Introduction to IEA DSM Task 24 on behaviour change
1. IEA DSM Task 24
Phase I
Closing the Loop – Behaviour Change in DSM: From
Theory to Practice
Dr Sea Rotmann
Operating Agent Task 24
Toronto, May 28, 2015
2. What is special about Task 24?
For more information, visit www.ieadsm.org
3. Some numbers of Task 24
For more information, visit www.ieadsm.org
• July 2012 – April 2015: Official start and end dates
• 8 participating countries: Netherlands, New Zealand, Sweden, Norway,
Switzerland, Belgium, Italy, Austria
• 9 countries gave in-kind (expert) support: the UK, Spain, Portugal, UAE,
France, Australia, South Africa, Canada and the US.
• 228 behaviour change and DSM experts from 21 countries participate in
Subtask 5, the invite-only Task 24 Expert Platform
(www.ieadsmtask24.ning.com)
• 15 successful expert workshops/webinars have been held to date
• 137 videos and presentations of these events on the Expert Platform
• Over 35 publications have been created and disseminated
• Almost 60 case studies showing the successful (or not so successful) use of
diverse models of understanding behaviour in the areas of transport, SMEs,
smart meters and building retrofits have been collected to date from 16
countries in a Wiki (www.ieadsmtask24wiki.info)
4. Our Audience: ‘Behaviour
Changers’ from:
- Government
- Industry
- Researchers
- The Third Sector
- Intermediaries
For more information, visit www.ieadsm.org
5. Subtasks of Task 24
5 – Social network and expert platform
1 –
Helicopter
view of
models,
frameworks,
contexts and
evaluation
metrics
2 –
In-depth
case study
analysis
3 –
Evaluation
Tool for
different
stakeholders
4 –
Country-
specific
to do’s and
not to do’s,
guidelines
and
recommenda
tions
For more information, visit www.ieadsm.org
6. Subtasks
For more information, visit www.ieadsm.org
Subtask 1 –
Helicopter Overview of different
models of understanding,
frameworks, contexts, case
studies and evaluation metrics
7. The Story of Task 24
For more information, visit www.ieadsm.org
8. Subtask 1 - Definitions of Task 24
For more information, visit www.ieadsm.org
http://www.slideshare.net/drsea/definitions-for-task-24
9. Subtask 1 – What is behaviour?
For more information, visit www.ieadsm.org
Energy behaviour refers to all human actions that affect the way that fuels
(electricity, gas, petroleum, coal, etc) are used to achieve desired
services, including the acquisition or disposal of energy-related
technologies and materials, the ways in which these are used, and the
mental processes that relate to these actions.
Behaviour Change in the context of this Task thus refers to any changes
in said human actions which were directly or indirectly influenced by a
variety of interventions (e.g. legislation, regulation, incentives, subsidies,
information campaigns, peer pressure etc.) aimed at fulfilling specific
behaviour change outcomes. These outcomes can include any changes
in energy efficiency, total energy consumption, energy technology uptake
or demand management but should be identified and specified by the
Behaviour Changer designing the intervention for the purpose of outcome
evaluation.
10. Subtask 1 – What is behaviour?
For more information, visit www.ieadsm.org
persistence
“unfrozen”
half-yearlyyearly
Conscious, or well-considered action
Once in a lifetime
Active information-seeking
monthlyrarely
Little information-seeking
Hardly thinking – taking action
Habitualised routinesOnce-off
“frozen”
consciousness
frequency
weekly daily
cookinggroceriesholidayingChoosing
energy supplier
Buying a
car
Buying a
house
11. Subtask 1 –
Overview of different models of understanding behaviour
For more information, visit www.ieadsm.org
12. Subtask 1 – The ‘Monster’ and its Wiki
For more information, visit www.ieadsm.org
13. Subtask 1 – More definitions
For more information, visit www.ieadsm.org
Models of behaviour help us to understand specific behaviours,
by identifying the underlying factors which influence them.
There are individualistic models and social models.
By contrast, theories of change show how behaviours change
over time, and how they can be changed.
Behavioural theory is diagnostic, and change theory is more
pragmatic.
Both are important to understand when designing
interventions!
14. Subtask 1 –
Models of Understanding Behaviour – some caution
For more information, visit www.ieadsm.org
Models are concepts, not representations of
behaviour
Behaviour is complex, models are deliberately
simple
There is a limit to how far models will stretch
Models don’t tend to differentiate between
people
Attitudes/awareness don’t always precede
behaviour
Factors are not barriers
15. Subtask 1 –
Looking at different models of understanding behaviour
For more information, visit www.ieadsm.org
16. Subtask 1 – Main models of understanding behaviour
For more information, visit www.ieadsm.org
INDIVIDUALISTIC (A-B-C Models)
Rational choice models based on cost-benefit calculations (neoclassical
economics)
Information deficit models are based on linear assumptions: information
generates knowledge, which shapes attitudes, which lead to behaviour
(neoclassical economics)
Bounded rationality models include psychological principles such as
cognitive biases and environmental constraints (behavioural economics)
Value Action Gap shows the difference of what people say and what they
do (social psychology)
17. Subtask 1 – Main models of understanding behaviour
For more information, visit www.ieadsm.org
INDIVIDUALISTIC (A-B-C Models)
Figure 3: The ABC model based on Shove 2010
However, in practice people usually make more complex trade-offs between costs and gains (both
financial and non- financial) and, consequently, the models are most likely not accurate. In Paul
attitudes and values
influence:
behaviour
and people chose to
behave a certain
way based on these
values and attitudes
18. Subtask 1 – Main models of understanding behaviour
For more information, visit www.ieadsm.org
INDIVIDUALISTIC (A-B-C Models) DUAL PROCESS Models of Cognition
Triandis’ Theory of Interpersonal Behaviour (TIB)
19. Subtask 1 – Main models of understanding behaviour
For more information, visit www.ieadsm.org
SOCIALLY-ORIENTED MODELS
Theories of Consumption as Social Practices (Practice Theory)
DECC running header
24
20. Subtask 1 – Practice Theory – worked example
For more information, visit www.ieadsm.org
54
Figure 6: 3 Elements Worked Example: Linedrying
21. Subtask 1 – Theories of Change
For more information, visit www.ieadsm.org
Central to many concepts of change is the merging of theory
and practice
Applied approaches: Social Marketing, Intervention Mapping,
Defra’s 4E Model…
22. Subtask 1 – Theories of Change – Changing habits
For more information, visit www.ieadsm.org
Unfreezing/Refreezing
Vigilant Monitoring
Implementation Intentions
But: Individuals only, they need to be pre-motivated, it
needs to be done quickly and intensely and they may not
be easily scalable
MOMENTS OF CHANGE!
23. Subtask 1 – Comparison between indvidual and social
approaches
For more information, visit www.ieadsm.org
Darnton, A, Verplanken, B, White, P and Whitmarsh, L (2011). Habits, Routines and Sustainable Lifestyles: A summary report to the
Department for Environment, Food and Rural Affairs. AD Research & Analysis for Defra, London.
24. Subtask 1 – Comparison between indvidual and social
approaches – Pros and Cons of each
For more information, visit www.ieadsm.org
Individual Models Social Models
Pros Cons Pros Cons
Some have
understanding of dual
process of cognition
Easy to follow A+B+C=
behaviour change
Can look at various
(mostly influencing)
contexts affecting
individuals
Known and tested
Very powerful with
segmentation and
bottom-up tailoring
Scaleability
Inclusivity
Breadth of Scope
Causal relationship hard
to determine
Not shown to be that
effective, especially if
based on intentions
More complex models
hard to use
Takes systemic approach
thus easily scaled up
If you change a practice,
it can be a global change
Looped, re-enforcing
Influencing and
contextual factors
Fosters collaboration
among all sectors
More realistic?
Too complex to
understand
Dependent on many
elements to work
together
Frustrating if right
collaboration can’t be
fostered
Hard to put into practice
May only speed up
change
25. Subtask 1 – Main differences between disciplines
For more information, visit www.ieadsm.org
The programmes based (explicitly and implicitly) on economic theories usually
translate into approaches that:
- focus mainly or even solely on individuals
- are mostly technocratic thus seem to be generating biggest benefits for the supply
side, not the end user
- regard individuals as instrumentally/economically rational creatures (‘Homo
economicus’)
- regard information deficits as an important cause of ‘non-rational’ behaviours
- focus often on short and one-off financial incentives
- focus on extrinsic motivations mainly (ie are dependent on the response they evoke
from others)
- do not normally tailor their approach to the individual characteristics
- lack flexibility and room for engagement, co-creation and participation
- monitor mainly quantitative aspects and work with calculated or modeled savings
Behavioural economics-based approaches also include insights from social
psychology, and for instance focus on the power of nudging people into different
behaviours through their infrastructural, institutional or design environment.
26. Subtask 1 – Main differences between disciplines
For more information, visit www.ieadsm.org
Social marketing, or insights from psychology, sociology and collaborative learning
and practice theory approaches are increasingly being used.
These programmes are often cross-sectoral and use elements of theories and models in
an eclectic manner. Very often, user engagement is central to the design. They do take
account of the impact of the wider context and environment and social norms and are
thus clearly based on a more systemic perspective/theory or model. They:
- focus on collaboration and institutional capacity building
- focus on building trust in market parties and information sources
- target end user needs and multiple benefits
- use multiple definitions of success
- perform pre-scoping
- allow for engagement and participation
- allow for flexibility and iteration of programmes
- focus on institutional change
- focus on lifestyles
- use the power of social norms
27. Language can be a problem!
For more information, visit www.ieadsm.org
28. That was our Eureka!
moment
For more information, visit www.ieadsm.org
29. We’re all expert story tellers
YOUTUBE:
https://www.youtube.com/watch?v=wbe83S8FfO0&list=U
U_p3PlWDpLyDBh8TwUBmVHQ
30. The art and scientific methodology of storytelling
Narratives = social science tool aimed at providing way to
explore how big events (policies) impact on small scale
(individuals)
Allow for quick, practical and useful understanding of
complexity of interconnected factors in behaviour research
We all turn everything into a narrative in order to remember
it
For more information, visit www.ieadsm.org
31. Subtask 1 – Some ‘Monster’ findings
For more information, visit www.ieadsm.org
Each of the domains of Task 24 also had some unique story lines, eg:
Transport: driving is a very routine behaviour, with built-in capacity
for adaptation/adoption to new cars/routes/traffic;
Buildings: retrofitting-related behaviour deals with investment
decisions at the planning stage where unappealing new behaviours
can be quickly rejected or even result in cancelling a planned action.
Smart metering: many cases demonstrate that this domain deals
with an entirely new behaviour, presenting opportunities for impact
through training and feedback but also almost no existing
behavioural context to use as a starting point.
SMEs: many of the behaviours that need to change require a lot of
risk taking and senior leadership, with potentially big impacts on
staff and productivity.
32. Subtask 1 – Some ‘Monster’ findings
For more information, visit www.ieadsm.org
Each of the domains also had some highlights, eg:
Buildings: Warm Up NZ programme for its use of intermediaries
and strong stakeholder engagement, as well as evaluation focus on
health improvements as main metric and Swedish Sustainable
Järva Project
Transport: NZ Post’s Driver Behaviour Training for its use of
trusted trainers and the Dutch Spitsmijden congestion avoidance
pilot
Smart metering: NOT working were time of use tariffs (NZ and
Italy) and solely economic incentives (though NLs Jouw Energie
Moment had a more systemic approach); the Swedish Energy
AWARE Clock was a great example of using smart design in
providing better feedback
SMEs: EECA’s Crown Loan Subsidy was thought to be great once
it learnt to use trusted intermediaries and shared learning. A good
example of nudging was shown in the Belgian Build4Change case.
33. Subtask 1 – Sustainable Järva (Building Retrofits)
For more information, visit www.ieadsm.org
Once upon a time… there were 6 neighbourhoods around the field of Järva that were in
urgent need of improvement. They were constructed in the 60s as part of the 1million Home
programme to tackle a growing housing deficit in urban areas in Sweden. They contained
housing units for more than 60000 people, but times had changed a lot since then…
Every day… People in the area were experiencing economic and social challenges. Many of
the foreign residents were unemployed and struggling with the Swedish language, and youth
was lacking good opportunities for education. The houses were terribly inefficient and the
area in general did not work for the needs of its residents. Several investments had been
undertaken but nothing worked and people felt no one was listening to them.
But, one day… the City of Stockholm decided to improve the living conditions once and for
all. But this time would be different, this time they realised that circumstances were radically
different to the 60s and that, in order for upgrading the area successfully, they needed to
involve the residents. From the beginning.
Because of that… the Järva dialogue was initiated during the Fall 2009 and for one week
10000 residents left over 30000 opinions and suggestions how the area should be developed
and improved. Based on their contribution a vision was formulated and measures were
planned in 4 areas: 1) improved housing and urban environment, 2) everyday security, 3)
better education and language teaching, 4) more jobs and entrepreneurship.
But then… it was also realised that the area and the buildings had been constructed before
the energy crisis without considering the environment, and thus the project Sustainable Järva
was born to include an energy, environment and climate-focus to the vision.
Until, finally… the dialogue with the residents continued and together with all stakeholders
many great measures were planned to promote sustainable lifestyles, satisfaction and well-
being. The ultimate goal with the project was to serve as a model for sustainable
development.
And, ever since then… the neighbourhoods around the field of Järva have become a place
where people want to live, work and play. The end.
34. Subtask 1 – Spitsmijden(Transport)
For more information, visit www.ieadsm.org
Once upon a time… in a small country with many cars, enthusiastic, hardworking Dutch
people left for work every morning around 9am, five days of the week, to be returning home
just as eagerly around 5pm.
Every day… they would bore themselves to death driving in peak traffic every morning and
afternoon. Such a waste of time, that could otherwise be spent on making wooden clogs
and picking tulips, the favourite past-time of any Dutch(wo)man, as you know.
But, one day… a cooperation between universities, government and business started a
project called Spitsmijden (congestion avoidance). Two types of incentives were used to
achieve this change in behaviour: a price incentive for every avoided drive in the city and
information supply (feedback) through a hand computer in the form of navigation and
suggestions for other modes of transport.
Because of that… several pilots in the Netherlands were set up to see if it would help
people in avoiding rush hour. And indeed, people were tempted: 4 months after the pilots,
when the financial incentive was gone, 47% of the participants were still avoiding rush hour
one way or another.
But then… the initiators came up with an additional incentive: the personal avoidance plan.
People were told to plan their congestion avoidance behaviour, using a scheme of when
and how the behaviour would be conducted. A theory of Cialdini states that when a person
commits herself to something, (s)he will be inclined to be consistent with that commitment.
But it was uncertain if this theory would fly in light of driving behaviour.
Until, finally… the results showed that the avoidance plan indeed caused an additional
effect on rush hour avoidance behaviour: there were now 27% more people avoiding rush
hour than without the avoidance plan!
And, ever since then… former participants are still avoiding traffic jams. Although the %
slightly dropped after the project ended, an increasing number of Dutchmen now have more
time to do what they love best: making wooden clogs and picking tulips. The end.
35. The story of neoclassical economics in building retrofits
www.ieadsm.org
Money makes the world go round!
You need to change your home’s energy use and we will help you pay (part of) its
retrofitting
By the way, you need to pay up first and it might take a while before we pay you back, if
ever
The info we need from you will teach you all you need to know
You only need to make a one-off decision to invest
We have the technology you need, contractors or installers (you will need to find/choose)
will put it in
If you don’t understand the technology just don’t touch the buttons!
You will save money for a nice weekend in Marbella
You only need to give us a bill from your installer, we probably won’t check how much
energy you saved
Neither will we tell you, you need to figure that out yourself
What counts for us is how many m2 we get insulated, how many homes we retrofitted or
how much money has been spent against the budget.
We will do the number crunching, don’t worry, we don’t need to know what you actually
saved, that’s what national models are for
But if you do want to know how much energy you saved, buy a metering device!
36. The story of systemic approaches in building retrofits
www.ieadsm.org
Together we’ll make the world go round!
We will co-create and co-design our interventions with you
You embody what we need to know and change: what you do, feel, learn…
We will help you understand and use the technology and train those that install and
sell it to you to tailor it to your needs
We will create a supportive material, institutional and social environment
Your needs are important so we need to do this together, as if this were your kitchen
Your life will change
It’s all about us now, our grandchildren and their future we have in our hands
Quality matters, and we will keep learning and sharing those learnings with you
If we need to be flexible, we will
This is only the start and your home is only the first step
We will monitor, calculate and report on energy, money, health, welfare, comfort,
wellbeing
And learnings based on qualitative and quantitative inputs will be shared (with you)
We will help you figure out what your impact is to be able to make sure you get where
we all collectively want to!
37. The pros and cons of each approach
• They do well with what they intend to do
and fit well within the current economic
and political system and way of thinking
• The programmes are relatively easy to
evaluate in quantitative terms and often
show good results
• The (retrofitting) market can grow
• Subsidies are often used up to the max
• Many homes do get insulated
• Behavioural economics does manage to
nudge a certain percentage
• Free riders upgrade their plans and
retrofit more comprehensively
• Sometimes even a new norm seems to
be emerging…
• These types of interventions are very
complex with many partners who have
different mandates, needs and
restrictions
• They cannot be driven by policy alone,
need all levels collaborating
• Not everyone wants to change
everything or their lifestyle
• Not everyone wants to engage but it is
important to ensure that the naysayers
are not becoming the over-riding voice
• The flexibility of changing goals, aims
and interrelatedness of issues etc makes
it difficult to evaluate
But people tend to like them much
more!
For more information, visit www.ieadsm.org
Economic approaches Systemic approaches
38. Different energy efficiency stories
*See Janda &Topouzi (2013). Closing the Loop: Using Hero Stories and Learning Stories to Remake Energy Policy ECEEE Summer Study
Proceedings.
39. Examples of a love and a horror story in Building Retrofits
www.ieadsm.org
40. The New Zealand love story with insulation
www.ieadsm.org
Once upon a time... there was a beautiful country called New Zealand,which had very
cold, damp houses.
Every day...Kiwis shivered and froze, but they just told each other to stop being a sissy
and put on another jumper.
But, one day...the new right-wing Government decided it needed to show it wasn’t
uncaring and evil and created a programme called Warm Up New Zealand. It was meant
to insulate a quarter of the housing stock, create many jobs and a new market, and
reduce energy use, energy bills and CO2.
Because of that...the Energy Efficiency and Conservation Authority set about tendering
for the best contractors in the country to fulfil this lofty goal.
But then...they realised that people weren’t that interested in insulation, they rather spent
their money on a new kitchen and kept putting on those jumpers!
Because of that... they concentrated on using Third Party Providers and other community
groups to ensure that at least the most needy and vulnerable people got free insulation
and clean heating installed.
So, finally... they did an evaluation and found that the real benefits - $5 for every $1
spent, lay in the health improvements, not a new market or energy savings or lower bills.
And, ever since then... The other Kiwis also slowly realised that being warm and cozy in
your home was maybe just as important as having a new kitchen. The End.
42. What’s the morale of the story?
www.ieadsm.org
A mix of interventions that are tailored to different
(national, local, organisational, domestic) levels; tailored at
both the individual and social level; aiming at changing
both the investment and habitual behaviour; targeting
multiple motivations (not only economic and informational
ones); adding strong quantitative and qualitative evaluation
(of actual and perceived/modelled behaviour changes) into
project design; making sure that intermediaries are well-
trained and customer-focused; and focusing on the
lifestyle in which energy is key to performing functions will
probably get you long-term success.
43. The main lessons of Subtask 1
www.ieadsm.org
A helicopter overview of models of understanding behaviour
is/not:
A great way to provide a general understanding/overview
Good representation of different models/disciplines
Collection of international best practice (but snapshot)
Good storytelling platform
Creating a ‘monster’ that keeps on growing
X An in-depth review of all literature on the subject
X Adhering to a ‘subjective’ disciplinary or sectoral view
X Easy to read, hence the Wiki and ST8 decision-making tool
It will continue to grow as a ‘living monster’ on the Wiki
throughout the Task 24 extension (and ST8 tool)
44. Subtasks
For more information, visit www.ieadsm.org
Subtask 2 –
In depth analysis in areas of greatest
need
(buildings, transport, SMEs, smart
metering)
45. Subtask 2 – Norwegian Finnfjord Case
For more information, visit www.ieadsm.org
46. Subtask 2 – Norwegian Finnfjord Case
For more information, visit www.ieadsm.org
Duration 2007-13
Country Norway (Finnsnes)
Type Project
Cost Initially: NOK850m (€108m), incl ENOVA grant of
NOK175m (€22m)
Overrun: NOK100m (€16m)
Specification/Goal Become world’s first carbon neutral smelter by
installing innovative heat recovery system and
other energy efficiency measures
Behavioural model/s or theories of
change
Not individual, but organisational culture
Rogers diffusion of innovation
Context Almost 100% RES, subsidies for intensive industry,
national policy on EE including national funding that
takes on some risk, small family-owned business
47. Subtask 2 – Norwegian Finnfjord Case
For more information, visit www.ieadsm.org
Becoming the world’s first carbon neutral ferro-silicon smelter is/not:
A massive goal that needs buy-in from all levels
Proof that company culture is everything
That a performative vision is essential
A technologically innovative showcase/best practice
A forerunner that breaks down hurdles for fast followers (replicable)
Possible because of strong CBA (barriers/drivers)
X Easy, cheap or straightforward
X Without a very large level of shared risk
X Possible without behavioural change in management
48. Subtask 2 – Austria’s Smart metering Cases
€CO2 Management
For more information, visit www.ieadsm.org
49. Subtask 2 – Austria’s Smart metering Cases
Die Energiejagd (the Energy Hunt)
For more information, visit www.ieadsm.org
50. Subtask 2 – Austria’s Smart metering Cases
For more information, visit www.ieadsm.org
social approach individualistic approach
social norm (MoU)social learning (ToC)Freezing/unfreezing (ToC)
classical economics (MoU)
Gamification, competition, feedback,
tailored advice, champions
Feedback, Advice & Incentive (iPod!)
Goal: CO2 savings
Huge success Unexpected failure
51. Subtask 2 – Main lessons
For more information, visit www.ieadsm.org
An in-depth review of case studies is/not:
A great way to drill deeper into most interesting cases
Showing impact of country-specific contexts
Providing some comparison between cases and countries
A way of standardising the analysis across countries
Important to collect different points of view
X In-depth as it focuses only on one issue per country
X Quantitative analysis, as it is empirical research by nature
X Available to countries that provided in-kind expertise only
We will continue to collect case studies for ST6 of the
extension
53. Subtask 3 – Evaluation Tool, definitions
For more information, visit www.ieadsm.org
What is it?
• Monitoring: measuring progress and achievements and production
of planned outputs
• Evaluation: structured process of assessing success in meeting
goals and reflect on learnings. Explicitly places a value judgement
on the data and information gathered in an intervention
Why do it the way we do now?
Establish effect of policies
Assess need for improvements
Assessing value for money
Contribution to evidence base for effectiveness of behavioural
interventions at population level
How to do it…….???
54. Subtask 3 – Evaluation Tool, outputs
For more information, visit www.ieadsm.org
- Individual evaluation and monitoring metrics for each domain can be found in the
Subtask 1 Monster/Wiki
- Subtask 3 Deliverable 3 - Methodological review of the scientific literature (smart
meter/feedback and building retrofits only) called ‘What do we know about what we
know?’ which will feed into Subtask 9
- An overview of how different disciplines evaluate behaviour, main challenges and
recommendations on monitoring and evaluation can be found in Subtask 3
Deliverable 3A report ‘Did you behave as we designed you to?’
- Specific guidelines and fact sheets for 3 main intervention tools in the building
retrofit area (Energy Performance Certificates, Mass Marketing and Subsidies and
Loans) can be found in Subtask 3 Deliverable 3B From “I think I know” to “I
understand what you did and why you did it”
55. Subtask 3 – Deliverable 3 ‘What do we know about
what we know?’
For more information, visit www.ieadsm.org
- Methodological review of behaviour-based energy intervention studies in the
customer feedback and residential building retrofit areas, which were
conducted over the past 10 years to determine what data has been collected
and how it has been collected (out of 315 papers, 85 were coded in detail for
analysis).
- No standard way of measuring behaviour change, which means no ability to
compare across studies and incorporate questions about context, attitudes,
knowledge and user experience.
- In future we should make better use of mixed methods for data collection, eg
surveys, focus groups, interviews, scales to allow for triangulation.
- Also need better transparency into the methods used to evaluate (only 4 out
of 85 published their actual evaluation instrument).
- Need to create and share validated data collection instruments which facilitate
a consistency of measurement
This will be done in Subtask 9
56. Subtask 3 – Deliverable 3A ‘Did you behave as we
designed you to?’
For more information, visit www.ieadsm.org
Positioning Paper providing an overview of:
- Definitions used in Task 24, particularly around monitoring and
evaluation (M&E)
- Evaluating efficiency and effectiveness of behaviour interventions
- Disciplinary basis for interventions and consequences on M&E
- Multiple challenges of M&E (benchmarking, mismatch of needs,
M&E
team not included in design, no longitudinal M&E, based on proxies
and models not actual measures, multiple benefit analyses, how do
other stakeholders assess success, monitoring individuals not
practices, no feedback loops, no shared learning)
- How to open up interventions to include end users
- A collective learning process: single vs double-loop learning
57. Subtask 3 – Deliverable 3A ‘Did you behave as we
designed you to?’
For more information, visit www.ieadsm.org
Single-loop learning is about the effectiveness and/or efficiency of a technology,
measure, instrument, arrangement, or intervention to achieve pre-defined goals.
Double-loop learning is process-oriented, focused on the how, when, where, how,
how long, for whom and is about questioning goals and the prevailing norms and
rules underlying these goals. In addition, double-loop learning is focused on
interactions, the quality of participation, learning by doing and doing by learning,
aligning expectations, in short, double-loop learning is about reflexive governance of
interventions.
58. Subtask 3 – Deliverable 3B From “I think I know” to
“I understand what you did and why you did it?
For more information, visit www.ieadsm.org
Guidelines and Factsheets in Building Retrofits:
- Identification and development of context-sensitive indicators,
metrics and ways to monitor and evaluate both short- and long-term,
identifiable and/or measurable (one-off investment- and more
frequent habitual) behaviour change outcomes of DSM tools (being
elements of larger interventions)
- Focus on investment vs habitual behaviours
- Examining different tools of building retrofit interventions
- Detailed factsheets of M&E in single- and double-loop learning
processes of three tools (Energy Performance Certificates, Mass
Marketing and Subsidies and Loans)
This work will be continued in Subtask 8
59. Subtask 3 – Deliverable 3B From “I think I know” to
“I understand what you did and why you did it?
For more information, visit www.ieadsm.org
60. Subtask 3 – Main lessons
For more information, visit www.ieadsm.org
A behavioural evaluation tool for stakeholders is/not:
Something everyone wants but no one has created yet
Important, as it is the only way to show impact and compare between
studies
Usually dependent on models and estimates, not measures
Collection of different metrics ‘beyond kWh’
Methodological review of behavioural interventions
Positioning paper recommending double-loop learning
Guidelines and factsheet examples of building intervention tools
X Possible to complete in the scope of Task 24’s Phase I
X Easy, as different Behaviour Changers have different needs/outcomes
X Finished, but created important building blocks
Will be developed as ST9 and become part of ST8 toolbox
61. Subtasks
For more information, visit www.ieadsm.org
Subtask 4 –
Country-specific recommendations,
to do’s and not to do’s
62. Subtask 4 – Do’s and
don’t’s
For more information, visit www.ieadsm.org
Intervention Phase DO DON'T
DESIGN PHASE use models of understanding behaviour and
theories of change to design interventions
spend some time pre-intervention researching
your audience, its motivations, needs and
heterogeneity
collaborate with other Behaviour Changers,
especially researchers and intermediaries to
design your interventions
segment your audience where you can as it
will help tailor the intervention
design evaluation into the intervention up
front, including the evaluation team (if
different)
learn from mistakes and (re)iterate your
intervention
put a lot of thought into dissemination and
don't be afraid to use unusual means like
social media, group learning and storytelling
believe that there is one silver bullet model for
behaviour change
always use the same model, neoclassical
economics is a valid model that fits our socio-
economic and political reality but it does not
explain peoples' mostly habitual energy-using
behaviour well enough
be afraid to mix models and create a toolbox of
interventions
think you can design, implement, evaluate and
disseminate a (national) behaviour change
programme all by yourself
think all people are rational, utility-maximising
automatons, even in each household you will find
very different attitudes, behaviours and motivations
think you can leave evaluation til after the
programme is finished
just think in kWh and cost savings, most people
don't think of energy in this way but of the services
they derive from it
IMPLEMENTATION
PHASE
collaborate with other behaviour changers in
rolling out the intervention
use trusted intermediaries and messengers
target your audience with tailored information
and feedback that makes sense to them
keep learning during the implementation by
evaluating ex durante
listen to peoples' stories and especially the
nay-sayers and laggards
not underestimate the power of moments of
change, use them wisely
operate in a silo, you need help
stop looking in unusal places for allies
let your (conflicting) mandates stop you from
working with other Behaviour Changers
let technology overwhelm the intervention, it is a
means to an end
ever forget that you are dealing with people and
their homes are their castles and their cars their
steeds
think you know better than your audience how they
should use energy
keep a successful intervention to yourself, share it
widely
EVALUATION PHASE evaluate ex ante, ex durante and ex post
put 10-15% of your resources into evaluation,
it's worth it
benchmark!
think of the most relevant metrics and
indicators, not just for you but for your target
audience and the other Behaviour Changers
use double-loop learning methods
provide strong, ongoing, targeted feedback to
your audience
think it's just about kWh, evaluate beyond it (eg
health, comfort, safety...)
think you need to do all evaluation yourself, use
your collaborators to evaluate the bits they know
best
leave evaluation til the end or ignore its importance
in showing that your intervention worked
just model, measure as well
ignore the pathway of behaviour change that led to
a kWh change – ask people
(RE)-ITERATION PHASE (re)iterate your intervention often
learn from your mistakes
listen to your collaborators and end users
ignore your evaluation
hide your mistakes and horror storries, they are
often the ones we can learn the most from
DISSEMINATION PHASE understand your audience, collaborators and
stakeholders, tailor your dissemination
accordingly
tell stories, use social media and word of
mouth
use trusted intermediaries to tell your story
spend all your money on (social) marketing
campaigns
keep doing the same thing, peoples' willingness or
brand awareness doesn't usually translate to
behaviour change
tell a boring story about kWh
think you know better, ever
63. Subtask 4 – Country context in form of stories
For more information, visit www.ieadsm.org
64. Subtask 4 – Summary of recommendations
For more information, visit www.ieadsm.org
- Design Phase (Subtask 1) Q: What are the best models and
theories to underpin intervention design? A: It depends on the
intervention, there is no silver bullet.
- Intervention Phase (Subtask 2) Q: What can we learn from best
practice in Task 24 countries? A: Context is everything, but there is
opportunity for shared learning.
- Evaluation Phase (Subtask 3) Q: How shall we monitor and
evaluate behaviour change outcomes? A: By using double-loop
learning processes and standardised data collection.
- (Re)iteration Phase (Subtask 4) Q: What can we do better? A:
Some things we do well, others can be improved on a lot.
- Dissemination Phase (Subtask 5) Q: How can we best share our
learnings? A: Close network of experts, building on relationships and
storytelling.
65. Subtask 4 – Reiteration of case studies based on
recommendations from ST1 analysis
For more information, visit www.ieadsm.org
66. Subtask 4 – Pilots and research questions
For more information, visit www.ieadsm.org
Building Retrofits: How to deal with the Split
Incentives/Principal Agent issue in rental properties?
SMEs: How to deal with the Split Incentives/Principal Agent
issue in a chosen SME segment?
Smart Metering/Feedback: How to link smart meters to
better feedback, using ICT?
Transport: How to get people out of their cars and into
healthier and/or more environmentally-friendly modes of
transport?
67. Subtask 4 – Main lessons
For more information, visit www.ieadsm.org
A country-specific list of recommendations is/not:
A major drawcard of Task 24
A collection of country-specific contexts, based on stories
Different for the different countries
But has some similarities, and overall conclusions
Based on country experts’ knowledge and stakeholder analyses
A detailed summary of each country’s involvement in Task 24
X Conclusive
X Entirely objective, may miss some sector and disciplinary views
X Available to countries that are not financially participating
Recommendations will form basis for Subtasks 6 and 7
69. Subtask 5 – Join our Expert Platform
www.ieadsmtask24.ning.com
For more information, visit www.ieadsm.org
230 members
137 videos and presentations
115 photos
6 blogs
21 Events
21 Discussion Fora
70. Subtask 5 – Main lessons
For more information, visit www.ieadsm.org
A social media platform is/not:
A good place to collect experts and information about the Task
A good broadcasting tool
A good way of measuring impact
X A silver bullet for making people talk
X A way of making busy experts use social media
X A way of easily managing files
We created a Wiki to make case studies more accessible, will
continue this Subtask during Phase II
71. Our main methodology– Storytelling
For more information, visit www.ieadsm.org
Storytelling is/not:
A valid social science tool
A great way to break down silos and jargon
Something we all innately do, and do well
Fun, engaging, social and importantly: memorable
Universally understood
A way to reduce bias by removing complexity?
X A way of getting around ‘proper’ analysis
We will continue to use narratives and investigate ways of
measuring the impact of storytelling
72. So… what’s the story?
For more information, visit www.ieadsm.org
• There is no silver bullet anywhere but the potential remains huge
• Homo economicus doesn’t exist (in energy)
• Habits are the most difficult thing to break
• This means we have to get even smarter & embrace complexity
• We are at a crossroads, and shouldn’t turn back
• We need to look at whole-system, societal change
• This can’t be done in isolation by one sector - collaboration is key
• Social media and networks are really good (theoretically) for it
• But: professionals are weary to use them, face-to-face still key
• It’s also hard to find the right Behaviour Changers and break down the
silos
• Everyone has a piece of the puzzle but we haven’t fit it together
• We need a shared learning and collaboration platform that works
• We also need a shared language based on narratives
It’s all about the people!
73. IEA DSM Task 24
Phase II
Helping the Behaviour Changers
74. The Subtasks of Phase II
5 – Expert Platform (upgraded)
6 –
Understanding
Behaviour
Changer
Practices inTop
DSM Areas
‘The Issues’
7 –
Identifying
Behaviour
Changers in
these areas
‘The People’
8 –
Developing a
toolbox of
interventions
to help
Behaviour
Changers
‘TheTools’
9 –
Standardising
Evaluation
beyond kWh
‘The Measures’
For more information, visit www.ieadsm.org
10 –Telling an Overarching Story ‘The Story”
75. Task 24 – Phase II
Objective in a tweet (or two)
To develop, in collaboration with the Behaviour Changers, a
toolbox of interventions that works for their specific DSM
issues, contexts (sectoral and national), mandates and
needs. We also aim to extract cohesive, overarching themes
to tell a coherent international story.
76. Task 24 – Phase II
How it all fits together
What?
Subtask 6
‘The Issues’
Who?
Subtask 7
‘The People’
How?
Subtask 8
‘The Tools’
Why?
Subtask 9
‘The
Measure’
So
what?
Subtask 10
‘The Story’
Subtask 1
Subtask 2
Subtask 4 Subtask 5
Subtask 1
Subtask 4 Subtask 3
77. Thank you very much for your
attention!
Any comments or questions?
For more information, visit www.ieadsm.org