Presented on Thursday 7 September at the NCVO Campaigning Conference 2017. Sarah Gilbert, head of campaigns and activism, Mencap Dima Alazi, monitoring and evaluation officer, Mencap Jennifer Pennington, senior research officer, Shelter Paul Donnelly, interim head of campaigns, Shelter If you would like to find out more about our training and events, visit our website at https://www.ncvo.org.uk/training-and-events.

1. Tech sponsor:
PRACTICAL 2: USING
EVALUATION TO KEEP YOUR
CAMPAIGN ON TRACK
CHAIR:
SARAH GILBERT
HEAD OF CAMPAIGNS AND ACTIVISM, MENCAP
SPEAKERS:
DIMA ALAZZI
MONITORING AND EVALUATION OFFICER, MENCAP
JENNY PENNINGTON
SENIOR RESEARCH OFFICER, SHELTER
PAUL DONNELLY
INTERIM HEAD OF CAMPAIGNS, SHELTER

2. Health Campaign
Impact & Evaluation Workshop
July 2017
Dima Alazzi
Impact & Evaluation, Mencap

3. Introduction

4. Impact
• The longer lasting changes, differences or benefits for the person we
support as a result of the support.
• It is long term
• Requires conditions to happen ( the Results Chain)
• Could be positive or negative
• Sometimes not changing a current situation is an impact
Change – Striking of one thing against the other – Results

5. Inputs Activity Output Outcome Impact
Process Results
• The resources we
use to create an
output
• Financial, human
resources, Technical
experience…etc
• Actions taken or
intervention
performed through
which inputs are
mobilised to
produce specific
outputs
• Conducted
training,
services…etc
• Direct
product/support
which are
delivered through
the
activity/interventio
n
• Staff trained,
PWLD knows how
to write CVs
• The achieved
effects of an
intervention’s
outputs
• Where the change
you want to see
starts happening
• Could be divided
into immediate,
intermediate and
long-term
outcomes
The longer lasting
changes,
differences or
benefits for the
person we
support as a
result of the
support.
The Results Chain

6. How does the Results Chain work?
Inputs Activity Output Outcome Impact
Act this way
Think this way
If this happens This will happen
This should
happen
For this to happen

7. More about planning for Impact…
• Theory of Change (TOC):
Specific type of methodology
for planning, participation,
and evaluation. It defines
long-term goals and then
maps backward to identify
necessary preconditions.
Shows causalities and links
between high level outcomes
Change/Impact
Long term outcomes Long term outcomes
Preconditions Preconditions

8. Narrative
description (The
Results Chain)
Target Indicator Data
Collectio
n tool
Respons
ible
function
Data
locati
on
Risks &
Assumption
s
Impact
Long term Impact
Intermediate
Impact
Immediate Impact
Outcome
Long-term
outcome
Intermediate
outcome
Immediate
Outcome
Output
Activity
• The Logical Framework
(Logframe)
• Tool/matrix for planning and measurement
• Illustrates a programme (implementation)
level understanding of the change
process.
• It is like a microscopic lens that zooms in
on a specific pathway within the TOC.

9. Target and Indicator
Target:
Mark, point shoot, figure... of what I want to
achieve
Indicator:
Tells me how close or far I am from achieving the
target

10. Monitoring, evaluation and Impact evaluation
Monitoring: The gathering of evidence to show what progress has been made in
the implementation of programs.
Evaluation: periodic, objective assessments of a planned, ongoing, or completed
project, program, or policy. Evaluations are used to answer specific questions
related to design, implementation, and results.
Impact Evaluation: a particular type of evaluation that seeks to answer cause-
and-effect questions. It looks for the changes in outcome that are directly
attributable to the program.

11. How to make sure we keep on the right track?
• Set clear objectives of the campaign (what is the change we
want to make?)
• Clear idea of how you will achieve the objectives (causal chain,
results chain)
• Outcome focused:
• Answer the question: What visible changes in behavior can be
expected among targeted audience as a result of the campaign, thus
validating the causal chain?
It would be great if we can do this at planning/preparation stage!
When planning…

12. • What?
• Type of information and data to be consolidated
• Why?
• How the collected data will support the campaign’s
monitoring and evaluation processes
Would this data:
tells me where my campaign is now and where it will be?
tells me whether I should change or consider new things in
my plan?
be important to any of the campaign’ stakeholders?
And of course, DATA…but what about data!?

13. • How?
• Procedures and approaches including methods for data
collection and analysis
• When?
• Frequency of data collection and reporting
• Who?
• Focal points, resource persons and responsibilities

14. If we have a clear TOC/logframe from the
beginning, then answering the previous
questions become

15. If we have a clear TOC/logframe from the
beginning, then answering the previous
questions become

17. Shelter – Evaluating Active Campaigns

18. 1. The background to the
campaign
2. The role of research and
evaluation
3. Our use of research and
evaluation in the campaign
Contents

19. Why we launched the campaign

21. Add map from benefit cap
The impact of the lowered benefit cap
Areas where
family with two
children in a two
bedroom home
would have
housing benefit
capped
Cap pre 2016 Cap post 2016

22. One part of our response: the
‘Neutralising Attitudes to Welfare’
campaign

23. • Aim to soften public attitudes to
welfare, so the Government feel
less emboldened to make
further cuts
• We’ve run two pilot bursts of the
campaign (Autumn 2016,
Spring 2017)
• We are now looking to share the
results of these in order to
assist others
Shameless event plug – Morning
of the September 13th
‘Neutralising Attitudes to Welfare’

24. The role of research and evaluation
through the campaign

25. The role of research
Before During
Understanding
drivers of attitudes
Audience insight Formative
evaluations
Analysis of
campaign data
Historical and
international
comparative
research
Audience selection
and profiling
Quantitative testing Facebook analytics
Qualitative
research into
emotional drivers
Campaign
materials testing
Focus groups

26. Main takeaways
Making people feel in
need of support can
make people feel
more in favour of cuts
Changing policy
doesn’t necessarily
change attitudes
Highlighting
‘deserving cases’
reminds people of
‘undeserving cases’
1. Understanding drivers of attitudes
Takeaways for campaign design
Tread carefully – you could make
things worse
Don’t make people fearful
Focus on the ‘missing middle’

27. Campaign development

28. Main takeaways
Reading about others
struggling reminds
them of own
difficulties
Welfare helps you to
live your life
2. Audience insight
Main takeaways for campaign
design
Keep the campaign optimistic,
dwelling on opportunity and
aspiration
Show people actively doing tasks
rather than passive

29. Campaign launch!

30. • We ran Facebook adverts for 6 weeks and measured
conversion rate
• This allowed us to tweak the campaigns in real time:
• We changed the copy to make it more ‘clickbaity’
• We changed the headline, call to action and body copy to be more
direct (i.e. ‘click here to read X’s story)
• We used this information to inform our choice of out of home
and direct mail advertising. We picked the case study that had
performed best after 2 weeks digitally.
3. Digital evaluations – during campaign

31. 4. Formative evaluation I
Not clear exactly
what situation the
person was in
She looks a bit
‘downtrodden’
Could look more
dynamic
Like the sentiment
71% of people agreed the situation was realistic
50% of people said they could relate to the situation
• Show a diverse range of people
• We need to include a campaign
action
• The image of the person needs
to be more dynamic
• Shelter branding makes the
audience trust us more

32. Campaign refinement

33. 5. Formative evaluation II
• The call to action needs to be
clearer
• It needs to be made clearer who
Shelter are/what we do – this
could link into a wider brand
awareness piece of work
• Visuals could be diversified –
i.e. showing more images of the
same person to reflect their
day-to-day lives
33%-47% people (depending on the case study) who weren’t
opposed to further cuts said the campaign made them agree we
need to improve the welfare system
“‘Even the strongest of
us’ sticks in your head”
“You want them to
be happy for the
help, not
complacent.”

34. Event – morning of the 13th September
Chat to us afterwards if you would like to know more
Campaign reflection and information
sharing

35. Thank you!