Delivered by Sally Bagwell and Lynn Simmonds NPC Resource Social Impact Seminars As part of the Cultural Commissioning Programme, New Philanthropy Capital (NPC) delivered a series of Social Impact Seminars aimed at arts and cultural organisations. These seminars took place in Birmingham, Leeds, Exeter, London and Peterborough between October and November 2016. Many arts and cultural organisations need to show how their work contributes to social outcomes. These may be outcomes required by public service commissioners or outcomes wanted by funders. These seminars aimed to help participants to identify, collect and interpret evidence which: • Could be used to inform and influence funders and commissioners of their social impact • Was realistic to collect, in keeping with the individuals and communities they work with • Focus on current best practice and make use of existing research. The Cultural Commissioning Programme runs until June 2016 and is funded by Arts Council England. It is delivered by NCVO in partnership with NPC (New Philanthropy Capital) and nef (New Economics Foundation).

1. v
SOCIAL IMPACT SEMINAR
Part of the Cultural Commissioning Programme
Sally Bagwell, Lynn Simmonds, New Philanthropy Capital, Autumn 2015.

2. WELCOME AND
INTRODUCTIONS
2

3. AIM FOR TODAY
Think about how to demonstrate social outcomes required by public service
commissioners or outcomes wanted by funders, in a way which
• aligns with your organisation’s creative mission as well as its social ambitions,
• supports your strategic planning, and
• helps deliver what is important for you as well as funders / commissioners
To help you identify, collect and interpret evidence which:
• Can inform and influence funders / commissioners of your social impact
• Is realistic to collect, in keeping with the communities you work with
• Draws on current best practice and makes use of existing research.

4. INTRODUCTION TO
IMPACT MEASUREMENT
4

5. DISCUSSION
• Why is measuring social impact important?
• Why is measuring social impact difficult?

6. WHY MEASURING IMPACT IS IMPORTANT?
Saves staff
time
Influences the
debate on
“what works”
Improved
services
Raises profile Motivates
frontline staff
Taken from: Rickey, B, Lumley, T and Ni Ogain, E . (2011) A Journey to Greater
Impact. New Philanthropy Capital.
Helps secure
funding
6

7. WHAT ARE THE CHALLENGES OF IMPACT
MEASUREMENT?
Lack of
skills
Cost
Complexity
Risk of weak
results
Lack of
confidence in
value of soft
outcomes
Risk of
negative impact
on service
delivery
Lack of
technology
Wary of
losing sight of
ethos
7

8. WHAT DO COMMISSIONERS WANT?
8
“Public health people thrive on using
data…if you can demonstrate that the
people who use libraries are from
lower socioeconomic groups and
you have a product that contributes to our
objectives then Public Health should be
interested”
Tracey Polak, Devon County Council
“There is quite a spectrum of
commissioners, but… more and
more of us are evidence based
commissioners, and we’re looking
to commission for outcomes”
Gary Wallace, Plymouth City Council
“We are open to innovative
proposals to show how
services deliver wellbeing
outcomes. We are also
happy for providers to
draw on evidence
produced by others
where they have similar
approaches to delivery.”
Christian Markandu, Tri-
borough adult services
commissioner

9. MEASURING TOGETHER EXERCISE:
UNDERSTANDING PRIORITIES
• What is important to your commissioners
and funders?

10. COMPONENTS OF NPC’S APPROACH TO
MEASURING SOCIAL IMPACT
10
Map your
theory of
change
Prioritise
what you
measure
Choose
your level
of evidence
Select your
sources and
tools
Effective measurement
framework developed
Strategic vision
Leadership
Case for impact measurement
Download: NPC’S FOUR PILLARS APPROACH

11. ADVANTAGES OF APPROACH
Efficient: Use other people’s evidence / data where possible
Grounded: Based on the underlying outcomes you want to influence
Integrated: Ensures what is measured is embedded in a shared understanding of the
planned outcomes of your activities
Buy-in: Provides mechanism for staff to input into what is measured
Understandable: Provides a narrative about your impact
11
Using a structured approach to measurement allows you to get onto the front
foot in conversations with commissioners, and help them to understand what
you’re hoping to achieve

12. DEVELOPING AND USING
A THEORY OF CHANGE

13. WHAT IS A THEORY OF CHANGE?
Links activities intermediate outcomes  final outcomes
A description of how activities lead to outcomes
13
- Clarifies what the activities aim to
achieve and how
- Provides the case for why achieving
intermediate outcomes is important
- Provides a structure for identifying
what can be measured

14. Tracker 1 Tracker 2 Tracker 3 Tracker 4
WHERE TO BEGIN?
Why are we measuring?
Alice: "Would you tell me, please, which way I
ought to go from here?"
Cheshire Cat: "That depends a good deal on
where you want to get to."
Alice: "I don’t much care where."
Cheshire Cat: "Then it doesn’t matter which
way you go."
14

15. THE IMPORTANCE OF OUTCOMES
15
Mentoring Reduction in youth unemployment
?

16. HOW TO REPRESENT A THEORY OF CHANGE
16
Planning Triangle Logic Model Outcomes Chain
However you represent your theory of change, it should be supported by a written description.

17. LOGIC MODEL: OLDER PEOPLES’ DAY
CENTRE ACTIVITIES IN A CULTURAL CENTRE
17
Group based participatory activity
Enabling
factor
Intermediate
outcomes
Long-term
goal
Mainstream cultural activities
Activity
Appealing and affordable arts provision
Wellbeing improves
Increased social connections
Increased artistic skills
Increased intellectual engagement
Older people stay well independently for longer

18. THEORY OF CHANGE: OLDER PEOPLES’ DAY
CENTRE ACTIVITIES IN A CULTURAL CENTRE
Older people
stay well
independently
for longer
Group based
participatory
arts activity
Sense of
achievement
Wellbeing
improves
Increased
interest in
trying new
things
Meaningful
shared
experiences
Mainstream
cultural
activities
18
Increased
intellectual
engagement
Increased
artistic skills
Activity
Enabling
factor
Intermediate
outcome
Long-term
goal
Increased
social
connections
Appealing and
affordable arts
provision

19. BENEFITS OF THEORY OF CHANGE
 Helps to understand the important outcomes
• Help teams work together towards a shared understanding
• Help determine what needs to be measured
 Helps understand how change happens
• Help identify and open up ‘black boxes’ in thinking
 Helps to track progress to ultimate goal
 Quickly and effectively communicate aims
 Structure reporting and make claims about impact
19
Download: CREATING YOUR THEORY OF CHANGE: NPC’S PRACTICAL GUIDE

20. PRIORITISING YOUR
OUTCOMES

21. PRIORITISE OUTCOMES THAT:
you directly influence (rather than indirectly support)
are important / material to your mission
are not too costly to measure
will produce credible data
21

22. THEORY OF CHANGE: OLDER PEOPLES’ DAY
CENTRE ACTIVITIES IN A CULTURAL CENTRE
Older people
stay well
independently
for longer
Group based
participatory
arts activity
Sense of
achievement
Wellbeing
improves
Increased
interest in
trying new
things
Meaningful
shared
experiences
Mainstream
cultural
activities
22
Increased
intellectual
engagement
Increased
artistic skills
Activity
Enabling
factor
Intermediate
outcome
Long-term
goal
Increased
social
connections
Appealing and
affordable arts
provision

23. MEASURING TOGETHER EXERCISE:
OUTCOMES
• What are the outcomes or changes you are
looking to achieve through your work?
• How similar are these across different
organisations?

24. CHOOSING THE RIGHT
LEVEL OF EVIDENCE

25. HOW ROBUST DOES EVIDENCE OF YOUR
IMPACT NEED TO BE?
• What do your target stakeholders (funders?) think?
• What is possible, given the nature of your service?
• What is plausible, given resources etc?
25

26. DATA TO COLLECT
Monitoring data
• Demographic profile of participants / attendees
• Attendance and completion rates
Quality
• Participant satisfaction and feedback
• Practitioner feedback
• Info to convey the richness of the experience (artwork, performance, photos or video
records)
Intermediate and long term outcomes
• Outcomes tools eg, Wellbeing measures such as (WEMWEBS), Outcomes stars
• Before and after measures using psychometric scales
• Session logs and observations
• Cost and value for money of achieving outcomes
26

27. WHAT TYPE OF DATA :
OUTCOMES MEASUREMENT
Quantitative data (numbers)
•Surveys/questionnaires
•Psychometric scales
•Case work tools/records
•Statutory data
Qualitative data (words)
•Interviews
(telephone or face-to-face)
•Observation
•Focus groups
•Anecdotes and feedback
•Photographs, video, capturing experiences
How many people
completed the
course, felt positive,
increased their
aspirations, gained
qualifications
What did
beneficiaries think,
did it make a
difference to them?
How? How could it
have been better?

28. NESTA STANDARDS OF EVIDENCE
Level 1: You can describe what you do and why it matters logically,
coherently, and convincingly
Level 2: You capture data that shows positive change but you cannot
confirm you caused this
Level 3: You can demonstrate causality using a control or comparison
group
Level 4: You have one + independent replication evaluations that
confirm these conclusions
Level 5: You have manuals, systems and procedures to ensure
consistent replication and positive impact (fidelity)
28http://www.nesta.org.uk/publications/standards-evidence-impact-investing

29. SELECT YOUR SOURCES
AND TOOLS

30. THEORY OF CHANGE: OLDER PEOPLES’ DAY
CENTRE ACTIVITIES IN A CULTURAL CENTRE
Older people
stay well
independently
for longer
Group based
participatory
arts activity
Sense of
achievement
Wellbeing
improves
Increased
interest in
trying new
things
Meaningful
shared
experiences
Mainstream
cultural
activities
30
Increased
intellectual
engagement
Increased
artistic skills
Activity
Enabling
factor
Intermediate
outcome
Long-term
goal
Increased
social
connections
Appealing and
affordable arts
provision
?
?
? ?
?
?
?
?
?
?

31. SOURCES OF EVIDENCE AND TOOLS
Research including academic
research base
Research docs may specify tools
Orgs who share your outcomes
(may be non-arts)
Commissioner preferences
Other arts and cultural orgs
(including umbrella bodies)
High quality evaluations from
partners doing similar interventions
Data from external sources
(partners, government)
Evidence from your own data
collection
Evidence Tools
Libraries of tools (eg Impact Hub)

32. THEORY OF CHANGE: OLDER PEOPLES’ DAY
CENTRE ACTIVITIES IN A CULTURAL CENTRE
Older people
stay well
independently
for longer
Group based
participatory
arts activity
Sense of
achievement
Wellbeing
improves
Increased
interest in
trying new
things
Meaningful
shared
experiences
Mainstream
cultural
activities
32
Increased
intellectual
engagement
Increased
artistic skills
Activity
Enabling
factor
Intermediate
outcome
Long-term
goal
Increased
social
connections
Appealing and
affordable arts
provision
?
?
? ?
?
?
?
?
?
?

33. WHAT: BEFORE YOU THINK ABOUT
COLLECTING DATA ASK YOURSELF:
Has anyone
already
proved the
causal link
you want to
make?
Yes
No
Is it really
important
you need
data on
this
outcome?
Do you
already
collect this
data?
Don’t collect
unless you
decide it is
really
important
Use research to
limit your data
collection
Yes
No
Yes
Can you
access this
data?
No
Develop
your own
data
source
Use this
data (if it is
fit for
purpose)
Does anyone
else collect
this data?
No
No
Yes
Yes
33

34. MEASURING TOGETHER EXERCISE:
EVIDENCE
• What evidence do you have at the moment?
…generated by you?
…generated by others?
• Can you fill the gaps?

35. MEASURING TOGETHER EXERCISE:
TOOLS TO DEMONSTRATE OUTCOMES
• What tools do you use to measure outcomes
and why?
• Is this similar across different organisations?
• Are some tools more / less suitable for your
users?

36. THE SOCIAL VALUE ACT
36

37. PUBLIC SERVICES (SOCIAL VALUE) ACT 2012
Requires public authorities to consider
how the procurement of a service may
improve the social, economic &
environmental well-being of an area.
Barriers to take-up
1. Mixed awareness and take-up of
the Act is mixed
2. Varying understanding of how to
apply the act can lead to
inconsistent practice
3. Measurement of social value
needs development
37

38. MEASUREMENT OF SOCIAL VALUE: THE
CHALLENGE FOR COMMISSIONERS
• The legal framework around commissioning and procurement
encourages commissioners to seek comparable ways of measuring
things
• Risk of measuring what is easy, rather than what is important
• Tempting to seek financial proxies which appear easily comparable
• Effective shared measurement is a promising alternative
38

39. DISCUSSION
• What are your thoughts on Social Value Act
• What could be its effects?
• Does anyone have any experience of it?

40. VALUE FOR MONEY AND
ECONOMIC EVALUATION
40

41. ECONOMIC ANALYSIS – THE BLACK BOX
41
Change in
outcomes
Costs
(£)
Net
economic
benefit of
programme
(£)
Value of
outcomes
(£)
Value of programme = Σ (∆ Outcomes x Value of outcomes) – Costs

42. FINANCIAL PROXIES FOR SOCIAL OUTCOMES
A few places where the financial value of certain social outcomes have been estimated:
• New Economy Manchester Unit Cost database
http://neweconomymanchester.com/stories/832-unit_cost_database
• Housing Association Charitable Trust Social Value Bank http://www.hact.org.uk/social-
value-bank
• Personal social services research unit (Unit costs for health and social care)
http://www.pssru.ac.uk/project-pages/unit-costs/
42

43. DIFFERENT TYPES OF ECONOMIC ANALYSIS
43
If you have cost data: Unit costs
What does it cost you to deliver the intervention per person?
Does this change depending on how many people?
Eg, Cartwheel’s mental health projects cost £70 per head for a 10 week course of 15 participants.
If you have cost and change in outcomes data: Cost effectiveness
What outcomes can you achieve for the cost of your intervention?
How does this compare to other interventions aiming to achieve the same outcomes?
Eg, Age Exchange supports dementia patients through arts-based reminiscence activity. It can
secure a 15.3% improvement in mood and engagement for £1,369 per person.
If you have cost, change outcomes & value outcomes: Cost benefit analysis / SROI
What is the overall value of the change in outcomes you achieve?
What is the saving or value to society of you achieving this outcome?
Eg, Artlift—an arts on prescription service—shows a 27% reduction in healthcare spend for
participants, with a saving to the NHS of £576 per person.

44. DISCUSSION
• Would any of these types of economic analysis be useful for your
intervention?
• Do you have any of the necessary information at the moment?
• How might you decide whether its worth making the investment?

45. TOWARDS SHARED
MEASUREMENT
45

46. SHARED MEASUREMENT
A common understanding of what to
measure and how to measure.
46

47. WHY SHARED MEASUREMENT?
47
• Saves time and resources
• Improves standards of impact measurement
• Promotes systems thinking
• Understanding what works
• Stronger voice

48. DEVELOPING A SHARED MEASUREMENT
APPROACH
48
Five step to remember…

49. ASSESSING READINESS
Sector
• Coherency
• Consistency
• Infrastructure
• Well defined
needs
• History of
collaboration
Evidence
• Some
understanding
of ‘what
works’
• Measurement
• Pressure to
prove
Momentum
• Awareness of
benefits
• Committed
funder
49

50. ACKNOWLEDGING CHALLENGES
50
• Takes time to agree on shared values and goals
• Doesn’t reflect nuances of different organisations
• May not meet funder / commissioner requirements
• Risk comparing apples: oranges
• Reputational risk of comparability
• Data protection prevents sharing

51. Discussion
• What are the benefits of sharing evidence?
• What questions does it raise for your organisation?
• Who needs to be involved?
• What actions will you take away from today?

52. CLOSING
52

53. HOW TO COMMUNICATE IMPACT THROUGH
REPORTING
Five key questions:
1. What’s the problem you’re trying to
tackle?
2. What are you doing to address it? (your
activities)
3. What does that achieve? (your
outcomes)
4. How do you know? (your evidence)
5. How are you learning & improving?
53
External evidence
Articulating your
approach through
the theory of
change
http://www.thinknpc.org/publications/talking-about-results/
http://www.clinks.org/sites/default/files/ReportWritingGuide.pdf
Evidence you
collect
Your reflections

54. DISCUSSION
• How does this approach to measuring social impact relate to your own
experiences of working with commissioners?
• Do you have any space to negotiate about the most appropriate impact
measurement?
• Are there additional issues you had hoped we could cover?

55. CULTURAL COMMISSIONING PROGRAMME
RESOURCES
Cultural Commissioning Programme website
www.ncvo.org/CCProg
• E-bulletin
• Case studies
• Articles, blogs and resources
• Insights from commissioners and influencers
Resources library http://knowhownonprofit.org/funding/cultural-
commissioning?=ncvo-cultural-commissioning
Databases and evidence sources http://knowhownonprofit.org/funding/cultural-
commissioning/evidencing-the-social-value-of-arts-and-culture

56. v
THANK YOU
New Philanthropy Capital – transforming the charity sector
sally.bagwell@thinkNPC.org lynn.simmonds@thinkNPC.org www.thinkNPC.org