Sharing Impact Practice (outcomes measurement) from third sector (community & voluntary sector) wellbeing projects for public health mental health & alcohol interventions

1. Impact Practice
Sharing experience from
Third Sector wellbeing projects
Catherine Greaves, Project Manager

2. Outcomes for this session
A greater understanding of
• What the third sector means by
“impact practice” & associated terms.
• How to use a Planning triangle and a
Monitoring & Evaluation Framework.
An awareness of
• The Theory of Change.
• Third sector resources for Impact
Practice

3. What & Why?

4. “Impact is the difference you make. By focusing on your impact, you can make
more of a difference through your work. This means planning what impact you
want to have and how best to achieve it, collecting information about your
impact, assessing what impact you’re having, communicating this information
and learning from it. We call this cycle of activities impact practice. This
includes, but is significantly bigger than, the tasks of measuring, monitoring
and evaluating your work.
Many terms are used to talk about the difference you make and the work
you’re doing, including outcomes, activities and outputs. The most important
distinction is between the work you do and the difference it makes: it is all too
easy to focus on your day-to-day work, when it is the impact you’re having,
and how you can have more, that is arguably more important.”
From: the Code of Good Impact Practice

5. Why is it more important than your day-to-day
work?
Professional development
• Fresh thinking
• Improve performance
• Show effectiveness
Accountability - stakeholders
• Service users & communities
• Funders
• Commissioners
Sustainability
• Secure funding and commissioning

6. Jargon buster
Impact: the broad or longer-term effects of your project or your
organisation’s work (also referred to as the difference you make). This can
include effects on people who are direct users, effects on those who are not
direct users, or effects on a wider field such as NHS practice or government
policy.
Impact practice: the activities that your team or your organisation does to
focus on its impact. This can include planning desired impact, planning how
to measure it, collecting information about it, making sense of that
information, communicating it and learning from it.
From: the Code of Good Impact Practice

7. Jargon Buster – over to you
1. Activities
2. Evaluation
3. Indicator
4. Monitoring
5. Overall aim
6. Outputs Outcomes
7. Processes
8. Theory of change
9. User Satisfaction

8. What is a Theory of Change?

9. Theory of change
“ A description of why a particular way of working will be effective,
showing how change happens in the short, medium and long term to
achieve the intended outcome.”
Usually represented as a visual map but can be narrative.
From: NCVO Charities Evaluation Services

10. What makes it different & potentially difficult?
• A critical, reflective process.
• Challenges assumptions that we make about our work.
• Starts with impact and works backwards.
• Shows interrelationships between outputs and outcomes.
• Suggests new possibilities.
• Acknowledges that there are a range of stakeholders that contribute
to the impact we are working towards.
• A thinking process as well as a product.

11. A good place to start

12. NCVO CES example (Lambeth)
To reduce isolation among older people
To increase the take up
of mental health
services by older
people
To increase older
people’s understanding
of mental health issues
To offer
information
To organise
social events
To Run outreach
events
To improve the
mental health
of older people
in Lambeth
Overall Aim
Specific Aims
Objectives

13. Local Example 1
Reduced depression levels for clients
Reduced anxiety levels for clients
Reduced social isolation
Improved family relationships
Improved economic wellbeing
1:1 counselling for clients who have experienced baby loss and infertility.
Counsellor-led support groups for baby loss and infertility.
Support and training for trainee counsellors on placement.
Professional development opportunities for practitioners working with families.
Improved health &
wellbeing of Berkshire
families affected by
baby loss and infertility
Aim
Outcome
Activities

14. Over to you…
Alcohol Service/Project
No more than 5 specific aims.
Use ‘action’ words e.g. to reduce, to
increase.
No more than 6 objectives.
Keep it general e.g. professional development opportunities rather
than ‘workshops’
One
long-term
change you
contribute to.
Aim
Outcome
Activities

15. Writing your own planning triangle – more tips
• Whose health are you seeking to improve?
• What behaviour are you seeking to change?
• How will you know if you have succeeded in changing behaviour?
• How will you know if you have succeeded in contributing to improving
the health of your target population?
• What processes and inputs need to be taken into account
• What assumptions have been made about the theoretical links
between the services/interventions and outcome?

16. Writing your own planning triangle – more tips
Outcomes for alcohol – some ideas
• Reduction in problematic drinking behaviour
• Clients have improved wellbeing
• Improved social support/functioning
• Improved help-seeking behaviour
• Improved capacity for families to cope with negative consequences of
drinking behaviour
From http://www.effectiveservices.org/downloads/BIG_Impact_of_Alcohol_evaluation_framework_website_dec15.pdf

18. Local Example 2

19. Adding to the triangle
Inputs
• Funding
• Staff time & skills
• Volunteer time & skills
• Premises
• Partner’s resources
Processes
• Developing relationships with
partners
• Developing relationships with
users
• Training

20. Evaluation Framework – Outputs
A
Outputs Output indicators Data collection &
storage
When & by whom Reporting
Specialist, accessible.
parenting training.
Number of courses &
workshops.
Number & profile of
people attending.
Times and venues of
courses & workshops.
Level of user
satisfaction.
Online event
management
(EventBrite)
CRM system
(CharityLog)
Online survey
(surveymonkey)
Online booking form &
registration forms
completed by
parent/carer.
Online survey form
completed by
attendees at end of
the course or
workshop.
Quarterly
for
Trustees.
End of
project
for
funders

21. Evaluation Framework – Outcomes
A
Outcomes Outcome indicators Data collection &
storage
When & by
whom
Reporting
Increased
confidence to access
mental health
services
Level of confidence is
moderate to high.
Likelihood of
accessing services is
moderate to high.
(internally developed
rating scale)
Paper questionnaires
Database system.
Service users –
end of outreach
work.
Project workers –
remote access
Quarterly
for
Directors.
End of
project for
funders

22. Evaluation Framework – Outcomes
A
Outcomes Outcome indicators Data collection &
storage
When & by
whom
Reportin
g
Reduced anxiety GAD-7 scores are all
in the mild range at
the end of
counselling.
Paper
questionnaires
Counselling notes
Spreadsheet
Clients –each
counselling
session.
Counsellors each
counselling
session.
Administrator –
end of
counselling.
Quarterly
for
funders.

23. Evaluation Framework – Outcomes
A
Outcomes Outcome indicators Data collection &
storage
When & by whom Reporting
Improved wellbeing
of parent/carers
Warwick –Edinburgh
score – improvements
in all questions.
Loneliness and social
isolation score –
improvements in all
questions. (Campaign
to end Loneliness
rating scale)
Number and type of
services accessed by
parent/carer has
increased.
Paper
questionnaires
Peer Supporter
notes
CRM system Ladder
outcomes
functionality.
(CharityLog)
Clients – start &
end of peer
support.
Peer Supporters
each contact with
parent/carer.
Administrator –
end of peer
support.
Quarterly
for
trustees.
End of
funding
report.

24. Over to you…..
Alcohol Project /Service
A
Outcomes Outcome indicators Data collection &
storage
When & by
whom
Reportin
g
.

25. Over to you… Evaluation Framework for an
Alcohol Misuse Service
Some ideas for outcome indicators:
• Reduced levels of alcohol intake
• Increased knowledge of harmful effects
• Better coping/abstinence strategy
• Increased self-esteem/self-confidence
• Reduced anti-social behaviour
• Increased knowledge of the services available.
From http://www.effectiveservices.org/downloads/BIG_Impact_of_Alcohol_evaluation_framework_website_dec15.pdf

26. Recommended resources
• https://knowhownonprofit.org/
• www.ncvo.org.uk/charities-evaluation-services
• inspiringimpact.org
• https://www.biglotteryfund.org.uk/research/making-the-most-of-
funding/impact-and-outcomes/measuring-the-difference-projects-
make

27. Outcomes for this session
A greater understanding of:
• What the third sector means by
“impact practice” & associated terms.
• How to use a Planning triangle and a
Monitoring & Evaluation Framework.
An awareness o:
• The Theory of Change.
• Third sector resources for Impact
Practice

28. Catherine Greaves
catherinegreaves@gmail.com
Twitter: @categreaves @pscsleep
LinkedIn: https://www.linkedin.com/in/catherineagreaves