Monitoring and evaluation can provide several benefits to charities and funders. It helps to increase their impact by focusing on theories of change and effective commissioning. NPC works at the intersection of charities and funders to strengthen collaboration toward shared goals. Measuring impact is important as it raises an organization's profile, motivates staff, and influences debates on "what works". However, before collecting data, organizations should determine if the data is important and if others already collect it to avoid duplicating efforts. Both quantitative and qualitative data can be used, and measurements should be taken before and after a program or service to assess impact while considering alternative explanations.
How to Effectively Monitor SD-WAN and SASE Environments with ThousandEyes
Anne Kazimirski 2013 Future Focus Workshop: Benifits of Monitoring & Evaluation
1. BENEFITS OF MONITORINGvAND EVALUATION
Anne Kazimirski
VAL future focus VCS conference, 18th September 2013
2. NPC: TRANSFORMING THE CHARITY SECTOR
Charity
Increasing the
impact of charities
eg, impact-focused
theories of change
Funder
Increasing the
impact of funders
eg, effective
commissioning
Sector
NPC works at the
nexus between
charities and
funders
Consultancy
Think tank
Strengthening the
partnership
eg, collaboration
towards shared
goals
2
4. WHY MEASURING IMPACT IS IMPORTANT
Raises profile
Motivates
staff
The Brandonsecure provides
Helps Centre
Influences the
counselling funding
and psychotherapy to
debate on
young people between the ages of 12 “what works”
and 21
Improved
services
Taken from: Rickey, B, Lumley, T and Ni Ogain, E . (2011) A Journey to Greater
Impact New Philanthropy Capital.
4
5. THEORY OF CHANGE
• Links activities intermediate outcomes final outcomes
– clarifies what the activities aim to achieve and how
– provides a structure for identifying what can be measured
– provides the case for why achieving intermediate outcomes is
important
A conceptual map of how activities lead to outcomes
5
6. EXAMPLE THEORY OF CHANGE
PARENTS’ COUNSELLING SERVICE
Counselling
Clients‟ emotional or
psychological
difficulties decrease
Clients‟ capacity
for self care
increases
Clients‟ ability to support
their children's healthy
development improves
Activities
Intermediate Outcomes
Parent / Child
interaction
improves
Children‟s emotional
resourcefulness
improves
Final outcomes
6
7. WHAT: BEFORE YOU THINK ABOUT
COLLECTING DATA ASK YOURSELF:
Do you
already
collect this
data?
Yes
Use this
data (if it is
fit for
purpose)
No
Has anyone
already
No
proved the
causal link
you want to
make?
Yes
Is it really
important
you need
data on
this
outcome?
Yes
Does anyone
else collect
this data?
No
Use research to
limit your data
collection
Yes
Don‟t collect
unless you
decide it is
really
important
Yes
Can you
access this
data?
No
No
Develop
your own
data
source
7
8. LEVELS OF EVIDENCE
Credibility
Anecdotes /
quotes
Case Self-reported
studies
change
Basic
Before and
after survey
Control
groups
Randomised
control trial
Advanced
8
10. QUANTITATIVE AND QUALITATIVE DATA
• Quantitative data (numbers)
– Statistical estimates
– Prevalence of views,
attitudes and experiences
How many
people find
our newsletter
useful?
– Admin data/ questionnaires
(paper, web, etc.)
• Qualitative data (words)
– Detailed understanding
– In depth interviewing
(telephone/ face to face)
What factors
influence
people‟s views
of our
newsletter?
10
11. WHEN TO MEASURE?
• Pre-post design:
Before
measure
Programme / service
After
measure
• What attribution can you claim?
• Perception of beneficiaries and those delivering the programme:
important but subjective
11
12. COMPARISON GROUPS
Intervention group
Before
measure
Programme / service
After
measure
Comparison group
Before
measure
After
measure
• Compare before and after data for participants and nonparticipants, ideally matching on their key characteristics
12
13. ALTERNATIVES
Comparison groups
• Waiting lists
• Dosage: users who‟ve received a smaller „dose‟ of a service
• Type of service: comparing users who‟ve received one type of service
compared to another
No comparison group
• Combine pre and post data with respondents‟ perspectives
• Collect data on other events and changes in circumstances which
might have contributed
13
14. 5 TIPS FOR MEASURING YOUR IMPACT
1. Decide what‟s important
2. Keep it simple
3. Know your limits
4. Consider investing in training
5. Think about the future
14
Notes de l'éditeur
We work with charities and fundersI focus on support with measuring impactProjects all about strengthening the sector (Research, collaboration with other organisations working towards the same goals)
Setting out measuring impact: know what you’re trying to doA parallel world : what you really need to understand the difference you make. Being able to track down the child, the school, the town or the country that your work tries to help, and find out what happened to them without you. What better way is there to assess your impact?Going to go through all the key issues you need to think about when collecting data
We work with charities and funders on their ToCKey step: identifying prioritiesWe usually identify intermediate and final outcomesGO THROUGH ToCParents’ counselling service where referral was through childIdentified final outcomes made explicit that important focus was the parent
Understand different levels of evidence, to help you decide where to pitch your approach, and what matches your resources and the type of work you’re trying to evaluate
Like a wedding, lots of important decisions to takeStakeholders: family & friendsParticipants: guestsTools: food, wine, music – don’t want to overburden with questions, you don’t want party guests to keel overNumber of decisions can be overwhelming
When planning research, good to be aware of some basic principles, like the difference between quant and qual data, and what they provide.Some of you will be very familiar with these distinctions…
how outcomes change over time by collecting data before (pre) and after (post) the programme. Counselling service: 1st session, then 10 sessions in, then 20 How deal with attribution?You’d want to know perception of beneficiaries
Difficult to identify, let alone collect data
Counselling service: feedback form supplemented standardised scales to assess change in outcomes
Decide what’s important. Define your research question and be clear what you want to achieve.Keep it simple. Concentrate on what you really want to know and don’t try to be too ambitious in the scope of your evaluation.Know your limits. You need to choose an approach that matches your resources.Consider investing in training. If you are not confident, get training or seek external help.Think about the future. Don’t build a system that will quickly go out of date and look at the year-on-year costs.