Anne Kazimirski 2013 Future Focus Workshop: Benifits of Monitoring & Evaluation
•
0 j'aime•368 vues
Monitoring and evaluation can provide several benefits to charities and funders. It helps to increase their impact by focusing on theories of change and effective commissioning. NPC works at the intersection of charities and funders to strengthen collaboration toward shared goals. Measuring impact is important as it raises an organization's profile, motivates staff, and influences debates on "what works". However, before collecting data, organizations should determine if the data is important and if others already collect it to avoid duplicating efforts. Both quantitative and qualitative data can be used, and measurements should be taken before and after a program or service to assess impact while considering alternative explanations.
Recommandé
Recommandé
Contenu connexe
Tendances
Tendances (20)
Similaire à Anne Kazimirski 2013 Future Focus Workshop: Benifits of Monitoring & Evaluation
Similaire à Anne Kazimirski 2013 Future Focus Workshop: Benifits of Monitoring & Evaluation (20)
Plus de Voluntary Action LeicesterShire
Plus de Voluntary Action LeicesterShire (20)
Dernier
Dernier (20)
Anne Kazimirski 2013 Future Focus Workshop: Benifits of Monitoring & Evaluation
- 1. BENEFITS OF MONITORINGvAND EVALUATION Anne Kazimirski VAL future focus VCS conference, 18th September 2013
- 2. NPC: TRANSFORMING THE CHARITY SECTOR Charity Increasing the impact of charities eg, impact-focused theories of change Funder Increasing the impact of funders eg, effective commissioning Sector NPC works at the nexus between charities and funders Consultancy Think tank Strengthening the partnership eg, collaboration towards shared goals 2
- 3. COUNTERFACTUAL • Comparing the world with your organisation in it with what the world would be like without it. 3
- 4. WHY MEASURING IMPACT IS IMPORTANT Raises profile Motivates staff The Brandonsecure provides Helps Centre Influences the counselling funding and psychotherapy to debate on young people between the ages of 12 “what works” and 21 Improved services Taken from: Rickey, B, Lumley, T and Ni Ogain, E . (2011) A Journey to Greater Impact New Philanthropy Capital. 4
- 5. THEORY OF CHANGE • Links activities intermediate outcomes final outcomes – clarifies what the activities aim to achieve and how – provides a structure for identifying what can be measured – provides the case for why achieving intermediate outcomes is important A conceptual map of how activities lead to outcomes 5
- 6. EXAMPLE THEORY OF CHANGE PARENTS’ COUNSELLING SERVICE Counselling Clients‟ emotional or psychological difficulties decrease Clients‟ capacity for self care increases Clients‟ ability to support their children's healthy development improves Activities Intermediate Outcomes Parent / Child interaction improves Children‟s emotional resourcefulness improves Final outcomes 6
- 7. WHAT: BEFORE YOU THINK ABOUT COLLECTING DATA ASK YOURSELF: Do you already collect this data? Yes Use this data (if it is fit for purpose) No Has anyone already No proved the causal link you want to make? Yes Is it really important you need data on this outcome? Yes Does anyone else collect this data? No Use research to limit your data collection Yes Don‟t collect unless you decide it is really important Yes Can you access this data? No No Develop your own data source 7
- 8. LEVELS OF EVIDENCE Credibility Anecdotes / quotes Case Self-reported studies change Basic Before and after survey Control groups Randomised control trial Advanced 8
- 9. LOTS OF DECISIONS • Budget • Stakeholders • Timing • Participants • Tools 9
- 10. QUANTITATIVE AND QUALITATIVE DATA • Quantitative data (numbers) – Statistical estimates – Prevalence of views, attitudes and experiences How many people find our newsletter useful? – Admin data/ questionnaires (paper, web, etc.) • Qualitative data (words) – Detailed understanding – In depth interviewing (telephone/ face to face) What factors influence people‟s views of our newsletter? 10
- 11. WHEN TO MEASURE? • Pre-post design: Before measure Programme / service After measure • What attribution can you claim? • Perception of beneficiaries and those delivering the programme: important but subjective 11
- 12. COMPARISON GROUPS Intervention group Before measure Programme / service After measure Comparison group Before measure After measure • Compare before and after data for participants and nonparticipants, ideally matching on their key characteristics 12
- 13. ALTERNATIVES Comparison groups • Waiting lists • Dosage: users who‟ve received a smaller „dose‟ of a service • Type of service: comparing users who‟ve received one type of service compared to another No comparison group • Combine pre and post data with respondents‟ perspectives • Collect data on other events and changes in circumstances which might have contributed 13
- 14. 5 TIPS FOR MEASURING YOUR IMPACT 1. Decide what‟s important 2. Keep it simple 3. Know your limits 4. Consider investing in training 5. Think about the future 14
Notes de l'éditeur
- We work with charities and fundersI focus on support with measuring impactProjects all about strengthening the sector (Research, collaboration with other organisations working towards the same goals)
- Setting out measuring impact: know what you’re trying to doA parallel world : what you really need to understand the difference you make. Being able to track down the child, the school, the town or the country that your work tries to help, and find out what happened to them without you. What better way is there to assess your impact?Going to go through all the key issues you need to think about when collecting data
- We work with charities and funders on their ToCKey step: identifying prioritiesWe usually identify intermediate and final outcomesGO THROUGH ToCParents’ counselling service where referral was through childIdentified final outcomes made explicit that important focus was the parent
- Understand different levels of evidence, to help you decide where to pitch your approach, and what matches your resources and the type of work you’re trying to evaluate
- Like a wedding, lots of important decisions to takeStakeholders: family & friendsParticipants: guestsTools: food, wine, music – don’t want to overburden with questions, you don’t want party guests to keel overNumber of decisions can be overwhelming
- When planning research, good to be aware of some basic principles, like the difference between quant and qual data, and what they provide.Some of you will be very familiar with these distinctions…
- how outcomes change over time by collecting data before (pre) and after (post) the programme. Counselling service: 1st session, then 10 sessions in, then 20 How deal with attribution?You’d want to know perception of beneficiaries
- Difficult to identify, let alone collect data
- Counselling service: feedback form supplemented standardised scales to assess change in outcomes
- Decide what’s important. Define your research question and be clear what you want to achieve.Keep it simple. Concentrate on what you really want to know and don’t try to be too ambitious in the scope of your evaluation.Know your limits. You need to choose an approach that matches your resources.Consider investing in training. If you are not confident, get training or seek external help.Think about the future. Don’t build a system that will quickly go out of date and look at the year-on-year costs.