Conceptual inputs on monitoring and result measurement in private sector
•Télécharger en tant que PPTX, PDF•
4 j'aime•984 vues
Conceptual inputs on monitoring and result measurement in private sector
Recommandé
Contenu connexe
Tendances
Tendances (20)
Similaire à Conceptual inputs on monitoring and result measurement in private sector
Similaire à Conceptual inputs on monitoring and result measurement in private sector (20)
Plus de HELVETAS Swiss Intercooperation Viet Nam
Plus de HELVETAS Swiss Intercooperation Viet Nam (8)
Dernier
Dernier (20)
Conceptual inputs on monitoring and result measurement in private sector
- 1. Logo Zenebe Uraguchi HELVETAS Swiss Intercooperation Bangladesh Regional Workshop 25.09.2013 Ben Tre, Vietnam Monitoring and Results Measurement (MRM) System
- 2. LogoWhy implement MRM system based on the DCED standard ? Internally & externally Cornerstone of good project steering Guidance for strategic direction for achieving impacts sustainably Externally Demonstrates impact to the donors & other stakeholders Contributes to credibility of projects Internally Fundamental to the learning process by providing vital information that projects need Enhances making good decisions in order to maximize impacts DCED: Donor Committee for Enterprise Development
- 3. LogoWhat is included in the (DCED) standard The DCED is a forum; 23 members (bilateral donors, including SDC, multilateral agencies and private foundations Member agencies and others share experiences and formulate a common understanding of good practices 7. Reporting The DCED Standard 8. Managing the System 4. Estimating attributable changes 5. Capturing wider changes in the system 1. Articulating the results chains 2. Defining indicators of change 3. Measuring changes in indicators 6. Tracking programme costs
- 4. LogoWhat is new as a system? • Do events unfold as planned? Yes? No? Why? • An internal system for MRM instead of external consultants One-off events External consultants Internal MRM system Improving intervention logics
- 5. Logo Source: Peter Roggekamp, Hans Posthumus What we don’t want to do? series of changes resulting from activities of the project local systems –people, businesses, government, civil society organizations and the relationships, rules, policies and cultural norms that govern how they interact and work together Goal
- 6. Logo Source: Peter Roggekamp, Hans Posthumus What we want to do?
- 7. LogoWhy MRM is important! Source: Peter Roggekamp, Hans Posthumus
- 8. Logo Source: Peter Roggekamp, Hans Posthumus
- 9. Logo Let us talk about the ball & goal Systematic Intervention Market trigger Market uptake Enterprise performance Impact The ball is the series of changes resulting from activities of the project A results chain or impact logic clearly outlines the series of changes project staff expect, starting with project activities and ending with goal level impacts
- 10. Logo An example from RLDP, Tanzania Farmers attend demo plot sessions Farmers understand technology Farmers apply technology MSK establishes demo plots Farmers‘ groups sign contract with MSK MSK distributes inputs Farmers apply inputs Farmers increase yield Farmers increase net income MSK sensitize farmers and fgroups
- 11. Logo An example from Samriddhi, Bangladesh Facilitating technical capacity of market actors Improving access to technologies/good practices & inputs Forming & strengthening producers’ groups Facilitating access to adapted financial services & products Systemic intervention Market trigger Enhanced capacity of local service providers on technical knowledge & skills Increased engagement of private sector enterprises Enhanced capacity of local service providers & producers’ groups in business planning & financial literacy Improved organizational & business skills of local service providers Service provision for affordable & quality services & inputs available; improved linkages with private sector enterprises & financial service providers established; improved &/or new technologies & practices in bull fattening available Better functioning market system Market uptake Increased production & productivity Increased additional employment & net income by producers Improved performance Impact Producers start establishing functional relationships with private sector enterprises Producers apply improved bull rearing practices using quality inputs & services tPoducers start developing & implementing business plans by identifying & negotiating with a range of financial service providers
- 12. Logo An example on results chain: direct & indirect outreach Training manual developed with the support of the project / other players e.g., Department of Livestock Services Activity Market trigger Market uptake Improved performance Impact Appropriate private sector company identified; deal made Local service providers selected Private sector company provided training to local service providers on improved bull fattening technology Local service providers improved their skills & knowledge on improved bull fattening technology Local service providers provided information on services to producers Producers are aware of service availability & are willing to receive & pay for the services Producers started using the services & adopted improved bull fattening technology Farmers fattened bulls in short cycles Farmers received better price for their bulls Increased profit for farmers Additional income of xx Taka for yy farmers Other farmers are influenced to fatten bulls using new technology Additional farmers received better price for their bulls Increased profit for indirect farmers Indirect outreach
- 13. LogoHow about developing a results chain from your project? Systemic intervention Market system changes Market uptake Improved performance Impact Tips: 1)Please select one intervention 2)Please be as detailed as possible in the impact logic (changes at each level)
- 14. Logo Results Chain and Logframes Compatible Results chain increase the relevance of Logframe as management tool Results chain provide details of the process for achieving results
- 15. Logo Compatible with Logframe • • • • • • • • • • Log Frame Results chain
- 17. Logo How to formulate indicators • Each box needs at least 1 indicator • Use quantitative and qualitative indicators - Behaviour, perception, motivation, … • Indicators should be S M A R T ! - Specific - Measureable - Attributable - Relevant - Timely
- 18. Logo Indicators…. We should always ask: “Is each piece of information essential to decision making?” What is a good indicator? Relevance Precision % growth in a business membership organization % change (year-on-year) of total businesses paying full membership fees in business membership organization Lazy & loose Tight & precise Time bounded- ness (e.g., profits in past month vs. in past year ) Realistic
- 19. Logo Defining indicators of change An indicator shows what success looks like outputs are produced & outcomes are achieved if indicators specify how the project will measure therefore the changes anticipated in the results chains are really occurring whether if so to what extent? + how? if not then why?
- 20. Logo Indicators indicators to be derived from the logic of the results chain Farmers reduced post- harvest loss of vegetables Farmers adopt improved plastic crates for harvesting and grading vegetables Number of farmers who adopted plastic crates % reduction of wastage or % increase in the sales of produce Quality of produce + Quantitative indicator Quantitative indicator tracking these indicators very important frequency of tracking will depend on the specific targets or projections set by the project Quantitative indicator
- 21. Logo Indicators & quantitative projection Projections: change in the value of an indicator due to the project by a specific date What to do? 1) At the start of activities, make clear projections about expected impacts Why we do this? give staff targets to aim for provide staff with feedback if an intervention is on track 2) Clear projection of key quantitative indicators give an indication of whether particular investments are worth their cost & impacts 3) Do market research, have field observations or other credible sources (e.g., secondary resources) provide realistic / credible projection 4) Periodically update projections (actual vs. projection) Help reflect new data collected on indicators of change
- 22. Logo An example from Pakistan Assumption Calucation Projection 50% success rate 445*50% 222 garment producers improve Oct-13 GPs produce 6 units per month 150Rs/unit* 6units Garment producers increase by 900Rs/month. Apr-13 Margins increase by 150Rs/unit 445 garment producers earn +150/unit Apr-13 75% success rate 594*75% 445 garment producers produce Apr-13 90% success rate 660*90% 594 garment producers improve Feb-13 Each sales agent services 30 garment producers 22*30 660 garment producers serviced Feb-13 75% uptake rate 30*75% 22 women adopt Dec-12 30 women trained Oct-12 Women adopt new service agents business model Sales agents deliver services to garment producers Garment producers improve knowledge of production techniques Garment producers produce high quality garments Garment producers earn higher margins Garment producers increase incomes Garment producers improve social status in house holds & community Project trains women to adopt new sales agents business model Constraint: Limited mobility of female garment producers (GPs) Identified solution: Sales agents (SAs) travel and provide services
- 23. Logo Data collection tools Observation In-depth interview Key informant interview (KII) Company / partner records Focus Group Discussions (FGD) Outsourced surveys Secondary Sources In-house surveys
- 25. Logo Secondary sources Source: HPC 2012
- 26. Logo Company records Source: HPC 2012
- 27. Logo Focus Group Discussion (FGD) Source: HPC 2012
- 29. Logo In-depth interviews Source: HPC 2012
- 30. Logo In house surveys Source: HPC 2012
- 31. Logo Outsourced surveys Source: HPC 2012
- 32. Logo Balance between rigor and cost Rigour Costs Observations Big surveys
- 33. Logo Mix of methods and triangulation 2 3 4 5 6 7 8 9 1 10 11 12 Same tool for several boxes For one box: use several simple tools for triangulation Tool 1 Tool 3 Document the process, not only findings; state the limitations
- 34. Logo Measuring change • Necessary to measure change • Ideally Pre-Intervention • Sources: - Previous monitoring data - Previous market analyses - Additional data collection and analysis e.g. through surveys, interiviews, FGD, etc. • Usually multiple baseline are needed across interventions Impact Indicator
- 35. Logo Missing baseline • What if baseline has not been collected? • What if the beneficiaires have not been known at the start of the intervention? Use secondary data; Derive a baseline from retrospective research; Use data from other regions with similar demographics etc. Use partners„ records
- 36. Logo Results Measurement Plans: example Level Impact chain Questions Indicators How? Who ? When? What do we have? Impact Additional income Whether farmers and laborers have increased income? Net increase in income Survey; secondary sources Researc h firm; sector team June xx Census from xx; secondary source Market uptake Farmers cultivated early variety Vegetables Do the farmers cultivate early variety vegetable? Do they get proper information? No. of farmers who cultivate early variety vegetable? Farmers' knowledge level Survey Researc h firm; sector team July XX Producers improved their knowledge on cultivation techniques of early variety Vegetables How many producers have been reached through promotional activities? How much have they learnt from the promotional activities? No. producers aware; change in knowledge of producers Observati on of the organizers , reports Sector team June xx
- 37. Logo Estimating attributable changes “it is better to be approximately right than precisely wrong.“John Maynard Keynes The outcomes might have happened without the intervention (counterfactual) Target groups might have been better off without the intervention (displacement or unintended consequences) The outcomes might not be due to the intervention, but due to other factors (e.g., projects, weather changes)
- 38. Logo b= before Impact= a-c a= after c= changes without the project Time Impact Intervention starts Intervention ends
- 39. Logo Attribution: Influencing factors Impact External factors Causality Many Few Easy to establish More difficult to establish Adapted from A. Miehlbradt
- 40. Logo Attribute plausibly Impact External factors Causality Many Few Easy to establish More difficult to establish Demonstrate rigorously Validate transparently Estimate contribution transparently Adapted from A. Miehlbrandt
- 41. Logo Lessons from Samriddhi Samriddhi had its “mock” audit for the compliance of its MRM system with the DCED standard in 2013. = Complies with the Standard = Additional work required = Current system does not address The project was assessed to have 75% of compliance with the standard, which was good. Areas of improvement include: 1)Developing results chain not per value chain but by intervention 2)Collecting specific and detailed information from individual producers, and not from producers‟ groups
- 42. Logo Success factors for Samriddhi’s MRM system 1) Designing MRM system to support decision making at all levels 2) Availability of MRM manual describing and elaborating the “nitty-gritty” of the MRM system 3) Making MRM system part of everyone’s job 4) Giving equal emphasis to qualitative and quantitative data with core indicators 5) Developing an integrated MRM tool 6) Ensuring staff training as a priority 7) Cooperation from the donor and good steering of the project
- 43. Logo Thank you
- 44. Logo Comparing different tools Tool What is it Purpose Sampling guide Tips Observation quick assessment of activity effectiveness gathers qualitative information; collects preliminary information before the use of other tools validate findings from other tools; understand the process of change required for regular monitoring purposes undertaken in the normal course of work or planned effective if a checklist of required information is prepared before a visit May include informal discussions; familiarity with context required In-depth interviews are detailed and require a lot of time to extract explanations In-depth analysis of change Validating facts through details number depends on the context of the study or impact assessment respondents are intentionally chosen for relevant information watch out for coached responses from the respondents use follow up questions and probes to elicit more extensive responses
- 45. Logo Comparing different tools Tool What is it Purpose Sampling guide Tips Key Informant interviews (KII) qualitative in‐depth interviews with individuals who have first‐hand knowledge of the issues collect information from a wide range of individuals who can provide important and significant insight into the issue Sample size ranges from 2‐5 individuals must have first‐hand knowledge about the issues associated with the Intervention diverse set of representatives are preferable Pocket surveys gather information on quantitative indicators when significant rigor is not needed validate the detailed findings of in‐depth interviews, with a larger sample size sample size ranges from 20‐40 questions should be straight‐forward and generally directed at getting facts Focus Group Discussions (FGD) conducted when consensus is needed on a topic, or to get a common view of attendants explore the process of change extensively; provide deeper understanding of why changes are taking place the group consists of 8‐15 people guideline should be prepared on need of info
- 46. Logo Comparing different tools Tool What is it Purpose Sampling guide Tips Formal surveys conducted for 'special cases' which require us to generate specific quantitative information validate the findings of an informal survey or an FGD Depends on the size of the population; usually between 200-400 Questionnaire is structured with closed ended questions, based on the checklist Simple questions to get facts
Notes de l'éditeur
- This is another way of seing it: Imagine that:We are the orange guy at the top – the development agency that wants to create impact The green goal at the bottom is our goal – income increases, job creation, poverty reductionThe blue line is local systems –people, businesses, government, civil society organizations and the relationships, rules, policies and cultural norms that govern how they interact and work togetherThe ball is the series of changes resulting from program activitiesCLICK: It used to be that most PSD programs, kicked the ball really hard – creating and funding parallel systems to achieve a their goalCLICK: But now many PSD practitioners realize that that strategy does not promote sustainable and growing impactCLICK: So, programs kick the ball gently, facilitating the development of local systems so that the local systems themselves carry the ball through the goal – this is much more sustainable than previous strategiesWe want to help develop local systems to carry the ball through the goal, that is our main role
- Gathering information on if and how the ball is rolling down the hill enables PSD projects to monitor if they are on track – if it’s likely that their activities will lead to the expected impact.It’s important to gather information on if and how the ball is rolling down the hill from the top, all the way down to the bottom – because the ball can stop anywhere.We need to know where the ball gets stuck and why it gets stuck. we cannot very precisely foresee how we will influence the economy, we always experiment and hence need to analyze continuously. And one intervention is not likely to take away a constraint; so we need to analyze what did we achieve, what do we need to do next.
- Gathering information on if and how the ball is rolling down the hill enables PSD projects to monitor if they are on track – if it’s likely that their activities will lead to the expected impact.It’s important to gather information on if and how the ball is rolling down the hill from the top, all the way down to the bottom – because the ball can stop anywhere.We need to know where the ball gets stuck and why it gets stuck. we cannot very precisely foresee how we will influence the economy, we always experiment and hence need to analyze continuously. And one intervention is not likely to take away a constraint; so we need to analyze what did we achieve, what do we need to do next.
- the conclusion is that to produce systemic change, sustainable development, we must know the system beforehand and monitor changes in the system afterwards!!M4P is based on capacities and incentives, implies that one needs to understand much more of the system in which one is intervening
- The results chain is a tool to show how programme activities will influence particular systems, how changes in these systems will affect enterprises, and how those changes in enterprises will ultimately reduce poverty and/or contribute to other development goals. Managers need to be explicit about the results chain of their programme: what actions are expected to lead to what results? For more complex programmes, this may include multiple, nested results chains. Log-frames, while originally intended for this sort of function, are not adequate, in their current form, for systemic programmes, as the format is too narrow and linear. In order to establish the basis for measuring impact, programme results chains must show how changes at each level lead to changes at the next level, ultimately impacting on poverty and/or other development goals among defined target group(s). Modelling is a useful tool to enable programme staff to be explicit and deliberate about the system(s) they are working with and how system changes will lead to enterprise changes and poverty reduction and/or other specific development goals. The programme results chain(s) will need regular review, because of changing circumstances and unintended outcomes. A causal model - also often called a theory of change, results chain, causal chain or logic model - is a tool used to consolidate, in one summary graphic: • The overall project goal and purpose • The linkages between project interventions and the effects these are expected to yield in support of the goal and purpose • The assumptions being made about how these linkages work and will play out in practice. Causal models allow project designers, managers and staff to be explicit about the ways in which they expect the project’s interventions to lead to positive effects on impact group and target group members over time. Importantly, in contrast to logical frameworks - which also remain a key aspect of many M&E plans - causal models are non-linear, allowing users to illustrate how interventions, effects and impacts are related to one another vertically, horizontally, diagonally, etc. This flexibility is important for systemic interventions like value chain programs, which often defy linear logic.
- Indicators tighten RCs
- Not one tool per box but combine measuring level changes with the same toolNot one tool but a range of tools because triangulation reinforces findingsDocument the process, not only findings, and state the limitationsMeasuring intervals: once equals zero!ZEWO: Méthodes de recensementDan le cadre d'une mesure de l'efficacité, diverses méthodes de recensement peuvent être utilisées. D'une façongénérale, on peut faire la distinction entre les méthodes de recensement Qualitatives et celles Quantitatives.En-dehors du choix de la méthode, il convient aussi, dans une seconde étape, de considérer si la totalité ouseulement une partie des unités, des groupes cibles ou des cas doivent être pris en compte dans ladite investigation(Unités d'analyse). Enfin, il s'agit d'examiner sous un angle critique la qualité des données recensées (Qualité desdonnées).Mix de méthodesDans la pratique actuelle de la recherche, on a pour habitude de combiner les méthodes quantitatives et qualitatives,de sorte à bénéficier des avantages des deux démarches. On parle alors d’un mix de méthodes ou d’unetriangulation. L’avantage d’une telle combinaison entre méthodes quantitatives et méthodes qualitatives estincontestablement reconnu, devenant la norme de toute mesure de l’efficacité riche en instructions. Ainsi, dans lamesure de l’efficacité d’un programme, il est d’une part possible de distribuer un questionnaire standardisé auxgroupes cibles et, d’autre part, de mener un entretien avec des collaboratrices et collaborateurs ou une discussionde groupe avec des experts. Selon la méthode de recensement retenue, les données collectées auront une formespécifique (compte rendu d’entretien, compte rendu de discussion avec des experts, pourcentages résultant d’unsondage, fréquence d’observation, etc.) et devront donc être analysées à l’aide de procédés d’évaluation appropriés.
- In order to predict the changes in indicators as a result of a programme’s planned interventions and also to measure the changes over time, the M&E team must know the status of the indicators at the time the intervention starts. This can be accomplished through a pre-intervention baseline study of key variables and measures, or it can be done by a retrospective study that compares the present with a previous point in time in order to assess changes (see below)10. Pre-Intervention Baseline Study When: A baseline assessment should be conducted as soon as programme participants can be identified, or as soon thereafter as possible. The key is to establish the participants’ condition before they have been significantly affected by programme activities11; assessment should therefore take place after the provider has been chosen but before significant capacity building has taken place12. How: If comprehensive market research has been conducted (see Section 1.1), it may not be necessary to gather additional data to measure the status of the indicators; there may be sufficient information from the Market Study/initial market research and analysis. Those responsible for establishing the baseline indicators may therefore wish to take the following steps:o Determine what data on the indicators is available from previous market research and analysis o Based on this data, calculate and record a baseline figure for as many indicators as possible o Make a plan to gather any additional data needed to complete calculations, that is not available from previous market research o Gather the relevant data e.g. though surveys, interviews, focus group discussions etc. (see Section 8.1) o Process the data and calculate the status of the remaining indicators o Make a record of the status of all the indicators at the baseline Establishing a Retrospective Baseline While quantitative data from two or more points in time are important for measuring or estimating change more reliably, in some cases such a pre-intervention baseline study may not be feasible, for example where the results chain or area of geographic concentration change significantly during implementation. If a baseline was not established at the time of project design, programmes will need to be innovative. Gathering information from a variety of sources will enable you to triangulate to gain a reasonably accurate picture of the base situation. For example13: - Recall: either in individuals or groups, ask people about their recollection of a situation and what would have happened if there had been no project. However, information that depends on recall over an extended period can be unreliable: “this is especially true for measuring change in areas where recall is weak, or if attitudes, opinions and behaviours are likely to change over time. For example, recall data on income, regular expenditures (e.g., on food) or self-esteem are not very reliable, especially when using a long reference period.”14 - Written records of partners: BDS providers, BMOs and governments should all have written records of some kind that throw light on where they were at the start of a project. - Other written and verbal sources: especially in more developed economies, there may be general economic and other data that can be drawn on.