Patrick Grémillet, UNDP Bratislava Regional Center, Management Practice 2011

1. Results-Based Management Patrick Grémillet Bratislava Regional Center Management Practice 2011

2. Session Schedule Results-based Management (RBM) Principles Introduction to RBM Setting a Results Management Strategy Problem Analysis and Results Mapping UNDP planning instruments Results Matrix Understanding the RBM Typology Differentiating outcomes and outputs Indicators, baseline and targets August 2011 Management Practice - BRC 2

3. What is a Result ? August 2011 Management Practice - BRC 3 Results are changes in a state or condition that derive from a cause-and-effect relationship. There are three types of such changes (intended or unintended, positive and/or negative) that can be set in motion by a development intervention – outputs, outcomes and impacts. - UNDG agreed RBM terminology

4. What are Result?* RESULTS are changes in a state or condition that derive from a cause-and-effect relationship. CONTROL & RESPOSIBILITY MAKING A DIFFERENCE * using UN harmonized terminology (based on OECD/DAC Glossary of Key Terms in Evaluation and Results-based Management)

5. August 2011 Management Practice - BRC 5 What is results based management? RBM is a management strategy by which all actors ensure that their processes, products and services contribute to the achievement of desired results (outputs, outcomes and higher level goals or impact) A key component of RBM is performancemonitoring which is to objectively measure how well results are being achieved, and report on measures taken to improve them.

6. Why RBM? Stated rationale/intended gains: Improved focus on results instead of activities Improved transparency Improved accountability Improved measurement of programme achievements (performance rather than utilization) Enhanced strategic focus Industry standard To get more funds!! August 2011 Management Practice - BRC 6

8. General problems when applying RBM? Difficult to apply Difficult to learn Difficult to integrate Difficult to revise (... or reluctance to revise? ) Difficult to measure Difficult to ‘attribute’ (at outcome level, the UN is accountable but not fully responsible) August 2011 Management Practice - BRC 8

9. Common elements of RBM Problem analysis to understand causes Stakeholder analysis Structuring of programmes around a chain of desired results - addressing causes Causality in the chain of results (if… then logic) Use of ‘change language’ (future conditional) Reliance on indicators to measure performance Costing of results rather than isolated activity budgeting August 2011 Management Practice - BRC 9

10. The RBM life-cycle approach August 2011 Management Practice - BRC 10

12. Setting a Results Management Strategy

13. Problem Analysis and Results Mapping

14. Results Matrix

15. Understanding the RBM Typology

16. Differentiating outcomes and outputs

18. Strategic Priority Setting August 2011 Management Practice - BRC 13 Capacity Value 1. “Just do it”/strategic priority Major national challenge UNDP comparative advantage 2. Potential high priority, if consensus can be built 3. Potential high priority, if others cannot meet demand and internal capacity development is feasible 4. Lower priority: does not meet major national challenge Alignment of key actors to support action Support

20. Setting a Results Management Strategy

21. Problem Analysis and Results Mapping

22. Results Matrix

23. Understanding the RBM Typology

24. Differentiating outcomes and outputs

26. Formulate problem in a neutral manner “Minorities and marginalized groups do not have the right to vote.” Versus “Minorities and other marginalized groups do not participate in elections.” Or “Low levels of participation by minorities in elections.” August 2011 Management Practice - BRC 16 33-35

27. Cause-effect analysis August 2011 Management Practice - BRC 17 Effects Causes Are there capacity constraints? – policy level, institutional, individual Are there Social or Cultural Constraints?

28. Causality analysis – Why? Negative outcomes, manifestations of problems, unfulfilled rights Immediate causes Affecting individuals and households Underlying causesand capacity issues Policies, laws, budgetsSystems for service deliveryBehaviors and practices, low household incomes Root causes Beliefs, attitudes, culture, traditionsNatural resources, natural disastersPolitical and economic systems, ideologies, conflict August 2011 Management Practice - BRC 18

29. Problem Tree August 2011 Management Practice - BRC 19 Manifestations Malnutrition & death Immediate causes effect Inadequate dietary intake Disease Underlying causes Insufficient food security Inadequate Maternal & Child Care services Insufficient health services & unhealthy environment Resource Control + Organizational structures Root causes Political, Ideological, Economic structures cause

30. Results Map August 2011 Management Practice - BRC 20 Poor rural women earn meaningful wages and have improved life skills Women gain marketable skills in traditional and non-traditional areas Women learn good work attitudes and workplace behavior Women develop self-confidence Potential employers educated and committed to programme Women trained in vocational skills Some women receive traditional schooling Advocacy and sensitization initiative Women enroll Long-term Outcome Intermediate Outcomes Partnerships Outputs Small stipend provided to women Child care services provided Programme info provided to women Training agency, child care agency, NGO, and National Women’s Bureau etc collaborate to design and deliver programmes

31. Session Schedule Results-based Management (RBM) Introduction to RBM Key Principles RBM Life cycle UNDP planning instruments Results Matrix Understanding the RBM Typology Differentiating outcomes and outputs Indicators, baseline and targets August 2011 Management Practice - BRC 21

33. Results matrix streamlinedPLANNING MONITORING National plans/strategies; PRSP; … Country Analysis UN Development Assistance Framework (UNDAF) Country Programme Document (CPD) Country Program Action Plan (CPAP) Annual Work Plan (AWP) Jointprogrammes Projectdocument

34. UNDAF BEFORE…. Good Governance and the protection of Human Rights By 2013, achieve effective participation of citizens, and government accountability and integrity Laws on public demonstrations and freedom of association improved in accordance with international standards. Increased participation of civil society and citizens in decision making Strengthened community participation in the planning and implementation of local development activities Capacity of civil society strengthened through civic education and engagement with Parliament Advocacy programmes conducted for accession to the UN convention against corruption. Corruption in government management significantly reduced Advocacy programmes conducted for the adoption of a national anti-corruption law Effectiveness of decentralized government structures improved to deliver basic services Government assisted to meet its obligation under the International Covenant on Economic, Social and Cultural Rights. Capacities of communes for decentralized planning, management and delivery of public goods and services further strengthened. August 2011 Management Practice - BRC 23

37. Verbatim from UNDAFAugust 2011 Management Practice - BRC 25

38. Session Schedule Results-based Management (RBM) Introduction to RBM Key Principles RBM Life cycle UNDP planning instruments Results Matrix Understanding the RBM Typology Differentiating outcomes and outputs Indicators, baseline and targets August 2011 Management Practice - BRC 26

39. Ways how to depict the Results chain Narrative IMPACTA. XXXXXXXX OUTCOMEA.1. XXXXXXXXA.2. XXXXXXX XXX OUTPUTSA.1.1.XXX XXX XXXXA.1.2. XXXXXXXA.2.1. XXXXXXXX Logframe Results Tree Results Pyramid

41. Common results language to describe changes

42. Collective accountability increases as you move up the chain of results towards outcomes and impacts (Key message  No agency can do it alone!)

43. A results matrix is a mean not an end

44. A results matrix is contextual August 2011 Management Practice - BRC 28

45. August 2011 Management Practice - BRC 29 Our counterparts may use different language (i.e. logframe), but the basis is the same…

46. CPAP Template August 2011 Management Practice - BRC 30 UNDAF/CP D outcome #1: Extracted verbatim from the UNDAF/CPD Relevant UNDP Strategic Plan result: Extracted verbatim from the UNDAF / CPD Indicative Resources by outcome (per year, US$) Outcome - le vel indicators, baselines and Implementation modality(ies) and targets for UNDP contribution to Country programme outputs implementing partner(s) UNDAF/CPD outcomes Year Year Year Year Year Total Output 1: State the implementing modality Regular Resources (i.e. NIM, UN Agency, NGO, IGO or Annual Targets (recommended for multi - DIM); And year outputs) Government and/or UN · Gender Marker Rating and [2] Implementing Partner [1] Motivation Output 2: State the implementing modality Other Resources (i.e. NIM, UN Agency, NGO, IGO or Annual Targets (recommended for multi - DIM); And year outputs) Government and/or UN · Gender Marker Rating and Implementing Partner Motivation UNDAF/CPD outcome #1: Extracted verbatim from the UNDAF/CPD Relevant UNDP Strategic Plan result: Extracted verbatim from the UNDAF / CPD Regular Resources Other Resources [1] Gender Rating: 3 - Gender equality is a principal objective of the output; 2 - Gender equality is a significant objective of the output; 1 - Outputs that will contribute in some way to gender equality but not significantly; 0 - Outputs that are not expected to contribute noticeably to gender equality. Include a one sentence motivation as to the reason for the chosen rating. [2] State also IGO Implementing Partners who have already signed SBEAA with UNDP. Other IGOs and NGOs will not be mentioned here in CPAP. They would be selec ted through a competitive process later.

47. August 2011 Management Practice - BRC 31 UNDP Projects Results & Resources Framework

48. A few caveats The results matrix is a storyline It is a description of a programme or project strategy Shows the intended paths we think are required to achieve desired results The storyline comprises results, indicators & targets and assumptions & risks August 2011 Management Practice - BRC 32

50. Not measurable

51. Are new resultsAugust 2011 Management Practice - BRC 33

52. Results Chain August 2011 Management Practice - BRC 34 Inputs Outputs outcomes Impact > Experts; equipment; funds People capacities improved; Laws/policies drafted Policies adopted; Laws enacted; Conditions improved health/ longevity Partnerships and other interventions

53. Results Chain August 2011 Management Practice - BRC 35

54. Results Chain August 2011 Management Practice - BRC 36 Positive and negative, primary and secondary long-term effects produced by a development intervention, directly or indirectly, intended or unintended. Impact: Human Change Outcome: The likely or achieved short-term and medium-term effects of an intervention’s outputs. Institutional & Behavioural Change Outputs: Tangible products and services which result from the completion of activities within a development intervention. Products & Services Skills & Abilities

55. Results Chain August 2011 Management Practice - BRC 37 Positive and negative, primary and secondary long-term effects produced by a development intervention, directly or indirectly, intended or unintended. Changes in the lives of people Impact: Human Change Institutional Change: values, ethic, rules, laws – associated with/to institutional performance, access.. Behavioural change: knowledge, skills acquisition, practices (individual level) Outcome: The likely or achieved short-term and medium-term effects of an intervention’s outputs. Institutional & Behavioural Change Outputs: Tangible products and services which result from the completion of activities within a development intervention. Products & Services Skills & Abilities

56. Results Chain August 2011 Management Practice - BRC 38 Positive and negative, primary and secondary long-term effects produced by a development intervention, directly or indirectly, intended or unintended. Changes in the lives of people Reduced infant and maternal mortality by 2018 Impact: Human Change Institutional Change: values, ethic, rules, laws – associated with/to institutional performance, access.. Behavioural change: knowledge, skills acquisition, practices (individual level) Outcome: Improved provision of public sanitary services to rural communities by 2015 The likely or achieved short-term and medium-term effects of an intervention’s outputs. Institutional & Behavioural Change Outputs: Tangible products and services which result from the completion of activities within a development intervention. National Public Works Agency has the management systems, equipment, and skills to provide sanitation services to rural communities Products & Services Skills & Abilities

57. August 2011 Management Practice - BRC 39 Results Like… Focus @ Timeframe more 5-10 yrs Impact Impact Impact then Collective Accountability if 5 yrs UN Outcome UN Outcome UN Outcome then <3 yrs Output Output Output if then less <1 yr Activity Activity Activity if A Typology for RBM HIV incidence reduced Human! Assumptions Leadership empowered Institutional/ Behavioural Skills of Nat’l Aids Comm. strengthened Operational/ skills, abilities, products & services Train 250 district AIDS officers

58. August 2011 Management Practice - BRC 40 more Collective Accountability less A Typology for RBM: Governance Results Like… Focus @ Timeframe then Impact More transparent governance 5-10 yrs Human! if then Outcome National capacity to implement governance reforms increased 5 yrs Institutional/ Behavioural if then Output Legislative code for local self-governance developed <5 yrs Operational/ skills, abilities, products & services if Activity - Training legislators - National consultation - Assessment of laws.. <1 yr

60. Training for chambers of commerce - National consultation Activity <1 yr

62. Community orientation conducted

63. Small grants provided <1 yr

64. Outcomes Let’s look at some examples…

66. Typical pitfalls Wordy (..and no change language) To promote equitable economic development and democratic governance in accordance with international norms by strengthening national capacities at all levels and empowering citizens and increasing their participation in decision-making processes Too ambitious Strengthened rule of law, equal access to justice and the promotion of rights Containing multiple results The state improves its delivery of services and its protection of rights—with the involvement of civil society and in compliance with its international commitments August 2011 Management Practice - BRC 45

67. Typical pitfalls Wishy-washy (ie. Support provided to improve..) Support to institutional capacity building for improved governance So general, they could mean anything To promote sustainable development and increase capacity at municipal level Overlapping with National goals/ MDGs (impacts) Substantially reduce the level of poverty and income inequality in accordance with the MDGs and PRSP Confusing means and ends Strengthen the protection of natural resources through the creation of an enabling environment that promotes sound resources management August 2011 Management Practice - BRC 46

68. Examples Legal and regulatory framework reformed to provide people with better access to information and communication technologies. The poor in x region have better access to capital and other financial services. Reduction in the level of domestic violence against women by 2016 Increased regional and sub-regional trade Higher and more sustainable employment and income for urban slum dwellers. August 2011 Management Practice - BRC 47

69. Examples By the end of 2010, user-friendly and sustainable health care and nutrition services are provided in compliance with international standards at national and sub-national levels. Increased access to and completion of basic education, especially for girls. By 2012, increased and more equitable access to and utilization of quality, integrated and sustainable basic services by the poor and vulnerable. August 2011 Management Practice - BRC 48

70. Outputs Tangible, deliverable, promises

72. Must be achieved within the project period

74. 3 to 6 outputs per agency outcome

75. Unless under a joint programme, outputs are NOT collective resultsAugust 2011 Management Practice - BRC 50

77. Indicators usually coming from existing data, assessments, analysis or from routine progress reports

78. Indicators can be ‘yes-no’

79. Or qualitative

80. 1 to 4 indicator enough - the fewer the better to reduce costsAugust 2011 Management Practice - BRC 51

81. Short exercise (10 mins) Please reword/optimize the following draft output statement: “To strengthen the capacity of civil servants to do X by undertaking Y” August 2011 Management Practice - BRC 52

82. Refining results… August 2011 Management Practice - BRC 53 To strengthen the capacity of civil servants to do X by undertaking Y,… Let’s use results language to emphasis the future condition we want to achieve. The capacity of civil servants is strengthened to do X by undertaking Y,… All civil servants, everywhere? Can you be more specific? Are there particularly weak or under-resourced civil servants we should emphasise? The capacity of civil servants in the 4 poorest districts is strengthened to do X by undertaking Y,… We can take out information that relates to either strategy or activities. The capacity of civil servants in the 4 poorest districts is strengthened to do X by undertaking Y,… Now, let’s try bringing the subject of change to the front, and shifting from passive to active language. Civil servants in the 4 poorest districts are better able to X

83. Typical pitfalls Wordy Unclear logic and confused indicators Over-ambitious Passive voice and wishy-washy wording (ie. Support provided to improve, including but not limited to..) Overlapping with Country Program outcomes or repeating activities August 2011 Management Practice - BRC 54

84. Wordiness August 2011 Management Practice - BRC 55

85. Good Outputs examples (?) Market-based vocational training programme developed A legislative framework and code for local self governance is drafted National budget process more effectively incorporates inputs from local governance structures All immunization centers have a functioning cold-chain, and adequate supply of vaccines and Vitamin A. August 2011 Management Practice - BRC 56

87. Study of environment-poverty linkages completed

88. Police forces and judiciary trained in understanding gender violence

89. National, participatory forum convened to discuss draft national anti-poverty strategy

90. National human development report produced

91. Revised electoral dispute resolution mechanism established

92. Business processes reengineered

95. Objectively verifiable, repeatable measures of a particular condition

96. They force clarification of what is meant by the result …….the fine print!

97. Must be accompanied by baselines and targetsAugust 2011 Management Practice - BRC 59

98. Baseline, Target and Achievement August 2011 Management Practice - BRC 60 Performance Commitment Achievement At end of period Planned Level of Achievement Target Achievement Current Level of Achievement Baseline

99. What to specify for each indicator? Unit of analysis; Existing baseline information; Target for subsequent comparison; Expected perceptions or judgments of progress by stakeholders; Detailed description of expected conditions or situations to be observed; 1-4 indicators for each result are adequate – fewer the better to reduce cost. August 2011 Management Practice - BRC 61

100. Indicators ‘indicate’ that change is happening or not happening They can Clarify the scale and scope of a result in the results framework Demonstrate progress when things go right Provide early warning when things go wrong Assist in identifying changes that need to be made in strategy and practice Inform decision making Facilitate effective evaluation August 2011 Management Practice - BRC 62

102. Monitoring reports

103. SurveysBaseline: -(based on analysis, capacity assessment, reports, national statistics etc) Target: actual value corresponding to each indicator, defined per year

104. Types of indicators August 2011 Management Practice - BRC 64

105. August 2011 Management Practice - BRC 65

106. Indirect (Proxy) Indicators Direct indicators are preferable. However, they do sometimes not exist, are too expensive or inefficient to obtain. In such cases, indirect indicator have to be used. August 2011 Management Practice - BRC 66

107. August 2011 Management Practice - BRC 67 OUTPUT INDICATORS

108. Indicators and Means of Verification A key test in ensuring whether an indicator is really good is to define “means of verification” for each indicator; Are data sources available? What does it take to obtain the data? Who needs to be involved? How much will it cost to obtain the data? If means of verification are unclear or unrealistic, the indicator and, possibly, the result need to be revisited; The definition of results, indicators and means of verification is an iterative process. August 2011 Management Practice - BRC 68

109. Baselines and Targets The indicator should be neutral The target is what signals how much change and in what direction The baseline and target should use the same unit of measurement as the indicator August 2011 Management Practice - BRC 69

110. The “SMART” way… Specific Measurable Attainable Relevant Trackable August 2011 Management Practice - BRC 70

111. What are good indicators? SPECIFIC The indicator needs to be as specific as possible in terms of quantity, quality, time, location, target groups, baseline and target for the indicator MEASURABLE Will the indicator show desirable change? Is it a reliable and clear measure of results? Is it sensitive to changes in policies & programmes? Do stakeholders agree on exactly what to measure? ACHIEVABLE Are the result(s) realistic and based on risk assessment, partnership strategy and other factors contributing to the underlying result RELEVANT Is it relevant to the intended result? Does it reflect the expectations and success criteria for change in the target groups? TRACKABLE Are data actually available at reasonable cost & effort? Can proxy indicators be used? Are data sources known? August 2011 Management Practice - BRC 71

112. Exercise for 10 mins

113. August 2011 Management Practice - BRC 73 Exercise: Result Chain

114. August 2011 Management Practice - BRC 74 Exercise: Result Chain

115. Thank you! Resource Person Patrick Grémillet Management Practice Coordinator UNDP Bratislava Regional Center Patrick.gremillet@undp.org This presentation is using slides/inputs from training materials developed by the UN Staff College and RBM consultants August 2011 Management Practice - BRC 75