1. Unpacking Impact and Influence:
Approaches to Monitoring & Evaluating
April 28, 2014
TTI PEC Africa workshop - Nairobi
Courtney Tolmie (Results for Development Institute)

2. A few notes to start
 This session focuses on Organization M&E. That said, most of the
lessons can be scaled down to the project level.
 You will guide the session – and giving real time feedback:
2 | R4D.org
Way too fast.
Please slow
down.
Just right.
You can
speed up.

3. How do we think about Impact and Influence?
1. What IMPACT does your organization seek to have?
2. What and who do you work to INFLUENCE as an organization?
3. What is the link between IMPACT and INFLUENCE in the work
that you do?
3 | R4D.org

4. Monitoring and Evaluation – the link to Impact and
Influence
In other words, a way of following whether you are doing what you
planned to do and whether you are achieving the IMPACT you wanted
to, including Policy INFLUENCE.
4 | R4D.org
Monitoring: a continuing function
that aims primarily to provide the
management and main
stakeholders of an ongoing
intervention with early
indications of progress, or lack
thereof, in the achievement of
results.
Evaluation: systematic and objective
assessment of an on-going or completed
project, program, or policy, and its
design, implementation and results. The
aim is to determine the relevance and
fulfillment of objectives, development
efficiency, effectiveness, impact, and
sustainability.
Definitions from the World Bank -
http://siteresources.worldbank.org/INTBELARUS/Resources/M&E.pdf

5. What Makes an organization-level M&E System Effective?
Defining Impact
and how we get
there (TOC)
Defining how to
measure impact
& outputs
(indicators)
Carrying out
M&E (Methods)
Using results to
improve work
(Feedback Loop)
5 | R4D.org

6. M&E Systems – Defining a Theory of Change
Defining Impact
and how we get
there (TOC)
Defining how to
measure impact
& outputs
(indicators)
Carrying out
M&E (Methods)
Using results to
improve work
(Feedback Loop)
6 | R4D.org
Defining a Theory of Change
 Clarifying your organization
objectives and desired
impacts
 Mapping out what is
required to achieve these
impacts
 Linking to the activities and
actions your organization
will take to achieve the
impact
 Also highlighting
assumptions (including
context issues)
 Similar structures -
logframes

7. M&E Systems – Defining Indicators
Defining Impact
and how we get
there (TOC)
Defining how to
measure impact
& outputs
(indicators)
Carrying out
M&E (Methods)
Using results to
improve work
(Feedback Loop)
7 | R4D.org
Defining Indicators
 Turning what you hope to
achieve into what you can
actually measure
 Making indicators SMART
 Value (and tradeoff) in
collecting indicators on
impact and outputs

8. M&E Systems – Designing Methods
Defining Impact
and how we get
there (TOC)
Defining how to
measure impact
& outputs
(indicators)
Carrying out
M&E (Methods)
Using results to
improve work
(Feedback Loop)
8 | R4D.org
Defining and designing Methods
 Identifying how to actually
collect data on the
indicators that you have
defined
 Timing – monitoring,
evaluating or both
 Things to consider –
capacity, resources, what
you are trying to achieve

9. M&E Systems – Ensuring Feedback Loops
Defining Impact
and how we get
there (TOC)
Defining how to
measure impact
& outputs
(indicators)
Carrying out
M&E (Methods)
Using results to
improve work
(Feedback Loop)
9 | R4D.org
Ensuring Feedback Loops
 The most forgotten step of
M&E systems
 Results for M&E can help
improve programs and
ultimately impact
 Best to have formal
structure to build results
back into future work
 Moving from culture of
M&E to MEL

10. Mapping areas for development
 Based on your existing M&E system, assess how you are doing in
each element of effective M&E
 Scale from “we don’t do this at all” to “we do this very well on all
areas of work”
 10 minutes to map your organization – then we will come back to
review
10 | R4D.org

11. Defining Theories of Change
11 | R4D.org

12. Defining Theories of Change
 Examples from other organizations
 Tips from the group about how to do it well
 Ideas for moving forward
12 | R4D.org

13. 13 | R4D.org
Build evidence on
better ways to
budget for higher
education
Share
recommendations
with Minster of
Education and
Finance
New budget passed
with
recommendation
It would be easier if organization TOCs looked like this:
But this is not realistic
WHY NOT?
• TTs are trying to achieve bigger changes than just this
• A single TT is going to likely need to take a lot of different actions to
get to that end impact
• A single TT is not going to individually achieve this impact

14. 14 | R4D.org

15. 15

16. Note about Theories of Change
 A good TOC for influencing policy is incredibly complex.
 Your organization may not try to affect all parts of the TOC – and
thus may not choose to do M&E for all factors that go into policy
influence.
 But good to be aware.
16

17. Defining Theories of Change
 Examples from other organizations
 Tips from the group about how to do it well
 Ideas for moving forward
17 | R4D.org

18. Defining Theories of Change – TOC Pitfalls to avoid from
Stanford Social Science Review
1. Confusing accountability with hope
2. Creating a mirror instead of a target
3. Failing to take external context into account
4. Not confirming the plausibility of your theory
5. Creating a theory that is not measurable
6. Assuming you have figured it all out
18 | R4D.orghttp://www.ssireview.org/blog/entry/six_theory_of_change_pitfalls_to_avoid
What lessons can you share from your organizations –
things that have worked and what has not worked?

19. Defining Indicators
19 | R4D.org

20. Defining Indicators
 What types of indicators work well, especially for measuring
policy influence and impact?
 Tips from the group about how to do it well
 Ideas for moving forward
20 | R4D.org

21. Examples of good indicators – for the full organization
 While the focus of this program is on PEC, it is worth noting that
there are multiple components of a think tank’s performance worth
monitoring:
21 | R4D.org
From Raymond Struyk’s forthcoming book – Improving Think Tank Management
Public
Policy
Perspective
Funder
Perspective
Internal
Business
Perspective
Innovation &
Learning
Perspective
Financial
Perspective

22. Examples of good indicators – for the full organization
 While the focus of this program is on PEC, it is worth noting that
there are multiple components of a think tank’s performance worth
monitoring:
22 | R4D.org
From Raymond Struyk’s forthcoming book – Improving Think Tank Management
Public
Policy
Perspective
Funder
Perspective
Internal
Business
Perspective
Innovation &
Learning
Perspective
Financial
Perspective

23. Examples of good indicators for PEC impact and influence
 Some helpful indicators from the center for International Governance Innovations:
23 | R4D.org
From www.cigionline.org/blogs/tank-treads/communications-and-impact-metrics-think-tanks
Indicators of Exposure:
• Media mentions
• # and type of publications
• Scholarly citations
• Government citations
• Think tank ratings
Indicators of Resources:
• Quality, diversity, stability of funding
• Characteristics of researchers
• Quality, # of networks, partnerships
Indicators of Demand:
• Events
• Digital traffic and engagement
• Access to officials
• Publications sold, downloaded
Indicators of
Policy Impact &
Quality of Work:
• Policy
recommendations
considered or
adopted
• Testimonials
• Quality of think tank
work

24. Other indicators we have seen used
24 | R4D.org
If the objective is …
Specific policy
recommendation
implemented
Change in the debate
… Possible indicators are:
 Invitation to share findings with specific audiences
(government, others)
 Attendance of decision makers at dissemination
events
 (if available) Information on issue being debated in
policy discussions
 Request for research by policymakers on the new
topics
 Greater debate on topic in media
 (if available) Information on issue being debated in
policy discussions
Longer term – access
to policymakers
 Individual meetings with policymakers

25. A note about impact indicators …
 In general, these are not going to be attributable.
 Attribution is difficult
 Almost without exception, it takes a village.
 This does not mean impact indicators are not helpful – especially if
seeking to improve results.
25 | R4D.org

26. Defining Indicators
 What types of indicators work well, especially for measuring policy
influence and impact?
 Tips from the group about how to do it well
 Ideas for moving forward
26 | R4D.org

27. Defining Indicators
 What types of indicators work well, especially for measuring policy
influence and impact?
 Tips from the group about how to do it well
 Ideas for moving forward
27 | R4D.org

28. Designing methods for M&E
28 | R4D.org

29. Designing methods for M&E
 What common options are others using?
 Tips from the group about how to do it well
 Ideas for moving forward
29 | R4D.org

30. Options for M&E approaches
30 | R4D.orgTable from ODI Background Note – A guide to monitoring and evaluating policy influence (Harry Jones)
Does not include common impact evaluation methods – RCTs,
process tracing, etc.

31. Designing methods for M&E
 What common options are others using?
 Tips from the group about how to do it well
 Ideas for moving forward
31 | R4D.org

32. Designing methods for M&E
 What common options are others using?
 Tips from the group about how to do it well
 Ideas for moving forward
32 | R4D.org

33. Ensuring Feedback Loops
33 | R4D.org

34. Ensuring feedback loops
 Do results feed back into the think tank automatically?
 Tips from the group about how to do it well
 Ideas for moving forward
34 | R4D.org

35. Connecting M&E Results to Think Tank Changes
 Two potential reasons to undertake M&E:
 To achieve the second, M&E can’t stop at data analysis – it needs to
feed back into the think tank.
35 | R4D.org
ACCOUNTABILITY to
funders and others
BETTER RESULTS for think
tank impact and influence

36. Questions to ask to design a strong Feedback Loop
(Knowledge Management)
36 | R4D.org
The Question
WHO should hear the
M&E results?
WHEN should they
get the information?
The Answer
 (1) Who is the best position to exploit the
information – capacity and interest
 (2) More generally, anyone who SHOULD know
 (1) At a time when they can act – capacity and
opportunity
 (2) Generally more than once
WHAT should they
receive?
 (1) Format – useful but not necessarily fancy
 (2) Ideally comparisons to baseline data
In general, the Feedback Loop is more likely to be effective if designed early –
proactive rather than reactive.
Built from Raymond Struyk’s forthcoming book – Improving Think Tank Management

37. Ensuring feedback loops
 Do results feed back into the think tank automatically?
 Tips from the group about how to do it well
 Ideas for moving forward
37 | R4D.org

38. Ensuring feedback loops
 Do results feed back into the think tank automatically?
 Tips from the group about how to do it well
 Ideas for moving forward
38 | R4D.org

39. A final word
39 | R4D.org

40. Conclusion
 M&E is not easy – and requires capacity. This is especially true for
organization-level M&E and for policy influence M&E.
 Match M&E plan to capacity
 Moving from M&E to MEL (L = “learning”)
 Linking the two objectives of M&E – accountability and better results
 Some methods and indicators can address both – where
possible, don’t double your work
 Example – how can you use the IDRC monitoring indicators for
PEC to help you achieve better results?
40 | R4D.org

41. Other Great Resources (in addition to those referenced on
this powerpoint)
 ODI work on RAPID
 On Think Tanks page on Monitoring and Evaluating Influence
(http://onthinktanks.org/topic-pages/topic-page-monitoring-and-
evaluating-influence/)
 Research 2 Action PEC page
41 | R4D.org