2. Table of Contents
Some Basic Things 4
References 72
Using Experimental Method 66
Quasi-experimental Method 69
Other Evaluation Approach? 71
Re-thinking Our M&E Practice 5
A. Changing the M&E Approach and Mindset 6
B. Donor: Let’s Get Better Together 9
Strengthening M&E:Insights for Donor 11
C. Partner Organization: Building the Learning Culture! 17
Strengthening M&E: 13 Insights for Organization 20
D. M&E for Partnership Program 27
M&E: Weaving Learning Into Change 33
A. Essentials of Monitoring, Evaluation & Learning 34
B. M&E: Some Tools, Methods & Approaches 35
Most Significant Change 36
Outcome Mapping 39
Theory of Change 44
C. Hybrid Approaches 49
Measuring the “Immeasurable” 53
A. Utilizing Good Governance Indicators 54
B. Tackling the Evaluation Challenge 60
C. Methodological Challenges in Evaluating Governance Intervention 63
D. Rigorous Evaluation of Governance Intervention: Is it Possible? 66
Chapter 2
Chapter 3
Chapter 1
3. List of Picture
Picture 1. Change in the Approach & Practice of Monitoring and Evaluation
List of Box
Box 1. IKAT US Program Approach in Linking Partners & Manage Knowledge
Box 2. Evaluation Approach in Oxfam GB
Box 3. AWID’s Strategic Initiative
Box 4. Experiences of IKAT US’s Lead Organizations in Initiating Partnership
Box 5. POWER: Showing Leadership & High Quality Training based on Needs
Box 6. Illustrative Use of MSC for ASIA CALLING
Box 7. Outcome Mapping in ACCESS Phase II
Box 8. Illustrative Use of ToC in IKAT US Partnership Program
Box 9. Experience of HIVOS: Building Southern Organizational Capacity Through Theory of
Change
Box 10. Women’s Learning Partnership (WLP) M&E Framework
Box 11. AWID (Association for Women’s Rights in Development) ME Framework
Box 12. Olken’s Study on Corruption in Road Infrastructure Projects in Indonesia
Box 13. The Evaluation of Impact of international Election Observers on Election Quality in
Indonesia
Box 14. The effect of information on the performance of local government officials on people’s
participation during decentralization in the Philippines
List of Tables
Table 1. Users and Uses of Governance Indicators
Table 2. List of Governance Indicators and Online Resources
Table 3. Four Quasi-experimental Methods
3
4. Some Basic Things…
About this Guide
This Guide will not talk much about definition of monitoring & evaluation nor types of
monitoring of evaluation in great details. There are many good monitoring & evalua-
tion guidelines out there that serve this purpose. The goal of this user guide is to
extract monitoring and evaluation lessons learned from the common practice and
share it to all of you as a part of learning process.
Nevertheless, it is important to lay down some basic principles before we move
forward in this document:
How do we define Monitoring & Evaluation?
Although the term“monitoring and evaluation”tends to be run together as if it is only
one integrated process, monitoring and evaluation are, in fact, two distinct sets of
organizational activities, related but not identical.
4
Source: Designing for Results: Integrating ME in Conflict Transformation Program, Search for Common Ground, p.96
What is it? Ongoing collection and analysis of data on
progress toward results, changes in the context,
strategies and implementation
Reviewing what has happened and why, and
determining relevance, efficiency, effectiveness,
impact, etc.
Inform day-to-day decision-making
Accountability and reporting
Strengthen future programming
Provide evidence of success
Deepen our understanding of how and why things
work
Program Staff and/or Partners and/or Participants External consultant, staff, participants or combination
of these groups
At design stage Core decisions taken at design stage and refined prior
to implementation
Why do it?
Who does it?
When to plan?
Throughout the program – periodically, frequently
or continuously
Mid-term (formative)
Completion (summative)
After completion (impact)
When to
implement?
Level Monitoring Evaluation
6. A. Changing the M&E Approach and Mindset
Limited time
Lack of technical
capacity
“Getting something wrong is not a crime. Failing to learn from past mistakes because
you are not monitoring and evaluating is.”
Source: www.civicus.org/new/media/Monitoring%20and%20Evaluation.pdf
No one will argue about the importance of monitoring and evaluation. However,
many get to really feel and experience the true benefit of doing monitoring & evalua-
tion.
So the question remains: why Monitoring & Evaluation fail to deliver its benefit for
development purposes?
There can be several reasons forwarded in answering this question:
These reasons are basically true, but there is actually other fundamental cause if we
trace further. It goes beyond the internal issue of one’s organization such as limited
capacity and limited resources. And it has to do with the way the overall development
sector - aid agencies and non-profit organizations - view M&E itself:
• M&E is perceived as a mean to justify the development aid expenditures and
thus, focus limitedly on logical framework activities and financial reporting, such as
financial audits, field visits, mid-term review and end of project evaluation.
• M&E is treated as an “add-on” to the project implementation, a mechanical
operation separate from daily program operations and done separately by an M&E
Officer or M&E Unit. Investment to M&E is less prioritized and given by very limited
M&E activities.
6
Limited resources
M&E is less-priority
7. • M&E is also perceived as a measurement of performance and success. Good
news is preferred over the bad ones, leading to a level of M&E that gives minimum
assurance of effectiveness without raising difficult questions.
• We want to achieve outcome but we want a short cut. M&E is dominated by
questions of measuring outcomes because we want to achieve the outcome level.
However little investment is made in developing a proper project/program interven-
tion (design) with proper project period that enables the achievement of outcome. As
a result, there is often a weak link between the project’s output and outcome. Further-
more, little investment is made also to encourage and facilitate a robust evaluation
method to really produce lessons learned.
7
Added with the fact that organization has limited resources and capacity to carry out
M&E, these have led to bad practices of M&E. This phenomena might satisfy the
requirement of donors, but have little or no real benefit for the organization,
intended beneficiaries and even the donors themselves because only little learning
can be captured from it. A change of approach and mindset is needed not only within
the internal organization but also the donors so that M&E could be useful!
8. 8
Source: Designing for Results: Integrating ME in Conflict Transformation Program, Search for Common Ground, p.96
Picture 1. Change in the Approach & Practice of Monitoring and Evaluation
9. B. Donors: Let’s Get Better Together!
9
The key principle is to Get Better Together. Donors/aid agencies have to acknowl-
edge that most partner organizations have limited resources & capacity to achieve
unrealistic targets. Thus, more active role of donors/ aid agencies is imperative.
Commitment and support, not only in terms of funds, are needed. Pursuing real
partnership and linking partners to knowledge and other resources are some inno-
vative ways to move forward.
One of the biggest issues the donors/aid agencies here and often becomes a chal-
lenge is in managing relations between donors/aid agencies and its partner
organizations (grantees). Even though these organizations are called as ‘partners’
but in reality, they are still trapped in the donor-grantee relations - where donors
avoid to invest in strengthening grantee’s or partner’s management and infrastruc-
ture whereas on the other hand partners are afraid to speak the truth because it might
risk their funding. They are trapped by a mere compliance to donors’required reports
in which substantially don’t lead to performance improvement Without a well
planned M&E framework and execution, reports are least useful other than just to
fulfill the formal accountability requirement. Partners then spend time and resources
on measurement that doesn’t do any good, instead of measurement that fosters
learning.
Establishing international standards for methodological rigor, ethical practice and
efficient management processes in M&E is another critical challenge. Key issues
include how donors/aid agencies should oversee evaluations being outsourced
to consultants, how to coordinate evaluations of joint donor program effectively
and further on, how to feed the result of evaluations to the development com-
munities, particularly their CSOs/NGOs partners.
This means – that merely designing an M&E system is not enough: donors need to
commit to supporting M&E at all stages of implementation, which include in the plan-
ning to the implementation and post implementation phase: selecting appropriate
indicators, establishing baselines, collecting quality data and reporting and using
findings effectively. Badly designed and managed evaluations can do more harm than
good: e.g. misleading results can undermine the effective channeling of resources for
poverty reduction. Therefore, learning – feeding the results of evaluation and link-
ing knowledge resources that the donors/aid agencies have is also crucial.
10. 10
Box 1. IKAT US Program Approach in Linking Partners & Manage KnowledgE
IKAT US 1 partnership network was established in 2010 to promote partnerships
among Indonesian, U.S. and Southeast Asian CSOs to expand democracy experiences
and expertise in the region. The three-year program aims to increase the capability of
Indonesian CSOs to cascade their expertise and experiences inside, outside Indonesia
and across South East Asian (SEA) countries.
The medium term goal of the program is to improve democratic development, gov-
ernance, and respect for human rights in the Southeast Asia region, while the long-
term goal of the program is to sustain South-South partnerships between Indonesian
CSOs and their counterparts throughout the region to advance human rights
nd democracy.
Currently, IKAT US Component 1 partnership consists of 50 organizations from 7 SEA
countries (Annex 1). Mainly all six partnerships identify themselves as ABA-ROLI, RWI,
PPMN, POWER, R4D and AGENDA. All lead partners are based in Indonesia and formed
a local partnership with local organizations but at the same time, through IKAT US
program, each partnership expands their partnership to Timor Leste, Malaysia, Cam-
bodia, The Philippines, Laos, Burma and Thailand
USAID then initiated the IKAT-US Component 2, aiming to provide support to the
members of the IKAT-US Component 1 partnerships, in order to enhance their capac-
ity to implement regional programs. Component 2 links the partnerships and draws
lessons learned from the partnership program implementation to facilitate learning &
manage knowledge among partners. It hosts annual conferences that bring together
the Indonesian, US and Southeast Asian partner CSOs from all partnerships to share
experiences and lessons learned as well as coordinate their program activities and
assist the US Government with evaluating the IKAT-US initiative as a whole and seek
ways to strengthen these regional partnerships.
In the next page, we will take a further look at 7 Insights for Donors adapted from
intensive research and analysis by AWID (Association for Women’s Rights in Develop-
ment). Between 2009 and mid-2010, AWID has been engaged in an intensive research
into the challenges of monitoring & evaluating the progress of women’s rights work.
The consultant team adapted and synthesized AWID principles according to the
lesson learnt from other resources and field experiences.
11. 11
1. Make M&E a learning partnership, not a performance test
2. Approaches that include multiple M&E frameworks/tools/methods are more
effective
3. Balanced quantitative & qualitative assessment techniques
4. Seek contribution to impactful changes instead of achievement in numbers It is
more important to seek contribution to impactful changes rather than claiming
big achievements by using statistical numbers.
5. Make M&E systems flexible and adaptable
6. Invest resources in developing M&E capacity
7. Invest in the creation of baselines
Strengthening M&E: Seven Insights for Donors1
1: Batlilawa, Srilatha, “Strengthening Monitoring & Evaluation for Women’s Rights: Twelve Insights for Donors” for
12. 1. Make M&E a learning partnership, not a performance test.
3. Balance of quantitative and qualitative assessment techniques.
2. Approaches that include multiple M&E frameworks/tools methods are
more effective.
12
The first and most important overarching lesson from AWID’s research and conversa-
tions with both donors and women’s organizations, and other recent work, is that
assessment is best achieved when it is approached as a learning avenue for both
donors and grantees. We need to change the paradigm from“proving that you did
what you were supposed to”to a more collaborative paradigm“let’s learn together on
how to alter the deeply embedded causes of these issues/problems”.
Of course, donors themselves are under pressure to show results and thus, are forced
to adopt stringent, rigid or overly quantitative M&E. But if we do not change the
current practice and adopt a more collaborative approach to assess the results and
impacts, engage grantee partners in M&E design and/or subsidizing the capacity
building of the grantee partners, then the work is less likely to be sustainable due to
unrealistic expectations and lesser impact since the program has been“projectized”.
In the spirit of learning, we also have to not only focus on our performance or the
positive impact of our intervention, but also to the negative impacts. Current assess-
ment tools do not capture many negative. Designing instruments that enable to pick
up these negative effects are vital, since they may radically alter the assessment of a
project’s“success”or“failure”, by placing achievements against a more realistic yard-
stick.
No single M&E framework can capture all aspects of the change, impact or results of a
women’s rights/ empowerment of project or strategy – in short, one size does not fit
all! Expected changes in democracy & governance as well as policies are influenced
by complex power relations, embedded in multiple social, cultural, economic and
political structures and institutions. Thus, a single M&E instrument – such as logical
framework, theory of change, outcome mapping – can assess the some dimensions
of the change process and indicate impacts, but not all. So while one instrument
cannot tell us the whole story, strategic and intelligent combinations can bring us
much closer to a more comprehensive understanding of the change process, its gains
and limitations.
There is a widespread belief that donors prefer quantitative – or“hard”– evidence of
results, rather than“soft”data that tends to appear“anecdotal”and hence not rigor-
ous. In reality, though, the most complete picture of positive change – or of reversals
– emerges when quantitative and qualitative tools of assessment are combined.
13. 13
Qualitative method could produce explanation on the contributing factors behind the
targeted numbers gained with quantitative method. An overemphasis on quantitative
data actually undermines our ability to understand how changes happen, and there-
fore, of how to make it happen more effectively.
Indicators that spells numbers such as 100 women received training on improvement
livelihoods, 200 sex workers increased knowledge on HIV/AIDS, and so on. This
approach could be more complete if there are tracking system which will produce
sample of stories from the beneficiaries on enabling or supporting factors that could
make the training results more sustain and how they use the knowledge in their daily
practice.
Example: IKAT US-Kemitraan Partnership Framework Vs M&E Framework
IKAT US-Kemitraan program basically use two frameworks to see the progress on
progress of each partnership with the assumption that “better quality of partnership
will lead to better results”. M&E Framework to measure achievements and progress for
each partnership and Partnership Framework to assess depth and quality of the part-
nership. The Partnership Framework contains three main phases of the partnership
development from pre-partnership, development and Partnership phase. All indica-
tors used are more on qualitative aspects and use survey method using sample of all
the partners. With this combined approach, deep and sharp lesson learned reports
could be produced and next round of programming could get more feedbacks on the
achievements as well as the pitfalls in designing the next intervention.
(Source: www.kemitraan/or/id/krc)
In order to balance quantitative and qualitative evidence, it is important to give due
weight to participatory tools and methods – such as beneficiaries’narratives of change
– rather than privileging so-called “objective” evidence alone. This is because no one
can as accurately assess change in their situation as the beneficiaries themselves as the
target of our intervention.
Far from being merely anecdotal, these narratives and other participatory tools are
often the most sensitive indicators of project impact. While it is true that these meth-
ods can be manipulated to present a rosy picture of achievement, the solution is not to
dismiss them. In fact, even“hard”data can be manipulated to hide less flattering truths
by highlighting some data and burying others – as in the case of South Asian NGO who
highlighted survey findings that their micro-credit program had increased women’s
income by over 30% - but hid the fact that school drop-out rate among these women’s
daughters was equally high!
4. Seek contribution to impactful changes instead of achievement in numbers
14. 14
It is vital for both donors and organizations to recognize that macro-level changes –
such as more effective public spending and service delivery – occur over longer peri-
ods of time, when a number of stakeholders buys in to the change, and when a variety
of cultural, political and economic force coalesce. No single project can work on all
these fronts, much less in a three or five-year time frame. What we can measure, how-
ever are interim changes within specific stakeholder groups. It is therefore important
for donor and grantee representatives to discuss, at the outset of a project, what can
be most realistically measured and reported within the time frame of the project. This
preempts grantees from making exaggerated claims and donors from feeling disap-
pointed with the results.
Rigid approaches that insist on the initially set out targets / indicators / outputs /
outcomes are not always useful for multi-dimensional interventions - they reflect the
limited understanding of those imposing them, rather than the contextual realities in
which such interventions are actually situated. But this is not a peculiarity of social
change work – even major corporations and businesses have been forced to revise
their targets and indicators when macro- economic and market realities change!
A simple example is that of quantitative targets that might have been set at the begin-
ning of the project, but the process of implementation shows that these were over-
estimated – or under-estimated. The M&E indicators may quickly have to be revised,
under these circumstances, to reflect the level of response a project was able to do. A
timely re-negotiation also ensures that indicators are not changed arbitrarily because
the earlier ones reflected negatively on the organization’s work, but because external
factors have forced changes in intervention’s trajectory.
The attribution-based approaches like the logical framework are perfectly acceptable
when tracking performance or implementation of project activities, but do not work
well when we are trying to understand how the change process produces intended –
and unintended - results. They are also not designed to capture the interim goals that
must be achieved in order to reach final intended outcomes.
Logical Framework results tend to jump right to measuring the overall goal of a
program which are achievable only in the longer term, and consequently, in measuring
macro-indicators (like poverty rates, literacy rates, mortality rates, or other indicators)
that go beyond the scope of a single program’s sphere of influence or impact.
So in developing the right“mix”of M&E instruments, it is advisable to ensure the use of
approaches which strive to track the organization’s contribution to these shifts, rather
than those that seek to claim the entire credit for the change, which is hard, if not
impossible, to prove.
5. Make M&E system flexible, adaptable and suitable to the nature of the issues
15. 15
The point is that M&E systems must be agile and flexible, since evidence shows that
even the most carefully-chosen approaches and measures may have to be changed
during midstream if the ground-realities shift radically in the course of the project
implementation.
The world of development’s interventions, including in democratic and governance
projects, is populated by organizations with diverse and complex architectures –
networks, local-to-national-to-regional-to-global structures, coalitions and other
factors.
These entities often gain donor support because of their architecture, which gives
them a greater reach, bandwidth, and impact at multiple levels of policy and activism.
Yet, when it comes to monitoring their work, assessing their results, or evaluating their
impact, they are compelled to use M&E frameworks and tools that were designed for
far simpler, grassroots-based, direct-action or service-delivery organizations. These
tools ask questions like“how many beneficiaries have you reached / benefitted?”which
are almost impossible for multi-layered structures to answer.
Anti Corruption, democratic, human rights, and more contextual based issues are
more complex. For an instance, these type programs generally involve the secretariat
of a large, geographically dispersed network or coalition, with multiple organizations
as members, is typically the recipient or pass-through agency for funds. They cannot
answer this question without rolling it through all the layers of its structure. And even
when they provide the answer to the question, it tells us nothing about the value that
has been added by supporting this type of structure, rather than a simpler one. But
formations with multi-layered architectures are in fact aiming not only to scale up, but
to deepen the impact of the work of their individual members by fostering their knowl-
edge, capacity and strategies, and through value-adding approaches that a single
organization may not have the ability, resources, or reach, to undertake.
Complex governance or democratic based programs require M&E approaches that
combine the assessment of (i) their effectiveness as networks (such as the Wilson-
Grau/Nunez) or partnership, (ii) their advocacy impact (through tools and recommen-
ndations such as offered by Patton and Klugman), and (iii) a judicious combination
of the more conventional frameworks for assessing local- or member-level results.
If M&E is intended to increase operational effectiveness and contribute to learning, it
cannot be treated as an afterthought. Yet sometimes, complex M&E requirements are
attached to projects after they are approved, assuming this is something that every
organization should be able to deliver.This fails to take into account the fact that many
frameworks and methods require a certain level of skill or training – not to mention
staff time - for their effective use.
6. Invest resources in developing M&E capacity.
16. 16
As mentioned, above, this is not always the case – but more problematically, grantees
find that their M&E needs are not seen as part of the project cost, or built into their
budgets. Donor expectations on how lean an organization should look and results or
changes to be achieved within a short period of time is somewhat unrealistic. These
organizations are on the front lines and doing the heavy lifting while they rarely have
enough money for program, much less operations. So, success, if not survival,
depends highly on fundraising efforts. Thus, donors and aid agencies have to
acknowledge that to achieve their expected results, they have to invest in building
capacity of their grantee organizations to overcome these technical capacity issue.
The time has come to integrate M&E into project design much more consciously, and
to allocate serious resources for this purpose – this investment, after all, yields rich
dividends in terms of both results and learning that can be leveraged for future work
by both grantee and donor.
Many organizations find that the M&E frameworks and approaches required by some
donors demand resources that are beyond their capacity, such as the abilities of their
staff, the time-intensiveness of their implementation, or the need to bring in external
expertise that they cannot always access or afford due to resource constraints, loca-
tion, or other factors. Often, the complexity and amount of data required is excessive,
and does not necessarily provide a more comprehensive picture of implementation
or impact – for effective M&E, sometimes less is more, if the approach includes fewer
but more sensitive and intelligent indicators. In addition, in organizations with multi-
layered structures, capacity to manage sophisticated M&E tools and data require-
ements will vary across the network. All this implies a need for grant-seekers to assess
their organizational M&E capacity in transparent and non-threatening ways, without
feeling exposed or fearing loss of funding. Overall, balancing M&E needs with capac-
ities require both donor and grantee to interrogate different M&E frameworks and
tools through the capacity lens during the grant negotiation process itself, identify
fewer but more sensitive indicators of progress and change, and create approaches
that are feasible for the grantee organization to use, given their particular capacity
profile
It is in this context that generating periodic baseline becomes a powerful tool in
accurately assessing project achievements – and directions for the next phase of
work. When a clear situational analysis is generated at the outset of a project, organi-
zations can be more accurately place the changes that have occurred in the course of
their works – both positive and negative – against the baseline, identify what worked
and what did not, and refine their strategies accordingly. They can make much
stronger cases as well, for continued investment in their works. Donors can in turn
leverage this kind of evidence to raise their case to their governments or other
contributors, for why such work needs support. Together, donors and grantees can
advocate more convincingly to the world at large. For all these reasons, baselines are
highly important and worthy of investing time, money, and people resources.
7. Invest in the creation of baselines.
17. C. Partner Organization: Building the Learning Culture!
Organizations (grantees) confront the M&E challenges from a different vantage point.
They rarely have enough money for programs, much less operations, so priorities are
always given to somewhere else. In many ways, M&E has become about compliance
and a means – if not a checklist – to satisfy the donor requirement in the most mini-
mum level. Moreover, they are often undermanaged, in one part because unrestricted
funding for organizational expertise is hard to come by and in other parts because of
their own inclinations and abilities. Many are trapped in a kind of “starvation cycle,”
which begins with unrealistic donor expectations about how lean the organization
should look, and continue as nonprofits by cutting overhead to the bone to meet
those expectations—gutting the organization, but further encouraging those unreal-
istic expectations.
So from the grantee’s perspective, the donor-grantee trap can feel inevitable.
Grantees don’t have the means to“resource it right” on their own. They may want
to pursue genuine donor partnerships – but they are in a position where they have
less bargaining power. As for getting better together, that also takes investment -
not to mention discipline, rigor and a willingness to appear vulnerable by identify-
ing performance shortfalls and strategic gaps. It seems far safer to comply with
donors’existing reporting requirements and declare victory whenever possible.
One of the most difficult challenges is related to the resilience towards M&E. M&E is
often received as a measurement of our performance. Therefore, resistance is
common – it is not always easy to receive feedback. Engagement of M&E Unit and
Project Management Unit (PMU) should be built and strengthened since the begin-
ning. It is crucial, since the planning process that both M&E and PMU sit together to
discuss about the project designs and ways to measure it, ensuring that both units’
staff participated in the process. The focus of M&E should also be about solution and
improvement instead of solely on who is not doing what.
Overall, from the organization side, strong leadership’s support and commitment are
important to change the M&E practice internally. Leadership has to be able to encour-
age culture of learning and determine sufficient investment to enable the learning
culture to occur even under limitation of resources and time. Finding ways to optimize
the utilization of internal resources and external resources through collaboration
might be one way.
Here are three organizations, which don’t have the resources of the World Bank or
Gates Foundation, yet each one has demonstrated the willingness to question their
assumptions and try to meet the evaluation challenge in different ways :
17
2
2: Savedoff, William, “Impact Evaluations Everywhere: What a Small NGO to do?”, for Center for Global Development,
17 Sept 2012, accessed at: http://international.cgdev.org/blog/impact-evaluations-everywhere-what%E2%80%99s-small-ngo-do
18. 18
Box 2. Evaluation Approach in Oxfam GB
Oxfam GB is a relatively large NGO that has an evaluation approach that would be
recognizable to a large development bank or government agency, yet on a scale that
tries to balance precision with staff time and costs. Oxfam GB randomly picks about 40
projects a year and tries to infer impact using different methods. When projects
involve affecting the lives of many people, they compare project beneficiaries with a
comparison group. When the projects are related to influencing policy or empower-
ment, they use process-tracing. If projects are randomly chosen, it helps Oxfam GB
avoid cherry-picking the most promising projects and therefore allows them to say
something more general about the performance of their portfolio. Randomly choos-
ing projects also increases the probability of discovering things that staff might not
have expected. The value and use of the studies will depend, as always, on their qual-
ity – particularly the degree to which their conclusions are credible. Are the counter-
factuals appropriate? Is the causal chain clearly described and can alternative expla-
nations be discarded?
Source:Savedoff, William, “Impact Evaluations Everywhere: What a Small NGO to do?”
19. 19
Box 3. AWID’s Strategic Initiative
The Association for Women's Rights in Development (AWID) is an international, femi-
nist, membership organization committed to achieving gender equality, sustainable
development and women's human rights. A dynamic network of women and men
around the world, AWID members are researchers, academics, students, educators,
activists, business people, policy-makers, development practitioners, funders, and
more. AWID’s mission is to strengthen the voice, impact and influence of women’s
rights advocates, organizations and movements internationally to effectively advance
the rights of women.
AWID’s work is structured through multi-year programs known as Strategic Initiatives.
Each strategic initiative includes a range of activities from membership consultations
and surveys, primary research and dialogues with policy makers (including targeted
advocacy) to capacity building institutes, regional networking and information
dissemination. In addition, AWID works to ensure that the specific priorities and voices
of young women are strongly represented in all their initiatives.
Recognizing that movements without resources cannot be sustained, this initiative
undertakes research and advocacy in order to significantly increase the amount
and quality of funding to support women’s rights work. The initiative was primarily
developed to tackle the urgent need of women’s organizations and movements to
access more resources on better terms and to transform their relationship to funding
– moving beyond a culture of“scarcity”to see funding and resource mobilization as a
critical aspect of their political agendas and key for building strong feminist move-
ments. At the same time, the initiative also works to improve the ability of women’s
organizations to use these funds in ways that are strategic, bold and effective.
Source: www.awid.org
It is in this context that through 2009 and 2010, AWID initiated multiple action
research projects to study the challenges faced by both women’s organizations and
their donors in effectively monitoring and evaluating women’s rights work, and to
enhance our collective capacity to assess the influence and impact of such work. An
in-depth quantitative and qualitative study of the experiences and challenges faced
by 37 out of the 45 organizations that received the Dutch Government’s MDG3 Fund
grants in 2008. The study led to the following insights that are drawn from Learning
More from the MDG3 Fund Experience (Batliwala and Pittman) . It is important to take
a look at these insights since it can also be applied to the works that our organizations
do!
3
3: Batlilawa, Srilatha, “Strengthening Monitoring & Evaluation for Women’s Rights:
Thirteen Insights for Women’s Organizations” for AWID, 2011.
20. 20
Strengthening M&E: Seven Insights for Organizations4
4 : Slightly adapted from “Strengthening Monitoring & Evaluation for Women’s Rights: Thirteen Insights for Women’s Organizations”
by Srilatha Batlilawa for AWID, 2011
1. Make M&E key ingredient in our learning & accountability (1&4)
2. One size does not fit all (3, 5)
3. Prioritize approaches that assess our contribution to impactful change instead
of achievement in numbers (4 & 5)
4. Make M&E system flexible and adaptable (7,8,9)
5. Negotiate M&E systems & result indicators with donors (10 & 11)
6. Develop M&E capacity (2 & 13)
7. Create baselines
21. 21
AWID’s research and conversations with both donors and women’s organizations
reveal that M&E is most useful and relevant when it is approached as a learning
process, rather than a reporting or fundraising requirement. Solid, comprehensive
and rigorous assessment of our effectiveness is a critical expression of our account-
ability to our constituency, and to our longer-term mission. We often engage in this
learning process subconsciously or informally. In our internal meetings and planning
processes, or in conversations with each other, for instance, we are constantly assess-
ing the progress of our work. We frequently share valuable insights about what we are
learning, about what we have achieved, the setbacks we have suffered; we identify
who and what is behind both the challenges to our work and the progress we have
made; and we analyze why we think change has—or has not—happened. This is
exactly what monitoring and evaluation means. The task is thus to transform this inter-
nalized habit of analysis and learning into more systematic and articulated forms that
can be shared with others—not just the donors who may require the information, but
others who could learn from our experiences and insights. Even more, we have to
apply this learning more consciously in reviewing our practices and strategies, and
shaping new interventions.The challenge, therefore, is to make M&E a central part of
the way we learn and strengthen our work at every level, a vibrant expression of
organizational and individual learning and growth, and a critical contribution to
the collective learning worldwide. “M&E should become part of our knowledge
management system”.
To address this learning goal, we have to also acknowledge that M&E system should
not only track positive impact as an indication of our performance. M&E systems that
allow space for us to track and document negative changes, resistance and backlash
are essential. For example, measuring the result of affirmative action program for
women in parliament in terms of increased number of women in parliament might
only tell one part of the story. Whether having more women in parliament lead to
better development policies or gender-sensitive policies is another issue or even,
whether the affirmative actions basically further the view that women can only sit in
parliament due to the quota. Nonetheless, most assessment tools are not designed to
track or capture these negative impacts. Designing instruments that pick up these
negative effects and reactions are vital, since it can radically alter the assessment of a
project’s“success” or“failure”, by placing our achievements in a more realistic context.
In fact, in many cases, negative reactions or reversals are actually evidence of positive
impact. Similarly, we need approaches that give due value to processes that success-
fully hold on to past gains that “hold the line”, such as preventing the repeal of a law
entitling women’s access to abortion, or protection from domestic violence. Holding
the line, in this context, is a success story, not evidence of a failure to move forward.The
sign of positive impact might actually simply be that“things haven’t gotten worse”.
1. Make M&E a key ingredient of our learning and accountability.
22. 22
Therefore, the final and possibly most important principle emerging from our M&E
system is that if we make the time, effort, and resources available to design and
implement the best M&E system possible for tracking and assessing our work –
make it a part of knowledge management, chances are that it will also serve the
needs of other stakeholders to whom we may be accountable— our governments,
donors or the public that we seek to serve. It can actually be useful to other stake-
holders in helping them design a better program or intervention.
AWID’s research has shown clearly that no single M&E framework can capture all
aspects of change, impact, or results of women’s rights / empowerment project or
strategy. No single tool or method can respond to all of our learning needs, since each
has been designed to track or capture specific dimensions of change or operational
effectiveness but not others. In any M&E instrument—whether it is the logical frame-
work, theory of change, outcome mapping, or gender impact analysis—it only
assesses a particular set of dimensions, but not all. Consequently, a comprehensive
assessment requires the application of multi frameworks, methods, and tools, working
together in a complementary fashion. In fact, many women’s organizations are already
doing this: over half (51%) of the women’s organizations in AWID’s recent study of
MDG3 Fund grantees use more than four M&E approaches and tools, or elements from
several, to document their progress and impact. This is quite logical given that the
nature of gender and social power relations are complex and that organizations oper-
ate in different social, cultural, economic and political contexts. Our study also found
that “organizations that used more than four M&E approaches, experienced a slightly
higher level of satisfaction with their M&E system (and conversely, lower levels of
dissatisfaction) than those that use one or two methods.”(Batliwala and Pittman, np).
Just as we need to consider combining multiple approaches, we also need to combine
both quantitative and qualitative M&E tools and their results in an appropriate
balance. The experience of most women’s rights activists and organizations is that
quantitative, or “hard”, evidence of results is taken more seriously than “soft” data like
stories of change, which are treated as anecdotal and lacking in rigor. There is also a
tendency to believe that our work can only be assessed qualitatively, and that our
processes of change are too complex or subtle to be measured in numbers. While it is
true that no one can as accurately assess change as the communities beneficiaries who
are the subjects and agents of a change process, we should not see these as substi-
tutes for, but complements to, harder assessment methods. In fact, an overemphasis
on qualitative information often limits our ability to demonstrate that our work is
making a difference, especially with audiences like government policy makers or the
donor community.
2. One size does not fit all.
23. 23
For example, we could combine surveys which generate quantitative data on changes
in women’s political participation, mobility, income, awareness of rights, literacy,
health-seeking behavior, and changes in male attitudes, with qualitative methods like
narratives of individual and collective struggles, stories of change, and focus group
discussions that describe how change happened. In fact, quantitative data often
validates and provide nuances for qualitative evidence, and vice-versa. The
quantitative-qualitative balance can be achieved by organizations regardless of their
size, location, or context, especially when combined with the “less is more” approach
(see point 7). Building our stories of change by combining quantitative data and quali-
tative evidence can help us make our case far more effectively.
Another example is IKAT US-Kemitraan Partnership Framework Vs M&E Framework
refer to the previous section on“Insights for Donors). IKAT US-Kemitraan program basi-
cally use two frameworks to see the progress on progress of each partnership with the
assumption that“better quality of partnership will lead to better results”.
So, while an M&E instrument cannot tell us the whole story, strategic combinations
can bring us much closer to a more comprehensive understanding of the change
process, including its strengths and limitations.
We are often tempted to claim credit for all the changes that occur in a development
process, or feel pressured to do so by the struggle to secure funding for our work. And
sometimes we are reluctant or too modest to take credit for our contribution to
change, fearing it will be seen as exaggerated or self-promoting. Some M&E frame-
works - such as the logical framework or Results Based Management—are in fact
designed to attribute results to our interventions in a simplistic way. But in reality, such
approaches are more appropriate for tracking performance or implementation of
project activities. They do not work as well when we are trying to understand how the
change process produced results, both intended and unintended. What is more, such
“attribution-seeking” approaches are not designed to capture the interim steps that
must be achieved in order to reach the final intended outcomes. For example, a height-
ened awareness of domestic violence as a crime is a necessary first step to reducing
such violence. This results in jumping straight to measuring the overall goal of a
program or intervention —which we know can only be achieved in the longer-term
(e.g., reduction or elimination of domestic violence)—and in making exaggerated
claims of attribution that can rarely be supported. Worse, both having held the line, or
reversals and backlashes will also be placed at our door in a negative way. This is why
contribution-based approaches – such as Outcome Mapping, Most Significant Change
orTheoryofChange,-shouldbeimportantcomponentsofourM&Esystems,sincethey
allow us to make more realistic, but modest, claims about our role in the change
process.
3. Prioritize approaches that assess our contribution to impactful change
instead of achievement in numbers
24. 24
Rigid approaches to reporting on targets / indicators / outputs / outcomes that were
planned at the start of a change intervention are often not useful for our line of work.
When we plan an intervention and design the M&E system to monitor and assess its
results, we are engaged in intelligent guesswork rather than infallible certainties. The
broader context can affect planned interventions, which requires the
organization/activists to adapt to the new circumstances. There is always an element
of unpredictability in women’s rights work, so that the best laid plans can go wrong,
often for reasons beyond our control. On the other hand, things can also go right, or
proceed much faster, than we had anticipated.This lack of predictability is not a peculi-
arity of social change work - even major corporations and businesses have been forced
to revise their targets and indicators when macro-economic and market realities
change! Under these circumstances, M&E indicators may quickly have to be revised, to
reflect what the organization was able to do in response. As such, M&Esystemsmustbe
agile and flexible, since evidence shows that even the most carefully-chosen
approaches and measures may have to be changed midstream if the ground-realities
shift radically in the course of project implementation (Batliwala and Pittman, np). Of
course this flexibility should not be misused —to hide, for instance, our own mistakes
or strategic errors. It should be applied only when it is clear that the trajectory of our
change intervention has been altered by external factors beyond our control or as a
result of new information that suggests a change in course.
The world of development intervention, including in democratic and governance
project, is populated by organizations with diverse and complex architectures –
networks, local-to-national-to-regional-to-global structures, coalitions, etc. But most
M&E frameworks and tools at our disposal were designed for far simpler, grassroots-
based, direct-action or service-delivery organizations. These tools ask questions like
“Howmanybeneficiarieshaveyoureached?Howmanybeneficiarieshavebenefitted?”
which are often quite inappropriate for complex, multi-layered, multi-locational stru-
ctures. Networks, coalitions, and transnational organizations have created complex
structures that enable them to work in multiple countries, at multiple levels (local,
national and global) and with multiple organizational members or partners. Their
complex architecture also often enables them to deepen the impact of the work of
their individual members by bringing in expertise from other locations or levels of the
structure. This results in strengthening knowledge, capacity and strategies of all their
members / units, and in stronger collective advocacy, multi-centric research studies,
and other activities that a single organization may not have the ability, resources or
reach to undertake. These organizational structures, therefore, require more complex
M&E systems. They may need to combine, for instance, the assessment of their effe-
ctiveness as networks (such as the Wilson-Grau/Nunez framework), their advocacy
impact (through tools and recommendations offered by Patton and Klugman), as well
as a judicious combination of the more conventional frameworks for assessing unit- or
member-level results.Thisisyet another exampleof whymultipleassessmenttoolsare
essential to build effective M&E systems
4. Make M&E system flexible and adaptable
25. 25
Many women’s organizations in the AWID’s study find that the M&E frameworks and
approaches required by some donors demand resources that are beyond their capac-
ity, such as the abilities of their staff, the time required for their implementation, or the
need for external expertise to which they may not have access. They may also be diffi-
cult to use for groups working in politically unstable or risky contexts of violence and
conflict. Sometimes, the complexity and amount of data required is excessive, and
does not necessarily give a better picture of implementation or impact. This usually
happens because M&E requirements are not prioritized in the grant-negotiation
process, but treated as an afterthought or add-on by both donor and grantee, result-
ing in a nasty shock when the reality of what is involved becomes clear.
Furthermore, we have to remember that macro-level changes, such as lower violence
rates, can only be reliably assessed in the long term, and only after accounting for the
role of multiple actors involved in the change process, as well as the influence of a
variety of cultural, political, and economic forces. A single project or organization
cannot work on all these factors, much less in a three- or five-year timeframe. What we
can measure, instead, are the interim changes within specific stakeholder groups and
our contribution to enabling that change. Therefore, indicators of results should also
be realistic and it is important for both the organization and donors to have the same
understanding on this.
But this can be pre-empted if we seize the initiative in designing a sound M&E frame-
work and indicators for our proposed work—based on quantitative-qualitative
balance,“less is more”, and other principles offered here—but which are tailored to
our contextual realities and organizational capacity, and which demonstrate a
serious approach to tracking and evaluating our work. This also ensures that the
basis for negotiating the assessment of our work is created by us, rather than others
less experienced or expert in the strategies and contexts of our work. Understanding
and negotiating M&E expectations at the outset of a project or funding cycle is a useful
strategy to avoid tensions and misunderstandings at a later stage.
The research shows that many of AWID’s organization members reported that it has
become harder than ever to mobilize resources for their work, particularly when their
strategies are seen hard to monitor or their results difficult to measure. Similarly, this is
also experienced by many other organizations. So there is a need to generate more
rigorous and convincing data about the effectiveness and impact of our strategies.
However, the research indicates that assessing our work effectively often demands the
use of tools and methods that require skills and capacities that may not be available
within our organizations, or which it may feel strange to our cultures and traditions of
learning. So, when setting up an M&E system either trying to redefine or re-design
an existing one, it is important to assess the types of staff capacities, time and other
resources it requires, and to develop a mix of tools and methods that are feasible
within these constraints.
5. Negotiate M&E systems & indicators of results with donors
6. Develop & Invest in M&E Capacity and Programs
26. 26
In fact we are often launching projects and change strategies without a clear vision of
the change we seek, the values and politics that inform the vision, and without a
strong diagnosis of the problems we are attempting to address. Baselines can greatly
advance this clarity by providing the basis for a clear situational analysis at the outset.
This in turn enables us to more accurately place the changes that have occurred in the
course of our work—both positive and negative—against this baseline, identify what
worked and what did not, and refine our strategies accordingly. For women’s rights
work, for example, there are many examples of baseline studies conducted by NGOs
and women’s organizations assessing the state of human rights, gender relations,
violence against women, etc., in many parts of world. The evidence generated through
these has helped to monitor and assess the impacts of their interventions in more
convincing and accurate ways by generating concrete evidence of where things were
at the outset of the project. This has enabled them to better learn about the impact of
their change interventions over time, but also to make a much stronger case to exter-
nal audiences about what worked—and what didn’t. It is not that only large, well-
funded organizations can conduct baseline studies—they can be done even with
limited resources through interesting combinations of participatory methods, secon-
dary data (gleaned, for example, throughofficial statistics or census data, or surveys and
studies done by other organizations), as well as conventional “objective” methods like
surveys. Our ability to create baselines, and place our progress along selected indic-
ators within these, also enhances the ability of our supporters (including our donors) to
make a stronger case to the larger world for why such work needs support
7. Create baselines. Generating periodic baseline data is a powerful
but underutilized tool in accurately assessing our role and achievements
in the change process—and in locating the most strategic directions for
the next phase of our work
27. D. M&E for Partnership Program
27
5
5: http://partnershipsresourcecentre.org/knowledge-platform/learning-modules/module-2#entry
Who Evaluates
what?
How to Elauate?
Who Evaluates
Whom?
Partnership raises critical questions about the extent to which collaboration actually
adds value in terms of both process and outcomes and how judgments are made
(Atkinson, 2005). Evaluating partnerships is difficult for various reasons such as long
timescales for achieving impact, different perspectives on what success means, the
complexity and variability of partnership interventions and the different context
within which partnerships work. Neither efficiency of the partnership process itself,
nor its effectiveness in addressing set goals is easy.
The key point in building and implementing M&E for Partnership is in investing to
build a strong partnership. Thus, open and honest communication matters, between
donors/aid agencies and the grantees, and among partner organizations, themselves.
Articulating and agreeing on shared goals, and how those goals will be achieved, is
another essential element. With adequate strategic clarity, both donor and grantee
can thoughtfully address the realities of what resources it takes to achieve the
goals—including overhead and a reasonable timeframe for results to be achieved.
Strategy should drive cost, not the other way around.But, it takes trust and mutual
commitment to identify what is and isn’t working, to alter strategies, and to get better
together.
So, how do we make partnership works? What should we do to make M&E for partner-
ship works?
28. #1 Select potential partner carefully
28
6: Quoted from interview with BIGS (one of the partner of Building Bridges Partnership) on December 18, 2002
and PGRI (Lead Organization of POWER) on December 20, 2012 – both are a part of IKAT US Program
First it is essential for both donors and grantees to pick potential partners carefully.
Foundations refer to this process as“due diligence”: evaluating a potential grantee in
much the same way, as an investor would analyze a venture capital opportunity. For
donors, investing time up front in carefully selecting grantees will reduce the risk of
subsequent problems. And for grantees, it is important to understand your donor’s
characteristic and approach and be prepared for engagement with the donor. And for
grantees, it’s important to understand that sometimes it is better to walk away from a
donor than to get trapped in an ineffective and burdensome relationship.
In partnership program, the lead organization also has to select its partners carefully.
Both donors/aid agencies and organization have to invest time to assess and under-
stand:
Social, political and cultural context in the area which the potential partner
organization work
Strengths & weaknesses of the potential partner organization
BIGS’Strategy to initiate a partnership at regional level in Building Bridges – IKAT US
Partnership Program:
Ideally, the process should be started with Country Scan Study (Initial Mapping) first to
understand the political dynamic and CSOs map in each country. However, BB did not do
that and R4D used strictly CFP (Call for Proposal) approach that they had done previously
in other programs as well. CFP was announced in public CSOs network in regional and
national level. The plus point from CFP process: more neutral.
BIGS considered that CFP process is not enough; it is better to conduct Country Scan study
first. BIGS had previously become the subject of Country Scan Study for IBP (International
Budget Partnership); a process using qualitative instruments (FGDs). This process allows
more understanding of the country context and can prevent issues such as what BB expe-
rienced in Cambodia (In Cambodia, the CSO has to obtain permission letter from related
Ministry to conduct research). Aside from that, the Country Scan will also inform a sense of
time frame in relation to dates of significance in the relevant country that can influence
activity implementation, based on BIGS experience However, the downside is that Country
Scan Study requires more time and more fund.
Box 4. Experiences of IKAT US’s Lead Organizations in Initiating Partnership
6
29. 29
7: Many adjustments of system, legal framework, financial management were required. For example, the partner in Malaysia
is not used to manage budget with dual-currency.
Kemitraan’s Strategy to initiate a partnership at regional level in POWER – IKAT US
Partnership Program:
The names of organizations included in the POWER partnership originated from inform
ation of individual who has worked in the country instead of resulted from Mapping of
Women Organization or Needs Assessment. After the program has been approved and
initiated, during the implementation, we realized that there are other organizations that
perhaps are better to be included in the partnership (more experience, acknowledgement
in their respective country).
From the proposal submitted, actually Kemitraan could firstly conduct capacity assess-
ment of potential partners in making a proper proposal and in capacity building. How-
ever, because it will take longer time to initiate the partnership, especially each organiza-
tion has different capacity in program and financial management , the Partnership did
not pursue deeper due diligence.
Kemitraan admitted that there is no budget to be allocated for a new partner and they
cannot increase the budget platform. Finally, they keep involving the organization by
inviting them to the workshop and include them in the partnership‘s network.
7
30. #2 Launch the partnership earlier
30
8: “Successful Partnership: A Guide” by OECD LEED Forum on Partnership and Local Governance
M&E capacity building and implementation are likely to be most effective if incorpo-
rated from the inception of program design. This way both partners can develop the
program and M&E components, and the partner providing M&E assistance has an
opportunity to provide input into both program and M&E design.
The partnerships discussed here were not formed until after the programs were
designed. M&E capacity building is still possible at this stage; ideally, however, the
partnerships would have begun earlier so that the M&E goals could have informed
program design.
The following elements need to be taken into account :
a. The partners must be involved in the entire exercise, including the M&E frame
work development
•Partners must be able, among themselves, to set the results to be reached and the
impacts and outcome to be sought
•Partners must agree on the key indicators that will best demonstrate the relevance
and efficiency of their actions and the long-term impact of the partnership
•Partners must participate in developing the monitoring & evaluation strategy that
includes various methods (e.g. collection of statistical data, satisfaction surveys, etc.)
•Partners must identify monitoring procedures and be involved in developing data
collection tools
b. M&E process must be implemented at the beginning of the partnership
• Establish initial baseline as quickly as possible and compile key data regularly and
on-going basis
• Budget must be provided to accommodate proper M&E from the start
• Establish a monitoring mechanism friendly to the user;it should be constructed in
such a way as to facilitate the work of the partners appointed to follow up the activi-
ties and procedures.
8
31. #3 Learn about our partners
31
It is important early in the partnership to understand some key characteristics about
the program partners and the context in which they work. Be open to their ideas and
constraints. This process paves the way to building partners' M&E capacity, and
convincing the partners that M&E will improve their programs without unduly reduc-
ing their already limited resources.
Furthermore, we need to recognize that in order to be able to achieve the shared goal,
each partner will need to be supported to overcome their organization’s weaknesses.
This is also the key for sustainability that is often demanded in each project/program.
In M&E for example, specialist must be engaged and responsible for some of the M&E
stages, particularly in evaluation:
• Specialist must lend their expertise to the partners and ensure that there are
shared understanding on evaluation methods
• They must guide the partners through the process and verify and validate their
choice of key indicators or follow-up methods
• Partners have to come into an agreement to compile necessary data & informa
tion
This approach helps partners familiarize themselves with the evaluation methods and
develop a certain expertise in the field, guaranteeing both the credibility of the
process and the results obtained and validated by the external specialists.
The partnerships may choose among various methods of evaluation that best suits
their case, both in terms of membership and in terms of scope and objectives. The use
of an external evaluator combined with the partnership’s involvement in the proce-
dure shall ensure demonstrable better results.
Many projects are now introducing a start-up phase where a more intensive process
of consultation leads to adjustments to project design after funding is secured. Some-
times, however, partners feel they are presented with a fait accompli; they are invited
to make suggestions, but only within the confines of the project budget, staffing
structures and strategies that have already been defined.
32. 32
Box 5. POWER: Showing Leadership & High Quality Training based on Needs
POWER is a partnership of seven civil organizations from 5 (five) different countries.
This partnership is led by Kemitraan as the lead partner to manage six regional part-
ners in initiating changes in each of their respective country. With generous support
from USAID, POWER share, learn and improve their capacity together. Six regional
partners are National Democratic Institute (NDI), Koalisi Perempuan Indonesia (KPI-
Indonesia), Persatuan Kesedaran Komuniti Selangor (Empower-Malaysia), Caucus
Feto iha Politica (Timor Leste), The Cambodian Center for Human Rights (CCHR-
Cambodia) and the Center for Popular Empowerment (CPE-Philippines).
POWER applied series of activities to improve partner’s capacities in advocacy.
Through such approach, the mutuality has been defined among partners due to
concrete exchange of assistance among themselves. Within two years of partnership,
multi-mentoring mechanism has been effectively performed. in terms of program-
matic as well as organizational capacity building. These are some examples:
Kemitraan-NDI-CCHR had carried out training on casework/constituency outreach for
elected women as well as specific training on facilitator skills for CCHR staff;
Kemitraan-NDI-Empower-Caucus in joint-training for women candidates; Kemitraan-
CPE-Caucus in joint-training for media person on gender sensitive training; and
Kemitraan-KPI-Caucus in joint-training for women candidates on leadership training.
Another enabling condition created by POWER to increase the sense of belonging
towards their new network identity is by trying to obtain engagement and commit-
ment from all partner members by accommodating partner's inputs.
Through various activities such as above, the likelihood to impact at national level is
expanding as each partner has enhanced capacity to run stronger advocacy program
in each country. The effort to expand awareness was done through many approaches
among others shown by the success of Empower to launch their online database for
women in parliament and CCHR to launch online interactive map on women candi-
dates for and women elected in the Commune/Sangkat election. In Cambodia, the
database became the first information media access for public database of women
candidate and elected women, while Empower became the first group ever to present
the result of women in parliament via interactive map in Malaysia.
Source: Lessons Learned Report IV – IKAT US Partnership
34. A. Essentials of Monitoring, Evaluation & Learning
As we have concluded in Chapter 1, the key of a meaningful M&E is learning. Then the
next question is Which M&E approach to be chosen to facilitate the learning?
In the following section, we will explore several M&E tools & approaches. But before
we move further, there are several things that we have to remember:
1) It takes a learning culture within an organization or program to take up the chal-
lenge of customizing and implementing different PME approaches that are
relevant for a specific context. In several cases we observed that a crucial element of
a learning culture is the presence of a group of people who have the motivation, the
courage, and the mandate to address PME challenges in their organizations or
program by introducing PME approaches that are new to their organization.The cases
further show how support from higher management and trustful relationships can
nurture such learning culture.
34
2) Each M&E tool & approach has its own strengths and weaknesses. Organizations
and its partners might have to use a mix of M&E approaches to complement each
other and address the complex context of their project or program as well as the
learning needs. In a partnership program, we need to also consider which approach
that can address the situation and needs of the partners. Agreement among part-
ners on which approach or mix of approaches to be used is essential in pursuing M&E
in a partnership program.
3) Sharing:Learning through collaboration. Collaboration is a fashionable buzzword
– and for a good reason. Undeniably, we simply have to work with a wide range of
people and organizations to be able to contribute towards real and sustainable
results. A single organization hardly has enough resources and skill to achieve
intended impact. But collaborating is difficult, it is hard enough for people inside an
organization to collaborate, let alone get two or more organizations to work together.
This is where learning together has a valuable role to play. There is now enough expe-
rience to show that if organizations or communities learn together, sharing their prac-
tices and experiences, their reflections and learning, then this starts to lay foundation
for collaboration. It helps people learn about each other and understand each other
before they decide to work together.
35. B. M&E: Some Tools, Methods & Approaches
35
Please do remember that this list is not an exhaustive list of M&E approaches. There are
other M&E approaches developed out there and it will be good to assess them one by one
to see whether it fits with our organizational or partnership architecture. In this chapter
we will try to look at several of the approaches, its strengths and weaknesses and how it
can be applied to our work. The logical framework approach will no longer be discussed
since many organizations are already familiar with the approach.
36. Most Significant Change
36
What:
Most Significant Change (MSC) is a method of participatory evaluation that involves
the collection of significant change stories at different levels of the intervention (for
example project staff, change agents, intervention participants) and collectively
deciding on the most significant change stories based on selected themes (called
domains). The domains reflect broad categories, such as change in capacity to take
action, or a change in participation in an activity, as well as lessons learned.
Main steps to design an MSC process :
• A good means of identifying unexpected
changes, both positive or negative
• Enables monitoring of initiatives that do not
have pre-defined outcomes against which to
evaluate
• It is a participatory form of monitoring that
requires no special professional skills
• It encourages analysis as well as data collec
tion because people have to explain why they
believe one change is more important than
another
• It can build staff capacity in analyzing data and
conceptualizing impact
This is where you should introduce the MSC process to all the stakeholders and clarify its
purpose. It is important to also consider who are the best people involved in the project to
collect stories from.
Domains are broad and often fuzzy categories of possible changes. For example, participants
in the MSC could be asked to look for significant changes in four domains:
- Changes in the quality of people’s lives
- Changes in the nature of people’s participation in sustainability activities
- Any other changes…
1. Starting
2. Establishing
'domains of
change'
• MSC is not meant to be used as a stand-alone
methodology
• Generally not as useful in situations when the
implementation processes and outcomes are
straightforward and the causal paths connect
ing inputs and outputs are clear
• Outcomes that are recorded through signifi
cant change stories may be positively biased.
MSC also raises the important issue of voice
and power associated with who participates in
the story selection process
Strengths Limitations
9:Referring to “Storytelling” http://evaluationtoolbox.net.au/index.php?option=com_content&view=article&id=61&Itemid=157
9
37. 37
The central part of MSC is an open question to participants, such as:
‘Looking back over the last month, what do you think was the most significant change in the
participants practice of sustainability in their home life?’
The question has six parts:
‘Looking back over the last month…’– It refers to a specific time period.
‘…what do you think was...’– It asks respondents to exercise their own judgment.
‘…the most significant…’ – It asks respondents to be selective, not to try to comment on
everything, but to focus in and report on one thing.
‘…change…’– It asks respondents to be more selective, to report a change rather than static
aspects of the situation or something that was present in the previous reporting period.
‘…in the participants practice of sustainability …’ – It asks respondents to be even more
selective, not to report just any change but a change in the participants understanding of
sustainability. This tag describes a domain of change and can be modified to fit other
domains of change (for example, participants understanding of sustainability in the commu-
nity).
‘…in their home life?’– Like the first part of the sentence, this establishes some boundaries. In
this particular case we are not asking about people’s actions in the community or what others
are doing in Australia, but focusing about people’s home life.
Selecting stories usually involves a hierarchical process, where the lower levels select
significant stories for the upper levels to review, and select the most significant ones from
the lower levels
Recommended Use of MSC:
Community development & empowerment as well as behavior change intervention
Exploring the impact of an intervention both intended and unintended impacts
Distill outstanding experience or initiatives and lessons learn
Remember that MSC is not meant to be used as a stand-alone methodology. MSC also is not suitable for
interventions that require quantitative measurement.
MSC will also be a very useful tool in assessing the impact of collaboration in a partnership program: what are
the impacts of partnership programto the organizations within the partnership? As elaborated in the previous
chapter that the current world of development intervention is populated with organizations with diverse and
complex architectures, including in forms of collaboration or partnership. And therefore, the M&E system also
needs to cater the question of partnership’s effectiveness aside from their overall impact. However, most do not
have a specific pre-determined partnership framework. MSC can be used to draw the experiences and learning of
each member of partnership in order to find ways to strengthen it.
4. Reviewing stories
and selecting most
significant change
story
It is really important to feed back your selected SC stories and the reasons for the choice to
the relevant people
5. Feedback
38. 38
Box 6. Illustrative Use of MSC for ASIA CALLING
ASIA CALLING: Telling the Untold Stories
Telling the unheard stories is the core mission of the Asia Calling Network (ACN). After
two years expansion, ACN has become one of the most extensive independent media
networks in Asia working to promote democratic values across the region. With a 3
year program of support from USAID Indonesia, since April 2011, ACN has expanded
its outreach by diversifying channels and media, including radio broadcasting, televi-
sion, online and print media. Under the IKAT-US partnership program, the Indonesian
Association for Media Development or Perhimpunan Pengembangan Media Nusan-
tara (PPMN), as the leading partner, works together with KBR68H and Tempo TV in
Indonesia, Media Development Investment Fund, Malaysiakini, 100 FM Chiang Mai
University (CMU)–Thailand, Democratic Voice of Burma (DVB) and Cambodian Centre
for Independent Media (CCIM) Cambodia. PPMN distributes its program in Indonesia,
Thailand, Cambodia, Burma, Singapore, Malaysia, Philippines and Timor Leste.
Apart from the geographical outreach that has expanded beyond the program’s
targets, the program content of ACN has had made significant impacts on beneficiar-
ies as reported in the midterm evaluation result. Views from Asia Calling correspond-
ents who have been trained by ACN felt that training provided by Asia Calling actually
improved their journalistic skills. The evaluator stated that IKAT-US funding for train-
ing at Asia Calling is being leveraged in multiple ways. There is no better journalism
training anywhere in the world than that offered by Asia Calling and KBR68H.
In terms of programmatic content, the most interesting outcomes beyond the Asia
Calling program have been the concrete impacts of some stories. A significant
number of Asia Calling’s features have led either to additional media coverage or to
political action and increased community awareness in many countries.
Clarence Chua of Malaysia, a country where mainstream media outlets are controlled
by parties close to the ruling coalition -- reports that his stories have been picked up
by other independent and even international media. Meanwhile Khortieth Him of
Cambodia reported significant impact in response to a story on education for Muslim
girls:
A story that had“real National”impact was the story on Young Muslim girls who have
less chance to enter school. After having this story aired, the source who was a Parlia-
ment member and was interviewed by Asia Calling took action to reduce the obsta-
cles to young Muslim girls going to school. It was because the teacher did not allow
them to wear Hijab in class and finally the government officially allowed Muslim girls
wearing Hijab in class and as a result many Muslim girls now have come back to
school.
Stories of changes starting from how journalism training brings changes to the media
network, how media networking brings changes among the members of the network
and how the “untold stories” broadcasted by the media network affect the lives of
people just like the previous story.
Source: Lessons Learned Report IV, IKAT US Partnership
39. Outcome Mapping
39
What:
Outcome Mapping (OM) focuses on measuring changes in the behavior of the people
with whom a development initiative works most closely. The goal is to identify which
stakeholders contribute to shifting behaviors, actions, activities, beliefs, policies,
etc.—focusing on contribution, rather than attribution, to change. Outcome mapping
limits its concern to those results – or “outcomes” – that fall strictly within the
program’s sphere of influence. It considers only those activities where the program
can claim its contribution to a direct effect.
Three main terms in OM:
• Behavioral change: Outcomes are defined as changes in the behavior, relation-
ships, activities, or actions of the people, groups, and organizations with whom a
program works directly. These outcomes can be logically linked to a program’s activi-
ties, although they are not necessarily directly caused by them.
• Boundary partners: Those individuals, groups, and organizations with whom
the program interacts directly and with whom the program anticipates opportunities
for influence. Most activities will involve multiple outcomes because they have multi-
ple boundary partners.
• Contributions: By using Outcome Mapping, a program is not claiming the
achievement of development impacts; rather, the focus is on its contributions to
outcomes. These outcomes, in turn, enhance the possibility of development impacts
– but the relationship is not necessarily a direct one of cause and effect.
• OM provides a focus on people and institu
tional transformation that is often lacking in
techniques which emphasize the delivery of
outputs as indicators of achievement.
• Milestones that indicate a possible process, not
final indicators; these indicate a path of
change that makes it possible to assess the
development in short time period and
therefore to assess / change / adapt strategies
within a short time.
• OM is not intended for a technical evaluation
to assess the relevance of the programming
area or an evaluation of the cost-effectiveness
of one approach compared to another
• OMis only concerned with the change
processes which occurred in those whom the
program directly interacts with
Strengths Limitations
40. 40
Three general stages of Outcome Mapping :
In this stage, the project establish a consensus on what changes in the macro level that it will
help to bring about and then plan the strategies to be used to achieve the changes. Organi-
zation should clearly express the long-term downstream impacts that they want to achieve,
bearing in mind that the project will not achieve the goal single-handedly.
It helps to answer four questions:
- Why? (What is the vision to which the program wants to contribute?);
- Who? (Who are the program’s boundary partners ?);
- What? (What are the tangible changes that are being sought?);
- How? (How will the program contribute to the change process among its bound
ary partners?)
This stage provides a framework for the on-going monitoring of a project’s actions and the
boundary partners’ progress toward the achievement of outcomes. It is based largely on
systematized self-assessment. It provides the following data collection tools for elements
identified in the Intentional Design stage: an Outcome Journal’ (to track impact against
progress markers); a Strategy Journal’(that seeks to test and adapt the programs strategy in
ever changing circumstances); and a‘Performance Journal’(that logs organizational practices
and gauges the need for improvements).
It helps the project identifies evaluation priorities (more in-depth review of progress) and
develop an evaluation plan that makes good use of resources and provides strategic benefit
to the project
1. Design stage
(Intentional Design
Stage)
2. Monitoring stage
(Outcome and
Performance
Monitoring)
3. Evaluation planning
10: “Outcome Mapping: A Basic Introduction” by Research to Action, http://www.researchtoaction.org/2012/01/
outcome-mapping-a-basic-introduction
11 : Boundary partners are individuals, groups or organizations with which the program interacts directly and
which the program hopes to influence
10
11
41. 41
Recommended Use of OM:
For interventions that the target outcomes are in the forms ofchanges in attitude,
knowledge, practices and behavior at stakeholder level (boundary partners).
Based on the indicative assessment by OutcomeMapping Learning Community from
September 2011 – March 2012, OM is dominantly used in Governance Sector.
Potential use of OM in IKAT US Partnership:
Building Bridges.
Its main intervention is to develop partner civil society organizations’ (CSO) research
and advocacy skills. OM can be useful because it enables the partnership to measure
changes in the knowledge & practice of its regional partners in using PETS, CRC and
Social Audits in each distinct national context as well as to measure changes in
attitude of its advocacy targets.
Human Rights in ASEAN.
OM tool can be used to map outcomes related to changes in capacity of Indonesian
lawyers and legal CSOs to serve as regional human rights advocacy resources and
changes among boundary partners due to regional advocacy strategy pursued by this
intervention.
AGENDA.
OM will be a great tool to see changes in knowledge and practice of DPOs (Disabled
People’s Organizations) and CSOs (Civil Society Organizations) within AGENDA’s
outreach and intervention in relation to the issue of political rights of people with
disabilities as well as the outcome in capacity building for electoral monitoring
bodies. Furthermore, it is also useful to track changes in the lobbying to influence
ASEAN policy-making.
Similarly, OM can also be applied to intervention with strong capacity building and
advocacy element, such as POWER as well as for exchange of knowledge in the case
of South-East Asia Partnership for Better Governance in Extractive Industries.
Remember!
OM is meant to be flexible, complimentary approach to traditional M&E methods. OM is
particularly effective when used from the planning stage, as it helps a program to focus on
supporting specific changes in its partners. With some adaptations, its various elements
and tools can be used separately or in conjunction with other processes (for example, a
SWOT, a situational analysis or an LFA).
42. 42
Box 7. Outcome Mapping in ACCESS Phase II
ACCESS Phase II has the following strategic directions:
• Strengthening engagement between civil society and government.
• A focus on empowering citizens’participation for democratization, from the
grassroots to village, sub-district, district and national levels.
• Scaling up impact through working across the systemic issues of district wide
governance and through contiguous geographic expansion.
ACCESS acknowledges that local actors – government and citizens - are the driving
force for better local democratic governance. Governance is a complex adaptive
system involving multiple actors and factors (of which ACCESS Phase II is one) so that
sustainable system change evolves from continually improving behavior changes,
including interactions with others. As people experience positive results in working
together, they are motivated to continue and scale up the benefits they have created.
This reinforces development of norms for a more inclusive, equitable and transparent
society and leads to self-directed local democracy.
ACCESS Phase II brings together interested actors, particularly those traditionally
excluded, to work out how to best set and respond to their own development priori-
ties, drawing on local capacities and assets. Outcome Mapping is well suited to this
approach with its systems and actor-centered orientation. It is interpretive and non-
linear and recognizes that change is influenced by many external factors and is not
always predictable. By encouraging collective sense making and continuous learning
for improvement, OM supports development of new understanding and places local
actors as the central agents of change rather than as implementers or recipients
The OM’s approach provides rationale for selection of CSOs as Boundary Partners
since they are the organizations best positioned to be influenced directly by the
Program and in turn have high potential to influence others (Ultimate Beneficiaries).
With ACCESS support, these Partners together with ultimate beneficiaries develop a
Vision, Outcome Challenge (anticipated behaviors of different actors in system by the
end of the Program), Progress Markers (sequence of desired behavior changes for
themselves) and Strategy Maps (how they will support their partners to make
change). Six monthly reflections with partners and stakeholders create shared under-
standing about change processes, provide data on the contributions of different
actors including ACCESS and identify what strategies need to be adjusted to
strengthen performance.
In Indonesia, there are two development programs ‐ VECO Indonesia and ACCESS
Phase II ‐ using Outcome Mapping (OM) as the basis for their intentional design and
respective monitoring process.
43. 43
Since ACCESS is only the second organization to implement OM in Indonesia (and the
first to do it in a large scale). Its process has been a trial-learning case. Nevertheless,
the overall ACCESS’ CSO partners and government partners have found the OM
approach useful in providing a clear rationale for determining who should work with
whom and tools for planning and monitoring through progress markers.
However, it is also the case that building awareness and capacity to use OM requires
time, ongoing reiteration of key ideas and expert technical assistance should be done
continuously. This is partially because it is conceptually challenging for CSOs to adjust
to using OM when they are used to the log frame approach and there is a lot of new
terminology (in English) that is difficult for non-English speakers to grasp easily. In
response to this learning, ACCESS has trained provincial ACCESS staff and local facilita-
tors in using OM to provide ongoing support for CSO partners and developed an
Indonesian Outcome Mapping which fits more with local linguistic and cultural
contexts of Indonesia, provides Indonesian terminology and explains differences
between OM and the log frame approach.
44. Theory of Change
44
What:
A Theory of Change (TOC) is a tool for developing solutions to complex social prob-
lems. A basic TOC explains how a group of early and intermediate accomplishments
sets the stage for producing long-range results. A more complete TOC articulates the
assumptions about the process through which change will occur and specifies the
ways in which all of the required early and intermediate outcomes related to achiev-
ing the desired long-term change will be brought about and documented as they
occur. (Adapted from Anderson, A. (2005). The community builder’s approach to theory of change: A
practical guide to theory and development. New York: The Aspen Institute Roundtable on Community
Change)
Generally, a theory of change includes:
• A logic model/results chain;
• The assumptions, risks and, in some cases, the mechanisms associated with
each link in the logic model/results chain;
• The external factors that may influence the expected results; and
• Any empirical evidence supporting the assumptions, risks and external factors.
Theories of change are referred to by a variety of names, including program theories,
impacts pathways, and pathways of change.
• Providing means to reach agreement among
stakeholders on just how the intervention is
expected to contribute to its intended aims. In
more complicated situations, it can help clarify
the intended contribution by the various
subcomponents of a broader intervention.
• Helps identify not only which results should be
monitored, but also which other factors should
be followed so that the intervention can be
on track and its progress monitored.
• Identifying where research may be needed to
better understand how the intervention works.
• They do not necessarily provide a quantitative
measure of the size of the contribution an
intervention is making.
• More than one theory of change may emerge.
If multiple theories of change emerge and are
strongly held, they may have to be tested
against the evidence to see which theory best
reflects reality
• While context is accounted for in the Theory of
Change, potential unexpected consequences
are not accounted.
Strengths Limitations
45. 45
Main Steps to Create a Theory of Change :
Start to design a simple map of the preconditions required to bring about the long-term
goal. The mapping process helps stakeholders to visualize and prioritize their goals as well as
specify what they expect to change and for which outcomes they want to be held account-
able for.
A more detailed stage of the mapping process. Building upon the initial framework, we
continue to map backwardly until we have a framework that tells the story we think is appro-
priate for the purposes of planning.
Because this work is challenging and most social change programs or broader initiatives
have a lot of moving parts, change frameworks usually go through many revisions. Outcomes
are added, moved and deleted until a map eventually emerges that tells a story the group
can agree on.
Identify the interventions that your initiative will perform to create these preconditions.
1) Identifying
long-term goals
2) Backward
mapping and
connecting
outcomes
3) Identify
interventions
The Indicators stage is when details are added to the changed framework. This stage focuses
on how to measure the implementation and effectiveness of the initiative. By collecting data
on each outcome, the initiative can identify what it is or isn’t happening and find out why.
Remember that each indicator has four parts: population, target, threshold and timeline.
4) Developing
indicators
5) Narrative of ToC Write a narrative that can be used to summarize the various moving parts in your theory.
12: Quoted from www.theoryofchange.org
12
46. 46
Recommended use of ToC:
Theory of Change (ToC), as its name, focus on changes. Different type of changes can
be mapped out – changes in attitudes, knowledge, awareness, skills, behavior,
health, financial status, social & economic conditions, visibility of issues, com-
munity norms and changes in partnerships.
Theory of change is increasingly used in research, advocacy and policy change
outcome as well as other projects seeking to achieve social change. An increasing
number of research donors are now recommending that researchers should develop
a theory of change (ToC) to help think through how they can do and communicate
the results of the research in a way that will maximize the value of the research for
policy and practice.
However there are few pre-conditions for optimal use of ToC approaches: i) It is
suitable for more open & learning organization, and ii) programs that are at early
stage in the grant process. It is found that it was harder for an organization to think
openly about its theory of change once it was already well into a project and had set
up many of its systems.
Box 8. Illustrative Use of ToC in IKAT US Partnership Program
As expressed by various reviews, ToC is great for complex program seeking to
achieve social changes. Thus, for complex-regional partnership program, such as
IKAT US Partnership, ToC will be a great tool!
However,ToC should also be developed at the beginning of grant process involv-
ing the partners,where each partnership will develop their own distinct theory
of change! And this process should be strongly facilitated since many partners
might not be familiar with the approach.
ToC will enable us to map out:
1) Changes that each intervention seeks to achieve
• Changes in human rights policy (in ASEAN level) or mechanism.
• Changes in knowledge or skills of legal advocates and CSOs in
advocating for human rights.
• Changes in knowledge and awareness of relevant stakeholders
on Convention on the Rights of Persons with Disabilities (CRPD)
leading to changes in policies or regulation in relation to
political rights of people with disabilities.
• Pathway of change: from access to knowledge (Regional
Election Access Index) to better advocacy and policy regarding
disability access to elections.
StrengtheningASEAN’s Human
Rights System
AGENDA
47. 47
• Changes in capacity and skills of Southeast Asian media and
regionally-based journalists leading to better quality media
coverage with strong emphasis on democracy, good govern
ance and human rights issues.
• Changes from visibility of issues to awareness and knowledge
of audiences in the region on democracy, good governance
and human rights issues leading to promotion of such values.
Asia Calling
• Changes in capacity of regional partners in using tools such as
PETS (Public Expenditure Tracking System), CRC (Citizen Report
Cards) among regional partners.
• Changes in budgeting and service delivery of government
counterparts.
• Pathway to change: from access to knowledge (online Social
Accountability Atlas) to improved performance of CSO-led
transparency and accountability projects in Southeast Asia.
Building Bridges
• Changes in knowledge and capacity of CSOs in Cambodia,
Malaysia, Philippines and Vietnam that lead to the adoption
and implementation of EITI (Extractive Industries Transparency
Initiative) by their respective government.
• Changes in attitude and policy of ASEAN to endorse transpar
ency and accountability in managing extractiveindustry
resources, including but not limited to EITI.
• Pathway to change: from research (knowledge) to capacity
building of women leaders and women’s political caucuses
leading to improvement of women’s political representation.
• Changes in legal framework for elections that increase women’s
political representation.
The partnership becomes more strategic, increased sharing of resources,
foster learning that leads to better project or program performance
Better Governance in Extrac-
tive Industries
IKAT US Partnership
POWER
2. Changes in Partnership
48. 48
Box 9. Experience of HIVOS: Building Southern Organizational
Capacity Through Theory of Change
Dutch funder, Hivos, became interested in theory of change as a way to support its
800 southern partners to focus more on outcomes than activities and to improve their
planning, learning, monitoring and evaluation. Hivos did not prescribe a rigid format
for partners to present information to them, but they noted that many have to use
logframe for other donors and rarely find them helpful for their own monitoring – and
even less for learning.
So Hivos tried a flexible approach to results-oriented planning, monitoring and evalu-
ation, based on indicators developed by their partners. Hivos found, though, that the
quality of planning, reporting and learning did not improve as much as they expected.
They therefore decided to explore further theory of change – which they had already
reflected on in an internal policy paper in 2004.
Since 2007, Hivos therefore carried out a number of workshops with partners in differ-
ent regions, facilitated by consultants. The workshops went well and most partici-
pants were initially very positive about the approach. But Hivos found that very few
managed to continue the process, even where they were offered consultancy and
funding support to do so. It was not always clear what held them back. Some partners
complained that it was just a way of imposing yet another kind of logic model on
them. Others feared that it would mean extra work as other donors continue to
demand logframe.
For that, Hivos reached to a wall. They still thought theory of change could be useful
in helping partners improve the effectiveness of their strategies and their learning, as
well as dealing with donor demands. But the capacity building workshops were not
working in the long-term. Hivos also recognized that to integrate theory of change
thinking in organizational practice needs very good facilitators who can adapt to the
process of different kinds of organizations.
To take the process further, Hivos engaged a small group of consultants – from CDRA
in South Africa, Wageningen University in the Netherlands and their Latin American
consultant. They began an action learning process with partners in Southern Africa
and South America to explore the value of applying theory of change thinking in
different ways and in different contexts. In South America, Hivos invited partners to
apply to be part of a longer term learning process – and received a very enthusiastic
response. Some partners in Bolivia are developing a group theory of change together,
as well as reflecting on their own organizational theory.
As the processes in the regions develop, Hivos hopes that the groups will gradually
expand to incorporate more partners and more consultants, generating interest
locally, rather than imposed by them as the funder. Hivos is also thinking about
setting up a web-based resource base.
Source: Theory of Change Review”, Comic Relief , September 2011
