IPC-IG's Research Coordinator, Fábio Veras Soares, presentation at the "International Conference on the Institutionalization of Public Policies Evaluation", held in Rabat, on 5 October.

1. The role of monitoring and evaluation
in improving public policies –
challenges and achievements.
International Conference on the
Institutionalization of public policies Evaluation
Rabat, 5 October 2015
Fabio Veras Soares (IPC-IG)

2. M&E: what for?
Monitoring and evaluation are long acknowledged as essential
elements of the policy making process.
Monitoring and evaluation activities are meant to inform…
 governments,
 policy makers,
 practitioners and society at large
whether programmes are working…
 as planned,
 over-performing, or..
 under-performing.

3. M&E: connected but not the same
 Monitoring and evaluation should be interconnected but they are
essentially two different process.
 It is possible to have a good monitoring system without having an
evaluation and vice-versa.
 However, a good monitoring system will never be able to respond to
the key evaluation question: What would have happened to the
target population in the absence of the intervention that has been
monitored? Would they have been better off or worse off or just the
same?
 The possibility to attribute observed changes in outputs and
outcomes to a specific intervention is at the heart of the evaluation
problem.

4. M&E: monitoring role
 a good monitoring process provides information with a periodicity
that enable quick action to rectify implementation hurdles.
 Through the record of the inputs, process and outputs of a
programme, a monitoring system provides constant feedback on the
extent to which the intervention is achieving its targets.
 Besides identifying potential problems at an early stage, it should
also be able to suggest possible solutions.
 The provision of a reliable flow of accurate information during
implementation allows managers to keep track of progress and
adjust activities accordingly.
 The monitoring system also allows to assess how accessible the
intervention has been to the target population.
 Finally, it gauges the efficiency with which the different components
of the intervention are being implemented and suggests
improvements, providing internal and external accountability of the
resources used and the results obtained.

5. M&E: the importance of monitoring for the
evaluation process
 Monitoring systems per se are not able to answer the evaluation
question, but…
 They offer the inputs – indicators and qualitative information– that
will support the evaluation process…
• by enhancing our understanding of the reasons for a good, bad or
normal performance of a programme.
• by proving key information on costs that will allow the measurement
of the cost-effectiveness of the programmes and interventions.

6. M&E: the importance of monitoring for the
evaluation process
The Evaluation component of an intervention analyses and interprets
the data collected to understand the causal relationship between inputs
invested into the programme, its components and the overall impact on
outputs and outcomes.
In order to do so in a rigorous way, particularly, when performing an
impact evaluation, it is important to be able to measure a
counterfactual.

7. Impact evaluation: the counterfactual
The counterfactual is the potential output/outcome indicator in the
absence of the programme.
The difference between the observed output/outcome and the
counterfactual is the estimate of the impact of the programme.
As one never observes the same population under both states –
affected by programme and not affected by the programme – it is
necessary to build a control or comparison group as similar as possible
to the target/treated group in order to proxy this counterfactual and
estimate the impact of the intervention.

8. Impact Evaluation: counterfactual
 The requirements to build a credible counterfactual can be quite
cumbersome (and sometimes expensive vis-à-vis the overall cost of
the intervention). Thus one should be able to prioritize what should
be evaluated and how using some clear criteria.
 Ideally, the comparison group should be identified at the design
phase of the intervention. Data collection includes both
target/treated population as well as the comparison/control group.
 As data is also collected for population groups not immediately
benefiting from the interventions, most likely the monitoring and
information system of the programme will not suffice for this
process. The evaluation component, in general, needs to have its
own data collection process. Early definition of the evaluation
procedure potential can allow the incorporation of the comparison
group in the MIS system, improving the quality of the evaluation
and, in some cases, reducing the costs of the evaluation.

9. Evaluation starts with the design of an
intervention
 Treating the M&E process as an ex-post phase of an intervention
increases its costs and jeopardizes its credibility as second-best
approaches and methodologies need to be used.
 However, the rule has not been to implement the M&E component
from scratch and the reason behind it is that many governments and
policymakers still see monitoring and evaluation as a source of
potential criticism and either do not actively search to make it
current practice or do not take full advantage of it – overlooking its
findings.

10. Investment case for M&E and political will
 To show that the benefits of a proper M&E system outweighs its
costs (including political costs) is one of the challenges faced by
practitioners worldwide.
 During the International Conference on National Evaluation
Capacity in 2013 (São Paulo, Brazil) practitioners from 52 countries
identified among the challenges to implement and strengthening the
evaluation components of public policy issues related to the
engagement of parliamentarians and the development of legislation
and institutional arrangements that could incentivize the
implementation and use of evaluation results in the improvement of
public policies.

11. Important successful cases…
The CONEVAL experience in Mexico is a clear example of the type of
institutions that has the ability to foster evaluation of public policies,
ensure accountability and put forward suggestions on how to improve
their efficiency.
A similar role is developed by the ONDH whose institutional mission in
Morroco is to assess and evaluate in a continuous way the impact of
human development programmes and to put forward suggestions to
improve these programmes.
In Tunisia the CRES has been supporting the government in the
reform of their Social Protection System through rigorous research and
evaluation components.

12. Strong M&E for the achievement of the SDGs.
The SDG is due to set an important role for the evaluation community
both at the international and national level. "Evaluation" has been
integrated in the new Agenda within the "follow-up and review" section.
This section suggests that the follow-up and review processes of
implementation of the post-2015 Development Agenda will be informed
by country-led evaluations and data and calls for strengthening
national evaluation capacity.
In this conference we will have the opportunity to learn more about the
work of these institutions and to reflect how they can contribute to the
strengthening of national and international evaluation capacities and
the improvement of public policies.

13. Merci beaucoup
‫شكرا‬
Gracias
Thank you
Obrigado