1. EuropeAid
ENPI
CROSS BORDER COOPERATION
SEMINAR ON MONITORING
METHODOLOGY
2. The EC ROM system:
BASIC CONCEPTS
BRUXELLES, 9-10 SEPTEMBER 2010

2. Origin of ROM EuropeAid
• In May 1999 the Council of the European Union invited
the European Commission to establish a performance-
based monitoring system with the objective to strengthen
Monitoring, Evaluation and Transparency
• Monitoring is considered crucial for timely adjustments in
programming.
• Each Directorate General (DG) of the EC is called upon to
“design monitoring arrangements that ensure that
information on outcomes and use of resources is regularly
collected”.
• EuropeAid responded to this call with the establishment of
the Results-Oriented Monitoring system (ROM).
2

3. What is ROM?
EuropeAid
A “snapshot” project by project
According to a standardized methodology
It essentially assesses results and outcomes and not
just activities and use of resources

4. What ROM does? EuropeAid
• ….provides the EC with a wide range of quantitative and
qualitative data on the performance of the funded actions
• …..serves as a tool for day-to-day project management by
informing stakeholders about the performance of a specific
project, but it also contributes to general EuropeAid policy
articulation, implementation and review.
• …..thanks to the monitoring data collected in EuropeAid
database, provides an overview of the performance of the
overall EC portfolio.
• …thanks to data collected by independent experts through
regular onsite missions through a consistent, highly
structured methodology, it ensures the quality and the
comparability of the collected data.
4

5. ROM in the Project Cycle EuropeAid
5

6. Purpose of ROM EuropeAid
• ROM system gives an overview how the project portfolio
is progressing towards results, helping the answers to the
main questions:
- are projects and programmes (likely) to deliver the
intended improvements in the lives of people?
- is the project, with all due flexibility pursuing its planned
course and will it attain its objectives? Is it on track?
ROM can serve as a useful tool for identifying trends
which should then be complemented by more in-depth
analysis such as evaluations.
6

7. Main features of ROM system EuropeAid
• There are 5 main features characterizing the ROM
system:
The adoption of the accepted DAC evaluation criteria
The use of external consultants to ensure
independency and transparency
The application of a common standard methodology
throughout all missions
The capacity to consolidate data and information to
offer larger views on portfolio performance
The utilization of standard PCM / LFA approach
7

8. 1. ROM assessment criteria
EuropeAid
• The DAC evaluation criteria are the basic points to assess
the project performance.
• Relevance describes how well a project addresses a real
problem of the beneficiaries and how well it matches the
EC development policies strategic objectives.
• Efficiency stands for how well the inputs are transformed
into output and outcomes.
• Effectiveness measures the degree to which the project’s
outputs have provided benefits and contributed to the
project purpose.
• Impact describes how and to which degree the project has
contributed to the solution of the problem and to the
achievement of the overall objective.
• Sustainability measures to the likelihood of a continuation
of the benefits produced by the project after the period of 8
external support has ended.

9. DAC concepts - 21 Assessment criteria EuropeAid
•Relevance & Quality of design •Impact Prospects
•1.1 Present level of relevance •4.1 Wider planned impact
•1.2 Present design •4.2 Monitoring on wider impact
•Efficiency •4.3 Follow up on
•2.1 Availability of means / inputs recommendations
•2.2 Implementation of activities •Potential Sustainability
•2.3 Results achievement •5.1 Financial/ economic viability
•2.4 Partner contribution •5.2 Ownership
•Effectiveness •5.3 Policy Support
•3.1 Access to project results / •5.4 Capacity development
services •5.5 Socio-cultural adequacy
•3.2 Use of project results / services •5.6 Gender equality
•3.3 Achievement of Project Purpose •5.7 Technological adequacy
•3.4 Follow up on recommendations •5.8 Environmental adequacy
9

10. 2. Who are the monitors? EuropeAid
• ROM Monitors are external independent experts recruited
by the ROM contractors (private consulting companies)
after approval by the EC.
• Externality is required to avoid conflict of interest and
ensure independence of judgment
• Selection criteria:
Sector/thematic experience
Regional/country knowledge
Mix of Senior/Medium/Junior/Local
Availability
No previous direct involvement in the project
• Risks: costs, quality not always adequate, availability,
dependence on each expert commitment 10

11. 3. Basics elements of the standard EuropeAid
methodology
• The use of a common standardized methodology is one of
the most decisive features of the ROM system
• The standard methodology is based on four main
instruments:
A) common criteria to select the actions to be monitored
B) common procedures to implement the missions
C) use of common formats for collecting the information
through standardized question and grading criteria
D) Common outputs
11

12. 3a. Selection of actions to be monitored EuropeAid
• All projects with budgets exceeding 1 million €
• All projects that have been in execution for at least six
months (as of signature of the Financial Agreement etc.)
• All projects that have at least six months remaining before
closure
Start Implementation End
€ 1 million
6 months 6 months
Monitoring period
- Monitoring visits are avoided when a mid-term review is
taking place (period of six months before and after)
- When AIDCO or Delegations specifically request it,
projects below 1M€ can be included (special flexibility with
thematic / NGO / SA budget lines)
12

13. 3b. Procedures to implement the missions EuropeAid
Time allocation for ROM missions:
• The monitoring exercise is to be speedy and quick.
• Five working days are usually allocated for the field visit
of an ongoing project.
• Monitors have one day before the mission to get
acquainted with the project and study the files and two
more days after the mission to complete the reports.
• So total working days per monitoring report are 8 plus
travel (1 or 2 more)
• Ex-post ROM incl. an adequate preparation time to identify
interlocutors can require more days (up to 10 working
days).
• A ROM mission usually assigns two operations per
monitor. 13

14. EuropeAid
14

15. 3c. Common formats EuropeAid
• 3 special formats:
Project Synopsis provides a concise overview on project
background, logic of intervention and administrative data.
Background Conclusion Sheet is the key methodological
instrument for ROM providing the standard instrument
through a set of questions for monitoring and ensure
objectivity, consistency and comprehensiveness.
Monitoring Report is the main document to present
findings of the monitoring mission. It includes general and
financial information on the project, grades for 5 ROM
criteria (obtained via the embedded calculation in the
BCS) and a summary of conclusions.
15

16. 3c. Common scoring system EuropeAid
• The grading of a project/programme for each of the five
evaluation criteria mentioned gives a quick overview of its
performance. In addition, it allows for a comparison
between different operations and for the aggregation of
overall portfolio performance data.
• The grade is meant to be a summary of the answer, rather
than the answer being a justification for the grade
• However, grades strongly reduce the complexity of the
issues at hand and their context. They risk overshadowing
the actual findings.
• In the BCS, monitors should tick the performance
conclusion checkboxes next to the questions after having
given an answer in writing.
16

17. 3c. Common scoring system EuropeAid
17

18. 3c. Using the scoring system EuropeAid
18

19. 3.d Standard ROM Outputs EuropeAid
1. A two page monitoring report: it provides, short
report
explanations and recommendations on quality
improvement + important complementary information to
internal monitoring
2. In order to guarantee uniformity and consistency it is
supported by the ‘Background Conclusion Sheets’,
Sheets’
which ensure that all monitors address the same
questions and issues. Each sub criteria has to be graded
and the overall scores for each criterion are automatically
calculated and then used in the MR.
3. Project synopsis
4. A response from the field on quality and usefulness of
the report is also uploaded
19

20. EuropeAid
20

21. EuropeAid
21

22. Overview of EU ROM 10 years after EuropeAid
• ROM: a useful instrument at “micro” level. To inform
stakeholders on project performance and help project
managers “to think in result oriented terms”. ROM’s main
objective
• ROM: provides statistics on overall portfolio performance.
An overview for management decisions in AIDCO. ROM
added value (quantitative analysis)
• ROM reports (stored in CRIS) are one of the sources to
inform the programming cycle. Lessons learnt and
experiences: e.g. to the Quality Support Groups. ROM in
the learning cycle
22

23. Who benefits from ROM ? EuropeAid
• EuropeAid HQ – broad overview of project performance
and identification of patterns of problems that need to be
analyzed and supported.
• EC Delegations and Task Managers – timely,
independent review of portfolio of projects with
identification of principal weaknesses and corresponding
recommendations.
• Project Management – provision of independent advice
from appropriately qualified experts. Opportunity to assess
progress and make necessary adjustments.
• Final beneficiaries - opportunity to discuss progress and
improved delivery of project services and outputs
23

24. ROM results after 10 years EuropeAid
ROM on ongoing projects:
• More than 13,000 monitoring reports produced on 5500
project
• More than 180 countries visited
• In 2009 1,548 monitoring reports were produced
• The total amount of investments monitored was €8.3
billion.
• ROM covered 28% of the expenditures managed by
EuropeAid and 21% of the projects and programmes
ongoing in 2009
ROM Ex post:
• 700 projects/reports in 100 countries
ROM on SPSP :
• 34 missions in all regions from 2007 until the mid 2010
24

25. ROM results EuropeAid
Large majority of projects are performing well,
Slight improvement in 2009

26. ROM results
Performance by evaluation criteria 2009 EuropeAid
2

27. Using ROM data: EuropeAid
Factors for good and bad performance
ROM data, especially those collected Ex-
Post, can provide valuable lessons learned
for future programming and implementation.
A Study on “Causes underlying
Effectiveness and Impact of EC
Development Projects” identified 6 factors
each for good and bad performance:
27

28. Factors for poor performance: EuropeAid
I. Weak Analysis Stage/Identification Phase
II. Strategic relevance (for EC or partner
government) that does not correspond to real
relevance for the target groups and final
beneficiaries
III. Mismatch between allocated resources and
planned objectives
IV. Inadequate risk management
V. Non-existent or very poor internal monitoring
systems 28

29. Factors for good performance: EuropeAid
I. Proactive managers and inclusive
management approaches
II. Implementing partners with strong
organisational / technical capacity
III. Strong analysis stage to target real
problems/priorities with well-suited strategies
IV. Relevance for beneficiaries addressed
throughout the project’s lifetime
V. Demand-driven interventions resulting in
widespread access and use of services and
29
high levels of ownership

30. Some final points for your consideration EuropeAid
• ROM system is very effective in the production of large
amount of information. However to be fully exploited it
needs to be “re-processed” and “re-analyzed” in order to
extract sound and credible lessons learnt for the future
• ROM is a complex system based on a sophisticated
methodology that needs highly qualified and professional
human resources at every level (from general management
to field missions)
• This entails that the system could be relatively expensive:
a single report cost is estimated now around 9/10,000
euros, raising then the question of the “value added” of the
investment in case of project of minor size
• Giving the cost and the needed overall structure, the
frequency of the missions on same project (re-monitoring) 30
should then be decided with caution

31. Activity B – Identify theory / reality gaps EuropeAid
• Now you have a basic idea of what Monitoring is, how is
implemented , what can produce.
• Try now to identify theory/reality ‘gaps with the
environment of the ENPI CBC programme you are
working in
• Are there potential changes required and decision making
implications for ENPI CBC management?
• Think in terms of expected results, capacity building,
availability of resources, human resources, inter-country
agreement, etc.
31