Monitoring and Evaluation of Private Sector Development

Monitoring and Evaluation of
Private Sector Development
Hiroyuki Hatashima
Private Sector Evaluation,
Independent Evaluation Group (IEG)
World Bank Group
Canadian Evaluation Society Conference
June 11, 2013

IEG is part of a larger array of
evaluation and control mechanisms
2
World Bank Group
President / Management
Management’s Internal
Accountability Framework
And Systems
•Self Evaluation
•Monitoring (QAG, DOTS, AS)
•Institutional Integrity
•Internal Audit
•Compliance Advisor/ Ombudsman (IFC,
MIGA)
Independent
Evaluation
Group
Inspection
Panel
Board of
Executive Directors

IEG seeks to address both
accountability and learning
• Accountability for results:
– Is the Bank Group doing what it says?
– Is it achieving its objectives?
– To what extent is it contributing to development outcomes?
– With what results and development impact?
• Learning:
– What projects and programs achieve results – relevance,
efficacy, and efficiency – and why?
– How can M&E be strengthened in the Bank Group and in
client countries?
3

Three main messages
Evaluation Across Boundaries
• Private Sector operations have specific data
gathering advantage/disadvantage based on
their business model
• M&E systems should adopt to the business
practices to be effective and efficient
• M&E can influence learning, quality of work
and outcome

Outline of Presentation
1. Context of Private Sector Development Assistance
2. Types of World Bank Group Private Sector
Development interventions
3. Technical Assistance projects M&E
4. Private Sector Investment projects M&E
5. Use and Influence of M&E

1.1 Context: Private Sector As Important Player
1. The private sector has taken on an essential
role in contributing a wide range of
development issues.
2. Potential of private sector to contribute to
development solution is still underutilized.
3. M&E essential for track progress, assess
outcome & performance and generating
knowledge of what works, what does not, and
why.

Are the monitoring and evaluation systems of IFC
and MIGA equipped to inform the organizations on
their performance and results?
Systems functions
• To what extent
does the
mechanism in
place ensure
that M&E
systems
generate
credible, timely,
and relevant
information?
Use and Influence
• To what extent
does M&E
information
support
evidence-based
decision making
and learning?
Effectiveness and
Efficiency
• What has been
the impact of
the M&E
outputs and use
on project
quality and
development
outcomes?
1.2 Assessing M&E systems –
Evaluation Questions

4. Private sector Investment projects M&E

The World Bank Group consists of
five closely related institutions:
• IBRD: The International Bank for
Reconstruction and Development
• IDA: The International Development
Association
• IFC: International Finance Corporation
• MIGA: The Multilateral Investment
Guarantee Agency
• ICSID: The International Center for the
Settlement of Investment Disputes

1. Types of WBG Private Sector Development
interventions
1. Interventions through the Public Sector to influence
the Private Sector
 World Band Financial and Private Sector Development
projects,
 IFC’s Advisory Services projects to government (Investment
Climate, PPP)
2. Direct interventions in partnership with the private
sector and public entities operating commercially
 IFC’s investments and MIGA’s guarantees
3. Blend interventions
 IFC’s Advisory Services to private companies, but with a
strong public good component

5. Business Models and M&E

3.1. Technical Assistances follow
logical model results measurement
ImpactOutcomesOutputsActivitiesInputs
 Resources used
for advisory
services projects
 Programs
associated with an
advisory project
 Example:
– Provide
advisory
services to
local
companies
 Immediate
deliverables of
the project
 Example:
– Diagnostic
report and
recommendati
ons made
– 10 managers
were trained
– Legislation
drafted
 Changes in
knowledge,
behaviors, and
attitudes as a
result of
intervention with
attribution of
changes to the
project
 Example:
– Legislative or
inst reforms
– Changes in
agricultural
techniques, or
CG practices
 Consequences
resulting from an
intervention
including
improvements in
clients’
performance and
effects beyond
direct
beneficiaries
 Example:
– increased
investment,
sales, jobs

Track indicators
Project Supervision Report
(PSR)
Supervise
Approve Disburse Completion
Commit
M&E system
Logframe,
Identify indicators
Project
Completion
Report
(PCR)
3.2. M&E follows project cycle

Development Effectiveness Framework
Strategic
Relevance
Was it the right
project at the
right time?
Output
Achievement
Were “key”
deliverables
achieved? To
what extent were
clients satisfied
with the advisory
service?
Outcome
Achievement
Were intended
short & medium-
term changes in
behavior
achieved?
Efficiency
Were costs
reasonable vis-à-
vis results?
Impact
Achievement
Were intended longer-
term effects achieved?
What would have
happened without the
intervention? (What is
the counterfactual?)
• These combine into a “synthesis” rating
In addition: Assessment of IFC Role & Contribution:
– Was IFC particularly catalytic or innovative in its advisory services?
– Was IFC able to play a unique role not easily filled by others?
3.2. Advisory Services Project
Completion Report as Self-Evaluation

3.4 Technical Assistance M&E –
ways to adopt to business practices
Progress
 Integration with project cycle
 M&E officer playing important and
formal roles in quality assurance
 Quality control cycle
established
 M&E officers sign-off, Development
Impact Dept reviews
 M&E quality improved
 Quality of logical framework, baseline
data, quality of results discussion – all
improved
Challenges
 Standard indicators not
sufficient to track results
 Inadequate information in
PCR to justify
outcome/impact ratings
 Post completion follow-up
system not systematic
 Work quality assessment not
covered in PCR

3. IFC Advisory Services (Technical Assistance)
projects M&E
4. Private Sector investment projects M&E

4.1. Characteristics of Private Sector investments
supported by IFC and MIGA
 Operations subject to market test
Direct Beneficiaries pay for service
Some degree of competition
Sponsors motivated by profitability
 Third party effects (TPE) with no market feedback
Economic distortions
Positive and negative externalities
Provide rationale for public intervention
 Development Agency’s involvement
Consistent with market principles
Focused on third party effects (TPE)

Neighbors /
Environment
CustomersGovernment /
Taxpayers
Employees
4.2 Assessing Development Results - Stakeholder
Framework
Competitors
Producers of
Complementary
Products
Suppliers
New Entrants
Project
Company
(financiers)

4.3. Performance dimensions of IFC
investments and MIGA guarantees
Objective Stakeholders Criteria Standards Assessment
Development
Outcome
Financiers Business
success
(Market test)
Market
benchmarks for
ROIC
Actual financial
statements and
objectives at approval
Customers,
suppliers,
employees,
government
Economic
sustainability
(Third party
effect)
Social benchmarks
for ERR
Actual financial
statements, adjusted
by economic prices
and values
Neighbors,
environment
Environment and
social effects
(Third party
effects)
WBG performance
standards
Actual impact and
compliance with
performance
standards
Competitors, new
entrants, producers
of complementary
products
Private sector
development
(Third party
effect)
Objectives on
competition,
ownership
structure,
demonstration
effects
Qualitative
assessment of
achievements of
objectives on
competition,
ownership structure,
demonstration effects

Track indicators and
development outcomes
Supervise
DOTS*
*IFC’s Development Outcome Tracking System
Approve Disburse Repaid / Exit
Commit
Identify indicators
Expanded
Supervision
Report (XPSR)
4.4. M&E to adopt investment
project cycle

21
Development Outcome Framework
Financial
Performance
Financiers
Economic
Performance
Gov’t, taxpayers
employees, custome
rs,
suppliers
Envi. & Social
Performance
Neighbors &
Environment
Private Sector
Development
Demo. effects,
linkages, corp. gov,
investment climate
& new entrants
+
Standard Development Reach Indicators
IFC-wide: Taxes, employment (incl. female)
Plus sector-specific: e.g. MSME loans, utility connections
• Development Outcome: Overall effect on country’s development
• Framework: Derived from Good Practice Standards for Private Sector
Evaluation (investments)
• Each with specific indicators, benchmarks
3.2. Performance Assessment

2.2 Investment project M&E
Challenges
•Difficult to gather info beyond
company”
•Gaps in tracking private sector
development outcome
– Missing indicators, discrepancies
between stated outcome and
indicators
•Reliance on client-submitted
data
– Some data based largely on
assumptions
•Financial Institutions –
challenges in getting sub-
project level information
Advantages
•Financial data- good quality
and frequently updated;
•Enforcement via contracts /
covenants
•Integration with project cycle
– Use of standard indicators, with
baseline and target
•Quality control cycle
established
–Dept. review, Development Impact
Dept review, External Assurance for
Annual Report

Political Risk Guarantee Business model
and M&E challenges (MIGA)
Business Model
•Core competency as
guarantee underwriting,
not financing
•Arms-length relationship
to project enterprises;
•No statutory reporting
requirement from projects;
•Constraints to access –
–No membership on project
company boards of directors
Challenges
In data gathering
In communicating with
project entities
Challenges in
contextual analysis
Industry / sector
intelligence
23

Function of M&E in MIGA:
significant progress in intro of self-evaluation and
indicators.
Achievements
•New monitoring system
for Environment & Social
•Mainstreaming self-
evaluation
•Development indicator
tracking system
introduced
Areas of improvement
Challenges in data gathering
Inconsistent E&S follow-up on
missing documents
DEIS –ex-ante data not
covering all indicators.
Pool of evaluation not
adequate to report portfolio
results
24

2. Types of WBG Private Sector Development
interventions
5. Use and influence of M&E

3.1 Lessons can have a positive influence on
results
IFC Investment
•Lack of lessons / application of
lessons at appraisal associated
with poor outcome
– 80% of unsuccessful projects – low
quality at entry was main factor
– Low DO with low quality at entry =
40% of them without lessons
60% of them lessons were not
integrated in design.
•Improved performance by
learning from past failures
(Agribusiness case study)
– Newer successful projects applied
lessons from earlier failed projects
IFC Advisory
 Lessons influence design and
implementation
 New template asks “how
lessons have been applied to
project design”
 Wealth of lessons from PCR
 Segmented categories of lessons
 High use of past project PCR in
new projects
 Low use of “SmartLessons”
 M&E officers review at entry

3.2 Issues have been detected during
supervision and actions have been taken
IFC Investment
•High quality of supervision work:
83% high (XPSR)
•Half of XPSR identified issues
during supervision, 90% of them
were detected, 63% of them
were fully addressed during
supervision.
•Credit and Environment risk
systems as the most frequently
used monitoring systems
IFC Advisory
 PSR widely used
 95% of staff used PSR to
make adjustment.
 88% of supervision issues
identified in PCR captured in
PSR.
 Poor supervision critical factor
of 25% of projects rated low in
DE

3.3 M&E information is increasingly
used in IFC Strategies
• Increasing use of M&E data
• Annual Strategy process
– Development impact Dept. provide data and
participate discussion.
– Guidance: asks development issues to cover
• Ratings and reach data used in strategy formulation
– Historical DOTS rating comparisons between
sectors – prioritization of subsector
– DOTS ratings vs. profitability
– Gap analysis using “reach indicators” – strategy to
sharpen focus, areas to reach (i.e. BOP)
– Development return of different options using reach
data (reach per $ cost of capital)

Three main messages
• Private Sector operations have specific data
gathering advantage/disadvantage based on
their business model
• M&E systems should adopt to the business
practices to be effective and efficient
• M&E can influence learning, quality of work
and outcome

Outstanding issues
• Difficult to estimate benefit of M&E
–Need innovative approach, some
assumptions about counterfactuals;

Monitoring and Evaluation of Private Sector Development

Recommandé

Recommandé

Contenu connexe

Tendances

Tendances (20)

En vedette

En vedette (20)

Similaire à Monitoring and Evaluation of Private Sector Development

Similaire à Monitoring and Evaluation of Private Sector Development (20)

Plus de Independent Evaluation Group /World Bank Group

Plus de Independent Evaluation Group /World Bank Group (20)

Dernier

Dernier (20)

Monitoring and Evaluation of Private Sector Development

Notes de l'éditeur