This presentation describes lessons and recommendations from the evaluation of World Bank Group response to global financial crisis of 2009-2010.

1. Evaluation Approaches, Methods and Data in a Rapidly
Changing Context: Global Crisis Response Evaluations
at the World Bank Group
Independent Evaluation Group
World Bank / IFC / MIGA
European Evaluation Society
Helsinki, October 2012
Anjali Kumar, Andaleeb Alam and Ali Khadr
IEG World Bank Group
1

2. The Global Economic Crisis and the Challenge
to WBG
Rapid movement of economic events
during the global crisis of 2008-2009
 Some stock markets lost 50% of value in days
 Global growth slowdown (3.9 % to -2.1% )
 Unprecedented scale, contagion
 Estimated 50-64 million more poor people
Motivating strong response from WBG
 117 countries received Bank loans during 2009-
10; tripling of IBRD lending to $100m
 17 received crisis support during 1993–2003
Call for ‘real time’ evaluation of response
2

3. Crisis Evaluation: Nature of the Challenge
Daily data show the decline in value of the UK Complexity of Crisis
FTSE index from 5,600 to 3,800 in weeks
Response Evaluations
Events are sudden,
unexpected and fast-evolving in
character
Evaluative lessons are needed
in a compressed time frame
Conventional evaluation
techniques that use information
based on a sequential logical
framework are of limited use
3
Uncertainty prevails

4. Crisis Evaluation: Nature of the Challenge
Scope of the present paper
Questions asked in crisis evaluation
Data usable to answer such questions
Use of high frequency data
Crisis evaluations at other IFIs / MDBs
Are such economic events
becoming more frequent?
‘Black swan’ events may increase
because they are probabilistic outliers
Increased need for corresponding
4
evaluative techniques

5. Crisis Evaluation –Nature of Evaluative
Questions
Evaluative Questions – complex
evaluations under uncertainty
 Skills demonstrated in dealing
with complexity and change
Speed of response; reflecting
information on events as they occur
Identification of risks involved
Results framework, learning:
Quality at entry
5

6. Crisis Evaluation: IEG’s Phased Response
Monthly data on emerging markets showed
IEG prepared a series of ‘real time’
declines in industrial production and credit evaluations on WBG Crisis
growth not revealed in annual averages Response
12% 1.5
Review of WBG Response to Past Crises
10% (2009)
1.0
Industrial Production (sd units)
Private Credit Growth (q-o-q)
17 Country Case studies – no current crisis data
8%
0.5 Phase I Evaluation of WBG Crisis
6% Response (2010)
4%
0.0
Real time evaluation focused on volume, speed,
and early results – selective use of data to
2%
-0.5
benchmark Bank actions
0% -1.0
Phase II Evaluation of WBG Crisis
Response (2011)
Jan-08
Jan-10
Jan-07
Jan-09
Jul-07
Jul-08
Jul-09
Jul-10
Oct-07
Oct-08
Oct-09
Oct-10
Apr-07
Apr-08
Apr-10
Apr-09
Used high frequency data to analyze patterns of
Private Credit Growth (q-o-q) Industrial Production (sd units) Bank support relative to crisis incidence
6

7. 1. Retrospective Evaluation - WB Response to
Past Crises – 17 case studies
Evaluative questions based on preceding principles
Focus on:
Scale and modality of the lending response
Impact on the Bank’s own balance sheet
Partnerships with other multilateral agencies
Content of the WB crisis-response operations
Macroeconomic trade and financial and fiscal
Use of Data
The crisis retrospective did not look at data on the
current crisis,
It did trace data on previous crises-
–extent of World Bank’s response relative to its
baseline lending,
–and relative to other international financial
institutions;
7 –time taken to return to normal lending patterns.

8. 2. WBG Response to the Global Crisis:
Phase I Evaluation
Private Capital Flows during the crisis: Evaluative questions :
January 2007–December 2010 Design aspects –readiness,
relevance, poverty focus,
Implementation aspects –
Speed, internal organization,
instruments, monitoring and
evaluation
Early Outcomes and
Prospects- Meeting Objectives,
additionality, debt sustainability
Data Use:
Annual economic data used
to trace effects of crisis on Bank
borrower / client countries
8

9. 2. WBG Response to the Global Crisis
Phase I Evaluation
Alternative Scenarios – Poverty in Phase I Example:
Developing Countries in 2015 and 2020
Data Use in Evaluation
under Uncertainty:
Scenario analysis
Ten year projections of
poverty impact of the crisis
Based on a series of
recent data on GDP growth,
trade, capital flows and
demand assumptions
9

10. WBG Response to the Global Crisis -Phase II
Scope of the Evaluation
Utilization of high frequency data across a
range of economic variables
 To track multiple dimensions of stress
 These multiple measures of stress were
only partially correlated.
 Analysed relative to other IFIs and MDBs
Multidimensionality of crisis
 Exchange rate and foreign exchange
reserve stress
 Financial Stress, including market, credit
and banking system
 Social indicators – unemployment,
private consumption; and
 Fiscal deficit and public debt as a
percentage of GDP)
10

11. WB Response to the Global Crisis
Phase II Evaluation Methods
Use of High Frequency data to measure multiple dimensions of stress that
were not necessarily correlated
11

12. Evaluating the Lending Response relative to
stress
Incremental Lending Relative to Levels of Simple and Composite stress measures
Crisis: World Bank
 Simple - changes in GDP;
 Composite - principal factor analysis
700%
 Peak to trough, period average
600% Illustrative bands ranking countries according
to stress in diagram
500%
 Are compared with the distribution of
% Increase in Lending
400%
incremental lending.
300%
Underlying these data are regressions
 That measure crisis intensity and
200% incremental Bank response on a
100%
continuous basis
Caveats and limitations of the analysis
 Country demand, country performance,
0%
other IFIs
Growth Decline  Controls can however be introduced for
some of these factors
12

13. Volume and Distribution of Support -
Other IFIs / MDBs
Incremental Lending Relative to Levels of Increased WB lending patterns were also
Crisis: Comparisons with Other Donors compared with other MDBs
12 1.2
 Correlations of each institution’s
incremental support to crisis intensity
Increase in Other Major Donor Lending
in client countries
10 1
Increase in World Bank Lending
8 0.8  Based on data obtained from relevant
IFIs
as % of GDP
as % of GDP
 Comparisons undertaken at multiple
6 0.6
4 0.4 levels
 Response to changes in GDP alone, to
2 0.2
composite overall stress
measures, and compared to specific
measures of stress
0 0
Do not analyse other MDBs’ overall
Other Major Donors (incl. IMF)
Other Major Donors (excl. IMF/EIB/EU)
response
 Comparisons limited to countries that
World Bank
13
were common borrowers

14. Crisis Evaluation and High Frequency Data - EBRD
EBRD’s crisis response evaluation established a three phase typology
Based on two high frequency data series on interest rates
 European Central Bank deposit facility rate
 ‘FED funds rate
14

15. Crisis Evaluation and High Frequency Data -
EBRD
EBRD’s evaluation then traces crisis response in each phase
 Eg: evolution of EBRD loan pricing in response to market signals
15

16. IMF – Evaluation of the Fund in the Run Up to
the Crisis
The IMF evaluated the quality of its surveillance in the run up to the crisis
 Evolution of crisis traced also using high frequency data:
 EMBI Global Spread (Left)
 TED spread (right)
16

17. Conclusions and Suggestions
Rapidly changing situations pose special challenges
for evaluators
 Different questions
 That can use high speed data for their response
 In order to give real time feedback
Financial and some macroeconomic data exist
 In multiple dimensions and high frequencies
 However more systematic monthly data on
GDP, fiscal deficits etc are still to be produced
More difficult are data on social dimensions
 Eg integrated global databases on employment or
consumption
Greater global cooperation on such data would be a
benefit
 To policy makers and to evaluators
17