This presentation by Margaret Miller was made at the High-level Global Symposium on Financial Education: Promoting Long-term Savings and Investments in Korea which explored policies and good practices for supporting long-term savings and investments through financial education and financial consumer protection. Find out more at http://www.oecd.org/daf/fin/financial-education/globalsymposiumonfinancialeducationforlong-termsavingsandinvestments.htm

1. Can you teach someone to
be financially capable?
Resultsfrom meta-analysis of the
impact evaluation literature
February 27, 2014
HIGH-LEVEL GLOBAL SYMPOSIUM ON FINANCIAL EDUCATION
Margaret Miller
Senior Economist, Financial Inclusion and Infrastructure Practice
The World Bank Group

2. Organization for Talk
I. Literature review for financial literacy / capability
II. Introduction to / rationale for meta-analysis
III. Descriptive statistics on the data
IV. Discussion of meta-analysis results
V. Concluding remarks

3. Motivation for work on financial
literacy / capability
• Significant body of evidence that people are making financial
mistakes (Shui & Ausubel 2004; Gross & Souleles 2002; Firestone,
Van Order and Zorn 2007)
• Research suggests that less financially literate / knowledgeable
consumers are more prone to making financial errors (Lusardi &
Tufano 2009; Stango & Zinman 2009)
• Regulations and consumer protection can help address some of
these problems but there are limits (Gine and Mazer 2013;
Christelis, Jappelli and Padula 2010; Willis 2011)
• This suggests a role for efforts to increase financial knowledge
and skills – but can this be done effectively?

4. Literature reviews focus on different
studies and provide inconclusive results
Pearson Correlation Table
References
M
artin
2007
H
athawayand
K
hatiw
ada2008
M
cCorm
ick
2009Agarw
aletal2010
Collinsand
O
’Rourke2010
G
aleand
Levine2010
H
astings,M
adrian,Skim
m
yhorn
2012
Xu
and
Zia
2012
Lusardiand
M
itchell2013
Martin 2007
1.000
(1.000)
0.416 1.000
(0.173) (1.000)
-0.067 0.001 1.000
(0.004) (0.000) (1.000)
0.118 0.097 -0.017 1.000
(0.014) (0.009) (0.000) (1.000)
0.161 0.104 0.026 0.425 1.000
(0.026) (0.011) (0.001) (0.181) (1.000)
0.171 0.190 -0.088 0.174 0.047 1.000
(0.029) (0.036) (0.008) (0.030) (0.002) (1.000)
-0.057 0.009 -0.060 0.075 0.036 0.084 1.000
(0.003) (0.000) (0.004) (0.006) (0.001) (0.007) (1.000)
-0.077 -0.099 -0.065 -0.088 -0.100 -0.083 -0.175 1.000
(0.006) (0.010) (0.004) (0.008) (0.010) (0.007) (0.031) (1.000)
-0.104 -0.038 -0.155 -0.105 -0.167 0.039 -0.002 -0.217 1.000
(0.011) (0.001) (0.024) (0.011) (0.028) (0.002) (0.000) (0.047) (1.000)
Total Number of
References 44 35 45 82 57 61 100 132 187
Xu andZia 2012
Lusardi and
Mitchell 2013
Agarwal et al 2010
Gale andLevine
2010
Hathaway and
Khatiwada 2008
Martin 2007
Collins and
O’Rourke 2010
McCormick 2009
Hastings, Madrian,
Skimmyhorn 2012

5. Our approach: first focus more narrowly on
interventions then cast a wide net
• Papers were included if they discussed the impact of a financial
literacy / capability intervention
• Papers were identified through a variety of approaches:
• Search of peer-reviewed articles in Econlit (search terms financial
capability, financial literacy and financial education)
• Including all papers from leading literature surveys published over the
past 5 years
• Impact evaluation papers funded through the Russia Trust Fund for
Financial Education and Literacy

7. 6%
9%
2%
6%
75%
2%
Region in which Intervention Conducted
(n=188)
Africa
Asia
Europe
Latin America
USA
Other

9. 54%
9%
5% 3%
6%
3%
18%
3%
0%
10%
20%
30%
40%
50%
60%
Course
taught in
person
Individual
Counseling
Mass
Media
Phone Print Online Mixed Other
Intervention Channel
(n=188)

13. Difficulties in using meta-analysis
for this research question
• Diverse interventions across many dimensions
• Type of intervention
• Duration and intensity of exposure to intervention
• Topic / objective of intervention
• Target population
• Lack of uniformity in defining outcomes
• Lack of statistical rigor / information including many studies
with small sample sizes
• Comparability of effect sizes is difficult
• Clustering and additional controls vary across studies
• Actual effect sizes/impact depend on control group means

14. Results of the meta-analysis
• Comparable information only available for a few outcomes
(all binary variables)
• Savings
• Retirement savings
• Default on loan
• Record keeping
• Information also available on financial literacy test scores
but not comparable due to diverse tests, scoring scales

15. NOTE: Weights are from random effects analysis
Overall (I-squared = 54.5%, p = 0.051)
Drexler Fischer Schoar
Authors
Bruhn Leão et al
Bruhn Ibarra McKenzie
Berg & Zia
Cole Sampson Zia
Doi McKenzie Zia
2011
Date
2013
2012
2013
2011
2012
Dominican Rep
Country
Brazil
Mexico
South Africa
Indonesia
Indonesia
15
Intensity
36
4
26
2
13
1
Channel
1
7
3
1
1
0.03 (0.00, 0.06)
0.14 (-0.02, 0.30)
ES (95% CI)
0.05 (0.04, 0.06)
0.03 (-0.01, 0.06)
-0.03 (-0.09, 0.03)
0.01 (-0.07, 0.09)
0.10 (0.00, 0.20)
100.00
3.73
%
Weight
36.13
24.33
15.98
11.54
8.29
0.03 (0.00, 0.06)
0.14 (-0.02, 0.30)
ES (95% CI)
0.05 (0.04, 0.06)
0.03 (-0.01, 0.06)
-0.03 (-0.09, 0.03)
0.01 (-0.07, 0.09)
0.10 (0.00, 0.20)
100.00
3.73
%
Weight
36.13
24.33
15.98
11.54
8.29
0-.297 0 .297
Papers testing savings behavior after financial education intervention

16. NOTE: Weights are from random effects analysis
Overall (I-squared = 87.7%, p = 0.000)
Hung Yoong
Muller
Authors
Beshears, Choi, Laibson, Madrian, Milkman
Bernheim & Garret
Lusardi and Mitchell - ALP article
2010
2003
Date
2011
2003
2007
USA
USA
Country
USA
USA
USA
Varied
Varied
Intensity
One letter sent
Varied
Seminars
2
1
Channel
5
7
NA
0.08 (0.01, 0.16)
0.06 (0.01, 0.11)
0.09 (-0.06, 0.24)
ES (90% CI)
-0.04 (-0.07, -0.01)
0.12 (0.07, 0.18)
0.23 (0.13, 0.33)
100.00
%
22.98
12.98
Weight
24.21
22.23
17.60
0.08 (0.01, 0.16)
0.06 (0.01, 0.11)
0.09 (-0.06, 0.24)
ES (90% CI)
-0.04 (-0.07, -0.01)
0.12 (0.07, 0.18)
0.23 (0.13, 0.33)
100.00
%
22.98
12.98
Weight
24.21
22.23
17.60
0-.327 0 .327
Papers testing retirement savings behavior after intervention

17. NOTE: Weights are from random effects analysis
Overall (I-squared = 69.1%, p = 0.012)
Sonobe Suzuki Otsuka Nam
Mano Akoten Otsuka Sonobe
Bruhn Ibarra McKenzie
Gibson McKenzie Zia
Authors
Doi Mckenzie Zia
2011
2010
2012
2012
Date
2012
Vietnam
Ghana / Kenya
Mexico
Australia/NZ
Country
Indonesia
NA
45
4
2
Intensity
13
1
1
7
1
Channel
1
0.04 (-0.00, 0.09)
0.06 (-0.00, 0.12)
0.18 (0.05, 0.31)
0.01 (-0.03, 0.04)
-0.05 (-0.11, 0.02)
ES (90% CI)
0.09 (0.04, 0.14)
100.00
21.11
9.16
27.17
19.07
%
Weight
23.49
0.04 (-0.00, 0.09)
0.06 (-0.00, 0.12)
0.18 (0.05, 0.31)
0.01 (-0.03, 0.04)
-0.05 (-0.11, 0.02)
ES (90% CI)
0.09 (0.04, 0.14)
100.00
21.11
9.16
27.17
19.07
%
Weight
23.49
0-.314 0 .314
Papers testing record keeping behaviors after intervention

18. NOTE: Weights are from random effects analysis
Overall (I-squared = 87.6%, p = 0.000)
Berg & Zia
Bruhn Ibarra McKenzie
Gine Mansuri
Agarwal et al
Authors
2013
2012
2011
2010
Date
South Africa
Mexico
Pakistan
USA
Country
26
4
46
35
Intensity
3
7
1,2
1
Channel
-0.02 (-0.06, 0.02)
0.04 (-0.04, 0.11)
0.02 (-0.01, 0.05)
-0.02 (-0.04, -0.00)
-0.09 (-0.12, -0.06)
ES (90% CI)
100.00
16.83
26.81
%
28.82
27.54
Weight
-0.02 (-0.06, 0.02)
0.04 (-0.04, 0.11)
0.02 (-0.01, 0.05)
-0.02 (-0.04, -0.00)
-0.09 (-0.12, -0.06)
ES (90% CI)
100.00
16.83
26.81
%
28.82
27.54
Weight
0-.115 0 .115
Papers testing loan defaults after intervention

19. Focus on Savings and
Retirement
Interventions

28. Assessing the findings
• The small sample sizes that we ultimately have for the meta-
analysis limit our ability to draw conclusions
• Effect sizes are not comparable as typically data are not
available for this estimation from the published works
• Studies using RCTs show limited impact of financial education
interventions at best
• With these caveats, some insights do emerge:

29. Assessing the findings
• Meta-analysis suggests that financial education can impact some financial
behaviors:
• Increase saving in general and retirement saving in particular
• Promote record keeping
• Descriptive analysis of savings and retirement interventions shows
impact across a much larger portion of the literature than is included in the
meta-analysis
• Results also suggest that it may be more useful in some circumstances than
others
• Not found to be significant factor for avoiding default – may be due to importance
of circumstances beyond borrower’s control
(health, loss of job, etc.)
• Some surprises too – for example the intensity (hours of exposure) of the
interventions was found to have statistical significance only for record-keeping
• not conclusive evidence but worth further research

30. For the future
• More attention to cost:benefit ratio: Virtually no discussion of
cost of the interventions or alternative methods to achieve
the desired outcomes (Cole, Sampson, Zia 2011 is exception)
• Importance of strengthening / expanding use of rigorous
evaluation methods
• Potential value from defining outcome variables in common
terms (including use of binary dependent variables) to
facilitate comparisons
• Importance of reporting more complete statistical
information
• Value from developing / contributing to a registry for
financial literacy interventions