Science Forum 2013 (www.scienceforum13.org) Plenary session: Evaluating nutrition and health outcomes of agriculture Linxiu Zhang, Chinese Academy of Sciences, main respondent

1. Better nutrition and health through
impact evaluation!?
Response to “How to evaluate nutrition and health
impacts of agricultural innovations?”
Linxiu Zhang
Center for Chinese Agricultural Policy
Chinese Academy of Sciences

2. Contents
• Why we want evaluate impacts
• Managing impact evaluations
• Nutrition and health – more than ag.
Innovations
• Communicating findings to policy makers –
scaling up, REAP example

3. Challenge/ Excuses
• We don’t have time
• It costs too much to do rigorous impact
evaluation
• It is unethical
• Project implementation is completely site and
context specific
• We already know!

4. • So many of the innovations/programs that we heard
about on the news …
… how many of the new technologies/programs that
we have become excited about …
… how many have been rigorously evaluated?
• Do we have empirical evidence, based on a carefully
constructed counterfactuals, that these
breakthroughs/programs work, can positively affect
the lives of the poor and do so in a cost effective
way? Yes but not many

5. KEY to IE: the attribution problem:
factual and counterfactual
Impact varies over time
Impacts also are defined over time … Little attention has
been given to the dynamics over time … though people
think about this …
With project
Without
project
Project
impact

6. Change in the CAL program effect on the standardized
math test scores over time
-0.04
-0.02
0
0.02
0.04
0.06
0.08
0.1
0.12
0.14
0.16
0.18
0.2
Baseline (Sept. 1) Midterm (early Nov) Final (late Dec)
Time
Standardizedmathtestscore
The CAL program effect occurred by the midterm evaluation, less
than two months after the start of the program.
0
0.12
0.11

7. Change in the CAL program effect on the standardized
math test scores over time
-0.04
-0.02
0
0.02
0.04
0.06
0.08
0.1
0.12
0.14
0.16
0.18
0.2
Baseline (Sept. 1) Midterm (early Nov) Final (late Dec)
Time
Standardizedmathtestscore
There is no improvement between month two and month three..
0
0.12
0.11

8. Impact of nutrition intervention at
infancy in Guatemala
• After 2 years greater BMI
• After 10 years higher grades in school
• After 15 years higher school attainment
• After 40 years higher wages / income
Source: Maluccio et al. 2009

9. When to use what methods?
• RCTs Gold standard
– Where random assignment is possible
• Matching methods
– Quasi-experimental methods (construct a
comparison group)
– Propensity score matching (PSM)
– Regression discontinuity design (RDD)
– ‘Intuitive matching’
• Regression-based/Modeling
– Instrumental variables: need to be well-
motivated

10. Managing Impact Evaluation
• The importance of ex ante designs with
baseline (building evaluation into
design)
- A theory-based approach (using causal chain) allowing
analysis of causal pathways
• The importance of a credible design
with a strong team.
– Clear objectives, simple interventions
• Ensure management feedback loops

11. Budget and timeline
• Ex post or ex ante?
• Existing data or new data?
• Qualitative and quantitative?
• How many rounds of data collection?
• How large is sample?
• When is it sensible to estimate impact?

12. Nutrition and health: more than
ag. innovations
Example of China:
•Annual growth rate at 8-10%
•Agricultural growth rate at 3-4%
•High speed of urbanization and
industrialization
•Income for both rural and urban increased
rapidly
•Yet, China’s challenge in nutrition and health
is HUGE

13. Between 2008 and 2012 we tested
nearly 60,000 children across
China for iron-deficiency anemia

14. In fact, anemia is all over China
Luo, R., X. Wang, C. Liu, et al. (2011) “Alarmingly High Anemia
Prevalence in Western China.” Southeast Asian Journal of Tropical
Medicine and Public Health Vol. 42 No. 5
Total
Total 33.7
Shaanxi—2008 (Dataset 1) 37.5
Shanxi—2009a (Dataset 2) 31.6
Gansu—2010 (Dataset 3) 31.2
Qinghai—2009 (Dataset 4) 51.1
Ningxia—2009 (Dataset 5) 25.4
Sichuan—2010 (Dataset 6) 24.8
Guizhou—2010 (Dataset 7) 33.1

15. Poor areas of China
Children
with anemia
(≈ 33%)
Children with
out≈ 30 to 35 million school aged
children are estimated to be suffering
from malnutrition!

16. Non-poor areas
of China
Children
with anemia
(≈ 8%)
Children with
out (92%)< 5 million school aged children in
all of the rest of China

17. THE SCOURGE WITHIN:
INTESTINAL WORMS IN RURAL CHINA
We have tested more than 4000 children for:

18. Incidences of Intestinal Worms,
Guizhou Province, 2010 & 2013
3 to 5 year olds 8 to 10 year olds
33.9%
with
worms
40.1%
with
worms
Without Without
Zhang et al., 2013
… millions of children are infested with these …

19. Even earlier
(malnutrition during the first 1000 days)
• Testing ≈1000
babies and their
Mom’s in
Southern Shaanxi
(11 counties)
(these areas are 2 to 3 hour drive
from Xi’an – one of
China’s fastest growing
cities)

20. Main Outcomes of Interest
At the time of enrollment and at each follow-up
survey, we test babies for:
• Hemoglobin (anemia)
• Length & weight
• Gross & fine motor skills
• Cognitive development
• Social-Emotional
development

21. One of these test is the Bayley Scales
of Infant Development
Like an IQ test for
babies

22. Some Initial Results from the
Baseline Survey
Surveyed about 1,000
mothers and babies in
rural Shaanxi Province

23. Of the nearly 1,000 babies tested….
55% were anemic
But less than 20% were stunted/wasted, indicating that this
is a micronutrient problem—the babies are getting enough
calories, but not enough nutrients

24. Of the nearly 1,000 babies tested….
35% were cognitively delayed – scored outside of
the “normal” range for other babies their age around
the world

25. Of the nearly 1,000 babies tested….
57% were significantly delayed in their motor
development

26. Ultimate Consequences:
If the micronutrient deficiencies of infants / toddlers
are not corrected before baby is 30 months old,
there will be permanent and irreversible effects on:
– IQ
– Mental health
– Height
– Weight
– Health
We hope that we can help to address these issues through our
intervention project, and ultimately bring national policy attention
to this important problem!

27. What does this mean?
In harshest terms:
Between 20 to 30 percent of China’s
future population are in danger of
becoming PERMANENTLY physically
and mentally HANDICAPPED

28. What’s the challenge?
1. Rural China’s missing
babies – broken chain
– During our survey we
found that many
families with babies had
moved out of the
villages.
– Where did they go?
– What is the nutritional
status of “migrant
babies”?
– If you cannot find them,
how to help?

29. To sum up the “China story”
• Nutrition and health improvement does not
necessary go hand in hand with income and
productivity growth
• We need to identify and map complete
causal-chain between agriculture/income –
nutrition/health, that would include care,
knowledge, health system reform, gender
and governance and MORE

30. Communicating findings to
policy makers- scaling up
Less discussed topic at this forum

31. Find solutions by helping 100s and
1000s …
… and work with the government to
help 1,000,000s …
Our experience of doing
project and its evaluation
is to:

32. REAP Experiments (Projects) in
China’’’’s Poor Rural Areas (and
Migrant Communities)
>30

33. REAP has touched the lives of thousands of
China's poorest and most needy children
REAP has tested more than school kids on anemia
Passed out more than multivitamins（to address anemia）
Tested more than school kids for vision problems
Passed out more than pairs of glasses
REAP has tested more than rural kids on worms
Introduced computer assisted learning (CAL) to more than school kids
So far has given out million USD scholarships to keep kids in school

34. From Results to Policy Action
Approximately Million children eat daily egg
Over million children eat a free heather meal a day
Over kids have been prevented from drooping out of schoo
schools have introduced CAL into regular clas
In the past years, REAP submitted policy briefs to Governmen

35. Thank you!
Please visit us / contact us:
• http://www.reapchina.com
• http://reap.stanford.edu