Practical Research 1 Lesson 9 Scope and delimitation.pptx
Evaluation and philanthropy: lessons from the third sector / Lisa Jordan
1. Evaluation and Philanthropy
Lessons from the Third
Sector
Lisa Jordan
Executive Director
Bernard van Leer Foundation
2. Mission Statement
Our mission is to improve opportunities for children up to
age 8 who are growing up in socially and economically
difficult circumstances. We see this both as a valuable end
in itself and as a long-term means to promoting more
cohesive, considerate and creative societies with equal
opportunities and rights for all.
3. Strategic Goals 2010-2015
v
Taking quality early learning to scale
v
Improving young children's physical environments
v
Reducing violence in young children's lives
4. AGEND
A
v
The Bernard van Leer Foundation
v
Our Impact Assessment
v
Common myths about evaluation
v
Lessons from the third sector
5. Philanthropic Tools for Greater
1.
Impact Development
Knowledge
o
research, documentation, evaluation
2. Service product development and delivery
o
investments, grants
3. Capacity Enhancement and Skills Development
o
training, technical assistance
4. Behaviour Change Programmes
o
campaigns, awareness
5. Policy Development & Implementation
Click to edit Master subtitle style
o
community organising, legal empowerment, lobby
6. Enabling Systems & Infrastructure
o
networks, markets
6. ISRAEL
Universal access to quality pre- Reduced incidence of Reduced rates of malnutrition
school among 3 to 6 year old depression, anxiety and among young Bedouin children
children. aggression among young Jewish growing up in unhealthy physical
and Arab children exposed to environments in the Negev.
political violence.
The National Ministry of Education, Parents and children experience Improved knowledge about dietary
local municipalities and members of reduced levels of stress from and infrastructure causes of child
the business sector have increased exposure to political violence illness and malnutrition
their financial investments in
preschools for children
The ratio of pre-school to supervisorThe Israeli government has an Increased access to clean water,
has decreased appropriate public system for improved waste management and
Click to edit Master subtitle style
prevention and treatment of electricity
depression, anxiety and aggression
among young children
supervisors are better equipped to Improved access to transport
improve pre-school quality among Bedouin communities,
especially women and children
The quality of teaching and learning
in pre-schools has improved
The curriculum and reflects the
7. Pilot
health
Reduced rates of malnutrition among young Bedouin children education
growing up in unhealthy physical environments in the Negev programm Lobby
es for
nurse
Improved child health (esp. s
reduction in digestive tract Improved access to
illness) mother-child
wellbeing clinics
Better access to Improved waste Healthy dietary
clean water management practices to combat
Regional vitamin deficiencies
and Improved housing Electricity
municipal conditions (hygienic,
ability to regulate Increased awareness among
advocacy
temperature) parents about dietary and
(water,
Government provision infrastructure causes of child
sanitation, Bedouin
of infrastructure in Increased malnutrition
electricity) families
Bedouin towns and incomes
invest in own
villages among
infrastructure
Bedouin
Increased mobility (e.g.
families
Government transport) among
planners Legal recognition Bedouin communities
Joint
municipal
Click to edit Master subtitle style
and of villages More Bedouin (esp. women)
Bedouins women join the
and
agree on a workforce
communit
y planning child-friendly
Research
plan
and
Bedouin communities are well organized, have more
communic
positive public image and are supported by professional
ations
planners
about PE
and child Organizin
health link g for
transport
Private
sector
transport
options
8. What are the most cost-
effective interventions to
improve Bedouin
children’s health and
nutrition?
Health and nutrition
outcomes for children
Individual evaluations
combined with meta-
analysis
How are our strategies working to
influence investment and service
delivery in the Negev region for
Review and Bedouin children?
recommendations by Budgets
advisory team with
knowledge of broader Evaluation of cluster of Effective coalitions (Arab,
political context grants from advocacy Jewish, government,
perspective private sector, parents)
Click to edit Master subtitle style Access to basic services
How are our efforts to empower Health and nutrition
Bedouin communities in the outcomes
planning process working to
influence investment and service
delivery?
Case study of selected
municipalities
9. Main goals of the philanthropic activity
The Foundation cannot solve directly any social
Test and validate problems. Its mission aims therefore at testing
innovative (especially with projects directly managed) innovative
policies solutions to social problems and at disseminating
successful solutions (“what works”).
Solutions to certain problems are well known and
Reward best organisations implementing related initiatives are
practices numerous: in such cases the Foundation selects and
funds the best projects through specific calls for
proposals.
Aimed at supporting (with institutional grants)
Support worthy deserving nonprofit organisations (operating in the
institutions sectors of Arts & Culture, Environment, Scientific
Research, Social Services) prevailingly based in
Lombardy (Cariplo Foundation’s traditional intervention
territory)
25/05/2011 Strategic Unit for 9
11. Main purposes of evaluation
Test and validate
innovative
policies
Reward best
practices
Support worthy
institutions
Critical Knowledge
Accountability analysis/ sharing
Learning
25/05/2011 Strategic Unit for 11
12. What are the most cost-
effective interventions to
improve Bedouin
children’s health and
nutrition?
Health and nutrition
outcomes for children
Individual evaluations
combined with meta-
analysis
How are our strategies working to
influence investment and service
delivery in the Negev region for
Review and Bedouin children?
recommendations by Budgets
advisory team with
knowledge of broader Evaluation of cluster of Effective coalitions (Arab,
political context grants from advocacy Jewish, government,
perspective private sector, parents)
Click to edit Master subtitle style Access to basic services
How are our efforts to empower Health and nutrition
Bedouin communities in the outcomes
planning process working to
influence investment and service
delivery?
Case study of selected
municipalities
13. Tools: FIT FOR PURPOSE
http://trasi.foundationcenter.org
14. AGEND
A
v
The Bernard van Leer Foundation
v
Our Impact Assessment
v
Common myths about evaluation
v
Lessons from the third sector
18. Which problem do you think was most
commonly reported?
1. Defining & agreeing on the purpose of
the evaluation
2. Finding capable evaluators
3. Making decisions based on evaluation
results
22. Myth 1: Evaluation is for donors
WHAT ARE YOU EVALUATI NG THE
DOING UP IMPAC OF OUR C
T LEAN
THERE? WATER PROGRAMME.
DID YOU
NOTICE THE SORRY, THAT’S NOT
FLOODING IT I N MY
CAUSED? QUES ONNAIRE.
TI
24. Evaluation & Accountability
Evaluatio at thehe o ac o
n art f c untab : A fully ac o
ility c untable
philanthro wo dob tte at g undingitswo in rig ro prac e
py uld e r ro rk o us tic
kno dg ; e
wle e xplainingno o fundings g sb thepre is sb hind
t nly trate ie ut me e
the ; ac wle ingins
m kno dg titutio value andb e ; and adm
nal s ias s ittingwhat
wedo ’t kno A fully ac o
n w. c untab philanthro wo o r m rethan
le py uld ffe o
surfac data dis o c dfro there
e c nne te m ality o iss sando anizatio that
f ue rg ns
foundatio s
ns uppo andfro thepurpo e andvalue that fram the
rt m ss s e ir
de is ns Weo it too c ns nc stob thisc ar-m dab ut
c io . we ur o titue ie e le inde o
whe wearehe d andwhy. Witho pro ingc
re ade ut b larity ando n inq ,
pe uiry
foundatio atte ptsat ac o
n m c untab will b mre s b lic
ility e e ly ym o .
25. Myth 2: Scientific methods are too
rigid to understand my reality
OUR PROGRAMSARE TOO
YOU MEAN 9000 YEARSOF
C OMPLI CATED TO
S IENTIFI CPROGRES ISNOT
C S
EVALUATE!
SUFFIC IENT TO MAKE SENS OF
E
YOUR PROGRAMS ?
26. Myth 2: Scientific methods are too
rigid to understand my reality
WE DEC IDE IT ISA
S C S
UC ES FUL PROGRAM
IF THREE CHERRIES
SHOW UP
27. Myth 2: Scientific methods are too
rigid to understand my reality
Community-Driven
Reconstruction: led by the
International Rescue Committee
with support from Fearon, Macartan
and Weinstein
The challenge: Attribution of
improvements in “community
cohesion” and “democratic practice”
The solution: Randomization at
village level, standard surveys +
tools from behavioral economics
28. Community
cohesion
Social
inclusion
Democratic
practice and
values
Material
wellbeing
29. But, let’s use RIGOR WITHIN
REASON
I ’M HERE TO EVALUATE HOW
TRANS PORTATI ON I MPACTS
C LDREN’SEDUCATI ON.
HI I HOPE THE BUS
ARRIVESTODAY. I
I PROPOS TO RANDOMI ZE YOUR
E DON’T WANT TO MI SS
FAMI LY AND NAME YOUR KI DSWI TH
S HOOL AGAIN!
C
ACRONYMS.
30. Myth 3: Evaluation is too expensive –
we should just spend the money on the
kids
I AM NOT GOI NG TO WASTE ISUPPOS THEY KNOW
E
MONEY. THE CHI LDREN NEED WHAT TO DO, THEY ARE
PROFES I ONALS
S …
ALL THE RES OURCESTHEY CAN
GET!
ISUPPOS THEY
E
KNOW WHAT TO
DO, THEY ARE
ADULTS …
31. Myth 3: Evaluation is too expensive –
we should just spend the money on the
kids
Review of 140 community-
based child protection
DIMINISHED
evaluations RETURNS?
•
rarely measured kids’
outcomes
WASTED MONEY?
•
84% had only ex-post
measures HARM DONE?
•
only 3% had comparison
groups and pre- and post- INTEGRITY?
measures
32. AGEND
A
v
The Bernard van Leer Foundation
v
Our Impact Assessment
v
Common myths about evaluation
v
Lessons from the third sector
33. The Barry Knight Mantra
www.centris.org.uk
1. Owned – People who use the evaluation feel that the system is
theirs, rather than being imposed on them. The system is integrated with
their day-today work.
2. Useful – Results are relevant, and can be applied in day-to-day
work to promote learning.
3. Robust – Results are valid and reliable. The system needs to be
sensitive to the complexity of what is likely to be involved in shifting the
deep-seated issues being worked on.
4. Simple – The system works smoothly and easily without the need
to have high technical knowledge. Note, however, that simple does not
mean simplistic. Things should be as simple as possible but no simpler.
34. The Pareto Principle
•
At least 80 percent of the assessment should be
driven by you and your learning needs
•
Mastering 20 percent of the jargon will get you
80 percent of the results you need
•
The first 20 percent of the cost/ time/ energy
spent on impact assessment yields 80 percent of
the learning
36. Theoretical calculation of efficiency
savings for UNICEF Child Protection
UNICEF child Investment
alone Efficiency Efficiency
Cost of 42
protection potential lost RCTs (one per savings over savings over
budget for 5 as a result of a country with 5 years 10 years
years weak evidence low HDI)
1% - 4 million + 13 million
10% + 149 million + 319 million
1.7 billion 25% 21 million + 404 million + 829 million
50% + 829 million + 1.68 billion
75% + 1.25 billion + 2.53 billion
Note: efficiency savings would only be felt after evaluation results began to feed programming.