1. Building the evidential quality
of evaluations
ALNAP March 5, 2013
Tony Redmond

2. Humanitarian vs Development

3. Hierarchy of Evidence

4. Most evidence in the
humanitarian sphere is in the
weakest categories or outside
the pyramid altogether

5. Expert Opinion
• Consensus Statements Regarding the
Multidisciplinary Care of Limb Amputation Patients
in Disasters or Humanitarian Emergencies: Report of
the 2011 Humanitarian Action Summit Surgical
Working Group on Amputations Following Disasters
or Conflict
• Delphi Studiens

8. Gathering the Evidence

9. bias
• Incentive
• Observer
• Systematic (non random) error
– Confidence intervals

10. RCT
Sibbald B, BMJ 1998;316:201
• the most rigorous way of determining
whether a cause-effect relation exists
between intervention and outcome and
• for assessing the cost effectiveness of an
intervention.

11. Dangerous Data
• Wrong inference
• Randomisation
• Open and transparent

12. HERR
• Medical teams are 200 times as cost
effective as UKISAR
• That is they cost less, stay longer, treat
more people and save more lives

13. sampling

14. Convenience sampling
• Using those who are willing to
volunteer, or cases which are presented to
you as a sample.

15. Judgement sampling
• Based on deliberate choice and excludes
any random process.
• Normal application is for small samples
from a population that is well understood
and there is a clear method for picking the
sample.
• Is used to provide illustrative examples or
case studies.

16. Simple random sampling
• Ensures every member of the population
has an equal chance of selection.
• Produces defensible estimates of the
population and sampling error.
• Simple sample design and interpretation.

17. Cluster sampling
Units in the population can often be found in geographical groups or clusters. A
random sample of clusters is taken, then all units within those clusters are
examined.
• Quicker, easier and cheaper than other forms of random sampling.
• Does not require complete population information.
• Useful for face-to-face interviews.
• Works best when each cluster can be regarded as a microcosm of the
population.
• Larger sampling error than other forms of random sampling.
• If clusters are not small it can become expensive.
• A larger sample size may be
needed to compensate for
greater sampling error.

18. Why is evidence ignored?
• Evidence influencing policy
• Policy influencing evidence
• We've always done it this way…
• ……and it’s done no harm (regardless of
the evidence)
• Maybe the evidence was always there….

19. Epidemiology/statistics
• Observation
• Pattern Recognition
• Cause and effect or coincidence
• Change of practice

20. CONFIDENCE
INTERVALS (CI)
MEAN VALUE
You have a 95%
confidence that
the TRUE VALUE
should fall
between the CI if
the study is
repeated multiple
times in a similar
population
95% CONFIDENCE INTERVAL

21. assessments
• Early warning
• Risk analysis
• Trending
• Needs assessment

22. Evidence based medicine
• Clinical trials in the field – in an
emergency?
• Multiple disciplines

24. Evaluation
• What was done?
• What was the reason for doing it?
• What did they find?
• SO WHAT?

25. Benefit cost

26. What you have to evaluate
adapted from DFID WASH Evaluation 2011
• Internal validity – was what they did good
enough?
• External validity – relevance to you?
• Can you implement it ?
• Sustainability?
• Wider impact?
• COST

27. Is good enough, good enough?

28. Humanitarian
• Historically draws on concept of
compassion, individuality and reform
(transformation)
• It is evidence and evaluation that will help
us reform and lead us to prevention