1. A M Y A . G E R M U T H , P H . D .
E v a l u a t i o n 2 0 1 0
A m e r i c a n E v a l u a t i o n A s s o c i a t i o n
S a n A n t o n i o , T X , N o v e m b e r 1 3 , 2 0 1 0
1
Evaluation Post Mortems
Dissecting what went right or wrong and learning from it!
2. What’s a Post Mortem?
2
A formal analysis of the success and failures of the
project to-date.
Findings are added to the general knowledge base for
future use.
3. Post Mortems in Practice
3
Conducting post-mortems is a common practice in
business.
Appears to be less common in evaluation.
Very little written about evaluations that don’t work,
including why they don’t work.
- Fear of reporting failures?
4. Post Mortems in Evaluation
4
What if we made post-mortems part of the
evaluation process?
- What would it look like?
- Where would it fit?
- Would it be beneficial?
- If so, why and how?
5. Post Mortems: The Process
5
Two-step process:
1. Provide the persons involved a list of questions
about the evaluation that they think about and
respond to on their own.
2. Bring all persons together in order to share what
they thought and discuss lessons learned.
6. Who to Involve
6
May depend upon where you are in the evaluation
cycle.
Ultimately want all stakeholders to weigh in.
Remember – about evaluation process – NOT
evaluation findings.
7. General Questions to Consider
7
Are you/we proud of our finished deliverables
(project work products)?
- If yes, what's good about them?
- If no, what's wrong with them?
What was the single most frustrating part of the
evaluation?
How would you/we do things differently next time
to avoid this frustration?
8. General Questions Cont.
8
What was the most gratifying or professionally
satisfying part of the evaluation?
Which methods or processes worked particularly
well?
Which methods or processes were difficult or
frustrating to use?
If you could wave a magic wand and change anything
about the evaluation, what would you change?
9. Evaluation-focused Questions
9
Did our stakeholders, senior managers,
customers, and sponsor(s) participate
effectively? If not, how could we improve their
participation?
How accurate were our original estimates of the
time, cost, and other resources required of the
evaluation? What did we over- or under-
estimate?
Knowing what we know now, would we have
chosen the same type of evaluation design as the
one we used? If not, what could have pointed us
to a design that would have been better suited for
such a project?
10. Evaluation-focused Questions Cont.
10
Were our evaluation questions the best ones, or
were there other questions we did not fully
explore with stakeholders, through our
evaluation, etc. that needed addressing?
How would we rate the quality of the data we
gathered and what could we have done to have
collected more convincing data for formative and
summative purposes?
Did our presentation of results highlight the data
so that stakeholders could make their own
interpretations or understand the ones we made?
What did we do to help stakeholders understand
and use the evaluation findings?
11. Further Notes on Post Mortems
11
Separate from just following the program evaluation
standards post-mortems have a very formal outcome, the
identification of lessons learned.
Separate from a meta-evaluation as meta-evaluations are
themselves evaluations and not designed to identify
lessons learned, as much as to identify the value in the
evaluation that was conducted.
Also, for meta-evaluation to be viewed as unbiased, they
do need to be conducted by someone outside of the
original evaluation, whereas post-mortems are
specifically designed to engage the original evaluators.
