Community-based Evaluation Methods and Practice

1. Community based evaluationCommunity based evaluation
methods & practicemethods & practice
Program Science in PracticeProgram Science in Practice
CAHR 2015 Ancillary EventCAHR 2015 Ancillary Event
April 30, 2015April 30, 2015
Javier MignoneJavier Mignone
Javier.mignone@umanitoba.caJavier.mignone@umanitoba.ca

2. Outline of the presentationOutline of the presentation
- Clarifying the terminologyClarifying the terminology
- Issues that must be addressedIssues that must be addressed
- Program evaluation: a seven stepProgram evaluation: a seven step
processprocess
- Examples of collaborationExamples of collaboration
J Mignone - University of Manitoba -
2015 2

3. What is program evaluation?What is program evaluation?
““Program evaluation is theProgram evaluation is the systematicsystematic
collection of informationcollection of information about theabout the
activities, characteristics, and outcomes ofactivities, characteristics, and outcomes of
programs to make judgments about theprograms to make judgments about the
program, improve program effectiveness,program, improve program effectiveness,
and/or inform decisions about futureand/or inform decisions about future
programming.” (Patton, 1997)programming.” (Patton, 1997)
J Mignone - University of Manitoba -
2015

4. Why?Why?
“…“…..for use by specific peoplefor use by specific people to reduceto reduce
uncertainties, improve effectiveness, anduncertainties, improve effectiveness, and
make decisions with regard to what thosemake decisions with regard to what those
programs are doing and affecting.” Itprograms are doing and affecting.” It
means, “gathering data that are meant tomeans, “gathering data that are meant to
be, and actually are, used for programbe, and actually are, used for program
improvement and decision making.”improvement and decision making.”
(Patton, 1997)(Patton, 1997)
J Mignone - University of Manitoba -
2015

5. Difference between evaluation andDifference between evaluation and
researchresearch
Basic scientific research is undertaken toBasic scientific research is undertaken to
discover new knowledge, test theories,discover new knowledge, test theories,
establish truth, and generalize across timeestablish truth, and generalize across time
and space.and space.
Program evaluation is undertaken toProgram evaluation is undertaken to
inform decisions, clarify options, reduceinform decisions, clarify options, reduce
uncertainties, and provide informationuncertainties, and provide information
about programs and policies withinabout programs and policies within
contextual boundaries of time, place,contextual boundaries of time, place,
values, and politics.values, and politics.
J Mignone - University of Manitoba -
2015

6. Difference between evaluation andDifference between evaluation and
research (cont’d)research (cont’d)
The difference between research andThe difference between research and
evaluation has been described byevaluation has been described by
Cronbach and Suppes as: Research isCronbach and Suppes as: Research is
aimed at truth; Evaluation is aimed ataimed at truth; Evaluation is aimed at
action.action.
J Mignone - University of Manitoba -
2015

7. Core questionsCore questions
Who uses the systematically producedWho uses the systematically produced
information?information?
How is the systematically producedHow is the systematically produced
information actually used?information actually used?
J Mignone - University of Manitoba -
2015

8. Community-based evaluationCommunity-based evaluation
What community?What community?
◦ PlacePlace
◦ IdentityIdentity
◦ InterestInterest
What does community-based mean?What does community-based mean?
◦ It depends on the purpose and the processIt depends on the purpose and the process
J Mignone - University of Manitoba -
2015 8

9. Community-based evaluation (cont’d)Community-based evaluation (cont’d)
Purpose of evaluationPurpose of evaluation
Process of evaluationProcess of evaluation
J Mignone - University of Manitoba -
2015 9

10. Some specific issues to addressSome specific issues to address
Expectations about evaluationExpectations about evaluation
Plural or conflicting agendasPlural or conflicting agendas
TrustTrust
Funding requirements and agenciesFunding requirements and agencies
Evaluation plan developmentEvaluation plan development
Cultural interpretations/knowledgeCultural interpretations/knowledge
J Mignone - University of Manitoba -
2015 10

11. Issues to address (cont’d)Issues to address (cont’d)
Appropriate methodologyAppropriate methodology
TimelinesTimelines
Ownership of data (e.g., OCAP)Ownership of data (e.g., OCAP)
PoliticsPolitics
ParticipationParticipation
EthicsEthics
Use/dissemination of findingsUse/dissemination of findings
J Mignone - University of Manitoba -
2015 11

12. Program evaluation: A seven stepProgram evaluation: A seven step
processprocess
1.1. Identify the primary intended users ofIdentify the primary intended users of
the evaluationthe evaluation
2.2. Identify and focus the relevantIdentify and focus the relevant
evaluation questionsevaluation questions
3.3. Make design methods and measurementMake design methods and measurement
decisionsdecisions
4.4. Collect dataCollect data
5.5. Organize data for stakeholder analysisOrganize data for stakeholder analysis
6.6. Involve users in interpretation ofInvolve users in interpretation of
findingsfindings
7.7. Facilitate intended use by intended usersFacilitate intended use by intended usersJ Mignone - University of Manitoba -
2015 12

13. Examples of collaborationExamples of collaboration
Summer Institute on Program EvaluationSummer Institute on Program Evaluation
(University of Manitoba)(University of Manitoba)
Community Health Information andCommunity Health Information and
Research Partnerships (CHIRP) (UniversityResearch Partnerships (CHIRP) (University
of Manitoba students)of Manitoba students)
J Mignone - University of Manitoba -
2015 13

14. ??????
J Mignone - University of Manitoba -
2015 14

15. ReferencesReferences
 Chomsky N (2003)Chomsky N (2003) Necessary illusions: Thought control in democratic societiesNecessary illusions: Thought control in democratic societies. Toronto:. Toronto:
House of Anansi Press.House of Anansi Press.
 First Nations Centre at NAHO (2007) OCAP: Ownership, control, access andFirst Nations Centre at NAHO (2007) OCAP: Ownership, control, access and
possession. NAHO: Ottawapossession. NAHO: Ottawa
 Horne T (1995)Horne T (1995) Making a difference: Program evaluation for health promotionMaking a difference: Program evaluation for health promotion..
WellQuest Consulting: Edmonton.WellQuest Consulting: Edmonton.
 Kaufman Kantor G & Kendall-Tackett K (2000)Kaufman Kantor G & Kendall-Tackett K (2000) A guide to family intervention andA guide to family intervention and
prevention program evaluationprevention program evaluation. University of New Hampshire. Electronic version.. University of New Hampshire. Electronic version.
 King JA, Stevahn L, Ghere G & Minnema J (2001) Toward a taxonomiy of essentialKing JA, Stevahn L, Ghere G & Minnema J (2001) Toward a taxonomiy of essential
program evaluator competencies.program evaluator competencies. American Journal of Evaluation, 22American Journal of Evaluation, 22(2), 229-247.(2), 229-247.
 Morra Imas LC & Rist RC (2009) The road to results: Designing and conductingMorra Imas LC & Rist RC (2009) The road to results: Designing and conducting
effective development evaluations. The World Bank: Washington DC.effective development evaluations. The World Bank: Washington DC.
 Patton MQ (1997)Patton MQ (1997) Utilization-focused evaluation: The new century text.Utilization-focused evaluation: The new century text. 33rdrd
edition,edition,
Thousand Oaks: Sage.Thousand Oaks: Sage.
 Patton MQ (2008)Patton MQ (2008) Utilization-focused evaluationUtilization-focused evaluation, 4th Edition, Thousand Oaks: Sage., 4th Edition, Thousand Oaks: Sage.
 Patton MQ (2012) Essentials of utilization-focused evaluation. Thousand Oaks: Sage.Patton MQ (2012) Essentials of utilization-focused evaluation. Thousand Oaks: Sage.
 The Provincial Centre for Excellence for Child and Youth Mental Health at CHEOThe Provincial Centre for Excellence for Child and Youth Mental Health at CHEO
(2007)(2007) Doing more with program evaluation: A toolkit for conducting program evaluationDoing more with program evaluation: A toolkit for conducting program evaluation..
Ottawa, OntarioOttawa, Ontario www.onthepoint.cawww.onthepoint.ca
 Shadish WR, Cook TD, Campbell DT (2002)Shadish WR, Cook TD, Campbell DT (2002) Experimental and quasi-experimentalExperimental and quasi-experimental
designs for generalized causal inferencedesigns for generalized causal inference. Boston: Houghton Mifflin.. Boston: Houghton Mifflin.
 Suchman EA (1967) Evaluative research: Principles and practice in public service andSuchman EA (1967) Evaluative research: Principles and practice in public service and
social action programs. New York: Russell Sage.social action programs. New York: Russell Sage.
J Mignone - University of Manitoba -
2015 15