Spotlight Webinar: Evaluability assessments in public health

Follow us @nccmt Suivez-nous @ccnmo
Funded by the Public Health Agency of Canada | Affiliated with McMaster University
Production of this presentation has been made possible through a financial contribution from the Public Health Agency of Canada. The
views expressed here do not necessarily reflect the views of the Public Health Agency of Canada..
Evaluability Assessments in Public Health
Presenters:
Laura C. Leviton, PhD
Laura Kettel Khan, PhD
November 10, 2017 1:00 – 2:30 PM ET

2
Housekeeping
Use Chat to post comments and/or
questions during the webinar
• ‘Send’ questions to All (not
privately to ‘Host’)
Connection issues
• Recommend using a wired
Internet connection (vs.
wireless),
• WebEx 24/7 help line
• 1-866-229-3239
Participant Side
Panel in WebEx
Chat

3
After Today
The PowerPoint presentation (in English and
French) and English audio recording will be made
available.
These resources will be available at:
http://www.nccmt.ca/previous-webinars

4
How many people are watching
today’s session with you?
Poll Question #1
A. Just me
B. 1-3
C. 4-5
D. 6-10
E. >10

Your profession?
Put a √ on your answer (or RSVP via email)
/
Epidemiologist Management (director,
supervisor, etc.)
Allied health
professionals (nurse,
dietician, dental
hygenist, etc.)
Librarian Physician / Dentist Other
5

Evaluability assessments in
public health
http://www.nccmt.ca/resources/search/290
Episode 35
6

NCC
Infectious
Diseases
Winnipeg, MB
NCC
Methods
and Tools
Hamilton, ON
NCC Healthy
Public Policy
Montreal, QC
NCC
Determinants
of Health
Antigonish, NS
NCC
Aboriginal
Health
Prince George, BC
NCC
Environmental
Health
Vancouver, BC
7

Registry of Methods and Tools
Online Learning
Opportunities
WorkshopsMultimedia
Public Health+
Networking and
Outreach
NCCMT Products and Services
8

9
Poll Question #2
How familiar are you with the
method or tool we are discussing
today?
A. I am not familiar with the method or tool
B. I have heard of the method or tool
C. I have used the method or tool

10
Presenters
Dr. Laura C. Leviton, PhD
Senior Advisor for Evaluation
Robert Wood Johnson Foundation
USA

■ Summative evaluation:
o Designing and using evaluation to judge merit
■ Formative evaluation:
o Designing and using evaluation to improve intervention
■ Evaluability assessment: Assessing whether
o the intervention is ready to be managed for results
o what changes are needed to do so
o whether evaluation would contribute to improved
performance
Definitions

Resources
• Wholey, Hatry & Newcomer, Handbook of Practical Program
Evaluation, Wiley 2010
• Leviton et al., Evaluability assessment to improve public
health. In Annual Review of Public Health, Volume 31:213-
234.
• Leviton, Kettel Khan & Dawkins, New Directions in Evaluation,
No. 125, January 2010-- chapter 3 has templates, procedures

Evaluations are Often Handed to Us
“Here, evaluate this.”

Some of the Interventions Just Aren’t Very Good
“Here, evaluate this.”

Why?
• Perhaps 95% of interventions are not fully developed at the
time of evaluation.
• Also, ongoing problems with measurement, design and
analysis.
• Wilson and Lipsey, 2001 review of 319 meta-analyses:
• Proportion of effect sizes associated with study features
• Variance accounted for:
Study methods = features of the intervention
Biggest sources: research design, operationalizing the dependent variable, sampling
error

Don’t Rush to Summative Evaluation

Steps in Evaluability Assessment
1. Involve intended users of evaluation information
2. Clarify the intended intervention
3. Explore intervention reality
4. Reach agreement on needed changes in activities or goals
5. Explore alternative evaluation designs
6. Agree on evaluation priorities and intended uses of information.

It’s Cyclic,
Not Linear
Involve end
users of
evaluation
Determine
scope of
project
Review
program
documents
Consult
stakeholders
Agree-
ment on
goals?
no
noCreate / revise
logic model or
theory of change
Agree-
ment on
model?
yes
Feedback to
program
manager or
policy maker
Interview staff;
“scout” the
program reality
yes
Report on assessment of:
 Plausibility
 Areas for program development
 Evaluation feasibility
 Options for further evaluation
 Critique of data (quality,
availability)
Develop
Program
Stop

Early Steps
Involve end
users of
evaluation
Determine
scope of
project
Review
program
documents
Consult
stakeholders
Agree-
ment on
goals?
no
Feedback to
program
manager or
policy maker
Stop

Middle Steps
Review
program
documents
Consult
stakeholders
Agree-
ment on
goals?
no
noCreate / revise
logic model or
theory of change
Agree-
ment on
model?
yes
Feedback to
program
manager or
policy maker
Develop
Program
Stop

Logic Models
Inputs Outputs Outcomes (Impact)
Resources
&
Staff
Activities,
Outreach to
Target Group
Products,
Target Group
Participation
Achieve
Short Term
Objectives
Achieve
Intermediate
Objectives
Achieve
Long Term
Objectives

Theory of Change
Goals Strategies Target Group If-Then
Statements
Short Term
Outcomes
Long Term
Outcomes
Statements How goal will
be
accomplished
Define group If (activity)
then outcome
Measurement Was strategy
achieved?
Were
participants in
the target
group?
Did the
activity result
in the
outcome?
Was short
term objective
achieved?
Was long term
objective
achieved?
Data Sources E.g., program
records
E.g., survey of
group

Later Steps
noCreate / revise
logic model or
theory of change
Agree-
ment on
model?
Interview staff;
“scout” the
program reality
yes
Report on assessment of:
 Plausibility
 Areas for program development
 Evaluation feasibility
 Options for further evaluation
 Critique of data (quality,
availability)

What Now?
• Intervention development
• Data collection to inform improvements
• When to do summative (outcome) studies:
• Logic model or TOC is sharpened and agreed to
• The model looks like the reality, and vice versa
• It’s plausible to achieve the outcome(s)
• Formative evaluation indicates intermediate steps are being
accomplished.

Presenters
25
Dr. Laura Kettel Khan, PhD
Senior Scientist and Advisor
Centers for Disease Control and
Prevention
USA

Systematic Screening and
Assessment: Past & Present
Examples from the Field

Overview
•4 Examples from the Field
• Community Obesity Prevention – New York City Child
Care
• Healthy eating & Active living
• Hypertension Control
• Traumatic Brain Injury
•Coming full circle with SSA
• Childhood Obesity Declines
•Benefit for Public Health
27

Obesity Prevention: NYC Child Care
Policies
Convene a panel of experts to identify and review
potential programs and policies
Assess programs and policies’ readiness for evaluation
Synthesize findings and share promising practices with the
field
Develop a network of professionals with the skills to
conduct evaluability assessments
Inform funders of programs and policies ready for
evaluation
28

Systematic Screening & Assessment
29
Inputs Steps Products
Guidance 1. CHOOSE priorities
2. SCAN environmental
interventions
3. REVIEW AND IDENTIFY
INTERVENTIONS that warrant
evaluability assessment
4. EVALUABILITY
ASSESSMENTS of priority
interventions
5. REVIEW & RATE for
promise/ readiness for eval
6. USE information
7. SYNTHESIZE what is known
Focus
Brief descriptions
Constructive feedback
Plan for rigorous
evaluation
List of
interventions
Nominations, existing
inventories, descriptions
Communicate with all
stakeholders
Expert review panel
Report of intervention an
evaluation issues
Ratings and Reports
List of Interventions
Brief DescriptionsNominations, existing
inventories, and
descriptions
Expert review panel
Network of
practitioners/research
Communicate with all
stakeholders
Expert review panel

Systematic Screening
•Criteria expert panel used for screening:
– Potential impact
– Innovativeness
– Reach
– Acceptability to stakeholders
– Feasibility of implementation
– Feasibility of adoption
– Sustainability
– Generalizability/transportability
– Staff/organization capacity for evaluation
30

Systematic Screening & Assessment
Results
Nominations
Met Inclusion
Criteria
Selected for
EA
After School/Day Care 167 61 23
Food Access 84 34 18
School District Local
Wellness Policies
146 58 6
Comprehensive Physical
Activity
39 7 2
Built Environment for
Physical Activity 22 14 4
TOTAL 458 174 53
31

Evaluability Assessment
Review of documents
• Draft logic model
2-3 day site visit
• Interviews: program description, logic model,
staffing, funding, sustainability, evaluation activities
• Observations
• TA /debriefing session
Reports and recommendations
Follow-up TA call with CDC experts
32

Readiness for Evaluation
•Review of site visit reports identified
classifications:
1. Ready for stand-alone, outcome evaluation
2. Appropriate for cluster evaluation
3. Theoretically sound but need further development
4. Technical assistance needed in specific areas
33

Big Pay Off: NYC Child Care Policy
Evaluation
 NYC child care regulations on nutrition, physical
activity & screen time
 Positive evaluation of implementation and
outcome
 Large sample of centers in low-income urban
neighborhood population
 Outcome verified with direct observation &
accelerometry
 Reference study for national recommendations for
center policy and practice
34

CDC: Healthy Eating & Active Living
 Identify innovative policies and strategies
related to public property shared & open Use
 Schools, swimming pools, parks
 Opportunity to identify promising practices
feasible for state and local implementation
 Focused on interventions for communities with
fewer resources
 Factsheets on successful policies and
strategies
35

CDC: Hypertension Control
 7 HTN Control project using SSA &/or SSA and
new Enhanced Evaluability Assessment (EEA)
 Depending on the topic area and amount of
evidence, both methodologies are used.
 EEA is an adaptation of SSA which permits
rapid assessment & reports effectiveness of
intervention
 JL Losby, et al. Arriving at Results Efficiently: Using the Enhanced
Evaluability Assessment Approach. Prev Chronic Dis 2015;12:150413.
DOI: http://dx.doi.org/10.5888/pcd12.150413
36

CDC: Return to School post-Traumatic
Brain Injury
 Best practices of Return to School after TBI
 Students who sustain TBI need post-injury
supports at school
 Families have difficulties accessing resources
 What is the best program model or components
for understanding outcomes
 Scalability and applicability for all 50 states
 Improving identification & management of TBI in
healthcare systems & schools
37

Childhood Obesity Declines
 A retrospective use of SSA….
 Is the evaluation valid? Is the BMI/outcome data
correct?
 If so, what factors could have played a role?
 Same process, different questions for site visit
 Used SSA as a confirmatory process for
outcome and an exploratory process for what
happened.
 4 sites, similar patterns of interventions and
programs yields anticipated outcome
38

It’s a Process…..
39
1. Choose priorities for the scan
2. Scan environmental programs & policies
3. Review and identify those that warrant
evaluability assessment
4. Evaluability assessment of programs & policies
5. Review and rate for promise and readiness for
evaluation
6. Use Information:
• Position for rigorous evaluation
• Feedback to innovators
• Cross-site synthesis

Why SSA adds Value…..
What is the same?
• Review documents
• Discuss with
stakeholders
• Develop logic model
• Iterate the process
• Determine what can
be evaluated
What is different?
• EA as one component of a
process of discovery
• SSA explicitly provides
feedback to innovators
• SSA provided insights on
clusters of projects
• SSA helped to identify
policies and programs
worthy of further
attention.
40

Finding the best bet faster…..
Of 458 innovations nominated in Year I:
• 174 met criteria for inclusion;
o 53 were selected for evaluability assessments;
– 27 were ready for stand alone evaluation.
• Yet all of the nominations were viewed as
important by stakeholders.
• If all of them underwent evaluation,
o there would be a 5% chance of encountering something
with even a chance of concluding success!
41

AND finding it cheaper…..
Without a systematic process,
one would need to conduct at least 20 evaluations to
discover 1 that might be successful.
The process is cost-effective for
and decision makers.
It reduces uncertainty about
investments.
42

Experts & practitioners benefit….
Expert and practice “Innovators” found the
process a learning experience.
 Learned to value interventions outside their
area of expertise
The evaluability assessment itself plays a program
development role.
 Opportunity to provide technical assistance
43

Impact on the field
Themes and issues emerged for clusters of
policies and programs
Evaluability assessments can be configured to
cast new light on
• developments in the field
• families or clusters of policies and programs
Researchers are fascinated by what practice is doing
Stimulated discussion of new research agendas
44

45
Your Comments/Questions
• Use Chat to post comments
and/or questions
• ‘Send’ questions to All (not
privately to ‘Host’)
Chat
Participant Side
Panel in WebEx

46
Poll Question #3
Could this method or tool be useful
in practice?
A. Very useful
B. Somewhat useful
C. Not at all useful
D. Don’t know

47
Your Feedback is Important
Please take a few minutes to share your thoughts
on today’s webinar.
Your comments and suggestions help to improve
the resources we offer and plan future webinars.
The short survey is available at:
https://nccmt.co1.qualtrics.com/jfe/form/SV_71yjf5
AwP51pRdP

48
Poll Question #4
What are your next steps? (Check all
that apply)
A. Access the method/tool referenced in the
presentation
B. Read the NCCMT summary about the
method/tool described today
C. Consider using the method/tool in
practice
D. Tell a colleague about the method/tool

Share your story!
• Are you using EIDM in your practice? We want
to hear about it!
• Email us: nccmt@mcmaster.ca
• Need support for EIDM? Contact us for help!
• Email us: nccmt@mcmaster.ca
• We typically respond within 24 business hours
49

50
Join us for our next webinar
Spotlight on Methods and Tools: EENet:
Mental health and addictions knowledge
translation resources
Date: Thursday January 25, 2018
Time: 1:00 – 2:30pm ET
Do you work in mental health and addiction? Interested in a
knowledge exchange network that helps to create and share
evidence to build a better mental health and substance use system?
Join us to find out!
Register at: https://health-evidence.webex.com/health-
evidence/onstage/g.php?MTID=ef97dccfad971be665e8e7550172d7
ada

Webinar Series from NCCMT
www.nccmt.ca/webinar-series
• Spotlight on Methods and Tools
• Topic-Specific Methods and Tools
• Online Journal Club
• Peer-to-peer Webinars
51

For more information about the
National Collaborating Centre
for Methods and Tools:
NCCMT website www.nccmt.ca
Contact: nccmt@mcmaster.ca

Systematic Selection
•7 After School/3 Daycare Programs
• 5 programs: PA time, nutritious snacks
• 4 programs: PA time, nutrition education
• 1 policy: PA, nutrition, TV screen time
•10 Food Access Programs
• 5 farmers’ markets
• 3 supermarket or corner store programs
• 2 restaurant programs
•6 School District Local Wellness Policies
• All selected addressed PA and nutrition
53

Results
•Expert panel determined:
– 14 ready for stand-alone, outcome evaluation
– 2 best suited for cluster evaluation
– 3 theoretically sound but need further development
– 6 need TA in specific areas
54

Spotlight Webinar: Evaluability assessments in public health

Recommandé

Recommandé

Contenu connexe

Tendances

Tendances (20)

Similaire à Spotlight Webinar: Evaluability assessments in public health

Similaire à Spotlight Webinar: Evaluability assessments in public health (20)

Plus de The National Collaborating Centre for Methods and Tools

Plus de The National Collaborating Centre for Methods and Tools (20)

Dernier

Dernier (20)

Spotlight Webinar: Evaluability assessments in public health

Notes de l'éditeur