Interested in learning how to evaluate your policy influence?
Do you promote the uptake and dissemination of population health interventions? Are you interested in exploring public health–related case studies of policy influence? The Guide to Policy-Influence Evaluation can help!
This guide was developed by the Public Health Agency of Canada’s Innovation Strategy and produced by Cathexis Consulting.
How can the Guide to Policy-Influence Evaluation help you?
The Guide to Policy-Influence Evaluation was developed to help organizations use policy influence to improve the uptake and evaluation of evidence-based population health interventions. This process is divided into the four steps of evaluation planning. Each step includes two or more resources to support it. The resources are then summarized and important highlights are presented as they related to each step.
This webinar includes an overview of the Guide by its developers, followed by a presentation from a community based organization who evaluated the impact on policies within their work to promote healthier weights.
The Guide to Policy-Influence Evaluation includes three public health–related case studies:
•Healthy weights among Aboriginal children and youth
•Anti-bullying for primary schools
•Food security and healthy weights
To see the summary statement of this method developed by NCCMT, click here: http://www.nccmt.ca/resources/search/241
The National Collaborating Centre for Methods and Tools is funded by the Public Health Agency of Canada and affiliated with McMaster University. The views expressed herein do not necessarily represent the views of the Public Health Agency of Canada.
NCCMT is one of six National Collaborating Centres (NCCs) for Public Health. The Centres promote and improve the use of scientific research and other knowledge to strengthen public health practices and policies in Canada.
NCCMT Spotlight Webinar: Guide to Policy-Influence Evaluation
1. Follow us @nccmt Suivez-nous @ccnmo
Funded by the Public Health Agency of Canada | Affiliated with McMaster University
Production of this presentation has been made possible through a financial contribution from the Public Health Agency of Canada. The
views expressed here do not necessarily reflect the views of the Public Health Agency of Canada..
Guide to Policy-Influence Evaluation
Presenters:
Marla Steinberg, PhD CE
Anima Anand, PhD
Michele Hopkins, RSW, MSW
Shannon Bradley Dexter, MSc
January 25, 2017 1:00 – 2:30 PM ET
2. Follow us @nccmt Suivez-nous @ccnmo
2
Housekeeping
Use Chat to post comments and/or
questions during the webinar
• ‘Send’ questions to All (not
privately to ‘Host’)
Connection issues
• Recommend using a wired
Internet connection (vs.
wireless),
• WebEx 24/7 help line
• 1-866-229-3239
Participant Side
Panel in WebEx
Chat
3. Follow us @nccmt Suivez-nous @ccnmo
3
After Today
The PowerPoint presentation (in English and French)
and English audio recording will be made available.
These resources are available at:
PowerPoint: http://www.slideshare.net/NCCMT/
Audio Recording:
https://www.youtube.com/user/nccmt/videos
4. Follow us @nccmt Suivez-nous @ccnmo
4
How many people are watching
today’s session with you?
Poll Question #1
A. Just me
B. 1-3
C. 4-5
D. 6-10
E. >10
5. Follow us @nccmt Suivez-nous @ccnmo
Your profession?
Put a √ on your answer (or RSVP via email)
/
Epidemiologist Management (director,
supervisor, etc.)
Allied health
professionals (nurse,
dietician, dental
hygenist, etc.)
Librarian Physician / Dentist Other
5
6. Follow us @nccmt Suivez-nous @ccnmo
Guide to Policy-Influence
Evaluation
http://www.nccmt.ca/resources/search/241
Episode 30
6
8. Registry of Methods and Tools
Online Learning
Opportunities
WorkshopsMultimedia
Public Health+
Networking and
Outreach
NCCMT Products and Services
8
9. Follow us @nccmt Suivez-nous @ccnmo
9
Poll Question #2
How familiar are you with the
method or tool we are discussing
today?
A. I am not familiar with the method or tool
B. I have heard of the method or tool
C. I have used the method or tool
10. Follow us @nccmt Suivez-nous @ccnmo
10
Presenter
Shannon Bradley Dexter, MSc
Senior Policy Analyst,
Innovation Strategy, Public
Health Agency of Canada
11. Follow us @nccmt Suivez-nous @ccnmo
11
Presenter
Marla Steinberg, PhD CE
Evaluation Consultant
12. Follow us @nccmt Suivez-nous @ccnmo
12
Presenter
Anima Anand, PhD
Project Lead, Healthy
Weights for Children, The
Bridge Youth & Family
Services
13. Follow us @nccmt Suivez-nous @ccnmo
13
Presenter
Michele Hopkins, RSW, MSW
Project Coordinator, Healthy
Weights for Children, The
Bridge Youth & Family
Services
14. Evaluating Policy-Influence
NCCMT Webinar
January 25, 2017
Marla Steinberg, Ph.D. CE
Evaluation Consultant
Anima Anand, Ph.D. and Michele Hopkins M.S.W.
Healthy Weights for Children Project
Shannon Bradley Dexter
Innovation Strategy
Public Health Agency of Canada
15. Welcome
By the end of this webinar, you should be able to:
1. List the complexities of evaluating policy-influence.
2. Assess how the Guide to Policy-Influence Evaluation can support your work.
16. Establishing our Learning Community
Poll: What is your interest in policy-influence?
A. I am a practitioner who does policy-influence work
B. I am a public health manager or decision-maker with a mandate
for policy-influence work
C. I am a funder who funds policy-influence work
D. I am an evaluator who evaluates policy-influence work
E. I am a student interested in policy-influence work
F. Other
G. I don’t really know what policy-influence work is……………..
Poll Question #3
17. Have you ever…..
• Wondered how you get senior decision
makers and politicians to take an
interest in your program or issue?
• Wondered how best to support the
scale-up of an intervention?
• Wondered how to evaluate your
influence on policy?
• Wondered what should be considered
a realistic achievement for policy-
influence work?
18. The Guide to Policy-Influence Evaluation may
be just what you need….
19. Needs, definitions, and the complexity of policy-influence work and evaluation
Setting the context for the guide
20. Origins of the Guide: The Public Health
Agency of Canada’s Innovation Strategy (IS)
• Funds population health intervention research in
diverse communities across Canada to build evidence
about "what works" "for who" and "in what context"
to reduce health inequities.
• Promotes and shares intervention results to inform
future policy, program design, and other actions that
will improve population health.
• Funding provided in three phases.
Current focus areas: Mental Health
Promotion and Achieving Healthier Weights
21. IS Evaluation Requirements
• IS projects are required to undertake a
comprehensive evaluation including assessing the
uptake of evidence or knowledge generated
through the project.
• Uptake is often achieved through the “influence
on policy.”
THE ISSUE: Projects needed guidance, tools, and resources to
effectively report on how IS projects were influencing policy. They
wanted to be able to tell their policy-influence stories!
24. What is policy?
• Formal and wordy definition:
Policy is a means of governing action with the aim of attenuating or promoting
particular phenomena occurring in the population.
Policies can outline rules, provide principles that guide action, set roles and
responsibilities, reflect values and principles, as well as state intentions.
Policies can be enacted by all levels of government (federal, provincial, regional
and municipal), community organizations, businesses, and schools.
Polices can guide programs, practice or education.
• Adapted from the National Collaborating Centre for Healthy Public Policy
(2010) Accessed at: http://www.ncchpp.ca/docs/MethodPP_EN.pdf
• Endorsed by 95% of IS project respondents in the policy evaluation needs
assessment conducted in the fall of 2013.
25. What is policy-influence work?
Policy-influence work supports the
uptake or spread of evidence-based
interventions.
26. An example
“With partner organizations, we developed a food
security questionnaire that went to all party leaders
during the recent provincial election to see where
they stand on food security issues. The next step will
be holding them to some promises. We have also
been working with the municipality to develop a food
charter and food strategy. Prior to this we prepared
a fact sheet on food security as a municipal issue,
which was distributed to supporters and encouraged
people to write in support of including food security
in the Regional Plan”
Policy maker education
Policy development
Public awareness
campaign
Advocacy
30. “Policy-influence is a highly complex
process shaped by a multitude of
interacting forces and actors.
Outright success, in terms of
achieving specific, hoped-for changes
in policy, is rare, and the work that
does influence policy is often unique
and rarely repeated or replicated,
with many incentives working against
the sharing of ‘good practice’”
(Jones, 2011, p. 1)
31. A lot of policy-influence is
about being there when the
window opens and being
nimble to jump on board. A
lot of policy work is stand
and wait and then jump
when you need to act”
Bonnie Leadbeater – WITS
(Mental Health IS project)
33. All roads lead
to Rome…..
There are a
wide variety
of activities
that can
influence
policy
34. There are a variety of
policy impacts:
(1) Conceptual – changing the
thinking of key
stakeholders
(2) Instrumental – changing
actions
Carol Weiss
35. Jones (2011) offers further elaborations on the
range of policy impacts:
1. Framing debates and getting issues on to the political
agenda; drawing attention to new issues and affecting
the awareness, attitudes or perceptions of key
stakeholders
2. Encouraging discursive commitments; affecting
language and rhetoric to promote the recognition of
specific groups or endorsements of policy
recommendations
3. Securing procedural change; changes in the process
whereby policy decisions are made, such as opening
new spaces for policy dialogue
4. Affecting policy content
5. Influencing behaviour change in key actors: policy
change requires changes in behavior and
implementation at various levels in order to be
meaningful and sustainable
41. Recommends a
limited number
of useful (and
free) resources
that are
organized
around a four-
step evaluation
planning
process
42. Four steps are
aligned with other
evaluation planning
frameworks
CDC Framework for
Evaluation in Public
Health
43. Includes three case studies:
WITS - anti-bullying program
for primary schools from the
mental health stream
Healthy Weights Connections
-
System change intervention
to improve public health
services for aboriginal
children and families
Our Food - Creating a food
strategy for Halifax
46. Set of “raw” ingredients
http://www.otisfundraisingideas.com/product/our-ingredients
To be selected,
prepared, adapted,
and consumed as
needed to suit your
context.
47. What it is not……
A step-by-step recipe to be rigidly followed
48. What it is not……
A step-by-step recipe to be rigidly followed
A resource on how to evaluate the implementation, effectiveness or
impact
49. Who is the audience
Funders
Projects or organizations involved in policy-influence work
Evaluators
Assumes a basic understanding of evaluation
50. Uses for the guide
Read it to understand the complexities of and
options for policy-influence work and its evaluation
51. Uses for the guide
Read it to understand the complexities of policy-
influence evaluation
Use some of the frameworks to develop evaluation
requirements for funded projects
52. Uses for the guide
Read it to understand the complexities of policy-
influence evaluation
Use some of the frameworks to develop project-
specific evaluation requirements
Use the tools to work with projects to plan their
policy-influence work and evaluation
54. • 21 sites in 8 Provinces and Territories
(BC, AB, SK, MB, ON, NL, NB, NWT)
• 150+ trained professionals across Canada
• 1,000+ children, youth and caregivers
• Community context:
• Northern, rural, urban, Aboriginal, multi-cultural
• Low income, immigrant/refugee, foster families
Snapshot of HT
55. Program effectiveness
• HT demonstrated that inclusion of healthy
cooking/eating and physical activity within group
learning was key to participant engagement, leading to
positive behaviour change.
• Reduced consumption of sugar sweetened beverages
• Increased physical activity
• Reduced screen time
• Relationships that developed during HT sessions led to
building social support for families, particularly in
underserved communities.
56. HT ‘policy influence’ needs
• An effective policy development framework
• Concerted effort by multi-sectorial partners to promote
healthier weight practices and policy
• Tools to measure the impact of policy influence work
57. Complexity of HT policy influence
• In order to be effective, HT needed to
– Be responsive
– Be adaptable
– Build capacity
– Enhance knowledge uptake
58. HT policy influence - process
Policy
Development
Placement on
Policy Agenda
Policy
Adoption
Policy
Implementation
Policy
Maintenance
59. Tool for HT Policy Influence Work –Goals & Activities
Practice/ Policy Goal Activities/Tactics Timeline
What is an example of a practice/ policy related to achieving healthier weights?
How do you envision Healthy Together influencing practice/policy discussions?
What practice/policy goal(s) do you think is possible within your sphere of influence?
What activities/tactics will you engage in, to achieve the above goals, and the estimated timeline?
Comments
Thank you for your participation!
Office/Agency
Organization/Network
Community
Province
Other
60. HT policy influence activities
• Program uptake/practice change by sites
• Establishing relationships with key decision makers
• Policy makers’ education/ capacity building
• Policy development/maintenance
• Policy implementation/evaluation
• Demonstration projects/pilots
• Electronic outreach through social media
• Briefings/presentations
61. • Adoption of HT approach into core practice
• Communities/systems engaged in ‘buy-in’
• Stakeholders leverage partnerships to support uptake
Ultimate outcome:
‘Vulnerable children will experience greater equality of health
outcomes and achieve healthier weights. ‘
HT policy outcomes
62. HT theory of change
• If families at-risk of developing unhealthy
weights come together through programming
that enables healthy eating, increases enjoyment
of physical activity and strengthens the family
bond, then members will improve their overall
health and overall quality of life with skills and
habits that will last.
63. How do I get the guide?
NCCMT:
http://www.nccmt.ca/uploads/media/media/0001/01/c3374b8be4b35
e1340385f1b593d3bb9f50f6a38.pdf
the guide?
64. Wrap Up – Key Messages
• Policy-influence in order to support scale-up of evidence-based
programs is a concern to many funders and organizations
• There is a burgeoning literature on how to do and evaluate policy-
influence work
• The Guide to Policy-Influence Evaluation will help you navigate this
literature with a select number of high quality useful resources
• The Guide can help to:
• Develop evaluation requirements for policy-influence work
• Support projects and organization to do and evaluate policy-influence work
66. Follow us @nccmt Suivez-nous @ccnmo
66
Your Comments/Questions
• Use Chat to post comments
and/or questions
• ‘Send’ questions to All (not
privately to ‘Host’)
Chat
Participant Side
Panel in WebEx
67. Follow us @nccmt Suivez-nous @ccnmo
67
Poll Question #4
Could this method or tool be useful
in practice?
A. Very useful
B. Somewhat useful
C. Not at all useful
D. Don’t know
68. Follow us @nccmt Suivez-nous @ccnmo
68
Your Feedback is Important
Please take a few minutes to share your thoughts
on today’s webinar.
Your comments and suggestions help to improve
the resources we offer and plan future webinars.
The short survey is available at:
https://nccmt.co1.qualtrics.com/SE/?SID=SV_8ib8Yz9pRao
GLBz
69. Follow us @nccmt Suivez-nous @ccnmo
69
Poll Question #5
Would you be willing to receive an
email from the NCCMT with a
follow-up survey in 2-3 months?
A. Yes
B. No
70. Follow us @nccmt Suivez-nous @ccnmo
70
Join us for our next webinar
NCCMT Spotlight on Methods & Tools: CDC Clear
Communication Index
Date: Wednesday, February 1, 2017
Time: 1:00 – 2:30pm EST
Interested in evidence-based criteria for clear communication
of public health material? Do you develop and review public
health communication materials? Are you interested in
enhancing clarity and ease of understanding of these
materials? Join us for a webinar on how the Clear
Communication Index can help you.
Register at: https://health-evidence.webex.com/health-
evidence/onstage/g.php?MTID=eb180a59b9994f1d4da0729d
c3d31345d
71. Follow us @nccmt Suivez-nous @ccnmo
71
Poll Question #6
What are your next steps? (Check all
that apply)
A. Access the method/tool referenced in the
presentation
B. Read the NCCMT summary about the
method/tool described today
C. Consider using the method/tool in practice
D. Tell a colleague about the method/tool
72. Follow us @nccmt Suivez-nous @ccnmo
Funded by the Public Health Agency of Canada | Affiliated with McMaster University
Production of this presentation has been made possible through a financial contribution from the Public Health Agency of Canada. The
views expressed here do not necessarily reflect the views of the Public Health Agency of Canada..
For more information about the
National Collaborating Centre
for Methods and Tools:
NCCMT website www.nccmt.ca
Contact: nccmt@mcmaster.ca