1. Sheryl Ryan, PhD, OTR/L

2. Week 5

3.  Program evaluation:
 Definition
 Purposes
 Describe several approaches
 Discuss the process
 Describe the differences between experimental, quasi-experimental, and non-
experimental evaluation designs.
 Discuss the appropriate use of qualitative methods in program evaluation.
 Identify the uses of evaluation results.
 Discuss the importance of disseminating the results of program development and
evaluation.
 Identify the ethical considerations in designing and conducting evaluation research.

4. Quiz 1
Vocabulary database
2 Quotes/2 Questions
Case: Ocean Therapy
Program Evaluation
Photo: http://www.surfersway.org/

5.  Appreciative inquiry approach
 Conceptual, instrumental, and process
use
 Contingency perspective
 Efficiency evaluations
 Impact evaluations
 Outcome evaluations
 Process evaluations
 Experimental, non-experimental, and
quasi-experimental designs
 Formative and Summative evaluation
 Logic models
 Managerial approach
 Objectives approach
 Participatory approach
 Program efficiency evaluation
 Program evaluation
 Qualitative
 Quantitative
 Stakeholders
 4D Model
 Indicators

6. Chapter 6

7. Carly Rogers:
Surfing – Infinite
Possibilities to
Heal
https://www.youtube.com/watch?v=
Wfb8tHn8Xv4
http://jimmymillerfoundation.org/ocean-therapy/

8. Collect data and
information
Use it to inform future
action
“Evaluators rely on stakeholders to
help develop the essential questions
that will be answered in the
evaluation” (Scaffa and Reitz, 2014:97)

9.  Evaluation is vital to helping design
and shape your practice
 Evaluation of your program should be
one of the first things you consider
during the development stage
 Evaluation should be continuous,
throughout the life of your program
https://www.opendemocracy.net/openglobalrights/evaluation-and-human-rights

10. :
Formative or Process Evaluations
 Determines whether program
activities have been implemented as
intended and resulted in certain
outputs. The results are used for
program improvement.
 Theoretical framework
 Design
 Activities
 Operation
Summative
 When the results are used to
determine effects or
continue/discontinue the program.
 Outcomes
 Impact
 Effectiveness/Efficiency
https://education.uky.edu/evaluationcenter/

11. Needs assessment
Program theory
Program implementation
Program outcome – Long-term effects
Program impact – Immediate effects
Program efficiency – Cost/benefit analysis justifies funding
https://www.isasurf.org/first-ever-isa-world-adaptive-surfing-championship-set-to-be-held-in-la-jolla-california-september-24-27/

12. Objectives Approach
 Collect data by previously agreed
upon methods to determine whether
goals and objects are being met.
 Relatively easy to quantify, but
limited to evaluating program
objectives only.
 Rarely used in isolation.
Managerial Approach
 Goal is to provide useful information
to managers to improve their decision
making.
 Evaluations look at how the program
operates as well as goals.
 Value of a program is determined by
managers and is reflected in decisions
they make based on evaluation
results.

13. Participatory Approach
 A formative/process approach
 Qualitative
 Considers the needs of all stakeholder
groups rather than just managers –
360* approach.
 Determines critical information
needed by each of stakeholder groups
to develop program improvement
plans.
Utilization-Focused Approach
 A process for including “users” in the
design of the evaluation process.
 Shifts attention from the program to
be evaluated and focuses on those
who will use the program evaluation
results.
 Highly situation and context specific.
 Mixed qualitative and quantitative
methods.

14. Appreciative Inquiry Approach
 4D Model
 Discovery – appreciate what is good
 Dream – envision what might be
 Design – consider what should be
 Destiny – implementing change
AKA Transtheoretical Model of Health
Behavior Change
 5 Principles
 Constructivist – multiple realities exist
 Simultaneity – inquiry is intervention
 Poetic – programs narrating own stories
and can change directions at any time
 Anticipatory – imagination of program
guides its current actions
 Positive – focus on positive experiences
increases motivation, inspiration, and
engagement
http://dancumberworth.co.uk/practice/

15. Develop evaluation questions
Determine
data needs
Choose evaluation
methods
Utilize evaluation
results
Identify stakeholders
http://www.theintrinsicvalue.com/tag/picture-of-a-airplane

16.  Determine OT Related Data Needs  Occupational performance
 Adaptation
 Health and wellness
 Participation
 Prevention
 Quality of life
 Role competence
 Self-advocacy
 Occupational justice
https://www.amazon.com/Occupational-Therapy-Practice-Framework-Process/dp/1569003610

17. Choosing Evaluation Methods
 Quantitative Designs
 Non-experimental
 Cross-sectional, cohort
 Quasi-experimental
 Compare two groups, one receiving
intervention
 Experimental
 Randomized controlled trial
 Qualitative Designs
 Provides an in-depth perspective on
phenomena that are not easily
quantifiable.
 Thick description, naturalistic,
experience oriented
 Interviews, focus groups, observation,
document review

18.  Instrumental use – to inform future action
 Conceptual use – impact on decision makers’
thought processes
 Process use – cognitive and behavioral changes result
from participating in the process
 Symbolic use – for political gain
COMMUNICATION!

19.  Beneficence
 Select appropriate evaluation approach
and outcome measures
 IRB approval
 Using current assessments, following
copyright laws
 Nonmaleficence
 Train evaluators and staff to ensure
competency
 Autonomy/Confidentiality
 Secure evaluation materials and data
 Compliance with evaluation protocol
and confidentiality as needed
 Procedural Justice
 IRB approval
 Veracity
 Report accurate results to participants
and stakeholders in a timely manner
 Fidelity
 Avoid conflicts of interest