Lee Frelich's "Climate Change & Forests" Presentation
Empowering youth to be evaluators: Involving Young People in Evaluating Informal Education Programs Presentation
1. 3/23/10
Empowering Youth To Be Evaluators:
Involving young people in evaluating
informal education programs
Amy Grack Nelson, Evaluation & Research Associate
Science Museum of Minnesota
Overview
Overview of participatory evaluation
Participatory evaluation examples
Sampling of interactive techniques
What is participatory evaluation?
1
2. 3/23/10
It’s All About Utility
Utility - one of the four essential features of all
evaluations
(Joint Committee on Standards for Educational Evaluation, 1994)
A way to help ensure use is to increase the primary
intended users’ level of participation in the
evaluation. (Cousins & Earl, 1995; Patton, 2008)
Participatory Evaluation
“Applied social research that involves trained evaluation
personnel and practice-based decision makers working
in partnership.” (Cousins & Earl, 1995, pg. 8)
Core purpose increasing use
Characteristics of Participatory Evaluation
Balanced control of evaluation process
Involvement of primary users
Extensive participation throughout the evaluation
(Cousins & Earl, 1995; Cousins & Whitmore, 1998)
Interactive Evaluation Practice Continuum
(King & Stevahn, 2002)
2
3. 3/23/10
Benefits of Participatory Evaluation
Increases use of evaluation results by:
Enhancing relevance of the evaluation
Increasing understanding of the data
Increasing ownership of the findings
(Cousins & Whitmore, 1998; King & Stevahn, 2002; Patton, 2008)
Evaluation capacity building
Develop analytic and evaluative skills
Stakeholders develop a more “critical eye”
(Cousins & Earl, 1992, 1995)
Participatory Evaluation Examples
Science Museum of Minnesota’s
Kitty Andersen Youth Science Center
Kitty Andersen Youth Science Center’s Park Crew
Facilitate earth science and environmental education
activities in the Big Back Yard and on outreaches
3
4. 3/23/10
Example 1: Summative Evaluation
Youth will…
Learn about water related earth surface processes
Develop teaching skills
Learn about related science, technology, engineering, and
math (STEM) careers
Evaluation design
Observations and interviews of youth staff at the
beginning and end of summer
Evaluation workshop to engage youth in results
Reviewing the Work
Keep/Change
Discussion
4
5. 3/23/10
Incorporating
Evaluation Data
How often youth talked about why something is considered a pollutant (n=27)
Keep/Change Discussion
Benefits Limitations
Engages users with data to Takes considerable amount
think about successes and of time
areas of improvement
Can be used to generate
recommendations
5
6. 3/23/10
Youth Benefits
Meaningful involvement in evaluation can help youth
develop higher order thinking skills, specifically
analytic and evaluative skills. (London et al., 2003)
Youth became more reflective of their work.
Youth comments reflected increased knowledge of the
activities and confidence in sharing that knowledge with
visitors.
Youth had a stronger sense of ownership and control.
Adult Staff Benefits
Provided important feedback about the crew’s work.
Gained deeper understanding of the participants’
experience and could proactively identify and respond
to their needs.
Increased understanding of evaluation and ability to
interpret data and generate recommendations.
Developed capacity to include participatory evaluation
in future work.
6
7. 3/23/10
Outcomes of the Process
Participants experience a sense of empowerment and
pride when they have an influence on the way
programs are run and see their ideas acted upon.
(Checkoway et al., 2003; Horsch et al., 2002; London et al., 2003)
Youth used suggestions to develop their own training.
They created a visitor survey and collected data.
Youth shared their ideas with a museum operations staff
member.
Example 2:
Formative Evaluation of Outreaches
Evaluation Process
Identify daily Craft survey
Pilot surveys
objectives questions
Discuss pilot data
Enter and code Administer
and revise
data surveys
surveys
Analyze and Generate Improve outreach
discuss data recommendations activities
7
8. 3/23/10
Organizational Requirements for PE
Evaluation must be valued
Sufficient time and resources
Commitment to organizational learning as a means to
improvement
Motivated individuals
Interest and ability to learn evaluative skills
(Cousins & Earl, 1992)
Evaluator Requirements for PE
Sufficient technical and facilitator skills
Accessible for participatory activities and support
Necessary resources and time
Serve an instructional role
Motivation and commitment to participate
A tolerance for imperfection
Flexibility
(Burke, 1998; Cousins & Earl, 1992; King, 1998)
8
10. 3/23/10
Interactive Graphs
Benefits Limitations
Quick data collection People may be
Everyone can see the influenced by others
process and results People may be hesitant
Can be used as a
to place a rating where
starting point for no one else has
deeper conversations
Can see data by
various characteristics
(King, 2009)
Carousel Sheets
Benefits Limitations
Alternative to traditional Participants may influence
brainstorming each others’ responses
Lots of information in a Tend to get first responses
short timeframe and gut reaction; not
Quick way to see patterns deep and thoughtful
Promotes high involvement Responses may be too
brief
Involves users in analysis
May need to reanalyze
some of the data
(King, 2009)
10