“Evaluation” and “data-based decision making” are common terms that are understood to varying degrees, but not always put into practice. People use evaluative thinking and make “data-based” decisions every day. For example, if you are trying to lose weight, you might look at nutrition information to decide whether to eat an apple or a candy bar. Or, your organization might decide to use one training program over another based on employee feedback and previous results of the program.
However, formal evaluation that elicits information that represents reality and facilitates accurate data-based decisions can be challenging. We will provide an interactive introduction to formal evaluation that will answer:
What is evaluation?
Why would I do evaluation?
How do I do evaluation?
6. What is Evaluation?
• Definition
– Evaluation is the systematic assessment of the
worth or merit of some object
– Evaluation is the systematic acquisition and
assessment of information to provide useful
feedback about some object
10. What is Evaluation?
• Formative:
– On-going process that allows for feedback to be
implemented during a program cycle
– Adjustments “on the fly” to help achieve program goals
11. What is Evaluation?
• Formative Types:
– Needs Assessment
– Evaluability Assessment
– Structured Conceptualization
– Implementation Evaluation
– Process Evaluation
12. What is Evaluation?
• Summative:
– occurs at the end of a program cycle and provides
an overall description of program effectiveness.
– Focuses on answering questions
27. Activity
• THINK- Think about the following questions:
– What question(s) do you want to answer for
your organization??
– What do you need to collect to answer the question?
– How would you collect data?
– How would you report the results?
• PAIR- Brainstorm with the
people around you
• SHARE- With everyone
Talk about examples of what we have done.School DistrictsHigher Ed Non-profitsFoundationsFederalStateFoundation funded projectsOrganizationally funded projects – need assessments, specific project focused analysis, logic modeling
This definition is hardly perfect. There are many types of evaluations that do not necessarily result in an assessment of worth or merit -- descriptive studies, implementation analyses, and formative evaluations, to name a few. Better perhaps is a definition that emphasizes the information-processing and feedback functions of evaluation. Note: What is evaluation?
This definition is hardly perfect. There are many types of evaluations that do not necessarily result in an assessment of worth or merit -- descriptive studies, implementation analyses, and formative evaluations, to name a few. Better perhaps is a definition that emphasizes the information-processing and feedback functions of evaluation.
Can answer all or any of these.
needs assessment determines who needs the program, how great the need is, and what might work to meet the needevaluability assessment determines whether an evaluation is feasible and how stakeholders can help shape its usefulnessstructured conceptualization helps stakeholders define the program or technology, the target population, and the possible outcomesimplementation evaluation monitors the fidelity of the program or technology deliveryprocess evaluation investigates the process of delivering the program or technology, including alternative delivery procedures
Focuses on answering questionsWere your program objectives met?Will you need to improve and modify the overall structure of the program?What is the overall impact of the program?What resources will you need to address the program’s weaknesses?
outcome evaluations investigate whether the program or technology caused demonstrable effects on specifically defined target outcomesimpact evaluation is broader and assesses the overall or net effects -- intended or unintended -- of the program or technology as a wholecost-effectiveness and cost-benefit analysis address questions of efficiency by standardizing outcomes in terms of their dollar costs and valuessecondary analysis reexamines existing data to address new questions or use methods not previously employedmeta-analysis integrates the outcome estimates from multiple studies to arrive at an overall or summary judgment on an evaluation question
Empowerment evaluation – emphasizes that the evaluation process and methods should be empowering to those who are being evaluation (Fetterman, Kaftarian et al. 1996).Goal-free evaluation – in which the evaluator’s task is to examine all of the outcomes of a program, not just its formal outcomes as identified in its objectives (Scriven 1972).Naturalistic or 4th generation evaluation – emphasizes the qualitative uniqueness of program and is a reaction against the limitation of quantitative evaluation approaches (Guba and Lincoln 1989).Stakeholder evaluation – looks at the differing perspectives of all of a program’s stakeholders (those who have an interest in it) (Greene 1988).Strategic evaluation – emphasizes that evaluation design decisions should be driven by the strategic value of the information they will provide for solving social problems (Duignan 1997).Theory Based Evaluation – puts an emphasis on detailing the assumptions on which a program is based (intervention logic) and follows those steps to see if they occur (Birckmayer and Weiss 2000).Utilization-focused evaluation – determines methods on the basis of what is going to be most useful to different audiences (Patton 1986). It helps make sure that evaluation is not only useful to funders by measuring successes, but also:•Helps staff make real improvements in the program•Helps administrators make decisions about future programs•Helps the community make decisions about resources.
Note: why would you do evaluation? What are they currently doing? (get them to talk about what they are doing) talk about when to formalize the process… when it would be necessaryFor OD:Decision makingRetentionEmployee developmentStrategic planningTo measure hiring practices
