1. Program Evaluation A Practical/Painless Approach NLA/NEMA Conference – Oct. 14, 2010

2. About the presenter Kathryn Brockmeier Research Analyst/ Special Projects Associate, Nebraska Library Commission Grant writer Evaluator kathryn.brockmeier@nebraska.gov 402-471-4002

3. About you Library staff Trustee Friends/Foundation board member

4. About this presentation Introduction to program evaluation: designs, methods, and techniques of analysis Real-world examples NLA/NLC & IMLS Internship program NLC/IMLS Scholarship program NLC One Book for Nebraska Kids & Teens Q&A

5. What is a program? Stems from mission and goals Goal becomes program A collection of organizational resources geared to accomplish a certain major goal or set of goals

6. What is program evaluation? Collecting information Making decisions “What do we need to know?” For… Needs assessment, return on investment study, grant application/report

7. Benefits of evaluation Investigate what works/doesn’t work Share results with stakeholders Demonstrate effectiveness Improve staff’s involvement Model the program

8. Types of program evaluationto meet program objectives Implementation evaluation (process evaluation) Program implementation objectives Outcome evaluation Participant (stakeholder and individual) outcome objectives

9. Steps in the evaluation process #1: Assemble an evaluation team #2: Prepare for the evaluation #3: Develop an evaluation plan #4: Collect evaluation information #5: Analyze your evaluation information #6: Prepare the evaluation report

10. 1. Who will conduct the evaluation? Outside evaluator supported by in-house staff In-house evaluator supported by program staff and outside consultant In-house evaluator supported by program staff

11. 2. How to prepare for the evaluation a. Decide what to evaluate b. Build a logic model of your program c. State your program implementation and participant outcome objectives in measurable terms d. Identify the context for your evaluation

12. 2a. Decide what to evaluate Entire program, program component, service, activity Depends on financial and staff resources Grant-driven or funding-driven At start-up of program To check effectiveness

13. 2b. Logic models Assumptions about your target audience Program interventions Implementation objectives Immediate outcomes Immediate participant outcome objectives Intermediate outcomes Long-term outcomes

14. 2c. SMART objectives Specific Measurable Achievable Relevant Time-framed By the end of ___________________ (enter timeframe), increase _______________ (what will change?) by _______ (enter a percentage) % or more for ___________________ (who will experience the change?) by _________________ (how will the change occur?). http://www.marchofdimes.com/HI_SMART_objectives.pdf

15. 2d. Evaluation context Assumptions about your target audience Program interventions Implementation objectives Immediate outcomes Immediate participant outcome objectives Intermediate outcomes Long-term outcomes

16. 3. Develop the plan Discuss the evaluation framework Evaluating implementation objectives Types of info needed, Sources of info, Selection of info sources, Methods for collecting info, Confidentiality, Timeframe, Analysis methods Evaluating participant outcome objectives Do program participants demonstrate changes in knowledge, attitudes, behaviors, or awareness? Were the changes the result of the program’s interventions? Evaluation practices and procedures

17. 4. “Data” collection When and where info will be collected Who are the respondents? Methods Survey/questionnaire Interview Focus group Existing records Observation

18. 5. Analyze the information Descriptive statistics Frequency distributions Summary statistics—measures of central tendency, range Cross-tabulation Filters Content analysis Case studies

19. 6. Write the report Audience Format Writing tips Oral presentations Visual aids Considerations

20. Resources The Program Manager’s Guide to Evaluation, from the Administration for Children and Families http://bit.ly/cNVlQ8 Basic Guide to Program Evaluation, from the Free Management Library http://www.managementhelp.org/evaluatn/fnl_eval.htm