This AEA presentation gives results from systematic review of all published reports from 2002 to 2009

1. A 8–Year Review and Lessons Learned from Federal Education Evaluations: 2002-2009 Prepared by: Margaret Cahalan Sharon Stout Prepared for: AEA Meeting November 13 2009 Draft not for Quotation or Citation: All views and data presented are solely those of the authors and do not reflect any review or authorization from the US Department of Education

4. Published Reports from IESCEE and OPEPD/PPSS: 2002-2009 117 ---Evaluation Reports IES/NCEE: Primarily experimental design 27 reports (2003 to 2009) OPEPD/PPSS: Primarily Implementation Studies 90 reports (2002-2009) 15 Studies (study with multiple reports counted once) 50 Studies (study with multiple reports counted once)

8. Topic Summary: IES/NCEE

9. Topics for OPEPD/PPSS Elementary-Secondary Studies

10. Topics/Programs of Studies with Reports Published in Secondary-Postsecondary by PPSS: 2002-2009

11. Major Method IES/NCEE: 15 Studies

12. OPEPD/PPSS Major Methods: 50 Studies

13. Summary of Effects IES/NCEE: 15 Studies

14. Effects Summary 23 PPSS Studies that report comparative effects or outcomes

15. 15 IES/NCEE Studies Topics Effects Summary Reading 6 studies (one 9 th grade positive; 3 small positive on decoding not comprehension; 2 not detected; findings that practices were different but not outcomes; the one study with positive involved more time on reading—9 th grade study) DC Voucher 1 study --First two years no effects; third year began to see some effects for reading not for math; 26 percent did not take opportunity Mentoring 1 study-- not overall positive effects--some positive for some subgroups—60 percent of controls also had mentoring; Reading and Math Software 1 study ( not detected positive overall ) Teacher—Reading PD, Induction, Alternative Certification 3 studies (not detected overall on student outcomes; some on teacher practice for reading) 4 Math Curricula 1-study 2 curriculum’s had positive effects—relative to each other Afterschool 2 studies—1--not effects on academics or behavior; second study—positive on math not reading–48 hours more instruction

17. K-12 PPSS Studies Topics Effects Summary Title 1-general & Funding Proficiency increased; however most states will not meet goals by 2013-14; Over 1100 schools in need of improvement by 2006 and most were in advanced stages of corrective action; did not show changes in time on subjects; poorest schools get less per capita; federal more targeted than state and local Teacher Over 90 percent of teachers met highly qualified; however those in high poverty have less experience and less likely to have degree in subject teaching and less professional development than less poverty schools; Supplemental ED Services (SES) Increase in number of schools eligible; however participation rate is stable--about 17 percent taking advantage ; Those who participated were more likely to show growth than eligible non-participants in sub-study of districts—selection effects issues; Waiver studies—mixed results—higher proportion eligible participating when district service—in two large districts do better in two small did not do better Choice/Voluntary School Choice About 1 percent of students participating; Mixed results—some show no difference in outcomes when compared with those not participating; one that students did better. Most districts kept choice within district and most did not go to high performing schools when transferred out of low performing schools Reading First More time on reading and adopted practices; limited evidence 3rd and 4th graders increased reading more than those not in programs; regression discontinuity

18. K-12 PPSS Studies (continued) Topics Effects Summary Charter Facilities study found loans were given to schools that served more needy population than other charter schools in areas; most would not have qualified for private loans-too high risk Descriptive study on characteristics focus on accountability Comprehensive School Reform No effects for those who got grants; evidence that those who implemented practices and longer term had increase in growth especially in math Small Learning Communities and class size Primarily implementation study; performance reports before and after for schools found increase in and decrease in violence incidence; 9 th grade indicators and drop out rate; plains for college; no change in test Even Start Random assignment—no difference detected on major child or parent outcomes; counter factual issues Technology Meta-analysis found significant effects for on-line courses; small number of studies; extensive reports on status of technology use access in schools and class rooms. Documents growth of use computer and internet and also less use in poorer schools. Use generally not related to test score differences Teaching History & Libraries History professional development---most who took were experienced teachers; increased knowledge. Library grants increased materials and associated with higher than expected growth in test scores

20. Part 2: Example of Issues Faced in Using Evaluations for PART and Budget PPSS Postsecondary College Access and Retention Studies Case History

22. Rough Estimates of Percent of Eligible Population Served

23. PPSS College Access and Postsecondary Studies Topics Effects Summary and Policy History Student Support Services (SSS) Propensity matching found positive impact for participation in SSS supported services whether funded by the SSS project or some other sources on college retention and completion Moderately effective- 2003 Talent Search Propensity matching found positive impacts in 3 states using longitudinal data bases on high school completion, financial aid and postsecondary entrance Moderately effective-2005 GEAR Up Matched comparison of 7 th graders in original cohort when re-surveyed in 8 th grade found increases in parent knowledge and aspirations and student knowledge. Did not find impact on academic grades or course taking. Found impact on course taking for some sub-groups. Adequate- 2003 Upward Bound Math/ Science Propensity matching found positive impacts on college entrance, 4-year degree completion, planning to major in math or science and majoring in social science; majoring in math or natural science not statistically significant although positive direction—small n’s

32. Bias in Favor of Control Group: Imbalance in Project 69

33. Almost Balance in 66 other Projects

34. Uncontrolled for Bias in Favor of Control Group in Overall Sample with project 69 included

35. Impact Estimates from Two Stage Instrumental Variables Regression for Percent Obtaining a BA in +6 years based on UB Random Assignment Evaluation Note: All estimates significant at the .01 level or higher. Estimates based on 66 of 67 projects in sample representing 74 percent of UB at the time of the study. One project removed due to introducing bias into estimates and representational issues.

36. Estimated tax impact of UB compared to estimated cost based on average length of participation Increased taxes collected are 4.9 times higher than per participant cost of program

40. Process of Evaluations for Improvement