1.
Organizing and Managing Program Evaluation

2.
Presentation Outline <ul><li>Setting the Stage: U.S. Housing Ministry </li></ul><ul><li>Broad Organization of the Evaluation Function </li></ul><ul><li>Defining Evaluation Topics </li></ul><ul><li>Managing Evaluation Projects </li></ul><ul><li>Concrete Examples </li></ul>

3.
Principal tasks <ul><li>Insuring mortgages – FHA </li></ul><ul><li>Mortgage refinance – GNMA </li></ul><ul><li>Direct loans for elderly and handicapped housing </li></ul><ul><li>Housing subsidies to assist low income families </li></ul><ul><li>Grants to localities for community development </li></ul><ul><li>Promote “fair housing” </li></ul>

4.
Budget and Program Overview: Fiscal Year 2000 Program Block Budget ($ billion) No. of Major Programs Percent of budget total Community development 7.8 12 24 Housing Assistance 23.3 11 72 Housing finance 0.8 25-30 2.5 Fair housing 0.4 2 1.4 total 32.2 60-65 100

5.
Organization <ul><li>Minister </li></ul><ul><li>Deputy Minister </li></ul><ul><ul><li>Program Assistant Ministers (A/M) </li></ul></ul><ul><ul><li>A/M for Administration (budget, personnel) </li></ul></ul><ul><ul><li>CFO (financial management; efficiency) </li></ul></ul><ul><ul><li>Inspector General </li></ul></ul><ul><ul><li>A/M for Policy Development & Evaluation (PDE) </li></ul></ul>

6.
Office of Policy Development and Evaluation <ul><li>Provide advice and information to minister for decision making </li></ul><ul><li>Monitor and evaluate ministry’s programs </li></ul><ul><li>Conduct research on priority housing and community development issues </li></ul><ul><li>Key participant in the budget process </li></ul><ul><li>Feeds evaluation results into policy </li></ul>

7.
2. Organization of the Evaluation Function

8.
Division of tasks among offices <ul><li>Program monitoring, including performance measurement </li></ul><ul><ul><li>Program A/Ms </li></ul></ul><ul><ul><li>CFO </li></ul></ul><ul><ul><li>PDE </li></ul></ul><ul><li>Program evaluation – PDE </li></ul><ul><ul><li>Within PDE, Office of Research & Evaluation </li></ul></ul>

9.
Office of Research & Evaluation <ul><li>Defines evaluation agenda </li></ul><ul><li>Working with program offices, defines the objectives of the evaluations and methods </li></ul><ul><li>Conducts competitions to select firms to do work </li></ul><ul><li>Oversees works of contractors </li></ul>

10.
3. Defining Evaluation Topics

11.
Who nominates programs for evaluation? <ul><li>Congressional mandates </li></ul><ul><li>Office of Management & Budget (Office of the President) </li></ul><ul><li>Program A/Ms </li></ul><ul><li>Office of Policy Development and Evaluation </li></ul>

12.
Federal Government Budget Cycle --Illustrated for the FY2001 Budget-- Date Action August 1998 OMB sends “management letter to agency January 1999 Minister issues instructions for the budget process, including guidance on priorities February-March Offices work on submission April – July Internal review process July-August Finalization September Submission to OMB October-November Formal hearings with OMB mid-November OMB “pass-back” mid-Nov to mid-December Appeals; final decisions mid-January 2000 Submission to Congress May-August / October 1 Congress acts ; new fiscal year

13.
4. Managing evaluation projects

14.
Basic steps in the process <ul><li>Defining the questions to be addressed </li></ul><ul><li>Determining the methodolgy </li></ul><ul><li>Estimating the budget </li></ul><ul><li>Preparing the TOR or RFP </li></ul><ul><li>Competition and award </li></ul><ul><li>Overseeing the contractor </li></ul><ul><li>Getting policy impacts </li></ul>

15.
1. Defining the questions to be addressed <ul><li>Understand the program before meeting with program officials about the evaluation </li></ul><ul><li>“ No client, no impact.” </li></ul><ul><ul><li>Make the program office your partner </li></ul></ul><ul><ul><li>Understand who else may be a “primary intended user” </li></ul></ul><ul><ul><li>Consult them at the beginning and keep them fully informed </li></ul></ul>

16.
Reality check <ul><li>For each question ask </li></ul><ul><ul><li>Can the question be answered with program evaluation tools? </li></ul></ul><ul><ul><li>What is the relevant indicator? </li></ul></ul><ul><ul><li>What is the source of the data? What does it cost? </li></ul></ul><ul><li>Is the overall cost affordable? </li></ul><ul><li>If the answers are positive proceed. If not, redesign. </li></ul>

17.
2. Determine the methodology <ul><li>Depends on the type of evaluation—process, impact or benefit-cost </li></ul><ul><li>Prepare clear statement of method that can be critically reviewed </li></ul><ul><li>Where data are to be collected, estimate sample sizes </li></ul><ul><li>If you do not have the capacity on staff, hire an expert to help </li></ul>

18.
3. Estimate the budget <ul><li>Labor for structuring the project, analysis, reporting (build up task-by-task) </li></ul><ul><li>Travel </li></ul><ul><li>Data collection </li></ul><ul><ul><li>Cost per household or program manager interviewed; office file checked </li></ul></ul><ul><li>Overheads </li></ul><ul><ul><li>Office space, communications, support staff, graphics, computers… </li></ul></ul>

19.
Staff loading chart for analysis and results write-up for impact analysis using econometric methods (days) Task Senior staff Mid-level Junior staff Data analysis --specify models 12 8 -- --estimation 3 5 15 --sensitivity tests 4 3 9 Results write-up --outline development 3 1 -- --writing 10 15 2 --table preparation -- -- 8 TOTAL 32 32 34

20.
4. Preparing the Analysis Plan or RFP <ul><li>Share the statement with the client even if they are not experts in evaluation or social sciences </li></ul><ul><li>Even if internal staff will do the analysis, the discipline of the Analysis Plan is very useful </li></ul><ul><li>The RFP should be read by the review panel to be sure they understand it and agree with it. </li></ul>

21.
The RFP <ul><li>Be clear about </li></ul><ul><ul><li>Objectives; questions to be answered </li></ul></ul><ul><ul><li>Expected products, schedule </li></ul></ul><ul><ul><li>Sample sizes, illustrative list of locations, etc. to facilitate analysis of budgets </li></ul></ul><ul><ul><li>Do not give too much guidance on methodology </li></ul></ul><ul><ul><li>Provide an indication of level-of-effort </li></ul></ul><ul><ul><li>Results dissemination: buy some time </li></ul></ul><ul><ul><li>Include the evaluation criteria and weight assigned to each </li></ul></ul>

22.
5. Competition and award <ul><li>Review panel is formed before the RFP is issued </li></ul><ul><li>Includes people with the necessary competence (use consultant if necessary) </li></ul><ul><li>Have someone from the client’s staff participate </li></ul><ul><li>Formal criteria and scoring sheets </li></ul><ul><li>“ Orals” and “Best and Finals” </li></ul><ul><li>Absolute confidentiality </li></ul>

23.
Standard factors evaluated <ul><li>Contractor’s understanding of the issues </li></ul><ul><li>Quality of the methodological design </li></ul><ul><li>Quality of the management plan </li></ul><ul><li>Quality of the proposed staff </li></ul><ul><li>Past experience of organization </li></ul><ul><li>Price </li></ul>

24.
6. Overseeing the contractor <ul><li>Standard tools </li></ul><ul><ul><li>Detailed review of </li></ul></ul><ul><ul><ul><li>Work plan </li></ul></ul></ul><ul><ul><ul><li>methodology </li></ul></ul></ul><ul><ul><li>Monthly progress reports </li></ul></ul><ul><ul><li>Briefings at key points </li></ul></ul><ul><ul><li>“ Milestone” reports </li></ul></ul><ul><ul><li>Draft final report </li></ul></ul>

25.
What to do if the contractor underperforms? <ul><li>More frequent, intense reviews; be clear on areas of weakness </li></ul><ul><li>Informal contact with senior management at the firm </li></ul><ul><li>Official letter to firm stating problems </li></ul><ul><li>Legal action to deny payment (very unusual) </li></ul><ul><li>Important that end-of-project records of performance be maintained </li></ul>

26.
7. Getting policy impacts <ul><li>Work with the client </li></ul><ul><ul><li>Don’t overlook the positive points </li></ul></ul><ul><ul><li>Make sure they understand the basis for criticisms </li></ul></ul><ul><ul><li>Have specific recommendations for improvements </li></ul></ul><ul><li>Report the results to other interested parties—but do it in a constructive way </li></ul>

27.
Sharing results <ul><li>Never “slant” the results or your credibility will be severely undermined </li></ul><ul><li>Make sure you are informed by the contractor about their plans for release of the results </li></ul>

28.
<ul><li>Specific examples of program </li></ul><ul><li>evaluations </li></ul>

29.
<ul><li>Demonstration project— </li></ul><ul><li>“ Moving to Opportunity” </li></ul>

30.
Subject -- Using housing vouchers to help poor families leave inner-city neighborhoods for middle class areas -- Evidence from non rigorous studies that such moves had significant positive impacts on the families who moved

31.
Context & Client <ul><li>Use of housing vouchers as a successful anti- poverty instrument was very exciting </li></ul><ul><li>Congress could mandate changes in the use of vouchers, increasing use for this purpose </li></ul><ul><li>Congress mandated a rigorous evaluation to determine if there is a sound basis for change </li></ul><ul><li>Funds included in HUD budget </li></ul>

32.
Specific objectives <ul><li>Determine the impacts on participating families of moving to middle-class neighborhoods </li></ul><ul><li>Impacts on both </li></ul><ul><ul><li>adults (employment, welfare dependence) </li></ul></ul><ul><ul><li>children (school achievement, crime, drugs) </li></ul></ul>

33.
Participants in determining questions & design <ul><li>HUD-evaluation office </li></ul><ul><li>Congressional staff </li></ul><ul><li>Advisory panel, including foundations (who contributed funds for the evaluation) </li></ul>

34.
Evaluation design <ul><li>10-year impact evaluation; 5-cities, almost 5,000 households </li></ul><ul><li>Random assignment </li></ul><ul><li>Three groups selected from residents of public or assisted housing </li></ul><ul><ul><li>Families given vouchers to rent housing in middle class areas </li></ul></ul><ul><ul><li>Families give vouchers to use anywhere </li></ul></ul><ul><ul><li>Control group, starting in public housing </li></ul></ul>

35.
Measurement Example: education achievement <ul><li>Achievement measured by </li></ul><ul><ul><li>Scores on series of standardized tests </li></ul></ul><ul><ul><li>School performance—grades </li></ul></ul><ul><ul><li>Advanced coursework </li></ul></ul><ul><ul><li>Application of seniors for college </li></ul></ul>

36.
Procurement—interim evaluation <ul><li>Contract awarded through a competition </li></ul><ul><li>5 firms competed </li></ul><ul><li>Evaluation board--only PDE staff </li></ul><ul><li>Size of contract: $8.2 million </li></ul><ul><li>3 year contract </li></ul>

37.
Monitoring contractor performance (only interim report) <ul><li>Workplan required, including detailed analysis plan </li></ul><ul><li>Monthly progress reports </li></ul><ul><li>Several briefings to HUD, Hill staff, and foundations </li></ul><ul><li>HUD staff time: 1 staff year </li></ul>

38.
Conclusions for education achievement <ul><li>Results are for MTO-vouchers in comparison with the both other groups </li></ul><ul><li>Findings for 1994 thru 1997 </li></ul><ul><li>No significant effect on achievement by all measures </li></ul><ul><li>No significant effect on the quality of schools attended </li></ul>

39.
Policy impact <ul><li>Results are very fresh </li></ul><ul><li>Interim findings; 5 years to go </li></ul><ul><li>Results contrary to expectations </li></ul><ul><li>Sufficient to prevent major change in use of housing vouchers </li></ul><ul><li>Congress will probably wait for final results </li></ul>

40.
<ul><li>Lending for Economic Development </li></ul><ul><li>in Low-Income Neighborhoods </li></ul>

41.
Subject-1 <ul><li>-- HUD has several programs that lend money to </li></ul><ul><li>private businesses, guarantee loans made by banks </li></ul><ul><li>or provide credit enhancements for guaranteed </li></ul><ul><li>loans. All involve subsidies </li></ul><ul><li>CDBG – loans of grants funds by city through banks </li></ul><ul><li>S.108 – loan guarantees </li></ul><ul><li>Credit enhancements on S.108 loans </li></ul>

42.
Subject-2 -- These loans are to businesses in CDBG target neighborhoods or primarily serving poor households --Evaluation Issue: how successful are these programs in generating economic development?

43.
Context & Client <ul><li>Programs not previously carefully evaluated </li></ul><ul><li>Client: Office of Community Planning and Development—request of the A/M </li></ul><ul><li>Request to evaluation office </li></ul>

44.
Specific questions (selected) <ul><li>Impact of programs on business development and job creation </li></ul><ul><li>How do these loans perform? </li></ul><ul><li>What is feasibility of creating a secondary market for these loans? </li></ul><ul><li>So combination impact and process evaluation </li></ul>

45.
Participants in determining questions & design <ul><li>HUD-evaluation office </li></ul><ul><li>HUD program office </li></ul>

46.
Evaluation design: business development & job creation <ul><ul><li>Measurements </li></ul></ul><ul><ul><li>-- No. of jobs created vs. no. planned </li></ul></ul><ul><ul><li>-- Cost per job created vs. cost per job of other federal econ-develop programs </li></ul></ul><ul><ul><li>--survival rate of businesses compared with all small businesses </li></ul></ul>

47.
Program experience analyzed <ul><li>Loans originated </li></ul><ul><ul><li>CDBG: 1996 – 1999 </li></ul></ul><ul><ul><li>S.108: 1994 - 1999 </li></ul></ul>

48.
Evaluation design: loan performance <ul><ul><li>Measurement: default rates compared with those of banks to similar business </li></ul></ul><ul><ul><li>-- closed loans, 1996-99 </li></ul></ul><ul><ul><li>-- loans still open at the time of the study </li></ul></ul><ul><ul><li>Loss rates </li></ul></ul>

49.
Procurement <ul><li>Contract through competition </li></ul><ul><li>3 firms competed </li></ul><ul><li>Evaluation board--only PDE staff </li></ul><ul><li>Size of contract: $2.1million </li></ul><ul><ul><li>Huge data collection effort </li></ul></ul><ul><li>2.5 year contract </li></ul>

50.
Monitoring contractor performance <ul><li>Workplan, analysis design report, and data collection plan required </li></ul><ul><li>Monthly progress reports—narrative, issues, financial report </li></ul><ul><li>Two reports: </li></ul><ul><ul><li>Interim on data assembled </li></ul></ul><ul><ul><li>final </li></ul></ul><ul><li>Two briefings </li></ul><ul><li>HUD staff time: 4 staff months </li></ul>

51.
Conclusions for business and job development <ul><li>Jobs created equaled 93% of those indicated in loan applications </li></ul><ul><li>Average cost to create one job </li></ul><ul><ul><li>CDBG = $2,675 </li></ul></ul><ul><ul><li>S.108 (loan guarantee) = $7,865 </li></ul></ul><ul><ul><li>Compared with: Other gov’t econ develop: $936 - $6,250 </li></ul></ul><ul><li>Overall positive impact on business survival rates </li></ul>

52.
Conclusions for loan performance <ul><li>Default rates on closed loans </li></ul><ul><ul><li>CDBG = 26% </li></ul></ul><ul><ul><li>S.108 = 42% </li></ul></ul><ul><li>Unsubsidized loans have lower default rates but are made to the most creditworthy borrowers </li></ul>

53.
Policy impact <ul><li>Results are only a few months old </li></ul><ul><li>Likely that the program office will use results to provide better guidance to cities on use of loan programs, particularly the characteristics of excessively risky loans </li></ul><ul><li>Explicit guidance on underwriting standard may result </li></ul>

54.
Key Steps in Managing a Program Evaluation <ul><li>Have a strongly interested client </li></ul><ul><li>Rigorously define the questions to be addressed </li></ul><ul><li>Have a general idea of the methodology </li></ul><ul><li>Open competition </li></ul><ul><li>Invest in monitoring contractor performance </li></ul><ul><li>Work with the program office to get the results used </li></ul>