Presentation by Dr Shannon McDermott on 'Evaluating permanent supportive housing, Elizabeth Street' at the recent Permanent Supportive Housing Forum in Melbourne, Sept 2012.
2. Overview
• Evidence base for
permanent supportive
housing
• Challenges for building
the evidence
• The Evaluation of the
Elizabeth Street Common
Ground Supportive
Housing Project
• Ensuring evaluation
resources are well spent
3. What is the evidence?
• Pathways to Housing has been the most rigorously
studied
• Have high retention rates (75-88%) in the first two
years
• Non housing outcomes are less clear (drug & alcohol,
isolation; Johnson, Parkinson and Purcell, 2012)
• Research has not kept pace with new models
• We need to be cautious about applying overseas
research to the Australian context
4. Evidence hierarchy for Australian policymakers
(Leigh, 2009)
1. Systematic reviews of multiple randomized trials
2. High quality randomized trials
3. Systematic reviews of natural experiments or pre-post studies
4. Quasi experiments
5. Pre-post studies
6. Expert opinion and theoretical conjecture
5. Some challenges in evaluation design
• Randomised controlled trials can be costly
• Require relinquishing some control to
researchers
• Results are quantitative and can be devoid of
context
• Essentially, there is no one truth out there
waiting to be discovered; evidence can and
should be contested
6. The Elizabeth Street Program
• Targeted at chronically
homeless and highly
vulnerable people with
complex needs
• 131 units – 65 for formerly
homeless
• Includes office space for onsite
support services
• Partnership between Yarra CH
(owns and manages the
building) and Homeground
(onsite support and concierge
service)
7. Evaluation questions
1. What is the Elizabeth Street project service model and to
what extent does it reflect what was originally intended?
2. What impact has the Elizabeth Street project had on the
mental and physical health, social well being and economic
participation of the formerly homeless residents?
3. How efficiently have Elizabeth Street resources been used?
4. What lessons have been learnt about the supported housing
model: its structure, establishment and implementation?
5. What lessons have been learnt about how to support
previously homeless residents?
8. Evaluation methods
Evaluation methods Brief description
Quantitative data De-identified resident data from HomeGround (demographics,
collection and service use, incidents), Yarra CH (tenancy issues and outcomes).
analysis Health data from DOH; resident survey with small sample of
residents
Qualitative data Interviews with formerly homeless and low income residents,
collection and HomeGround and Yarra CH staff and DHS, DoH staff.
analysis
Cost and benefit data Financial data on the cost of the program is being collected
collection and from key stakeholders
analysis
Observation The researchers spent time at 660 Elizabeth Street to
understand how the program operates in practice.
9. Some reflections - NGOs
• Plan to collect excellent program data and use clever data collection
systems
• Partner very early on with academics
• Make research and evaluation part of the organization culture
• Define and document what you think the program will achieve from
the beginning. Develop a logic model detailing the inputs, activities,
outputs and outcomes of your program.
• Think through ethical issues regarding data sharing and use in
research
• Keep good financial records by tracking your budgeted and actual
costs. This is particularly crucial for economic evaluations.
10. Some reflections – government funders
• Involve academics earlier in program design so that
sound research design is possible
• Administrative data should become more widely
available for research purposes.
• Reconsider intellectual property restrictions to encourage
data sharing and publication of results
• Think about other models of funding evaluations to
ensure that they remain independent from government
politics
11. Some reflections – academics
• Need to partner with governments and NGOs to produce
more high quality program evaluation
• We need to work on producing research that can be
inform project design and operation, and to produce
these outputs in a timely fashion
• We need to develop more ways of implementing theory
in practice, especially in relation to economic evaluation
12. Acknowledgements and references
I would like to thank:
• Kristy Muir and my project team
• Our funders, DHS
• All the residents and stakeholders who contributed to the evaluation
Selected references
Johnson, G., Parkins, S., & Parsell, C. (2012). Policy shift or program drift?
Implementing Housing First in Australia. Melbourne: Australian Housing and
Urban Research Institute.
Leigh, A. (2009). What evidence should social policymakers use? Economic
Roundup, 1, 27-43.
