Evaluation and Policy: How Are They Connected?Geoff Nelson, Psychology, Wilfrid Laurier UniversityTim Aubry, Psychology, University of OttawaAimee Watson, Mental Health Commission of Canada

Agenda

2:45-3:00 – introductions and expectations for the workshop3:00-3:10 – policy 3:10-3:20 – evaluation3:20-3:30 – knowledge transfer3:30-3:50 – small group discussion3:50-4:00 – reporting back on small groups

PolicyThe renewal and repurposing of HPS• March 21, 2013, federal budget announcement, Economic Action Plan

2013 proposes $119 million per year over five years, nearly $600 million in total, to the Homelessness Partnering Strategy using a “Housing First” approach.

• Building on the successful At Home/Chez Soi project, the Government will continue to work in partnership with provinces and territories, communities, the private sector and other stakeholders, such as the Federation of Canadian Municipalities, to reduce homelessness. Economic Action Plan 2013 proposes $119 million per year over five years for the Homelessness Partnering Strategy using a Housing First approach. Over the coming months, the Minister of Human Resources and Skills Development will work with Government partners at the community level to confirm the implementation details of this approach.

PolicyThe renewal and repurposing of HPS• The evidence shows that the Housing First approach: can be

implemented across Canada; improves the lives of those who are homeless and have a mental illness; and makes better use of public dollars, especially for those who are high service users—as demonstrated by a reduction in the use of hospital in-patient care, emergency rooms, police detentions and justice services.

• The project will develop a body of evidence to help provinces and territories provide services to homeless people living with a mental illness. In addition, funding arrangements are being established to ensure that people participating in the project will have supports in place once the project is completed.

PolicyHow did this renewal and repurposing of HPS happen? Evaluation and trusted messengers

• At Home/Chez Soi research and MHCC knowledge translation• The conservative heartland – the Alberta experience and the

emergence of the Canadian Alliance to End Homelessness led by Tim Richter

• Looking south of the border – US research on Housing First, support from the G.W. Bush administration, development of 10-year plans to end homelessness in many US cities

PolicyHow did this renewal and repurposing of HPS happen? Political will

• Dissatisfaction with progress in reducing homelessness in Canada• Review of HPS provided an opportunity to refocus its approach• Desire to do a better job of reducing homelessness in Canada• An existing funding mechanism and program (HPS)

Program Evaluation (Rossi et al., 2004)Definition

“The systematic collection of information about the activities, characteristics, and outcomes of programs”

Why Evaluate?1. to make judgements about the program, 2. to improve program effectiveness, 3. to inform decisions about future programming”

PROGRAM MISSION PROGRAM PROCESSES

PROGRAM OUTCOMES

NEEDS INPUTS INITIAL OUTCOMES

GOALS & OBJECTIVES

VALUES & PRINCIPLES

ACTIVITIES

OUTPUTS

INTERMEDIATE OUTCOMES

LONGER-TERM OUTCOMES

RELATIONSHIP TO THE THREE MAIN TYPES OF PROGRAM EVALUATION

NEEDS ASSESSMENT

IMPLEMENTATION / PROCESS

EVALUATION

IMPACT / OUTCOME EVALUATION

Make-Up of a Program

Housing First Program Mission Program Processes Outcomes Individual:- Reduce HOUSING + Housing Stability Homelessness FIRST + Functioning + Quality of life Community: - Chronic & episodic

homelessness

Housing First: Assessment of Fidelity

Housing First:Assessment of Fidelity

Options1. Self-asessment of fidelity (Gilmer,

Stefanic, Sklar, & Tsemberis, 2013)2. External assessment of fidelity

(Mental Health Commisssion of Canada, 2014)

Outcome Monitoring“The ongoing collection and analysis of information on indicators of how well programs are performing with regard to the achievement of results” (Rossi, Freeman, & Lipsey, 1999).

Standardized self-report measures Simple counts of client status Collateral rating scales Consumer satisfaction measures

Types of Outcome Measures

Housing First: Outcome Indicators & Potential

Measures1. Housing status of clients (Simple count)2. Community functioning (Multnomah Community

Ability Scale – Collateral rating scale)3. Quality of Life (Lehman’s Quality of Life Interview

– Standardized self-report measure)4. Consumer Satisfaction Measure – (Service

Satisfaction Scale)

A PIT is a snapshot of sheltered and unsheltered homeless people in a community on a single night.

Provides a baseline Identifies needs of the populatioin Enables you to measure community level

outcomes

Helps identify priority populations

Community-Level Outcomes Point-in-Time Counts

Why do a count?

Canadian Point-in-Time Count

Methodology and Toolkit

Piloted in Seven Cities in AlbertaOctober 16, 2014

Who to Count?OPERATIONAL CATEGORY LIVING SITUATION COUNTED

1 Unsheltered o People living in public or private spaces without consent or contract ✔

1.2 People living in places not intended for permanent human habitation ✔

2 Emergency Sheltered

Emergency overnight shelters for people who are homeless ✔

2.2 Violence-Against-Women (VAW) shelters ✔

2.3 Emergency shelter for people fleeing a natural disaster or destruction of accommodation due to fires, floods etc.

3 Provisionally Accommodated

3.1 Interim Housing for people who are homeless ✔

3.2 People living temporarily with others, but without guarantee of continued residency or immediate prospects for accessing permanent housing.

3.3 People accessing short term, temporary rental

accommodations without security of tenure 3.4 People in institutional care who lack permanent

housing arrangements. ✔ 3.5 Accommodation / Reception centres for recently

arrived immigrants and refugees 4 At-Risk of

Homelesness

4.1 People at imminent risk of homelessness

4.2 Individuals and families who are precariously housed.

1

Mandatory questions

• Screening • Consent

• Gender• Age• Ethnicity• Migration• Immigration• Homelessness History• Veteran Status

Using Program Evaluation to Develop and Improve Housing First Programs

1. Conduct regular fidelity assessments 2. Integrate outcome monitoring into service delivery3. Conduct regular point-in-time counts4. Provide training of service providers on data

collection5. Provide regular feedback on collected data6. Keep analyses simple7. Make program improvements based on collected

data

Knowledge Exchange“Evidence of effectiveness alone is rarely enough to ensure adoption of interventions” (Leff & Mulkern, 2002)

What is Knowledge Exchange?The Canadian Institutes of Health Research defines KE as: a dynamic and iterative process that includes synthesis, dissemination, exchange and ethically-sound application of knowledge to improve the health of Canadians, provide more effective health services and products and strengthen the health care system www.cihr irsc.gc.ca/e/29418.html‐

Knowledge Exchange

How do you do it?Moving Knowledge to Action:

• What?• Who?• How?

Synthesis

Dissemination

Exchange

- 1 and 3 pagers, summaries, early findings

- media, community events, social media

- Engagement, training, collaboration

Knowledge ExchangeInfluencing Policy

Lessons from the At Home/ Chez Soi experience:

Throughout At Home/Chez Soi, the project team engaged in an ongoing process to work with government. Strategies included:• Worked with all levels of government (municipal, provincial, federal)• Regularly held in-person meetings and briefings throughout course of

project• Developed briefing material- PowerPoint presentations, Early Findings

reports• Used key messages to ensure consistent and focused messaging• Resulted in strong government engagement in At Home project sites

and ongoing policy change

Focusing the Frame

- 1 and 3 pagers, summaries, early findings

- media, community events, social media

- Advisory panels, collaboration, meetings

A Participant Photography Project

My Kitchen__________________________I love to cook. On the street it's hard to get a decent meal. Now my fridge is full and I can choose what and when I eat. I'm so much healthier now, and I get so much satisfaction in doing something I love to do. I can even invite people over for a meal. Beats the hell out of a food kitchen.

View from my Window________________________

I am so grateful for this view. I am also grateful to not have anyone able to knock on my

window at all hours of the night. I have some sense of safe.

Focusing the Frame

- 1 and 3 pagers, summaries, early findings

- media, community events, social media

- Advisory panels, collaboration, meetings

A Participant Photography Project

Strings____________________

The name “chaos theory” comes from the fact that the systems that the theory

describes are apparently disordered. But Chaos Theory is really about finding the

underlying order in seemingly random data.

The “butterfly effect” is a sensitive dependence on initial conditions. Just a small change in the initial conditions can

drastically change the long term behaviour of a system...

I am the system

Small Group Discussion Questions• How can your communities incorporate evaluation into your Housing

First programs?• What knowledge transfer strategies could you use with policy-makers

in your jurisdictions to promote the adoption of Housing First