1

CASE STUDIES:USING DATA FOR INNOVATION

•Ozone & Corner Health Center

•Community Action Network

•Food Gatherers

•Avalon Housing

6

7

SBIRT FOR YOUTH!

• Douglas Manigault III, Grants andEvaluation Director, Ozone House

• Raina LaGrand, Health Coach, The CornerHealth Center

2

AGENDA

• Why do this project?• Overview of SBIRT, the CRAFFT, and the

collaboration• Data Collection

• What data we have• Preliminary findings• How the results have been used

• Lessons Learned• Accomplishments and challenges

8

WHY THIS PROJECT?

• Opioid epidemic• Limited data:

• Most data on local youth collected in schools• SBIRT with youth delivered mostly in physician’s

offices

• (Often) not a priority for youth facing complextrauma, abuse, homelessness, acute healthcrises

• Early intervention• Integrated health

9

3

SCREENING, BRIEF INTERVENTION, &

REFERRAL TO TX

10

CRAFFT TOOL

11

4

THE COLLAB

• Collect and provide essential localized datafrom high risk youth ages 12-25, who visit theCorner and Ozone’s Drop-In Center

• Provide services that foster integration ofmental health, substance use, and primary careservices

• Recovery oriented goals:• Intervening early

• Supporting recovery

• Integrating substance use services within primary caresettings

12

NOW, THE DATA…

5

THE DATA WE HAVE…

14

PRELIMINARY FINDINGS…

• The average CRAFFT score is ~2.21 for just over 192youth.

• We screen and serve majority African Americanyouth in this program (54%), in comparison to 34%White youth.

• We screen and serve majority female youth in theprogram (62%), in comparison to 38% male youth.No youth identified as transgender.

• Over 50% of the youth who receive a briefintervention come back for a second session.

15

6

UTILIZING THE DATA…

• Advocacy efforts about youth needs in WashtenawCounty

• Corner facilitated faster appointments for participants tosee Addiction Medicine physician

• Inform skills groups• Real Talk Substance Use Groups (Drop-In Center)• Orange Cards (Drop-In Center)• What then… (Corner Health)

• Outreach• Ypsilanti Community High School’s Eagles Nest Program• Dawn Farm• Growing Hope• Student Advocacy Center• Juvenile Justice System

16

LESSONS LEARNED…

• Accomplishments

• There is now some data on this issue forWashtenaw County youth

• Challenges

• Two different data collection methods

• Support for substance abuse treatment foryouth

17

7

QUESTIONS?

Douglas Manigault III, Grants and Evaluation Director,

Ozone House

Raina LaGrand, Health Coach, The Corner Health Center

CAN:

A LogicModel Lens

&EvaluationStrategies

8

CAN’s 3 Pillars-i.e. Logic Model Goals

Educating Youth& Children

Goal: Prepareyouth to fulfill theiracademic potential

and becomesuccessful, self-sufficient adults.

StabilizingFamilies

Goal: Assistfamilies in

meeting theirbasic needs and

create betterfutures for

themselves.

Building StrongCommunities

Goal: Create andmaintain clean,

safe, andsupportive

neighborhoodswhere families

can thrive.

20

Built Like a Brick House-i.e. Logic Model Inputs

What we are made of…in addition to passion & determination of course

6 Full-Time Staff ~40 Seasonal, Part-Time, and/or

Work Study Staff 12+ Interns (MSW, UROP, BSW,

Public Health) Per Year 1200+ Volunteers 11 Full Time AmeriCorps Vistas 15+ AmeriCorps Summer

Associates

21

9

CAN Summary of Programs & Services-i.e. Logic Model Input Activities

Food Distributions (over $740,000 worth offood resources per year in collaboration withFood Gatherers.After School Program (tutoring, meals, lifeskill enrichment, mentoring, truancyremediation)Summer Camp (tutoring, meals, life skillenrichment, recreation)CAN Art and DesignYouthWorks (soft skills training, internships,work stipends)Community Events (Back to School BBQ,Thanksgiving potlucks, holiday party, etc)

A2 ExpeditionsCommunity MeetingsCollaborators (Food Gatherers, NationalKidney Foundation, SLATE, GLAAM, AA, NA,WIC. CSS, Girl Scouts, A2 Reskilling, UMSSW, UM College of Pharmacy, UM School ofPublic Health, UM Law, EMU SSW, WCCHuman Services, Foster Grandparent, ToledoZoo, and so many more.)Internet AccessComputer AccessHoliday GiftsTurkey BasketsSchool Supplies…And the list goes on and on.

22

CAN Food Security Outputsi.e. Logic Model Outputs

Food and Meal Programs

23

10

24

CAN ASP Outcomesi.e. Logic Model Outcomes (Short & Long Term)

Efforts to Outcomes- After School Programs

25

11

CAN Evaluation Strategies

Youth Matrix• Created in 2010 and based on Arizona Self-Sufficiency Matrix.

• Undergoes intensive analysis including thesis review, analysis for statisticalsignificance, and multi-year UM UROP project reviews.

• Consists of 9 developmental domains with research supported indicators on youthacademic achievement and post-secondary success

• Influenced by ACEs study among other key research

Domains• Academic Performance

• School Behavior

• School Attendance

• Homework Completion

• ASP Involvement

• Role of Education

• Perceived Social Support(Peer Relations)

• Perceived ParentalSupport

• Child Safety

26

CANElementary

Youth MatrixTraining Video

CAN Teen (MS& HS) Youth

Matrix TrainingVideo

27

12

CAN Evaluation Strategies Cont’d

Collecting Data:

• Try to use flexible/versatile data fields (ex. use date of birth vs age)even when a funder/stakeholder requests reports on “age brackets.”

• Leverage technology when feasible and recognize when low-tech isthe best tech.

• Align data collection systems to reduce duplication of effort andsimplify analysis.

• Remember to communicate results to your staff and stakeholders.Highlight their accomplishments!

28

Other CAN Evaluation Strategies

• Most Significant Change

• Focus Groups

• Satisfaction Surveys

• Yelping Your Programs and Services

29

13

Evaluation Soapbox Moment

Assumption of Client Performancew/ No Intervention

Common Funder/StakeholderAssumption of Progress

Often Realistic Client PerformanceTrajectory w/No Intervention

Real Success w/Intervention

30

Questions

31

14

32

Contact Information

33

15

34

16

How we use data

Case study – Healthy Pantry Conversion Project

1. What data did we have?

2. What did we do?

3. How did we set up our evaluation?

4. How have we used the results?

5. Lessons learned

36

What did we already know?

1. Client ProduceConsumption Survey

- Eating 2 c Fruit andVeg

(nearly 5 c recommended)

2. Nearly all (92%) FG Partnersdistribute produce

3. Clients could still take moreproduce and other healthyfoods

37

17

Test Run

• In Food Gatherers Warehouse Pantry (foragencies) we tripled the amount of producedistributed by using nudge strategies

38

Healthy Pantry Conversion Project

Strategies

- Encourage HealthyFood Selection

- “Nudges”

- Unlimited produce

- Stock a healthy pantry

- Indirect NutritionEducation

- Shelf talkers, recipes

39

18

Program Evaluation

• Collect client feedback • “It makes me feel likeyou care about me.Youmake the food lookgood and it makes mefeel good about thefood I pick.” – an SOSCommunity Servicesclient.

40

Program Evaluation

• Track pounds ofproduce and healthyfoods distributed

• Depending on pantrydesign and capacity forgrowth, partners sawvarying increases inproduce distributed

Increase

A 36%

B 142%

C 33%

D 5%

E 109%

F 3%41

19

Phase 2

• Work with additionalpartners to implementHealthy Pantrystrategies

• Expand program topilot intervention atmobile distributions(temporary fooddistribution locations)

42

Lessons Learned

• What data already exist? (secondary data orprimary data collected for another purpose)

• What do we know we can affect throughintervention or program?

• How can we measure what we think we canaccomplish?

• What other information do we need to havein order to interpret the evaluation?

43

20

Thank you!

Markell MillerDirector of Community Food Programs

markell@foodgatherers.org

Healthy PantryConversion Project

Shaira DayaNutrition Projects Manager

sharia@foodgatherers.org

44

21

46

Daily Data Quality

“Kaizen” (PQI/Quality Assurance)

Line of Sight

• Examples: Case note challenge

Dashboard

Peer review process

IGoR (reviewed by BLT,board, all staff – leadershipupdates to all staff)

Avalon facilitates the workof its front-line staffthrough continuouslyworking to make our toolsmore “user friendly”through making the manynecessary (but often time-consuming) data entrytasks more simple andautomated.

Foundation Automation

47

22

Daily (De-Identified) DQ Example 1:

We keep massiveamounts of data.Certain staff, however,don’t need to access allof it. Avalon’s Eval staffbuild automated lookupsand list builders thatshow staff only theinformation that isrelevant to them.

Automation Automation

48

Data on Demand

Avalon’s use of secureonline data managementtools has allowed us toexport live aggregateinformation such as currentclient demographics forinstant access for staffwho need this informationfor communication,reporting, etc.

Live Data: Instant Information:

49

23

Data on Demand

Linking informationbetween different teamsand departments helpsus to produce productslike our Data Dashboardwhile also letting thosewho enter data focusmore on only the qualityof the information that isrelevant to them.

Unique, yet United: The “Beehive” Model

Results in…

This…

50

Reporting

In most cases, theinformation staff arerequired to report onalready exists. Insteadof requiring redundantre-entry, staff caninstantly import data byusing clients’ uniqueidentification numbers.

Remove Redundancy The “Beehive” Model

Not any more!

This time-consuming report is empty…

51

24

FUSE Data

3rd party evaluation

1. Baseline and 1 year follow up surveys

2. Site visits (process evaluation)

3. Claims data

4. Data tracking sheets

52

PDCA Cycles

Example:

Learn from other sites, review data tracking sheets,and review IGoR : high incidence of medicalemergencies, poor health outcomes

Use data for PA2 grant

Provide home based primary care with PackardHealth = Dr. Ravi has seen 91 patients since August: 73% ER utilization yr prior to engagement

40% ER utilization after initial engagement

68% decrease in inappropriate ER utilization

53

25

Co

nn

ecti

cut

San

Fran

cisc

o

Lo

sA

ng

eles

Was

hte

naw

Co

un

ty

To

tal

Demographics of All Clients

Average Age 46.5 51.8 51.5 47.7 49.7

Gender

Female 31% 29% 37% 29% 30%

Male 69% 67% 63% 71% 62%

Male-to-Female 0% 5% 0% 0% 2%

Race/ethnicity

African-American 31% 43% 39% 37% 38%

Latino or Hispanic 21% 9% 15% 2% 12%

White 48% 33% 39% 56% 42%

Asian 0% 2% 2% 1% 1%

Native American 0% 0% 3% 0% 0%

Hawaiian/Pacific Islander 0% 1% 0% 0% 0%

Multi-ethnic/Multi-racial 0% 5% 1% 1% 2%

Other/Declined/NA 0% 7% 2% 4% 4%

% Veteran 3% 4% 4% 1% 3%

FUSE Demographics54

Health Indicators (1 year post housing)

60% rate health as fair/poor

1 in 5 experience a medical problem every day inpast month

Half report difficulty walking or climbing stairs

55

26

Co

nn

ecti

cut

San

Fran

cisc

o

Lo

sA

ng

eles

Was

hte

naw

Co

un

ty

To

tal

Health conditions of clients

Morbidity

% with chronic health condition 89% 92% 100% 100% 93%

% with mental health issues 86% 76% 83% 79% 80%

% with substance use 86% 79% 70% 83% 80%

% with chronic health condition and substance use (noMH) 11% 16% 12% 14% 13%

% with chronic health condition and mental healthissues (no SU) 11% 15% 20% 11% 14%

% substance use and mental health issues (no chronicconditions) 6% 5% 7% 4% 5%

% tri-morbid (all three indicated) 64% 55% 33% 61% 55%

Health Conditions56

Housing Retention

As of January 2016

Co

nn

ecti

cut

San

Fran

cisc

o

Lo

sA

ng

eles

Was

hte

naw

Co

un

ty

To

tal

Clients in Housing

# Ever housed 176 244 121 161 702

# Currently housed 144 172 96 123 535

# Newly housed in last 6 months 7 12 13 14 46

# Left housing (any reason) 32 72 25 38 167

# Left housing (negative reason) 20 26 10 20 76

Retention rate (using negative exits) 88% 87% 91% 86% 88%

Reason for Leaving:

Became homeless 2 2 1 0 5

Deceased 4 25 6 11 46

Evicted/Avoid eviction 11 14 6 9 40

Hospitalized/Higher level of care 4 6 0 2 12

Incarcerated 3 4 3 9 19

Moved in with family 2 15 3 2 22

Moved to independent living 0 5 1 2 8

Other/Unknown 6 1 5 3 15% Exits Negative outcomes(homeless/incarcerated/hospitalized/evicted) 63% 36% 40% 53% 46%

57

27

Access to Health Care

ED as main source ofcare:

58%

Needed but could notfind a dentist:

76%

ED as main source ofcare:

31%

Needed but could notfind a dentist:

32%

Baseline 1 year follow up

58

Mental Health and Social Support

Frequent Loneliness:

48%

Feeling Life isUnstable:

57%

Frequent Loneliness:

34%

Feeling Life isUnstable:

12%

Baseline 1 year follow up

59

28

Mental Health and Social Support

Feeling Stable AboutOne’s Life:

30%

Life is Organized:

24%

Feeling Stable AboutOne’s Life:

61%

Life is Organized:

75%

Baseline 1 year follow up

60

Client Satisfaction

Nearly all clients reported that the program met all ormost needs, and that they would recommend the programto friends. High levels of satisfaction were reported with:

Ease contacting social worker

Choice of when to see social worker

Choice over whether or not to take meds

Proximity to shopping, public transport, etc.

Independence in daily life

Condition and affordability of the apartment

61

29

Cost Impact

Statistically significant savings and reducedhospitalizations were found with the highest costindividuals

Estimated Cost impacts varied with the level of pre-period costs (pre-period is the year prior torandomization)

62

Plan for Further Analysis

Extend the follow up to see whether impacts on health careutilization grow over time (that is, after individuals arehoused, stabilized, and engaged in primary care). Note: Theperiod from someone being placed into the intervention groupand getting housed was long for Washtenaw County (up to 9months in some cases). Sequestration

Look at control group in MI with systematic approach toaddressing homelessness shift at program start up with thegoal of assessing for “treatment difference” between controland intervention. Zero 2016, Coordinated Assessment

63

30

Environmental Factors

•Limited housing availability

•Limited access to medical detox and othersubstance use treatment programs (ignoringabstinence only programs)

•Limited access to mental health services if substanceuse is a primary diagnosis

•No medical staff on team during evaluation period

64

How does data tell a story?

Regression to the mean among high users of health care iscommon. When pre-post studies lack a strong comparisongroup, it may be inappropriately attributed to intervention.

There is more room to reduce costs for the most costly thanfor people with lower service use who account for themajority of chronically homeless people – what does thattell us about targeting?

Unintended consequences of sub population focus - doesstaking the future of Housing First on the expectation that itwill save money undermine efforts to deliver an effectiveintervention to the majority of the population it’s intended toserve?

65