1

1

DATA IS/PROVOKES/ALLOWS…

F E B R U A R Y 2 7 , 2 0 1 79 : 0 0 A M – 1 2 : 0 0 P M W I S D

2016-2018 PROGRAM OPERATIONS GRANTEE CONVENING

2

AGENDA

• Networking & Data Walk• Welcome and Introductions• Using Data for Learning• FY17 Learning Community• Breakouts: Workshop the Case Studies• Reflection, Next Steps & Networking

3

WELCOME

• COFU Team Introductions- Pam

• Update: 6-month program reports- Elisabeth

• Boundary Spanning Issues of Note- Mercedes

• Affordable Care Act• Changes to SNAP • Immigrant Rights• Transgender Rights

4

3

DATA WALK

•Questions•Reactions•Reflections

5

CASE STUDIES: USING DATA FOR INNOVATION

•Ozone & Corner Health Center•Community Action Network•Food Gatherers•Avalon Housing

Bridget 6

4

7

SBIRT FOR YOUTH!

• Douglas Manigault III, Grants and Evaluation Director, Ozone House

• Raina LaGrand, Health Coach, The Corner Health Center

AGENDA

• Why do this project?• Overview of SBIRT, the CRAFFT, and the

collaboration• Data Collection

• What data we have• Preliminary findings• How the results have been used

• Lessons Learned• Accomplishments and challenges

5

WHY THIS PROJECT?

• Opioid epidemic• Limited data:

• Most data on local youth collected in schools• SBIRT with youth delivered mostly in physician’s

offices• (Often) not a priority for youth facing complex

trauma, abuse, homelessness, acute health crises

• Early intervention • Integrated health

SCREENING, BRIEF INTERVENTION, & REFERRAL TO TX

6

CRAFFT TOOL

THE COLLAB

• Collect and provide essential localized data from high risk youth ages 12-25, who visit the Corner and Ozone’s Drop-In Center

• Provide services that foster integration of mental health, substance use, and primary care services

• Recovery oriented goals:• Intervening early• Supporting recovery• Integrating substance use services within primary care

settings

7

NOW, THE DATA…

THE DATA WE HAVE…

8

PRELIMINARY FINDINGS…

• The average CRAFFT score is ~2.21 for just over 192 youth.

• We screen and serve majority African American youth in this program (54%), in comparison to 34% White youth.

• We screen and serve majority female youth in the program (62%), in comparison to 38% male youth. No youth identified as transgender.

• Over 50% of the youth who receive a brief intervention come back for a second session.

UTILIZING THE DATA…

• Advocacy efforts about youth needs in Washtenaw County

• Corner facilitated faster appointments for participants to see Addiction Medicine physician

• Inform skills groups• Real Talk Substance Use Groups (Drop-In Center)• Orange Cards (Drop-In Center)• What then… (Corner Health)

• Outreach• Ypsilanti Community High School’s Eagles Nest Program• Dawn Farm• Growing Hope• Student Advocacy Center• Juvenile Justice System

9

LESSONS LEARNED…

• Accomplishments• There is now some data on this issue for

Washtenaw County youth

• Challenges• Two different data collection methods• Support for substance abuse treatment for

youth

QUESTIONS?

Douglas Manigault III, Grants and Evaluation Director,Ozone House

Raina LaGrand, Health Coach, The Corner Health Center

10

CAN:A Logic

Model Lens &

Evaluation Strategies

CAN’s 3 Pillars-i.e. Logic Model Goals

Educating Youth & Children

Goal: Prepare youth to fulfill their academic potential

and become successful, self-sufficient adults.

Stabilizing Families

Goal: Assist families in

meeting their basic needs and

create better futures for

themselves.

Building Strong Communities

Goal: Create and maintain clean,

safe, and supportive

neighborhoods where families

can thrive.

11

Built Like a Brick House-i.e. Logic Model Inputs

What we are made of…in addition to passion & determination of course

6 Full-Time Staff ~40 Seasonal, Part-Time, and/or

Work Study Staff 12+ Interns (MSW, UROP, BSW,

Public Health) Per Year 1200+ Volunteers 11 Full Time AmeriCorps Vistas 15+ AmeriCorps Summer

Associates

CAN Summary of Programs & Services-i.e. Logic Model Input Activities

Food Distributions (over $740,000 worth of food resources per year in collaboration with Food Gatherers.After School Program (tutoring, meals, life skill enrichment, mentoring, truancy remediation)Summer Camp (tutoring, meals, life skill enrichment, recreation)CAN Art and DesignYouthWorks (soft skills training, internships, work stipends)Community Events (Back to School BBQ, Thanksgiving potlucks, holiday party, etc)

A2 ExpeditionsCommunity MeetingsCollaborators (Food Gatherers, National Kidney Foundation, SLATE, GLAAM, AA, NA, WIC. CSS, Girl Scouts, A2 Reskilling, UM SSW, UM College of Pharmacy, UM School of Public Health, UM Law, EMU SSW, WCC Human Services, Foster Grandparent, Toledo Zoo, and so many more.)Internet AccessComputer AccessHoliday GiftsTurkey BasketsSchool Supplies…And the list goes on and on.

12

CAN Food Security Outputsi.e. Logic Model Outputs

Food and Meal Programs

13

CAN ASP Outcomesi.e. Logic Model Outcomes (Short & Long Term)

Efforts to Outcomes- After School Programs

CAN Evaluation Strategies

Youth Matrix• Created in 2010 and based on Arizona Self-Sufficiency Matrix.

• Undergoes intensive analysis including thesis review, analysis for statistical significance, and multi-year UM UROP project reviews.

• Consists of 9 developmental domains with research supported indicators on youth academic achievement and post-secondary success

• Influenced by ACEs study among other key research

Domains• Academic Performance

• School Behavior

• School Attendance

• Homework Completion

• ASP Involvement

• Role of Education

• Perceived Social Support (Peer Relations)

• Perceived Parental Support

• Child Safety

14

CAN Elementary Youth Matrix

Training Video

CAN Teen (MS & HS) Youth

Matrix Training Video

CAN Evaluation Strategies Cont’d

Collecting Data:• Try to use flexible/versatile data fields (ex. use date of birth vs age)

even when a funder/stakeholder requests reports on “age brackets.”

• Leverage technology when feasible and recognize when low-tech is the best tech.

• Align data collection systems to reduce duplication of effort and simplify analysis.

• Remember to communicate results to your staff and stakeholders. Highlight their accomplishments!

15

Other CAN Evaluation Strategies

• Most Significant Change• Focus Groups• Satisfaction Surveys• Yelping Your Programs and Services

Evaluation Soapbox Moment

Assumption of Client Performance w/ No Intervention

Common Funder/Stakeholder Assumption of Progress

Often Realistic Client Performance Trajectory w/No Intervention

Real Success w/Intervention

16

Questions

17

Contact Information

18

How we use data

Case study – Healthy Pantry Conversion Project1. What data did we have?2. What did we do?3. How did we set up our evaluation?4. How have we used the results?5. Lessons learned

19

What did we already know?

1. Client Produce Consumption Survey- Eating 2 c Fruit and Veg

(nearly 5 c recommended)2. Nearly all (92%) FG Partners distribute produce3. Clients could still take more produce and other healthy foods

Test Run

• In Food Gatherers Warehouse Pantry (for agencies) we tripled the amount of produce distributed by using nudge strategies

20

Healthy Pantry Conversion Project

Strategies- Encourage Healthy

Food Selection- “Nudges”- Unlimited produce

- Stock a healthy pantry- Indirect Nutrition

Education- Shelf talkers, recipes

Program Evaluation

• Collect client feedback • “It makes me feel like you care about me. You make the food look good and it makes me feel good about the food I pick.” – an SOS Community Services client.

21

Program Evaluation

• Track pounds of produce and healthy foods distributed

• Depending on pantry design and capacity for growth, partners saw varying increases in produce distributed

Increase

A 36%

B 142%

C 33%

D 5%

E 109%

F 3%

Phase 2

• Work with additional partners to implement Healthy Pantry strategies

• Expand program to pilot intervention at mobile distributions (temporary food distribution locations)

22

Lessons Learned

• What data already exist? (secondary data or primary data collected for another purpose)

• What do we know we can affect through intervention or program?

• How can we measure what we think we can accomplish?

• What other information do we need to have in order to interpret the evaluation?

Thank you!

Markell MillerDirector of Community Food Programsmarkell@foodgatherers.org

Healthy Pantry Conversion ProjectShaira DayaNutrition Projects Managersharia@foodgatherers.org

23

24

Daily Data Quality

“Kaizen” (PQI/Quality Assurance)

Line of Sight• Examples:

Case note challenge Dashboard Peer review process IGoR (reviewed by BLT,

board, all staff – leadership updates to all staff)

Avalon facilitates the work of its front-line staff through continuously working to make our tools more “user friendly” through making the many necessary (but often time-consuming) data entry tasks more simple and automated.

Foundation Automation

Daily (De-Identified) DQ Example 1:

We keep massive amounts of data. Certain staff, however, don’t need to access all of it. Avalon’s Eval staff build automated lookups and list builders that show staff only the information that is relevant to them.

Automation Automation

25

Data on Demand

Avalon’s use of secure online data management tools has allowed us to export live aggregate information such as current client demographics for instant access for staff who need this information for communication, reporting, etc.

Live Data: Instant Information:

Data on Demand

Linking information between different teams and departments helps us to produce products like our Data Dashboard while also letting those who enter data focus more on only the quality of the information that is relevant to them.

Unique, yet United: The “Beehive” Model

Results in…

This…

26

Reporting

In most cases, the information staff are required to report on already exists. Instead of requiring redundant re-entry, staff can instantly import data by using clients’ unique identification numbers.

Remove Redundancy The “Beehive” Model

Not any more!

This time-consuming report is empty…

FUSE Data

3rd party evaluation

1. Baseline and 1 year follow up surveys

2. Site visits (process evaluation)

3. Claims data

4. Data tracking sheets

27

PDCA Cycles

Example: Learn from other sites, review data tracking sheets,

and review IGoR : high incidence of medical emergencies, poor health outcomes

Use data for PA2 grant Provide home based primary care with Packard

Health = Dr. Ravi has seen 91 patients since August: 73% ER utilization yr prior to engagement 40% ER utilization after initial engagement 68% decrease in inappropriate ER utilization

Con

nec

ticu

t

San

Fra

nci

sco

Los

An

gele

s

Was

hte

naw

Cou

nty

Tot

al

Demographics of All Clients

Average Age 46.5 51.8 51.5 47.7 49.7

Gender

Female 31% 29% 37% 29% 30%

Male 69% 67% 63% 71% 62%

Male-to-Female 0% 5% 0% 0% 2%

Race/ethnicity

African-American 31% 43% 39% 37% 38%

Latino or Hispanic 21% 9% 15% 2% 12%

White 48% 33% 39% 56% 42%

Asian 0% 2% 2% 1% 1%

Native American 0% 0% 3% 0% 0%

Hawaiian/Pacific Islander 0% 1% 0% 0% 0%

Multi-ethnic/Multi-racial 0% 5% 1% 1% 2%

Other/Declined/NA 0% 7% 2% 4% 4%

% Veteran 3% 4% 4% 1% 3%

FUSE Demographics

28

Health Indicators (1 year post housing)

60% rate health as fair/poor

1 in 5 experience a medical problem every day in past month

Half report difficulty walking or climbing stairs

Con

nec

ticu

t

San

Fran

cisc

o

Los

An

gele

s

Was

hte

naw

Cou

nty

Tot

al

Health conditions of clients

Morbidity

% with chronic health condition 89% 92% 100% 100% 93%

% with mental health issues 86% 76% 83% 79% 80%

% with substance use 86% 79% 70% 83% 80%

% with chronic health condition and substance use (no MH) 11% 16% 12% 14% 13%

% with chronic health condition and mental health issues (no SU) 11% 15% 20% 11% 14%

% substance use and mental health issues (no chronic conditions) 6% 5% 7% 4% 5%

% tri-morbid (all three indicated) 64% 55% 33% 61% 55%

Health Conditions

29

Housing RetentionAs of January 2016

Con

nec

ticu

t

San

Fran

cisc

o

Los

An

gele

s

Was

hten

aw

Cou

nty

Tot

al

Clients in Housing

# Ever housed 176 244 121 161 702

# Currently housed 144 172 96 123 535

# Newly housed in last 6 months 7 12 13 14 46

# Left housing (any reason) 32 72 25 38 167

# Left housing (negative reason) 20 26 10 20 76

Retention rate (using negative exits) 88% 87% 91% 86% 88%

Reason for Leaving:

Became homeless 2 2 1 0 5

Deceased 4 25 6 11 46

Evicted/Avoid eviction 11 14 6 9 40

Hospitalized/Higher level of care 4 6 0 2 12

Incarcerated 3 4 3 9 19

Moved in with family 2 15 3 2 22

Moved to independent living 0 5 1 2 8

Other/Unknown 6 1 5 3 15% Exits Negative outcomes (homeless/incarcerated/hospitalized/ evicted) 63% 36% 40% 53% 46%

Access to Health Care

ED as main source of care:

58%

Needed but could not find a dentist:

76%

ED as main source of care:

31%

Needed but could not find a dentist:

32%

Baseline 1 year follow up

30

Mental Health and Social Support

Frequent Loneliness:

48%

Feeling Life is Unstable:

57%

Frequent Loneliness:

34%

Feeling Life is Unstable:

12%

Baseline 1 year follow up

Mental Health and Social Support

Feeling Stable About One’s Life:

30%

Life is Organized:

24%

Feeling Stable About One’s Life:

61%

Life is Organized:

75%

Baseline 1 year follow up

31

Client Satisfaction

Nearly all clients reported that the program met all or most needs, and that they would recommend the program to friends. High levels of satisfaction were reported with:

Ease contacting social worker

Choice of when to see social worker

Choice over whether or not to take meds

Proximity to shopping, public transport, etc.

Independence in daily life

Condition and affordability of the apartment

Cost Impact

Statistically significant savings and reduced hospitalizations were found with the highest cost individuals

Estimated Cost impacts varied with the level of pre-period costs (pre-period is the year prior to randomization)

32

Plan for Further Analysis

Extend the follow up to see whether impacts on health care utilization grow over time (that is, after individuals are housed, stabilized, and engaged in primary care). Note: The period from someone being placed into the intervention group and getting housed was long for Washtenaw County (up to 9 months in some cases). Sequestration

Look at control group in MI with systematic approach to addressing homelessness shift at program start up with the goal of assessing for “treatment difference” between control and intervention. Zero 2016, Coordinated Assessment

Environmental Factors

•Limited housing availability

•Limited access to medical detox and other substance use treatment programs (ignoring abstinence only programs)

•Limited access to mental health services if substance use is a primary diagnosis

•No medical staff on team during evaluation period

33

How does data tell a story?

Regression to the mean among high users of health care is common. When pre-post studies lack a strong comparison group, it may be inappropriately attributed to intervention.

There is more room to reduce costs for the most costly than for people with lower service use who account for the majority of chronically homeless people – what does that tell us about targeting?

Unintended consequences of sub population focus - does staking the future of Housing First on the expectation that it will save money undermine efforts to deliver an effective intervention to the majority of the population it’s intended to serve?

BREAKOUT SESSIONS

•Ozone & Corner Health Center:Seminar 2

•Community Action Network: Seminar 3

•Food Gatherers: Seminar 4•Avalon Housing: Seminar 5

66

34

REFLECTION, NEXT STEPS AND NETWORKING

• Based on what you heard today, what will you carry forward for further conversation within your organizations?

• What additional secondary data do you want to see to help understand what’s happening within your system?

• What’s one thing we can do/get behind in the next 12 months to better use data for learning as a community?

Thank you!

67