The New Outcomes System

Stage I: Adopting the Youth Outcomes Survey

AIM AgenciesJune 2009

Today’s workshop focus:

• Vision, goals, and components of BBBSA’s Outcomes System

• Development of the Youth Outcome Survey (YOS)

• Guidelines and tips to effectively implement the YOS

The vision…We stand for quality.

We will track the length, strength and outcomes of each Little in a consistent and accurate way so we may know how many successful matches we have in an agency and nationwide.

Our outcomes are central to our case for support.

We seek to communicate our results in effective and powerful terms.

The goals:

• To track current outcomes for each Little, each agency, and nationally.

• To use methodologies that are compelling to supporters, funders and researchers.

• To improve agency program performance and build our case for community support using outcomes.

The components:

Length • 6 and 12 month RR• average match length (AML)

Strength • Strength of Relationship (SoR)

Outcomes • Youth Outcomes Survey (YOS)• School records (future)

other quality processes?

• quality assurance ratings• match support compliance %

BBBSA’s Outcome System

What is the Youth Outcomes Survey?

• Built on the findings of our past P/PV studies that indicate specific areas where our work yields impact

• Designed by BBBSA in consultation with Rhodes, Grossman, and Herrera

• “Stronger” than POE

Why “stronger?”

• Pre/post and age cohort comparisons tell us more

• Different levels of impacts at different lengths?

• What makes some matches more successful than others?

• Continues to validate the methodologically sound and rigorous results from the P/PV studies

And the pilot results?

• Sound scales• Changed some questions• Results showed statistical significance

in key outcome areas• Positive feedback from all levels of

staff about feasibility of network-wide roll out

• Great recommendations for large-scale implementation

Summary of theYouth Outcomes Survey

• Academics– Scholastic Competence– Educational Expectations– Self-Assessment of Academic

Performance

• Relationships– Social Acceptance– Parental Trust

• Risky Behaviors– Attitudes Towards Risk

More about the YOS

• Pre-test (baseline surveys) and post-test (follow-up surveys)

• For youth ages 9 and up

• 32 question survey

• For both CB and SB matches

Measure Related Long-Term Outcomes

Rationale

Scholastic Competence

Academic achievement Increases in this scale often precede performance improvement. Higler levels of scholastic competence are associated with better mental health and high grades.

Educational Expectations

Academic achievement Youth’s academic expectations and plans have shown longitudinal associations with school performance and standardized test scores.

Self-Assessment of

Academic Performance

Academic Achievement Correlation with actual academic performance.

Social Acceptance

Academic achievement and delinquency avoidance

Improvements in peer relations have been linked to school achievement, improved conduct, and lower likelihood of dropping out.

Parental Trust Academic achievement and delinquency avoidance

Improved parental relationships are associated with improved grades and attendance and reduced substance abuse.

Attitudes Towards Risk

Delinquency avoidance Correlation with actual risk behaviors.

Scholastic Competence

I have trouble figuring out the answers in school.

I feel that I am just as smart as other kids.

I am very good at my schoolwork.

I’m pretty slow in finishing my school work.

I often forget what I learn.

I do very well at my class work.

Educational Expectations

How sure are you that you will finish high school?

How sure are you that you will go to college?

How sure are you that you will finish college?

Self-Assessment of Academic Performance

Mathematics

Reading or language arts

Social studies

Science

Social Acceptance

I am always doing things with a lot of kids.

I wish that more people my age liked me.

I find it hard to make friends

I would like to have a lot more friends.

I am popular with others my age.

I have a lot of friends.

Parental Trust

How often do I feel that my parents respect my feelings.

How often do I feel that my parents accept me as I am.

How often do I feel that when I’m angry about something, my parents try to be understanding.

Attitudes Towards Risk Using tobacco (cigarettes, cigars, smokeless or chewing tobacco)?

Taking drugs that aren’t given to them by a doctor or parent?

Drinking alcohol without their parents knowing?

Skipping school without permission?

Hitting someone because they didn’t like something they said or did?

Breaking rules in school?

Being late for school?

Plus…

• 2 truancy questions– In the past 30 days, have you been

absent from school?– In the past 30 days, have you been

late for school?

• 1 “special adult” question

But should we rely on youth responses??

• Widely accepted in research community

• Unique position of youth as most accurate source of data

• Potential for high response rates• Valuable and essential complement to

hard data• Positive indicators of accuracy

Administration Guidelines for Youth Outcomes Survey

• Who gets it?– All Littles ages 9 and over– Both CB and SB programs

• What about younger Littles?– Can use youth POE if needed

Administration Guidelines for Youth Outcomes Survey

• When do they get it?– Baseline Surveys: before match is

made (up to 30 days prior)– Follow-up Surveys:

• CB: at 12 month match anniversaries• SB: at end of school year

Administration Guidelines for Youth Outcomes Survey

• Who gives it?– BBBS staff – Never our Bigs!

• How should it be completed?– Preferably in person – then by phone– No e-Form functionality– Staff should read survey to Littles, even if

they know how to read!• Where does it go?

– In AIM

Consents

• Update your child application

• Passive consent form for all children/parents in current processing

What about the data?

• 2 methodologies:– Pre/post comparison– Age cohort comparison

• Value of “statistically significant” differences

Outcomes Scorecard

• Template provided

• AIM will help with data gathering

• Share/discuss with fund development team

Possible / Expected Program Implications?

• Survey fatigue? • Time needed to complete

baseline and follow up surveys• Need to train staff• How to expect higher response

rates?• Implications if results, at first,

aren’t as positive as POE?

Suggested Implementation

• Inform “up” about evolving outcome evaluation system

• Train staff • Update program manual to reflect use

of new forms and new procedures• Look out for continued webinars and

check out resources on Agency Connection