measuring program outcomes boys & girls clubs of america
TRANSCRIPT
Measuring Program Outcomes
Boys & Girls Clubs of America
Workshop Climate
• Hours
• Breaks
• Lunch
• Cell phones
• Rest Rooms
• Learning Credit
• Other?
Climate - Introductions
• Interview:– Name– Club– Years working with youth– Role in Club– Role in measurement of program
Expectations
• At your tables:– Appoint a note keeper– Identify your expectations for this workshop
Training Objectives Be aware of the board, growing interest in outcomes; Understand the uses and limitations of outcome
measurement; Be able to identify appropriate program outcomes and
outcome indicators; Understand the basic steps in creating and using logic
models; Learn the benefits and how to utilize the Outcomes
Measurement Tool Kit. Develop a plan to begin outcome measurement at your
Club.
BGCA’s Mission Statement
To inspire and enable all young people, especially those from disadvantaged
circumstances, to realize their full potential as productive,
responsible and caring citizens.
What Makes Clubs Great?
• In groups of three:– Discuss what makes Clubs GREAT
• What would you do if someone asked you to “prove it?”
• How would you back up the statement “We are great?”
• Measuring Outcomes is an effective backup. Benefits include
Benefits of Measurement• Determines effectiveness of services and
programs
• Becomes resource development tool
• Increases awareness of Club
• Enables Clubs to respond to increased levels of accountability (United Way, public dollars, parents, funders, etc.)
Overview of Topic 2: The Logic Model
• Terms and Definitions – A common Language
• The Logic Model – A look at structure• Outcomes – The fourth column• Logic Model Shuffle – The if. . .
than. . . exercise• Building a Logic Model – putting it all
together
Logic Model Terminology
• Inputs - Resources a program uses to achieve program objectives. • Activities - What a program does with inputs. This is our program.• Outputs - Products of the program (activities), which can be counted.• Outcomes – Benefits for participants during or after involvement with
the program. • Initial Outcomes – First benefits or changes participants experience.• Intermediate Outcomes – Link the initial outcomes to the longer-
term outcomes desired for participants.• Long-Term Outcomes – The ultimate goals of the program. • Outcome Indicators - The specific information collected to track a
program’s success in achieving the targeted outcome.• Outcome Targets – A numerical objective for a program’s level of
achievement.
Handout: Selected Terms
Logic Model – What do we now measure?
• What do you measure at your club?Flip Chart: Fit in outline of Logic Model
Logic ModelInputs Activities Outputs Outcomes
Initial:
Intermediate:
Long Term:
Logic Model Handouts
Terminology
Program Outcome Model
Inputs Activities Outputs Outcomes
Resources Services Products Benefits for Participants
Money Staff
Volunteers Equipment and Supplies
CConstraints Laws Regulations
Training Education Counseling Mentoring Internships
Classes Taught Counseling sessions conducted Educational Materials Distributed Hours of Service Delivered Participants Served
New Knowledge Increased Skills Changed Attitudes or Values Modified Behavior Improved Condition Altered Status
Handout: Program Outcome Model
Outcomes - are what happens to members involved in the program – how members are better off.
They include changes in the following:
• Knowledge
• Skills
• Attitudes
• Values
• Behavior
• Condition
• Status
BGCA’s Youth Development Outcomes
• Positive Self-Identity• Community and Civic
Involvement• Health and Well-Being• Moral Compass• Competencies:
Educational Social Emotional Cultural Employment
Handout: BGCA Youth Development Outcomes
Program Outcome Criteria
• For each outcome:– It is reasonable to think the program can influence the outcome
in a non-trivial way, even though it can’t control it.– Would measurement of the outcome help identify program
successes and pinpoint problems?– Will the program’s various “publics” accept as a valid outcome
of the program?
• For the set of outcomes:– Do they reflect the program logic – the chain of changes
program outputs are intended to set in motion for participants?– Do the longer-term outcomes represent meaningful benefits or
changes in participants’ conditions or quality of life?– Are potential negative outcomes identified?
Uses of Outcome Findings
• INTERNAL– Provides direction for
staff– Identify training
needs– Support strategic
planning– Suggest outcome
targets– Focus board
members’ attention on programmatic issues
• EXTERNAL– Recruit talented staff
and volunteers– Promote program to
potential participants and referral sources
– Identify partners for collaboration
– Enhance program’s public image
– Retain and increase funding
Work with the Logic Model
• Logic Model Shuffle– Review Start Smart Logic Model– Arrange Pieces of a Logic Model
• Build Logic Models:1. A Program you are familiar with2. A Core Program Area
Handouts: Conceptual Chain
What is the Tool Kit?
• Web-based Application
• Based on member survey administration
• Resource for documenting and assessing change
Development of The Tool Kit
• Created in 1998• Initial instrument was a
paper version • Contracted with Policy
Studies Associates• 12 Organizations piloted
in Fall 2003
What’s New in the Tool Kit
• Reliability of Questions Improved
• Scoring is Simplified
• Charts Available with a Click of a button
Handout: Outline of Outcome Measurment Tool Kit
Handout: Outcome Indicator Definitions
Three Distinct Surveys
• Elementary School
• Middle School
• High School
B&GCA Outcomes Survey Summary Graph for All Boys & Girls Clubs - Fall 2003 Pilot Data
25
23
22
24
27
24
26
20
22
22
25
49
53
51
42
50
47
50
58
50
51
49
26
24
27
34
24
30
24
23
28
27
27
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
Club Benefits
Club Connections
Activities Participation
Risk Avoidance
Educational Commitment
Employment Competence
Caring Behavior
Tolerance
Moral Compass
Leadership
Technology Connection
Percent of Youth Surveyed
Room to Grow Doing Fine Doing Great
Reporting the Results• Compare the Club activities against
scores.
• Convene staff to help decide how data should be analyzed and reported.
• Create an action plan.
Use the 75/25 Rule
A good rule of thumb is …•Have 75% of all survey takers respond in the “Doing Great” or “Doing Fine” categories.•Think about how to affect change if less than 25%
Bonus slide
Getting Started •Log on www.bgca.net and request an ID and password•Designate a lead Club staff•Provide Staff Training •Determine what information you want to learn
Sampling
•More Increases Reliability
•Target 30 Members Per Age Group
•Make Random Selections
•Just Get Started!
Consent
Youth younger than age 18 need parental consent. It’s the law! There are several means to secure consent:
• Passive
• Active
Two Consent Handouts
When Should I Survey?•Surveys should be conducted annually, possibly in the fall after enrollment has been stabilized.•Conduct surveys at the same time each year.•Surveys should be linked with the need for outcomes information.
Helpful Tips
•Have a survey administrator available to provide oral instructions•Have survey copies available for parents•Use lots of incentives•Have fun!
Target Audience
• Chief Professional Officers• Club Staff• Funders• Boards• Regional Service Directors
Other BGCA Evaluation Resources
• Commitment to Quality• Program Reviews• Peer Reviews• SOEs• Proven Results: A Compendium
B&GCA Assessment Tools
• Organization:– Standards of Organizational Effectiveness
(SOE)
• Club Experience:– Commitment to /Quality
• Youth Development Outcomes– Youth Development Outcome Measurement Kit
Getting Started
• Develop anOutcome Measurement Workgroup• Create plan for the Tool Kit• Identify programs to be measured• Build logic models for programs
– Manage expectations
• Create data collection strategies• Analyze and present results• Collect praise for job well done!
Responsibilities of Outcome Measurement Work Group
1. Create measurement strategy
2. Develop a timeline
3. Identify the outcomes to measure
4. Construct a program logic model when necessary
5. Identify outcome indicators
6. Prepare and test data collection instruments
7. Plan for and monitor a pilot process
8. Analyze and share results
9. Evaluate the results and make changes
Criteria for Choosing First Program for Outcome
Measurement• It has a recognizable and reasonably
defined mission and clientele.• It represents a substantial portion of the
Club’s activity.• Funders or others are interested in the
program’s results.• Program supervisors and staff are likely to
be supportive of an outcome measurement effort.
Reporting the Results
• Description of goals• Description of outcomes explored and Club’s
goals and strategies• Description of survey instruments, sample
process and approach to administration• Description of the sample and comparison
to overall characteristics of Club members• Description of results• Uses of results
Who Is Interested Tool Kit Results
• At your tables, brainstorm and list “What’s In It” for your assigned “interested party.”– Boards– CPO’s– Staff– Funders– B&GCA (RSD’s, Others?)– Others?
Wrap Up
• What was learned?
• How might you use it?
• What will each of you do:– Write what you will do:
• Within 30 days
• Within 90 days
• Share one of the things you will do.
Questions
