Why evaluate?Benefits of program evaluation

YD as EVAL: Webinar Agenda

• Why evaluate?• So what is evaluation?• How does evaluation fit youth development?• How can evaluation benefit a YD program?

• What resources are available to learn about evaluation?

What happened at the fair

What difference did the volunteer make?

Benefits of Informal Evaluation

• Building relationships• Gathering facts• Offering support, recognition• Troubleshooting problems• Offering guidance

Evaluation by the ‘net

What difference did it make?

Where was evaluation happening?

Where was evaluation happening?

Where was evaluation happening?

Where was evaluation happening?

Where –else-- was evaluation happening?

Benefits of Formal EvaluationCompleting Organizational Evaluation

• Targeted outcomes and tools• Efficient reporting process• Data for funding proposals• Feedback for improving programs

Who needs to know?

StakeholdersIndividuals and

groups with interest in a program

Rethinking EvaluationRevitalizing Programs

Ask the right questions…

…in a user-friendly way…

…and put the answers to good use…

So what is Evaluation?

EvaluationCritical examination of a program’s resources, activities, and outcomes to judge effectiveness, improve quality and results, and inform decisions. (Patton, 1997)

Evaluation Activities

• Assessment: Investigating needs and assets

• Monitoring: Tracking processes and progress

• Evaluation: Documenting quality and outcomes

Merit: Program Quality and Potential

Merit:Quality, Best Practices, potential to promote outcomes

•Overall: Traits that promote positive development(safety, support, skill-building, self-efficacy…

•Particular: Afterschool, Camps, Science, Horsemanship

Formative Evaluation: Is the program working as planned?Summative Evaluation: What program components contribute to positive outcomes

Worth: Program Outcome and Impact

WorthValue of a program, grounded in outcomes that accrue to targeted units (individuals, groups, organizations)

•Short-term: Knowledge, Attitudes, Skills, Aspirations•Mid-term: Practices, Knowledge Application, Behaviors•Long-term: Sustained impact of the program on Social, Economic, and Environmental conditions

Significance: Importance and priority

Significance:Importance of change to program partners

•Overall: Importance to a broad range of stakeholders•Particular: Importance to a particular individual or unit

Evaluation Cycle

Preparation•Planning--Setting goals, matching strategies•Focusing--Refining plans, identifying sources•Designing--Developing strategies for collecting data•Selecting Methods--Identifying tools to measure for goals

Evaluation Cycle

Implementation•Collecting Data—gathering info from diverse sources•Analyzing Data—reviewing and interpreting data

Evaluation Cycle

Utilization•Communicating and Using Results—Sharing data about quality and impact; applying findings to improvement

Positive Youth Development and Evaluation(Formal and Informal)

• Life skill and life-blood of program management– Goal Setting/Project Management– Communication– Problem Solving

Positive Youth Development and Evaluation

(Formal and Informal)

• Life skill and life-blood of program management– Goal Setting/Project Management– Communication– Problem Solving

• Key strategy in positive youth development– Experiential Learning = Action & Reflection

Youth Development as Evaluationbased on Hamilton, Hamilton, & Pittman, 2004

• Processes: Child Development knowledge base

• Principles: Program Theory and Philosophy

• Practices: Strategies for Programming…and Evaluation

• Products: Results of PYD Programming

Youth Development as Evaluationbased on Hamilton, Hamilton, & Pittman, 2004

Processes: Child Development knowledge base

DomainsPhysicalCognitive

Social-Emo StagesEarly Child

Middle ChildAdolescence

DynamicsMaturationInteractionAdaptation

Youth Development as Evaluationbased on Hamilton, Hamilton, & Pittman, 2004

Principles: Program theory and philosophy

Integrated across EcosystemsTheories of Change

Theories of Action

Developmentally Appropriate

Universal and Inclusive

Intentional and Iterative

Holistic and Communal

Ongoing in Engagement

Strengths-based, Risk-aware

Youth Development as Evaluationbased on Hamilton, Hamilton, & Pittman, 2004

Practices: Strategies for program implementation

ExperientialLearning

•Project-based•Inquiry-based•Problem-based•Adventure-based•Participatory•Service-learning

Youth-led/Y-A partnership

SafetySupport

Skill-OpportunitiesSelf-efficacy-

buildingQuality

PracticesSocial Norms

Social InclusionStructureSynergy

choiceslevels

teamwork

active listening

buddy system

YD as EVAL: Authentic Assessmentbased on Hamilton, Hamilton, & Pittman, 2004

Practices: Strategies for program implementation…and evaluation

SafetySupport

Skill-OpportunitiesSelf-efficacy-

buildingQuality

PracticesSocial Norms

Social InclusionStructureSynergy

choiceslevels

teamwork

active listening

buddy system

Skills cardExpert obs/intx

RubricTrained obs/intx

Skills cardExpert obs/intx

Skills cardCheck-in/Obs

Photo evidence

Physical evidence Demo/Confirmation

Youth Development as Evaluationbased on Hamilton, Hamilton, & Pittman, 2004

Products: Outcomes of programs

Character

ConfidenceCompetence

Caring

Contribution

Connection

Youth Development as Evaluationbased on Hamilton, Hamilton, & Pittman, 2004

Processes: Child Development knowledge base

DomainsPhysicalCognitive

Social-Emo StagesEarly Child

Middle ChildAdolescence

DynamicsMaturationInteractionAdaptation

Youth Development as Evaluationbased on Hamilton, Hamilton, & Pittman, 2004

Principles: Program theory and philosophy

Integrated across EcosystemsTheories of Change

Theories of Action

Developmentally Appropriate

Universal and Inclusive

Intentional and Iterative

Holistic and Communal

Ongoing in Engagement

Strengths-based, Risk-aware

YD as EVAL: Minimum Effort

Outdoor Skills

Program Quality

Cabin Group Life Skills

FocusIntegrate

Document

FrequencyIntensityDuration

YD as EVAL: Maximum Effort

Cabin Group Life Skills

Outdoor Skills

Registration

Program QualityParentSurvey

AgentFocus Groups

Benefits of Evaluation

• Systematic planning, implementation• Monitoring risks, opportunities• Accountability and reporting• Improving and expanding quality• Sustaining, growing funding• Support and recognition of partners• Recruiting, training new partners• Sharing with peers• Building broader support, insight

Barriers to Evaluation

“We’re too busy…

“Evaluation can only hurt us…

“We just don’t have the goods….

“We don’t know what to evaluate…

Resources for Evaluation

https://youthdevelopment.ces.ncsu.edu/

E-Basic BasicsFactsheets

A difference that counts

How evaluation makes a fairer fair…•Safety…Support…Skill-building•Typical/Exceptional care for animals•Problem solving about projects, careers•Individual resilience and group teamwork

Connectedness Competence Confidence

So…Why evaluate?

• Satisfy our own curiosity…

• Track quality and impact…

• Use results to improve, show impact, inform partners, and invite new participants

• Make the most of our method: action and reflection, learning by doing

Youth Development as Evaluation: Application

SafetySupport

Skill-OppsSelf-efficacy-building

Social NormsSocial Inclusion

StructureSynergy

Cabin Group Life Skills

Outdoor Skills

Program Quality

Registration

Program Quality

ParentSurvey

“What I do doesn’t make that much difference…”

A Fair Appraisal

•How did the volunteer promote youth development best practices (safety, support, skill-building, etc.)?

•How would these qualities over time promote connectedness, competence, confidence, character, caring, and contribution?

Behold the Benefits

Professional LevelFormal Evaluation Functions

•Reporting achievements to sponsors and decision-makers

•Sustaining, improving upon, or enhancing program activities

…and evaluation is experiential learning!

Eval benefits at the Professional Level(Formal evaluation)

•Accountability to program stakeholders

•Improvement efforts with staff and volunteers