the formative & summative evaluation process ts & eoc saeopp priority 1 training py 2009
TRANSCRIPT
The Formative & Summative The Formative & Summative Evaluation ProcessEvaluation Process
TS & EOC TS & EOC
SAEOPP Priority 1 Training
PY 2009
PresenterPresenter
Elizabeth C. Retamozo, Executive Director
NOSOTROS Education Center NOSOTROS Education Center www.nosotrosedu.org
Phone: 469-450-7137
Email:[email protected]
2
Presentation MaterialsPresentation Materials
• PowerPoint Slide Handouts• Additional Documents & Materials located at:
• www.nosotrosedu.org– Go to Resources Page– Register Your Email Address– Click the Presentations & Links Tab– Click SAEOPP Priority 1 Training
3
4
Session Summary Session Summary
This session will focus on providing Talent Search staff with an overview of:
• Program Evaluation• Types of Program Evaluations • Methods & Types of Data Collection • Quantitative & Qualitative Data • Formative & Summative Methods
5
EvaluationEvaluation
• Evaluation - Systematic method for collecting, analyzing, and using information to answer basic questions about your program.
• Helps to identify effective & ineffective services, practices, & approaches.
• Informs & assists program staff to make changes to improve program effectiveness.
6
Evaluation PlanEvaluation Plan
• Evaluation Plan - A written & comprehensive plan to determine the effectiveness of the program in meeting stated objectives & goals.
• Includes what you plan to do, how you plan to do it, who will do it, and when it will be done.
• Provides a mechanism for evaluating & implementing programmatic changes to improve program effectiveness
• Relates your project to GPRA & the TRIO Programs Performance Indicators
7
The GPRA ConnectionThe GPRA Connection
1) Government Performance and Results Act of 1993 (GPRA)
2) DOE Response to GPRA Requirements: Goals 1-63) TRIO Programs Performance Indicator: Goal 54) TRIO Programs’ Specific Program Measures:
Upward Bound, Talent Search, Educational Opportunity Centers Student Support Services & Ronald E. McNair
5) Prior Experience Criteria & Mandatory Objectives – APR6) Project Evaluation Plan7) Activities & Services Specific to Each Objective
8
Types of Evaluation PlansTypes of Evaluation Plans
Goals-Based Evaluation
Outcome-Based Evaluation
Process-Based Evaluation
9
Evaluation Plan “Buzz” WordsEvaluation Plan “Buzz” Words
Qualitative Values
Quantitative Values
Formative Methods
Summative Methods
10
Qualitative ValuesQualitative Values
• Measurement of “feelings”, impressions, & perception
Ways to describe effectiveness that are measurable, but may not aggregate in traditional, quantifiable methods
Information that is difficult to measure, count, or express in numerical terms
11
Collection of Qualitative Data Collection of Qualitative Data
• Interviews & Focus Groups Used to fully understand someone's impressions or
experiences, or learn more about their answers to questionnaires
• ObservationsUsed to gather accurate information about how a program actually operates, particularly about processes
• Questionnaires, Surveys, Checklists
Used to quickly and/or easily get info in a non-threatening way
12
Quantitative ValuesQuantitative Values
• Information that can be expressed in numerical terms, counted, or compared on a scale
Promotion to Next Grade Level Rates
Graduation From High School Rates
# of College Admissions Forms Completed
# of Financial Aid Applications Completed
College Enrollment Rates
13
Collection of Quantitative Data Collection of Quantitative Data
• Report Cards
• Transcripts
• Test Scores (NCLB, PSAT, SAT, ACT)
• National Student Clearinghouse Reports
14
Formative MethodsFormative Methods
• Continuous (daily, weekly, monthly, quarterly, semester) assessment of services/activities & progress towards achieving objectives
• Allows for changes in activities during the year so final objectives/outcomes will be met (changes in delivery of services)
15
Formative MethodsFormative Methods
• Weekly/monthly student service reports# of continuing student with 0 services # students with less than 2 services # of students with equal to or more than # services # students that have received a specific services
• Student evaluation of workshops & field trips
• Grading period report cards (6/9 weeks)
16
Progress Towards Achieving Goals & Objectives During the Year -
Formative Methods:
• Project staff document all services provided to students as they occur on Individual or Group Service Logs
• Project staff enter all data on service logs into the project database on a weekly basis
• Project database generates monthly/quarterly contact reports documenting which students have been served, the types of services provided, & the frequency of services provided.
17
Progress Towards Achieving Goals & Objectives During the Year -
Formative Methods:
• Project Director tracks progress towards the accomplishment of all objectives via reports (quantitative) generated by the project database on a weekly/ monthly basis
• Participants rate specific group activities and workshops to rate their level of satisfaction with the quality of services on evaluation forms (qualitative).
18
Summative MethodsSummative Methods
• End of year assessment that measures the results or outcomes of a project.
Concerned with a program's overall effectiveness
Quantitative data is analyzed for a given period
Qualitative evaluation supplements the quantitative data
19
Achieving Goals & Objectives at End of the Year - Summative Methods:
• Project database combines all monthly reports to produce a report documenting & confirming the achievement of project objectives at the end of the year (quantitative).
• Students complete end-of-year surveys (qualitative) to measure level of satisfaction with the overall quality of project services.
20
Achieving Objectives at the End of the Project Period - Summative Methods:
• Target school/agency/project personnel complete end of year evaluation forms (qualitative) to measure level of satisfaction with the overall quality of project services.
• Project Director reviews evaluation results to implement any corrective measures to eliminate potential problems by appropriately modifying & improving activities & services for the next program.
21
TRIO Evaluation PlansTRIO Evaluation Plans
• Goals Based Evaluation (Internal & Formative) Evaluates the extent to which projects are meeting
predetermined goals or objectives. INTERNAL GOALS
• Outcomes Based Evaluation (Internal & Summative) Looks at impacts/benefits/changes to your students as a
result of your project’s efforts during and/or after their participation in your project. PE OBJECTIVES
• Process Based Evaluation (External & Summative) Geared to fully understanding how a program works -- how
does it produce that results that it does.
22
Goals Based Evaluation Goals Based Evaluation
1) What project goals were established?
2) What is the status of the project’s progress towards achieving the goals?
3) Will the goals be achieved according to the timelines specified in the program implementation or operations plan?
4) Do personnel have adequate resources (supplies, facilities, training, etc)?
23
Goals Based Evaluation Goals Based Evaluation
5) How should priorities be changed to more focus on achieving the goals?
6) How should timelines be changed?
7) How should goals be established in the future?
24
Sample TS Internal Goals Sample TS Internal Goals
1) Identification, Recruitment, & Selection of Participants:
To Identify, recruit, & select new students to have
a total of 600 participants with 67% classified as
both low-income & potential first generation
college students within 90 days of the beginning
of the program year.
25
Sample TS Internal Goals Sample TS Internal Goals
2) Development of Needs Assessment & Individual Service Plans:
100% of all participants will have their needs
assessed and an Annual Individual Service Plan
developed within 30 days of their initial selection
or renewal into the project each program year.
26
Sample TS Internal Goals Sample TS Internal Goals
3) Delivery of Promotion Services To 9th - 11th Participants:
70% of all 9th - 11th grade participants will
receive a minimum of three (3) college, career,
and financial aid exploration services each
semester for a total of six (6) services by the end
of the program year.
27
Sample TS Internal Goals Sample TS Internal Goals
4) Delivery of Individual College Admissions & Financial Aid Application Assistance to all College Ready Participants:
80% of all college ready participants will receive
a minimum of five (5) individual advising sessions
each semester for a total of ten (10) sessions by
the end of the program year.
28
Sample TS Internal Goals Sample TS Internal Goals
5) Increased Awareness & Knowledge of College, Financial Aid, & Career Information:
80% of participants will demonstrate an increase
in knowledge of college, financial aid & career
information as demonstrated by a comparison
pre/post tests by the end of the program year.
29
Sample TS Internal Goals Sample TS Internal Goals
6) Perception & Level of Satisfaction with Project Services:
85% of all participants and target school
personnel surveyed will rate the project as
"Excellent" on the Project Evaluation Form by the
end of each project year.
30
Outcome Based Evaluation Outcome Based Evaluation
TS Goals Based Evaluation Plan
31
Sample EOC Internal Goals Sample EOC Internal Goals
1) Identification, Recruitment, & Selection of Clients:
To Identify, recruit, & select new students to have
a total of 1000 clients with 67% classified as both
low-income & potential first-generation college
students within 180 days of the beginning of the
program year.
32
Sample EOC Internal Goals Sample EOC Internal Goals
2) Development of Needs Assessment & Individual Service Plans:
100% of all clients will have their needs assessed
and an Individual Service Plan developed within
30 days of their initial selection into the project
each program year.
33
Sample EOC Internal Goals Sample EOC Internal Goals
3) Delivery of GED/Continuing Education Services to Non-College Ready Clients:
100% of all Non-College Clients will receive a
minimum of three (3) GED/Continuing Education
services (individual advising, referrals,
information, etc.) by the end of the program year.
34
Sample EOC Internal Goals Sample EOC Internal Goals
4) Delivery of Individual College Admissions & Financial Aid Application Assistance to all College Ready Clients:
100% of all college ready clients will receive a
minimum of three (3) individual advising sessions
by the end of the program year.
35
Sample EOC Internal Goals Sample EOC Internal Goals
5) Perception & Level of Satisfaction with Project Services:
85% of all Clients and target agency personnel
surveyed will rate the project as "Excellent" on
the Project Evaluation Form by the end of each
project year.
36
Outcome Based Evaluation Outcome Based Evaluation
EOC Goals Based Evaluation Plan
37
Outcome Based Evaluation Outcome Based Evaluation
1) Identify Major Outcomes (Mandatory Objectives)
2) Specify Observable Measures & Indicators
3) Identify what Info is Needed to Show these Indicators
38
Outcome Based Evaluation Outcome Based Evaluation
4) Decide How this Info Can be Realistically & Efficiently Gathered
5) Assign Staff Responsibility & State Timelines
6) Analyze & Report Findings
39
Outcome Based Evaluation Outcome Based Evaluation
TS Outcome Based Evaluation Plan
EOC Outcome Based Evaluation Plan
40
External EvaluationExternal Evaluation
• Recommended in last grant proposal competitions
• Must be conducted if included in grant proposals
• Allowable cost for the project
• May be paid by the grantee agency/institution
• May be provided free of charge – in-kind/donation
41
External EvaluationExternal Evaluation
• May be conducted by person working/employed by agency/institution BUT not for the project
• Must have experience conducting evaluations for educational opportunity programs & TRIO like programs
• Must state what type of evaluation will be conducted by the evaluator.
42
External EvaluationExternal Evaluation
• Process-Based Evaluation
Process-based evaluations are geared to fully understanding how a program works -- how does it produce that results that it does.
• Useful if programs are long-standing and have changed over the years.
• Useful for accurately portraying to outside parties how a program truly operates.
43
Process-Based EvaluationProcess-Based Evaluation
• Examines the extent to which a program is operating as intended by assessing ongoing program operations & whether the targeted population is being served.
• Helps program staff identify needed interventions and/or change program components to improve service delivery.
44
Process Based Evaluation Process Based Evaluation
1) How does the project identify, recruit, & select students?
2) How does the project decide what specific services & activities to provide to its students?
3) How does the project monitor students’ academic progress?
45
Process Based Evaluation Process Based Evaluation
4) How does the project deliver services to its students?
5) How does the project document services provided to student?
6) What is the project’s goals based evaluation plan?
7) What is the project’s outcome based evaluation plan?
8) How does the project plan for the next program year?
46
Process Based Evaluation Process Based Evaluation
Process Based Evaluation Plan
47
Basic Guide to Program Evaluation
Written by Carter McNamara, MBA, PhD, Authenticity Consulting, LLC. Copyright 1997-2007.
Adapted from the Field Guide to Nonprofit Program Design, Marketing and Evaluation.
Online Guide: A Basic Guide to Program Evaluation.pdf
Bibliography & Resources
48
Basic Guide to Goals-Based Evaluation for Nonprofit Organizations with Very Limited Resources
Written by Carter McNamara, MBA, PhD, Authenticity Consulting, LLC. Copyright 1997-2007.
Online Guide: Basic Guide to Outcomes-Based Evaluation for Nonprofit Organizations with Very Limited Resources.mht
Bibliography & Resources
49
US Department of Health & Human Services Administration for Children & Families - Office of Planning, Research & Development
The Program Manager's Guide to Evaluation
http://www.acf.hhs.gov/programs/opre/other_resrch/pm_guide_eval/reports/pmguide/pmguide_toc.html
This guide explains program evaluation -what it is, how to understand it, and how to do it. It answers your questions about evaluation and explains how to use evaluation to improve programs and benefit students, families, and staff.
Bibliography & Resources
50
Questions & Answers