program evaluation follow-up...
TRANSCRIPT
Program Evaluation Follow-up Webinar
Lee R. Gaseoma, Ed.D.
Suzanne Malson, M.S.
American Indian
Vocational Rehabilitation
Training and
Technical Assistance Center
The content of this presentation was developed under a grant from the U.S. Department of Education (CFDA 84.250Z; PR/Award Number
H250Z150002). However, this content does not necessarily represent the policy of the Department of Education, and you should not assume endorsement by the Federal
Government.
Do you have questions or comments to share on the first webinar regarding program evaluation or the second webinar regarding the community of practice (CoP)?
Learning Objectives
AIVRS STAFF WILL…
LO1 …understand the purpose of program evaluation
LO2 …make distinctions in the required AIVRS performance reports
LO3 …be able to work through the program evaluation process, methods, and analysis
LO4 …identify qualitative and quantitative data methods
LO5 …understand formative and summative evaluation methods
LO6 …utilize basic descriptive statistics to enhance the AIVRS annualreport, the final 5-year AIVRS report and Tribal reports
LO7 …become familiar with sources of data for program evaluation
Topics of Interest from CoP 1. Purpose of program evaluation and CoP
2. Example of techniques used in program evaluation
3. Implement program evaluation described in the AIVRS grant proposal
4. Data gathering, management, and analysis
5. Use of data to prepare annual AIVRS performance report
6. Program evaluation provides performance feedback
Purpose of program
evaluation
Purpose of Program Evaluation
•Demonstrate AIVRS program effectiveness to funders
• Improve the implementation and effectiveness of programs
• The better management of limited resources
•Document program accomplishments
Purpose of Program Evaluation
• Justify current AIVRS program funding
•Support the need for increased levels of funding
•Document AIVRS program development and activities to help ensure successful replication
Purpose of CoP
Purpose of Community of Practice (CoP)
•CoP is a group of people who work together to solve a problem or improve practice in their area of professional work
•Deepen participants’ knowledge and expertise by interacting on an ongoing basis
Purpose of Community of Practice (CoP)
•CoPs may be large in scale and deal with complex problems
•CoPs may be small in scale that focus on a problem at a specific level
Example of Techniques used
in Program Evaluation
Quantitative and Qualitative Techniques
Quantitative Techniques Qualitative Techniques
Surveys/Questionnaires Observations
Pre/post Tests Interviews
Existing Databases Focus Groups
Statistical Analysis Non-statistical (methods vary)
Implement Program Evaluation Described in the AIVRS Grant
Proposal
AIVRS Grant Proposal
•Describes the quality of Program Evaluation
•The evaluation methods must be thorough, feasible and appropriate to measure the program goals, objectives, and outcomes
AIVRS Grant Proposal•Evaluation methods should include the use of objective performance measures clearly related to the intended program outcomes
•This means the use of an unbiased evaluation tool that will measure your outcomes
AIVRS Grant Proposal
•Two evaluation criteria to consider is the validity and reliability of your evaluation tool
•Evaluation methods will provide quantitative and qualitative data
AIVRS Grant ProposalA well-designed program evaluation will provide accurate performance feedback and periodic assessment of progress toward achieving intended outcomes
AIVRS Grant Proposal•Evaluation strategies describe how appropriate and reliable data will be obtained and evaluated
•It is important to use external evaluators who have the experience and knowledge of AIVRS programs
AIVRS Grant Proposal•Program evaluation procedures should be phased in with other project tasks. For example the use of consumer satisfaction surveys
•Evaluation plans provide performance feedback to influence program operations
Data Gathering,
Management, and Analysis
What methods are you using to
gather, manage and analyze
program data?
Program Evaluation Data
Quantitative data•Consist of numbers or quantities•Closed-ended questions•Limited choices in answers
Data Source
Case Management produces quantitative data•Number of Applicants•Number of Individual Plan of Employment, (IPE)•Successful Employment Outcomes
Program Evaluation Data•Qualitative•Looks at quality•Open-ended questions•Unlimited choices in answers that is obtained in focus groups •This approach is time consuming but worth the effort
Data Source
Program Outreach Activities•Disability Awareness Day•Cultural Awareness Day
Data SourceCommunity Education•Presentations on Program at Tribal and Local Functions •Meetings with Local Primary Employers
Data SourceCommunity Education •Partner with State Independent Living Council (SILC) and State Rehabilitation Council (SRC)
•Implement co-operative agreement with State VR Agencies
Use of Data to Prepare Annual
AIVRS Performance
Report
AIVRS Annual Performance Report Includes 9-Sections
1. Program budget
2. Program goals and objectives
3. Number of consumers served
4. Types of VR services provided
5. Educational goals and employment outcomes
AIVRS Annual Performance Report
6. Job Training Common Measures
7. Interaction with State VR agency
8. Program self-evaluation
9. Consumer satisfaction
Program Evaluation Provides
Performance Feedback
The program evaluation cycle provides AIVRS
programs with performance feedback while the program is in progress, as well as at
the end of the program period
AIVRS Purpose Statement
AIVRS Purpose Statement
AIVRS Purpose Statement
Thank you!
Contact Information• Lee R. Gaseoma, Ed.D., Director
• Email: Lee Roy Gaseoma [[email protected]]
• PH: 928-523-6829
• Suzanne Malson, M.S.
• American Indian VR Training & Technical Assistant Specialist
• Email: [email protected]
• Phone: 928-523-8602
Physical NAU Location
•American Indian VR Training & TA Center
•PO Box 5630
• (912 Riordan Ranch Rd.)
•Flagstaff, AZ 86011-5630
•FAX: 928-523-9127
•TTY: 928-523-1695