program evaluation follow-up...

Program Evaluation Follow-up Webinar

Lee R. Gaseoma, Ed.D.

Suzanne Malson, M.S.

American Indian

Vocational Rehabilitation

Training and

Technical Assistance Center

The content of this presentation was developed under a grant from the U.S. Department of Education (CFDA 84.250Z; PR/Award Number

H250Z150002). However, this content does not necessarily represent the policy of the Department of Education, and you should not assume endorsement by the Federal

Government.

Do you have questions or comments to share on the first webinar regarding program evaluation or the second webinar regarding the community of practice (CoP)?

Learning Objectives

AIVRS STAFF WILL…

LO1 …understand the purpose of program evaluation

LO2 …make distinctions in the required AIVRS performance reports

LO3 …be able to work through the program evaluation process, methods, and analysis

LO4 …identify qualitative and quantitative data methods

LO5 …understand formative and summative evaluation methods

LO6 …utilize basic descriptive statistics to enhance the AIVRS annualreport, the final 5-year AIVRS report and Tribal reports

LO7 …become familiar with sources of data for program evaluation

Topics of Interest from CoP 1. Purpose of program evaluation and CoP

2. Example of techniques used in program evaluation

3. Implement program evaluation described in the AIVRS grant proposal

4. Data gathering, management, and analysis

5. Use of data to prepare annual AIVRS performance report

6. Program evaluation provides performance feedback

Purpose of program

evaluation

Purpose of Program Evaluation

•Demonstrate AIVRS program effectiveness to funders

• Improve the implementation and effectiveness of programs

• The better management of limited resources

•Document program accomplishments

Purpose of Program Evaluation

• Justify current AIVRS program funding

•Support the need for increased levels of funding

•Document AIVRS program development and activities to help ensure successful replication

Purpose of CoP

Purpose of Community of Practice (CoP)

•CoP is a group of people who work together to solve a problem or improve practice in their area of professional work

•Deepen participants’ knowledge and expertise by interacting on an ongoing basis

Purpose of Community of Practice (CoP)

•CoPs may be large in scale and deal with complex problems

•CoPs may be small in scale that focus on a problem at a specific level

Example of Techniques used

in Program Evaluation

Quantitative and Qualitative Techniques

Quantitative Techniques Qualitative Techniques

Surveys/Questionnaires Observations

Pre/post Tests Interviews

Existing Databases Focus Groups

Statistical Analysis Non-statistical (methods vary)

Implement Program Evaluation Described in the AIVRS Grant

Proposal

AIVRS Grant Proposal

•Describes the quality of Program Evaluation

•The evaluation methods must be thorough, feasible and appropriate to measure the program goals, objectives, and outcomes

AIVRS Grant Proposal•Evaluation methods should include the use of objective performance measures clearly related to the intended program outcomes

•This means the use of an unbiased evaluation tool that will measure your outcomes

AIVRS Grant Proposal

•Two evaluation criteria to consider is the validity and reliability of your evaluation tool

•Evaluation methods will provide quantitative and qualitative data

AIVRS Grant ProposalA well-designed program evaluation will provide accurate performance feedback and periodic assessment of progress toward achieving intended outcomes

AIVRS Grant Proposal•Evaluation strategies describe how appropriate and reliable data will be obtained and evaluated

•It is important to use external evaluators who have the experience and knowledge of AIVRS programs

AIVRS Grant Proposal•Program evaluation procedures should be phased in with other project tasks. For example the use of consumer satisfaction surveys

•Evaluation plans provide performance feedback to influence program operations

Data Gathering,

Management, and Analysis

What methods are you using to

gather, manage and analyze

program data?

Program Evaluation Data

Quantitative data•Consist of numbers or quantities•Closed-ended questions•Limited choices in answers

Data Source

Case Management produces quantitative data•Number of Applicants•Number of Individual Plan of Employment, (IPE)•Successful Employment Outcomes

Program Evaluation Data•Qualitative•Looks at quality•Open-ended questions•Unlimited choices in answers that is obtained in focus groups •This approach is time consuming but worth the effort

Data Source

Program Outreach Activities•Disability Awareness Day•Cultural Awareness Day

Data SourceCommunity Education•Presentations on Program at Tribal and Local Functions •Meetings with Local Primary Employers

Data SourceCommunity Education •Partner with State Independent Living Council (SILC) and State Rehabilitation Council (SRC)

•Implement co-operative agreement with State VR Agencies

Use of Data to Prepare Annual

AIVRS Performance

Report

AIVRS Annual Performance Report Includes 9-Sections

1. Program budget

2. Program goals and objectives

3. Number of consumers served

4. Types of VR services provided

5. Educational goals and employment outcomes

AIVRS Annual Performance Report

6. Job Training Common Measures

7. Interaction with State VR agency

8. Program self-evaluation

9. Consumer satisfaction

Program Evaluation Provides

Performance Feedback

The program evaluation cycle provides AIVRS

programs with performance feedback while the program is in progress, as well as at

the end of the program period

AIVRS Purpose Statement

AIVRS Purpose Statement

AIVRS Purpose Statement

Thank you!

Contact Information• Lee R. Gaseoma, Ed.D., Director

• Email: Lee Roy Gaseoma [Lee.Gaseoma@nau.edu]

• PH: 928-523-6829

• Suzanne Malson, M.S.

• American Indian VR Training & Technical Assistant Specialist

• Email: Suzanne.Malson@nau.edu

• Phone: 928-523-8602

Physical NAU Location

•American Indian VR Training & TA Center

•PO Box 5630

• (912 Riordan Ranch Rd.)

•Flagstaff, AZ 86011-5630

•FAX: 928-523-9127

•TTY: 928-523-1695