are we making a difference? application of a five-level evaluation model to determine impact of...
TRANSCRIPT
Are We making a Difference? Application of a Five-Level Evaluation Model to Determine Impact of State Capacity BuildingPaula D. Kohler
Western Michigan University
David W. TestUNC Charlotte
Paper presented at the OSEP Project Director’s Conference, July 20, 2010
What’s an NSTTAC?
National Secondary Transition Technical Assistance Center Federally funded technical assistance and
dissemination center (TA&D) January 1, 2006 through December 31,
2010 By the U.S. Department of Educations
Office of Special Education Programs (OSEP)
Award #H326J050004
Our Charge
Assist SEAs in collecting data on federally mandated State Performance Plan Indicator 13 (content of the IEP) and using these data to improve transition education and services.
Build state capacity to implement evidence-based secondary education and transition services for youth with disabilities.
Model for Extending Transition Research
Effective Transition Practices
Increase Capacity to Implement Effective Transition Practices
Facilitate Implementation of Effective Transition
Practices
Data-Based Decision Making
Professional Development
Policy Analysis and
Change
Technical Assistance
The Guskey Model
Evaluating the Impact of Professional DevelopmentLevel 1 – Participants’ reactions
Level 2 – Participants’ learning
Level 3 – Organizational impact
Level 4 – Participant implementation
Level 5 – Student learning outcomes
Level 1 – Participant Satisfaction
Questions Did they like it? Was their time well spent? Did the material make sense? Will it be useful?
What’s measured Initial satisfaction with the experience
NSTTAC Examples - Level 1
Level 1 – Participant reactionsLikert-like scale evaluations of institutes, cadre meetings, workshops
Achievement of intended outcomes Usefulness of information Relevance of materials
Qualitative open ended questions What worked and what didn’t
Participant Satisfaction
Your Thoughts?
Level 2 – Participant Learning
Questions Did participants acquire the intended
knowledge and skills?
What’s measured New knowledge and skills of
participants
NSTTAC Examples - Level 2
Level 2 – Participant learningPre-post tests
New knowledge and skills of participants: student, teacher, and parent instruments
Analysis of products Development of IEPs
Participant Learning
Note. Frequency (f) represents the number of participants who answered the item on both the pretest and posttest. Dependent t test (across all items) revealed a significant difference between pretest scores and posttest scores, t(396)=-22.06, p < .0001.
Participant Learning
Your Thoughts?
Level 3 – Organization Factors
Questions What was the impact on the organization? Did it affect organizational climate and
procedures? Was implementation advocated, facilitated, and
supported? Were sufficient resources available?
What’s measured The organization’s advocacy, support,
accommodation, facilitation, and recognition
NSTTAC Examples - Level 3
Level 3 – Organization support and changeAnalysis of teacher reports regarding curriculum implementation
Identification of facilitators and barriers to curriculum implementation, including administrative support
Analysis of annual performance reports (APRs) to determine
Change in data collection procedures Alignment of strategic plans (from institutes) with
improvement activities in “determination ” areas Change in target indicators
Curriculum Implementation
Your Thoughts?
Level 4 -- Participant Implementation
Questions Did participants effectively apply the
new knowledge and skills?
What’s measured Degree and quality of implementation
NSTTAC Examples - Level 4
Level 4 – Participant use of new knowledge and skillsAnalysis of state and local strategic plans (from institutes)
To document and improve the implementation of program content
To assess growth from year to year
Evaluation of local curriculum implementation To assess if and how participants applied their
new knowledge at the classroom level
Teacher Involvement
Your Thoughts?
Level 5 – Student Learning
Questions What was the impact on students? Did it affect student performance or achievement? Did it influence students’ physical or emotional well-
being? Is student attendance improving? Are dropouts decreasing?
What’s measured Student learning outcomes:
Cognitive, affective, psychomotor
NSTTAC Examples - Level 5
Level 5 – Student learningAnalysis of APRs and SPP/APR Indicators
To determine school and student improvement on federal performance and compliance indicators
To demonstrate the overall impact of capacity building
To assess impact of capacity building model at the state and local levels
Student portfolios and oral reports To measure student learning outcomes
Student Participation in IEP
LEVEL 5 EVALUATION: STUDENT LEARNING OUTCOMES
EVALUATION
Durant High School Tool 4: Student Self-Assessment of Student InvolvementDurant, OK, March 11, 2009
PURPOSE To measure extent self-determination courses have impacted student learning outcomes as seen in student involvement on the IEP
EVALUATION
QUESTIONS
Q1. Did the student attend their IEP?Q2. How much did the student contribute in the IEP meeting?
DATASOURCES
Tool 4: Student Assessment of Student Involvement
OUTCOME Q1. The percentage of students that attended their IEP: 100% Students attended their IEPQ2. The percentage of students that felt they contributed somewhat to yes in the IEP meeting: 100% Identified their post-secondary goals 100% Provided information about their strengths 100% Provided information about their limitations or problem areas 100% Provided information about their interests 100% Provided information about the courses they want to take 100% Reviewed their past goals and performance 100% Asked for feedback or information from the other participants at their IEP meeting 100% Identified the support they need 100% Summarized the decisions made at the meeting
IMPLICATION
A significant increase in the number and extent of students involved in their own IEP as self-assessed
Student Learning
Dress nice and appropriately (12)
Be on time (4) Don’t rush Work hard (2) Respect (2) Turn off cell phones (3) Resumes (2) Different types of jobs (2) Don’t chew gum (3)
Be nice in the work place
How to find jobs (6) How to interview (3) How to use
community resources to find a job (3)
How to apply for a job (2)
How to act during an interview (5)
How to look-up jobs in the Internet (5)
List 3 things you learned today (n=16)
Your Thoughts?
Challenges
Easy to collect and analyze data regarding Level 1 -- satisfaction
Somewhat more difficult to collect and analyze data regarding Level 2 – participant learning
More difficult to collect and analyze data regarding Levels 3, 4, 5 – organization, application, and student learning
Questions?
Resources
www.nsttac.org NSTTAV Evaluation Toolkit
NSTTAC Indicator 13 Checklist
NSTTAC’s training materials
NSTTAC Transition Institute Toolkit
Paula Kohler ( [email protected])
David Test ([email protected])