School Improvement Network Impact Assessment:Higher Engagement Schools
versus Lower Engagement Schools
Steven H. Shaha, PhD, DBAProfessor, Center for Public Policy and Administration
Independent EvaluatorDecember 2012
Overarching Research Question:
Does engagement in PD 360 and Observation 360,
tools within the Educator Effectiveness System,
significantly affect student success
and school-wide metrics?
Sample Description• High Video Utilizers– 39 States– 211 Districts– 734 Schools
• Metrics:• Educator
Engagement• Student Success• School Impacts
• 32 data elements collected or computed through PD 360 and Observation 360
• Contrasted higher engagement schools versus lower engagement schools– Improvement in percentages of students who tested
advanced or proficient in math and reading– Classified into four quartiles– Analyses of highest and lowest quartiles only
Study of Educator Engagement
Metrics for Differentiating Advantages for Higher Engagement Organizations:
• Focus Objectives Set Up• Observations Performed• Percent Registered Users• Percent of Users in Communities
• Minutes Viewed• Forums Viewed• Programs Viewed• Segments Viewed• Links Viewed
• Follow-up Questions Answered• Reflection Questions Answered• Focus Objectives Set Up• Forums Posted• Downloaded Files• Uploaded Files• Participation in Communities
Leadership, Implementation
and Accountability
Educator Participation
Educator Engagement
These are the 15 metrics for which higher engagement schools were significantly higher than their lower engagement counterparts
Sample of Differentiating Metrics of Utilization and Engagement
Links Viewed
Minutes Viewed Follow-up Questions Answered
Observations Performed
Passive participation (e.g. video viewing alone) is LESS influential than Active engagement
63.8% advantage (p<.001)
39.0% advantage (p<.001)
70.3% advantage (p<.001)
4.3% advantage (p<.001)
Passive participation (e.g. video viewing alone) is LESS influential than Active engagement
Uploaded Files
Downloaded Files Forums Viewed
Forums Posted
Sample of Differentiating Metrics of Utilization versus Engagement
47.3% advantage (p<.001)
30.5% advantage (p<.001)79.5% advantage (p<.001)
68.6% advantage (p<.001)
Who Cares?Who cares if educators used it
more?
Did it make a difference for kids and schools?
Student Success:• Performance on standardized tests– Percent either proficient or advanced in the
following subjects:• Reading• Math
Study of Student Success
ImprovedStudent Performance
4.9% gain for lower engagement schools (p<.01)
ImprovedStudent Performance
4.9% gain for lower engagement schools (p<.01)
18.0% gain for higher engagement schools (p<.001)
Closed the Gap:267% advantage in gains for higher engagement schools (p<.001)
Nearly 4 times the impact
ImprovedStudent Performance
0.5% gain for lower engagement schools (p=ns)
Actually Important Gains:For every 200 students, 1 more performed at proficient or advanced levelthan in the previous year
ImprovedStudent Performance
0.5% gain for lower engagement schools (p=ns)
18.9% gain for higher engagement schools (p<.001)
Surpassed the Gap:3,520% advantage in gains for higher engagement schools (p<.001)
36 times greater impact
• Performance on key indicators from Internet (when publicly available) and structured telephone interviews:– Dropout Rates– Student Discipline Rates– Teacher Retention Rates– College-Bound Rates
Metrics of School Impact
Improved Dropout Rates
4.9% improvement for lower engagement schools (p<.01)
For every 100 students, 5 fewer dropped out than in the previous year
Figures reflect rounding, projections reflect correct math.
Improved Dropout Rates
20.0% improvement for higher engagement schools (p<.001)
309.1% advantage in gains for higher engagement schools (p<.001)
Executive SummaryHigher engagement schools
began statistically equal, then significantly
outperformed their counterparts (p<.01)
4.9% improvement for lower engagement schools (p<.01)
For every 100 students, 20 fewer dropped out than in the previous year.
Improved Student Discipline Rates
Figures reflect rounding, projections reflect correct math.Y-Axis is inverted to reflect improvement as intuitively upward trend.
7.4% fewer disciplinary incidents for lower engagement schools (p<.01)
33.2% fewer disciplinary incidents for higher engagement schools (p<.001)
Improved Student Discipline Rates
Executive SummaryHigher engagement schools significantly outperformed their counterparts (p<.01)
7.4% fewer disciplinary incidents for lower engagement schools (p<.01)
351% advantage in gains for higher engagement schools (p<.001)
4 ½ times the impact
For every 100 students, 33 fewer problem students than in the previous year.
Figures reflect rounding, projections reflect correct math.
ImprovedTeacher Retention Rates
1.7% more teachers stayed for lower engagement schools (p<.01)
ImprovedTeacher Retention Rates
Executive SummaryHigher engagement schools significantly outperformed their counterparts (p<.01)
65.9% advantage in gains for higher engagement schools (p<.001)
Nearly twice the impact
1.7% more teachers stayed for lower engagement schools (p<.01)
2.8% more teachers stayed for higher engagement schools (p<.01)
For every 100 teachers, nearly 3 fewer left than in the previous year.
Figures reflect rounding, projections reflect correct math.
Improved College-Bound Rates
No decrease or gain for lower engagement schools (ns)
Figures reflect rounding, projections reflect correct math.
Percentage of students schools report as being college-bound.
Improved College-Bound Rates
Executive SummaryHigher engagement
schools began statistically equal, then significantly
outperformed their counterparts (p<.01)
Incalculable advantage in gains for higher engagement schools (p<.001)
9.6% improvement for higher engagement schools (p<.001)
No decrease or gain for lower engagement schools (ns)
For every 100 students, 10 more were college-bound than in the previous year.
Figures reflect rounding, projections reflect correct math.
• Dropout Rates– Approx. 15 fewer dropouts per 100 students than lower engagement
school counterparts
• Student Discipline Rates– Approx. 33 fewer students “in the office” per 100 students than for
lower engagement school counterparts
• Teacher Retention Rates– Approx. 3 fewer teachers leaving per 100 teachers, which is 1 fewer
than for lower engagement school counterparts
• College-Bound Rates– Approx. 10 more college-bound students per 100 students than for
lower engagement school counterparts
Summary of School Impacts
Student Success
School Impacts
Leadership, Implementation
and Accountability
Educator Participation
Educator Engagement
A Model for Educational Success