Evaluating the Vermont Mathematics Initiative (VMI) in a Value Added Context

H. ‘Bud’ Meyers, Ph.D.College of Education and Social Services

University of VermontDouglas Harris, Ph.D.

The Evaluation Center The Vermont Institutes, Montpelier, VT

Purpose

Describe the context, purpose, structure, staffing and content of the instructional program for teachers

Describe the logic model that provides the framework for evaluation of the program

Examine the methodology for assessing the value added to student learning by teacher preparation in mathematics

Present findings from cross-sectional analyses of eight cohorts of students who have been taught by VMI teachers and four cohorts of control students.

Present findings from a qualitative evaluation of the process of implementation of the components of the program from the perspectives of participants and administrators

Present recommendations arising from the findings of the evaluation

Context, purpose, structure, staffing

A field based masters degree in Education with a primary emphasis on mathematics content

Partially funded by local districts, partially MSP

Focus on Grades k-8 Includes focus on teacher leadership Includes action research Three years Mathematicians and educators

Content-12 Courses

Course 1: Mathematics as a Second Language (3 credits)

Course 2: Functions and Algebra for Elementary Teachers (3 credits)

Course 3: Trigonometry for Elementary Teachers (2 credits) and Algebra and Geometry II (1 credit)

Course 4: Measurement, Geometry, and Probability for Elementary Teachers (3 credits)

Course 5: Number Theory for Elementary Teachers (3 credits)

Courses…contd.

Course 6: Statistics, Action Research, and Inquiry into Effective Practice, I (3 credits)

Course 7: Statistics, Action Research, and Inquiry into Effective Practice, II (3 credits)

Course 8: Algebra and Geometry for Elementary Teachers, III (3 credits)

Course 9: Statistics, Action Research, and Inquiry into Effective Practice, III (3 credits)

Course 10: Calculus for Elementary Teachers, I (3 credits)

Courses…contd.

Course 11: Calculus for Elementary Teachers, II (3 credits)

Course 12: Capstone VMI Experience (3 credits)

Methods

Quantitative- Cross-sectional and longitudinal Two intervention groups

Group 1=16 schools, multiple VMI teachers (HiEnd)

Group 2=22 schools, single teacher per school (RegIntervention)

Control group=22 schools, matched to Groups 1 and 2 (combined) on demographics

Longitudinal comparisons for 2 cohorts, 1999 and 2000, grades 4 to 8. (See Figure 2)

Methods

Qualitative Interviews

Participants Graduates Administrators

Observations Review of course materials and participant

portfolios Course evaluations Interviews of School Staff and Leadership

Sample Characteristics

Comparison of Free/Red Lunch- Schools (%)

10

15

20

25

30

35

40

Groups of Schools

Per

cen

t F

ree/

Red

Lu

nch

2001 Free/Red Lunch- School (%)

2001 Free/RedLunch- School (%)

32 36

VMI Control

Figure 1

Free or reduced price lunch eligibility is a metric for poverty used by the Vermont Department of Education in the calculation of Adequate Yearly Progress. At grade 4 for each cohort when it was measured in the present study, it yields a statewide average of about 34%. Both the VMI and Control schools were about at the state average. Despite a 4 percent differential in poverty level between the VMI and Control groups, comparisons of the VMI and Control groups of schools for both of the cohorts indicated no significant differences among the matched schools with respect to poverty levels as documented by the Vermont Department of Education in 2001.

Current and Potential Trend Studies with 3 data points for VT Data

Others are cross-sectional, independent group comparisons

Year Gr 4 Gr 5 Gr 6 Gr 7 Gr 8 Gr 9 Gr10

1999 x

2000 x

2001 x

2002

2003 x

2004 x x

2005 x x x

2006 xx = Comparisons

Longitudinal comparisonsFigure 2

Question 1: What do comparisons with comparable schools show, over time, when students are matched from grade 4 through grade 8 and grade 10 during the two cohort groups (1999-2005, and 2000-2006)?

Intervention and Control Groups from 1999 to 2005 Matched Pairs Scaled Scores

620

670

720

Years of Intervention

Sca

led

Sco

res

VMI

Control

VMI 662 720 745

Control 657 716 734

1999 2003 2005

Significant differences in 2005. t=3.22, p<.001, df 1, 1300.

Intervention and Control Groups from 2000 to 2006 Matched Pairs Scaled Scores

620

670

720

Years of Intervention

Sca

led

Sco

res

VMI

Control

VMI 671 720 728

Control 668 708 720

2000 2004 2006

Significant differences in 2004. t = 5.27, p<.001, df 1, 1488Significant differences in 2006. t = 3.39, p<.001, df 1, 1107

Figure 3

Figure 4

Discussion

As indicated in Figures 3 and 4, each of the VMI cohorts outperforms the Control schools in similar patterns of difference. Results for comparisons at the first data point for each cohort (1999 and 2000) are not significantly different while they emerge as significant in 2005 and 2006.

Intervention and Control Groups from 1999 to 2005 Matched Pairs Scaled Scores

650

675

700

725

750

Years of Intervention

Scal

ed S

core

s Reg I

Control

HiEnd

Reg I 650 708 735

Control 653 709 735

HiEnd 667 725 753

1999 2003 2005

Significant differences in 2005. adjusted for differences in 1999,

HiEnd v Reg and Control, F = 7.96, df 2, 1235, p <.001

Question 2: How do levels of implementation of the VMI curriculum in classrooms relate to levels of achievement of students who experience them? (1999 Cohort)

Figure 5

Intervention and Control Groups from 2000 to 2006 Matched Pairs Scaled Scores

650

675

700

725

750

Years of Intervention

Sca

led

Sco

res

Reg I

Control

HiEnd

Reg I 664 710 722

Control 667 706 719

HiEnd 674 727 730

2000 2004 2006

Significant differences in 2006 adjusted for differences in 2000, HiEnd v Reg and Control, F = 9.91, df 2, 1093, p <.001 Note: The average standard deviation for groups is about 40 points across all groups.

Mean differences between groups is thus approximately .25 S.D. between HiEnd and Control groups.

Question 2: How do levels of implementation of the VMI curriculum in classrooms relate to levels of achievement of students who experience them? (2000 Cohort)

Figure 6

Discussion:

VMI HiEnd Schools significantly out-performed the matched schools and appear to carry most of the difference between the VMI and matched schools in both cohorts from 1999 through 2006. Analyses of covariance adjusted the score differences at the first two data points in each cohort. The final data point for each cohort represents group differences at grade 10.

Question 3. Are school level effects related to income level of students and levels of content implementation of the VMI curriculum? (1999 Cohort) Do gaps close?

Free Lunch vs Others Scaled Scores 1999 CohortANCOVA

580

600

620

640

660

680

700

720

740

Year

Sca

led

Sco

res

Reg I (E)

Control (E)

HiEnd (E)

Reg I (NE)

Control (NE)

HiEnd (NE)

Reg I (E) 638 692 694

Control (E) 646 697 720

HiEnd (E) 650 699 725

Reg I (NE) 654 715 722

Control (NE) 656 714 721

HiEnd (NE) 670 729 727

1999 2003 2005

Figure 7

Question 3. Are school level effects related to income level of students and levels of content implementation of the VMI curriculum? (2000 Cohort) Do gaps close?

Free Lunch vs Others Scaled Scores 2000 CohortANCOVA

580

600

620

640

660

680

700

720

740

Year

Sc

ale

d S

co

res

Reg I (E)

Control (E)

HiEnd (E)

Reg I (NE)

Control (NE)

HiEnd (NE)

Reg I (E) 635 695 723

Control (E) 639 700 722

HiEnd (E) 643 706 723

Reg I (NE) 652 713 727

Control (NE) 656 705 726

HiEnd (NE) 661 723 728

2000 2004 2006

Figure 8

Discussion:

The baseline, grade 4 performance is comparable for all groups. Not surprising is the observation that whether in intervention schools or matched schools, students eligible for free or reduced lunch score significantly lower than their non-eligible for free or reduced price lunch peers. This pattern continues through the grade 8 testing. Secondly, the grade 10 Free-lunch eligible students are significantly outscored by their non-eligible peers in the matched schools. No surprise there. But, rather surprising is that given the lack of cohort differences at grades 4 and 8, the free or reduced lunch eligible students are significantly out-scoring their eligible peers at grade 10 and they are gaining on students who are not eligible for free or reduced lunch in the matched schools. The mean score for the group of VMI Free-lunch eligible students (723, in 2006) is approaching that of the Matched Schools non-eligible students (726). In other words, the gap is narrowing between the free or reduced lunch eligible students and their non-eligible peers in the matched schools. These results were not obtained for the regular VMI schools in 2005 (Figure 5) and we have no explanation for that result at this time.

Quantitative Summary

Longitudinal comparisons of VMI and Control schools indicate a persistent pattern of advantage for VMI schools

Longitudinal comparisons indicate that the VMI schools, with more VMI teachers, had a significantly greater long term gain on percentile ranks. Stronger teachers may not have as great an effect in the shorter term when isolated.

Longitudinal comparisons suggest that SES differences are reduced over time when high quality mathematics instruction is offered to all students

Qualitative Findings (2005 and 2006)

Impact on Participants and Their Teaching Finding 1: Mathematics Content An overarching theme is the impact of the VMI experience on the teacher’s own

understanding of mathematics content. This is consistent among participants in all three years of the VMI experience and remains the highest priority of graduates as they reflect on the VMI experience

Finding 2: Increased Confidence Related to Mathematics Increased understanding of mathematics content impacts the confidence of participants as

related to mathematics, to teaching mathematics, and to their enthusiasm for and enjoyment of mathematics.

Finding 3: Impact on Instructional and Assessment Practice Participants graduates (many of who are math teacher leaders), and principals report that

the instruction in VMI, increased content knowledge, and increased confidence have had major impact on instructional and assessment practice in the classrooms of participants and of teachers impacted by VMI-trained math teacher leaders

Finding 4: The Impact of Action Research on Classroom Practice The impact of action research is mixed, with some participants, graduates, and principals

reporting considerable impact, others less so. Finding 5: Principal Support The active support of principals makes a profound difference in VMI participants’ work in

the classroom and in leadership positions.

More Qualitative Findings

Finding 6: Personal Impact on Participants Beyond the impact of VMI on teachers in relation to math content, instruction, and leadership, the program

also has profound personal impact on many participants. Impact on Students Finding 7: Transfer of VMI Content to the Classroom Teachers report direct transfer of mathematics content used in VMI to the math experiences of their

students. Finding 8: Impact on Student Problem Solving Teachers and teacher leaders report that the problem solving emphasis in VMI has significant impact on

their understanding of how to engage students in problem solving in the classroom. Finding 9: Impact of Action Research on Students Some teachers believe that the interventions begun in their action research projects will continue to impact

their students over time. Impact on Teacher Leadership in Schools and Districts Finding 10: Impact on Teacher Leaders Teachers working as teacher leaders credit VMI for providing the knowledge, confidence, and support for

them to take on leadership roles. This is true of teachers working in leadership at the school, district, and state levels.

Finding 11: Impact on curriculum, instruction, and assessment Teacher leaders report that the knowledge base and contacts developed during VMI have significant

impact on the development and implementation of mathematics curriculum, instruction, and assessment. Finding 12: Need for Ongoing Professional Community There is a common desire among VMI graduates to maintain the type of professional learning community

afforded them through VMI.

Recommendations

Recommendations Related to Continued Study of the Vermont Mathematics Initiative

Continue to gather longitudinal data from the State of Vermont’s Assessment System. .

As Vermont transitions to statewide assessment utilizing the New England Common Assessment, carefully analyze the logic and structure of the NECA and review VMI course content in relation to the Grade Level Expectations upon which this assessment is built.

Continue qualitative analysis utilizing existing data sources and consider adding series of observations within VMI participants’ classrooms to better understand what exactly is happening in those classrooms.

Select and implement a “theory into practice” change model considering, for example:

o the IBM/Harvard School of Business Change Toolkit and/or o the McREL Balanced Leadership Model. o the CBAM model of school change o the New Structure of School Improvement model (Joyce and Calhoun)

Educational or Scientific Importance of the StudyThe National Research Council, (2004) US Department

of Education (2004) and researchers at RAND (2004) have cited the need for better evaluations of teacher professional development programs. The combination of mixed method approaches with multiple strategies such as longitudinal studies of students acting as their own controls over time is the present attempt to respond to current concerns. In addition, intervention students in combination with matched groups at multiple intervals is a further attempt to provide additional confidence in results.

Want a copy of the paper or PowerPoint?? Bud.Meyers@uvm.edu