Measuring the Effect of the Measuring the Effect of the Milwaukee Mathematics Milwaukee Mathematics Partnership on Student Partnership on Student
AchievementAchievementCindy M. WalkerJacqueline K. GoszDeAnn HuinkerUniversity of Wisconsin - Milwaukee
Structure of MMPMath TeachingSpecialist (MTS)
Learning Team(LT)
Literacy Coach (LC)
Math TeacherLeader (MTL)
Math Teacher
Concerns
• Difficulty measuring impact of MMP on student achievement in mathematics– Much of the effect of the MMP is not under
the direct control of the MMP– There are uncontrollable variables at the
school level that must be taken into consideration
• Finding an appropriate statistical model– Status-based models– Value-added Assessment (VAA) models
Status-Based Growth Models
• Rely upon cohort-to-cohort comparison to assess improvement in achievement
• Compare unadjusted average student performance on standardized tests across successive cohorts
• Inherent shortcomings– Based on a non-equivalent group case
study design– Do not take into consideration pre-
existing differences between students
VAA Growth Models
• Consider the growth of students from one year to the next through repeated measures analyses
• Use longitudinal data to compare a students’ progress to his/her past academic achievements
• Provides some control over the individual differences between students because students serve as their own controls
Purposes
1) To determine if reported differences in how the model of professional development is being implemented by the MMP are related to differences in student achievement gains in mathematics
2) To compare and contrast status-based and VAA models in terms of assessing the impact of the effectiveness of the MMP
3) To address issues specific to the MMP that have been encountered in attempting to implement a system of accountability
Methodology – Participants
• Schools– 92 of the 143 (64%) elementary & middle
schools that participated in the MMP during its 1st year were considered in this study.
• Students – 3rd – 8th graders who participated in the district’s standardized testing – VAA
• Only students who had test scores for both the 2003-4 and 2004-5 school years, did not change schools, and did not repeat/skip a grade could be considered
• 14,259 total students
– Status-based• 27,117 students for the 2003-4 school year• 27,079 students for the 2004-5 school year
Methodology - Instrumentation• Self-report instrument administered to MTLs
1) How often MTLs meet with their Math Specialist and others in their building;
2) How satisfied MTLs feel with the amount of time they have to discuss issues centered around mathematics with their Math Specialist and others in their building;
3) How supported MTLs feel in their role as a MTL by their Math Specialist and others in their building;
4) The MTLs perceptions of the functioning of school’s Learning Team in terms of the general climate of the meetings as well as the overall cohesion in accomplishing goals related to their mathematics curriculum.
• MTL Tracking variables– Number of MMP meetings attended and whether or
not a vision statement and/or action plan was turned in
Methodology – Instrumentation• Mathematics scale scores from the
November administration of the Terra Nova (grades 3, 5-7) and WKCE (grades 4 and 8) for 2003-4 and 2004-5.– Both tests published by CTB-McGraw Hill– Mathematics scale scores from Terra
Nova and WKCE shown to be highly correlated by CTB and therefore, according to the test publishers, it is appropriate to directly compare the results of the WKCE and Terra Nova
• DV = school level difference score representing growth in student mathematics achievement• Weighted average of individual difference scores was
calculated to obtain an overall measure of growth for each school
• Weighted average used because expected growth for each student is not equal to zero ((Dj-i) ≠ 0)
I = the lowest tested grade level served by a particular schoolj = the highest grade level served by a particular schooln(j - i) = the # of students with scale scores for both grades i
and j = the average of individual difference scores for students from grade j to grade I, and
N = total number of students used to calculate the overall growth
Dependent Variables - VAA
N
Dn ijij
j
Ii)()( −−
−∑
)( ijD −
Dependent Variables – Status based• DV = school level difference score
representing growth in student mathematics achievement• The difference in average scale scores for
each grade between the 2004-5 and 2003-4 school years was calculated and then the average difference across all grade levels was calculated for an overall measure of growth for each school
• Simple average used because the expected difference for each grade level is zero ((D2-1) ≠ 0)
• Does the degree in which schools’ learning teams embrace the vision of the MMP influence student growth in mathematics achievement test scores?
– Stepwise regression analysis used with five independent variables with four interaction terms• Measure of the frequency in which the MTL met with
the LT and had useful discussions about how to help students succeed in mathematics
• Climate of the LT LT collaboration
• Vision Plan submission Action Plan Submission
– Analyses exploratory in nature
Evaluation Question #1
Evaluation Question #2
• Does the level of satisfaction with time spent discussing mathematics, as reported by MTLs, and frequency of MTL attendance at district-wide MMP meetings influence student growth in mathematics achievement test scores?
– Stepwise regression analyses used with two independent variables with one interaction term• Number of district-wide MMP meetings attended by
the MTL• Satisfaction the MTL felt with the amount of time
spent discussing mathematics with others in their school
• The interaction of the above two variables
– Analysis exploratory in nature
Evaluation Questions #3 and #43) Is there a difference in student growth
in mathematics for schools for which MTLs report feeling a greater ability to accomplish goals regarding mathematics?
4) Is there a difference in student growth in mathematics for schools in which MTLs feel more supported by other math teachers in their school?
• Each used a one-way ANOVA with three levels on the DV
Results
• Very little statistically significant results were found– This is not surprising given that the
analyses conducted were based only on approximately six months of MMP activity
• Evaluation Question #1 – degree in which LTs embrace vision of the MMP– VAA – no IVs were significant enough to be
entered into the regression model– Status based – LT collaboration found to be
a significant predictor of student growth in math (F(1,88) = 5.06, p=.027, = -2.23, R2 = .05)
Results (con’t)• Evaluation Question #2 – MTL level of
satisfaction with time spent math and frequency of MTL attendance at MMP meetings– VAA and Status based – MTL attendance at
district-wide MMP meetings found to be a statistically significant predicator of overall school level growth in math• VAA: F(1,88) = 4.17, p=.044, = .98, R2 = .05
• Status based: F(1,90) = 6.37, p=.011, = 1.31, R2 = .07
– Encouraging that attendance seems to play a role in increasing student achievement
Results (con’t)
• Evaluation Questions #3 and #4 – MTLs feeling of ability to accomplish goals and perception of support from other math teachers– No statistically significant results
were found for either evaluation question for either accountability system
Discussion
• Statistical analyses did not detect many relationships between survey responses (IVs) and overall school level growth in mathematics– Not surprising given that the data
collected from the survey was obtained only six months after the inception of MMP activities
– Encouraging that a small positive effect on overall growth in student achievement in mathematics was found in terms of MTL attendance at MMP meetings
Discussion
• Differences found between the models highlight major flaws of using a status based growth model– Namely, the status based model
utilizes different cohorts of students to ascertain a measure of change in achievement
Discussion• Data exemplifying the major flaw of status based growth modelsMeans and SD of Difference Scores Obtained from VAA and Status Based Accountability Systems
VAA Mean SD Status Based
Mean SD
Overall 16.27 28.30 Overall -2.55 7.39
4th - 3rd Gr 23.29 33.30 3rd Gr -3.05 14.77
5th - 4th Gr 8.37 26.55 4th Gr -5.10 10.77
6th - 5th Gr 11.69 26.38 5th Gr -5.59 11.52
7th - 6th Gr 15.17 26.01 6th Gr 0.37 9.83
8th - 7th Gr 20.69 25.58 7th Gr -3.66 12.03
8th Gr 7.91 11.79
Discussion
• Support for VAA models– Use of longitudinal data to estimate
the impact of large-scale educational reform on student achievement•Compares student’s progress to his/her
past academic achievement rather than to a generic cohort of students
•Offers a picture of the rate at which a student is learning
•By tracking individual student’s progress from one year to the next, VAA models provide a control for pre-existing student differences
Discussion
• Practical disadvantages of VAA models– Require an extensive database of
longitudinal data •Student achievement scores•Student characteristic variables•Means of linking students to educational reform activities
– Requires reliable and valid methods of quantifying educational reform endeavors and outcomes
Discussion
• Theoretical concerns with using VAA– Model specification– Viability of using the model to claim
causal effects of educational reform on student learning
– Inclusion of covariates and possible confounding variables
– Use of achievement test data as the primary outcome measure
Discussion
• Theoretical concerns (con’t)– Effect of having missing data that is NOT
completely missing at random (CMAR)•CMAR data – there are not discernable
features that characterize the students who have missed a test from those who have not
•However, students who are more likely to miss tests often have certain characteristics that distinguish them from the population of students who take the tests
•Data that is NOT missing at random cannot be ignored because it introduces bias into the estimates of school-level effects
Discussion
• Theoretical concerns (con’t)– How best to take into account the
time at which student achievement tests are administered•Unclear how to best model growth in
achievement so that the growth is accurately distributed among the schools responsible for that growth
•Especially salient when school level variables are collected at a different point in time as when students are tested
Discussion
• Theoretical concerns (con’t)– How best to take into account the time at
which student achievement tests are administered is a major issue in trying to model how variability at the school level, which is related to the efforts of MMP, is having an impact on increasing student achievement in mathematics.
MMP CONUNDRUM: While the efforts of the MMP start in September and end in August, the school year runs from September to June, students are tested in November, and data is collected from teachers, learning team members, and MTLs in June.
Discussion• Theoretical concerns (con’t)
– Using growth as a DV implies the scale of measurement is constant across years• Vertical scaling assumes each test is measuring
a common trait so that the scales have identical meanings
• However, as students progress from one grade to the next, their curriculum progresses in content and difficulty and so tests designed for different grade levels may not allow for a measure of true growth
– Standardized achievement tests are subject to measurement error and the amount of error varies depending on student’s level of achievement
Discussion
• Problems encountered in MMP evaluation– Currently only school level variables
can be utilized– Lack of means to link teacher
responses to student achievement• Issues associated with teacher
confidentiality
– Lack of participation in data collection efforts•Likely that modeling classroom effects
would be statistically biased due to the magnitude of missing responses
Future Direction
• Conduct more refined analyses using data collected at the end of year two– On-line survey made available to all
MPS participants in the MMP that may have an effect on increasing student achievement in mathematics including•Learning Team Members•Literacy Coaches•Math Teacher Leaders•Math Teachers
Future Direction
• Data collected at the end of year two– Six primary areas were evaluated
1.Professional Development2.School/District Functioning3.Learning Team Functioning4.Classroom Practice5.Role of the MTL6.Role of the LC
Future Direction
• Data collected at the end of year two– Professional Development (PD)– the
extent to which those surveyed participated in, and were influenced by, PD activities supported by the MMP•Both formal and informal professional
development activities were considered•PD items were administered to all
participants
Future Direction
• Data collected at the end of year two– School/District Functioning
• The extent to which those surveyed feel involved in helping shape their school’s mathematics program
• Participant’s perception regarding the degree of alignment between the school and district goals
• Participant’s perception regarding the level of support provided by the Math Teaching Specialists and Learning Targets
• Items were administered to all participants
Future Direction
• Data collected at the end of year two– Learning Team Functioning
•The extent to which the Learning Team is functioning as a cohesive unit
•The degree to which the Learning Team is emphasizing school improvement efforts in mathematics in their work
• Items administered to all MTLs and all Learning Team members
Future Direction
• Data collected at the end of year two– Classroom Practice
•The extent to which math teachers engage their students in interactive vs. didactic instruction
•The amount of time math teachers devoted to each of the learning targets in MPS
• Items administered to all participants that reported teaching math in the classroom
Future Direction
• Data collected at the end of year two– Role of Math Teacher Leader (MTL)
•The extent to which MTLs met with others in their school to discuss the teaching and learning of mathematics
•The level of support MTLs felt was provided to them by others in their school and the MTSs
• Items administered only to MTLs
– Role of the Literacy Coach (LC)•The extent to which the LC collaborated with
and supported the MTL at their school• Items only administered to the LCs