the impact of including predictors and using various hierarchical linear models on evaluating school...
TRANSCRIPT
The Impact of The Impact of Including Predictors Including Predictors and Using Various and Using Various
Hierarchical Linear Hierarchical Linear Models on Evaluating Models on Evaluating
School Effectiveness in School Effectiveness in MathematicsMathematics
Nicole Traxel & Cindy WalkerNicole Traxel & Cindy Walker
University of Wisconsin - University of Wisconsin - MilwaukeeMilwaukee
April 14, 2009April 14, 2009The Milwaukee Mathematics Partnership (MMP) is supported by the National Science Foundation under Grant No. 0314898.
IntroductionIntroduction Value added modelsValue added models
Fair and accurate way to assess the effectiveness Fair and accurate way to assess the effectiveness of schoolsof schools
Determine how much value a school adds to Determine how much value a school adds to student learning by examining student progress student learning by examining student progress over timeover time
Operational definition: Effectiveness=growthOperational definition: Effectiveness=growth Hierarchical linear models can be used to Hierarchical linear models can be used to
implement a value added accountability implement a value added accountability systemsystem Hierarchical because students nested within Hierarchical because students nested within
schoolsschools Can determine how much of growth can be Can determine how much of growth can be
attributed to the student and to the schoolattributed to the student and to the school
Types of Hierarchical Types of Hierarchical Linear ModelsLinear Models
Several different hierarchical linear models Several different hierarchical linear models can be used to assess school effectiveness, can be used to assess school effectiveness, so which is best?so which is best? 2-level hierarchical model predicts final 2-level hierarchical model predicts final
achievement from initial achievement. Can achievement from initial achievement. Can include student level and school level include student level and school level predictors of achievement, not growthpredictors of achievement, not growth
2-level growth model predicts change in test 2-level growth model predicts change in test scores from one year to the next. Can include scores from one year to the next. Can include student level and school level predictors of student level and school level predictors of growth, not achievementgrowth, not achievement
3-level individual growth model predicts 3-level individual growth model predicts achievement and change over time. Can include achievement and change over time. Can include student level and school level predictors of student level and school level predictors of growth and achievementgrowth and achievement
Research QuestionsResearch Questions
Do effectiveness rankings differ Do effectiveness rankings differ depending on which type of model is depending on which type of model is used?used?
Does predictor significance remain Does predictor significance remain constant across model types?constant across model types?
Does including predictors change Does including predictors change effectiveness rankings of school?effectiveness rankings of school?
Sample & MeasuresSample & Measures 7,232 students from 128 school from a 7,232 students from 128 school from a
large urban school district in the Midwestlarge urban school district in the Midwest 33rdrd to 4 to 4thth grade grade 87% minority, 79% receive free/reduced lunch87% minority, 79% receive free/reduced lunch
Mathematics scores on a state mandated Mathematics scores on a state mandated standardized teststandardized test
Math Focus score for each school – “There Math Focus score for each school – “There is a strong focus on increasing student is a strong focus on increasing student achievement in mathematics at my school.”achievement in mathematics at my school.” Gain in Math Focus calculated by subtracting Gain in Math Focus calculated by subtracting
Math Focus score from 1Math Focus score from 1stst year from Math year from Math Focus score from 2Focus score from 2ndnd year year
The Models That Were The Models That Were FitFit
2-Level2-Level 2-Level Gain2-Level Gain 3-Level3-Level
Initial ScoreInitial Score NoneNone Initial ScoreInitial Score
Student RaceStudent Race Student RaceStudent Race Student RaceStudent Race
Student SESStudent SES Student SESStudent SES Student SESStudent SES
Student Race & Student Race & SESSES
Student Race & Student Race & SESSES
Student Race & Student Race & SESSES
Student & School Student & School RaceRace
Student & School Student & School RaceRace
Student & School Student & School RaceRace
Student & School Student & School SESSES
Student & School Student & School SESSES
Student & School Student & School SESSES
Student & School Student & School Race & SESRace & SES
Student & School Student & School Race & SESRace & SES
Student & School Student & School Race & SESRace & SES
Note: Initial Score was included as a student level covariate in all 2-level and 3-level models, but not in the 2-level gain models.
ComparisonsComparisons
Predictor significance across modelsPredictor significance across models Effectiveness rankings across Effectiveness rankings across
predictors being included within each predictors being included within each model typemodel type
Effectiveness rankings across model Effectiveness rankings across model types for models including only initial types for models including only initial score or no predictorsscore or no predictors
Effectiveness rankings across model Effectiveness rankings across model types for models including only initial types for models including only initial score or no predictors validated using score or no predictors validated using gain in Math Focus scoregain in Math Focus score
Predictor SignificancePredictor Significance Student-level SES and Student-level Student-level SES and Student-level
Race were significant predictors of the Race were significant predictors of the average achievement of students within average achievement of students within schools but not of the average growth of schools but not of the average growth of students within schoolsstudents within schools
School-level Race and School-level SES School-level Race and School-level SES were significant predictors of the were significant predictors of the average achievement among schools but average achievement among schools but not of the average growth among schoolsnot of the average growth among schools But not when both were included, due to But not when both were included, due to
collinearitycollinearity
Predictors Do Not Change Predictors Do Not Change EffectivenessEffectiveness2-Level Model 2-Level Gain Model 3-Level Model
Student Level Race 0.999 1.000 1.000
Student Level SES 0.991 1.000 0.999
Student Level Race & SES
0.991 0.999 0.999
Student & School Level Race
0.996 0.999 0.998
Student & School Level SES
0.989 0.999 0.998
Student & School Level Race & SES
0.988 0.998 0.997
Spearman Correlations between the null model and models including predictors for each model type.
Correlations Among Model Correlations Among Model Types (Null Models)Types (Null Models)
2-level and 2-level gain models 2-level and 2-level gain models rr = .090 = .090
2-level and 3-level models2-level and 3-level models rr = .101 = .101
2-level gain and 3-level models2-level gain and 3-level models rr = .993 = .993
Therefore, rankings from 2-level Therefore, rankings from 2-level gain and 3-level models are very gain and 3-level models are very similar to one anothersimilar to one another
Validating Effectiveness Validating Effectiveness RankingsRankings
Pearson correlations between gain Pearson correlations between gain in math focus and effectiveness in math focus and effectiveness estimates from null models of each estimates from null models of each model type.model type. 2-level: 2-level: rr = .034, = .034, pp = .747 = .747 2-level gain: 2-level gain: rr = .200, = .200, pp = .055 = .055 3-level: 3-level: rr = .177, = .177, pp = .089 = .089
Conclusions, Part OneConclusions, Part One
Including predictors, even if they are Including predictors, even if they are significant, does not change significant, does not change effectiveness estimateseffectiveness estimates
Effectiveness estimates from null 2-level Effectiveness estimates from null 2-level model were different from effectiveness model were different from effectiveness estimates from null 2-level gain and 3-estimates from null 2-level gain and 3-level modelslevel models
Effectiveness estimates from 2-level gain Effectiveness estimates from 2-level gain and 3-level models were very similarand 3-level models were very similar
Conclusions, Part TwoConclusions, Part Two
Correlation between 2-level gain and Correlation between 2-level gain and 3-level models and gain in math 3-level models and gain in math focus had higher magnitudes than focus had higher magnitudes than correlation between 2-level model correlation between 2-level model and gain in math focusand gain in math focus
Effectiveness estimates from 2-level Effectiveness estimates from 2-level gain and 3-level models are more gain and 3-level models are more valid than those from 2-level modelvalid than those from 2-level model
So which model type should So which model type should I use?I use?
3-level model has several advantages 3-level model has several advantages over 2-level gain modelover 2-level gain model Includes ALL available data-all Includes ALL available data-all
participants with at least one participants with at least one observation are includedobservation are included
Can include many years of dataCan include many years of data Provides more information (growth and Provides more information (growth and
achievement estimates)achievement estimates)