pattern analysis and data visualization of student data ... · pattern analysis and data...
TRANSCRIPT
Pattern Analysis and Data Visualization of
Student Data for Decision Making
Alex J. Bowers, Ph.D.
Associate Professor of Education Leadership
Teachers College, Columbia University
February 14, 2013
A research seminar for the Teachers College
Learning Analytics Program
Teacher Assigned Grades as Useful Data to
Inform Decision Making in Schools
• What are the most accurate predictors of who
will dropout or graduate? Do we know?
– Grade Trajectories (Growth Mixture Models)
– Grade Patterns (Cluster Analysis Heatmaps)
• What do grades represent as useful data for
decision making?
Bowers, A.J. (2013)
Teacher Assigned Grades as Useful Data to
Inform Decision Making in Schools
• What are the most accurate predictors of who
will dropout or graduate? Do we know?
– Grade Trajectories (Growth Mixture Models)
– Grade Patterns (Cluster Analysis Heatmaps)
• What do grades represent as useful data for
decision making?
Bowers, A.J. (2013)
Issues of Accuracy in Predicting High School Dropout
• Dropping out of high school in the U.S. is associated with a multitude of negative outcomes.
• However, other than a select number of demographic and background variables, we know little about the accuracy of current dropout predictors that are school-related.
• Current predictions accurately predict only about 50-60% of students who actually dropout.
• According to Gleason & Dynarski (2002), accurate prediction of who will dropout is a resource and efficiency issue:
– A large percentage of students are mis-identified as at-risk.
– A large percentage of students who are at-risk are never identified.
• Current reporting of dropout “flags” across the literature is haphazard. – Almost none report accuracy
– Many report specificity or sensitivity, but rarely both
• A dropout flag may be highly precise, in that almost all of the students with the flag dropout, but may not be accurate since the flag may identify only a small proportion of the dropouts.
Balfanz et al. (2007); Bowers (2010); Gleason & Dynarski (2002); Pallas (2003); Rumberger (2004, 2011) Bowers, A.J. (2013)
Re-analyzing Past Dropout Flags for Accuracy,
Precision, Sensitivity and Specificity
• Literature search. • Queried multiple databases:
– JSTOR, Google Scholar, EBSCO, Educational Full Text Wilson Web
• Studies were included that: – Were published since 1979 – Examined High School dropout – Examined school-wide characteristics and included all students – A focus on the student level – Reported frequencies for re-analysis
• Initially yield 6,434 overlapping studies – 301 studies were read in full – 140 provided school-wide samples and quantifiable data – 36 articles provided enough data for accuracy re-calculations – Yield 110 separate dropout flags
• Relative Operating Characteristic (ROC) – Hits versus False-Alarms
Bowers, Sprott & Taff (2013) Do we Know Who Will Drop Out? A Review of the Predictors of Dropping out of
High School: Precision, Sensitivity and Specificity. The High School Journal. 96(2), 77-100.
Bowers, A.J. (2013)
Dropout Graduate
Dropout a
True-positive
(TP)
Correct
b
False-positive
(FP)
Type I Error
a+b
Graduate c
False-negative
(FN)
Type II Error
d
True-negative
(TN)
Correct
c+d
a+c b+d a+b+c+d=N
Event
Pre
dic
tor
Precision = a/(a + b) Positive Predictive Value
True-Positive Proportion = a/(a + c) Sensitivity
True-Negative Proportion = d/(b + d) Specificity
False-Positive Proportion = b/(b + d) 1-Specificity
Event table for calculating dropout contingency proportions
Fawcett (2004); Hanley & McNeil (1982); Swets (1988); Zwieg & Campbell (1993)
“Hits”
“False Alarms”
Bowers, A.J. (2013)
0
0.2
0.4
0.6
0.8
1.0
Tru
e-p
ositiv
e p
roport
ion (
Sensitiv
ity)
0 0.2 0.4 0.6 0.8 1.0
False-positive proportion (1-Specificity)
Perfect prediction
Better prediction
Worse prediction
Low attendance
An example of the true-positive proportion plotted against the false-positive
proportion for Balfanz et al. (2007) comparing the relative operating characteristics
(ROC) of each dropout flag.
Bowers, A.J. (2013)
0
0.2
0.4
0.6
0.8
1.0
Tru
e-p
ositiv
e p
roport
ion (
Sensitiv
ity)
0 0.2 0.4 0.6 0.8 1.0
False-positive proportion (1-Specificity)
Relative operating characteristics (ROC) of all dropout flags reviewed, plotted
as the true-positive proportion against the false-positive proportion. Numbers
refer to dropout indicator IDs.
Bowers, A.J., Sprott, R., Taff, S.A. (2013) Do we Know Who Will Drop Out? A Review of the Predictors of Dropping out of High School: Precision, Sensitivity and Specificity. The High School Journal. 96(2), 77-100. http://muse.jhu.edu/journals/high_school_journal/v096/96.2.bowers.html
0
0.2
0.4
0.6
0.8
1.0
Tru
e-p
ositiv
e p
roport
ion (
Sensitiv
ity)
0 0.2 0.4 0.6 0.8 1.0
False-positive proportion (1-Specificity)
Relative operating characteristics (ROC) of all dropout flags reviewed, plotted
as the true-positive proportion against the false-positive proportion. Numbers
refer to dropout indicator IDs.
Bowers, A.J., Sprott, R., Taff, S.A. (2013) Do we Know Who Will Drop Out? A Review of the Predictors of Dropping out of High School: Precision, Sensitivity and Specificity. The High School Journal. 96(2), 77-100. http://muse.jhu.edu/journals/high_school_journal/v096/96.2.bowers.html
Growth mixture model for the simultaneous estimation of latent trajectory classes
using non-cumulative GPA from the first three semesters of high school.
GPA 9S1 GPA 9S2 GPA 10S1
Intercepts Slopes
High School
Dropout
Behavior & Structure:
Student:
Extracurricular
Retained
Negative Behavior
School:
Student-Teacher Ratio
Academic Press
Small School
Large School
Extra-Large School
Demographics:
Student:
Female
African American
Asian
Hispanic
Non-Traditional Family
SES
School:
Urban
Rural
% Students Free Lunch
Latent
Trajectory
Classes
C
Bowers, A.J., Sprott, R. (2012) Examining the Multiple Trajectories Associated with Dropping Out of High School: A Growth Mixture Model Analysis. The Journal of Educational Research, 105(3), 176-195.doi:10.1080/00220671.2011.552075
Bowers, A.J. (2013)
A:
B:
0
1
2
3
4
9S1 9S2 10S1Semester
No
n-C
um
ula
tive
GP
A
Mid-DecreasingLow-IncreasingMid-AchievingHigh-Achieving
0
1
2
3
4
9S1 9S2 10S1
Semester
0
1
2
3
4
9S1 9S2 10S1
0
1
2
3
4
0
1
2
3
4
9S1 9S2 10S1
Semester
0
Non
-cum
ula
tive
GP
A
1
2
3
4
9S1 9S2 10S1
Semester
0
1
2
3
4
9S1 9S2 10S1
Semester
0
1
2
3
4
9S1 9S2 10S1
0
1
2
3
4
9S1 9S2 10S1
Semester
0
1
2
3
4
9S1 9S2 10S1
Semester
0
1
2
3
4
9S1 9S2 10S1
0
1
2
3
4
9S1 9S2 10S1
Semester
Low-Increasing: 13.8%Mid-Decreasing: 10.8% Mid-Achieving: 56.5% High-Achieving: 18.9%
Dropout: 39.7% 52.1% 7.0% 1.2%
Longitudinal Non-Cumulative GPA Trajectories in the First
Three Semesters of High School
n=5,400
Mid-decreasing & Low-increasing accounted for:
24.6% of the sample
91.% of the dropouts
Bowers, A.J., Sprott, R. (2012) Examining the Multiple Trajectories Associated with Dropping Out of High School: A Growth Mixture Model Analysis. The Journal of Educational Research, 105(3), 176-195.doi:10.1080/00220671.2011.552075
Bowers, A.J. (2013)
Teacher Assigned Grades as Useful Data to
Inform Decision Making in Schools
• What are the most accurate predictors of who
will dropout or graduate? Do we know?
– Grade Trajectories (Growth Mixture Models)
– Grade Patterns (Cluster Analysis Heatmaps)
• What do grades represent as useful data for
decision making?
Bowers, A.J. (2013)
Grades as Useful Information for
Data Driven Decision Making (3DM)
Bowers, A.J. (2013)
Grades as Useful Information for
Data Driven Decision Making (3DM)
• In the U.S. we have a dual assessment system
– Standardized test scores are reported to administrators, policymakers and
the state
– Teacher assigned grades that are reported to students and parents
– Bowers (2010,2011); Farr (2000)
• However little criterion validity exists for standardized tests and their link
to overall schooling outcomes, while grades are well known to be
predictive of high school graduation and college success.
• Can we leverage grades, as data we already collect in schools, to help school leaders predict future student performance and on-time graduation in an effort to improve educational practice by providing fine-grained data to help direct the limited resources of a school district to specific student needs?
Bowers, A.J. (2010) Analyzing the Longitudinal K-12 Grading Histories of Entire Cohorts of Students: Grades, Data Driven Decision Making, Dropping Out and Hierarchical Cluster Analysis. Practical Assessment, Research & Evaluation (PARE), 15(7), 1-18. http://pareonline.net/pdf/v15n7.pdf
Bowers, A.J. (2013)
Research Questions
1. To what extent do past student grade patterns predict future student grade patterns?
2. To what extent do student grade patterns predict on-time graduation or student dropout?
Sample • Two student cohorts: The entire 2006 graduating cohorts for two small
Midwestern first-ring suburban school districts, enrollment < 3000 for both districts.
• 188 students included in the sample
• Overall, as defined by the state, these two districts serve students who are about 60% economically disadvantaged, 60% European American,13% Hispanic, 6% African American, 2% Asian, 2% multi-ethnic, with 17% non-reporting
Bowers, A.J. (2010) Analyzing the Longitudinal K-12 Grading Histories of Entire Cohorts of Students: Grades, Data Driven Decision Making, Dropping Out and Hierarchical Cluster Analysis. Practical Assessment, Research & Evaluation (PARE), 15(7), 1-18. http://pareonline.net/pdf/v15n7.pdf Bowers, A.J. (2013)
Method
• Recorded all assessment records for each student in each of the two cohorts, K-12. About 650 variables per student.
• The dataset mirrors typical district assessment datasets that often go unexamined by district and school leaders
• Converted all grades to a 4 point scale
• Hierarchical Cluster Analysis to identify patterns predictive of high school dropout
Bowers, A.J. (2010) Analyzing the Longitudinal K-12 Grading Histories of Entire Cohorts of Students: Grades, Data Driven Decision Making, Dropping Out and Hierarchical Cluster Analysis. Practical Assessment, Research & Evaluation (PARE), 15(7), 1-18. http://pareonline.net/pdf/v15n7.pdf Bowers, A.J. (2013)
Cluster Analysis
• A descriptive statistic
• Fewer assumption violation issues than with regression statistics
• Uses all of the available data
• Rarely used in education
Bowers, A.J. (2010) Analyzing the Longitudinal K-12 Grading Histories of Entire Cohorts of Students: Grades, Data Driven Decision Making, Dropping Out and Hierarchical Cluster Analysis. Practical Assessment, Research & Evaluation (PARE), 15(7), 1-18. http://pareonline.net/pdf/v15n7.pdf Bowers, A.J. (2013)
Student 1
Student 2
Student 3
Student 4
Student 5
Student 6
Student 7
Student 8
A
B
C
D
F
K 1 2 3 4 5 6 7 8 9 10 11 12
Grade Year
Gra
de M
ark
ing
Hypothetical Hierarchical Clustering Data
Bowers, A.J. (2010) Analyzing the Longitudinal K-12 Grading Histories of Entire Cohorts of Students: Grades, Data Driven Decision Making, Dropping Out and Hierarchical Cluster Analysis. Practical Assessment, Research & Evaluation (PARE), 15(7), 1-18. http://pareonline.net/pdf/v15n7.pdf Bowers, A.J. (2013)
K 1 2 3 4 5 6 7 8 9 10 1112
Grade Year
A B C D F
Hypothetical Hierarchical Clustering Data: Clustergram
Student 3
Student 7
Student 5
Student 4
Student 8
Student 6
Student 1
Student 2
Bowers, A.J. (2010) Analyzing the Longitudinal K-12 Grading Histories of Entire Cohorts of Students: Grades, Data Driven Decision Making, Dropping Out and Hierarchical Cluster Analysis. Practical Assessment, Research & Evaluation (PARE), 15(7), 1-18. http://pareonline.net/pdf/v15n7.pdf Bowers, A.J. (2013)
K Elementary MS 9th 10th 11th 12th
Hierarchical Clustering of Teacher Assigned Subject-Specific Grades
Clu
ste
r T
ree
Su
bje
cts
Students
+3 -3 0 No Data Bowers, A.J. (2013)
Hierarchical Clustering of Grades
K Elementary MS 9th 10th 11th 12th
Bowers, A.J. (2013)
Hierarchical Clustering of Grades
K Elementary MS 9th 10th 11th 12th High-High
Low-Low
Low-High
High-Low
Bowers, A.J. (2013)
0
0.5
1
1.5
2
2.5
3
3.5
4
K 1 2 3 4 5 6 7 8 9S1 9S2 10S1 10S2 11S1 11S2 12S1 12S2
Grade-Level
Mean N
on-c
um
ula
tive G
PA
Mean non-cumulative GPA trends for clusters high-high,
low-low, low-high and high-low, K-12
High-High Low-Low High-Low Low-High
Bowers, A.J. (2013)
• Examine why teacher assigned grades are predictive of dropout
• Further examine different types of high school dropouts
• Replicate and extend cluster & pattern analysis – Collaborating with local districts in Texas
– Longitudinal patterns of 10,000 student cohorts
– Test additional clustering algorithms • Self Organizing Maps
• Multi-level GMM
• Survival models
– Testing interventions for 3DM
Future Work K12 Schooling Outcomes
Bowers, A.J. (2010) Analyzing the Longitudinal K-12 Grading Histories of Entire Cohorts of Students: Grades, Data Driven Decision Making, Dropping Out and Hierarchical Cluster Analysis. Practical Assessment, Research & Evaluation (PARE), 15(7), 1-18. http://pareonline.net/pdf/v15n7.pdf Bowers, A.J. (2013)
Teacher Assigned Grades as Useful Data to
Inform Decision Making in Schools
• What are the most accurate predictors of who
will dropout or graduate? Do we know?
– Grade Trajectories (Growth Mixture Models)
– Grade Patterns (Cluster Analysis Heatmaps)
• What do grades represent as useful data for
decision making?
Bowers, A.J. (2013)
Grades and Standardized Tests
• In the U.S. we have a dual assessment system – Standardized test scores that are reported to administrators,
policymakers and the state – Teacher assigned grades that are reported to students and
parents
• However little criterion validity exists for standardized tests and their link to overall schooling outcomes, while grades are well known to be highly predictive of high school graduation and college success
• The purpose of the study is to describe the multi-
dimensional nature of grades as an assessment of both academic knowledge and an “other” dimension
Bowers, 2009; Farr, 2000
Bowers, A.J. (2013)
Hodge-podge & Kitchen Sink Grading
• Grades are historically seen as “hodge-podge” and “kitchen sink” subjective measures of student performance that include student: – Academic knowledge – Participation – Discipline – Attendance – Homework – Engagement
• Researchers encourage teachers to move to more standardized grading
Brookhart, 1991; Cizek, 2000; Cross & Frary, 1999; Kuncel, 2005; Trumbull 2000
Bowers, A.J. (2013)
What is it that Grades Assess?
We must ask, "if hodgepodge grading is so deplorable, why haven't students, parents and administrators, or the general public called for reform?" It may well be that they share a common understanding that grades often do, in fact, represent a hodgepodge of attitude, effort, conduct, growth, and achievement and that is what they expect and endorse. (Cross & Frary, 1999, p.70)
• “Conative” skills - Engagement – Willingham, Pollack & Lewis, 2002
• Common grade dimension – Klapp Lekholm & Cliffordson, 2008, 2009, Thorsen & Cliffordson, 2012
• Success at school factor (SSF) – Bowers, 2009, 2011
Bowers, A.J. (2013)
Standardized Tests
Grades
25% Shared Variance
The Difference in What Standardized Tests and Grades May Assess
Bowers, A.J. (2011) What's in a Grade? The Multidimensional Nature of What Teacher Assigned Grades Assess in High School. Educational Research & Evaluation, 17(3), 141-159. doi: 10.1080/13803611.2011.597112
Bowers, A.J. (2013)
Core Grades
Non-Core Grades
Standardized Tests Academic Knowledge
???
The Difference in What Standardized Tests and Grades May Assess
Bowers, A.J. (2013)
Research Questions
• To what extent are teacher assigned grades and standardized test scores related?
• To what extent is the relationship multi-dimensional across core and non-core grades versus standardized tests.
• To what extent does the relationship between test scores and grades vary across different high school course subjects?
Bowers, A.J. (2011) What's in a Grade? The Multidimensional Nature of What Teacher Assigned Grades Assess in High School. Educational Research & Evaluation, 17(3), 141-159. doi: 10.1080/13803611.2011.597112
Bowers, A.J. (2013)
Sample & Method
• Education Longitudinal Study 2002 (ELS:2002) Transcript Survey Component – NCES collected all grades in all subjects 9-12 – Tested students in reading and mathematics grade 10
• Standardized test equated to NAEP and PISA.
– n=5,230 students with complete data
• Teacher Assigned Grades – Extracted 638,000 individual course grades across 1,600 different course
subject titles – Letter grades scaled from 4.0 (A) to 0 (F) – Classification of Secondary School Courses (CSSC) to aggregate grades into
seven main course topics: • Core courses: mathematics, English, science, social studies • Non-core courses: foreign/Non-English Lang, physical education (PE), art
– Average grades calculated for each semester of grades 9-12, and for each core and non-core subject for high school
• Method – Multi-Dimensional Scaling (MDS) – Grades and standardized test scores z-scored – Ordinal MDS with a Euclidean distance measure – 2-dimensional MDS across time, 3-dimensional MDS across subjects
Bowers, A.J. (2013)
MDS of standardized tests and core and non-core GPA – Grades 9-12
-3 -2 -1 0 1 2 3
Dimension 1
0
-1
1.0
1.5
0.5
-0.5
-1.5
Dim
ensio
n 2
Reading
Mathematics
12S1
12S2
12S1
12S2
9S1
9S2
11S2
11S1 10S210S1
11S2
11S1 10S210S1
9S19S2
Standardized Test Core GPA Non-Core GPA
Legend:
-3 -2 -1 0 1 2 3
Dimension 1
0
-1
1.0
1.5
0.5
-0.5
-1.5
Dim
ensio
n 2
Reading
Mathematics
12S1
12S2
12S1
12S2
9S1
9S2
11S2
11S1 10S210S1
11S2
11S1 10S210S1
9S19S2
-3 -2 -1 0 1 2 3
Dimension 1
-3 -2 -1 0 1 2 3
Dimension 1
0
-1
1.0
1.5
0.5
-0.5
-1.5
Dim
ensio
n 2
Reading
Mathematics
12S1
12S2
12S1
12S2
9S1
9S2
11S2
11S1 10S210S1
11S2
11S1 10S210S1
9S19S2
Standardized Test Core GPA Non-Core GPA
Legend:
Standardized Test Core GPA Non-Core GPA
Legend:
Bowers, A.J. (2011) What's in a Grade? The Multidimensional Nature of What Teacher Assigned Grades Assess in High School. Educational Research & Evaluation, 17(3), 141-159. doi: 10.1080/13803611.2011.597112 Bowers, A.J. (2013)
-3 -2 -1 0 1 2 3
Dimension 1
0
-1
1.0
1.5
0.5
-0.5
-1.5
Dim
ensio
n 2
Reading
Mathematics
ArtPE
Foreign Languages
Mathematics
Science
English
Social Studies
-3 -2 -1 0 1 2 3
Dimension 1
0
-1
1.0
1.5
0.5
-0.5
-1.5
Dim
ensio
n 2
Reading
Mathematics
ArtPE
Foreign Languages
Mathematics
Science
English
Social Studies
-3 -2 -1 0 1 2 3
Dimension 1
0
-1
1.0
1.5
0.5
-0.5
-1.5
Dim
ensio
n 3 Reading
Mathematics
Art
PE
Foreign Languages
Mathematics
Science
English
Social Studies
-3 -2 -1 0 1 2 3
Dimension 1
0
-1
1.0
1.5
0.5
-0.5
-1.5
Dim
ensio
n 3 Reading
Mathematics
Art
PE
Foreign Languages
Mathematics
Science
English
Social Studies
Dimension 1 Dimension 3
Dim
ensio
n 2
Reading
Math
PE
MathScience
Social Studies
Foreign LangArt
English
Standardized Test Core Subject GPA Non-Core Subject GPA
Legend:
A.
B.
C.-3 -2 -1 0 1 2 3
Dimension 1
0
-1
1.0
1.5
0.5
-0.5
-1.5
Dim
ensio
n 2
Reading
Mathematics
ArtPE
Foreign Languages
Mathematics
Science
English
Social Studies
-3 -2 -1 0 1 2 3
Dimension 1
0
-1
1.0
1.5
0.5
-0.5
-1.5
Dim
ensio
n 2
Reading
Mathematics
ArtPE
Foreign Languages
Mathematics
Science
English
Social Studies
-3 -2 -1 0 1 2 3
Dimension 1
0
-1
1.0
1.5
0.5
-0.5
-1.5
Dim
ensio
n 3 Reading
Mathematics
Art
PE
Foreign Languages
Mathematics
Science
English
Social Studies
-3 -2 -1 0 1 2 3
Dimension 1
0
-1
1.0
1.5
0.5
-0.5
-1.5
Dim
ensio
n 3 Reading
Mathematics
Art
PE
Foreign Languages
Mathematics
Science
English
Social Studies
Dimension 1 Dimension 3
Dim
ensio
n 2
Reading
Math
PE
MathScience
Social Studies
Foreign LangArt
English
Dimension 1 Dimension 3
Dim
ensio
n 2
Dimension 1 Dimension 3
Dim
ensio
n 2
Dimension 1 Dimension 3
Dim
ensio
n 2
Reading
Math
PE
MathScience
Social Studies
Foreign LangArt
English
Standardized Test Core Subject GPA Non-Core Subject GPA
Legend:
A.
B.
C.
-3 -2 -1 0 1 2 3
Dimension 1
0
-1
1.0
1.5
0.5
-0.5
-1.5
Dim
ensio
n 2
Reading
Mathematics
ArtPE
Foreign Languages
Mathematics
Science
English
Social Studies
-3 -2 -1 0 1 2 3
Dimension 1
0
-1
1.0
1.5
0.5
-0.5
-1.5
Dim
ensio
n 2
Reading
Mathematics
ArtPE
Foreign Languages
Mathematics
Science
English
Social Studies
-3 -2 -1 0 1 2 3
Dimension 1
0
-1
1.0
1.5
0.5
-0.5
-1.5
Dim
ensio
n 3 Reading
Mathematics
Art
PE
Foreign Languages
Mathematics
Science
English
Social Studies
-3 -2 -1 0 1 2 3
Dimension 1
0
-1
1.0
1.5
0.5
-0.5
-1.5
Dim
ensio
n 3 Reading
Mathematics
Art
PE
Foreign Languages
Mathematics
Science
English
Social Studies
Dimension 1 Dimension 3
Dim
ensio
n 2
Reading
Math
PE
MathScience
Social Studies
Foreign LangArt
English
Standardized Test Core Subject GPA Non-Core Subject GPA
Legend:
A.
B.
C.-3 -2 -1 0 1 2 3
Dimension 1
0
-1
1.0
1.5
0.5
-0.5
-1.5
Dim
ensio
n 2
Reading
Mathematics
ArtPE
Foreign Languages
Mathematics
Science
English
Social Studies
-3 -2 -1 0 1 2 3
Dimension 1
0
-1
1.0
1.5
0.5
-0.5
-1.5
Dim
ensio
n 2
Reading
Mathematics
ArtPE
Foreign Languages
Mathematics
Science
English
Social Studies
-3 -2 -1 0 1 2 3
Dimension 1
0
-1
1.0
1.5
0.5
-0.5
-1.5
Dim
ensio
n 3 Reading
Mathematics
Art
PE
Foreign Languages
Mathematics
Science
English
Social Studies
-3 -2 -1 0 1 2 3
Dimension 1
0
-1
1.0
1.5
0.5
-0.5
-1.5
Dim
ensio
n 3 Reading
Mathematics
Art
PE
Foreign Languages
Mathematics
Science
English
Social Studies
Dimension 1 Dimension 3
Dim
ensio
n 2
Reading
Math
PE
MathScience
Social Studies
Foreign LangArt
English
Dimension 1 Dimension 3
Dim
ensio
n 2
Dimension 1 Dimension 3
Dim
ensio
n 2
Dimension 1 Dimension 3
Dim
ensio
n 2
Reading
Math
PE
MathScience
Social Studies
Foreign LangArt
English
Standardized Test Core Subject GPA Non-Core Subject GPA
Legend:
A.
B.
C.
Three-dimensional MDS of standardized tests, grades in core subjects, and grades in non-core subjects, n=5230
Bowers, A.J. (2013)
-3 -2 -1 0 1 2 3
Dimension 1
0
-1
1.0
1.5
0.5
-0.5
-1.5
Dim
ensio
n 2
Reading
Mathematics
ArtPE
Foreign Languages
Mathematics
Science
English
Social Studies
-3 -2 -1 0 1 2 3
Dimension 1
0
-1
1.0
1.5
0.5
-0.5
-1.5
Dim
ensio
n 2
Reading
Mathematics
ArtPE
Foreign Languages
Mathematics
Science
English
Social Studies
-3 -2 -1 0 1 2 3
Dimension 1
0
-1
1.0
1.5
0.5
-0.5
-1.5
Dim
ensio
n 3 Reading
Mathematics
Art
PE
Foreign Languages
Mathematics
Science
English
Social Studies
-3 -2 -1 0 1 2 3
Dimension 1
0
-1
1.0
1.5
0.5
-0.5
-1.5
Dim
ensio
n 3 Reading
Mathematics
Art
PE
Foreign Languages
Mathematics
Science
English
Social Studies
Dimension 1 Dimension 3
Dim
ensio
n 2
Reading
Math
PE
MathScience
Social Studies
Foreign LangArt
English
Standardized Test Core Subject GPA Non-Core Subject GPA
Legend:
A.
B.
C.-3 -2 -1 0 1 2 3
Dimension 1
0
-1
1.0
1.5
0.5
-0.5
-1.5
Dim
ensio
n 2
Reading
Mathematics
ArtPE
Foreign Languages
Mathematics
Science
English
Social Studies
-3 -2 -1 0 1 2 3
Dimension 1
0
-1
1.0
1.5
0.5
-0.5
-1.5
Dim
ensio
n 2
Reading
Mathematics
ArtPE
Foreign Languages
Mathematics
Science
English
Social Studies
-3 -2 -1 0 1 2 3
Dimension 1
0
-1
1.0
1.5
0.5
-0.5
-1.5
Dim
ensio
n 3 Reading
Mathematics
Art
PE
Foreign Languages
Mathematics
Science
English
Social Studies
-3 -2 -1 0 1 2 3
Dimension 1
0
-1
1.0
1.5
0.5
-0.5
-1.5
Dim
ensio
n 3 Reading
Mathematics
Art
PE
Foreign Languages
Mathematics
Science
English
Social Studies
Dimension 1 Dimension 3
Dim
ensio
n 2
Reading
Math
PE
MathScience
Social Studies
Foreign LangArt
English
Dimension 1 Dimension 3
Dim
ensio
n 2
Dimension 1 Dimension 3
Dim
ensio
n 2
Dimension 1 Dimension 3
Dim
ensio
n 2
Reading
Math
PE
MathScience
Social Studies
Foreign LangArt
English
Standardized Test Core Subject GPA Non-Core Subject GPA
Legend:
A.
B.
C.
-3 -2 -1 0 1 2 3
Dimension 1
0
-1
1.0
1.5
0.5
-0.5
-1.5
Dim
ensio
n 2
Reading
Mathematics
ArtPE
Foreign Languages
Mathematics
Science
English
Social Studies
-3 -2 -1 0 1 2 3
Dimension 1
0
-1
1.0
1.5
0.5
-0.5
-1.5
Dim
ensio
n 2
Reading
Mathematics
ArtPE
Foreign Languages
Mathematics
Science
English
Social Studies
-3 -2 -1 0 1 2 3
Dimension 1
0
-1
1.0
1.5
0.5
-0.5
-1.5
Dim
ensio
n 3 Reading
Mathematics
Art
PE
Foreign Languages
Mathematics
Science
English
Social Studies
-3 -2 -1 0 1 2 3
Dimension 1
0
-1
1.0
1.5
0.5
-0.5
-1.5
Dim
ensio
n 3 Reading
Mathematics
Art
PE
Foreign Languages
Mathematics
Science
English
Social Studies
Dimension 1 Dimension 3
Dim
ensio
n 2
Reading
Math
PE
MathScience
Social Studies
Foreign LangArt
English
Standardized Test Core Subject GPA Non-Core Subject GPA
Legend:
A.
B.
C.-3 -2 -1 0 1 2 3
Dimension 1
0
-1
1.0
1.5
0.5
-0.5
-1.5
Dim
ensio
n 2
Reading
Mathematics
ArtPE
Foreign Languages
Mathematics
Science
English
Social Studies
-3 -2 -1 0 1 2 3
Dimension 1
0
-1
1.0
1.5
0.5
-0.5
-1.5
Dim
ensio
n 2
Reading
Mathematics
ArtPE
Foreign Languages
Mathematics
Science
English
Social Studies
-3 -2 -1 0 1 2 3
Dimension 1
0
-1
1.0
1.5
0.5
-0.5
-1.5
Dim
ensio
n 3 Reading
Mathematics
Art
PE
Foreign Languages
Mathematics
Science
English
Social Studies
-3 -2 -1 0 1 2 3
Dimension 1
0
-1
1.0
1.5
0.5
-0.5
-1.5
Dim
ensio
n 3 Reading
Mathematics
Art
PE
Foreign Languages
Mathematics
Science
English
Social Studies
Dimension 1 Dimension 3
Dim
ensio
n 2
Reading
Math
PE
MathScience
Social Studies
Foreign LangArt
English
Dimension 1 Dimension 3
Dim
ensio
n 2
Dimension 1 Dimension 3
Dim
ensio
n 2
Dimension 1 Dimension 3
Dim
ensio
n 2
Reading
Math
PE
MathScience
Social Studies
Foreign LangArt
English
Standardized Test Core Subject GPA Non-Core Subject GPA
Legend:
A.
B.
C.
-3 -2 -1 0 1 2 3
Dimension 1
0
-1
1.0
1.5
0.5
-0.5
-1.5
Dim
ensio
n 2
Reading
Mathematics
ArtPE
Foreign Languages
Mathematics
Science
English
Social Studies
-3 -2 -1 0 1 2 3
Dimension 1
0
-1
1.0
1.5
0.5
-0.5
-1.5
Dim
ensio
n 2
Reading
Mathematics
ArtPE
Foreign Languages
Mathematics
Science
English
Social Studies
-3 -2 -1 0 1 2 3
Dimension 1
0
-1
1.0
1.5
0.5
-0.5
-1.5
Dim
ensio
n 3 Reading
Mathematics
Art
PE
Foreign Languages
Mathematics
Science
English
Social Studies
-3 -2 -1 0 1 2 3
Dimension 1
0
-1
1.0
1.5
0.5
-0.5
-1.5
Dim
ensio
n 3 Reading
Mathematics
Art
PE
Foreign Languages
Mathematics
Science
English
Social Studies
Dimension 1 Dimension 3
Dim
ensio
n 2
Reading
Math
PE
MathScience
Social Studies
Foreign LangArt
English
Standardized Test Core Subject GPA Non-Core Subject GPA
Legend:
A.
B.
C.-3 -2 -1 0 1 2 3
Dimension 1
0
-1
1.0
1.5
0.5
-0.5
-1.5
Dim
ensio
n 2
Reading
Mathematics
ArtPE
Foreign Languages
Mathematics
Science
English
Social Studies
-3 -2 -1 0 1 2 3
Dimension 1
0
-1
1.0
1.5
0.5
-0.5
-1.5
Dim
ensio
n 2
Reading
Mathematics
ArtPE
Foreign Languages
Mathematics
Science
English
Social Studies
-3 -2 -1 0 1 2 3
Dimension 1
0
-1
1.0
1.5
0.5
-0.5
-1.5
Dim
ensio
n 3 Reading
Mathematics
Art
PE
Foreign Languages
Mathematics
Science
English
Social Studies
-3 -2 -1 0 1 2 3
Dimension 1
0
-1
1.0
1.5
0.5
-0.5
-1.5
Dim
ensio
n 3 Reading
Mathematics
Art
PE
Foreign Languages
Mathematics
Science
English
Social Studies
Dimension 1 Dimension 3
Dim
ensio
n 2
Reading
Math
PE
MathScience
Social Studies
Foreign LangArt
English
Dimension 1 Dimension 3
Dim
ensio
n 2
Dimension 1 Dimension 3
Dim
ensio
n 2
Dimension 1 Dimension 3
Dim
ensio
n 2
Reading
Math
PE
MathScience
Social Studies
Foreign LangArt
English
Standardized Test Core Subject GPA Non-Core Subject GPA
Legend:
A.
B.
C.
-3 -2 -1 0 1 2 3
Dimension 1
0
-1
1.0
1.5
0.5
-0.5
-1.5
Dim
ensio
n 2
Reading
Mathematics
ArtPE
Foreign Languages
Mathematics
Science
English
Social Studies
-3 -2 -1 0 1 2 3
Dimension 1
0
-1
1.0
1.5
0.5
-0.5
-1.5
Dim
ensio
n 2
Reading
Mathematics
ArtPE
Foreign Languages
Mathematics
Science
English
Social Studies
-3 -2 -1 0 1 2 3
Dimension 1
0
-1
1.0
1.5
0.5
-0.5
-1.5
Dim
ensio
n 3 Reading
Mathematics
Art
PE
Foreign Languages
Mathematics
Science
English
Social Studies
-3 -2 -1 0 1 2 3
Dimension 1
0
-1
1.0
1.5
0.5
-0.5
-1.5
Dim
ensio
n 3 Reading
Mathematics
Art
PE
Foreign Languages
Mathematics
Science
English
Social Studies
Dimension 1 Dimension 3
Dim
ensio
n 2
Reading
Math
PE
MathScience
Social Studies
Foreign LangArt
English
Standardized Test Core Subject GPA Non-Core Subject GPA
Legend:
A.
B.
C.-3 -2 -1 0 1 2 3
Dimension 1
0
-1
1.0
1.5
0.5
-0.5
-1.5
Dim
ensio
n 2
Reading
Mathematics
ArtPE
Foreign Languages
Mathematics
Science
English
Social Studies
-3 -2 -1 0 1 2 3
Dimension 1
0
-1
1.0
1.5
0.5
-0.5
-1.5
Dim
ensio
n 2
Reading
Mathematics
ArtPE
Foreign Languages
Mathematics
Science
English
Social Studies
-3 -2 -1 0 1 2 3
Dimension 1
0
-1
1.0
1.5
0.5
-0.5
-1.5
Dim
ensio
n 3 Reading
Mathematics
Art
PE
Foreign Languages
Mathematics
Science
English
Social Studies
-3 -2 -1 0 1 2 3
Dimension 1
0
-1
1.0
1.5
0.5
-0.5
-1.5
Dim
ensio
n 3 Reading
Mathematics
Art
PE
Foreign Languages
Mathematics
Science
English
Social Studies
Dimension 1 Dimension 3
Dim
ensio
n 2
Reading
Math
PE
MathScience
Social Studies
Foreign LangArt
English
Dimension 1 Dimension 3
Dim
ensio
n 2
Dimension 1 Dimension 3
Dim
ensio
n 2
Dimension 1 Dimension 3
Dim
ensio
n 2
Reading
Math
PE
MathScience
Social Studies
Foreign LangArt
English
Standardized Test Core Subject GPA Non-Core Subject GPA
Legend:
A.
B.
C.
Three-dimensional MDS of standardized tests, grades in core subjects, and grades in non-core subjects, n=5230
Bowers, A.J. (2013)
-3 -2 -1 0 1 2 3
Dimension 1
0
-1
1.0
1.5
0.5
-0.5
-1.5
Dim
ensio
n 2
Reading
Mathematics
ArtPE
Foreign Languages
Mathematics
Science
English
Social Studies
-3 -2 -1 0 1 2 3
Dimension 1
0
-1
1.0
1.5
0.5
-0.5
-1.5
Dim
ensio
n 2
Reading
Mathematics
ArtPE
Foreign Languages
Mathematics
Science
English
Social Studies
-3 -2 -1 0 1 2 3
Dimension 1
0
-1
1.0
1.5
0.5
-0.5
-1.5
Dim
ensio
n 3 Reading
Mathematics
Art
PE
Foreign Languages
Mathematics
Science
English
Social Studies
-3 -2 -1 0 1 2 3
Dimension 1
0
-1
1.0
1.5
0.5
-0.5
-1.5
Dim
ensio
n 3 Reading
Mathematics
Art
PE
Foreign Languages
Mathematics
Science
English
Social Studies
Dimension 1 Dimension 3
Dim
ensio
n 2
Reading
Math
PE
MathScience
Social Studies
Foreign LangArt
English
Standardized Test Core Subject GPA Non-Core Subject GPA
Legend:
A.
B.
C.-3 -2 -1 0 1 2 3
Dimension 1
0
-1
1.0
1.5
0.5
-0.5
-1.5
Dim
ensio
n 2
Reading
Mathematics
ArtPE
Foreign Languages
Mathematics
Science
English
Social Studies
-3 -2 -1 0 1 2 3
Dimension 1
0
-1
1.0
1.5
0.5
-0.5
-1.5
Dim
ensio
n 2
Reading
Mathematics
ArtPE
Foreign Languages
Mathematics
Science
English
Social Studies
-3 -2 -1 0 1 2 3
Dimension 1
0
-1
1.0
1.5
0.5
-0.5
-1.5
Dim
ensio
n 3 Reading
Mathematics
Art
PE
Foreign Languages
Mathematics
Science
English
Social Studies
-3 -2 -1 0 1 2 3
Dimension 1
0
-1
1.0
1.5
0.5
-0.5
-1.5
Dim
ensio
n 3 Reading
Mathematics
Art
PE
Foreign Languages
Mathematics
Science
English
Social Studies
Dimension 1 Dimension 3
Dim
ensio
n 2
Reading
Math
PE
MathScience
Social Studies
Foreign LangArt
English
Dimension 1 Dimension 3
Dim
ensio
n 2
Dimension 1 Dimension 3
Dim
ensio
n 2
Dimension 1 Dimension 3
Dim
ensio
n 2
Reading
Math
PE
MathScience
Social Studies
Foreign LangArt
English
Standardized Test Core Subject GPA Non-Core Subject GPA
Legend:
A.
B.
C.
-3 -2 -1 0 1 2 3
Dimension 1
0
-1
1.0
1.5
0.5
-0.5
-1.5
Dim
ensio
n 2
Reading
Mathematics
ArtPE
Foreign Languages
Mathematics
Science
English
Social Studies
-3 -2 -1 0 1 2 3
Dimension 1
0
-1
1.0
1.5
0.5
-0.5
-1.5
Dim
ensio
n 2
Reading
Mathematics
ArtPE
Foreign Languages
Mathematics
Science
English
Social Studies
-3 -2 -1 0 1 2 3
Dimension 1
0
-1
1.0
1.5
0.5
-0.5
-1.5
Dim
ensio
n 3 Reading
Mathematics
Art
PE
Foreign Languages
Mathematics
Science
English
Social Studies
-3 -2 -1 0 1 2 3
Dimension 1
0
-1
1.0
1.5
0.5
-0.5
-1.5
Dim
ensio
n 3 Reading
Mathematics
Art
PE
Foreign Languages
Mathematics
Science
English
Social Studies
Dimension 1 Dimension 3
Dim
ensio
n 2
Reading
Math
PE
MathScience
Social Studies
Foreign LangArt
English
Standardized Test Core Subject GPA Non-Core Subject GPA
Legend:
A.
B.
C.-3 -2 -1 0 1 2 3
Dimension 1
0
-1
1.0
1.5
0.5
-0.5
-1.5
Dim
ensio
n 2
Reading
Mathematics
ArtPE
Foreign Languages
Mathematics
Science
English
Social Studies
-3 -2 -1 0 1 2 3
Dimension 1
0
-1
1.0
1.5
0.5
-0.5
-1.5
Dim
ensio
n 2
Reading
Mathematics
ArtPE
Foreign Languages
Mathematics
Science
English
Social Studies
-3 -2 -1 0 1 2 3
Dimension 1
0
-1
1.0
1.5
0.5
-0.5
-1.5
Dim
ensio
n 3 Reading
Mathematics
Art
PE
Foreign Languages
Mathematics
Science
English
Social Studies
-3 -2 -1 0 1 2 3
Dimension 1
0
-1
1.0
1.5
0.5
-0.5
-1.5
Dim
ensio
n 3 Reading
Mathematics
Art
PE
Foreign Languages
Mathematics
Science
English
Social Studies
Dimension 1 Dimension 3
Dim
ensio
n 2
Reading
Math
PE
MathScience
Social Studies
Foreign LangArt
English
Dimension 1 Dimension 3
Dim
ensio
n 2
Dimension 1 Dimension 3
Dim
ensio
n 2
Dimension 1 Dimension 3
Dim
ensio
n 2
Reading
Math
PE
MathScience
Social Studies
Foreign LangArt
English
Standardized Test Core Subject GPA Non-Core Subject GPA
Legend:
A.
B.
C.
-3 -2 -1 0 1 2 3
Dimension 1
0
-1
1.0
1.5
0.5
-0.5
-1.5
Dim
ensio
n 2
Reading
Mathematics
ArtPE
Foreign Languages
Mathematics
Science
English
Social Studies
-3 -2 -1 0 1 2 3
Dimension 1
0
-1
1.0
1.5
0.5
-0.5
-1.5
Dim
ensio
n 2
Reading
Mathematics
ArtPE
Foreign Languages
Mathematics
Science
English
Social Studies
-3 -2 -1 0 1 2 3
Dimension 1
0
-1
1.0
1.5
0.5
-0.5
-1.5
Dim
ensio
n 3 Reading
Mathematics
Art
PE
Foreign Languages
Mathematics
Science
English
Social Studies
-3 -2 -1 0 1 2 3
Dimension 1
0
-1
1.0
1.5
0.5
-0.5
-1.5
Dim
ensio
n 3 Reading
Mathematics
Art
PE
Foreign Languages
Mathematics
Science
English
Social Studies
Dimension 1 Dimension 3
Dim
ensio
n 2
Reading
Math
PE
MathScience
Social Studies
Foreign LangArt
English
Standardized Test Core Subject GPA Non-Core Subject GPA
Legend:
A.
B.
C.-3 -2 -1 0 1 2 3
Dimension 1
0
-1
1.0
1.5
0.5
-0.5
-1.5
Dim
ensio
n 2
Reading
Mathematics
ArtPE
Foreign Languages
Mathematics
Science
English
Social Studies
-3 -2 -1 0 1 2 3
Dimension 1
0
-1
1.0
1.5
0.5
-0.5
-1.5
Dim
ensio
n 2
Reading
Mathematics
ArtPE
Foreign Languages
Mathematics
Science
English
Social Studies
-3 -2 -1 0 1 2 3
Dimension 1
0
-1
1.0
1.5
0.5
-0.5
-1.5
Dim
ensio
n 3 Reading
Mathematics
Art
PE
Foreign Languages
Mathematics
Science
English
Social Studies
-3 -2 -1 0 1 2 3
Dimension 1
0
-1
1.0
1.5
0.5
-0.5
-1.5
Dim
ensio
n 3 Reading
Mathematics
Art
PE
Foreign Languages
Mathematics
Science
English
Social Studies
Dimension 1 Dimension 3
Dim
ensio
n 2
Reading
Math
PE
MathScience
Social Studies
Foreign LangArt
English
Dimension 1 Dimension 3
Dim
ensio
n 2
Dimension 1 Dimension 3
Dim
ensio
n 2
Dimension 1 Dimension 3
Dim
ensio
n 2
Reading
Math
PE
MathScience
Social Studies
Foreign LangArt
English
Standardized Test Core Subject GPA Non-Core Subject GPA
Legend:
A.
B.
C.
Three-dimensional MDS of standardized tests, grades in core subjects, and grades in non-core subjects, n=5230
Bowers, A.J. (2013)
Conclusions 1. Grades appear to be a multi-dimensional assessment of student
academic achievement and an “other” grades dimension
2. The “other” grades dimension appears to be consistent across core and non-core subjects
3. The “other” grades dimension appears to conform to the “engagement” or “success at school factor” hypothesis as a component of grades.
4. Recommendation is to consider grades as a useful multi-dimensional assessment of both academic achievement and a student’s ability to negotiate the social aspects of the schooling process. – Low grades may indicate that a student needs either academic tutoring or
tutoring in how to negotiate the social and engagement aspects of school.
• Future work: – Replication using a larger dataset – Replication in multiple contexts – Addition of more grade levels and standardized tests
Bowers, A.J. (2011) What's in a Grade? The Multidimensional Nature of What Teacher Assigned Grades Assess in High School. Educational Research & Evaluation, 17(3), 141-159. doi: 10.1080/13803611.2011.597112
Bowers, A.J. (2013)
Summary
• Grades and grade trajectories appear to be some of the best predictors of dropping out or graduating.
• Pattern analysis through GMM or HCA combined with data visualization can help schools make data informed decisions.
• Grades may represent a multidimensional assessment of behaviors and outcomes we want assessed in school.
Bowers, A.J. (2013)
Thank you!
Graduate Student Research Team:
Jared Boyce
Youmna Dbouk
Jooyoung Lee
Ryan Sprott
Sherry Taff
Angela Urick
Bowers, A.J. (2013)