Kathy Booth, RP Group
John Hetts, Long Beach City College
Andrew LaManque, RP Group
What Role Does Transcript Analysis Play in Multiple Measures Assessment?
Regional Research GroupsOctober 19, 2012
2
Session Outcomes1. Clarify the current status of common assessment
efforts in the state
2. Explain how one college used an analysis of transcript data to change its placement practices
3. Share preliminary findings from a study replicating this analysis at 20 colleges
What Role Does Transcript Analysis Play in Multiple Measures Assessment? | October 2012
3
STATEWIDE EFFORTS ON COMMON ASSESSMENT
Building common systems for sharing assessment data
What Role Does Transcript Analysis Play in Multiple Measures Assessment? | October 2012
4
SSTF Recommendation 2.1
Community colleges will develop and implement a common centralized assessment for English reading and writing, mathematics, and ESL that can provide diagnostic information to inform curriculum development and student placement and that, over time, will be aligned with the K-12 Common Core State Standards (CCSS) and assessments.
What’s Completion Got to Do with It? – April 2012
Statewide Efforts on Common Assessment: the Task Force
Statewide Efforts on Common Assessment: Some History
5What Role Does Transcript Analysis Play in Multiple Measures Assessment? | October 2012
AB743Centralized Assessment Bill signed by the Governor in 2011 but a permanent funding source has yet to be identified.
2012 TTAC Recommendations, April 2012Develop a universally available assessment system that provides access to assessment results, as well as K-12 and CCC transcript data, to inform local placement.
CCC Assessment Committee, June 2012Bring together an ad hoc workgroup to flesh out the details of the data warehouse and bring back the results to this committee. This should include the crosswalk as well.
Statewide Efforts on Common Assessment: TTAC
6What Role Does Transcript Analysis Play in Multiple Measures Assessment? | October 2012
• The topic of centralized assessment has highlighted the importance of utilizing a variety of data points in the process of student assessment and predictive behaviors.
• Since an ongoing funding source has yet to be
identified, the Centralized Assessment Advisory Committee approved the plan to pursue the development of a centralized data warehouse
• The idea of predictive analytics offers additional possibilities for the area of student placement and support services.
Statewide Efforts on Common Assessment: TTAC
7What Role Does Transcript Analysis Play in Multiple Measures Assessment? | October 2012
• The Data Warehouse Work Group proposes to link data in phases. Phase 1 will focus on data that is accessible to the project team, Cal-PASS, EAP, CCCApply and MIS.
• A key challenge with these data sets will be to link student records.
• The project team recognizes the value of transcript information among community colleges and will work to develop a plan for collecting college transcript information.
• K-12 transcript information is dependent upon K-12 institutions implementing an electronic transcript system; currently no K-12 institution has a system in place.
8What Role Does Transcript Analysis Play in Multiple Measures Assessment? | October 2012
Source: Tim Calhoon
Statewide Efforts on Common Assessment
Q & A
What questions do you have about the statewide work on common assessment?
Use the chat box on the left side of your screen to type in questions.
What Role Does Transcript Analysis Play in Multiple Measures Assessment? | October 2012
10
HOW THINGS GOT STARTED
Looking closer at assessment and placement
What Role Does Transcript Analysis Play in Multiple Measures Assessment? | October 2012
LBCC’s ResearchSought local answers to 3 questions:
• What predicts how students assess and place into developmental courses?
• What predicts how students perform in those courses?
• How well are placement and performance aligned?
11What Role Does Transcript Analysis Play in Multiple Measures Assessment? | October 2012
Our Research
• Five longitudinal cohorts tracking more than 7,000 high school graduates who attended LBCC from a large local unified school district
• Simplified presentation of utility of 11th grade California Standards Test (CST) scores and high school grades for predicting:
• How students are assessed and placed into the levels of our developmental skills sequences
• How students perform in those classes
12What Role Does Transcript Analysis Play in Multiple Measures Assessment? | October 2012
13
CST ELA (z) Eng Grade (12)
GPA (other)0.0
0.2
0.4
0.6
0.8
1.0
1.2
1.4 1.34x
.00
.30**
Predicting Placement
Alignment in English
* p <.05 **, p <.01, *** p<.001, x = p< 1 x 10-10
CST ELA (z) Eng Grade (12)
GPA (other)0.0
0.2
0.4
0.6
0.8
1.0
.17*
.37***
.88x
Predicting Performance
14
CST Math (z)
Math GPA
GPA (12) GPA (11)0.0
0.2
0.4
0.6
0.8
1.0
.75x
.20
.00.07
Predicting Placement
CST Math (z)
Math GPA
GPA (12) GPA (11)0.0
0.2
0.4
0.6
0.8
1.0
.20*.25**
.73x
.21*
Predicting Performance
Alignment in Math
* p <.05 **, p <.01, *** p<.001, x = p< 1 x 10-10
15
The Take-Aways
•Standardized tests best predict standardized tests
•Classroom performance best predicts classroom performance
•More information tells you more about a student than less information.
What Role Does Transcript Analysis Play in Multiple Measures Assessment? | October 2012
Evidence-Based Change:Fall 2012, Promise Pathways966 students who completed matriculation by May 5 received
• Pilots predictive placement model in English & math using multi-method, evidence-based assessment of students’ preparation and readiness
• Prescriptive, full-time course load Emphasis on early completion of foundational skills
Taken together, what does this mean for our students?
• Elephant in the subway slides
16What Role Does Transcript Analysis Play in Multiple Measures Assessment? | October 2012
English: Full Implementation of Placement and Prescriptive EnrollmentAlternative vs. Traditional Placement
English 1 (or higher) One level below Three levels below
Traditional 14% 38% 47%
Pathways 56% 17% 27%
5%
15%
25%
35%
45%
55%
65%
14%
38%
47%56%
17%
27%
Traditional Pathways
17What Role Does Transcript Analysis Play in Multiple Measures Assessment? | October 2012
Math: Partial Implementation of Placement (but only partial prescriptive enrollment)
Transfer Level Intermediate Algebra (1)
Algebra (2) Pre-Algebra (3)
Arithmetic (4)
Traditional 9% 27% 42% 20% 2%
Pathways 31% 30% 23% 16% 1%
5%
15%
25%
35%
45%
9%
27%
42%
20%
2%
31% 30%
23%
16%
1%
Traditional Pathways
18What Role Does Transcript Analysis Play in Multiple Measures Assessment? | October 2012
Placement is Not Enough: Full-Time Students
Total Full-time0
200
400
600
800
1000
1200
1400
1600
1800 1650
400
1514
966
Typical LUSD 2012-2013
19What Role Does Transcript Analysis Play in Multiple Measures Assessment? | October 2012
20
What LBCC Gained By Using Multiple Measures
• Gives students earlier but appropriate access to transfer level courses
• Increases the number of students that attain meaningful educational outcomes & decreases time to do so
• Challenges conventional wisdom about students’ readiness for college
• Opens new avenues of discussion about research and kick-starts opportunities for experimentation and innovation
• Provides concrete steps that any college—like yours—could take to dramatically improve all students’ futures
20What Role Does Transcript Analysis Play in Multiple Measures Assessment? | October 2012
Q & A
What questions do you have about Long Beach City College’s research and placement changes?
Use the chat box on the left side of your screen to type in questions.
What Role Does Transcript Analysis Play in Multiple Measures Assessment? | October 2012
22
A BRIEF OVERVIEW
The Student Transcript-Enhanced Placement Project (STEPS)
What Role Does Transcript Analysis Play in Multiple Measures Assessment? | October 2012
23
What Are We Trying to Do?• Examine the value of using transcripts as part of
the assessment process• Create predictive models using Cal-PASS data to
study students who had already taken community college courses
• The models analyze how well transcript data predicts the first English & math courses students take and how well they do in them
• Recruit colleges to do local analyses to generate campus-specific insights and trigger CCC/K-12 conversations
What Role Does Transcript Analysis Play in Multiple Measures Assessment? | October 2012
24
Related Work• Brown & Conely. (2007). Comparing state high school
assessments to standards for success in entry-level university courses.
• Brown & Niemi. (2007). Investigating the alignment of high school and community college assessments in California.
• Belfield & Crosta. (2012). Predicting Success in College: The Importance of Placement Tests and High School Transcripts.
• Willett, Hayward, & Dahlstrom. (2007). Leveraging the CST’s.• Wurtz. (2007). Relations among high school math performance
and college placements and achievement.• Fuenmayor, Hetts, & Rothstein. (2011). Assessing Assessment:
Evaluating Models of Assessment and Placement.• English Curriculum Alignment Project (ECAP) between
Grossmont College and Grossmont UHSD.
What Role Does Transcript Analysis Play in Multiple Measures Assessment? | October 2012
25
What Is Happening Now?• Missing data weakened the predictive models
(this highlights the value of strengthening data sharing among segments)
• Most colleges are still working on their analyses
• Bottom line: So far, high school performance is partially predictive of college performance
What Role Does Transcript Analysis Play in Multiple Measures Assessment? | October 2012
26
In Englishtests
predict tests,grades and courses
matter but vary by college
27
Cox & Snell pseudo R-square ~ 0.35
0
1
2
3
4
5
CST's A-GCourses
HSCourseLevel
HSCourseGrade
HS GPA*
Coun
t of C
olle
ges
Show
ing
Sign
ifica
nce
Predictor Variable Category
Predicting College English Level
Strongest
Intermediate
Weakest
What Role Does Transcript Analysis Play in Multiple Measures Assessment? | October 2012
28
In math, thespecific
classes taken in high schoolpredict placement
most strongly
29
Cox & Snell pseudo R-square ~ 0.50
0
1
2
3
4
5
CST's A-GCourses
HSCourseLevel
HSCourseGrade
HS GPA*
Coun
t of C
olle
ges
Show
ing
Sign
ifica
nce
Predictor Variable Category
Predicting College Math Level
Strongest
Intermediate
Weakest
What Role Does Transcript Analysis Play in Multiple Measures Assessment? | October 2012
30
In English,
gradespredict
grades
31
Cox & Snell pseudo R-square ~ 0.20
0
1
2
3
4
5
CST's A-GCourses
HSCourseLevel
HSCourseGrade
HSGPA*
CollegeCourseLevel
Coun
t of C
olle
ges
Show
ing
Sign
ifica
nce
Predictor Variable Category
Predicting College English Success
Strongest
Intermediate
Weakest
What Role Does Transcript Analysis Play in Multiple Measures Assessment? | October 2012
32
In math, success
predictors vary by
college
33
Cox & Snell pseudo R-square ~ 0.20
0
1
2
3
4
5
CST's A-GCourses
HSCourseLevel
HSCourseGrade
HSGPA*
CollegeCourseLevel
Coun
t of C
olle
ges
Show
ing
Sign
ifica
nce
Predictor Variable Category
Predicting College Math Success
Strongest
Intermediate
Weakest
What Role Does Transcript Analysis Play in Multiple Measures Assessment? | October 2012
34
What Happens Next?
• Due to challenges with missing data, additional techniques will be applied with participating colleges
• We are launching a larger study using Cal-PASS data that will help answer additional research questions about using transcripts for assessment
What Role Does Transcript Analysis Play in Multiple Measures Assessment? | October 2012
What Can You Do?Replicate this research at your campus
• http://www.rpgroup.org/projects/STEPS or http://bit.ly/RPSTEPS
Learn more about research on placement exams• “Where to Begin? The Evolving Role of Placement Exams for Stu
dents Starting College” – Jobs for the Future/Achieving the Dream http://bit.ly/AlternativeAssessment
Find out more about Long Beach’s efforts• http://www.rpgroup.org/resources/promising-pathways
Find out more about the previous Cal-PASS study• http://www.rpgroup.org/content/2009-rp-statewide-research-
award-winner-using-california-standards-test-identify-remediation
What Role Does Transcript Analysis Play in Multiple Measures Assessment? | October 2012
36
Find Out More
Terrence Willett, RP [email protected]
John Hetts, Long Beach City [email protected]