Research review and resources to help to make your case

Northern California Convening of MMAP Pilot Colleges

October 28, 2015

Being prepared

Excellent overviews MMAP Status Report coming very soon In the meantime

– Burdman, 2012: Where to begin? The evolving role of placement exams for students starting college. http://bit.ly/Burdman2012

– Bracco et al, 2014: Exploring the use of multiple measures for placement into college-level courses: Seeking alternatives or improvements to the use of a single standardized test. http://bit.ly/MMWestEd

Know the law Know your matriculation handbook, esp. Chap 2: http://bit.ly/

SSSPHandbook – “Assessment is a holistic process through which each college collects

information about students to facilitate their success by ensuring their appropriate placement into math, English, and ESL curricula. Student assessments should reflect a variety of informational sources that create a profile of a student’s academic strengths and weaknesses.” p. 2.3

Colleges must adhere to the following regulations and guidelines when implementing and managing any assessment instrument used for course placement: – …– Course placement recommendations must be based on multiple measures (sections 55502(i)

and 55522(a)). Additional indicators of student readiness for math, English, and ESL course content must be used together with placement test results. p. 2.4

Know the law - II Know your Title 5, esp. Division 6 (CCCs), Subchapter 6 (Matriculation

programs): http://bit.ly/Title5Matriculation– 55502.(i) “Multiple measures” are a required component of a district's assessment system and refer to the

use of more than one assessment measure in order to assess the student. Other measures that may comprise multiple measures include, but are not limited to, interviews, holistic scoring processes, attitude surveys, vocational or career aptitude and interest inventories, high school or college transcripts, specialized certificates or licenses, education and employment histories, and military training and experience. (See also 55522(a)

– 55502 (e) “Disproportionate impact” in broad terms is a condition where access to key resources and supports or academic success may be hampered by inequitable practices, policies, and approaches to student support or instructional practices affecting a specific group. For the purpose of assessment, disproportionate impact is when the percentage of persons from a particular racial, ethnic, gender, age, or disability group, who are directed to a particular service or course placement based on an assessment test or other measure is significantly different from the representation of that group in the population of persons being assessed, and that discrepancy is not justified by empirical evidence demonstrating that the assessment test or other measure is a valid and reliable predictor of performance in the relevant educational setting.

– (also 55003 (d)(2): Prerequisites or corequisites may be established only for any of the following purposes:

o (2) the prerequisite will assure, consistent with section 55002, that a student has the skills, concepts, and/or information that is presupposed in terms of the course or program for which it is being established, such that a student who has not met the prerequisite is highly unlikely to receive a satisfactory grade in the course (or at least one course within the program) for which the prerequisite is being established)

Know the Academic Senate’s position

Academic Senate for California Community Colleges (ASCCC)

passed resolution in strong support of using multiple measures

for placement (ASCCC, 2013). http://bit.ly/MMAACCC2013

ASCCC task force concluded that “inclusion of multiple

measures in our assessment processes is an important step

toward improving the accuracy of placement processes”

(Grimes-Hillman, Holcroft, Fulks, Lee, & Smith, 2014, p. 7).

http://bit.ly/MMAACCC

Know what advocates of standardized tests say “’We’ve been advocating for almost everything that’s been indicated in the report [Pamela Burdman’s

Where To Begin? The Evolving Role Of Placement Exams For Students Starting College] for quite a few

years now,’ said David Parmele, executive director in the ACCUPLACER program for the

College Board. …’We do not believe that the placement score alone should be

the only factor used to decide a student’s placement into college-level classes,’

Parmele said, echoing a key aspect of the report—namely, how some systems are weighing the merits

of moving away from the widespread practice of using the test scores as the only basis for assigning

students to remedial classes and toward using multiple measures, such as high school grades.

http://bit.ly/Diverse2012

Mr. Parmele and Mr. Sconing [ACT assistant vice president for applied research] said both Accuplacer

and Compass include tools to allow colleges to weigh test results along with other academic

indicators, such as high school grades and course credits, and work with colleges to use broader

measures of student readiness than just the test. Neither testing representative, however, knew how

many of its client colleges actually use those tools. http://bit.ly/MMEdWeek2013

Know what advocates of standardized tests say - II

“The College Board agrees that the most successful placement models

are those that take a comprehensive approach. This means utilizing

the extensive range of tools available within ACCUPLACER to assess

multiple variables, including high school GPA, to develop a more robust

picture of a student’s preparation for college and careers.”

http://bit.ly/MMACCU

But it [the US Department of Education] also said that tests should be

“just one of multiple measures” of student achievement, and that “no

single assessment should ever be the sole factor in making an educational

decision about a student, an educator or a school.” http://bit.ly/MMUSDOE

Know current state of affairs >92% of two year institutions administer high-stakes placement exams (Hughes &

Scott-Clayton, 2011): http://bit.ly/Hughes2011

Only 21% of two year institutions use anything other than an admissions or placement test in mathematics, 13% in reading (Fields & Parsad, 2012) http://bit.ly/NAGB2012– Wide variability in cut scores with those at 2-year institutions typically higher than at 4-

year institutions

68% of students in two year institutions take at least one developmental education course (Scott-Clayton & Belfield, 2015). http://bit.ly/CCRCPlacementAccuracy

Placement below transfer level is significant barrier to completion (Bailey, 2009; Bailey, Jeong, & Cho, 2010) http://bit.ly/Bailey2010

– <50% complete the sequence, ~30% never attempt a course in the sequence and ~10% fail to re-enroll after successfully completing at least one course in the sequence

50-60% of equity gap in college completions occur during assessment and matriculation (Stoup, 2015)

Conventional Wisdom

It is a problem with today’s students

– Students are simply, vastly unprepared for college

– Kids these days ….

That seems awfully familiar

Too familiar(Bye Bye Birdie – 1963)

The conventional wisdom is likely wrong National Assessment of Educational Progress: at all-time highs

in virtually every demographic category:bit.ly/NAEPInfo

The evidence mounts Research increasingly questions effectiveness of current standardized

assessment for understanding student capacity

– Little relation to college course outcomeso (e.g., Belfield & Crosta, 2012; Edgescombe, 2011; Jaggars & Stacey, 2014; Scott-Clayton, 2012; Scott-Clayton & Rodriguez, 2012):

bit.ly/CCRCAssess and http://bit.ly/DevEdOutcomes

– 20-35% of students in developmental education sequences are severely underplaced

(e.g., would likely earn a B or better in the transfer-level course) with many more

underplaced. (Scott-Clayton and Belfield, 2015).bit.ly/CCRCPlacementAccuracy

– Underestimates capability of students of color, women, first generation college

students, low SES (Hiss & Franks, 2014) bit.ly/DefiningPromise

– May increasingly be confounded with income (Geiser, 2015) http://bit.ly/Geiser2015

o Controlling for SES, the utility drops meaningfully

Potential change in placements

Implementing Multiple Measures Placement:Transfer-level Placement Rates LBCC F2012

Transfer Level English Transfer Level Math0%

10%

20%

30%

40%

50%

60%

70%

11%7%

13%9%

14%9%

60%

31%

F2011 First time studentsF2011 LBUSDF2012 Promise Pathways - Ac-cuplacer OnlyF2012 Promise Pathways - Mul-tiple Measures

SDCCD MMAP F2015 Pilot (N = ~1000)

English Math0%

10%20%30%40%50%60%70%80%

24% 28%

58%68%

Accuplacer

Accuplacer + MM

http://bit.ly/MMAPPilot

But doesn’t that just flood transfer-level courses with unqualified students? …

Comparison against traditional sequence: Success rates in transfer-level courses

English Math0%

10%20%30%40%50%60%70%80%90%

64%55%

62%51%

First Cohort, F2012

Non-Pathways Promise PathwaysNeither of these differences approach significance, p >.30 http://bit.ly/RPMMEarly

Cohort 1 English 1 Success Rates by Placement(vs. 4 year completion)

F2012 Non-Pathways F2012 Promise Pathways0%

10%20%30%40%50%60%70%80%

56%

68%65% 67%67%

52%

Transfer 1 Level Below3 Levels Below

F2008 English 1 Cohort Attempt Rate

F2008 English 1 Cohort Complete Rate

0%10%20%30%40%50%60%70%80%

62%

47%43%35%

12%9%

Transfer 1 Level Below 3 Levels Below

http://bit.ly/RPMMEarly

Cohort 3: Success rates in transfer-level courses

English Math0%

10%20%30%40%50%60%70%80%90%

67%

49%

79%

49%

Most recent cohort, F2014

Non-Pathways Promise Pathways

English difference, p < .001 http://bit.ly/RPMMEarly

Overall Success Rate in Transfer Level English by Method of Qualification

ENGL1 Success Rate Percentage of Transfer English Placements0%

10%20%30%40%50%60%70%80%90%

100%

66%

20%

70%

90%

Accuplacer (All) Multiple Measures (All)(among students with high school data available) http://bit.ly/RPMMEarly

Success Rate by Method of Qualification in Transfer Level English

ENGL1 Success Rate Percentage of Transfer English Placements0%

10%20%30%40%50%60%70%80%90%

56%

8%

73%

12%

69%

79%

Accuplacer Only MM and Accuplacer Multiple Measures Onlyhttp://bit.ly/RPMMEarly

Sierra College F2014 Transfer-Level English Success Rates by Placement

Fall 2011

Fall 2012

Fall 2013

Fall 2014 ALL

Fall14 - Accupl...

F14 HS Data

F14 Other

60%

65%

70%

75%

80%

72% 73%

70%

73% 73%

79%

71%

http://bit.ly/RPMMEarly

… what about grade inflation/social promotion in HS?

Concerns about grade inflation and social promotion do not fit evidence

Suggests that there should be little to no relation between HS

grades and college grades because HS grades unrelated to

performance

– Everyone gets As and Bs would mean no variation to predict

outcomes

Yet, predictive utility strongly observed

– Stronger than standardized tests

– Even by standardized test companies

Westrick & Allen, 2014: ACT COMPASS Validation Median Logistic R (Table 4) http://bit.ly/ACTandGPA

Course Compass Test Compass HSGPA HSGPA + Compass

English 1 Writing Skills .31 .57 .62

Arithmetic Pre-Algebra .57 .34 .66

Algebra Pre-Algebra .36 .65 .80

Intermediate Algebra Algebra .47 .66 .84

College Algebra Algebra .41 .76 .88

College Algebra College Algebra .51 .76 .94

Westrick & Allen, 2014: Conditional Success Rates for English 1 (Table 6) http://bit.ly/ACTandGPA

Compass Score(30 extremely low to 90 extremely high)

HSGPA 30 50 60 70 90

2.00 23% 26% 28% 29% 32%

3.00 43% 47% 49% 51% 55%

4.00 65% 69% 70% 72% 75%

Compass Score(30 extremely low to 90 extremely high)

HSGPA 30 50 60 70 90

2.00 23% 26% 28% 29% 32%

3.00 43% 47% 49% 51% 55%

4.00 65% 69% 70% 72% 75%

Evidence for grade inflation low at best

Little evidence for grade inflation

over last decade

Earlier observations of grade

inflation may have been partly

artifactual

– adjustments to GPA for

AP/IB/Honors

Zhang & Sanchez, 2014:

http://bit.ly/ACTGradeInflation

… didn’t that work only because Long Beach is special/has special relationship between LBCC and LBUSD?

Not just Long Beach LBCC now includes multiple additional districts including ABC Unified and Los Alamitos Unified

Long thread of research in the CCCs alone– 2008: Willett, Hayward, & Dahlstrom http://bit.ly/WIllett2008

o 11th grade HS variables as early alert mechanism for discipline assessment

– 2011: Martinez http://bit.ly/Martinez2011 o self-reported HS variables as more powerful predictors of college completion

– 2014: Willett & Karanjeff http://bit.ly/RPSTEPS o replication of LBCC research with 12 additional colleges (STEPS)

Replication of implementation

– Bakersfield College and Sierra College began similar implementation in 2014

o http://bit.ly/RPMMEarly

CCRC research

MMAP Statewide Research & local replications: bit.ly/MMAP2015

– MMAP Pilot colleges: http://bit.ly/MMAPPilot

… we’re happy with our placement. Why should we change?

Powerful reasons for change:1) Basic assessment theory and methods

Self-reported satisfaction with assessment by instructors and students

is most common measure and has grave methodological flaws:

– Selection bias

– Confirmation bias

– Effort justification

– System justification

– Self-fulfilling prophecy effects and stereotype threat

HSGPA is effectively gold standard of assessment/measurement theory

– Triangulates capacity across assessment methods, content domains, evaluators, and time

eliminating most sources of systematic and random error

Powerful reasons for change:2) It’s poorly assessing students

Substantial evidence of systemic &

severe underplacement– placing students in developmental education

who could get a B or better in the transfer level

course

– Up to 36% of students placed into development

English and 25% of students placed into

developmental Math

Using multiple measures reduces error

and has clear potential to increase

success rates and sequence completion

– http://bit.ly/CCRCPlacementAccuracy

Powerful reasons for change:3) Transformational impacts for students

Potential for dramatic increases in rates and time to completion of– Transfer-level course in discipline

– Subsequent courses in discipline

– Other early educational milestones.

F2012 Promise Pathways vs. Fall 2011 2-year rates of achievement

0%

20%

40%

60%

13%24%

3%

31%23%

52%

20%

54%

F2011 LBUSD (N=1654) F2012 Promise Pathways (N=933)

http://www.lbcc.edu/PromisePathways/

Equity impact LBCC: F2011 Baseline Equity Gaps for 2-year rates of achievement

-10%10%30%50%70%

4% 13% 2% 15%12%25% 3%

32%21% 24% 1%

33%18%

34%

6%

41%

F11 African Americans F11 HispanicF11 Asian F11 White

http://www.lbcc.edu/PromisePathways/

Equity impact LBCC: F2012 2-year rates of achievement

0%

20%

40%

60%

12%

39%18%

42%

21%

51%

17%

52%

26%

58%

23%

59%

36%

64%

28%

66%

F12 African American F12 Hispanic F12 Asian F12 Whitehttp://www.lbcc.edu/PromisePathways/

… what about students for whom high school transcript data aren’t available/easy to get?

Just ask: self-reported HSGPA appears to be reliable alternative

College of the Canyons Research (Gribbons, 2014)

– Self-report of last course and grade in Fall term very accurate

– Errors that do occur in part because of timing of assessment

University of California admissions

– Uses self-report HSGPA but verifies after admission

– 2008: 9 campuses, 60000 students. No campus had more than 5 discrepancies b/w

reported grades and student transcripts:

o http://bit.ly/UCSelfReportGPA

Much of the ACT research uses self-report GPA and finds it to be a more powerful

predictor than students actual scores on the standardized tests

– ACT, 2013: r(1978) = .84

ACT, 2013:http://bit.ly/ACTSelf-ReportedGPA

Actual HSGPA Level N

Mean HSGPAMean diff.

Accuracy

Actual Student-reported % within 0.25 % within 0.50

3.50–4.00 599 3.79 3.75 –0.04 87% 98%

3.00–3.49 451 3.24 3.23 –0.01 60% 90%

2.50–2.99 408 2.81 2.76 0.05 47% 82%

2.00–2.49 265 2.24 2.35 0.11 40% 73%

1.50–1.99 172 1.77 2.04 0.27 30% 55%

0.00–1.49 85 1.03 1.85 0.82 14% 35%

Total 1,980 2.95 3.02 0.07 58% 83%

… what about non-traditional students?

Multiple measures continues to have utility for delayed matriculants

HSGPA continues to be predictively useful up to the point

where we have data we can meaningfully connect

– Delay of 9-10 years.

How long is High School GPA good for?

0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 180

0.050.1

0.150.2

0.250.3

0.350.4

f(x) = − 0.00763859649122807 x + 0.341964912280702R² = 0.620115928375629f(x) = − 0.0116122807017544 x + 0.363070175438596R² = 0.833614197361731

MMAP: Decay function for the predictive utility of HSGPA on Eng-lish grade

HS 11 GPA Linear (HS 11 GPA) HS 12 GPALinear (HS 12 GPA) Accuplacer

Semesters of delay (approx. 6 months each)

Corr

elati

on b

etw

een

HSPG

A an

d 1s

t CC

Engl

ish g

rade

Westrick & Allen, 2014: ACT COMPASS Validation Standardized Logistic Regression Coefficients(Table 5) http://bit.ly/ACTandGPA

Course Compass Test Student Type Compass HSGPA DiffEnglish 1 Writing Skills Traditional .25 .72 .47

Nontraditional .21 .36 .15Arithmetic Pre-Algebra Traditional .67 .51 -.16

Nontraditional .43 .08 -.35Algebra Pre-Algebra Traditional .43 .78 .35

Nontraditional .32 .47 .15Int. Algebra Algebra Traditional .52 .76 .24

Nontraditional .44 .25 -.19Coll. Algebra Algebra Traditional .36 .88 .52

Nontraditional .43 .59 .16Coll. Algebra Coll. Algebra Traditional .50 .82 .32

Nontraditional .26 .47 .21