1 presented to the fall 2011 meeting of the michigan educational research association may 17, 2011...

1

PRESENTED TO THE FALL 2011 MEETING OF THE MICHIGAN EDUCATIONAL RESEARCH ASSOCIATION

MAY 17, 2011

Methods for Setting Cut Scores to Represent College Readiness

2

College Ready Cut Scores for MEAP and MME • May 17, 2011 • MERA

Resources

ContributorsMDE Measurement Research &

Psychometrics UnitACT Research & Development UnitNational Center for Educational

AchievementMichigan Technical Advisory Committee

3


Defining College Readiness

ACT Methodology Good basis for a starting point What score on the ACT gives an 11th grader a

50/50 chance of obtaining a B or better in a… first-semester of freshman year credit-bearing (non-remedial)

…course in a related subject?

4


Defining College ReadinessACT Methodology

ACT Methodology for identifying College Ready Benchmarks http://www.act.org/research/researchers/reports/pdf/ACT_RR2005-3.pdf A logistic regression methodology identifying the point on the ACT subject

score scale where a student has a 50% or better probability of achieving a B or better in the first-semester, credit-bearing, adequately related, freshman course

Start with data from Michigan Public Institutions of Higher Education Identify appropriate credit-bearing freshman courses against which to

analyze the relationship between MME scores and course grades Analyze proportion of students receiving a B or better against MME scale

scores

5



Each dot represents a group of students with a specific test score, plotting the test score against the percent of students with that test score who earned a B or better

6



The curved line is the logistic regression line of best fit through the cloud of points

7



Horizontal line represents a 50% probability of earning a B or better

8



Vertical line represents the test score that give a 50% probability of earning a B or better

9



Issues With the ACT Methodology for the Purpose of Setting Cut Scores on the MME

Non-symmetric, regression-based method Focused on probability prediction

ACT’s purpose Appropriate for ACT’s purpose

Identifying equivalent points on different scales requires the use of a symmetric procedure.

Using a non-symmetric procedure can result in a biased cut score because of regression to the mean when identifying a cut score not at the state average

10


Defining College ReadinessProposed Methodology

Proposed Methodology A Method Based on Signal Detection Theory (SDT)

A symmetric method Focused on maximizing classification consistency

Maximizes the percentage of students who are both1. Classified as proficient on 11th grade MME2. Achieved a B or better in their first-semester, freshman-year, credit-

bearing college course Classification consistency is the core of identifying both college

readiness in high school and being on track in lower grades Results in an unbiased cut score because regression to the mean is not

an issue

11



SDT in a nutshell, for our purposesFocused on the ability to…

Make correct decisions when data contain both “signal” and “noise”

Identify thresholds of “signal” in the data that maximize the ability to make correct classification decisions

12



Other uses of SDT-based methods germane to our purposes Maximally accurate detection of…

Medical anomalies in reading radiological reports Severe cases in Emergency Room triage Medical impacts in clinical drug trials Objects in naval sonar, military radar, and civilian air traffic control Impact of interventions on memory and cognition

Our purpose Maximally accurate detection of…

College readiness on MME Being on-track to college readiness on MEAP

13



Basic application of SDT-based methods for our purposes

Using the known outcomes of college course grades, set the MME cut score to… Maximize the number of students who are consistently classified Minimize the number of students who are inconsistently classified

Grade 11 MME(unknown cut

score)

College Grades (known outcome)

Less than B B or Better

Proficient Inconsistent Consistent

Not Proficient Consistent Inconsistent

14



Start with data from Michigan Public Institutions of Higher Education Identify appropriate first-semester, freshman, credit-bearing (non-

remedial) courses against which to analyze the relationship between MME scores and course grades

Identify students who took both the MME as 11th graders and took those courses

Split those students into those who received a B or higher and those who received a B- or lower

Depicted graphically in the next slides

15


Each dot on the left side of the vertical line represents one student who achieved a B- or lower in a first-semester freshman year, credit-bearing course

Each dot on the right side of the vertical line represents one student who achieved a B or higher in a first-semester freshman year, credit-bearing course

16


Each dot on the upper side of the horizontal line represents one student who achieved proficiency on the MME

Each dot under the horizontal line represents one student who was not proficient on the MME

17


Inconsistent Proficient in 11th grade B- or lower in college

Consistent Not proficient in 11th grade B- or lower in college

Consistent Proficient in 11th grade B or higher in college

18


19


20


75.00% Consistent

21


79.69% Consistent

22



Intend to create three cut scores based on this methodology Produces four performance levels All three cut scores can have externally validated meaning:

Example Cut Score Set 1: A or better in college course as Advanced B or better in college course as Proficient C or better in college course as Partially Proficient

Example Cut Score Set 2: B or better in selective enrollment 4-year Universities as Advanced B or better in non-selective 4-year Universities as Proficient B or better in community college as Partially Proficient

Graphical example shows one cut at a time being set For convenience Statistical methodology can do all three cuts simultaneously

23


Defining On-Track for the Highest Grade Level of Each MEAP Subject

From the new MME cut scores, identify new cut scores for MEAP as follows… Mathematics – grade 8 Reading – grade 8 Science – grade 8 Social Studies – grade 9

Use test scores of students who took both the MME in Spring 2010 and the grade 9 MEAP in Fall of 2007 for Social Studies

Use test scores of students who took both the MME in Spring 2010 and the grade 8 MEAP in Fall of 2006 for Mathematics, Reading, and Science

24


Defining On-Track for the Highest Grade Level of Each MEAP SubjectEach dot on the graph represents the grade 11

MME score and the score on the highest grade level of MEAP for an individual student

25


Defining On-Track for the Highest Grade Level of Each MEAP SubjectMME cut scores

previously identified using SDT-based methods

26



Not Proficienton MME

Partially Proficient on MME

Proficienton MME

Advancedon MME

27



MEAP grade 8 cut scores to be identified

28



29



Adjust the Grade 8 MEAP Cut Scores Up or Down to

Obtain Maximum Classification Consistency

30


Defining On-Track for the Remaining Grade Levels of Each MEAP Subject

Use test scores of students who took MEAP in both Fall 2009 and Fall 2010 for Reading and Math

Use test scores of students who took MEAP in both Fall 2007 and Fall 2010 for Science and Social Studies

Why not use the same cohort all the way back to the lowest MEAP grade? Data are not available Need to use the most recent data for identifying each grade level’s

cut score Need to reflect the current state of Michigan education

31


Defining On-Track for the Remaining Grade Levels of Each MEAP Subject

Use same SDT-based method to go systematically down the line one grade at a time Mathematics and Reading

Connect grade 8 to grade 7 Connect grade 7 to grade 6 Connect grade 6 to grade 5 Connect grade 5 to grade 4 Connect grade 4 to grade 3

Science Connect grade 8 to grade 5

Social Studies Connect grade 9 to grade 6

32


Next Steps

1. Obtain data from Institutions of Higher Education (currently being done, from CEPI)

2. Determine the appropriate credit-bearing freshman courses against which to analyze MME scores (currently being done, with ACT and CEPI)

3. Identify college-ready cut scores for all subjects on the MME

4. Identify on-track cut scores for all subjects on the MEAP5. Recommend to the State Board of Education the final

cut scores for Fall 2011 MEAP and Spring 2012 MME

33


References

Boutis, K. , Pecaric, M. , Seeto, B. and Pusic, M. (2010). Using signal detection theory to model changes in serial learning of radiological image interpretation. Advances in health sciences education: theory and practice, 15(5), 647-58.

Despins, L., Scott-Cawiezell, J., and Rouder, J. (2010). Detection of patient risk by nurses: a theoretical framework. Journal of advanced nursing, 66(2), 465-74.

Green, D. M., & Swets, J. A. (1966). Signal detection theory and psychophysics. New York: Wiley.Klein, S. and Levi, D. (2009) Stochastic model for detection of signals in noise. Journal of the Optical

Society of America: Optics, image science, and vision, 26(11),110-26.Merlo-Pich, E. and Gomeni, R. (2008). Model-based approach and signal detection theory to evaluate

the performance of recruitment centers in clinical trials with antidepressant drugs. Clinical pharmacology and therapeutics, 84(3), 378-84.

Neal, A. and Kwantes, P. (2009) An evidence accumulation model for conflict detection performance in a simulated air traffic control task. Human factors, 51(2), 164-80.

Pigeau, R., Angus, R., O’Neill, P. and Mack, I. (1995). Vigilance latencies to aircraft detection among NORAD surveillance operators. Human Factors, 37(3), 622-634.

34


Contact Information

Joseph A. Martineau, Ph.D. Executive Director Bureau of Assessment & Accountability Michigan Department of Education

[email protected]

1 presented to the fall 2011 meeting of the michigan educational research association may 17, 2011...

Documents