measuring up on college-level learning

Measuring Up on Measuring Up on College-Level College-Level

LearningLearning

Margaret Miller, Project Director

September 2003

Measuring Up 2000

Measuring Up 2000

Learning in the States: Incomplete

[Add state map on incomplete]

State Efforts to Measure Learning

(taxonomy: Peter Ewell, Change magazine)• Certification of individual students

– E.g., Texas’s TASP, Florida’s CLAST• Institutional assessment for

improvement– E.g., Tennessee's performance

measures– Missouri’s accountability program– Campus-based assessment

• Institutional assessment for accountability– E.g., S. Dakota and Arkansas

National Attention to College-Level Learning

• Pew’s Quality of Undergraduate Education and writing assessment projects

• American Association of Colleges and Universities’ general education assessment project

• Council on Higher Education Accreditation’s project on institutional effectiveness

• Secretary's Commission on Achieving Necessary Skills (SCANS) skills

• Equipped for the Future• National Skills Standards Board

Key Questions

What do the state’s college-educated citizens know and what can they do that contributes to the social good? What kind of educational capital do they represent?

and

Key Questions (cont.)

How well do the state’s public and private, two- and four-year colleges and universities collectively contribute to that capital? What do those whom they educate know, and what can they do?

Key Decisions

• Whose learning will we measure?• What learning will we measure?• How will we use the information?• What strategies will we pursue?

Whose Learning

The college-educated in the states

and

college students

• Whose learning• What learning• The policy uses for the information• Assessment strategies

What Learning

National Education Goal 6:

“By the year 2000, every adult American will be

literate and will possess the knowledge and skills

necessary to compete in a global economy and exercise

the rights and responsibilities of

citizenship”

• Whose learning• What learning• The policy uses for the information• Assessment strategies

What Learning (cont.)

National Goal 6, objective for college education:

“By the year 2000, every adult American will be

literate and will possess the knowledge and skills

necessary to compete in a global economy and exercise

the rights and responsibilities of

citizenship”

• Whose learning• What learning• The policy uses for the information• Assessment strategies

Policy Purposes

Higher education policyand

K-12 education + economic development +

adult literacy policy

• Whose learning• What learning• The policy uses for the information• Assessment strategies

Direct Strategies

• National Assessment of Adult Literacy

• Graduate-admissions and licensing exams

• General intellectual skills tests

• Whose learning• What learning• The policy uses for the information• Assessment strategies

National Assessment of Adult Literacy (NAAL)

concludes12/03 Disadvantages:• Labor-intensive,

expensive • Decadal federal

survey --timing• National sample

only, except in 6 states

• Not what colleges think they teach

Advantages:• Advanced literacy

levels of a good measure of educational capital

• Assesses general population

• Comparison group of non-college-educated

• Household survey – respondent motivation high

Existing Exams• Graduate-admissions exams

– Dental– Graduate Management– Graduate Record– Law School, – Medical College – Optometry– Pharmacy

• Licensing exams– Clinical Pathology– Dental Hygiene– Occupational Therapy– Physical Therapy– Physician Assistant– Nursing– Respiratory Therapy– Teaching

• Whose learning• What learning• The policy uses for the information• Assessment strategies

Existing Examsdata gathered by 03/04

Disadvantages:• Selection bias• Uneven coverage

by discipline• Variable (and

sometimes small) numbers of test- takers in each state

• Most in health professions

Advantages:• Established,

credible instruments

• Highly motivated test-takers

• Admissions tests assess general intellectual abilities

• Availability• Low cost

General Intellectual Skills Tests

administered fall 03

• WorkKeys to a sample of two-year students in each state– Applied Math– Locating Information– Reading for Information– Business Writing

• Collegiate Learning Assessment (CLA) to a sample of four-year students in each state

WorkKeys and CLA

Disadvantages:

• Institutional motivation

• Test-taker motivation

• Expense

Advantages:• Excellent tests

of general & functional intellectual skills

• Can impart useful information to student and school

Indirect MeasuresNSSE/CCSSE co-administered with

testsCRS summer through fall, 03

• National Survey of Student Engagement (NSSE)

• Community College Survey of Student Engagement (CCSSE)

• College Results Survey (CRS)

• Whose learning• What learning• The policy uses for the information• Assessment strategies

Surveys

Disadvantages:• Not direct

learning measures

• Not yet cross-correlated with direct measures

Advantages:• Excellent and

recently developed instruments

• Process measure could lead to improvement

• Both have face validity

• Respondent motivation good

Challenges

• Political instability in states: gubernatorial, SHEEO

• Personnel changes among key players

• Institutional skepticism• Faculty resistance• Data-collection hurdles• Test-taker motivation

General Timeline

• Measuring Up 2002: model tested with incomplete data from Kentucky

• 2002-2004: Five-state pilot to test assessment model: IL, KY, NV, OK, SC

• Measuring Up 2004: publish the results of the pilot

• Measuring Up 2006: if enough states adopt the model, grade states on learning

Reasons to Act

• It is the right thing to do.• We can determine how to

do it right.• This initiative will generate

information useful to states, institutions, and students.

• State-level analysis can promote collaborations to serve underachieving subpopulations or regions of the state.

• State resources can be effectively targeted.

http:///collegelevellearning.org