assessment: designs and approaches

Assessment:

Designs and Approaches

Gail Matthews-DeNatale

Northeastern University

Dave Consiglio

Bryn Mawr College

The Cycle

Identify

Scope

Interpret

Data

Share &

Act Upon

Findings

Form

Research

Question

Design &

Implement

ResearchYou Are

Here

On a Dime

http://www.flickr.com/photos/jdhancock/3544855402

Gold Standard

http://www.flickr.com/photos/bullionvault/3591745555

Three Scenarios

Activity

Online Enrollment

Your Dean wants to raise the cap

on enrollment in online courses.

The cap limit will be revisited on an

annual basis. What research can

you perform with your students

and your faculty at your institution

to inform these decisions?

Clickers

Your institution has been using

classroom response systems

(a.k.a. “clickers”) for several years.

Usage is robust. You’re leading a

task force to assess the impact of

clickers on learning.

Blended Learning

A pilot initiative is underway at

your school to integrate blended

courses across the curriculum.

What will be the research

component for the pilot, how will

the research be performed, and

how will it be used?

North Star Questions

What do you want to know/learn?

What’s the purpose?

What’s the population?

Who will receive the results?

Who could/should be involved?

Scenario Planning

20Minutes

What do you want to know?

What’s the population?

What’s the purpose?

Who will be involved?

Who will receive the results?

Gold Standard

http://www.flickr.com/photos/bullionvault/3591745555

Gold Standard

Reliability

and Validity

Gold Standard

Controls

Gold Standard

Example: Try Science

Qualitative Methods (Mostly)

Background information (years teaching, education, prior OL)

Activity and work products(discussion texts, projects, reports)

Formative feedback responses

Teacher follow-up

interviewsProgram retention rates

Pre-post thought experiments

Self-reported increase in confidence

Change in classroom practice (pre- post -observation/video)

Tool

NVIVO

Interviews

I thought I knew everything

about density. When I did my

physics unit in class, I thought I

was all set. As long as you know

the formula, you know density --

that's the way I taught it before.

Try Science blew my mind. At first I was a little bit angry

because I found out that for the three years I taught

density, in some cases, I know truly believe that I was

teaching it incorrectly. The assumptions that I made from

reading it instead of doing it were wrong.

Turning Qual into Quant

96% Program Retention Rate

Cost: $600,000

Report: 121 pages

Some Numbers

Quantitative Example: CATS

Goal: Study effect of interactive instruction on student learning

Tools: Standardized concept inventory, demographic and

attitudinal survey of students, teaching practice survey of instructors

Population: Undergraduate students taking Astronomy 101

Study group: Thousands of students of dozens of higher

education institutions from across the country

Process: Pre and post tests using concept inventory, student

survey demographic and attitudinal survey, measure classroom and

institutional variables

Results: Statistical analysis of individual and classroom

influences on learning gains

Your Task Over Lunch

What would these “gold standard”

methodologies would look like if

they were done “on a dime”?

How might you adapt them for use

in situations that are under-

resourced and pressed for time?