assessment: designs and approaches

21
Assessment: Designs and Approaches Gail Matthews-DeNatale Northeastern University Dave Consiglio Bryn Mawr College

Upload: gail-matthews-denatale

Post on 15-Jul-2015

77 views

Category:

Education


1 download

TRANSCRIPT

Page 1: Assessment: Designs and Approaches

Assessment:

Designs and Approaches

Gail Matthews-DeNatale

Northeastern University

Dave Consiglio

Bryn Mawr College

Page 2: Assessment: Designs and Approaches

The Cycle

Identify

Scope

Interpret

Data

Share &

Act Upon

Findings

Form

Research

Question

Design &

Implement

ResearchYou Are

Here

Page 3: Assessment: Designs and Approaches

On a Dime

http://www.flickr.com/photos/jdhancock/3544855402

Page 4: Assessment: Designs and Approaches

Gold Standard

http://www.flickr.com/photos/bullionvault/3591745555

Page 5: Assessment: Designs and Approaches

Three Scenarios

Activity

Page 6: Assessment: Designs and Approaches

Online Enrollment

Your Dean wants to raise the cap

on enrollment in online courses.

The cap limit will be revisited on an

annual basis. What research can

you perform with your students

and your faculty at your institution

to inform these decisions?

Page 7: Assessment: Designs and Approaches

Clickers

Your institution has been using

classroom response systems

(a.k.a. “clickers”) for several years.

Usage is robust. You’re leading a

task force to assess the impact of

clickers on learning.

Page 8: Assessment: Designs and Approaches

Blended Learning

A pilot initiative is underway at

your school to integrate blended

courses across the curriculum.

What will be the research

component for the pilot, how will

the research be performed, and

how will it be used?

Page 9: Assessment: Designs and Approaches

North Star Questions

What do you want to know/learn?

What’s the purpose?

What’s the population?

Who will receive the results?

Who could/should be involved?

Page 10: Assessment: Designs and Approaches

Scenario Planning

20Minutes

What do you want to know?

What’s the population?

What’s the purpose?

Who will be involved?

Who will receive the results?

Page 11: Assessment: Designs and Approaches

Gold Standard

http://www.flickr.com/photos/bullionvault/3591745555

Page 12: Assessment: Designs and Approaches

Gold Standard

Page 13: Assessment: Designs and Approaches

Reliability

and Validity

Gold Standard

Page 14: Assessment: Designs and Approaches

Controls

Gold Standard

Page 15: Assessment: Designs and Approaches

Example: Try Science

Page 16: Assessment: Designs and Approaches

Qualitative Methods (Mostly)

Background information (years teaching, education, prior OL)

Activity and work products(discussion texts, projects, reports)

Formative feedback responses

Teacher follow-up

interviewsProgram retention rates

Pre-post thought experiments

Self-reported increase in confidence

Change in classroom practice (pre- post -observation/video)

Tool

NVIVO

Page 17: Assessment: Designs and Approaches

Interviews

I thought I knew everything

about density. When I did my

physics unit in class, I thought I

was all set. As long as you know

the formula, you know density --

that's the way I taught it before.

Try Science blew my mind. At first I was a little bit angry

because I found out that for the three years I taught

density, in some cases, I know truly believe that I was

teaching it incorrectly. The assumptions that I made from

reading it instead of doing it were wrong.

Page 18: Assessment: Designs and Approaches

Turning Qual into Quant

96% Program Retention Rate

Page 19: Assessment: Designs and Approaches

Cost: $600,000

Report: 121 pages

Some Numbers

Page 20: Assessment: Designs and Approaches

Quantitative Example: CATS

Goal: Study effect of interactive instruction on student learning

Tools: Standardized concept inventory, demographic and

attitudinal survey of students, teaching practice survey of instructors

Population: Undergraduate students taking Astronomy 101

Study group: Thousands of students of dozens of higher

education institutions from across the country

Process: Pre and post tests using concept inventory, student

survey demographic and attitudinal survey, measure classroom and

institutional variables

Results: Statistical analysis of individual and classroom

influences on learning gains

Page 21: Assessment: Designs and Approaches

Your Task Over Lunch

What would these “gold standard”

methodologies would look like if

they were done “on a dime”?

How might you adapt them for use

in situations that are under-

resourced and pressed for time?