NASPA IARC 2009 – New Orleans, LA

Thank you for downloading this presentation from the NASPA IARC Web site. For additional information on student motivation in low-stakes

assessment contexts, please visit JMU’s Center for Assessment and Research Studies: http://www.jmu.edu/assessment/

Questions?Please contact Peter Swerdzewski at pswerdzewski@collegeboard.org

Suggested Citation:Swerdzewski, P., Anderson, R. D., & Sundre, D.L. (2009, June). Tackling the big question: Motivation in Low-Stakes Assessment. Educational session presented at the NASPA International Assessment and Retention Conference, New Orleans, LA.

TACKLING THE BIG QUESTION: Techniques for Improving Motivation in Low-

Stakes Assessment

Peter Swerdzewski

Robin Anderson

Donna L. Sundre

James Madison Universitywww.jmu.edu/assessment/

One Source of Error: Examinee Effort

• The test giver assumes that examinees will give good effort.

• Low effort will lead to biased proficiency estimates (which produces construct irrelevant variance).

• This leads to test scores with lower validity.

• This problem is most prevalent in testing situations without personal consequences for examinees (e.g., TIMMS, PIRLS).

1. perceived success on the test,

2. perceived level of effort the test will consume,

3. perceived importance of the test, and

4. affective and emotional reaction to the various test items.

(Wise & Demars, 2005)

A student’s effort on a low-stakes test is a function of his or her:

Confronting Low Examinee Motivation

The JMU Assessment Model

What is Learning Assessment?Assessment is the systematic basis for making inferences about

the learning and development of students.

EstablishingObjectives

Selecting/Designing

Instruments

CollectingInformation

Analyzing/MaintainingInformation

UsingInformation

Continuous Cycle

Stages of the Assessment Process

The Assessment Culture at JMU

JMU requires students to take a series of student outcomes assessments prior to their graduation. These assessments are held at four stages of students’ academic careers: as entering first-year students at the mid-undergraduate point when they have earned 45 to 70

credit hours, typically the sophomore year as graduating seniors in their academic major(s) Students will also complete an alumni survey after graduation

-JMU Undergraduate Catalog

• Two institution-wide Assessment Days Fall (August): Incoming freshmen tested at orientation Spring (February): Students with 45-70 credits ; typically the sophomore

year

• Classes are cancelled on this day• All students are required to participate, else course registration is blocked• Students are randomly assigned using the last two digits of their JMU ID

number to testing rooms where a particular series of instruments are administered

– This results in large, representative samples of students– Student ID numbers do not change; therefore, we can assure that students

complete the same instruments at time 2 as they did at time 1

JMU just completed its 23rd Spring Assessment Day• The Spring Assessment Day is also used by many majors to collect data

on their graduating seniors

Data Collection Strategies

Data Collection Scheme:Repeated Measures

Fall 2006Spring 2007

Fall 2007Spring 2008

Fall 2008Spring 2009

Fall 2009Spring 2010

COHORT 1

COHORT 2

COHORT 3

Students in each Students in each cohort are tested cohort are tested twice twice on the same on the same instrument – once as instrument – once as incoming freshmenincoming freshmen and again in the and again in the second semester of second semester of thethe sophomoresophomore year. year.

Prerequisites for Quality Assessment

We must have three important components

Excellence in sampling of students

– Either large, representative student samples or a census

Sound assessment instrumentation

– Reliable, valid assessment methods

– Instruments that faculty find meaningful

Motivated students to participate in assessment activities

– Can we tell if students are motivated?

– Can we influence examinee motivation?

What Techniques Can Be Used Prior to And

During an Assessment?

Opportunities to Impact Motivation

Students: Communication

Proctors: Defining the Role

Effort Scores by Test Room Spring 2007effort

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

21

22

23

24

25

Effort by Room spring 2008

5.006.007.008.009.00

10.0011.0012.0013.0014.0015.0016.0017.0018.0019.0020.0021.0022.0023.0024.0025.00

Room Name

Eff

ort

Sco

res

Testing: Respect Students’ Time

Scholarship at JMU related to Assessment Day Motivation

Correlates of student test-taking motivation Feedback study: Type of feedback and effect on motivation Differences in proctor behavior across test rooms Focus groups & the “non-attenders” survey Change in motivation during a test session

What Techniques Can Be Used After an

Assessment?

Assessing Examinee Motivation

• We have spent over 15 years assessing examinee motivation in a variety of testing contexts High stakes and Low stakes

• We have worked on an instrument to assess motivation in testing contexts The Student Opinion Scale (SOS)

• You can freely download and use this instrument

• www.jmu.edu/assessment/resources/Overview.htm

• You can view a podcast on motivation research

• www.jmu.edu/assessment/resources/podcasts.htm

Student Opinion Scale (SOS)

This is a 10-item instrument: provides two scores Importance-perceived importance of the task(s) Effort-Examinee self-report of level of effort expended in task

completion Both measures result in reliability estimates in mid .80s SOS scores are NOT correlated with SAT scores! This instrument, scoring instructions, and manual are freely

available and downloadable from www.jmu.edu/assessment/

Using the SOS Scores, we have:

• Described and quantified the level of our students’ motivation

• Shared this information with our faculty

• Included SOS scores in our data analysis

• Positively impacted student motivation levels

• Improved our proctor selection and training

• Worked to provide feedback to students on their assessment performances

• Pete will share more about statistical controls, motivation filtering, and RTE

Other Approaches to Measuring Effort

Avoidance

Item response time-based measures Requires computer-based testing Useful with multiple-choice items Unobtrusive Provides item-by-item measure of effort. Rapid-guessing behavior vs.

solution behavior Response time effort

An Item’s Response Time Distribution

(Wise, 2007)

Statistical Control

• ANCOVA

• Regression Effect

• Decile Categories

The Challenge: Motivation is not necessarily a continuous construct

Final Thoughts on Bringing These

Interventions to Your Campus