aera 2012 presentation on nudge

+

Nudge

Computer support forstudying at the right times

Aleahmad, Koedinger and Zimmerman

1

+Usability

Research often difficult to translate into practice

Need usable knowledge (Lagemann, 2002)

Tension between internal and ecological validity

Modern computing lets us do controlled lab-style research in real-world settings

Lower the costs of deploying research in broad contexts to understand contextual factors

AERA 2012 — Aleahmad, Koedinger and Zimmerman

2

Usable systems

+Process

AERA 2012 — Aleahmad, Koedinger and Zimmerman

5

Informed Exploration

Enactment

Evaluation: Local Impact

Evaluation: Broader Impact

Integrative Learning Design Framework (Bannan-Ritland 2003)

+Informed Exploration

Used HCI methods for usability: Focus setting College lecture courses Contextual inquiry Interviews with stakeholders Ideation ~60 ideas for systems Scenario development 17 that express needs Needs validation Interview stakeholders to check

Example observation for Nudge need:“by second semester freshman year I was trying to learn how to study, pretty much teaching myself.”

AERA 2012 — Aleahmad, Koedinger and Zimmerman

6

+Nudge Scenario

AERA 2012 — Aleahmad, Koedinger and Zimmerman

7

+Needs Validation

IES practice guides to see what practice recommendations could be operationalized usably in college lecture courses E.g. “help students allocate study time efficiently” from

Organizing Instruction and Study to Improve Student Learning (Pashler et al., 2007) principle #6

Ruled out all but a few systems because either1. not interesting theoretically

2. required too much change by stakeholders

3. rejected by the culture

e.g. study leaderboard. students didn't want competition.

AERA 2012 — Aleahmad, Koedinger and Zimmerman

8

+Principles of Nudge Design

Feature Claim Warrant

Course task assigned to dates and organized centrally

Explicit and salient dates more likely to be met

External deadlines boost task performance more than self-determined deadlines (Ariely & Wertenbroch, 2002);

Students generally do whatever’s due soonest (Kornell & Bjork, 2007)

Break-down of study activity into smaller actions

Decomposition of tasks improves time allocation and decreases aversiveness

Smaller tasks abate the planning fallacy (Forsyth & Burt, 2008; Kruger & Evans, 2003);

Students procrastinate largely due to fear of failure (Solomon & Rothblum, 1984);

In shared task lists, vague information preferred (Blandford, 2001)

Maintaining and tracking assignments, study time and progress

Recording task status increases awareness and inclination

Self-monitoring of study behaviors improves learning (Richards, 1975)

AERA 2012 — Aleahmad, Koedinger and Zimmerman

9

+Iterative Development: Mockup

AERA 2012 — Aleahmad, Koedinger and Zimmerman

10

+Iterative Development: Pilot

AERA 2012 — Aleahmad, Koedinger and Zimmerman

11

+Final Iteration for Evaluation

AERA 2012 — Aleahmad, Koedinger and Zimmerman

12

+Study

Large lecture-based intro chemistry class 15 week fall semester

60 tasks defined for the semester (14 required, 43 advised, 3 suppl.) Do and submit HW-n (required) Take a practice text for Exam-n (advised) Review notes for Exam IV (advised)

AERA 2012 — Aleahmad, Koedinger and Zimmerman

13

+Conditions

AERA 2012 — Aleahmad, Koedinger and Zimmerman

14

Introductory Chemistry Course

N=136

Nudge?

All nudges polled on tasks before due (n=45)

No nudges, tasks batched after exam to poll (n=48)

Everyone else gets message before due, at a schedule they choose. (n=42)

+Hypotheses

H-grades: Student sent all Nudge messages perform better on exams than students sent no Nudge messages.

H-disposition: Students with poor study time use benefit more from Nudge messages.

Time/Environment scale from Motivated Strategies for Learning Questionnaire (Pintrich et al. 2001)

AERA 2012 — Aleahmad, Koedinger and Zimmerman

15

+Evaluation of Operation

H-grade: No main effects…

AERA 2012 — Aleahmad, Koedinger and Zimmerman

16

H-disposition: Better time management led to better exam scores (F(1,76.9)=6.4, p=.014) but Nudge interacts to help students with poor management (F(1,76.9)=4.6, p=.036).

…and hinders with good management?Nudge helps students with poor management

+Mechanism

The worse a student’s time use the greater the benefit of opening each Nudge message.

AERA 2012 — Aleahmad, Koedinger and Zimmerman

17

+Broader Impact

Group

Replied to task polls ever

Agreement with “What I enter is accurate”.

(7pt Likert)

No nudges 83% (40/48) 5.5 (n=26, sd=1.4)

All nudges 87% (39/45) 5.8 (n=29, sd=1.3)

AERA 2012 — Aleahmad, Koedinger and Zimmerman

18

75% of nudged respondents (n=28) rated “Email reminders about course work” as “Good” or “Great”

40% agreed, “The reminder emails helped me in the class”

46% agreed, “I wish I could have email reminders for all my classes”

44% agreed, “Without the reminders I would have forgotten to do something.”

80% of students who could stop the message didn’t

+Discussion

Need validation ruled out systems I thought would work; saved resources

Indications that it is easy to adopt and would be used voluntarily by students and teachers Need to test adoption empirically

Usability focus needs more methods for validation Examplify Future work

AERA 2012 — Aleahmad, Koedinger and Zimmerman

19

+Thanks for your attention!

@tfunq

Turadg Aleahmad

http://coursecheck.org

http://turadg.aleahmad.net

http://openeducationresearch.org

AERA 2012 — Aleahmad, Koedinger and Zimmerman

20