aera 2012 presentation on nudge
DESCRIPTION
AERA talk on Nudge, a Computer support for studying at the right times. Goes through design process and system evaluation. Some slides with animation will be confusing or missing pieces because of Slideshare's importer.TRANSCRIPT
+
Nudge
Computer support forstudying at the right times
Aleahmad, Koedinger and Zimmerman
1
+Usability
Research often difficult to translate into practice
Need usable knowledge (Lagemann, 2002)
Tension between internal and ecological validity
Modern computing lets us do controlled lab-style research in real-world settings
Lower the costs of deploying research in broad contexts to understand contextual factors
AERA 2012 — Aleahmad, Koedinger and Zimmerman
2
Usable systems
+Process
AERA 2012 — Aleahmad, Koedinger and Zimmerman
5
Informed Exploration
Enactment
Evaluation: Local Impact
Evaluation: Broader Impact
Integrative Learning Design Framework (Bannan-Ritland 2003)
+Informed Exploration
Used HCI methods for usability: Focus setting College lecture courses Contextual inquiry Interviews with stakeholders Ideation ~60 ideas for systems Scenario development 17 that express needs Needs validation Interview stakeholders to check
Example observation for Nudge need:“by second semester freshman year I was trying to learn how to study, pretty much teaching myself.”
AERA 2012 — Aleahmad, Koedinger and Zimmerman
6
+Nudge Scenario
AERA 2012 — Aleahmad, Koedinger and Zimmerman
7
+Needs Validation
IES practice guides to see what practice recommendations could be operationalized usably in college lecture courses E.g. “help students allocate study time efficiently” from
Organizing Instruction and Study to Improve Student Learning (Pashler et al., 2007) principle #6
Ruled out all but a few systems because either1. not interesting theoretically
2. required too much change by stakeholders
3. rejected by the culture
e.g. study leaderboard. students didn't want competition.
AERA 2012 — Aleahmad, Koedinger and Zimmerman
8
+Principles of Nudge Design
Feature Claim Warrant
Course task assigned to dates and organized centrally
Explicit and salient dates more likely to be met
External deadlines boost task performance more than self-determined deadlines (Ariely & Wertenbroch, 2002);
Students generally do whatever’s due soonest (Kornell & Bjork, 2007)
Break-down of study activity into smaller actions
Decomposition of tasks improves time allocation and decreases aversiveness
Smaller tasks abate the planning fallacy (Forsyth & Burt, 2008; Kruger & Evans, 2003);
Students procrastinate largely due to fear of failure (Solomon & Rothblum, 1984);
In shared task lists, vague information preferred (Blandford, 2001)
Maintaining and tracking assignments, study time and progress
Recording task status increases awareness and inclination
Self-monitoring of study behaviors improves learning (Richards, 1975)
AERA 2012 — Aleahmad, Koedinger and Zimmerman
9
+Iterative Development: Mockup
AERA 2012 — Aleahmad, Koedinger and Zimmerman
10
+Iterative Development: Pilot
AERA 2012 — Aleahmad, Koedinger and Zimmerman
11
+Final Iteration for Evaluation
AERA 2012 — Aleahmad, Koedinger and Zimmerman
12
+Study
Large lecture-based intro chemistry class 15 week fall semester
60 tasks defined for the semester (14 required, 43 advised, 3 suppl.) Do and submit HW-n (required) Take a practice text for Exam-n (advised) Review notes for Exam IV (advised)
AERA 2012 — Aleahmad, Koedinger and Zimmerman
13
+Conditions
AERA 2012 — Aleahmad, Koedinger and Zimmerman
14
Introductory Chemistry Course
N=136
Nudge?
All nudges polled on tasks before due (n=45)
No nudges, tasks batched after exam to poll (n=48)
Everyone else gets message before due, at a schedule they choose. (n=42)
+Hypotheses
H-grades: Student sent all Nudge messages perform better on exams than students sent no Nudge messages.
H-disposition: Students with poor study time use benefit more from Nudge messages.
Time/Environment scale from Motivated Strategies for Learning Questionnaire (Pintrich et al. 2001)
AERA 2012 — Aleahmad, Koedinger and Zimmerman
15
+Evaluation of Operation
H-grade: No main effects…
AERA 2012 — Aleahmad, Koedinger and Zimmerman
16
H-disposition: Better time management led to better exam scores (F(1,76.9)=6.4, p=.014) but Nudge interacts to help students with poor management (F(1,76.9)=4.6, p=.036).
…and hinders with good management?Nudge helps students with poor management
+Mechanism
The worse a student’s time use the greater the benefit of opening each Nudge message.
AERA 2012 — Aleahmad, Koedinger and Zimmerman
17
+Broader Impact
Group
Replied to task polls ever
Agreement with “What I enter is accurate”.
(7pt Likert)
No nudges 83% (40/48) 5.5 (n=26, sd=1.4)
All nudges 87% (39/45) 5.8 (n=29, sd=1.3)
AERA 2012 — Aleahmad, Koedinger and Zimmerman
18
75% of nudged respondents (n=28) rated “Email reminders about course work” as “Good” or “Great”
40% agreed, “The reminder emails helped me in the class”
46% agreed, “I wish I could have email reminders for all my classes”
44% agreed, “Without the reminders I would have forgotten to do something.”
80% of students who could stop the message didn’t
+Discussion
Need validation ruled out systems I thought would work; saved resources
Indications that it is easy to adopt and would be used voluntarily by students and teachers Need to test adoption empirically
Usability focus needs more methods for validation Examplify Future work
AERA 2012 — Aleahmad, Koedinger and Zimmerman
19
+Thanks for your attention!
@tfunq
Turadg Aleahmad
http://coursecheck.org
http://turadg.aleahmad.net
http://openeducationresearch.org
AERA 2012 — Aleahmad, Koedinger and Zimmerman
20