1 cognitive analysis of student learning using learnlab brett van de sande, kurt vanlehn, & tim...
TRANSCRIPT
1
Cognitive Analysis of Student Learning Using LearnLab
Brett van de Sande, Kurt VanLehn, & Tim Nokes
University of Pittsburgh
2
Agenda
I. LearnLab methodology
II. Demonstration of Andes,
an intelligent homework tutor
III. Log File Analysis
3
Goal: To understand physics learning
• Policy level– e.g., Physics for high school freshman?
• Instructional level– e.g., How much assistance to give?– e.g., How much practice per topic?– e.g., How to handle errors?
• Neurocognitive level– e.g., Can neuroimaging distinguish deep from
shallow studying of a text?
Our focus
4
Traditional methodsfor studying learning
• Design experiment– Modify text, classroom activities, tests…– e.g., Project Scale-up
• Lab experiment– Modify just one factor– Brief; money instead of grades, …
5
PSLC methods
• Educational data mining– Logs from instrumented courses– Some analysis is automated
• In vivo experiments– Control of variables– Instrumented courses
Next
6
Instrumented courses (Called LearnLab courses)
• Existing class + data collection– Homework done on a tutoring system or
photocopied and analyzed– Photocopies of quizzes, exams– FCI given before and after the course– Demographics, GPAs, Majors…– Handouts, slides, clicker data,…
• Instructor, student & IRB cooperation– Anonymity
7
Existing Physics LearnLab Course(s)
• US Naval Academy– Course take by all 2nd year students– LearnLab is in 4 of about 20 sections– Profs. Wintersgill, McClanahan
• Your course here
8
Basic data mining question
• What features of students’ histories are statistically associated with learning gains?
• e.g., What are the differences between histories of Student A and Student F?
Student A: 25% on pretest
Semester-long history 85% on post-test
Student F: 25% on pretest
Semester-long history 20% on post-test
9
Knowledge decomposition hypothesis
• Decompose knowledge to be learned into a set of knowledge components– e.g., Newton’s third law– e.g., Centripetal acceleration
• Assume each knowledge component is learned independently– An approximation/idealization
10
Data mining with knowledge components (KCs)
Student KC Pre-test History Post-test
A 1 35% … 85%
A 2 15% … 10%
A 3 25% … 20%
B 1 50% … 20%
B 2 10% … 10%
B 3 25% … 80%
For each KC, find statistical associations between histories and gains.
11
History decomposition hypothesis
• Decompose the student’s history into events such that each event addresses only one (or a few) knowledge components.– Reading a paragraph about Newton’s 3rd law– Drawing a reaction force vector– Seeing the instructor draw a reaction force– Drawing a centripetal acceleration vector
• Assume that a KC’s learning gain depends only on that KC’s events
12
Events for 1 student on 1 KC(e.g., Newton’s 3rd law)
Time Context Behavior8/27/07 9:05 FCI item 3 Incorrect
8/27/07 9:12 FCI item 10 Incorrect
9/13/07 18:06 Textbook, pg. 111 Highlighted
9/13/07 21:11 Problem 5-11, drawing FBD
Omitted force on the hand due to block
9/14/07 9:12 Lecture, slide 20 Taking notes
9/15/07 22:05 Problem 5-11, drawing FBD
Draws force on the hand due to block
etc.
13
Some events are not currently available
Time Context Behavior8/27/07 9:05 FCI item 3 Incorrect
8/27/07 9:12 FCI item 10 Incorrect
9/13/07 18:06 Textbook, pg. 111 Highlighted
9/13/07 21:11 Problem 5-11, drawing FBD
Omitted force on the hand due to block
9/14/07 9:12 Lecture, slide 20 Taking notes
9/15/07 22:05 Problem 5-11, drawing FBD
Draws force on the hand due to block
etc.
14
More feasible data mining
• Predict learning gains of a KC given the sequence of events relevant to that KC
• On an event that assesses mastery of a KC, predict the student’s performance during that event given the sequence of preceding events relevant to that KC
15
Predicting correctness ofevents that assess mastery
Context Event type P(Correct)FCI item 3 Assessment 0.10
FCI item 10 Assessment 0.12
Reading textbook, pg. 111, paragraph 3
Instruction Not applicable
Problem 5-11, drawing force on hand due to block
Assessment 0.30
Lecture, slide 20 Instruction Not applicable
Problem 5-11, drawing force on hand … with remedial feedback if needed
Assessment then instruction
0.55
16
Learning curves
• Plot assessment events on x-axis– Ordered chronologically
• Plot measure of mastery on y-axis– Usually aggregated across subjects
e.g., proportion of 100 subjects who performed correctly on this event
17
An expected learning curveF
requ
ency
of
corr
ect
0
0.5
1.0
1 2 3 4 5 6 7 8
Assessment events
18
Summary of PSLC educational data mining
• Given knowledge to be learned– Decompose into knowledge components
• Given students’ histories from an instrumented course– Divide into assessment/instruction events– such that one KC (or a few) per event
• For each KC, find a function on a sequence of events that predicts the KC’s– learning gain during the course– learning curve
19
PSLC methods
• Educational data mining– Logs from instrumented courses– Some analysis is automated
• Andes produces logs with KCs • DataShop draws learning curves, etc.
• Correlation ≠ Causation• In vivo experiments
– Control of variables– Instrumented courses
Next
20
Two major typesof in vivo experiments
• Short & fat– During one lesson or one unit
• Long & skinny– During whole course– “invisibly”
21
Example of a short, fat, in vivo experiment (Hausmann 07)
• During a 2-hour period (usually used for lab work)
• ~25 students in the room, each with a laptop and a headset mike
• Repeat 3 times: – Study a video while explaining it into the mike– Solve a problem
• 4 experimental conditions, varying the content of the video and the instructions for explaining it
• Random assignment of students to conditions• Dependent measures include learning curves• Result: Instructions to self-explain worked best regardless
of content of the video
22
Example of a long, skinny in vivo experiment (Katz 07)
• During 8 weeks of a 13-week course• Random assignment to 2 conditions:
– Experimental group: After solving certain homework problems, the student discussed the solution with a natural language tutoring system
– Control group: Extra homework problems
• Result: Experiment > Control on some conceptual measures
23
Robust Learning
• Immediate learning– During an immediate post-test– Similar content to training (near transfer)
• Robust learning– Far transfer– Retention– Acceleration of future learning
• Does manipulation of instruction on topic A affect rate of learning of a later topic, B?
24
Summary of PSLC methodology
• Data mining– Instrumented (LearnLab) courses– Knowledge components– Instructional and assessment events– Learning curves
• In vivo experiments– Short & fat vs. long & skinny– Robust learning
25
Agenda
I. LearnLab methodology
II. Demonstration of Andes,
an intelligent homework tutor
III. Log File Analysis Next
26
27
Define variables
Draw free body diagram (3 vectors and body)
Define coordinates (3 choices for this problem)
Upon request, Andes gives hints for what to do next
28
Principle-based help for incorrect entry
Red/green gives immediate feedback for student actions
29
30
# Log of Andes session begun Tuesday, July 17, 2007 12:12:28 by [User] on [Computer]
...05:03 DDE (read-problem-info "S2E" 0 0) ...02:35 Axes Axes-671 64 335 143 29602:35 Axes-dlg Axes-671 || …02:38 C dir 4002:42 BTN-CLICK 1 OK02:42 DDE (assert-x-axis NIL 40 Axes-671 "x" "y" "z")02:42 DDE-COMMAND assoc step (DRAW-AXES 40)02:42 DDE-COMMAND assoc op DRAW-VECTOR-ALIGNED-AXES02:42 DDE-COMMAND set-score 3902:42 DDE-RESULT |T| ...10:02 E 0 F1_y+F2_y=010:02 EQ-SUBMIT 010:02 DDE (lookup-eqn-string "F1_y+F2_y=0" 0)10:47 DDE-COMMAND assoc parse (= (+ Yc_Fn_BALL_WALL1_1_40
Yc_Fn_BALL_WALL2_1_40) 0)10:47 DDE-COMMAND assoc error MISSING-FORCES-IN-Y-AXIS-SUM10:47 DDE-COMMAND assoc step (EQN (= (+ Yc_Fw_BALL_EARTH_1_40
Yc_Fn_BALL_WALL2_1_40 Yc_Fn_BALL_WALL1_1_40) 0))10:47 DDE-COMMAND assoc op WRITE-NFL-COMPO10:47 DDE-RESULT |NIL| ...10:50 DDE-RESULT |!show-hint There is a force acting on the ball at T0 that you
have not yet drawn.~e| ...16:38 END-LOG
problem name
session time
student action (equation)
error analysis:intended action
student action (draw axes)
interpretation:compare to model
green
red
31
Demonstration by Tim Nokes# Log of Andes session begun Wednesday, April 18, 2007 21:08:07 by [user] on [computer]...0:02 DDE (read-problem-info "FARA9" 0 0)...0:13 Help-Hint 0:13 DDE (Get-Proc-Help)0:13 DDE-COMMAND assoc (NSH NEW-START-AXIS 0)0:13 DDE-RESULT |!show-hint It is a good idea to begin most problems by drawing an
axis. This helps to ground your work and will be useful later on in the process.~e|…0:17 Begin-draw 50001 Axes-1 185 331...0:30 New-Variable resistance...0:39 DDE (define-variable "R" |NIL| |resistance| |R| |NIL| |NIL| Var-2 "20 ohm")0:39 DDE-COMMAND assoc step (DEFINE-VAR (RESISTANCE R))0:39 DDE-COMMAND assoc op DEFINE-RESISTANCE-VAR0:39 DDE-COMMAND assoc parse (= R_R (DNUM 20 ohm))0:39 DDE-COMMAND set-score 30:39 DDE-RESULT |T|....0:50 DDE (lookup-vector "B" Unspecified B-field |s| NIL 0 |NIL| Vector-3)0:50 DDE-COMMAND assoc entry (VECTOR (FIELD S MAGNETIC UNSPECIFIED
TIME NIL) ZERO)0:50 DDE-COMMAND assoc error DEFAULT-SHOULD-BE-NON-ZERO0:50 DDE-COMMAND assoc step (VECTOR (FIELD S MAGNETIC UNSPECIFIED TIME
NIL) OUT-OF)0:50 DDE-COMMAND assoc op DRAW-FIELD-GIVEN-DIR0:50 DDE-COMMAND set-score 20:50 DDE-RESULT |NIL|...9:51 DDE-RESULT |T|9:55 END-LOG
32
Agenda
I. LearnLab methodology
II. Demonstration of Andes,
an intelligent homework tutor
III. Log File Analysis Next
33
Model Solution Set
Solution 0
Principle A
Op1
Op3
Op6
Op7
Principle B
Op2
Op3
Op5
Op8
Op10
Solution 1
Principle C
Op10
Op11
Op12
Principle A
Op1
Op3
Op6
Op7
Principle D
Assumption: Opi = KC
34
# Log of Andes session begun Friday, July 27, 2007 14:29:38 by bobh on BOBH
…
0:02 DDE (read-problem-info "S2E" 0 0)
…
11:45 Vector-dlg Vector-673 ||
…
11:48 CLOSE type instantaneous
11:48 SEL type 1 instantaneous
11:51 BTN-CLICK 1 OK
11:51 DDE (lookup-vector "a" instantaneous Acceleration |ball| NIL 0 |T0| Vector-673)
11:51 DDE-COMMAND assoc step (VECTOR (ACCEL BALL :TIME 1) ZERO)
11:51 DDE-COMMAND assoc op ACCEL-AT-REST
11:51 DDE-RESULT |T|
…
problem name
student actions
match model solution:assoc step = entryAssoc op = operator
green
35
14:03 E 8 Fearth_y = m*g
14:11 EQ-SUBMIT 8
14:11 DDE (lookup-eqn-string "Fearth_y = m*g" 8)
14:11 DDE-COMMAND assoc parse (= Yc_Fw_BALL_EARTH_1_0 (* m_BALL g_EARTH))
14:11 DDE-COMMAND assoc error MISSING-NEGATION-ON-VECTOR-COMPONENT
14:11 DDE-COMMAND assoc step (EQN (= Fw_BALL_EARTH_1 (* m_BALL g_EARTH))),(EQN (= Yc_Fw_BALL_EARTH_1_0 (- Fw_BALL_EARTH_1)))
14:11 DDE-COMMAND assoc op WT-LAW,COMPO-PARALLEL-AXIS
14:11 DDE-COMMAND set-score 74
14:11 EQ-F 8
14:11 DDE-RESULT |NIL|
student actions
red
guessintended
errorinterpretation
36
Review Video
Match steps in video to log file
37
Researchable questions
Timing Sequencing(order of steps)
Hint Usage
Problem solving skills
Errors as window to mental state
Self-correction of errors
38
DataShophttps://learnlab.web.cmu.edu/
> Launch DataShop> New user? Sign up now!> (Create account)