![Page 1: Automatic Generation of Programming Feedback: A Data-Driven Approach](https://reader036.vdocument.in/reader036/viewer/2022081515/56813b00550346895da39d7a/html5/thumbnails/1.jpg)
Automatic Generation of Programming Feedback:
A Data-Driven Approach
Kelly Rivers and Ken Koedinger
1
![Page 2: Automatic Generation of Programming Feedback: A Data-Driven Approach](https://reader036.vdocument.in/reader036/viewer/2022081515/56813b00550346895da39d7a/html5/thumbnails/2.jpg)
Programming is Hard
2
![Page 3: Automatic Generation of Programming Feedback: A Data-Driven Approach](https://reader036.vdocument.in/reader036/viewer/2022081515/56813b00550346895da39d7a/html5/thumbnails/3.jpg)
The Importance of Feedback
3
Corbett & Anderson, 1991
![Page 4: Automatic Generation of Programming Feedback: A Data-Driven Approach](https://reader036.vdocument.in/reader036/viewer/2022081515/56813b00550346895da39d7a/html5/thumbnails/4.jpg)
But This Takes Time!
4
![Page 5: Automatic Generation of Programming Feedback: A Data-Driven Approach](https://reader036.vdocument.in/reader036/viewer/2022081515/56813b00550346895da39d7a/html5/thumbnails/5.jpg)
Research Question: Can We Make it Automatically?
5
![Page 6: Automatic Generation of Programming Feedback: A Data-Driven Approach](https://reader036.vdocument.in/reader036/viewer/2022081515/56813b00550346895da39d7a/html5/thumbnails/6.jpg)
What We Currently Have
6
![Page 7: Automatic Generation of Programming Feedback: A Data-Driven Approach](https://reader036.vdocument.in/reader036/viewer/2022081515/56813b00550346895da39d7a/html5/thumbnails/7.jpg)
And more…
• Knowledge Modeling– Syntax Patterns
– Plans/Templates/Clichés
– Error models (Singh et al, 2013)
7
![Page 8: Automatic Generation of Programming Feedback: A Data-Driven Approach](https://reader036.vdocument.in/reader036/viewer/2022081515/56813b00550346895da39d7a/html5/thumbnails/8.jpg)
The Solution Space
8
Each node is a solution state that a student might reach, incorrect or correct
Each edge is the transition between the first state and the second, the edits made to the solution
![Page 9: Automatic Generation of Programming Feedback: A Data-Driven Approach](https://reader036.vdocument.in/reader036/viewer/2022081515/56813b00550346895da39d7a/html5/thumbnails/9.jpg)
Our Approach
9
![Page 10: Automatic Generation of Programming Feedback: A Data-Driven Approach](https://reader036.vdocument.in/reader036/viewer/2022081515/56813b00550346895da39d7a/html5/thumbnails/10.jpg)
Step 0: Solution Space Setup
10
![Page 11: Automatic Generation of Programming Feedback: A Data-Driven Approach](https://reader036.vdocument.in/reader036/viewer/2022081515/56813b00550346895da39d7a/html5/thumbnails/11.jpg)
We start with a collection of solution states.
11
![Page 12: Automatic Generation of Programming Feedback: A Data-Driven Approach](https://reader036.vdocument.in/reader036/viewer/2022081515/56813b00550346895da39d7a/html5/thumbnails/12.jpg)
The programming solution space for a given problem is very crowded.
12
![Page 13: Automatic Generation of Programming Feedback: A Data-Driven Approach](https://reader036.vdocument.in/reader036/viewer/2022081515/56813b00550346895da39d7a/html5/thumbnails/13.jpg)
Program Normalization
• Remove dead code and comments• Variable propagation• Commutative Expression Ordering• Anonymize variable names
13
Rivers, K., and Koedinger, K. (2012). A Canonicalizing Model for Building Programming Tutors. In Proceedings of the 11th International Conference on Intelligent Tutoring Systems (pp. 591-593).
![Page 14: Automatic Generation of Programming Feedback: A Data-Driven Approach](https://reader036.vdocument.in/reader036/viewer/2022081515/56813b00550346895da39d7a/html5/thumbnails/14.jpg)
Program Normalization
Original Student Programdef findPattern(s, pattern, startIndex): # This should return the location #of the pattern l = len(s) for i in range(l): if (findPatternAtIndex(s, pattern, startIndex + i) == True): return i + startIndex # return ??
Normalized Versiondef findPattern(s, pattern, startIndex): l = len(s) for i in range(l): if (findPatternAtIndex(s, pattern, startIndex + i) == True): return i + startIndex
14
![Page 15: Automatic Generation of Programming Feedback: A Data-Driven Approach](https://reader036.vdocument.in/reader036/viewer/2022081515/56813b00550346895da39d7a/html5/thumbnails/15.jpg)
Program Normalization
Original Student Programdef findPattern(s, pattern, startIndex): l = len(s) for i in range(l): if (findPatternAtIndex(s, pattern, startIndex + i) == True): return i + startIndex
Normalized Versiondef findPattern(s, pattern, startIndex): for i in range(len(s)): if (findPatternAtIndex(s, pattern, startIndex + i) == True): return i + startIndex
15
![Page 16: Automatic Generation of Programming Feedback: A Data-Driven Approach](https://reader036.vdocument.in/reader036/viewer/2022081515/56813b00550346895da39d7a/html5/thumbnails/16.jpg)
Program Normalization
Original Student Programdef findPattern(s, pattern, startIndex): for i in range(len(s)): if (findPatternAtIndex(s, pattern, startIndex + i) == True): return i + startIndex
Normalized Versiondef findPattern(s, pattern, startIndex): for i in range(len(s)): if findPatternAtIndex(s, pattern, startIndex + i): return i + startIndex
16
![Page 17: Automatic Generation of Programming Feedback: A Data-Driven Approach](https://reader036.vdocument.in/reader036/viewer/2022081515/56813b00550346895da39d7a/html5/thumbnails/17.jpg)
Program Normalization
Original Student Programdef findPattern(s, pattern, startIndex): for i in range(len(s)): if findPatternAtIndex(s, pattern, startIndex + i): return i + startIndex
Normalized Versiondef findPattern(s, pattern, startIndex): for i in range(len(s)): if findPatternAtIndex(s, pattern, startIndex + i): return startIndex + i
17
![Page 18: Automatic Generation of Programming Feedback: A Data-Driven Approach](https://reader036.vdocument.in/reader036/viewer/2022081515/56813b00550346895da39d7a/html5/thumbnails/18.jpg)
Program Normalization
Original Student Programdef findPattern(s, pattern, startIndex): for i in range(len(s)): if findPatternAtIndex(s, pattern, startIndex + i): return startIndex + i
Normalized Versiondef findPattern(v0, v1, v2): for v3 in range(len(v0)): if findPatternAtIndex(v0, v1, v2 + v3): return v2 + v3
18
![Page 19: Automatic Generation of Programming Feedback: A Data-Driven Approach](https://reader036.vdocument.in/reader036/viewer/2022081515/56813b00550346895da39d7a/html5/thumbnails/19.jpg)
Once normalized, the solution space has a more reasonable scope, and some common states are evident
19
![Page 20: Automatic Generation of Programming Feedback: A Data-Driven Approach](https://reader036.vdocument.in/reader036/viewer/2022081515/56813b00550346895da39d7a/html5/thumbnails/20.jpg)
When state correctness is added, common paths can be found as well.
20
![Page 21: Automatic Generation of Programming Feedback: A Data-Driven Approach](https://reader036.vdocument.in/reader036/viewer/2022081515/56813b00550346895da39d7a/html5/thumbnails/21.jpg)
Step 1: Find Optimal Learning Progression
21
![Page 22: Automatic Generation of Programming Feedback: A Data-Driven Approach](https://reader036.vdocument.in/reader036/viewer/2022081515/56813b00550346895da39d7a/html5/thumbnails/22.jpg)
Insert New State
22
![Page 23: Automatic Generation of Programming Feedback: A Data-Driven Approach](https://reader036.vdocument.in/reader036/viewer/2022081515/56813b00550346895da39d7a/html5/thumbnails/23.jpg)
Finding the Best Path
• Distance function– Tree edits– Levenshtein string distance– Feature vectors
• Chains of actions– Sequence of states to closest correct
23
![Page 24: Automatic Generation of Programming Feedback: A Data-Driven Approach](https://reader036.vdocument.in/reader036/viewer/2022081515/56813b00550346895da39d7a/html5/thumbnails/24.jpg)
Step 2: State Transition - Edits
24
• Deletions: ([code lines], [])– Semantically unnecessary code
• Changes: ([code fragment], [code fragment])– Switching from one version to another
• Additions: ([], [code lines])– Missing a step in the solution
![Page 25: Automatic Generation of Programming Feedback: A Data-Driven Approach](https://reader036.vdocument.in/reader036/viewer/2022081515/56813b00550346895da39d7a/html5/thumbnails/25.jpg)
State Transition - Trace
def findPattern(v0, v1, v2): for v3 in range(len(v0)): if findPatternAtIndex(v0, v1, v3): return v2 + v3 return -1
25
![Page 26: Automatic Generation of Programming Feedback: A Data-Driven Approach](https://reader036.vdocument.in/reader036/viewer/2022081515/56813b00550346895da39d7a/html5/thumbnails/26.jpg)
Step 3: Generating Individual Feedback
26
![Page 27: Automatic Generation of Programming Feedback: A Data-Driven Approach](https://reader036.vdocument.in/reader036/viewer/2022081515/56813b00550346895da39d7a/html5/thumbnails/27.jpg)
Levels of Hints
• Location: What line needs to be changed?– Make a change in line 26.
• Content: Which code fragment is wrong?– Change v3 in line 26.
• Edit: What is the correct code?– Replace v3 with v2+v3 in line 26.
27
![Page 28: Automatic Generation of Programming Feedback: A Data-Driven Approach](https://reader036.vdocument.in/reader036/viewer/2022081515/56813b00550346895da39d7a/html5/thumbnails/28.jpg)
The Feedback Doesn’t Match
23 def findPattern(s, pattern, startIndex): 24 l = len(s) 25 for i in range(l): 26 if findPatternAtIndex(s, pattern, startIndex + i) == True: 27 return i 28 return -1
Replace v3 with v2+ v3 in line 26.
28
![Page 29: Automatic Generation of Programming Feedback: A Data-Driven Approach](https://reader036.vdocument.in/reader036/viewer/2022081515/56813b00550346895da39d7a/html5/thumbnails/29.jpg)
But it’s Normalized!
23 def findPattern(v0, v1, v2): 24 for v3 in range(len(v0)): 25 if findPatternAtIndex(v0, v1, v2 + v3): 26 return v3 27 return -1
Replace v3 with v2+ v3 in line 26.
29
![Page 30: Automatic Generation of Programming Feedback: A Data-Driven Approach](https://reader036.vdocument.in/reader036/viewer/2022081515/56813b00550346895da39d7a/html5/thumbnails/30.jpg)
Undo the Transformations
30
![Page 31: Automatic Generation of Programming Feedback: A Data-Driven Approach](https://reader036.vdocument.in/reader036/viewer/2022081515/56813b00550346895da39d7a/html5/thumbnails/31.jpg)
Many to One
31
![Page 32: Automatic Generation of Programming Feedback: A Data-Driven Approach](https://reader036.vdocument.in/reader036/viewer/2022081515/56813b00550346895da39d7a/html5/thumbnails/32.jpg)
Unrolling the Trace
def findPattern(v0, v1, v2): for v3 in range(len(v0)): if findPatternAtIndex(v0, v1, v2 + v3): return v3 return -1
32
![Page 33: Automatic Generation of Programming Feedback: A Data-Driven Approach](https://reader036.vdocument.in/reader036/viewer/2022081515/56813b00550346895da39d7a/html5/thumbnails/33.jpg)
Unrolling the Trace
def findPattern(s, pattern, startIndex): for i in range(len(s)): if findPatternAtIndex(s, pattern, startIndex + i) == True: return i return -1
33
![Page 34: Automatic Generation of Programming Feedback: A Data-Driven Approach](https://reader036.vdocument.in/reader036/viewer/2022081515/56813b00550346895da39d7a/html5/thumbnails/34.jpg)
Mapping the Transformations?
• Deleted lines• Extra code• Reordered expressions
• How?
34
![Page 35: Automatic Generation of Programming Feedback: A Data-Driven Approach](https://reader036.vdocument.in/reader036/viewer/2022081515/56813b00550346895da39d7a/html5/thumbnails/35.jpg)
35
![Page 36: Automatic Generation of Programming Feedback: A Data-Driven Approach](https://reader036.vdocument.in/reader036/viewer/2022081515/56813b00550346895da39d7a/html5/thumbnails/36.jpg)
Overview
36
![Page 37: Automatic Generation of Programming Feedback: A Data-Driven Approach](https://reader036.vdocument.in/reader036/viewer/2022081515/56813b00550346895da39d7a/html5/thumbnails/37.jpg)
Let’s try it!
37
![Page 38: Automatic Generation of Programming Feedback: A Data-Driven Approach](https://reader036.vdocument.in/reader036/viewer/2022081515/56813b00550346895da39d7a/html5/thumbnails/38.jpg)
So What?
38
![Page 39: Automatic Generation of Programming Feedback: A Data-Driven Approach](https://reader036.vdocument.in/reader036/viewer/2022081515/56813b00550346895da39d7a/html5/thumbnails/39.jpg)
Research Question
• Automatically generate feedback
39
![Page 40: Automatic Generation of Programming Feedback: A Data-Driven Approach](https://reader036.vdocument.in/reader036/viewer/2022081515/56813b00550346895da39d7a/html5/thumbnails/40.jpg)
Research Question
• Automatically generate feedback
• … in order to make programming less painful for novices
40
![Page 41: Automatic Generation of Programming Feedback: A Data-Driven Approach](https://reader036.vdocument.in/reader036/viewer/2022081515/56813b00550346895da39d7a/html5/thumbnails/41.jpg)
How to Measure?
• Rate relevance of messages– Relation to test results– Targeted solution
• Test with real students!– Fall 2013
41
![Page 42: Automatic Generation of Programming Feedback: A Data-Driven Approach](https://reader036.vdocument.in/reader036/viewer/2022081515/56813b00550346895da39d7a/html5/thumbnails/42.jpg)
For students, we hope…
• Help them squash ‘impossible’ bugs
• Recommend how correct solutions can become better
42
![Page 43: Automatic Generation of Programming Feedback: A Data-Driven Approach](https://reader036.vdocument.in/reader036/viewer/2022081515/56813b00550346895da39d7a/html5/thumbnails/43.jpg)
For teachers, we hope…
• Help them target struggling students
• Discover missing knowledge components
43
![Page 44: Automatic Generation of Programming Feedback: A Data-Driven Approach](https://reader036.vdocument.in/reader036/viewer/2022081515/56813b00550346895da39d7a/html5/thumbnails/44.jpg)
Discussion
• Limitation: reliance on previously collected data
• Learning to debug: when do we stop giving feedback?
• Open-ended problems: how to approach?
44
![Page 45: Automatic Generation of Programming Feedback: A Data-Driven Approach](https://reader036.vdocument.in/reader036/viewer/2022081515/56813b00550346895da39d7a/html5/thumbnails/45.jpg)
Next Steps
• Generalize for many teachers…
• … and other languages…
• … and even other domains?
45
![Page 46: Automatic Generation of Programming Feedback: A Data-Driven Approach](https://reader036.vdocument.in/reader036/viewer/2022081515/56813b00550346895da39d7a/html5/thumbnails/46.jpg)
Questions?
46
This work was supported in part by Graduate Training Grant awarded to Carnegie Mellon University by the Department of Education (# R305B090023).
![Page 47: Automatic Generation of Programming Feedback: A Data-Driven Approach](https://reader036.vdocument.in/reader036/viewer/2022081515/56813b00550346895da39d7a/html5/thumbnails/47.jpg)
Time on Task
47
Perkins & Martin, 1986; Jadud, 2006
Stoppers Tinkerers Movers