measuring the effectiveness of error messages designed for novice programmers guillaume marceau...
TRANSCRIPT
![Page 1: Measuring the Effectiveness of Error Messages Designed for Novice Programmers Guillaume Marceau Kathi Fisler (WPI) Shriram Krishnamurthi (Brown) Danny](https://reader030.vdocument.in/reader030/viewer/2022032702/56649cb95503460f94980e16/html5/thumbnails/1.jpg)
1
Measuring the Effectiveness of Error Messages Designed for Novice Programmers
Guillaume MarceauKathi Fisler (WPI)Shriram Krishnamurthi (Brown)Danny Yoo (WPI/Brown, Utah)
![Page 2: Measuring the Effectiveness of Error Messages Designed for Novice Programmers Guillaume Marceau Kathi Fisler (WPI) Shriram Krishnamurthi (Brown) Danny](https://reader030.vdocument.in/reader030/viewer/2022032702/56649cb95503460f94980e16/html5/thumbnails/2.jpg)
2
![Page 3: Measuring the Effectiveness of Error Messages Designed for Novice Programmers Guillaume Marceau Kathi Fisler (WPI) Shriram Krishnamurthi (Brown) Danny](https://reader030.vdocument.in/reader030/viewer/2022032702/56649cb95503460f94980e16/html5/thumbnails/3.jpg)
define: expected only one expression for the function body, but found at least one extra part
![Page 4: Measuring the Effectiveness of Error Messages Designed for Novice Programmers Guillaume Marceau Kathi Fisler (WPI) Shriram Krishnamurthi (Brown) Danny](https://reader030.vdocument.in/reader030/viewer/2022032702/56649cb95503460f94980e16/html5/thumbnails/4.jpg)
How well do error messages support learning(or fail to?)
When errors fail to teach,in which ways do they fail?
What makes a good error message?What is a valid metric of quality?
Can we make recommendations to the creators of pedagogical IDEs/compilers/ languages?
![Page 5: Measuring the Effectiveness of Error Messages Designed for Novice Programmers Guillaume Marceau Kathi Fisler (WPI) Shriram Krishnamurthi (Brown) Danny](https://reader030.vdocument.in/reader030/viewer/2022032702/56649cb95503460f94980e16/html5/thumbnails/5.jpg)
Prior Approaches to Message Effectiveness
“Were the messages helpful?”
No alternative
Did grades improve?
Too coarse
5
![Page 6: Measuring the Effectiveness of Error Messages Designed for Novice Programmers Guillaume Marceau Kathi Fisler (WPI) Shriram Krishnamurthi (Brown) Danny](https://reader030.vdocument.in/reader030/viewer/2022032702/56649cb95503460f94980e16/html5/thumbnails/6.jpg)
6
= execution
= receives an error message
The Student’s Response to the Error Message
= keystroke
![Page 7: Measuring the Effectiveness of Error Messages Designed for Novice Programmers Guillaume Marceau Kathi Fisler (WPI) Shriram Krishnamurthi (Brown) Danny](https://reader030.vdocument.in/reader030/viewer/2022032702/56649cb95503460f94980e16/html5/thumbnails/7.jpg)
[DEL] Deletes the problematic code wholesale.
[UNR]Unrelated to the error message, anddoes not help.
[DIFF]Unrelated to the error message, but it correctly addresses a different error or makes progress in some other way
[PART] Evidence that the student has understood the error message (though perhaps not wholly,) and is trying to take an appropriate action (though perhaps not well.)
[FIX] Fixes the proximate error (though other cringing errors might remain.)
![Page 8: Measuring the Effectiveness of Error Messages Designed for Novice Programmers Guillaume Marceau Kathi Fisler (WPI) Shriram Krishnamurthi (Brown) Danny](https://reader030.vdocument.in/reader030/viewer/2022032702/56649cb95503460f94980e16/html5/thumbnails/8.jpg)
8
Rater #1
goodbad
goodgoodbad
Rater #2goodbad
goodgoodgood
![Page 9: Measuring the Effectiveness of Error Messages Designed for Novice Programmers Guillaume Marceau Kathi Fisler (WPI) Shriram Krishnamurthi (Brown) Danny](https://reader030.vdocument.in/reader030/viewer/2022032702/56649cb95503460f94980e16/html5/thumbnails/9.jpg)
9
Rater #2Rater #1
Inter-rater agreement : κ
0.80 0.58
0.580.38
Expected Agreement = 0.702 + 0.302
= 0.58
0.0 1.0
slight fair moderate substantial excellent
0.20
0.40
0.60
0.80
0.84
0.38
eq. to chance
perfect agreement
worse than chance
![Page 10: Measuring the Effectiveness of Error Messages Designed for Novice Programmers Guillaume Marceau Kathi Fisler (WPI) Shriram Krishnamurthi (Brown) Danny](https://reader030.vdocument.in/reader030/viewer/2022032702/56649cb95503460f94980e16/html5/thumbnails/10.jpg)
10
Lab number #1 #2 #3# of errors a studentreceived during this lab(on average)
8.5 16.3 14.4
% error % bad # bad % error % bad # bad % error % bad # bad
arg. count 5% 48% 0.22 17% 27% 0.74 14% 17% 0.33parens matching 28% 24% 0.58 12% 14% 0.27 17% 0% 0.00runtime cond 3% 0% 0.00 3% 100% 0.49 4% 20% 0.12runtime type 2% 100% 0.15 8% 73% 0.91 16% 40% 0.93syntax cond 14% 51% 0.59 4% 50% 0.31 6% 26% 0.24syntax defi ne 16% 50% 0.68 14% 50% 1.14 6% 15% 0.14syntax func. ca l l 14% 64% 0.74 14% 17% 0.37 12% 14% 0.26syntax s truct 0% 0% 0.00 8% 32% 0.43 5% 92% 0.73unbound id. 16% 16% 0.21 13% 40% 0.85 16% 14% 0.32Total: 3.16 5.51 3.07
#error × %error × %bad = #bad
![Page 11: Measuring the Effectiveness of Error Messages Designed for Novice Programmers Guillaume Marceau Kathi Fisler (WPI) Shriram Krishnamurthi (Brown) Danny](https://reader030.vdocument.in/reader030/viewer/2022032702/56649cb95503460f94980e16/html5/thumbnails/11.jpg)
11
Lab number #1 #2 #3# of errors a studentreceived during this lab(on average)
8.5 16.3 14.4
% error % bad # bad % error % bad # bad % error % bad # bad
arg. count 5% 48% 0.22 17% 27% 0.74 14% 17% 0.33parens matching 28% 24% 0.58 12% 14% 0.27 17% 0% 0.00runtime cond 3% 0% 0.00 3% 100% 0.49 4% 20% 0.12runtime type 2% 100% 0.15 8% 73% 0.91 16% 40% 0.93syntax cond 14% 51% 0.59 4% 50% 0.31 6% 26% 0.24syntax defi ne 16% 50% 0.68 14% 50% 1.14 6% 15% 0.14syntax func. ca l l 14% 64% 0.74 14% 17% 0.37 12% 14% 0.26syntax s truct 0% 0% 0.00 8% 32% 0.43 5% 92% 0.73unbound id. 16% 16% 0.21 13% 40% 0.85 16% 14% 0.32Total: 3.16 5.51 3.07
Lab number #4 #5 #6# of errors a studentreceived during this lab(on average)
9.0 9.8 9.8
% error % bad # bad % error % bad # bad % error % bad # bad
arg. count 13% 20% 0.24 35% 21% 0.74 12% 31% 0.36parens matching 14% 0% 0.00 13% 0% 0.00 10% 15% 0.15runtime cond 6% 72% 0.40 8% 78% 0.62 1% 100% 0.06runtime type 8% 22% 0.17 6% 44% 0.26 3% 38% 0.13syntax cond 10% 28% 0.25 9% 20% 0.17 11% 11% 0.12syntax defi ne 7% 24% 0.14 2% 17% 0.03 3% 38% 0.10syntax func. ca l l 23% 27% 0.55 4% 29% 0.12 13% 38% 0.48syntax s truct 0% 0% 0.00 1% 0% 0.00 0% 0% 0.00unbound id. 16% 0% 0.00 20% 7% 0.14 34% 13% 0.44Total: 1.75 2.07 1.84
Vocabulary difficulties
n-ary tree
Difficult structure accessor names
![Page 12: Measuring the Effectiveness of Error Messages Designed for Novice Programmers Guillaume Marceau Kathi Fisler (WPI) Shriram Krishnamurthi (Brown) Danny](https://reader030.vdocument.in/reader030/viewer/2022032702/56649cb95503460f94980e16/html5/thumbnails/12.jpg)
Reflection
Rubric identifies pain points
Identifying fix needs deeper evaluation
Curriculum Error Messages
12
![Page 13: Measuring the Effectiveness of Error Messages Designed for Novice Programmers Guillaume Marceau Kathi Fisler (WPI) Shriram Krishnamurthi (Brown) Danny](https://reader030.vdocument.in/reader030/viewer/2022032702/56649cb95503460f94980e16/html5/thumbnails/13.jpg)
13
For the Community
Look at fine-grained edits
Gather around a common rubric
![Page 14: Measuring the Effectiveness of Error Messages Designed for Novice Programmers Guillaume Marceau Kathi Fisler (WPI) Shriram Krishnamurthi (Brown) Danny](https://reader030.vdocument.in/reader030/viewer/2022032702/56649cb95503460f94980e16/html5/thumbnails/14.jpg)
14
Interviews
Four interviews
One hour long each
Done around the midterm
Average-to-good students
![Page 15: Measuring the Effectiveness of Error Messages Designed for Novice Programmers Guillaume Marceau Kathi Fisler (WPI) Shriram Krishnamurthi (Brown) Danny](https://reader030.vdocument.in/reader030/viewer/2022032702/56649cb95503460f94980e16/html5/thumbnails/15.jpg)
15
Observation From Interviews
1 of 2
![Page 16: Measuring the Effectiveness of Error Messages Designed for Novice Programmers Guillaume Marceau Kathi Fisler (WPI) Shriram Krishnamurthi (Brown) Danny](https://reader030.vdocument.in/reader030/viewer/2022032702/56649cb95503460f94980e16/html5/thumbnails/16.jpg)
16
Interviewer: The error message says “the function body.”Do you know what “function body” means?
Student: Nah, the input, everything that serves
as a piece of input? Interviewer: Actually, it's this. When DrScheme
says “function body” it means this part. Student: Oh man! I didn't… [The student proceeds to fix the error successfully]
![Page 17: Measuring the Effectiveness of Error Messages Designed for Novice Programmers Guillaume Marceau Kathi Fisler (WPI) Shriram Krishnamurthi (Brown) Danny](https://reader030.vdocument.in/reader030/viewer/2022032702/56649cb95503460f94980e16/html5/thumbnails/17.jpg)
17
What DrRacket Says:
define: expected only one expression for the function body, but found at least one extra part.
![Page 18: Measuring the Effectiveness of Error Messages Designed for Novice Programmers Guillaume Marceau Kathi Fisler (WPI) Shriram Krishnamurthi (Brown) Danny](https://reader030.vdocument.in/reader030/viewer/2022032702/56649cb95503460f94980e16/html5/thumbnails/18.jpg)
18
What the Student Sees:
define: expected only one thingamabob for the geewhiz’s whosiewhatsit, but found one extra whatchamacallit.
![Page 19: Measuring the Effectiveness of Error Messages Designed for Novice Programmers Guillaume Marceau Kathi Fisler (WPI) Shriram Krishnamurthi (Brown) Danny](https://reader030.vdocument.in/reader030/viewer/2022032702/56649cb95503460f94980e16/html5/thumbnails/19.jpg)
19
Circle an identifier in this program
(define (celsius->fahrenheit temp) (+ (* temp 9/5) 32))
RIGHT WRONG
![Page 20: Measuring the Effectiveness of Error Messages Designed for Novice Programmers Guillaume Marceau Kathi Fisler (WPI) Shriram Krishnamurthi (Brown) Danny](https://reader030.vdocument.in/reader030/viewer/2022032702/56649cb95503460f94980e16/html5/thumbnails/20.jpg)
20
Circle one instance of each vocabulary term in the code below
Vocabulary term Sample usage
Q1. Argument >: expects at least 2 arguments, given 1Q2. Selector this selector expects 1 argument, here it is provided 0 argumentsQ3. Procedure this procedure expects 2 arguments, here it is provided 0 argumentsQ4. Expression expected at least two expressions after `and', but found only one expressionQ5. Predicate this predicate expects 1 argument, here it is provided 2 arguments
;; (make-book number string string number number bst bst) (define-struct book (isbn title author year copies left right))
;; this-edition?: bst number number -> boolean;; Consumes a binary search tree, an ISBN number, and a year, and produces true ;; if the book with the given ISBN number was published in the given year(define (this-edition? a-bst isbn-num year) (cond [(symbol? a-bst) false] [(book? a-bst) (cond [(= isbn-num (book-isbn a-bst)) (= year (book-year a-bst))] [(< isbn-num (book-isbn a-bst)) (this-edition? (book-left a-bst) isbn-num year)] [else (this-edition? (book-right a-bst) isbn-num year)])]))
![Page 21: Measuring the Effectiveness of Error Messages Designed for Novice Programmers Guillaume Marceau Kathi Fisler (WPI) Shriram Krishnamurthi (Brown) Danny](https://reader030.vdocument.in/reader030/viewer/2022032702/56649cb95503460f94980e16/html5/thumbnails/21.jpg)
21
< 50%
![Page 22: Measuring the Effectiveness of Error Messages Designed for Novice Programmers Guillaume Marceau Kathi Fisler (WPI) Shriram Krishnamurthi (Brown) Danny](https://reader030.vdocument.in/reader030/viewer/2022032702/56649cb95503460f94980e16/html5/thumbnails/22.jpg)
22
Serendipitous Controlled Trials
BrownNEU WPI
Primitive name Procedure Primitive operator Field name Procedure application Predicate Defined name Type name Identifier Function body Function header
Argument Clause Expression Selector
= USED IN CLASS
![Page 23: Measuring the Effectiveness of Error Messages Designed for Novice Programmers Guillaume Marceau Kathi Fisler (WPI) Shriram Krishnamurthi (Brown) Danny](https://reader030.vdocument.in/reader030/viewer/2022032702/56649cb95503460f94980e16/html5/thumbnails/23.jpg)
Number of times the word was presented to a student in an error message (WPI)
Quiz score residual(after factoring out word difficulty and university strength)
Wordsused in class
Words NOTused in class
![Page 24: Measuring the Effectiveness of Error Messages Designed for Novice Programmers Guillaume Marceau Kathi Fisler (WPI) Shriram Krishnamurthi (Brown) Danny](https://reader030.vdocument.in/reader030/viewer/2022032702/56649cb95503460f94980e16/html5/thumbnails/24.jpg)
Wordsused in class
Words NOTused in class
p = 0.034
(which is statistically significant at the
0.05 level)
For words used in class:more exposure in error messages better quiz scores
For words not used in class:more exposure in error messages worse quiz scores
![Page 25: Measuring the Effectiveness of Error Messages Designed for Novice Programmers Guillaume Marceau Kathi Fisler (WPI) Shriram Krishnamurthi (Brown) Danny](https://reader030.vdocument.in/reader030/viewer/2022032702/56649cb95503460f94980e16/html5/thumbnails/25.jpg)
25
Old term New termProcedurePrimitive namePrimitive operatorPredicateSelectorConstructor
Function
NameIdentifier ArgumentDefined name
Variable, argument(“argument” is reserved for actualarguments in function calls)
Sequence At least oneStructure type name Structure nameQuestion—answer clause A clause is expected to have a question and an answer
Function header KeywordType< > ‘ ’
These words and notations are removed entirely and reworded in terms of other vocabulary words.
Function bodyExpressionField nameType nameTop levelBindingClausePart
These words stay unchanged
![Page 26: Measuring the Effectiveness of Error Messages Designed for Novice Programmers Guillaume Marceau Kathi Fisler (WPI) Shriram Krishnamurthi (Brown) Danny](https://reader030.vdocument.in/reader030/viewer/2022032702/56649cb95503460f94980e16/html5/thumbnails/26.jpg)
26
Observation From Interviews
2 of 2
![Page 27: Measuring the Effectiveness of Error Messages Designed for Novice Programmers Guillaume Marceau Kathi Fisler (WPI) Shriram Krishnamurthi (Brown) Danny](https://reader030.vdocument.in/reader030/viewer/2022032702/56649cb95503460f94980e16/html5/thumbnails/27.jpg)
27
Interviewer: When you get these highlights, what do they mean to you?
Student #1: The problem is between here and here, fix the problem between these two bars.
![Page 28: Measuring the Effectiveness of Error Messages Designed for Novice Programmers Guillaume Marceau Kathi Fisler (WPI) Shriram Krishnamurthi (Brown) Danny](https://reader030.vdocument.in/reader030/viewer/2022032702/56649cb95503460f94980e16/html5/thumbnails/28.jpg)
28
1. This expression contains the error
2. The parser didn’t expect to find this
3. The parser expected to see something after this, but nothing is there
4. This parenthesis is unmatched
5. This expression is inconsistent with another part of the code
DrRacket’s Highlight Semantics
![Page 29: Measuring the Effectiveness of Error Messages Designed for Novice Programmers Guillaume Marceau Kathi Fisler (WPI) Shriram Krishnamurthi (Brown) Danny](https://reader030.vdocument.in/reader030/viewer/2022032702/56649cb95503460f94980e16/html5/thumbnails/29.jpg)
29
Summary of Findings
1. Error messages need to explicate the meaning of the highlight.
2. Students need an avenue through which they will learn the vocabulary.
3. Error messages are hard to get right;user studies are important.
![Page 30: Measuring the Effectiveness of Error Messages Designed for Novice Programmers Guillaume Marceau Kathi Fisler (WPI) Shriram Krishnamurthi (Brown) Danny](https://reader030.vdocument.in/reader030/viewer/2022032702/56649cb95503460f94980e16/html5/thumbnails/30.jpg)
30
Read Understand Formulate
goodbad
goodgoodbad
Lab number #4 #5 #69.0 9.8 9.8
% error % bad # bad % error % bad # bad % error % bad # bad
arg. count 13% 20% 0.24 35% 21% 0.74 12% 31% 0.36parens matching 14% 0% 0.00 13% 0% 0.00 10% 15% 0.15runti me cond 6% 72% 0.40 8% 78% 0.62 1% 100% 0.06runti me type 8% 22% 0.17 6% 44% 0.26 3% 38% 0.13s yntax cond 10% 28% 0.25 9% 20% 0.17 11% 11% 0.12s yntax defi ne 7% 24% 0.14 2% 17% 0.03 3% 38% 0.10s yntax func. ca l l 23% 27% 0.55 4% 29% 0.12 13% 38% 0.48s yntax s truct 0% 0% 0.00 1% 0% 0.00 0% 0% 0.00unbound id. 16% 0% 0.00 20% 7% 0.14 34% 13% 0.44Total: 1.75 2.07 1.84
n-ary tree
[FIX] Fixes the proximate error (though other cringing errors might remain.)
[PART] Evidence that the student has understood the error message (though perhaps not wholly,) and is trying to take an appropriate action (though perhaps not well.)
[UNR] Edit unrelated to the error message, and does not help.
[DEL] Deletes the problematic code wholesale.
[DIFF] Edit unrelated to the error message, but it correctly addresses a different error or makes progress in some other way.
Kathi Fisler (WPI)Shriram Krishnamurthi (Brown)
[email protected]@cs.brown.edu