The Mystery of “b := (b = false)”

Stuart Reges

University of Washingtonhttp://www.cs.washington.edu/homes/reges/mystery

Data overview 1988 AP Computer Science Exam (Pascal) Divided into A and B (roughly CS1 and CS2) 35 A multiple choice 15 B multiple choice 3 A free response 2 B free response 7,374 AB tests; 3,344 A-only tests

Analysis overview Correlations between multiple choice

questions Any correlation below 0.2 ignored Correlations between multiple choice

and free response questions Didn't find much of interest in correlating

free response with free response

The mystery? In general, not much correlated at the

level of 0.2 or higher But, five multiple choice questions kept

popping up over and over nickname: the powerhouse questions One question in particular (question 23)

stood out

A multiple choice correlations

Minus powerhouse questions

B multiple choice correlations

Minus powerhouse questions

Free response correlations

1 (A) 2 (A) 3 (A) 4 (B) 5 (B)

1 (A) 1 0.46 0.37 0.35 0.35

2 (A) 0.46 1 0.42 0.37 0.39

3 (A) 0.37 0.42 1 0.42 0.48

4 (B) 0.35 0.37 0.42 1 0.48

5 (B) 0.35 0.39 0.48 0.48 1

Free response vs powerhouse

1 (A) 2 (A) 3 (A) 4 (B) 5 (B)

14 0.30 0.35 0.34 0.27 0.30

15 0.34 0.38 0.36 0.31 0.31

18 0.32 0.39 0.30 0.28 ?

20 0.34 0.41 0.31 0.28 0.30

23 0.36 0.44 0.37 0.32 0.32

Greatest AB vs A-only deltas

Question AB correct A correct Delta

20 69.2% 38.7% 30.5

23 60.0% 35.3% 24.7

15 66.3% 43.8% 22.5

32 46.6% 25.7% 20.9

14 65.2% 46.0% 19.2

18 71.7% 52.6% 19.1

Mysterious question 23If b is a Boolean variable, then the statement

b := (b = false) has what effect?A. It causes a compile-time error message.B. It causes a run-time error message.C. It causes b to have value false regardless of its value

just before the statement was executed.D. It always changes the value of b.E. It changes the value of b if and only if b had value

true just before the statement was executed.

And question 20procedure Wow(n : integer);begin if n > 1 then Wow(n div 2); write(n, ' ')end;The procedure call Wow(16) will yield as output which of the following

sequences of numbers?A. 10 8 6 4 2B. 16 8 4 2 1C. 1 2 4 8 16D. 32 16 8 4 2E. 2 4 8 16 32

What does this mean? Bob Floyd: These questions seem to

test whether a student has a model of computation; whether they can play computer in their head

Don Knuth: "I conclude that roughly 2% of all people ‘think algorithmically,’ in the sense that they can reason rapidly about algorithmic processes."

More Knuth“The other missing concept that seems to separate mathematicians from computer scientists is related to the ‘assignment operation’ :=, which changes values of quantities. More precisely, I would say that the missing concept is the dynamic notion of the state of a process. ‘How did I get here? What is true now? What should happen next if I’m going to get to the end?’ Changing states of affairs, or snapshots of a computation, seem to be intimately related to algorithms and algorithmic thinking.”

Snapshot examplepublic static int mystery(int n) { int x = 0; while (n % 2 == 0) { // Point A n = n / 2; x++; // Point B } // Point C return x;}

Is (n % 2 == 0) always true, never true, or sometimes true/sometimes false at points A, B and C?

What next? More exploration, more studies Powerhouse questions suggest certain

kinds of questions to explore Is this measuring some innate aptitude?

Some acquired ability? Or is it a fluke? Join the discussion:http://www.cs.washington.edu/homes/reges/mystery