changes in probabilistic causal models

Changes in Probabilistic Causal Models

or: Do we stink at probability or what?

James Baker

PSY446

Participation Mostly NU undergrads Some NU grad students Early data collection error / internet preteens filtered

67 data records 53 finished samples

13 of type 0 14 of type 1 13 of type 2 13 of type 3

Gametype 0 (control)

A: Display "33%" Prob: .35 Price: $3 B: Display "66%" Prob: .68 Price: $6 C: Display "97%" Prob: .95 Price: $10 No changes

Responses to gametype 0 I can get 20 right if I always choose the most expensive... I didn't really know how the computer was operating. I just

figured that if the probability was high I would follow that suggestion.

I felt like 'B' was more than 66 percent, so after trying the other two, I stuck with B.

I chose to go with query B every time, because, on average, this would return 22 correct answers. Query A would do the same. And, I think, query C would return about 19, on average. Unless my math is wrong (which is very possible)...consistently choosing A or B is the best bet.

so i guess i got more 'correct' answers when i just played many times with low probablility of winning- but again- it's a little too late in the day for me to actually strategize.

Responses to gametype 0 I think the computer worked on the probability given,

but when you moved to a higher cost value, it decreased the likelihood that the probability given would work at a lower cost value. (Therefore, if you ask at $3, then move to $6, $3 will be less likely to work in subsequent rounds.) I mostly stayed with the $6 option, only moving to $10 when I wanted a correct answer and $3 when I wanted $6 to be more likely, which seemed to work.

I spent nearly all my money on the 97% question. i would stick with c until it stopped being correct,

then move to b for a while until it stopped working then go back to c, etc

Results-0

> 90% A’s: 3 > 75% B’s: 4 > 70% C’s: 4 30% < B and C < 60%: 2

Conclusion: Tendency to pick one favorite

Gametype 1

A: Display "30%" Prob: .32 Price: $3 B: Display "40%" Prob: .22 Price: $4 C: Display "50%" Prob: .52 Price: $5 Equally good choices B ramps from .22 to .92 Hypothesis: People will like C the best,

might detect something off about B.

Responses to gametype 1 If the computer was trying to pick up on my strategy, it would be

pretty easy to learn the strategy. I picked C every time. Although it was the most expensive option, it had the highest probability of a right answer. So I picked it each time.

i would stick with c until it stopped being correct, then move to b for a while until it stopped working then go to a, then start over

totaly random Honestly, I have no idea how the computer was operating. I think

asking the same thing, then changing to something less likely, then going back to the original query, makes it more likely to get the correct answer, so I continued to follow that strategy.

Probablility shouldn't necessarily play into what results occur because it is still a tossup and 40% still seemed to produce positive results about 50 % of the time..

Responses to gametype 1

I figured it was random, so I always picked A which was had the highest return rate.

I didn't make any changes; selecting C everytime seemed to be working.

It seems that 50% of a chance would be better odds despite the cost.

i figured i would last the longest if i kept choosing C The odds for this game were that I would get twenty

queries right, so I always went with the most expensive query because it was quicker.

Results-1

> 95% A’s: 2 > 70% B’s: 1 > 70% C’s: 6 Mixed: 4 Among mixed: 57 A, 61 B, 77 C

Conclusion: We like round numbers?

Gametype 2

A: Display "35%" Prob: .37 Price: $3 B: Display "65%" Prob: .67 Price: $5 C: Display "95%" Prob: 1.0 Price: $7 C starts as best choice C ramps from 1.0 to .3 Hypothesis: People will start on C, then

change to B/A when C gets worse.

Responses to gametype 2 Same logic as last time (divided $200 by the cost/query and then

multiplied by the odds of that query returning a "correct answer"). I predicted 27 correct answers for choice C, which would have been the most.

considering that the correct answer changed (I haven't figured out the pattern yet) when I stopped getting right answers I would switch.

Option C maximizes expected profit in the long run. Hence, I believe in following it should return in the best outcome in the long-run.

I just tried the 95% to play it safe. i didn't change my strategy at all, i just kept going for what i

thought was the most consistent, c every time. stick with highest odds

Responses to gametype 2 I think the computer worked on the probability given, but when

you moved to a higher cost value, it decreased the likelihood that the probability given would work at a lower cost value. (Therefore, if you ask at $3, then move to $5, $3 will be less likely to work in subsequent rounds.) I mostly stayed with the $5 option, since it was a better than 50% chance and it cost an average amount.

just played C as long as I could (in writing, it seemed like the best deal)

The correct answer was not returned at anywhere near 95%, so I don't know how I'm supposed to guess which query to go with.

I stayed with the highest probability option It seemed that the computer started returning wrong answers for

C more than 5% of the time, even when I would vary my answers.

Results-2

>90% B: 1 >75% C: 10 50 < C < 60: 2 The last two started with strings of C

Conclusion: Very slow to change mental model from samples

Gametype 3

A: Display "30%" Prob: .10 Price: $1 B: Display "60%" Prob: .35 Price: $6 C: Display "90%" Prob: .70 Price: $10 A appears to be best, but isn’t A ramps from .1 to .6 Hypothesis: People will start on A, discover

it’s bad, switch to C/B, move around a bit, perhaps find A again when it’s good

Responses to gametype 3 I stuck mostly with the cheapest one this time becuse it only cost

$1. I was able to score pretty high by being able to ask such cheap questions.

When the $1 started winning I just kept going with those. I assumed that the probabilities set out in the beginning of the task were going to be followed so I just tried to mix up the type of answer I chose and was more patient with A and B.

Same strategy as always. Loyally played the odds. I only predicted 60 correct answers...but I guess I got rewarded for hanging in there. Vegas here I come!

i chose a alot since it was so cheap i thought that there might have been different questions

eventually- since there weren't i decided to stick with the most likely 'correct' answer from the computer.

Responses to gametype 3

the cheapest query gave the best odds by about a factor of 3. I got even more right than the odds predicted.

tried A, but it failed 10 in a row; so then I tested C, and I was convined it was legit, so I plkayed it utnil the end, when I could only afford B

I tried the riskier ones when I got low on money. seemed as though ninety percent wasn't actually

appropriate.. it seemed more like 66%

Results

> 80% A: 5 > 90% B: 1 > 70% C: 0 Mixed: 7 Of A’s, 2 were ALL As and 3 had other

selections in the middle section.

Conclusion: More churn with bad outcomes, again slow to change model of A

Aggregate Results

Tendency to pick one favorite We like round numbers? Very slow to change mental model from

samples More churn with bad outcomes, again slow to

change model of A Disconnect between comments and

unconscious trends?

Confounding Factors

Display of percentages reinforcing static mental model

Fin

James Baker

Backup Gametype 0 20 choices: 0 A's, 0 B's, 20 C's. .00% A, .00% B, 1.00% C. 55 choices: 50 A's, 0 B's, 5 C's. .91% A, .00% B, .09% C. 20 choices: 0 A's, 0 B's, 20 C's. .00% A, .00% B, 1.00% C. 32 choices: 3 A's, 25 B's, 4 C's. .09% A, .78% B, .12% C. 30 choices: 6 A's, 15 B's, 9 C's. .20% A, .50% B, .30% C. 24 choices: 4 A's, 3 B's, 17 C's. .17% A, .12% B, .71% C. 35 choices: 5 A's, 29 B's, 1 C's. .14% A, .83% B, .03% C. 20 choices: 0 A's, 0 B's, 20 C's. .00% A, .00% B, 1.00% C. 33 choices: 0 A's, 33 B's, 0 C's. .00% A, 1.00% B, .00% C. 27 choices: 1 A's, 16 B's, 10 C's. .04% A, .59% B, .37% C. 60 choices: 55 A's, 4 B's, 1 C's. .92% A, .07% B, .02% C. 33 choices: 0 A's, 33 B's, 0 C's. .00% A, 1.00% B, .00% C. 66 choices: 66 A's, 0 B's, 0 C's. 1.00% A, .00% B, .00% C.

Backup Gametype 1 42 choices: 3 A's, 4 B's, 35 C's. .07% A, .10% B, .83% C. 40 choices: 0 A's, 0 B's, 40 C's. .00% A, .00% B, 1.00% C. 47 choices: 7 A's, 21 B's, 19 C's. .15% A, .45% B, .40% C. 51 choices: 20 A's, 15 B's, 16 C's. .39% A, .29% B, .31% C. 49 choices: 5 A's, 35 B's, 9 C's. .10% A, .71% B, .18% C. 47 choices: 16 A's, 3 B's, 28 C's. .34% A, .06% B, .60% C. 50 choices: 14 A's, 22 B's, 14 C's. .28% A, .44% B, .28% C. 65 choices: 63 A's, 0 B's, 2 C's. .97% A, .00% B, .03% C. 40 choices: 0 A's, 0 B's, 40 C's. .00% A, .00% B, 1.00% C. 40 choices: 0 A's, 0 B's, 40 C's. .00% A, .00% B, 1.00% C. 40 choices: 0 A's, 0 B's, 40 C's. .00% A, .00% B, 1.00% C. 41 choices: 2 A's, 2 B's, 37 C's. .05% A, .05% B, .90% C. 40 choices: 0 A's, 0 B's, 40 C's. .00% A, .00% B, 1.00% C. 66 choices: 66 A's, 0 B's, 0 C's. 1.00% A, .00% B, .00% C.

Backup Gametype 2 29 choices: 1 A's, 0 B's, 28 C's. .03% A, .00% B, .97% C. 35 choices: 7 A's, 9 B's, 19 C's. .20% A, .26% B, .54% C. 29 choices: 1 A's, 0 B's, 28 C's. .03% A, .00% B, .97% C. 29 choices: 0 A's, 2 B's, 27 C's. .00% A, .07% B, .93% C. 29 choices: 1 A's, 0 B's, 28 C's. .03% A, .00% B, .97% C. 29 choices: 1 A's, 0 B's, 28 C's. .03% A, .00% B, .97% C. 29 choices: 1 A's, 0 B's, 28 C's. .03% A, .00% B, .97% C. 31 choices: 2 A's, 5 B's, 24 C's. .06% A, .16% B, .77% C. 40 choices: 2 A's, 37 B's, 1 C's. .05% A, .93% B, .03% C. 29 choices: 1 A's, 0 B's, 28 C's. .03% A, .00% B, .97% C. 29 choices: 1 A's, 0 B's, 28 C's. .03% A, .00% B, .97% C. 29 choices: 1 A's, 0 B's, 28 C's. .03% A, .00% B, .97% C. 33 choices: 2 A's, 12 B's, 19 C's. .06% A, .36% B, .58% C.

Backup Gametype 3 115 choices: 98 A's, 17 B's, 0 C's. .85% A, .15% B, .00% C. 38 choices: 12 A's, 18 B's, 8 C's. .32% A, .47% B, .21% C. 200 choices: 200 A's, 0 B's, 0 C's. 1.0% A, .00% B, .00% C. 35 choices: 2 A's, 33 B's, 0 C's. .06% A, .94% B, .00% C. 119 choices: 106 A's, 9 B's, 4 C's. .89% A, .08% B, .03% C. 84 choices: 68 A's, 7 B's, 9 C's. .81% A, .08% B, .11% C. 39 choices: 18 A's, 7 B's, 14 C's. .46% A, .18% B, .36% C. 32 choices: 4 A's, 21 B's, 7 C's. .12% A, .66% B, .22% C. 44 choices: 24 A's, 6 B's, 14 C's. .55% A, .14% B, .32% C. 200 choices: 200 A's, 0 B's, 0 C's. 1.0% A, .00% B, .00% C. 33 choices: 10 A's, 10 B's, 13 C's. .30% A, .30% B, .39% C. 28 choices: 8 A's, 2 B's, 18 C's. .29% A, .07% B, .64% C. 26 choices: 4 A's, 6 B's, 16 C's. .15% A, .23% B, .62% C.

Backup Gametype 0 C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C, A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,

A,A,A,A,A,A,A,A,A,A,A,A,A,A,C,C,C,C,C, C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C, B,B,B,A,A,A,B,B,B,C,C,C,C,B,B,B,B,B,B,B,B,B,B,B,B,B,B,B,B,B,B,B, C,C,C,C,C,C,B,B,B,B,B,B,B,B,B,B,B,A,A,A,A,A,B,B,B,B,C,C,C,A, C,C,C,C,C,C,C,C,C,C,B,B,C,C,C,C,C,C,C,A,A,A,B,A, B,A,A,A,B,B,B,B,B,B,B,B,B,B,B,C,A,B,B,B,B,B,B,B,B,B,B,B,B,B,B,B,B,B,A, C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C, B,B,B,B,B,B,B,B,B,B,B,B,B,B,B,B,B,B,B,B,B,B,B,B,B,B,B,B,B,B,B,B,B, B,B,C,C,C,C,B,B,B,B,B,B,B,B,B,B,B,B,C,C,C,C,B,B,C,C,A, A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,B,A,A,A,A,A,A,A,A,A,B,C,B,A,A,A,

A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,B,A, B,B,B,B,B,B,B,B,B,B,B,B,B,B,B,B,B,B,B,B,B,B,B,B,B,B,B,B,B,B,B,B,B, A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,

A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,

Backup Gametype 1 A,B,C,C,C,C,B,A,B,C,A,C,B,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C, C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C, C,C,C,C,B,C,A,C,C,B,B,A,C,C,C,B,A,A,A,C,C,C,C,C,C,C,B,B,B,A,C,B,A,C,B,B,B,B,B,B,B,B,B,B,B,

B,B, B,A,A,A,A,A,A,A,A,A,A,A,A,A,B,B,B,B,B,B,B,C,C,C,C,C,C,C,C,A,A,A,A,A,B,B,B,B,B,C,C,C,C,C,C,C

,B,B,A,A,C, A,A,A,B,B,B,B,C,C,C,C,C,C,C,C,C,A,B,B,B,B,B,B,B,B,B,B,B,B,B,B,B,B,B,B,B,B,B,B,B,B,B,B,B,B,B,

B,B,A A,A,A,A,A,C,C,C,C,C,C,B,B,A,A,A,A,A,A,A,A,A,A,A,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,

C,B, B,B,B,B,B,B,B,B,B,B,C,C,C,C,C,C,B,B,B,A,A,A,B,B,B,B,C,C,C,C,A,A,A,A,A,A,A,A,A,C,C,B,B,B,B,B

,A,C,C,A, C,C,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,

A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A, C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C, C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C, C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C, C,A,A,B,B,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C, C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C, A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,

A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,

Backup Gametype 2 C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,A, C,C,C,C,C,C,C,A,A,C,A,C,C,C,C,C,B,B,B,B,B,A,A,A,A,C,C,C,C,C,C,B,B,B,B

, C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,A, C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,B,C,C,C,C,C,C,C,C,B, C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,A, C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,A, C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,A, C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,B,B,A,B,B,A,C,C,B,C,C, B,B,B,B,B,B,B,B,B,B,B,A,B,B,B,B,B,B,B,C,B,B,B,A,B,B,B,B,B,B,B,B,B,B,B,B

,B,B,B,B, C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,A, C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,A, C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,A, C,C,C,A,C,C,C,C,C,B,B,B,C,C,C,B,B,B,B,B,B,C,C,C,B,C,C,C,B,C,C,B,A,

Backup Gametype 3 A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,B,B,

B,B,B,B,B,B,B,B,B,B,B,B,B,B,B,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,

B,C,A,A,A,B,B,B,A,B,B,C,C,C,C,B,B,B,B,B,A,A,A,A,A,A,A,A,B,B,B,B,B,B,B,C,C,C, A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,

A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,

B,B,B,B,B,B,B,B,B,B,B,B,B,B,B,B,B,B,B,B,B,B,B,B,B,B,B,B,B,B,B,B,B,A,A, B,B,B,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,C,C,C,C,B,B,B,B,B,B,A,A,A,A,A,A,

A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,

A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,C,C,C,C,A,A,B,B,A,A,A,A,C,C,A,B,B,B,A,C,A,A,A,A,A,A,C,B,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,C,B,A,

C,B,B,A,C,C,C,C,C,B,A,C,C,C,C,C,C,B,B,B,A,C,C,A,A,A,A,A,A,A,A,A,B,A,A,A,A,A,A, C,B,C,C,C,C,C,C,B,B,B,B,B,B,B,B,B,B,B,B,B,B,B,B,B,B,B,B,A,A,A,A, B,B,C,C,C,C,C,B,B,A,A,A,A,A,A,A,A,A,A,A,A,A,A,C,C,C,C,C,C,C,C,C,B,B,A,A,A,A,A,A,A,A,A,A, A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,

A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,

A,A,A,A,A,A,A,A,A,A,C,C,C,B,B,B,B,B,B,B,C,C,C,C,C,C,C,C,C,C,B,B,B, C,B,B,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,A,A,A,A,A,A,A,A, C,B,B,C,C,C,B,C,A,C,B,C,A,C,C,B,C,C,C,C,C,C,C,B,A,A