![Page 1: Hierarchical Bayesian Models for Aggregating Retrieved Memories across Individuals](https://reader036.vdocument.in/reader036/viewer/2022062321/56813454550346895d9b3a85/html5/thumbnails/1.jpg)
Hierarchical Bayesian Models for Aggregating Retrieved Memories across Individuals
Mark Steyvers
Department of Cognitive Sciences
University of California, Irvine
Joint work with:Michael LeeBrent Miller
Pernille HemmerBill Batchelder
Paolo Napoletano
![Page 2: Hierarchical Bayesian Models for Aggregating Retrieved Memories across Individuals](https://reader036.vdocument.in/reader036/viewer/2022062321/56813454550346895d9b3a85/html5/thumbnails/2.jpg)
Thomas Jefferson
Andrew Jackson
James Monroe
George Washington
John AdamsGeorge Washington
Ordering problem:
time
what is the correct order of these Presidents?
![Page 3: Hierarchical Bayesian Models for Aggregating Retrieved Memories across Individuals](https://reader036.vdocument.in/reader036/viewer/2022062321/56813454550346895d9b3a85/html5/thumbnails/3.jpg)
Goal: aggregating responses
3
D A B C A B D C B A D C A C B D A D B C
Aggregation Algorithm
A B C D A B C D
ground truth
=?
group answer
![Page 4: Hierarchical Bayesian Models for Aggregating Retrieved Memories across Individuals](https://reader036.vdocument.in/reader036/viewer/2022062321/56813454550346895d9b3a85/html5/thumbnails/4.jpg)
Bayesian Approach
4
D A B C A B D C B A D C A C B D A D B C
Generative Model
A B C D
ground truth =latent common cause
![Page 5: Hierarchical Bayesian Models for Aggregating Retrieved Memories across Individuals](https://reader036.vdocument.in/reader036/viewer/2022062321/56813454550346895d9b3a85/html5/thumbnails/5.jpg)
Important notes:
No communication between individuals
There is always a true answer (ground truth)
Aggregation algorithm never has access to ground truth ground truth only used for evaluation
5
![Page 6: Hierarchical Bayesian Models for Aggregating Retrieved Memories across Individuals](https://reader036.vdocument.in/reader036/viewer/2022062321/56813454550346895d9b3a85/html5/thumbnails/6.jpg)
Matching problem:
6
RembrandtVan Gogh Monet Renoir
A BC D
![Page 7: Hierarchical Bayesian Models for Aggregating Retrieved Memories across Individuals](https://reader036.vdocument.in/reader036/viewer/2022062321/56813454550346895d9b3a85/html5/thumbnails/7.jpg)
Wisdom of crowds phenomenon
Crowd estimate is often better than any individual in the crowd
(Think of independent noise influencing each individual)
7
![Page 8: Hierarchical Bayesian Models for Aggregating Retrieved Memories across Individuals](https://reader036.vdocument.in/reader036/viewer/2022062321/56813454550346895d9b3a85/html5/thumbnails/8.jpg)
Examples of wisdom of crowds phenomenon
8
Who wants to be a millionaire?Galton’s Ox (1907): Median of individual estimates comes close to true answer
![Page 9: Hierarchical Bayesian Models for Aggregating Retrieved Memories across Individuals](https://reader036.vdocument.in/reader036/viewer/2022062321/56813454550346895d9b3a85/html5/thumbnails/9.jpg)
Limitations of Current “Wisdom of Crowds” Research
Studies restricted to numeric or categorical judgments simple averaging schemes:
Mode Median Mean
No treatment of individual differences every “vote” is treated equally downplayed role of expertise
9
![Page 10: Hierarchical Bayesian Models for Aggregating Retrieved Memories across Individuals](https://reader036.vdocument.in/reader036/viewer/2022062321/56813454550346895d9b3a85/html5/thumbnails/10.jpg)
Cultural Consensus Theory (CCT)E.g. Romney, Batchelder, and Weller (1987)
Finds the “answer key” to multiple choice questions when ground truth is lost takes person and item differences into account
Informal version of CCT also developed for ranking data
10
![Page 11: Hierarchical Bayesian Models for Aggregating Retrieved Memories across Individuals](https://reader036.vdocument.in/reader036/viewer/2022062321/56813454550346895d9b3a85/html5/thumbnails/11.jpg)
Research Goals
Generalize “wisdom of crowds” effect to more complex data
Aggregation of permutations Ranking data Matching (assignment) data
11
![Page 12: Hierarchical Bayesian Models for Aggregating Retrieved Memories across Individuals](https://reader036.vdocument.in/reader036/viewer/2022062321/56813454550346895d9b3a85/html5/thumbnails/12.jpg)
Hierarchical Bayesian Models
Probability distributions over all permutations of items with N items, there are N! combinations e.g., when N=44, we have 44! > 10^53 combinations Approximate inference methods: MCMC
Cognitively plausible generative processes
Treatment of individual differences
12
![Page 13: Hierarchical Bayesian Models for Aggregating Retrieved Memories across Individuals](https://reader036.vdocument.in/reader036/viewer/2022062321/56813454550346895d9b3a85/html5/thumbnails/13.jpg)
Part IOrdering Problems
13
![Page 14: Hierarchical Bayesian Models for Aggregating Retrieved Memories across Individuals](https://reader036.vdocument.in/reader036/viewer/2022062321/56813454550346895d9b3a85/html5/thumbnails/14.jpg)
Experiment 1
Task: order all 44 US presidents
Methods 26 participants (college undergraduates) Names of presidents written on cards Cards could be shuffled on large table
14
![Page 15: Hierarchical Bayesian Models for Aggregating Retrieved Memories across Individuals](https://reader036.vdocument.in/reader036/viewer/2022062321/56813454550346895d9b3a85/html5/thumbnails/15.jpg)
= 1= 1+1Measuring performance
Kendall’s Tau: The number of adjacent pair-wise swaps
Participant Ordering1 2 5 3 4
Ground Truth1 2 3 4 5
3 451 2
1 2 5 3 4
1 2 3 4 5= 2
![Page 16: Hierarchical Bayesian Models for Aggregating Retrieved Memories across Individuals](https://reader036.vdocument.in/reader036/viewer/2022062321/56813454550346895d9b3a85/html5/thumbnails/16.jpg)
Empirical Results
16
1 10 200
100
200
300
400
500
Individuals (ordered from best to worst)
(random guessing)
![Page 17: Hierarchical Bayesian Models for Aggregating Retrieved Memories across Individuals](https://reader036.vdocument.in/reader036/viewer/2022062321/56813454550346895d9b3a85/html5/thumbnails/17.jpg)
Probabilistic models Thurstone (1927) Mallows (1957) Plackett-Luce (1975) Lebanon-Mao (2008)
Spectral methods Diaconis (1989)
Heuristic methods from voting theory Borda count
… however, many of these approached developed for preference rankings
Many approaches for analyzing rank data…
17
![Page 18: Hierarchical Bayesian Models for Aggregating Retrieved Memories across Individuals](https://reader036.vdocument.in/reader036/viewer/2022062321/56813454550346895d9b3a85/html5/thumbnails/18.jpg)
Bayesian Thurstonian Approach
18
Each item has a true coordinate on some dimension
A B C
![Page 19: Hierarchical Bayesian Models for Aggregating Retrieved Memories across Individuals](https://reader036.vdocument.in/reader036/viewer/2022062321/56813454550346895d9b3a85/html5/thumbnails/19.jpg)
Bayesian Thurstonian Approach
19
A B C
… but there is noise because of encoding and/or retrieval error
Person 1
![Page 20: Hierarchical Bayesian Models for Aggregating Retrieved Memories across Individuals](https://reader036.vdocument.in/reader036/viewer/2022062321/56813454550346895d9b3a85/html5/thumbnails/20.jpg)
Bayesian Thurstonian Approach
20
Each person’s mental representation is based on (latent) samples of these distributions
B C
A B C
Person 1
A
![Page 21: Hierarchical Bayesian Models for Aggregating Retrieved Memories across Individuals](https://reader036.vdocument.in/reader036/viewer/2022062321/56813454550346895d9b3a85/html5/thumbnails/21.jpg)
Bayesian Thurstonian Approach
21
B C
A B C
The observed ordering is based on the ordering of the samples
A < B < C
Observed Ordering:
Person 1
A
![Page 22: Hierarchical Bayesian Models for Aggregating Retrieved Memories across Individuals](https://reader036.vdocument.in/reader036/viewer/2022062321/56813454550346895d9b3a85/html5/thumbnails/22.jpg)
Bayesian Thurstonian Approach
22
People draw from distributions with common mean but different variances
Person 1
B C
A B CA < B < C
Observed Ordering:
Person 2
A B C
BC
Observed Ordering:
A < C < BA
A
![Page 23: Hierarchical Bayesian Models for Aggregating Retrieved Memories across Individuals](https://reader036.vdocument.in/reader036/viewer/2022062321/56813454550346895d9b3a85/html5/thumbnails/23.jpg)
Graphical Model Notation
23
jx
1x
2x 3xj=1..3
shaded = observednot shaded = latent
![Page 24: Hierarchical Bayesian Models for Aggregating Retrieved Memories across Individuals](https://reader036.vdocument.in/reader036/viewer/2022062321/56813454550346895d9b3a85/html5/thumbnails/24.jpg)
Graphical Model of Bayesian Thurstonian Model
24
j individuals
jx
jy
μ
j
| , ~ N ,ij j jx
( )j jranky x
~ Gamma ,1 /j
Latent ground truth
Individual ability
Mental representation
Observed ordering
![Page 25: Hierarchical Bayesian Models for Aggregating Retrieved Memories across Individuals](https://reader036.vdocument.in/reader036/viewer/2022062321/56813454550346895d9b3a85/html5/thumbnails/25.jpg)
Inference
Need the posterior distribution
Markov Chain Monte Carlo Gibbs sampling on Metropolis-hastings on and
Draw 400 samples group ordering based on average of across samples
25
jxμ j
μ
, , | jp μ σ x y
![Page 26: Hierarchical Bayesian Models for Aggregating Retrieved Memories across Individuals](https://reader036.vdocument.in/reader036/viewer/2022062321/56813454550346895d9b3a85/html5/thumbnails/26.jpg)
1 10 200
50
100
150
200
250
300
350
Individuals
Thurstonian ModelIndividuals
Wisdom of Crowds effect
26
model’s ordering is as good as best individual
![Page 27: Hierarchical Bayesian Models for Aggregating Retrieved Memories across Individuals](https://reader036.vdocument.in/reader036/viewer/2022062321/56813454550346895d9b3a85/html5/thumbnails/27.jpg)
Inferred Distributions for 44 US Presidents
27
George Washington (1)John Adams (2)
Thomas Jefferson (3)James Madison (4)James Monroe (6)
John Quincy Adams (5)Andrew Jackson (7)
Martin Van Buren (8)William Henry Harrison (21)
John Tyler (10)James Knox Polk (18)
Zachary Taylor (16)Millard Fillmore (11)Franklin Pierce (19)
James Buchanan (13)Abraham Lincoln (9)
Andrew Johnson (12)Ulysses S. Grant (17)
Rutherford B. Hayes (20)James Garfield (22)Chester Arthur (15)
Grover Cleveland 1 (23)Benjamin Harrison (14)
Grover Cleveland 2 (25)William McKinley (24)
Theodore Roosevelt (29)William Howard Taft (27)
Woodrow Wilson (30)Warren Harding (26)Calvin Coolidge (28)Herbert Hoover (31)
Franklin D. Roosevelt (32)Harry S. Truman (33)
Dwight Eisenhower (34)John F. Kennedy (37)
Lyndon B. Johnson (36)Richard Nixon (39)
Gerald Ford (35)James Carter (38)
Ronald Reagan (40)George H.W. Bush (41)
William Clinton (42)George W. Bush (43)
Barack Obama (44)
median and minimumsigma
![Page 28: Hierarchical Bayesian Models for Aggregating Retrieved Memories across Individuals](https://reader036.vdocument.in/reader036/viewer/2022062321/56813454550346895d9b3a85/html5/thumbnails/28.jpg)
Model is calibrated
28
0 0.1 0.2 0.3 0.450
100
150
200
250
300
R=0.941
Individuals with large sigma are far from the truth
![Page 29: Hierarchical Bayesian Models for Aggregating Retrieved Memories across Individuals](https://reader036.vdocument.in/reader036/viewer/2022062321/56813454550346895d9b3a85/html5/thumbnails/29.jpg)
Alternative Models
Many heuristic methods from voting theory E.g., Borda count method
Suppose we have 10 items assign a count of 10 to first item, 9 for second item, etc add counts over individuals order items by the Borda count
i.e., rank by average rank across people
29
![Page 30: Hierarchical Bayesian Models for Aggregating Retrieved Memories across Individuals](https://reader036.vdocument.in/reader036/viewer/2022062321/56813454550346895d9b3a85/html5/thumbnails/30.jpg)
1 10 200
50
100
150
200
250
300
350
Individuals
Thurstonian ModelBorda countIndividuals
Model Comparison
30
![Page 31: Hierarchical Bayesian Models for Aggregating Retrieved Memories across Individuals](https://reader036.vdocument.in/reader036/viewer/2022062321/56813454550346895d9b3a85/html5/thumbnails/31.jpg)
Experiment 2
78 participants 17 problems each with 10 items
Chronological Events Physical Measures Purely ordinal problems, e.g.
Ten Amendments Ten commandments
31
![Page 32: Hierarchical Bayesian Models for Aggregating Retrieved Memories across Individuals](https://reader036.vdocument.in/reader036/viewer/2022062321/56813454550346895d9b3a85/html5/thumbnails/32.jpg)
Ordering states west-east
32
Oregon (1)
Utah (2)
Nebraska (3)
Iowa (4)
Alabama (6)
Ohio (5)
Virginia (7)
Delaware (8)
Connecticut (9)
Maine (10)
0 1 2 3
0
5
10
15
20
25
30
35
40
45
R=0.961
![Page 33: Hierarchical Bayesian Models for Aggregating Retrieved Memories across Individuals](https://reader036.vdocument.in/reader036/viewer/2022062321/56813454550346895d9b3a85/html5/thumbnails/33.jpg)
Ordering Ten Amendments
33
Freedom of speech & religion (1)
Right to bear arms (2)
No quartering of soldiers (4)
No unreasonable searches (3)
Due process (5)
Trial by Jury (6)
Civil Trial by Jury (7)
No cruel punishment (8)
Right to non-specified rights (10)
Power for the States & People (9)
ten ammendments
0 0.5 1 1.5 2 2.50
5
10
15
20
25
30
35
R=0.889
![Page 34: Hierarchical Bayesian Models for Aggregating Retrieved Memories across Individuals](https://reader036.vdocument.in/reader036/viewer/2022062321/56813454550346895d9b3a85/html5/thumbnails/34.jpg)
Ordering Ten Commandments
34
Worship any other God (1)
Make a graven image (7)
Take the Lord's name in vain (2)
Break the Sabbath (3)
Dishonor your parents (4)
Murder (6)
Commit adultery (8)
Steal (5)
Bear false witness (9)
Covet (10)
0 0.5 1 1.5 20
5
10
15
20
25
30
35
R=0.722
![Page 35: Hierarchical Bayesian Models for Aggregating Retrieved Memories across Individuals](https://reader036.vdocument.in/reader036/viewer/2022062321/56813454550346895d9b3a85/html5/thumbnails/35.jpg)
Average results over 17 Problems
35
1 10 20 30 40 50 60 70 800
5
10
15
20
25
Individuals
Me
an
Thurstonian ModelBorda countModeIndividuals
![Page 36: Hierarchical Bayesian Models for Aggregating Retrieved Memories across Individuals](https://reader036.vdocument.in/reader036/viewer/2022062321/56813454550346895d9b3a85/html5/thumbnails/36.jpg)
Effect of Group Composition
How many individuals do we need to average over?
36
![Page 37: Hierarchical Bayesian Models for Aggregating Retrieved Memories across Individuals](https://reader036.vdocument.in/reader036/viewer/2022062321/56813454550346895d9b3a85/html5/thumbnails/37.jpg)
Effect of Group Size: random groups
37
0 10 20 30 40 50 60 70 807
8
9
10
11
12
13
14
Group Size
T=0T=2
T=12
![Page 38: Hierarchical Bayesian Models for Aggregating Retrieved Memories across Individuals](https://reader036.vdocument.in/reader036/viewer/2022062321/56813454550346895d9b3a85/html5/thumbnails/38.jpg)
Experts vs. Crowds
Can we find experts in the crowd? Can we form small groups of experts?
Approach Form a group for some particular task Select individuals with the smallest sigma (“experts”) based on
previous tasks Vary the number of previous tasks
38
![Page 39: Hierarchical Bayesian Models for Aggregating Retrieved Memories across Individuals](https://reader036.vdocument.in/reader036/viewer/2022062321/56813454550346895d9b3a85/html5/thumbnails/39.jpg)
Group Composition based on prior performance
39
0 10 20 30 40 50 60 70 807
8
9
10
11
12
13
14
Group Size
T=0T=2
T=12
T = 0
# previous tasks
T = 2T = 8
Group size (best individuals first)
![Page 40: Hierarchical Bayesian Models for Aggregating Retrieved Memories across Individuals](https://reader036.vdocument.in/reader036/viewer/2022062321/56813454550346895d9b3a85/html5/thumbnails/40.jpg)
Methods for Selecting Experts
40
Endogenous: no feedback
required
Exogenous: selecting people based on
actual performance
0 10 20 30 407
8
9
10
11
12
13
14
0 20 407
8
9
10
11
12
13
14
![Page 41: Hierarchical Bayesian Models for Aggregating Retrieved Memories across Individuals](https://reader036.vdocument.in/reader036/viewer/2022062321/56813454550346895d9b3a85/html5/thumbnails/41.jpg)
Model incorporating overall person ability
41
j individuals
jmx
jmy
mμ
jm
| , ~ N ,ijm m jm m jmx
( )jm jmranky x
~ Gamma ,1 /jm j j
Overall ability
Task specific ability
m tasks
j ~ Gamma ,1 /j j individuals
![Page 42: Hierarchical Bayesian Models for Aggregating Retrieved Memories across Individuals](https://reader036.vdocument.in/reader036/viewer/2022062321/56813454550346895d9b3a85/html5/thumbnails/42.jpg)
1 10 20 30 40 50 60 70 800
5
10
15
20
25
Individuals
Mea
n
Thurstonian Model v1Thurstonian Model v2Borda countModeIndividuals
Average results over 17 Problems
42
Me
an
new model
![Page 43: Hierarchical Bayesian Models for Aggregating Retrieved Memories across Individuals](https://reader036.vdocument.in/reader036/viewer/2022062321/56813454550346895d9b3a85/html5/thumbnails/43.jpg)
Part IIOrdering Problems in Episodic Memory
43
![Page 44: Hierarchical Bayesian Models for Aggregating Retrieved Memories across Individuals](https://reader036.vdocument.in/reader036/viewer/2022062321/56813454550346895d9b3a85/html5/thumbnails/44.jpg)
Another ordering problem:
44
http://www.youtube.com/watch?v=29VGZtnCD30&feature=related
A
B
C
D
time
![Page 45: Hierarchical Bayesian Models for Aggregating Retrieved Memories across Individuals](https://reader036.vdocument.in/reader036/viewer/2022062321/56813454550346895d9b3a85/html5/thumbnails/45.jpg)
Experiment 3
26 participants
6 videos 3 videos with stereotyped event sequences (e.g. wedding) 3 videos “unpredictable” videos (e.g., example video) extracted 10 stills for testing
Method study video followed by immediate ordering test of 10 items
45
![Page 46: Hierarchical Bayesian Models for Aggregating Retrieved Memories across Individuals](https://reader036.vdocument.in/reader036/viewer/2022062321/56813454550346895d9b3a85/html5/thumbnails/46.jpg)
Bayesian Thurstonian Model
46
event1 (1)
event2 (2)
event3 (3)
event4 (4)
event5 (7)
event6 (6)
event7 (5)
event8 (8)
event9 (9)
event10 (10)
yogurt commercial
0 0.5 1 1.5 2
0
5
10
15
20
R=0.890
= 3
![Page 47: Hierarchical Bayesian Models for Aggregating Retrieved Memories across Individuals](https://reader036.vdocument.in/reader036/viewer/2022062321/56813454550346895d9b3a85/html5/thumbnails/47.jpg)
Two other examples
47
event1 (1)
event2 (2)
event3 (3)
event4 (4)
event5 (6)
event6 (5)
event7 (7)
event8 (8)
event9 (9)
event10 (10)
clay animation
= 1 event1 (1)
event2 (2)
event3 (3)
event4 (4)
event5 (5)
event6 (6)
event7 (7)
event8 (8)
event9 (9)
event10 (10)
wedding
= 0
![Page 48: Hierarchical Bayesian Models for Aggregating Retrieved Memories across Individuals](https://reader036.vdocument.in/reader036/viewer/2022062321/56813454550346895d9b3a85/html5/thumbnails/48.jpg)
Overall Results
48
1 10 20 300
5
10
15
Individuals
Thurstonian ModelBorda countModeIndividuals
Me
an
![Page 49: Hierarchical Bayesian Models for Aggregating Retrieved Memories across Individuals](https://reader036.vdocument.in/reader036/viewer/2022062321/56813454550346895d9b3a85/html5/thumbnails/49.jpg)
Part IIIMatching Problems
49
![Page 50: Hierarchical Bayesian Models for Aggregating Retrieved Memories across Individuals](https://reader036.vdocument.in/reader036/viewer/2022062321/56813454550346895d9b3a85/html5/thumbnails/50.jpg)
Example Matching Problem (one-to-one)
50
Dutch
Danish
Yiddish
Thai
Vietnamese
Chinese
Georgian
Russian
Japanese
A
B
C
D
E
F
G
H
I
godt nytår
gelukkig nieuwjaar
a gut yohr
С Новым Годом
สวั�สดี�ปี�ใหม่�
Chúc Mừng Nǎm Mới
გილოცავთ ახალწელს
![Page 51: Hierarchical Bayesian Models for Aggregating Retrieved Memories across Individuals](https://reader036.vdocument.in/reader036/viewer/2022062321/56813454550346895d9b3a85/html5/thumbnails/51.jpg)
Experiment
17 Participants
8 matching problems, e.g. car logo’s and brand names first and last names philosophers flags and countries greek symbols and letter names
Number of items varied between 10 and 24 with 24 items, we have 24! possibilities
51
![Page 52: Hierarchical Bayesian Models for Aggregating Retrieved Memories across Individuals](https://reader036.vdocument.in/reader036/viewer/2022062321/56813454550346895d9b3a85/html5/thumbnails/52.jpg)
Overall Results
52
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 170
0.2
0.4
0.6
0.8
1
Individuals
Mea
n A
ccur
acy
![Page 53: Hierarchical Bayesian Models for Aggregating Retrieved Memories across Individuals](https://reader036.vdocument.in/reader036/viewer/2022062321/56813454550346895d9b3a85/html5/thumbnails/53.jpg)
Heuristic Aggregation Approach
Combinatorial optimization problem maximizes agreement in assigning N items to N responses
Hungarian algorithm construct a count matrix M Mij = number of people that paired item i with response j find row and column permutations to maximize diagonal sum O( n3 )
53
![Page 54: Hierarchical Bayesian Models for Aggregating Retrieved Memories across Individuals](https://reader036.vdocument.in/reader036/viewer/2022062321/56813454550346895d9b3a85/html5/thumbnails/54.jpg)
Hungarian Algorithm Example
54= correct
Dutch
Danish
Fren
ch
Japa
nese
Span
ish
Arabic
Chinese
German
Italia
n
Russian
Thai
Vietnam
ese
Wels
h
Georg
ian
Yiddis
h
gelukkig Nieuwjaar 7 3 0 0 0 1 0 0 0 0 0 0 2 0 2
godt nytår 2 3 0 0 0 0 0 2 0 2 0 0 1 3 2
bonne année 0 0 14 0 1 0 0 0 0 0 0 0 0 0 0
0 0 0 9 0 0 2 0 1 0 3 0 0 0 0
feliz año nuevo 0 0 0 0 14 0 0 0 0 0 1 0 0 0 0عامسعيد 0 1 0 0 0 14 0 0 0 0 0 0 0 0 0
0 0 0 2 0 0 12 0 0 0 0 1 0 0 0
ein gutes neues Jahr 3 1 0 0 0 0 0 9 0 0 0 0 1 0 1
felice anno nuovo 0 0 0 0 0 0 0 0 14 1 0 0 0 0 0
С Новым Годом 0 0 1 0 0 0 0 0 0 11 0 0 1 2 0
สวั�สดี�ปี�ใหม่ � 0 0 0 1 0 0 1 0 0 0 7 1 1 4 0Chúc Mừng Nǎm Mới 0 0 0 0 0 0 0 0 0 1 0 11 1 2 0
Blwyddyn Newydd Dda 0 4 0 1 0 0 0 0 0 0 1 0 6 1 2
გილოცავთ ახალ წელს 0 0 0 2 0 0 0 1 0 0 3 2 0 1 6
a gut yohr 3 3 0 0 0 0 0 3 0 0 0 0 2 2 2
= incorrect
![Page 55: Hierarchical Bayesian Models for Aggregating Retrieved Memories across Individuals](https://reader036.vdocument.in/reader036/viewer/2022062321/56813454550346895d9b3a85/html5/thumbnails/55.jpg)
Hungarian Algorithm Results (2)
55
Leonardo da Vinci 9 5 1 0 0 0 0 0 0 0Jan Vermeer 0 7 3 0 2 2 0 0 1 0
Rembrandt van Rijn 1 0 4 3 1 1 1 1 3 0Pablo Picasso 0 0 1 6 6 0 0 1 1 0
Vincent van Gogh 0 0 0 1 6 4 0 1 2 1Renoir 1 0 1 0 0 3 7 2 1 0Monet 0 1 1 0 0 2 5 3 0 3
Jan Van Eyck 0 2 3 2 0 0 0 5 3 0Edvard Munch 0 0 1 0 0 3 0 2 4 5
Salvador Dali 4 0 0 3 0 0 2 0 0 6
![Page 56: Hierarchical Bayesian Models for Aggregating Retrieved Memories across Individuals](https://reader036.vdocument.in/reader036/viewer/2022062321/56813454550346895d9b3a85/html5/thumbnails/56.jpg)
Bayesian Matching Model
56
Proposed process:
- match “known” items- guess between remaining ones
Individual differences:
-some items easier to know-some participants know more
Dutch
Danish
Yiddish
Russian
godt nytår
gelukkig nieuwjaar
a gut yohr
С Новым Годом
![Page 57: Hierarchical Bayesian Models for Aggregating Retrieved Memories across Individuals](https://reader036.vdocument.in/reader036/viewer/2022062321/56813454550346895d9b3a85/html5/thumbnails/57.jpg)
Graphical Model
57
i items
jx
jy
z
ja
Latent ground truth
Observed matching
Knowledge State
jsProb. of knowing
id
j individuals
logitj i js d a
~ Bernoulliij ijx s
1 1( )
1 / ! 0ij
ij ij ij
xp y z
n x
person abilityitem easiness
![Page 58: Hierarchical Bayesian Models for Aggregating Retrieved Memories across Individuals](https://reader036.vdocument.in/reader036/viewer/2022062321/56813454550346895d9b3a85/html5/thumbnails/58.jpg)
Overall Modeling Results
58
1 10 200
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
Individuals
Mea
n A
ccur
acy
Bayesian MatchingHungarian AlgorithmIndividuals
![Page 59: Hierarchical Bayesian Models for Aggregating Retrieved Memories across Individuals](https://reader036.vdocument.in/reader036/viewer/2022062321/56813454550346895d9b3a85/html5/thumbnails/59.jpg)
Calibration at level of items and people(for paintings problem)
59
0 0.5 10
0.2
0.4
0.6
0.8
1
D (inferred)
D (
act
ual)
Greek symbols
R=0.953
0 0.5 10
0.2
0.4
0.6
0.8
1
D (inferred)
D (
act
ual)
Philosophers
R=0.978
0 0.5 10
0.2
0.4
0.6
0.8
1
D (inferred)
D (
act
ual)
Flags
R=0.973
0 0.5 10
0.2
0.4
0.6
0.8
1
D (inferred)
D (
act
ual)
Paintings
R=0.916
0 0.5 10
0.2
0.4
0.6
0.8
1
D (inferred)
D (
act
ual)
US presidents
R=0.960
0 0.5 10
0.2
0.4
0.6
0.8
1
D (inferred)
D (
act
ual)
Car logos
R=0.918
0 0.5 10
0.2
0.4
0.6
0.8
1
D (inferred)
D (
act
ual)
Languages
R=0.947
0 0.5 10
0.2
0.4
0.6
0.8
1
D (inferred)
D (
act
ual)
Sport balls
R=0.963
0 0.5 10
0.2
0.4
0.6
0.8
1
A (inferred)
A (
act
ual)
Greek symbols
R=0.990
0 0.5 10
0.2
0.4
0.6
0.8
1
A (inferred)
A (
act
ual)
Philosophers
R=0.992
0 0.5 10
0.2
0.4
0.6
0.8
1
A (inferred)
A (
act
ual)
Flags
R=0.987
0 0.5 10
0.2
0.4
0.6
0.8
1
A (inferred)
A (
act
ual)
Paintings
R=0.975
0 0.5 10
0.2
0.4
0.6
0.8
1
A (inferred)
A (
act
ual)
US presidents
R=0.992
0 0.5 10
0.2
0.4
0.6
0.8
1
A (inferred)
A (
act
ual)
Car logos
R=0.992
0 0.5 10
0.2
0.4
0.6
0.8
1
A (inferred)
A (
act
ual)
Languages
R=0.968
0 0.5 10
0.2
0.4
0.6
0.8
1
A (inferred)
A (
act
ual)
Sport balls
R=0.995
ITEMS INDIVIDUALS
![Page 60: Hierarchical Bayesian Models for Aggregating Retrieved Memories across Individuals](https://reader036.vdocument.in/reader036/viewer/2022062321/56813454550346895d9b3a85/html5/thumbnails/60.jpg)
How predictive are subject provided confidence ratings?
60
0 1-2 3-4 5+0
0.2
0.4
0.6
0.8
1
0 1-2 3-4 5+0
0.2
0.4
0.6
0.8
1
# guesses estimatedby individual
Acc
urac
y
# guesses estimatedby model
(based on variable A)
r=-.42 r=-.77
![Page 61: Hierarchical Bayesian Models for Aggregating Retrieved Memories across Individuals](https://reader036.vdocument.in/reader036/viewer/2022062321/56813454550346895d9b3a85/html5/thumbnails/61.jpg)
Part IVOpen Issues
61
![Page 62: Hierarchical Bayesian Models for Aggregating Retrieved Memories across Individuals](https://reader036.vdocument.in/reader036/viewer/2022062321/56813454550346895d9b3a85/html5/thumbnails/62.jpg)
When do we get wisdom of crowds effect?
Independent errors different people knowing different things
Population response centered around ground truth
Some minimal number of individuals 10-20 individuals often sufficient
62
![Page 63: Hierarchical Bayesian Models for Aggregating Retrieved Memories across Individuals](https://reader036.vdocument.in/reader036/viewer/2022062321/56813454550346895d9b3a85/html5/thumbnails/63.jpg)
What are methods for finding experts?
1) Self-reported expertise: unreliable has led to claims of “myth of expertise”
2) Based on explicit scores by comparing to ground truth but ground truth might not be immediately available
3) Endogenously discover experts Use the crowd to discover experts Small groups of experts can be effective
63
![Page 64: Hierarchical Bayesian Models for Aggregating Retrieved Memories across Individuals](https://reader036.vdocument.in/reader036/viewer/2022062321/56813454550346895d9b3a85/html5/thumbnails/64.jpg)
What to do about systematic biases?
In some tasks, individuals systematically distort the ground truth spatial and temporal distortions memory distortions (e.g. false memory) decision-making distortions
Does this diminish the wisdom of crowds effect? maybe… but a model that predicts these systematic distortions might be
able to “undo” them
64
![Page 65: Hierarchical Bayesian Models for Aggregating Retrieved Memories across Individuals](https://reader036.vdocument.in/reader036/viewer/2022062321/56813454550346895d9b3a85/html5/thumbnails/65.jpg)
Can we build domain specific models?
Thurstonian model applied to wide variety of problems
How about domain specific models? e.g., apply serial recall models to serial recall better specify sources of noise model systematic biases
65
![Page 67: Hierarchical Bayesian Models for Aggregating Retrieved Memories across Individuals](https://reader036.vdocument.in/reader036/viewer/2022062321/56813454550346895d9b3a85/html5/thumbnails/67.jpg)
Other slides
67
![Page 68: Hierarchical Bayesian Models for Aggregating Retrieved Memories across Individuals](https://reader036.vdocument.in/reader036/viewer/2022062321/56813454550346895d9b3a85/html5/thumbnails/68.jpg)
Results separated by problem
68
Problem PC τ * C τ Rank C τ Rank C τ Rank C τ Rank C τ Rankbooks .000 10 0 6 88 0 6 88 0 5 91 0 7 82 0 12 40
city population europe .000 15 0 12 77 0 12 77 0 12 77 0 11 81 0 17 42city population us .000 14 0 9 90 0 8 91 0 7 96 0 12 67 0 16 45
city population world .000 18 0 16 73 0 16 73 0 16 73 0 15 77 0 19 44country landmass .000 9 0 5 95 0 6 85 0 5 95 0 5 95 0 7 76
country population .000 15 0 10 87 0 10 87 0 11 82 0 11 82 0 15 53hardness .000 15 0 14 64 0 13 73 0 14 64 0 11 91 0 15 46holidays .051 8 0 5 77 0 5 77 0 5 77 0 4 78 1 0 100
movies releasedate .013 6 0 1 99 0 1 99 0 2 95 0 2 95 0 2 95oscar bestmovies .013 10 0 4 90 0 3 97 0 4 90 0 3 97 0 3 97
oscar movies .000 10 0 1 100 0 1 100 0 1 100 0 2 96 0 2 96presidents .064 7 0 1 94 1 0 100 0 1 94 0 3 79 1 0 100
rivers .000 15 0 11 91 0 12 86 0 14 67 0 11 91 0 16 42states westeast .026 6 0 1 97 0 1 97 0 2 88 0 3 78 0 1 97
superbowl .000 17 0 13 86 0 12 88 0 15 71 0 10 96 0 19 40ten amendments .013 13 0 2 97 0 1 99 0 3 96 0 5 90 0 4 95
ten commandments .000 17 0 7 91 0 7 91 0 7 91 0 12 74 0 17 51AVERAGE .011 12.1 .00 6.94 88.0 .06 6.71 88.8 .00 7.29 85.1 .00 7.47 85.3 .12 9.67 68.2
Mallows Model Borda Counts ModeThurstone v1Humans Thurstone v2
![Page 69: Hierarchical Bayesian Models for Aggregating Retrieved Memories across Individuals](https://reader036.vdocument.in/reader036/viewer/2022062321/56813454550346895d9b3a85/html5/thumbnails/69.jpg)
Notes
Noise in Thurstonian models acquisition / encoding noise retrieval noise
Link to crowd within (Ed Vul) are our results due to wisdom of crowds or individuals? Probably a bit of both and we cannot tell with our experiments However, there is probably a fair amount of encoding noise that
would not benefit from repeated measurements within individuals Different individuals probably do know different things
69
![Page 70: Hierarchical Bayesian Models for Aggregating Retrieved Memories across Individuals](https://reader036.vdocument.in/reader036/viewer/2022062321/56813454550346895d9b3a85/html5/thumbnails/70.jpg)
To Do
Compare explicitly estimated number of guesses with latent confidence
Identifiability issue fix mean A?
Hierarchical model test on small numbers of subjects
Model comparisons on small sets of subjects
70
TO DO: look at kurtosis of sigma distributions
![Page 71: Hierarchical Bayesian Models for Aggregating Retrieved Memories across Individuals](https://reader036.vdocument.in/reader036/viewer/2022062321/56813454550346895d9b3a85/html5/thumbnails/71.jpg)
Modeling Group Serial Recall
Goal: infer distribution over orderings of events given verbal reports i.e., P( original order | verbal report )
Many models for serial recall, e.g. Estes Perturbation model (1972) Shiffrin & Cook (1978) SOB (2002) Simple (2007)
but many of these models do not have a likelihood function p( item 1, item 2, …, item N | memory contents )
71
![Page 72: Hierarchical Bayesian Models for Aggregating Retrieved Memories across Individuals](https://reader036.vdocument.in/reader036/viewer/2022062321/56813454550346895d9b3a85/html5/thumbnails/72.jpg)
Bayesian Algorithm: not every person has equal weight
72= correct = incorrect
DutchDan
ish
Frenc
h
Japan
ese
Spanish
Arabic
Chinese
German
Italia
n
Russian
ThaiViet
names
e
Wels
h
Georg
ian
Yiddis
h
gelukkig Nieuwjaar 7 3 0 0 0 1 0 0 0 0 0 0 2 0 2
godt nytår 2 3 0 0 0 0 0 2 0 2 0 0 1 3 2
bonne année 0 0 14 0 1 0 0 0 0 0 0 0 0 0 0
0 0 0 9 0 0 2 0 1 0 3 0 0 0 0
feliz año nuevo 0 0 0 0 14 0 0 0 0 0 1 0 0 0 0عامسعيد 0 1 0 0 0 14 0 0 0 0 0 0 0 0 0
0 0 0 2 0 0 12 0 0 0 0 1 0 0 0
ein gutes neues Jahr 3 1 0 0 0 0 0 9 0 0 0 0 1 0 1
felice anno nuovo 0 0 0 0 0 0 0 0 14 1 0 0 0 0 0
С Новым Годом 0 0 1 0 0 0 0 0 0 11 0 0 1 2 0
สวั�สดี�ปี�ใหม่ � 0 0 0 1 0 0 1 0 0 0 7 1 1 4 0Chúc Mừng Nǎm Mới 0 0 0 0 0 0 0 0 0 1 0 11 1 2 0
Blwyddyn Newydd Dda 0 4 0 1 0 0 0 0 0 0 1 0 6 1 2
გილოცავთ ახალ წელს 0 0 0 2 0 0 0 1 0 0 3 2 0 1 6
a gut yohr 3 3 0 0 0 0 0 3 0 0 0 0 2 2 2
![Page 73: Hierarchical Bayesian Models for Aggregating Retrieved Memories across Individuals](https://reader036.vdocument.in/reader036/viewer/2022062321/56813454550346895d9b3a85/html5/thumbnails/73.jpg)
Summary of Findings
Extended wisdom of crowds to combinatorial problems approximate inference (MCMC) to infer probability distributions
over permutations
Bayesian methods that are calibrated we can tell who is likely to be accurate without having ground
truth available
73
![Page 74: Hierarchical Bayesian Models for Aggregating Retrieved Memories across Individuals](https://reader036.vdocument.in/reader036/viewer/2022062321/56813454550346895d9b3a85/html5/thumbnails/74.jpg)
Graphical Model
74
i items
jx
jy
z
ja
Latent ground truth
Observed matching
Knowledge State
jsProb. of knowing
id
j individuals
logitj i js d a
~ Bernoulliij ijx s
1 1( )
1 / ! 0ij
ij ij ij
xp y z
n x
item and person parameters
![Page 75: Hierarchical Bayesian Models for Aggregating Retrieved Memories across Individuals](https://reader036.vdocument.in/reader036/viewer/2022062321/56813454550346895d9b3a85/html5/thumbnails/75.jpg)
When do we get Wisdom of Crowds effect?
Analyze model performance in a variety of tasks
75
![Page 76: Hierarchical Bayesian Models for Aggregating Retrieved Memories across Individuals](https://reader036.vdocument.in/reader036/viewer/2022062321/56813454550346895d9b3a85/html5/thumbnails/76.jpg)
MDS solution of pairwise tau distances
76-15 -10 -5 0 5 10 15 20 25 30 35-20
-15
-10
-5
0
5
10
15
7
26
3
16
7 9
61
22
2
13
12
7
11
14
9
5
7
11
8
3
24
3
7
10
10
4
03
6
9
6
26
5
18
44 3
14
6
2
5
3
5
1
4210
11
4
3
42
0
8
21
7
3
5
1
1
8
1
33
14
3
20
6
8
16
7
22
23
2 3710
states westeast
IndividualsTruthThurstonian Model
distance to truth
![Page 77: Hierarchical Bayesian Models for Aggregating Retrieved Memories across Individuals](https://reader036.vdocument.in/reader036/viewer/2022062321/56813454550346895d9b3a85/html5/thumbnails/77.jpg)
MDS solution of pairwise tau distances
77-20 -15 -10 -5 0 5 10 15 20 25
-20
-15
-10
-5
0
5
10
15
20
14
23
25
24
1824
13
14
10
5
9
20
8
20
15
18
12
33
25
29
171
14
20
27176
13
11
15
3
17
17
17
24
7
26
9
13
17
27
13
15
11
15
15
23
2811
26
16
4
27
9
23
24
11
17
19
15
22
2
15
14
12
21
11
26
11
18
35
22
10
20
24
25
1
19
7
0
ten commandments
IndividualsTruthThurstonian Model
![Page 78: Hierarchical Bayesian Models for Aggregating Retrieved Memories across Individuals](https://reader036.vdocument.in/reader036/viewer/2022062321/56813454550346895d9b3a85/html5/thumbnails/78.jpg)
Modeling Performance Across Task
Current model is applied independently across tasks
Extend hierarchical model with random effects approach to tasks Each person has a an overall ability (Pearson’s “g” ) Ability in a specific task is varies around overall ability
78