1

Differences Between Observed and Latent Confidence in Rank Ordering

Brent J. MillerMark Steyvers

University of California, Irvine

2

Getting Expert Information

3

Getting Expert Information

Delphi Method Iterative Collaboration

4

George Washington

John Adams

James Monroe

James Madison

Thomas Jefferson

John Adams

James Madison

James Monroe

Independent Rank-Ordering Tasks

Order these U.S. Presidents, by term of office:

George Washington

1st Participant 2nd Participant 3rd Participant

John Adams

George Washington

Thomas Jefferson

James Madison

James Monroe

Thomas Jefferson

5

James Madison

Thomas Jefferson

Thomas Jefferson

John Adams

James Madison

James Monroe

James Monroe

Iterative Rank-Ordering Tasks

Order these U.S. Presidents, by term of office:

George Washington

1st ParticipantGeorge Washington

John Adams

George Washington

James Madison

Thomas Jefferson

James Monroe

2nd Participant 3rd Participant

John Adams

6

• 100 Subjects• 17 Ranking Questions

– Chronological Events (Holidays, U.S. Presidents)– Physical Measures (Landmass, City Pop., State Locations)– Ordinal Information (Ten Amendments, Ten Commandments)

• 2 Conditions

Rank-Ordering Experiment

7

Rank Ordering – Independent & Iteratives

5 10 15 20 25 30 35 40 45 504

6

8

10

12

14

16

18

Group Size

Ave

rage

Tau

17 Questions via Mechanical Turk

5 10 15 20 25 30 35 40 45 506

7

8

9

10

11

12

13

14

Group Size

Ave

rage

Tau

17 Questions from University Students

x 4

88

Participant1 Y1: A < B < CX11 X12 X13

A B C

Thurstonian Model for Rank Ordering

σ1

99

Participant1 Y1: A < B < CX11 X12 X13

Y2 : A < C < BParticipant2 X22 X23X21

A B C

σ2

Thurstonian Model for Rank Ordering

10

Iterative Collaboration

George Washington

John Adams

Thomas Jefferson

James Monroe

Andrew Jackson

Theodore Roosevelt

Woodrow Wilson

Franklin D. Roosevelt

Harry S. Truman

Dwight D. Eisenhower

76 77 78 79 80 81 correct

1

2

3

4

5

6

7

8

9

10

Participants

Pos

ition

U.S. Presidents

33 34 35 36 37 38 39

Participants

Posi

tion

11

Participant1 Y1: A < B < CX11 X12 X13

A B C

Thurstonian Model for Item Representation

σ11 σ12 σ13

12

• 50 Subjects• 8 Ranking Questions

– Chronological Events (Holidays, U.S. Presidents)– Physical Measures (Landmass, City Pop., State Locations)– Ordinal Information (Ten Amendments, Ten Commandments)

Rank-Ordering Experiment

7 5 6 4 1 8 3 2 7 5 6 4 1 8 3 2

Independent (8) Iterative (8)

13

Independent vs. Iterative Aggregate Performance

5 10 15 20 25 30 35 40 457

8

9

10

11

12

13

14

15

16

Group Size

Ave

rage

Tau

Independent

Iterative

14

Independent Subject Generative Model

μi

xij σij

yj

i Items

j Subjects

µi ≡ Ground truthσ2

ij ~ InvGamma(.2,1)xi ~ Norm(µi,σ2

ij)Yj ~ Rank(xj)

15

Iterative Subject Generative Model

μij

xi,j-1

σi,j-1

yj-1

i Items

j Subjects

xij

σij

cutyj

µi ≡ Ground truthσ2

ij ~ InvGamma(.2,1)Xij-1 ~ Norm(µi,σ2

ij-1)Yj-1 ~ Rank(xj-1)

16

Simulated Aggregate Performance

1 10 20 30 40 50 652

4

6

8

10

12

14

16

Group Size

Ave

rag

et

Independent Subjects

Iterative Subjects

17

Reduction in Iterative Error by Item Confidence

−0.4 −0.05 0.3 0.65 1

1

2

3

4

5

DError

Sub

ject

Con

fide

nce

Confidence for Items with Error > 4 Tau

No Change

18

Conclusions

• Item-level knowledge may allow partial expertise between participants, allowing for improvement in group aggregation.

• Subject item-level confidence appears to support the validity of our proposed model.

• We should be able to use subjects’ confidence as an estimate of underlying item-level expertise.

19

Danke!

Differences Between Observed and Latent Confidence in Rank Ordering