learning from the wisdom of crowds - microsoft.com from the wisdom of crowds by minimax entropy...
TRANSCRIPT
![Page 1: Learning from the Wisdom of Crowds - microsoft.com from the Wisdom of Crowds by Minimax Entropy Denny Zhou, John Platt, Sumit Basu and Yi Mao Microsoft Research, Redmond, WA](https://reader034.vdocument.in/reader034/viewer/2022051801/5adfbbdc7f8b9a8f298d4c4c/html5/thumbnails/1.jpg)
Learning from the Wisdom of
Crowds by Minimax Entropy
Denny Zhou, John Platt, Sumit Basu and Yi Mao Microsoft Research, Redmond, WA
![Page 2: Learning from the Wisdom of Crowds - microsoft.com from the Wisdom of Crowds by Minimax Entropy Denny Zhou, John Platt, Sumit Basu and Yi Mao Microsoft Research, Redmond, WA](https://reader034.vdocument.in/reader034/viewer/2022051801/5adfbbdc7f8b9a8f298d4c4c/html5/thumbnails/2.jpg)
Outline
1. Introduction
2. Minimax entropy principle
3. Future work and conclusion
Microsoft Research Redmond 2
![Page 3: Learning from the Wisdom of Crowds - microsoft.com from the Wisdom of Crowds by Minimax Entropy Denny Zhou, John Platt, Sumit Basu and Yi Mao Microsoft Research, Redmond, WA](https://reader034.vdocument.in/reader034/viewer/2022051801/5adfbbdc7f8b9a8f298d4c4c/html5/thumbnails/3.jpg)
1. Introduction
Microsoft Research Redmond 3
![Page 4: Learning from the Wisdom of Crowds - microsoft.com from the Wisdom of Crowds by Minimax Entropy Denny Zhou, John Platt, Sumit Basu and Yi Mao Microsoft Research, Redmond, WA](https://reader034.vdocument.in/reader034/viewer/2022051801/5adfbbdc7f8b9a8f298d4c4c/html5/thumbnails/4.jpg)
Machine Learning Meets Crowdsourcing
• To Improve a machine learning model:
– Add more training examples
– Create more meaningful features
– Invent more powerful learning algorithms
More and more efforts, less and less gain
![Page 5: Learning from the Wisdom of Crowds - microsoft.com from the Wisdom of Crowds by Minimax Entropy Denny Zhou, John Platt, Sumit Basu and Yi Mao Microsoft Research, Redmond, WA](https://reader034.vdocument.in/reader034/viewer/2022051801/5adfbbdc7f8b9a8f298d4c4c/html5/thumbnails/5.jpg)
Machine Learning Meets Crowdsourcing
• To Improve a machine learning model:
– Adding more training examples
– Creating more meaningful features
– Inventing more powerful learning algorithms
More and more efforts, less and less gain
![Page 6: Learning from the Wisdom of Crowds - microsoft.com from the Wisdom of Crowds by Minimax Entropy Denny Zhou, John Platt, Sumit Basu and Yi Mao Microsoft Research, Redmond, WA](https://reader034.vdocument.in/reader034/viewer/2022051801/5adfbbdc7f8b9a8f298d4c4c/html5/thumbnails/6.jpg)
Crowdsourcing for Labeling
6 Microsoft Research Redmond
![Page 7: Learning from the Wisdom of Crowds - microsoft.com from the Wisdom of Crowds by Minimax Entropy Denny Zhou, John Platt, Sumit Basu and Yi Mao Microsoft Research, Redmond, WA](https://reader034.vdocument.in/reader034/viewer/2022051801/5adfbbdc7f8b9a8f298d4c4c/html5/thumbnails/7.jpg)
Low Cost, but also Low Quality
(Stanford dogs dataset)
Norfolk Terrier Norwich Terrier
Irish Wolfhound Scottish Deerhound
7 Microsoft Research Redmond
![Page 8: Learning from the Wisdom of Crowds - microsoft.com from the Wisdom of Crowds by Minimax Entropy Denny Zhou, John Platt, Sumit Basu and Yi Mao Microsoft Research, Redmond, WA](https://reader034.vdocument.in/reader034/viewer/2022051801/5adfbbdc7f8b9a8f298d4c4c/html5/thumbnails/8.jpg)
Workers: 𝑖 = 1, 2, ⋯ , 𝑚 Items: 𝑗 = 1, 2, ⋯ , 𝑛 Categories: 𝑘 = 1, 2, ⋯ , 𝑐
Problem Setting and Notations
8 Microsoft Research Redmond
Response matrix 𝑍𝑚×𝑛×𝑐
𝑧𝑖𝑗𝑘 = 1, if worker 𝑖 labels item 𝑗 as category 𝑘
𝑧𝑖𝑗𝑘 = 0, if worker 𝑖 labels item 𝑗 as other (not 𝑘)
𝑧𝑖𝑗𝑘 = 𝑢𝑛𝑘𝑛𝑜𝑤𝑛, if worker 𝑖 does not label item 𝑗
Goal: Estimate the ground truth {𝑦𝑗𝑘}
![Page 9: Learning from the Wisdom of Crowds - microsoft.com from the Wisdom of Crowds by Minimax Entropy Denny Zhou, John Platt, Sumit Basu and Yi Mao Microsoft Research, Redmond, WA](https://reader034.vdocument.in/reader034/viewer/2022051801/5adfbbdc7f8b9a8f298d4c4c/html5/thumbnails/9.jpg)
Toy Example: Binary Labeling
Item 1 Item 2 Item 3 Item 4 Item 5 Item 6
1 2 1 1 1 2
2 2 1 2 1 1
1 1 2 1 1 2
1 1 1 1 1 2
1 1 1 2 2 2
Worker 1
Worker 2
Worker 3
Worker 4
Worker 5
Problem: What are the true labels of the items?
9 Microsoft Research Redmond
![Page 10: Learning from the Wisdom of Crowds - microsoft.com from the Wisdom of Crowds by Minimax Entropy Denny Zhou, John Platt, Sumit Basu and Yi Mao Microsoft Research, Redmond, WA](https://reader034.vdocument.in/reader034/viewer/2022051801/5adfbbdc7f8b9a8f298d4c4c/html5/thumbnails/10.jpg)
A Simple Method: Majority Voting
Item 1 Item 2 Item 3 Item 4 Item 5 Item 6
1 2 1 1 1 2
2 2 1 2 1 1
1 1 2 1 1 2
1 1 1 1 1 2
1 1 1 2 2 2
Worker 1
Worker 2
Worker 3
Worker 4
Worker 5
By majority voting, the true label of item 4 should be class 1: # {workers labeling it as class 1} = 3 # {workers labeling it as class 2} = 2
Improve: More skillful workers should have more weight
10 Microsoft Research Redmond
![Page 11: Learning from the Wisdom of Crowds - microsoft.com from the Wisdom of Crowds by Minimax Entropy Denny Zhou, John Platt, Sumit Basu and Yi Mao Microsoft Research, Redmond, WA](https://reader034.vdocument.in/reader034/viewer/2022051801/5adfbbdc7f8b9a8f298d4c4c/html5/thumbnails/11.jpg)
Dawid & Skene’s Method
• Maximum Likelihood Estimation (MLE): jointly estimate confusion matrices and ground truth
• Implementation: EM algorithm
• Assume that each worker is associated with a 𝑐 × 𝑐 confusion matrix
{𝑝𝑘𝑙𝑖
= Prob[𝑧𝑖𝑗 = 𝑙|𝑦𝑗 = 𝑘, 𝑖]}
• For any labeling task, the label by a worker is generated according to her confusion matrix
11 Microsoft Research Redmond
![Page 12: Learning from the Wisdom of Crowds - microsoft.com from the Wisdom of Crowds by Minimax Entropy Denny Zhou, John Platt, Sumit Basu and Yi Mao Microsoft Research, Redmond, WA](https://reader034.vdocument.in/reader034/viewer/2022051801/5adfbbdc7f8b9a8f298d4c4c/html5/thumbnails/12.jpg)
Probabilistic Confusion Matrices
Item 1 Item 2 Item 3 Item 4 Item 5 Item 6
1 2 1 1 1 2
2 2 1 2 1 1
1 1 2 1 1 2
1 1 1 1 1 2
1 1 1 2 2 2
Worker 1
Worker 2
Worker 3
Worker 4
Worker 5
Assume that the true labels are: Class 1 = {item 1, item 2, item 3} Class 2 = {item 4, item 5, item 6}
Class 1 Class 2
Class 1 1 0
Class 2 2/3 1/3
12 Microsoft Research Redmond
![Page 13: Learning from the Wisdom of Crowds - microsoft.com from the Wisdom of Crowds by Minimax Entropy Denny Zhou, John Platt, Sumit Basu and Yi Mao Microsoft Research, Redmond, WA](https://reader034.vdocument.in/reader034/viewer/2022051801/5adfbbdc7f8b9a8f298d4c4c/html5/thumbnails/13.jpg)
EM in Dawid & Skene’s Method
• Initialize the ground truth by majority vote
• Iterate the following procedure till converge:
o Estimate the worker confusion by using the estimated ground truth
o Estimate the ground truth by using the estimated worker confusion
Microsoft Research Redmond 13
![Page 14: Learning from the Wisdom of Crowds - microsoft.com from the Wisdom of Crowds by Minimax Entropy Denny Zhou, John Platt, Sumit Basu and Yi Mao Microsoft Research, Redmond, WA](https://reader034.vdocument.in/reader034/viewer/2022051801/5adfbbdc7f8b9a8f298d4c4c/html5/thumbnails/14.jpg)
Simplified Dawid & Skene’s Method
Each worker 𝑖 is associated with a single number 𝑝𝑖 ∈ 0,1 such that
Prob 𝑧𝑖𝑗 = 𝑦𝑗|𝑖 = 𝑝𝑖
Prob 𝑧𝑖𝑗 ≠ 𝑦𝑗|𝑖 = 1 − 𝑝𝑖
14 Microsoft Research Redmond
![Page 15: Learning from the Wisdom of Crowds - microsoft.com from the Wisdom of Crowds by Minimax Entropy Denny Zhou, John Platt, Sumit Basu and Yi Mao Microsoft Research, Redmond, WA](https://reader034.vdocument.in/reader034/viewer/2022051801/5adfbbdc7f8b9a8f298d4c4c/html5/thumbnails/15.jpg)
Simplified Dawid & Skene’s Method
Each worker 𝑖 is associated with a single number 𝑝𝑖 ∈ 0,1 such that
Prob 𝑧𝑖𝑗 = 𝑦𝑗|𝑖 = 𝑝𝑖
Prob 𝑧𝑖𝑗 ≠ 𝑦𝑗|𝑖 = 1 − 𝑝𝑖
15 Microsoft Research Redmond
worker coin
![Page 16: Learning from the Wisdom of Crowds - microsoft.com from the Wisdom of Crowds by Minimax Entropy Denny Zhou, John Platt, Sumit Basu and Yi Mao Microsoft Research, Redmond, WA](https://reader034.vdocument.in/reader034/viewer/2022051801/5adfbbdc7f8b9a8f298d4c4c/html5/thumbnails/16.jpg)
2. Minimax Entropy Principle
Microsoft Research Redmond 16
![Page 17: Learning from the Wisdom of Crowds - microsoft.com from the Wisdom of Crowds by Minimax Entropy Denny Zhou, John Platt, Sumit Basu and Yi Mao Microsoft Research, Redmond, WA](https://reader034.vdocument.in/reader034/viewer/2022051801/5adfbbdc7f8b9a8f298d4c4c/html5/thumbnails/17.jpg)
Our Basic Assumption
Observed labels
unobserved distributions
![Page 18: Learning from the Wisdom of Crowds - microsoft.com from the Wisdom of Crowds by Minimax Entropy Denny Zhou, John Platt, Sumit Basu and Yi Mao Microsoft Research, Redmond, WA](https://reader034.vdocument.in/reader034/viewer/2022051801/5adfbbdc7f8b9a8f298d4c4c/html5/thumbnails/18.jpg)
Our Basic Assumption
Separated distribution per work-item!
![Page 19: Learning from the Wisdom of Crowds - microsoft.com from the Wisdom of Crowds by Minimax Entropy Denny Zhou, John Platt, Sumit Basu and Yi Mao Microsoft Research, Redmond, WA](https://reader034.vdocument.in/reader034/viewer/2022051801/5adfbbdc7f8b9a8f298d4c4c/html5/thumbnails/19.jpg)
Our Basic Assumption
Separated distribution per work-item!
![Page 20: Learning from the Wisdom of Crowds - microsoft.com from the Wisdom of Crowds by Minimax Entropy Denny Zhou, John Platt, Sumit Basu and Yi Mao Microsoft Research, Redmond, WA](https://reader034.vdocument.in/reader034/viewer/2022051801/5adfbbdc7f8b9a8f298d4c4c/html5/thumbnails/20.jpg)
Maximum Entropy
• To estimate a distribution, it is typical to use the maximum entropy principle
E. T. Jaynes
![Page 21: Learning from the Wisdom of Crowds - microsoft.com from the Wisdom of Crowds by Minimax Entropy Denny Zhou, John Platt, Sumit Basu and Yi Mao Microsoft Research, Redmond, WA](https://reader034.vdocument.in/reader034/viewer/2022051801/5adfbbdc7f8b9a8f298d4c4c/html5/thumbnails/21.jpg)
Column and Row Matching Constraints
![Page 22: Learning from the Wisdom of Crowds - microsoft.com from the Wisdom of Crowds by Minimax Entropy Denny Zhou, John Platt, Sumit Basu and Yi Mao Microsoft Research, Redmond, WA](https://reader034.vdocument.in/reader034/viewer/2022051801/5adfbbdc7f8b9a8f298d4c4c/html5/thumbnails/22.jpg)
Column Constraints
Item 1 Item 2 Item 3 Item 4 Item 5 Item 6
1 2 1 1 1 2
2 2 1 2 1 1
1 1 2 1 1 2
1 1 1 1 1 2
1 1 1 2 2 2
Worker 1
Worker 2
Worker 3
Worker 4
Worker 5
For each item: Count # workers labeling it as class 1 Count # workers labeling it as class 2
column matching
22 Microsoft Research Redmond
![Page 23: Learning from the Wisdom of Crowds - microsoft.com from the Wisdom of Crowds by Minimax Entropy Denny Zhou, John Platt, Sumit Basu and Yi Mao Microsoft Research, Redmond, WA](https://reader034.vdocument.in/reader034/viewer/2022051801/5adfbbdc7f8b9a8f298d4c4c/html5/thumbnails/23.jpg)
Row Constraints
Item 1 Item 2 Item 3 Item 4 Item 5 Item 6
1 2 1 1 1 2
2 2 1 2 1 1
1 1 2 1 1 2
1 1 1 1 1 2
1 1 1 2 2 2
Worker 1
Worker 1
Worker 3
Worker 4
Worker 5
For each worker: Count # misclassifications from class 1 to 2 Count # misclassifications from class 2 to 1
23 Microsoft Research Redmond
row matching
![Page 24: Learning from the Wisdom of Crowds - microsoft.com from the Wisdom of Crowds by Minimax Entropy Denny Zhou, John Platt, Sumit Basu and Yi Mao Microsoft Research, Redmond, WA](https://reader034.vdocument.in/reader034/viewer/2022051801/5adfbbdc7f8b9a8f298d4c4c/html5/thumbnails/24.jpg)
Subject to
(column constraint)
(row constraint)
Maximum Entropy
![Page 25: Learning from the Wisdom of Crowds - microsoft.com from the Wisdom of Crowds by Minimax Entropy Denny Zhou, John Platt, Sumit Basu and Yi Mao Microsoft Research, Redmond, WA](https://reader034.vdocument.in/reader034/viewer/2022051801/5adfbbdc7f8b9a8f298d4c4c/html5/thumbnails/25.jpg)
Subject to
(column constraint)
(row constraint)
To Estimate True Labels, Can We …
![Page 26: Learning from the Wisdom of Crowds - microsoft.com from the Wisdom of Crowds by Minimax Entropy Denny Zhou, John Platt, Sumit Basu and Yi Mao Microsoft Research, Redmond, WA](https://reader034.vdocument.in/reader034/viewer/2022051801/5adfbbdc7f8b9a8f298d4c4c/html5/thumbnails/26.jpg)
Subject to
(column constraint)
(row constraint)
To Estimate True Labels, Can We …
Leading to a uniform distribution for 𝑦𝑗𝑙
![Page 27: Learning from the Wisdom of Crowds - microsoft.com from the Wisdom of Crowds by Minimax Entropy Denny Zhou, John Platt, Sumit Basu and Yi Mao Microsoft Research, Redmond, WA](https://reader034.vdocument.in/reader034/viewer/2022051801/5adfbbdc7f8b9a8f298d4c4c/html5/thumbnails/27.jpg)
Subject to
(column constraint)
(row constraint)
To Estimate True Labels, Can We …
Leading to a uniform distribution for 𝑦𝑗𝑙
![Page 28: Learning from the Wisdom of Crowds - microsoft.com from the Wisdom of Crowds by Minimax Entropy Denny Zhou, John Platt, Sumit Basu and Yi Mao Microsoft Research, Redmond, WA](https://reader034.vdocument.in/reader034/viewer/2022051801/5adfbbdc7f8b9a8f298d4c4c/html5/thumbnails/28.jpg)
Subject to
(column constraint)
(row constraint)
![Page 29: Learning from the Wisdom of Crowds - microsoft.com from the Wisdom of Crowds by Minimax Entropy Denny Zhou, John Platt, Sumit Basu and Yi Mao Microsoft Research, Redmond, WA](https://reader034.vdocument.in/reader034/viewer/2022051801/5adfbbdc7f8b9a8f298d4c4c/html5/thumbnails/29.jpg)
Justification of Minimum Entropy
• Assume true measurement are available:
Microsoft Research Redmond 29
Subject to
true measurements
![Page 30: Learning from the Wisdom of Crowds - microsoft.com from the Wisdom of Crowds by Minimax Entropy Denny Zhou, John Platt, Sumit Basu and Yi Mao Microsoft Research, Redmond, WA](https://reader034.vdocument.in/reader034/viewer/2022051801/5adfbbdc7f8b9a8f298d4c4c/html5/thumbnails/30.jpg)
Justification of Minimum Entropy
• Theorem. Minimizing the KL divergence
is equivalent to minimize entropy.
Microsoft Research Redmond 30
![Page 31: Learning from the Wisdom of Crowds - microsoft.com from the Wisdom of Crowds by Minimax Entropy Denny Zhou, John Platt, Sumit Basu and Yi Mao Microsoft Research, Redmond, WA](https://reader034.vdocument.in/reader034/viewer/2022051801/5adfbbdc7f8b9a8f298d4c4c/html5/thumbnails/31.jpg)
Lagrangian Dual
Microsoft Research Redmond 31
• The Lagrangian dual can be written as
Lagrangian multipliers
![Page 32: Learning from the Wisdom of Crowds - microsoft.com from the Wisdom of Crowds by Minimax Entropy Denny Zhou, John Platt, Sumit Basu and Yi Mao Microsoft Research, Redmond, WA](https://reader034.vdocument.in/reader034/viewer/2022051801/5adfbbdc7f8b9a8f298d4c4c/html5/thumbnails/32.jpg)
Microsoft Research Redmond 32
Lagrangian Dual
• KKT conditions lead to a closed-form:
𝑍 is the normalization factor given by
![Page 33: Learning from the Wisdom of Crowds - microsoft.com from the Wisdom of Crowds by Minimax Entropy Denny Zhou, John Platt, Sumit Basu and Yi Mao Microsoft Research, Redmond, WA](https://reader034.vdocument.in/reader034/viewer/2022051801/5adfbbdc7f8b9a8f298d4c4c/html5/thumbnails/33.jpg)
Worker Expertise & Task Confusability
• Explanation of dual variables:
item confusability worker expertise
33 Microsoft Research Redmond
![Page 34: Learning from the Wisdom of Crowds - microsoft.com from the Wisdom of Crowds by Minimax Entropy Denny Zhou, John Platt, Sumit Basu and Yi Mao Microsoft Research, Redmond, WA](https://reader034.vdocument.in/reader034/viewer/2022051801/5adfbbdc7f8b9a8f298d4c4c/html5/thumbnails/34.jpg)
Measurement Objectivity: Item
• Objective item confusability. The difference of difficulty between labeling two items should be independent of the chosen workers
• Mathematical formulation. Let
Microsoft Research Redmond 34
Then the ratio 𝑐(𝑖, 𝑗, 𝑘)/𝑐 𝑖′, 𝑗, 𝑘 should be Independent of the choices of 𝑖, 𝑖′
![Page 35: Learning from the Wisdom of Crowds - microsoft.com from the Wisdom of Crowds by Minimax Entropy Denny Zhou, John Platt, Sumit Basu and Yi Mao Microsoft Research, Redmond, WA](https://reader034.vdocument.in/reader034/viewer/2022051801/5adfbbdc7f8b9a8f298d4c4c/html5/thumbnails/35.jpg)
Measurement Objectivity: Worker
• Objective worker expertise. The difference of expertise between two workers should be independent of the item being labeled
• Mathematic Formulation. Let
Microsoft Research Redmond 35
Then the ratio 𝑐(𝑖, 𝑗, 𝑘)/𝑐 𝑖, 𝑗′, 𝑘 should be Independent of the choices of 𝑗, 𝑗′
![Page 36: Learning from the Wisdom of Crowds - microsoft.com from the Wisdom of Crowds by Minimax Entropy Denny Zhou, John Platt, Sumit Basu and Yi Mao Microsoft Research, Redmond, WA](https://reader034.vdocument.in/reader034/viewer/2022051801/5adfbbdc7f8b9a8f298d4c4c/html5/thumbnails/36.jpg)
The Labeling Model Is Objective
Theorem. For deterministic labels, the labeling model given by
uniquely satisfies the measurement objectivity principle
36 Microsoft Research Redmond
![Page 37: Learning from the Wisdom of Crowds - microsoft.com from the Wisdom of Crowds by Minimax Entropy Denny Zhou, John Platt, Sumit Basu and Yi Mao Microsoft Research, Redmond, WA](https://reader034.vdocument.in/reader034/viewer/2022051801/5adfbbdc7f8b9a8f298d4c4c/html5/thumbnails/37.jpg)
Constraint Relaxation
Item 1 Item 2 Item 3 Item 4 Item 5 Item 6
1 2 1 1 1 2
2 2 1 2 1 1
1 1 2 1 1 2
1 1 1 1 1 2
1 1 1 2 2 2
Worker 1
Worker 1
Worker 3
Worker 4
Worker 5
For each item: Count # workers labeling it as class 1 Count # workers labeling it as class 2
column matching
37 Microsoft Research Redmond
![Page 38: Learning from the Wisdom of Crowds - microsoft.com from the Wisdom of Crowds by Minimax Entropy Denny Zhou, John Platt, Sumit Basu and Yi Mao Microsoft Research, Redmond, WA](https://reader034.vdocument.in/reader034/viewer/2022051801/5adfbbdc7f8b9a8f298d4c4c/html5/thumbnails/38.jpg)
Constraint Relaxation
Item 1 Item 2 Item 3 Item 4 Item 5 Item 6
1 2 1 1 1 2
2 2 1 2 1 1
1 1 2 1 1 2
1 1 1 1 1 2
1 1 1 2 2 2
Worker 1
Worker 1
Worker 3
Worker 4
Worker 5
For each worker: Count # misclassifications from class 1 to 2 Count # misclassifications from class 2 to 1
38 Microsoft Research Redmond
row matching
![Page 39: Learning from the Wisdom of Crowds - microsoft.com from the Wisdom of Crowds by Minimax Entropy Denny Zhou, John Platt, Sumit Basu and Yi Mao Microsoft Research, Redmond, WA](https://reader034.vdocument.in/reader034/viewer/2022051801/5adfbbdc7f8b9a8f298d4c4c/html5/thumbnails/39.jpg)
Relaxing moment constraints to prevent overfitting
Constraint Relaxation
Subject to
![Page 40: Learning from the Wisdom of Crowds - microsoft.com from the Wisdom of Crowds by Minimax Entropy Denny Zhou, John Platt, Sumit Basu and Yi Mao Microsoft Research, Redmond, WA](https://reader034.vdocument.in/reader034/viewer/2022051801/5adfbbdc7f8b9a8f298d4c4c/html5/thumbnails/40.jpg)
Implementation
• Convert the primal problem to its dual form
• Coordinate descent
– Split the variables into two blocks: 𝑦 , {𝜏, 𝜎}
– Each subproblem is convex and smooth
– Initialize ground truth by majority vote
Microsoft Research Redmond 40
![Page 41: Learning from the Wisdom of Crowds - microsoft.com from the Wisdom of Crowds by Minimax Entropy Denny Zhou, John Platt, Sumit Basu and Yi Mao Microsoft Research, Redmond, WA](https://reader034.vdocument.in/reader034/viewer/2022051801/5adfbbdc7f8b9a8f298d4c4c/html5/thumbnails/41.jpg)
Model Selection
• 𝑘-fold cross validation to choose (𝛼, 𝛽)
– Split the data matrix into 𝑘 folds
– Each fold used as a validation set once
– Compute average likelihood over validations
Microsoft Research Redmond 41
We don’t need ground truth for model selection!
![Page 42: Learning from the Wisdom of Crowds - microsoft.com from the Wisdom of Crowds by Minimax Entropy Denny Zhou, John Platt, Sumit Basu and Yi Mao Microsoft Research, Redmond, WA](https://reader034.vdocument.in/reader034/viewer/2022051801/5adfbbdc7f8b9a8f298d4c4c/html5/thumbnails/42.jpg)
Experiments: Image Labeling
• 108 bird images, 2 breeds, 39 workers
• Each image was labeled by all workers
42 Microsoft Research Redmond
From: P. Welinder, S. Branson, S. Belongie and P. Perona. The Multidimensional Wisdom of Crowds. NIPS 2010.
![Page 43: Learning from the Wisdom of Crowds - microsoft.com from the Wisdom of Crowds by Minimax Entropy Denny Zhou, John Platt, Sumit Basu and Yi Mao Microsoft Research, Redmond, WA](https://reader034.vdocument.in/reader034/viewer/2022051801/5adfbbdc7f8b9a8f298d4c4c/html5/thumbnails/43.jpg)
Experiments: Image Labeling
• Experimental results (accuracy, %)
Microsoft Research Redmond 43
Worker Number 10 20 30
Minimax Entropy 85.18 92.59 93.52
Dawid & Skene 79.63 87.04 87.96
Dawid & Skene (S)* 45.37 57.41 75.93
Majority Voting 67.59 83.33 76.85
Average Worker 62.78
* Dawid & Skene (S): simplified Dawid and Skene’s method
![Page 44: Learning from the Wisdom of Crowds - microsoft.com from the Wisdom of Crowds by Minimax Entropy Denny Zhou, John Platt, Sumit Basu and Yi Mao Microsoft Research, Redmond, WA](https://reader034.vdocument.in/reader034/viewer/2022051801/5adfbbdc7f8b9a8f298d4c4c/html5/thumbnails/44.jpg)
Experiments: Image Labeling
• Experimental results (accuracy, %)
Microsoft Research Redmond 44
Worker Number 10 20 30
Minimax Entropy 85.18 92.59 93.52
Dawid & Skene 79.63 87.04 87.96
Dawid & Skene (S) 45.37 57.41 75.00
Majority Voting 67.59 83.33 76.85
Average Worker 62.78
It is risky to model worker expertise by a single number
![Page 45: Learning from the Wisdom of Crowds - microsoft.com from the Wisdom of Crowds by Minimax Entropy Denny Zhou, John Platt, Sumit Basu and Yi Mao Microsoft Research, Redmond, WA](https://reader034.vdocument.in/reader034/viewer/2022051801/5adfbbdc7f8b9a8f298d4c4c/html5/thumbnails/45.jpg)
Experiments: Web Search
• 177 workers and 2665 <query, URL> pairs
• 5 classes: perfect, excellent, good, fair and bad
• Each pair was labeled by 6 workers
Minimax Entropy 88.84
Dawid & Skene 84.09
Majority Voting 77.65
Average worker 37.05
45 Microsoft Research Redmond
![Page 46: Learning from the Wisdom of Crowds - microsoft.com from the Wisdom of Crowds by Minimax Entropy Denny Zhou, John Platt, Sumit Basu and Yi Mao Microsoft Research, Redmond, WA](https://reader034.vdocument.in/reader034/viewer/2022051801/5adfbbdc7f8b9a8f298d4c4c/html5/thumbnails/46.jpg)
Comparing with More Methods
• Other methods: Raykar et al (JMLR 2010, adding beta/Dirichlet prior), Welinder et al (NIPS 2010, matrix factorization), Karger et al (NIPS, 2011, BP-like iterative algorithm)
• From the evaluation in (Liu et al. NIPS 2012) – None of them can outperform Dawid and Skene’s
– Karger et al (NIPS, 2011) is even much worse than majority voting
Microsoft Research Redmond 46
![Page 47: Learning from the Wisdom of Crowds - microsoft.com from the Wisdom of Crowds by Minimax Entropy Denny Zhou, John Platt, Sumit Basu and Yi Mao Microsoft Research, Redmond, WA](https://reader034.vdocument.in/reader034/viewer/2022051801/5adfbbdc7f8b9a8f298d4c4c/html5/thumbnails/47.jpg)
Microsoft Research Redmond 47
3. Future Work and Conclusion
![Page 48: Learning from the Wisdom of Crowds - microsoft.com from the Wisdom of Crowds by Minimax Entropy Denny Zhou, John Platt, Sumit Basu and Yi Mao Microsoft Research, Redmond, WA](https://reader034.vdocument.in/reader034/viewer/2022051801/5adfbbdc7f8b9a8f298d4c4c/html5/thumbnails/48.jpg)
Budget-Optimal Crowdsourcing
• Assume that we have a budget to get 6 labels. Which one deserves another label, item 2 or 3?
• How about having a budget of 7 labels or even more?
Microsoft Research Redmond 48
1st round 2nd round
Item 1 1 1
Item 2 1 -1
Item 3 1
![Page 49: Learning from the Wisdom of Crowds - microsoft.com from the Wisdom of Crowds by Minimax Entropy Denny Zhou, John Platt, Sumit Basu and Yi Mao Microsoft Research, Redmond, WA](https://reader034.vdocument.in/reader034/viewer/2022051801/5adfbbdc7f8b9a8f298d4c4c/html5/thumbnails/49.jpg)
Contextual Minimax Entropy
• Contextual information of items and workers
• (An example) Label a web page as spam or nonspam by a group of workers
– For each web page: its URL ends with .edu or not, popularity of its domain, creating time
– For each worker: education degree, reputation history, working experience
Microsoft Research Redmond 49
![Page 50: Learning from the Wisdom of Crowds - microsoft.com from the Wisdom of Crowds by Minimax Entropy Denny Zhou, John Platt, Sumit Basu and Yi Mao Microsoft Research, Redmond, WA](https://reader034.vdocument.in/reader034/viewer/2022051801/5adfbbdc7f8b9a8f298d4c4c/html5/thumbnails/50.jpg)
Beyond Labeling
Mobile crowdsourcing platform Crowdsourcing machine translation Crowdsourcing indoor/outdoor navigation Crowdsourcing design Wikipedia …
![Page 51: Learning from the Wisdom of Crowds - microsoft.com from the Wisdom of Crowds by Minimax Entropy Denny Zhou, John Platt, Sumit Basu and Yi Mao Microsoft Research, Redmond, WA](https://reader034.vdocument.in/reader034/viewer/2022051801/5adfbbdc7f8b9a8f298d4c4c/html5/thumbnails/51.jpg)
ICML’13 Workshop Machine Learning Meets Crowdsourcing
Microsoft Research Redmond 51
http://www.ics.uci.edu/~qliu1/MLcrowd_ICML_workshop/
![Page 52: Learning from the Wisdom of Crowds - microsoft.com from the Wisdom of Crowds by Minimax Entropy Denny Zhou, John Platt, Sumit Basu and Yi Mao Microsoft Research, Redmond, WA](https://reader034.vdocument.in/reader034/viewer/2022051801/5adfbbdc7f8b9a8f298d4c4c/html5/thumbnails/52.jpg)
Summary
• Proposed minimax entropy principle for estimating ground truth from noisy labels
• Both task confusability and worker expertise are taken into account in our method
• Measurement objectivity is implied
52 Microsoft Research Redmond
![Page 53: Learning from the Wisdom of Crowds - microsoft.com from the Wisdom of Crowds by Minimax Entropy Denny Zhou, John Platt, Sumit Basu and Yi Mao Microsoft Research, Redmond, WA](https://reader034.vdocument.in/reader034/viewer/2022051801/5adfbbdc7f8b9a8f298d4c4c/html5/thumbnails/53.jpg)
Acknowledgments
• Daniel Hsu (MSR New England)
• Xi Chen (Carnegie Mellon University)
• Gabriella Kazai (MSR Cambridge)
• Chris Burges (MSR Redmond)
• Chris Meek (MSR Redmond)
53 Microsoft Research Redmond