budget-optimal task allocation for reliable crowdsourcing
Post on 28-Dec-2021
10 Views
Preview:
TRANSCRIPT
Budget-Optimal Task Allocation forReliable Crowdsourcing Systems
Sewoong Oh
Massachusetts Institute of Technologyjoint work with David R. Karger and Devavrat Shah
September 28, 2011
1 / 13
Crowdsourcing
Image classification
Character recognition
Transcription
Proofreading
2 / 13
Budget-optimal Crowdsourcing
Microtasks Workers
Add redundancy to cope with errors
Objective: Get reliable answers at minimum cost
Challenges
1. Task Allocation
→ Solution: Random Graph
2. Inference Problem
→ Solution: Low-rank Matrix Approximation
3 / 13
Budget-optimal Crowdsourcing
Microtasks Workers
Add redundancy to cope with errors
Objective: Get reliable answers at minimum cost
Challenges
1. Task Allocation → Solution: Random Graph2. Inference Problem → Solution: Low-rank Matrix Approximation
3 / 13
Previous Work on Reliable Crowdsourcing
Focuses on Inference problem
EM-based heuristics with no guarantees
I Dawid, Skene (’79)I Smyth et al. (’95)I Whitehill et al. (’09)I Welinder et al. (’10)
4 / 13
Task AllocationMicrotasks Batches
` r
Random (`, r)-regular bipartite graphs have good properties
I Locally Tree-like
→ Sharpen Analysis
I Good Expander
0
5
10
15
20
25
30
︸ ︷︷ ︸Gap
→ High Signal-to-Noise Ratio
5 / 13
Modeling the Crowd
−+
−−+
+−−+
−−−
−
+
−
−
1.0
0.5
0.0
0.9
0.7
0.6
i j
Binary tasks: si ∈ {+1,−1}Worker reliability: pj ∈ [0, 1]
Aij =
{si with probability pj−si with probability 1− pj
Assume we know if 1n
∑j pj > 0.5
6 / 13
Inference Problem
Given: Responses from the crowd {Aij}Find: Estimate of the answer {si}
si = sign(∑
j
Wij︸︷︷︸reliability
Aij︸︷︷︸response
)−+
−−+
+−−+
−
−−
?
?
?
?
?
?
?
?
?
?
1e-05
0.0001
0.001
0.01
0.1
1
0 5 10 15 20 25
Resources
Error rate
Majority VotingWij = 1
Oracle Estimator who knows pj ’sWij = log(
pj1−pj
)
7 / 13
Inference Problem
Given: Responses from the crowd {Aij}Find: Estimate of the answer {si}
si = sign(∑
j
Wij︸︷︷︸reliability
Aij︸︷︷︸response
)−+
−−+
+−−+
−
−−
?
?
?
?
?
?
?
?
?
?
1e-05
0.0001
0.001
0.01
0.1
1
0 5 10 15 20 25
Resources
Error rate
Majority VotingWij = 1
Oracle Estimator who knows pj ’sWij = log(
pj1−pj
)
Iterative Algorithm learns Wij ’s
7 / 13
Inference Problem
Given: Responses from the crowd {Aij}Find: Estimate of the answer {si}
si = sign(∑
j
Wij︸︷︷︸reliability
Aij︸︷︷︸response
)−+
−−+
+−−+
−
−−
?
?
?
?
?
?
?
?
?
?
Iteratively learn the weights
Task-likelihood update
i j
Sij
Lij︸︷︷︸likelihood
=∑j′ 6=j
Aij′ Wij′︸︷︷︸reliability
Worker-reliability update
j i
Wij
Wij︸︷︷︸reliability
=∑i′ 6=i
Ai′j Li′j︸︷︷︸likelihood
A task is likely to be ‘+’ if reliable
workers agree that it is ‘+’
A worker is reliable if the worker agreed
with our belief on other tasks7 / 13
Iterative Algorithm as Singular Vector Computation
A E[A|s, p] Random Perturbation
+=
︸ ︷︷ ︸data
︸ ︷︷ ︸low-rank signal
︸ ︷︷ ︸noise
−+−
−
+
+
−
−+−
−
−
−+−−
1. Why are the singular vectors good for inference?→ Good expanders have high SNR
2. Why not use the singular vectors directly?→ Exploit tree-like structure to prove a sharp bound
8 / 13
Performance Analysis
p1
p2
p3
p4
p5
p6` r
The performance depends on the worker reliability through
q ≡ 1
n
n∑j=1
(2pj − 1)2
Theorem. [Karger, O., Shah ’11]
In the large system limit, for σ2 ≡(
3 + 1qr
)q2lr
q2lr−1and `r > 1/q2
Perror ≤ exp{− q`
2σ2
}9 / 13
How Good is the Performance?
1e-05
0.0001
0.001
0.01
0.1
1
0 1 2 3 4 5 6 7 8 9
Majority Voting
EM Algorithm
Iterative Algorithm
Oracle Estimatorq`
Perror
Iterative algorithm (r > 1/q):
Perror ≤ e−116 q`
Matching minimax lower bound:
infAlg,G(`)
sup{si},{pj}∈F(q)
Perror & e−(q`+O(q2`))
10 / 13
Implications
PError ≤ e−116q`
How much do we need to spend to achieve PError ≤ ε ?
I Sufficient to choose ` ∼ 1q log( 1
ε )
I Necessary to have ` ∼ 1q log( 1
ε )
I Need q to determine `
I Can search for q using bisection
11 / 13
Resource Allocation
Which crowd is ‘better’?
Cost c1 = $0.04 c2 = $0.05
Worker Quality 0 0.2 0.4 0.6 0.8 1
P1 0 0.2 0.4 0.6 0.8 1
P2
q1 = E[(2P1 − 1)2] q2 = E[(2P2 − 1)2]
Invest all resources on arg max qkCk
12 / 13
Resource Allocation
Which crowd is ‘better’?
Cost c1 = $0.04 c2 = $0.05
Worker Quality 0 0.2 0.4 0.6 0.8 1
P1 0 0.2 0.4 0.6 0.8 1
P2
q1 = E[(2P1 − 1)2] q2 = E[(2P2 − 1)2]
Invest all resources on arg max qkCk
12 / 13
Conclusion
Problem: Reliable crowdsourcing with minimum resources
Task allocation: random regular graphs
Inference algorithm: low-rank matrix approximation
Required budget is order-optimal
13 / 13
top related