lessons learned from large scale crowdsourced data...
TRANSCRIPT
![Page 1: Lessons Learned from Large Scale Crowdsourced Data ...image-net.org/tutorials/cvpr2015/crowdsourcing_slides.pdf · Classification Overview • 1.4M images • 1,000 classes Pelican](https://reader035.vdocument.in/reader035/viewer/2022070909/5f8ebfc41e90d25d73727415/html5/thumbnails/1.jpg)
Lessons Learned from Large‑Scale Crowdsourced Data Collection for ILSVRC
Jonathan Krause
![Page 2: Lessons Learned from Large Scale Crowdsourced Data ...image-net.org/tutorials/cvpr2015/crowdsourcing_slides.pdf · Classification Overview • 1.4M images • 1,000 classes Pelican](https://reader035.vdocument.in/reader035/viewer/2022070909/5f8ebfc41e90d25d73727415/html5/thumbnails/2.jpg)
OverviewClassification
Localization
Detection Pelican
![Page 3: Lessons Learned from Large Scale Crowdsourced Data ...image-net.org/tutorials/cvpr2015/crowdsourcing_slides.pdf · Classification Overview • 1.4M images • 1,000 classes Pelican](https://reader035.vdocument.in/reader035/viewer/2022070909/5f8ebfc41e90d25d73727415/html5/thumbnails/3.jpg)
OverviewClassification
Localization
Detection Pelican
![Page 4: Lessons Learned from Large Scale Crowdsourced Data ...image-net.org/tutorials/cvpr2015/crowdsourcing_slides.pdf · Classification Overview • 1.4M images • 1,000 classes Pelican](https://reader035.vdocument.in/reader035/viewer/2022070909/5f8ebfc41e90d25d73727415/html5/thumbnails/4.jpg)
OverviewClassification
Localization
Detection BirdFrog
![Page 5: Lessons Learned from Large Scale Crowdsourced Data ...image-net.org/tutorials/cvpr2015/crowdsourcing_slides.pdf · Classification Overview • 1.4M images • 1,000 classes Pelican](https://reader035.vdocument.in/reader035/viewer/2022070909/5f8ebfc41e90d25d73727415/html5/thumbnails/5.jpg)
Classification Overview• 1.4M images • 1,000 classes
Pelican
By hand: • 5 sec/image • 50% images correct • 12 hours worked/day
= 324 days!
![Page 6: Lessons Learned from Large Scale Crowdsourced Data ...image-net.org/tutorials/cvpr2015/crowdsourcing_slides.pdf · Classification Overview • 1.4M images • 1,000 classes Pelican](https://reader035.vdocument.in/reader035/viewer/2022070909/5f8ebfc41e90d25d73727415/html5/thumbnails/6.jpg)
CrowdsourcingLet the crowd do the work for you!
![Page 7: Lessons Learned from Large Scale Crowdsourced Data ...image-net.org/tutorials/cvpr2015/crowdsourcing_slides.pdf · Classification Overview • 1.4M images • 1,000 classes Pelican](https://reader035.vdocument.in/reader035/viewer/2022070909/5f8ebfc41e90d25d73727415/html5/thumbnails/7.jpg)
Classification Pipeline1. Collect candidate images for each category
2. Put candidate images on Amazon Mechanical Turk (AMT)
3. AMT workers click on images containing each class
4. Aggregate worker responses into labels
![Page 8: Lessons Learned from Large Scale Crowdsourced Data ...image-net.org/tutorials/cvpr2015/crowdsourcing_slides.pdf · Classification Overview • 1.4M images • 1,000 classes Pelican](https://reader035.vdocument.in/reader035/viewer/2022070909/5f8ebfc41e90d25d73727415/html5/thumbnails/8.jpg)
Collecting ImagesCategory: “Whippet”
Google Image Search:
![Page 9: Lessons Learned from Large Scale Crowdsourced Data ...image-net.org/tutorials/cvpr2015/crowdsourcing_slides.pdf · Classification Overview • 1.4M images • 1,000 classes Pelican](https://reader035.vdocument.in/reader035/viewer/2022070909/5f8ebfc41e90d25d73727415/html5/thumbnails/9.jpg)
Problem: Limited Images• Web searches are limited • Solution: Query Expansion
• WordNet: Whippet: “a small slender dog of greyhound type developed in England”
→ “whippet dog”, “whippet greyhound”
→ translate into other languages
![Page 10: Lessons Learned from Large Scale Crowdsourced Data ...image-net.org/tutorials/cvpr2015/crowdsourcing_slides.pdf · Classification Overview • 1.4M images • 1,000 classes Pelican](https://reader035.vdocument.in/reader035/viewer/2022070909/5f8ebfc41e90d25d73727415/html5/thumbnails/10.jpg)
Deploying on AMT
Annotate many images at once!
![Page 11: Lessons Learned from Large Scale Crowdsourced Data ...image-net.org/tutorials/cvpr2015/crowdsourcing_slides.pdf · Classification Overview • 1.4M images • 1,000 classes Pelican](https://reader035.vdocument.in/reader035/viewer/2022070909/5f8ebfc41e90d25d73727415/html5/thumbnails/11.jpg)
Make sure workers
understand the classes!
![Page 12: Lessons Learned from Large Scale Crowdsourced Data ...image-net.org/tutorials/cvpr2015/crowdsourcing_slides.pdf · Classification Overview • 1.4M images • 1,000 classes Pelican](https://reader035.vdocument.in/reader035/viewer/2022070909/5f8ebfc41e90d25d73727415/html5/thumbnails/12.jpg)
Understanding ClassesWikipedia and Google links
![Page 13: Lessons Learned from Large Scale Crowdsourced Data ...image-net.org/tutorials/cvpr2015/crowdsourcing_slides.pdf · Classification Overview • 1.4M images • 1,000 classes Pelican](https://reader035.vdocument.in/reader035/viewer/2022070909/5f8ebfc41e90d25d73727415/html5/thumbnails/13.jpg)
Understanding ClassesGive them a definition delta: a low triangular area of alluvial deposits where a river divid before entering a larger body of water: “the Mississippi River delta”; “the Nile delta”
![Page 14: Lessons Learned from Large Scale Crowdsourced Data ...image-net.org/tutorials/cvpr2015/crowdsourcing_slides.pdf · Classification Overview • 1.4M images • 1,000 classes Pelican](https://reader035.vdocument.in/reader035/viewer/2022070909/5f8ebfc41e90d25d73727415/html5/thumbnails/14.jpg)
Understanding ClassesTest them on the definition
![Page 15: Lessons Learned from Large Scale Crowdsourced Data ...image-net.org/tutorials/cvpr2015/crowdsourcing_slides.pdf · Classification Overview • 1.4M images • 1,000 classes Pelican](https://reader035.vdocument.in/reader035/viewer/2022070909/5f8ebfc41e90d25d73727415/html5/thumbnails/15.jpg)
Understanding ClassesTest them on the definition
![Page 16: Lessons Learned from Large Scale Crowdsourced Data ...image-net.org/tutorials/cvpr2015/crowdsourcing_slides.pdf · Classification Overview • 1.4M images • 1,000 classes Pelican](https://reader035.vdocument.in/reader035/viewer/2022070909/5f8ebfc41e90d25d73727415/html5/thumbnails/16.jpg)
Understanding ClassesGive example images (if you have them)
Hard Easya small slender dog of greyhound type developed in England
a small slender dog of greyhound type developed in England
+
![Page 17: Lessons Learned from Large Scale Crowdsourced Data ...image-net.org/tutorials/cvpr2015/crowdsourcing_slides.pdf · Classification Overview • 1.4M images • 1,000 classes Pelican](https://reader035.vdocument.in/reader035/viewer/2022070909/5f8ebfc41e90d25d73727415/html5/thumbnails/17.jpg)
Quality ControlWorkers on AMT are:
• Fast • Inexpensive • Plentiful
But they are not: • Highly trained
Solution: Multiple responses, merge results
![Page 18: Lessons Learned from Large Scale Crowdsourced Data ...image-net.org/tutorials/cvpr2015/crowdsourcing_slides.pdf · Classification Overview • 1.4M images • 1,000 classes Pelican](https://reader035.vdocument.in/reader035/viewer/2022070909/5f8ebfc41e90d25d73727415/html5/thumbnails/18.jpg)
Quality ControlGiven
• Set of (worker, image, response)
Want • P(image has label) for each image • (Optionally) worker quality estimates
![Page 19: Lessons Learned from Large Scale Crowdsourced Data ...image-net.org/tutorials/cvpr2015/crowdsourcing_slides.pdf · Classification Overview • 1.4M images • 1,000 classes Pelican](https://reader035.vdocument.in/reader035/viewer/2022070909/5f8ebfc41e90d25d73727415/html5/thumbnails/19.jpg)
A Simple Method• Majority vote
Q: Is this a whippet?Responses:
Yes No Yes Yes No No Yes
Yes
![Page 20: Lessons Learned from Large Scale Crowdsourced Data ...image-net.org/tutorials/cvpr2015/crowdsourcing_slides.pdf · Classification Overview • 1.4M images • 1,000 classes Pelican](https://reader035.vdocument.in/reader035/viewer/2022070909/5f8ebfc41e90d25d73727415/html5/thumbnails/20.jpg)
Majority VoteProblems: • Doesn’t give confidence • Hard to measure worker quality
Responses: Yes No Yes Yes No No Yes
How sure are we it’s positive?
How good are these workers?
![Page 21: Lessons Learned from Large Scale Crowdsourced Data ...image-net.org/tutorials/cvpr2015/crowdsourcing_slides.pdf · Classification Overview • 1.4M images • 1,000 classes Pelican](https://reader035.vdocument.in/reader035/viewer/2022070909/5f8ebfc41e90d25d73727415/html5/thumbnails/21.jpg)
One Approach• Annotate a subset of images with many
annotations • Majority vote to determine ground truth • Determine confidence given fewer annotations
Deng et al. 2009
![Page 22: Lessons Learned from Large Scale Crowdsourced Data ...image-net.org/tutorials/cvpr2015/crowdsourcing_slides.pdf · Classification Overview • 1.4M images • 1,000 classes Pelican](https://reader035.vdocument.in/reader035/viewer/2022070909/5f8ebfc41e90d25d73727415/html5/thumbnails/22.jpg)
Pro & ConPro
• Simple • Gives image confidence
Con • Treats all workers the same • Relies on initial majority vote
![Page 23: Lessons Learned from Large Scale Crowdsourced Data ...image-net.org/tutorials/cvpr2015/crowdsourcing_slides.pdf · Classification Overview • 1.4M images • 1,000 classes Pelican](https://reader035.vdocument.in/reader035/viewer/2022070909/5f8ebfc41e90d25d73727415/html5/thumbnails/23.jpg)
Another ApproachModel:
• Prior of label correct • Worker confusion matrix
Dawid, Skene. 1979
Max-likelihood with EM
![Page 24: Lessons Learned from Large Scale Crowdsourced Data ...image-net.org/tutorials/cvpr2015/crowdsourcing_slides.pdf · Classification Overview • 1.4M images • 1,000 classes Pelican](https://reader035.vdocument.in/reader035/viewer/2022070909/5f8ebfc41e90d25d73727415/html5/thumbnails/24.jpg)
Another ApproachWorker Quality
• Compute Soft Label: distribution over labels given worker response
• Calculate expected cost of soft label q:
Ipeirotis, Provost, Wang. 2012
![Page 25: Lessons Learned from Large Scale Crowdsourced Data ...image-net.org/tutorials/cvpr2015/crowdsourcing_slides.pdf · Classification Overview • 1.4M images • 1,000 classes Pelican](https://reader035.vdocument.in/reader035/viewer/2022070909/5f8ebfc41e90d25d73727415/html5/thumbnails/25.jpg)
Pro & ConPro
• Gives image confidence • Gives worker quality
Con • More complex • Need to run optimization
![Page 26: Lessons Learned from Large Scale Crowdsourced Data ...image-net.org/tutorials/cvpr2015/crowdsourcing_slides.pdf · Classification Overview • 1.4M images • 1,000 classes Pelican](https://reader035.vdocument.in/reader035/viewer/2022070909/5f8ebfc41e90d25d73727415/html5/thumbnails/26.jpg)
OverviewClassification
Localization
Detection Pelican
![Page 27: Lessons Learned from Large Scale Crowdsourced Data ...image-net.org/tutorials/cvpr2015/crowdsourcing_slides.pdf · Classification Overview • 1.4M images • 1,000 classes Pelican](https://reader035.vdocument.in/reader035/viewer/2022070909/5f8ebfc41e90d25d73727415/html5/thumbnails/27.jpg)
Localization Overview• Classification images • 1,000 classes • 600k training bounding boxes
Pelican
Main Challenge: Collecting and verifying bounding boxes
![Page 28: Lessons Learned from Large Scale Crowdsourced Data ...image-net.org/tutorials/cvpr2015/crowdsourcing_slides.pdf · Classification Overview • 1.4M images • 1,000 classes Pelican](https://reader035.vdocument.in/reader035/viewer/2022070909/5f8ebfc41e90d25d73727415/html5/thumbnails/28.jpg)
Bounding BoxesRequirements:
• Tight around object • Around all object instances • Not around other objects
Su, Deng, Fei-Fei. 2012
bounding boxes for “bottle”
![Page 29: Lessons Learned from Large Scale Crowdsourced Data ...image-net.org/tutorials/cvpr2015/crowdsourcing_slides.pdf · Classification Overview • 1.4M images • 1,000 classes Pelican](https://reader035.vdocument.in/reader035/viewer/2022070909/5f8ebfc41e90d25d73727415/html5/thumbnails/29.jpg)
Tasks1. Draw a bounding box around a single instance
2. Quality verification of bounding box
3. Coverage verification
![Page 30: Lessons Learned from Large Scale Crowdsourced Data ...image-net.org/tutorials/cvpr2015/crowdsourcing_slides.pdf · Classification Overview • 1.4M images • 1,000 classes Pelican](https://reader035.vdocument.in/reader035/viewer/2022070909/5f8ebfc41e90d25d73727415/html5/thumbnails/30.jpg)
DrawingIntuitively simple…..
But the devil is in the details
![Page 31: Lessons Learned from Large Scale Crowdsourced Data ...image-net.org/tutorials/cvpr2015/crowdsourcing_slides.pdf · Classification Overview • 1.4M images • 1,000 classes Pelican](https://reader035.vdocument.in/reader035/viewer/2022070909/5f8ebfc41e90d25d73727415/html5/thumbnails/31.jpg)
DrawingThings vision researchers take for granted
• Include all visible parts • Include only visible parts • Make the bounding box tight • Only include a single instance • Don’t draw over any instances that already have
bounding boxes • What if there are no unannotated objects?
→ Provide instructions and use a qualification task!
![Page 32: Lessons Learned from Large Scale Crowdsourced Data ...image-net.org/tutorials/cvpr2015/crowdsourcing_slides.pdf · Classification Overview • 1.4M images • 1,000 classes Pelican](https://reader035.vdocument.in/reader035/viewer/2022070909/5f8ebfc41e90d25d73727415/html5/thumbnails/32.jpg)
DrawingInclude all visible parts
Good Bad
![Page 33: Lessons Learned from Large Scale Crowdsourced Data ...image-net.org/tutorials/cvpr2015/crowdsourcing_slides.pdf · Classification Overview • 1.4M images • 1,000 classes Pelican](https://reader035.vdocument.in/reader035/viewer/2022070909/5f8ebfc41e90d25d73727415/html5/thumbnails/33.jpg)
DrawingInclude only all visible parts
• Don’t try to “complete” the object
Good Bad
![Page 34: Lessons Learned from Large Scale Crowdsourced Data ...image-net.org/tutorials/cvpr2015/crowdsourcing_slides.pdf · Classification Overview • 1.4M images • 1,000 classes Pelican](https://reader035.vdocument.in/reader035/viewer/2022070909/5f8ebfc41e90d25d73727415/html5/thumbnails/34.jpg)
DrawingMake the bounding box tight
• Even though loose is much faster
Good Bad
![Page 35: Lessons Learned from Large Scale Crowdsourced Data ...image-net.org/tutorials/cvpr2015/crowdsourcing_slides.pdf · Classification Overview • 1.4M images • 1,000 classes Pelican](https://reader035.vdocument.in/reader035/viewer/2022070909/5f8ebfc41e90d25d73727415/html5/thumbnails/35.jpg)
DrawingOnly include a single instance
Good Bad
![Page 36: Lessons Learned from Large Scale Crowdsourced Data ...image-net.org/tutorials/cvpr2015/crowdsourcing_slides.pdf · Classification Overview • 1.4M images • 1,000 classes Pelican](https://reader035.vdocument.in/reader035/viewer/2022070909/5f8ebfc41e90d25d73727415/html5/thumbnails/36.jpg)
DrawingDon’t draw over instances that already have bounding boxes
• Can enforce this in the UI
Good Bad
![Page 37: Lessons Learned from Large Scale Crowdsourced Data ...image-net.org/tutorials/cvpr2015/crowdsourcing_slides.pdf · Classification Overview • 1.4M images • 1,000 classes Pelican](https://reader035.vdocument.in/reader035/viewer/2022070909/5f8ebfc41e90d25d73727415/html5/thumbnails/37.jpg)
DrawingWhat if there are no unannotated objects?
• Give option to annotate no bounding boxes
Good Bad
→ No more objects anything else
![Page 38: Lessons Learned from Large Scale Crowdsourced Data ...image-net.org/tutorials/cvpr2015/crowdsourcing_slides.pdf · Classification Overview • 1.4M images • 1,000 classes Pelican](https://reader035.vdocument.in/reader035/viewer/2022070909/5f8ebfc41e90d25d73727415/html5/thumbnails/38.jpg)
Quality VerificationSimpler than bounding box drawing
Still has some details
Is this bounding box good? YES
![Page 39: Lessons Learned from Large Scale Crowdsourced Data ...image-net.org/tutorials/cvpr2015/crowdsourcing_slides.pdf · Classification Overview • 1.4M images • 1,000 classes Pelican](https://reader035.vdocument.in/reader035/viewer/2022070909/5f8ebfc41e90d25d73727415/html5/thumbnails/39.jpg)
Quality VerificationDetails:
• Still need to know about good bounding boxes • Quality control
Is this bounding box good? YES
![Page 40: Lessons Learned from Large Scale Crowdsourced Data ...image-net.org/tutorials/cvpr2015/crowdsourcing_slides.pdf · Classification Overview • 1.4M images • 1,000 classes Pelican](https://reader035.vdocument.in/reader035/viewer/2022070909/5f8ebfc41e90d25d73727415/html5/thumbnails/40.jpg)
Quality VerificationQuality control
• Embed “gold standard” images • Positives: Majority vote • Negatives: Perturb the positives • Reject annotations if bad answers to these • Can be used for almost any type of task!
• (Optionally) require agreement of more than one annotator
![Page 41: Lessons Learned from Large Scale Crowdsourced Data ...image-net.org/tutorials/cvpr2015/crowdsourcing_slides.pdf · Classification Overview • 1.4M images • 1,000 classes Pelican](https://reader035.vdocument.in/reader035/viewer/2022070909/5f8ebfc41e90d25d73727415/html5/thumbnails/41.jpg)
Coverage Verification
Any unannotated raccoons?
Similar in style to quality verification • Just a different question • Still need instructions, quality control
Nope!
![Page 42: Lessons Learned from Large Scale Crowdsourced Data ...image-net.org/tutorials/cvpr2015/crowdsourcing_slides.pdf · Classification Overview • 1.4M images • 1,000 classes Pelican](https://reader035.vdocument.in/reader035/viewer/2022070909/5f8ebfc41e90d25d73727415/html5/thumbnails/42.jpg)
Bounding Boxes: Misc.Provide definitions and example images!
• Especially if uncommon objects • But also helps with common objects • Annotators from different cultures
Make sure objects being annotated are actually in your images
• Do the classification task first
![Page 43: Lessons Learned from Large Scale Crowdsourced Data ...image-net.org/tutorials/cvpr2015/crowdsourcing_slides.pdf · Classification Overview • 1.4M images • 1,000 classes Pelican](https://reader035.vdocument.in/reader035/viewer/2022070909/5f8ebfc41e90d25d73727415/html5/thumbnails/43.jpg)
Bounding Boxes: Misc.Make qualification tasks
Verification tasks are much faster than drawing
Corner cases: Each task needs plan for when previous task goes wrong.
![Page 44: Lessons Learned from Large Scale Crowdsourced Data ...image-net.org/tutorials/cvpr2015/crowdsourcing_slides.pdf · Classification Overview • 1.4M images • 1,000 classes Pelican](https://reader035.vdocument.in/reader035/viewer/2022070909/5f8ebfc41e90d25d73727415/html5/thumbnails/44.jpg)
Detection Overview• 456k training images • 61k fully-annotated val+test • 200 classes
BirdFrog
![Page 45: Lessons Learned from Large Scale Crowdsourced Data ...image-net.org/tutorials/cvpr2015/crowdsourcing_slides.pdf · Classification Overview • 1.4M images • 1,000 classes Pelican](https://reader035.vdocument.in/reader035/viewer/2022070909/5f8ebfc41e90d25d73727415/html5/thumbnails/45.jpg)
Detection Overview• 456k training images • 61k fully-annotated val+test • 200 classes
BirdFrog
Main Challenge: Annotating all 200 classes in every image.
![Page 46: Lessons Learned from Large Scale Crowdsourced Data ...image-net.org/tutorials/cvpr2015/crowdsourcing_slides.pdf · Classification Overview • 1.4M images • 1,000 classes Pelican](https://reader035.vdocument.in/reader035/viewer/2022070909/5f8ebfc41e90d25d73727415/html5/thumbnails/46.jpg)
Detection Pipeline1. Collect images
2. Class presence annotation
3. Bounding box annotation
BirdFrog
![Page 47: Lessons Learned from Large Scale Crowdsourced Data ...image-net.org/tutorials/cvpr2015/crowdsourcing_slides.pdf · Classification Overview • 1.4M images • 1,000 classes Pelican](https://reader035.vdocument.in/reader035/viewer/2022070909/5f8ebfc41e90d25d73727415/html5/thumbnails/47.jpg)
Detection Pipeline1. Collect images
2. Class presence annotation
3. Bounding box annotation
BirdFrog
Same as previous
![Page 48: Lessons Learned from Large Scale Crowdsourced Data ...image-net.org/tutorials/cvpr2015/crowdsourcing_slides.pdf · Classification Overview • 1.4M images • 1,000 classes Pelican](https://reader035.vdocument.in/reader035/viewer/2022070909/5f8ebfc41e90d25d73727415/html5/thumbnails/48.jpg)
Detection Pipeline1. Collect images
2. Class presence annotation
3. Bounding box annotation
BirdFrog
![Page 49: Lessons Learned from Large Scale Crowdsourced Data ...image-net.org/tutorials/cvpr2015/crowdsourcing_slides.pdf · Classification Overview • 1.4M images • 1,000 classes Pelican](https://reader035.vdocument.in/reader035/viewer/2022070909/5f8ebfc41e90d25d73727415/html5/thumbnails/49.jpg)
Collecting ImagesNeed images that aren’t single object-centric
Additional queries: • Compound object queries (“tiger lion”,
“skunk and cat”) • Complex scene queries (“kitchenette”,
“dining table”, “orchestra”)
![Page 50: Lessons Learned from Large Scale Crowdsourced Data ...image-net.org/tutorials/cvpr2015/crowdsourcing_slides.pdf · Classification Overview • 1.4M images • 1,000 classes Pelican](https://reader035.vdocument.in/reader035/viewer/2022070909/5f8ebfc41e90d25d73727415/html5/thumbnails/50.jpg)
Detection Pipeline1. Collect images
2. Class presence annotation
3. Bounding box annotation
BirdFrog
![Page 51: Lessons Learned from Large Scale Crowdsourced Data ...image-net.org/tutorials/cvpr2015/crowdsourcing_slides.pdf · Classification Overview • 1.4M images • 1,000 classes Pelican](https://reader035.vdocument.in/reader035/viewer/2022070909/5f8ebfc41e90d25d73727415/html5/thumbnails/51.jpg)
Deng, Russakovsky, Krause, Bernstein, Berg, Fei-‐Fei. CHI 2014
Naive approach: ask for each object
Answer
Question
Machine Crowd
Is there a table?
Yes
Table Chair Horse Dog Cat Bird
? ? ? ? ? ?
![Page 52: Lessons Learned from Large Scale Crowdsourced Data ...image-net.org/tutorials/cvpr2015/crowdsourcing_slides.pdf · Classification Overview • 1.4M images • 1,000 classes Pelican](https://reader035.vdocument.in/reader035/viewer/2022070909/5f8ebfc41e90d25d73727415/html5/thumbnails/52.jpg)
Deng, Russakovsky, Krause, Bernstein, Berg, Fei-‐Fei. CHI 2014
Naive approach: ask for each object
Table Chair Horse Dog Cat Bird
+ ? ? ? ? ?
Answer
Question
Machine Crowd
Is there a table?
Yes
![Page 53: Lessons Learned from Large Scale Crowdsourced Data ...image-net.org/tutorials/cvpr2015/crowdsourcing_slides.pdf · Classification Overview • 1.4M images • 1,000 classes Pelican](https://reader035.vdocument.in/reader035/viewer/2022070909/5f8ebfc41e90d25d73727415/html5/thumbnails/53.jpg)
Deng, Russakovsky, Krause, Bernstein, Berg, Fei-‐Fei. CHI 2014
Table Chair Horse Dog Cat Bird
+ + ? ? ? ?
Answer
Question
Machine Crowd
Is there a chair?
Yes
Naive approach: ask for each object
![Page 54: Lessons Learned from Large Scale Crowdsourced Data ...image-net.org/tutorials/cvpr2015/crowdsourcing_slides.pdf · Classification Overview • 1.4M images • 1,000 classes Pelican](https://reader035.vdocument.in/reader035/viewer/2022070909/5f8ebfc41e90d25d73727415/html5/thumbnails/54.jpg)
Deng, Russakovsky, Krause, Bernstein, Berg, Fei-‐Fei. CHI 2014
Table Chair Horse Dog Cat Bird
+ + -‐ ? ? ?
Answer
Question
Machine Crowd
Is there a horse?
No
Naive approach: ask for each object
![Page 55: Lessons Learned from Large Scale Crowdsourced Data ...image-net.org/tutorials/cvpr2015/crowdsourcing_slides.pdf · Classification Overview • 1.4M images • 1,000 classes Pelican](https://reader035.vdocument.in/reader035/viewer/2022070909/5f8ebfc41e90d25d73727415/html5/thumbnails/55.jpg)
Deng, Russakovsky, Krause, Bernstein, Berg, Fei-‐Fei. CHI 2014
Table Chair Horse Dog Cat Bird
+ + -‐ -‐ ? ?
Answer
Question
Machine Crowd
Is there a dog?
No
Naive approach: ask for each object
![Page 56: Lessons Learned from Large Scale Crowdsourced Data ...image-net.org/tutorials/cvpr2015/crowdsourcing_slides.pdf · Classification Overview • 1.4M images • 1,000 classes Pelican](https://reader035.vdocument.in/reader035/viewer/2022070909/5f8ebfc41e90d25d73727415/html5/thumbnails/56.jpg)
Deng, Russakovsky, Krause, Bernstein, Berg, Fei-‐Fei. CHI 2014
Table Chair Horse Dog Cat Bird
+ + -‐ -‐ -‐ ?
Answer
Question
Machine Crowd
Is there a cat?
No
Naive approach: ask for each object
![Page 57: Lessons Learned from Large Scale Crowdsourced Data ...image-net.org/tutorials/cvpr2015/crowdsourcing_slides.pdf · Classification Overview • 1.4M images • 1,000 classes Pelican](https://reader035.vdocument.in/reader035/viewer/2022070909/5f8ebfc41e90d25d73727415/html5/thumbnails/57.jpg)
Deng, Russakovsky, Krause, Bernstein, Berg, Fei-‐Fei. CHI 2014
Table Chair Horse Dog Cat Bird
+ + -‐ -‐ -‐ -‐
Answer
Question
Machine Crowd
Is there a bird?
No
Naive approach: ask for each object
![Page 58: Lessons Learned from Large Scale Crowdsourced Data ...image-net.org/tutorials/cvpr2015/crowdsourcing_slides.pdf · Classification Overview • 1.4M images • 1,000 classes Pelican](https://reader035.vdocument.in/reader035/viewer/2022070909/5f8ebfc41e90d25d73727415/html5/thumbnails/58.jpg)
Deng, Russakovsky, Krause, Bernstein, Berg, Fei-‐Fei. CHI 2014
Table Chair Horse Dog Cat Bird
+ + -‐ -‐ -‐ -‐
+ -‐ -‐ -‐ + -‐
+ + -‐ -‐ -‐ -‐
Cost: O(NK) for N images and K objects
Naive approach: ask for each object
![Page 59: Lessons Learned from Large Scale Crowdsourced Data ...image-net.org/tutorials/cvpr2015/crowdsourcing_slides.pdf · Classification Overview • 1.4M images • 1,000 classes Pelican](https://reader035.vdocument.in/reader035/viewer/2022070909/5f8ebfc41e90d25d73727415/html5/thumbnails/59.jpg)
Deng, Russakovsky, Krause, Bernstein, Berg, Fei-‐Fei. CHI 2014
Table Chair Horse Dog Cat Bird
+ + -‐ -‐ -‐ -‐
Furniture Mammal
AnimalHierarchy
![Page 60: Lessons Learned from Large Scale Crowdsourced Data ...image-net.org/tutorials/cvpr2015/crowdsourcing_slides.pdf · Classification Overview • 1.4M images • 1,000 classes Pelican](https://reader035.vdocument.in/reader035/viewer/2022070909/5f8ebfc41e90d25d73727415/html5/thumbnails/60.jpg)
Deng, Russakovsky, Krause, Bernstein, Berg, Fei-‐Fei. CHI 2014
Table Chair Horse Dog Cat Bird
+ + -‐ -‐ -‐ -‐
+ -‐ -‐ -‐ + -‐
+ + -‐ -‐ -‐ -‐
SparsityCorrelation
FurnitureMammal
AnimalHierarchy
![Page 61: Lessons Learned from Large Scale Crowdsourced Data ...image-net.org/tutorials/cvpr2015/crowdsourcing_slides.pdf · Classification Overview • 1.4M images • 1,000 classes Pelican](https://reader035.vdocument.in/reader035/viewer/2022070909/5f8ebfc41e90d25d73727415/html5/thumbnails/61.jpg)
Deng, Russakovsky, Krause, Bernstein, Berg, Fei-‐Fei. CHI 2014
Table Chair Horse Dog Cat Bird
? ? ? ? ? ?
Answer
Question
Machine Crowd
Furniture Mammal
Animal
Better approach: exploit label structure
![Page 62: Lessons Learned from Large Scale Crowdsourced Data ...image-net.org/tutorials/cvpr2015/crowdsourcing_slides.pdf · Classification Overview • 1.4M images • 1,000 classes Pelican](https://reader035.vdocument.in/reader035/viewer/2022070909/5f8ebfc41e90d25d73727415/html5/thumbnails/62.jpg)
Deng, Russakovsky, Krause, Bernstein, Berg, Fei-‐Fei. CHI 2014
Table Chair Horse Dog Cat Bird
? ? ? ? ? ?
Answer
Question
Machine Crowd
Is there an animal?
No
Furniture Mammal
Animal
Better approach: exploit label structure
![Page 63: Lessons Learned from Large Scale Crowdsourced Data ...image-net.org/tutorials/cvpr2015/crowdsourcing_slides.pdf · Classification Overview • 1.4M images • 1,000 classes Pelican](https://reader035.vdocument.in/reader035/viewer/2022070909/5f8ebfc41e90d25d73727415/html5/thumbnails/63.jpg)
Deng, Russakovsky, Krause, Bernstein, Berg, Fei-‐Fei. CHI 2014
Table Chair Horse Dog Cat Bird
? ? -‐ -‐ -‐ -‐
Answer
Question
Machine Crowd
Is there an animal?
No
Furniture Mammal
Animal
Better approach: exploit label structure
![Page 64: Lessons Learned from Large Scale Crowdsourced Data ...image-net.org/tutorials/cvpr2015/crowdsourcing_slides.pdf · Classification Overview • 1.4M images • 1,000 classes Pelican](https://reader035.vdocument.in/reader035/viewer/2022070909/5f8ebfc41e90d25d73727415/html5/thumbnails/64.jpg)
Deng, Russakovsky, Krause, Bernstein, Berg, Fei-‐Fei. CHI 2014
Table Chair Horse Dog Cat Bird
? ? -‐ -‐ -‐ -‐
Answer
Question
Machine Crowd
Is there furniture?
Yes
Mammal
Animal
Furniture
Better approach: exploit label structure
![Page 65: Lessons Learned from Large Scale Crowdsourced Data ...image-net.org/tutorials/cvpr2015/crowdsourcing_slides.pdf · Classification Overview • 1.4M images • 1,000 classes Pelican](https://reader035.vdocument.in/reader035/viewer/2022070909/5f8ebfc41e90d25d73727415/html5/thumbnails/65.jpg)
Deng, Russakovsky, Krause, Bernstein, Berg, Fei-‐Fei. CHI 2014
Answer
Question
Machine Crowd
Is there a table?
Yes
Mammal
Animal
Table Chair Horse Dog Cat Bird
? ? -‐ -‐ -‐ -‐
Furniture
Better approach: exploit label structure
![Page 66: Lessons Learned from Large Scale Crowdsourced Data ...image-net.org/tutorials/cvpr2015/crowdsourcing_slides.pdf · Classification Overview • 1.4M images • 1,000 classes Pelican](https://reader035.vdocument.in/reader035/viewer/2022070909/5f8ebfc41e90d25d73727415/html5/thumbnails/66.jpg)
Deng, Russakovsky, Krause, Bernstein, Berg, Fei-‐Fei. CHI 2014
Table Chair Horse Dog Cat Bird
+ ? -‐ -‐ -‐ -‐
Answer
Question
Machine Crowd
Is there a chair?
Yes
Mammal
Animal
Furniture
Better approach: exploit label structure
![Page 67: Lessons Learned from Large Scale Crowdsourced Data ...image-net.org/tutorials/cvpr2015/crowdsourcing_slides.pdf · Classification Overview • 1.4M images • 1,000 classes Pelican](https://reader035.vdocument.in/reader035/viewer/2022070909/5f8ebfc41e90d25d73727415/html5/thumbnails/67.jpg)
Deng, Russakovsky, Krause, Bernstein, Berg, Fei-‐Fei. CHI 2014
Answer
Question
Machine Crowd
Is there a chair?
Yes
Mammal
Animal
Table Chair Horse Dog Cat Bird
+ + -‐ -‐ -‐ -‐
Furniture
Better approach: exploit label structure
![Page 68: Lessons Learned from Large Scale Crowdsourced Data ...image-net.org/tutorials/cvpr2015/crowdsourcing_slides.pdf · Classification Overview • 1.4M images • 1,000 classes Pelican](https://reader035.vdocument.in/reader035/viewer/2022070909/5f8ebfc41e90d25d73727415/html5/thumbnails/68.jpg)
Deng, Russakovsky, Krause, Bernstein, Berg, Fei-‐Fei. CHI 2014
Selecting the Right Question
Goal: Get as much utility (new labels) as possible, for as little cost (worker time) as possible, given a desired level of accuracy.
![Page 69: Lessons Learned from Large Scale Crowdsourced Data ...image-net.org/tutorials/cvpr2015/crowdsourcing_slides.pdf · Classification Overview • 1.4M images • 1,000 classes Pelican](https://reader035.vdocument.in/reader035/viewer/2022070909/5f8ebfc41e90d25d73727415/html5/thumbnails/69.jpg)
Deng, Russakovsky, Krause, Bernstein, Berg, Fei-‐Fei. CHI 2014
Accuracy constraint
• User-specified accuracy threshold, e.g., 95% • Might require only one worker, might require
several based on the task
![Page 70: Lessons Learned from Large Scale Crowdsourced Data ...image-net.org/tutorials/cvpr2015/crowdsourcing_slides.pdf · Classification Overview • 1.4M images • 1,000 classes Pelican](https://reader035.vdocument.in/reader035/viewer/2022070909/5f8ebfc41e90d25d73727415/html5/thumbnails/70.jpg)
Deng, Russakovsky, Krause, Bernstein, Berg, Fei-‐Fei. CHI 2014
Cost: worker time (time = money)
Question (is there …) Cost (second)
a thing used to open cans/bottles 14.4an item that runs on electricity (plugged in or using batteries) 12.6a stringed instrument 3.4a canine 2.0
expected human time to get an answer with 95% accuracy
![Page 71: Lessons Learned from Large Scale Crowdsourced Data ...image-net.org/tutorials/cvpr2015/crowdsourcing_slides.pdf · Classification Overview • 1.4M images • 1,000 classes Pelican](https://reader035.vdocument.in/reader035/viewer/2022070909/5f8ebfc41e90d25d73727415/html5/thumbnails/71.jpg)
Deng, Russakovsky, Krause, Bernstein, Berg, Fei-‐Fei. CHI 2014
Utility: expected # of new labelsTable Chair Horse Dog Cat Bird
? ? ? ? ? ?
Is there a table?
Yes
No
Table Chair Horse Dog Cat Bird
+ ? ? ? ? ?
Table Chair Horse Dog Cat Bird
-‐ ? ? ? ? ?
utility = 1
![Page 72: Lessons Learned from Large Scale Crowdsourced Data ...image-net.org/tutorials/cvpr2015/crowdsourcing_slides.pdf · Classification Overview • 1.4M images • 1,000 classes Pelican](https://reader035.vdocument.in/reader035/viewer/2022070909/5f8ebfc41e90d25d73727415/html5/thumbnails/72.jpg)
Deng, Russakovsky, Krause, Bernstein, Berg, Fei-‐Fei. CHI 2014
Table Chair Horse Dog Cat Bird
? ? ? ? ? ?
Is there a table?
Yes
No
Table Chair Horse Dog Cat Bird
+ ? ? ? ? ?
Table Chair Horse Dog Cat Bird
-‐ ? ? ? ? ?
Table Chair Horse Dog Cat Bird
? ? ? ? ? ?
Is there an animal?
Table Chair Horse Dog Cat Bird
? ? ? ? ? ?
Table Chair Horse Dog Cat Bird
? ? -‐ -‐ -‐ -‐
utility = 1
utility = 0.5 * 0 + 0.5 * 4 = 2
Pr(Y) = 0.5
Pr(N) = 0.5
Utility: expected # of new labels
![Page 73: Lessons Learned from Large Scale Crowdsourced Data ...image-net.org/tutorials/cvpr2015/crowdsourcing_slides.pdf · Classification Overview • 1.4M images • 1,000 classes Pelican](https://reader035.vdocument.in/reader035/viewer/2022070909/5f8ebfc41e90d25d73727415/html5/thumbnails/73.jpg)
Deng, Russakovsky, Krause, Bernstein, Berg, Fei-‐Fei. CHI 2014
Pick the question with the most labels per second
Query: Is there a... Utility (num labels)
Cost (worker time in secs)
Utility-‐Cost Ratio (labels per sec)
mammal with claws or fingers
12.0 3.0 4.0
living organism 24.8 7.9 3.1mammal 17.6 7.4 2.4creature without legs 5.9 2.6 2.3land or avian creature 20.8 9.5 2.2
Selecting the Right Question
![Page 74: Lessons Learned from Large Scale Crowdsourced Data ...image-net.org/tutorials/cvpr2015/crowdsourcing_slides.pdf · Classification Overview • 1.4M images • 1,000 classes Pelican](https://reader035.vdocument.in/reader035/viewer/2022070909/5f8ebfc41e90d25d73727415/html5/thumbnails/74.jpg)
Deng, Russakovsky, Krause, Bernstein, Berg, Fei-‐Fei. CHI 2014
• Dataset: 20K images from ImageNet Challenge 2013. • Labels: 200 basic categories (dog, cat, table…) • 64 internal nodes in hierarchy
Results
![Page 75: Lessons Learned from Large Scale Crowdsourced Data ...image-net.org/tutorials/cvpr2015/crowdsourcing_slides.pdf · Classification Overview • 1.4M images • 1,000 classes Pelican](https://reader035.vdocument.in/reader035/viewer/2022070909/5f8ebfc41e90d25d73727415/html5/thumbnails/75.jpg)
Deng, Russakovsky, Krause, Bernstein, Berg, Fei-‐Fei. CHI 2014
Results: accuracy
Accuracy Threshold per question (parameter)
Accuracy (F1 score) Naive approach
Accuracy (F1 score) Our approach
0.95 99.64 (75.67) 99.75 (76.97)0.90 99.29 (60.17) 99.62 (60.69)
Annotating 10K images with 200 objects
![Page 76: Lessons Learned from Large Scale Crowdsourced Data ...image-net.org/tutorials/cvpr2015/crowdsourcing_slides.pdf · Classification Overview • 1.4M images • 1,000 classes Pelican](https://reader035.vdocument.in/reader035/viewer/2022070909/5f8ebfc41e90d25d73727415/html5/thumbnails/76.jpg)
Deng, Russakovsky, Krause, Bernstein, Berg, Fei-‐Fei. CHI 2014
Results: cost
Accuracy Threshold per question (parameter)
Cost saving (our approach compared to
naive approach)
0.95 3.93x0.90 6.18x
Annotating 10K images with 200 objects
6 times more labels per second
![Page 77: Lessons Learned from Large Scale Crowdsourced Data ...image-net.org/tutorials/cvpr2015/crowdsourcing_slides.pdf · Classification Overview • 1.4M images • 1,000 classes Pelican](https://reader035.vdocument.in/reader035/viewer/2022070909/5f8ebfc41e90d25d73727415/html5/thumbnails/77.jpg)
OverviewClassification
Localization
Detection BirdFrog
![Page 78: Lessons Learned from Large Scale Crowdsourced Data ...image-net.org/tutorials/cvpr2015/crowdsourcing_slides.pdf · Classification Overview • 1.4M images • 1,000 classes Pelican](https://reader035.vdocument.in/reader035/viewer/2022070909/5f8ebfc41e90d25d73727415/html5/thumbnails/78.jpg)
Final Thoughts• Provide good instructions • Do quality control • Visualize results • Listen to your workers
![Page 79: Lessons Learned from Large Scale Crowdsourced Data ...image-net.org/tutorials/cvpr2015/crowdsourcing_slides.pdf · Classification Overview • 1.4M images • 1,000 classes Pelican](https://reader035.vdocument.in/reader035/viewer/2022070909/5f8ebfc41e90d25d73727415/html5/thumbnails/79.jpg)
Questions?