crowdflow integrating machine learning with mechanical turk for speed-cost-quality flexibility
DESCRIPTION
CrowdFlow Integrating Machine Learning with Mechanical Turk for Speed-Cost-Quality Flexibility. Alex Quinn, Ben Bederson, Tom Yeh , Jimmy Lin. Human Computation. Translation Photo tagging Face recognition Human detection Speech recognition Text analysis Planning. Things COMPUTERS can do. - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: CrowdFlow Integrating Machine Learning with Mechanical Turk for Speed-Cost-Quality Flexibility](https://reader036.vdocument.in/reader036/viewer/2022062501/56816231550346895dd26320/html5/thumbnails/1.jpg)
CrowdFlowIntegrating Machine Learning with Mechanical Turk for Speed-Cost-Quality Flexibility
Alex Quinn, Ben Bederson, Tom Yeh, Jimmy Lin
![Page 2: CrowdFlow Integrating Machine Learning with Mechanical Turk for Speed-Cost-Quality Flexibility](https://reader036.vdocument.in/reader036/viewer/2022062501/56816231550346895dd26320/html5/thumbnails/2.jpg)
Human Computation
ThingsHUMANS
can do
ThingsCOMPUTERS
can do
TranslationPhoto tagging
Face recognitionHuman detection
Speech recognitionText analysis
Planning
![Page 3: CrowdFlow Integrating Machine Learning with Mechanical Turk for Speed-Cost-Quality Flexibility](https://reader036.vdocument.in/reader036/viewer/2022062501/56816231550346895dd26320/html5/thumbnails/3.jpg)
Human Computation
ThingsHUMANS
can do
ThingsCOMPUTERS
can do
TranslationPhoto tagging
Face recognitionSpeech recognitionHuman detection
Text analysisPlanning
![Page 4: CrowdFlow Integrating Machine Learning with Mechanical Turk for Speed-Cost-Quality Flexibility](https://reader036.vdocument.in/reader036/viewer/2022062501/56816231550346895dd26320/html5/thumbnails/4.jpg)
Example: Human detection
![Page 5: CrowdFlow Integrating Machine Learning with Mechanical Turk for Speed-Cost-Quality Flexibility](https://reader036.vdocument.in/reader036/viewer/2022062501/56816231550346895dd26320/html5/thumbnails/5.jpg)
Trade-off space
Quality
Spee
d, A
fford
abili
ty
Computers
Human Workers(traditional)
Human Computation
![Page 6: CrowdFlow Integrating Machine Learning with Mechanical Turk for Speed-Cost-Quality Flexibility](https://reader036.vdocument.in/reader036/viewer/2022062501/56816231550346895dd26320/html5/thumbnails/6.jpg)
Trade-off space
Quality
Spee
d, A
fford
abili
ty
Computers
Human Computation
Human Workers(traditional)
![Page 7: CrowdFlow Integrating Machine Learning with Mechanical Turk for Speed-Cost-Quality Flexibility](https://reader036.vdocument.in/reader036/viewer/2022062501/56816231550346895dd26320/html5/thumbnails/7.jpg)
Man-Computer Symbiosis
Automation with human
post-correction
Supervisedmachinelearning
humans
computer
speedcostquality
computer
humans
speedcostquality
![Page 8: CrowdFlow Integrating Machine Learning with Mechanical Turk for Speed-Cost-Quality Flexibility](https://reader036.vdocument.in/reader036/viewer/2022062501/56816231550346895dd26320/html5/thumbnails/8.jpg)
Man-Computer Symbiosis
CrowdFlowAutomation with human
post-correction
Supervisedmachinelearning
humanscomputer
speedcostquality
humans
computer
speedcostquality
computer
humans
speedcostquality
![Page 9: CrowdFlow Integrating Machine Learning with Mechanical Turk for Speed-Cost-Quality Flexibility](https://reader036.vdocument.in/reader036/viewer/2022062501/56816231550346895dd26320/html5/thumbnails/9.jpg)
Mechanical Turk
![Page 10: CrowdFlow Integrating Machine Learning with Mechanical Turk for Speed-Cost-Quality Flexibility](https://reader036.vdocument.in/reader036/viewer/2022062501/56816231550346895dd26320/html5/thumbnails/10.jpg)
Human Detection – Starting point
![Page 11: CrowdFlow Integrating Machine Learning with Mechanical Turk for Speed-Cost-Quality Flexibility](https://reader036.vdocument.in/reader036/viewer/2022062501/56816231550346895dd26320/html5/thumbnails/11.jpg)
Human Detection – Task
![Page 12: CrowdFlow Integrating Machine Learning with Mechanical Turk for Speed-Cost-Quality Flexibility](https://reader036.vdocument.in/reader036/viewer/2022062501/56816231550346895dd26320/html5/thumbnails/12.jpg)
Human Detection – Results
Quality
Spee
d, A
fford
abili
ty
60% 90%
119 images took3 hrs 50 mins and cost $2.38
![Page 13: CrowdFlow Integrating Machine Learning with Mechanical Turk for Speed-Cost-Quality Flexibility](https://reader036.vdocument.in/reader036/viewer/2022062501/56816231550346895dd26320/html5/thumbnails/13.jpg)
Human Detection – Scenarios
Quality
Spee
d, A
fford
abili
ty
60% 90%
1000 photos at 72% accuracy would take 12 hrs 20 mins and cost $8.00
119 images took3 hrs 50 mins and cost $2.38
![Page 14: CrowdFlow Integrating Machine Learning with Mechanical Turk for Speed-Cost-Quality Flexibility](https://reader036.vdocument.in/reader036/viewer/2022062501/56816231550346895dd26320/html5/thumbnails/14.jpg)
Vision: Richer modelInput with computer results
Validator
Appraiser
Fixer Worker
Correct Incorrect
Fix Start over
Output
![Page 15: CrowdFlow Integrating Machine Learning with Mechanical Turk for Speed-Cost-Quality Flexibility](https://reader036.vdocument.in/reader036/viewer/2022062501/56816231550346895dd26320/html5/thumbnails/15.jpg)
Lessons Learned Design for overall needs/constraints
Practical advice: Pay consistently and reasonably Reject only work that is definitely cheating Build in fair cheating deterrence from the start Keep instructions short, but always clear
Contact: Alex Quinn [email protected]