amazon mechanical turk - courses · pdf fileservice overview based on human computation...
TRANSCRIPT
AMAZON MECHANICAL TURK
JOAN ROIG-ARDERIU
AGENDA
HUMAN COMPUTATION
SERVICE OVERVIEW
THE NAME: “TURK”
HISTORY
TERMINOLOGY
HOW DOES IT WORK? (DEMONSTRATION)
WHO USES THIS?
CONCLUSIONS
HUMAN COMPUTATION
CERTAIN TASKS ARE BETTER PERFORMED BY HUMANS THAN BY COMPUTERS (I.E. PICTURE TAGGING)
INVERTED THINKING: INSTEAD OF CALLING A COMPUTER SERVICE TO PERFORM AN ACTION, THE COMPUTER CALLS A HUMAN.
WHAT IF WE COULD COORDINATE MANY HUMANS TO PERFORM A CERTAIN TASK?
SERVICE OVERVIEW
BASED ON HUMAN COMPUTATION CONCEPT: HUMANS OUTPERFORM COMPUTERS IN MANY TASKS
AMAZON MECHANICAL TURK PROVIDES AN “ON-DEMAND, SCALABLE, HUMAN WORKFORCE TO COMPLETE JOBS THAT HUMANS CAN DO BETTER THAN COMPUTERS”.
DEVELOPERS NEEDING TASKS THAT REQUIRE HUMAN INTELLIGENCE CAN USE MECHANICAL TURK AND INTEGRATE THE RESULTS WITH THEIR SOFTWARE (WEB) APPLICATIONS
THE NAME: “TURK”
IN THE 18TH CENTURY, W. VON KEMPELEN CREATED A CHESS-PLAYING AUTOMATON CALLED “THE TURK”
“THE TURK” TRAVELED AROUND EUROPE, BEATING GOOD CHESS PLAYERS
IT WAS REVEALED THAT “THE TURK” WAS NOT AN AUTOMATON, BUT A BOX WITH A CHESS MASTER HIDDEN INSIDE
THE TURKTHE MECHANICAL TURK: FAKE CHESS-PLAYING MACHINE
HISTORY
INITIALLY CREATED FOR SELF-USAGE BY AMAZON, TO SOLVE PROBLEMS THAT REQUIRED HUMANS: FIND PAGES THAT SHOW THE SAME PRODUCT
AMAZON REALIZED THIS WAS A UNIQUE SERVICE
LAUNCHED TO GENERAL PUBLIC IN 2005
IN MARCH 2007, IT WAS REPORTED THAT THERE WERE MORE THAN 100,000 WORKERS IN OVER 100 COUNTRIES
TERMINOLOGY: HIT
ANY TASK THAT CAN BE COMPLETED BY USING A COMPUTER CONNECTED TO INTERNET
DEVELOPERS (OR REQUESTERS) PUBLISH HITS, AND WORKERS WORK ON THEM
EXAMPLES
SPAM IDENTIFICATION
IMAGE TAGGING
TERMINOLOGY: HIT
ANY TASK THAT CAN BE COMPLETED BY USING A COMPUTER CONNECTED TO INTERNET
DEVELOPERS (OR REQUESTERS) PUBLISH HITS, AND WORKERS WORK ON THEM
EXAMPLES
SPAM IDENTIFICATION
IMAGE TAGGING
EXAMPLE OF EXCLUSIVELY HUMAN TASK:CAPTCHA
TERMINOLOGY: REQUESTER
THE INDIVIDUAL THAT CREATES THE HIT
MUST HAVE MONEY IN AN AMAZON ACCOUNT
CAN REQUEST QUALIFICATIONS FROM THE WORKERS
CAN SPECIFY HOW MANY WORKERS CAN WORK ON A TASK
CAN APPROVE OR REJECT THE RESULTS
CAN TEST HITS IN A “SANDBOX” : PLATFORM THAT ALLOWS REQUESTERS TO TEST THEIR HITS WITHOUT PAYING
TERMINOLOGY: QUALIFICATION
THE REQUESTER CAN POST A TEST THAT THE WORKERS MUST PASS IN ORDER TO WORK ON A PARTICULAR HIT
QUALIFICATIONS ALLOW REQUESTERS TO SELECT COMPETENT WORKERS
TERMINOLOGY: WORKER
THE INDIVIDUAL THAT PERFORMS THE TASKS SUBMITTED BY A REQUESTER
MUST COMPLETE QUALIFICATION TEST FOR SOME TASKS
WORKER CAN PREVIEW THE HIT AND ACCEPT IT OR SKIP IT. ONCE ACCEPTED, THE HIT CAN BE SUBMITTED OR REJECTED
HOW DOES IT WORK?
REQUESTER CAN MANAGE HITS BY:
WEB INTERFACE
COMMAND LINE TOOL
SDK OR USING THE API (SUPPORTS SOAP AND REST)
WORKER CAN LOOK FOR HITS ON THE WEBSITE
LET’S DO IT!
USING THE WEB INTERFACE OF AMAZON TURK.....
AS A REQUESTER: WE ARE GOING TO CREATE A HIT THAT ASKS WHAT KIND OF EMOTIONS A PICTURE PRODUCES.
2 WORKERS PER HIT
PAY $0.03 PER HIT COMPLETED
AS A WORKER: FIND A JOB, AND MAKE SOME MONEY...
WHO USES THIS?
AMAZON TURK:
EXAMPLE OF CLOTHING COMPANY THAT USES MECHANICAL TURK TO CATEGORIZE THEIR CLOTHES: HTTP://WWW.STYLIGHT.DE/
SORTFOLIO USES MECHANICAL TURK TO PREVENT NSFW IMAGES: HTTP://SORTFOLIO.COM/
HUMAN COMPUTATION:
GWAP (GAMES WITH A PURPOSE) HTTP://WWW.GWAP.COM/
CONCLUSIONS
NEW WAY OF THINKING ABOUT GETTING WORK DONE
FAST: TASKS ARE DONE IN PARALLEL (24 X 7)
ACCURATE: MONITOR AND MANAGE QUALITY OF WORK
COST EFFECTIVE: BIG SCALE WORK DONE QUICKLY. REQUESTER CAN REVIEW RESULTS BEFORE PAYING
CONS:
PAYOUTS ARE LOW
MISUSED : I.E. WRITE GOOD REVIEWS ON PRODUCTS
REFERENCES
HTTPS://WWW.MTURK.COM/MTURK/WELCOME
HTTP://BEHIND-THE-ENEMY-LINES.BLOGSPOT.COM/SEARCH/LABEL/MECHANICAL%20TURK
HTTP://WWW.CS.CMU.EDU/~BIGLOU/RESEARCH.HTML
HTTP://VIDEO.GOOGLE.COM/VIDEOPLAY?DOCID=-8246463980976635143#
HTTP://AWS.AMAZON.COM/DOCUMENTATION/MTURK/
HTTP://WWW.NYTIMES.COM/2007/03/25/BUSINESS/YOURMONEY/25STREAM.HTML?_R=1
GRÀCIES! ~ THANK YOU! ~