Artificial, Artificial Artificial, Artificial IntelligenceIntelligenceUsing Amazon Mechanical Turk and .NET to Using Amazon Mechanical Turk and .NET to Create a New Breed of Web AppCreate a New Breed of Web App
Jeff BarrJeff BarrWeb Services EvangelistWeb Services EvangelistAmazon Web ServicesAmazon Web Services
NGW012NGW012
AgendaAgenda
Amazon Web Services OverviewAmazon Web Services Overview
Artificial, Artificial IntelligenceArtificial, Artificial Intelligence
DemosDemos
Writing Mechanical Turk ApplicationsWriting Mechanical Turk Applications
Q&AQ&A
Introduction And Introduction And BackgroundBackground
Software development backgroundSoftware development background
Veteran of several startupsVeteran of several startups
Visual Studio team at MicrosoftVisual Studio team at Microsoft(DHTML, XML, Web Services) (DHTML, XML, Web Services)
3.5 Years with Amazon3.5 Years with Amazon
Amazon Web Services EvangelistAmazon Web Services Evangelist
What Is Amazon?What Is Amazon?
Online RetailerOnline RetailerOver 55 million active customer accountsOver 55 million active customer accounts
Seven countries: US, UK, Germany, Japan, France, Seven countries: US, UK, Germany, Japan, France, Canada, ChinaCanada, China
Technology ConsumerTechnology ConsumerMulti-National Web SitesMulti-National Web Sites
Vast Data Warehouse – 25 TBVast Data Warehouse – 25 TB
World-Class Logistics – 21 fulfillment centers; 9 million ft2World-Class Logistics – 21 fulfillment centers; 9 million ft2
Technology ProviderTechnology ProviderHundreds of thousands of Amazon AssociatesHundreds of thousands of Amazon Associates
Over 1,050,000 active seller accountsOver 1,050,000 active seller accounts
Over 150,000 software developers registered to use Over 150,000 software developers registered to use Amazon Web ServicesAmazon Web Services
Web Service ArchitectureWeb Service Architecture
Operating SystemOperating System
Libraries / DLLs / AssembliesLibraries / DLLs / Assemblies
ApplicationApplication
InternetInternet
Web ServicesWeb Services
ApplicationApplication
What Is Amazon Web What Is Amazon Web Services? Services?
APIs that give developers programmatic APIs that give developers programmatic access to Amazon’s data and technologyaccess to Amazon’s data and technology
Building-block web servicesBuilding-block web services
Web-scale infrastructureWeb-scale infrastructure
E-commerce capabilityE-commerce capability
Content, data, and informationContent, data, and information
New business modelsNew business models
Customer-created contentCustomer-created content
AWS Product FamilyAWS Product Family
Amazon E-Commerce ServiceAmazon E-Commerce ServiceComplete access to Amazon’s Complete access to Amazon’s product catalogproduct catalogFree + Associates Free + Associates commissions paidcommissions paid
Amazon Historical PricingAmazon Historical PricingData warehouse access for Data warehouse access for product pricingproduct pricingMonthly FeeMonthly Fee
Amazon Mechanical Turk Amazon Mechanical Turk Artificial Artificial IntelligenceArtificial Artificial Intelligence10% Commission 10% Commission Paid workforcePaid workforce
Amazon Simple Queue ServiceAmazon Simple Queue ServiceIT building blockIT building blockIn betaIn beta
Amazon S3Amazon S3Storage for the internetStorage for the internetCharge by storage/bandwidth Charge by storage/bandwidth usageusage
Alexa Web Information ServiceAlexa Web Information ServiceData warehouse access for web Data warehouse access for web crawl datacrawl data10K calls per month free, then 15 10K calls per month free, then 15 cents per 1000 callscents per 1000 calls
Alexa Web Search PlatformAlexa Web Search PlatformRoll your own search engineRoll your own search enginePay for time, storage, bandwidthPay for time, storage, bandwidth
Alexa Top SitesAlexa Top SitesTop sites by Alexa traffic rankTop sites by Alexa traffic rankCharges by URLCharges by URL
Amazon S3Amazon S3Simple storage serviceSimple storage service
Storage for the internet - web service to read Storage for the internet - web service to read and write dataand write data
15 cents per Gigabyte-Month to store data15 cents per Gigabyte-Month to store data
20 cents per Gigabyte to access data20 cents per Gigabyte to access data
Private and public storagePrivate and public storage
Scalable, reliable, cost-effective, and simple!Scalable, reliable, cost-effective, and simple!
Amazon Amazon Mechanical Mechanical
TurkTurk
Man Versus MachineMan Versus Machine
But Not ReallyBut Not Really
Chair Or Table?Chair Or Table?
Survey Says…Survey Says…
Amazon Mechanical TurkAmazon Mechanical Turk
Provides Web Services API for Provides Web Services API for computers to integrate artificial artificial computers to integrate artificial artificial
intelligence directly into their intelligence directly into their processing by making requests processing by making requests
of humansof humans
Put The Human In The LoopPut The Human In The Loop
while (read (photo))while (read (photo))
{{
ContainsHuman = CallMechanicalTurk(photo);ContainsHuman = CallMechanicalTurk(photo);
if (ContainsHuman)if (ContainsHuman)
{{ acceptPhoto(photo); acceptPhoto(photo); } } else else
{{ rejectPhoto(photo); rejectPhoto(photo); } }
} }
How It WorksHow It Workswww.mturk.com
WorkersArtificial, Artificially
Intelligent Software
Requester(Developer)
Human Intelligence Tasks (HITs)
Completed HITs
Worker Qualifications
What This MeansWhat This Means
For Software DevelopersFor Software DevelopersSolves problem of building applications that until now have not Solves problem of building applications that until now have not worked well without human intelligenceworked well without human intelligence
For BusinessesFor BusinessesEfficiently, effectively and inexpensively complete millions of Efficiently, effectively and inexpensively complete millions of tasks that require human intelligencetasks that require human intelligence
For For AnyoneAnyoneMake money using the skills you already have whenever your Make money using the skills you already have whenever your schedule allowsschedule allowsPreviously unimaginable business ideas are now feasible…Previously unimaginable business ideas are now feasible…
High Quality, Inexpensive, High Quality, Inexpensive, Massive Scale Image Clean-UpMassive Scale Image Clean-Up
Which picture best represents this Which picture best represents this business?business?
Easy-to-use application to build HITs Easy-to-use application to build HITs for Mturkfor Mturk
Requester-oriented interface:Requester-oriented interface: Create and issue a series of HITs (Human Create and issue a series of HITs (Human Intelligence Tasks)Intelligence Tasks)
There's no need to deal with source code or There's no need to deal with source code or with raw XML datawith raw XML data
Free Use of HIT-Builder Functions during Beta Test period Free Use of HIT-Builder Functions during Beta Test period (requires Amazon Requester account)(requires Amazon Requester account)
Building A HITBuilding A HIT
Hit-Hit-Builder.comBuilder.com
Building Applications Building Applications WithWith
Amazon Mechanical Amazon Mechanical TurkTurk
API StatusAPI Status
RequestersRequestersMust use APIsMust use APIs
Must have development skillsMust have development skills
Expect some small tweaks during the betaExpect some small tweaks during the beta
WorkersWorkersMust use mturk.comMust use mturk.com
Worker API in the futureWorker API in the future
API Basics - #1API Basics - #1
Use REST or SOAP, your choiceUse REST or SOAP, your choice
Be sure to check for IsValid in Be sure to check for IsValid in each responseeach response
URL encode Question and Answer URL encode Question and Answer XML dataXML data
Include proper namespace in XML dataInclude proper namespace in XML data
API Basics - #2API Basics - #2
Sign each request using your Sign each request using your Secret Key:Secret Key:
HMAC(Hashing Function)
Service
Operation
Timestamp
Secret Key
Base 64Encode
SOAPRequest
RESTRequest
URLEncode
System Design ProcessSystem Design Process
Design Great HITsDesign Great HITs
Identify Worker QualificationsIdentify Worker Qualifications
Determine approval strategyDetermine approval strategy
Build ApplicationBuild Application
Deposit funds into Amazon accountDeposit funds into Amazon account
Load QualificationsLoad Qualifications
Load HITsLoad HITs
Approve completed AssignmentsApprove completed Assignments
Retrieve resultsRetrieve results
Designing A Good HIT - #1Designing A Good HIT - #1
Question elementsQuestion elementsTitleTitle
Text BlockText Block
ListList
BinaryBinary
Answer elementsAnswer elementsFree TextFree Text
Selection (1 or multiple)Selection (1 or multiple)
Build XML using the Build XML using the Question BuilderQuestion Builder
Designing A Good HIT - #2Designing A Good HIT - #2
HIT design considerationsHIT design considerationsBe clearBe clearBe verbose, if necessaryBe verbose, if necessaryBe specific about needs & expectationsBe specific about needs & expectationsLeave no ambiguitiesLeave no ambiguitiesTest drive and monitor before going Test drive and monitor before going full-borefull-boreAlways have an appropriate “none of the Always have an appropriate “none of the above” choiceabove” choicePunctuation, capitalization, etc.Punctuation, capitalization, etc.Approval strategyApproval strategy
Identify Worker Identify Worker QualificationsQualifications
Create a test to admit only workers with the Create a test to admit only workers with the skills or other qualifications that are needed skills or other qualifications that are needed for your particular type of HITsfor your particular type of HITs
Qualification often looks very much like a HITQualification often looks very much like a HIT
Objective (multiple choice) or subjective Objective (multiple choice) or subjective (free text)(free text)
If necessary, use “Secret Code” If necessary, use “Secret Code” qualificationsqualifications
Use or build on existing QualificationsUse or build on existing Qualifications
Application FlowApplication Flow
LoadQualifications
Load HITs
AnyCompleted
Assignments?
Retrieve and Approve
Do LocalProcessing
More Work?
Y
YDone!
Do LocalProcessing
N
Load QualificationsLoad Qualifications
Relevant API CallsRelevant API CallsCreateQualificationTypeCreateQualificationType
GetQualificationRequestsGetQualificationRequests
GrantQualificationGrantQualification
NotesNotesQualifications which include an Answer Key are Qualifications which include an Answer Key are automatically granted when passedautomatically granted when passed
Application must explicitly check and grant others Application must explicitly check and grant others using GrantQualificationusing GrantQualification
You cannot currently attach a Qualification to a You cannot currently attach a Qualification to a worker without a prior requestworker without a prior request
Load HITsLoad HITs
Relevant API CallsRelevant API CallsCreateHIT - Returns HIT identifierCreateHIT - Returns HIT identifier
DisableHIT – Turn it offDisableHIT – Turn it off
ExtendHIT – Add additional timeExtendHIT – Add additional time
NotesNotesMultiple assignments per HIT are importantMultiple assignments per HIT are important
Auto approval timeAuto approval time
Qualification requirements can be attached Qualification requirements can be attached to HITsto HITs
Attach application data using Attach application data using RequesterAnnotationRequesterAnnotation
Poll And Approve WorkPoll And Approve Work
Relevant API CallsRelevant API CallsGetReviewableHITsGetReviewableHITs
GetHITsGetHITs
GetAssignmentsForHITGetAssignmentsForHIT
ApproveAssignment – Pay the WorkerApproveAssignment – Pay the Worker
RejectAssignment – Don’t pay the WorkerRejectAssignment – Don’t pay the Worker
DisposeHITDisposeHIT
ExtendHITExtendHIT
NotifyWorkers – Send e-mail notificationNotifyWorkers – Send e-mail notification
Retrieve ResultsRetrieve Results
Relevant API callsRelevant API callsGetAssignmentsForHIT – Get the GetAssignmentsForHIT – Get the actual answersactual answers
NotesNotesGetReviewableHITs returns list of non-GetReviewableHITs returns list of non-assignable HITsassignable HITs
Dispose or Extend (or they’ll keep Dispose or Extend (or they’ll keep coming back)coming back)
Perform Local ProcessingPerform Local Processing
Specific to your applicationSpecific to your application
TypesTypesIncremental (per Assignment)Incremental (per Assignment)
Batch (per group of related HITs)Batch (per group of related HITs)
Connect application logic back to TurkConnect application logic back to TurkIf application’s needs change, Dispose of If application’s needs change, Dispose of Associated HITAssociated HIT
Adjust pricing and timing dynamicallyAdjust pricing and timing dynamically
Adjust necessary QualificationsAdjust necessary Qualifications
Next Steps…Next Steps…
Find potential HITs:Find potential HITs:Within your organizationWithin your organization
Unmet or underserved marketUnmet or underserved market
Design workflow process around the Design workflow process around the Mechanical TurkMechanical Turk
Start writing code and loading HITsStart writing code and loading HITs
Process resultsProcess results
Get your work done!Get your work done!
Q & AQ & A
Thanks!Thanks!
© 2006 Microsoft Corporation. All rights reserved.This presentation is for informational purposes only. Microsoft makes no warranties, express or implied, in this summary.
For More Information:For More Information:
Mechanical Turk: Mechanical Turk: www.mturk.comwww.mturk.com
AWS Blog: AWS Blog: aws.typepad.comaws.typepad.com
Amazon Web Services: Amazon Web Services: aws.amazon.comaws.amazon.com
© 2006 Microsoft Corporation. All rights reserved.This presentation is for informational purposes only. Microsoft makes no warranties, express or implied, in this summary.