crowdflower university oct. 21 2013

CrowdFlower University October 21, 2013

1:00 - 1:05 // Arrive, Plug-in

1:05 - 1:30 // Intro to Workforce Management and the CrowdFlower Platform

1:30 - 2:30 // Process Deconstruction & Job Creation

2:30 - 2:40 // Break

2:40 - 3:30 // Job Monitoring and Reports

3:30 - 4:00 // From the Field: Case Studies

4:00 - 4:10 // Break

4:10 - 4:30 // What’s to come: Product Roadmap

4:30 - 5:00 // Q&A

Schedule for the day

2 CrowdFlower, Inc. – Proprietary and Confidential

CrowdFlower University will teach you the CrowdFlower

Platform, empowering you to manage workforces online.

Purpose

3 CrowdFlower, Inc. – Proprietary and Confidential

4 CrowdFlower, Inc. – Proprietary and Confidential

Success Center

Success Center

5 CrowdFlower, Inc. – Proprietary and Confidential

success.crowdflower.com

6 CrowdFlower, Inc. – Proprietary and Confidential

Workforce Management

Geography

Performance Levels and Skills

Workflows

Test Questions

Channels

Workforce Management

7 CrowdFlower, Inc. – Proprietary and Confidential

8 CrowdFlower, Inc. – Proprietary and Confidential

Geography

9 CrowdFlower, Inc. – Proprietary and Confidential

Geography

10 CrowdFlower, Inc. – Proprietary and Confidential

Performance Levels and Skills

Skills and Performance Levels

11 CrowdFlower, Inc. – Proprietary and Confidential

The CrowdFlower Platform enables you to add

Skill requirements and Performance Levels to

your job. Enabling the Skilled Crowds will likely

increase the quality of your results and may

decrease the throughput of your judgments.

12 CrowdFlower, Inc. – Proprietary and Confidential

Workflows

13 CrowdFlower, Inc. – Proprietary and Confidential

Workflows 6RXUFH�

�,QSXW�3D\ORDG�

&RPSUHVV�WR���URZ�SHU�FRPSDQ\

9HULI\��)LQG�2IILFLDO�85/V)LQG�3DJH�1XPEHUV

)LQG�2IILFHU��'LUHFWRU�3DJHV

$GG�1HZ

([SDQG�WR���URZ�SHU�OHDGHU

/RFDWH�/HDGHUV�RQ�3DJHV��9HULI\�'HOHWH�

)LQG�&XUUHQW�7LWOHV��%LRV

$GG�1HZ

([SDQG�WR���URZ�SHU����OHDGHUV

/RFDWH�/HDGHUV�RQ�3DJHV��9HULI\�'HOHWH�

([SDQG�WR���URZ�SHU�OHDGHU

)LQG�&XUUHQW�7LWOHV��%LRV

)LQG�8QLYHUVLWLHV )LQG�$JH)LQG�*HQGHU

)LQG�0DMRU��'HJUHH�DQG�*UDGXDWLRQ�<HDU

)LQG�8QLYHUVLWLHV)LQG�$JH )LQG�*HQGHU

)LQG�0DMRU��'HJUHH�DQG�*UDGXDWLRQ�<HDU

)LQG�)XOO�1DPH )LQG�)XOO�1DPH

1DPH�3DUVLQJ /515��0DWFK�JHQGHU�EDVHG�RQ�ELR�NH\ZRUGV

/515��0DWFK�JHQGHU�EDVHG�RQ�ELR�NH\ZRUGV

HQG�QRGH

HQG�QRGH

PLVVLQJ�SGI PLVVLQJ�RIILFLDO�XUO

XQYHULILHG

XQYHULILHG

HQG�QRGH

QR�XQLYHUVLWLHV

HQG�QRGH

QR�XQLYHUVLWLHV

HQG�QRGH

QR�ELRV

HQG�QRGH

QR�ELRV

3DFNDJLQJ

3') 85/

)LQG�&HUWLILFDWLRQV )LQG�&HUWLILFDWLRQV

-RE

6FULSW

HQG�QRGH

6RXUFH��,QSXW�3D\ORDG�

.(<

/515

/515�7LWOH�1RUPDOL]DWLRQ

14 CrowdFlower, Inc. – Proprietary and Confidential

Test Questions

Test Questions

15 CrowdFlower, Inc. – Proprietary and Confidential

Test Questions are a powerful method for

measuring the quality of the contributors in your

job. By providing the answers to a small

percentage of units, you can tell how accurate

someone is, and automatically remove anyone

who doesn't meet your requirements.

16 CrowdFlower, Inc. – Proprietary and Confidential

Channels

Channels

17 CrowdFlower, Inc. – Proprietary and Confidential

CrowdFlower Terminology

18 CrowdFlower, Inc. – Proprietary and Confidential

•  Judgment - One (of many) answers given by an individual contributor

•  Unit - One row on a spreadsheet (One company, one officer)

•  Dataset - The entire set of data for a project

•  Task - A subset of data from the whole dataset

•  Workflow - Multiple jobs that strung together

•  Channel - A website in which a contributor can access a task

•  Test Questions - Training data created in the platform

•  Contributor - Person completing assignments online

CrowdFlower Terminology

19 CrowdFlower, Inc. – Proprietary and Confidential

Process Deconstruction and Job Creation

20 CrowdFlower, Inc. – Proprietary and Confidential

Let’s learn how to build a job!

Today we’re going to build and run a sentiment analysis job about CrowdFlower.

21 CrowdFlower, Inc. – Proprietary and Confidential

Process Deconstruction & Job Creation

22 CrowdFlower, Inc. – Proprietary and Confidential

Best Practices – Job Construction

•  Break down problem into smallest components

•  Leverage the Template Library

•  Use the Instructions Template

•  Help contributors find the data (provide links, images, etc)

•  Provide examples in the instructions

•  Leverage CML logic wherever possible

•  Use validations to control text inputs

•  Allow an "escape" option for multiple choice questions regarding information in the job or a link: "I don't know", "I couldn't find it", "the link is broken", etc.

Best Practices – Job Construction

23 CrowdFlower, Inc. – Proprietary and Confidential

24 CrowdFlower, Inc. – Proprietary and Confidential

Best Practices – Test Questions

•  Try to create an even distribution of answers

•  Enable random data for Test Questions

•  Copy units to source

•  Be as thorough in your Test Question reasons as possible – these will help train the crowd throughout the task

•  Create Test Questions out of edge case (subjective or tricky) units to train the crowd

•  You don’t need to answer every question

•  For subjective questions you can allow for multiple correct answers

25 CrowdFlower, Inc. – Proprietary and Confidential

Best Practices – Test Questions

26 CrowdFlower, Inc. – Proprietary and Confidential

Best Practices – Job Calibration

•  Set the Max Judgments Per IP at the number of units in an assignment multiplied by the amount of gold in the task

•  This ensures that a contributor will not see the same gold unit twice

•  Set variable judgments on trickier CML elements

•  Set threshold for trust of contributors

Best Practices – Job Calibration

27 CrowdFlower, Inc. – Proprietary and Confidential

Launch!

28 CrowdFlower, Inc. – Proprietary and Confidential

29 CrowdFlower, Inc. – Proprietary and Confidential

Task Monitoring and Reports

Task Monitoring

Interpreting Reports

Interpreting Results

*This session will teach you how to understand your results.

Task Monitoring and Reports

30 CrowdFlower, Inc. – Proprietary and Confidential

31 CrowdFlower, Inc. – Proprietary and Confidential

Task Monitoring

• Review Test Questions

• Monitor Contributors

Task Monitoring

32 CrowdFlower, Inc. – Proprietary and Confidential

33 CrowdFlower, Inc. – Proprietary and Confidential

Interpreting Reports

Interpreting Reports

34 CrowdFlower, Inc. – Proprietary and Confidential

35 CrowdFlower, Inc. – Proprietary and Confidential

Interpreting Results

• Confidence Scores

• Aggregation Methods

• Auditing for Accuracy

Interpreting Results

36 CrowdFlower, Inc. – Proprietary and Confidential

37 CrowdFlower, Inc. – Proprietary and Confidential

Case Studies

Case Studies

38 CrowdFlower, Inc. – Proprietary and Confidential

In this session we will cover several practical applications

of the CrowdFlower Platform.

39 CrowdFlower, Inc. – Proprietary and Confidential

Search Relevance Testing

40 CrowdFlower, Inc. – Proprietary and Confidential

Example 1

41 CrowdFlower, Inc. – Proprietary and Confidential

Example 2

42 CrowdFlower, Inc. – Proprietary and Confidential

Master Data Management

43 CrowdFlower, Inc. – Proprietary and Confidential

Find URL

44 CrowdFlower, Inc. – Proprietary and Confidential

Find About Us Page

45 CrowdFlower, Inc. – Proprietary and Confidential

Categorize Businesses

46 CrowdFlower, Inc. – Proprietary and Confidential

De-Duplication

47 CrowdFlower, Inc. – Proprietary and Confidential

Content Categorization

48 CrowdFlower, Inc. – Proprietary and Confidential

Photo Categorization

49 CrowdFlower, Inc. – Proprietary and Confidential

Product Categorization

50 CrowdFlower, Inc. – Proprietary and Confidential

Content Moderation

51 CrowdFlower, Inc. – Proprietary and Confidential

Photo Moderation

52 CrowdFlower, Inc. – Proprietary and Confidential

Text Moderation

53 CrowdFlower, Inc. – Proprietary and Confidential

Image and Text Interpretation

Image Transcription

54 CrowdFlower, Inc. – Proprietary and Confidential

OCR Verification

55 CrowdFlower, Inc. – Proprietary and Confidential

56 CrowdFlower, Inc. – Proprietary and Confidential

Product Roadmap

Automatic Test Question access for Enterprise customers

only

A wizard for first-time users so that your team needs less

training

A reputation score for our contributors and improvements

to our performance levels

Q4 2013

57 CrowdFlower, Inc. – Proprietary and Confidential

In-platform templates for you to get started quickly

Improvements to our GUI

Team collaboration on jobs

In-platform auditing tool so that your internal team and

our strongest contributors can easily evaluate the output

of your job

Q1 2014

58 CrowdFlower, Inc. – Proprietary and Confidential

New task analytics for improved slicing and dicing of your

data

Workflow building tool 

Minimize human input (e.g. variable pay where you set

the budget, we set the pay for contributors)

And beyond…

59 CrowdFlower, Inc. – Proprietary and Confidential

60 CrowdFlower, Inc. – Proprietary and Confidential

Questions?

Nathan Zukoff

Platform Program Manager

nathan@crowdflower.com

973-997-1921

Contact Me

61 CrowdFlower, Inc. – Proprietary and Confidential