Contenu connexe
Similaire à AWS Summit 2011: Amazon Mechanical Turk for Enterprises
Similaire à AWS Summit 2011: Amazon Mechanical Turk for Enterprises (20)
Plus de Amazon Web Services
Plus de Amazon Web Services (20)
AWS Summit 2011: Amazon Mechanical Turk for Enterprises
- 1. Introducing
Amazon
Mechanical
Tu r k
Presented by:
John Hoskins
Sr. Manager
June 10, 2011
- 2. Agenda
Introduction
A brief overview of Mechanical Turk
Use Cases & Case Studies
Practical applications of Mechanical Turk and examples of
how others have used it.
Getting Started
Options for implementing with Mechanical Turk
© 2011 Amazon.com, Inc. or its Affiliates.
- 3. Business Challenges
Content Moderation
Our community is generating 10,000 comments per day –
how can we eliminate SPAM?
Product Feedback
We just released a new product or service – how do we
judge how it’s being received?
Discoverability
We have thousands of legacy media assets – how do we
make them searchable?
© 2011 Amazon.com, Inc. or its Affiliates.
- 4. When to use Mechanical Turk?
Tasks that require human judgment:
Recognition
Reasoning
Context
Prior Knowledge
Algorithmic solutions are inefficient or impractical
Requires flexibility due to unpredictable fluctuations in
volume
© 2011 Amazon.com, Inc. or its Affiliates.
- 5. What is
Mechanical Turk?
© 2011 Amazon.com, Inc. or its Affiliates.
- 6. Mechanical Turk is a
marketplace for work.
Mechanical Turk gives businesses and developers access to
an on-demand, scalable workforce.
Flexibility: Scale your workforce up and down quickly
Accuracy: Get high quality, efficient and cost effective
results.
Price: Pay only when you are satisfied with the results.
Speed: Start receiving results in minutes
© 2011 Amazon.com, Inc. or its Affiliates.
- 7. Workforce
500,000Workers
190+ Countries
Who are the Workers?
Workers are global, available 24x7
Workers choose the work they want to do and work from
anywhere, anytime
Workers are dedicated to your tasks and can work in
parallel
Valuable source of feedback on how to improve your HITS
© 2011 Amazon.com, Inc. or its Affiliates.
- 8. Your Workers
Identify and manage your best Workers
• All available Workers
All Workers • 500,000 +
• 24x7, 365
• Any Worker who
Your completes a HIT
Workers • Evaluate performance
on your HITS
Your Trusted • Your best Workers
with proven quality
Workers and availability
© 2011 Amazon.com, Inc. or its Affiliates.
- 9. How are businesses using it?
Data Management Categorization
Data Verification Classification
Data Entry & Collection Tagging
Data De-duplication Sentiment Analysis
Algorithm Training
Content & Media Business Services
Moderate Photos & Content Search Relevancy
Content Creation & Editing Product Usability Testing
Transcription Research
© 2011 Amazon.com, Inc. or its Affiliates.
- 10. Search Enhancement / Relevance
Search discovery requires categorization and expansion
or classification of attributes – keywords that users can
use to find content that relates to them.
Adding meta data – such as detailed descriptions,
attribute tags and categories can improve discoverability.
How it works:
Item description is Content is more
Workers add Tags are added as
sent to Mechanical easily discovered
creative & attributes to the
Turk for review by through Web and
descriptive tags item
Workers Site search
© 2011 Amazon.com, Inc. or its Affiliates.
- 11. Moderation of User Generated
Content
The volume and speed at which user generated content
is being generated – and maintaining site guidelines is a
real problem faced by site publishers.
Filtering content for appropriateness and usefulness
requires review..
How it works:
Content is sent Web site
Workers review Inappropriate
to Mechanical maintains
according to content is
Turk for review community
rules deleted
by Workers expectation
© 2011 Amazon.com, Inc. or its Affiliates.
- 12. Data Handling
Resolve Data conflicts directly in your business process
Remove duplicates
Normalize data entries
Resolve conflicts and collisions
Enhance entries
How it works:
Items flagged
Business Workers review Business
for conflict are
Process Ingests item and Process
sent to
Data Elements resolve conflict continues
Mechanical Turk
© 2011 Amazon.com, Inc. or its Affiliates.
- 13. Advantages of Mechanical Turk
Gives you access to an on-demand workforce
Eliminates
100% Efficiency
Staffing Issues
Maintain the flexibility your business demands
Use only the capacity you need, when you need it
Converts a fixed cost into a variable cost
Lowers
Significantly reduces ongoing headcount expenses
Costs
Offers a pay-as-you-go staffing model
Eliminates lead time of interviewing and staffing
Reduces work
Work gets done faster, by working in parallel
turnaround time
Work continues around the globe, around the clock
© 2011 Amazon.com, Inc. or its Affiliates.
- 14. Getting Started
Easy, scalable options for organizations of all sizes
© 2011 Amazon.com, Inc. or its Affiliates.
- 15. How it works:.
“Validate,
“Design & Publish”
Pay & Go”
© 2011 Amazon.com, Inc. or its Affiliates.
- 16. Your Project Tasks
“Working Backwards”
What are the business results that you want?
How will you use those results?
What format do the results need to be in?
Which tasks benefits from human intelligence?
Break your overall project into smaller “tasks”
Consolidate like items and processes
Define instructions for each task
© 2011 Amazon.com, Inc. or its Affiliates.
- 17. Example Workflow
Content Moderation with API integration
© 2011 Amazon.com, Inc. or its Affiliates.
- 18. The Key components of your HIT
Instruction
Guidelines for how to
do a HIT
Data
“batch” concept
What kind of dress is this?
Question(s) Cocktail
Sun dress
Bridal dress
© 2011 Amazon.com, Inc. or its Affiliates.
- 19. HIT Design
Instructions
The guidelines you provide on how to do your HIT
Define what an acceptable answer looks like
Account for ambiguity
Ergonomics – get work done faster
Design for Worker and reviewer efficiency
Make sure all Workers do equal work for equal $
Defensive design – make sure your questions can be answered as
expected
Workflow
Don‟t try to make one HIT do all of the work
Split the process into logical stages
Maintain „Like Reward for Like Work‟
© 2011 Amazon.com, Inc. or its Affiliates.
- 20. Work Catalog
Workers
discover
available HITs
through the
listings catalog
Name of
Requester,
Name of HIT,
Price and
Qualifications
are displayed
© 2011 Amazon.com, Inc. or its Affiliates.
- 21. Adjudication Strategies
Leverage different methods of determining the correct
answer by designing for accuracy in your HIT:
Multiple Choice
Plurality
Known Answers (“Gold Standards”)
Free-form
Multiple HIT Workflow‟s where a new Worker “edits” or
“grades” previous Worker submissions
Creative Tasks
Bonus Payments
© 2011 Amazon.com, Inc. or its Affiliates.
- 22. Balances & Levers
Leverage market dynamics to optimize speed, price and
accuracy to meet your business needs.
Speed
Price
Accuracy
© 2011 Amazon.com, Inc. or its Affiliates.
- 23. Requester access:
Apps & Solution
Web UI API & CLT
Providers
• Easy Access • Programmatic • Experts at
• Create HITS in Access Workflow
HTML • Multiple SDKS design
• Upload and • Massive scale • They do the
manage data training and
via CSV management
of Workers for
specific kinds
of tasks
© 2011 Amazon.com, Inc. or its Affiliates.
- 24. Applications & Solution Providers
Mechanical Turk supports a robust ecosystem of providers
that have built or can build solutions to meet your needs.
© 2011 Amazon.com, Inc. or its Affiliates.
- 25. Thank you.
John Hoskins
hoskins@amazon.com
206.266.8105
@amazonmturk
Facebook.com/amazonmturk
mechanicalturk.typepad.com
© 2011 Amazon.com, Inc. or its Affiliates.