Crowdsourcing applications in business world. Netflix prize and other examples. Team D9 Boston University School of Management MBA '2014 Gian Calvesbert, Kobi Magnezi, Will Reid, Sarah Rubinton, Abhishek Sinha

1. Crowdsourcing the Lab:
Learning from “The Netflix Prize”
Team D9
Gian Calvesbert
Kobi Magnezi
Will Reid
Sarah Rubinton
Abhishek Sinha

3. Agenda
Innovation
Crowdsourcing
Netflix Prize
Analysis of Process and Results
Updates
Key Learnings and Takeaways

4. The traditional approach for
problem solving
Many expensive R&D processes
eventually fail.
Success rates are low, and the cost of
failure is high.

5. The Crowdsourcing approach for
problem solving
“The act of taking a job traditionally performed by a
designated agent (usually an employee) and outsourcing it
to an undefined, generally large group of people in the
form of an open call.” -Jeff Howe ,Wired

6. Why yes
•Less expensive
•Larger and broader knowledge
base
•Greater potential for
creativity, “outside the box”
thinking
•Fun, attractive for more casual
contributors
Why not
•Potential to expose strategic or
product weaknesses to competitors
•Complicates intellectual property
issues
•Contributors may not be aligned
behind larger goals

7. Two Models for Crowdsourcing
Innovation
Third Party Platforms
 Manage relationship
between solvers and
searchers
 Provide hub for potential
solvers with a variety of
problems
 Build a community and
network
 Protect anonymity
Internally Run Contests
 Control over all aspects of
contest rules and entry
capability
 Immediate access to all
solutions
 “Test as you go” process
 Control over collaboration

8. The Netflix Prize
Start 1%: $50K
10%:
$1M
Announced
October
2, 2006
Provided the
public with a
real data set of
1.4M+ unique
records
Netflix blog:
updated
contest leaders
and online
collaborative
environment
for solvers

9. The Netflix Prize: Results
Several leading teams
combined over 800
algorithms, developing
a solution that beat
the Netflix algorithm
by 10.05%
Collaboration between
teams allowed shared-
resources to create a
more powerful
Algorithm
Lots of publicity over
success of
crowdsourcing and the
“future” of science
“A $1 Million Research Bargain for Netflix, and
Maybe a Model for Others”
“The Netflix Prize Was Brilliant -- Google
and Microsoft should steal the idea”

10. Would you tell
me, please, which
way I ought to go
from here?
That depends a good deal on
where you want to get to
I don't much care where.
Then it doesn't much
matter which way you
go.

11. Success?
As of April 16th, still had
not implemented the
10.05% improvement
solution, over 5 years
from the announcement
date
Based grading criteria on
Root Mean Squared Error
Media backlash over
“Success”

12. Intellectual Property
 Ownership of IP for the new algorithm stayed with the
solvers
 Winning team from: AT&T, Commendo Research and
Consulting, and individual contributors
 Netflix gained a non-exclusive license to the solutions
 Encouraged more solutions, from employees of other
companies

13. The Fallout
2006
2009
“additional accuracy gains that
we measured did not seem to
justify the engineering effort
needed to bring them into a
production environment.”
Netflix business primarily
based on through the mail
DVDs
Market shift to VOD:
Much larger, more
complex datasets and
analytics

14. Why did Netflix fail?
Implementation
team not
involved in
winner
selection
Contest
Period
too long
Solution
not aligned
with goals

15. Crowdsourcingworkbest
• Everyone can interpret the facts
in their own way
Diversity of
Opinion
• Opinions of one actor are not
influenced by the opinions of
others
Independence
• Different specializationsDecentralization
• Method for successfully making a
decisionAggregation

16.  10 $10,000 awards for various cloud service applications
 Best feature
 Best Code Contribution
 Runs March-September, 2013
 Smaller overall awards for smaller projects
 “Quick hit”, easy to implement contributions

17. Problem Definition
Knowledge Selection
Experimentation Cycles
Determining Value

18. Problem Definition
Knowledge Selection
Experimentation Cycles
Determining Value

19. Learning from Past Mistakes
Key Improvements
• Shorter timeframe
reduces risk of major
technological change
• Smaller prize and
shorter time serve to
limit scope of work
• Nomination process
screens submissions
Remaining Concerns
• Unclear goal of
innovation
• Unmanaged
collaboration
environment
• Licensing model
denies Netflix full
access to intellectual
property they paid to
develop

20. The crowd doesn’t solve all
problems, but appropriate and
intentional use of crowdsourcing can
produce results for companies looking
to innovate.