Ce diaporama a bien été signalé.
Nous utilisons votre profil LinkedIn et vos données d’activité pour vous proposer des publicités personnalisées et pertinentes. Vous pouvez changer vos préférences de publicités à tout moment.
Quantifying the Value of
Better Recommendations
The Value of Better Recommendations
Stakeholders:
■ Business value
■ Consumer value
■ Producer value
■ Cultural and Societ...
Neil Hunt
1999: Started work on recommendations at Netflix
Goal was to improve satisfaction, while solving the %New proble...
Context
Technology Eliminates
Constraints on Personal Choice
Constraints on Personal Choice Falling Away
■ Geographic “trade radiu...
Recommenders Enable Long Tail Media
■ There are no bad shows,
just shows with small audiences
■ It’s our job to find and m...
Linear TV Channel - one choice available
■ Only watch what’s being broadcast
■ 21 hours/week of prime time - nothing else ...
No More Commercials
… or only relevant and interesting ads
(chosen by recommenders…)
Richer Storytelling
Freed of the constraints of linear TV,
not all shows must be 42 minutes with a cliffhanger end
Discove...
Business Objectives
Why Do Businesses Invest in Recommenders?
Better Economics…
■ Makes a traditional business better, or
(Netflix, Amazon, Sp...
Why Do Businesses Invest in Recommenders?
The Tension:
■ Enhancing customer satisfaction
■ Better choices
■ Shorter time t...
Netflix Choices
All our content is licensed to a fixed fee:
Each possible choice has same cost impact
We don’t sell advert...
Quantifying Netflix Benefits
7B hours per quarter
50M subscribers worldwide
90 minutes/day average
150M choices/day
Quantifying Netflix Benefits
A good choice leads to a complete viewing
A poor choice leads to abandonment, and risk of can...
Our Business Metrics
Business Value
New Trials Retention
Hours of ViewingWord of Mouth
Retention is a Blunt Measuring Instrument
Retention is a Blunt Measuring Instrument
Measuring Users-at-Threshold
Hours of Viewing
Frequency
Medians Averages
Baseline
Measuring Users-at-Threshold
Hours of Viewing
Frequency
Medians Averages
Baseline Higher Avg
Measuring Users-at-Threshold
Hours of Viewing
Frequency
Medians Averages
BaselineHigher Median Higher Avg
But We’re Still Measuring the Wrong Thing...
We optimize hours of viewing…
But all hours are not created equal
Implication...
What if the Retention Driver is Something
Else?
Avoiding Failed Sessions (user found nothing to watch)
Reducing Time-to-Pl...
Consumer Objectives
What Consumers Say...
“I don’t need suggestions, just show me the good stuff”
“Don’t hide anything - I want to evaluate it...
Winning The Moment of Truth
■ Moment of truth
■ 1-2 minutes to find something
■ 20-50 chances to connect
■ Or the user has...
The Intrinsic/Social Spectrum
Oracle vs. Advisor
✗ ✓
Content Producer Implications
The Cliff of Conventional Media
Producers must aim for broad
audience or be irrelevant
Target Audience
Consumption
Too sma...
Recommender Systems Level The Cliff
Economics of high-end
producers is less exponential
Producers can target the
audience ...
Recommender Systems Level The Cliff
Long-tail producers aren’t excluded
Much greater cultural diversity is enabled
Does Data Drive the Product?
■
■
■
■
■
■
Netflix Use of Data for Content
✓ Predict reach & hours for a project given what we know
? Give insight to choice of cast,...
Cultural and Societal Implications
Democratization of Media
The cultural implication of the media cliff is lack of access
to less prominent voices, channels,...
The Cultural Exception
■ Marketing economics drives large commercial culture
to displace local, regional, niche culture
■ ...
Filter Bubbles and Echo Chambers
Proposition:
■ Recommendation systems reinforce existing taste,
don’t expose users to the...
Final Thoughts
We are just scratching the surface of what’s possible
We are just scratching the surface of what’s possible
We depend upon our users trusting us with their data
-- they might l...
We are just scratching the surface of what’s possible
We depend upon our users trusting us with their data
-- they might l...
We are just scratching the surface of what’s possible
We depend upon our users trusting us with their data
-- they might l...
Prochain SlideShare
Chargement dans…5
×

Recsys 2014 Keynote: The Value of Better Recommendations - For Businesses, Consumers, Producers, and Society

4 579 vues

Publié le

A keynote at RecSys 2014: The Value of Better Recommendations - For Business, Consumer, Producer, and Society. A story, told from the Netflix perspective, of Internet TV and how recommendations systems enable the long tail, improve economics, and spread a global culture, with thoughts on objective metrics, measurement techniques, AB testing.

Publié dans : Business

Recsys 2014 Keynote: The Value of Better Recommendations - For Businesses, Consumers, Producers, and Society

  1. 1. Quantifying the Value of Better Recommendations
  2. 2. The Value of Better Recommendations Stakeholders: ■ Business value ■ Consumer value ■ Producer value ■ Cultural and Societal value Recommenders have power for great benefit, but also for harm Use your power wisely!
  3. 3. Neil Hunt 1999: Started work on recommendations at Netflix Goal was to improve satisfaction, while solving the %New problem 2006-2009: Netflix Prize Public recognition of the importance of recommenders 2007-2011: Transition to streaming Complete catalog, short supply → curated catalog, unlimited supply 2014: 300 people working on “content discovery” $150M investment
  4. 4. Context
  5. 5. Technology Eliminates Constraints on Personal Choice Constraints on Personal Choice Falling Away ■ Geographic “trade radius” ■ Production / manufacturing ■ Shelf-space ■ Channel (TV, radio) Recommenders allow access to the long-tail of choices: ■ Discovery ■ Evaluation
  6. 6. Recommenders Enable Long Tail Media ■ There are no bad shows, just shows with small audiences ■ It’s our job to find and motivate exactly the right audience
  7. 7. Linear TV Channel - one choice available ■ Only watch what’s being broadcast ■ 21 hours/week of prime time - nothing else matters On-demand - unlimited catalog accessible instantly ■ Paradox of choice: 1000s of possibilities, most not interesting Need a custom channel for each user (50M channels): ■ 20-50 personalized choices Netflix - TV of the Future - 50M Channels
  8. 8. No More Commercials … or only relevant and interesting ads (chosen by recommenders…)
  9. 9. Richer Storytelling Freed of the constraints of linear TV, not all shows must be 42 minutes with a cliffhanger end Discovery from outside a channel grid liberates the format The same was true for novels in Dicken’s time: Pickwick Papers was published in 20 weekly magazines with 32 pages of text (a 4-fold broadsheet) and 16 pages of advertising support They too, were liberated by advances in technology - making books possible
  10. 10. Business Objectives
  11. 11. Why Do Businesses Invest in Recommenders? Better Economics… ■ Makes a traditional business better, or (Netflix, Amazon, Spotify, Pandora, ...) ■ Enables new businesses not possible before (LinkedIn, GoogleNews, Instagram, Waze, Pinterest, any free service with ads, …)
  12. 12. Why Do Businesses Invest in Recommenders? The Tension: ■ Enhancing customer satisfaction ■ Better choices ■ Shorter time to choose ■ Suggesting more profitable products ■ Choices with better margins ■ Advertising ■ Long-term vs. Short-term tradeoff?
  13. 13. Netflix Choices All our content is licensed to a fixed fee: Each possible choice has same cost impact We don’t sell advertising on our service. Never will. We don’t sell our recs or data to third parties in any form. For Netflix, it’s all about customer satisfaction
  14. 14. Quantifying Netflix Benefits 7B hours per quarter 50M subscribers worldwide 90 minutes/day average 150M choices/day
  15. 15. Quantifying Netflix Benefits A good choice leads to a complete viewing A poor choice leads to abandonment, and risk of cancel 10% “better” choices → +500M/month good outcomes If 1% of those avoids a cancellation → $500M/year Our measurement thresholds: 0.1% retention improvement ($5..50M/year) 0.1% more viewing per time period
  16. 16. Our Business Metrics Business Value New Trials Retention Hours of ViewingWord of Mouth
  17. 17. Retention is a Blunt Measuring Instrument
  18. 18. Retention is a Blunt Measuring Instrument
  19. 19. Measuring Users-at-Threshold Hours of Viewing Frequency Medians Averages Baseline
  20. 20. Measuring Users-at-Threshold Hours of Viewing Frequency Medians Averages Baseline Higher Avg
  21. 21. Measuring Users-at-Threshold Hours of Viewing Frequency Medians Averages BaselineHigher Median Higher Avg
  22. 22. But We’re Still Measuring the Wrong Thing... We optimize hours of viewing… But all hours are not created equal Implication: ■ We machine-learn addictive over compelling ■ Partly innoculated by also measuring retention What signal can we find for valued hours?
  23. 23. What if the Retention Driver is Something Else? Avoiding Failed Sessions (user found nothing to watch) Reducing Time-to-Play Maximizing fraction-of-content-viewed Maximizing velocity of episode consumption
  24. 24. Consumer Objectives
  25. 25. What Consumers Say... “I don’t need suggestions, just show me the good stuff” “Don’t hide anything - I want to evaluate it all”
  26. 26. Winning The Moment of Truth ■ Moment of truth ■ 1-2 minutes to find something ■ 20-50 chances to connect ■ Or the user has moved on...
  27. 27. The Intrinsic/Social Spectrum
  28. 28. Oracle vs. Advisor ✗ ✓
  29. 29. Content Producer Implications
  30. 30. The Cliff of Conventional Media Producers must aim for broad audience or be irrelevant Target Audience Consumption Too small No-one knows No-one cares Consum ption m atches target audience
  31. 31. Recommender Systems Level The Cliff Economics of high-end producers is less exponential Producers can target the audience of their choice New producers with niche product can emerge Target Audience Consumption Even small audiences can be engaged Consum ption m atches target audience
  32. 32. Recommender Systems Level The Cliff Long-tail producers aren’t excluded Much greater cultural diversity is enabled
  33. 33. Does Data Drive the Product? ■ ■ ■ ■ ■ ■
  34. 34. Netflix Use of Data for Content ✓ Predict reach & hours for a project given what we know ? Give insight to choice of cast, location, etc. if requested ✗ DO NOT dictate “she has to die at the end of S2-E1” The director’s choices matter!
  35. 35. Cultural and Societal Implications
  36. 36. Democratization of Media The cultural implication of the media cliff is lack of access to less prominent voices, channels, products Recommendations systems can provide the market. Producers are stepping in to fill that niche
  37. 37. The Cultural Exception ■ Marketing economics drives large commercial culture to displace local, regional, niche culture ■ France: l’exception culturelle under GATT ■ Canada: cultural exemption under NAFTA ■ Recommendation systems can reduce the swamping effect of large commercial culture ■ More to gain by exporting French culture to the world than by limiting import of global culture to France Protectionism can yield to multiculturalism
  38. 38. Filter Bubbles and Echo Chambers Proposition: ■ Recommendation systems reinforce existing taste, don’t expose users to the new, unexpected or different If this keeps users happy, it’s likely to be true Our experience is that diversity and serendipity play a large role in delivering recommendations that win ?
  39. 39. Final Thoughts
  40. 40. We are just scratching the surface of what’s possible
  41. 41. We are just scratching the surface of what’s possible We depend upon our users trusting us with their data -- they might lose that trust
  42. 42. We are just scratching the surface of what’s possible We depend upon our users trusting us with their data -- they might lose that trust We have the ability to do amazing things for culture or distort it horribly by following a false north-star
  43. 43. We are just scratching the surface of what’s possible We depend upon our users trusting us with their data -- they might lose that trust We have the ability to do amazing things for culture or distort it horribly by following a false north-star Be creative, but humble, and amaze the world!

×