10 lessons learned from Netflix Personalization - 10 Lessons Learned from Netflix

PERSONALIZATION 10 LESSONS LEARNED from NETFLIX SUCCESS (and how to apply them to your video services)

Personalization 10 Lessons Learned from Netflix’s Success (and how to apply them to video services)

Greetings! Kauser Kanji Pancrazio Auteri Managing Editor, VOD Professional CTO, Contentwise

Today’s objectives 2 1 Share the findings of observing how Netflix uses personalization as a competitive advantage in engaging and retaining users and planning content acquisition Show how you can connect the dots and take advantage of those findings for your online video services

So what is personalization?

ph. Thinkstock Not like this Personalized like this

It’s about the pleasure of choice and the abundance of options It’s understanding needs and narrowing the options to make choosing a pleasant experience

Personalization can go wrong

It’s not just high-tech Brian & Doreen remember customers’ taste, curate and organize shelves, plan promotions, update the display windows, listen to requests and recommend products and let customers browse their shop in Somerset, UK

Netflix Hulu Amazon Home page (web) Pure S-VOD S-VOD Prime Free, ad-based VOD Upsell S-VOD Hulu+ T-VOD

So what’s the problem? Broad range of user types and tastes Fragmentation of content sources and applications: too many places to look at Content availability can be very dynamic over time Most UI solutions rely on drill-down and user efforts User’s attention span and screen real estate are very limited You name it!

See the opportunity? Consumption increases Habits formation occurs Users feel more engaged Things can happen when people have a truly personalized experience Your service becomes a destination for unified discovery People talk about your brand with passion Subscribers perceive the value and the abundance of what you offer

Let’s start!

10 lessons learned from Netflix The information, interpretations, advice and recommendations contained in this presentation are not endorsed in any way by Netflix and are based on information publicly shared by Netflix or its employees.

10 lessons learned from Netflix 1. Set objectives and pick metrics

10 lessons learned from Netflix 1. Set objectives and pick metrics Netflix: maximize member satisfaction and month-to-month subscription retention Example Metrics

10 lessons learned from Netflix 1. Set objectives and pick metrics Example Metrics Canceled subscriptions per month Interactive sessions resulting in a playback Played minutes per user per month Fully watched playbacks Interaction time before starting a playback Returning users

10 lessons learned from Netflix 2. Consider UX as mission-critical

10 lessons learned from Netflix 2. Consider UX as mission-critical secret sauce best practices innovative functionalities well-tuned business rules UX Engine Users Content offer Audiovisual quality Delivery infrastructure Editorial curation Customer support

10 lessons learned from Netflix 2. Consider UX as mission-critical UX Engine = Code Users Rules Widgets Behaviors Configurations Management tools

10 lessons learned from Netflix 2. Consider UX as mission-critical Changes in UI behavior can have a dramatic impact on key metrics. Multiscreen? Make sure behavior is consistent across devices Pay special attention to cross-screen consistency of Welcome screens Frequent user actions User “lost” actions Leverage the UX Engine to control UI behavior across all screens

10 lessons learned from Netflix 3. Personalize UX as much as possible

10 lessons learned from Netflix 3. Personalize UX as much as possible At Netflix, more than 75% of views come from some sort of recommendation or personalized ranking Views % from personalized ranking source: Netflix 0 25 50 75 100

10 lessons learned from Netflix 3. Personalize UX as much as possible User’s attention span is very limited The first 8-12 seconds are critical Weinreich et al. - ACM 2008 Desired outcomes - find something to watch - engage in some sort of exploration

10 lessons learned from Netflix 3. Personalize UX as much as possible Screen real estate is very limited too Ideally user should find relevant content in the first screen

10 lessons learned from Netflix 3. Personalize UX as much as possible A Ineffective sort criteria Effective criteria B Alphabetic C By release year Personalized order By ingestion order D Canned categories Even “computed” lists such as E Most popular Most viewed F Recently added and My list!

10 lessons learned from Netflix 3. Personalize UX as much as possible There are two “folds” Netflix personalizes in both directions 1. ranking of items in a carousel 2. ranking of carousels in the layout 2 Real estate “above-the-fold” 1

10 lessons learned from Netflix 3. Personalize UX as much as possible Featured content Resume play + My list { Most likely actions Popular Top picks for you Recently added Main genres Pseudo-genres Because you watched… Friends watching… Watch again Displayed in a personalized order" " Some of them disappear for a while if never “touched” " or because of A/B Testing

10 lessons learned from Netflix 3. Personalize UX as much as possible Personalized reordering may be disorienting for some people Indeed some Netflix users complain about this; but it seems to be effective and we’ll show you how to handle it

Aha! My Aha! moment with NETFLIX

10 lessons learned from Netflix 4. Understand user’s lifestyle and context

10 lessons learned from Netflix 4. Understand user’s lifestyle and context Netflix mines usage data to extract behavior patterns Personalization may be affected by context elements Device type Time of the day Day of the week Season of the year User is at-home or out-of-home Geo-location (traveling, commuting, weekend-home…) Local weather Popular news Other users in close proximity (phones/wearables)

10 lessons learned from Netflix 5. Use interaction data then ask for feedback Priority on high-value usage events Playback start/stop/resume View asset details Add to personal list Other interactions Trick-play control Search Sharing Navigation paths… Ask for feedback 5-stars Like Dislike Love it!

10 lessons learned from Netflix 6. Let users know how the service is adapting to their tastes

10 lessons learned from Netflix 6. Let users know how the service is adapting to their tastes Promote trust in the system Encourage users to give feedback Better personalization

10 lessons learned from Netflix 6. Let users know how the service is adapting to their tastes Use meaningful labels referring to past behavior user can recognize Because you watched Breaking Bad Because of your interest for Time Travel Because you loved Kill Bill Vol.1

10 lessons learned from Netflix 7. Ensure metadata captures content nuances and is consistent

10 lessons learned from Netflix 7. Ensure metadata captures content nuances and is consistent Actors, Directors, Writers Genres Synopsis Release Year Duration Country Studio Language Characters Topics Themes Moods Locations Time Periods Keywords - Microtags “NETFLIX QUANTUM THEORY” A set of best practices for manual micro-tagging of video content Social acceptability of the lead character

10 lessons learned from Netflix 7. Ensure metadata captures content nuances and is consistent Let users search for content you don’t have

10 lessons learned from Netflix 7. Ensure metadata captures content nuances and is consistent With richer content metadata you can use analytics to understand content performance and drive content acquisition (or even original production) And add meaning to user profiles

Movie Iron Man 3 Data from Gracenote-TMS

TV Series Breaking Bad Data from Gracenote-TMS

10 lessons learned from Netflix 8. Give reasons to come back often Refresh catalog frequently - OR - Let the UX Engine do it for you (virtually)

10 lessons learned from Netflix 8. Give reasons to come back often Re-shuffle top items to periodically change the ones above-the-fold Items outside the first screen are still highly relevant for the user User perceives novelty and will be keen to return more often

10 lessons learned from Netflix 9. Run frequent UI experiments There is no “perfect way” and there are many types of users: experiments and adaptation seem to be the most effective ways Identify the UI elements on the path to the key goals Roll-out the variations and look at 2-5 metrics Run the experiments for two weeks or until statistical validity Design and plan experiments not to interfere with each other Experiments consume interaction events: make sure there is enough activity to feed all of the active variations

10 lessons learned from Netflix 10. Close the loop, base decisions upon data

10. Close the loop, base decisions upon data Netflix was the only network that said “We believe in you. We’ve run our data, and it tells us that our audience would watch this series. We don’t need you to do a pilot” Kevin Spacey, actor and producer Listen to Kevin saying this (video)

10 lessons learned from Netflix 10. Close the loop, base decisions upon data Netflix uses analytics to heavily influence the content acquisition policy Netflix proved to be agile and effective in rolling out variations and track several metrics across hundreds of client platforms Netflix team is very disciplined on reporting UI events. This enables full visibility in analytics and higher ROI Yes. At Netflix they go nuts for analytics! And they look to be right

10 Lessons from Netflix - Recap 1. Set objectives, pick metrics and share them with the team 2. Consider UX as mission-critical 3. Personalize UX as much as possible 4. Understand user’s lifestyle and context 5. Use interaction data then ask for feedback 6. Let users know your service is adapting to their tastes 7. Ensure metadata captures content nuances and is consistent 8. Give reasons to come back often 9. Run frequent UI experiments 10. Close the loop and base your decisions upon data

Netflix solutions are applicable (and applied) at… Netflix Other services may include S-VOD as well as Linear TV, DVR, Transactional VOD, Pay TV, Pay-per view, music videos, sports highlights, Advertising or User-generated Content… We need a way to turn these lessons into practice touching all the stakeholders in our projects

" UIDO A set of checklists to guide you while introducing personalization in your video service

What you deliver How you start User Experience Integrator Experience UX IX DX OX Developer Experience Operator Experience How you build it Tools to manage " UIDO

UX User Experience What you deliver

UX User Experience What you deliver Content types ✓ Movies Aggregates ✓ Collections ✓ Series ✓ Episodes ✓ Extras ✓ Music videos ✓ Playlists ✓ News ✓ Sports events ✓ Sports highlights ✓ Scheduled programs ✓ Channels ✓ _____________________ ✓ Seasons ✓ Channel bundles ✓ Movie bundles ✓ Sports Team bundles ✓ Sports League bundles ✓ __________________

UX User Experience What you deliver Key UX features ✓ Manually curated collections ✓ Search results ✓ Search suggestions while you type (single/multi-type) ✓ Search refine with smart filters (facets) ✓ Similar content ✓ Personalized picks for user ✓ Critics-based feed (Rotten Tomatoes, Metacritic…) ✓ Series you watch (with next-episode) ✓ VOD bookmarking (resume playback) ✓ User’s list ✓ Predictive browsing (surfacing folders) ✓ Personalized pseudo-genres

Reference UI " Showing most of the personalization use cases supported by ContentWise

UX User Experience What you deliver Key UX features (cont’d) ✓ Social graph (e.g. friends, followers) ✓ Sharing actions ✓ Content can be embedded ✓ Co-watching (blended profiles) ✓ Profile explanation with content metadata ✓ User can rate content (stars, like, dislike, love, etc.)

UX User Experience What you deliver For kids ✓ Parental ratings ✓ Kids mode ✓ Specialized metadata (e.g. Commonsense) ✓ Editorial curation ✓ Curation by parents ✓ Analytics for parents

UX User Experience What you deliver Content sources ✓ Linear schedule (line-ups) ✓ Start-over TV system ✓ VOD Catalog ✓ Local DVR ✓ Network DVR ✓ Reverse EPG (catch-up) ✓ ______________

UX User Experience What you deliver Device types ✓ Phone ✓ Tablet ✓ PC ✓ TV ✓ Watch Access models ✓ S-VOD ✓ T-VOD ✓ Ad-VOD ✓ Free-Linear ✓ Pay-Linear ✓ PPV Profile types ✓ Personal ✓ Household ✓ Main account powers ✓ Blended ✓ Personas templates ✓ Personal on device Access locations ✓ At-home, OOH ✓ On-net, off-net

UX User Experience What you deliver Entitlements ✓ S-VOD packages ✓ Rented movies ✓ Purchased movies ✓ Purchased seasons ✓ Purchased episodes ✓ Subscribed channels ✓ Subscribed bundles (e.g. Channel + S-VOD) ✓ ______________________

Explaining a recommendation Because you liked these other movies Affinity between the user’s taste and the recommended movie (using the tag structure) ContentWise Reference UI

OX Operator Experience How to manage

OX Operator Experience How to manage ✓ Managing UI Elements with UX Engine ✓ Creating and updating editorial lists ✓ Generating and curating pseudo-genres ✓ Accessing analytics ✓ Content planning using analytics ✓ Managing variations and experiments for A/B Testing ✓ Understanding the impact of business rules on key metrics

Personalized pseudo-genres INTENSE ACTION MOVIES mood genre type 2000s AUSTRALIAN THRILLER MOVIES release prod genre type year country AMERICAN DRAMA MOVIES STARRING TOM HANKS ContentWise Reference UI

The magic of richer metadata MOVIES FROM FEMALE DIRECTORS type person role MOVIES STARRING A ROCKSTAR type gender from semantic enrichment looking into actors person role from semantic enrichment ContentWise Reference UI

Curation of Pseudo-genres Metadata fields considered for labels Status of the pseudo-genre Type: Editorial or Computed ContentWise Management Console

Driving from the UX Engine Rendered by UI code Configured by UX Engine ContentWise Management Console

UX Engine - Personalized Order of Carousels

Content planning - Choosing items to retire Find movies with a small number of “estimated” residual views and are “expiring” Automatically create a business rule The rule can be used in A/B Testing to anticipate the impact of removing these movies from the catalog. ContentWise Management Console

A/B/C Testing Biz rule #1 Biz rule #N Variation A Biz rule #1 Biz rule #N Variation B Experiment Group A Group B Control Group Results Metrics Normal behavior

Launching an experiment This is an experiment Treated user base Variations

Monitoring experiment results Variations Affected users Metrics ContentWise Management Console

DX Developer Experience How to build

DX Developer Experience How to build UI Element ✓ Carousel ✓ Group of carousel ✓ Item ✓ Item attribute ✓ __________

DX Developer Experience How to build UI Events to be reported ✓ Item displayed ✓ Item selected ✓ Item details accessed ✓ User rating submitted ✓ Explanation displayed ✓ Tuned-in ✓ Tuned-out ✓ Playback started ✓ Playback resumed ✓ Playback paused ✓ Playback completed ✓ Playback stopped ✓ Item saved to list ✓ Item removed from list ✓ Preview playback started

IX Integrator Experience How to start

IX Integrator Experience How to start ✓ Content model map ✓ Event model map ✓ User ID map ✓ Data refresh policy ✓ Bulk ingestion automation ✓ Delta updates automation ✓ Client applications map ✓ UI elements to be managed from UX Engine

Thank you! For more information, please visit our website or contact us pancrazio.kauser.kanji@vodprofessional.com auteri@contentwise.tv Digital TV. Personalized www.vodprofessional.com www.contentwise.tv