RecSys 2015 - Unifying the Problem of Search and Recommendations at OpenTable
•Télécharger en tant que PPTX, PDF•
5 j'aime•1,044 vues
When we talk about search and recommendations, we frequently talk about them as polar opposites. In this talk, I will discuss the different technology components we employ when thinking about Search and Recommendations at OpenTable. We will describe how we can use the same technology stack to power the many different modes of decision-making users have, ranging from helping people locate a restaurant they have in mind, to browsing based on a general set of criteria, to suggesting ideas for what might be best for a given context.
Recommandé
Recommandé
Contenu connexe
Similaire à RecSys 2015 - Unifying the Problem of Search and Recommendations at OpenTable
Similaire à RecSys 2015 - Unifying the Problem of Search and Recommendations at OpenTable (20)
Dernier
Dernier (20)
RecSys 2015 - Unifying the Problem of Search and Recommendations at OpenTable
- 1. Unifying the Problem of Search and Recommendations at OpenTable Jeremy Schiff, Ph.D. RecSys 2015 09/20/2015
- 3. BEFORE DURING AFTER DINERSRESTAURANTS Understanding & Evolving Attracting & Planning OpenTable: Deliver great experiences at every step, based on who you are Proprietary 3
- 4. OpenTable in Numbers • Our network connects diners with more than 32,000 restaurants worldwide. • Our diners have spent approximately $35 billion at our partner restaurants. • OpenTable seats more than 17 million diners each month. • Every month, OpenTable diners write more than 475,000 restaurant reviews 4
- 5. OpenTable Data Ecosystem Search (Context & Intent) Restaurant Profile (Decision Confidence) Reservation History (Verifying the Loop) Reviews (Verifying the Loop) User’s Location Search Location Date, Time Query Reviews, Ratings (Overall, Food, Noise Level, etc) Seating Logs Photos, Reviews, Ratings, Menus
- 6. OpenTable Data Ecosystem Search (Context & Intent) Restaurant Profile (Decision Confidence) Reservation History (Verifying the Loop) Reviews (Verifying the Loop) User’s Location Search Location Date, Time Query Reviews, Ratings (Overall, Food, Noise Level, etc) Seating Logs Photos, Reviews, Ratings, Menus User Interaction Logs
- 7. So what are recommendations?
- 8. So what are recommendations?
- 9. What’s the Goal Minimizing Engineering Time to Improve The Metric that Matters • Make it Easy to Measure • Make it Easy to Iterate • Reduce Iteration Cycle Times 9
- 10. Pick Your Business Metric Revenue, Conversions • OpenTable • Amazon Retention, Engagement • Netflix • Pandora • Spotify 10
- 11. Importance of A/B Testing • If you don’t measure it, you can’t improve it • Metrics Drive Behavior • Continued Forward Progress 11
- 12. The Optimization Loops Introspect Offline Learning Online Learning Hours Days Weeks 12
- 13. The ingredients of a spectacular dining experience… 13
- 14. … and a spectacularly bad one 14
- 15. Examples of Topics (using MF) 15
- 16. Edit via the Header & Footer menu in PowerPoint 1616
- 18. Query Logs • Effective mechanism for understanding what users are trying to do • Reducing 0 result queries - Anecdote: should we support zipcodes next?
- 19. Search to Recommendations Continuum • Common Themes - Ranking always tries to move key metric (like conversion) - Always leverage implicit signals (time of day, day of week, location, etc) - User Control vs. Paradox of Choice Advantage Example Stage Item Count Search User Control $$, French, Takes Credit Card Retrieval Many Browse Use Case Control Great View / Romantic Ranking Many Recommend Data-Driven Flexibility Best around me Ranking Few
- 20. Differences in Recommender Usage Right now vs. Planning Cost of Being Wrong Search vs. Recommendations 20
- 21. Search vs. Recommendations Collaborative Filtering Models • Personalized • Without Context Search • Leverage Context • Using CF as One of Many Inputs
- 22. Search & Recommendation Stack Query Interpretation Retrieval Ranking – Item & Explanation Index Building Context for Query & User Model Building Explanation Content Visualization Collaborative Filters Item / User Metadata 22
- 23. Using Context, Frequency & Sentiment • Context - Implicit: Location, Time, Mobile/Web - Explicit: Query • High End Restaurant for Dinner - Low Frequency, High Sentiment • Fast, Mediocre Sushi for Lunch - High Frequency, Moderate Sentiment 23
- 24. Offline Models with Limited Data • Minimize Confusing User Experience • Little to No Data - Heuristics Encoding Product Expectations • Eg: Romantic Dates are not $. Sushi is not good for Breakfast • Limited Data - Data-Informed Eg: Analyze what Cuisines Users Click on when they Query for Lunch
- 25. Offline Models with Significant Data • Compensate for Sparseness • As Signals Improve, Popular -> Personalized • OpenTable Example - Context: User Location, Searched Location, Query, etc. • Learning to Rank - E [ Revenue | Query, Position, Item, User ] - E [ Engagement | Query, Position, Item, User ] - Regression, RankSVM, LambdaMart…
- 26. The Metric Gap 26 Training Test Training Error Generalization Error RMSE Precision @ K Stage Example
- 27. The Metric Gap 27 Training Error Generalization Error RMSE Precision @ K Stage Example Generalization Gap Training Test
- 28. The Metric Gap 28 Training Error Generalization Error RMSE Precision @ K Stage Example Learning to Rank Training Test
- 29. The Metric Gap Training Error Generalization Error A/B Metric RMSE Precision @ K Conversion Learning to Rank Stage Example Offline (Hours) Online (Weeks)
- 30. The Metric Gap 30 Training Error Generalization Error A/B Metric RMSE Precision @ K Conversion Learning to Rank Stage Example Offline (Hours) Online (Weeks) Offline -> Online Gap
- 31. Online Learning – Overview • Naïve Online Learning is A/B testing - Try different sets of parameters, pick the winner • Multi-Arm Bandit - Exploiting the parameter sets that do well - Exploring parameters that we don’t understand well yet (high variance)
- 32. Online Learning – Implementation • Iteration Loop - Add Sets of Parameters - Explore vs. Exploit Current Parameters • Validate Online Learning with A/B testing • Note: Tradeoff in Time to Statistical Significance
- 33. Example – Start with 1 arm Parameter Metric
- 34. Example – Resample arm Parameter Metric
- 35. Example – Determine 2nd Arm Parameter Metric
- 36. Example – Select Arm Parameter Metric
- 37. Example – Improve Arm’s Estimation Parameter Metric
- 38. Example – Select Arm Parameter Metric
- 39. Example – Learn from Arm Parameter Metric
- 40. Example – Determine New Arm Parameter Metric
- 41. Training DataFlow Collaborative Filter Service (Realtime) Collaborative Filter HyperParameter Tuning (Batch with Spark) Collaborative Filter Training (Batch with Spark)
- 42. Training DataFlow Collaborative Filter Service (Realtime) Collaborative Filter HyperParameter Tuning (Batch with Spark) Collaborative Filter Training (Batch with Spark) Search Service (Realtime) Search HyperParameter Tuning (Batch with Spark) Search Training (Batch with Spark)
- 43. Training DataFlow Collaborative Filter Service (Realtime) Collaborative Filter HyperParameter Tuning (Batch with Spark) Collaborative Filter Training (Batch with Spark) Search Service (Realtime) Search HyperParameter Tuning (Batch with Spark) Search Training (Batch with Spark) User Interaction Logs (Kafka) Frontends & Backend Services
- 44. Training DataFlow Collaborative Filter Service (Realtime) Collaborative Filter HyperParameter Tuning (Batch with Spark) Collaborative Filter Training (Batch with Spark) Search Service (Realtime) Search HyperParameter Tuning (Batch with Spark) Search Training (Batch with Spark) User Interaction Logs (Kafka) Online Learning Frontends & Backend Services
- 45. Training DataFlow Collaborative Filter Service (Realtime) Collaborative Filter HyperParameter Tuning (Batch with Spark) Collaborative Filter Training (Batch with Spark) Search Service (Realtime) Search HyperParameter Tuning (Batch with Spark) Search Training (Batch with Spark) User Interaction Logs (Kafka) Online Learning Frontends & Backend Services A/B Validation
- 47. Recommendation Explanations • Amazon • Ness • Netflix • Ness - Social 47
- 48. Summarizing Content • Essential for Mobile • Balance Utility With Trust? - Summarize, but surface raw data • Example: - Initially, read every review - Later, use average star rating 48
- 50. Dish Recommendation • What to try once I have arrived? 50
- 51. Thanks! Jeremy Schiff, Ph.D. jschiff@opentable.com Other OpenTable Members @ RecSys: Sudeep Das & Pablo Delgado
Notes de l'éditeur
- 70,80,95%