Learning over Sequences of Decisions with Reinforcement Learning
•
8 j'aime•1,722 vues
Web Optimization is a Reinforcement Learning problem. Q-Learning is introduced as a way to integrate AB Testing, Attribution, and Predictive Targeting.
Recommandé
Recommandé
Contenu connexe
Similaire à Learning over Sequences of Decisions with Reinforcement Learning
Similaire à Learning over Sequences of Decisions with Reinforcement Learning (20)
Dernier
Dernier (20)
Learning over Sequences of Decisions with Reinforcement Learning
- 1. Learning over Sequences of Decisions
- 2. CEO / Co-Founder Conductrics www.conductrics.com Past: Database Marketing Education: Artificial Intelligence & Economics twitter:@mgershoff, @conductrics Email:matt@conductrics.com www.conductrics.com/blog Who is this guy?
- 3. AI in the News www.Conductrics.com @conductrics
- 4. AI in the News www.Conductrics.com @conductrics
- 6. What’s In it For You www.Conductrics.com @conductrics
- 7. • Reinforcement Learning (RL): • AB Testing What’s In it For You www.Conductrics.com @conductrics
- 8. • Reinforcement Learning (RL): • AB Testing • Attribution What’s In it For You www.Conductrics.com @conductrics
- 9. • Reinforcement Learning (RL): • AB Testing • Attribution • Predictive Targeting What’s In it For You www.Conductrics.com @conductrics
- 10. • Reinforcement Learning (RL): • AB Testing • Attribution • Predictive Targeting • An RL Solution from AI What’s In it For You www.Conductrics.com @conductrics
- 11. • Reinforcement Learning (RL): • AB Testing • Attribution • Predictive Targeting • An RL Solution from AI • Tell EveryoneYou Know AI!!! What’s In it For You www.Conductrics.com @conductrics
- 13. Reinforcement Learning is a Problem not a Solution www.Conductrics.com @conductrics
- 14. Reinforcement Learning Problem: Learn to make a Sequence of Decisions by Trial & Error in order to Achieve some Goal(s) www.Conductrics.com @conductrics
- 19. 19 AB Testing Sequential Decisions Targeting Part 1: AB TEST = Trial & Error Learning www.Conductrics.com @conductrics
- 20. Single Location Decisions/AB Test Home Page Hero Decision Point Decision Simple Image Fancy Version Fancy Version RL Agent www.Conductrics.com @conductrics
- 21. Page A Location Single Location Decisions/AB Test www.Conductrics.com @conductrics
- 22. A B Page A Location Decision Single Location Decisions/AB Test www.Conductrics.com @conductrics
- 23. A B Page A Convert Location Decision Objective/Payoff Single Location Decisions/AB Test
- 24. A B Page A Convert Location Decision Objective/Payoff Single Location Decisions/AB Test www.Conductrics.com @conductrics
- 25. A B Page A Convert Don’t Convert Location Decision Objective/Payoff Single Location Decisions/AB Test
- 26. A B Page A Convert Don’t Convert Location Decision Objective/Payoff Single Location Decisions/AB Test www.Conductrics.com @conductrics
- 27. How to Solve: A B Page A Convert Don’t Convert Location Decision Objective/Payoff Single Location Decisions/AB Test
- 28. How to Solve: 1. AB/MV Testing A B Page A Convert Don’t Convert Location Decision Objective/Payoff Single Location Decisions/AB Test www.Conductrics.com @conductrics
- 29. How to Solve: 1. AB/MV Testing 2. Multi-Arm Bandit A B Page A Convert Don’t Convert Location Decision Objective/Payoff Single Location Decisions/AB Test
- 30. Only need Conversion Data Option Value A 5% B 6% Single Location Decisions/AB Test www.Conductrics.com @conductrics
- 31. 31 Part 2: Attribution as Sequential Decisions AB Testing Attribution Targeting www.Conductrics.com @conductrics
- 32. Single Location Compound: MVT Home Page Banner Hero Decision Point Decision #1 Decision #2 Banner A Banner B Banner C Simple Image Fancy Version Banner C Fancy Version RL Agent Search Results Special Offers CheckoutSignup Home Page www.Conductrics.com @conductrics
- 33. Sequential Decisions -> Dynamics Enter Site Page 1 Page 2 www.Conductrics.com @conductrics
- 34. Enter Site Page 1 Page 2 C D A B Sequential Decisions -> Dynamics www.Conductrics.com @conductrics
- 35. Enter Site Exit Site Page 1 Page 2 C D A B Sequential Decisions -> Dynamics
- 36. Enter Site Exit SiteGoal Page 1 Page 2 C D A B Sequential Decisions -> Dynamics www.Conductrics.com @conductrics
- 37. Enter Site Exit SiteGoal Page 1 Page 2 C D A B Sequential Decisions -> Dynamics www.Conductrics.com @conductrics So far just AB Testing
- 38. Enter Site Exit SiteGoal Page 1 Page 2 C D A B Sequential Decisions -> Dynamics www.Conductrics.com @conductrics Now add Dynamics
- 39. Enter Site Exit SiteGoal Page 1 Page 2 C D A B Sequential Decisions -> Dynamics Transitions = Dynamics www.Conductrics.com @conductrics
- 40. Enter Site Exit SiteGoal Page 1 Page 2 C D A B Sequential Decisions -> Dynamics www.Conductrics.com @conductrics
- 41. 1. Conversion Rates Option Value Page1:A 3% Page1:B 4% Page2:C 10% Page2:D 12% Sequential Decisions
- 42. 1. Conversion Rates 2. Transition Frequencies Page:Action Page 1 Page 2 Page1:A - 30% Page1:B - 20% Page2:C 2% - Page2:D 1% - Sequential Decisions -> Dynamics www.Conductrics.com @conductrics
- 43. This is Complicated! Sequential Decisions -> Dynamics
- 44. How to Assign Value? Backward Calculation: AssignValues BACK to events AFTER the Conversion www.Conductrics.com @conductrics
- 45. Agent Calculating Attribution: Backward Looking Search Results Special Offers Signup Home Page
- 46. How Does Google Do it? www.Conductrics.com @conductrics
- 47. How Does Google Do it?
- 48. How to Assign Value? Backward Calculation: Is this the ONLY way? www.Conductrics.com @conductrics
- 49. Q Learning ሿ𝑸 𝒔 𝒕, 𝒂 𝒕 + 𝜶[𝒓 𝒕+𝟏 + 𝜸 ∗ 𝒎𝒂𝒙 𝒂 𝑸 𝒔 𝒕+𝟏, 𝒂 𝒕 − 𝑸 𝒔 𝒕, 𝒂 𝒕 www.Conductrics.com @conductrics
- 50. Q Learning ሿ𝑸 𝒔 𝒕, 𝒂 𝒕 + 𝜶[𝒓 𝒕+𝟏 + 𝜸 ∗ 𝒎𝒂𝒙 𝒂 𝑸 𝒔 𝒕+𝟏, 𝒂 𝒕 − 𝑸 𝒔 𝒕, 𝒂 𝒕 www.Conductrics.com @conductrics
- 51. Q Learning ሿ𝑸 𝒔 𝒕, 𝒂 𝒕 + 𝜶[𝒓 𝒕+𝟏 + 𝜸 ∗ 𝒎𝒂𝒙 𝒂 𝑸 𝒔 𝒕+𝟏, 𝒂 𝒕 − 𝑸 𝒔 𝒕, 𝒂 𝒕
- 52. Q Learning ሿ𝑸 𝒔 𝒕, 𝒂 𝒕 + 𝜶[𝒓 𝒕+𝟏 + 𝜸 ∗ 𝒎𝒂𝒙 𝒂 𝑸 𝒔 𝒕+𝟏, 𝒂 𝒕 − 𝑸 𝒔 𝒕, 𝒂 𝒕 www.Conductrics.com @conductrics
- 53. Forward View: Q Learning ሿ𝑸 𝒔 𝒕, 𝒂 𝒕 + 𝜶[𝒓 𝒕+𝟏 + 𝜸 ∗ 𝒎𝒂𝒙 𝒂 𝑸 𝒔 𝒕+𝟏, 𝒂 𝒕 − 𝑸 𝒔 𝒕, 𝒂 𝒕 www.Conductrics.com @conductrics
- 54. Forward View: Q Learning Analytics Interpretation of Q-Learning 1)Treat Landing on the Next Page like a regular conversion! www.Conductrics.com @conductrics
- 55. Forward View: Q Learning Analytics Interpretation of Q-Learning 1)Treat Landing on the Next Page like a regular conversion! 2)Use the estimates at the next step as the conversion value! www.Conductrics.com @conductrics
- 56. Page 1 A B 1) Take an action Forward View: Q Learning www.Conductrics.com @conductrics
- 57. Page 1 A 1) Take an action – Pick A Forward View: Q Learning www.Conductrics.com @conductrics
- 58. Page 1 A 2) Measure what user does after Forward View: Q Learning www.Conductrics.com @conductrics
- 59. 2) Do they Convert? $10 Page 1 A Forward View: Q Learning www.Conductrics.com @conductrics
- 60. 2) Yes! $10 Page 1 A Forward View: Q Learning www.Conductrics.com @conductrics
- 61. ሿ𝑸 𝒔 𝒕, 𝒂 𝒕 + 𝜶[𝒓 𝒕+𝟏 + 𝜸 ∗ 𝒎𝒂𝒙 𝒂 𝑸 𝒔𝒕+𝟏, 𝒂 𝒕 − 𝑸 𝒔 𝒕, 𝒂 𝒕 2) Set r =$10 $10 Page 1 A Forward View: Q Learning www.Conductrics.com @conductrics
- 62. EXACTLY the SAME as AB TESTING $10 Page 1 A Forward View: Q Learning www.Conductrics.com @conductrics
- 63. 3) Do they next go to Page 2? Goal Page 1 A Page 2 Forward View: Q Learning
- 64. 3) Yes! Goal Page 1 Page 2 A Forward View: Q Learning www.Conductrics.com @conductrics
- 65. 3) Yes! Now in Dynamic part of Path Goal Page 1 Page 2 A Forward View: Q Learning www.Conductrics.com @conductrics
- 66. Page 2 C D 4) Check Current Estimated Values ‘C’ & ‘D’ Forward View: Q Learning www.Conductrics.com @conductrics
- 67. 4) Check Current Estimated Values ‘C’ & ‘D’ Of course initially C=$0; D=$0 Page 2 C D $0 $0 Forward View: Q Learning www.Conductrics.com @conductrics
- 68. 4) Check Current Estimated Values ‘C’ & ‘D’ But lets just assume a mean of C=$1; D=$5 Page 2 C D $1 $5 Forward View: Q Learning www.Conductrics.com @conductrics
- 69. ሿ𝑸 𝒔 𝒕, 𝒂 𝒕 + 𝜶[𝒓 𝒕+𝟏 + 𝜸 ∗ 𝒎𝒂𝒙 𝒂 𝑸 𝒔𝒕+𝟏, 𝒂 𝒕 − 𝑸 𝒔 𝒕, 𝒂 𝒕 4) Set max(Q(st,at)) = $5 Page 2 C D $1 $5 Forward View: Q Learning www.Conductrics.com @conductrics
- 70. ሿ𝑸 𝒔 𝒕, 𝒂 𝒕 + 𝜶[𝒓 𝒕+𝟏 + 𝜸 ∗ 𝒎𝒂𝒙 𝒂 𝑸 𝒔𝒕+𝟏, 𝒂 𝒕 − 𝑸 𝒔 𝒕, 𝒂 𝒕 1. 𝛄 𝐢𝐬 the 𝐝𝐢𝐬𝐜𝐨𝐮𝐧𝐭 𝐫𝐚𝐭𝐞 2. Related to Google’s Half Life 3. 7 day half life 0.9 Forward View: Q Learning www.Conductrics.com @conductrics
- 71. ሿ𝑸 𝒔 𝒕, 𝒂 𝒕 + 𝜶[𝒓 𝒕+𝟏 + 𝜸 ∗ 𝒎𝒂𝒙 𝒂 𝑸 𝒔𝒕+𝟏, 𝒂 𝒕 − 𝑸 𝒔 𝒕, 𝒂 𝒕 5) 𝐏𝐚𝐠𝐞𝟏: 𝐀 = 𝟏 + 𝟎. 𝟗 ∗ 𝟓 Goal Page 1 Page 2 A Forward View: Q Learning www.Conductrics.com @conductrics
- 72. Forward View: Q Learning Direct Credit: $1.0 Attribution Credit: $4.5 www.Conductrics.com @conductrics
- 73. Forward View: Q Learning Direct Credit: $1.0 Attribution Credit: $4.5 Total Page1|A: $5.5 www.Conductrics.com @conductrics
- 74. 5) 𝐂𝐫𝐞𝐝𝐢𝐭 𝐏𝐚𝐠𝐞𝟏: 𝐀 = 𝟓. 𝟓 Goal Page 1 Page 2 A Forward View: Q Learning www.Conductrics.com @conductrics
- 75. Attribution in just two simple steps: Forward View: Q Learning www.Conductrics.com @conductrics
- 76. Forward View: Q Learning Attribution in just two simple steps: 1)Treat Landing on Next Page like a regular conversion! www.Conductrics.com @conductrics
- 77. Attribution in just two simple steps: 1)Treat Landing on Next Page like a regular conversion! 2)Use Predictions of future values at the next step as the conversion value! Forward View: Q Learning www.Conductrics.com @conductrics
- 78. Attribution in just two simple steps: 1)Treat Landing on Next Page exactly like a conversion! 2)Use estimates at the next step as the conversion value! 3)This is guaranteed to converge to optimum result!!! Forward View: Q Learning www.Conductrics.com @conductrics
- 79. 79 Part 3: Targeting Trial & Error Learning Sequential Decisions Targeting www.Conductrics.com @conductrics
- 80. Targeting = Decision Logic IF [Customer] THEN [Experiences?] www.Conductrics.com @conductrics
- 81. Q Learning + Targeting User: Is a New User and from Rural area Page 1 Page 2 A www.Conductrics.com @conductrics
- 82. User: Is a New User and from Rural area Page 1 Page 2 A Q Learning + Targeting www.Conductrics.com @conductrics
- 83. Attribution calculation depends on [Rural;New] Page 1 Page 2 A Q Learning + Targeting www.Conductrics.com @conductrics
- 84. 84 www.Conductrics.com @conductrics At Page2: Evaluate Value New & Rural Customer
- 85. 85 At Page2: Evaluate Value New & Rural Customer www.Conductrics.com @conductrics
- 86. 86 www.Conductrics.com @conductrics At Page2: Evaluate Value New & Rural Customer
- 88. 88 Predicted Value=43% Q Learning + Targeting www.Conductrics.com @conductrics
- 89. ሿ𝑸 𝒔 𝒕, 𝒂 𝒕 + 𝜶[𝒓 𝒕+𝟏 + 𝜸 ∗ 𝒎𝒂𝒙 𝒂 𝑸 𝒔𝒕+𝟏, 𝒂 𝒕 − 𝑸 𝒔 𝒕, 𝒂 𝒕 Page 1 Page 2 A 𝐏𝐚𝐠𝐞𝟏: 𝐀 = 𝟎 + 𝟎. 𝟗 ∗ 𝟒𝟑% Q Learning + Targeting www.Conductrics.com @conductrics
- 90. 90 Case Study: Web & Call Center Optimize Marketing Site and Call Center IVR 1. WebSite • Initial Offer • Online Chat option 2. Call Center • Choice of IVR prompts Result: Call Center Conversion by 26% Increased average call value by $9.28www.Conductrics.com @conductrics
- 91. Sales drive Optimization User Features •Time of Day •Geo-Data •Browser Data Get Decision Website IVR Options Response IVR Prompts: A;B;C;…. Get Decision Call Agent IVR ‘Rewards’ Web Decision Case Study: Web and Call Center www.Conductrics.com @conductrics
- 92. Application Layer Targeted Multi-Touch Optimization Touch-Point 1 User 2) Option Response 3) Touch-Point Transition 1) Option Request Local Options Optimizer F1 F2 Fn S Local Model 1 Touch-Point 2 User 7) Option Response 4) Option Request Local Options Optimizer F1 F2 Fn S Local Model 25) Calculates Attribution Credit and sends to Model1 8)Conversion 6) Update Local Model1 using credit as a conversion 9) Update Local Model2 using conversion value • Attribution Credit enables Local Optimizers to Solve Global Multi-Touch Optimization www.Conductrics.com @conductrics
- 93. 1) Attribution can be solved by hacking ‘AB Testing’ (Q-Learning) 2) Extended Attribution to include decisions/experiments 3) Looked into the eye of AI and Lived What did we Do/Learn? www.Conductrics.com @conductrics
- 94. References www.Conductrics.com @conductrics 1) https://conductrics.com/data- science-resources-2 2) http://videolectures.net/mlss09uk_ littman_rl (model based RL) 3) https://en.wikipedia.org/wiki/Mar kov_decision_process