Ce diaporama a bien été signalé.
Nous utilisons votre profil LinkedIn et vos données d’activité pour vous proposer des publicités personnalisées et pertinentes. Vous pouvez changer vos préférences de publicités à tout moment.
Prochain SlideShare
What to Upload to SlideShare



What is AB, MVT and MAB?

What is the difference between them? When to use one or the other? Who use?
Talk was on Halloween on Globo.com.

What is AB, MVT and MAB?

  1. 1. with a/b testing
  2. 2. A GUY WHO LIKES DATA ON GLOBO.COM @renan_oliveira
  3. 3. What is A/B?
  4. 4. Experimentwith a group of users who will notbe impactedby change and other groups is impacted withit
  5. 5. Who use?
  6. 6. Netflix
  7. 7. When you have highly talented individuals who have lots of great ideas, it’s important to have a framework where any new idea can be developed and tested, and data, notopinion, is used to make decisions. https://medium.com/netflix-techblog/a-b-testing-and-beyond-improving-the-netflix-streaming-experience-with-experimentation-and-data-5b0ae9295bdf
  8. 8. AirBnb
  9. 9. The outsideworld often has a much larger effectonmetrics than product changes do. Users can behave very differently depending on the day of week, the time of year, the weather, or whether they learned about the website through an online ad or found the site organically. Controlled experimentsisolatetheimpact of the product change while controlling for the aforementioned external factors. https://medium.com/airbnb-engineering/experiments-at-airbnb-e2db3abf39e7
  10. 10. globo.com
  11. 11. We use it to validate our products both in the UX area and in the APIs. One of the steps we are seeking and increasingly integrate our A/B solution with the platforms developed by other teams.
  12. 12. A/B vs
  13. 13. First step to make your A/B
  14. 14. So… What's a hypothesis?
  15. 15. Aquestion! After observing a behavior create a question to try to guess the answer with the experiment.
  16. 16. When I get 99% confidence can I stop my experiment?
  17. 17. NO! This test says that the difference between the alternative and the control is real. If you do not include the time axis you lose the seasonal behavior.
  18. 18. How long do I leave my test running?
  19. 19. IDON’TKNOW! Each product has its own user seasonal behavior. You'll know it better than I do.
  20. 20. How do I analyze this?
  21. 21. Trust thedashboardsand consult a datascientist
  22. 22. Statistic? Data Science?
  23. 23. It is a skill that will notbeableto escape, everyone should at some time know how to metrify their products in a correct way. Until there, look for specialists toguideyou.
  24. 24. More than one test on the same page?
  25. 25. The default answer is no.
  26. 26. No solution at all?
  27. 27. MVT - multivariate testing
  28. 28. For pages with a lotof audience all the experiments and alternatives of your product can be combined.
  29. 29. Think of settheory and the intersection of it, so we know what the bestcombination of experiments and alternatives for your product.
  30. 30. IsolatedMVT
  31. 31. If we build all the combination then we have one where the useronly sawthisexperiment.
  32. 32. AB vs MVT?
  33. 33. AB MVT High statistical confidence Does not depend on too much volume Non-visible noise may occur Best combination Population between experiments must be significant Intercessions are seen
  34. 34. https://blog.acolyer.org/2017/09/27/an-efficient-bandit-algorithm-for-real-time-multivariate-optimization/ 27% MVT + MAB conversion in 1 week
  35. 35. Multi-Armed Bandit
  36. 36. Minimize the consumption of the worst
  37. 37. The best options may vary over time ARMsmostdeliveryperhour
  38. 38. Exploration vs Exploitation
  39. 39. Quem é Díaz-Canel, indicado para suceder Raúl Castro em Cuba? Indicado para suceder Raúl Castro nasceu após a Revolução Cubana https://g1.globo.com/mundo/noticia/quem-e-miguel-diaz-canel-o-discipulo-predileto-de-raul-castro-que-assume-o-poder-em-cuba.ghtml TITLE1 TITLE2 TITLE1 >15%
  40. 40. Viewability Use of an optimization algorithm (MAB) to set the best time to start advertising loading on the page and distribute users accordingly ALTERNATIVA A ALTERNATIVA B ALTERNATIVA C 200px 250px 300px
  41. 41. THANK YOU @renan_oliveira www.renanoliveira.net
  • GilmaraAlvesdeCastro

    Jan. 23, 2019
  • MatheusJorge1

    Nov. 9, 2018
  • dannyeuu

    Nov. 8, 2018
  • brunofms

    Nov. 8, 2018
  • LuanaHohmann

    Nov. 8, 2018

What is the difference between them? When to use one or the other? Who use? Talk was on Halloween on Globo.com.


Nombre de vues


Sur Slideshare


À partir des intégrations


Nombre d'intégrations









Mentions J'aime