When in doubt, go live

6. PO Establishing Feedback Loops Users Team Stakeholders Users

7. PO Establishing Feedback Loops Users Team Stakeholders Users

8. PO Delivery Pipeline Feature Toggle Shadow TraﬃcLab Test Focus Group Survey Visual Report A/B Test Establishing Feedback Loops

30. A/B testing © 2020 ThoughtWorks “You want your data to inform, to guide, to improve your business model, to help you decide on a course of action.” Lean Analytics Focus on the understanding of the underlying statistics that drives the calculation of a sample size. STATS

31. A/B testing © 2020 ThoughtWorks “You want your data to inform, to guide, to improve your business model, to help you decide on a course of action.” Lean Analytics A/B testing ≡ a set of statistical tests that evaluate two independent groups, a control and a test group “Independent groups” -> between-subjects design STATS Focus on the understanding of the underlying statistics that drives the calculation of a sample size. groups = variants “Independent groups” -> between-subjects design

33. A/B testing A/B testing mostly uses statistical hypothesis testing to calculate the likelihood of a change in your website being meaningful. Null hypothesis (H0): The state of the world. There is no effect, no difference when you apply changes. H0: Our <KPIs> remained the “same” in the control group and in the test group Alternative hypothesis (H1): the changes in the test group had a real effect. H1: Our users are actively engaged in clicking the button and therefore our A2B is relatively increased by 5% © 2020 ThoughtWorks

34. A/B testing © 2020 ThoughtWorks Alternative hypothesis (H1): the changes in the test group had a real eﬀect. H1: Our users are actively engaged in clicking the button and therefore our A2B is relatively increased by 5%

38. A/B testing © 2020 ThoughtWorks Source: https://abtestguide.com/abtestsize/ Metrics we know We decide from previous data or knowledge about this variable [eﬀect size] Dependent on the variable and what we are looking for [normally two-sided]

39. A/B testing © 2020 ThoughtWorks Source: https://abtestguide.com/abtestsize/ Metrics we know We decide from previous data or knowledge about this variable [eﬀect size] We can play but mostly by convention and dependent on traﬃc [accuracy] Dependent on the variable and what we are looking for [normally two-sided]

41. A/B testing © 2020 ThoughtWorks Source: https://abtestguide.com/abtestsize/ Test conversion rate = 15 * 2 + 2 = 2.3% (± 0.3%) Effect size The magnitude of the effect, how important the difference is Improvement that is meaningful for your business Test conversion rate - Control conversion rate Control conversion rate Relative improvement*100 = 100

42. A/B testing © 2020 ThoughtWorks Source: https://abtestguide.com/abtestsize/ One-sided or two-sided? ControlTest Mean test Mean control Is the difference significant enough to reject the null hypothesis? H0 : 𝝻t = 𝝻c 𝝻t : mean test 𝝻c : mean control difference in means

43. A/B testing © 2020 ThoughtWorks Source: https://abtestguide.com/abtestsize/ One-sided or two-sided? H1 : 𝝻t > 𝝻c (one-sided) directional H1 : 𝝻t ≠ 𝝻c (two-sided) Two-sided tends to be the best option 𝝻t : mean test 𝝻c : mean control

45. A/B testing © 2020 ThoughtWorks Power of a test: the probability of finding an effect when it is really there. It is the inverse of the type II error (false negatives) Source: https://towardsdatascience.com/a-guide-for-selecting-an-appropriate-metric-for-your-a-b-test-9068cccb7fb Typical value is 80% (a convention) Power Chance to miss a true effect Sample size Power, significance level & confidence level

46. A/B testing © 2020 ThoughtWorks Source: https://www.youtube.com/watch?v=CSBCKVQLf8c Our study Effect present Effect absent Real World Effect present Reject H0 Type II error (miss) Effect absent Type I error (false alarm) Reject H1 Type II error : probability to miss an effect that is really there (the odds to not detect it)

47. A/B testing © 2020 ThoughtWorks Source: https://www.youtube.com/watch?v=CSBCKVQLf8c Our study Effect present Effect absent Real World Effect present Reject H0 (power 1-𝛃) Type II error (miss) ( 𝛃 risk) Effect absent Type I error (false alarm) Reject H1 Type II error : miss -> probability less than 20% (𝛃) Power is 1-𝛃 -> 80% Power Chance to miss a true effect Sample size

48. A/B testing © 2020 ThoughtWorks Source: https://towardsdatascience.com/a-guide-for-selecting-an-appropriate-metric-for-your-a-b-test-9068cccb7fb Typical value is 95% (a convention) Significance level (𝛂): the probability of detecting an effect that is really not there Power, significance level & confidence level

49. A/B testing © 2020 ThoughtWorks Source: https://www.youtube.com/watch?v=CSBCKVQLf8c Type I error : false alarm -> probability less than 5% (𝛂) Confidence level is 1- 𝛂 : 95% Significance level 𝛂 related to p-value: 𝛂 > p-value Our study Effect present Effect absent Real World Effect present Reject H0 Type II error (miss) Effect absent Type I error (false alarm) (𝛂 risk) Reject H1

50. A/B testing © 2020 ThoughtWorks Confidence level: the inverse of the significance level. The probability that the value of a parameter falls within a specified range of values Source: https://towardsdatascience.com/a-guide-for-selecting-an-appropriate-metric-for-your-a-b-test-9068cccb7fb Typical value is 95% (a convention) Significance level (𝛂) Confidence level Sample size (significance level 𝛂 tells you about the probability that the effect you found was just chance; 𝛂 > p-value) Power, significance level & confidence level Significance level ~ 0.05 (5%) P-value < 0.05

51. A/B testing © 2020 ThoughtWorks Source: https://abtestguide.com/abtestsize/ Meaningful for your business Power and confidence level influence your sample size and the probability of finding a true effect

55. A/B testing © 2020 ThoughtWorks High traffic Low traffic Choose KPIs wisely, low effect size Preferably AB but also MVT Choose KPIs with high increase (large effect size) AB Run Qualitative tests Never stop an experiment before time even if you “find” significant results (danger! False positives raising!) Source: https://www.evanmiller.org/how-not-to-run-an-ab-test.html https://vwo.com/blog/ab-split-testing-low-traffic-sites/ Important points

58. Focus Group Survey © 2020 ThoughtWorks Delivery Pipeline Feature Toggle Shadow TraﬃcFocus Group Survey Visual Report Iterative and Incremental development Independent Teams What is it Study using inferential statistics to verify an hypothesis. When As part of the discovery of a feature, during development Why Short feedback loops Data-driven decisions Caution! You need experience designing and analysing statistical tests.

60. Focus Group Survey © 2020 ThoughtWorks Stronglydisagree Disagree Neutral Agree Stronglyagree Likert Scale [categorical variable] The shopteaser survey Your research question will drive the design of the experiment and also the analysis of your data trial trial trial trial

61. Focus Group Survey © 2020 ThoughtWorks Stronglydisagree Disagree Neutral Agree Stronglyagree Likert Scale The shopteaser survey trial trial trial Things that could go wrong: - Familiarity bias Methodology examples: - Gave 5s per trial so the answers would be spontaneous - The ﬁrst trials were discarded [categorical variable that can be transformed to continuous - scale 1-5]

62. During the design phase we also took into account: ● Collect demographic data: there is no such thing as enough data ● Collect feedback at the end of the survey: did they understand the task, did something go wrong? ● Make clear instructions: if you are not there, they cannot ask and will “assume” © 2020 ThoughtWorks Focus Group Survey The shopteaser survey

65. UX designers test the design and usability of a feature on a test group. ● Small group of people in-person (~5-10 pp) ● Web-based testing of users remote ● Qualitative questions ○ e.g. did you like it? Was it easy to ﬁnd? UX Lab tests © 2020 ThoughtWorks

67. PO Delivery Pipeline Feature Toggle Shadow TraﬃcLab Test Focus Group Survey Visual Report A/B Test Techniques for faster and better decisions Iterative and Incremental development In- dependent Team

68. When is your next release? Could it be earlier? Do you have a solid hypothesis and measurable KPIs for it? Which measurements could you be using instead of assuming the user’s preference? Which of your meetings in the next 2 weeks could be replaced by a lean experiment? © 2020 ThoughtWorks

When in doubt, go live

Vous êtes en train de lire un aperçu.

Consultez-les par la suite

When in doubt, go live

Plus De Contenu Connexe

Diaporamas pour vous (20)

Similaire à When in doubt, go live (20)

Plus par Thoughtworks (20)

Plus récents (20)