Agile Summit 希平方-攻其不背消費管道入口A/B Testing實戰分享

1. 希平方 - 攻其不背消費管道入口 A/B Testing 實戰分享希平方科技 Jasper Chang, Ph.D. 張修齊 June 29, 2018

2. Hello! I am Jasper • CTO of CSquare Technology • Started from 2014 • Research on students’ learning behavior & product development • Medium • E-mail https://medium.com/@hsiuchichang jasperchang@youtubelearn.com

3. • Brief introduction • Background • Purpose • Iterative A/B testing • Round 1 • Round 2 • Roll out to the website • Takeaway Abstract

4. Iterative Learning

5. • How do we attract people to buy our products? • From Facebook(FB) ads or fans page Background

6. From Designing with Data , O’REILLY pp.212 Fig. 6-2 Some considerations before kickoff

7. • Agile UX • UX and Dev sprint Agile UX https://www.agriya.com/blog/lean-ux-vs-agile-ux-shift-working-methodology/ https://www.nngroup.com/articles/doing-ux-agile-world/

8. • Things to be considered • Will the design of context affect people’s reading behavior? • Will people press the Call To Action (CTA) button in the bottom of the page? Purpose

9. Image from: https://www.dynamicyield.com/wp-content/uploads/2014/05/testing.jpg A/B testing or multivariate testing

10. http://www.azquotes.com/quote/530630

11. • Period: 2017/12/26(1 day) • Target: • 18~65 years on FB • Male, female • Except students in the system and colleagues of CSquare Technology Target criterion

12. A B C PC version for sharing page

13. A & B C Extended reading link design

14. 沒有任何滑動就離開滑動一次離開滑動 30% 滑動 60% 滑動 90% Scroll for reading Testing parameters

15. 沒有任何滑動就離開滑動一次離開滑動 30% 滑動 60% 滑動 90% 學員心得精選心得同場加映 _5 步撇同場加映 _15 歲文末 _ 好評不斷文末 _ 特訓成效驚人 Scroll for reading Extended click Testing parameters

16. 沒有任何滑動就離開滑動一次離開滑動 30% 滑動 60% 滑動 90% 學員心得精選心得同場加映 _5 步撇同場加映 _15 歲文末 _ 好評不斷文末 _ 特訓成效驚人前往購買頁前往體驗看更多心得 Scroll for reading Extended click Call To Action (CTA) Testing parameters

17. 版本 A (PC/ 現行版本 ) 版本 B (PC/ 新版按鈕 / 舊內文 ) 版本 C (PC/ 新版按鈕 / 新內文 ) 沒有任何滑動就離開 22.73% 18% 17% 滑動一次離開 9.09% 6% 10% 滑動 30% 4.55% 6% 10% 滑動 60% 13.64% 3% 7% 滑動 90% 45.45% 56% 53% Testing result of scroll for reading

20. 版本 A (PC/ 現行版本 ) 版本 B (PC/ 新版按鈕 / 舊內文 ) 版本 C (PC/ 新版按鈕 / 新內文 ) 前往購買頁 * 1.51A% A% 2.33A% 前往體驗 NA 0% 3% 看更多心得 NA 0% 3% Call To Action(CTA) 前往購買頁的轉換率，因涉及商業機密，接下來都會用 A ％做為基本單位的表示

21. 版本 A (PC/ 現行版本 ) 版本 C (PC/ 新版按鈕 / 新內文 ) 沒有任何滑動就離開 22.73% 17% 滑動一次離開 9.09% 10% 滑動 30% 4.55% 10% 滑動 60% 13.64% 7% 滑動 90% 45.45% 67% 點擊「同場加映 _5 步」撇 0.00% 3% 點擊「同場加映 _15 歲」 4.55% 3% 點擊「文末 _ 好評不斷」 0.00% 0% 點擊「文末 _ 特訓成效驚人」 0.00% 0% 前往購買頁 1.51A% 2.33A% 前往體驗無此按鈕 3% 看更多心得無此按鈕 3%

22. F GE Mobile version for sharing page

23. 版本 E (Mobile/ 現行版本 ) 版本 F (Mobile/ 新按鈕 / 舊內文 ) 版本 G (Mobile/ 新按鈕 / 新內文 ) 沒有任何滑動就離開 8% 16% 17% 滑動一次離開 6% 8% 5% 滑動 30% 6% 10% 11% 滑動 60% 10% 10% 11% 滑動 90% 42% 35% 35% 點擊「學員心得」 0% 0% 0% 點擊「精選心得」 0% 0% 2% 點擊「同場加映 _5 步」撇 2% 0% 0% 點擊「同場加映 _15 歲」 0% 0% 0% 點擊「文末 _ 好評不斷」 0% 0% 0% 點擊「文末 _ 特訓成效驚人」 0% 0% 0% 前往購買頁 0.67A% 4.33A% 1.66A% 前往體驗無此按鈕 5% 2% 看更多心得無此按鈕 0% 0%

24. • Pros • New design increase people’s intent for reading • New CTA button perform better than the old one • Both PC and mobile version have higher conversion rates for CTA button Round 1 summary

25. Source: Designing with Data Iterative design and testing

26. A B C Delete version A for next round

27. • Cons • Contradiction for 『前往購買頁』 with different device and version • The sample size is not big enough • P value is 0.0571(The parameter: 『滑動 90% 』 ) • Suppose other constraints are fixed, maybe we could have a p value around 0.01 Round 1 版本 B (PC/ 新版按鈕 / 舊內文 ) 版本 C (PC/ 新版按鈕 / 新內文 ) 版本 F (Mobile/ 新版按鈕 / 舊內文 ) 版本 G (Mobile/ 新版按鈕 / 新內文 )前往購買頁 A% 2.33A% 4.33A% 1.67A% Round 1 summary

28. From: http://mindbodysouljourney.com/our-greatest-weakness-lies-in-giving-up-the- most-certain-way-to-succeed-is-always-to-try-just-one-more-time-thomas-edison/ The Good & Bad Habits of Smart People

29. • Contradiction • Contradiction for 『前往購買頁』 with different device and version • Sample size • P value is 0.0571(The parameter: 『滑動 90% 』 ) • Suppose other constraints are fixed, maybe we could have a p value around 0.01 Round 1 版本 B (PC/ 新版按鈕 / 舊內文 ) 版本 C (PC/ 新版按鈕 / 新內文 ) 版本 F (Mobile/ 新版按鈕 / 舊內文 ) 版本 G (Mobile/ 新版按鈕 / 新內文 ) 前往購買頁 A% 2.33A% 4.33A% 1.67A% Round 2 purpose

30. • Period: 2017/12/26~2018/1/26(1 month) • Target: • 18~65 years on fb • Male, female • Except students in the system and colleagues of CSquare Technology Target criterion

31. B C PC version for sharing page

32. F G Mobile version for sharing page

33. Round 2 版本 B (PC/ 新版按鈕 / 舊內文 ) 版本 C (PC/ 新版按鈕 / 新內文 ) 版本 F (Mobile/ 新版按鈕 / 舊內文 ) 版本 G (Mobile/ 新版按鈕 / 新內文沒有任何滑動就離開 13% 13% 16% 17% 滑動一次離開 4% 4% 8% 5% 滑動 30% 7% 4% 10% 11% 滑動 60% 4% 4% 10% 11% 滑動 90% 42% 36% 35% 35% Round 1 版本 B (PC/ 新版按鈕 / 舊內文 ) 版本 C (PC/ 新版按鈕 / 新內文 ) 版本Ｆ (Mobile/ 新版按鈕 / 舊內文 ) 版本Ｇ (Mobile/ 新版按鈕 / 新內文沒有任何滑動就離開 18% 17% 13% 15% 滑動一次離開 6% 10% 5% 2% 滑動 30% 6% 10% 10% 11% 滑動 60% 3% 7% 7% 10% 滑動 90% 56% 53% 40% 38% Round 1 testing vs. Round 2 testing

34. Round 2 版本 B (PC/ 新版按鈕 / 舊內文 ) 版本 C (PC/ 新版按鈕 / 新內文 ) 版本 F (Mobile/ 新版按鈕 / 舊內文 ) 版本 G (Mobile/ 新版按鈕 / 新內文沒有任何滑動就離開 13% 13% 16% 17% 滑動一次離開 4% 4% 8% 5% 滑動 30% 7% 4% 10% 11% 滑動 60% 4% 4% 10% 11% 滑動 90% 42% 36% 35% 35% Round 1 版本 B (PC/ 新版按鈕 / 舊內文 ) 版本 C (PC/ 新版按鈕 / 新內文 ) 版本Ｆ (Mobile/ 新版按鈕 / 舊內文 ) 版本Ｇ (Mobile/ 新版按鈕 / 新內文沒有任何滑動就離開 18% 17% 13% 15% 滑動一次離開 6% 10% 5% 2% 滑動 30% 6% 10% 10% 11% 滑動 60% 3% 7% 7% 10% 滑動 90% 56% 53% 40% 38% Round 1 testing vs. Round 2 testing

35. Round 1 版本 B (PC/ 新版按鈕 / 舊內文 ) 版本 C (PC/ 新版按鈕 / 新內文 ) 版本 F (Mobile/ 新版按鈕 / 舊內文 ) 版本 G (Mobile/ 新版按鈕 / 新內文 ) 前往購買頁 A% 2.33A% 4.33A% 1.67A% 前往體驗 0% 3% 5% 2% 看更多心得 0% 3% 0% 0% CTA testing result Round1, 前往購買頁 C>B, but F>G

36. Round 2 版本 B (PC/ 新版按鈕 / 舊內文 ) 版本 C (PC/ 新版按鈕 / 新內文 ) 版本 F (Mobile/ 新版按鈕 / 舊內文 ) 版本 G (Mobile/ 新版按鈕 / 新內文 ) 前往購買頁 1.33A% 1.33A% 2.67A% 3A% 前往體驗 2% 2% 4% 2% 看更多心得 0% 1% 0% 2% Round 1 版本 B (PC/ 新版按鈕 / 舊內文 ) 版本 C (PC/ 新版按鈕 / 新內文 ) 版本 F (Mobile/ 新版按鈕 / 舊內文 ) 版本 G (Mobile/ 新版按鈕 / 新內文 ) 前往購買頁 A% 2.33A% 4.33A% 1.67A% 前往體驗 0% 3% 5% 2% 看更多心得 0% 3% 0% 0% CTA testing result Round1, 前往購買頁 C>B, but F>G Round2, 前往購買頁 C=B, F and G 相近

37. • Each version performed the same as well • Solve the contradiction in Round 1 testing • Sample size • People count • B version:164 • C version:168 • F version:133 • G version:130 Round 2 discussion

38. • Things to be considered • Will the design of context affect people’s reading behavior? • Will people press the CTA button in the bottom of the page? Summary

39. • Things to be considered • Will the design of context affect people’s reading behavior? • Will people press the CTA button in the bottom of the page? • Answers • No significant result imply the relation between design of context and the CTA button in the bottom of the page. Summary

40. • Things to be considered • Will the design of context affect people’s reading behavior? • Will people press the CTA button in the bottom of the page? • Answers • No significant result imply the relation between design of context and the CTA button in the bottom of the page. • However, only one page can not certainly prove this statement Summary

41. • Things to be considered • Will the design of context affect people’s reading behavior? • Will people press the CTA button in the bottom of the page? • Answers • No significant result imply the relation between design of context and the CTA button in the bottom of the page. • However, only one page can not certainly prove this statement • Maybe Round 3 experiment? Summary

42. So, what’s next?

43. Roll out to the website

44. B C Delete version C

45. Roll out to the website (2018/5/12~2018/6/12) Conversion rate improves 18% Conversion rate improves 73%

46. http://www.azquotes.com/quote/573389

47. • Resources are limited Multivariate testing Takeaway

48. • Resources are limited Multivariate testing • Iterative testing and analysis Analyze the result from the previous Further explore the questions yet to be answered Takeaway

49. • Resources are limited Multivariate testing • Iterative testing and analysis Analyze the result from the previous Further explore the questions yet to be answered • Focus on the target Conversion rate is the king! Takeaway

50. • 下載希平方—攻其不背 APP 試用• Medium • E-mail • 想知道 Jasper 更多資訊？ https://medium.com/@hsiuchichan g jasperchang@youtubelearn.co m 謝謝大家！

Agile Summit 希平方-攻其不背消費管道入口A/B Testing實戰分享

Vous êtes en train de lire un aperçu.

Consultez-les par la suite

Agile Summit 希平方-攻其不背消費管道入口A/B Testing實戰分享

Plus De Contenu Connexe

Plus récents (20)

À la une (20)