Recommandé
Recommandé
Contenu connexe
Similaire à Driving Product Success through Experimentation
Similaire à Driving Product Success through Experimentation (20)
Plus de NUS-ISS
Plus de NUS-ISS (20)
Dernier
Dernier (20)
Driving Product Success through Experimentation
- 1. Driving Product Success through Experimentation Talk + Workshop Scott Si Felicitas Seah #ISSLearningFest
- 2. What we will cover today #ISSLearningFest A New Way of Building Products Workshop: How to Get Started Which Experiments Matter Most? 1 2 3 Key Takeaway: Viewing product management in a more outcome-led, hypothesis-driven way – build with intention by framing potential features as experiments
- 3. Your Facilitators #ISSLearningFest Scott SI Felicitas SEAH Senior Lecturer & Consultant, Digital Products & Platforms Practice Principal Lecturer & Consultant, Digital Products & Platforms Practice
- 4. Goal as Product Builders… Deliver Value for clients and stakeholders #ISSLearningFest But… What is Value?
- 7. #ISSLearningFest https://blog.statsig.com/free-beer-6b4a14a8c250 Our assumptions can be misleading
- 8. #ISSLearningFest Build 10 things, find out only 3 worked Run 10 experiments, build the 3 that worked Image from https://www.hellowonderful.co/post/blast- off-with-these-8-fun-ways-to-make-a-rocket/ OLD NEW
- 9. #ISSLearningFest “Instead of saying 'I have an idea,' what if you said 'I have a new hypothesis, let's go test it, see if it's valid, ask how quickly can we validate it.' And if it's not valid, move on to the next one.” Satya Nadella, Microsoft CEO https://www.businessinsider.com/microsoft-ceo-brilliant-career-advice- 2017-5?IR=T&jwsource=cl
- 10. #ISSLearningFest Treat product changes as hypotheses you can prove/disprove with experiments Outcomes > Output From Features to Experiments
- 11. Value = Driving Metrics that Matter #ISSLearningFest Product Initiatives User Actions/KPIs User/Business Outcome New activation flow Product 360 view User Comments Seller chat North Star Metric Revenue/ Shareholder Value # of sign-ups App downloads % of returning users e.g. Total value transacted, services booked Ave. shopping cart items
- 12. So what is a product experiment? #ISSLearningFest Drive business and user outcome A B vs. Control Treatment
- 13. #ISSLearningFest We are all guinea pigs!
- 14. Getting Started: Crafting your hypothesis #ISSLearningFest We believe that [making product change] will lead to [meaningful outcome] We are right if [improvement in measurable user action versus baseline] and we can validate this by [type of experiment]
- 15. Example: Page Registrations #ISSLearningFest We believe that [changing ‘Register’ button to blue] will lead to [more membership registrations] We are right if [we see 2x improvement in ‘Register’ CTR versus the baseline] and we can validate this by [running A/B test for the homepage] https://hbr.org/2020/03/building-a-culture-of-experimentation
- 16. Why do we experiment? #ISSLearningFest • We don’t have all the answers • Drive metrics that matter - measure the impact of a change o New features/functions may hurt KPIs or metrics! o Differing/polarized opinions within the team on a product change • Avoid feature creep
- 17. Workshop Time! Time to practice what you learned #ISSLearningFest
- 18. Product Experiment Mini Case Study: E-commerce You are a product manager for an apparel e-commerce platform. While your channel traffic is on a historic high, your team flagged that total sales (gross merchandise value) are flat compared to the previous quarter. You review your product backlog for the purchase page to see which of these product changes your team should work on to drive the most business impact: Choose one of the features above. How would you define and reframe it as a hypothesis that you can test with experiments - with the goal of driving up purchases? #ISSLearningFest Delivery time estimation Additional pick-up options Seller Q&A Product 360-degree view A C B D
- 19. Workshop Template #ISSLearningFest We believe that [feature or product change] will lead to [meaningful user/business outcome] We are right if [proof of measurable improvement on desired user action over the baseline, usually a % increase]. and we can validate this by [the experiment you will run] 1 2 3 4
- 20. Example #ISSLearningFest We believe that [adding delivery time estimation prompt to the purchase page] will lead to [higher ‘add to shopping cart’ conversions] We are right if [# of purchases increase by 20% versus the baseline]. and we can validate this easily by [running A/B test on Singapore user base over the next 6 weeks]
- 21. Product Experimentation Steps 1. Know which metrics matter 2. Ask questions and brainstorm 3. Create testable hypothesis 4. Conduct experiment and monitor data 5. Make decision and implement changes 6. Repeat! #ISSLearningFest
- 22. #ISSLearningFest What is my riskiest assumption? What is the impact if I am proven right? What is the least expensive way to test it? Should we experiment on everything?
- 23. Experimentation is a Spectrum #ISSLearningFest Optimization Product Discovery Are we solving the right problem? Is this the right solution to solve the problem? Is this the best way to build the solution? More risky Assumptions Assumptions 1 2 3 ‘North Star’
- 24. #ISSLearningFest Goal of Product Experimentation To Win To Learn
- 25. DPP Practice Review-v1.0 © Copyright National University of Singapore. All Rights Reserved 25 Idea -> Failed Feature 1 -> Failed Feature 2 -> Successful Feature 3-> Outcome Idea -> Hypothesis -> Experiment -> Successful Feature -> Outcome Less Waste, More Value https://kromatic.com/blog/what-type-of-lean-startup-experiment-should-i-run/ Driving Product Success through Experimentation
- 27. Give Us Your Feedback #ISSLearningFest Day 1 Programme