How to Run Product Discovery Experiments in FinTech
•Télécharger en tant que PPTX, PDF•
4 j'aime•830 vues
From this presentation you will learn about running Product Discovery
Recommandé
Contenu connexe
Tendances
Tendances (20)
Similaire à How to Run Product Discovery Experiments in FinTech
Similaire à How to Run Product Discovery Experiments in FinTech (20)
Plus de Product School
Plus de Product School (20)
Dernier
Dernier (20)
How to Run Product Discovery Experiments in FinTech
- 1. www.productschool.com How to Run Product Discovery Experiments in FinTech
- 2. Join 35,000+Product Managers on Free Resources Discover great job opportunities Job Portal prdct.school/PSJobPortalprdct.school/events-slack
- 3. CERTIFICATES Your Product Management Certificate Path Product Leadership Certificate™ Full Stack Product Management Certificate™ Product Management Certificate™ 20 HOURS40 HOURS40 HOURS
- 4. C O U R S E S Corporate Training Level up your team’s product management skills
- 5. EXPERIMENTATION & PRODUCT DISCOVERY Google for Startups campus - London November 2019
- 7. 2011 2011 2013 2015 2016 2019 BSC Computer Science
- 8. The Goal Photo by Júnior Ferreira on Unsplash As your guest speaker, I want to give you an idea of 1. What experimentation really means 2. Why it is important 3. How to run a successful one So that you see the real value and hopefully you will be motivated to learn more about experimentation and apply it to your way of thinking and doing.
- 9. What I am going to cover… Demystifying Experimentation 1 Running Experiments Experimentation Culture 2 3 The What The Why The Framework Identifying Problems Create Hypothesis Design Run Learn The Big Picture The When The Culture 8 Pitfalls
- 10. I have an idea!
- 11. I have an idea!
- 12. Should we call it an idea yet? “Instead of saying 'I have an idea,' what if you said 'I have a new hypothesis, let's go test it, see if it's valid, ask how quickly can we validate it.' And if it's not valid, move on to the next one." Satya Nadella, Microsoft CEO
- 13. What’s wrong with the idea? Idea = People need a mobile application to help them find the best financial product that’s suitable to their needs and situation. Who do we mean by ‘People’?1 Why do we think a mobile application should be the solution?2 What is the actual problem we are trying to solve here?3 How are we so sure? Is there any evidence?4
- 14. Idea or Assumption? An Idea on its own is an Assumption in need of testing and evidence
- 15. Moving from the Idea to an Assumption Idea = People need a mobile application to help them find the best financial product that’s suitable to their needs and situation. Assumption = We assume there is a population of people that would want to find a suitable mortgage product on the move and receive advice whenever they need it. We know that this need to be tested.1 We assume the problem is not for everyone.2 We are now shifting from a Solution to the actual Problems.3 Multiple hypothesis can be derived from this assumption.4
- 16. Assumption and Hypothesis An Assumption can be broken down to multiple specific and measurable Hypothesis that can be tested.
- 17. The Hypothesis We believe that Mortgage Brokers can have access to a mortgage advisor on the move and 24/7 to advise them with the best mortgage products for their clients. To verify that we will implement a virtual mortgage assistant in our mobile banking app that can analyse financial status of a potential client and advise the best option And measure the number of times that the virtual mortgage assistant could offer the right product and also the number of calls made to our advisors during our experiment We are right if our VA could advise a right product by 80% and the number of calls reduced by 30%
- 18. Definitions Idea = A general Assumption with no or not enough Evidence Assumption = An un-testable statement consists of multiple Hypothesis with no clear benchmark or target Metrics Hypothesis = A Specific, Measurable and Testable statement derived from the assumption with current benchmark and the target metrics we want to improve (reduce or increase).
- 19. Experimentation: The What A Problem-solving Framework that gives us all the necessary Tools in order to test and validate our Hypothesis Problem-solving: The focus is on finding the right problems1 Framework: It’s a process and not a one step task2 Tools: It gives us all the different assets we need to find the problems and then validate our hypothesis. 3
- 20. Experimentation: The Why Establishing data-driven culture and making decisions based on data and insight and eliminating bias 1 Identifying the right solution that adds more value than existing mechanisms2 Measuring the usability and impact on users3 De-risking future investment by identifying risk and measuring practicality4 Finding the right problem and building the right solution
- 21. Experimentation: The Outcome Enough evidence that can lead us to propose a right solution to solve a real problem faced by target customers
- 23. 2 More things to say … Demystifying Experimentation 1 Running Experiments Experimentation Culture 2 3 The What The Why The Framework Identifying Problems Create Hypothesis Design Run Learn The Big Picture The When The Culture 8 Pitfalls
- 25. Finding the Right Problem(s) to Solve At it’s core, the job of Entrepreneurs and Product People is to Live, Breathe and Stay with a Problem until it’s not a problem any more! Find the right problem Test till you hit the right solution Build the right product, right Build an organisation Build an industry
- 26. Start with the ‘Who’ Who is facing the problem? Who? What do they do? Jobs to be done Why? What are they trying to achieve? Pains What pains are They facing?
- 27. Sources of Information Go out and speak to the customers1 Run effective workshop2 Conduct surveys3 Speak to your support team4 Look at complaints5 Get input from Marketing and Sales6 Use existing data7 Check previous experiments8 (user groups, external reports, competitors, blogs …)
- 28. Our Customer Mortgage Brokers Married and have a family Single under 30 Assumption = We assume there is a population of people that would want to find a suitable mortgage product on the move and receive advice whenever they need it.
- 29. Customer Segments: Build up Empathy with them The Empathy Map Think & Feel? Hear? See? Say? What matters to them? What worries them? How are they influenced by hearing others say? What are they saying? What do we think they are saying? What’s their surrounding and Environment?
- 30. The Characteristics of our Customer The Empathy Map Think & Feel? Hear? See? Say? It’s important to find a right mortgage for their clients and build up trust and credit Their colleagues say our advisors are not always helpful It’s hard to get hold of an advisor in the bank and each advisor might advise differently Lots of information on the bank website Mortgage Broker
- 31. Build Customer Profile & Prioritise Customer Jobs Gains Pains + Insignificant - Important + Nice to have - Essential + Moderate - Extreme
- 32. Broker’s Profile Customer Jobs Gains Pains + Insignificant - Important + Nice to have - Essential + Moderate - Extreme Learn about different products Find a suitable mortgage for their client Know how to submit an application High chance of mortgage approval Build up knowledge so can advise other clients Submit application with no hassle Picking a wrong product for client information on the website is not consistent A lot of back and forth with client and the bank
- 33. Build Value Map & Prioritise Product/Service Gains Creator Pain Relievers + Insignificant - Important + Nice to have - Essential + Moderate - Extreme
- 34. Value Proposition + Insignificant - Important + Nice to have - Essential + Moderate - Extreme Improve information architecture of the website Virtual Mortgage Assistant Can advise brokers with the right product Easier to find information Available on the move and 24/7 and Reduction in volumes of calls Avoiding confusion Product/Service Gains Creator Pain Relievers
- 37. Hypothesis: The Structure We believe that, {the hypothesis} To verify that we will, {the experiment} And measure, {the metrics we want to measure} We are right it, {the success measure}
- 38. Our Hypothesis We believe that Mortgage Brokers can have access to a mortgage advisor on the move and 24/7 to advise them with the best mortgage products for their clients. To verify that we will implement a virtual mortgage assistant in our mobile banking app that can analyse financial status of a potential client and advise the best option And measure the number of times that the virtual mortgage assistant could offer the right product and also the number of calls made to our advisors during our experiment We are right if our VA could advise a right product by 80% and the number of calls reduced by 30%
- 39. Metrics Primary Secondary Measurement of a task that your customer does with your product/service/feature Measures the success of the overall experiment Measures the extra events related to the primary metric
- 40. Metrics Qualitative Quantitative Usability testing Customer interview Voice of the customer Analytics Events coded in prototypes Surveys Both Qualitative and Quantitative metrics required in measuring experiments
- 41. Prioritise Hypothesis David J. Bland https://www.precoil.com/ Desirable Feasible Viable Do customers want this? Should we do this?Can we do this? (Product, UX) (Product, Finance, Business)(Engineering, Brand, Legal)
- 42. Prioritise Hypothesis Important Unimportant Have Evidence No Evidence David J. Bland https://www.precoil.com/
- 43. Test Cards The hypothesis The experiment Metrics we want to measure The success measure
- 45. The Experimentation Team Cross Functional Product Engineering User Experience Data Science Dedicated , Customer Centric & Analytical Interested Parties+ SMEs Leadership
- 46. Choose Type of Experiment Low-fidelity High-fidelity Pen & Paper Prototype Clickable Prototype Front-end / Client-side Back-end / Server side (A/B testing, Call to Action, landing page, Painted door, …) The nature of the product and your hypothesis defines the type of experiment Customer interaction is limited Realistic customer interaction required (Building SDK, Databases, …)
- 47. Shape User Group or Model Office
- 48. Shape User Group or Model Office A sample size of the customer segment who will try the solution for a pre-defined period of time. 1 Form the group prior to running the experiment 2 Share the experiment goal with the group 3 Put in place a set of instructions to mitigate potential risks
- 50. Run the Experiment Implement mechanisms to continuously observe and collect data during the experiment Continuously observe impacts and identify potential risks Have an Exit Plan in place and be ready for the worst case scenario The team should stay available to make any necessary changes while the experiment is happening 1 2 3 4
- 52. Results Collect all qualitative and quantitative data1 Work with the Data Science team to analyse the data2 Issue the draft report and share with few more people to validate your interpretation of data 3 Produce the final result and Share it!4
- 54. 1 More … Demystifying Experimentation 1 Running Experiments Experimentation Culture 2 3 The What The Why The Framework The Big Picture The When The Culture 8 Pitfalls Identifying Problems Create Hypothesis Design Run Learn
- 55. The Big Picture How does experimentation fit in to the product development lifecycle?
- 56. What is right to build Building it right Feedback and Insight Results
- 57. The When Experimentation is part of the iterative process of the Product Discovery and Development lifecycle. Think of it as a good Habit that happens all the time. Product Discovery Optimisation Heavy weight Experimentation Frequent Experimentation
- 58. The Culture It’s how we think It’s the way we act A thought process Follow a framework Hypothesis over Idea Test it Disproving over Failure Disprove safe, Learn fast Making decisions Follow data Impact on the customer Iterative process, Continuous feedback and improvement Outcome over Output
- 59. 8 pitfalls to avoid! Running lengthy experiments1 Designing experiments for scale2 Running not enough experiments with not enough users 3 Disproving hypothesis is not failure4 Selecting wrong metrics5 Getting confused by analytics6 Thinking that it’s all done by UX7 Expecting that everyone knows what experimentation is 8
- 60. Recapping The Goal Photo by Júnior Ferreira on Unsplash As your guest speaker, I tried to give you an idea of 1. What experimentation really means 2. Why it is important 3. How to run a successful one So hopefully you have seen the real value and you are now motivated to learn more about experimentation and apply it to your way of thinking and doing.
- 61. Recommendations David J. Bland Alex Osterwalder https://www.strategyzer.com/
- 62. Recommendations Teresa Torres https://www.producttalk.org/ Product Experimentation Masterclass https://www.optimizely.com/produc t-experimentation-masterclass/ https://www.optimizely.com/uk/tho mke-talks/ The Thomke Talks https://read.realstartupbook.com/
- 64. www.productschool.com Part-time online Product Management Training Courses and Corporate Training
Notes de l'éditeur
- If you’re interested to connect with other Product Managers, aspiring PMs, or those within tech, join our Slack community of over 40,000 professionals. It’s a great place to network and to find interesting content. We host a weekly AMA through our Slack channel on Tuesdays from 11:15am - 12pm PST. We have also recently launched the Job Portal where you can find the latest Product Management opportunities! As members of the Product School community, we'd like to provide you with these resources at your disposal.
- Product School’s Product Management Certificate Path comprises of 3 part-time courses. During Product Management Training you will learn Product Management fundamentals. You will deep dive into the technical knowledge to enhance your ability to work with agile teams, during Full Stack Product Management Training. Finally, Product Leadership Training will elevate your product knowledge to become an effective Product Leader.
- As well as individual courses we provide corporate training across the world! If you’d like to upskill your product team this is the best option for you. We have trained employees from multiple companies such as Deloitte, Salesforce, JP Morgan, Bank of America amongst many other companies across all industries.
- But nothing is better than doing experiments.