Grace Ng | SearchLove San Diego, 'Designing Effective Experiments for Product Decisions'

1. Designing Effective Experiments for Product Decisions Grace Ng Co-founder, Javelin.com creators of QuickMVP @uxceo

2. "Business has only two functions - marketing and innovation." - Peter Drucker

3. Build

4. BUILD EXPERIMENT Hypothesize Test METRICS PIVOT/PERSEVERE Analyze @uxceo LEARN MEASURE

5. Javelin Experiment Board Start here. Brainstorm with stickies, pull it over to the right to start your experiment. Experiments Customer Problem Solution Riskiest Assumption Method & Success Criterion Result & Decision Learning 1 2 3 4 5 >OVPZ`VYJZ[VTLY)LHZZWLJPÄJHZWVZZPISL Time Limit: 5 Min What is the problem? Phrase it from your customer’s perspective. Time Limit: 5 Min +LÄUL[OLZVS[PVUVUS`HM[LY`VOH]L]HSPKH[LKHWYVISLT^VY[OZVS]PUNTime Limit: 5 Min List the assumptions that must hold true, for your hypothesis to be true. Time Limit: 10 Min Need help? Use these sentences to help construct your experiment. To form a Customer/Problem Hypothesis: I believe my customer has a problem achieving this goal. To form your Assumptions: 0UVYKLYMVY̸hypothesis to be true, assumption needs to be true. Determine how you will test it: The least expensive way to test my assumption is... To form a Problem/Solution Hypothesis: I believe this solution will result in XHU[PÄHISLV[JVTL. To identify your Riskiest Assumption: The assumption with the least amount of data, and core to the viability of my hypothesis is... Determine what success looks like: I will run experiment with # of customers and expect a strong signal from # of customers. Project Name: Team Leader Name: GET OUT OF THE BUILDING! Download Experiment Board and watch case studies at www.javelin.com © 2014 Javelin. You are free to use it and earn money with it as an entrepreneur, consultant, or executive, as long as you are not a software company (the latter need to license it from us). @uxceo http://bit.ly/xB0@Dux ceo grace@QuickMVP.com

6. Step 1 Define Your Hypothesis @uxceo

7. Who is your customer? Must be specific to be easily testable! @uxceo

8. What is his/her problem? Must be specific to the customer @uxceo grace@QuickMVP.com @uxceo

9. Problem INCORRECT: “People are not buying enough cameras” is not a problem someone has CORRECT: “The outdoor adventurer hates the bulkiness” “The amateur photographer doesn’t think his skills are good enough to invest in a DSLR ” Different customers, different problems @uxceo @uxceo grace@QuickMVP.com

10. Problem Hypothesis I believe [my customer] has a problem [achieving this goal] @uxceo @uxceo grace@QuickMVP.com

11. Solution Do not define until the problem is validated Why? Because every problem has multiple solutions @uxceo @uxceo grace@QuickMVP.com

12. Discover @uxceo

13. What is the solution? What will solve your customer’s problem? @uxceo

14. Solution Hypothesis I believe [my solution] will result in [my customer] taking [specific action] @uxceo grace@QuickMVP.com @uxceo

15. Step 2 Plan your experiment @uxceo @uxceo grace@QuickMVP.com

16. What must we LEARN? What are the Core Assumptions of our business? Assumption: A behavior, mentality, or action that needs to happen in order for hypothesis to be true. @uxceo @uxceo grace@QuickMVP.com

17. What must we TEST? Prioritize: Which Assumption is the Riskiest? The one that is most unknown, and crucial to the business @uxceo @uxceo grace@QuickMVP.com

18. Minimum Viable Product Use the least amount of effort to get the most amount of customer feedback @uxceo @uxceo grace@QuickMVP.com

19. What What do must we BUILD we BUILD? to test? 1. Exploration: test validity of problem 2. Pitch: test value of your solution 3. Concierge: test customer satisfaction or behavior @uxceo @uxceo grace@QuickMVP.com

20. How to MEASURE success? Minimum Success Criteria : What’s the weakest outcome you need to see in order to move forward? @uxceo @uxceo grace@QuickMVP.com

21. How to MEASURE success? 1. Cost • Project Cost • Opportunity Cost 2. Existing metrics • Business KPI • Specific user behavior @uxceo @uxceo grace@QuickMVP.com

22. Step 3 Run your experiment @uxceo @uxceo grace@QuickMVP.com

23. @uxceo

24. Step 4 What did you learn? @uxceo @uxceo grace@QuickMVP.com

25. Invalidated? PIVOT! @uxceo

26. Validated? PERSEVERE! @uxceo

27. Keep Testing @uxceo

28. Case Study @uxceo

29. @uxceo Founders UX Help

30. experiment. Experiments Customer Problem Solution Riskiest Assumption Success Criterion Result Decision 1 2 3 4 5 Time Limit: 5 Min Time Limit: 5 Min OZVS]PUNTime Limit: 5 Min Time Limit: 10 Min Hypothesis: solution will result XHU[PÄHISLV[JVTL. Assumption: the least core to the hypothesis is... Boot-strapped startup founders Getting UX feedback No reliable source Interview: 6/10 GET OUT OF THE BUILDING!

31. Exploration Interviews @uxceo

32. Effective Interviews 1. I’m doing some research around... 2. Have you ever [experienced problem]? 3. Tell me about that time, what happened? 4. What did you do to resolve it? 5. How satisfied were you with the outcome? 6. If unsatisfied, how else have you tried to solve it? @uxceo @uxceo grace@QuickMVP.com

33. Collect Data • Level of pain/frustration • Frequency of encounter with pain • Existing behavior to solve problem • Time/money spent searching for a solution @uxceo @uxceo grace@QuickMVP.com

34. Solution Riskiest Assumption Success Criterion Result Decision Learning OZVS]PUNTime Limit: 5 Min Time Limit: 10 Min Hypothesis: solution will result XHU[PÄHISLV[JVTL. Assumption: the least core to the hypothesis is... like: with # of expect a strong customers. No reliable source Interview: 6/10 GET OUT OF THE BUILDING! 9/12 Persevere Friends. Classes. Paid UX designers hourly for help @uxceo

35. experiment. Experiments Customer Problem Solution Riskiest Assumption Success Criterion Result Decision 1 2 3 4 5 Time Limit: 5 Min Time Limit: 5 Min OZVS]PUNTime Limit: 5 Min Time Limit: 10 Min Hypothesis: solution will result XHU[PÄHISLV[JVTL. Assumption: the least core to the hypothesis is... Boot-strapped startup founders Getting UX feedback No reliable source Interview: 6/10 GET OUT OF THE BUILDING! Boot-strapped startup founders Getting UX feedback @uxceo

36. experiment. Experiments Customer Problem Solution Riskiest Assumption Success Criterion Result Decision 1 2 3 4 5 Time Limit: 5 Min Time Limit: 5 Min OZVS]PUNTime Limit: 5 Min Time Limit: 10 Min Hypothesis: solution will result XHU[PÄHISLV[JVTL. Assumption: the least core to the hypothesis is... Boot-strapped startup founders Getting UX feedback No reliable source Interview: 6/10 GET OUT OF THE BUILDING! Boot-strapped startup founders Getting UX feedback Virtual UX feedback Pay for virtual UX feedback Pitch: 8/10 @uxceo

37. @uxceo grace@QuickMVP.com $10 @uxceo

38. No Backend @uxceo grace@QuickMVP.com @uxceo

39. Solution Riskiest Assumption Success Criterion Result Decision Learning OZVS]PUNTime Limit: 5 Min Time Limit: 10 Min Hypothesis: solution will result XHU[PÄHISLV[JVTL. Assumption: the least core to the hypothesis is... like: with # of expect a strong customers. No reliable source Interview: 6/10 GET OUT OF THE BUILDING! Pay for virtual UX feedback Pitch: 8/10 10/10 Test 2nd side of market Don’t mind virtual at all. Price is cheap 9/12 Persevere Friends. Classes. Paid UX designers for help @uxceo

40. experiment. Experiments Customer Problem Solution Riskiest Assumption Success Criterion Result Decision 1 2 3 4 5 Time Limit: 5 Min Time Limit: 5 Min OZVS]PUNTime Limit: 5 Min Time Limit: 10 Min Hypothesis: solution will result XHU[PÄHISLV[JVTL. Assumption: the least core to the hypothesis is... Boot-strapped startup founders Getting UX feedback No reliable source Interview: 6/10 GET OUT OF THE BUILDING! Boot-strapped startup founders Getting UX feedback Virtual UX feedback Pay for virtual UX feedback Pitch: 8/10 UX Designers Getting clients for quick UX consulting @uxceo

41. Boot-strapped startup founders Getting UX feedback Boot-strapped startup founders Getting UX feedback No reliable source No reliable source Interview: 6/10 Interview: 6/10 Boot-strapped startup founders Getting UX feedback Virtual UX feedback Pay for virtual UX feedback Pitch: 8/10 UX Designers Getting clients for quick UX consulting Virtual UX feedback Will consult virtually Pitch: 8/10 experiment. Experiments Customer Problem Solution Riskiest Assumption Success Criterion Result Decision 1 2 3 4 5 Time Limit: 5 Min Time Limit: 5 Min OZVS]PUNTime Limit: 5 Min Time Limit: 10 Min Hypothesis: solution will result XHU[PÄHISLV[JVTL. Assumption: the least core to the hypothesis is... GET OUT OF THE BUILDING! Boot-strapped startup founders Getting UX feedback Virtual UX feedback Pay for virtual UX feedback Pitch: 8/10 UX Designers Getting clients for quick UX consulting Virtual UX feedback Will consult virtually Pitch: 8/10 @uxceo

42. Seriously? @uxceo

43. Solution Riskiest Assumption Success Criterion Result Decision Learning OZVS]PUNTime Limit: 5 Min Time Limit: 10 Min Hypothesis: solution will result XHU[PÄHISLV[JVTL. Assumption: the least core to the hypothesis is... like: with # of expect a strong customers. No reliable source Interview: 6/10 GET OUT OF THE BUILDING! Pay for virtual UX feedback Pitch: 8/10 10/10 Test 2nd side of market Don’t mind virtual at all. Price is cheap Will consult virtually Pitch: 8/10 0/10 Pivot! 9/12 Persevere Friends. Classes. Paid UX designers for help @uxceo

44. Solution Riskiest Assumption Success Criterion Result Decision Learning OZVS]PUNTime Limit: 5 Min Time Limit: 10 Min Hypothesis: solution will result XHU[PÄHISLV[JVTL. Assumption: the least core to the hypothesis is... like: with # of expect a strong customers. No reliable source Interview: 6/10 GET OUT OF THE BUILDING! Pay for virtual UX feedback Pitch: 8/10 10/10 Test 2nd side of market Don’t mind virtual at all. Price is cheap Will consult virtually Pitch: 8/10 0/10 Pivot! No! 5 yrs exp. Established 9/12 Persevere Friends. Classes. Paid UX designers for help @uxceo

45. Boot-strapped startup founders Getting UX feedback Boot-strapped startup founders Getting UX feedback No reliable source No reliable source Interview: 6/10 Interview: 6/10 Boot-strapped startup founders Getting UX feedback Virtual UX feedback Pay for virtual UX feedback Pitch: 8/10 UX Designers Getting clients for quick UX consulting Virtual UX feedback Will consult virtually Pitch: 8/10 Recent grad UX designers experiment. Experiments Customer Problem Solution Riskiest Assumption Success Criterion Result Decision 1 2 3 4 5 Time Limit: 5 Min Time Limit: 5 Min OZVS]PUNTime Limit: 5 Min Time Limit: 10 Min Hypothesis: solution will result XHU[PÄHISLV[JVTL. Assumption: the least core to the hypothesis is... GET OUT OF THE BUILDING! Boot-strapped startup founders Getting UX feedback Virtual UX feedback Pay for virtual UX feedback Pitch: 8/10 UX Designers Getting clients for quick UX consulting Virtual UX feedback Will consult virtually Pitch: 8/10 Recent grad UX designers Getting clients for quick UX consulting @uxceo

46. Boot-strapped startup founders Getting UX feedback Boot-strapped startup founders Getting UX feedback No reliable source No reliable source Interview: 6/10 Interview: 6/10 Boot-strapped startup founders Getting UX feedback Virtual UX feedback Pay for virtual UX feedback Pitch: 8/10 UX Designers Getting clients for quick UX consulting Virtual UX feedback Will consult virtually Pitch: 8/10 Recent grad UX designers Getting clients for quick UX consulting Virtual UX feedback Give quick online feedback Pitch: 60% experiment. Experiments Customer Problem Solution Riskiest Assumption Success Criterion Result Decision 1 2 3 4 5 Time Limit: 5 Min Time Limit: 5 Min OZVS]PUNTime Limit: 5 Min Time Limit: 10 Min Hypothesis: solution will result XHU[PÄHISLV[JVTL. Assumption: the least core to the hypothesis is... GET OUT OF THE BUILDING! Boot-strapped startup founders Getting UX feedback Virtual UX feedback Pay for virtual UX feedback Pitch: 8/10 UX Designers Getting clients for quick UX consulting Virtual UX feedback Will consult virtually Pitch: 8/10 Recent grad UX designers Getting clients for quick UX consulting Virtual UX feedback Give quick online feedback Pitch: 60% @uxceo

47. Pitch @uxceo @uxceo grace@QuickMVP.com

48. Google Ads @uxceo @nycuxpa #ngnycuxpa @uxceo

49. No Backend @uxceo

50. Quick MVP @uxceo grace@QuickMVP.com @uxceo

51. 3 Data Points @uxceo @uxceo @nycuxpa #ngnycuxpa

52. Solution Riskiest Assumption Success Criterion Result Decision Learning OZVS]PUNTime Limit: 5 Min Time Limit: 10 Min Hypothesis: solution will result XHU[PÄHISLV[JVTL. Assumption: the least core to the hypothesis is... like: with # of expect a strong customers. No reliable source Interview: 6/10 GET OUT OF THE BUILDING! 8/10 Persevere Friends. Classes. 2 paid UX designers for help Pay for virtual UX feedback Pitch: 8/10 10/10 Test 2nd side of market Don’t mind virtual at all. Price is cheap Will consult virtually Pitch: 8/10 0/10 Pivot! No! 5 yrs exp. Established Give quick online feedback Pitch: 60% 37/184 emails Persevere Started referring friends @uxceo

53. experiment. Experiments Customer Problem Solution Riskiest Assumption Success Criterion Result Decision 1 2 3 4 5 Time Limit: 5 Min Time Limit: 5 Min OZVS]PUNTime Limit: 5 Min Time Limit: 10 Min Hypothesis: solution will result XHU[PÄHISLV[JVTL. Assumption: the least core to the hypothesis is... Boot-strapped startup founders Getting UX feedback No reliable source Interview: 6/10 GET OUT OF THE BUILDING! Boot-strapped startup founders Getting UX feedback Virtual UX feedback Pay for virtual UX feedback Pitch: 8/10 UX Designers Getting clients for quick UX consulting Virtual UX feedback Will consult virtually Pitch: 8/10 Recent grad UX designers Getting clients for quick UX consulting Virtual UX feedback Give quick online feedback Pitch: 60% Boot-strapped startup founders Getting UX feedback Virtual UX feedback Online platform is sufficient Concierge: 60% satisfied @uxceo

54. Customer Submits Request UX Designer gets email and submits feedback Concierge @uxceo grace@QuickMVP.com @uxceo

55. Customer Submits Request UX Designer gets email and submits feedback Manually @uxceo grace@QuickMVP.com @uxceo

56. Solution Riskiest Assumption Success Criterion Result Decision Learning OZVS]PUNTime Limit: 5 Min Time Limit: 10 Min Hypothesis: solution will result XHU[PÄHISLV[JVTL. Assumption: the least core to the hypothesis is... like: with # of expect a strong customers. No reliable source Interview: 6/10 GET OUT OF THE BUILDING! 8/10 Persevere Friends. Classes. 2 paid UX designers for help Pay for virtual UX feedback Pitch: 8/10 10/10 Test 2nd side of market Don’t mind virtual at all. Price is cheap Will consult virtually Pitch: 8/10 0/10 Pivot! No! 5 yrs exp. Established Give quick online feedback Pitch: 60% 37/184 emails Persevere Started referring friends Online platform is sufficient Concierge: 60% satisfied 20% Iterate Wanted more function-ality @uxceo

57. @uxceo High Frequency Low Pain High Pain Low Frequency Focus

58. @uxc@eolissijean

59. experiment. Experiments Customer Problem Solution Riskiest Assumption Success Criterion Result Decision 1 2 3 4 5 Time Limit: 5 Min Time Limit: 5 Min OZVS]PUNTime Limit: 5 Min Time Limit: 10 Min Hypothesis: solution will result XHU[PÄHISLV[JVTL. Assumption: the least core to the hypothesis is... OpenSky Shoppers Think curators not authentic GET OUT OF THE BUILDING! @uxceo

60. experiment. Experiments Customer Problem Solution Riskiest Assumption Success Criterion Result Decision 1 2 3 4 5 Time Limit: 5 Min Time Limit: 5 Min OZVS]PUNTime Limit: 5 Min Time Limit: 10 Min Hypothesis: solution will result XHU[PÄHISLV[JVTL. Assumption: the least core to the hypothesis is... OpenSky Shoppers Think curators not authentic CMS Block GET OUT OF THE BUILDING! @uxceo

61. experiment. Experiments Customer Problem Solution Riskiest Assumption Success Criterion Result Decision 1 2 3 4 5 Time Limit: 5 Min Time Limit: 5 Min OZVS]PUNTime Limit: 5 Min Time Limit: 10 Min Hypothesis: solution will result XHU[PÄHISLV[JVTL. Assumption: the least core to the hypothesis is... OpenSky Shoppers Think curators not authentic Shoppers will buy GET OUT OF THE BUILDING! @uxceo CMS Block

62. experiment. Experiments Customer Problem Solution Riskiest Assumption Success Criterion Result Decision 1 2 3 4 5 Time Limit: 5 Min Time Limit: 5 Min OZVS]PUNTime Limit: 5 Min Time Limit: 10 Min Hypothesis: solution will result XHU[PÄHISLV[JVTL. Assumption: the least core to the hypothesis is... OpenSky Shoppers Think curators not authentic CMS Block Concierge: Purchase up 20% GET OUT OF THE BUILDING! @uxceo Shoppers will buy

63. @uxc@eolissijean

64. Solution Riskiest Assumption Success Criterion Result Decision Learning OZVS]PUNTime Limit: 5 Min Time Limit: 10 Min Hypothesis: solution will result XHU[PÄHISLV[JVTL. Assumption: the least core to the hypothesis is... like: with # of expect a strong customers. Shoppers will buy from CMS Concierge: Purchase up 20% GET OUT OF THE BUILDING! @uxceo $0 in Revenue No CT

65. Solution Riskiest Assumption Success Criterion Result Decision Learning OZVS]PUNTime Limit: 5 Min Time Limit: 10 Min Hypothesis: solution will result XHU[PÄHISLV[JVTL. Assumption: the least core to the hypothesis is... like: with # of expect a strong customers. Shoppers will buy from CMS Concierge: Purchase up 20% GET OUT OF THE BUILDING! @uxceo $0 in Revenue No CT Shoppers don’t looks at the home feed, only emails

66. experiment. Experiments Customer Problem Solution Riskiest Assumption Success Criterion Result Decision 1 2 3 4 5 Time Limit: 5 Min Time Limit: 5 Min OZVS]PUNTime Limit: 5 Min Time Limit: 10 Min Hypothesis: solution will result XHU[PÄHISLV[JVTL. Assumption: the least core to the hypothesis is... OpenSky Shoppers Think curators not authentic CMS Block Shoppers will buy from CMS GET OUT OF THE BUILDING! @uxceo Concierge: Purchase up 20%

67. experiment. Experiments Customer Problem Solution Riskiest Assumption Success Criterion Result Decision 1 2 3 4 5 Time Limit: 5 Min Time Limit: 5 Min OZVS]PUNTime Limit: 5 Min Time Limit: 10 Min Hypothesis: solution will result XHU[PÄHISLV[JVTL. Assumption: the least core to the hypothesis is... OpenSky Shoppers Think curators not authentic CMS Block Shoppers will buy from CMS GET OUT OF THE BUILDING! Opensky Shoppers Not seeing curator messages @uxceo Concierge: Purchase up 20%

68. experiment. Experiments Customer Problem Solution Riskiest Assumption Success Criterion Result Decision 1 2 3 4 5 Time Limit: 5 Min Time Limit: 5 Min OZVS]PUNTime Limit: 5 Min Time Limit: 10 Min Hypothesis: solution will result XHU[PÄHISLV[JVTL. Assumption: the least core to the hypothesis is... OpenSky Shoppers Think curators not authentic CMS Block Shoppers will buy from CMS GET OUT OF THE BUILDING! Opensky Shoppers Not seeing curator messages @uxceo Concierge: Purchase up 20% Email messages

69. experiment. Experiments Customer Problem Solution Riskiest Assumption Success Criterion Result Decision 1 2 3 4 5 Time Limit: 5 Min Time Limit: 5 Min OZVS]PUNTime Limit: 5 Min Time Limit: 10 Min Hypothesis: solution will result XHU[PÄHISLV[JVTL. Assumption: the least core to the hypothesis is... OpenSky Shoppers Think curators not authentic CMS Block Shoppers will buy from CMS GET OUT OF THE BUILDING! Opensky Shoppers Not seeing curator messages @uxceo Concierge: Purchase up 20% Email messages Shoppers will buy

70. experiment. Experiments Customer Problem Solution Riskiest Assumption Success Criterion Result Decision 1 2 3 4 5 Time Limit: 5 Min Time Limit: 5 Min OZVS]PUNTime Limit: 5 Min Time Limit: 10 Min Hypothesis: solution will result XHU[PÄHISLV[JVTL. Assumption: the least core to the hypothesis is... OpenSky Shoppers Think curators not authentic CMS Block Shoppers will buy from CMS GET OUT OF THE BUILDING! Opensky Shoppers Not seeing curator messages @uxceo Concierge: Purchase up 20% Email messages Shoppers will buy Concierge: Purchase up 20%

71. @uxceo

72. Success! 2x Open Rate 3x Purchases @uxceo

73. Solution Riskiest Assumption Success Criterion Result Decision Learning OZVS]PUNTime Limit: 5 Min Time Limit: 10 Min Hypothesis: solution will result XHU[PÄHISLV[JVTL. Assumption: the least core to the hypothesis is... like: with # of expect a strong customers. Shoppers will buy from CMS Concierge: Purchase up 20% GET OUT OF THE BUILDING! $0 in Revenue No CT Shoppers don’t looks at the home feed, only emails Shoppers will buy Concierge: Purchase up 20% 300% on purchases Shoppers like videos, photos, from curator in email @uxceo

74. Guess? @uxceo

75. 8 Weeks + 5 People -$77,000 @uxceo

76. 2 Weeks + 2 People +$10,000 @uxceo

77. 1. Form your hypothesis 2. Identify your riskiest assumption 3. Decide how you’ll test it 4. Define success 5. Analyze results and try again @uxceo

78. Thanks! Download resources at Javelin.com @uxceo

Grace Ng | SearchLove San Diego, 'Designing Effective Experiments for Product Decisions'

Vous êtes en train de lire un aperçu.

Consultez-les par la suite

Grace Ng | SearchLove San Diego, 'Designing Effective Experiments for Product Decisions'

Plus De Contenu Connexe

Diaporamas pour vous (20)

Les utilisateurs ont également aimé (20)

Similaire à Grace Ng | SearchLove San Diego, 'Designing Effective Experiments for Product Decisions' (20)

Plus par Distilled (20)

Plus récents (20)