Every product decision is an experiment. If we can reframe our thinking from launching campaigns and building features to challenging assumptions and running experiments, then we can increase our productivity and get better results with less effort. Grace will share a structured process for designing effective experiments, and talk about some of the common pitfalls and challenges teams face when running experiments in their companies. Using examples from her work helping product teams within large companies, Grace will reveal a few ways to overcome these challenges.

1. Designing Effective
Experiments for
Product Decisions
Grace Ng
Co-founder, Javelin.com
creators of QuickMVP
@uxceo

2. "Business has only two functions -
marketing and innovation."
- Peter Drucker

3. Build

4. BUILD
EXPERIMENT
Hypothesize
Test
METRICS
PIVOT/PERSEVERE
Analyze
@uxceo
LEARN
MEASURE

5. Javelin Experiment Board
Start here. Brainstorm with stickies, pull it over to the right to start your experiment. Experiments
Customer
Problem
Solution
Riskiest
Assumption
Method &
Success
Criterion
Result &
Decision
Learning
1 2 3 4 5
>OVPZ`VYJZ[VTLY)LHZZWLJPÄJHZWVZZPISL Time Limit: 5 Min
What is the problem? Phrase it from your customer’s perspective. Time Limit: 5 Min
+LÄUL[OLZVS[PVUVUS`HM[LY`VOH]L]HSPKH[LKHWYVISLT^VY[OZVS]PUNTime Limit: 5 Min
List the assumptions that must hold true, for your hypothesis to be true. Time Limit: 10 Min
Need help? Use these sentences to help construct your experiment.
To form a Customer/Problem Hypothesis:
I believe my customer has a
problem achieving this goal.
To form your Assumptions:
0UVYKLYMVY̸hypothesis to be
true, assumption needs to
be true.
Determine how you will test it:
The least expensive way to
test my assumption is...
To form a Problem/Solution Hypothesis:
I believe this solution will result
in XHU[PÄHISLV[JVTL.
To identify your Riskiest Assumption:
The assumption with the least
amount of data, and core to the
viability of my hypothesis is...
Determine what success looks like:
I will run experiment with # of
customers and expect a strong
signal from # of customers.
Project Name: Team Leader Name:
GET OUT OF THE BUILDING!
Download Experiment Board and watch case studies at www.javelin.com © 2014 Javelin. You are free to use it and earn money with it as an entrepreneur, consultant, or executive, as long as you are not a software company (the latter need to license it from us).
@uxceo http://bit.ly/xB0@Dux ceo grace@QuickMVP.com

6. Step 1
Define Your Hypothesis
@uxceo

7. Who is your customer?
Must be specific to be easily testable!
@uxceo

8. What is his/her problem?
Must be specific to the customer
@uxceo grace@QuickMVP.com
@uxceo

9. Problem
INCORRECT:
“People are not buying enough cameras” is not a
problem someone has
CORRECT:
“The outdoor adventurer hates the bulkiness”
“The amateur photographer doesn’t think his skills are
good enough to invest in a DSLR ”
Different customers, different problems
@uxceo @uxceo grace@QuickMVP.com

10. Problem Hypothesis
I believe [my customer] has a
problem [achieving this goal]
@uxceo @uxceo grace@QuickMVP.com

11. Solution
Do not define until the problem is
validated
Why? Because every problem
has multiple solutions
@uxceo @uxceo grace@QuickMVP.com

12. Discover
@uxceo

13. What is the solution?
What will solve your customer’s problem?
@uxceo

14. Solution Hypothesis
I believe [my solution] will
result in [my customer]
taking [specific action]
@uxceo grace@QuickMVP.com
@uxceo

15. Step 2
Plan your experiment
@uxceo @uxceo grace@QuickMVP.com

16. What must we LEARN?
What are the Core Assumptions of
our business?
Assumption:
A behavior, mentality, or action that needs to
happen in order for hypothesis to be true.
@uxceo @uxceo grace@QuickMVP.com

17. What must we TEST?
Prioritize: Which Assumption is
the Riskiest?
The one that is most unknown,
and crucial to the business
@uxceo @uxceo grace@QuickMVP.com

18. Minimum Viable Product
Use the least amount of effort to get
the most amount of customer feedback
@uxceo @uxceo grace@QuickMVP.com

19. What What do must we BUILD we BUILD?
to test?
1. Exploration: test validity of problem
2. Pitch: test value of your solution
3. Concierge: test customer
satisfaction or behavior
@uxceo @uxceo grace@QuickMVP.com

20. How to MEASURE success?
Minimum Success Criteria :
What’s the weakest outcome you
need to see in order to move
forward?
@uxceo @uxceo grace@QuickMVP.com

21. How to MEASURE success?
1. Cost
• Project Cost
• Opportunity Cost
2. Existing metrics
• Business KPI
• Specific user behavior
@uxceo @uxceo grace@QuickMVP.com

22. Step 3
Run your experiment
@uxceo @uxceo grace@QuickMVP.com

23. @uxceo

24. Step 4
What did you learn?
@uxceo @uxceo grace@QuickMVP.com

25. Invalidated?
PIVOT!
@uxceo

26. Validated?
PERSEVERE!
@uxceo

27. Keep Testing
@uxceo

28. Case Study
@uxceo

29. @uxceo
Founders UX Help

30. experiment. Experiments
Customer
Problem
Solution
Riskiest
Assumption
Success
Criterion
Result
Decision
1 2 3 4 5
Time Limit: 5 Min
Time Limit: 5 Min
OZVS]PUNTime Limit: 5 Min
Time Limit: 10 Min
Hypothesis:
solution will result
XHU[PÄHISLV[JVTL.
Assumption:
the least
core to the
hypothesis is...
Boot-strapped
startup
founders
Getting
UX
feedback
No
reliable
source
Interview:
6/10
GET OUT OF THE BUILDING!

31. Exploration Interviews
@uxceo

32. Effective Interviews
1. I’m doing some research around...
2. Have you ever [experienced problem]?
3. Tell me about that time, what happened?
4. What did you do to resolve it?
5. How satisfied were you with the outcome?
6. If unsatisfied, how else have you tried to solve it?
@uxceo @uxceo grace@QuickMVP.com

33. Collect Data
• Level of pain/frustration
• Frequency of encounter with pain
• Existing behavior to solve problem
• Time/money spent searching for a
solution
@uxceo @uxceo grace@QuickMVP.com

34. Solution
Riskiest
Assumption
Success
Criterion
Result
Decision
Learning
OZVS]PUNTime Limit: 5 Min
Time Limit: 10 Min
Hypothesis:
solution will result
XHU[PÄHISLV[JVTL.
Assumption:
the least
core to the
hypothesis is...
like:
with # of
expect a strong
customers.
No
reliable
source
Interview:
6/10
GET OUT OF THE BUILDING!
9/12
Persevere
Friends.
Classes.
Paid
UX designers
hourly for help
@uxceo

35. experiment. Experiments
Customer
Problem
Solution
Riskiest
Assumption
Success
Criterion
Result
Decision
1 2 3 4 5
Time Limit: 5 Min
Time Limit: 5 Min
OZVS]PUNTime Limit: 5 Min
Time Limit: 10 Min
Hypothesis:
solution will result
XHU[PÄHISLV[JVTL.
Assumption:
the least
core to the
hypothesis is...
Boot-strapped
startup
founders
Getting
UX
feedback
No
reliable
source
Interview:
6/10
GET OUT OF THE BUILDING!
Boot-strapped
startup
founders
Getting
UX
feedback
@uxceo

36. experiment. Experiments
Customer
Problem
Solution
Riskiest
Assumption
Success
Criterion
Result
Decision
1 2 3 4 5
Time Limit: 5 Min
Time Limit: 5 Min
OZVS]PUNTime Limit: 5 Min
Time Limit: 10 Min
Hypothesis:
solution will result
XHU[PÄHISLV[JVTL.
Assumption:
the least
core to the
hypothesis is...
Boot-strapped
startup
founders
Getting
UX
feedback
No
reliable
source
Interview:
6/10
GET OUT OF THE BUILDING!
Boot-strapped
startup
founders
Getting
UX
feedback
Virtual
UX
feedback
Pay for
virtual UX
feedback
Pitch:
8/10
@uxceo

37. @uxceo grace@QuickMVP.com
$10
@uxceo

38. No Backend
@uxceo grace@QuickMVP.com
@uxceo

39. Solution
Riskiest
Assumption
Success
Criterion
Result
Decision
Learning
OZVS]PUNTime Limit: 5 Min
Time Limit: 10 Min
Hypothesis:
solution will result
XHU[PÄHISLV[JVTL.
Assumption:
the least
core to the
hypothesis is...
like:
with # of
expect a strong
customers.
No
reliable
source
Interview:
6/10
GET OUT OF THE BUILDING!
Pay for
virtual UX
feedback
Pitch:
8/10
10/10
Test
2nd side
of market
Don’t mind
virtual at all.
Price is cheap
9/12
Persevere
Friends.
Classes.
Paid
UX designers
for help
@uxceo

40. experiment. Experiments
Customer
Problem
Solution
Riskiest
Assumption
Success
Criterion
Result
Decision
1 2 3 4 5
Time Limit: 5 Min
Time Limit: 5 Min
OZVS]PUNTime Limit: 5 Min
Time Limit: 10 Min
Hypothesis:
solution will result
XHU[PÄHISLV[JVTL.
Assumption:
the least
core to the
hypothesis is...
Boot-strapped
startup
founders
Getting
UX
feedback
No
reliable
source
Interview:
6/10
GET OUT OF THE BUILDING!
Boot-strapped
startup
founders
Getting
UX
feedback
Virtual
UX
feedback
Pay for
virtual UX
feedback
Pitch:
8/10
UX
Designers
Getting
clients for
quick UX
consulting
@uxceo

41. Boot-strapped
startup
founders
Getting
UX
feedback
Boot-strapped
startup
founders
Getting
UX
feedback
No
reliable
source
No
reliable
source
Interview:
6/10
Interview:
6/10
Boot-strapped
startup
founders
Getting
UX
feedback
Virtual
UX
feedback
Pay for
virtual UX
feedback
Pitch:
8/10
UX
Designers
Getting
clients for
quick UX
consulting
Virtual
UX
feedback
Will
consult
virtually
Pitch:
8/10
experiment. Experiments
Customer
Problem
Solution
Riskiest
Assumption
Success
Criterion
Result
Decision
1 2 3 4 5
Time Limit: 5 Min
Time Limit: 5 Min
OZVS]PUNTime Limit: 5 Min
Time Limit: 10 Min
Hypothesis:
solution will result
XHU[PÄHISLV[JVTL.
Assumption:
the least
core to the
hypothesis is...
GET OUT OF THE BUILDING!
Boot-strapped
startup
founders
Getting
UX
feedback
Virtual
UX
feedback
Pay for
virtual UX
feedback
Pitch:
8/10
UX
Designers
Getting
clients for
quick UX
consulting
Virtual
UX
feedback
Will
consult
virtually
Pitch:
8/10
@uxceo

42. Seriously?
@uxceo

43. Solution
Riskiest
Assumption
Success
Criterion
Result
Decision
Learning
OZVS]PUNTime Limit: 5 Min
Time Limit: 10 Min
Hypothesis:
solution will result
XHU[PÄHISLV[JVTL.
Assumption:
the least
core to the
hypothesis is...
like:
with # of
expect a strong
customers.
No
reliable
source
Interview:
6/10
GET OUT OF THE BUILDING!
Pay for
virtual UX
feedback
Pitch:
8/10
10/10
Test
2nd side
of market
Don’t mind
virtual at all.
Price is cheap
Will
consult
virtually
Pitch:
8/10
0/10
Pivot!
9/12
Persevere
Friends.
Classes.
Paid
UX designers
for help
@uxceo

44. Solution
Riskiest
Assumption
Success
Criterion
Result
Decision
Learning
OZVS]PUNTime Limit: 5 Min
Time Limit: 10 Min
Hypothesis:
solution will result
XHU[PÄHISLV[JVTL.
Assumption:
the least
core to the
hypothesis is...
like:
with # of
expect a strong
customers.
No
reliable
source
Interview:
6/10
GET OUT OF THE BUILDING!
Pay for
virtual UX
feedback
Pitch:
8/10
10/10
Test
2nd side
of market
Don’t mind
virtual at all.
Price is cheap
Will
consult
virtually
Pitch:
8/10
0/10
Pivot!
No!
5 yrs exp.
Established
9/12
Persevere
Friends.
Classes.
Paid
UX designers
for help
@uxceo

45. Boot-strapped
startup
founders
Getting
UX
feedback
Boot-strapped
startup
founders
Getting
UX
feedback
No
reliable
source
No
reliable
source
Interview:
6/10
Interview:
6/10
Boot-strapped
startup
founders
Getting
UX
feedback
Virtual
UX
feedback
Pay for
virtual UX
feedback
Pitch:
8/10
UX
Designers
Getting
clients for
quick UX
consulting
Virtual
UX
feedback
Will
consult
virtually
Pitch:
8/10
Recent
grad UX
designers
experiment. Experiments
Customer
Problem
Solution
Riskiest
Assumption
Success
Criterion
Result
Decision
1 2 3 4 5
Time Limit: 5 Min
Time Limit: 5 Min
OZVS]PUNTime Limit: 5 Min
Time Limit: 10 Min
Hypothesis:
solution will result
XHU[PÄHISLV[JVTL.
Assumption:
the least
core to the
hypothesis is...
GET OUT OF THE BUILDING!
Boot-strapped
startup
founders
Getting
UX
feedback
Virtual
UX
feedback
Pay for
virtual UX
feedback
Pitch:
8/10
UX
Designers
Getting
clients for
quick UX
consulting
Virtual
UX
feedback
Will
consult
virtually
Pitch:
8/10
Recent
grad UX
designers
Getting
clients for
quick UX
consulting
@uxceo

46. Boot-strapped
startup
founders
Getting
UX
feedback
Boot-strapped
startup
founders
Getting
UX
feedback
No
reliable
source
No
reliable
source
Interview:
6/10
Interview:
6/10
Boot-strapped
startup
founders
Getting
UX
feedback
Virtual
UX
feedback
Pay for
virtual UX
feedback
Pitch:
8/10
UX
Designers
Getting
clients for
quick UX
consulting
Virtual
UX
feedback
Will
consult
virtually
Pitch:
8/10
Recent
grad UX
designers
Getting
clients for
quick UX
consulting
Virtual
UX
feedback
Give quick
online
feedback
Pitch:
60%
experiment. Experiments
Customer
Problem
Solution
Riskiest
Assumption
Success
Criterion
Result
Decision
1 2 3 4 5
Time Limit: 5 Min
Time Limit: 5 Min
OZVS]PUNTime Limit: 5 Min
Time Limit: 10 Min
Hypothesis:
solution will result
XHU[PÄHISLV[JVTL.
Assumption:
the least
core to the
hypothesis is...
GET OUT OF THE BUILDING!
Boot-strapped
startup
founders
Getting
UX
feedback
Virtual
UX
feedback
Pay for
virtual UX
feedback
Pitch:
8/10
UX
Designers
Getting
clients for
quick UX
consulting
Virtual
UX
feedback
Will
consult
virtually
Pitch:
8/10
Recent
grad UX
designers
Getting
clients for
quick UX
consulting
Virtual
UX
feedback
Give quick
online
feedback
Pitch:
60%
@uxceo

47. Pitch
@uxceo
@uxceo grace@QuickMVP.com

48. Google Ads
@uxceo @nycuxpa #ngnycuxpa
@uxceo

49. No Backend
@uxceo

50. Quick MVP
@uxceo grace@QuickMVP.com
@uxceo

51. 3 Data Points
@uxceo
@uxceo @nycuxpa #ngnycuxpa

52. Solution
Riskiest
Assumption
Success
Criterion
Result
Decision
Learning
OZVS]PUNTime Limit: 5 Min
Time Limit: 10 Min
Hypothesis:
solution will result
XHU[PÄHISLV[JVTL.
Assumption:
the least
core to the
hypothesis is...
like:
with # of
expect a strong
customers.
No
reliable
source
Interview:
6/10
GET OUT OF THE BUILDING!
8/10
Persevere
Friends.
Classes.
2 paid
UX designers
for help
Pay for
virtual UX
feedback
Pitch:
8/10
10/10
Test
2nd side
of market
Don’t mind
virtual at all.
Price is cheap
Will
consult
virtually
Pitch:
8/10
0/10
Pivot!
No!
5 yrs exp.
Established
Give quick
online
feedback
Pitch:
60%
37/184
emails
Persevere
Started
referring
friends
@uxceo

53. experiment. Experiments
Customer
Problem
Solution
Riskiest
Assumption
Success
Criterion
Result
Decision
1 2 3 4 5
Time Limit: 5 Min
Time Limit: 5 Min
OZVS]PUNTime Limit: 5 Min
Time Limit: 10 Min
Hypothesis:
solution will result
XHU[PÄHISLV[JVTL.
Assumption:
the least
core to the
hypothesis is...
Boot-strapped
startup
founders
Getting
UX
feedback
No
reliable
source
Interview:
6/10
GET OUT OF THE BUILDING!
Boot-strapped
startup
founders
Getting
UX
feedback
Virtual
UX
feedback
Pay for
virtual UX
feedback
Pitch:
8/10
UX
Designers
Getting
clients for
quick UX
consulting
Virtual
UX
feedback
Will
consult
virtually
Pitch:
8/10
Recent
grad UX
designers
Getting
clients for
quick UX
consulting
Virtual
UX
feedback
Give quick
online
feedback
Pitch:
60%
Boot-strapped
startup
founders
Getting
UX
feedback
Virtual
UX
feedback
Online
platform is
sufficient
Concierge:
60%
satisfied
@uxceo

54. Customer Submits Request
UX Designer gets
email and submits
feedback
Concierge
@uxceo grace@QuickMVP.com
@uxceo

55. Customer Submits Request
UX Designer gets
email and submits
feedback
Manually
@uxceo grace@QuickMVP.com
@uxceo

56. Solution
Riskiest
Assumption
Success
Criterion
Result
Decision
Learning
OZVS]PUNTime Limit: 5 Min
Time Limit: 10 Min
Hypothesis:
solution will result
XHU[PÄHISLV[JVTL.
Assumption:
the least
core to the
hypothesis is...
like:
with # of
expect a strong
customers.
No
reliable
source
Interview:
6/10
GET OUT OF THE BUILDING!
8/10
Persevere
Friends.
Classes.
2 paid
UX designers
for help
Pay for
virtual UX
feedback
Pitch:
8/10
10/10
Test
2nd side
of market
Don’t mind
virtual at all.
Price is cheap
Will
consult
virtually
Pitch:
8/10
0/10
Pivot!
No!
5 yrs exp.
Established
Give quick
online
feedback
Pitch:
60%
37/184
emails
Persevere
Started
referring
friends
Online
platform is
sufficient
Concierge:
60%
satisfied
20%
Iterate
Wanted
more
function-ality
@uxceo

57. @uxceo
High Frequency
Low Pain High Pain
Low Frequency
Focus

58. @uxc@eolissijean

59. experiment. Experiments
Customer
Problem
Solution
Riskiest
Assumption
Success
Criterion
Result
Decision
1 2 3 4 5
Time Limit: 5 Min
Time Limit: 5 Min
OZVS]PUNTime Limit: 5 Min
Time Limit: 10 Min
Hypothesis:
solution will result
XHU[PÄHISLV[JVTL.
Assumption:
the least
core to the
hypothesis is...
OpenSky
Shoppers
Think
curators
not
authentic
GET OUT OF THE BUILDING!
@uxceo

60. experiment. Experiments
Customer
Problem
Solution
Riskiest
Assumption
Success
Criterion
Result
Decision
1 2 3 4 5
Time Limit: 5 Min
Time Limit: 5 Min
OZVS]PUNTime Limit: 5 Min
Time Limit: 10 Min
Hypothesis:
solution will result
XHU[PÄHISLV[JVTL.
Assumption:
the least
core to the
hypothesis is...
OpenSky
Shoppers
Think
curators
not
authentic
CMS
Block
GET OUT OF THE BUILDING!
@uxceo

61. experiment. Experiments
Customer
Problem
Solution
Riskiest
Assumption
Success
Criterion
Result
Decision
1 2 3 4 5
Time Limit: 5 Min
Time Limit: 5 Min
OZVS]PUNTime Limit: 5 Min
Time Limit: 10 Min
Hypothesis:
solution will result
XHU[PÄHISLV[JVTL.
Assumption:
the least
core to the
hypothesis is...
OpenSky
Shoppers
Think
curators
not
authentic
Shoppers
will buy
GET OUT OF THE BUILDING!
@uxceo
CMS
Block

62. experiment. Experiments
Customer
Problem
Solution
Riskiest
Assumption
Success
Criterion
Result
Decision
1 2 3 4 5
Time Limit: 5 Min
Time Limit: 5 Min
OZVS]PUNTime Limit: 5 Min
Time Limit: 10 Min
Hypothesis:
solution will result
XHU[PÄHISLV[JVTL.
Assumption:
the least
core to the
hypothesis is...
OpenSky
Shoppers
Think
curators
not
authentic
CMS
Block
Concierge:
Purchase
up 20%
GET OUT OF THE BUILDING!
@uxceo
Shoppers
will buy

63. @uxc@eolissijean

64. Solution
Riskiest
Assumption
Success
Criterion
Result
Decision
Learning
OZVS]PUNTime Limit: 5 Min
Time Limit: 10 Min
Hypothesis:
solution will result
XHU[PÄHISLV[JVTL.
Assumption:
the least
core to the
hypothesis is...
like:
with # of
expect a strong
customers.
Shoppers
will buy
from CMS
Concierge:
Purchase
up 20%
GET OUT OF THE BUILDING!
@uxceo
$0 in
Revenue
No CT

65. Solution
Riskiest
Assumption
Success
Criterion
Result
Decision
Learning
OZVS]PUNTime Limit: 5 Min
Time Limit: 10 Min
Hypothesis:
solution will result
XHU[PÄHISLV[JVTL.
Assumption:
the least
core to the
hypothesis is...
like:
with # of
expect a strong
customers.
Shoppers
will buy
from CMS
Concierge:
Purchase
up 20%
GET OUT OF THE BUILDING!
@uxceo
$0 in
Revenue
No CT
Shoppers don’t
looks at the
home feed,
only
emails

66. experiment. Experiments
Customer
Problem
Solution
Riskiest
Assumption
Success
Criterion
Result
Decision
1 2 3 4 5
Time Limit: 5 Min
Time Limit: 5 Min
OZVS]PUNTime Limit: 5 Min
Time Limit: 10 Min
Hypothesis:
solution will result
XHU[PÄHISLV[JVTL.
Assumption:
the least
core to the
hypothesis is...
OpenSky
Shoppers
Think
curators
not
authentic
CMS
Block
Shoppers
will buy
from CMS
GET OUT OF THE BUILDING!
@uxceo
Concierge:
Purchase
up 20%

67. experiment. Experiments
Customer
Problem
Solution
Riskiest
Assumption
Success
Criterion
Result
Decision
1 2 3 4 5
Time Limit: 5 Min
Time Limit: 5 Min
OZVS]PUNTime Limit: 5 Min
Time Limit: 10 Min
Hypothesis:
solution will result
XHU[PÄHISLV[JVTL.
Assumption:
the least
core to the
hypothesis is...
OpenSky
Shoppers
Think
curators
not
authentic
CMS
Block
Shoppers
will buy
from CMS
GET OUT OF THE BUILDING!
Opensky
Shoppers
Not seeing
curator
messages
@uxceo
Concierge:
Purchase
up 20%

68. experiment. Experiments
Customer
Problem
Solution
Riskiest
Assumption
Success
Criterion
Result
Decision
1 2 3 4 5
Time Limit: 5 Min
Time Limit: 5 Min
OZVS]PUNTime Limit: 5 Min
Time Limit: 10 Min
Hypothesis:
solution will result
XHU[PÄHISLV[JVTL.
Assumption:
the least
core to the
hypothesis is...
OpenSky
Shoppers
Think
curators
not
authentic
CMS
Block
Shoppers
will buy
from CMS
GET OUT OF THE BUILDING!
Opensky
Shoppers
Not seeing
curator
messages
@uxceo
Concierge:
Purchase
up 20%
Email
messages

69. experiment. Experiments
Customer
Problem
Solution
Riskiest
Assumption
Success
Criterion
Result
Decision
1 2 3 4 5
Time Limit: 5 Min
Time Limit: 5 Min
OZVS]PUNTime Limit: 5 Min
Time Limit: 10 Min
Hypothesis:
solution will result
XHU[PÄHISLV[JVTL.
Assumption:
the least
core to the
hypothesis is...
OpenSky
Shoppers
Think
curators
not
authentic
CMS
Block
Shoppers
will buy
from CMS
GET OUT OF THE BUILDING!
Opensky
Shoppers
Not seeing
curator
messages
@uxceo
Concierge:
Purchase
up 20%
Email
messages
Shoppers
will buy

70. experiment. Experiments
Customer
Problem
Solution
Riskiest
Assumption
Success
Criterion
Result
Decision
1 2 3 4 5
Time Limit: 5 Min
Time Limit: 5 Min
OZVS]PUNTime Limit: 5 Min
Time Limit: 10 Min
Hypothesis:
solution will result
XHU[PÄHISLV[JVTL.
Assumption:
the least
core to the
hypothesis is...
OpenSky
Shoppers
Think
curators
not
authentic
CMS
Block
Shoppers
will buy
from CMS
GET OUT OF THE BUILDING!
Opensky
Shoppers
Not seeing
curator
messages
@uxceo
Concierge:
Purchase
up 20%
Email
messages
Shoppers
will buy
Concierge:
Purchase
up 20%

71. @uxceo

72. Success!
2x Open Rate
3x Purchases
@uxceo

73. Solution
Riskiest
Assumption
Success
Criterion
Result
Decision
Learning
OZVS]PUNTime Limit: 5 Min
Time Limit: 10 Min
Hypothesis:
solution will result
XHU[PÄHISLV[JVTL.
Assumption:
the least
core to the
hypothesis is...
like:
with # of
expect a strong
customers.
Shoppers
will buy
from CMS
Concierge:
Purchase
up 20%
GET OUT OF THE BUILDING!
$0 in
Revenue
No CT
Shoppers don’t
looks at the
home feed,
only
emails
Shoppers
will buy
Concierge:
Purchase
up 20%
300%
on
purchases
Shoppers
like videos,
photos, from
curator in
email
@uxceo

74. Guess?
@uxceo

75. 8 Weeks + 5 People
-$77,000
@uxceo

76. 2 Weeks + 2 People
+$10,000
@uxceo

77. 1. Form your hypothesis
2. Identify your riskiest assumption
3. Decide how you’ll test it
4. Define success
5. Analyze results and try again
@uxceo

78. Thanks!
Download resources at
Javelin.com
@uxceo