The document discusses the importance of experiments in validating ideas and making decisions. It provides examples from Urban Ladder of how they designed and conducted experiments to test assumptions and gather user feedback. Key aspects of experiment design discussed are identifying the problem, developing hypotheses to test, running targeted experiments to collect meaningful data, and using the results to inform next steps. Conducting many small, low-cost experiments is advocated over large projects to reduce risks and costs from failed assumptions.

1. Experiments never
killed anybody…
Rajiv Srivatsa (Urban Ladder) / Rajan (Intuit)
#PNCamp
Dec 04 2013

2. 1. Vision, Target, Problem
These are more important to crystallize than the solution or the idea itself…

3. 1. Vision, Target, Problem
• The Urban Ladder Example
• Vision: Beautiful homes for millions of Indians
Tip: The vision is always user-facing!
Bold.
Specific.
Solution-free.
Customer-focused.
Memorable.
• Target: ‘Upper Middle-Class’ ‘Home-proud’ ‘Digitally-Savvy’
‘Urban Indian’ ‘Earning > 1 lak family income’ ‘Well connected’
’Travelled internationally’
Tip: The sharper it is defined, the better it is for engaging with a
clear set of customers for each experiment
• Problem: Getting good quality, well-designed, trust-worthy
furniture at reasonable prices
Tip: Balance going too wide or too narrow

4. 1. Exercise
• Vision, Target Audience & Problem Statement
• Pair with 1 other person
• 2-min to write the following on a post-it
• The vision
• Target audience
• Problem statement
• Your partner to introduce you from the post-it you have
prepared!

5. 2. Why Experiments

6. 2. Experiments – the pre-work
Pick a broad decision / feature / initiative that you last did.
Please answer the following questions (write in a post – it )
•
•
•
•
What was the decision?
Was it a successful one? (Y/N)
How long did it take to realize the learning about the decision?
What were some implicit assumptions made in the decision?

7. 2. Why experiments
observe
idea
analyze
&
present
decision
design
build
launch
&
marketing
users

8. 2. Why experiments
observe idea
analyze
&
present
decision
design
build
launch
&
marketing
users

9. 2. Why experiments
observe
idea
Design
analyze
Build
&
Test
present
decision
design
build
launch
&
marketing
users

10. 2. Why experiments
In the age of rapid changes...
observe
idea
Design
analyze
Build
&
Test
present
From: Decisions by
“Opinions and
Powerpoint”
users
decision
To: Decisions by
“ Experimentation &
Learning”

11. 3. Designing the experiments
There are no failed experiments, only negated hypotheses…
need-gap
analysis
product-market
concept test
[ Testing if the need-gap is a big enough need; understanding priorities ]
[ A feasible brand and product concept that delivers on the vision: how much
are users willing to pay for this ]
leap of faith
assumption
[ Key behavioral assumption about our idea that’s keeping us up at night if this assumption is false, nothing else matters ]
hypothesis
[ Solutions/features that could support our leap of faith assumption . It can
also be considered as a restatement of leap of faith in a numerical way]
repeat tests
experiment
reflect
[ Conditions created to measure behavioral response to learn about the
hypothesis ]
[ Pivot the idea or persevere ]

12. 3. Designing the experiments
need-gap
analysis
product-market
concept test
leap of faith
assumption
• The Urban Ladder Examples
• Confirm need-gap priority
Discovery: Quality and design were more important than price
Tools: SurveyMonkey, Customer Interviews
Dataset: 100 Responses; Over 40 in-depth interviews
• Is the brand promise exciting?
Question: People should relate to the brand-name, logo and promise
Tools: LaunchRock, FB, Customer Interviews
Dataset: 350 sign-ups in 2.5 months; 25 FB shares; Over 100 conversations
• Validate product-market concept fit
Validation: People should like the product at a feasible price-point
Tools: Polls, PPT, Email, A/B Tests
Dataset: Over 40 responses; 25 in-depth interviews

13. 3. Designing the experiments
need-gap
analysis
product-market
fit
leap of faith
assumption
• The Urban Ladder Example
• Hypothesis: People get a sense of the furniture quality online and buy it
• Dataset: Friends and family
• MVP definition: Beta version of the site using outsourced technology,
design integrated with Google Analytics; Calls to check source and
feedback; Basic range in 2 categories; Merchandised products with story
• Target Metrics: If 2% of the visitors buy items for at least Rs. 5k, then we
can say that this experiment can be taken to the next level
• Data Gathering: Spread over a 2 week period to get to the first 25
transactions, largely from family and friends
• Results Analysis: Strong interest in buying product; ability to get a sense
of colors and size from images; strong interest in other cities, categories
Experiment Repeats: Neutralize the audience, check with friends of friends
who probably don’t have direct affinity to people or brand; Test with
service / without service; test repeat rates

14. 3. Designing the experiments
• Write down on a post – it ( 5 min)
• Problem
•
•
•
•
Leap of Faith Assumption
Numerical Hypothesis
Experiment
Metric
• Pair up & review

15. 4. Also important are…
•
•
•
•
•
•
A strong business plan with valid numbers and realistic targets
The right questions to ask during the probe phase
Messaging, Communication, Design, Brand
Focus on doing few things well
Clear milestones
A smile 

16. Q&A
What would you do differently in your work?

17. Credits
Thanks to www.dilbert.com and Scott Adams for the
awesomeness that is Dilbert!

18. MVP Fidelity