GOOD
HYPOTHESIS
TESTING
IS SURPRISING
Timothy Fitz (.com)

Timothy Fitz
Coined Continuous Deployment
Technical Lead at IMVU
CTO at Canvas
Software Consultant
TimothyFitz.com
SystemsLive.com
@TimothyFitz

WHICH SHADE OF BLUE?
Buy Buy Buy
Buy Buy Buy
Buy Buy Buy
Buy Buy Buy

MULTI-ARMED BANDIT

GOOD HYPOTHESIS
• Documented
• Testable
• Actionable
• Interesting
• Surprising

DOCUMENTED
NCSU Libraries Mobile Redesign for 2012
Objective
The redesign of the NCSU Libraries Mobile website will improve the extensibility of the service and introduce several user experience
enhancements to the existing set of mobile library services.
Requirements
● Redesign the homepage interface in a way that enables deployment of new apps with minimal effort
● Migrate mobile dev framework from forked MIT Mobile Web framework to jQuery Mobile framework
● Implement editorial recommendations on app names, taglines, and graphic identity; update the color scheme to be consistent with
the NCSU Libraries website
● Perform a comprehensive review of the usage stats; use this information to remove underutilized services
● Maintain or improve usage data collection; improve usage data reporting tools
● Retire legacy MoboiLIB service
Out of Scope
● Development of brand new apps
● Deep redesign of existing apps
● Custom iPad mobile website
● Native app deployment (iOS/Android)
Team
Project Manager: Tito Sierra
UI/UX Lead: Adam Rogers
Tech Lead: Cory Lown
Schedule
Launch: 2012.12.08

DOCUMENTED
Hypothesis
Students want to use their phones to track their
outstanding library books.

The details really matter.
Tiny code changes can change
results dramatically.

TESTABLE
• Are you collecting the data?
• Can you collect and analyze it quickly?
• Is it statistically significant?

HIGHTEMPOTESTING
3 experiments per week!

Hypothesis:
Adding Group Chat (a 12-month effort) will result in
a 25% increase in 3-month retention.

Hypothesis: Users want to chat with strangers.
Hypothesis: Users want to chat with strangers
about specific topics.
Hypothesis: Users want a directory of group chats.
Hypothesis: Users who group chat will
have higher retention.

Hypothesis: Users want to chat with strangers.
Hypothesis: Users want to chat with strangers
about specific topics.
Hypothesis: Users want a directory of group chats.
Hypothesis: Users who group chat will
have higher retention.
✓

Hypothesis: Users want to chat with strangers.
Hypothesis: Users want to chat with strangers
about specific topics.
Hypothesis: Users want a directory of group chats.
Hypothesis: Users who group chat will
have higher retention.
✓

Hypothesis: Users want to chat with strangers.
Hypothesis: Users want to chat with strangers
about specific topics.
Hypothesis: Users want a directory of group chats.
Hypothesis: Users who group chat will
have higher retention.

ACTIONABLE
The results of testing your
hypothesis cause action.

ACTIONABLE
Hypothesis: Adding a new payment method will
increase overall revenue by 5%
• If confirmed, we will keep the payment method

INTERESTING
The action you take is
important and
dependent on the outcome.

Hypothesis: Adding a new payment method will
increase overall revenue by 5%
• If confirmed, we will keep the payment method
• If denied, we will keep the payment method
INTERESTING+DOCUMENTED

GOOD PRODUCT MANAGERS
Can rationalize ANY action
from ANY outcome.
:( :( :(

Hypothesis: Adding a new payment method will
increase overall revenue by 2%
• If confirmed, we will keep the payment method
• If denied, we will DELETE the payment method
INTERESTING+DOCUMENTED

SURPRISING
Bayesian definition of surprise:
“The total number of wows experienced when
simultaneously considering all models is obtained
through the integration over the model class.”
http://ilab.usc.edu/surprise/

SURPRISING
surprise
=
change in beliefs

SURPRISING
The results of your hypothesis tests
should frequently and significantly
change your beliefs about your product!

Don’t let a single hypothesis
test invalidate
your whole vision.

GOOD HYPOTHESIS
• Documented
• Testable
• Actionable
• Interesting
• Surprising