SlideShare une entreprise Scribd logo
1  sur  26
Télécharger pour lire hors ligne
GOOD

HYPOTHESIS

TESTING
IS SURPRISING

Timothy Fitz (.com)
Timothy Fitz
Coined Continuous Deployment
Technical Lead at IMVU
CTO at Canvas
Software Consultant
TimothyFitz.com
SystemsLive.com
@TimothyFitz
WHICH SHADE OF BLUE?
Buy Buy Buy
Buy Buy Buy
Buy Buy Buy
Buy Buy Buy
MULTI-ARMED BANDIT
GOOD HYPOTHESIS
• Documented
• Testable
• Actionable
• Interesting
• Surprising
DOCUMENTED
NCSU Libraries Mobile Redesign for 2012
Objective
The redesign of the NCSU Libraries Mobile website will improve the extensibility of the service and introduce several user experience
enhancements to the existing set of mobile library services.
Requirements
● Redesign the homepage interface in a way that enables deployment of new apps with minimal effort
● Migrate mobile dev framework from forked MIT Mobile Web framework to jQuery Mobile framework
● Implement editorial recommendations on app names, taglines, and graphic identity; update the color scheme to be consistent with
the NCSU Libraries website
● Perform a comprehensive review of the usage stats; use this information to remove underutilized services
● Maintain or improve usage data collection; improve usage data reporting tools
● Retire legacy MoboiLIB service
Out of Scope
● Development of brand new apps
● Deep redesign of existing apps
● Custom iPad mobile website
● Native app deployment (iOS/Android)
Team
Project Manager: Tito Sierra
UI/UX Lead: Adam Rogers
Tech Lead: Cory Lown
Schedule
Launch: 2012.12.08
DOCUMENTED
Hypothesis
Students want to use their phones to track their
outstanding library books.
The details really matter.
Tiny code changes can change
results dramatically.
TESTABLE
• Are you collecting the data?
• Can you collect and analyze it quickly?
• Is it statistically significant?
HIGHTEMPOTESTING
3 experiments per week!
Hypothesis:
Adding Group Chat (a 12-month effort) will result in
a 25% increase in 3-month retention.
Hypothesis: Users want to chat with strangers.
Hypothesis: Users want to chat with strangers
about specific topics.



Hypothesis: Users want a directory of group chats.
Hypothesis: Users who group chat will

have higher retention.
Hypothesis: Users want to chat with strangers.
Hypothesis: Users want to chat with strangers
about specific topics.



Hypothesis: Users want a directory of group chats.
Hypothesis: Users who group chat will

have higher retention.
✓
Hypothesis: Users want to chat with strangers.
Hypothesis: Users want to chat with strangers
about specific topics.



Hypothesis: Users want a directory of group chats.
Hypothesis: Users who group chat will

have higher retention.
✓
Hypothesis: Users want to chat with strangers.
Hypothesis: Users want to chat with strangers
about specific topics.



Hypothesis: Users want a directory of group chats.
Hypothesis: Users who group chat will

have higher retention.
ACTIONABLE
The results of testing your
hypothesis cause action.
ACTIONABLE
Hypothesis: Adding a new payment method will
increase overall revenue by 5%
• If confirmed, we will keep the payment method
INTERESTING
The action you take is
important and

dependent on the outcome.
Hypothesis: Adding a new payment method will
increase overall revenue by 5%
• If confirmed, we will keep the payment method
• If denied, we will keep the payment method
INTERESTING+DOCUMENTED
GOOD PRODUCT MANAGERS
Can rationalize ANY action
from ANY outcome.

:( :( :(
Hypothesis: Adding a new payment method will
increase overall revenue by 2%
• If confirmed, we will keep the payment method
• If denied, we will DELETE the payment method
INTERESTING+DOCUMENTED
SURPRISING
Bayesian definition of surprise:





“The total number of wows experienced when
simultaneously considering all models is obtained
through the integration over the model class.”
http://ilab.usc.edu/surprise/
SURPRISING
surprise
=
change in beliefs
SURPRISING
The results of your hypothesis tests
should frequently and significantly
change your beliefs about your product!
Don’t let a single hypothesis
test invalidate

your whole vision.
GOOD HYPOTHESIS
• Documented
• Testable
• Actionable
• Interesting
• Surprising

Contenu connexe

Similaire à Good hypothesis testing is surprising!

Soup to Nuts: Rapid Iteration for Innovative Customer Service
Soup to Nuts: Rapid Iteration for Innovative Customer ServiceSoup to Nuts: Rapid Iteration for Innovative Customer Service
Soup to Nuts: Rapid Iteration for Innovative Customer ServiceChris Bulin
 
Urykea Presentation : Light Color Version
Urykea Presentation : Light Color VersionUrykea Presentation : Light Color Version
Urykea Presentation : Light Color VersionMadlis
 
Urykea Presentation : Dark Color Version
Urykea Presentation : Dark Color VersionUrykea Presentation : Dark Color Version
Urykea Presentation : Dark Color VersionMadlis
 
Embedded Analytics: 5 Steps to App Modernization
Embedded Analytics: 5 Steps to App ModernizationEmbedded Analytics: 5 Steps to App Modernization
Embedded Analytics: 5 Steps to App ModernizationPoojitha B
 
What is User Research? How to Build a User Research Culture
What is User Research? How to Build a User Research CultureWhat is User Research? How to Build a User Research Culture
What is User Research? How to Build a User Research CultureVani Jain
 
Big Data Day LA 2015 - Building a Big Data Culture in the Entertainment Indus...
Big Data Day LA 2015 - Building a Big Data Culture in the Entertainment Indus...Big Data Day LA 2015 - Building a Big Data Culture in the Entertainment Indus...
Big Data Day LA 2015 - Building a Big Data Culture in the Entertainment Indus...Data Con LA
 
PUBLISHING-AND-SUSTAINING-ICT-PROGECTS-FOR-SOCIAL-CHANGE-2.pptx
PUBLISHING-AND-SUSTAINING-ICT-PROGECTS-FOR-SOCIAL-CHANGE-2.pptxPUBLISHING-AND-SUSTAINING-ICT-PROGECTS-FOR-SOCIAL-CHANGE-2.pptx
PUBLISHING-AND-SUSTAINING-ICT-PROGECTS-FOR-SOCIAL-CHANGE-2.pptxKevinKregelCampollo1
 
Horua Presentation : Dark Color Theme
Horua Presentation : Dark Color ThemeHorua Presentation : Dark Color Theme
Horua Presentation : Dark Color Themepunkl.
 
Horua Presentation : Light Color Theme
Horua Presentation : Light Color ThemeHorua Presentation : Light Color Theme
Horua Presentation : Light Color Themepunkl.
 
Rapid User Research - a talk from Agile 2013 by Aviva Rosenstein
Rapid User Research - a talk from Agile 2013 by Aviva RosensteinRapid User Research - a talk from Agile 2013 by Aviva Rosenstein
Rapid User Research - a talk from Agile 2013 by Aviva RosensteinAviva Rosenstein
 
Using analytics in ux design my view
Using analytics in ux design   my viewUsing analytics in ux design   my view
Using analytics in ux design my viewOuti Aramo
 
Best Practices on Optimization with Jessie Becker, CMO of Optimizely
Best Practices on Optimization with Jessie Becker, CMO of OptimizelyBest Practices on Optimization with Jessie Becker, CMO of Optimizely
Best Practices on Optimization with Jessie Becker, CMO of OptimizelyOptimizely
 
Transitioning to-lean-at-infochimps
Transitioning to-lean-at-infochimpsTransitioning to-lean-at-infochimps
Transitioning to-lean-at-infochimpsAsh Maurya
 
Labaiik Marketing Research Plan 2021 (1!).pptx
Labaiik Marketing Research Plan 2021 (1!).pptxLabaiik Marketing Research Plan 2021 (1!).pptx
Labaiik Marketing Research Plan 2021 (1!).pptxJunaidAhmed374694
 
The Ultimate Guide To Embedded Analytics
The Ultimate Guide To Embedded Analytics The Ultimate Guide To Embedded Analytics
The Ultimate Guide To Embedded Analytics Poojitha B
 
From Sticker Sheet to Mature Design System: Improving User Experiences while ...
From Sticker Sheet to Mature Design System: Improving User Experiences while ...From Sticker Sheet to Mature Design System: Improving User Experiences while ...
From Sticker Sheet to Mature Design System: Improving User Experiences while ...Karissa Woodward-Hobson
 
Orchestrating experiments at scale
Orchestrating experiments at scaleOrchestrating experiments at scale
Orchestrating experiments at scalematteo cavucci
 
Burrows orofino unified_cx_04042016
Burrows orofino unified_cx_04042016Burrows orofino unified_cx_04042016
Burrows orofino unified_cx_04042016Patricia Burrows
 
Product development at NerdWallet: Why user research is pivotal
Product development at NerdWallet: Why user research is pivotalProduct development at NerdWallet: Why user research is pivotal
Product development at NerdWallet: Why user research is pivotalUserTesting
 
Elusi Presentation : Light Color Theme
Elusi Presentation : Light Color ThemeElusi Presentation : Light Color Theme
Elusi Presentation : Light Color Themepunkl.
 

Similaire à Good hypothesis testing is surprising! (20)

Soup to Nuts: Rapid Iteration for Innovative Customer Service
Soup to Nuts: Rapid Iteration for Innovative Customer ServiceSoup to Nuts: Rapid Iteration for Innovative Customer Service
Soup to Nuts: Rapid Iteration for Innovative Customer Service
 
Urykea Presentation : Light Color Version
Urykea Presentation : Light Color VersionUrykea Presentation : Light Color Version
Urykea Presentation : Light Color Version
 
Urykea Presentation : Dark Color Version
Urykea Presentation : Dark Color VersionUrykea Presentation : Dark Color Version
Urykea Presentation : Dark Color Version
 
Embedded Analytics: 5 Steps to App Modernization
Embedded Analytics: 5 Steps to App ModernizationEmbedded Analytics: 5 Steps to App Modernization
Embedded Analytics: 5 Steps to App Modernization
 
What is User Research? How to Build a User Research Culture
What is User Research? How to Build a User Research CultureWhat is User Research? How to Build a User Research Culture
What is User Research? How to Build a User Research Culture
 
Big Data Day LA 2015 - Building a Big Data Culture in the Entertainment Indus...
Big Data Day LA 2015 - Building a Big Data Culture in the Entertainment Indus...Big Data Day LA 2015 - Building a Big Data Culture in the Entertainment Indus...
Big Data Day LA 2015 - Building a Big Data Culture in the Entertainment Indus...
 
PUBLISHING-AND-SUSTAINING-ICT-PROGECTS-FOR-SOCIAL-CHANGE-2.pptx
PUBLISHING-AND-SUSTAINING-ICT-PROGECTS-FOR-SOCIAL-CHANGE-2.pptxPUBLISHING-AND-SUSTAINING-ICT-PROGECTS-FOR-SOCIAL-CHANGE-2.pptx
PUBLISHING-AND-SUSTAINING-ICT-PROGECTS-FOR-SOCIAL-CHANGE-2.pptx
 
Horua Presentation : Dark Color Theme
Horua Presentation : Dark Color ThemeHorua Presentation : Dark Color Theme
Horua Presentation : Dark Color Theme
 
Horua Presentation : Light Color Theme
Horua Presentation : Light Color ThemeHorua Presentation : Light Color Theme
Horua Presentation : Light Color Theme
 
Rapid User Research - a talk from Agile 2013 by Aviva Rosenstein
Rapid User Research - a talk from Agile 2013 by Aviva RosensteinRapid User Research - a talk from Agile 2013 by Aviva Rosenstein
Rapid User Research - a talk from Agile 2013 by Aviva Rosenstein
 
Using analytics in ux design my view
Using analytics in ux design   my viewUsing analytics in ux design   my view
Using analytics in ux design my view
 
Best Practices on Optimization with Jessie Becker, CMO of Optimizely
Best Practices on Optimization with Jessie Becker, CMO of OptimizelyBest Practices on Optimization with Jessie Becker, CMO of Optimizely
Best Practices on Optimization with Jessie Becker, CMO of Optimizely
 
Transitioning to-lean-at-infochimps
Transitioning to-lean-at-infochimpsTransitioning to-lean-at-infochimps
Transitioning to-lean-at-infochimps
 
Labaiik Marketing Research Plan 2021 (1!).pptx
Labaiik Marketing Research Plan 2021 (1!).pptxLabaiik Marketing Research Plan 2021 (1!).pptx
Labaiik Marketing Research Plan 2021 (1!).pptx
 
The Ultimate Guide To Embedded Analytics
The Ultimate Guide To Embedded Analytics The Ultimate Guide To Embedded Analytics
The Ultimate Guide To Embedded Analytics
 
From Sticker Sheet to Mature Design System: Improving User Experiences while ...
From Sticker Sheet to Mature Design System: Improving User Experiences while ...From Sticker Sheet to Mature Design System: Improving User Experiences while ...
From Sticker Sheet to Mature Design System: Improving User Experiences while ...
 
Orchestrating experiments at scale
Orchestrating experiments at scaleOrchestrating experiments at scale
Orchestrating experiments at scale
 
Burrows orofino unified_cx_04042016
Burrows orofino unified_cx_04042016Burrows orofino unified_cx_04042016
Burrows orofino unified_cx_04042016
 
Product development at NerdWallet: Why user research is pivotal
Product development at NerdWallet: Why user research is pivotalProduct development at NerdWallet: Why user research is pivotal
Product development at NerdWallet: Why user research is pivotal
 
Elusi Presentation : Light Color Theme
Elusi Presentation : Light Color ThemeElusi Presentation : Light Color Theme
Elusi Presentation : Light Color Theme
 

Plus de Timothy Fitz

Continuous Deployment: Beyond Continuous Delivery
Continuous Deployment: Beyond Continuous DeliveryContinuous Deployment: Beyond Continuous Delivery
Continuous Deployment: Beyond Continuous DeliveryTimothy Fitz
 
Gdc 2010 architecture final slideshare edition
Gdc 2010 architecture final slideshare editionGdc 2010 architecture final slideshare edition
Gdc 2010 architecture final slideshare editionTimothy Fitz
 
Scaling Up Continuous Deployment
Scaling Up Continuous DeploymentScaling Up Continuous Deployment
Scaling Up Continuous DeploymentTimothy Fitz
 
The Hard Problems of Continuous Deployment
The Hard Problems of Continuous DeploymentThe Hard Problems of Continuous Deployment
The Hard Problems of Continuous DeploymentTimothy Fitz
 
Continuous Deployment
Continuous DeploymentContinuous Deployment
Continuous DeploymentTimothy Fitz
 
Continuous Deployment
Continuous DeploymentContinuous Deployment
Continuous DeploymentTimothy Fitz
 
Continuous Deployment
Continuous DeploymentContinuous Deployment
Continuous DeploymentTimothy Fitz
 
Just In Time Scalability Agile Methods To Support Massive Growth Presentation
Just In Time Scalability  Agile Methods To Support Massive Growth PresentationJust In Time Scalability  Agile Methods To Support Massive Growth Presentation
Just In Time Scalability Agile Methods To Support Massive Growth PresentationTimothy Fitz
 

Plus de Timothy Fitz (12)

Continuous Deployment: Beyond Continuous Delivery
Continuous Deployment: Beyond Continuous DeliveryContinuous Deployment: Beyond Continuous Delivery
Continuous Deployment: Beyond Continuous Delivery
 
Gdc 2010 architecture final slideshare edition
Gdc 2010 architecture final slideshare editionGdc 2010 architecture final slideshare edition
Gdc 2010 architecture final slideshare edition
 
Scaling Up Continuous Deployment
Scaling Up Continuous DeploymentScaling Up Continuous Deployment
Scaling Up Continuous Deployment
 
The Hard Problems of Continuous Deployment
The Hard Problems of Continuous DeploymentThe Hard Problems of Continuous Deployment
The Hard Problems of Continuous Deployment
 
Realtime web2012
Realtime web2012Realtime web2012
Realtime web2012
 
Continuous Deployment
Continuous DeploymentContinuous Deployment
Continuous Deployment
 
Continuous Deployment
Continuous DeploymentContinuous Deployment
Continuous Deployment
 
Shdh
ShdhShdh
Shdh
 
Socket.io
Socket.ioSocket.io
Socket.io
 
Shdh
ShdhShdh
Shdh
 
Continuous Deployment
Continuous DeploymentContinuous Deployment
Continuous Deployment
 
Just In Time Scalability Agile Methods To Support Massive Growth Presentation
Just In Time Scalability  Agile Methods To Support Massive Growth PresentationJust In Time Scalability  Agile Methods To Support Massive Growth Presentation
Just In Time Scalability Agile Methods To Support Massive Growth Presentation
 

Dernier

Fields in Java and Kotlin and what to expect.pptx
Fields in Java and Kotlin and what to expect.pptxFields in Java and Kotlin and what to expect.pptx
Fields in Java and Kotlin and what to expect.pptxJoão Esperancinha
 
eAuditor Audits & Inspections - conduct field inspections
eAuditor Audits & Inspections - conduct field inspectionseAuditor Audits & Inspections - conduct field inspections
eAuditor Audits & Inspections - conduct field inspectionsNirav Modi
 
IA Generativa y Grafos de Neo4j: RAG time
IA Generativa y Grafos de Neo4j: RAG timeIA Generativa y Grafos de Neo4j: RAG time
IA Generativa y Grafos de Neo4j: RAG timeNeo4j
 
AI Embracing Every Shade of Human Beauty
AI Embracing Every Shade of Human BeautyAI Embracing Every Shade of Human Beauty
AI Embracing Every Shade of Human BeautyRaymond Okyere-Forson
 
Transforming PMO Success with AI - Discover OnePlan Strategic Portfolio Work ...
Transforming PMO Success with AI - Discover OnePlan Strategic Portfolio Work ...Transforming PMO Success with AI - Discover OnePlan Strategic Portfolio Work ...
Transforming PMO Success with AI - Discover OnePlan Strategic Portfolio Work ...OnePlan Solutions
 
How Does the Epitome of Spyware Differ from Other Malicious Software?
How Does the Epitome of Spyware Differ from Other Malicious Software?How Does the Epitome of Spyware Differ from Other Malicious Software?
How Does the Epitome of Spyware Differ from Other Malicious Software?AmeliaSmith90
 
online pdf editor software solutions.pdf
online pdf editor software solutions.pdfonline pdf editor software solutions.pdf
online pdf editor software solutions.pdfMeon Technology
 
OpenChain Webinar: Universal CVSS Calculator
OpenChain Webinar: Universal CVSS CalculatorOpenChain Webinar: Universal CVSS Calculator
OpenChain Webinar: Universal CVSS CalculatorShane Coughlan
 
Kawika Technologies pvt ltd Software Development Company in Trivandrum
Kawika Technologies pvt ltd Software Development Company in TrivandrumKawika Technologies pvt ltd Software Development Company in Trivandrum
Kawika Technologies pvt ltd Software Development Company in TrivandrumKawika Technologies
 
Optimizing Business Potential: A Guide to Outsourcing Engineering Services in...
Optimizing Business Potential: A Guide to Outsourcing Engineering Services in...Optimizing Business Potential: A Guide to Outsourcing Engineering Services in...
Optimizing Business Potential: A Guide to Outsourcing Engineering Services in...Jaydeep Chhasatia
 
Watermarking in Source Code: Applications and Security Challenges
Watermarking in Source Code: Applications and Security ChallengesWatermarking in Source Code: Applications and Security Challenges
Watermarking in Source Code: Applications and Security ChallengesShyamsundar Das
 
Why Choose Brain Inventory For Ecommerce Development.pdf
Why Choose Brain Inventory For Ecommerce Development.pdfWhy Choose Brain Inventory For Ecommerce Development.pdf
Why Choose Brain Inventory For Ecommerce Development.pdfBrain Inventory
 
Big Data Bellevue Meetup | Enhancing Python Data Loading in the Cloud for AI/ML
Big Data Bellevue Meetup | Enhancing Python Data Loading in the Cloud for AI/MLBig Data Bellevue Meetup | Enhancing Python Data Loading in the Cloud for AI/ML
Big Data Bellevue Meetup | Enhancing Python Data Loading in the Cloud for AI/MLAlluxio, Inc.
 
Cybersecurity Challenges with Generative AI - for Good and Bad
Cybersecurity Challenges with Generative AI - for Good and BadCybersecurity Challenges with Generative AI - for Good and Bad
Cybersecurity Challenges with Generative AI - for Good and BadIvo Andreev
 
Sales Territory Management: A Definitive Guide to Expand Sales Coverage
Sales Territory Management: A Definitive Guide to Expand Sales CoverageSales Territory Management: A Definitive Guide to Expand Sales Coverage
Sales Territory Management: A Definitive Guide to Expand Sales CoverageDista
 
Growing Oxen: channel operators and retries
Growing Oxen: channel operators and retriesGrowing Oxen: channel operators and retries
Growing Oxen: channel operators and retriesSoftwareMill
 
Enterprise Document Management System - Qualityze Inc
Enterprise Document Management System - Qualityze IncEnterprise Document Management System - Qualityze Inc
Enterprise Document Management System - Qualityze Incrobinwilliams8624
 
JS-Experts - Cybersecurity for Generative AI
JS-Experts - Cybersecurity for Generative AIJS-Experts - Cybersecurity for Generative AI
JS-Experts - Cybersecurity for Generative AIIvo Andreev
 
Deep Learning for Images with PyTorch - Datacamp
Deep Learning for Images with PyTorch - DatacampDeep Learning for Images with PyTorch - Datacamp
Deep Learning for Images with PyTorch - DatacampVICTOR MAESTRE RAMIREZ
 

Dernier (20)

Fields in Java and Kotlin and what to expect.pptx
Fields in Java and Kotlin and what to expect.pptxFields in Java and Kotlin and what to expect.pptx
Fields in Java and Kotlin and what to expect.pptx
 
eAuditor Audits & Inspections - conduct field inspections
eAuditor Audits & Inspections - conduct field inspectionseAuditor Audits & Inspections - conduct field inspections
eAuditor Audits & Inspections - conduct field inspections
 
IA Generativa y Grafos de Neo4j: RAG time
IA Generativa y Grafos de Neo4j: RAG timeIA Generativa y Grafos de Neo4j: RAG time
IA Generativa y Grafos de Neo4j: RAG time
 
AI Embracing Every Shade of Human Beauty
AI Embracing Every Shade of Human BeautyAI Embracing Every Shade of Human Beauty
AI Embracing Every Shade of Human Beauty
 
Transforming PMO Success with AI - Discover OnePlan Strategic Portfolio Work ...
Transforming PMO Success with AI - Discover OnePlan Strategic Portfolio Work ...Transforming PMO Success with AI - Discover OnePlan Strategic Portfolio Work ...
Transforming PMO Success with AI - Discover OnePlan Strategic Portfolio Work ...
 
How Does the Epitome of Spyware Differ from Other Malicious Software?
How Does the Epitome of Spyware Differ from Other Malicious Software?How Does the Epitome of Spyware Differ from Other Malicious Software?
How Does the Epitome of Spyware Differ from Other Malicious Software?
 
online pdf editor software solutions.pdf
online pdf editor software solutions.pdfonline pdf editor software solutions.pdf
online pdf editor software solutions.pdf
 
OpenChain Webinar: Universal CVSS Calculator
OpenChain Webinar: Universal CVSS CalculatorOpenChain Webinar: Universal CVSS Calculator
OpenChain Webinar: Universal CVSS Calculator
 
Kawika Technologies pvt ltd Software Development Company in Trivandrum
Kawika Technologies pvt ltd Software Development Company in TrivandrumKawika Technologies pvt ltd Software Development Company in Trivandrum
Kawika Technologies pvt ltd Software Development Company in Trivandrum
 
Optimizing Business Potential: A Guide to Outsourcing Engineering Services in...
Optimizing Business Potential: A Guide to Outsourcing Engineering Services in...Optimizing Business Potential: A Guide to Outsourcing Engineering Services in...
Optimizing Business Potential: A Guide to Outsourcing Engineering Services in...
 
Watermarking in Source Code: Applications and Security Challenges
Watermarking in Source Code: Applications and Security ChallengesWatermarking in Source Code: Applications and Security Challenges
Watermarking in Source Code: Applications and Security Challenges
 
Why Choose Brain Inventory For Ecommerce Development.pdf
Why Choose Brain Inventory For Ecommerce Development.pdfWhy Choose Brain Inventory For Ecommerce Development.pdf
Why Choose Brain Inventory For Ecommerce Development.pdf
 
Big Data Bellevue Meetup | Enhancing Python Data Loading in the Cloud for AI/ML
Big Data Bellevue Meetup | Enhancing Python Data Loading in the Cloud for AI/MLBig Data Bellevue Meetup | Enhancing Python Data Loading in the Cloud for AI/ML
Big Data Bellevue Meetup | Enhancing Python Data Loading in the Cloud for AI/ML
 
Cybersecurity Challenges with Generative AI - for Good and Bad
Cybersecurity Challenges with Generative AI - for Good and BadCybersecurity Challenges with Generative AI - for Good and Bad
Cybersecurity Challenges with Generative AI - for Good and Bad
 
Sales Territory Management: A Definitive Guide to Expand Sales Coverage
Sales Territory Management: A Definitive Guide to Expand Sales CoverageSales Territory Management: A Definitive Guide to Expand Sales Coverage
Sales Territory Management: A Definitive Guide to Expand Sales Coverage
 
Growing Oxen: channel operators and retries
Growing Oxen: channel operators and retriesGrowing Oxen: channel operators and retries
Growing Oxen: channel operators and retries
 
Enterprise Document Management System - Qualityze Inc
Enterprise Document Management System - Qualityze IncEnterprise Document Management System - Qualityze Inc
Enterprise Document Management System - Qualityze Inc
 
JS-Experts - Cybersecurity for Generative AI
JS-Experts - Cybersecurity for Generative AIJS-Experts - Cybersecurity for Generative AI
JS-Experts - Cybersecurity for Generative AI
 
Deep Learning for Images with PyTorch - Datacamp
Deep Learning for Images with PyTorch - DatacampDeep Learning for Images with PyTorch - Datacamp
Deep Learning for Images with PyTorch - Datacamp
 
Salesforce AI Associate Certification.pptx
Salesforce AI Associate Certification.pptxSalesforce AI Associate Certification.pptx
Salesforce AI Associate Certification.pptx
 

Good hypothesis testing is surprising!

  • 2. Timothy Fitz Coined Continuous Deployment Technical Lead at IMVU CTO at Canvas Software Consultant TimothyFitz.com SystemsLive.com @TimothyFitz
  • 3. WHICH SHADE OF BLUE? Buy Buy Buy Buy Buy Buy Buy Buy Buy Buy Buy Buy
  • 5. GOOD HYPOTHESIS • Documented • Testable • Actionable • Interesting • Surprising
  • 6. DOCUMENTED NCSU Libraries Mobile Redesign for 2012 Objective The redesign of the NCSU Libraries Mobile website will improve the extensibility of the service and introduce several user experience enhancements to the existing set of mobile library services. Requirements ● Redesign the homepage interface in a way that enables deployment of new apps with minimal effort ● Migrate mobile dev framework from forked MIT Mobile Web framework to jQuery Mobile framework ● Implement editorial recommendations on app names, taglines, and graphic identity; update the color scheme to be consistent with the NCSU Libraries website ● Perform a comprehensive review of the usage stats; use this information to remove underutilized services ● Maintain or improve usage data collection; improve usage data reporting tools ● Retire legacy MoboiLIB service Out of Scope ● Development of brand new apps ● Deep redesign of existing apps ● Custom iPad mobile website ● Native app deployment (iOS/Android) Team Project Manager: Tito Sierra UI/UX Lead: Adam Rogers Tech Lead: Cory Lown Schedule Launch: 2012.12.08
  • 7. DOCUMENTED Hypothesis Students want to use their phones to track their outstanding library books.
  • 8. The details really matter. Tiny code changes can change results dramatically.
  • 9. TESTABLE • Are you collecting the data? • Can you collect and analyze it quickly? • Is it statistically significant?
  • 11. Hypothesis: Adding Group Chat (a 12-month effort) will result in a 25% increase in 3-month retention.
  • 12. Hypothesis: Users want to chat with strangers. Hypothesis: Users want to chat with strangers about specific topics.
 
 Hypothesis: Users want a directory of group chats. Hypothesis: Users who group chat will
 have higher retention.
  • 13. Hypothesis: Users want to chat with strangers. Hypothesis: Users want to chat with strangers about specific topics.
 
 Hypothesis: Users want a directory of group chats. Hypothesis: Users who group chat will
 have higher retention. ✓
  • 14. Hypothesis: Users want to chat with strangers. Hypothesis: Users want to chat with strangers about specific topics.
 
 Hypothesis: Users want a directory of group chats. Hypothesis: Users who group chat will
 have higher retention. ✓
  • 15. Hypothesis: Users want to chat with strangers. Hypothesis: Users want to chat with strangers about specific topics.
 
 Hypothesis: Users want a directory of group chats. Hypothesis: Users who group chat will
 have higher retention.
  • 16. ACTIONABLE The results of testing your hypothesis cause action.
  • 17. ACTIONABLE Hypothesis: Adding a new payment method will increase overall revenue by 5% • If confirmed, we will keep the payment method
  • 18. INTERESTING The action you take is important and
 dependent on the outcome.
  • 19. Hypothesis: Adding a new payment method will increase overall revenue by 5% • If confirmed, we will keep the payment method • If denied, we will keep the payment method INTERESTING+DOCUMENTED
  • 20. GOOD PRODUCT MANAGERS Can rationalize ANY action from ANY outcome.
 :( :( :(
  • 21. Hypothesis: Adding a new payment method will increase overall revenue by 2% • If confirmed, we will keep the payment method • If denied, we will DELETE the payment method INTERESTING+DOCUMENTED
  • 22. SURPRISING Bayesian definition of surprise:
 
 
 “The total number of wows experienced when simultaneously considering all models is obtained through the integration over the model class.” http://ilab.usc.edu/surprise/
  • 24. SURPRISING The results of your hypothesis tests should frequently and significantly change your beliefs about your product!
  • 25. Don’t let a single hypothesis test invalidate
 your whole vision.
  • 26. GOOD HYPOTHESIS • Documented • Testable • Actionable • Interesting • Surprising