As a product manager, you probably know specific ways to gather data to inform your product decisions, like the ever-popular A/B test. But as they say, you wouldn't put a round peg in a square hole. What about the times when it doesn't make sense to A/B test, because you have too small a sample size? Do you just do it anyway, because your company's product culture requires that you have those numbers?
Tim Herbig will share his hands-on approach to working with analytics in agile product management. Tim will discuss the line between being data-informed versus data-driven. The audience will leave with an analysis toolkit filled with the right data tools for every scenario. Data validation won't be an issue again.
Fun all Day Call Girls in Jaipur 9332606886 High Profile Call Girls You Ca...
Analytics in Action: How to Build Data-Informed Products
1. How To Build Data-Informed Products
Tim Herbig Hannah Flynn
With: Moderated by:
TO USE YOUR COMPUTER'S AUDIO:
When the webinar begins, you will be connected to audio using
your computer's microphone and speakers (VoIP). A headset is
recommended.
Webinar will begin:
11:00 am, PST
TO USE YOUR TELEPHONE:
If you prefer to use your phone, you must select "Use
Telephone" after joining the webinar and call in using the
numbers below.
United States: +1 (415) 655-0052
Access Code: 985-561-257
Audio PIN: Shown after joining the webinar
--OR--
2. Revulytics gives any software producers deep and actionable insight into
who is using their software products and how they are being used, and the
out-of-box analytics that enable them to grow incremental revenue, convert
and retain customers, and make decisions about licensing and cloud
transformation strategies.
3. Click on the Questions panel to
interact with the presenters
www.productmanagementtoday.com/webinar-series/analytics-in-action
www.projectmanagementupdate.com/webinar-series/analytics-in-action
www.businessinnovationbrief.com/webinar-series/analytics-in-action
4. About Tim Herbig
Tim is a product and business leader, as well as a prolific speaker and author of the book ‘Lateral Leadership: A
Practical Guide for Agile Product Managers’. Currently, he’s responsible for the overall sales, product,
engineering, and marketing efforts behind the leading conversion optimization platform iridion.
He previously shaped digital products at XING, Gruner+Jahr, and several smaller startups where he held product
leadership roles for more than seven years. Tim also regularly co-organizes the Product Tank Hamburg meetup to
promote the local product management community.
About Hannah Flynn
Hannah went to The University of Chicago, where she majored in Environmental Studies with a concentration in
Economics and Policy. She now works with Aggregage on social media strategy and webinar production on sites
such as Product Management Today, B2B Marketing Zone, and Customer Experience Update.
8. Product Vision & Strategy
Product Discovery Product Delivery
Analytics
Story
Mappings
User
Interviews
Webinar: How To Build Data-Informed Products by @herbigt
9. Product Vision & Strategy
Product Discovery Product Delivery
Analytics
Story
Mappings
User
Interviews
Webinar: How To Build Data-Informed Products by @herbigt
13. 1. Get caught up in micro-conversions.
2. Confuse data with why people are doing things.
3. Leaving interpretation of data to everyone themselves
(confirmation bias).
4. Confuse today’s behavior with tomorrow’s potential.
Analytics DON’Ts
Webinar: How To Build Data-Informed Products by @herbigt
14. Analytics DOs
Webinar: How To Build Data-Informed Products by @herbigt
1. Sanity check your first assumptions about user
problems.
2. Use it as a foundation for your hypotheses.
3. Objectify understanding about outcomes.
4. Monitor your product’s ongoing performance.
15. Analytics Tool Landscape
Webinar: How To Build Data-Informed Products by @herbigt
Web/App
Data
A/B Testing Heatmaps
DWH/Back-
end
Tracking
Data Viz
16. Analytics Tool Examples
Webinar: How To Build Data-Informed Products by @herbigt
Web/App
Data
● Used by Product
Managers/Analytics Managers
● Mirrors current behavior
17. Analytics Tool Examples
Webinar: How To Build Data-Informed Products by @herbigt
A/B Testing
● Used by Product
Manager/Conversion Optimization
Managers
● Helps to decide which
design/version performs better
● Lots of craftsmanship required to
nail
18. Analytics Tool Examples
Webinar: How To Build Data-Informed Products by @herbigt
Heatmaps
● Used by Product Managers and UX
Designers to spot Usability gaps
● Helps to move closer to the why
behind analytics data
● Requires few “expertise” to distill
information
19. Analytics Tool Examples
Webinar: How To Build Data-Informed Products by @herbigt
DWH/Back-
end
Tracking
● Backbone behind analytics tools
● Used by data analytics managers
● Helpful to double check frontend
analytics data
● Allows to unify data from various
silos
20. Analytics Tool Examples
Webinar: How To Build Data-Informed Products by @herbigt
Data Viz
● Makes analytics data tangible
● Allows to spot similarities between
and compare multiple sources
● Unmatched flexibility for building
custom dashboards
23. Why not?
Webinar: How To Build Data-Informed Products by @herbigt
● People fancy A/B Testing for the sake of it
● Red flags to watch out for:
○ Lack of hypotheses
○ Used to resolve conflicts → getting consensus and
leaving potential on the table for progress
○ Only marginal differences in variations
25. Sharing Results too early
Webinar: How To Build Data-Informed Products by @herbigt
● Tests take time to become valid
● Don’t give in to the nagging of stakeholders to ‘take a
look’
● Instead, make stats and significance calculation visible
26. Solely relying on calculators
Webinar: How To Build Data-Informed Products by @herbigt
● Calculators help you to objectify, but factor in all
environmental factors
● E.g. Competing A/B tests, changing discount campaigns
changing selection criteria
27. Bonus: Validity Rule of thumb
Webinar: How To Build Data-Informed Products by @herbigt
1. 2 weeks minimum duration to capture ‘seasonal’
fluctuations
2. 500 conversions per variation (ideally 1.000)
3. Significance of >95%
28. ● Don’t assume you can automatically influence bottom
line through micro-conversions at top of the funnel
● We tend to run smaller A/B tests without connecting
these KPIs to the overall company goals
● Track and compare cohorts over time to identify impact
of test change on e.g. revenue
Focussing on wrong KPI
Webinar: How To Build Data-Informed Products by @herbigt
29. Focussing on wrong KPI
Webinar: How To Build Data-Informed Products by @herbigt
+20% Top of Funnel
Doesn’t carry through
Increase in Quantity
can lead to decrease in
Quality
= Lower Bottom Line
Conversions
30. Webinar: How To Build Data-Informed Products by @herbigt
Travel Group Syndrome
31. A/B Test Results ≠ User Feedback
Webinar: How To Build Data-Informed Products by @herbigt
● ‘Version A won, so our users like this one more!’
● A/B Tests don’t tell you why people are doing sth.
● It’s the unfortunate discrepancy between what people
say they do and what they actually do.
● Check back it’s mid- to long-term impact on your
product by conducting some 1:1 user interviews digging
into what users actually made them e.g. click more
32. Don’t A/B Test entire Projects
Webinar: How To Build Data-Informed Products by @herbigt
● A/B Tests work for further evolution of existing products
● Don’t help much for new ideas or innovation
● You can’t test business models
● Instead, try quantitative or qualitative experiments to
validate individual core hypotheses
37. Low Traffic B2B Environments
● Don’t look for something what’s not there
● Focus on key metrics like weekly active users or NPS
● 10 well executed and recruited user interviews beat
obsessing over micro metrics
● Make capturing user feedback a habit and core part of
your teams processes
