Ce diaporama a bien été signalé.
Nous utilisons votre profil LinkedIn et vos données d’activité pour vous proposer des publicités personnalisées et pertinentes. Vous pouvez changer vos préférences de publicités à tout moment.
www.productschool.com
How to Think Product Analytics in PM
Interviews by Amazon Sr PM
Join 35,000+Product
Managers on
Free Resources
Discover great job
opportunities
Job Portal
prdct.school/PSJobPortalprdct.s...
C O U R S E S
Product
Management
Learn the skills you need to land
a Product Manager job
C O U R S E S
Coding
for Managers
Build a website and gain the
technical knowledge to lead
software engineers
C O U R S E S
Data Analytics
for Managers
Learn the skills to understand web
analytics, SQL and machine learning
concepts
C O U R S E S
Learn how to acquire more users
and convert them into clients
Digital Marketing
for Managers
C O U R S E S
UX Design
for Managers
Gain a deeper understanding of your
users and deliver an exceptional end-to-
end expe...
C O U R S E S
For experienced Product Managers
looking to gain strategic skills needed
for top leadership roles
Product
Le...
C O U R S E S
Corporate
Training
Level up your team’s Product
Management skills
Vivek Pandey
T O N I G H T ’ S S P E A K E R
Thinking Product Analytics
Vivek Pandey
1. Who is this guy?
2. Why should I listen to him?
3. Does he have dog pictures? (Oh yes!)
Vivek Pandey
What will we talk about today?
Analytics +
Product Mgmt
● Role of analytics in Product Management
● What does the intervie...
What will we NOT talk about today?
Product
design
● Design an alarm clock for cats
Fermi
problems
● How many golf balls wi...
Part.01
Analytics as a part of Product Management
What does a
Product Manager
do?
What does a Product Manager do?
Ideates
Design new products, services, features, and
improvements
Prioritizes Which ideas ...
What is the interviewer looking for?
Can you identify the levers that can be
controlled?
05
Can you use data to persuade o...
Part.02
Analytics Questions and how to think about them
I group Product analyses into four buckets
Success definition and measurement03
Prioritization02
Opportunity sizing01
Diag...
Example Questions
Bucket Typical Question Structure
Opportunity sizing ● Should we launch product/ feature X?
● How many I...
Product analysis: Opportunity sizing
Identify purchase price05
Identify addressable segments03
Identify the base populatio...
Let’s run through an example:
Should we launch a bicycle-based
food delivery service in Seattle?
There are two parts to the answer
Also to consider (not covered):
● Will this be profitable?
○ Competitors
○ Achievable ma...
Going through the product analysis steps:
In this case, we are only talking about Seattle.
Clarify sources of revenue. Del...
Who has access to your service? Base your calculations on a geographical
area, industry, or demographic.
What’s your unit ...
What’s your unit of measure for your customers? Individuals? Households?
Which measure makes most sense for a service like...
This is the part where you show your creativity!
● How are you identifying your segments? By geography? By income?
○ How b...
So… what fraction of the base population is addressable?
Answer:
Bicycle speed is a limitation, so we are starting with ti...
How many times will people use your service?
● Pick a reasonable unit of measure: Weekly? Monthly?
○ For some services it ...
How many times will people use your service?
Answer:
Let’s assume people who order in are doing so 3 times a month on
aver...
What is the unit price?
● Pick a reasonable price.
○ Base it on industry standards, if similar products exits.
○ Base it o...
What is the unit price?
Answer:
$5 Per delivery sounds reasonable? (Based on UberEats, Amazon
Restaurants etc.)
Identify p...
Total Households = 320K
Addressable Households = 30K
Monthly Purchase Frequency = 3
Revenue Per Delivery = $5
Total Monthl...
$5.4M per year
● Competitors (Uber Eats, Amazon Restaurants, DoorDash etc.)
● What is our competitive advantage?
● What is...
Summary: Opportunity sizing
Identify purchase price05
Identify addressable segments03
Identify the base population02
Clari...
Prioritization, Success Metrics, and Issue
Diagnosis
Questions around these topics -
● Of all the ideas we discussed, whic...
How well do you understand ‘the funnel’?
Total Site
Visitors
100%
View Item
Detail Page
70%
Hit ‘Add to
Cart’ button
20%
G...
Inputs and outputs
Total Site
Visitors
Inputs:
Marketing
SEO
Seasonality
100%
Visit Item
Detail Page
Inputs:
# Items shown...
PMs design features to improve inputs
Total Site
Visitors
Inputs:
Marketing
SEO
Seasonality
100%
Visit Item
Detail Page
In...
Prioritize for impact
View Item
Detail Page
70%
View Item
Detail Page
80%
Feature 1 Improves
Detail Page Views from
70% to...
Total Site
Visitors
Breakdown:
50% Mobile
50% Web
100%
Visit Item
Detail Page
Breakdown:
60% Mobile
80% Web
70%
Hit ‘Buy
B...
Part.03
A/B tests
Control Treatment
A Bvs.
• A ‘Control’ group of users is presented with
an experience ‘A’ that is unchanged from
the statu...
Common mistakes with A/B tests
Picking the wrong test metric01
Financial metrics such as ‘Revenue per order’, ‘Profit’ etc...
Common mistakes with A/B tests
Misunderstanding ‘Statistical Significance’02
You do not run a test until it shows statisti...
Common mistakes with A/B tests
‘Peeking’ at results to draw ‘early data’03
Do not try to read into results until Sample Si...
Common mistakes with A/B tests
Ignoring ‘seasonality’04
On a high traffic website (think Netflix, Google, Amazon), I may r...
When you can’t use A/B tests….
● Legal issues
● Speed / Urgency
● Content based features
A pre/post analysis is a messy bu...
“If you can’t measure it,
you can’t manage it”
- Peter Drucker
“If you can’t measure it,
you can’t manage it”
- Peter Drucker
- William Edwards Deming
“It is wrong to suppose that if
you can’t measure it, you can’t
manage it – a costly myth.”
- William Edwards Deming,
(The...
Thank you!!!
www.productschool.com
Part-time Product Management, Coding, Data Analytics, Digital
Marketing, UX Design, Product Leadersh...
Vous avez terminé ce document.
Prochain SlideShare
What to Upload to SlideShare
Suivant
Prochain SlideShare
What to Upload to SlideShare
Suivant

16

Partager

How to Think Product Analytics in PM Interviews by Amazon Sr PM

Main takeaways:
- Knowing what metrics to measure and how to measure them are key skills for a Product Manager. Interviewers are always going to gauge this aspect.
- How should we think about setting Product Metrics for every situation? How should we think about measuring these?
- What are the strengths and limitations of A/B testing. When can you use it and when should you rely on other methods? What are the different methods for measuring metrics and when to employ those.

How to Think Product Analytics in PM Interviews by Amazon Sr PM

  1. 1. www.productschool.com How to Think Product Analytics in PM Interviews by Amazon Sr PM
  2. 2. Join 35,000+Product Managers on Free Resources Discover great job opportunities Job Portal prdct.school/PSJobPortalprdct.school/events-slack
  3. 3. C O U R S E S Product Management Learn the skills you need to land a Product Manager job
  4. 4. C O U R S E S Coding for Managers Build a website and gain the technical knowledge to lead software engineers
  5. 5. C O U R S E S Data Analytics for Managers Learn the skills to understand web analytics, SQL and machine learning concepts
  6. 6. C O U R S E S Learn how to acquire more users and convert them into clients Digital Marketing for Managers
  7. 7. C O U R S E S UX Design for Managers Gain a deeper understanding of your users and deliver an exceptional end-to- end experience
  8. 8. C O U R S E S For experienced Product Managers looking to gain strategic skills needed for top leadership roles Product Leadership
  9. 9. C O U R S E S Corporate Training Level up your team’s Product Management skills
  10. 10. Vivek Pandey T O N I G H T ’ S S P E A K E R
  11. 11. Thinking Product Analytics Vivek Pandey
  12. 12. 1. Who is this guy? 2. Why should I listen to him? 3. Does he have dog pictures? (Oh yes!)
  13. 13. Vivek Pandey
  14. 14. What will we talk about today? Analytics + Product Mgmt ● Role of analytics in Product Management ● What does the interviewer *really* want to know? Analytics in PM Interviews ● Structure of questions and why they are asked ● Opportunity sizing Measurement & Metrics ● Funnel Design, A/B tests
  15. 15. What will we NOT talk about today? Product design ● Design an alarm clock for cats Fermi problems ● How many golf balls will fit into the Space Needle? Product strategy ● Should Google launch a LinkedIn competitor?
  16. 16. Part.01 Analytics as a part of Product Management
  17. 17. What does a Product Manager do?
  18. 18. What does a Product Manager do? Ideates Design new products, services, features, and improvements Prioritizes Which ideas to build and implement? Persuades Convince leadership, engineering, and other stakeholders Executes Build and launch products.Track, and manage performance
  19. 19. What is the interviewer looking for? Can you identify the levers that can be controlled? 05 Can you use data to persuade others?03 Can you correctly interpret data to make decisions?02 Can you identify what data is relevant?01 Are you realistic about what data can be collected? 04
  20. 20. Part.02 Analytics Questions and how to think about them
  21. 21. I group Product analyses into four buckets Success definition and measurement03 Prioritization02 Opportunity sizing01 Diagnosing issues04
  22. 22. Example Questions Bucket Typical Question Structure Opportunity sizing ● Should we launch product/ feature X? ● How many Italian restaurants are in Seattle? Prioritization ● Given X amount of time/money, which ideas should be prioritized? ● Of all the features we discussed, which one would you build? Success definition and measurement ● Pick one metric to manage the feature you designed? ● How will you measure success after you launch your idea? Diagnosing issues ● Ad revenue dropped by 20%; how will you identify the issue? ● Why are ‘Product Page Views’ down?
  23. 23. Product analysis: Opportunity sizing Identify purchase price05 Identify addressable segments03 Identify the base population02 Clarify scope01 Identify purchase frequency04
  24. 24. Let’s run through an example: Should we launch a bicycle-based food delivery service in Seattle?
  25. 25. There are two parts to the answer Also to consider (not covered): ● Will this be profitable? ○ Competitors ○ Achievable market share ○ Unit economics ○ Does this align with the company’s strategy? ○ Will you find enough riders/delivery-people? What is the size of the opportunity?
  26. 26. Going through the product analysis steps: In this case, we are only talking about Seattle. Clarify sources of revenue. Delivery fee? Commissions? Clarify scope01
  27. 27. Who has access to your service? Base your calculations on a geographical area, industry, or demographic. What’s your unit of measure for your customers? Individuals? Households? Which measure makes most sense for a service like this? Identify the base population02
  28. 28. What’s your unit of measure for your customers? Individuals? Households? Which measure makes most sense for a service like this? Identify the base population02 Answer: # of Households = Population of Seattle / Average Household Size so.. 800K Individuals / 2.5 Individuals per household = 320K Households
  29. 29. This is the part where you show your creativity! ● How are you identifying your segments? By geography? By income? ○ How big is the serviceable area? What are the constraints? ○ How many households are in serviceable areas? ○ How many households order in? How many can afford to? ● So… what fraction of the base population is addressable? Identify addressable segments03
  30. 30. So… what fraction of the base population is addressable? Answer: Bicycle speed is a limitation, so we are starting with tight population and restaurant clusters. Let’s assume serviceable areas are just Downtown, Ballard, Green Lake, and Capitol Hill. I will assume top 50% households by income are ordering in. Adjusting for all these factors, about 30K households are serviceable. Identify addressable segments03
  31. 31. How many times will people use your service? ● Pick a reasonable unit of measure: Weekly? Monthly? ○ For some services it gets very intuitive. ● What assumptions are you making? ○ What are you basing them on? ● Will different segments have a different purchase frequency? ○ Would singles order in more often that families with kids? Note: For physical products, are you considering replacement sales? Identify purchase frequency04
  32. 32. How many times will people use your service? Answer: Let’s assume people who order in are doing so 3 times a month on average. So, now the service will be used 30K times 3 = 90K times per month. Identify purchase frequency04
  33. 33. What is the unit price? ● Pick a reasonable price. ○ Base it on industry standards, if similar products exits. ○ Base it on the next best option, it it is something new to world. Identify purchase price05
  34. 34. What is the unit price? Answer: $5 Per delivery sounds reasonable? (Based on UberEats, Amazon Restaurants etc.) Identify purchase price05
  35. 35. Total Households = 320K Addressable Households = 30K Monthly Purchase Frequency = 3 Revenue Per Delivery = $5 Total Monthly Revenue = 30K * 3 * 5 = $450K Monthly Or $450K *12 = $5.4M annually Bringing it together:
  36. 36. $5.4M per year ● Competitors (Uber Eats, Amazon Restaurants, DoorDash etc.) ● What is our competitive advantage? ● What is the achievable Market Share ● Unit Economics per delivery ● Does this align with the company’s strategy? ● Risks? So, should we do it? ¯_(ツ)_/¯ What is the size of the opportunity? Will this be profitable?
  37. 37. Summary: Opportunity sizing Identify purchase price05 Identify addressable segments03 Identify the base population02 Clarify scope01 Identify purchase frequency04 Of course, discuss competition, seasonality, regional difference, profitable vs. unprofitable segments etc. to complete your answer.
  38. 38. Prioritization, Success Metrics, and Issue Diagnosis Questions around these topics - ● Of all the ideas we discussed, which one would you implement first? ● If you had to pick one metric to manage this feature, which one would it be? ● How will you measure performance of the feature you described? ● Revenue/visits/deliveries dropped by 20%; how will you identify the issue? All these questions have one common theme.
  39. 39. How well do you understand ‘the funnel’? Total Site Visitors 100% View Item Detail Page 70% Hit ‘Add to Cart’ button 20% Go to ‘Checkout’ 7% Complete Purchase 3% This example depicts a simple eCommerce website’s funnel
  40. 40. Inputs and outputs Total Site Visitors Inputs: Marketing SEO Seasonality 100% Visit Item Detail Page Inputs: # Items shown Relevance Price UX quality 70% Hit ‘Buy Button’ Inputs: Product description # of Images Product ratings and reviews 20% Go to ‘Checkout’ Inputs: Calls to action Promotions 7% Complete Purchase Inputs: Card on file # of fields to complete Registered user vs guest 3%
  41. 41. PMs design features to improve inputs Total Site Visitors Inputs: Marketing SEO Seasonality 100% Visit Item Detail Page Inputs: # Items shown Relevance Price UX quality 70% Hit ‘Buy Button’ Inputs: Product description # of Images Product ratings and reviews 20% Go to ‘Checkout’ Inputs: Calls to action Promotions 7% Complete Purchase Inputs: Card on file # of fields to complete Registered user vs guest 3%
  42. 42. Prioritize for impact View Item Detail Page 70% View Item Detail Page 80% Feature 1 Improves Detail Page Views from 70% to 80% Complete Purchase 3% Complete Purchase 3.5% Feature 2 Improves Purchase Completion from 3% to 3.5% R evenue +14% R evenue +17%
  43. 43. Total Site Visitors Breakdown: 50% Mobile 50% Web 100% Visit Item Detail Page Breakdown: 60% Mobile 80% Web 70% Hit ‘Buy Button’ Breakdown: 15% Mobile 25% Web 20% Go to ‘Checkout’ Breakdown: 10% Mobile 4% Web 7% Complete Purchase Breakdown: 4% Mobile 2% Web 3% Troubleshooting - identify the problematic step and narrow down possible causes. The funnel varies by segment, source of traffic etc.
  44. 44. Part.03 A/B tests
  45. 45. Control Treatment A Bvs. • A ‘Control’ group of users is presented with an experience ‘A’ that is unchanged from the status quo. • A ‘Treatment’ group is presented with an alternate experience ‘B’. • The behavior of these two groups is compared over time. A/B Test Primer B Select a test Metric Calculate Sample Size Run Test Analyze Results
  46. 46. Common mistakes with A/B tests Picking the wrong test metric01 Financial metrics such as ‘Revenue per order’, ‘Profit’ etc. have a very high natural degree of variance. These make lousy candidates for A/B testing. When variance is high, you need a huge sample size. Tests can run for years! Speed is key, nobody has time for this. Tip - Pick low variance metrics such as ‘Orders per user’, ‘Click thru rate’ etc. Click thrus and conversion are more telling anyway.
  47. 47. Common mistakes with A/B tests Misunderstanding ‘Statistical Significance’02 You do not run a test until it shows statistical significance. You run an A/B test until you hit the required sample size and then you check the results. At that point - 1. There is a statistically significant difference between Control and Treatment. OR 1. There is no statistically significant difference between Control and Treatment. Tip - Use Evan Miller’s A/B testing blog for calculators.
  48. 48. Common mistakes with A/B tests ‘Peeking’ at results to draw ‘early data’03 Do not try to read into results until Sample Size is reached. We plotted the results of 100 experiments. Even though ~60% of them showed significance at some point, only ONE had a statistically significant difference after sample sizes were reached.
  49. 49. Common mistakes with A/B tests Ignoring ‘seasonality’04 On a high traffic website (think Netflix, Google, Amazon), I may reach sample size in a few hours. But does that give me the full picture? Users may behave differently over weekends. Make sure you run the test long enough to capture nuances.
  50. 50. When you can’t use A/B tests…. ● Legal issues ● Speed / Urgency ● Content based features A pre/post analysis is a messy but acceptable last-resort alternative. i.e. measure performance before and after the change, accounting for other impacts.
  51. 51. “If you can’t measure it, you can’t manage it” - Peter Drucker
  52. 52. “If you can’t measure it, you can’t manage it” - Peter Drucker - William Edwards Deming
  53. 53. “It is wrong to suppose that if you can’t measure it, you can’t manage it – a costly myth.” - William Edwards Deming, (The New Economics, Page 35)
  54. 54. Thank you!!!
  55. 55. www.productschool.com Part-time Product Management, Coding, Data Analytics, Digital Marketing, UX Design, Product Leadership courses and Corporate Training
  • ssuser0228f61

    Sep. 12, 2021
  • syerram

    Aug. 17, 2021
  • ChenHuang3

    Jun. 27, 2021
  • asutosh333

    Jun. 8, 2021
  • GirmaGessesse

    Apr. 20, 2021
  • SouvikMaji2

    Feb. 23, 2021
  • AlokDharpure

    Feb. 14, 2021
  • HamzaIshaq15

    Nov. 24, 2020
  • CarlJamilkowski

    Nov. 21, 2020
  • arnabbasu73

    Jun. 15, 2020
  • noflaps

    May. 30, 2020
  • EvaTang

    Apr. 30, 2020
  • KingChen2008

    Apr. 13, 2020
  • AlexanderChan13

    Mar. 31, 2020
  • DavytheDavidDicochea

    Mar. 5, 2020
  • AnsalMishra1

    Jan. 7, 2020

Main takeaways: - Knowing what metrics to measure and how to measure them are key skills for a Product Manager. Interviewers are always going to gauge this aspect. - How should we think about setting Product Metrics for every situation? How should we think about measuring these? - What are the strengths and limitations of A/B testing. When can you use it and when should you rely on other methods? What are the different methods for measuring metrics and when to employ those.

Vues

Nombre de vues

1 152

Sur Slideshare

0

À partir des intégrations

0

Nombre d'intégrations

17

Actions

Téléchargements

0

Partages

0

Commentaires

0

Mentions J'aime

16

×