Getting Started with Product Analytics - A 101 Implementation Guide for Beginner and Aspiring PMs

1. Getting Started with Product Analytics
A Workshop for Beginner & Aspiring Product Managers
March 19, 2017 (Sunday)
UpGrad Xchange, Bangalore

2. What is all the fuss about?
Making a case for product analytics & feedback loops
“In God we
trust. Everyone
else must bring
along data.”
Proctor & Gamble operates on this
“You cannot
improve upon
something you
can’t measure.”
Pretty obvious no?
“Plan - Design - Spec - Build - Launch - Measure - Iterate”
Typical product cycle for PMs
“Without data,
you’re just
another person
with an opinion.”
You don’t want to fight on opinions!

3. PMs and data (hence,
analytics) are interrelated
Data is all pervasive!
Let’s take a PM walk through the how products
are developed to understand why you cannot fly
blind when building a product or it features.
Associated Costs with Product Development
• Time to build & grow
• Money to build & grow
• Resources to build & grow
• Opportunity Cost (!)
often ignored

4. Meet the “Empathy Debt”
A gauge of whether you understand your users and how they use your product
Let customers do the
talking for you
Customer research, interviews, ethnography
(External Feedback Loop)
Let numbers do the
talking for you
Product & Marketing Analytics, Deductions
(Internal Feedback Loop)

5. Why is it absolutely critical to get your product analytics implementation right?
“Data is the new oil.
It’s only useful
when it’s refined.
Jess Greenwood, Contagious
Getting it right
Measurement vs Analytics

6. How analytics data is being used to drive product & business decisions
Real-World Examples

7. Let’s all get on the same page
Some essentials before we begin…
• What will we NOT discuss about?
• Data Analytics Techniques
• Data Mining & Manipulation
• Data Analytics Algorithms
• and so on…
• What will we discuss about?
• Product Analytics
• Product Metrics
• User-level vs aggregate
• Marketing & Campaign Metrics
• Implementing product analytics
• What should you keep in mind?
• Datapoints - Identifier, Attributes,
Derived Metrics, Dimensions
• Definition – Sessions, Events, User
• Paradigms – Segments, Cohorts,
Funnels, Experiments
• Reporting and Visualization
• Analysis, Hypothesis Verification
• Analytics Tools
• Correlation between Metrics
• Misleading Metrics
• Vanity vs Real/Actionable metrics

8. Let’s put on our thinking caps!
Quiz Time
• # of Downloads
• App Install Base
• Total Revenue
• # of Daily App Opens
• Session Interval
• Session Length
• # of Daily Users
What do you think of these?
Are these vanity or insightful metrics?

9. Startups (or SMBs) vs Established Enterprises
Implementing Product Analytics
STARTUP / SMB
• Starved of capital, analysts and engineering
resources, hence prefer an out-of-box 3P solution
• Focus is on integration to hit the ground running
• Often different analytics tools each plugged into
different properties
ENTERPRISES
• Abundance of capital and resources, hence
prefer in-house, custom built solutions also
owing to IP protection & tight data-
sharing/privacy policies
• Often uniform analytics platform powering
different product lines

10. Learning through a Group Activity
Implementing Product Analytics
>
Implementation Pipeline
Plan Design
>
Spec
>
Execute
>
Rollout

11. Learning through a Group Activity
Implementing Product Analytics
Our Case Companions
• Industry: Ecommerce, Shopping
• Product: Physical (Groceries)
• #1 Business Goal: Purchase
• Industry: Real-time Communication
• Product: Virtual, Software
• #1 Business Goal: Usage (+Conversion)

12. Learning through a Group Activity
Implementing Product Analytics
Some pointers towards what to measure?
• Acquisition: Downloads, Clicks, Visits etc.
• Activation: Onboarding, Login/Register
• Engagement: category & product views, size select
• Retention: Loyalty/stickiness, repeat visit/purchase
• Referral: Viral growth, Share, Invitation
• Monetization: Conversion, Subscription
• Acquisition: Downloads, Clicks etc.
• Activation: Onboarding, Register
• Engagement: Opens, # of chats, calls etc.
• Retention: Frequency of users coming back, duration etc.
• Referral: Viral growth, Share, Invitation
• Monetization: Subscription, In-app purchases

13. Commandment #1
Planning Your Product Analytics
What business
decisions and actions
should your analytics
implementation aid?
“ “
How does analytics tie up with larger business goals?

14. Commandment #1 (contd.)
Planning Your Product Analytics
Examples of Business Goals / Decisions
• Product Usage
• Effectiveness of product discovery properties (CTRs etc.)
• User flow between screens
• Efficacy of Traffic & Marketing Channels
• Traffic Sources – effectiveness and quality
• Attribution of a sale to a traffic source
• Merchandise Performance
• Product impressions, views/clicks, conversions
• Product categories performance
• Communication & Campaign Management
• Single tool vs different tools for marketing campaigns
• Real-time performance tracking of marketing campaigns
What business
decisions and actions
should your analytics
implementation aid?
“ “

15. Commandment #2
Planning Your Product Analytics
Why does it matter?
• Product requirements
• Delivery timelines
• Selection of analytics tools
• Infrastructure
• Program management
• Engineering responsibilities
• Single vs multiple code bases
• Sync between platform-specific PMs
• Sync in terms of data points, definitions, terminology, event attributes,
reporting dashboards etc.
• Unified Campaign Management
• Adoption costs by stakeholders
Concept of “Earned Differences”
Does your analytics
implementation span
across platforms or is
limited only to one?
“ “
You’d most likely not be revisiting
this fundamental consideration.

16. Commandment #3
Planning Your Product Analytics
Become a “Business PM” at this stage - Each stakeholder will use
your implementation differently as per their needs.
• Product team
• Engineering team
• Business Analysts
• Data Scientists
• Business / Revenue team
• Marketing team
• Digital and Growth team
• Operations team
• Category / Merchandise team
• Customer Support team
• Management & Executive Leadership team
Gather business requirements NOT data points to empower stakeholders.
PM needs to understand needs of each function to the determine:
• Data points to be collected to fulfil requirements
• Aggregation, reporting and visualization to be done thereafter
Double down on
gathering business
requirements from
different functions or
stakeholders.
“ “
Listing down their requirements
should come naturally to you.

17. Commandment #4
Planning Your Product Analytics
Build vs Buy: Choose
the analytics tool and
infrastructure best
suited to satisfy your
requirements.
“ “
What should you keep in mind?
• Native Platform Support
• Designed to play equally well across platforms?
• Support for unifying data across platforms?
• Ease of Instrumentation
• How easy is it to integrate in your codebase?
• Is basic out-of-box reporting possible with minimal effort?
• Is the implementation scalable as your platform expands?
• Will the tool support your present & future scale?
• Tier-pricing
• Processing scale
Huge decision - once implemented, you’ll
stick with the tool for at least a few years!

18. Commandment #4 (contd.)
Planning Your Product Analytics
• Does it address your data processing needs?
• Events and attributes are hygiene
• Does it support user-level profiles?
• How many default and custom dimensions?
• Does it unify data signals and profiles across platforms?
• Reporting & Visualization Support
• How easy is it to setup reporting in the tool?
• What kind of reports are supported?
• What technical expertise is required to operate the tool?
• How sharply can you segment your reports?
• What level of pivoting and aggregation is supported?
• Does it support private/public reports/dashboards?
• Does it support “smart” dashboards (eg: derived metrics?)
• Does it support exporting data for offline consumption?
Build vs Buy: Choose
the analytics tool and
infrastructure best
suited to satisfy your
requirements.
“ “

19. Commandment #4 (contd.)
Planning Your Product Analytics
• Action Orientation
• Can you configure marketing interventions, campaigns,
communication (eg: emails, push notifications etc.)?
• What behavioural and profile conditions can be used?
• Can the action be taken in real-time?
• Are there nuances to the data on which you’ll take action?
• Support for Ingesting External Data
• Does the tool allow you to upload data sitting elsewhere?
• Particularly useful if you’ve historic data
• Also if you’re enriching campaigns from outside info
• User Management
• Support for user management & different access rights?
• You would not want your employees and vendors both to
have access to the same view of data!
Build vs Buy: Choose
the analytics tool and
infrastructure best
suited to satisfy your
requirements.
“ “

20. Commandment #4 (contd.)
Planning Your Product Analytics
• Automation
• Does the tool support putting conditional alerts?
• Can it send an automatic summary of metrics & reports?
• Programmability and API Support
• Does it support APIs to access data & run campaigns?
• Raw Data Access
• Does it let you consume full raw data eg: clickstreams?
• Is it provided in real-time, near real-time or deferred?
• Integration and Technical Support
• Will your provider handhold you during integration?
• How responsive will support be? Is there team in India?
• What level of support if free vs paid?
Build vs Buy: Choose
the analytics tool and
infrastructure best
suited to satisfy your
requirements.
“ “

21. Commandment #5
Planning Your Product Analytics
Back it up!
At least 2 tools (maybe one free and another paid) so that:
• You can compare and cross-check data
• You can troubleshoot any discrepancies
• Tools stay available to fight downtime/outages
• You can compensate for missing features in each tool (for eg:
visualization capabilities vs real-time data collection
capabilities)
Always back up your
primary analytics
implementation with
a secondary one!
“ “
Never ever put all your eggs in only
a single basket.

22. Commandment #6
Design, Spec & Execute the Implementation
I can guarantee that reading documentation, user guides is worth
a PM’s time to avoid getting caught unaware down the road.
Know the tools you’ve
chosen inside out. Do
not assume or leave
any doubt about how
they work.
“ “
Yes, it means reading the dev docs
and user guides as each tool
internally behaves little different.

23. Commandment #6 (contd.)
Design, Spec & Execute the Implementation
It’s worth investing one’s time to understand the following:
• Definitions of Common Terms
• Tools always promise a very brief integration period
• It’s essential to understand (and not assume!) what the
metrics mean in the context of your tool
• Sessions, Users, Browsers, Visitors, Bounce
• Attribution Window, Lookback Window
• Funnel definition
• Reporting, Dashboard’ing and Visualization Capabilities:
• How many and what pivots do you need?
• Which pivots and how many are supported in dashboard?
• Can your engineers tactfully get more out of the tool?
Know the tools you’ve
chosen inside out. Do
not assume or leave
any doubt about how
they work.
“ “

24. Commandment #6 (contd.)
Design, Spec & Execute the Implementation
• Attribution
• A change in attribution model may result in dramatically
different RoI splits between traffic sources.
• What schema(s) of attribution is supported?
• Degree of configurability
• Multi-Platform Behaviour
• Same tool may have different definitions of common terms
and attribution across platforms
• Design and spec needs to account for data points and user
profiles that will be shared across each of your properties
• How you implement can make or break such use cases
Know the tools you’ve
chosen inside out. Do
not assume or leave
any doubt about how
they work.
“ “

25. Commandment #7
Design, Spec & Execute the Implementation
Some pointers to approach this activity:
• Spreadsheet it out
• The tabular format lends a natural structure
• No compromise on detailed spec’ing
• Track engg. status against each scenario/data point
Design and spec your
requirements like a
Yoda. Do not leave
even minute details.
“ “
Being brief or assuming here doesn’t
count, but being crystal clear does.

26. Commandment #7 (contd.)
Design, Spec & Execute the Implementation
• Single living document across platforms
• Something I picked up from my stint at Microsoft
• Living spec also a handy source of future reference
• Scalability for instrumentation
• Extensive instrumentation of your codebase is needed
• Does it make sense to introduce a standard middleware?
• Decide basis the following:
• Scalability - “instrument once, use everywhere”
• Ease of Troubleshooting
• Ease of Correction – correct at only a single place
• Extensibility – add new logic quickly & at low risk
Design and spec your
requirements like a
Yoda. Do not leave
even minute details.
“ “

27. Commandment #7 (contd.)
Design, Spec & Execute the Implementation
• Translate all known use-cases into data points
• Which events will you trigger?
• How many & which dimensions to instrument for?
• What information is required for every individual event?
• Which events will be interaction vs non-interaction?
• What will be the micro-funnel steps be?
• Which data points will you capture per user profile?
• In which context should information be captured in?
• Decide how you'd ingest external data
• Which data points sitting in your own systems do you wish
to ingest into the analytics tool you’ve chosen?
• How do you upload these data points to your tool and in
which format?
• How frequently will you need to do this?
• Do you need tool’s API integration with internal systems?
Design and spec your
requirements like a
Yoda. Do not leave
even minute details.
“ “

28. Commandment #8
Design, Spec & Execute the Implementation
Your stakeholders are your “internal customers”
• Essential to share your design - even boring, technical nuances
• Call out limitations (if any) of the tool
• Capabilities of reporting dashboards
• Get stakeholders to sign-off before engineering picks up your
spec for implementation
Do not start your
implementation
without reviewing it
with stakeholders and
getting sign-offs.
“ “
PMs need to behave as business manager
and change managers in their organization.

29. Commandment #9
Design, Spec & Execute the Implementation
Become a “Technical PM” at this stage
• Doesn’t matter how much overview and briefing you provide
• Your developers will almost always have lesser context than you
• Typical queries you may expect:
• Source of data to be sent to the analytics tool
• Exact trigger of data event
• Format required for instrumenting a data-point
Engineering
implementation of
your analytics spec is
high-availability time
for you as a PM.
“ “
Always be around for your developers to
clarify ambiguity.

30. Commandment #10
Finalizing and Deploying Your Implementation
The undisputed onus lies on the PM. The buck stops at you!
While QA engineers are generally more aware of user scenarios
than developers, they will still not have as much context as a PM.Testing & validating
your analytics
implementation is the
PM's job too!
“ “
Is the analytics implementation ready to
power reports, visualizations & campaigns?

31. Commandment #11
Finalizing and Deploying Your Implementation
Dependency alignment with stakeholders is an often neglected
but a critical step. It’s time to ensure data inputs are clean!
Be obsessed about
not polluting your
brand new analytics
implementation.
“ “
Align stakeholders to ensure correct data is
passed on to your analytics infrastructure.

32. Commandment #12
Finalizing and Deploying Your Implementation
Empower all
consumers of
analytics data and
reports through
thorough training.
“ “
A new implementation will never be 100% at
par with what your stakeholders were used to.
Handhold first and then let self-serve follow!

33. Commandment #13
Finalizing and Deploying Your Implementation
A successful rollout represents the beginning of a culture change.
• It’s unlikely you will get it 100% right in the first go
• Improving your analytics is a continuous process.
• With time and new features getting added to your product, with
new business requirements and use cases, you’ll discover:
• Plenty of missing scenarios over time (eg: filters)
• Edge cases not taken care of
• Data corruptions you will find only at scale
• Logic errors
• Reporting/visualization needs that will demand design
changes in the way you’ve implemented the analytics
• New requirements
Remember, enriching
and fine-tuning
analytics is a
continuous process.
“ “

34. Here’s your brownie!
Time to celebrate your first implementation

35. Here’s an elaborate version of what we covered in 2 hours!
Want to read more about the 13 commandments?
• Part 1
5 Things I Learnt While Planning Out My Product
Analytics Implementation
• Part 2
Rolling Up Your Sleeves to Design, Spec &
Execute Your First Product Analytics
• Part 3
This is how I made my 1st product analytics
implementation truly effective

36. Vishrut Shukla
in.linkedin.com/in/vishrutshukla
Thank You! Now get started.
Feedback, comments and suggestions welcome.