As a Product Manager, you benefit from mixing anecdotes and data to have an understanding of customer needs. - You can track the success metrics of your product launch better by defining output metrics vs intermediary signals of progress. - Sometimes you need to drive stakeholder alignment and internal process changes to improve customer experience - You can build your own checklist for product launch and for communicating with customers to ensure you have your metrics ready, your communication is well received, and you are driving the desired customer behavior.

1. Measuring and Improving CX as
a PM by fmr Twilio Staff PM
productschool.com

2. Your Product Management
Certificate Path
Certificates
Product Manager
Certification™
Senior Product Manager
Certification™
Product Leader
Certification™

3. Corporate Training
Level Up Your Team Management Skills
productschool.com

4. Free Product Management Resources
Resources
Events Courses Podcasts
Newsletters Communities eBooks and
Reports

5. Measure &
Improve
Customer
Experience
Harshal Patil, Consultant
and Product Coach

6. 2/21
Where does the content (and I) come from?
● I run Spark Creative Technologies, where we provide Product
Management coaching and business consultancy services.
Most recently, I was a Staff Product Manager at Twilio.
● Some of my foundational experiences were in improving
customer experience and different ways of measuring CX,
which form the basis of today’s content.
● Questions? See follow-up and FAQs at harshalp.com/talk

7. 3/21
● Data + Anecdotes: Merge product analytics, surveys, and interviews
to understand your customers
● Metrics + Signals: Apart from output metrics, use input metrics and
signals to get leading measures of customer experience (CX)
● Product + Process: Beyond product features, work on eliminating
bottlenecks via process changes to improve CX
● Launch + Checklist: Boost your launch success by using an
information checklist
What can you take away today?
🕗

8. 4/21
● In Customers’ KYC Registration funnel
● Normal: Analytics showed some drop-off at each step
● Surprising: But, mid-way through funnel, Analytics
showed 60% drop-off of customers
● Step: Simple step that asks customers to wait for
verification. Once verified, may congratulate on
success...
#1 Where did 60% customers drop-off?
60% ↓

9. 5/21
#1 Why did 60% customers drop-off?
How to understand?
● Qualitative Data from surveys and support requests
● Customers complain about receiving reminders although they
think they’ve completed the KYC registration
● Empathy through Interviews.
Why dropoff?
● Customers thought the step felt (so easy that) like the last step
and they were done.
● Customers thought we would notify them when the wait is
done.
confusions
60%
drop?
Empathy
Quantitative
Analytics sw
Survey, Support
Interviews
Qualitative

10. 6/21
#1 Solving for 60% drop-off
Solved for wait…
● Notify customers after wait time, instead of wait.
● Targeted email reminders for drop-offs.
● Notifications and emails bring customers back at that step.
Solved for step sequence…
● Merge the earlier and later part of the funnel
● Clear Call-To-Action to move forward, instead of congratulations.

11. 7/21
#1 Mix anecdotes and data
● Understand and measure customer experience in 3 ways
Qualitative anecdotes
● Surveys and Customer support
02
● Patterns and high-level “why”
● Clarify hypotheses of “why”
Empathy from conversations
● Interview customers
03
● Understand their mental model
● Figure out “why”
Quantitative data
● Product and Web Analytics
● Identify patterns
● Prioritize problems to solve
● Build hypotheses of “why”
01
Few
Interviews
Lots
of
Data
Lots
of
Why
Very
little
Why
Quantitative
Qualitative
Empathy

12. 8/21
#1 Anecdotes + Data
Merge product analytics, surveys, and interviews to understand your customers

13. 9/21
● 100s of support requests to change billing settings.
● Metric to improve is # of support requests.
● Built and launched a self-service feature.
● Yet no improvement in metric. No reduction in requests.
● How to know whether it is working or what is failing?
#2 Why didn’t the metrics improve?
🕗

14. 10/21
● Support tickets are still high.
● Latency in benefit from launch time.
● Launch is the input and Support tickets reduction is
the output.
● what are early indicators?
#2 Is the launch working or failing?
Launch FAQ views
Support
redirects Page views API usage
Support
reduction
timeline
🕗

15. 11/21
#2 Look at Signals along with Metrics
● Track success using input and output metrics.
● You can control input metrics - launch of product.
● Find signals or early indicators of progress.
NPS
timeline
Revenue
Launches
Sales
Training
hours
Sales
calls
# Support
shortcuts Drop-offs
Page
latency
Support
tickets
Support
costs
Outages
# Blogs
Page
views
🕗

16. 12/21
#2 Metrics + Signals
Apart from output metrics, use input metrics and signals to get leading measures of
customer experience
🕗

17. 13/21
#3 Why are 10,000 customers angry?
● Angry emails from customers that we didn’t respond to them,
even though that was their first email to Customer Support.
● Reviewed few emails. Found a pattern.
● Customers were replying to emails from us. But their replies not
reviewed.
● 10,000 customers emails unopened.

18. 14/21
#3 Product is good but customers are angry?
● Customers are angry although this isn’t a bug in the product.
● What to do?
● Process changes with stakeholders.

19. 15/21
#3 Process changes too, not just Product
● Improve process, not just product. Whatever is the bottleneck.
● Process improvements might be cheaper than product launches.
● Product launches also need process changes. Examples below
metrics
dashboard
reviews
beta product
support
Internal
roadshows
escalation
policies
manual
outage
monitoring
support
redirections

20. 16/21
#3 Product + Process
Beyond product features, work on process changes to improve customer experience
by eliminating bottlenecks

21. 17/21
● Launched a product redesign - Some features removed. Visual look changed.
● “Good product doesn’t need documentation.” or does it?
● “Why inform support? Anyway there won’t be support tickets.”
● “Did the launch succeed?” No metrics before launch so no idea of improvement.
● Lack of launch hygiene.
● Software worked but customers were not aware, so no customer benefit. Launch fail.
#4 Why did the product launch fail?

22. 18/21
● Next time, when removing a feature and disrupting CX, build a checklist.
● Beyond, build a gradual release cycle.
#4 How to make a disruption a success?
● Wrote escalation policy and educated Customer Support.
Support
● Advance heads-up to sales team of large customers.
GTM
● Added information in email, bills, and FAQs but not on web portal.
In-situ
● SQL Queries to tell which customers are impacted.
Identify

23. 19/21
#4 Use a Checklist to boost launch success
● Product launch involves steps beyond deploying code to production.
● But also, too many small steps.
Inspiration - Checklist Manifesto by Dr. Atul Gawande
● How will customers find out about this?
● Announce in conferences, webinars, and periodic marketing emails.
Announce
● What will customers ask?
● Provide information, redirection shortcuts, and escalation policies for
Customer Support.
Support
● How will we sell this to customers?
● Provide information to Sales to share with their customers or answer
questions.
GTM
● Where will customers think about this?
● Add information of a launch to appropriate parts of the product.
In-situ
● Which customers are interested or impacted?
● Based on product analytics, support queries, and sales insights.
Identify

24. 20/21
#4 Launch + Checklist
Boost your launch success by using a information checklist

25. 21/21
Data + Anecdotes: Merge the 3 of product analytics, surveys, and interviews to
understand your customers
Metrics + Signals: Apart from output metrics, use input metrics and signals to
get leading measures of customer experience (CX)
Product + Process: Beyond product features, work on process changes to
improve CX by eliminating bottlenecks
Launch + Checklist: Boost your launch success by using a
information checklist
What can you take away today?
🕗
Questions? See follow-up and FAQs at harshalp.com/talk