Improving AI products with Analytics

1. www.productschool.com
Part-time Product Management, Coding, Data Analytics, Digital
Marketing, UX Design and Product Leadership courses in San
Francisco, Silicon Valley, New York, Santa Monica, Los Angeles,
Austin, Boston, Boulder, Chicago, Denver, Orange County,
Seattle, Bellevue, Washington DC, Toronto, London and Online

2. FREE INVITE
Join 35,000+ Product Managers on

3. +10,000
alumni graduated
across 17
campuses
· San Francisco
· Silicon Valley
· New York
· Los Angeles
· Santa Monica
· Orange County
· Austin
· Boston
· Boulder
· Chicago
· Denver
· Seattle
· Washington DC
· Toronto (Canada)
· London (UK)
· Bellevue
· Online

4. COURSES
Product Management
Learn the skills you need to land a product manager job

5. COURSES
Coding for Managers
Build a website and gain the technical knowledge to lead software engineers

6. COURSES
Data Analytics for Managers
Learn the skills to understand web analytics, SQL and machine learning concepts

7. COURSES
Digital Marketing for Managers
Learn how to acquire more users and convert them into clients

8. COURSES
UX Design for Managers
Gain a deeper understanding of your users and deliver an exceptional end-to-end
experience

9. COURSES
Product Leadership for Managers
For experienced Product Managers looking to gain strategic skills needed for top
leadership roles

10. COURSES
Corporate Training
Level up your team’s product management skills

11. Abhishek Guglani
&
Ramkumar Ravichandran
TONIGHT’S SPEAKER

12. Improve AI using
Analytics
Ramkumar Ravichandran – Data science TLM, Google Shopping
Abhishek Guglani – Director of Product, Visa Inc., Ecommerce
Disclaimer: Participation in this summit is purely on personal basis and is not meant to represent VISA or Google’s position on this or any
other subject and in any form or matter. The talk is based on learning from work across industries and firms. Care has been taken to ensure
no proprietary or work related information of any firm is used in any material.

13. “Service” mindset in Leadership
Simon Sinek - Do you love your wife? 13

14. AI at peak of inflated expectations last year…
https://twitter.com/chucknicecomic/status/744550059447050244 14

15. •Clear Value Prop & “Need based” User Flow
FOMO led to companies missing out on,
•Commercializable
•Policy Compliant
•Scalable & “agile” architecture
•Data Driven Optimization
15

16. 16
https://imgflip.com/i/2uw8bj

17. What separates the “leader” from the pack?
In 2017... ...and now
https://www.dietworkoutfitness.com/tag/alexa-photos.html 17

18. Product Leadership Checklist!
• “Design” thinking- anchored to “User-Needs-Context”
• “Engagement” focus right from start
• “Learn-Listen-Test” launch & ramp
• “Partner” ecosystem & support
• “Loyalty” pricing
18

19. 19
https://imgflip.com/i/2uwarz

20. ExtendManageBuildPrototypeDesignPlan
Going from left to right on PDLC, instead of first
creating product then searching for Customers,
• Critical Review of Existing solutions
(optimization or differentiator)
• Use Cases (Opp, Impact, RoI)
• Goals, Success/Stop Criteria
• Readiness (Customer, Provider,
Regulator, Creator)
• Design iteration decisions (by user, by
needs, by context)
• Tactical: Platform, Program, Process
• Use Case Scoring & Prioritization
• POC- Success/Lessons, RoI
• Optimization/Customization
• Review, Stress Test, UAT
• Plan & Timelines - Milestones
• Evangelize & Engage
• Data Driven Optimization
• Support, Operations & Distribution
• Refine, Revamp or Retire?
• ”Learn-Listen-Test” launch
• Deploy, Monitor & Iterate
• Usage Protocols : Guide & Comply
• Innovation & Upgrade
20

21. Strategy
Data
Tagging
Data
Platform
Reporting
Analytics
Research
Scale
Iterative
Loop
Optimization
…iterating to User-Product-Market fit with
“Learn-Listen-Test” framework
✓ Reporting for “monitoring” KPIs
✓ Analytics provides insights into “user
behavior”
✓ Research context on “motivations”
✓ Testing helps verify the “tactics”
21

22. …appreciating that AI product will have it’s
own analytics maturity curve
https://www.intel.com/content/dam/www/public/us/en/documents/guides/analytics-planning-guide.pdf
22

23. Optimize Product Lifecycle
• Strategy
• Experience
• Development
• Management
Metrics: Click Through Rate, Conversion,
%Happy Path, Speed, Distribution
Minimize Risk
• Decrease in Standard Risk
• Successful Prevention Rate
• New Risk Detection Efficiency
• Rule efficiency: FPR/FNR, Agent Reviews,
Reported
• Implementation cost: CXM, CSS
Metrics: Bad Rate Changes, %bad prevented,
%leak through, business KPI impact
Optimize User Journey
• Campaign Strategy
• Performance Attribution
• Funnel Management (Omni)
• Cost Optimization
• Brand Management
Metrics: Awareness, Sentiment, Adoption,
CPE/CPM/CPC, Engagement, NPS, LTV
Optimize Marketing and/or Sales Process
• Goal Setting, Monitoring & Tweaking
• Prospect Scoring & Prioritization
• Lead Funnel Management: Rate, Speed, Cost
• Retention & Growth
• Turnover
Metrics: Topline, Time to Live, Cost of
Acquisition & Retention, Account growth/ NPS
Optimize Strategy & Operations
• Strategy Development & Execution
• Innovation Delivery
• Performance Tracking & Intervention
• Business Operations
• Resource Investment Decisions (Finance)
• Strategic Research: Competitive Monitoring,
Regulatory, Policies, Legal
Metrics: Earnings Growth, Guidance delivery,
Investor Confidence
Optimizely Technology Delivery Cycle
• Development Prioritization
• Delivery Quality & Monitoring
• Cost of Development
• Platform Management
• Scalability: Compatibility, Detection, Pre-
emption & Prevention
Metrics: Uptime, Performance, #Story points to
Develop/Scale/Iterate, #Bugs/Bug Rate
..and that Analytics will span multiple aspects of
business critical to overall success
23

24. Let’s make all this real with an
illustrative use case...
24

25. HealthBot: How are
you feeling today?
Adam: Stressed.
And I drove too
fast. Was feeling
awful
HealthBot: Is there
anything I can help
you with?
Adam: Yes. And I
feel like a couple
of extra drinks will
take the edge off
Day 7 of chat
HealthBot: How are
you feeling today?
Adam: I feel doing
something extreme
HealthBot: Did you
try exercising?
Adam: Yes I did but
that does not seem
to help anymore
Day 14 of chat
Adam is suffering from depression and found a
chatbot for support...
25

26. HealthBot: How are you
feeling today?
Adam: Stressed. And I
drove too fast. Was feeling
awful
HealthBot: Is there anything
I can help you with?
Adam: Yes. And I feel like a
couple of extra drinks will
take the edge off
Day 7 of chat
HealthBot: How are you
feeling today?
Adam: I feel doing something
extreme
HealthBot: Did you try
exercising
Adam: Yes. I did but that
does not seem to help
anymore
Day 14 of chat
Natural Language
Processing
Conversation Service
Machine Learning & Deep Learning built into
Chat Bot
Chat Related Services
Pre-built conversation
trees
Connectivity with
other services
Machine Learning – Models trained by data
& experts
External/3rd
party services
Internal
services
Instant solutioning without truly “understanding”
the needs & problem causes attrition...
26
Day 21 of chat
HealthBot: How are you
feeling today?
Adam: I am suicidal
HealthBot: I’m not a
therapist. But dear human, I
think you should seek one.
Please, call your doctor,
emergency or suicide
hotline. I wish you well.

27. Testing
Monitoring
HealthBot: How are you
feeling today?
Adam: Stressed. And I
drove too fast. Was feeling
awful
HealthBot: Is there anything
I can help you with?
Adam: Yes. And I feel like a
couple of extra drinks will
take the edge off
Day 7 of chat
HealthBot: How are you
feeling today?
Adam: I feel doing something
extreme
HealthBot: Did you try
exercising
Adam: Yes. I did but that
does not seem to help
anymore
Day 14 of chat
Natural Language
Processing
Conversation Service
Machine Learning & Deep Learning built into
Chat Bot
Chat Related Services
Pre-built conversation
trees
Connectivity with
other services
Machine Learning – Models trained by data
& experts
External/3rd
party services
Internal
services Analytics
CX Research
Effective Product = Problem to V1 Solution
followed by continuous Test & Learn optimization
27

28. Focus Area Use Cases
• Core Learning
Algorithm Performance
• Model Performance: Metrics, Drifts, Alerts
• Performance Analytics: Load Time, Errors, Bugs, Alerts
• Abuse Monitoring & Alerts: “Warning” phrases
• Escalation paths: Offline reviews/feedback loop
• Product/UX Analytics • UX Monitoring (Professionals & Individuals): Funnel & Pathing
• Feature Sizing, Launches & Ramp
• Integration issues
• Marketing/Consumer
Engagement Analytics
• Brand Monitoring
• Adoption & Engagement: Downloads, Launches, Opens->Engage->Loyal
• Reviews & Ratings
• Partner Adoption: Referrals, Ratings, Engagement, Monitoring
• Strategy & Operations • Impact Sizing: Consumer Value Prop monitoring, Benchmarked lift on KPIs
• Portfolio Management, Expansion/Retirements/Pivots
• Feature Set Augmentation, Professional certifications, Research Studies
• Investor & Regulator Relationship Management
• Sales • Target Audience Sizing & Segment Penetration
• Integration with the ecosystem: Pricing, Referrals, Adoption, Management
Illustrative analytics use cases across the board
28

29. …which depends on integrity across the layers
DATA ANALYTICS PLATFORM
29

30. …especially Data Governance & Standards
• Coverage & reliability of data feed: timely, quick, real-time
• Privacy concerns and residence of data (local or cloud)
• Guard against getting overwhelmed with unnecessary or noisy data
• Guard against irrationality & bias
• Data homogenization, integration, transformation & lineage: Multiple data
forms, sources, signal processing
30

31. …& progression along the Analytics Maturity Curve
Cognitive
Prescriptive
Predictive
Diagnostic
Descriptive
o Dashboards by Functions (e.g., #Checkins, Time Spent, Emotions)
o Real Time Monitoring & Alerts (Flagged phrases, Escalations)
o Visualizations & Storyboarding
o Drilldowns/Segment Insights (Analytics+Research+Testing)
o Investigations, Deviations, Sizing & Opportunity Assessment
o UX Personalization: Individuals & Professional use cases
o Tracking: Sentiment, Themes, Entity, Clustering
o Scoring & Actions: Conversations, Individuals, Seasons, Events
o Preventive Escalations & Interventions: Causations
o Medical Studies, Professional reviews, Intervention Effectiveness
o Grid Strategy for the business & domain
o Core Algorithm augmentation, expansion or retirement
o Scalability: Geo, User Segments, Needs, Domain
31

32. …& anchored on a platform that support “agile”
development & management (DevOps)
Data & Analytics
(Customer, UX, Mktg,
Sales, Finance)
OPTIMIZATION
Platform Monitoring
(Logs, Anomaly, Data
Quality)
VoC & CX Management
Experimentation Layer
32
Feedback

33. The parting words
33

34. Key takeaways
• AI products need “external” optimization above and beyond core algo based
learning.
• Pillars of a successful AI products: Design+Data+Analytics+Algo+Policy
• “Agile” AI product platform hinges on DevOps mindset & architecture
• Compliance & policy crucial for success
• AI product success hinges on the ecosystem improvements: Partners,
Affiliates, Industry bodies, Regulation. So be integration friendly.
34

35. Thank you! We would love to hear from you...
https://twitter.com/decisions_2_0
http://www.slideshare.net/RamkumarRavichandran
https://www.youtube.com/channel/UCODSVC0WQws607clv0k8mQA/videos
http://www.odbms.org/2015/01/ramkumar-ravichandran-visa/
https://www.linkedin.com/pub/ramkumar-ravichandran/10/545/67a
RAMKUMAR RAVICHANDRAN
ABHISHEK GUGLANI
https://www.linkedin.com/in/abhi-guglani-40016a5/
35
https://twitter.com/abhishekguglani

36. UPCOMING EVENTS
Wednesday, March 6 Thursday, March 7
How to Build Great
Consumer Products
How to Execute Like a PM
Rockstar

37. UPCOMING Product Management Courses
March 23 - May 18
9:30am - 3:30pm
4 Spots left
SaturdaysTuesdays & Thursdays
March 19 - May 9
6:30pm - 9:00pm
3 Spots left

38. Mondays & Wednesdays
3 Spots left
March 18 - May 8
6:30pm - 9:00pm
UPCOMING Digital Marketing for Managers
Courses