Everybody seems to be talking about big data, data science and predictive analytics these days, but what does it really mean? How does it apply to your business? Can it help you identify the early indicators of churn and growth in your customers? And if you had the data, what can you do with it? In this webinar, you will learn all about data science from David Gerster, former head of data science at Groupon. Mike Stocker, from the Marketo Customer Success team, will then dive into how Marketo is using data science to drive success for its customers. Finally, the Gainsight data science team will share best practices on turning analytics into Customer Success actions.

1. Monthly
Webinar
August 20,
2013
#customersuccess

2. Some Housekeeping
1
We will have a Q&A
session at the
end, so keep your
questions and
comments coming
Video recording
and slides will be
made available
soon after the
webinar
Please fill the survey
at the end – we
promise it’s short!
It’ll help us bring
you more
#CustomerSuccess
events and content
2 3 4
Use the
GoToWebinar chat
box and/or tweet
using
#CustomerSuccess

3. Dan Steinman
Chief Customer
Officer
David Gerster
VP Data Science
Mike Stocker
Team Lead –
Customer Success
Our Panelists

4. David Gerster
VP Data Science
Introduction
• I’m David Gerster, and I led the
Mobile Data Science team at
Groupon
• Spent 3 years applying Data Science
to practical business problems
• Example: predicting customer churn
on Groupon’s iPhone app
– Concept of customer churn also
applies to B2B companies
• Recently joined BigML as Vice
President of Data Science

5. David Gerster
VP Data Science
What Is Data Science?
• Your business has gathered
large amounts of data
• How do we find meaningful
patterns in all this data?
– Which of my customers are most
likely to churn?
• How do we take action on
these patterns?
– If I know a customer is likely to
churn, what do I do about it?

6. David Gerster
VP Data Science
Simple Churn Example
• You’re a daily deal site with an
iPhone app
• You log three types of user events:
deal views, deal clicks and deal
purchases
• Based only on a user’s activity in
the first month after downloading
the app, how do we predict if that
user will still be using the app in six
months?

7. David Gerster
VP Data Science
Simple Churn Example
There’s a
pattern in this
data: “Users
who do 1 or
more clicks in
Month 1 come
back and use
the app in
Month 6.”
Even this simple
pattern is
challenging for
a person to find.
Why not have
machines do it
instead?

8. David Gerster
VP Data Science
Using BigML to Predict Churn
• BigML is a cloud-
based tool that finds
meaningful patterns
in data
• Instead of three
users, let’s try 243,000!
• Instead of three
columns, let’s try 10!
• BigML finds useful
patterns that predict
which users will churn

9. David Gerster
VP Data Science
Using BigML to Predict Churn
• This segment of users is 80% likely to churn
in 6 months:
– Doesn’t use the iPhone app in last 2 weeks of
the first month, and
– Doesn’t log in during the first month.
– 32% of users fit this description!
• This segment of users is 75% likely to come
back in 6 months:
– Logged 5 or more events in the last 2 weeks of
the first month, and
– Had an average gap between visits of 2 days
or less in the first month.
– 11% of users fit this description!
• [Quick glimpse of SunBurst visualization]

10. David Gerster
VP Data Science
Acting on Predictions
• These predictions only need data from the
first month of activity, so we can act
quickly when we see a user who’s likely to
churn
• Reach out to that high-risk 32% of users
using marketing promotions, better daily
deals, etc.
• Analyze the 11% of users who do come
back in more detail. What are we doing
right that makes them so active?
• More info: gerster@bigml.com

11. Mike Stocker
Team Lead –
Customer Success
Introduction
Marketing software from Marketo automates lead
scoring, email nurturing, landing pages, events, social
campaigns, and ROI reporting from a single, integrated
platform.
It’s not just about automating tasks; it’s about making
marketers better.
Marketing automation software is fundamentally
different from other kinds of business applications, like
CRM or ERP.
Marketing is much more dynamic – users need to
constantly conceive, build and launch new marketing
campaigns every few days or weeks, with minimum
effort and minimal IT support.

12. Mike Stocker
Team Lead –
Customer Success
Marketo CSM Goals
To become a long-term partner and grow with
our customers
To have a closed-loop feedback system in
place
To bring customer data and workflow front and
center to the whole organization
To have more relevant and strategic
engagements with our customers
To be more proactive and improve customer
experience

13. Mike Stocker
Team Lead –
Customer Success
Success Across the Enterprise
CSM
Team
Installed
Base Sales
Team
Renewals
Team
Marketing
Department

14. Mike Stocker
Team Lead –
Customer Success
Important Features Used Everyday
Alerts: Provide early warning for customers who are at-risk helping
to prioritize Customer Success outreach. Alerts may be triggered
by survey scores, observations of usage, support tickets, etc.
Customer 360: Holistic view of customers’ health—includes
adoption, support tickets, NPS score, and Services
engagements, all in one view.
Adoption: Quick and easy way to identify out-of-compliance and
underutilizing customers based on product adoption metrics.
Survey/NPS: Results are pushed across the organization; alerts are
triggered based on NPS score.
Insights: Executive Dashboards to monitor customers’
health, CSM status, up-sell/cross-sell opportunities.

15. Mike Stocker
Team Lead –
Customer Success
Alerts Applied Before Data Science
Logins
• No logins last X days
• Logins have not increased in last four weeks
• Logins have dropped by 10% in the last two
weeks
Adoption
• # of emails sent dropped by 10% vs. last week
Support Tickets
• No support tickets in the past 30 days
Survey Scores
• Low survey score

16. Mike Stocker
Team Lead –
Customer Success
Alerts Applied Before Data Science

17. Mike Stocker
Team Lead –
Customer Success
Alerts Applied Before Data Science

18. Dan Steinman
Chief Customer
Officer
Data Science @ Gainsight
Intuition
Best
Practices
Data
Science
Alert Rules and Playbooks
New
Insights
Counter
Intuition
Confirm
Intuition

19. Dan Steinman
Chief Customer
Officer
Data Science Approach
Phase Hard Problems
Data science strategy What data to start with?
Feature extraction How do group / aggregate data?
Feature selection Which data matters?
Machine learning Which algorithms to use?
Iteration Where do we go next?
Alert configuration How do we operationalize this?

20. Dan Steinman
Chief Customer
Officer
Case Study
Data Set Selection
• Isolated data to customers with at least 9 months of history
• Split customers into two – ―Test‖ and ―Control‖ groups
• The ―Control‖ group compared Churn vs Active
• We back-test the prediction model against the ―Test‖
Time Selection
• Narrowed focus to ―normal‖ active periods
• Compared 3-month time frames to find secular trends

21. Dan Steinman
Chief Customer
Officer
Case Study
Variables Analyzed
• > 40 usage features
• Support data
• Account attributes
• NPS data
Best Practices
• Remove ―noisy‖ data (e.g., big companies with
large ACV)
• Remove incomplete or inconsistent data
(e.g., manually-coded Account attributes)
• Remove variables that are ―too heavy‖ (ones
that are true for >80% of all customers)

22. Dan Steinman
Chief Customer
Officer
Generate New Alert Rules
Alert Rules and Playbooks
No proactive
outreach
within 6
months of
renewal
53% decline
in number of
sessions in
middle of
subscription
15% decline
in key usage
metric

23. Dan Steinman
Chief Customer
Officer
Counter-Intuitive Findings
Average NPS score of
churned customers = 7.2
(scale is 1 to 10)
Only a small number of
churned customers had
highest level Support tickets

24. Questions?

25. Thank
You