Advanced Spark and TensorFlow Meetup

1. MARKETING AUTOMATION:
BEGGING FOR MACHINE LEARNING
Seth Myers
Senior Data Scientist
Demandbase
smyers@demandbase.com
Advanced Spark and TensorFlow Meetup

2. © 2016 DEMANDBASE｜SLIDE 2
What We Do
 We automate how companies market to, sell to, and advertise to other
companies (B2B)
 Old methods of B2B
 Target as many companies as possible
 Example: Send out generic spam sales emails
 New method
 Customized/Personalized approached to each customer
 Focus on the right companies instead of all companies
 Scaling personalized approach is huge opportunity for machine learning

3. © 2016 DEMANDBASE｜SLIDE 3
Examples
 Business Knowledge Graph
 A monthly crawl of 2 to 5 Billion web pages to build a network of companies, people, and
products
 Uses in-house-built NLP to infer relationships between entities from natural text (e.g.
which companies buy from/compete with/acquire other companies
 Technologies: Elastic MapReduce
 Predicting company buying behavior
 Currently tracking 700 Billion web interactions a year.
 Predicts when a company is shopping around for a new product.
 Technologies: Spark, BigQuery
 Predicting our client’s next customer (Who to Target)
 Takes current customers as positive training examples, trains a classifier
 Scores 14 Million known companies inside of 20 minutes (other solutions take months to
implement)
 Technologies: Spark/mllib
 Content Recommendations for our clients’ websites

4. © 2016 DEMANDBASE｜SLIDE 4
Generating Page Recommendations
 An important potential customer is on your website – how do we make
sure they find the right pages that will sell them?
 Our clients are big companies that sell a lot of products
 Different visitors are looking for very different things
 Using IP addresses and cookies, we know the company where each
website visitor works
 Based on visitor’s company, generate personalized recommendations

5. © 2016 DEMANDBASE｜SLIDE 5

6. © 2016 DEMANDBASE｜SLIDE 6
Learning to Recommend
 Build a supervised classifier that predicts which page a website visitor will
go to next: Prob(current page next page)
 Recommend the most likely next page.
 Training data: our client’s historical website visitor logs
 Feature vectors are extracted from:
 Visitor’s company (revenue, industry, buying behaviors, corporate structure, etc)
 Current and next pages
 Previous visitor behaviors
Visitor Page view sequence
Visitor 1 Page 7, Page 4, Page 8
Visitor 2 Page 2, Page 9

7. © 2016 DEMANDBASE｜SLIDE 7
Planning Sequences of Recommendations
 Our clients first reaction: “meh.”
 They only really care about traffic to certain target pages
 Signing up for webinars
 Downloading whitepapers
 Requesting a demo
 Becomes a reinforcement learning problem:
 Generate a sequence of page recommendations r1, r2, r3,… that maximize

8. © 2016 DEMANDBASE｜SLIDE 8
Closed-Form Solution
 Assume Markov property – next page is independent of previous pages
 Define V(pi) = Prob(eventually landing on target page | current page = pi)
 If pi is a target page, then V(pi) = 1
 The probability of going to a particular page pj, and then to the target page is
 V(pi) is the probability of going to any page times that page’s V()

9. © 2016 DEMANDBASE｜SLIDE 9
 If we recommend page pj, we assume the user will land on target page
with probability
 We recommend the best pj
Closed-Form Solution
 So Bounce rate,
measured directly
Original ML
Classifier

10. © 2016 DEMANDBASE｜SLIDE 10
Implementation
 For each client, train the Pr(pi pj ) classifier in Spark (mllib) from web
logs, and serialize the classifier in Redis.
 For each new visitor to the website – calculate Pr(pi pj ) between all
pages pi, pj on the site
 Solve for V(pi) for all pages.
 On each page view, generate next-page recommendations:

11. © 2016 DEMANDBASE｜SLIDE 11
But Does it work?
 A/B Test on Demandbase.com with recommendations
 Bounce Rate: -52%
 Page views per visitor: +107%
 Time on Site: + 231%
 Target Page # 1 – Sales Demo Requests: +311%
 Target Page # 2 – Downloading Whitepapers: +593%
 Similar results across all clients

12. © 2016 DEMANDBASE｜SLIDE 12
Shameless Plug
 We’re hiring very aggressively
 Pre-IPO
 Just raised $65M, mostly earmarked for expanding our data science
team and infrastructure
 We build customer-facing machine learning solutions – no revenue
modeling for the finance team here
 High impact – all of our recent hires are already building new products.
 Proprietary datasets - a lot of fun to work with
 Voted “Best 10 Places to Work” multiple times by Glassdoor and SF
Business Times
 Interested?
 Seth – smyers@demandbase.com