What does a successful partnership between product and analytics teams look like? What can analysts do to ensure a successful partnership with other teams? Some strategies and tips from my work at ZipRecruiter.
2. 2
Today’s topics
● About me and ZipRecruiter
● What does the analytics team do? What are our projects like?
● Strategies for successful partnerships
● Some things I wish I knew when I started
4. 4
About Me
● Columbia University - Computer Science and Machine Learning
● Worked at startups in New York, work on the credit risk analytics system
at JP Morgan
● Currently a data scientist on the analytics team at ZipRecruiter
5. 5
ZipRecruiter
● Short version: We help people find jobs
● Employers post jobs, we help them find qualified candidates
● #1 job search app on iPhone!
● Located in Santa Monica
● We’re hiring! For analytics and product positions - Resumes to
louisc@ziprecruiter.com
6. 6
Some challenges we face at ZipRecruiter
● How can we match employers with jobseekers in a way that benefits
both?
● Who should we market our service to?
● How can we improve our user experience?
● How can we guard against fraud on our platform?
7. 7
What we do and how we do it
Analytics at ZipRecruiter
8. 8
The Role of analytics at ZipRecruiter
● Help business stakeholders make data driven decisions
● Other departments have domain knowledge and problems to solve, we
supply statistical skills
● Define metrics
● Answer vague business questions with well-defined data analysis
10. 10
ZipRecruiter’s Analytics team
● Follows the “centralized” model
● Independent department which provides data and statistical analysis
● Advisory capacity - help other teams understand their data and figure out
which decisions will benefit them the most
11. 11
ZipRecruiter’s Analytics team
● Follows the “centralized” model
● Pros:
○ Makes it easy to build up institutional knowledge
○ Can build and share analytics tools
○ Independent incentive structure
● Cons:
○ Requires skilled analytics-specific leadership
○ Further from domain experts
● My opinion: Given the rapidly changing state of industrial analytics, pros
outweigh cons
13. 13
Analytics can help make great products
● Product managers have vision, and data analysis can help advise on the
best way to execute it
● Two main ways:
○ Product Optimization:
■ Example: A/B testing different user experiences
○ Machine intelligence Integration:
■ Example: Making recommendations to users, dynamic pricing,
fraud detection
14. 14
Successful partnerships
● A balancing act:
○ Product people are
incentivized to do cool things
as fast as they reasonably
can
○ Analytics people incentivized
to be rigorous and careful
● Pragmatism vs rigor
● Fast vs slow
15. 15
Analytics projects
● It helps to understand how an analytics project is structured
● For a data-driven project to succeed, we need to:
○ Collect the data
■ Run experiments, look at historical data, etc
○ Analyze the data
■ Build a model, extract insights, etc
16. 16
Analytics projects
What insight will
be extracted?
What information
will the model give
us?
What mechanism
collects the data?
Where will it be
stored and
accessed?
What is the
question we want
to answer?
What data will be
used?
What modelling
approach will be
used?
What software will
be used?
Data Collection
Data Analysis
Design Implementation
17. 17
Analytics projects
● Some anti-patterns:
○ “Crunch these numbers for me”
■ Analysts need context
○ “Do some data stuff with all our historical data”
■ Analysts benefit from a clear product vision
○ “We ran this experiment, what does it tell you”
■ If an analyst did not help design the experiment, the data may
not be useful
19. 19
A typical request
● ZipRecruiter has a free trial for our employer subscription product
● During the free trial, employers post jobs and get applications
● If we knew how the number of applications related to the conversion rate,
we could design our experience to improve this
● How can we understand the relationship between number of
applications per job and likelihood of conversion?
20. 20
A first look
● We have historical examples of how many applications/job were received,
as well as whether or not a user converted
● We could compare the response counts of the convert/non-convert
groups to describe the data
● But this doesn’t allow us to make predictions - what we want is a model!
21. 21
Understanding our data
● We have a few years of historical data about:
○ Hundreds of thousands of free trials
○ How many jobs each one posted, which jobs received applications
○ Whether or not they converted
○ Geographic and industry information
● Our model will need to relate the features (geo/industry info, number of
applications) to the output (conversion event)
22. 22
Error bars
● Just specifying the expected value of the conversion rate at each response
count can give a false impression of precision
● We want to quantify our uncertainty
● Solution: Add confidence intervals to estimates (via bootstrap)
23. 23
Analytics projects
Can we produce a
curve defining the
FT conversion vs
response
relationship?
The data used is
collected in our
SQL (Redshift)
database
What is the
relationship
between FT
conversion and
responses?
Use historical data
Model: Logistic
regression
Software: SQL
and Python
Data Collection
Data Analysis
Design Implementation
24. 24
The result
● The curve which describes the
relationship between application
count and conversion
● Can use this to optimize the free
trial give our customers the best
experience (for example, tuning
the length of time the trial lasts)
26. 26
Uncertainty is the only certainty
● Analytics helps us understand where there is uncertainty, but it usually
can’t be brought to zero
● Nate Silver: “[Some forecasters] see uncertainty as the enemy...this tends
to leave us less prepared when a deluge hits.”
● Analysts deal with uncertainty by mitigating it where possible, and
communicating it where not possible
● Use confidence intervals and similar techniques, which provide best/worst
cases
27. 27
Don’t be afraid to experiment
● Experiments have become common in the industry (A/B tests)
● But there is a cost associated with running them - when should we run
them?
● Answer - whenever possible!
● Product can often get great insight by running an experiment
● Analytics can often provide much more definitive results and mitigate
uncertainty as much as possible
28. 28
Understand Everyone’s Incentives
● Analysts should report revenue/profit/cost impact, dollars and cents
● Product folks can help by making it clear what needle you want to move,
even if it’s a big picture metric
● Analysts are responsible for translating their findings into a language that
a business user can use for decision making
29. 29
Understand the choices, make a recommendation
● Try and help people make data driven decisions by understanding the
choices they want to evaluate
● Product - present the strategies you are considering
● Analytics - focus on making specific recommendations, rather than simply
conveying the results of number crunching
30. 30
Focus on the small picture
● It’s tempting to totally overhaul a system and replace it with DEEP
LEARNING THE BIG DATA
● But big overhauls are risky
● At each step in modelling, you may make bad assumptions - rolling out
incremental improvements allows you to check these assumptions
● When trying to improve a process, don’t overhaul it from the very
beginning - start with small improvements to the existing method
31. 31
Summary
● ZipRecruiter’s centralized analytics team model has a lot of
advantages
● Analytics + Product =
● Uncertainty is part of the process, but we can do a lot to mitigate it
and communicate it clearly
32. 32
About my work? About ZipRecruiter? About working in data science?
Q & A
