Talk given at Strata Hadoop World conference March 2016. http://conferences.oreilly.com/strata/hadoop-big-data-ca/public/schedule/detail/48305 In this talk we review the culture, process and tools needed for a data driven organization. We review an example of how companies like LinkedIn use data to make business decisions, and then walk through the culture, process, and tools needed to foster this. We review the spectrum of data science used within an organization and explore organizational needs, such as the democratization of data via self-serve data platforms for experimentation, monitoring, and data exploration, as well as the challenges that come with such systems. Participants leave this session with the ability to identify opportunities for data scientists to contribute within their organization and with an understanding of what investments are needed to drive transformation into a data-driven organization.

1. Data 101:
How to use your data science team:
Becoming a data-driven organization
March 29, 2016
Yael Garten
Director of Data Science, LinkedIn
http://www.linkedin.com/in/yaelgarten, @yaelgarten

2. Data is the present: every industry, organization,
product, decision
data driven, or at least data informed.

3. Good vs Great? Data is Expected, Discussed, & Accessible
Expected
For decision making
e.g. Determining strategy, goal setting, impact estimates of initiatives
Discussed
Talked about
e.g. product strategy reviews, design discussions, in the hallway
Accessible
When questions come up, easy to get hands on data?
e.g. democratized data, self-serve tools
Culture
Process
Tools

4. How do we get started?
(whether from scratch or to transform existing organization)
Invest in the data
Hire the right “data scientists”
Culture Process Tools

5. Data Scientist – which kind?
human machine
Find & communicate data insights
Producing for:
Deliverables:
Skills:
Machine learning models

6. This talk will be a success if we:
1. Review the steps of data driven product innovation
2. Understand what is needed to best enable fostering of (or transformation into)
a data-driven organization: culture, process, and tools.

7. 1. Culture: Create a culture which expects decisions are informed by
data & which treats data as a first class citizen
2. Process: Consciously map how you use data
(in each phase of the product lifecycle, in making exec decisions)
3. Tools: invest in your data ecosystem
(data quality, pipeline, and access tools)
To create a data driven organization

8. Data driven product innovation
Product Lifecycle
Ideation Design and Specs Development Test & Iterate Release
Data driven product innovation framework:
Use data to measure, understand, and improve the product:
Build Measure Track Ship Tweak
If data scientist is not involved until
this stage, it may be too late.

9. Well-connected.
Get relevance right.
Few connections.
Give them inventory.
1. Opportunity sizing: how big or important
is the problem?
2. Use data to predict successful product
initiatives:
• Show news articles
• Suggest new connections
• Suggest following active content creators
Build Measure Track Ship Tweak

10. Hypothesis: Following active sources leads to improved user
experience with LinkedIn Feed
Success Metric – progression of definition:
• Total clicks on Follow
• # clicks / #impressions of Follow suggestions
• % Feed Inventory created by new followees
• Downstream sustained engagement with items created by these
followees
• What is engaging? # of clicks? Time spent? # Shares?
Combo?
Actionable insights lead
to product feature
Success Metric
Definition
Tracking
Instrumentation
spec
Experimentation
Setup and Analysis
Post-Launch
Analysis
Build Measure Track Ship Tweak
is creating content
we think you’ll find
interesting
Invest in developing the right success metric.

11. ~Metric = Downstream engagement with items created by these followees
Must enable attributing future clicks on feed items to that campaign as a
source for the Follow.
FeedActivityClick
{
memberID = 77777
actor = 55555
}
FollowSources
{
followCampaign666
memberID = 77777
followeeID = 55555
}
Need accurate reliable standardized data logging to enable metric computation.
Actionable insights lead
to product feature
Success Metric
Definition
Tracking
Instrumentation
spec
Experimentation
Setup and Analysis
Post-Launch
Analysis
Build Measure Track Ship Tweak

12. Actionable insights lead
to product feature
Success Metric
Definition
Tracking
Instrumentation
spec
Experimentation
Setup and Analysis
Post-Launch
Analysis
Design: How long to run experiment, on whom?
Implement: properly set up & randomize to ensure no bias
Analyze: Go or no-go? monitor success metric, ideally automated
on company-wide platform for holistic view of impacts
Build Measure Track Ship Tweak
is creating content
we think you’ll find
interesting
Rigorously set up, then identify whether the feature increased the success metric.

13. Reporting, monitoring, ad hoc analysis
Long term measures of engagement/success
Analysis to inform revision of design
Actionable insights lead
to product feature
Success Metric
Definition
Tracking
Instrumentation
spec
Experimentation
Setup and Analysis
Post-Launch
Analysis
Build Measure Track Ship Tweak
is creating content
we think you’ll find
interesting
Iterate. How can we revise? How can we tweak to optimize?

14. Invest in foundations. Culture Process Tools

15. 1. Culture: Create a culture which expects decisions are informed by
data & which treats data as a first class citizen
2. Process: Consciously map how you use data
(in each phase of the product lifecycle, in making exec decisions)
3. Tools: invest in your data ecosystem
(data quality, pipeline, and access tools)
To create a data driven organization

16. Data scientist as a partner, not a service – give context and ownership
Strong bias to actionable impactful insights, speed of iteration & feedback
• Data Foundations: governed datasets, consistent shared datasets and metrics
• Data Democratization: self serve data exploration platform
• Enable innovation: environment supports speedy ad-hoc analysis
Culture
Process
Tools
Actionable insights lead
to product feature
Success Metric
Definition
Tracking
Instrumentation
spec
Experimentation
Setup and Analysis
Post-Launch
Analysis
Build Measure Track Ship Tweak

17. Actionable insights lead
to product feature
Success Metric
Definition
Tracking
Instrumentation
spec
Experimentation
Setup and Analysis
Post-Launch
Analysis
Build Measure Track Ship Tweak
Democratize data – self served data exploration platform
Enable people in your organization (execs, product managers, designers) to have data at
their fingertips – to ask and answer questions

18. s
• Invest in creating the right metrics
• user centric mindset; optimize for user value not team success
all stakeholders agree upon success metrics prior to launching the
feature test
• Platform of shared tiered metrics visible to entire company
• a metrics pipeline that enables easy implementation of metrics
(and not manual one-offs)
Actionable insights lead
to product feature
Success Metric
Definition
Tracking
Instrumentation
spec
Experimentation
Setup and Analysis
Post-Launch
Analysis
Build Measure Track Ship Tweak
Culture
Process
Tools

19. sActionable insights lead
to product feature
Success Metric
Definition
Tracking
Instrumentation
spec
Experimentation
Setup and Analysis
Post-Launch
Analysis
Build Measure Track Ship Tweak

20. • Data as a first class citizen
• Data tracking bugs as a launch blocker
• Proactive joint definition of data requirements and contract (schemas)
by producers and consumers
• Data Model Review Committee
• Data tracking spec tool for source of truth
Automated testing of data tracking
• Data Quality monitoring tool to ensure data contract
is met
Actionable insights lead
to product feature
Success Metric
Definition
Tracking
Instrumentation
spec
Experimentation
Setup and Analysis
Post-Launch
Analysis
Build Measure Track Ship Tweak
Culture
Process
Tools

21. s
• Belief in proactive controlled experiment as decision making tool
• Efficient and principled lifecycle, from inception to decision
• Before: Review experiment design & implementation
• After: stakeholders discuss impacts and implications of tradeoffs
• Company-wide experimentation platform, with tiered key metrics
• Automated metric reporting & analysis capability
Actionable insights lead
to product feature
Success Metric
Definition
Tracking
Instrumentation
spec
Experimentation
Setup and Analysis
Post-Launch
Analysis
Build Measure Track Ship Tweak
Culture
Process
Tools

22. sActionable insights lead
to product feature
Success Metric
Definition
Tracking
Instrumentation
spec
Experimentation
Setup and Analysis
Post-Launch
Analysis
Build Measure Track Ship Tweak

23. s
• Iteration and innovation
• Metrics meeting: weekly to understand
performance and product value
• Combine qualitative user feedback with
quantitative results
• Long term hold-out groups for deep
understanding of impacts
Actionable insights lead
to product feature
Success Metric
Definition
Tracking
Instrumentation
spec
Experimentation
Setup and Analysis
Post-Launch
Analysis
Build Measure Track Ship Tweak
Culture
Process
Tools

24. This talk will be a success if we:
1. Review the steps of data driven product innovation
2. Understand what is needed to best enable fostering of (or transformation
into) a data-driven organization: culture, process, and tools.

25. 1. Culture: Create a culture which expects decisions are informed by
data & which treats data as a first class citizen
2. Process: Consciously map how you use data
(in each phase of the product lifecycle, in making exec decisions)
3. Tools: invest in your data ecosystem
(data quality, pipeline, and access tools)
To create a data driven organization

26. Thank you.
http://www.linkedin.com/in/yaelgarten
@yaelgarten