At Netflix, we've spent a lot of time thinking about how we can make our analytics group move quickly. Netflix's Data Engineering & Analytics organization embraces the company's culture of "Freedom & Responsibility". How does a company with a $40 billion market cap and $6 billion in annual revenue keep their data teams moving with the agility of a tiny company? How do hundreds of data engineers and scientists make the best decisions for their projects independently, without the analytics environment devolving into chaos? We'll talk about how Netflix equips its business intelligence and data engineers with: the freedom to leverage cloud-based data tools - Spark, Presto, Redshift, Tableau and others - in ways that solve our most difficult data problems the freedom to find and introduce right software for the job - even if it isn't used anywhere else in-house the freedom to create and drop new tables in production without approval the freedom to choose when a question is a one-off, and when a question is asked often enough to require a self-service tool the freedom to retire analytics and data processes whose value doesn't justify their support costs Speaker Bios Monisha Kanoth is a Senior Data Architect at Netflix, and was one of the founding members of the current streaming Content Analytics team. She previously worked as a big data lead at Convertro (acquired by AOL) and as a data warehouse lead at MySpace. Jason Flittner is a Senior Business Intelligence Engineer at Netflix, focusing on data transformation, analysis, and visualization as part of the Content Data Engineering & Analytics team. He previously led the EC2 Business Intelligence team at Amazon Web Services and was a business intelligence engineer with Cisco. Chris Stephens is a Senior Data Engineer at Netflix. He previously served as the CTO at Deep 6 Analytics, a machine learning & content analytics company in Los Angeles, and on the data warehouse teams at the FOX Audience Network and Anheuser-Busch.

1. Rapid Data Analytics @ Netflix
Jason Flittner
Senior BI Engineer
Chris Stephens
Senior Data Engineer
Monisha Kanoth
Senior Data Architect

2. What We Do

3. 633643 DEA @ Netflix
Content Analytics

4. Global
Expansion &
Content Spend

5. Freedom & Responsibility
Highly Aligned, Loosely
Coupled
Context, not Control
Culture + Technology
Courage
Judgement
Honesty
Communication
Curiosity
Passion
Innovation
Impact
Selflessness

6. Parquet FF
Storage Compute Tools BI
AWS
S3
(Hadoop
clusters)

7. Deploy Fast, Fix Faster
● Improve & Iterate vs Perfect
● Have a Rollback Plan Ready

8. Develop Business
Logic not ETL
● Think in Patterns

9. The Path of Least Resistance is the
Right Path
● Make Smart Engineering
Tradeoffs

10. The Clock starts Ticking when you
Deploy
● Every Data Pipeline comes with
an Expiration Date
● Deprecate and Prune

11. No Man’s Land
is Expensive
● Ownership

12. Be a Noob
● User Groups

14. What You Could Do
in your Data Warehouse

15. Let everyone drop tables in production

16. Cost / Benefit
Conscientious people make mistakes,
but not very often
Data warehouse is not an operational system
What happens if a table is accidentally dropped?
● Do you have backups?
● How quickly can you restore a table?
Is the benefit of worth the tax on every data /
analytical product your team produces?

17. We have some protection

18. In Hive, all tables are external tables pointing to S3 locations.
ETL writes a new “batch” of data then updates the metastore.
s3://[bucket]/hive/schema.db/table/batchid=1459364911
ALTER TABLE table SET LOCATION [path to new batch ID];
DROP TABLE does not delete any data.

19. In our MPP databases, we have a procedure for upgrading and
downgrading our privileges.
CALL admin.UpgradePrivileges('me')
Lasts for several hours. Usage is logged.
Accidents? Restore from backups. Or reload from Hive.

20. When other teams are ready to move to production ...
We’re done. And moving on to the next thing.
You can trust your people to work the same way.

21. Don’t have an “on call”
(Use a “first responder” instead)

22. Everyone on the team takes a shift: both BI and data engineers
(even managers every once in a while!)
First Responder = the first one to respond
● handles most common failures (restarting jobs)
● reaches out directly to ETL owner if escalation is required
● handles communication surrounding ETL delays

23. Goal is to protect the team’s time and focus

24. How we do this
● visually define what needs attention and what doesn’t
○ “above the line” vs “below the line”
● email alerts for “above the line” jobs that take longer than normal
● playbook for fixing common stuff
○ the more complete your entries are, the less you get called!

25. Have a very clear sense of what is urgent, and what isn’t

26. Treating every failure like it’s urgent bleeds your team of the time they
need to do work
Build your processes so they can be ignored for 3 days
● don’t load data if it’s incomplete
● reprocess fact data for several days instead of picking up the latest
Gives you the freedom to judge whether a failure is worth an
interruption

27. Everybody owns ETL
(when they need to)

28. BI engineer needs data structured a certain way for a report
Many environments:
● Ask a data engineer to build them a table
Our environment:
● Let them schedule a Hive script and adjust as necessary

29. We focus on centers of excellence, not role boundaries

30. More Examples:
● our BI engineers use Python to automate tasks
● our data engineers have Tableau licenses, and use them for
quick visualizations and report deployments
For small tasks, this helps us avoid the overhead of interruption and
knowledge transfer

31. What You Could Do
on the Front-end

32. Parquet FF
(Hadoop
clusters)
Storage Compute Data Interface Data Access, Analytics and Visualization
AWS
S3

33. Do Not Limit Yourself to Conventional Tools
○ Tableau - Data Visualization and Dashboards
○ MicroStrategy - Dynamic SQL and Metadata
○ Python or Custom Reporting - Emails

34. Give your BI Engineers
Superpowers (like this guy)
○ Provide a data platform
○ BI + Data Engineering
○ Context not Requirements
○ Be early adopters

35. Simple is
Often Best

36. Dismantle your Data Warehouse Team
○ Integrate with the business
○ Data Engineering and Data Science teams
○ Open and honest communication

37. Fast is better than perfect
○ Build, iterate… repeat
○ How to handle adhocs
○ Freedom - make the right call
○ Responsibility - Ownership

38. Encourage
Hacking

39. Questions?
Want to chill with us!?
jobs.netflix.com