Petabytes Scale Analytics Infrastructure @Netflix
•
4 j'aime•1,508 vues
Video and slides synchronized, mp3 and slide download available at URL http://bit.ly/2l2Wrs8. Tom Gianos and Dan Weeks discuss Netflix' overall big data platform architecture, focusing on Storage and Orchestration, and how they use Parquet on AWS S3 as their data warehouse storage layer. Filmed at qconsf.com. Daniel Weeks manages the Big Data Compute team at Netflix and is responsible for integrating and enhancing open source big data processing technologies including Spark, Presto, Hadoop. Tom Gianos is a Senior Software Engineer, Big Data Platform at Netflix. He leads development of Genie and has a passion for merging web and big data technologies to solve interesting distributed systems problems.
Recommandé
Recommandé
Contenu connexe
En vedette
En vedette (14)
Plus de C4Media
Plus de C4Media (20)
Dernier
Dernier (20)
Petabytes Scale Analytics Infrastructure @Netflix
- 1. Daniel C. Weeks Tom Gianos Netflix: Petabyte Scale Analytics Infrastructure in the Cloud
- 2. InfoQ.com: News & Community Site • Over 1,000,000 software developers, architects and CTOs read the site world- wide every month • 250,000 senior developers subscribe to our weekly newsletter • Published in 4 languages (English, Chinese, Japanese and Brazilian Portuguese) • Post content from our QCon conferences • 2 dedicated podcast channels: The InfoQ Podcast, with a focus on Architecture and The Engineering Culture Podcast, with a focus on building • 96 deep dives on innovative topics packed as downloadable emags and minibooks • Over 40 new content items per week Watch the video with slide synchronization on InfoQ.com! https://www.infoq.com/presentations/ netflix-big-data-infrastructure
- 3. Purpose of QCon - to empower software development by facilitating the spread of knowledge and innovation Strategy - practitioner-driven conference designed for YOU: influencers of change and innovation in your teams - speakers and topics driving the evolution and innovation - connecting and catalyzing the influencers and innovators Highlights - attended by more than 12,000 delegates since 2007 - held in 9 cities worldwide Presented at QCon San Francisco www.qconsf.com
- 4. ● Data at Netflix ● Netflix Scale ● Platform Architecture ● Data Warehouse ● Genie ● Q&A Overview
- 10. Our Biggest Challenge is Scale
- 11. Netflix Key Business Metrics 86+ million members Global 1000+ devices supported 125+ million hours / day
- 12. Netflix Key Platform Metrics 60 PB DW Read 3PB Write 500TB500B Events
- 13. Big Data Platform Architecture
- 14. Data Pipelines Cloud apps Kafka Ursula Cassandra Aegisthus Dimension Data Event Data 5 min Daily S3 SS Tables
- 15. Storage Compute Service Tools Big Data APIBig Data Portal S3 Parquet Transport Visualization Quality Workflow Vis Job/Cluster Vis Interface Orchestration Metadata
- 16. Production Ad-hoc Other ~2300 d2.4xl ~1200 d2.4xl
- 18. Why S3? • Lots of 9’s • Features not available in HDFS • Decouple Compute and Storage
- 19. Decoupled Scaling Warehouse Size HDFS Capacity All Clusters 3x Replication No Buffer
- 20. Production Ad-hoc S3 Decouple Compute / Storage
- 21. Tradeoffs - Performance • Split Calculation (Latency) – Impacts job start time – Executes off cluster • Table Scan (Latency + Throughput) – Parquet seeks add latency – Read overhead and available throughput • Performance Converges with Volume and Complexity
- 23. Metadata • Metacat: Federated Metadata Service • Hive Thrift Interface • Logical Abstraction
- 24. Partitioning - Less is More ab_test telemetry etl data_science country_d catalog_d playback_f search_f date=20161101 date=20161102 date=20161103 date=20161104 Database Table Partition
- 26. Parquet
- 27. Parquet File Format Column Oriented ● Store column data contiguously ● Improve compression ● Column projection Strong Community Support ● Spark, Presto, Hive, Pig, Drill, Impala, etc. ● Works well with S3
- 28. Footer RowGroup Metadata row count, size, etc. schema, version, etc. Column Chunk Metadata [encoding, size, min, max] Column Chunk Metadata [encoding, size, min, max] Column Chunk Metadata [encoding, size, min, max] Dict Page Data Page Data Page Column Chunk Data Page Data Page Data Page Column Chunk Dict Page Data Page Data Page Column Chunk Dict Page Data Page Data Page Column Chunk Data Page Data Page Data Page Column Chunk Dict Page Data Page Data Page Column Chunk RowGroupRowGroup
- 29. Staging Data • Partition by low cardinality fields • Sort by high cardinality predicate fields
- 34. A Nascent Data Platform Gateway
- 35. Need Somewhere to Test Test Gateway Test Prod Gateway Prod
- 36. More Users = More Resources Prod Prod Gateway Prod Gateway Prod Gateways Test Prod Gateway Prod Gateway Test Gateways
- 37. Clusters for Specific Purposes Prod Prod Gateway Prod Gateway Prod Gateways Test Prod Gateway Prod Gateway Test Gateways Backfill Prod Gateway Prod Gateway Backfill Gateways
- 39. No one is happy
- 40. Genie to the Rescue Prod Test Backfill
- 41. Problems Netflix Data Platform Faces • For Administrators – Coordination of many moving parts • ~15 clusters • ~45 different client executables and versions for those clusters – Heavy load • ~45-50k jobs per day – Hundreds of users with different problems • For Users – Don’t want to know details – All clusters and client applications need to be available for use – Need to provide tools to make doing their jobs easy
- 42. Genie for the Platform Administrator
- 43. An administrator wants a tool to… • Simplify configuration management and deployment • Minimize impact of changes to users • Track and respond to problems with system quickly • Scale client resources as load increases
- 44. Genie Configuration Data Model • Metadata about cluster – [sched:sla, type:yarn, ver:2.7.1] • Executable(s) – [type:spark-submit, ver:1.6.0] • Dependencies for an executable Cluster Command ApplicaLon 1 0..* 1 0..*
- 45. Search Resources
- 47. Updating a Cluster • Start up a new cluster • Register Cluster with Genie • Run tests • Move tags from old to new cluster in Genie – New cluster begins taking load immediately • Let old jobs finish on old cluster • Shut down old cluster • No down time!
- 48. Load Balance Between Clusters • Different loads at different times of day • Copy tags from one cluster to another to split load • Remove tags when done • Transparent to all clients!
- 49. Update Application Binaries • Copy new binaries to central download location • Genie cache will invalidate old binaries on next invocation and download new ones • Instant change across entire Genie cluster
- 50. Genie for Users
- 51. User wants a tool to… • Discover a cluster to run job on • Run the job client • Handle all dependencies and configuration • Monitor the job • View history of jobs • Get job results
- 52. Submitting a Job 1. https://analyticsforinsights.files.wordpress.com/2015/04/superman-data-scientist-graphic.jpg { … “clusterCriteria”:[ “type:yarn”, “sched:sla” ], “commandCriteria”:[ “type:spark”, “ver:1.6.0” ] … } Commands Clusters
- 53. Genie Job Data Model Cluster Command ApplicaLon Job Job Request Job ExecuLon Job Metadata 0..* 1 1 1 1 1 1 1 1 1 1 1
- 54. Job Request
- 56. Job History
- 57. Job Output
- 58. Wrapping Up
- 59. Data Warehouse • S3 for Scale • Decouple Compute & Storage • Parquet for Speed
- 60. Genie at Netflix • Runs the OSS code • Runs ~45k jobs per day in production • Runs on ~25 i2.4xl instances at any given time • Keeps ~3 months of jobs (~3.1 million) in history
- 61. Resources • http://netflix.github.io/genie/ – Work in progress for 3.0.0 • https://github.com/Netflix/genie – Demo instructions in README • https://hub.docker.com/r/netflixoss/genie- app/ – Docker Container
- 62. Questions?
- 63. Watch the video with slide synchronization on InfoQ.com! https://www.infoq.com/presentations/ netflix-big-data-infrastructure