Data lineage

•

4 j'aime•463 vues

GirishLingappa

Overview of Building and Scaling Data Lineage at Netflix and our vision of covering the diverse landscape

Données & analyses

Building and Scaling
Data Lineage at Netflix
Strata SF 2019
Di Lin, Girish Lingappa, Jitender
Aswani (@jaswani)

CTA: Intentionally Purposeful to
Improve Diversity in Tech

People, Security,
& Infra Data Eng
(DEI)
Data Lineage Asset Inventory Anomaly Detection
Leverage data to
improve cloud infra

#1 Data Scientist
validate
source of the
metrics?

#2 Software Engineer
who
downstream to
you service is
impacted?

#3 Reliability Engineer
identify and
alert on jobs /
table at risk of
missing SLA?

Detection and
Data Cleansing
Use Cases Inspired Lineage
Retention and
Data Efficiency
Cost Attribution Platform Reliability
Data Integrity

intentionality
behind complex
data landscape

Freedom & Responsibility
distributed decision
making =>
● move fast,
● scale complexity,
● Judicious about
what’s best for Netflix

Challenge # 1
Diverse Data Landscape
First time expanding beyond traditional data
warehouse realm

Challenge # 2
Platform Evolution
New Platforms Upgrades

Challenge # 3
Data Conformance
Variance in granularity of the meta data

Design Principle # 1
Ensure data coverage

Design Principle # 2
Enable seamless integration

Design Principle # 3
Envision a flexible data model

Data Coverage - data gets pushed
● Platform tools publish events (preferred approach)
○ Compute engines
○ Data movement tools
○ Reporting tools

Data Coverage - we pull & parse data
● Ingestion scripts parse logs and metadata
○ Enumerate jobs and scripts
○ Parse plan info
○ Parse queries

#1: Platform Reliability
● SLA Recommendation
● Predict SLA miss
● Instructive notifications
● Anomaly detection

#2 : Data visibility
● Reinstate key tables
● Deprecate tables
● Schema changes
● Contract no longer held

#3: Efficiency
● Cost evaluation for entities
○ Storage
○ Compute
○ Usage
● Retention
● Disaster recovery

Please visit us at
jobs.netflix.com
Continue your journey...
Lineage blog
Infra DE team blog

Netflix @ Strata
● Wednesday
○ Self-service data platform
● Thursday
○ App performance measurement
○ Security detection platform
○ Personalization enhancement 4:40 PM @ 2002
11:50 AM @ 2002
11:50 AM @ 2006
2:40 PM @ 2024

“Provide a complete and accurate
data lineage system enabling
decision-makers to win moments
of truth”
Lineage Mission

Contenu connexe

Tendances

Data Mesh for Dinner

Kent Graziano

Organizations have been chasing the dream of data democratization, unlocking and accessing data at scale to serve their customers and business, for over a half a century from early days of data warehousing. They have been trying to reach this dream through multiple generations of architectures, such as data warehouse and data lake, through a cambrian explosion of tools and a large amount of investments to build their next data platform. Despite the intention and the investments the results have been middling. In this keynote, Zhamak shares her observations on the failure modes of a centralized paradigm of a data lake, and its predecessor data warehouse. She introduces Data Mesh, a paradigm shift in big data management that draws from modern distributed architecture: considering domains as the first class concern, applying self-sovereignty to distribute the ownership of data, applying platform thinking to create self-serve data infrastructure, and treating data as a product. This talk introduces the principles underpinning data mesh and Zhamak's recent learnings in creating a path to bring data mesh to life in your organization.

How to Build the Data Mesh Foundation: A Principled Approach | Zhamak Dehghan...

HostedbyConfluent

This is the first in a series of five webinars that look 'under the covers' of Denodo's industry leading Data Virtualization Platform. The webinar will provide an overview of the architecture and key modules of the Denodo Platform - subsequent webinars in the series will take a deeper look at some of the key modules and capabilities of the platform, including performance, scalability, security, and so on. More information and FREE registrations to this webinar: http://goo.gl/fLi2bC To learn more click to this link: http://go.denodo.com/a2a Join the conversation at #Architect2Architect Agenda: The Denodo Platform Platform Architecture Key Modules Connectors Data Services and APIs

Denodo Data Virtualization Platform: Overview (session 1 from Architect to Ar...

Denodo

Watch full webinar here: https://buff.ly/2mHGaLA What started to evolve as the most agile and real-time enterprise data fabric, data virtualization is proving to go beyond its initial promise and is becoming one of the most important enterprise big data fabrics. Attend this session to learn: • What data virtualization really is • How it differs from other enterprise data integration technologies • Why data virtualization is finding enterprise-wide deployment inside some of the largest organizations

Why Data Virtualization? An Introduction

Denodo

So many buzzwords of late: Data Lakehouse, Data Mesh, and Data Fabric. What do all these terms mean and how do they compare to a modern data warehouse? In this session I’ll cover all of them in detail and compare the pros and cons of each. They all may sound great in theory, but I'll dig into the concerns you need to be aware of before taking the plunge. I’ll also include use cases so you can see what approach will work best for your big data needs. And I'll discuss Microsoft version of the data mesh.

Data Lakehouse, Data Mesh, and Data Fabric (r2)

James Serra

Netflix processes trillions of events and petabytes of data a day in the Keystone data pipeline, which is built on top of Apache Flink. As Netflix has scaled up original productions annually enjoyed by more than 150 million global members, data integration across the streaming service and the studio has become a priority. Scalably integrating data across hundreds of different data stores in a way that enables us to holistically optimize cost, performance and operational concerns presented a significant challenge. Learn how we expanded the scope of the Keystone pipeline into the Netflix Data Mesh, our real-time, general-purpose, data transportation platform for moving data between Netflix systems. The Keystone Platform’s unique approach to declarative configuration and schema evolution, as well as our approach to unifying batch and streaming data and processing will be covered in depth.

Virtual Flink Forward 2020: Netflix Data Mesh: Composable Data Processing - J...

Flink Forward

Watch full webinar here: https://bit.ly/2Y0vudM What is Data Virtualization and why do I care? In this webinar we intend to help you understand not only what Data Virtualization is but why it's a critical component of any organization's data fabric and how it fits. How data virtualization liberates and empowers your business users via data discovery, data wrangling to generation of reusable reporting objects and data services. Digital transformation demands that we empower all consumers of data within the organization, it also demands agility too. Data Virtualization gives you meaningful access to information that can be shared by a myriad of consumers. Register to attend this session to learn: - What is Data Virtualization? - Why do I need Data Virtualization in my organization? - How do I implement Data Virtualization in my enterprise?

Data Virtualization: An Introduction

Denodo

Data mesh is a relatively recent term that describes a set of principles that good modern data systems uphold. A kind of “microservices” for the data-centric world. While the data mesh is not technology-specific as a pattern, the building of systems that adopt and implement data mesh principles have a relatively long history under different guises. In this talk, we share our recommendations and picks of what every developer should know about building a streaming data mesh with Kafka. We introduce the four principles of the data mesh: domain-driven decentralization, data as a product, self-service data platform, and federated governance. We then cover topics such as the differences between working with event streams versus centralized approaches and highlight the key characteristics that make streams a great fit for implementing a mesh, such as their ability to capture both real-time and historical data. We’ll examine how to onboard data from existing systems into a mesh, modelling the communication within the mesh, how to deal with changes to your domain’s “public” data, give examples of global standards for governance, and discuss the importance of taking a product-centric view on data sources and the data sets they share.

Apache Kafka and the Data Mesh | Michael Noll, Confluent

HostedbyConfluent

Future of Data Engineering

C4Media

The world of data architecture began with applications. Next came data warehouses. Then text was organized into a data warehouse. Then one day the world discovered a whole new kind of data that was being generated by organizations. The world found that machines generated data that could be transformed into valuable insights. This was the origin of what is today called the data lakehouse. The evolution of data architecture continues today. Come listen to industry experts describe this transformation of ordinary data into a data architecture that is invaluable to business. Simply put, organizations that take data architecture seriously are going to be at the forefront of business tomorrow. This is an educational event. Several of the authors of the book Building the Data Lakehouse will be presenting at this symposium.

Data Lakehouse Symposium | Day 1 | Part 2

Databricks

Data catalogs are in wide use today across hundreds of enterprises as a means to help data scientists and business analysts find and collaboratively analyze data. Over the past several years, customers have increasingly used data catalogs in applications beyond their search & discovery roots, addressing new use cases such as data governance, cloud data migration, and digital transformation. In this session, the founder and CEO of Alation will discuss the evolution of the data catalog, the many ways in which data catalogs are being used today, the importance of machine learning in data catalogs, and discuss the future of the data catalog as a platform for a broad range of data intelligence solutions.

Data Catalog as the Platform for Data Intelligence

Alation

Azure data analytics platform - A reference architecture

Rajesh Kumar

Watch full webinar here: https://bit.ly/30mHuYH What started to evolve as the most agile and real-time enterprise data fabric, data virtualization is proving to go beyond its initial promise and is becoming one of the most important enterprise big data fabrics. Denodo’s vision is to provide a unified data delivery layer as a logical data fabric, to bridge the gap between the IT and the business, hiding the underlying complexity and creating a semantic layer to expose data in a business friendly manner. Attend this webinar to learn: - What data virtualization really is - How it differs from other enterprise data integration technologies - Why data virtualization is finding enterprise-wide deployment inside some of the largest organizations - Business Value of data virtualization and customer use cases - Highlights of the newly launched Denodo Platform 8.0

Data Virtualization: Introduction and Business Value (UK)

Denodo

This describes a conceptual model approach to designing an enterprise data fabric. This is the set of hardware and software infrastructure, tools and facilities to implement, administer, manage and operate data operations across the entire span of the data within the enterprise across all data activities including data acquisition, transformation, storage, distribution, integration, replication, availability, security, protection, disaster recovery, presentation, analytics, preservation, retention, backup, retrieval, archival, recall, deletion, monitoring, capacity planning across all data storage platforms enabling use by applications to meet the data needs of the enterprise. The conceptual data fabric model represents a rich picture of the enterprise’s data context. It embodies an idealised and target data view. Designing a data fabric enables the enterprise respond to and take advantage of key related data trends: • Internal and External Digital Expectations • Cloud Offerings and Services • Data Regulations • Analytics Capabilities It enables the IT function demonstrate positive data leadership. It shows the IT function is able and willing to respond to business data needs. It allows the enterprise to meet data challenges • More and more data of many different types • Increasingly distributed platform landscape • Compliance and regulation • Newer data technologies • Shadow IT where the IT function cannot deliver IT change and new data facilities quickly It is concerned with the design an open and flexible data fabric that improves the responsiveness of the IT function and reduces shadow IT.

Designing An Enterprise Data Fabric

Alan McSweeney

Data Lakehouse, Data Mesh, and Data Fabric (r1)

James Serra

Mario Molina, Software Engineer CDC systems are usually used to identify changes in data sources, capture and replicate those changes to other systems. Companies are using CDC to sync data across systems, cloud migration or even applying stream processing, among others. In this presentation we’ll see CDC patterns, how to use it in Apache Kafka, and do a live demo! https://www.meetup.com/Mexico-Kafka/events/277309497/

CDC patterns in Apache Kafka®

confluent

Building Lakehouses on Delta Lake with SQL Analytics Primer

Databricks

Data Mesh

Piethein Strengholt

Infographic: Data Governance Best Practices

Enterprise Management Associates

Watch Paul's session from Fast Data Strategy on-demand here: https://goo.gl/3veKqw "Through 2020, 50% of enterprises will implement some form of data virtualization as one enterprise production option for data integration" according to Gartner. It is clear that data virtualization has become a driving force for companies to implement an agile, real-time and flexible enterprise data architecture. Attend this session to learn: • What data virtualization actually means and how it differs from traditional data integration approaches • The most important use cases and key patterns of data virtualization • The benefits of data virtualization

Data Virtualization: An Introduction

Denodo

Tendances (20)

Data Mesh for Dinner

How to Build the Data Mesh Foundation: A Principled Approach | Zhamak Dehghan...

Denodo Data Virtualization Platform: Overview (session 1 from Architect to Ar...

Why Data Virtualization? An Introduction

Data Lakehouse, Data Mesh, and Data Fabric (r2)

Virtual Flink Forward 2020: Netflix Data Mesh: Composable Data Processing - J...

Data Virtualization: An Introduction

Apache Kafka and the Data Mesh | Michael Noll, Confluent

Future of Data Engineering

Data Lakehouse Symposium | Day 1 | Part 2

Data Catalog as the Platform for Data Intelligence

Azure data analytics platform - A reference architecture

Data Virtualization: Introduction and Business Value (UK)

Designing An Enterprise Data Fabric

Data Lakehouse, Data Mesh, and Data Fabric (r1)

CDC patterns in Apache Kafka®

Building Lakehouses on Delta Lake with SQL Analytics Primer

Data Mesh

Infographic: Data Governance Best Practices

Data Virtualization: An Introduction

Similaire à Data lineage

Neo4j GraphDay Seattle- Sept19- Connected data imperative

Neo4j

Resume (1)

naveenreddytamma

Resume (1)

naveenreddytamma

Watch full webinar here: https://bit.ly/32c6TnG Advanced data science techniques, like machine learning, have proven an extremely useful tool to derive valuable insights from existing data. Platforms like Spark, and complex libraries for R, Python and Scala put advanced techniques at the fingertips of the data scientists. However, these data scientists spent most of their time looking for the right data and massaging it into a usable format. Data virtualization offers a new alternative to address these issues in a more efficient and agile way. Attend this webinar and learn: - How data virtualization can accelerate data acquisition and massaging, providing the data scientist with a powerful tool to complement their practice - How popular tools from the data science ecosystem: Spark, Python, Zeppelin, Jupyter, etc. integrate with Denodo - How you can use the Denodo Platform with large data volumes in an efficient way - About the success McCormick has had as a result of seasoning the Machine Learning and Blockchain Landscape with data virtualization

Advanced Analytics and Machine Learning with Data Virtualization

Denodo

I presented this at a 2014 Tableau Conference session with Albert Wong. Netflix relies on data to make decisions ranging from buying and recommending content, to improving the streaming experience on devices. This presentation shares our Big Data analytics architecture and the tools used to make data accessible throughout our business, focusing on how Tableau fits into our organization and why it aligns well with our culture.

DATA @ NFLX (Tableau Conference 2014 Presentation)

Blake Irvine

Watch full webinar here: https://bit.ly/3eBEoKH Presented at BIGIT's World Tech Festival 2022, ASEAN Ongoing digital transformation is generating new data assets that have the potential of offering organisations unprecedented insights into operations, business processes, customer behaviour, the competition, and much more. But, if organisations cannot effectively access, integrate, and govern their data that is distributed across on-premises and multiple cloud providers’ data platforms, they are doomed to fall short of realizing its value. A logical data fabric that uses virtualization capabilities can avoid the traditional approach of integrating data. In this session, you will learn how organisations can create a logical data fabric with data virtualization technology to: - Minimize data movement and data replication which can be time-consuming, expensive and pose security and compliance risks - Virtually integrate, manage and govern enterprise data across on-premises and cloud for insight generation and business decision making - Examine how and why a logical data fabric could benefit your organization today and future-proof your data architecture to meet new demands

Innovative and Agile Data Delivery, using 'A Logical Data Fabric'

Denodo

Watch: https://bit.ly/2DYsUhD Advanced data science techniques, like machine learning, have proven an extremely useful tool to derive valuable insights from existing data. Platforms like Spark, and complex libraries for R, Python and Scala put advanced techniques at the fingertips of the data scientists. However, these data scientists spent most of their time looking for the right data and massaging it into a usable format. Data virtualization offers a new alternative to address these issues in a more efficient and agile way. Attend this webinar and learn: - How data virtualization can accelerate data acquisition and massaging, providing the data scientist with a powerful tool to complement their practice - How popular tools from the data science ecosystem: Spark, Python, Zeppelin, Jupyter, etc. integrate with Denodo - How you can use the Denodo Platform with large data volumes in an efficient way - How Prologis accelerated their use of Machine Learning with data virtualization

Advanced Analytics and Machine Learning with Data Virtualization

Denodo

In this deck from the Stanford HPC Conference, Ryan Quick from Providentia Worldwide presents: High Availability HPC ~ Microservice Architectures for Supercomputing. "Microservices power cloud-native applications to scale thousands of times larger than single deployments. We introduce the notion of microservices for traditional HPC workloads. We will describe microservices generally, highlighting some of the more popular and large-scale applications. Then we examine similarities between large-scale cloud configurations and HPC environments. Finally we propose a microservice application for solving a traditional HPC problem, illustrating improved time-to-market and workload resiliency." Watch the video: https://insidehpc.com/2018/02/high-availability-hpc-microservice-architectures-supercomputing/ Learn more: http://www.providentiaworldwide.com/ and http://hpcadvisorycouncil.com Sign up for our insideHPC Newsletter: http://insidehpc.com/newsletter

High Availability HPC ~ Microservice Architectures for Supercomputing

inside-BigData.com

Many organizations are immature when it comes to data use. The answer lies in delivering a greater level of insight from data, straight to the point of need. Enter: machine learning. In this webinar, William will look at categories of organizational response to the challenge across strategy, architecture, modeling, processes, and ethics. Machine learning maturity levels tend to move in harmony across these categories. As a general principle of maturity models, you can’t skip levels in any category, nor can you advance in one category well beyond the others. Vis-à-vis ML, attaining and retaining momentum up the model is paramount for success. You will ascend the model through concerted efforts delivering business wins utilizing progressive elements of the model, and thereby increasing your machine learning maturity. The model will evolve. No plateaus are comfortable for long. With ML maturity markers, sequencing, and tactics, this webinar provides a plan for how to build analytic Data Architecture maturity in your organization.

ADV Slides: How to Improve Your Analytic Data Architecture Maturity

DATAVERSITY

Large-scale enterprise migration can be a complex undertaking, especially for organizations that re-architect solutions to leverage the benefits of the Cloud. FINRA, which regulates US equities and options markets, recently completed a 2.5-year migration and re-architecture of its Big Data platform. Their platform consumes billions of market events every day. FINRA has developed scalable platforms and services on AWS that enable migrating enterprise applications and business functions to the Cloud quickly. Their data management platform takes advantage of AWS storage and compute products. In this session, IT influencers and decision makers will learn lessons from FINRA’s migration, including how to create an enterprise-class Cloud architecture and which technology skills are required for transitioning to the Cloud. We also share examples of the business value FINRA has realized.

AWS re:Invent 2016: FINRA in the Cloud: the Big Data Enterprise (ENT313)

Amazon Web Services

Data in Motion - tech-intro-for-paris-hackathon

Cisco DevNet

Implement a Universal Data Distribution Architecture to Manage All Streaming ...

Timothy Spann

Simply Business' Data Platform

Dani Solà Lagares

Watch full webinar here: https://bit.ly/3offv7G Presented at AI Live APAC Advanced data science techniques, like machine learning, have proven an extremely useful tool to derive valuable insights from existing data. Platforms like Spark, and complex libraries for R, Python and Scala put advanced techniques at the fingertips of the data scientists. However, these data scientists spend most of their time looking for the right data and massaging it into a usable format. Data virtualization offers a new alternative to address these issues in a more efficient and agile way. Watch this on-demand session to learn how companies can use data virtualization to: - Create a logical architecture to make all enterprise data available for advanced analytics exercise - Accelerate data acquisition and massaging, providing the data scientist with a powerful tool to complement their practice - Integrate popular tools from the data science ecosystem: Spark, Python, Zeppelin, Jupyter, etc.

How Data Virtualization Puts Enterprise Machine Learning Programs into Produc...

Denodo

Watch full webinar here: https://bit.ly/3xj6fnm Presented at Chief Data Officer Live 2021 A/NZ The world is changing faster than ever. And for companies to compete and succeed they need to be agile in order to respond quickly to market changes and emerging opportunities. Data plays an integral role in achieving this business agility. However, given the complex nature of the enterprise data architecture finding and analysing data is an increasingly challenging task. Data virtualization is a modern data integration technique that integrates data in real-time, without having to physically replicate it. Watch on-demand this session to understand what data virtualization is and how it: - Delivers data in real-time, and without replication - Creates a logical architecture to provide a single view of truth - Centralises the data governance and security framework - Democratises data for faster decision making and business agility

Quicker Insights and Sustainable Business Agility Powered By Data Virtualizat...

Denodo

At the StampedeCon 2015 Big Data Conference: As a frequent recipient of the J.D. Powers award for excellence in customer service, T-Mobile takes great pride in the quality of care that we provide our customers. As smartphone technologies advance (and fragment), the challenge of providing quality technical support can be daunting. To address this challenge, T-Mobile is reinventing many of its traditional practices and embracing DevOps, cloud deployment and lambda architecture. Specifically: * Cassandra for fast and consistent writes (at scale), as well as low-latency reads * Apache Spark and EMR for processing data archived in S3 * Kafka for flexibility in data ingestion * Chef and CloudFormation Templates to automate deployments * Graphite and Riemann for monitoring The goals of this presentation are: * showcase how these technologies are helping T-Mobile be * successful in addressing these business challenges * share tactics for tackling customer preference management and data collection transparency * specific “lessons learned” while migrating to NoSQL, Big Data and The Cloud.

From SQL to NoSQL - StampedeCon 2015

StampedeCon

Tejas bichave m tech python

tejas bichave

Big data and the cloud are perfect partners for companies who want to unlock maximum value from all of their unstructured, semi-structured, and structured data. The challenge has been how to create and manage a reliable end-to-end solution that spans data ingestion, storage and analysis in the face of the volume, velocity and variety of big data sources. In this webinar, we will show you how to achieve big data bliss by combining StreamSets Data Collector, which specializes in creating and running complex any-to-any dataflows, with Microsoft's Azure Data Lake and Azure analytic solutions. We will walk through an example of how a major bank is using StreamSets to transport their on-premise data to the Azure Cloud Computing Platform and Azure Data Lake to take advantage of analytics tools with unprecedented scale and performance.

Enabling Next Gen Analytics with Azure Data Lake and StreamSets

Streamsets Inc.

2021 was a year full of unexpected data integration challenges, but one thing that didn’t change was the continued growth of the importance and value of data. By watching our customers adapt and cope through the consistent application of technology, we’ve learned that the future can be quickly adjusted to if we have up-to-date and readily available data to make decisions. As we consider the data integration landscape and look forward into 2022, we see a set of trends (some new, some old) that data leaders will need to consider as they work to provide competitive business value to their organizations: - The Continued Importance of Spatial - Data Ops as a Practice - Rising Data Volumes Demand Data Quality - Ubiquitous Hardware Supporting Augmented Reality - Agile Enterprise Integration Effortlessly Connects Systems - Real-Time Data Stream Processing - Flexible, Hybrid Deployment Options - Cost effective ARM based processing In this webinar, join co-founders Don Murray and Dale Lutz as they offer insight and predictions on what’s to come in these areas. To follow, they’ll host a Q&A session where you can get feedback and advice on solutions to your data challenges.

7 Emerging Data & Enterprise Integration Trends in 2022

Safe Software

The Briefing Room with Dean Abbott and Tableau Software Live Webcast July 23, 2013 http://www.insideanalysis.com Today’s desire for analytics extends well beyond the traditional domain of Business Intelligence. That’s partly because business users are realizing the value of mixing and matching all kinds of data, from all kinds of sources. One emerging market driver is Cloud-based data, and the desire companies have to analyze this data cohesively with their on-premise data sets. Register for this episode of The Briefing Room to learn from Analyst Dean Abbott, who will explain how the ability to access data in the cloud can play a critical role for generating business value from analytics. He’ll be briefed by Ellie Fields of Tableau Software who will tout Tableau’s latest release, which includes native connectors to cloud-based applications like Salesforce.com, Amazon Redshift, Google Analytics and BigQuery. She’ll also demonstrate how Tableau can combine cloud data with other data sources, including spreadsheets, databases, cubes and even Big Data.

Bridging the Gap: Analyzing Data in and Below the Cloud

Inside Analysis

Similaire à Data lineage (20)

Neo4j GraphDay Seattle- Sept19- Connected data imperative

Resume (1)

Advanced Analytics and Machine Learning with Data Virtualization

DATA @ NFLX (Tableau Conference 2014 Presentation)

Innovative and Agile Data Delivery, using 'A Logical Data Fabric'

Advanced Analytics and Machine Learning with Data Virtualization

High Availability HPC ~ Microservice Architectures for Supercomputing

ADV Slides: How to Improve Your Analytic Data Architecture Maturity

AWS re:Invent 2016: FINRA in the Cloud: the Big Data Enterprise (ENT313)

Data in Motion - tech-intro-for-paris-hackathon

Implement a Universal Data Distribution Architecture to Manage All Streaming ...

Simply Business' Data Platform

How Data Virtualization Puts Enterprise Machine Learning Programs into Produc...

Quicker Insights and Sustainable Business Agility Powered By Data Virtualizat...

From SQL to NoSQL - StampedeCon 2015

Tejas bichave m tech python

Enabling Next Gen Analytics with Azure Data Lake and StreamSets

7 Emerging Data & Enterprise Integration Trends in 2022

Bridging the Gap: Analyzing Data in and Below the Cloud

Dernier

Vip Mumbai Call Girls Thane West Call On 9920725232 With Body to body massage with sex Booking Contact Details :- WhatsApp Chat :- +91-9920725232 4-May-2024(SMW) Call Girls In Model Towh +91-9920725232 !! Best Woman Seeking Man Call Girls Service, Escorts Service in Home Hotel in NCR 24 Hours Available Service Call Girls, Contact Us +91-9920725232 (Any Time. Any Where) Call Girls in , Noida, Gurgaon, Ghaziabad,Sexy Indian Female Escorts Service NCRWelcome To Escorts Service – An All Over New Very Sexy Hot Call Girls Agency Service Escorts In South NCR’s No. 1 High Profile Independent Female Escorts Service. We Provide Good Quality Educated Profile At Very Regnebal Price 100% Safe And Original.We Are Provide Escorts Service All OYO Hotels ,3*,4*,5* Star Hotel And Home Flat, Apartment. Guest-House. Services In -Call And Out – Call Both Are Services Available. 24Hrs. Any Time Any Where. In All Over Noida Gurgaon Ghaziabad Faridabad.More Information And Contact Profile Real Pic Visit Our Website City Wise Escorts Service Agency.Good Looking Cheap And Best Models Girls U Can Get Best Click On Link……Night Call Girls Now In Hotel Le Meridien Gurgaon Near Female Escort One Shot — 5000/in call (time 1 hour), 6000/out call Two shot with one girl — 8000/in call (time 2 hour), 10000/out call Body to body massage with sex- 8000/in call (time 1 hour) Full night Service for one person– 12000/in call, 13000/out call (shot limit 3-4 shots) Full night Service for more than 1 person — please contact Us —9920725232 We are available 24*7 all days of the year. Call us — 9920725232 Thank you for Visiting.

Vip Mumbai Call Girls Thane West Call On 9920725232 With Body to body massage...

amitlee9823

Discover Why Less is More in B2B Research

michael115558

Anomaly detection and data imputation within time series

Paris Women in Machine Learning and Data Science

VIP Model Call Girls Hinjewadi ( Pune ) Call ON 8005736733 Starting From 5K to 25K High Profile Escorts In Pune Booking Now open +91- 8005736733 Why you Choose Us- +91- 8005736733 HOT⇄ 8005736733 Mr ashu ji Call Mr ashu Ji +91- 8005736733 (V030524]N) 𝐇𝐨𝐭𝐞𝐥 𝐑𝐨𝐨𝐦𝐬 𝐈𝐧𝐜𝐥𝐮𝐝𝐢𝐧𝐠 𝐑𝐚𝐭𝐞 𝐒𝐡𝐨𝐭𝐬/𝐇𝐨𝐮𝐫𝐲🆓 .█▬█⓿▀█▀ 𝐈𝐍𝐃𝐄𝐏𝐄𝐍𝐃𝐄𝐍𝐓 𝐆𝐈𝐑𝐋 𝐕𝐈𝐏 𝐄𝐒𝐂𝐎𝐑𝐓 Hello Guys ! High Profiles young Beauties and Good Looking standard Profiles Available , Enquire Now if you are interested in Hifi Service and want to get connect with someone who can understand your needs. Service offers you the most beautiful High Profile sexy independent female Escorts in genuine ✔✔✔ To enjoy with hot and sexy girls ✔✔✔ ★providing:- • Models • vip Models • Russian Models • Foreigner Models • TV Actress and Celebrities • Receptionist • Air Hostess • Call Center Working Girls/Women • Hi-Tech Co. Girls/Women • Housewife

VIP Model Call Girls Hinjewadi ( Pune ) Call ON 8005736733 Starting From 5K t...

SUHANI PANDEY

Call Girls Begur Just Call 👗 7737669865 👗 Top Class Call Girl Service Bangalore Booking Contact Details :- WhatsApp Chat :- +91-7737669865 2-May-2024(SMW) Call Girls In Model Towh Bangalore +91-7737669865 !! Best Woman Seeking Man Call Girls Service, Escorts Service in Home Hotel in Bangalore NCR 24 Hours Available Service Call Girls, Contact Us +91-7737669865 (Any Time. Any Where) Call Girls in Bangalore, Noida, Gurgaon, Ghaziabad,Sexy Indian Female Escorts Service Bangalore NCRWelcome To Bangalore Escorts Service – An All Over New Bangalore Very Sexy Hot Call Girls Agency Service Escorts In South BangaloreNCRBangalore’s No. 1 High Profile Independent Female Escorts Service. We Provide Good Quality Educated Profile At Very Regnebal Price 100% Safe And Original.We Are Provide Escorts Service All OYO Hotels ,3*,4*,5* Star Hotel And Home Flat, Apartment. Guest-House. Services In -Call And Out – Call Both Are Services Available. 24Hrs. Any Time Any Where. In All Over Bangalore Noida Gurgaon Ghaziabad Faridabad.More Information And Contact Profile Real Pic Visit Our Website City Wise Escorts Service Agency.Good Looking Cheap And Best Models Girls U Can Get Best Click On Link……Night Call Girls Now In Hotel Le Meridien Gurgaon Near Female Escort One Shot — 5000/in call (time 1 hour), 6000/out call Two shot with one girl — 8000/in call (time 2 hour), 10000/out call Body to body massage with sex- 8000/in call (time 1 hour) Full night Service for one person– 12000/in call, 13000/out call (shot limit 3-4 shots) Full night Service for more than 1 person — please contact Us —7737669865 We are available 24*7 all days of the year. Call us — 7737669865 Thank you for Visiting.

Call Girls Begur Just Call 👗 7737669865 👗 Top Class Call Girl Service Bangalore

amitlee9823

Call Girls Indiranagar Just Call 👗 7737669865 👗 Top Class Call Girl Service Bangalore Booking Contact Details :- WhatsApp Chat :- +91-7737669865 2-May-2024(SMW) Call Girls In Model Towh Bangalore +91-7737669865 !! Best Woman Seeking Man Call Girls Service, Escorts Service in Home Hotel in Bangalore NCR 24 Hours Available Service Call Girls, Contact Us +91-7737669865 (Any Time. Any Where) Call Girls in Bangalore, Noida, Gurgaon, Ghaziabad,Sexy Indian Female Escorts Service Bangalore NCRWelcome To Bangalore Escorts Service – An All Over New Bangalore Very Sexy Hot Call Girls Agency Service Escorts In South BangaloreNCRBangalore’s No. 1 High Profile Independent Female Escorts Service. We Provide Good Quality Educated Profile At Very Regnebal Price 100% Safe And Original.We Are Provide Escorts Service All OYO Hotels ,3*,4*,5* Star Hotel And Home Flat, Apartment. Guest-House. Services In -Call And Out – Call Both Are Services Available. 24Hrs. Any Time Any Where. In All Over Bangalore Noida Gurgaon Ghaziabad Faridabad.More Information And Contact Profile Real Pic Visit Our Website City Wise Escorts Service Agency.Good Looking Cheap And Best Models Girls U Can Get Best Click On Link……Night Call Girls Now In Hotel Le Meridien Gurgaon Near Female Escort One Shot — 5000/in call (time 1 hour), 6000/out call Two shot with one girl — 8000/in call (time 2 hour), 10000/out call Body to body massage with sex- 8000/in call (time 1 hour) Full night Service for one person– 12000/in call, 13000/out call (shot limit 3-4 shots) Full night Service for more than 1 person — please contact Us —7737669865 We are available 24*7 all days of the year. Call us — 7737669865 Thank you for Visiting.

Call Girls Indiranagar Just Call 👗 7737669865 👗 Top Class Call Girl Service B...

amitlee9823

Mature dropshipping via API with DroFx.pptx

olyaivanovalion

ALSO dropshipping via API with DroFx.pptx

olyaivanovalion

Call Girl In Dwarka ☎92055#41914 ¶¶ Indian,Russian Best Quality full Educated And Full Cooperative Independent Call Girls Escort Services In New Delhi- I Have Extremely Beautiful Broad Minded Cute Sexy & Hot Call Girls and Escorts, We Are Located in 3* 4* 5* Hotels in Delhi. Safe & Secure High Class Services Affordable Rate 100% Satisfaction, Unlimited Enjoyment. Any Time for Model/Teens Escort in Delhi High class luxury and premium escorts agency Indian Russian Call Girls In Delhi Booking Good High Profile Escorts (Call Girls) In Delhi 5 Star Hotel ,Incall Service,OutCall Service, We provide services by Call Girls,College Girls,Modals Get High Profile queens,Well Educated,Good Looking,Full Cooperative Model, Russian Models,Punjabi Girls Kashmeri Girls Services etc… We Provide Hottest Female With Safe And Consensual With Most Limits Respected Complete Satisfaction Guaranteed…Service. Call Me Spacial For Including Incall//outcall Service In New Delhi Indian Russian Escorts Service

Call Girls in Sarai Kale Khan Delhi 💯 Call Us 🔝9205541914 🔝( Delhi) Escorts S...

Delhi Call girls

Halmar dropshipping via API with DroFx

olyaivanovalion

Chintamani Call Girls: 🍓 7737669865 🍓 High Profile Model Escorts | Bangalore Escorts Service Booking Contact Details :- WhatsApp Chat :- +91-7737669865 2-May-2024(SMW) Call Girls In Model Towh Bangalore +91-7737669865 !! Best Woman Seeking Man Call Girls Service, Escorts Service in Home Hotel in Bangalore NCR 24 Hours Available Service Call Girls, Contact Us +91-7737669865 (Any Time. Any Where) Call Girls in Bangalore, Noida, Gurgaon, Ghaziabad,Sexy Indian Female Escorts Service Bangalore NCRWelcome To Bangalore Escorts Service – An All Over New Bangalore Very Sexy Hot Call Girls Agency Service Escorts In South BangaloreNCRBangalore’s No. 1 High Profile Independent Female Escorts Service. We Provide Good Quality Educated Profile At Very Regnebal Price 100% Safe And Original.We Are Provide Escorts Service All OYO Hotels ,3*,4*,5* Star Hotel And Home Flat, Apartment. Guest-House. Services In -Call And Out – Call Both Are Services Available. 24Hrs. Any Time Any Where. In All Over Bangalore Noida Gurgaon Ghaziabad Faridabad.More Information And Contact Profile Real Pic Visit Our Website City Wise Escorts Service Agency.Good Looking Cheap And Best Models Girls U Can Get Best Click On Link……Night Call Girls Now In Hotel Le Meridien Gurgaon Near Female Escort One Shot — 5000/in call (time 1 hour), 6000/out call Two shot with one girl — 8000/in call (time 2 hour), 10000/out call Body to body massage with sex- 8000/in call (time 1 hour) Full night Service for one person– 12000/in call, 13000/out call (shot limit 3-4 shots) Full night Service for more than 1 person — please contact Us —7737669865 We are available 24*7 all days of the year. Call us — 7737669865 Thank you for Visiting.

Chintamani Call Girls: 🍓 7737669865 🍓 High Profile Model Escorts | Bangalore ...

amitlee9823

Carero dropshipping via API with DroFx.pptx

olyaivanovalion

Edukaciniai dropshipping via API with DroFx

olyaivanovalion

Vip Mumbai Call Girls Marol Naka Call On 9920725232 With Body to body massage with sex Booking Contact Details :- WhatsApp Chat :- +91-9920725232 4-May-2024(SMW) Call Girls In Model Towh +91-9920725232 !! Best Woman Seeking Man Call Girls Service, Escorts Service in Home Hotel in NCR 24 Hours Available Service Call Girls, Contact Us +91-9920725232 (Any Time. Any Where) Call Girls in , Noida, Gurgaon, Ghaziabad,Sexy Indian Female Escorts Service NCRWelcome To Escorts Service – An All Over New Very Sexy Hot Call Girls Agency Service Escorts In South NCR’s No. 1 High Profile Independent Female Escorts Service. We Provide Good Quality Educated Profile At Very Regnebal Price 100% Safe And Original.We Are Provide Escorts Service All OYO Hotels ,3*,4*,5* Star Hotel And Home Flat, Apartment. Guest-House. Services In -Call And Out – Call Both Are Services Available. 24Hrs. Any Time Any Where. In All Over Noida Gurgaon Ghaziabad Faridabad.More Information And Contact Profile Real Pic Visit Our Website City Wise Escorts Service Agency.Good Looking Cheap And Best Models Girls U Can Get Best Click On Link……Night Call Girls Now In Hotel Le Meridien Gurgaon Near Female Escort One Shot — 5000/in call (time 1 hour), 6000/out call Two shot with one girl — 8000/in call (time 2 hour), 10000/out call Body to body massage with sex- 8000/in call (time 1 hour) Full night Service for one person– 12000/in call, 13000/out call (shot limit 3-4 shots) Full night Service for more than 1 person — please contact Us —9920725232 We are available 24*7 all days of the year. Call us — 9920725232 Thank you for Visiting.

Vip Mumbai Call Girls Marol Naka Call On 9920725232 With Body to body massage...

amitlee9823

Cheap Rate Call girls Sarita Vihar Delhi 9205541914 shot 1500 night

Delhi Call girls

Ravak dropshipping via API with DroFx.pptx

olyaivanovalion

Accredited-Transport-Cooperatives-Jan-2021-Web.pdf

adriantubila

Call Girls Hsr Layout Just Call 👗 7737669865 👗 Top Class Call Girl Service Bangalore Booking Contact Details :- WhatsApp Chat :- +91-7737669865 2-May-2024(SMW) Call Girls In Model Towh Bangalore +91-7737669865 !! Best Woman Seeking Man Call Girls Service, Escorts Service in Home Hotel in Bangalore NCR 24 Hours Available Service Call Girls, Contact Us +91-7737669865 (Any Time. Any Where) Call Girls in Bangalore, Noida, Gurgaon, Ghaziabad,Sexy Indian Female Escorts Service Bangalore NCRWelcome To Bangalore Escorts Service – An All Over New Bangalore Very Sexy Hot Call Girls Agency Service Escorts In South BangaloreNCRBangalore’s No. 1 High Profile Independent Female Escorts Service. We Provide Good Quality Educated Profile At Very Regnebal Price 100% Safe And Original.We Are Provide Escorts Service All OYO Hotels ,3*,4*,5* Star Hotel And Home Flat, Apartment. Guest-House. Services In -Call And Out – Call Both Are Services Available. 24Hrs. Any Time Any Where. In All Over Bangalore Noida Gurgaon Ghaziabad Faridabad.More Information And Contact Profile Real Pic Visit Our Website City Wise Escorts Service Agency.Good Looking Cheap And Best Models Girls U Can Get Best Click On Link……Night Call Girls Now In Hotel Le Meridien Gurgaon Near Female Escort One Shot — 5000/in call (time 1 hour), 6000/out call Two shot with one girl — 8000/in call (time 2 hour), 10000/out call Body to body massage with sex- 8000/in call (time 1 hour) Full night Service for one person– 12000/in call, 13000/out call (shot limit 3-4 shots) Full night Service for more than 1 person — please contact Us —7737669865 We are available 24*7 all days of the year. Call us — 7737669865 Thank you for Visiting.

Call Girls Hsr Layout Just Call 👗 7737669865 👗 Top Class Call Girl Service Ba...

amitlee9823

Invezz.com - Grow your wealth with trading signals

Invezz1

Call Girls Jalahalli Just Call 👗 7737669865 👗 Top Class Call Girl Service Bangalore Booking Contact Details :- WhatsApp Chat :- +91-7737669865 2-May-2024(SMW) Call Girls In Model Towh Bangalore +91-7737669865 !! Best Woman Seeking Man Call Girls Service, Escorts Service in Home Hotel in Bangalore NCR 24 Hours Available Service Call Girls, Contact Us +91-7737669865 (Any Time. Any Where) Call Girls in Bangalore, Noida, Gurgaon, Ghaziabad,Sexy Indian Female Escorts Service Bangalore NCRWelcome To Bangalore Escorts Service – An All Over New Bangalore Very Sexy Hot Call Girls Agency Service Escorts In South BangaloreNCRBangalore’s No. 1 High Profile Independent Female Escorts Service. We Provide Good Quality Educated Profile At Very Regnebal Price 100% Safe And Original.We Are Provide Escorts Service All OYO Hotels ,3*,4*,5* Star Hotel And Home Flat, Apartment. Guest-House. Services In -Call And Out – Call Both Are Services Available. 24Hrs. Any Time Any Where. In All Over Bangalore Noida Gurgaon Ghaziabad Faridabad.More Information And Contact Profile Real Pic Visit Our Website City Wise Escorts Service Agency.Good Looking Cheap And Best Models Girls U Can Get Best Click On Link……Night Call Girls Now In Hotel Le Meridien Gurgaon Near Female Escort One Shot — 5000/in call (time 1 hour), 6000/out call Two shot with one girl — 8000/in call (time 2 hour), 10000/out call Body to body massage with sex- 8000/in call (time 1 hour) Full night Service for one person– 12000/in call, 13000/out call (shot limit 3-4 shots) Full night Service for more than 1 person — please contact Us —7737669865 We are available 24*7 all days of the year. Call us — 7737669865 Thank you for Visiting.

Call Girls Jalahalli Just Call 👗 7737669865 👗 Top Class Call Girl Service Ban...

amitlee9823

Dernier (20)

Vip Mumbai Call Girls Thane West Call On 9920725232 With Body to body massage...

Discover Why Less is More in B2B Research

Anomaly detection and data imputation within time series

VIP Model Call Girls Hinjewadi ( Pune ) Call ON 8005736733 Starting From 5K t...

Call Girls Begur Just Call 👗 7737669865 👗 Top Class Call Girl Service Bangalore

Call Girls Indiranagar Just Call 👗 7737669865 👗 Top Class Call Girl Service B...

Mature dropshipping via API with DroFx.pptx

ALSO dropshipping via API with DroFx.pptx

Call Girls in Sarai Kale Khan Delhi 💯 Call Us 🔝9205541914 🔝( Delhi) Escorts S...

Halmar dropshipping via API with DroFx

Chintamani Call Girls: 🍓 7737669865 🍓 High Profile Model Escorts | Bangalore ...

Carero dropshipping via API with DroFx.pptx

Edukaciniai dropshipping via API with DroFx

Vip Mumbai Call Girls Marol Naka Call On 9920725232 With Body to body massage...

Cheap Rate Call girls Sarita Vihar Delhi 9205541914 shot 1500 night

Ravak dropshipping via API with DroFx.pptx

Accredited-Transport-Cooperatives-Jan-2021-Web.pdf

Call Girls Hsr Layout Just Call 👗 7737669865 👗 Top Class Call Girl Service Ba...

Invezz.com - Grow your wealth with trading signals

Call Girls Jalahalli Just Call 👗 7737669865 👗 Top Class Call Girl Service Ban...

Data lineage

1. Building and Scaling Data Lineage at Netflix Strata SF 2019 Di Lin, Girish Lingappa, Jitender Aswani (@jaswani)

2. CTA: Intentionally Purposeful to Improve Diversity in Tech

3. People, Security, & Infra Data Eng (DEI) Data Lineage Asset Inventory Anomaly Detection Leverage data to improve cloud infra

4. Show of hands

5. in thinking / planning stages?

6. currently building the system?

7. done building?

8. Imagination Trip (User Stories)

9. #1 Data Scientist validate source of the metrics?

10. #2 Software Engineer who downstream to you service is impacted?

11. #3 Reliability Engineer identify and alert on jobs / table at risk of missing SLA?

12. Detection and Data Cleansing Use Cases Inspired Lineage Retention and Data Efficiency Cost Attribution Platform Reliability Data Integrity

13. intentionality behind complex data landscape

14. Freedom & Responsibility distributed decision making => ● move fast, ● scale complexity, ● Judicious about what’s best for Netflix

15. Data Landscape

16. Top Challenges

17. Challenge # 1 Diverse Data Landscape First time expanding beyond traditional data warehouse realm

18. Challenge # 2 Platform Evolution New Platforms Upgrades

19. Challenge # 3 Data Conformance Variance in granularity of the meta data

20. Design Principles

21. Design Principle # 1 Ensure data coverage

22. Design Principle # 2 Enable seamless integration

23. Design Principle # 3 Envision a flexible data model

24. Data Coverage

25. Data Coverage - data gets pushed ● Platform tools publish events (preferred approach) ○ Compute engines ○ Data movement tools ○ Reporting tools

26. Data Coverage - we pull & parse data ● Ingestion scripts parse logs and metadata ○ Enumerate jobs and scripts ○ Parse plan info ○ Parse queries

27. Lineage Data Flow

28.

29. Use Cases

30. #1: Platform Reliability ● SLA Recommendation ● Predict SLA miss ● Instructive notifications ● Anomaly detection

31. #2 : Data visibility ● Reinstate key tables ● Deprecate tables ● Schema changes ● Contract no longer held

32. #3: Efficiency ● Cost evaluation for entities ○ Storage ○ Compute ○ Usage ● Retention ● Disaster recovery

33. Please visit us at jobs.netflix.com Continue your journey... Lineage blog Infra DE team blog

34. Netflix @ Strata ● Wednesday ○ Self-service data platform ● Thursday ○ App performance measurement ○ Security detection platform ○ Personalization enhancement 4:40 PM @ 2002 11:50 AM @ 2002 11:50 AM @ 2006 2:40 PM @ 2024

35. ???

36. Thank you.

37. “Provide a complete and accurate data lineage system enabling decision-makers to win moments of truth” Lineage Mission

Data lineage

Recommandé

Recommandé

Contenu connexe

Tendances

Tendances (20)

Similaire à Data lineage

Similaire à Data lineage (20)

Dernier

Dernier (20)

Data lineage