Brandon obrien streaming_data

•Télécharger en tant que PPTX, PDF•

2 j'aime•1,604 vues

Nitin Kumar

Brandon O'Brien, Principal Software Engineer at Expedia, Inc

Ingénierie

Streaming Data
Ecosystems
Brandon O’Brien
@hakczar
Expedia, Inc

Outline
 Streaming Data Opportunities & Challenges
 Streaming Data Ecosystem at Expedia
 Use Cases
 Questions?

Streaming Data: Opportunities
 Data -> Insights -> Data Products -> $$$
 Fresher data = better
 However…

Streaming Data: Challenges
 Today’s Challenge: System integration
 N^2 integrations
 Team roadmaps need to align

Streaming Data: Solutions
 Solution:
 Kafka with Streaming Data Ecosystem
 Decoupled systems & roadmaps
 Easy publish
 Easy consume
 Central data depot
 Talk Focus:
 Streaming Data Ecosystem that makes life easier for
Kafka producers and consumers

Streaming Data Ecosystem
 Self-Service
 Centralized Data Depot (logical)
 Primer, CICD, Cloud
 Elastic Components
 Monitoring: Velocity, Lag

Streaming Data Ecosystem cont’d
 Producers
 Publish data to HTTP Ingestor endpoint
 Ingestor -> Kafka
 Ingestor -> S3
 Consumers
 Create skeleton apps with built-in Kafka integration
 CICD pipeline for deployments

Streaming Data
Ecosystems
Brandon O’Brien
@hakczar
Expedia, Inc
Questions?

Recommandé

Moving eBay’s Data Warehouse Over to Apache Spark – Spark as Core ETL Platfor...Databricks

Kappa vs Lambda Architectures and Technology ComparisonKai Wähner

Deploying Kafka Streams Applications with Docker and Kubernetesconfluent

Building a healthy data ecosystem around Kafka and Hadoop: Lessons learned at...Yael Garten

Frame - Feature Management for Productive Machine LearningDavid Stein

High Performance Data Streaming with Amazon Kinesis: Best Practices (ANT322-R...Amazon Web Services

Tech Modernization: A Cloud MigrationVMware Tanzu

Serving models using KFServingTheofilos Papapanagiotou

Recommandé

Moving eBay’s Data Warehouse Over to Apache Spark – Spark as Core ETL Platfor...Databricks

Kappa vs Lambda Architectures and Technology ComparisonKai Wähner

Deploying Kafka Streams Applications with Docker and Kubernetesconfluent

Building a healthy data ecosystem around Kafka and Hadoop: Lessons learned at...Yael Garten

Frame - Feature Management for Productive Machine LearningDavid Stein

High Performance Data Streaming with Amazon Kinesis: Best Practices (ANT322-R...Amazon Web Services

Tech Modernization: A Cloud MigrationVMware Tanzu

Serving models using KFServingTheofilos Papapanagiotou

Data Streaming Ecosystem Management at Booking.com confluent

Graph Computing with JanusGraphJason Plurad

Algorithmic Music Recommendations at SpotifyChris Johnson

Introduction To Streaming Data and Stream Processing with Apache Kafkaconfluent

Kafka Tutorial - Introduction to Apache Kafka (Part 1)Jean-Paul Azar

Comparison of dbmsTech_MX

Cassandra at eBay - Cassandra Summit 2012Jay Patel

Distributed Tracing for Kafka with OpenTelemetry with Daniel Kim | Kafka Summ...HostedbyConfluent

Scaling Machine Learning Feature Engineering in Apache Spark at FacebookDatabricks

Simple math for anomaly detection toufic boubez - metafor software - monito...tboubez

Dynamic Rule-based Real-time Market Data AlertsFlink Forward

Application modernization patterns with apache kafka, debezium, and kubernete...Bilgin Ibryam

RedisConf17- Using Redis at scale @ TwitterRedis Labs

Bucket your partitions wisely - Cassandra summit 2016Markus Höfer

[Pcamp19] - Escalando o uso de dados no Nubank - André Tavares | NubankProduct Camp Brasil

Engineering Fast Indexes for Big-Data Applications: Spark Summit East talk by...Spark Summit

How to Build an Apache Kafka® Connectorconfluent

Using Modular Topologies in Kafka Streams to scale ksqlDB’s persistent querie...HostedbyConfluent

Real-Time Detection of Anomalies in the Database Infrastructure using Apache ...Spark Summit

Netflix Recommendations Using Spark + Cassandra (Prasanna Padmanabhan & Roopa...DataStax

Confluent kafka meetupseattle jan2017Nitin Kumar

Avvo fkafkaNitin Kumar

Contenu connexe

Tendances

Data Streaming Ecosystem Management at Booking.com confluent

Graph Computing with JanusGraphJason Plurad

Algorithmic Music Recommendations at SpotifyChris Johnson

Introduction To Streaming Data and Stream Processing with Apache Kafkaconfluent

Kafka Tutorial - Introduction to Apache Kafka (Part 1)Jean-Paul Azar

Comparison of dbmsTech_MX

Cassandra at eBay - Cassandra Summit 2012Jay Patel

Distributed Tracing for Kafka with OpenTelemetry with Daniel Kim | Kafka Summ...HostedbyConfluent

Scaling Machine Learning Feature Engineering in Apache Spark at FacebookDatabricks

Simple math for anomaly detection toufic boubez - metafor software - monito...tboubez

Dynamic Rule-based Real-time Market Data AlertsFlink Forward

Application modernization patterns with apache kafka, debezium, and kubernete...Bilgin Ibryam

RedisConf17- Using Redis at scale @ TwitterRedis Labs

Bucket your partitions wisely - Cassandra summit 2016Markus Höfer

[Pcamp19] - Escalando o uso de dados no Nubank - André Tavares | NubankProduct Camp Brasil

Engineering Fast Indexes for Big-Data Applications: Spark Summit East talk by...Spark Summit

How to Build an Apache Kafka® Connectorconfluent

Using Modular Topologies in Kafka Streams to scale ksqlDB’s persistent querie...HostedbyConfluent

Real-Time Detection of Anomalies in the Database Infrastructure using Apache ...Spark Summit

Netflix Recommendations Using Spark + Cassandra (Prasanna Padmanabhan & Roopa...DataStax

Tendances (20)

Data Streaming Ecosystem Management at Booking.com

Graph Computing with JanusGraph

Algorithmic Music Recommendations at Spotify

Introduction To Streaming Data and Stream Processing with Apache Kafka

Kafka Tutorial - Introduction to Apache Kafka (Part 1)

Comparison of dbms

Cassandra at eBay - Cassandra Summit 2012

Distributed Tracing for Kafka with OpenTelemetry with Daniel Kim | Kafka Summ...

Scaling Machine Learning Feature Engineering in Apache Spark at Facebook

Simple math for anomaly detection toufic boubez - metafor software - monito...

Dynamic Rule-based Real-time Market Data Alerts

Application modernization patterns with apache kafka, debezium, and kubernete...

RedisConf17- Using Redis at scale @ Twitter

Bucket your partitions wisely - Cassandra summit 2016

[Pcamp19] - Escalando o uso de dados no Nubank - André Tavares | Nubank

Engineering Fast Indexes for Big-Data Applications: Spark Summit East talk by...

How to Build an Apache Kafka® Connector

Using Modular Topologies in Kafka Streams to scale ksqlDB’s persistent querie...

Real-Time Detection of Anomalies in the Database Infrastructure using Apache ...

Netflix Recommendations Using Spark + Cassandra (Prasanna Padmanabhan & Roopa...

En vedette

Confluent kafka meetupseattle jan2017Nitin Kumar

Avvo fkafkaNitin Kumar

Microsoft kafka load imbalanceNitin Kumar

Age of "Stockal" Media ComethStockal

PIXTAにおけるABテストPIXTA Inc.

WALをバックアップとレプリケーションに使う方法Takashi Hoshino

PIXTAにおけるCloudSearch運用 - JAWS DAYS 2016 LTPIXTA Inc.

AWS Black Belt Techシリーズ Amazon CloudSearchAmazon Web Services Japan

Kafka at trivagoClemens Valiente

Time Series Analysis/ Forecasting Samudra Kanankearachchi

PIXTAの紹介PIXTA Inc.

Spark Streaming& Kafka-The Future of Stream Processing by Hari Shreedharan of...Data Con LA

AWS Black Belt Tech Webinar 2016 〜 Amazon CloudSearch & Amazon Elasticsearch ...Amazon Web Services Japan

Cloudera's FlumeCloudera, Inc.

Real Time Data Processing Using Spark StreamingHari Shreedharan

Kafka connectAndrew Stevenson

Iranian Startup EcosystemTechly

Real Time Data Processing using Spark Streaming | Data Day Texas 2015Cloudera, Inc.

Spark+flume seattleHari Shreedharan

並行実行制御の最適化手法Sho Nakazono

En vedette (20)

Confluent kafka meetupseattle jan2017

Avvo fkafka

Microsoft kafka load imbalance

Age of "Stockal" Media Cometh

PIXTAにおけるABテスト

WALをバックアップとレプリケーションに使う方法

PIXTAにおけるCloudSearch運用 - JAWS DAYS 2016 LT

AWS Black Belt Techシリーズ Amazon CloudSearch

Kafka at trivago

Time Series Analysis/ Forecasting

PIXTAの紹介

Spark Streaming& Kafka-The Future of Stream Processing by Hari Shreedharan of...

AWS Black Belt Tech Webinar 2016 〜 Amazon CloudSearch & Amazon Elasticsearch ...

Cloudera's Flume

Real Time Data Processing Using Spark Streaming

Kafka connect

Iranian Startup Ecosystem

Real Time Data Processing using Spark Streaming | Data Day Texas 2015

Spark+flume seattle

並行実行制御の最適化手法

Similaire à Brandon obrien streaming_data

ADF+Course+Deck.pdfChiquteRobledo

Architecting for change: LinkedIn's new data ecosystemYael Garten

Strata 2016 - Architecting for Change: LinkedIn's new data ecosystemShirshanka Das

Delta Lake OSS: Create reliable and performant Data Lake by Quentin AmbardParis Data Engineers !

(ARC346) Scaling To 25 Billion Daily Requests Within 3 Months On AWSAmazon Web Services

Software Defined Storage - Open Framework and Intel® Architecture TechnologiesOdinot Stanislas

Planning your Next-Gen Change Data Capture (CDC) Architecture in 2019 - Strea...Impetus Technologies

Outgrowing your-datacenterMike Kavis

The Scout24 Data Platform (A Technical Deep Dive)RaffaelDzikowski

Open Source Reliability for Data Lake with Apache Spark by Michael ArmbrustData Con LA

Making Big Data Analytics with Hadoop fast & easy (webinar slides)Yellowfin

Creating a Modern Data Architecture for Digital TransformationMongoDB

How to govern and secure a Data Mesh?confluent

Making Apache Spark Better with Delta LakeDatabricks

Stream Processing – Concepts and FrameworksGuido Schmutz

AWSomeBuilder3-v12-clean.pdfSal Marcus

AWS Partner Webcast - Analyze Big Data for Consumer Applications with Looker ...Amazon Web Services

Introduction to Stream ProcessingGuido Schmutz

Getting Started with Amazon RedshiftAmazon Web Services

AWS Summit Berlin 2013 - Big Data AnalyticsAWS Germany

Similaire à Brandon obrien streaming_data (20)

ADF+Course+Deck.pdf

Architecting for change: LinkedIn's new data ecosystem

Strata 2016 - Architecting for Change: LinkedIn's new data ecosystem

Delta Lake OSS: Create reliable and performant Data Lake by Quentin Ambard

(ARC346) Scaling To 25 Billion Daily Requests Within 3 Months On AWS

Software Defined Storage - Open Framework and Intel® Architecture Technologies

Planning your Next-Gen Change Data Capture (CDC) Architecture in 2019 - Strea...

Outgrowing your-datacenter

The Scout24 Data Platform (A Technical Deep Dive)

Open Source Reliability for Data Lake with Apache Spark by Michael Armbrust

Making Big Data Analytics with Hadoop fast & easy (webinar slides)

Creating a Modern Data Architecture for Digital Transformation

How to govern and secure a Data Mesh?

Making Apache Spark Better with Delta Lake

Stream Processing – Concepts and Frameworks

AWSomeBuilder3-v12-clean.pdf

AWS Partner Webcast - Analyze Big Data for Consumer Applications with Looker ...

Introduction to Stream Processing

Getting Started with Amazon Redshift

AWS Summit Berlin 2013 - Big Data Analytics

Plus de Nitin Kumar

Deep learning with kafkaNitin Kumar

2019 04 seattle_meetup___kafka_machine_learning___kai_waehnerNitin Kumar

Kafka meetup seattle 2019 mirus reliable, high performance replication for ap...Nitin Kumar

Processing trillions of events per day with apacheNitin Kumar

Ren cao kafka connectNitin Kumar

Insta clustr seattle kafka meetup presentation bbNitin Kumar

EventHub for kafka ecosystems kafka meetupNitin Kumar

Kafka eosNitin Kumar

Microsoft challenges of a multi tenant kafka serviceNitin Kumar

Net flix kafka seattle meetupNitin Kumar

Map r seattle streams meetup oct 2016Nitin Kumar

Linked in multi tier, multi-tenant, multi-problem kafkaNitin Kumar

Seattle kafka meetup nov 2015 published siphonNitin Kumar

Plus de Nitin Kumar (13)

Deep learning with kafka

2019 04 seattle_meetup___kafka_machine_learning___kai_waehner

Kafka meetup seattle 2019 mirus reliable, high performance replication for ap...

Processing trillions of events per day with apache

Ren cao kafka connect

Insta clustr seattle kafka meetup presentation bb

EventHub for kafka ecosystems kafka meetup

Kafka eos

Microsoft challenges of a multi tenant kafka service

Net flix kafka seattle meetup

Map r seattle streams meetup oct 2016

Linked in multi tier, multi-tenant, multi-problem kafka

Seattle kafka meetup nov 2015 published siphon

Dernier

Engineering Drawing focus on projection of planesRAJNEESHKUMAR341697

AIRCANVAS[1].pdf mini project for btech studentsvanyagupta248

Hostel management system project report..pdfKamal Acharya

1_Introduction + EAM Vocabulary + how to navigate in EAM.pdfAldoGarca30

FEA Based Level 3 Assessment of Deformed Tanks with Fluid Induced LoadsArindam Chakraborty, Ph.D., P.E. (CA, TX)

School management system project Report.pdfKamal Acharya

Online electricity billing project report..pdfKamal Acharya

Verification of thevenin's theorem for BEEE Lab (1).pptxchumtiyababu

Unit 4_Part 1 CSE2001 Exception Handling and Function Template and Class Temp...drmkjayanthikannan

kiln thermal load.pptx kiln tgermal loadhamedmustafa094

Generative AI or GenAI technology based PPTbhaskargani46

GEAR TRAIN- BASIC CONCEPTS AND WORKING PRINCIPLEselvakumar948

"Lesotho Leaps Forward: A Chronicle of Transformative Developments"mphochane1998

NO1 Top No1 Amil Baba In Azad Kashmir, Kashmir Black Magic Specialist Expert ...Amil baba

Online food ordering system project report.pdfKamal Acharya

HAND TOOLS USED AT ELECTRONICS WORK PRESENTED BY KOUSTAV SARKARKOUSTAV SARKAR

PE 459 LECTURE 2- natural gas basic concepts and propertiessarkmank1

Employee leave management system project.Kamal Acharya

DeepFakes presentation : brief idea of DeepFakesMayuraD1

data_management_and _data_science_cheat_sheet.pdfJiananWang21

Dernier (20)

Engineering Drawing focus on projection of planes

AIRCANVAS[1].pdf mini project for btech students

Hostel management system project report..pdf

1_Introduction + EAM Vocabulary + how to navigate in EAM.pdf

FEA Based Level 3 Assessment of Deformed Tanks with Fluid Induced Loads

School management system project Report.pdf

Online electricity billing project report..pdf

Verification of thevenin's theorem for BEEE Lab (1).pptx

Unit 4_Part 1 CSE2001 Exception Handling and Function Template and Class Temp...

kiln thermal load.pptx kiln tgermal load

Generative AI or GenAI technology based PPT

GEAR TRAIN- BASIC CONCEPTS AND WORKING PRINCIPLE

"Lesotho Leaps Forward: A Chronicle of Transformative Developments"

NO1 Top No1 Amil Baba In Azad Kashmir, Kashmir Black Magic Specialist Expert ...

Online food ordering system project report.pdf

HAND TOOLS USED AT ELECTRONICS WORK PRESENTED BY KOUSTAV SARKAR

PE 459 LECTURE 2- natural gas basic concepts and properties

Employee leave management system project.

DeepFakes presentation : brief idea of DeepFakes

data_management_and _data_science_cheat_sheet.pdf

Brandon obrien streaming_data

1. Streaming Data Ecosystems Brandon O’Brien @hakczar Expedia, Inc

2. Outline  Streaming Data Opportunities & Challenges  Streaming Data Ecosystem at Expedia  Use Cases  Questions?

3. Outline  Streaming Data Opportunities & Challenges  Streaming Data Ecosystem at Expedia  Use Cases  Questions?

4. Streaming Data: Opportunities  Data -> Insights -> Data Products -> $$$  Fresher data = better  However…

5. Streaming Data: Challenges  Today’s Challenge: System integration  N^2 integrations  Team roadmaps need to align

6. Streaming Data: Solutions  Solution:  Kafka with Streaming Data Ecosystem  Decoupled systems & roadmaps  Easy publish  Easy consume  Central data depot  Talk Focus:  Streaming Data Ecosystem that makes life easier for Kafka producers and consumers

7. Outline  Streaming Data Opportunities & Challenges  Streaming Data Ecosystem at Expedia  Use Cases  Questions?

8. Ecosystem Architecture

9. Streaming Data Ecosystem  Self-Service  Centralized Data Depot (logical)  Primer, CICD, Cloud  Elastic Components  Monitoring: Velocity, Lag

10. Streaming Data Ecosystem cont’d  Producers  Publish data to HTTP Ingestor endpoint  Ingestor -> Kafka  Ingestor -> S3  Consumers  Create skeleton apps with built-in Kafka integration  CICD pipeline for deployments

11. Outline  Streaming Data Opportunities & Challenges  Streaming Data Ecosystem at Expedia  Use Cases  Questions?

12. Use Case: Streaming Analytics

13. Streaming Data Ecosystems Brandon O’Brien @hakczar Expedia, Inc Questions?