Kafka high-throughput persistent streams

•Télécharger en tant que PPT, PDF•

16 j'aime•6,959 vues

Yahoo Developer Network

Art & Photos Technologie Formation

Kafka high-throughput, persistent, multi-reader streams http://sna-projects.com/kafka

[object Object],[object Object],[object Object]

Problem How do you model and process stream data for a large website?

Examples ,[object Object],[object Object],[object Object],[object Object],[object Object]

Tracking Basics ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Existing messaging systems ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Ideas ,[object Object],[object Object],[object Object],[object Object]

Performance Test ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Performance Summary ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

How can we get high performance with persistence?

Some tricks ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Implementation ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Distribution ,[object Object],[object Object]

Consumer State ,[object Object],[object Object],[object Object],[object Object]

APIs ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Stream processing (0.06 release) Data published to persistent topics, and redistributed by primary key between stages

Also coming soon ,[object Object],[object Object],[object Object],[object Object]

The End http://sna-projects.com/kafka https://github.com/kafka-dev/kafka [email_address] [email_address]

Recommandé

Yahoo! Hadoop User Group - May Meetup - Extraordinarily rapid and robust data...Hadoop User Group

Building a Scalable Web Crawler with HadoopHadoop User Group

Spark Streaming & Kafka-The Future of Stream ProcessingJack Gudenkauf

Facebook Presto presentationCyanny LIANG

Building Realtim Data Pipelines with Kafka Connect and Spark StreamingGuozhang Wang

Jack Gudenkauf sparkug_20151207_7Jack Gudenkauf

An evening with Jay Kreps; author of Apache Kafka, Samza, Voldemort & Azkaban.Data Con LA

HBase at MendeleyDan Harvey

Recommandé

Yahoo! Hadoop User Group - May Meetup - Extraordinarily rapid and robust data...Hadoop User Group

Building a Scalable Web Crawler with HadoopHadoop User Group

Spark Streaming & Kafka-The Future of Stream ProcessingJack Gudenkauf

Facebook Presto presentationCyanny LIANG

Building Realtim Data Pipelines with Kafka Connect and Spark StreamingGuozhang Wang

Jack Gudenkauf sparkug_20151207_7Jack Gudenkauf

An evening with Jay Kreps; author of Apache Kafka, Samza, Voldemort & Azkaban.Data Con LA

HBase at MendeleyDan Harvey

Flurry Analytic Backend - Processing Terabytes of Data in Real-timeTrieu Nguyen

Prestogres, ODBC & JDBC connectivity for PrestoSadayuki Furuhashi

Bullet: A Real Time Data Query EngineDataWorks Summit

Presto at TwitterBill Graham

Apache kafkaDaan Gerits

Apache Drill (ver. 0.1, check ver. 0.2)Camuel Gilyadov

Efficient State Management With Spark 2.0 And Scale-Out DatabasesJen Aman

Understanding Presto - Presto meetup @ Tokyo #1Sadayuki Furuhashi

Architecture of a Kafka camus infrastructuremattlieber

Automatic Scaling Iterative ComputationsGuozhang Wang

HBaseConAsia2018 Track1-5: Improving HBase reliability at PInterest with geo ...Michael Stack

Yahoo! Hadoop User Group - May Meetup - HBase and Pig: The Hadoop ecosystem a...Hadoop User Group

PrestoChen Chun

Searching At ScaleHadoop User Group

Building Realtime Data Pipelines with Kafka Connect and Spark StreamingJen Aman

Impala presentation ahad ranaData Con LA

Cassandra Introduction & FeaturesPhil Peace

Marc Schwering – Using Flink with MongoDB to enhance relevancy in personaliza...Flink Forward

HBaseConAsia2018 Track2-6: Scaling 30TB's of data lake with Apache HBase and ...Michael Stack

Apache Kafka at LinkedInGuozhang Wang

January 2011 HUG: Pig PresentationYahoo Developer Network

January 2011 HUG: Howl PresentationYahoo Developer Network

Contenu connexe

Tendances

Flurry Analytic Backend - Processing Terabytes of Data in Real-timeTrieu Nguyen

Prestogres, ODBC & JDBC connectivity for PrestoSadayuki Furuhashi

Bullet: A Real Time Data Query EngineDataWorks Summit

Presto at TwitterBill Graham

Apache kafkaDaan Gerits

Apache Drill (ver. 0.1, check ver. 0.2)Camuel Gilyadov

Efficient State Management With Spark 2.0 And Scale-Out DatabasesJen Aman

Understanding Presto - Presto meetup @ Tokyo #1Sadayuki Furuhashi

Architecture of a Kafka camus infrastructuremattlieber

Automatic Scaling Iterative ComputationsGuozhang Wang

HBaseConAsia2018 Track1-5: Improving HBase reliability at PInterest with geo ...Michael Stack

Yahoo! Hadoop User Group - May Meetup - HBase and Pig: The Hadoop ecosystem a...Hadoop User Group

PrestoChen Chun

Searching At ScaleHadoop User Group

Building Realtime Data Pipelines with Kafka Connect and Spark StreamingJen Aman

Impala presentation ahad ranaData Con LA

Cassandra Introduction & FeaturesPhil Peace

Marc Schwering – Using Flink with MongoDB to enhance relevancy in personaliza...Flink Forward

HBaseConAsia2018 Track2-6: Scaling 30TB's of data lake with Apache HBase and ...Michael Stack

Apache Kafka at LinkedInGuozhang Wang

Tendances (20)

Flurry Analytic Backend - Processing Terabytes of Data in Real-time

Prestogres, ODBC & JDBC connectivity for Presto

Bullet: A Real Time Data Query Engine

Presto at Twitter

Apache kafka

Apache Drill (ver. 0.1, check ver. 0.2)

Efficient State Management With Spark 2.0 And Scale-Out Databases

Understanding Presto - Presto meetup @ Tokyo #1

Architecture of a Kafka camus infrastructure

Automatic Scaling Iterative Computations

HBaseConAsia2018 Track1-5: Improving HBase reliability at PInterest with geo ...

Yahoo! Hadoop User Group - May Meetup - HBase and Pig: The Hadoop ecosystem a...

Presto

Searching At Scale

Building Realtime Data Pipelines with Kafka Connect and Spark Streaming

Impala presentation ahad rana

Cassandra Introduction & Features

Marc Schwering – Using Flink with MongoDB to enhance relevancy in personaliza...

HBaseConAsia2018 Track2-6: Scaling 30TB's of data lake with Apache HBase and ...

Apache Kafka at LinkedIn

En vedette

January 2011 HUG: Pig PresentationYahoo Developer Network

January 2011 HUG: Howl PresentationYahoo Developer Network

Yahoo compares Storm and SparkChicago Hadoop Users Group

August 2016 HUG: Better together: Fast Data with Apache Spark™ and Apache Ign...Yahoo Developer Network

August 2016 HUG: Open Source Big Data Ingest with StreamSets Data Collector Yahoo Developer Network

August 2016 HUG: Recent development in Apache OozieYahoo Developer Network

En vedette (6)

January 2011 HUG: Pig Presentation

January 2011 HUG: Howl Presentation

Yahoo compares Storm and Spark

August 2016 HUG: Better together: Fast Data with Apache Spark™ and Apache Ign...

August 2016 HUG: Open Source Big Data Ingest with StreamSets Data Collector

August 2016 HUG: Recent development in Apache Oozie

Similaire à Kafka high-throughput persistent streams

Big Data Streams Architectures. Why? What? How?Anton Nazaruk

Open HFT libraries in @JavaPeter Lawrey

AI&BigData Lab 2016. Сарапин Виктор: Размер имеет значение: анализ по требова...GeeksLab Odessa

Luxun a Persistent Messaging System Tailored for Big Data Collecting & AnalyticsWilliam Yang

Apache ActiveMQ - Enterprise messaging in actiondejanb

Apache Kafka - Scalable Message-Processing and more !Guido Schmutz

Scalable Apache for Beginnerswebhostingguy

In search of the perfect IoT Stack - Scalable IoT Architectures with MQTTDominik Obermaier

Spinnaker VLDB 2011sandeep_tata

NoSQL afternoon in Japan Kumofs & MessagePackSadayuki Furuhashi

NoSQL afternoon in Japan kumofs & MessagePackSadayuki Furuhashi

Apache kafkaCristiano Altmann

Messaging - RabbitMQ, Azure (Service Bus), Docker and Azure FunctionsJohn Staveley

[@NaukriEngineering] Messaging QueuesNaukri.com

Voldemort & Hadoop @ Linkedin, Hadoop User Group Jan 2010Bhupesh Bansal

Hadoop and Voldemort @ LinkedInHadoop User Group

Cluster_Performance_Apache_Kafak_vs_RabbitMQShameera Rathnayaka

Distributed Systems: scalability and high availabilityRenato Lucindo

APIs & Underlying Protocols #APICraftSFAndrei Savu

Data Pipelines with Apache KafkaBen Stopford

Similaire à Kafka high-throughput persistent streams (20)

Big Data Streams Architectures. Why? What? How?

Open HFT libraries in @Java

AI&BigData Lab 2016. Сарапин Виктор: Размер имеет значение: анализ по требова...

Luxun a Persistent Messaging System Tailored for Big Data Collecting & Analytics

Apache ActiveMQ - Enterprise messaging in action

Apache Kafka - Scalable Message-Processing and more !

Scalable Apache for Beginners

In search of the perfect IoT Stack - Scalable IoT Architectures with MQTT

Spinnaker VLDB 2011

NoSQL afternoon in Japan Kumofs & MessagePack

NoSQL afternoon in Japan kumofs & MessagePack

Apache kafka

Messaging - RabbitMQ, Azure (Service Bus), Docker and Azure Functions

[@NaukriEngineering] Messaging Queues

Voldemort & Hadoop @ Linkedin, Hadoop User Group Jan 2010

Hadoop and Voldemort @ LinkedIn

Cluster_Performance_Apache_Kafak_vs_RabbitMQ

Distributed Systems: scalability and high availability

APIs & Underlying Protocols #APICraftSF

Data Pipelines with Apache Kafka

Plus de Yahoo Developer Network

Developing Mobile Apps for Performance - Swapnil Patel, Verizon MediaYahoo Developer Network

Athenz - The Open-Source Solution to Provide Access Control in Dynamic Infras...Yahoo Developer Network

Athenz & SPIFFE, Tatsuya Yano, Yahoo JapanYahoo Developer Network

Athenz with Istio - Single Access Control Model in Cloud Infrastructures, Tat...Yahoo Developer Network

CICD at Oath using ScrewdriverYahoo Developer Network

Big Data Serving with Vespa - Jon Bratseth, Distinguished Architect, OathYahoo Developer Network

How @TwitterHadoop Chose Google Cloud, Joep Rottinghuis, Lohit VijayaRenuYahoo Developer Network

The Future of Hadoop in an AI World, Milind Bhandarkar, CEO, AmpoolYahoo Developer Network

Apache YARN Federation and Tez at Microsoft, Anupam Upadhyay, Adrian Nicoara,...Yahoo Developer Network

Containerized Services on Apache Hadoop YARN: Past, Present, and Future, Shan...Yahoo Developer Network

HDFS Scalability and Security, Daryn Sharp, Senior Engineer, OathYahoo Developer Network

Hadoop {Submarine} Project: Running deep learning workloads on YARN, Wangda T...Yahoo Developer Network

Moving the Oath Grid to Docker, Eric Badger, OathYahoo Developer Network

Architecting Petabyte Scale AI ApplicationsYahoo Developer Network

Introduction to Vespa – The Open Source Big Data Serving Engine, Jon Bratseth...Yahoo Developer Network

Jun 2017 HUG: YARN Scheduling – A Step BeyondYahoo Developer Network

Jun 2017 HUG: Large-Scale Machine Learning: Use Cases and Technologies Yahoo Developer Network

February 2017 HUG: Slow, Stuck, or Runaway Apps? Learn How to Quickly Fix Pro...Yahoo Developer Network

February 2017 HUG: Exactly-once end-to-end processing with Apache ApexYahoo Developer Network

February 2017 HUG: Data Sketches: A required toolkit for Big Data AnalyticsYahoo Developer Network

Plus de Yahoo Developer Network (20)

Developing Mobile Apps for Performance - Swapnil Patel, Verizon Media

Athenz - The Open-Source Solution to Provide Access Control in Dynamic Infras...

Athenz & SPIFFE, Tatsuya Yano, Yahoo Japan

Athenz with Istio - Single Access Control Model in Cloud Infrastructures, Tat...

CICD at Oath using Screwdriver

Big Data Serving with Vespa - Jon Bratseth, Distinguished Architect, Oath

How @TwitterHadoop Chose Google Cloud, Joep Rottinghuis, Lohit VijayaRenu

The Future of Hadoop in an AI World, Milind Bhandarkar, CEO, Ampool

Apache YARN Federation and Tez at Microsoft, Anupam Upadhyay, Adrian Nicoara,...

Containerized Services on Apache Hadoop YARN: Past, Present, and Future, Shan...

HDFS Scalability and Security, Daryn Sharp, Senior Engineer, Oath

Hadoop {Submarine} Project: Running deep learning workloads on YARN, Wangda T...

Moving the Oath Grid to Docker, Eric Badger, Oath

Architecting Petabyte Scale AI Applications

Introduction to Vespa – The Open Source Big Data Serving Engine, Jon Bratseth...

Jun 2017 HUG: YARN Scheduling – A Step Beyond

Jun 2017 HUG: Large-Scale Machine Learning: Use Cases and Technologies

February 2017 HUG: Slow, Stuck, or Runaway Apps? Learn How to Quickly Fix Pro...

February 2017 HUG: Exactly-once end-to-end processing with Apache Apex

February 2017 HUG: Data Sketches: A required toolkit for Big Data Analytics

Dernier

Jagat Puri Call Girls : ☎ 8527673949, Low rate Call Girlsashishs7044

Benjamin Portfolio Process Work Slideshowssuser971f6c

Triangle Vinyl Record Store, Clermont FloridaGabrielaMiletti

San Jon Motel, Motel/Residence, San Jon NMroute66connected

Call Girl in Bur Dubai O5286O4116 Indian Call Girls in Bur Dubai By VIP Bur D...dajasot375

Laxmi Nagar Call Girls : ☎ 8527673949, Low rate Call Girlsashishs7044

Khanpur Call Girls : ☎ 8527673949, Low rate Call Girlsashishs7044

Downtown Call Girls O5O91O128O Pakistani Call Girls in Downtowndajasot375

Kristy Soto's Industrial design PortfolioKristySoto

Burari Call Girls : ☎ 8527673949, Low rate Call Girlsashishs7044

Low Rate Call Girls in Laxmi Nagar Delhi Call 9990771857delhimodel235

8377087607, Door Step Call Girls In Gaur City (NOIDA) 24/7 Availabledollysharma2066

Russian⚡ Call Girls In Sector 39 Noida✨8375860717⚡Escorts Servicedoor45step

FULL ENJOY - 9953040155 Call Girls in Gandhi Vihar | DelhiMalviyaNagarCallGirl

FULL ENJOY - 9953040155 Call Girls in Paschim Vihar | DelhiMalviyaNagarCallGirl

Bobbie goods colorinsssssssssssg book.pdflunavro0105

Hauz Khas Call Girls : ☎ 8527673949, Low rate Call Girlsashishs7044

Retail Store Scavanger Hunt - Foundation College Parkjosebenzaquen

Russian⚡ Call Girls In Sector 104 Noida✨8375860717⚡Escorts Servicedoor45step

Iffco Chowk Call Girls : ☎ 8527673949, Low rate Call Girlsashishs7044

Dernier (20)

Jagat Puri Call Girls : ☎ 8527673949, Low rate Call Girls

Benjamin Portfolio Process Work Slideshow

Triangle Vinyl Record Store, Clermont Florida

San Jon Motel, Motel/Residence, San Jon NM

Call Girl in Bur Dubai O5286O4116 Indian Call Girls in Bur Dubai By VIP Bur D...

Laxmi Nagar Call Girls : ☎ 8527673949, Low rate Call Girls

Khanpur Call Girls : ☎ 8527673949, Low rate Call Girls

Downtown Call Girls O5O91O128O Pakistani Call Girls in Downtown

Kristy Soto's Industrial design Portfolio

Burari Call Girls : ☎ 8527673949, Low rate Call Girls

Low Rate Call Girls in Laxmi Nagar Delhi Call 9990771857

8377087607, Door Step Call Girls In Gaur City (NOIDA) 24/7 Available

Russian⚡ Call Girls In Sector 39 Noida✨8375860717⚡Escorts Service

FULL ENJOY - 9953040155 Call Girls in Gandhi Vihar | Delhi

FULL ENJOY - 9953040155 Call Girls in Paschim Vihar | Delhi

Bobbie goods colorinsssssssssssg book.pdf

Hauz Khas Call Girls : ☎ 8527673949, Low rate Call Girls

Retail Store Scavanger Hunt - Foundation College Park

Russian⚡ Call Girls In Sector 104 Noida✨8375860717⚡Escorts Service

Iffco Chowk Call Girls : ☎ 8527673949, Low rate Call Girls

Kafka high-throughput persistent streams

1. Kafka high-throughput, persistent, multi-reader streams http://sna-projects.com/kafka

3. Problem How do you model and process stream data for a large website?

6. Tracking

10.

11.

12.

13.

14. How can we get high performance with persistence?

15.

16.

17.

18. Hadoop InputFormat

19.

20.

21. Stream processing (0.06 release) Data published to persistent topics, and redistributed by primary key between stages

22.

23. The End http://sna-projects.com/kafka https://github.com/kafka-dev/kafka [email_address] [email_address]

Notes de l'éditeur

No state