Presented at Orange County Advanced Analytics and Big Data Meetup, June 21 2019. Apache Kafka has fast become the dominant messaging technology for the enterprise; if you're a data scientist or data engineer and you have not yet worked with Kafka, that situation will likely change soon! In this session, Pat Patterson, director of evangelism at StreamSets, explains what Kafka is, why it has disrupted the previous generation of messaging products, and how you can use open source products to build dataflow pipelines with Kafka, without writing code.

1. 1© StreamSets, Inc. All rights reserved.
Data Integration with Apache Kafka:
What, Why, How
Orange County Advanced Analytics and Big Data Meetup
Pat Patterson
pat@streamsets.com | @metadaddy

2. 2© StreamSets, Inc. All rights reserved.
Who Am I?
Pat Patterson / pat@streamsets.com / @metadaddy
Past: Sun Microsystems, Salesforce
Present: Director of Evangelism, StreamSets
I run far ☺

3. 3© StreamSets, Inc. All rights reserved.
Agenda
What is Kafka?
Why use Kafka?
How do I use Kafka?

4. 4© StreamSets, Inc. All rights reserved.
Open-source stream-processing software platform
Developed at LinkedIn
Donated to the Apache Software Foundation
Commercialized as part of Confluent Platform
In use by > 40% of F5001 across all industries.
(1) Apache Kafka Refcard - https://dzone.com/refcardz/apache-kafka
What is Apache Kafka?

5. 5© StreamSets, Inc. All rights reserved.
Core - publish/subscribe messaging
Streams - library for real-time apps and microservices
Connect - integration framework
Kafka Components

6. 6© StreamSets, Inc. All rights reserved.
Topic - logical stream of records
Partition - disjoint, ordered set of records within a topic
Offset - unique sequential record index, per partition
Record - key, value, timestamp, headers
Broker - stores records in partitions
Cluster - comprises multiple brokers
Kafka Terminology

7. 7© StreamSets, Inc. All rights reserved.
From https://kafka.apache.org/intro.html
Kafka Partition

8. 8© StreamSets, Inc. All rights reserved.
Kafka Topic
From https://kafka.apache.org/intro.html

9. 9© StreamSets, Inc. All rights reserved.
Putting It All Together
Kafka Cluster
Topic
Partition
Partition
Partition
Topic
Partition
Partition
Partition
Topic
Partition
Partition
Partition
Broker
Broker
Broker

10. 10© StreamSets, Inc. All rights reserved.
Kafka Consumer Group
From https://kafka.apache.org/documentation/

11. 11© StreamSets, Inc. All rights reserved.
Contrast with Traditional Message Queues
Message Queue Kafka
~30-40K messages/sec per node ~600K-800K messages/sec per node
Any payload size Optimized for small payload: 1KB
FIFO queue Commit log - ‘topic’
Messages stored until consumer ack Messages stored according to policy
Messages in a queue are consumed once,
typically by a single application
Messages in a topic may be consumed by
multiple applications; consumers may
replay messages from any point in time
See also https://jack-vanlightly.com/blog/2017/12/3/rabbitmq-vs-kafka-series-introduction

12. 12© StreamSets, Inc. All rights reserved.
Throughput - a single server can handle 100s of MB/sec
Availability - data can be stored redundantly across multiple servers in a cluster
No single point of failure
Scalability - cluster can scale out by adding more servers
Integration - client libraries for Java, Python, C/C++, Go, .Net and many more
Benefits of Kafka

13. 13© StreamSets, Inc. All rights reserved.
Impedance - decouple producers from consumers, apply back pressure
Buffering - handle spiky workloads
Aggregation - store record stream, replay records from a given instant
When to Use Kafka

14. 14© StreamSets, Inc. All rights reserved.
kafka.apache.org/quickstart
Getting Started with Kafka

15. 15© StreamSets, Inc. All rights reserved.
Kafka Basics Demo

16. 16© StreamSets, Inc. All rights reserved.
Example Log Ingest Use Case
UDP
Many producers
One record per message
Asynchronous message flow
Unreliable transport
Single cloud service
Thousands of messages/object
Synchronous object creation
Reliable storage

17. 17© StreamSets, Inc. All rights reserved.
UDP to S3 Demo

18. 18© StreamSets, Inc. All rights reserved.
Consuming records from flat files
Low velocity/volume/variety of messages
High throughput AND global ordering is required
When NOT to Use Kafka

19. 19© StreamSets, Inc. All rights reserved.
Kafka Is Not the Only Game in Town
Apache Pulsar
Second mover advantage
Has queue semantics as well as streaming
More flexible message retention

20. 20© StreamSets, Inc. All rights reserved.
Conclusion
Apache Kafka is the de facto stream-processing platform for enterprises
Kafka can handle Internet-scale load
but may be overkill for simple use cases
StreamSets Data Collector is a great, open source, tool to work with Kafka

21. 21© StreamSets, Inc. All rights reserved.
September 3-5, 2019
Tue, Sep 3 - Training & Tutorials
Wed-Thu, Sep 4-5, Keynote & Breakouts
Hilton Financial District
(Tue|Wed|Thur)

22. 22© StreamSets, Inc. All rights reserved.
QUESTIONS?