Watch this talk here: https://www.confluent.io/online-talks/integrating-apache-kafka-into-your-environment-on-demand Integrating Apache Kafka with other systems in a reliable and scalable way is a key part of an event streaming platform. This session will show you how to get streams of data into and out of Kafka with Kafka Connect and REST Proxy, maintain data formats and ensure compatibility with Schema Registry and Avro, and build real-time stream processing applications with Confluent KSQL and Kafka Streams. This session is part 4 of 4 in our Fundamentals for Apache Kafka series.

1. 1
Fundamentals for Apache Kafka®
Integrating Kafka intoYour Environment
Russ Sayers, Sr. Technical Trainer, Confluent

2. 2
Session Schedule
● Session 1: Benefits of Stream Processing and Apache Kafka Use Cases
● Session 2: Apache Kafka Architecture & Fundamentals Explained
● Session 3: How Apache Kafka Works
● Session 4: Integrating Apache Kafka into your Environment

3. 3
Learning Objectives
After this module you will be able to:
● Explain how the Confluent REST Proxy works
● Justify why Confluent Schema Registry is an
essential piece of a streaming platform powered by
Kafka
● Sketch where and how Kafka Connect is used in
streaming ETL
● Name main features of Kafka Streams
● Elaborate on goals of Confluent KSQL

4. 4
Development and Connectivity
How do I get streams of data
into and out of Kafka?

5. 5
Kafka Connect
Import and Export Data In & Out of Kafka
Fault Tolerant
Manage hundreds of data
sources and sinks
Preserves data schema
Integrated within Confluent
Control Center

6. 6
Kafka Connect
● Framework for Streaming Data between Kafka and other Systems
● Open Source
● Simple, Scalable, and Reliable

7. 7
Kafka Connect
Connectors: Connect Kafka Easily with Data Sources and Sinks

8. 8
Kafka Connect

9. 9
Confluent REST Proxy
Talk to non-native Kafka Apps and outside the Firewall
Provides a RESTful interface
to a Kafka cluster
Simplifies message creation
and consumption
Simplifies administrative
actions

10. 10
Confluent REST Proxy

11. 11
Data Compatibility
How do I maintain my
data formats and ensure
compatibility?

12. 12
The Challenge of Data Compatibility at Scale
Many sources without a policy
causes mayhem in a centralized
data pipeline
Ensuring downstream systems can
use the data is key to an
operational stream pipeline
Even within a single application,
different formats can be presented

13. 13
Confluent Schema Registry
Make Data Backwards Compatible and Future-Proof
• Define the expected fields for each
Kafka topic
• Automatically handle schema
changes (e.g., new fields)
• Prevent backwards incompatible
changes
• Support multi-data center
environments

14. 14
Confluent Schema Registry

15. 15
AVRO

16. 16
Schema Evolution

17. 17
Stream Processing
How do I build real-time
applications?

18. 18
Kafka Streams
Transform Data with Real-Time Applications
Overview
• Write standard Java applications
• No separate processing cluster required
• Exactly-once processing semantics
• Elastic, highly scalable, fault-tolerant
• Fully integrated with Kafka security
Example Use Cases
• Microservices
• Continuous queries
• Continuous transformations

19. 19
Your Kafka Streams App

20. 20
Kafka Streams

21. 21
Confluent KSQL:
Streaming SQL Engine for Apache Kafka
Develop real-time stream processing apps writing only SQL!
No Java, Python, or other boilerplate to wrap around it.
ksql> CREATE STREAM vip_actions AS
SELECT userid, page, action
FROM clickstream c
LEFT JOIN users u ON c.userid = u.user_id
WHERE u.level = ’Platinum’;

22. 22
Confluent KSQL
Enable Stream Processing using SQL-like Semantics
Leverage Kafka Streams API
without any coding required
Use any programming language
Connect via Control Center, CLI,
REST, or headless
Example Use Cases
• Streaming ETL
• Anomaly detection
• Event monitoring

23. 23
Confluent KSQL & Apache Kafka = easy

24. 25
Continue your Apache Kafka Education!
● Confluent Operations for Apache Kafka
● Confluent Developer Skills for Building Apache Kafka
● Confluent Stream Processing using Apache Kafka
Streams and KSQL
● Confluent Advanced Skills for Optimizing Apache Kafka
For more details, seehttp://confluent.io/training

25. 2626
Certifications
Confluent Certified Developer
for Apache Kafka
(aligns to Confluent Developer Skills
for Building Apache Kafka course)
Confluent Certified
Administrator for Apache
Kafka
(aligns to Confluent Operations Skills
for Apache Kafka)
What you Need to Know
○ Qualifications: 6-to-9 months hands-on
experience
○ Duration: 90 mins
○ Availability: Live, online 24/7
○ Cost: $150
○ Register online:
www.confluent.io/certification

26. 2727
cnfl.io/slack
Stay in touch!
cnfl.io/kafka-trainingcnfl.io/download

27. 2828
cnfl.io/blog cnfl.io/slack
Stay in touch!
cnfl.io/kafka-training

28. 29
Thank you for attending!
• Thank you for attending thesession!
• Feedback to: training-admin@confluent.io

29. 30
Copyright ©Confluent, Inc. 2014-2019. Privacy Policy | Terms &Conditions.
Apache, Apache Kafka, Kafka and the Kafka logo are trademarks of
the Apache Software Foundation