As a streaming platform, Apache Kafka provides low-latency, high-throughput, fault-tolerant publish and subscribe pipelines and excels at processing streams of real-time events. Kafka provides reliable, millisecond delivery for connecting downstream systems with real-time data. In this talk, we will show how easy it is to leverage Kafka and the Elasticsearch connector to keep your indices populated with the latest data from the rest of your enterprise, as it changes.

1. 1
Integrating Apache Kafka
and Elastic using the
Connect Framework

2. 2
I ❤ Elastic 😁

3. Kafka
Cluster
3
Apache Kafka®
Kafka
A Distributed Commit Log. Publish and subscribe to
streams of records. Highly scalable, high throughput.
Supports transactions. Persisted data.
Reads are a single seek &
Writes are
append only

4. 4
Apache Kafka®
Kafka Streams API
Write standard Java applications & microservices
to process your data in real-time
Kafka Connect API
Reliable and scalable integration of Kafka
with other systems – no coding required.
Orders
Table
Customers
Kafka Streams API

5. 5
Many Systems are a bit of a mess…

6. 6
The Streaming Platform

7. 7
The Streaming Platform

8. 8
Why Kafka & Elastic?

9. Event-Centric Thinking
Streaming
Platform
“A product was viewed”
Elasticsear
ch
web
app

10. Event-Centric Thinking
Streaming
Platform
“A product was viewed”
web
app
mobile
app
APIs
Elasticsear
ch

11. mobile
app
web
app
APIs
Streaming
Platform
Hadoop
Security
Monitoring
Elastic
search
“A product was
Event-Centric Thinking

12. System Availability and Event Buffering
Producer Elasticsearch

13. System Availability and Event Buffering
Producer Elasticsearch

14. Native Stream Processing
Raw
SLA
breaches
Alert
Stream
Processing
App
Serve

15. Visualise & Analyse data from Kafka

16. 16
Integrating Elastic and Kafka

17. 17
Integrating Elastic with Kafka - Beats, Logstash
output.kafka:
hosts: ["localhost:9092"]
topic: 'logs'
required_acks: 1
output {
kafka {
topic_id => "logstash_logs_json"
bootstrap_servers => "localhost:9092"
codec => json
}
}
Beats
Logstash

18. 18

19. 19
Kafka Connect
Kafka Brokers
Kafka Connect
Tasks Workers
Sources Sinks
Amazon S3
syslog
flat file

20. 20
Kafka -> Elasticsearch

21. 21
Kafka Connect's Elasticsearch Sink
{
"name": "es-sink",
"config": {
"connector.class":
"io.confluent.connect.elasticsearch.ElasticsearchSinkConnector",
"connection.url": "http://localhost:9200",
"type.name": "type.name=kafka-connect",
"topics": "foobar"
}
}

22. 22
Kafka Connect to stream Kafka Topics to Elasticsearch

23. 23
Kafka Connect
Elasticsearch Sink Properties
https://docs.confluent.io/current/connect/connect-elasticsearch/docs/configuration_options.html

24. 24
Sink properties : Converters
• Json, Avro, String, Protobuf, etc
• Specify the converter in the Kafka Connect configuration, e.g.
key.converter=org.apache.kafka.connect.json.JsonConverter
value.converter=org.apache.kafka.connect.json.JsonConverter
• Kafka Connect uses pluggable converters for both message key and
value deserialisation

25. 25
Schemas & Document Mappings

26. 26
Schemas in Kafka Connect - JSON
{"schema":
{"type":"struct",
"fields":[{"type":"int32","optional":true,"field":"c1"},
{"type":"string","optional":true,"field":"c2"},
{"type":"int64","optional":false,
"name":"org.apache.kafka.connect.data.Timestamp","field":"create_ts"},
{"type":"int64","optional":false,
"name":"org.apache.kafka.connect.data.Timestamp","field":"update_ts"}],
"optional":false,
"name":"foobar"
},
"payload":{ "c1":100,
"c2":"bar",
"create_ts":1516747629000,
"update_ts":1516747629000}
}

27. 27
Kafka Connect + Schema Registry = WIN
Avro
Messag
e
Schema
Registry
Avro
Schema
Kafka
Connect

28. 28
Single Message Transform (SMT) -- Extract, TRANSFORM, Load…
• Modify events before storing in Kafka:
• Mask/drop sensitive information
• Set partitioning key
• Store lineage
• Cast data types
• Modify events going out of Kafka:
• Direct events to different Elasticsearch
indexes
• Mask/drop sensitive information
• Cast data types to match destination

29. 29
Confluent Platform: Enterprise Streaming based on Apache Kafka®
Database Changes Log Events loT Data Web Events …
CRM
Data Warehouse
Database
Hadoop
Data
Integration
…
Monitoring
Analytics
Custom Apps
Transformations
Real-time Applications
…
Apache Open Source Confluent Open Source Confluent Enterprise
Confluent Platform
Confluent Platform
Apache Kafka®
Core | Connect API | Streams API
Data Compatibility
Schema Registry
Monitoring & Administration
Confluent Control Center | Security
Operations
Replicator | Auto Data Balancing
Development and Connectivity
Clients | Connectors | REST Proxy | CLI
Apache Open Source Confluent Open Source Confluent Enterprise
SQL Stream Processing
KSQL

30. 30
https://www.confluent.io/download/
Streaming ETL, powered by Apache Kafka and Confluent Platform
https://www.confluent.io/blog/simplest-useful-kafka-connect-data-pipeline-world-thereabouts-part-1/
https://docs.confluent.io/current/connect/connect-elasticsearch/docs/