Jay Kreps is the CEO of Confluent, Inc., a company backing the popular Apache Kafka® messaging system. Prior to founding Confluent, he was formerly the lead architect for data infrastructure at LinkedIn. He is among the original authors of several open source projects including Project Voldemort (a key-value store). Apache Kafka (a distributed messaging system) and Apache Samza (a stream processing system).
3. 33
The world is changing.
Then
Hardware product
Up front purchase
Opaque
No data
Now
Hardware, Software & Global Internet Service
On-demand
Real-time visibility
Built on a foundation of data
Transportation
12. 1212
This is a fundamental paradigm shift...
Infrastructure
as code
Data as a continuous
stream of events
Future of the
datacenter
Future of data
Cloud
Event
Streaming
14. 1414
Two Problems in Application Infrastructure
What’s the state
of the world?
What’s happening
in the world?
Solution:
Databases
Solution:
Messaging, RPC, ETL, etc.
18. 1818
ETL/Data Integration Messaging
Transient MessagesStored records
ETL/Data Integration MessagingMessaging
Batch
Expensive
Time Consuming
Difficult to Scale
No Persistence
Data Loss
No Replay
High Throughput
Durable
Persistent
Maintains Order
Fast (Low Latency)
Event Streaming Paradigm
High Throughput
Durable
Persistent
Maintains Order
Fast (Low Latency)
19. 1919
To rethink data as not stored records
or transient messages, but instead as
a continually updating stream of events
Event Streaming Paradigm
28. 28
Stream Processing with KSQL
Stream 01:
Stream 02:
Stream
Table
Processing streams of data to create more streams or tables
QueryQuery
Continuous
Query
29. 29
KSQL: a few use cases
A big step towards a
streaming-first world:
● Real-time monitoring
and analytics
● Streaming ETL, not
Batch ETL
● Application development
33. 33
Confluent Products: Software and SaaS
Confluent Platform
The Enterprise Distribution of
Apache Kafka
Self-Managed Software
Deploy on any platform, on-prem or cloud
VM
Confluent Cloud
Apache Kafka Re-engineered
for the Cloud
Fully-Managed Service
Available on the leading public clouds
34. 34
Announcing Confluent Platform 5.2
Accelerate
Development
● Free forever Confluent
Platform on single
broker
● Librdkafka 1.0
● New KSQL expressions
Operate
at Scale
● Control Center
○ Schema management
○ Dynamic broker config
○ Multi-cluster
KSQL/Connect
○ Improved scalability
Enable Hybrid
Streaming
● Schema Registry
replication from
Confluent Platform to
Confluent Cloud
Apache Kafka 2.2.0