Kafka in Context, Cloud
and Community
Simon Elliston Ball, Product Manager
Data-in-Motion
Cloudera

© 2020 Cloudera, Inc. All rights reserved. 2

© 2020 Cloudera, Inc. All rights reserved. 3
Kafka and Friends
UNIFIED SCHEMA MANAGEMENT - SCHEMA REGISTRY
UNIFIED LINEAGE MANAGEMENT - APACHE ATLAS
UNIFIED MONITORING AND CONTROL - STREAMS MESSAGING MANAGER

© 2020 Cloudera, Inc. All rights reserved. 4
Apache NiFi: Collecting, sending and integrating
MQTT Proxy
REST Proxy

© 2020 Cloudera, Inc. All rights reserved. 5
Apache Flink: ANSI SQL on streams, CEP, advanced analytics
CREATE VIEW TransactionsInArea AS
SELECT
toAreaId(lat, lon) AS area,
COUNT(DISTINCT transactionId) AS transactionCount,
TUMBLE_ROWTIME(transactionTime, INTERVAL '30' MINUTE) AS rowTime,
TUMBLE_START(transactionTime, INTERVAL '30' MINUTE) AS startTime,
TUMBLE_END(transactionTime, INTERVAL '30' MINUTE) AS endTime
FROM
Transactions
GROUP BY
toAreaId(lat, lon),
TUMBLE(transactionTime, INTERVAL '30' MINUTE)

© 2020 Cloudera, Inc. All rights reserved. 6
Apache Flink: ANSI SQL on streams, CEP, advanced analytics
SELECT * FROM TransactionsInArea MATCH_RECOGNIZE(
PARTITION BY area ORDER BY rowTime
MEASURES
FIRST(UP.startTime) as peakStart,
LAST(DOWN.endTime) AS peakEnd,
SUM(UP.transactionCount) + SUM(DOWN.transactionCount) AS transactionSum
AFTER MATCH SKIP PAST LAST ROW
PATTERN (UP{4,} DOWN{2,} E)
DEFINE
UP AS UP.transactionCount > LAST(UP.transactionCount, 1) OR
LAST(UP.transactionCount, 1) IS NULL,
DOWN AS DOWN.transactionCount < LAST(DOWN.transactionCount, 1) OR
LAST(DOWN.transactionCount, 1) IS NULL,
E AS E.rideCount > LAST(DOWN.transactionCount)
)

© 2020 Cloudera, Inc. All rights reserved. 7
Apache Ranger: Security and Audit

© 2020 Cloudera, Inc. All rights reserved. 8
Apache Atlas: Lineage and Governance - linking the rest of the world
CDF
CDE
CML

© 2020 Cloudera, Inc. All rights reserved. 9
Schema Registry: consistent governance across the ecosystem
Schema for more than just Kafka
Record Based
Processors
Serializer
Deserializer
FlinkKafkaConsumer &
FlinkKafkaProducer
Stream Reader
Event
Explorer

© 2020 Cloudera, Inc. All rights reserved. 10
Wow. That’s a lot.

© 2020 Cloudera, Inc. All rights reserved. 11
Better together: complete solution for data
A fully integrated edge-to-cloud platform
for streaming and batch data with
advanced analytics and machine learning

© 2020 Cloudera, Inc. All rights reserved. 12
Managing complexity: Streams Messaging Manager

© 2020 Cloudera, Inc. All rights reserved. 13
Managing complexity: Streams Messaging Manager

© 2020 Cloudera, Inc. All rights reserved. 14
Managing complexity: Streams Messaging Manager

© 2020 Cloudera, Inc. All rights reserved. 15
Managing complexity: Streams Messaging Manager
Control lifecycle, configuration, and monitoring for
Kafka Connect instances

© 2020 Cloudera, Inc. All rights reserved. 16
Operations made easy: Cruise Control
Automatic Topic
Balancing
New Brokers Scale Up and Down Failure Leads to
Hot-Spot

© 2020 Cloudera, Inc. All rights reserved. 17
Operations made easy: Cruise Control

Kafka in the community

© 2020 Cloudera, Inc. All rights reserved. 19
Mirror Maker 2:
● Circular Replication
● Offset adjustment
● Open source
● Managed and
integrated into Streams
Replication Manager
and Streams
Messaging Manager
Mirror Maker 2 and Streams Replication Manager

© 2020 Cloudera, Inc. All rights reserved. 20
As Kafka Grows…..
Kafka Connect
Simple Data
Movement
In/Out of Kafka
Kafka Streams
Management and
Monitoring Services
Simpler Kafka… we all grow with it

Kafkas, clouds, kubes, and consumers

© 2020 Cloudera, Inc. All rights reserved. 22
Not just Kafka as a service… Topics as a service

© 2020 Cloudera, Inc. All rights reserved. 23
… with all the things you need around them

© 2020 Cloudera, Inc. All rights reserved. 24
… simple quick flow-based applications
Instant flows based for
common use cases
Autoscaling based on SLAs

© 2020 Cloudera, Inc. All rights reserved. 25
On cloud, on prem, hybrid, kubernetes: Kafka, where you need it.
● Run on any cloud
● Control your VPC
● Public or Private
● Auto-scaling
● Auto-balancing
● Machine learning
based sizing

© 2020 Cloudera, Inc. All rights reserved. 26
TH N Y U