The document discusses Kafka in the context of cloud platforms and open source communities. It describes several Apache projects that can be used with Kafka, such as Apache NiFi for data collection, Apache Flink for stream processing, and Apache Ranger for security. It also outlines features of Cloudera's platform for managing Kafka deployments, including unified governance tools, monitoring, and services to simplify operations. Finally, it discusses how Kafka can be deployed across cloud, on-premise, and hybrid environments with auto-scaling and other management capabilities.

1. Kafka in Context, Cloud
and Community
Simon Elliston Ball, Product Manager
Data-in-Motion
Cloudera

2. © 2020 Cloudera, Inc. All rights reserved. 2

3. © 2020 Cloudera, Inc. All rights reserved. 3
Kafka and Friends
UNIFIED SCHEMA MANAGEMENT - SCHEMA REGISTRY
UNIFIED LINEAGE MANAGEMENT - APACHE ATLAS
UNIFIED MONITORING AND CONTROL - STREAMS MESSAGING MANAGER

4. © 2020 Cloudera, Inc. All rights reserved. 4
Apache NiFi: Collecting, sending and integrating
MQTT Proxy
REST Proxy

5. © 2020 Cloudera, Inc. All rights reserved. 5
Apache Flink: ANSI SQL on streams, CEP, advanced analytics
CREATE VIEW TransactionsInArea AS
SELECT
toAreaId(lat, lon) AS area,
COUNT(DISTINCT transactionId) AS transactionCount,
TUMBLE_ROWTIME(transactionTime, INTERVAL '30' MINUTE) AS rowTime,
TUMBLE_START(transactionTime, INTERVAL '30' MINUTE) AS startTime,
TUMBLE_END(transactionTime, INTERVAL '30' MINUTE) AS endTime
FROM
Transactions
GROUP BY
toAreaId(lat, lon),
TUMBLE(transactionTime, INTERVAL '30' MINUTE)

6. © 2020 Cloudera, Inc. All rights reserved. 6
Apache Flink: ANSI SQL on streams, CEP, advanced analytics
SELECT * FROM TransactionsInArea MATCH_RECOGNIZE(
PARTITION BY area ORDER BY rowTime
MEASURES
FIRST(UP.startTime) as peakStart,
LAST(DOWN.endTime) AS peakEnd,
SUM(UP.transactionCount) + SUM(DOWN.transactionCount) AS transactionSum
AFTER MATCH SKIP PAST LAST ROW
PATTERN (UP{4,} DOWN{2,} E)
DEFINE
UP AS UP.transactionCount > LAST(UP.transactionCount, 1) OR
LAST(UP.transactionCount, 1) IS NULL,
DOWN AS DOWN.transactionCount < LAST(DOWN.transactionCount, 1) OR
LAST(DOWN.transactionCount, 1) IS NULL,
E AS E.rideCount > LAST(DOWN.transactionCount)
)

7. © 2020 Cloudera, Inc. All rights reserved. 7
Apache Ranger: Security and Audit

8. © 2020 Cloudera, Inc. All rights reserved. 8
Apache Atlas: Lineage and Governance - linking the rest of the world
CDF
CDE
CML

9. © 2020 Cloudera, Inc. All rights reserved. 9
Schema Registry: consistent governance across the ecosystem
Schema for more than just Kafka
Record Based
Processors
Serializer
Deserializer
FlinkKafkaConsumer &
FlinkKafkaProducer
Stream Reader
Event
Explorer

10. © 2020 Cloudera, Inc. All rights reserved. 10
Wow. That’s a lot.

11. © 2020 Cloudera, Inc. All rights reserved. 11
Better together: complete solution for data
A fully integrated edge-to-cloud platform
for streaming and batch data with
advanced analytics and machine learning

12. © 2020 Cloudera, Inc. All rights reserved. 12
Managing complexity: Streams Messaging Manager

13. © 2020 Cloudera, Inc. All rights reserved. 13
Managing complexity: Streams Messaging Manager

14. © 2020 Cloudera, Inc. All rights reserved. 14
Managing complexity: Streams Messaging Manager

15. © 2020 Cloudera, Inc. All rights reserved. 15
Managing complexity: Streams Messaging Manager
Control lifecycle, configuration, and monitoring for
Kafka Connect instances

16. © 2020 Cloudera, Inc. All rights reserved. 16
Operations made easy: Cruise Control
Automatic Topic
Balancing
New Brokers Scale Up and Down Failure Leads to
Hot-Spot

17. © 2020 Cloudera, Inc. All rights reserved. 17
Operations made easy: Cruise Control

18. Kafka in the community

19. © 2020 Cloudera, Inc. All rights reserved. 19
Mirror Maker 2:
● Circular Replication
● Offset adjustment
● Open source
● Managed and
integrated into Streams
Replication Manager
and Streams
Messaging Manager
Mirror Maker 2 and Streams Replication Manager

20. © 2020 Cloudera, Inc. All rights reserved. 20
As Kafka Grows…..
Kafka Connect
Simple Data
Movement
In/Out of Kafka
Kafka Streams
Management and
Monitoring Services
Simpler Kafka… we all grow with it

21. Kafkas, clouds, kubes, and consumers

22. © 2020 Cloudera, Inc. All rights reserved. 22
Not just Kafka as a service… Topics as a service

23. © 2020 Cloudera, Inc. All rights reserved. 23
… with all the things you need around them

24. © 2020 Cloudera, Inc. All rights reserved. 24
… simple quick flow-based applications
Instant flows based for
common use cases
Autoscaling based on SLAs

25. © 2020 Cloudera, Inc. All rights reserved. 25
On cloud, on prem, hybrid, kubernetes: Kafka, where you need it.
● Run on any cloud
● Control your VPC
● Public or Private
● Auto-scaling
● Auto-balancing
● Machine learning
based sizing

26. © 2020 Cloudera, Inc. All rights reserved. 26
TH N Y U