Compacted topics grow over time and are often utilizing high performance, low latency and relatively expensive storage solutions. Reducing duplicated data plays a critical role in the size of compacted topics. with less data on the topics, the Kafka cluster consumes less disk space which in turn it leads to lower operation cost. in this use case-driven talk, we are going to demonstrate how our team at UnitedHealth Group leveraged existing transformers to extract data from the message metadata in the topic as well as how we developed our customized transformers to minimize the amount of duplicated data in each message in the topic.

1. Optum Technology
Provider Data Platform
Siavash Sedghi
Lead Software Engineer
September 2021

2. Developing Custom Transformations
in the Kafka Connect to Minimize Data
Redundancy
Sep 2021 – Kafka Summit Americas

3. Our Mission
To help people live healthier lives and to help
make the health system work better for everyone.

4. Optum by numbers
o Optum serves 127 million individual consumers
o Optum serves 9 out of 10 U.S. hospitals
o Optum serves ~9 out of 10 Fortune 100 employers
o Optum serves more than 90,000 physicians, practices and other health care facilities as
well as non-profit associations and organizations
o Optum serves a network of over 67,000 pharmacies
4

5. Architecture
5

6. The Journey Begins
Data ingestion from multiple resources (Kafka Connect)
Data enrichment (KTable Joins & Streaming API)
Aggregation and metrics calculation (Kafka Streaming API)
Sinking data to database (Kafka Connect)
Near real-time APIs to serve the data
6

7. Broker Instance Fee Broker Storage Fee Data Transfer Fee
Cost of Operating a Kafka Cluster
7

8. Broker Instance Fee Broker Storage Fee Data Transfer Fee
Cost of Operating a Kafka Cluster
8
Reducing Broker Storage Cost by Minimizing Data
Redundancy in Topics

9. Build-in Transformations
9

10. Example:
10

11. Example:
11

12. Example:
12

13. Example:
13

14. Example:
14

15. Implementing a Transformation
15

16. Implementing a Transformation
16
Add your Jar file to the base docker image

17. Implementing a Transformation
17
docker logs -f kafka-connect
cat /var/log/kafka/connect.log

18. DEMO

19. Siavash Sedghi
19

20. Optum® is a registered trademark of Optum, Inc. in the U.S. and other jurisdictions. All other brand or product names
are the property of their respective owners. Because we are continuously improving our products and services,
Optum reserves the right to change specifications without prior notice. Optum is an equal opportunity employer.
© 2021 Optum, Inc. All rights reserved.