Apache Kafka is an open source event streaming platform. It is often used to complement or even replace existing middleware to integrate applications and build microservice architectures. Apache Kafka is already used in various projects in almost every bigger company today. Understood, battled-tested, highly scalable, reliable, real-time. Blockchain is a different story. This technology is a lot in the news, especially related to cryptocurrencies like Bitcoin. But what is the added value for software architectures? Is blockchain just hype and adds complexity? Or will it be used by everybody in the future, like a web browser or mobile app today? And how is it related to an integration architecture and event streaming platform? This session explores use cases for blockchains and discusses different alternatives such as Hyperledger, Ethereum and a Kafka-native tamper-proof blockchain implementation. Different architectures are discussed to understand when blockchain really adds value and how it can be combined with the Apache Kafka ecosystem to integrate blockchain with the rest of the enterprise architecture to build a highly scalable and reliable event streaming infrastructure. Speakers: Kai Waehner, Technology Evangelist, Confluent Stephen Reed, CTO, Co-Founder, AiB

1. Building a Secure, Tamper-Proof &
Scalable Blockchain on Top of Apache
Kafka
Kai Waehner
Technology Evangelist
Confluent
kai.waehner@confluent.io
LinkedIn
@KaiWaehner
confluent.io
kai-waehner.de
Stephen Reed
CEO, Co-founder
AiB
stephen@ai-blockchain.com
LinkedIn
ai-blockchain.com

2. Building a Secure, Tamper-Proof & Scalable Blockchain on Top of Apache Kafka
1. Use Cases for Secure and Tamper-Proof Data Processing
2. Distributed Ledger and Blockchain Technologies
3. Event Streaming with Apache Kafka
4. Relation between Kafka and Blockchain
5. A Tamper-Proof Blockchain implemented with Kafka
6. Live Demo
Agenda

3. Building a Secure, Tamper-Proof & Scalable Blockchain on Top of Apache Kafka
PRODUCTS
Built FIRST DocuSign Blockchain
product.
Allows for the writing of contracts on the
blockchain with proof of trust
Software agents keep the blockchain
tamper proof and immutable
Current POC’s across eCommerce,
Healthcare and Supply Chain
SOLUTIONS
• Artificial Intelligence, Blockchain, Ecommerce,
Healthcare, Media, Payments and Supply Chain
• Securely store commercial and non-commercial
transactions
• Energy efficient and ESG
GROWTH OPPORTUNITIES
How do we expand globally with new customers
using AiB products?
AiB offers services for scalability to meet all of the
demands of global businesses offering architecture
assessment diagrams to clients.
TECHNOLOGY
We employ one of a kind Artificial Agents on
the Blockchain to give world class security
and to ensure all data is immutable.
PLATFORM
Its most visible success is in the unique
approach to financial transactions through its
implementation of its Artificial Intelligent
Blockchain Cloud Payment Platform.
Enterprise
Grade
Platform
COMPANY
Since its founding in 2014,
Ai-Blockchain has built a
combination of innovative IP
and blockchain products
that have been recognized in
the marketplace and USPTO.
Potential
Customer
Use Cases
and
Problems
Ai-Blockchain
Products
Ai-Blockchain
Solutions and
advantages
Industry
Expansion
Ai-Blockchain – Patented Private-Permissioned Blockchain
for Smart Contracts

4. Building a Secure, Tamper-Proof & Scalable Blockchain on Top of Apache Kafka
4
CUSTOMER CLIENT EXPERIENCE WITH AiB:
● Immediate go-to-market product execution
using blockchain
● Artificial Intelligence and Machine Learning
recommendation systems
Ai-Blockchain – Patented Private-Permissioned Blockchain
for Smart Contracts

5. Building a Secure, Tamper-Proof & Scalable Blockchain on Top of Apache Kafka
Ai-Blockchain – Co-Founders
6

6. Building a Secure, Tamper-Proof & Scalable Blockchain on Top of Apache Kafka
Blockchain for Supply Chain Management
(Manufacturing, Logistics, …)
7
https://coletiv.com/blog/blockchain-supply-chain-management-system/
https://www.disruptordaily.com/blockchain-use-cases-supply-chain-management/

7. Building a Secure, Tamper-Proof & Scalable Blockchain on Top of Apache Kafka
Blockchain in Healthcare
8
• Use Cases
• Management of patient data,
especially access control
• RVU Agreements with Ai-Blockchain
• Management and use of academic
research data
• Logistics / Supply chain for drugs,
medical technology, and other
products
• Often no need for Blockchain /
DLT
• Only for Identity Management
• Processing and (big) data storage
outside blockchain
https://digiforest.io/en/blog/blockchain-examples-in-healthcare

8. Building a Secure, Tamper-Proof & Scalable Blockchain on Top of Apache Kafka
Blockchain for
Financial Services
9
https://www.accenture.com/t20170120T074124Z__w__/us-en/_acnmedia/Accenture/Conversion-Assets/DotCom/Documents/Global/PDF/Consulting/Accenture-Banking-on-Blockchain.pdf%23zoom=50

9. Building a Secure, Tamper-Proof & Scalable Blockchain on Top of Apache Kafka
1. Use Cases for Secure and Tamper-Proof Data Processing
2. Distributed Ledger and Blockchain Technologies
3. Event Streaming with Apache Kafka
4. Relation between Kafka and Blockchain
5. A Tamper-Proof Blockchain implemented with Kafka by
AiB
6. Live Demo
Agenda

10. Building a Secure, Tamper-Proof & Scalable Blockchain on Top of Apache Kafka
Bitcoin – The Tip of the Iceberg!
11
‘Bitcoin’ compared to
‘blockchain’, ‘distributed
ledger’, etc.

11. Building a Secure, Tamper-Proof & Scalable Blockchain on Top of Apache Kafka
Blockchain vs. Distributed Ledger Technology
(DLT)
12
https://www.da.ventures/post/gaining-clarity-on-key-terminology-bitcoin-versus-blockchain-versus-distributed-ledger-technology

12. Building a Secure, Tamper-Proof & Scalable Blockchain on Top of Apache Kafka
Blockchain
14
https://www.investopedia.com/terms/b/blockchain.asp

13. Building a Secure, Tamper-Proof & Scalable Blockchain on Top of Apache Kafka
Blockchain Concepts
• Permissioned or permissionless (or consortium)
• Consensus algorithm (Proof of work, Proof of Stake, or Voting systems)
• Blockchain: Global Consensus across all nodes
• DLT: Consensus without having to validate across the entire Blockchain
• Blockchain
• Growing list of records, called blocks, linked using cryptography
• Each block contains a cryptographic hash of the previous block, a timestamp, and transaction data
16

14. Building a Secure, Tamper-Proof & Scalable Blockchain on Top of Apache Kafka
Challenges and Concerns of Blockchains
• Technical and
organizational Complexity
• Transaction speed
• Energy consumption
• Security
• Data tenancy
• Lifecycle costs
• Ease of integration

15. Building a Secure, Tamper-Proof & Scalable Blockchain on Top of Apache Kafka
Purpose-built solutions
• Ripple: Payment settling, currency exchange and remittance system
• Libra: Global payment system and financial infrastructure
• IOTA: IoT transactions
Non-Blockchain DLTs
to solve the challenges to provide better performance and scalability
18
Open and flexible solution for many use
cases
• Apache Kafka: Open, scalable, real time
event streaming
Technology Partners
• Ai-Blockchain: “AiB” USA based

16. Building a Secure, Tamper-Proof & Scalable Blockchain on Top of Apache Kafka
1. Use Cases for Secure and Tamper-Proof Data Processing
2. Distributed Ledger and Blockchain Technologies
3. Event Streaming with Apache Kafka
4. Relation between Kafka and Blockchain
5. A Tamper-Proof Blockchain implemented with Kafka
6. Live Demo
Agenda

17. Building a Secure, Tamper-Proof & Scalable Blockchain on Top of Apache Kafka
Event Streaming in the Finance Industry
Check past Kafka Summit videos for details about the use cases:
https://kafka-summit.org/past-events/

18. Building a Secure, Tamper-Proof & Scalable Blockchain on Top of Apache Kafka
Tamper-Proof Event Streaming for
(some of the) Use Cases in the Finance Industry
Real Time Processing Digital Transformation Strategic Goals
Short-Sale Risk
Calculation / Trade
Approval
Mainframe Offloading
and Replacement
Instant Payment
Next-Best
Offer
Robot Process Automation
(e.g. Know Your Customer,
KYC)
Customer Service
(e.g. Chat Bots)
IT
ModernizationRegulatory
Reporting
Fraud Detection
Anomaly Detection
Across Assets and Locations
Derivatives
Pricing Compliance
Trading
Post-Processin
g
Strategic
Planning and
Simulations

19. Building a Secure, Tamper-Proof & Scalable Blockchain on Top of Apache Kafka
STREAM
PROCESSING
Create and store
materialized views
Filter
Analyze in-flight
Time
C CC
Event Streaming

20. Building a Secure, Tamper-Proof & Scalable Blockchain on Top of Apache Kafka
Event Streaming Platform –
The Immutable Commit Log
Time
P
C1 C2
C3

21. Building a Secure, Tamper-Proof & Scalable Blockchain on Top of Apache Kafka
Event Streaming Platform –
A Distributed System for 24/7 and Zero Data Loss
Broker 1
Topic1
partition1
Broker 2 Broker 3 Broker 4
Topic1
partition1
Topic1
partition1
Leader Follower
Topic1
partition2
Topic1
partition2
Topic1
partition2
Topic1
partition3
Topic1
partition4
Topic1
partition3
Topic1
partition3
Topic1
partition4
Topic1
partition4

22. Building a Secure, Tamper-Proof & Scalable Blockchain on Top of Apache Kafka
An Event Streaming Platform
is the Underpinning of an Event-driven Architecture
Microservices
Mainframes
SaaS apps
Mobile
Customer 360
Real-time fraud
detection
Data warehouse
Producers
Consumers
Database
change
Microservices
events
SaaS
data
Customer
experiences
Streams of real time events
Stream processing apps
Connectors
Connectors
Stream processing apps

23. Building a Secure, Tamper-Proof & Scalable Blockchain on Top of Apache Kafka
Kafka Connect
with Ethereum Connector
Kafka Cluster
Blockchain
Integration
Domain-Driven Design and Decoupled Applications
Legacy
Integration
Custom
Application
ETL Tool
with Mainframe Connector
Java / C++ /
Go / Python / KSQL
Schema
Registry
Event Streaming Platform
Blockchain
Domain
Payment Domain Fraud Domain
Audit Logs,
RBAC, etc.

24. Building a Secure, Tamper-Proof & Scalable Blockchain on Top of Apache Kafka
Reasons for Event Streaming with Apache Kafka
Real Time
Scalable
High Throughput
Cost Reduction
24/7 – Zero downtime, zero data loss
Decoupling – Storage, Domain-driven Design
Data (re-)processing and stateful client applications
Integration – Connectivity to IoT, legacy, big data, everything
Hybrid Architecture – On Premises, multi cloud, edge computing
Fully managed cloud
No vendor locking
29

25. Building a Secure, Tamper-Proof & Scalable Blockchain on Top of Apache Kafka
Kafka Clusters
can stretch over
Regions
Zero Downtime + Zero Data loss
(RPO=0 and RTO=0)
e.g. Stretched over US East + Mid +
West
Automate Disaster Recovery
Sync or Async Replication per Topic
Offset Preserving
Automated Client Failover without
Custom Code
Multi-Region Cluster
(Only available in Confluent Platform)
31

26. Building a Secure, Tamper-Proof & Scalable Blockchain on Top of Apache Kafka
Example of a Multi-Region Cluster in a Bank
Large FinServ Customer
Payment
Log
Payment
Log
Location Location
synchronous
asynchronous
● ‘Payment’ transactions enter
from us-east and us-west with
fully synchronous replication
● ‘Log’ and ‘Location’ information
in the same cluster use async -
optimized for latency
● Automated disaster recovery
(zero downtime, zero data loss)
Result: Clearing time from ‘deposit’ to
‘available’ goes from 5 days to 5
seconds (including security checks)
(Only available in Confluent Platform)

27. Building a Secure, Tamper-Proof & Scalable Blockchain on Top of Apache Kafka
1. Use Cases for Secure and Tamper-Proof Data Processing
2. Distributed Ledger and Blockchain Technologies
3. Event Streaming with Apache Kafka
4. Relation between Kafka and Blockchain
5. A Tamper-Proof Blockchain implemented with Kafka
6. Live Demo
Agenda

28. Building a Secure, Tamper-Proof & Scalable Blockchain on Top of Apache Kafka
Real Time
High Throughput
Decentralized database
Distributed log of records
Immutable log
Replication
High availability
Decoupling of applications /
clients
Role-based access control to
data
Tamper-Proof
Encrypted payloads
is not a Blockchain!
X

29. Building a Secure, Tamper-Proof & Scalable Blockchain on Top of Apache Kafka
Kafka AND Blockchain
37
Kafka
Real Time
Instant Payment
App
(Java, C++,
Python, etc.)
Batch
Analytics Platform
(Spark, Splunk,
etc.)
Blockchain
(Bitcoin)
Blockchain
(Ethereum)
Kafka-native
Blockchain
Kafka AS Blockchain
Real Time
Instant Payment
App
(Java, C++,
Python, etc.)
Batch
Analytics Platform
(Spark, Splunk,
etc.)

30. Building a Secure, Tamper-Proof & Scalable Blockchain on Top of Apache Kafka
1. Use Cases for Secure and Tamper-Proof Data Processing
2. Distributed Ledger and Blockchain Technologies
3. Event Streaming with Apache Kafka
4. Relation between Kafka and Blockchain
5. A Tamper-Proof Blockchain implemented with Kafka
6. Live Demo
Agenda

31. Building a Secure, Tamper-Proof & Scalable Blockchain on Top of Apache Kafka

32. Building a Secure, Tamper-Proof & Scalable Blockchain on Top of Apache Kafka
AiB’s BRICK Messaging Platform
For security, privacy and scalability, the AiB’s BRICK
platform is a platform agnostic layer that is built for
enterprise and micro payments in the cloud.
Third-Party Audited
AiB’s technology stack has been
security audited and stress tested by
third-party global cloud providers.
Full-Stack Capability
Deploying nodes and writing smart contracts
up to front end development and deployment.
Thought Leaders
Consistently asked to
speak at international
events to share our
Blockchain expertise.
Blockchain Experts
Established in 2014
Ai Blockchain’s team of over
100 Blockchain developers &
experts have been building
solutions for over six years
and are ready today for any
and all market
opportunities.
Ai-Blockchain – Proprietary Blockchain Technology Stack

33. Building a Secure, Tamper-Proof & Scalable Blockchain on Top of Apache Kafka
41
KafkaBlockchain Architecture
Producers reuse the KafkaBlockchain
library and Tamper-Evident Records.
Consumers reuse the KafkaBlockchain library and Tamper-Evident Records.

34. Building a Secure, Tamper-Proof & Scalable Blockchain on Top of Apache Kafka
KafkaBlockchain – Producer, Consumer & Verifier
Open Source Apache
Licensed on GitHub

35. Building a Secure, Tamper-Proof & Scalable Blockchain on Top of Apache Kafka
43

36. Building a Secure, Tamper-Proof & Scalable Blockchain on Top of Apache Kafka
1. Use Cases for Secure and Tamper-Proof Data Processing
2. Distributed Ledger and Blockchain Technologies
3. Event Streaming with Apache Kafka
4. Relation between Kafka and Blockchain
5. A Tamper-Proof Blockchain implemented with Kafka
6. Live Demo
Agenda

37. Building a Secure, Tamper-Proof & Scalable Blockchain on Top of Apache Kafka
Demonstration: run-kafka-blockchain.demo.sh
• Demonstrate putting messages into a Kafka blockchain with one partition, one producer
and one consumer.
• Describe the single blockchain partition
• Demonstrate verifying messages with one consumer
Demonstration: run-kafka-blockchain-multiple-partition.demo.sh
• Demonstrate putting messages into a Kafka blockchain with five partitions, one producer
and one consumer.
• Describe the five blockchain partitions
• Demonstrate verifying messages with five cooperating consumers - one for each
partition.
45
KafkaBlockchain – Live Demo

38. Building a Secure, Tamper-Proof & Scalable Blockchain on Top of Apache Kafka
When to use Kafka vs. a “real blockchain
framework”?
46
Use Apache Kafka with
AIBlockchain for
• Enterprise infrastructure
• Open, scalable, real-time
requirements
• Flexible architectures for
many use cases
Use Hyperledger, Ethereum,
et al. for
• Deployment over various
independent organizations
• Participants verify the
distributed ledger contents
themselves.
• Specific use cases
• Server-side managed and
controlled by multiple
organizations
• Scenarios where the
business value overturns the
added complexity and
project risk
Use Kafka and
Blockchain together to
combine the benefits of
both (only where this
makes sense!)

39. Building a Secure, Tamper-Proof & Scalable Blockchain on Top of Apache Kafka
What about long-term storage in Kafka?
48

40. Building a Secure, Tamper-Proof & Scalable Blockchain on Top of Apache Kafka
Today, Kafka works well
for recent events, short
horizon storage, and
manual data balancing
Kafka’s present-day design offers
extraordinarily low messaging latency by
storing topic data on fast disks that are
collocated with brokers. This is usually good.
But sometimes, you need to store a huge
amount of data for a long time.
Blockchain is such a use case!
Kafka
Processing
App
Storage
Transactions, auth,
quota
enforcement,
compaction, ...

41. Building a Secure, Tamper-Proof & Scalable Blockchain on Top of Apache Kafka
Confluent Tiered Storage for Kafka
Object Store
Processing Storage
Transactions,
auth, quota
enforcement,
compaction, ...
Local
Remote
Kafka
Apps
Store Forever – AiB’s Tamper-Proof
Blockchain with KafkaBlockchain
Older data is offloaded to inexpensive object storage,
permitting it to be consumed at any time. Using AiB storage
can be made tamper-proof and immutable
Save $$$
Storage limitations, like capacity and duration, are effectively
uncapped.
Instantaneously scale up and down
Your Kafka clusters will be able to automatically self-balance
load and hence elastically scale
(Only available in Confluent Platform)

42. Questions? Feedback?
Let’s connect!
Kai Waehner
Technology Evangelist
kai.waehner@confluent.io
LinkedIn
@KaiWaehner
confluent.io
kai-waehner.de
Stephen Reed
CEO, Co-founder
AiB
stephen@ai-blockchain.com
LinkedIn
ai-blockchain.com