apidays LIVE India 2021 - Connecting 1.3 billion digital innovators May 20, 2021 Asynchronous and Broadcasting APIs using Kafka Rohit Saxena, Software Development Consultant at Guardian Life

1. Rohit Saxena
Software Development
Consultant
Guardian Life
Connecting 1.3 billion digital
innovators
India | May 20, 2021

2. Speaker Profile
• 11 Years exp Software Consultant
• Enterprise Solution Architect
• Ecommerce, Finance, Travel & Insurance domain Knowledge
• Backend engineering enthusiastic
• Technical author and Speaker
Asynchronous and Broadcasting APIs using Kafka

3. Agenda
1. Where do we come from?
2. What about Microservices?
3. Why asynchronous ?
4. Can we do evenbetter?
5. How does Apache Kafka help?
6. What about Streaming Data Sources?
7. Summary / Conclusion
8. Questions ?

4. Where do we come
from?

5. Shop Rich UI
Shop Backend Application
Layered Architecture” Approach
Search Facade
Customer DAO
Order DAO
Order Facade
UI Logic
Product DAO
Data
Business
Shop UI
GUI
Customer Fat Client App
Customer BO
Data
Customer UI
GUI
Data
Storage
Shared
Database
sync request/response

6. Shop UI App
Business
Activity Service
SOA Approach
• Contract-
first Web
Services
• Technical
layers
offer their
own
interfaces
• Reuse on
each level
• Lower layer
often
wraps
legacy code
Search BAS
rder
O
DAO
Order BAS
Product DAO
UI Logic
Shop UI
GUI
Business Entity
Service
Shop Web App
UI Logic
Shop UI
GUI
Data
Storage
Customer
Database
Customer BES
Payment BES
ProCduusctot
mBEeSr DAO
Order BES
Custer BAS
Order and
Product DB
SOAP
SOAP
SOAP
SOAP
SOAP
SOAP
SOAP

7. What about Microservices?

8. Customer Microservice
Microservice Approach
• Tightly Scoped behind interfaces
• Highly decoupled
• Independently deployable
• Bounded Context/Aggregate
(DDD)
• Responsible for their data (does
not mean they need their own
DB!)
• Smart Endpoints and Dump
Pipes
• just SOA done right ? What
about capabilities the “smart
pipes” provided?
{ }
Customer
Customer API Customer Logic
Order Microservice
{ }
Order API
Order
Order Logic
Product Microservice
Product API
Product
Product Logic
Stock Microservice
Stock API
Stock
Stock Logic
Shop Web App
UI Logic
Shop UI
GUI
REST
REST
RES
{T
}
RES
{T
}

9. Synchronous Request-Response lead to tight, point- to-point
couplings
problem in lower end of chain have a ripple
effect on other service
• crash of service
• overloaded service / slow response time
• change of interface
Service 2
Service 1
{ }
API
Logic
}
API Logic
State
REST{
State
Service 3
{ }
API Logic
State
Service 4
{ }
API Logic
State
Service 5
API Logic
State
Service 7
API Logic
State
Service 6
{ }
API Logic
State
REST
REST
REST
REST { } REST REST { }

10. Why Asynchronous?

11. Events
Distribute to all
handlers strong
ordering req’s
No results
Queries
Route with load
balancing Sometimes
scatter-gather Provide
result
Three mechanisms through which services can
interact
Commands
Route to single handler
Use consistent
hashing Provide
Result
Adapted from Axon IQ

14. Types of Async
APIs
● The task is big, and may take a while to
complete?
○
○ Uploading a video for
encoding Analyzing a big
data set
● The service may not be able to handle the flow of events /
response not needed
○ Analytics events
● You want the service to contact you
(push)
○
○ Push notifications in
app Reacting to data
changes
Task
Management
Events
Flow
Push
API

15. Types of Async
APIs
Task
Management
Events
Flow
Push
API
Queu
e
Strea
m
Message
Broker

16. Async API can be implemented using any of
below
Queu
e
Strea
m

17. Can we do even(t)
better?

18. Stock Microservice
Event-Driven (Async) Microservice
Event
Hub
• Pub / Sub messaging
• Message transparent to topic
• Message coupling
• Domain Events from DDD
• This is event-driven interaction but
not
event
sourcing!
Customer Microservice
{ }
Customer
Customer API Customer Logic
Order Microservice
{ }
Order API
Order
Order Logic
Product Microservice
{ }
Product
Product API
Product Logic
{ }
Stock API
Stock
Stock Logic
Shop Web App
Shop UI UI Logic
REST
REST
REST
REST
API
Gateway
Event
Hub
sync request/response
async, event pub/sub
async request/response
Customer
Mat View

19. How does Apache Kafka
help?

20. Apache Kafka – highly scalable message broker
Consumer 1 Consume 2r
Broker 1
Kafka Cluster
Broker 2 Broker 3
Zookeeper
Ensemble
ZK 1 ZK 3 ZK 2
Schema
Registry
Service 1
Management
Control Center
Kafka Manager
KAdmin
Producer 1 Producer 2
kafkacat
Data Retention:
• Never
• Time (TTL) or Size-based
• Log-Compacted based
PProrodduu
cceer3r3
CConosnus
mumere3r
Event Hub

21. What about Streaming Data
Sources?

22. Streaming & (Big) Data Analytics Architecture
Event
Stream
Hadoop Clusterd
Hadoop
Cluster Big Data
Cluster
Parallel
Processing
Storage
Storage
Raw
Refined
Results
Microservice Cluster
Microservice State
{ }
API
Stream Processing Cluster
Stream
Processor
State
{ }
API
Event
Stream
Event
Stream
SQL
Search
BI Tools
Enterprise Data
Warehouse
Search / Explore
Online & Mobile
Apps
Event
Hub
Service
Location
Social
Click
stream
Sensor
Data
Mobile
Apps
Weather
Data
Billing &
Ordering
CRM /
Profile
Marketing
Campaigns
File Import / SQL Import

23. CONCLUSIO
N
• Not all communication need to be
synchronous => distinguish into commands,
events, queries
• Events should use a schema with support for
backward and forward compatibility
• Kafka handles event streaming very well
• brings many more interesting
features beyond just “message
passing”, i.e. Log compaction
• Kafka broker is not a full-fledged Event Store
• For Event Sourcing additional capabilities
are needed (Kafka Streams, Axon, …)
• See also: Kafka as an Event Store: is it

24. The opportunity: The shift to streams & digital transformation
Streaming ingestion and
analytics will become a
must-have for digital
winners. - Forrester |
Nov.
3
By 2021, 85% of
organizations adopted
data streaming to enable
real-time analytics.

25. CONNECT WITH ME FOR MORE…
rohitsxn2@gmail.com
https://www.linkedin.com/in/rohitsaxena3
1/

26. 20
A n y Q
u e s t I
o n s ?

27. THANK
YOU