A microservices architecture makes it easier to develop and scale rapidly with a growing data load. However, synchronous API calls tightly couples microservices and introduces problems such as fault intolerance, poor performance, and decelerating time-to-marker for new features. Event driven architecture is an effective design pattern to solve these problems. However, implementing this pattern can be challenging too, because each component must manage the state of each event separately. In this talk, learn how to decouple the communication between disparate microservices using Apache Kafka and manage the state of the events separately using Apache Flink Stateful functions. You will learn how this architecture simplifies the pub / sub pattern by handling the event routing differently without requiring creation of too many topics within the Apache Kafka cluster.

1. Build event driven microservices with
Apache Kafka and Apache Flink
Ali Alemi (Sr. Streaming SA)
Amazon Web Services

2. © 2023, Amazon Web Services, Inc. or its affiliates. © 2023, Amazon Web Services, Inc. or its affiliates.
Agenda
Why Microservices?
Why Event Driven Architecture?
Why Data Streaming?
Why is State important?
Why Apache Flink?
Call to action

3. © 2023, Amazon Web Services, Inc. or its affiliates. © 2023, Amazon Web Services, Inc. or its affiliates.
Why microservices?
Microservices
Agility
Easier to scale
Accelerating time to
market for new features
Developer choice
Faster to develop
Enabling innovation

4. © 2023, Amazon Web Services, Inc. or its affiliates. © 2023, Amazon Web Services, Inc. or its affiliates.
Are you Well Architected?
Anticipate Failure?
Can recover by itself?
Performant?
Secure?
Cost effective?
Sustainable?
Network
Products Recommendation
DB
Orders
DB
User profile
DB
Marketing
DB
Inventory
DB
DB

5. © 2023, Amazon Web Services, Inc. or its affiliates. © 2023, Amazon Web Services, Inc. or its affiliates.
Loosely couple your microservices with
Event Driven Architecture
User
registration
User
verification
User profile Connections Recommendation
Apache Kafka

6. © 2023, Amazon Web Services, Inc. or its affiliates. © 2023, Amazon Web Services, Inc. or its affiliates.
Oh! it is more difficult than we thought!

7. © 2023, Amazon Web Services, Inc. or its affiliates. © 2023, Amazon Web Services, Inc. or its affiliates.
Change the synchronous way of writing code
1. profile = userService.getProfile(‘user_id’);
2. profile.views += 1;
3. userService.saveProfile(profile);
4. Print (“Yaaay!!”)

8. © 2023, Amazon Web Services, Inc. or its affiliates. © 2023, Amazon Web Services, Inc. or its affiliates.
Because… “The World is A-synchronous”
Bank Corporate

9. © 2023, Amazon Web Services, Inc. or its affiliates. © 2023, Amazon Web Services, Inc. or its affiliates.
Build Asynchronous Event Driven Microservices
Web service
Central State Manager
Web service Web service
Web service
Web service
Web service
Choreography

10. © 2023, Amazon Web Services, Inc. or its affiliates. © 2023, Amazon Web Services, Inc. or its affiliates.
Application /
Business logic
Application /
Business logic
Application /
Business logic
Application /
Business logic
I want my microservices “Stateless”
Input events
stream
Output events
stream
Application/
Business logic
Database
Traditional
database-backed
microservice
JDBC/REST
Input events
stream
Output events
stream
State
Manager
Service
Application/
Business logic
HTTP/gRPC
Event-driven
function
state
state

11. © 2023, Amazon Web Services, Inc. or its affiliates. © 2023, Amazon Web Services, Inc. or its affiliates.
However, State is not your only problem
Number of topics = Number of Microservices X 2 (request + response)
Number of Partitions = Max. Parallelism of consumers X Number of topics

12. © 2023, Amazon Web Services, Inc. or its affiliates. © 2023, Amazon Web Services, Inc. or its affiliates.
Why Apache Flink? is a framework and
distributed processing engine for stateful
computations over data stream. Flink has been
designed to perform computations at in-memory
speed and at any scale.
Composable event processing
State management
Performance and scale

13. © 2023, Amazon Web Services, Inc. or its affiliates. © 2023, Amazon Web Services, Inc. or its affiliates.
Apache Flink Stateful Functions
P O L Y G L O T E V E N T - D R I V E N F U N C T I O N S F O R D I S T R I B U T E D S T A T E F U L A P P L I C A T I O N S
FaaS
HTTP/gRPC
K8s deployment
(containerized functions)
Event streams ingress Event streams egress
Apache Flink StateFun cluster

14. © 2023, Amazon Web Services, Inc. or its affiliates. © 2023, Amazon Web Services, Inc. or its affiliates.
Apache Flink Stateful Functions
event ingress
Apache Flink
StateFun
event egress
fx
fx
fx
fx
fx
fx
state
state
state
state
state
state
State snapshots

15. © 2023, Amazon Web Services, Inc. or its affiliates. © 2023, Amazon Web Services, Inc. or its affiliates.
Logical co-location, physical separation
Each function has a unique address which is used as a
reference
Function instances are virtual
Functions can run remotely, with the event and state
provided as part of the request
Event sourcing, state access/updates and function
invocations are managed tightly together
If an Function instance has no state and is not actively
running, then it occupies no CPU, no threads, and no
memory.
fx Address
Function Type
+ Namespace
+ Name
ID
fx
instance
fx
instance
fx
instance
fx
instance
fx
instance
fx
instance
fx
instance
fx
instance
fx
instance

16. © 2023, Amazon Web Services, Inc. or its affiliates. © 2023, Amazon Web Services, Inc. or its affiliates.
Here's how it all comes together
Stateful Functions
cluster
Application/Business logic
HTTP/gRPC
Amazon MSK Amazon MSK
Amazon Kinesis Data Analytics
for Apache Flink
Input streams
Output
streams
State snapshots on
Amazon S3
Amazon API Gateway
state
state
Remote functions Remote functions
Amazon Kinesis Data
Analytics for Apache Flink
Realtime Analytics
Amazon Kinesis
Data Streams
Amazon MSK
JDBC End Points
Amazon Kinesis
Data Firehose
Amazon S3
Amazon
Elasticsearch

17. © 2023, Amazon Web Services, Inc. or its affiliates. © 2023, Amazon Web Services, Inc. or its affiliates.
Event-driven applications with Data Streaming
ü Clearly defined ingress and egress
ü Simplified composability
ü Decoupling of application logic and state management
ü Locally embedded state – no databases required
ü Idempotency and Exactly once semantics

18. © 2023, Amazon Web Services, Inc. or its affiliates. © 2023, Amazon Web Services, Inc. or its affiliates.
Call to action
• https://aws.amazon.com/architecture/well-architected
• https://flink.apache.org/
• https://nightlies.apache.org/flink/flink-statefun-docs-stable/
• https://aws.amazon.com/msk/features/msk-serverless/
• https://aws.amazon.com/kinesis/data-analytics/
• https://aws.amazon.com/lambda/
• https://aws.amazon.com/blogs/aws/introducing-amazon-kinesis-
data-analytics-studio-quickly-interact-with-streaming-data-using-sql-
python-or-scala/
• https://aws-blogs-prod.amazon.com/big-data/how-to-choose-the-
right-amazon-msk-cluster-type-for-you/

19. © 2023, Amazon Web Services, Inc. or its affiliates. © 2023, Amazon Web Services, Inc. or its affiliates.
Thank you!