On July 18th, we got together at Campus Madrid to discover all about Kafka. Discover with Óscar Gómez, Software Architect at Stratio, how Kafka can help us on our event-driven Microservices Architectures. Find out more: http://www.stratio.com/blog/events/all-about-kafka-origins-ecosystem-and-future-directions/

1. Using Kafka
on Event-Driven
Microservices Architectures.

2. Who am I?
Oscar Gómez Soriano
What I Do?
Software Architect / Developer @Stratio BD
How can you contact me?
ogomez@stratio.com
Oscar Gómez Soriano
@oskarflesh

3. 1. Event Driven Architectures
2. Real Use Case: Appointment Creation System
3. Architectural Solution
4. Improving the Architecture
AGENDA

4. EVENT-DRIVEN
ARCHITECTURES1

5. “An enterprise application is a
system that reacts to events
from the outside world”.
Martin Fowler

6. Event-driven Architectural Patterns
We can find 4 great types of event-driven architectural
patterns depending on the way the events are used:
● Event Notification - This happens when a system
sends event messages to notify other systems of a
change in its domain. A key element of event
notification is that the source system doesn't really
care much about the response.
● Event-Carried State Transfer - The event sended
contains all the info needed to update the
“recipient system” in such a way that it doesn't
need to contact the source system in order to do
further work.

7. Event-driven Architectural Patterns
● Event-Sourcing - The core idea
of event sourcing is that
whenever we make a change to
the state of a system, we record
that state change as an event,
and we can confidently rebuild
the system state by reprocessing
the events at any time in the
future.
The event store becomes the principal source of truth, and the system state is
purely derived from it. For programmers, the best example of this is a version-
control system. The log of all the commits is the event store and the working copy
of the source tree is the system state.

8. Event-driven Architectural Patterns
● CQRS - Command Query
Responsibility Segregation
(CQRS) is the notion of
having separate data
structures for reading and
writing information. Strictly
CQRS isn't really about
events, since you can use
CQRS without any events
present in your design. But
commonly people do
combine CQRS with the
earlier patterns here, hence
their presence at the
summit.

9. Implementing Event-driven Architectures
“A event-messaging-based application typically uses a message broker, which is an infrastructure
service through which the services communicates. However, a broker-based architecture is not the
only messaging architecture. You can also use a brokerless-based messaging architecture in which
the services communicate with one another directly”. Chris Richardson - Microservices Patterns

10. Broker-Based Messaging Benefits & Drawbacks
There are many advantages to using broker-based messaging:
● Loose Coupling - a client makes a request simply sending a message to the appropriate channel.
The client is completely unaware of the service instances. It does not need to use a discovery
mechanism to determine the location of a service instance.
● Message Buffering - the message broker buffers messages until they can be processed. This
means, for example, that an online store can accept orders from customers, even when the order
fulfillment system is slow or unavailable. The Order messages will simply queue up.
● Explicit Inter-Process Communication - Messaging makes differences between remote and local
services very explicit so developers are not lulled into a false sense of security.

11. Broker-Based Messaging Benefits & Drawbacks
There are some, however, downsides to using messaging:
● Potential performance bottleneck - there is a risk that the message broker could be a
performance bottleneck. Fortunately, many modern message brokers are designed to be highly
scalable.
● Potential single-point of failure - its essential that the message broker is highly available
otherwise system reliability will be impacted. Fortunately, most modern brokers have are designed
to be highly available.
● Additional operational complexity - the messaging system is yet another system component that
must be installed, configured and operated.

12. REAL USE CASE:
APPOINTMENT
CREATION SYSTEM
2

13. The Challenge
The basic idea is to construct an Appointment Creation System.
On matter of the performance & functional requirements we got 3 different
scenarios depends on the user who interacts with the system:
● Final user
● Admin User
● BI User
● AI User

14. The Challenge: Final User Requirements
1. The user can consult calendars to be able to see the available appointments
& create it.
2. The user can check their created appointments and reschedule/cancel it.
3. By default, a same hole can not be reserved by two users at the same time,
an overload algorithm determines if this restriction is disabled.
4. An AI model determines the probability that the patient attends the
appointment and an algorithm determines the overload capacity.

15. The Challenge: Admin User Requirements
1. A admin user can consult existing appointments. Using filters they can adjust
their search.
2. The Admin user can create appointments for patients.

16. The Challenge: BI User Requirements
1. The BI users can calculate business metrics from visits events in near real
time.
2. The BI user can visualize the values ​​of these metrics through dashboards.

17. The Challenge: AI User Requirements
1. A data scientist has an advanced analytical environment in which they can
develop models from the information obtained through the system.
2. The data scientist can do data discovery using Stratio Discovery

18. The Challenge: Performance Requirements
1. Response time for more complex operations less than 250 ms.
1. Transactional process supporting 10,000 online users interacting
appointment API (creation/cancel/Rescheduling) and consulting
calendars in a two minute window.
1. Replication of the data to the query model must respond to the principle
of eventual consistency with a delay of less than 1 minute, regardless of
the size of the HDFS block.

19. ARCHITECTURAL
SOLUTION3

20. Architectural Solution: The Big Picture

21. Architectural Solution: Microservice Architecture

22. Architectural Solution On Detail : Appointment Service

23. Architectural Solution On Detail : Calendar Search

24. Architectural Solution On Detail : KPIs (BI Users)

25. Performance Test Results: Test Scenario
Infrastructure:
● Stratio DataCentric Platform deployed on Azure Cloud
● 500,000 pre-reserved appointments.
● > 84,000 agendas registered in the system.
● Production Masters Data
● Test Distribution by functionality:
○ 37% Calendar Search
○ 16,67% Appointment Inserts
○ 15,15% Appointment Reschedule/Cancel
○ 30,30% Created Appointment Search
● Microservices Resource Configuration:
○ Calendar: 4 cores y 4GB, Autoscaler Config from 2 to 5 instances.
○ Appointment: 1 Core, 1GB, Autoscaler Config from 2 to 5 instances.
○ Masters: 1 Core, 1GB Autoscaler Config from 2 to 3 instances.

26. Performance Test Results
Performance Test Results:
2 Hour Performance Test
88 tps (10483 2 minute window)
Microservice
Request
Distribution
Request/
s
Media
Calendar
Search
37,88 % 33 20 ms
Appointment
Insert
16,67 % 15 64 ms
Reschedule /
Cancel
15,15 % 14 74 ms
Appointment
Search
30,30 % 26 26 ms
2 Hour Performance Test
178 tps (10483 1 minute window)
Microservice
Request
Distribution
Request/s Media
Calendar
Search
37,88 % 67 19 ms
Appointment
Insert
16,67 % 30 106 ms
Reschedule /
Cancel
15,15 % 28 91 ms
Appointment
Search
30,30 % 53 46 ms

27. IMPROVING THE
ARCHITECTURE4

28. Knowing the Gaps
With this numbers in mind we identify two possible bottlenecks on this Architecture:
● Created Appointment Search: As the search and insertion go against the same service and
database model both operations are penalized. In addition, the model applied to the insertion
is not optimal for the search since it forces the front to make auxiliary requests to obtain the
data it needs.
● Calendar Service Responsibilities: Another gap we found is that Calendar service has to
receive & process the kafka events (overbooking calculations & update the cache model) and at
the same time attend to the get calendar requests so in times of high demand we will have a
lot of kafka events to process which could slow down the search speed of the calendars.
● Another consideration to take is that despite having demonstrated a high performance,
Memcached is not a solution that can scale well having a series of limitations such as not
having a persistence mechanism of its own, having a high availability solution or not being a
distributed system
Once we know our service needs in matter of performance and scalability it’s time to take advantage of
the decoupled way of development we use and separate those subdomains on separate services.

29. Revisiting the Architecture

30. THANK YOU