Apache Kafka is changing the way we build scalable and highly available software systems. Providing a simplified path to eventual consistency and event sourcing Kafka gives us the platform to make these patterns a reality for a much broader segment of applications and customers than was possible in the past. Cloud Events is an interoperable specification for eventing that is part of the CNCF. This session will combine open source and open standards to show you how you can build highly reliable application that scale linearly, provide interoperability and are easily extensible leveraging both push and pull semantics. Concrete real world examples will be shown of how Kafka makes event sourcing more approachable and how streams and events complement each other including the difference between business events and technical events.

1. Building Event Driven
Architectures with
Kafka and Cloud Events
@DanRosanova
Group Principal Program Manager
Azure Messaging
Microsoft

2. Who am I and why am I here?
• Product owner for Azure’s
messaging services
• Career long Messenger
• Background in trading and
financial systems

3. Examining the parts of this talk
EVENT DRIVEN
ARCHITECTURE
APACHE KAFKA CNCF CLOUD EVENTS

4. What is Event Driven Architecture (EDA)
LOOSELY COUPLED ASYNCHRONOUS (NORMALLY) DOES NOT PROVIDE A
MEANINGFUL DIRECT RESPONSE
An architectural style (paradigm) built around reactive extensibility
with the following core principles

5. What do you mean by “Event-Driven”?
• Event Notification (what we will talk about)
• Event-carried state transfer (partially what we’ll talk about)
• Event Sourcing (not covered today)
https://martinfowler.com/articles/201701-event-driven.html
photo: Christopher Ferguson
Martin Fowler

6. Why use EDA?
Designed for distributed
systems
Simplifies horizontal
scale
Builds in resiliency

7. What is an event
A SIGNIFICANT CHANGE IN STATE E.G. A LIGHT TURNING FROM OFF
TO ON OR A TRADE FROM OPEN
TO FILLED
FROM HERE FORWARD WHAT WE
REALLY MEAN IS AN EVENT
NOTIFICATION

8. Attributes of
events
SELF CONTAINED AND
INDEPENDENT
CONTAINS
PROVENANCE
NOT TIED TO THE
LOGIC THAT IT WILL
ULTIMATELY TRIGGER
NORMALLY CONSISTS
OF A HEADER AND A
BODY
MAY OR MAY NOT
CONTAIN THE ACTUAL
STATE CHANGE
EVENTS ARE NOT
COMMANDS

9. Provenance
Who the
source is
“When” it
happened
What was the
cause of the
event

10. Deep thoughts
If an event happens and there’s
no one to receive it

11. Components of Event Driven Architecture
Emitter ConsumerChannel
• Generates and sends events
• Unaware of the existence of
consumers
• Does not expect responses
directly
• Reacts to events
• Directly processes
• Forwards
• May raise events of its own
• Communication conduit
• Exclusively responsible for the
distribution of events
• Does not imply transport or
protocol

12. Event Processing Styles
Simple event
processing (SEP)
•Individual single events
•Consumer directly
processes
•No further action
Events stream
processing (ESP)
•Unit of Work is stream
•Unending sequence
(codata)
•Consumer will have
some memory / state
Complex event
processing (CEP)
•Matching patterns
across events and
sources
•Not bound by time

13. Example stream processor
simpleMA(float[] prices, int period)
float result;
for(int i=0;i<period;i++)
result +=
prices[prices.length – i];
return result/period

14. Another example stream processor
Delay ReverbFlange

15. Extended from SEP to CEP
• Creating derivative
events
• Joining streams
• Over differing time
windows

16. Beyond the simple diagram – how EDAs really
look
Price Emitter
Moving
Average
Processor
RSI Processor
MACD
Processor
Trade Signal
CEP
Trading
Component

17. Consumer
Benefits of EDA: Scale out design
• Flowing changes
through the EDA
makes scaling out
trivial
• As scale grows just
add more nodes
• Easy!
Emitter
Consumer
Consumer

18. Benefits of EDA: Regionally distributed systems
• Add content-based
routing to deliver data for
where it’s needed
• Zero change for emitter
• Each region emits
• Each region receives

19. Benefits of EDA: Side by side deployment
• Start at the end and
work backwards
• Add a new version of
the consumer FIRST
• Consumers and
Emitters exist side by
side for some amount
of time
Emitter v1
Application v2
Application v1Application v3
Emitter v2
Emitter v3

20. Benefits of EDA: Side by side deployment
• Real systems are
not so simple
• There may be
hundreds of
components
• Downtime is not
as widely
accepted as it
used to be
Web Frontend Inventory Management Fraud Detection

21. Obvious drawbacks of EDA
Eventual
consistency
The channel is
now REALLY
important
End to end
visibility
Missed events
Replay Event storms CQRS

22. Hidden drawback of EDA: Semantic coupling
SUBSCRIPTIONS TYPES (SCHEMA) PATTERNS USED FOR
MATCHING EVENTS
Behavioral dependencies between components or services
<or> How to build a distributed monolith

23. What is
Apache
Kafka®

24. If you’re asking me, on
the last session of the
last day about what
Kafka is?

25. Where have
you been all
week?

26. Key
characteristics
of Apache
Kafka
A DISTRIBUTED
STREAMING PLATFORM
MANIFESTED AS
PARTITIONED APPEND
ONLY LOG
WITH CLIENT-SIDE CURSOR AND FAULT TOLERANCE

27. What Apache
Kafka is not

29. Cloud Events
• A specification for describing event data in common formats to
provide interoperability across services, platforms and systems
• An open standard built in an open forum via the CNCF’s Serverless
Working Group
• With many contributors across our industry (from Alibaba to
VMWare: can we a W,X,Y, or Z contributor?)

30. Why I believe strongly in open standards
REDUCES BARRIERS TO
ENTRY
SPURS INNOVATION PROTECTS EVERYONE
INVOLVED

31. https://ourworldindata.org/economic-growth

32. An example event
{
"id" : "A234-1234-1234",
"source" : "https://github.com/cloudevents/spec/pull",
"specversion" : "1.0-rc1",
"type" : "com.github.pull.create",
"subject" : "123",
"time" : "2018-04-05T17:31:00Z",
"datacontenttype" : "text/xml",
"data" : "<much wow="xml"/>"
}
RequiredOptional

33. Bringing it all
together

34. Kafka as a channel: Topics
Emitter
Consumer
• Obvious choice
• Solves replay
• Each consumer gets its
own offsets
• Decouples time
• Cloud Events already has a
Kafka Transport Binding Consumer

35. Does this
mean Kafka
for every
channel?
No!

36. Kafka as a consumer: KStreams
• Kstreams can materialize a
view for the scale out
model we saw earlier
• Can filter and forward
• Joins and aggregations
make CEP possible
KStream
Topic 1
Topic 2
Topic 3

37. Kafka for end to end visibility and logging
• Anything making
changes can write its
own events to a
centralized event log
• You can forward the
actual channel data as
well
KStream
Topic 1
Topic 2
Topic 3
Service 1
Service 2
Service 3
Emitter Topic
Global Log
Consumer
Topic ConsumerService N

38. Kafka as a router: Connect, MirrorMaker
• Write to local Kafka,
replicate and route as
needed
• Outbox for local services
in app, DC, or Geo
App 1
App
Health

39. Key take aways
AVOID THE ONE TOOL /
TOPIC TO RULE THEM
ALL
GO LOCAL – USE A
LOCAL CLUSTER /
COMPUTE
USE PORTABLE AND
OPEN FORMATS
PLAN FOR VERSIONING
AHEAD OF TIME –
CHANGE HAPPENS

40. Q&A

41. Open Stand