Speaker: Ben Stopford, Technologist, Office of the CTO, Confluent Are events the new API? Event driven systems provide some unique properties, particularly for microservice architectures, as they can be used both for notification as well as for state transfer. This lets systems run in a broad range of use cases that cross geographies, clouds and devices. In this talk we will look at what event driven systems are; how they provide a unique contract for services to communicate and share data and how stream processing tools can be used to simplify the interaction between different services, be them closely coupled or largely disconnected. Ben is a technologist working in the Office of the CTO at Confluent Inc (the company behind Apache Kafka®). He’s worked on a wide range of projects, from implementing the latest version of Kafka’s replication protocol through to developing strategies for streaming applications. Before Confluent Ben led the design and build of a company-wide data platform for a large investment bank. His earlier career spanned a variety of projects at ThoughtWorks and UK-based enterprise companies. He is the author of the book “Designing Event Driven Systems,” O’Reilly, 2018. Watch the recording: https://videos.confluent.io/watch/8MLuNHnE3uSZPgstdzSk4Q?.

1. Leveraging Microservice
Architectures & Event-Driven
Systems for Global APIs
Ben Stopford

2. About me:
- Office of the CTO @Confluent
- Kafka Core (replication protocol,
throttling, data balancing…)
- Built a central data service for a large
financial intuition
- Background in High Performance
Computing
- Ex-Thoughtworks
http://bit.ly/designing-event-driven-systems

3. What is an API?
Ordering fish and chips
Chef
Customer

4. What is an API?
ChefCustomer
Choose your
food

5. What is an API?
Ask the chef for
cod and chips

6. What is an API?
Chef does work:
making your food

7. What is an API?
You get your food and eat it

8. An API just describes this
interchange
• Choose what you want
• Ask for your food
• Wait for it to be prepared
• Get your food

9. With Events
Events are just facts that describe the real world:
• User: ordered fish and chips
• Kitchen: fish and chips prepared
• User: fish and chips consumed

10. Use a ‘Broker’: A middle man.
KitchenTable
The waiter records what you
chose to eat
The waiter brings your
food back to you

11. Why?

12. What if the chef has taken
a break?

13. What if the chef is busy?

14. What if orders need to go
to another kitchen

15. What if want to optimize
service

16. Contract (API), from the user’s
perspective, is two events
Create: fish and chips ordered
Listen for: fish and chips prepared
Schema

17. APIs work well for simple
applications
App

18. EcosystemsApp
Future of software lies in
ecosystems

19. At some point most businesses
outgrow the monolith

20. Startup that grows

21. Enterprise that already exists

22. This isn’t simply about scale

23. We can scale a
monolith!

24. Well we can load balance

25. Even in the data layer!

26. Ecosystems are about people
People need to form groups and operate independently

27. We have many teams

28. That need to change and
evolve independently
Change

29. So this is about more than just
growing. It’s about evolving in time.

30. The Problem is DATA

31. Catalog
Most services share the same core facts.
Most services
live in here

32. Orders
Service
Payments
Service
Customers
Service
APIs lead to data fragmentation

33. Service A Service B
Service C
One option is to share a database

34. Brings us back to the
‘One big app’ approach
App

35. Microservices shouldn’t
share a database!
" Sam Newman
(Author of

36. Better idea is to
use events

37. Businesses have an implicit
flow
Order
Requested
Order
Validated
Payment
Processed
Shipment
Dispatched
Shipment
Delivered
Shipment
Prepared
Order
Completed
Order
Confirmed

38. Fit services around this flow
Order
Requested
Order
Validated
Payment
Processed
Shipment
Dispatched
Shipment
Delivered
Shipment
Prepared
Order
Completed
Order
Confirmed
Basket Service
Order Service
Shipment Service
Payment Service

39. Let’s explore this event driven
approach with an example
Orders
Service
Shipping
Service
Customer
Service
Webserver

40. Buying an iPad (with REST)
Submit
Order
shipOrder() getCustomer()
Orders
Service
Shipping
Service
Customer
Service
Webserver

41. Events for Notification Only
Message Broker (Kafka)
Submit
Order
Order
Created
getCustomer()
REST
Notification
Orders
Service
Shipping
Service
Customer
Service
Webserver
KAFKA

42. Pluggability
Submit
Order
Order
Created
Orders
Service
Shipping
Service
Customer
Service
Webserver
KAFKA
Repricing
getCustomer()
REST
Notification

43. Events for Data Locality
Customer
Updated
Submit
Order
Order
Created
Data is
replicated
Orders
Service
Shipping
Service
Customer
Service
Webserver
KAFKA

44. Events have two hats
Notification Data
replication

45. => Immediacy,
Pluggability
=> Autonomy,
Locality
Notification
Data replication

46. Validation
Service
Email
Service
T2’’
Tax
Service
Order
Service
Services connected via a log of events

47. --------------------------- Internet ---------------------------
FinanceFulfillment
Web
Query
Web
Command
Scale to multiple departments

48. Kafka
US East
Tokyo
EU West
Scale to multiple clouds and geographies

49. The Global API
==
The Global Source of Truth

50. Stream Processing

51. Events are the key to
scalable service
ecosystems

52. Streaming is the toolset for dealing
with events as they move!

53. Kafka: a Streaming Platform
The Log ConnectorsConnectors
Producer Consumer
Streaming Engine

54. Kafka: a Streaming Platform
The Log ConnectorsConnectors
Producer Consumer
Streaming Engine

55. What is a Distributed Log?

56. An event log is a simple idea
Messages are added at the end of the log
Old New

57. Readers have a position all
of their own
Sally
is here
George
is here
Fred
is here
Old New
Scan Scan
Scan

58. You can rewind and replay,
just like Tivo!
Old New
Sally
is here Scan

59. Scalability in every layer
Producing
Services
Kafka
Consuming
Services

60. Each shard is a queue
Producing
Services
Kafka
Consuming
Services

61. Consumers share load
Producing
Services
Kafka
Consuming
Services

62. Load Balanced Services

63. Fault Tolerant Services

64. Build ‘Always On’ Services
Rely on Fault
Tolerant Broker

65. Distributed logs can hold huge
datasets (100TB+)

66. Kafka: a Streaming Platform
The Log ConnectorsConnectors
Producer Consumer
Streaming Engine

67. Kafka Connect
Kafka
Connect
Kafka
Connect
Kafka

68. Kafka: a Streaming Platform
The Log ConnectorsConnectors
Producer Consumer
Streaming Engine

69. A database engine for data-in-flight

70. SELECT card_number, count(*)
FROM authorization_attempts
WINDOW (SIZE 5 MINUTE)
GROUP BY card_number
HAVING count(*) > 3;
Continuously Running Queries

71. KAFKA
Buffer 5 mins
Windows, Buffers & Joins

72. Lookup tables
KAFKA

73. Features: similar to database query engine
Join Filter
Aggr-
egate
View
Window

74. Scales Out

75. Streaming leverages the
two hats
1. Processing data incrementally
2. Moving data to where it needs
to be processed (quickly and
efficiently)
On Notification
Data Replication

76. Key Features
• ‘Big data’ throughput & storage
• High availability and always on execution
• Messaging separated from processing/routing
• Routing/inspection/analytics provided via SQL

77. Kafka: a Streaming Platform
The Log ConnectorsConnectors
Producer Consumer
Streaming Engine

78. So what do real-
world streaming
ecosystems look
like?

79. KAFKA
Serving
Layer
Streaming
Handling Scale: Internet Workloads

80. Handling Disconnectedness
KAFKA KAFKA

81. Analyzing in real-time
RF
HQ
Analytics

82. Small ecosystems
(real-time loan processing)
HTTP To middle
office
To back
office
Rules
Aggregate
Post
validation

83. Large Ecosystems
(Event Driven Bank)
APP KAFKA
APP
APP
APP
APP
APP
Front Office
Operations
Risk
Finance
Many locations

84. So…

85. Good architectures have little to do
with this:

86. It’s about how systems
evolves over time

87. Evolution
• Evolution of People
• Evolution of Software
• Evolution of Data

88. Data in manyData in one place
You need to make a decision
around data

89. If the latter you need
something to tie it all together

90. You need a global middle
man

91. Kafka
Kafka
US East
Tokyo
EU West
Departments, Regions, Clouds

92. Events: A Global API
Notification Data
replication

93. Streaming is the toolset
for dealing with events
as they move!

94. http://bit.ly/designing-event-driven-systems

95. Twitter:
@benstopford