One of the trickiest problems with microservices is dealing with data as it becomes spread across many different bounded contexts. An event architecture and event-streaming platform like Kafka provide a respite to this problem. Event-first thinking has a plethora of other advantages too, pulling in concepts from event sourcing, stream processing, and domain-driven design. In this talk, Ben and Cornelia will tackle how to do the following: ● Transform the data monolith to microservices ● Manage bounded contexts for data fields that overlap ● Use event architectures that apply streaming technologies like Kafka to address the challenges of distributed data Speakers: Cornelia Davis, Author & VP, Technology, Pivotal Ben Stopford, Author & Technologist, Office of CTO, Confluent

1. © Copyright 2019 Pivotal Software, Inc. All rights Reserved. Version 1.0
Cornelia Davis
Vice President, Technology, Pivotal
@cdavisafc
February 2019
Microservices, Events, and Breaking
the Data Monolith Webinar
Ben Stopford
Office of the CTO, Confluent
@benstopford

2. Cornelia
Developer (wasn’t Ops)
Web architectures for >10 years
Cloud-native for 6+ years
Cloud Foundry for 6+ years
Discount code 40% off!: 40cloudnat
https://www.manning.com/books/cloud-native
@cdavisafc

3. Ben Stopford
• Leads Office of the CTO team at
Confluent.
• 20 years in engineering, mostly
distributed data.
• Worked on Kafka’s replication protocol,
rebalancing, throttling.
Download book: bit.ly/benstopford
@benstopford

4. @cdavisafc
Where we’re
going today
Microservices Goodness
Microservices Challenges
How Event-driven Approaches Help

5. Why Microservices
● Scale Applications
● Scale Teams
● Independent Development Cycles
● Experimentation
● Resilience

6. http://highscalability.com/blog/2014/4/8/microservices-not-a-free-lunch.html

7. The Microservices Death-star

8. Assumption:
Each microservice
boasts 99.999%
availability
(and there are
100 of them)
Assumption:
The network is
reliable
Result:
Home Page
achieves
99.9%
availability
https://en.wikipedia.org/wiki/Fallacies_of_distributed_computing

9. How we try to add a level of resilience?

10. Retries
▸ Client must consider failure
▸ Decide on fall-back
behavior
▸ Likely including retries
▸ But then we need to handle
downstream consequences of
these (retry) behaviors
CLIENT
SERVICE
Timeouts?
If we don’t hear back,
try again

11. But…
CLIENT
SERVICE
CLIENT
CLIENT
CLIENT
▸ Client must consider failure
▸ Decide on fall-back
behavior
▸ Likely including retries
▸ But then we need to handle
downstream consequences of
these (retry) behaviors

12. Circuit Breakers
▸ Client must consider failure
▸ Decide on fall-back
behavior
▸ Likely including retries
▸ But then we need to handle
downstream consequences of
these (retry) behaviors
CLIENT
SERVICE
CLIENT
CLIENT
CLIENT

13. But then, is stale data better than no data? - often it is
▸ Client must consider failure
▸ Decide on fall-back
behavior
▸ Likely including caches
▸ But then we need to handle
things like cache expiry
SERVICE
APP
SERVICE
APP
APP
SERVICE
APP
SERVICE
APP
SERVICE
APP

14. And there’s more…

15. New LIGHTWEIGHT ARCHITECTURES are emerging Microservices addressing speed to market and cloud scale
Monolithic / Layered Microservices

17. Pattern: Bounded Context
● Domain-Driven Design
● Each bounded context has a single,
unified model
● Relationships between models are
explicitly defined
● A product team usually has a strong
correlation to a bounded context
● Ideal pattern for Data APIs - do not fall
into the trap of simply projecting current
data models

18. Pattern: Database per Service
● Supports Polyglot
persistence
● Independent availability,
backup/restore, access
patterns, etc.

19. Pattern: Database per Service
● Supports Polyglot
persistence
● Independent availability,
backup/restore, access
patterns, etc.
But then what about…
● Overlapping data
● Data ownership
● Data synchronization

20. More patterns to the rescue

21. Request/Response - root to leaf; Events - leaf to roots

22. Connections’
Posts
API
Abundant Sunshine
Your source for healthy cooking
New From Your
Network
Seasonal Favorites
! Response
" Response
# Response
$ Request
%Request
&Request
Connections
API
User:
- id
- username
- name
Connections:
- id
- follower
- followed
Posts
API
Posts:
- id
- date
- userId
- title
- body
Starting with the familiar (Root to Leaf)

23. %
$
!
&
"
#
Connections’
Posts
API
Abundant Sunshine
Your source for healthy cooking
New From Your
Network
Seasonal Favorites
Response
Response
Response
Request
Request
Request
Connections
API
User:
- id
- username
- name
Connections:
- id
- follower
- followed
Posts
API
Posts:
- id
- date
- userId
- title
- body
Layer in caching to improve performance
Cache
How do we handle out of data
data?

24. µService
Event
Consumer
µService
Event
Producer
event store
… …
Introduce a broker/event store to decouple microservices
Key Properties:
- Messaging
- Scalability/HA
- Event Storage & Replay
- Stream Processing

25. Connections’
Posts
API
Abundant Sunshine
Your source for healthy cooking
New From Your
Network
Seasonal Favorites
Connections
API
Posts
API
Response
Request
1
2
event store
… …
1
Turn the flow on its head (leaves to root)
1
Connection Created Event - id
- follower
- followed
1
Post Created Event - id
- date
- userId
- title
- body
Cache
(Materialized View)
When the request comes in, the
Connections’ Post service already
has the answer!

26. Connections’
Posts
API
Abundant Sunshine
Your source for healthy cooking
New From Your
Network
Seasonal Favorites
Connections
API
Posts
API
Response
Request
1
1
1
1
2
event store
… …
Cache is populated when things change, not on request
Materialized View
View is formed to match what we need:
“New from your Network”

28. Event Sourcing

29. Traditional systems use mutable state
DB
This isn’t wrong, it’s just lossy

30. Events record the user’s journey
Shopping Cart
Events
2 Trousers added
1 Jumper added
1 Trousers removed
1 Hat added
Checkout
Shopping Cart
Event
User
Journey
12.42
12.44
12.49
12.50
12.59

31. Stored as a stream Stored statefully (think DB)
12.42
12.44
12.49
12.50
12.59 Information lost!
Event
User
Journey

32. 12.42
12.44
12.49
12.50
12.59
We can derive the current state
(but not the other way around)
DERIVE
Stream Processor

33. Connections’
Posts
API
Abundant Sunshine
Your source for healthy cooking
New From Your
Network
Seasonal Favorites
Connections
API
Posts
API
Response
Request
event store
… …
Our “Materialized View” is a form of Event Sourcing
View
Cache the view somewhere
convenient
Define the view in our code

34. Connections’
Posts
API
Abundant Sunshine
Your source for healthy cooking
New From Your
Network
Seasonal Favorites
Connections
API
Posts
API
Response
Request
event store
… …
When the applications starts it loads the cache from the log
View

35. Abundant Sunshine
Your source for healthy cooking
New From Your
Network
Seasonal Favorites
event store
… …
Event Store becomes the Source of Truth
Service A
Service E
C
Service D
DB
Apps
Search
NoSQL
Apps Apps
S T R E A M I N G P L A T F O R M
Service C
DB
Apps
Search
NoSQL
Apps App
DWH
Hadoop
S T R E A M I N G P L A T F O R M
C
Service B
DB
Apps
Search
NoSQL
Apps Apps
DWH
Hadoop
S T R E A M I N G P L A T F O R M

36. Event Streaming

37. Connections’
Posts
API
Abundant Sunshine
Your source for healthy cooking
New From Your
Network
Seasonal Favorites
Connections
API
Posts
API
Response
Request
event store
… …
Event Streaming
View
Recommendations
Service
1
2
Recommendation Service plugs
right into the existing flow

38. event store
… …
Event Streaming
Recommendations
Service
KafkaStreamsAPI
Posts
Connections
Page Views by User
Recommendation
Kafka Streams wraps up the .
solutions to these problems in a single API:
- Joins real-time streams
- Tables (from streams)
- Summarises streams

39. Connections’
Posts
API
Abundant Sunshine
Your source for healthy cooking
New From Your
Network
Seasonal Favorites
Connections
API
Posts
API
Response
Request
event store
… …
Event Streaming
View
Recommendations
Service
Create ‘views’ derived from
events in the log
Build event streaming services
using KStreams API
Decouple services from one
another using the event store

40. Summing it up…

41. Evolution of software systems
App AppApp App AppApp App AppApp App
AppApp App
Kafka
Monolith Distributed Monolith Microservices
Event-Driven
Microservices
Microservices
or?
AppApp App
AppApp App

42. Kafka
Event-Driven Microservices
AppApp App
AppApp App

43. Kafka
Event-Driven Microservices
AppApp App
AppApp App

44. AppApp App
AppApp App
Kafka
Event-Driven Microservices

45. Service Bindings
AppApp App
AppApp App
Kafka
Event-Driven Microservices
Service Bindings

46. © Copyright 2019 Pivotal Software, Inc. All rights Reserved. Version 1.0
Cornelia Davis
Vice President, Technology, Pivotal
@cdavisafc
February 2019
Microservices, Events, and Breaking
the Data Monolith Webinar
Ben Stopford
Office of the CTO, Confluent
@benstopford

47. Join us!
Keynote Speaker:
James Watters
Senior Vice President of Pivotal’s Product
and Business Development
Organizations
Register now at Kafka-Summit.org, and
get a special 25% off using this
code: KSNYPivotal25