An ode to the underrepresented and underused pattern of events and asynchrony in the design and development of Microservices.
Prepared by Saul Caganoff, and delivered by Saul at Melbourne Microservices, and by Yamen Sader at Sydney Microservices.
2. Introduction
• Asynchrony is considered a microservice
characteristic
– But not often discussed
• In the top 20 Google results
– 7 mentioned async/choreography or events
– 3 of those by Chris Richardson or Martin Fowler
• Mostly mentioned in passing
• Honourable Mentions:
– http://www.infoq.com/articles/microservices-intro
– http://highscalability.com/blog/2014/4/8/microservices-not-a-free-
lunch.html
9. Coupling & Cohesion
• Cohesion – how well a subsystem forms a
uniting whole
– How well a bounded context does its job
• Coupling – the propensity for change in one
part of a system to disrupt another
– how badly “stuff” leaks across the boundaries
• Coupling is evil – it sneaks into solutions in all
kinds of insidious ways
– But… distributed monolith is the “worst kind of
evil”
10.
11. Coupling & Cohesion
• Microservices strive to eliminate coupling
• Insidious sources of coupling
– Shared databases
– Single Common Data Model (vs bounded contexts)
– DRY – shared code
– APIs generated from code
– Poor client code
– Misunderstood ‘transactions’
– Assumptions about business processes
– Organizational structure
– Services…
12. Business Processes
• End-to-end business processes require
coordination across bounded contexts
Finance
Sales
Ful-
filment
Product
13. Business Processes
• Browse the inventory
• Create order
• Enter Shipping Details
• Quote shipping charge
• Make the payment
• Update inventory
• Create Shipment
• Pick & Pack
• Courier
• Accept delivery
GET /products
POST /order
PUT /order/:id/address
GET /3pl/charges?zip=…
POST /finance/payment?order=
POST /products
POST /3pl/shipment
GET /3pl/shipment/:id
GET /3pl/shipment/:id/address
PUT /3pl/shipment/:id?status=
14. Business Processes
• Orchestration
– Centralized coordination
– E.g. the user-agent, BPM engine, service facade
– the “god” object
• Hypermedia
– HATEOAS
– But what about at the edges?
– How much should one BC know about another BC?
“microservices architecture will never be SOA done right unless it means building
hypermedia systems of microservices” – Jason Bloomberg
http://intellyx.com/2015/07/20/are-microservices-soa-done-right/
16. Data Synchronization
• How to handle data “owned” by multiple
domains
– Glibness: “don’t” have multiple data owners
– …in reality it is impossible to eliminate
– …especially if legacy or COTS systems involved.
• Some data synchronization is naturally
conveyed by the business process
• …some is not
• Another topic!
17. Choreography
• Use events to coordinate activity across bounded
contexts
• Publishers & Subscribers are highly decoupled
– Event subscribers can come & go minimal or no code
or configuration changes
• Publishers & Subscribers are more resilient to
change – either planned or unplanned
– E.g. events redelivered in the event of subscriber
failure – either to the restored subscriber, or a
replacement
19. Choreography
• But trade-offs
• Asynchronous events lie at the heart of some of
the microservices challenges
– Centralised orchestration is harder to change, but
easier to control/monitor
– Event-based choreography is easier to change but
harder to control/monitor
• Distributed logging/monitoring systems are
needed to provide a view into asynchronous
interactions
20. An Aside
• No global view may seem like a major downside
• But very rare for anyone to know this anyway
• When X happens, we do A and B
• When A happens, we do F
• Etc
• The more ‘inter-domain’ concerns can be solved
by just depending on events, the more likely you
are to have the ‘correct’ boundaries
21. Coordination
• Think about hypermedia inside boundaries
• Events outside the boundaries
Finance
Sales
Ful-
filment
Product
23. Commands & Events
• Duality between commands & events
– Commands cause state transitions
– Events reflect state transitions
• An external observer can infer the state of a
resource from its event-stream
– This is the essence of event-sourcing & CQRS
26. Event Structure
• Immutable structures
• Typical properties
– Timestamp
– Source
– Entity URI
– Action (state transition)
• The MVP is to notify that a resource has changed
and allow subscribers to pull the update if
needed
• Don’t forget Postel’s Law, versioning etc
27. Event Granularity
• Customer details changed?
• Or email address changed?
• Some considerations:
– Event storm with the business or the technical
specialists, they will call out ‘special’ events
– ‘Eye of the Beholder’ ie the subscriber determines
deltas important to them
– Expect to get this wrong once or twice
28. Concurrency
• Race Conditions
• The importance of order
• Single Master vs Multi-Master
• Deltas vs Copies
• Action indicators
• CRDTs – conflict-free replicated data types
• …think in terms of events rather than copies
30. Pull vs Push
• Pull = subscribers beat
– Subscriber never get overwhelmed
– Messages pile up on publisher
• Push = publishers beat
– Publisher never gets backed up
– Subscribers can be overwhelmed
• Answer: use an intermediary tuned to this
problem
32. Known Quantity
• Events are highly regular
– Append only series
– Generic attributes (topics, metadata)
– Timestamped
– Consumed in sequence
– Like a very simple database
• Common to standardise on an intermediary
– Like standardising on a database
– Usually exposed directly, a platform service
33. Syndication (Pull)
• Service exposes an RSS/atom feed
• Consumers do all the hard work:
– Subscription & Concurrency
– State Management (where am I up to?)
– Retries
• Positives
– Very easy…well-known pattern
– Web-scale
• Negatives
– Polling
– High latency
– Consumer concurrency very difficult
34. Web Hooks (Push)
• Service POSTs events to subscribers
• Publishers do all the hard work:
– Manage subscribers
– Managed guaranteed delivery/ retries
• Positives
– Not polling, relatively lower latency
– Web technology (HTTP endpoints and load balancers)
• Negatives
– Consumer concurrency a little harder
– Retries and history difficult
35. Message Queues (Push)
• Events published to queues/topics
– E.g JMS, AMQP, ZMQ, SQS/SNS, IronMQ
• Positives
– Mature, well-known technologies
– Good open-source implementations available
– Competing consumer pattern
• Negatives
– Extra infrastructure
– Reliability depends on reliable file-systems
• SANs, Distributed file-lock managers
– No history or replay
– Slow consumer problem
36. Pull vs Push (redux)
• Pull
– Subscriber tracks state
– Intermediary simplified
– Consumer concurrency more difficult
– Natural history is kept
• Push
– Intermediary tracks consumer state
– Can create a bottleneck and contention
– Consumer concurrency simpler
– Generally no history or replay
37.
38. Apache Kafka
• Distributed log
– Consumers tail the logs
• Consumers tail this log, but:
– Server manages consumer position (if desired)
– Server manages consumer concurrency
– Subscription is very simple
– Pull is very low latency (push-like)
– Replay and history is native
– Application level replication
• Others (eg Event Store) provide some similar features
(history, pull + push), but none as rich in capability
39. Event Processing
• Simple events are very useful for data
synchronization or process coordination
• Continuum of complexity
• Some tools: Drools, Riemann.io, Apache Storm,
Apache Spark (Streaming), Samza, etc
• This is a whole ‘nother topic
Cardinality State Tools
Single event Simple Seriously?
Multiple events/stream Simple Stream processor
Multiple events/stream Complex / Rules Complex Event Processor
40. Wrap-Up
• Bounded contexts are important
– they promote cohesion and reduce coupling
• You need to decide how to coordinate data &
processes across boundaries
– Orchestration (relatively higher coupling)
– Hypermedia (couples across boundaries)
– Events
• Prefer Hypermedia within bounded contexts & events
across bounded contexts
Events have many benefits but also some downsides -
they are an important tool in your toolbox
Notes de l'éditeur
Thank you Tyro for hosting again
Introductions?
ATP announcement
Hosting meetup: Many benefits, especially educating co-workers. We’re happy here at home with Tyro, but nice to go on holiday every now and again
Talking: Very broad, touches many things, services, events, platform, cloud, devops
My ‘inadvertent’ turn to step up again: all the credit, none of the blame. Interestingly you’ll see me potentially disagree with some points here, which should be exciting for you and frustrating for me.
Microservices 101: Why
Microservices Reference Architecture: What
Zooming in now…
If I read another Microservice blog or presentation that doesn’t mention events I’m going to break something
Communication across bounded contexts, async is important to decouple
Articles would say “very important” but “we’ll talk about it later”
Chris Richardson
Benjamin Wootton – very balanced on events, shows that its quite hard
Why by example
Bounded Context – self-contained domains in a solution, high cohesion inside, low coupling between.
Sales – Customers, orders
Product – Catalogues and inventory
Fulfilment – Getting it to them. Boring for a digital bookstore
Finance – Money and accounts
Zooming into the information model in each bounded context to show how the bounds work.
If coming from an integration background, you may be familiar with the concept of a common information model. Don’t want to couple the data structures of two systems together. Hundreds of these are difficult. One path was creating a common model, that you map to and from. Eg SAP to system neutral, from system neutral to Salesforce.
Good theory.. In practice. One data model to rule them all: doesn’t work. Too many concerns, too brittle and hard to change.
Note: will focus on information models here, but same applies to ‘services’ ‘processes’ etc in each domain.
Don’t want one, want different lenses and views.
Describe sales context.
Within Finance, interested in credit cards, PCI (maybe payments etc). Don’t care about the sales component.
Now about getting it to the customer. Care about their address mostly, the details of the shipment and how to get it to them.
And this is catalogue and inventory. Maybe historical data on the sales of each different product. Pivots from “Customer” to “Product” view.
Data models are slightly different in each context. But slightly shared too…
Two attributes.
Each component dealing with a ‘single’ concern. Eg sales just sales, not supply chain, not customers etc. Fulfilment, just cares about the order.
Flip side = coupling. Propensity for change to propagate.
Monolithic information model = heavily coupled. Change in one equals change in another.
Balancing act. Want all related concerns together, but to break apart beyond that or it gets too big and unwieldy. But break on the wrong seams and the coupling is too tight between parts and you have a distributed monolith.
Everyone says it
To eliminate all forms of coupling
While maintaining cohesion!
Microservices are not a free lunch, must be this tall etc, come with a lot of cost
Cost is there to eliminate as much coupling as possible
Can do it inside a monolith, but much harder to find the insidious sources
Services are insidious forms of coupling… ask/answer on demand.
End to end business process.
Attract them in, offers, recommendations etc.
Fill in order.
Pay.
Ship it out and modify inventory.
Crosses bounded contexts… They are ‘related’ in the end to end view. How do we support these?
Note: one sales order can become multiple ‘fulfilment’ orders or shipments. Can’t just expect a sales order = 1:1 fulfilment order.
End may not be the end. Restocking, returns etc. Breakages in packing etc.
So how do we orchestrate this?
Most common = the god object
Knows about all the other domains
Or the domains know about each other
Centralised, easy to see, is a little more brittle / harder / scarier to change – adding more interested parties…?>
Hypermedia makes it a little more pliable
Each step shows the links to the next step
Less hardcoding in the ‘god’ object
But some more insidious linking -> good within a bounded context (clients decoupled), but Sales will need to know about cross-domain concepts
Eg sales context knowing the next step is fulfilment or finance?
What’s missing is composition.
The y-axis with useless numbers!
Coupling as it relates to end to end business processes…
The ‘business process’ to update a customer for example, may need to update it across many bounded contexts. Not quite a business process, but we bleed data. Not the main concern today.
Now we’re talking about events!
Choreography is holding a process’ hand and moving it along. Choreography is more about the protocol of interactions between various applications, without an explicit controller or co-ordinator. Its more emergent, and is often best done with events. Each interaction is a little more ‘bounded’ – knows very little about everything else.
Eg sales domain goes “sales order published, I’m done!”
Each other domain gets to decide if they are interested in that.
New marketing system interested in sales orders – just do it!
Outages = less coupled, just continue (more resilient).
The primary difference between Services and Events.
Should be able to describe in detail what happens in then Sales domain to get this event to happen, and each other domain should be able to describe in detail what they do in response to everything. But end to end becomes more ‘academic’.
No free lunch. Visibility is very difficult. Harder to control and monitor.
Eg Splunk, Elasticsearch. Infer the state of a process from the outside – correlate them and infer.
Actually a benefit – natural breakdown of the problem space, and enables a solution space.
Or why this is so much simpler than any other modelling you will ever do!
Who has heard of event storming? Anyone done it?
Won’t get too deep into CQRS and event sourcing. A great way of thinking about the interactions between contexts even if you don’t use it to build the system inside the context.
Starting from the state model and transitions – or from the events – is very natural. This gives you the commands.
HATEOAS – what’s allowed at each point
Events are the Notifier of Application State EATNOAS – infer state based on the events
These are much more powerful than the ‘snapshot’ of current state that services often give
They naturally fulfil more use cases because by nature they are the fundamental building blocks of the system
Atomic..
Hypermedia commands to create a purchase
Then events notifying significant events across the contexts
The behaviour in the system is emergent. Easy to add new behaviour, easy to understand ‘isolated’ hops. Very little coupling – microservices are not exposed to details not relevant to their operations.
At the very least = what changed.
Problem with by reference only is no longer immutable.
Super ideal = event + current state, but with an understanding that ‘real current state’ is back in the source
At the very least = what changed.
Problem with by reference only is no longer immutable.
Super ideal = event + current state, but with an understanding that ‘real current state’ is back in the source
What granularity makes sense? Start with the important business processes. Eg is this an email address change, or a general address change? This is tough. No silver bullet. Know your domain and be prepared to change. Note: delta can be in the ‘eye of the beholder’.
Can happen, often a design issue, but can be infrastructure related: getting events out of order is a problem.
CRDTs can help with this, a talk for another day. Makes events more commutative, ie removing order as a problem.
Usually used for ‘copying data’ problems.
Cheap and cheerful. Very well known.
In
Yes, no security Graham! Not yet.
This is different from event processing – it’s about event distribution. Processing is another conversation.
Most processing is simple. When it gets more complex, it’s a whole another topic. Anyone want to talk about it?
An important tool. If its in your toolbox, worth considering it ‘first’ between contexts and seeing what happens.