Deep-dive into Microservices Patterns with Replication and Stream Analytics
Target Audience: Microservices and Data Architects
This is an informational presentation about microservices event patterns, GoldenGate event replication, and event stream processing with Oracle Stream Analytics. This session will discuss some of the challenges of working with data in a microservices architecture (MA), and how the emerging concept of a “Data Mesh” can go hand-in-hand to improve microservices-based data management patterns. You may have already heard about common microservices patterns like CQRS, Saga, Event Sourcing and Transaction Outbox; we’ll share how GoldenGate can simplify these patterns while also bringing stronger data consistency to your microservice integrations. We will also discuss how complex event processing (CEP) and stream processing can be used with event-driven MA for operational and analytical use cases.
Business pressures for modernization and digital transformation drive demand for rapid, flexible DevOps, which microservices address, but also for data-driven Analytics, Machine Learning and Data Lakes which is where data management tech really shines. Join us for this presentation where we take a deep look at the intersection of microservice design patterns and modern data integration tech.
WSO2Con2024 - Enabling Transactional System's Exponential Growth With Simplicity
Microservices Patterns with GoldenGate
1. Microservices Patterns with GoldenGate:
Deep-dive into Microservices Patterns with Replication and Stream Analytics
O R A C L E D E V E L O P M E N T , M A Y - 2 0 2 0
4. Microservices are Good!
4
Service Mesh Revolution
Emergence and widespread use of microservices have
directly led to revolution in DevOps, massive uptake in
Kubernetes and by 2020 the Service Mesh revolution
Key Benefits:
• Decomposition, of monolithic architecture
• Modularity, smaller services and improved
speed of initial development
• Independence, loosely-coupled systems that can
be created using different languages or data
• With the loose coupling also comes much
greater flexibility around deployment and
upgrades, eliminating complex dependencies
• Flexibility at Scale, deployments may start small,
run locally and later scale very wide, running
across multi-cloud environments and containers
• Sidecar pattern for “Mesh” frameworks are
further revolutionizing microservices
Chris Richardson: https://www.nginx.com/blog/introduction-to-microservices/
5. Dogma of Received Wisdom at the Data Tier
5
https://docs.microsoft.com/en-us/dotnet/architecture/microservices/architect-
microservice-container-applications/data-sovereignty-per-microservice
• “DB per Service” is a founding concept of Microservices
Architectures
• Some architects are more dogmatic than others:
• “All microservices must have an isolated data store” vs. “Some
microservices may share a data store”
• At scale, usually means that microservice application
must accept “eventual consistency” for data tier
Graph of Microservices vs. Classical App Architecture
“Drawbacks”: https://www.nginx.com/blog/introduction-to-microservices/
https://microservices.io/patterns/data/database-per-service.html
Encapsulate the data to simplify system
dependencies, create loose-coupling and
empower a good CICD/DevOps process!
Does this pattern always make sense?
Mr. Microservice
6. 6
https://en.wikipedia.org/wiki/Rorschach_test
• Service Granularity, DDD offers a weak heuristic for defining the domain
scope of a microservice (decomposition is hard)
• Enterprise Domains are Big, for larger applications (ERP etc) domains are
inherently large. domain decomposition can create artificial, subjective
boundaries that proliferate incompatible business ontologies
• Business Processes are Cross-Functional, a given business process may
cut across many data domains, corporate business units and even 3rd party
entities altogether (shared kernels and domain aggregates are needed)
• Context Matters, reasonable differences of object definitions and
taxonomy will naturally occur across large organizations (context
mappings)
• Legacy Applications, they exist. And, they don’t fit neatly into idealistic
Bounded Context mappings (bubble contexts for Legacy)
Context mapping is more art than science:
http://www.ouarzy.com/2016/07/25/micro-
service-and-bounded-context-clarification/
Context is in the mind of the modeler:
However…
Domain Driven Design (DDD) principles guide developers to create microservices that align to Bounded
Contexts, which “defines tangible boundaries of applicability of some sub-domain”
https://microservices.io/patterns/decomposition/decompose-by-subdomain.html
Challenges with Bounded Context & DDD
Sounds hard!
7. Ledger
Service
Command
Model
Domain
Event
Hardest Part About Microservices: Your Data
7
Availability & Durability – what happens with the
data when there is a service failure? How is the state
of data saved and restored? (non-volatility)
CAP Theorem & Network Partitions (CP or AP)
• Strong Consistency – which services require guaranteed
consistency, recovery and a single record of the truth?
• Eventual Consistency – if a microservice end-point is
unavailable, what is the retry protocol? Do all the
services in the Saga/Process adhere to the same
transaction semantics for retry and rollbacks?
ACID – what is the business impact of data
transactions ending in an inconsistent state? Will
business processes fail? Are there any regulatory or
compliance impacts? (ACID vs CAP, Google F1)
findOrderHistory()
findOrder()
Order Events
GL EventsShipping
Events
Logistics
Service
Command
Model
Domain
Event
Order
Service
Command
Model
Domain
Event
Order
History Service
Command
Model
Event
Handlers
Event
Handlers
Event
Handlers
Event
Store
many microservice design patterns
have been created specifically to
try and compensate for the data
consistency guarantees that
databases inherently provide
Eg: CQRS and Event Sourcing
DB encapsulation seems
to shift the very hard data
consistency issues to the
App Tier…
just accept “eventual consistency” and “lack of isolation” for transactions, and we’ll all be ok!
8. Data and Events are First-Class Citizens
8
Service Service Service
Analytics, Data Science and Data Lakes are too important to my
business’ Digital Transformation and Data-Driven initiatives…
we need architecture focus on Data and Events too
Application Microservices
Data Stores
Event Logs
produce events
consume events
produce events
consume events
read write
App
Events
Data
Events
DB Log
Events
Control Plane
• “State of the Truth”
at a point in time
(current or historic)
• Durable storage used
for Data Recovery /
Archives/years of data
• Polyglot, each service
may determine its own
data structures
• “Narrative of the Truth”
sequence of events
(between data snapshots)
• Days/months of event
data available as Time
Series or Messaging
• Strict ordering of events
& Idempotency
• Strong Consistency of
DB logs (eg; when using
GoldenGate)
• “Systems of Record”
at application tier
• APIs, business rules
and business object
semantics
• Not durable storage
The microservice API is king!
9. Can we take some of the best ideas of a Service Mesh and apply them to
Data and Events, to create a kind of Data Mesh?
Microservice Mesh and Data Mesh
9
Immutable
Raw Data Events Prepared Data Canonical Data
App
Events
Data
Events
DB Log
Events
Data Domain
Projection …n
Data Domain
Projection 1
Application
Microservices
Control Plane
11. What is a Data Mesh?
11
Microservice
Patterns
Log-based
Integrations
Polyglot Data
Movement
Data Mesh is a data-tier architecture to integrate and
govern enterprise data assets across distributed multi-
cloud environments
Microservices-centric:
• For the administration, deployment and monitoring of the core
frameworks of data movement and governance
• Aligns with for Service Mesh frameworks (K8S, Istio, etc)
• “Sidecar Proxy” style pattern for Events and Data
Immutable event-logs for data integrations:
• Messaging and data store events are globally accessible via
immutable event logs
• Logs may be used to drive Streaming or Batch integrations
Distributed data movement of all types of data
• A data mesh moves data: Relational, NoSQL, JSON, Graph…
• Relational data consistency (ACID) during data movement
• Must work reliably with enterprise OLTP data sets
https://en.wikipedia.org/wiki/ACID
Data
Mesh
Event
Streaming
Immutable
Logs
Data
Replication
Polyglot
Persistence
Edge / 5G
Frameworks
Domain
Driven
Design
Service Mesh
“Sidecars”
Data
Mesh
20. First Things First – GoldenGate Microservices
20
data replications
bi-directional
ms/sec updates
consistency guarantees
Cloud, Containers or Edge Devices
Extracts Replicats Client Libraries
Native GUIs and Full REST APIs
API Gateway or Proxy Service
GG GUI
GoldenGate is itself a set of microservices
that human users or other services may
interact with
Embedded User Interface:
• C-based microservices with embedded HTTP client for
native JavaScript based GUI
• Oracle Jet frameworks for intuitive interaction model
REST native APIs:
• Fully REST native
• Also available, a Command Line Interface (CLI) produces
REST calls to the native services
Full GoldenGate Replication Capabilities:
• 100% coverage for all traditional GG replication
patterns; fully capable of HA/DR use cases
GG
Admin
Service
GG
Distro
Service
GG
Metrics
Service
GG
Receiver
Service
GG
Service
Manager
Your
Services
21. Microservice Design Patterns for Data
21
Patterns for MicroservicesInherent to the Microservice Architecture is the developer
using specific patterns, sometimes the patterns are partially
embodied in a Programming Framework, but typically the
developers must choose to follow certain heuristics while
programming.
This presentation’s focus:
• “Database Patterns” & “Integration Patterns” …using DB Event
Replication (AKA: Change Data Capture) to improve them
• Simplify the pattern, make the microservice application more resilient
and provide better data consistency guarantees
DB Patterns for Discussion:
• Database per Service (covered earlier)
• CQRS – Command Query Responsibility Segregation
• Event Sourcing
• Saga Pattern
• Transactional Outbox
• Aggregates (AKA: Domain Events)
Transaction
Outbox
https://medium.com/@madhukaudantha/microservice-architecture-and-design-patterns-for-microservices-e0e5013fd58a