Data Management Patterns of Microservice CQRS and Event Souring

1. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
CQRS & Event Sourcing
Better Distributed Applications through CQRS, Event Sourcing, and
Immutable Logs
Inho Kang
Principal Sales Consultant
Infrastructure, Cloud Platform
Confidential – Oracle Internal/Restricted/Highly Restricted

2. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Safe Harbor Statement
The following is intended to outline our general product direction. It is intended for
information purposes only, and may not be incorporated into any contract. It is not a
commitment to deliver any material, code, or functionality, and should not be relied upon
in making purchasing decisions. The development, release, and timing of any features or
functionality described for Oracle’s products remains at the sole discretion of Oracle.
Confidential – Oracle Internal/Restricted/Highly Restricted 2

3.  .Net Developer
 CBD, SOA Methodology Consulting
 ITA/EA, ISP Consulting
 Oracle Corp.
 Middleware
 Open Source, Cloud Native Application
 OCAP Team
 DevRel
innoshom@gamil.com

4. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Program Agenda
Microservices
Event Sourcing
CQRS
Reference
Hands-On
1
2
3
4
Confidential – Oracle Internal/Restricted/Highly Restricted 4
4

5. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Microservice
Confidential – Oracle Internal/Restricted/Highly Restricted 5

6. Middleware/App Server
Transactions/Synchronous API Calls/Asynchronous
Monolith Problem : 1. Scaling 2. Performance 3. Deploy 4. Down

7. User Interface
Application
Datastore
Infrastructure
Resulting SoftwareTypical Enterprise Organization Structure
Head of IT
Head of
Operation
Head of DBAs
Head of
Infrastructure
Head of App
Dev
Head of UI
Head of
Development
An Enormous Monolith
Conway’s Law: Software reflects the structure of the
organization that produced it

8. Build small product-focused teams – strict one team
to one microservice mapping
Many Small Microservices
Resulting SoftwareMicroservices Organization Structure
API
Application
Datastore
Infrastructure
API
Application
Datastore
Infrastructure
API
Application
Datastore
Infrastructure
API
Application
Datastore
Infrastructure
Product Lead
Project Manager Sys Admin DBA
JavaScript
Developer
Developer
Developer
Sys Admin
Storage Admin
Graphic ArtistNoSQL Admin
Product Lead
Project Manager Sys Admin DBA
JavaScript
Developer
Developer
Developer
Sys Admin
Storage Admin
Graphic ArtistNoSQL Admin
Product Lead
Project Manager Sys Admin DBA
JavaScript
Developer
Developer
Developer
Sys Admin
Storage Admin
Graphic ArtistNoSQL Admin
Product Lead
Project Manager Sys Admin DBA
JavaScript
Developer
Developer
Developer
Sys Admin
Storage Admin
Graphic ArtistNoSQL Admin

9. Edge
Load
Balancer
Zuul
(Proxy Svc)
Playback
(Legacy Dev.)
API
(g/w)
Middle Tier & Platform
EVCache
Cassandra

10. Reference: http://microservices.io
http://microservices.io
 Patterns of Microserivce

11. Reference: http://microservices.io

13. Online Shopping Mall
Order Customer
ID CUSTOMER_ID STATUS TOTAL ID CREDIT_LIMIT …….
Order Table Customer Table

14. BEGIN TRANSACTION
…
SELECT ORDER_TOTAL FROM ORDERS WHERE CUSTOMER_ID = ?
…
SELECT CREDIT_LIMIT FROM CUSTOMERS WHERE CUSTOMER_ID = ?
…
INSERT INTO ORDERS …
…
COMMIT TRANSACTION
Traditional
 First Challenge - ACID
User Interface
Application
Datastore
Infrastructure
One Application
 Transaction

15. Consistency across multiple services
Order Customer
API
Application
Datastore
Infrastructure
Order
Microservice
API
Application
Datastore
Infrastructure
Customer
Microservice
 First Challenge - ACID

16. 2-PC (Phase Commit)
 First Challenge - ACID

17. Rules of Distributed Computing
Consistency
Each node shows the same data at all times
Availability
Each node is available for writes at all times
Partition Tolerance
Able to handle network outages
CAP Theorem
C
A P
Theory Practice
Pick One
Partition Tolerance is non-negotiable because
we have networks that can always fail
Enterprise IT Systems: Often CP
Microservice Systems: Often AP
Each microservice can be CP, AP or CA but the
system as a whole is always CP or AP
Pick
Any
Two

18.  First Challenge - BASE
Basic Availability
• The database appears to work most of the time.
Soft-state
• Stores don’t have to be write-consistent, nor do different replicas have to
be mutually consistent all the time.
Eventual consistency
• Stores exhibit consistency at some later point (e.g., lazily at read time).

19. Traditional
 Second Challenge - Query
User Interface
Application
Datastore
Infrastructure
One Application
SELECT *
FROM CUSTOMER c, ORDER o
WHERE
c.id = o.ID
AND o.ORDER_TOTAL > 100000
AND o.STATE = 'SHIPPED'
AND c.CREATION_DATE > ?
 Join Query

20.  Event Driven Architecture
Order
Customer
Message
Broker
(Pub/Sub)
Order
Created Event
Order
Created Event
Credit
Reserved
ID CUSTOMER_ID STATUS TOTAL
900 202 NEW 1000
ID CREDIT_LIMIT …….
202 5,000,000
Order Table
Customer Table
CUST_ID ORDER_ID AMOUNT
202 900 1000
Reserved_Credit Table
Credit
Reserved
1
2
3
4
Credit Limit
Exceeded
APPROVED
CANCELLED
Eventual Consistency

21. Order
Service
Customer
Service
 Event Driven Architecture
Message Broker
Customer Order
View Query Service
Customer Created
Customer Cancelled
Customer Shipped
……
Order Created
Order Cancelled
Order Shipped
…..
Customer
Order
View
Update
Find
Customer & Order Query

22.  Event Driven Architecture - Example
Profile
WebApp
Message Broker
Database Search App
Cache
Monitoring
App
Newsfeed
App
Memcached
HBASE/
CASSANDRA/
RDBMS
ELASTICSEARCH/
SOLR

23. ACCOUNT_ID ACCOUNT_TYPE BALANCE
12345 A 5,000,000
23456 B 250,000
34567 C 3,500,000
… …. ….
Account Table
SEQ ACCOUNT_ID … UPDATE_DATE
1 12345 .. ….
2 23456 … …
3 34567 … …
4
5
Account History Table
입금
출금
최종 시점의 상태 정보만을 보관
잔고 정보가 맞지 않는 경우?
원인 파악은???

24. Event Sourcing = 데이터 저장의 새로운 Pattern
Order
event #1 @ T1
event #2 @ T2
event #3 @ T3
event #4 T4
“Capture all changes to an application
state as a sequence of events”
”어플리케이션의 모든 상태 변화를
발생순서에 따라 이벤트로 보관한다.

26. https://martinfowler.com/eaaDev/EventSourcing.html
“Capture all changes to an application
state as a sequence of events”
”어플리케이션의 모든 상태 변화를
발생순서에 따라 이벤트로 보관한다.
– Martin Fowler

27. Feature
 Immutable
 Append Only
(No Update/Delete)
Contents
 Event ID(Aggregate ID)
 Type
 Revision
 TimeStamp
 PayloadOrderCreated
ItemAddedToOrder
OrderShipped
OrderDeliverred
…
…….
Event 속성

28. OrderCreated
OrderApproved
OrderShipped
OrderDeliverred
…
…….
EventReplay
잔고, 재고량
특정시점의
상태정보

29. OrderCreated
OrderApproved
OrderShipped
OrderDeliverred
…
…….
Snapshot

30.  장점
 Event Driven Architecture를 구현하면서 Atomic한 데이터 Publish가 가능
– Event 자체가 Atomic
 Domain Event(Aggregate) 를 적용함으로써 Object–Relation 의 Impedance
Mismatch 문제를 회피
 특정 시점의 상태값을 조회할 수 있다.
 단점
 상대적을 낯선 스타일 – Learning Curve
 Event Store기반의 쿼리에 한계점으로 인해 CQRS가 필수
 적용 분야 예시
 추천(장바구니), 분석(Debugging), 감사(Audit)
 Netflix Download Service License

31.  CQRS (Command and Query Responsibility Segregation)
명령과 쿼리의 역할 구분
 CQS
 Bertrand Meyer가 “Object Oriented Construction”이란 책에서
거론한 개념으로, 모든 메소드는 한번의 액션에서 1) 상태를 변경하는
커맨드든 2) 데이터를 반환하는 쿼리든 한가지 액션만 취해야 한다는
것이다. 다시 말하자면 질문을 할때 대답을 변경하지 말라는 것이다.
 이 개념이 Object로 확대되어, 상태를 변경하는 Command(CUD)와
Query(Read)를 분리해야 한다는 원칙

32. 단일 모델 모델 분리

33.  CQRS (Command Query Responsibility Segregation)
 Problem :
최근 일주일간 장바구니 아이템 추가 중간에 주문을 취소한 주문만 조회
 Solution
- 명령과 조회의 책임을 분리
- 상태변경을 처리하는 Command Model과
데이터를 조회하는 Query Model을 분리 구현
- 조회에 위한 Materialized View을 미리 생성

34. API Gateway
Message
Broker
Event Store
Command
Processor
(Write Logic)
API(Commands)
Event
View Store
Materialized
View
(Read Logic)
API(Query)
Event Projection

35. From Early in 2016

36. Download Business Requirements
● Devices with downloads
● Downloads per studio

37. Working With Event Sourcing

38. Debugging
● Possibly the biggest win
● Current tooling is rudimentary
{
"Event" : "LicenseAcquiredEvent",
"Event Time" : "2017-06-25 05:23:00 PM",
"aggregateId" : "aggregateId1",
"committed" : true,
"eventTimeStamp" : 1498411380903,
"eventOrderNumber" : 0
}

39. Event Sourcing

40. Domain Model

41. Make Purple
Command
Commands

42. Commands
Make Purple
Command
Command
Handler

43. Added Red
Event
Added Blue
Event
Make Purple
Command
Command
Handler
Commands

44. Events
Added Red
Event
Added Blue
Event
Event
Handler

45. Events
Added Blue
Event
Event
Handler

46. Aggregate

47. Aggregate
ServiceREST
Endpoint
Aggregate
Repository
Event Store
Query Query Query

48. Event Store
Row ID Events

49. Event Store
Row ID Events

50. Row ID Events
e1 e2
e3 e5
e6
e7
e4
Event Store
Aggregate id 1 Aggregate id 2

51. Aggregate
Repository
Query Event Store

52. Aggregate
Repository
Event StoreQuery
select * from events
where rowId =

53. Aggregate
Repository Event StoreResponse
e1 e2 e3 e5 e6 e7e4

54. Aggregate Repository
e1 e2 e3 e5 e6 e7e4
e1
Aggregate id 1
e2 e5 e6
Aggregate id 2
e3 e4 e7

55. Aggregate Repository
Aggregate id 1
e2 e5 e6
Aggregate id 2
Aggregate 1
Aggregate 2 e3 e4
e7
e1

56. Aggregate Repository
Aggregate id 2
Aggregate id 1
Aggreeg2ate 1 e5 e6
Aggre ate 2
3
e7

57. Aggregate Repository
Aggregate id 2
Aggregate id 1
Aggreeg5ate 1 e6
Aggre ate 2
7

58. Aggregate Repository
Aggregate 2
Aggregate id 2
Aggregate id 1
Aggreeg6ate 1

59. Aggregate Repository
Aggregate 2
Aggregate 1

60. Aggregate Repository
Aggregate 1
Command
Handler
Command

61. Aggregate Repository
Command
Handler
e8
e9

62. Aggregate Repository
e8 e9Aggregate 1

63. Aggregate Repository
Aggre8
ate1 e9

64. Aggregate Repository
Aggre9
ate1

65. Aggregate Repository
Aggregate 1

66. Aggregate
Repository Event Store
Update
append events
where rowId =
e8 e9

67. Event Store
Aggregate id 1 Aggregate id 2
Row ID Events
e1 e2 e3
e5 e6 e7e4
e8
e9

68. http://edisonxu.com/2017/03/23/hello-cqrs.html

69.  Axonframework.org / AxonIQ.com
 Since 2010, 8 years
 Version : 3.x
 Many systems in production
 Focus on Messaging in Commands,
Events and Queries
 Plug in Event Store
 JPA, JDBC and MongoDB OOTB
 Scalability with JGroups
 Eventuate.io vs RBMH Eventuate?
 Since 2015
 Version : Still 0.x
 Focus of Event Sourcing and Event
Publication
 Own Event Store implementation
 Scalability with Akka
https://stackoverflow.com/questions/43136291/axon-framework-vs-eventuate-comparison

70. • Chris Richardson : http://Micoservice.io
• http://www.slideshare.net/RichardHarvey7/micro-services-and-containers
• Thought Works - Real Case : http://vvgomes.com/microservices-event-
sourcing/
• Mastering Chaos : https://www.slideshare.net/JoshEvans2/mastering-chaos-
a-netflix-guide-to-microservices
• Scaling Event Sourcing for Netflix Downloads :
https://www.infoq.com/presentations/netflix-scale-event-sourcing

71.  CQRS is not a silver bullet...CQRS is not Event Sourcing...CQRS is not a
good wife...but...CQRS can open many doors. - Greg Young

73. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | Confidential – Oracle Internal/Restricted/Highly Restricted 73