77. 77
Insert data Materialized View
Create via a query
(select * from Orders
Where Region == ‘USA’)
Query is rerun on data as it arrives
Base
Table
Materialized Views in a DB
78. 78
In practice often more complex
select Order.region,
count(order.quantity)
from Orders, Product, Customer
where Product.group = ‘Houshold’
and Customer.type = external
group by Order.region
Orders Product Customer
View
Read
Optimized
79. 79
Same problem but with services
select Order.region,
count(order.quantity)
from Orders, Product, Customer
where Product.group = ‘Houshold’
and Customer.type = external
group by Order.region
My Service View
Read
Optimized
Orders
Service
Product
Service
Customer
Service
86. 86
High Throughput Data Movement
Service
Service
Service
Service instance 1
Service instance 2
Service instance 3
Service instance 4
Kafka Cluster
(many machines)
92. 92
Take only the data you need
{“Stock Inventory”: {
“Id”: “Foo1234”,
“Vendor”: “Foo Industries”,
“Description”: “This is a …”,
“Delivery Category”: “ND”,
“Stock Status”: [
“Items in Stock”: 53,
“Items on Order”: 0
]…etc…}
93. 93
Data Movement
Be realistic:
• Network is no longer the bottleneck
• Indexing is:
• In memory indexes help
• Keep datasets focussed
94. 94
12. Use the log instead as a ‘database’
(for data-on-the-inside)
104. 104
(2) Stateless, Data Enabled Processor
Similar to star schema
• Facts are stream
• Dimensions are GKTables
e.g. Enrich Orders with Customer Info & Account details
Stream
GKTable
GKTable
Stream-Table
Join
105. 105
(3) Gates
e.g. rules engines or event correlation
When Order & Payment then …
Stream 1
Window
Window Stream-Stream
Join
Stream 2
106. 106
(4) Stateful Processor
e.g.
- Average order amount by region
- Posting Engine (reverse replace of previous position)
State store
backed up to Kafka
113. 113
All order logic,
stateless service
Orders-by-Customer view
(KStreams + Queryable State API)
UI Service
Stream
Maintenance &
Monitoring
Highly available, stateful service
UI uses Orders-by-
Customer View directly
via Queryable State
History Service pushes data to
a local Elastic Search Instance
Orders
Service
Derived
View
UI joins data
Tables & Streams
Fulfillment Service
Analytics
OrdersProduct Custo-
mers KTables
KTables KTables
Schemas