Flipp is an e-commerce company that promotes weekly shopping opportunities. We began our migration to event-driven microservices in November 2016, and have since moved to nearly 300 Kafka-powered microservices. In this presentation we will explore the major strategies we have used in our migration from distributed monoliths to event-driven microservices. There have been a number of painful learnings and pitfalls along the way that we will share with you. Lastly, we will provide recommendations for each step of the way on your journey from monoliths to effective event-driven microservices. The first major section of this presentation deals with the liberation of data from monolithic services. In this section we will cover: Kafka Connect vs System Production, Event Schematization, Entities and Events, The importance of the Single Source of Truth, Consumption patterns and Event update verbosity. The second major section of this presentation discusses the usage of liberated event data in conjunction with other event streams.In this section we will cover common access patterns, handling (lots) of relational data, Stateful Foreign-Key Joins in Kafka Streams (See Kafka KIP-213), High frequency updates (price, stock) vs static properties and how to handle too many data streams. The third major section details how to abstract event complexity away, leverage the single source of truth and the usage of Core Events across a company. In this section we cover abstracting data streams, Core Events as detailed by the Single Source of Truth, Core Events in relation to bounded contexts and using Core Events successfully as a business.

1. Monoliths to Microservices

2. The Flipp app is used by millions of
people each week to find deals and
plan their shopping
50M
Downloads
90%
of North America’s
leading retailers

3. Agenda
1. Background
2. Principles
3. Liberating Data
4. Building Event-Driven Microservices
5. Core Events

4. 1 - Background

5. Classic Monolithic Architecture
Monolith sprawl
Teams sharing ownership (slow)
Tangled web of dependencies
Significant Technical Debt
Existing implementations hindering change
Multiple sources of truth

6. Moving Forwards
Modularize and decompose monoliths
While…
Event-driven microservice
• Decouple systems, teams and products
• Streamline application development

7. 2 - Principles

8. 1 - Single Source of Truth
Kafka

9. 2 - Use Event-Driven Microservices
MicroserviceTopic Topic
Microservice
MicroserviceTopic
REST
API
Request/Response

10. 3 - Avro Schemas
• Confluent Schema Registry
• Avro-encode everything published to Kafka
• Compatibility & Isolation
Schema
Registry
Producer ConsumerTopic
1) Validate /
Register
2) Publish 3) Consume
4) Obtain Schema
5) Process

11. 4 - Bounded Contexts - Services
Microservice
Topic - User Images
Business Functions
Technological Functions
Topic - Partner
Microservice
File
Store
Technological Functions
Microservice
File
Store
Business Functions
Microservice
Topic - User Images
Topic - Partner

12. 5 - Align Systems, Teams and Products
Team 1 Team 2
PII
Financial
Outgoing: 3
Incoming: 0
Outgoing: 0
Incoming: 3

13. 6a - Language and Technology Toolbox
Free-for-All Fully RestrictedToolbox
Criteria
& Selection
Process
Toolbox
Languages
Frameworks
Datastores
Services
Accepted

14. 6b - Toolbox -Templates & Generators
~Topic- Topic-
State
State
~Topic-
REST
API
~Topic-
Topic-
REST
API
~
Topic-

15. 7 - Single Unit of Deployability
State~Topic-
REST
API
Service 1 Service 2 Service 3
Single Unit of Deployability

16. 3 - Liberating Data

17. Why Liberation
• Decouple existing systems
• Support single-source of truth Principle
• Necessary for EDM

18. Kafka Connect - Pros
• Easy to get started
• Supports many connectors
• Can be use DB CDC, like Debezium
• Can source and sink data

19. Kafka Connect - Cons
• Inadvertently expose inner data structures
• Performance Issues (Source)
• Cluster Overhead
• Inter-team dependencies (owners vs users)
• Setting Precedence - Lethargy towards moving to EDM

20. Direct Production of Records
Monolith
Tabl
e
Tabl
e
Tabl
e
Tabl
e
Event
Producing
Logic
Isolate the Internals
Topic
Topic
Topic
Uses Confluent Schema Registry

21. Monolith - Forked Writer Anti-Pattern
Monolith
Data Table
1) Update
Data
2) Update
Table
3) Write to
Kafka
Topic

22. Lesson - Transactions and Output Table
ActiveReco
rd
(Rails)
Event Output
Table
Data Table
2) Transaction
1) Update
Data
Producer
(Separate
Thread)
3)
Asynchronous
Publish
Topic

23. 4 - Building Event-Driven
Microservices

24. What does building an app look like?
Topics &
Schemas
1) Data Discovery

25. Data Discovery - Data and Topic Lookup
Single Source of Truth

26. What does building an app look like?
Topics &
Schemas
1) Data Discovery 2) Create microservice
code
repo
register
microservic
3) Register for R/W Topic permissions
Topic-A
Topic-B
Write
Read
Read
4) Obtain Schemas +
Copy to Code Base
Schema-A.avsc
Schema-B.avsc
code
repo
5) Generate Code from Schemas
Schema-A.avsc
Schema-B.avsc
SchemaB.java
SchemaB.java
6) Write Business Logic
Topic-
Topic-
Busines
s
Logic
Topic-
Topic-
CI &
CD

27. Denormalization + Join Strategy
Exposed relational data as streams
Exists in our data models
Foreign keys
Denormalization vs. Joining

28. Relational Data - Downstream Join
Monolith
Tabl
e
Tabl
e
Tabl
e
Tabl
e
Topic
Topic
Topic
Tabl
e
Tabl
e
Topic
Topic
Consumer
Consumer

29. Relational Data - Upstream Denormalize
Monolith
Tabl
e
Tabl
e
Tabl
e
Tabl
e
Tabl
e
Tabl
e
Merge
Topic
Topic Consumer
Consumer

30. Denormalization + Join Strategy
Denormalize:
Closely related properties
Commonly referenced properties (Data domain knowledge)
Consumers perform joins on:
Distant data
Verbose data

31. Example - Item Availability
Items
Merchants
Many Items to
One Merchant
Topics
Item: Key=123, Value={size:M, color: Red, merchant: 500, … }
Merchant: Key=500, Value={name: Josh’s Fishing & Beards, … }
Event Examples
Stock Stock: Key=123, Value={LocationId:9000, Ordered:500, Online:

32. What do we do
Items
Merchants
Many Items to
One Merchant
Topics
Stock
One Stock to
One Item
Item
Eventification
Service
Enriched-
Topic
Item-Stock
Service

33. Foreign-Key Joins (Stateful)
Kafka Streams (KIP-213 - Used Internally)
External Data Store - Relational Database
Kafka Streams - Rekey + Group By Key?
- Limited by message space - List(A,B,C,D,E,F,G,H,I…)
- Reconciling additions and deletions is extremely problematic

34. 5 - Core Events

35. Core Events
Primitives
Basic entities (merchants, stock, price, items)
Basic events (user clicked on item)
Core events
Abstractions
Based on primitives or other core events
Business-wide meaning

36. Example - Budget Fulfillment
Budget
Merchant
UserClick
Topics
Budget
Calculation
Budget
Calculation
Budget
Calculation
App
Backend
Merchant
Reporting
Search
Team 1
Team 2
Team 3

37. Example - Budget as Core Event
Budget
Merchant
UserClick
Topics
Budget
Calculation
App
Backend
Merchant
Reporting
Search
BudgetFill
Team 1
Team 2
Team 3
Core Event

38. Other Core Event Examples
Resource Availability
Budget, Stock, Promotion, Order Fulfillment
Explicit Patterns of User Behaviour (aka Audiences)
Fraud-risk, churn-risk, geo-fencing

39. Primitives
Core

40. Questions?