2. WHY?
*LONG INTRO
Before diving in we need to
understand context and reasoning
As some of you are from the outside
the company, I need to give a bit more
details on how things work
So, bear with me
3. Behind the scenes of any internet company
Any projects begins with real life
And real life shows that every
company has a mess of a various
scale
Separate parts or subsystems can be
very clean and pretty, but we never
stop our progress and even clean
systems deteriorate with time, due to
project evolution and new features,
nigher loads that require new
architectures, etc.
Engineers are the creators and the
cleaners of this mess, today we’ll talk
about cleaning-up
One example would be:
4. Shopee app We’re Shopee, we’re doing e-
commerse :D
People buy and sell stuff, and when
they do, they have this useful info
numbers on their “orders” page
To ship, etc. Now, we found out that
having these numbers can cause
some nasty pain during sale events
Why?
5. CORE SERVER
Mobile app
&
Web clients
DB1 DB2 DB3
CORE SERVER*
*INSANELY SIMPLIFIEDVIEW
Some intro about core server
6. create transaction
commit transaction
query
query
query
query
query
SLOW (locking) query
query
query
CORE SERVER
Mobile app
&
Web clients
DB1 DB2 DB3
BOTTLENECKS!
When ppl buy stuff, number “to_ship”
changes for the seller.
All those numbers, to_ship, to_receive
(returns), etc are bunch of values in a
single row in a table.
When a lot of ppl buy stuff, this row
has many simultaneous updates
which leads to row locks, which leads
to transactions being timed out and
we have this avalanche effect
avalanche, when users can’t make a
purchase, they retry this whole big
transactions again and again, we
can’t serve new users, they
accumulate, everyone’s retrying to
make a purchase again and the whole
system is bought to a crawl
Shopee users are not happy, out DBA
are not happy, we gotta do something
7. CORE SERVER
Mobile app
&
Web clients
DB1 DB2 DB3
Let’s process slow (locking)
queries in background,
asynchronously
?
?
??
?
This info numbers are not absolutely
important in the big scheme of things,
they can be processed in background.
They can even be a bit delayed, it’s no
problem.
So, we need some new system
outside of core server, that could
handle these requests in background
8. DB3
Let’s process slow (locking)
queries in background,
asynchronously
?
?
??
?
This info numbers are not absolutely
important in the big scheme of things,
they can be processed in background.
They can even be a bit delayed, it’s no
problem.
So, we need some new system
outside of core server, that could
handle these requests in background
In fact we don’t need core server to
care about this logic at all. External
system could track buyer actions from
DB changes and update seller records
accordingly
10. CORE SERVER
Mobile app
&
Web clients
SERVICE
redis queue A
redis queue B
transformation
server
DB1 DB2 DB3
CODE / INFRA BLOAT!
Let’s continue cleaning things up!Another
example!
Explain what’s going on.
It’s already outside the core server, but
requires core server to have additional code
(that needs support, monitoring and is not a
general solution)
External system can be a complicated mess
that’s reinvented over and over again by
different teams
13. CORE SERVER
Mobile app
&
Web clients
DB1 DB2 DB3
CODE / INFRA BLOAT
This magic piece of infra is a bicycle
reinvention every time. It needs
servers, maintenance and it’s a
custom solution every time
SERVICE
redis queue A
redis queue B
transformation
server
17. EXISTING DBTOOLS?
TRIGGERS?
FUNCTIONS?
- Triggers allows to modify only storage itself using set of
predefined functions. React to insert/update/delete queries,
executes before or after the query
- Works only on DB host itself
- Limited in data processing capabilities
- Are bound to specific DB (mysql, oracle, etc)
- Can not send request to outside systems, queues
- Extending functionality is pretty much impossible
19. Data
Source
Data
Destination
Magic Data
Pipeline??
Again, looks like we need something
like this? General solution
- Works as independent service
- Has flexible data processing capabilities
- Not bound to specific data sources or destinations
- Connects completely unrelated systems in generic way
- Is easily extensible to support new systems
- It’s like DB functions/triggers taken to another level
24. OTHERS
LinkedIn's Change Data Capture Pipeline
SOCC 2012
We looked at other systems there are
not many in open source. It’s all
mostly internal systems never shared
with the outside world
This specific system is closely
connected with Oracle DB that’s used
at linkedin
34. DEC ARCHITECTURE:
1) Reads events from datasource
2) Applies transformations to events using simple transforms or (LUA script)
3) Serializes resulting queries to internal format using msgpack
4) Writes resulting binary queries to configured kafka topics
1) Reads binary queries
2) Deserializes queries and sends them to specified destination
3) Takes care of retry logic and events deduplication
36. Make sure DEC configuration has
correct data source and data destination
Step 2
CONFIGURATION
37. Implement and deploy necessary data
transformation scripts.
Step 3
CONFIGURATION
38. CONSUMER
EVENT TRANSFORMATIONS
1) DEC Consumer takes event from GDS queue
2) Filters event by table/event type (insert/update/delete)
3) Process with corresponding LUA script
48. CONCLUDING
All software projects are evolving and it’s always a mess
but we need to create decent tools to keep the entropy at bay
and DEC is one such attempt in this never ending battle :)