Video and slides synchronized, mp3 and slide download available at URL https://bit.ly/2xlBfZ8. Adrian Trenaman describes how HBC Tech leverages event-sourced architectures that allow core services to reliably share their information with other services asynchronously. At the heart of each solution is the snapshot, the stream, and a bunch of deltas. Trenaman gives examples of these architectures in practice in their product, pricing, inventory and search services. Filmed at qconlondon.com. Adrian Trenaman is SVP Engineering at HBC Digital. He leads the engineering and infrastructure teams for Gilt in New York and Dublin. He is an experienced, outspoken software engineer, communicator and leader with over 20 years of experience working with teams throughout Europe, US and Asia in diverse industries such as financial services, telecoms, retail, and manufacturing.

1. A snapshot, a
stream, and a
bunch of deltas
Applying Lambda Architectures in a
post-Microservice World
Q-Con London March 6th 2018
Ade Trenaman, SVP Engineering, Raconteur,
HBC Tech
t: @adrian_trenaman
http://tech.hbc.com
t: @hbcdigital fa: @hbcdigital in: hbc_digital

2. InfoQ.com: News & Community Site
Watch the video with slide
synchronization on InfoQ.com!
https://www.infoq.com/presentations/
lambda-streams-delta
• Over 1,000,000 software developers, architects and CTOs read the site world-
wide every month
• 250,000 senior developers subscribe to our weekly newsletter
• Published in 4 languages (English, Chinese, Japanese and Brazilian
Portuguese)
• Post content from our QCon conferences
• 2 dedicated podcast channels: The InfoQ Podcast, with a focus on
Architecture and The Engineering Culture Podcast, with a focus on building
• 96 deep dives on innovative topics packed as downloadable emags and
minibooks
• Over 40 new content items per week

3. Purpose of QCon
- to empower software development by facilitating the spread of
knowledge and innovation
Strategy
- practitioner-driven conference designed for YOU: influencers of
change and innovation in your teams
- speakers and topics driving the evolution and innovation
- connecting and catalyzing the influencers and innovators
Highlights
- attended by more than 12,000 delegates since 2007
- held in 9 cities worldwide
Presented at QCon London
www.qconlondon.com

7. ~$3.5Bn
annual e-commerce revenue

8. 00’s of
Stores

9. What this talk is about
Solving the problem of microservice dependencies with lambda architectures:
> performance, scalability, reliability
Lambda architecture examples:
> product catalog, search, real-time inventory, third-party integration
Lessons learnt:
> It’s not all rainbows and unicorns
> Kinesis vs. Kafka

10. Some context: a minimalist abstraction of our
architectural evolution
2007
Monolith
2010
Service
Oriented
2012
µ-Services
λ λ λ λ λ λ λ λ λ λ λ λ λ λ
λ λ λ
λ λ λ λ λ λ
λ
2016
Rise of
Serverless λ
λ λ λ λ λ λ λ λ λ λ λ λ λ λ
λ λ λ
λ λ λ λ λ λ
λ
2018+
Multi-banner
Multi-tenant
Multi-region
λ architectures
Streams
GraphQL In the
seams

11. Slam on the breaks! Dublin Microservices Meetup, Feb 2015

12. Part 0
In which we briefly describe lambda architecture, and the Hollywood Principle

13. Lambda architecture: making batch processing sexy again.
Some kind of
data-source
A view of the data
Batch processing ‘the
baseline’
Stream processing
‘real-time’
Provide low-latency,
high-throughput,
reliable, convenient
access to the data.
Preserve the
integrity and
purpose of the
source of truth.
Stream: an
append-only,
immutable log store of
interesting events.

14. Lambda architecture: making batch processing sexy again.
Some kind of
data-source
A view of the data
Batch processing ‘the
baseline’
Stream processing
‘real-time’
Need to rebuild the
view? Take latest
snapshot, and replay
all events with a
greater timestamp.
T_7:
po
T_6:
dipsy
T_8:
tinky
T_1: foo
T_2: bar
T_3: pepe
T_4: pipi
T_5: lala
T_1: foo
T_2: bar
T_3: pepe
T_4: pipi
T_5: lala
T_6: dipsy
...

15. “Don’t call us,
we’ll call you.”

16. Inversion of control: previously, we ask for data when we need it.
Some kind of
data-source
A view of the data
Provide low-latency,
high-throughput,
reliable, convenient
access to the data.
Preserve the
integrity and
purpose of the
source of truth.

17. Inversion of control: now, when the data changes, we are
informed.
Some kind of
data-source
A view of the data
Provide low-latency,
high-throughput,
reliable, convenient
access to the data.
Preserve the
integrity and
purpose of the
source of truth.

18. Part I
In which we learn the perils of caching in a microservices architecture, and how
lambda architecture helped us out.

19. Gilt: we source luxury brands...

20. … we shoot the product in our studios

21. … we receive

22. … we sell every day at noon

23. … stampede!

25. The Gilt Problem
Massive pulse of traffic, every day.
=> serve fast
Low inventory quantities of high value merchandise, changing rapidly
=> can’t cache
Individually personalised landing experiences
=> can’t cache

26. Caching
“Just say no.”
“Until you have to say yes.”
“Then, just say maybe.”

27. consumer (e.g. web-pdp)
A stateless, cache-free library, busted.
product-service inventory-service price-service
commons <<lib>>
Hmm, engineer adds a local brand cache
to reduce network calls..
… and then later, another cache for
product information.
Leads to (1) arbitrary caching policies, &
(2) duplicated cache information.
product cache
brand cache

28. A caching library. Worked well initially, but...
product-service inventory-service price-service
consumer (e.g. web-pdp)
commons <<lib>>
We changed the commons library to
cache products with a consistent, timed
refresh (20m).
Worked well, until the business changed
its mind about one small thing: let’s
make everything in the warehouse
sellable.
Orders of magnitude more SKUs:
* JSON from product service > 1Gb
* Startup time > 10m
* JVM garbage collection every 20m on
cache clear
* ~1hr to propagate a change.
* m4.xlarge, w/ 14Gb JVM Heap
product cache
brand cache

29. Near real-time caching at scale
Source of Truth - PG
admin
web-pdp
commons
L1
* Startup time ~1s
* No more stop-the-world GC
* ~seconds to propagate a change.
* c4.xlarge (CPU!!!), w/ 6Gb JVM Heap
Next: replace JSON marshalling with binary
OTW format (e.g. AVRO)
S3
Brands, products,
sales, channels, ...
s
Elasticache
product
-service
Kinesis
Calatrava λ

30. https://github.com/gilt/calatrava - soon to be public

31. Part 2
In which we learn how we’ve used Lambda architecture to implement a near
real-time search index, but needed an additional relational ‘view of truth’.

32. Problem: polling a polling service means changes to
product data are not reflected in realtime.
Source of Truth - PG
admin
product
-service
search
-indexer

33. Source of Truth - PG
admin
View of Truth - PG
svc-search
-feed
Kinesis
Calatrava
VOT
* Changes are propagated in real-time to Solr
* Rebuild of index (s + *) with zero down time
* Same logic for batch & stream (thank you akka-streams)
* V.O.T.: “We needed a relational DB to solve a relational
problem”
S3
Brands, products,
sales, channels, ...
s

34. Part 3
In which we use a lambda architecture to facade an unscalable unreliable system
as a reliable R+W API… and benefit from always using the same flow.

35. Real-time inventory: bridging bricks’n’clicks
internet
OMS
warehouse
stores
Inventory
SOT
?

36. Real-time inventory: bridging bricks’n’clicks
OMS
warehouse
stores
Inventory
SOT RTAM
* Every sku
inventory level
every 24hrs
* Threshold (O,
LWM, HWM)
inventory events.
λ
Elasticache
R+W
* Absolute
inventory values
REST API
* APIBuilder.io

37. Making a web reservation
OMS
warehouse
stores
Inventory
SOT RTAM
λ
Elasticache REST API
R+W
1. Is inventory >0 ?
2. Attempt a reservation with OMS. IF it fails, generate a
random reservation ID.
3. Put the change on the RTAM stream
4. Update the cache (and stream, not shown)
5, 6, 7. Trigger a best effort to true-up inventory with ATP
(available to purchase)
1.
2.
3.
4.
λ
5.
6.
7.

38. X
THERE IS ONLY ONE PATH

39. Part 4
In which we learn that the paradigm generalises across third-party boundaries.

40. International E-Commerce: Taxes, Shipping & Duty is
HARD. Performance is critical!

41. Typical solution: cache for PDP & PA, go direct at checkout. Asymmetric,
with chance of sticker-shock.
Third Party
Shipping Partner
Intl Pricing
Cache
Product Listing
Product Details
Checkout
pricing
service

42. Elasticache
Stream driven solution with flow.io
Pricing
Service
Product
Listing
Product
Details
Checkout

43. Part 5
In which we consider Kafka vs. Kinesis

44. ∞
Stream: an immutable, append-only log.
Except it isn’t.
Which makes us use snapshots, and complicates our architecture.

45. LOG
COMPACTION

46. “Log compaction”: always remember the
latest version of the same object.
Source of Truth
(1, janes bond)
(2, dr. who)
(1, james bond)
(3, fr. ted)
(1, janes bond)
(2, dr. who)
(1, james bond)
(3, fr. ted)

47. TABLE
STREAM
DUALITY

48. KTable & Kafka Streams Library

49. K-Table & Kafka Streams...

50. #thanks @adrian_trenaman @gilttech @hbcdigital
(0) Apply lambda arch to create scalable, reliable offline systems.
(1) Replicate and transform the one source of truth
(2) It’s not all unicorns and rainbows: complex VOT, snapshots
(3) Kinesis is the gateway drug; Kafka is the destination.

51. Watch the video with slide
synchronization on InfoQ.com!
https://www.infoq.com/presentations/
lambda-streams-delta