AI/ML Infra Meetup | Perspective on Deep Learning Framework
Caching 101: Caching on the JVM (and beyond)
1. @alexsnaps @ljacomet#Devoxx #cache101
Caching 101
Caching on the JVM
and beyond
alex snaps - principal software engineer at terracotta
louis jacomet - lead software engineer at terracotta
3. @alexsnaps @ljacomet#Devoxx #cache101
Introductory Poll
•Who knows nothing about caching?
•Who already uses caching in production?
•Who had caching related problems in
production?
•Who is interested in advanced caching patterns?
7. @alexsnaps @ljacomet#Devoxx #cache101
Cache coherence
• P write A to X, P read X => returns A when no writes
happened in between, must be valid for single processor too
• P2 writes A to X, P1 reads X => returns A if “enough time” has
elapsed
• P1 writes A to X, P2 writes B to X => no one can ever see B
then A at X
9. @alexsnaps @ljacomet#Devoxx #cache101
Numbers explained
L1 cache reference 1 s 2 heartbeats
Branch mispredict 3 s Yawn
L2 cache reference 4 s (Longer) Yawn
Mutex lock / unlock 17 s Coffee preparation
Main memory reference 100 s Brushing your teeth
Send 2k over commodity network 4.2 s A song
10. @alexsnaps @ljacomet#Devoxx #cache101
Numbers explained
Compress 1kB with zippy 33 m Sitcom episode
Read 1MB from memory 2 h Bad commute
SSD random read 4 h Half day at work
Read 1MB from SSD 2.6 d Long week-end
Round trip in datacenter 10 d Two weeks delivery
Packet roundtrip CA to Be 4.8 y PhD thesis
14. @alexsnaps @ljacomet#Devoxx #cache101
Further reading
• Gustafson’s law states that
computations involving
arbitrarily large data sets
can be efficiently
parallelized.
• As computing powers
grow, larger problems can
be solved in a given time
16. @alexsnaps @ljacomet#Devoxx #cache101
What is a cache in an application?
• Data structure holding a temporary copy of some data
• Trade off between higher memory usage for reduced latency
• Targets:
• Data which is reused
• Data which is expensive to compute or retrieve
19. @alexsnaps @ljacomet#Devoxx #cache101
JSR 107
• Java Community Process driven standard
• Specifies API and semantics for temporary,
in-memory caching of Java objects, including object creation,
shared access, spooling, invalidation, and consistency across JVM's
26. @alexsnaps @ljacomet#Devoxx #cache101
Cache aside conclusions
• Solution most seen out there: Spring, Play, Grails, …
• Most often based on annotations
• Tricky to get the concurrency and / or atomicity right
• Especially when rolling your own
• Does not resolve doing multiple real invocations when warming cache
33. @alexsnaps @ljacomet#Devoxx #cache101
Cache through conclusions
• Requires different abstraction / modelling
• Viewing the system of record through the cache may not be easy
• Provides better guarantees and consistency as invalidation is no
longer required
• Still falls apart if data is modified by other applications / processes
• Cost of writing is paid by thread putting in the cache
39. @alexsnaps @ljacomet#Devoxx #cache101
Write behind conclusions
• Scales out your writes
• Batching and coalescing
• Persistent queue or not
• Idempotent operations are important in distributed and failure
conditions
45. @alexsnaps @ljacomet#Devoxx #cache101
Moving off heap
• JVMs have issues with large heaps due to garbage collection
• OffHeap is a nice alternative for data which has a well known
lifecycle
• Perfect match for cache data
• Performance hit due to binary representation compared to
objects on heap
47. @alexsnaps @ljacomet#Devoxx #cache101
Clustering
• Cache shared between multiple machines
• Different topologies
• Peer to peer
• Client server - with or without client cache
• Consistency becomes a much harder problem
51. @alexsnaps @ljacomet#Devoxx #cache101
Probabilistically Bounded Staleness
• In how “long” will a write be “readable” by all ?
• How many “old versions” are still around ?
• Depending on
• Network delay
• Node processing time
• … delayed replication
(e.g. batching)
• LinkedIn’s data stores returned consistent data 99.9 percent of the time
within 13.6 ms, and on SSDs (solid-state drives) within 1.63 ms.