Video and slides synchronized, mp3 and slide download available at URL http://bit.ly/2h34ibv.
Yao Yue introduces Pelikan, a framework to implement distributed caches such as Memcached and Redis. Filmed at qconsf.com.
Yao Yue is a Distributed Systems Engineer Working on Cache at Twitter.
2. InfoQ.com: News & Community Site
• 750,000 unique visitors/month
• Published in 4 languages (English, Chinese, Japanese and Brazilian
Portuguese)
• Post content from our QCon conferences
• News 15-20 / week
• Articles 3-4 / week
• Presentations (videos) 12-15 / week
• Interviews 2-3 / week
• Books 1 / month
Watch the video with slide
synchronization on InfoQ.com!
https://www.infoq.com/presentations/
pelikan
3. Purpose of QCon
- to empower software development by facilitating the spread of
knowledge and innovation
Strategy
- practitioner-driven conference designed for YOU: influencers of
change and innovation in your teams
- speakers and topics driving the evolution and innovation
- connecting and catalyzing the influencers and innovators
Highlights
- attended by more than 12,000 delegates since 2007
- held in 9 cities worldwide
Presented at QCon San Francisco
www.qconsf.com
4. ABOUT ME
• 6 years at Twitter, on cache
• maintainer of Twemcache (OSS), Twitter’s Redis fork
• operations of thousands of machines
• hundreds of (internal) customers
• Now working on Pelikan, a next-gen cache framework to replace the above @twitter
• Twitter: @thinkingfish
10. “MILLIONS OF QPS PER MACHINE”
“SUB-MILLISECOND LATENCIES”
“NEAR LINE-RATE THROUGHPUT”
…
KING OF PERFORMANCE
11. “USUALLY PRETTY FAST”
“HICCUPS EVERY ONCE IN A WHILE”
“TIMEOUT SPIKES AT THE TOP OF THE HOUR”
“SLOW ONLY WHEN MEMORY IS LOW”
…
GHOSTS OF PERFORMANCE
12. I SPENT FIRST 3 MONTHS AT TWITTER
LEARNING CACHE BASICS…
…AND THE NEXT 5 YEARS CHASING
GHOSTS
31. LOCKING FACTS
• ~25ns per operation
• more expensive on NUMA
• much more costly when contended
source
32. MEMCACHE RESTART
…
EVERYTHING IS FINE
REQUESTS SUDDENLY GET SLOW/TIMED-OUT
CONNECTION STORM
CLIENTS TOPPLE
SLOWLY RECOVER
(REPEAT A FEW TIMES)
…
STABILIZE
A TIMELINE
lock!
lock!
52. WHAT WE KNOW
• alloc-free cause fragmentation
• internal vs external fragmentation
• OOM/swapping is deadly
• memory alloc/copy relatively
expensive
source
56. WHAT IS PELIKAN CACHE?
• (Datacenter-) Caching framework
• A summary of Twitter’s cache ops
• Perf goal: deterministically fast
• Clean, modular design
• Open-source
waitless logging lockless metrics composed config
channels buffers timer alarm
poo
ling
streams events
data store
parse/compose/tracedata model
request response
server
orchestration
threading
common
core
cache
process
pelikan.io
57. A COMPARISON
PERFORMANCE DESIGN DECISIONS
latency-oriented
threading
Memory/
fragmentation
Memory/
buffer caching
Memory/
pre-allocation, cap
locking
Memcached partial internal partial partial yes
Redis no->partial external no partial no->yes
Pelikan yes internal yes yes no
58. MEMCACHED REDIS
TO BE FAIR…
• multiple worker threads
• binary protocol + SASL
• rich set of data structures
• master-slave replication
• redis-cluster
• modules
• tools