Soumettre la recherche
Mettre en ligne
Redis to the Rescue?
•
40 j'aime
•
3,992 vues
Tim Lossen
Suivre
O'Reilly MySQL Conference 2011, Santa Clara
Lire moins
Lire la suite
Technologie
Signaler
Partager
Signaler
Partager
1 sur 76
Télécharger maintenant
Télécharger pour lire hors ligne
Recommandé
Redis to the Rescue?
Redis to the Rescue?
Tim Lossen
Facilitando a Programação concorrente com o Fork/Join
Facilitando a Programação concorrente com o Fork/Join
Mario Amaral
NOSQL101, Or: How I Learned To Stop Worrying And Love The Mongo!
NOSQL101, Or: How I Learned To Stop Worrying And Love The Mongo!
Daniel Cousineau
MongoDB Performance Tuning
MongoDB Performance Tuning
MongoDB
Mythbusting: Understanding How We Measure the Performance of MongoDB
Mythbusting: Understanding How We Measure the Performance of MongoDB
MongoDB
Json tutorial, a beguiner guide
Json tutorial, a beguiner guide
Rafael Montesinos Muñoz
Talk about ReactiveMongo at MSUG May
Talk about ReactiveMongo at MSUG May
Andrey Neverov
MySQL Without The SQL -- Oh My! PHP Detroit July 2018
MySQL Without The SQL -- Oh My! PHP Detroit July 2018
Dave Stokes
Recommandé
Redis to the Rescue?
Redis to the Rescue?
Tim Lossen
Facilitando a Programação concorrente com o Fork/Join
Facilitando a Programação concorrente com o Fork/Join
Mario Amaral
NOSQL101, Or: How I Learned To Stop Worrying And Love The Mongo!
NOSQL101, Or: How I Learned To Stop Worrying And Love The Mongo!
Daniel Cousineau
MongoDB Performance Tuning
MongoDB Performance Tuning
MongoDB
Mythbusting: Understanding How We Measure the Performance of MongoDB
Mythbusting: Understanding How We Measure the Performance of MongoDB
MongoDB
Json tutorial, a beguiner guide
Json tutorial, a beguiner guide
Rafael Montesinos Muñoz
Talk about ReactiveMongo at MSUG May
Talk about ReactiveMongo at MSUG May
Andrey Neverov
MySQL Without The SQL -- Oh My! PHP Detroit July 2018
MySQL Without The SQL -- Oh My! PHP Detroit July 2018
Dave Stokes
MongoDB at GUL
MongoDB at GUL
Israel Gutiérrez
Building Apps with MongoDB
Building Apps with MongoDB
Nate Abele
Building a Scalable Inbox System with MongoDB and Java
Building a Scalable Inbox System with MongoDB and Java
antoinegirbal
Pinterest的数据库分片架构
Pinterest的数据库分片架构
Tommy Chiu
Operational Intelligence with MongoDB Webinar
Operational Intelligence with MongoDB Webinar
MongoDB
MongoDB @ Frankfurt NoSql User Group
MongoDB @ Frankfurt NoSql User Group
Chris Harris
Become a Java GC Hero - All Day Devops
Become a Java GC Hero - All Day Devops
Tier1app
MongoDB San Francisco 2013: Hash-based Sharding in MongoDB 2.4 presented by B...
MongoDB San Francisco 2013: Hash-based Sharding in MongoDB 2.4 presented by B...
MongoDB
Nested and Parent/Child Docs in ElasticSearch
Nested and Parent/Child Docs in ElasticSearch
BeyondTrees
Introduction to Scala for Java Programmers
Introduction to Scala for Java Programmers
sbjug
Scala for the doubters
Scala for the doubters
Max Klyga
Introduction to MongoDB
Introduction to MongoDB
antoinegirbal
Apocalypse Later
Apocalypse Later
Tim Lossen
Embracing Failure
Embracing Failure
Tim Lossen
Embracing Failure
Embracing Failure
Tim Lossen
Enemy of the State?
Enemy of the State?
Tim Lossen
Beyond Devops
Beyond Devops
Tim Lossen
All Your Core Are Belong To Us
All Your Core Are Belong To Us
Tim Lossen
Podularity FTW!
Podularity FTW!
Tim Lossen
Cubic Foot Gardening
Cubic Foot Gardening
Tim Lossen
Dashboard Mania
Dashboard Mania
Tim Lossen
Event-Stream Processing with Kafka
Event-Stream Processing with Kafka
Tim Lossen
Contenu connexe
Tendances
MongoDB at GUL
MongoDB at GUL
Israel Gutiérrez
Building Apps with MongoDB
Building Apps with MongoDB
Nate Abele
Building a Scalable Inbox System with MongoDB and Java
Building a Scalable Inbox System with MongoDB and Java
antoinegirbal
Pinterest的数据库分片架构
Pinterest的数据库分片架构
Tommy Chiu
Operational Intelligence with MongoDB Webinar
Operational Intelligence with MongoDB Webinar
MongoDB
MongoDB @ Frankfurt NoSql User Group
MongoDB @ Frankfurt NoSql User Group
Chris Harris
Become a Java GC Hero - All Day Devops
Become a Java GC Hero - All Day Devops
Tier1app
MongoDB San Francisco 2013: Hash-based Sharding in MongoDB 2.4 presented by B...
MongoDB San Francisco 2013: Hash-based Sharding in MongoDB 2.4 presented by B...
MongoDB
Nested and Parent/Child Docs in ElasticSearch
Nested and Parent/Child Docs in ElasticSearch
BeyondTrees
Introduction to Scala for Java Programmers
Introduction to Scala for Java Programmers
sbjug
Scala for the doubters
Scala for the doubters
Max Klyga
Introduction to MongoDB
Introduction to MongoDB
antoinegirbal
Tendances
(12)
MongoDB at GUL
MongoDB at GUL
Building Apps with MongoDB
Building Apps with MongoDB
Building a Scalable Inbox System with MongoDB and Java
Building a Scalable Inbox System with MongoDB and Java
Pinterest的数据库分片架构
Pinterest的数据库分片架构
Operational Intelligence with MongoDB Webinar
Operational Intelligence with MongoDB Webinar
MongoDB @ Frankfurt NoSql User Group
MongoDB @ Frankfurt NoSql User Group
Become a Java GC Hero - All Day Devops
Become a Java GC Hero - All Day Devops
MongoDB San Francisco 2013: Hash-based Sharding in MongoDB 2.4 presented by B...
MongoDB San Francisco 2013: Hash-based Sharding in MongoDB 2.4 presented by B...
Nested and Parent/Child Docs in ElasticSearch
Nested and Parent/Child Docs in ElasticSearch
Introduction to Scala for Java Programmers
Introduction to Scala for Java Programmers
Scala for the doubters
Scala for the doubters
Introduction to MongoDB
Introduction to MongoDB
Plus de Tim Lossen
Apocalypse Later
Apocalypse Later
Tim Lossen
Embracing Failure
Embracing Failure
Tim Lossen
Embracing Failure
Embracing Failure
Tim Lossen
Enemy of the State?
Enemy of the State?
Tim Lossen
Beyond Devops
Beyond Devops
Tim Lossen
All Your Core Are Belong To Us
All Your Core Are Belong To Us
Tim Lossen
Podularity FTW!
Podularity FTW!
Tim Lossen
Cubic Foot Gardening
Cubic Foot Gardening
Tim Lossen
Dashboard Mania
Dashboard Mania
Tim Lossen
Event-Stream Processing with Kafka
Event-Stream Processing with Kafka
Tim Lossen
Into the Void
Into the Void
Tim Lossen
Treasure Island -- Concurrency in JRuby
Treasure Island -- Concurrency in JRuby
Tim Lossen
Dashboard Mania
Dashboard Mania
Tim Lossen
JRuby vs. Rubinius
JRuby vs. Rubinius
Tim Lossen
The Smallest Cluster in the World
The Smallest Cluster in the World
Tim Lossen
Announcing Euruko 2011
Announcing Euruko 2011
Tim Lossen
Memory: The New Disk
Memory: The New Disk
Tim Lossen
Key-Value-Stores -- The Key to Scaling?
Key-Value-Stores -- The Key to Scaling?
Tim Lossen
Cassandra vs. Redis
Cassandra vs. Redis
Tim Lossen
An Introduction to Membase
An Introduction to Membase
Tim Lossen
Plus de Tim Lossen
(20)
Apocalypse Later
Apocalypse Later
Embracing Failure
Embracing Failure
Embracing Failure
Embracing Failure
Enemy of the State?
Enemy of the State?
Beyond Devops
Beyond Devops
All Your Core Are Belong To Us
All Your Core Are Belong To Us
Podularity FTW!
Podularity FTW!
Cubic Foot Gardening
Cubic Foot Gardening
Dashboard Mania
Dashboard Mania
Event-Stream Processing with Kafka
Event-Stream Processing with Kafka
Into the Void
Into the Void
Treasure Island -- Concurrency in JRuby
Treasure Island -- Concurrency in JRuby
Dashboard Mania
Dashboard Mania
JRuby vs. Rubinius
JRuby vs. Rubinius
The Smallest Cluster in the World
The Smallest Cluster in the World
Announcing Euruko 2011
Announcing Euruko 2011
Memory: The New Disk
Memory: The New Disk
Key-Value-Stores -- The Key to Scaling?
Key-Value-Stores -- The Key to Scaling?
Cassandra vs. Redis
Cassandra vs. Redis
An Introduction to Membase
An Introduction to Membase
Dernier
AI as an Interface for Commercial Buildings
AI as an Interface for Commercial Buildings
Memoori
Powerpoint exploring the locations used in television show Time Clash
Powerpoint exploring the locations used in television show Time Clash
charlottematthew16
Dev Dives: Streamline document processing with UiPath Studio Web
Dev Dives: Streamline document processing with UiPath Studio Web
UiPathCommunity
Designing IA for AI - Information Architecture Conference 2024
Designing IA for AI - Information Architecture Conference 2024
Enterprise Knowledge
Scanning the Internet for External Cloud Exposures via SSL Certs
Scanning the Internet for External Cloud Exposures via SSL Certs
Rizwan Syed
"Federated learning: out of reach no matter how close",Oleksandr Lapshyn
"Federated learning: out of reach no matter how close",Oleksandr Lapshyn
Fwdays
Kotlin Multiplatform & Compose Multiplatform - Starter kit for pragmatics
Kotlin Multiplatform & Compose Multiplatform - Starter kit for pragmatics
carlostorres15106
Nell’iperspazio con Rocket: il Framework Web di Rust!
Nell’iperspazio con Rocket: il Framework Web di Rust!
Commit University
CloudStudio User manual (basic edition):
CloudStudio User manual (basic edition):
comworks
What's New in Teams Calling, Meetings and Devices March 2024
What's New in Teams Calling, Meetings and Devices March 2024
Stephanie Beckett
Advanced Test Driven-Development @ php[tek] 2024
Advanced Test Driven-Development @ php[tek] 2024
Scott Keck-Warren
Vertex AI Gemini Prompt Engineering Tips
Vertex AI Gemini Prompt Engineering Tips
Miki Katsuragi
New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024
New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024
BookNet Canada
Install Stable Diffusion in windows machine
Install Stable Diffusion in windows machine
Padma Pradeep
Unraveling Multimodality with Large Language Models.pdf
Unraveling Multimodality with Large Language Models.pdf
Alex Barbosa Coqueiro
WordPress Websites for Engineers: Elevate Your Brand
WordPress Websites for Engineers: Elevate Your Brand
gvaughan
Search Engine Optimization SEO PDF for 2024.pdf
Search Engine Optimization SEO PDF for 2024.pdf
RankYa
E-Vehicle_Hacking_by_Parul Sharma_null_owasp.pptx
E-Vehicle_Hacking_by_Parul Sharma_null_owasp.pptx
null - The Open Security Community
"Subclassing and Composition – A Pythonic Tour of Trade-Offs", Hynek Schlawack
"Subclassing and Composition – A Pythonic Tour of Trade-Offs", Hynek Schlawack
Fwdays
Anypoint Exchange: It’s Not Just a Repo!
Anypoint Exchange: It’s Not Just a Repo!
Manik S Magar
Dernier
(20)
AI as an Interface for Commercial Buildings
AI as an Interface for Commercial Buildings
Powerpoint exploring the locations used in television show Time Clash
Powerpoint exploring the locations used in television show Time Clash
Dev Dives: Streamline document processing with UiPath Studio Web
Dev Dives: Streamline document processing with UiPath Studio Web
Designing IA for AI - Information Architecture Conference 2024
Designing IA for AI - Information Architecture Conference 2024
Scanning the Internet for External Cloud Exposures via SSL Certs
Scanning the Internet for External Cloud Exposures via SSL Certs
"Federated learning: out of reach no matter how close",Oleksandr Lapshyn
"Federated learning: out of reach no matter how close",Oleksandr Lapshyn
Kotlin Multiplatform & Compose Multiplatform - Starter kit for pragmatics
Kotlin Multiplatform & Compose Multiplatform - Starter kit for pragmatics
Nell’iperspazio con Rocket: il Framework Web di Rust!
Nell’iperspazio con Rocket: il Framework Web di Rust!
CloudStudio User manual (basic edition):
CloudStudio User manual (basic edition):
What's New in Teams Calling, Meetings and Devices March 2024
What's New in Teams Calling, Meetings and Devices March 2024
Advanced Test Driven-Development @ php[tek] 2024
Advanced Test Driven-Development @ php[tek] 2024
Vertex AI Gemini Prompt Engineering Tips
Vertex AI Gemini Prompt Engineering Tips
New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024
New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024
Install Stable Diffusion in windows machine
Install Stable Diffusion in windows machine
Unraveling Multimodality with Large Language Models.pdf
Unraveling Multimodality with Large Language Models.pdf
WordPress Websites for Engineers: Elevate Your Brand
WordPress Websites for Engineers: Elevate Your Brand
Search Engine Optimization SEO PDF for 2024.pdf
Search Engine Optimization SEO PDF for 2024.pdf
E-Vehicle_Hacking_by_Parul Sharma_null_owasp.pptx
E-Vehicle_Hacking_by_Parul Sharma_null_owasp.pptx
"Subclassing and Composition – A Pythonic Tour of Trade-Offs", Hynek Schlawack
"Subclassing and Composition – A Pythonic Tour of Trade-Offs", Hynek Schlawack
Anypoint Exchange: It’s Not Just a Repo!
Anypoint Exchange: It’s Not Just a Repo!
Redis to the Rescue?
1.
Redis to the
Rescue? O’Reilly MySQL Conference 2011-04-13
2.
Who? • Tim Lossen
/ @tlossen • Berlin, Germany • backend developer at wooga
3.
4.
Redis Intro Case 1:
Monster World Case 2: Happy Hospital Discussion
5.
Redis Intro Case 1:
Monster World Case 2: Happy Hospital Discussion
6.
What? • key-value-store • in-memory
database • “data structure server”
7.
8.
Data Types • strings
(integers)
9.
Data Types • strings
(integers) • lists • hashes
10.
Data Types • strings
(integers) • lists • hashes • sets • sorted sets
11.
flickr.com/photos/atzu/2645776918
12.
Performance • same for
reads / writes
13.
Performance • same for
reads / writes • 50 K ops/second - regular notebook, EC2 instance
14.
Performance • same for
reads / writes • 50 K ops/second - regular notebook, EC2 instance • 200 K ops/second - intel core i7 X980 (3.33 GHz)
15.
Durability • snapshots • append-only
log
16.
Other Features • master-slave
replication • virtual memory • ...
17.
Redis Intro Case 1:
Monster World Case 2: Happy Hospital Discussion
18.
19.
Daily Active Users April
May June July Aug Sept Oct 2010
20.
Daily Active Users April
May June July Aug Sept Oct 2010
21.
Challenge • traffic growing
rapidly
22.
Challenge • traffic growing
rapidly • bottleneck: write throughput - EBS volumes pretty slow
23.
Challenge • traffic growing
rapidly • bottleneck: write throughput - EBS volumes pretty slow • MySQL already sharded - 4 x 2 = 8 shards
24.
Idea • move write-itensive
data to Redis
25.
Idea • move write-itensive
data to Redis • first candidate: inventory - integer fields - frequently changing
26.
27.
Solution • inventory =
Redis hash - atomic increment / decrement !
28.
Solution • inventory =
Redis hash - atomic increment / decrement ! • on-demand migration of users - with batch roll-up
29.
Results • quick win
- implemented in 2 weeks - 10% less load on MySQL servers
30.
Results • quick win
- implemented in 2 weeks - 10% less load on MySQL servers • decision: move over more data
31.
But ... • “honeymoon
soon over”
32.
But ... • “honeymoon
soon over” • growing memory usage (fragmentation) - servers need periodic “refresh” - replication dance
33.
Current Status • hybrid
setup - 4 MySQL master-slave pairs - 4 Redis master-slave pairs
34.
Current Status • hybrid
setup - 4 MySQL master-slave pairs - 4 Redis master-slave pairs • evaluating other alternatives - Riak, Membase
35.
Redis Intro Case 1:
Monster World Case 2: Happy Hospital Discussion
36.
37.
Challenge • expected peak
load: - 16000 concurrent users - 4000 requests/second - mostly writes
38.
“Memory is the
new Disk, Disk is the new Tape.” ⎯ Jim Gray
39.
Idea • use Redis
as main database - excellent (write) performance - virtual memory for capacity
40.
Idea • use Redis
as main database - excellent (write) performance - virtual memory for capacity • no sharding = simple operations
41.
Data Model • user
= single Redis hash - each entity stored in hash field (serialized to JSON)
42.
Data Model • user
= single Redis hash - each entity stored in hash field (serialized to JSON) • custom Ruby mapping layer (“Remodel”)
43.
44.
{“level”: 4, 1220032045
u1 “xp”: 241} u1_pets [“p7”, “p8”] {“pet_type”: p7 “Cat”} {“pet_type”: p8 “Dog”} {“level”: 1, 1234599660 u1 “xp”: 22} u1_pets [“p3”] ... ...
45.
{“level”: 4, 1220032045
u1 “xp”: 241} u1_pets [“p7”, “p8”] {“pet_type”: p7 “Cat”} {“pet_type”: p8 “Dog”} {“level”: 1, 1234599660 u1 “xp”: 22} u1_pets [“p3”] ... ...
46.
{“level”: 4, 1220032045
u1 “xp”: 241} u1_pets [“p7”, “p8”] {“pet_type”: p7 “Cat”} {“pet_type”: p8 “Dog”} {“level”: 1, 1234599660 u1 “xp”: 22} u1_pets [“p3”] ... ...
47.
{“level”: 4, 1220032045
u1 “xp”: 241} u1_pets [“p7”, “p8”] {“pet_type”: p7 “Cat”} {“pet_type”: p8 “Dog”} {“level”: 1, 1234599660 u1 “xp”: 22} u1_pets [“p3”] ... ...
48.
{“level”: 4, 1220032045
u1 “xp”: 241} u1_pets [“p7”, “p8”] {“pet_type”: p7 “Cat”} {“pet_type”: p8 “Dog”} {“level”: 1, 1234599660 u1 “xp”: 22} u1_pets [“p3”] ... ...
49.
{“level”: 4, 1220032045
u1 “xp”: 241} u1_pets [“p7”, “p8”] {“pet_type”: p7 “Cat”} {“pet_type”: p8 “Dog”} {“level”: 1, 1234599660 u1 “xp”: 22} u1_pets [“p3”] ... ...
50.
Virtual Memory • server:
24 GB RAM, 500 GB disk - can only keep “hot data” in RAM
51.
Virtual Memory • server:
24 GB RAM, 500 GB disk - can only keep “hot data” in RAM • 380 GB swap file - 50 mio. pages, 8 KB each
52.
Dec 2010: Crisis •
memory usage growing fast
53.
Dec 2010: Crisis •
memory usage growing fast • cannot take snapshots any more - cannot start new slaves
54.
Dec 2010: Crisis •
memory usage growing fast • cannot take snapshots any more - cannot start new slaves • random crashes
55.
Analysis • Redis virtual
memory not compatible with: - persistence - replication
56.
Analysis • Redis virtual
memory not compatible with: - persistence - replication • need to implement our own!
57.
Workaround • “dumper” process
- tracks active users - every 10 minutes, writes them into YAML file
58.
ruby
redis disk
59.
ruby
redis disk
60.
ruby
redis disk
61.
ruby
redis disk
62.
ruby
redis disk
63.
Workaround • in case
of Redis crash - start with empty database - restore users on demand from YAML files
64.
Real Solution • Redis
“diskstore” - keeps all data on disk - swaps data into memory as needed
65.
Real Solution • Redis
“diskstore” - keeps all data on disk - swaps data into memory as needed • under development (expected Q2)
66.
67.
Results • average response
time: 10 ms
68.
Results • average response
time: 10 ms • peak traffic: - 1500 requests/second - 15000 Redis ops/second
69.
Current Status • very
happy with setup - simple, robust, fast - easy to operate • still lots of spare capacity
70.
Redis Intro Case 1:
Monster World Case 2: Happy Hospital Discussion
71.
Advantages • order-of-magnitude performance
improvement - removes main bottleneck - enables simple architecture
72.
Disadvantages • main challenge:
durability - diskstore very promising
73.
Disadvantages • main challenge:
durability - diskstore very promising • no ad-hoc queries - think hard about data model - hybrid approach?
74.
Conclusion • Ruby +
Redis = killer combo
75.
Q&A
76.
redis.io github.com/tlossen/remodel wooga.com/jobs
Télécharger maintenant