How a little quiz achieved Olympic Glory

1. Nov 19, 2008
Scaling the Rails
How a little quiz achieved Olympic Glory
Josh Schwartzman, Newzwag

2. Get Ready For:
• Database Enhancements & Gotchas
• Hardware Basics
• Load Testing
• Rails Backends & Web Servers

3. From Humble Beginnings:

8. (on limited hardware)
30,000 concurrent users 9 million pageviews/hour

9. [question]

10. [answer]

11. Find
Your MySQL
Bottlenecks

12. Analyze Your Logs
• Look for slowest requests
• Verbose Logging
• Detailed DB logging (a small script)
Completed in 0.06763 (14 reqs/sec) | Rendering:
0.01377 (20%) |
DB: 0.00587 (8%) 2.6k, 54 rows, 7 queries |
200 OK [http://url]

13. Database Enhancements
(Reads)
• Index all of the keys you are going to use to
find your data
• Turn on debugging and check the slowest
queries
• Use EXPLAIN on your SQL statements
(Some MySQL-specific enhancements)

14. Database Enhancements
(Reads)
• Minimize your # of queries
• Use Eager Loading

15. Database Enhancements
(Writes)
• Only save the data that has changed
• Rails 2.1: 2x faster on some actions, due to:
• Partial Updates (automatic
performance)
• Dirty Tracking

16. Database Enhancements
(Writes)
• Check your model validations
• Transactions w/InnoDB

17. Database Enhancements
(Writes)
• Bulk Writes with AR-Extensions
• 8-12x faster on InnoDB
• 8-40x faster on MyISAM*
• Master/Slave Replication
• Master DB for writes with replication to
read slaves
• Distributes load to multiple machines
*40x if you don’t use validations

18. Useful Plugins
• ActiveRecord Extensions
• Bulk writes
• Dr. Nic’s magic_multi_connections
• Offload reads/writes to another DB
• Single-Write Multi-Read (SWMR)
• Our plugin to write to one DB & use
multiple read slaves

19. Hardware
• Cache everything, so lots of RAM
• Make your reads/writes fast
• RAID helps:
RAID 10 - faster writes
RAID 5 - faster reads

20. Hardware
• Minimize hits to your Rails Stack
• Cache images or use a fast web server
• Move all your routes.rb routing logic to
dedicated hardware (load balancer)

21. Hardware
• Keep App & DB on separate servers
• Infrastructure specific to it’s purpose
• Easy to isolate & debug bottlenecks:
Graph from Cacti

22. Load Testing
• Test the path a user will take
• One Tool: JMeter
• Session and full hardware stack testing
• jmeter-server
• Distributed testing
• Test from any computer @ ~1500
threads per box

24. Load Testing
• You’ll always find new problems under
load
• We discovered issues with lots of
new users: checking login twice
• Simple queries (especially joins)
became slow
• Tweaking how often we wrote scores
to the database, and how many at a
time

25. Load Testing
• The type and number of web servers*
is critical. Quick comparison:
Mongrel Ebb Thin
•Stable •Some bugs •Almost as stable
•RAM & CPU heavy •RAM & CPU light as Mongrel (it uses
•Slowest per action •Fastest per Mongrel’s parser)
•Lightest hardware
action (uses C for
heavy lifting) load
•Almost as fast as
Ebb
*Yeah, you still need a web server like Lighttpd or Nginx to handle your static files and proxy to your Rails backends

26. Load Testing
Using 18 Mongrels
# of questions
1 2 3 4 5 6 7 8 9 10
minutes
Gruff graphs from our
custom user actions

27. Load Testing
Using 18 Ebbs
# of questions
1 2 3 4 5 6 7 8 9 10
minutes
949 questions / minute: 40% faster than mongrel!

28. When You Need a
Little Help
• After all of that performance work...
• Servers were crashing spontaneously
• Nothing was apparent in the logs
• Sleepless nights...

30. We Found Stability in God
Simple monitoring for Rails apps:

31. What Was the
Problem?
• Post requests on older browsers (Firefox
1.5, Navigator 7)
• Killed our backends, and as each failed, it
passed on the request to another server
• Solution: switch from Nginx to Lighttpd

32. A Summary
• 200 (scaling) 5000 simultaneous users
a 25x performance gain
• Achieved by adding plugins, custom
Memcached methods, optimizing the DB,
changing to ebbs, and numerous other code
tweaks (and a slight hardware upgrade)

34. Any Questions:
Josh Schwartzman
Lead Developer, Newzwag
josh@newzwag.com

35. Resources
•Active Record Extensions:
http://www.continuousthinking.com/are/activerecord-extensions-0-0-5
•Detailed DB Logging: http://gurge.com/blog/2006/11/09/rails-sql-logging-improvements/
•Ebb: http://ebb.rubyforge.org/
•God (Monitoring): http://god.rubyforge.org/
•Gruff Graphs: http://nubyonrails.com/pages/gruff
•Jmeter: http://jakarta.apache.org/jmeter/
•Magic Multi Connections: http://magicmodels.rubyforge.org/magic_multi_connections/
•Seattle.rb’s Memcache Client: http://github.com/fiveruns/memcache-client/tree/master
•Thin: http://code.macournoyer.com/thin/