2. What we develop
Frontend products
-Collaborative platforms, ‘forum-like’, white-labeled, for
big accounts
(a kind of GetSatisfaction / UserVoice, but.. Better ;) )
Backend product
-SocialAPIs client
-(a kind of TweetDeck, but for multiple channels, and
designed for teams/multiple users per account)
Copyright Dimelo SA www.dimelo.com
3. Stats on Dynamic requests
- on HTML/XHTML dynamic requests (we don’t count CSS, JS, images, etc)
20 req/s
Web
Peaks of 40 req/s
10 req/s APIs
Peaks of 25 req/s
Copyright Dimelo SA www.dimelo.com
4. Problem
Medium response time ~ 400 ms
WTF??
Copyright Dimelo SA www.dimelo.com
6. Finding the problem(s)
- DB queries (unoptimized queries, n+1)
- External services (EG external RSS feeds)
- Lots of partials, low or inexistent cache
- Slow session storage
- Memory issues
- Unoptimized GC
- Memory leaks
Copyright Dimelo SA www.dimelo.com
7. Memory / GC issues
- How do I know it’s a memory/GC issue?
- Server logs
- Profilers (memprof)
- Monitorization services (newrelic)
Copyright Dimelo SA www.dimelo.com
8. Ruby Garbage Collector
- Conservative
- Stop-the-world
- Mark-and-sweep process
Copyright Dimelo SA www.dimelo.com
9. Ruby default GC settings
RUBY_HEAP_MIN_SLOTS=10000
-Initial number of heap slots. It also represents the
minimum number of slots, at all times
-a new rails app boots up with almost 500k objects
on the heap(mostly nodes)
RUBY_HEAP_SLOTS_INCREMENT=10000
-The number of new slots to allocate when all initial
slots are used
RUBY_HEAP_SLOTS_GROWTH_FACTOR=1.8
-Next time Ruby needs new heap slots it will use a
multiplicator
-New stab is almost double sized
Copyright Dimelo SA www.dimelo.com
10. Ruby default GC settings (2)
RUBY_GC_MALLOC_LIMIT=8000000
-force GC after 8 MB malloc/realloc
-High traffic Rails servers can easily allocate more
than 8 MB per request
RUBY_HEAP_FREE_MIN=4096
-The number of free slots that should be present
after GC finishes running. If there are fewer slots
than those defined it will allocate new ones
Copyright Dimelo SA www.dimelo.com
11. Find your numbers
- Start with enough memory to hold Rails (Ruby’s
default is practically nothing)
- Increase it linearly if you need more (Ruby’s
default is exponential increase)
- Only garbage-collect every XX million malloc
calls (Ruby’s default is way too small)
- Benchmark, benchmark, benchmark..
Copyright Dimelo SA www.dimelo.com
14. Conclusions
- use REE on prod (or Ruby 1.9)
- measure, measure, measure (don’t forget to
GC.enable_stats)
- optimise what matters
- And yes, 5 lines of code can make a difference ;)
Copyright Dimelo SA www.dimelo.com
15. Thank you!
• Questions?
Copyright Dimelo SA www.dimelo.com
