Scaling small apps
•Download as KEY, PDF•
1 like•496 views
In March 2010, the UK had an election with a televised leadership debate. We built an application that scaled to 2.7 million hits in 90 minutes on top of AppEngine which we built in about 3 days. See the mistakes we made and what went well.
Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Similar to Scaling small apps
Similar to Scaling small apps (20)
Recently uploaded
Recently uploaded (20)
Scaling small apps
- 1. Michael Brunton-Spall Developer Advocate guardian.co.uk @bruntonspall
- 2. Scaling small apps on AppEngine
- 3. Our coverage
- 4. Poll Charts • "Let people viewing the page vote at anytime whether they like or dislike what the party leader is saying. Oh, and lets show it with a real time graph" • Bad words here • anytime • real-time
- 5. The Design
- 6. The reality
- 7. Timeline • Thursday 8th April 2010 - Project Start • Thursday 15th April 2010 - Debate 1 - Go Live • Thursday 22th April 2010 - Debate 2 • Thursday 29th April 2010 - Final Debate
- 9. Counters incr(self, val): self.val += val
- 10. Locks are bad, Mkay! • transaction write locks • all threads are serialised
- 11. Sharded Counters incr(self, val=1): c = randomcounter c.val += val
- 12. Sharded Counters get(self): return sum(counters)
- 13. Shards are better, Mkay! • writes lock only 1 shard at a time • n simultaneous writes
- 14. On the first night
- 15. The poll itself
- 17. In Detail
- 18. Numbers • Average: 100-120 req/s • Max: 400 req/s • Total requests: 1,000,000 in 90 mins • Little cheating - 2000 requests
- 19. Request Duration
- 20. Request Duration • 1 to 8 seconds • Thread contention • Not enough shards?
- 21. Not enough shards? • Increase number of shards? • by factor of 10? • No transaction failures • Each request still takes 200ms
- 22. 200ms per incr? • Datastore write • requires disk access
- 23. Memcached? • Change architecture • atomic increment/decrement • get from memcache • cronjob 1/min • memcache => datastore • votes now take 20 ms each
- 24. The Flu
- 25. The second night
- 26. The Results?
- 27. The Results?
- 28. Some notes • Total of around 2,727,000 requests • Average of around 454 req/s • Peaked at 750 req/s
- 30. Request Duration
- 31. The problems
- 32. Errors / Second
- 33. Google AppEngine • Request limit: • x requests per day • But also... • y requests per minute • approx 1/3 reqs refused
- 34. Guardian • Serving up json via guardian domain • Heavier process than we thought • cronjob to upload to cdn 1/min
- 35. Request Duration
- 36. The solution!
- 37. The third night
- 38. The results
- 41. Some notes • Total of around 750,000 requests • Average of around 138 req/s • Peaked at around 230 req/s
- 44. £££ (or $$$) • Appengine Cost: $Negligable • plus cdn bandwidth • plus dev costs • etc...
- 45. Any Questions? Michael Brunton-Spall @bruntonspall michael.brunton-spall@guardian.co.uk
Editor's Notes
- \n
- Scalability on journalistic deadlines.\nOn Thursday 15th April 2010, there was an historic occasion, the UK’s first televised party leader debate for an election.\nAs a news organisation we wanted to not only report the news and information about the debate, but also get feedback from our readers as to what they thought of the debate.\n
- Our coverage on the website looked something like this. This was the picture that we put on the main page to encourage people to view our live blog.\n
- \n
- \n
- \n
- \n
- \n
- \n
- \n
- \n
- \n
- \n
- \n
- \n
- \n
- \n
- \n
- \n
- \n
- \n
- \n
- \n
- \n
- \n
- \n
- \n
- \n
- \n
- \n
- \n
- \n
- \n
- \n
- \n
- \n
- \n
- \n
- \n
- \n
- \n
- \n
- \n
- \n
- \n