In March 2010, the UK had an election with a televised leadership debate. We built an application that scaled to 2.7 million hits in 90 minutes on top of AppEngine which we built in about 3 days. See the mistakes we made and what went well.

1. Michael Brunton-Spall
Developer Advocate
guardian.co.uk
@bruntonspall

2. Scaling small apps on AppEngine

3. Our coverage

4. Poll Charts
• "Let people viewing the page vote at
anytime whether they like or dislike what
the party leader is saying. Oh, and lets
show it with a real time graph"
• Bad words here
• anytime
• real-time

5. The Design

6. The reality

7. Timeline
• Thursday 8th April 2010 - Project Start
• Thursday 15th April 2010 - Debate 1 - Go
Live
• Thursday 22th April 2010 - Debate 2
• Thursday 29th April 2010 - Final Debate

8. Counters
get(self):
return self.val

9. Counters
incr(self, val):
self.val += val

10. Locks are bad, Mkay!
• transaction write locks
• all threads are serialised

11. Sharded
Counters
incr(self, val=1):
c = randomcounter
c.val += val

12. Sharded
Counters
get(self):
return sum(counters)

13. Shards are better, Mkay!
• writes lock only 1 shard at a time
• n simultaneous writes

14. On the first night

15. The poll itself

16. Requests Per Second

17. In Detail

18. Numbers
• Average: 100-120 req/s
• Max: 400 req/s
• Total requests: 1,000,000 in 90 mins
• Little cheating - 2000 requests

19. Request Duration

20. Request Duration
• 1 to 8 seconds
• Thread contention
• Not enough shards?

21. Not enough shards?
• Increase number of shards?
• by factor of 10?
• No transaction failures
• Each request still takes 200ms

22. 200ms per incr?
• Datastore write
• requires disk access

23. Memcached?
• Change architecture
• atomic increment/decrement
• get from memcache
• cronjob 1/min
• memcache => datastore
• votes now take 20 ms each

24. The Flu

25. The second night

26. The Results?

27. The Results?

28. Some notes
• Total of around 2,727,000 requests
• Average of around 454 req/s
• Peaked at 750 req/s

29. Requests per Second

30. Request Duration

31. The problems

32. Errors / Second

33. Google AppEngine
• Request limit:
• x requests per day
• But also...
• y requests per minute
• approx 1/3 reqs refused

34. Guardian
• Serving up json via guardian domain
• Heavier process than we thought
• cronjob to upload to cdn 1/min

35. Request Duration

36. The solution!

37. The third night

38. The results

39. Requests per second

41. Some notes
• Total of around 750,000 requests
• Average of around 138 req/s
• Peaked at around 230 req/s

44. £££ (or $$$)
• Appengine Cost: $Negligable
• plus cdn bandwidth
• plus dev costs
• etc...

45. Any Questions?
Michael Brunton-Spall
@bruntonspall
michael.brunton-spall@guardian.co.uk