Facebook has over 500 million active users, with half logging in every day. It processes over 4 trillion feed actions per day and caches over 2 trillion objects. Facebook has scaled to over 1 million active users per engineer, significantly more efficient than other large tech companies. To achieve this scale, Facebook relies on techniques like frequent small releases, dark launching of major changes, and shedding load during outages to maintain reliability as the site grows enormously.

2. 500 Million Active Users
0M
60M
120M
180M
240M
300M
2004 2005 2006 2007 2008 2009
500M

3. #2 property on the Internet (time on site)
100s of billions of monthly page views
>4 trillion feed actions processed per day
100s of millions of cache queries per second
Over 2 trillion objects cached
Over 500 million active users, half log in every day
100 billion photo files stored
Over 20 billion minutes spent every Day
Over 2 billion pieces of content uploaded every week

4. Move Fast
Server Scaling
Reliability

5. 500 Million Active Users
0M
60M
120M
180M
240M
300M
2004 2005 2006 2007 2008 2009
500M

6. Over 1 million active users per engineer
0
240,000
480,000
720,000
960,000
1,200,000
Facebook Google* Amazon* Microsoft*
75,000
94,000
190,000
1,100,000
* Conservative estimates based on publicly available data

7. Few external deadlines... but

8. The site can’t go down

9. Frequent small changes
Never a delay waiting for a push
Easier to isolate bugs
Major changes dark launched

12. Traditional websites
Bob
Bob’s data

13. Traditional websites
Bob
Bob’s data
Julie
Julie’s data
Dan
Dan’s data
Beth
Beth’s data
Sue
Sue’s data
Erin
Erin’s data

14. Facebook the data is interconnected
Bob Erin
Beth
Servers

17. Scale Horizontally

18. Database
Memcache
Web Server

19. Database
Memcache
Web Server

20. Database
Memcache
Web Server

21. Database
Memcache
Web Server

22. Network Incast
Memcache Memcache Memcache Memcache
Switch
PHP Client

23. Network Incast
Many Small
Get Requests
Memcache Memcache Memcache Memcache
Switch
PHP Client

24. Memcache Memcache Memcache Memcache
Switch
PHP Client
Many big
data packets
Network Incast

25. Memcache Memcache Memcache Memcache
Switch
PHP Client
Network Incast

26. Memcache Memcache Memcache Memcache
Switch
PHP Client
Network Incast

27. Memcache Memcache Memcache Memcache
Switch
PHP Client
Network Incast

28. Reliability

29. Single Points of Failure

30. Single Points of Failure
Software can be a SPOF

31. Don’t make small problems big

32. Don’t make small problems big
Don’t push problems upstream

33. Don’t make small problems big
Don’t push problems upstream
Be wary of “smart” failover

34. Don’t make small problems big
Don’t push problems upstream
Be wary of “smart” failover
Shed load when you’re in trouble

35. If you lose half of your machines
you’re doing well if you’re serving
half of your traffic

36. Measure Everything

37. p95 vs p99

39. Culture
Always do a post-mortem
Release often
Control and Responsibility

40. Facebook Platform

41. Lessons Learned
Federate everything
Keep failures contained
Measure distributions
Understand every problem
Make a person responsible

42. (c) 2009 Facebook, Inc. or its licensors. "Facebook" is a registered trademark of Facebook, Inc.. All rights reserved. 1.0