Overview of understanding your stack at the KL Facebook Developer Garage

1. Your backend architecture
is what matters
Scaling your application
Colin Charles, colin@montyprogram.com
@bytebot / http://bytebot.net/blog/
KL Facebook Developer Garage, February 26 2011

3. What are you building?
• a bunch of static fbml pages + fql+ database
triggers+views?
• the next cool game zynga wants to acquire?
• needs to be database driven with proper
architecture planning

4. http://xkcd.com/327/

5. Don’t prematurely
optimise
Just remember the 7P’s:
Prior & Proper Planning Prevents Piss Poor Performance

6. Reference
Architectures
• Is there one for the Web world?
• Your choices are to:
• scale up
• scale out
• Which do you pick?

7. Scaling out
• Buying (renting) commodity hardware
• Using the cloud to expand
• Or using the cloud totally: e.g. http://
heroku.com/facebook

8. OS
• If you didn’t go opensource, you’re silly
• Tuning Linux/BSD is mandatory
• filesystem: xfs, ext3(4)
• swap is the devil
• different schedulers work better for different
tasks (web, database, etc.)
• NFS? You’d better tune that! (stopgap, scaling is
hard)

9. Web server
• Apache, lighthttpd, nginx
• They all require configuration (httpd.conf)
• Simple things like maximum connections,
worker MPM, usually go unconfigured

10. Language
• “PHP doesn’t scale.” - Cal Henderson, when
he was at Flickr.com
• Languages are not meant to scale for you
• Use bytecode caches (PHP, Python, etc.)
• Compile away -- HipHop
• Library, driver support; developer
communities

11. Databases
• are slow, period.
• partition data into shards
• tune that database

12. And that was your
basic LAMP stack

13. How do you scale
easily?
• Use caches
• Disk-based caching (cache_lite via php-
pear). RAM disks on SSDs... fast!
• In-memory caching (APC, memcached)
• Cloud-based caching (S3, MogileFS)

14. memcached
• Easy to setup and use
• Very fast over the network
• Scales, has failover, widely supported
• Centralised and shared across the site

15. S3
• Databases are good for storing relational
data, but suck for blob storage
• S3 is a file & data store, running over HTTP
• In theory, infinitely scalable
• Centralised & shared across site
• Costs money, no Malaysian POP
• See OpenStack’s Swift Object Store

16. CDN
• Outsource it
• Costs a lot of money
• Aflexi is a Malaysian company making a
pretty darn good CDN (resold via
Exabytes?)
• Out of your control but will help you scale,
scale, scale

17. Back to the database...
• Sharding
• not all data lives in one place
• hash records to partitions
• partition alphabetically? put n-users/
shard? organise by postal code?
• horizontal vs vertical partitioning

18. Horizontal vs Vertical
Partitioning
192.168.0.1 192.168.0.1
User
User
id int(10)
Better if INSERT
id int(10)
username char(15)
username char(15)
password char(15)
email char(50) 192.168.0.3 heavy and there’s
password char(15)
email char(50)
User
less frequently
192.168.0.2 id int(10)
192.168.0.2 changed data
username char(15)
password char(15)
User email char(50) UserInfo
id int(10) login datetime
username char(15) md5 varchar(32)
password char(15) guid varchar(32)
email char(50)

19. MySQL has engines
• InnoDB (XtraDB) for transactional use
• MyISAM for “data warehousing” use
• Maria in time

20. MySQL has replication!
• Simple, easy to implement (async)
• Row based replication is better than
statement based replication
• You do not need mysql cluster (ndb)
• Look at Tungsten Replicator, Galera, etc. for
other topologies (e.g. many masters)

21. Use INDEXes
• Covering index: all fields in SELECT for
specific table are contained in index
• EXPLAIN will say “Using index”

22. Monitor everything!
• Benchmarking allows tracking performance
over time
• Nagios
• MySQL (MariaDB/Percona Server)
• slow query log, extended stats in slow
query log, use EXPLAIN, microsecond
process list, userstats v2, SHOW
PROCESSLIST, etc.

23. Fulltext Search
• Don’t use the database!
• Sphinx
• SphinxSE for MariaDB
• Lucene

24. Don’t
• SELECT * FROM room WHERE room_date
BETWEEN ‘2011-02-25’ AND ‘2011-02-27’
• not have an INDEX on field being
operated on by range operator => full
table scan
• not allocate a primary key
• over-normalise (3NF is fine)

25. Keeping state
• Session data in DB
• PHP has files, doesn’t scale. DB
+Memcached goes far
• Replicate/Partition/Cache state
• Cookies can be validated by checksums and
timestamps (encryption consumes CPU)

26. General advice
• Your DB servers are not your web servers and they’re not
your load balancers
• Write non-locking code
• Don’t block loading unnecessarily
• Cache partially (esp. w/dynamic pages)
• Use UTC for time (replication across geographies?)
• Keep everything in version control
• Migrations are never recommended unless you’ve exceeded
capabilities of current solutions. Beware v2 disasters.

27. NoSQL
• MongoDB
"I don't foresee StumbleUpon ever giving up on all of
its MySQL instances. RDBMSs are just too useful. The
plan, though, is to shrink what MySQL does over
• Redis time, let MySQL do what its good at and have HBase
take over where MySQL is running up against limits
handling ever-growing write rates, table sizes, etc." -
• hBase/Hadoop Michael Stack, hbase project chair, StumbleUpon DBA
http://www.theregister.co.uk/2011/01/19/
• CouchDB
hbase_on_the_rise/
• And the 45 other solutions out there...

28. A lot of web scale tech
comes from...
• Brad Fitzpatrick
• LiveJournal infrastructure
• memcached (distributed caching, hits less
DB), MogileFS (distributed file system),
Perlbal (reverse proxy load balancer),
Gearman (remotely run code, load balanced,
in parallel)
• next: camlistore (http://camlistore.org/)

29. “Without money the site can't function. Okay, let me tell
you the difference between Facebook and everyone
else, we don't crash EVER! If those servers are down for
even a day, our entire reputation is irreversibly
destroyed! Users are fickle, Friendster has proved that.
Even a few people leaving would reverberate through
the entire userbase. The users are interconnected, that is
the whole point. College kids are online because their
friends are online, and if one domino goes, the other
dominos go, don't you get that?” -- Mark Zuckerberg
(okay, not really, Jesse Eisenberg, in The Social Network)

30. Resources
• High Performance Web Sites (Steve
Sounders)
• High Performance MySQL (Jeremy
Zawodny, Baron Schwartz, Peter Zaitsev, et
al)
• Study HyperDB (Powers wordpress.com)
• http://kb.askmonty.org/

31. Thanks/Q&A
Colin Charles, colin@montyprogram.com
@bytebot | http://bytebot.net/blog/