1. Building a Website To Scale
Target: 200 Million page views per day and beyond!
By Eric Pickup
eric.pickup@manwin.com Twitter: EricPickupYP
www.ManwinJobs.com

2. Contents
1. The Context
2. The Requirements
3. The Architecture
4. The Good and the Bad
www.ManwinJobs.com

3. What are we talking about?
Aug Apr Dec Feb Apr
2006 2007 2007 2008 2011
YP First 1 Million 100,000 100 million Acquired
Launched daily Uploads daily by Manwin
visitors page views
www.ManwinJobs.com

4. Traffic In Perspective
Source: Alexa.com
Alexa global rank 95
100 Gb/s – 3 full DVDs streamed every single second
www.ManwinJobs.com

5. The Context
 Written in PERL with a very complex architecture
 First few months dedicated to learning the site,
maintain it, and plan the re-write.
 Re-write started in August 2011 and was originally
planned for a delivery in mid-November.
 Actually launched at the end of January.
www.ManwinJobs.com

6. The requirements
1 Support 200 million+ daily requests
2 100% transparent to users
3 Six years of legacy data
4 Even faster site
www.ManwinJobs.com

7. The Architecture
www.ManwinJobs.com

8. The Architecture
www.ManwinJobs.com

9. The Architecture
 Fast and reliable load-balancing.
 Intelligent load distribution.
 Performs health-checks
www.ManwinJobs.com

10. The Architecture
www.ManwinJobs.com

11. The Architecture
 Reverse proxy optimized for better speed
 Reduces web and database server load
 Very rich and flexible configuration
www.ManwinJobs.com

12. The Architecture
 Cache management (what, for how long)
 Edge Side Includes (ESIs)
 Health check on Web servers
www.ManwinJobs.com

13. The Architecture
www.ManwinJobs.com

14. The Architecture
 Custom logging of page views
 Used for tasks like view counters or related videos
 Between 8GB and 15GB of logs per hour!
www.ManwinJobs.com

15. The Architecture
www.ManwinJobs.com

16. The Architecture
 High-performance HTTP server.
 PHP-FPM
 External CDNs for Static files like CSS, images and JS
www.ManwinJobs.com

17. The Architecture
www.ManwinJobs.com

18. The Architecture
 FPM hosts our framework of choice: Symfony2.
 Fast and feature rich.
 A wealth of bundles already available.
www.ManwinJobs.com

19. The Architecture
www.ManwinJobs.com

20. The Architecture
 A messaging component
 Designed for large scale deployments
 ActiveMQ to do writes (MySQL and Redis)
www.ManwinJobs.com

21. The Architecture
 Partially implemented with mitigated results.
 Too rigid for a site requiring constant changes.
 Gains not justifying Java and a separate infrastructure.
www.ManwinJobs.com

22. The Architecture
www.ManwinJobs.com

23. The Architecture
 Ability to manage pools of servers with health checks.
 We maintain 2 pools:
 Write pool with fail-over to backup-Master.
 Read pool with all servers except Master.
www.ManwinJobs.com

24. The Architecture
www.ManwinJobs.com

25. The Architecture
 Open source, advanced key-value store
 Read operations on Redis are FAST
 Primary data source
www.ManwinJobs.com

26. The Architecture
 Updated in real time as with MySQL.
 Redis Sorted Sets for all lists.
 Pipelining is VERY important for performance.
www.ManwinJobs.com

27. The Architecture
 Persistence needs tuning.
 RDB does a snapshot but is very IO extensive.
 AOF does incremental backups and is IO pain-free.
www.ManwinJobs.com

28. The Architecture
www.ManwinJobs.com

29. The Architecture
 Very normalized database since not used directly for site.
 Some tables have over 100 million rows.
 Used to populate Redis lists for new features
www.ManwinJobs.com

30. The good and the bad
 Main reasons for the delays:
 Decisions concerning some of the technologies to use.
 Learning curve for new technologies longer than expected.
 Data transfer and restructuring in MySQL and Redis
 Staffing issues.
www.ManwinJobs.com

31. The good and the bad
 Was it a success?
 Launch without any downtime
 New site about 10% faster
 Valuable expertise gained
A GOOD SUCCESS STORY WITH LESSONS LEARNED
www.ManwinJobs.com

32. Eric Pickup
eric.pickup@manwin.com Twitter: EricPickupYP
www.ManwinJobs.com