Good Stuff Happens in 1:1 Meetings: Why you need them and how to do them well
Scalability meetup-05-2013-presentation
1. How to scale your
platform when you
are not Facebook
2013 Carlos Herrera
caherrerapa@gmail.com
2. CTO - Binumi, former Head of Product and IT –
Lazada TH (Rocket Internet)
MBA and B. Computer Science
7 countries
10 years in tech. Contributed to some OSS
Product Management, Software dev, ethical
hacking, corporate governance
Coca Cola FEMSA, Infosys, among others
Who am i?
3. Be humble
“Normal” scale platforms are different.
–Customer paying vs Like a cat
–Sensitivity*
–Structured information vs Unstructured
–Concurrency
–Reach across regions
–GBs vs Several TB or PB
Why I shouldn’t do exactly as fb?
4. You scale a platform not a language*
Language selection drivers
– Problem
– Maturity
– Community
– Talent pool
Nice to experiment but focus on the
problem (yes, im talking about you
mongodb guys)
Different solutions / Creativity
7. How does it start?
Everything on the
same server
Reading the DB
all the time
JS, CSS, Images
served from your
server
8. 1. Define, measure, benchmark
b. Measure
Munin
Icinga
Statsd
New Relic $
Pingdom $
c. Benchmark
Siege
Jmeter
Sysbench
a. Baremetal vs Cloud (Amazon)
Physical vs Virtualized
Powerful vs Flexible
Normal skills vs Hard core skills
Support vs On your own
Outages?
Avoid anything with Cpanel for God’s sake
International gateway
9. Cache?
Set content in memory/drive
Faster than DB
Key = Value
TTL
Memcached (> 1 server), APC (1
server)
Other Common use: sessions
Extremely important in cloud
Live without cache
Constant hits to DB
DB easily the bottleneck
Wasting $$ you paid for memory
2. Cache system
10. 2. Cache system
a. Customer goes to your website’s homepage
b. The page requested needs to load a list of
products
c. Is the list of products in the cache by the key
“XXXXX”?
a. Yes. Retrieve from cache using key
“XXXXX”, use it and return page.
b. No. Go to the Database, perform the query
and save it in cache for later, use the info
and return the page.
Introducing
Cache
12. 4. Decouple and revisit your
web node Separation of concerns
Webnodes apart from DB
More visibility
Easier to scale
Horizontal
Vertical
Evaluate Nginx, Puma(RoR)
PHP-FPM
13. 5. Adding web nodes / load balancing
Load Balancer
Sends traffic to internal web nodes
Easier to pay. ELB*, Rackspace
No budget? Nginx, HAProxy
Location depends if you pay or you
build it*
www.yourapp.com
web01.yourapp.com
web02.yourapp.com
14. Beware of the session
management
If you don’t use sticky sessions
be careful you can end up not
remembering your customers as
the load balancer sends traffic
to the less busy
5. Adding web nodes / load balancing
15. 6. Scaling the database:
vertical
Vertical
Bigger server
RDS (downtime)
Easier
But there is a limit
16. 7. Scaling the database:
horizontal
Master / Slave
More of effort for sysadmin
Master
Important reads
Always writes
Slave
Only Reads
Consider delay
Enough for
1000RPM per
node*
17. Need more?
Separate memcache servers
Sharding
RAIDs
If you are doing heavy search add
Apache Solr
Revisit your problem
Do you need Hadoop
Can you use NoSQL (Cassandra,
Mongo)?
18. Something is not working?
Revisit your database design
Indexes anyone?
Revisit your app design
Pessimistic locks?
Lack of good algorithms?
19.
20. Bottleneck => web nodes
We added one web and db
RPM
Used historical performance data
Rackspace (no choice)
Reduce loaded elements on frontpage
Standby - Monitor APDEX
We did well. No downtimes
TV and BTS at Lazada
21.
22. 20-25% of the traffic just to one page
That page was on a small server we
controlled in TH
2 hours handling our own end of the
world
12/12/12 Campaign
23.
24. Tech communication with business
Monitoring
VPS
Static files
Cache
What could be done better?
Solution
Delay traffic / Business guys
EC2
Cloudfront
Cache and fix code
New relic
25. 1M video clips
20K+ video clips per month
Heavy search
Streaming
Fast quick preview. Faster than Animoto or
Wevideo
Rendering
45. Where do I see how the big guys do?
https://github.com/blog/530-how-we-made-
github-fast
http://nerds.airbnb.com/
http://www.facebook.com/Engineering
http://engineering.twitter.com/
http://highscalability.com/blog/2012/4/16/ins
tagram-architecture-update-whats-new-with-
instagram.html