1. Black Friday and Cyber Monday:
Best Practices for Your E-
Commerce Database
Tim Vaillancourt
Sr. Technical Operations Architect
@ Percona

2. Agenda
●Synchronous versus Asynchronous Applications
●Scaling a Synchronous/Latency-sensitive Application
●Scaling an Asynchronous Application
●Efficient Usage of Data at Scale
Secondary/Slave Hosts
Caching
Queuing
●Efficient Usage of Data at Scale
Moving Expensive Work
Caching Techniques
Counters and In-memory Stores
Connection Pooling

3. Agenda
●Scaling Out (Horizontal) Tricks
Pre-Sharding
Kill Switches
Limits and Graphs
●Scaling with Hardware (Vertical Scaling)
●Testing Performance and Capacity
●Knowing Your Application and Questions to Ask at Development
Time
●Questions

4. About Me
●Started at Percona in January 2016
●Experience
Web Publishing
Big-scale LAMP-based Websites
Ecommerce
Large Inventory SaaS
Gaming
DevOps
50-100 Microservices
5-7+ x Massive Launches / Year
Design, launch and maintain apps

5. About Me
DBA at EA DICE
2 x New Titles
5+ x Legacy Titles
Technologies
MySQL
MongoDB
Cassandra
Redis and Memcached
RabbitMQ, Kafka and ActiveMQ
Solr and Elasticsearch
(Sort of) AWS, HDFS, HBase, Postgres, etc…

6. Services
Monolith
One application that does everything
Example: Chrome, MySQL, huge Python app
Microservice
Different purposes, pain points, SLA apps are discreet services
Often easier to scale/troubleshoot
Reduces risk of outage
Example: frontend PHP app, messaging app, encoding app, etc
In Practice
Both can be scaled up and down with the right features
Microservices offer more flexibility
Monolith services bring problems at scale

7. Application Operations
Synchronous
Blocking operation until success or failure
Slower requests
Example: a file uploading app
Asynchronous
Request and response are separated
Fast response time back to user/application
Example: a social media site
Slow Operations
Can cause pileups in a tiered system

8. Applications
Synchronous
Pros: less code, always the right answer
Cons: blocking operations and poorer efficiency
Example: a file uploading app
Latency/Integrity Sensitive
Pros: always the right answer
Cons: less scalability tricks available
Example: a stock trading app that cannot accept “slave lag”
Asynchronous
Pros: light operations and more scalability
Cons: eventual consistency (and sometimes more code)
Example: a social media site

9. Types of Data Designs
Decentralised
Data is duplicated in several places
Pros: lighter to read, decreased locking, easy to shard
Cons: increased storage space, extra duplication effort
Centralised
Data is kept in one (or few) places and referenced
Pros: less storage, one source-of-truth
Cons: locking, inefficiencies, sharding issues

10. Balancing Request Impact
Read-focused Apps
Benefit from
Values pre-computed at write/change-time
Indices and/or few “scans” for data
No/few JOINs/operations to get result
Write-focused Apps
Benefit from
No pre-computing of values (compute at read-time)
No/few indices to update
Insert/Append > Update
Reads: compute read summaries with replicas, add indices to
secondaries only, etc

11. Event Metadata
Example: “UserX has the new top score!”
Without Queue example
Update Top Score in Database(s)
Send Email to Friends
Post to Facebook Page
Update cache
...
With Queue example
Add event to queue ‘topscore’
Apps read queue
Queuing Updates

12. Queuing Updates
Update Buffering
Scenario: there is a high rate of updates to buffer
Queue-based example
App adds to update buffer (queue)
Worker app works from the bottom of buffer
Queue Operational Benefits
Spikes in traffic
Backend downtime
Communication bus

13. Scaling Sync./Latency-Sensitive Apps
Rethink the Flow Using Async
Use lots of database RAM
Shard the database
Reduce impact of request flow
Apache Cassandra
Synchronous
Very write optimized
Percona XtraDB Cluster, NDB
Use memory-based storage
Queue persistence to database

14. Expensive DB Work
Focus on lightweight user-facing operations
Move aggregations/summaries/reporting to background
Use replicas for expensive jobs
Avoid or reduce (maybe cache) “JOINs”
Enable and monitor metrics
MySQL
log_queries_not_using_indexes
MongoDB
Enable operationProfiling
Review metrics and improve!
Percona Monitoring and Management
Efficient Usage of Data at Scale

15. Efficient Usage of Data at Scale
Caching / In-Memory Stores
Alleviates load from database
Very fast lookups
Low connection overhead
MySQL connection buffers: ~1MB+
MongoDB connection buffers: ~1MB
Redis or Memcache connection buffers: 0-limit/infinity**
Server-Side
Hit/Miss Caching
If something is not in the cache: find + add it. TTL expiry
Inline/Preemptive Caching
Update/Delete cache data at change time/preemptively

16. Efficient Usage of Data at Scale
Caching / In-Memory Stores (continued)
Client-Side
Cache client data in the client app/browser/etc
In-memory Stores
Memcached
Redis
Percona Server for MongoDB with Memory Engine :)
Use TTLs to trim data

17. Efficient Usage of Data at Scale
Storing Numerical Counters and Stats
Offload to in-memory stores
Incremented/decremented counters
Aggregations, summaries, counts
Count-style Queries to Counters
Increment counter at request/change time
Read counter value at read-request time
Or, try to use an index

18. Efficient Usage of Data at Scale
Connection Pooling
Removes 3-way TCP “handshake” from request (more w/SSL)
Reduces threading overhead on databases
Proxies on App server localhost/loopback
Reduces 1 x TCP ‘hop’, ie: faster connect time
Can create a LOT of DB connections with many app servers

19. Efficient Usage of Data at Scale
Connection Pooling (continued)
MySQL Proxies
ProxySQL
HAProxy
Maxscale
Others…
MongoDB Proxies
Mongos (sharding) process
Proxy-on-Localhost or direct is fastest

20. Virtualization
Pretends to be a real computer from BIOS up
OS + Software run under a hypervisor layer
Pros
Full hardware-level emulation, eg: CentOS, Redhat, Win 10
Automation of platform (sometimes)
Cons
Emulation overhead
Slow boot-up time
Lots of OSs to update
Virtualization, Containers, etc

21. Virtualization, Containers, etc
Containers (cgroups, jails)
Several can run inside a single operating system and kernel
Offers controls to limit resources like RAM, CPU time, etc
Pros
Low overhead
Container creation is very fast

22. Virtualization, Containers, etc
Mesos, Kubernetes, etc
Make a lot of servers distribute work, containers, etc
Apache Mesos: “Distributed systems kernel”
Agent on every host and manager servers give out work
Kubernetes

23. Virtualization, Containers, etc
Many Processes per Host
Run un-related processes on hosts
Add/remove from load balancers
Not advised for disk-bound or high-bandwidth apps

24. Scaling Out Tricks
Sharding
Techniques
Modulus
Even distribution of keys
Hard to reshape data
Map-based
1-to-1 shard mapping using another table, config, etc
Easy to reshape data
Launch with many shards in advance
1-4 MySQL/MongoDB Instance/host
1 MySQL/MongoDB Instance/host, 4 x databases as shards
1 MySQL/MongoDB Instance/host, small hardware

25. Scaling Out Tricks
Sharding
Modulus: Mapping:

26. Scaling Out Tricks
Hardware
Have a strategy to add/remove capacity quickly
Cloud Instances
Mesos/Kubernetes
Automation
Use cheap application servers for in-memory stores and apps
Launch with lots of RAM, scale down post-launch

27. Scaling Out Tricks
Elasticity
Ensure there is a way to add/remove hosts, examples:
Load Balancers
Good health-checks are important
Application Configs
File
Database
Zookeeper

28. Scaling Out Tricks
At Launch...
Scale-out
Keep spare servers online, partially configured
Launch with extra database replicas (slave/secondary)
Monitor usage and remove extra hardware post-launch
Monitor and adjust capacity
Scale-up
Launch with lots of RAM
Traffic Control
Launch one region at a time
Launch with rate limits

29. Scaling Out Tricks
Application “Kill switches”
A switch to disable certain app features/functions
Useful when there is:
Too much traffic/scale-up
DDoS
A maintenance

30. Scaling Out Tricks
Limiting Graph Structures
“Friends” / ”Followers” features are often graphs
If Katy Perry or Barack Obama used your “friends” feature…
Limit the size of graphs, or queue events for fan-out updating

31. Scaling Out Tricks
Batching and Parallel Work
Do large queries in parallel
Modern CPUs have many cores (2, 4, 8+)
1 connection = 1 thread = 1 CPU core
Batch inserts/updates
1 x update with 1000 items > 1000 x updates with 1 item

32. Scaling Up Tricks
Test provider turn-around time on hardware upgrading
Test application performance on improved hardware in advance
Scale up only resources needed

33. Databases
General
Monitoring/reviewing slow queries reduces most inefficiencies
More memory will reduce disk requests
SSDs will reduce disk request time
Proper database and kernel tunings will help further
Linux has very inefficient defaults!
Try to use real local-disks, not EBS, NFS, etc
Queries
Don’t try to make MySQL/MongoDB a queue or search engine!
Decentralizing data and pre-computing answers for reads will take
you far
The best query is no query (cache)

34. Testing Performance and Capacity
General
Try to emulate the real user traffic
Add micro-pauses to simulate reality
Cloud-based providers are great for running load generation
Applications
Component testing
Test the max volume of each component on a single host
Test the max volume of each component on many hosts
Calculate host scalability, ie: “+1 host = +80% more traffic”
Feature capacity
Test the impact of each feature if not separate

35. Testing Performance and Capacity
Databases
Replay real user traffic on real backups
Load test tools: Linkbench, Sysbench, TPCC, JMeter, etc
Single feature/query testing
Understand host capacity per feature, eg: “2000 user login
queries/sec per db replica”
Know your slowest query!

36. Development-time Questions
General
What does the app do?
If I break X, what happens?
Are connections to data stores “pooled”?
Replicas
Can the app use replicas (with possible lag)?
Tip: start early, deploy replication from the start
Can we Add/Remove replicas without disruption?
Sharding
Can the app understand shards/partitions?
How is data balanced post-sharding?
Are there cross-shard references?

37. Development-time Questions
Caching
What data can be cached?
Will an change be read immediately?
Can we pre-cache this change?
When should the cache delete an item?
Can we set TTLs on our keys?
How do we add/remove cache servers easily?

38. Knowing Your App
If you see…
The app is write heavy
Remove overhead from immediate write path
Batch writes if possible
The app is read heavy
Reduce scans/operations from the read path (index, etc)
Add as many replicas (slave/secondary) as needed
The app queries for counts often, ie: # of items, friends, etc
Move count-queries to incremented in-memory counters
Or, create an index for the count query
The app uses references or joins often
Consider decentralising the data (with fan-out updates)

39. Themes
Make all features, apps, databases elastic
Request Flow
Make the heavy workload easy / make the light workload hard
Move graph updates to background (queues, async, etc)
Move ‘counts’ to counters
Caching
Cheaper/faster to access than DB
Try to cache before anyone reads data
Queues
Great for replicating events while simplifying update
Great for batching changes
Monitor everything! Try Percona Monitoring and Management!

40. Join us at Percona Live Europe
When: October 3-5, 2016
Where: Amsterdam, Netherlands
The Percona Live Open Source Database Conference is a great event for users of any level
using open source database technologies.
Get briefed on the hottest topics
Learn about building and maintaining high-performing deployments
Listen to technical experts and top industry leaders
Use promo code “WebinarPLAM16” and receive €15 off the current registration price!
Sponsorship opportunities available as well here.

41. Questions?

42. Thanks for joining!
Be sure to checkout
the Percona Blog for more
technical blogs and topics!