How to manage 1 Billion+ Inserts a day into a MySQL resource -Lightning talk presented at Super Mondays October http://www.supermondays.org/2011/10/24/lightning-talks-october-31st-2011/

1. HOW TO MANAGE 1 BILLION+ INSERTS A DAY INTO A MYSQL RESOURCE
Sam Lambert, October, 2011
PHP & Continuous Data Processing, PHPNW 2011
1

2. SMITH ELECTRIC VEHICLES & TELEMETRY
• Worlds largest manufacturer of Commercial, all-electric vehicles
• Smith Link – on-board vehicle telematics system, capturing over 2500 data
points each second on the vehicle and broadcasting them over mobile network
• ~400 telemetry enabled vehicles on the road
• Worlds largest telemetry project outside of F1
• Number of inserts per day increases by around 250,000,000 per month
• Record on truck all the way to our web interface in a maximum of 30 seconds
How to manage 1 billion + inserts a day into a MySQL resource
2

3. SYSTEM ARCHITECTURE
How to manage 1 billion + inserts a day into a MySQL resource
3

4. ORIGINAL PROBLEMS
Unable to insert data quick enough
Problems with concurrently processing data
Data processing scripts taking an unacceptable amount of time
No obvious path for scaling
How to manage 1 billion + inserts a day into a MySQL resource
4

5. INNODB
Huge scaling potential
Transactional
Tuneable
Well supported/Strong community
How to manage 1 billion + inserts a day into a MySQL resource
5

6. MEASURE
Measure, measure, measure
Always measure as many metrics as possible
De ne a scope
Decide on metrics that can best determine performance improvements
How to manage 1 billion + inserts a day into a MySQL resource
6

7. TOOLS
Sysbench
Cacti (mysql-cacti-templates)
Iostat
Iotop
mysqltuner.pl
How to manage 1 billion + inserts a day into a MySQL resource
7

8. RESOURCES
Large amount of RAM (all machines 100GB+) Our latest server roll outs
have minimum of 144GB
innodb_buffer_pool_size=???
How to manage 1 billion + inserts a day into a MySQL resource
8

9. STRIKING A BALANCE
Maximum safe RAM utilisation
Make the sure buffer pool never swaps
How to manage 1 billion + inserts a day into a MySQL resource
9

10. LOAD DATA INFILE
Our application pulls messages from the queue and creates les for
importing
Have a well optimised perl application that loads the data into MySQL
Inserting data in a batch gave great improvements
How to manage 1 billion + inserts a day into a MySQL resource
10

11. OTHER SETTINGS
innodb_ le_per_table=1
innodb_ ush_log_at_trx_commit=2
innodb_ ush_method=O_DIRECT
innodb_log_ le_size=1G
How to manage 1 billion + inserts a day into a MySQL resource
11

12. SAN
Open-iSCSI settings:
node.session.cmds_max
node.session.queue_depth
How to manage 1 billion + inserts a day into a MySQL resource
12

13. MESSAGE QUEUE
Makes it much easier to plan downtime
Can comfortably tweak the live system
Can pull data in different directions for testing etc
Fast way to receive data
How to manage 1 billion + inserts a day into a MySQL resource
13

14. SUMMARY
www.samlambert.com
@isamlambert
Questions?
How to manage 1 billion + inserts a day into a MySQL resource
14