1. Evolution of a “big data”
project
Michael Peacock

2. @michaelpeacock

3. @michaelpeacock
Head Developer @groundsix

4. @michaelpeacock
Head Developer @groundsix
Author of a number of web related books

5. @michaelpeacock
Head Developer @groundsix
Author of a number of web related books
Occasional conference / user group speaker

6. Ground Six

7. Ground Six
Tech company based in the North East of
England

8. Ground Six
Tech company based in the North East of
England
Specialise in developing web and mobile
applications

9. Ground Six
Tech company based in the North East of
England
Specialise in developing web and mobile
applications
Provide investment (financial and tech) to
interesting app ideas

10. Ground Six
Tech company based in the North East of
England
Specialise in developing web and mobile
applications
Provide investment (financial and tech) to
interesting app ideas
Got an idea? Looking for investment?
www.groundsix.com

11. Whats in store

12. Whats in store
Challenges, solutions and approaches when dealing with
billions of inserts per day

13. Whats in store
Challenges, solutions and approaches when dealing with
billions of inserts per day
Processing and storing the data

14. Whats in store
Challenges, solutions and approaches when dealing with
billions of inserts per day
Processing and storing the data
Querying the data quickly

15. Whats in store
Challenges, solutions and approaches when dealing with
billions of inserts per day
Processing and storing the data
Querying the data quickly
Reporting against the data

16. Whats in store
Challenges, solutions and approaches when dealing with
billions of inserts per day
Processing and storing the data
Querying the data quickly
Reporting against the data
Keeping the application responsive

17. Whats in store
Challenges, solutions and approaches when dealing with
billions of inserts per day
Processing and storing the data
Querying the data quickly
Reporting against the data
Keeping the application responsive
Keeping the application running

18. Whats in store
Challenges, solutions and approaches when dealing with
billions of inserts per day
Processing and storing the data
Querying the data quickly
Reporting against the data
Keeping the application responsive
Keeping the application running
Legacy project, problems and code

19. Vehicle Telematics

20. Electric Vehicles: Need
for Data

21. Electric Vehicles: Need
for Data
We need to receive all of the data

22. Electric Vehicles: Need
for Data
We need to receive all of the data
We need to keep all of the data

23. Electric Vehicles: Need
for Data
We need to receive all of the data
We need to keep all of the data
We need to be able to display data in real time

24. Electric Vehicles: Need
for Data
We need to receive all of the data
We need to keep all of the data
We need to be able to display data in real time
We need to transfer large chunks of data to
customers and government departments

25. Electric Vehicles: Need
for Data
We need to receive all of the data
We need to keep all of the data
We need to be able to display data in real time
We need to transfer large chunks of data to
customers and government departments
We need to be able to calculate performance
metrics from the data

26. Some stats

27. Some stats
500 (approx) telemetry enabled vehicles
using the system

28. Some stats
500 (approx) telemetry enabled vehicles
using the system
2500 data points captured per vehicle per
second

29. Some stats
500 (approx) telemetry enabled vehicles
using the system
2500 data points captured per vehicle per
second
> 1.5 billion MySQL inserts per day

30. Some stats
500 (approx) telemetry enabled vehicles
using the system
2500 data points captured per vehicle per
second
> 1.5 billion MySQL inserts per day
Worlds largest vehicle telematics project
outside of Formula 1

31. More stats
Constant minimum of 4000 inserts per
second within the application
Peaks:
3 million inserts per second

32. Processing and storing
the data

33. Receiving continuous
data streams

34. Receiving continuous
data streams
We need to be online

35. Receiving continuous
data streams
We need to be online
We need to have capacity to process the
data

36. Receiving continuous
data streams
We need to be online
We need to have capacity to process the
data
We need to scale

39. Message Queue
Fast, secure, reliable and scalable
Hosted: they worry about the server
infrastructure and availability
We only have to process what we can

40. AMQP + PHP
php-amqplib (github.com/videlalvaro/php-
amqplib)
OR install it via composer: videlalvaro/php-amqplib
Pure PHP implementation
Handles publishing and consuming messages
from a queue

41. AMQP: Consume
// connect to the AMQP server
$connection = new AMQPConnection($host,$port,$user,$password);
// create a channel; a logical stateful link to our physical connection
$channel = $connection->channel();
// link the channel to an exchange (where messages are sent)
$channel->exchange_declare($exchange, ‘direct’);
// bind the channel to the queue
$channel->queue_bind($queue, $exchange);
// consume by sending the message to our processing callback function
$channel->basic_consume($queue, $consumerTag, false, false, false,
$callbackFunctionName);
while(count($channel->callbacks))
{
$channel->wait();
}

42. Buffers

43. Pulling in the data
Dedicated application and hardware to
consume from the Message Queue and
convert to MySQL Inserts
MySQL: LOAD DATA INFILE
Very fast
Due to high volumes of data, these “bulk
operations” only cover a few seconds of
time - still giving a live stream of data

44. Optimising MySQL
innodb_flush_method=O_DIRECT
Lets the buffer pool bypass the OS cache
InnoDB buffer pools more efficient that OS
Can have negative side effects
Improve write performance:
innodb_flush_log_at_trx_commit=2
Prevents per-commit log flushing
Query cache size (query_cache_size)
Measure your applications usage and make a judgement
Our data stream was too frequent to make use of the cache

45. Sharding (1)
Evaluate data, look for natural break points
Split the data so each data collection unit
(vehicle) had a seperate database
Gives some support for horizontal scaling
Provided the data per vehicle is a
reasonable size

46. System architecture

47. But the MQ can store data...why
do you have a problem?
Message Queue isn’t designed for storage
Messages are transferred in a compressed
form
Nature of vehicle data (CAN) means that a 16
character string is actually 4 - 64 pieces of
data

48. Sam Lambert
Solves big-data MySQL problems for
breakfast
Constantly tweaking the servers and
configuration to get more and more
performance
Pushing the capabilities of our SAN,
tweaking configs where no DBA has gone
before
www.samlambert.com
http://www.samlambert.com/2011/07/how-
to-push-your-san-with-open-iscsi_13.html
http://www.samlambert.com/
2011/07/diagnosing-and-fixing-
mysql-io.html
Twitter: @isamlambert

49. Querying the data
QUICKLY!

50. Graphs! Slow!

51. Long Running Queries
More and more vehicles came into service
Huge amount of data resulted in very slow
queries
Page load
Session locking
Slow exports
Slow backups

52. Real time information
Original database schema dictated all
information was accessed via a query, or a
separate subquery. Expensive.
Live information:
Up to 30 data points
Refreshing every 5 - 30 seconds via AJAX
Painful

53. Requests
Asynchronous requests let the page load before
the data
Number of these requests had to be monitored
Real time information used Fusion Charts
1 AJAX call per chart
10 - 30 charts per vehicle live screen
Refresh every 5 - 30 seconds

54. Requests: Optimised

55. Single entry point
Multiple entry points make it difficult to
dynamically change the time out and
memory usage of key pages, as well as
dealing with session locking issues
effectively.
Single point of entry is essential
Checkout the symfony routing component...

56. Symfony Routing
// load your routes
$locator = new FileLocator( array(__DIR__ . '/../../' ) );
$loader = new YamlFileLoader( $locator );
$loader->load('routes.yml');
$request = ( isset( $_SERVER['REQUEST_URI'] ) ) ? $_SERVER['REQUEST_URI'] : '';
$requestContext = new RequestContext( $request );
// Setup the router
$router = new RoutingRouter( new YamlFileLoader( $locator ), "routes.yml",
array('cache_dir' => null), $requestContext );
$requestURL = ( isset( $_SERVER['REQUEST_URI'] ) ) ? $_SERVER['REQUEST_URI'] : '';
$requestURL = (strlen( $requestURL ) > 1 ) ? rtrim( $requestURL, '/' ) : $requestURL;
// get the route for your request
$route = $this->router->match( $requestURL );
// act on the route

57. Sharding: split the data into
smaller buckets

58. Sharding (2)
Data is very time relevant
Only care about specific days
Don’t care about comparing data too much
Split the data so that each week had a
separate table

59. Supporting Sharding
Simple PHP function to run all queries
through. Works out the table name. Link
with a sprintf to get the full query string
/**
* Get the sharded table to use from a specific date
* @param String $date YYYY-MM-DD
* @return String
*/
public function getTableNameFromDate( $date )
{
// ASSUMPTION: todays database is ALWAYS THERE
// ASSUMPTION: You shouldn't be querying for data in the future
$date = ( $date > date( 'Y-m-d') ) ? date('Y-m-d') : $date;
$stt = strtotime( $date );
if( $date >= $this->switchOver ) {
$year = ( date( 'm', $stt ) == 01 && date( 'W', $stt ) == 52 ) ? date('Y', $stt ) - 1 : date('Y', $stt );
return 'datavalue_' . $year . '_' . date('W', $stt );
}
else {
return 'datavalue';
}
}

60. Sharding: an excuse
Alterations to the database schema
Code to support smaller buckets of data
Take advantage of needing to touch queries
and code: improve them!

61. Index Optimisation
Two sharding projects left the schema as a
frankenstien
Indexes still had data from before the first shard
(the vehicle ID)
Wasting storage space
Increasing the index size
Increasing query time
Makes the index harder to fit into memory

62. Schema Optimisation
MySQL provides a range of data-types
Varying storage implications
Does that need to be a BIGINT
Do you really need DOUBLE PRECISION when a
FLOAT will do?
Are those tables, fields or databases still required?
Perform regular schema audits

63. Query Optimisation
Run your queries through EXPLAIN
EXTENDED
Check they hit the indexes
For big queries avoid functions such as
CURDATE - this helps ensure the cache is hit

64. Reporting against the
data

65. Performance report

66. Reports & Intensive
Queries
How far did the vehicle travel today
Calculation involves looking at every single
motor speed value for the day
How much energy did the vehicle use today
Calculation involves looking at multiple
variables for every second of the day
Lookup time + calculation time

67. Group the queries
Leverage indexes
Perform related queries in succession
Then perform calculations
Catching up on a backlog of calculations and
exports?
Do a table of queries at a time
Make use of indexes

68. Save the report
Automate the queries in dead time, grouped
together nicely
Save the results in a reports table
Only a single record per vehicle per day of
performance data
Means users and management can run
aggregate and comparison queries
themselves quickly and easily

69. Enables date-range aggregation

70. Check for efficiency
savings
Initial export scripts maintained a MySQLi
connection per database (500!)
Updated to maintain one per server and
simply switch to the database in question

71. Leverage your RAM
Intensive queries might only use X% of your
RAM
Safe to run more than one report / export
at a time
Add support for multiple exports / reports
within your scripts e.g.

72. $numberOfConcurrentReportsToRun = 2;
$reportInstance = 0;
$counter = 0;
foreach( $data as $unit ) {
! if( ( $counter % $numberOfConcurrentReportsToRun ) == $reportInstance ) {
! ! $dataToProcess[] = $unit;
! }!
$counter++;
}

73. Extrapolate & Assume
Data is only stored when it changes
Known assumptions are used to extrapolate
values for all seconds of the day
Saves MySQL but costs in RAM
“Interlation”

74. Interlation
* Add an array to the interlation
public function addArray( $name, $array )
* Get the time that we first receive data in one of our arrays
public function getFirst( $field )
* Get the time that we last received data in any of our arrays
public function getLast( $field )
* Generate the interlaced array
public function generate( $keyField, $valueField )
* Beak the interlaced array down into seperate days
public function dayBreak( $interlationArray )
* Generate an interlaced array and fill for all timestamps within the range
of _first_ to _last_
public function generateAndFill( $keyField, $valueField )
* Populate the new combined array with key fields using the common field
public function populateKeysFromField( $field, $valueField=null )
http://www.michaelpeacock.co.uk/interlation-library

75. Food for thought
Gearman
Tool to schedule and run background jobs

76. Keeping the application
responsive

77. Session Locking
Some queries were still (understandably, and
acceptably) slow
Sessions would lock and AJAX scripts would
enter race conditions
User would attempt to navigate to another
page: their session with the web server
wouldn’t respond

78. Session Locking:
Resolution
Session locking caused by how PHP handles
sessions;
Session file is closed once it has finishes
executing the request
Potential solution: use another method e.g.
database
Our solution: manually close the session

79. Closing the session
session_write_close();
Caveats:
If you need to write to sessions again in
the execution cycle, you must call
session_start() again
Made problematic by the lack of template
handling

80. Live real-time data
Request consolidation helped
Each data point on the live screen was still a
separate query due to original design
constraints
Live fleet information spanned multiple
databases e.g. a map of all vehicles
belonging to a customer
Solution: caching

81. Caching with memcached
Fast, in-memory key-value store
Used to keep a copy of the most recent
data from each vehicle
$mc = new Memcache();
$mc->connect($memcacheServer, $memcachePort);
$realTimeData = $mc->get($vehicleID . ‘-’ . $dataVariable);
Failover: Moxi Memcached Proxy

82. Caching enables large range of
data to be looked up quickly

83. Legacy Project
Constraints, problems and code. Easing
deployment anxiety.

84. Source Control
Management
Initially SVN
Migrated to git
Branch per feature strategy
Automated deployment

85. Dependencies
Dependency Injection framework missing
from the application, caused problems with:
Authentication
Memcache
Handling multiple concurrent database
connections
Access control

86. Autoloading
PSR-0

87. Templates and sessions
Closing and opening sessions means you need
to know when data has been sent to the
browser
Separation of concerns and template systems
help with this

88. Database rollouts
Specific database table defines how the data should
be processed
Log database deltas
Automated process to roll out changes
Backup existing table first
DATE=`date +%H-%M-%d-%m-%y`
mysqldump -h HOST -u USER -pPASSWORD DATABASE TABLENAME > /backups/dictionary_$DATE.sql
cd /var/www/pdictionarypatcher/repo/
git pull origin master
cd src
php index.php
Rollout changes

89. private function applyNextPatch( $currentPatchID ) {
$patchToTry = ++$currentPatchID;
if( file_exists( FRAMEWORK_PATH . '../patches/' . $patchToTry . '.php' ) ) {
$sql = file_get_contents( FRAMEWORK_PATH . '../patches/' . $patchToTry . '.php' );
$this->database->multi_query( $sql );
return $this->applyNextPatch( $patchToTry );
}
else {
return $patchToTry-1;
}
}

90. The future

91. Tiered SAN hardware

92. NoSQL?
MySQL was used as a “golden hammer”
Original team of contractors who built the
system knew it
Easy to hire developers who know it
Not necessarily the best option
We had to introduce application-level
sharding for it to suite the growing needs

93. Rationalisation
Do we need all that data? Really?
At the moment: probably
In the future: probably not

94. Direct queue interaction
Types of message queue could allow our live
data to be streamed direct from a queue
We could use this infrastructure to share
the data with partners instead of providing
them regular processed exports

95. More hardware
More vehicles + New components = Need for
more storage

96. Conclusions
So you need to work with a crap-load of data?

97. PHP needs lots of
friends
PHP is a great tool for:
Displaying the data
Processing the data
Exporting the data
Binding business logic to the data
It needs friends to:
Queue the data
Insert the data
Visualise the data

98. Continually Review
Your schema & indexes
Your queries
Efficiencies in your code
Number of AJAX requests

99. Message Queue: A
safety net
Queue what you can
Lets you move data around while you process
it
Gives your hardware some breathing space

100. Code Considerations
Template engines
Dependency management
Abstraction
Autoloading
Session handling
Request management

101. Compile Data
Keep related data together
Look at storing summaries of data
Approach used by analytics companies: granularity
changes over time:
This week: per second data
Last week: Hourly summaries
Last month: Daily summaries
Last year: Monthly summaries

102. Thanks; Q+A
Michael Peacock
mkpeacock@gmail.com
@michaelpeacock
www.michaelpeacock.co.uk

103. Photo credits
flickr.com/photos/itmpa/4531956496/
flickr.com/photos/eveofdiscovery/3149008295
flickr.com/photos/gadl/89650415/
flickr.com/photos/brapps/403257780