The increasing availability of mobile phones with embedded GPS devices and sensors has spurred the use of vehicle telematics in recent years. Telematics provides detailed and continuous information of a vehicle such as the location, speed, and movement. Vehicle telematics can be further linked with other spatial data to provide context to understand driving behaviors. The collection of high-frequency telematics data results in huge volumes of data that must be processed efficiently. We present a solution that uses Apache Spark to load and transform large-scaled telematics data. We then present how to use machine learning on telematics data to derive insights about driving safety.

1. Wayne Zhang, Uber
Neil Parker, Uber
Large-Scaled Telematics
Analytics in Apache
Spark
#DS3SAIS

2. Agenda
• Telematics introduction
• Eng pipeline
2

3. Telematics
3
Source: Smartphone-based Vehicle Telematics - A Ten-Year Anniversary
- Wide availability
- Cheap
- Short upgrade cycle
- Lower quality
- Measure phone motion

4. Core Pipeline
4
Preprocessing &
Transformation
Sensor Data
Collection
Vehicle
Movement
Inference
Driving Behavior
Inference

5. Phone Sensor Data
5
● GPS
○ Absolute location, velocity and time
○ Low frequency (<= 1 point per second)
● IMU
○ Relative motion of phone
■ Accelerometer: 3D linear acceleration
■ Gyroscope: 3D angular velocity
○ High frequency (>20 points per second)

6. GPS Map-Matching
6

7. Phone Re-Orientation
7

8. Long Stop Detection
8
Long Stop
Long Stop

9. Vehicle
Movement
9

10. Phone Mounting
10

11. Engineering
11
• Pipeline
– Past
– Present
• Problems millions trips per day
xTbs sensor data per day
xPbs per year
How big is
our data?

12. Data Pipeline - Past (Streaming)
12
Input
Schema Topic
(Kafka)
Transform
Samza Job
Output
Schema Topic
(Kafka)
• Realtime

13. Data Pipeline - Present (Batch)
13
Input
Hive Table
(HDFS)
Transform
Spark Job
Output
Hive Table
(HDFS)
Select logic in SparkSQL
• Flexible

14. Data Pipeline - Actuality (λ)
14
Input
Schema Topic
(Kafka)
Transform
Samza Job
Output
Schema Topic
(Kafka)
Input
Hive Table
(HDFS)
Transform
Spark Job
Output
Hive Table
(HDFS)
Business Logic
(JVM)

15. Data Pipeline - Present
15
Input
Hive Table
(HDFS)
Transform
Spark Job
Output
Hive Table
(HDFS)
Select logic in SparkSQL

16. Data Pipeline
16
Input
Hive Table
(HDFS)
Transform
Spark Job
Output
Hive Table
(HDFS)
Input
Hive Table
(HDFS)
Join logic in SparkSQL

17. Data Pipeline
17
Input
Hive Table
(HDFS)
Transform
Spark Job
Output
Hive Table
(HDFS)
Input
Hive Table
(HDFS)
Output
Hive Table
(HDFS)

18. Scheduler (Every 24hrs)
Data Pipeline
18
Input
Hive Table
(HDFS)
Transform
Spark Job
Output
Hive Table
(HDFS)
Input
Hive Table
(HDFS)
Output
Hive Table
(HDFS)

19. Data Pipeline
19
Input
Hive Table
(HDFS)
Transform
Spark Job
Output
Hive Table
(HDFS)

20. Data Pipeline
20
Input
Hive Table
(HDFS)
Transform
Spark Job
Output /
Input
Hive Table
(HDFS)
Transform
Spark Job
Output
Hive Table
(HDFS)

21. Data Pipeline - Actuality
21

22. • Data Sources
• OOM Errors
• Too many Namenodes
Eng Problems
22

23. Eng Problems - Data Sources
23
Input
Hive Table
(HDFS)
Transform
Spark Job
Output
Hive Table
(HDFS)
Input
Thrift Binaries
(S3)
Join logic in SparkSQL
Doesn’t work

24. 24
Eng Problems - Data Sources
Input
Thrift Binaries
(S3)
Transform
Spark Job
Output
Hive Table
(HDFS)
github.com/airbnb/airbnb-spark-thrift

25. 25
Aside: Encode Decode Invariant
Java Thrift
Class Instance
Spark SQL
Row

26. 26
Aside: Encode Decode Invariant
Java Thrift
Class Instance
Spark SQL
Row
Generate random data to test
(ScalaCheck Library)

27. Eng Problems - Data Sources
27
Input
Hive Table
(HDFS)
Transform
Spark Job
Output
Hive Table
(HDFS)
Join logic in SparkSQL
Works
Input
Thrift Binaries
(S3)
Transform
Spark Job
Output
Hive Table
(HDFS)

28. • If hitting OOM related issues, usually increasing
partitions help
– `spark.sql.shuffle.partitions=X
• Where x > 200 (default)
• Might also play with executor-cores and
memory settings
Eng Problems - OOM Errors
28

29. • If we have more partitions, we create more files
• Solution: Merge file after job run
– github.com/apache/parquet-mr/tree/master/parquet-tools
Eng Problems - # Namenode
29

30. Looking Towards the Future
30

31. Thank You!
(And thanks for everyone @ Uber who helped us)
31
Email us if you have any questions:
actuaryzhang@uber.com
nwparker@uber.com