1. The document presents a Spark application that alerts for irregular traffic patterns in real-time using machine learning and automatic data profiling of real-time traffic data from sources like GPS. 2. Key performance indicators (KPIs) for road congestion are defined and used to automatically profile traffic patterns and compare to historical patterns to detect anomalies. 3. A prediction model is used to predict event types for irregular traffic events based on the road section's data profile and spatial-temporal prediction algorithms.

1. 6/3/2017
1
Alerting Real-time Irregular Traffic Patterns in Spark
with Automatic Data-profiling and Machine Learning
Cognitive insight from spatial temporal big data 时空大数据认知与洞察
Smarter Transportation 智慧交通
1
With the growing use of real time data, it is highly desired to alert irregular traffic
patterns to both the public and traffic management administration of the city.
In this session, we present a Spark application that alerts in real-time irregular
events with a predicted event type using a pre-built prediction model for the
transportation network in a large city in China. Real-time traffic data (such as
GPS, RFID, and surveillance video) are fed to Spark via Spark Streaming and
automatically profiled into discrete traffic indicators and patterns per road
section, the result of which are categorized against historic patterns to determine
the irregularity of the incoming pattern based upon the data profile of the road
section. An event type is therefore be predicted for a found irregular event using
a pre-built spatial-temporal prediction algorithm.
概要 Abstract
2

2. 6/3/2017
2
目标 Goals
3
指标体系
KPIs for Congestion
•路段行程车速 Speed
•路段行驶时间 Time
•拥堵点位置 congestion point
•拥堵点首尾位置 positions
•拥堵队列长度 length
•路段拥堵状态 status change
•路段低速里程占比 ratio-speed
•路段低速时间占比 ratio-time
•路段拥堵区间位置 locations
历史规律
Historic Pattern
•拥堵总体趋势变化 congestion
trend
•判定早晚高峰 identify peaks
•拥堵扩散规律描述 disperse
pettern
•每日累计拥堵时长 congested time
•拥堵状态转换间隔 interval
•路段状态频率统计 statistics
•自由流车速统计 free flow speed
•区间拥堵频率 frequency
•堵点改善判断 finding status
change
趋势预测
Traffic Prediction
•未来车速预测 predict speed
•未来拥堵状态预测 predict status
•未来车流量预测 predict
throughput
•指标变化与历史趋势 historic trend
异常报警
Anomaly Alert
•历史车速描述
•准实时检测交通指标（如流量，车速）
KPIs for realtime traffic
•记录并报告异常发生 monitor and
alert traffic anomaly against
historic patterns and predicted
status.
1. Defined a new set of KPI for road congestions and their patterns of disperse
and use data profiling to automate the process
2. Uncovered the historic patterns for formation, peaking, and dispersing of
road congestions and their correlated factors.
3. Predicts traffic status using spatio-temporal prediction algorithms and alert
the traffic administration for proper preparation to ease the upcoming
congestions.
4. Monitors real time traffic events to be compared with historic patterns and
predicted status to determine whether an anomaly occurs and alerts
路径 the approach
4

3. 6/3/2017
3
指标体系 Data profiled KPI for traffic congestion
5
路段指标刻画
Data
profiled KPI
平均速度
Ave speed
低速里程占比
Ratio – low
speed
路段拥堵状态
Status
低速时间占比
Ratio-time
拥堵点位置
Position
拥堵区间长度
Duration
平均行驶时间
Ave time
历史规律-常发拥堵路段 Historic pattern mining – congested roads
6
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
435 539 848 818 153 680 146 321 236 642 453 229 737 567
拥堵频数占比
路段编号

4. 6/3/2017
4
历史规律-工作日 VS 休息日 Weekdays vs Weekend
7
0
0.05
0.1
0.15
0.2
0.25
0.3
0.35
0:05
0:45
1:35
2:35
3:20
4:30
5:30
6:20
7:05
7:45
8:25
9:05
9:45
10:25
11:05
11:45
12:25
13:05
13:45
14:25
15:05
15:45
16:25
17:05
17:45
18:25
19:05
19:45
20:25
21:05
21:45
22:40
23:30
拥堵频率占比
Road-737 Road-146
计算每日累计
拥堵时长
路段平
均速度
路段状态频率
统计
工作日、休
息日拥堵时
长相关性检
验
路段平
均速度
工作日、休息日
拥堵存在差异
Road A工作日
work-day
Road A休息日
weekend
Road B工作日
work day
Road B休息日
weekend
HS路
Road
A
Road
B
历史规律-拥堵总体趋势 Historic pattern mining – congestion trends
8
0
10
20
30
40
50
60
70
80
90
每日累计拥堵时长（分钟）
Road-453
0
10
20
30
40
50
60
每日累计拥堵时长（分钟）
Road-539
0
100
200
300
400
500
600
700
800
每日累计拥堵时长（分钟）
Road-737

5. 6/3/2017
5
• 采用15.7.1-16.1.31的数据预测
20151105（周四）的道路平均车速。
• 右图为快速路的一个路段的预测结果。
• 预测过程中还用到了天气预报数据。
YL Blvd大道
预测（中长期） – 路段平均车速Predicted speed (fast roads)
9
交通状态预测 Short term prediction: 5-30 minutes
10
0
5
10
15
20
25
30
35
路段平均速度（km/h）
Road-642 2015/8/8 平均速度
预测速度 平滑速度 原始速度
邻近法补缺失值
Time
<T
均值滤波
计算每一时刻历史速度
（区分休息日）
合并
（速度、历史速度、天气）
STP预测
未来5分钟到
30分钟速度
速度、天
气、时间戳
历史规律
未来1天，3
天，一周

6. 6/3/2017
6
THANKS