1. Driver
Drowsiness and
Distraction
Detection by
Sensor Fusion
D4SF
Johan Karlsson, Autoliv
Transportforum 2011
Fordonsstrategisk Forskning och Innovation FFI – D4SF
ALR-JKAR/Jan2011/Transportforum - 1 Copyright Autoliv Inc., All Rights Reserved
2. Overview
Background
Goals
Drowsiness detection
(Distraction detection)
Method
Data collection
Training/optimization of classifier
Sensor fusion
Results
Reference – ground truth
Improvement by (f)using parallel detectors
Fordonsstrategisk Forskning och Innovation FFI – D4SF
ALR-JKAR/Jan2011/Transportforum - 2 Copyright Autoliv Inc., All Rights Reserved
3. Driver Drowsiness detection
Drowsy driving is a road safety problem
- drowsiness contributing in 10-30% of accidents (Anund & Patten 2010)
What can be done?
Commercial fleet traffic
Fatigue Risk Management
Work time regulation
Detection and warning
Privately owned vehicles
Detection and warning
Detection?
Detection systems offered as option from several OEMs
So far, performance is far from ideal...
Fordonsstrategisk Forskning och Innovation FFI – D4SF
ALR-JKAR/Jan2011/Transportforum - 3 Copyright Autoliv Inc., All Rights Reserved
4. Target and Goals
Sensitivity
Different indicators exist Specificity Availability
- ’Physiology’ A
Indicator measures+ blink duration etc.
- – +
- Driving performance measures - lane keeping measures
- EnvironmentBmeasures - time of day, traffic, road type
Indicator –
(previous sleep possible in commercial fleet vehicles??)
+ +
Indicator C + + –
Various indicators have different strengths and weaknesses
Improve performance by fusing data from multiple indicators+
Fusion ++ ++ +
The fusion algorithm shall show an improvement in:
- Overall performance
- Reduced number of faulty detections
- Increased number of correct detections
Fordonsstrategisk Forskning och Innovation FFI – D4SF
ALR-JKAR/Jan2011/Transportforum - 4 Copyright Autoliv Inc., All Rights Reserved
5. Data collection
On-road tests were conducted with
Data collection governmental approval (N2007/5326/TR)
and ethical approval by Regional ethics
Relevant vehicle data approval board (EPN 142-07 T34-09).
Speed, lane position, SW angle, pedals etc.
Video based gaze direction, eyelid opening, head pos
KSS value every 5 minute
EEG, EOG and EMG
Video recordings (road scenery and cabin)
In total: 43 drivers have completed 3 drives each
Procedure: Each driver drove three times during one day
(day, evening and night). Trip duration 80-90 minutes
Fordonsstrategisk Forskning och Innovation FFI – D4SF
ALR-JKAR/Jan2011/Transportforum - 5 Copyright Autoliv Inc., All Rights Reserved
6. Test Route
Road RV34
Mostly 9 m width
Driving lane width 3,75 m
Speed limit - mostly 90 km/h
Numbers on map are Yearly day traffic
volume in January 2002
We know of only a few similar studies
performed on public roads
90 minute driving,
approx 115 km distance
Rested safety driver –
dual command
Fordonsstrategisk Forskning och Innovation FFI – D4SF
ALR-JKAR/Jan2011/Transportforum - 6 Copyright Autoliv Inc., All Rights Reserved
7. Ground Truth – KSS
KSS Description in Swedish Verbal description
1 extremt pigg extremely alert
2 mycket pigg very alert
3 pigg alert
4 ganska pigg rather alert
5 varken pigg eller sömnig neither alert nor sleepy
6 första tecknen på sömnighet some signs of sleepiness
7 sömnig, ej ansträngande vara vaken sleepy, but no effort to keep alert
8 sömnig, viss ansträngning vara vaken sleepy, some effort to keep alert
9 mycket sömnig, ansträngande vara vaken, very sleepy, great effort to keep alert,
kämpar mot sömnen fighting sleep
+ Simple to collect
+ Simple to understand – immediately ready for analysis
- Training needed for participants
- Some offset for inexperienced participants?
Fordonsstrategisk Forskning och Innovation FFI – D4SF
ALR-JKAR/Jan2011/Transportforum - 7 Copyright Autoliv Inc., All Rights Reserved
8. Example indicators of driver sleepiness
Closing Opening Open
200 ms Closed
Blink duration (AS):
Amplitude
400 ms
Closed
Mean blink duration
Short Blink Long Blink
GVI (Sandberg 2008)
Lane keeping variability (Lane): G=
1 N
∑ w(zi ) | zi |k
N i=1
Variability in Steering wheel position or
zi = xi − (δ x + (1 − δ ) p)
Lane Position. e.g. using Generic
cL cR
Variability Indicator (Sandberg 2008) . w( z) = −α L ( z −β L )
+ −α R ( z −β R )
1+ e 1+ e
Time-of-day (TPM):
Expected drowsiness with regard to time
of day (circadian rythm)
* Each indicators has several parameters
that needs to be tuned for optimal
performance
Fordonsstrategisk Forskning och Innovation FFI – D4SF
ALR-JKAR/Jan2011/Transportforum - 8 Copyright Autoliv Inc., All Rights Reserved
9. Video examples
Video examples
Fordonsstrategisk Forskning och Innovation FFI – D4SF
ALR-JKAR/Jan2011/Transportforum - 9 Copyright Autoliv Inc., All Rights Reserved
10. Sensor fusion
SVM (Support Vector Machine):
Machine learning method using data from field tests to calculate “best fit” function
between indicator values and ground truth (KSS rating scale)
Indicator parameters optimized simultaneously with training of SVM
Data sets for SVM training and validation are from separate drivers.
Thus, validation is done on truly “never-before-seen” data.
Drowsy data
Goal:
Find the maximum margin hyperplane
Indicator B
Alert data
Indicator A
Fordonsstrategisk Forskning och Innovation FFI – D4SF
ALR-JKAR/Jan2011/Transportforum - 10 Copyright Autoliv Inc., All Rights Reserved
11. Evaluation Method
Assuming a binary classification,
A
alert or =
sensitivitydrowsy
A+C Ground truth
Performance is the mean value of
D Non-
specificity =
sensitivity and specificity Drowsy
Performance + D
Drowsy
B is related to the
Algorithm output
A B
sensitivity + specificity
proportion of the time where the Detect
performancis =
algorithm e correct
(hit) (false hit)
2 Non- C D
Detect (miss) (correct reject)
KSS = ground truth
Ground truth cutoff Sum A+C B+D
Binary Algo output
KSS
Time
Fordonsstrategisk Forskning och Innovation FFI – D4SF
ALR-JKAR/Jan2011/Transportforum - 13 Copyright Autoliv Inc., All Rights Reserved
12. Example of results from sensor fusion
Model Fitness Sens Spec
Blink 0.66 (0.64) 0.36 (0.32) 0.96 (0.95)
Blink + Circadian 0.80 (0.83) 0.77 (0.79) 0.83 (0.87)
Blink + Lane + Circ. 0.80 (0.78) 0.68 (0.68) 0.92 (0.88)
Blink + Steer + Circ. 0.80 (0.85) 0.76 (0.81) 0.84 (0.89)
First figure is training data performance
second figure is test data performance Decision every 1 minute
KSS >= 7 drowsy
KSS < 7 alert
Fordonsstrategisk Forskning och Innovation FFI – D4SF
ALR-JKAR/Jan2011/Transportforum - 14 Copyright Autoliv Inc., All Rights Reserved
13. Fulfillment of goals
The fusion algorithm shall show an improvement in:
- Improved performance true
- Increased number of correct detections true
- Reduced number of faulty detections (?)
Clearly improved overall performance
– Minor differences between different combinations
Fordonsstrategisk Forskning och Innovation FFI – D4SF
ALR-JKAR/Jan2011/Transportforum - 15 Copyright Autoliv Inc., All Rights Reserved
14. Summary
Controlled experiment on public roads
43 drivers so far
What is ideal performance?
Method developed with focus on mathematical performance
Most important goal is to have relevant warnings
More data is needed:
Different road types
Different conditions (weather, drive duration etc.)
Different driver types (age, cultural differences etc.)
Fordonsstrategisk Forskning och Innovation FFI – D4SF
ALR-JKAR/Jan2011/Transportforum - 16 Copyright Autoliv Inc., All Rights Reserved
15. Thank you for you attention!
Fordonsstrategisk Forskning och Innovation FFI – D4SF
ALR-JKAR/Jan2011/Transportforum - 17 Copyright Autoliv Inc., All Rights Reserved