Sensors are everywhere, and context detection can improve almost any service, from media recommendations and advertising, to mobility management and mHealth. These two use cases show what mobile sensor data and machine learning can do to help make technologies more 'smart' and context-aware.
Presented at Strata Hadoop Barcelona, Internet of Things track, November 2014
2. “ Sense, understand and predict the context, behavior
and mood of your mobile audience ”
3. Argus has created a platform that enables
you to build context-aware solutions
4. Turning sensor data into behavioral, context,
and emotional awareness
ARGUS MOBILE SENSING
SD
K
ARGUS
PLATFORM
…
LVL 3
PROFILE
S
LVL 2
MOMENTS
LVL 1
EVENTS
5. Mobile Profiling
EVENTS
Sense and interpret the
contextual cues of a mobile user
MOMENTS
Uncover habits and predict
human behavior
PROFILES
Learn about the ever-changing
personalities of a
mobile user
LAYER 3
ACTIVENESS
DRIVING STYLE BEHAVIOUR
LAYER 2
ARRIVING AT HOME, WORK, ..
LAYER 1
…
WAKING UP …
…
SOCIALNESS
SLEEPING
IN A MEETING
DRIVING (CAR) TRAIN SUBWAY
WALKING RUNNING
BUS
SITTING STANDING
TRAM MOTORCYCLE
AIRPLANE BIKING
BUSY
BORED TIRED
LOUD ENVIRONMENT
ALONE
HOME
WORK
COMPANY
COMMUTING TYPE
CHATTY
CALM
DRIVING
BEHAVIOUR
FOR EXAMPLE PROFILES
7. ARGUS
YOU @ ARGUS
We are looking to expand our machine learning and data
research department. Email: vincent.spruyt@arguslabs.com, if
you..
1. Want to work with state-of-the-art machine learning
2. Find the use cases I’ll present on music and mobility
fascinating and want to work on this with us.
3. Most definitely, if you feel you can improve upon what we did
in these two use cases, or can suggest a better approach
8. FLEET &
MOBILITY
Two case studies
Detecting
transport types
based on low
level sensor data
MUSIC &
MOOD
Estimating a
user’s mood
based on
acoustic features
9. Transport type detection
1. Time series data: accelerometer, gyroscope
2. Categorical enrichment: road type, train stations, etc.
3. Missing/partially observable data: GPS locations
4. Small data!
HOW DO WE SOLVE THIS?
10. Transport type detection
Our general prediction pipeline
Pre-processing Feature calculation Data abstraction
Post-processing Temporal smoothing Temporal prediction
11. Transport type detection
Our general prediction pipeline
Pre-processing Feature calculation Data abstraction
Post-processing Temporal smoothing Temporal prediction
12. Prediction pipeline: pre-processing
1. Remove noise: low-pass filter
2. Isolate signal components: band-pass filter
3. Resample and interpolate
4. Sanity checks: sampling rate, sequence length, etc.
14. Transport type detection
Our general prediction pipeline
Pre-processing Feature calculation Data abstraction
Post-processing Temporal smoothing Temporal prediction
15. Prediction pipeline: Feature calculation
1. Periodicity and rhythm
- Autocorrelation, beats, zero-crossings, etc.
2. Timbre
- Spectral envelope
1. Pitch
- fundamental frequencies and harmonics
2. Spectral Flux
- Temporal spectral behavior
3. Loudness
- Power/RMS
…
16. Prediction pipeline: Feature calculation
Deep Learning
- Convolutional neural network
- 1D convolutions across frequency axis!
- Max-pooling and dropout
=> avoid the curse of dimensionality
- Automatically discovers important non-linearities
- Disadvantage: needs lots of training data!
17. Transport type detection
Our general prediction pipeline
Pre-processing Feature calculation Data abstraction
Post-processing Temporal smoothing Temporal prediction
18. Prediction pipeline: Data abstraction
Huge input dimensionality
- E.g. 6D input data (accelerometer and gyroscope) @50Hz
- 5-second fragments: 1500D!
Huge feature space dimensionality
- ± 500D for each 5-second fragment
Dimensionality reduction needed!
- Traditional methods are unsupervised:
Kernel PCA, SOM, IsoMAP, Spectral clustering, etc.
20. Prediction pipeline: Data abstraction
Idea:
- Learn non-linear abstraction in a supervised manner
- E.g. Random Forest, or deep CNN
- RF output: class probabilities
- Use these as input features for a temporal classifier
21. Transport type detection
Our general prediction pipeline
Pre-processing Feature calculation Data abstraction
Post-processing Temporal smoothing Temporal prediction
22. Prediction pipeline: Temporal prediction
Goal:
- Learn temporal correlations between input data (co-adaptations)
- Cope with missing or partially observable data
Constraints:
- Small training dataset!
- For some features more than 70% missing data
- Simple imputation techniques won’t work!
30. Automatic tracking of
automotive journeys
• Start and stop time
• Traveled distance
• Time and duration
• Way points
Contextual driver
profiles through
clustering techniques
• Long term driver
profile classifications
• Real time anomaly
detections
Reliable
differentiation
between multiple
cars used
(Bluetooth, frequency,
charger, USB, …)
Back-end SAAS
platform providing
extensive API, reports
and dashboard
Track changes for
individuals, groups and
vehicles
31. EXTERNAL CONTEXTUAL INFLUENCERS
WEATHER
TIME OF
DAY
BASE EVENTS
ROAD
TYPES
TRAFFIC
BRAKE STOP LANE CHANGE TURN ACCELERATE
BEHAVIOURAL INFLUENCERS (OPTIONAL)
SPEED
LIMITS
STOP
LIGHTS
VEHICLE
TYPES
VEHICLE
CONDITION
Examples of human behaviour and mindset that we can take into account are phone interaction and usage,
alertness and stress, schedule, amount of time slept, ..
32. LEFT LANE
DRIVER
These drivers
consistently
opt for the fast
lane.
ZEN
DRIVER
Courteous and
calm. Nobody
more pleasant
to encounter
on the road.
THE
TAILGATER
Let us hope
the person in
front does not
decide to hit
the breaks.
ASOCIAL
DRIVER
Familiar to all
of us, these
drivers that do
not realize
there are
others on the
road as well.
LANE
SWITCHER
Left, right,
left.. . Then
right seems
faster. Maybe
middle lane
now?
10%
DRIVER
At least
there’s
consistency in
their
speeding. An
average 10%
above the
limit.
33. FLEET &
MOBILITY
Two case studies
Detecting
transport types
based on low
level sensor data
MUSIC &
MOOD
Estimating a
user’s mood
based on
acoustic features
40. Music & Mood
Research questions:
1. Are emotions encapsulated in a raw music signal?
2. How can we automatically label millions of songs?
41. Music & Mood
Transfer learning:
1. For 200 songs, we have per-second valence/arousal data
-> Learned a prediction model based on this
2. For 1 million songs, we only have meta-data
-> Tags (e.g. ‘happy’, ‘sleepy’, ‘metal’, ‘super’, ‘cool’)
3. For 100 songs, we have both!
-> Transfer knowledge from 1 to 2 using LSA
43. Music & Mood
Transfer knowledge:
1. For each of the 1 million song
1. Find KNN of the 200 songs in latent space (cosine distance)
2. Interpolate
DEMO
44. Conclusion
1. Sensors are everywhere!
2. Context can improve almost any service, e.g.
1. Media recommendation
2. Insurance: driving behavior
3. Fleet and mobility
4. Advertising
3. We are hiring the best!
2. Data scientists and machine learning specialists
3. Big data analysts and architects