This document discusses computational behavior modeling for the Internet of Things. It begins by posing questions about the future of connected devices and apps. The author then discusses using sensors, learning, and actions to create useful IoT devices like a smart thermostat. However, many current IoT devices are dismissed as solving no real problems. The document outlines using wireless networks and wearables to understand human context and behavior through large datasets of user activity traces and applying machine learning. Challenges are discussed around privacy and participation guarantees. Applications include people analytics, productivity management, and space management to quantify enterprises. The future may include contextual automation through multi-modal sensing and deep learning on devices, networks, and infrastructure to better understand environments and activities.

1. Fahim Kawsar
Computational Behaviour Modelling for the Internet of Things
1
@raswak
http://www.fahim-kawsar.net

2. Everything is Connected

3. 3Questions

4. 1980
1990
2000
2010
2014
7
8 5
6
Which device you will carry in 2025?

5. Will "one app per service model” scale?

6. If everything is connected why would you need a device?

7. Sense
Learn
Act
The learning thermostat

8. Experience

9. Internet of useless things

10. and that smart egg tray

11. They do not solve a real problem

12. Sense
Learn
Act
The learning thermostat

13. Life logging, memory cues and assisting the blind

14. Quantified self towards better health and lifestyle

15. Remote awareness of families and friends

16. The awareness mirror

17. Right information at the right time at the right place

18. Accessing
everything
Controlling
everything
Understanding
everything
Sensing + Understanding
you and the world around you

19. Sensing + Understanding
you and the world around you

20. To what extent can we leverage existing wireless network and wearable devices as a sensing
modality to understand human context?
Sensorless Sensing with Network and Devices

21. Sensing + Understanding
you and the world around you
UbiComp 2014, MobiQuitous 2015, UbiComp 2016
Quantifying and Understanding Mobile User Behaviour
through Cellular Network

22. Dataset
10000 Users
30 Days
77000000 records
User Data Records (UDR) Include
Web URL
Up and Down Traffic
Start and End Time
eNodeB Id
User Demography
Understanding Mobile Users
Behavioural
Change
Interest
Modelling
System
Optimisation

23. UDR to Activity Trace Transformation Pipeline
03/03 10:02:05
03/03 10:02:06
03/03 10:12:05
03/03 10:02:06
http:/www.bbc.co.uk
http:/www.img1. bbc.co.uk
http:/www.facebook.com
http:/www.img2.bbc.co.uk
....................................................................
UDR Trace Activity Burst Detection Landmark Clustering at Each Burst
Tagging Landmark with
Activity Type for Each Session
Session Start Time Session End Time Activity
03/03 10:02:05 03/03 10:06:05 News
03/03 10:12:05 03/03 10:17:05 Social Network
Activity Trace
Detect session
by duration or with
time weighted TF-IDF
for multiple small
bursts
Landmark
Detection
90%Coverage of the Transactions across all users.
23

24. Daily Activity Pattern
Identical patterns for most of the activities with a dip during early hours of the day.
Also, it pretty much reflects typical activity cycle, e.g., valley in the early morning for relevant activities, e.g., news,
travel, etc. and bump in evening for entertainment activities, e.g., shopping, video, etc.
Blogs Finance News Online Games
Social NetworkOnline VideoOnline ShoppingOnline Music
Travel Weather Web Communication Web Search
Time of the Day
NumberofActivitySessions

25. Activity Level Analysis
Majority of the sessions are less than 10 min reflecting typical mobile usage.
Shorter durations indicate the abundance of Push Notifications....
52% of the sessions’ durations
are 5 minutes or less
66% of the sessions’ durations
are 10 minutes or less
76% of the sessions’ durations
are 15 minutes or less
37% of the sessions’ durations
are 1 minute or less
Cumulative Distribution of the Activity Session Durations
25

29. Push Notification and RRC State Machine
A mobile device is in the CELL_DCH state for a specific time period (which is called tail time)
without any or small data transmission after a data session, and this tail time corresponds to
60%of the total energy consumed by UE.

30. Bringing the Network into the Story

31. 1 1 1 0
1 0 1 0
1 0 1 0
0 0 0 0
0 0 1 0
1 1 1 0
Day j-4 Day j-3 Day j-2 Day j-1
Look up day, Nl=4
T
12:00
12:01
12:02
.
.
.
.
23: 59
2 Selected Candidate Days
1
1
1
?
?
?
Day jc
Lookupwindow(Lk)
DSj-4=0.6 DSj-4=0.7
Over 75% of
network activity can be
predicted with accuracy
of over 80%
Discovering the Human Routine

32. Average latency
is 157
seconds.
±15% energy
can be saved by
buffering notifications.
ô
RNC UE
UTRAN
GGSN SGSN
CN
Internet
Burst Detector
Deep Packet Inspector
Notification
Detector
Notification Scheduler
Prediction
Model
NS
Request
TTL
TTL
Push to Model
For Update
Request
Scheduler

33. Sensing + Understanding
you and the world around you
Quantifying and Understanding People Dynamics in the Workplace
through Wi-Fi Network
UbiComp 2015, WPA 2015, MUM 2015, UbiComp 2016, MobileHCI 2016, ICMI 2016

34. Quantified Self Quantified Team Quantified Enterprise
People
Analytics
Productivity
Management
Space
Management
People
á
n
Places
7
Activity
Quantified Enterprise
Understand and quantify how people interact and work together
in the real enterprise for personal, group and larger
organization efficiency.

35. “Only of large companies can make meaningful predictions about their workforces,
while can accurately predict business metrics such as budgets, financial
results, and expenses”
4%
90%
- Bersin Research
Employee Survey

36. Spontaneous Interactions
Key to Flow of Ideas
A third of team performance can be
predicted merely by the number of Face
to Face exchanges among team
members.
The “data signature” of natural leaders
can be discovered.
Daily Productivity and Creativity can be
rightly assessed.

37. Primary Signals
Reliable and high precision spatiotemporal trajectories of people and the spatial configuration of the venue

38. People
á
n
Places
7
Activity
Location
Location Co-Location Face to Face Interaction
Trajectory Physical Activity
Location Trajectory + Co-Location Personality and Mood Happiness
Location Co-Location Space Management and Recommendation
Location Trajectory State of Use
(Co-)Location is the key, and behaviour models can extract higher order contexts

39. Design Decisions
Dedicated Badges
Feedback Application
• Data Transport by Wi-Fi probes for minimum energy expenditure
The object tags leverage the IEEE 802.11 standard’s management
frames to propagate the states to the home node.
Challenges
• Detection Reliability (Sparsity, Path Loss, Congestion)
• Participation Guaranty
• A custom designed PCB
• ESP82662 SoC
• A Freescale MMA8452Q accelerometer
• A ultra thin 180mAH LiPo battery
A
Applications

40. Radio Signal
Capturing
Co-presence
Detection
Interaction
Inference
Exploiting Every Day Radio Signals to Detect Human Social Interactions
A network-centric
architecture that captures
existing radio signals (WiFi
probes) from the user’s
device.
Co-location detection based on
similarity of wireless channel
propagation characteristics.
An empirically defined model
grounded upon sociology
theories, by leveraging the size
and duration of the encounter.
Accuracy (Precision)
60%of Encounters Detected
90%
Face to Face Interaction Detection

41. Behaviour Modelling
Extracting high order behavioral traits
Location -> Face to Face Interaction
Location -> Personality
F2F Encounter Diversity, Number, Frequency, regularity and Spatial Behaviour are used to extract Big Five
Personality Traits
Location -> Happiness
Spatial Behaviour and Movement Trajectory are used to estimate Physical Activeness and then map to mental
wellbeing (baseline Happiness Index Survey)
This has been used to build connectivity graph and show collaboration intensity in the application.

42. Interaction Intensity
Interaction Intensity represents the relative exchange between different individuals and captures two aspects
Interaction Frequency: Number of times of Face to Face Interaction.
Interaction Duration: Total Durations of of Face to Face Interaction
A Higher P Value indicates a intense interaction, and a lower P value indicates the reverse
Total Number of Face to Face Interaction
Number of Face to Face Interaction of
Specific Person | Program
Total Duration of a specific Face to Face Interaction
Maximum Duration for a single Face to Face
Interaction across all
P = wai
d + (1 w)ai
f

43. Interaction Diversity
Interaction Diversity represents two aspects
Richness: Number of different individuals a user is engaged with
Evenness: Relative contribution of each individual (program | department) to user’s overall interaction time
H =
X
piln(pi)
Proportion of Interaction Sessions
with User i
A Higher H Value indicates a diverse user who engages with many
users and spends time with them evenly.
A Lower H value indicates a more stable and periodic user who
spend large amount of time with a small number of users.
40 Random Users
As connection increases, diversity
and evenness also increase mostly
but not always.

44. Application Experience
People
Analytics
á
Enterprise Wearables Market to Reach $6.3 Billion by 2020
Tractica Research

45. Self Quantification @ Workspace

46. Timeline of Face to Face Interactions

47. Realtime Recommendation to New Contacts

48. Locating Colleagues and Empty Rooms at Realtime

49. Happiness Map of the Workspace

50. Playful Visualisation of the Workplace Mood

51. Collaboration Uncovered
Insights from Quantified BL Dublin and Antwerp Workplaces
3Secrets Revealed
Its all about
Relationship
Key to Engagement is the
visualization of the
relationship structure
Subtle Hints
Users awareness of their collaboration
nature is crucial, however presentation
needs to subtle but meaningful
Recommend
Opportunities
Users are willing to compromise their
privacy when the value is higher.
Recommendations of right opportunities
are key to create that value
51

52. Web Summit
Largest Tech Conference in the Planet
2015 @ Dublin
40K+ Attendees, 134 Countries
± 6000 Sq. Meter
Startups, Entrepreneurs, Investors …
Permission to monitor
• 50 Investors + 50 Entrepreneurs
• Passive Anonymous Crowd

53. Quantification Metric
Investors and Entrepreneurs’ trajectories during the event are modelled as a weighted directed graph G <V,E> where
the vertices are the zones and the edges correspond to the trajectories between the zones.
Dwelling Time : How much time they spent in their home zone
In-flow and Outflow Degrees : How frequently they moved between zones
Average Path Length : How long and diverse was the exploration
Observations
Zone X
Zone Y
Zone Z
WXZ
WXY
WYZ
Investors Entrepreneurs
Investors spent most of the time in their home zone but
not the entrepreneurs
Investors' visit to other zones were
short and sporadic
• Investors follow a caveman model, and entrepreneurs an explorer model.
• Investors tend to stay together and show strong behavioural homophily

54. Sensing + Understanding
you and the world around you
IoT-App 2015, UbiComp 2016, IPSN 2016, SenSys 2016, MobiCASE 2016, MobiSys 2017
Quantifying and Understanding User Behaviour
through Mobile and Wearable Devices

55. System
User
Content
• User behaviour
• Available attention over time
• Ability to get involved
• Usability
• Endurability
• Aesthetics
• Content Novelty
Engagement-Aware Computing
Can we use context and past usage information from a smartphone to predict
moments of high engagement?

56. Key Features 5 [Last CallLast Notification Recent Active Time

57. The most personal device ever

58. Poor inference Exposes private data Drains battery
But…

59. Breakthroughs in Deep Learning
Speech Recognition, Image Recognition, Object Recognition, Language Translation,
Natural Language Processing….
Remarkable Performance in

60. How can we scale down Deep Learning algorithms to run locally on
wearables and IoT devices?

61. DeepX: A Software Accelerator for Embedded Deep Learning
Online Model Compression
Novel algorithms to compress deep neural networks with negligible degradation in accuracy
Dynamic Model Fusion
Dynamic Model Fusion Simultaneous Execution of Multiple Models through parallelisation of parameter heavy and
computation heavy layers
Optimal Resource Allocation
Scalable algorithms to reduce energy footprints of neural networks and allocate an optimal set of resources at runtime
Online
Compression
Model
Fusion
Optimal
Resource
Allocation

62. Dynamic Model Fusion
Dynamic Model Fusion Simultaneous Execution of Multiple Models through parallelisation of parameter heavy and
computation heavy layers
Online Model Compression
Novel algorithms to compress deep neural networks with negligible degradation in accuracy
Factorisation reduces memory and
computational requirements
Compression by online Insertion
of approximating layers using SVD
Compression by offline layer
sparsification by dictionary learning
Runtime interleaved execution of memory-heavy
and compute heavy layers
Aggressive caching of memory-heavy layers
Perceptual hashing for pre-empt execution
1.5x gains in overall execution time

63. DeepX enables future mobile, wearable and IoT devices to make incredible leaps in understanding you
DeepX: Example of Execution Optimisation
Across a broad set of deep models, data sets and hardware prototypes, DeepX is able to, on average, significantly reduce latency
(10x), and energy (10x) constraints, and remove memory bottlenecks - at a cost of 5% in accuracy

64. Accessing
everything
Controlling
everything
Understanding
everything
The Future : Contextual automation in an everything-digital world
Modelling through RF Sensing
• Deep quantification of the parameter space to determine the bounds
• Exploration of Multi-band and multi-stage inference pipeline e.g., Wi-Fi + Radar for better qualifications
Modelling through Network Infrastructure Sensing
• Exploration of cross device and service interactions to better understand activity semantics
• Fusion of local (immediate environment) and global sensing space
Modelling through Device Sensing
• Exploration of bleeding-edge deep architectures for spatiotemporal data, e.g., RNN, LSTM
• Binarisation of parameter space to simplify memory and compute footprint
Research Challenges Ahead

65. Thank You
Fahim Kawsar
@raswak
eMail: fahim.kawsar@nokia-bell-labs.com