This presentation outlines a rule-based approach for both offline and real-time recognition of Activities of Daily Living (ADL), leveraging events produced by a non-intrusive multi-modal sensor infrastructure deployed in a residential environment. Novel aspects of the approach include: the ability to recognise arbitrary scenarios of complex activities using bottom-up multi-level reasoning, starting from sensor events at the lowest level; an effective heuristics-based method for distinguishing between actual and ghost images in video data; and a highly accurate indoor localisation approach that fuses different sources of location information. The proposed approach is implemented as a rule-based system using Jess and is evaluated using data collected in a smart home environment. Experimental results show high levels of accuracy and performance,proving the effectiveness of the approach in real world setups.

1. Rule-based Real-Time Activity Recognition
in a Smart Home Environment
Przemyslaw Woznowski
Grigoris Antoniou
10th International Web Rule Symposium (RuleML) 2016, Stony Brook, New York, USA
George Baryannis

2. Outline
2
Introduction
• Activity Recognition and the Internet of Things
• The SPHERE Project
• Related Work
Rule-based ADL Recognition
• Offline Version
• Online Version
• Experimental Evaluation
Conclusions & Future Work

3. Activity Recognition and
the Internet of Things
• Sensors have become cheaper, small, widely available
• Interconnected within an Internet of Things (IoT)
setting, benefitting from
– Distribution of resources
– Support for common naming schemas and ontologies
– Common access strategies
– Availability of computational resources
• Automated Activity Recognition (AR) requires a fusion
of multiple sensor-related low-level events
• Challenge: to locate and fuse the right pieces of
information from an IoT instance (e.g. sensor network) in
order to realise AR at the best quality of information
possible
3

4. Approaches for sensor-based AR
• Data-driven – exploiting machine learning techniques
Noise and uncertainty are handled well
Require large, annotated training datasets
Data conflicts are not handled well
• Knowledge-driven – leveraging logical modelling and
reasoning
No training data needed
Not as robust against noise and uncertainty
Require carefully crafted rules
4

5. Activity Recognition in a multi-modal
smart home environment
• Focuses on the so-called Activities of Daily Living
(ADL), with the purpose of supporting Ambient
Assisting Living (AAL) efforts
– Long-term monitoring of health-related features
– Direct assistance
• Main requirements
– Increased need for robustness against noise (due to multiple
sensors)
– Support for complex, uncertain and non-sequential scenarios
– Support for user localization within the smart home, with
minimal user involvement
– Inference of real-time, continuous streams of meaningful and
actionable events
– Less reliance on training data, since they are difficult to acquire
due to them being environment-dependent 5

6. The SPHERE Project
Woznowski et al. (2015)
6
• SPHERE: a
Sensor Platform
for Healthcare in a
Residential
Environment
– Common platform
of non-medical
networked sensors
– Deployed on a
home environment
testbed, the
SPHERE house
– Impact a range of
healthcare needs
simultaneously

7. • Chen et al.: equivalence and subsumption reasoning on
ontologies modelling both sensors and activities
Both offline and real-time modes, incrementally-specific
recognition
Requires activities to be performed in a predefined, strictly
sequential order and fixed time intervals
• MetaQ: SPARQL-based reasoning on sensor data
represented as RDF graphs
Recognition building from atomic gestures to complex activities
Works only offline, does not take into account missing activities
Related Work (1)
7

8. • Skarlatidis et al.: hybrid approach, combining event
calculus reasoning with Markov Logic Networks
High recognition rates, robustness against missing data
Only focuses on posture and movement-related activities, as
opposed to complex ADL scenarios
• Helaoui et al.: hybrid approach, employing a probabilistic
DL reasoner
Recognition building from atomic gestures to complex activities
Requires training data, works only offline, no support for
temporal features
Related Work (2)
8

9. Outline
9
Introduction
• Activity Recognition
• The SPHERE Project
• Related Work
Rule-based ADL Recognition
• Offline Version
• Online Version
• Experimental Evaluation
Conclusions & Future Work

10. 10
• Rule base
– rules defined by examining collected
sensor data from scripted experiments
• Fact base
– derived from sensor data
• The system operates in two modes
– Offline: precollected sensor data are stored as individual facts
• Can provide activity reports for past periods (e.g. hourly or daily)
– Real-time: facts represent each deployed sensor node and store
its current state/value (as well as its previous one)
• Recognises activities as soon as the associated sensor events
happen
Rule
Base
Fact
Base
Inference Engine
(JESS)
“Expert” knowledge Sensor data
Rule-based System Overview

11. 11
• Environmental Sensors
– Door contact, electricity meters, water flow meters, PIR
– Ambient light useful only when the effect of sunlight is minimal
(i.e. the sun is below the horizon)
– Scripted experiment data do not yield patterns from ambient
noise, dust, humidity and temperature
• Video Sensors
– 2D bounding box coordinates
– Depth coordinates of 3D bounding box
Fact Base

12. 12
PIR-based
Location
Camera-
based
Location
Fused
Location
Second Level Rules
Higher Level Rules
Door
Interaction
Electrical
Devices
Water
Flow
Water
Flow
Clean-up
Atomic Activity Rules
Rule Base
rules assume single inhabitant scenarios

13. 13
• Detect changes in sensor values within their reporting
windows
– From >0 to 0: OpenDoor / SwitchOff
– From 0 to >0: CloseDoor / SwitchOn
Doors and Electrical Devices

14. 14
• Water meters do not have a reporting period, only report
instantaneously
– Positive flow value: OpenTap
– Zero flow value: CloseTap
• “Clean up” rules follow to keep only the earliest events
for each distinct opening or closing occurrence
– If there is no close tap activity between two consecutive open tap
activities, remove the latest one
Water Flow

15. 15
• Rules so far recognise atomic activities
• Higher-level rules progressively combine recognised
activities to infer activities of increasingly higher
complexity
– SwitchOn(device,t1) and SwitchOff(device,t2)  Use(device, t1,
t2)
– SwitchOn(tv,t1) and SwitchOff(tv,t2)  WatchingTV(t1, t2)
– Use of taps in kitchen or bathroom  WashHands or
WashFace
– Use of taps in bathroom  BrushTeeth or Bathing/Showering
– Use kettle and close tap in kitchen  PreparingDrink
– Open fridge and use toaster  PreparingSnack
– PreparingDrink or PreparingSnack and use of taps in kitchen 
WashDishes
Complex Activities

16. 16
• Basic PIR rule places user in a specific room, from the
time PIR is activated till it’s deactivated
– Sequences in the same room merged if temporally close or user
not in a different room in between
• Basic video rule places user in a specific room, for as
long as the associated camera reports bounding box
coordinates
• Detect ghost sequences since they severely
compromise validity
– Length of less than 30 frames
– Stuck in the same coordinates for more than 30 frames
– Width and/or height of box consistently and unjustifiably small,
in correlation with depth
Localisation Rules

17. 17
• Assign confidence values to PIR and camera location reports
– PIR: confidence inversely proportional to the number of PIR sensors
simultaneously reporting motion
– Video: confidence depends on the probability of being a ghost,
based on the detection heuristics
• If only a single source reports a location, it is assumed to
hold (with the associated confidence value)
• If PIR and video report the same location, it is assumed to
hold (with confidence values summed)
• If PIR and video disagree, the correct location is the one
associated with a recognised atomic activity
• If both or neither disagreeing reports are supported by an
activity, we assume the one with the higher confidence
holds
– If confidences are equal, we trust PIR
Fused Location

18. 18
• Facts now represent the state of each distinct sensor
– Instead of the history of sensor events
– To detect state change, previous state is also stored
• Changes are necessary only for rules at the lowest level
– Second and higher-level rules remain unchanged
• Transparent to the way sensor events are generated
• Any state change event is linked to a related atomic
activity
– Holds for DC sensors, electricity and water flow meters
– Rules fire only once when sensor values change – no need for
“clean up” rules
From Offline to Real-time

19. 19
• Each consecutive activation/deactivation of a PIR sensor
corresponds to the user being in the associated room
• Subsequent activations extend the user’s stay when
– Activation directly follows the last deactivation
– The elapsed time between them does not exceed a threshold
– No activation has taken place in a different room in the
meantime
• State-based approach is not applicable for video sensors
– Video cameras do not broadcast a single value
• Each reported bounding box is stored briefly
– Combined to create facts that represent a period of stay in a
room
– The same heuristics used for ghost detection
Online Localisation (1)

20. 20
• Each time a PIR sensor is activated, the system fuses
available information to decide on its validity
– If there is no active video sequence and no activity detected, we
assume PIR is valid
– If the active video sequence with the highest confidence agrees
with PIR, we conclude the user is in the room, summing
confidence values
– If video reports a different room, we assume the user is in the
room where the most recently recognised atomic activity was
performed
Online Localisation (2): Fusion

21. 21
• Offline and real-time versions implemented in Java,
using Jess as a rule engine
– Implemented rules designed to accommodate variable reporting
periods
• Real-time version built as an MQTT client
– Sensor messages are broadcast in separate threads
• 10 participants executed an ADL script of half-hour
duration, twice, in the SPHERE house
– Ground-truth data acquired through annotation of video images
collected using a head-mounted camera
– Subset of performed activities that are recognised: interaction
with doors, electrical devices and water taps, preparing a
snack/drink, washing hands/dishes, brushing teeth,
bathing/showering
Implementation and Data Collection

22. 22
• TP (true positive): activity performed and recognised
• FP (false positive): activity not performed but recognised
• FN (false negative): activity performed but not recognised
• 𝑝𝑟𝑒𝑐𝑖𝑠𝑖𝑜𝑛 =
𝑇𝑃
𝑇𝑃+𝐹𝑃
%, 𝑟𝑒𝑐𝑎𝑙𝑙 =
𝑇𝑃
𝑇𝑃+𝐹𝑁
%
Evaluation Results

23. Outline
23
Introduction
• Activity Recognition
• The SPHERE Project
• Related Work
Rule-based ADL Recognition
• Offline Version
• Online Version
• Experimental Evaluation
Conclusions & Future Work

24. Concluding remarks
• Rule-based system capable of operating on both
historical and real-time, multi-modal sensor data
acquired in a smart home
– Bottom-up, multi-level rules to support complex ADL scenarios
– Non-deterministic patterns to account for missing activities
• Sensor fusion and heuristics to achieve robustness
against noise
– 95% recall and 88% precision on average for a significant subset
of activities
– 93% room-level localisation accuracy due to effective ghost
detection and location fusion rules
24

25. 25
• Integrate wearable sensor data
– Infer activities unidentifiable with only the other sensors
– Improve localisation accuracy or provide an alternative to video
cameras when they are not available/allowed
• Explore multi-inhabitant scenarios
– Use localisation results to pin down activities to the person
performing them
– For some activities, localisation needs to be more fine-grained
than room-level
• Explore hybrid approach with Machine Learning
research within SPHERE
– Incorporate rules as features in ML algorithms
– Use rules that act on the results of ML algorithms
– Devise ML techniques to learn rules
Current and Future Work

26. 26
Thank you! Questions?
g.bargiannis@hud.ac.uk
http://www.irc-sphere.ac.uk

27. 27
• Chen, L., Nugent, C.D., Wang, H.: A knowledge-driven approach to activity
recognition in smart homes. IEEE Trans. Knowl. Data Eng. 24(6), 961–974
(2012)
• Filippaki, C., Antoniou, G., Tsamardinos, I.: Using constraint optimization for
conflict resolution and detail control in activity recognition. In: Keyson, D.V.,
Maher, M.L., Streitz, N., Cheok, A., Augusto, J.C., Wichert, R., Englebienne, G.,
Aghajan, H., Krose, B.J.A. (eds.) AmI 2011. LNCS, vol. 7040, pp. 51–60.
Springer, Heidelberg (2011)
• Helaoui, R., Riboni, D., Stuckenschmidt, H.: A probabilistic ontological
framework for the recognition of multilevel human activities. In: Mattern, F.,
Santini, S., Canny, J.F., Langheinrich, M., Rekimoto, J. (eds.) UbiComp 2013,
pp. 345–354. ACM (2013)
• Meditskos, G., Dasiopoulou, S., Kompatsiaris, I.: MetaQ: a knowledge-driven
framework for context-aware activity recognition combining SPARQL and OWL
2 activity patterns. Pervasive Mob. Comput. 25, 104–124 (2016)
• Skarlatidis, A., Paliouras, G., Artikis, A., Vouros, G.A.: Probabilistic event
calculus for event recognition. ACM Trans. Comput. Log. 16(2), 11:1–11:37
(2015)
• Woznowski, P., Fafoutis, X., Song, T., Hannuna, S., Camplani, M., Tao, L.,
Paiement, A., Mellios, E., Haghighi, M., Zhu, N., et al.: A multi-modal sensor
infrastructure for healthcare in a residential environment. In: 2015 IEEE
International Conference on Communication Workshop, pp. 271–277. IEEE
(2015)
References