1. Rule-Based High-Level Situation Recognition
from Incomplete Tracking Data
David Münch1, Joris IJsselmuiden1, Ann-Kristin Grosselfinger1,
Michael Arens1, and Rainer Stiefelhagen1,2
1 Fraunhofer IOSB, Germany, david.muench@iosb.fraunhofer.de
2 Karlsruhe Institute of Technology, Germany.
The 6th International Symposium on Rules - Rule ML2012,
Montpellier, France, August 27-29, 2012
1
© Fraunhofer IOSB

2. Motivation
Motivation
• Security systems: Persons
and their behavior are
the focus of attention:
Person centric analysis
• Threat detection
• Visual surveillance
• Activity logging
• Video search
• Driver Assistance Systems
Input data is incomplete
and noisy.
2
© Fraunhofer IOSB

3. Overview
• Cognitive Vision System as a whole.
• High-level knowledge and situation recognition.
• Handling Incomplete Data.
• Experiments.
3
© Fraunhofer IOSB

4. Cognitive Vision System
4
© Fraunhofer IOSB

5. Cognitive Vision System
5
© Fraunhofer IOSB

6. Cognitive Vision System
6
© Fraunhofer IOSB

7. Cognitive Vision System
7
© Fraunhofer IOSB

8. Cognitive Vision System
8
© Fraunhofer IOSB

9. Cognitive Vision System
9
© Fraunhofer IOSB

10. Conceptual Layer – Conceptual Primitives Level
• Quantitative information (from Quantitative Layer) is transformed into primitive
conceptual knowledge (Logic predicates).
• Dictionary of basic rules.
• Mainly domain independent.
• Support of uncertainty and vagueness.
• The rules in the CPL are mostly concerned with spatial relations and temporal
relations on short time intervals.
10
© Fraunhofer IOSB

11. Dictionary of basic rules
Dictionary of basic rules for every
domain.
Fuzzy Metric Temporal Logic
(FMTL):
Extension of first order logic by
notions of fuzziness, time, and
metrics on time.
Inference engine: F-LIMETTE
11
© Fraunhofer IOSB

12. “Numbers” mapped to Concepts
Fuzzy membership functions 𝜇 𝑠𝑝𝑒𝑒𝑑 𝑣𝑎𝑙𝑢𝑒 for the subset {zero, small, normal,
high, very_high} of discrete conceptual speed values.
Figure from: H.-H. Nagel, Steps toward a Cognitive Vision System, 2004, AI Mag.
12
© Fraunhofer IOSB

13. Conceptual Layer – Behavior Representation Level
How to represent the expected Situations?
• Knowledge represented in Situation Graph Trees.
• Graphically editable
• Easy to extend and edit
• Interpretable (vs. black box)
• Exhaustive situation recognition.
13
© Fraunhofer IOSB

14. Behavior Representation Level
Basic logic predicates from Conceptual Primitives Level are aggregated and
structured in Situation Graph Trees (SGT)
 high-level conceptual situations.
An SGT consists of situation schemes:
• Can be start and/or end nodes.
• Unique name.
• State scheme (Precondition).
• Action scheme (Postcondition).
Specialize a situation scheme:
• Prediction edges link to a possible subsequent situation scheme.
• Specialization edges link to more specific situation graphs in a hierarchical
structure.
14
© Fraunhofer IOSB

15. Behavior Representation Level
15
© Fraunhofer IOSB

16. Handling Incomplete Data – Low Level in Scene Domain Level
Perfect input data.
Truth value
Time
16
© Fraunhofer IOSB

17. Handling Incomplete Data – Low Level in SDL
Incomplete input data.
Truth value
Time
17
© Fraunhofer IOSB

18. Handling Incomplete Data – Low Level in SDL
Interpolation of input data:
Truth value
Time
18
© Fraunhofer IOSB

19. Handling Incomplete Data – Low Level in SDL
Interpolation of input data:
Time
19
© Fraunhofer IOSB

20. Handling Incomplete Data – High Level in BRL
20
© Fraunhofer IOSB

21. Handling Incomplete Data – High Level in BRL
What if is_open(trunk, Car) fails?
21
© Fraunhofer IOSB

22. Handling Incomplete Data – High Level in BRL
Hallucination (abduction) of missing evidence.
What if is_open(trunk, Car) fails?
 Hallucinate is_open(trunk, Car) and continue!
22
© Fraunhofer IOSB

23. VIRAT Video Dataset
VIRAT Video Dataset Release 1.0
Input data: annotated ground truth: persons and their situations.
Place: 0000: Place: 0002:
Videos: 02 03 04 06 Videos: 00 06
23
© Fraunhofer IOSB

24. Situations
1. Person loads object into car.
2. Person unloads object from car.
3. Person gets into car.
4. Person gets out of car.
VIRAT Video Dataset Release 1.0
24
© Fraunhofer IOSB

25. Evaluation
• The annotated ground truth is regarded as complete information.
• Randomly make gaps of a distinct length into the data.
• Increase the amount of missing data in steps of 5%.
• Repeat each experiment several times.
25
© Fraunhofer IOSB

26. Evaluation
Original, F-Score
unmodified
Gap size
5 seconds.
Precision Recall
Gap size Gap size
5 seconds. 5 seconds.
26
© Fraunhofer IOSB

27. Evaluation
Original, F-Score
unmodified
Gap size
5 seconds.
Precision Recall
Gap size Gap size
5 seconds. 5 seconds.
27
© Fraunhofer IOSB

28. Evaluation
ROC-curves for video (d) with gap size 5 seconds.
TPR TPR
FPR FPR
With interpolation and hallucination. Without interpolation and hallucination.
28
© Fraunhofer IOSB

29. Evaluation
Video (d),
F-Score Gap size
5 seconds.
Gap size Gap size
5 seconds. 5 seconds.
29
© Fraunhofer IOSB

30. Conclusion
• low level: interpolation of data and its uncertainty.
 ordinary incomplete data.
• high level: extension of the situation recognition inference algorithm
(hallucination, abduction).
 high-level incomplete data such as occlusions.
• Knowledge base for vehicle-centered situations.
• Runs in real-time on off-the-shelf hardware.
30
© Fraunhofer IOSB

31. Finis.
31
© Fraunhofer IOSB