Event detection and understanding is an important area in computer science and especially multimedia. The term event is very broad, and we want to propose a novel event based view on endoscopic surgeries. Thus, with the novel view on surgery in this paper, we want to provide a better un-derstanding and possible way of segmentation of the whole event surgery but also the included sub-events. To achieve this sophisticated goal, we present an annotation tool in combination with a thinking aloud test with an experienced surgeon.

1. Event Understanding in
Endoscopic Surgery Videos
Mario Guggenberger*, Michael Riegler°, Mathias Lux*, Pål Halvorsen°
Workshop on Human Centered Event Understanding from Multimedia (HuEvent14)
*Institute of Information Technology
Alpen-Adria-Universität Klagenfurt
Austria
°Media Performance Group
Simula Research Laboratory AS
Norway

2. The problem: Analysis of endoscopic
surgeries
Mario Guggenberger (ITEC AAU), Michael Riegler (MPG Simula)
/ HuEvent14
2

3. What we did
• Development of an annotation tool for endoscopic surgery videos
• Evaluation with a renowned surgeon
• Thinking aloud test & interview
• Evaluation of the evaluation
• Conceptual findings (event model)
• Technical findings (annotation tool requirements)
Mario Guggenberger (ITEC AAU), Michael Riegler (MPG Simula)
/ HuEvent14
3

4. Take home message
• Surgeries can be seen as events
• Which contain subevents
• Hierarchical model of endoscopic
surgery events
• Granularity of the events is
connected to the type of annotation
Mario Guggenberger (ITEC AAU), Michael Riegler (MPG Simula)
/ HuEvent14
4

5. Annotation tool
Mario Guggenberger (ITEC AAU), Michael Riegler (MPG Simula)
/ HuEvent14
5

6. Thinking aloud test
Mario Guggenberger (ITEC AAU), Michael Riegler (MPG Simula)
/ HuEvent14
6

7. Conceptual findings 1/2
• Surgery can be segmented into events and subevents.
• Granularity of event is directly connected to annotation type.
Mario Guggenberger (ITEC AAU), Michael Riegler (MPG Simula)
/ HuEvent14
7

8. The event model
Mario Guggenberger (ITEC AAU), Michael Riegler (MPG Simula)
/ HuEvent14
8

9. Technical findings 1/3
• Prepopulate timeline
• With automatically detected intervals (preprocessing step).
• With bookmarks marked as key events
during the surgery (planned).
• No interest in classical image processing methods
• E.g. image sharpness, dominant colors, movement intensity.
• Except for out-of-patient detection!
• Drawn annotations on moving video are pointless
• Except maybe with object tracking  study with Rikshospital.
Mario Guggenberger (ITEC AAU), Michael Riegler (MPG Simula)
/ HuEvent14
9

10. Technical findings 2/3
• Tablet size (7‘‘ too small, min 10‘‘)
• Full screen video, control overlay
• Pen!
• Frame-exact seeking
• Surgeon noticed even single-frame misplacements
• Non computer savvy surgeons do not know UI paradigms
• Action feedback needs greater visual or tactile impact
• Zoomed timeline
• Separation of voice and drawn annotations anticipated
• But brings up many problems
Mario Guggenberger (ITEC AAU), Michael Riegler (MPG Simula)
/ HuEvent14
10

11. Technical findings 3/3
• Identified stroke usage types:
• Marking borders with solid stroke
• Marking areas with dots or hatched lines (optional solid stroke border)
• Arrows to indicate directional actions
• thin pen, thick felt pen, semitransparent marker
• Zoom-in
• Export annotated stills
• FF unimportant segments, SlowMo important segments
• Render standalone video
• Multimedia integration (e.g. linking x-ray images)
• Measurements (size, area, structure, color)
Mario Guggenberger (ITEC AAU), Michael Riegler (MPG Simula)
/ HuEvent14
11

12. A little bit of advertisement…
• ITEC MediaPlayer Library for Android
• GPL licensed, sources on GitHub
• http://git.io/SoH2ig
• API-compatible
• Frame-exact seeking
• Playback speed adjustment
• GLES support
• Shader effects, Gesture Zoom & Pan
• Frame extraction
• Supports local and network sources
• incl. MPEG-DASH
• Demo-App on the Play Store
Mario Guggenberger (ITEC AAU), Michael Riegler (MPG Simula)
/ HuEvent14
12

13. Conclusion
• Deduced a general event model of a surgery
• direct mapping between event granularity and annotation type
• Evaluation of an annotation tool for endoscopic surgery videos
• Evaluated with a renowned surgeon
• On his own recordings
• Provided many insights, ideas and a better understanding for
developing appropriate tools and techniques to understand surgeries
better
Mario Guggenberger (ITEC AAU), Michael Riegler (MPG Simula)
/ HuEvent14
13

14. Thank You!
Questions?
Mario Guggenberger (ITEC AAU), Michael Riegler (MPG Simula)
/ HuEvent14
14