This document discusses Henning Müller's work on medical image retrieval projects. It provides an overview of his background and research interests. Müller's research focuses on developing image retrieval systems to help radiologists and other medical professionals by providing similar past case examples. His projects involve collecting and analyzing medical images and text, extracting visual features, developing multimodal retrieval methods, and evaluating systems through user studies. The goal is to create clinical decision support tools that integrate different data sources to help diagnosis.

1. MedGIFT projects in medical imaging
Henning Müller

2. Where we are
2

3. Who I am
• Medical informatics studies in
Heidelberg, Germany (1992-1997)
• Exchange with Daimler Benz research, USA
• PhD in image processing, image retrieval,
Geneva, Switzerland (1998-2002)
• Exchange with Monash University, Melbourne, AUS
• Titular professor in radiology and medical
informatics at the University of Geneva (2014-)
• Postdoc, assistant professor between 2002-2013
• Professor in Computer Science at the
HES-SO, Sierre, Switzerland (2007-)
3

4. Why working on image retrieval?
• Much imaging data is produced
• Imaging data is very complex
• And getting more complex
• Imaging is essential in diagnosis
and treatment planning
• Images out of their context
loose most of their sense
• Clinical data is necessary
• Diagnoses are often not precise
• Evidence-based medicine case-based reasoning
4

5. Khresmoi – retrieval
5

6. The informed patient
6

7. Steps for our retrieval system
7
Image pre-treatment
Visual feature
extraction
Feature
modeling
Multimodal
fusion
Classification,
detection,
retrieval
Resource
creation
Visualization,
results
presentation

8. Diagnosis aid
8

9. Identifying user requirements
• Surveys among radiologists
• Also GPs and patients
• Observing diagnosis processes
• Analyzing search log files (Goldminer, PubMed, HON)
• Eye tracking on a radiology viewing station
• What are information needs and what are
tasks that are hard and where help is needed?
• Test the developed systems in user studies
• Analyze feedback
• Record the system use for understanding problems
9

10. Eye tracking
10

11. Data used for ParaDISE
• Scientific data of the biomedical literature
• 600’000 articles and 1.6 mio figures of the open access
literature (4 mio images if separating compound figures)
• Public data source but only 2D data
• Clinical data from the Vienna Medical University
image archive
• 5TB of data of two consecutive months
• Radiology reports for each case (in German)
• Private data source, so access only with password
• Link medical cases with similar cases from the
literature based on image data and text
11

12. Creation of the VISCERAL database
12

13. Annotations (20 organs, 55 landmarks)
13

14. Connecting different data levels
14
EHR, PACS

15. Classification of journal figures
• Most figures in articles
are not diagnostic
imaging
• Captions do not always
allow to identify the
image type
• Visual information can help
• All these image types are
mapped to RadLex and
UMLS/MeSH
• Allows reusing information and search in related terms
H Müller, J Kalpathy-Cramer, D Demner-Fushman, S
Antani, Creating a classification of image types in the
medical literature for visual categorization, SPIE medical
15
imaging, San Diego, USA, 2012.

16. Context is important (25 yo vs. 88 yo)!
16

17. Visual feature extraction
• Colors grey levels
• Shapes after segmentations
• Texture information
• In 2D, 3D, 4D
• In several scales and directions
• Local vs. global information extraction
• Finding interest points
• Finding regions or volumes of interest
• Combination of features is usually best
17

18. Visual feature modeling
• Visual words instead of raw visual features
• Reducing the curse of dimensionality
• Find models similar to text (synonyms, polysemy)
18
A Foncubierta, AG Seco de Herrera, H Müller, Medical Image Retrieval using a Bag of Meaningful Visual Words,
ACM MM workshop on medical multimedia retrieval, Barcelona, Spain, 2013.

19. Feature extraction and detection
• Learn combinations of Riesz wavelets as digital
signatures using SVMs
• Create signatures to detect small local lesions and
visualize them
19
A Depeursinge, A Foncubierta–Rodriguez, D Van de Ville, H Müller, Rotation–covariant feature learning
using steerable Riesz wavelets, IEEE Transactions on Image Processing, 2014.

20. Information fusion
• Combine information from
text or structured data
with visual information
• Text data can be mapped
to semantics to understand
links
• Also language-independent
• Early fusion
• Late fusion
• Rank-based vs. score-based
20
⎛ 1
⎞
⎜ ⎜ M
⎟
⎟
⎜ ⎟
⎜ M
⎟
⎜ 1
⎟
⎜ ⎟
⎜⎜ ⎟⎟
⎝ ⎠
M SVM
N
t
t
c
c
⎛ 1
⎞
⎜ ⎟
⎜ M
⎟
⎜ ⎟
⎝ M
⎠
t
t
t q
SVM
⎛ 1
⎞
⎜ ⎟
⎜ M
⎟
⎜ ⎟
⎝ N
⎠
c
c
c q
SVM
mod N Π
( ) t i p w
( ) c i p w

21. Detection and retrieval of similar cases
A Depeursinge, D Van de Ville, A Platon, A Geissbuhler, PA Poletti, H Müller, Near-Affine-Invariant Texture Learning for Lung
Tissue Analysis Using Isotropic Wavelet Frames, IEEE Transactions on Information Technology in Biomedicine, 16(4), 2012.
21

22. Khresmoi4radiology interface
22

23. Khresmoi4professionals interface
23

24. Semantic search, also for images
24

25. Khresmoi4everyone interface
25

26. Shambala – a simple web interface
26

27. Much involvement in benchmarking
• ImageCLEF
• Has had a medical task since 2004
• 2013: modality classification, compound figure separation,
image-based and case-based retrieval
• 2014: liver annotation
• VISCERAL
• Organ segmentation and landmark detection (ISBI)
• Lesion detection and retrieval task
• Khresmoi LinkedIn group, …
27

28. Cloud-based evaluation in VISCERAL
28
Test

29. VISCERAL data
29

30. 4D data analysis
30
Material Attenuation Coefficient vs keV
10
0
1
0
0.
1
• Dual Energy CT for perfusion analysis in
pulmonary embolism
• Collaboration with emergency radiology
• Epileptogenic lesion detection in several MRI
image series (T1, T2, DTI)
OA Jimenez del Toro, A Foncubierta-Rodriguez, MI Vargas Gomez, H Müller, A Depeursinge, Epileptogenic lesion
quantification in MRI using contralateral 3D texture comparisons, MICCAI 2013, Springer LNCS, Nagoya, Japan, 2013.
A Depeursinge, A Foncubierta-Rodriguez, A Vargas, D Van de Ville, A Platon, PA Poletti, H Müller, Rotation-covariant texture
analysis of 4D dual-energy CT as an indicator of local pulmonary perfusion, ISBI 2013, San Francisco, USA, 2013.
1
40
50
60
70
80
90
100
110
120
130
140
Photon Energy (keV)
m(E) (cm2/
mg)
Iodine
Water
80 keV 140 keV

31. 4D visualization
31
• Visualization of two (min and max) energy
levels to visualize pulmonary embolisms

32. Another view on 4D
32

33. An infrastructure supporting the load
• Small, fixed experiments are easy, large routine
updates and use are difficult!! Big data is hard!
• Workflow for data re-indexation, maximum automation
• Khresmoi: Private cloud
• All components in virtual machines connected with a
SOA infrastructure, reattribution of resources possible
• Local computation
• Hadoop/MapReduce to distribute the computation
• Needs some optimization
• Cloud use when local resources are not sufficient
33

34. System overview
34

35. Infostructure in MD-Paedigree
35

36. Conclusions
• Visual information retrieval has many
interesting challenges in the medical field
• Many supporting techniques are required
• Treating big data is a challenge and digital
medicine is really big data
• Many techniques can and need to be used with image
analysis and machine learning as the basis
• Digital medicine is a reality and more is yet to
come … genetics, molecular imaging, …
• We also need corresponding infrastructures
36

37. Contact and more information
• More information can be found at
• http://khresmoi.eu/
• http://visceral.eu/
• http://medgift.hevs.ch/
• http://publications.hevs.ch/
• Contact:
• Henning.mueller@hevs.ch
37