The Virtual Dimension Center (VDC) Fellbach has created a whitepaper that provides an up-to-date overview of applications of virtual reality technologies in medical applications.

1. Whitepaper
Virtual Reality in Medicine and Medical
Engineering
Techniques, Applications, Results
© Competence Centre for Virtual Reality and Cooperative Engineering w. V. – Virtual Dimension Center (VDC)
Dr.-Ing. Dipl.-Kfm. Christoph Runde
Virtual Dimension Center (VDC) Fellbach
Auberlenstr. 13
70736 Fellbach
www.vdc-fellbach.de
Techniques, Applications, Results

2. Intraoperative
Support
Digital
Atlases
VR-based
Diagnosis
Analysis
Simulation Data
Planning
Intervention
Tele-
Medicine
Rehabilitation Education
and Training
Workflow and
Arrangement
Overview Illustration
Environment:
Necessary in training:
o realistic
o rare pathologies
o objective evaluation of performance
o easy access
scenario-simulation
Challenges
Permanent advancement of treatment
methods
Coordination task team
Limited visibility, indirect working
Complex 3D data, difficult image
interpretation
Overview
2
o scenario-simulation
Training still on the patient
Assessment of the surgeon/ student
Mastery of many knowledge domains
interpretation
Multimodal interaction
Planning of complex operations
Complex device technology
Application for device development
Scenario simulation for therapy

3. Intraoperative
Support
Digital
Atlases
VR-based
Diagnosis
Analysis
Simulation Data
Planning
Intervention
Tele-
Medicine
Rehabilitation Education
and Training
Workflow and
Arrangement
Overview Illustration
Application fields
VR-based diagnosis
Analysis of 3D data (volume-based
method)
Analysis simulation data
o Strength/ stress calculation
o Kinematics, multibody systems
Flow, fluidics
Rehabilitation
o Motor skills
o Psychiatric therapy
Digital atlases
Training
o Endoscopic procedures
Overview
3
o Flow, fluidics
Planning intervention
o Procedures, routes
o Irradiation
Intraoperative support
o Navigation support
o Expanded surgical field of view
Tele-Medicine
Tele-presence with VR
o Endoscopic procedures
o Surgical activities
o Procedures and processes,
device technology
Workflow and arrangement
planning
Illustration medical technology for
marketing communication

4. Intraoperative
Support
Digital
Atlases
VR-based
Diagnosis
Analysis
Simulation Data
Planning
Intervention
Tele-
Medicine
Rehabilitation Education
and Training
Workflow and
Arrangement
Overview Illustration
VR-based diagnosis
Immersive analysis of results from volume-
based procedures: CT, MRI, ultrasound
Spatial 6DOF interaction
Using stereoscopy and motion parallax (via
head tracking): improving spatial depth
perception
Image: Fh-IPA
DigiHom
VR-based
Diagnosis
4
perception
Image: ZGDV Darmstadt
Image: VPH/ZGDV
3D-representation
brain tumor with
blood vessels
3D-scene extracted from
contents of volume-based
procedures

5. Intraoperative
Support
Digital
Atlases
VR-based
Diagnosis
Analysis
Simulation Data
Planning
Intervention
Tele-
Medicine
Rehabilitation Education
and Training
Workflow and
Arrangement
Overview Illustration
Time lapse, slow motion:
animation: for the recognition
of temporal relationships
Reinforcement: physical
quantity is increased in order
to make it visible
VR-based diagnosis – Applicable techniques
False color representation:
a physical quantity (such as
temperature) is visualized
via color coding
Samples: the viewer can
use a measurement sensor
to display the model´s
Selective display: only areas
with certain measurement
values are displayed: problem
zones
Superposition: 2 simulation
results or one test part and a
simulation result are
VR-based
Diagnosis
Comparative presentation:
alternative processes are
shown simultaneously to
detect differences
Subtractive representation:
only the difference between
two alternative processes is
displayed. This makes process
differences even easier to
identify
5
to display the model´s
measurement values. The
sensor may be variable
regarding the measured
quantity.
Sections: the 3D model is
cutted into sections in
order to make the viewer´s
areas of interest clearly
visible
simulation result are
graphically superimposed:
optimization potential of the
simulation model or
fluctuations of the process
Inline-support with AR:
overlay measurement values/
process parameters on situs
( intraoperative support)

6. Intraoperative
Support
Digital
Atlases
VR-based
Diagnosis
Analysis
Simulation Data
Planning
Intervention
Tele-
Medicine
Rehabilitation Education
and Training
Workflow and
Arrangement
Overview Illustration
Analysis simulation data
Strength and stress calculation:
Inspection of
Geometry implant
Location of insertion
Functionality
Image: Fh-IGD
Analysis of statics
knee implant
Analysis
Simulation Data
6
Functionality
Strains, tensions
on the digital model Imagee CADFEM
Adjust implant on
individual vortex
geometry
Tension calculation
implant
Tension calculation
thighbone with implant;
Decubitus simulation:
tensions on bones
Image: Visenso
Image: CADFEM
Image: Visenso

7. Intraoperative
Support
Digital
Atlases
VR-based
Diagnosis
Analysis
Simulation Data
Planning
Intervention
Tele-
Medicine
Rehabilitation Education
and Training
Workflow and
Arrangement
Overview Illustration
Analysis simulation data
Kinematics and multibody systems:
Inspection of
Movements, limits of movement,
speed, accuracy of fit
Forces, torques
Hip joint:
analysis forcesImage: Blundell
Analysis
Simulation Data
7
Forces, torques
Tensions
on the digital model
Image: Fh-IGD
Image: AnyBody
Jaw:
analysis of movement
pathologies, validation of
implantations
Leg:
digital model of
muscles and bones

8. Intraoperative
Support
Digital
Atlases
VR-based
Diagnosis
Analysis
Simulation Data
Planning
Intervention
Tele-
Medicine
Rehabilitation Education
and Training
Workflow and
Arrangement
Overview Illustration
Analysis simulation data
Flow and fluidics:
Inspection of
Flow rates
Turbulences
Areas with/ without flows
Image: Fh-IGD
3D modeling,
simulation model
flow nose interior
Analysis
Simulation Data
8
Areas with/ without flows
on the digital model
Image: Fh-IGD
Image: Visenso
Flow simulation
nose flow
Simulation model
flow aneurysm

9. Intraoperative
Support
Digital
Atlases
VR-based
Diagnosis
Analysis
Simulation Data
Planning
Intervention
Tele-
Medicine
Rehabilitation Education
and Training
Workflow and
Arrangement
Overview Illustration
Preoperative planning – Simulation of intervention
Process planning
Route planning
Joining
Application examples:
Image: HS Bern
Virtual osteosynthesis
Planning
Intervention
Eingriff
9
Application examples:
Osteosynthesis planning
Various surgery applications
Irradiation planning
Image: HS Bern
Image: BMBF
3D interaction with
3D data for
osteosynthesis
Surgery planning
metastases (yellow),
portal vein (orange)
and hepatic veins
(blue)

10. Intraoperative
Support
Digital
Atlases
VR-based
Diagnosis
Analysis
Simulation Data
Planning
Intervention
Tele-
Medicine
Rehabilitation Education
and Training
Workflow and
Arrangement
Overview Illustration
Planning – Example of liver resection
Various distribution
tools to simulate an
atypical resection:
Planning
Intervention
Eingriff
10
atypical resection:
a) Plane
b) Sphere
c) Malleable surface
/ŵĂŐĞƐ͗ dh 'ƌĂnj
3D measurement tools
for spatial analysis:
a) Distance
measurement
b) Volume measurement
of individual objects
c) Overall volume
through virtual
measuring cup

11. Intraoperative
Support
Digital
Atlases
VR-based
Diagnosis
Analysis
Simulation Data
Planning
Intervention
Tele-
Medicine
Rehabilitation Education
and Training
Workflow and
Arrangement
Overview Illustration
Excursion: Augmented Reality – AR
Basic idea: superimposition of natural view
with computer-generated information
Context sensitivity, alpha-numeric or 3D
content
Simple comparison of reality vs. model:
Data glasses
Overview
11
Simple comparison of reality vs. model:
consistency checks digital model - phys. world
Digitization is not always required
High comprehensibility by docking the
computer graphic to a real object
Usage when manuals are difficult to apply
Instruction, clarification, support in the
operating process
Image: metaio
Image: Volkswagen
CAD model superimposed
with physical prototype
Equipment planning in
factory

12. Intraoperative
Support
Digital
Atlases
VR-based
Diagnosis
Analysis
Simulation Data
Planning
Intervention
Tele-
Medicine
Rehabilitation Education
and Training
Workflow and
Arrangement
Overview Illustration
Das Bild kann nicht angezeigt werden. Dieser Computer verfügt möglicherweise über zu wenig Arbeitsspeicher, um das Bild zu öffnen, oder das Bild ist beschädigt. Starten Sie den Computer neu, und öffnen Sie dann erneut die Datei. Wenn weiterhin das rote x angezeigt
wird, müssen Sie das Bild möglicherweise löschen und dann erneut einfügen.
Intraoperative support
Provision of 3D data from volumetric
methods
Technical data, measured data
Different system configurations are
conceivable (tested prototypical)
Image: ZGDV
Possible hardware
arrangement: display
above patient
Intraoperative
Support
12
conceivable (tested prototypical)
o Display between surgeon and situs
(video-see-through or optical-see-
through)
o Projection on situs
o Side by side on separated displays (next
page)
Image: ZGDV
Image: Surgical Planning
Laboratory, Harvard
Gaze through display:
additional information
to normal view
Projection of
3D data

13. Intraoperative
Support
Digital
Atlases
VR-based
Diagnosis
Analysis
Simulation Data
Planning
Intervention
Tele-
Medicine
Rehabilitation Education
and Training
Workflow and
Arrangement
Overview Illustration
Intraoperative support
9th of October, 2012: At the ARTORG Center
for Biomedical Engineering in Bern, surgical
navigation technology is combined with
augmented reality displays. This technology
makes it possible to map the internal
structure of an organ during an operation.
Intraoperative
Support
13
/ŵĂŐĞ͗ hŶŝǀĞƌƐŝƚćƚ ĞƌŶ
structure of an organ during an operation.
The new procedure was now used for the
first time in a complex liver operation.
The liver is shown in triplicate: A screen in the
operating room shows 3D computed
tomography (CT) and magnetic resonance
(MRI) scans illustrating the internal anatomy
of the patient's liver. On another screen
endoscopic images of the miniature camera
can be seen and a third screen shows a
combination of both, a superimposed,
reality-enhanced image. All the relevant
information is summarized in a surgical
cockpit on several screens – this method
helps to simplify orientation and make better
decisions.

14. Intraoperative
Support
Digital
Atlases
VR-based
Diagnosis
Analysis
Simulation Data
Planning
Intervention
Tele-
Medicine
Rehabilitation Education
and Training
Workflow and
Arrangement
Overview Illustration
Das Bild kann nicht angezeigt werden. Dieser Computer verfügt möglicherweise über zu wenig Arbeitsspeicher, um das Bild zu öffnen, oder das Bild ist beschädigt. Starten Sie den Computer neu, und öffnen Sie dann erneut die Datei. Wenn weiterhin das rote x angezeigt
wird, müssen Sie das Bild möglicherweise löschen und dann erneut einfügen.
Tele-Medicine
Tele-surgery: Telepresence applications
where the surgeon and the patient are in
different locations
Application scenarios:
On-site treatment of accident victims,
soldiers, persons who are not transportable
Tele-
Medicine
14
soldiers, persons who are not transportable
Remote diagnosis
Long-distance vs. short-distance
Inclusion of distant specialists:
Teleconsultation
Images: Internet-Blog „Geste, image, sens“
A gallbladder removal
performed in 2001 is
seen as a paradigm for
the long-distance
application. Here a
patient in Strasbourg
underwent a surgery by
robotic arms operated by
a surgeon in New York.

15. Intraoperative
Support
Digital
Atlases
VR-based
Diagnosis
Analysis
Simulation Data
Planning
Intervention
Tele-
Medicine
Rehabilitation Education
and Training
Workflow and
Arrangement
Overview Illustration
Rehabilitation
Motor skills:
Support movement
Deliberate restriction of unintended
directions of movement
Performing exercises
Image: Gupta, O’Malley
MAHI Exoskeleton
Rehabilitation
15
Performing exercises
Objective measurement of success
Image: Blundell
Image: O‘Malley, Ambrose
Haptic work station
for training after stroke
6DOF JPL arm master

16. Intraoperative
Support
Digital
Atlases
VR-based
Diagnosis
Analysis
Simulation Data
Planning
Intervention
Tele-
Medicine
Rehabilitation Education
and Training
Workflow and
Arrangement
Overview Illustration
Rehabilitation
Treatment of phobias and addictions
through individual training in the simulator
Treatment of Posttraumatic Stress Disorder
(PTSD)
Examination of patients behavior and
feelings
Image: Uni Würzburg
VR flight scene
Rehabilitation
16
feelings
Cognitive-behavioral methods
Accompanied by the analysis
of problematic thought
patterns, specific behavioral
exercises and the influence
of physical processes
(e.g. through precise
relaxation techniques)
Image: VT+
Image: Uni Würzburg
VR hight scene
VR spider sceneImage: Uni Würzburg
VR drive through tunnel

17. Intraoperative
Support
Digital
Atlases
VR-based
Diagnosis
Analysis
Simulation Data
Planning
Intervention
Tele-
Medicine
Rehabilitation Education
and Training
Workflow and
Arrangement
Overview Illustration
Rehabilitation
Rehabilitation
17
Image: VT+ VR hight scene

18. Intraoperative
Support
Digital
Atlases
VR-based
Diagnosis
Analysis
Simulation Data
Planning
Intervention
Tele-
Medicine
Rehabilitation Education
and Training
Workflow and
Arrangement
Overview Illustration
Pain treatment
In research since 1996
Central idea: distract patients
2004: patients suffering from burns were
sent to virtual ice worlds
Brain scans show less brain activity with VR
in the areas which are responsible for pain
Rehabilitation
HitLab
VR for the treatment of
burn patients
18
in the areas which are responsible for pain
perception
Endorphins do not play a central role (no
effects of blockers)
Even simple game consoles show no effect
New study (2016) of a hospital in Los
Angeles: 25% reduction of pain perception
through VR
AppliedVR
HitLab
Patient with Samsung
GearVR Headset
Brain scan: less brain
activity in areas which
are responsible for pain
perception with VR

19. Intraoperative
Support
Digital
Atlases
VR-based
Diagnosis
Analysis
Simulation Data
Planning
Intervention
Tele-
Medicine
Rehabilitation Education
and Training
Workflow and
Arrangement
Overview Illustration
Motoric skillsImage: VRMMP
Excursion: Virtual Reality in knowledge management
Knowledge types in virtual environments:
Position knowledge: What is where?
Structural knowledge: How is something
related?
Behavioral knowledge: How does the system
behave? How do I behave?
Overview
VR scene for the
application of a
stethoscope
Virtual surgery with
associated haptic
interface
19
Image: The Medical News (2010): http://www.thaimedicalnews.com
Image: VRMMP
behave? How do I behave?
Procedure knowledge: Which processes cause
what?
Learning opportunities in virtual environments:
Spatial exploration
Conceptual learning
Learning motoric abilities
Procedural learning

20. Intraoperative
Support
Digital
Atlases
VR-based
Diagnosis
Analysis
Simulation Data
Planning
Intervention
Tele-
Medicine
Rehabilitation Education
and Training
Workflow and
Arrangement
Overview Illustration
Digital atlases
Mediation of position knowledge
Mediation of structural knowledge
Anatomy, physiology, biomechanics
Clipping (cutting away graphical
areas in order to get a free view on
Image: Fh-IPA
DigiHom
Digitale
Atlases
20
areas in order to get a free view on
relevant areas)
Transparency
Animation
Image: Google
Image: Bratz
3D representation of
brain and heart
Google Body Browser
Image: BMBF

21. Intraoperative
Support
Digital
Atlases
VR-based
Diagnosis
Analysis
Simulation Data
Planning
Intervention
Tele-
Medicine
Rehabilitation Education
and Training
Workflow and
Arrangement
Overview Illustration
Training scenarios with
visual and haptic
modalities
Learning levels:
demonstrate, accompany,
examine
Training
Simulation
environment eyesi:
Education
and Training
examine
Virtual endoscopy,
laparoscopy
Virtual palpation
Images: Projekt HASASEM
21
environment eyesi:
ophtalmo surgery
Electrorheological fluid
for haptic display of
elasograms.
Application: palpation
for identification of
pathological changes of
tissue
Image: BioSkill
Image: VRmagic
BioSkill simulator
for endoscopy and laparoscopy

22. Intraoperative
Support
Digital
Atlases
VR-based
Diagnosis
Analysis
Simulation Data
Planning
Intervention
Tele-
Medicine
Rehabilitation Education
and Training
Workflow and
Arrangement
Overview Illustration
Training
Diagnosis:
Provision of training scenarios
Examples of application:
Rhinoscopy
Image: Fh-IGD
Virtual rhinoscopy
Education
and Training
22
Rhinoscopy
Ophtalmoscopy
Arthroscopy
Hysteroscopy
Images: VR Magic
Image: Fh-IGD
Virtual ophtalmoscopy
Virtual hystereoscopyVirtual arthroscopy Image: Fh-IGD

23. Intraoperative
Support
Digital
Atlases
VR-based
Diagnosis
Analysis
Simulation Data
Planning
Intervention
Tele-
Medicine
Rehabilitation Education
and Training
Workflow and
Arrangement
Overview Illustration
Training
Intervention
Provision of training scenarios
Examples of application:
Ophthalmic surgery
Image: Uni Heidelberg
Ophthalmic surgery
Education
and Training
23
Ophthalmic surgery
Vitro-retinal interventions,
cataract
Colonoscopy
Images: Uni Heidelberg
Images: Uni Heidelberg
Ophthalmic surgery
EndoSim,
scheme colonoscopy,
3D scene virtual
colonoscopy

24. Intraoperative
Support
Digital
Atlases
VR-based
Diagnosis
Analysis
Simulation Data
Planning
Intervention
Tele-
Medicine
Rehabilitation Education
and Training
Workflow and
Arrangement
Overview Illustration
Training
Procedures and processes:
Provision of training scenarios for
Procedures
Sequences
Involvement of staff
Image: Texas AM University
Virtual inspection
eye adaption
(pupil reflex)
Education
and Training
24
Involvement of staff
Aids
Points of provision
Use of device technology
Image: Texas AM University
Image: Texas AM University
Virtual palpation of spine
Handling equipment
technology

25. Intraoperative
Support
Digital
Atlases
VR-based
Diagnosis
Analysis
Simulation Data
Planning
Intervention
Tele-
Medicine
Rehabilitation Education
and Training
Workflow and
Arrangement
Overview Illustration
Training
Procedures and processes - emergency response
simulators:
Gaining overview,
securing
Self organisation,
team organisation
Image: ETC Simulation
Simulation
ADMS
Education
and Training
25
team organisation
Coordination staff
Image: Texas AM University
Image: E-Semble
Staff control
Simulation Pulse
Simulation
XVR

26. Intraoperative
Support
Digital
Atlases
VR-based
Diagnosis
Analysis
Simulation Data
Planning
Intervention
Tele-
Medicine
Rehabilitation Education
and Training
Workflow and
Arrangement
Overview Illustration
Workflow and arrangement planning
Layout-optimization regarding
reachability, minimization of routes,
overview, flow principle
High occupancy rate of equipment
technology and low patient waiting
times at the same time (resource-
Image: FH Bern
Virtual surgery scene
(project Hovisse)
Workflow and
Arrangement
26
times at the same time (resource-
oriented clocking)
Amongst others use of material flow
simulations
Procedure acceleration
Image: FH Bern
Image: FH Bern
Work group at planning
desk (project Hovisse)
Top view and 4
perspectives
(project Hovisse)

27. Intraoperative
Support
Digital
Atlases
VR-based
Diagnosis
Analysis
Simulation Data
Planning
Intervention
Tele-
Medicine
Rehabilitation Education
and Training
Workflow and
Arrangement
Overview Illustration
Illustration in marketing communication
Realistic representation based on 3D data
Create understanding
Brochures, web, (stereo) films
Interaction is possible
2D drafting based on configuration
Image: LumoGraphics
Computer graphics
operating room for
Berchtold
Illustration
27
Mediation of product variance
Customization: configuration of modular
designs
Consistent sales and quotation processes
Reliable and correct offers even for
complex products
Innovative appearance
Virtual operating room:
extract of 3D stereo film
for KLSmartin
Surgical instruments:
extract of 3D stereo
film for KLSmartin
Image: Visenso
Image: Visenso

28. Intraoperative
Support
Digital
Atlases
VR-based
Diagnosis
Analysis
Simulation Data
Planning
Intervention
Tele-
Medicine
Rehabilitation Education
and Training
Workflow and
Arrangement
Overview Illustration
Conclusion
Very wide range of applications of VR in medicine and medical engineering
Medicine is the most specialized field of application of VR
Very specific device technology
Challenge:
o Registration
Generation of 3D data in real-time
Overview
28
o Generation of 3D data in real-time
Complex models that reflect all facets of reality
Germany, Switzerland, USA very far ahead – local specialists

29. VDC members and partners regarding this topic
29

30. Thank you very much for your interest!
You are interested in this topic and you are looking for contact persons/
implementation partners? Please contact us.
© Competence Centre for Virtual Reality and Cooperative Engineering w. V. – Virtual Dimension Center (VDC)
Virtual Dimension Center (VDC) Fellbach
Auberlenstraße 13
70736 Fellbach
www.vdc-fellbach.de
30