Mastering MySQL Database Architecture: Deep Dive into MySQL Shell and MySQL R...
FOCUS K3D Research Road Map
1. FOCUS K3D D1.4.1
Deliverable D1.4.1 of Task T1.4
Road map for Future Research
Circulation: PU1
Partners: ALL
Authors: Chiara Catalano (IMATI), Michela Spagnuolo (IMATI), Michela
Mortara (IMATI), Bianca Falcidieno (IMATI), Andre Stork (FRAUNHOFER),
Marianne Koch (FRAUNHOFER), Pierre Alliez (INRIA), Frederic Cazals
(INRIA), Mariette Yvenec (INRIA), Wolfgang Huerst (UU), Remco
Veltkamp (UU), Marios Pitikakis (CERETETH), Caecilia Charbonnier
(MIRALab), Lazhari Assassi (MIRALab), Jinman Kim (MIRALab), Nadia
Magnenat-Thalmann (MIRALab), Patrick Salamin (EPFL), Daniel
Thalmann (EPFL), Tor Dokken (SINTEF), Ewald Quak (SINTEF)
Version: 02
Date: Thursday, 01 April 2010
1
Please indicate the dissemination level using one of the following codes:
PU=Public
PP=Restricted to other programme participants (including the Commission Services).
RE= Restricted to a group specified by the Consortium (including the Commission Services).
CO= Confidential, only for members of the Consortium (including the Commission Services).
01/04/2010 1
3. FOCUS K3D D1.4.1
Document History
Vers. Issue Date Stage Content and changes
01 2nd February 2010 80% First version submitted
st
02 31 March 2010 100% Final submitted version
03
04
Executive Summary
This document contains the deliverable D1.4.1 of the FOCUS K3D Coordination Action.
Deliverable D1.4.1 is the final technical report of Task 1.4 entitled “Road map for Future
Research”. This deliverable defines the guidelines envisaged by the FOCUS K3D Consortium for
future research, consolidating input from all the different kinds of AWG activities (i.e. ad hoc
meetings, questionnaires, workshops, dissemination events) carried out during the two years
of the project. The document also includes the outcomes of the discussion with AWG members
about the challenges of the road map, as presented during the final FOCUS K3D conference.
01/04/2010 3
4. FOCUS K3D D1.4.1
Table of Contents
Chapter 0 : Preface................................................................................................. 5
Chapter 1 : Visionary scenarios .............................................................................. 7
1.1 Visionary scenario in BioTech, health & medicine .................................................. 7
1.2 Visionary scenario in Robotics ............................................................................ 8
1.3 Visionary scenario in Business & education, home & leisure.................................... 9
1.4 Discussion ...................................................................................................... 9
Chapter 2 : Definition of the field ......................................................................... 12
2.1 3D media representation: from geometry to semantics........................................ 12
2.2 Knowledge sources in 3D applications ............................................................... 14
2.3 Discussion .................................................................................................... 16
Chapter 3 : 3D in application domains .................................................................. 17
3.1 Medicine ....................................................................................................... 17
3.2 Bioinformatics ............................................................................................... 20
3.3 CAD/CAE and Virtual Product Modelling ............................................................. 23
3.4 Gaming and Simulation................................................................................... 26
3.5 Cultural Heritage and Archaeology.................................................................... 28
3.6 Discussion and conclusion ............................................................................... 30
Chapter 4 : The FOCUS K3D research road map .................................................... 32
4.1 Derive symbolic representations....................................................................... 34
4.1.1 Time line and dependence diagram ......................................................... 43
4.2 Goal-oriented 3D model synthesising ................................................................ 44
4.2.1 Time line and dependence diagram ......................................................... 48
4.3 Documenting the life cycle of 3D objects ........................................................... 49
4.3.1 Time line and dependence diagram ......................................................... 55
4.4 Semantic interaction and visualisation............................................................... 56
4.4.1 Time line and dependence diagram ......................................................... 62
4.5 Standards ..................................................................................................... 63
4.5.1 Time line and dependence diagram ......................................................... 64
4.6 Trends in Semantic Web research..................................................................... 65
4.7 Final discussion ............................................................................................. 68
4.7.1 Medicine scenario: liver segmentation ..................................................... 69
4.7.2 Bioinformatics scenario: from models to annotation................................... 70
4.7.3 Gaming and Simulation scenario: vessel design workflow ........................... 70
4.7.4 CAD/CAE and Virtual Product Modelling scenario: semantics based virtual 3D
product modelling............................................................................................. 72
4.7.5 Archaeology and Cultural Heritage scenario: large-scale repository of 3D digital
artefacts ......................................................................................................... 73
4.7.6 Robust geometry processing: practical benefits across the AWGs ................ 75
4.7.7 Conclusion........................................................................................... 77
References ........................................................................................................... 81
01/04/2010 4
5. FOCUS K3D D1.4.1
Chapter 0 : Preface
3D media are digital representations of either physically existing objects or virtual objects
that can be processed by computer applications. They may be defined either directly in the
virtual world with a modelling system, or acquired by scanning the surfaces of a real physical
object.
The production and processing of digital 3D content was and still is a traditional field of
expertise of Computer Graphics, but only recently 3D entered the multimedia world: only in
the last decade, indeed, has Computer Graphics reached a mature stage where fundamental
problems related to the modelling, visualisation and streaming of static and dynamic 3D
shapes are well understood and solved. Considering that nowadays most PCs connected to the
Internet are equipped with high-performance 3D graphics hardware, it seems clear that in the
near future 3D data will represent a huge amount of the data stored and transmitted using
Internet technologies.
Therefore, 3D media introduce a new kind of content in the established multimedia scenario,
with the term multimedia characterised by the possible multiplicity of content, by its
availability in digital form and its accessibility via electronic media. Text, audio, animations,
videos, and graphics are typical forms of content combined in multimedia, and their
consumption can be either linear (e.g. sound) or non-linear (e.g. hypermedia), usually allowing
for some degree of interactivity.
At the same time, research on semantic multimedia, defined by the deep integration of
semantic web techniques with multimedia analysis tools, has shown how to use and share
content of multiple forms, endowed with some kind of intelligence, accessible in digital form
and in distributed or networked environments. The success of semantic multimedia largely
depends on the extent to which we will be able to use them in systems that provide efficient
and effective search capabilities, analysis mechanisms, and intuitive re-use and creation
facilities, at the level of content, semantics and context (Golshani06). Going one step further,
we could easily envisage semantic multimedia systems of a higher level of complexity, in which
the generic architectural framework underpinning semantic multimedia systems could be
extended to knowledge and data intensive applications, which have been historically developed
in sectors that were quite far from multimedia and knowledge technologies.
Throughout this document, we will illustrate this idea focusing on prospective applications of
3D content, as a rapidly emerging new form of media in the semantic multimedia panorama
and an extremely challenging application context for semantic multimedia.
3D media, indeed, encompass all forms of digital content concerning 3D objects used and
managed in networked environments, and not only fancy-looking graphics used in
entertainment applications. 3D media are endowed with a high knowledge value carried either
by the expertise needed to design them or by the information content itself. Currently,
research on multimedia and semantic multimedia is largely devoted to pixel-based content,
which is at most two-dimensional (e.g. images), possibly with the addition of time and audio
(e.g., animations or videos), while 3D media are defined by vector-based representations. Due
to its distinctive properties, 3D media make it necessary to develop ad hoc solutions for
content analysis, content-based and context-based retrieval, modelling and presentation,
simply because most 2D methods do not generalise directly to 3D (SpFa2009).
The AIM@SHAPE Network of Excellence (www.aimatshape.net) was the first EC project
addressing the issue of adding semantics to plain geometrical shapes. The prototype
infrastructure created by AIM@SHAPE and the ontologies developed by it demonstrated the
potential gain for the Computer Graphics community of having detailed metadata attached to
3D models.
Based on the experience of AIM@SHAPE, the FOCUS K3D project (www.focusk3d.eu)
tackled the more complex problem of raising the interest of users in a number of application
domains for semantics-driven processing of 3D data. 3D graphics are key media in many
01/04/2010 5
6. FOCUS K3D D1.4.1
sectors, among them the ones we selected for the application working groups in FOCUS K3D:
Medicine and Bioinformatics, Gaming and Simulation, CAD/CAE and Virtual Product Modelling,
Archaeology and Cultural Heritage. In these areas, representing a complex shape in the
various stages of its complete life-cycle is known to be highly non-trivial, due to the sheer
mass of information involved and the complexity of the knowledge that a shape can reveal as
the result of a modelling process.
The research road map presented in this deliverable is an attempt to synthesise the vision
of the FOCUS K3D project on these themes, after extensive discussions with a variety of users
in the application domains. The perspective of the road map is oriented towards challenges
that exist more in the content production and sharing phase, rather than in the networking
phase.
Thinking about life fifteen years ago, you will discover how it is different from now and how
some technologies we regularly use today were unimaginable at that time, mobile phones and
internet above all. Therefore, we decided to start the document with three visionary scenarios,
which imagine different aspects of life in 2040. What emerged from the stories in the end is
that semantic 3D content will strongly permeate everyday life, even if not directly perceived. A
discussion of where, how and which kind of 3D data will be involved follows to make the link
between geometry and semantics explicit.
In the second chapter, we take a step backward and briefly define our view of semantic 3D
media, as originated from AIM@SHAPE, while, in the third chapter, we describe their current
status in the application working groups selected in FOCUS K3D. The state-of-the-art reports
produced by FOCUS K3D are an important source of information for the interested readers
(see deliverables D2.1.1, D2.2.1, D2.3.1, D2.4.1), as they give an idea of the level of
acquaintance with 3D semantics in these fields, with expected and natural differences between
more established fields, such as CAD/CAE and Product Design, when compared to fields such
as the Gaming and Simulation domains, where the potential of a concurrent use of semantics
and digital 3D data is really high but relatively less perceived and reflected by current practices.
Finally, chapter 4 provides the real road map, in which we have identified high-level goals
that somehow represent the long-term challenges that the Computer Graphics community
should face as targets for new and disruptive research, with a strong need to breach the
borders of a single discipline, calling for a truly multi-disciplinary effort. Within each of these
high-level challenges, we have also identified a number of mid-term challenges that, without
being exhaustive of course, could be required building blocks for further important research
advances.
After the experience of FOCUS K3D and as reflected by the research road map, the path
towards really effective semantic 3D content is still complex, but doable. As the research road
map tries to convey, we believe that the potential of semantic multimedia technologies could
be fully exploited in 3D and knowledge intensive application areas, where the processes deal
with contents of multiple form and type, the processing workflows are guided by knowledge
and semantics, and the working environment is usually distributed. Computer Graphics is
ready to answer these challenges, but it needs a closer connection with Knowledge
Management, Machine Learning, and Cognitive Modelling.
01/04/2010 6
7. FOCUS K3D D1.4.1
Chapter 1 : Visionary scenarios
To illustrate how pervasive semantic 3D technology will be in about 30 years, we present in
the following three visionary scenarios and discuss which aspects related to 3D and semantics
are involved and should be tackled in the future. The three scenarios cover some areas where
3D media are a horizontal technology enabling the megatrends of the future, which are, as
many experts agree (Fut09):
• BioTech, health, medicine;
• Robotics;
• Business & education, home & leisure.
1.1 Visionary scenario in BioTech, health & medicine
The scenario takes place in 2040. The current challenge for Dr Edwardes is to improve the
quality of life of his patient Bill Thomson who is suffering from a serious arthritic problem in
one of his knees. Bill not only suffers from unbearable pain, but also from the serious
discomfort of having limited flexion.
To begin with and check whether the pathology is amenable to drug treatment, Dr
Edwardes decides to have Bill's genome fully sequenced using the latest high throughput
sequencing technique. The output of the procedure being the collection of Bill's genes, Dr
Edwardes aligns portions of this genome against those of other patients suffering from related
disorders, so as to precisely identify the genetic determinants of the disease and check for
possible treatments. The matching portions of the genome are visualised through a
holographic projector. Interestingly for Dr Edwardes – but not for Bill – three genes appear as
mutated, respectively HOOK-01, ESS-21, and RGid-07, each of them being involved in a
different form of arthritis. While the first two mutations are amenable to drug treatment, no
drug is known for the third one. Therefore, Dr Edwardes carries out a homology search in the
Advanced Protein Data Bank to see whether the structure of proteins whose sequence is
similar to that of the protein coded by RGid-07 is known, and indeed finds one such structure.
Screening a library of candidate drugs through a virtual panel against this protein, Dr
Edwardes identifies two of them with good binding affinity. These molecules are candidate
drugs, and he hands the investigation to a colleague of his at the National Institute of Arthritis,
so as to check whether the protein coded by RGid-07 can be patched by one of these drugs.
Meanwhile, he and Bill decide to go for a surgical treatment.
The latest advances in computer-aided medicine enable Dr Edwardes to tackle the challenge
through function-driven modelling and simulation, with precise estimates of the risks so as to
optimise both the decisions and the treatments. The novel methodology consists of driving the
whole medical process by modelling, simulating and monitoring it with a constant eye on the
end goal, here restoring the function (knee motion) while minimizing the pain. The first crucial
decision is to move from non-operative to operative treatment: on the one hand, pain
medication controls pain but does not change the underlying arthritic problem; on the other
hand, it is possible to go ahead with prosthesis surgery albeit with some non-negligible risks.
In the present case the results of a simulation on patient-specific data led Bill to decide to go
ahead with surgery in order to place an implant. Different from previous approaches the shape
of the implant is not chosen among a predefined set but instead optimised through the
simulation so as to best restore the original motion amplitude and flexion of the knee. The
implant is manufactured by a 3D printer, which can make physical 3D shapes from a variety of
biologically compatible materials, while the actual surgical operation is robot-assisted. Another
novelty is that the body is treated as a global system in order to best engineer the implant
together with the treatment that comes with it in order to avoid infections and other
complications such as late mechanical dysfunction of the implant. The present system mixes
knowledge and geometric modeling of the implant. This requires simulating and monitoring
01/04/2010 7
8. FOCUS K3D D1.4.1
Bill’s organism at different scales ranging from the proteins to the organs through the cells,
tissues and ligaments. After surgery Bill has to live with a machine, which monitors his body
until the risk of complication is considered low enough to return to a normal life. One year later
Bill asks Dr Edwardes whether or not he would be able to pursue an athletic activity, like when
he was 25. To answer this question Dr Edwardes runs a new series of acquisitions and
simulations (not reimbursed by social security!), which provide precise quantitative risk
estimates. While looking at the simulation Bill has the strange feeling of being engineered as
an industrial product by a computational engineer. This is, however, nothing compared to the
fun of running once more the Boston marathon.
1.2 Visionary scenario in Robotics
It is Monday morning and Cynthia is starting her new working week. She is a product
development engineer at Robotical Inc., a company focusing on the development of the next
generation of humanoids. Cynthia is specialised in behaviour modelling and simulation with a
special focus on swarm behaviour. At the beginning of her career she developed a 3D sensor
system and the embedded software for the interpretation of hand gestures of human beings
for robot-human-interaction. Her system was able to build a 3D model of the situation and to
interpret the gestures in a semantically described context.
After breakfast Cynthia gets into her car. The car is the new AUTONObile 2040 model.
Cynthia just tells the car the destination of her ride. The autonomous automobile checks the
traffic situation, plans the trip and starts to move. All of a sudden a child runs across the street
and the AUTONObile brakes instantly. Its semantic reasoning systems have come to the
decision to brake after analysing the 3D scene, which is constantly generated and updated by
various 3D sensors (global positioning system, lasers, radars, and cameras). The decision to
brake is taken after considering the distance between the car and the child and rejecting the
possibility to drive around the child because of opposing traffic. After the situation is cleared,
the car continues. Having ‘delivered’ Cynthia to her office, the car searches for a parking space
and waits for the next order.
During the day, Cynthia is trying to optimise the shape and behaviour of the new humanoid
she is developing with her colleagues at Robotical Inc. She is using the newest version of
CAHIA (Computer-Aided Humanoid Interactive Application), which provides function-based
constraints on free-space deformation techniques. With CAHIA all kinds of behaviour models
can be embedded and simulation models are generated on the fly and evaluated in real-time.
Not only one humanoid can be optimised but also strategies for jointly solving problems can be
used. The humanoid is to work as a fire fighter to replace humans in this dangerous job. To
achieve this, group behaviour in 3D environments needs to be trained and simulated.
While waiting for Cynthia, her intelligent house communicates with AUTONObile and asks to
bring some beverages from the supermarket. The car uses the drive-through option of the
supermarket and a service robot is putting the beverages into its trunk. In the meantime the
car’s traffic system receives the information that weather conditions will change rapidly within
the next hours and snowy roads are expected. So on the way back to Cynthia’s office it drives
to the next garage to have the tires changed.
After her working day, Cynthia calls the car to fetch her at the entrance of her office and
they start their way home. During the drive, the car is informed about an accident that
happened on the road ahead. Thus, it is starting to re-plan the trip. This is done in online-
communication with the other vehicles participating in traffic to avoid jams due to too many
cars planning the same route. In addition to the street map, also the 3D profile of the
landscape is taken into account to optimise energy consumption.
AUTONObile returns Cynthia safely to her home. Cynthia is very happy with her personal
mobility assistant. Also her children appreciate their personal chauffeur and Cynthia is
unworried since she knows that the grandparents will supervise the safe and comfortable drive
she already programmed to allow only for certain routes, e.g. to their friends and the football
stadium. AUTONObile recognises the people authorised to drive using 3D face recognition
together with voice analysis, a system Cynthia developed herself after finishing the gesture
01/04/2010 8
9. FOCUS K3D D1.4.1
recognition system.
1.3 Visionary scenario in Business & education, home & leisure
When Semantha gets into her next sleeping phase, the mattress adjusts automatically, and
the room lighting changes slightly, to anticipate her waking up soon. After she gets up, she
prepares for going to work, and leaves a message to her daughter Alice, that will be
synthesised by her private avatar with her own voice.
Semantha is a 3D modeller at the gaming company 'Iterative Life'. When she approaches
the company building, her gait and face are recognised, and entrance is permitted. She is
currently working on the design and modelling of a collection of characters for a new game:
"Parallel Universe Loophole Paradise". The characters in the game, the PULPese, have feet like
wheels, arms like fins, and ears like wings. In her 'Evolving Personalised Information
Construct' (Epico, marketed by GoogleZone, the joint venture of Google and Amazone)
Semantha sees that she has modelled the feet last week. According to the schedule, she now
must design the accompanying gadgets for chiropody. She invokes the encyclopaedic
dictionary to learn about that, and the semantic network helps her to link these concepts to
the maintenance tools for wheels. In the collection of instruments she wants to select those
instruments that somewhat fit to the feet and can be used as a start for modelling new ones.
Then, she activates her “FindMeShape”, the shape matcher tool able to select the appropriate
models for the redesign process.
She is interrupted by her iWant, her personal digital life assistant, a seventh sense to her.
She has to collect Alice from school. On her way, she consults City 3.0 to find a restaurant
near school and near the transportation dock, so that they do not lose time on their way to the
afternoon event. One of her friends that are currently also signed in to City 3.0 tells her of a
good place to eat. They decide to spend the afternoon visiting the Clonehenge area, a replica
of the Stonehenge site that disappeared under water after the sudden ice meltdown
catastrophe in 2025. When Alice puts on her glasses, she sees the enhanced environment
consisting of projected 3D media and the real world. Through the looking glass, Alice sees a
wonderland with a lot of cultural information on this piece of heritage, and she gets really
affected by experiencing the past. When they finally get back home, Semantha continues her
work to meet the planning schedule, while Alice is doing homework with her iCat.
1.4 Discussion
In the three stories above, we tried to envisage how technology will influence and support
different aspects of our future life. Although it may not be obvious, 3D multimedia play a key
role in the megatrends identified. In all the three scenarios futuristic technology makes strong
use of 3D data: since our perception of the world is 3D, the advanced tools of the future
should be able to support it as efficiently and effectively as (and together with) any other kind
of multimedia.
The interaction between real and virtual world will be tighter: virtual avatars will replace
humans, while intelligent environments will become the human-computer interface. As a
consequence, one of the main research trends refers to the description and understanding of
the surrounding world, which have to rely on efficient methodologies for modelling and
processing 3D data. Another prominent aspect is the personalisation of the digital content and
technology according to the context and the user. Therefore, the formalisation of the data
cannot be static, but it should vary dynamically according to the purpose, the context and the
user.
In particular, the BioTech, health & medicine scenario implies a function-centric modelling
and simulation of the human body, which relates different scales ranging from proteins to
organs. In such a context where the human body is simulated as a complex system, semantic
plays a key role to model the knowledge that links the various scales and the border between
medicine and computational engineering becomes fuzzier. The future of computer-assisted
medicine will certainly require thinking in terms of goal-oriented modelling and simulation
01/04/2010 9
10. FOCUS K3D D1.4.1
(with the goal being a knee that functions well in terms of motion amplitude, and with no pain),
where some parts of the data come from knowledge about the generic human anatomy and
other parts come from patient-specific acquisition and semantic annotation both before and
after surgery. Furthermore, as medicine is not an exact science and developed countries care
increasingly about legal issues and quality assurance, precise semantic description and
estimates of the risks of any physical medical procedure will probably be of increasing
importance. While generic knowledge about the human body model will certainly be of crucial
importance for statistical risk estimation, a patient-specific estimate will require semantics-
based analysis of massive data sets coming both from past simulations and real experiments
carried out on other patients, which show some similarity.
In addition, the adoption of theragnostics is investigated more and more, which is a
treatment strategy that combines therapeutics with diagnostics. It associates both a diagnostic
(semantically rich) test that identifies patients most likely to be helped or harmed by a new
medication, and a targeted drug therapy based on the test results. Bioinformatics, genomics,
proteomics, and functional genomics are molecular biology tools essential for the progress of
molecular theragnostics.
Generic human anatomy, as well as patient specific data, will be modelled and integrated on
3D replicas of the physical parts and used to attach information and run simulation models
that will be tailored for each patient. Finally, and instead of thinking of computer-aided
medicine only when the disease or pain has already occurred, the future of computer-aided
medicine will most probably consist of increasingly preventive medicine, where modelling and
simulation will again be put to use for monitoring, predicting and avoiding dysfunction as much
as possible. In addition, robots will have an increased role in assisting surgery with high
precision.
Not only in medical applications the number of robots will be huge in the future, as
technology and market forecasts indicate. Not only their population will increase in terms of
sheer numbers but they will also drive some key technology developments and will become
ever more intelligent.
Marvin J. Cetron, President of Forecasting International and a member of the world Future
Society board, forecasts (Fut09) that:
• until 2020 we will have self-diagnostic and self-repairing robots in self-monitoring
infrastructures for almost any job in homes and hospitals;
• until 2030 the robot population will surpass human population in the developed
world, and
• in 2040 robots will completely replace humans in the workforce.
Robots can be regarded as intelligent machines, and semantic 3D models are key to their
development and for their successful use.
As pointed out in the robotics scenario, to orient themselves in the real world and to
communicate with human beings and other robots, they first and foremost need to build up an
understanding about the scene they are surrounded by. It is not sufficient to create just a
point cloud or a static (or dynamic) distance field – e.g. to avoid collisions –, the robot has to
create a ‘mental’ model from the objects it is surrounded with. It needs to understand the
meaning of those objects, it needs to know the actions they may perform, it needs to
understand their dynamics if they are moving, in order to be able to plan its own actions.
These are issues related to understanding the 3D world, valid not only for robots but for all
kinds of interaction between virtual entities and humans. The outdoor navigation domain is a
great challenge for three-dimensional perception. One key to successful navigation in real
world environments is the robot’s ability to reason about its environment in three dimensions,
to handle unknown space in an effective manner and to detect and navigate in environments
with obstacles of varying shapes and sizes. We believe that the problem should be addressed
by a tighter coupling of geometry processing methods with cognitive science, so that the
properties characterising the shapes can be better aligned with features that are used by
humans to “understand and classify” shapes. Also machine learning or similar statistical
01/04/2010 10
11. FOCUS K3D D1.4.1
methods are expected to play a key role to cope with the complexity and variability of 3D
object shapes, and as tools supporting the automatic segmentation of massive datasets.
To be able to do so, the robot needs semantic and geometric 3D descriptions of reference
objects (learnt/trained knowledge about objects potentially being present in the scene and
their functionality). Such models have to be ‘implanted’ into the robot possibly along with self-
learning mechanisms, i.e. they have to exist when the robot ‘comes to life’, which takes us to
the development phase for robots. There is already a growing tendency to introduce high-level
semantic information into robotic systems. This tendency is visible in many areas, including
mapping and localisation, robotic vision, human-robot interaction, and the increasing use of
ontologies in robotics.
Today, robots are systems consisting of mechanics, electronics, and software. Some behave
in a pre-programmed way, some react to their surroundings. In any case the development
process of robots also requires semantic 3D models (not only the use phase of a robot).
Robots can be built more rapidly if functional-oriented design and reuse of functional
components are intrinsically supported by virtual product development tools along with
advanced simulation techniques. Ontological information is has increasingly being used in
distributed systems in order to allow for automatic reconfiguration in the areas of flexible
automation and of ubiquitous robotics. There is a need to use ontological information to
improve the inter-operability of robotic components developed for different systems.
Also in the Business & education, home & leisure scenario, we see that a number of new
technologies will be present, which allow for an integrated approach to search for and find
functionalities, are built on the massive semantic annotation of 3D media, exploit the re-use of
personalised information, depend on the connectedness between real and virtual worlds, and
utilise location-aware ambient intelligence and smart objects, which altogether make the
production and management workflows much more effective, and the leisure and care tasks
much more affective and satisfying than in the old days before 2010.
Although some of the above technologies are currently being developed, there is still a long
road to go. Through professional production, user content generation, and digitization and
preservation of cultural heritage, large amounts of digital media such as images, video, music,
three-dimensional objects and environments are becoming available. Tools for creation, editing,
searching and interaction have been developed by now. New techniques are being developed
to build games, model 3D worlds, and create virtual prototypes of new products or
architectural plans.
The next generation of applications and environments will require more natural interaction
and intelligence to become more dynamic, affective, and satisfying. Tools are necessary to
create virtual persons that are endowed with speech and social and emotional intelligence.
Toolboxes and interfaces must be created to add ambient intelligence to working and living
spaces. These will lead to newly created products and services for personalised information
finding, connecting real and virtual worlds, experiencing past culture, and adopting its lifestyle.
Functionality and elements that are currently missing are rich semantic annotations of the
world and environments, massive semantic annotation of 3D objects, localisation in 3D
environments, and tangible 3D visual interfaces.
In the next chapter, our view of 3D semantic multimedia will be introduced and the current
status of the field will be briefly described; chapter 3 will then give an overview of some
application fields.
01/04/2010 11
12. FOCUS K3D D1.4.1
Chapter 2 : Definition of the field
In this chapter, we will introduce our view of semantic 3D media, taking into account the
perspective of researchers in the field of Computer Graphics as well as the perspective of the
users in the reference application domains. Researchers in Computer Graphics are indeed key
players in the process of formalising the semantics through knowledge technologies as they
are the experts able to develop the appropriate tools and schemes for documenting and
sharing 3D media representations, in close cooperation with experts in different domains. In
the following, Section 2.1 will discuss the evolution of 3D modelling paradigms, from the
traditional geometry-oriented to the emerging semantics-driven approaches. In Section 2.2,
we will outline the sources of knowledge that we believe need a formalisation and a more
effective management much more integrated with the geometric modelling pipeline of the
object’s shape. A discussion in Section 2.3 concludes this chapter.
2.1 3D media representation: from geometry to semantics
Knowledge technologies can be effectively used in complex 3D application fields if the
underlying 3D modelling approach is able to support and encapsulate the different levels of
abstraction needed for describing an object’s form, function and meaning in a suitable way.
The EC Network of Excellence project AIM@SHAPE proposed an evolution of the traditional
geometric modelling paradigm towards a semantics-based modelling paradigm, which is
consistent with the different description levels used for traditional 2D media and reflects an
organisation of the information content that ontologies dealing with 3D application domains
could nicely exploit.
The use of computers has revolutionised the approach to shape modelling, opening new
frontiers in research and application fields: Computer Aided Design, Computer Graphics and
Computer Vision, whose main goal is to discover basic models for generating and representing
shapes. In the beginning, this effort gave rise to research in geometric modelling, which
sought to define the abstract properties, which completely describe the geometry of an object,
and the tools to handle this embedding into a symbolic structure. The visual aspect of shapes
has deeply influenced the development of techniques for digital shape modelling, which have
mainly focused on the definition of mathematical frameworks for approximating the outer form
of objects using a variety of representation schemes.
The same geometry can be represented by different approximations (e.g. meshes,
parametric surfaces, unstructured point clouds, to cite a few), each representation being
chosen according to its advantages and drawbacks with respect to application purposes. For
example, complex triangulated meshes are badly suited to interactive animation; simpler
representations such as point set surfaces can be sufficient for special classes of applications
(e.g. ray-tracing techniques only require ray-surface interrogations). The conversion between
distinct representations is still a delicate issue in most systems but there are tools to derive
triangular meshes from the majority of representation schemes. The selected representation
model will eventually be mapped into an appropriate data structure, which is computer-
understandable and devised according to optimisation and efficiency criteria.
It is important to point out that for the development of semantic 3D media the
conceptualisation of the geometry is not an issue: geometric modelling is, by itself, the answer
of the Computer Graphics community to the need of defining and expressing in formal terms
concepts and relationships concerning the geometric representation of 3D media. Shape
models and related data structures encapsulate all the geometric and topological information
needed to store and manipulate 3D content in digital form.
While the technological advances in terms of hardware and software have made available
plenty of tools for using and interacting with the geometry of shapes, the interaction with the
semantic content of digital shapes is still far from being satisfactory: we still miss effective and
robust tools to manage non-spatial information, i.e. the information not related to the shape of
01/04/2010 12
13. FOCUS K3D D1.4.1
the object (e.g. ownership, material, price) and the information related to the high-level
conceptualisation of the object, either expressed as features (e.g. size, volume, compactness,
presence of holes) or as synthetic concepts (e.g. what it is, what it is used for). Stated
differently, the semantic gap - the lack of coincidence between the information that one can
extract from the visual data and the interpretation that the same data have for a user in a
given situation (Smeulders00) –is still far from being filled.
To tackle this problem we may follow two complementary paths: on the one hand, it is
possible to integrate the missing information through the definition of concept-based metadata
(either by following a formal conceptualisation encoded in an ontology or by allowing a free
tagging of the resources); on the other hand, it is possible to exploit state-of-the-art
descriptors and analysis tools to extract content-based information from spatial information.
Both of the mentioned approaches help to narrow the semantic gap: the design of ad-hoc
ontologies and the development of tools for a versatile annotation of 3D objects are part of the
plan, but yet another important element is crucial for the description process: the role of the
user. In fact, it is not sufficient to provide a generic interpretation of the data, the
interpretation should take the context into account. The awareness of the context should
enable a dynamic and user-oriented description in which only the most relevant content-based
descriptors are activated and combined.
The integration of concept-based and content-based approaches and their embedding in a
context dynamically defined by the user could be an efficient solution, whose evaluation was
one of the main goals of FOCUS K3D through the Application Working Groups.
This integration can be achieved if the 3D content is organised in a way that takes into
account and supports reasoning at different levels of abstraction and that goes beyond the
limits of the pure geometry. The conceptualisation of the knowledge domain has to adhere to a
suitable organisation of the geometric data about the shapes. In AIM@SHAPE, shapes were
defined as characterised by a geometry (i.e. the spatial extent of the object), describable by
structures (e.g. form features or part-whole decomposition), having attributes (e.g. colours,
textures, or names, attached to an object, its parts and/or its features), having a semantics
(e.g. meaning, purpose, functionality), and possibly having interaction with time (e.g. history,
shape morphing, animation) (Falcidieno et al. 2004).
Leaving the traditional modelling paradigm, a simple one-level geometric model has to be
replaced by a multi-layer view, where both the geometry and the semantics contribute to the
representation of the shape. At the same time the structure is seen as an important bridge
towards the semantics, as it supports a natural manner to annotate the geometry with
semantic information. The key idea is that, while the geometry of a shape is unique, its
structure can be defined in many possible ways, each contributing to the creation of a specific
structural model, or view, of the shape. These views of the shape make it easier to interpret
the geometry according to different semantic contexts and attach appropriate content to the
relevant parts of the shape.
To realise the shift towards this new modelling paradigm, it is also necessary to develop
new and advanced tools for supporting semantics-based analysis, synthesis and annotation of
shape models. They correspond to image analysis and segmentation for 2D media: features of
a 3D model are equivalent to regions-of-interests in images. There is, however, a different
input and a slightly different target, as image analysis is trying to understand what kind of
objects are present in the scene captured by the image, while in 3D segmentation the object is
known and it has to be decomposed into meaningful components that might be used to
manipulate and modify the shape at later stages. From a high-level perspective, the main
differences concern the different nature of the content: descriptors used for 2D images are
concerned with colour, textures, and properties that capture geometric details of the shapes
segmented in the image. While one-dimensional boundaries of 2D shapes have a direct
parameterisation (e.g. arc length), the boundary of arbitrary 3D objects cannot be
parameterised in a natural manner, especially when the shape exhibits a complex topology,
e.g. many through-holes or handles. Most notably, feature extraction for image retrieval is
intrinsically affected by the so-called sensory gap: “The sensory gap is the gap between the
01/04/2010 13
14. FOCUS K3D D1.4.1
object in the world and the information in a (computational) description derived from a
recording of that scene” (Smeulders00). This gap makes the description of objects an ill-posed
problem and casts an intrinsic uncertainty on the descriptions due to the presence of
information, which is only accidental in the image or due to occlusion and/or perspective
distortion. However, the boundary of 3D models is represented in vector form and therefore
does not need to be segmented from a background. Hence, while the understanding of the
content of a 3D vector graphics remains an arduous problem, the initial conditions are different
and allow for more effective and reliable analysis results and offer more potential for
interactivity since they can be observed and manipulated from different viewpoints.
Finally, at the semantic level, which is the most abstract level, there is the association of
specific semantics to structured and/or geometric models through annotation of shapes, or
shape parts, according to the concepts formalised by a specific domain ontology. In the Figure
below, a possible shape analysis pipeline is shown: in (a) the digital model of a chair obtained
by acquisition is shown, in (b) the shape represented in structural form can be analysed in the
domain of knowledge related to Computer Animation and the regions of possible grasping are
annotated accordingly (see (c)). The aim of structural models is, therefore, to provide the user
with a rich geometry organisation, which supports the process of semantic annotation.
Consequently, a semantic model is the representation of a shape embedded into a specific
context, and the multi-layer architecture emphasises the separation between the various levels
of representations, depending on the knowledge embedded as well as on their mutual
relationships.
(a) (b) (c)
From geometry to semantics
The multi-layer view of 3D media resembles the different levels of description used for other
types of media, but there is a kind of conceptual shift when dealing with 3D media: here, we
have the complete description of the object and we want to be able to describe its main parts,
or features, usually in terms of low-level characteristics (e.g. curvature, ridges or ravines).
These features define segmentations of the shape itself, which is independent of a specific
domain of application but that carry a geometric or morphological meaning (e.g. protrusions,
depressions, and through holes).
2.2 Knowledge sources in 3D applications
Effective and efficient information management, knowledge sharing and integration have
become an essential part of more and more professional tasks and workflows in Product
Modelling, one of the first fields where semantics came into play. However, it is clear that the
same applies also to other contexts, from the personal environment to other applied sectors.
There is a variety of information related to the shape itself, to the way it has been acquired or
modelled, to the style in which it is represented, processed, and even visualised, and many
more aspects to consider.
The description of a shape is intrinsically not unique and varies according to both the
application and user contexts. Therefore, the abstraction levels used to process or reason
01/04/2010 14
15. FOCUS K3D D1.4.1
about 3D media should correspond to the mental models used to answer questions such as
“what does it look like?”, “what is its function?”, thus making it possible to model, manipulate
and compare the various 3D media in a semantics-oriented framework.
Therefore, the ingredients needed to implement a 3D semantic application should definitely
include a conceptualisation of the shape itself, in terms of geometry, structure and semantics,
and of the knowledge pertaining to the application domain. In order to fulfil the requirements
of complex 3D applications, we need tools and methods to formalise and manage knowledge
related to the media content and to the application domain, at least at the following levels
[SpFa2007]:
• knowledge related to the geometry of 3D media: while the descriptions of digital 3D
media can vary according to the contexts, the geometry of the object remains the same
and it is captured by a set of geometric and topological data that define the digital
shape;
• knowledge related to the application domain in which 3D media are manipulated: the
application domain casts its rules on the way the 3D shape should be represented,
processed, and interpreted. A big role is played by the knowledge of the domain experts,
which is used to manipulate the digital model: for example, the correct manner to
compute a finite element mesh of a 3D object represented by free-form surfaces is
subject also to informal rules that should be captured in a knowledge formalisation
framework;
• knowledge related to the meaning of the object represented by 3D media: 3D media
may represent objects that belong to a category of shapes, either in broad unrestricted
domains (e.g. chair, table in home furniture) or narrow specific domains (e.g. T-slots,
pockets in mechanical engineering). The shape categories can also be defined or
described by domain-specific features, which are the key entities to describe the media
content, and these are obviously dependent on the domain.
The first bullet point of the list is concerned with knowledge, which has geometry as its
background domain. There are a variety of different representations for the geometry of 3D
media that cannot be simply reduced to the classical Virtual Reality Modelling Language
(VRML) descriptions and its variations, as currently supported by MPEG-4. Here, the view of 3D
media is more concerned with the visualisation, streaming and interaction aspects than with
requirements imposed by applications. 3D geometry, as used in applications, has to do with a
much richer variety of methods and models, and users might have to deal with different
representation schemes for the same product within the same modelling pipeline. In this sense,
describing the content of 3D media in terms of geometric data is much more complex for 3D
than for 2D media. There are many attributes and properties of 3D models that scientists and
professionals use to exchange, process and share content, and all these have to be classified
and formalised.
The second bullet point refers to the knowledge pertaining to the specific application domain,
but it has to be linked to the geometric content of the 3D media. Therefore, if we want to
devise semantic 3D media systems with some reasoning capabilities, we have to formalise also
expert knowledge owned by the professionals of the field.
Finally, the third bullet point has to do with the knowledge related to the existence of
categories of shapes; as such, it is related both to generic and specific domains. Usually in 3D
applications, it is neither necessary nor feasible to formalise the rules that precisely define
these categories in terms of geometric properties of the shape, besides very simple cases.
However, due to the potential impact of methods for search and retrieval of 3D media, there is
a growing interest in methods that can be used to derive feature vectors or more structured
descriptors that could be used to automatically classify 3D media.
01/04/2010 15
16. FOCUS K3D D1.4.1
2.3 Discussion
3D media content is growing both in volume and importance in more and more applications
and contexts. In this chapter, we have introduced the issues related to handling this form of
content from the point of view of semantic media systems, focusing on the level at which we
should be able to capture knowledge pertaining to 3D media.
3D applications are characterised by several knowledge-intensive tasks that are not limited
to browsing and retrieving properly annotated material, but deal with the manipulation,
analysis, modification and creation of new 3D content out of the existing one. While this is true
also for traditional 2D media applications, the tools and expertise needed to manipulate and
analyse 3D vector-based media are still the prerogative of a rather specialised community of
professionals and researchers in the Computer Graphics field. In this sense, the Computer
Graphics community could make a significant contribution: on the one hand, it could provide
modelling and processing tools able to handle and exploit the semantics of the shape, and
interaction and visualisation tools, which allow users to interact with the shape in an intelligent
way; on the other hand, it could contribute to comprehensive schemes for documenting and
sharing 3D media and related processing tools, to be linked and further specialised by experts
in different domains.
Semantics-aware classification of available tools for processing the geometry and
manipulating 3D shapes will be an important building block for the composition and creation of
new easy-to-use tools for 3D media, whose use and selection can be made available also to
professional users of 3D who are not experts in Computer Graphics. The use of 3D is indeed
spreading out of the traditional communities of professional users and will soon reach a
general and inexperienced audience.
The role of experts in 3D modelling for the development of semantic 3D media is twofold:
on the one side, the identification of key properties for the description of 3D media and
processing tools, and on the other side, the contribution to the development of advanced tools
for the interpretation, analysis and retrieval of 3D content.
It is evident that if we want to be able to reason about shapes at the geometric, structural
and semantic level, then we have to be also able to annotate, retrieve and compare 3D
content at each of the three layers (SpFa2009).
In the next chapter the current status of semantic 3D media in the four Application Working
Groups explored during the FOCUS K3D project will be described, while in chapter 4 the grand
challenges identified along the way will be stated together with a list of open issues necessary
to reach these goals.
01/04/2010 16
17. FOCUS K3D D1.4.1
Chapter 3 : 3D in application domains
FOCUS K3D has identified four representative application areas that are both consolidated
(Medicine and Bioinformatics and CAD/CAE and Virtual Product Modelling) and emerging
(Gaming and Simulation and Archaeology and Cultural Heritage) in the massive use of 3D
digital resources coupled with knowledge. The selection of these different application areas was
motivated by two reasons. First, all these fields face a huge amount of available 3D data;
second, the use of 3D data is not only related to visual aspects and rendering processes, but
also involves an adequate representation of domain knowledge to exploit and preserve both
the expertise used for their creation and the information content carried.
Among the first actions of FOCUS K3D, a state-of-the-art report (STAR) was produced for
each of the four application areas to report on the differences and similarities in the
approaches and desiderata of users in these representative areas (see deliverables D2.1.1,
D2.2.1, D2.3.1, D.2.4.1). These four STARs were meant to identify the use of current
methodologies for handling 3D content and coding knowledge related to 3D digital content in
the different contexts, and also to identify possible gaps. Each STAR therefore reflects a
snapshot of the current situation in the four subfields that all massively use 3D data, but which
are distinct in their characteristics, requirements, status, and condition.
In this chapter, we provide a summary of the STARs discussed from the perspective of the
research road map, aiming at describing each of the five application areas (Medicine and
Bioinformatics are presented separately for the sake of clarity) and discussing the current
situations and limitations as perceived by the users in the different domains.
3.1 Medicine
Computer-aided medicine is a rapidly growing field and recent
advances in medical imaging yield 3D shape data that are more and
more reliable, accurate and massive. Different medical applications
such as diagnosis, radiotherapy planning, image-guided surgery,
prosthesis fitting or legal medicine now heavily rely on the analysis
and processing of 3D information (e.g., data measuring the spatial
extent of organs, data about the electrical activity of the brain),
requiring different criteria for the specific 3D content they use. 3D
content is sometimes used for shape measurements, as basis for
diagnostic information, or to establish statistical reference data. Therapy planning, image-
guided surgery and assessments of post-operative results are other typical examples of use of
3D content.
The starting point of a typical computer-assisted diagnosis session is often a set of image
data acquired from CT, MRI, MicroCT or other medical imaging devices. The current goal is to
generate a digital 3D model suited to visual inspection and analysis, but the goal could be
much more ambitious, such as simulation, electromagnetic wave propagation and temperature
distribution for hyperthermia, soft tissue deformation for cranial-maxillofacial surgery,
biomechanics for orthopaedic surgery, computational fluid dynamics for rhino-surgery,
electrical potentials for electro-cardiology, and many more. In between, we find not only a
sequence of automatic image processing algorithms but also a true reconstruction pipeline,
which involves data registration, segmentation, 3D geometry generation and processing.
While it is now commonplace to store medical images in digital form, 3D content used in
medical applications is far from easy to obtain: there is a labour-intensive pipeline to get from
the raw images generated by the medical imaging devices to the digital reconstructed
anatomical models suitable to medical applications, such as therapy planning or surgical
simulation.
The current assessment reveals that the pipeline is far from being automatic and requires a
01/04/2010 17
18. FOCUS K3D D1.4.1
lot of knowledge-based interaction, in particular for the segmentation of the images. Even after
decades of research, a major challenge is to recover the inherent 3D models of anatomical
structures from 3D medical images, as accurately and efficiently as possible. Surprisingly, the
most common clinical practice for this task is an interactive correction of an automatic
segmentation algorithm by a medical expert, which incorporates some knowledge throughout
the pipeline. One important thread of research to facilitate this process is therefore devoted to
the elaboration of semi-automatic tools such as magic wands, intelligent scissors or
deformable models. As the resolution of the images always increases, it is however commonly
admitted that such a labour-intensive interactive approach is not a long term approach. This
motivates another thread of research devoted to the elaboration of fully automatic methods,
which incorporate the knowledge through statistical or model-based approaches. One required
feature is to let a door open for the expert to adjust the segmentation as a post-process in the
few cases where the automated process fails. Similar issues arise for geometry modelling and
processing, where one of the goals is to preserve the multi-domain labels during surface
extraction and mesh generation.
The practitioners resort to creating 3D models of organs or of human beings for simulation
purposes. Those models are frequently input to finite element software, for example to
simulate problems related to the hip joint from patient-specific data. This is a typical example
of goal-oriented modeling, where the geometry of the organ to be modelled needs to fulfil the
specific requirements imposed by the technical solution required to solve the objective, or
function to be simulated, which might differ from those of the simple visual inspection: for
finite element analysis, for instance, a fine volumetric mesh representing the boundary and
interior of the bone is required, while for the simple visualisation of the bone shape a coarser
triangulation of its boundary surface would suffice.
Another example is the simulation of the human articulations for replacement by prosthesis.
The different steps are bone resection, prosthesis alignment, and checking of motion
amplitudes. In terms of knowledge, the parameters of interest are contact pressure within the
cartilages, forces and muscle biomechanics. The connections to physiology are related to
material properties as well as to bone motion and geometry. The current state of the art
assesses that such a data process is considerably simplified when these parameters are stored
into an ontology, and even more so if parts of the models of the geometry could be annotated
with functional information or comments by the medical doctors.
3D data and knowledge are again tightly coupled in the simulation of laparoscopic surgery
for training and practising. The goal is to simulate the planned surgery with a robot acting on a
3D model, so as to rehearse the gestures and to check the feasibility of the complete process.
The knowledge involved is related to the anatomy, the forces applicable to the organs and the
physical properties of a tumour.
Virtual anatomy is perceived as a central problem concerned with the elaboration of 3D
geometry models of human anatomy. 3D models must be augmented with knowledge about
the function of the organs, as the primary end goal in computer-aided medicine is to obtain a
human body, which functions well after surgery or treatment. The function itself cannot be
decoupled from the topological and geometrical relationships between the organs. Virtual
anatomy is important in medicine with respect to patient-specific computer-assisted therapy
planning, as well as for industrial applications, such as virtual crash tests where human models
are involved. It is common to distinguish between individual anatomy and generic anatomic
atlases, which demonstrate anatomical structures and their relationships. Individual anatomies
are of special interest when pathological situations need specific attention with respect to a
medical treatment. Generic anatomy models are carefully designed using modelling software,
whereas patient-specific anatomy models come as output of a scanning device, associated with
point set based or image-based reconstruction algorithms.
Another noticeable trend, in simulating and monitoring surgery and therapy is the
replacement of generic anatomic atlases by patient-specific digital anatomical models. This
trend results in the generation of more and more massive 3D content, thus requiring the help
of knowledge technologies for classification, storage and retrieval of those data.
01/04/2010 18
19. FOCUS K3D D1.4.1
Monitoring therapy or following disease evolution requires storing and retrieving various
data for each patient over long time scales. A typical workflow example is provided by a
clinician in a department of neuro-rehabilitation, which routinely performs transcranial
magnetic simulations (TMS). The data are initially acquired by MRI. The MRI images are then
segmented and a so-called 3D anatomical project is created for each patient. Such project,
which corresponds to a geometry reconstruction process, includes surface reconstruction and
mesh generation for the head as well as for the cortical surfaces. This anatomical project is
stored and reused for repeated TMS sessions performed for the same patient.
Overall the evolution toward patient-specific modelling and simulation is recognised as an
advance. However, the fact that either little or no reuse of data and processes is currently
performed across patients is recognised as being very detrimental to the productivity.
Intuitively, the more patients processed the more productive (and hence assisted) the expert
would like to be. The experts and clinicians are conscious that the goal here is not only to
reuse the data, but also to reuse the processes themselves through documenting the whole
process. This challenge is where knowledge-based technologies must play a role.
Knowledge technologies are increasingly required to manage structural, functional and
topological information extracted from the medical data. More specifically, semantics is
required at every step of the modeling and processing pipeline, which ranges from raw data to
goal-oriented 3D numerical models through the crucial and yet current bottleneck step of data
segmentation.
Similarly, visualisation is always at the core of many medical applications such as diagnosis,
surgical simulations, image-guided surgery and training. In the context of knowledge-based
computer-aided medicine visualisation should be not only concerned with the plain rendering
of the geometry, but also with semantics. While our current assessment is that geometry and
semantics are most often decoupled when it comes to interaction and visualisation, some
recent research experiments show that it would be very beneficial to combine both. For
instance, the process of simulating a complex orthopaedic surgery is considerably simplified
when both geometry and knowledge are stored into an ontology, as the practitioner can get an
interactive, faithful simulation including both geometry and semantic information. Such
simplification translates into a reduced surgery planning duration, which is critical for the
practitioners. Having knowledge such as organ dependency and material properties combined
with geometry is already a significant step toward efficient and reliable surgery planning.
What is missing in the current knowledge representation is a detailed description of the link
between the geometry (and topology) of an organ or tissue and its function. For example, the
shape of a joint (e.g., a spherical cap) partially explains its function in terms of motion. The
geometry and topology of an organ also often explains its function, such as connection to a
bone or to another organ, etc. The simulation could thus be improved by integrating into its
model the inherent functional constraints of the musculoskeletal system.
Standards already play a prominent role in what is currently the most important use of
knowledge technology in medical applications: description of anatomical features, pathologies
and protocols. Examples are the Unified Medical Language System (UMLS), the Current
Procedural Terminology (CPT), and the Foundational Model of Anatomy (FMA). These standards
serve different purposes such as training and education, communication among practitioners,
statistical analysis for health insurance, preventive medicine and legal issues.
Current standards are primarily required to ensure the continuity of medical imaging data
and the sharing of those data among care and research centres. The development of standards
will most likely continue fostering the mutual enhancement of different imaging technologies.
Standards for 3D medical content are however currently lacking, although they are likely to
have in the near future a fundamental role in the field of computer-aided medicine where
patient-specific anatomical modeling is of increasing importance. Similarly, the development of
standardised procedures (including geometry and knowledge) for goal-oriented modelling and
processing will be of increasing importance both for the sake of quality and for legal issues,
when knowledge-intensive modeling and processing techniques will be routinely used.
01/04/2010 19
20. FOCUS K3D D1.4.1
3.2 Bioinformatics
While computer science is concerned with the development of
(generic) algorithms, which in general may be applied in a variety of
settings, biology is more of a system-centric activity, since in general
a panel of methods are used to investigate a particular system
(molecule, cell, tissue, organ, organism). In biology, knowledge
technologies are especially suited to structure the knowledge relative
to a particular piece of this multi-scale puzzle, as well as to integrate
data across the different scales. In this broad context, knowledge
technologies in general and ontologies in particular are characterised
by two main features. First, their development is under the guidance of biologists, since a
precise understanding of the systems under investigation is necessary in order to select
relevant annotations. Second, the knowledge technologies used cover the full range, from
simple databases all the way to complex ontologies, as evidenced by the numerous tools which
are made available from the web portal of the Protein Databank in Europe (PDBe).
If one narrows down the application scope of knowledge technologies to applications dealing
with 3D shapes, the natural setting in biology is that of structural bioinformatics. With
structural biology being the sub-domain of biophysics and biology concerned with the
investigation of the connection between the structure of macromolecules and their function,
structural bioinformatics is concerned with the development of methods and algorithms meant
to complement experiments in this perspective. Structural biology poses a number of difficult
problems, among them the acquisition of reliable physical signals, the reconstruction of
macromolecular models from these signals, the description of these models so as to design
parameters best describing their biophysical properties, and finally, the utilisation of these
models for prediction. In particular, predicting the 3D shape of a protein (the folding problem)
and predicting the interaction of two or several partners (the docking problem) are clearly two
of the main challenges in structural biology.
When working on these challenges, since the shape of molecules is so important, all
scientists resort to 3D content. This 3D content consists of a mix between geometry and
biochemistry. The geometric side corresponds to molecular representations and conformations,
either represented as a collection of balls, using Cartesian coordinates, or using internal
coordinates (bond angles, valence and dihedral angles). On the knowledge side, one specifies
the types of atoms and bonds, as well as a number of biochemical annotations (positional
fluctuations for experimentally resolved structures, known partners of a protein, affinity
constants, diseases the protein is involved with, etc.). The complexity of the concepts dealt
with can again be seen from the various tools, which are accessible from the web portal to the
European PDB at http://www.ebi.ac.uk.
From a modelling perspective, the topics of interest depend on the application scenario.
Experts in reconstruction from experimental measurements wish to output a model coherent
with the physical measurements. Scientists designing drugs are especially interested in
pockets on protein surfaces, since these accommodate drugs. Similarly, the affinity and
specificity of protein–protein interactions are tightly coupled to geometric and biophysical
properties of the molecules. Biologists wish to understand how the structure of molecules
accounts for their function. This requires making progress on the two aforementioned
mechanisms, namely folding and docking, which involves manipulating a mix of quantitative
and symbolic information. We substantiate this claim by examining four situations, which
exhibit a wide range of difficulties:
• Characterising the packing properties of atoms. Native forms of proteins, that is
biologically active forms, exhibit very specific patterns in terms of spatial properties, in
particular regarding the packing properties of atoms. Packing properties are obviously
numeric properties (for example atomic volumes). On the other hand, annotations of
proteins are key: well-packed proteins are called native like, while proteins with loose
packing are called decoys. In between these two extremes, a wide variety of situations
occur, and some of them are particularly important for the description of diseases (e.g.,
01/04/2010 20
21. FOCUS K3D D1.4.1
misfolding in amyeloid diseases);
• Computing the depressions and/or protrusions of molecules. When docking two
molecules in a blind fashion, i.e., without any a-priori knowledge, knowing where the
binding sites is a must. Depressions/pockets and protrusions/knobs on molecular
surfaces are good hints. In general, a pocket is defined in geometric terms (e.g., a
concave region of sufficient volume, or the volume corresponding to the stable manifold
of a maximum of the distance function to the molecular surface, etc.), but such
quantitative models are not easily handled, and annotations such as flat pocket, a
pocket with two crevices, etc. would help the design process.
• Developing proper notions of curvature, in conjunction with solving models and binding
modes. Curvature inherently refers to differential geometry, and there are indeed
several strategies to quantify the curvature of a molecular surface. Aside from these
quantitative statements, the qualitative description of regions of molecular surfaces in
qualitative terms such as flat, convex, concave, saddle-like, etc. would help biochemists.
• Partitioning a molecule into regions, which are likely to be flexible and rigid. Such
regions are of utmost interest for flexible docking algorithms. Rigidity characterisation
typically resorts to normal modes – the eigenvalues and eigenvectors of the internal
energy of the molecular system about a minimum. These quantities are used to define
molecular motion involved in deformations coupled to docking. Practically, annotations
such as hinge motion, shear motion, etc. are more easily handled than the underlying
geometric descriptions.
Documenting the life cycle of 3D objects manipulated in bioinformatics naturally depends on
this context. We shall discuss three examples illustrating the complexity of the documentation
problem.
Consider first a biophysicist who is experimentally solving the 3D structure of a protein:
he/she aims at producing a final product, namely the 3D structure of the molecule. The
structure submitted to the Protein Data Bank contains specific pieces of information, namely
those imposed by the PDB file format. However, the difficulties handled along the process may
or may not be documented. To make a long story short, a biophysicist interested in solving at
the atomic level the structure of a given protein using X-ray crystallography has to go through
the following steps: (i) introduction of the gene coding for this protein into a host cell, (ii) large
scale protein expression and purification, (iii) crystallisation, i.e., crystal growth, (iv) X-ray
diffraction analysis of this crystal and model reconstruction. Each of these steps might be
challenging and may require years of work. Experiments along the way are written down in the
Laboratory Information Management Systems, which is the memory of the process followed.
Still for complex cases the whole process might be more similar to a handcrafted activity, and
rationalising might be very difficult. It is also important to note that a person who managed to
carry out a complex process will first exploit it before releasing the details.
Consider now a person dealing with molecular modeling, who is working on the problem of
inferring a putative complex from two unbound partners. Key questions contributing to the
ability to run a reliable prediction are: which evidence of interactions between homologous
proteins does one find in the biochemistry literature (this requires a thorough literature search,
together with a precise compilation of experimental conditions)? What are the key regions in
the interaction? Are additional molecules involved to stabilise the interactions (ions or water
molecules)? Are there flexible regions? What is the role of electrostatics – which often plays a
crucial role at early stages of the binding?
These questions involve a subtle mix of relatively simple and complex tasks. Simple
questions are those related to the search for proteins within a given homology threshold with
respect to either a given reference protein, the classification and the search of protein folds,
the search of protein atoms with prescribed biophysical properties (e.g., temperature factors),
or the search for bibliographical pieces of information related to a molecule or a complex.
01/04/2010 21
22. FOCUS K3D D1.4.1
These tasks clearly benefit from knowledge technologies and can be easily documented.
Nevertheless, there are other and clearly very difficult questions, for example handling flexible
regions, computing precisely free energies, handling mutagenesis data, and understanding the
precise connection between coarse and atomic modeling results. These questions in their full
generality are still open. The route taken to produce a prediction cannot be fully and easily
traced and there is no standard way to rationalise and document it.
Finally, consider a pharmaceutical company designing a drug. The docking process just
described is now a mere step in a much longer process. To begin with, the chemist will aim at
identifying candidate drugs for the disease of interest. Drugs are small molecules, which
underwent a wealth of experiments in the chemistry and biochemistry communities over the
past five decades. The zoo of small organic molecules is rather stable. Simple rules involving
the molecular weight or the solubility of a drug allow one to infer pretty reliably the chances
for a molecule to be a lead – cf. Lipinski's rule of five (Lipinski et al. 2007). These pieces of
information can be rationalised and organised into ontologies, and can also be easily traced in
Laboratory Information Management Systems.
The second step, docking, is more difficult to handle as seen above. Finally, upon identifying
good hits thanks to a docking prediction, wet lab tests must be carried out, and finally clinical
trials implemented. Overall, the whole process lasts more than a decade, and there is no
standard way to trace it thoroughly. In fact, the Product Life Management within
pharmaceutical companies is a major problem and certainly an obstacle to benefiting from the
experience accumulated over a large variety of projects. This observation motivates the
strategy of companies such as Dassault Systemes: having to some extent mastered the life
cycle of manufactured objects, Dassault Systemes now aims at providing their tools and
environments to pharmaceutical companies.
To conclude, there is no general and unified way to handle the documentation of the life
cycle of 3D objects encountered in bioinformatics and applications. In the most general setting,
our understanding is not yet sufficient to document and elaborate easily simple knowledge
technology models.
The semantic interaction found in structural bioinformatics is not different from that found in
other domains. As mentioned before, a number of tools have actually been developed and
made accessible. Let us mention three of them, which are available from the EBI institute at
http://www.ebi.ac.uk, and allow running queries on the Protein Data Bank. The rationale for
mentioning these three tools is that they are of increasing complexity in terms of knowledge
technologies.
The first one, PDBelite, allows the user to pre-define criteria to be used by a search engine.
The second one, MSDpro allows the user to define its own logical queries using a drag-and-
drop interface. Finally, the third one, MSDmine gives access to a complex data model allowing
the user to take full advantage of the features of a relational database. Interestingly, the
knowledge data made available from these services are rather elementary in terms of
geometry, that is, no advanced annotation of geometric nature is provided. This is consistent
with the complexity of the aforementioned mechanisms.
The role and status of visualisation in the field of structural bioinformatics is rather
interesting. On the one hand, molecules are 3D objects, and it is rather tempting to inspect
their geometry so as to guess what the binding patches might be, or to assess the
complementarities of the molecular surfaces of interacting molecules. On the other hand, the
binding of two molecules is a very subtle mechanism, with a subtle interplay between enthalpic
and entropic phenomena, and a mere 3D visualisation cannot account for these delicate
mechanisms. However, the visualisation of 3D structures is important to develop one's
intuition. It is also of prime importance to communicate with large audiences and promote this
kind or research.
Standards do not play a prominent role in structural bioinformatics. File formats, for
example those that are used in the Protein Data Bank, come with standards. However, as seen
above, the complexity of the biological processes involved, e.g., in the experimental resolution
of a protein structure, is such that typically non standard problems are faced, which calls for
01/04/2010 22
23. FOCUS K3D D1.4.1
an unstructured part in the files themselves. These difficulties and also the variety of scenarios
handled are clearly a hindrance to the exchange of information.
3.3 CAD/CAE and Virtual Product Modelling
CAD/CAE and Virtual Product Modelling cover all ranges of man-
made objects: electronics, transportation devices, tooling machines
and many more.
There are important differences between CAD/CAE and the other
fields considered in FOCUS K3D. Medicine and Cultural Heritage are
of global interest and collaboration amongst researchers, building
ontologies, etc., has a long tradition, whereas product-developing
companies face much harder competition amongst each other.
Although the human body can be looked at as 'the most complex
machine' known, it is just one type of object that Medicine and Bioinformatics are trying to
decode, understand and conceptualise/formalise fully.
In Virtual Product Modelling the objects (machines) modelled are much less complex than
the human body but the range of different types is enormous and cannot be easily mapped
into one scheme.
This explains to a certain degree the different status in terms of ontologies (in virtual
product modelling there are few) and standards in these areas. Nevertheless, the main
problems of CAD/CAE are still comparable to those of the other fields, e.g.:
• How to generate a model from a digitally acquired object?
• How to model a shape easily?
• How to model function or behaviour?
• How to retrieve a model with certain properties?
• How to handle all the information involved?
Virtual Product Modelling is part of the virtual product developing process. The 3D modelling
phase is typically fed with goal market characteristics, market/cost figures, requirements
(aesthetic, functional, environmental, safety, etc.), concept sketches, and so on and so forth.
Stylists and designers use CAD (computer-aided design) and CAS (computer-aided styling)
systems to create virtual 3D models that foremost represent shape. The product functionality
is usually modelled using a variety of behaviour/function modelling tools and is simulated with
corresponding solvers. In the process, physical models still need to be created for styling
reviews or to validate behaviour.
In the styling phase, the shape of physical models is changed manually, which requires
scanning the resulting 3D surface and re-creating a virtual model for it (re-engineering).
In the simulation phase data from real experiments are used to tweak material models (to
just name an example).
The data created along the process typically reside in a multitude of data sources with a
tendency to integrate product relevant data into a product data management or product
lifecycle management system. Today most of the content of those systems is restricted to
geometric 3D information, the whole semantics being in the heads of the engineers, and not
explicitly documented in a computer-interpretable way.
Although being the field where 3D media really started off (Sutherlands SketchPad 1965,
company-owned CAD systems in the 1960s, etc.), the software in use today still requires a lot
of manual intervention and tedious work and typically imposes certain concepts onto the user
instead of giving him/her full freedom in choosing the best-suited tools at will.
In the following we shortly present the state of play for the most important phases and
kinds of software tools and give hints on current limitations.
01/04/2010 23
24. FOCUS K3D D1.4.1
CAD systems enable designers to create geometric models of products with the computer,
to be reused and manipulated by the designers as needed. CAD systems were, and remain,
highly technical software with an extremely rich feature set and functions for detailed design
work.
CAE tools are widely used in the automotive and aerospace industry. In fact, their use has
enabled engineers to reduce product development cost and time while improving the safety,
comfort, and durability of their products. The power of CAE tools has progressed to the point
where much of the design verification is now done using computer simulations rather than
physical prototype testing.
Today’s shape representations in CAD (mainly NURBS) do not feed all purposes well,
especially not those of simulations. For designing, the concept of creating a larger base surface
from which certain parts are cut out by trim curves to create the final piece of surface is
certainly artificial – it has no natural equivalent. Other representation schemes are needed that
allow for more intuitive and easy shape manipulation and also enable easily integrating
concepts of function-oriented design, etc.
Simulations typically require a different kind of shape representation than provided by CAD
systems, a discrete instead of a continuous one. This implies a tedious manual process of
model preparation to suit different simulation needs.
The integration of CAD and CAE using FEA (Finite Element Analysis) is today very complex
due to the different principles for shape representation employed. In CAD, a volumetric object
is described by shells described by a patchwork of mathematical surfaces representing the
outer hull and inner hulls of the object, and data structures conveying the volumetric
relationships. There is no requirement that adjacent surfaces match exactly, only that they
match within specified tolerances. In FEA, the object has a complete mathematical description
through watertight structures of trivariate parametric volumes. There is a fast growing
research field in both the US and Europe concerning this discrepancy, addressing how to solve
better this great interoperability challenge of CAD and FEA. This research field can be denoted
as “Isogeometric representation and analysis” aiming at a common shape representation for
both CAD and FEA to integrate the two disciplines and drastically improve the quality of Finite
Element Analysis. Consequently, there is a need for investigating the concepts of this new
approach both from a purely mathematical and from a semantic perspective to integrate it into
current industrial information pipelines. Isogeometric representation and analysis employ new
concepts in both the shape representation phase and the analysis phase. We will need
ontologies that encompass these concepts and their relationships to the traditional concepts to
facilitate interoperability between the new isogeometric systems and traditional CAD and FEM.
Today, many companies use product data management (PDM) systems to store product
related information, such as geometry, assembly structure, and materials. Considering the full
product development process and the mechatronic disciplines (mechanics, electronics,
software), there exists much more information spanning from requirements, to specifications,
to mechanic, electronic and software behaviour models, to cross-domain dependencies,
influences and effects, largely carrying the semantics of the product and not represented in
PDM.
The question is whether all the data should be in one PDM system. A more promising idea is
to bridge the existing islands with semantic linking to manage knowledge effectively in order to
reuse past positive experiences and solutions and avoid repeating past efforts or errors. When
talking about knowledge in industrial companies, one of the important islands of information is
the one that deals with the knowledge of the geometrical properties of the products
manufactured, bought or integrated, and to be maintained during product changes. Therefore,
models and methods for making explicit and maintaining links between geometric properties
and domain specific knowledge have to be developed for improving both the specific process –
either computer-aided or manually performed by the operator – and also the inter-process
communication. This would be beneficial for the integration between design and simulation in
product development, and in simulation during maintenance.
Currently, searching in distributed information sources is hampering efficient work
01/04/2010 24
25. FOCUS K3D D1.4.1
procedures. When linking those information islands together, new search functionality is
needed that spreads out over these sources, takes shape, function, metadata and semantics
into account, and combines partial results to meaningful answers of the user’s questions. New
partial/local 3D retrieval approaches and combined searches contribute to overcoming the
current limitations of text-based or current global 3D content-based retrieval. Another aspect
is the appropriate visualisation and navigation through results being returned from combined
searches and user feedback for self-improving and self-learning mechanisms.
Another current limitation in the downstream process of virtual assembly is the tedious way
to create virtual kinematics and simulate this interactively within the virtual environment.
Nowadays kinematics simulation is performed in the domain of CAD and CAE software
packages, which need specially trained personnel and are designed to work on detailed product
models without providing the degree of realism and interactivity virtual environments offer.
Here, using semantics from CAD and parametric feature-based design for virtual kinematics in
a Virtual Conceptual Design (VCD) platform would ease the process and support the early
conceptual design phase as well as the embodiment design, allowing the immediate validation
of assemblies and mechanisms by experiencing their behaviour during the modelling,
simulation and assembling process.
3D model exchange between different systems has been and is still a great challenge with
respect to model quality, as different CAD-systems and CAD-system kernels employ several
approaches (and tolerances) to handle the inherit inaccuracies of STEP-type CAD-models.
CAD-model check and repair have been on the agenda for many years and resulted in a
number of standardisation initiatives:
• AIA/ASD EN-9300 Long Term Archiving (LOTAR) addressing High Quality Geometry
verification and validations and rules to execute.
• ISO 14721.4, Open archival information system – Reference model.
• ISO 10303-59 Product Data Quality (PDQ), Part 59: Integrated generic resource –
Quality of product shape data.
Since standards for CAD model quality are in place, the focus of advanced industry has now
turned to the validation of the CAD-models exchanged (e.g., the STEP file, or the CAD-model
generated by the receiving system), being an exact representation of or equivalent to the
CAD-model in the sending system. This demand for equivalence checking poses new
challenges to the CAD-model representation, to the semantic annotation of the CAD-model and
to CAD-vendors in general, as industry requires more than the basic CAD-technology, CAD
models, representation and algorithms to be available for validation.
The ‘standard file format problem’ has to be addressed by academia and research institutes
in the future. We need to develop a 3D file format addressing the following problems that are
still open:
• compatibility / interoperability / semantic mapping;
• shape representation (procedural, parametric);
• functional expressiveness;
• transport of objects together with their interactively modifiable parameters.
01/04/2010 25
