Gabriele Guidi is responsible for the “Computer Vision and Reverse Engineering Laboratory” at Politecnico di Milano (Italy). Since the late 1990s, it has focused on 3D acquisition and modeling techniques of cultural heritage artifacts on very small to very large scales. An interesting quality of a polytechnic institution like the one in Milan is to have in its DNA both a technical mind, coming from the Engineering departments, and a humanistic soul, linked to its departments of Architecture and Design. This dual point of view is critical when applying advanced technologies such as 3D data capture, opto-electronics, image processing, metrology and computer graphics to 3D documentation of a cultural artifact in a way that is useful for archaeologists, architects and officers of institutions responsible for the conservation of cultural heritage.
This presentation introduces the research group at Politecnico di Milano and presents an overview of the technological evolution of 3D capturing techniques since 2000. Several major examples of the researches done are shown, as well as how such discoveries have been applied to concrete problems of cultural heritage documentation and visualization. In the conclusion some of the major challenges we intend to confront in the near future are mentioned.
Bajaj Allianz Life Insurance Company - Insurer Innovation Award 2024
3D Acquisition and Modeling in Cultural Heritage
1. Indiana University, Bloomington (IN) USA - Nov. 21, 2014
GABRIELE GUIDI, PHD
DEPT. OF MECHANICAL ENGINEERING
POLITECNICO DI MILANO, ITALY
3D ACQUISITION AND MODELING
IN CULTURAL HERITAGE:
EVOLUTION AND PERSPECTIVES
2. 3D FOR CULTURAL HERITAGE
RESEARCH GROUP
Gabriele
Guidi
Coordinator
Electronic
Engineer
Michele Russo
Temporary
Researcher
Architect
Laura Micoli Post Doc Architect
Davide
Angheleddu
Phd Student Architect
Sara Gonizzi Phd Student Archaeologist
+ 1 to 3 intern students and 1 to 3 thesis students depending on the period
3. RESEARCH ACTIVITIES
• Integration of Passive/Active technologies
• Characterization of 3D acquisition technologies
• 3D post processing
• Applications of 3D acquisition and modeling to:
• Cultural Heritage (reality based & reconstruction)
• Industrial applications
4. A BIT OF MY/OUR STORY
1988!
•!GG: Master degree in Electronic Engineering, Univ. Florence!
•!Thesis on real-time signal processing of Doppler signals!
1992!
•!GG: PhD in Bioengineering, Univ. Bologna!
•!Thesis on measurement of blood speed in 3D!
1998!
•!Marc Levoy scans the David by Michelangelo!
1999!
•!Parnaso project!
•!First experiments with 3D scanning of CH at Univ. Florence!
2000!
•!First large 3D scanning project at the Univ. of Florence!
•!Maddalena by Donatello!
5. MADDALENA BY DONATELLO
• Sculpted in 1455 approx.
• Height 180 cm
• Width 40 cm
• Complex shape involving shades and fragmented
range maps
• Wooden statue originally gold coated: currently
dark with reflective spots (optically non
cooperative)
7. EQUIPMENT USED
• Generates 3D images (range maps)
• Working principle: triangulation
• Pattern projection of vertical strips
MEA S U R ING R A N G E 0 . 5 - 1 . 2 M
MEA S U R EMENT UNC E R TA INT Y 0 . 0 5 - 0 . 2 MM
S E N S O R S I Z E ( P I X E L ) 7 6 8 X 5 7 6
MEA S U R EMENT R E S O LU T ION 0 . 5 - 0 . 1 MM
8. PROJECT PLANNING
•! First stage: model skeleton!
–! Required resolution: 0.4 mm!
–! Framed field (focal plane): 30x23 cm!
–! Uncertainty along z: 0.125 mm!
–! Volume divided in 11 stripes 23 cm tall,
vertically overlapped (29%)!
–! Each stripe divided in 8-10 images (range
maps) horizontally overlapped (~30%)!
–! Supplemental images for hands, legs and
arms!
•! Second stage: Hi-res model!
–! Final required resolution: 0.25 mm!
–! Framed area: 19x14 cm!
–! Uncertainty along z : 0.070 mm!
12. SUMMARY
Uncertainty!70-125 "m!21 "m!21 "m!
Triangles!4.6 M!1.2M!724 k!
Size (Mbyte)!115.3!30.89!19.65!
2.64!
Total data
(Gbyte)!
374!13!23!
Number of
range maps!
Full model!Face!Foot!
Phase 1!
!200 range maps, 170 used!
!155 work hours!
Phase 2!
205 additional range maps!
160 man-hours!
13. QUALITY CHECK
• At this stage an acquisition work was usually
considered completed
• In our project a quality control was arranged in order
to check the metric reliability of the whole model
• A complementary method was used in order to
achieve such purpose: photogrammetry
14. d1 – d5!
Agreement between 3D
and photogrammetry!
d6 – d8!
Positive deviations
worst case: 4.3 mm (0.25 %)!
d9 – d11!
Negative deviations
worst case : -4.2 mm (1.66 %) !
d1!d2!
d8!
d3!
d4! d5!
d6!d7!
d9!
d10!d11!
15. 2001!
•!Visiting Researcher at NRC Canada with Angelo J. Beraldin!
•!Integration of photogrammetry and 3D scanning!
16. TARGET EXTRACTION MIXING
2D AND 3D INFORMATION
Geometry! Texture!
xt, yt!
3D
plane!
Geometry
Projection! x,y,z!
17. 3D model!
Alignment and
merging!
A!
B! C!
D!
Photogrammetric
X Y Z coordinates!
A!
B!
C!
D!
Roto-translation
matrices!
3D images in the
photogrammetric
coordinate system!
Quaternion!
A!
B! C!
D!
18. FINAL CHECK
• Mesurements on the new model were coherent with
photogrammetry: the new model grown in height of few
millimeters
• By comparing the two models other lateral unexpected
distortions became evident
19. LESSON LEARNED
• the usual approach for creating 3D models from small
range images may involve a loss of metric accuracy
even when the single images are highly accurate
• A sensor fusion between the two methods allowed to
overcome the alignment problems
• As a general criteria 3D scanning should always be
coupled to a complementary measurement method at
least for checking global accuracy
20. “ADORAZIONE DEI MAGI” BY LEONARDO DA VINCI
Grey coded
pattern projection
range camera!
2002!
•!First 3D acquisition and modeling of a wooden painting!
21. 3D MODEL OF THE PAINTING
393 range maps, H&V res= 0.3 mm! 222 range maps, H&V res=0.4!
23. LESSON LEARNED
• High resolution dimensional monitoring appears to be
extremely useful for applications in wood restoration,
specially when it is the support of a delicate painting
• However, due to the natural deformations of wood,
the possibility of repeating the same monitoring in
different times seems a key feature for gaining the
information needed by restorators
24. 2003!
•!First 3D scan with a Laser Radar in the CH field!
Pietà (Michelangelo)
1997, IBM
Pattern projection (triangulation)
Madonna col Bambino (G. Pisano)
1997, Univ. Padova / NRC Canada
Laser scanning (triangulation)
David (Michelangelo)
1999, Stanford University
Laser scanning (triangulation)
Maddalena (Donatello)
2001, Univ. Firenze, NRC Canada, Optonet Srl
Pattern projection (triangulation)
25. TOF VS. TRIANGULATION
(METROLOGY)
Measurement uncertainty
Triangulation
range device
0.1 mm
Time-of-flight
range device
4-8 mm
27. METROLOGY IMPROVEMENT
Measurement uncertainty
Triangulation
range device
0.1 mm
Time-of-flight
range device
4-8 mm
Frequency modulated
Laser Radar 0.1 mm
28. 3D DATA ALIGNMENT
Triangulation based camera
• Mostly local to the camera
• Range maps have to be aligned by means of semi-automatic
procedures (ICP)
• Range maps have to be redundant in order to make ICP work
FM laser radar
• All 3D data are directly re-oriented in a global reference system
thanks to special targets over the scene (metallic spheres)
29. MODEL GENERATION PIPELINE
Triangulation
sensor!
3D scanning!
ICP!
method!
Camera referenced!
Range maps!
Aligned range maps
(referenced to a single
coordinate system)!
Merge! Polygonal
model!
FM Laser Radar!
3D scanning!
Range maps !
Referenced to a !
single coordinate system!
Merge! Polygonal
model!
30. DAV I D B Y
DONATELLO
• Height 160 cm
• Located over a 1m basement
Critical points
• Non cooperative material
• Hidden surfaces
Lateral resolution needed
• 1mm on low curvature surfaces
• 0.5mm on compex surfaces
31.
32. • System capable to work
through a Front Surface
Mirror (FSM)
• From the same point of view
front and rear points can be
captured
• 80 hours for acquisition
• 20 hours for merge &
preliminary editing (much
less than in previous project!)
34. LESSON LEARNED
• Acquisitions from a single point of view dramatically
enhance the amount of surface captured in a single
acquisition
• The possibility to use mirrors further increases this
feature, solving also problems of data alignment in
objects with small thickness
• Metallic rectified sphere added on the scene allow
automatic 3D data orientation
35. 2003-6!
•! 3D acquisition of a large and detailed object:“Plastico di Roma Antica”!
•! CAD remodeling on the scanned data: “Rome Reborn”!
36. MOTIVATIONS
• Digital Roman Forum project
(Frischer et al. 1999-2003)
• Rome Reborn project (Frischer
et al. 2004-2008): extend this
virtual model of ancient Rome
up to the exterior walls
• Idea: reverse engineering
Gismondi’s “plastico” for
creating a good starting point
• Updated with the most recent
archaeological discoveries
37. 3D DIGITIZATION CONSTRAINTS
17.4 m
16.0 m
• No measurement machinery
flying over the “plastico”
• Long range (7-24 m)
• Wide area (about 200 sq. m)
• Small buildings (2-20 cm) Low
uncertainty (<0.5 mm)
• Balcony pavement at 2.7
meters respect to the model
• Balustrade 1.2m high
1.2 m
20 cm
5 cm
! 3mm
Plaster
plane
Balcony
plane
2.7 m
24
7
Observation
point
Plaster plane
Balcony
plane
38. METRIS
LASER RADAR
• Known: same equipment used for the
David’s work
• Range = up to 24 m
• Uncertainty (1σ): 300 μm (metrology mode)
• Framed area: 360° H x 90° V (from -45° to
+45°)
• Beam spot size = 400 μm with automatic
refocusing (metrology mode)
• Stacking mode: reduce uncertainty
averaging repeated measures (metrology
mode)
➜ Metrologically Ok
39. …BUT, WHAT ABOUT SPEED?
• Triangulation range device: >150 000 points/s
• TOF range device: > 20 000 points/s
• Laser radar in metrology mode: 1 point/s (!)
➜ time for one complete scan: 40 days (nights included).
Not feasible!!
40. SYSTEM
CUSTOMIZATION
The most time consuming
activity in metrology mode is
refocusing ☟
• Scanning on circular
scanlines
• Focusing only once (at the
beginning of each scan line)
• Stacking level optimized for
the best tradeoff
41. FEATURES OPTIMIZATION
(mm)
No averaging
Average on 2
Average on 5
Average on 10
• Several averaging test
were made using planar
targets
• Best tradeoff: average on 4
values
• !=0.3 mm
• Speed: 170 points/s
42. SYSTEM SETTINGS
• 2 mm resolution
• 0.3 mm uncertainty
• 200 m2 per scan
• 50 millions of points per scan
• Registration with external targets, no need for redundancy
➜ Time for a complete scan of the “plastico”: 4 days
(nights included). Not fast but feasible
43. TYPICAL SCANNING SESSION
1. Locate the scanner in place
2. Measure targets for determining scanner position
3. Measure the plaster perimeter from that particular
location
4. Off-line calculation intersections between circular
scan-lines and the perimeter ➭ pass them as input of
the custom control software ➭ start scanning
44.
45. TYPICAL SCANNING SESSION
1. Locate the scanner in place
2. Measure targets for determining scanner position
3. Measure the plaster perimeter from that particular
location
4. Off-line calculation intersections between circular
scan-lines and the perimeter ➭ pass them as input of
the custom control software ➭ start scanning
46.
47.
48. TYPICAL SCANNING SESSION
1. Locate the scanner in place
2. Measure targets for determining scanner position
3. Measure the plaster perimeter from that particular
location
4. Off-line calculation intersections between circular
scan-lines and the perimeter ➭ pass them as input of
the custom control software ➭ start scanning
49.
50. TYPICAL SCANNING SESSION
1. Locate the scanner in place
2. Measure targets for determining scanner position
3. Measure the plaster perimeter from that particular
location
4. Off-line calculation intersections between circular
scan-lines and the perimeter ➭ pass them as input of
the custom control software ➭ start scanning
51. PLANNING
First stage
• Acquisition from 3 locations on the balcony for a first massive data
capture
Second stage
• Searching several optimal locations for small data integrations
• Actual acquisition
• Data merge
• Editing
52. FIRST STAGE
• Laser radar only
• 3 locations on the balcony
• Blind areas below 45° (to
be integrated)
• 12 days total scanning
time
Blind areas
53. DATA SUBDIVISION
AND MESHING
• Each scan 50 MPoints
• huge data set, not
manageable at that time
(2004-5)
• sets 2m x 2m blocks
generated with a 3D grid
• aligment made globally,
integration and meshing
singularly on each block
54. SECOND
STA G E ( 1 )
• Laser radar for integrations
of the central area
• 1 more locations from the
balcony (4)
• 6 locations at ground level
(5-10)
55. SECOND
STA G E ( 2 )
• Minolta Vivid 900 sensor
• Range maps all around the
Aurelian Walls
• Integrated with LR data
through ICP alignment
56. AT THE END OF SUCH PROCESS THE WHOLE MESH WAS COMPLETED
57. R E A L V S .
DIGITIZED
• resolution and uncertainty
chosen resulted sufficient
to detect all the details
• the result was significant
considering the technical
and logistic difficulties
• however…
58. DRAWBACKS
• a lot of occlusions " these nice
meshes required a considerable
amount of editing work
• a mesh is still a mesh (e.g. e
static representation of a 3D
geometry)
• LOD might be implemented up
to acquisition resolution, while
in a VR application closeups
might be needed
! remodeling over the mesh
Edited mesh
Simplified unedited mesh
59. REMODELING
THE MESH
• different approaches are
possible
• very different in terms of
time-consumption and
visual result
• the squared area has been
processed differently in
the next slides
67. REMODELING CHOICES
• remodeling all at the maximum level of details would
have required 1 week x about 7000 buildings: not
feasible
• a simplified approach could have been acceptable for
the simpler structures, not for the monumental
buildings, hoverer still time-consuming!
68. TWO CATEGORIES OF BUILDINGS WERE IDENTIFIED
Urban fabric
Monumental buildings
69. URBAN FABRIC HAS TO BE MODULAR…
• The extension of the
model and the relatively
short time needed for sure
an optimized assembly line
• Monumental buildings
developed singularly
• Urban fabric developed
with archetypes
70. OTHER CLUES
• Gismondi left few
documents
• however some preparatory
drawings have been found
• they shows domus types
studies
71. Approach #1: search of recurring elements and
Maya modelling of a limited set of modules
(library)
–JANEZ DONNO, MASTER THESIS (2006)
74. RECURRING ELEMENTS IN BUILDINGS
• Pattern analysis on
horizontal and vertical
sections of the mesh
• Classification of similarities
75. RESULTS
• About 20 types of elementary building archetypes
employed for 90% of the physical model
• Used in the “Plastico” with variation of scale and in
different combination hiding geometric repetitions
77. Approach #2: search of recurring elements and
procedural modeling of a class of buildings
(object oriented)
–IGNAZIO LUCENTI, MASTER THESIS (2007)
78. PROCEDURAL MODELING
• Sofware used: Side Effects Houdini
• General purpose procedural modeling package
• Every item is considered as a flow of data and can be
manipulated through a network of operators
• Users can make their own custom operators and
custom “prototypes” (called digital assets)
80. BENEFITS OF THIS APPROACH
• Models are made up of reusable parametric modules (e.g. a column asset
can be used in every object that contains a column)
• Updating the model became very easy because only the asset needs to be
modified and all the instances are updated accordingly
• For example, to update the temple models, add a texture or a new
parameter, user needs to modify the prototype only and the changes will be
reflected in every existing temple
• It is possible to have different versions of the model, switching them
automatically (e.g. different levels of detail based on camera distance)
• Object parameters can be controlled manually, by algorithms, by data
sources (database) or even by image maps
81. WORKFLOW
• Definition of the object parameters (analysis)
• Making a parametric model of the object
• Turning it into a Digital Asset (prototype) with its own
custom interface
• Placing instances of the prototype on the 3D model of
the “plastico” (manually or driven by a rule)
86. DTM FROM THE PLASTICO MESH
• All the buildings in the 3D scan have been deleted
• All the consequent holes filled
• The resulting mesh have been sliced in order to
separate river, land, and paved roads (for assigning
them different shaders)
87. ME RGE IN S INGL E DIGI TAL MODE L
Included:
• DTM (original mesh)
• vernacular buildings (library)
• Temples (procedural)
• Bridges (procedural)
• Walls (procedural)
88. Such 3D model, once integrated by Bernard Frischer’s
group with the various high-detail models of monumental
buildings (Forum, Coliseum, Circus Maximum, etc.),
became the model known as Rome Reborn 1.0
It was also the starting point of the following fully
procedural versions of RR, based on CityEngine:
http://romereborn.frischerconsulting.com
89. LESSON LEARNED
• Laser Radar technology can solve the difficult task of
acquiring large artifacts with small details
• The acquired data is always valuable but sometimes it
is visually not sufficient for virtual reality
• In that case the right post-processing approach may
change dramatically the time needed for completing
the model
90. 2007-8!
•! 3D survey of the Pompeii Forum (Scuola Normale di Pisa)!
•! POLIMI coordination, 3D scanning and modeling, integration, rendering!
•! FBK Trento contributing with photogrammetry!
Large area:
• 150 m maximum length
• 80 m maximum width
• 8 large structures included
• 377 small finds spread all
over the area
N
150 m
80 m
91. • Level of detail ranging from the geographical scale to the object scale
• For each scale the more suitable survey technology was adopted
• The consequent resolution varied from 25 cm to 0.2 mm
92. LOW-RESOLUTION (GEOGRAPHICAL AREA
FRAMING)
Digital surface model (DSM) of about 1
square km around the forum:
• existing aerial Images for geometry
capture
• 1:3500 photogram scale
• geometric resolution: 25 cm
Acquisition of single points for image
registration
• GPS
• Standard topographic approach
Texture mapping from above
• Pictometry images
• 15 cm texture resolution
93. MEDIUM-RESOLUTION (LARGE STRUCTURES)
Leica HDS 3000 laser scanner
for long-range acquisitions
(3D framing of the forum)
Resolution 5-20 mm
Leica HDS 6000 for fast and
massive acquisitions (3D
acquisition of areas with
many occlusions)
Resolution 5-10 mm
Close range
photogrammetry digital
reflex cameras with manual
processing
Resolution: dynamically changing
94. HIGH-RESOLUTION (DETAILS)
Close range photogrammetry with digital
reflex cameras and automatic matching
(ETH multi-photo matcher)
Resolution: up to 0.5 mm
95. CATALOGUING THE RUINS
• Each geometrical entity was
identified, catalogued,
photographed and coded
• All the following work has been
referred to such IDs for image storing
and models management
96. LASER SCANNING
• 10 days of scanning in two stages
• 1.2 G points acquired
• 100 M points used for modeling (1:10 ratio)
• Heavy hand cleaning for deleting artifacts (visitors, spurious data)
• ICP alignment
• Sorting and subdivision with two outputs:
• General reference for the whole model
• Single sets of data for each structure
97. DIGITAL PHOTOGRAMETRY
• 3200 images acquired with precalibrated cameras
• Photogrammetric models metrically generated in their own reference
• Aligned with the 3D scanned reference in the final integration stage
100. 2011!
•!3D survey and modeling of Temples in My Son (Vietnam)!
•!Virtual reconstruction with strong integration between actual
3D data and other sources!
3D scanning with Faro Focus 3D
Image based 3D and textures
104. 2012!
•! 3D survey and modeling of Certosa di Pavia (Italy)!
•! Virtual reconstruction of several historical phases based on integration
between real 3D and historical sources!
105. 2012-15!
•! EU project 3D-ICONS: 3000+ models for EUROPEANA!
•! POLIMI: massive digitization of 527 items!
•!Wide use of automatic photogrammetry based on SFM!
107. 2007-!
•! Metrologic analysis of 3D devices and methods!
!"#$%&'$('#%)*+,-#
./#%)&(0#
1/#%)&(0#
2(3$4,(%)(*#
5&*)4$&6#7)+&3$,4#
89)4&*,4#0:$66#
;(<)4*&$(*=#
!"#$%&'()*+,-.,)*/0'1'2$%*30*45$1'(1*$(&*67&"%'(18*.*5"2#7%71'9$%*$::#7$9;<)*='5"*>75:#"??'7(*
="9;(7%71'"?*6$1$@'(")*33-3A*BCDDEF*
108. ACCURACY, PRECISION &
UNCERTAINTY
!"#$%&''
())%)''!'
./-0(&"0,+' *)(+,-,%#'
1))%)''!'
2++3)"+/'
• In “modern” metrology
uncertainty incorporates
both concepts
• Useful in standards for
acceptance tests
• Not for separately
analyzing systematic and
random error components
109. TEST OBJECTS FOR 3D
NRC, Canada
NPL, UK
POLIMI, small volumes
!"##$%%$
!"##$%%$
64 mm
1 mm
!"#$%#
!&#$%#
POLIMI, large volumes
111. METROLOGY FOR CH MODELING
Laser scanned reference Ref. vs. Photogrammetric model
- Agisoft Photoscan SW
- Nikon D90
- Mean error = 1.18 mm
- Std. dev. = 3.38 mm
Ref. vs. Photogrammetric model
- Autodesk Recap Web service
- Nikon D600
- Mean error = 0.34 mm
- Std. dev. = 1.80 mm
Gabriele GUIDI, Bernard FRISCHER, Photomodeling vs. traditional 3D data capture of cultural heritage artifacts, Conference on
Cultural Heritage and New Technologies, November 3-5 2014, Vienna, Austria
112. CONCLUDING REMARKS
• In order to solve complex problems you have to go deeply into them.
Many of the CH models shown could not be feasible if electronic
engineering, informatics, archaeology, statistics, architecture,
geomatics, computer graphics and metrology would not have interacted
positively. The keyword is interdisciplinarity intended as action giving a
result larger than the sum of the single disciplinar contributions
• The 3D model is important but often it is not enough. In many case it is just
a (fundamental) starting point for a documentation activity that necessarily
involves an enrichment of such models, both geometric (3D semantics),
visual (computer graphics) and informative (metadata & ontologies)
• Similarly the 3D model can be used for communication purposes where the
main issues are related with both local and remote 3D visualization
(including virtual reality and augmented reality)
113. CONCLUDING REMARKS (2)
• The technologies seen show that many of those models required months to
be created. Although any experimentation is important it is clear that the
future of 3D documentation can’t be that. It has to be quick! Only in this way
it will be possible to handle problems of massive 3D digitization. Image
based modeling integrated with laser scanning and smart 3D post
processing techniques seems nowadays the most promising way
• The quality of what your 3D data indicates what you can do with them. The
traceability of the whole 3D acquisition pipeline (sensor, process, 3D model)
is fundamental for a scientific use of 3D
• The same concept can be extended to any 3D modeling activity in CH,
including reconstructive modeling of something not anymore existing
(philological traceability), obtained through a wise use of metadata
documenting the process and the sources for generating the 3D model
114. CREDITS
• Carlo Atzeni (Emeritus, retired from University of
Florence, Italy)
• Jean-Angelo Beraldin (National Research
Council, Ottawa, Canada)
• Bernard Frischer (University of Indiana,
Bloomington, USA)
• Fabio Remondino (FBK, Trento, Italy)
• Alessandro Spinetti (Florence Engineering,
Florence, Italy)
• Tommaso Grasso (3dHPM, Rome, Italy)
• Sara Lazzari (formerly Optonet, Brescia, Italy)
• Grazia Tucci (University of Florence, Italy)
• Monica De Simone (Director of Museo
Archeologico di Rieti, Italy)
• Claudia Angelelli (Università degli Studi di
Padova,Italy)
• Salvatore Barba (University of Salerno, Italy)
• Carlo Bianchini (University of Rome “La
Sapienza”, Italy)
• Maurizio Seracini (UC, San Diego, USA)
• Federico Uccelli (Leica Geosystems, Lodi,
Italy)
• Achim Lupus (Leica GEOSYSTEMS AG,
Switzerland)
• Patrizia Zolese (Fondazione Lerici, Rome, Italy)
• Mara Landoni (Politecnico di Milano, Italy)
• Sebastiano Ercoli (Politecnico di Milano, Italy)