The document discusses technologies for cultural heritage applications, focusing on accurate digitization of sites and artifacts. It describes how modern sensors produce large amounts of data that are difficult to process, store, distribute and visualize at scale. Research is needed into scalable enabling technologies for acquisition, geometric processing, visualization and more. The goal is to efficiently acquire and process 3D models' geometry and color from sites and objects, and to effectively archive, distribute and visualize the resulting models.
2. R. Pintus – CRS4/ViC, October 2012
Tecnologie per i beni culturali
• Focus: digitalizzazione accurata (forma e
colore) di siti e manufatti + …
– Partire dai dati: Acquisizione -> Trattamento !
– Modelli misurabili
• Molti usi oltre la visualizzazione
– Riproduzione materica
– Studio di opere d’arte
– Documentazione in-situ di scavi archeologici
– Supporto al restauro e sua documentazione
– Valorizzazione
3. R. Pintus – CRS4/ViC, October 2012
Tecnologie per i beni culturali
• Le quantità di dati prodotte dai moderni sensori
sono però difficili da trattare, archiviare,
distribuire, visualizzare
– Scalabilità!
• Tecniche attuali sub-ottimali
– Costi, tempi, qualità
• Bisogno di ricerca in tecnologie abilitanti
scalabili
– Acquisizione
– Processamento geometrico
– Visualizzazione
– …
4. R. Pintus – CRS4/ViC, October 2012
Tecnologie per i beni culturali
• Come acquisire e processare efficacemente forma e colore di
siti e manufatti?
– Tecniche di fusione multi-sensore, stream-processing, multiresolution, external
memory algorithms, parallel programming, GPGPUs
• Come archiviare e distribuire efficacemente i modelli?
– Multiresolution, adaptive streaming, compression
• Come visualizzarli efficacemente?
– Multiresolution, adaptive rendering, out-of-core methods, GPU programming,
parallelization, rasterization, ray-casting
• Come esplorarli?
– Novel 3D displays, specific interaction techniques
– Portable devices
5. R. Pintus – CRS4/ViC, October 2012
Alcuni esempi
• Allineamento geometria/colore
• Colorazione di modelli 3D
• Fusione di dati e ricostruzione geometrica
• Visualizzazione scalabile ed interattiva
• Distribuzione di dati in rete
• Esplorazione su display innovativi
(… e molto altro)
7. R. Pintus – CRS4/ViC, October 2012
Modelling vs Acquisition
Modelling
Subjective Reality
Acquisition
Objective reality
7
8. R. Pintus – CRS4/ViC, October 2012
3D Reconstruction
• Acquire geometry and color
• A lot of techniques
– Structured light, laser scanning (triangulation or
time-of-flight), photometric stereo, shape-from-X,
…
• Which technique?
– Object type (big/small, material….)
– Cost
– Accuracy/Resolution
– Time
– Complexity
8
9. R. Pintus – CRS4/ViC, October 2012
Outline
• 3D Reconstruction Techniques
• 3D Reconstruction Pipeline
– Photo mapping/blending
– Printing
• Case study
9
37. R. Pintus – CRS4/ViC, October 2012
Cultural Heritage
• Techniques
– Triangulation (laser scanner)
– Time of Flight
– Texture Mapping
– Multi-view reconstruction
– Photometric Stereo
• Deal with multiple acquisitions
• Manage a huge amount of data for visualization
purposes
37
38. R. Pintus – CRS4/ViC, October 2012
3D Reconstruction Pipeline
Real Object Acquisition Devices
Photos
3D Digital Model Geometry
=== Processing ===
-Cleaning
- Merging
- Photo Alignment
- Color Projection
-…
38
39. R. Pintus – CRS4/ViC, October 2012
3D Reconstruction Pipeline
• Real Model Inspection (onsite)
• Scans design (offsite/onsite)
• Acquisition (onsite)
• Alignment (offsite)
• Editing (offsite)
• Merge (offsite)
• Texture (offsite)
• Final Model (offsite)
• 3D Printing (offsite)
39
40. R. Pintus – CRS4/ViC, October 2012
3D Reconstruction Pipeline
• Real Model Inspection (onsite)
• Scans design (offsite/onsite)
• Acquisition (onsite)
• Alignment (offsite)
• Editing (offsite)
• Merge (offsite)
• Texture (offsite)
• Final Model (offsite)
• 3D Printing (offsite)
40
41. R. Pintus – CRS4/ViC, October 2012
Goal
• Fast and low-cost technique for creating
accurate colored models
• Acquisition
– 3D – laser scanners
– Color – digital cameras
• Mapping photo-to-geometry
– Fast and Robust Semi-Automatic Registration of Photographs
to 3D Geometry
• Photo blending
– A Streaming Framework for Seamless Detailed Photo
Blending on Massive Point Clouds
42. www.crs4.it/vic/
Photo Mapping
Ruggero Pintus, Enrico Gobbetti, and Roberto Combet. “Fast and Robust Semi-Automatic
Registration of Photographs to 3D Geometry”. In The 12th International Symposium on Virtual
Reality, Archaeology and Cultural Heritage, October 2011.
43. R. Pintus – CRS4/ViC, October 2012
Problem Statement
3D Geometry Unordered Set
Of n Uncalibrated
Photos
n Camera Poses
(2D/3D Registration)
44. R. Pintus – CRS4/ViC, October 2012
Related work
• Manual selection of 2D-3D matches
– Massive user intervention – Tiring and time-consuming
• Automatic feature matching
– Not robust enough for a generic dataset
• Semi-automatic statistical correlation
– Point cloud attributes not always provided
• Geometric multi-view reconstruction
– 2D-3D problem 3D-3D registration task
– dense and ordered frame sequence
• Our contribution
– Minimize user intervention / Large datasets / Semi-
automatic / Multi-view based approach / No Attributes
45. R. Pintus – CRS4/ViC, October 2012
Input Data
User Dense 3D n Photos • Dense Geometry
– Point cloud, triangle
mesh, etc.
SfM Reconstruction
– No attributes
– No particular features
Coarse Registration • n photos
– Naïve constraints:
Refinement • Blur, Noise, Under- or
over-exposured
– Sufficient overlap
Output Data
46. R. Pintus – CRS4/ViC, October 2012
Multi-
Multi-view
User Dense 3D n Photos • Bundler [Snavely et al.
2006]
– SfM system for unordered
SfM Reconstruction image collections
– http://phototour.cs.washingto
n.edu/bundler/
Coarse Registration • Output
– A sparse point cloud
– n camera poses
Refinement – SIFT keypoints (projections of
sparse 3D points)
Output Data
47. R. Pintus – CRS4/ViC, October 2012
Coarse registration
User Dense 3D n Photos • Register two point clouds
with different:
– scales
– reference frames
SfM Reconstruction – resolutions
• Automatic methods are not
robust and efficient enough
Coarse Registration • User aligns few images (one
or more) to the dense
geometry
Refinement • Affine transformation is
applied to all cameras and
sparse points
Output Data
48. R. Pintus – CRS4/ViC, October 2012
Refinement
User Dense 3D n Photos Pj
C1
Q (C2 , p j ) s1, j
pj
SfM Reconstruction
s2 , j NN F ( p j )
Coarse Registration
Q (C2 , NN F ( p j ))
Refinement C2
E (C , P ) = ∑∑ vij Q (Ci , NN F ( p j )) − si , j
N P NC
2
Output Data j =1 i =1
49. R. Pintus – CRS4/ViC, October 2012
Refinement
User Dense 3D n Photos • Sparse Bundle
Adjustment (SBA)
– Constants – SIFT keypoints,
SfM Reconstruction dense 3D points
– Variables – Camera poses,
sparse 3D points
Coarse Registration – SBA
• A Generic SBA C/C++ Package
Based on the Levenberg-
Marquardt Algorithm
Refinement • http://www.ics.forth.gr/~loura
kis/sba/
Output Data
50. R. Pintus – CRS4/ViC, October 2012
Output data
User Dense 3D n Photos • n camera poses
SfM Reconstruction • Input of photo
blending
– n photos
Coarse Registration
– n camera poses
– Dense 3D geometry
Refinement
Output Data
51. R. Pintus – CRS4/ViC, October 2012
Results – Photo mapping
52. www.crs4.it/vic/
Photo Blending
Ruggero Pintus, Enrico Gobbetti, and Marco Callieri. A Streaming Framework for Seamless Detailed
Photo Blending on Massive Point Clouds. In Proc. Eurographics Area Papers. Pages 25- 32, 2011.
53. R. Pintus – CRS4/ViC, October 2012
Problem Statement
Point Cloud Calibrated
Photos
54. R. Pintus – CRS4/ViC, October 2012
Problem Statement
Point Cloud Calibrated
Photos
P
55. R. Pintus – CRS4/ViC, October 2012
Problem Statement
Calibrated Colored
Point Cloud
Photos Point Cloud
P
56. R. Pintus – CRS4/ViC, October 2012
Problem Statement
Calibrated Colored
Point Cloud
Photos Point Cloud
P
• Problem Unlimited size of 3D model (Gpoints) and unlimited
number of images
57. R. Pintus – CRS4/ViC, October 2012
Related work
• State-of-the-art techniques
– Image quality estimation
– Stitching or blending
• Data representation
– Triangle meshes – exploit connectivity
– Meshless approaches
• Both triangle meshes and point clouds
• Memory settings
– All in-core – no massive geometry/images
– 3D in-core and images out-of-core – no massive geometry
– All out-of-core – Low performances
• Our contribution
– Blending function / Streaming framework / Massive point cloud /
Adaptive geometry refinement
58. R. Pintus – CRS4/ViC, October 2012
Pipeline
Masked
Per-pixel
Photo Stencil Per-pixel
Weight
Weight
59. R. Pintus – CRS4/ViC, October 2012
Simple blending
60. R. Pintus – CRS4/ViC, October 2012
Edge extraction and Distance
Transform
61. R. Pintus – CRS4/ViC, October 2012
Smooth weight
62. R. Pintus – CRS4/ViC, October 2012
Smooth weight
63. R. Pintus – CRS4/ViC, October 2012
Single band blending
64. R. Pintus – CRS4/ViC, October 2012
Multi band blending
65. R. Pintus – CRS4/ViC, October 2012
Adaptive point refinement
66. R. Pintus – CRS4/ViC, October 2012
Adaptive point refinement
67. R. Pintus – CRS4/ViC, October 2012
Adaptive point refinement
68. R. Pintus – CRS4/ViC, October 2012
Adaptive point refinement
69. R. Pintus – CRS4/ViC, October 2012
Results
David • Callieri et. al 2008 – David
28M
470Mpoints – Disk space occupancy –
6.2GB
– Computation time – 15.5
hours
70. R. Pintus – CRS4/ViC, October 2012
Results – Church’s Apse
14 Mpoint Geometry 40 photos
71. R. Pintus – CRS4/ViC, October 2012
Results – Church’s Apse
72. R. Pintus – CRS4/ViC, October 2012
Results – Grave
21 photos
8 Mpoint Geometry
73. R. Pintus – CRS4/ViC, October 2012
Results – Grave
78. R. Pintus – CRS4/ViC, October 2012
Conclusion
• Image-to-geometry registration approach
• Minimum user intervention
• No constraints on geometry, attributes and features
• Specific robust cost function and SBA
• Out-of-core photo blending approach (Point clouds of unlimited size)
• Incremental color accumulation (Unlimited number of images)
• Smooth weight function (Seamless color blending)
• Streaming framework (Performance improvement)
• Adaptive point refinement
• Future work
– Automatic sparse-to-dense geometry registration
– Interactive blending - adding and removing images in an interactive tool
– Fast visual check of previous alignment step
80. R. Pintus – CRS4/ViC, October 2012
3D Printing
80
81. R. Pintus – CRS4/ViC, October 2012
Printing Process
• Original model
• Slice
representation
• Layer by layer
deposition
• Cleaning
• Printed model
81
82. R. Pintus – CRS4/ViC, October 2012
Printing Process
• Original model
• Slice
representation
• Layer by layer
deposition
• Cleaning
• Printed model
82
83. R. Pintus – CRS4/ViC, October 2012
Printing Process
• Original model
• Slice
representation
• Layer by layer
deposition
• Cleaning
• Printed model
83
84. R. Pintus – CRS4/ViC, October 2012
Printing Process
• Original model
• Slice
representation
• Layer by layer
deposition
• Cleaning
• Printed model
84
85. R. Pintus – CRS4/ViC, October 2012
Printing Process
• Original model
• Slice
representation
• Layer by layer
deposition
• Cleaning
• Printed model
85
86. R. Pintus – CRS4/ViC, October 2012
Geometry processing
86
87. R. Pintus – CRS4/ViC, October 2012
Geometry processing
87
88. R. Pintus – CRS4/ViC, October 2012
Geometry processing
88
89. R. Pintus – CRS4/ViC, October 2012
Sub-
Sub-surface scattering
89
90. R. Pintus – CRS4/ViC, October 2012
Color enhancement
90
91. R. Pintus – CRS4/ViC, October 2012
Color enhancement
91
92. R. Pintus – CRS4/ViC, October 2012
Color enhancement
92
93. R. Pintus – CRS4/ViC, October 2012
Conclusioni
• Lavorare su dati misurati è un pre-
requisito di molti lavori (tutti?) nel
contesto dei beni culturali
– Applicazioni specialistiche o per grande pubblico
• Le moderne tecnologie di acquisizione
consentono di acquisire una grande
quantità di informazioni (forma e colore)
– Laser scanning, camere digitali, ecc.
• Uso potenziale vasto!
– Valorizzazione, restauro, studio, ecc.
94. R. Pintus – CRS4/ViC, October 2012
Conclusioni
• Queste quantità di dati sono però difficili
da trattare, archiviare, distribuire,
visualizzare
– Scalabilità!
• Tecniche attuali sub-ottimali
– Costi, tempi, qualità
95. R. Pintus – CRS4/ViC, October 2012
Conclusioni
• Il CRS4 è impegnato in attività di ricerca
per migliorare le tecnologie…
– Stato dell’arte internazionale
– Collaborazioni e ricadute locali
• PMI, Contro Restauro SS, Soprintendenze, CNR, UniCA
• … e per applicarle a casi concreti
– Collaborazioni multidisciplinari!
96. R. Pintus – CRS4/ViC, October 2012
Conclusioni
96
97. R. Pintus – CRS4/ViC, October 2012
Questions & Contacts
• CRS4 – VIC
www.crs4.it/vic/
• Ruggero Pintus
ruggero@crs4.it