Ivan Sahumbaiev "Deep Learning approaches meet 3D data"
•Télécharger en tant que PPTX, PDF•
0 j'aime•184 vues
1. The document discusses deep learning applications for 3D data including classification, segmentation, shape deformation, and remeshing. 2. PointNet and PointNet++ are introduced for 3D point cloud classification and segmentation using deep neural networks. 3. Non-Euclidean methods for processing manifolds include spectral and spatial convolutional neural networks on graphs and manifolds. 4. The RoButcher project aims to use 3D data and deep learning for autonomous robotic cutting of meat in abattoirs.
Recommandé
Recommandé
Contenu connexe
Tendances
Tendances (20)
Similaire à Ivan Sahumbaiev "Deep Learning approaches meet 3D data"
Similaire à Ivan Sahumbaiev "Deep Learning approaches meet 3D data" (20)
Plus de Fwdays
Plus de Fwdays (20)
Dernier
Dernier (20)
Ivan Sahumbaiev "Deep Learning approaches meet 3D data"
- 2. Agenda 1. Introduction to 3D data a) Data collection b) Data representation 2. Deep Learning applications based on 3D data a) Classification problem b) Segmentation problem 3. Remeshing 4. Non-Euclidean methods for manifold processing a) Spectral b) Spatial 5. Libraries and datasets 6. RoButcher project 7. Q&A 2
- 4. Data representation – 1 R G B
- 5. v1 v2 v3 f1 f2 Base components: 1. Vertices – v1 = [x1 y1 z1] 2. Edges 3. Faces – f1 = [v1 v2 v3 ] 5 Data representation – 2 class TriangularMesh: Vertices -> float array Triangles -> int array class PointCloud: Vertices -> float array
- 7. 7 PointNet [Charles R. Qi ] PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation Properties 1. Permutation invariance 2. Rigid transformation invariance 3. No local context
- 8. 8 PointNet++ PointNet++, Qi et al., NeurIPS, 2018 Properties 1. Permutation invariance 2. Rigid transformation invariance 3. Hierarchical features learning 3. Acces to the local context
- 9. 9 PointNet & PointNet++ Results PointNet++, Qi et al., NeurIPS, 2018
- 10. 10 PointNet & PointNet++ Results
- 11. 11 Shape deformation Rigid transformations: • Translation • Rotation • Scaling Not Rigid transformations: • Internal structure has changed • Rotation of some parts
- 12. 12 Shape deformation [Boscani] Learning shape correspondence with anisotropic convolutional neural networks Extrinsic Intrinsic
- 13. Point cloud 13 Remeshing [Edelsbrunner1983]; [Bernardini1999] https://pytorch3d.readthedocs.io/en/latest/ Alpha shapes Ball Pivoting; Poisson Triangulation
- 14. 14 Non-Euclidean CNN Spectral domain Spacial domain Embedding domain
- 15. 15 Laplac-Bertrami operator MacNeal 1949; Duffin 1959; Pinkall, Polthier 1993
- 16. 16 Laplac-Bertrami operator ∆𝚽 = 𝚽𝚲 • Λ – diag(𝜆1, … , 𝜆 𝑛) diagonal matrix of non-negative eigenvalues • Φ – (𝜙1, … , 𝜙 𝑛) orthonormal matrix of eigenvectors a3 http://geometry.cs.ucl.ac.uk/SGP2017/?p=gradschool
- 17. 17 Spectral convolution d’Alembert 1754; Borel 1899
- 21. 21 Correspondences result Functionalmaps:a flexiblerepresentationofmapsbetweenshapes, O.,Ben-Chen,Solomon,Butscher,Guibas,SIGGRAPH 2012
- 22. 22 Non-Euclidean convolutions Embedding domain Spacial domainSpectral domain Geodesic convolutional neural networks on Riemannian manifolds
- 24. 24 Spacial patches j • Local system of coordinates Uij around vertex i (geodesic polar) • Spatial convolution with filter g 𝑥𝑖 ′ ∝ 𝜃=1 Θ 𝑔 𝜃 𝑗∈𝑁(𝑥) 𝑤 𝜃 𝒖𝑖,𝑗 𝑥𝑗 Geometric deep learning on graphs and manifolds using mixture model CNNs
- 25. 25 Spatial convolutions 26 Spatial convolutions Geodesic convolutional neural networks on Riemannian manifolds
- 27. 27 Spatial convolutions Geometric deep learning on graphs and manifolds using mixture model CNNs
- 28. 28 Spatial convolutions Shape correspondence quality obtained by different methods on the FAUST humans dataset Geometric deep learning on graphs and manifolds using mixture model CNNs
- 29. ModelNet (2015) (ModelNet10: 4899 models, 10 categories) ModelNet40: 12311 models, 40 categories PartNet (2019) A Large-scale Benchmark for Fine- grained and Hierrarchical Part-level 3D Object Understanding ABC (2018) A Big CAD Model Dataset For Geometric Deep Learning MPI FAUST 300 human meshes 10 objects un 30 poses ~7k vertices 29 Datasets
- 30. 1. Open3d 2. Pytorch3d 3. Nvidia-Kaolin 4. Pytorch-Points 5. Pytorch-Geometric Python C++ 1. LibIgl (has python bindings) 2. CGAL 3. Point cloud library (PCL) 4. Geometry Central 5. Open3d 30 Libraries Easy to install Contains visualizer tools Compute some descriptors, normals, reconstructions Algorithms o Pytorch integration easier o Need to compile Contains visualizer tools Compute descriptors, normals, reconstructions Algorithms Speed
- 32. 32 A Robust, Flexible and Scalable Cognitive Robotics Platform RoBUTCHER – a European funded Horizon 2020 innovation project that aims to replace the conventional line production in abattoirs, with parallel production in autonomous «meat factory cells». In the RoBUTCHER concept the robotic system will be able to understand and plan cutting trajectories based on the carcass that is presented. To achieve this it will use a combination of detailed computed tomography (CT) data, real-time 3D imagery and human-expert cutting data for neural network training toward cutting trajectory planning. https://robutcher.eu/
- 33. 33 Pipeline Data collection Point cloud segmentaiton Point cloud to CAD morphing Extraction correspondences RealSense camera
- 34. 34 PointNet++ Data collection and Segmentation Data collection Point cloud segmentaiton Point cloud to CAD morphing Extraction correspondences
- 35. 35 • Point clouds from cameras is not manifold. • Reconstructing the mesh is hard. Point cloud morphing and Correspondence Data collection Point cloud segmentaiton Point cloud to CAD morphing Extraction correspondences
- 36. 36 Point cloud morphing and Correspondence Data collection Point cloud segmentaiton Point cloud to CAD morphing Extraction correspondences
- 37. Partners 37
- 38. Thank you for your attention! Дякую за увагу! Спасибо за внимание! Дзякуй за ўвагу! 38