SlideShare une entreprise Scribd logo
1  sur  22
Télécharger pour lire hors ligne
Sensor fusion for autonomous
driving perception
Bagni Fabio
228594@studenti.unimore .it
San Francisco - 15 May 2019
1
Fabio Bagni
San Francisco - 15 May 2019
Autonomous driving perception
Precision and reactivity
Risks of accidents, traffic laws respect.
2
Fabio Bagni
San Francisco - 15 May 2019
Embedded platforms
3
Fabio Bagni
San Francisco - 15 May 2019
Goals
Real time 3D detection
● High performances
○ required 10 Hz frequency
● High precision
● Real time on embedded platforms
4
Fabio Bagni
San Francisco - 15 May 2019
UNIMORE HiPeRT prototype
5
Fabio Bagni
San Francisco - 15 May 2019
Sensors
● LiDAR
○ 3D point cloud
● Cameras
○ images
Heterogeneous outputs
6
Fabio Bagni
San Francisco - 15 May 2019
Sensor fusion
What is sensor fusion:
● Matching of different sensors
views
Why sensor fusion:
● Heterogeneous information of
environment
7
Fabio Bagni
San Francisco - 15 May 2019
Fields of view
LiDAR: 360 degrees around the vehicle
Camera: wide angle, disposed do cover 360
degrees.
8
Cameras
LiDAR
Fabio Bagni
San Francisco - 15 May 2019
Deep learning approach
Camera:
● Bird’s Eye View estimation
● Object detection
LiDAR
● Bird’s Eye View calculation
● Object detection
9
Image
Point
cloud
3D
detection
3D
detection
Fusion
Fabio Bagni
San Francisco - 15 May 2019
Deep learning approach time
Image stitching for multiple camera
fusion
10
➔ Under10 Hz
Notenoughresponsive
1. Ming Liang, Bin Yang, Shenlong Wang and Raquel Urtasun: Deep Continuous
Fusion for Multi-Sensor3D ObjectDetection.In: ECCV (2018).
Fabio Bagni
San Francisco - 15 May 2019
Static alignment sensor fusion
Alignment of LiDAR with each camera
individually.
11
Calibration
Preprocessing
Fabio Bagni
San Francisco - 15 May 2019
Features extraction
Matching is made by a calibration with a
perforated panel.
Holes are detected by all sensors.
Holes’ centers are used as features to be
matched.
12
Fabio Bagni
San Francisco - 15 May 2019
Cylindrical projection
Fields of view matching is made projecting sensors’ outputs on the surface of a
cylinder.
13
Fabio Bagni
San Francisco - 15 May 2019
Projections complexity
LiDAR cylindrical projection: Depthmap
O(n) : one operation per point
14
Camera cylindrical projection
O(n) : one operation per pixel
PARALLELIZABLE
Fabio Bagni
San Francisco - 15 May 2019
Calibration results
15
Fabio Bagni
San Francisco - 15 May 2019
Colored points
Alignment of each point with the
corresponding pixel on the image.
16
Fabio Bagni
San Francisco - 15 May 2019
Object detection
Neural network for object detection on
camera frame
Clustering of LiDAR points
Bounding boxes matching
17
Fabio Bagni
San Francisco - 15 May 2019
Object detection results
18
Fabio Bagni
San Francisco - 15 May 2019
Object detection
average time
19
LiDAR output : 10 Hz
Time limit : 100 ms
Thanks for your attention
Fabio Bagni - 228594@studenti.unimore.it
20
Fabio Bagni
San Francisco - 15 May 2019
21
Fabio Bagni
San Francisco - 15 May 2019
Homography
Homography allow to translate camera projection plane on the LiDAR projection plane.
This method make cylindrical surfaces match.
22

Contenu connexe

Tendances

V2X Communications: Getting our Cars Talking
V2X Communications: Getting our Cars TalkingV2X Communications: Getting our Cars Talking
V2X Communications: Getting our Cars TalkingAlison Chaiken
 
Introductory Level of SLAM Seminar
Introductory Level of SLAM SeminarIntroductory Level of SLAM Seminar
Introductory Level of SLAM SeminarDong-Won Shin
 
Object Detection using Deep Neural Networks
Object Detection using Deep Neural NetworksObject Detection using Deep Neural Networks
Object Detection using Deep Neural NetworksUsman Qayyum
 
Principle of FMCW radar
Principle of FMCW radarPrinciple of FMCW radar
Principle of FMCW radartobiasotto
 
"Fundamentals of Monocular SLAM," a Presentation from Cadence
"Fundamentals of Monocular SLAM," a Presentation from Cadence"Fundamentals of Monocular SLAM," a Presentation from Cadence
"Fundamentals of Monocular SLAM," a Presentation from CadenceEdge AI and Vision Alliance
 
Faster R-CNN: Towards real-time object detection with region proposal network...
Faster R-CNN: Towards real-time object detection with region proposal network...Faster R-CNN: Towards real-time object detection with region proposal network...
Faster R-CNN: Towards real-time object detection with region proposal network...Universitat Politècnica de Catalunya
 
Lecture 15 DCT, Walsh and Hadamard Transform
Lecture 15 DCT, Walsh and Hadamard TransformLecture 15 DCT, Walsh and Hadamard Transform
Lecture 15 DCT, Walsh and Hadamard TransformVARUN KUMAR
 
Deep learning for object detection
Deep learning for object detectionDeep learning for object detection
Deep learning for object detectionWenjing Chen
 
HIGH PASS FILTER IN DIGITAL IMAGE PROCESSING
HIGH PASS FILTER IN DIGITAL IMAGE PROCESSINGHIGH PASS FILTER IN DIGITAL IMAGE PROCESSING
HIGH PASS FILTER IN DIGITAL IMAGE PROCESSINGBimal2354
 
Object tracking survey
Object tracking surveyObject tracking survey
Object tracking surveyRich Nguyen
 
Action Recognition (Thesis presentation)
Action Recognition (Thesis presentation)Action Recognition (Thesis presentation)
Action Recognition (Thesis presentation)nikhilus85
 
Computer Vision: Feature matching with RANSAC Algorithm
Computer Vision: Feature matching with RANSAC AlgorithmComputer Vision: Feature matching with RANSAC Algorithm
Computer Vision: Feature matching with RANSAC Algorithmallyn joy calcaben
 
Lecture 13 (Usage of Fourier transform in image processing)
Lecture 13 (Usage of Fourier transform in image processing)Lecture 13 (Usage of Fourier transform in image processing)
Lecture 13 (Usage of Fourier transform in image processing)VARUN KUMAR
 
Object Detection & Tracking
Object Detection & TrackingObject Detection & Tracking
Object Detection & TrackingAkshay Gujarathi
 
Blur and un blur an image using matlab
Blur and un blur an image using matlabBlur and un blur an image using matlab
Blur and un blur an image using matlabWaqarAhmad444
 
Object tracking presentation
Object tracking  presentationObject tracking  presentation
Object tracking presentationMrsShwetaBanait1
 
2D mapping using omni-directional mobile robot equipped with LiDAR
2D mapping using omni-directional mobile robot equipped with LiDAR2D mapping using omni-directional mobile robot equipped with LiDAR
2D mapping using omni-directional mobile robot equipped with LiDARTELKOMNIKA JOURNAL
 
Lidar for Autonomous Driving II (via Deep Learning)
Lidar for Autonomous Driving II (via Deep Learning)Lidar for Autonomous Driving II (via Deep Learning)
Lidar for Autonomous Driving II (via Deep Learning)Yu Huang
 

Tendances (20)

Cw and fm cw radar
Cw and fm cw radarCw and fm cw radar
Cw and fm cw radar
 
V2X Communications: Getting our Cars Talking
V2X Communications: Getting our Cars TalkingV2X Communications: Getting our Cars Talking
V2X Communications: Getting our Cars Talking
 
Introductory Level of SLAM Seminar
Introductory Level of SLAM SeminarIntroductory Level of SLAM Seminar
Introductory Level of SLAM Seminar
 
Object Detection using Deep Neural Networks
Object Detection using Deep Neural NetworksObject Detection using Deep Neural Networks
Object Detection using Deep Neural Networks
 
Principle of FMCW radar
Principle of FMCW radarPrinciple of FMCW radar
Principle of FMCW radar
 
YOLO
YOLOYOLO
YOLO
 
"Fundamentals of Monocular SLAM," a Presentation from Cadence
"Fundamentals of Monocular SLAM," a Presentation from Cadence"Fundamentals of Monocular SLAM," a Presentation from Cadence
"Fundamentals of Monocular SLAM," a Presentation from Cadence
 
Faster R-CNN: Towards real-time object detection with region proposal network...
Faster R-CNN: Towards real-time object detection with region proposal network...Faster R-CNN: Towards real-time object detection with region proposal network...
Faster R-CNN: Towards real-time object detection with region proposal network...
 
Lecture 15 DCT, Walsh and Hadamard Transform
Lecture 15 DCT, Walsh and Hadamard TransformLecture 15 DCT, Walsh and Hadamard Transform
Lecture 15 DCT, Walsh and Hadamard Transform
 
Deep learning for object detection
Deep learning for object detectionDeep learning for object detection
Deep learning for object detection
 
HIGH PASS FILTER IN DIGITAL IMAGE PROCESSING
HIGH PASS FILTER IN DIGITAL IMAGE PROCESSINGHIGH PASS FILTER IN DIGITAL IMAGE PROCESSING
HIGH PASS FILTER IN DIGITAL IMAGE PROCESSING
 
Object tracking survey
Object tracking surveyObject tracking survey
Object tracking survey
 
Action Recognition (Thesis presentation)
Action Recognition (Thesis presentation)Action Recognition (Thesis presentation)
Action Recognition (Thesis presentation)
 
Computer Vision: Feature matching with RANSAC Algorithm
Computer Vision: Feature matching with RANSAC AlgorithmComputer Vision: Feature matching with RANSAC Algorithm
Computer Vision: Feature matching with RANSAC Algorithm
 
Lecture 13 (Usage of Fourier transform in image processing)
Lecture 13 (Usage of Fourier transform in image processing)Lecture 13 (Usage of Fourier transform in image processing)
Lecture 13 (Usage of Fourier transform in image processing)
 
Object Detection & Tracking
Object Detection & TrackingObject Detection & Tracking
Object Detection & Tracking
 
Blur and un blur an image using matlab
Blur and un blur an image using matlabBlur and un blur an image using matlab
Blur and un blur an image using matlab
 
Object tracking presentation
Object tracking  presentationObject tracking  presentation
Object tracking presentation
 
2D mapping using omni-directional mobile robot equipped with LiDAR
2D mapping using omni-directional mobile robot equipped with LiDAR2D mapping using omni-directional mobile robot equipped with LiDAR
2D mapping using omni-directional mobile robot equipped with LiDAR
 
Lidar for Autonomous Driving II (via Deep Learning)
Lidar for Autonomous Driving II (via Deep Learning)Lidar for Autonomous Driving II (via Deep Learning)
Lidar for Autonomous Driving II (via Deep Learning)
 

Plus de NECST Lab @ Politecnico di Milano

Embedding based knowledge graph link prediction for drug repurposing
Embedding based knowledge graph link prediction for drug repurposingEmbedding based knowledge graph link prediction for drug repurposing
Embedding based knowledge graph link prediction for drug repurposingNECST Lab @ Politecnico di Milano
 
PLASTER - PYNQ-based abandoned object detection using a map-reduce approach o...
PLASTER - PYNQ-based abandoned object detection using a map-reduce approach o...PLASTER - PYNQ-based abandoned object detection using a map-reduce approach o...
PLASTER - PYNQ-based abandoned object detection using a map-reduce approach o...NECST Lab @ Politecnico di Milano
 
EMPhASIS - An EMbedded Public Attention Stress Identification System
 EMPhASIS - An EMbedded Public Attention Stress Identification System EMPhASIS - An EMbedded Public Attention Stress Identification System
EMPhASIS - An EMbedded Public Attention Stress Identification SystemNECST Lab @ Politecnico di Milano
 
Maeve - Fast genome analysis leveraging exact string matching
Maeve - Fast genome analysis leveraging exact string matchingMaeve - Fast genome analysis leveraging exact string matching
Maeve - Fast genome analysis leveraging exact string matchingNECST Lab @ Politecnico di Milano
 

Plus de NECST Lab @ Politecnico di Milano (20)

Mesticheria Team - WiiReflex
Mesticheria Team - WiiReflexMesticheria Team - WiiReflex
Mesticheria Team - WiiReflex
 
Punto e virgola Team - Stressometro
Punto e virgola Team - StressometroPunto e virgola Team - Stressometro
Punto e virgola Team - Stressometro
 
BitIt Team - Stay.straight
BitIt Team - Stay.straight BitIt Team - Stay.straight
BitIt Team - Stay.straight
 
BabYodini Team - Talking Gloves
BabYodini Team - Talking GlovesBabYodini Team - Talking Gloves
BabYodini Team - Talking Gloves
 
printf("Nome Squadra"); Team - NeoTon
printf("Nome Squadra"); Team - NeoTonprintf("Nome Squadra"); Team - NeoTon
printf("Nome Squadra"); Team - NeoTon
 
BlackBoard Team - Motion Tracking Platform
BlackBoard Team - Motion Tracking PlatformBlackBoard Team - Motion Tracking Platform
BlackBoard Team - Motion Tracking Platform
 
#include<brain.h> Team - HomeBeatHome
#include<brain.h> Team - HomeBeatHome#include<brain.h> Team - HomeBeatHome
#include<brain.h> Team - HomeBeatHome
 
Flipflops Team - Wave U
Flipflops Team - Wave UFlipflops Team - Wave U
Flipflops Team - Wave U
 
Bug(atta) Team - Little Brother
Bug(atta) Team - Little BrotherBug(atta) Team - Little Brother
Bug(atta) Team - Little Brother
 
#NECSTCamp: come partecipare
#NECSTCamp: come partecipare#NECSTCamp: come partecipare
#NECSTCamp: come partecipare
 
NECSTCamp101@2020.10.1
NECSTCamp101@2020.10.1NECSTCamp101@2020.10.1
NECSTCamp101@2020.10.1
 
NECSTLab101 2020.2021
NECSTLab101 2020.2021NECSTLab101 2020.2021
NECSTLab101 2020.2021
 
TreeHouse, nourish your community
TreeHouse, nourish your communityTreeHouse, nourish your community
TreeHouse, nourish your community
 
TiReX: Tiled Regular eXpressionsmatching architecture
TiReX: Tiled Regular eXpressionsmatching architectureTiReX: Tiled Regular eXpressionsmatching architecture
TiReX: Tiled Regular eXpressionsmatching architecture
 
Embedding based knowledge graph link prediction for drug repurposing
Embedding based knowledge graph link prediction for drug repurposingEmbedding based knowledge graph link prediction for drug repurposing
Embedding based knowledge graph link prediction for drug repurposing
 
PLASTER - PYNQ-based abandoned object detection using a map-reduce approach o...
PLASTER - PYNQ-based abandoned object detection using a map-reduce approach o...PLASTER - PYNQ-based abandoned object detection using a map-reduce approach o...
PLASTER - PYNQ-based abandoned object detection using a map-reduce approach o...
 
EMPhASIS - An EMbedded Public Attention Stress Identification System
 EMPhASIS - An EMbedded Public Attention Stress Identification System EMPhASIS - An EMbedded Public Attention Stress Identification System
EMPhASIS - An EMbedded Public Attention Stress Identification System
 
Luns - Automatic lungs segmentation through neural network
Luns - Automatic lungs segmentation through neural networkLuns - Automatic lungs segmentation through neural network
Luns - Automatic lungs segmentation through neural network
 
BlastFunction: How to combine Serverless and FPGAs
BlastFunction: How to combine Serverless and FPGAsBlastFunction: How to combine Serverless and FPGAs
BlastFunction: How to combine Serverless and FPGAs
 
Maeve - Fast genome analysis leveraging exact string matching
Maeve - Fast genome analysis leveraging exact string matchingMaeve - Fast genome analysis leveraging exact string matching
Maeve - Fast genome analysis leveraging exact string matching
 

Dernier

Environmental and Climate Justice Programby Karen Campblin
Environmental and Climate Justice Programby Karen CampblinEnvironmental and Climate Justice Programby Karen Campblin
Environmental and Climate Justice Programby Karen CampblinForth
 
Equity Lab: Inked with Intent by The Greenlining Institute
Equity Lab: Inked with Intent by The Greenlining InstituteEquity Lab: Inked with Intent by The Greenlining Institute
Equity Lab: Inked with Intent by The Greenlining InstituteForth
 
Nosfdsfsdfasdfasdfasdfsadf asdfasdfasdfasdf
Nosfdsfsdfasdfasdfasdfsadf asdfasdfasdfasdfNosfdsfsdfasdfasdfasdfsadf asdfasdfasdfasdf
Nosfdsfsdfasdfasdfasdfsadf asdfasdfasdfasdfJulia Kaye
 
Transportation Electrification Funding Strategy.pptx
Transportation Electrification Funding Strategy.pptxTransportation Electrification Funding Strategy.pptx
Transportation Electrification Funding Strategy.pptxForth
 
USDA’s EV Charging Infrastructure Solutions by Chris McLean
USDA’s EV Charging Infrastructure Solutionsby Chris McLeanUSDA’s EV Charging Infrastructure Solutionsby Chris McLean
USDA’s EV Charging Infrastructure Solutions by Chris McLeanForth
 
Study on Financing of zero-emission trucks and their infrastructure
Study on Financing of zero-emission trucks and their infrastructureStudy on Financing of zero-emission trucks and their infrastructure
Study on Financing of zero-emission trucks and their infrastructureEuropeanCleanTruckin
 
Lakshitha maduranga CV - for data entry clerck
Lakshitha maduranga CV - for data entry clerckLakshitha maduranga CV - for data entry clerck
Lakshitha maduranga CV - for data entry clerckLakshanMadhushanka3
 
Design and Fund Equitable Electric Transportation For Communities by Jasmine ...
Design and Fund Equitable Electric Transportation For Communities by Jasmine ...Design and Fund Equitable Electric Transportation For Communities by Jasmine ...
Design and Fund Equitable Electric Transportation For Communities by Jasmine ...Forth
 
Commercial Extractor fan repair services
Commercial Extractor fan repair servicesCommercial Extractor fan repair services
Commercial Extractor fan repair servicesmb1294198
 
Centering Equity Presentation by Brenna Rivett
Centering Equity Presentation by Brenna RivettCentering Equity Presentation by Brenna Rivett
Centering Equity Presentation by Brenna RivettForth
 
Building a Budget by Jeff Allen and Josh Rodriguez
Building a Budget by Jeff Allen and Josh RodriguezBuilding a Budget by Jeff Allen and Josh Rodriguez
Building a Budget by Jeff Allen and Josh RodriguezForth
 
Building a Future Where Everyone Can Ride and Drive Electric by Linda Bailey
Building a Future Where Everyone Can Ride and Drive Electric by Linda BaileyBuilding a Future Where Everyone Can Ride and Drive Electric by Linda Bailey
Building a Future Where Everyone Can Ride and Drive Electric by Linda BaileyForth
 
Centering Equity and Community in Transportation by Benito Perez
Centering Equity and Community in Transportation by Benito PerezCentering Equity and Community in Transportation by Benito Perez
Centering Equity and Community in Transportation by Benito PerezForth
 
Program Design by Prateek Suri and Shakaya Cooper
Program Design by Prateek Suri and Shakaya CooperProgram Design by Prateek Suri and Shakaya Cooper
Program Design by Prateek Suri and Shakaya CooperForth
 
Centering equity and the community in Transportation by Richard Ezike
Centering equity and the community in Transportation by Richard EzikeCentering equity and the community in Transportation by Richard Ezike
Centering equity and the community in Transportation by Richard EzikeForth
 

Dernier (17)

Environmental and Climate Justice Programby Karen Campblin
Environmental and Climate Justice Programby Karen CampblinEnvironmental and Climate Justice Programby Karen Campblin
Environmental and Climate Justice Programby Karen Campblin
 
Equity Lab: Inked with Intent by The Greenlining Institute
Equity Lab: Inked with Intent by The Greenlining InstituteEquity Lab: Inked with Intent by The Greenlining Institute
Equity Lab: Inked with Intent by The Greenlining Institute
 
Nosfdsfsdfasdfasdfasdfsadf asdfasdfasdfasdf
Nosfdsfsdfasdfasdfasdfsadf asdfasdfasdfasdfNosfdsfsdfasdfasdfasdfsadf asdfasdfasdfasdf
Nosfdsfsdfasdfasdfasdfsadf asdfasdfasdfasdf
 
Transportation Electrification Funding Strategy.pptx
Transportation Electrification Funding Strategy.pptxTransportation Electrification Funding Strategy.pptx
Transportation Electrification Funding Strategy.pptx
 
USDA’s EV Charging Infrastructure Solutions by Chris McLean
USDA’s EV Charging Infrastructure Solutionsby Chris McLeanUSDA’s EV Charging Infrastructure Solutionsby Chris McLean
USDA’s EV Charging Infrastructure Solutions by Chris McLean
 
Study on Financing of zero-emission trucks and their infrastructure
Study on Financing of zero-emission trucks and their infrastructureStudy on Financing of zero-emission trucks and their infrastructure
Study on Financing of zero-emission trucks and their infrastructure
 
Lakshitha maduranga CV - for data entry clerck
Lakshitha maduranga CV - for data entry clerckLakshitha maduranga CV - for data entry clerck
Lakshitha maduranga CV - for data entry clerck
 
Design and Fund Equitable Electric Transportation For Communities by Jasmine ...
Design and Fund Equitable Electric Transportation For Communities by Jasmine ...Design and Fund Equitable Electric Transportation For Communities by Jasmine ...
Design and Fund Equitable Electric Transportation For Communities by Jasmine ...
 
Reinventing the Car - as I reported it in 1985!
Reinventing the Car - as I reported it in 1985!Reinventing the Car - as I reported it in 1985!
Reinventing the Car - as I reported it in 1985!
 
Commercial Extractor fan repair services
Commercial Extractor fan repair servicesCommercial Extractor fan repair services
Commercial Extractor fan repair services
 
Centering Equity Presentation by Brenna Rivett
Centering Equity Presentation by Brenna RivettCentering Equity Presentation by Brenna Rivett
Centering Equity Presentation by Brenna Rivett
 
Building a Budget by Jeff Allen and Josh Rodriguez
Building a Budget by Jeff Allen and Josh RodriguezBuilding a Budget by Jeff Allen and Josh Rodriguez
Building a Budget by Jeff Allen and Josh Rodriguez
 
Building a Future Where Everyone Can Ride and Drive Electric by Linda Bailey
Building a Future Where Everyone Can Ride and Drive Electric by Linda BaileyBuilding a Future Where Everyone Can Ride and Drive Electric by Linda Bailey
Building a Future Where Everyone Can Ride and Drive Electric by Linda Bailey
 
Centering Equity and Community in Transportation by Benito Perez
Centering Equity and Community in Transportation by Benito PerezCentering Equity and Community in Transportation by Benito Perez
Centering Equity and Community in Transportation by Benito Perez
 
Program Design by Prateek Suri and Shakaya Cooper
Program Design by Prateek Suri and Shakaya CooperProgram Design by Prateek Suri and Shakaya Cooper
Program Design by Prateek Suri and Shakaya Cooper
 
EVAT - Future Mobility Transformation in Thailand
EVAT - Future Mobility Transformation in ThailandEVAT - Future Mobility Transformation in Thailand
EVAT - Future Mobility Transformation in Thailand
 
Centering equity and the community in Transportation by Richard Ezike
Centering equity and the community in Transportation by Richard EzikeCentering equity and the community in Transportation by Richard Ezike
Centering equity and the community in Transportation by Richard Ezike
 

Sensor fusion of LiDAR and Camera for real time object detection - talk version

  • 1. Sensor fusion for autonomous driving perception Bagni Fabio 228594@studenti.unimore .it San Francisco - 15 May 2019 1
  • 2. Fabio Bagni San Francisco - 15 May 2019 Autonomous driving perception Precision and reactivity Risks of accidents, traffic laws respect. 2
  • 3. Fabio Bagni San Francisco - 15 May 2019 Embedded platforms 3
  • 4. Fabio Bagni San Francisco - 15 May 2019 Goals Real time 3D detection ● High performances ○ required 10 Hz frequency ● High precision ● Real time on embedded platforms 4
  • 5. Fabio Bagni San Francisco - 15 May 2019 UNIMORE HiPeRT prototype 5
  • 6. Fabio Bagni San Francisco - 15 May 2019 Sensors ● LiDAR ○ 3D point cloud ● Cameras ○ images Heterogeneous outputs 6
  • 7. Fabio Bagni San Francisco - 15 May 2019 Sensor fusion What is sensor fusion: ● Matching of different sensors views Why sensor fusion: ● Heterogeneous information of environment 7
  • 8. Fabio Bagni San Francisco - 15 May 2019 Fields of view LiDAR: 360 degrees around the vehicle Camera: wide angle, disposed do cover 360 degrees. 8 Cameras LiDAR
  • 9. Fabio Bagni San Francisco - 15 May 2019 Deep learning approach Camera: ● Bird’s Eye View estimation ● Object detection LiDAR ● Bird’s Eye View calculation ● Object detection 9 Image Point cloud 3D detection 3D detection Fusion
  • 10. Fabio Bagni San Francisco - 15 May 2019 Deep learning approach time Image stitching for multiple camera fusion 10 ➔ Under10 Hz Notenoughresponsive 1. Ming Liang, Bin Yang, Shenlong Wang and Raquel Urtasun: Deep Continuous Fusion for Multi-Sensor3D ObjectDetection.In: ECCV (2018).
  • 11. Fabio Bagni San Francisco - 15 May 2019 Static alignment sensor fusion Alignment of LiDAR with each camera individually. 11 Calibration Preprocessing
  • 12. Fabio Bagni San Francisco - 15 May 2019 Features extraction Matching is made by a calibration with a perforated panel. Holes are detected by all sensors. Holes’ centers are used as features to be matched. 12
  • 13. Fabio Bagni San Francisco - 15 May 2019 Cylindrical projection Fields of view matching is made projecting sensors’ outputs on the surface of a cylinder. 13
  • 14. Fabio Bagni San Francisco - 15 May 2019 Projections complexity LiDAR cylindrical projection: Depthmap O(n) : one operation per point 14 Camera cylindrical projection O(n) : one operation per pixel PARALLELIZABLE
  • 15. Fabio Bagni San Francisco - 15 May 2019 Calibration results 15
  • 16. Fabio Bagni San Francisco - 15 May 2019 Colored points Alignment of each point with the corresponding pixel on the image. 16
  • 17. Fabio Bagni San Francisco - 15 May 2019 Object detection Neural network for object detection on camera frame Clustering of LiDAR points Bounding boxes matching 17
  • 18. Fabio Bagni San Francisco - 15 May 2019 Object detection results 18
  • 19. Fabio Bagni San Francisco - 15 May 2019 Object detection average time 19 LiDAR output : 10 Hz Time limit : 100 ms
  • 20. Thanks for your attention Fabio Bagni - 228594@studenti.unimore.it 20 Fabio Bagni San Francisco - 15 May 2019
  • 21. 21
  • 22. Fabio Bagni San Francisco - 15 May 2019 Homography Homography allow to translate camera projection plane on the LiDAR projection plane. This method make cylindrical surfaces match. 22