1.
Sensor fusion for autonomous
driving perception
Bagni Fabio
228594@studenti.unimore .it
San Francisco - 15 May 2019
1

2.
Fabio Bagni
San Francisco - 15 May 2019
Autonomous driving perception
Precision and reactivity
Risks of accidents, traffic laws respect.
2

3.
Fabio Bagni
San Francisco - 15 May 2019
Embedded platforms
3

4.
Fabio Bagni
San Francisco - 15 May 2019
Goals
Real time 3D detection
● High performances
○ required 10 Hz frequency
● High precision
● Real time on embedded platforms
4

5.
Fabio Bagni
San Francisco - 15 May 2019
UNIMORE HiPeRT prototype
5

6.
Fabio Bagni
San Francisco - 15 May 2019
Sensors
● LiDAR
○ 3D point cloud
● Cameras
○ images
Heterogeneous outputs
6

7.
Fabio Bagni
San Francisco - 15 May 2019
Sensor fusion
What is sensor fusion:
● Matching of different sensors
views
Why sensor fusion:
● Heterogeneous information of
environment
7

8.
Fabio Bagni
San Francisco - 15 May 2019
Fields of view
LiDAR: 360 degrees around the vehicle
Camera: wide angle, disposed do cover 360
degrees.
8
Cameras
LiDAR

9.
Fabio Bagni
San Francisco - 15 May 2019
Deep learning approach
Camera:
● Bird’s Eye View estimation
● Object detection
LiDAR
● Bird’s Eye View calculation
● Object detection
9
Image
Point
cloud
3D
detection
3D
detection
Fusion

10.
Fabio Bagni
San Francisco - 15 May 2019
Deep learning approach time
Image stitching for multiple camera
fusion
10
➔ Under10 Hz
Notenoughresponsive
1. Ming Liang, Bin Yang, Shenlong Wang and Raquel Urtasun: Deep Continuous
Fusion for Multi-Sensor3D ObjectDetection.In: ECCV (2018).

11.
Fabio Bagni
San Francisco - 15 May 2019
Static alignment sensor fusion
Alignment of LiDAR with each camera
individually.
11
Calibration
Preprocessing

12.
Fabio Bagni
San Francisco - 15 May 2019
Features extraction
Matching is made by a calibration with a
perforated panel.
Holes are detected by all sensors.
Holes’ centers are used as features to be
matched.
12

13.
Fabio Bagni
San Francisco - 15 May 2019
Cylindrical projection
Fields of view matching is made projecting sensors’ outputs on the surface of a
cylinder.
13

14.
Fabio Bagni
San Francisco - 15 May 2019
Projections complexity
LiDAR cylindrical projection: Depthmap
O(n) : one operation per point
14
Camera cylindrical projection
O(n) : one operation per pixel
PARALLELIZABLE

15.
Fabio Bagni
San Francisco - 15 May 2019
Calibration results
15

16.
Fabio Bagni
San Francisco - 15 May 2019
Colored points
Alignment of each point with the
corresponding pixel on the image.
16

17.
Fabio Bagni
San Francisco - 15 May 2019
Object detection
Neural network for object detection on
camera frame
Clustering of LiDAR points
Bounding boxes matching
17

18.
Fabio Bagni
San Francisco - 15 May 2019
Object detection results
18

19.
Fabio Bagni
San Francisco - 15 May 2019
Object detection
average time
19
LiDAR output : 10 Hz
Time limit : 100 ms

20.
Thanks for your attention
Fabio Bagni - 228594@studenti.unimore.it
20
Fabio Bagni
San Francisco - 15 May 2019

21.
21

22.
Fabio Bagni
San Francisco - 15 May 2019
Homography
Homography allow to translate camera projection plane on the LiDAR projection plane.
This method make cylindrical surfaces match.
22