Qiu, Kejie, and Shaojie Shen. "Model-aided monocular visual-inertial state estimation and dense mapping." Intelligent Robots and Systems (IROS), 2017 IEEE/RSJ International Conference on. IEEE, 2017.

1. National Chung Cheng University, Taiwan
Robot Vision Laboratory
2018/03/22
Jacky Liu
(Research Note)
Model-Aided Monocular Visual-Inertial
State Estimation and Dense Mapping

2. About this work
Model-Aided Monocular Visual-Inertial State Estimati
on and Dense Mapping
Kejie Qiu1 , Shaojie Shen1
IROS2017 - IEEE/RSJ International Conference on Intelligent Robots an
d Systems
1. Department of Electronic and Computer Engineering, Hong Kong University of Science an
d Technology, Hong Kong, China
2018/03/22 Model-Aided Monocular Visual-Inertial State Estimation and Dense Mapping 2

3. Related works (1/5)
Global localization solutions based on place recognition [1][2] can only
obtain topological localization which is not accurate enough for closed-l
oop control
[1] G. Schindler, M. Brown, and R. Szeliski, “City-scale location recog- nition,” in Computer Vi
sion and Pattern Recognition, 2007. CVPR’07. IEEE Conference on. IEEE, 2007, pp. 1–7.
[2] M. Cummins and P. Newman, “Fab-map: Probabilistic localization and mapping in the spa
ce of appearance,” The International Journal of Robotics Research, vol. 27, no. 6, pp. 647–665,
2008.
2018/03/22 Model-Aided Monocular Visual-Inertial State Estimation and Dense Mapping 3
Problem 1: Global localization

4. Related works (2/5)
The odometry-based methods suffer from long-term drifting while SLA
M-based approaches can not guarantee global consistency before a m
ajor loop closures detection.
Fusion of odometry, SLAM and GNSS may resolve the localization pro
blem in most cases, but it still does not guarantee drift-free localization
at all times.
2018/03/22 Model-Aided Monocular Visual-Inertial State Estimation and Dense Mapping 4
Problem 2: Drift

5. Related works (3/5)
Depth camera
• It has the intrinsic detection limitation impedes outdoor applications.
Stereo camera
• It has limited baseline constrains detection range.
• The extrinsic calibration is another issue for easy use.
2018/03/22 Model-Aided Monocular Visual-Inertial State Estimation and Dense Mapping 5
Problem 3: Sensing range

6. Related works (4/5)
Computation Map
[10] high Dense
LSD-SLAM low Semi-dense
DTAM high Dense
REMODE high Dense(mono)
3D model based[15] low Dense
2018/03/22 Model-Aided Monocular Visual-Inertial State Estimation and Dense Mapping 6

7. Related works (5/5)
Multistate constraint Kalman filter (MSCKF) [6] is a light-weight filter-ba
sed solution of fusing visual odometry and IMU data.
[6] A. I. Mourikis and S. I. Roumeliotis, “A multi-state constraint kalman filter for vision-aided
inertial navigation,” in Robotics and automation, 2007 IEEE international conference on. IEEE,
2007, pp. 3565–3572.
2018/03/22 Model-Aided Monocular Visual-Inertial State Estimation and Dense Mapping 7
MSCKF

8. Contributions (1/2)
1. GPS not available => 3D model based localization
2. Drift problem => 3D model based localization
3. Sensing range => Monocular temporal stereo
2018/03/22 Model-Aided Monocular Visual-Inertial State Estimation and Dense Mapping 8
Solutions

9. Contributions (2/2)
1. Handling simultaneously global localization and real-
time dense mapping problem with minimum sensing
and a rough prior 3D model.
2. Integrating the model-based global localization
method with a tightly-coupled visual-inertial fusion
method to get all-the-time global localization with
high local accuracy.
3. Implementing motion stereo with depth prior
rendered form a prior 3D model to realize accurate
environment awareness.
2018/03/22 Model-Aided Monocular Visual-Inertial State Estimation and Dense Mapping 9

10. Overview
2018/03/22 Model-Aided Monocular Visual-Inertial State Estimation and Dense Mapping 10

11. Overview
2018/03/22 Model-Aided Monocular Visual-Inertial State Estimation and Dense Mapping 11

12. Method
2018/03/22 Model-Aided Monocular Visual-Inertial State Estimation and Dense Mapping 12
1. Global pose fusion (MSCKF)
2. Fused state estimation
3. Semi-global matching

13. Global pose update for visual-inertial odometry
The original visual-inertial odometry (VIO) can already handle local are
a autonomous navigation robustly.
They use MSCKF(multi-state constraint kalman filter) as the VIO imple
mentation that is based on Kalman filter and treat the global pose upda
te as an additional EKF update.
2018/03/22 Model-Aided Monocular Visual-Inertial State Estimation and Dense Mapping 13
Multi-view cost aggregation
VIO
3D model
global pose
All-the-time
global-
consistent
property
Global
Pose
update
Cost
aggregation
Semi-global
matching

14. 2018/03/22 Model-Aided Monocular Visual-Inertial State Estimation and Dense Mapping 14
MSCKF parameters
Global pose update for visual-inertial odometry
Global
Pose
update
Cost
aggregation
Semi-global
matching
rotation
positionvelocity
Bias of gryo and accelemeter
frame 1 frame N
Kalman
gain
residual

15. 2018/03/22 Model-Aided Monocular Visual-Inertial State Estimation and Dense Mapping 15
MSCKF parameters
Global pose update for visual-inertial odometry
Global
Pose
update
Cost
aggregation
Semi-global
matching

16. 2018/03/22 Model-Aided Monocular Visual-Inertial State Estimation and Dense Mapping 16
Global pose update for visual-inertial odometry
Observation (sensing)
Global
Pose
update
Cost
aggregation
Semi-global
matching
MSCKF camera state
global observation
symbol ⊗ denotes quaternion multiplication

17. EKF
2018/03/22 Model-Aided Monocular Visual-Inertial State Estimation and Dense Mapping 17
http://ais.informatik.uni-freiburg.de/teaching/ws12/mapping/pdf/slam04-ekf-slam.pdf

18. Quaternion Multiplication
2018/03/22 Model-Aided Monocular Visual-Inertial State Estimation and Dense Mapping 18
https://www.mathworks.com/help/aeroblks/quaternionmultiplication.html

19. 2018/03/22 Model-Aided Monocular Visual-Inertial State Estimation and Dense Mapping 19
Global pose update for visual-inertial odometry
Update (prediction)
Global
Pose
update
Cost
aggregation
Semi-global
matching

20. Monocular dense mapping with depth prior constrains
Different from spatial stereo where only two calibrated views are used for depth e
stimation, multiple temporal camera views are used for depth estimation with prec
ise pose estimation for every camera frame.
The advantage of using multiple temporal camera views:
1. No baseline limitation (camera mounting distance limitation)
Therefore, the same depth estimation scheme can be used both small indoor
environments and large outdoor cases.
2018/03/22 Model-Aided Monocular Visual-Inertial State Estimation and Dense Mapping 20
A) Multi-view cost aggregation
Cost
aggregation
Global
Pose
update
Cost
aggregation
Semi-global
matching

21. 2018/03/22 Model-Aided Monocular Visual-Inertial State Estimation and Dense Mapping 21
Global
Pose
update
Cost
aggregation
Semi-global
matching

22. Every re-projection pixel is found by back-projecting a pixel in the reference frame
to a 3D point and re-projecting this 3D point into the measurement frame.
2018/03/22 Model-Aided Monocular Visual-Inertial State Estimation and Dense Mapping 22
Reference image measurement image
Global
Pose
update
Cost
aggregation
Semi-global
matching

23. 2018/03/22 Model-Aided Monocular Visual-Inertial State Estimation and Dense Mapping 23
The multiplication with the inverse of the came
ra matrix gives you a ray along which the 3D
point is located.
Depth pixel intensityworld frame rotation matrix
world frame trans. matrix
Global
Pose
update
Cost
aggregation
Semi-global
matching

24. 2018/03/22 Model-Aided Monocular Visual-Inertial State Estimation and Dense Mapping 24
Reference image measurement image
Global
Pose
update
Cost
aggregation
Semi-global
matching

25. Global
Pose
update
Cost
aggregation
Semi-global
matching
2018/03/22 Model-Aided Monocular Visual-Inertial State Estimation and Dense Mapping 25
Δ-Δ
Huber loss

26. 2018/03/22 Model-Aided Monocular Visual-Inertial State Estimation and Dense Mapping 26
Huber loss
Δ-Δ
Global
Pose
update
Cost
aggregation
Semi-global
matching

27. Monocular dense mapping with depth prior constrains
2018/03/22 Model-Aided Monocular Visual-Inertial State Estimation and Dense Mapping 27
B) Semi-global matching
Pixel-wise cost
Smoothness constraints (depth diff = 1)
Smoothness constraints (depth diff > 1)
Global minimization is an NP-complete problem which
cannot be solved in polynomial time.
Global
Pose
update
Cost
aggregation
Semi-global
matching

28. 2018/03/22 Model-Aided Monocular Visual-Inertial State Estimation and Dense Mapping 28
Global
Pose
update
Cost
aggregation
Semi-global
matching

29. Monocular dense mapping with depth prior constrains
TSDF (truncated signed distance function)
Each 3D voxel contains
1. TSDF Depth
2. Photometric intensity
3. Confidence weight of the measurements.
2018/03/22 Model-Aided Monocular Visual-Inertial State Estimation and Dense Mapping 29
C) 3D reconstruction

30. Experimental results
• Implementation detail
2018/03/22 Model-Aided Monocular Visual-Inertial State Estimation and Dense Mapping 30
752x480
12Hz
100Hz
160x108
12Hz
3D model constructed by Altizure.com
MAV
Nvidia Tegra X1
4 CPU 256 GPU core

31. Online SfM service Altizure.com
2018/03/22 Model-Aided Monocular Visual-Inertial State Estimation and Dense Mapping 31

32. State estimation results
2018/03/22 Model-Aided Monocular Visual-Inertial State Estimation and Dense Mapping 32
• The comparison of position, orientation of MSCKF and MSCKF+Model.
(Motion capture Ground truth)

33. State estimation results
2018/03/22 Model-Aided Monocular Visual-Inertial State Estimation and Dense Mapping 33
• The comparison of position, orientation of MSCKF and MSCKF+Model.
(Motion capture Ground truth)

34. Mapping result
• Depth prior-based (2nd row) has better mapping performance at the
texture-less areas such as the green circular areas.
2018/03/22 Model-Aided Monocular Visual-Inertial State Estimation and Dense Mapping 34
Motion stereo
Depth prior-based

35. Object removal (dynamic env.)
2018/03/22 Model-Aided Monocular Visual-Inertial State Estimation and Dense Mapping 35

36. Conclusion
1. Global localization and dense mapping utilizing a known 3D texture
d model
2. Combine several techniques to achieve real-time onboard dense S
LAM.
1. Fast model view rendering
2. Image stabilization
3. Edge-based image alignment
4. Global pose fusion
5. Monocular dense mapping
2018/03/22 Model-Aided Monocular Visual-Inertial State Estimation and Dense Mapping 36

37. Future work
1. Utilizing larger image to enhance mapping quality
2. Refining 3D model online using onboard visual information to acco
unt for the differences between the model and the environment.
2018/03/22 Model-Aided Monocular Visual-Inertial State Estimation and Dense Mapping 37

38. 2018/03/22 Model-Aided Monocular Visual-Inertial State Estimation and Dense Mapping 38
J.Delmerico andD.Scaramuzza, “A Benchmark Comparison of M
onocular Visual-Inertial Odometry Algorithms for Flying Robot
s,” 2018.

39. 2018/03/22 Model-Aided Monocular Visual-Inertial State Estimation and Dense Mapping 39