1. The document describes a visual-inertial odometry algorithm that directly couples inertial and visual observations into a nonlinear optimization problem. 2. It presents batch and online versions of the algorithm and shows accuracy improves as the active window size increases. 3. Experimental results on a benchmark dataset show the algorithm achieves higher accuracy than previous methods and can handle real-world mobile phone data.

1. National Chung Cheng University, Taiwan
Robot Vision Laboratory
2018/05/24
Jacky Liu
(Research Note)
Visual Inertial Odometry using Coupled
Nonlinear Optimization

2. About this work
Visual Inertial Odometry using Coupled Nonlinear O
ptimization
Euntae Hong and Jongwoo Lim1
2017 IEEE/RSJ International Conference on Intelligent Robots and Syst
ems (IROS) September 24–28, 2017, Vancouver, BC, Canada
1. Division of Computer Science and Engineering, Hanyang University, Seoul
2018/05/24 Visual Inertial Odometry using Coupled Nonlinear Optimization 2

3. Overview
Joint optimization of camera pose with noisy IMU data
and visual feature locations.
Achieves good accuracy, and can be easily implemented
using publicly available non-linear optimization toolkits.
2018/05/24 Visual Inertial Odometry using Coupled Nonlinear Optimization 3
Visual inertial odometry (VIO) algorithm

4. Contributions
1. They propose a simple and unified framework that directly
couples inertial and visual observations into one non-linear
optimization problem for the VIO task.
2. The batch and online versions of VIO is presented and we
show that the accuracy-speed tradeoff according to the
active window size.
3. The experimental results show that the proposed algorithm
can handle real-world data captured by a mobile phone,
and achieves higher accuracy compared to the prior art in a
well-known benchmark dataset.
2018/05/24 Visual Inertial Odometry using Coupled Nonlinear Optimization 4

5. ABOUT IMU (INERTIAL MEASUREMENT UNIT )
2018/05/24
Visual Inertial Odometry using Coupled Non
linear Optimization
5

6. About IMU (Inertial measurement unit )
IMU
Accelerometers
acceleration
Gyroscope
Rotation
Magnetometer
Earth magnetic field
2018/05/24 Visual Inertial Odometry using Coupled Nonlinear Optimization 6
Further reading
[1] Inertial Measurement Units I - https://stanford.edu/class/ee267/lectures/lecture9.pdf
[2] Introduction to Navigation Systems - https://www.slideshare.net/JosephHennawy/introduc
tion-to-navigation-systems

7. About IMU - Accelerometers
2018/05/24 Visual Inertial Odometry using Coupled Nonlinear Optimization 7

8. About IMU - Gyroscope
2018/05/24 Visual Inertial Odometry using Coupled Nonlinear Optimization 8
(科氏力)
https://www.slideshare.net/JosephHennawy/introduction-to-navigation-systems

9. About IMU - Magnetometer
2018/05/24 Visual Inertial Odometry using Coupled Nonlinear Optimization 9
Hall effect

10. Introduction
Odometry
• IMU only: severe drift due to sensor noise.
• GPS-based: cannot estimate small-scale (sub-meter) motions.
• Visual: error-prone under motion blur, low illumination and texture-less scene.
• VIO: combine 2 sensor to overcome their own weakness.
2018/05/24 Visual Inertial Odometry using Coupled Nonlinear Optimization 10
Why do we need VIO(Visual-inertial odometry)?
VIO
Filtering Optimization

11. Introduction
Filtering Optimization
Computation fast slow
Accuracy
Less accurate
than opt method
More accurate
2018/05/24 Visual Inertial Odometry using Coupled Nonlinear Optimization 11
Method comparison
Because opt method can
use more information
Opt could be fast!
We can adjust the “window size” of optimization method to
meet the computation requirement.

12. RELATED WORK
2018/05/24
Visual Inertial Odometry using Coupled Non
linear Optimization
12

13. Related work
2018/05/24 Visual Inertial Odometry using Coupled Nonlinear Optimization 13
Filtering based
• Filtering based: [1] EKF, UKF, MSCKF[4]
fusing different sensor data by operating on a probabilistic state
representation with the mean and covariance.
• [2] proposed a loosely coupled SLAM system using IMU observations to get
the real scale of the estimated map.
[1] E. A. Wan and R. Van Der Merwe, “The unscented kalman filter for nonlinear estimation,” in Adaptive Systems for Sig
nal Processing, Communications, and Control Symposium 2000. AS-SPCC. The IEEE 2000. Ieee, 2000, pp. 153–158.
[4] A. I. Mourikis and S. I. Roumeliotis, “A multi-state constraint kalman filter for vision-aided inertial navigation,” in Proc
eedings 2007 IEEE International Conference on Robotics and Automation. IEEE, 2007, pp. 3565–3572.

14. Related work
• Optimization based: [7] reduce the computation cost by only optimize the
poses in a small local window. [17] using incremental smoothing techniques.
• These approaches use a method of integrating IMU sensor reading, they
suffer from the need for re-evaluate summation to be performed again
according to the changed rotation at time of optimization.
2018/05/24 Visual Inertial Odometry using Coupled Nonlinear Optimization 14
Optimization based
[7] E. Mouragnon, M. Lhuillier, M. Dhome, F. Dekeyser, and P. Sayd, “Generic and real-time structure from motion using local bundle a
djustment,” Image and Vision Computing, vol. 27, no. 8, pp. 1178– 1193, 2009.
[17] M. Kaess, H. Johannsson, R. Roberts, V. Ila, J. J. Leonard, and F. Dellaert, “isam2: Incremental smoothing and mapping using the bay
es tree,” The International Journal of Robotics Research, p. 0278364911430419, 2011.

15. Method
2018/05/24 Visual Inertial Odometry using Coupled Nonlinear Optimization 15

16. Method
• Kalman filter is used to mitigate the noise in the sensor readings.
• The acquisition frequency of the IMU data is not consistent with and is much
higher than the video frame rate.
2018/05/24 Visual Inertial Odometry using Coupled Nonlinear Optimization 16
Spline interpolated IMU measurements

17. Method
• The integration of noisy raw IMU data in the filter is in
general less accurate than optimizing the states with the
IMU data.
• Instead of modeling the bias in the filtering framework, the
bias parameters are estimated in the optimization together
with the camera poses and landmark positions by fully
utilizing the visual feature observations.
2018/05/24 Visual Inertial Odometry using Coupled Nonlinear Optimization 17
Raw IMU filter
Optimize with filtered
IMU and image feature
Raw IMU
Optimize with raw IMU and
image feature

18. Estimate error (1/3)
2018/05/24 Visual Inertial Odometry using Coupled Nonlinear Optimization 18

19. Estimate error (2/3)
2018/05/24 Visual Inertial Odometry using Coupled Nonlinear Optimization 19

20. Estimate error (3/3)
2018/05/24 Visual Inertial Odometry using Coupled Nonlinear Optimization 20
Active VO keyframes Active landmarks

21. 2018/05/24 Visual Inertial Odometry using Coupled Nonlinear Optimization 21
Harris corner
detector & KLT

22. Scale
To determine the initial scale factor 𝑠 they compute the actual travel
distance 𝑑𝐼𝑀𝑈 between the first two keyframes by integrating the IMU data:
2018/05/24 Visual Inertial Odometry using Coupled Nonlinear Optimization 22
Distance estimated by IMU
Distance estimated by VO
Calculate the scale factor

23. Experimental results
Dataset: EuRoC
2018/05/24 Visual Inertial Odometry using Coupled Nonlinear Optimization 23
Average distance error on EuRoC dataset (unit: meters).
Fast and accurate
enough
window size
Optimization-based
approaches

24. Process time
2018/05/24 Visual Inertial Odometry using Coupled Nonlinear Optimization 24

25. Trajectory
2018/05/24 Visual Inertial Odometry using Coupled Nonlinear Optimization 25
Room1.1

26. 2018/05/24 Visual Inertial Odometry using Coupled Nonlinear Optimization 26
Averageofaccumulatederror(m) OKVIS
Batch
Online
frame
Room1.1
Room2.1
Room1.2
Room2.2

27. Mobile phone exp.
2018/05/24 Visual Inertial Odometry using Coupled Nonlinear Optimization 27
Samsung Galaxy Note4
• Whole trajectory is 70m
• Start to end error is 1.3m

28. Conclusion
• They propose a novel VIO algorithm which directly
optimizes the camera poses using the visual and
inertial measurements.
• Algorithm is easy to implement
• Online version can process the incoming data online
by using sliding-window
• Work with noisy mobile phone data
2018/05/24 Visual Inertial Odometry using Coupled Nonlinear Optimization 28