1. Localization of MAV in GPS-
denied Environment Using
Embedded Stereo Camera
Syaril Azrad, Farid Kendoul,Fadhil
Mohamad,, Kenzo Nonami
Department of Mechanical
Engineering,Nonami Lab, Chiba
University
2. Research Background
• Vision based Autonomous Q-Rotor
Research in Chiba University
– Started in 2008 , Participate in
1st US-Asian Demonstration and Assessment of Micro Aerial and
, MAV’08 Agra, India,
– Single camera approach
• Color-based object tracking algoritm (visual-servoing)
• Feature-based/Optical Flow (OF)
– Stereo camera approach
• Ground stereo camera based hovering
• Fully-embedded based object tracking
3. Current Research Background
• Localization of MAV using embedded
camera
– Research has been conducted by Farid et. al
using single embedded camera using Optical
Flow to localize rotorcraft position
– Height estimation improvement
• Fusion with Pressure Sensor
• Using stereo camera for higher-precision
4. Altitude Estimation For MAVs ?
• GPS
• Pressure sensor
• Laser Range Finder
• Radar Sensors
• Vision System
– Single Camera
– Stereo Camera
5. For Small UAVs (MAV) Altitude Estimation
We propose…
• Embedded Lightweight Stereo Camera
• Fusion Optical Flow (SFM algorithm) with
Image Homography
• Fusion Optical Flow (SFM algorithm) with
Scale Invariant Feature Transform (SIFT)
–based feature matching
7. 18th
Aug. 2010 MOVIC 2010 Tokyo,JAPAN
7
•Size : 54cm x 54cm x 20cm
•Total weight : < 700 grams
•Flight time : 10 min
•Range : 1~2 km
•Total cost : < 4000 USD
8. Proposed Vision-based MAV
localization algorithm
• Horizontal Position using Optical Flow-
based algorithm
• Altitude Position using Optical Flow fused
with Homography/SIFT-based matching
approach
9. 18th
Aug. 2010 MOVIC 2010 Tokyo,JAPAN
9
Computing Optical Flow
Implementation
We use Lucas-Kanade Method
10. 18th
Aug. 2010 MOVIC 2010 Tokyo,JAPAN
10
the feature point (x, y) in the next image I2 will be at (x+dx, y+dy),
and the same process will be repeated in the next step (t +Δt), providing the
optical flow and a simple feature tracking algorithm.
12. Or can be expressed as below
Then we can express the OF equation as below
WHAT DOES THE EQUATION ABOVE MEANS?
The velocity of can be expressed as Pi(xi
,yi)
A perspective-central camera model
maps Pi(xi ,yi)
13. If we have data from IMU about the rotational velocity of the camera on the body
of our MAVs we can get purely translational part
Meaning = velocity and position
Because OF calculated from images contains rotational and translational part
Kendoul et. al (2007)
14. The strategy is an estimation problem with the state vector
So for KF dynamics model, we can write the measurement equation plus noise
With H is as below, from our optical flow expression in equation (5)
15. Now, after estimating Oftrans, we can use them for measurement vector for
(MAV)translational velocity and structure parameter
Both our cameras are mounted on the MAV, and assuming camera motion is smooth,
we can write the dynamic model as below with γ is the camera acc.
We use the model proposed by Kendoul et. al for depth (altitude) as below
We can write the discrete dynamic system using the state vector X
16. Which is non-linear, we Implement Extended Kalman Filter as proposed by Kendoul et. A
and estimate translational velocity and depth (altitude)
While the observation of the discrete model is as below
17. OF raw data from camera,
Attitude data from IMU
Estimate the translational
part of OF, separate the
rotational part
Estimate the camera
velocity and depth(altitude)
18. Move along x-axis with various
attitude
X
Y
Verification Experiment of Image Algorithm Fused with IMU data
5 10 15 20 25 30
- 0.4
- 0.2
0
0.2
0.4
Time [s ]
X,Ydistance[m]
19. Localization of MAV using Optical
Flow and SIFT-matching technique
• SIFT?
– Detect & describe feature in local image
• In our computation we use SIFTGPU (Wu,2009)
to speed up the matching
• Matching result is filtered by RANSAC algorithm
to separate between outlier in inlier
• Applying threads in computations we can get the
Optical Flow based algorithm 50fps and SIFT
matching 7-8fps. This inculding triangulation
23. Implementation Strategy
1. Vicon Data
2. Control Program
3. Receive Image Data
4. Process Image Data
Image Processing
1.Optical Flow Base
2.Stereo Feature Matching
GPU Graphical Processing Unit
Socket
1 Computer with two separate Process, or
2 Computers
24. Implementation
• Implemented on 1 Computer due to high
capability, but have 2 share GPU
• Core-i7 (4 Cores 8 Threads)
– Separate Core for Image Processing
– Separate Thread for Control
– Separate Core for Receiving Image Data
– Separate Thread for Vicon Data
25. Results
• Process that has been implemented with
stable frequency
– PID Control (30Hz)
– Vicon Data Acquisition (30Hz)
– Image Processing SIFT (8Hz) and Optical
Flow (15Hz)
Standard GPS vertical precision between 25m to 50m
Pressure sensor depends on environment will have an error between 6% and 7% moreover, pressure sensor only calculate relative height
Laser altimeter is good but for our type of MAV it would be not suitable due to its energy consumption and surface requirement
Radar sensor would not be suitable due to its high cost and energy consuming
Thus we still think the vision system is one of the best alternative. However due to its high computing cost requirement, depending on algorithm there is an issue whether we can implement on-board or host-based.
Our platform is off-the shelf platform from Ascending Technologies, German. The size of our platform is 54 cm in diameters, and 20cm of height, with total weight of less than 700 grams, and flight time up to 12 minutes. However we state here 10 minutes for safety purposes. The flight range would be 1 to 2 kilometers. And the total cost is less than 4000 USD, which is around 350 thousand yen. The details of the price can be seen on the right corner table. In this platform we incorporated the flight computer GUMSTIX, which has a computing speed of 400MHz, equipped with wi-fi. The GUMSTIX is linux-based. We also use the Crossbow MNAV IMU, pressure, gyro sensor for avionics sensor, and mount a small analog camera on the bottom of the platform, which operates using 1.3GHz band.
The Optical Flow algorithm can be summarized by the picture here. When we select the object of interest as shown as the blue shape here. The Tomasi-Shi algorithm will automatically specify good features to track. We can limit the good features to track to a certain number of features. In this example five point is selected. Even though it is not exactly the center of the OOI, we output the center of the five features as the object center, and relay it to the controller as an input. Based on Optical Flow algorithm, the five same features will be tracked in the next frame. A process of reselection of feature is executed when the feature of interest is lost in detection. We can summarize the Optical Flow algorithm as in the next slide.
As shown in the formula above, feature point x,y will be x+dx and y+dy in the next image frame I2. dx and dy is defined when the argument of the minimum of this equation is fulfilled. The same process then will be repeated in the next step, to provide an optical flow and simple feature tracking algorithm. However there is a problem when feature detection. First it only selected limited points or features, next it will lose the object profile when the object is lost from the screen due to movement or just simply image blur due to bad video streaming reception. The feature based algorithm does not “remember” the object when it first selected. Thus we introduce a hypothetical algorithm to solve the problem, combining the color based and feature based algorithm.
The study has been done by Kendoul et al. using single camera, however we want to exploit the potential of stereo camera especially to predict correctly depth (Zi) used in the equation.