For the full video of this presentation, please visit: http://www.embedded-vision.com/platinum-members/perceptonic/embedded-vision-training/videos/pages/may-2014-embedded-vision-summit For more information about embedded vision, please visit: http://www.embedded-vision.com Goksel Dedeoglu, Ph.D., Founder and Lab Director of PercepTonic, presents the "Embedded Lucas-Kanade Tracking: How It Works, How to Implement It, and How to Use It" tutorial at the May 2014 Embedded Vision Summit. This tutorial is intended for technical audiences interested in learning about the Lucas-Kanade (LK) tracker, also known as the Kanade-Lucas-Tomasi (KLT) tracker. Invented in the early 80s, this method has been widely used to estimate pixel motion between two consecutive frames. Dedeoglu presents how the LK tracker works and discuss its advantages, limitations, and how to make it more robust and useful. Using DSP-optimized functions from TI's Vision Library (VLIB), he also shows how to detect feature points in real-time and track them from one frame to the next using the LK algorithm. He demonstrates this on Texas Instruments' C6678 Keystone DSP, where he detects and tracks thousands of Harris corner features in 1080p HD resolution video.

1. Goksel Dedeoglu
May 29, 2014
Embedded Lucas-Kanade Tracking:
How it Works, How to Implement It,
and How to Use It

2. Copyright © 2014 Texas Instruments, Inc. 2
Contributors for Algorithms, Optimization,
and Prototypes
Goksel Dedeoglu
goksel@PercepTonic.com
Andrew Miller
amiller@dappervision.com

3. Copyright © 2014 Texas Instruments, Inc. 3
Understanding Motion: Sensors, Algorithms,
Applications
Application
Requirements
• static / moving camera
• frame rate
• motion range
Algorithms
• model fitting
• object tracking
• segmentation
• 3D reconstruction
Automotive
Camera
(visual)
Motion Field
• 2D / 3D
• dense / sparse
• resolution
• Accuracy
(angular/magnitude)
GPS
Inertial
Sensor
Range
Sensor
• cross-traffic alert
• collision avoidance
• parking assist
Human-Device
Interaction
• gesture recognition
• sign recognition
• facial expression
analysis
Video Security
• crowd analysis
• action recognition
• traffic analysis

4. Copyright © 2014 Texas Instruments, Inc. 4
Putting the Lucas-Kanade Tracker on the Map
temporal
support
model
complexity
2
frames
1
frame
N
frames…
single pixel
kxk patch
of pixels
(u,v)
Lucas-Kanade

5. • “An Iterative Image Registration Technique with an Application to Stereo
Vision”, Bruce Lucas and Takeo Kanade, published in 1981.
• Tested and proven over 30+ years in practical applications
• There exist generalizations to more complex object & motion models
• There exist much-simplified versions that work for very small
displacements
• Good understanding of how & when it works well
• “Good Features to Track”, Jianbo Shi and Carlo Tomasi, published in 1994
Copyright © 2014 Texas Instruments, Inc. 5
Lucas-Kanade Estimates Motion Between
Consecutive Frames
frame (t) frame (t+1)
+
estimated motion
(flow vectors)
LK tracker

6. • Assumption: brightness constancy
• Underdetermined system of equations; additional constraints needed.
• Assuming the flow is constant in a small neighborhood of pixels,
estimate the displacement (optical flow) vector h by minimizing
Copyright © 2014 Texas Instruments, Inc. 6
Understanding Lucas-Kanade
kxk
frame (t) frame (t+1)displacement
to be estimated
h
=

7. h
1. The objective function is
2. Linearize
3. Compute the least-squares solution for the displacement h
4. Translate the image G by hx and hy
5. Iterate until convergence!
The Lucas-Kanade Tracking Algorithm
kxk
kxk
Copyright © 2014 Texas Instruments, Inc. 7

8. h
• When the image regions are “textured”
• When the displacements are small
• Common remedy: multi-resolution pyramids
• At the expense of increased computation, more
robust results expected with faster numerical
convergence
• The higher frame-rate the sensor, the better!
Copyright © 2014 Texas Instruments, Inc. 8
When Does Lucas-Kanade Work Best?

9. Copyright © 2014 Texas Instruments, Inc. 9
How to Choose Typical Parameters
Ground
Truth
Motion
Motion Key
(in pixels)
-5 +5
+5
-5
Scene
patch size
pyramidlevels

10. Copyright © 2014 Texas Instruments, Inc. 10
Typical Lucas-Kanade Tracking Pipeline
Gaussian
Pyramid
Gradient
Pyramid
Saliency
Score
Non-maximum
Suppression
Lucas-Kanade
Feature Tracker
frame t
output: flow vectors
locate
salient
features
estimate motion of
selected features
multi-scale
gradients
multi-scale
images
frame t-1

11. Copyright © 2014 Texas Instruments, Inc. 11
Lucas-Kanade with TI’s Vision Library VLIB
Gaussian
Pyramid
Gradient
Pyramid
Harris ,
FAST,ORB
Non-maximum
Suppression
Lucas-Kanade
Feature Tracker
frame t
output: flow vectors
locate
salient
features
estimate motion of
selected features
multi-scale
gradients
multi-scale
images
VLIB
VLIBVLIB
VLIB
VLIB
frame t-1

12. S32 VLIB_trackFeaturesLucasKanade_7x7_track_error(
const U08 * restrict im1,
const U08 * restrict im2,
const S16 * restrict gradX,
const S16 * restrict gradY,
S32 width,
S32 height,
S32 nfeatures,
const S16 x[], // SQ11.4
const S16 y[], // SQ11.4
S16 outx[], // SQ11.4
S16 outy[], // SQ11.4
S32 max_iters,
const U08 * restrict scratch_klt,
U32 track_error[],
U08 patch)
Copyright © 2014 Texas Instruments, Inc. 12
The API for the Lucas-Kanade Tracker in VLIB

13. Copyright © 2014 Texas Instruments, Inc. 13
Studying the Fixed-Point Approximation
Floating Point
(e.g., OpenCV)
Fixed Point
Optimized VLIB
Ground
Truth Motion
Motion Key
(in pixels)
-5 +5
+5
-5
Scene

14. Copyright © 2014 Texas Instruments, Inc. 14
Live Demonstration on TI’s C6678 Keystone DSP

15. Copyright © 2014 Texas Instruments, Inc. 15
Prototype: Gesture Recognition
rotate image
left / right
navigate to
select image

16. • Computer Vision is evolving, with Open Source libraries expanding fast,
but with limited scrutiny & quality control. There exist two “Lucas-
Kanade” optical flow functions in OpenCV:
• Embedded developers & architects beware!
Copyright © 2014 Texas Instruments, Inc. 16
Quality Control for “Lucas-Kanade”
calcOpticalFlowPyrLK
pyramid-based
good results
“LK-brand”
cvCalcOpticalFlowLK
single-step algorithm
can be 100x faster
handles very small motion
now obsolete
ground truth
optical flow

17. • Lucas-Kanade is a well-understood & widely deployed method for
tracking feature points
• We have implemented an embedded Lucas-Kanade tracker on the
Keystone C6678 SoC; APIs are available in TI’s Vision Library VLIB
• Three key messages:
• Advantages: tested & proven over 30+ years, works reliably in
textured image regions and small motion vectors
• Challenge: computationally demanding, algorithmic extensions
continue
• With the right programmable processor and careful design trade-
offs, Lucas-Kanade can be implemented with cost & power
consumption suitable for embedded systems
Copyright © 2014 Texas Instruments, Inc. 17
Conclusions

18. • “An Iterative Image Registration Technique with an Application to
Stereo Vision”, Bruce Lucas and Takeo Kanade, Proceedings of the 7th
International Joint Conference on Artificial Intelligence (IJCAI '81),
April, 1981, pp. 674-679. (URL)
• “Lucas-Kanade 20 Years On”, project lead by Simon Baker & Iain
Matthews at the Robotics Institute of Carnegie Mellon University (URL)
• “Determining Optical Flow”, Berthold Horn and Brian Schunck, Artificial
Intelligence, vol. 17, pp. 185-203, 1981.
• For related algorithms, benchmarks and datasets, see
• Middlebury dataset: http://vision.middlebury.edu/flow
• KITTI dataset: http://www.cvlibs.net/datasets/kitti
Copyright © 2014 Texas Instruments, Inc. 18
Resources for Further Investigation

19. PercepTonic is hosting a demo
of the Lucas-Kanade tracker
on a TI Keystone DSP
at the Technology Showcase.
Please stop by!
Copyright © 2014 Texas Instruments, Inc. 19
Hands-On Demonstration