1. Software
Feature implementation and computation
Andrea Vedaldi, University of Oxford
https://sites.google.com/site/eccv12features/
goo.gl/jKC2J

2. 2
VLFeat
MATLAB ease of use
Covariant detectors, MSER, SIFT, etc ...
OpenCV
One-shop experience
The most popular vision library
Others
There is certainly no lack of software!

3. 3
VLFeat
MATLAB ease of use
Covariant detectors, MSER, SIFT, etc ...
OpenCV
One-shop experience
The most popular vision library
Others
There is certainly no lack of software!

4. VLFeat 4
An open and portable library of CV software
http://www.vlfeat.org/benchmarks/index.html
Quick Start with MATLAB
>>"untar"http://www.vlfeat.org/download/vlfeat50.9.165bin.tar.gz"
>>"run vlfeat-0.9.16/toolbox/vl_setup
Mac Linux Windows
Since 2008
ACM OSS award

5. SIFT detector and descriptor 5
1. load an image
imPath = fullfile(‘oxford.jpg’) ;
im = imread(imPath) ;
2. convert it to single precision gray scale
imgs = im2single(rgb2gray(im)) ;
3. run SIFT
[frames, descrs] = vl_sift(imgs) ;
4. visualise keypoints
imagesc(im) ; colormap gray; hold on ;
vl_plotframe(frames) ;
5. visualise descriptors
vl_plotsiftdescriptor(...
descrs(:,432), frames(:,432)) ;

6. Example: matching features 6
684 tentative matches
nr : :h f t :a
r / /h f t /a
r h f t : /a

7. Example: stitching 7
684 tentative matches
492 (76%) inliers mosaic
• Live demonstration

8. Other covariant detectors 8
frames = vl_covdet(imgs, 'verbose') ;
imagesc(im) ; vl_plotframes(frames) ;
Do not forget to convert the images to
grayscale in single precision. Use
im2single(im) and not single(im) !

9. Understanding feature frames 9
x
   
x a11 a12 b
x t1
affine tf. = +
1 y a21 a22 b
y t2
b
x
b
y
y
• frames has a feature frame for each column
containing a stacked affine transformation (A,T)
2 3
t1
6 t2 7
6 7    
6a11 7 x a11 a12 b
x t1
frames(:,i) = 6
6a21 7
7 = +
6 7 y a21 a22 b
y t2
4a12 5
a22

10. A frame as an oriented ellipse 10
reference = two vectors
uniquely identifies
the affine
1 transformation
b
x
b
y
reference = oriented ellipse

11. Different cornerness measures 11
frames = vl_covdet(imgs, 'method', 'Hessian') ;
DoG Hessian
• Difference of Gaussian (Laplacian, trace of Hessian, or SIFT)
• Hessian (determinant of)
HarrisLaplace HessianLaplace

12. Different cornerness measures 12
frames = vl_covdet(imgs, 'method', 'HarrisLaplace') ;
HarrisLaplace HessianLaplace
• Harris Laplace (Harris cornerness, Laplacian for scale selection)
• Hessian Laplace
MultiscaleHarris MultiscaleHessian

13. Different cornerness measures 13
frames = vl_covdet(imgs, 'method', 'MultiscaleHarris') ;
MultiscaleHarris MultiscaleHessian
• Multiscale Harris (just a bunch of Harris)
• Multiscale Hessian

14. Affine adaptation 14
frames = vl_covdet(imgs, 'EstimateAffineShape', true) ;

15. Upright features 15
By default the rotation is
selected so that the vertical
axis stays vertical.
In formulas
   
x a11 0 b
x t1
= +
y a21 a22 b
y t2
Remark: for general affine transformations
the horizontal direction may appear rotated
even for upright features.

16. Orientation detection 16
frames = vl_covdet(imgs, 'EstimateOrientations', true) ;

17. Computing descriptors 17
[frames, descrs] = vl_covdet(imgs) ;
• Each column of descrs is a stacked SIFT descriptor
• How is a SIFT descriptor associated to a feature?
• What about other descriptors?

18. Computing descriptors 18
[frames, descrs] = vl_covdet(imgs, 'Descriptor', 'patch') ;
Each column of descrs is a stacked
normalised image patch.

19. The geometry of an normalised patch 19
• Normalisation and patch resampling is controlled by three parameters:
(antialiasing effect)
affine tf.
(size of patch in normalised units)
(number of pixels)

20. Example: normalised patch for SIFT 20
=1
1 3 6
f v w
= 15

21. Custom frames 21
Example: running affine adaptation on a grid
frames = vl_covdet(imgs, ...
'Frames', frames, ...
'EstimateAffineShape', true, ...
'EstiamteOrientation', true) ;

22. Obtaining additional information on the features 22
[frames, descrs, info] = vl_covdet(imgs);
info =
gss: [1x1 struct]
css: [1x1 struct]
peakScores: [1x351 single]
edgeScores: [1x351 single]
orientationScores: [1x351 single]
laplacianScaleScores: [1x351 single]
120 120
100 100
80 80
Occurences
Occurences
60 60
40 40
20 20
0 0
−0.1 −0.05 0 0.05 0.1 0 2 4 6 8 10
Peak Score Edge Score

23. Obtaining the scale space 23
[frames, descrs, info] = vl_covdet(imgs);
info =
gss: [1x1 struct] Gaussian
css: [1x1 struct] scale space
peakScores: [1x351 single]
edgeScores: [1x351 single]
orientationScores: [1x351 single]
laplacianScaleScores: [1x351 single]
octave −1 octave 0 octave 1
octave 2 octave 3

24. Obtaining the scale space 24
[frames, descrs, info] = vl_covdet(imgs);
info =
gss: [1x1 struct] cornerness
css: [1x1 struct] measure
peakScores: [1x351 single]
edgeScores: [1x351 single]
orientationScores: [1x351 single]
laplacianScaleScores: [1x351 single]
octave −1 octave 0 octave 1
octave 2 octave 3

25. Performance comparison 25
Oxford 5K lite retrieval benchmark
Num. features / image mAP
VL HessianLaplace (double) VL HessianLaplace (double)
7131 79.9%
VL DoG (double) 5631 DoG (double)
VL 85.1%
VL Hessian (double) 3411 VL Hessian (double) 77.8%
VL DoG (no aff) 2794 VL DoG (no aff) 74.0%
VL HarrisLaplace (double) 2316 VL HarrisLaplace (double) 77.8%
CMP Hessian 2292 CMP Hessian 77.1%
VL HessianLaplace 2020 VL HessianLaplace 73.8%
VL DoG 1854 VL DoG 80.1%
VGG hes 1517 VGG hes 74.6%
VGG har 1384 VGG har 71.9%
VL Hessian 1089 VL Hessian 68.7%
VL HarrisLaplace 765 VL HarrisLaplace 70.4%
0 2000 4000 6000 8000 0.6500 0.7125 0.7750 0.8375 0.9000

26. Repeatability: viewpoint 26
Repeatability (graf)
100
VL DoG
VL Hessian
90
VL HessianLaplace
VL HarrisLaplace
80
VL DoG (double)
VL Hessian (double)
70
VL HessianLaplace (double)
Repeatability (graf)
VL HarrisLaplace (double)
60
VLFeat SIFT
CMP Hessian
50 VGG hes
VGG har
40
30
20
10
0
30 40 50 60 20
Viewpoint angle

27. Repeatability: viewpoint 27
Repeatability (wall)
100
VL DoG
VL Hessian
90
VL HessianLaplace
VL HarrisLaplace
80
VL DoG (double)
VL Hessian (double)
70
VL HessianLaplace (double)
Repeatability (wall)
VL HarrisLaplace (double)
60
VLFeat SIFT
CMP Hessian
50 VGG hes
VGG har
40
30
20
10
0
30 40 50 60 20
Viewpoint angle

28. Repeatability: scale 28
Repeatability (boat)
100
VL DoG
VL Hessian
90
VL HessianLaplace
VL HarrisLaplace
80
VL DoG (double)
VL Hessian (double)
70
VL HessianLaplace (double)
Repeatability (boat)
VL HarrisLaplace (double)
60
VLFeat SIFT
CMP Hessian
50 VGG hes
VGG har
40
30
20
10
0
1.38 1.9 2.35 2.8 1.12
Scale changes

29. Repeatability: JPEG compression 29
Repeatability (ubc)
100
VL DoG
VL Hessian
90
VL HessianLaplace
VL HarrisLaplace
80
VL DoG (double)
VL Hessian (double)
70
VL HessianLaplace (double)
Repeatability (ubc)
VL HarrisLaplace (double)
60
VLFeat SIFT
CMP Hessian
50 VGG hes
VGG har
40
30
20
10
0
80 90 95 98 60
JPEG compression %

30. Maximally Stable Extremal Regions (MSER) 30
[Matas et al. 2002]
• Extracts MSER regions regions and fitted ellipses frames:
imgs = uint8(rgb2gray(I)) ;
[regions,"frames]"="vl_mser(imgs)";
• Control region stability:
[regions,"frames]"="vl_mser(imgs,"...
""""""""""""""""""""""""""""‘MinDiversity’,"0.7,"...
""""""""""""""""""""""""""""‘MaxVariation’,"0.2,"...
""""""""""""""""""""""""""""‘Delta’,"10)";

31. Many other features and tools 31
superpixels MSER agglomerative information bottleneck distance transform
di erent mode seeking algo-
sent (some of) the data points
nal to the Parzen density esti-
wards the mode approximately
ates mean shift trajectories by
xt and in Fig. 2, medoid shifts
circles. This disconnects por-
alleviated (but not solved) by
ect. 3) seeks the energy modes
s, trading-o mode over- and
randomized kd-trees fast k-means feature maps, stochastic gradient SVM
s, where d is the dimension-
is only O(dN 2 T ), where T Fast Dense SIFT
lever implementations yield

32. Many other features and tools 32
SLIC superpixels Histogram of Oriented Gradients (HOG)
HARVEST Programme
New SGD Learning of SVMs

33. OpenCV 33
• The largest, best known vision library
• Features
- OpenCV 2.1:
Harris, Tomasi&Shi, MSER, DoG, SURF, STAR and FAST,
DoG + SIFT, SURF, HARRIS, GoodFeaturesToTrack
- OpenCV 2.4:
ORB, BRIEF, and FREAK
Example: computing ORB features.
#include <opencv2/core/core.hpp>
#include <opencv2/features2d/features2d.hpp>
#include <vector>
Mat image = importImage8UC1(in[IN_I]);
Mat descriptors;
std::vector<KeyPoint> frames;
ORB orb(numFeatures, scaleFactor, numLevels, edgeThreshold, 0,
wtak, scoreTypeToORBPar(scoreType), patchSize);
orb(image, Mat(), frames);

34. This is not all folks 34
• Abundance of open source / free software for CV
- Always check out authors home pages
• Examples
Rob Hess SIFT SURF
http://robwhess.github.com/opensift/ http://www.vision.ee.ethz.ch/~surf/
Open SURF
http://www.chrisevansdev.com/computer-vision-opensurf.html
FAST corners
http://www.edwardrosten.com/work/fast.html
LIOP descriptors
http://vision.ia.ac.cn/Students/bfan/index.htm ...

35. Credits 35
Karel Lenc Daniele Perrone Michal Perdoch
HARVEST Programme
Krystian Mikolajczyk Tinne Tuytelaars Jiri Matas Cordelia Schmid
Andrew Zisserman