Michal's presentation on David Lowe's Scale Invariant Feature Transform (IJCV 2004)

4. Location Recognition

5. Panoramic Image Matching

14. Scale-Space

15. Scale-Space To scale: take every second pixel in each row and column (another approach: average 4 pixels)

16. Difference of Gaussians (DOG) Sigma 4 Sigma2-Sigma4 Sigma 2

17. Scale-Space with DOG

18. Scale-Space with DOG

20. Keypoints Too many keypoints, some are unstable

21. Step II: Keypoint Localization Reject points with low contrast Reject points that are localized along an edge.

24. Keypoint Localization Use Quadric Taylor Expansion of the scale-space function, so that the origin is at the sample point: (x is the offset from this point) Calculate extermum: if X > 0.5: the extermum lies closer to a different point (Need to recalculate…) Otherwise: add offset to the sample point location to get the estimated extremum ^

25. Reject Low Contrast Keypoints Calculate value of D at extremum point X: if |D(X)| < 0.03: discard keypoint for having a low contrast

26. Reject Low Contrast Keypoints

27. Eliminate Edge Responses: DoG function might have strong response along edges, even if unstable to small amounts of noise Edge identification: large principal curvature across the edge, but small one in perpendicular direction. Note ♥ : It's easy to show that the two principle curvatures (i.e., the min and max curvatures) are always along directions perpendicular to each other. In general, finding the principle directions amounts to solving a nxn eigenvalue problem

28. Eliminate Edge Responses: No need to explicitly calculate the eigenvalues – we only need their ratio!! a = small eigenvalue b = large eigenvalue r = ratio between large and small eigenvalues (r=a/b) (r+1)^2/r is at min when a=b, and increases as the ratio increases

29. Eliminate Edge Responses: To check if the ratio of the principal curvatures is below a threshold r, we only need to check if: Use r = 10 to reject keypoints that lay along an edge

30. Reject Near-Edge Keypoints

31. 832 keypoints 729 keypoints (eliminate low contrast) 536 keypoints (eliminate edge keypoints)

32. Step III: Orientation Assignment Each keypoint is assigned 1 or more orientations, based on local image gradient directions. Data is trasformed relative to the assigned orientation, scale and location hence providing invariance to these transformations

33. Gradient Calculation The scale of the keypoint is used to select the Gaussian image L we’ll work on (image with closest scale) – All computations are performed in a scale-invariant manner. We calculate gradient magnitue and orientation using pixel differences:

34. Gradient Calculation

35. Gradient Calculation

36. Orientation Histogram Orientation histogram with 36 bins (each bin covers 10 degrees) Each sample added to the histogram bin is weighted by its gradient magnitude and by a Gaussian weighted circular window with theta = 1.5 times that of the keypoint scale

37. Orientation Histogram: Detect highest peak and local peaks that are within 80% of the highest peak. Use these to assign (1 or more) orientations

38. Step IV: Local Image Descriptor Previous operations imposed a local 2D coordination system, which provides invariance to image location, scale and orientation We wish to compute descriptors for the local image regions: 1. Highly distinctive 2. Invariant as possible to remaining variations (illumination, 3D viewpoint…)

39. Descriptor Representation Use the scale of the keypoint to select the level of Gaussian blur. Sample the gradient magnitude and orientation around the keypoint Assign weight to magnitude using a Gaussian weighted function with theta = ½ width of descriptor window (provides gradual change & gives less emphasis to gradients far from the keypoint Use a descriptor array with histogram bins

40. Descriptor Representation

41. Descriptor Representation :

42. Invariance to Affine Illumination Changes: * Multiplication by a constant: Normalize vector to unit length: A change in each pixel: pixel -> a * pixel (each pixel multiplied by a constant) will result – gradient -> gradient * a. This will be canceled by the normalization * Addition of a constant: pixel -> pixel + a Has no effect on the gradient

43. Partial Invariance To Non Affine Illumination changes: Will cause large change in relative magnitude, but is unlikely to affect gradient orientations. Solution: reduce the influence of large gradient magintudes by thresholding the values to be no larger than 0.2, then normalize them to unit length.

44. Partial Invariance To Affine Change In Viewpoint Angle:

45. Object Recognition: Best candidate match for each keypoint is nearest neighbour in database Problem: many background features will not have a matching pair in database resulting in a false match Global threshold to descriptors does not perform well since some descriptors are more discriminating than others Solution: Compare distance to closet neighbour to that of the second closet neighbour (that comes from a different object)

46. Results:

47. More Results:

48. More Results (not as successful…):

49. Image matching:

53. The End…