BSc thesis by Jordi Sánchez Escué. ETSETB UPC (25/07/2014) More details:

1. Bundling interest points
for object classification
Jordi Sánchez Escué
Supervised by
Xavier Giró i Nieto
Carles Ventura Royo

2. Contents
● Introduction
● State of the art
● System Architecture
○ Feature extraction
○ Classification
○ Evaluation
● Experiments
○ Class aggregation of interest points
○ Bundling interest points
○ Class aggregation & Bundling
● Conclusions & Future work
1

3. Contents
● Introduction
● State of the art
● System Architecture
○ Feature extraction
○ Classification
○ Evaluation
● Experiments
○ Class aggregation of interest points
○ Bundling interest points
○ Class aggregation & Bundling
● Conclusions & Future work
2

4. Introduction
● Does this image contain a plane?
3

5. Introduction
● Does this image contain a plane?
● Which type of flower is it?
4

6. Introduction
● Mobile Visual Search
○ Generalist: Google Goggles
○ Leaf-based: Leafsnap
● Fine-grained classification
○ Mushrooms
○ Flowers
5

7. Introduction
● Textures around some interest points
6

8. Introduction
● Features based on regions
7

9. Introduction
● Explore combination: points & regions
8

10. Introduction
● Project Requirements and Goals
○ Comparative study bundling interest points
9

11. Introduction
● Project Requirements and Goals
○ Software Development
10

12. Contents
● Introduction
● State of the art
● System Architecture
○ Feature extraction
○ Classification
○ Evaluation
● Experiments
○ Class aggregation of interest points
○ Bundling interest points
○ Class aggregation & Bundling
● Conclusions & Future work
11

13. State of the art
● In Defense of Nearest-Neighbor Based
Image Classification, Oren Boiman
12

14. State of the art
● Building contextual visual vocabulary for
large-scale image applications, S. Zhang
13

15. Contents
● Introduction
● State of the art
● System Architecture
○ Feature extraction
○ Classification
○ Evaluation
● Experiments
○ Class aggregation of interest points
○ Bundling interest points
○ Class aggregation & Bundling
● Conclusions & Future work
14

16. System Architecture
● Interest points and feature extraction
○ Sparse extraction
15

17. System Architecture
● Interest points and feature extraction
○ Interest Points: SURF
16

18. System Architecture
● Binary Partition Tree (BPT)
○ Partition: 20 reg. SLIC
17

19. ● Binary Partition Tree
○ A scale is chosen (ex, N = 3)
System Architecture
18

20. Contents
● Introduction
● State of the art
● System Architecture
○ Feature extraction
○ Classification
○ Evaluation
● Experiments
○ Class aggregation of interest points
○ Bundling interest points
○ Class aggregation & Bundling
● Conclusions & Future work
19

21. System Architecture
● Classification: Training
20
Trainer
1
2
3
4

22. System Architecture
CLASSIFIER
1-NN, euclidean
distance
1
3
4
2
● Classification: Detection
21

23. Target image
System Architecture
Query image
22

24. System Architecture
Target image
Query image
23

25. System Architecture
Query image
Target image
24

26. System Architecture
Query image
Target image
25

27. System Architecture
Query image
Target image
26

28. System Architecture
Query image
Target image
27

29. System Architecture
Query image
Nearest Target image
11
28

30. Contents
● Introduction
● State of the art
● System Architecture
○ Feature extraction
○ Classification
○ Evaluation
● Experiments
○ Class aggregation of interest points
○ Bundling interest points
○ Class aggregation & Bundling
● Conclusions & Future work
29

31. System Architecture
● Evaluation
○ Development of an evaluation tool
30

32. Tools
System Architecture
● Software development
31
Trainer
Detector
Evaluation
SVM adapted
to a flexible
architecture
New tool for
evaluation
Can be adapted
to any classifier
or descriptor

33. Contents
● Introduction
● State of the art
● System Architecture
○ Feature extraction
○ Classification
○ Evaluation
● Experiments
○ Class aggregation of interest points
○ Bundling interest points
○ Class aggregation & Bundling
● Conclusions & Future work
32

34. M-E. Nilsback & A. Zisserman, «A Visual Vocabulary for Flower Classification» Proceedings of the IEEE
Conference on Computer Vision and Pattern Recognition, 2006. http://www.robots.ox.ac.uk/~vgg/data/flowers/17/
33

35. 0.591769
0.381372
0.463660
Experiments: basic approach
● Results
34

36. Contents
● Introduction
● State of the art
● System Architecture
○ Feature extraction
○ Classification
○ Evaluation
● Experiments
○ Class aggregation of interest points
○ Bundling interest points
○ Class aggregation & Bundling
● Conclusions & Future work
35

37. Experiments: Class aggregation
● Aggregation of the interest points of all the
images of the same class to do the matching
36

38. Experiments: Class aggregation
● Results
37
0.59
0.38
0.46
0.78
0.43
0.56

39. Contents
● Introduction
● State of the art
● System Architecture
○ Feature extraction
○ Classification
○ Evaluation
● Experiments
○ Class aggregation of interest points
○ Bundling interest points
○ Class aggregation & Bundling
● Conclusions & Future work
38

40. ● Region restriction
Experiments: Bundling interest points
39

41. Experiments: Bundling interest points
40

42. ● Why the results did not improve?
○ Image flower segmentation
Experiments: Bundling interest points
41

43. ● Why the results did not improve?
○ Bad flower segmentation (N = 2)
Experiments: Bundling interest points
42

44. ● Why the results did not improve?
○ Bad flower segmentation (N = 2)
● Future work to improve results
○ Using perfect manual segmentation
Experiments: Bundling interest points
43

45. ● Why the results did not improve?
○ Good region matching (flower to flower)
Experiments: Bundling interest points
44

46. ● Why the results did not improve?
○ Bad region matching (flower to background)
Experiments: Bundling interest points
45

47. ● Why the results did not improve?
○ Bad region matching (flower to background)
● Future work to improve results
○ Avoid using edge regions
○ Using object candidates
Experiments: Bundling interest points
46

48. Contents
● Introduction
● State of the art
● System Architecture
○ Feature extraction
○ Classification
○ Evaluation
● Experiments
○ Class aggregation of interest points
○ Bundling interest points
○ Class aggregation & Bundling
● Conclusions & Future work
47

49. Experiments: Class aggregation & Bundling
● Class aggregation with points bundled in regions
48

50. ● Comparative study
Experiments: Class aggregation & Bundling
49

51. Contents
● Introduction
● State of the art
● System Architecture
○ Feature extraction
○ Classification
○ Evaluation
● Experiments
○ Class aggregation of interest points
○ Bundling interest points
○ Class aggregation & Bundling
● Conclusions & Future work
50

52. Conclusions & Future Work
● Comparative study done
○ Bundling interest points into regions worsens the
F1-score between 1% and 7%
○ Class aggregation improves the F1-score by 9.2%
● State of the art comparative study
○ Pointless having bad results
● Software development
● Future Work
51

53. Bundling interest points
for object classification
Jordi Sánchez Escué
Supervised by
Xavier Giró i Nieto
Carles Ventura Royo

55. System Architecture
● Classification: Training
○ Semantic annotation & Ontology
...
...
...
...
...
...
1
2
3
4

56. System Architecture
● Binary Partition Tree (BPT)
○ 20 SLIC superpixels

57. Future work
● Add new approaches
○ Class aggregation in the query
○ Bundling query image, not bundling target
images (with certain spatial restriction).
● Optimize k, change classifier, more descriptors