DevEX - reference for building teams, processes, and platforms
Visual Saliency: Learning to Detect Salient Objects
1. Visual Attention: Detecting Saliency on Images Vicente Ordonez Department of Computer Science State University of New York Stony Brook, NY 11790
2. I will be working mainly on the following paper Learning to Detect a Salient Object. T. Liu, J. Sun, N. Zheng, X. Tang, H. Shum. (Xian Jiaotong University and Microsoft Research Asia) from CVPR 2007. http://research.microsoft.com/en-us/um/people/jiansun/papers/SalientDetection_CVPR07.pdf
3. What is Saliency? What is Visual Attention? “Everyone knows what attention is...” —William James, 1890
4. This is a problem of… Arbitrary object detection? Background / Foreground segmentation? Modeling Visual Attention?
5. The Method Features: Multiscale Contrast (Done!) Center surround histogram (Mostly Done!) (Done!) Color spatial distribution (Done!) Supervised learning using Conditional Random Fields to determine the parameters to combine the features obtained above. (Done!) [I will use a labeled dataset of 5000 images provided by Microsoft Research Asia!]
6. Multiscale Contrast Function Generate the Gaussian Pyramid for the input image. For each level in the pyramid Do gaussian blurring Do resampling I’m using a 6 levels Gaussian pyramid for each RGB channel.
7. How a Gaussian pyramid looks like Figure from David Forsyth
8. Generate contrast maps for each level of the Pyramid. Sum all of the results to produce the final multiscale contrast map. The two steps mentioned above are described in this formula: Multiscale Contrast Function
15. Create a surrounding rectangle and calculate the histogram of the rectangle and the surrounding area.
16.
17. Center Surround Histogram Feature The algorithm as described before is computationally expensive… It is required to use a technique called Integral Histogram. It allows you fast calculation of the histogram of any given rectangular region of an image. The algorithm was introduced in: “Integral Histogram: A Fast Way to Extract Histograms in Cartesian Spaces” by FatihPorikli, Mitsubishi Electric Research Lab in CVPR 2005.
18. Center Surround Histogram Feature Use the Chi Square Distances Map and the Map of Most Salient Rectangle Regions per pixel to generate the Center Surround Histogram Feature using the next formula:
19. Center Surround Histogram Results Using my Implementation (15.2 sec, size = 245x384) Results Reported in the Paper
20. Center Surround Histogram Results Using my Implementation (13.6 sec, size = 247x346) Results Reported in the Paper
34. Color Spatial Distribution Make an initial clustering of the colors in the image using k-means. Further refine the clusters by using Gaussian Mixture Models. The Gaussian Mixture Model parameters are calculated using the EM algorithm. I am using 5 clusters (5 colors) per image. And the results look similar to those presented in the paper with an execution time of around 17 seconds per image.
35. Color Spatial Distribution Calculate the vertical variance of the horizontal positions of the pixels for each cluster. And then the same for the vertical positions. Sum the variances and use this value to weight more those clusters with less spatial variance. Penalize the clusters that contain the majority of its pixels away from the center of the image.
45. Conditional Random Field Training and Inference Accelerated Training of Conditional Random Fields with Stochastic Meta-Descent S Vishwanathan, N. Schraudolph, M. Schmidt, K. Murphy. ICML'06 (Intl Conf on Machine Learning). I did the training using this toolbox from the above paper: http://people.cs.ubc.ca/~murphyk/Software/CRF/crf.html
46. Mask outputs using CRF inference Input M-Contrast-map Center Surr. Hist. Color Spatial Var. Input Combined features Ground truth
47. Mask outputs using CRF inference Input M-Contrast-map Center Surr. Hist. Color Spatial Var. Input Combined features Ground truth
48. Mask outputs using CRF inference Input M-Contrast-map Center Surr. Hist. Color Spatial Var. Input Combined features Ground truth
49. Mask outputs using CRF inference Input M-Contrast-map Center Surr. Hist. Color Spatial Var. Input Combined features Ground truth
51. Some Conclusions The results of the original research paper on computing the visual features have been successfully replicated in a considerable extent. The Conditional Random Field framework used in this project turned out to perform well for this task. The center-surround histogram map turned out to be the feature that gave the higher precision. The amount of time required for computing the individual features is in the order of several seconds.