Ce diaporama a bien été signalé.
Nous utilisons votre profil LinkedIn et vos données d’activité pour vous proposer des publicités personnalisées et pertinentes. Vous pouvez changer vos préférences de publicités à tout moment.

Introducing context-dependent and spatially-variant viewing biases in saccadic models

820 vues

Publié le

O. Le Meur and A. Coutrot, Introducing context-dependent and spatially-variant viewing biases in saccadic models, Accepted for publication in Vision Research, 2016.

Previous research showed the existence of systematic tendencies in viewing behavior during scene exploration. For instance, saccades are known to follow a positively skewed, long-tailed distribution, and to be more frequently initiated in the horizontal or vertical directions. In this study, we hypothesize that these viewing biases are not universal, but are modulated by the semantic visual category of the stimulus. We show that the joint distribution of saccade amplitudes and orientations significantly varies from one visual category to another. These joint distributions are in addition spatially variant within the scene frame. We demonstrate that a saliency model based on this better understanding of viewing behavioral biases and blind to any visual information outperforms well-established saliency models. We also propose a saccadic model that takes into account classical low-level features and spatially-variant and context-dependent viewing biases.

Publié dans : Sciences
  • Soyez le premier à commenter

  • Soyez le premier à aimer ceci

Introducing context-dependent and spatially-variant viewing biases in saccadic models

  1. 1. Visual attention O. Le Meur Introduction Le Meur & Liu’s saccadic model Description Performance Le Meur & Coutrot’s improvements Intuitions Are viewing biases context-dependent? Are viewing biases spatially-variant? New model Conclusion Introducing context-dependent and spatially-variant viewing biases in saccadic models Olivier Le Meur 1 Antoine Coutrot 2 olemeur@irisa.fr 1 IRISA - University of Rennes 1, France 2 CoMPLEX, University College London, UK January 29, 2016 1 / 30
  2. 2. Visual attention O. Le Meur Introduction Le Meur & Liu’s saccadic model Description Performance Le Meur & Coutrot’s improvements Intuitions Are viewing biases context-dependent? Are viewing biases spatially-variant? New model Conclusion Preamble These slides are based on the following papers: O. Le Meur & Z. Liu, Saccadic model of eye movements for free-viewing condition, Vision Research, 2015, doi:10.1016/j.visres.2014.12.026. O. Le Meur & A. Coutrot, Introducing context-dependent and spatially-variant viewing biases in saccadic models, Vision Research, 2016. 2 / 30
  3. 3. Visual attention O. Le Meur Introduction Le Meur & Liu’s saccadic model Description Performance Le Meur & Coutrot’s improvements Intuitions Are viewing biases context-dependent? Are viewing biases spatially-variant? New model Conclusion Outline 1 Introduction 2 Le Meur & Liu’s saccadic model 3 Le Meur & Coutrot’s improvements 4 Conclusion 3 / 30
  4. 4. Visual attention O. Le Meur Introduction Le Meur & Liu’s saccadic model Description Performance Le Meur & Coutrot’s improvements Intuitions Are viewing biases context-dependent? Are viewing biases spatially-variant? New model Conclusion Introduction Computational models of visual attention aim at predicting where we look within a scene. Bottom-Up models of overt attention compute a 2D static saliency map from an input image. Performances significantly increases during the past decades but they suffers from severe limitations. One solution to deal with these limitations is saccadic models. This presentation aims to describe a new saccadic model. From an input image, this kind of model outputs: plausible visual scanpaths, from which scanpath-based saliency maps can be computed. 4 / 30
  5. 5. Visual attention O. Le Meur Introduction Le Meur & Liu’s saccadic model Description Performance Le Meur & Coutrot’s improvements Intuitions Are viewing biases context-dependent? Are viewing biases spatially-variant? New model Conclusion Le Meur & Liu’s saccadic model 2 Le Meur & Liu’s saccadic model Description Performance 5 / 30
  6. 6. Visual attention O. Le Meur Introduction Le Meur & Liu’s saccadic model Description Performance Le Meur & Coutrot’s improvements Intuitions Are viewing biases context-dependent? Are viewing biases spatially-variant? New model Conclusion Le Meur & Liu’s saccadic model (1/6) O. Le Meur & Z. Liu, Saccadic model of eye movements for free-viewing condition, Vision Research, 2015. ª The model is stochastic: the subsequent fixation cannot be completely specified (given a set of data). ª The model generates plausible scanpaths i.e. scanpaths that are similar to those generated by humans in similar conditions: distribution of saccade amplitudes and orientations, center bias... ª The model predicts well the salient areas, i.e. predicted fixations are mainly located on salient areas. 6 / 30
  7. 7. Visual attention O. Le Meur Introduction Le Meur & Liu’s saccadic model Description Performance Le Meur & Coutrot’s improvements Intuitions Are viewing biases context-dependent? Are viewing biases spatially-variant? New model Conclusion Le Meur & Liu’s saccadic model (2/6) Three main components Let I : Ω ⊂ R2 → R3 an image and xt a fixation point at time t. We consider the 2D discrete conditional probability: p (x|xt−1, . . . , xt−T ) ∝ pBU (x)pB(d, φ)pM (x, t) ª pBU : Ω → [0, 1] is the grayscale saliency map; ª pB(d, φ) represents the joint probability distribution of saccade amplitudes and orientations. d is the saccade amplitude between two fixation points xt and xt−1 (expressed in degree of visual angle), and φ is the angle (expressed in degree between these two points); ª pM (x, t) represents the memory state of the location x at time t. This time-dependent term simulates the inhibition of return. 7 / 30
  8. 8. Visual attention O. Le Meur Introduction Le Meur & Liu’s saccadic model Description Performance Le Meur & Coutrot’s improvements Intuitions Are viewing biases context-dependent? Are viewing biases spatially-variant? New model Conclusion Le Meur & Liu’s saccadic model (3/6) Bottom-up saliency map p (x|xt−1, . . . , xt−T ) ∝ pBU (x)pB(d, φ)pM (x, t) ª pBU is the bottom-up saliency map. • Computed by GBVS model (Harel et al., 2006). According to (Borji et al., 2012)’s benchmark, this model is among the best ones and presents a good trade-off between quality and complexity. • pBU (x) is constant over time. (Tatler et al., 2005) indeed demonstrated that bottom-up influences do not vanish over time. 8 / 30
  9. 9. Visual attention O. Le Meur Introduction Le Meur & Liu’s saccadic model Description Performance Le Meur & Coutrot’s improvements Intuitions Are viewing biases context-dependent? Are viewing biases spatially-variant? New model Conclusion Le Meur & Liu’s saccadic model (4/6) Viewing biases p (x|xt−1, . . . , xt−T ) ∝ pBU (x)pB(d, φ)pM (x, t) ª pB(d, φ) represents the joint probability distribution of saccade amplitudes and orientations. d and φ represent the distance and the angle between each pair of successive fixations, respectively. From left to right: distributions estimated from Le Meur, Bruce, Kootstra and Judd’s fixation datasets. 9 / 30
  10. 10. Visual attention O. Le Meur Introduction Le Meur & Liu’s saccadic model Description Performance Le Meur & Coutrot’s improvements Intuitions Are viewing biases context-dependent? Are viewing biases spatially-variant? New model Conclusion Le Meur & Liu’s saccadic model (5/6) Memory effect and inhibition of return (IoR) p (x|xt−1, . . . , xt−T ) ∝ pBU (x)pB(d, φ)pM (x, t) ª pM (x, t) represents the memory effect and IoR of the location x at time t. It is composed of two terms: Inhibition and Recovery. • The spatial IoR effect declines as a Gaussian function Φσi (d) with the Euclidean distance d from the attended location (Bennett and Pratt, 2001); • The temporal decline of the IoR effect is simulated by a simple linear model. 10 / 30
  11. 11. Visual attention O. Le Meur Introduction Le Meur & Liu’s saccadic model Description Performance Le Meur & Coutrot’s improvements Intuitions Are viewing biases context-dependent? Are viewing biases spatially-variant? New model Conclusion Le Meur & Liu’s saccadic model (6/6) Selecting the next fixation point p (x|xt−1, . . . , xt−T ) ∝ pBU (x)pB(d, φ)pM (x, t) ª Optimal next fixation point (Bayesian ideal searcher proposed by (Najemnik and Geisler, 2009)): x∗ t = arg max x∈Ω p (x|xt−1, · · · , xt−T ) (1) Problem: this approach does not reflect the stochastic behavior of our visual system and may fail to provide plausible scanpaths (Najemnik and Geisler, 2008). ª Rather than selecting the best candidate, we generate Nc = 5 random locations according to the 2D discrete conditional probability p (x|xt−1, · · · , xt−T ). The location with the highest saliency gain is chosen as the next fixation point x∗ t . 11 / 30
  12. 12. Visual attention O. Le Meur Introduction Le Meur & Liu’s saccadic model Description Performance Le Meur & Coutrot’s improvements Intuitions Are viewing biases context-dependent? Are viewing biases spatially-variant? New model Conclusion Performance (1/3) Are the simulated scanpaths plausible? Top row: Bruce’s dataset. Bottom row: Judd’s dataset. 12 / 30
  13. 13. Visual attention O. Le Meur Introduction Le Meur & Liu’s saccadic model Description Performance Le Meur & Coutrot’s improvements Intuitions Are viewing biases context-dependent? Are viewing biases spatially-variant? New model Conclusion Performance (2/3) Scanpath-based saliency map ª We compute, for each image, 20 scanpaths, each composed of 10 fixations. ª For each image, we created a saliency map by convolving a Gaussian function over the fixation locations. (a) original image; (b) human saliency map; (c) GBVS saliency map; (d) GBVS-SM saliency maps computed from the simulated scanpaths. 13 / 30
  14. 14. Visual attention O. Le Meur Introduction Le Meur & Liu’s saccadic model Description Performance Le Meur & Coutrot’s improvements Intuitions Are viewing biases context-dependent? Are viewing biases spatially-variant? New model Conclusion Performance (3/3) Scanpath-based saliency map 14 / 30
  15. 15. Visual attention O. Le Meur Introduction Le Meur & Liu’s saccadic model Description Performance Le Meur & Coutrot’s improvements Intuitions Are viewing biases context-dependent? Are viewing biases spatially-variant? New model Conclusion Le Meur & Coutrot’s improvements 3 Le Meur & Coutrot’s improvements Intuitions Are viewing biases context-dependent? Are viewing biases spatially-variant? New model 15 / 30
  16. 16. Visual attention O. Le Meur Introduction Le Meur & Liu’s saccadic model Description Performance Le Meur & Coutrot’s improvements Intuitions Are viewing biases context-dependent? Are viewing biases spatially-variant? New model Conclusion Le Meur & Coutrot’s improvements (1/11) O. Le Meur & A. Coutrot, Introducing context-dependent and spatially-variant viewing biases in saccadic models, Vision Research, 2016. What are the main intuitions.... ª Viewing biases are not universal but depend on the semantic visual category of the stimulus... ª Viewing biases are likely spatially-variant... Are viewing biases spatially-variant and context-dependent? 16 / 30
  17. 17. Visual attention O. Le Meur Introduction Le Meur & Liu’s saccadic model Description Performance Le Meur & Coutrot’s improvements Intuitions Are viewing biases context-dependent? Are viewing biases spatially-variant? New model Conclusion Le Meur & Coutrot’s improvements (2/11) Are viewing biases context-dependent? (1/2) The joint distribution of saccade amplitudes and orientations is computed for different categories of visual scenes: ª Natural scenes, webpages & Conservational videos Strong horizontal bias. Horizontal bias but mainly in the rightward direction. Three modes in the distribution. 17 / 30
  18. 18. Visual attention O. Le Meur Introduction Le Meur & Liu’s saccadic model Description Performance Le Meur & Coutrot’s improvements Intuitions Are viewing biases context-dependent? Are viewing biases spatially-variant? New model Conclusion Le Meur & Coutrot’s improvements (3/11) Are viewing biases context-dependent? (2/2) From observed distributions: ª we draw 5000 samples; ª we test whether both data sets are drawn from the same distribution (two-sample two-dimensional Kolmogorov-Smirnov (Peacock, 1983)). For all conditions, the difference is significant, i.e. p << 0.001. Visual strategy is influenced by the scene content. 18 / 30
  19. 19. Visual attention O. Le Meur Introduction Le Meur & Liu’s saccadic model Description Performance Le Meur & Coutrot’s improvements Intuitions Are viewing biases context-dependent? Are viewing biases spatially-variant? New model Conclusion Le Meur & Coutrot’s improvements (4/11) Are viewing biases spatially-variant? (1/3) Rather than computing a unique joint distribution per image, we evenly divide the image into a N × N equal base frames. N = 3 19 / 30
  20. 20. Visual attention O. Le Meur Introduction Le Meur & Liu’s saccadic model Description Performance Le Meur & Coutrot’s improvements Intuitions Are viewing biases context-dependent? Are viewing biases spatially-variant? New model Conclusion Le Meur & Coutrot’s improvements (5/11) Are viewing biases spatially-variant? (2/3) Estimation of the joint distribution pB(d, φ|F, S), given the frame index F (F ∈ {1, ..., 9}) and the scene category S (Natural scenes, webpages, conversational...): Dynamic landscape. Natural scenes. ª Re-positioning saccades allowing us to go back to the screen’s center. Interesting to reproduce the center bias! 20 / 30
  21. 21. Visual attention O. Le Meur Introduction Le Meur & Liu’s saccadic model Description Performance Le Meur & Coutrot’s improvements Intuitions Are viewing biases context-dependent? Are viewing biases spatially-variant? New model Conclusion Le Meur & Coutrot’s improvements (6/11) Are viewing biases spatially-variant? (3/3) Estimation of the joint distribution pB(d, φ|F, S), given the frame index F (F ∈ {1, ..., 9}) and the scene category S (Natural scenes, webpages, conversational...): Conversational videos. Webpages. ª Specific distributions for these visual contents. 21 / 30
  22. 22. Visual attention O. Le Meur Introduction Le Meur & Liu’s saccadic model Description Performance Le Meur & Coutrot’s improvements Intuitions Are viewing biases context-dependent? Are viewing biases spatially-variant? New model Conclusion Le Meur & Coutrot’s improvements (7/11) The joint distribution of saccade amplitudes and orientations is spatially-variant and context-dependent. 22 / 30
  23. 23. Visual attention O. Le Meur Introduction Le Meur & Liu’s saccadic model Description Performance Le Meur & Coutrot’s improvements Intuitions Are viewing biases context-dependent? Are viewing biases spatially-variant? New model Conclusion Le Meur & Coutrot’s improvements (8/11) Improving Le Meur & Liu’s model Le Meur & Liu’s model p (x|xt−1, . . . , xt−T ) ∝ pBU (x)pB(d, φ)pM (x, t) ⇓ Le Meur & Coutrot’s model p (x|xt−1, . . . , xt−T , S) ∝ pBU (x)pB(d, φ|F, S)pM (x, t) 23 / 30
  24. 24. Visual attention O. Le Meur Introduction Le Meur & Liu’s saccadic model Description Performance Le Meur & Coutrot’s improvements Intuitions Are viewing biases context-dependent? Are viewing biases spatially-variant? New model Conclusion Le Meur & Coutrot’s improvements (9/11) Effectiveness of viewing biases to predict where we look at (1/1) Can viewing biases predict where we look at, when we are blind to image information? p (x|xt−1, . . . , xt−T ) ∝ XXXXpBU (x)pB(d, φ|F, S)pM (x, t) 24 / 30
  25. 25. Visual attention O. Le Meur Introduction Le Meur Liu’s saccadic model Description Performance Le Meur Coutrot’s improvements Intuitions Are viewing biases context-dependent? Are viewing biases spatially-variant? New model Conclusion Le Meur Coutrot’s improvements (10/11) Joint distribution of saccade amplitudes and orientations of predicted scanpaths (1/1) Are the predicted scanpaths similar to human ones? p (x|xt−1, . . . , xt−T ) ∝ pBU (x)pB(d, φ|F, S)pM (x, t) Yes, predicted scanpaths show similar patterns as the human scanpaths! 25 / 30
  26. 26. Visual attention O. Le Meur Introduction Le Meur Liu’s saccadic model Description Performance Le Meur Coutrot’s improvements Intuitions Are viewing biases context-dependent? Are viewing biases spatially-variant? New model Conclusion Le Meur Coutrot’s improvements (11/11) Bottom-up salience and viewing biases for predicting visual scanpaths (1/1) Mixing together bottom-up saliency and viewing biases. (i) When the quality of the input saliency map increases, performance of saccadic model increases; (ii) The gain brought by spatially-variant and context-dependent distributions is not significant; (iii) Spatially-variant and context-dependent distributions are required to generate plausible visual scanpaths (see previous slides). 26 / 30
  27. 27. Visual attention O. Le Meur Introduction Le Meur Liu’s saccadic model Description Performance Le Meur Coutrot’s improvements Intuitions Are viewing biases context-dependent? Are viewing biases spatially-variant? New model Conclusion Conclusion 4 Conclusion 27 / 30
  28. 28. Visual attention O. Le Meur Introduction Le Meur Liu’s saccadic model Description Performance Le Meur Coutrot’s improvements Intuitions Are viewing biases context-dependent? Are viewing biases spatially-variant? New model Conclusion Conclusion (1/2) Contributions: ª A new saccadic model performing well to: • produce plausible visual scanpaths; • detect the most salient regions of visual scenes. ª Signature of viewing tendencies. This signature is spatially-variant and context-dependent; Extension of current model to predict salient areas when observers perform a task: O. Le Meur A. Coutrot, How saccadic models help predict where we look during a visual task? Application to visual quality assessment, SPIE Electronic Imaging, Image Quality and System Performance XIII, 2016. 28 / 30
  29. 29. Visual attention O. Le Meur Introduction Le Meur Liu’s saccadic model Description Performance Le Meur Coutrot’s improvements Intuitions Are viewing biases context-dependent? Are viewing biases spatially-variant? New model Conclusion Conclusion (2/2) Future works: ª Dealing with the limitations of the current implementation (fixation durations, duration of IoR...); ª Spatio-temporal signature of viewing tendencies. See the paper for a thorough analysis. 29 / 30
  30. 30. Visual attention O. Le Meur Introduction Le Meur Liu’s saccadic model Description Performance Le Meur Coutrot’s improvements Intuitions Are viewing biases context-dependent? Are viewing biases spatially-variant? New model Conclusion Thanks! 30 / 30
  31. 31. Visual attention O. Le Meur References References P. J. Bennett and J. Pratt. The spatial distribution of inhibition of return:. Psychological Science, 12:76–80, 2001. A. Borji, D. N. Sihite, and L. Itti. Quantitative analysis of human-model agreement in visual saliency modeling: A comparative study. IEEE Transactions on Image Processing, 22(1):55–69, 2012. J. Harel, C. Koch, and P. Perona. Graph-based visual saliency. In Proceedings of Neural Information Processing Systems (NIPS), 2006. J. Najemnik and W.S. Geisler. Eye movement statistics in humans are consistent with an optimal strategy. Journal of Vision, 8(3): 1–14, 2008. J. Najemnik and W.S. Geisler. Simple summation rule for optimal fixation selection in visual search. Vision Research, 42: 1286–1294, 2009. JA Peacock. Two-dimensional goodness-of-fit testing in astronomy. Monthly Notices of the Royal Astronomical Society, 202(3): 615–627, 1983. B.W. Tatler, R. J. Baddeley, and I.D. Gilchrist. Visual correlates of fixation selection: effects of scale and time. Vision Research, 45:643–659, 2005. 30 / 30

×