Description of the main features of a scene which are extracted by most computational models. Comparison of two different computational models.

1. Modeling the Shape of the
Scene: A Holistic
Representation of the Spatial
Envelope
Oliva. A., Torralba A. (2001).
International Journal of Computer
Vision 42(3), 145-175.

2. Scene Recognition
- What is a Scene?
- S.R. Studies
Spatial Envelope
- Holistic
Representation
Scene Recognition
- Definition
- Spatial Properties
Scene Structure 2 different approaches:
- DFT
- WFT  Details
 General Picture
- Mean
Spectrograms  General Picture  Details
Sp.E. Properties
- DST
- WDST
- Naturalness  Scene Recognition is…
- Natural Scenes’
Properties a reconstruction of the properties of a 3D
- Urban Scenes’
Properties Scene that are important for tasks such as
- Correlations
- Semantic
movement or grasping
Categories  the providing of information about the
Conclusions
semantic category and the function of the
environment.

3. Scene Recognition
- What is a Scene?
- S.R. Studies
Spatial Envelope
- Holistic
Representation
Scene Recognition
- Definition
- Spatial Properties
Scene Structure  Object  Scene
- DFT
- WFT 1 – 2 meters  >5meters
- Mean
Spectrograms
 “Hand distance”  “Movement”
Sp.E. Properties
- DST
- WDST
- Naturalness
- Natural Scenes’
Properties
- Urban Scenes’
Properties
- Correlations
- Semantic
Categories
Conclusions

4. Scene Recognition
- What is a Scene?
- S.R. Studies
Spatial Envelope
- Holistic
Representation
Scene Recognition
- Definition
- Spatial Properties
Scene Structure  “Gist”
Recognition Speed: 200ms
- DFT
- WFT
- Mean  View of a Scene for short time
Spectrograms
Sp.E. Properties
- DST
 Recognition of Functional and
- WDST
- Naturalness Categorical Properties
- Natural Scenes’
Properties  Object information – ignored
- Urban Scenes’
Properties
- Correlations  Environmental pictures – rapid
- Semantic
Categories
Conclusions categorization

5. Scene Recognition
- What is a Scene?
- S.R. Studies
Spatial Envelope
- Holistic
Representation
Spatial Envelope
- Definition
- Spatial Properties
Scene Structure Scene is not just an arrangement of objects and
- DFT
- WFT
it’s something more than the relationship
- Mean between them.
Spectrograms
Sp.E. Properties
- DST
- WDST
- Naturalness
- Natural Scenes’
Properties
- Urban Scenes’
Properties
- Correlations
- Semantic
Categories
Conclusions
Scenes of the same category have a
stable spatial structure.

6. Scene Recognition
- What is a Scene?
- S.R. Studies
Spatial Envelope
- Holistic
Representation
Spatial Envelope
- Definition
- Spatial Properties
Scene Structure Spatial Envelope: boundaries that give a shape to the Scene.
- DFT
- WFT
e.g. ground, horizon, elevations, slants
- Mean
Spectrograms
Sp.E. Properties
- DST
- WDST
- Naturalness
- Natural Scenes’
Properties
- Urban Scenes’
Properties
- Correlations
- Semantic
Categories
Conclusions

7. Scene Recognition
- What is a Scene?
- S.R. Studies
Spatial Envelope
- Holistic
Representation
Spatial Envelope
- Definition
- Spatial Properties
Scene Structure
81 pictures
- DFT
- WFT Experiment A:
- Mean
Spectrograms 17 subjects - 81 pictures
Sp.E. Properties Criteria: A B
- DST
1)Global Aspect
- WDST
- Naturalness 2)Global Structure
- Natural Scenes’ 3)Elements A1 A2 B1 B2
Properties
- Urban Scenes’
Properties
a b a b a b a b
- Correlations
- Semantic
Categories 8 subgroups
Conclusions

8. Scene Recognition
- What is a Scene?
- S.R. Studies
Spatial Envelope
- Holistic
Representation
Spatial Envelope
- Definition 5 Main Properties
- Spatial Properties
Scene Structure
1. Naturalness
- DFT sharp VS smooth edges
- WFT
- Mean
Spectrograms 1. Openness
Sp.E. Properties
- DST No Horizon / Visual references
- WDST VS Horizon / lack of Visual
- Naturalness References
- Natural Scenes’
Properties
- Urban Scenes’ 1. Roughness
Properties
- Correlations Complexity VS No complexity
- Semantic
Categories
Conclusions 1. Expansion
Depth VS No depth
1. Ruggedness
Deviation VS No deviation

9. Scene Recognition
- What is a Scene?
- S.R. Studies
Spatial Envelope Modeling the Scene Structure
Image-based representations
- Holistic
Representation
- Definition
Hanning
- Spatial Properties Discrete Fourier Transform Window
Scene Structure Intensity
- DFT
- WFT
- Mean
Spectrograms
Sp.E. Properties
- DST
- WDST Amplitude Phase
- Naturalness Spectrum Function
- Natural Scenes’
Properties
- Urban Scenes’
Properties Windowed Fourier Transform
- Correlations
- Semantic
Categories
Hamming
Conclusions Intensity window with
radius r

10. Scene Recognition
- What is a Scene?
- S.R. Studies
Spatial Envelope
- Holistic
Representation
Modeling the Scene Structure
- Definition Energy Spectrum
- Spatial Properties
Scene Structure
- DFT
- WFT
- Mean
Spectrograms
Sp.E. Properties
- DST
- WDST
- Naturalness Energy Spectrum Principal Components of Energy Spectrum
- Natural Scenes’
Properties
- Urban Scenes’
Spectrogram
Properties
- Correlations
- Semantic
Categories
Conclusions
Spectrogram Principal Components of the Spectrogram

11. Scene Recognition
- What is a Scene?
- S.R. Studies
Spatial Envelope
Modeling the Scene Structure
- Holistic
Representation
- Definition
- Spatial Properties
Scene Structure
- DFT
- WFT
- Mean
Spectrograms
Sp.E. Properties Man-made open urban | Streets City Center Buildings close-up
- DST Urban Str.
- WDST
- Naturalness
- Natural Scenes’
Properties
- Urban Scenes’
Properties
- Correlations
- Semantic
Categories
Natural open natural enclosed mountains enclosed forest natural,
Conclusions
non-textured

12. Scene Recognition
- What is a Scene?
- S.R. Studies
Spatial Envelope
Estimation of the Spatial Envelope
- Holistic
Representation Properties
- Definition
- Spatial Properties Scene
Scene Structure attribute s
- DFT
- WFT
- Mean
Spectrograms
Sp.E. Properties
- DST
- WDST
- Naturalness
- Natural Scenes’
Properties
- Urban Scenes’
Properties
- Correlations
- Semantic
Categories
Conclusions
Discriminant Spectral Template - DST

13. Scene Recognition
- What is a Scene?
- S.R. Studies
Spatial Envelope
Estimation of the Spatial Envelope
- Holistic
Representation Properties
- Definition
- Spatial Properties
Scene Structure
- DFT
- WFT
- Mean
Spectrograms
Sp.E. Properties
- DST
- WDST
- Naturalness
- Natural Scenes’
Properties
- Urban Scenes’
Properties
- Correlations
- Semantic
Categories
Conclusions
Windowed Discriminant Spectral Template -
WDST

14. Scene Recognition
- What is a Scene?
- S.R. Studies
Spatial Envelope
Spatial Envelope Properties
- Holistic
Representation ~Naturalness~
- Definition
- Spatial Properties
Scene Structure 2000 images of natural and man-made scenes
- DFT
- WFT
- Mean
Spectrograms
Sp.E. Properties
- DST
- WDST
- Naturalness
DST(fx,fy) WDST(fx,fy)
- Natural Scenes’
Properties
- Urban Scenes’
93.5% Correct!
Properties
- Correlations
- Semantic
Categories
Conclusions
Low Naturalness High Naturalness

15. Scene Recognition
- What is a Scene?
- S.R. Studies
Spatial Envelope
Spatial Envelope Properties
- Holistic
Representation ~Natural Images~
- Definition
- Spatial Properties 500 images of natural scenes
Scene Structure Openness Ruggedness Roughness
- DFT
- WFT
- Mean
Spectrograms
Sp.E. Properties DST(fx,fy)
- DST
- WDST
- Naturalness
- Natural Scenes’
Properties
- Urban Scenes’
Properties
- Correlations WDST(fx,fy)
- Semantic
Categories
Conclusions
DST: 93% DST: 89% DST: 92%
WDST: 96,5% WDST: 91% WDST: 94%

16. Scene Recognition
- What is a Scene?
- S.R. Studies
Spatial Envelope
Spatial Envelope Properties
- Holistic
Representation ~Natural Images~
- Definition
- Spatial Properties
Scene Structure
- DFT
- WFT
- Mean
Spectrograms
Sp.E. Properties
- DST
- WDST
- Naturalness
- Natural Scenes’
Properties
- Urban Scenes’
Properties
- Correlations
- Semantic
Categories
Conclusions

17. Scene Recognition
- What is a Scene?
- S.R. Studies
Spatial Envelope
Spatial Envelope Properties
- Holistic
Representation ~Urban Environments~
- Definition
- Spatial Properties 500 urban scenes
Scene Structure
Openness Expansion Roughness
- DFT
- WFT
- Mean
Spectrograms
Sp.E. Properties DST(fx,fy)
- DST
- WDST
- Naturalness
- Natural Scenes’
Properties
- Urban Scenes’
Properties
- Correlations WDST(fx,fy)
- Semantic
Categories
Conclusions
DST: 94% DST: 90% DST: 91%
WDST: 96% WDST: 94% WDST: 92%%

18. Scene Recognition
- What is a Scene?
- S.R. Studies
Spatial Envelope
Spatial Envelope Properties
- Holistic
Representation ~Urban Environments~
- Definition
- Spatial Properties
Scene Structure
- DFT
- WFT
- Mean
Spectrograms
Sp.E. Properties
- DST
- WDST
- Naturalness
- Natural Scenes’
Properties
- Urban Scenes’
Properties
- Correlations
- Semantic
Categories
Conclusions

19. Scene Recognition
- What is a Scene?
- S.R. Studies
Spatial Envelope
- Holistic Spatial Envelope Properties
Representation
- Definition
- Spatial Properties
Natural Scenes
Scene Structure
- DFT
- WFT
- Mean
Spectrograms
Sp.E. Properties
- DST
- WDST
- Naturalness
- Natural Scenes’
Properties
- Urban Scenes’
Properties Urban Scenes
- Correlations
- Semantic
Categories
Conclusions

20. Scene Recognition
- What is a Scene?
- S.R. Studies
Spatial Envelope
- Holistic Spatial Envelope Properties
Representation
- Definition
- Spatial Properties
Scene Structure
- DFT
- WFT
- Mean
Spectrograms
Sp.E. Properties
- DST
- WDST
- Naturalness
- Natural Scenes’
Properties
- Urban Scenes’
Properties
- Correlations
- Semantic
Categories
Conclusions

21. Scene Recognition
- What is a Scene?
- S.R. Studies
Spatial Envelope
- Holistic Spatial Envelope Properties
Representation
- Definition
- Spatial Properties
Scene Structure
- DFT
- WFT
- Mean
Spectrograms
Sp.E. Properties
- DST
- WDST
- Naturalness
- Natural Scenes’
Properties
- Urban Scenes’
Properties
- Correlations 400 “target” images
- Semantic
Categories XXX + XXXX
Conclusions
XXXOXOO + OOXOXOX
XX+XX
DST: 86% WDST: 92%

22. Scene Recognition
- What is a Scene?
- S.R. Studies
Spatial Envelope
- Holistic Conclusions / Important Points
Representation
- Definition
- Spatial Properties
Scene Structure
•Introduction of the “Spatial Envelope” model
- DFT •A set of important perceptual properties (Naturalness, openness,
- WFT ruggedness, expansion, roughness)
- Mean
Spectrograms •Provides a holistic description of the scene
Sp.E. Properties
- DST
•Spatial Envelope properties are strongly correlated with
- WDST the templates of DST and WDST
- Naturalness
- Natural Scenes’
•The S.E. model organizes scenes as humans subjects do
Properties
- Urban Scenes’ •S.E. model is able to retrieve images sharing the same
Properties semantic category
- Correlations
- Semantic •Object identification is not a necessary stage for
Categories
identification a scene
Conclusions
•S.E. model provides a meaningful representation of
complex scenes