Roman Fedorov seminar "Applying Computer Vision Techniques to Web Public Images for the Environmental Monitoring Purposes" on Nov 11, 2015 at Indiana University Bloomington, USA.
CHEAP Call Girls in Pushp Vihar (-DELHI )🔝 9953056974🔝(=)/CALL GIRLS SERVICE
Applying Computer Vision Techniques to Web Public Images for the Environmental Monitoring Purposes
1. DIPARTIMENTO DI ELETTRONICA, INFORMAZIONE E BIOINGEGNERIA
SnowWatch
Applying Computer Vision Techniques to Public
Web Images for Environmental Monitoring
Roman Fedorov: roman.fedorov@polimi.it
2. DIPARTIMENTO DI ELETTRONICA, INFORMAZIONE E BIOINGEGNERIA
UGC
2
Media User Generated Content (UGC) publicly
available on the Web is reaching unprecedented size
nowadays:
• Facebook ~ 350 MM daily photo uploads
• Flickr ~ 8.000 MM photographs with 3.5 MM daily
uploads
• Panoramio reached ~ 75 MM geo-tagged photograph
It contains an enormous amount of latent implicit
knowledge, the challenge is to extract it.
Applying Computer Vision Techniques to Public Web Images for Environmental Monitoring
3. DIPARTIMENTO DI ELETTRONICA, INFORMAZIONE E BIOINGEGNERIA
Introduction
Applying Computer Vision Techniques to Public Web Images for Environmental Monitoring
3
• Challenge: transform geolocated and
timestamped public images depicting
natural scenarios into virtual
environmental monitoring stations.
• Complement existing measurements.
4. DIPARTIMENTO DI ELETTRONICA, INFORMAZIONE E BIOINGEGNERIA
Project
4
SnowWatch aims at:
• improving snow analysis and water management through
the acquisition and analysis of public media content
depicting alpine mountains
• Creating alpine snow media dataset with open contribution
of new media and access to processed media and its
results for any environmental researchers
Applying Computer Vision Techniques to Public Web Images for Environmental Monitoring
5. DIPARTIMENTO DI ELETTRONICA, INFORMAZIONE E BIOINGEGNERIA
Content Providers
SnowWatch Engine
SnowWatch Architecture
User Photos
5
Flickr Photos
Content
Consumers
Webcams
API Media Storage
Get Image Orientation,
Identify Mountain Peaks
Get Image Snow Pixel-
level classification
Applying Computer Vision Techniques to Public Web Images for Environmental Monitoring
6. DIPARTIMENTO DI ELETTRONICA, INFORMAZIONE E BIOINGEGNERIA
Flickr Photographs Crawling
6
Does an image contain a
clear a mountain slope view?
- Heavily depends on the
photograph shooting point altitude
Applying Computer Vision Techniques to Public Web Images for Environmental Monitoring
7. DIPARTIMENTO DI ELETTRONICA, INFORMAZIONE E BIOINGEGNERIA
Flickr Crawling Implementation
7
SVM Image Content-based Classifier:
Bag of Visual Words / Fisher Vector
SIFT/SSIM/GIST/HOG2x2 Descriptors
Spatial Histogram, Vertical Bands
Manually annotated 1658 samples (~70%) for training and
validation and 710 samples (~30%) for testing.
Applying Computer Vision Techniques to Public Web Images for Environmental Monitoring
8. DIPARTIMENTO DI ELETTRONICA, INFORMAZIONE E BIOINGEGNERIA
Webcam Images Aggregation
8
Identified ~2.000 webcams depicting mountains in Alpine
regions.
Each webcam produces an image every 1’ – 15’.
• 67% of the images are not suitable due to bad weather
• More than 50.000 images and 6 GB of data acquired daily
Through the extraction of the edges we can compute the percentage of the
skyline clearly visible, and discard all images with visibility lower than a certain
threshold, obtaining the following ROC curve varying the threshold:
Applying Computer Vision Techniques to Public Web Images for Environmental Monitoring
9. DIPARTIMENTO DI ELETTRONICA, INFORMAZIONE E BIOINGEGNERIA
Webcam Images Aggregation
9
Remaining small moving obstacles (such as low clouds,
persons, vehicles) and varying illumination conditions are
managed through the median aggregation of all daily images
Applying Computer Vision Techniques to Public Web Images for Environmental Monitoring
10. DIPARTIMENTO DI ELETTRONICA, INFORMAZIONE E BIOINGEGNERIA
Photo Peak Identification
10
An orientation of a geo-tagged photograph is
needed.
Given a Digital Elevation Model (a 3D terrain
model) we can simulate the terrain
silhouettes that should be seen from the
photographer point of view.
Steps
Panorama
Scale
Edges
Filter
Match
Applying Computer Vision Techniques to Public Web Images for Environmental Monitoring
11. DIPARTIMENTO DI ELETTRONICA, INFORMAZIONE E BIOINGEGNERIA
Photo Panorama
11
Steps
Panorama
Scale
Edges
Filter
Match
Applying Computer Vision Techniques to Public Web Images for Environmental Monitoring
12. DIPARTIMENTO DI ELETTRONICA, INFORMAZIONE E BIOINGEGNERIA
Photo Scaling
12
Photo must be scaled properly, so the
mountains have the same pixel dimension on
the photograph and on the panorama.
This can be obtained by calculating the Field
Of View (FOV) of the photo:
Steps
Panorama
Scale
Edges
Filter
Match
Applying Computer Vision Techniques to Public Web Images for Environmental Monitoring
13. DIPARTIMENTO DI ELETTRONICA, INFORMAZIONE E BIOINGEGNERIA
Photo Edge Extraction
13
Steps
Panorama
Scale
Edges
Filter
Match
Applying Computer Vision Techniques to Public Web Images for Environmental Monitoring
14. DIPARTIMENTO DI ELETTRONICA, INFORMAZIONE E BIOINGEGNERIA
Photo Edge Filtering
14
Too many edges on the image that do not
belong to a terrain contour!
• Skyline detection algorithm is employed, and all the edge
pixels above the skyline are removed, being considered
obstacles or clouds.
• Weighting mechanism is applied, which assigns
decreasing weights to the edge pixels as the distance
from the skyline increases.
Steps
Panorama
Scale
Edges
Filter
Match
Applying Computer Vision Techniques to Public Web Images for Environmental Monitoring
15. DIPARTIMENTO DI ELETTRONICA, INFORMAZIONE E BIOINGEGNERIA
Photo Alignment
15
Find the best overlap maximizing the vector
cross correlation score:
Steps
Panorama
Scale
Edges
Filter
Match
Applying Computer Vision Techniques to Public Web Images for Environmental Monitoring
16. DIPARTIMENTO DI ELETTRONICA, INFORMAZIONE E BIOINGEGNERIA
Photo Peak Alignment
16
A global overlap between the photo and the
panorama is often not enough, a
neighborhood vector cross correlation
adjustment is applied to each identified
peak:
Steps
Panorama
Scale
Edges
Filter
Match
Local Match
Applying Computer Vision Techniques to Public Web Images for Environmental Monitoring
17. DIPARTIMENTO DI ELETTRONICA, INFORMAZIONE E BIOINGEGNERIA
Photo Peak Identification
17
Fraction of photographs with orientation correctly identified
Applying Computer Vision Techniques to Public Web Images for Environmental Monitoring
18. DIPARTIMENTO DI ELETTRONICA, INFORMAZIONE E BIOINGEGNERIA
Snow Pixels Classification
18
Classifying each pixel as snow covered/terrain is
much more than simply checking bright/dark pixels:
Applying Computer Vision Techniques to Public Web Images for Environmental Monitoring
19. DIPARTIMENTO DI ELETTRONICA, INFORMAZIONE E BIOINGEGNERIA
Snow Pixels Classification
19
SnowGround GroundGroundSnow Snow Snow
Applying Computer Vision Techniques to Public Web Images for Environmental Monitoring
20. DIPARTIMENTO DI ELETTRONICA, INFORMAZIONE E BIOINGEGNERIA
Snow Pixels Classification
20
Random Forest Classifier that considers pixel
and pixel neighbors color, global and local
intensity of the image.
Trained on 59 manually
labeled images
(~ 7 Million pixels),
outperform current
methods:
Applying Computer Vision Techniques to Public Web Images for Environmental Monitoring
21. DIPARTIMENTO DI ELETTRONICA, INFORMAZIONE E BIOINGEGNERIA
Environmental Impact Evaluation
21
Environmental consistency evaluation of
extracted snow cover indexes.
No direct ground truth, how do we
evaluate it?
Practical utility w.r.t. an environmental
model
Applying Computer Vision Techniques to Public Web Images for Environmental Monitoring
22. DIPARTIMENTO DI ELETTRONICA, INFORMAZIONE E BIOINGEGNERIA
Environmental Impact Evaluation
22
Use case: Como Lake water management
Flooding vs Irrigation trend, what is the
best solution?
Applying Computer Vision Techniques to Public Web Images for Environmental Monitoring
23. DIPARTIMENTO DI ELETTRONICA, INFORMAZIONE E BIOINGEGNERIA
SnowWatch 2°°°° year
23
If the physical snow estimations provided
by ARPA (Italian EPA) are not available,
Adding Virtual Snow Measures improves
the performance by ~100%!
Applying Computer Vision Techniques to Public Web Images for Environmental Monitoring
24. DIPARTIMENTO DI ELETTRONICA, INFORMAZIONE E BIOINGEGNERIA
SnowWatch 2°°°° year
24
With some irrigation <-> flooding target
solutions, using SnowWatch virtual
measurements is even better than using
ARPA snow estimation!
Applying Computer Vision Techniques to Public Web Images for Environmental Monitoring
25. DIPARTIMENTO DI ELETTRONICA, INFORMAZIONE E BIOINGEGNERIA
Current and Future Work
25
Current:
• MountainWatch: real time augmented reality mobile
application
• Expand tests on environmental impact
Future:
• Expand area of interest and content providers
• Orthophotos and new snow indexes
Applying Computer Vision Techniques to Public Web Images for Environmental Monitoring