http://imatge-upc.github.io/egocentric-2016-saliency/ This project focuses on the creation of a new type of egocentric (first person) vision dataset. For that purpose, the EgoMon Gaze & Video Dataset is presented. This EgoMon dataset was recorded using the eye gaze tracking technology that studies the movement and position of the eyes. The Tobii glasses (wearable, eye tracker and head-mounted device) were the main tool used to record and extract the gaze data for this dataset. The dataset consists in 7 videos of 34 minutes each one of average, 13428 frames extracted from each video (with a frequency of 1 fps), and 7 files with the gaze data (fixations points of the wearer of the glasses) for each frame and video. The videos were recorded in the city of Dublin (Ireland) both indoor and outdoor. The generated dataset has been used to evaluate the performance of a state of art model for visual saliency prediction on egocentric video.

1. Mònica Chertó Sarret, “EgoMon Gaze and Video Dataset for Visual Saliency Prediction”. ESEIAAT 11/07/2016.
EgoMon Gaze and Video
Dataset for Visual Saliency
Prediction
Mònica Chertó Sarret Supervised by: Cathal Gurrin and Xavier Giró

2. Mònica Chertó Sarret, “EgoMon Gaze and Video Dataset for Visual Saliency Prediction”. ESEIAAT 11/07/2016.
Outline
1. Introduction
2. State of the art
3. EgoMon Gaze & Video Dataset
4. Visual Saliency Prediction
5. Conclusions and Future Works
2

3. Mònica Chertó Sarret, “EgoMon Gaze and Video Dataset for Visual Saliency Prediction”. ESEIAAT 11/07/2016.
1. Introduction
3

4. Mònica Chertó Sarret, “EgoMon Gaze and Video Dataset for Visual Saliency Prediction”. ESEIAAT 11/07/2016.
Introduction. Main goals and project planning
4
Goals February March April May June
Construct the Dataset
Run state of the art saliency estimator
with a single image
Frames extraction
Run saliency estimator with the
extracted frames
Compare Results

5. Mònica Chertó Sarret, “EgoMon Gaze and Video Dataset for Visual Saliency Prediction”. ESEIAAT 11/07/2016.
Equipment and Software. Eye tracker, Tobii Glasses
5

6. Mònica Chertó Sarret, “EgoMon Gaze and Video Dataset for Visual Saliency Prediction”. ESEIAAT 11/07/2016.
Equipment and Software. Tobii studio Software
6

7. Mònica Chertó Sarret, “EgoMon Gaze and Video Dataset for Visual Saliency Prediction”. ESEIAAT 11/07/2016.
Equipment and Software.
7

8. Mònica Chertó Sarret, “EgoMon Gaze and Video Dataset for Visual Saliency Prediction”. ESEIAAT 11/07/2016.
Equipment and Software.
8

9. Mònica Chertó Sarret, “EgoMon Gaze and Video Dataset for Visual Saliency Prediction”. ESEIAAT 11/07/2016.
Publication
9
Repositori of Egocentric-saliency in GitHub [online] Available: https://github.com/imatge-upc/egocentric-saliency
EgoMon Dataset [online] Available: https://imatge.upc.edu/web/sites/default/files/resources/1720/saliency/2016-egomon/

10. Mònica Chertó Sarret, “EgoMon Gaze and Video Dataset for Visual Saliency Prediction”. ESEIAAT 11/07/2016.
Outline
1. Introduction
2. State of the art
3. EgoMon Gaze & Video Dataset
4. Visual Saliency Prediction
5. Conclusions and Future Works
10

11. Mònica Chertó Sarret, “EgoMon Gaze and Video Dataset for Visual Saliency Prediction”. ESEIAAT 11/07/2016.
2. State of the art
11
GTEA Dataset UT Ego Dataset
GTEA (Georgia Tech Egocentric Activities) – Gaze Dataset [online] Available: http://ai.stanford.edu/~alireza/GTEA_Gaze_Website/
UT (University of Texas) Ego Dataset [online] Available: http://vision.cs.utexas.edu/projects/egocentric_data/UT_Egocentric_Dataset.html

12. Mònica Chertó Sarret, “EgoMon Gaze and Video Dataset for Visual Saliency Prediction”. ESEIAAT 11/07/2016.
Outline
1. Introduction
2. State of the art
3. EgoMon Gaze & Video Dataset
4. Visual Saliency Prediction
5. Conclusions and Future Works
12

13. Mònica Chertó Sarret, “EgoMon Gaze and Video Dataset for Visual Saliency Prediction”. ESEIAAT 11/07/2016.
Acquisition. Calibration process of the Tobii Glasses
13
Video tutorial uploaded on YouTube.

14. Mònica Chertó Sarret, “EgoMon Gaze and Video Dataset for Visual Saliency Prediction”. ESEIAAT 11/07/2016.
Acquisition. Results of the calibration process of the Tobii Glasses
14

15. Mònica Chertó Sarret, “EgoMon Gaze and Video Dataset for Visual Saliency Prediction”. ESEIAAT 11/07/2016.
EgoMon Gaze & Video Dataset
15
...
7 x text files
(gaze data)
7 x RAW (videos)
7 x Gaze (videos with
the gaze information
plotted)
13428 x frames extracted
75 x
narrative
images

16. Mònica Chertó Sarret, “EgoMon Gaze and Video Dataset for Visual Saliency Prediction”. ESEIAAT 11/07/2016.
EgoMon Gaze & Video Dataset
16

17. Mònica Chertó Sarret, “EgoMon Gaze and Video Dataset for Visual Saliency Prediction”. ESEIAAT 11/07/2016.
EgoMon Gaze & Video Dataset
17
INDOOR OUTDOOR

18. Mònica Chertó Sarret, “EgoMon Gaze and Video Dataset for Visual Saliency Prediction”. ESEIAAT 11/07/2016.
EgoMon. Oral Presentation
18

19. Mònica Chertó Sarret, “EgoMon Gaze and Video Dataset for Visual Saliency Prediction”. ESEIAAT 11/07/2016.
EgoMon. DCU and Albert College Park
19

20. Mònica Chertó Sarret, “EgoMon Gaze and Video Dataset for Visual Saliency Prediction”. ESEIAAT 11/07/2016.
EgoMon. Spanish Omelette
20

21. Mònica Chertó Sarret, “EgoMon Gaze and Video Dataset for Visual Saliency Prediction”. ESEIAAT 11/07/2016.
EgoMon. Playing cards
21

22. Mònica Chertó Sarret, “EgoMon Gaze and Video Dataset for Visual Saliency Prediction”. ESEIAAT 11/07/2016.
EgoMon. Botanic Gardens
22

23. Mònica Chertó Sarret, “EgoMon Gaze and Video Dataset for Visual Saliency Prediction”. ESEIAAT 11/07/2016.
EgoMon. Botanic Gardens (Narrative Clip)
23

24. Mònica Chertó Sarret, “EgoMon Gaze and Video Dataset for Visual Saliency Prediction”. ESEIAAT 11/07/2016.
EgoMon. Bus Ride
24

25. Mònica Chertó Sarret, “EgoMon Gaze and Video Dataset for Visual Saliency Prediction”. ESEIAAT 11/07/2016.
EgoMon. Walking to the Office
25

26. Mònica Chertó Sarret, “EgoMon Gaze and Video Dataset for Visual Saliency Prediction”. ESEIAAT 11/07/2016.
EgoMon. Privacy
26

27. Mònica Chertó Sarret, “EgoMon Gaze and Video Dataset for Visual Saliency Prediction”. ESEIAAT 11/07/2016.
EgoMon. Problems with the Gaze (Losses)
27
static
non-static

28. Mònica Chertó Sarret, “EgoMon Gaze and Video Dataset for Visual Saliency Prediction”. ESEIAAT 11/07/2016.
EgoMon. Processing, Eye Gaze data
28

29. Mònica Chertó Sarret, “EgoMon Gaze and Video Dataset for Visual Saliency Prediction”. ESEIAAT 11/07/2016.
EgoMon. Frame extraction
29
DURATION FRAMES EXTRACTED
TOTAL 3:43:41 13428
AVERAGE: 0:34:30 1918
1 fps

30. Mònica Chertó Sarret, “EgoMon Gaze and Video Dataset for Visual Saliency Prediction”. ESEIAAT 11/07/2016.
Outline
1. Introduction
2. State of the art
3. EgoMon Gaze & Video Dataset
4. Visual Saliency Prediction
5. Conclusions and Future Works
30

31. Mònica Chertó Sarret, “EgoMon Gaze and Video Dataset for Visual Saliency Prediction”. ESEIAAT 11/07/2016.
4. Visual Saliency Predictor.
31

32. Mònica Chertó Sarret, “EgoMon Gaze and Video Dataset for Visual Saliency Prediction”. ESEIAAT 11/07/2016.
Saliency Predictor. SalNet
32

33. Mònica Chertó Sarret, “EgoMon Gaze and Video Dataset for Visual Saliency Prediction”. ESEIAAT 11/07/2016.
EgoMon Gaze & Video Dataset
33
...
7 x text files
(gaze data)
7 x RAW (videos)
7 x Gaze (videos with
the gaze information
plotted)
13428 x frames extracted
75 x
narrative
images
...13428 x saliency models

34. Mònica Chertó Sarret, “EgoMon Gaze and Video Dataset for Visual Saliency Prediction”. ESEIAAT 11/07/2016.
Results of the Dataset
34

35. Mònica Chertó Sarret, “EgoMon Gaze and Video Dataset for Visual Saliency Prediction”. ESEIAAT 11/07/2016.
Quantitative Evaluation. Comparison Metric
35
Location-based Distribution-based
AUC-Judd, sAUC, NSS SIM, CC, EMD, KL
NORMALIZED SCANPATH SALIENCY
MIT Saliency Benchmark [online] Available: http://saliency.mit.edu/results_mit300.html

36. Mònica Chertó Sarret, “EgoMon Gaze and Video Dataset for Visual Saliency Prediction”. ESEIAAT 11/07/2016.
Results. Quantitative Evaluation
36

37. Mònica Chertó Sarret, “EgoMon Gaze and Video Dataset for Visual Saliency Prediction”. ESEIAAT 11/07/2016.
Results. Qualitative Evaluation
37
Example of GOOD results

38. Mònica Chertó Sarret, “EgoMon Gaze and Video Dataset for Visual Saliency Prediction”. ESEIAAT 11/07/2016.
Results. Qualitative Evaluation
38
Example of BAD results

39. Mònica Chertó Sarret, “EgoMon Gaze and Video Dataset for Visual Saliency Prediction”. ESEIAAT 11/07/2016.
Outline
1. Introduction
2. State of the art
3. EgoMon Gaze & Video Dataset
4. Visual Saliency Prediction
5. Conclusions and Future Works
39

40. Mònica Chertó Sarret, “EgoMon Gaze and Video Dataset for Visual Saliency Prediction”. ESEIAAT 11/07/2016.
40
Conclusions
Dataset Amount of Data Recorded
Device
Environment Number of
participants
GTEA 17 sequences Tobii eye-tracker
Glasses
Indoor 14
UT Ego 4 videos of 4 hours (16
h)
Looxcie
wearable camera
Indoor + Outdoor 4
EgoMon 7 clean videos (4 h)
7 gaze videos
13428 extracted frames
13428 saliency maps
7 files with eye gaze data
75 Narrative images
Tobii eye tracker
glasses +
Narrative Cip
Indoor + Outdoor 3

41. Mònica Chertó Sarret, “EgoMon Gaze and Video Dataset for Visual Saliency Prediction”. ESEIAAT 11/07/2016.
Future Works
Fine-tuning of saliency estimator based on the
comparison metric
41

42. Mònica Chertó Sarret, “EgoMon Gaze and Video Dataset for Visual Saliency Prediction”. ESEIAAT 11/07/2016.
Publication
42
http://imatge-upc.github.io/egocentric-2016-saliency/

43. Mònica Chertó Sarret, “EgoMon Gaze and Video Dataset for Visual Saliency Prediction”. ESEIAAT 11/07/2016.
43