Master thesis at Dublin City University, September 2013. Co-advised by Xavier Giró-i-Nieto, Kevin McGuinness and Noel O'Connor. More details: https://imatge.upc.edu/web/publications/investigating-eeg-saliency-and-segmentation-applications-image-processing

1. Investigating EEG for Saliency and
Segmentation Applications in Image
Processing
Eva Mohedano
1

2. CONTENT
1 - Problem statement
2 – Related Work
3 – Local exploration of the image
4 – Experimental set-up
5 – Signal Processing of EEG Signals
6 – Conclusions
2

3. CONTENT
1 - Problem statement
2 – Related Work
3 – Local exploration of the image
4 – Experimental set-up
5 – Signal Processing of EEG Signals
6 – Conclusions
3

4. 1- PROBLEM STATEMENT
Design a system based on a Brain Computer Interface (BCI) wich measure
Electroencephalography (EEG) signals to answer the following questions:
1 - Are the EEG signals to compute Saliency Maps?
2 - Are the EEG signals useful to images segmentation?
EEG Signals
Visual stimulus
BCI
Data Processing
4

5. 1- PROBLEM STATEMENT
Design a system based on a Brain Computer Interface (BCI) wich measure
Electroencephalography (EEG) signals to answer the following questions:
1 - Are the EEG signals to compute Saliency Maps?
2 - Are the EEG signals useful to images segmentation?
Visual stimulus
BCI
Data Processing
5

6. 1- PROBLEM STATEMENT
Design a system based on a Brain Computer Interface (BCI) wich measure
Electroencephalography (EEG) signals to answer the following questions:
1 - Are the EEG signals to compute Saliency Maps?
2 - Are the EEG signals useful to images segmentation?
Visual stimulus
BCI
Data Processing
6

7. 1- PROBLEM STATEMENT
1 - Are the EEG signals to compute Saliency Maps?
Motivation:
- New way to compute maps of the atention of the image
based directly in the reaction of the brain and not in the
features of the images (Niebur and Koch (1996) algorithm).
Visual stimulus
BCI
Data Processing
7

8. 1- PROBLEM STATEMENT
2 - Are the EEG signals useful to images segmentation?
Motivation:
- Reduce the user interaction to the minimun expression.
- Measure the brain reaction at local scale of the image.
Visual stimulus
BCI
Data Processing
8

9. CONTENT
1 - Problem statement
2 – Related Work
3 – Local exploration of the image
4 – Experimental set-up
5 – Signal Processing of EEG Signals
6 – Conclusions
9

10. 2- RELATED WORK
2.1 – BCI in image processing applications
The oddball paradigm
10

11. 2- RELATED WORK
2.1 – BCI in image processing applications
The oddball paradigm
P300
11

12. 2- RELATED WORK
2.1 – BCI in image processing applications
The oddball paradigm
P300
• Speed rate around 10Hz
• Usually experiements centered to
find target images not target regions
12

13. 2- RELATED WORK
Event-Related Potential
•8 electrodes placed mainly in the
posterior points on the scalp.
• Which is consistent with the
discriminating
activity
typically
produced by a P300 ERP.
[Optimising the Number of Channels in
EEG-Augmented Image Search. Graham
Healy]
13

14. 2- RELATED WORK
Event-Related Potential
•8 electrodes placed mainly in
How to present the scalp. to the
image
posterior points on the
generate and is consistent signal?
• Which detect this with the
discriminating
activity
produced by a P300 ERP.
typically
[Optimising the Number of Channels in
EEG-Augmented Image Search. Graham
Healy]
14

15. CONTENT
1 - Problem statement
2 – Related Work
3 – Local exploration of the image
4 – Experimental set-up
5 – Signal Processing of EEG Signals
6 – Conclusions
15

16. 3- LOCAL EXPLORATION OF THE IMAGE
First Design: Sliding Window
http://www.youtube.com/watch?v=bKTGKVx58Ps
16

17. 3- LOCAL EXPLORATION OF THE IMAGE
CHALLENGE 1
• Eyes movement affect to the EEG signals – Introduce Artifacts to the signal
Opened eyes / Closed eyes. Image from the slides Dr. Ranjith Polusani
17

18. 3- LOCAL EXPLORATION OF THE IMAGE
CHALLENGE 2
• Progressive inspection may not generate a useful reaction in the EEG waves.
Suggestions meeting Thomas Ward and Nima Bidgely Shamlo :
- Follow Oddball Paradigm and perform RSVP od the windows
P300
SNAP - Simulation and Neuroscience
Application Platform
18

19. 3- LOCAL EXPLORATION OF THE IMAGE
CHALLENGE 3
•Syncronitzation Problem
19

20. 3- LOCAL EXPLORATION OF THE IMAGE
CHALLENGE 4
•Size of the object / window
Grabcut Dataset – Objects of different size
Suggestions meeting Thomas Ward and Nima Bidgely Shamlo :
To use images with an homogeneus background with a salient object.
The number of distractors (windows with background) must be higher
than the number of targets (windows with object).
20

21. 3- LOCAL EXPLORATION OF THE IMAGE
CHALLENGE 5
•What am I seeing?
21

22. 3- LOCAL EXPLORATION OF THE IMAGE
CHALLENGE 5
•What am I seeing?
22

23. 3- LOCAL EXPLORATION OF THE IMAGE
CHALLENGES
SOLUTIONS
1 - Eyes movement
Display a fixed window on the screen
2 - Progressive inspection
SNAP to perform a random RSVP
3 - Syncronitzation Problem
First test with flashes to find ERPS
4 - Size of the object / window
Generate my own dataset
5 - What am I seeing?
• Is it just noise?
• Am I able to detect something?
Real time visualitzation of the signal
23

24. CONTENT
1 - Problem statement
2 – Related Work
3 – Local exploration of the image
4 – Experimental set-up
5 – Signal Processing of EEG Signals
6 – Conclusions
24

25. http://www.youtube.com/watch?v=KsgtvQkOE
lQ&feature=youtu.be
4- EXPERIMENTAL SET-UP
Second Design: Starting from the easiest case
a) Device Calibration
I. Real time visualization of Alpha waves
II. Detecting ERPS
b) Synthetic Images
I. RSVP synthetic images fitted in the window.
c) RSVP real images
25

26. 4- EXPERIMENTAL SET-UP
a) Device Calibration
?
Is the device well
connected?
Is the syncronitzation
method correct ?
Am I able to detect
something?
26

27. 4- EXPERIMENTAL SET-UP
a) Device Calibration
SIGNAL EXPECTED - Closed eyes – Alpha waves (8-12 Hz)
Closed-eye EEG alpha waves (10-20 channels Pz-Top, Fz-Bottom) extracted from http://blog.grahamhealy.com/
27

28. 4- EXPERIMENTAL SET-UP
a) Device Calibration
SIGNAL OBTAINED
5 seconds Closed Eyes
Amplitude (uV)
40
20
0
-20
-40
0
0.5
1
1.5
0
0.5
1
1.5
2
2.5
3
3.5
Time (sec)
5 seconds Opened Eyes
4
4.5
5
4
4.5
5
Amplitude (uV)
40
20
0
-20
-40
2
2.5
3
Time (sec)
3.5
28

29. 4- EXPERIMENTAL SET-UP
a) Device Calibration
Finding ERPS response after a white flash
SIGNAL EXPECTED: After the flash P100 and a negative peak between 150-200ms
Presenting a serie of white flashes (2 seconds between the flashes)
29

30. 4- EXPERIMENTAL SET-UP
a) Device Calibration
Finding ERPS response after a white flash
SIGNAL OBTAINED: 60 Flashes to get the response.
Averaged ERP waveform per channel
Channel
P100 (ms)
1
130
320
2
90
210
3
90
210
4
10
220
5
110
220
6
90
210
7
10
220
8
100
22
Mean
10
N1 (ms)
80
23
Amplitude (uV)
5
0
-5
-10
-15
0
100
200
300
400
500
600
Time (ms)
700
800
900
1000
30

31. CONTENT
1 - Problem statement
2 – Related Work
3 – Local exploration of the image
4 – Experimental set-up
5 – Signal Processing of EEG Signals
6 – Conclusions
31

32. 5- SIGNAL PROCESSING OF EEG SIGNALS
Data adquisition
100 windows per Image
1 Target
99 Distractors
32

33. 5- SIGNAL PROCESSING OF EEG SIGNALS
Preprocessing: Single trial
One Image
1
2
3
4
5
6
7
8
33

34. 5- SIGNAL PROCESSING OF EEG SIGNALS
Preprocessing: Single trial - PROBLEM
- Signal very noisy
- Single Targets and Single
Distractors very similar
34

35. 5- SIGNAL PROCESSING OF EEG SIGNALS
Preprocessing: Feature
96 Distractors
Mean Absolute Amplitude
looks different
96 Targets
Energy from 0 to
600ms
Feature for the window
presented
35

36. 5- SIGNAL PROCESSING OF EEG SIGNALS
Preprocessing: Averaged trials
1 averaged target
99 averaged
distractors
Energy from
0-600 ms
36

37. 5- SIGNAL PROCESSING OF EEG SIGNALS
Preprocessing: Averaged trials
SINGLE
AVERAGED
1 x 32 single repeats
1 averaged target
99 x 32 single
repeats
99 averaged
distractors
1 x 32 window repeats
1 averaged target
99 x 32 single
repeats
Average
by 32
99 averaged
distractors
1 x 32 window repeats
1 averaged target
99 x 32 single
repeats
99 averaged
distractors
37

38. 5- SIGNAL PROCESSING OF EEG SIGNALS
Preprocessing: Averaged trials
1 averaged target
Problem:
Too few (target) samples
for training
99 averaged
distractors
1 averaged target
99 averaged
distractors
1 averaged target
99 averaged
distractors
38

39. 5- SIGNAL PROCESSING OF EEG SIGNALS
Bootstrapping
SINGLE
AVERAGED
1 x 32 single repeats
1 x 32 averaged target
99 x 32 single
repeats
99 x 1
averaged distractors
1 x 32 window repeats
99 x 32 single
repeats
1 x 32 window repeats
99 x 32 single
repeats
1 x 32 averaged target
Bootstrapping
99 x 1
averaged distractors
1 x 32 averaged target
99 x 1
averaged distractors39

40. 5- SIGNAL PROCESSING OF EEG SIGNALS
Classification
Problem:
Unbalanced dataset for
binary classification
AVERAGED
1 x 32 averaged target
99 x 1
averaged distractors
1 x 32 averaged target
99 x 1
averaged distractors
1 x 32 averaged target
99 x 1
averaged distractors40

41. 5- SIGNAL PROCESSING OF EEG SIGNALS
Classification
AVERAGED
AVERAGED
1 x 32 averaged targets
99 x 1
averaged distractors
1 x 32 averaged targets
Subsample
1 x 32 averaged targets
99 x 1
averaged distractors
1 x 32 averaged targets
Subsample
1 x 32 averaged targets
99 x 1
averaged distractors
32 x 1 avgd distractors
32 x 1 avgd distractors
1 x 32 averaged targets
Subsample
32 x 1 avgd distractors
41

42. 5- SIGNAL PROCESSING OF EEG SIGNALS
Classification
AVERAGED
HISTOGRAM
1 x 32 averaged targets
32 x 1 avgd distractors
1 x 32 averaged targets
32 x 1 avgd distractors
1 x 32 averaged targets
32 x 1 avgd distractors
42

43. 5- SIGNAL PROCESSING OF EEG SIGNALS
Classification
AVERAGED
1 x 32 averaged targets
32 x 1 avgd distractors
1 x 32 averaged targets
32 x 1 avgd distractors
SVMTRAIN
(linear
kernel)
Classifier
1 x 32 averaged targets
32 x 1 avgd distractors
43

44. 5- SIGNAL PROCESSING OF EEG SIGNALS
Classification
100 x 32 single repeats
Average
by 32
100 avgd windows
44

45. 5- SIGNAL PROCESSING OF EEG SIGNALS
Classification
100 x 32 single repeats
Average
by 32
100 avgd samples
Classifier
SVM
PREDICT
45

46. 5- SIGNAL PROCESSING OF EEG SIGNALS
8 samples feature vectors
Cross validation approach 3 train + 1 test

47. CONTENT
1 - Problem statement
2 – Related Work
3 – Local exploration of the image
4 – Experimental set-up
5 – Signal Processing of EEG Signals
6 – Conclusions
47

48. 6- CONCLUSIONS
- Results from sythetic images provide and evidence that BCI devices could be
used to located an object into an image.
- Simplicity of the system: Energy value from 8 channels to train SVM with
lineal kernel.
Future work
-Study the impact of the number of repetitions.
- Extract better features.
-Analize data from real images.
-Tool to evaluate and compare the EEG mask (ROC, Jaccard index)
48