This document provides an overview of Nilo Sarraf's PhD dissertation which examines how emotions impact search performance. Sarraf proposes including human emotion data from wearable devices as an additional input to artificial neural networks to improve search results based on neurological feedback. His dissertation aims to determine the effects of different emotional dimensions like valence and arousal on search effectiveness and efficiency. Sarraf conducted experiments using the Emotiv EEG headset to measure brain activity during searches under neutral, pleasant and unpleasant emotional states induced by IAPS pictures. The results will contribute to developing "smart emotional neuro search engines" that can improve search results based on reading a user's brain waves through wearable devices.

1. BayCHI Speaker Series
April 14, 2015
Nilo Sarraf
Towards Smart Emotional Neuro Search
Engines: An Extension of the Human Brain
PhD Gateway Program
SJSU & QUT School of Information

2. Disclaimer
❖ This presentation is an overview of my ‘in-progress’ PhD
dissertation (Third year)
❖ I am not a neuroscientist but have research background in
neuroscience and HCI
❖ My PhD dissertation work is NOT affiliated with the
company where I work
❖ This deck was presented at the BayCHI April 2015
speaker series: http://www.baychi.org/calendar/20150414/
2

3. Research Background &
Interests
❖ Research at Stanford University - Neuroscience
❖ Research in the industry - User Experience Research
❖ Passion for Neuroscience and Information Retrieval
❖ Positioning in ‘Neuro Information Science’
❖ Neurophysiological methods in Information Science
❖ Wearable Computing
❖ Artificial Neural Networks
3

4. Two Parts of this Presentation
1.Overview of my doctoral dissertation (in-progress)
❖ Positioning in Neuro Information Science (Gwizdka, 2012)
❖ “How does Affect Impact Search Performance?”
2.Proposal to the industry
❖ Improve search results based human neurological
feedback through wearable computing devices
❖ Artificial Neural Networks architecture improvements by
adding human emotion data input

5. Dissertation Overview

6. Introduction
❖ Purpose: Examine the potential effects of emotions on
information retrieval, as revealed in search processes
❖ Focus: Examine the effects of emotions on search
performance, in terms of search effectiveness and search
efficiency
❖ Conclusions will be drawn about the effects, if any, of
different dimensions of emotions and their effects on
search performance
6

7. Affect in Design
❖Affective component of information retrieval system
design is becoming increasingly essential
❖Expressions such as “pleasurable engineering” or
“emotional design” have become the driving factors
in system design and these expressions have also
been extended to information retrieval system design
(Nahl & Bilal, 2007)

8. Historical Evolution of Information Science
Research
❖ System Oriented Approach
❖ Precision and Recall
❖ User Oriented Approach
❖ User behavior
❖ Cognitive Oriented Approach
❖ Affect Oriented Approach
8

9. Research Questions
❖ The gap: The effects of physiological and neurological emotion
responses in information retrieval, more specifically on web search
❖ Two dry runs and two pilot studies
❖ Q1: How do dimensions of emotions affect search effectiveness?
❖ Q2: How do dimensions of emotions affect search efficiency?
❖ Q3: Are there any interactional effects between dimensions of
emotions and search performance?
❖ The hypothesis is that positive emotional states have positive effects
on information retrieval and negative emotional states affect users’
web search performance negatively
9

10. Human Emotions

11. Theoretical Models and
Framework
❖Emotion in Information Seeking – ISP Model (Kuhlthau (1991)
❖Six steps of the ISP model: Initiation, Selection, Exploration,
Formulation, Collection, Presentation
❖Emotion in Information Processing – CPM Model (Scherer,
2001)
❖Emotions elicit as individuals evaluate continuous events,
objects, or situations
❖Evaluations with respect to the effect they may have on
individual’s goals
11

12. Structures of Emotions
❖ Discrete
❖ Darwin, the father of the discrete approach, claimed that there exist six basic
emotions: fear, happiness, surprise, anger, sadness, and disgust (Darwin, 1872;
Ekman, 1992)
❖ Continuous
❖ Addresses different ‘dimensions’ of emotions (Russel & Mehrabian, 1977; Russel,
1994). These theorists state that there are two dimensions of emotions, valence and
arousal (Russell, 1994; Russell & Mehrabian, 1977; Russell & Steiger, 1982; Barrett
& Russell, 1999)
❖ Valence: indicates the positivity versus the negativity of an emotion, ranging from
highly positive to negative states
❖ Arousal: measures the calmness versus the excitement of an emotion, ranging
from calming to exciting (or agitating) states
12

14. Emotional Dimensions Associated with Brain
Waves
❖ Russel’s (1989) research shows that the following two emotional dimensions
are associated with various brain waves:
❖ Theta waves, also seen in meditative states (Cahn & Polich, 2006), show
arousal or drowsiness in adults
❖ Alpha waves are exhibited when closing the eyes and during relaxation
❖ Beta waves, linked with motor behavior, occur when the individual is actively
moving (Pfurtscheller and da Silva, 1999)
❖ Low beta frequencies are often associated with concentration and/or active
thinking
❖ Gamma waves represent cognitive or motor functions (Niedermeyer & da
Silva, 2004)
14

16. Emotiv EPOC Neuro Headset

17. Research Design
❖Experimental Research Design
❖Two-Way Repeated Measures ANOVA
❖Independent Variable (IV) - Emotional State
❖Three sets of 20 IAPS pictures at neutral, pleasant, and unpleasant level
❖Brain Activities: Alpha and Beta waves
❖SAM Self-Report: Valence and Arousal
❖Dependent Variable (DV) - Search Performance
❖Time on Task
❖Task Completion
❖Number of Search Queries
❖Number of Web Pages
17

18. EEG Data Analysis
❖Careful data processing and analysis must be done when collecting EEG raw data sets
❖To analyze EEG raw data:
❖Pass each channel through a high/low pass filter
❖Perform a transform on the data
❖Filter for a key frequency band
1.Preprocessing: The raw EEG data usually is not clean and some preprocessing steps are needed:
❖ Applying high-pass and low-pass filters
❖ High-pass filter: Removes the low frequencies
❖ Low-pass filter: Removes the high frequency brain waves
2.Feature Extraction: Divide the signals in chunks of time in order to extract features out of each one of
these pieces
❖ MatLab has many functions for filtering these signals where one could set band pass filters
❖ E.g. alpha waves are between 8Hz and 12Hz
❖ EEGLab
18

19. EEG Data Analysis: Recognizing Dimensions of
Emotions
❖In recognizing the dimensions of emotions (Valence &
Arousal) through EEG devices studies appear to suggest
that:
❖Valence - Alpha asymmetry (present on states of low
cognitive load) on the frontal lobes
❖Arousal - The ratio of beta-alpha waves (present on
states of high cognitive load) on the frontal lobes
19

21. Challenges & Limitations
❖ Some of the challenges and limitations when it comes to
the proposed model for developing Affective Neuro
Search
❖ The complex human motor movements may contribute
to ‘noise’ when it comes to reading the brain signals
❖ The EEG devices in the market today may not be able
to fully suppress all the noise emanating from major
body movements, such as head/hand movements
21

22. Proposal to the Industry
Designers & Architects

23. Problem Statement
❖The existing Artificial Neural Networks architectures are based
solely on digital data input
❖System programmers and architectures fail to approach
modeling the human brain holistically
❖ The main component, human emotion, is missing from this
equation
❖I propose that adding one additional data input of human
emotion may improve these artificial neural networks
23

24. Why Ponder on These
Issues?
❖This is the era of brain-controlled devices
❖This is the era of physical connection/collaboration between the
human brain and physical devices
❖In an era when humans are creating brain controlled airplanes,
neuro-gaming, and robots that learn behavior by reading human
emotions, there appear to be no limits in having search engines
read human emotions in order to improve search results based
on the neurological feedback they receive from brain waves
24

27. Research Contribution
❖ One of the main contributions of my
dissertation is my proposal to include human
emotions readings via wearable computing
devices as an additional data input for
statistical learning algorithms when creating
artificial neural networks
27

28. Smart Emotional Neuro Search
Engines
❖ I envision my dissertation contribute to the body of knowledge of Neuro
Information Science in developing search engines that, through wearable
computing devices that are able to read brain waves and dimensions of
emotions in order to improve search results based on the neurological
feedback that the search engines receive from brain waves
❖ Search engines become an extension of the human brain by receiving
brain waves that constantly provide neurological feedback in terms of the
search results that they provide
❖ Search engine reads brain waves by receiving the brain signals through
wearable computing devices and ‘learn to improve’
28

29. Thank You!
nilosarraf.com
@nilosarraf