Nero-IR is a novel area of research under cognitive psychology, neuro-physiological methods (eye tracking, EEG, EOG, and GSR) and machine learning to understand information searchers and to improve search experience. Neuro-IR is useful in investigating the search as a learning process and to employ these sensory data as assessment of reading, mind-wandering and in inferring metadata features for machine learning models. In this talk, I will introduce a unification framework for neuro-physiological data; practically these models provide context for user interactions. I will show how we can take advantage of many existing interactions combining various sensory platforms (e.g., PupilLabs, Emotiv, Empatica E4). Information fusion can provide numerous benefits in combining multiple-sources of neuro-physiological data. The most obvious among them is the expected performance gain due to combination of evidence from multiple cues. As a practical matter, acquisition of physiological metadata is a research frontier.

1. Assistant Professor, Computer Science
Web Science & Digital Libraries Research Group
@OpenMaze, @WebSciDL
Multi-Modal Sensing with
Neuro-Information Retrieval

2. Joined ODU Fall 2018
Assist. Professor Cal Poly 2016-2018
PhD @ Texas A&M University, 2016
Originally from Sri Lanka
• Neuro-IR, HCI, Machine Learning, Data Science, Digital Library
• http://www.cs.odu.edu/~sampath/
• sampath@cs.odu.edu, (757) 683-7787
• @OpenMaze
2
My Journey!!

3. My Research
NIRDS
Neuro-Information
Retrieval
&
Data Science
User
Modeling
Oculomotor
Plant
Digital
Library
Data Analytics
3
@NirdsLab
https://www.cs.odu.edu/~sampath/nirds-lab/

4. Big Data Aspect of User Behavior Analytics
• Lately, the term "big data" tends to refer to the use
of predictive analytics, user behavior analytics, or certain
other advanced data analytics methods that extract value
from data, and seldom to a particular size of data set.
4

5. Big Data Aspect of User Behavior Analytics
• Data sets grow rapidly - in part because they are increasingly
gathered by cheap and numerous information-
sensing Internet of things devices such as mobile devices,
wearables, cameras, microphones and appliances.
5

6. Information Retrieval
• Information Retrieval (IR) is finding material
(usually documents) of an unstructured nature
(usually text) that satisfies an information
need from large collections (usually stored on
computers).
• Most prominent example: Web Search
Engines
6

7. Neuro-Information Retrieval
• Neuro-IR is an emerging field that aims to take advantage of
advances in cognitive psychology and psycho-physiological
methods and apply them to answering Information Science
questions.
– e.g., low-cost consumer-level devices raise the possibility of introducing these
devices into the home environment as novel ways of collecting data about
users and in support of new interaction modalities.
7

8. Neuro-IR: My Focus
• Eye Tracking measures what people see. It has wide variety of uses in
neuro-IR as it provides valuable indications of interest, attention and
attraction.
• Electroencephalography (EEG) accounts the most popular neuro-IR
technology because of its relatively low costs and
manageable equipment requirements. It can
measure moment-to-moment changes and
identify memory activation in real time.
• Galvanic Skin Response (GSR) is calculated
from the skin conductance which is an
indication of psychological or
physiological arousal.
8

9. Applications: ASD
• About 1 in 59 children has been identified with autism
spectrum disorder (ASD)
• The total costs per year for children with ASD in the United
States were estimated to be between $11.5 billion – $60.9
billion
9

10. Applications: ASD
• Collaboration with Psychology professors, Dr. Mark Jaime
(Early Sensory Experience Lab at IUPUC) and Chris Harshaw
(Mechanism Underlying Sociality Lab at University of New
Orleans
• Early brain markers of social impairment that can potentially be used as
“red flags” for Autism in infants or toddlers that do not yet speak
• EEG Cap
10

11. Temporal Relationship : ASD and EEG
• We analyzed the short-term and long-term relationships
between ASD and brain activity using Electroencephalography
(EEG) readings taken during the administration of ADOS-2.
11

12. Frequency Decomposition Method
• We decomposed signals from each electrode into frequency
band counterparts, to observe characteristics at each band
– We performed band pass filtering using Butterworth filters of order (n)
= 5
12
Source Signal
Frequency Band Counterparts

13. Wavelet Transformation Method
• We used wavelet transformation to obtain information about
the signal at a given frequency (f) and time (t)
13
Ψ = wavelet function
Morlet wavelet
fc = 1 Hz, fb = 1.5 Hz
x(t) = time series signal
a = scale (w.r.t. frequency)
b = translation (w.r.t. time)
Wavelet Transformation Equation

14. Visualization
14

15. Evaluation - Deep Learning Model
• Convolutional Neural Network (CNN) to classify power
matrices obtained from wavelet transformation method into
ASD or TD
15

16. Results (I)
16

17. ASD Fever Effect
• Thermoregulatory abnormalities are often noted in ASD,
but have received limited attention from researchers to
date.
• ‘Fever effect’ – children with ASD show surprising improvement in
social functioning during high fevers.
• EEG Cap + Thermal Imaging Camera
17

18. Extending to Multi Label Classification
18

19. Results (II)
19

20. 20
Social Interaction with ASD
• Connecting Brain Function and Visual Behavior during
Social Interaction with ASD
– Distinct eye movement scan patterns, in combination with
electroencephalographic (EEG) recordings, during naturalistic
and dynamic social interaction, can be used to delineate
biomarkers of Autism Spectrum Disorder (ASD)
– Collaboration with Andrew Duchowski, Clemson University

21. 21
Social Interaction with ASD
• Naturalistic, dynamic, joint attention tasks for children with
and without a diagnosis of ASD.
– Synchronization of eye tracker to EEG devices in terms of temporal
synchronization and masking of EEG noise produced by extraocular muscles
across people in the environment and the communicative tasks.
– AOIs to label individuals’ faces to allow estimation of gaze switching within
faces between the eyes, mouth, and nose and as well as between faces

22. Eye Movements
• Fixations
• High acuity vision
• Eye is stable in regard to the object of interest
• Saccades
• No vision
• Move eyes between eye fixations very rapidly
• Smooth pursuits
• Various quality of vision
• Eyes follow an object
22
Human eye provides a plethora of information useful for user modeling

23. Eye Movement Control
• Brain
• Oculomotor Plant
• Extraocular muscles
• eye globe with surrounding tissues
23

24. Eye Movements: Recorded Signal
24

25. Applications: ADHD
• Collaboration with ODU Special Education faculty, Dr. Anne
Michalek.
• Distinct eye movement scan patterns within specific areas
of interests (AOIs) to estimate psychometric measures to
understand ADHD
25

26. WMC Task - RSPAN
• We analyzed the eye movements of ADHD and Non-ADHD during a
Working Memory Capacity (WMC) task, Reading Span (RSPAN)
• Participants are asked to read a sentence and letter they see on a computer
screen.
• Sentences are presented in varying sets of 2-5 sentences.
• Participants are asked to judge sentence coherency by saying 'yes' or 'no' at
the end of each sentence.
• Then, participants are asked to remember the letter printed at the end of the
sentence. After a 2-5 sentence set, participants are asked to recall all the
letters they can remember from that set.
26

27. Participants
• We recruited a total of 14 adult participants with and without a
diagnosis of ADHD for this study.
27

28. Areas Of Interest (AOI)
• We used Tobii Pro X2-60 computer screen-based eye tracker with
Tobii Studio analysis software
• We used Tobii Studio analysis software's Area of Interest (AOI) tool
to draw the boundaries around elements of the eye tracking
stimulus.
• The three AOI groups
• AOI 1 - Stimulus (the whole sentence)
• AOI 2 - Critical word (critical word when determining the coherency of the
sentence)
• AOI 3 - Determiner (the decision point with the letter to be remembered)
28

29. Feature Sets
• We derived two feature sets for the investigation of fixations and
saccades within AOIs based on the following qualifiers:
• Number of fixations in AOI 1, 2 and 3
• Fixation duration in AOI 1, 2 and 3
• Average fixation duration in AOI 2
• Fixation standard deviation in AOI 2, pupil diameter of both eyes in AOI 2
and 3
• Maximum and minimum saccade amplitude in AOI 1, 2 and 3
• Average saccade amplitude in AOI 1, 2 and 3
• Standard deviation of saccade amplitude in AOI 1, 2 and 3
• Scene-based Feature set - includes the above qualifiers within the
AOIs of sets of 2-5 sentences
• Sentence-based Feature set - includes the above qualifiers within
the AOIs of all the sentences
29

30. Results
30

31. Simulated Learning Environment for ADHD
• SLE will facilitate realistic academic stimuli eliciting cognitive
activity and eye movements for analysis and correspondence to
ADHD diagnostic criteria, executive attention, and audio/visual
retention in college students with ADHD for the design of
accessible classrooms.
31

32. Neuro-Physiological Fusion
32

33. Neuro-Physiological Fusion
33

34. Integrated Simulation Environment
• The aim of this project is to design an innovative Integrated Simulation
Environment (ISE) that leverages the integration of multiple sensory
systems, including eye movements and brain electrical signals measured
using eye tracking and EEG, respectively.
– assemble the hardware to allow controlled display of visual stimulus (e.g., the VR/AR
HMD) along with component facilitating physiological measures of eye movement (eye
tracker) and brain activity (EEG) during the Cognition test battery for spaceflight
– physiological measure analytics, tuned with parameters designed to detect significant
events deemed indicative of cognitive health, e.g., EEG synchronized with eye movement
metrics in response to specifically selected cognitive tasks
34

35. DFS: Dataset File System
35
• If datasets contain metadata that is sufficient to determine
the pipeline needed to reach the target format, it is safe to
assume that it could be automated.

36. NeuroPype
36
• NeuroPype is a powerful platform for real-time brain-
computer interfacing, neuroimaging, and bio/neural signal
processing with an open-source visual pipeline designer and
tools for interfacing with diverse sensor hardware, recording
data, and other functions.

37. NeuroPype: Muse and PupilLabs
37

38. Thanks to my Collaborators/Students
38

39. Thank you!
39
• Web : https://www.cs.odu.edu/~sampath/
• Contact : sampath@cs.odu.edu
• WS-DL : https://ws-dl.cs.odu.edu/ @WebSciDL
• Twitter : @OpenMaze
Questions and Comments