CONISOFT 2015

1. Cognitive Informatics:
Intersection of Software Engineering and
Cognitive Science
Dr. Robert Atkinson
Assistant Professor
Director of the Advancing Next Generation Learning Environments Lab
School of Computing, Informatics, and Decision Systems Engineering
Ira A. Fulton School of Engineering

2. Agenda
1. Introduction
§ Definition
§ Three reasons why this is important
§ Impact
2. Challenges
3. Tools and Methods
4. Impact on Software Testing
§ Examples
5. Impact on Software Design
§ Examples
6. Conclusion

3. 1. Introduction

4. Definition | Cognitive Informatics
Software
Engineering
Cognitive
Science
Understand human information processing mechanisms (cognition)
aiming to achieve their implementation to create or improve systems (engineering)
All about systematic,
disciplined, and quantifiable
software production
All about the mind and its
processes: perception,
memory, reasoning, and
emotions

5. Definition | Cognitive Science
Cognitive
Science
Understand human information processing mechanisms (cognition)
aiming to achieve their implementation to create or improve systems (engineering)
All about the mind and its
processes: perception,
memory, reasoning, and
emotions

6. Definition | Cognitive Science
§ Information representation
§ Information processing
§ Information transformation
§ Human perception
§ memory
§ Reasoning
§ Emotions

7. Why this is important?
A. Improve human-computer interfaces (HCI) and user experience (UX)
Software that understands and adapts to its user necessities in real time,
such as: cognitive robots, cognitive networks, cognitive computers,
cognitive cars, as well as brain-machine interfaces for physically-
impaired persons, and cognitive binaural hearing instruments.

8. A. Improve HCI and UX
Mul$modal)emo$on)recogni$on)system)
User)
Brainwaves)
Eye)movements)
Facial)expressions)
Physiological)signals)
Sensing))
Devices)
Percep$on)
mechanisms)
Integra$on)
Algorithm)
Raw)data) Beliefs) State)

9. Why this is important?
B. Improve Artificial intelligence
Approaches to endow computers with human capabilities, such as
computer vision, consciousness, automated reasoning, learning, and
problem-solving. The AI research has not produced major breakthrough
recently due to a lack of understanding of human brains and natural
intelligence. Ignoring what goes on in human brain and focusing instead
on behavior has been a large impediment to understanding complex
human adaptive, distributed reasoning and problem solving

10. B. Improve Artificial intelligence
“... neurons combine so that each one helps with many memories at a
time, exponentially increasing the brain’s memory storage capacity to
something closer to around 2.5 petabytes [1 petabyte ≈ 1,000 terabytes].
For comparison, if your brain worked like a digital video recorder in a
television, 2.5 petabytes would be enough to hold three million hours of
TV shows. You would have to leave the TV running continuously for more
than 300 years to use up all that storage.”

11. Why this is important?
C. Understand Human Memory System
Understanding the principles and mechanisms of information
organization, retrieval and selection in human memory aims to find more
cognition-inspired methods of information memory system, problem
solving and reasoning at the web scale. Based on many investigations
on information retrieval and selection in human memory system, we can
view the human brain as a huge parallel distributed knowledge base
with multiple information granule networks. This supports the
improvement of parallel and web computing.

12. C. Understand Human Memory System

13. Impact
§ Multimodal interactive systems
§ Communicative robots
§ Web, text, and data mining
§ Multimedia, real time, or virtual environments for distributed collaborative work
§ Intelligent tutoring systems
§ Affect-driven adaptive games
§ Empathic and decision-capable health care applications

14. 2. Challenges

15. Definition | Software Engineering
Software
Engineering
Understand human information processing mechanisms (cognition)
aiming to achieve their implementation to create or improve systems (engineering)
All about systematic,
disciplined, and quantifiable
software production

16. Definition | Software Engineering
Software
Engineering
Challenges must be tackled from diverse perspectives since software engineering
includes several sub-disciplines (requirements design, testing, maintenance, quality,
configuration management among others).
But, let us concentrate our attention in two: software design and software testing.
Software
Testing
Software
Design

17. Challenges for Designing and Testing
§ Multidisciplinary and their Inherent complexity and diversity
§ Requirements of varying problem domain knowledge
§ Increased changeability or malleability of software
§ Abstraction and intangibility of software products
§ Dependability of interactions between software, hardware, and
human beings

18. 3. Tools and Methods

19. Tools and Methods
Tools
§ Brain-computer-interfaces
§ eye-tracking systems
§ face-based emotion recognition systems
§ Arousal or skin conductance sensing
§ Pressure and posture sensing
§ Hearth rate monitoring
§ Voice analysis

20. Tools | Brain-Computer Interfaces (BCI)
It is a particular type of a physiological instrument that uses brainwaves as
information sources (electrical activity along the scalp produced by the firing of
neurons within the brain).
Emotiv | EEG System | Brain Computer Interface Technology. Retrieved February 18th, 2014, from http://www.emotiv.com.
Sharbrough F, Chatrian G-E, Lesser RP, Lüders H, Nuwer M, and Picton TW. American Electroencephalographic Society Guidelines for Standard Electrode Position Nomenclature.
Journal of Clinical Neurophysiologyl, 1991, April 8:200-2.
Electroencephalography. Retrieved February 18th, 2014, from Electric and Magnetic Measurement of the Electric Activity of Neural Tissue: www.bem.fi/book/13/13.htm

21. Tools | Eye-tracking systems
These are instruments that measure eye position and eye movement in order to
detect zones in which the user has particular interest in a specific time and
moment.
Tobii Technology - Eye Tracking and Eye Control. Retrieved February 18th, 2014, from http://www.tobii.com.

22. Tools | Face-based emotion recognition systems
These systems infer affective states by capturing images of the users’ facial
expressions and head movements.
R. E. Kaliouby and P. Robinson, “Real-Time Inference of Complex Mental States from Facial Expressions and Head Gestures,” In Proceedings of the Real-Time
Vision for Human-Computer Interaction, In B. Kisačanin, V. Pavlović, and T.S. Huang (Eds.), IEEE Computer Society, 2005, pp 181-200. DOI: 10.1007/0-387-27890-7_11

23. Tools | Arousal or skin conductance sensing
Arousal detection. Measures the electrical conductance of the skin, which varies
with its moisture level that depends on the sweat glands, which are controlled by
the sympathetic and parasympathetic nervous systems.
M. Strauss, C. Reynolds, S. Hughes, K. Park, G. McDarby, and R.W. Picard, “The HandWave Bluetooth Skin Conductance Sensor,” In
Proceedings of First International Conference on Affective Computing and Intelligent Interaction (ACII 05), Springer-Verlang, October 2005,
pp 699-706, DOI:10.1007/11573548_90.

24. Tools | Pressure Sensing
Pressure sensors are able to detect the increasing amount of pressure (correlated
with levels of frustration) that the user puts on a mouse, or any other controller
(such as a game controller).
Y. Qi, and R. W. Picard, "Context-Sensitive Bayesian Classifiers and Application to Mouse Pressure Pattern Classification," In Proceedings of International Conference on Pattern
Recognition (ICPR 02), August 2002, Volume 3, pp 30448, DOI:10.1109/ICPR.2002.1047973.

25. Tools | Posture Sensing
Posture detection using a low-cost, low-resolution pressure sensitive seat cushion
and back pad.
S. Mota, and R. W. Picard, "Automated Posture Analysis for Detecting Learners Interest Level," In Proceedings of Computer Vision and Pattern Recognition Workshop
(CVPRW 03), IEEE Press, June 2003, Volume 5, pp 49, DOI:10.1109/CVPRW.2003.10047.

26. Tools | Summary
!"#$%&'(#)*%&$&+),*-&.+/$&+)0102*%)
30*,)
4,(/+5(6*0)
71*)%&6*%*+20)
8(-/(#)*9:,*00/&+0)
;<10/&#&./-(#)0/.+(#0)
=*+0/+.))
>*6/-*0)
;*,-*:$&+)
%*-<(+/0%0)
?+2*.,($&+)
@#.&,/2<%)
A(5)'(2() 4*#/*B0) =2(2*)

27. Methods

28. 4. Software Testing

29. Software Testing
It Provides stakeholders (business owners) with information about the
quality of the product, for instance:
1. Functionality. It meets the requirements that guided its design and
development.
2. Performance. It performs its functions within an acceptable time.
3. Robustness. It can be installed and run in its intended environments
with low or not risk of its failure.
4. Usability and Learnability. It is sufficiently usable and feasible to learn
its operation.

30. Software Testing | Usability
§ Cognitive science theories and affective computing tools are
applied for testing product interface and how the product interacts
with its users.
§ It helps to recognize things that can be improved.
§ For instance, UX on:
Web pages
Mobile applications
Tutor systems interfaces
Videogames

31. Software Testing | Examples
Visualization of emotions and fixation points for an expert Guitar Hero® player
playing in expert mode.
Engagement Frustration
Boredom

32. Software Testing | Examples
Visualization of emotions and gaze points for an undergraduate student reading
a screen with and without illustrations.
Boredom
Engagement Frustration

33. Software Testing | Examples
Office of N Research (ONR) damage Control Simulation. We measure the
emotions of reclutas while working in a damage control scenarios.

34. 5. Software Design

35. Software Design
Lets focus on two principles of software design:
• The design should minimize the intellectual distance between the
software and the problem as it exists in the real world. That is, the
structure of the software design should (whenever possible) mimic the
structure of the problem domain. For instance: for an Intelligent
Tutoring System, real-world student are cognitive and emotional
subjects.
• The design should be structured to accommodate change. A lot of
effort has been done accommodating to changes in computers
resources, networks capabilities, and error handling. But, what about
a changing cognitive and emotional user.

36. Software Design | Closed-Loop Model
Affective Adaptive Systems Architecture

37. Software Design | Goal
§ Cognitive science or affective computing well-know tools and
techniques added inside the new-generation software, make them
human-centered self-adaptive.
§ New generation software systems take advantage of sensing user
status in real time to change, improving system behavior
(functionality), and user experience in real-time.
§ For instance:
Affective Tutors that keep engagement and avoid frustration
Games that keep excitement and avoid boredom

38. Software Design | Example - Persuasive Game

39. Software Design | Example - Persuasive Game
Concept:
A 3D maze inside a cylinder that considers affective inputs: Excitement,
Meditation, and Engagement inferred from a BCI device
Goal:
Persuade the user to learn to control their emotional reactions.
Behavior:
§ Excitement will open a door
§ Engagement will trigger louder musical feedback.
§ Meditation will push back the darkness and improve your visibility
Implementation:
Undergrad students following the provided design model and
framework of tools; implemented it in 6 months.

40. Software Design | Example - Affective Pac-Man
!
Meditation
!
Engagement
!
Boredom
!
Frustration

41. Software Design | Example - Affective Pac-Man
Concept:
Modified version of Pac-Mac to add affective inputs: Meditation, Frustration,
Boredom, and Engagement.
Goal:
Improve the player experience accordingly with their emotional reactions.
Behavior:
§ speed of Pac-Man can increase or decrease
§ number of ghosts can increase or decrease
§ speed of the ghost can increase or decrease
§ music tempo can be faster or upbeat or slower or ballad
§ special features (such as fruits, power pellets, and 1-up component) can be
enabled or disabled
§ difficulty level of the next maze, which can increase or decrease
Implementation:
Undergrad students following the provided design model and framework of tools;
implemented it in 6 months.

42. Software Design | Example - Affective Tutor

43. Software Design | Example - Affective Tutor
Concept:
An Affective Tutor System that considers affective inputs to guide
interaction (messages) with the learner.
Goal:
Improve learning gains and deep modeling by providing affective
support
Behavior:
The combination of environment events (performance and modeling
behavior) and affective inputs trigger particular affective support
Implementation:
Undergrad and grad students following the provided design model and
framework of tools; implemented it in 1 year.

44. Software Design | ANGLE Lab
Our lab research related to Pattern Languages of Programing and
Software Architecture
Gonzalez-Sanchez, J., Chavez-Echeagaray, M.E., Atkinson, R., and Burleson, W. (2012).
Towards a Pattern Language for Affective Systems. Proceedings of the 19th Conference on
Pattern Languages of Programs (PLoP). Tucson, Arizona, USA. October 2012. ACM.
Gonzalez-Sanchez, J., Chavez-Echeagaray, M.E., Atkinson, R., and Burleson, W. (2011).
Affective Computing Meets Design Patterns: A Pattern-Based Model of a Multimodal Emotion
Recognition Framework. Proceedings of the 16th European Conference on Pattern
Languages of Programs (EuroPLoP). Irsee, Germany. July 2011. ACM, New York, NY, USA,
Article 14, 11 pages. ISBN: 978-1-4503-1302-5. doi=10.1145/2396716.2396730.
Gonzalez-Sanchez, J., Chavez-Echeagaray, M.E., Atkinson, R., and Burleson, W. (2011). ABE:
An Agent-Based Software Architecture for a Multimodal Emotion Recognition Framework.
Proceedings of the 9th Working IEEE/IFIP Conference on Software Architecture (WICSA).
Boulder, Colorado, USA. June 2011. IEEE, pp 187-193. ISBN: 978-1-61284-399-5. doi=10.1109/
WICSA.2011.32.

45. 6. Conclusion

46. Dr. Robert Atkinson
Assistant Professor
Director of the Advancing Next Generation Learning Environments Lab
angle.lab.asu.edu