In recent years, the interest to the area of computer games for educational purposes increased due to their positive outcomes and effects in technology-enhanced learning. One of their chief merits is retaining the learning motivation and engagement of players during all time of the game. Therefore, it is necessary educational games to be able to adjust their features such as task difficulty, object speed, learning content, etc. according to the current emotional state of the player and, as well, to his/her playing style. In this paper, we present a dynamic mechanism for affective game adaptation based on both emotion and arousal estimation. The mechanism is implemented within an applied video game named “Rush for Gold” designed for implicit recognition of playing or learning styles. The paper outlines, analyzes and discusses results of an experimental study related to player´s engagement in affective applied adaptation.

1. Assessing Engagement
in an Emotionally-Adaptive
Applied Game
Boyan Bontchev and Dessislava Vassileva
Dep. of Software Technologies,
Fac. of Mathematics and Informatics, Sofia Univ., Bulgaria
TEEM-2016
Salamanca, November 2, 2016

2. Player-centric adaptation
Player-centric adaptation in video games needs to
answer some questions:
Who – player character structural and behavioral
changes => requires player modelling
How – how adaptation will be realized => requires
adaptive loop design
What – which game features can be adapted =>
requires adaptive gameplay design
Why – advantages of game adaptation => requires
analysis of outcomes in playability and learning results
02/11/2016Assessing Engagement in an Emotionally-Adaptive Applied Game
2

3. The ADAPTIMES FP7 project
 ADAPTIve player-centric serious video gaMES
 ADAPTIMES aims at investigating how
cognitive abilities (the cognitive part),
psycho-emotional status (the affective part) and
playing style (the conative part – that of “desire, volition, and
striving”)
 can be used in a
holistic approach
for efficient and
effective
player-centric
adaptation
02/11/2016Assessing Engagement in an Emotionally-Adaptive Applied Game
3

4. Adaptation Model
02/11/2016Assessing Engagement in an Emotionally-Adaptive Applied Game
4
Game dynamics
Player
Player style metrics
Playing styles
Performance
and efficiency
(abilities, skills
and knowledge)
Emotions
(flow, immersion &
intrinsic
motivation)

5. Measuring process
Measuring player’s performance – by tracking player’s
results, incl. time, task difficulties and efficiency
Measuring playing styles – based on tracking specific
gaming metrics describing player’s style of playing
Measuring player’s affect (emotions and arousal):
By means of analysis of psychophysiology signals
(arousal)
By means of visual expressions analysis (emotional
states)
02/11/2016Assessing Engagement in an Emotionally-Adaptive Applied Game
5

6. Affect inferred by psychophysiology signals
EDA (Electro-Dermal Activity)
Custom EDA meter – custom hardware device and
software (H2020 RAGE asset developed by SU)
02/11/2016Assessing Engagement in an Emotionally-Adaptive Applied Game
6

7. Emotions inferred by visual expressions analysis
Custom emotion recognizer using Affectiva SDK
Recognizes: sadness, disgust, anger, surprise, fear, joy;
engagement, attention (fixation), eye closures (saccades)
02/11/2016Assessing Engagement in an Emotionally-Adaptive Applied Game
7

8. Workflow of emotion-based
game dynamic adaptation control
02/11/2016Assessing Engagement in an Emotionally-Adaptive Applied Game
8
Implicit
Emotional
Indicators
(via sensors &
other devices)
Facial expr.
GSR
Heart Rhythm
ECG
EEG
Pressure
…
Change
Procedures
Time
Velocity
Acceleration
Route
Size
Defeat range
Luminosity
Contrast
…
Real-Time
Explicit
Player
Feedback
through
Adaptation
Control Panel
Analysis
Change
Visuali-
zation
Triggering
Events
on Behavior
Templates

9. Emotional Game Adaptation 1/2
02/11/2016Assessing Engagement in an Emotionally-Adaptive Applied Game
9
Adaptation of flying objects’ velocity, puzzle difficulty, and
ambient light for lower phasic arousal

10. Emotional Game Adaptation 2/2
02/11/2016Assessing Engagement in an Emotionally-Adaptive Applied Game
10
Adaptation of flying objects’ velocity, puzzle difficulty, and
ambient light for higher phasic arousal

11. Box plots of player’s average
emotions inferred by facial
expressions
02/11/2016Assessing Engagement in an Emotionally-Adaptive Applied Game
11

12. Box plots of measured GEngQ terms
02/11/2016Assessing Engagement in an Emotionally-Adaptive Applied Game
12
GEngQ = Game Engagement Questionnaire (Brockmyer et al., 2009)

13. Player´s joy, engagement, attention, eye
closure, and SCR (skin conductance response)
02/11/2016Assessing Engagement in an Emotionally-Adaptive Applied Game
13
Successful shot Unsuccessful shot

14. Correlations of psychological states (by facial
expressions) and arousal with GEngQ terms
02/11/2016Assessing Engagement in an Emotionally-Adaptive Applied Game
14
Game Engagement Questionnaire (GEngQ)
State /
arousal
Presen-
ce
Absorp-
tion
Flow
Immer-
sion
Engage-
ment
Atten-
tion
0.3878 0.0645 0.1779 0.0139 0.1978
Enga-
gement
0.0087 -0.0510 0.1870 -0.0532 0.0221
Eye
closure
0.3818 0.0885 0.3711 0.1737 0.1554
SCL
level
0.1341 -0.1312 0.0459 -0.1067 -0.0318
SCR
level
0.0119 -0.0558 -0.2177 0.1042 -0.0674

15. Correlations of psychological states and arousal
with emotions inferred by facial expressions
02/11/2016Assessing Engagement in an Emotionally-Adaptive Applied Game
15
Emotions
State /
arousal
Sad-
ness
Disgust Anger
Surp-
rise
Fear Joy
Atten-
tion
-0.0226 0.3712 0.2329 0.1835 -0.1677 0.2040
Enga-
gement
0.0782 0.7049 0.1653 0.4296 0.1013 0.8346
Eye
closure
-0.1257 0.2293 0.0918 0.4657 0.1713 0.3196
SCL
level
0.0226 -0.1484 -0.0759 0.3873 0.3007 -0.0027
SCR
level
-0.0275 0.3616 -0.0307 0.1285 -0.0959 0.4008

16. Conclusions 1/2
Statistically significant correlations found:
attention and presence
eye closure and presence
eye closure and flow
engagement and surprise
engagement and joy
SCL arousal level and surprise
SCR and joy
eye closure & average efficiency
SCL & relative time
Facial expressions (emotions) are complementary to
EDA (arousal)
16
02/11/2016
Assessing Engagement in an Emotionally-Adaptive Applied Game

17. Conclusions 2/2
Player-centric adaptive game play possesses essential
advantages compared to the non-adaptive gameplay
The synergy of using player´s performance, emotions,
and playing styles is very promising
All they are identified at run time and used for adapting
game mechanics, dynamics and audio-visual content, in
a dynamic and implicit way
More experiments are needed for revealing how such
emotion-based adaptations affect game experience
and playability, learning outcomes and consequences
of flow
17
02/11/2016
Assessing Engagement in an Emotionally-Adaptive Applied Game

18. Future work
Emotionally-adaptive games and other software app’s
Dynamic adaptation control
Software app’s using user affectation:
HRV
TEMP
….
Web-based platforms for generation of educational
games targeted to non-ICT people
Semantic content management in applied games
02/11/2016Assessing Engagement in an Emotionally-Adaptive Applied Game
18

19. Thank you for your attention!
Discussion
Videos at:
https://www.youtube.com/watch?v=rSzM2RrceNw&index=1&li
st=PLs4ZK9XAjRxpROiCNQ1kdDR8SFpH-itmY
More info at: http://adaptimes.eu/
19
02/11/2016
Assessing Engagement in an Emotionally-Adaptive Applied Game