The document summarizes a study investigating how different sensory modalities (visual, auditory) impact immersion in digital games. Participants played a game where they dodged virtual cars using motion controls. Sessions included visual-only, audio-only, and audio-visual feedback conditions. Questionnaires found audio-visual provided the highest immersion. Performance was best with audio-visual and worse with audio-only. Most participants preferred audio-visual and found visual-only muted and audio-only intense. The study suggests achieving immersion requires relatable scenarios and natural inputs, while removing primary senses leads to more intense experiences.

1. Designing Games for All: Exploring
Output and Immersion.
Investigating the impact of modality perception
on immersion in digital games

2. The Senses
Five Primary Sensory Modalities
• Vision
• Auditory
• Somatosensory (Touch and Proprioception)
• Olfactory (Sense of Smell)
• Gustatory (Sense of Taste)

3. Multimodal Interaction
•
•
Human Interaction and Communication is inherently
multimodal
– An everyday conversation includes more than just
auditory cues…
– A touch can convey a whole idea…
Human Computer Interaction remains limited
– Multimodal Input is common.
• Voice, Touch, Gaze, Motion, Gesture, BMI, etc.
– Output tends to be limited
• Visual, Auditory and occasionally haptic (vibration)

4. Immersion
•
•
•
Subjective by nature
Difficult to achieve and easy to destroy
Engagement is a vital ingredient
– Mental engagement is intensely subjective but promotes
immersion very effectively.
– Physical engagement is easier to achieve and directly
promotes immersion.

5. The importance of immersion
•
•
•
Immersion is an acknowledged ingredient of human crafted
experiences
– Books, Films, Theatre, Games…
The majority of these constructs are limited by their medium
with regards to their form and function
Even when technology comes into play, the extents are
often limited
– 3D Films, Force Feedback

6. The Project
Investigates the role of two sensory modalities, specifically
visual and auditory perception, in achieving effective
immersion and presence.
• Establishing an experimental design to provide balanced and
comparable sets of modal cues.
Do so with natural interfaces for low physical engagement
barriers.
• Performing a user study that would evaluate the modalities
both in isolation and in combination in order to study the
impact of each modality.

7. The Experiment
What did we do?
• We made participants dodge cars.
• Sometimes while blindfolded.

10. The Setup
•
•
•
Simulated a collision avoidance scenario
– Virtual Cars on a highway
– Mental Engagement
Multimodal Input
– Motion sensing technology (Kinect)
– High Physical Engagement
Multimodal Output
– Visual and Auditory Feedback

15. The Specifics
•
•
The Participants where asked to dodge the incoming cars for
as long as possible.
Nine measured sessions in total
– 3 With both visual and auditory feedback
– 3 With only visual feedback
– 3 With only auditory feedback

16. Data Collection and Evaluation
Primary desired data element:
• Qualitative Immersion Measures
– Method: IMEX questionnaire
Secondary data element
• Quantitative performance data
– Participant survival time per session

17. Quantitative Results - Immersion
Mean IMX score on 0-100 scale
90
The analysis showed that the modal output had a
significant impact on immersion.
80
70
60
50
Both Modalities
40
Audio Only
30
Visual Only
20
10
0
General
Immersion
Action Visceral Mental Visceral

18. Quantitative Results Performance and Preference
•
•
Performance was measured by the survival time of each
participant in each play through
– Insignificant difference between Audio-Visual and Visualonly sessions
– Significant difference between Audio-Visual and Audioonly sessions – as was expected
With regards to preference
– 75% preferred the Audio-Visual version
– 25% preferred the Audio-only version
– Noce preferred the visual only version

19. Qualitative Results & User Feedback
Overall, feedback from the users shows promising results:
• Positive reactions to the experience
• High levels of Immersion
• Interesting responses regarding value of audio
– A/V is optimal
– Visual only is “muted” and “disconnected”
– Audio only is “intense” and “thrilling”

20. Qualitative Results & User Feedback
Three themes surfaced:
• Rapid Immersion
– Physical and Mental engagement equals rapid immersion
– “Damn this is intense!”
• The importance of sound
– Removing a “secondary” modality has an impact
– “Something was missing, I was waiting for the crunch.”
• Engaging audio experience
– Removing a “primary” modality has a severe impact
– “That was absolutely terrifying! I was all there!”

21. Conclusions
•
The Starting point
– The objective: Design and implement a highly immersive
digital experience
– The hypothesis: Unimodal and multimodal experiences
create different levels of immersive response.
•
The findings
– Achieving Immersion: Relatable scenarios and natural
input greatly promote immersion
– Unimodal Output: Removing primary senses leads to
intense experiences

22. Potential Application
•
•
•
Enhanced Entertainment Experiences
Games Designed for All
– Vision Impairment
• AudioGames.net
– Dexterity/Mobility Impairment
• OneSwitch.org.uk
Games Designed for All vs. Accessible Games
http://www.gameaccessibilityguidelines.com

23. The next step
•
•
•
•
Contextual modality suppression
– Scenario based sensory deprivation
Further immersion enablers
– Oculus rift
Introduce additional output modalities
– Haptic feedback
Investigate immersive scenarios

24. Thanks for listening!