This document provides recommendations for subjectively evaluating sensory experiences. It discusses different test setups and stimulus methods, including single stimulus and double stimulus studies. Single stimulus continuous quality evaluation was found to stress participants and an indirect voting device is recommended. Double stimulus studies like degradation category rating and double stimulus continuous quality scale were found to be more suitable for assessing sensory enhancements. Combining stimulus methods requires caution to avoid biases.
2. TABLE OF CONTENTS
Introduction
Motivation
Test Setup
Single Stimulus Studies
SSCQE + ACR
Double Stimulus Studies
DCR
DSCQS + single stimulus
3. INTRODUCTION
• Multimedia enriched by sensory effects like wind,
vibration and light
For quality degradations by lowering the bitrate of the actual video
sequence
For genres like action, documentary, commercial, and sports
Intensity of emotions are influenced
• Passive emotions are decreased
• Active emotions are increased
4. MOTIVATION
How has the test setup and design to be, for subjectively
assessing the QoE of sensory experience?
Which stimulus methods are useful for subjectively
assessing sensory experience?
Single stimulus?
Double stimulus?
Does mixing up single stimulus and double stimulus end
up in a bias when assessing sensory experience?
5. TEST CONDITIONS, TEST SETUP, AND
TEST DESGIN
In-Lab assessments
Setup according to ITU-R BT.500-11/13
Ergonomic placement of keyboard, mouse and seat
Test setup
6. TEST CONDITIONS, TEST SETUP, AND
TEST DESGIN
Textual introduction
Each participant is presented the same introduction
Allows for training phases
Includes demographic questions aswell as the possiblity for the
participants to give feedback
Supports different test methods
7. SINGLE STIMULUS EVALUATION -
SSCQE + ACR
Single Stimulus Continuous Quality Evaluation
with categorical rating scale
Combined with an Absolute Category Rating (ACR)
8. SINGLE STIMULUS EVALUATION -
SSCQE + ACR
24 Participants
11 female
13 male
Assess whether sensory effects unmask quality degradations
Quality in terms of bitrate
Dedicated voting device
Stressed the participants for the SSCQE part
No intiuitive handling of the voting device
More intuitive voting devices should be used
• Indirect devices e.g., EEG
Assessing Sensory Experience with SSCQE is not recommended
10. DOUBLE STIMULUS EVALUATIONS -
DCR
Degradation Category Rating
Double stimulus with a categorical scale
• veryannoying (1), annoying (2), imperceptible (3),
littleenhancement (4) andbigenhancement (5)
Assess the enhancement of sensory effects
No randomization of reference and test condition
Reference can be easily identified (no effects)
25 Participants
12 female
13 male
12. DOUBLE STIMULUS EVALUATIONS -
DSCQS
Double Stimulus Continuous Quality Scale
Modified by combining it with a single stimulus
• After each stimulus presentation emotion rating
• Double stimulus for the enhancement of QoE
The rating scale for emotions and the
enhancement in QoE was on an interval of
[0,100]
13. DOUBLE STIMULUS EVALUATIONS -
DSCQS
In total 68 participants
36 female
32 male
Conducted at three sites
AAU (Austria)
UOW (Australia)
RMIT (Australia)
For identyfing a bias of the combined test method studies from UOW
and RMIT were used
15. CONCLUSIONS
Single stimulus method with categorical rating
scales are suitable for assessing the sensory
experience
SSCQE may stress the participants, the voting
device plays an important role
Even with an intuitive voting device particpants may be
stressed due to effects and the task of the continuous
voting
Indirect voting devices such as EEG should be used
16. CONCLUSIONS
The DSCQS was modified to fit the needs of an
subjective quality evaluation in the domain of
sensory experience
The DCR method was used as defined, only the
rating scale had to be changed slightly (from
impairment to enhancement)
When combining stimulus methods caution is
advised that no bias is introduced