VERDIKT conference 2013

1. 1
PERCEVAL - Perceptual and Cognitive
Quality Evaluation Techniques for
Audiovisual Systems
Ulrich Reiter
ulrich.reiter@ntnu.no

2. 2
Outline
•
•
•
•
•
•
•
PERCEVAL overview
Motivation (long duration)
Method description (long duration)
Experiment (long duration)
Results (long duration)
Conclusions
Food for thought
Ulrich Reiter – PERCEVAL

3. 3
Outline
•
•
•
•
•
•
•
PERCEVAL overview
Motivation (long duration)
Method description (long duration)
Experiment (long duration)
Results (long duration)
Conclusions
Food for thought
Ulrich Reiter – PERCEVAL

4. 4
Multimedia systems rely on
compression and encoding to speed
up transmission
Compression reduces the file
size by removing redundant
or unessential information
As a result, media quality goes
down and the audio and video
streams can become
misaligned
Ulrich Reiter – PERCEVAL

5. 5
Perceptual consequences of distortion:
-
-
Face identification
Scene/object recognition
Speech comprehension
Audiovisual speech
integration
Temporal order of events
Audiovisual temporal
integration?
Long duration quality
perception?
Ulrich Reiter – PERCEVAL

6. 6
PERCEVAL project
Ulrich Reiter – PERCEVAL

7. 7
Outline
•
•
•
•
•
•
•
PERCEVAL overview
Motivation (long duration)
Method description (long duration)
Experiment (long duration)
Results (long duration)
Conclusions
Food for thought
Ulrich Reiter – PERCEVAL

8. 8
Questions to be answered
Do quality expectations decrease over time and
with increased involvement in the content?
How fast can users notice quality changes and at
which quality level does this happen?
Is the quality level at which the change is noticed
similar to the desired quality level?
Ulrich Reiter – PERCEVAL

9. 9
Quality assessment methods for
long duration AV content
Ulrich Reiter – PERCEVAL

10. 10
SSCQE
(Single Stimulus Continuous Quality Evaluation)
Ulrich Reiter – PERCEVAL

11. 11
Outline
•
•
•
•
•
•
•
PERCEVAL overview
Motivation (long duration)
Method description (long duration)
Experiment (long duration)
Results (long duration)
Conclusions
Food for thought
Ulrich Reiter – PERCEVAL

12. 12
Method description
Quality adjustment
Ulrich Reiter – PERCEVAL

13. 13
Outline
•
•
•
•
•
•
•
PERCEVAL overview
Motivation (long duration)
Method description (long duration)
Experiment (long duration)
Results (long duration)
Conclusions
Food for thought
Ulrich Reiter – PERCEVAL

14. 14
Experimental conditions
Ulrich Reiter – PERCEVAL

15. 15
Stimulus material
30min 55sec long
HD resolution (1080p)
H.264/AVC compression
– Quantization parameter
11 versions, JND
Conversion to 16 bit
YUV 4:2:0
Original audio track
Ulrich Reiter – PERCEVAL

16. 16
Experimental procedure
Participants
22 subjects (8 female, 14 male) in the age of 24-61 years
Test procedure and duration
Quality degradation initiated every 3 mins
One JND drop every 10 sec
Ulrich Reiter – PERCEVAL

17. 17
Outline
•
•
•
•
•
•
•
PERCEVAL overview
Motivation (long duration)
Method description (long duration)
Experiment (long duration)
Results (long duration)
Conclusions
Food for thought
Ulrich Reiter – PERCEVAL

18. 18
Questions to be answered
Do the quality expectations decrease over time
and with increased involvement in the content?
How fast can participants notice quality changes
and at which quality level does this happen?
Is the quality level at which the change is noticed
similar to the quality level set by the participant?
Ulrich Reiter – PERCEVAL

19. 19
Data preparation
Three subsets of data (20x10 each)
created:
1) average quality level of the last minute of each 3min
time slot
2) response time to the automatic quality degradation
right after the start of each 3 min time slot
3) quality level at the time when a user reacted to
quality change
Ulrich Reiter – PERCEVAL

20. 20
Results for subset data 1)
Main effects plot for quality levels averaged over last
minute of each 3 min time section.
Ulrich Reiter – PERCEVAL

21. 21
Results for subset data 2)
Main effects plot for reaction time to quality changes.
Ulrich Reiter – PERCEVAL

22. 22
Results for subset data 3)
Main effects plot for quality levels at time when a user
reacted to quality change
Ulrich Reiter – PERCEVAL

23. 23
Results cont.
QLRT: Quality Level
at Reaction Time
AQL: Absolute
Quality Level
Comparison of sensitivity to quality changes under
different conditions.
Ulrich Reiter – PERCEVAL

24. 24
Outline
•
•
•
•
•
•
•
PERCEVAL overview
Motivation (long duration)
Method description (long duration)
Experiment (long duration)
Results (long duration)
Conclusions
Food for thought
Ulrich Reiter – PERCEVAL

25. 25
Conclusions
Time dimension does not influence participants’
expectations with respect to the perceived quality.
The reaction time to quality changes is relatively constant
across all time intervals.
Participants are less sensitive to quality changes when
the process is controlled externally than when they are in
charge of the quality adjustment.
Ulrich Reiter – PERCEVAL

26. 26
Outline
•
•
•
•
•
•
•
PERCEVAL overview
Motivation (long duration)
Method description (long duration)
Experiment (long duration)
Results (long duration)
Conclusions
Food for thought
Ulrich Reiter – PERCEVAL

27. 27
Food for thought
• Content dependency
– Ideal bandwidth trade-off for audio vs. video quality depends
on content type
• Impact of emotional state, fatigue
– On perceived quality, on quality requirements / expectations
– Use of EEG and body sensors
• Data analysis of non-traditional data
– E.g. quantitative and qualitative, combined
– Multivariate statistical methods
Ulrich Reiter – PERCEVAL

28. 28
Thanks for your attention!
ulrich.reiter@ntnu.no
Ulrich Reiter – PERCEVAL

29. 29
References
[1] http://perceval.no
[2] ITU-R Recommendation BT.500-7: Methodology for the Subjective Assessment of the Quality
of Television Pictures. International Telecommunication Union, Geneva (1996)
[3] N. Staelens, S. Moens, W. Van den Broeck, I. Marien, B. Vermuelen, P. Lambert, R. Van de
Walle and P. Demeester, “Assessing Quality of Experience of IPTV and Video on Demand
Services in Real-Life Environments”, IEEE Transactions on Broadcasting, Vol.56, No.4, Dec.
2010.
[4] Borowiak, A., Reiter, U., Svensson, U.P.: Quality Evaluation of Long Duration Audiovisual
Content. In: The 9th Annual IEEE Consumer Communications and Networking Conference –
Special Session on Quality of Experience (QoE) for Multimedia Communications, pp. 353--357,
Las Vegas (2012)
[5] Yang, X., Tan, Y., Ling, N.: Rate control for H.264 with two-step quantization parameter
determination but single-pass encoding. In: EURASIP Journal on Applied Signal Processing, pp.
1--13, (2006)
[6] Wang, H., Qian, X., Liu, G.: Inter Mode Decision Based on Just Noticeable Difference Profile.
In: Proceedings of 2010 IEEE 17th International Conference on Image Processing, Hong Kong
(2010)
Thanks to Adam Borowiak and Ragnhild Eg for providing figures and slides!
Ulrich Reiter – PERCEVAL