ConférenSquad #3 : Subjective Quality and HTTP Adaptive Streaming: a Review of Psychophysical Studies (Frédéric Dufaux, Telecom Paris Tech)

Institut Mines-Télécom
Subjective quality and HTTP adaptive
streaming: a review of psychophysical studies
Francesca De Simone, Frédéric Dufaux
Institut Mines-Télécom; Télécom ParisTech; CNRS LTCI

Content
Basic concepts
• Quality of Service (QoS) vs Quality of Experience (QoE)
• Subjective vs objective assessment
• QoE and DASH
Subjective QoE and DASH: what do we know
• Perception of stalling events
• Perception of quality switches
• Tradeoffs
Conclusions
2

Basic Concepts
3

QoS vs QoE
Example of Quality of Service (QoS) metrics
• Available bandwidth
• Round Trip Time (RTT)
• Packet loss rate (PLR)
• Etc..
Quality of Experience (QoE)?
4

QoS vs QoE
Example of Quality of Service (QoS) metrics
• Available bandwidth
• Round Trip Time (RTT)
• Packet loss rate (PLR)
• Etc..
Quality of Experience (QoE)? [Qualinet WhitePaper 2012]
5
multimedia experiencemultimedia experience
MULTIMEDIA
SERVICE
Visual
signal
Price
Novelty
Interface
CONTEXT
etc…
Audio
signal
etc…USER
Human Visual
System
Expectations
Emotions
Auditory
System
Attention
etc…

Subjective vs objective assessment
Subjective quality assessment
In controlled environment
• standard methodologies for video
designed for analog television [ITU
Rec BT.500-11]
• adapted for multimedia applications
[ITU Rec. P.910]
In uncontrolled environment
• crowdsourcing
• field experiments
• users’ feedback from real products
6

Subjective vs objective assessment
Objective quality assessment
• Full-Reference (FR)
• Reduced-Reference (RR)
• No-Reference (NR)
7
Input/Reference
signal
Output/Processed signalImage/ video
processing
FR METRIC
Input/Reference
signal
processing NR METRIC
Input/Reference
signal
processing
Feature
extraction RR METRIC
Example of standardized solutions
• FR video quality metric for HDTV for digital cable television [ITU-T
Rec. J.341]
• NR model for audiovisual quality assessment in IPTV [ITU-T Rec.
P.1201.2]

QoE and DASH
HTTP server TCP transport layer DASH client
128 kbps
500kbps
1Mbps
• Terminology
0 2 4 6 8 10 12 14 time (s)
Realization
Realization
Realization
Chunk Adaptation Set
128 kbps
500kbps
1Mbps

QoE and DASH
No packet losses
• Reliable transport layer
Stalling events
• Playout interruptions
due to video player
buffer underflow
Quality switches
• Quality variation due to
switch between
representations
9

QoE and HAS
No packet losses
• Reliable transport layer
Stalling events
• Playout interruptions
due to video player
buffer underflow
Quality switches
• Quality variation due to
switch between
representations
10
Not addressed in traditional
video QoE research!

Goal of this talk
11
Identify key questions
concerning subjective video
quality perception and DASH
Review existing answers
• Perception of video stalling
• Perception of video quality switches
• Tradeoffs

Subjective QoE and DASH:
what do we know
12

Perception of stalling events
13
Stallings should be avoided at all times
[Dobrian2011, Mok2011, Floris2012,
Atzori2013, Hobfeld2013] but if they
occur…
“what is the impact of stalling frequency and
duration?”

14
Stallings should be avoided at all times
[Dobrian2011, Mok2011, Floris2012,
Atzori2013, Hobfeld2013] but if they
occur…
“what is the impact of stalling frequency and
duration?”
• Studies before DASH [Pastrana-Vidal2004, Staelens2010]
• Studies on DASH [Moorthy2011, vanKester2011]

Conclusions
• Threshold to detect stalling: 80ms [Pastrana-Vidal2004]
• Subjects prefer a single stalling of longer duration compared
to multiple short stallings [Moorthy2011]
• Subjects prefer regular stallings over irregular ones
[Moorthy2011]
• Stallings up to 400ms can still be tolerated in the case of few
stalling events and long video sequences [vanKester2011]
[Staelens2010]
15

Perception of quality switches
16
Assumptions
• Each representation corresponds to a
different bitrate
• Representations may differ in terms of
spatial resolution, temporal resolution and
encoding quantization settings
• Three kinds of switches are possible
• Encoding switch (ESW)
• Spatial switch (SSW)
• Temporal switch (TSW)
128 kbps
500kbps
1Mbps

17
“What is the impact of switching amplitude and
frequency?”

18
frequency?”
Question 1: “shall the bitrate
be reduced (increased) once
with a big quality drop or
gradually?”
time
Layer
or
Rate
time
Layer
or
Rate

19
– Studies on layer encoded videos and SVC [Zink 2003, Ni 2011]
– Studies on HAS [Moorthy 2012, Mok 2012, Yitong 2013]
frequency?”
Question 1: “shall the bitrate
be reduced (increased) once
with a big quality drop or
gradually?”
time
Layer
or
Rate
time
Layer
or
Rate

20
Conclusions
• Smooth variations preferred to abrupt variations
• Amplitude of the variation must be kept as small as possible
• Examples of amplitude thresholds to deliver generally
acceptable quality [Ni 2011]
• Quality differences should not exceed 4 QPs for ESW, one third of
the original frame rate for TSW, half the original size for SSW

21
But…
• Conclusion might not be generalizable to switches between
levels exhibiting small quality separation [Moorthy 2012]
Conclusions
• Smooth variations preferred to abrupt variations
• Amplitude of the variation must be kept as small as possible
• Examples of amplitude thresholds to deliver generally
acceptable quality [Ni 2011]
• Quality differences should not exceed 4 QPs for ESW, one third of
the original frame rate for TSW, half the original size for SSW

22
Question 2: “Do users prefer shorter but more
frequent variation or longer but less frequent
variations?”

23
• [Zink 2003, Ni 2011] layer
encoded videos and SVC
• [Moorthy 2012, Robinson
2012, Yitong 2013]
Question 2: “Do users prefer shorter but more
frequent variation or longer but less frequent
variations?”
time
Layer
or
Rate
time
Layer
or
Rate

Institut Mines-Télécom24
Conclusions
• Less frequent quality variations are preferred to more frequent
variations
• Frequency of variations should be kept as small as possible
• For SNR or spatial resolution switches, low frequency can relieve
the annoyance of quality switch [Ni 2011]

Institut Mines-Télécom25
Conclusions
• Less frequent quality variations are preferred to more
frequent variations
• Frequency of variations should be kept as small as
possible
• For SNR or spatial resolution switches, low frequency can
relieve the annoyance of quality switch [Ni 2011]
But…
• More frequent switches are preferred over fewer
switches, if the subject is able to view the highest quality
video for longer duration [Moorthy 2012]
• For temporal resolution switches, the frequency does not
seem to have significant influence on QoE [Ni 2011]

26
Question 3: “is it better to switch quality levels or to
stay at a lower quality? i.e. to switch or not to switch?”

27
Question 3: “is it better to switch quality levels or to
stay at a lower quality? i.e. to switch or not to switch?”
• [Zink 2003, Ni 2011] layer
encoded videos and SVC
• [Moorthy 2012, Robinson
2012, Mok 2012, Yitong
2013, Villa 2013, Rehman
2013, Tavakoli 2014]
time
Layer
or
Rate
time
Layer
or
Rate

28
Conclusions
• Constant quality preferred to varying quality
• Short-term spikes degrade QoE [Yitong 2013]
• Constant (lower) quality better than decreasing quality (from higher
to lower) [Zink 2003]
• Constant or nearly constant quality is preferable to frequently
varying quality (even if mean quality is lower) [Robinson 2012] [Ni
2011]
• Providing as high bitrate as possible does not necessarily lead to
the highest QoE [Mok 2012]

29
But…
• Preferable to switch to a higher rate, if the duration of the
higher rate is at least half the duration of the lower rates
[Moorthy 2012]
• If constant quality is too low, any adaptation is preferred
[Tavakoli 2014] [Ni 2011]
• Maintaining a reasonable quality for longer duration
results in a small bias towards better subjective
experience [Rehman 2013]

30
Temporal trend
• The end quality of the video makes a definite impact on the perceived
quality: all is well that ends well [Zink 2003][Moorthy 2012]
• [TavakoliSPIE2014] For increasing quality, overall results did not show
any statistically significantly difference between different strategies.
For the decreasing scenarios, there was a statistically significantly
preference to gradual change with 10 seconds chunk compared to the
other scenarios.
Content dependency
• [NiMMsys2011] [RobinsonBellTJ2012] Effect of switch of spatial and
temporal resolution varies on different content types, even for the
same amplitude
• difficult to spot quality oscillations when frequent scene changes in the
scene.
• Quality change is more noticeable in steady, pan, tracking shots and when
there are number of hard edges.
• More complex texture details , more strongly affected by the loss of spatial
resolution
• The lack of relative movement between objects in the scene may limit the
visible effect of frame dropping.

When the bitrate has to be lowered (increased), for
instance due to restricted (better) network
conditions, …
Question 1: “is it better to reduce (increase)
encoded bitrate, frame rate, or resolution?”
31

32
When the bitrate has to be lowered (increased), for
instance due to restricted bandwidth (better network
conditions), …
Question 1: “is it better to reduce (increase) the
encoded bitrate, the video frame rate, or the video
resolution?”
H.264/SVC encoded versions at same bitrate
1/16 of the original
resolution upscaledStronger quantization
• Many subjective
studies on optimal
combinations (SNR,
spatial resolution,
temporal resolution)
for scalable video
coding [Lee 2012]

33
Conclusions
• Spatial quality more important
than frame rate at low bit rates
and viceversa at high bit rates
• Low vs high bit rate threshold
higher for content with faster
motion
• Among quality switch of SNR levels, spatial resolutions or
temporal resolutions, SNR switch is the most recommended
as less noticeable [Ni 2011]

34
Methodological findings:
• [RehmanQomex2013] Subjects are resistent in updating their
opinions: when there is a small quality variation between consecutive
scenes, subjects tend to keep their opinions or change only slightly.
• [RehmanQomex2013] [ChenIP2014] Subjects use asymmetric
strategies in updating their opinions. A significant quality degradation
between consecutive scenes results in a large penalty, as compared to
the reward obtained by a similar quality improvement between
consecutive scenes. Behavioral response of viewers to quality
variation is more sensitive in low quality region than in high quality
region.
• [ChenIP2014] The QoE of a video at a particular moment depends on
the viewing experience before the moment: the current video quality
can affect the perceived quality in the next 15 second
• [ChenIP2014] A viewer suffering a previous unpleasant viewing
experience tends to penalize the perceived quality in the future
• [VillaNIK2013] Significant differences in terms of self-reported
pleasure, arousal, and the degree of delight between the two usage
scenarios (user alone vs group of users). All significantly higher in the
first usage scenario.

Tradeoffs: initial delay vs stalling
35
Stallings can be avoided by employing larger
client buffers but this will affect the initial startup
delay of the video, so…
Question 2: “what is the impact of initial loading?”
Question 3: “what is the impact of combined effect
(stalling + initial loading)?”

36
Stallings can be avoided by employing larger
client buffers but this will affect the initial startup
delay of the video, so…
Question 2: “what is the impact of initial loading?”
Question 3: “what is the impact of combined effect
(stalling + initial loading)?”
[Hobfeld2013, HoBfeldQomex2012, Staelens2010]

Conclusions
• Initial startup delay does not significantly influence
quality perception [Hoßfeld 2013]
• End-users are willing to tolerate larger startup delays,
if this results in less video stallings [Staelens 2010]
• Different amounts of startup delay are tolerated,
depending on the specific type of application
[HoBfeld 2012]
37

Conclusions
38

Conclusions
Agreed upon conclusions:
• Frequency of stalling is the main factor affecting QoE
• The amplitude of the quality switch is the most
dominant factor for the perception of quality switch
(more than the frequency)
Issue:
• How generalizable are the findings of these studies?
39

Challenges
40
Why is it difficult to design a
subjective test for HAS?
• Temporal variation of quality
─ Longer sequences may be needed: which
test method to use?
• Variety of terminals in the real scenario
─ Different network conditions
─ Different users’ expectations

Thank You for your
Attention!
francesca.de-simone@telecom-paristech.fr
frederic.dufaux@telecom-paristech.fr
Thanks to:
Marie-Neige Garcia and Alexander Raake (T-Labs TU
Berlin), Nicolas Staelens (Ghent University), Samira
Tavakoli (Universidad Politecnica de Madrid), Sebastian
Egger (ftw), Kjell Brunnstrom (Acreo)
41

Further reading and resources…
1. Qualinet White Paper on Definitions of Quality of Experience (2012), European Network on Quality of Experience in
Multimedia Systems and Services (COST Action IC 1003), Patrick Le Callet, Sebastian Möller and Andrew Perkis, eds.,
Lausanne, Switzerland, Version 1.1, June 3, 2012.
2. Zink et al., Subjective impression of variations in layer encoded videos, ACM 2003
3. R.R. Pastrana-Vidal, J.C. Gicquel, C. Colomes, and H. Cherifi, “Sporadic frame dropping impact on quality perception,” in
Proceedings of SPIE, 2004, vol. 5292, p. 182
4. Q. Huynh-Thu and M. Ghanbari, “Temporal Aspect of Perceived Quality in Mobile Video Broadcasting”, IEEE Transactions on
Broadcasting, vol.54, no.3, pp.641-651, Sept. 2008
5. J.-S. Lee, F. De Simone, N. Ramzan, E. Izquierdo and T. Ebrahimi, Quality assessment of multidimensional video scalability,
in IEEE Communications Magazine, vol. 50, num. 4, p. 38-46, 2012
6. J.-S. Lee, F. De Simone and T. Ebrahimi, Subjective quality evaluation via paired comparison: application to scalable video
coding, in IEEE Transactions on Multimedia, vol. 13, num. 5, p. 882-893, 2011
7. Venkataraman et al., Effects of Internet path selection on video QoE, MMSys 2011
8. Chen et al., Quantifying QoS requirements of network services: a cheat-proof framework, MMSys 2011
9. R. Mok, E. Chan, and R. Chang. Measuring the quality of experience of HTTP video streaming. In Proc. IEEE/IFIP
10. IM (pre-conf.), 2011
11. R. Mok, E. Chan, X. Luo, and R. Chang., Inferring the QoE of HTTP Video Streaming from User-Viewing Activities, ACM W-
MUST’11, 2011
12. Zhou Wang; Bovik, A.C., "Mean squared error: Love it or leave it? A new look at Signal Fidelity Measures," Signal Processing
Magazine, IEEE , vol.26, no.1, pp.98-117, Jan. 2009
13. Kusching at el., An evaluation of TCP-based rate-control algorithms for adaptive internet streaming of H.264/SVC, MMSys
2010
14. Lederer at el., Dynamic adaptive streaming over HTTP dataset, MMSys 2012
15. K. Seshadrinathan and A. C. Bovik, Motion Tuned Spatio-temporal Quality Assessment of Natural Videos, vol. 19, no. 2, pp.
335-350, IEEE Transactions on Image Processing, Feb. 2010
16. NTIA VQM http://www.its.bldrdoc.gov/resources/video-quality-research/software.aspx
17. Yim et al., Evaluation of temporal variation of video quality in packet loss network, Signal Processing Image Communication,
2011
18. Singh, K.D.; Hadjadj-Aoul, Y.; Rubino, G.; , "Quality of experience estimation for adaptive HTTP/TCP video streaming using
H.264/AVC," Consumer Communications and Networking Conference (CCNC), 2012 IEEE , vol., no., pp.127-131, 14-17 Jan.
2012
42

ConférenSquad #3 : Subjective Quality and HTTP Adaptive Streaming: a Review of Psychophysical Studies (Frédéric Dufaux, Telecom Paris Tech)

Recommandé

Recommandé

Contenu connexe

En vedette

En vedette (16)

Similaire à ConférenSquad #3 : Subjective Quality and HTTP Adaptive Streaming: a Review of Psychophysical Studies (Frédéric Dufaux, Telecom Paris Tech)

Similaire à ConférenSquad #3 : Subjective Quality and HTTP Adaptive Streaming: a Review of Psychophysical Studies (Frédéric Dufaux, Telecom Paris Tech) (20)

Dernier

Dernier (20)

ConférenSquad #3 : Subjective Quality and HTTP Adaptive Streaming: a Review of Psychophysical Studies (Frédéric Dufaux, Telecom Paris Tech)