Quality of Experience Past, Present and Future Trends
1. 1
Quality of Experience
Past, Present
and Future Trends
Touradj Ebrahimi
Touradj.Ebrahimi@epfl.ch
Colloquium on Quality of Experience Alpen-Adria-Universität Klagenfurt
in Multimedia Systems and Services 23rd November 2012, Austria
2. Today
we
will
talk
about… 2
•What
is
“quality”?
•Origins
of
“quality”
evalua7on
in
telecommunica7ons
•How
is
“quality”
measured
in
mul7media
communica7ons
today
?
•What
are
the
trends
on
“quality”
in
mul7media
communica7on?
•
What
are
the
challenges
ahead?
Colloquium on Quality of Experience Alpen-Adria-Universität Klagenfurt
in Multimedia Systems and Services 23rd November 2012, Austria
3. 3
Quality: a simple yet difficult concept
• Like many human sensations, quality is easy to
understand but difficult to define
• Quality according to Wikipedia:
– A quality (from Latin - qualitas) is an attribute or a
property
– Some philosophers assert that a quality cannot be
defined
Colloquium on Quality of Experience Alpen-Adria-Universität Klagenfurt
in Multimedia Systems and Services 23rd November 2012, Austria
4. A fundamental, ancient, but largely under investigated concept 4
Aristotle classified every object of human
apprehension into 10 Categories
– Substance
– Quantity
– Quality
– Relation
– Place
– Time
– Position
– State
– Action
– Affection
Colloquium on Quality of Experience Alpen-Adria-Universität Klagenfurt
in Multimedia Systems and Services 23rd November 2012, Austria
5. 5
Quality According to ISO 9000
• ISO 9000: a family of standards for quality
management systems
• Quality of something can be determined by
comparing a set of inherent characteristics with
a set of requirements
– High quality: if characteristics meet requirements
– Low quality: if characteristics do not meet all
requirements
• Quality is a relative concept
– Degree of quality
Colloquium on Quality of Experience Alpen-Adria-Universität Klagenfurt
in Multimedia Systems and Services 23rd November 2012, Austria
6. 6
Qualinet white paper
• White Paper produced by COST Action
IC1003 (Qualinet):
– http://www.qualinet.eu/images/stories/
whitepaper_v1.1_dagstuhl_output_corrected.pdf
• Several definitions of quality in multimedia
systems and services and other related
concepts
Colloquium on Quality of Experience Alpen-Adria-Universität Klagenfurt
in Multimedia Systems and Services 23rd November 2012, Austria
7. 7
Quality is like an elephant …
The blind men and the elephant: Poem by John Godfrey Saxe
Colloquium on Quality of Experience Alpen-Adria-Universität Klagenfurt
in Multimedia Systems and Services 23rd November 2012, Austria
8. 8
Quality in
telecommunications
Colloquium on Quality of Experience Alpen-Adria-Universität Klagenfurt
in Multimedia Systems and Services 23rd November 2012, Austria
9. 9
Quality
in
telecommunica4ons
Network Quality Link Quality
Capacity Bitrate
Coverage Frame/Bit/Packet loss
Handoff Delay
User Quality
Speech fidelity
Audio fidelity
Image fidelity
Video fidelity
Colloquium on Quality of Experience Alpen-Adria-Universität Klagenfurt
in Multimedia Systems and Services 23rd November 2012, Austria
10. 10
Quality in telecommunications
Objective: input = output
Colloquium on Quality of Experience Alpen-Adria-Universität Klagenfurt
in Multimedia Systems and Services 23rd November 2012, Austria
11. 11
QoS in telecommunications
• Quality of Service (QoS)
– Resource reservation control mechanisms
– Ability to provide different priority to different
applications or data flows
– Guarantee a certain level of performance (quality) to
a data flow
• Requires measuring or estimating quality
• (Service) Provider-centric concept
– Tightly related to the concept of Mean Opinion Score
(MOS)
Colloquium on Quality of Experience Alpen-Adria-Universität Klagenfurt
in Multimedia Systems and Services 23rd November 2012, Austria
12. 12
Mean Opinion Score (MOS)?
• Widely used in many fields:
– Politics/Elections
– Marketing/Advertisement
– Food industry
– Multimedia
– …
• The likely level of satisfaction of a service or product as
appreciated by an average user in a typical use
• Should be performed such that it generates reliable and
reproducible results
– Subjective evaluation methodology
– More complex and difficult that it seems
Colloquium on Quality of Experience Alpen-Adria-Universität Klagenfurt
in Multimedia Systems and Services 23rd November 2012, Austria
13. 13
What is behind a MOS?
Colloquium on Quality of Experience Alpen-Adria-Universität Klagenfurt
in Multimedia Systems and Services 23rd November 2012, Austria
14. 14
What is wrong with MOS?
• Ground truth
– It is assumed to be the optimal solution (ground truth)
• Defies user preference
– It is assumed to be independent of specific subjects
• Ignores key parameters
– It is often assumed to be independent of content and
context
• Misuse as a comparison approach
– It is often used to compare performance between
alternative tools for a same task/operation
• Impractical or expensive
• Statistical assumptions not always realistic
Colloquium on Quality of Experience Alpen-Adria-Universität Klagenfurt
in Multimedia Systems and Services 23rd November 2012, Austria
15. 15
Ground truth
Colloquium on Quality of Experience Alpen-Adria-Universität Klagenfurt
in Multimedia Systems and Services 23rd November 2012, Austria
16. 16
Defies user preference
Colloquium on Quality of Experience Alpen-Adria-Universität Klagenfurt
in Multimedia Systems and Services 23rd November 2012, Austria
17. 17
Content
p01 $$$$$$p06 $$$$$$$$$$p10 $$$$$$$$$$$$$$$bike$$$$$$$$$$$$$$$$$$cafe$$$$$$$$$$$$$$$woman$
Colloquium on Quality of Experience Alpen-Adria-Universität Klagenfurt
in Multimedia Systems and Services 23rd November 2012, Austria
18. 18
Context
Colloquium on Quality of Experience Alpen-Adria-Universität Klagenfurt
in Multimedia Systems and Services 23rd November 2012, Austria
19. Objec4ve
quality
metrics 19
• Subjec7ve
tests
are
7me
consuming,
expensive,
difficult
to
design,
and
cannot
be
used
in
real-‐7me
systems
• Objec7ve
algorithms,
i.e.
metrics,
es7ma7ng
MOS
with
high
level
of
correla7on
are
desired
• Full
reference
metrics
• No
reference
metrics
• Reduced
reference
metrics
Colloquium on Quality of Experience Alpen-Adria-Universität Klagenfurt
in Multimedia Systems and Services 23rd November 2012, Austria
20. Objec4ve
quality
metrics
based
on
fidelity 20
• Full
Reference
scenario
• Metrics
which
look
at
the
fidelity
of
the
signal
when
compared
to
an
explicit
reference:
processed
signal
=
perfect
quality
reference
signal
+
error
signal
Colloquium on Quality of Experience Alpen-Adria-Universität Klagenfurt
in Multimedia Systems and Services 23rd November 2012, Austria
21. Objec4ve
quality
metrics
based
on
fidelity 21
• Examples
of
fidelity
metrics
for
image
quality
quality
assessment
− Mean
Square
Error
(MSE)
− Peak
Signal
to
Noise
Ra7o
(PSNR)
− Maximum
Pixel
Devia7on
(Linf)
− Weighted
PSNR
− Masked
PSNR
− Structural
SIMilarity
(SSIM)
− Mul7scale
Structural
Similarity
(MSSIM)
− Visual
Informa7on
Fidelity
(VIF)
− etc…
Colloquium on Quality of Experience Alpen-Adria-Universität Klagenfurt
in Multimedia Systems and Services 23rd November 2012, Austria
22. 22
What is wrong with objective quality metrics ?
• Poor performance in predicting MOS
– Especially in real life situations
• How to deal with color components?
– Even in PSNR!
• How to deal with video?
– Time versus spatial dimension
• How to take into account context?
– Display size/type
– Environmental illumination
– …
• How to deal with user preference/expectation?
• ...
Colloquium on Quality of Experience Alpen-Adria-Universität Klagenfurt
in Multimedia Systems and Services 23rd November 2012, Austria
23. MVC Coding Performance
23
MVC assessment using PSNR as metric
Sample comparison of simulcast vs inter-view prediction
8 views (640 480), consider rate for all views
~25% bit rate savings over all views
Ballroom Race1
40 42
39 41
40
38
39
37
38
PSNR (db)
PSNR (db)
36
37
35
36
34
35
33
Simulcast 34 Simulcast
32 MVC 33 MVC
31 32
0 200 400 600 800 1000 1200 1400 1600 1800 0 200 400 600 800 1000 1200 1400 1600
Bitrate (Kb/s) Bitrate (Kb/s)
CORESA
24-25 mai 2012, Lille, MPEG 3D Video Coding Activity
Vetro: Update on
France
24. MVC:assessment by subjective evaluation
MVC Subjective Performance 24
• Main finding: MVC achieves comparable quality to
simulcast with as little as 25% rate for dependent view
4.50
Mean Opinion Score
4.00
3.50
3.00
2.50
2.00
1.50
1.00
al C
) c t c t c t c t c t c t c t
in 0P 5P 5P 0P 5P 0P 5P
r ig AV _
O + _5 _3 _2 _2 _1 _1 L
VC 12
L
12
L
12
L
12
L
12
L
12
L 12
(A
st
ca
ul
m
Si
Base view fixed at 12Mbps
Dependent view at varying percentage of base view rate
CORESA Vetro: Update on MPEG 3D Video Coding Activity
24-25 mai 2012, Lille, France
25. 25
Evolving Quality Paradigms
Colloquium on Quality of Experience Alpen-Adria-Universität Klagenfurt
in Multimedia Systems and Services 23rd November 2012, Austria
26. 26
User centered evaluation
• It is becoming increasing important to evaluate quality from a user
perspective rather than from a provider/system perspective
• Constrains the tests to potential users and stimuli and evaluates the
acceptability with respect to the task and the context
• Identify underlying quality factors beside the overall quality
CORESA
24-25 mai 2012, Lille, France
27. 27
What people mean when they say QoE ?
• “The degree of fulfillment of an intended experience
on a given user”
as defined by Touradj Ebrahimi, 2001
• “The overall acceptability of an application or service
… by the end user”
as defined by the ITU-T
• “The degree of delight ... of the user of an
application or service ... user’s personality and
current state.”
as defined in white paper by Qualinet
The term ‘experience’ promises individual engagement …
Colloquium on Quality of Experience Alpen-Adria-Universität Klagenfurt
in Multimedia Systems and Services 23rd November 2012, Austria
28. Quality
of
Service
vs
Quality
of
Experience 28
• Quality
of
Service:
Value
of
the
average
user’s
experience
es7mated
by
a
service/product/content
provider
• Quality
of
Experience:
Value
(es7mated
or
actually
measured)
of
a
specific
user’s
richness
of
experience
• Quality
of
Experience
is
the
dual
(and
extended)
view
of
QoS
problem
QoS=provider-‐centric
QoE=user-‐centric
Colloquium on Quality of Experience Alpen-Adria-Universität Klagenfurt
in Multimedia Systems and Services 23rd November 2012, Austria
29. 29
A simple model for QoE
• User attributes
– individual attributes – expectation, age, sex, personality, background…
– sensorial attributes – including limitations and deficiencies
– perceptual attributes
– emotional attributes
• System attributes
– technical attributes (as in QoS) QoE
• Contextual attributes user
QoS
context
– environmental attributes
– device attributes
– service attributes
– content attributes
Colloquium on Quality of Experience Alpen-Adria-Universität Klagenfurt
in Multimedia Systems and Services 23rd November 2012, Austria
30. Personas and scenarios [Strohmeier2009] 30
• Personas (user preference)
– Archetypical user representing the needs, behaviors
and goals of a particular group of users
– Not real people but fictional characters derived from
user requirements
• Scenarios (context)
– Realistic usage environment
CORESA
24-25 mai 2012, Lille, France
31. 31
Trends in QoE
• Digital world has (re-)discovered the notion of
quality
– Lower quality content is increasingly less tolerated by
end-users
– Digital technology have surpassed barriers of
attainable quality, while remaining cost effective
• Increasing interest in QoE
– Extending from device-centric and system-centric
quality optimization to end-to-end and especially
user-centric optimization
Colloquium on Quality of Experience Alpen-Adria-Universität Klagenfurt
in Multimedia Systems and Services 23rd November 2012, Austria
32. 32
Trends in QoE community building
• Increased interest in workshops and conferences
around the notion of quality assessment and metrics
– QoMEX: International Workshop on Quality of Multimedia
Experience (http://www.qomex.org)
– VPQM: International Workshop on Video Processing and Quality
Metrics for Consumer Electronics (http://www.vpqm.org)
– …
• QoE is one of the issues referred to in research
programs funded by the EC
– Various calls in the recent year include QoE issues
– QoE is one of the top topics in future framework program
– Already ongoing initiatives such as COST Action IC1003
(Qualinet)
Colloquium on Quality of Experience Alpen-Adria-Universität Klagenfurt
in Multimedia Systems and Services 23rd November 2012, Austria
33. 33
Trends in standardization
• Standardization efforts in quality assessment and metrics
– Video Quality Experts Group (VQEG)
– ITU-T SG 12 (Performance, QoS and QoE)
– JPEG (Advanced Image Coding - AIC)
– MPEG (HEVC, 3DVC)
– …
Colloquium on Quality of Experience Alpen-Adria-Universität Klagenfurt
in Multimedia Systems and Services 23rd November 2012, Austria
34. 34
Challenges ahead
• Some key issues in QoE:
– Content-dependent quality assessment methods and metrics
– Context-dependent quality assessment methods and metrics
– Quality assessment methods and metrics beyond AV (haptics, …)
– Multi-modal quality assessment methods and metrics (AV, …)
– 3D quality assessment methods and metrics (3D sound, 3D video, …)
– HDR content quality assessment methods and metrics
– Interaction quality metrics (closely related to usability)
– Presence/immersion quality metrics
– …
• Need for Quality Certification Mechanisms of multimedia
services and products
– Similar in idea to ISO 9000 series
Colloquium on Quality of Experience Alpen-Adria-Universität Klagenfurt
in Multimedia Systems and Services 23rd November 2012, Austria
35. 35
What does this all mean to you?
• Era of user-centric multimedia has already started
– It is not anymore sufficient to merely add new features and
functionalities to multimedia systems
– True added value in terms of impact on user experience of such
features and functions should be demonstrated
– Quality of Experience plays a central role in this new game
• Opportunities in research, technology, business, art and
entertainment
Colloquium on Quality of Experience Alpen-Adria-Universität Klagenfurt
in Multimedia Systems and Services 23rd November 2012, Austria
36. Thank you for your attention 36
Questions?
Colloquium on Quality of Experience Alpen-Adria-Universität Klagenfurt
in Multimedia Systems and Services 23rd November 2012, Austria