1. Ultra-High-Definition Quality of
Experience with MPEG-DASH
Priv.-Doz. Dr. Christian Timmerer
Daniel Weinberger, Christopher Mueller and Stefan Lederer
Chief Innovation Officer (CIO) at bitmovin GmbH
http://www.bitmovin.com  christian.timmerer@bitmovin.com
Alpen-Adria-Universität Klagenfurt (AAU)  Faculty of Technical Sciences (TEWI)  Department of Information
Technology (ITEC)  Multimedia Communication (MMC)  Sensory Experience Lab (SELab)
http://blog.timmerer.com  http://selab.itec.aau.at/  http://dash.itec.aau.at  christian.timmerer@itec.aau.at
http://www.slideshare.net/christian.timmerer

2. Outline
• Introduction
• Quality, Quality of Experience, and DASH
• Evaluation strategies
• Results
• Conclusions
• Acknowledgment [some slides]: Ali C. Begen, CISCO
April 16, 2015 NAB2015 - BEC - QoE-DASH 2

3. Introduction
• Real-time entertainment
– Streaming video and audio
– > 60% of Internet traffic
• All delivered over-the-top (OTT)
• MPEG Dynamic Adaptive
Streaming over HTTP (DASH)
– Coding format agnostic
• DASH Industry Forum
– Interoperability Points (IOPs) for
common codecs and others (v3.0)
– E.g., AVC/H.264, HEVC/H.265, and
AAC
April 16, 2015 NAB2015 - BEC - QoE-DASH 3

4. Over-The-Top – Adaptive Media Streaming
April 16, 2015 NAB2015 - BEC - QoE-DASH 4
Adaptation logic is within the
client, not normatively specified
by the standard, subject to
research and development

5. Open Digital Media Value Chain
April 16, 2015 NAB2015 - BEC - QoE-DASH 5
Create
Content
Aggregate
Monetize
Distribute
Content
Consume
Content
Any Content Any Storefront Any Network Any Device
CDNsMedia
Protocols
Internet
Transport
DRM
Encoding
Encapsulation
Dynamic
Ads
Clients

6. Simplified Example Workflow: bitcodin/bitdash
April 16, 2015 NAB2015 - BEC - QoE-DASH 6
Source: http://www.bitmovin.net/bitcodin-cloud-based-transcoding-streaming-platform/

7. Internet TV vs. Traditional TV in 2010
• Areas most important to
overall TV experience are
– Content
– Timing control
– Quality
– Ease of use
• While traditional TV surpasses
Internet TV only in quality, it
delivers better “overall
experience”
April 16, 2015 NAB2015 - BEC - QoE-DASH 7
When comparing traditional and Internet TV,
which option is better?
Traditional Internet
Content 7%  79%
Timing / Control 7%  83%
Quality  80% 16%
Ease of Use 23%  52%
Control (FF, etc.) 9%  77%
Portability 4%  92%
Interactivity 31%  52%
Sharing 33%  56%
Overall Experience  53% 33%
Source: Cisco IBSG Youth Survey, Cisco IBSG Youth Focus Group Sessions, 2010

8. Quality (of Experience)
• QoE as evolution of QoS [ITU-T
P.10/G.100]
• QoS: totality of characteristics
of a telecommunications
service that bear on its ability
to satisfy stated and implied
needs of the user of the service
• QoE: the overall acceptability
of an application or service, as
perceived subjectively by the
end-user
April 16, 2015 NAB2015 - BEC - QoE-DASH 8
Many definitions but in general, it’s like an elephant

9. Quality of Experience
• COST Action IC1003 – QUALINET (http://www.qualinet.eu/)
“the degree of delight or annoyance of the user of an application or service. It results
from the fulfillment of his or her expectations with respect to the utility and/or
enjoyment of the application or service in the light of the user’s personality and
current state”
• QoE influence factors
– Any characteristic of a user, system, service, application, or context
– Grouped into human, system, and context
• QoE features
– Perceivable, recognized and namable characteristic of the individual’s experience
– Depends on the level of direct perception, interaction, the usage situation
April 16, 2015 NAB2015 - BEC - QoE-DASH 9

10. QoE for DASH
• Different application domains have different QoE
requirements
– Need to provide specializations of the general QoE definition
– Take into account requirements formulated by means of
influence factors and features of QoE
• QoE influence factors for DASH
– Initial/start-up delay (low)
– Buffer underruns, stalls, freezes (zero)
– Quality switches (low)
– Media throughput (high)
– …
April 16, 2015 NAB2015 - BEC - QoE-DASH 10

11. QoE Evaluation for DASH-based Services
• Test sequence
– Many datasets available
– Adopted Big Buck Bunny & DASHed it with bitcodin
• Players
– bitdash
– …and compare it with ten different adaptation algorithms
• Objective evaluation
– Test setup
– Predefined bandwidth trajectory (or real network traces)
• Subjective evaluation
– Lab vs. crowdsourcing
April 16, 2015 NAB2015 - BEC - QoE-DASH 11
http://www.bitcodin.com/
http://www.dash-player.com/

12. Objective Evaluations
April 16, 2015 NAB2015 - BEC - QoE-DASH 12
Stalls (lower is better)Average Bitrate (higher is better)

13. Stalls are really bad…
April 16, 2015 NAB2015 - BEC - QoE-DASH 13
Conviva: Viewer Experience Report. 2014

14. DASH-JS vs. bitdash
April 16, 2015 NAB2015 - BEC - QoE-DASH 14

15. Subjective Evaluation
• Microworker platform
– Limited to Europe, USA/Canada, India
• DASH clients
– DASH-JS (dash.itec.aau.at)
– dash.js (DASH-IF)
– YouTube
• Tears of Steal trailer according to YouTube
configuration
• Screening techniques
– Browser fingerprinting
– Presentation time
– QoE ratings and Pre-Questionnaire
April 16, 2015 NAB2015 - BEC - QoE-DASH 15

16. What about 4K and 8K?
April 16, 2015 NAB2015 - BEC - QoE-DASH 16
• Why? – because we can!
• Supported on the Web
– HTML5, MSE
– AVC/H.264
– [HEVC/H.265 needed
to lower bitrate]
• See demo @
http://www.dash-
player.com/
• UHD-QoE evaluation

17. Conclusions
• QoE for DASH-based services (a rule of thumb)
– Startup delay (low [but live vs. on-demand & short vs. long-tail
content])
– Buffer underrun / stalls (zero)
– Quality switches (low) and media throughput (high)
– Energy- and cost-awareness (data plan)
• No general applicable QoE model for DASH
– (Too) many factors influencing / features of QoE for DASH-based
services
– Methodology for reproducible research is in place and well established
– Ample research opportunities
April 16, 2015 NAB2015 - BEC - QoE-DASH 17
Main QoE
factors for DASH

18. Thank you!
April 16, 2015 NAB2015 - BEC - QoE-DASH 18
Source: http://www.bitmovin.net/bitcodin-cloud-based-transcoding-streaming-platform/