HTTP Adaptive StreamingState of the Art and Challenges Ahead

HTTP Adaptive Streaming
State of the Art and Challenges Ahead
Priv.-Doz. Dr. Christian Timmerer
[Ack: Ali C. Begen, Networked Media & Ozyegin University | Thomas Stockhammer, Qualcomm Inc. | Iraj Sodagar, Consultant]
Alpen-Adria-Universität Klagenfurt (AAU)  Faculty of Technical Sciences (TEWI) 
Department of Information Technology (ITEC)  Multimedia Communication (MMC)
http://blog.timmerer.com  http://dash.itec.aau.at  christian.timmerer@itec.aau.at
Chief Innovation Officer (CIO) at bitmovin GmbH
http://www.bitmovin.com  christian.timmerer@bitmovin.com
Internet QoE, July 2, 2018
http://www.slideshare.net/christian.timmerer

Importance of Multimedia Delivery
• Real-time entertainment: Streaming video and audio; >70% of Internet traffic at peak periods
• Popular services
– YouTube (17.53%), Netflix (35.15%), Amazon Video (4.26%), Hulu (2.68%); all delivered over-the-top (OTT)
• Forecast: Visual Networking Index (VNI) 2016-2021 (Sep’17)
– IP video traffic will be 82% of all consumer Internet traffic by 2021 (up from 73% in 2016); will grow
threefold from 2016 to 2021
– Live Internet video will account for 13% of Internet video traffic by 2021; will grow 15-fold from 2016 to
2021
• More people now subscribe to Netflix (50.85M) than cable TV (48.61M) in the US (Q1 2017)
July 2, 2018 Dr. Timmerer [AAU/Bitmovin] 2
Global Internet Phenomena Report: 2016
>70% Downstream Traffic

$44,000,000

Example Platform/Infrastructure
https://bitmovin.com/

Common Annoyances in Streaming
• Wrong format
• Wrong protocol
• Plugin requirements
• DRM issues
• Long start-up delay
• Poor quality
• Frequent stalls
• Quality oscillations
• No seeking features

Over-The-Top – Adaptive Media Streaming
• In a nutshell…
Adaptation logic is within the
client, not normatively
specified by the standard,
subject to research and
development

Multi-Bitrate Encoding and Representation Switching
Contents on the Web Server
Movie A – 200 Kbps
Movie A – 400 Kbps
Movie A – 1.2 Mbps
Movie A – 2.2 Mbps
. . .
. . .
Movie K – 200 Kbps
Movie K – 500 Kbps
Movie K – 1.1 Mbps
Movie K – 1.8 Mbps
. . .
. . .
Time (s)
Start quickly
Keep requesting
Improve quality
Loss/congestion detection
Revamp quality
...
. . .
Segments

Adaptive Streaming over HTTP
…
…
…
…
HTTP GETs
Client
Buffer
Media
Player
HTTP
Server

Formats and Standards
• Adobe
– HTTP Dynamic Streaming (HDS)
– Switched to DASH
• Apple
– HTTP Live Streaming (HLS)
– Required for iOS
• Microsoft
– Smooth Streaming
– Switched to DASH, almost..
• MPEG Dynamic Adaptive Streaming over HTTP (DASH)
– Supported by Netflix, YouTube, Bitmovin, etc.
• MPEG Common Media Application Format (MPEG-A Part 19)
– The new kid on the block – support for “fragmented mp4 in HLS”
– DASH/HLS convergence at segment level – some open issues with encryption format
Source: http://xkcd.com/927/

Scope of DASH: what is specified?
Media Presentation on
HTTP Server
DASH-enabled ClientMedia Presentation
Description
.
.
.
Segment
…
.
.
.Segment
…
.
.
.
Segment
…
.
.
.Segment
…
…
Segments located by
HTTP-URLs
DASH Control Engine
HTTP/1.1
HTTP
Client
MPD
Parser
Media
Engine
On-time HTTP
requests to
segments

DASH Data Model
MPD
Period id = 1
start = 0 s
Period id = 3
start = 300 s
Period id = 4
start = 850 s
Period id = 2
start = 100 s
Adaptation Set 0
subtitle turkish
Adaptation Set 2
audio english
Adaptation Set 1
BaseURL=http://abr.rocks.com/
Representation 2
Rate = 1 Mbps
Representation 4
Rate = 3 Mbps
Representation 1
Rate = 500 Kbps
Representation 3
Rate = 2 Mbps
Resolution = 720p
Segment Info
Duration = 10 s
Template:
3/$Number$.mp4
Segment Access
Initialization Segment
http://abr.rocks.com/3/0.mp4
Media Segment 1
start = 0 s
Media Segment 2
start = 10 s
Adaptation Set 3
audio german
Adaptation Set 1
video
Period id = 2
start = 100 s
Representation 3
Rate = 2 Mbps
Selection of
components/tracks
Well-defined
media format
Selection of
representations
Splicing of arbitrary
content like ads
Chunks with addresses
and timing

MPEG-DASH Features

https://bitmovin.com/demo/

MPEG CMAF: Threat or Opportunity?
+ CMAF Media Object
Model compatible with
DASH Data Model
+ Segment formats based
on ISOBMFF
- Different manifest
formats (MPD vs. m3u8)
- CENC: AES-128 CBC
(HLS) vs. AES-128 CTR (all
others) modeJuly 2, 2018 Dr. Timmerer [AAU/Bitmovin] 15
https://bitmovin.com/what-is-cmaf-threat-opportunity/

CMAF ISO-BMFF Media Objects
• Manifests typically provide URLs to
– CMAF track files
– CMAF header + CMAF segments
• single/multiple fragment(s)
– CMAF header + CMAF chunk
Encoder
Encryption
Packaging
CMAF
Header
CMAF
Fragment
CMAF
Fragment
CMAF
Chunk
CMAF
Chunk
CMAF
Chunk
CMAF
Fragment
R
A
P
R
A
P
R
A
P
R
A
P
CMAF
Fragment
CMAF
Segment
CMAF
Segment
CMAF Track File

Common Problems in DASH
• Encoding | Packaging | Encryption
– Guidelines: 3-20 different representations (mobile to UHD)
– Segment length: 4s shows good tradeoff (2s vs. 9s)
– Offline vs. on-the-fly
• Delivery, distribution, CDN
– MMSys’16 keynote by Neill Kipp: https://mmsys2016.itec.aau.at/
• Consumption and Quality of Experience (QoE)
– Adaptation strategies: buffer- vs. throughput-based
– Multi-client competition: on-off behavior
– Quality-aware streaming: highest possible bitrate vs. highest quality
– Inter-Destination Media Synchronization (IDMS): new applications
– Virtual Reality / 360-degree video: tiled streamingJuly 2, 2018 Dr. Timmerer [AAU/Bitmovin] 17
S. Lederer, C. Müller and C. Timmerer, “Dynamic Adaptive Streaming
over HTTP Dataset”, In Proceedings of the ACM Multimedia Systems
Conference 2012, Chapel Hill, North Carolina, February 22-24, 2012.

Quality of Experience for DASH
• Objective
– Initial or startup delay (low)
– Buffer underrun / stalls (zero)
– Quality switches (low)
– Media throughput (high)
– [Other media-related configuration: encoding,
representations, segment length, …]
• Subjective
– Mean Opinion Score (MOS) – various scales
– Various methodologies (e.g., DSCQS, DSIS, ACR, PC, …)
July 2, 2018 Dr. Timmerer [AAU/Bitmovin] 18M. Seufert, et al., "A Survey on Quality of Experience of HTTP Adaptive Streaming,"IEEE
Communications Surveys & Tutorials, vol.17, no.1, 2015. doi:10.1109/COMST.2014.2360940

DASH QoE in Real-World
B. Rainer, C. Timmerer, Quality of Experience of Web-based Adaptive HTTP Streaming Clients in
Real-World Environments using Crowdsourcing, Proceedings of International Workshop on
VideoNext: Design, Quality and Deployment of Adaptive Video Streaming, Sydney, Australia, Dec.
2014. https://doi.org/10.1145/2676652.2676656
Stalls and low quality
are bad for QoE but
not startup delay

10 different
adaptation
algorithms
C. Timmerer, M. Maiero, B. Rainer, Which Adaptation
Logic? An Objective and Subjective Performance
Evaluation of HTTP-based Adaptive Media Streaming
Systems, In arXiv.org [cs.MM], N.N., vol. abs/1606.00341,
N.N., pp. 11, 2016. https://arxiv.org/pdf/1606.00341

AdViSE
• Adaptive Media Content
[DASH, HLS, CMAF]
• Players/Algorithms
• Network Parameters
Impaired
Media Sequences
Generate Impaired
Media Sequences
Templates [Startup
Delay, Stalling, …]
WESP
QoE Evaluation
Parameters
[Questionnaire,
Methodology,
Crowdsourcing
Platform, …]
QoS/QoE
Metrics
Subjective Results +
Other Data
Reports
Analysis
AdViSE: Anatoliy Zabrovskiy, Evgeny Kuzmin, Evgeny Petrov, Christian Timmerer, and Christopher Mueller. 2017. AdViSE: Adaptive Video Streaming Evaluation Framework
for the Automated Testing of Media Players. In Proceedings of the 8th ACM on Multimedia Systems Conference (MMSys'17). ACM, New York, NY, USA, 217-220. DOI:
https://doi.org/10.1145/3083187.3083221
WESP: Benjamin Rainer, Markus Waltl, Christian Timmerer, A Web based Subjective Evaluation Platform, In Proceedings of the 5th International Workshop on Quality of
Multimedia Experience (QoMEX'13) (Christian Timmerer, Patrick Le Callet, Martin Varela, Stefan Winkler, Tiago H Falk, eds.), IEEE, Los Alamitos, CA, USA, pp. 24-25, 2013.
https://doi.org/10.1109/QoMEX.2013.6603196
How to evaluate HAS systems?
①
②
③④
⑤
Log of
Segment
Requests

C. Timmerer, A. Zabrovskiy, and Ali C. Begen, Automated Objective and Subjective
Evaluation of HTTP Adaptive Streaming Systems, Proceedings of the 1st IEEE
International Conference on Multimedia Information Processing and Retrieval
(MIPR), Miami, FL, USA, April 2018.
Example Evaluation Results
• Test sequence encoded 15 different representation (Amazon Prime configuration:
400x224@100Kbps – 1920x1080@15Mbps) with 4s segment length
• Bandwidth trajectory based on prior work
proposed in literature; network delay 70ms

Bandwidth Index vs. QoE
• Bandwidth
Index
– Avg. bitrate
– Efficiency
– Stability
• QoE Model
– Startup delay
– Stalls
C. Timmerer, A. Zabrovskiy, E. Kuzmin, E. Petrov, Quality of experience of commercially deployed
adaptive media players, In 2017 21st Conference of Open Innovations Association (FRUCT) (Sergey
Balandin, ed.), pp. 330-335, 2017. https://doi.org/10.23919/FRUCT.2017.8250200

Omnidirectional Video / 360° Video
Capturing Devices
• Stitching, Projection Formats
• Encoding, Encryption, Encapsulation
• Storage, Content Distribution, Delivery
• Processing, Decoding, Rendering, … Consumer Devices
C. Timmerer, A. C. Begen, A Framework for Adaptive Delivery of Omnidirectional Video, In
Electronic Imaging – Human Vision and Electronic Imaging (HVEI), vol. 2018, no. 16, 2018.

Adaptive Streaming Options for 360
• Traditional, viewport-agnostic streaming
– Simple, easy, deployed today
– Bandwidth waste, quality issues
• Viewport-adaptive streaming
– Multiple versions for predefined viewports
– Various projection techniques (pyramid)
– Bandwidth waste reduced, increased storage
and CDN costs, limited flexibility
• Tile-based streaming
– Use tiling technique of modern video codecs
– High complexity, full flexibility
– Multiple challenges
X. Corbillon, et al., "Viewport-adaptive navigable 360-degree video
delivery," 2017 IEEE International Conference on Communications
(ICC), Paris, 2017, https://doi.org/10.1109/ICC.2017.7996611
M. Graf, et al. 2017. Towards Bandwidth Efficient Adaptive
Streaming of Omnidirectional Video over HTTP: Design,
Implementation, and Evaluation. Proc. ACM MMSys'17.
https://doi.org/10.1145/3083187.3084016
C. Concolato, et al., "Adaptive Streaming of HEVC
Tiled Videos using MPEG-DASH," IEEE TCSVT, 2017.
https://doi.org/10.1109/TCSVT.2017.2688491

Tile-based Adaptive Streaming
Tile 1 Tile 2 Tile 3 Tile 4 Tile 5 Tile 6
Tile 7 Tile 8 Tile 9
Tile
10
Tile
11
Tile
12
Tile
13
Tile
14
Tile
15
Tile
16
Tile
17
Tile
18
Tile
19
Tile
20
Tile
21
Tile
22
Tile
23
Tile
24
Encoding &
Packaging
Tile
10
Tile
11
Tile
12
Tile
13
Tile
14
Tile
15
Tile
16
Tile
17
Tile
18
Tile
19
Tile
20
Tile
21
Tile
22
Tile
23
Tile
24
Tile
10
Tile
11
Tile
12
Tile
13
Tile
14
Tile
15
Tile
16
Tile
17
Tile
18
Tile
19
Tile
20
Tile
21
Tile
22
Tile
23
Tile
24
Tile
10
Tile
11
Tile
12
Tile
13
Tile
14
Tile
15
Tile
16
Tile
17
Tile
18
Tile
19
Tile
20
Tile
21
Tile
22
Tile
23
Tile
24
Delivery
MPEG-HEVC/H.265
Tiles in ISOBMFF
Adaptive
Player
Tile
10
Tile
11
Tile
12
Tile
13
Tile
14
Tile
15
Tile
16
Tile
17
Tile
18
Tile
19
Tile
20
Tile
21
Tile
22
Tile
23
Tile
24
Tile-based streaming of VR/360°
content with MPEG-DASH SRD /
MPEG-I OMAF
…
…
…
…
Adaptive Streaming using MPEG-
DASH SRD / MPEG-I OMAF
Head Mounted Displays
Browsers, Smart (Mobile) Devices
(Stereo) 2D, (Stereo) 3D

Bitrate Overhead due to Tiling

Bandwidth Requirements

Immersive Media Streaming
• 3DoF, 3DoF+, windowed &
omnidirectional 6DoF, 6DoF
• Point cloud, light field, volumetric
video, holograms, volograms, …
– Content Creation for AR, VR, and Free
Viewpoint Video, Aljosa Smolic
https://www.qomex2018.org/keynotes/#AljT
• Theses today… (Philip A. Chou, 8i)
– Hologram compression today is like
video compression in 1988
– Hologram streaming today is like video
streaming in 1997
• MPEG-I
– First
standardization activity comprising now eight parts
July 2, 2018 Dr. Timmerer [AAU/Bitmovin] 30Holograms are the Next Video, Philip A. Chou, 8i,
http://www.mmsys2018.org/program/keynotes/
C. Timmerer, "Immersive Media Delivery: Overview of Ongoing Standardization Activities," in
IEEE Communications Standards Magazine, vol. 1, no. 4, pp. 71-74, Dec. 2017.
https://doi.org/10.1109/MCOMSTD.2017.1700038
M. Hosseini, C. Timmerer, Dynamic Adaptive Point Cloud Streaming, 23rd Packet Video Workshop,
Amsterdam, The Netherlands, June 2018. https://doi.org/10.1145/3210424.3210429
https://multimediacommunication.blogspot.com/2017/04/vr360-streaming-standardization-related.html

Multimedia Systems Tradeoff
Quality
Content Time
Quality of {Content, Service,
Experience, Life, …}
Content complexity:
traditional AV, AR/VR/360,
multi-modality/-sensory
Based on Klara Nahrstedt at IEEE MIPR’18 Retreat
End-to-end delay, startup
delay, channel switching,
synchronization, interaction

Deployment Thoughts
• Proprietary ecosystems disappeared (Silverlight, Flash),
no more plugins – HTML5!
– MSE/EME available on all major browser platforms
– Support for both DASH/HLS (+CMAF) and CENC
• Rich feature set: multiple codecs, ads, DRM, multi-language/-audio,
subtitles, VR/360, UHD, HFR, HDR, live, on-demand, analytics, …
• Common implementation issues: start-up, buffering, high-quality,
seamless switching, platform support, cost-effective, …
• Solutions available for adaptive streaming, advertising, VR/360, live
streaming, and DRM
– Per-title/-scene/-shot adaptation, AV1, fast encoding, managed on-premise
– Details available at https://bitmovin.com/

Bitmovin kicks off NAB
show 2018 with a bang
● Announcing $30M in Series B
funding and exciting new
product launches
● And yes, we are hiring…
https://bitmovin.com/careers

References
• B. Rainer, C. Timmerer, Quality of Experience of Web-based Adaptive HTTP Streaming Clients in Real-World Environments using Crowdsourcing,
Proceedings of International Workshop on VideoNext: Design, Quality and Deployment of Adaptive Video Streaming, Sydney, Australia, Dec. 2014.
https://doi.org/10.1145/2676652.2676656
• C. Timmerer, M. Maiero, B. Rainer, Which Adaptation Logic? An Objective and Subjective Performance Evaluation of HTTP-based Adaptive Media
Streaming Systems, In arXiv.org [cs.MM], N.N., vol. abs/1606.00341, N.N., pp. 11, 2016. https://arxiv.org/pdf/1606.00341
• A. Zabrovskiy, E. Kuzmin, E. Petrov, C. Timmerer, and C. Mueller. 2017. AdViSE: Adaptive Video Streaming Evaluation Framework for the Automated
Testing of Media Players. In Proceedings of the 8th ACM on Multimedia Systems Conference (MMSys'17). ACM, New York, NY, USA, 217-220. DOI:
https://doi.org/10.1145/3083187.3083221
• B. Rainer, M. Waltl, C. Timmerer, A Web based Subjective Evaluation Platform, In Proceedings of the 5th International Workshop on Quality of
Multimedia Experience (QoMEX'13) (Christian Timmerer, Patrick Le Callet, Martin Varela, Stefan Winkler, Tiago H Falk, eds.), IEEE, Los Alamitos, CA,
USA, pp. 24-25, 2013. https://doi.org/10.1109/QoMEX.2013.6603196
• C. Timmerer, A. Zabrovskiy, and Ali C. Begen, Automated Objective and Subjective Evaluation of HTTP Adaptive Streaming Systems, Proceedings of
the 1st IEEE International Conference on Multimedia Information Processing and Retrieval (MIPR), Miami, FL, USA, April 2018.
• C. Timmerer, A. Zabrovskiy, E. Kuzmin, E. Petrov, Quality of experience of commercially deployed adaptive media players, In 2017 21st Conference of
Open Innovations Association (FRUCT) (Sergey Balandin, ed.), pp. 330-335, 2017. https://doi.org/10.23919/FRUCT.2017.8250200
• C. Timmerer, A. C. Begen, A Framework for Adaptive Delivery of Omnidirectional Video, In Electronic Imaging – Human Vision and Electronic Imaging
(HVEI), vol. 2018, no. 16, 2018.
• M. Graf, C. Timmerer, C. Mueller. Towards Bandwidth Efficient Adaptive Streaming of Omnidirectional Video over HTTP: Design, Implementation, and
Evaluation. Proc. ACM MMSys'17. https://doi.org/10.1145/3083187.3084016
• C. Timmerer, "Immersive Media Delivery: Overview of Ongoing Standardization Activities," in IEEE Communications Standards Magazine, vol. 1, no. 4,
pp. 71-74, Dec. 2017. https://doi.org/10.1109/MCOMSTD.2017.1700038
• M. Hosseini, C. Timmerer, Dynamic Adaptive Point Cloud Streaming, 23rd Packet Video Workshop, Amsterdam, The Netherlands, June 2018.
https://doi.org/10.1145/3210424.3210429

HTTP Adaptive StreamingState of the Art and Challenges Ahead

Recommandé

Recommandé

Contenu connexe

Tendances

Tendances (20)

Similaire à HTTP Adaptive StreamingState of the Art and Challenges Ahead

Similaire à HTTP Adaptive StreamingState of the Art and Challenges Ahead (20)

Plus de Alpen-Adria-Universität

Plus de Alpen-Adria-Universität (20)

Dernier

Dernier (20)