Advance video distribution

Advance Video Distribution Fast Forward Your Development www.dsp-ip.com

Day I - Agenda • Current Delivery technologies (RTP/RTSP) • Components in depth ▫ File Formats ▫ Codecs • Video Streaming Evolution ▫ Requirements ▫ Solutions • CDN • Basic Adobe Flash Fast Forward Your Development www.dsp-ip.com

Day II - Agenda • Microsoft “Smooth Streaming” • Adobe “HTTP Streaming” • Apple “HTTP Streaming” • Internet Gorillas positioning • Adobe OSMF Fast Forward Your Development www.dsp-ip.com

Current Delivery Methods Fast Forward Your Development www.dsp-ip.com

Progressive Download • Simple • Utilizing existing protocols & servers (HTTP) • Media file is prepared: metadata up front • Playback - after metadata is received • Cache-ability - supported • Seek-ability – very limited support • Poor user experience - seek, multi-rate • Waste of bandwidth when not watched fully • Low cost Fast Forward Your Development www.dsp-ip.com

Pseudo Streaming • Media is sent as a regular file like Progressive DW • Server must understand how the media is structured • Playback: after metadata is received • Existing protocols ▫ Non standard server ▫ Non standard client component • Cache-ability – Limited ! • Seek-ability – supported • User experience – better than PD, support seek. • Waste of bandwidth when not watched full Fast Forward Your Development www.dsp-ip.com

Streaming • Seek-ability – supported • Server side – proprietary technology (FMS) • Cache-ability – requires special servers for streaming • User experience – very good • Cost – high Fast Forward Your Development www.dsp-ip.com

HTTP Streaming Intro • HTTP Streaming offers the advantages of: ▫ Progressive download in terms of Cost Standard Server Scalability Standard client components (OSMF) ▫ Streaming in terms of User experience Seek-ability of streaming Fast Forward Your Development www.dsp-ip.com

RTSP Streaming Fast Forward Your Development www.dsp-ip.com

RTSP Protocol • Real Time Streaming Protocol • Used for controlling streaming data over the web. • Designed to efficiently broadcast audio/video- on-demand to large groups. • Using Directives to control the stream ▫ Options, Describe, Setup, Play, Pause, Record, Teardown. Fast Forward Your Development www.dsp-ip.com

SDP Protocol • Describes the metadata of the stream. • Mainly used in: SIP, RTSP and other Multicast sessions. Protocol Version • Sample SDP description: Session ID ▫ v=0 Session Name ▫ o=jdoe 2890844526 2890842807 IN IP4 10.47.16.5 Session Info. ▫ s=SDP Seminar Description URI ▫ i=A Seminar on the session description protocol Connection Info. u=http://www.example.com/seminars/sdp.pdf e=j.doe@example.com (Jane Doe) Active session time ▫ c=IN IP4 224.2.17.12/127 Session Attribute lines ▫ t=2873397496 2873404696 Media Name and Transport address ▫ a=recvonly ▫ m=audio 49170 RTP/AVP 0 Media Attribute lines ▫ m=video 51372 RTP/AVP 99 ▫ a=rtpmap:99 h263-1998/90000 Fast Forward Your Development www.dsp-ip.com

Client-Server flow Client Server Web HTTP GET Web Stream URI Server Browser OPTIONS DESCRIBE SDP Information SETUP Media Media PLAY Player RTP Media Stream Server RTP Media Stream PAUSE TEARDOWN Fast Forward Your Development www.dsp-ip.com

RTSP Protocol Parameters • version ▫ The version of rtsp. (RTSP/1.0) • URL [rtsp/rtspu]://host:port/path Reliable unreliable legal domain port used to the server protocol protocol name or IP control the stream path (TCP) (UDP) address stream *port – the actual stream will be delivered in other port Fast Forward Your Development www.dsp-ip.com

RTSP Protocol Parameters (Ctnd.) • Session ID ▫ Generated by the server ▫ Stays constant for the entire session • SMPTE – Relative timestamp ▫ A relative time from the beginning of the stream. ▫ Nested types: smpte-range, smpte-type, smpte-time. ▫ smpte-25=(starttime)-(endtime) • UTC – Absolute time ▫ Absolute time using GMT. ▫ Nested types: utc-range, utc-time. utc-date ▫ utc-time = (utcdate)T(utctime).(fraction)Z • NPT - Normal Play Time ▫ Absolute position from the beginning of the presentation. ▫ npt=123.45-125 Fast Forward Your Development www.dsp-ip.com

RTSP Session Details Initiation Handling Termination Fast Forward Your Development www.dsp-ip.com

RTSP - OPTIONS request Media URL Client Player Request ID OPTIONS – Request for information about the communication options available by the Request-URI. • CSeq – the request id, a response with the same id will be sent from the server. •Media URL – the URL of the video. •Client Player – the user agent of the client. Fast Forward Your Development www.dsp-ip.com

RTSP – OPTIONS response Response Code Available Options • All RTSP response codes are divided into 5 ranges (RFC 2326 7.1.1) : 1xx – Informational, 2xx – Success 3xx – Redirection, 4xx – Client Error, 5xx – Server Error. CSeq has the same value as the request CSeq field. • The server response will return the available methods that it supports. It May contain any arbitrary data the server want to expose. Fast Forward Your Development www.dsp-ip.com

RTSP – DESCRIBE request Description readers DESCRIBE is used to retrieve the description of the media URL and the session. The description response MUST contain all media and streaming data needed in order to initialize the session. Fields: Accept - Used to inform the server which description methods the client supports. Session Description Protocol (SDP) is highly used. Notice that CSeq field is increased by one. Fast Forward Your Development www.dsp-ip.com

RTSP – DESCRIBE response The media URL the response is referring to The description method used The length of the SDP message Description readers SDP The response will always return the details of the media. SDP details will be next Fast Forward Your Development www.dsp-ip.com

RTSP – GET_PARAMETER request GET_PARAMETER is used to retrieve information about the stream. The request can be initiated from the Client or from the Server. The request/response message body is left to server/client implementation. The parameters can be: packets received, jitter, bps or any other relevant information about the stream. Fast Forward Your Development www.dsp-ip.com

RTSP – SETUP request Transport protocol Unicast/Multicast RTP/RTSP client Track ID media port SETUP is used to specify the transport details used to stream the media. The request/response message body is left to server/client implementation. The parameters can be: packets received, jitter, bps or any other relevant information about the stream. Fast Forward Your Development www.dsp-ip.com

Transport Unicast/Multicast Unicast Last gateway The client port The server protocol server option destination ip source ip to receive port to receive media data media data SETUP response will contain the session ID. For each track ( audio/video ) a different SETUP request will be made After the response is received, a PLAY request can be made to start receiving the media stream. Fast Forward Your Development www.dsp-ip.com

RTSP – PLAY request Normal Play TIme PLAY request tells the server to start send data through the streaming details defined in the SETUP process. PLAY request maybe queued so that a PLAY request arriving while a previous PLAY request is still active is delayed until the first has been completed. Fast Forward Your Development www.dsp-ip.com

RTSP – PAUSE request Stream URL PAUSE request tells the server to pause the streaming. When the user will want to start the stream again he’ll send a PLAY request to the same URL. The request may contain time information to handle when the pause will take effect. Fast Forward Your Development www.dsp-ip.com

RTSP – TEARDOWN Description readers TEARDOWN stops the stream delivery for the URL specified. Informs the server that the client is disconnecting from it. The response will include only the response code. Fast Forward Your Development www.dsp-ip.com

RTSP – More Request types • RECORD: ▫ Initiates recording operation given a time information and stream URL. • REDIRECT: ▫ Server to Client request that informs the client he needs to switch the server he connected to. The request will contain the new server URL. • SET_PARAMETER: ▫ sends a request to change a value of the presentation stream. The response code will contain the answer. • ANNOUNCE: ▫ Can be initiated both by client/server. Informs the recipient that the SDP table of the object has changed. Fast Forward Your Development www.dsp-ip.com

Components •Containers •CODECs Fast Forward Your Development www.dsp-ip.com

File Format Movie (meta-data) Video track trak moov Audio track trak Media Data sample sample sample sample mdat frame frame Fast Forward Your Development www.dsp-ip.com

Agenda • Intro to file formats • Second Generation formats ▫ RIFF: AVI, WAV • Third Generation Containers ▫ MPEG4 FF ▫ MKV Fast Forward Your Development www.dsp-ip.com

File Format Segmentation File Formats 3rd 2nd 1st Generation Generation Generation Object Media Raw / XML Based Based Muxer Proprietary Fast Forward Your Development www.dsp-ip.com

2nd Generation File Formats Fast Forward Your Development www.dsp-ip.com

2ND Generation Files features • Multiple media track in the same file • Identification of codec ▫ Usually by FourCC • Interleaving Fast Forward Your Development www.dsp-ip.com

2nd Generation File Formats 2nd Generation FF RIFF ASF MPEG2 FLV MP2PS WAV AVI WMA WMV MP2TS VOB Fast Forward Your Development www.dsp-ip.com

AVI FILE FORMAT Fast Forward Your Development www.dsp-ip.com

AVI Overview • AVI files use the AVI RIFF format (like WAV) • Introduced by Microsoft on 1992 • File is divided into: ▫ Streams – Audio, Video, Subtitles ▫ Blocks “Chunks” - Fast Forward Your Development www.dsp-ip.com

Blocks / Chunks • A RIFF File logical unit • Chunks are identified by four letters (FOUR-CC) • RIFF file has two mandatory sub-chunks and one optional sub-chunk • Mandatory Chunks: RIFF ('AVI ' ▫ hdrl – File header LIST ('hdrl‘ 'avih'(<Main AVI Header>) ▫ movi - Media Data LIST ('strl’ ... ) . . . ) • Optional Chunk LIST ('movi‘ . . . ) ['idx1 ['idx1'<AVI Index>] ▫ idx1 - Index ) *This order is fixed Fast Forward Your Development www.dsp-ip.com

AVI main header RIFF 'AVI ' - Identifies the file as RIFF file. LIST 'hdrl' - Identifies a chunk containing sub- chunks that define the format of the data. 'avih' - Identifies a chunk containing general information about the file. Includes: ▫ dwMicrosecPerFrame - Time between frames ▫ dwMaxBytesPerSec – number of bytes per second the player should handle ▫ dwReserved1 - Reserved ▫ dwFlags - Contains any flags for the file. Fast Forward Your Development www.dsp-ip.com

Example - headers Avi file header Initial frame chunk ID chunk size format chunk ID Data rate flages Time between streams frames Total no. of frames Frame Stream header width 320 Frame height reserved Size of padding Junk chunk identifier Fast Forward Your Development www.dsp-ip.com

Example – data chunks Audio data chunk (stream 01) video data chunk (stream 00) Fast Forward Your Development www.dsp-ip.com

AVI Summary • Advantages ▫ Includes both audio and video ▫ Index-able • Disadvantage ▫ Not suited for progressive DW ▫ Very rigid format ▫ Insufficient support for: seeking, metadata multi- reference frames Fast Forward Your Development www.dsp-ip.com

3RD GENERATION FILE FORMATS Fast Forward Your Development www.dsp-ip.com

Why “Fix it”? • 2nd Generation Formats are missing: • Metadata ▫ Separate from Media ▫ Info on angle, language, Synchronization ▫ Versioning • Better Streaming Support ▫ Reduce CPU per stream ▫ Better seeking support • Better parsing ▫ XML ▫ Atom Based Fast Forward Your Development www.dsp-ip.com

Main Attributes • File format is not just a Video / Audio multiplexer • Separation between ▫ Media – Audio, Video, Images, Subtitles ▫ Metadata – Indexing, frame length, Tags Fast Forward Your Development www.dsp-ip.com

3rd Generation File Formats 3rd Generation XML Based Object Based Matruska (MKV) MOV MPEG4 FF Fragmented 3GPP FF MPEG4 FF Fast Forward Your Development www.dsp-ip.com

MPEG4 FILE FORMAT Fast Forward Your Development www.dsp-ip.com

MP4 File Format • File Structuring Concepts ▫ Separate the media data from descriptive (meta) data. ▫ Support the use of multiple files. ▫ Support for hint tracks: support of real time streaming over any protocol Fast Forward Your Development www.dsp-ip.com

Separate Metadata and Media • Key meta-information is compact ▫ The type of media present ▫ Time-scales ▫ Timing ▫ Synchronization points etc. • Enables ▫ Random access ▫ Inspection, composition, editing etc. ▫ Simplified update Fast Forward Your Development www.dsp-ip.com

Multiple file support • Use URLs to ‘point to’ media ▫ Distinct from URLs in MPEG-4 Systems • URLs use file-access service ▫ e.g. file://, http://, ftp:// etc. • Permits assembly of composition without requiring data-copy • Referenced files contain only media ▫ Meta-data all in ‘main’ file Fast Forward Your Development www.dsp-ip.com

Logical File Structure • Presentation (‘movie’) contains… • Tracks which contain… • Samples Fast Forward Your Development www.dsp-ip.com

Physical Structure—File • Succession of objects (atoms, boxes) • Exactly one Meta-data object • Zero or more media data object(s) • Free space etc. Fast Forward Your Development www.dsp-ip.com

Example Layout Movie (meta-data) Video track trak moov Audio track trak Media Data sample sample sample sample mdat frame frame Fast Forward Your Development www.dsp-ip.com

Meta-data tables • Sample Timing • Sample Size and position • Synchronization (random access) points, priority etc. • Temporal/physical order de-coupled ▫ May be aligned for optimization ▫ Permits composition, editing, re-use etc. without re-write • Tables are compacted Fast Forward Your Development www.dsp-ip.com

Multi-protocol Streaming support • Two kinds of track • Media (Elementary Stream) Tracks ▫ Sample is Access Unit • Protocol ‘hint’ tracks ▫ Sample tells server how to build protocol transmission unit (packet, protocol data unit etc.) Fast Forward Your Development www.dsp-ip.com

Track types • Visual—’description’ formats ▫ MPEG4 ▫ JPEG2000 • Audio—’description’ formats ▫ MPEG4 compressed tracks ▫ ‘Raw’ (DV) audio • Other MPEG-4 tracks • Hint Tracks (streaming) Fast Forward Your Development www.dsp-ip.com

Track Structure • Sample pointers (time, position) • Sample description(s) • Track references ▫ Dependencies, hint-media links • Edit lists ▫ Re-use, time-shifting, ‘silent’ intervals etc. Fast Forward Your Development www.dsp-ip.com

Hint Tracks • May include media (ES) data by ref. • Only ‘extra’ protocol headers etc. added to hint tracks — compact ▫ Make SL, RTP headers as needed • May multiplex data from several tracks • Packetization/fragmentation/multiplex through hint structures • Timing is derived from media timing Fast Forward Your Development www.dsp-ip.com

Hint track structure Movie (meta-data) Video track trak moov Hint track trak Sample Data sample sample hint sample hint sample mdat header header frame frame pointer pointer Fast Forward Your Development www.dsp-ip.com

Extensibility • Other media types. ▫ Non-sc29 sample descriptions (e.G. Other video). ▫ Non-sc29 track types (e.G. Laboratory instrument trace). • Copyright notice (file or track level) etc. • General object extensions (GUIDs). Fast Forward Your Development www.dsp-ip.com

Advantages • Compatibility ▫ files can be played by other companies players. Real Player with envivo plug-in. Windows media player etc. ▫ Files can be streamed by other companies streaming server Darwin Streaming Server. Quick Time Streaming Server. Fast Forward Your Development www.dsp-ip.com

Single File-Multiple data types • No need to do an export process for files, one file type is used for storage of video, audio, events, continues telemetry data from sensors and JPEG images in one file. Audio Métadonnées Video JPEG1 JPEG1 Sensor Continues data events Fast Forward Your Development www.dsp-ip.com

Single file playback • All video track of a site could be stored in one file. In order to view many cameras in a synchronized manner the MPEG-4 file format can hold all the views of multiple cameras in one file. Audio Métadonnées Video cam 1 Video cam 2 Video cam ……. Video cam N Fast Forward Your Development www.dsp-ip.com

Skimming • Skimming – shortening a long movie to its interesting points, much like creating a “promo”. For example skimming a surveillance movie of two hours to 2 minutes where there is movement and people are entering the building. • MPEG-4 FF enables the creation of skims within the file through the use of edit-list (part of the standard) without overhead. Fast Forward Your Development www.dsp-ip.com

MKV FILE FORMAT XML Based File-Format Fast Forward Your Development www.dsp-ip.com

MKV - File Format • Container file format for videos, audio tracks, pictures and subtitles all in one file. • Announced on Dec. 2002 by Steve Lhomme. • Based on Binary XML format called EBML (Extensible Binary Meta Language) • Complete Open-Standard format. (Free for personal use). • Source is licensed under GNU L-GPL. Fast Forward Your Development www.dsp-ip.com

MKV - Specifications • Can contain chapter entries of video streams • Allows fast in-file seeking. • Metadata tags are fully supported. • Multiple streams container in a single file. • Modular – Can be expanded to company special needs. • Can be streamed over HTTP, FTP, etc. Fast Forward Your Development www.dsp-ip.com

MKV Support software & hardware • Players: ▫ All Player, BS.Player, DivX Player, Gstreamer-Based players, VLC media, xine, Zoom Player, Mplayer, Media Player Classic, ShowTime, Media Player Classic and many more… • Media Centers: ▫ Boxee, DivX connected, Media Portal, PS3 Media Server, Moovida, XBMC etc. • Blu-Ray Players: ▫ Samsung, LG and Oppo. • Mobile Players: ▫ Archos 5 android device, Cowon A3 and O2. Fast Forward Your Development www.dsp-ip.com

MKV - EBML in details • A binary format for representing data in XML- like format. • Using specific XML tags to define stream properties and data. • MKV conforms to the rules of EBML by defining a set of tags. ▫ Segment , Info, Seek, Block, Slices etc. • Uses 3 Lacing mechanisms for shortening small data block (usually frames). ▫ Uses: Xiph, EBML or fixed-sized lacing. Fast Forward Your Development www.dsp-ip.com

MKV – Simple representation Type Description Header Version info, EBML type ( matroska in our case ). Meta Seek Optional, Allows fast seeking of other level 1 elements in file. Information Segment File information - title, unique file ID, part number, next file Information ID. Track Basic information about the track – resolution, sample rate, codec info. Chapters Predefines seek point in media. Clusters Video and audio frames for each track Cueing Data Stores cue points for each track. Allows fast in track seeking. Attachment Any other file relates to this. ( subtitles, Album covers, etc… ) Tagging Tags that relates to the file and for each track (similar to MP3 ID3 tags). Fast Forward Your Development www.dsp-ip.com

MKV – Streaming • Matroska supports two types of streaming. • File Access ▫ Used for reading file locally or from remote web server. ▫ Prone to reading and seeking errors. ▫ Causes buffering issues on slow servers. • Live Streaming ▫ Usually over HTTP or other TCP based protocol. ▫ Special streaming structure – no Meta seek, Cues, Chapters or attachments are allowed. Fast Forward Your Development www.dsp-ip.com

File Format Summary - Trends • Metadata is important ▫ Simple metadata or XML ▫ Separated from media • Forward compatibility ▫ Not crash if don’t understand a data entry • Progressive download oriented • Multi-bitrate oriented • Fragmentation -> Lower granularity ▫ Self contained File fragments • CDN-ability Fast Forward Your Development www.dsp-ip.com

Video Codecs Movie (meta-data) Video track trak moov Audio track trak Media Data sample sample sample sample mdat frame frame Fast Forward Your Development www.dsp-ip.com

Why Advance ? MPEG2 Works…. • Coding efficiency • Packetization • Robustness • Scalable profiles • Internet requires Interaction ▫ Scalable & On demand ▫ Fast-Forward / Fast Rewind / Random Access ▫ Stream switching • Multi ▫ Bitrate ▫ resolution /screen Fast Forward Your Development www.dsp-ip.com

Coding efficiency Motivation Fast Forward Your Development www.dsp-ip.com

Sorenson Spark video Codec • H263 variant • Low footprint (code size) ~100K • Good performance for 2002 • Quality SPARK vs Optimal MPEG (H263+) ▫ 20-30% less efficient • SPARK Quality RT vs Offline ▫ RT has Considerably lower quality due to processing power and RT (delay) constraints Fast Forward Your Development www.dsp-ip.com

Sorenson Spark - 2 • Does Not support: ▫ Arithmetic coding ▫ Advance prediction ▫ B-frames • Features ▫ De-blocking filter mode ▫ UMV - Unrestricted Motion Vector mode ▫ Arbitrary frame dimensions ▫ Supported by FFMPEG ▫ D – Frames Fast Forward Your Development www.dsp-ip.com

D-Frames • D (Disposable) frames ▫ One way prediction ▫ Provides flexible bit-rate: I-D-P-D-P-D-P ▫ D-frames used only when feeding a flash communication server Fast Forward Your Development www.dsp-ip.com

On2 TrueMotion VP6 • Features ▫ Compressed I-frames (Intra-compression makes use of spatial predictors) ▫ unidirectional predicted frames (P-frames) ▫ Multiple reference P-frames ▫ 8x8 iDCT-class transform (4x4 in VP7) ▫ improved quantization strategy (preserves image details) ▫ Advance Entropy Coding Fast Forward Your Development www.dsp-ip.com

VP6 Features • Entropy Coding ▫ various techniques are used based on complexity and frame size including: VLC Context modeled binary coding (like H264 CABAC) • Bit Rate Control ▫ To reach the requested data rate, VP6 adjusts Quantization levels Encoded frame dimensions Entropy Coding Drop frames Fast Forward Your Development www.dsp-ip.com

VP6 motion prediction • Motion Vectors ▫ One vector per MacroBlock (16x16) or ▫ 4 vectors for each block (8x8) • Quarter pel motion compensation support • Unrestricted motion compensation support • Two reference frames: ▫ The previous frame or ▫ Previously bookmarked frame Fast Forward Your Development www.dsp-ip.com

VP6 vs H264 • VP6 is much simpler than H.264 ▫ Requires less CPU resourced for decoding & encoding ▫ Code size is considerably smaller. • Simpler means less efficient? NO! Techniques used: ▫ Mix of adaptive sub-pixel motion estimation ▫ Better prediction of low-order frequency coefficients ▫ Improved quantization strategy ▫ de-blocking and de-ringing filters ▫ Enhanced context based entropy coding, Fast Forward Your Development www.dsp-ip.com

PSNR Graphs are used for comparative 720p High Profile H.264 vs VP7 This axis represents quality. Higher is better analysis of compression quality. Each line represents the encode quality on a given clip at multiple datarates. The highest line Draw a line straight represents the codec withAlexander Trailer the best quality. across until you intersect In this case VP7 clearly is better than x264. 47 the lower line ( in this Pick any point on case x264. i.e. keep the 46.5 Tips for reading this kind of a the top line, in this quality/ psnr constant ) 46 graph (a PSNR graph): case it’s VP7. What this means: 45.5 On this clip VP7 at 2750 kbps has the same quality / PSNR as x264 high profile 45 at 3620 kbps. i.e. you’d need 30%Draw a line straight higher PSNR Draw a line straight 44.5 datarate to get the same quality out of from that point down from that point down Vp7 to the datarate axis. that you got from vP7. x264 x264 to the datarate axis. 44 The crossing point tells The crossing point tells 43.5 you the datarate at that you the datarate at that point. point. 43 42.5 1400 1900 2400 2750 kbps 2900 3400 3620 kbps 3900 4400 Kbps This axis represents datarate in kilobits per second. Fast Forward Your Development www.dsp-ip.com

VP6 vs. H264 • There is a difference between the codec technology and a codec implementation. Fast Forward Your Development www.dsp-ip.com

On2 VP7 • Not open source • Non-standard royalties model • Better video quality than H264 • Used by: ▫ Part of EVD – China standard for HD-DVD ▫ Skype Beta (V 2.0) ▫ Flash Player Fast Forward Your Development www.dsp-ip.com

Windows Media • Windows media is a format used by Microsoft for encoding and distributing Audio and Video. • Windows Media has two types of media: ▫ Windows Media Audio (WMA) ▫ Windows Media video (WMV) • Windows Media Video ▫ A modified version of MPEG 4 ▫ Codec version has initially started from version 7 for windows media player 7 and then evolved to version 8-10 Fast Forward Your Development www.dsp-ip.com

Windows Media 9 - VC1 Format • Microsoft has submitted Version 9 codec to the Society of Motion Picture and Television Engineers (SMPTE), for approval as an international standard. SMPTE is reviewing the submission under the draft-name "VC-1") • This codec is also used to distribute high definition video on standard DVDs in a format Microsoft has branded as WMV HD. This WMV HD content can be played back on computers or compatible DVD players. • The Trial version of standards were published by SMPTE in September 2005 • WMV9 was approved by SMPTE, April 2006 Fast Forward Your Development www.dsp-ip.com

H.264 Fast Forward Your Development www.dsp-ip.com

H.264 Terminology • The following terms are used interchangeably: ▫ H.26L ▫ “JVT CODEC” ▫ The “AVC” or Advanced Video CODE • Proper Terminology going forward: ▫ MPEG-4 Part 10 (Official MPEG Term) ISO/IEC 14496-10 AVC ▫ H.264 (Official ITU Term) Fast Forward Your Development www.dsp-ip.com

H264 Standard ideas • “Blocks” size fixed ->Variable ▫ Slice ▫ Block • Block Size order/scanning –> different orders ▫ Zig-zag, Flexible Macroblock Order • Additional spatial prediction - >Intra prediction • Inter prediction 1 frame only ->Multiple frames ▫ P and B picture ▫ Multiple reference frame Fast Forward Your Development www.dsp-ip.com

H264 Standard Ideas ▫ Pixel interpolation ▫ Motion vectors • In-loop Deblocking filter • Improved Entropy coding Fast Forward Your Development www.dsp-ip.com

New Features of H.264 - summarized • SP, SI - Additional picture types • NAL (Network Abstraction Layer) • CABAC - Additional entropy coding mode • ¼ & 1/8-pixel motion vector precision • In-loop de-blocking filter • B-frame prediction weighting • 4×4 integer transform • Multi-mode intra-prediction • NAL - Coding and transport layers separation • FMO - Flexible MacroBlock ordering. Fast Forward Your Development www.dsp-ip.com

Block diagram Fast Forward Your Development www.dsp-ip.com

Profiles and Levels • Profiles: Baseline, Main, and X ▫ Baseline: Progressive, Videoconferencing & Wireless ▫ Main: esp. Broadcast ▫ Extended: Mobile network • Wireless <> Mobile Fast Forward Your Development www.dsp-ip.com

Fast Forward Your Development www.dsp-ip.com

Baseline Profile • Baseline profile is the minimum implementation ▫ No CABAC, 1/8 MC, B-frame, SP-slices • 15 levels ▫ Resolution, capability, bit rate, buffer, reference # ▫ Built to match popular international production and emission formats ▫ From QCIF to D-Cinema • Progressive (not interlaced) • I and P slices types Fast Forward Your Development www.dsp-ip.com

Baseline Profile • 1/4-sample Inter prediction • Deblocking filter, Redundant slices • VLC-based entropy coding (no CABAC) • 4:2:0 chroma format • Flexible Macroblock Ordering (FMO) • Arbitrary Slice Order (ASO) ▫ Decoder process slices in an arbitrary order as they arrive to the decoder. ▫ The decoder dose not have a wait for all slices to be properly arranged before it starts processing them. ▫ Reduces the processing delay at the decoder. Fast Forward Your Development www.dsp-ip.com

Baseline Profile • FMO: Flexible Macroblock Ordering ▫ With FMO, macroblocks are coded according to a macroblock allocation map that groups, within a given slice. ▫ Macroblocks from spatially different locations in the frame. ▫ Enhances error resilience • Redundant slices: ▫ allow the transmission of duplicate slices. Fast Forward Your Development www.dsp-ip.com

H.264 Profiles & Levels - Main • All Baseline features Plus Interlace B slice types (bi directional reference ) CABAC Weighted prediction • All features included in the Baseline profile except: Arbitrary Slice Order (ASO) Flexible Macroblock Order (FMO) Redundant Slices Fast Forward Your Development www.dsp-ip.com

Main Profile • CABAC • Good performance (bit rate reduction) by ▫ Selecting models by context ▫ Adapting estimates by local statistics ▫ Arithmetic coding reduces computational complexity • Improve computational complexity more than 10%~20% of the total decoder execution time at medium bitrate • Average bit-rate saving over CAVLC 10-15% Fast Forward Your Development www.dsp-ip.com

Extended Profile • All Baseline features plus ▫ Interlace ▫ B slice types ▫ Weighted prediction Fast Forward Your Development www.dsp-ip.com

Variable block size Slices A picture split into 1 or several slices Slices are a sequence of macroblocks Macroblock Contains 16x16 luminance samples and two 8x8 chrominance samples Macroblocks within a slices depend on each others Macroblocks can be further partitioned Slice 0 Slice 1 Slice 2 Fast Forward Your Development www.dsp-ip.com

Basic Marcoblock Coding Structure Input Coder Video Control Control Signal Data Transform/ Quant. Scal./Quant. - Transf. coeffs Decoder Scaling & Inv. Split into Macroblocks Transform 16x16 pixels Entropy Coding De-blocking Intra-frame Filter Prediction Output Motion- Video Compensation Signal Intra/Inter Motion Data Motion Estimation Fast Forward Your Development www.dsp-ip.com

Motion Compensation Input Coder Video Control Control Signal Data Transform/ Quant. Scal./Quant. - Transf. coeffs Decoder Scaling & Inv. Split into Macroblocks Transform 16x16 pixels Entropy Coding De-blocking 16x16 16x8 8x16 8x8 Intra-frame MBFilter 0 0 1 Prediction Types 0 0 1 Output1 2 3 Motion- Video Compensation 8x8 8x4 Signal 4x8 4x4 Intra/Inter 8x8 0 0 1 0 0 1 Types Motion 1 2 3 Data Motion Various block sizes and shapes Estimation Fast Forward Your Development www.dsp-ip.com

Variable block size Block sizes of 0 0 1 16x8, 8x16, 8x8, 8x4 , 4X8 and 0 0 1 2 3 1 4X4 are available. Mode 1 Mode 2 Mode 3 Mode 4 1 16x16 block 2 16x8 blocks 2 8x16 blocks 4 8x8 blocks 0 1 0 1 2 3 Using seven different 0 1 2 3 2 3 4 5 6 7 block sizes can translate 4 5 8 9 1 1 into bit rate savings of 4 5 6 7 0 1 6 7 more than 15% as 1 2 1 3 1 4 1 5 compared to using only a Mode 5 Mode 6 16x16 block size. 8 8x4 blocks 8 4x8 blocks Mode 7 16 4x4 blocks Fast Forward Your Development Yossi Cohen www.dsp-ip.com DSP-IP

How to select the partition size? The partition size that minimizes the coded residual and motion vectors Fast Forward Your Development www.dsp-ip.com

Inter prediction modes motion vectors • MVs for neighboring partitions are often highly correlated. • So we encode MVDs instead of MVs • MVD = predicted MV – MVp • ¼ pixel accurate motion compensation Fast Forward Your Development www.dsp-ip.com

Multiple Reference Frames Input Coder Video Control Control Signal Data Transform/ Quant. Scal./Quant. - Transf. coeffs Decoder Scaling & Inv. Split into Macroblocks Transform 16x16 pixels Entropy Coding De-blocking Intra-frame Filter Prediction Output Motion- Video Compensation Signal Intra/Inter Motion Multiple Reference Data Frames for Motion Motion Compensation Estimation Fast Forward Your Development www.dsp-ip.com

Multiple Reference Frames Fast Forward Your Development www.dsp-ip.com

Intra prediction modes 4x4 luminance prediction modes 0(vertical) 1(Horizontal) 2(DC) 3(Diagonal 4(Diagonal Down/left) Down/right) 5(Vertical-right) 6(Horizontal-down) 7(Vertical-left) 8(Horizontal-top) Mode 2 (DC) Predict all pixels from (A+B+C+D+I+J+K+L+4)/8 or (A+B+C+D+2)/4 or (I+J+K+L+2)/4 Fast Forward Your Development www.dsp-ip.com

Intra prediction modes 4x4 luminance prediction modes Fast Forward Your Development www.dsp-ip.com

Intra prediction modes Intra 16x16 luminance and 8x8 chrominance prediction modes Fast Forward Your Development www.dsp-ip.com

Inter prediction modes chrominance Pixel interpolation A B Quarter chrominance Pixels are interpolated by tacking weighted dy dx averages of distance from the new S-dx pixel to four surrounding original S-dy pixels. C D (s-dx)(s-dy)A+dx(s-dy)B+(s-dx)dyC+dxdyD+s2/2 V= S2 Fast Forward Your Development www.dsp-ip.com

In-loop De-blocking Filter • Highly compressed decoded inter picture • Significantly reduces prediction residuals Without filter with H.264/AVC De-blocking Fast Forward Your Development www.dsp-ip.com

Entropy coding Fast Forward Your Development www.dsp-ip.com

Entropy coding • Entropy coding methods: • CABAC - Discussed • UVLC ▫ H.264 offers a single Universal VLC (UVLC) table for all symbol • CAVLC ▫ CAVLC (Context-based variable Length Coding ) ▫ Probability distribution is static ▫ Code words must have integer number of bits (Low coding efficiency for highly peaked pdfs) Fast Forward Your Development www.dsp-ip.com

CABAC - Binarization Fast Forward Your Development www.dsp-ip.com

CABAC: Technical Overview update probability estimation Context Binarization Probability Coding modeling estimation engine Adaptive binary arithmetic coder Chooses a model Maps non-binary Uses the provided model conditioned on symbols to a for the actual encoding past observations binary sequence and updates the model Fast Forward Your Development www.dsp-ip.com

H.264 NAL & RTP Fast Forward Your Development www.dsp-ip.com

H264 Layer Structure Control Data Video Coding Layer Macroblock Data Partitioning Slice/Partition Network Abstraction Layer H.320 H.324 H.323/IP MPEG2 Fast Forward Your Development www.dsp-ip.com

H264 & NAL • Motivation ▫ Many delivery methods are based on packet based networks ▫ Its better to do the packetization inside the encoder where all coding information than in other separate modules • Architecture: NAL units as the transport entity ▫ NAL units may be mapped into a bit stream ▫ NAL units are self-contained-independently decodable ▫ The decoding process assumes NAL units are in decoding order Fast Forward Your Development www.dsp-ip.com

Network Abstraction Layer (NAL) • H.264 encoder is composed of two layers: • VCL - Video Coding Layer – unit which translates the video information into a stream of bits • NAL - Network Abstraction Layer (NAL). Which maps and packetize the VCL biststream into unitsprior to transmission or storage • Each NAL unit contains: ▫ Payload – RBSP (Raw Byte Sequence Payload), which contains set of data corresponding to coded video data or header information ▫ NAL Unit Header – which contains the NAL header Fast Forward Your Development www.dsp-ip.com

NAL • The coded video sequence is represented by a sequence of NAL units that can be transmitted over a packet-based network or a bitstream transmission link or stored in a file • There are two NAL Types ▫ VCL Units – NAL units which represents encoded video data ▫ Non-VCL Units – Parameters sets Fast Forward Your Development www.dsp-ip.com

NAL Unit Header NAL unit header NAL unit payload NAL unit header- 1 byte consisting of: • Forbidden_bit(1 bit) may be used to signal that a NAL unit is corrupt • nal_storage_idc(2 bit): signal relative importance, and if the picture is stored in the reference picture buffer. • nal_unit_type(5 bit): signal 1 of 10 different NAL unit types: ▫ Coded slice (regular VCL data) ▫ Coded data partition. (DPA,DPB,DPC) ▫ Instantaneous decoder refresh (IDR) ▫ Supplemental enhancement information (SEI) ▫ Sequence and picture parameter sets (SPS,PPS) ▫ Picture delimiter (PD) and filler data (FD) Fast Forward Your Development www.dsp-ip.com

RBSP (NAL Payload) Types • Parameter Set – global parameters for a sequence includes: Resolution, video format, macroblock, allocation map • Supplemental Enhancement Information • Picture Delimiter – boundary between video pictures • Coded slice – header and data for a slice, this unit contains actual coded video data • Data Partition A,B or C – Data Partitioned slice layer data (A – header data for all MBs in the slice, B – intra coded data, C – inter coded data) • End of sequence • End of stream • Filler data Fast Forward Your Development www.dsp-ip.com

RTP payload format for H.264 • Based on IETF RFC 3984, February 2005 • Described how to use H.264 NAL inside RTP with proper packetization Employs the native NAL (Network Abstraction Layer) interface, based on NAL units (NALUs) • NALU – byte string of variable length that contains syntax elements of a certain class • NALU header – defines the information within the NAL Unit (Corrupted, Type etc) • There are two basic methods for RTP packetization of NAL units: ▫ Non fragmented NAL units ▫ Fragmented NAL units Fast Forward Your Development www.dsp-ip.com

RTP Payload for H.264 NAL • The most common method is to configure the encoder to output one NAL unit for each RTP packet. Each NAL unit is ~1.4KB • Fragment a large NAL unit (Frame) into many RTP Packets. The difference is in the RTP Header information Fast Forward Your Development www.dsp-ip.com

RTP and H.264 • RTP Packetization of NAL allows both aggregation of many NAL units into one RTP Packet and fragmentation of one NAL units into many RTP packets Fast Forward Your Development www.dsp-ip.com

Comparison Fast Forward Your Development www.dsp-ip.com

Summary • New key features are: ▫ Enhanced motion compensation ▫ Small blocks for transform coding ▫ Improved de-blocking filter ▫ Enhanced entropy coding • Substantial bit-rate savings (up to 50%) relative to other standards for the same quality • The complexity of the encoder triples that of the prior ones • The complexity of the decoder doubles that of the prior ones Fast Forward Your Development www.dsp-ip.com

Google VP8 Fast Forward Your Development www.dsp-ip.com

Before we start • VP8 goal is NOT to delivery the best video quality in any given bitrate • VP8 was designed as a mobile video decoder and should be examined in this context: ▫ VP8 vs H.264 base profile Fast Forward Your Development www.dsp-ip.com

Google VP8 • Last month, in Google IO (its developer confrence), Google released VP8 as open source • VP8 is a light weight video codec developed by On2. • VP8 provide quality which is the same/higher than H.264 base profile • VP8 memory requirements are lower than H.264 base profile • After optimization, VP8 might have better MIPS performance than H.264 base profile Fast Forward Your Development www.dsp-ip.com

Genealogy • VP8 is part of a well know codec family • VP3 was released to open source to become XIPH Theora Sorenson • VP6 is used in Flash video Spark • VP7 is used in Skype VP3 • Motivation: Theora ▫ “No Royalties” CODEC VP7 VP6 VP8 Fast Forward Your Development www.dsp-ip.com

ADAPTATION – WHO USE IT? Software Hardware Platform & Publishers Fast Forward Your Development www.dsp-ip.com

Software Adaptation • Android, Anystream, Collabora • Corecodec, Firefox, Adobe Flash • Google Chrome, iLinc, • Inlet, Opera, ooVoo • Skype, Sorenson Media • Theora.org, Telestream, Wildform. Fast Forward Your Development www.dsp-ip.com

Hardware adaptation • AMD, ARM, Broadcom • Digital Rapids, Freescale • Harmonic ,Logitech, ViewCast • Imagination Technologies, Marvell • NVIDIA, Qualcomm, Texas Instruments • VeriSilicon, MIPS Fast Forward Your Development www.dsp-ip.com

Platforms and Publishers • Brightcove • Encoding.com • HD Cloud • Kaltura • Ooyala • YouTube • Zencoder Fast Forward Your Development www.dsp-ip.com

VP8 MAIN FEATURES (According to On2/Google) Fast Forward Your Development www.dsp-ip.com

Adaptive Loop Filter • Improved Loop filter provides better quality & performance in comparison to H.264 • ALF is a feature of H.265 Source: On2 Fast Forward Your Development www.dsp-ip.com

Golden Frames • Golden frames enables better decoding of background which is used for prediction in later frames • Could be used as resync-point: ▫ Golden frame can reference an I frame • Could be hidden (not for display) Fast Forward Your Development www.dsp-ip.com Source: On2

Decoding efficiency • CABAC is an H.264 feature which improves coding efficiency but consumes many CPU cycles • VP8 has better entropy coding than H.264, this leads to relatively lower CPU consumption under the same conditions • Decoding efficiency is important for smooth operation and long battery life in netbooks and mobile devices Fast Forward Your Development www.dsp-ip.com Source: On2

Resolution up-scaling & downscaling • Supported by the decoder • Encoder could decide dynamically (RT applications) to lower resolution in case of low bit rate and let the decoder scale. • Remove decision from the application • No need for an I frame Fast Forward Your Development www.dsp-ip.com

VP8 BASICS Definitions Bitstream structure Frame structure Fast Forward Your Development www.dsp-ip.com

Definitions • Frame – same as H.264 • Segment – Parallel to slice in H.264. MB in the same segment will use the settings such as: ▫ Entropy encoder/decoder context ▫ De-blocking filter settings • Partition – block of byte aligned compressed video bits. Fast Forward Your Development www.dsp-ip.com

Definitions • Block – 8x8 matrix of pixels • Macro-block –processing unit, contains a 16x16 Y pixels, and 2 8x8 matrix of U and V: ▫ 4* 8x8Y block ▫ 1* 8x8U block ▫ 1* 8x8V block • Sub-block – 4x4 matrix of pixels. All DCT / WHT operations are done on sub-blocks. Fast Forward Your Development www.dsp-ip.com

Frame Types • I Frame • P Frame • No B Frames due to patents / delays but we have “Future Alt-Ref” frame. What is the diff? • Prediction ▫ Previous frame ▫ Golden Frame ▫ Alt-ref frame Fast Forward Your Development www.dsp-ip.com

Frame Structure • Include three sections: • Frame Header • Partition I • Partition II Frame Header Partition I Partition II partitions Fast Forward Your Development www.dsp-ip.com

Frame Header • Byte aligned uncompressed information • Frame type - 1-bit frame type ▫ 0 for key frames, 1 for inter-frame. • Level - A 3-bit version number ▫ 0 - 3 are defined as four different profiles with different decoding complexity; other values for future use • show_frame - A 1-bit show_frame flag ▫ 0 – current frame not for display ▫ 1 - current frame is for display • Length - A 19-bit field containing the size of the first data partition in bytes. Fast Forward Your Development www.dsp-ip.com

Partition I • Header information for the entire frame • Per-macroblock information specifying how each macroblock is predicted. • This information is presented in raster-scan order Fast Forward Your Development www.dsp-ip.com

Partition II • Texture information - DCT/WHT quantized coefficients • Optionally each macroblock row could be mapped to a separate partition. • Partition II might be divided to several partitions for parallel processing Frame Header Partition I Partition IIA Partition IIB Partition IIn Texture Data Fast Forward Your Development www.dsp-ip.com

Decoder • Holds 4 frames: ▫ Current remonstrated frame ▫ Previous frame ▫ Previous “Golden Frame” ▫ Previous Alt-ref frame • Frame dimension can change in every frame Fast Forward Your Development www.dsp-ip.com

VP8 block diagram Input Coder Video Control Control Signal Data Transform/ Quant. Scal./Quant. - Transf. coeffs Decoder Scaling & Inv. Split into Macroblocks Transform Entropy Coding Dynamic Intra-frame De-blocking Prediction Output Motion- Video Compensation Intra/Inter Motion Data Motion Estimation Fast Forward Your Development www.dsp-ip.com

VP8 BLOCK CODING Fast Forward Your Development www.dsp-ip.com

VP8 Macroblock coding DC/AC Coeff 4x4 Divide to Divide to Process as DCT 16x16 8x8 4x4 Macroblock blocks sub blocks 4x4 WHT Each Macroblock is divided into 25 sub-blocks •16Y sub-blocks •4 U sub-blocks, •4 V sub-blocks •1 Y2 DC values sub-block (WHT) aka Hierarchical transform Fast Forward Your Development www.dsp-ip.com

VP8 Transforms • Very inefficient raw implementation of the transform – uses 16bit multipliers • Uses exact values of pixels ▫ +Memory ▫ +Accuracy and no drift static const int cospi8sqrt2minus1 = 20091; //sqrt(2) * cos(pi/8) static const int sinpi8sqrt2 = 35468; //sqrt(2) * sin (pi/8) temp1 = (ip[4] * sinpi8sqrt2 + rounding) >> 16; Fast Forward Your Development www.dsp-ip.com

iDCT • Like in H.264 or VP6 the 2D transform is done by two separable 1D transforms (LLM method) there are two methods of implementation: 1. Transpose, vertical T, Transpose, vertical T. 2. Vertical T., transpose, vertical T, transpose. • Duo to SIMD command its better to do it as 1 and eliminate the first transpose like H.264 does • VP8 does it as 2 (Why?!) wasted 1-2% CPU Fast Forward Your Development www.dsp-ip.com

Quantization • There are 6 quantizer types each has its own levels • The quantizer depends on (multiplication of) ▫ Plane: Y,U, V ▫ Coefficient AC, DC • Quantizer level is indicated by a 7 digit number which is an entry into one of the 6 quantization levels Fast Forward Your Development www.dsp-ip.com

VP8 PREDICTION Inter-prediction Intra prediction Fast Forward Your Development www.dsp-ip.com

Macroblock Intra Prediction • Intra-prediction exploits the spatial coherence between Macro-blocks without referring to other frames. • Modes ▫ Same as H.264 in i16x16 and i4x4 ▫ Missing modes like i8x8 which exists in H.264 Fast Forward Your Development www.dsp-ip.com

Intra prediction - blocks used Not Relevant Not Available Not Available M Not Available Not Available Not Available Not Available Fast Forward Your Development www.dsp-ip.com

Luma i4x4 Intra Prediction • 4x4 block are predicated by ▫ Orig four 16x16 prediction methods ▫ six “diagonal” prediction methods Diagonal Down/left Diagonal Down/right Vertical-right Horizontal-down Vertical-left Horizontal-top Fast Forward Your Development www.dsp-ip.com

8x8 Chroma prediction modes • U,V, Y prediction are done separately and one channel prediction does not affect the other channels. Fast Forward Your Development www.dsp-ip.com

Inter-frame prediction • Definition - Inter-prediction exploits the temporal coherence between frames to save bitrate. • Luma sub-block prediction ▫ Method - each Y 4x4 sub-blocks is related to a 4x4 sub-block of the prediction frame. ▫ Precision – motion vectors precision is q-pel. ▫ interpolation pixel is calculated by applying a kernel filter three pixels horizontally and vertically. ▫ Based on deltas from previous MVs “NEARST” and NEAR modes Fast Forward Your Development www.dsp-ip.com

Inter-frame prediction • SPLITMV mode ▫ Supports up to 16 MV inside a single MB ▫ MV reuse (delat) is done not only on MB level but also on sub-block level • Example: Partition of Block into 3 MV areas 16 Fast Forward Your Development www.dsp-ip.com 3

Inter-frame prediction - Chroma • Chroma prediction - motion vector for each 8X8 chroma block is calculated separately by one of four prediction methods listed below: 1. Vertical - Copying the row from above throughout the prediction buffer. 2. Horizontal - Copying the column from left throughout the prediction buffer. 3. DC - Copying the average value of the row and column throughout the prediction buffer. 4. Extrapolation from the row and column using the (fixed) second difference (horizontal and vertical) from the upper left corner. Fast Forward Your Development www.dsp-ip.com

Inter-frame Prediction - Chroma • Chroma precision - the calculated chroma motion vectors have 1/8 pixel resolution • averaging the vectors of the four Y sub-blocks that occupy the same area of the frame. Fast Forward Your Development www.dsp-ip.com

Entropy Coding • Entropy coding is based on binary tree like CABAC but unlike H.264 its not context adaptive like in H.264 and does not adapt on every operation • Tables does not change all the time • Unlike H.264 which makes all Symbol->Bit decision as text, VP8 has a tree which represent the transform of each symbol -> faster implementation (not faster processing time) 16 Fast Forward Your Development www.dsp-ip.com 6

Entropy Coding • According to X264Developer “probability values are constant over the course of the frame” • According to Google document, there could be up to 4 “Entropy coding context” in a frame • We can view this as “switched context BAC” and not as CABAC • Obviously “switched context BAC”<<CABAC on a CPU level • However setting the context for each fragment is hard to implement in hardware 16 Fast Forward Your Development www.dsp-ip.com 7

PARALLEL PROCESSING Segment Partition Fast Forward Your Development www.dsp-ip.com

Segment Processing • Segmentation enables creation of MB groups within one logical unit. • MB are associated with a segment by the MB Segment ID • All MBs in a segment has the same adaptive adjustments which includes: ▫ Same Quantization level ▫ Loop filter strength (0-2) Fast Forward Your Development www.dsp-ip.com

Frame Processing Architecture • Frame Header and Partition I are processed first to initialize probabilistic decoder and prediction scheme for each MB. A Serial operation • Each sub-partition might be processed in parallel to other partitions. probabilistic model of one sub-partition does not interact with another sub- partition Frame Length Partition Partition Partition Partition I Header IIA-IIn-1 IIA IIB IIn Sub-partition Fast Forward Your Development www.dsp-ip.com

VP VS H.264 DIFFRENCES Fast Forward Your Development www.dsp-ip.com

H.264 Loop Filter • H.264 Loop filter strength depends on boundary strength, MB type 17 Fast Forward Your Development www.dsp-ip.com 2

VP8 Loop Filter • Lop filter is adjusted by 6 bit variable on two levels: ▫ Global Frame Loop filter level ▫ Per MB level • VP8 supports two filter types. • Per MB level is set as “delta” from frame filter level • L1 – enable/Disable Loop filter in code mode_ref_lf_delta_update 17 Fast Forward Your Development www.dsp-ip.com 3

VP8 Loop filter complexity • Before final optimization 70% of CPU time is deblocking filter!!! According to http://x264dev.multimedia.cx/?p=486 • a 4×4 transform requires a total of 8 length-16 and 8 length-8 loopfilter calls per macroblock while theora 8x8 filter requires half the amount of calls 17 Fast Forward Your Development www.dsp-ip.com 4

AltRef Noise Reduction • One of the usages of Alt-Ref frame is for noise reduction • For noisy sources using a filtered image as reference improves compression (so is pre- processing) • Improvement 0.25DB for noisy clip • Example with noise reduction set to 5 17 Fast Forward Your Development www.dsp-ip.com 5

AltRef Frame Future Frame • Just don’t call it a B-Frame and it will give you 1db 17 Fast Forward Your Development www.dsp-ip.com 6

H.264 vs VP8 Transforms differences • No 8x8 transforms • H.264 simplifies the DCT transform and implement it as a series of: 1. Add 2. Subtract 3. Right shift by 1 It sacrifice accuracy (~1%) for speed/CPU VP8 uses large,16bit multipliers for accuracy (20091, 35468), this is redundant unlike VC which uses small multiplier 17 Fast Forward Your Development www.dsp-ip.com 7

H.264 vs VP8 Transforms differences • Unlike H.264 which de-corellates DC values (Hadamard transform) ONLY in Intra i116x16 MB, VP8 uses this method also on some p16x16 MBs 17 Fast Forward Your Development www.dsp-ip.com 8

VP8 / H.264 summary • “Golden Frames” – exist at some level in H.264 slice group map type 2 • Slice granularity is better in H.264 ▫ support MB instead of MB line • Interlacing – not supported • AltRef Frame – a reference frame which is not displayed. Could aggregate all sorts of useful MBs • Filter – VP has an adaptive complex and slow filter 17 Fast Forward Your Development www.dsp-ip.com 9

Install • Visual Studio 2010 version – I couldn’t make it work • Download Visual Studio 2005/2008 code version • Download and Install YASM • Follow instructions on http://www.tortall.net/projects/yasm/wiki/Visu alStudio2005 to integrate with Visual Studio 2005/2008 (copy files and set paths) 18 Fast Forward Your Development www.dsp-ip.com 0

Basic Decoder work • Compile IVFDEC Project • Set parameters ▫ --codec=vp8 -o a.i420 qcif1.ivf • Compile IVFEnc Project ▫ --codec=vp8 --i420 -v a.i420 qcif1.ivf • Check that the decoder unders 18 Fast Forward Your Development www.dsp-ip.com 1

COMPARISON (FINALLY) Fast Forward Your Development www.dsp-ip.com

Talking heads, Low motion • Low motion videos like talking heads are easy to compress, so you'll see no real difference Fast Forward Your Development www.dsp-ip.com

Low motion In another low motion video with a terrible background for encoding (finely detailed wallpaper), the VP8 video retains much more detail than H.264. Interesting result. Fast Forward Your Development www.dsp-ip.com

Medium motion VP8 holds up fairly well Fast Forward Your Development www.dsp-ip.com

High motion • In high motion videos, H.264 seems superior. In this sample, blocks are visible in the pita where the H.264 video is smooth. The pin-striped shirt in the right background is also sharper in the H.264 video, as is the striped shirt on the left. Fast Forward Your Development www.dsp-ip.com

Very High motion In this very high motion skateboard video, H.264 also looks clearer, particularly in the highlighted areas in the fence, where the VP8 video has artifacts. Fast Forward Your Development www.dsp-ip.com

Final In the final comparison, I'd give a slight edge to VP8, which was clearer and showed fewer artifacts. Fast Forward Your Development www.dsp-ip.com

Quality Comparison Fast Forward Your Development www.dsp-ip.com

Test yourself 1. Why VP8 is less effective in high motion? 2. Is it patent free? 3. Will you use it? Fast Forward Your Development www.dsp-ip.com

Adobe Flash Video Fast Forward Your Development www.dsp-ip.com www.dsp-ip.com

Flash File Format Web Page File type: .htm, Flash video / .html, .asp, FLV etc File type: .flv Flash Movie / SWF Includes: File type: .swf Video Includes: Plays in: Graphics, text, video .swf Video controls Client logic Served from: Plays in: Web server Flash Player Flash Served from: Communicati Web server on Server Synonyms: Application Video player Fast Forward Your Development www.dsp-ip.com

FLV (Flash Video) File Format • Headers in the beginning of the file. Why? • FLV and MP4 File Formats • Video support: ▫ SPARK ▫ TrueMotion VP6 ▫ H264 and VP8 (future support) • Audio ▫ Nelly Moser codec by ASAO ▫ MP3 Codec ▫ ADPCM (not compressed) • Alpha Channel Fast Forward Your Development www.dsp-ip.com

SWF File • Vector Graphic Format • Container file • Includes FLV files • Action Scripts • Players Fast Forward Your Development www.dsp-ip.com

SWF and Personalized Ads • Encoding a special video version for each user is: ▫ Expensive ▫ Degrades the video • SWF Enables: ▫ Using one video version stored in FLV ▫ Changing the text or image stored in the outer SWF per user Video & Images, Audio Scripts, Personalized special Video FLV File Effects Fast Forward Your Development www.dsp-ip.com

HTTP Live Streaming by Apple Fast Forward Your Development www.dsp-ip.com

Agenda • System Overview • Components • Session Fast Forward Your Development www.dsp-ip.com

Apple’s Note Note: Many existing streaming services require specialized servers to distribute content to end users. It requires specialized skills to set up and maintain these servers, and in a large-scale deployment these servers can be costly. Apple has designed a system that avoids this by using standard HTTP to deliver the streams. Fast Forward Your Development www.dsp-ip.com

System Overview Fast Forward Your Development www.dsp-ip.com

Components Review • Server ▫ Encoder ▫ Segmenter • Distributer ▫ Basic HTTP Server • Client Fast Forward Your Development www.dsp-ip.com

Server • Receives Digital / Analog input stream • Encodes / Transcode video/audio ▫ H.264 Video ▫ AAC audio (HE-AAC or AAC-LC) • Encodes / Transcode audio only: ▫ MPEG-2 elementary streams, HE-AAC or AAC-LC files, or MP3 files • Encapsulate in MPEG2 ▫ Transport Stream ▫ Program Stream Fast Forward Your Development www.dsp-ip.com

Segmenter • All segments should be with the same duration • All segments are placed in a separate file • Creates Index file with references to segment files • For protection, the Segmenter might encrypt each media segment and create a key file Fast Forward Your Development www.dsp-ip.com

Distribution • Distribution system is a regular HTTP Server • Could be Apache or small embedded Server Fast Forward Your Development www.dsp-ip.com

Files • Segments – stored as *.ts files • Index files – stored as *.m3u8 • Index file format example: #EXTM3U #EXT-X-TARGETDURATION:10 #EXTINF:10, http://media.example.com/segment1.ts #EXTINF:10, http://media.example.com/segment2.ts #EXTINF:10, http://media.example.com/segment3.ts #EXT-X-ENDLIST Fast Forward Your Development www.dsp-ip.com

Session types • Live Stream Broadcast ▫ Index file is continues updated ▫ Include a moving window of segments around “live” part of the session ▫ Client should continuously refresh the Index file • VoD Session ▫ Index file static ▫ Includes ALL the segments of the file ▫ Enables “Seek” operation Fast Forward Your Development www.dsp-ip.com

Multi-bitrate multi–device support • Multi-bitrate is enabled via multiple index files • Index files are pointed by a global index files • Client can select a stream according to: ▫ Device properties ▫ Available bit rate • This method is less efficient than Silverlight Global File Fast Forward Your Development www.dsp-ip.com

Test yourself 1. What are the two Live streaming file types? 2. What is the role of the Segmenter? 3. On which delivery protocol is the live streaming based? Fast Forward Your Development www.dsp-ip.com

Silverlight Smooth Streaming Fast Forward Your Development www.dsp-ip.com

SILVERLIGHT INTRODUCTION Fast Forward Your Development www.dsp-ip.com

Smooth Streaming • Microsoft’s implementation of HTTP-based adaptive streaming • A hybrid media delivery method that acts like streaming but is in fact a series of short progressive downloads • Leverages existing HTTP caches • Client can seamlessly switch video quality and bit rate based on perceived network bandwidth and CPU resources Fast Forward Your Development www.dsp-ip.com

Streaming or Progressive Download? Traditional Progressive Streaming Download • Responsive User • Works from a Experience Web Server • Bandwidth Use • World-wide • User Tracking scale w/HTTP Challenges Challenges • No cache-ability • Limited User • Separate, Experience smaller • User tracking streaming • Bandwidth Use networks (20% watched) Fast Forward Your Development www.dsp-ip.com

Smooth Streaming Design • Smooth Streaming File Format based on MP4 (ISO Base Media File Format) • Video is encoded and stored on disk as one contiguous MP4 file ▫ Separate file for each bit rate • Each video Group of Pictures (GOP) is stored in a Movie Fragment box ▫ This allows easy fragmentation at key frames • Contiguous file is virtually split up into chunks when responding to a client request Fast Forward Your Development www.dsp-ip.com

Content Provider Benefits • Cheaper to deploy ▫ Can utilize any generic HTTP caches/proxies ▫ Doesn’t require specialized servers at every node • Better scalability and reach ▫ Reduces “last mile” issues because it can dynamically adapt to inferior network conditions • Audience can adapt to the content, rather than requiring the content providers to guess which bit rates are most likely to be accessible to their audience Fast Forward Your Development www.dsp-ip.com

End User Benefits • Fast start-up and seek times ▫ Start-up/seeking can be initiated on the lowest bit rate before moving up to a higher bit rate • No buffering, no disconnects, no playback stutter ▫ As long as the user meets the minimum bit rate requirement • Seamless bit rate switching based on network conditions and CPU capabilities. • A generally consistent, smooth playback experience Fast Forward Your Development www.dsp-ip.com

Evolution • Previous versions of MS streaming divide the file into many chunkc 0001.vid 0002.vid etc • Problematic in caching, CDNs, CMS etc • Today all fragments of a file are contained in a single bitstream container. Typically 1 fragment = 1 video GOP. Fast Forward Your Development www.dsp-ip.com

SILVERLIGHT FILES Containers & Configuration files Fast Forward Your Development www.dsp-ip.com

Format options • ASF/WMV – native Microsoft Format • MPEG4 File-Format • AVI • OGG Fast Forward Your Development www.dsp-ip.com

MP4 over ASF file format • MP4 is a lightweight container format with less overhead than ASF • MP4 is easier to parse in managed (.NET) code • MP4 is based on a widely used standard, making 3rd party adoption and support easier • MP4 has native H.264 video support • MP4 was designed to natively support payload fragmentation within the file Fast Forward Your Development www.dsp-ip.com

MP4 File format • MP4 has two format types ▫ Disk Format - for file storage ▫ Wire format - for transport • Wire format enables easy CDN support and integration Fast Forward Your Development www.dsp-ip.com

Smooth Streaming File Format Fast Forward Your Development www.dsp-ip.com

Smooth Streaming Wire Format Fast Forward Your Development www.dsp-ip.com

File extensions • Media Files ▫ *.ismv - Audio & Video ▫ *.isma – Audio only • Manifest Files ▫ *.ism – Server manifest. Describes to the server Relation between tracks, bitrates & files on disk. Based on SMIL 2.0 XML format specification ▫ *.ismc – Describes to the client the available streams, CODECS used, bitrates encoded, video resolutions, markers, captions. First file delivered to client. It’s the first file delivered to client (“SDP” like). Fast Forward Your Development www.dsp-ip.com

Directory Structure Media file in different Manifest Files bitrates Fast Forward Your Development www.dsp-ip.com

Manifest files • VC-1, WMA, H.264 and AAC codecs • Text streams • Multi-language audio tracks • Alternate video & audio tracks (i.e. multiple camera angles, director’s commentary, etc.) • Multiple hardware profiles (i.e. same bitrates targeted at different playback devices) • Script commands, markers/chapters, captions • Client manifest Gzip compression • URL obfuscation • Live encoding and streaming Fast Forward Your Development www.dsp-ip.com

ISM file sample <?xml version="1.0" encoding="utf-16" ?> -  - <smil xmlns="http://www.w3.org/2001/SMIL20/Language"> - <head> <meta name="clientManifestRelativePath" content="NBA.ismc" /> </head> - <body> - <switch> - <video src="NBA_3000000.ismv" systemBitrate="3000000"> <param name="trackID" value="2" valuetype="data" /> </video> - <video src="NBA_2400000.ismv" systemBitrate="2400000"> <param name="trackID" value="2" valuetype="data" /> </video> - <video src="NBA_1800000.ismv" systemBitrate="1800000"> <param name="trackID" value="2" valuetype="data" /> </video> Fast Forward Your Development www.dsp-ip.com

ISM file sample - <video src="NBA_1300000.ismv" systemBitrate="1300000"> <param name="trackID" value="2" valuetype="data" /> </video> - <video src="NBA_800000.ismv" systemBitrate="800000"> <param name="trackID" value="2" valuetype="data" /> </video> - <video src="NBA_500000.ismv" systemBitrate="500000"> <param name="trackID" value="2" valuetype="data" /> </video> - <audio src="NBA_3000000.ismv" systemBitrate="64000"> <param name="trackID" value="1" valuetype="data" /> </audio> </switch> </body> </smil> Fast Forward Your Development www.dsp-ip.com

*.ISMC sample <?xml version="1.0" encoding="utf-16" ?> -  - <SmoothStreamingMedia MajorVersion="1" MinorVersion="0" Duration="4084405506"> - <StreamIndex Type="video" Subtype="WVC1" Chunks="208" Url="QualityLevels({bitrate})/Fragments(video={start time})"> <QualityLevel Bitrate="3000000" FourCC="WVC1" Width="1280" Height="720" CodecPrivateData="250000010FD3FE27F1678A27F859E80C9082DB8D44A9 C00000010E5A67F840" /> <QualityLevel Bitrate="2400000" FourCC="WVC1" Width="1056" Height="592" CodecPrivateData="250000010FD3FE20F1278A20F849E80C9082493DEDDC C00000010E5A67F840" /> <QualityLevel Bitrate="1800000" FourCC="WVC1" Width="848" Height="480" CodecPrivateData="250000010FCBF81A70EF8A1A783BE80C908236EE52654 00000010E5A67F840" /> <QualityLevel Bitrate="1300000" FourCC="WVC1" Width="640" Height="352" CodecPrivateData="250000010FCBE813F0AF8A13F82BE80C9081A7ABF704 400000010E5A67F840" /> Fast Forward Your Development www.dsp-ip.com

ISMC File - 2 - <SmoothStreamingMedia MajorVersion="1" MinorVersion="0" Duration="5965419999"> - <StreamIndex Type="video" Subtype="WVC1" Chunks="299" Url="QualityLevels({bitrate})/Fragments(video={start time})"> <QualityLevel Bitrate="2750000" FourCC="WVC1" Width="1280" Height="720" CodecPrivateData="250000010FD3BE27F1678A27F859E804508253EBE8E6C00000010E5AE7F8 40" /> ….. <c n="0" d="20000000" /> <c n="1" d="20000000" /> ..... <c n="298" d="5000001" /> </StreamIndex> - <StreamIndex Type="audio" Subtype="WmaPro" Chunks="299" Url="QualityLevels({bitrate})/Fragments(audio={start time})"> <QualityLevel Bitrate="64000" WaveFormatEx="6201020044AC0000451F0000CF05100012001000030000000000000000000 000E00042C0" /> <c n="0" d="20433560" /> .... <c n="297" d="20433560" /> <c n="298" d="4393197" /> </StreamIndex> </SmoothStreamingMedia> Fast Forward Your Development www.dsp-ip.com

SILVERLIGHT SESSION Initiation and Flow Fast Forward Your Development www.dsp-ip.com

Smooth Streaming Protocol • Smooth Streaming Protocol uses HTTP [RFC2616] as its underlying transport . • The Server role in the protocol is stateless ▫ Enabling (potentially) different instance of the server to handle client requests ▫ Request can utilize any generic HTTP caches/proxies - > Lowering CDN costs Fast Forward Your Development www.dsp-ip.com

Messages • Smooth Streaming Protocol uses 4 different messages: ▫ Manifest Request ▫ Manifest Response ▫ Fragment Request ▫ Fragment Response • All messages follow the HTTP/1.1 specification Fast Forward Your Development www.dsp-ip.com

Messages Flow Server Client Manifest Request Manifest Response Fragment Request Fragment Response Fragment Request(s) Fast Forward Your Development www.dsp-ip.com

Messages • Manifest Request and Fragment Request message MUST use the HTTP "GET" method, generated by the client. • Manifest Request and Fragment Request message use the HTTP Response messages. Status-Code SHOULD be 200. Fast Forward Your Development www.dsp-ip.com

Smooth Streaming Transport Protocol Session Details Manifest Request Manifest Response Video Fragment Request Audio Fragment Request Fragment Response Fast Forward Your Development www.dsp-ip.com

Session Details - Manifest Request • In order to initiate a presentation the Client MUST send the server a Manifest Request using the HTTP GET method. Fast Forward Your Development www.dsp-ip.com

Session Details - Manifest Response • The Response is a ISMC Manifest file describing the session. - <SmoothStreamingMedia MajorVersion="1" MinorVersion="0" Duration="5965419999"> - <StreamIndex Type="video" Subtype="WVC1" Chunks="299" Url="QualityLevels({bitrate})/Fragments(video={start time})"> <QualityLevel Bitrate="2750000" FourCC="WVC1" Width="1280" Height="720" CodecPrivateData="250000010FD3BE27F1678A27F859E804508253EBE8E6C00000010E5AE7F840" /> ….. <c n="0" d="20000000" /> <c n="1" d="20000000" /> ..... <c n="297" d="20000000" /> <c n="298" d="5000001" /> </StreamIndex> - <StreamIndex Type="audio" Subtype="WmaPro" Chunks="299" Url="QualityLevels({bitrate})/Fragments(audio={start time})"> <QualityLevel Bitrate="64000" WaveFormatEx="6201020044AC0000451F0000CF05100012001000030000000000000000000000E00042C0" /> <c n="0" d="20433560" /> .... <c n="297" d="20433560" /> <c n="298" d="4393197" /> </StreamIndex> </SmoothStreamingMedia> Fast Forward Your Development www.dsp-ip.com

Manifest Response reviewed • We can see in the ISMC file that the server can support 8 different levels of quality (bitrate) for the client can chose from between 2.75Mbit to 0.35 Mbit. - <SmoothStreamingMedia MajorVersion="1" MinorVersion="0" Duration="5965419999"> - <StreamIndex Type="video" Subtype="WVC1" Chunks="299" Url="QualityLevels({bitrate})/Fragments(video={start time})"> <QualityLevel Bitrate="2750000" FourCC="WVC1" Width="1280" Height="720" CodecPrivateData="250000010FD3BE27F1678A27F859E804508253EBE8E6C00000010E5AE7F840" /> <QualityLevel Bitrate="2040000" FourCC="WVC1" Width="1056" Height="592" CodecPrivateData="250000010FD3BE20F1278A20F849E80450823E414DD1400000010E5AE7F840" /> <QualityLevel Bitrate="1520000" FourCC="WVC1" Width="848" Height="480" CodecPrivateData="250000010FCBAE1A70EF8A1A783BE8045081AE62F3F7400000010E5AE7F840" /> <QualityLevel Bitrate="1130000" FourCC="WVC1" Width="704" Height="400" CodecPrivateData="250000010FCBA215F0C78A15F831E8045081A27BD635C00000010E5AE7F840" /> <QualityLevel Bitrate="845000" FourCC="WVC1" Width="576" Height="320" CodecPrivateData="250000010FCB9A11F09F8A11F827E804508199C94077400000010E5AE7F840" /> <QualityLevel Bitrate="630000" FourCC="WVC1" Width="448" Height="256" CodecPrivateData="250000010FCB920DF07F8A0DF81FE804508113396020C00000010E5AE7F840" /> <QualityLevel Bitrate="470000" FourCC="WVC1" Width="368" Height="208" CodecPrivateData="250000010FC38E0B70678A0B7819E80450810E5747B6C00000010E5AE7F840" /> <QualityLevel Bitrate="350000" FourCC="WVC1" Width="320" Height="176" CodecPrivateData="250000010FC38A09F0578A09F815E80450808AADEACF400000010E5AE7F840" /> Fast Forward Your Development www.dsp-ip.com

Manifest Response – reviewed • The client also receives the number of chunks for audio and video tracks and the duration of each chunk so it can request the chunk which fits the desired position in the file <c n="0" d="20000000" /> <c n="1" d="20000000" /> <c n="2" d="20000000" /> <c n="3" d="20000000" /> .... <c n="297" d="20000000" /> <c n="298" d="5000001" /> </StreamIndex> - <StreamIndex Type="audio" Subtype="WmaPro" Chunks="299" Url="QualityLevels({bitrate})/Fragments(audio={start time})"> <QualityLevel Bitrate="64000" WaveFormatEx="6201020044AC0000451F0000CF05100012001000030000000000000000000000E00042C0" /> <c n="0" d="20433560" /> <c n="1" d="19969161" /> <c n="2" d="19969161" /> <c n="3" d="20433560" /> <c n="4" d="20433560" /> <c n="297" d="20433560" /> <c n="298" d="4393197" /> </StreamIndex> </SmoothStreamingMedia> Fast Forward Your Development www.dsp-ip.com

Session Details – Fragment Request • Client-Server requests are based on RESTFull URLs: GET /mediadl/iisnet/smoothmedia/Experience/BigBuckBunny_720p.ism/QualityLevels(350000)/Fragments(video=0) • The URL includes reference to: ▫ Bitrate as QualityLevels which maps to a media file ▫ Fragment number Fast Forward Your Development www.dsp-ip.com

Session Details – Fragment Response • The Server: ▫ checks “BigBuckBunny_720p.ism” server manifest file to find the media file associated with the quality level(350000) ▫ Opens and parses the associated media file to get the chunk with requested time offset (0). ▫ Sends the requested media fragment to the client as HTTP response with status code set to 200 Fast Forward Your Development www.dsp-ip.com

Refrences • Most valuable refrence: http://alexzambelli.com/blog/2009/02/10/smo oth-streaming-architecture/ Fast Forward Your Development www.dsp-ip.com

SILVERLIGHT PROGRAMMING MediaElement Code Walkthrough Fast Forward Your Development www.dsp-ip.com

Creating New Silverlight Project • Choose new “Silverlight Application” in the new project dialog. C# or VB.Net is available. • Choose the default settings at the dialog opened Fast Forward Your Development www.dsp-ip.com

New Project configuration • A new project has been created. • The starting point of the application is “App.xaml”. ▫ The starting page of the application is called by default: “MainPage.xaml” ( can be changed at “App.xaml.cs”) Fast Forward Your Development www.dsp-ip.com

• Media element Represents a control that contains audio and/or video. • Media files cannot be used as project resource. ▫ Must be delivered separately. • To preserve video original aspect ratio DON’T change both Width and Height. Fast Forward Your Development www.dsp-ip.com

Media Element – Main Events • Media Events ▫ MediaOpened Occurs when media loading has finished. ‘ActualWidth’ and ‘ActualHeight’ are set after this event is called. ▫ MediaEnded Occurs when the media has ended. ▫ MediaFailed Occurs when an error is encountered. Use ‘ExceptionRoutedEventArgs’ to get failure details. • Buffering Events: ▫ BufferingStarted Occurs when media buffering has begun. ▫ BufferingEnded Occurs when media buffering has ended. Fast Forward Your Development www.dsp-ip.com

Controlling Media Element • Setting Audio/Video input. • Playing file from current position. ▫ Call this only after MediaOpened event is called. • Stop Playing current ▫ Stops the media playing and sets current to start • Pause command Fast Forward Your Development www.dsp-ip.com

Full Source - MainPage.xaml Fast Forward Your Development www.dsp-ip.com

Full Source - MainPage.xaml.cs Fast Forward Your Development www.dsp-ip.com

Adobe HTTP Streaming Fast Forward Your Development www.dsp-ip.com

Agenda • Overview • Components • Files • Session Fast Forward Your Development www.dsp-ip.com

HTTP Streaming Intro • HTTP Streaming offers the advantages of: ▫ Progressive download in terms of Cost Standard Server Scalability Standard client components (OSMF) ▫ Streaming in terms of User experience Seek-ability of streaming Fast Forward Your Development www.dsp-ip.com

Overview Adobe HTTP Dynamic Streaming is a solution that allows you to stream live and on-demand content over HTTP to Adobe Flash Player. When content streams over HTTP, clients can seek quickly to any location. HTTP Dynamic Streaming supports adaptive streaming, DVR functionality, and Adobe Flash Access protection (DRM). Fast Forward Your Development www.dsp-ip.com

COMPONENTS • Content Ingest • Server • Client Fast Forward Your Development www.dsp-ip.com

Main components • Preparation ▫ File Packager ▫ Live Packager for HTTP Dynamic Streaming • Server ▫ Apache module (HTTP Origin Module) ▫ Flash Access • Client ▫ Player with OSMF classes ▫ Flash Player version 10.1+ ▫ Air 2.0+ Fast Forward Your Development www.dsp-ip.com

Adobe Solution main components Fast Forward Your Development www.dsp-ip.com

Ingest- File Packager • A command-line tool • Used for converting offline content to formats required for Adobe HTTP streaming • Translates on-demand media files into fragments and writes the fragments to F4F files. Fast Forward Your Development www.dsp-ip.com

Ingest - Live Packager • The Live Packager for HTTP Dynamic Streaming is part of Adobe Flash Media Server. • The server ingests a live stream over RTMP and translates it into F4F files in real-time. • The built-in Apache HTTP Server uses the HTTP Origin Module to deliver the live content over HTTP. Fast Forward Your Development www.dsp-ip.com

Server - Apache module (HTTP Origin) • Extension to Apache HTTP Server version 2.2. • Enables processes of Adobe Files: ▫ F4F, F4M,F4X ▫ .bootstrap and ▫ .drmmeta • Flash Media Interactive Server 3.8 includes Apache HTTP Server. Fast Forward Your Development www.dsp-ip.com

Server - Flash Access • DRM Server • Flash Access delivers protected media to Flash Player • For content protection, both File Packager and Flash Media Server are required to package and encrypt the content Fast Forward Your Development www.dsp-ip.com

Client - OSMF classes • The OSMF Player uses the ActionScript 3.0 NetStream.appendBytes() API to deliver data to Flash Player. • OSMF is a robust framework designed to deliver high-quality video. • Adobe strongly recommends using OSMF to build HTTP Dynamic Streaming players. Fast Forward Your Development www.dsp-ip.com

ADOBE HTTP STREAMING FILES F4F, F4M,F4X .bootstrap and .drmmeta Fast Forward Your Development www.dsp-ip.com

Files • The files required for HTTP streaming are: ▫ F4F - MPEG4 media format. Holds the media ▫ F4M – Media description file(codec, resolution) ▫ F4X - Fragments location file ▫ .bootstrap – bootstrap information for each segment ▫ .drmmeta – DRM encryption information Fast Forward Your Development www.dsp-ip.com

*.F4F File • Standard MP4 format with open file specification • Each file contains a segment of the source file. • Each segment contains one or more fragments of content. • The file formats stores any flash supported codec • A player can use a URL to address each fragment. Fast Forward Your Development www.dsp-ip.com

HTTP Streaming file types • *.F4X File ▫ Flash Index file. ▫ Contains the location of specific fragments within a stream. • *.F4M File ▫ Flash Media Manifest file. ▫ Contains information about the media codecs, resolution, and the availability of multi-bitrate files. Fast Forward Your Development www.dsp-ip.com

F4M File Sample <?xml version="1.0" encoding="utf-8" ?> - <manifest xmlns="http://ns.adobe.com/f4m/1.0"> <id>Dynamic Streaming</id> <duration>253</duration> <mimeType>video/mp4</mimeType> <baseURL>rtmp://cp67126.edgefcs.net/ondemand</baseURL> <media url="mp4:mediapm/ovp/content/demo/video/elephants_dream/elephants_d ream_768x428_24.0fps_408kbps" bitrate="408" width="768" height="428" /> <media url="mp4:mediapm/ovp/content/demo/video/elephants_dream/elephants_d ream_768x428_24.0fps_608kbps" bitrate="608" width="768" height="428" /> <media url="mp4:mediapm/ovp/content/demo/video/elephants_dream/elephants_d ream_1024x522_24.0fps_908kbps" bitrate="908" width="1024" height="522" /> </manifest> Fast Forward Your Development www.dsp-ip.com

HTTP Streaming file types • .bootstrap ▫ Bootstrap file. ▫ Contains the bootstrap information for each segment of the file • .drmmeta ▫ DRM Header file. ▫ Contains additional header information about the encryption. Fast Forward Your Development www.dsp-ip.com

Folder example • Example for File “foo” Folder will include : ▫ fooSeg#.f4f (many fragments) ▫ foo.f4x ▫ foo.meta ▫ foo.bootstrap ▫ foo.drmmeta (if the stream is configured for encryption) Fast Forward Your Development www.dsp-ip.com

Messages Flow Server Client Manifest Request Manifest Response Fragment Request Fragment Response Fragment Request(s) Fast Forward Your Development www.dsp-ip.com

Session • The player fetches foo.f4m manifest. • The player request fragment# • HTTP Origin Module locates the required f4f files by looking at F4X index file and send to client. • If the client seeks further, the Origin module searches at the f4m and again locates the f4f file that fits, starts streaming again. • If the player constantly monitor bandwidth and performance and ask the Origin Module for the proper bit rate associated f4f. Fast Forward Your Development www.dsp-ip.com

HTML5 Video Fast Forward Your Development www.dsp-ip.com

HTML5 • Drafts by WHAT WG ▫ Web Hypertext Application Technologies • Merging into W3C specifications • “One of HTML5’s goals is to move the Web away from proprietary technologies such as Flash, Silverlight, and JavaFX, says Ian Hickson, co-editor of the HTML5 specification.” —Paul Krill, reporting for InfoWorld, June 16, 2009 • Browser support Fast Forward Your Development www.dsp-ip.com

Fragmented Web - Description • Multimedia coding on the web is fragmented • Many video codecs: ▫ DIVX, XVID, H.264 ▫ WMV, VC-1, VP6 • Many containers (File Format) ▫ AVI, MKV ▫ MPEG4 FF, 3GPP • Many delivery methods ▫ RTSP/RTP Streaming, Progressive download ▫ Live HTTP, Smooth Streaming Fast Forward Your Development www.dsp-ip.com

Fragmented Web - Challenges • Proprietary Plug-ins - like Flash • Vertical market control on media distribution – like Apple • Media Distributers need to support many: ▫ Codecs ▫ Containers ▫ Delivery Formats • in order to support all device and audiences Fast Forward Your Development www.dsp-ip.com

XIPH • XIPH.org is a non profit organization which aims to create free multimedia coding standards • XIPH defined ▫ Vorbis – Audio codec ▫ Ogg – a free file format media container ▫ Speex – voice codec ▫ Theora – Video Codec • HTML5 Video first based its video codec and container standard on XIPH Standards Fast Forward Your Development www.dsp-ip.com

HTML5 Video • HTML5 video first defined XIPH formats as the base HTML5 video: “User agents should support Theora video and Vorbis audio, as well as the Ogg container format.” December 10, 2007, the HTML5 specification • This was later replaced by a statement which basically stated: we cant make up our mind, use whatever you like. Fast Forward Your Development www.dsp-ip.com

HTML5 Video - Fragmented • Support Theora (version of VP3) ▫ Old codec ▫ Poor performance (BR/Quality ratio) ▫ Free no royalties ▫ Hardware support? • Also H.264 ▫ Much better quality per bitrate ▫ But it requires royalties…. • Google opens VP8 ▫ Good Quality ▫ No Royalties (?) Fast Forward Your Development www.dsp-ip.com

HTML5 Video Code <videosrc="movie.ogg" controls="controls">If you can see this text, your browser does not support the HTML5 video tag.</video> Source W3C School Fast Forward Your Development www.dsp-ip.com

Browser CODEC Support Browser Ogg Theora H.264/MPEG-4 AVC Internet Explorer NO 9.0 Mozilla Firefox 3.5 No Google Chrome 3.0 3.0 Safari No 3.1 Opera 10.50 Fast Forward Your Development www.dsp-ip.com

What is missing • Standard Multi-bitrate support • HTTP Streaming (not PD) • Option for live streams • Transmit your camera (ChatRoulette Style) • P2P Interaction Is that the Flash Killer? Fast Forward Your Development www.dsp-ip.com

WebM Project Fast Forward Your Development www.dsp-ip.com

WebM Overview • Google Sponsored Project • Aims to create: Open, Royalty free media coding formats for the open web • Defines ▫ File Format / Container ▫ Audio CODEC ▫ Video CODEC Fast Forward Your Development www.dsp-ip.com

H.264 vs Theora • Theora is a royalty free code but has lower quality than H.264 • H.264 requires royalties for IP owners but has better quality • Some browser used H.264 video some stayed with Theora. Source: Wikipedia • HTML5 video fragments Browser Ogg Theora H.264 the web instead of unit it Internet NO 9.0 Explorer Mozilla 3.5 No Firefox Google 3.0 3.0 Chrome Safari No 3.1 Fast Forward Your Development www.dsp-ip.com Opera 10.50

WebM • WebM fills the gap left by HTML5 standardization. • Defines: video, audio and container formats • Solves the royalty free Theora vs the superior quality H.264 by providing a royalty free video codec with the same (or better) video quality as H.264 Fast Forward Your Development www.dsp-ip.com Source: On2

CDNs Fast Forward Your Development www.dsp-ip.com Source: Limelight

Previous CDN classification Classify CDN based on delivery technology • Server based – Incumbents, Akamai • Traditional P2P ▫ “Bittorrent”-s ▫ Pando Networks • Grid Based • Hybrid Architecture ▫ “Peer Assisted Streaming” ▫ Joost Fast Forward Your Development www.dsp-ip.com

Classic CDN Segmentation CDN PoP Multi- P2P Grid Based stream Peer Distributed Centralized Pure P2P Assisted Fast Forward Your Development www.dsp-ip.com

Modern CDN Classification Classify CDN based on ▫ Who pays for the service? ▫ What is the service goal? not on the delivery technology • Content Provider CDN • External Service provider CDN • Internal Service Provider CDN • Application CDN Fast Forward Your Development www.dsp-ip.com

Modern CDN Classification CDN Service Service Content Provider - Provider - Based Internal External Premium OTT Content Fast Forward Your Development www.dsp-ip.com

Content Provider CDN • Goal – Assure delivery, Reduce load and delay from content providers’ sites. • Examples ▫ Akamai ▫ Limelight • Located – in main service provider’s POP deployed globally • Client - Content distributers and owners such as CNN, Hulu and “Youtube”-like sites. Fast Forward Your Development www.dsp-ip.com

External Service providers CDN • Goal - Reduce EXTERNAL bandwidth cost for ISP by caching content within the ISP network. • Examples ▫ Oversi ▫ PeerApp • Located – in main ISP POPs • Equipment is PC server based or embedded • Clients - service providers like 012 or 014 (Israeli examples) Fast Forward Your Development www.dsp-ip.com

Internal Service Provider CDN • Goal - reduce INTERNAL bandwidth load inside the service provider network by caching. • Used mainly in IPTV to distribute premium content (due to cost) • Examples: Mostly VoD service providers ▫ Bitband (Motorola) ▫ Velolcix (Alcatel-Lucent) • Located – Distributed within service provider’s network • Equipment is PC server based or embedded • Clients – IPTV service providers Fast Forward Your Development www.dsp-ip.com

Peer based CDNs • Bittorrent • Giraffic • RayV Fast Forward Your Development www.dsp-ip.com

Video Technology Positioning Microsoft Apple Adobe Google Fast Forward Your Development www.dsp-ip.com

Microsoft • Large Install base 60%+ for Silverlight • Good connection to media and content companies (DRM – Play Ready) • Innovative in video technologies ▫ HTTP Streaming ▫ CDN-oriented ▫ Moving from proprietary to standard technology Fast Forward Your Development www.dsp-ip.com

Apple • User base ▫ Relatively small and fanatic • Video innovation – minor ▫ HTTP Streaming FF is based on MPEG2 ▫ Many fragments - hard to manage and CDN ▫ First use of front video cam on iPhone 4.0 • Closed platform • Vertical market control • Relaying too much on HTML5 • Annoying clients and business partners Fast Forward Your Development www.dsp-ip.com

Adobe • Install base ▫ Biggest 99% of connected computers ▫ Company image suffers from bad PR lately • Innovation ▫ Very innovative in the past ▫ First massive use of progressive download ▫ Good on Integration Solution – OSMF ▫ Losing its edge: Last to implement HTTP Streaming Fast Forward Your Development www.dsp-ip.com

Adobe • Acquisition strategy - Poor ▫ Can’t monetize past acquisitions: Macromedia, Amicima ▫ Paid too much for last acquisition: Omniture • Technology ▫ Got good technologies ▫ Focused on developing technologies and less on monetizing them Fast Forward Your Development www.dsp-ip.com

Google • Install base ▫ Medium ▫ Good growth potential on mobile / browser • Innovation ▫ Good Innovation Internally ▫ Aggressive acquisition strategy (Android, On2) ▫ Open system as innovation • Open platform strategy ▫ Not technology oriented • Partnerships ▫ Good relations with partners Fast Forward Your Development www.dsp-ip.com

Google • Acquisition strategy – Great! ▫ Does not pay too much on companies ▫ Monetize and build acquired companies IP: Android ▫ Great move with On2 IP • Technology ▫ Good mix of internal and external technologies ▫ Use standard API & Protocols ▫ Standardize proprietary Technologies (VP8) ▫ Does not depend on a single delivery technology: HTML5 or Flash Fast Forward Your Development www.dsp-ip.com

How do we define success? • Who is first in the video race: ▫ Install base – Speed Install ▫ Innovation - acceleration Base Innovation Fast Forward Your Development www.dsp-ip.com

Positioning • This is my understanding. • What do you think? Install Base Innovation Fast Forward Your Development www.dsp-ip.com

Summary • Scaling Video delivery: ▫ CDNs ▫ HTTP Streaming ▫ Standards ▫ Interoperability • Lowering costs ▫ HTTP Streaming • User experience ▫ PD -> Streaming ▫ Multi-bit-rate Fast Forward Your Development www.dsp-ip.com

DSP-IP Contact information Download slides at: www.dsp-ip.com www.dsp- Course materials & lecture request Adi Yakov Training Manager adi@dsp- adi@dsp-ip.com +972-9-8651933 972- Projects development services: Alona Ashkenazi Development Services www.dsp-ip. www.dsp-ip.com alona@dsp- alona@dsp-ip.com Mail : info@dsp-ip.com info@dsp-ip. +972-9-8850956 972- Phone: 972- 8850956, Phone: +972-9-8850956, Fax : +972-50- 8962910 972-50- Fast Forward Your Development www.dsp-ip.com

APPENDIX OPTIONAL SLIDES Fast Forward Your Development www.dsp-ip.com

OSMF Fast Forward Your Development www.dsp-ip.com

Goal • Accelerate Development • Standardize API between modules For Video Players Fast Forward Your Development www.dsp-ip.com

Support Media Types • Streaming audio ▫ Mp3 , AAC , Speex and Nellymoser • Streaming video ▫ FLV, F4V, MP4, MPEG:MP4, M4V, 3GPP • Progressive audio & video ▫ Mp3, FLV, F4V, MP4, MP4V-ES, M4V,3GPP, QuickTime Fast Forward Your Development www.dsp-ip.com

Basic Structure an OSMF player Media player Media container Control playback of a media Display media element (s) element On the stage Play, Stop, Pause etc Add to stage, show element(s) Media element Loads any support media type Video, images, sound, etc or Composite element load stuff Fast Forward Your Development www.dsp-ip.com

Imports Import org. osmf. containers. Media container; Import org. osmf. Elements. Video element; Import org. osmf. Media .media player ; Import org. osmf. Media. URL Resource; Fast Forward Your Development www.dsp-ip.com

The media container Import org. osmf. containers. Media container; Import org. osmf. Elements. Video element; Import org. osmf. Media .media player ; Import org. osmf. Media. URL Resource; // create the container class that displays the media . var container: media container = new media container ( ) ; Fast Forward Your Development www.dsp-ip.com

The media container Import org. osmf. containers. Media container; Import org. osmf. Elements. Video element; Import org. osmf. Media .media player ; Import org. osmf. Media. URL Resource; // create the container class that displays the media . var container: media container = new media container ( ) ; //add the Media container instance to the stage Add child (container) ; Fast Forward Your Development www.dsp-ip.com

The media container Import org. osmf. containers. Media container; Import org. osmf. Elements. Video element; Import org. osmf. Media .media player ; Import org. osmf. Media. URL Resource; // create the container class that displays the media var container: media container = new media container ( ) ; //add the Media container instance to the stage Add child (container) ; Var video Element = new video Elements ( new URL resource ) ; Fast Forward Your Development www.dsp-ip.com

The media container Import org. osmf. containers. Media container; Import org. osmf. Elements. Video element; Import org. osmf. Media .media player ; Import org. osmf. Media. URL Resource; // create the container class that displays the media var container: media container = new media container ( ) ; //add the Media container instance to the stage Add child (container) ; Var video Element = new video Elements ( new URL resource ) ; // cerate the media player instance Var media player : media player = mew media player ( ); Fast Forward Your Development www.dsp-ip.com

The media container Import org. osmf. containers. Media container; Import org. osmf. Elements. Video element; Import org. osmf. Media .media player ; Import org. osmf. Media. URL Resource; // create the container class that displays the media var container: media container = new media container ( ) ; //add the Media container instance to the stage Add child (container) ; Var video Element = new video Elements ( new URL resource ) ; // cerate the media player instance Var media player : media player = mew media player ( ); //Set the media Elements on the media player // because auto play defaults to true , play back , begin immediately media player . media = Video Elements ; Fast Forward Your Development www.dsp-ip.com

ADDING CAPABILITIES Playback control Media Factory Layouts Fast Forward Your Development www.dsp-ip.com

Default Playback // set auto play to false Media player .auto play=false; Fast Forward Your Development www.dsp-ip.com

Default Playback // listen for the click event on the stage Stage . Add Event Listener ( mouse Event . CLICK,_ on click ); Function _on click (evt ): void { // if the media player is playing If (media player .playing ) { // pause it media player. pause ( ); } else { // resume it media player . Play ( ); } } Fast Forward Your Development www.dsp-ip.com

Media Factory Media Factory Create Media Elements, with a URL Media Elements: Image Element video Element sound Elements Fast Forward Your Development www.dsp-ip.com

The Media Factory Import org .osmf .media .default media factory ; // create the default media factory instance Var media factory = new default media factory (); // have the media factory create the video element Var Media Element = media factory create media element ( new URL resource ) Fast Forward Your Development www.dsp-ip.com

Layouts Media container Layout metadata: Controls how a media element is Displayed in the media container Media element Fast Forward Your Development www.dsp-ip.com

Layouts Layout metadata Width Height X Y Can be absolute or relative Fast Forward Your Development www.dsp-ip.com

Layout metadata // Create a Layout metadata object Var layout = new Layout metadata( ) ; // set the width and height of the Layout metadata object Layout . width = 640 Layout height =480 // tie the Layout metadata instance to the Media element instance Media element . add metadata (Layout metadata. LAYOUT_MANESPACE, layout ) ; Fast Forward Your Development www.dsp-ip.com

Netloader Media element Netloader Fast Forward Your Development www.dsp-ip.com

Netloader HTTP Streaming NetLoader Net Loader RTMPDynamic Streaming NetLoader DVRCastNetLoader Fast Forward Your Development www.dsp-ip.com

Change var video element : video element = new video element ( resource ) ; To var video element : video element = new video element ( resource , new RTMPDynamicstreamingNetLoader ( ) ) Fast Forward Your Development www.dsp-ip.com

Media Element Vector.<DynamicstreamingReousorcecletem> Fast Forward Your Development www.dsp-ip.com

Dynamic Streaming // create a new Dynamic streaming Resource pointing to FMS application Var resource : Dynamic streaming Resource = new Dynamic streaming Resource ( RMTP_ URL ) ; // create a Vector to store the Dynamic Streaming Item object Var vector : vector . < Dynamic Streaming Item > = New vector . < Dynamic Streaming Item > ( 3 ) ; Fast Forward Your Development www.dsp-ip.com

Dynamic Streaming // create 3 entries in the vector, each a DynamicStreamingItem , with a file name and a minimum bit rate Vector ( 0 ) = new DynamicStreamingItem ( “ 2012 _ high “ , 1500 ) ; Vector ( 1 ) = new DynamicStreamingItem ( “ 2012 _ low “ , 400 ) ; Vector ( 2 ) = new DynamicStreamingItem ( “ 2012 _ medium “ , 600 ) ; Fast Forward Your Development www.dsp-ip.com

Dynamic Streaming // set the StreamingItems property on the DynamicStreamingResource to our vector Resource. StreamingItems = vector ; // create a new VideoElement , based on the resource Var videoElement = new Video Element ( resource ) ; Fast Forward Your Development www.dsp-ip.com

Dynamic Streaming in a few lines Var resource = new Dynamic Streaming resource ( RMTP_ URL ) ; Var vector = new vector . < DynamicStreamingItem > ( 3 ) ; Vector ( 0 ) = new DynamicStreamingItem ( “ 2012 _ high “ , 1500 ) ; Vector ( 1 ) = new DynamicStreamingItem ( “ 2012 _ low “ , 400 ) ; Vector ( 2 ) = new DynamicStreamingItem ( “ 2012 _ medium “ , 600 ) ; Resource . StreamItems = vector ; Var VideoElement = new VideoElement (resource ) ; Fast Forward Your Development www.dsp-ip.com

http://www.scoop.it	1654
http://www.videomarketingpunch.com	92
http://videomarketingpunch.com	73
http://translate.googleusercontent.com	6
http://www.slideshare.net	1
http://www.linkedin.com	1

Advance video distribution

by Yossi Cohen

Accessibilité

Catégories

Détails de l'import

Droits d'utilisation

6 Ajouts 1,827

Statistiques

Advance video distribution Presentation Transcript