This document discusses video coding standards and techniques. It summarizes the development of standards including H.264/AVC and the newer H.265/HEVC, which can provide 50% bitrate reductions. It also covers research topics like immersive video communication and measuring human visual perception. Overall the document traces advances in video coding and compression from early standards to current research frontiers like high resolution 3D video and improved subjective quality assessment.

2. slide2

3. slide3

4. slide4
§ H.264/MPEG-AVC
§ Lagrangian Coder Control
§ H.265/MPEG-HEVC
§ Immersive Video Communication
§ Human Perception Measurement

5. slide5
0 100 200 300
28
30
32
34
36
38
40
Rate [kbit/s]
PSNR
[dB] Half-pel
motion compensation
(MPEG-1 1993
MPEG-2 1994)
Integer-pel
motion
compensation
(H.261, 1991)
Variable block size
(16x16 – 8x8)
(H.263, 1996) +
quarter-pel
motion compensation
(MPEG-4, 1998)
Variable block size
(16x16 – 4x4) +
quarter-pel +
multi-frame
motion compensation
(H.264/AVC, 2003)
Intra frame
DCT coding
(JPEG, 1990)
Bit-rate Reduction: 75%
35
Foreman
10 Hz, QCIF
100 frames

6. slide6
0 100 200 300
28
30
32
34
36
38
40
Rate [kbit/s]
PSNR
[dB]
H.264/AVC (2003)?

7. slide7
1989: Digital TV – Digital Broadcast, DVD
1999: Birth of H.26L in Berlin
Today: >3 Billion devices with H.264/AVC
50% of all bits on the Internet
Every HDTV Receiver
Every Blu-Ray Player
Most Internet Video
Countless Mobile Video

8. slide8
Entropy
Coding
Scaling & Inv.
Transform
Motion-
Compensation
Control
Data
Quant.
Transf. coeffs
Motion
Data
Intra/Inter
Coder
Control
Decoder
Motion
Estimation
Transform/
Scal./Quant.
-
Input
Video
Signal
Split into
Macroblocks
16x16 samples
Intra-frame
Prediction
De-blocking
Filter
Output
Video
Signal

9. slide9
§ How to run the video encoder?
§ Decision between many options denoted with vector p
§ Unconstrained Lagrangian Formulation:
€
min
p
D(p)+ λ ⋅ R(p)
D - Distortion
R - Rate
RT - Target Rate
p - Parameter Vector
with λ controlling the rate-distortion trade-off
min
p
D(p) s.t. R(p)≤ RT
§ Minimization tests the various modes in video coding
[Wiegand, et al., 1996]
§ Constrained Problem:

10. slide10
[Shoham & Gersho, 1989]

12. slide12
Entropy
Coding
Scaling & Inv.
Transform
Motion-
Compensation
Control
Data
Quant.
Transf. coeffs
Motion
Data
Intra/Inter
Coder
Control
Decoder
Motion
Estimation
Transform/
Scal./Quant.
-
Input
Video
Signal
Split into
Macroblocks
16x16 pixels
Intra-frame
Prediction
De-blocking
Filter
Output
Video
Signal
8x8
0
4x8
0 1
0 1
2 3
4x48x4
1
08x8
Types
0
16x16
0 1
8x16
MB
Types
8x8
0 1
2 3
16x8
1
0

13. slide13
§ Division of a picture into square blocks
§ Blocks are assigned to quadtrees
§ Maximum block size is signalled (e.g. 64x64)
§ Quadtree-based
subdivision of
tree block into
prediction and
transform blocks

14. slide14
§ Transform sizes range from 4x4 to 32x32
§ Fast integer transforms specified
§ Additional new rectangular transforms proposed

15. slide15
15
0 100 200 300
28
30
32
34
36
38
40
bit rate (kbit/s)
PSNR
(dB)
Foreman
10 Hz, QCIF
100 frames
H.265 /
MPEG-HEVC
H.264/
MPEG-AVC
MPEG-2 H.261H.263 +
MPEG-4 Visual
JPEG
35
Bit-rate Reduction: 50%

16. slide16
PSNR [dB]
Bit Rate [kbit/s]
50%50%

17. slide17
PSNR [dB]
Bit Rate [kbit/s]
2.5 dB
3.5 dB

18. slide18

19. slide19
Source:
D. Grois et al.

20. slide20
Source:
D. Grois
et al.

21. slide21
Final approval of version 1: April 14, 2013
What comes after version 1 of H.265/MPEG-HEVC?
èThe following H.265/MPEG-HEVC extensions are
work in progress:
• Range Extensions (January 2014)
Higher bit-depths (>10bit), More chroma formats (4:4:4,
4:2:2),...
• Scalable Coding (Mid. 2014)
• 3D Multiview and Depth (January 2014 and 2015)

23. slide23
§ Whole conference situation is not sufficiently natural
§ Provision of eye-contact is limited
§ Awareness of gestures and body language is not fully supported

24. slide24
Courtesy:

25. slide25

26. slide26
26
§ Multi-view video analysis
§ Calculation of a 3D model
§ Rendering of a novel virtual view

27. slide27
27

28. slide28
©
28

29. slide29
©
29

30. slide30

31. slide31
8k theater (NHK) at IBC 2011
Virtual Stadium (NTT) at IBC 2005 Laser Dream Theatre (Sony) at Expo 2005
5k system (HHI) at NAB 2007

32. slide32
Omnidirectional
6k camera
system
(OMNICAM)
Stitching
Segmentation
Coding
Transmission
Decoding
Warping
Blending
Multiprojection
system
7 HD Projectors
6k Video

34. slide34
§ Steady-state visual evoked
potentials (VEPs) to
objectively evaluate
Visual Cortex response
§ Event-Related Potentials
(ERPs) to objectively evaluate
subjective processes
§ ERPs are eventually leading
to judgments and evaluation

35. slide35
Display Human Subjective
Assessment:
MOS
excellent
good
fair
poor
bad
Cornea
Lens
Retina
V1 V2 P3

36. slide36

37. slide37
§ H.264/MPEG-AVC: More than 3 Billion devices
and 50% of all bits on the Internet
§ H.265/MPEG-HEVC:
§ Lagrangian approach to coder control
§ 50% bit-rate reduction relative to H.264
§ Immersive rooms
§ Seamless integration and eye contact
§ Walls are becoming displays
§ H.266 – Research Frontier
§ Even higher resolutions and 3D
§ Improved subjective measures

38. slide38
ITU-T VCEG & ISO/IEC MPEG Colleagues:
• Gary J. Sullivan
• Gisle Bjontegaard
• …
Vidyo
• Alex Eleftheriadis
• Ofer Shapiro
• …
HHI/TUB members and research associates
§ H. Schwarz, D. Marpe & D. Grois
§ P. Kauff & R. Schäfer
§ K.-R. Müller & A. Norcia
§ …

39. slide39