This document discusses two approaches to using neural networks for video compression: conservative and disruptive. The conservative approach involves using neural networks to filter or process blocks within an existing video coding framework like MPEG. Experiments showed potential bitrate savings of up to 7.9% for intra-frame coding. The disruptive approach aims to replace the entire video coding chain with end-to-end neural networks. This could jointly learn motion estimation, motion compression and residual compression in an optimized way. Challenges include effectively compressing the optical flow for motion information. The researchers are exploring end-to-end models and issues like motion compensation network design.

1. Reti neurali e
compressione video
Roberto Iacoviello, Angelo Bruccoleri
Rai Centre for Research, Technological Innovation and
Experimentation (CRITS)

2. Typical hybrid block based approach

3. Two approaches:
 NON Video approach: coded representation of neural network
Neural Network Video approach
Conservative Disruptive
One to One End to End
Replace one MPEG block with one Deep
Learning block
Replace the entire chain MPEG

4. NN Video approach: Conservative
Neural Network based Filter for Video Coding
Core Experiment 13 on neural network based filter for video coding
Investigate the following problems:
 The impact of NN filter position in the filter chain
 The benefit of the CTU/block level NN filter adaptive on/off
 The generalization capability of the NN: performance change when the test QP is not the same as the
training QP
4
JVET-N0840-v1

5. CE13-2.1: Convolutional Neural Network Filter (CNNF) for
Intra Frame
JVET-N0169
Over VTM-4.0 All Intra
Y U V EncT DecT
DF+CNNF+SAO+ALF -3.48% -5.18% -6.77% 142% 38414%
CNNF+ALF -4.65% -6.73% -7.92% 149% 37956%
CNNF -4.14% -5.49% -6.70% 140% 38411%

6. Concat
Conv1, (5,5,64)
Conv2, (3,3,64)
Conv3, (3,3,64)
Conv4, (3,3,64)
Conv5, (3,3,64)
Conv6, (3,3,64)
Conv7, (3,3,64)
Convolution8, (3,3,1)
Summation
Normalized QP MapNormalized Y/U/V
N: kernel size
K:kernel number
ConvM, (N,N,K)
Convolution (N,N,K)
ReLU
CE13-2.1: Convolutional Neural Network Filter (CNNF) for Intra
Frame
JVET-N0169

7. CE13-1.1: Convolutional neural network loop filter
JVET-N0110-v1
Over VTM-4.0
Random Access
Y U V EncT DecT
-1.36% -14.96% -14.91% 100% 142%

8. Each category will investigate the following problems:
 The impact of NN filter position in the filter chain: there is always objective gain
 The benefit of the CTU/block level NN filter adaptive on/off: the objective gain of adaptive is minor
 The generalization capability of the NN: results indicate that the difference is minor
Neural Network based Filter for Video Coding
JVET-N_Notes_dD
What we have decided in the last meeting (25/3/2019):
The performance/complexity tradeoff indicates that the NN technology
currently is not mature enough to be included in a standard

9. Neural Network Video approach: Disruptive
 Videos are temporally highly redundant
 No deep image compression can compete with state-of-the-art video
compression, which exploits this redundancy
Optical Flow

10. Optical Flow
 In the computer vision tasks, optical flow is widely used to exploit temporal relationship
 Learning based optical flow methods can provide accurate motion information at pixel-level
 Only artificial/synthetic data set

11. SpyNet

12. SpyNet training & loss
Target Residual
Network residual flow

13.  Each Gk has 5 convolutional layers
 Each convolutional layer is followed by a Rectified Linear Unit (ReLU)
 7x7 convolutional kernel for each of the layers
 Reduction in model parameters
 The warping function and learning of residual flow
 SPyNet also has a small memory footprint
 Space required to store all the model parameters is 9.7 MB
SpyNet

14. • Learning based optical flow estimation is utilized to obtain the motion information and reconstruct
the current frame
• End-to-end deep video compression model that jointly learns motion estimation, motion
compression, and residual compression
DVC: An End-to-end Deep Video Compression
Framework

15. DVC: An End-to-end Deep Video Compression
Framework
MPEG NN

16. DVC: An End-to-end Deep Video Compression
Framework
Optical Flow Net

17. DVC: An End-to-end Deep Video Compression
Framework
Motion Compression

18.  MV Encoder and Decoder Network
DVC: An End-to-end Deep Video Compression
Framework

19. DVC: An End-to-end Deep Video Compression
Framework
Motion Compensation Network

20.  Motion Compensation Network
DVC: An End-to-end Deep Video Compression Framework

21. DVC: An End-to-end Deep Video Compression
Framework
Residual Encoder Net
Bit Rate Estimation Net

22. Loss Function DVC: An End-to-end Deep Video
Compression Framework
 The whole compression system is end-to-end optimized: Rate Distortion
Optimization

23.  Results
DVC: An End-to-end Deep Video Compression
Framework

24. Advantages of Neural Networks
 Excellent content adaptivity
 Improve coding efficiency by leveraging samples from far distance
 Neural Network can well represent both texture and feature
 The whole compression system is end-to-end optimized

25. Rai Centre for Research, Technological Innovation and
Experimentation (CRITS): what we are doing
 End to end chain
 Issues:
 Optical flow compression
 Next:
 Motion compensation network?

26. Reti neurali e compressione video
Roberto Iacoviello, Angelo Bruccoleri
roberto.iacoviello@rai.it
angelo.bruccoleri@rai.it
Rai Centre for Research, Technological Innovation and
Experimentation (CRITS)
Grazie per l’attenzione
This work is licensed under the Creative Commons Attribution-NonCommercial-NoDerivs 3.0 Unported License
To view a copy of this license, visit http://creativecommons.org/licenses/by-nc-nd/3.0/