Light field imaging enables some post-processing capabilities like refocusing, changing view perspective, and depth estimation. As light field images are represented by multiple views they contain a huge amount of data that makes compression inevitable. Although there are some proposals to efficiently compress light field images, their main focus is on encoding efficiency. However, some important functionalities such as viewpoint and quality scalabilities, random access, and uniform quality distribution have not been addressed adequately. In this paper, an efficient light field image compression method based on a deep neural network is proposed, which classifies multiple views into various layers. In each layer, the target view is synthesized from the available views of previously encoded/decoded layers using a deep neural network. This synthesized view is then used as a virtual reference for the target view inter-coding. In this way, random access to an arbitrary view is provided. Moreover, uniform quality distribution among multiple views is addressed. In higher bitrates where random access to an arbitrary view is more crucial, the required bitrate to access the requested view is minimized.

1. All rights reserved. ©2020
All rights reserved. ©2020
SLFC: Scalable Light Field Coding
Hadi Amirpour, Christian Timmerer, and Mohammad Ghanbari
1

2. All rights reserved. ©2020
● Introduction
● Light field compression
● SLFC
● Experimental results
● Conclusion
All rights reserved. ©2020
2

3. All rights reserved. ©2020
Introduction
3

4. All rights reserved. ©2020
Light field photography
● conventional camera can only record 2D light intensity on an
imaging sensor
● light field camera will capture a 4D light field; that is, the intensity of
light in a scene, and also the direction that the light rays are
traveling in space
4

5. All rights reserved. ©2020
Light field photography
5

6. All rights reserved. ©2020
Light field photography
u x
v y
6

7. All rights reserved. ©2020
Introduction
http://lightfield-forum.com/what-is-the-lightfield/
7

8. All rights reserved. ©2020
Introduction
● refocusing
● changing viewport
● depth estimation
● 3D modeling
● synthesizing new views
Post-processing task
8

9. All rights reserved. ©2020
Introduction
Refocusing
Lytro Light Field Camera: from scientific research to a $50-million business
9

10. All rights reserved. ©2020
Introduction
Changing viewport
10

11. All rights reserved. ©2020
Introduction
Changing viewport
11

12. All rights reserved. ©2020
● Light field compression and its challenges
All rights reserved. ©2020
12

13. All rights reserved. ©2020
Light field compression
13

14. All rights reserved. ©2020
Light field compression
14

15. All rights reserved. ©2020
Light field compression
15

16. All rights reserved. ©2020
Light field compression
16

17. All rights reserved. ©2020
Light field compression
Encoder
Decoder
Bitstream
17

18. All rights reserved. ©2020
Light field compression
Encoder
Decoder
Bitstream
Dependency between views
18

19. All rights reserved. ©2020
Light field compression challenges
● encoding efficiency
● viewport scalability
● quality scalability
● random access
● uniform quality distribution
19

20. All rights reserved. ©2020
Viewport scalability
To increase compatibility with:
● display
● capturing device
● network condition
● processing power
● storage capacity
20

21. All rights reserved. ©2020
Viewport scalability
To increase compatibility with:
● display
● capturing device
● network condition
● processing power
● storage capacity
21

22. All rights reserved. ©2020
Quality scalability
To increase compatibility with:
● network condition
● processing power
● storage capacity
22

23. All rights reserved. ©2020
Random access
Navigating between views can increase:
● decoding delay
● bandwidth requirement
● processing power
23

24. All rights reserved. ©2020
Random access
Navigating between views can increase:
● decoding delay
● bandwidth requirement
● processing power
24

25. All rights reserved. ©2020
Quality distribution
It is undesirable to face different quality levels when navigating
between viewports
25

26. All rights reserved. ©2020
Quality distribution
It is undesirable to face different quality levels when navigating
between viewports
26

27. All rights reserved. ©2020
● SLFC: Scalable Light Field Coding
All rights reserved. ©2020
27

28. All rights reserved. ©2020
SLFC
● light field image views are divided into seven layers
○ red view belong to that layer
○ gray views belong to the previous layers
○ black views belong to the next layers
L1 L2 L3 L4 L5 L6 L7
28

29. All rights reserved. ©2020
SLFC
● The central view is intra-coded and can be accessed independently.
● Views in the second layer, are encoded independent of each other
but using the central view as their reference image.
● We see encoding of the remaining views as view/frame interpolation
problem.
L1 L2 L3 L4 L5 L6 L7
29

30. All rights reserved. ©2020
SLFC
Residual with
target view
Residual with
target view
Residual
with
target
view
30

31. All rights reserved. ©2020
SLFC
SepConv: Video Frame Interpolation via Adaptive Separable Convolution
31

32. All rights reserved. ©2020
SLFC
● In each layer (3 to 7), two views from the previously encoded layers
are used to synthesize their intermediate view.
● When the intermediate view is synthesized, it is added as a virtual
reference frame to the reference list in the video encoder.
● To encode each intermediate view four references are used for
inter-coding: (i) the most central view, (ii, iii) two views that are used
for interpolation, (v) the synthesized view.
32

33. All rights reserved. ©2020
SLFC
33

34. All rights reserved. ©2020
Encoding efficiency
Rate distortion curves for Table light field images.
Anchor: JPEG Pleno anchor
WaSP: Hierarchical warping, merging, and sparse prediction for light field image compression
MuLE: A 4D DCT-Based lenslet light field codec
PSB: Pseudo sequence based 2-D hierarchical coding structure for light-field image compression
34

35. All rights reserved. ©2020
Viewport scalability
Bitrate allocated to each layer at bpp4 =
0.75
The number of views in each
layer
35

36. All rights reserved. ©2020
Quality scalability
SLFC provides two quality levels :
● quality 1: When the most central view and corner views are available,
SLFC can synthesize all remaining views by utilizing SepConv.
● quality 2: SLFC first synthesize a target view, then utilize the
synthesized view for inter-coding at the cost of additional bitrate.
36

37. All rights reserved. ©2020
Random access
Ap
for all views of greek light field images at bpp4 = 0.75
37

38. All rights reserved. ©2020
Uniform quality distribution
PSNR heatmap plot for greek light field
image
38

39. All rights reserved. ©2020
Conclusion
● SLFC utilizes a DNN to synthesize intermediate views.
● Synthesized views are used as virtual references for inter-coding.
● SLFC provides:
○ High encoding efficiency
○ Viewport scalability
○ Quality scalability
○ Random access
○ Uniform quality distribution
39

40. All rights reserved. ©2020
Thanks for your attention
40
www.athena.itec.aau.at