An introduction about lightfields

1. Lightfields
Aidan Goh, John Lee, Ujjayanta Bhaumik
UNIVERSITY COLLEGE LONDON

2. Overview of seminar
• Principles of lightfield capture
• Applications of lightfields in VR displays
• Improvements to lightfield storage and representation

3. Digital Camera
• Images are made using light readings focused on the photo sensor by the
aperture and the diaphragm of the lens
• Images are computed using a demosaicing algorithm
• Resolution of images are limited to resolution of photo sensor

4. Focus Problem
• Difficulty to accurately focus on a subject:
Sometimes foreground can be indistinguishable from the background, which
can blur them together
• Dependency between the lens aperture size and the depth of field represented in
an image:
A narrower lens will allow for a larger range of depths but will require more
exposure and is more prone to blurriness due to human error
• The imperfect design of lenses in cameras:
Leads to aberrations, where light rays from one point in the world do not
converge to a single image point.

5. Lightfield Imaging for Single Images
• Invented by Gabriel Lippmann. Under the term Integral Photography
• Designed a camera with a lens made up of tiny lenses to allow a scene to be
captured from different views.
Lightfield imaging allows the amount of light traveling along
individual rays, which contribute to an image, to be
recorded.
The amount of light traveling along individual rays to
the image can be defined as the total geometric
distribution of light passing through the lens at a
single exposure.
Lightfield imaging and the total geometric distribution of light

6. Plenoptic Function
• Plenoptic function which are 5D functions where 3D (x,y,z) comes from spatial
position and 2D (u,v) for each direction of phase
• Can be reduced by one dimension by ignoring any ray blocked by an object.
• The light field is parameterized so that each ray can be described by its
intersection points between the aperture of the lens and the photo sensor.

7. Plenoptic Camera
• Uses an array of microlens that covers multiple
pixels of the photo sensor, which can simply be
inserted in between the lens and sensor.
• Uses different image resolutions to sample light
rays that contribute to the final image
• The microlens array then separates the light
rays that strike it into a tiny image on the pixels
of the photo sensor
• The size of each of microlens determines the
resolution of the spatial sampling pattern of the
light field.

8. Ray Space
• A ray space is used to visualise which parts of the
light field are measured.
• Represents each pixel with a strip in the space
• Can be sheared to refocus the image.

9. Computing Lightfield Images
• Estimating the sum of light rays on a strip with any orientation by synthesizing the
value of the pixel
• Tracing the ideal rays from the world through the aperture, and microlens array to
the photo sensor by to find the radiance along the rays in the each strip in the ray
space
Concepts for computing Lightfield images
• Balancing the resolution/size of the microlenses
• Adjusting microlenses size to reduce effects of
diffraction
Limitations

10. Lightstages
• Used to capture the reflectance field of the surface
of a subject
• Works by activating lights to produce an even field
of illumination
• Can be used to place a subject in artificial lighting
conditions and determine a surface’s svBRDF

11. [4]

12. Capturing light field - Welcome to Lightfield [4][6]
● Limited to static scene
● Portable, easy to set up
● GoPro: People, outdoor
● DSLR: Highest quality images

13. Processing - Welcome to Lightfield [4][6]
● Calibration → View synthesis
● Extract depth map → Renderable mesh

14. Rendering - Welcome to Lightfield [4][6]
- Each disk correspond to one of the camera views

15. High fidelity
● Highly realistic environment and object
● Focus cue
○ Parallax
○ Avoid vergence-accommodation conflict
[13]

16. Correcting eye aberration[5]

17. Microlens-array-based Near-eye Display [7]
● Trade off spatial resolution for wide field of view
● Thin and lightweight (1cm for eye piece, 109g)
● Programmable remapping of rays

18. Light field Stereoscope [8]
● Multiplicative multilayer display
● Focus cue
● Higher resolution, better accommodation range
● Also able to correct aberration

19. Light field atoms are the essential building blocks of natural light fields - most
light fields can be represented by the weighted sum of very few atoms

20. A light field is divided into small 4D patches and represented by only few coefficients. Light field atoms
achieve a higher image quality than DCT coefficients.
DCT vs Lightfield atoms

22. Sparse coding and the proposed 4D dictionaries can remove noise from 4D
light fields.
Light Field Denoising

23. Lightfield Super Rays

24. Refocusing in Fourier Domain

26. Light field view dependent performance
improvement
In general, the viewer cannot watch all views at the same time. Here, the method of weighted compressive
light field display is used, based on the viewing position.

27. Increasing light field resolution

28. References
• [1] Acquiring the Reflectance Field of a Human Face. (2000). SIGGRAPH 2000 Conference, (pp. 145-156).
• [2] Ng, R. (2006). Digital Lightfield Photography. California: Stanford University.
• [3] Ng, R., Levoy, M., Bredif, M., Duval, G., Horowitz, M., & Hanrahan, P. (2005). Light Field Photography with a Hand-held Plenoptic Camera. California:
Stanford University, Duval Design.
• [4] YouTube. (2018). Tripping the Light VR - YouTube. [ONLINE] Available at: https://www.youtube.com/watch?v=lpu4GJZLXEw. [Accessed 04
December 2018].
• [5] Huang, F., G. Wetzstein, B. Barsky, R. Raskar. (2014). Eyeglasses-free Display: Towards Correcting Visual Aberrations with Computational Light
Field Displays. ACM Proc. of SIGGRAPH (Transactions on Graphics 33, 4), 2014.
• [6] Overbeck, R. S., D. Erickson, D. Evangelakos, and P. Debevec. (2018). The making of welcome to light fields VR. In ACM SIGGRAPH 2018 Talks
(SIGGRAPH '18). ACM, New York, NY, USA, Article 63, 2 pages. DOI: https://doi.org/10.1145/3214745.3214811
• [7] Lanman, D. (NVIDIA), D. Luebke. (2013). Near-Eye Light Field Displays. ACM Transactions on Graphics (TOG), Volume 32 Issue 6, November 2013
(Proceedings of SIGGRAPH Asia)
• [8] Huang, F. (NVIDIA), D. Luebke, G. Wetzstein (Stanford University). (2015). The Light Field Stereoscope. ACM SIGGRAPH 2015 Emerging
Technologies
• [9] Shi, L. (NVIDIA & MIT CSAIL), F. Huang (NVIDIA), W. Lopes, W. Matusik (MIT CSAIL), D. Luebke. 2017. Near-eye Light Field Holographic Rendering
with Spherical Waves for Wide Field of View Interactive 3D Computer Graphics. ACM SIGGRAPH ASIA 2017
• [10] Matthieu Hog, Neus Sabater, Christine Guillemot Dynamic Super-Rays for Efficient Light Field Video Processing, BVMC 2018 - 29th British Machine
Vision Conference, September 2018, Newcastle Upon Tyne, United Kingdom, pp.1-12. <hal-01649342v2>
• [11] M.Umair Mukati , Bahadir.K.Gunturk Light Field Super Resolution through Controlled Micro Shifts of Light Field Sensor
• [12] Kshitij Marwah, Gordon Wetzstein, Yosuke Bando, Ramesh Raskar, Compressive Light Field Photography
• [13] YouTube. (2018). Real-time Rendering of Animated Light Fields - YouTube. [ONLINE] Available at: https://www.youtube.com/watch?v=-
Hc92mP3GLw [Accessed 04 December 2018].