2. The 5D Plenoptic Function
Q: What is the set of all things that one can ever see?
A: The Plenoptic Function [Adelson and Bergen 1991]
(from plenus, complete or full, and optic)
P(q, f, l, t)
3. The 5D Plenoptic Function
Q: What is the set of all things that one can ever see?
A: The Plenoptic Function [Adelson and Bergen 1991]
(from plenus, complete or full, and optic)
P(q, f, l, t)
13. Static Camera Arrays
Stanford Multi-Camera Array
125 cameras using custom hardware
[Wilburn et al. 2002, Wilburn et al. 2005]
Distributed Light Field Camera
64 cameras with distributed rendering
[Yang et al. 2002]
16. Controlled Camera or Object Motion
Stanford Spherical Gantry
[Levoy and Hanrahan 1996]
Relighting with 4D Incident Light Fields
[Masselus et al. 2003]
17. Uncontrolled Camera or Object Motion
Unstructured Lumigraph Rendering
[Gortler et al. 1996; Buehler et al. 2001]
19. Parallax Barriers (Pinhole Arrays)
[Ives 1903]
sensor
barrier
Spatially-multiplexed light field capture using masks (i.e., barriers):
• Cause severe attenuation long exposures or lower SNR
• Impose fixed trade-off between spatial and angular resolution
(unless implemented with programmable masks, e.g. LCDs)
20. Integral Imaging (“Fly’s Eye” Lenslets)
[Lippmann 1908]
sensor
lenslet
f
Spatially-multiplexed light field capture using lenslets:
• Impose fixed trade-off between spatial and angular resolution
24. Modern, Digital Implementations
Digital Light Field Photography
• Hand-held plenoptic camera [Ng et al. 2005]
• Heterodyne light field camera [Veeraraghavan et al. 2007]
25. Marc Levoy
Light Field = Array of (Virtual) Cameras
Slide by Marc Levoy
26. Marc Levoy
Sub-aperture
Virtual Camera =
Sub-aperture View
Light Field = Array of (Virtual) Cameras
Slide by Marc Levoy
27. Marc Levoy
Sub-aperture
Virtual Camera =
Sub-aperture View
Light Field = Array of (Virtual) Cameras
Slide by Marc Levoy
33. Lytro: microlens array in camera
Main lens
Microlens
array (MLA)
Digital
sensor
Lens is not
really thin,
but can be
treated as so.
330 × 330 hex
array, 13.9
micron pitch
Occluders are
not required.
14 Mpixel,
square cropped
to 11 Mpixels
Choose pixel in
same location
behind each
microlens
34. Microlens
array (MLA)
Main lens
Digital
sensor
# of sub-apertures
=
# of pixels behind
each microlens
(10 × 10)
# of pixels per
sub-aperture image
=
# of microlenses
(~ 120,000)
All rays pass through a
“sub-aperture”
Sub-aperture captures on camera view
Choose pixel in
same location
behind each
microlens
So why put the
microlens array
inside the camera?
35. Lytro camera has unusual shape
8x f/2 lens
Light field
sensor
Battery
43-343 mm
equivalent
37. Lens Glare Reduction
[Raskar, Agrawal, Wilson, Veeraraghavan SIGGRAPH 2008]
Glare/Flare due to camera lenses reduces contrast
38. MERL, MIT Media Lab Glare Aware Photography: 4D Ray Sampling for Reducing Glare Raskar, Agrawal,
Wilson & Veeraraghavan
Reducing Glare
Glare Reduced Image
After removing outliersConventional Photo
39. MERL, MIT Media Lab Glare Aware Photography: 4D Ray Sampling for Reducing Glare Raskar, Agrawal,
Wilson & Veeraraghavan
Enhancing Glare
Glare Enhanced ImageConventional Photo
40. MERL, MIT Media Lab Glare Aware Photography: 4D Ray Sampling for Reducing Glare Raskar, Agrawal,
Wilson & Veeraraghavan
41. MERL, MIT Media Lab Glare Aware Photography: 4D Ray Sampling for Reducing Glare Raskar, Agrawal,
Wilson & Veeraraghavan
42. MERL, MIT Media Lab Glare Aware Photography: 4D Ray Sampling for Reducing Glare
Raskar, Agrawal,
Wilson & Veeraraghavan
Glare due to Lens Inter-Reflections
a
Sensor
b
43. MERL, MIT Media Lab Glare Aware Photography: 4D Ray Sampling for Reducing Glare Raskar, Agrawal,
Wilson & Veeraraghavan
Effects of Glare on Image
• Hard to model, Low Frequency in 2D
• But reflection glare is outlier in 4D ray-space
Angular Variation
at pixel a
Lens Inter-reflections
a
Sensor
b
44. MERL, MIT Media Lab Glare Aware Photography: 4D Ray Sampling for Reducing Glare Raskar, Agrawal,
Wilson & Veeraraghavan
Key Idea
• Lens Glare manifests as low frequency in 2D Image
• But Glare is highly view dependent
– manifests as outliers in 4D ray-space
• Reducing Glare == Remove outliers among rays
a
Sensor
b
45. MERL, MIT Media Lab Glare Aware Photography: 4D Ray Sampling for Reducing Glare Raskar, Agrawal,
Wilson & Veeraraghavan
Reducing Glare
using
a Light Field Camera
46. MERL, MIT Media Lab Glare Aware Photography: 4D Ray Sampling for Reducing Glare Raskar, Agrawal,
Wilson & Veeraraghavan
Single Shot Light Field Cameras
Using Mask, this paper
Using Lenslets, Ng et al. 2005
Mask
Adelson and Wang, 1992, Ng et al. 2005
Kanolt 1933, Veeraraghavan et al. 2007
47. MERL, MIT Media Lab Glare Aware Photography: 4D Ray Sampling for Reducing Glare Raskar, Agrawal,
Wilson & Veeraraghavan
Captured Photo: LED off
48. MERL, MIT Media Lab Glare Aware Photography: 4D Ray Sampling for Reducing Glare Raskar, Agrawal,
Wilson & Veeraraghavan
Captured Photo: LED On
49. MERL, MIT Media Lab Glare Aware Photography: 4D Ray Sampling for Reducing Glare Raskar, Agrawal,
Wilson & Veeraraghavan
Each Disk: Angular Samples at that Spatial Location
No Glare
50. MERL, MIT Media Lab Glare Aware Photography: 4D Ray Sampling for Reducing Glare Raskar, Agrawal,
Wilson & Veeraraghavan
With Glare
51. MERL, MIT Media Lab Glare Aware Photography: 4D Ray Sampling for Reducing Glare Raskar, Agrawal,
Wilson & Veeraraghavan
x
y
u
v
52. MERL, MIT Media Lab Glare Aware Photography: 4D Ray Sampling for Reducing Glare Raskar, Agrawal,
Wilson & Veeraraghavan
Sequence of Sub-Aperture Views
Average of all the
Light Field views
One of the
Light Field views
Low Res Traditional Camera
Photo
Glare Reduced Image
53. MERL, MIT Media Lab Glare Aware Photography: 4D Ray Sampling for Reducing Glare Raskar, Agrawal,
Wilson & Veeraraghavan
Key Idea
• Reducing Glare == Remove outlier among angular samples
a
Sensor
b