study of "Dappled Photography: Mask Enhanced Cameras for Heterodyned Light Fields and Coded Aperture Refocusing"

1. Dappled Photography: Mask Enhanced Cameras forHeterodyned Light Fields and Coded Aperture Refocusing study Ashok Veeraraghavan, RameshRaskar, AmitAgrawal Mitsubishi Electric Research Labs (MERL), Cambridge, MA Ankit Mohan, Jack Tumblin Northwestern University, Evanston, IL

5. Low resolution with different focus settings4D light field L(u,v,x,y)

9. Heterodynes Light Field camera Based on modulation theorem in 4D frequency domain – mask carries rays = rays x mask F

10. Modulation Theorem [Oppenheim et al. 99] http://en.wikipedia.org/wiki/Amplitude_modulation

11. Heterodynes Light Field camera Based on modulation theorem in 4D frequency domain – mask carries rays = rays x mask F A good mask carriers the rays ! A poor mask blends the rays ! good mask ! α depends on (d,v) rays

12. Heterodynes Light Field camera Based on modulation theorem in 4D frequency domain – mask carries rays = rays x mask F

13. Heterodynes Light Field camera Based on modulation theorem in 4D frequency domain – mask carries rays recover the light field by rearranging the tiles of 2D Fourier transfer into 4D plane to get the full resolution image information for the in-focus parts of the scene = rays x mask F Rearrange F-1

14. Heterodynes Light Field camera Based on modulation theorem in 4D frequency domain – mask carries rays recover the light field by rearranging the tiles of 2D Fourier transfer into 4D plane to get the full resolution image information for the in-focus parts of the scene A raw sensor holds a modulated 4D light filed = rays x mask Raw sensor (modulate 4D light field data) In-focus at full resolution (demodulated)

15. Optical Heterodyning Baseband Audio Signal Software Demodulation Main Lens Object Mask Sensor RecoveredLight Field Receiver: Demodulation High Freq Carrier 100.1 MHz Incoming Signal ReferenceCarrier 99 MHz Incident Modulated Signal Photographic Signal(Light Field) Carrier ReferenceCarrier

16. Coded Aperture camera Base on Convolution Aperture as a Modulator sinc function depends on θ Pinhole camera has a very very broadband modulator Design broadband mask = rays x mask

17. Outline Introduction Related Work Theory & Framework Heterodyne Light Field Camera Encoded Blur Camera Implements & Analysis Contributions & Future Work

18. Introduction

19. Light Field

20. Sensed image (in-focus) θ x red : the in-focus line yellow : sample object x θ Imaginary film object Sensor Lens http://graphics.stanford.edu/papers/fourierphoto/fourierphoto.ppt

21. Sensed image (in-focus) θ x x θ Imaginary film Sensor Lens http://graphics.stanford.edu/papers/fourierphoto/fourierphoto.ppt

22. Sensed image (in-focus) θ x red : the in-focus line yellow : sample x u Imaginary film Sensor Lens http://graphics.stanford.edu/papers/fourierphoto/fourierphoto.ppt

23. Sensed image (out of focus, far) θ x red : the in-focus line yellow : sample x θ Imaginary film Sensor Lens http://graphics.stanford.edu/papers/fourierphoto/fourierphoto.ppt

24. Sensed image (out of focus, far) θ x red : the in-focus line yellow : sample x θ Imaginary film Sensor Lens http://graphics.stanford.edu/papers/fourierphoto/fourierphoto.ppt

25. Sensed image (out of focus, near) red : the in-focus line yellow : sample θ x x θ Imaginary film Sensor Lens http://graphics.stanford.edu/papers/fourierphoto/fourierphoto.ppt

26. Sensed image (out of focus, near) θ x red : the in-focus line yellow : sample x θ Imaginary film Sensor Lens http://graphics.stanford.edu/papers/fourierphoto/fourierphoto.ppt

28. Integral camera [Okano et al. 99; Martnez-Corral et al. 04; Javidi and Okano 02]Light field Camera Virtual viewpoint [Levoy and Hanrahan 96] [Gertler et al 96] Virtual aperture [Levoy and Hanrahan 96] [Isaksen et al. 00] Synthetic appearture photography (similar virtual aperture) [Levoy et al. 04] [Vaish et al. 04]

30. [Levoy and HanrahanSiggraph 96]

31. [Gortler et al 96]

32. Hand-held light field camera [R Ng et al 05]

33. Fourier slice photography [R Ng, SIGGRAPH05]

35. Theory & Framework

36. For different focus settings, the obtained images correspond to slices at different angles, “Fourier Slice Photography ” [Ng, R. 05]

37. Assumption:simulate the aperture as mask placed at lens Open Aperture

38. Open Aperture Assumption:a planar Lambertianobject at the focus plane Because no angular variations in the irradiance of rays from a Lambertian object, the content of light field is restricted to be along the fx axis The sensed image is a slice of the modulated light field

39. Open Aperture In-focus sensor The in-focus image corresponds to a slice of LA(fx, fθ) along fx(fθ=0) No information lost Out of focus sensor The sensor image is a slanted slice The slant angle depends on the degree of mis-focus

40. Heterodyne Light Field Camera

41. Mask as Modulator

42. Mask as Modulator d = v (at aperture stop, θ plane) Mask affects the all rays at an angle θ in a similar way ! m(x, θ) = c (y = θ) α = 900 d = 0 (at sensor, conjugate plane) Mask attenuates all rays for the same x equally ! m(x, θ) = c (y = x) α = 00

45. Notes 4D light field Aliasing When band-limit assumption is not valid in the spatial dimension, the energy in the higher spatial frequencies of the light field masquerade as energy in the lower angular dimension. Post-filter the recovered light field using a Kaiser-Bessel filter with a filer width of 1.5 [Ng 05]

46. Encoded Blur Camera

47. Mask as modulator Assumption:layered Lambertian scene ∵ Because no angular variations in the irradiance of rays from a Lambertian scene, the content of light field is restricted to be along the fx axis

49. 7x7 Binary mask as initial guess

51. Heterodyne Light Field Camera 210 mm f/5.6 Nikkor-W Lens Mask CanoScanLiDE 70 scanner sensor 80 dots/mm

52. Raw sensor image

53. Scene parts which are in-focus can be recovered at full resolution

54. Far Focused Near Focused

55. In-focus – full resolution Low resolution refocused image Out of focus

58. Failure Cases If Assumption of a band-limited light field is invalid, the aliasing artifacts in recovered light field 2D cosine mask needs to be moved away from the sensor because it results in diffraction

59. Encoded Blur Camera 100 mm f/2.8 USM Macro Lens Mask Sensor Canon Rebel XT SLR camera

60. ISO-12233 Chart

61. Modulation Transfer Function(MTF) of ISO-12233 MTF: low MTF: high

62. Full resolution digital refocusing using encoded blur camera Captured photo Refocused photo

64. A is the block-Toeplitz matrix representing 2D blur

65. W is a weighting matrix which sets the weights corresponding to the occluded pixels in the blurred image to zero In-focus fence + blurred person Deblurring without taking the occluders into account Weighted deconvolution Eq. Binary mask for the occluders

66. Failure Cases Scenes with large variation in depths and those with view dependencies can not be handle Practice value 7x7 mask : blur size of about 20 pixels Finer resolution mask can handle large defocus blur but lead to diffraction blur

67. Contributions = rays x mask A theoretical framework of modulating 4D light fields camera working on frequency domain A new class of 4D light filed camera holds full resolution modulated 4D light field Don’t require additional optical elements such as lens arrays Analyze defocus blur as a special case of the frequency domain re-mapping and demonstrate that a broadband mask at aperture can preserve high spatial frequencies in defocused image

68. Future Work Light Fields for Dynamic Scenes Changing masks with time Coding in time and space General Ray Modulators Tilted/curved/multiple masks Wavelength dependent masks Angular/Spatial Resolution Tradeoff Applications Estimating lens aberration Microscopy Light Field Applications

69. End Source