Computational Displays in 4D, 6D, 8D We have explored how light propagates from thin elements into a volume for viewing for both automultiscopic displays and holograms. In particular, devices that are typically connected with geometric optics, like parallax barriers, differ in treatment from those that obey physical optics, like holograms. However, the two concepts are often used to achieve the same effect of capturing or displaying a combination of spatial and angular information. Our work connects the two approaches under a general framework based in ray space, from which insights into applications and limitations of both parallax-based and holography-based systems are observed. Both parallax barrier systems and the practical holographic displays are limited in that they only provide horizontal parallax. Mathematically, this is equivalent to saying that they can always be expressed as a rank-1 matrix (i.e, a matrix in which all the columns are linearly related). Knowledge of this mathematical limitation has helped us to explore the space of possibilities and extend the capabilities of current display types. In particular, we have designed a display that uses two LCD panels, and an optimisation algorithm, to produce a content-adaptive automultiscopic display (SIGGRAPH Asia 2010). (Joint work with R Horstmeyer, Se Baek Oh, George Barbastathis, Doug Lanman, Matt Hirsch and Yunhee Kim) http://cameraculture.media.mit.edu In other work we have developed a 6D optical system that responds to changes in viewpoint as well as changes in surrounding light. Our lenticular array alignment allows us to achieve such a system as a passive setup, omitting the need for electrical components. Unlike traditional 2D flat displays, our 6D displays discretize the incident light field and modulate 2D patterns in order to produce super-realistic (2D) images. By casting light at variable intensities and angles onto our 6D displays, we can produce multiple images as well as store greater information capacity on a single 2D film (SIGGRAPH 2008). Ramesh Raskar joined the Media Lab from Mitsubishi Electric Research Laboratories in 2008 as head of the Lab’s Camera Culture research group. His research interests span the fields of computational photography, inverse problems in imaging and human-computer interaction. Recent inventions include transient imaging to look around a corner, next generation CAT-Scan machine, imperceptible markers for motion capture (Prakash), long distance barcodes (Bokode), touch+hover 3D interaction displays (BiDi screen), low-cost eye care devices (Netra) and new theoretical models to augment light fields (ALF) to represent wave phenomena. In 2004, Raskar received the TR100 Award from Technology Review, which recognizes top young innovators under the age of 35, and in 2003, the Global Indus Technovator Award, instituted at MIT to recognize the top 20 Indian technology innovators worldwide. In 2009, he was awarded a Sloan Research Fellowship. In 2010, he received the Darpa Young Faculty award. He holds over 40 US patents and has received four Mitsubishi Electric Invention Awards. He is currently co-authoring a book on Computational Photography. http://raskar.info

1. Camera Culture Ramesh Raskar Camera Culture MIT Media Lab Computational Displays in 4D, 6D and 8D

2. Slow Glass: Time Shift http:// baens-universe.com/articles/otherdays Light of Other Days by Bob Shaw http://www.fantasticfiction.co.uk/s/bob-shaw/other-days-other-eyes.htm

3. Shift Glass

4. Shift Glass Space Shifting Angle Shifting Time Shifting Illumination Shifting 4D 4D t 4D 4D

5. Capture Analyze Display Shift Glass

6. Capture Analyze Display 5D: Looking around corners 4D: Plenoptic Camera 3D: Flutter Shutter Camera 6D: View and Lighting Aware 4D: Rank Deficient 4D: Netra for Optometry 4D, 6D, 8D: Augmented Light Field Shift Glass

7. Can you look around a corner ? Without any device in the line of sight

8. Femto-Photography: Higher Dimensional LF FemtoFlash UltraFast Detector Computational Optics Serious Sync

9. Kirmani, Hutchinson, Davis, Raskar ICCV’2009, Marr Prize Honorable Mention

10. Streak Camera = Inverse of CRT/CRO Femto-laser

11. Multi-Dimensional Light Transport 5-D Transport

13. Rescue and Planning

14. Robot, Car Path Planning

15. Endoscopy

16. Camera Culture Ramesh Raskar Team Moungi G. Bawendi, Professor, Dept of Chemistry, MIT James Davis, UC Santa Cruz Andreas Velten, Postdoctoral Associate, MIT Media Lab Ahmed Kirmani, RA, MIT Media Lab Tyler Hutchison, RA, MIT Media Lab Rohit Pandharkar, RA, MIT Media Lab Andrew Matthew Bardagjy, RA, MIT Media Lab Everett Lawson, MIT Media Lab Ramesh Raskar, MIT Media Lab

17. Capture Analyze Display 5D: Looking around corners 4D: Plenoptic Camera 3D: Flutter Shutter Camera 6D: View and Lighting Aware 4D: Rank Deficient 4D: Netra for Optometry 4D, 6D, 8D: Augmented Light Field Shift Glass

18. Capture Analyze Display 5D: Looking around corners 4D: Plenoptic Camera 3D: Flutter Shutter Camera 6D: View and Lighting Aware 4D: Rank Deficient 4D: Netra for Optometry 4D, 6D, 8D: Augmented Light Field

19. Slow Display

20. Light Reactive Monostable Materials 16 Megapixel, 2 Watt

21. Day/Night visible

22. g SlowDisplay.org Saakes, Chiu, Hutchison, .., Inami, Raskar, Siggraph 2010 Etech Demo

23. 6D Photo Frames One Pixel of a 6D Display = 4D Display Single Pixel of 6D Frame Martin Fuchs, Ramesh Raskar, Hans-Peter Seidel, Hendrik P. A. Lensch 1 2 1 1 2D 2D 2D

24. Respond to Viewpoint + Ambient Light

25. 6D Display Light sensitive 4D display One Pixel of a 6D Display = 4D Display Raskar, Saakes, Fuchs, Siedel, Lensch, 2008

26. Beyond Multi-touch: Thin LCD for touch+hover Laptops Mobile

27. BiDi Screen: Multi-touch + Hover 3D interface

28. Converting LCD Screen = large Camera for 3D Interactive HCI and Video Conferencing Matthew Hirsch, Henry Holtzman Doug Lanman, Ramesh Raskar Siggraph Asia 2009 BiDi Screen

29. Theory: Benefits of Tiled-broadband Masks 0 10 20 30 40 50 60 0 0.1 0.2 0.3 0.4 0.5 0.6 Angular Resolution Average Transmission (%) Pinholes Sum-of-Sinusoids MURA 11x11 23x23 43x43 Angular Resolution Tiled-Broadband Code MURA Sum-of-Sinusoids Pinholes

30. Overview: Sensing Depth from Array of Virtual Cameras in LCD

31. Bits Photons CV / Machine Learning Optics Sensors Computational Displays Signal Processing Light Transport Displays HCI

33. View Dependent Appearance and Iridescent color Cross section through a single M. rhetenor scale

34. Two Layer Displays PB = dim displays Lenslets = fixed spatial and angular resolution Dynamic Masks = Brighter, High spatial resolution barrier sensor/display lenslet sensor/display

35. Limitations of 3D Display Lanman, Hirsch, Kim, Raskar Siggraph Asia 2010 Front Back Parallax barrier LCD display

36. ` Light Field Analysis of Barriers L[i,k] L[i,k] i k light box g[k] k f[i] i

37. f[i] g[k] L[i,k] light box ` Content-Adaptive Parallax Barriers k i

39. f[i] g[k] L[i,k] light box ` Content-Adaptive Parallax Barriers k i

40. = Content-Adaptive Parallax Barriers `

42. rear mask: f 1 [i,j] front mask: g 1 [k,l] reconstruction (central view) Optimization: Iteration 1 Daniel Lee and Sebastian Seung. Non-negative Matrix Factorization. 1999. Vincent Blondel et al . Weighted Non-negative Matrix Factorization. 2008.

43. Optimization: Iteration 10 rear mask: f 1 [i,j] front mask: g 1 [k,l] reconstruction (central view) Daniel Lee and Sebastian Seung. Non-negative Matrix Factorization. 1999. Vincent Blondel et al . Weighted Non-negative Matrix Factorization. 2008.

44. Optimization: Iteration 20 rear mask: f 1 [i,j] front mask: g 1 [k,l] reconstruction (central view) Daniel Lee and Sebastian Seung. Non-negative Matrix Factorization. 1999. Vincent Blondel et al . Weighted Non-negative Matrix Factorization. 2008.

45. Optimization: Iteration 30 rear mask: f 1 [i,j] front mask: g 1 [k,l] reconstruction (central view) Daniel Lee and Sebastian Seung. Non-negative Matrix Factorization. 1999. Vincent Blondel et al . Weighted Non-negative Matrix Factorization. 2008.

46. Optimization: Iteration 40 rear mask: f 1 [i,j] front mask: g 1 [k,l] reconstruction (central view) Daniel Lee and Sebastian Seung. Non-negative Matrix Factorization. 1999. Vincent Blondel et al . Weighted Non-negative Matrix Factorization. 2008.

47. Optimization: Iteration 50 rear mask: f 1 [i,j] front mask: g 1 [k,l] reconstruction (central view) Daniel Lee and Sebastian Seung. Non-negative Matrix Factorization. 1999. Vincent Blondel et al . Weighted Non-negative Matrix Factorization. 2008.

48. Optimization: Iteration 60 rear mask: f 1 [i,j] front mask: g 1 [k,l] reconstruction (central view) Daniel Lee and Sebastian Seung. Non-negative Matrix Factorization. 1999. Vincent Blondel et al . Weighted Non-negative Matrix Factorization. 2008.

49. Optimization: Iteration 70 rear mask: f 1 [i,j] front mask: g 1 [k,l] reconstruction (central view) Daniel Lee and Sebastian Seung. Non-negative Matrix Factorization. 1999. Vincent Blondel et al . Weighted Non-negative Matrix Factorization. 2008.

50. Optimization: Iteration 80 rear mask: f 1 [i,j] front mask: g 1 [k,l] reconstruction (central view) Daniel Lee and Sebastian Seung. Non-negative Matrix Factorization. 1999. Vincent Blondel et al . Weighted Non-negative Matrix Factorization. 2008.

51. Optimization: Iteration 90 rear mask: f 1 [i,j] front mask: g 1 [k,l] reconstruction (central view) Daniel Lee and Sebastian Seung. Non-negative Matrix Factorization. 1999. Vincent Blondel et al . Weighted Non-negative Matrix Factorization. 2008.

52. Content-Adaptive Front Mask (1 of 9)

53. Content-Adaptive Rear Mask (1 of 9)

54. Emitted 4D Light Field

56. Lightfield vs Hologram Displays

57. Is hologram just another ray-based light field? Can a hologram create any intensity distribution in 3D? Why hologram creates a ‘wavefront’ but PB does not? Why hologram creates automatic accommodation cues? What is the effective resolution of HG vs PB?

58. Parallax Barrier: N p =10 3 pix. Hologram: N H =10 5 pix. θ p =10 pix w θ H =1000 pix ϕ P ∝w/d ϕ H ∝λ/t H Fourier Patch Horstmeyer, Oh, Cuypers, Barbastathis, Raskar, 2009

59. Augmenting Plenoptic Function Wigner Distribution Function Traditional Light Field WDF Traditional Light Field Augmented LF Interference & Diffraction Interaction w/ optical elements ray optics based simple and powerful wave optics based rigorous but cumbersome Oh, Raskar, Barbastathis 2009: Augmented Light Field

60. Light Fields Goal: Representing propagation, interaction and image formation of light using purely position and angle parameters Reference plane position angle LF propagation (diffractive) optical element LF LF LF LF LF propagation light field transformer

61. Augmented Lightfield for Wave Optics Effects Wigner Distribution Function Light Field LF < WDF Lacks phase properties Ignores diffraction, interferrence Radiance = Positive ALF ~ WDF Supports coherent/incoherent Radiance = Positive/Negative Virtual light sources LF Augmented Light Field WDF

63. L(x, θ ) W(x,u) W m = sinc d = delta q W m p q p d(θ) p q q d(θ) * p W m * * Rays: No Bending 1 Fresnel HG Patch θ u * Zooming into the Light Field

64. Algebraic Rank Constraint s 1 m 2 s 1 m 2 s 1 * s 1 s 1 s 1 * Rank-1 Rank-1 Rank-3

65. - Transform <t(x+x ʹ /2)t*(x-x ʹ /2)> Interference xʹ x (a) Two Slits, Coherent t(x+x ʹ /2)t*(x-x ʹ /2) W(x,u) Rank-1 t(x 1 )t*(x 2 ) R 45 , D Transform -1 u

66. L2 L1 L3 ϕ 1 ϕ 1 ϕ 1 ϕ 1 L 1 (x,θ) L 2 (x,θ) L 3 (x,θ) d z 1 h H r z 2 L 1 (x,θ) L 2 (x,θ) L 3 (x,θ) s 1 m 2 (a)

67. A B C Vary Illumination Direction: -5 ̊ , 0 ̊, 5 ̊ A B C A … -5 ̊ 5 ̊ 0 ̊ No Slits 24mm 36mm t H =25μm w=125μm z H =10cm (c)

68. M2 M1 M3 ϕ 1 ϕ 1 L 1 (x,θ) L 2 (x,θ) L 3 (x,θ) d z 1 r z 2 s 1 m 2 s 1 m 2 s 1 m 2 s 1 * s 1 s 1 s 1 * Rank-1 Rank-1 Rank-3

69. Is hologram just another ray-based light field? Can a hologram create any intensity distribution in 3D? Why hologram creates a ‘wavefront’ but PB does not? Why hologram creates automatic accommodation cues? What is the effective resolution of HG vs PB?

70. NETRA: Interactive Display for Estimating Refractive Errors and Focal Range Vitor Pamplona Ankit Mohan Manuel Oliveira Ramesh Raskar

71. Vitor Pamplona Ankit Mohan Manuel Oliveira Ramesh Raskar NETRA: N ear E ye T ool for R efractive A ssessment

72. 0.6B uncorrected refractive errors NETRA at LVP Eye Institute 6.5 Billion people 4.5B with Mobile phone 2B refractive errors

73. * Phoropter-based: $5,000.00 Needs expert, Moving parts, Shining lasers Retino scope w/ Lenses Auto-refracto-meter Chart with Lenses In-Focus: Focometer Optiopia Solo-health: EyeSite NETRA Technology Shining Light plus lenses Fundus Camera Moving lenses + target Moving lenses + target Reading chart on monitor Cellphone + eyepiece Cost to buy $2,000* ~$10,000 ~$100 ~$495 ~$200 -- $30 Cost per test ~$36 ~$36 ~$5 -- -- -- ~$1 Data capture No Comp. No No No Comp. Phone Mobility <500g >10Kg 2kg 1kg <5kg >10Kg <100g Speed Fast Fast Medium Medium -- Fast Fast Scalability No No No Yes Probably No Yes Accuracy 0.15 0.15 0.5 0.75 -- -- <0.5 Self evaluation No No Yes Yes Yes Yes Yes Electricity Req No Yes No No -- Yes No Astigmatism Yes Yes Yes/No No -- Yes Yes Network No Yes No No No Yes Yes Training High High High Medium Medium Low Low

74. Shack-Hartmann Wavefront Sensor Expensive; Bulky, Requires trained professionals Wavefront aberrometer

75. Shack-Hartmann Wavefront Sensor Laser Sensor Microlens Array Planar Wavefront Shack & Platt 1971 Liang et al 1994 David Williams et al, Rochester Spot Diagram

76. Laser Sensor Displacement = Local Slope of the Wavefront Spot Diagram Shack-Hartmann Wavefront Sensor Shack-Hartmann ~ Lightfields Levoy et al 2009 Zhang and Levoy 2009: Observable Light Field Oh, Raskar, Barbastathis 2009: Augmented Light Field

77. NETRA = Inverse of Shack-Hartmann Spot Diagram on LCD Cell Phone Display Eye Piece

78. NETRA = Inverse of Shack-Hartmann Spot Diagram on LCD Cell Phone Display Eye Piece

79. Spot Diagram on LCD Inverse of Shack-Hartmann User interactively creates the Spot Diagram Displace 25 points

80. Spot Diagram on LCD Inverse of Shack-Hartmann User interactively creates the Spot Diagram Displace 25 points but 3 parameters

82. Capture Analyze Display 5D: Looking around corners 6D: View and Lighting Aware 4D: Rank Deficient, multilayer 4D: Netra for Optometry 4D, 6D, 8D: Augmented Light Field MIT Media Lab Ramesh Raskar http://raskar.info Shift Glass ` = WDF Light Field Augmented LF