Computational Displays in 4D, 6D, 8D
We have explored how light propagates from thin elements into a volume for viewing for both automultiscopic displays and holograms. In particular, devices that are typically connected with geometric optics, like parallax barriers, differ in treatment from those that obey physical optics, like holograms. However, the two concepts are often used to achieve the same effect of capturing or displaying a combination of spatial and angular information. Our work connects the two approaches under a general framework based in ray space, from which insights into applications and limitations of both parallax-based and holography-based systems are observed.
Both parallax barrier systems and the practical holographic displays are limited in that they only provide horizontal parallax. Mathematically, this is equivalent to saying that they can always be expressed as a rank-1 matrix (i.e, a matrix in which all the columns are linearly related). Knowledge of this mathematical limitation has helped us to explore the space of possibilities and extend the capabilities of current display types. In particular, we have designed a display that uses two LCD panels, and an optimisation algorithm, to produce a content-adaptive automultiscopic display (SIGGRAPH Asia 2010).
(Joint work with R Horstmeyer, Se Baek Oh, George Barbastathis, Doug Lanman, Matt Hirsch and Yunhee Kim) http://cameraculture.media.mit.edu
In other work we have developed a 6D optical system that responds to changes in viewpoint as well as changes in surrounding light. Our lenticular array alignment allows us to achieve such a system as a passive setup, omitting the need for electrical components. Unlike traditional 2D flat displays, our 6D displays discretize the incident light field and modulate 2D patterns in order to produce super-realistic (2D) images. By casting light at variable intensities and angles onto our 6D displays, we can produce multiple images as well as store greater information capacity on a single 2D film (SIGGRAPH 2008).
Ramesh Raskar joined the Media Lab from Mitsubishi Electric Research Laboratories in 2008 as head of the Lab’s Camera Culture research group. His research interests span the fields of computational photography, inverse problems in imaging and human-computer interaction. Recent inventions include transient imaging to look around a corner, next generation CAT-Scan machine, imperceptible markers for motion capture (Prakash), long distance barcodes (Bokode), touch+hover 3D interaction displays (BiDi screen), low-cost eye care devices (Netra) and new theoretical models to augment light fields (ALF) to represent wave phenomena.
In 2004, Raskar received the TR100 Award from Technology Review, which recognizes top young innovators under the age of 35, and in 2003, the Global Indus Technovator Award, instituted at MIT to recognize the top 20 Indian technology innovators worldwide. In 2009, he was awarded a Sloan Research Fellowship. In 2010, he received the Darpa Young Faculty award. He holds over 40 US patents and has received four Mitsubishi Electric Invention Awards. He is currently co-authoring a book on Computational Photography. http://raskar.info
What Are The Drone Anti-jamming Systems Technology?
Raskar Keynote at Stereoscopic Display Jan 2011
1. Camera Culture Ramesh Raskar Camera Culture MIT Media Lab Computational Displays in 4D, 6D and 8D
2. Slow Glass: Time Shift http:// baens-universe.com/articles/otherdays Light of Other Days by Bob Shaw http://www.fantasticfiction.co.uk/s/bob-shaw/other-days-other-eyes.htm
16. Camera Culture Ramesh Raskar Team Moungi G. Bawendi, Professor, Dept of Chemistry, MIT James Davis, UC Santa Cruz Andreas Velten, Postdoctoral Associate, MIT Media Lab Ahmed Kirmani, RA, MIT Media Lab Tyler Hutchison, RA, MIT Media Lab Rohit Pandharkar, RA, MIT Media Lab Andrew Matthew Bardagjy, RA, MIT Media Lab Everett Lawson, MIT Media Lab Ramesh Raskar, MIT Media Lab
17. Capture Analyze Display 5D: Looking around corners 4D: Plenoptic Camera 3D: Flutter Shutter Camera 6D: View and Lighting Aware 4D: Rank Deficient 4D: Netra for Optometry 4D, 6D, 8D: Augmented Light Field Shift Glass
18. Capture Analyze Display 5D: Looking around corners 4D: Plenoptic Camera 3D: Flutter Shutter Camera 6D: View and Lighting Aware 4D: Rank Deficient 4D: Netra for Optometry 4D, 6D, 8D: Augmented Light Field
23. 6D Photo Frames One Pixel of a 6D Display = 4D Display Single Pixel of 6D Frame Martin Fuchs, Ramesh Raskar, Hans-Peter Seidel, Hendrik P. A. Lensch 1 2 1 1 2D 2D 2D
28. Converting LCD Screen = large Camera for 3D Interactive HCI and Video Conferencing Matthew Hirsch, Henry Holtzman Doug Lanman, Ramesh Raskar Siggraph Asia 2009 BiDi Screen
42. rear mask: f 1 [i,j] front mask: g 1 [k,l] reconstruction (central view) Optimization: Iteration 1 Daniel Lee and Sebastian Seung. Non-negative Matrix Factorization. 1999. Vincent Blondel et al . Weighted Non-negative Matrix Factorization. 2008.
43. Optimization: Iteration 10 rear mask: f 1 [i,j] front mask: g 1 [k,l] reconstruction (central view) Daniel Lee and Sebastian Seung. Non-negative Matrix Factorization. 1999. Vincent Blondel et al . Weighted Non-negative Matrix Factorization. 2008.
44. Optimization: Iteration 20 rear mask: f 1 [i,j] front mask: g 1 [k,l] reconstruction (central view) Daniel Lee and Sebastian Seung. Non-negative Matrix Factorization. 1999. Vincent Blondel et al . Weighted Non-negative Matrix Factorization. 2008.
45. Optimization: Iteration 30 rear mask: f 1 [i,j] front mask: g 1 [k,l] reconstruction (central view) Daniel Lee and Sebastian Seung. Non-negative Matrix Factorization. 1999. Vincent Blondel et al . Weighted Non-negative Matrix Factorization. 2008.
46. Optimization: Iteration 40 rear mask: f 1 [i,j] front mask: g 1 [k,l] reconstruction (central view) Daniel Lee and Sebastian Seung. Non-negative Matrix Factorization. 1999. Vincent Blondel et al . Weighted Non-negative Matrix Factorization. 2008.
47. Optimization: Iteration 50 rear mask: f 1 [i,j] front mask: g 1 [k,l] reconstruction (central view) Daniel Lee and Sebastian Seung. Non-negative Matrix Factorization. 1999. Vincent Blondel et al . Weighted Non-negative Matrix Factorization. 2008.
48. Optimization: Iteration 60 rear mask: f 1 [i,j] front mask: g 1 [k,l] reconstruction (central view) Daniel Lee and Sebastian Seung. Non-negative Matrix Factorization. 1999. Vincent Blondel et al . Weighted Non-negative Matrix Factorization. 2008.
49. Optimization: Iteration 70 rear mask: f 1 [i,j] front mask: g 1 [k,l] reconstruction (central view) Daniel Lee and Sebastian Seung. Non-negative Matrix Factorization. 1999. Vincent Blondel et al . Weighted Non-negative Matrix Factorization. 2008.
50. Optimization: Iteration 80 rear mask: f 1 [i,j] front mask: g 1 [k,l] reconstruction (central view) Daniel Lee and Sebastian Seung. Non-negative Matrix Factorization. 1999. Vincent Blondel et al . Weighted Non-negative Matrix Factorization. 2008.
51. Optimization: Iteration 90 rear mask: f 1 [i,j] front mask: g 1 [k,l] reconstruction (central view) Daniel Lee and Sebastian Seung. Non-negative Matrix Factorization. 1999. Vincent Blondel et al . Weighted Non-negative Matrix Factorization. 2008.
57. Is hologram just another ray-based light field? Can a hologram create any intensity distribution in 3D? Why hologram creates a ‘wavefront’ but PB does not? Why hologram creates automatic accommodation cues? What is the effective resolution of HG vs PB?
58. Parallax Barrier: N p =10 3 pix. Hologram: N H =10 5 pix. θ p =10 pix w θ H =1000 pix ϕ P ∝w/d ϕ H ∝λ/t H Fourier Patch Horstmeyer, Oh, Cuypers, Barbastathis, Raskar, 2009
59. Augmenting Plenoptic Function Wigner Distribution Function Traditional Light Field WDF Traditional Light Field Augmented LF Interference & Diffraction Interaction w/ optical elements ray optics based simple and powerful wave optics based rigorous but cumbersome Oh, Raskar, Barbastathis 2009: Augmented Light Field
60. Light Fields Goal: Representing propagation, interaction and image formation of light using purely position and angle parameters Reference plane position angle LF propagation (diffractive) optical element LF LF LF LF LF propagation light field transformer
61. Augmented Lightfield for Wave Optics Effects Wigner Distribution Function Light Field LF < WDF Lacks phase properties Ignores diffraction, interferrence Radiance = Positive ALF ~ WDF Supports coherent/incoherent Radiance = Positive/Negative Virtual light sources LF Augmented Light Field WDF
62.
63. L(x, θ ) W(x,u) W m = sinc d = delta q W m p q p d(θ) p q q d(θ) * p W m * * Rays: No Bending 1 Fresnel HG Patch θ u * Zooming into the Light Field
65. - Transform <t(x+x ʹ /2)t*(x-x ʹ /2)> Interference xʹ x (a) Two Slits, Coherent t(x+x ʹ /2)t*(x-x ʹ /2) W(x,u) Rank-1 t(x 1 )t*(x 2 ) R 45 , D Transform -1 u
66. L2 L1 L3 ϕ 1 ϕ 1 ϕ 1 ϕ 1 L 1 (x,θ) L 2 (x,θ) L 3 (x,θ) d z 1 h H r z 2 L 1 (x,θ) L 2 (x,θ) L 3 (x,θ) s 1 m 2 (a)
67. A B C Vary Illumination Direction: -5 ̊ , 0 ̊, 5 ̊ A B C A … -5 ̊ 5 ̊ 0 ̊ No Slits 24mm 36mm t H =25μm w=125μm z H =10cm (c)
68. M2 M1 M3 ϕ 1 ϕ 1 L 1 (x,θ) L 2 (x,θ) L 3 (x,θ) d z 1 r z 2 s 1 m 2 s 1 m 2 s 1 m 2 s 1 * s 1 s 1 s 1 * Rank-1 Rank-1 Rank-3
69. Is hologram just another ray-based light field? Can a hologram create any intensity distribution in 3D? Why hologram creates a ‘wavefront’ but PB does not? Why hologram creates automatic accommodation cues? What is the effective resolution of HG vs PB?
70. NETRA: Interactive Display for Estimating Refractive Errors and Focal Range Vitor Pamplona Ankit Mohan Manuel Oliveira Ramesh Raskar
71. Vitor Pamplona Ankit Mohan Manuel Oliveira Ramesh Raskar NETRA: N ear E ye T ool for R efractive A ssessment
72. 0.6B uncorrected refractive errors NETRA at LVP Eye Institute 6.5 Billion people 4.5B with Mobile phone 2B refractive errors
73. * Phoropter-based: $5,000.00 Needs expert, Moving parts, Shining lasers Retino scope w/ Lenses Auto-refracto-meter Chart with Lenses In-Focus: Focometer Optiopia Solo-health: EyeSite NETRA Technology Shining Light plus lenses Fundus Camera Moving lenses + target Moving lenses + target Reading chart on monitor Cellphone + eyepiece Cost to buy $2,000* ~$10,000 ~$100 ~$495 ~$200 -- $30 Cost per test ~$36 ~$36 ~$5 -- -- -- ~$1 Data capture No Comp. No No No Comp. Phone Mobility <500g >10Kg 2kg 1kg <5kg >10Kg <100g Speed Fast Fast Medium Medium -- Fast Fast Scalability No No No Yes Probably No Yes Accuracy 0.15 0.15 0.5 0.75 -- -- <0.5 Self evaluation No No Yes Yes Yes Yes Yes Electricity Req No Yes No No -- Yes No Astigmatism Yes Yes Yes/No No -- Yes Yes Network No Yes No No No Yes Yes Training High High High Medium Medium Low Low
75. Shack-Hartmann Wavefront Sensor Laser Sensor Microlens Array Planar Wavefront Shack & Platt 1971 Liang et al 1994 David Williams et al, Rochester Spot Diagram
76. Laser Sensor Displacement = Local Slope of the Wavefront Spot Diagram Shack-Hartmann Wavefront Sensor Shack-Hartmann ~ Lightfields Levoy et al 2009 Zhang and Levoy 2009: Observable Light Field Oh, Raskar, Barbastathis 2009: Augmented Light Field
77. NETRA = Inverse of Shack-Hartmann Spot Diagram on LCD Cell Phone Display Eye Piece
78. NETRA = Inverse of Shack-Hartmann Spot Diagram on LCD Cell Phone Display Eye Piece
79. Spot Diagram on LCD Inverse of Shack-Hartmann User interactively creates the Spot Diagram Displace 25 points
80. Spot Diagram on LCD Inverse of Shack-Hartmann User interactively creates the Spot Diagram Displace 25 points but 3 parameters
81.
82. Capture Analyze Display 5D: Looking around corners 6D: View and Lighting Aware 4D: Rank Deficient, multilayer 4D: Netra for Optometry 4D, 6D, 8D: Augmented Light Field MIT Media Lab Ramesh Raskar http://raskar.info Shift Glass ` = WDF Light Field Augmented LF
Notes de l'éditeur
This kind of a technique can be used in other scenarios as well such as Rescue and Planning
Robot and car navigation to avoid collisions by estimating position of objects around the bend
Martin Fuchs, Ramesh Raskar, Hans-Peter Seidel, Hendrik P. A. Lensch Siggraph 2008
This video is only for 4D display that responds to light Bonny’s lenticular prints outside
Since we are adapting LCD technology we can fit a BiDi screen into laptops and mobile devices.
Here’s a quick teaser to illustrate the capabilities I’m describing. <wait for multi- to hover part to pass> Here you see a user pulling her hands away to rotate and zoom a 3-D model. We also show a use of 3D gesture to navigate a 3D world. We support these modes by creating an array of virtual cameras on an LCD using a technique known as Spatial Heterodyning. Because we’re using an optical technique, we also enable dynamic relighting applications, where real-world lighting is transfered to a rendered scene.
So here is a preview of our quantitative results. I’ll explain this in more detail later on, but you can see we’re able to accurately distinguish the depth of a set of resolution targets. We show above a portion of portion of views form our virtual cameras, a synthetically refocused image, and the depth map derived from it.
A cross section through a single M. rhetenor scale. Light reflected off each level of the “Christmas tree structure†gives the butterfly its iridescent color. Credit: Pete Vukusic, University of Exeter
Augmented plenoptic function the motivation, to augment lf, model diffraction in light field formulation
In this paper, we show a self-optometry solution. You look at a cell phone display thru a clip-on eye piece, interactively align a few patterns, hit calculate and get data for your eye prescription.
We call our tool NETRA: near eye tool for refractive assessment such as nearsightedness/far/astigmatism Basic idea is to create a unique interactive lightfield display near the eye and is possible due to the highresolution of modern LCDs.
2 billion people have refractive errors And half a billion in developing countries worldwide have uncorrected vision that affects their daily livelyhood. They don’t have access to an optometrist or it simply too expensive. While making and distributing of lenses has become quite easy now, surprisingly there is still no easy solution for measuring eyesight. Can we use a fraction of the 4.5B cellphone displays to address this problem?
For better precision, there are many kinds of solutions, some really clever. The beauty of netra is that it avoids moving parts or shining lasers, and all intelligence is in the software.
The most accurate method is based on a so called SH WS. It involves shining a laser at the back of the retina and observing the wavefront using a sophisticated sensor. We ask user to generate a spot diagram. But navigating in a high dimensional space is challenging so we come up with a strikingly simple approach to let the user interactively create the spot diagram. We are first to make connection between Shack Hartmann and Lightfields (and it goes well with recent work in computational photography about ALF and Zhang/Levoy). Connection to Adaptive optics/ Astronomy. The way that this device works is that, it shines a lasers in the eye, the laser is reflected in the retina and comes out of the eye being distorted by the cornea. These light rays reaches an array of lenses that focus them to dots in a sensor. The device measures how much this dots deviate from the ideal case. Since it uses lasers, the device is expensive and requires trained professionals
For a normal eye, the light coming out of the eye forms a parallel wavefront. The sensor has a lenslet array and we get a spot diagram of uniform dots. This lenslet should remind you of a lightfield camera, and in fact Levoy and others showed last year that there is a close relationship between the two. In addition, Zhang and Levoy, plus our grp has shown the relationship between wavefront sensing and lightfield sensing.
When the eye has a distortion, the spot diagram is not uniform. And the displacement of the spots from the center indicates the local slope of the wavefront. From the slope one can integrate and recover the wave shape.
NETRA uses an exact inverse of this sensor. We get rid of the laser and we instead show the same spot diagram in a cellphone display. For normal eye, it will appear as a dot to the user. And then we replace the sensor for a light field display. If the user sees a single red dot, he does not need glasses, but if he sees more than one, he interacts with this display.
NETRA uses an exact inverse of this sensor. We get rid of the laser and we instead show the same spot diagram in a cellphone display. For normal eye, it will appear as a dot to the user. And then we replace the sensor for a light field display. If the user sees a single red dot, he does not need glasses, but if he sees more than one, he interacts with this display.
For eye with distortion, the user will interactively displace the 25 points so that he will see a single spot. Of course changing 25 spot locations is cumbersome, but we realize that there are only 3 parameters for eye-prescription and we help the user navigate thru this space efficiently. But if you think about these theory, you will realize that we have the dual of the shack-hartmann. First we though out the laser.
For eye with distortion, the user will interactively displace the 25 points so that he will see a single spot. Of course changing 25 spot locations is cumbersome, but we realize that there are only 3 parameters for eye-prescription and we help the user navigate thru this space efficiently. But if you think about these theory, you will realize that we have the dual of the shack-hartmann. First we though out the laser.
Since we are relying on the user interaction, the subject has to be aware of the alignment tasks. So, very young Children may not be able to run the test. Instead of just one eye, one may use both eyes to exploit convergence. And of course, the resolution of NETRA itself is a function of the resolution of the display. With a 326 dpi display, resolution is 0.14 diopters and presciption glasses come in increments of 0.25 diopters. So our system is already sufficiently accurate.