Can we create a display that adapts itself to improve one's eyesight? Top figure compares the view of a 2.5-diopter farsighted individual in regular and tailored displays. We use currently available inexpensive technologies to warp light fields to compensate for refractive errors and scattering sites in the eye.
13. and there is no mismatch between
convergence and accommodation
14. Tailoring Process Myopic View: -3D
Focusing Here He can focus up to 33cm (12in)
Blurred
Image
Distance Display-Eye: 50cm
15. Tailoring Process Myopic View: -3D
Light-field Focusing Here He can focus up to 33cm (12in)
Display
Distance Display-Eye: 50cm
16. Tailoring Process Myopic View: -3D
Light-field Focusing Here He can focus up to 33cm (12in)
Display
Distance Display-Eye: 50cm
17. Tailoring Process Myopic View: -3D
Light-field Focusing Here He can focus up to 33cm (12in)
Display
Pixel Size of
96um at 33cm
1 arc-minute
Distance Display-Eye: 50cm
Resolution
18. Pixel Size and Alignment
1 arc
minute
Pixel on the
Virtual image
Zoomed Version
19. Pixel Size and Alignment
1 arc
minute
Pixel on the
Virtual image
Zoomed Version
24. Tailoring Process Subject’s prescription
-2D -1D @ 90
Light-field Two Points in Focus He focus at 30cm to 50cm.
Display
50cm
30cm
Where the
Subject’s
Accommodate
25. Wavefront Maps
90 degrees
Sphere: -2D
Cylinder: -1D
Axis: 90°
0 degrees
k
Lens focal length in k
Zernike Polynomials
35. 3.25D Myopic Eye
1mm
using a display at
37cm from the eye
Input Image
As Seen on a Regular Display As Seen on a Tailored Display
Resolution of 0.87 arc minutes
36. 3.5D Myopic Eye with
0.9mm
Cataract
Nuclear Cataracts
Display at 47cm from the eye
7.9mm
Input
p = 20mm
No Cataracts No Cataract Tailoring Tailored for Cataracts
Resolution of 0.87 arc minutes
55. Limitations
• Eyes fixed relative to the display
– Similar to glasses-free 3D Displays
– Higher-order requires more fixation
• High-resolution LCD panels (PPI)
– Giga-pixel displays for monitors
• Other ocular diseases may not be corrected for
61
56. Tailored Displays to Compensate
for Visual Aberrations
• The First Multi-depth Multi-focus Display
– Time-varying Optical Corrections
– Resolution around 1 arc minute
• Correcting for a Wide Range of Conditions:
– Myopia, Hyperopia and Presbyopia
– Astigmatism and High-order aberrations
– Cataracts
www.TailoredDisplays.com
57. Future Works
• Retinal Diseases on the Tailoring Pipeline
– Retinal Displacements (Image Warping)
– Color Blindness (Re-coloring)
• Light-Field to Hardware Mapping
– Temporal and Context-Based Optimizations for Brightness
• Stereo Multi-focus Multi-depth Displays
58. Vitor Pamplona TailoredDisplays.com
Manuel M. Oliveira
Daniel Aliaga
Ramesh Raskar
Tailored Displays to Compensate
for Visual Aberrations
59. Tailored Displays to Compensate
for Visual Aberrations
• The First Multi-depth Multi-focus Display
– Time-varying Optical Corrections
– Resolution around 1 arc minute
• Correcting for a Wide Range of Conditions:
– Myopia, Hyperopia and Presbyopia
– Astigmatism and High-order aberrations
– Cataracts
www.TailoredDisplays.com
60. Cell Phone-based
Refractive Measurements
The Inverse of
Shack-Hartmann
Vitor Pamplona, Manuel Oliveira, Ankit Mohan, Ramesh Raskar
63. Tailored Displays to Compensate
for Visual Aberrations
• The First Multi-depth Multi-focus Display
– Time-varying Optical Corrections
– Resolution around 1 arc minute
• Correcting for a Wide Range of Conditions:
– Myopia, Hyperopia and Presbyopia
– Astigmatism and High-order aberrations
– Cataracts
www.TailoredDisplays.com
65. Challenge
Billions of People with
Uncorrected Refractive Error, by Region
1.70
1.80
1.60
1.40
1.20
1.00
4 billion people estimated 0.80
to need eyeglasses 0.60 0.50
worldwide 0.40
0.20 0.13 0.10
0.02
0.00
Emerging Africa & Latin Europe North
Asia Middle America America
East
Source: Essilor, Infomarket 2009, CPB Research,
66. Challenge
Due to:
1. Heavy “near work”
2. Intensity of education.
90% of high-school
kids are myopic in 3. Games, computers and TV.
major Asian cities 4. Insufficient time spent
outdoors.
Morgan et al. 2012. Science 2012.
68. Needs expert, Annoying Experience, Limited
Eye Glasses Contact Multi- Surgery Adaptive Adaptive Tailoring
Lenses focals optics spectacles
Technology Coated Hard Flexible/Hard Hard Lenses LASIK and Dynamic Liquid / Parallax
Lenses+Frame Lenses with many Cataract micro mirror Electronic / Barrier
focal Surgery array on Mechanic
powers contacts Lenses
Cost in the US <$100 <$50 ~$200 ~$2,000 Expensive $30 Feature
Quality 1 arc-min 1 arc-min 1 arc-min 1 arc-min 1 arc-min 1 arc-min 1 arc-min
Astigmatism Yes Yes Yes Yes Yes No Yes
Cataracts No No No Yes Yes No Yes
Keratoconus No Yes No No No No Yes
Adaptive No No No No Yes Yes Yes
Carry on Yes No Yes No No Yes No
Need doctor Yes Yes Yes Yes ? No No
Maintenance Easy Medium Easy Nothing Hard Medium Easy
Invasive No Yes No Yes Yes No No
Imagine a special kind of display that adjusts itself to compensate for vision problems in such a way that it avoids the use of eyeglasses when looking at these displays. We call our solution the Tailored Display. The basic idea is to create a light-field rendering technique that projects partial holograms in space for the visually impared using a low-cost 3D hardware, such as simple stack of LCDs.
Driving may be a problem for farsighted individuals that usually rely on reading glasses or bi-focals to see both the streets and the dashboard of the car at, kind of, the same time.
If they don’t wear them, the dashboard, of course, gets blurry.
A Tailored display allow a safe drive by compensating for whatever refractive condition the driver has in software in real time.
Although wearing eye glasses appear to be an easy solution, they are annoying in many ways, Specially if you have to put them on and off all the time, such as when you need them just to check you phone.
OR when you are running and you don’ t want to carry your glasses but you still want to read to your newest gadget to check for you performance.
The beauty of tailoring is that it avoids extra carry-ons for certain activities. And since all the intelligence is in the software, you can share the same hardware with many people and all of them will be corrected for their problems.
So, how does it work? We have low order aberratons and high order aberrations. Let’ s start easy. Low-order refractive aberrations are Myopia, Hyperopia, And presbiopia. And they essentially shift the focal range or in a better word the accoomodation range. MyopiaHyperopya shifts it farther from the viewreAnd Presbyopia decreases the range.
Usually a person that does not need eyeglasses has the display inside his focal range.He can accommodate anywhere in this range to see the display in focus.
Myopia, on the other way, shifts the focal range closer to the viewer The device is placed outside his accommodation range, so he cannot focus there But his desire to see the display will make his eyes accommodate to the closest point in focus. The tailoring process creates a hologram image at the plane the subject is accommodating. And this is one of the big points of the paper, the subject’s point of focus or accommodation actually does not change.
For hyperopes/nearsighted the accommodation range is shifted against the viwers. These individuals usually have the display before their closest point in focus and thus they see a blurred image. The tailoring process then moves the image away from the eye, inside their focal range. And this intuition of working where the subject is focusing instead of trying to chang his accommodation state led us to our first lesson we learned during this work
That we can easily create a hologram for the visually impaired.
Mainly because we do not need to create any accommodation clues to allow them to focus in mid air. They are already focusing there. The fact that they are not focusing in the display plane, makes things a lot easier. // We know how tricky is to drive accommodation to create a multi-focus display. Specially using low-cost hardware such as stack of LCDs.
And there is no clear mismatch between convergence and accommodation, as standard 3D displays have. The viewer is already used to the situation they have. We are not doing anything unnatural for him.
So, an emmetrope can see the image clearly because his accommodation moves to conjugate the real object with the retina.
But say a 3D myope don’t have enough accommodation to get to focus at 50cm-away display. They light coming from this device converges before the retina and he only sees a blur. In order to minimize the blur, this subject will accommodate at 33cm from his eye.
3 dioptermyope doesn’t have enough accommodation to get to focus at 50cm-away display. In order to minimize the blur, this subject will accommodate at 33cm from his eye. Trying to focus in in mid air. To create the tailored display, we need to replace the standard display by a stack of LCDs
that allows us to cast directed light. And Then recreate the image, pixel by pixel at 33cm from that eye. Every pixel on this image, will be composed by a set of pixels from both LCD that converge at 33cm, and spread again simulating this virtual point in space.
We just repeat the process for all the pixels and we have a hologram right there.
Another important fact we learned while doing this work is that we have the resolution required to create that. In a ophthalmology perspective, a corrected human eye can distinguish features on a resolution of about 1 arc minute, which would mean in this case that we need 96 microns pixels at 33cm. A pretty reasonable resolution,
Except for the fact that we are playing on a discrete domain and pixels on LCD1 and 2 have a given size, And we probably won’ t be able to align them all into a single spot. If the misalignment is bigger than 1-arc minute the individual will end up seeing a blured pixelAnd building a tailored display would not make any sense.
We want this misalignment to be smaller than the human resolution. Meaning the person cannot see this blur. And for that we need high-resolution displays.
And we ended using a resolution that is around 1800 dpi, in which color chaneels are down to 4 microns.These displays you can find in many nonexpensive equipment such as on head mounted displays (in which for 150 dollars you get two) and LCD projectorsOn that resolution we can pretty much correct for a large set of conditions and optical powers.
Things get more exciting when you have more interesting aberrations. Astigmatism, for instance, create two main focal points for each accommodative state.
In this one we have a zero power at horizontal
And positive power at 90 degrees. The intermediate meridians between these two are an non-linear interpolation between these powers.
So, let say we have a 2D myope individual with a 1D astigmatism at 90 degrees, looking to our display. This means that the horizontal meridian he will focus at 30cm while the vertical at 50cm. The subject will usually accommodate in between them, to try to get both meridians with the minimum blur . What we need to do is to create an image at 50cm and another one at 30. That would be enoug, but you can also go beyond and compensate for all other meridians.
To make that happen we need to usea good interpolation function that will create this wavefront map. Essentially every prescription can be translated to these map of focal lengths and we can use them as an input for the tailoring procedure.
The procedure takes a wavefront map and the image that will be displayedEssentially, We raytrace for each pair of pixels from both lcds, we check where the light ray is falling on the retina and we sample the input image. to set the color of the pixel on LCD1Doing this for all the pixels we end up creating these holograms in space according to the subjects wavefront map. This one in particular has a 2D optical coma. In the end, we have a distorted light-field to be rendered.
AT these stage,A single pixel in the retina can come from many virtual depths , each depth for each part of the corneaYou can create as much as you want to increase brightness. `This is why we call it a multi-depth display, because for a given pixel in the retina and a accommodative state we may have objects in many depths
And of course, by distributing the images in space we also have to scale them accodingly.
scale them accordingly.
So, everything I explained since the beginning reduces to this very simple equation based on the parameters of the system.Which is a mapping between the light-field and the image in the retina. The equation can be easily converted to a matrix representation and parallelized in the gpu.
And what about cataracts? Can we do something about it? Cataracts are these clouds you may see in someone’ s eye. They scatter light, decreasing contrast and increasing glare.
They can be measured in terms of density maps, just like the wavefront maps. Each point there represents an attenuation coefficient. And if we use them as input
the procedure can detect where the rays are going throught the blob and remove them.
WE then normalize the image by the number of rays each pixel in the retina is receiving. And we have the image ready to be displayed.
We tested the system in two prototypes, one uses a dual stack of LCDs from a low-cost LCD Projector. These lenses you see are optometric lenses to simulate aberrations we want to correct.
And the second prototype comes from a Vizux Head Mounted display with a tiny Lens Array on top of the display. Both are 1800 DPI displays.
So here is a tiny snoopy as an input image for a 3.25 diopter myopic subjectAnd here is how he see it on a regular display at a 37cm away display. And on a Tailored display
The tinysmile is corrected for a 3.5 Myopic subject with Nuclear Cataracts. Here you see a picture without the cataract simulation, With the cataract on the lens of the cameraAnd tailored to avoid scattering.
We also experimented with color images with the size of a tablet for a presbyopic which needs +3D on reading glasses. The display is at 20cm from the eye, but the subject can only focus at 50cm.
So, here is how he would see the image in a regular display without his glasses
And by looking to a tailored display. This tiling effect you see comes from our array of light-field displays. Since our testing prototypes are about 1x1cm, we tiled them up to generate a complete image
For the user evaluations, the idea is to test the method, not the individual. So we prepared 5 holograms from 1 to 5 diopters to be seen by testersWe asked individuals to look at them, one at a time, and vote for the best view,
with a 5D lens, which creates myopia
Or without the lens, in which the subject is supposed to be able to sharply focus on holograms from 1D to 3D (meaning from 1meter to 33cm). The idea is that, if the displayed hologram is inside their focal range, they would prefer seeing it without the lens.
And for the 5 diopter hologram, the subject is supposed to choose the lens all the time.
And the experiment confirmed the expectations we hadAs you can see, from 1 to 3 diopters, the volunteers preffered no lens. For 4 diopters we have a tie. And for 5 diopters, our testers choose the lens 98% of the time.
A similar test for astigmatism indicated that 66% of the testers choose a 4D astigmatic lens when the hologram was corrected for 4D of astigmatism.
For high-order aberrations we asked a individual to vote whether he could see the multiple copies effect in a corrected and non-corrected set of holograms.
And 92% of the corrected holograms he could not see the multiple copies.
We also made a simple test with a Multi-focus, Multi-depth display in which The three first letters of SIGGRAPH are placed in front of the display and the five least letters behind it. And for each depth we have a multi-depths to compensate for a miopic individual. So the viewer will be able to see only part of the word according to his accommodation state.
We have 3 main limitations: The eyes of the viewer must be fixed relative to the display, similar to glasses-free 3D displays. However, real-time eyetracking systems can eliminate this limitation. We need high-resolution LCD panels. But we hope the display industry follows the path of the camera industry, and start using a exceeding resolution to exploit refocusing features.And other ocular diseases may influence the results.
So, in a summany,We have the first multi-focus, multi-depth display which generates images with the resolution around 1 arc minute. We have shown evaluations with cameras and users for myopia, hyperopia, presbyopia, astigmatism, keratoconus and cataracts.,
For the future works we can easily see: Retinal diseases being integrated into the systemBetter light-field to hardware mappings exploring optimizations algorithms. And the evolution to Stereo Multi-focus, Multi-depths Displays.,
Thanks,
So joining low-cost techniques to measure refractive error.
And cataracts
We can think on a new trend in which consumers would be able to measure and get corrected instantly at their homes.
Before I finish, I would like to leave you with a thought to think about. The standard human acuity of an emmetrope - the person that does not need glasses - is about 1 arc-minute. We know that the theoretical resolution of our retina is about 0.3 arc-minute, which means that the average human being is under corrected. And his visual experience could be 3 times superior than what he sees right now. That’s like moving from a you tube video to a blue ray. So, who will be the first to make us free?
Since we have a dual stack of LCDs, we have to deal with Cross-talk.
And the way to do that is to keep the front-lcd apertures at a certain distance form each other. In our prototypes, they are about 0.3mm apart. Pretty much a 90 DPI setup.
Using a regular grid of apertures, you will find that you can actually have multiple points of view, because it does not really matter if the aberration designed for one aperture is going through another.
Two thirds of the world’s population, 4 billion people, have refractive errors. 2 billion are corrected and 2 billion have no idea they need glasses. Majority of the uncorrected cases are in the Emerging Asia mostly due to the lack of eye care but Also to an increasing prevalence of myopia.
Which Today reaches about 90% of the high-school dropouts in major Asian cities This increase of prevalence of the past years has been associated with education, reading, gaming, computers and TVs, an scope we pretty much own these days, specially with the new ebook readers. So, can we help mitigate visual problems by transforming the way we display information and accepting this kind of variability among the population?
In Computer Graphics, we try to understand how perception works to come up with nice graphics, overcome the computing limitations and extend the human abilities to play with the computer. Although we usually design for an average individual, it is clear that we all have distinct abilities to process the sensing data we capture. Can we build a common framework to deal with this intrinsic variability, at least in the perception space, that allows our techniques behave better even for outliers, say un uncorrected myope?
Although wearing eye glasses appear to be an easy solution, they are annoying in many ways. And we can find several reasons just googling. They require periodic visits to a doctor, if you have access to one in your country. The good ones are expensive, having the price of a mobile phone these days. They change your visual, your style.
Although wearing eye glasses appear to be an easy solution, they are annoying in many ways, Specially if you have to put them on and off all the time, such as when you need them just to check you phone. OR when you are running and you don’ t want to carry your glasses but you still want to look to your newst gadget. Good eyeglasses are expensive, having the price of a mobile phone these days. The beauty of tailoring is that it avoids extra carry-ons for certain activities. And since all the intelligence is in the software, you can share the same hardware with many people and all of them will be corrected for their problems.
For the user evaluations, the idea is to test the method, not the individual. So we prepare 5 holograms from 1 to 5 diopters. We asked individuals to look at them, one at a time and vote for the best view,
with a 5D lens, which creates myopia
Or without the lens that only holograms from 1D to 3D (meaning from 1meter to 33cm) are sharp to the tester. The idea was that, if the displayed hologram is inside their focal range, they would prefer seeing it without the lens because they can focus on any of them.
Two thirds of the world’s population, 4 billion people, have refractive errors.
2 billion are corrected and 2 billion have no idea they need glasses. So, can we help mitigate visual problems by transforming the way we display information Accepting this variability among the population making our techniques behave even for outliers, say un uncorrected myope?