The document summarizes two approaches to implementing foveated imaging in CMOS image sensors: (1) A pyramidal architecture with multiple rings of pixels having different integration times, allowing for dynamic range enhancement. (2) A universal multiresolution sensor using a 3T pixel design that allows pixels to be grouped and averaged, enabling adaptive resolution. Both designs aim to mimic the human retina and improve efficiency over traditional sensors. The pyramidal and multiresolution sensors were fabricated in 0.18um CMOS technology and are being tested for applications like video conferencing and industrial inspection.
ICT role in 21st century education and it's challenges.
Ei2004 presentation
1. Foveated Architectures for
CMOS Image Sensors
Fayçal Saffih1, Richard Hornsey2
1
Integrated Camera Group, Electrical & Computer Engineering
Department, University of Waterloo, ON, Canada
2
Centre for Vision Research, Department of Computer Science,
York University, Toronto, Ontario, Canada
Electronic Imaging 2004, San Jose
2. Outline
Introduction & Motivation
Motivation Implementation
Pyramidal CMOS Image Sensor
Applications
Universal Multiresolution CMOS Image Sensor
Applications
Final Conclusion.
Electronic Imaging 2004, San Jose
3. Introduction & Motivation
CMOS image sensor development levels:
Device level: Optical charge devices such as photodiodes.
Goal: Dynamic range enhancement, Fill factor increase…etc.
Circuit Level: Amplifiers, buffers, ADC’s…etc.
Goal: Efficient charge transfer, high speed acquisition, noise filtering…etc
System Level: Signal processing blocks such as motion detection, image
processing blocks, …etc.
Goal: Image information extraction (motion, segmentation…etc), performance
enhancement (data reduction, multiresolution, foveated vision…etc).
Motivation
Imitation of biological visual systems such as Human Visual
System.
Efficient Data Reduction.
Efficient Image Information Transfer.
Minimization of Power Consumption.
Emphasize on the importance of architecture design in CMOS Imagers.
Philosophy: Image acquisition is a Vision needs vision
Electronic Imaging 2004, San Jose
4. Motivation Implementation
System level implementation
The pixel structure is the 3T Active Pixel Sensor (APS).
The acquisition system is a non-classical (non-
orthogonal) architecture.
Solution: Pyramidal CMOS image sensor
Device level implementation
The pixel structure is a non-classical (3T active pixel
sensor).
The acquisition system is a classical orthogonal
architecture.
Solution: Universal Multiresolution CMOS Image Sensor
Electronic Imaging 2004, San Jose
6. Classical Acquisition System
The orthogonal acquisition
architecture along with raster
scanning suffers from the
following issues:
1D sampling architecture.
Unique integration time
Different sampling speed
between horizontal and vertical
axis of the image. This leads to
anisotropic distribution of
motion blur, being higher in the
vertical axis than in the
horizontal.
Electronic Imaging 2004, San Jose
7. Human Fovea Architecture
Dynamic Range is higher in cones than Circular symmetry of the fovea photocells
rods. (rods and cones) has lead to symmetrical yet
HVS system DR (~106) is much larger co-centric image sampling.
than the photoreceptor DR (~102).
Electronic Imaging 2004, San Jose
9. Pyramidal Acquisition
Architecture
Floorplan of the imager is
composed of square
pixels (16µmx16µm)
orthogonally compacted.
Reset and select signals
are shared among each
ring
The output buses are
diagonals to dump the ring
output into 8 CDS block at
the base of the pyramid
Electronic Imaging 2004, San Jose
10. Bouncing Scanning Scheme
Estimated Integration time of a
ring r for inward Scanning:
r +1
Trin = 2 ∑ iT s + ( R − r )Tspl + rT s
i = R
i → i −1
Estimated Integration time of a
ring r for outward Scanning:
r −1
Trout = 2 ∑ iTs + ( r − 1)Tspl + rTs
i =1
i → i +1
Electronic Imaging 2004, San Jose
11. Ring Integration Time
Estimated of all rings’ integration time for inward (ISc) and Rings’ RMS of inward (ISc) and bounced scanning
bounced scanning (BSc) at 40Fps. (BSc) at 8Fps under light intensity of 43.33uW
(current testing)
Electronic Imaging 2004, San Jose
12. Dynamic Range Fovea
Each ring in the pyramidal imager The resulting dynamic range
will have two different integration enhancement is made with two
time. Fusing their two output will acquired scenes. Thus, this type
enhance the ring’s dynamic range of dynamic range enhancement is
by:20log(Tint1/Tint2), where Tint1 ≥Tint2 called, intrascene dynamic range.
Electronic Imaging 2004, San Jose
13. Experimental Results
Inward and Bounced Images from
the Pyramid Imager at 14lux@29Fps
Fused Image of the inward Pyramidal CMOS Image Sensor
and bounced images Layout
Electronic Imaging 2004, San Jose
14. Applications
Applications that need optimal data transfer such as:
Video-Phone
Internet cameras
Applications that need foveated vision:
IndustrialInspection
Surveillance Cameras
Low Vision Enhancement
Consumers Cameras
Electronic Imaging 2004, San Jose
15. Disadvantages
Complex data structure for image re-construction
Mismatch between sampling capacitors between the
different pyramid clusters may create some undesired
artifacts.
The central diagonal in the pyramid is sampled by
capacitors with twice the capacity of their neighbor pixels.
Although the above are real problems in our CMOS imager, they
are not limitations as they are all solvable problems.
Electronic Imaging 2004, San Jose
16. Conclusion
New architecture (hardware) and scanning scheme
(software) for CMOS imagers have been suggested.
The benefits of the new approach in image sampling are
due to:
The 2D nature of the sampling rings
Centricity of the sampling units has lead to foveated dynamic
range enhancement topology.
The difference in sampling architectures in CMOS
imagers greatly impacts their performances.
Electronic Imaging 2004, San Jose
17. Device Level approach
Universal
Multiresolution
CMOS Image Sensor
Electronic Imaging 2004, San Jose
18. Multiresolution CMOS Image Sensor
Motivation
Only small parts of image are regions of interest.
Regions of low interest need to be sub-sampled or
averaged.
With increasing image resolution the ultimate limitation
is image transfer especially for video broadcasting.
Minimizing of power consumption requires a
minimization of power consumption of the imager OR
minimization of “data of interest” to be readout
Programmability and expandability are the most
important feature of such an architecture in order for it
to span the largest field of applications:
Universality
Electronic Imaging 2004, San Jose
19. The Kernelling
Just two clock cycles are enough for the kernel averaging: Fast Kernelling
Electronic Imaging 2004, San Jose
23. Application: Programmability
Fundamental Foveated Multiresolution Fundamental Foveated Multiresolution
With horizontally-rectangular kernels With vertically-rectangular kernels
Effect of kernel topology on spatial filtering
Electronic Imaging 2004, San Jose
25. Final Conclusion
Two different approaches for implementing
foveated imaging have been suggested.
The two imagers were fabricated in dual voltage
1P6M standard 0.18µm CMOS technology. The
two imagers under test.
Parallelism, programmability and expandability
were the keys behind the proposed CMOS
image sensors’ architectures
Electronic Imaging 2004, San Jose
26. Acknowledgement
The authors are grateful to:
Canadian Microelectronics Corporation.
NSERC Canada.
Betacom Inc.
Thank you !
Electronic Imaging 2004, San Jose
Notes de l'éditeur
Efficient data reduction=transferring just important (central) image information. Enhance Image Information Transfer=creating enough room for transferring important image information (details)
Viewpoint: Sampling architecture is not necessarily the same as the display architecture. Raster scanning was originally suggested for CRT displays but was adopted since 60’s for MOS imagers. Biological sampling architectures are very promising to implement efficient smart sampling architectures. Human fovea is a clear example.
1- Integration time is the time elapsed between successive ring reset and ring signal readout. 2- Ring integration time in the bouncing scanning scheme is deduced including the bouncing at the inner and outer rings. 3- Due to the different of cardinals of the inner and outer rings, bouncing scanning at the inner ring will result lower integration time for outward scanning than bouncing at the outer ring for the inward scanning.