An Overview of Market Opportunities for Intelligent Systems Using a Calibrated Camera as an Input Measuring Tool

1. Camera Calibration
and Software Image
Enhancement
Guy Martin BEng MASc
Marie Josée Perreault BAA
Serge Gagné BCom

2. Digital Imaging and Intelligent Systems
Digital imaging is an enabling technology, the key building block for
a broad range of high end, highly automated technologies.
The use of digital cameras as measuring tools in Intelligent Systems
(IS) requires the camera to be calibrated.
We corrected systematic errors found in every camera calibration
model and technique known.
Using our technique as a software lens correction filter,
we demonstrated to National Defence Canada*
8:1 higher measurement accuracy (image or 3D telemetry)
4:1 faster computation time
30% added lossless video compression
… *
(June 2nd 2009)

3. Camera Calibration
Market Wise
Besides Aerospace and Military applications, the technology
can be found applied in a broad range of Intelligent Systems
Major Markets Market Annual
billions $US Growth
Robotics 7.37 7-15%
Service Robots 0.25 40-70%
Artificial Vision 13.44 12.25%
Tele-medicine 95.0 10%
Robot Surgeon 1.0 100%
Transportation 40.04 17%
Security 14.00 20%
Biometrics 1.20 20%
Facial Recognition 0.14 100%
Digital Imaging is forecasted to become in the present century the
next big technological trend setter, as big as the advent of the
silicon chip in the 20th century.

4. Camera Calibration
Market Wise
Our innovation reaches 60% of the optics and photonics market
- 75 billions $ annual sales worldwide
- 2/3 shared between 5,000 US companies
Military and Space Opportunities
- Satellite imaging: digital compression, infrared fusion, sub pixel image
analysis…
- Enhanced night vision: infrared and color image fusion
- Surveillance and object tracking: sub pixel contouring
- Aerial cartography using UAV (Unmanned Airborne Vehicle): 3D from
motion and 3D mapping using a camera video feed
- Video feedback and robotics guidance: docking to ISS and Canadarm-
Dextre visual feedback, rover exploration…
…

5. Camera Calibration
Is knowing how an image projects
through the lens to print on the CCD
camera surface
We measure the behaviour related to
- Camera posture with respect to the scene
- Image deformation introduced by the lens
- Image sensor and lens mechanical assembly tolerances
Once calibrated, through software we can remove image artefacts:
Errors introduced in the image by the imperfect nature of lenses.
We can then use the corrected image as a feedback sensor for
automated tasks in Intelligent Systems.

6. Camera Calibration
We correct the biggest error found in digital imaging
The distortion introduced by the lens
Lens distortion is known to cause
•errors in object measurement and tracking using cameras
•blurring of object edges
•imaging software sensitivity to lighting conditions
•lower digital image and video compression ratios
Using off the shelf cameras and lenses, we demonstrated to National
Defense Canada (DND)
•an accuracy increase of 8:1 in 3D measurements using digital images
•a 4:1 speed increase in image geometric distortion correction
•a model extension for using zooming lenses in telemetry, a world first …
We corrected a systematical error found in every camera model and
calibration technique published.
We can retrieve the shape of a camera pixel with an accuracy better than
10-10 mm (roughly the size of an hydrogen molecule), removing a bias in
the image center as big as 2 pixels.
Every lens parameter is ‘polluted’ by the image center bias.
Our results surpass MIT, NASA, JPL, Stanford, INRIA, INO, Creaform, …

7. Camera Calibration
Lens distortion introduces the biggest
error found in digital imaging
Geometric distortion curves straight lines and
shears the squareness of rectangles with
increasing magnitude, going from the image
center towards the outside border.
Chromatic distortion splits white light as in a
prism blurring the rectangle edges in blue on
the short side, and in red on the far side with
respect to the image center.
Chromatic distortion introduces a systematic
edge measurement error with varying object
color or light spectrum changes.
Both distortions are seen by image analysis
algorithms as false image point motion while it
actually is an imperfection created by the
optics.
We like to use short focal length lenses to gain
image and surface definition, to see more
details. Up until our work, we were faced with
a trade off between image definition and
measurement accuracy because of lens
distortion. We are now able to compensate
through software.

8. Camera Calibration
How ? By taking the picture of a 3D known
structure, and modeling how the image
prints on the camera electronic surface.

9. Camera Calibration
We corrected everyone’s error in
locating the image center
The Image center is the center of symmetry
needed to model and correct lens distortion,
both geometric and chromatic…
Finding the true image center position corrects
a systematic bias in all other lens parameters
• the lens focal distance
• the lens skew compensation
• the lens geometric distortion
• the lens chromatic distortion
• the camera position w.r. to the outside world
and all 3D measurements taken from
camera images in telemetry…

10. Camera Calibration
3D from a stereo camera pair
All advanced imagery techniques require a
pinhole ‘errorless’ camera image, where
geometric and chromatic distortion are
removed, and without bias in the image center.
It enables sub pixel edge extraction and
increases 3D triangulation accuracy.

11. Camera Calibration
3D from a video sequence
3D from motion techniques behaves essentially like a
camera stereo pair. Two pictures of a still scenery taken
from two different vantage points will yield a 3D model.
Still, lens geometric and chromatic distortion have to be
removed from the images.
This is the technique used by imaging satellites, UAV
planes, rovers, panoramic cameras, … to achieve a 3D
map of a given area.
In satellite imaging, the earth’s atmosphere behaves like a
lens and adds distortion to the image. It can be
compensated for only if the camera distortion is accurately
known.

12. Camera Calibration
Earth Observation and Roving
A rover relies on its camera to map its Sub Pixel edge detection is biased
surroundings. In aerial photography, we by lens distortion.
can retrieve edge locations at a fraction of Geometric and chromatic
a pixel. In order to make the distortion correction require an
measurement significant, lens chromatic accurate knowledge of the image
and geometric distortion have to be center.
removed from the image. We removed a 2 pixel image
center bias in image calibration
using COTS cameras and lenses

13. Camera Calibration
Image fusion and Night Vision
Seeing through fog, smoke, snow, camouflage
The US Army is spending billions to gain a tactical
advantage in night operations.
They are seeking the next generation of wide angle imaging
to replace the current NVG light amplification devices.
Our proposed modification of SWIR (Short Wave Infra Red)
cameras to DND Canada provides that kind of performance,
and will cut down their supplying cost by 125 to 150$
millions with a smaller/lighter and more accurate night
vision camera.
Our initial involvement in this venture was to show DND
Canada how camera calibration would solve the image
fusion problem between SWIR and color images needed for
Enhanced Night Vision Goggles (ENVG)
We then proceeded to increase the camera sensitivity
The same technology goes for ground or airborne
surveillance cameras, soldier helmet, and satellite imaging.

14. Camera Calibration
Landing an Airplane
Civil Aviation produces more CO2 emissions than the entire
African continent.
At 300,000 $ a piece, with a 2 year payback in fuel economy
alone, a cryogenic infrared setup helps to locate the runway
in low visibility conditions.
The next generation of supersonic airplanes will rely on
cameras so no cockpit or hull windows are needed, making
the plane lighter.
Our proposal to DND is an augmented reality helmet using
the human eye as a pointing device to replace the mouse
click, a device that doesn’t require cryogenic cooling.
Synthetic vision with
Vision Amplification
should appear in civilian
airline transportation
around year 2018

15. Camera Calibration
Medical Imaging
Historically, imaging technology
developed for military uses has seen its
way to medical imaging.
The use of robotic surgery shortens the
patient’s recovery period in hospital by
more than half. Less invasive, it relies on
cameras to provide access to the
operating area.
The US Army is developing a robot
surgeon for NASA’s space program.
Removing lens distortion makes the use
of cameras more intuitive and
comfortable for the surgeon, increasing
his skill.
Used with a robot, lens distortion
removal makes measurements 8 times
more accurate.

16. Camera Calibration
Telecommunications
We demonstrated an added 30% lossless
compression over existing codec.
Removing lens distortion allowed us to
use geometric redundancy to increase the
compression ratio of images and video
sequences. Video content is therefore
faster to transmit.
Internet saturates because of the
increased use of digital video. The
hardware cost for North America is
estimated at 55 $billions.
In satellite applications, our accuracy
gain impacts two ways:
• Smaller image files because of
added compression
• Less images because larger
areas can be scanned
In wireless transmission, added
compression means more devices
serviced using the same network.
Since 75% of all Internet data content is
video, a given telco operator could be
more profitable than the owner of the
network it is roaming on.

17. Camera Calibration
Other market targets include…
•Pharmaceutical packaging
•Robotics parcel handling
•Highway traffic surveillance
•Airport runway traffic management
•Border surveillance control
…
As part of the Optics and Photonics market segment, computer
vision and digital imaging will impact on most of humanity’s
economic activities, part of a technological break trough as
significant as the advent of the microchip and the electron in
the 20th century (according to ISO’s market study).
The use of zooming lenses cameras for telemetry was thought
to be the only major technological barrier for the explosion of
this specific market niche. We showed quite a few more…