Image Processing and Cartography with the NASA Vision Workbench

Image Processing and
Cartography with the NASA
Vision Workbench

Matthew D. Hancher
Intelligent Systems Division
NASA Ames Research Center
September 26, 2007
Intelligent Systems Division NASA Ames Research Center

Talk Overview

• Who We Are

• Introduction to the Vision Workbench

• Example Applications

• FOSS and NASA


NASA Ames Research Center

• NASA’s Silicon Valley
research center
• Small spacecraft
• Supercomputers
• Intelligent Systems
• Human Factors
• Thermal protection systems
• Aeronautics
• Astrobiology


GIS & Imaging at Ames

MASTER
NASA World Wind
(MODIS/ASTER simulator)

NASA/Google Western States
Planetary Content Fire Monitoring Mission


IRG & ACES

Intelligent Robotics Adaptive Control &
Group Evolvable Systems Group


Intro to the Vision Workbench


NASA Vision Workbench

• Open-source image processing and machine vision
library in C++

• Developed as a foundation for unifying raster
image processing work at NASA Ames

• A “second-generation” C++ image processing
library, drawing on lessons learned by VXL, GIL,
VIGRA, etc.

• Designed for easy, expressive coding of
efﬁcient image processing algorithms


Open-Source VW Modules
• Core: Low-level types & platform support

• Math: General-purpose mathematical tools VW
“Foundation”
• Image: Basic image operations, filters, etc. Modules

• FileIO: Simple, flexible image file IO layer

• Camera: Camera models & related tools

• Cartography: Geospatial image manipulation

• Mosaic: Image mosaicing & multi-band blending

• HDR: High-dynamic-range imaging

(Open source as of now)

VW Modules Underway

• InterestPoint: Interest point detection & matching

• Stereo: Stereo correlation & 3D reconstruction

• Python: Python bindings to many VW capabilities

• GPU: GPU-accelerated image operations

• Texture: Texture analysis & matching

• Display: Image display and user interaction

(The ﬁrst four to be released later this year)

Design Goals & Approach

• A simple, clean API for easy hacking
• Simple syntax: Write what you mean!

• Easy to manipulate arbitrarily large images

• Automatic memory management

• Generates high-performance code
• Optimized processing via lazy evaluation

• Function inlining via “generic” (template-based) C++ style


API Philosophy

• Simple, natural, mathematical, expressive

• Treat images as first-class mathematical data
types whenever possible
• Example: IIR filtering for background subtraction
background += alpha * ( image - background );

• Direct, intuitive function calls
• Example: A Gaussian smoothing filter
result = gaussian_filter( image, 3.0 );


Image Module Basics


Under the Hood: Image Views
• The core “image view” concept:
• Can be evaluated at a location to return a pixel value
• Has a width and height in pixels

• Cannonical example: the ImageView class
• ImageView<PixelRGB<uint8> > image(1024,768);

• Data processing represented as views
• image2 = gaussian_filter(image1, 3.0);

• Lazy container for arbitrary views
• ImageViewRef<PixelRGB<uint8> > image3
= gaussian_filter(image1, 3.0);


Image Views II
• Eliminates unnecessary temporaries
• background += alpha * ( image - background );

• Supports procedurally generated images
• image2 = fixed_grid(10,10,white,black,1024,768);

• Allows greater control over processing
• image2 = block_rasterize( gaussian_filter(image1, 3.0) );

• Views of images on disk
• DiskImageView<PixelRGB<uint8> > disk_image(filename);


Applications & Modules


GigaPan Panorama Stitcher

(As featured in the GigaPan layer in Google Earth.)

Mosaic Module

• ImageComposite
• Composite an arbitrary number of arbitrarily large images

• It’s “just another image view”

• Supports multi-band blending for seamless composites

• QuadTreeGenerator
• Generates a tiled pyramid representation of an arbitrary
image view on disk

• Great for building e.g. KML superoverlays or TMS maps


Cartographic Reprojection

(As seen in the newly updated Google Moon.)

Cartography Module
• GeoReference
• Uses PROJ.4 for standard projections, GDAL to read/write

• GeoTransform
• Reprojects image data between GeoReferences
• Makes “just another image view”

• OrthoImageView
• Ortho-rectiﬁes an aerial or satellite image against an
arbitrary DEM (in conjunction with the Camera module).
• Also “just another image view”


Automated Image Alignment
• Problem: Given two images, ﬁnd and align the overlap region.


Image Alignment w/ Interest Points

Point correspondencesto be aligned image
Locate interest points inin ﬁrst image
Locate interest points second alignment
Images determine image


Interest Point Module
• Interest point detectors, descriptors, and matching
ScaledInterestPointDetector<LoGInterest> detector;
InterestPointList ip1 = interest_points( image1, detector );
InterestPointList ip2 = interest_points( image2, detector );

PatchDescriptor descriptor;
compute_descriptors( image1, ip1, descriptor );
compute_descriptors( image2, ip2, descriptor );

DefaultMatcher matcher(threshold);
InterestPointList matched1, matched2;
matcher.match( ip1, ip2, matched1, matched2 );

Matrix2x2 homography = ransac( matched1, matched2,
SimilarityFittingFunctor(),
InterestPointErrorMetric() );


The Ames Stereo Pipeline
Fast, high quality, automated stereogrammetric
surface reconstruction originally developed for
Mars Pathﬁnder science operations

Disparity

Now a Vision Workbench application.

The Ames Stereo Pipeline
Primary Image Secondary Image

Ephemeris or
Registration
Automated Interest
Points
Mask / Sign of Laplacian of Gaussian

Fast Stereo Correlation

Outlier Rejection / Hole Filling /
Smoothing
Disparity Map
Camera Model (e.g. Linear Pushbroom)
Point Cloud/DTM
Mesh Generation
3D Mesh

Surprise: It’s all just Vision Workbench image views!

Mars Stereo: MOC NA
MGS MOC-Narrow Angle
• Malin Space Science Systems
• Altitude: 388.4 km (typical)
• Line Scan Camera: 2048 pixels
• Focal length: 3.437m
• Resolution: 1.5-12m / pixel
• FOV: 0.5 deg


NE Terra Meridiani

!%
quot;quot;
#$
!!

$
#
quot;

quot;quot;
quot;quot;

!%
#$

!!quot;quot;quot; $
!%quot;quot;#$

Upper Left: This DTM was generated from MOC images E04-01109 and M20-01357 (2.38°N, 6.40°E). The contour lines (20m
spacing) overlay an ortho-image generated from the 3D terrain model. Lower Right: An oblique view of the corresponding VRML
model.


Preliminary MOLA Comparison
Elevation at boresight pixel (m)

Scanline Capture Time (s)


Lunar Stereo: Apollo Orbiter Cameras

ITEK Panoramic Camera
• Focal length: 610 mm (24”)
• Optical bar camera
• Apollo 15,16,17 Scientific
Instrument Module (SIM)
• Film image: 1.149 x 0.1149 m
• Resolution: 108-135 lines/mm


Apollo 17 Landing Site

Top: Stereo reconstruction

Right: Handheld photo taken by an
orbiting Apollo 17 astronaut


Public Outreach: Hayden Planetarium


Application: Image Matching
• Problem: Given an image, ﬁnd others like it.

Example database: Apollo Metric Camera images


Texture-Based Image Matching
Model
image

Texture bank filtering
Filtering
(Gaussian 1st derivative and LOG)

Grouping to remove orientation
Output Representation
Energy in a window

E-M Gaussian mixture model
Segmentation
Iterative tryouts, MDL

Max vote
Post-processing

Grouping
Summarization
Mean energy in segment

Euclidian distance
Vector Comparison

Matched
image


Image Matching: Results


FOSS and NASA


The NOSA
• The NASA Open Source Agreement, an OSI-approved
non-viral open source license

• Intended to protect users from contributor patent
licensing issues.

• Yes, we know: The current version (1.3) has several
well-known peculiarities.


U.S. Contractor Rights
• The University and Small Business Patent
Procedures Act of 1980, a.k.a. “Bayh-Dole”.
• A university, small business, or non-proﬁt can claim patent
ownership of a federally-funded invention before the government.

• The government must actively promote and attempt to
commercialize the invention.

• Severely complicates open-source
initiatives within the government that involve
universities, small businesses, or non-proﬁts.


The Open Source Process

• Open-source approval stages include:
• Invention disclosure

• Copyright assignment (all parties)

• Legal review (copyright & patent issues)

• Export control review (e.g. ITAR)

• Computer security review

• more....


Signs of Improvement
• The old model: (e.g.VW 1.0)
• Seek approvals after code completion
• Long, slow, high-latency release cycle

• The new model: ?? (e.g. WV 2.0 ??)
• Seek periodic approval for upcoming development
• Allows regular updates within prescribed bounds

• On the horizon: ??
• User contribution process ?
• Publicly-accessible subversion repository ???


Free and Open Data
• Free and open data has received much
less attention than free and open software.

• The National Aeronautics & Space Act:
• The Administration, in order to carry out the purpose of this
Act, shall... provide for the widest practicable and
appropriate dissemination of information concerning
its activities and the results thereof.

• Alas, NASA does not own much of what
is often imagined to be “NASA data”.


Outreach: Google Earth

Astronaut Photography MODIS Coverages

• Make more datasets publicly available as KML (and soon
WMS) for mash-ups.

• Increase the visibility of existing public repositories
of NASA data and imagery.


Outreach: Google Moon

Data coming soon via KML and WMS from NASA.


Obtaining the Vision Workbench

• VW version 1.0.1 available now.
• VW version 2.0 coming this fall!
http://ti.arc.nasa.gov/visionworkbench/

• To contact me:
Matthew.D.Hancher@nasa.gov


Image Processing and Cartography with the NASA Vision Workbench

Recommandé

Recommandé

Contenu connexe

En vedette

En vedette (8)

Similaire à Image Processing and Cartography with the NASA Vision Workbench

Similaire à Image Processing and Cartography with the NASA Vision Workbench (20)

Dernier

Dernier (20)

Image Processing and Cartography with the NASA Vision Workbench