image processing and cartography with the nasa vision workbench
TRANSCRIPT
Intelligent Systems Division NASA Ames Research Center
Image Processing and Cartography with the NASA
Vision Workbench
Matthew D. HancherIntelligent Systems Division
NASA Ames Research CenterSeptember 26, 2007
Intelligent Systems Division NASA Ames Research Center
Talk Overview
• Who We Are
• Introduction to the Vision Workbench
• Example Applications
• FOSS and NASA
Intelligent Systems Division NASA Ames Research Center
NASA Ames Research Center
• NASA’s Silicon Valley research center
• Small spacecraft
• Supercomputers
• Intelligent Systems
• Human Factors
• Thermal protection systems
• Aeronautics
• Astrobiology
Intelligent Systems Division NASA Ames Research Center
GIS & Imaging at Ames
NASA World Wind MASTER(MODIS/ASTER simulator)
NASA/GooglePlanetary Content
Western StatesFire Monitoring Mission
Intelligent Systems Division NASA Ames Research Center
IRG & ACES
Intelligent RoboticsGroup
Adaptive Control &Evolvable Systems Group
Intelligent Systems Division NASA Ames Research Center
Intro to the Vision Workbench
Intelligent Systems Division NASA Ames Research Center
NASA Vision Workbench
• Open-source image processing and machine vision library in C++
• Developed as a foundation for unifying raster image processing work at NASA Ames
• A “second-generation” C++ image processing library, drawing on lessons learned by VXL, GIL, VIGRA, etc.
• Designed for easy, expressive coding of efficient image processing algorithms
Intelligent Systems Division NASA Ames Research Center
Open-Source VW Modules
• Core: Low-level types & platform support
• Math: General-purpose mathematical tools
• Image: Basic image operations, filters, etc.
• FileIO: Simple, flexible image file IO layer
• Camera: Camera models & related tools
• Cartography: Geospatial image manipulation
• Mosaic: Image mosaicing & multi-band blending
• HDR: High-dynamic-range imaging
(Open source as of now)
VW “Foundation”
Modules
Intelligent Systems Division NASA Ames Research Center
VW Modules Underway
• InterestPoint: Interest point detection & matching
• Stereo: Stereo correlation & 3D reconstruction
• Python: Python bindings to many VW capabilities
• GPU: GPU-accelerated image operations
• Texture: Texture analysis & matching
• Display: Image display and user interaction
(The first four to be released later this year)
Intelligent Systems Division NASA Ames Research Center
Design Goals & Approach
• A simple, clean API for easy hacking
• Simple syntax: Write what you mean!
• Easy to manipulate arbitrarily large images
• Automatic memory management
• Generates high-performance code
• Optimized processing via lazy evaluation
• Function inlining via “generic” (template-based) C++ style
Intelligent Systems Division NASA Ames Research Center
API Philosophy
• Simple, natural, mathematical, expressive
• Treat images as first-class mathematical data types whenever possible
• Example: IIR filtering for background subtraction
background += alpha * ( image - background );
• Direct, intuitive function calls
• Example: A Gaussian smoothing filter
result = gaussian_filter( image, 3.0 );
Intelligent Systems Division NASA Ames Research Center
Image Module Basics
Intelligent Systems Division NASA Ames Research Center
Under the Hood: Image Views
• The core “image view” concept:• Can be evaluated at a location to return a pixel value
• Has a width and height in pixels
• Cannonical example: the ImageView class•ImageView<PixelRGB<uint8> > image(1024,768);
• Data processing represented as views
•image2 = gaussian_filter(image1, 3.0);
• Lazy container for arbitrary views
•ImageViewRef<PixelRGB<uint8> > image3 = gaussian_filter(image1, 3.0);
Intelligent Systems Division NASA Ames Research Center
Image Views II
• Eliminates unnecessary temporaries• background += alpha * ( image - background );
• Supports procedurally generated images•image2 = fixed_grid(10,10,white,black,1024,768);
• Allows greater control over processing•image2 = block_rasterize( gaussian_filter(image1, 3.0) );
• Views of images on disk
•DiskImageView<PixelRGB<uint8> > disk_image(filename);
Intelligent Systems Division NASA Ames Research Center
Applications & Modules
Intelligent Systems Division NASA Ames Research Center
GigaPan Panorama Stitcher
(As featured in the GigaPan layer in Google Earth.)
Intelligent Systems Division NASA Ames Research Center
Mosaic Module
• ImageComposite
• Composite an arbitrary number of arbitrarily large images
• It’s “just another image view”
• Supports multi-band blending for seamless composites
• QuadTreeGenerator
• Generates a tiled pyramid representation of an arbitrary image view on disk
• Great for building e.g. KML superoverlays or TMS maps
Intelligent Systems Division NASA Ames Research Center
Cartographic Reprojection
(As seen in the newly updated Google Moon.)
Intelligent Systems Division NASA Ames Research Center
Cartography Module
• GeoReference• Uses PROJ.4 for standard projections, GDAL to read/write
• GeoTransform• Reprojects image data between GeoReferences
• Makes “just another image view”
• OrthoImageView• Ortho-rectifies an aerial or satellite image against an
arbitrary DEM (in conjunction with the Camera module).
• Also “just another image view”
Intelligent Systems Division NASA Ames Research Center
Automated Image Alignment
• Problem: Given two images, find and align the overlap region.
Intelligent Systems Division NASA Ames Research Center
Image Alignment w/ Interest Points
Images to be alignedLocate interest points in first imageLocate interest points in second imagePoint correspondences determine image alignment
Intelligent Systems Division NASA Ames Research Center
Interest Point Module
• Interest point detectors, descriptors, and matching
ScaledInterestPointDetector<LoGInterest> detector;InterestPointList ip1 = interest_points( image1, detector );InterestPointList ip2 = interest_points( image2, detector );
PatchDescriptor descriptor;compute_descriptors( image1, ip1, descriptor );compute_descriptors( image2, ip2, descriptor );
DefaultMatcher matcher(threshold);InterestPointList matched1, matched2;matcher.match( ip1, ip2, matched1, matched2 );
Matrix2x2 homography = ransac( matched1, matched2, SimilarityFittingFunctor(), InterestPointErrorMetric() );
Intelligent Systems Division NASA Ames Research Center
The Ames Stereo PipelineFast, high quality, automated stereogrammetric surface reconstruction originally developed for
Mars Pathfinder science operations
Disparity
Now a Vision Workbench application.
Intelligent Systems Division NASA Ames Research Center
Primary Image Secondary Image
RegistrationEphemeris or
Automated Interest Points
Fast Stereo Correlation
Outlier Rejection / Hole Filling / Smoothing
Disparity Map
Camera Model (e.g. Linear Pushbroom)
Mesh Generation
Point Cloud/DTM
3D Mesh
Mask / Sign of Laplacian of Gaussian
The Ames Stereo Pipeline
Surprise: It’s all just Vision Workbench image views!
Intelligent Systems Division NASA Ames Research Center
Mars Stereo: MOC NA
MGS MOC-Narrow Angle• Malin Space Science Systems• Altitude: 388.4 km (typical) • Line Scan Camera: 2048 pixels• Focal length: 3.437m• Resolution: 1.5-12m / pixel• FOV: 0.5 deg
Intelligent Systems Division NASA Ames Research Center
NE Terra Meridiani
Upper Left: This DTM was generated from MOC images E04-01109 and M20-01357 (2.38°N, 6.40°E). The contour lines (20m spacing) overlay an ortho-image generated from the 3D terrain model. Lower Right: An oblique view of the corresponding VRML model.
!!"""#$
!!"""$
!%""#$
!%""#$
!%""#$
Intelligent Systems Division NASA Ames Research Center
Preliminary MOLA Comparison
Scanline Capture Time (s)
Elev
atio
n at
bor
esig
ht p
ixel
(m
)
Intelligent Systems Division NASA Ames Research Center
Lunar Stereo: Apollo Orbiter Cameras
ITEK Panoramic Camera• Focal length: 610 mm (24”)• Optical bar camera• Apollo 15,16,17 Scientific
Instrument Module (SIM)• Film image: 1.149 x 0.1149 m• Resolution: 108-135 lines/mm
Intelligent Systems Division NASA Ames Research Center
Apollo 17 Landing Site
Top: Stereo reconstruction
Right: Handheld photo taken by an orbiting Apollo 17 astronaut
Intelligent Systems Division NASA Ames Research Center
Public Outreach: Hayden Planetarium
Intelligent Systems Division NASA Ames Research Center
Public Outreach: Hayden Planetarium
Intelligent Systems Division NASA Ames Research Center
Application: Image Matching
• Problem: Given an image, find others like it.
Example database: Apollo Metric Camera images
Intelligent Systems Division NASA Ames Research Center
Texture-Based Image Matching
Texture bank filtering (Gaussian 1st derivative and LOG)
Grouping to remove orientationEnergy in a window
E-M Gaussian mixture modelIterative tryouts, MDL
Max vote
GroupingMean energy in segment
Euclidian distance
Summarization
Post-processing
Output Representation
Filtering
Model image
Segmentation
Vector Comparison
Matched image
Intelligent Systems Division NASA Ames Research Center
Image Matching: Results
Intelligent Systems Division NASA Ames Research Center
FOSS and NASA
Intelligent Systems Division NASA Ames Research Center
The NOSA
• The NASA Open Source Agreement, an OSI-approved non-viral open source license
• Intended to protect users from contributor patent licensing issues.
• Yes, we know: The current version (1.3) has several well-known peculiarities.
Intelligent Systems Division NASA Ames Research Center
U.S. Contractor Rights
• The University and Small Business Patent Procedures Act of 1980, a.k.a. “Bayh-Dole”.
• A university, small business, or non-profit can claim patent ownership of a federally-funded invention before the government.
• The government must actively promote and attempt to commercialize the invention.
• Severely complicates open-source initiatives within the government that involve universities, small businesses, or non-profits.
Intelligent Systems Division NASA Ames Research Center
The Open Source Process
• Open-source approval stages include:
• Invention disclosure
• Copyright assignment (all parties)
• Legal review (copyright & patent issues)
• Export control review (e.g. ITAR)
• Computer security review
• more....
Intelligent Systems Division NASA Ames Research Center
Signs of Improvement
• The old model: (e.g. VW 1.0)• Seek approvals after code completion
• Long, slow, high-latency release cycle
• The new model: ?? (e.g. WV 2.0 ??)• Seek periodic approval for upcoming development
• Allows regular updates within prescribed bounds
• On the horizon: ??• User contribution process ?
• Publicly-accessible subversion repository ???
Intelligent Systems Division NASA Ames Research Center
Free and Open Data
• Free and open data has received much less attention than free and open software.
• The National Aeronautics & Space Act:
• The Administration, in order to carry out the purpose of this Act, shall... provide for the widest practicable and appropriate dissemination of information concerning its activities and the results thereof.
• Alas, NASA does not own much of what is often imagined to be “NASA data”.
Intelligent Systems Division NASA Ames Research Center
Outreach: Google Earth
MODIS CoveragesAstronaut Photography
• Make more datasets publicly available as KML (and soon WMS) for mash-ups.
• Increase the visibility of existing public repositories of NASA data and imagery.
Intelligent Systems Division NASA Ames Research Center
Outreach: Google Moon
Data coming soon via KML and WMS from NASA.
Intelligent Systems Division NASA Ames Research Center
Obtaining the Vision Workbench
• VW version 1.0.1 available now.
• VW version 2.0 coming this fall!
• To contact me:
http://ti.arc.nasa.gov/visionworkbench/