stabilization and georegistration of aerial video over mountain terrain by means of lidar

Stabilization and Georegistration of Aerial

Video Over Mountain Terrain by Means of LIDAR

Mark Pritt, PhDLockheed Martin

Gaithersburg, Marylandmark.pritt@lmco.com

Kevin LaTouretteLockheed MartinGoodyear, Arizonakevin.j.latourette@lmco.com

IGARSS 2011, Vancouver, CanadaJuly 24-29, 2011

2

Problem: Georegistration

· Georegistration is the assignment of 3-D geographic coordinates to the pixels of an image.

· It is required for many geospatial applications: Fusion of imagery with other sensor data Alignment of imagery with GIS and map graphics Accurate 3-D geolocation

· Inaccurate georegistration can be a major problem:

Misaligned GIS

Correctly aligned

3

Solution

· Our solution is image registration to a high-resolution digital elevation model (DEM): A DEM post spacing of 1 or 2 meters yields good results. It also works with 10-meter post spacing.

· Works with terrain data derived from many sources: LIDAR: BuckEye, ALIRT, Commercial Stereo Photogrammetry: Socet Set® DSM SAR: Stereo and Interferometry USGS DEMs

4

· Create predicted images from the DEM, illumination conditions, sensor model estimates and actual images.

· Register the images while refining the sensor model.· Iterate.

Methods

Aerial Video Sensor

Image Plane

Scene

Occlusion

Illumination

ShadowPredicted

Images

5

Methods (cont)

Predicted Image

from DEM

Predicted Image from

Aerial Image

Registration Tie Point

Detections

The algorithm identifies tie points between the

predicted and the actual images by means of NCC

(normalized cross correlation) with RANSAC

outlier removal.

6

· The algorithm uses the refined sensor model as the initial guess for the next video frame:

· The refined sensor model enables georegistration. Exterior orientation: Platform position and rotation angles Interior orientation: Focal length, pixel aspect ratio, principal point

and radial distortion

Methods (cont)

Initial Camera

•Estimate camera model

•Use camera focal length & platform GPS if avail.

Register

•Predict images from DEM and camera

•Register images with NCC

Refine

•Compose registration fcn & camera

•LS fit for better cam estimate

• Iterate

Next Frame

•Register to previous frame

•Compose with cam of prev. frame for init. cam estimate

Iterate

• Iterate for each video frame

Finish

•Trajectory•Propagate geo data from DEM

•Resample images for orthomosaic

7

Example 1: Aerial Motion Imagery

Inputs:Aerial Motion Imagery over

Arizona, U.S.

16 Mpix, 3.3 fps, panchromatic

1/3 Arc-second USGS DEM

Area: 64 km2

Post Spacing: 10 m

8

Example 1 (cont)

Problem: Too shaky to find moving objects

Zoomed to full resolution (1 m)

9

· Outputs: Sensor camera models Images georegistered to DEM Platform trajectory

Example 1: Results

10

Example 1 Results (cont)

ATV Vehicle Human

Pickup Truck

Video is now stabilized, and as a

result, moving objects are easily

detected.

11

Example 2: Oblique Motion Imagery

Inputs:Oblique Motion Imagery Over

Arizona, U.S.

16 Mpix, 3.4 fps, pan

LIDAR DEM

Area: 24 km2

Post Spacing: 1 m

12

Example 2: Results

Map coordinates

Stabilized Video Inset

Orthorectified Video

Background LIDAR DEM Aligned

Map Graphics

Target Tracking

Aligned Map

Graphics

13

Example 2 Results (cont)

IMAGE 1 Camera Iteration1 2 3

Num tie points: 319 318 282

RMSE: 17.4 4.8 2.9Mean Δx: 1.4 -0.7 0.1Mean Δy: -3.8 -0.1 0Sigma Δx: 15.8 4 2.5Sigma Δy: 6 2.6 1.5

IMAGE 591 Camera Iteration1 2 3

Num tie points 681 687 681

RMSE 2.7 0.6 0.3Mean Δx 1 0 0Mean Δy 0.9 0 0Sigma Δx 2.1 0.5 0.3Sigma Δy 0.9 0.2 0.1

· How fast does the algorithm converge?

1 2 30

5

10

15

20

Tie Point Residuals

RMSEmeansigma

Camera IterationIm

age

Pixe

ls

1 2 30

0.5

1

1.5

2

2.5

3

Tie Point Residuals

RMSEmeansigma

Camera Iteration

Imag

e Pi

xels

The initial error is high, but it

decreases after only several iterations.

Subsequent frames have better initial

sensor model estimates and require only 2

iterations.

14

Example 3: Aerial Video

Inputs:Aerial Video Over

Arizona, U.S.

720 x 480 Color 30 fps

LIDAR DEM

Area: 24 km2

Post Spacing: 1 m

15

Example 3: Results

Map coordinates

Orthorectified Video

Background Image

Draped Over DEM

Aligned Map

Graphics

16

Example 3 Results (cont)

Map Graphics Stay Aligned with Features in Video

17

Example 4: Thermal Infrared Video

Inputs:MWIR Video Over White

Tank Mountains in Arizona

1 Mpix, 3.3 fps

Commercial LIDAR DEM

Post Spacing: 2 m

18

Example 4: Results

BackgroundLIDAR DEM

Video Mosaic

Inset: Original Video

with Map Graphics Overlay

Video Mosaic Georegistered and

Draped Over Mountains in Google Earth

19

Demo

Click picture to play video

20

Conclusion

· We have introduced a new method for aerial video georegistration and stabilization.

· It registers images to high-resolution DEMs by: Generating predicted images from the DEM and sensor model; Registering these predicted images to the actual images; Correcting the sensor model estimates with the registration results.

· Processing speed is 1 sec per 16-Mpix image on a PC.· Absolute geospatial accuracy is about 1-2 meters.

We are developing a rigorous error propagation model to quantify the accuracy.

· Applications: Video stabilization and mosacs Cross-sensor registration Alignment with GIS map graphics