http://www.nshorter.com 1

Autonomous 3D Reconstruction From Irregular LiDAR and Aerial Imagery

Nicholas ShorterMonday, May 19, 2007

Beginner’s Guide to LiDAR Research

Email: nshorter@cfl.rr.com

Website: http://www.nshorter.com

http://www.nshorter.com 2

Presentation Objectives

Help enable fellow researchers to be able to apply their image processing and machine learning algorithms to LiDAR challenges by providing the following:

• Reviewing key conference and journal LiDAR related papers

• Explain physical setup and noise sources associated with LiDAR procurement

• Reviewing previous works

http://www.nshorter.com 3

Presentation Layout

1. Overview

2. LiDAR Capture Characteristics

3. Applications

4. LiDAR Noise

5. ‘Solved’ LiDAR Problems

6. Current LiDAR Challenges

7. Existing Methodology

8. Training/Clustering Features

http://www.nshorter.com 4

LIDAR Overview• Data Collection

– Plane Equipped with GPS, INS & LIDAR – LIDAR – Light Detection and Ranging (active sensor)

• LiDAR sensor works day or night, cloud coverage or not• Collection of 3D points• Laser sent out from Emitter, reflects off of Terrain, Returns to Receiver [1],

[2]– Receiver measures back scattered electromagnetic radiation (laser intensity)– In [1], Yu Sun et. al. propose using LS-SVM for denoising LiDAR

» sufficiently explains how LiDAR works• Time Difference Determines Range to Target

http://www.toposys.com/

http://www.nshorter.com 5

Sampling the following scene…

http://www.nshorter.com 6

Raw Elevation Plot

http://www.nshorter.com 7

Angled, Zoomed In View

http://www.nshorter.com 8

Triangulated, Angled, Zoomed In View

http://www.nshorter.com 9

LIDAR Captured Characteristics

• Range to Target (elevation from INS & LiDAR)

• Longitude and Latitude (GPS)• First and Last Return Pulses

– First – shrubbery, vegetation, power lines, birds and buildings

– Last – buildings (unless vegetation is really dense, then vegetation too)

• Returned Laser Intensity

http://www.nshorter.com 10

First and Last Return Pulse Difference

http://www.nshorter.com 11

LiDAR Returned Intensity Plot

http://www.nshorter.com 12

LiDAR Characteristics Continued

• Typically stored as an ASCII File

• The (x,y,z) coordinates are irregularly distributed – “swarm of angry bees”

http://www.nshorter.com 13

Applications for 3D Reconstruction

• Military Applications– Automatic Target Recognition [3]– Reconstructed Models of Opponent Terrain

(UAV) [5]

• Tourism/Entertainment– Virtual Walkthrough of Theme Park

• Commercial– Change Detection (Natural Disasters) [4]– Network Planning for Mobile Communication– Noise Nuisance (Universal Studios, 408

Expressway - walls)– Urban Planning

http://www.nshorter.com 14

LiDAR Noise

• Geo-location results from LiDAR, GPS and INS sensor systems– Accuracy Limitations [1]– Offset and Drift in both GPS and INS [13]– Misalignment between INS and LiDAR [13]

• Atmosphere – Intensity and Path Distortion• Shadowing Effect from Tall buildings• Artifacts from non uniform sampling from

multiple strips

http://www.nshorter.com 15

‘Solved’ LiDAR Challenges

• LiDAR Triangulation [6] – Wang et. al. in [6] benchmark different triangulation

methods on LiDAR

• LiDAR interpolation to fixed point spacings [7]– Goncalves in [7] compares several interpolation

methods and the errors they produce for LiDAR rasterizing

• DTM and DSM generation [25]– Commercial packages/companies provide DTM and

DSM generation from raw LiDAR

http://www.nshorter.com 16

Current LiDAR Challenges• Building Detection [8], [9], [11], [17], [26]

– In [8], Dr. Zhang overcomes Morphological Filtering Windowing Problem by using several window sizes (LiDAR only)

– [9] unfortunately makes very limiting assumptions (single aerial image only)• Parallel building sides, rectilinear buildings only, only considers aerial imagery

– [17] assumes roof top colors are within red spectrum of RGB and takes 45 to 70 minutes on 6400x6400 pixel image (single aerial image only)

• Several buildings in Fairfield dataset have white and grey roof tops– [26] uses thresholding between DTM and DSM

• No ideal window size for entire data set for Morphological Filtering (as shown in [8])

• Building Reconstruction [14], [15], [16], [18]– [14] reports trouble with small houses and building roof tops not being simple

Gable or Hip– [15], [16] Authors assume roofs on all buildings are flat– [18] Some stages during reconstruction process rely on human intervention

http://www.nshorter.com 17

Current LiDAR Challenges

• Multiple Source Registration [10]– In [10] Zitova and Flusser refer to over 200 sources

when reviewing current Image Registration methods• Note [10] covers not only image, but registration between

different other sources as well

– Most have trouble automating the relation of control points from the source image to the reference image

• LiDAR Strip Registration [12]• LiDAR Noise Removal [1], [2], [13] • Detecting Structure under Vegetation Canopy

http://www.nshorter.com 18

Data Sources• All Sources Typically procured by Commercial Geospatial

Solution Company and/or Funded Academic Researchers– More commonly researchers receive commercially donated data

(instead of procuring themselves)• LIDAR (both air and ground)

– Overhead – readily available if not procurable– Ground – rare

• Aerial Imagery (single, stereo pair, video sequences)– Single Aerial Image – readily available if not procurable– Stereo Pair – most existing data sets only have single nadir image

but stereo pair procurable– Video Sequence – rare but procurable

• GIS Ground Plans, Architectural Plans, GIS Models– All rare and very difficult to get updated

• Ground Truth (extremely rare)

http://www.nshorter.com 19

LiDAR – What’s the Big Deal?• Literally hundreds of commercial companies providing

several Geo-spatial solutions developed for previously listed applications

• Lockheed Martin [22], Harris Corporation [23], and Boeing [24] have all (within the last 5 years) been involved in a LiDAR related project

• Literally hundreds of publications (many of which are funded), from as early as 90s, being made about LiDAR

• Most of the following companies use LiDAR technology for their offered Geo-spatial services

• This is an actively pursued topic by both academia and industry!

http://www.nshorter.com 20

Commercial Geospatial Solutions3D Laser Mapping http://www.3dlasermapping.com/

AAMHatch http://www.aamhatch.com.au/index.cfm

Advanced LiDAR Technology http://www.advlidar.com/

Aerial Cartographics of America, Inc. http://www.aca-net.com/

Aerial Data Service http://www.aerialdata.com/

Aerial Services, Inc. http://www.aerialservicesinc.com/index.html

Aerials, Inc. http://www.floridaphoto.com/index.html

Aerodata International Surveys http://www.aerodata-surveys.com/

Aero-Metric http://www.aerometric.com/

Airborne 1 Corporation http://www.airborne1.com/#

Airborne Sensing Corporation http://www.airsensing.com/

BKS Surveys http://www.bks.co.uk/

COWI http://www.cowi.com/cowi/en/menu/home

Digital Mapping, Inc. http://www.admap.com/

Digital Mapping Services http://www.dmslp.com/

http://www.nshorter.com 21

Commercial Geospatial SolutionsEarthData http://www.earthdata.com/

GeoAnalytic Inc. http://www.geoanalytic.com/

Geofiny Technologies http://www.geofiny.com/

GRW Aerial Surveys http://www.grwinc.com/

Helica http://www.helica.it/

Horizons http://www.horizonsinc.com/

InfoTerra http://www.infoterra-global.com/

Intermap http://www.intermap.com/index.cfm

John Chance Land Surveys http://www.flimap.com/site1.php

Kucera International http://www.kucera-gis.com/

Laser Map Image Plus http://www.lasermap.com/laserM/home_en.htm

Laser Mapping Specialists http://www.lasermaps.com/default.asp

LIDAR Services International http://www.lidarservices.ca/index.html

M7 Visual Intelligence LP http://www.visidata.com/HOME.htm

Merrick & Company http://www.merrick.com/servicelines/gis/

http://www.nshorter.com 22

Commercial Geospatial SolutionsMD Atlantic Technologies http://www.atlantictech.com/

North West Geomatics http://www.nwgeo.com/

Precision Terrain Surveys LTD http://www.btinternet.com/~r_chiles/

Sanborn http://www.sanborn.com/

Simmons AeroFilms http://www.simmonsaerofilms.com/

Spencer B. Gross http://www.sbgmaps.com/

Survey Inspection System http://www.survey-inspection.com/

SwissPhoto http://www.swissphoto.ch/index_e.html

TerraImaging http://www.terraimaging.de/

TerraPoint USA http://www.terrapoint.com/

TopEye http://www.topeye.com/

Topographic Imaging http://www.lidarmapping.com/

Tuck Mapping Solutions http://www.tuckengineering.com/

VeriMap http://verimap.com/

Woolpert http://www.woolpert.com/

http://www.nshorter.com 23

Existing 3D Reconstruction Research

• Model Based Reconstruction [19], [20], [21]– Pre-defined models with parameters– Minimize error between models and data– [21] Assumes flat, gable or hip roof– [19] relies on user intervention

• Data Driven– Group Coplanar Pts

• Clustering, planar equation thresholds, etc.

– Identify Break Lines• planar intersections

– Derive Model to Minimize Error

http://www.nshorter.com 24

Other Data Driven Approaches

• In [27] Chen et. al use Patented Split Shape Merge (SMS) algorithm

• In [28] Overby et. al. use 3D Hough to extract planes and then use geometric constraints to refine and voting to accept/reject

http://www.nshorter.com 25

Triangulation Based Methods

• Triangular Irregular Network (TIN)– Series of Non-Overlapping Triangles Modeling

given Surface

• TIN 3D Reconstruction Methods– Clustering approach [29]

• Spherical Normal Vectors of Triangles

– TIN region growing approach [27], [30]• Merge Triangles to Same Region if Normal Vectors

within Threshold

http://www.nshorter.com 26

Building Features• Vegetation Points

– Significant first and last return elevation diff.– Corresponding (mostly) green aerial image color– Nearest points have diff. elev. or adjacent triangles have

significantly diff. norm vectors (jagged canopy)– Nonlinear/jagged exterior boundaries

• Common Building Points– Spatially close in terms of long. and lat.– Bounded by aerial img. edges and exterior wall tri.– Bounded building does not contain terrain pts – or - – Triangulation of all points, building points immediately connected– Relatively (mostly) smooth canopy

http://www.nshorter.com 27

Building Reconstruct Cluster Features

• Surface change - normal vector orientation difference between adjacent triangles

• (X,Y,Z) - Longitude, Latitude, Elevation• Edge - Aerial imagery edge detection• Color - Aerial imagery corresponding color

(building surface differs from clutter)• Triangle planar coef, pt. height diff., same normal

vector, or planar equation• Feedback – difference between reconstruction and

raw LiDAR and Aerial Image Edges

http://www.nshorter.com 28

Acknowledgements• Data Contributors

– Dr. Simone Clode, Dr. Franz Rottensteiner, AAMHatch– Mr. John Ellis, AeroMap– Mr. Steffen Firchau, TopoSys– Mr. Paul Mrstik, Terra Point

• Funding Contributors– Harris Corporation

• Advisor (Committee Chair)– Dr. Takis Kasparis

• Committee Members– Dr. Michael Georgiopoulos , Dr. Georgios Anagnostopoulos, Dr.

Andy Lee, Dr. Abhijit Mahalanobis, and Dr. Wasfy Mikhael

http://www.nshorter.com 29

References• [1] Sun, Bing Yu; Huang, De-Shang; Fang, Hai-Tao; “LiDAR Signal Denoising Using Least Square Support Vector Machine.” IEEE Signal

Processing Letters, Vol. 12, No. 2, February 2005• [2] Yu, Shirong; Wang, Weiran; “LiDAR Signal Denoising Based on Wavelet Domain Spatial Filtering.” International Conference on Radar,

October 2006• [3] Mahalanobis, Abhijit; “Multidimensional Algorithms for Target Detection in LiDAR Imagery.” University of Central Florida, Electrical and

Computer Engineering Seminar Series. Orlando. 28 March 2007• [4] Vu, Tuong Thuy; Matsoka, Matashi; Yamazaki, Fumio; “LiDAR based Change Detection of Buildings in Dense Urban Areas.” Geoscience

and Remote Sensing Symposium, 2004. IGARSS '04. Proceedings. 2004 IEEE International• [5] Lammons, George; “After The Storm.” Military Geospatial Technology, Volume 4, Issue 1, March 2006.• [6] Wang, Kai; Lo, Chor Pang; Brook, George; Arabnia, Hamid; “Comparison of existing triangulation methods for regularly and irregularly

spaced height fields.” INT. J. Geographical Information Science, vol. 15, no. 8, pp 743-762, 2001• [7] Goncalves, Gil; "Analysis of Interpolation Errors in Urban Digital Surface Models Created from LiDAR Data." 7th International Symposium

on Spatial Accuracy Assessment in Natural Resources and Environmental Sciences, 2006, pp 160-168

• [8] Zhang, Keqi; Chen, Shu-Ching; Whitman, Dean; Shyu, Mei-Ling; Yan, Jianhua, Zhang, Chengcui; "A Progressive Morphological Filter for Removing Nonground Measurements from Airborne LIDAR Data." IEEE Transactions on Geoscience and Remote Sensing, Vol. 41, No. 4, April 2003.

• [9] Zitova, Barbara; Flusser, Jan; “Image Registration Methods: A Survey”; Image and Vision Computing, Vol 21, 2003, pp 977-1000

• [10] Katartzis, Antonis; and Sahli, Hichem; “A Stochastic Framework for the Identification of Building Rooftops Using a Single Remote Sensing Image.” IEEE Transactions on Geoscience and Remote Sensing, vol. 46, no. 1, January 2008

• [11] Rottensteiner, Franz; Trinder, John; Clode, Simon; Kubik, Kurt; “Building Detection Using LIDAR Data and Multispectral Images.” 2003

• [12] Vosselman, G.; Maas, H.-G.; "Adjustment and Filtering of Raw Laser Altimetry Data." Proceedings OEEPE Workshop on Airborne Laserscanning and Interferometric SAR for Detailed Digital Elevation Models, OEEPE Publication No. 40, on CD-ROM, 11 pages.

http://www.nshorter.com 30

References• [13] Filin, S. Elimination of systematic errors from airborne laser scanning data. Geoscience and Remote Sensing Symposium, 2005.

IGARSS apos;05. Proceedings. 2005 IEEE International. Volume 1, Issue , 25-29 July 2005 Page(s): 4 pp.• [14] Forlani, G.; Nardinocchi, C.; Zingaretti, P.; Scaioni, M. ; "Complete classification of raw LIDAR data and 3D reconstruction of

buildings." Pattern Analysis and Applications vol.8, no.4 (2006),p.357-74• [15] Xie, Minghong; Fu, Kun; Wu, Yirong; "Building Recognition and Reconstruction from Aerial Imagery and LIDAR Data." Radar,

2006. CIE '06. International Conference on, Oct. 2006, pp. 1-4.• [16] Chen, Liang-Chien; Teo, Tee-Ann; Shao, Yi-Chen; Lai, Yen-Chung; and Rau, Jiann-Yeou; “Fusion of LIDAR Data and Optical

Imagery for Building Modeling.” ISPRS XXth Congress - Comission 4, 2004, p732• [17] Muller, Sonke; Zaum, Daniel; "Robust Building Detection in Aerial Images." IAPRS, Vol. XXXVI, Part3/W24, Vienna, Austria,

August 29-30, 2005• [18] Vosselman, G.; Gorte, B.G.H.; Sithole, G.; Rabbani, T.; "Recognising structure in laser scanner point clouds." International Archives of

Photogrammetry, Remote Sensing and Spatial Information Sciences, vol. 46, part 8/W2, Freiburg, Germany, October 4-6, (2004) pp. 33-38.• [19] Hu, Jinhui; You, Suya; Neumann, Ulrich; Park, Kyung Kook; "Building Modeling From LiDAR and Aerial Imagery." Proceedings of

ASPRS, May 2004.• [20] Mass, H; and Vosselman, G. “Two Algorithms for Extracting Building Models from Raw Lser Altimetry Data.” ISPRS Journal of

Photogrammetry & Remote Sensing, vol. 54, pp. 153-155, 1999• [21] Suveg, Ildiko; and Vosselman, George. “Automatic 3D Building Reconstruction.” Proceedings of SPIE, vol. 4661, pp. 59 – 69, 2002• [22] http://www.laserfocusworld.com/display_article/309074/12/none/none/INDUS/Lockheed-Martin-to-combine-electro-optics,-LIDAR-

for-urban-environment-surveillanc • [23] http://www.harris.com/view_pressrelease.asp?act=lookup&pr_id=2001 • [24] http://www.boeing.com/news/releases/2004/q4/nr_041203m.html

• [13] Filin, S. Elimination of systematic errors from airborne laser scanning data. Geoscience and Remote Sensing Symposium, 2005. IGARSS apos;05. Proceedings. 2005 IEEE International. Volume 1, Issue , 25-29 July 2005 Page(s): 4 pp.

• [14] Forlani, G.; Nardinocchi, C.; Zingaretti, P.; Scaioni, M. ; "Complete classification of raw LIDAR data and 3D reconstruction of buildings." Pattern Analysis and Applications vol.8, no.4 (2006),p.357-74

• [15] Xie, Minghong; Fu, Kun; Wu, Yirong; "Building Recognition and Reconstruction from Aerial Imagery and LIDAR Data." Radar, 2006. CIE '06. International Conference on, Oct. 2006, pp. 1-4.

• [16] Chen, Liang-Chien; Teo, Tee-Ann; Shao, Yi-Chen; Lai, Yen-Chung; and Rau, Jiann-Yeou; “Fusion of LIDAR Data and Optical Imagery for Building Modeling.” ISPRS XXth Congress - Comission 4, 2004, p732

• [17] Muller, Sonke; Zaum, Daniel; "Robust Building Detection in Aerial Images." IAPRS, Vol. XXXVI, Part3/W24, Vienna, Austria, August 29-30, 2005

• [18] Vosselman, G.; Gorte, B.G.H.; Sithole, G.; Rabbani, T.; "Recognising structure in laser scanner point clouds." International Archives of Photogrammetry, Remote Sensing and Spatial Information Sciences, vol. 46, part 8/W2, Freiburg, Germany, October 4-6, (2004) pp. 33-38.

• [19] Hu, Jinhui; You, Suya; Neumann, Ulrich; Park, Kyung Kook; "Building Modeling From LiDAR and Aerial Imagery." Proceedings of ASPRS, May 2004.

• [20] Mass, H; and Vosselman, G. “Two Algorithms for Extracting Building Models from Raw Lser Altimetry Data.” ISPRS Journal of Photogrammetry & Remote Sensing, vol. 54, pp. 153-155, 1999

• [21] Suveg, Ildiko; and Vosselman, George. “Automatic 3D Building Reconstruction.” Proceedings of SPIE, vol. 4661, pp. 59 – 69, 2002• [22] http://www.laserfocusworld.com/display_article/309074/12/none/none/INDUS/Lockheed-Martin-to-combine-electro-optics,-LIDAR-

for-urban-environment-surveillanc • [23] http://www.harris.com/view_pressrelease.asp?act=lookup&pr_id=2001 • [24] http://www.boeing.com/news/releases/2004/q4/nr_041203m.html • [25] Arefi, H.; Hahn, M.; “A Morphological Reconstruction Algorithm For Separating Off-Terrain Points from Terrain Points in Laser

Scanning Data.” ISPRS Workshop Laser Scanning, 2005

http://www.nshorter.com 31

References• [26] Rottensteiner, F.; and Briese, Ch.; “A New Method for Building Extraction in Urban Areas from High-Resolution LIDAR Data.”

IAPRSIS, vol. 34/3A, pp. 295-301, 2002• [27] Chen, Liang Chien; Teo, Tee-Ann; Shao, Yi-Chen; Lai, Yen-Chung; Rau, Jiann-Yeo; “Fusion of LIDAR Data and Optical Imagery for

Building Modeling.” International Archives of Photogrammetry and Remote Sensing, vol. 35, no. B4, pp. 732-737, 2004• [28] Overby, Jens; Bodum, Lars; Kjems, Erik; and Ilsoe, Pee M; “Automatic 3D Building Reconstruction from Airborne Laser Scanning

and Cadastral Data Using Hough Transform.” Geo-Imagery Bridging Continents 20th ISPRS Congress, 2004• [29] Hofmann, A.D.; “Analysis of TIN-Structure Parameter Spaces In Airborne Laser Scanning Data for 3-D Building Model Generation.”

Geo-Imagery Bridging Continents XXth ISPRS Congress, 2004• [30] Morgan, Michel; Habib, Ayman; “Interpolation of LIDAR Data and Automatic Building Extraction.” ACSM-ASPRS2002 Annual

Conference Proceedings, 2002