european space agency - matching sparse networks of...

Group of Robotics and Cognitive Systems http://robotics.pme.duth.gr/

Democritus University of Thrace Department of Production and Management Engineering, Greece

European Space Agency, European Space Research and Technology Centre (ESTEC)

Network/Partnering Initiative (NPI)

Matching Sparse Networks of Semantic ROIs Among Rover and Orbital Imagery

Evangelos Boukas, Antonios Gasteratos, Gianfranco Visentin

http://robotics.pme.duth.gr/

Outline

• Motivation • ESA Network/Partnering Initiative (NPI) • Related Work • Global Localization System • Repeatability of Matching • Discussion - Conclusions

5/19/2015 Advanced Space Technologies in Robotics and Automation (ASTRA 2015) 2

Outline



Motivation

• Advanced space missions require increased autonomy. • Localization is a sine qua non for Space Exploratory Rovers. • Is relative localization enough?


Motivation

• Advanced space missions require increased autonomy. • Localization is a sine qua non for Space Exploratory Rovers. • Is relative localization enough? • Mars Sample Return (MSR) mission:

• Fetch sample within a very narrow time window. • Even more with multiple “drop-off” of samples as a possibility for

Mars2020.

• ESA NPI: “Methods to Refine the Self-Localization of Planetary Rovers Using Orbital Imaging”.


Outline



European Space Agency Network/Partnering Initiative (NPI) • European Space Agency co-funds and supports PhD and post-doc

research • Main objectives:

• Strengthen the links among ESA, Universities, Research institutes and Industry.

• Apply recent non-space technology to space programs.

• Provides: • Co-funding. • Access to ESTEC laboratories (6-12 months). • Technical support. • Networking via ESA links.


European Space Agency Network/Partnering Initiative (NPI) • Laboratory of Robotics and Automation, School of Engineering,

Democritus University of Thrace. • “Methods to Refine the Self-Localization of Planetary Rovers

Using Orbital Imaging”. • Improve the global localization of Space Rovers. • Urge from previous ESA activities on localization and rover

integration: • SPARTAN project – VO for Rovers on FPGA

• Real Data. • Multiple scenarios.


European Space Agency Network/Partnering Initiative (NPI) • Key features:

• Extraction of commonly observed Regions of Interest (ROIs) on both orbital and rover imagery

• Opportunistic approaches to refine the localization • Integration of multiple localization techniques:

• Locally on the rover • Globally on Georeferenced images


Outline



Related Work

• OSU Mapping and GIS Laboratory: • Global Localization and daily rover traverses for MER using DEM generated

from satellite imagery. • Tie points on rover-orbital network, Bundle adjustment to find position. • The tie points were selected manually.

• JPL (L. Mathies, et al., Isairas 1997, Tokyo): • Combining lander, rover and descent imagery to improve localization near the

rover landing site. • Skyline Based Approach (“VIPER”, Stein and Medioni, 1992, 1995):

• Extract skylines from rover • Create possible skylines on DEMs • Match the rover skylines to the orbital ones

• LIDAR based (Furgale et al., 2010) • Create peaks on LIDAR scans • Match from DEM peaks


Related Work

• Terrain Matching (Van Pham et al., ASTRA 2013): • Create local DEMs. • Match using Particle Filters to Global DEMs. (Also Hwangbo et al. (2009)).

• Boulder based VO (Lourakis and Hourdakis, ASTRA 2015): • Utilize boulders from robot stereo and orbital imagery to improve the

visual odometry.


Mars-like Real Dataset

• Atacama Seeker Dataset (Woods et al., Astra 2013) • Mars like scenery:

• Large Rocks. • Outcrops (mostly salars). • Ground (sand).

• UAV: • Georefernced, Orthorectified Images (spatial resolution up to 0.1m). • Digital Elevation Models (DEM).


Orthorectified Images

Digital Elevation Model

Mars-like Real Dataset

• Atacama Seeker Dataset 2012 (Woods et al. ASTRA 2013) • Robovolc based rover, max 1𝑚𝑚

𝑠𝑠 up to 7 hours.

• More than 5km (in a day ) • Sensors

• Stereo Camera. • Inertial Measurement Unit (IMU). • Wheel Odometry. • DGPS

• Features • Rocky and smooth terrain. • Direct sunlight (at some parts). • Great instantaneous alterations (mostly around the roll axis).


Outline



Global Localization System


Detection of ROIs

Classification & Outlier Rejection

ROI Local Network

Relative Localization (VO & IMU)

Inertial Measurements

Rover Stereo Pair

ROI Local Network Update

ROI Sub-Network Match

Rover Global Localization

ONBOARD PROCEDURE

Detection of ROIs


Orbital Images

ROI Global Network

OFFLINE PROCEDURE

Detection on Orbital Imagery


Detection of ROIs


ROI Local Network



Rover Stereo Pair




ONBOARD PROCEDURE

Detection of ROIs


Orbital Images

ROI Global Network

OFFLINE PROCEDURE


• Detection of prominent shapes (inc. blob-like, ellipse-like)


∂2I∂x∂x

∂2I∂x∂y

∂2I∂y∂x

∂2I∂y∂y

H(x, y)=

• Hessian Analysis: • Second order partial derivatives along image directions. • Hessian eigenvalues define the principal curvature. • Extreme eigenvalues of Hessian correspond to protruding and well defined shapes in the

image .

• Local Entropy: • Weighted Entropy • Gaussian coefficient w(i) to make low differentiations negligible • Only extreme values are considered as blob candidates.

Orbital Image

Hessian Analysis

Entropy Response

Adaptive Thresholding for Rock

Detection

Adaptive Thresholding for

Outcrop Detection

Rock Candidates

Outcrop Candidates


• Methods that detect different and complementary regions • Logical OR operation • Binary image filling • Filtering based on the size of ROIs


An example of the detection employing the mixture of Hessian and

Entropy

An example of the detection employing solely the Hessian analysis

Classification on Orbital Imagery


Detection of ROIs


ROI Local Network



Rover Stereo Pair




ONBOARD PROCEDURE

Detection of ROIs


Orbital Images

ROI Global Network

OFFLINE PROCEDURE

Classification on Orbital Imagery


Detection of ROIs


ROI Local Network



Rover Stereo Pair




ONBOARD PROCEDURE

Detection of ROIs


Orbital Images

OFFLINE PROCEDURE

ROI Global Network

Classification: Feature Space

• Intensity Values: • RGB. • HSV. • Grayscale.

• Local Entropy. • Gabor Features:

• σ, standard deviation of the 2D Gaussian. • θ, orientation of the sinusoidal. • φ, phase offset.


Classification: Feature Generation

• Feature vector: N-Dimensional vector • Data matrix: MxN matrix, M number of samples • “Raw” Feature vectors size:

• 20px by 20px ROIs. • Intensity, 400 values. • Entropy, 400 values. • Gabor, 28800 values.

• Principal Component Analysis (PCA) • Linear mapping to a subspace. • 32800 dimensional vector 119 dimensional vector, 99% of information.

5/19/2015 Advanced Space Technologies in Robotics and Automation

(ASTRA 2015) 26

Classification: k-NN

• k Nearest Neighborhoods vote • Simple • Competes (some cases) SVM and Perceptor based methods • Only one parameter to setup


An example “Outcrop vs Ground" classication result.

An example “Rock vs Ground" classication result.

Global ROI network (GN)

Relative Localization


Detection of ROIs


ROI Local Network



Rover Stereo Pair




ONBOARD PROCEDURE

Detection of ROIs


Orbital Images

OFFLINE PROCEDURE

ROI Global Network



Detection of ROIs


ROI Local Network



Rover Stereo Pair




ONBOARD PROCEDURE

Detection of ROIs


Orbital Images

ROI Global Network

OFFLINE PROCEDURE


• Visual Odometry: • 3D Reconstruction. • Feature Matching. • Filtering. • EPnP.

• IMU & VO fussion: (loosely coupled)


Left Image (t-1)

Right Image (t-1)

Left Image (t)

Right Image (t)

Features: Detection, Description

Features: matching

Local 3D Map Disparity Calculation 3D Reconstruction

Local 3D Map Disparity Calculation 3D Reconstruction

3D Reconstruction

Matched 3D Points

Matched 2D Points

3-point RANSAC 3D to 3D

Motion Estimation

Inliers

Best Transform

Efficient PnP 3D to 2D

Pose Estimation

Visual Odometry Motion Estimation

Inertial Measurement Unit (IMU)

Extended Kalman Filter (EKF)

Rover Relative Localization

Relative Localization: 3D Map Building • Disparity calculation from stereo pairs • 3D points are reconstructed for every consequent frame. • Intensity info from the left image of the stereo pair.

_


Relative Localization: Results

Advanced Space Technologies in Robotics and Automation (ASTRA 2015) 32

• Tested on 64000 stereo frames, without dropping any between. Most relative long range implementations are • either managing the data to be utilized by dropping some frames with low

movement as in Konolige’s Work or • realigning the relative visual odometry with the groundtruth for the first 50

meters of several traverse sections (with varying lengths of 300 to 600 m) as in Lambert’s work.

• 64000 stereo frames, a traverse of more than 3080m • the algorithm is executed without any user intervention or data selection.

• The respective IMU measurements at each frame are utilized as they offer a great correction to the relative localization system.

[1] Konolige, K., Agrawal, M. and Sola, J. Large-scale visual odometry for rough terrain. Springer 2011. [2] Lambert, A., Furgale, P., Barfoot, T.D. and Enright, J. Field testing of visual odometry aided by a sun sensor and inclinometer. Wiley 2012.

Detection on Rover Imagery


Detection of ROIs


ROI Local Network



Rover Stereo Pair




ONBOARD PROCEDURE

Detection of ROIs


Orbital Images

ROI Global Network

OFFLINE PROCEDURE



Detection of ROIs


ROI Local Network



Rover Stereo Pair




ONBOARD PROCEDURE

Detection of ROIs


Orbital Images

ROI Global Network

OFFLINE PROCEDURE


• Hessian Analysis • Entropy • Horizon Detection • 3D reconstruction


Left Stereo Image

Right Stereo Image

Disparity Calculation

3D Reconstruction

Hessian Analysis

Entropy Response

Horizon Detection

Appearance Adaptive Thresholding for Rock

Detection

Appearance Adaptive Thresholding for

Outcrop Detection

Geometrical Thresholding for Rock

Detection

Geometrical Thresholding for

Outcrop Detection

Rock Candidates

Outcrop Candidates

Example rover image containing

rocks. Hessian. Entropy

Example rover image containing

outcrops. Hessian Entropy

Horizon Rock Detection Mask Horizon Outcrop

Detection Mask

Classification on Rover Imagery


Detection of ROIs


ROI Local Network



Rover Stereo Pair




ONBOARD PROCEDURE

Detection of ROIs


Orbital Images

ROI Global Network

OFFLINE PROCEDURE



Detection of ROIs


ROI Local Network



Rover Stereo Pair




ONBOARD PROCEDURE

Detection of ROIs


Orbital Images

ROI Global Network

OFFLINE PROCEDURE



Local Network (LN)

ROI Matching


Detection of ROIs


ROI Local Network



Rover Stereo Pair




ONBOARD PROCEDURE

Detection of ROIs


Orbital Images

ROI Global Network

OFFLINE PROCEDURE

ROI Matching

• Data-Aligned Rigidity-Constrained Exhaustive Search (DARCES) • Create constellations (triangles) of control points on the LN. • Calculate all hypotheses of triangle matches among LN and GN. • Prune the number of valid hypotheses with constraints:

• Edges length • Orientation • Semantic information (Outcrop, Rock)

• Select the affine transformation that best fits the LN-GN


ROI Matching: Results

• The final error of the localization is 38.46 m after a route of 3076.76 m in an area of approximately 6 km2.

• Most importantly the rover is globally localized!


Outline



Repeatability of Matching

• Create simulated traverses: • Based on the GN • Simulate the system’s detection

capabilities (~87%) • Simulate the localization error

(+1%,-1%)


• 174 ROIs on GN • >58000 simulated trajectories • 99.3% of correct matching. The algorithm is robust (given the assumptions of ROI plethora).

Conclusions

• Achievements: • Global Localization System • Rover ability to autonomously detect ROIs • Tested the system for robustness

• Next steps: • Improvements in:

• Detection of candidate ROI • Machine Learning (GP, Deep Learning for classification) • Infuse our system with a Bayesian Recursive framework

• Building a rover for long range navigation

• No method is panacea


Group of Robotics and Cognitive Systems http://robotics.pme.duth.gr/

Democritus University of Thrace Department of Production and Management Engineering, Greece

European Space Agency, European Space Research and Technology Centre (ESTEC)

Network/Partnering Initiative (NPI)

Matching Sparse Networks of Semantic ROIs Among Rover and Orbital Imagery

Evangelos Boukas, Antonios Gasteratos, Gianfranco Visentin

http://robotics.pme.duth.gr/

european space agency - matching sparse networks of...

Documents