Biologically inspired Mobile Robot Vision Localization

Presenter Folami Alamudun

AuthorsChristian Siagian

Laurent Itti

IntroductionVision-based LocalizationScene RecognitionTopological MapsBiological Vision Localization System Experimental ResultsDiscussionRelated work

Introduction

What? ◦Robot localization system using biologically

inspired visionWhy?

◦Provide machines with a human-like perceptual system capable conducting intelligent localization in an unstructured environment.

How?◦Biologically inspired scene summarization (gist)

and landmark identification (saliency).

IntroductionVision-based LocalizationScene RecognitionTopological MapsBiological Vision Localization System Experimental ResultsDiscussionRelated work

Vision-based Localization

Vision◦Primary perceptual system for localization in most

animals (including humans).◦Effective in most environments where sonar, radar and

GPS are unavailable or inoperable.The human process of localization is performed

using two processes:◦Gist – A holistic statistical signature of the image,

thereby yielding abstract scene classification and layout.

◦Saliency – A measure of interest at every image location and landmark-identification.

Vision-based Localization

Vision-based localization systems use vision information to classify systems using:

Global features – A general summary of information over the entire image.

Local features – Computed over a limited area of the image

Vision-based Localization – Global Features

Global feature methods generally consider an input image as a whole and extract a low-dimensional signature.

Advantage:◦ Provides a summary of the image statistics or semantics.◦ Robust because random local pixel noise averages out on global

scale.

Disadvantages:◦ Sacrifices spatial information such as feature location and

orientation.◦ Unable to define accurate pose estimation◦ Harder to deduce positional change even with significant robot

movement.

Vision-based Localization – Local Features

Local feature methods limit their scope to image regions and their respective configuration relationships to form a signature of a location.

Advantage◦ local features encode scene characteristics that are more

focused in scope.◦ Invariant in scale, in-plane rotation, viewpoint and lighting

invariance.

Disadvantage◦ Very slow.

IntroductionVision-based LocalizationScene RecognitionTopological MapsBiological Vision Localization System Experimental ResultsDiscussionRelated work

Scene Recognition

Human visual processing system uses visually interesting regions within the field of view.

Saliency-based selection of landmarks that are most reliable in a particular environment.

Focusing on specific regions for comparing different images makes for a less computationally expensive process

Scene Recognition

Scene Recognition

IntroductionVision-based LocalizationScene RecognitionTopological MapsBiological Vision Localization System Experimental ResultsDiscussionRelated work

Topological Maps

A topological map is a graph annotation of an environment.

Topological Maps assign nodes to particular places and edges as paths if direct passage between pairs of places (end nodes) exist.

Humans manage spatial knowledge primarily by topological information.

This information is used to construct a hierarchical topological map that describes the environment.

IntroductionVision-based LocalizationScene RecognitionTopological MapsBiological Vision Localization System Experimental ResultsDiscussionRelated work

Biological Vision Localization System

Biological Vision Localization System

The localization system is divided into three stages:

Feature extraction – Processes image to produce:◦Gist features; ◦Salient regions.

Recognition - compares features with memorized environment visual information.

Localization – compute where the robot is situated.

Biological Vision Localization System – Feature extraction

Feature extraction involves processing of raw low-level filter outputs into gist and saliency modules.

Gist feature extraction ◦Computes average values from sub-regions of

feature maps.◦Dimensionality reduction using PCA/ICA

Salient region selection and segmentation ◦Uses feature maps to detect conspicuity regions

in each channel.

Biological Vision Localization System – Gist Feature extraction

Biological Vision Localization System – Gist Feature extraction

Biological Vision Localization System – Segment and Salient Region Recognition

This stage attempts to match salient regions and gist features with stored environment information.

Segment estimator: ◦Three-layer neural network classifier trained using the

back-propagation on gist features

Salient Region Recognition:◦Recalls stored salient regions◦Uses SIFT key points and salient feature vector to

recognize regions.

Biological Vision Localization System

Segment Estimation computes likelihood that a scene belongs to a segment:

Salient region localization provides a saliency map which highlights coordinates of peak values (salient points).◦These points are used for identification

purposes in subsequent viewing.

Biological Vision Localization System – Salient Region Recognition

Recollection of stored salient regions for localization involves:

SIFT keypoints◦SIFT recognition system using parameters and

thresholds.Salient feature vector

◦A set of values taken from 5x5 window centered at the salient point location.

Biological Vision Localization System – Salient Region Recognition

(continued)Salient feature vectors form two salient

regions (sreg1, sreg2) are compared using:◦Similarity

◦Proximity

Biological Vision Localization System – Salient Region Recognition

Biological Vision Localization System – Monte Carlo Localization

When a landmark is recognized its associated location is used to deduce robot location.

Accumulated temporal context is used to distinguish between identical landmarks.

Robot position is estimated by implementing Monte-Carlo Localization (MCL) which utilizes Sampling Importance Resampling (SIR).

Biological Vision Localization System – Monte Carlo Localization

St as a set of weighted particles:◦ St = {xt,i, wt,i}, (i = 1, . . . , N)◦ xt,i = {snumt,i , ltravt,i} (possible robot location)

snum – segment number Ltrav – length traveled along segment edge

◦ wt,i = weight likelihood.◦ at time t; and ◦ N is the number of particles.

◦ Bel(St) = location belief at time t.◦ ut = motion measurement

Biological Vision Localization System – Monte Carlo Localization

Belief estimation algorithm:Apply motion model to St−1 to create St’ .Apply segment observation model to St’ to create St’’.IF (Mt > 0):

◦ apply salient region observation model to St’’ to yield St ;ELSE St = St’’.

Where:◦ St’ = is the belief state after application of motion model.◦ St’’ = is the state after the segment observation

Biological Vision Localization System – Monte Carlo Localization

IntroductionVision-based LocalizationScene RecognitionTopological MapsBiological Vision Localization System Experimental ResultsDiscussionRelated work

Experimental Results – rigid environment

Lighting conditions test

Experimental Results – rigid environment

Lighting conditions test

Experimental Results – rigid environment

Test System response on sparser scenes

Experimental Results – rigid environment

Test System response on sparser scenes

IntroductionVision-based LocalizationScene RecognitionTopological MapsBiological Vision Localization System Experimental ResultsDiscussionRelated work

Discussion

This paper introduced new ideas (the use of complementary gist and saliency features).

Saliency model lets the system automatically select persistent salient regions as localization cues.

Low computation cost gist features approximate the image layout and provide segment estimation.

System is able to compute coordinate level localization in multiple environments

Performance is comparable to GPS database guided systems.

Related Work

Determining Patch Saliency Using Low-Level Context. European Conference on Computer Vision (ECCV), 2008. D. Parikh et. al.