Miguel Lourenço, João P. Barreto, Abed Malti

Institute for Systems and Robotics,

Faculty of Science and Technology

University of Coimbra, Portugal

Feature Detection and Matching in Images with Radial Distortion

Presentation Outline

SIFT Features – brief overview

RD problems in keypoint detection and matching Theoretical reasoning Experimental validation

Improvement to the SIFT algorithm to enhance it with RD

Real experiments – a comparison study

Motion estimation and 3D reconstruction in endoscopic images

Name / Location / Date Slide 2

Motivation for keypoint detection and matching

Miguel Lourenço – Anchorage, Alaska – ICRA 2010 Slide 3

Point correspondence across multiple views Camera calibration Sparse 3D reconstruction Recover camera/robot motion Visual Slam

Representation of image content Image retrieval applications Recognition tasks (e.g. Voc-tree) Image compression

Partioning of the descriptor space

SIFT Features (Lowe, IJCV 2004 – 6725 citations on google scholar)

Miguel Lourenço – Anchorage, Alaska – ICRA 2010 Slide 4

SIFT is probably the most broadly used algorithm for keypoint detection and matching

How does SIFT work ? Image salient points detected in a scale space framework

Incr

ease

sca

le

Gaussian pyramid DoG pyramid

(x,y,sigma)

SIFT Features (Lowe, IJCV 2004 – 6725 citations on Scholar google)

Miguel Lourenço – Anchorage, Alaska – ICRA 2010 Slide 5

SIFT is probably the most broadly used algorithm for keypoint detection and matching

How does SIFT work ? Image salient points detected in a scale space framework SIFT descriptor is computed based on local image gradient on a scale and rotation normalized

patch

Problem statement (1/2)

Miguel Lourenço – Anchorage, Alaska – ICRA 2010 Slide 6

XX

ZZO'O'

Q

What is radial distortion? Bending of the light rays pulling image points towards the center along

radial direction

OO

Cameras with radial lens distortion are often used in computer and robotic vision applications

Mini-lens Fish-eye lensBoroscope

Problem statement (2/2)

Miguel Lourenço – Anchorage, Alaska – ICRA 2010 Slide 7

SIFT is invariant to rotation and scale but it is not invariant to RD

Our Contribution: Modifications to the original SIFT for invariance to image RD

Assumptions: RD can be fairly described by the division model (Fitzgibbon, CVPR 2001)

RD is roughly known ( e.g. line stretching ) (Barreto, CVIU 2006)

336 correct matches421 correct matches

Tracking RD effects in SIFT

How does RD affect the SIFT algorithm? Study using images with artificially added distortion

Isolate the RD effect in SIFT detection and matchingReliable ground truth

Miguel Lourenço – Anchorage, Alaska – ICRA 2010 Slide 8

Improvement to the SIFT algorithm to handle RD issues

Results on real imagery

RD = 0% RD = 15% RD = 35% RD = 55%

How does RD affect keypoint detection?

Repeatability of keypoint detection decreases with increasing distortion

Miguel Lourenço – Anchorage, Alaska – ICRA 2010 Slide 9

Filt

erin

g bo

unds

Regular DoG pyramid ‘Distorted’ DoG pyramid

Small features (fine scale) tend to disappear during the blurring process Coarse features tend to be detected at finer levels of scale Flat regions (e.g. edges) start gain to strong gradient variations

Proposed Solution: Adaptive smoothing

We can avoid the reconstruction artifacts by using an adaptive filter

Miguel Lourenço – Anchorage, Alaska – ICRA 2010 Slide 10

Radial distortion must be removed before the Gaussian smoothing

Rectification (~ 1.5 seconds in Matlab)

Standard vs Adaptive Gaussian smoothing

Inherent properties of the standard Gaussian filter Decouple the convolution mask in X and Y directions

Miguel Lourenço – Anchorage, Alaska – ICRA 2010 Slide 11

Advantages of the Simplified Adaptive Filter Shape only depends on the radius of the convolution window Isotropic filter that can be decoupled for each image radius

Simplification of the adaptive filter

Detection repeatability (synthetic adding of RD)

Miguel Lourenço – Anchorage, Alaska – ICRA 2010 Slide 12

Better Repeatability results for keypoint detection

Repeatability

More robust to calibration errors

Error in calibration

Lower computational time than image rectification

Computational time

How does RD affect matching?

RD modifies the local structures in the image and by consequence the gradients are affected

Miguel Lourenço – Anchorage, Alaska – ICRA 2010 Slide 13

Changes in local gradients of the image deteriorates SIFT descriptor performance

Proposed solution: Compute gradients in the distorted image and perform implicit correction using the jacobian matrix of the distortion function

Matching evaluation (synthetic adding of RD)

Miguel Lourenço – Anchorage, Alaska – ICRA 2010 Slide 14

Compressive effect adds new contributions to the descriptor that do not occur in undistorted views

The matching performance can be improved by correcting image gradients before building the descriptor

Implicit gradient correction outperforms explicit image rectification for distortion amounts up to 25%.

Implicit gradient correction SIFT in Rectified Images SIFT in RD Images

Experiments with Real Images

Planar scenes for repeatability test and scenes with depth variation for motion estimation

Firewire camera with regular lens (~ 10% ) of distortion

Dragonfly camera with mini lens (~ 25% ) of distortion

Firewire camera with fish-eye lens (~ 45% ) of distortion

Miguel Lourenço – Anchorage, Alaska – ICRA 2010 Slide 15

Planar images

Miguel Lourenço – Anchorage, Alaska – ICRA 2010 Slide 16

10% distortion

25% distortion

45% distortion

SIFT in RD Images SIFT in Rectified Images Our method

176 matches 294 matches 364 matches

201 matches 310 matches 401matches

112 matches 253 matches 326 matches

Motion recovery / Sparse 3D reconstruction

Scenes with depth variation where wrong matches are discarded using epipolar geometric constraints

Miguel Lourenço – Anchorage, Alaska – ICRA 2010 Slide 17

10% distortion

25% distortion

45% distortion

Main Scene

Experimental evaluation

Miguel Lourenço – Anchorage, Alaska – ICRA 2010 Slide 18

Main Scene Number of Inliers RMS rotation angle 3D reconstruction Inliers Distribution

Conclusions / Future Work

We proposed a set of modifications to the original SIFT algorithm (RD-SIFT) for achieving invariance to radial distortion. The additional computational overhead is minimum

Miguel Lourenço – Anchorage, Alaska – ICRA 2010 Slide 19

RD – SIFT proved to be superior to explicit image correction Better repeatability and retrieval performance Less computational overhead Increased robustness to calibration errors

Future Work Extend the approach to other keypoint detectors (e.g. MSER and SURF) Real-time implementation using GPGPU (to make available to the

community) Get rid of calibration dependence

Name / Location / Date Slide 20

THANKS FOR COMING

Miguel Lourenço – Anchorage, Alaska – ICRA 2010 Slide 21

Detection after explicit RD correction

Correct the radial distortion via image rectification

Miguel Lourenço – Anchorage, Alaska – ICRA 2010 Slide 22

Rectification (~ 1.5 seconds in Matlab)

Drawbacks of this approach Signal reconstruction introduces artifacts affecting SIFT performance

Image 1.5x Image (Bilinear) 1.5x Image (Bicubic)