object recognition in 3d scans - ros.org · bastian steder - object recognition in 3d scans -...

Object Recognition in 3D scans

Willow Garage Intern Presentation 09/23/2010

Bastian StederUniversity of Freiburg, Germany

Bastian Steder - Object Recognition in 3D scans - intern presentation 2

Object Recognition in 3D scans

Goal: Object detection andGoal: Object detection andlocalization in point cloudslocalization in point clouds

Bastian Steder - Object Recognition in 3D scans - intern presentation 3

Working on Range images

Main advantage:Main advantage:Encodes negative informationEncodes negative information

Bastian Steder - Object Recognition in 3D scans - intern presentation 4

• Match features between model and scene

• Estimate object candidate poses

• Score and filter candidates

General Approach

Bastian Steder - Object Recognition in 3D scans - intern presentation 5

Feature extraction

Mainly two steps:

• Interest point detection:

Where should features be extracted

• Calculation of a descriptor:

Describe the area around the interest point in a way that makes it easy to compare

Bastian Steder - Object Recognition in 3D scans - intern presentation 6

Desired properties of our Features

• Use positions where the surface is stable,

but with significant changes in surrounding (I)

• Encode the surface structure (I+D)

• Encode outer shape (I+D)

• Provide local 6DOF transformation

=> NARF: Normal aligned radial features

Bastian Steder - Object Recognition in 3D scans - intern presentation 7

Border information

• Three kinds of border points

– Obstacle borders

– Shadow borders

– Veil points

Bastian Steder - Object Recognition in 3D scans - intern presentation 8

How to detect borders?

• Possibilities:

– Impact angles of the sensor beams

– Changes in normals

– …

• What actually works well:

– Changes in distances toneighboring points

Bastian Steder - Object Recognition in 3D scans - intern presentation 9

Border extraction video

Bastian Steder - Object Recognition in 3D scans - intern presentation 10

Interest points extraction

• Calculate a score for each image point

– How much do the dominant directions of the surface changes in the vicinity vary?

(border information & principal curvature)

– How stable is the surface on the point itself?

• Find maximums above a threshold

Bastian Steder - Object Recognition in 3D scans - intern presentation 11

NARF Interest point extraction video

Bastian Steder - Object Recognition in 3D scans - intern presentation 12

NARF descriptor video

Bastian Steder - Object Recognition in 3D scans - intern presentation 13

Results

ROC curve

Bastian Steder - Object Recognition in 3D scans - intern presentation 14

Example – Step 1

Bastian Steder - Object Recognition in 3D scans - intern presentation 15

Spatial validation – Validation points

Validation points on the model views for a fast spatial validationValidation points on the model views for a fast spatial validation

Bastian Steder - Object Recognition in 3D scans - intern presentation 16

Example – Step 2

Bastian Steder - Object Recognition in 3D scans - intern presentation 17

Overlapping poses + ICP

• Removal of multiple instances of the same object in the same position, coming from different feature matches

• Refinement of the object poses using ICP

Bastian Steder - Object Recognition in 3D scans - intern presentation 18

Example – Step 3

Bastian Steder - Object Recognition in 3D scans - intern presentation 19

Spatial validation – View Simulation

Comparison of what was actually seen and Comparison of what was actually seen and what it should look like according to the modelwhat it should look like according to the model

Bastian Steder - Object Recognition in 3D scans - intern presentation 20

Example – Step 4

Bastian Steder - Object Recognition in 3D scans - intern presentation 21

Video

Bastian Steder - Object Recognition in 3D scans - intern presentation 22

Final scene

Bastian Steder - Object Recognition in 3D scans - intern presentation 23

Software and Documentation

• Currently some parts in PCL and some in point_cloud_perception_experimental

– Range image class, border extraction, NARF interest point extraction, NARF descriptor, model class, spatial verification, visualization tools …

• Some Tutorials already exists – there will be more

• Paper submitted to ICRA11

Bastian Steder - Object Recognition in 3D scans - intern presentation 24

Thank you!

• Willow Garage in general for making this possible

• Radu for always being available for (sometimes stupid) questions and discussions

• Kurt for giving valuable input

• The other interns and normal employees for making it a great time

• The people in the kitchen for protecting me from months of Fast Food

Bastian Steder - Object Recognition in 3D scans - intern presentation 25

Questions?