november 6, 2011 - pcl - point cloud library (pcl) cad models whyand how i training on 3d models is...

39
Object Recognition, Classification and Evaluation with PCL November 6, 2011

Upload: trinhthuan

Post on 07-May-2018

221 views

Category:

Documents


4 download

TRANSCRIPT

Page 1: November 6, 2011 - PCL - Point Cloud Library (PCL) CAD models Whyand How I Training on 3D models is ... I easy: all you need is a CAD model and PCL I fast: some seconds / model I complete:

Object Recognition, Classification and Evaluation with PCLNovember 6, 2011

Page 2: November 6, 2011 - PCL - Point Cloud Library (PCL) CAD models Whyand How I Training on 3D models is ... I easy: all you need is a CAD model and PCL I fast: some seconds / model I complete:

Outline

1. Introduction

2. Object Recognition with PCL

3. Training data

4. Recognition & Pose

5. Databases & Evaluation

Aitor Aldoma & Walter Wohlkinger / PCL :: Object Recognition, Classification and Evaluation

Page 3: November 6, 2011 - PCL - Point Cloud Library (PCL) CAD models Whyand How I Training on 3D models is ... I easy: all you need is a CAD model and PCL I fast: some seconds / model I complete:

Object RecognitionCan we use PCL for object recognition?

filters io

features

octree search segmentation sample_consensus

surface

visualization

keypoints

Aitor Aldoma & Walter Wohlkinger / PCL :: Object Recognition, Classification and Evaluation

Page 4: November 6, 2011 - PCL - Point Cloud Library (PCL) CAD models Whyand How I Training on 3D models is ... I easy: all you need is a CAD model and PCL I fast: some seconds / model I complete:

Object RecognitionCan we use PCL for object recognition? Yes, we can

filters io

features

search segmentation sample_consensus

surface

visualization

Aitor Aldoma & Walter Wohlkinger / PCL :: Object Recognition, Classification and Evaluation

Page 5: November 6, 2011 - PCL - Point Cloud Library (PCL) CAD models Whyand How I Training on 3D models is ... I easy: all you need is a CAD model and PCL I fast: some seconds / model I complete:

Object RecognitionBuilding a PCL based object recognition app

Aitor Aldoma & Walter Wohlkinger / PCL :: Object Recognition, Classification and Evaluation

Page 6: November 6, 2011 - PCL - Point Cloud Library (PCL) CAD models Whyand How I Training on 3D models is ... I easy: all you need is a CAD model and PCL I fast: some seconds / model I complete:

Segmentationpcl::apps::DominantPlaneSegmentation

Segmentation of a Kinect scene

typedef pcl::PointCloud<OpenNIFrameSource::PointT> Cloud;std::vector<typename Clout::Ptr>, ... > clusters;frame = camera.snap();

pcl::apps::DominantPlaneSegmentation<OpenNIFrameSource::PointT> dps;dps.setInputCloud (frame);dps.setMaxZBounds (1.0);dps.setObjectMinHeight (0.005);dps.setMinClusterSize (1000);dps.setWSize(3);dps.setDistanceBetweenClusters(0.1);dps.setDownsamplingSize (0.02);dps.compute_fast (clusters);

Aitor Aldoma & Walter Wohlkinger / PCL :: Object Recognition, Classification and Evaluation

Page 7: November 6, 2011 - PCL - Point Cloud Library (PCL) CAD models Whyand How I Training on 3D models is ... I easy: all you need is a CAD model and PCL I fast: some seconds / model I complete:

Object RecognitionWhat else do we need? Descriptors

I Encode whole object with descriptor (globally)I VFHI CVFHI SHOTI SpinImageI ShapeContextI ShapeDistributionsI ...

Aitor Aldoma & Walter Wohlkinger / PCL :: Object Recognition, Classification and Evaluation

Page 8: November 6, 2011 - PCL - Point Cloud Library (PCL) CAD models Whyand How I Training on 3D models is ... I easy: all you need is a CAD model and PCL I fast: some seconds / model I complete:

VFHViewpoint Feature Histogram

I Encodes the surface of the object and the viewpointI Relative to the centroid and the average of the normals

I Efficient to compute

Aitor Aldoma & Walter Wohlkinger / PCL :: Object Recognition, Classification and Evaluation

Page 9: November 6, 2011 - PCL - Point Cloud Library (PCL) CAD models Whyand How I Training on 3D models is ... I easy: all you need is a CAD model and PCL I fast: some seconds / model I complete:

FeaturesVFH Estimation in PCL

1 using namespace pcl;2 typedef PointCloud<PointXYZ> CloudXYZ;3 typedef PointCloud<Normal> CloudNormal;4 typedef PointCloud<VFHSignature308> VFHSig;5 typedef CloudXYZ::Ptr CloudXYZPtr;6 typedef CloudNormal::Ptr CloudNormalPtr;78 CloudXYZPtr cloud(new CloudXYZ());9 CloudNormalPtr normals(new CloudNormal());10 // Compute normals11 ...1213 // Compute VFH14 VFHEstimation<PointXYZ, Normal, VFHSignature308> vfh;15 vfh.setInputCloud (cloud);16 vfh.setInputNormals (normals);17 KdTreeFLANN<PointXYZ>::Ptr tree (new KdTreeFLANN<PointXYZ> ());18 vfh.setSearchMethod (tree);19 VFHSig::Ptr vfhs (new VFHSig ());20 // Compute the features21 vfh.compute (*vfhs);

Aitor Aldoma & Walter Wohlkinger / PCL :: Object Recognition, Classification and Evaluation

Page 10: November 6, 2011 - PCL - Point Cloud Library (PCL) CAD models Whyand How I Training on 3D models is ... I easy: all you need is a CAD model and PCL I fast: some seconds / model I complete:

Training dataWhat else do we need? Training data

I Given partial views from the KinectI we also want views for training

Aitor Aldoma & Walter Wohlkinger / PCL :: Object Recognition, Classification and Evaluation

Page 11: November 6, 2011 - PCL - Point Cloud Library (PCL) CAD models Whyand How I Training on 3D models is ... I easy: all you need is a CAD model and PCL I fast: some seconds / model I complete:

Training dataSources:

I Capturing views with real sensors

Aitor Aldoma & Walter Wohlkinger / PCL :: Object Recognition, Classification and Evaluation

Page 12: November 6, 2011 - PCL - Point Cloud Library (PCL) CAD models Whyand How I Training on 3D models is ... I easy: all you need is a CAD model and PCL I fast: some seconds / model I complete:

Training dataSources:

I Scanned object models

Aitor Aldoma & Walter Wohlkinger / PCL :: Object Recognition, Classification and Evaluation

Page 13: November 6, 2011 - PCL - Point Cloud Library (PCL) CAD models Whyand How I Training on 3D models is ... I easy: all you need is a CAD model and PCL I fast: some seconds / model I complete:

Training dataSources:

I CAD models

Aitor Aldoma & Walter Wohlkinger / PCL :: Object Recognition, Classification and Evaluation

Page 14: November 6, 2011 - PCL - Point Cloud Library (PCL) CAD models Whyand How I Training on 3D models is ... I easy: all you need is a CAD model and PCL I fast: some seconds / model I complete:

Using CAD modelsWhy and How

I Training on 3D models is ...

I easy: all you need is a CAD model and PCLI fast: some seconds / modelI complete: no missed views, viewpoints evenly spaced

around the objectI parameterizable: easily re-trainable with different

parametersI number of viewsI resolutionI noise levelI distance to object

I sensor independent: adapt code to simulate your favoritesensor characteristics (fov, aliasing, noise, errors, holes )

I providing additional information: entropy, aspect graphs,common reference frame

Aitor Aldoma & Walter Wohlkinger / PCL :: Object Recognition, Classification and Evaluation

Page 15: November 6, 2011 - PCL - Point Cloud Library (PCL) CAD models Whyand How I Training on 3D models is ... I easy: all you need is a CAD model and PCL I fast: some seconds / model I complete:

Using CAD modelsWhy and How

I Training on 3D models is ...I easy: all you need is a CAD model and PCL

I fast: some seconds / modelI complete: no missed views, viewpoints evenly spaced

around the objectI parameterizable: easily re-trainable with different

parametersI number of viewsI resolutionI noise levelI distance to object

I sensor independent: adapt code to simulate your favoritesensor characteristics (fov, aliasing, noise, errors, holes )

I providing additional information: entropy, aspect graphs,common reference frame

Aitor Aldoma & Walter Wohlkinger / PCL :: Object Recognition, Classification and Evaluation

Page 16: November 6, 2011 - PCL - Point Cloud Library (PCL) CAD models Whyand How I Training on 3D models is ... I easy: all you need is a CAD model and PCL I fast: some seconds / model I complete:

Using CAD modelsWhy and How

I Training on 3D models is ...I easy: all you need is a CAD model and PCLI fast: some seconds / model

I complete: no missed views, viewpoints evenly spacedaround the object

I parameterizable: easily re-trainable with differentparameters

I number of viewsI resolutionI noise levelI distance to object

I sensor independent: adapt code to simulate your favoritesensor characteristics (fov, aliasing, noise, errors, holes )

I providing additional information: entropy, aspect graphs,common reference frame

Aitor Aldoma & Walter Wohlkinger / PCL :: Object Recognition, Classification and Evaluation

Page 17: November 6, 2011 - PCL - Point Cloud Library (PCL) CAD models Whyand How I Training on 3D models is ... I easy: all you need is a CAD model and PCL I fast: some seconds / model I complete:

Using CAD modelsWhy and How

I Training on 3D models is ...I easy: all you need is a CAD model and PCLI fast: some seconds / modelI complete: no missed views, viewpoints evenly spaced

around the object

I parameterizable: easily re-trainable with differentparameters

I number of viewsI resolutionI noise levelI distance to object

I sensor independent: adapt code to simulate your favoritesensor characteristics (fov, aliasing, noise, errors, holes )

I providing additional information: entropy, aspect graphs,common reference frame

Aitor Aldoma & Walter Wohlkinger / PCL :: Object Recognition, Classification and Evaluation

Page 18: November 6, 2011 - PCL - Point Cloud Library (PCL) CAD models Whyand How I Training on 3D models is ... I easy: all you need is a CAD model and PCL I fast: some seconds / model I complete:

Using CAD modelsWhy and How

I Training on 3D models is ...I easy: all you need is a CAD model and PCLI fast: some seconds / modelI complete: no missed views, viewpoints evenly spaced

around the objectI parameterizable: easily re-trainable with different

parametersI number of viewsI resolutionI noise levelI distance to object

I sensor independent: adapt code to simulate your favoritesensor characteristics (fov, aliasing, noise, errors, holes )

I providing additional information: entropy, aspect graphs,common reference frame

Aitor Aldoma & Walter Wohlkinger / PCL :: Object Recognition, Classification and Evaluation

Page 19: November 6, 2011 - PCL - Point Cloud Library (PCL) CAD models Whyand How I Training on 3D models is ... I easy: all you need is a CAD model and PCL I fast: some seconds / model I complete:

Using CAD modelsWhy and How

I Training on 3D models is ...I easy: all you need is a CAD model and PCLI fast: some seconds / modelI complete: no missed views, viewpoints evenly spaced

around the objectI parameterizable: easily re-trainable with different

parametersI number of viewsI resolutionI noise levelI distance to object

I sensor independent: adapt code to simulate your favoritesensor characteristics (fov, aliasing, noise, errors, holes )

I providing additional information: entropy, aspect graphs,common reference frame

Aitor Aldoma & Walter Wohlkinger / PCL :: Object Recognition, Classification and Evaluation

Page 20: November 6, 2011 - PCL - Point Cloud Library (PCL) CAD models Whyand How I Training on 3D models is ... I easy: all you need is a CAD model and PCL I fast: some seconds / model I complete:

Using CAD modelsWhy and How

I Training on 3D models is ...I easy: all you need is a CAD model and PCLI fast: some seconds / modelI complete: no missed views, viewpoints evenly spaced

around the objectI parameterizable: easily re-trainable with different

parametersI number of viewsI resolutionI noise levelI distance to object

I sensor independent: adapt code to simulate your favoritesensor characteristics (fov, aliasing, noise, errors, holes )

I providing additional information: entropy, aspect graphs,common reference frame

Aitor Aldoma & Walter Wohlkinger / PCL :: Object Recognition, Classification and Evaluation

Page 21: November 6, 2011 - PCL - Point Cloud Library (PCL) CAD models Whyand How I Training on 3D models is ... I easy: all you need is a CAD model and PCL I fast: some seconds / model I complete:

Using CAD modelsWhy and How

Obtain synthetic partial views of CAD models with PCL

std::string PLYModel = std::string(argv[1]);float resx = atof(argv[2]);float resy = resx;typedef pcl::PointCloud<pcl::PointXYZ> Cloud;std::vector <Cloud, ... > views_xyz;std::vector < Eigen::Matrix4f, ... > poses;std::vector<float> entropies;

pcl::visualization::PCLVisualizer vis;vis.addModelFromPLYFile (PLYModel, "mesh1", 0);vis.setRepresentationToSurfaceForAllActors ();vis.renderViewTesselatedSphere (resx, resy, views_xyz,

poses, entropies);

Aitor Aldoma & Walter Wohlkinger / PCL :: Object Recognition, Classification and Evaluation

Page 22: November 6, 2011 - PCL - Point Cloud Library (PCL) CAD models Whyand How I Training on 3D models is ... I easy: all you need is a CAD model and PCL I fast: some seconds / model I complete:

Object Recognition

I What we have so farI Input from KinectI SegmentationI DescriptorsI Training data

I Matching against DB: L1,L2, Chi-Square,... with FLANNI best matching objects+views

Aitor Aldoma & Walter Wohlkinger / PCL :: Object Recognition, Classification and Evaluation

Page 23: November 6, 2011 - PCL - Point Cloud Library (PCL) CAD models Whyand How I Training on 3D models is ... I easy: all you need is a CAD model and PCL I fast: some seconds / model I complete:

Object Recognition

I What we have so farI Input from KinectI SegmentationI DescriptorsI Training dataI Matching against DB: L1,L2, Chi-Square,... with FLANNI best matching objects+views

Aitor Aldoma & Walter Wohlkinger / PCL :: Object Recognition, Classification and Evaluation

Page 24: November 6, 2011 - PCL - Point Cloud Library (PCL) CAD models Whyand How I Training on 3D models is ... I easy: all you need is a CAD model and PCL I fast: some seconds / model I complete:

Object RecognitionTime for a live-demo...

I Recognition/Classification Demo

Aitor Aldoma & Walter Wohlkinger / PCL :: Object Recognition, Classification and Evaluation

Page 25: November 6, 2011 - PCL - Point Cloud Library (PCL) CAD models Whyand How I Training on 3D models is ... I easy: all you need is a CAD model and PCL I fast: some seconds / model I complete:

Object Recognition with PCLHow much code is to write?

pcl::visualization::PCLVisualizer vis("kinect");pcl::apps::DominantPlaneSegmentation<OpenNIFrameSource::PointT> dps;while(camera.isActive()) {frame = camera.snap();dps.setInputCloud (frame);dps.setMaxZBounds (Z_DIST_);dps.setObjectMinHeight (0.005);dps.setMinClusterSize (1000);dps.setWSize(3);dps.setDistanceBetweenClusters(0.1);dps.setDownsamplingSize (0.02);dps.compute_fast (clusters);vis.addPointCloud<OpenNIFrameSource::PointT>(filtered,"frame");for(size_t i=0; i < clusters.size(); i++) {vis.addPointCloud<OpenNIFrameSource::PointT> (clusters[i], random_handler, cluster_name.str());

recognizer.recognize (*cluster[i], model_ids, poses, confidences);

std::string category = getCategory(model_ids[0]);Eigen::Vector4f centroid;pcl::compute3DCentroid(*cluster[i], centroid);vis.addText3D(category, centroid, text_scale, 1, 0, 1, cluster_name.str(), 0);

}vis.spinOnce();

Aitor Aldoma & Walter Wohlkinger / PCL :: Object Recognition, Classification and Evaluation

Page 26: November 6, 2011 - PCL - Point Cloud Library (PCL) CAD models Whyand How I Training on 3D models is ... I easy: all you need is a CAD model and PCL I fast: some seconds / model I complete:

Object Recognitionits even easier

I some generic classes do most of the work

Aitor Aldoma & Walter Wohlkinger / PCL :: Object Recognition, Classification and Evaluation

Page 27: November 6, 2011 - PCL - Point Cloud Library (PCL) CAD models Whyand How I Training on 3D models is ... I easy: all you need is a CAD model and PCL I fast: some seconds / model I complete:

Object Recognition with PCLRecognition + Pose

I Aligned CAD models with objects

Aitor Aldoma & Walter Wohlkinger / PCL :: Object Recognition, Classification and Evaluation

Page 28: November 6, 2011 - PCL - Point Cloud Library (PCL) CAD models Whyand How I Training on 3D models is ... I easy: all you need is a CAD model and PCL I fast: some seconds / model I complete:

Object Recognition with PCLRecognition + Pose

I classes for

Aitor Aldoma & Walter Wohlkinger / PCL :: Object Recognition, Classification and Evaluation

Page 29: November 6, 2011 - PCL - Point Cloud Library (PCL) CAD models Whyand How I Training on 3D models is ... I easy: all you need is a CAD model and PCL I fast: some seconds / model I complete:

Object RecognitionTime for a live-demo...

I Recognition/Classification Demo with Pose

Aitor Aldoma & Walter Wohlkinger / PCL :: Object Recognition, Classification and Evaluation

Page 30: November 6, 2011 - PCL - Point Cloud Library (PCL) CAD models Whyand How I Training on 3D models is ... I easy: all you need is a CAD model and PCL I fast: some seconds / model I complete:

DatabasesWhere to get training data?

I Are there freely available sources out there?

Aitor Aldoma & Walter Wohlkinger / PCL :: Object Recognition, Classification and Evaluation

Page 31: November 6, 2011 - PCL - Point Cloud Library (PCL) CAD models Whyand How I Training on 3D models is ... I easy: all you need is a CAD model and PCL I fast: some seconds / model I complete:

DatabasesWhere to get data...

I Training data is freely availableI 3D-Net ( 3d-net.org )I Google’s 3D Warehouse, www.123dapp.com, turbosquid, ...I RoboEarthI Princeton Shape BenchmarkI Willow Garage household objects DBI KIT household object database, TUM, ...I ...

Aitor Aldoma & Walter Wohlkinger / PCL :: Object Recognition, Classification and Evaluation

Page 32: November 6, 2011 - PCL - Point Cloud Library (PCL) CAD models Whyand How I Training on 3D models is ... I easy: all you need is a CAD model and PCL I fast: some seconds / model I complete:

DatabasesHaving lots of CAD models...

I Nearest neighbor classification video

Aitor Aldoma & Walter Wohlkinger / PCL :: Object Recognition, Classification and Evaluation

Page 33: November 6, 2011 - PCL - Point Cloud Library (PCL) CAD models Whyand How I Training on 3D models is ... I easy: all you need is a CAD model and PCL I fast: some seconds / model I complete:

Evaluation

I Are there test databases out there?

Aitor Aldoma & Walter Wohlkinger / PCL :: Object Recognition, Classification and Evaluation

Page 34: November 6, 2011 - PCL - Point Cloud Library (PCL) CAD models Whyand How I Training on 3D models is ... I easy: all you need is a CAD model and PCL I fast: some seconds / model I complete:

EvaluationTest databases

I from 3D-NetI 1600 Kinect scenes, 10 Classes, single object, multiple

posesI +1600 scenes for 40 additional classes

Aitor Aldoma & Walter Wohlkinger / PCL :: Object Recognition, Classification and Evaluation

Page 35: November 6, 2011 - PCL - Point Cloud Library (PCL) CAD models Whyand How I Training on 3D models is ... I easy: all you need is a CAD model and PCL I fast: some seconds / model I complete:

EvaluationTest databases

I from SHRECI http://www.aimatshape.net/event/SHREC/I Shape Retrieval Contest of Range ScansI Range images captured using a Minolta Laser ScannerI segmented objects, as mesh

Aitor Aldoma & Walter Wohlkinger / PCL :: Object Recognition, Classification and Evaluation

Page 36: November 6, 2011 - PCL - Point Cloud Library (PCL) CAD models Whyand How I Training on 3D models is ... I easy: all you need is a CAD model and PCL I fast: some seconds / model I complete:

EvaluationTest databases

I from Lai’s RGBD-DBI http://www.cs.washington.edu/rgbd-dataset/I 300 common household objects, 51 classesI on Turntable with Kinect, 3 diff. viewing anglesI segmented objects, as colored point cloud

Aitor Aldoma & Walter Wohlkinger / PCL :: Object Recognition, Classification and Evaluation

Page 37: November 6, 2011 - PCL - Point Cloud Library (PCL) CAD models Whyand How I Training on 3D models is ... I easy: all you need is a CAD model and PCL I fast: some seconds / model I complete:

EvaluationEvaluation code

I e.g. to test your new feature/descriptorI implement your feature/descriptor in PCLI get training models, put into directoryI get a test databaseI use provided evaluation code

./category_evaluation-models_dir ../ICCV/ModelDatabase-training_dir ../ICCV/ModelDatabase_trained-test_dir ../3DNET/Cat10_TestDatabase-descriptor_name vfh-n_nearest_neighbours 4

Aitor Aldoma & Walter Wohlkinger / PCL :: Object Recognition, Classification and Evaluation

Page 38: November 6, 2011 - PCL - Point Cloud Library (PCL) CAD models Whyand How I Training on 3D models is ... I easy: all you need is a CAD model and PCL I fast: some seconds / model I complete:

EvaluationEasy evaluation

appl

e

bana

na

bottl

e

bow

l

car

donu

t

ham

mer

mug

tetr

a pa

k

toile

t pap

er

toilet paper

tetra pak

mug

hammer

donut

car

bowl

bottle

banana

apple

0.0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1.0

1 2 3 4 5 6 7 8 9 10# Nearest Neighbours

0.0

0.2

0.4

0.6

0.8

1.0

Clas

sific

atio

n Ra

te

97% | 100% apple 82% | 92% banana 86% | 94% bottle 69% | 82% bowl 94% | 98% car 82% | 98% donut 100% | 100% hammer 99% | 100% mug 71% | 89% tetra pak 24% | 73% toilet paper 80% | 93% OVERALL

Aitor Aldoma & Walter Wohlkinger / PCL :: Object Recognition, Classification and Evaluation

Page 39: November 6, 2011 - PCL - Point Cloud Library (PCL) CAD models Whyand How I Training on 3D models is ... I easy: all you need is a CAD model and PCL I fast: some seconds / model I complete:

The end

I PCL demo-apps and code to simplify evaluation andtesting

I helps you concentrate on your research area

Aitor Aldoma & Walter Wohlkinger / PCL :: Object Recognition, Classification and Evaluation