Sean M. Ficht

Problem Definition

Previous Work

Methods & Theory

Results

Track and follow specific person with a mobile robot

Cluttered environments

Brief occlusion

Long occlusion

Cooperative user

Helper robot

Carry items for a person• Example: Hospital situation

Problem Definition

Previous Work

Methods & Theory

Results

Person following with a mobile robot

• Appearance based

• Optical flow based

• Stereo vision based

Segmentation of image

Classification

Detection

Limitations

Sidenbladh, Kragic, and Christensen; ICRA; 1999Tarokh and Ferrari; Journal of Robotic Systems; 2003Schlegel, Illman, Jaberg, Schuster, and Worz; British Machine Vision Conference; 2005

Calculate optical flow

Use to segment image

Limitations

Chivilo, Mezzaro, Sgorbissa, and Zaccaria; IROS; 2004Piaggio, Fornaro, Piombo, Sanna, and Zaccaria; IEEE ISIC/CIRA/ISAS joint conference; 1998

Find features

Segment from background

Use to track

Limitations

Zhichao and Birchfield; IROS; 2007

Problem Definition

Previous Work

Methods & Theory

Results

Kinect

Provides a depth image

Provides a RGB color image

Packaged solution

Detection and Tracking• Generic detector

• Specific appearance model

• Integrating particle filter

Robot Control

HOG person detector (OpenCV)

• HOG descriptoro Cells -> Block -> Windowo 4 cells in a blocko 105 blocks in a windowo 64x128 window

• Training

Dalal and Triggs, CVPR, 2005

Gradient of the Image

Binning of pixels in cells

Grouping of cells into blocks

Normalization

Kernel convolution

• Magnitude = (gx2 + gy

2)

• Angle = arctan(gy/gx)

Directional change in intensity

Bins apply to each cell

Nine separate bins

Gradient magnitude added to bin

Cells grouped into blocks

4 cells per block

Blocks overlap one another

HOG person detector

• HOG descriptoro Cells -> Block -> Windowo 4 cells in a blocko 105 blocks in a windowo 64x128 window

• Training

Support Vector Machine (SVM) classifier

Binary classifier

Trained on images

Detection and Tracking• Generic detector

• Specific appearance model

• Integrating particle filter

Robot Control

Color Histogram

Segmentation by depth to create template

Represents distribution of colors

10 bins for each color channel• 1000 element color histogram

Pixel classification

2 bin example• Bin 1: 0-127• Bin 2: 128-255

Average depth

Threshold (0.3 meters)

Template used to make color histogram

Detection and Tracking• Generic detector

• Specific appearance model

• Integrating particle filter

Robot Control

System State

Motion model

Observation model

Expected state

Resample

Hybrid state space

X and Y in image coordinates• Scaled according to depth

Z in depth coordinates

Detection and Tracking• Generic detector

• Specific appearance model

• Integrating particle filter

Robot Control

Input: tracking information from tracking algorithm

Uses tracking information to make movement decisions

Executes movement and returns to tracking algorithm

Problem Definition

Previous Work

Methods & Theory

Results

No occlusion

• Other people present (different depth)• Other people present (similar depth)• Pose change

Brief occlusion

Long occlusion

Initial template

Initial template Non-occluded target Occluded target

Average between 73% and 74%

Problem• Follow a person in different scenarios

System• RGB-D sensor• Generic detector• Specific appearance model• Particle filter• Robot control architecture

Performance• Performed in three separate test scenarios• Rapid side to side target motion trade-off• Large target scale changes

Train a new HOG detector to handle scale issues

Using more particles

KLT features for trajectory histories

Adaptive appearance model