folien ss2009 machine vision

8/3/2019 Folien SS2009 Machine Vision

1/163

Machine VisionMarkus Vincze

Automation and Control InstituteTechnische Universitt Wien

[email protected]


2/163

2

Contents

Motivation: many projects Terminology: from computer to machine vision Components: from light to images Machine Vision Segmentation Edge detection Grouping

Application: XPERO

robot learning


3/163

3

Industrial Projects FESTO Checkbox

(inspection of parts and part recpgnition)

AVL

Engine

Videoscope

(temperature measurement inthe engine during operation) IAEA/FAO Seperation ofmale and female Tsetse fliesto fight malaria Rauscher Inspection oftampoos, quality control


4/163

4

Prototypes

Holzer

Training-Optimisation-System

(TOS)


5/163

5

Research Projects CARAK Measuring 3D shape of retina FlexPaint Spray painting any industrial part

FibreScope

automatic

finding

of bore

holes

REDUX follow seam of carbon mats (EADS)


6/1636

Research Projects RobVision Navigation in ship ActIPret Interpretation of ahuman handling objects MOVEMENT reliable vision indoors Kognitives Sehen understanding to see,hide and seek XPERO Learning by experimentation


7/1637

3D (6 DoF) Object TrackingRobVision

EU Project 1998 -

2001

Model-based: Lines, ellipses estimate object pose

Robustness: integration of model cue and image cues Real time, 10 Hz


8/1638

RobVision: Vision Based NavigationRobVision


9/1639

Video: Navigation in Mock-up

RobVision


10/16310

Robot Navigation in Office

Vision for

Robotics

(V4R Tracking

Tool)


11/16311

Object Recognition and Tracking

Interest

points

and local

description

of the

objects appearance Code book as compact storage of many objects Recognition: relation of interest points to object centre


12/16312

Contents

Motivation: many projects Terminology: from computer to machine vision Components: from light to images Machine Vision Segmentation Edge detection Grouping

Application: XPERO

robot learning


13/16313

Computer Vision Computer Vision is a subfield of AI concerned withprocessing of images from the real world. Purpose:program a computer to"understand" a scene or features Methods:detection, segmentation,tracking, pose estimation, mapping to 3Dmodel, recognition of objects (e.g., human faces) Achieved bymeans of pattern recognition,statistical learning, projective geometry, imageprocessing, graph

theory

and other fields.

[1]

Dana H. Ballard, Christopher M. Brown, (1982) Computer Vision(2nd edition), Prentice Hall.


14/16314

Pattern Recognition

"the act of taking in raw data and taking an

action based on the category of the data" [1] Methods:statistics, machine learning, ... Problem:how to determine category of data

[1]

Richard O. Duda, Peter E. Hart, David G. Stork (2001) Patternclassification(2nd edition), Wiley, New York.


15/16315

Image Processing (Bildverarbeitung) Image to image processing [1] Subarea of computer / machine vision

[1]

Rafael C. Gonzales, Richard E. Woods, (2002) Digital ImageProcessing(2nd edition), Prentice Hall.

Fig.: Recognising number plates; 7 Spektra of NASA LANDSAT


16/16316

Machine Vision

=

application of computer

vision

to factory

automation.

A MV system is a computer that makes decisionsbased on the analysis of digital images.

Light

Lighting

ObjectSensor

Image

DataResultsControl

Media Reflection

Processing

Figure: Components of a machine vision system.


17/16317

Machine Vision Problem 1:narrow aplications Problem 2:understanding?

Lesson learned:more

options

to control

Lesson learned:consider complete systemLight

Lighting

ObjectSensor

Image

DataResultsControl

Media Reflection

Processing

Figure: Components of a machine vision system.

Light

Lighting

ObjectSensor

Image

DataResultsControl

Media Reflection

Processing


18/16318

Information searched for in Images

Determination of

Geometry: form, size Position and orientation (pose)

Properties

of materials: colour,

texture, irregularities (errors) Detect and recognise objects

Faces, persons, activities, ... Number plates, tables, mugs,industrial

parts, ...


19/16319

Contents

Motivation: many projects Terminology: from computer to machine vision Components: from light to images Machine Vision Segmentation Edge detection Grouping Stereo images Application: MOVEMENT robot navigation


20/163

20

From Light Source to Image

Many different influencing factors

Complete

modeling

is

impossible, yet

Solutions only under very constrained conditions


21/163

21

Electromagnetic Spectrum

[Encarta]


22/163

22

Radiometry Photometry

Radiometryis the

science of the measurement ofillumination

(light,

electromagnetic radiation)

Photometrieincludes

the

aspect of how lightappears to the humanobserver Radiometric units matched to human eye

Human average

eye

normed in trials of CIE(CommissionInternationale de

lEclairage)


23/163

23

Standard Observer

Fig.: CIE StandardObserver curve [B1.19]Fig.: Sensitivity of humancolour vision [Padgham

1975]


24/163

24

Day and Night Sight

Fig.: Spectral intensity function

(efficiency) of the human eye as

defined in CIE for night sight (1951) and sum daylight [B1.17]


25/163

25

Radiometric

und Photometric

Units

Lm Lumen

W Watt = J/s

Cd Candela

Sr Steradiant

= m2/m2

Lx Lux

Einheiten:


26/163

26

Radiant Flux Luminous Flux

(Strahlungsfluss)

Amount of radiation (photons) per second in Watt [W]

(Lichtstrom)

Amount of radiation perceived by standardobserver in Lumen [lm]

Fig.: Spectral sensitivity ofhuman eye defeined by CIE for

day light, 1931, 1964


27/163

27

Radiant Flux Luminous Flux

is

the

integral over

the entire spectrum S

weighs

the

radiant

flux according to the intensity function of thestandard

observer

eF

=0

)/( dFSF ee =750

380

)/(683 dFSF ev

vF

Example: Source

wit

radiant

flux

of 1W at a wave length

of 555nm

emits 683 LumenQuestion: How many Lumens emits an infra red 0.7 mW LED?


28/163

28

Radiant Inten. Luminous Intensity

(Strahlstrke)

Radiated power per unit of solid angle [W/sr]

(Lichtstrke)

Measured in the SIbasic unit Candela [cd]]/[1/1][1

222

mmrAsr =

Fig.: Definition of the solid angle orsteradians [sr] [B1.8]

Solid angle

(Raumwinkel) ofunit

sphere: 4


29/163

29

Irradiance Illuminance

(Bestrahlungsstrke)

Amount of radiation falling onto the innersurface of a unit sphere [W/m]

(Beleuchtungsstrke)

Visible illumination

Unit: Lux

[lux]

eE vE

Fig.: Definition of unitLux [B1.21]


30/163

30

Examples for IlluminanceLux Description

100,000 Sun light, noon32,000 Cloudy, noon

2,000 Cloudy, 1 hour after sun set

600 Illumination in supermarket450 Average office illumination

175 Street light, at night

10 Candel in 20 cm distance, pocket lamp0.3 Bright moon light, clear sky


31/163

31

Radiance Luminance

(Strahldichte)

Emitted intensity perunit area (e.g. from object) [W/m/sr]

(Leuchtdichte)

Response of humaneye or sensor toradiance [cd/m]

eL vL

Fig.: Definition of luminance [B1.24]


32/163


33/163

33

Fore-shortening

Fig.: Reduction of intensity [lux] withdistance and angle [B1.22].

Fig.: Effect of fore-shortening ishandled using the cosine

of tilt angle[H10.3].


34/163

34

Characteritics of Surfaces

Lambertian surface

Appears equally bright from all viewing directions

All incident

light is

reflected

Specular surface Mirror, all incident light is reflected Albedo Coefficent between 0 and 1 indicating how much

light a surface

reflects

relative to

ideal surface with no absorption Lambertian surface: albedo = .


35/163

35

Dichromatic Reflection Model Combines Lambertian and specular model

Fig.: Body and surface reflection.

Ee()


36/163

36

Reflection Models

Fig.: Summary of reflection models [Bajscy]


37/163

37

Dichromatic Reflection Model In general, radiance: Dichromatic model: Lambertian geometry function Gb independent of

viewing angle

Fig.: Actual, typical reflectivity pattern [B3.7]

)()( SEL ee =

bbeess GSGSS )(),()()( +=


38/163

38

Geometry of a Scene

Fig.: Angles relative toobject normal [H10.7].

Fig.: Definition of polarandazimuth angles [H10.6].


39/163

39

Image Formation

42

cos

4

=

f

DLE ee

Fig.: Relation between image irradianceEe and radiance of the surfaceLe tacking into account the size of regions in the image corresponding

to the patch on the object surface [H10.4].

Assumption:

is

small const.

Application:shape from shading shape of surface

Ee

Le .... radiance

f z


40/163

40

Orthographic Projection Normal- or parallel projection: x = X, y = Y Simplification when object is far away


41/163

41

Perspective Projection

Lochmodell (pin-hole model) einer Kamera

Fig.: Basic model of imaging process.

xz

fu =

yz

f

v =

Fig.: Pinhole model.


42/163

42

Pose of Camera in 3D Space Homogenous matrix:

ppRp += ic

=

10

pRAo

c

o

cAFig.: Camera and object coordinate system.

Pose estimation: possible from 3 (or more)points or lines of known object size


43/163

43

Lenses

Gaussian

lens

equations: and

Focal length:gbf

111+=

g

b

G

B=

BG

gBf

+=

Fig.: The lens law [DBS].


44/163

44

Field of View, FOV

small

focal

length wide FOV:

Depth of View Is obtained by Small aperture

Short focus length Large object distanceFig.: Field of view angle

[DBS].

=

f

B

2

atan2 max


45/163

45

Image Blur

Blur

circle

(Unschrfekreis)

d ... Aperture

opening

Power of alens

= 1/f [diopters]

babdu ='

Fig.: Diameter d of blur circle depending onaperture [DBS].


46/163

46

IlluminationDark

field

Bright field

diffuse: bell or

ring light

Through

light transparent object

Cast

schadow

Figs.: lighting options [DBS]


47/163

47

CCD Flchensensor


48/163

48

Farbkameras

3-Chip Kamera

1-Chip Kamera

Bayer

Farbmaske

[DCO]

[DCO]


49/163

49

Aliasing

[DCO]

Artefacte bei Farbbergngen Farb- und Intensittskante an verschiedenen Orten

Bildaufnahme bei


50/163

50

Bildaufnahme bei

Bewegung

Interlaced

Kamera: 2 Aufnahmen bei

je 25

(30) Hz

Abhilfe: Full-frame Kamera Ganzes Bild mit 25/50 Hz

odd field

even field

original

imageinterlaced image

of moving object

[DCO]


51/163

51

Images

Information in images: Intensity, Colour, multi-

spectral images, depth/distance/disparity, ...

Fig.: Intensity image (left) and range/depth image extracted fromthree intensity images [PointGrey]


52/163

52

Resolution, Image Pyramid

Fig.: Bildpyramide: Bilder wie oben nur mit gleicher Gre [G. Sandini]

Fig.: Bild bei verschiedenen Auflsungen (120x120, 60x60, 30x30)


53/163

53

Space-variant Images

Fig.: Images where resolution and field of view changes so that each imagecontains the same number of pixels

Fig.: Superimposing the above images [G. Sandini]


54/163

54

Log-polar Images, Fovea

Fig.: Log-polar pixel tessellation [G. Sandini]Fig.: Transformationof log-polar image

into a uniform,

rectangular pixeltesselation[G. Sandini]

[IBIDEM retina]

Advantage: similar tocharacteristics of human eye Efficient coding, e.g. video conferencing (64 kPixel)


55/163

55

Log-polar Images: ApplicationsFig.: Concentriccircles in log-polar

image project tovertical line, spokesto (nearly) horizontallines

Fig.: When focusingon the vanishing point

the radial lines (floorand ceiling to walls)project to horizontallines [Peters, Bishay,

1996].


56/163

56

Contents

Motivation: many projects Terminology: from computer to machine vision Components: from light to images Machine Vision Segmentation

Edge detection

Grouping

Application: XPERO

robot learning


57/163

57

Segmentation Partitioning the image into its constituent parts Constituent parts depend on the task Detect object, object class, foreground/background Stop whenobject of

interest is

found


58/163

58

Segmentation Partitioning the image into its constituent parts Constituent parts depend on the task Detect object, object class, foreground/background Stop whenobject of

interest is

found


59/163

59

Gaussian Convolution

2

22

22

1),(

yx

eyxG

+

=

Fig.: Original image =1, 5x5 =2, 9x9 =4, 15x15

Better smoothingthan mean


60/163

60

Salt and Pepper Noise

1% of Pixels either black or weight

Smoothing: outliers are spread but not eliminated

Fig.: Salt and Pepper Gauss filter=8, 5x5 Original image


61/163

61

Median Filter

Fig.: Median filter 3x3 Median filter 7x7 3x Median filter 3x3

Salt and Pepper (5%) Eliminates outliers (up to 50%)


62/163

62

Edge Detection 1D

Perfectedge

Perfectspreadedge

Edge withnoise

=1

Edge with noiseafter filtering

(Gauss)

Signal 1D

= Input

First

derivative= gradient

Second

derivative(Laplaceoperator)


63/163

63

Edge Detection 2D Fast: Roberts cross Strength:Angle:

Fig.: Original image Edge strength Binary image, threshold:80

,22

yx GGG += yx GGG +=4/3)/arctan( = xy GG xIGx /= yIGy /=


64/163

64

Edge Detection SobelAngle:

Fig.: Edge strength

)/arctan( xy GG=

Fig.: Original imageand after Sobel

Prewitt:


65/163

65

Edge Detection Canny Processing steps:

1. Gaussian smoothing2. Edge detection (e.g. Sobel)3. Ridge following with hystereses (t1 >t2

)

Optimised, standard method Good compromise Thin, 1 pixel edge (ridge) Smoothing can eliminate detail

Fig.:= 1, t1=255, t2=1


66/163

66

Canny - Examples Properties

Y-Effect: 3 edges meeting in a point are not connected Adaptive: detail and edge elements, but image dependent

Fig.:= 1, t1

=255, t2

=1 = 1, t1

=255, t2

=220 = 2, t1

=255, t2

=1


67/163

67

Laplacian of Gaussian - LoG Laplacian:

2

2

2

2

),(y

I

x

IyxL

+

=

2

22

22

22

4 21

1),(

yx

eyx

yxLoG

+

+=

Fig.: LoG or Mexican hat Example filter 9x9, =1.4

Fig.: Example of masks


68/163

68

LoG Example

Edges

are

zero

crossings

Ideal case: closed curves

Fig.: LoG with= 1 All zero crossings


69/163

69

LoG Edge Detection Derivative (differencing) amplifies noise ! Better results if scaled correctly

Fig.: zero crossings at= 2 = 1, strong zerocrossings (difference toneighbours >40)

Zero crossings at= 3


70/163

70

Contents

Motivation: many

projects

Terminology: from computer to machine vision Components: from light to images Machine Vision Segmentation

Edge detection

Region detection Grouping Application: XPERO robot learning


71/163

71

Histogram

Fig.: Original image t1=150 t1=130, t2=150 Regions (blobs)

Statistics over many pixel

Robust, fastIntensity

Numberofpixel

Intensity

t1

t1

t2

Numberofpixel


72/163

72

Thresholds - Examples

Fig.: Original image Histogram Binary image for t1=80 and t1=120

Fig.: Original image Histogram Binary image for t1=120


73/163

73

Finding Local Thresholds Statistics in neighborhood (e.g. 7x7) of a pixel

t = (mean value), or t = med (median) t = ( + med) / 2, or t = C (C .. constant)

Fig.: Original image Window 7x7, C=4 Window 140x140, C=8


74/163

74

Contents

Motivation: many

projects

Terminology: from computer to machine vision Components: from light to images Machine Vision Segmentation

Edge detection


Object Recognition Approaches


75/163

75

Object Recognition - Approaches Model based CAD Model of objects

Geometric features Finding features and their relationships Appearance based Interest points or entire object Gestalt Principles Regard structure in data and features

Perceptual grouping

Object Recognition Approaches


76/163

76

Object Recognition - Approaches Model based CAD Model of objects

Geometric features Finding features and their relationships Appearance based Interest points or entire object Gestalt Principles Regard structure in data and features

Perceptual grouping


77/163

77

Relation to the State of the Art Object Recognition = re-cognising a known object

Point features

and signatures

(e.g., SIFT [Lowe04])

Object categorisation = recognising objects belongingto a category

(bottle, animal)

Codebooks of features (blob size, colour, circle, e.g. [Pinz07,Skocaj07, Poggio07])

Object

detection

= find (relevant) entities

(to task)

Attention, grouping, figure-ground segmentation: task dependent methods (e.g., [Itti06] [Zillich07]) Perceived affordances = features related to actions Learned feature clusters, 2D laser data (e.g. [Kuipers06,07])


78/163

78

Perceptual Grouping Idea: exploit all the known structure in the data Learn what are objects (what they mean foroverall system), rather than specific objects Uses Gestalt principles

built-in knowledge

levels of abstraction

P t l G i


79/163

79

Perceptual Grouping

State of the Art

Recognition by components: A theory of

human image understanding [Biederman 1987,Dickinson, Bergevin, Biederman 1997]

Function depends on part relationship

36 geons

3D Object Recognition from single 2D images[Lowe 1987] Integration of regions and contours for objectrecognition [Schlter 2000]


80/163

80

A ComputationalStructure forPreattentive PerceptualOrganization[Sarkar 1994]


81/163

81

Principles of Perceptual Grouping

Perceptual organization

to form groupings andstructures in image

Proximity

Parallelism Collinearity Probabilistic ranking 13 principles [Wilson ]

[Lowe 87]


82/163

82

Model-based Recognition

[Lowe 87]

ModelImage Edges

Groupings Reprojections


83/163

83

Grouping, Objects and Semantics

Pixels

Object

Shortcut

Pixels

Object

Features

Proto Objects

Gestalts

Shortcut

Classical object recognition

Use

Vision(+phys.knowl.)

Vision(+model)

Cognitive system


84/163

84

Detection from Grouping Abstraction fromfeatures

to

Gestalts

Hierarchical grouping Learning is possiblefrom all features Problem:scaleability of feature search

Approach:

Incremental indexing [Zillich 2007]


85/163

85

Non-Accidentalness

Helmholtz

principle: non-accidental groupings

of features are perceived as Gestalts

The less likely randomness, the more likely an

underlying cause

Low likelihood of a grouping = highsignificance


86/163

86

Poisson Process Possible outcomes of a trial are 0 and 1 Trials are independent Expected number of occurrences in given

interval = Probability that there areexactly koccurrences

is

= Probability Mass/Distribution Function, discrete k
http://upload.wikimedia.org/wikipedia/commons/c/c1/Poisson_distribution_PMF.pnghttp://upload.wikimedia.org/wikipedia/commons/c/c1/Poisson_distribution_PMF.png


87/163

87

Poisson Process in 2D

Randomly distributed points


88/163

88


Adding some points


89/163

89


How non-accidental are k points in the circle?

Interval: two dimensional or area


90/163

90

Ellipse Detection

Joint work

of

Michael Zillich and Jiri Matas, CMP, Prague Exploit local information where possible Take any edge segmented image, e.g. Canny Minimal number of parameters


91/163


92/163

92

Approach1. Connected edgels

2. Local shape: fit circular arcs3.

Geometric constraints: convexity

4. Fit Ellipses

Canny Edges


93/163

93

Canny Edges

Canny Edges


94/163

94

Canny Edges


95/163

95

Approach1.

Connected edgels

2. Local shape: fit circular arcs

3.


4. Fit Ellipses


96/163

96

ArcsComparing three methods

SPLIT

GROW

RANSAC


97/163

97

SPLIT (Rosin, West 1995)


98/163

98

SPLIT

Problem:

When to stopsplitting?


99/163

99

GROW

1. Locally fit cirle2.

Grow

when

radius

similar


100/163

100

GROW

Problem:

Sensitive to end

points


101/163

101

RANSAC

1.

Grow

with

random

seeds

2. Take MDL optimum


102/163

102

Approach1.

Connected edgels

2. Local shape: fit circular arcs3. Geometric constraints: convexity

4. Fit Ellipses


103/163

103

Convex Arcs

convex non-convex


104/163

104

ConvexityInner side

of arc

Beyond endpoint of arc


105/163

105

Image Space VotingVote = Intersection of arc tangents/normals


106/163

106

Voted Arcs

Limited length Unlimited length

Depends strongly on length of tangent/normal


107/163

107

Run Time Complexity

Testing all pairs: O(n2

) Image space voting: O(n)


108/163

108

Growing Groups

Extend group of arcs to form ellipse

hypothesis

Exhaustive search Greedy search, heuristics: Co-curvilinearity

Ellipse centers Fit quality


109/163

109

Exhaustive vs. Greedy

17 groups 1 group

Greedy Search Heuristics


110/163

110

Greedy Search Heuristics

Co-curvilinearity

Yuen et. al.,1989:3 points on ellipse (P, Q, R)+ tangents

centre (C)

Ellipse fit, relative support


111/163

111

True Positive Rate (TPR)


112/163

112

Runtime


113/163

113

Approach1.

Connected edgels

2. Local shape: fit circular arcs3.


4. Fit Ellipses

-

B2AC [Fitzgibbon, Pilu, Fisher 1999], OpenCV

All Ellipses


114/163

114

All Ellipses

Good Ellipses


115/163

115

Good Ellipses


116/163

116


117/163

117

TPR vs. Runtime

Office scene in different sizes320x240 - 1280x960


118/163

118

More Structure

Lines, arcs, ellipses, parallelities,

continuations, junctions, ... higher level shapes, object, symbolic?

Vocabulary sentences (Prolog) C Perceptual grouping + symbolic reasoning


119/163

119

Hierarchical Perceptual Grouping1.

Connected edgels

2. Local shape: fit lines or circular arcs3.

Hierarchical

grouping

L- and T-junctions Collinear lines, parallel lines Ellipses Rectangle, flaps; cylinder


120/163

120

Detection from Grouping Abstraction fromfeatures

to

Gestalts

Hierarchical grouping Learning is possiblefrom all features Problem:scaleability of feature search

Approach:

Incremental indexing [Zillich 2007]

Rectangle


121/163

121

Rectangle

rect(A,B,C,D) :-

u(A,B,C), u(C,D,A)

line(A,B) :- left(A,X), right(B,X).rect(A,B,C,D) :- line(A,B), line(B,C), line(C,D), line(D,A)


122/163

122

Flap


123/163

123

Book Scene

3847 edges

First 2 rectangles All rectangles

Original image


124/163

124

Office Scene


125/163

125

Shelves scene


126/163

126

Cylindercyl(E,A,B) :-

tangent_l(E,A), tangent_r(E,B), parallel(A,B)


127/163

127

Cup Scene

Original image Edges First cylinder


128/163

128

Kitchen Scene

Original image First 3 cylinders


129/163

129

Illusion Kanizsa Square


130/163

130

Problem

Fig.: fast detection difficult, longer time detection
http://c/thesis/rectangles/incvote.sh


131/163

131

Grouping and Incremental Indexing Incremental growing of search lines given aprocessing-time Image as index reduction of search space to imagespace Incremental detection of Gestalts (e.g., closures,proto-object)

Searchlines after 100ms Searchlines after 500ms

ExamplesExamples forfor ClosuresClosures andand CubesCubes


132/163

132

Results ofincremental indexing Specify onlyprocessing-time No thresholds Closures detected after 160, 240 and 420 ms

Cubesdetected

from

Eddy


133/163

133

VS2 Example

Vision System 2

tool

for

grouping


134/163

134

Finding Closures

Closed

Polygon

Any size Smaller ones

better Better ratio ofedgel

support

over perimeter

Ontology: Adjacency and Rule


135/163

135

Base

Adjacency map from segments/closures polygon relationships, e.g.,

2 (blue) is_StablySupported_by

1 (violet),

[barycenter

within x-limits]

3 (green) is_InstablySupported_by

2 (blue),

[barycenter too much right]4 (brown) is_right_of 1 (violet),...

Relationships

are

input

to reasoning engine

Finds concepts according to rules

2 rules

for

stack, one

for

arch, one

for

transitivity

Processing time typically less than a second(Java)

1

2

3

4

RealReal--WorldWorld EExamplexample


136/163

136

Segmentation

Some arches found:

Best from functionalp.o.v.Correct, yet irrelevant

from functionalp.o.v.


137/163

137

Scenario - Exit Test on door in office

378 segments, 86 closures,

35 stacks

Example

of two

columns:

Original image closures best arch window arch

Exit in XPERO Setup


138/163

138

p

Onotology operates

on

segments andclosures

Results

comparable

Exit

-

arch

Closing the loop Arch as relevant entity Learn object concepts from interaction

Occlusion Handling and Depth


139/163

139

Ordering

Perceptual Grouping Clustering based onGestalt principles

Improved Tracking

Interest point based Lucas-Kanade algorithm

Delaunay

representation

Local affine assumption

Spatial Reasoning

Occlusion Detection Occlusion Reasonin

Occlusion Handling


140/163

140

g

Only closures

T-junctions

indicate occlusion Occlusion indicates

depth ordering of closures More occlusion more closures ordered

Future work

Exploitation of motion Estimate plane normals


141/163

141

Conclusion Same method to detect different shapes Exploit local information (smoothness, ...) Avoid local decisions and early pruning ofhypotheses (thresholds) Only threshold: good ellipses Use ranking

Perceptual grouping = well-defined pruning ofhypotheses reduce search space Image Space Indexing: O(n2) O(n)


142/163

142

Further Work Difficult to find a good local quality measure Global measure: combinatorial searchproblem Interaction with scene,mobile observer:

false positives vs. stable percepts

Used to initialise tracking

^
http://c/tmp/office_movie.shhttp://c/tmp/office_movie.sh


143/163

143

Detection and Tracking

143Edges


144/163

144144

Arcs



145/163

145145

Detection



146/163

146146Tracking



147/163

147



148/163

148

Contents

Motivation: many

projects

Terminology: from computer to machine vision

Components: from

light to images

Machine Vision Segmentation

Edge detection


MatchObservations and

BRS

Execute Observe

BRS

TUWUVR

Hypothesisformation

AUPBRS

AUP

UVR

BRS

TUW

ntation


149/163

149

Innateknowledge

Evaluation of

experiment(s)

Design of

experiment(s)

Learning/Gaining Insights/

KnowledgeBase

A priori/existingknowledge

TUW UVR

ULJ

intelligent data collection/

experimental feedback(may not involve the generation

of new hypotheses)

Predictions

Educated Guess(time series analysis,reasoning by analogy)

feedback loopon the

actualexecution

of anexperiment

Observation of

experiment(s)

Planning andexecution ofexperiment(s)

TUW UVR

Plan Predict

UVR BRS

by analogy

ULJAUP

BRSULJ

BRSUVR

ULJTUW

TUW

UVR

BRSAUP

Anarchitecturefor

BRS

BRSUVR

BRSULJ

UVR

UVR

AUPULJ

ULJ

ULJ

UVR

AUP

BRSAUP

ULJTUW

TUW

TUW

TUW

UVR

learnin

gbyex

perimen

intelligent datacollection: reportlocation of robot

and object(s)

Showcase


150/163

150

Robot needs to know Where am I and Where are relevantentities (objects)

Robot pose

Egomotion relativeto environment Odometry, vision

Object pose Basic features todescribe

objects

Blobs: centre, size Sha e: cube, ...

Goal: observe object and agents


151/163

151

j

g

Robot observes experiments Start: overhead camera Goal: robot view 3rd 1st person

perspective Blobs shapes

Object Pose: Proto-object


152/163

152

Detection

Objective: detect

entities

relevant to experiment

Goal: abstract features to groups or objects Attributes such as size, location, shape Detection from grouping (VS3) Incremental indexing

Closures, Cube, cylinder, cone

Occlusion handling Tracking and depth ordering Ontology to label

Tracking and Grouping


153/163

153


Hypothesis

generation

from

tracked

Gestalts



154/163

154

ac g a d G oup g

Combining

infor-

mation over time Tracking through

degenerate views

Detecting Proto-Objects


155/163

155

Cube, Cylinder and Cone

Rule based grouping of Gestalts to obtain

hypotheses

Masking of proto-objects by the use of geometricfeatures from the hypothesis

(a) edge image

(b) closures (c) proto-objects

Detecting proto-objects from an edge image

XPERO - Ontology to Label Objects


156/163

156

Tackling last gap betweenfeatures (proto-objects) andsemantically meaningful(e.g.,in the sense of function)objects

Affords thoughts on thedefining substrate of objectconcepts spatial configuration of parts

Needs abstraction techniques arbitrary parts can take a roleina concept (e.g., being thesupport of another part)

Conclusion Machine Vision


157/163

157

System approach to solve a task (Passive) understanding scene is too hard Solving a particular task in a scenario is much easierto resolve Understanding features is under way E.g., structure, grouping, interest points, patches Segmentation in itself is impossible it depends on task Cameras cheap but processing difficult

Machine Vision Lessons Learned


158/163

158

Design vision system for task(s)

50% of solution

is

good illumination

Move out into the real world

Databases

and videos

are

nice

But reality is richer Avoid thresholds robustness is parameter-free A lot can be done with a camera and a PC

EU Project robots@home


159/163

159

j @

Mile stones 2007: Start 2008: Learn one room, touch screen 2009: Learn entire apartment 2010: Navigate in 4 apartments

Classify objects according to function (table, sofa, ...)

Using

stereo and

Time-of-flight cameras

Recognising Furniture


160/163

160

g g Using vanishing point and 3D Group using Gestalt principles surfaces

Some Detected Furniture


161/163

161

Thesis Topics


162/163

162

p

Ontology

for

furniture

Any-door detector and trying it in many homes

Shape

and shade

(Make3D)

Poke object and predict what will happen

Thank you


163/163

y

folien ss2009 machine vision

Documents