ifma – lasmea clermont-ferrand, france

IOCoViST – UJI / UMH International Online Course on Visual Servoing Techniques

September 17th – December 16th, 2005 Philippe Martinet


October 17th – December 16th, 2005

Professor Philippe Martinet

IFMA – LASMEA

Clermont-Ferrand, France

http://wwwlasmea.univ-bpclermont.fr/Philippe.Martinet/Welcome.html

Introduction to visual servoing

[email protected]

Philippe Martinet

2


September 17th – December 16th, 2005

Outline of the presentation

Basic concepts •Introduction •Classification

Modeling toolsComputer vision

History and Bibliography•Seventies and heighties•Nineties•Current research

Control •Kinematic control •Task function

Philippe Martinet

3



A set of basic notation

v=(v ω ) Kinematic Screw

e Task Function (Error function)

s Sensor signal (At each iteration)

iVjFrame change matrix for L

s* Sensor signal (At equilibrium)

L Interaction matrix

C Combination matrix

iTj

Homogeneous

Transformation matrix

itj = (tx,ty,tz) Translation vector

u = (ux,uy,uz) Rotation axle

θ Rotation angle

irj= θ.u Rotation vector

iRj=exp([irj]x) Rotation matrix

Philippe Martinet

4



Basic

concepts

History and

Bibliography

Modeling

toolsControl Conclusion

Basic concepts : Introduction

)()(

)(

..

)(

)(

2

1

tfts

th

th

th

n

===>

⎪⎪

⎭

⎪⎪

⎬

⎫

2D Visual Features

Low and

intermediate

levelCNA M

I

)(thReconstruction

(Model))(

)(

..

)(

)(

ts

ts

ts

ts

n

2

1

==>

⎪⎪

⎭

⎪⎪

⎬

⎫Low and

intermediate

levelCNA M

3D Visual Features

I

)(th

Reconstruction

(Model))(

)(

..

)(

)(

ts

ts

ts

ts

n

2

1

==>

⎪⎪

⎭

⎪⎪

⎬

⎫Low and

intermediate

levelCNA M

2D/3D Visual Features

I

e.v λ−=

v.s

Ls =&

( )*)(. stsCe −=

Philippe Martinet

5



Basic

concepts

History and

Bibliography

Modeling



x

y

z

Fo

x

y

z

Fc

x

y

z

Fa

⎟⎟⎠

⎞⎜⎜⎝

⎛=

c

c

c

ω

ν

vx

y

z

Fc

cω

cν

t

sLs

cs

δ

δ+= v.&

Interaction matrix

csLs v.=&

0t

s=

δ

δIf (fixed object)

( )tpss ,=

t

s

dt

dp

p

ss

dt

ds

δ

δ

δ

δ+== .

&

Case of embedded camera

Philippe Martinet

6



Basic

concepts

History and

Bibliography

Modeling



Features

Extraction

)(IhVisual feedback

Sensor of 2D features

Controller

Power

Joint feedback

q&

inverse

Jacobian

v

)(ts

Control

law

*s

-+

( )*)(..v stsLs

−−=

+

λ

Allen, Andreff, Asada, Berry, Cervera, Chaumette, Christensen, Collewet, Corke,Cowan,Cretual, Crowley, Daney, Debain,

Degushi, Devy, Dornaika, Espiau, Feddema, Gangloff, Ginoux, Hager, Hamel, Hashimoto, Horaud, Hutchinson,

Jagersand, Kanade, Kelly, Khosla, Khadraoui, Kragic, Iwatsuki, Lee, Malis, Marchand, Martinet, Mezouar, Motyl,

Myasaki, Nelson, Hosoda, Papanikolopoulos, Piepmier, Pissard-Gibollet, Rives, Sanderson, Soueres, Swain, Urban,

Weiss, ….. (non exhaustive)

IBVSImage Based

Visual Servoing

I

Philippe Martinet

7



Features

Extraction

)(IhVisual feedbackPose

Estimation


Basic

concepts

History and

Bibliography

Modeling



Daucher, Dhome, Grosso, Krupa, Martinet, Malis, Morel, Rizzi, Sandini, Sharifi, Siciliano, Wilson, Zanne, ….. (non

exhaustive)

PBVSPosition Based

Visual Servoing

Controller

Power

Joint feedback

q&

Inverse

Jacobian

v

( )*)(..v stsLs

−−=

+

λ

Control

law

)(ts

*s-+

I

Philippe Martinet

8



Features

Extraction


Estimation


Basic

concepts

History and

Bibliography

Modeling



Andreff, Basri, Chaumette, Chesi, Daney, Degushi, Dixon, Martinet, Malis, Mahony, Morel, Ostrowski, Taylor….. (non

exhaustive)

HBVSHybrid 2D/3D Based

Visual Servoing

Controller

Power

Joint feedback

q&

Inverse

Jacobian

v

( )*)(..v stsLs

−−=

+

λ

Control

law

)(ts

*s-+

I

Philippe Martinet

9



Basic

concepts

History and

Bibliography

Modeling


History and Bibliography : Seventies and heighties

G.J. Agin in 79 Embedded camera

Bolles and Paul in 73 Shirai and Inoue in 73

Fixed cameraAssembly tasks« Look and move »

Sanderson and Weiss 87 Classification

Espiau 87 Sensor based control

Samson, Espiau 89 Task function approach

(non exhaustive bibliography)

Feddema 89 Feature based trajectorygeneration

…….

Philippe Martinet

10



Basic

concepts

History and

Bibliography

Modeling


History and Bibliography : Nineties

Sandini 96, Wilson 96 Martinet 98-99

Position based visual servoing

Chaumette 90, 92 Visual servoing

Malis 97 2D+1/2 visual servoing

Corke 94 Dynamic visual servoing

Maru 93 Stereo

De Schutter 97 Morel/Malis 98

Force/vision coupling

Chaumette 98 Local minima


Motyl 93-96 Camera/laser stripe sensor

Cretual 98 2D+dt visual servoing

Corke 93, Gangloff 96, Hashimoto 96 Bensalah 96, Piepmeir 02

Target tracking

…….

Philippe Martinet

11



Basic

concepts

History and

Bibliography

Modeling


History and Bibliography : Current research (2000-2004)

Mezouar 00-02 Trajectory generation

Kragic 02 Robust visual servoing

Martinet 02 , Siciliano 02 Wilson 02

Position based visual servoing

Malis 02 Invariant visual servoing

Corke, Hutchinson 02 Partitioned visual servoing

Malis 00, Martinet/Cervera 01-02

Multi-camera , Stereo


Philippe Martinet

12



Basic

concepts

History and

Bibliography

Modeling


History and Bibliography : Current research (2000-2004)(non exhaustive bibliography)

Horaud 02, Mezouar 04 Central catadioptric Cameras

Chesi 03, Corke 02, Malis 99

Mezouar 02, Morel 03Thuilot 02

Field of view

Malis 04 Efficient Second order control

Malis 03 Stability pb in depth distribution

Non exhaustive Bibliography : see the web site for a longer one

Philippe Martinet

13



Basic

concepts

History and

Bibliography

Modeling


Modeling tools : Computer vision Pinhole cameraPinhole camera

Normalized

coordinates

Z

Xx

n=

Z

Yyn=

Philippe Martinet

14



Basic

concepts

History and

Bibliography

Modeling


Modeling tools : Computer vision Pinhole cameraPinhole camera

⎥⎥⎥

⎦

⎤

⎢⎢⎢

⎣

⎡

⋅

⋅

=

100

00

0

vrf

usff

K

Intrinsic paramaters:

f : focal length

s : skew

r : aspect ratio

u0,v

0: principal point

Philippe Martinet

15



Basic

concepts

History and

Bibliography

Modeling


Modeling tools : Computer vision Projection MatrixProjection Matrix

Philippe Martinet

16



Basic

concepts

History and

Bibliography

Modeling


Modeling tools : Computer vision Camera calibrationCamera calibration

Philippe Martinet

17



Basic

concepts

History and

Bibliography

Modeling


Modeling tools : Computer vision Pose estimationPose estimation

Philippe Martinet

18



Basic

concepts

History and

Bibliography

Modeling


Control : Kinematic control

pqJq && ⋅=

− )(1

⎟⎟⎠

⎞⎜⎜⎝

⎛

∂

∂

∂

∂

∂

∂

∂

∂=

− n

k

n

kkk

kq

p

q

p

q

p

q

pqJ

121

)(

J(q) represents the robot jacobian

The kth line is given by :

Use of Kinematic model

)(ts

Filter

h


Estimation

Controller

Power

Joint feedback

q&

Inverse

Jacobian

Control

law

p*s -

+

.

I

Philippe Martinet

19



Basic

concepts

History and

Bibliography

Modeling


Control : Task function

p represents the position/orientation of the robot end effector

Simplified modeling of the loop Integration due to the sensor

pp&

q&

( )qJ1−

SensorRobot

System

Velocity control basis

Modeling of the sensor by the DGM

Using the current state of the robot

q&

IKM DGM1/p

1/p

pp&

pp&

Philippe Martinet

20



Basic

concepts

History and

Bibliography

Modeling


Control : Task function Velocity control basis

Simplified modeling of the loop

e

C

p

p* +-

p&

1/p p

Error Function ppe −=

*

Proportional control law λ=C ( )ppep −==

*

.. λλ&

pe && −=

ee .λ−=&

Exponential decrease of the

Error Function

Philippe Martinet

21



Basic

concepts

History and

Bibliography

Modeling


Control : Task function Main concepts

The task function characterizes the robotic

task to be performed and allows to establish

a virtual link between the sensor and the

environment.

q*(t) represents a desired trajectory in the joint space

The robotic task can be described as

a regulation to zero of a task function

Samson, Espiau

End of 80

( )tqe ,

( ) ( ) ( )tqtqtqe*

, −=

Some task function :

( ) ( ) ( )tpqptqe*

, −=

( ) ( ) *, dqdtqe −=

p*(t) a desired trajectory in the operational space

(i.e cartesian space for a robot with 6 d.o.f.)

d* represents a desired distance between the

object and the end effector

Philippe Martinet

22



Basic

concepts

History and

Bibliography

Modeling



C represents a combination matrix

(dim = m x k, full rank m); It allows

to take into account more sensor

informations than necessary to

perform the robot control.

When the sensor deliver the information s and dim(s)= k , the task function to be regulated can written like:

km ≤)dim(qn =( ) ( )( )*

1,., stpsCtpe −=

m x 1 k x 1 m x k

)dim(em =

)dim(sk =

nm ≤

If s is well chosen, the m components

of e1

are independent and allow to

control m d.o.f. (necessary to

perform the robotic task)

Philippe Martinet

23



Basic

concepts

History and

Bibliography

Modeling



Task redondancy

If the interaction

matrix is not

a full rank matrix q

s

δ

δ

p

s

δ

δou ≥

⎪⎭

⎪⎬

⎫

)dim(

)dim(

q

ou

p Number of

independent

components of s

One hybrid task can be defined :

- one primary task e1

(maintain an interaction constraint

during the execution of the task)

- one secondary task (minimize a cost function hs)

T

sT

st

hg ⎟

⎟⎠

⎞⎜⎜⎝

⎛=

δ

δGradient of the

Cost function hs

Philippe Martinet

24



Basic

concepts

History and

Bibliography

Modeling



( ) T

smgWWIeWe ...

1

++

−+= γ

Represent two operators which guarantee that the induced motions

due to the secondary task (included in the kernel of W) are

compatible with the convergence of s to s* (primary task)

Global task functionWWI

m

+

−

γ

( ) ( )s

LW kerker =

A represents a full rank matrix as :

Orthogonal projector on

the kernel of W

gain allowing to tune the

preponderance of the primary

task in regard with the

secondary task

WWIm

+

−

+

W

Philippe Martinet

25



Basic

concepts

History and

Bibliography

Modeling



( ) ( )( )*,., strsCtre1

−=

( )( )*,..v strsLsc

−−=

+

λ

0t

e1=

δ

δFixed object

Task function

Without a secondary task With a secondary task

Task function

Condition of convergence

Control law

+

=s

LC

Condition of convergence

Control law

+

=s

LWC .

( )t

gWWItre

T

s

mcδ

δγλ ...),(.v +

−−−=

( ) T

smgWWIeWe ...

1

++

−+= γ

Philippe Martinet

26



Basic

concepts

History and

Bibliography

Modeling


Conclusion

Different kind of works :

• Application of known methods

• Innovation (new methods)

• Application fields

Real world objective (thanks to image processing improvements)

A cooking approach : spherical, cylindrical, mixture…

A large number of robot applications

Research community is growing (thanks for all of us)


September 17th – December 16th, 2005 Philippe Martinet

Professor Philippe Martinet

IFMA – LASMEA

Clermont-Ferrand, France

http://wwwlasmea.univ-bpclermont.fr/Philippe.Martinet/Welcome.html

Applications in visual servoing

[email protected]


October 17th – December 16th, 2005

Philippe Martinet

28




Manipulator robot

Visual servoing : 2D and 3D approach, Stereo

Force/Vision control

Mobile Robot

Automatic Guided vehicule in agricultural context.

Automatic Guided vehicule in indoor context.

Philippe Martinet

29



3D … 3D Visual servoing

Positioning task face to an unknown object

Positioning task

Dhome, Jurie

Philippe Martinet

30



ST … Stereo Visual servoing

Cervera [99,01,02,03]

Comparaison study :

3D point

2D Stereo point

2D image point + depth

2D image point + disparity

Case study for grasping:

Oriented blob

Philippe Martinet

31



FV … Force and vision coupling

Robot control

By vision/force

coupling

LASMEA

IFMA-SOCRATES

External Position/force

External Vision/force

Hybrid Position Force

Hybrid Vision Force

Visual servoing

Force feedback Control

Hybrid Vision Force : case studyDone by M. Prats

from Castellon, Spain

Philippe Martinet

32




Manipulator robot

Visual servoing : 2D and 3D approach, Stereo

Force/Vision control

Mobile Robot

Automatic Guided vehicule in agricultural context.

Automatic Guided vehicule in indoor context.

Philippe Martinet

33



2D … Automatic Guided Vehicles

Debain [96]

Control laws for agricultural machines

Combine-harvester

Harvesting work

Philippe Martinet

34



2D … Automatic Guided Vehicles

Debain [96] Control

Guiding on a sloping ground

Philippe Martinet

35



PTZ

Controlled

Camera

Our Vision

sensorFixed focal

length camera

Wide angle

ActuatorsSensor Decision Module

2D … Automatic Guided Vehicles : target tracking

Clady [02]

Philippe Martinet

36



2D … Automatic Guided Vehicles : target tracking

Clady [02]

Philippe Martinet

37



2D … Automatic Guided Vehicles : visual memory

Blanc [04] Ait Ader[04]

Navigation using visual memory

Philippe Martinet

38



2D … Automatic Guided Vehicles : visual memory

Blanc [04] Ait Ader[04]

Navigation using visual memory

ifma – lasmea clermont-ferrand, france

Documents