Download - IFMA – LASMEA Clermont-Ferrand, France
IOCoViST – UJI / UMH International Online Course on Visual Servoing Techniques
September 17th – December 16th, 2005 Philippe Martinet
IOCoViST – UJI / UMH International Online Course on Visual Servoing Techniques
October 17th – December 16th, 2005
Professor Philippe Martinet
IFMA – LASMEA
Clermont-Ferrand, France
http://wwwlasmea.univ-bpclermont.fr/Philippe.Martinet/Welcome.html
Introduction to visual servoing
Philippe Martinet
2
IOCoViST – UJI / UMH International Online Course on Visual Servoing Techniques
September 17th – December 16th, 2005
Outline of the presentation
Basic concepts •Introduction •Classification
Modeling toolsComputer vision
History and Bibliography•Seventies and heighties•Nineties•Current research
Control •Kinematic control •Task function
Philippe Martinet
3
IOCoViST – UJI / UMH International Online Course on Visual Servoing Techniques
September 17th – December 16th, 2005
A set of basic notation
v=(v ω ) Kinematic Screw
e Task Function (Error function)
s Sensor signal (At each iteration)
iVjFrame change matrix for L
s* Sensor signal (At equilibrium)
L Interaction matrix
C Combination matrix
iTj
Homogeneous
Transformation matrix
itj = (tx,ty,tz) Translation vector
u = (ux,uy,uz) Rotation axle
θ Rotation angle
irj= θ.u Rotation vector
iRj=exp([irj]x) Rotation matrix
Philippe Martinet
4
IOCoViST – UJI / UMH International Online Course on Visual Servoing Techniques
September 17th – December 16th, 2005
Basic
concepts
History and
Bibliography
Modeling
toolsControl Conclusion
Basic concepts : Introduction
)()(
)(
..
)(
)(
2
1
tfts
th
th
th
n
===>
⎪⎪
⎭
⎪⎪
⎬
⎫
2D Visual Features
Low and
intermediate
levelCNA M
I
)(thReconstruction
(Model))(
)(
..
)(
)(
ts
ts
ts
ts
n
2
1
==>
⎪⎪
⎭
⎪⎪
⎬
⎫Low and
intermediate
levelCNA M
3D Visual Features
I
)(th
Reconstruction
(Model))(
)(
..
)(
)(
ts
ts
ts
ts
n
2
1
==>
⎪⎪
⎭
⎪⎪
⎬
⎫Low and
intermediate
levelCNA M
2D/3D Visual Features
I
e.v λ−=
v.s
Ls =&
( )*)(. stsCe −=
Philippe Martinet
5
IOCoViST – UJI / UMH International Online Course on Visual Servoing Techniques
September 17th – December 16th, 2005
Basic
concepts
History and
Bibliography
Modeling
toolsControl Conclusion
Basic concepts : Introduction
x
y
z
Fo
x
y
z
Fc
x
y
z
Fa
⎟⎟⎠
⎞⎜⎜⎝
⎛=
c
c
c
ω
ν
vx
y
z
Fc
cω
cν
t
sLs
cs
δ
δ+= v.&
Interaction matrix
csLs v.=&
0t
s=
δ
δIf (fixed object)
( )tpss ,=
t
s
dt
dp
p
ss
dt
ds
δ
δ
δ
δ+== .
&
Case of embedded camera
Philippe Martinet
6
IOCoViST – UJI / UMH International Online Course on Visual Servoing Techniques
September 17th – December 16th, 2005
Basic
concepts
History and
Bibliography
Modeling
toolsControl Conclusion
Basic concepts : Introduction
Features
Extraction
)(IhVisual feedback
Sensor of 2D features
Controller
Power
Joint feedback
q&
inverse
Jacobian
v
)(ts
Control
law
*s
-+
( )*)(..v stsLs
−−=
+
λ
Allen, Andreff, Asada, Berry, Cervera, Chaumette, Christensen, Collewet, Corke,Cowan,Cretual, Crowley, Daney, Debain,
Degushi, Devy, Dornaika, Espiau, Feddema, Gangloff, Ginoux, Hager, Hamel, Hashimoto, Horaud, Hutchinson,
Jagersand, Kanade, Kelly, Khosla, Khadraoui, Kragic, Iwatsuki, Lee, Malis, Marchand, Martinet, Mezouar, Motyl,
Myasaki, Nelson, Hosoda, Papanikolopoulos, Piepmier, Pissard-Gibollet, Rives, Sanderson, Soueres, Swain, Urban,
Weiss, ….. (non exhaustive)
IBVSImage Based
Visual Servoing
I
Philippe Martinet
7
IOCoViST – UJI / UMH International Online Course on Visual Servoing Techniques
September 17th – December 16th, 2005
Features
Extraction
)(IhVisual feedbackPose
Estimation
Sensor of 3D features
Basic
concepts
History and
Bibliography
Modeling
toolsControl Conclusion
Basic concepts : Introduction
Daucher, Dhome, Grosso, Krupa, Martinet, Malis, Morel, Rizzi, Sandini, Sharifi, Siciliano, Wilson, Zanne, ….. (non
exhaustive)
PBVSPosition Based
Visual Servoing
Controller
Power
Joint feedback
q&
Inverse
Jacobian
v
( )*)(..v stsLs
−−=
+
λ
Control
law
)(ts
*s-+
I
Philippe Martinet
8
IOCoViST – UJI / UMH International Online Course on Visual Servoing Techniques
September 17th – December 16th, 2005
Features
Extraction
)(IhVisual feedbackPose
Estimation
Sensor of 3D features
Basic
concepts
History and
Bibliography
Modeling
toolsControl Conclusion
Basic concepts : Introduction
Andreff, Basri, Chaumette, Chesi, Daney, Degushi, Dixon, Martinet, Malis, Mahony, Morel, Ostrowski, Taylor….. (non
exhaustive)
HBVSHybrid 2D/3D Based
Visual Servoing
Controller
Power
Joint feedback
q&
Inverse
Jacobian
v
( )*)(..v stsLs
−−=
+
λ
Control
law
)(ts
*s-+
I
Philippe Martinet
9
IOCoViST – UJI / UMH International Online Course on Visual Servoing Techniques
September 17th – December 16th, 2005
Basic
concepts
History and
Bibliography
Modeling
toolsControl Conclusion
History and Bibliography : Seventies and heighties
G.J. Agin in 79 Embedded camera
Bolles and Paul in 73 Shirai and Inoue in 73
Fixed cameraAssembly tasks« Look and move »
Sanderson and Weiss 87 Classification
Espiau 87 Sensor based control
Samson, Espiau 89 Task function approach
(non exhaustive bibliography)
Feddema 89 Feature based trajectorygeneration
…….
Philippe Martinet
10
IOCoViST – UJI / UMH International Online Course on Visual Servoing Techniques
September 17th – December 16th, 2005
Basic
concepts
History and
Bibliography
Modeling
toolsControl Conclusion
History and Bibliography : Nineties
Sandini 96, Wilson 96 Martinet 98-99
Position based visual servoing
Chaumette 90, 92 Visual servoing
Malis 97 2D+1/2 visual servoing
Corke 94 Dynamic visual servoing
Maru 93 Stereo
De Schutter 97 Morel/Malis 98
Force/vision coupling
Chaumette 98 Local minima
(non exhaustive bibliography)
Motyl 93-96 Camera/laser stripe sensor
Cretual 98 2D+dt visual servoing
Corke 93, Gangloff 96, Hashimoto 96 Bensalah 96, Piepmeir 02
Target tracking
…….
Philippe Martinet
11
IOCoViST – UJI / UMH International Online Course on Visual Servoing Techniques
September 17th – December 16th, 2005
Basic
concepts
History and
Bibliography
Modeling
toolsControl Conclusion
History and Bibliography : Current research (2000-2004)
Mezouar 00-02 Trajectory generation
Kragic 02 Robust visual servoing
Martinet 02 , Siciliano 02 Wilson 02
Position based visual servoing
Malis 02 Invariant visual servoing
Corke, Hutchinson 02 Partitioned visual servoing
Malis 00, Martinet/Cervera 01-02
Multi-camera , Stereo
(non exhaustive bibliography)
Philippe Martinet
12
IOCoViST – UJI / UMH International Online Course on Visual Servoing Techniques
September 17th – December 16th, 2005
Basic
concepts
History and
Bibliography
Modeling
toolsControl Conclusion
History and Bibliography : Current research (2000-2004)(non exhaustive bibliography)
Horaud 02, Mezouar 04 Central catadioptric Cameras
Chesi 03, Corke 02, Malis 99
Mezouar 02, Morel 03Thuilot 02
Field of view
Malis 04 Efficient Second order control
Malis 03 Stability pb in depth distribution
Non exhaustive Bibliography : see the web site for a longer one
Philippe Martinet
13
IOCoViST – UJI / UMH International Online Course on Visual Servoing Techniques
September 17th – December 16th, 2005
Basic
concepts
History and
Bibliography
Modeling
toolsControl Conclusion
Modeling tools : Computer vision Pinhole cameraPinhole camera
Normalized
coordinates
Z
Xx
n=
Z
Yyn=
Philippe Martinet
14
IOCoViST – UJI / UMH International Online Course on Visual Servoing Techniques
September 17th – December 16th, 2005
Basic
concepts
History and
Bibliography
Modeling
toolsControl Conclusion
Modeling tools : Computer vision Pinhole cameraPinhole camera
⎥⎥⎥
⎦
⎤
⎢⎢⎢
⎣
⎡
⋅
⋅
=
100
00
0
vrf
usff
K
Intrinsic paramaters:
f : focal length
s : skew
r : aspect ratio
u0,v
0: principal point
Philippe Martinet
15
IOCoViST – UJI / UMH International Online Course on Visual Servoing Techniques
September 17th – December 16th, 2005
Basic
concepts
History and
Bibliography
Modeling
toolsControl Conclusion
Modeling tools : Computer vision Projection MatrixProjection Matrix
Philippe Martinet
16
IOCoViST – UJI / UMH International Online Course on Visual Servoing Techniques
September 17th – December 16th, 2005
Basic
concepts
History and
Bibliography
Modeling
toolsControl Conclusion
Modeling tools : Computer vision Camera calibrationCamera calibration
Philippe Martinet
17
IOCoViST – UJI / UMH International Online Course on Visual Servoing Techniques
September 17th – December 16th, 2005
Basic
concepts
History and
Bibliography
Modeling
toolsControl Conclusion
Modeling tools : Computer vision Pose estimationPose estimation
Philippe Martinet
18
IOCoViST – UJI / UMH International Online Course on Visual Servoing Techniques
September 17th – December 16th, 2005
Basic
concepts
History and
Bibliography
Modeling
toolsControl Conclusion
Control : Kinematic control
pqJq && ⋅=
− )(1
⎟⎟⎠
⎞⎜⎜⎝
⎛
∂
∂
∂
∂
∂
∂
∂
∂=
− n
k
n
kkk
kq
p
q
p
q
p
q
pqJ
121
)(
J(q) represents the robot jacobian
The kth line is given by :
Use of Kinematic model
)(ts
Filter
h
)(IhVisual feedbackPose
Estimation
Controller
Power
Joint feedback
q&
Inverse
Jacobian
Control
law
p*s -
+
.
I
Philippe Martinet
19
IOCoViST – UJI / UMH International Online Course on Visual Servoing Techniques
September 17th – December 16th, 2005
Basic
concepts
History and
Bibliography
Modeling
toolsControl Conclusion
Control : Task function
p represents the position/orientation of the robot end effector
Simplified modeling of the loop Integration due to the sensor
pp&
q&
( )qJ1−
SensorRobot
System
Velocity control basis
Modeling of the sensor by the DGM
Using the current state of the robot
q&
IKM DGM1/p
1/p
pp&
pp&
Philippe Martinet
20
IOCoViST – UJI / UMH International Online Course on Visual Servoing Techniques
September 17th – December 16th, 2005
Basic
concepts
History and
Bibliography
Modeling
toolsControl Conclusion
Control : Task function Velocity control basis
Simplified modeling of the loop
e
C
p
p* +-
p&
1/p p
Error Function ppe −=
*
Proportional control law λ=C ( )ppep −==
*
.. λλ&
pe && −=
ee .λ−=&
Exponential decrease of the
Error Function
Philippe Martinet
21
IOCoViST – UJI / UMH International Online Course on Visual Servoing Techniques
September 17th – December 16th, 2005
Basic
concepts
History and
Bibliography
Modeling
toolsControl Conclusion
Control : Task function Main concepts
The task function characterizes the robotic
task to be performed and allows to establish
a virtual link between the sensor and the
environment.
q*(t) represents a desired trajectory in the joint space
The robotic task can be described as
a regulation to zero of a task function
Samson, Espiau
End of 80
( )tqe ,
( ) ( ) ( )tqtqtqe*
, −=
Some task function :
( ) ( ) ( )tpqptqe*
, −=
( ) ( ) *, dqdtqe −=
p*(t) a desired trajectory in the operational space
(i.e cartesian space for a robot with 6 d.o.f.)
d* represents a desired distance between the
object and the end effector
Philippe Martinet
22
IOCoViST – UJI / UMH International Online Course on Visual Servoing Techniques
September 17th – December 16th, 2005
Basic
concepts
History and
Bibliography
Modeling
toolsControl Conclusion
Control : Task function Main concepts
C represents a combination matrix
(dim = m x k, full rank m); It allows
to take into account more sensor
informations than necessary to
perform the robot control.
When the sensor deliver the information s and dim(s)= k , the task function to be regulated can written like:
km ≤)dim(qn =( ) ( )( )*
1,., stpsCtpe −=
m x 1 k x 1 m x k
)dim(em =
)dim(sk =
nm ≤
If s is well chosen, the m components
of e1
are independent and allow to
control m d.o.f. (necessary to
perform the robotic task)
Philippe Martinet
23
IOCoViST – UJI / UMH International Online Course on Visual Servoing Techniques
September 17th – December 16th, 2005
Basic
concepts
History and
Bibliography
Modeling
toolsControl Conclusion
Control : Task function Main concepts
Task redondancy
If the interaction
matrix is not
a full rank matrix q
s
δ
δ
p
s
δ
δou ≥
⎪⎭
⎪⎬
⎫
)dim(
)dim(
q
ou
p Number of
independent
components of s
One hybrid task can be defined :
- one primary task e1
(maintain an interaction constraint
during the execution of the task)
- one secondary task (minimize a cost function hs)
T
sT
st
hg ⎟
⎟⎠
⎞⎜⎜⎝
⎛=
δ
δGradient of the
Cost function hs
Philippe Martinet
24
IOCoViST – UJI / UMH International Online Course on Visual Servoing Techniques
September 17th – December 16th, 2005
Basic
concepts
History and
Bibliography
Modeling
toolsControl Conclusion
Control : Task function Main concepts
( ) T
smgWWIeWe ...
1
++
−+= γ
Represent two operators which guarantee that the induced motions
due to the secondary task (included in the kernel of W) are
compatible with the convergence of s to s* (primary task)
Global task functionWWI
m
+
−
γ
( ) ( )s
LW kerker =
A represents a full rank matrix as :
Orthogonal projector on
the kernel of W
gain allowing to tune the
preponderance of the primary
task in regard with the
secondary task
WWIm
+
−
+
W
Philippe Martinet
25
IOCoViST – UJI / UMH International Online Course on Visual Servoing Techniques
September 17th – December 16th, 2005
Basic
concepts
History and
Bibliography
Modeling
toolsControl Conclusion
Control : Task function Main concepts
( ) ( )( )*,., strsCtre1
−=
( )( )*,..v strsLsc
−−=
+
λ
0t
e1=
δ
δFixed object
Task function
Without a secondary task With a secondary task
Task function
Condition of convergence
Control law
+
=s
LC
Condition of convergence
Control law
+
=s
LWC .
( )t
gWWItre
T
s
mcδ
δγλ ...),(.v +
−−−=
( ) T
smgWWIeWe ...
1
++
−+= γ
Philippe Martinet
26
IOCoViST – UJI / UMH International Online Course on Visual Servoing Techniques
September 17th – December 16th, 2005
Basic
concepts
History and
Bibliography
Modeling
toolsControl Conclusion
Conclusion
Different kind of works :
• Application of known methods
• Innovation (new methods)
• Application fields
Real world objective (thanks to image processing improvements)
A cooking approach : spherical, cylindrical, mixture…
A large number of robot applications
Research community is growing (thanks for all of us)
IOCoViST – UJI / UMH International Online Course on Visual Servoing Techniques
September 17th – December 16th, 2005 Philippe Martinet
Professor Philippe Martinet
IFMA – LASMEA
Clermont-Ferrand, France
http://wwwlasmea.univ-bpclermont.fr/Philippe.Martinet/Welcome.html
Applications in visual servoing
IOCoViST – UJI / UMH International Online Course on Visual Servoing Techniques
October 17th – December 16th, 2005
Philippe Martinet
28
IOCoViST – UJI / UMH International Online Course on Visual Servoing Techniques
September 17th – December 16th, 2005
Outline of the presentation
Manipulator robot
Visual servoing : 2D and 3D approach, Stereo
Force/Vision control
Mobile Robot
Automatic Guided vehicule in agricultural context.
Automatic Guided vehicule in indoor context.
Philippe Martinet
29
IOCoViST – UJI / UMH International Online Course on Visual Servoing Techniques
September 17th – December 16th, 2005
3D … 3D Visual servoing
Positioning task face to an unknown object
Positioning task
Dhome, Jurie
Philippe Martinet
30
IOCoViST – UJI / UMH International Online Course on Visual Servoing Techniques
September 17th – December 16th, 2005
ST … Stereo Visual servoing
Cervera [99,01,02,03]
Comparaison study :
3D point
2D Stereo point
2D image point + depth
2D image point + disparity
Case study for grasping:
Oriented blob
Philippe Martinet
31
IOCoViST – UJI / UMH International Online Course on Visual Servoing Techniques
September 17th – December 16th, 2005
FV … Force and vision coupling
Robot control
By vision/force
coupling
LASMEA
IFMA-SOCRATES
External Position/force
External Vision/force
Hybrid Position Force
Hybrid Vision Force
Visual servoing
Force feedback Control
Hybrid Vision Force : case studyDone by M. Prats
from Castellon, Spain
Philippe Martinet
32
IOCoViST – UJI / UMH International Online Course on Visual Servoing Techniques
September 17th – December 16th, 2005
Outline of the presentation
Manipulator robot
Visual servoing : 2D and 3D approach, Stereo
Force/Vision control
Mobile Robot
Automatic Guided vehicule in agricultural context.
Automatic Guided vehicule in indoor context.
Philippe Martinet
33
IOCoViST – UJI / UMH International Online Course on Visual Servoing Techniques
September 17th – December 16th, 2005
2D … Automatic Guided Vehicles
Debain [96]
Control laws for agricultural machines
Combine-harvester
Harvesting work
Philippe Martinet
34
IOCoViST – UJI / UMH International Online Course on Visual Servoing Techniques
September 17th – December 16th, 2005
2D … Automatic Guided Vehicles
Debain [96] Control
Guiding on a sloping ground
Philippe Martinet
35
IOCoViST – UJI / UMH International Online Course on Visual Servoing Techniques
September 17th – December 16th, 2005
PTZ
Controlled
Camera
Our Vision
sensorFixed focal
length camera
Wide angle
ActuatorsSensor Decision Module
2D … Automatic Guided Vehicles : target tracking
Clady [02]
Philippe Martinet
36
IOCoViST – UJI / UMH International Online Course on Visual Servoing Techniques
September 17th – December 16th, 2005
2D … Automatic Guided Vehicles : target tracking
Clady [02]
Philippe Martinet
37
IOCoViST – UJI / UMH International Online Course on Visual Servoing Techniques
September 17th – December 16th, 2005
2D … Automatic Guided Vehicles : visual memory
Blanc [04] Ait Ader[04]
Navigation using visual memory
Philippe Martinet
38
IOCoViST – UJI / UMH International Online Course on Visual Servoing Techniques
September 17th – December 16th, 2005
2D … Automatic Guided Vehicles : visual memory
Blanc [04] Ait Ader[04]
Navigation using visual memory