visual servo control using stereo image jacobian - saetang

8/22/2019 Visual Servo Control Using Stereo Image Jacobian - Saetang

1/5

Visual Servo Control using Stereo Image Jacobian

ARSWIN SAETANG and ARTHIT SRIKAEWRobotics and Automation Research Unit for Real-World Applications (RARA)

School of Electrical Engineering

Suranaree University of Technology111 University Ave., Suranaree Subdistrict, Muang District, Nakhon Ratchasima 30000

THAILAND

Abstract: - Visual servo is a method to control robot manipulator using both image and robot information. Inthis paper, stereo image Jacobian has been developed and applied to control a 5-DOF robot manipulator withtwo cameras mounted on it. The proposed dynamic estimation of stereo image Jacobian allows the system toachieve visual servoing without any calibration of the camera or robot and any robot kinematics. The robottrajectory can then be planned online which provides the system to work in an undetermined environment. The

proposed technique can also be used to directly drive motors of the robot joints.

Key-words: -Visual Servo, robot manipulator control, calibration, stereo image jacobian

1 IntroductionPresently, human-robot interaction has becomemore varied. In order for the robot to interact withhuman accurately, it must relatively know physical

positions of the target being interacted with. Themethod to acknowledge such positions can bevaried such as using infrared or laser device withadditional signal processing. Using such sensorshas some disadvantages and limitations, especiallywhen human directly involves. The development ofusing visual servo which employs computer visionfor target localization is then one of the mostsuitable methods.

Visual servo is mainly composed ofrobotics and computer vision. It was firstlyintroduced in [1]. In early 80s, all related work ofvisual feedback can be categorized into 4 groups ofmethods: look and move, dynamic look and move,

position and image based visual servo. In 1987, [3]proposed the method to directly use object

parameters from image without having to computeactual positions of the target. This work, however,was only presented in mathematical model andlimited to only 3-DOF system. Later on, [4]

presented the use of digital camera to track movingobject via adaptive control. In 1996, all relatedconcept of visual servo was summarized in [5].

Nowadays, there are various visual servo methods,e.g. cooperation of eye-in-hand and eye-to-handcameras for visual servoing [6][7], dynamic visualservo [8][9], visual servo together with artificialintelligent techniques [10][11] etc.

Image Jacobian-based visual servo has beenintroduced in 1984 by [2]. The most interesting ofusing this method is the ability to directly use image

parameters without having to calculate actualpositions of the target in which the end effector ismoving toward. The relationship between the image

parameters and the end effector has been calledimage Jacobian by [12] and has been widely usedever since [13][14]. This work presents the online

trajectory planning for a robot manipulator. Theobjective is to develop a robot system to work in anundetermined environment without having tocalibrate the system. The purposed method is basedon visual servo technique which does not require a

priori knowledge about robot kinematics. Thesystem has been able to apply stereo image Jacobianfor controlling robot manipulator to grab singleobject efficiently.

2 Image JacobianRobot control can be defined in term of parametersof end-effector position in Cartesian space. Thechanges in robot joints relate to robot Jacobian [15].These cause corresponding changes in image

parameters which relate to image Jacobian [5]. Let arobot manipulator has n joints and n DOF. Thecontrol target is defined in term of m image

parameters. Let v

be n -joint position vector andrvbe a robot end-effector position vector with size of

p . Let fv

be m -image-parameter vector. The

Proceedings of the 8th WSEAS Int. Conference on Automatic Control, Modeling and Simulation, Prague, Czech Republic, March 12-14, 2006 (pp1-5)


2/5

relationship of robot joint velocities

and end-effector velocity r& can be described by

= Jr (1)where J is the robot Jacobian and

=

n

pp

n

rr

rr

J

L

MOM

L

1

1

1

1

(2)

Changes in robot end-effector

r provide changes

in image parameters

f by the relationship

= rJf r& (3)

where rJ is pm matrix and

=

p

mm

p

r

r

f

r

f

r

f

r

f

J

L

MOM

L

1

1

1

1

(4)

Consequently, relationship between f& and & can

be related as image Jacobian qJ which is

&& = qJf (5)

where JJJ rq = and

=

n

mm

n

q

ff

ff

J

L

MOM

L

1

1

1

1

(6)

2.1 Dynamic Image Jacobian EstimationImage Jacobian can be dynamically estimated fromchanges of robot joints and changes of image

parameters using linear relationship in (5). Thus, a

joint motion )( that reduces the image error can

be computed by

fJq =1

(7)

Consider image parameteri

f and joint

vectori

at time index i . Changes of image

parameters can be computed by1= iii ffdf

and changes of robot joints can be obtained by 1= iiid . Both can be used for image

Jacobian calculation. In order to be able to controlrobot joints using estimated Jacobian, a number of

image parameters must be greater than or equal to anumber of robot joints, i.e. mn . There are n pairsof image parameters in total and robot joints defined

by

]),(,,),[( 11 iinini dfddfd L++ (8)

Then nn matrix DQ and nm matrix DF can bedefined by

[ ]ini ddDQ L1+=

=+

+

i

n

ni

n

ini

dd

dd

L

MOM

L

1

1

1

1

(9)

inidfdfDF L

1+=

=+

+

i

m

ni

m

ini

dfdf

dfdf

L

MOM

L

1

1

1

1

(10)

where [ ]

Ti

n

ii

ddd

,,1L

= and[ ]Ti

m

iidfdfdf ,,1 L= .The image Jacobian can

then be estimated by1 = DQDFJq (11)

The estimated Jacobian can be used to

compute1+i

df and1+i

d for the next robot drive.

Then, information of1+idf and 1+id are observed

and stored to employ in a new pair of DQ and DFmatrix for next image Jacobian estimation.

2.2 Dynamic Estimation of Stereo Image

JacobianUsing image Jacobian from one camera is proved tohave some limitations. For examples, when the targetin the image is obscured, calculation of imageJacobian might not be achieved. Consequently,stereo image Jacobian then has been developed inthis work. Two cameras are used to obtain requiredinformation for each computation of image Jacobian.The main advantage of using two cameras is that if

one camera fails to provide information about thetarget, the other camera can still be able to track the



3/5

target. Details of using stereo image Jacobian aredescribed in the following.

Consider 1rJ and 2rJ as image Jacobian

of each camera, they can be calculated by usingii

r

idrJdf =

11and

ii

r

idrJdf =

22. Obviously,

both image Jacobians are related byi

dr .Consequently, both equation can be combined to

i

i

r

i

r

i

i

drJ

J

df

df

=

2

1

2

1(12)

From [ ]Tiririr JJJ 21= which is the Jacobian ofimage parameters from both cameras and robot end-effector velocity, the stereo image Jacobian can then

be defined by

[ ]Ti

q

i

q

i

q JJJ 21= (13)

wherei

rJ is the pm matrix andii

r

i

q JJJ = 2,12,1 is

the nm matrix.

Fig. 1 Diagram of visual servo system using stereo image Jacobian.

3. System ConfigurationsIn order to employ the stereo image Jacobian in therobot controller, the system is implemented on a 5-DOF SCORBOT-ER III as seen in Figure 2 inwhich two cameras are mounted on the robot arm

as shown in Figure 3. Both cameras are aligned tocover the work space of the robot gripper. A yellowball is used as a target. It can then be segmentedand located in the image plane by using simplecolor segmentation. Finally, the objective of thesystem is to drive robot arm to reach for the target

by keeping it in the center of the images of bothcameras.

Once the system starts, initial values ofJacobian are computed. There are 2 methods usedin this work. Firstly, robot joints are movedrandomly until enough information is obtained.

Secondly, each robot joint is moved andcorresponding parameters are observed and

employed for the next move of the robot arm. Theimage from each camera has a size of 320x240

pixels. The target is said to be in the center of theimage if it locates within a radius of 10 pixels ofthe image center.

Fig. 2 System configurations.



4/5

Fig. 3 Two cameras mounted on the robot arm.

4. Experimental ResultsFigure 4 shows a sequence of robot actionsreaching the target at random position within therobot workspace. The image sequences taken from

each camera are shown in Figure 5. The errordistance of the target position from the imagecenter of each camera is displayed in Figure 6. Theresults clearly show that the robot arm can reachthe target where the final location of the target isinside the gripper. This can be seen by the target islocated in the image center for both camera. Therobot manipulator can reach the target withinapproximately 25 iterations. This also shows thatthe estimated stereo image Jacobian can beefficiently achieved without having to calibrate

both cameras and robot manipulator.

Fig. 4 Robot arm moves toward the target.

(a) (b) (c)

(d) (e) (f)

Fig. 5 Image sequences taken while the robot end-effector is moving toward the target.

5. Discussion and ConclusionIn this paper, stereo image Jacobian has beendeveloped and applied to control a 5-DOF robot

arm. By using two cameras mounted on the robotarm, image parameters from both camera can beachieved and used in image Jacobian calculation.The proposed dynamic estimation of stereo imageJacobian allows the system to achieve visualservoing without any calibration of the camera orrobot and any robot kinematics. The robottrajectory can then be planned online which allowsthe system to work in an undeterminedenvironment. The proposed technique can also beused to directly drive motors of the robot joints.The results show the accuracy of both position and

direction while the robot arm moves toward thetarget. The implementation is also as convenient as

using single image Jacobian. Due to its fast andsimple characteristic, the use of stereo imageJacobian allows the system to be improved for the

better control performance.

Reference:

[1] J. Hill and W.T. Park, Real Time Control ofa Robot with a Mobile Camera, Proceedingsof the 9th ISIR, 1979, pp. 233-246.

[2] L.E. Weiss, A.C. Sanderson and C.P.Neuman, Dynamic Visual Servo Control ofRobots: an Adaptive Image-Based Approach,

Robotics and Automation. Proceedings. 1985

IEEE International Conference, 1984, pp.662-668.

[3] L.E. Weiss, A.C. Sanderson and C.P.Neuman, Dynamic Sensor-Based Control of



5/5

Robots with Visual Feedback, IEEE Journalon Robotics and Automation, Vol. RA-3, No.5, 1987, pp. 404-417.

[4] M. Kabuka, E. McVey and P. Shironoshita,An Adaptive Approach to Video Tracking,

IEEE Journal on Robotic and Automation,Vol. 4, No. 2, 1998.

[5] P.I. Corke, S. Hutchinson and G. Hager, ATutorial on Visual Servo Control. IEEETrans. on Robotic and Automation, Vol. 12,

No. 5, 1996.[6] G. Flandin, F. Chaumette and E. Marchand,

Eye-in-hand/Eye-to-hand Coorperation forVisual Servoing, Proceedings of the 2000

IEEE Int. Conf. on Robotic and Automation,2000, pp. 2741-2746.

[7] M. Elena, M. Cristiano, F. Damiano and M.Bonf, Variable structure PID Controller forcooperative eye-in-hand/eye-to-hand VisualServoing, Proceedings of the 2003 IEEE Int.Conf. on Robotic and Automation, 2003, pp.989-994.

[8] P.J.S. Gonalves and J.R.C. Pinto, DynamicVisual Servoing of Robotic Manipulators.Proceedings of EIFA03, Vol. 2, 2003, pp.560-565.

[9] J.A. Piepmeier, G.V. McMurray and H.Lipkin, Uncalibrated Dynamic Visual

Servoing. IEEE Trans. on Robotics andAutomation, Vol. 20, No. 1, 2004

[10] H. Liu, D. Liu and Y.X. Yang, Research ofReal Time Robot Visual Servoing Based onGenetic Algorithm. Proceedings of the Int.Conf. on Machine Learning and Cybernetics,2002, pp. 87-90.

[11] S, Lonard and M. Jgersand,Approximating the Visuomotor Function forVisual Servoing. Computer and RobotVision, 2004. Proceedings. First Canadian

Conference, 2004, pp.112-119.[12] B.H. Yoshimi and P.K. Allen, Active,

uncalibrated visual servoing. In Proc. IEEEInternational Conference, 1994, pp. 2670 2675

[13] Y.H. Liu, H. Wang, and K. Lam, DynamicVisual Servoing of Robots in UncalibratedEnvironments, IEEE InternationalConference, 2005, pp. 2670 2675.

[14] Q. Jiang and J. Su, Online Estimation ofImage Jacobian Matrix by IKalman-BucyFilter for Uncalibrated Stereo VisionFeedback. IEEE International Conference,2002, pp. 562-567.

[15] J.J. Craig, Introduction to ROBOTICS:mechanics and control, Addison-WesleyPublishing Company, USA, 1989.

0 5 10 15 20 25 30-50

0

50

100

150

Error(Pixel)

Step

error x axis

error y axis

(a)

0 5 10 15 20 25 30-50

0

50

100

150

Error(Pixel)

Step

error x axis

error y axis

(b)

Fig. 6 Error distance of the target position from center of the image (a) from CCD camera #1 and (b) from CCDcamera #2


visual servo control using stereo image jacobian - saetang

Documents