calibration using ga

8/3/2019 Calibration Using Ga

1/6

Proceedings of the 2001 IEEEInternational Conference on Robotics & AutomationSeoul, Korea - Ma y 21-26, 2001

Camera Calibration With Genetic AlgorithmsYongniian Zhang and Qiang Jit

Department of Computer ScienceUniversity of Nevada, Reno

Department of Electrical, Computer, and Systems Engineering+Rensselaer Polytechnic Instituteqj eecse. rpi.edu

AbstractIn thi s paper I, we present a novel approach basedon gene t i c a l gor i thms f o r per f ormi ng camera c d i -bra t i on . Cont ra i y t o the c lass ical non- l inea r pho-t ogrammet r i c appr*oach the proposed technique cancorrect l y f i nd t he near -op t i mal so l u t i on w i t hou t t heneed of initid guesses (w i t h on l y very l oose param-e t er bounds ) u n d ,with a minimum number of con-trol poin.ts (7 poirits) . Results f r o m our ex t ens i ves t u d y usin,g both synthet ic and real image data aswell as perforirzaizce co mpa riso n with Tsa i 's proce-dure dem,ons tru t e t he exce l len t per f ormance of t h eproposed technique an t e r m s of convergence, accu-racy, and robrrstness.

1 IntroductionCamera calibration is an essential step in many

machine vision and photogrammetric applicationsincluding robotics, 3D reconstruction, and mensu-ration. It addresses the issue of determining intrin-sic and extrinsic cainera parameters using 2D im-age data arid the corresponding known 3D modeldat a. Classical camera calibration techniques can bebroadly classificcl into linear approaches [ l , , 31 andnon-linear approaches [ l , 1. Linear methods havethe advantage of coniputa tional efficiency but sufferfrom a lack of accuracy and robustness. Non-linearmethods, on the other hand, offer a more accurateand robust solution but computationally intensiveand require good initial estimates. To get aroundthis problem, one common strategy in computer vi-sion is to attack tlic camera calibration problem byusing two steps [4, 51. The first step generates anapproximate solution using a linear technique, whilethe second step improves the linear solution using anon-linear iterative procedure.

The first step utilizing linear approaches is key tothe success of two-step methods . Approximate solu-

'A longer version of this paper will appear in March issueof IEEE Transactioiis on System, Man, Cybernetics

tioiis provided by the linear techniques must be goodenough for the subsequent non-linear techniques tocorrectly converge. Being susceptible to noise in im-age coordinates, existing linear techniques ar e, how-ever, notorious for their lack of robustness and ac-curacy [6], [ 7 ] . The use of more points can helprelieve this problem. However, fabrication of morecontrol points often proves to be difficult, expensive,and time-consuming. Given a small number (e.g.,minimum) of control points, it is therefore question-able whether linear methods can consistently androbustly provide good enough initial guesses for thesubsequent non-linear procedure to correctly con-verge to the optimal solution.

Another problem is th at almost all nonlinear tech-niques employed in the second step use variants ofconventional optimization techniques like gradient-descent, conjugate gradient descent or the Newtonmethod. They therefore all inherit well known prob-lems plaguing these conventional optimization meth-ods, namely, poor convergence and susceptibility togetting trapped in local extrema. If the startingpoint of the algorithm is not well chosen, the solutioncan diverge, or get trapped at a local minimum.

To alleviate the problems with existing cameracalibration techniques, we explore an alternativeparadigm based on genetic algorithms (GAS) o con-ventional non-linear optimization methods. GASwere designed to efficiently search a large, non-linear,poorly-understood spaces and have been widely ap-plied in solving difficult search and optimizationproblems including camera calibration [8]. Resultsfrom our study are encouraging and promising. Theproposed GA approach can quickly converge to thecorrect solution without initial guesses and with theminimum number of points (7 points).2 Background

In this section we provide short introductions togenetic algorithms and perspective geometry usedfor camera calibration.

0-7803-6475-9/01/$10.000 2001 IEEE 21 77


2/6

2.1 Genetic AlgoritlhmsGAS are stochastic, parallel search algorithms

based on the mechanics of natural selection and theprocess of evolution. The thre e basic GA opera-tions are evaluation, selection, and recombination.Evaluation of each string which encodes a candi-date solution is based on a fitness fuiiction that isproblem dependent. Selection is done on the basisof relative fitness and it probabilistically culls solu-tions from the populat ion that have relatively low fit-ness. Good individuals (with high fitness) are thenassigned for fur the r reproduction. Recombination,which consists of mutation and crossover, imitatessexual reproduction. Mutation insures against thepermanent loss of genetic materi al during the selec-tion process. Crossover is a structured yet stochasticoperator that allows information exchange betweencandidate solutions.

2.2 Perspective GeometryFigure 1shows a pinhole camera model that does

not consider any type of lens distortions. Let X =(x y , z ) ~ e a 3D point in an object frame andU = (U w) the corresponding image point in the im-age frame. Let X, = ( p q s ) ~e the coordinates ofX n the camer a frame and p = ( e r ) T be the coor-dinates of U in the row-colunin frame as illustratedin figure 1. The image plane, which corresponds tothe image sensing array, is assumed to be parallelto the (Xc,Yc)plane and at a distance f to the ori-gin, where f denotes the focal length of th e camer a.The relationship between the camera frame C, andobject frame CO s given by

CiImagc Framc Object FrameI/ rinciple Point

Figure 1. Perspective projection geomet,ry

where R is a 3 x 3 rotation matrix defining the cam-era orientation and T is a translation vector repre-senting the camera position. R and T can further

be parameterized asR = (;: '11 7'12 ~ 1 3:) T = ( ) (2 )The T i J in matrix R can be expressed as the func-

tion of camera pan angle w , tilt angle 4 and swingangle. The collinearity of 3D world coordinate Xand 2D image coordinate p can be written as

where s x and sy are scale factors(pixels/~nnz) ueto spatial quantization. uo arid 210 are the coordi-nates of the principle point in pixels relative to im-age frame. q is a vector of all camera parameters asdefined in equation 4.3 Objective Function'The main task of camera calibration in 3D ma-chine vision is to obtain an opt imal se t of the interiorcamera parameters ( ( u g ,vug), s ,sg, ) T and exteriorcamera parameters (w,4 ,K , t,. t, , using knowncontrol points in the 2D image and their correspond-ing 3D points in the world coordinate system. Let qbe an vector consisting of the unknown interior andexterior camera parameters, tha t is,

q = b o , VI, f , sx, ,, w ,4 , 4 t x , ,, LIT (4 )For notational convenience, vie rewrite q as q =( q l , q 2 , . . . , q 1 1 ) ~ , here 4 1 , q 2 and 411 correspond toU O , wo and t , respectively in toheprevious notationused for q. Assume q is a solution of interior and ex-terior camera parameters and 'qC Q, then we have

where 4% an d q+ are the lower and upper bounds ofq i . The bounds on parameters can be obtained basedon the knowledge of camera. Any reasonable intervalwhich may covers possible parameter values may bechosen as the bound of paramleter q i . For example,we may have w , 4 , K E [-T, ], E [20 ,70]as in ourtest cases and so on. An optimal solution of q withM control points can be achieved by minimizing

i= l

where Xi = ( z i , y i , z i ) is the i t h 3D point, g and ware defined in equation 3

2178


3/6

A key issue tliat arises in this approach is the ex-tremely large search space caused by the presenceof uncertainties. Thi s presents a serious challengeto conventional minimization procedures since the remay be several local iiiininia , and the choice of st art -ing point will determine which minimum the proce-dure converges to, or whether it will converge at all.If s tar tin g points are far away from the desired min-imum, the traditional optimization techniques couldconverge erroneously.4 GA Operators and Representation

Designing an appropr iate encoding and /or recom-bination method is often crucial to the success ofthis algorithm. To improve GA's convergence, weproposed a n e w mutation operator that determinesthe amount and direction of perturbation in thesearch space. Mutation can be viewed as one di-mensional or local search, while crossover performsmulti-dimension, or more global, search.4.1 Representation

GA chromosome are usually encoded as bitstrings and a long binary string is required in or-der to represent a large continuous range for eachparameter. Instead, we encode the GA chromosomeas a vector of real numbers. Each camera param-eter q i , i = 1, .., 1 is initialized to a value withinits respective bounds as defined in equation 5. Thechromosome vector may be defined as

where q: is an iiidividual from population N at t thgeneration. 9: is an individual from the new gen-eration after genetic selection; and q;C indicates theparameter that is niodified during the evolutionaryprocess.4.2 Mutation

Our mutation scheme comprises two steps: de-termining the search direction and simultaneouslydetermining the step size in the selected search di-rection. In Equat ion 5, the search space Q shouldbe a convex space S. The task of the G A is to de-termine an uii~ciiowiioptimal point q l E [ q i ,q t ] inthe convex space S which minimizes the global errorfunction of equation G at that point.

Assuniing that the probability of receiving a cor-rect step size from the G A is p , whenever the currentq k 5 q i the G A correctly increases q k with a prob-ability p . It ma y also, however, incorrectly decrease

2179

qk with a probability 1- . It is reasonable to ex-pect that it is equally likely for a GA t o increase q kas to decrease q k . Then the next current point q;+'is given by

4; + I A ( t , ;) + (1 )A( t , Jq; if > q; (8 )-{ q k i f q;+l < 4;q;+,+'=where A(t,q;) and A(t,q1;)are the step sizes for

the upper and lower bounds of qk respectively. I isan indicator function assuming the value of 0 and 1depending on the outcome of a coin toss.

The crucial issue is the amount of perturbation(step size) of point q k in the interval [q; , g z ] . TOOsmall pertur batio n may lead to sluggish convergence,while too large perturbation may cause the GA toerroneously converge or even oscillate. Since q k E[q1;,q;] E S holds, q k must be a fraction, say e ,of the way between its lower bound qc and upperbound q t , i.e.

Assuming the successive point is q;+', where q;" E[ q i , q ; ] , we can utilize the golden sect ion to deter-mine the optimal step size of 4 as

where c is the golden f ract ion with value of ( 3 -&) / 2 ; and a and b represent, respectively, 14 - k land 1qk - q i l . Since equation 10 linearly convergesto it s ideal value, to improve the GA's efficiency andskip away unimportant search region in the earlystage of evolution processing, we then incorporateevolution time t in equation 10, i.e,

where r is a random variable distributed on the unitinterval [0, I] ,a lies in [I,1.51 an d T denotes the tot alnumber of iterations.

The essence of our mutation scheme, termedgolden sect ion, lies in integrating equations 10, 11,1 2 together and in each generation (or iteration) thescheme stochastically chose one of them to deter-mine the position of the new current point. Therandom determination of step size allows discontinu-ous jumps in the parameter interval, and then golden


4/6

section is used to control the search direction. Thisultimately makes the GA coiiverge more accuratelyto a value arbitrarily close to the optimal solution.Additionally, proposed mutat.ion scheme requires in-significant computati onal time.

4.3 CrossoverCrossover produces new points in the search

space. The initial population forms a basis of theconvex space S and a new individual in current gen-eration is generated by a linear combination of theparent individuals in the previous generation.

Let qt and qf be two individuals from popula-tion N at generation t . They satisfyq, = { q3 E [ q , , q ,+ ] c S } , = 1. .., N j = 1, . ,n (13)where N denotes the population size, and n the nuni-ber of parameters (or chromosome length). Follow-ing equation 13, a new individual q, in generationt + 1 can be expressed as a linear combination oftwo arbitrarily selected individuals from the previ-ous generation t , that is,

5.1 Synthetic Images;We generated 20 0 independently perturbed sets

of control points for each noise level so that an ac-curate ensemble avcrage of the results could be ob-tained. To ensure fair comparison, GA parameterswere ideiitical in all test cases. We first investigatedhow image noise and control points affect the per-formance of our approac h. For this study, the initi alcamera parameter bounds and their ground truthare given in Table 1.

Table 1. Camera parameter groundtruthand bounds

20 , 40144.0 110, 160144.0 110, 160

200, 300170, 330-38.0 -SO, 5035.0 -80, 501210.0 900, 1400-1.901460.20916 -7r, 7r0.15152 -7r, 7r

where cy ranges within [O, I] . p is a bias f a c t o r thatincreases the contribution from the dominating in-dividual with a better fitness at current stage. As -suming non-negative fitness function, p can be de-termined from the following equations.

where f ( * ) denotes the GAs fitness function definedin equation 6.5 Experimental Results

This section describes experiments performedwith synthetic and real images to evaluate the per-formance of our approach in terms of its accuracyand robustness under 1) varying amounts of imagenoise, 2 ) different numbers of control points, a nd 3)different ranges of parameter bounds. Furthermore,we describe results from a comparative performancestudy of our appr oach with tha t of Tsais calibrationalgorithm.

Results from this experiment show that sufficientaccuracy can be achieved with the minimum num-ber of control points without loss of accuracy androbustness. Compared with conventional calibrationmethods, this represents a practical advantage sincecreating many redundant control points usually is anexpensive and time-consuming procedure. Second,this experiment demonstrates the algorithms con-siderable accuracy and robustness in the presence ofdifferent noise levels as seen from both camera errorsand image pixel errors. The camera errors are alsowithin acceptable margins given considerable imagenoises ( 3 pixels).

We now present results from the experiment car-ried out to test the stability of the proposed methodunder varying initial bounds. Practically, camerascale factors (sz, g) and principle point ( u 0 , U O ) canbe restricted in a relative small ranges based on cam-era manufact ure information and possible hardwareerrors. Accordingly, in this experiment we assumethat bounds of s,, sy, 0 and vo can be maintainedin a reasonable range (which is often practiced inother existing calibration techniques) and then ex-amine the change in performance by means of vary-ing bounds of focal length and extrinsic parameters.We investigated 4 different cases using minimumcontrol points (7 in this case). The initial boundsand corresponding est imate d c.amera parameters un-der free-noise (0 = 0) are summarized in Table 2(Ground tru th see Table I ) .

2180


5/6

Table 2. Estimated camera parameters under dif-ferent parameter bounds ( o = 0)

&

f

Case 1

0, 501100, 1400-x,-x,-?r, x 0.1525811 -?r, I 0.173463Case 3 Estimated Case 4 1 Est imated 20,50 27.2 i in ,60 1 25.42

Est imated26.29138.40244.06-33.9135.03192.00

1223.05-1.901560.206670

Ground Truth70 . 043.4s43.48258.0204.0-100.0-85.02000.00 . 5 2 : ~ m0.0174530.034907

Case 220,40120,150230, 270180, 200-50, 501000, 1400-50, 50-x, r-x, r

Parameter Bound40 , 8540, 6040 , 60250, 270190, 210-150.0. 150.0-150.0, 150.014on.0, ~ 6 n n . n-x . r-x, r-x,

Est imated27.87132.37245.17191.82-34.4035.001240.430 . ~ 2 6 4 0 7-1.875521; 180, 200 1 194.91

-110, 110 33.917 0 0 , 14nn 1207.32-x, r n . m m 5-7r, x 0 . 195439

-110, 110 -34.21

-x, r -1.851877

197.46-37.261235.120.275590

33.01-1.8200670.225426

In four cases shown in Table 2 , we gradually en-large the bounds of focal length and the translationparameters, and keep all camera angles range from-n o Sn.Table 2 indicates that enlarging parame-ter bounds only causes marginal impact on the cal-ibration accuracy. In all cases, the pixel errors andcamera errors are within acceptable margin. Errorsof some parameters such as t , become moderatelylarge with an increase in their initial ranges, How-ever, the pixel errors maintain almost the same levelregardless of their bounds. This is because our G Acan always find a correct search direction towardsthe global niiiiimuiri by gradually reproducing a fitparameter set , and errors caused by deflection of oneparameter may offset by others. Particularly, if wecan restrict the range of intrinsic parameters (exceptfor focal length) in a reasonable range as practiced byother calibration techniques, the calibration wouldbe more accurate as demonstrated in case 4, and thiswill be of riiore practica l interest . More importantly,the results imply that the bounds of parameters canbe any reasonable interval which may cover possibleparameter values without the need of expert knowl-edge. This means that regardless of the parametersbounds, the G R algorithm can always converge to asolution very close to the optimal solution. This issignificant and i s exactly what we set out to achieve,i.e., our m ethod ca*n avoid local rninima without theneed of good iw i t in l es t i mat es .5.2 Comparison with Tsais

To further study the performance of the proposedmethod, we con-qarecl our method with Tsais twostep calibration technique [4], which perhaps is the

most popular camera calibration method for bothcomputer vision and photogrammetry communities.We synthetically generate 108 non-coplanar controlpoints to test Tsais calibration method and use thesame methodology as described earlier in this sec-tion to produce the perturbed dat a sets. The exper-iments were performed over 16 different noise levelsand the final results for each noise level is the aver-age of results from 200 independently perturbed setsof control points. To see if equivalent results can heachieved by o ur proposed method with fewer cali-bration points, we use 108 and 8 of control points inour approach. T he parameter ground tru th and ini-tial parameter bounds in this experiment are givenin Table 3 .

Table 3. Camera parameter ground truth andbounds

Figure 2 depicts the comparison results with dif-ferent noise levels. Note tha t in Figure 2 , we ig-nored results froin Tsais method as noise level (0)is over 2 . 2 since parameter errors tend to skyrocketfor Tsais method if D > 2.2. We can concludefrom Figure 2 that when synthetic image data hasno perturbations or very little perturbations, Tsaiscalibration technique is extremely accurate. Theaccuracy of the Tsais method, however, decreasesdramatically after a short interval of noise level(0 5 0 5 0.4). For example, if the noise level cr isover 1.5, some parameters deteriorate severely andthe pixel errors almost exponentially increase. Wecan therefore conclude that the accuracy potentialof Tsais methods is very limited in noise situation .In contrast, our approach shows that camera errorsremain approximately constant for various noise lev-els and pixel error increases linearly. This provesthe accuracy and robustness of our technique un-der no i sy cond i t i ons . The method therefore hashigher capability of immunization against the im-age perturbation. Furthermore, increasing controlpoints is almost not helpful for the algorithm accu-racy and robustness in our approach as illustrated inFigure 2. Once again, it implies then, tha t with min-imum control points (8 in this case), our approach is

2181


6/6

able to achieve accuracy and robustness equivalentto or even better than that wi.th highly redundantcontrol points.

0 0. 4 0 8 1.2 1 6 2 2. 4 2 . 8GanSSlan Pixel NoiseiSlgmaI

0 0. 4 0.8 I 2 1.6 2 2.4 2.8Gaussian Pixel Nalse(Sigma!0.03

k 0 . 0 2 5-5 0.022" 0.01x5 0.0152 0.005= o

.- I

0. 4 0. 8 1 2 1.6 2 2 4 2 .8Gaussian Pixel NolseIS19maI

1 8 1 , , , , , -

0 0. 4 0.8 1.2 I 2 2 4 2.8Gaussian Pixel ~ a i s e ( S i m a !

Figure 2. Side-by-side comparison with Tsai'scalibration algorithm. First ccllumn: estimated er-ror of camera extrinsic parameters (T,, Ty T, ,R);second column: estimated errcc of camera intrinsicparameters ( f , sz , sy, U O , O) ; a.nd last figure on thesecond column: mean pixel error.

5.3 Experiments wi th Real ImagesWe applied our method to real images of indus-

trial parts and the accuracy is judged by visual in-spection of the alignment between the image of apart and the re-projected outline of the part usingestimated camera parameters. Results of visual in-spection shows excellent alignment between the orig-inal images and the projected outlines, which furtherprove the performance of our proposed approach onreal images.

6 ConclusionsWe presented a new approach to the problem of

camera calibration based on genetic algorithms withnovel genetic opcrators. Our study performed withboth synthetic and real iinagcs demonstrates the ex-cellent performance of our technique in terms of con-vergence, accuracy, and robustness. The comparisonwith Tsai's calibration technique shows that our ap-proach has high potential in the various noise situ-ation. Specifically, the proposed method enjoys sev-eral favorable properties:

0 It does not require initial guesses of camera pa-rameters (with only very loose bounds) to con-verge correctly.It achieves the sufficient accuracy and robust-ness with the minimum number (7 ) of calibra-tion points and for noisy images.It is tolerant to the large range of image pertur -bations.

References[l] Y. . Abdel-Aziz and H. M. Kara-ra, "Direct linear tran s-

formation into object space coordinates in close-rangephotogrammetry," in Proceedin.gs of Symp. Close-RangePhotogrammetry , pp. 1-18, 1971.

[a ] D. C. Brown, "Close-range camera calibration," Ph o -togrammetric Engeering and Remote Sensing , vol. 37>no. - , pp. 855-866, 1971.

[3] W . Faig, "Calibration of close-range photogrammetry sys-tems: Mathematical formulation," Photogrammetric En-geering and R emote Sensing , vol. 41, no. -,pp. 1479-1486,1975.

[4] R. Y. Tsai, "A versatile camera c-alibration technique forhighaccuracy 3d machine vision metrology using offthe-shelf tv cameras and lenses," IEEE Journal of Roboticsa n d Au t o ma t i o n , vol. RA-3, no. 4, pp. 323-344, 1987.

[5] 3 . Weng, P. Cohen, and M. Herniou, "Camera calibra-tion with distortion models and accuracy evaluation,"IEEE Tra n sa c t i o n of Pat tern Analysi s and Machine In-tell igence, vol. 10, no. 14, pp . 965-980, 1992.[6] X . Wang and G. Xu, "Camera parameters estimation and

evaluation in active vision system," Pat ter n Recognit ion ,vol. 29, no . 3, pp. 439-447, 1996.

[7] It . M. Haralick, H. J o o , C. Lee, X. Zhang, V. Vaidya,and M. Kim, "Pose estimation from corresponding pointdata," I E E E D a n . on Sys tems , Man, and Cybernetics,vol. 19 , no. 6, pp. 1426-1446, 1989.

[S j If . Y. Huang and F. H. Qi, "A genetic algorithm approachto accurate calibration of camera,'' Infrared and millmeterwaves , vol. 19, no. 1, pp. 1-6, 2000.

21 a2

calibration using ga

Documents