Active Calibration of Cameras: Theory and Implementation Anup Basu

Active Calibration of Cameras:Theory and ImplementationAnup BasuSung HuhCPSC 643 Individual Presentation IIMarch 4th, 20091OutlineIntroductionTheoretical DerivationStrategies for Active CalibrationTheoretical Error AnalysisExperimental ResultConclusion and Future Work

2OutlineIntroductionTheoretical DerivationStrategies for Active CalibrationTheoretical Error AnalysisExperimental ResultConclusion and Future Work

3IntroductionImportant step for a 2D image to relates to the 3D worldInvolves relating the optical features of a lens to the sensing devicePose estimation, 3D motion estimation, automated assemblyParameters: image center and focal lengthExpressed in terms of image pixelsLinear vs. Nonlinear, Lens distortion consideration vs. w/o consideration4Technique: LinearityLinearNonlinearSimpler to implementMost cannot model camera distortionsCapable to consider complicated imaging model with many parametersComputationally expensive search procedureReasonable good initial guess for convergence of the solutions5Major Drawback of existing algorithmCalibrate with predefined patternRelating image projections to the camera parametersRecent algorithms suffer from the same limitationNew discovery: Active Calibration6Active CalibrationCamera capable of panning and tilitng can automatically calibrate itselfModeled from eye movementActive machines can keep track of object of interestFacilitate region-of-interest process7Active Calibration How different?Does not need a starting estimate for focal length and image centerDoes not need prior information about focal lengthDoes not need to match points or feature b/w imagesReasonably accurate localization of contourEstimate of center (Not too far from true value)

8Method of CalibrationUsing perspective distortion to measure calibration parametersWithout using perspective distortion9OutlineIntroductionTheoretical DerivationStrategies for Active CalibrationTheoretical Error AnalysisExperimental ResultConclusion and Future Work

1010Theoretical Derivation11

Theoretical DerivationLemma 1Camera rotates by R and translate by TNew image contours

12

Theoretical DerivationLemma 1 - ProofUse two set of equation

,13

Theoretical DerivationProposition 1Depth (Z) is larger than X, Y, ZCamera moves by small tilt angle14

Theoretical DerivationProposition 1 ProofRotation matrix R at small tilt angle

are negligibleFrom Lemma 115

Theoretical DerivationProposition 1 ProofExpand right side of equation with Taylor series, because of small t With the same assumption, if camera moves by small pan angle p

16

OutlineIntroductionTheoretical DerivationStrategies for Active CalibrationTheoretical Error AnalysisExperimental ResultConclusion and Future Work

1717Strategy for Active CalibrationWant A relation b/w lens parameters and image information w/ given image contours before & after camera motionRelate focal length to other camera parameters and the pan/tilt angles18Strategy for Active CalibrationProposition 2Similar assumption as Proposition 1Center of the lens is estimated with a small error (x, y)19

Strategy for Active CalibrationProposition 2 ProofFrom Proposition 1

Estimate image Center with error (x, y)

Ignore xy term20

Strategy for Active CalibrationProposition 3 (Plan A)Using tilt (or pan) movement and considering three independent static contours, two linear equation in xy can be obtained if negligible terms are ignored21Strategy for Active CalibrationProposition 3 ProofTwo different contour, C1 and C2Point lying on C1 & C2, (x(1),y(1)) and (x(2),y(2))From Proposition 2

22

Strategy for Active CalibrationProposition 3 ProofEquate right side equations and simplify

where23

Strategy for Active CalibrationProposition 3 ProofThird contour, C3 Point on C3, (x(3),y(3))

where24

Strategy for Active CalibrationProposition 3 ProofFinding fx and fy with estimated center

e denote the estimate of a certain parameter25

Strategy for Active Calibration Procedure Summary for Plan AEstimate x and y using (3) and (4) with three distinct image contourObtain estimate for fx and fy by substituting resulting estimate into (5) and (6)Term and make (5) and (6) unstable26

Strategy for Active Calibration Procedure Summary for Plan AVariation in x-coord. for any point is due to change in perspective distortion (tilt)Little change in the image y-coord. corresponding to a given 3-d point (pan) and are small (few pixel)Relative error can be large presence of noise and inaccuracies in localization of a contourEstimate in (5) & (6) are often unreliable27

Strategy for Active Calibration Proposition 4 (Plan B)Using a single contour and pan/tilt camera movements fx and fy can be obtained if negligible terms are ignored28Strategy for Active Calibration Proposition 4 Proofx and y are non-zero in the second equation in Proposition 1

Simplify

The last three terms are negligible even if x and y are large29

Strategy for Active Calibration Proposition 4 ProofSimplifying eq (7)

fx can be obtained with similar way30

Strategy for Active Calibration Proposition 4 CorollaryGiven two independent contours, pan/tilt camera movements, and estimate of fx and fy given by and respectively, x and y can be obtained by solving

Considering from two independent contour from Proposition 231

Strategy for Active Calibration Proposition 4 ProofConsider (8)Most practical systemy < 300, fy > 500(8) is in formA = 1, B < 0, C is small compare to B

32

Strategy for Active Calibration Procedure Summary for Plan BEstimate fx and fy from (12) and (13) using a single image contourSolve for x and y by substituting resulting estimates into (10) and (11) and using another independent contour33Strategy for Active Calibration Proposition 5When there is error in contour localization after pan/tilt movements, the ratio of the error in Plan A compared to Plan B for estimating fx (fy) is approximately34

Strategy for Active Calibration Proposition 5 ProofIntroduce similar error term in and in (13) and (5) respectivelySimplify the expressions and consider the approximate magnitude of error in both the expressionsTake the ratio of these two terms35

Strategy for Active Calibration Proposition 5 ImplicationError in Plan A can be as large as 30 times that of Plan B, for estimating focal lengthsPlan A is theoretically more precise, but not reliable for noisy real scenes36OutlineIntroductionTheoretical DerivationStrategies for Active CalibrationTheoretical Error AnalysisExperimental ResultConclusion and Future Work

3737Theoretical Error AnalysisEffect of errors from various sources on the estimation of different parametersErrors in measurements of pan/tilt angles Effect of noise in the extraction of image contours38Theoretical Error AnalysisRemark 1 Error in measurement of the pan (tilt) angle generates a proportional error in the estimate of fx(fy)ProofConsider(5)fx is proportional to the pan angleAny error in the measurement translates to a corresponding error in fxAny error from tilt angle generate a proportional error in fy

39Theoretical Error AnalysisRemark 2 Errors in measurement of the pan & tilt angles do not affect the estimate of the lens center independent contours from the same image are consideredProof Linear equations in x and y are obtained by equating the right hand sides of two equations

40Theoretical Error AnalysisConsider (1) & (2)Denote 1: error in tilt angleContour extracted from same imageThen (t+1) of (3) cancels out from both sidesError in pan/tilt angle do not affect the estimate of lens center

41Theoretical Error AnalysisConsider two independent images generating the contours in (1) & (2)K1 in (3) modifies to is not equal to 1 in generalErrors in angle can change the estimate of the lens center if contours from independent images are considered42

Theoretical Error AnalysisRemark 3The coefficients of the linear (3)-(6) are unbiased in the presence of uncorrelated noise with zero meanCoefficients involve a linear combination of terms

These terms are unbiased in the presence of uncorrelated noise with zero mean43

Theoretical Error AnalysisRemark 4The variance of the coefficients of (3)-(6) is inversely proportional to the number of points on a contourUncorrelated noise with zero mean is consideredForm of variances

Inversely proportional to the number of points for which the averages were computed44

OutlineIntroductionTheoretical DerivationStrategies for Active CalibrationTheoretical Error AnalysisExperimental ResultConclusion and Future Work

4545Experimental ResultSimulation Validity of AlgorithmsSynthetic data usedThree independent contour represented by three sets of 3D pointsPoints projected onto the image planeValues quantized to the nearest integer Without noise, A produced more accurate estimateLess than 1 percent relative error in focal length estimate46Experimental ResultSimulation Variation of error in focal length estimateChange fx and fy from 100 to 1000 with interval 100Keep other parameters fixedDiscretization error influence A more when focal length was smallA is not very robust to noiseLarger the focal length, smaller the error relative to the focal lengthBetter estimate production with A

47Experimental ResultSimulation Variation of error in focal length estimateError estimate from B does not drop off rapidlyB is theoretically less accurate than A48Experimental ResultSimulation Gaussian Noise AddedPoor performance with A20, 28, and 40 percent error with 3, 4, and 5 noise standard deviationHigh robustness with B49Experimental ResultTracking ContourMatch contours of interest during pan/tilt For automatic calibration Edges in the original image was thickened using the morphological operation of dilationEdges after pan/tilt was AND-ed with the dilated image to extract corresponding contours after camera rotation50A sequence of images for small pan movements of a camera51

Corresponding edge images52

Tracked over the sequence of image53

Experimental ResultCalibration With Real Images initial image and its edge54

Experimental ResultCalibration With Real Images panned image and its edge55

Experimental ResultCalibration With Real Images matching contour56

Experimental ResultCalibration With Real ImagesPlan AEstimate of fx and fy, 693 and 981With known pattern and refining initial estimate by trial and error, 890 and 1109Plan BEstimate of fx and fy, 917 and 1142Produces estimates fairly close to the true value57Experimental ResultOther Environment58

Experimental ResultOther EnvironmentEstimate produced using Plan B, 902 and 1123, 905 and 1099Average relative error < 1.5%A produced very inaccurate estimateB produced stable estimatesLens center estimates are not very accurate for both

59OutlineIntroductionTheoretical DerivationStrategies for Active CalibrationTheoretical Error AnalysisExperimental ResultConclusion and Future Work

6060ConclusionAlgorithm do not require unique patternOnly need scenes with strong and stable edgeA gives almost perfect estimates in an ideal environmentB is suitable for noisy synthetic images or real scenes61Future WorkSimplify algorithm further by considering roll movements of the cameraDesigning a simple method to obtain the optical center of the lens

62Question ?63