Feature Based Image Mosaicing

Satya Prakash Mallick

IntroductionMosaicing methods can be classified

broadly into

1. Direct Method: Uses information from all pixels. Iteratively updates an estimate of homography so

that a particular cost function is minimized. Sometimes Phase-Correlation is used to estimate

the a few parameters of the homography.

2. Feature Based Method: A few corresponding points are selected on the two images and homography is estimated using these reliable points only.

My Choice?

Feature Based Method: Because1. They are in general more accurate. 2. Can Handle large disparities.3. Convergence: Direct methods, may not

converge to the optimal solution is the presence of local minima.

4. For reliable performance direct methods rely on feature based initialization.

Problems in Image Mosaicing

1. Global Alignment: essentially means recovery of underlying homography.

2. Local Alignment: Correct the local mismatches left after global alignment.

3. Image Blending: a decision has to be made as to what color the overlapping region should take.

4. Image Warping: After calculation of homography, a decision has to be made as to how to warp one image w.r.t the other.

Problems in Image Mosaicing

5. Automatic selection of images to blend.

6. Auto-exposure compensation7. Camera error compensation.

I am looking at the first four problems.

The Algorithm ( Overview )

1. Detect corners in both the images2. Solve for correspondence. 3. Use RANSAC to estimate homography.4. Refine the estimate of homography

using a non-linear method.5. Make a decision on how to blend the

images.6. Warp the one image with respect to the

other taking the blending decision into consideration

Solving for correspondences

Corners were detected in both the images using Harris corner detector. Correspondences are solved using a version of Zhang’s relaxation algorithm:

Matching Through Correlation: Disambiguating Matches Through Relaxation:

Solving for correspondences

Matching through correlation:

Solving for correspondences

Disambiguating matches through relaxation:

Show correspondence results

Estimation of homography

RANSAC was implemented for estimating the homography relating the two scenes. Results show a comparison between using RANSAC and Least squares. The error function being minimized is: 2'2'1 ),(),( iiii HxxdxHxd

Image Blending

Weighted Image Blending:

Image Blending

Lightest Path Cut: The idea is to take the difference of the images in the overlapping region and cut the region along that curve of minimum intensity. The implementation was done using dynamic programming.

Note: Optimal implementation of the algorithm is non-trivial and can be a good problem to look into.

),,min( 1,1,11,1 jijijiijij EEEeE

Results of Blending

Try to find the curve along which the image overlap was cut! ALL THE BEST!

Homography Refinement

In the Weighted image blending example we saw, there was some blurring at the edges of the overlap. The obvious question is:

Are we sure that the homography estimate is good enough or can it be further refined?

Homography Refinement

The Homography was refined using Newton’s non-linear optimization technique. It’s “safe” to use Newton’s method because we are already close to the solution.

The vector X shown is made of inliers got using RANSAC

x1]x2 1,[)(

]2,2....,21,21,1,1......,12,12,11,11[

)(

HHHf

NyNxyxNyNxyxyxX

HfXT

Homography Refinement

Results: The blurring near the edge of overlapping region is almost gone.

Limitations and Mistakes

My biggest mistake: I assumed that, to make a mosaic with many images, it is sufficient to keep on adding images to the main mosaic. The mosaic doesn’t do well, if we try to blend more than four images. So there should have been a global refinement on different homographies

Conclusions

Lessons to be learnt: Non-Linear optimization of homography is not

an overkill. After RANSAC, it is usually very fast ( Most of the time, my results converged after 2-3 iterations only)

It’s a good idea to blend images along the lightest curve along the difference image. However, the image quilting algorithm cannot be directly used in that.

To blend a large number of images, there is “something” more to be done than finding pair-wise homography of images.

Conclusion

Image warping should always be done backward with bilinear interpolation.

References

[Hartley] Hartley, R. & Zisserman, A. (2000) Multiple View Geometry{Cambridge University Press, UK.[Shum] Shum, H. & Szeliski, R. (1998) Construction and refinement of panoramic mosaics with global and local alignment. IEEE Int'l Conf. Computer Vision, pp. 953-958.[Faugeras] Zoghlami, I. & Faugeras,O. & Deriche,R. (1997) Using geometric corners to build a 2d mosaic from as set of images.Computer Vision and Pattern Recognition, pp 421-425.[Zhang] Zhang, Z.& Deriche, R. & Faugeras, O & Luong, Q. A robust technique for matching two uncalibrated images through the recovery of the unknown epipolar geometry (1995) Artificial Intelligence Journal, 78:87-119, October 1995[Harris] Harris, C. & Stephens, M. A combined corner and edge detector.(1998) Proc. of 4th Alvey Vision Conf.,147-151.[Szeliski] Szeliski, R. Image Mosaicing for Tele-Reality Applications.(1994). Digital Equipment Corporation, Cambridge, USA.[Davis] Davis, J. Mosaics of scenes with moving objects.(1998).Computer Vision and Pattern Recognition[Capel] Capel,D & Zisserman,A. Automated mosaicing with super-resolutionzoom.(1998).Computer Vision and Pattern Recognition

THANKS !