Fast stitching algorithm for moving object Fast stitching algorithm for moving object detection and mosaic constructiondetection and mosaic construction

School of Electrical Engineering and Computer Science School of Electrical Engineering and Computer Science KyungpookKyungpook National Univ.National Univ.

Image and Vision ComputingImage and Vision Computingvol. 22, 2004vol. 22, 2004

JunJun--WeiWei Hsieh Hsieh Presented by DongPresented by Dong--Chang LeeChang Lee

22 /43/43

Proposed method– Edge-based stitching method to detect moving object

and constructing mosaics from images• Coarse-to-fine scheme

– Estimating a good initialization of camera parameters with two complementary methods

» Edge alignment» Correspondence-based approach

– Refining the solution through an optimization process

AbstractAbstract

33 /43/43

Application of image stitching– Video compression, video indexing, object tracking,

creation of virtual environments• For example, panorama

Using an affine camera model– Recovering parameters of this model from pair of

images by two common methods• Correlation based approach

– Phase-correlation method in frequency domain– Feature matching approach in spatial domain

• Optimization-based approach

IntroductionIntroduction

44 /43/43

Problem of conventional methods– Variant to nonlinear intensity changes

• Not correspondences

– If the starting point is not properly initialized• Trapped on a local minimum

In this paper– Edge-based stitching technique to detect moving

objects and constructing mosaics from consecutive images

– Using a planar perspective motion model

55 /43/43

– Using coarse-to-fine approach• Coarse stage

– Two complementary methods» Edge alignment and correspondence based approach

• Fine stage– Refining the found initial estimate through an optimization

process

66 /43/43

Camera motion modelCamera motion model

Assuming– Captured images by video camera from a static scene

Relationship between two adjacent images– Describing by a planar perspective motion model

1'

1'

76

543

76

210

++++

=++++

=ymxm

mymxmyandymxm

mymxmx (1)

where coordinate of a pixel in the current frame coordinate of its corresponding point in the next frame

parameters associated with the focal length,rotation angle, and scaling of the camera

),( yx)','( yx

),,,( 710 mmmM L=

77 /43/43

In the past– Obtaining the parameters of this model by

minimizing the error function

– Let– Obtaining M by the iterative form

• Using Levenber-Marquardt method

(2)∑∑ =−=i

iiiii

i eyxIyxIME 220

''1 )],(),([)(

TTT MMM Δ+← (3)

∑ ∑ ∂∂

∂∂

=∂∂

=i i n

i

k

ikn

k

iik m

emea

meeb ,

BIAM T 1)( −+=Δ λ

88 /43/43

Fast algorithm for camera Fast algorithm for camera compensation and mosaic constructioncompensation and mosaic construction

Overall flow chart

Fig. 1. Flowchart of the proposed method.

99 /43/43

Translation estimation using edge alignment– Estimating desired model translations from pair of

images

Fig. 2. Flowchart of our edge alignment method.

1010 /43/43

• Finding a set of positions of vertical edges

• Example

)),1(()),1(()),(( jipIjipIjipgx −−+=

∑ −−+=i

g jipIjipIH

iS )),1(()),1((1)(

Condition) 15)( => THiSg

Fig. 3. Edge results of two images.

1111 /43/43

– Position of vertical edges

• Proposed a novel method– Estimating desired translation parameters from images– No longer one-to-one– Defining a distance function to measure the distance of a

position to the translation solution

)470,390,360,325,310,200,180,115,100(=vaP

)390,310,280,245,230,120,100,35,20(=vbP

80)()(

=+=

x

xv

bv

a

ddjPiP

Condition) one-to-one

)()(min),(1

jPkiPkid vb

va

Njv v

b

−−=≤≤

(4)

)(iPva k

1212 /43/43

– Given a number k» Determining the number of elements in

– In addition» The average value of for these of elements

as» Using to measure the goodness of k

– Collecting k as an element of the set of possible horizontal translations

vpN v

aP

Condition) 4),( =< dv Tkid

),( kidvvpN

vkE

Condition) 5,2 =≥=≤ pvpe

vk TNTE

xS

Condition) bb IimageinputtheofwidthWk :<

1313 /43/43

– On the other hand, the sets of horizontal edge positions

– Obtaining the set of possible translations– Determining the best translation from through a correlation

technique» Sum of intensity differences

(5))()(min),(1

jPkiPkid hb

ha

Njh h

b

−−=≤≤

xyS

xyS

∑=

−=

+++−−++=Kyx

Kyxbyxbayxa qyqxIpypxIqpD

,

,),(),(),( μμ (6)

where local meanlocal variance

iμiσ

1414 /43/43

» Normalized cross-correlation

∑=

−=

−++−+++

=Kyx

Kyxbyxbayxa

ba

qyqxIpypxIK

qpC,

,2 ]),(][),([

)12(1),( μμ

σσ(7)

where area of matching window2)12( +K

1515 /43/43

Edge-based translation estimation algorithm– Step1

• Applying a vertical edge detector to find the sets and of vertical edge position

– Step2• Determining the set of possible horizontal translations

– Step3• Applying a horizontal edges detector to find the sets and

of horizontal edge positions

– Step4• Determining the set of possible vertical translations

vaP v

bP

xS

haP h

bP

yS

1616 /43/43

– Step5•

– Step6• Determining the best solution

– Using a correlation technique and a branch and bounded method

•

},|),{( yxxy SySxyxS ∈∈=

),( yx tt

0,0,,1,0,,0,1 76543210 ==−===−=== mmtmmmtmmm yx

1717 /43/43

Motion parameter estimation by feature matching– Describing correspondence-based method

Feature extraction– Using 2D gaussian smoothing function

)2

exp(),( 2

22

σσ yxyxG +

−=

)2

exp(),(

)2

exp(),(

2

22

2

2

22

2

σσ

σσσ

σ

yxyyxG

yxxyxG

y

x

+−−=

+−−=

),(*),(),,(*),( yxGIyxIyxGIyxI yyxxσσσσ ==

1818 /43/43

– Gradient vector of

• Two conditions to judge feature point– An edge point of the image – Local maxima of and– , → a neighborhood

of within a 27x27 window

22),(),(),( yxIyxIyxI yx

σσσ +=∇

),( yxI

),( yxI

2),(

=∇

σ

σ yxI 20),(2

=>∇=

THyxIσ

σ

),( yxP

})','({max),(2)','(2 =∈=

∇×=∇σ

σ

σ

σ yxIyxIpNyx pN

),( yxP

1919 /43/43

Correspondence establishment– Finding the matching pairs between– Let– Using normalized cross-correlation

ba IandI

)},({)},({ iy

ixiI

iy

ixiI qqqFPandpppFP

ba====

75.0),(),(max),( =≥=∈ ciiIIkiIIFPqiiII TqpCandqpCqpC

bababIk

ba(8)

2020 /43/43

Eliminating false matches– Refining more accurately matching results

• Consideration of the relative geometries of features

– Let– Let → neighbors of within a

disc of radius– Assuming that is the set of

matching pairs →– Basic concept of proposed method

• If are two good matches, the relation between should be similar to the one between

K,2,1}{ =⇔= iiiII qpMPba

)(),( iIiI qNepNeba

ii qandp200=R

K,2,121 }{ =⇔= ikkqp nnNP

ji

)(),( 21jIkiIk qNenpNen

ba∈∈

}{}{ jjii qpandqp ⇔⇔

ji pandp ji qandq

2121 /43/43

– With this concept, the measure of goodness

– Eliminating false matches• Average value of • → matching pair is eliminated• Remained pairs →

∑∈⇔ +

=iqipkk

baNPnn

kkiII kidist

kirnnCG}{

21

)(21 ),(1

),(),(

⎭⎬⎫

⎩⎨⎧ <

=

+=−

otherwiseTkiuife

kir

nqdnpdkidistTkiu

kiki

,0),(,

),(

2/)],(),([),(

2/),(

21

1

_)( GiG

baII →_

75.0)( GiGbaII < }{ ii qp ⇔

rMP

2222 /43/43

– Desired solution M

– Some false matching pairs• Using Monte-Carlo style method• Finding a good initialization of camera parameters

bAM T =

⎥⎥⎥⎥⎥⎥⎥⎥⎥

⎦

⎤

⎢⎢⎢⎢⎢⎢⎢⎢⎢

⎣

⎡

−−

−−−−

−−−−

=

4'444

1'111

4'444

2'222

1'111

1000

10000001

00010001

xyyx

xyyxxxyx

xxyxxxyx

A

LLLLLLL

LLLLLLL

4,1''

'4

'1

'4

'2

'1

}),(),{(, K

L

L

=⇔

⎥⎥⎥⎥⎥⎥⎥⎥⎥

⎦

⎤

⎢⎢⎢⎢⎢⎢⎢⎢⎢

⎣

⎡

= kT

kkT

kk yxyxand

y

yx

xx

b

(9)

2323 /43/43

Motion parameter estimation using Monte carlomethod– Integrating two different strategies to obtain different

motion parameters– Using Monte-Carlo-style method

• Concept– Many trying to find the wanted correct solution

• Assuming– Each try can generate a solution– Probability to find a correct solution for each try →– After k tries, probability of continuous failure to find a correct

solution →– Many tries → finding a correct solution

r

krs )1( −=

2424 /43/43

Finding camera motion parameters →– If has elements and ones are correct– Probability to select four correct pairs for each try

– Proposed method to improve the probability for each try• Separating images into grids• Assuming

– All the correct and false matching pairs distribute very randomly– Probability to select a correct matching pair in each grid

M

rMP rN cN

)3)(2)(1()3)(2)(1(

−−−−−−

rrrr

cccc

NNNNNNNN

r

c

NN

2525 /43/43

• Selecting four different grids and obtaining one matching pair from each grid

• Probability to select four correct matching pairs

– Proposed verification process to determine best try• Assuming

– Solution from the ith try →– Consistent error of this pair to

4

4

)3)(2)(1()3)(2)(1(

r

c

rrrr

cccc

NN

NNNNNNNN

<−−−−−−

),,,( 710iiii mmmM K=

),,( iMqpe iM

2

76

5432

76

210 )1

()1

(),,(++++

−+++++

−= yixi

iyixiy

yixi

iyixixi

pmpmmpmpmq

pmpmmpmpmqMqpe (10)

2626 /43/43

– For each matching pair in →– Counter counting numbers satisfying above condition

• Best solution

– Initialization (i=0) → obtaining from edge alignment approach

)(maxarg i

MMcM

i=

rMP

M

6),,( =< ei TMqpe

)( iMc

(11)

0M

2727 /43/43

Parameter refinement through optimization– Method to find desired parameters by minimizing errors

only on position of feature points– New matching set

– Defining an error function

},,2,1,{ MkkM NkqpMP K=⇔=

where ekkIkIk TMqpeandFPqFPpba

<∈∈ ),,(,,

∑=

=ΦMN

kkk MqpeM

1),,()( (12)

2828 /43/43

Updating with the iterative form– Minimization process

• Levenber-Marquardt method

Blending technique for mosaic construction– Proposed two-stage scheme

• Intensity adjustment method– Adjusting two adjacent images to have similar intensities

∑ ∑= =

−+

∂∂

=∂∂

∂∂

=

++=M MN

k

N

k i

kki

j

k

i

kij

Tt

Tt

meeB

me

meA

BAMM

1 1

11

][,][

)( λ (13)

M

2929 /43/43

– Blending technique to construct a seamless mosaic

(a) (b)

Fig. 4. Intensity adjustment: (a) original images ; (b) after adjusting, the intensities between are getting closer.

ba IandIba IandI

∑∈

−=ΔAi

iaib pIqIA

I ))()((1 (14)

bbbb

aaa

WIwxyxqIyxqI

WIxyxpIyxpI

2)()),(()),((

2)),(()),((

Δ−+=

Δ+= (15)

3030 /43/43

• Ray-casting method– More smoothly the overlapping area

Fig. 5. Example to explain the blending technique.

eb

ea

ibeaia

eb

ic ddqIdpIdrI

++

=)()()( (16)

where exponential order for weighting.e

3131 /43/43

Complexity analysis– Assuming

• All images → same dimension

– Complexity of extracting edge features →– Complexity of correlation matching →– Complexities of processing of the others

• Eliminating false matches → radius R = 200• Monte Carlo method → maxiterations = 300 • Final optimization method

– Total time complexity →

II NN ×

)( 2INO

)( 22INKO

)( 22INKO

3232 /43/43

Experimental resultsExperimental results

Analysis of the performanceTable 1. Two sets of synthetic camera motions used to generate the synthetic images of Fig.6 and 7, respectively.

3333 /43/43

Stitching result

(a)

(b)

(c)

Fig. 6. Stitching result of two synthetic temple images generated with the camera parameters.0,0,220,0.1,1.0,242,1.0,0.1 76543210 ======== mmmmmmmm

3434 /43/43

Stitching result

(a)

(b)

(c)

Fig. 6. Stitching result of the synthetic ‘White House’ images generated with the camera parameters.0,0,290,5.1,1.0,240,1.0,0.1 76543210 ====−==== mmmmmmmm

3535 /43/43

Examination of the robustness and sensibility of the proposed feature extraction and matching method– Using several masks with different sizes– Edge alignment method is not used

Table 2. Estimation results of camera parameters obtained from pairs of images shown in Fig. 8 when different feature masks are used.

3636 /43/43

Stitching result

(a)

(b)

(c)Fig. 8. Stitching results when different feature masks are used.

5151,3535,2323,1515 ××××

3737 /43/43

Using a series of panoramic images

(a)

(b)

Fig. 9. Stitching result of a series of panoramic images.

3838 /43/43

Images with larges intensity differences

(a) (b)Fig. 10. Stitching result of two images with larges lighting changes.

(c)

3939 /43/43

Images with some moving objects

(a) (b)

Fig. 11. Stitching result when images have moving objects.

(c)

4040 /43/43

Images with some rotation and skewing effects

(a) (b)

Fig. 12. Stitching result when the camera has some rotation change.(c)

4141 /43/43

Mosaic construction and object detection

(a) (b)

Fig. 13. Mosaic construction and object detection when images have a moving object.(c) (d)

4242 /43/43

Mosaic construction and object detection

(a) (b)

Fig. 14. Mosaics and object detection when images have a moving object.(c) (d)

4343 /43/43

Proposed method– Edge-based method for stitching series of images from

a video camera• Two different scheme

– Edge alignment approach– Correspondence-based approach

• Integrating two methods– Using Monte-Carlo style method to find the best motion

parameters

Conclusions Conclusions