. AD-A093 925 MASSACHUSETTS INST OF TECH CAMBRIDGE ARTIFICIAL INTE-ETC F/6 14/5DETERMINING OPTICAL FLOW.(U)APR 80 B K HORN, B 6 SCHUNCK N0OOI-75-CB043

UNCLASSIFIED A-M-52 Mt.

I muuuuuuuuuuuummum

9 T-

UNCLASS IFIEDSECURI Ty CLASSIFICATION OF THIS PAGE (Won Date Enfoted)

REPOR DOCMENTTIONPAGEREAD INSTRUC11"ISP RMI ORITIOCMENAN N AES BEFORE COMPLE T F0TM

REPORT iMBER nteOligecELab yRECIPIENT S CATAOr NUMR

AIM 572 JA -4" TITLE (and Subtitle) i '' S. TYPE OF REPORT A PERIOD COVERED

SDetermining Optical Flow memorandum.-- " . PERFO'RMING ORO. REPORT NUMBER

7.' AUTHOR(*) 8..I. CONTRACT OR GRANT NUMBER(s)

{ ! Berthold K.P iHorn & Brian G,/ chunck' I"- // 64

9. PERFORMING ORGANIZATION NAME AND ADDRESS 10. PROGRAM ELEMENT. PROJEC'T, TASK<

( Artificial IntellIigence Laboratory AREA & WORK UNIT INUMBERS

545 Technology Square , ...... // -- ;,J /

Cambridge, Massachusetts 02139 (l - -I I. CONTROLLING OFFICE NAME AND ADDRESS M -R--ORT DATE

Advanced Research Projects Agency// April 19801400 Wilson BlvdArlington, Virginia 22209 28

C4 MONITORING AGENCY NAME & ADDRESS(l! differeint from ControllingI Office) 15. SECURITY CLASS. (of this reporT,

Office of Naval Research UNCLASSIFIEDInformation SystemstArlington, Virginia 22217 jfu 15a. "'",EoLASIIAI, iONR

10. AISTRIBUTIONSTATEMENT , ..,, n I(of thisl i RouM

Distribution of this document is unlimited.

In DISTRIBUTION STATEMENT (on the obttrlo t atered in Block 20, I d hich as s rot Rhpttt) appa r

IS. SUPPLEMENTARY NOTES ",

None '

19 KEY WORDS (Continue r I necessapan i v s ttdenriy by block number)

Optical Flow Intrinsic ImagesMotion Perception

a.nCooperative ComputationQ-

20. ABSTRACT (Continue O reverse sIt nOcssary ad Identify by block number)

S>0ptical flow cannot be computed locally, since only one independentLImeasurement is available from the image sequence at a point, while the flow

__J velocity has two components. A second constraint is needed. A method for

LL. finding the optical flow pattern is presented which assumes that the apparent

velocity of the brightness pattern varies smoothly almost everywhere in the~image. An iterative implementation is shown which successfully computes

the optical flow for a number of synthetic image sequences. The algorithm..

DD IFAMPMR 1473 EDITION OF I NOV 65 1SOBSOLETE UNCLASSIFIED k'r /'Pe

SECURITY CLASSIFeICATION OF THIS PAG (to mf ntored)

- is robust in that it can handle image sequences that are quantized rather coarselyin space and time. It is also insensitive to quantization of brightness levelsand additive noise. Examples are included where the assumption of smoothness isviolated at singular points or along lines in the image. ..

0 r 4

T

NI ssachusct is In stiltte of, I ch nology

Arti ticiA Initelligeccnce I ah( uxtory

V. 1. NICInlo No. 572 Apli ii 980

IDtltcriing plical Flow

JL-i thold 1K. 1), I (1 jd Ihiaii G. Schiinck

.\fisi;nct 011ticall th ( 11(\ Ciiit tle coiiiptitcd I(4calti Siniceol &()Ile in dcpcndciit nciilliitIS iilai

~I .m i lilate SC1,g u11ciicc ;I( It point. Mille tilc 101 \'ciocity 11as t~kO coinon)McntIS. kA)I( Cond 1mntrint

IS ICCdcd. A HIciliod o6r tildi di tlw piLdI t10\N pIlMMcr k prcsciic1Cd M hiki &asilleNC 1ilu1 tilc .ippir-cli

\ Ctocii\ ili 111k, brlgilCSS p-ittcril slics siioodtii\ almoi~st cecr "s hcu ill [he iiiiac. An itciuii\k C unpienlienl-

tutioll IS ShiOV I \dlliCtl ,Icc('.jiIlj\ InlI)tIt0S tile opntijI tIowfo Ia iint1hcr ot s\ 1iutctic ini.gc Seqcetiics.

lic ag itcoil k1 5 obust ill (lit it canl liandic inliagc ScqnMIcncc 0th11 arcQ (UII~InIid rithctl ia'L~ inl spaCe

;ulk id 1111. It is 1lso iliscnSsli\ c to tianti/itionl of' brigtness Ickcis And aiddihi~c nloic. Fvimpics arc

indlidcd \de tile thcISSuliliptioll ot snuloothic~sS is \inlted at Singlilr poinlts ol along lilcs ill thle Image.

~ c hoi~kdginn I Th is icpoui dcscl i cs 'csc. i ch dolnc at tile Alii i tl I n(cii gccc I atIbolIlr of thle

Nliss,"Idiilts 1ISlitlt of, I cciinlouit Suipport 1,01 this r-CSCarch is piomidcd inl pairt bh [Ile \dunced

Pccn loje~ qcts A gcnc\ ol, ilc Decpartmnent of' Defenise uinder Mtice oit Ni\i Re'Ii(search couldiact

M00ttt1 !5-C-0(43.

81 19x1

I)(I'lnninfin ()plivaI low Padge /

1. Introdutctioni

Optical flowv is the dist ribuit ion of'appaircn t veloc ities of itimfcnieilt of' hrightness pan em s in ml imoage.

Optical flow can orisC Ijoin relative motion of objects and the jewer (Gibson 1950, 19(60). on~ctitently,

optical flow can give important information about the Spatial arrangement Or the objects %icy. ed and the

rate of change of IthIis arranigement I(Gibson 1977). 1 )iscontintities inI thle op~ticafl f1mk .1 I I,: Ip i II ,cgl I],I It-

ing imoages into regions that correspond to different objects (Naik.tyanti & I oonik 1 974?). .\ttefopts hive

been nIle topo suich segntcntation using difl'(rneflS hetAeeit S~lCCeCi%0 ntdgk-fi~ (1,61 0i a!

1977, J~tin ei a4. 1979, bill & Nagel 1979. hinhm & Ni iirphy 1975, Nagel 1977). Sonic cci pIm~em ' have

considei ed tile p rf ibi eml ofrev e CO Ii ng thle Motionis of'objects Felaltiv e to thbe '. 10% Cf Iil on 01 icp c~l .floi w

(I Ladani c/' a. 1980. Kociaderink & van 1)oorn 1975 & 1976, Longff Aiet- I Iigpiifs & h/Imrfdf 19)71. 1 P,il yn

1979 & 1980). 1n ti sme cases inlFormation about tilie shap i Pct a ohjci ma y ak he re i -c r-[ca ( K (efi no k

& xan lDoorff 1975 & 1976, Clocksin 1.978).

SIeSe ~iMPCS us hgi ii by aISSo ii fi ig that the flitical flow has already beetl dcteili filetI AlIthoftuighi Somle

reference hlfs beeff ma~de toi scheiles 1 or conPipiig tle 11 AV fr1ofi suIccessiveC viCIAS of fNric (I en ff~f tia

& Thomopson 1979. 1 Liaiii cA. 1980), the spcifics of a schenie fow deteIifInflig th10 flowk f10fi1 (Ile iflIfg

have not hcui dcecrihed. Related work has been doffe iii an atteniplt tot faammuuloa a Ilodel 1for the sifoit

frange Iliotiaff dctectiai pilfcesses ifi hiumuani isioff (Bmfo~di & I 1lu1i.iu 1981), Nhfif & I. lliwin 1979).

lTfe opticail fifiw Cannfot Ilt ctaiiiputed at 1 pofifft in theQ imffialle indepeuildenthy oh ffeighbofinge polt

withoot inftroducing additional coifstrailfus, becaueC thle vclocit) field ist each imaige point lifds wo coirn-

plleffts while thle Change ill imiage brightness ift .1 poift inl thie iige plan1fe dffe to fmotionf yields oni) (fli

constraint. Cfonsider for example a patch of a pattern Where hiuightoiess v;Irfes ifs af hiioii of, one image

cooirdiniate buft not thle otlher. Movi'emlent of' thle pattern iii fffe directioff AtN the brC iitufICY, Jt d pI)Irtii fflr

poitl. bilt moftionill inte otherl (lirectitfif \ields no change. Ilifi1" compionenfit't of' fffuvelifnt ifl [lie hafter

directionl cannot he computed locally, Additionial confstraifnts mfffst be ifftirfduc' to ff'lll detelilme thie

flow.

2. Iheat ioii~lip to Objet Nlto ionl

I lie rclionvlhip between tlie opticaIl flill [f h I e lille m(ithhe \locitie\, of' obtecil thle three

H~ornf . Schf flck . I pn I

Page 2 Determining Optical flow

dimensional world is not necessarily obvious. We perceive motion when a changing picture is proiectced

onto a stationary screen, for example. Conversely, a moving object may give rise to a constant brightness

pattern. Consider for example, a uniform sphere which exhibits shading because its surface elements are

oriented in many different directions. Yet, when it is rotated, the optical flow is zero at all points in the

image, since the shading does not move with the surface. Also, specular reflections move with a velocity

characteristic of the virtual image, not the surfiace in which light is rcflectcd.

3. The Restricted Prolbem I)omain

For convenience, we tackle a particularly simple world where the apparent velocity of brightness

patterns can be directly identified with the movement of surfaces in the scene. To avoid variations in

brightness due to shading effects we initially assume that the surface is flat. We further assume that

the incident illumination is uniform across the surface. The brightness at a point in the image is (hen

proportional to the reflectance of the surface at the corresponding point oi the object. Also, we assume

initially that relleciance varies smoothly and has no spatial discontinuities. This latter condition assures us

that the image brightness is diflferentiable.

In this simple situation, the motion of the brightness patterns in the image is directly determined by

the motions of corresponding points on the surface of the object. Computing the velocities of points on

the object is a matter of simple geometry once the optical flow is known.

4. Constraints

We w ill derive an equation that relates the change in image brightness at a point to the mo tion ofthe

brightness pIttern. I et the inmage brightness at the point (X, y) in the image plane at time t be denoted by

I(x, y, 1). Now consider what happens when the pattern moves. The brightness of a particular point in

the pattern is constant, so that

dE 0.dt

A, I Memo 572 IHorn fSchunck

IPacriiing Optical H/ow Page 3

1\ifng file chdin I-il forib di llcrent jationl we see that,

OE d OE d V 9pOx d t Oy dt Ot

(See Appendix A flait a ore dailecd derivation.) Ih'%c let

dz _ dyU = anddit d I

di citl it is eaIsN it) see that we haII e a single linlear eq Limil atf)1 tile tw Liiiknow iis it anid 1).

Eb.1 -j Y +C, EI- I 0,

Mdiere %wha a1l\Isoi initroduIced thle additional ahhre~ ationls 1', F1'. an1d PI fim tile paula1 der,.aifves

(1t itli'age brightness wkith respect to X. y and t, rcspecti~ely. Ilie coIlhtraiil i (ite loca1 lm %c \locity

exprIessed I)' this; equatioli is iIIlStraited inl lignie' 1. Wr tite equion110 in still an10ther Wily,

(,ii,.(It, V) -E $I'l I Lis I hie Compn pitenlt oft thle opt ical flow inl thle di rct ion Li 1the h rilit nciss grade it (EK LV', equLa.ls

WVe Llllohoe%.er determinle tile coniponcit (if moveneit inl tile directionif li,i jihicss on1-

oi-us. Mt Irli agls lo tile brightness giadieilt. As a Loniit-tice. tile fl %clocit% 01i, v) LcIiilot Ile

5. 1hi CSillooh lC%ine ( Coiws taint

II elef. polint of' the brighit ilss pilllei If Cani ll1(iSc independeitllY. there I, little hlope oh' recmi elIng (lie

N clot itics. More coilllofihy wc iQWk opatque ohjts oli fiiiite Sim1 unidergoinig Irigid Illitill or. dlComiiiltoii.

II ch~ unc~ik s.I f',2 7

Page 4 /)ciemunin I 1 ij ,',')V

V

(EXVEY)

0- U

constraint line

Figure 1. The basic rate of change of image brightness equation constrains theoptical flow velocity. The velocity (u,v) has to lie along a certain line perpendicularto the brightness gradient vector ( 4 l)in velocity space.

4. /. Ateini, 572 lHorn t( Xc/uick

Ietermuilning Opt jeW HFow I'age 5

III this Ca~sc neighboring points onl the objects haw similar velocities and tile Nelocity field of (fhe bright-

ness patterns in thle imagc v'aries smoothly almost everywhere. D iscontinuities in Ilo%% call hie expected

Mwhre one object occludes another.

One way ito express the additional constraint is to) limit the difference between the flow 'elocity at

at point and thle average velocity omer a small neighborhood containing thle point. I Llquivlentily we Canl

mlii oe thle stum of (lie squares of ilie I aplac ins i the z- anrd y -compo nen ts of thie fh iw. TIh e Iap.cau

of u and v atre definled ats

611u (!,u VV ov C)V 2u .- 4 +-- - and V-v +? 2

Inl simnple situat ioins, both11 I ap fac iansare zero. Iftic dietw ertiia uslates parallel to a h1It object, rotates ;1bout

aI fine pe rpend icuLla r to thle sourface or travels on hlog n il ly to dhe so if 'Iice (,aS'; liil ug e 'is eel i\ C po 0JeCr n),

then the second partial derivatives or both uL and v vanish. Note that our approach is in contrast " ith

that of' Fei c n a & Iiioiiipson ( 1979), who priopose an algo rithmi that in di rectify i net prafOr tes additional

assumlptions such ats surface snmoothness or object rigidity.

6. Quatiz at ion and Noise

images may be samipled at intervals onl a fixed grid of points. While tesselations oilher than tile

(tln ious one hla\ e certain adhitages (Mersereau 1979), Gray 197 1), for conlvenlience %\ e % ill assumnle that

the in iage is samip1led on] a squia re grid at regulia r intervals. ILet thle imieasu ired briigh tniess be L' ,.a tl the

intersect ion if the i - tIi row anrd j - I columi n i tilie k -tii image franie. Ideally. each measo enie n shouId

be anl aw rage m'er. thle area of at picture cell aiid over thle length ofi thimre Internal. in thle experirnents

cited liere we Ii aw t. ke ii samples at discrete point illo space and littile. in stead.

Ill additionl to being (jiiaiiti/ed in space and tinie, thle measm-iiieiOiu\% ill Ill practL'ie:C he quanii/ed in

h ugh itessas A elf. I u ilhe r. noiuse will be appa reiinti i in 5L m e eits obt a i ed ill 1aN I ICal Systeiii

Hiot-it C .%hunck A. 1. Ateno .S72

Page 6 Delerini'nig Optical Now

7. Estimating 1he Partial IDerivatives

We must estimate tie derivatives of brightncss from the discrete set of image brightness measure-

ments available. It is important that the estimates of E, E, and C, be consistent. That is, they should

all refer to the same point in the image at the same time. While there are many formulas for approximate

differentiation (Conte & de Boor 1965, 1 lamming 1962) we will use a set which gives its an estimate of E,

E, Et at a point in the center of a cube formed by eight measurements. The relationship in space and

lime between these measurements is shown in Figure 2. Each of the estimates is the average of four first

differences taken o~er adjacent measurements in the cube.

E. r{E,..j ,.k - E ,j,k + Ei-Iq,j+,,k - Ei-t,1j,k

+ Eij + I,kA- - Ei,j,k-l + Ei--,j-l,k+I - Ei+I,j,k+i}

-- E,,+lj,k l - EL,j+l -E i ,j t,kl - i-I ,k4t}

4

b", R 1 I{ Ei,j,k± + - Fl~~ + Ei--Ij,k-p-I - E4 ~~4

_1 Li,j-+-,k+i - EJi -kk +E,tI,j+1,k.f I - Et ,

I lere the unit of length is the grid spacing interval in each image frame and the unit of time is the inlage

frame sampling period.

8. lstiniating the Laplacian of the FIo- Velocities

We also must approximate the Laplacians of u and v. One convenient approximation takes the

following form

V'u F K(ft.j.k -- u1 ,.k.) and V2 v - K(f,,j, - vJi. p

licrc the local averages ft and f, are defined as follows

ft, ,. --- 1{ , I.j.4- + u I i ., -I-- It .4 u .k + itij 1,k)

+ 112{tIi I,j--l,k + Ui -- ,j4 1,k + U, f-I. 4 I --+ lt i 1 ) -I,k}

V",J,' = --{_ I,.j A- v,,. -..- , + V, 4- ,j,k + V. k - , }6

1,k+ Vai-Ij 4-.ik 4 V, I + 1,k + ?

4. 1. fe~no 572 Ib)" it Schujick

DI~ccrfifitlg Optical low Pa:ge 7

Figure 2. The three partial derivatives of image brightness at the center of thle cubeare each estimated from the average of first differenccs along four parallel edgesof the cube. Here the column index j corresponds to tho .r direction in the image,the row index i to the y direction, while k lies in the time direction.

Iforn S 1 /At iu *S7k

Page 8 Ielermining Oplica! P on

11Te Proportionality lactor x equals 3 if tile average is Comnnpu ted as shown and asso uing aga ini III;)[ the

unit t f'Icngth equal iis the grid spacinig intierv al. I iguii 3 illuLst rates thle assignment olI weights to n e glih w -

iitg points.

9, Minimization

[he Pioblen ithen is to miiinize the suni of the errors in) thle equation for tie rate of change of image

brightness,

a1nd the estimiate of the departure from smoothnecss in thc velocity flow,

S2 =f,_U) 2 + (jD __ V)2.

What shouild he the relative weight of' these two 111ctors? In practice thle imlage brigtness tInt1eeins

tt ll he corrupted 1). quantiiation eiior and noise so that we cannot eXl)CCI 9b to heC identiiCalIly ein 'I his

quntt~~ill tend to have anl error magitude that is proportional to) the noise inl the mleastirelient. This

Lict guides us in choosing a suitable \%eighting factor, denioted lby a ais will be seen later.

cI e(hle total error to be mininiied be

g2_ a~g' + S2

I hie in ii it 1,a1iml is to he accot np1ishcd by Finding sulitaible Nallues for1 thle optical tI~ \ e bc it x ( i, u).- We

dlicrniate 9 to obtain

- -2a2(ft -- uL) 4-2(/,.',. -f 1 v 4- ~)

c9u

- -- 2a2(ip - v) 4-2(Ik7,s +- bjv + E,)b7.Ov

Settinz these two derivatikcs equial ito /ero leads ito two equations in it and V,

( 2 L I2)U + p, ,v (ats- j)

I 'b,,s -- ( 2 + 1.,l) V =(aF I

A1. 1. .iImo.572 Ivn& tiw

1/12 1 /6 1 /12

16 -1 16

/12 6 12

Figjure 3. The Laplacian is estimated by subtracting the value at a point from aweigjhted average of the values at necighboring points. Shown here are suitableweights by which values can be multiplied.

Page /0 Detennining Oplical IHfow

The determinant of the coefficient matrix equals &(a - E' + E"). Solving for u and v we find that

(a 2 ± E)u = + (a'2 i f _ 1,,,p E ,

x + E Y+ u -- --/f)

(a2 + 2 + E2)V = -- :;fl + (a2 + I.,'I)b ;Et, .

10. Difference of Flow at a Point from local Average

These equations can be written in the alternate form

(a2 + E2 + E2)(U - - ',. f + EVf) + E,]

(a2 + E2 + E2)(v - 0) = - E[EfIt + E,, + Edj.

This shows that the value of dhe flow velocity (u, v) which minimizes the error g2 lies in the direction

towards de constraint line along a line that intersects the constraint line at right angles. This relationship

is illustrated geometrically in i'igurc 4. The distance from the local average is proportional to the error

in the basic formula for rate of change of brightness when fl, V arc substituted for u and v. IFinally we

can see that a2 plays a significant role only for areas where the brightness gradient is small, preventing $hap h,/ard adjustments to the estimated flow velocity occasioned by noise in the estimated derivatives.

This parameter should be roughly cqual to the expected noise in the estimate of E - E2.

II. Constrained Minimization

When we allow a2 to ted to zero we obtain the solution to a constrained linimi/iation problem.

Applying the method of Lagrange multipliers (Russell 1976, Yourgau & Mandelstani 1968) to the

problcni of minimiting . while maintaining 9b 0 leads to

_1 _ ,j_ )(E' + E",)(, - b)= -,,'Eft + L,,,, + E,.

We A ill not use these equations since we do expect errors in the estimation of the gradient (EI, Ey).

A. . Aleto 572 Hlorn d .chunck

'NNW

Ih't pnuinhl' Optiall IEh Ial 1

V

(U'V)

constraint line

Figure 4. The value of the flow velocity which minimizes the error lies of) a linedrawn from the local average of the flow velocity perpendicular to the constraintline.

Page/12 iemnli pcaHot

12. Iterative Solat ion

We now have at pail.r of equat11ions 6wo C.ach poin t ill tI e uii age. It %% ud ie v'ery co stlIy to st Iv OeLIc

equaLtions simlowtaneously by oine of the standard mi etIho ds. such ts ( lIss-JOrdaii eliiatiaton (II ii ii 1ig

1962, 1I ildehrand 1952). The corresponding mnatrix is sparse and very large since the number of tows anid

columins equals twice the numbller of'picturc cells in the image. Iteraitive met-hods, such ais the (auss-Seidel

method (I lamming 1962, 1I ildebrand 1956), suggest themselves. W,- canl compute at new set of'% clocity

estimates ( up V11 v' )From thie estimated derivatives and thc average of tic previous velocity estimtates

(it", v0) byUri1 =I'- EX!E.JSn + Efi" + h'j1/(a' - F' + E')

-Eu,(E.is + Ejjf"iP -F/(& + Ex + Ea)Y

It is in~terestinig thalt the neCw estimrates at a particuilar point do not depend directly on the previoos

estimates at the same point.

13. illing in Uniform Reg.ions

Inl parts of tie image where 1he brightness gradient is zero. the velocity estiniace m ll simply he

ave rag~es Of. the neighbhoring velocity esti mates. TIhere is mio local in Ii oat ion it) oiiONi:-.in tie ippmt

velocity ofimotion of the brightness pattern in these areas. Eventually tie valuies aron d suhl a regionl vmll

propagate inwards. If tile velocities on tile border of the region arc ill] tihe samne, then pn ts inl the region

%ill be assigned that value too after at stiflicicnt number of iterations. Velocity iniformation is thul[, filled in

from the boundary of a region of constant brightness.

14. Number of Iterations

I f the %allies on tIle border are not all[ the sam~e, thie values filled in will correspond to the Solnltiomi

of lb e I i1place eq nation For thle given bionidarmy cond it ion (Ameis 1 977., MIil1ne 19)53, Rici1tileCr & Nit ilt) l

1957). I he pro g ress ofthiis filfling iml phenomena is siiliar to the p~ropaga tion) e ffects inl the solUtil moftile

hecat eq oation, whle re tlie tillme riitc ol chanige Of tempe mlat i C is P roport io nalI to thle I ap lacia ii. ThIiis gives

US a1 mleans if' understanding tile iterative method iil phlysical teins and of estimlating tile numbher or steps

A L. Mtmo.572 Hornu 4 .'chunck

ItkimIcL!. I lie numinhc of' itchttioll h~toiIlk hcii ihaii1 (lie Crosy v-~oCCIInI' Ow 1 hiiw 1 ci '"

Ilk filled ll. It' tile -e of, such region~s is not know.n ill .Iucec oc iiu 111 ic WAlie c !h)!)c I, I,

I II , I \ Iuc I uve csti I I t C.

15. Uighitiii~s of Colot raint

'AtWhl;Iiun~ in ~i ieginis a' . liicarI tuctiol of' Iic igc~k c:00IdiI1A(C1 \\, L,111 OuiIk 01biil !h"

1.1 f opm .il !I('%% ill (Ile dirct ionl of' [tc gradient. HIC I colioPOncnt M I i~l it if nlI i n ",1 il,- iI I I

thc hooiiii!Ii I, 01c 1i~jou -is dibeC~hd lictoic. Ill gCI1mil (Ile SAU11iid 11(V nnd kIll if\ k (C ii[111l'

iii ic.'iuiii ulit'ic Ow~ Illii~'Ihi'ss gadiculit I; not too) suinall ail \,IIrics l 1iccluuuil Iliiil~ii P1!(fi l1W

~iIt i) 111iluoiiu of c"icl pint. Iuio x ioikuut fluictulutionus Ill bi'iics oll (ho uiii, il 'lV fit

16. Chijce of Iteratke Scem

AS .u pudI Li flfuCI- 011C IMs a Chioice of' how thec Iilns ire~ tIle ilcilucd '\ flh thc inch "t,

Oi fit-, 'inc limiud. onei~ Could i(tIc lIIUI ilti h Sliji Iis "IIhili/c(I hCkI0I- t\ I11iciii 10 Ill', Ihil11 li'C

Irliniu. Oil Ole utiu'i 11.11ud, tc .1 goodi inlitial1 fluC\\ iini' u\ lck oiii utic i(ciitiuuii SIY 1tiiiiC'iV.

goodi id nin'li :muui for (It t uld)ti 1301l vCOL]iC; is 1tlIuII\l u i.il.1hik: tiuii (tic ibS lll i1c WI.

I tic oft ii~~,u (ie tIttcr ;IIa)IOd)10 C Alld I Iitii It) d.it %kt h iiWCc? W iiiu 'Cl oS i ll lillic: 111d

thu1tubA (I' iui~I li i.J iliwAi %luoctc illculiul'u. lcu lt'i'i Atlcx u o ut, i tll uhf uluic I '-I.I lt Islui

*\I pII ii- (.ilf u iii kicui(.i No btauu ied huos 1kl 'Idulflo li o m O ,cu~'il pci li t'y I~ 11""k lcs',1111.1 0.l

\icmiiscd hunIdcP m upiuuith 111.1i s Iu )W )OP'ull wuucus.,111

'11)(I Wid td o..Nu/iiooeippintdnAotpx' 111cpl~k11wl 101h j. I ',nCl P~ /2

-i i*. iag

Page 14 Ietriining O)ptical / olt

17. Experiments

The iterative Scheme has been implemented and applied to image sequences correspondinig to at

number of' simple flow patternls. T[he resuilts shown here are on it relati~ ely smnall image of' 32 b) 32

p ictur e cellIs. T he hirigl tn Il east irelii lns Were intlention11ally c irri ied by app ro x iiiatel) V1% niiose amid

then (iianti/ed in to 2 50 levels to simul Ilate at real inmaginI g situiat ion. The unude rly inig sutrface ic hlcct.m ce

I I it tem In w as a h il Ctar con I b il n(I on It o spatiall y 01rth0oi i I Sinll s I ISO iT Ci I a r k AjICIgth Itwds ChOW1e to I 1%k

reaso nabl y St long brightnless gradient ISwit hout leading to tmnde rsanit p I Ile prItblemi s. I )i-sc i ntillinLiei were

J% ied to ernure that thle required derivatives exist everywhere.

S ho wn it i Figure 5 hfa r example are foumr friames of' ia seq tence of' images depicting a sphere II ital in

about anl axis inclined towards thie viewer. A stitoothlly %arying reflectance pattern is paiiited onl the surf'ice

tif the sphere. ['he spherie is illuminated unillbrmly from all directions so that there is no0 shading.

18. Resuilts $

[lie first flotm to he investigated was I Simple linear tritslatiiin o* thle entire brightness pattern. 11wl

,es ill 1)LM in110%ititd o is Shown as at needle diagram ill I igt are 6 hiir 1. 4. 10 i. d 0.1 ite it itils. I hie

ceftil lm a e %it l c it ics are dep ic ted as Sholrt lililes, show in g the apparlent displace nell t duin g 0 lle filiIC

step. Ill tis cumiipile it sinle (n me step wats ta ken so) tha~t tile coimptutat ion ill ie b asedl oil julst two ii i ,ml"es.

liiIanallk the cstlillitCls oh, floA velocity are zero. Consequently the first iterationl shows %ectors, in the

(Il actiti i oh' [lie bright nless glad il. a ter, thie est inae 1.1 Spp roach thle correct s'ah es inl .111 pa its of, tile

iIimi!,c. lew cha~nges occur ifter 32 itertitlns when (lte telticity %ectors liuxe error" of' abioutt 10%11. [hle

eStlili1i1Cs telld tol lie ttoo smiall. rather than too lirge, perhaps becautse of'. a tendency t ttinldelestilllLe thle

de ri at it s. I hle ttU Clri c it u ~tr. as (i ie inigh t ex pct, wI eio tIie h rigi liess g r.id icllt is silall.

Ill thle sectlid eXpelllleut tine iterationl w.s used per tunei step oil thie Same iiie.ir. traiislattiiil image

seqtlleikc I hie resilting conilpuled Iltiw is Shiownl Il ligule 7 tor 1, 4, ti. anld 0.1 lttlle steps. [hle estimiates

pr .1110k s I tIlcorrect il tICs Imore rapidly anid do no t have at tendency to bie tio 10smll, as. IS ite prIe ii lls

CS pI~v illiilt. I'Cek Changes oXCcur after 16 iterations when the velocity vectors has C errors of abotlt 7%1. [lie

storM errors txciir. as (tie migla explca. where tile noise il recniIeSiirmnents of NlIrgh ies's V% is Ivillst.

A. L 1. lint 5/2 lt rm J .c/,nck

At i

hiqure 5i 1out fitrues mil of a sequeonce of imaqies of aI !phowre ot'Itim) i1hout ani

A. 1% HIiwloil to)wardis tlit viewer. The sphere is cove-red with i polemv whichvaies! .moothrly from piaco to place. The SPhere I-; pImtrayod .qiota t,'xut, lrqhtly4 t~extured hok.1rmi;dir. tIiiwim sequenlces like thesie w )roce(!t( hi' the tiltical1 flow11(jorillim.

Page 16 DleCicnining Oplical flow

S,.~~~~ ~~~ 0i dii I//'d/ IddJd

1 1 Al~l Z//////

...... ~~ /I//// V llll I II/

% ~~~ ~~ ... . ./lt/ . I ... .% .llli .l~

A B

I I & La £LL dddil-/71 i S IdIddddg gld dddild//~ldd1J.J J

illililZZZ IlAill /]l I1//Z/11 I//I/Iiiljl

'll 1ZiZZZZ6 I iiii Z/Z '1 H IZZ// I

111111111 1'l Zlllll 1/ /// I///1//ii //i/Ill

I//i//I/I / ,,,,]iJJ]i/ I/ii/// Z ///ZY////// TI/////1111

ZZZZZ//ZZZZZ/ Y1Z1111 / 1/11/ IiiZZZZZ

cells in the x direction and 1.0 picture cells in the Vj direction per time interval.Two images are used as input, depicting the situation at two times separated byone time interval.

A L. Motto 5.?2 IHorn A.cwc

/ p i'1 I'loft,1

Y. 2

WMIY

z j Y V/ 'it Y I

I~~ I Z

A B~ue7 o atr o ptdfrsm l tasaino rgte! alr.Il

II ~ ~ ~ ~ ~ ~ lp are/111i1 /lon Her one Ill//i/IL' is/1/1411 / /// e'

1111iile /fe /, /, 1ti an 1 14 lim /jt 11i o /m1t/1/1117/1* !//vrelc is mor 1*'~ i 111n 1/111 th '1 /11111141

accurately 111l1111 'i/. 4 H/111 1 /11

Page 18 l)ehr, millng 0p)/ ll I h'

While individual estimates of velocity may not be %cry accuiatc, the .aclagc mver the whole Image %as

within 1% of the correct value.

Next. the method was applied t) simple rotation ind simple contraction of the brightness pattern.

The results after 32 time steps are shown in Figure 8. Note that the magnitude of the %clocit, is piopor-

tional to the distance from the origin of the flow in holi these cases.

In all of the examples so Cair the I aplacian of both flo% %clocity components is /cro evci "here. We

also studied more dillicult cases where this was not the case. II partictlar, if we let the inigniltode of the

velocity vary as tic inverse of the distance fron the origin we generate flow around a line vortex and two

dilensional flow into ia sink. The computed flow patterns are shown in Figure 9. I lere te colllutatioll

involved many iterations based on a single time step. The worst errors occur near the singularity at the

origin of the flow pattern, where velocities are found which are much larger thian one picture ct. II per time

step.

Finally we considered rigid body motions. Shown in Figure 10 are die flows computed for , cylinder

rotating about its axis and for a rotating sphere. In both cases the ILaplacian of the flow is not zero and in

fact tile I *placianl of one of' the velocity components becomes infinite on the occluding bound. Since tihe

vec;ities themselves remain finite, reasonable solutions are still obtained. The correct flow patterns ue

shown in Figtre 11. Cinpariiig the computed and exact values shows that the worst errors occur oii the

occluding boundary. These boundaries constitute a one dimensional subset of the plane and so one cal

Cxp'ct that Mhc relative number of points at which the estimated flow is seriously in error will decrease as

the resolution of the image is imade finer.

In Appendix II it is shown that there is direct relationship between the Laplacian of the flow velocity

colmponent, anld the ILaplacian of the surface height. This cal be used to see how our smoothness con-

stiant % ill tiii for different objects. For example, a rotating polyhedron will give rise to a flow which has

iCio I .l1,1kial exceept on tie iinmge lies which are die projections of die edges ofthle body.

19. Sutnu:iry

A ethod was developed for computing optical flow from a sequence of images. It is hased on the

observation that the flow velocity has two components ,ind that the basic equation for the r.te ofchange

A. . Afemo 572 lIh,, u .S'cltmck

- -- - ---

I', T I r III

. . . .. . . .

Figutire 8. Flow patterns computed for simple rotaition and simple contraction of abfiglitncss patterin. In tile first case, the pattern is rotaited about 2.18 deqines pertime step, while it is contracted abouit 5". per time step in (fie Second case. 1Meestimates aftei 32 time step% are shown.

kkk;UCAI I-s,.

Page 20 Delerminini Op/ical Flow

L QL i t e f ~ pt P P P P P PP P P P P P P

k~ II I m ~ r l L l mq t i II +P + P P P P P P P

P P P P

V

- - - - - - -

P P P P P

i . qqI I t r r t I / P , P

' A - -

I

A -B

e+++ ] ...........

Figure 9. Flow patterns computed for flow around, a line vortex and two dimensionalflow into a sink. In each case the estimates after 32 iterations are shown.

,f. 1. ,tleino .572 hIur if Sc'hutucck

- - - - - - - -

* P ~.o . . . . . . \ . .

• • t % " ..I I. . .. ..-.-... '... .... .. ..... .. -" "

..... j

. . o.

A B

Figure 10. Flow patterns computed for a cylinder rotating about its axis and for atotaling sphere. The axis of the cylinder is inclined 30 degrees towards the viewerand that of the sphere 45 legrees. Both are rotating at about 5 degrees per timestep. The estimates shown are obtained after 32 lime steps.

I,,,l ,' Xcliu & ,. /. .ihCmP, .c/2

. .. . . . - - --- -- ---.. . " . . . . .

Page 22 /)elonmillg )pila Hlo'

* * ~~ ~V- .-.-. . .

S...-.- . . • . . .. .

~do .. ...- . . . ...................................•"•............................... '' \.....

I- .... . . . .... . .

. .. .. . . . . .

Figure 11. Exact flow patterns for the cylinder adthe sphere.

.1 . W1ono.572 or CSiuc

d

Itcmuno,i O)ptical Flow PA ge 2

ot, iiiiai~c h[nghti@;s prom ides 0on1 0onC COnIStrinit. Sirio01lucss o( thie lo% %%,s Inn odueed is5 a scond

colistraiiit. Alil itcrilti~v iiiedlOd toir Soh ing (Ie I-S1iillo. CLMqit~n'l) IS [)hen de'C]Iopcd. A limlple i01-

pliinillntioii prlo. ied \ isii.ii Collfirlwlit ioli of, conlvel gk:iicC OF C th0e 101 saitoil in th M Irii it icedcl diaei Alis.

Illaiplc1) *rse eid diffrent types of opticail flow pAtterns e rc studied. 'I ie~e include:d e xse \\ hei ': the

I 1 ilaicianl or (thc flow wils z'ero 31nd eaIses where it beCamel"I infinlite at singular poinits ()I Ilml! buniding

0,We O* Lit:e Ikltlloi, (I barn) Nouhd like to thank Prol'essor lI A . Nage!c I'M his hlo-yitllit) .I h timsic

LjiI~t(IFS Akre coiceiked dii a visit to the hjoheisity of I Janbi, stimiulated b P'i or N~ieel\s

]l 'u-stI.Idiing initerest Inl notion vision. 'Ilhe other ziuihoi. (Schlick) would like to thank XV. F-. I ( l illiso

Jild V. I Iildreth tbr uiiu inter-esting discussions and much knowliedgahle criticism.

Hl pid chu mcA A1 1. lfco' '2

Iuge 24 Determiiing Optical/ low

Appendfix A. Rate ofr aage of Image Brightness

Consider a patch of the brightness pattern that is dispiwced a distance bx in the x-dircction and by in

the y-direction i tinie 6. The brightness of the patch is assumed to remain constant so that

E(z, V, 1) = E(z + bz, u + by, i -- 6).

Expanding the right hand side about the point (x, Y, 1) we get,

E(, y, t) + 6z +6 + 61 - + E.

Where c contains second and higher order terms in 6z, 6y, and 6t. After subtracting E(X, y, t) from both

sides and dividing through by 6t we have

b x (9, y ,9 + -+ 40 E + ()C 6t) 0 ,

where 0(6t) is a term of order 61, and we assumc that 6x and by vary as 6t. In the limit as 6t -, 0 this

becomesc7Edz + + L = 0.

S oy dt 0-a O

A 1. A feno 572 Horn d Schunck

Deternining Optical Flow Page 25

Appendix B. Sinootliness of Flow for Rigid Body Motions

Let a rigid body rotate about an axis (Lo, w,, w.), where the magnitude of the %cctor cqtla;, the

angular velocity of the motion. If this axis passes through the origin, then the velocity ot a poult (X, y, z)

equals the cross product of (w1, wu, wL), and (z, y, z). There is a direct relationship between the image

coordinatcs and the x and y coordinates here it' we assume that thc imagc is gcnerated by orthographic

projection. The x and y components of thc velocity can be written,

U =wUZ - zkY

V =WoZx - UoxZ.

ConscquCntly,V2 V = - wtV'z.

This illustrates that the smoothness of the optical flow is related directly to the smoothness of the rotating

body and that the Laplacian of the flow velocity will becomc infinite on the occluding hound, since the

partial derivatives ofz with respect to x and y become infinite there.

H1orn , Schunck A. I. Memo 572

.4

p--- --- _ - .--------------- - -

4

Page 26 i)etermining Optical low

References

Ames, William F. (1977) Numerical Methods for Partial Differential Equations, Academic Press, NewYork.

Batali, J. & Ullman. S. (1979) "Motion )etection and Analysis," ARPA Image Understanding Workshop,7-8 November 1979, pp. 69-75, Science Applications Inc., Arlington, Virginia.

Clocksin, W. (1978) "l)ctermining the Orientation or SurfacCs from Optical Flow," i'rc'eedings ThirdAISIB Conference, I lamburg, pp. 93-102.

Contc, S. 1). & de floor, C. (1%5, 1972) Elenentai, Numerical Analwysis McGraw-I fill, New York.

:ennerna, C. I. & Thompson, W. B. (1979) "Velocity Deteimination in Scenes Containing Several

Moving Objects," ('omputer Graphics and Image irocessing 9,4 (April 1979). pp. 301-315.

Gibson, J. J. (1950) The Perception of the Visual Word, Riverside Press, Cambridge, England.

Gibson, J. 1. (1966) The Senses Considered as Perceptual Systems, IHoughton-Mifflin, Boston.

Gibson, J. J. (1977) "On the Analysis of Change in the Optic Array," Scandinavian Journal of Ps),cholog,18, pp. 161-163.

Gray, S. 13. (1971) "L.ocal Properties of Binary Images in Two Dimensions," IEEE' Transactions onCoinputersC-20,5 (May 1971), pp. 551-561.

Hladani, ., Ishai, G. & Gur, M. (1980) "Visual Stability and Space Perception in Monocular Vision:Mathematical Model," Journal of the Optical Society of America 70.1 (January 1980), pp. 60-65.

Hamming, Richard W. (1%2) Numerical Methods for Scientists and Engineer.% McGraw- fill, New York.

I lildebrand, Francis B1. (1952, 1965) Methods of Applied Mathematics, Prentice-I fall, 'nglewood Cliffs,New Jersey.

lildcbrand, Francis B. (1956, 1974) Introduction to Numerical Analysis. McGraw-Ilill, New York.

Jain. R., Martin. W. N., & Aggarwal, J. K. (1979) "Segmentation Through the )ctcciion of Changes )ueto Motion." ('onuer Graphics and Image Processing 11, 1 (September 1979), pp. 13-34.

Jain. It., Militzer, I). & Nagel. 1.-! I. (1977) "Sepcrating Non-stationary from Stationary Scene Componentsin a Sequence of Real World TV-images," Proceedings of'the 5 h International Joint Conference onArtificial Intelligence, 22-25 August 1977, pp. 612-618.

Jain. R. & Nagel. 11.-It. (1979) "On a Motion Analysis Process for Image Sequences from Real World

A. I. Afeino 572 lion & Schunck

QJ

I)terming Optical Flow hage 27

Scenes," I1.1.'11' Twsactions oi Patlern Analysis andMachine Inclligence 2, (April 1979), pp. .06--214.

Koenderink, J. J. & van I)oorn, A. J. (1975) "Invariant Properties of the Motion Parallax Field I)ue to the

\io\ vcnct of Rigid Bodies Relativc to an Observer," Optica Acta 22,9, pp. 773-791.

Koenderink. J. J. & van 1)oorn, A. J. (1976) "Visual Perception of Rigidity of Solid Shape," .Iouu,,l ofalthemtical Biology- 3,79, pp. 79-85.

limb, J. 0. & Murphy, J. A. (1975) " stiilming the Velocity of' hm ing Images in 'ele\ision Sigllls,"

(",mpllter Graphics ,n1d Image Prot-essing 4,4 (I)ecember 1975), pp. 311-327.

I o. guct-I I iggins, II. (C. & Pra.zdny, K. (1980) "The Interpretation of INo\ ing Retinal Image."' Pro(C'cdng.vofthc Royal.So'ic' B, (in press).

Mnrr. I). & tlillman, S. (1979) ")irectional Selectivity and Its lse in Farly Visual Processing." Artificial

Inlc'lilclce I.ahorory Memo No. 524. Massachusetts Institute of I'echnology, June 1979.

Nt, isciimui Russcl M. (1979) "Ilie Processing of I lexagonlall, Sampled "lko-l)incmiolal Sigul.ds,"

Iho, ,,,mIO Ojt/io' //-'-T h7,0 (June 1979). pp. 930-949.

ilhi. , i linl Ia . (195.1. 979) Dou,,'ri<'d.ScI/uion ofDi]'renua/I:'quaziwnx I)over, New York.

Nicl. II.-H. (1977) "Analyzing Sequences of 'V-frames," Procectlings of h'e 5th lh mai'rional Joinl

('rn/li- -om ,n Ahilicial hbtelligene, 22-25 August 1977, pg. 626.

Nikxtmaua. K. & I oomis. J. M. (1974) "Optical Velocity Patterns, Velocity-Sensitivc Neurons and Space

I'crccption." I'r('cepion, 3, pp. 63-80.

I'ra/dny. K. (197)) 'lgonotion and Relative I)epth Map fi'oi Optical Flow," (Computer Scicrice

I )cp;tiII(ent, Ulini~rsity of E'ssex, Colchester, England, (unpublished).

I'.i,dnv, K. (1)9S) "The Ilnlfrmation iii Optical Flows." Con puter Science I)cparlmctit. I tni\ersity of

I Cv ("olchestc. INigIiild (unl1publisled).

H iclitin cr. Robct I). & Mortin, K. W. (1957. 1967) )if/'rence MeIcthods fil l, i- I'uh" I'1col

In1t'iSL IcuC. .lohn Wiley & Sons. New York.

I -','l. I . 1( I .(cd.) (1A76) (','ulus of'l aritions iol( otroli'e'. ,cadcmic Pre,, New York.

N oi i.l1 . W. & Mlmdktldalil. S. (1968, 1979) I'aialional Principles in IhnwMics ,ald Quniam/un Ihcor"

l)Do 'r. Nem, York.

Ifor, (X .'hunck A. I. femo 572