. AD-A093 925 MASSACHUSETTS INST OF TECH CAMBRIDGE ARTIFICIAL INTE-ETC F/6 14/5DETERMINING OPTICAL FLOW.(U)APR 80 B K HORN, B 6 SCHUNCK N0OOI-75-CB043
UNCLASSIFIED A-M-52 Mt.
I muuuuuuuuuuuummum
9 T-
UNCLASS IFIEDSECURI Ty CLASSIFICATION OF THIS PAGE (Won Date Enfoted)
REPOR DOCMENTTIONPAGEREAD INSTRUC11"ISP RMI ORITIOCMENAN N AES BEFORE COMPLE T F0TM
REPORT iMBER nteOligecELab yRECIPIENT S CATAOr NUMR
AIM 572 JA -4" TITLE (and Subtitle) i '' S. TYPE OF REPORT A PERIOD COVERED
SDetermining Optical Flow memorandum.-- " . PERFO'RMING ORO. REPORT NUMBER
7.' AUTHOR(*) 8..I. CONTRACT OR GRANT NUMBER(s)
{ ! Berthold K.P iHorn & Brian G,/ chunck' I"- // 64
9. PERFORMING ORGANIZATION NAME AND ADDRESS 10. PROGRAM ELEMENT. PROJEC'T, TASK<
( Artificial IntellIigence Laboratory AREA & WORK UNIT INUMBERS
545 Technology Square , ...... // -- ;,J /
Cambridge, Massachusetts 02139 (l - -I I. CONTROLLING OFFICE NAME AND ADDRESS M -R--ORT DATE
Advanced Research Projects Agency// April 19801400 Wilson BlvdArlington, Virginia 22209 28
C4 MONITORING AGENCY NAME & ADDRESS(l! differeint from ControllingI Office) 15. SECURITY CLASS. (of this reporT,
Office of Naval Research UNCLASSIFIEDInformation SystemstArlington, Virginia 22217 jfu 15a. "'",EoLASIIAI, iONR
10. AISTRIBUTIONSTATEMENT , ..,, n I(of thisl i RouM
Distribution of this document is unlimited.
In DISTRIBUTION STATEMENT (on the obttrlo t atered in Block 20, I d hich as s rot Rhpttt) appa r
IS. SUPPLEMENTARY NOTES ",
None '
19 KEY WORDS (Continue r I necessapan i v s ttdenriy by block number)
Optical Flow Intrinsic ImagesMotion Perception
a.nCooperative ComputationQ-
20. ABSTRACT (Continue O reverse sIt nOcssary ad Identify by block number)
S>0ptical flow cannot be computed locally, since only one independentLImeasurement is available from the image sequence at a point, while the flow
__J velocity has two components. A second constraint is needed. A method for
LL. finding the optical flow pattern is presented which assumes that the apparent
velocity of the brightness pattern varies smoothly almost everywhere in the~image. An iterative implementation is shown which successfully computes
the optical flow for a number of synthetic image sequences. The algorithm..
DD IFAMPMR 1473 EDITION OF I NOV 65 1SOBSOLETE UNCLASSIFIED k'r /'Pe
SECURITY CLASSIFeICATION OF THIS PAG (to mf ntored)
- is robust in that it can handle image sequences that are quantized rather coarselyin space and time. It is also insensitive to quantization of brightness levelsand additive noise. Examples are included where the assumption of smoothness isviolated at singular points or along lines in the image. ..
0 r 4
T
NI ssachusct is In stiltte of, I ch nology
Arti ticiA Initelligeccnce I ah( uxtory
V. 1. NICInlo No. 572 Apli ii 980
IDtltcriing plical Flow
JL-i thold 1K. 1), I (1 jd Ihiaii G. Schiinck
.\fisi;nct 011ticall th ( 11(\ Ciiit tle coiiiptitcd I(4calti Siniceol &()Ile in dcpcndciit nciilliitIS iilai
~I .m i lilate SC1,g u11ciicc ;I( It point. Mille tilc 101 \'ciocity 11as t~kO coinon)McntIS. kA)I( Cond 1mntrint
IS ICCdcd. A HIciliod o6r tildi di tlw piLdI t10\N pIlMMcr k prcsciic1Cd M hiki &asilleNC 1ilu1 tilc .ippir-cli
\ Ctocii\ ili 111k, brlgilCSS p-ittcril slics siioodtii\ almoi~st cecr "s hcu ill [he iiiiac. An itciuii\k C unpienlienl-
tutioll IS ShiOV I \dlliCtl ,Icc('.jiIlj\ InlI)tIt0S tile opntijI tIowfo Ia iint1hcr ot s\ 1iutctic ini.gc Seqcetiics.
lic ag itcoil k1 5 obust ill (lit it canl liandic inliagc ScqnMIcncc 0th11 arcQ (UII~InIid rithctl ia'L~ inl spaCe
;ulk id 1111. It is 1lso iliscnSsli\ c to tianti/itionl of' brigtness Ickcis And aiddihi~c nloic. Fvimpics arc
indlidcd \de tile thcISSuliliptioll ot snuloothic~sS is \inlted at Singlilr poinlts ol along lilcs ill thle Image.
~ c hoi~kdginn I Th is icpoui dcscl i cs 'csc. i ch dolnc at tile Alii i tl I n(cii gccc I atIbolIlr of thle
Nliss,"Idiilts 1ISlitlt of, I cciinlouit Suipport 1,01 this r-CSCarch is piomidcd inl pairt bh [Ile \dunced
Pccn loje~ qcts A gcnc\ ol, ilc Decpartmnent of' Defenise uinder Mtice oit Ni\i Re'Ii(search couldiact
M00ttt1 !5-C-0(43.
81 19x1
I)(I'lnninfin ()plivaI low Padge /
1. Introdutctioni
Optical flowv is the dist ribuit ion of'appaircn t veloc ities of itimfcnieilt of' hrightness pan em s in ml imoage.
Optical flow can orisC Ijoin relative motion of objects and the jewer (Gibson 1950, 19(60). on~ctitently,
optical flow can give important information about the Spatial arrangement Or the objects %icy. ed and the
rate of change of IthIis arranigement I(Gibson 1977). 1 )iscontintities inI thle op~ticafl f1mk .1 I I,: Ip i II ,cgl I],I It-
ing imoages into regions that correspond to different objects (Naik.tyanti & I oonik 1 974?). .\ttefopts hive
been nIle topo suich segntcntation using difl'(rneflS hetAeeit S~lCCeCi%0 ntdgk-fi~ (1,61 0i a!
1977, J~tin ei a4. 1979, bill & Nagel 1979. hinhm & Ni iirphy 1975, Nagel 1977). Sonic cci pIm~em ' have
considei ed tile p rf ibi eml ofrev e CO Ii ng thle Motionis of'objects Felaltiv e to thbe '. 10% Cf Iil on 01 icp c~l .floi w
(I Ladani c/' a. 1980. Kociaderink & van 1)oorn 1975 & 1976, Longff Aiet- I Iigpiifs & h/Imrfdf 19)71. 1 P,il yn
1979 & 1980). 1n ti sme cases inlFormation about tilie shap i Pct a ohjci ma y ak he re i -c r-[ca ( K (efi no k
& xan lDoorff 1975 & 1976, Clocksin 1.978).
SIeSe ~iMPCS us hgi ii by aISSo ii fi ig that the flitical flow has already beetl dcteili filetI AlIthoftuighi Somle
reference hlfs beeff ma~de toi scheiles 1 or conPipiig tle 11 AV fr1ofi suIccessiveC viCIAS of fNric (I en ff~f tia
& Thomopson 1979. 1 Liaiii cA. 1980), the spcifics of a schenie fow deteIifInflig th10 flowk f10fi1 (Ile iflIfg
have not hcui dcecrihed. Related work has been doffe iii an atteniplt tot faammuuloa a Ilodel 1for the sifoit
frange Iliotiaff dctectiai pilfcesses ifi hiumuani isioff (Bmfo~di & I 1lu1i.iu 1981), Nhfif & I. lliwin 1979).
lTfe opticail fifiw Cannfot Ilt ctaiiiputed at 1 pofifft in theQ imffialle indepeuildenthy oh ffeighbofinge polt
withoot inftroducing additional coifstrailfus, becaueC thle vclocit) field ist each imaige point lifds wo coirn-
plleffts while thle Change ill imiage brightness ift .1 poift inl thie iige plan1fe dffe to fmotionf yields oni) (fli
constraint. Cfonsider for example a patch of a pattern Where hiuightoiess v;Irfes ifs af hiioii of, one image
cooirdiniate buft not thle otlher. Movi'emlent of' thle pattern iii fffe directioff AtN the brC iitufICY, Jt d pI)Irtii fflr
poitl. bilt moftionill inte otherl (lirectitfif \ields no change. Ilifi1" compionenfit't of' fffuvelifnt ifl [lie hafter
directionl cannot he computed locally, Additionial confstraifnts mfffst be ifftirfduc' to ff'lll detelilme thie
flow.
2. Iheat ioii~lip to Objet Nlto ionl
I lie rclionvlhip between tlie opticaIl flill [f h I e lille m(ithhe \locitie\, of' obtecil thle three
H~ornf . Schf flck . I pn I
Page 2 Determining Optical flow
dimensional world is not necessarily obvious. We perceive motion when a changing picture is proiectced
onto a stationary screen, for example. Conversely, a moving object may give rise to a constant brightness
pattern. Consider for example, a uniform sphere which exhibits shading because its surface elements are
oriented in many different directions. Yet, when it is rotated, the optical flow is zero at all points in the
image, since the shading does not move with the surface. Also, specular reflections move with a velocity
characteristic of the virtual image, not the surfiace in which light is rcflectcd.
3. The Restricted Prolbem I)omain
For convenience, we tackle a particularly simple world where the apparent velocity of brightness
patterns can be directly identified with the movement of surfaces in the scene. To avoid variations in
brightness due to shading effects we initially assume that the surface is flat. We further assume that
the incident illumination is uniform across the surface. The brightness at a point in the image is (hen
proportional to the reflectance of the surface at the corresponding point oi the object. Also, we assume
initially that relleciance varies smoothly and has no spatial discontinuities. This latter condition assures us
that the image brightness is diflferentiable.
In this simple situation, the motion of the brightness patterns in the image is directly determined by
the motions of corresponding points on the surface of the object. Computing the velocities of points on
the object is a matter of simple geometry once the optical flow is known.
4. Constraints
We w ill derive an equation that relates the change in image brightness at a point to the mo tion ofthe
brightness pIttern. I et the inmage brightness at the point (X, y) in the image plane at time t be denoted by
I(x, y, 1). Now consider what happens when the pattern moves. The brightness of a particular point in
the pattern is constant, so that
dE 0.dt
A, I Memo 572 IHorn fSchunck
IPacriiing Optical H/ow Page 3
1\ifng file chdin I-il forib di llcrent jationl we see that,
OE d OE d V 9pOx d t Oy dt Ot
(See Appendix A flait a ore dailecd derivation.) Ih'%c let
dz _ dyU = anddit d I
di citl it is eaIsN it) see that we haII e a single linlear eq Limil atf)1 tile tw Liiiknow iis it anid 1).
Eb.1 -j Y +C, EI- I 0,
Mdiere %wha a1l\Isoi initroduIced thle additional ahhre~ ationls 1', F1'. an1d PI fim tile paula1 der,.aifves
(1t itli'age brightness wkith respect to X. y and t, rcspecti~ely. Ilie coIlhtraiil i (ite loca1 lm %c \locity
exprIessed I)' this; equatioli is iIIlStraited inl lignie' 1. Wr tite equion110 in still an10ther Wily,
(,ii,.(It, V) -E $I'l I Lis I hie Compn pitenlt oft thle opt ical flow inl thle di rct ion Li 1the h rilit nciss grade it (EK LV', equLa.ls
WVe Llllohoe%.er determinle tile coniponcit (if moveneit inl tile directionif li,i jihicss on1-
oi-us. Mt Irli agls lo tile brightness giadieilt. As a Loniit-tice. tile fl %clocit% 01i, v) LcIiilot Ile
5. 1hi CSillooh lC%ine ( Coiws taint
II elef. polint of' the brighit ilss pilllei If Cani ll1(iSc independeitllY. there I, little hlope oh' recmi elIng (lie
N clot itics. More coilllofihy wc iQWk opatque ohjts oli fiiiite Sim1 unidergoinig Irigid Illitill or. dlComiiiltoii.
II ch~ unc~ik s.I f',2 7
Page 4 /)ciemunin I 1 ij ,',')V
V
(EXVEY)
0- U
constraint line
Figure 1. The basic rate of change of image brightness equation constrains theoptical flow velocity. The velocity (u,v) has to lie along a certain line perpendicularto the brightness gradient vector ( 4 l)in velocity space.
4. /. Ateini, 572 lHorn t( Xc/uick
Ietermuilning Opt jeW HFow I'age 5
III this Ca~sc neighboring points onl the objects haw similar velocities and tile Nelocity field of (fhe bright-
ness patterns in thle imagc v'aries smoothly almost everywhere. D iscontinuities in Ilo%% call hie expected
Mwhre one object occludes another.
One way ito express the additional constraint is to) limit the difference between the flow 'elocity at
at point and thle average velocity omer a small neighborhood containing thle point. I Llquivlentily we Canl
mlii oe thle stum of (lie squares of ilie I aplac ins i the z- anrd y -compo nen ts of thie fh iw. TIh e Iap.cau
of u and v atre definled ats
611u (!,u VV ov C)V 2u .- 4 +-- - and V-v +? 2
Inl simnple situat ioins, both11 I ap fac iansare zero. Iftic dietw ertiia uslates parallel to a h1It object, rotates ;1bout
aI fine pe rpend icuLla r to thle sourface or travels on hlog n il ly to dhe so if 'Iice (,aS'; liil ug e 'is eel i\ C po 0JeCr n),
then the second partial derivatives or both uL and v vanish. Note that our approach is in contrast " ith
that of' Fei c n a & Iiioiiipson ( 1979), who priopose an algo rithmi that in di rectify i net prafOr tes additional
assumlptions such ats surface snmoothness or object rigidity.
6. Quatiz at ion and Noise
images may be samipled at intervals onl a fixed grid of points. While tesselations oilher than tile
(tln ious one hla\ e certain adhitages (Mersereau 1979), Gray 197 1), for conlvenlience %\ e % ill assumnle that
the in iage is samip1led on] a squia re grid at regulia r intervals. ILet thle imieasu ired briigh tniess be L' ,.a tl the
intersect ion if the i - tIi row anrd j - I columi n i tilie k -tii image franie. Ideally. each measo enie n shouId
be anl aw rage m'er. thle area of at picture cell aiid over thle length ofi thimre Internal. in thle experirnents
cited liere we Ii aw t. ke ii samples at discrete point illo space and littile. in stead.
Ill additionl to being (jiiaiiti/ed in space and tinie, thle measm-iiieiOiu\% ill Ill practL'ie:C he quanii/ed in
h ugh itessas A elf. I u ilhe r. noiuse will be appa reiinti i in 5L m e eits obt a i ed ill 1aN I ICal Systeiii
Hiot-it C .%hunck A. 1. Ateno .S72
Page 6 Delerini'nig Optical Now
7. Estimating 1he Partial IDerivatives
We must estimate tie derivatives of brightncss from the discrete set of image brightness measure-
ments available. It is important that the estimates of E, E, and C, be consistent. That is, they should
all refer to the same point in the image at the same time. While there are many formulas for approximate
differentiation (Conte & de Boor 1965, 1 lamming 1962) we will use a set which gives its an estimate of E,
E, Et at a point in the center of a cube formed by eight measurements. The relationship in space and
lime between these measurements is shown in Figure 2. Each of the estimates is the average of four first
differences taken o~er adjacent measurements in the cube.
E. r{E,..j ,.k - E ,j,k + Ei-Iq,j+,,k - Ei-t,1j,k
+ Eij + I,kA- - Ei,j,k-l + Ei--,j-l,k+I - Ei+I,j,k+i}
-- E,,+lj,k l - EL,j+l -E i ,j t,kl - i-I ,k4t}
4
b", R 1 I{ Ei,j,k± + - Fl~~ + Ei--Ij,k-p-I - E4 ~~4
_1 Li,j-+-,k+i - EJi -kk +E,tI,j+1,k.f I - Et ,
I lere the unit of length is the grid spacing interval in each image frame and the unit of time is the inlage
frame sampling period.
8. lstiniating the Laplacian of the FIo- Velocities
We also must approximate the Laplacians of u and v. One convenient approximation takes the
following form
V'u F K(ft.j.k -- u1 ,.k.) and V2 v - K(f,,j, - vJi. p
licrc the local averages ft and f, are defined as follows
ft, ,. --- 1{ , I.j.4- + u I i ., -I-- It .4 u .k + itij 1,k)
+ 112{tIi I,j--l,k + Ui -- ,j4 1,k + U, f-I. 4 I --+ lt i 1 ) -I,k}
V",J,' = --{_ I,.j A- v,,. -..- , + V, 4- ,j,k + V. k - , }6
1,k+ Vai-Ij 4-.ik 4 V, I + 1,k + ?
4. 1. fe~no 572 Ib)" it Schujick
DI~ccrfifitlg Optical low Pa:ge 7
Figure 2. The three partial derivatives of image brightness at the center of thle cubeare each estimated from the average of first differenccs along four parallel edgesof the cube. Here the column index j corresponds to tho .r direction in the image,the row index i to the y direction, while k lies in the time direction.
Iforn S 1 /At iu *S7k
Page 8 Ielermining Oplica! P on
11Te Proportionality lactor x equals 3 if tile average is Comnnpu ted as shown and asso uing aga ini III;)[ the
unit t f'Icngth equal iis the grid spacinig intierv al. I iguii 3 illuLst rates thle assignment olI weights to n e glih w -
iitg points.
9, Minimization
[he Pioblen ithen is to miiinize the suni of the errors in) thle equation for tie rate of change of image
brightness,
a1nd the estimiate of the departure from smoothnecss in thc velocity flow,
S2 =f,_U) 2 + (jD __ V)2.
What shouild he the relative weight of' these two 111ctors? In practice thle imlage brigtness tInt1eeins
tt ll he corrupted 1). quantiiation eiior and noise so that we cannot eXl)CCI 9b to heC identiiCalIly ein 'I his
quntt~~ill tend to have anl error magitude that is proportional to) the noise inl the mleastirelient. This
Lict guides us in choosing a suitable \%eighting factor, denioted lby a ais will be seen later.
cI e(hle total error to be mininiied be
g2_ a~g' + S2
I hie in ii it 1,a1iml is to he accot np1ishcd by Finding sulitaible Nallues for1 thle optical tI~ \ e bc it x ( i, u).- We
dlicrniate 9 to obtain
- -2a2(ft -- uL) 4-2(/,.',. -f 1 v 4- ~)
c9u
- -- 2a2(ip - v) 4-2(Ik7,s +- bjv + E,)b7.Ov
Settinz these two derivatikcs equial ito /ero leads ito two equations in it and V,
( 2 L I2)U + p, ,v (ats- j)
I 'b,,s -- ( 2 + 1.,l) V =(aF I
A1. 1. .iImo.572 Ivn& tiw
1/12 1 /6 1 /12
16 -1 16
/12 6 12
Figjure 3. The Laplacian is estimated by subtracting the value at a point from aweigjhted average of the values at necighboring points. Shown here are suitableweights by which values can be multiplied.
Page /0 Detennining Oplical IHfow
The determinant of the coefficient matrix equals &(a - E' + E"). Solving for u and v we find that
(a 2 ± E)u = + (a'2 i f _ 1,,,p E ,
x + E Y+ u -- --/f)
(a2 + 2 + E2)V = -- :;fl + (a2 + I.,'I)b ;Et, .
10. Difference of Flow at a Point from local Average
These equations can be written in the alternate form
(a2 + E2 + E2)(U - - ',. f + EVf) + E,]
(a2 + E2 + E2)(v - 0) = - E[EfIt + E,, + Edj.
This shows that the value of dhe flow velocity (u, v) which minimizes the error g2 lies in the direction
towards de constraint line along a line that intersects the constraint line at right angles. This relationship
is illustrated geometrically in i'igurc 4. The distance from the local average is proportional to the error
in the basic formula for rate of change of brightness when fl, V arc substituted for u and v. IFinally we
can see that a2 plays a significant role only for areas where the brightness gradient is small, preventing $hap h,/ard adjustments to the estimated flow velocity occasioned by noise in the estimated derivatives.
This parameter should be roughly cqual to the expected noise in the estimate of E - E2.
II. Constrained Minimization
When we allow a2 to ted to zero we obtain the solution to a constrained linimi/iation problem.
Applying the method of Lagrange multipliers (Russell 1976, Yourgau & Mandelstani 1968) to the
problcni of minimiting . while maintaining 9b 0 leads to
_1 _ ,j_ )(E' + E",)(, - b)= -,,'Eft + L,,,, + E,.
We A ill not use these equations since we do expect errors in the estimation of the gradient (EI, Ey).
A. . Aleto 572 Hlorn d .chunck
'NNW
Ih't pnuinhl' Optiall IEh Ial 1
V
(U'V)
constraint line
Figure 4. The value of the flow velocity which minimizes the error lies of) a linedrawn from the local average of the flow velocity perpendicular to the constraintline.
Page/12 iemnli pcaHot
12. Iterative Solat ion
We now have at pail.r of equat11ions 6wo C.ach poin t ill tI e uii age. It %% ud ie v'ery co stlIy to st Iv OeLIc
equaLtions simlowtaneously by oine of the standard mi etIho ds. such ts ( lIss-JOrdaii eliiatiaton (II ii ii 1ig
1962, 1I ildehrand 1952). The corresponding mnatrix is sparse and very large since the number of tows anid
columins equals twice the numbller of'picturc cells in the image. Iteraitive met-hods, such ais the (auss-Seidel
method (I lamming 1962, 1I ildebrand 1956), suggest themselves. W,- canl compute at new set of'% clocity
estimates ( up V11 v' )From thie estimated derivatives and thc average of tic previous velocity estimtates
(it", v0) byUri1 =I'- EX!E.JSn + Efi" + h'j1/(a' - F' + E')
-Eu,(E.is + Ejjf"iP -F/(& + Ex + Ea)Y
It is in~terestinig thalt the neCw estimrates at a particuilar point do not depend directly on the previoos
estimates at the same point.
13. illing in Uniform Reg.ions
Inl parts of tie image where 1he brightness gradient is zero. the velocity estiniace m ll simply he
ave rag~es Of. the neighbhoring velocity esti mates. TIhere is mio local in Ii oat ion it) oiiONi:-.in tie ippmt
velocity ofimotion of the brightness pattern in these areas. Eventually tie valuies aron d suhl a regionl vmll
propagate inwards. If tile velocities on tile border of the region arc ill] tihe samne, then pn ts inl the region
%ill be assigned that value too after at stiflicicnt number of iterations. Velocity iniformation is thul[, filled in
from the boundary of a region of constant brightness.
14. Number of Iterations
I f the %allies on tIle border are not all[ the sam~e, thie values filled in will correspond to the Solnltiomi
of lb e I i1place eq nation For thle given bionidarmy cond it ion (Ameis 1 977., MIil1ne 19)53, Rici1tileCr & Nit ilt) l
1957). I he pro g ress ofthiis filfling iml phenomena is siiliar to the p~ropaga tion) e ffects inl the solUtil moftile
hecat eq oation, whle re tlie tillme riitc ol chanige Of tempe mlat i C is P roport io nalI to thle I ap lacia ii. ThIiis gives
US a1 mleans if' understanding tile iterative method iil phlysical teins and of estimlating tile numbher or steps
A L. Mtmo.572 Hornu 4 .'chunck
ItkimIcL!. I lie numinhc of' itchttioll h~toiIlk hcii ihaii1 (lie Crosy v-~oCCIInI' Ow 1 hiiw 1 ci '"
Ilk filled ll. It' tile -e of, such region~s is not know.n ill .Iucec oc iiu 111 ic WAlie c !h)!)c I, I,
I II , I \ Iuc I uve csti I I t C.
15. Uighitiii~s of Colot raint
'AtWhl;Iiun~ in ~i ieginis a' . liicarI tuctiol of' Iic igc~k c:00IdiI1A(C1 \\, L,111 OuiIk 01biil !h"
1.1 f opm .il !I('%% ill (Ile dirct ionl of' [tc gradient. HIC I colioPOncnt M I i~l it if nlI i n ",1 il,- iI I I
thc hooiiii!Ii I, 01c 1i~jou -is dibeC~hd lictoic. Ill gCI1mil (Ile SAU11iid 11(V nnd kIll if\ k (C ii[111l'
iii ic.'iuiii ulit'ic Ow~ Illii~'Ihi'ss gadiculit I; not too) suinall ail \,IIrics l 1iccluuuil Iliiil~ii P1!(fi l1W
~iIt i) 111iluoiiu of c"icl pint. Iuio x ioikuut fluictulutionus Ill bi'iics oll (ho uiii, il 'lV fit
16. Chijce of Iteratke Scem
AS .u pudI Li flfuCI- 011C IMs a Chioice of' how thec Iilns ire~ tIle ilcilucd '\ flh thc inch "t,
Oi fit-, 'inc limiud. onei~ Could i(tIc lIIUI ilti h Sliji Iis "IIhili/c(I hCkI0I- t\ I11iciii 10 Ill', Ihil11 li'C
Irliniu. Oil Ole utiu'i 11.11ud, tc .1 goodi inlitial1 fluC\\ iini' u\ lck oiii utic i(ciitiuuii SIY 1tiiiiC'iV.
goodi id nin'li :muui for (It t uld)ti 1301l vCOL]iC; is 1tlIuII\l u i.il.1hik: tiuii (tic ibS lll i1c WI.
I tic oft ii~~,u (ie tIttcr ;IIa)IOd)10 C Alld I Iitii It) d.it %kt h iiWCc? W iiiu 'Cl oS i ll lillic: 111d
thu1tubA (I' iui~I li i.J iliwAi %luoctc illculiul'u. lcu lt'i'i Atlcx u o ut, i tll uhf uluic I '-I.I lt Islui
*\I pII ii- (.ilf u iii kicui(.i No btauu ied huos 1kl 'Idulflo li o m O ,cu~'il pci li t'y I~ 11""k lcs',1111.1 0.l
\icmiiscd hunIdcP m upiuuith 111.1i s Iu )W )OP'ull wuucus.,111
'11)(I Wid td o..Nu/iiooeippintdnAotpx' 111cpl~k11wl 101h j. I ',nCl P~ /2
-i i*. iag
Page 14 Ietriining O)ptical / olt
17. Experiments
The iterative Scheme has been implemented and applied to image sequences correspondinig to at
number of' simple flow patternls. T[he resuilts shown here are on it relati~ ely smnall image of' 32 b) 32
p ictur e cellIs. T he hirigl tn Il east irelii lns Were intlention11ally c irri ied by app ro x iiiatel) V1% niiose amid
then (iianti/ed in to 2 50 levels to simul Ilate at real inmaginI g situiat ion. The unude rly inig sutrface ic hlcct.m ce
I I it tem In w as a h il Ctar con I b il n(I on It o spatiall y 01rth0oi i I Sinll s I ISO iT Ci I a r k AjICIgth Itwds ChOW1e to I 1%k
reaso nabl y St long brightnless gradient ISwit hout leading to tmnde rsanit p I Ile prItblemi s. I )i-sc i ntillinLiei were
J% ied to ernure that thle required derivatives exist everywhere.
S ho wn it i Figure 5 hfa r example are foumr friames of' ia seq tence of' images depicting a sphere II ital in
about anl axis inclined towards thie viewer. A stitoothlly %arying reflectance pattern is paiiited onl the surf'ice
tif the sphere. ['he spherie is illuminated unillbrmly from all directions so that there is no0 shading.
18. Resuilts $
[lie first flotm to he investigated was I Simple linear tritslatiiin o* thle entire brightness pattern. 11wl
,es ill 1)LM in110%ititd o is Shown as at needle diagram ill I igt are 6 hiir 1. 4. 10 i. d 0.1 ite it itils. I hie
ceftil lm a e %it l c it ics are dep ic ted as Sholrt lililes, show in g the apparlent displace nell t duin g 0 lle filiIC
step. Ill tis cumiipile it sinle (n me step wats ta ken so) tha~t tile coimptutat ion ill ie b asedl oil julst two ii i ,ml"es.
liiIanallk the cstlillitCls oh, floA velocity are zero. Consequently the first iterationl shows %ectors, in the
(Il actiti i oh' [lie bright nless glad il. a ter, thie est inae 1.1 Spp roach thle correct s'ah es inl .111 pa its of, tile
iIimi!,c. lew cha~nges occur ifter 32 itertitlns when (lte telticity %ectors liuxe error" of' abioutt 10%11. [hle
eStlili1i1Cs telld tol lie ttoo smiall. rather than too lirge, perhaps becautse of'. a tendency t ttinldelestilllLe thle
de ri at it s. I hle ttU Clri c it u ~tr. as (i ie inigh t ex pct, wI eio tIie h rigi liess g r.id icllt is silall.
Ill thle sectlid eXpelllleut tine iterationl w.s used per tunei step oil thie Same iiie.ir. traiislattiiil image
seqtlleikc I hie resilting conilpuled Iltiw is Shiownl Il ligule 7 tor 1, 4, ti. anld 0.1 lttlle steps. [hle estimiates
pr .1110k s I tIlcorrect il tICs Imore rapidly anid do no t have at tendency to bie tio 10smll, as. IS ite prIe ii lls
CS pI~v illiilt. I'Cek Changes oXCcur after 16 iterations when the velocity vectors has C errors of abotlt 7%1. [lie
storM errors txciir. as (tie migla explca. where tile noise il recniIeSiirmnents of NlIrgh ies's V% is Ivillst.
A. L 1. lint 5/2 lt rm J .c/,nck
At i
hiqure 5i 1out fitrues mil of a sequeonce of imaqies of aI !phowre ot'Itim) i1hout ani
A. 1% HIiwloil to)wardis tlit viewer. The sphere is cove-red with i polemv whichvaies! .moothrly from piaco to place. The SPhere I-; pImtrayod .qiota t,'xut, lrqhtly4 t~extured hok.1rmi;dir. tIiiwim sequenlces like thesie w )roce(!t( hi' the tiltical1 flow11(jorillim.
Page 16 DleCicnining Oplical flow
S,.~~~~ ~~~ 0i dii I//'d/ IddJd
1 1 Al~l Z//////
...... ~~ /I//// V llll I II/
% ~~~ ~~ ... . ./lt/ . I ... .% .llli .l~
A B
I I & La £LL dddil-/71 i S IdIddddg gld dddild//~ldd1J.J J
illililZZZ IlAill /]l I1//Z/11 I//I/Iiiljl
'll 1ZiZZZZ6 I iiii Z/Z '1 H IZZ// I
111111111 1'l Zlllll 1/ /// I///1//ii //i/Ill
I//i//I/I / ,,,,]iJJ]i/ I/ii/// Z ///ZY////// TI/////1111
ZZZZZ//ZZZZZ/ Y1Z1111 / 1/11/ IiiZZZZZ
cells in the x direction and 1.0 picture cells in the Vj direction per time interval.Two images are used as input, depicting the situation at two times separated byone time interval.
A L. Motto 5.?2 IHorn A.cwc
/ p i'1 I'loft,1
Y. 2
WMIY
z j Y V/ 'it Y I
I~~ I Z
A B~ue7 o atr o ptdfrsm l tasaino rgte! alr.Il
II ~ ~ ~ ~ ~ ~ lp are/111i1 /lon Her one Ill//i/IL' is/1/1411 / /// e'
1111iile /fe /, /, 1ti an 1 14 lim /jt 11i o /m1t/1/1117/1* !//vrelc is mor 1*'~ i 111n 1/111 th '1 /11111141
accurately 111l1111 'i/. 4 H/111 1 /11
Page 18 l)ehr, millng 0p)/ ll I h'
While individual estimates of velocity may not be %cry accuiatc, the .aclagc mver the whole Image %as
within 1% of the correct value.
Next. the method was applied t) simple rotation ind simple contraction of the brightness pattern.
The results after 32 time steps are shown in Figure 8. Note that the magnitude of the %clocit, is piopor-
tional to the distance from the origin of the flow in holi these cases.
In all of the examples so Cair the I aplacian of both flo% %clocity components is /cro evci "here. We
also studied more dillicult cases where this was not the case. II partictlar, if we let the inigniltode of the
velocity vary as tic inverse of the distance fron the origin we generate flow around a line vortex and two
dilensional flow into ia sink. The computed flow patterns are shown in Figure 9. I lere te colllutatioll
involved many iterations based on a single time step. The worst errors occur near the singularity at the
origin of the flow pattern, where velocities are found which are much larger thian one picture ct. II per time
step.
Finally we considered rigid body motions. Shown in Figure 10 are die flows computed for , cylinder
rotating about its axis and for a rotating sphere. In both cases the ILaplacian of the flow is not zero and in
fact tile I *placianl of one of' the velocity components becomes infinite on the occluding bound. Since tihe
vec;ities themselves remain finite, reasonable solutions are still obtained. The correct flow patterns ue
shown in Figtre 11. Cinpariiig the computed and exact values shows that the worst errors occur oii the
occluding boundary. These boundaries constitute a one dimensional subset of the plane and so one cal
Cxp'ct that Mhc relative number of points at which the estimated flow is seriously in error will decrease as
the resolution of the image is imade finer.
In Appendix II it is shown that there is direct relationship between the Laplacian of the flow velocity
colmponent, anld the ILaplacian of the surface height. This cal be used to see how our smoothness con-
stiant % ill tiii for different objects. For example, a rotating polyhedron will give rise to a flow which has
iCio I .l1,1kial exceept on tie iinmge lies which are die projections of die edges ofthle body.
19. Sutnu:iry
A ethod was developed for computing optical flow from a sequence of images. It is hased on the
observation that the flow velocity has two components ,ind that the basic equation for the r.te ofchange
A. . Afemo 572 lIh,, u .S'cltmck
- -- - ---
I', T I r III
. . . .. . . .
Figutire 8. Flow patterns computed for simple rotaition and simple contraction of abfiglitncss patterin. In tile first case, the pattern is rotaited about 2.18 deqines pertime step, while it is contracted abouit 5". per time step in (fie Second case. 1Meestimates aftei 32 time step% are shown.
kkk;UCAI I-s,.
Page 20 Delerminini Op/ical Flow
L QL i t e f ~ pt P P P P P PP P P P P P P
k~ II I m ~ r l L l mq t i II +P + P P P P P P P
P P P P
V
- - - - - - -
P P P P P
i . qqI I t r r t I / P , P
' A - -
I
A -B
e+++ ] ...........
Figure 9. Flow patterns computed for flow around, a line vortex and two dimensionalflow into a sink. In each case the estimates after 32 iterations are shown.
,f. 1. ,tleino .572 hIur if Sc'hutucck
- - - - - - - -
* P ~.o . . . . . . \ . .
• • t % " ..I I. . .. ..-.-... '... .... .. ..... .. -" "
..... j
. . o.
A B
Figure 10. Flow patterns computed for a cylinder rotating about its axis and for atotaling sphere. The axis of the cylinder is inclined 30 degrees towards the viewerand that of the sphere 45 legrees. Both are rotating at about 5 degrees per timestep. The estimates shown are obtained after 32 lime steps.
I,,,l ,' Xcliu & ,. /. .ihCmP, .c/2
. .. . . . - - --- -- ---.. . " . . . . .
Page 22 /)elonmillg )pila Hlo'
* * ~~ ~V- .-.-. . .
S...-.- . . • . . .. .
~do .. ...- . . . ...................................•"•............................... '' \.....
I- .... . . . .... . .
. .. .. . . . . .
Figure 11. Exact flow patterns for the cylinder adthe sphere.
.1 . W1ono.572 or CSiuc
d
Itcmuno,i O)ptical Flow PA ge 2
ot, iiiiai~c h[nghti@;s prom ides 0on1 0onC COnIStrinit. Sirio01lucss o( thie lo% %%,s Inn odueed is5 a scond
colistraiiit. Alil itcrilti~v iiiedlOd toir Soh ing (Ie I-S1iillo. CLMqit~n'l) IS [)hen de'C]Iopcd. A limlple i01-
pliinillntioii prlo. ied \ isii.ii Collfirlwlit ioli of, conlvel gk:iicC OF C th0e 101 saitoil in th M Irii it icedcl diaei Alis.
Illaiplc1) *rse eid diffrent types of opticail flow pAtterns e rc studied. 'I ie~e include:d e xse \\ hei ': the
I 1 ilaicianl or (thc flow wils z'ero 31nd eaIses where it beCamel"I infinlite at singular poinits ()I Ilml! buniding
0,We O* Lit:e Ikltlloi, (I barn) Nouhd like to thank Prol'essor lI A . Nage!c I'M his hlo-yitllit) .I h timsic
LjiI~t(IFS Akre coiceiked dii a visit to the hjoheisity of I Janbi, stimiulated b P'i or N~ieel\s
]l 'u-stI.Idiing initerest Inl notion vision. 'Ilhe other ziuihoi. (Schlick) would like to thank XV. F-. I ( l illiso
Jild V. I Iildreth tbr uiiu inter-esting discussions and much knowliedgahle criticism.
Hl pid chu mcA A1 1. lfco' '2
Iuge 24 Determiiing Optical/ low
Appendfix A. Rate ofr aage of Image Brightness
Consider a patch of the brightness pattern that is dispiwced a distance bx in the x-dircction and by in
the y-direction i tinie 6. The brightness of the patch is assumed to remain constant so that
E(z, V, 1) = E(z + bz, u + by, i -- 6).
Expanding the right hand side about the point (x, Y, 1) we get,
E(, y, t) + 6z +6 + 61 - + E.
Where c contains second and higher order terms in 6z, 6y, and 6t. After subtracting E(X, y, t) from both
sides and dividing through by 6t we have
b x (9, y ,9 + -+ 40 E + ()C 6t) 0 ,
where 0(6t) is a term of order 61, and we assumc that 6x and by vary as 6t. In the limit as 6t -, 0 this
becomesc7Edz + + L = 0.
S oy dt 0-a O
A 1. A feno 572 Horn d Schunck
Deternining Optical Flow Page 25
Appendix B. Sinootliness of Flow for Rigid Body Motions
Let a rigid body rotate about an axis (Lo, w,, w.), where the magnitude of the %cctor cqtla;, the
angular velocity of the motion. If this axis passes through the origin, then the velocity ot a poult (X, y, z)
equals the cross product of (w1, wu, wL), and (z, y, z). There is a direct relationship between the image
coordinatcs and the x and y coordinates here it' we assume that thc imagc is gcnerated by orthographic
projection. The x and y components of thc velocity can be written,
U =wUZ - zkY
V =WoZx - UoxZ.
ConscquCntly,V2 V = - wtV'z.
This illustrates that the smoothness of the optical flow is related directly to the smoothness of the rotating
body and that the Laplacian of the flow velocity will becomc infinite on the occluding hound, since the
partial derivatives ofz with respect to x and y become infinite there.
H1orn , Schunck A. I. Memo 572
.4
p--- --- _ - .--------------- - -
4
Page 26 i)etermining Optical low
References
Ames, William F. (1977) Numerical Methods for Partial Differential Equations, Academic Press, NewYork.
Batali, J. & Ullman. S. (1979) "Motion )etection and Analysis," ARPA Image Understanding Workshop,7-8 November 1979, pp. 69-75, Science Applications Inc., Arlington, Virginia.
Clocksin, W. (1978) "l)ctermining the Orientation or SurfacCs from Optical Flow," i'rc'eedings ThirdAISIB Conference, I lamburg, pp. 93-102.
Contc, S. 1). & de floor, C. (1%5, 1972) Elenentai, Numerical Analwysis McGraw-I fill, New York.
:ennerna, C. I. & Thompson, W. B. (1979) "Velocity Deteimination in Scenes Containing Several
Moving Objects," ('omputer Graphics and Image irocessing 9,4 (April 1979). pp. 301-315.
Gibson, J. J. (1950) The Perception of the Visual Word, Riverside Press, Cambridge, England.
Gibson, J. 1. (1966) The Senses Considered as Perceptual Systems, IHoughton-Mifflin, Boston.
Gibson, J. J. (1977) "On the Analysis of Change in the Optic Array," Scandinavian Journal of Ps),cholog,18, pp. 161-163.
Gray, S. 13. (1971) "L.ocal Properties of Binary Images in Two Dimensions," IEEE' Transactions onCoinputersC-20,5 (May 1971), pp. 551-561.
Hladani, ., Ishai, G. & Gur, M. (1980) "Visual Stability and Space Perception in Monocular Vision:Mathematical Model," Journal of the Optical Society of America 70.1 (January 1980), pp. 60-65.
Hamming, Richard W. (1%2) Numerical Methods for Scientists and Engineer.% McGraw- fill, New York.
I lildebrand, Francis B1. (1952, 1965) Methods of Applied Mathematics, Prentice-I fall, 'nglewood Cliffs,New Jersey.
lildcbrand, Francis B. (1956, 1974) Introduction to Numerical Analysis. McGraw-Ilill, New York.
Jain. R., Martin. W. N., & Aggarwal, J. K. (1979) "Segmentation Through the )ctcciion of Changes )ueto Motion." ('onuer Graphics and Image Processing 11, 1 (September 1979), pp. 13-34.
Jain. It., Militzer, I). & Nagel. 1.-! I. (1977) "Sepcrating Non-stationary from Stationary Scene Componentsin a Sequence of Real World TV-images," Proceedings of'the 5 h International Joint Conference onArtificial Intelligence, 22-25 August 1977, pp. 612-618.
Jain. R. & Nagel. 11.-It. (1979) "On a Motion Analysis Process for Image Sequences from Real World
A. I. Afeino 572 lion & Schunck
QJ
I)terming Optical Flow hage 27
Scenes," I1.1.'11' Twsactions oi Patlern Analysis andMachine Inclligence 2, (April 1979), pp. .06--214.
Koenderink, J. J. & van I)oorn, A. J. (1975) "Invariant Properties of the Motion Parallax Field I)ue to the
\io\ vcnct of Rigid Bodies Relativc to an Observer," Optica Acta 22,9, pp. 773-791.
Koenderink. J. J. & van 1)oorn, A. J. (1976) "Visual Perception of Rigidity of Solid Shape," .Iouu,,l ofalthemtical Biology- 3,79, pp. 79-85.
limb, J. 0. & Murphy, J. A. (1975) " stiilming the Velocity of' hm ing Images in 'ele\ision Sigllls,"
(",mpllter Graphics ,n1d Image Prot-essing 4,4 (I)ecember 1975), pp. 311-327.
I o. guct-I I iggins, II. (C. & Pra.zdny, K. (1980) "The Interpretation of INo\ ing Retinal Image."' Pro(C'cdng.vofthc Royal.So'ic' B, (in press).
Mnrr. I). & tlillman, S. (1979) ")irectional Selectivity and Its lse in Farly Visual Processing." Artificial
Inlc'lilclce I.ahorory Memo No. 524. Massachusetts Institute of I'echnology, June 1979.
Nt, isciimui Russcl M. (1979) "Ilie Processing of I lexagonlall, Sampled "lko-l)incmiolal Sigul.ds,"
Iho, ,,,mIO Ojt/io' //-'-T h7,0 (June 1979). pp. 930-949.
ilhi. , i linl Ia . (195.1. 979) Dou,,'ri<'d.ScI/uion ofDi]'renua/I:'quaziwnx I)over, New York.
Nicl. II.-H. (1977) "Analyzing Sequences of 'V-frames," Procectlings of h'e 5th lh mai'rional Joinl
('rn/li- -om ,n Ahilicial hbtelligene, 22-25 August 1977, pg. 626.
Nikxtmaua. K. & I oomis. J. M. (1974) "Optical Velocity Patterns, Velocity-Sensitivc Neurons and Space
I'crccption." I'r('cepion, 3, pp. 63-80.
I'ra/dny. K. (197)) 'lgonotion and Relative I)epth Map fi'oi Optical Flow," (Computer Scicrice
I )cp;tiII(ent, Ulini~rsity of E'ssex, Colchester, England, (unpublished).
I'.i,dnv, K. (1)9S) "The Ilnlfrmation iii Optical Flows." Con puter Science I)cparlmctit. I tni\ersity of
I Cv ("olchestc. INigIiild (unl1publisled).
H iclitin cr. Robct I). & Mortin, K. W. (1957. 1967) )if/'rence MeIcthods fil l, i- I'uh" I'1col
In1t'iSL IcuC. .lohn Wiley & Sons. New York.
I -','l. I . 1( I .(cd.) (1A76) (','ulus of'l aritions iol( otroli'e'. ,cadcmic Pre,, New York.
N oi i.l1 . W. & Mlmdktldalil. S. (1968, 1979) I'aialional Principles in IhnwMics ,ald Quniam/un Ihcor"
l)Do 'r. Nem, York.
Ifor, (X .'hunck A. I. femo 572