ASS' :,S 00 E

REPORT DOCUMENTATION PAGE .READ INSTRUCTIONSo~~o ~.~e~ Z GVT ACESIONBEFORE COMPLETING FORM

12 OVTACESSONNO I. RECIPIENT'S CAT ALOG NUMBERAIM 803 K'i~

4 ~TILE 1-d S~b?1"e) S TYPE Olt REPORT A PERIOD COVERED

Multigrid Relaxation Methods and the Analysis of AI-MemoLightness, Shading, and Flow. G EFRIGOG EOTNME

AU ThOR(#) 5 CONTRACT OR GRANT NUMUER(e)

Demetri Terzopoulos iNOOO 14-80-C-0505

9 ~VQtGOGNZTO NAME ANO AODRESS 10. PROGRAM ELEMENT. PRO.JECT, TASKArtificial Intelligence Lab. AREA A WORK UNIT NUMBERS

* 545 Technology Sq.* 1% Cambridge, MA 02139

I CCNTROLLING Or"ICE ýAME ýNO ADDRESS12REOTDE

Advanced Research Projects Agency October,1984___001400 Wilson Blvd. !3. NUMBER OF PAGES

LnArlington, VA 22209 22______________14MONITORING AGENCY NAME A AOORIESS(,I differcmil 0001 CoAIOIntf OllinfOfic) W3 SIECURITY CLASS fol (hit Popefl)

Office of Naval Research UmnclassifiedInformation SystemsArlington, VA22217 ISO. OCCLASSIF ICATION/ODOWN GRADING

SCHEOULL

It DISTRIBUTION STATEMENT (of thi. ftepoft) ST.BF1NSATMN

Distribution is Unlimited AppwvedTIO 57ATlc E1lEN-r A

DistrffutmaB Unlimited

17, DISTRIBUTION STATEMENT (of We 8681rdte .eneopod In 8199A 2@. It diflotent fts Reps")

15. SUPPLEMENTARY NOTES 6 E

It KEY WORDS (Continue. on f@*@P.@ *id* It noese..' And Wen~tity by block OKWOF)

Computer Vision Multigrid RelaxationLightness Shape-From-ShadingýOptical Flow Variational Principles,Partial Differential Eqs. Parallel Algorithms.

*20 ABSTRACT (COflIID9 OR 101"00 Old* It MOCOeOWT GR4 00"100I 67 block Ofbtao)

Image Analysis problems, Posed mathematically as variational principlesor as partial differential equations, are amendable to numerical solution byrelaxation algorithms that are local, iterative, and often parallel. Althoughthey are well suited structurally for implementation on massively parallel,locally-interconnected computational architectures, such distributed algorithnsare seriously handicapped by an inherent inefficiency at propagating constrairtsbetween widely seperated processing elements. Hence, they converge extremely

D D 1473 E01TI0N OP 1NO011 t1 OBSOLETE UNCLASS IF IEDSECURITY ý.LASZIFICATION Of THIS PAGE OM"w. Del.Inlt

BestAvailable

Copy

20)slowly when confronted by the large representations necessary for low-levelvision. Application of muligrid methods can overcome this drawback, as we establishin previous work on 3-D surface reconstruction. In this paper, we developeefficient multiresolution iterative algoithms for computing lightness, shape-from-shading, and optical flow, and we evaluate the performance of these algorithmson synthetic images. The multigrid methodology that we describe is broadlyapplicable in low-level vision. Notably, it is an appealing stategy to use inconjuntion with regularization analysis for the efficient solution of awide range of ill-posed visual reconstruction problems.

I.I

0 - . ..... . .

*A V

Dist\UA",'t. .•.-,• ']

0D • .po:' ,

M..\SS.\CI ISt-'! S INSI IFII I: OF 1 FCIINOI.OGYp \TI(I -CI.AI I NI I-.,.1.I(1G ECE LAIIORA'IORY

A.]. %temo No. 803 OctobCI, 1984

MULTIGRID RELAXATION METHODS AND

THE ANALYSIS OF LIGHTNESS, SHADING, AND FLOW

Demetri Terzopoulos

Abstractlinrage n1;i!ysis prol'ieml, posed mathie ;,,tic;aly as variatic tal principles or as partial differentialCq0 ;hit Os, a;mIule, to nto 11¢1_,ii Solltion by relaxation algorithms that are local, mtciative,ar.'dt (etn parillel. ..,lhu,!ih they aic well suited structurally for implementation on massivelyprtdlkl, ii dally- l:iticui;,c.tcd coiipi~t;•tiiul archlitcc ies, such distr~~itcd alg,)ritlhns ,re :,criouldyh:Indipredb ar in Krcnt incfli.Jcncy -it pr(paating constraints between ,4idc!y s.paratcdpjocc,,ý;ing celmciv s I lcnICe, 01C., coin ic b cextrenie1> slomly' ýAhcn contftonL'• d by die largeSrvcpýcmV-tionr n,; ,e.,i'; For luv-,IcIel ,Isiunl. Application of multigrid nmchods can overcomet ', dita ' l kav. Ick, ; ',1 .ý %ý iih! slChh in prc ýii i us ok on 3-1) surface recLlonsti ction. In Ihis p.lpcr, wedLaehlop CfliCieILt 1iif '' esohutýo ti'ti \c algorithms for computing lightncss, shape-frorn-shading,,•nc oticnil lhv.. Mn v,.c cva'. (¢c he perfoniance of dicsc algorithms on synthetic images. Tfhe* ;;,iitwi ii c dcth .. I ', .1y that vý,C describe is broadly applicable in lok-levcl vision. Notuably. it is in,Tppc;,li ;ig stm i CC i (o ,, M C liW'..Llio wjth rcgulari.ation anal sis for tie eflicicrnt solution of a• In up.:" o! ill-i'i d ,.p,,. 1- 1 cc,, ,ction problerns. , .

,ic- i?,~ l s,•:hctts lnstimute of'cchnology 1984

1 -'.' .... , , . 't'!• ) '• , t the A tificial Intelligence L..aborat.ory of the i ss uhuLetts-I~aiit• " - i _l . t:pipoirt !', the I.ibirater,"'s Artificial Inteliinciic research iS Providedli, plot h% the Ch R.l c, Ks. ii •-hI.jcts Agency o1 die I)ecpartmcnt of l)et'enc under OfMice

0ul" •\;1\,,1 •,,.>,•',|1i. U,•t!. Y. ;i0 C-0505 and the System I)C'cli ilmnc t Iokindatmoi . "[he"* - .* aui~i',,, '.;.i,• ,il'li~ h , 1'ý N:, ,il S, ,1 5 c ' .and Iogin cering Rcsearch Council of C man da and

1he I on k,, I." \.. ( ) - ('imida

• . • * ' -- *- - - .-i - . ... - - .." . .'. .- - " ..i . / ' . . I - . ' ' ' ' ' . - . - -. "- .. .v-, - - -i - " -

4

1. Introduction

"";ari,ttiunal principIcs and p._tial difleCrCn tinll eqations halve played a si>gnificant role in

the irrathe mnca'ii,! oormf1 iUOtt ol I -le \cl 'i,,ul information proc.essing pl-ob1iCITs (represcntativeexamples include [thorn, 1974, 1)975; Ullman, 1979: tlon & Schunck, 1981: Ikeuchi & I oin:

19S 1; Nartpnai al an 1 19K), 82 13jcI &' & l1roit, 19S2; Itulinmel & /.icker, 1983; Crimson. 1983;

T Terzopoulos, 1982. 19S3: Nagel, 1983 1 Ilildrcth, 1984; Brady & Yuille, 19'.1-W. %n attracuive feat -,

ef variatin nal and differentia! fIrOrmulationIs (once dr, ccrCtiZcd) is the possibility of COniputing In

dcsir,'d ililiiions i a popular clar,; of nmierical relxation alpnorirhuis. "These iteratie alzeorithimc

require o iy local comput~itions vhich can usuall% be performed in parallel by many locally

Coilut1in.0 tine trncsors dist i ibtti in conp)utItional netw~orks or grids.

l poc , 1. parrle algi.rnI dr appealing in the cuotext of low-ijeel Nision [Rosenfeld ca

,., 19716,. LllunIT.. 1979; If;! lard c1 aX., 19831. At a ceitain level of absutraciion tiey do not

Ippcar in ninp<arihle v ith UiC ,plarc.; stucture , aidvaInced hh.,Iogical ,ision s sterns. loreover,

SrthcN are i c,icll d sitCd to IIUl)leilCiiIt.IU(',U on nassi'ely parallel computers Viah nurmrerous silnple,

. locdl', irl rconnected prucesing elemen0ts. SucIh potcntiall:) powerful archirectures will certainly

proh!icrate, pcniding ihiinent adX.,uc.C2s in \TSI tcchnolog) [Butcher. 1930; 1 l illis, 1981].

The dc2 i red suhliL [ions 1t0 n ;i S y IsIul problNems appear to posses certain global properties

• (cti,:.sitcn1._}", 'roln ,, Cinima l nge. etc.). ';%hiCh ar e \pre,,scd formally by dth variational

principle or associ<ted ) tial diff0rential equation tom iulatons.' Gi\en only local commrutnication

c;i;•mhilitics amnong proccssing olercits, however, global properties can only be satisfied imnirectly,

Utpically by itir.ti¼cl. propagating visual constraints across th1e grid network. Indirect propagation

can esuilt in substantial coinp UrationAl ineficiCncy., since the ctmputlational grids necessary for low-

lcCl ,isian ,ippi:;..iioos te:nd to be c\ticniely la cig. Convergence of the itCrative process is Often

,, "w is t % neal Iwoutrili/e ec ilput at ionul p)o%%cr offered by mrassive parallelism. Indeed,

for ii ne dlsrtan in, oil , Io, l,'i Z frid,. cxc ruc.iatiil21c slow convergence rates hase been obscrscd in

itcrlrýne algowith ors foir Lcorpinting iihotness [Blake, 1984, ,;e also H lorn, 1974]. shape-froin-:hading

[Ikecch'i ,: Ih1m. 19i"1: Silnithl i' I)21, optical I, Illorn & Schunck, 1981; Nagel, 198U, 3-)D

"1linM..s [( i lin i;. 1 9X-31 it'lopilo,. 19 2, 19,831. and Other visual reconsirtictiofl problems.

Sin f t ,patial Iccality of co tatia is deipendntllt Oiln SpA;d! iesnltitionl, local (e.g., nearest

nwi-hol) comoplitil ioioslM .l a It '.00!"c grid , ci a giscen region are aalogotus to imore global

C0fl1i L.nilll nS onil ai fil, emid er\ci lhe sa1e ipion. lIhis suggests the possibility Of coouitcracting

the secihness . -:lohInl intel .)"t :Ih by depio; ing local iterativc piOCesses over a nutiltiresolunion

h irebil s!'l) i, 'cli I. Ihi( is die I,&is, ()i" Pnrit,'r!, ro'il aioi t mct'hom which arc gaininig p utlality

m I;ll IM:Ic,:LI anl;.is ji l:C'kbuiseh & 1rmrerbcrg, 19S)]. lIhc conputtiat loni stirtctitlre of

:\,; i.m i il x:d ihlleirc ild iorl l ini t,.S l C ICl;nicil 111o10i0 i the ilrl -I agrý;ge CquaricIs of tCA e IC .AlC lus

t. it i l S. Qivcn pp ,puat,.e crnilli.':t and ,'illllrie.r (or seIf ,idjoiu,'n' ,) cLiWdt;t',,s [('ourril Sr Iiliheil.- 1953].1

1 FIH(OPOUI OS MLI I ICRI I \V\ 1\AISON Nil I IIIO)S

multigrid n,,edods bears an mtci csting an1,og) to the I111lii eolo non na(t ic of spatial frequency

channels II the human eiiI- visual system [Bladdick. ct (1.. 1978]. The methods are also related

to certain multiresolution ilnage processi ng C str:1'tores that has e been proposed, nlotably pyranlids

[Rosenfeld, 1984].

In earlier work, %ke developed an cfficient si face reconstrucuion algrithrn based on multigrid

relax.tion methods [lerl pulos. 0 I 9S2. 19 29831 d mwe sugO c sted, as hl; (i li-er [19S4], diat mulugrid

methods arc broadly applicoHlc in lo -ic\el co•nputcr ,,ion After a buief ox ri cw of multigrid

methodolog\ . wc app.1 it (1 duect ,thcr \x iion problems: the cll-known problems mr computing

lightness, shape-from-shading, and optical fly, from images. We de\clop noxcl muhtiresolution

algorithms for each pioblem. OT. empirical reIsulus indicate that these algorithms offer order-of-

magnitude gains in cflIcien).o mer their coaxentional bingle level counterparts.

2. Multigrid Methodology

Pionreti 6,'\csmetignts into n ultigrid methodology include the wkork of [Fcldorenko, 19611,

[ltdl. ;hohv,, 1966]. [, t. 1"273, 197-71. ind [Nicolaides. 19,77]. It has bccn applied to many

l)ounafjw \ai\ ,; probllem,, •c [irand, 1982j for an eteinm,\.e bhliograph•,) and ihere has also been

some de\clopenIcit in 1nc LOni).CXL 1 ','i,itaiamal p1,..ieoii [Nicolaides, 1977; li andt, 19801.

2.1. Multigrid Relaxation Methods

Iol tied relaxation inethods take ad\:imaiage of mu' iple discre:.iiations of a continuous

problem ovcr a ronce of rcslibtioni lescls. ihe co:,i ýr :vels trade off spatial resolution for

direct comnum,,ication paths o\cr larger distances. I lenc,. the) ieltis ly accelerate the global

propgation of in formation to ampihf\ thc ox.erail eflicieicy ) of th,: iteiati e elaxation process.

The inherent co:niutaitonal sluggishness of local iterative algorit1i;-s can 'c .-sdied from a

spatial frequency perspc-cfic. A lohca:l Fourier anal'.,sis of the error function (or, more conxeniently,

t[ie d. namic residu.l ct frlm nti r.:Y'; 1or,;ic 1tC ition to the next .imhvs that high-frequency con , ;iuents

of tic error - those coipo n'ieins xxth m en,!\clcixgihs0 tile order of the grid spacing - are short-

lixed. ,whcieas low -f!cqUIC'c. coipomicn,,s pe;'-,ist through many iterations 1l1randk 1977]. Hence,

Ctnmniltll (I., or I., ) ciifl fOrOlllim dccicaS.e shrpl 1.y during die first few i.eraloiiS, so long as there are

hlgh-f'IeqeLmCi:cv ComllpOnen1s 10 be iniuihilided, hot SOon degenerate to a slow, ,i5¢rnptotic dimillution

\when onl\ ',.- frequcncy c1 ,po•neits reitiain (see Fig. I). Iihis suggests dhat wh1ile relaxation is

inelliCenit at cnmiipItl CIt,] 1mn l nl.ting the cm or function. it a/ln be ser,. clliclent at sinoiihing it.

i t Il till',his iit o0 \ iCxx, ;:ie -iid ler;uchnl enables 'Jic etlicik uii:{ i siC'-, of flatiiio

ti be cxp!oitd om er J1 \ ide ;auO 4 spitiil Freqlencics.

i ' nipiri.Cal studies of model problems ( Poisson's cqtuatium in a rectamgle) indica'c that multigridcihi~~ds canl cn; •,e it C'catiallh order ()(N) umlber I ('I)rltior•, x, h Ae : I, te nuhr of 0 1 "

IIC !,d L,1- -CONi C1 th lul .h( tI. _ ._ 2

1 iR /OPOLI OS ML I I IUIID RI ..Ix,\1 ION MI mrlOm).

:.'

-.o

Figure 1 . Av.poKcr-rM rckiucliori h: I mit aion. '1h% riican su-e( nIicriul)error iplotted* as a I:n~t on t the ier~ti~flninib T !rA seq uciicc of ((iautm5-SciwcI) rcl;t.ation i3utoso surftrce

rcckiilbttuciio ifl ~gorohnir. I'lic curve c.x~ihi .it'-, -1Ipica!hcj i of IoeaLd iliwt~ ethiods:. Convergence israipid dmriig the first le-.% UriUcilos. bii qiiid.I dcgkcnizic~i- to slow \% o t~ error reduction.

nodcJC in the .!lidJMItvdt. !97~ T. Ih:s ca- be collipOcdl to typical Coii~llfxitiCS Of 0( .\' ) operAtoios

for the ;oliwiton (,F i o&Ic prohhlkins 1 standardJ (ingle level) relX~ation. AS a CoOSIýLCIIICC,

4 ~~ 0 011: T~ierid meiith'ods pomcnutalIy effc; dianimitic ailcleoes inl eflicieticy oxc Ct tandard rclaxaitioii weiciods

it'. ok-Ic' ci i-;ion ,:ýH cti ic. ~nc N tends to be '.cr, lr (older i Ol to 106, or more), F~or

conp;ra~ e omnplext. ana, c die total coemputitAtonal expense oif multigiid methods may beIi~~~dinl co)COO crut mal~chine indepenident ulin iS. Vihe basic work unit is defined as thc 3MOonnt.

of ciiiiphtaticfl rccquii ci to pcrform ni -)ie ratiofl on the finest grid in the hicrarchy.

M i II!:pta t",o:I (4 niltg~d ic ithods to isnall proccssi og has a niumber of features.- (i)

TifL:hl1ple visuall rcprcwwn.:ý'oris Coxcriitg ai ranie iof spatial resolutrios. 00i' local, iterati\e reclaxtion

Pro Cses dlxii prý 'ptc Conutlaits within each ieprcsewiational lcvCl, (iii) local coars(-to-fiflc

* J-m Ifjt:o/oI.o Pi! occCscs that allow coairser roproscntations to constrain ftiner oncs, (ik) finic-t-coarse

IC'ru'picC s'~ liit allovw fiiwCr ici~iiiU s o constrain and fi!,prove the cccurricy of coarscr

m'-.. lilt! (:-' (CII ( Pcii 0 c) c(, no'lH~~~i SchCiitht enable the hicrimchy oif representations and

C( tfpo icilii. proccs,,cs t,. coopeite: ,n %--'. linc reasiii a cli icienc%.

* ~~~~~in r withipid nictiln is the n1 .lcc piocesses tisoirlly are basic iclaixiitioii rietilodsS uch as

Gaus-Se dcl or Ji.oh)i velax sI c", tie prolonption processes are local I agrrrnuc (polynomial)

iiitc oLton, nd Wec rcsi i t;.,i; pr cesses are local ave tigIrilg Opetations. '11he eXaCt form11 of these

2.2. Discretlzatlon

A ppropix arc res.wrin -,iesc n he &cii cd by local discrtainofhccni llu0e."/.ti" o n

-i- . . . .

:It I/0P1LI OS \It I I lu1i1) RI] A\,.IIO\ XII I I10ID

problems. [he finte cicmcnt metiho.I [Strang & lix, 19?3j, a general a id pO cr1l'i local

disciletilation tcchnique, ca111 he applied diicctl o ariatioall prin.ciple Foiinii ulations of visual .

ru prwChlens [I cr.opulos. 19821. When dihe s isul problemn is posed as i pArtial differential equation,

louc1 discreti/dtion Imay be carried out using thie h.zil't d.worencc t,,iiwd ji-orsytdi & Wasom, 19601.

"'lhe basic idea h ibnd hic fnMite elleninl 1P.mtho1d is that a global app roxtI imation can result from

iniceiactions among imi\ \ ,,:I\ 'inilpic local aipproxinllitins. li-,is V, ciw:-np1lishcd by tessellating

thc contino•ots doniiin intao a larec nuimibcr oif smiall siubdoinii, @s elcinils It' /,l\\hte dimi/ensiollsd'pend onAa ,i d.l izai "ie A. I `C ,ippiox iination w ithin clcireint-, dcpciids on a simall number of

paraileters - the \:liies of the solhition. and/or sOilCe of its dcri\aci•es, ait ai set of nodes associated

walilh each clement. lhe ptm\eI of the method stems from the fact die local approximations can be

bahscd tn low-ordi pal nolanomi.i>. A his maikes it relatitkely easy to express the con: niioutS functional

as a discrete sunimattion o ,Ci all the element co'tributioits. If the arialiomnal prinnciple is quadratic,

dhe icsolting dic,:rete problemia takes the Foonn of a large system of linear CetitiMios .Atu" = 0,hliciCe uh is the L ccCLor (if nodal alesW!-cs. 'I lie finite element nietlOid can also be characterized as a

s\stcinlUtic pu•,.edure fIor generating finite element approximating spaiccs Alhose local-support basis

0 fo lictins m.kc A": se (i.e., miiost of its elemncits are /ero).

[1h fin( ite dikfuence id applied diffTrently. 1:. picallv a gil ()f nodes with spcings

proplirtlona0i to a pIraielCtCr hi set 01p) OCr the domiain. [he delrenti.al operiator is then replaced

b., tillite dilefrcmCcc eqiia:!'ons itol ing noda;.l •ariables at neighboring n,,des. The collection

oF" finite dcffcrcnce equations defines a distem etc Nlstem sAllich approximates the given diftcrcntial

ciiatololl. If the dmfl.'Crental Opcrit.,r is line,,r (,as amie tile :ulcr-I igrange equ,itions of quadratic

vi national plin•eple•) and a linear finite difference ,qp1,)xiIlJ'ta!) iei Ciilloy Cd, the discrete systemis agamin a aincir systemn V011 .'P. Althougli tie totl nunber of nJts 3.N is gcnerally large, each

fillire diffcrence Cquaition insow!es only a few Ilodal variibles. f i;el cbre. !:'c lear system is agahi

spalrse.

While the finite dIfference ii'iliod is gener,mlly easier to apply, the tinik CleFl mt Mi.iedid offers

a illuiicfh soundcr c(il, erg-ice thleor', as s Cll as a flexlill itv that alloA, thd -patially nonuai:Yriin

d!,,,Ir ei,/tioil of doii.iinrs li.Mtidý cnlplillctcd sh.pes. NIethI,,Css. both di,,cietit/ition techniques

yield large. sp A"rsc 'a ls s ('f iin1Cer count ions ill a x idC range of isuail appicaltions. A great deal

ol cldou n'11 ¾e. , i .m.ilis !<i<i, becn dirccted ',f0 the solutio1n of .,ctl V. stlls. Mllicli turn ouit to

he Ce pc-.iall, ) eftll ,iii t rd s-, iti I) y IO.,il. l',i.,l:l, itcr,lti¢c iliefhods,, p1,iwlclti ilali die relaxatiolniicti',,md, that ste h.i\v bee': L!'.sct-siumg.

2.3. Multigrid Structure and Coordination

Otlr spatial': oliifilWn dfi-scrt!/1tlol0, Of the conuitminus ,.,il [ri~me1l.s ill this papewr will

* yield ninifOrr gr-id2 at e,,dL. I \eel (Af the Wllltig.rid het! 'elmi,. A\l•mi~)Cc.a,, Of rtiilligrid i1iciflds .

"cain hC ,liiTpll i'd stibstaitpiil!, licia 2:3 deeie,ise I iii idf nsI'u•ion 1:11 Aii level ti the iICXt

'.'. .- .-. .' . -. -- - - - : .. .. . -,:':

r

-

I ,

IGI 10 1ýX\ o I I I] o~

i medium

rd . Icoarse* I

liutvre 2. Pos',ich eiij * ', c f ,:'csoltitn ;dv..rithm. A s.all] p'don of three lewls of the 2:1~ul tici id hWe r .- !: u.. (i hi , 'c.,,•-nighi k-ir i qiC r o cN,,or coinneclti i,, are included.

NI-.-lE,.c. I -\ i '1:1.i , I h . i,...:,l:' , ratio appears to be near optimal with regard Lo multigrid

co'.e,-cii,.ce ratc, jl; :: 19771 lip. 2 illu.,, c, a periton of three grids of a 2:1 mriltigrid

"*iC1i c rchv. Inl a I I , Uv' .i f, the ecrmIIal prf CV oi[ opcirtes at each grid node sequentially.

11xus , , I t*h 11', 1,H ,li ni1, lun '1htw,. each node represents a sCeparatC processing clenlent

Vl AN , diittMr ut: ,., - iiciW., j 1c ,ivt:ciuie (see 1ig. 2).

o Ihc t kitiictw. uth'" des.,:i, , ed utWilie simple h1jectin l:,_-.I forlhe ne-t cc o r'.w ; ,, tii*\ e..:r Int-rpol i i - ..> for the coarse-to-fine prolongation,

*and wjn iij.ai,(iv( ntigrkil C6.1dtUi 'c1e1Ce "l' hiehl wjs employed successfully in our surface

recontI~uc •,n al1-till ( 'e el l ipcs. 19S2. 1983] for details). he general coordination*• * airl,' fir ,t pCrt0 1.is a ,vifli.-iCni, ntnom r of relaation iterations to sohe the coarsest level discrete

- -. s•.ein .\S ½ 11 , -A" ;', red 1.corPct, (proaedmtre SOLVE). and then prI'cecds to die fine,;t level

. ---. acc( [ding to

"* -

n. - " ; " .- - -_.- - - -- ' - - - " .- - - " -- . ..- -.-- ' .- . .-- - . ..- -, . - -{ . - ". - - - - - '-' .- ;----- --- -, . , •

'ILRIOPOUI.OS MU[ I IORII) RIlAXA1 ION .MI1rIODS

procedure FMGuh. ,- SOLVE (1, uh', ei;-

for I +- 2 to L dobegin•h, _ Iil • h.-,

MG (1 •h f".)end;

applying the muluhgrid algorithm

procedure MG (1, u, g)if 1-1- I then u -- SOLVE (I, u, g)

elsebegin

f for i -- I to 7it [while ] do u F- RELAX (1, u, g);

d - .. h,- i, + 1 =.l-t1(g --. '\hU);

for i.- I to ',2 [while ... ] do MG (I-l, 1, d);u *- U I- Ii - ,(V - I - I U);for i•-l to n3 do [while ... ] Ius-- RELAX (I, u, g)

end;

,.\fter 7i relaxaiioi, iterations (procCdure RELAX) hac been perfbirmcd at level 1, MG pcrfomts a

restriction to tie ncxt Ctaoucr lcvel I -- 1. It then calls itself rccursivelk on the coarser level n 2

times. l'tIally. pe • i . po i,) olition from the coai.sci lcvel back to lc ci I, following tip with

13 more iterations or. Ic,,.l 1. I!.-h equations on the coarsest level I = I n;a) bc solved to desired

accuracy wAith suticienftl, n y itaerations (piocCdtirC SOLVE). One can rcadiý- show that when MG

is inl oked tin iCIC, X it -ill, RELAX a utu,; o fln' -(,, i .. i) times ofn lc. I / I and it calls

SOLVE n2'-' tinCs o|, ks:c:l1 1. In general, most of the relaxation iterations are performed on the

coarser levels [I lciikcr. 1980].

"The optional [whi 1 e . ] clauses denote conditions (hat. ma. be checked during the

coiMpouttioii and used to term| inate some iterations. l).$ntamnic cono iiito,,i, CoIl,,) coilCi gli;C,rates ilcast rcd hy ell norsq, ae-C lilncopolated into adaptive cotrdi natio;-,. fhllicses, ihcreas fixed

schemies ale controlled onl. by thie constants n1 , 112. and n?3 [lrandL 19771. Aldithoth adaptive

sl.heimes tend to be more Celiczient in practice, fixed schemces lend Lheni.ohlves better to thcoitical

anal)s'is and, mr•eiover, they a;c citsier to iTmpiclient on dismibuoted iocal-inteitconnect architectures

duc, in part, to the abselnce of cn or nornis Ahich require global compuaitions.

3. The Lightness Problem

"1thC IiIhtnICýs of a ,,uiht.C i, the ,&ceptiiid corrcl,itc of t, rclect.n c. Irr.Ji,ince at a point

in th1 image is frupiortioinad to the prodUct of the illulililtuMcC MiJ r,.ic'LI Xec aL hi!e] corrCes(nding

point oi -ic teUrfFce. 1 li light 1 ic. proble- i is to Cotllpte tl ohtnc- ll()ii iyic mi radiancc, ithout

any precise knowledgo of either rclccta.ncc or Illt Min:lncC.

6

-- -- - -- -- - -------.--.- --- -.--.'. - .-. -'."--- ..-,. --..-- " - .. ' . . - . ' " . ? 7 . ' ;- ... .. ).

%Y

. *B.*'BMUl~flI tLl101(1)4 R!)I AXAlI'ON NrTI 101 )S

3.1. Analysis

lThe retine\ ihcor•o')i oli•.lltnlc,•s and color proposed by I.and and Mc.ann 11971] is based on theobservation that ithi mince and relectlance patterns differ in their spatial properties. IIIhm inance

", challges are U1,'%ully graduh anid. theret1or'e typically give rise to smooth illumination gridienti,

while refl'ctance changes tend to he sharp, since they often originate from abnrpt pigmentationchange, and surflaee occlusions. 0rIIn[1974] proposed-a twodimitensional geeralization or the

1.and-McCann algcrithm for computing lightness in Alondrian scenes, consisting or planar areas

divided in to subregions or uniflorm matte reflectatncc.

".Let '?(X, Y) e)C the rellectance of the surface at a point corresponding to the image point (x, y)

and let S';,.,y) be die illuminance at that point. The irradiance at the image point is given by

, k,(, y) =: ,(., y) X I,?(., ,1). Denoting t(le logarithms of the above functions as lowercase quantities,

we have •x, (i) -.- , .y) , r(x, y). Applying the Laplacian operator A gives d(x, y) Ar(z, y)

•,(:, ,) + 4r(.r, y). In a Mondrian, illuminance is assumed to vary smoothly so that A(S(x, Y') is finite

. cvcrywhere, while Ar(-, y) exhibits pulse doublets at intensity edges separating neighboring legions.

S ).A tdresholding operator 7' can be applied to discard the illuminance component: T[d(z,y)]A r(x, Y) "(x, 11). 1 lence. the reflectance I? is given by the inverse logarithm of the solution to

Poisson's equation

~A i(• xr.,1 y)- f(x, ,V), i n n1,

where fl is the planar region covered by the image.

SIlorn solved the ahove partial differential equation by convolution with the appropriate

Green's function. We instead pursue 1 local, iterative solution based on the finite diflference

method. Suppose that I1 is covered by a uniform square grid with spacing h. We can

"" approximate Ar r, "i r.,1 u;in', the order h2 approximations r, -= (rý+ ,. - 2rhJ -i r,_,?)/h2

and', r 3 r' r-- )/th- to obtain a standard discrete version of Poisson's equationh -- r--I-,"- I,,j" 4 r,,,r. .,j This denotes a system of linear equations with

,Sparse coefficient matrix.

Rearrangting. the Ja•cobi relaxation step is given by

I I,j ,nI h 'Ih

where-ihe -)rackcted superscripts denote the iteration index. Jakbbi relaxation is suited to parallel

synchronot s hardware, whereas the Gatuss-Scidel relaxation step given by

11., 1" ) 11r' •. ) (n .,, - )- h (n) A (n 1) h'-j ,j -I1

"is more sui able on a sCriafl computer and, moreover, requires less storage.

7

* I~FR/OI(.)LI),S 1IM I OSt I (I) II ,AXAtION M11TIIOIS

N

i

Figure 3. Symtheitcd Mondri,tn ijfltj'-* t1ic ýc inages. input to th1e al00oriih11, kniatin patches or uniformrefleetmuice and a 1 t'-to-i ijh t il in ii nt ooa i craddi ent. "li t ihiree sinallcr iinagcs are in creasitngly coarser smunpledversions of the lae c t inu:i;tc which i l:i ",. t2!) pixels., qanhlited to 256 irradiamice levels.

We note in passing ihat IPoisson's Cquation Ar f I is the I'uler-li.agrange equation For the"variational prilci[)le associated with a menmbrane problem. The solution can bc characterized as

the deflection v(.r,y) - r(x, y) of a membrane subject to a load f(:,, ). and it milnimizes the

potential energy functional S(,) -: ( fn 21(,,2 ' ,I)- fdjXI y [C'ur;,i I lilbert, 1953]. Blake119841 olfers an alternative variational principle for lightness. ,, ; ,. '.:htncss problem asa variational principle permits the direct application of the f;.tt, elemei di,,icti,,ation method,

"which for instance does niot require a uniform discretization of 0.

0 3.2. Results

A four l\ d Outwit lh te lgorithm (with grid si.fI. 129, 129, 65 x 65. 33 X :13, and17 x 17) w\as, teted onl a stithesi/,cd Mcndrian scene conlsisting ol' patches of unifomi reflectance,subjected to .n illuiinaition %,hich increases qtualdrMtically from left to right. The original image,

"which is 12!9 x 129! pixels in sie. and three co,,rser-sampled versions oiwe shown in Fig. 3. All

images aire quanti/ed to 250 irridiaince levels. 'The grid hiunvioI> f -,Ig,' in Fig. 4, was

"computed bM inaincining only tie peaks in the ILaplacian of rqt /.ero boundary conditions were

provided around the edges of the images, and the compuLtition '.; as stirtcd from the zero initial

appirox imatuion r"- 0.I,,8

;'.- ap~ l.Olilililgl i ~t,, :_+ O

"o.

n ~ I R~Rii IlAX VI ION Mt( I II(DIS

--4I-A ice-r stieOb a'u ungo l ,

ek

W. jihl4.s'ls f(tc iii iaiolg a irt

I)j :1!11 Ik.e flom Coarsest W inot respectiveamgoly is1,12 100k,

62. mtid 1!1' *. I,~~. 17 l'c i!ittic'. ay cidmi inrequired '1101.t 5001 work utnits to

~~a)Ittj)Ilt:; *'' '" '" ;It rICh !!c Ai~ t c 1 in isoulation. T[he single-k'% el .ilgorithmncqutc. ~ .. . ~'t- cotjerin CUiii- -~s 1licro are notivs Iross the so rt~ce, since

ifl~~C4 L';l,(,hr~s in) one iterotion. The mnultilevel

li1'h~~~~~ilCN~il 1*1711 lol . 1 .i ~ ' 'IC ttrt Tnin 't effcclivel) it tile

4. Tim, S'~m 2~ir

* tmf ? *I * '. '. aL ge~ometry, scene iflumnitance, ;tirracc

rellct~tn'c.. ~q'.. :otna iiiding problem is to recover tile shape of'

Fr ~p * *~n''a~ i~hnmii-ce. refleetance, and imagintg geomnetry

* *~~ire Coiý!r..t A rI,*'*' in hv rvla,dti directly' it smi~e riCmio.

Let )!hv*'' ' . ~*v1.~t lbeo defIined o%cr a1 hminded planar region f).

r)

¾ .* .. ':* :: ': Ji~~.. :e.K~-.K :: - .

-r---. r-- 7r-- •: •£. -7a-, Y ,-r v-. -I. -. r - -,-.n- -- . -...-.- •. . .

ITR/OPOUIOS ML! IlKRtID RlA.I..\XAFiON NI HIG9S

Figure 5. The rCCOISIl, cted .1'wdnan. this i- the solution computed after 33.97 york LnIMs by the four-level

lightlness dlgktthin Ntu, t (A th.: ilhmml ;!ii ,,n gf ir,lnt .1 ig. 3 has ben clinainated.

"" he rclatiorsh p beteCCn the sL rt'acC oriCnt,,tion at a point (z, y) iand the image irradiance there

lN(S, y) is denoted b. l?(p,q), whcre p -- u, and q - a tre the first partial derivativcs of the surface

function at (x, y) The shape- from-shading prhebcm can be posed as a nonlinear, first-order partial

difAerential equati;Un in two onk noil s, called the image-irradi.hce equation: ,x(, Y) - R(p, q) = 0

[1Huorn, 1975]. SUrflace orientation cmanot be com)puted strictly locally I.)aeatse image irradiance

provides a single measurement, while su rnice orienta.ion has t\, 1' depe. ,l.i components. The

image irradiancC CqUntion pruo ides ol)y one explicit conmtraint on ,urfacc uiuentitmon.

Ikeuch andl i lorn [19811 proposed an additional surface snoomhness constraint and the

*S Oe o •suc eclutdlmn, contOnS, sJ s boundaIry co{ldiiollns. Since OWe p-q pirameterization (t

SLit fiaC orientation becomes unbounded at occluding contours, hmwvecr, surface orientation was

rcpadramcter/Cd in t oi as o" the (hbooled) stereographic mapping: f - 2ap, g = 2aq. where

( --: 1"( -, v"I 4 .:.) q'l.

Ilhcse cmnsidcr.ations are forini,ili/ed by a vili.ttioal principle insolvuig die tninimi4'atior. of

the functional

) J jw(f I ) I ( c4)(! dy + 2 J f ,) lf f, q)i2' dr dy.

The first integral inc iporate. rilite surface smootihness camtrai nt. I liC second is a lCest-squares term

'lit.h n.oerces die suoution t" atislJ l 1g the imal.gC irtad:.oucc equtitnil i1,, 1reatg the equation as

10

• ? -" ".- '... .. --• .. . "-' '. -.. .. - -.- - . - . . . . - '" - - " . .. .. - " . . . " . . .

7( 7

1 ITl']ot / 11Ot. I I)I i It ll) R11 AX lI0N Nil- 11lOl)S

i AgI

Fiouare 6. 1 '~i'', it phci I S. Tlhese ,yntllh ic iuil;•e• p lIptI ItIit ,algorilhInI show a VIIa bertian.phikiL dSI,,1!, , - ll.w'• 1'om t~he viewing direction. M1we three smaller im iges ,ire in.rc;sinaly coarsers:amlpled •ca';,,i., ! th Ij 1. iul:i;c which is 121) . 129 pixels, qutaltimld to 256 irracli;aice levels.

Sa penally cont ruint .Icwhitcd y;a- factor X.Other variational-formulations for shape-fronl-shading

have been ,c'K.'. .. t.. [ln()ioks & llorn, 19841.

The I1ulci- i , e';;i,•, Cquatiouns are given by the system ofcoupled partial ditlerenlial equations

-•.i~~A y/. [..(e) "le(/,g!)'1l1 0" ,I! ,_x.,/ - Xl¢' : R -I(/, y))]It, 0-.

S)iscreri.i ;',, , ,,,• on• a, uniflorm Lrid with spcing h using standard finite diffecrncc

Ip1, I11'c J.1ucnbi relaxation scheme

i" V e l~r!•' i(O XI • I y,,..',

~ I' I' I1ad *''"vh r 4- 1 •'. 1*"j i11, ,~ , '] j I h6 •,hee I'' . 3" • ,,,, " ]" ,.;t (i l-, .,( fa "t " of ' ,, '"I•1 g:,. 14,,,

e hc, .' node (i,j) litor of has been absorhed into X), Itf

d-l?/f. af, ,': ,',,q. ()in a sequenrtial computer, we prefcr )o use the analogous Gauss-Scidcl

relaxati mia i ,, i, ,..,,.• .di. due to its gre•ter stability, faster co•nvergeflcc, and reduced

1 merno r I..l ... i,,,,priu boundary condiions can be specified at occluding contours in

the image.

*.*ej .- % .S , ' * ,' :.. ." - .

TERZOPOULOS .MUI.IIGRID R-I AXA1ION METHODS

Figure 7. Surt'ice normnws of the I mhbcrtian sphere. The solution at the four rc),,luti)ns ,hat were obtainedafter 6.125 \,oi uniL, ,re shown.

4.2. -esults

A four le\cl sh,,e-trom-shading m•gollt n (vNith grid sizes 129 X 12q. 65 x 65, 33 x 33, and

17 x 17) wAas tested on a •,u thci.,ly-gcner,,, A image: of ii .anibmrtian sphere distantly illuminated

from the vie ijg dilection b) aI mn: Mouicc. The original image, which is 129 x 129 pixcls in size,

and dl ce coilC,-SaImp!cd vCrsions nrc show.in in Fig. 6. All images arc quanticd to 256 irradiance

lcvels. For the IF.nhcrtian su rface. we cmplohcd the expression R(f•.. rmax, cosi]. wherem2 -• (2+ ?11

Co,' _ I O(ff -+ , (1 - ' ) -6 ,- j ,( -f- f i J, '.1-4nd "shere f, and

y. are td light source direction componeiits (Ikeuchi & Hlorn. 1981]. and ,irialogous expressions

for its dcriati Vs 1•/ and R•. "11 h 1 tiltatti(il of die ,tjrfacc va,, Spccificd aounid the occluding

contour of die sphere. ,nd Iy ticatine the contour it.1lf as a possileIC orientation discontintuity, thc

grid functions f and y wcrc allow,.d to make discontinuous tran~itions aý.ross it. Computatiorn was

started from the tcro urinal ,tpproximation y : - 0.

120I

TERZOPOCLOS %I U.II CRItI)D RI--AXArioN %F: 1 IODS

x/

y% Y

~J

N / /4%

Figuc 8.'ii f ipic~rI~ f. f the Lxnbeltini sphere. ibe dcpth representaitions on the Ilof %~ere* ~~g~rcnraicd b y hr untucin lgorithmn in 8.9 work tinis timing the norm~al L'Ltors in

F~ig. 7 it, ricn aiin cowmmnY ~is. O n hlc right. the 'fflcniation cmi~rhin is are dcpicWd as *'needicb" on thefCCe)nStinK((d Sill I;LCCb. CI[\ 111e thiiec C ~irscst Iskarc shown, smnLC Ltic tirist resolution surface is tcx) denseto render it. ;i3-D per,,;ecmvcpl.

r 13

"IERZOPOULOS MULIIGRJD RELAXATION MEfIODS

The solution computed at the four levels after 6.125 work units are shown in Fig. 7. "11he

total number of iterations pcrfomicd on cach levcl from coarsest to finest respectively is 32, 10, 4,

and 4. In comparison, a single-levcl algorithm required close to 200 work units to obtain a soludon

of the same accuracy at the finest level in isolation. As in the casc of the lightness problem, the

single-leel algorithm requires at least as many iterations for convergence as there are nodes across

the surface, since information at a node propagates only to its nearest neighbors after each iteration.

Convergence is somewhat faster, however, because shading infonration is available at every node

inside the occluding contour to constrain surface shape according to the image irradiance cquation.

In any case, the multilevel shape-from-shading algorithm is again much more efficient because it

enables information to propagate quickly at the coarser scales.

To obtain a representation of the surface in depth, the surface normals in Fig. 7 were

introduced as orientation constraints to a four-level surface reconstruction algorithm with identical

grid siues [-erzopouloS, 1984a]. The normal vectors were first transformed from the f-g

stereographic parameterization used in the shape-from-shadding algnrithin to the p-q gradient space

parametcriation used in the surface reconstruction algorithm using the formulas p = -4f1/(fI2 +-2 _,i) and q --4g g' - 4). Nodes outside the occluding contour of the sphere were treated

as depth discontinuities. Fig. 8 shows the surfaces generated by the algorithm at the three coarsest

resolutions. The rcconstruction required an additional 8.8 work units.

5. The Optica! Flow Problem

Optical floA is the distribution of apparent velocities of irradiance patterns in the dynamic

image. The velocity field and its discontinuities can be ,,n important source of informlation about

the configurations and inotions of visible surfaces. The optical flow pioblem is to compute a

velocity field from a temporal series of images.

5.1. Analysis

I [orn and Schunck J1981] suggested a technique for determining optical flow in the rc-,ericted

case whcre the observed Nclocity of image irradiance patterns can be iattributed directly to small

intcrf~anrC motions of surfaces in the scene. Under these circurnstinces, the change in image

irradiance at a peint (.r,y) in the image plane at time and the motion of the irradiance pattern

can be icl,tted b) die flow equation L-'u-,',v., - -', =-O, w here E(y,y,I) is the image irradiince,

and u ý dx/,l and v -= d~i/dt are the optical flow component functions.

An iadditional con-iraint is needed to solve this linear equation for tic two unknowns u and t,.

If opaque 5l oject5 undeigo rigid motion oi d fonnation, most poimts [i:,.c a '.,elcity similar to that

of their neighbors, except where surfices occlude one another. Obsermrg that the velocity field

vvaries smoodtly almost eer),yhere. optical flow can be determined by finding the flow functions

u(.r, .1) anrd v(r-, y) which inininlize the functional

14

• . - - ._S. - . " i - - . " - . i •" - - . •

IERZOPOUI.OS MULTIGRID RIIAXA1 1ON MN1 I IODS

((U, V) r1 a JU 2 t42) +(t,2 Vý) didy +J j(/'u + E~v + Et )2 dx dy,

•here a is a constant. The first temi is the smoothness constraint, while (le second is a least-squares

penaity expression which coerces the flow field into satisfying the flow equation. Rclatcd variational

fonnulations of the optical flow problern havc bccn suggested (e.g., [Nagel, 1983], [Cornelius and

Kanade, 1983]).

The Euler-Lagrange equations for zhe functional ! are given by [Horn and Schunck, 19811

EI', u -+ - E 2 -V a 2EAV- _ h-.

Assuming a cubical network of nodes with spacing h, where i, j, and k index nodes along the x, y,

and t axCi rcspctively, v,c usc the f'ollowing finite ditTerence fornulas to discretizc the differential

operators

, h h /,,, ,:h~l ,)ILI,,k -- (

h 1 i,h h

h -I .- (ph

,h + h h-

I,(u,_. th %+ +~ u,"+ 1h%,.•* where 1(+,". ] -- _., L . ,,, and

-I(%.h % ,h vj.,[ , ( , . ,' . '). Other approximations are possible, including

those suggested hy i lorn and Schunck which, however, require uoer four tines the computation per

iteration to gain some impr)-d attcnutilon o.' high frequency error. Gien di namic images overtt i,:mt three fraims, it s:mmptric central ditere fre ormunlac h I eji .,I.k -- , )

%wu~ld bc preferable, pit;Ididep it is steble.

SUbI,titUtilg the a10\e approximatwon, iinto the lFuler--I agrange equations and solving for

uhl. and h '. Nwds the tollloting Jacobi rclaxition formulas

tpr (n) . ,

(,i)

V h (n + -l , h , h,. " ,• k ( h (n)

.. .. . . (1, , ) 2 -•- and

.-l . o, ' I' i ' . ., The naltr rl boundary conditions of Ole zero

niorniil derivatiwke ame appropriate on Ihe hound,iries of surfacce. lhey can be enforced by copying

"' x alues to boundamy roJdos firom neighhoring ii:tcrior nodes.

15

T.RZOPOUIOS MUi.TIGRID RELAXATION METHODS

IlI:

Figure 9. La mbertian sphere images. These synthetic images input to thc algorithmi at four rcsolutions depicta uniformly expanding Limbcruan sphere. diýuindy illuminated from the ,,iewig direction. Frames for thefirst time instait are shown to the left of frames for the zccond time instint.

5.2. Results

A four kIvcl optical flow algoridim (with grid si/es 129 x 129, 65 x 65. 33 x 33, and 17 X 17)was tctcd on a synthctica!ly-generatcd image of a L.amnbcrtian sphere disuntly' illuminated from

the viewing direction by a pM)ont source. The sphere expanded unilimnnlv over two frames. The

first frame, \%hich is 129 x 129 pixcls in si/e, and threc c(,arscr-sampled versions are shown in the

left half of Fig. 9. The next frame, in which the sphere has expanded is shown in the right half

of the figure. All iniagcs arc qunti,/ed to 256 irradian.e levels. [he %,ocit, field was specified

around die occluding corntosur of the sphere, and by trcauirg the contour as a p,'•sible flow field

discontinuity, u and , were illos ed to mak- discotMinuous tr,insitions across it. 'Ihe computation

Aas started from the /ero initial approximation u = t, - 0.

I he solution conmputcd on the three c arsest lees c aifter 4.938 work units arc shown in Fig.

16

p: ; : ' > " . " " - - . - - . : " . : , . . - i . i i . ? . - " i . ' . -

TERZOPOULOS MUILI IGRiI) RI-t X\V IION Mil I IODS

11

I III[lt!III\\\\\\ t I 1 . / .......

' '\\\\ ' /'I1"////

•.x\ \\\ t iI I Lii///,,/,IN.N. NN.N \N .\\ bI A A d III% • | / #/ ¢

at.~~ .........-.. ~

-**/ 1 - A A % ", x '.,.-.'

~ t I j ~ % . %..- - - - -- - - - --

F igure, 10. R -th O o•pan-,I g 1.•ril',-criwi~ sphcre. "lhc solutio n ait the tdifc,. coairsest resolut-ions

-. 4.9/, 1) Ieb to~l depict).~Figure -0. / 'I-I " *''%lS N

di;qt %%er,- oý)1,,:11_',;I ; .3 ,,,.oikt u c-.: i.', c :,hown (ýhc filiest-lc',cl solution i, tw e s o d pc)

10 ivs %elocityv ýeL.tOrs in .tj-sp,jcc. 'llc totali jiu]r.bcr of iterattions, pcit'Orrycd on each level from

L~O ýrCSIt o lwi cý t rc sp ccti elý is 40, 5, 4 , inJi 3. ()n j~n? ll '• , a i nigle,-lec ]c algo r fith m re~q u ired

37 "soik units, to Obtaill a 1,0ltiLIm (,.[ G-c saijme a1ccuracy it die finest level in isolation. Again,Fige 10. Vellccl a elforth is iste eipcanigicicnt bc,; i iaus it propagte. s infolination quickly tt (he coarser

scles. C ,l/ c..[ 119841 ;lso repwor;, ,Ape:,.tn'nl c n estten cith ours Aish regard to thedconergcnce

r sate of i t..u1lIt. ecti O et y 400W d.4lthm ran.)3Inc to a single aeve aiglee(1' i g, lie r employed the

I Io; u-Scht ni'k iclaxation foiti.•l.rs, for his im0plciient,• .

17

0 - - " . - - . - - -- . . . i i - -- -- .' i - - . i - -- i - .• - - - £ i"i ' i ". . - - i -- . -. - __ .- . i -- . i

TERLOPOULOS MULTIGRID RELAXATION METHIODS

6. Multigrid Methods, Regularization, and Stochastic Relaxation

A pririarv purpose of low-level visual processing is to reconstruct relevant physical charac-

teristics of 3-D scenes from their images. We have considered in this paper three different visual

rcconstruction problems - the computation of lightness from an image (a 2-D, static reconstruction

problem), shape-from-shading (a 3-D, static problem), and optical flow (a 2-D, dynamic problem).

It was possible to apply mulbgrid methods because each of these problems x&as formulated as a

variational principle or associated partial differential equation.

As inverse mathematical problems, visual reconstruction problems tend to be mathematically

ill-posed, in that existence, uniqueness, and stability of their solutions cannot be guaranteed a

priori [Poggio and Tforre, 1984]. Among the sstematic techniques that have been developed to

tackle ill-posed problems is the method of regularization [Fikhonov and Arsenin, 1977]. Through

regularization analysis, ill-posed visual problems can be restated as well-posed variational principles

by restricting the possible solutions with appropriate stabilizing functionals. In general, the

sm-oothiess properties of stabilizers must be controlled near discontinuities [rerzopoulos, 1984b].

I - tingly, the same stabilizer was used to impose the smoothness constraint in both the shape-

from -shading and optical flow problems.

A major attraction of regtilarization analysis is that it leads sstematically io variational

principles which pemuit advantageous use of multigrid relaxation methods. As a ,isual algorithm

design strategy, regularization analysis applied in conjunction with multigrid methocology promises

to inmpact on a broader spectrum of visual reconstruction problems, including image reconstruction

and discontinuity detection [Geman and Geman, 1984], stereopsis [Marr and Poggio, 1977],

registration [l1ajcsy & Broit, 1982], motion field interpolation [llildreth, 1984], shape-from-contour

S[Brady & Yuille, 1983], and structure-from-motion [Ullman, 1979].

An issue of concern is that the regularization of visual reconstruction problems cannot always

be expected to lead to conivex variational principles having a unique absolute extremum, without

relative extrema. Unfortunately, classical relaxation or gradient descent methods are not directly

applicable to nonconvex variational principles, since they oflen get trapped in relative extrema

S•,chaslic reIaxauion algorithms (such as simulated annealing) do not suffer this disadvantage

[Kirkpatrick ct a., 1983: lHinton & Sejnowski, 19831. Nonetheless, since stochastic relaxation

,carch:es for absolute extrema with processors thdat are restricted to local interactions, it too suffers

serious inefficiencies in propagating constraints. The inherently slow convergence rates are further

aggravated bý the nondeterministic nature of the local computations. Multigrid methods may

ameliorate these problems by facilitating constraint propagation through the use of coarser scales.

7. Conclusion

Many important problems in low-level computer ision can he tormulhited as variational -.-

18

4 " . - ? . . ° _ ? . - i i - i i G . . . . - ? • / . • - - . • - . . .

TERZOPOULOS MULTIGRID RELAXAI ]ON SMI-IIIODS

principles or as partial differential equations. A particular source of such formulations is theregularization analysis of iil-posed visual rcconstruction problems. Once discretized, variational anl

dif'crential fornulations are aiumenable to numerical solution by iterative relaxation methods, whichreadily map into massively parallel computer architectures. However, dist-ributed local-support

computations are inherently ineficient at propagating constraints over the large nctwork or grid

rcpresentaticns that are encountered in computer Nision applications.

In our previous work on surface reconstruction algorithms, we established that multiresolution

relaxation techniques can overcome this inefficiency, without sacrificing the local-interconnect nature

of the cot iputations. This has been corroborated in the present paper by successfully applying

muldgrid methods to the well-known problems of computing lightness, shape-from-shading, andoptical flok fromn images. Tlh i;c J ni ,'tiresolution algorithms that we designed in the contextof each of these problems were shown to be substantially more efficient than the published single

level versi.)ns.

Be.ond its effectiveness as a (local) convergence acceleration strategy, our adaptation ofmuhligrid nethodology also leads to iterative algorithms that compute mutually consistent visual

rcpicscnt;i.ions over a tage olfspatia! scales. Multircsolution representations appear to be crucial* in itef inig low-l)vel w isual processing to subsequent tasks such as recognition, manipulation.and navigation.

iL,

Acknowledgements

.'I his paper derives from thesis research supervised by Michael Brady and Shimon Ullman. Insightful

comments on a draft were provided by Michael Brady, Michael Brooks, and Berthold Horn.

19

E

,'," '"'.'"""'.i 1 >:2","_'-".'-".". .'2,"." .. "' , "'" .. ...""-' " ""-" ."""" ---"-" " """"- ""- - -"-'-• " '

TERZOPOULOS MULTIGRID RELAXATION MLrHO)DS

References

Bajcsy, R., and Broit, C., [1982], "Matching of deformed images," Proc. Sixth Int. J. Collf PatternvRecognition, Munich, 351-353.

Bakhnalov, N.S., [19661, "Convergence of a relaxation method with natural constraints on an cilipticoperator," Zh. vych. Mat. inat. F'z. (USSR Comp. Math. and Maih. Phys), 6, 861-885.

Ballard, D.H, Hinton, G.E., and Sejnoisski, T.J., [19831, "Parallel visual computation," Nature, 306,5938, 21-26.

Batcher, K.E., [1980]. "Design of a massively parallel processor," IEEE Trans. Computers, C-26.Blake, A., [1984]. "On lightness computation in Mondrian world," Central and Peripheral Mechanisms

of Color Vision, Macmillan.Braddick, 0.. Campbell, F.W., and Atkinson, J., [1978], "Channels in vision. basic aspects,"

Handbook of Sensory Physiology: Perception, Vol. 8, R. Held, H.W. Leibowitz, and H.L.T[cuber (ed.). Springer, Berlin, 3-38.

Brady J.NI., and N'uille, A., [19841, "An extremum principle for shape from contour," IEEE Trans.Pattern Anaiysis afd Machine Intelligence, PAMI-6, 288-301.

Brand, K., [19821, "Multigrid Bibliography," Multigrid Aethods. Lectuic Notes in Mathematics,Vol. 960. Ilackbusch, W.. and Trottcnberg, U. (cd.), Springer-Vcrlag, Ne, York, 631-650.

Brandt, A., [19731. "Multi-level adapti',e techniquc (MLAT) for fast numerical solution of boundaryvalue problems," Proc. Third Inter. Conf. Num. Mch. Fluid Mechanics, Paris (1972),l.ecturc Notes in Physics, Vol. 18, Springer-Verlag, Berlin,

Brandt, A., [1977], "Multi-level adaptive solutions to boundary-value problems," Math. Comp., 31,333-390.

Brandt, A., [1980], "Multi-level adaptivc finite clement methods: Variational problems," Specialo'opics of Applied Mlathematics, J. Frehsc, D. Pallaschke, and U. Trottcnberg (ed.), North-

l-lolland, New York, 91-128.Brooks, M.J., and Horn, B.K.P., [1984], A variational approach to shape from shading, MIT A.I.

L.ab., Cunbridge, MA, Al Memo No. 813.Cornelius, N., and Kai|ade, T., [1983], "Adapting optical flow to measure object motion in reflectance

and x-ray image scqucnces," Proc. ACM SIGGRAPH/SIGGART Interdisciplinary WorkshopMotion: Representation and Perception, 50-58.

Courant, R., and Hilbert, 1)., 11953], Methods of Alathentatical Plhysics, Vol. 1, Interscience, London.

Fcdorcnko, R.P., [1961], "A relaxation method for sol'ing elliptic difference equations," Zh. vych.Mat. mat. Iiz. (USSR Conip. Math. and AMath. Phys.). 1. 922-927.

Forsythe, G.E., and WNasov. W.R., [1960], Finite Difference Methods for Partial DiffertntialEquations. Wiley, New York.

Geman, S., and Geman, D., [1984], "Stochastic relaxation, Gibbs distributons, and the Bayesianrestoration of images," IE'EE Trans. Pattern Analysis & Machine Intelligence, to appear.

Glazer, F., 119841, "Multilevel relaxation in low level computer vision," Multiresolution ImageProcessing and Analysis, A. Rosenfeld (cd.), Springer-Verlag, New York, 312-330.

Crimson, W.F.L.. [1983], "An implementation of a computational theory of visual surface inter-polation," Computer Vision, Graphics. and Image Processing, 22, 39-69.

liackbusth, W., and lrottenberg, U., (cd_), [1982]. MultigridlMethodsx L.ecture Notes in Mathematics,Vol. 960, Springer-Vcliag, New York.

Heinker. P.\V., [19801, "Introduction to multigrid methods," Co/niqium Numerical Solution of Partial.Dfjorcuniial l:'quatiuns, V.G. Verwer (ed.), D)ept. of Numerical Mathematics, MathematicalCenter. Amsterdam, 59-97.

20

L.

i" "'°'" "- -'" " ' " ' ' '- " .-t . .. "" " - " " --"-". ' " -" "' " -"" ' ' " " -" ." - --- - " " -.. -i i-

fT-RZOl;OUi.OS .MULTIGRID RIIAXA1 ION M.- I IODS

11ildretlh, E.C., [1984], "Computations underlying the measuremcnt of visual motion," ArtificialInic/ligctice, 23, 309-354.

r Hlillis, W.I)., [1931]. The cornection machine, MIT A.I. Lab.. Caimbridge, MA, AM Mcnio No. 646.

Ilinton, G.E., and ScjnIomkski, ,J., 119831. "Arialyzing cooperative computation," Proc. 5"' AnnualCn. I. c" CL L,'o.'vc S'ci"/ of A lItcrica, Rochester, NY.

Hlorn, , [1974], "')etennining lightness from an image," Computer Graphics and ImageProcesi,;. 3. 111-299.

l'orn, B.K.P., 119751, "Obtaining shape from shading information," The Psychology of ComputerlI ishi, P.14, Winston (ed.), McGraw-tlill. New York, 115-155.

Illorn, II.L.P., and Schunck, II.G., [1981], ")ctermining optical flow," Artificial Intelligence, 17,185- 203.

Ilummcl, RA., and Zuckcr, S.V., [1983], "On the foundations of relaxation labeling processes,"IKFI: T?-,'n. fAztprpi .1t ;alvsis and Machine Intelligence, PAMI-5, 267-287.

Ikcuchi, h., and I lorii, R.K.P., [19S1], "Numerical shape from shading and occluding boundarics,"Al tt-,cia,' c:igjit', 17, 141-184.

Kirkpatrick, S.. G(elatt, C.D)., ,lr., vecchi, M1.P., [1983], "Optimization by simulated annealing,"Sc.e ice. 220l, 671-680.

I and, .t I., and McCann, ,l.J., [1971], "L.ightncss and retinex theory," J. Opt. Soc. A-tcr., 61,

\Marr, )., and Pogio, T., [1977], 'Cooperative cornputation of stereo disparity," Science, 195,"283- 287.

Nagel. II.- I.. [119S3h], tConisraints for the estimation of displacement vector fields t'roni imagescqu.cc\.' l'.',,. , n:i,. .1. (iuv A/1, Kairlsruhe, W. Genmany, 945-951.

r Nara.aan:i. ..\., O'lWeri, I).P'., and Roscnf'ld, A., 11982], "Irmage smoothing and segmentation bycv ,ri,.imi.,ati,...' fI Trans. S)'i ue Man, ,21nd (Cvbernetics, SNIC- 12. 91-96.

Nicolhide,, R..\., [197]. "'On the 12 convergence of an algorithm for solving finite elementeq, at iols,' ,Mlath. ( omp., 31, 892-906.

IPoggio. T.. Iand orre, V.. [1984]. Ill-posed problems and retilarizafion anidysis in early vision,N1I I A.I. 1ab.. Cambridge, MA, AI Memo No. 773.

Rosenfeld, :\. (Md.). 119841, .luluiresolution Inage Processing and Analysis, Springcr-Verlag, NewYoik.

Ro,;enfeld, A., tlliiuincl. R.. and Zucker, S.W., [19761. "Scene labeling by relaxation operations,"Ii.! 7,n.v S•; ic,'n! . Man. and () bermtins, 6, 420-433.

Sn iii, i . . [,'M I ,'.h reca c. f S1 Iu frace ol:ieiitaion from image irradia nce,'" Proc. D,) R PA* /';.i " ', "nd~ a'iI:I,; ,i~ lt' ,k zapl P,tlo Alto. CA\, 132-141.

Strmang, G., and Fix, (..1.. [19731. ,11 An.Nvsts of the tlinte Element Method, Prenticc-I hall,i~ o.,n~xs (1T.1l, NJ.

"l'Tcriojulo,, 1)., 119S21, NUltl06vel -eConstruction of visual ,wrfaces: Variational principles andhlitc elcincri repreentationsi [I* A.I. I al., Camuhitilc, MA, AI Memo No. 671. reprintedin Iah. .\,,',b, , I /'vccx%ýng i,;:1 .,:fc %i.%t, A. Roscnfeld (ed.), Springer-Vc'lag, NewYork, 1984.

Iertopoulos. D).. 119S3]. 'Nliihlilc'c computational processes for visual surfice reconstruction,"(, t c• a; i , ,-s. ,ad hmats' Procc.sing, 24. 52-96.

l'cr/oloiit lo, I)., [1 94;,I, M.N tie(sIution colmputahion of v isile-surface representations. Ph.D).hL',,I 'Clj.'.rItilct o![ Ilecc,ld Inuincering and Compputei Science, NI I1. Cambridge, MA.

SlcriOpoti ,, I).. [198-11]. "ConlIr(ýllc'd-',ll)(thllcss sLlhlicrs for the regulari/ation of ill-posed•p, uat' I, ,,Cl ii ing discont nuitics," Proc. I)4,RP.- 11mage L'Undertianding W.orkshop,

Ncv, 0-)m',, Nocenhcr, 225--229.

210

. . .

TERZOPOULOS MUI.TIGRID REL\XATION METI1iODS

"Tikhonov, A.N., and Arsenin, V.A., [1977], Solutions of Ill-Posed Problems, Winstorn & Sons,Washington, DC.

Ullman, S., [19791, "Relaxation and constraincd optinization by local processes," Computer Graphicsand Inagc Processing, 10, 115-125.

r

22

t"

!'? .-- ; '• -. i •- -:," " -•L ;- .. i. -. - '... . . -. . .--. ... .-. . .. . -. . . .-. .. . .. .--. " -'•,5 -'•,? -i'..-..