mumford-shah energy functionalkato/teaching/emm/01-mumford-shah.pdf · cartoon image! (f, Γ) is...

ZoltanZoltanZoltan Kato: PhD Course on Kato: PhD Course on Kato: PhD Course on VariationalVariationalVariational and Level Set Methods in Image processingand Level Set Methods in Image processingand Level Set Methods in Image processing

MumfordMumford--Shah Shah Energy FunctionalEnergy FunctionalZoltanZoltan KatoKato

http://www.cab.uhttp://www.cab.u--szeged.hu/~kato/variational/szeged.hu/~kato/variational/

22ZoltanZoltanZoltan Kato: PhD Course on Kato: PhD Course on Kato: PhD Course on VariationalVariationalVariational and Level Set Methods in Image processingand Level Set Methods in Image processingand Level Set Methods in Image processing

IntroductionIntroduction

!! Proposed in their influential paper by Proposed in their influential paper by !! David David MumfordMumford

!! http://http://www.dam.brown.edu/people/mumfordwww.dam.brown.edu/people/mumford//!! JayantJayant ShahShah

!! http://http://www.math.neu.eduwww.math.neu.edu/~shah//~shah/Optimal Approximations by Piecewise Smooth Optimal Approximations by Piecewise Smooth

Functions and Associated Functions and Associated VariationalVariationalProblemsProblems. . Communications on Pure and Applied Communications on Pure and Applied Mathematics, Vol. XLII, pp 577Mathematics, Vol. XLII, pp 577--685, 1989685, 1989


Images as functionsImages as functions

!! A grayA gray--level image represents the light level image represents the light intensity recorded in a plan domain intensity recorded in a plan domain RR!! We may introduce coordinates We may introduce coordinates xx, , yy!! Let Let g(x,yg(x,y)) denote the intensity recorded at the denote the intensity recorded at the

point point ((x,yx,y) ) of of R R !! The function The function g(x,yg(x,y)) defined on the domain defined on the domain RR is is

called an image.called an image.


What kind of function is g?What kind of function is g?

!! The light reflected by the The light reflected by the surfaces surfaces SSii of various of various objects objects OOii will reach the will reach the domain domain RR in various open in various open subsets subsets RRii

!! When When OO11 appears as the appears as the background to the sides of background to the sides of OO22 then the open sets then the open sets RR11and and RR22 will have a will have a common boundary (common boundary (edgeedge))

!! One usually expects One usually expects g(x,yg(x,y))to be discontinuous along to be discontinuous along this boundarythis boundary FigureFigure fromfrom D. D. MumfordMumford & J. & J. ShahShah: : Optimal Approximations by Piecewise Smooth Functions Optimal Approximations by Piecewise Smooth Functions

and Associated and Associated VariationalVariational ProblemsProblems. . Communications on Pure and Applied Mathematics, Communications on Pure and Applied Mathematics, Vol. XLII, pp 577Vol. XLII, pp 577--685, 1989685, 1989


Other discontinuitiesOther discontinuities

!! Surface orientation of visible objects (cube)Surface orientation of visible objects (cube)!! Surface markingsSurface markings!! Illumination (shadows, uneven light)Illumination (shadows, uneven light)


PiecePiece--wise smooth gwise smooth g

!! In all cases, we expect In all cases, we expect g(x,yg(x,y)) to be pieceto be piece--wise wise smooth to the first approximation.smooth to the first approximation.

!! It is well It is well modelledmodelled by a set of smooth functions by a set of smooth functions ffiidefined on a set of disjoint regions defined on a set of disjoint regions RRii covering covering RR..

!! Problems:Problems:!! Textured objects (regions perceived homogeneous but Textured objects (regions perceived homogeneous but

lots of discontinuities in intensity)lots of discontinuities in intensity)!! SahdowsSahdows are not true discontinuitiesare not true discontinuities!! Partially transparent objectsPartially transparent objects!! NoiseNoise

!! Still widely and Still widely and succesfullysuccesfully applied model!applied model!


Segmentation problemSegmentation problem

!! Consists in computing a decomposition of Consists in computing a decomposition of the domain of the image the domain of the image g(x,yg(x,y))

1.1. gg varies varies smootlysmootly and/or slowly within and/or slowly within RRii2.2. gg varies discontinuously and/or rapidly varies discontinuously and/or rapidly

across most of the boundary across most of the boundary ΓΓ between between regions regions RRii

Un

i

iRR1=

=


Optimal approximationOptimal approximation

!! Segmentation problem may be restated as Segmentation problem may be restated as !! finding optimal approximations of a general function finding optimal approximations of a general function gg!! by pieceby piece--wise smooth functions wise smooth functions ff, whose restrictions , whose restrictions ffii

to the regions to the regions RRii are are differentiabledifferentiable!! Many other applications:Many other applications:

!! Speech recognitionSpeech recognition!! Sonar, radar or laser range dataSonar, radar or laser range data!! CAT scansCAT scans!! etcetc……


Optimal segmentationOptimal segmentation

!! MumfordMumford and Shah studied 3 and Shah studied 3 functionalsfunctionals which which measure the degree of match between an image measure the degree of match between an image g(x,yg(x,y)) and a segmentation.and a segmentation.

!! First, they defined a general functional First, they defined a general functional EE (the (the famous famous MumfordMumford--Shah functional):Shah functional):!! RRii will be disjoint connected open subsets of the planar will be disjoint connected open subsets of the planar

domain domain RR, each one with a piece, each one with a piece--wise smooth boundarywise smooth boundary!! ΓΓ will be the union of the boundaries.will be the union of the boundaries.

CC Γ==

n

i

iRR1


MumfordMumford--Shah functionalShah functional

!! Let Let ff differentiable on differentiable on ∪∪RRii and allowed to and allowed to be discontinuous across be discontinuous across ΓΓ..

!! The smaller The smaller EE, the better , the better (f, (f, ΓΓ)) segments segments gg1.1. ff approximates approximates gg2.2. f f (hence (hence gg) does not vary much on ) does not vary much on RRiiss3.3. The boundary The boundary ΓΓ be as short as possible.be as short as possible.!! Dropping any term would cause Dropping any term would cause infinf E=0E=0..

Γ+∇+−=Γ ∫∫∫∫ Γ−νµ dxdyfdxdygffE

RR

222 )(),(


Cartoon imageCartoon image

!! (f, (f, ΓΓ) ) is simply a cartoon of the original image is simply a cartoon of the original image gg..!! Basically Basically ff is a new image with edges drawn sharply.is a new image with edges drawn sharply.!! The objects are drawn The objects are drawn smootlysmootly without texturewithout texture!! (f, (f, ΓΓ) ) is essentially an idealization of is essentially an idealization of gg by the sort of by the sort of

image created by an artist.image created by an artist.!! Such cartoons are perceived correctly as representing Such cartoons are perceived correctly as representing

the same the same scanescane as as gg "" ff is a simplification of the scene is a simplification of the scene containing most of its essential features.containing most of its essential features.


Cartoon image exampleCartoon image example


Related problemsRelated problems

!! D. D. GemanGeman & S. & S. GemanGeman: Stochastic relaxation, : Stochastic relaxation, Gibbs distribution and the Bayesian restoration of Gibbs distribution and the Bayesian restoration of images. images. IEEE Trans. on PAMI 6, pp 721IEEE Trans. on PAMI 6, pp 721--741, 741, 1984.1984.!! MRF modelMRF model

!! A. Blake & A. A. Blake & A. ZissermanZisserman: Visual Reconstruction. : Visual Reconstruction. MIT Press, 198MIT Press, 19877!! Weak membrane modelWeak membrane model

!! M. M. KassKass, A. , A. WitkinWitkin & D. & D. TerzopoulosTerzopoulos: Snakes: : Snakes: Active contour Models. Active contour Models. International Journal of International Journal of Computer Vision, vol. 1, pp 321Computer Vision, vol. 1, pp 321--332, 1988332, 1988..!! Active contour modelActive contour model


Piecewise constant Piecewise constant approximationapproximation!! A special case of A special case of EE where where f=f=aaii is constant is constant

on each open set on each open set RRii..

!! Obviously, it is minimized in Obviously, it is minimized in aaii by setting by setting aaiito the mean of to the mean of gg in in RRii::

Γ+−=Γ ∫∫∑−2

22 )(),(µνµ

iR ii

dxdyagfE

)()(i

RRi Rarea

gdxdygmeana i

i

∫∫==


Piecewise constant Piecewise constant approximationapproximation

!! It can be proven that minimizing It can be proven that minimizing EE00 is well is well posed:posed:!! For any continuous For any continuous gg, there exists a , there exists a ΓΓ made up made up

of of finitfinit number of singular points joined by a number of singular points joined by a finitfinitnumber of arcs on which number of arcs on which EE00 atteinsatteins a minimum.a minimum.

!! It can also be shown that It can also be shown that EE00 is the natural is the natural limit functional of limit functional of E E as as µµ##00

Γ+−=Γ ∫∫∑ 22

0 ))(()(µν

iiR R

idxdygmeangE


Relation to the Relation to the IsingIsing modelmodel

!! If we further restrict If we further restrict ff to to !! take only values of +/take only values of +/--1, 1, !! assume that assume that gg and and ff are defined on a lattice are defined on a lattice !! then then EE00 becomes the energy of the becomes the energy of the IsingIsing

modelmodel..


Relation to the Relation to the IsingIsing modelmodel

!! ΓΓ is the path between all pairs of lattice points on is the path between all pairs of lattice points on which which f f changes sign:changes sign:

∑∑ −+−=),(),,(

2

,

2*0 )),(),(()),(),(()( 2

lkjijilkfjifjigjiffE

µν

FigureFigure fromfrom D. D. MumfordMumford & J. & J. ShahShah: : Optimal Optimal Approximations by Approximations by Piecewise Smooth Piecewise Smooth Functions and Associated Functions and Associated VariationalVariational ProblemsProblems. . Communications on Pure and Communications on Pure and Applied Mathematics, Vol. Applied Mathematics, Vol. XLII, pp 577XLII, pp 577--685, 1989685, 1989


WeakWeak stringstring

!! FittingFitting anan elasticelastic splinespline withwith possiblepossible breaksbreaks ((linelineprocessprocess oror locallocal edgesedges))!! RemoveRemove noisenoise!! ApproximateApproximate withwith smoothsmooth curvescurves!! BreaksBreaks wherewhere smoothnesssmoothness is is notnot satisfiedsatisfied

Image Image fromfrom CMU 15CMU 15--385 Computer 385 Computer VisionVision coursecourse, , SpringSpring 2002 2002 byby TaiTai Sing Sing LeeLee


EnergyEnergy ofof a a weakweak stringstring

!! αα is is thethe costcost ofof insertinginserting a a breakbreak ((locallocal edgeedgeelementelement) ) llii

!! llii maymay taketake binarybinary valuesvalues [0,1][0,1]!! llii is is turnedturned onon whenwhen ((ffi+1i+1--ffi i ))22>>αα//λλ

∑∑∑ +−−+−= +i

iiii

iii

i llffdffE αλ )1()()()( 21

2


WeakWeak membranemembrane modelmodel


∑∑∑

∑∑+++−−+

+−−+−=

+

+

jic

jijijiji

jijiji

jiji

jijiji

jijijijiji

jiVhvvff

hffdfvhfE

,,,,,

,

2,1,

,,

2,,1

,

2,,,,,

),()()1()(

)1()()(),,(

καλ


ContourContour continuitycontinuityconstraintconstraint!! VVcc(i,j(i,j)) energyenergy termterm::

!! LowLow forfor 0, 2 0, 2 lineslines!! MediumMedium forfor 3 3 lineslines!! HighHigh forfor 1, 4 1, 4 lineslines



FirstFirst variationvariation andandthethe EulerEuler equationequation!! The The extremaextrema ofof a a functionfunction f(xf(x)) areare attainedattained wherewhere

f� = 0f� = 0!! SimilarlySimilarly, , thethe extremaextrema ofof thethe functionalfunctional E(uE(u)) areare

obtainedobtained wherewhere E� = 0E� = 0..!! E = (E = (∂∂ E / E / ∂∂ u)u) is is thethe firstfirst variationvariation..!! AssumingAssuming a a commoncommon formulationformulation wherewhere u(x):[0,1]u(x):[0,1]##RR, ,

u(0)=au(0)=a andand u(1)=bu(1)=b, , thethe basicbasic problemproblem is is toto minimizeminimize::

!! The The necessarynecessary conditioncondition forfor uutoto be be anan extremumextremum ofof E(uE(u)) is is thetheEulerEuler equationequationofof a a oneone dimensionaldimensional problemproblem..

∫=1

0)',()( dxuuFuE

0'=

∂∂

−∂∂

uF

dxd

uF


EnergyEnergy minimizationminimization::GradientGradient descentdescent!! The The EulerEuler equationequation cancan be be solvedsolved

byby numericallynumerically solvingsolving (1). (1). !! CanCan be be formulatedformulated asas anan evolutionevolution

equationequation ((tt –– timetime):):!! ForFor exampleexample, , updatingupdating ff whilewhile ll is is

fixed (fixed (stringstring):):


)3(

)2()1)((2)(2

)1(

1

1

i

ti

tii

iiiiii

i

i

fEfff

lffdffE

fE

dtdf

∂∂

−=−=∆

−−−−=∂∂

∂∂

−=

+

+

µ

λ

Works Works onlyonly forfor convexconvexfunctionsfunctions!!


EnergyEnergy minimizationminimization::SimulatedSimulated AnnealingAnnealing exampleexample

!! Works Works forfor nonnon--convexconvexenergyenergy functionsfunctions!! λλ=6=6, , αα=0.04=0.04

ImagesImages fromfrom CMU 15CMU 15--385 Computer 385 Computer VisionVision coursecourse, , SpringSpring2002 2002 byby TaiTai Sing Sing LeeLee

Line process (l) Surface signal (f)


EnergyEnergy minimizationminimization::GraduatedGraduated nonnon--convexityconvexity!! ProposedProposed byby BlakeBlake & & ZissermanZisserman forfor thethe

weakweak membrane’smembrane’s energyenergy!! Basic idea:Basic idea:

1.1. ApproximateApproximate thethe originaloriginal energyenergy functionalfunctional bybya a convexconvex oneone

2.2. DoDo a a gradientgradient descentdescent onon thethe approximationapproximation3.3. GraduallyGradually morphmorph backback thethe approximationapproximation intointo

thethe originaloriginal energyenergy whilewhile repeatingrepeating stepstep 2.2.!! InIn casecase ofof a a weakweak membranemembrane energyenergy, , thethe

morphingmorphing cancan be be parametrizedparametrized!!


GraduatedGraduated nonnon--convexityconvexity

!! GNC GNC runsruns downhilldownhill ononeacheach ofof a a sequencesequence ofoffunctionsfunctions

!! ItIt reachesreaches a a globalglobaloptimum optimum assumingassuming a a seuenceseuence ofofapproximatingapproximating andandlocallylocally convexconvex funtionsfuntionsexistexist..



ConvexConvex approximationapproximation ofofthethe weakweak membranemembrane energyenergy

!! FF(0)(0)=E=E thethe originaloriginalfunctionalfunctional,,

!! FF(1)(1)=F*=F* thethe convexconvexapproximationapproximation

∑∑ −−+−=i

iip

iii

p ffgdfF )()( 1)(2)(

≥<≤−−

<=

rtifrtqifrtc

qtifttg p

||||2/)|(|

||)( 22

22

)(

ααλ

rq

cr

membranestring

cpcc

22 12

4/12/1

*,/*

λα

λ=

+=

==

ContoursContours arearedefineddefined asasthethe setset ofof ii forforwhichwhich|f|fii--ffii--11|>0|>0


ConvexConvex approximationapproximation ofofthethe weakweak membranemembrane energyenergy!! ItIt is is shownshown thatthat FF(p(p)) is is

convexconvex forfor pp≥≥11!! FF(1)(1) cancan be be minimizedminimized

usingusing gradientgradient descentdescent!! AsAs pp##00

!! IncreasedIncreased localizationlocalizationofof boundariesboundaries ((ll))

!! GradualGradual anisotropicanisotropicsmoothingsmoothing ofof surfacesurface ((ff))

!! ParametersParameters usedused forforthethe test: test: λλ=6=6, , αα=0.03=0.03


ff ll


EnergyEnergy functionalfunctional –– MRF MRF equivalenceequivalence!! FormalFormal equivalenceequivalence betweenbetween thethe twotwo

approachesapproaches. . ForFor exampleexample::

!! TakingTaking exponentialexponential (~ (~ HammersleyHammersley--CliffordClifford))!! TT ~ ~ uncertaintyuncertainty („(„temperaturetemperature”)”)

∑∑∑ +−−+−= +i

iiii

iii

i llffdffE αλ )1()()()( 21

2

4444 34444 214434421(MRF)prior smoothness

/))1()((

(Gaussian) termdata

/)(/)( 21

2

∏∏ +−−−−−− +=i

Tllff

i

TdfTfE iiiiii eee αλ


EnergyEnergy functionalfunctional –– MRF MRF equivalenceequivalence!! SizeSize ofof thethe neighborhoodneighborhood inin thethe MRF (MRF (oror

GibbsGibbs fieldfield) ) correspondscorresponds toto thethe degreedegree ofofderivativesderivatives inin thethe energyenergy functionalfunctional!! MembraneMembrane: : (f(fi+1i+1--ffii))22

!! ThinThin plateplate: : (f(fi+1i+1--2f2fii+f+fii--11))22

mumford-shah energy functionalkato/teaching/emm/01-mumford-shah.pdf · cartoon image! (f, Γ) is...

Documents