8/6/2019 Lecture6 Clustering and Seg p2 Cs223b

1/67

Lecture 6 -Fei-Fei Li

Lecture6:Clusteringand

Segmentation Part2

ProfessorFeiFei Li

StanfordVisionLab

11Jan111

8/6/2019 Lecture6 Clustering and Seg p2 Cs223b

2/67

Lecture 6 -Fei-Fei Li

Recap:GestaltTheory

Gestalt:wholeorgroup

Wholeisgreaterthansumofitsparts

Relationshipsamongpartscanyieldnewproperties/features

Psychologistsidentifiedseriesoffactorsthatpredisposesetof

elementstobegrouped(byhumanvisualsystem)

Untersuchungen zur Lehre von der Gestalt,

Psychologische Forschung, Vol. 4, pp. 301-350, 1923http://psy.ed.asu.edu/~classics/Wertheimer/Forms/forms.htm

I st and at t he window and see a house, t rees, sky.Theoret ical ly I might say t here were 327 br ightnessesand nuances of colour. Do I have "327"? No. I have sky, house,and t rees.

Max Wertheimer(1880-1943)

11Jan112

8/6/2019 Lecture6 Clustering and Seg p2 Cs223b

3/67

Lecture 6 -Fei-Fei Li

Recap:GestaltFactors

Thesefactorsmakeintuitivesense,butareverydifficulttotranslateintoalgorithms.

11Jan113

8/6/2019 Lecture6 Clustering and Seg p2 Cs223b

4/67

Lecture 6 -Fei-Fei Li

Recap:ImageSegmentation

Goal:identifygroupsofpixelsthatgotogether

11Jan114

8/6/2019 Lecture6 Clustering and Seg p2 Cs223b

5/67

Lecture 6 -Fei-Fei Li

Recap:KMeansClustering

Basicidea:randomlyinitializethekclustercenters,and

iteratebetweenthetwostepswejustsaw.

1. Randomlyinitializetheclustercenters,c1,...,cK2. Givenclustercenters,determinepointsineachcluster

Foreachpointp,findtheclosestci. Putpintoclusteri

3. Givenpointsineachcluster,solveforci

Setci tobethemeanofpointsinclusteri4. Ifci havechanged,repeatStep2

Properties

Will

always

converge

to

some solution Canbealocalminimum

Doesnotalwaysfindtheglobalminimumofobjectivefunction:

11Jan115

8/6/2019 Lecture6 Clustering and Seg p2 Cs223b

6/67

Lecture 6 -Fei-Fei Li

Recap:ExpectationMaximization(EM)

Goal

Findblobparametersthatmaximizethelikelihoodfunction:

Approach:

1. Estep: givencurrentguessofblobs,computeownershipofeachpoint

2. Mstep: givenownershipprobabilities,updateblobstomaximizelikelihood

function

3. Repeatuntilconvergence

11Jan116

8/6/2019 Lecture6 Clustering and Seg p2 Cs223b

7/67

Lecture 6 -Fei-Fei Li

Recap:MeanShiftAlgorithm

IterativeModeSearch1. Initializerandomseed,andwindowW

2. Calculatecenterofgravity(themean)ofW:

3. Shiftthesearchwindowtothemean

4. RepeatStep2untilconvergence

11Jan117

8/6/2019 Lecture6 Clustering and Seg p2 Cs223b

8/67

Lecture 6 -Fei-Fei Li

Recap:MeanShiftClustering

Cluster:alldatapointsintheattractionbasinofamode

Attractionbasin:theregionforwhichalltrajectoriesleadto

the

same

mode

11Jan118

8/6/2019 Lecture6 Clustering and Seg p2 Cs223b

9/67

Lecture 6 -Fei-Fei Li

Recap:MeanShiftSegmentation

Findfeatures(color,gradients,texture,etc)

Initializewindowsatindividualpixellocations

Performmeanshiftforeachwindowuntilconvergence

Mergewindowsthatendupnearthesamepeakormode

11Jan119

8/6/2019 Lecture6 Clustering and Seg p2 Cs223b

10/67

Lecture 6 -Fei-Fei Li

BacktotheImageSegmentationProblem

Goal:identifygroupsofpixelsthatgotogether

Uptonow,wehavefocusedonwaystogrouppixelsintoimagesegmentsbasedontheirappearance

Segmentationasclustering. Wealsowanttoenforceregionconstraints.

Spatialconsistency

Smoothborders

11Jan1110

8/6/2019 Lecture6 Clustering and Seg p2 Cs223b

11/67

Lecture 6 -Fei-Fei Li

Whatwewilllearntoday? Graphtheoreticsegmentation

NormalizedCuts

Usingtexturefeatures

SegmentationasEnergyMinimization

MarkovRandomFieldsGraphcutsforimagesegmentation

stmincut algorithm

ExtensiontononbinarycaseApplications

(Midtermmaterials)

11Jan1111

8/6/2019 Lecture6 Clustering and Seg p2 Cs223b

12/67

Lecture 6 -Fei-Fei Li

Whatwewilllearntoday?

Graphtheoreticsegmentation

NormalizedCutsUsingtexturefeatures

SegmentationasEnergyMinimization

MarkovRandomFields

Graphcutsforimagesegmentation

stmincut algorithm

Extensiontononbinarycase

Applications

11Jan1112

8/6/2019 Lecture6 Clustering and Seg p2 Cs223b

13/67

Lecture 6 -Fei-Fei Li

ImagesasGraphs

Fullyconnectedgraph Node(vertex)foreverypixel

Linkbetweeneverypairofpixels,(p,q)

Affinityweightwpq foreachlink(edge) wpq measuressimilarity Similarityisinverselyproportionaltodifference

(incolorandposition)

q

p

wpq

w

Slide credit: Steve Seitz

11Jan1113

8/6/2019 Lecture6 Clustering and Seg p2 Cs223b

14/67

Lecture 6 -Fei-Fei Li

SegmentationbyGraphCuts

BreakGraphintoSegments Deletelinksthatcrossbetweensegments

Easiesttobreaklinksthathavelowsimilarity(lowweight) Similarpixelsshouldbeinthesamesegments

Dissimilarpixelsshouldbeindifferentsegments

w

A B C

Slide credit: Steve Seitz

11Jan1114

8/6/2019 Lecture6 Clustering and Seg p2 Cs223b

15/67

Lecture 6 -Fei-Fei Li

MeasuringAffinity

Distance

Intensity

Color

Texture

2 212( , ) exp daff x y x y

2 212( , ) exp ( ) ( )daff x y I x I y

(some suitable color space distance)

221

2( , ) exp ( ), ( )

d

aff x y dist c x c y

Source:Forsyth&

Ponce

221

2( , ) exp ( ) ( )

d

aff x y f x f y

(vectors of filter outputs)

11Jan1115

8/6/2019 Lecture6 Clustering and Seg p2 Cs223b

16/67

Lecture 6 -Fei-Fei Li

ScaleAffectsAffinity

Small :grouponlynearbypoints

Large :groupfarawaypoints

Slide credit: Svetlana Lazebnik Small Medium Large 11Jan1116

8/6/2019 Lecture6 Clustering and Seg p2 Cs223b

17/67

Lecture 6 -Fei-Fei Li

GraphCut:usingEigenvalues

Extractasinglegoodcluster

Whereelementshavehighaffinityvalueswitheachother

8/6/2019 Lecture6 Clustering and Seg p2 Cs223b

18/67

Lecture 6 -Fei-Fei Li

points

matrix

eigenvector

GraphCut:usingEigenvalues

8/6/2019 Lecture6 Clustering and Seg p2 Cs223b

19/67

Lecture 6 -Fei-Fei Li

Extractasinglegoodcluster

Extractweightsforasetofclusters

GraphCut:usingEigenvalues

8/6/2019 Lecture6 Clustering and Seg p2 Cs223b

20/67

Lecture 6 -Fei-Fei Li

GraphCut:usingEigenvalues

8/6/2019 Lecture6 Clustering and Seg p2 Cs223b

21/67

Lecture 6 -Fei-Fei Li

8/6/2019 Lecture6 Clustering and Seg p2 Cs223b

22/67

Lecture 6 -Fei-Fei Li

GraphCut

Setofedgeswhoseremovalmakesagraphdisconnected

CostofacutSumofweightsofcutedges:

AgraphcutgivesusasegmentationWhatisagoodgraphcutandhowdowefindone?

Slide credit: Steve Seitz

A B

BqAp

qpwBAcut,

,),(

11Jan1122

8/6/2019 Lecture6 Clustering and Seg p2 Cs223b

23/67

Lecture 6 -Fei-Fei Li

GraphCut

Image Source: Forsyth & Ponce

Here, the cut is nicely

defined by the block-diagonal

structure of the affinity matrix.

How can t his be general ized?

11Jan1123

8/6/2019 Lecture6 Clustering and Seg p2 Cs223b

24/67

Lecture 6 -Fei-Fei Li

MinimumCut

Wecandosegmentationbyfindingtheminimumcutinagraph

aminimumcut ofagraphisacutwhosecutset hasthesmallestnumber

ofelements(unweighted case)orsmallestsumofweightspossible.

Efficientalgorithmsexistfordoingthis

Drawback:

Weightofcutproportionaltonumberofedgesinthecut

Minimumcuttendstocutoffverysmall,isolatedcomponents

IdealCut

Cutswith

lesser

weightthanthe

idealcut

Slidecredit:Khurram

Hassan-Sha

fique

11Jan1124

8/6/2019 Lecture6 Clustering and Seg p2 Cs223b

25/67

Lecture 6 -Fei-Fei Li

NormalizedCut(NCut)

Aminimumcutpenalizeslargesegments

Thiscanbefixedbynormalizingforsizeofsegments

Thenormalizedcutcostis:

TheexactsolutionisNPhardbutanapproximationcanbe

computedbysolvingageneralizedeigenvalueproblem.

assoc(A,V)=sumofweightsofalledgesinVthattouchA

),(

),(

),(

),(

VBassoc

BAcut

VAassoc

BAcut

J.ShiandJ.Malik.Normalizedcutsandimagesegmentation. PAMI2000

11Jan1125

8/6/2019 Lecture6 Clustering and Seg p2 Cs223b

26/67

Lecture 6 -Fei-Fei Li

InterpretationasaDynamicalSystem

Treat

the

links

as

springs

and

shake

the

system Elasticityproportionaltocost Vibrationmodescorrespondtosegments

Cancomputethesebysolvingageneralizedeigenvectorproblem

Slidecredit:Ste

veSeitz

11Jan1126

8/6/2019 Lecture6 Clustering and Seg p2 Cs223b

27/67

Lecture 6 -Fei-Fei Li

NCuts asaGeneralizedEigenvectorProblem

Definitions

RewritingNormalizedCutinmatrixform:

,: ( , ) ;

: ( , ) ( , );

: {1, 1} , ( ) 1 .

the affinity matrix,

the diag. matrix,

a vector in

i j

j

N

W W i j w

D D i i W i j

x x i i A

Slide credit: Jitendra Malik

0

(A,B) (A,B)(A,B)

(A,V) (B,V)

( , )(1 ) ( )(1 ) (1 ) ( )(1 );

1 1 (1 )1 1 ( , )

...

i

T Tx

T T

i

cut cut NCut

assoc assoc

D i ix D W x x D W xk

k D k D D i i

11Jan1127

8/6/2019 Lecture6 Clustering and Seg p2 Cs223b

28/67

Lecture 6 -Fei-Fei Li

SomeMoreMath

Slidecredit:JitendraMalik

11Jan1128

8/6/2019 Lecture6 Clustering and Seg p2 Cs223b

29/67

Lecture 6 -Fei-Fei Li

NCuts asaGeneralizedEigenvalueProblem

Aftersimplification,weget

ThisisaRayleighQuotient Solutiongivenbythegeneralized eigenvalueproblem

Solvedbyconvertingtostandardeigenvalueproblem

Subtleties

Optimalsolutionissecondsmallesteigenvector Givescontinuousresultmustconvertintodiscretevaluesofy

( )( , ) , with {1, }, 1 0.

T

T

iT

y D W y NCut A B y b y D

y Dy

Slide credit: Alyosha Efros

This is hard,

as y is discrete!

Relaxation:

continuous y.

,1 1 1

2 2 2 D (D W)D z z where z D y

11Jan1129

8/6/2019 Lecture6 Clustering and Seg p2 Cs223b

30/67

Lecture 6 -Fei-Fei Li

NCuts Example

Smallest eigenvectors

Image source: Shi & MalikNCuts segments

11Jan1130

8/6/2019 Lecture6 Clustering and Seg p2 Cs223b

31/67

Lecture 6 -Fei-Fei Li

Discretization

Problem:eigenvectorstakeoncontinuousvalues Howtochoosethesplittingpointtobinarize theimage?

Possibleproceduresa) Pickaconstantvalue(0,or0.5).

b) Pickthemedianvalueassplittingpoint.

c) LookforthesplittingpointthathastheminimumNCutvalue:1. Choosen possiblesplittingpoints.

2. ComputeNCutvalue.

3. Pickminimum.

Image Eigenvector NCut scores

11Jan1131

8/6/2019 Lecture6 Clustering and Seg p2 Cs223b

32/67

Lecture 6 -Fei-Fei Li

NCuts:OverallProcedure

1. ConstructaweightedgraphG=(V,E)fromanimage.

2. Connecteachpairofpixels,andassigngraphedgeweights,

3. Solve forthesmallestfeweigenvectors.This

yields

a

continuous

solution.4. Thresholdeigenvectorstogetadiscretecut

Thisiswheretheapproximationismade(werenotsolvingNP).

5. Recursively

subdivide

if

NCut value

is

below

a

pre

specified

value.

( )W y Dy

( , ) Prob. that and belong to the same region.W i j i j

Slidecredit:JitendraMalik

NCuts Matlab codeavailableat

http://www.cis.upenn.edu/~jshi/software/

11Jan1132

C l I S i i h NC

8/6/2019 Lecture6 Clustering and Seg p2 Cs223b

33/67

Lecture 6 -Fei-Fei Li

ColorImageSegmentationwithNCuts

ImageSource:Shi&Malik

11Jan1133

8/6/2019 Lecture6 Clustering and Seg p2 Cs223b

34/67

Lecture 6 -Fei-Fei Li

UsingTextureFeaturesforSegmentation

Texturedescriptorisvectoroffilterbankoutputs

J.Malik,S.Belongie,T.LeungandJ.Shi."ContourandTextureAnalysisforImageSegmentation".

IJCV43(1),727,2001.Slidecredit:Sv

etlanaLazebnik

11Jan1134

8/6/2019 Lecture6 Clustering and Seg p2 Cs223b

35/67

Lecture 6 -Fei-Fei Li

UsingTextureFeaturesforSegmentation

Texturedescriptoris

vectoroffilterbankoutputs.

Textons arefoundby

clustering.

Slide credit: Svetlana Lazebnik

11Jan1135

8/6/2019 Lecture6 Clustering and Seg p2 Cs223b

36/67

Lecture 6 -Fei-Fei Li

UsingTextureFeaturesforSegmentation

Texturedescriptorisvectoroffilterbankoutputs.

Textons arefoundbyclustering.

Affinitiesaregivenbysimilaritiesoftextonhistogramsoverwindowsgivenbythelocalscaleofthetexture.

Slide credit: Svetlana Lazebnik

11Jan1136

8/6/2019 Lecture6 Clustering and Seg p2 Cs223b

37/67

Lecture 6 -Fei-Fei Li

ResultswithColor&Texture

11Jan1137

8/6/2019 Lecture6 Clustering and Seg p2 Cs223b

38/67

Lecture 6 -Fei-Fei Li

Summary:NormalizedCuts

Pros:Genericframework,flexibletochoiceoffunctionthat

computesweights(affinities)betweennodes

Doesnotrequireanymodelofthedatadistribution

Cons:

Time

and

memory

complexity

can

be

high Dense,highlyconnectedgraphs manyaffinitycomputations Solvingeigenvalueproblemforeachcut

Preferenceforbalancedpartitions Ifaregionisuniform,NCutswillfindthe

modesofvibrationoftheimagedimensions

Slide credit: Kristen Grauman

11Jan1138

8/6/2019 Lecture6 Clustering and Seg p2 Cs223b

39/67

Lecture 6 -Fei-Fei Li

Whatwewilllearntoday? Graphtheoreticsegmentation

NormalizedCutsUsingtexturefeatures

SegmentationasEnergyMinimization

MarkovRandomFieldsGraphcutsforimagesegmentation

stmincut algorithm

Extensiontononbinarycase

Applications

11Jan1139

8/6/2019 Lecture6 Clustering and Seg p2 Cs223b

40/67

Lecture 6 -Fei-Fei Li

MarkovRandomFields

Allowrichprobabilisticmodelsforimages

Butbuiltinalocal,modularway

Learnlocaleffects,getglobaleffectsout

Slidecredit:William

Freeman

Observed evidence

Hidden true states

Neighborhood relations

11Jan1140

8/6/2019 Lecture6 Clustering and Seg p2 Cs223b

41/67

Lecture 6 -Fei-Fei Li

MRFNodesasPixels

Reconstruction

from MRF modeling

pixel neighborhood

statistics

Degraded imageOriginal image

Slidecredit:BastianLeibe

11Jan1141

8/6/2019 Lecture6 Clustering and Seg p2 Cs223b

42/67

Lecture 6 -Fei-Fei Li

MRFNodesasPatches

Image

Scene

Image patches

Scene patches

( , )i ix y

( , )i jx x

Slidecredit:William

Freeman

11Jan1142

8/6/2019 Lecture6 Clustering and Seg p2 Cs223b

43/67

Lecture 6 -Fei-Fei Li

Network

Joint

Probability

,( , ) ( , ) ( , )i i i j

i i j x y x y x x Scene

Image

Image-scene

compatibilityfunction

Scene-scene

compatibilityfunction

Neighboring

scene nodesLocal

observations

Slidecredit:William

Freeman

11Jan1143

8/6/2019 Lecture6 Clustering and Seg p2 Cs223b

44/67

Lecture 6 -Fei-Fei Li

EnergyFormulation Jointprobability

Takingthelogp(.)turnsthisintoanEnergyoptimizationproblem

Thisissimilartofreeenergyproblemsinstatisticalmechanics

(spinglasstheory).WethereforedrawtheanalogyandcallEanenergy function.

andarecalledpotentials.

,

( , ) ( , ) ( , )i i i ji i j

P x y x y x x

,

,

log ( , ) log ( , ) log ( , )

( , ) ( , ) ( , )

i i i j

i i j

i i i j

i i j

x y x y x x

E x y x y x x

Slidecredit:BastianLeibe

11Jan1144

l

8/6/2019 Lecture6 Clustering and Seg p2 Cs223b

45/67

Lecture 6 -Fei-Fei Li

EnergyFormulation

Energyfunction

Singlenodepotentials

Encodelocalinformationaboutthegivenpixel/patch Howlikelyisapixel/patchtobelongtoacertainclass

(e.g.foreground/background)?

Pairwisepotentials

Encodeneighborhoodinformation Howdifferentisapixel/patchslabelfromthatofitsneighbor?(e.g.

basedonintensity/color/texturedifference,edges)

Pairwise

potentials

Single-node

potentials

( , )i iy

( , )i jx

,

( , ) ( , ) ( , )i i i j

i i j

E x y x y x x

Slidecredit:BastianLeibe

11Jan1145

E Mi i i i

8/6/2019 Lecture6 Clustering and Seg p2 Cs223b

46/67

Lecture 6 -Fei-Fei Li

EnergyMinimization

Goal: InfertheoptimallabelingoftheMRF.

Manyinferencealgorithmsareavailable,e.g. Gibbssampling,simulatedannealing Iteratedconditionalmodes(ICM) Variationalmethods

Beliefpropagation Graphcuts

Recently,GraphCutshavebecomeapopulartool

Onlysuitableforacertainclassofenergyfunctions Butthesolutioncanbeobtainedveryfastfortypicalvisionproblems(~1MPixel/sec).

( , )i iy

( , )i jx

Slidecredit:BastianLeibe

11Jan1146

8/6/2019 Lecture6 Clustering and Seg p2 Cs223b

47/67

Lecture 6 -Fei-Fei Li

What

we

will

learn

today? Graphtheoreticsegmentation

NormalizedCuts

Usingtexturefeatures

Extension:Multilevelsegmentation

SegmentationasEnergyMinimizationMarkovRandomFields

Graphcutsforimagesegmentation

stmincut algorithm

ExtensiontononbinarycaseApplications

11Jan1147

Graph Cuts for Optimal Boundary Detection

8/6/2019 Lecture6 Clustering and Seg p2 Cs223b

48/67

Lecture 6 -Fei-Fei Li

GraphCutsforOptimalBoundaryDetection

Idea:convertMRFintosourcesinkgraph

n-links

s

t a cuthard

constraint

hard

constraint

Minimumcostcutcanbe

computedinpolynomialtime

(maxflow/mincutalgorithms)

2

2

exp

pq

pq

Iw

pqI

Slide credit: Yuri Boykov

11Jan1148

Si l E l f E

8/6/2019 Lecture6 Clustering and Seg p2 Cs223b

49/67

Lecture 6 -Fei-Fei Li

SimpleExampleofEnergy

Npq

qppq

p

pp LLwLDLE )()()(

},{ tsLp

tlinks nlinks

BoundarytermRegionalterm

(binaryobjectsegmentation)

Slide credit: Yuri Boykov

22

exp

pq

pq

Iw

pqI

s

t a cut

)(sDp

)(tDp

11Jan1149

Addi R i l P ti

8/6/2019 Lecture6 Clustering and Seg p2 Cs223b

50/67

Lecture 6 -Fei-Fei Li

AddingRegionalProperties

pqw

n-links

s

ta cut)(tDp

)(sDp

NOTE:hardconstrainsarenotrequired,ingeneral.

Regional biasexample

Suppose aregiven

expectedintensities

ofobjectandbackground

ts II and 22 2/||||exp)( spp IIsD

22

2/||||exp)( t

pp IItD

Slide credit: Yuri Boykov

11Jan1150

Adding Regional Properties

8/6/2019 Lecture6 Clustering and Seg p2 Cs223b

51/67

Lecture 6 -Fei-Fei Li

AddingRegionalProperties

pqw

n-links

s

ta cut)(tDp

)(sDp

22 2/||||exp)( spp IIsD

22

2/||||exp)( t

pp IItD

EMstyleoptimization

expected intensities of

object and background

can be re-estimated

ts

II and

Slide credit: Yuri Boykov

11Jan1151

Adding Regional Properties

8/6/2019 Lecture6 Clustering and Seg p2 Cs223b

52/67

Lecture 6 -Fei-Fei Li

AddingRegionalProperties

Moregenerally,regionalbiascanbebasedonany

intensitymodelsofobjectandbackground

a cut ( ) logPr( | ) p p p p D L I L

givenobjectandbackgroundintensityhistograms

)(sDp

)(tDps

t

)|Pr( sIp

)|Pr( tIp

pI

Slide credit: Yuri Boykov

11Jan1152

How to Set the Potentials? Some Examples

8/6/2019 Lecture6 Clustering and Seg p2 Cs223b

53/67

Lecture 6 -Fei-Fei Li

HowtoSetthePotentials?SomeExamples

Colorpotentials e.g.modeledwithaMixtureofGaussians

Edgepotentials e.g.acontrastsensitivePottsmodel

where

Parameters, needtobelearned,too!

[Shotton & Winn, ECCV06]

( , , ( ); ) ( ) ( )T

i j ij ij i jx x g y g y x x

2

2 i javg y y 2

( ) i jy y

ij g y e

( , ; ) log ( , ) ( | ) ( ; , )i i i i i k k k

x y x k P k x N y y

11Jan1153

8/6/2019 Lecture6 Clustering and Seg p2 Cs223b

54/67

Lecture 6 -Fei-Fei Li

What

we

will

learn

today? Graphtheoreticsegmentation

NormalizedCuts

Usingtexturefeatures

Extension:Multilevelsegmentation

SegmentationasEnergyMinimizationMarkovRandomFields

Graphcutsforimagesegmentation

stmincut algorithm

ExtensiontononbinarycaseApplications

11Jan1154

G hC t A li ti G bC t

8/6/2019 Lecture6 Clustering and Seg p2 Cs223b

55/67

Lecture 6 -Fei-Fei Li

GraphCutApplications:GrabCut

Usersegmentationcues

Additionalsegmentation

cues

InteractiveImageSegmentation[Boykov &Jolly,ICCV01] Roughregioncuessufficient

Segmentationboundarycanbeextractedfromedges

Procedure Usermarksforegroundandbackgroundregionswithabrush.

Thisisusedtocreateaninitialsegmentationwhichcanthenbecorrectedbyadditionalbrushstrokes.

Slide credit: Matthieu Bray

11Jan1155

GrabCut: Data Model

8/6/2019 Lecture6 Clustering and Seg p2 Cs223b

56/67

Lecture 6 -Fei-Fei Li

Obtainedfrominteractiveuserinput Usermarksforegroundandbackgroundregionswithabrush

Alternatively,usercanspecifyaboundingbox

GrabCut:DataModel

Globaloptimumofthe

energy

Background

color

Foreground

color

Slide credit: Carsten Rother

11Jan1156

GrabCut: Coherence Model

8/6/2019 Lecture6 Clustering and Seg p2 Cs223b

57/67

Lecture 6 -Fei-Fei Li

GrabCut:CoherenceModel

Anobjectisacoherentsetofpixels:

Howtochoose ?

Slide credit: Carsten Rother

Error (%) over training set:

25

2

( , )

( , ) e m ny y

n m

m n C

x y x x

11Jan1157

Iterated Graph Cuts

8/6/2019 Lecture6 Clustering and Seg p2 Cs223b

58/67

Lecture 6 -Fei-Fei Li

IteratedGraphCuts

Energyafter

eachiteration

Result

Foreground &

Background

Background G

RForeground

Background G

R

1 2 3 4

Color model

(MixtureofGaussians)

Slide credit: Carsten Rother

11Jan1158

GrabCut: live demo

8/6/2019 Lecture6 Clustering and Seg p2 Cs223b

59/67

Lecture 6 -Fei-Fei Li

GrabCut:livedemo

IncludedinMSOffice2010(letstryit)

Reported

results

11Jan1159

GrabCut: live demo

8/6/2019 Lecture6 Clustering and Seg p2 Cs223b

60/67

Lecture 6 -Fei-Fei Li

GrabCut:livedemo

IncludedinMSOffice2010(letstryit)

Reportedresults

11Jan1160

GrabCut: live demo

8/6/2019 Lecture6 Clustering and Seg p2 Cs223b

61/67

Lecture 6 -Fei-Fei Li

GrabCut:livedemo

IncludedinMSOffice2010(letstryit)

11Jan1161

GraphCut ImageSynthesisResults

8/6/2019 Lecture6 Clustering and Seg p2 Cs223b

62/67

Lecture 6 -Fei-Fei Li 62

Source:Vivek

Kwatra

11Jan11

Application:TextureSynthesisintheMedia

8/6/2019 Lecture6 Clustering and Seg p2 Cs223b

63/67

Lecture 6 -Fei-Fei Li

Currently,stilldonemanually

Slide credit: Kristen Grauman

11Jan1163

ImprovingEfficiencyofSegmentation

8/6/2019 Lecture6 Clustering and Seg p2 Cs223b

64/67

Lecture 6 -Fei-Fei Li

p g y g

Problem:Imagescontainmanypixels Evenwithefficientgraphcuts,anMRF

formulationhastoomanynodesforinteractiveresults.

Efficiencytrick:Superpixels Grouptogethersimilarlooking

pixelsforefficiencyoffurther

processing. Cheap,localoversegmentation

Importanttoensurethatsuperpixelsdonotcrossboundaries

Severaldifferentapproachespossible Superpixelcodeavailablehere http://www.cs.sfu.ca/~mori/research/superpixels/

Image source: Greg Mori

11Jan1164

Superpixels forPreSegmentation

8/6/2019 Lecture6 Clustering and Seg p2 Cs223b

65/67

Lecture 6 -Fei-Fei Li

SpeedupGraph structure

11Jan1165

Summary: Graph Cuts Segmentation

8/6/2019 Lecture6 Clustering and Seg p2 Cs223b

66/67

Lecture 6 -Fei-Fei Li

Summary:GraphCutsSegmentation

ProsPowerfultechnique,basedonprobabilisticmodel(MRF).

Applicableforawiderangeofproblems.Veryefficientalgorithmsavailableforvisionproblems.

Becomingadefactostandardformanysegmentationtasks.

Cons/IssuesGraphcutscanonlysolvealimitedclassofmodels

Submodularenergyfunctions

CancaptureonlypartoftheexpressivenessofMRFsOnlyapproximatealgorithmsavailableformultilabelcase

6611Jan11

8/6/2019 Lecture6 Clustering and Seg p2 Cs223b

67/67

Lecture 6 -Fei-Fei Li

What

we

have

learned

today?

11Jan1167

Graphtheoreticsegmentation

NormalizedCuts

Usingtexturefeatures

SegmentationasEnergyMinimization

MarkovRandomFields

Graphcutsforimagesegmentation

stmincut algorithm

Extensiontononbinarycase

Applications

(Midtermmaterials)