mean-field theory and its applications in computer vision5 1

Mean-Field Theory and Its Applications In Computer Vision5

1

Global Co-occurrence Terms

2

• Encourages global consistency and co-occurrence of objects

Without cooc

With co-occurrence

Global Co-occurrence Terms

3

• Defined on subset of labels• Associates a cost with each possible subset

Properties of cost function

4

Non-decreasing

0.2 0.2 0.2

3.0 3.0 3.0

5.0

Properties of cost function

5

We represent our cost as second order cost function defined on binary vector:

Complexity

6

• Complexity: O(NL2)

• Two relaxed (approximation) of this form• Complexity: O(NL+L2)

Our model• Represent 2nd order cost by binary latent variables • Unary cost per latent variable

7

1l

2l

3l

label level variable node (0/1)

Our model• Represent 2nd order cost by binary latent variables • Pairwise cost between latent variable

8

1l

2l

3l

Global Co-occurrence Cost• Two approximation to include into fully connected CRF

9

Global Co-occurrence Terms• First model

10

1l2l

3l

Global Co-occurrence Terms• Model

11

1l2l

3l

Global Co-occurrence Terms• Constraints (lets take one set of connections)

12

1l2l

3l

1l2l

3l

If latent variable is on, atleast one of image variable take that label

If latent variable is off, no image variable take that label

Global Co-occurrence Terms• Pay a cost K for violating first constraint

13

1l2l

3l

Global Co-occurrence Terms• Pay a cost K for violating second constrait

14

1l2l

3l

Global Co-occurrence Terms• Cost for first model:

15

1l

3l

Global Co-occurrence Terms• Second model

• Each latent node is connected to the variable node

16

1l2l

3l

Global Co-occurrence Terms• Constraints (lets take one set of connections)

17

1l2l

3l

1l2l

3l

If latent variable is on, atleast one of image variable take that label

If latent variable is off, no image variable take that label

Global Co-occurrence Terms• Pay a cost K for violating the constraint

18

1l2l

3l

Global Co-occurrence Terms• Cost for second model:

19

1l2l

3l

Global Co-occurrence Terms• Expectation evaluation for variable Yl• Case 1: Y_l takes label 0

20

1l2l

3l

Global Co-occurrence Terms• Expectation evaluation for variable Yl• Case 1: Y_l takes label 0

21

1l2l

3l

Global Co-occurrence Terms• Expectation evaluation for variable Yl• Case 1: Y_l takes label 0

22

1l2l

3l

Global Co-occurrence Terms• Expectation evaluation for variable Yl• Case 1: Y_l takes label 1

23

1l

2l

3l

Global Co-occurrence Terms• Expectation evaluation for variable Yl• Case 1: Y_l takes label 1

24

1l

2l

3l

Global Co-occurrence Terms• Expectation evaluation for variable Yl

25

Global Co-occurrence Terms• Latent variable updates:

26

Global Co-occurrence Terms• Latent variable updates:

27

Global Co-occurrence Terms

Pay a cost K if variable takes a label l and corresponding latent variable takes label 0

28

1l2l

3l

ComplexityExpectation updates for latent variable Y_l

29

ComplexityExpectation updates for latent variable Y_l

30

Overall complexity:

Does not increase original complexity:

PascalVOC-10 dataset

31Qualitative analysis: observe an improvement over other comparative methods

PascalVOC-10 dataset

32

Algorithm Time (s) Overall Av. Recall Av. I/U

AHCRF+Cooc 36 81.43 38.01 30.09

Dense CRF 0.67 71.43 34.53 28.40

Dense + Potts 4.35 79.87 40.71 30.18

Dense + Potts + Cooc

4.4 80.44 43.08 32.35

Observe an improvement of almost 2.3% improvement Almost 8-9 times faster than alpha-expansion based method

Mean-field Vs. Graph-cuts

33

• Measure I/U score on PascalVOC-10 segmentation • Increase standard deviation for mean-field• Increase window size for graph-cuts method

• Both achieve almost similar accuracy

Window sizes

34

Algorithm Model Time (s) Av. I/U

Alpha-exp (n=10) Pairwise 326.17 28.59

Mean-field pairwise 0.67 28.64

Alpha-exp (n=3) Pairwise + Potts 56.8 29.6

Mean-field Pairwise + Potts 4.35 30.11

Alpha-exp (n=1) Pairwise + Potts + Cooc

103.94 30.45

Mean-field Pairwise + Potts + Cooc

4.4 32.17

• Comparison on matched energy

Impact of adding more complex costs and increasing window size

PascalVOC-10 dataset

35

Algorithm bkg plane Cycle bird Boat

AHCRF+Cooc

82.5 43.2 4.9 17.4 27.1

Dense + Potts + Cooc

82.9 44.6 15.8 18.9 26.3

Algorithm bottle Bus car cat Chair

AHCRF+Cooc

31.3 49.4 51.0 29.3 7.1

Dense + Potts + Cooc

31.7 48.9 55.2 33.3 7.9

Per class Quantitative results

PascalVOC-10 dataset

36

Algorithm Cow Dtb dog horse Mbike

AHCRF+Cooc

26.7 8.3 17.0 24.0 27.1

Dense + Potts + Cooc

27.0 16.1 16.8 23.4 43.8

Algorithm

pson Plant sheep sofa train TV Av

AHCRF+Cooc

41.9 21.8 25.2 16.4 43.8 43.4 30.9

Dense + Potts + Cooc

38.4 21.1 30.9 15.5 44.0 36.8 32.35

Per class Quantitative results

Mean-field Vs. Graph-cuts

37

• Measure I/U score on PascalVOC-10 segmentation • Increase standard deviation for mean-field• Increase window size for graph-cuts method

•Time complexity very high, making infeasible to work with large neighbourhood system