markov networks in computer vision - university at buffalosrihari/cse674/chap4/4.5-mrf-cv.pdf ·...
TRANSCRIPT
![Page 1: Markov Networks in Computer Vision - University at Buffalosrihari/CSE674/Chap4/4.5-MRF-CV.pdf · –Many formulated as a Markov network •Multiclass segmentation –Each variable](https://reader034.vdocument.in/reader034/viewer/2022050307/5f6f460870480735a05d0319/html5/thumbnails/1.jpg)
Probabilistic Graphical Models Srihari
1
Markov Networks in Computer Vision
Sargur [email protected]
![Page 2: Markov Networks in Computer Vision - University at Buffalosrihari/CSE674/Chap4/4.5-MRF-CV.pdf · –Many formulated as a Markov network •Multiclass segmentation –Each variable](https://reader034.vdocument.in/reader034/viewer/2022050307/5f6f460870480735a05d0319/html5/thumbnails/2.jpg)
Probabilistic Graphical Models Srihari
Markov Nets (MRFs) in Computer Vision
1. Image Denoising
2. Image segmentation
3. Stereo reconstruction
4. Object recognition2
![Page 3: Markov Networks in Computer Vision - University at Buffalosrihari/CSE674/Chap4/4.5-MRF-CV.pdf · –Many formulated as a Markov network •Multiclass segmentation –Each variable](https://reader034.vdocument.in/reader034/viewer/2022050307/5f6f460870480735a05d0319/html5/thumbnails/3.jpg)
Probabilistic Graphical Models Srihari
3
1. Binary Image de-noising• Image-to-image mapping• Observed noisy image
– Binary pixel values yi ∈ {-1,+1}, i=1,..,D• Unknown noise-free image
– Binary pixel values xi ∈ {-1,+1}, i=1,..,D• Noisy image assumed to randomly
flip sign of pixels with small probability
![Page 4: Markov Networks in Computer Vision - University at Buffalosrihari/CSE674/Chap4/4.5-MRF-CV.pdf · –Many formulated as a Markov network •Multiclass segmentation –Each variable](https://reader034.vdocument.in/reader034/viewer/2022050307/5f6f460870480735a05d0319/html5/thumbnails/4.jpg)
Probabilistic Graphical Models Srihari
4
Markov Random Field Model
• Known– Strong correlation between xi and yi– Neighbor pixels xi and xj are strongly
correlated• Prior knowledge captured using MRF
xi= unknown noise-free pixelyi= known noisy pixel
![Page 5: Markov Networks in Computer Vision - University at Buffalosrihari/CSE674/Chap4/4.5-MRF-CV.pdf · –Many formulated as a Markov network •Multiclass segmentation –Each variable](https://reader034.vdocument.in/reader034/viewer/2022050307/5f6f460870480735a05d0319/html5/thumbnails/5.jpg)
Probabilistic Graphical Models Srihari
5
Energy Functions
• Graph has two types of cliques– {xi,yi} expresses correlation between variables
• Choose simple energy function –η xiyi• Lower energy (higher probability) when xi and yi have
same sign
– {xi,xj} which are neighboring pixels• Choose β xixj
![Page 6: Markov Networks in Computer Vision - University at Buffalosrihari/CSE674/Chap4/4.5-MRF-CV.pdf · –Many formulated as a Markov network •Multiclass segmentation –Each variable](https://reader034.vdocument.in/reader034/viewer/2022050307/5f6f460870480735a05d0319/html5/thumbnails/6.jpg)
Probabilistic Graphical Models Srihari
6
Potential Function
• Complete energy function of model
– The hxi term biases towards pixel values that have one particular sign
– Which defines a joint distribution over x and y given by
{ , }
(x, y) i i j i ii i j i
E h x x x x yb h= - -å å å
1(x, y) exp{ (x, y)}p E
Z= -
![Page 7: Markov Networks in Computer Vision - University at Buffalosrihari/CSE674/Chap4/4.5-MRF-CV.pdf · –Many formulated as a Markov network •Multiclass segmentation –Each variable](https://reader034.vdocument.in/reader034/viewer/2022050307/5f6f460870480735a05d0319/html5/thumbnails/7.jpg)
Probabilistic Graphical Models Srihari
Parameters for Image denoising• Node potential ϕ(Xi) for each pixel Xi
• penalize large discrepancies from observed pixel value yi• Edge potential
– Encode continuity between adjacent pixel values• Penalize cases where inferred value of Xi is too far from
inferred value of neighbor Xj• Important not to over-penalize true edge disparities
(edges between objects or regions)– Leads to oversmoothing of image
• Solution: Bound the penalty using a truncated norm– ε(xi,xj) = min(c||xi-xj||p,distmax) for p ε {1,2}
7
![Page 8: Markov Networks in Computer Vision - University at Buffalosrihari/CSE674/Chap4/4.5-MRF-CV.pdf · –Many formulated as a Markov network •Multiclass segmentation –Each variable](https://reader034.vdocument.in/reader034/viewer/2022050307/5f6f460870480735a05d0319/html5/thumbnails/8.jpg)
Probabilistic Graphical Models Srihari
8
De-noising problem statement
• We fix y to observed pixels in the noisy image• p(x|y) is a conditional distribution over all noise-
free images– Called Ising model in statistical physics
• We wish to find an image x that has a high probability
![Page 9: Markov Networks in Computer Vision - University at Buffalosrihari/CSE674/Chap4/4.5-MRF-CV.pdf · –Many formulated as a Markov network •Multiclass segmentation –Each variable](https://reader034.vdocument.in/reader034/viewer/2022050307/5f6f460870480735a05d0319/html5/thumbnails/9.jpg)
Probabilistic Graphical Models Srihari
• Gradient ascent– Set xi=yi for all i– Take one node xj at a time
• evaluate total energy for states xi=+1 and xi=-1• keeping all other node variable fixed
– Set xj to value for which energy is lower• This is a local computation
• which can be done efficiently
– Repeat for all pixels until • a stopping criterion is met
– Nodes updated systematically • by raster scan or randomly
• Finds a local maximum (which need not be global)
• Algorithm is called Iterated Conditional Modes (ICM)
De-noising algorithm
![Page 10: Markov Networks in Computer Vision - University at Buffalosrihari/CSE674/Chap4/4.5-MRF-CV.pdf · –Many formulated as a Markov network •Multiclass segmentation –Each variable](https://reader034.vdocument.in/reader034/viewer/2022050307/5f6f460870480735a05d0319/html5/thumbnails/10.jpg)
Probabilistic Graphical Models Srihari
10
Image Restoration Results• Parametersβ=1.0 (for xixj)η=2.1 (for xiyi)h=0
Result of ICM Global maximum obtained byGraph Cut algorithm
Noise Free imageNoisy image where 10%of pixels are corrupted
![Page 11: Markov Networks in Computer Vision - University at Buffalosrihari/CSE674/Chap4/4.5-MRF-CV.pdf · –Many formulated as a Markov network •Multiclass segmentation –Each variable](https://reader034.vdocument.in/reader034/viewer/2022050307/5f6f460870480735a05d0319/html5/thumbnails/11.jpg)
Probabilistic Graphical Models Srihari
2. Image Segmentation Task• Partition the image pixels into regions
corresponding to distinct parts of scene• Different variants of segmentation task
– Many formulated as a Markov network• Multiclass segmentation
– Each variable Xi has a domain {1,..,K} pixels– Value of Xi represents region assignment for pixel i
, e.g., grass, water, sky, car– Classifying each pixel is expensive
• Oversegment image into superpixels (coherent regions) and classify each superpixels
– All pixels within region are assigned same value11
![Page 12: Markov Networks in Computer Vision - University at Buffalosrihari/CSE674/Chap4/4.5-MRF-CV.pdf · –Many formulated as a Markov network •Multiclass segmentation –Each variable](https://reader034.vdocument.in/reader034/viewer/2022050307/5f6f460870480735a05d0319/html5/thumbnails/12.jpg)
Probabilistic Graphical Models Srihari
Importance of Modeling Correlations between superpixels
12
car
road
building
cow
grass
(a) (b) (c) (d)Original image Oversegmentedimage-superpixelsEach superpixel isa random variable
Classification using node potentials alone-each superpixel classified independently
Segmentation usingpairwise Markov Network encoding interactionsbetween adjacentsuperpixels
![Page 13: Markov Networks in Computer Vision - University at Buffalosrihari/CSE674/Chap4/4.5-MRF-CV.pdf · –Many formulated as a Markov network •Multiclass segmentation –Each variable](https://reader034.vdocument.in/reader034/viewer/2022050307/5f6f460870480735a05d0319/html5/thumbnails/13.jpg)
Probabilistic Graphical Models Srihari
Graph from Superpixels• A node for each superpixel• Edge between nodes if regions are adjacent• This defines a distribution in terms of this graph
13
![Page 14: Markov Networks in Computer Vision - University at Buffalosrihari/CSE674/Chap4/4.5-MRF-CV.pdf · –Many formulated as a Markov network •Multiclass segmentation –Each variable](https://reader034.vdocument.in/reader034/viewer/2022050307/5f6f460870480735a05d0319/html5/thumbnails/14.jpg)
Probabilistic Graphical Models Srihari
Log-linear model with features
• Xi: Pixels or Super-pixels• Joint probability distribution over an image• Log-linear model:
F={fi}: Features between variables A,B ε Di:
14
P(X
1,..X
n;θ) = 1
Z(θ)exp θ
ifi(D
i)
i=1
k
∑⎧⎨⎩⎪
⎫⎬⎭⎪
fa0b0(a,b) = I a = a0{ }I b = b0{ }
Z θ( ) = exp θ
ifiξ( )
i∑⎧⎨
⎩
⎫⎬⎭ξ
∑
![Page 15: Markov Networks in Computer Vision - University at Buffalosrihari/CSE674/Chap4/4.5-MRF-CV.pdf · –Many formulated as a Markov network •Multiclass segmentation –Each variable](https://reader034.vdocument.in/reader034/viewer/2022050307/5f6f460870480735a05d0319/html5/thumbnails/15.jpg)
Probabilistic Graphical Models Srihari
Network Structure• In most applications structure is pairwise
– Variables correspond to pixels– Edges (factors) correspond to
• interactions between adjacent pixels in grid on image
– Each interior pixel has exactly four neighbors• Value space of variables and exact form of
factors depend on task
• Usually formulated:• Factors in terms of energies
– Negative log potentials– Values represent penalties:
» lower value = higher probability15
car
road
building
cow
grass
(a) (b) (c) (d)
![Page 16: Markov Networks in Computer Vision - University at Buffalosrihari/CSE674/Chap4/4.5-MRF-CV.pdf · –Many formulated as a Markov network •Multiclass segmentation –Each variable](https://reader034.vdocument.in/reader034/viewer/2022050307/5f6f460870480735a05d0319/html5/thumbnails/16.jpg)
Probabilistic Graphical Models Srihari
Model
• Edge potential between every pair of superpixels Xi, Xj
– Encodes a contiguity preference– With a penalty λ whenever Xi≠Xj
– Model can be improved by making penalty depend on presence of an image gradient between pixels
– Even better model:• Non default values for class pairs• Tigers adjacent to vegetation, water below vegetation
16
![Page 17: Markov Networks in Computer Vision - University at Buffalosrihari/CSE674/Chap4/4.5-MRF-CV.pdf · –Many formulated as a Markov network •Multiclass segmentation –Each variable](https://reader034.vdocument.in/reader034/viewer/2022050307/5f6f460870480735a05d0319/html5/thumbnails/17.jpg)
Probabilistic Graphical Models Srihari
Features for Image Segmentation• Features extracted for each superpixel
– Statistics over color, texture, location• Features either clustered or input to local classifiers to reduce
dimensionality• Node potential is a function of these features
– Factors depend upon pixels in image– Each image defines a different probability
distribution over segment labels for pixels or superpixels
• Model in effect is a Conditional Random Field
17
![Page 18: Markov Networks in Computer Vision - University at Buffalosrihari/CSE674/Chap4/4.5-MRF-CV.pdf · –Many formulated as a Markov network •Multiclass segmentation –Each variable](https://reader034.vdocument.in/reader034/viewer/2022050307/5f6f460870480735a05d0319/html5/thumbnails/18.jpg)
Probabilistic Graphical Models Srihari
Metric MRFs• Class of MRFs used for labeling• Graph of nodes X1,..Xn related by set of edges E• Wish to assign to each Xi a label in space V={v1,..vk}
• Each node, taken in isolation, has its preference among possible labels
• Also need to impose a soft ”smoothness” constraint that neighboring nodes should take similar values
18
![Page 19: Markov Networks in Computer Vision - University at Buffalosrihari/CSE674/Chap4/4.5-MRF-CV.pdf · –Many formulated as a Markov network •Multiclass segmentation –Each variable](https://reader034.vdocument.in/reader034/viewer/2022050307/5f6f460870480735a05d0319/html5/thumbnails/19.jpg)
Probabilistic Graphical Models Srihari
Encoding preferences
• Node preferences are node potentials in pairwise MRF
• Smoothness preferences are edge potentials• Traditional to encode these models in negative
log-space– using energy functions• With MAP objective we can ignore the partition
function
19
![Page 20: Markov Networks in Computer Vision - University at Buffalosrihari/CSE674/Chap4/4.5-MRF-CV.pdf · –Many formulated as a Markov network •Multiclass segmentation –Each variable](https://reader034.vdocument.in/reader034/viewer/2022050307/5f6f460870480735a05d0319/html5/thumbnails/20.jpg)
Probabilistic Graphical Models Srihari
Energy Function
• Energy function
• Goal is to minimize the energy
20
E(x
1,..x
n) = ε
i(x
i)
i∑ + ε
ij(x
ix
j)
{i,j}∑
arg
x1,..xn
min E(x1,..x
n)
![Page 21: Markov Networks in Computer Vision - University at Buffalosrihari/CSE674/Chap4/4.5-MRF-CV.pdf · –Many formulated as a Markov network •Multiclass segmentation –Each variable](https://reader034.vdocument.in/reader034/viewer/2022050307/5f6f460870480735a05d0319/html5/thumbnails/21.jpg)
Probabilistic Graphical Models Srihari
Smoothness definition
• Slight variant of Ising model
• Obtain lowest possible pairwise energy (0) when neighboring nodes Xi,Xj take the same value and a higher energy λi,j when they do not
21
εij(x
1,x
j) =
0λ
i,j
⎧⎨⎪
⎩⎪
xi= x
j
xi≠ x
j
![Page 22: Markov Networks in Computer Vision - University at Buffalosrihari/CSE674/Chap4/4.5-MRF-CV.pdf · –Many formulated as a Markov network •Multiclass segmentation –Each variable](https://reader034.vdocument.in/reader034/viewer/2022050307/5f6f460870480735a05d0319/html5/thumbnails/22.jpg)
Probabilistic Graphical Models Srihari
Generalizations
• Potts model extends it to more than two labels• Distance function on labels
– Prefer neighboring nodes to have labels smaller distance apart
22
![Page 23: Markov Networks in Computer Vision - University at Buffalosrihari/CSE674/Chap4/4.5-MRF-CV.pdf · –Many formulated as a Markov network •Multiclass segmentation –Each variable](https://reader034.vdocument.in/reader034/viewer/2022050307/5f6f460870480735a05d0319/html5/thumbnails/23.jpg)
Probabilistic Graphical Models Srihari
Metric definition
• A function µ: V � Và [0,∞) is a metric if it satisfies– Reflexivity: µ(vk,vl)=0 if and only if k=l– Symmetry:µ(vk,vl)=µ(vl,vk);– Triangle Inequality:µ(vk,vl)+µ(vl,vm) ≥ µ(vk,vm)
• µ is a semi-metric if it satisfies first two• Metric MRF is defined by defining
εi,j(vk,vj) = µ(vk,vl)• A common metric: ε(xi,xj) = min(c||xi-xj||,distmax)
![Page 24: Markov Networks in Computer Vision - University at Buffalosrihari/CSE674/Chap4/4.5-MRF-CV.pdf · –Many formulated as a Markov network •Multiclass segmentation –Each variable](https://reader034.vdocument.in/reader034/viewer/2022050307/5f6f460870480735a05d0319/html5/thumbnails/24.jpg)
Probabilistic Graphical Models Srihari
3. Stereo Reconstruction• Reconstruct depth disparity of each
image pixel – Variables represent discretized version
of depth dimension (more finely for discretized for close to camera and coarse when away)
• Node potential: a computer vision technique to estimate depth disparity
• Edge potential: a truncated metric– Inversely proportional to image gradient
between pixels– Smaller penalty to large gradient suggesting
occlusion24