advanced topics in computer vision spring 2006 video segmentation tal kramer, shai bagon video...
Post on 20-Jan-2016
221 views
TRANSCRIPT
Video SegmentationTal Kramer, Shai Bagon
Advanced Topics in Computer Vision Spring 2006
Video Segmentation
April 30th, 2006
Video SegmentationTal Kramer, Shai Bagon
Advanced Topics in Computer Vision Spring 2006
Introduction
• Recognition and Segmentation• Min Cut Max Flow• Single Image Methods
– GrabCut– Lazy Snapping– …
Video SegmentationTal Kramer, Shai Bagon
Advanced Topics in Computer Vision Spring 2006
Lazy Snapping
• Interactive User Interface
Video SegmentationTal Kramer, Shai Bagon
Advanced Topics in Computer Vision Spring 2006
Lazy Snapping
• Energy minimization
Video SegmentationTal Kramer, Shai Bagon
Advanced Topics in Computer Vision Spring 2006
Lazy Snapping
• Energy minimization
Video SegmentationTal Kramer, Shai Bagon
Advanced Topics in Computer Vision Spring 2006
Lazy Snapping
• Boundary overriding
Video SegmentationTal Kramer, Shai Bagon
Advanced Topics in Computer Vision Spring 2006
Lazy Snapping
• Boundary overriding
Video SegmentationTal Kramer, Shai Bagon
Advanced Topics in Computer Vision Spring 2006
Motivation
• Obvious Next Step• Video Cut & Paste• Video Manipulation and Editing
Video SegmentationTal Kramer, Shai Bagon
Advanced Topics in Computer Vision Spring 2006
Introduction
• Frame by Frame– Time Consuming and Tedious
• Error With Simple Methods– Fast motions– Deforming silhouettes – Changing topologies
Video SegmentationTal Kramer, Shai Bagon
Advanced Topics in Computer Vision Spring 2006
Introduction
• Two Papers– Video Object Cut
and Paste– Video Cutout
Video SegmentationTal Kramer, Shai Bagon
Advanced Topics in Computer Vision Spring 2006
Video Object Cut and Paste
Yin Li, Jian Sun, Heung-Yeung Shum
Video SegmentationTal Kramer, Shai Bagon
Advanced Topics in Computer Vision Spring 2006
Overview
Video SegmentationTal Kramer, Shai Bagon
Advanced Topics in Computer Vision Spring 2006
Pre-segmentation
• Pre-Segmentation to All Frames
Video SegmentationTal Kramer, Shai Bagon
Advanced Topics in Computer Vision Spring 2006
Key Frames
• Picking Key Frames.
Video SegmentationTal Kramer, Shai Bagon
Advanced Topics in Computer Vision Spring 2006
Key Frames
• User Fore/Background Segmentation
Video SegmentationTal Kramer, Shai Bagon
Advanced Topics in Computer Vision Spring 2006
3D Graph Cut Segmentation
• 3D Graph – G=(V,A)
• Labeling– Foreground = 1 – Background = 0
• Volume Between Successive Key Frames
Video SegmentationTal Kramer, Shai Bagon
Advanced Topics in Computer Vision Spring 2006
3D Graph Construction
• 2 Kinds of Arcs:– AI – Intra
Frames (BLUE)
– AT – Inter Frame (RED)
Video SegmentationTal Kramer, Shai Bagon
Advanced Topics in Computer Vision Spring 2006
3D Graph Construction
• Minimizing Equation:
• E1 – Global Color Models
• E2 – Penalizing Spatially
• E3 – Penalizing Temporally
Video SegmentationTal Kramer, Shai Bagon
Advanced Topics in Computer Vision Spring 2006
Likelihood Energy
• GMMs Decide Label• In Key Frames:
Video SegmentationTal Kramer, Shai Bagon
Advanced Topics in Computer Vision Spring 2006
GMM• Gaussian Mixture
Model
• Distance is Measured By:
Video SegmentationTal Kramer, Shai Bagon
Advanced Topics in Computer Vision Spring 2006
Prior Energies
• E2, E3 Are the Same
• Distance of Adjacent Regions.
• β = (2 E (||cr – cs||2 ))-1
Video SegmentationTal Kramer, Shai Bagon
Advanced Topics in Computer Vision Spring 2006
Prior Energies
• λ1 = 24
• λ2 = 12
Video SegmentationTal Kramer, Shai Bagon
Advanced Topics in Computer Vision Spring 2006
3D Graph Segmentation
Video SegmentationTal Kramer, Shai Bagon
Advanced Topics in Computer Vision Spring 2006
Errors
• Global Colors• Similarity to
Background• Thin Areas
Video SegmentationTal Kramer, Shai Bagon
Advanced Topics in Computer Vision Spring 2006
Error Overriding
• Video tubes• Manual corrections
Video SegmentationTal Kramer, Shai Bagon
Advanced Topics in Computer Vision Spring 2006
Video Tubes
• Local Color Models• Put Two Windows• Tracking Algorithm• Key Frames to
Solve
W1
WT
Video SegmentationTal Kramer, Shai Bagon
Advanced Topics in Computer Vision Spring 2006
Fixing graph cut segmentation
• Minimizing:
Video SegmentationTal Kramer, Shai Bagon
Advanced Topics in Computer Vision Spring 2006
Overriding Brush
• Fixing Boundary Manually
Video SegmentationTal Kramer, Shai Bagon
Advanced Topics in Computer Vision Spring 2006
Manual error overriding
Video SegmentationTal Kramer, Shai Bagon
Advanced Topics in Computer Vision Spring 2006
• Soften hard segmentation
Matting
Video SegmentationTal Kramer, Shai Bagon
Advanced Topics in Computer Vision Spring 2006
Coherent Matting
• Boundary is not 0/1• Prevent Bolting Pixels• Smooth Paste
Video SegmentationTal Kramer, Shai Bagon
Advanced Topics in Computer Vision Spring 2006
Coherent Matting
Video SegmentationTal Kramer, Shai Bagon
Advanced Topics in Computer Vision Spring 2006
Example
Video SegmentationTal Kramer, Shai Bagon
Advanced Topics in Computer Vision Spring 2006
Example
Video SegmentationTal Kramer, Shai Bagon
Advanced Topics in Computer Vision Spring 2006
Example
Video SegmentationTal Kramer, Shai Bagon
Advanced Topics in Computer Vision Spring 2006
Example
Video SegmentationTal Kramer, Shai Bagon
Advanced Topics in Computer Vision Spring 2006
Video CutOut
J. WANG, P. BHAT, A. COLBURN, M. AGRAWALA, M. COHEN. Interactive Video Cutout. ACM Trans. on Graphics
(Proc. of SIGGAPH2005), 2005
Video SegmentationTal Kramer, Shai Bagon
Advanced Topics in Computer Vision Spring 2006
Video Cutout introduction
What’s new?• Different user interface• 3D graph formation• Refinement mechanism
Video SegmentationTal Kramer, Shai Bagon
Advanced Topics in Computer Vision Spring 2006
System overview
Video SegmentationTal Kramer, Shai Bagon
Advanced Topics in Computer Vision Spring 2006
3D Graph construction
• Hierarchical graph nodes:1. Frame by frame mean shift
segmentation2. Aggregating segments across
frames
Video SegmentationTal Kramer, Shai Bagon
Advanced Topics in Computer Vision Spring 2006
• Pixel 26-neighborhood induce links• Lower level links induce higher level
link
3D Graph construction
Video SegmentationTal Kramer, Shai Bagon
Advanced Topics in Computer Vision Spring 2006
• Stroking foreground and background over the 3D spatio-temporal volume
• Not segmenting any frame
User Interface
Video SegmentationTal Kramer, Shai Bagon
Advanced Topics in Computer Vision Spring 2006
• Graph construction– User input propagates upward – Min cut uses yellow nodes
3D Min cut/Max flow
Video SegmentationTal Kramer, Shai Bagon
Advanced Topics in Computer Vision Spring 2006
3D Min cut/Max flow
• Weights / Energy function– The energy function:
– Data term: color similarity to F/B model– Link term: cut likelihood
1,
, , , , , ,i i i i j i ji nghbrs i j
E D x c L x x c c
Video SegmentationTal Kramer, Shai Bagon
Advanced Topics in Computer Vision Spring 2006
3D Min cut/Max flow
• Terms in energy function
Graph Cut Energy function
L LinkDF ForegroundDB Background
DB,L Pix. history DB,G Color DF,G Color LL Local temporal LG Gradient
Video SegmentationTal Kramer, Shai Bagon
Advanced Topics in Computer Vision Spring 2006
Data weight
• User input generates color model (GMM)
• Infinite weight preserves marked pixels
• Data weight = abiding to F/B color model
Graph Cut Energy function
L LinkDF ForegroundDB Background
DB,L Pix. history DB,G Color DF,G Color LL Local temporal LG Gradient
Video SegmentationTal Kramer, Shai Bagon
Advanced Topics in Computer Vision Spring 2006
Data weight
White – high probability ForegroundBlack – Low probability Foreground
Graph Cut Energy function
L LinkDF ForegroundDB Background
DB,L Pix. history DB,G Color DF,G Color LL Local temporal LG Gradient
Video SegmentationTal Kramer, Shai Bagon
Advanced Topics in Computer Vision Spring 2006
Data weight
White – high probability BackgroundBlack – Low probability Background
Graph Cut Energy function
L LinkDF ForegroundDB Background
DB,L Pix. history DB,G Color DF,G Color LL Local temporal LG Gradient
Video SegmentationTal Kramer, Shai Bagon
Advanced Topics in Computer Vision Spring 2006
• Strong gradients segment border
• Link cost encourage cut at edges
Link weight Graph Cut Energy function
L LinkDF ForegroundDB Background
DB,L Pix. history DB,G Color DF,G Color LL Local temporal LG Gradient
Video SegmentationTal Kramer, Shai Bagon
Advanced Topics in Computer Vision Spring 2006
Link weight
White – low cut probabilityBlack – high cut probability
Graph Cut Energy function
L LinkDF ForegroundDB Background
DB,L Pix. history DB,G Color DF,G Color LL Local temporal LG Gradient
Video SegmentationTal Kramer, Shai Bagon
Advanced Topics in Computer Vision Spring 2006
• Pixel span: (xo, yo, t)t>0
Data weight Graph Cut Energy function
L LinkDF ForegroundDB Background
DB,L Pix. history DB,G Color DF,G Color LL Local temporal LG Gradient
Video SegmentationTal Kramer, Shai Bagon
Advanced Topics in Computer Vision Spring 2006
Data weight Graph Cut Energy function
L LinkDF ForegroundDB Background
DB,L Pix. history DB,G Color DF,G Color LL Local temporal LG Gradient
• Local background model• Assuming camera is stabilized, video is
registered• Extracting “clean plate” • Weight per pixel span
Video SegmentationTal Kramer, Shai Bagon
Advanced Topics in Computer Vision Spring 2006
• d(zi) = minimum color distance {“clean plate”, B marked pixel}.
• “Clean plate” cannot be always trusted
• Weight:
Data weight Graph Cut Energy function
L LinkDF ForegroundDB Background
DB,L Pix. history DB,G Color DF,G Color LL Local temporal LG Gradient
1 100
2 1N
NumFramese
Video SegmentationTal Kramer, Shai Bagon
Advanced Topics in Computer Vision Spring 2006
Data weight
White – high probability BackgroundBlack – Low probability Background
Graph Cut Energy function
L LinkDF ForegroundDB Background
DB,L Pix. history DB,G Color DF,G Color LL Local temporal LG Gradient
Video SegmentationTal Kramer, Shai Bagon
Advanced Topics in Computer Vision Spring 2006
• Link span: links between two adjacent pixel spans
Link weight Graph Cut Energy function
L LinkDF ForegroundDB Background
DB,L Pix. history DB,G Color DF,G Color LL Local temporal LG Gradient
Video SegmentationTal Kramer, Shai Bagon
Advanced Topics in Computer Vision Spring 2006
• Strong edges exists within segment
• Small change over time• Local temporal link cost penalize
strong temporal gradient
Link weight Graph Cut Energy function
L LinkDF ForegroundDB Background
DB,L Pix. history DB,G Color DF,G Color LL Local temporal LG Gradient
Video SegmentationTal Kramer, Shai Bagon
Advanced Topics in Computer Vision Spring 2006
Link weight Graph Cut Energy function
L LinkDF ForegroundDB Background
DB,L Pix. history DB,G Color DF,G Color LL Local temporal LG Gradient
Video SegmentationTal Kramer, Shai Bagon
Advanced Topics in Computer Vision Spring 2006
Link weight
White – low cut probabilityBlack – high cut probability
Graph Cut Energy function
L LinkDF ForegroundDB Background
DB,L Pix. history DB,G Color DF,G Color LL Local temporal LG Gradient
Video SegmentationTal Kramer, Shai Bagon
Advanced Topics in Computer Vision Spring 2006
• Energy function
Graph Cut Energy function
L LinkDF ForegroundDB Background
DB,L Pix. history DB,G Color DF,G Color LL Local temporal LG Gradient
3D Min cut/Max flow
1,
, , , , , ,i i i i j i ji nghbrs i j
E D x c L x x c c
λ2
λ1
λ3
Video SegmentationTal Kramer, Shai Bagon
Advanced Topics in Computer Vision Spring 2006
Iterative process
• The user refines the cut• Adds F/B strokes• Graph is re-computed
Nth iterationN+1th iteration
Video SegmentationTal Kramer, Shai Bagon
Advanced Topics in Computer Vision Spring 2006
Post Processing
Video SegmentationTal Kramer, Shai Bagon
Advanced Topics in Computer Vision Spring 2006
Post processing
• Binary cut obtained• Edges need refinement
Video SegmentationTal Kramer, Shai Bagon
Advanced Topics in Computer Vision Spring 2006
• A pixel-level min cut around edges• Color model obtained form
boundary• Uniform edge cost = small
cut
Refinement
Video SegmentationTal Kramer, Shai Bagon
Advanced Topics in Computer Vision Spring 2006
Video SegmentationTal Kramer, Shai Bagon
Advanced Topics in Computer Vision Spring 2006
Matting
• Soften hard segmentation• Evaluate α Channel
Video SegmentationTal Kramer, Shai Bagon
Advanced Topics in Computer Vision Spring 2006
Matting
• Refinement fixed boundary locally
• Global 3D mesh• α Channel along
mesh normals
Video SegmentationTal Kramer, Shai Bagon
Advanced Topics in Computer Vision Spring 2006
Results
Video SegmentationTal Kramer, Shai Bagon
Advanced Topics in Computer Vision Spring 2006
Results
Video SegmentationTal Kramer, Shai Bagon
Advanced Topics in Computer Vision Spring 2006
Performance
Video SegmentationTal Kramer, Shai Bagon
Advanced Topics in Computer Vision Spring 2006
• Pros– Online 3D min cut – Spatio temporal smooth cut
• Cons– Does not handle shadows– Ignore motion blur (LPF to avoid
temporal aliasing)– Cannot separate translucent objects
Summary
Video SegmentationTal Kramer, Shai Bagon
Advanced Topics in Computer Vision Spring 2006
ComparisonVideo object cut
and pasteVideo Cutout
Features
•Graph nodes
•UI
2D segmentation
Frame base interface
3D segmentation in 2 stages
spatial-temporal manipulation
Performance
•Preprocessing•Artist time•Post processing•Total
4-5 min25 min
?30 min
25 min10 sec per Min cut
30 min60 min
Video SegmentationTal Kramer, Shai Bagon
Advanced Topics in Computer Vision Spring 2006
Questions?
Video SegmentationTal Kramer, Shai Bagon
Advanced Topics in Computer Vision Spring 2006
• The total energy:
• Foreground and background terms:
• Background terms:
• Link terms:
Energy function
1,
, , , , , ,i i i i j i ji nghbrs i j
E D x c L x x c c
;B Fi i
B F B F
D DD x B D x F
D D D D
2 , 2 ,1B i B L B GD x B D D
3 31i j G LL x x L L 3 0.3
1 100
2 1N
NumFramese