university of toronto aug. 11, 2004 learning the “epitome” of a video sequence information...
TRANSCRIPT
Aug. 11, 2004
Universityof Toronto
Learning the “Epitome”of a Video Sequence
Information Processing Workshop 2004
Vincent Cheung
Probabilistic and Statistical Inference Group
Electrical & Computer Engineering
University of Toronto
Toronto, Ontario, Canada
Advisor: Dr. Brendan J. Frey
Cheung2 / 12
Information Processing Workshop 2004
Outline
● Image epitome► What?► Why?
● Implementation computation issues► Efficiently implementing the learning algorithm
● Video epitome► Extension to videos► Video inpainting
Cheung3 / 12
Information Processing Workshop 2004Im
age
Lea
rnin
gV
ide
o
Image Epitome
● Jojic, N., Frey, B., & Kannan, A. (2003). Epitomic analysis of appearance and shape. In Proc. IEEE ICCV.
● Miniature, condensed version of the image
● Accurately accounts for the interesting properties of the image
● Applications► object detection► texture segmentation► image retrieval► compression
Cheung4 / 12
Information Processing Workshop 2004Im
age
Lea
rnin
gV
ide
o
Image Epitome Examples
Cheung5 / 12
Imag
eL
earn
ing
Vid
eo
Information Processing Workshop 2004
Epitome
Input image
Training Set
SamplePatches
UnsupervisedLearning
Learning the Image Epitome
e
Z1 Z2 ZM
…
TMT2T1Bayesiannetwork
e – epitomeTk – mappingZk – image patch
Cheung6 / 12
Imag
eL
earn
ing
Vid
eo
Information Processing Workshop 2004
Xe(i,j)
KK
N
N
Cumsum
2
N
N
Xe(i,j)
KK
N
N
Cumsum
2
N
N
Shifted Cumulative Sum Algorithm
(2, 2), (i+1, j+1)
(1, 2), (i, j+1)
(2, 1), (i+1, j)
– col 1+ col (P+1)
– row 1+ row (P+1)
–
+
+ + pixel (1,1)+ pixel (P+1, P+1)
(1, 1), (i, j)
(2, 2), (i+1, j+1)(2, 2), (i+1, j+1)
(1, 2), (i, j+1)(1, 2), (i, j+1)
(2, 1), (i+1, j)(2, 1), (i+1, j)
– col 1+ col (P+1)
– row 1+ row (P+1)
–
+
+ + pixel (1,1)+ pixel (P+1, P+1)
(1, 1), (i, j)(1, 1), (i, j)
+-
-+
Cheung7 / 12
Imag
eL
earn
ing
Vid
eo
Information Processing Workshop 2004
X e
P
P
P
P
Ta
Tß
X e
P
P
P
P
Ta
Tß
Collecting Sufficient Statistics
Cheung8 / 12
Imag
eL
earn
ing
Vid
eo
Information Processing Workshop 2004
Extending Epitomes to Videos
● Desire a miniature, condensed version of a video sequence
● Want it to accurately account for the interesting properties of the video
● Applications► optic flow► segmentation► texture transfer► layer separation► compression► noise reduction► inpainting
Cheung9 / 12
Imag
eL
earn
ing
Vid
eo
Information Processing Workshop 2004
Input Video
Frame 1Frame 2
Frame 3
Training Set
SamplePatches
Video Epitome
UnsupervisedLearning
Video Epitome
Cheung10 / 12
Imag
eL
earn
ing
Vid
eo
Information Processing Workshop 2004
Video Epitome Example
Temporally Compressed
Spatially Compressed
Cheung11 / 12
Imag
eL
earn
ing
Vid
eo
Information Processing Workshop 2004
Video Inpainting (1)
● Fill in missing portions of a video► damaged films► occluding objects
● Reconstruct the missing pixels from the video epitome
Cheung12 / 12
Imag
eL
earn
ing
Vid
eo
Information Processing Workshop 2004
Video Inpainting (2)
Cheung13 / 12
Information Processing Workshop 2004
Conclusion
● Improved the efficiency of learning image epitomes
● Extended the concept of epitomes to video sequences
● Demonstrated the ability of video epitomes to model motion patterns through video inpainting