![Page 1: Unwrap Mosaics: A new representation for video …Unwrap Mosaics: A new representation for video editing Alex Rav-Acha, Pushmeet Kohli, Carsten Rother, Andrew Fitzgibbon Microsoft](https://reader033.vdocument.in/reader033/viewer/2022043002/5f7e2b9ba81c64132f4e6d8a/html5/thumbnails/1.jpg)
Unwrap Mosaics: A new
representation for video editing
Alex Rav-Acha, Pushmeet Kohli, Carsten Rother, Andrew Fitzgibbon
Microsoft Research
SIGGRAPH 2008http://research.microsoft.com/en-us/um/cambridge/projects/visionimagevideoediting/unwrap/
![Page 2: Unwrap Mosaics: A new representation for video …Unwrap Mosaics: A new representation for video editing Alex Rav-Acha, Pushmeet Kohli, Carsten Rother, Andrew Fitzgibbon Microsoft](https://reader033.vdocument.in/reader033/viewer/2022043002/5f7e2b9ba81c64132f4e6d8a/html5/thumbnails/2.jpg)
Abstract
• New representation for common editing tasks
• Representation has the power of full 3D
surface reconstruction
• This representation is “Unwrap Mosaics”• This representation is “Unwrap Mosaics”
• Editing can be performed and re-composited
into the original sequence
![Page 3: Unwrap Mosaics: A new representation for video …Unwrap Mosaics: A new representation for video editing Alex Rav-Acha, Pushmeet Kohli, Carsten Rother, Andrew Fitzgibbon Microsoft](https://reader033.vdocument.in/reader033/viewer/2022043002/5f7e2b9ba81c64132f4e6d8a/html5/thumbnails/3.jpg)
What is a Mosaic ?
A group of pictures used to represent a broader/wider
image.
http://graphics.cs.cmu.edu/courses/15-463/2005_fall/www/hw/proj3/proj3_files/image006.jpg
![Page 4: Unwrap Mosaics: A new representation for video …Unwrap Mosaics: A new representation for video editing Alex Rav-Acha, Pushmeet Kohli, Carsten Rother, Andrew Fitzgibbon Microsoft](https://reader033.vdocument.in/reader033/viewer/2022043002/5f7e2b9ba81c64132f4e6d8a/html5/thumbnails/4.jpg)
What is a Mosaic ?
4 or 5 images used to form this mosaic
http://graphics.cs.cmu.edu/courses/15-463/2005_fall/www/hw/proj3/proj3_files/image006.jpg
![Page 5: Unwrap Mosaics: A new representation for video …Unwrap Mosaics: A new representation for video editing Alex Rav-Acha, Pushmeet Kohli, Carsten Rother, Andrew Fitzgibbon Microsoft](https://reader033.vdocument.in/reader033/viewer/2022043002/5f7e2b9ba81c64132f4e6d8a/html5/thumbnails/5.jpg)
Mosaic Example
http://www.infotech.oulu.fi/Annual/2005/pics/mvg_01.jpg
![Page 6: Unwrap Mosaics: A new representation for video …Unwrap Mosaics: A new representation for video editing Alex Rav-Acha, Pushmeet Kohli, Carsten Rother, Andrew Fitzgibbon Microsoft](https://reader033.vdocument.in/reader033/viewer/2022043002/5f7e2b9ba81c64132f4e6d8a/html5/thumbnails/6.jpg)
Mosaic Example
http://www.vision.deis.unibo.it/MosPerf/MosPerf-Files/MosaicRM.jpg
![Page 7: Unwrap Mosaics: A new representation for video …Unwrap Mosaics: A new representation for video editing Alex Rav-Acha, Pushmeet Kohli, Carsten Rother, Andrew Fitzgibbon Microsoft](https://reader033.vdocument.in/reader033/viewer/2022043002/5f7e2b9ba81c64132f4e6d8a/html5/thumbnails/7.jpg)
Goal
To recover the object’s texture map, rather than its 3D shape
THE CONCEPT
Video represented as several unwrap mosaics
• One per object
• One per background• One per background
![Page 8: Unwrap Mosaics: A new representation for video …Unwrap Mosaics: A new representation for video editing Alex Rav-Acha, Pushmeet Kohli, Carsten Rother, Andrew Fitzgibbon Microsoft](https://reader033.vdocument.in/reader033/viewer/2022043002/5f7e2b9ba81c64132f4e6d8a/html5/thumbnails/8.jpg)
Goal
It is possible in principle to recover the 3D shape.
For many editing tasks this is not necessary: edits can be
performed on the mosaic itself and re-rendered without ever
converting to a 3D representation.
![Page 9: Unwrap Mosaics: A new representation for video …Unwrap Mosaics: A new representation for video editing Alex Rav-Acha, Pushmeet Kohli, Carsten Rother, Andrew Fitzgibbon Microsoft](https://reader033.vdocument.in/reader033/viewer/2022043002/5f7e2b9ba81c64132f4e6d8a/html5/thumbnails/9.jpg)
Unwrap Mosaics
A method to recover and edit dense models of deforming 3D
surfaces from a video.
Accompanying the recovered texture map will be a 2D-to-2D
mapping describing the texture map’s projection to the mapping describing the texture map’s projection to the
images, and a sequence of binary masks modeling occlusion.
The combination of texture map, 2D–2D mapping, and
occlusion masks is called the unwrap mosaic.
![Page 10: Unwrap Mosaics: A new representation for video …Unwrap Mosaics: A new representation for video editing Alex Rav-Acha, Pushmeet Kohli, Carsten Rother, Andrew Fitzgibbon Microsoft](https://reader033.vdocument.in/reader033/viewer/2022043002/5f7e2b9ba81c64132f4e6d8a/html5/thumbnails/10.jpg)
Unwrap Mosaics
3D models – But not 3D !
Uses
• Wide range of editing operations on the input video
• Model easier to recover from video and helps us work with self
occlusion and deforming surfacesocclusion and deforming surfaces
The main contribution of this paper is the algorithm to recover the
unwrap mosaic from images. This algorithm is an energy minimization
procedure.
![Page 11: Unwrap Mosaics: A new representation for video …Unwrap Mosaics: A new representation for video editing Alex Rav-Acha, Pushmeet Kohli, Carsten Rother, Andrew Fitzgibbon Microsoft](https://reader033.vdocument.in/reader033/viewer/2022043002/5f7e2b9ba81c64132f4e6d8a/html5/thumbnails/11.jpg)
High level steps of Algorithm
Steps that correspond to minimization of the model energy via
coordinate descent
1. Segmentation
2. Tracking
3. Embedding3. Embedding
4. Mosaic stitching
5. Track refinement
6. Iterate
![Page 12: Unwrap Mosaics: A new representation for video …Unwrap Mosaics: A new representation for video editing Alex Rav-Acha, Pushmeet Kohli, Carsten Rother, Andrew Fitzgibbon Microsoft](https://reader033.vdocument.in/reader033/viewer/2022043002/5f7e2b9ba81c64132f4e6d8a/html5/thumbnails/12.jpg)
Segmentation
Steps
• Segmentation into independent moving objects
• Variant used here is “video cut and paste” – allows
segmentation maps to be obtained with relatively segmentation maps to be obtained with relatively
little effort
• Automatic segmentation maps computation is also
possible.
![Page 13: Unwrap Mosaics: A new representation for video …Unwrap Mosaics: A new representation for video editing Alex Rav-Acha, Pushmeet Kohli, Carsten Rother, Andrew Fitzgibbon Microsoft](https://reader033.vdocument.in/reader033/viewer/2022043002/5f7e2b9ba81c64132f4e6d8a/html5/thumbnails/13.jpg)
Tracking
AimTo recover the texture map of a deforming 3D object from a
sequence of 2D images.
=====================================Although the model is changing its shape, the texture map may be Although the model is changing its shape, the texture map may be
assumed to be constant, as on skin and cloth.
Consider a point on the object. As this point moves through the video, it
generates a 2D trajectory or track. Conversely, standard computer vision
algorithms for point detection and tracking can be used to compute such
tracks from the input sequence.
![Page 14: Unwrap Mosaics: A new representation for video …Unwrap Mosaics: A new representation for video editing Alex Rav-Acha, Pushmeet Kohli, Carsten Rother, Andrew Fitzgibbon Microsoft](https://reader033.vdocument.in/reader033/viewer/2022043002/5f7e2b9ba81c64132f4e6d8a/html5/thumbnails/14.jpg)
Embedding
- Computational cornerstone of the method
- The tracks that were generated are viewed as high
dimensional projection of 2D surfaces
-The point (u; v) in parameter space generates a vector of
image positions (x1; y1; x2; y2; :::; xT ; yT )image positions (x1; y1; x2; y2; :::; xT ; yT )
- We can recover the surface’s (u; v) parameter space by
computing an embedding of the point tracks into 2D, directly
yielding (u; v) coordinates for each track.
![Page 15: Unwrap Mosaics: A new representation for video …Unwrap Mosaics: A new representation for video editing Alex Rav-Acha, Pushmeet Kohli, Carsten Rother, Andrew Fitzgibbon Microsoft](https://reader033.vdocument.in/reader033/viewer/2022043002/5f7e2b9ba81c64132f4e6d8a/html5/thumbnails/15.jpg)
Mosaic stitching
- The embedding defines a map from the tracked points in
each image to the (u; v) parameter space.
- Interpolating this mapping allows each image to be warped
into the common frame.
- A variation of the standard mosaic stitching technique - A variation of the standard mosaic stitching technique
[Agarwala et al. 2004] emerges naturally from the energy
formulation.
- In simple terms, creating a mosaic from different images.
![Page 16: Unwrap Mosaics: A new representation for video …Unwrap Mosaics: A new representation for video editing Alex Rav-Acha, Pushmeet Kohli, Carsten Rother, Andrew Fitzgibbon Microsoft](https://reader033.vdocument.in/reader033/viewer/2022043002/5f7e2b9ba81c64132f4e6d8a/html5/thumbnails/16.jpg)
Track Refinement
- The mosaic is already available in this step (prev 3 steps
build the mosaic)
- The generated mosaic is not accurate, but good enough to
create a reference template to match the original frames
- Reduces drift that may have been present after the tracking - Reduces drift that may have been present after the tracking
phase
- Track info propagates to occluded areas giving complete
description of the object motion
![Page 17: Unwrap Mosaics: A new representation for video …Unwrap Mosaics: A new representation for video editing Alex Rav-Acha, Pushmeet Kohli, Carsten Rother, Andrew Fitzgibbon Microsoft](https://reader033.vdocument.in/reader033/viewer/2022043002/5f7e2b9ba81c64132f4e6d8a/html5/thumbnails/17.jpg)
Iterate-Repeat the above steps again to fine tune precision
-Hope to compute a better embedding and mosaic
by iterating
![Page 18: Unwrap Mosaics: A new representation for video …Unwrap Mosaics: A new representation for video editing Alex Rav-Acha, Pushmeet Kohli, Carsten Rother, Andrew Fitzgibbon Microsoft](https://reader033.vdocument.in/reader033/viewer/2022043002/5f7e2b9ba81c64132f4e6d8a/html5/thumbnails/18.jpg)
Two Input Video Frames
![Page 19: Unwrap Mosaics: A new representation for video …Unwrap Mosaics: A new representation for video editing Alex Rav-Acha, Pushmeet Kohli, Carsten Rother, Andrew Fitzgibbon Microsoft](https://reader033.vdocument.in/reader033/viewer/2022043002/5f7e2b9ba81c64132f4e6d8a/html5/thumbnails/19.jpg)
Unwrap Mosaic
![Page 20: Unwrap Mosaics: A new representation for video …Unwrap Mosaics: A new representation for video editing Alex Rav-Acha, Pushmeet Kohli, Carsten Rother, Andrew Fitzgibbon Microsoft](https://reader033.vdocument.in/reader033/viewer/2022043002/5f7e2b9ba81c64132f4e6d8a/html5/thumbnails/20.jpg)
Edited Mosaic
![Page 21: Unwrap Mosaics: A new representation for video …Unwrap Mosaics: A new representation for video editing Alex Rav-Acha, Pushmeet Kohli, Carsten Rother, Andrew Fitzgibbon Microsoft](https://reader033.vdocument.in/reader033/viewer/2022043002/5f7e2b9ba81c64132f4e6d8a/html5/thumbnails/21.jpg)
Two Output Video Frames
![Page 22: Unwrap Mosaics: A new representation for video …Unwrap Mosaics: A new representation for video editing Alex Rav-Acha, Pushmeet Kohli, Carsten Rother, Andrew Fitzgibbon Microsoft](https://reader033.vdocument.in/reader033/viewer/2022043002/5f7e2b9ba81c64132f4e6d8a/html5/thumbnails/22.jpg)
Limitations1. It requires quite well textured objects in order to work
-Textured surfaces are required for point tracking. Low
texture, such as on a face, requires good focus and
lighting. One dimensional textures (stripes, rather than
spots) suffer from the aperture problem.
2. Motion blur reduces efficacy of point tracking.2. Motion blur reduces efficacy of point tracking.
3. Object with protrusions do not yield useful mosaics
4. Strong shadows will disrupt tracking. Smooth lighting is
assumed
5. Limited to disc-topology objects. Eg : A rotating cylinder
will be reconstructed as a long tapestry rather than a
mosaic with cylindrical topology.
![Page 23: Unwrap Mosaics: A new representation for video …Unwrap Mosaics: A new representation for video editing Alex Rav-Acha, Pushmeet Kohli, Carsten Rother, Andrew Fitzgibbon Microsoft](https://reader033.vdocument.in/reader033/viewer/2022043002/5f7e2b9ba81c64132f4e6d8a/html5/thumbnails/23.jpg)
Disc topology limitation
Unwrap Mosaic
![Page 24: Unwrap Mosaics: A new representation for video …Unwrap Mosaics: A new representation for video editing Alex Rav-Acha, Pushmeet Kohli, Carsten Rother, Andrew Fitzgibbon Microsoft](https://reader033.vdocument.in/reader033/viewer/2022043002/5f7e2b9ba81c64132f4e6d8a/html5/thumbnails/24.jpg)
User interaction & Hinting
-Algorithm can work completely automatically
-Situations where lack of texture, repeated structures, or
motion blur could cause the recovered mosaic not to cover
all of the object.
-New brushes (brush strokes) are introduced to specify -New brushes (brush strokes) are introduced to specify
points that were not covered by automatic segmentation.
![Page 25: Unwrap Mosaics: A new representation for video …Unwrap Mosaics: A new representation for video editing Alex Rav-Acha, Pushmeet Kohli, Carsten Rother, Andrew Fitzgibbon Microsoft](https://reader033.vdocument.in/reader033/viewer/2022043002/5f7e2b9ba81c64132f4e6d8a/html5/thumbnails/25.jpg)
User interaction & Hinting
a. Automatic Computation
![Page 26: Unwrap Mosaics: A new representation for video …Unwrap Mosaics: A new representation for video editing Alex Rav-Acha, Pushmeet Kohli, Carsten Rother, Andrew Fitzgibbon Microsoft](https://reader033.vdocument.in/reader033/viewer/2022043002/5f7e2b9ba81c64132f4e6d8a/html5/thumbnails/26.jpg)
User interaction & Hinting
b. User brush strokes
![Page 27: Unwrap Mosaics: A new representation for video …Unwrap Mosaics: A new representation for video editing Alex Rav-Acha, Pushmeet Kohli, Carsten Rother, Andrew Fitzgibbon Microsoft](https://reader033.vdocument.in/reader033/viewer/2022043002/5f7e2b9ba81c64132f4e6d8a/html5/thumbnails/27.jpg)
User interaction & Hinting
c. New mosaic computed with only the
brush strokes
![Page 28: Unwrap Mosaics: A new representation for video …Unwrap Mosaics: A new representation for video editing Alex Rav-Acha, Pushmeet Kohli, Carsten Rother, Andrew Fitzgibbon Microsoft](https://reader033.vdocument.in/reader033/viewer/2022043002/5f7e2b9ba81c64132f4e6d8a/html5/thumbnails/28.jpg)
Results• For video editing, the primary criterion is obviously the
range and quality of effects that are enabled by the model.
• Computation times for all of these sequences (show in
the video) are of the order of a few hours.
Check out this video !!Check out this video !!
![Page 29: Unwrap Mosaics: A new representation for video …Unwrap Mosaics: A new representation for video editing Alex Rav-Acha, Pushmeet Kohli, Carsten Rother, Andrew Fitzgibbon Microsoft](https://reader033.vdocument.in/reader033/viewer/2022043002/5f7e2b9ba81c64132f4e6d8a/html5/thumbnails/29.jpg)
Thank you
Any Questions ??!??!