sp atiotemporal re gularity f low ( spref )
Post on 21-Jan-2016
18 Views
Preview:
DESCRIPTION
TRANSCRIPT
Spatiotemporal Regularity Flow (SPREF)
Mubarak ShahComputer Vision LabSchool of Electrical Engineering & Computer ScienceUniversity of Central FloridaOrlando, FL 32765
What are good features?
Color Histograms Eigen vectors Wavelet Coefficients Edges
Spatiotemporal Surfaces of edges XY, XT, YT slices Spatial/spatiotemporal tensors
SIFT Optical Flow
SPREF
New Spatiotemporal feature for VACE Generalization of Isophotes, Optical flow,… Can be computed when gradient is zero It analyzes whole region instead of a single pixel
Applications Image and Video In-painting Object removal Video Compression Tracking, Segmentation, …
Spatiotemporal Regularity Definition: A spatiotemporal volume is regular
along the directions, in which the pixels change the least.
SPatiotemporal REgularity Flow (SPREF) 3D vector field ζ models the directions of regularity
No motion (Spatial Regularity) Depends on the regularity of a single frame
Presence of motion (Temporal Regularity) Global motion
Single regularity model
Local motion Multiple regularity models
Estimating SPREF
…gives the directions, along which the sum of the gradients is minimum:
where F is the spatiotemporal volume, and H is a regularizing filter (Gaussian)
dxdydttyx
tyxHFE
2
),,(
),,)((
The SPREF Energy Functions The energy function is modified according to
the flow type: x-y Parallel: ζ(c1'[t], c2'[t],1)
y-t Parallel: ζ(1,c2'[x], c3'[x])
x-t Parallel: ζ(c1'[y],1, c3'[y])
dxdydtfxcfxcfE tyx2
32 ][']['
dxdydtfycffycE tyx2
31 ][']['
dxdydtfftcftcE tyx2
21 ][']['
Solving for the SPREF Approximate each flow component, cm'[p],
with a 1D spline Incorporates multiple frames in the solution.
i
lim ipbpc )2(]['
• Quadratic minimization of the energy functions • Solve for the spline parameters
Solving T-SPREF Equation
The original synthetic sequence (8 frames)
x-y Parallelism: ζ(c1'[t], c2'[t],1)
y-t Parallelism: ζ(1,c2'[x], c3'[x])
There are three types of planar parallelism constraints.
x-t Parallelism: ζ(c1'[y],1, c3'[y])
The SPREF Curves
… define the actual paths, along which the GOF is regular.
}3,2,1{]['][1
micpcp
imm
T-SPREF - An Overview
Demo
x-y Parallel SPREF
y-t Parallel SPREF
y
t
x
ζ(1,c2'[x], c3'[x])x
y
x-t Parallel SPREF
y
t
x
ζ(c1'[y],1, c3'[y]) x
y
T-SPREF Results (Flower Sequence)
Oblique View Top View Side View
T-SPREF Results (Alex Sequence)
x
y
t
y
t t
x
Oblique View Top View Side View
The Affine SPREF (A-SPREF)
When the directions of regularity depend on multiple axes (zooming, rotation and etc.) Precision of T-SPREF goes down Translational flow model to Affine flow model Affine (A-)SPREF
iV t
HFtyxc
y
HFtyxc
x
HFE
2
'2
'1 *],,[)(],,[)(
][][][],,[ 131211'1 taytaxtatyxc
][][][],,[ 232221'2 taytaxtatyxc
Flow energy equation:
Comparison of T- and A- SPREF
1st row: A synthetic sequence from the Lena image.
2nd row: T-SPREF approximation to the underlying directions of regularity.
3rd row: A-SPREF approximation of the directions of regularity.
More examples
T-SPREF A-SPREF
T-SPREF A-SPREF
Comparison of T- and A- SPREF
Optical Flow Vs SPREF
SPREF carries similar but not necessarily the same information as the optical flow. SPREF captures both the spatial and temporal
regularity Optical flow only cares about motion information
in temporal direction. When motion exists, the directions of xy parallel
SPREF depend on direction of motion. If the motion is globally translational, then xy-
parallel SPREF converges to the optical flow.
Optical Flow Vs SPREF
Optical Flow is not well-defined where the spatiotemporal gradients are insignificant.
Spline-based formulation of SPREF minimizes over multiple frames.
The true optical flow usually lacks plane parallelism.
Optical Flow Vs SPREF
Applications of SPREF
Inpainting
Filling in the regions of missing data
Image Inpainting Missing regions create spatial holes Inpainting the missing region in the SPREF direction
Video Inpainting Missing regions create spatiotemporal holes Inpainting these holes require using the information
from temporal neighbors.
Image Inpainting
Video Inpainting
Requires understanding the temporal behavior of the pixels.
The temporal behavior of the undamaged pixels gives clues about the behavior of the damaged pixels
Temporal behavior Modeled explicitly by x-y Parallel SPREF
Video Inpainting
The algorithm (cont’d)1. Estimate the x-y Parallel SPREF curves using the non-
missing regions. The pixels along the SPREF curves vary smoothly
2. Fit a spline to the non-missing pixels along each flow curve.
3. Interpolate the values of the missing pixels from the splines
Results
Big Bounce (Before)
Results
Big Bounce (Flow)
Results
Big Bounce (After)
Supervised Removal of Objects from Videos
Motivation
Object removal from videos Preceding step to video inpainting Manual selection of the object from each frame is
required. Time consuming
Use x-y Parallel SPREF to decrease the amount of manual work Removal along the SPREF curves
Algorithm
Given a group of frames (GOF):1) Compute the x-y Parallel SPREF, and the
SPREF curves
2) Remove the object from the first and the last frames of the GOF
3) Remove the pixels along the curves, whose first and last pixels have been removed.
Results
Golden Eye (Final)
86% reduction in manual work!
Video Compression Using SPREF
3D Wavelet Decomposition
Problem The spatiotemporal regularity of the GOF is not
taken into account
Solution Decompose the GOF along the SPREF directions Entropy along these directions is lower:
Higher compression rate
SPREF-based Video Compression
Warping the wavelet basis along the flow curves
x-y Parallel : G(x,y,t) = (x+c1[t], y+c2[t], t)
y-t Parallel : G(x,y,t) = (x,y+c2[x],t+c3[x])
x-t Parallel : G(x,y,t) = (x+c1[y], y, t+c3[t])
Choosing the correct SPREF type The correct SPREF type is the one that
minimizes the compression cost : Di + λRi
Di: Reconstruction error
λ: Lagrange multiplier Ri: Bit cost of the bandelet and flow coefficients
Segmentation for Optimal Compression Find the segmentation of the GOF (F) into
subGOFs (Fi), such that the total compression cost is minimized:
i
ii RDRD
Fi
Oct-tree Segmentation
Recursively partition the GOF (F) into rectangular prisms (cuboids), Fi.
Compute the best flow and the compression cost for each cuboid.
Use split/merge algorithm to achieve the final segmentation. Merge the child nodes if:
i
jjii RDRD
Compression results for frames 98-105 of the Alex sequence at 1000kbps
Compression results for frames 11-18 of the Akiyo sequence at 480kbps
Compression results for frames 99-106 of the Mobile sequence at 350kbps
Compression results for frames 26-33 of the Foreman sequence at 500kbps
Compression Results
The bit-rate vs PSNR plots of (a) Alex, (b) Akiyo. Both SPREF-based compression and LIMAT framework are shown in the results.
(a) (b)
LIMAT framework, Secker and Taubman, IEEE TIP, 2004.
Compression Results (cond.)
The bit-rate vs PSNR plots of (a) Foreman, (b) Mobile. Both SPREF-based compression and LIMAT framework are shown in the results.
(a) (b)
Summary
SPREF New Spatiotemporal Feature Computes direction of regularity simultaneously in
space & Time Similar to optical flow, edge direction.. SPREF is plane parallel (xy, xt, yt) SPREF is computed using region/image
information instead of a single pixel SPREF is defined even when gradient is zero
Summary
Applications Image and Video In-painting Object Removal Video Compression Tracking Segmentation
Orkun Alatas
August 16th, 1977 - September 3rd, 2005
Publications
Orkun Alatas, Omar Javed, and Mubarak Shah, “Video Compression Using Structural Flow", International Conference on Image Processing, Genova, Italy, September 11-14, 2005.
Orkun Alatas, Omar Javed, and Mubarak Shah, “Video Compression Using Spatiotemporal Regularity Flow, IEEE Transactions on Image Processing, December 2006.
Orkun Alatas, Pingkun Yan, and Mubarak Shah, “Spatiotemporal Regularity Flow, (SPREF): Its Estimation and Applications”, IEEE Transactions on Circuit & Systems Video Technology (accepted).
Computer Vision Lab
http://www.cs.ucf.edu/~vision
shah@cs.ucf.edu
Group
top related