webcam-synopsis: peeking around the world

Webcam-synopsis:Webcam-synopsis:Peeking Around the WorldPeeking Around the World

Young Ki Baik (CV Lab.)Young Ki Baik (CV Lab.)2008. 4. 4 (Fri)2008. 4. 4 (Fri)

Webcam-Synopsis:Webcam-Synopsis: Peeking Around the World Peeking Around the World

ReferencesReferencesWebcam Synopsis : Peeking Around the World

• Yael Pritch, Alex Rav-Acha, Avital Gutman Shmuel Peleg• (ICCV 2007)

Non-Chronological Video Synopsis and Indexing• Yael Pritch, Alex Rav-Acha, Shmuel Peleg (PAMI 2008)

Background Cut• J. Sun, W. Zhang, X. Tang, and H. Shum (ECCV 2006)


What is Video-Synopsis?What is Video-Synopsis?

Video synopsis is compact video data reducing spatio-temporal redundancy in input video.


Demo (Final result)Demo (Final result)


How can we make Video Synopsis?How can we make Video Synopsis?Basic concept

• Detect activity tubes… in input video I.

• Put activity tubes… to the S… considering relation between - tubes and background. - tube and other tubes.


SynopsisSynopsisProperties of video synopsis S

• The video synopsis S should be substantially shorter than the original video I.

• Maximum “activity” from the original video should appear in the synopsis video.

• The dynamics of the objects should be preserved in the synopsis video. (for example, regular fast-forward may fail to preserve the dynamics of fast objects.)

• Visible seams and fragmented objects should be avoided.


OBJECT-BASED OBJECT-BASED SYNOPSISSYNOPSIS


Object-based synopsis constructionObject-based synopsis constructionOnline Phase

• This phase is done in real time during video capture.• Preparing step for response phase.

Response Phase • started with user query.• This phase may take a few minutes, depending on

the activity in the time period of interest.


Overall flowOverall flow


ONLINE PHASEONLINE PHASE


Video AcquisitionVideo AcquisitionVideo Volume Data

y

t

x

I(x, y, t) : 3D Space Time Volume(x, y) : Spatial coordinates of the image t : Frame index


Background Video ConstructionBackground Video ConstructionTemporal Median

y

t

x

Sorting by intensity

Selecting median value

4 minutes

Temporal Window


Background Video ConstructionBackground Video Construction

y

t

x

y

t

xI B


Moving Objects(tubes) ExtractionMoving Objects(tubes) ExtractionUsing Background Cut (ECCV 2006)

• Using min-cut to get a smooth segmentation of foreground objects.

• Only considering 2D image (not video).


Moving Objects(tubes) ExtractionMoving Objects(tubes) ExtractionExtracting moving objects

• We have information for…

I

I(x, y, t) : 3D Space Time VolumeB(x, y, t) : 3D Space Time Volume for background

B

Current ~

: Current image = I (x, y, current time)

: Current background image = B (x, y, current time)


Moving Objects(tubes) ExtractionMoving Objects(tubes) ExtractionSimplify Object detection problem to…

• Binary labeling f : Foreground = 1, Background = 0 • Obtaining the desirable labeling f … → with Gibbs energy function.

1 2,

( ) ,r r sr I r s Neighbors

E f E f E f f

: target position of image: neighboring of target position (4 Neighbors): user defined weight

rs

Color term Contrast term



• Unary term (or color term)

1 2,


E f E f E f f

d I r B r

11

1

01

d kE

k d otherwise

2

1 1 2 10 - 0

d kE d k k d k

otherwise

1 2, : user defined thresholdsk k

Foreground energy Background energy

Color differences between image and background


Moving Objects(tubes) ExtractionMoving Objects(tubes) ExtractionUsing Background Cut (ECCV 2006)

• Using min-cut to get a smooth segmentation of foreground objects.

• Only considering 2D image (not video).



• Binary term (or contrast term)

1 2,


E f E f E f f

: user defined weight

2

2 ( , )r s r sE f f f f Exp I r I s

In case of same labeling, energy is decreased.In case of different labeling, energy defends on differences between neighboring intensities.


Moving Objects(tubes) ExtractionMoving Objects(tubes) ExtractionUsing min-cut algorithm…

- Moving objects are extracted.



• Constructing a mask of all foreground pixels in space-time volume.

• Applying a 3D morphological dilation on mask.• Finally we can obtain activity tubes (or objects).

x

t

Example of tubes


Moving Objects(tubes) ExtractionMoving Objects(tubes) ExtractionObject queue

• Obtained activity tubes are saved in queue.

End of online phase…


RESPONSE PHASERESPONSE PHASE


Response phaseResponse phaseUser query

• “I would like to watch in one minute a synopsis of the video from this camera captured during the last hour.”

• “I would like to watch in five minutes a synopsis of the last week.”…

Input Input videovideo SynopsisSynopsis

When user query occurred, response phase are started.


Response phaseResponse phaseUser query

• “I would like to watch in one minute a synopsis of the video from this camera captured during the last hour.”

• “I would like to watch in five minutes a synopsis of the last week.”…


Creating time lapse background videoCreating time lapse background videoTime lapse background video (Bout)

• The background of the synopsis video

• It should represent the background changes over time.• Day-night transitions, etc.

• It should represent the background of the activity tubes.

BackgroundVideo

Time lapseBGvideo

Sampling uniformly


Creating time lapse background videoCreating time lapse background videoTime lapse background video

• A temporal activity histogram Ha• A uniform temporal histogram Ht


Creating time lapse background videoCreating time lapse background videoTime lapse background video

• A temporal activity histogram Ha• A uniform temporal histogram Ht

• Interpolating the two histograms Hi = Ha + (1- λ)Ht

BackgroundVideo (b)

Time lapseBGvideo

Sampling with Hi

Bout


Selecting tubes and stitchingSelecting tubes and stitchingDefinition

Input Input videovideo SynopsisSynopsis

ˆ

Mb

b

b̂b M

: temporal mapping (or time shift): tube with time segment: shifted tube with time segment

,s eb b bt t t

ˆ ˆ ˆ,s eb b bt t t

ˆM b b


Select tubes and stitchingSelect tubes and stitchingEnergy for temporal mapping M

ˆM b b

, ,Q

: Queue: user selected weights

,

ˆ ˆ

ˆ ˆ ˆ ˆ , ,

a sb Q

t cb b Q

E M E b E b

E b b E b b

bb

: target tube: other tube


Select tubes and stitchingSelect tubes and stitchingGraph

SynopsisSynopsis

t

Longest tube tube tube

tube

tube

Node number = number of tube

Label = number of frame t

label

label

label

label



• Unary term

• Activity Cost

ˆ ˆ

a sb QE b E b

SynopsisSynopsis

ˆx,y,t b

0 tube region inside the Sˆ = , , , , tube region outside the SaE b I x y t B x y t

Only pixel that were not entered into the synopsis are added to the activity cost.



• Unary term

• Consistency with background

ˆ ˆ

a sb QE b E b

SynopsisSynopsis

b̂

b̂

ˆˆx,y b ,t

ˆ = , , , ,out

s outbt t

E b I x y t B x y t

: border of the mapped activity tube



• Binary term

• Collision Cost SynopsisSynopsis

b b

ˆ ˆˆ ˆx,y,t t t

ˆ = , , , , , , , ,c b bE b I x y t B x y t I x y t B x y t

,ˆ ˆ ˆ ˆ, ,t cb b Q

E b b E b b

This expression give a low penalty to pixel whose color is similar to the background…



• Binary term

• Temporal Consistency CostSynopsisSynopsis

ˆtE b

,ˆ ˆ ˆ ˆ, ,t cb b Q

E b b E b b

Preserving the chronological order of events


Select tubes and stitchingSelect tubes and stitchingE(M) is minimized by Min-cut algorithm.

Stitching the synopsis video• To make more reliable result… → The α–Poisson Image Blending

End of response phase…


ConclusionConclusionThe method to creating a short video that is a synopsis

of and video stream has been presented.

DiscussionDiscussionContribution

• Found out new interesting application.Can we find more contribution?

• Finding higher dimensional problem and Solving it. • 3D volume animation, etc.

webcam-synopsis: peeking around the world

Documents

synopsis video

chronological video

background webcamsynopsis

worldreferenceswebcam

current timewebcamsynopsis

frame indexwebcamsynopsis

input video

original video