lecture 4: video applications · 2016-07-27 · lecture 4: video applications topological time...

Lecture 4: Video Applications

Topological Time Series Analysis - Theory And Practice

Jose Perea, Michigan State University. Chris Tralie, Duke University

7/21/2016

Topological Time Series Analysis - Theory And Practice Lecture 4: Video Applications

Table of Contents

I Sliding window video definition / examples

B Sliding window video formalism

B Natural video dynamics/geometry

B Memory efficiency / preprocessing

B Interactive Example

B KTH Dataset


Sliding Window Videos

Y[n]=

X[n]

.

.

.

Time

M

X[n+1]

X[n+2]

X[n+M-1]

X[n]

X[n+M-1]

http://www.ctralie.com/Research/SlidingWindowVideo-SOCG2016/




Examples

Jumping jacks, heartbeat animation, my neck


Table of Contents

B Sliding window video definition / examples

I Sliding window video formalism




B KTH Dataset


Table of Contents



I Natural video dynamics/geometry



B KTH Dataset


Pure Cosine Composition Model

On the board


Oscillating Line Segment

0 20 40 60 80 100120140Pixel Index i

0

50

100

150

200

Tim

e t

Oscillating Line Segment Raw Embedding 2D PCA T-length Embedding 2D PCA

B Good Toy Example for Natural VideoB 2 questions

1. Why is it path-like without embedding?2. Why is it curved?



0 20 40 60 80 100120140Pixel Index i

0

50

100

150

200

Tim

e t



1. Why is it path-like without embedding?

2. Why is it curved?



0 20 40 60 80 100120140Pixel Index i

0

50

100

150

200

Tim

e t



1. Why is it path-like without embedding?2. Why is it curved?


Jumping Jacks Example

Show PCA Videos


Distance Matrix Interpretation

Oscillating Bar

Raw Embedding 2D PCA T-length Embedding 2D PCA

0 50 100 150Frame Number

0

50

100

150

Fram

e N

um

ber

0 20 40 60 80 100 120 140Frame Number

0

20

40

60

80

100

120

140

Fram

e N

um

ber



Convolving along diagonals (show video)



Jumping Jacks

0 2 4 6 8RHSV Number

0

20

40

60

80

100

Fram

e Nu

mbe

rRight Hand Singular Vectors 2D PCA

0 20 40 60 80 100Frame Number

0

20

40

60

80

100

Fram

e Nu

mbe

r

Self-Similarity Matrix

0 2 4 6 8RHSV Number

0

10

20

30

40

50

60

70

Fram

e Nu

mbe

r

Right Hand Singular Vectors 2D PCA

0 10 20 30 40 50 60 70Frame Number

0

10

20

30

40

50

60

70

Fram

e Nu

mbe

r

Self-Similarity Matrix



Why is it curved?B Eulerian pixel view

0 20 40 60 80 100 120 140 160Pixel Index i

0

50

100

150

200

250

Tim

e t


0 50 100 150 2002.0

1.5

1.0

0.5

0.0

0.5

1.0

1.5

2.0Pixel Index 40


Jumping Jacks Video Eulerian View

Show video


Principal Component Videos

Show videos for heartbeat and jumping jacks


Table of Contents




I Memory efficiency / preprocessing


B KTH Dataset


Naive Embedding

Naive video stacking blows up in memory!

B E.g. 400x400 video of length 300 frames, delay of 30frames

Solution?

B Do SVD


What About Drift?

e.g. X (t) = (cos(t), sin(t), t)Show video

B Take smoothed time derivative of each pixel

g(t) = te−t2/(2σ2)

B Can be interpreted as a bandpass filter


1D Drift Example


Table of Contents





I Interactive Example

B KTH Dataset


Table of Contents






I KTH Dataset


KTH Dataset

http://www.nada.kth.se/cvap/actions/


http://www.nada.kth.se/cvap/actions/

KTH Dataset

B We are not doing activity recognition!!Castrodad, Alexey, and Guillermo Sapiro. “Sparse modeling of human actionsfrom motion imagery.” International journal of computer vision 100.1 (2012):1-15.

B Instead, we will be ranking periodicity


KTH Dataset: Experiment 1

Window length vs persistence

B Keep dimension fixed. For window length W , τ = d/WB Keep number of points fixed. dT = (N − dτ)/NB Take blocks of 160 frames




B Keep dimension fixed. For window length W , τ = d/W

B Keep number of points fixed. dT = (N − dτ)/NB Take blocks of 160 frames




B Keep dimension fixed. For window length W , τ = d/WB Keep number of points fixed. dT = (N − dτ)/N

B Take blocks of 160 frames




B Keep dimension fixed. For window length W , τ = d/WB Keep number of points fixed. dT = (N − dτ)/NB Take blocks of 160 frames


KTH Dataset: Experiment 2 (Interactive)

Rank 4 videos as a class


KTH Dataset: Experiment 3 (Interactive)

Rank all videos for a particular person


lecture 4: video applications · 2016-07-27 · lecture 4: video applications topological time...

Documents