unit7 - cs.yorku.ca · 7 orientation in visual space-time: representation • goal is to analyze...
TRANSCRIPT
![Page 1: Unit7 - cs.yorku.ca · 7 Orientation in visual space-time: Representation • Goal is to analyze spatiotemporal data according to its local orientation structure. – Consider orientation](https://reader034.vdocument.in/reader034/viewer/2022050414/5f8b2578188c234886699595/html5/thumbnails/1.jpg)
1
CSE 4422/5323
Unit 7: Spatiotemporal analysis
![Page 2: Unit7 - cs.yorku.ca · 7 Orientation in visual space-time: Representation • Goal is to analyze spatiotemporal data according to its local orientation structure. – Consider orientation](https://reader034.vdocument.in/reader034/viewer/2022050414/5f8b2578188c234886699595/html5/thumbnails/2.jpg)
2
Outline
• Introduction• Orientation in visual space-time• A representation for spatiotemporal patterns• Spatiotemporal boundaries• A framework for spatiotemporal analysis• Applications• Summary
![Page 3: Unit7 - cs.yorku.ca · 7 Orientation in visual space-time: Representation • Goal is to analyze spatiotemporal data according to its local orientation structure. – Consider orientation](https://reader034.vdocument.in/reader034/viewer/2022050414/5f8b2578188c234886699595/html5/thumbnails/3.jpg)
3
Introduction
• We have considered the analysis of spatial structure.– Oriented, bandpass representations.
• We have considered analysis of the temporal dimension– Motion
• Now we consider the integrated analysis and interpretation of the spatial and temporal dimensions.– Spatiotemporal analysis
![Page 4: Unit7 - cs.yorku.ca · 7 Orientation in visual space-time: Representation • Goal is to analyze spatiotemporal data according to its local orientation structure. – Consider orientation](https://reader034.vdocument.in/reader034/viewer/2022050414/5f8b2578188c234886699595/html5/thumbnails/4.jpg)
4
Outline
• Introduction• Orientation in visual space-time• A representation for spatiotemporal patterns• Spatiotemporal boundaries• A framework for spatiotemporal analysis• Applications• Summary
![Page 5: Unit7 - cs.yorku.ca · 7 Orientation in visual space-time: Representation • Goal is to analyze spatiotemporal data according to its local orientation structure. – Consider orientation](https://reader034.vdocument.in/reader034/viewer/2022050414/5f8b2578188c234886699595/html5/thumbnails/5.jpg)
5
Orientation in visual space-time: Basics
• The local orientation (or lack thereof) of a pattern is one of its most salient characteristics.
• Geometrically, orientation captures the local first-order structure of a pattern.
• For vision, local spatiotemporal orientation can have additional interpretations.– Image velocity is manifest as spatiotemporal orientation.– And more...
![Page 6: Unit7 - cs.yorku.ca · 7 Orientation in visual space-time: Representation • Goal is to analyze spatiotemporal data according to its local orientation structure. – Consider orientation](https://reader034.vdocument.in/reader034/viewer/2022050414/5f8b2578188c234886699595/html5/thumbnails/6.jpg)
6
Orientation in visual space-time: Graphic
y
x
t
![Page 7: Unit7 - cs.yorku.ca · 7 Orientation in visual space-time: Representation • Goal is to analyze spatiotemporal data according to its local orientation structure. – Consider orientation](https://reader034.vdocument.in/reader034/viewer/2022050414/5f8b2578188c234886699595/html5/thumbnails/7.jpg)
7
Orientation in visual space-time: Representation
• Goal is to analyze spatiotemporal data according to its local orientation structure.
– Consider orientation in x-t and y-t planes, with local weighted averaging in orthogonal spatial dimension.
– Filter for multiple bands each tuned for certain orientations ina spatiotemporal plane.
– For example, select 4 orientations/plane: horizontal, vertical, 2 diagonals
• Consider single spatiotemporal scale (for now).
![Page 8: Unit7 - cs.yorku.ca · 7 Orientation in visual space-time: Representation • Goal is to analyze spatiotemporal data according to its local orientation structure. – Consider orientation](https://reader034.vdocument.in/reader034/viewer/2022050414/5f8b2578188c234886699595/html5/thumbnails/8.jpg)
8
Orientation in visual space-time: Filtering
• Apply filters tuned to 4 different orientations in both x-t and y-t domains.
• In general, might consider additional directions.
• Filter specifics: – Oriented bandpass filters in
spatiotemporal slice. – Lowpass filter in orthogonal
spatial dimension.– Pointwise squared to yield
local “oriented energy”.
x
t
xs
l
r
xf
![Page 9: Unit7 - cs.yorku.ca · 7 Orientation in visual space-time: Representation • Goal is to analyze spatiotemporal data according to its local orientation structure. – Consider orientation](https://reader034.vdocument.in/reader034/viewer/2022050414/5f8b2578188c234886699595/html5/thumbnails/9.jpg)
9
Orientation in visual space-time: Filtering
• Apply filters tuned to 4 different orientations in both x-t and y-t domains.
• In general, might consider additional directions.
• Filter specifics: – Oriented bandpass filters in
spatiotemporal slice. – Lowpass filter in orthogonal
spatial dimension.– Pointwise squared to yield
local “oriented energy”.
y
t
ys
u
yf
d
![Page 10: Unit7 - cs.yorku.ca · 7 Orientation in visual space-time: Representation • Goal is to analyze spatiotemporal data according to its local orientation structure. – Consider orientation](https://reader034.vdocument.in/reader034/viewer/2022050414/5f8b2578188c234886699595/html5/thumbnails/10.jpg)
10
Orientation in visual space-time: Normalization
• For any given orientation, the filter response is a joint function of– orientation – contrast
• Normalization yields purer measure of orientation
with a small bias added for stability.
• Similarly for , , and their y-t counterparts.
ε
xsl xf
ε++++=
),,(),,(),,(),,(),,(
),,(tyxftyxstyxltyxr
tyxrtyxR
xx
![Page 11: Unit7 - cs.yorku.ca · 7 Orientation in visual space-time: Representation • Goal is to analyze spatiotemporal data according to its local orientation structure. – Consider orientation](https://reader034.vdocument.in/reader034/viewer/2022050414/5f8b2578188c234886699595/html5/thumbnails/11.jpg)
11
Orientation in visual space-time: Velocity
• Consider the response to a horizontally moving pattern of r, l and sfilters.
velocity
Filterresponse
![Page 12: Unit7 - cs.yorku.ca · 7 Orientation in visual space-time: Representation • Goal is to analyze spatiotemporal data according to its local orientation structure. – Consider orientation](https://reader034.vdocument.in/reader034/viewer/2022050414/5f8b2578188c234886699595/html5/thumbnails/12.jpg)
12
Orientation in visual space-time: Velocity
• Consider the response to a horizontally moving pattern of r, l and sfilters.
velocity
Filterresponse
![Page 13: Unit7 - cs.yorku.ca · 7 Orientation in visual space-time: Representation • Goal is to analyze spatiotemporal data according to its local orientation structure. – Consider orientation](https://reader034.vdocument.in/reader034/viewer/2022050414/5f8b2578188c234886699595/html5/thumbnails/13.jpg)
13
Orientation in visual space-time: Velocity
• Consider the response to a horizontally moving pattern of r, l and sfilters.
velocity
Filterresponse
![Page 14: Unit7 - cs.yorku.ca · 7 Orientation in visual space-time: Representation • Goal is to analyze spatiotemporal data according to its local orientation structure. – Consider orientation](https://reader034.vdocument.in/reader034/viewer/2022050414/5f8b2578188c234886699595/html5/thumbnails/14.jpg)
14
Orientation in visual space-time: Velocity
• Consider the response to a horizontally moving pattern of r, l and sfilters.
velocity
Filterresponse
![Page 15: Unit7 - cs.yorku.ca · 7 Orientation in visual space-time: Representation • Goal is to analyze spatiotemporal data according to its local orientation structure. – Consider orientation](https://reader034.vdocument.in/reader034/viewer/2022050414/5f8b2578188c234886699595/html5/thumbnails/15.jpg)
15
Orientation in visual space-time: Velocity
• Consider the response to a horizontally moving pattern of r, l and sfilters.
velocity
Filterresponse
![Page 16: Unit7 - cs.yorku.ca · 7 Orientation in visual space-time: Representation • Goal is to analyze spatiotemporal data according to its local orientation structure. – Consider orientation](https://reader034.vdocument.in/reader034/viewer/2022050414/5f8b2578188c234886699595/html5/thumbnails/16.jpg)
16
Orientation in visual space-time: Velocity
• Consider the response to a horizontally moving pattern of r, l and sfilters.
• By taking the difference r-l we get a single response that is properly signed WRT velocity.
velocity
Filterresponse
velocity
Filterresponse
![Page 17: Unit7 - cs.yorku.ca · 7 Orientation in visual space-time: Representation • Goal is to analyze spatiotemporal data according to its local orientation structure. – Consider orientation](https://reader034.vdocument.in/reader034/viewer/2022050414/5f8b2578188c234886699595/html5/thumbnails/17.jpg)
17
Orientation in visual space-time: Velocity
• Consider the response to a horizontally moving pattern of r, l and sfilters.
• By taking the difference r-l we get a single response that is properly signed WRT velocity.
• Dividing to get (r-l)/syields a response that is (approximately) linear with velocity.
velocity
Filterresponse
velocity
Filterresponse
velocity
Filterresponse
![Page 18: Unit7 - cs.yorku.ca · 7 Orientation in visual space-time: Representation • Goal is to analyze spatiotemporal data according to its local orientation structure. – Consider orientation](https://reader034.vdocument.in/reader034/viewer/2022050414/5f8b2578188c234886699595/html5/thumbnails/18.jpg)
18
Orientation in visual space-time: Velocity
Comparison with optical flow constraint equation (OFCE) • Recall the OFCE as derived from constant brightness assumption
• Let us restrict consideration to one spatial dimension + time
• Now, we can directly solve for (1D) velocity
0=++ tyx EvEuE
xt EEu /−=
0=+ tx EuE
![Page 19: Unit7 - cs.yorku.ca · 7 Orientation in visual space-time: Representation • Goal is to analyze spatiotemporal data according to its local orientation structure. – Consider orientation](https://reader034.vdocument.in/reader034/viewer/2022050414/5f8b2578188c234886699595/html5/thumbnails/19.jpg)
19
Orientation in visual space-time: Velocity
Comparison with optical flow constraint equation (OFCE) • Recall the OFCE as derived from constant brightness assumption
• Let us restrict consideration to one spatial dimension + time
• Now, we can directly solve for (1D) velocity
0=++ tyx EvEuE
xt EEu /−=
0=+ tx EuE
![Page 20: Unit7 - cs.yorku.ca · 7 Orientation in visual space-time: Representation • Goal is to analyze spatiotemporal data according to its local orientation structure. – Consider orientation](https://reader034.vdocument.in/reader034/viewer/2022050414/5f8b2578188c234886699595/html5/thumbnails/20.jpg)
20
Orientation in visual space-time: Velocity
Comparison with optical flow constraint equation (OFCE) • In contrast, let us consider simple differential filters for leftward and rightward movement
![Page 21: Unit7 - cs.yorku.ca · 7 Orientation in visual space-time: Representation • Goal is to analyze spatiotemporal data according to its local orientation structure. – Consider orientation](https://reader034.vdocument.in/reader034/viewer/2022050414/5f8b2578188c234886699595/html5/thumbnails/21.jpg)
21
Orientation in visual space-time: Velocity
Comparison with optical flow constraint equation (OFCE) • In contrast, let us consider simple differential filters for leftward and rightward movement
2/)(~
xt EEl +=
![Page 22: Unit7 - cs.yorku.ca · 7 Orientation in visual space-time: Representation • Goal is to analyze spatiotemporal data according to its local orientation structure. – Consider orientation](https://reader034.vdocument.in/reader034/viewer/2022050414/5f8b2578188c234886699595/html5/thumbnails/22.jpg)
22
Orientation in visual space-time: Velocity
Comparison with optical flow constraint equation (OFCE) • In contrast, let us consider simple differential filters for leftward and rightward movement
2/)(~
xt EEl +=t
x
+
-
![Page 23: Unit7 - cs.yorku.ca · 7 Orientation in visual space-time: Representation • Goal is to analyze spatiotemporal data according to its local orientation structure. – Consider orientation](https://reader034.vdocument.in/reader034/viewer/2022050414/5f8b2578188c234886699595/html5/thumbnails/23.jpg)
23
Orientation in visual space-time: Velocity
Comparison with optical flow constraint equation (OFCE) • In contrast, let us consider simple differential filters for leftward and rightward movement
2/)(~
xt EEl +=t
x+-
![Page 24: Unit7 - cs.yorku.ca · 7 Orientation in visual space-time: Representation • Goal is to analyze spatiotemporal data according to its local orientation structure. – Consider orientation](https://reader034.vdocument.in/reader034/viewer/2022050414/5f8b2578188c234886699595/html5/thumbnails/24.jpg)
24
Orientation in visual space-time: Velocity
Comparison with optical flow constraint equation (OFCE) • In contrast, let us consider simple differential filters for leftward and rightward movement
2/)(~
xt EEl +=t
x
++
--
![Page 25: Unit7 - cs.yorku.ca · 7 Orientation in visual space-time: Representation • Goal is to analyze spatiotemporal data according to its local orientation structure. – Consider orientation](https://reader034.vdocument.in/reader034/viewer/2022050414/5f8b2578188c234886699595/html5/thumbnails/25.jpg)
25
Orientation in visual space-time: Velocity
Comparison with optical flow constraint equation (OFCE) • In contrast, let us consider simple differential filters for leftward and rightward movement
2/)(~
xt EEl += 2/)(~
xt EEr −=t
x
+-
-+
![Page 26: Unit7 - cs.yorku.ca · 7 Orientation in visual space-time: Representation • Goal is to analyze spatiotemporal data according to its local orientation structure. – Consider orientation](https://reader034.vdocument.in/reader034/viewer/2022050414/5f8b2578188c234886699595/html5/thumbnails/26.jpg)
26
Orientation in visual space-time: Velocity
Comparison with optical flow constraint equation (OFCE) • In contrast, let us consider simple differential filters for leftward and rightward movement
• Indeed, as we are concerned not with sign for a given direction, but rather magnitude, it suffices to consider the squared filter responses
4/)2( 22xtxt EEEEl ++= 4/)2( 22
xtxt EEEEr +−=
2/)(~
xt EEl += 2/)(~
xt EEr −=
![Page 27: Unit7 - cs.yorku.ca · 7 Orientation in visual space-time: Representation • Goal is to analyze spatiotemporal data according to its local orientation structure. – Consider orientation](https://reader034.vdocument.in/reader034/viewer/2022050414/5f8b2578188c234886699595/html5/thumbnails/27.jpg)
27
Orientation in visual space-time: Velocity
Comparison with optical flow constraint equation (OFCE) • In contrast, let us consider simple differential filters for leftward and rightward movement
• Indeed, as we are concerned not with sign for a given direction, but rather magnitude, it suffices to consider the squared filter responses
• To get a single (signed) measure of movement, we difference the two squared responses
txEElr −=−
2/)(~
xt EEl += 2/)(~
xt EEr −=
4/)2( 22xtxt EEEEl ++= 4/)2( 22
xtxt EEEEr +−=
![Page 28: Unit7 - cs.yorku.ca · 7 Orientation in visual space-time: Representation • Goal is to analyze spatiotemporal data according to its local orientation structure. – Consider orientation](https://reader034.vdocument.in/reader034/viewer/2022050414/5f8b2578188c234886699595/html5/thumbnails/28.jpg)
28
Orientation in visual space-time: Velocity
Comparison with optical flow constraint equation (OFCE) • In contrast, let us consider simple differential filters for leftward and rightward movement
• Indeed, as we are concerned not with sign for a given direction, but rather magnitude, it suffices to consider the squared filter responses
• To get a single (signed) measure of movement, we difference the two squared responses
• Finally, to avoid being biased by locally large values of image contrast, we divide through by the square of a first-order measure of local contrast
xt EEslr //)( −=−
2xEs =
2/)(~
xt EEl += 2/)(~
xt EEr −=
4/)2( 22xtxt EEEEl ++= 4/)2( 22
xtxt EEEEr +−=
txEElr −=−
![Page 29: Unit7 - cs.yorku.ca · 7 Orientation in visual space-time: Representation • Goal is to analyze spatiotemporal data according to its local orientation structure. – Consider orientation](https://reader034.vdocument.in/reader034/viewer/2022050414/5f8b2578188c234886699595/html5/thumbnails/29.jpg)
29
Orientation in visual space-time: Velocity
Comparison with optical flow constraint equation (OFCE) • In contrast, let us consider simple differential filters for leftward and rightward movement
• Indeed, as we are concerned not with sign for a given direction, but rather magnitude, it suffices to consider the squared filter responses
• To get a single (signed) measure of movement, we difference the two squared responses
• Finally, to avoid being biased by locally large values of image contrast, we divide through by the square of a first-order measure of local contrast
• We now recognize that this computation of velocity is equivalent to that based on the OFCE
xttx EEuEuE /0 −=�=+
2/)(~
xt EEl += 2/)(~
xt EEr −=
4/)2( 22xtxt EEEEl ++= 4/)2( 22
xtxt EEEEr +−=
txEElr −=−
xt EEslr //)( −=−
2xEs =
![Page 30: Unit7 - cs.yorku.ca · 7 Orientation in visual space-time: Representation • Goal is to analyze spatiotemporal data according to its local orientation structure. – Consider orientation](https://reader034.vdocument.in/reader034/viewer/2022050414/5f8b2578188c234886699595/html5/thumbnails/30.jpg)
30
Orientation in visual space-time: Velocity
Conclusion• Oriented filters in visual
space-time support the recovery of image velocity, along a particular direction.
• A “bank” of such filters can be used to span direction and provide an approach to optical flow estimation.
velocity
Filterresponse
velocity
Filterresponse
velocity
Filterresponse
![Page 31: Unit7 - cs.yorku.ca · 7 Orientation in visual space-time: Representation • Goal is to analyze spatiotemporal data according to its local orientation structure. – Consider orientation](https://reader034.vdocument.in/reader034/viewer/2022050414/5f8b2578188c234886699595/html5/thumbnails/31.jpg)
31
Outline
• Introduction• Orientation in visual space-time• A representation for spatiotemporal patterns• Spatiotemporal boundaries• A framework for spatiotemporal analysis• Applications• Summary
![Page 32: Unit7 - cs.yorku.ca · 7 Orientation in visual space-time: Representation • Goal is to analyze spatiotemporal data according to its local orientation structure. – Consider orientation](https://reader034.vdocument.in/reader034/viewer/2022050414/5f8b2578188c234886699595/html5/thumbnails/32.jpg)
32
A representation for spatiotemporal patterns: Motivation
• When confronted with spatiotemporal data, an intelligent system can be overwhelmed by sheer quantity.
• An initial organization would be a key enabler for dealing effectively with data of this nature.
• The organization should afford distinctions that can guide subsequent processing.
• Distinctions that go beyond that of simple velocity.
![Page 33: Unit7 - cs.yorku.ca · 7 Orientation in visual space-time: Representation • Goal is to analyze spatiotemporal data according to its local orientation structure. – Consider orientation](https://reader034.vdocument.in/reader034/viewer/2022050414/5f8b2578188c234886699595/html5/thumbnails/33.jpg)
33
A representation for spatiotemporal patterns: General approach
• Parse stream of spatiotemporal data into primitive, yet semantically meaningful categories at the earliest stages of processing.
• Make distinctions along the following lines– What is moving and what is stationary?– Are the moving objects behaving coherently?– How much of the variance in the data is due to temporal brightness
change?– Which portions of the data are simply too unstructured to support
further analysis?
![Page 34: Unit7 - cs.yorku.ca · 7 Orientation in visual space-time: Representation • Goal is to analyze spatiotemporal data according to its local orientation structure. – Consider orientation](https://reader034.vdocument.in/reader034/viewer/2022050414/5f8b2578188c234886699595/html5/thumbnails/34.jpg)
34
A representation for spatiotemporal patterns: General approach
• Integrate information across both the spatial and temporal dimensions.
• Build on analysis of local orientation.– The simplest non-trivial characterization of local geometric
structure.
![Page 35: Unit7 - cs.yorku.ca · 7 Orientation in visual space-time: Representation • Goal is to analyze spatiotemporal data according to its local orientation structure. – Consider orientation](https://reader034.vdocument.in/reader034/viewer/2022050414/5f8b2578188c234886699595/html5/thumbnails/35.jpg)
35
A representation for spatiotemporal patterns: Primitive patterns
• Consider a spatiotemporal slice– As an observer views a uniform pattern
x
t
![Page 36: Unit7 - cs.yorku.ca · 7 Orientation in visual space-time: Representation • Goal is to analyze spatiotemporal data according to its local orientation structure. – Consider orientation](https://reader034.vdocument.in/reader034/viewer/2022050414/5f8b2578188c234886699595/html5/thumbnails/36.jpg)
36
A representation for spatiotemporal patterns: Primitive patterns
• Consider a spatiotemporal slice– As an observer views a uniform pattern
x
t
![Page 37: Unit7 - cs.yorku.ca · 7 Orientation in visual space-time: Representation • Goal is to analyze spatiotemporal data according to its local orientation structure. – Consider orientation](https://reader034.vdocument.in/reader034/viewer/2022050414/5f8b2578188c234886699595/html5/thumbnails/37.jpg)
37
A representation for spatiotemporal patterns: Primitive patterns
• Consider a spatiotemporal slice– As an observer views a uniform pattern
x
t
![Page 38: Unit7 - cs.yorku.ca · 7 Orientation in visual space-time: Representation • Goal is to analyze spatiotemporal data according to its local orientation structure. – Consider orientation](https://reader034.vdocument.in/reader034/viewer/2022050414/5f8b2578188c234886699595/html5/thumbnails/38.jpg)
38
A representation for spatiotemporal patterns: Primitive patterns
• Consider a spatiotemporal slice– As an observer views a uniform pattern
x
t
![Page 39: Unit7 - cs.yorku.ca · 7 Orientation in visual space-time: Representation • Goal is to analyze spatiotemporal data according to its local orientation structure. – Consider orientation](https://reader034.vdocument.in/reader034/viewer/2022050414/5f8b2578188c234886699595/html5/thumbnails/39.jpg)
39
A representation for spatiotemporal patterns: Primitive patterns
• Consider a spatiotemporal slice– As an observer views a uniform pattern
x
t
![Page 40: Unit7 - cs.yorku.ca · 7 Orientation in visual space-time: Representation • Goal is to analyze spatiotemporal data according to its local orientation structure. – Consider orientation](https://reader034.vdocument.in/reader034/viewer/2022050414/5f8b2578188c234886699595/html5/thumbnails/40.jpg)
40
A representation for spatiotemporal patterns: Primitive patterns
• Consider a spatiotemporal slice– As an observer views a spatially structured,
temporally static patternx
t
![Page 41: Unit7 - cs.yorku.ca · 7 Orientation in visual space-time: Representation • Goal is to analyze spatiotemporal data according to its local orientation structure. – Consider orientation](https://reader034.vdocument.in/reader034/viewer/2022050414/5f8b2578188c234886699595/html5/thumbnails/41.jpg)
41
A representation for spatiotemporal patterns: Primitive patterns
• Consider a spatiotemporal slice– As an observer views a spatially structured,
temporally static patternx
t
![Page 42: Unit7 - cs.yorku.ca · 7 Orientation in visual space-time: Representation • Goal is to analyze spatiotemporal data according to its local orientation structure. – Consider orientation](https://reader034.vdocument.in/reader034/viewer/2022050414/5f8b2578188c234886699595/html5/thumbnails/42.jpg)
42
A representation for spatiotemporal patterns: Primitive patterns
• Consider a spatiotemporal slice– As an observer views a spatially structured,
temporally static patternx
t
![Page 43: Unit7 - cs.yorku.ca · 7 Orientation in visual space-time: Representation • Goal is to analyze spatiotemporal data according to its local orientation structure. – Consider orientation](https://reader034.vdocument.in/reader034/viewer/2022050414/5f8b2578188c234886699595/html5/thumbnails/43.jpg)
43
A representation for spatiotemporal patterns: Primitive patterns
• Consider a spatiotemporal slice– As an observer views a spatially structured,
temporally static patternx
t
![Page 44: Unit7 - cs.yorku.ca · 7 Orientation in visual space-time: Representation • Goal is to analyze spatiotemporal data according to its local orientation structure. – Consider orientation](https://reader034.vdocument.in/reader034/viewer/2022050414/5f8b2578188c234886699595/html5/thumbnails/44.jpg)
44
A representation for spatiotemporal patterns: Primitive patterns
• Consider a spatiotemporal slice– As an observer views a spatially structured,
temporally static patternx
t
![Page 45: Unit7 - cs.yorku.ca · 7 Orientation in visual space-time: Representation • Goal is to analyze spatiotemporal data according to its local orientation structure. – Consider orientation](https://reader034.vdocument.in/reader034/viewer/2022050414/5f8b2578188c234886699595/html5/thumbnails/45.jpg)
45
A representation for spatiotemporal patterns: Primitive patterns
• Consider a spatiotemporal slice– As an observer views a spatially uniform,
temporally flickering patternx
t
![Page 46: Unit7 - cs.yorku.ca · 7 Orientation in visual space-time: Representation • Goal is to analyze spatiotemporal data according to its local orientation structure. – Consider orientation](https://reader034.vdocument.in/reader034/viewer/2022050414/5f8b2578188c234886699595/html5/thumbnails/46.jpg)
46
A representation for spatiotemporal patterns: Primitive patterns
• Consider a spatiotemporal slice– As an observer views a spatially uniform,
temporally flickering patternx
t
![Page 47: Unit7 - cs.yorku.ca · 7 Orientation in visual space-time: Representation • Goal is to analyze spatiotemporal data according to its local orientation structure. – Consider orientation](https://reader034.vdocument.in/reader034/viewer/2022050414/5f8b2578188c234886699595/html5/thumbnails/47.jpg)
47
A representation for spatiotemporal patterns: Primitive patterns
• Consider a spatiotemporal slice– As an observer views a spatially uniform,
temporally flickering patternx
t
![Page 48: Unit7 - cs.yorku.ca · 7 Orientation in visual space-time: Representation • Goal is to analyze spatiotemporal data according to its local orientation structure. – Consider orientation](https://reader034.vdocument.in/reader034/viewer/2022050414/5f8b2578188c234886699595/html5/thumbnails/48.jpg)
48
A representation for spatiotemporal patterns: Primitive patterns
• Consider a spatiotemporal slice– As an observer views a spatially uniform,
temporally flickering patternx
t
![Page 49: Unit7 - cs.yorku.ca · 7 Orientation in visual space-time: Representation • Goal is to analyze spatiotemporal data according to its local orientation structure. – Consider orientation](https://reader034.vdocument.in/reader034/viewer/2022050414/5f8b2578188c234886699595/html5/thumbnails/49.jpg)
49
A representation for spatiotemporal patterns: Primitive patterns
• Consider a spatiotemporal slice– As an observer views a spatially uniform,
temporally flickering patternx
t
![Page 50: Unit7 - cs.yorku.ca · 7 Orientation in visual space-time: Representation • Goal is to analyze spatiotemporal data according to its local orientation structure. – Consider orientation](https://reader034.vdocument.in/reader034/viewer/2022050414/5f8b2578188c234886699595/html5/thumbnails/50.jpg)
50
A representation for spatiotemporal patterns: Primitive patterns
• Consider a spatiotemporal slice– As an observer views a spatially structured,
temporally moving patternx
t
![Page 51: Unit7 - cs.yorku.ca · 7 Orientation in visual space-time: Representation • Goal is to analyze spatiotemporal data according to its local orientation structure. – Consider orientation](https://reader034.vdocument.in/reader034/viewer/2022050414/5f8b2578188c234886699595/html5/thumbnails/51.jpg)
51
A representation for spatiotemporal patterns: Primitive patterns
• Consider a spatiotemporal slice– As an observer views a spatially structured,
temporally moving patternx
t
![Page 52: Unit7 - cs.yorku.ca · 7 Orientation in visual space-time: Representation • Goal is to analyze spatiotemporal data according to its local orientation structure. – Consider orientation](https://reader034.vdocument.in/reader034/viewer/2022050414/5f8b2578188c234886699595/html5/thumbnails/52.jpg)
52
A representation for spatiotemporal patterns: Primitive patterns
• Consider a spatiotemporal slice– As an observer views a spatially structured,
temporally moving patternx
t
![Page 53: Unit7 - cs.yorku.ca · 7 Orientation in visual space-time: Representation • Goal is to analyze spatiotemporal data according to its local orientation structure. – Consider orientation](https://reader034.vdocument.in/reader034/viewer/2022050414/5f8b2578188c234886699595/html5/thumbnails/53.jpg)
53
A representation for spatiotemporal patterns: Primitive patterns
• Consider a spatiotemporal slice– As an observer views a spatially structured,
temporally moving patternx
t
![Page 54: Unit7 - cs.yorku.ca · 7 Orientation in visual space-time: Representation • Goal is to analyze spatiotemporal data according to its local orientation structure. – Consider orientation](https://reader034.vdocument.in/reader034/viewer/2022050414/5f8b2578188c234886699595/html5/thumbnails/54.jpg)
54
A representation for spatiotemporal patterns: Primitive patterns
• Consider a spatiotemporal slice– As an observer views a spatially structured,
temporally moving patternx
t
![Page 55: Unit7 - cs.yorku.ca · 7 Orientation in visual space-time: Representation • Goal is to analyze spatiotemporal data according to its local orientation structure. – Consider orientation](https://reader034.vdocument.in/reader034/viewer/2022050414/5f8b2578188c234886699595/html5/thumbnails/55.jpg)
55
A representation for spatiotemporal patterns: Primitive patterns
• Consider a spatiotemporal slice– As an observer views a spatially structured,
temporally moving pattern, with > 1 motionx
t
![Page 56: Unit7 - cs.yorku.ca · 7 Orientation in visual space-time: Representation • Goal is to analyze spatiotemporal data according to its local orientation structure. – Consider orientation](https://reader034.vdocument.in/reader034/viewer/2022050414/5f8b2578188c234886699595/html5/thumbnails/56.jpg)
56
A representation for spatiotemporal patterns: Primitive patterns
• Consider a spatiotemporal slice– As an observer views a spatially structured,
temporally moving pattern, with > 1 motionx
t
![Page 57: Unit7 - cs.yorku.ca · 7 Orientation in visual space-time: Representation • Goal is to analyze spatiotemporal data according to its local orientation structure. – Consider orientation](https://reader034.vdocument.in/reader034/viewer/2022050414/5f8b2578188c234886699595/html5/thumbnails/57.jpg)
57
A representation for spatiotemporal patterns: Primitive patterns
• Consider a spatiotemporal slice– As an observer views a spatially structured,
temporally moving pattern, with > 1 motionx
t
![Page 58: Unit7 - cs.yorku.ca · 7 Orientation in visual space-time: Representation • Goal is to analyze spatiotemporal data according to its local orientation structure. – Consider orientation](https://reader034.vdocument.in/reader034/viewer/2022050414/5f8b2578188c234886699595/html5/thumbnails/58.jpg)
58
A representation for spatiotemporal patterns: Primitive patterns
• Consider a spatiotemporal slice– As an observer views a spatially structured,
temporally moving pattern, with > 1 motionx
t
![Page 59: Unit7 - cs.yorku.ca · 7 Orientation in visual space-time: Representation • Goal is to analyze spatiotemporal data according to its local orientation structure. – Consider orientation](https://reader034.vdocument.in/reader034/viewer/2022050414/5f8b2578188c234886699595/html5/thumbnails/59.jpg)
59
A representation for spatiotemporal patterns: Primitive patterns
• Consider a spatiotemporal slice– As an observer views a spatially structured,
temporally moving pattern, with > 1 motionx
t
![Page 60: Unit7 - cs.yorku.ca · 7 Orientation in visual space-time: Representation • Goal is to analyze spatiotemporal data according to its local orientation structure. – Consider orientation](https://reader034.vdocument.in/reader034/viewer/2022050414/5f8b2578188c234886699595/html5/thumbnails/60.jpg)
60
A representation for spatiotemporal patterns: Primitive patterns
• Consider a spatiotemporal slice– As an observer views a pattern, “randomly”
structured in space and time.x
t
![Page 61: Unit7 - cs.yorku.ca · 7 Orientation in visual space-time: Representation • Goal is to analyze spatiotemporal data according to its local orientation structure. – Consider orientation](https://reader034.vdocument.in/reader034/viewer/2022050414/5f8b2578188c234886699595/html5/thumbnails/61.jpg)
61
A representation for spatiotemporal patterns: Primitive patterns
• Consider a spatiotemporal slice– As an observer views a pattern, “randomly”
structured in space and time.x
t
![Page 62: Unit7 - cs.yorku.ca · 7 Orientation in visual space-time: Representation • Goal is to analyze spatiotemporal data according to its local orientation structure. – Consider orientation](https://reader034.vdocument.in/reader034/viewer/2022050414/5f8b2578188c234886699595/html5/thumbnails/62.jpg)
62
A representation for spatiotemporal patterns: Primitive patterns
• Consider a spatiotemporal slice– As an observer views a pattern, “randomly”
structured in space and time.x
t
![Page 63: Unit7 - cs.yorku.ca · 7 Orientation in visual space-time: Representation • Goal is to analyze spatiotemporal data according to its local orientation structure. – Consider orientation](https://reader034.vdocument.in/reader034/viewer/2022050414/5f8b2578188c234886699595/html5/thumbnails/63.jpg)
63
A representation for spatiotemporal patterns: Primitive patterns
• Consider a spatiotemporal slice– As an observer views a pattern, “randomly”
structured in space and time.x
t
![Page 64: Unit7 - cs.yorku.ca · 7 Orientation in visual space-time: Representation • Goal is to analyze spatiotemporal data according to its local orientation structure. – Consider orientation](https://reader034.vdocument.in/reader034/viewer/2022050414/5f8b2578188c234886699595/html5/thumbnails/64.jpg)
64
A representation for spatiotemporal patterns: Primitive patterns
• Consider a spatiotemporal slice– As an observer views a pattern, “randomly”
structured in space and time.x
t
![Page 65: Unit7 - cs.yorku.ca · 7 Orientation in visual space-time: Representation • Goal is to analyze spatiotemporal data according to its local orientation structure. – Consider orientation](https://reader034.vdocument.in/reader034/viewer/2022050414/5f8b2578188c234886699595/html5/thumbnails/65.jpg)
65
A representation for spatiotemporal patterns: Primitives
![Page 66: Unit7 - cs.yorku.ca · 7 Orientation in visual space-time: Representation • Goal is to analyze spatiotemporal data according to its local orientation structure. – Consider orientation](https://reader034.vdocument.in/reader034/viewer/2022050414/5f8b2578188c234886699595/html5/thumbnails/66.jpg)
66
Orientation in visual space-time: Filtering
• Apply filters tuned to 4 different orientations in both x-t and y-t domains.
• In general, might consider additional directions.
• Filter specifics: – Oriented bandpass filters in
spatiotemporal slice. – Lowpass filter in orthogonal
spatial dimension.– Pointwise squared to yield
local “oriented energy”.
x
t
xs
l
r
xf
![Page 67: Unit7 - cs.yorku.ca · 7 Orientation in visual space-time: Representation • Goal is to analyze spatiotemporal data according to its local orientation structure. – Consider orientation](https://reader034.vdocument.in/reader034/viewer/2022050414/5f8b2578188c234886699595/html5/thumbnails/67.jpg)
67
Orientation in visual space-time: Normalization
• For any given orientation, the filter response is a joint function of– orientation – contrast
• Normalization yields purer measure of orientation
with a small bias added for stability.
• Similarly for , , and their y-t counterparts.
ε
xsl xf
ε++++=
),,(),,(),,(),,(),,(
),,(tyxftyxstyxltyxr
tyxrtyxR
xx
![Page 68: Unit7 - cs.yorku.ca · 7 Orientation in visual space-time: Representation • Goal is to analyze spatiotemporal data according to its local orientation structure. – Consider orientation](https://reader034.vdocument.in/reader034/viewer/2022050414/5f8b2578188c234886699595/html5/thumbnails/68.jpg)
68
A representation for spatiotemporal patterns: Opponency and summation
• The R and L (U and D) components are ambiguous WRT coherent and incoherent motion.
R
L
![Page 69: Unit7 - cs.yorku.ca · 7 Orientation in visual space-time: Representation • Goal is to analyze spatiotemporal data according to its local orientation structure. – Consider orientation](https://reader034.vdocument.in/reader034/viewer/2022050414/5f8b2578188c234886699595/html5/thumbnails/69.jpg)
69
A representation for spatiotemporal patterns: Opponency and summation
• The R and L (U and D) components are ambiguous WRT coherent and incoherent motion.
• Solution: Combine via– opponency R-L (U-D)– summation R+L (U+D)
• Geometrically a rotation of coordinate axes.
R
L
R+L
R-L
![Page 70: Unit7 - cs.yorku.ca · 7 Orientation in visual space-time: Representation • Goal is to analyze spatiotemporal data according to its local orientation structure. – Consider orientation](https://reader034.vdocument.in/reader034/viewer/2022050414/5f8b2578188c234886699595/html5/thumbnails/70.jpg)
70
A representation for spatiotemporal patterns: Spatiotemporal representation
• Proposal: A four band representation for both the x-t and y-t dimensions.
x-t y-t
x
x
F
S
LR
LR
+− ||
y
y
F
S
DU
DU
+− ||
![Page 71: Unit7 - cs.yorku.ca · 7 Orientation in visual space-time: Representation • Goal is to analyze spatiotemporal data according to its local orientation structure. – Consider orientation](https://reader034.vdocument.in/reader034/viewer/2022050414/5f8b2578188c234886699595/html5/thumbnails/71.jpg)
71
A representation for spatiotemporal patterns: Primitives projected on representation
![Page 72: Unit7 - cs.yorku.ca · 7 Orientation in visual space-time: Representation • Goal is to analyze spatiotemporal data according to its local orientation structure. – Consider orientation](https://reader034.vdocument.in/reader034/viewer/2022050414/5f8b2578188c234886699595/html5/thumbnails/72.jpg)
72
A representation for spatiotemporal patterns: Examples I
• A natural image sequence of each proposed class was acquired, (x,y,t) = (64,64,40).– Unstructured: featureless sky– static: motionless tree– flicker: smooth surface illuminated by lightning flashes– coherent motion: field of flowers under camera motion– incoherent motion: overlapped legs in complex motion– scintillation: rain striking a puddle
• Each sequence brought under proposed representation.
![Page 73: Unit7 - cs.yorku.ca · 7 Orientation in visual space-time: Representation • Goal is to analyze spatiotemporal data according to its local orientation structure. – Consider orientation](https://reader034.vdocument.in/reader034/viewer/2022050414/5f8b2578188c234886699595/html5/thumbnails/73.jpg)
73
Unstructured
![Page 74: Unit7 - cs.yorku.ca · 7 Orientation in visual space-time: Representation • Goal is to analyze spatiotemporal data according to its local orientation structure. – Consider orientation](https://reader034.vdocument.in/reader034/viewer/2022050414/5f8b2578188c234886699595/html5/thumbnails/74.jpg)
74
Static
![Page 75: Unit7 - cs.yorku.ca · 7 Orientation in visual space-time: Representation • Goal is to analyze spatiotemporal data according to its local orientation structure. – Consider orientation](https://reader034.vdocument.in/reader034/viewer/2022050414/5f8b2578188c234886699595/html5/thumbnails/75.jpg)
75
Flicker
![Page 76: Unit7 - cs.yorku.ca · 7 Orientation in visual space-time: Representation • Goal is to analyze spatiotemporal data according to its local orientation structure. – Consider orientation](https://reader034.vdocument.in/reader034/viewer/2022050414/5f8b2578188c234886699595/html5/thumbnails/76.jpg)
76
Coherent motion
![Page 77: Unit7 - cs.yorku.ca · 7 Orientation in visual space-time: Representation • Goal is to analyze spatiotemporal data according to its local orientation structure. – Consider orientation](https://reader034.vdocument.in/reader034/viewer/2022050414/5f8b2578188c234886699595/html5/thumbnails/77.jpg)
77
Incoherent motion
![Page 78: Unit7 - cs.yorku.ca · 7 Orientation in visual space-time: Representation • Goal is to analyze spatiotemporal data according to its local orientation structure. – Consider orientation](https://reader034.vdocument.in/reader034/viewer/2022050414/5f8b2578188c234886699595/html5/thumbnails/78.jpg)
78
Scintillation
![Page 79: Unit7 - cs.yorku.ca · 7 Orientation in visual space-time: Representation • Goal is to analyze spatiotemporal data according to its local orientation structure. – Consider orientation](https://reader034.vdocument.in/reader034/viewer/2022050414/5f8b2578188c234886699595/html5/thumbnails/79.jpg)
79
A representation for spatiotemporal patterns: Results x-t
![Page 80: Unit7 - cs.yorku.ca · 7 Orientation in visual space-time: Representation • Goal is to analyze spatiotemporal data according to its local orientation structure. – Consider orientation](https://reader034.vdocument.in/reader034/viewer/2022050414/5f8b2578188c234886699595/html5/thumbnails/80.jpg)
80
A representation for spatiotemporal patterns: Results y-t
![Page 81: Unit7 - cs.yorku.ca · 7 Orientation in visual space-time: Representation • Goal is to analyze spatiotemporal data according to its local orientation structure. – Consider orientation](https://reader034.vdocument.in/reader034/viewer/2022050414/5f8b2578188c234886699595/html5/thumbnails/81.jpg)
81
A representation for spatiotemporal patterns: Remarks
• Have described a representation for distinguishing primitive spatiotemporal patterns.
• The representation makes use of oriented bandpassimage decomposition.
• Initial empirical results support the hypothesis that the proposed representation can afford the desired distinctions.
![Page 82: Unit7 - cs.yorku.ca · 7 Orientation in visual space-time: Representation • Goal is to analyze spatiotemporal data according to its local orientation structure. – Consider orientation](https://reader034.vdocument.in/reader034/viewer/2022050414/5f8b2578188c234886699595/html5/thumbnails/82.jpg)
82
Outline
• Introduction• Orientation in visual space-time• A representation for spatiotemporal patterns• Spatiotemporal boundaries• A framework for spatiotemporal analysis• Applications• Summary
![Page 83: Unit7 - cs.yorku.ca · 7 Orientation in visual space-time: Representation • Goal is to analyze spatiotemporal data according to its local orientation structure. – Consider orientation](https://reader034.vdocument.in/reader034/viewer/2022050414/5f8b2578188c234886699595/html5/thumbnails/83.jpg)
83
Boundaries: Motivation
• Detection and localization of spatiotemporal boundaries is an important aspect of chunking information into meaningful pieces.
• Complimentary to the area-based analysis considered so far.
• Differential operators matched to the juxtaposition of contrasting spatiotemporal structure can be assembled from the primitive filter responses, S, F, R-L, R+L, etc.
![Page 84: Unit7 - cs.yorku.ca · 7 Orientation in visual space-time: Representation • Goal is to analyze spatiotemporal data according to its local orientation structure. – Consider orientation](https://reader034.vdocument.in/reader034/viewer/2022050414/5f8b2578188c234886699595/html5/thumbnails/84.jpg)
84
Boundaries: Coherent motion
• Boundaries in coherent motion discriminate foreground/background.
• Coherent motion related to opponent bands R-Land U-D.
• Combine a spatial Laplacian with opponent filtering to yield double opponent operators.
-R+L
+R-L
![Page 85: Unit7 - cs.yorku.ca · 7 Orientation in visual space-time: Representation • Goal is to analyze spatiotemporal data according to its local orientation structure. – Consider orientation](https://reader034.vdocument.in/reader034/viewer/2022050414/5f8b2578188c234886699595/html5/thumbnails/85.jpg)
85
Boundaries: Coherent motion
• Boundaries in coherent motion discriminate foreground/background.
• Coherent motion related to opponent bands R-Land U-D.
• Combine a spatial Laplacian with opponent filtering to yield double opponent operators.
-U+D
+U-D
![Page 86: Unit7 - cs.yorku.ca · 7 Orientation in visual space-time: Representation • Goal is to analyze spatiotemporal data according to its local orientation structure. – Consider orientation](https://reader034.vdocument.in/reader034/viewer/2022050414/5f8b2578188c234886699595/html5/thumbnails/86.jpg)
86
Boundaries: Signature
• Zero-crossings in the double-opponent motion operator output indicate coherent motion boundaries.
• Slope magnitude taken as strength of boundary signal.
• Sum signals from x-t and y-t dimensions.
LR −
( )LRDx −
( )LRDxx −
![Page 87: Unit7 - cs.yorku.ca · 7 Orientation in visual space-time: Representation • Goal is to analyze spatiotemporal data according to its local orientation structure. – Consider orientation](https://reader034.vdocument.in/reader034/viewer/2022050414/5f8b2578188c234886699595/html5/thumbnails/87.jpg)
87
Boundaries: Examples II
• Two image sequences depicting boundaries of coherent motion, (x,y,t) = (256,256,16).– random dot cinematogram: left and right sides of display in opposite
horizontal motion.– natural image sequence: aerial view of tree canopy with movement
relative to undergrowth due to camera motion; homogeneous texture of vegetation obscures boundary in any one image.
• Each sequence processed by proposed method for indicating coherent motion boundaries.
![Page 88: Unit7 - cs.yorku.ca · 7 Orientation in visual space-time: Representation • Goal is to analyze spatiotemporal data according to its local orientation structure. – Consider orientation](https://reader034.vdocument.in/reader034/viewer/2022050414/5f8b2578188c234886699595/html5/thumbnails/88.jpg)
88
Boundaries: Results
Frame of input sequence. Motion boundary signal intensity.
![Page 89: Unit7 - cs.yorku.ca · 7 Orientation in visual space-time: Representation • Goal is to analyze spatiotemporal data according to its local orientation structure. – Consider orientation](https://reader034.vdocument.in/reader034/viewer/2022050414/5f8b2578188c234886699595/html5/thumbnails/89.jpg)
89
Boundaries: Results
Motion boundary signal intensity.Frame of input sequence.
![Page 90: Unit7 - cs.yorku.ca · 7 Orientation in visual space-time: Representation • Goal is to analyze spatiotemporal data according to its local orientation structure. – Consider orientation](https://reader034.vdocument.in/reader034/viewer/2022050414/5f8b2578188c234886699595/html5/thumbnails/90.jpg)
90
Outline
• Introduction• Orientation in visual space-time• A representation for spatiotemporal patterns• Spatiotemporal boundaries• A framework for spatiotemporal analysis• Applications• Summary
![Page 91: Unit7 - cs.yorku.ca · 7 Orientation in visual space-time: Representation • Goal is to analyze spatiotemporal data according to its local orientation structure. – Consider orientation](https://reader034.vdocument.in/reader034/viewer/2022050414/5f8b2578188c234886699595/html5/thumbnails/91.jpg)
91
Framework: Local orientation
ωx
ωy
ωt
unstructured flicker static coherent motion scintillation
superimposed motion
x
y
t
Appeal to orientation is not arbitrary• The local orientation (or lack thereof) of a pattern is one of its
most salient characteristics.• Geometrically, orientation captures the local first-order
correlation structure of a pattern, alternatively the local tangent.• Provides a formal prerequisite to analysis of higher-contact
constructions (e.g., curvature)• For vision, local spatiotemporal orientation can have additional
interpretations.– Image velocity is manifest as spatiotemporal orientation.– And more...
![Page 92: Unit7 - cs.yorku.ca · 7 Orientation in visual space-time: Representation • Goal is to analyze spatiotemporal data according to its local orientation structure. – Consider orientation](https://reader034.vdocument.in/reader034/viewer/2022050414/5f8b2578188c234886699595/html5/thumbnails/92.jpg)
92
Framework: Filtering
Orientation selectivity• Goal is to analyze spatiotemporal data,
I(x, y, t), according to its local orientation structure.
• Choose a representation with multiple bands each tuned for certain orientations, , and scales in 3D (x, y, t).
• Filter specifics: – 3D Gaussian second derivatives, – Corresponding Hilbert transforms, – Rectified and summed in quadrature
pairs to yield local “oriented energy”.
[ ] [ ]22
22 ),,(*),,(*),,( tyxIHtyxIGtyxE θθθ +=
θ
.2θG.2θH
input spatiotemporal volume
oriented filter bank
output oriented energy volumes
![Page 93: Unit7 - cs.yorku.ca · 7 Orientation in visual space-time: Representation • Goal is to analyze spatiotemporal data according to its local orientation structure. – Consider orientation](https://reader034.vdocument.in/reader034/viewer/2022050414/5f8b2578188c234886699595/html5/thumbnails/93.jpg)
93
Framework: Architecture
oriented energysequences
![Page 94: Unit7 - cs.yorku.ca · 7 Orientation in visual space-time: Representation • Goal is to analyze spatiotemporal data according to its local orientation structure. – Consider orientation](https://reader034.vdocument.in/reader034/viewer/2022050414/5f8b2578188c234886699595/html5/thumbnails/94.jpg)
94
Framework: Example
y
x
time
spatiotemporal volumeoriented energy volume
t
xy
left right
flicker static
![Page 95: Unit7 - cs.yorku.ca · 7 Orientation in visual space-time: Representation • Goal is to analyze spatiotemporal data according to its local orientation structure. – Consider orientation](https://reader034.vdocument.in/reader034/viewer/2022050414/5f8b2578188c234886699595/html5/thumbnails/95.jpg)
95
Framework: Distributions of oriented energy
energy
Key ideas• Build a distributed representation (i.e., histogram) at each point that
measures the amount of energy for various kinds of spacetime oriented structures.
• Base subsequent analysis on the distribution of oriented energies across space and time.
![Page 96: Unit7 - cs.yorku.ca · 7 Orientation in visual space-time: Representation • Goal is to analyze spatiotemporal data according to its local orientation structure. – Consider orientation](https://reader034.vdocument.in/reader034/viewer/2022050414/5f8b2578188c234886699595/html5/thumbnails/96.jpg)
96
Outline
• Introduction• Orientation in visual space-time• A representation for spatiotemporal patterns• Spatiotemporal boundaries• A framework for spatiotemporal analysis• Applications• Summary
![Page 97: Unit7 - cs.yorku.ca · 7 Orientation in visual space-time: Representation • Goal is to analyze spatiotemporal data according to its local orientation structure. – Consider orientation](https://reader034.vdocument.in/reader034/viewer/2022050414/5f8b2578188c234886699595/html5/thumbnails/97.jpg)
97
Applications
Examples provided in lecture.
![Page 98: Unit7 - cs.yorku.ca · 7 Orientation in visual space-time: Representation • Goal is to analyze spatiotemporal data according to its local orientation structure. – Consider orientation](https://reader034.vdocument.in/reader034/viewer/2022050414/5f8b2578188c234886699595/html5/thumbnails/98.jpg)
98
Summary
• Introduction• Orientation in visual space-time• A representation for spatiotemporal patterns• Spatiotemporal boundaries• A framework for spatiotemporal analysis• Applications• Summary