computing motion from images
DESCRIPTION
Computing Motion from Images. Chapter 9 of S&S plus otherwork. General topics. Low level change detection Region tracking or matching over time Interpretation of motion MPEG compression Interpretation of scene changes in video Understanding human activites. - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: Computing Motion from Images](https://reader035.vdocument.in/reader035/viewer/2022062410/568159c9550346895dc71b77/html5/thumbnails/1.jpg)
Stockman MSU Fall 2009 1
Computing Motion from Images
Chapter 9 of S&S plus otherwork.
![Page 2: Computing Motion from Images](https://reader035.vdocument.in/reader035/viewer/2022062410/568159c9550346895dc71b77/html5/thumbnails/2.jpg)
Stockman MSU Fall 2009 2
General topics
Low level change detection Region tracking or matching over
time Interpretation of motion MPEG compression Interpretation of scene changes in
video Understanding human activites
![Page 3: Computing Motion from Images](https://reader035.vdocument.in/reader035/viewer/2022062410/568159c9550346895dc71b77/html5/thumbnails/3.jpg)
Stockman MSU Fall 2009 3
Motion important to human vision
![Page 4: Computing Motion from Images](https://reader035.vdocument.in/reader035/viewer/2022062410/568159c9550346895dc71b77/html5/thumbnails/4.jpg)
Stockman MSU Fall 2009 4
What’s moving: different cases
![Page 5: Computing Motion from Images](https://reader035.vdocument.in/reader035/viewer/2022062410/568159c9550346895dc71b77/html5/thumbnails/5.jpg)
Stockman MSU Fall 2009 5
Image subtraction
Simple method to remove unchanging background from
moving regions.
![Page 6: Computing Motion from Images](https://reader035.vdocument.in/reader035/viewer/2022062410/568159c9550346895dc71b77/html5/thumbnails/6.jpg)
Stockman MSU Fall 2009 6
Change detection for surveillance
![Page 7: Computing Motion from Images](https://reader035.vdocument.in/reader035/viewer/2022062410/568159c9550346895dc71b77/html5/thumbnails/7.jpg)
Stockman MSU Fall 2009 7
Change detection by image subtraction
![Page 8: Computing Motion from Images](https://reader035.vdocument.in/reader035/viewer/2022062410/568159c9550346895dc71b77/html5/thumbnails/8.jpg)
Stockman MSU Fall 2009 8
What to do with regions of change?
Discard small regions Discard regions of non interesting
features Keep track of regions with
interesting features Track in future frames from motion
plus component features
![Page 9: Computing Motion from Images](https://reader035.vdocument.in/reader035/viewer/2022062410/568159c9550346895dc71b77/html5/thumbnails/9.jpg)
Stockman MSU Fall 2009 9
Some effects of camera motion that can cause problems
![Page 10: Computing Motion from Images](https://reader035.vdocument.in/reader035/viewer/2022062410/568159c9550346895dc71b77/html5/thumbnails/10.jpg)
Stockman MSU Fall 2009 10
Motion field
![Page 11: Computing Motion from Images](https://reader035.vdocument.in/reader035/viewer/2022062410/568159c9550346895dc71b77/html5/thumbnails/11.jpg)
Stockman MSU Fall 2009 11
FOE and FOC
Will return to use the FOE or FOC or detection of panning to determine what the camera is doing in video tapes.
![Page 12: Computing Motion from Images](https://reader035.vdocument.in/reader035/viewer/2022062410/568159c9550346895dc71b77/html5/thumbnails/12.jpg)
Stockman MSU Fall 2009 12
Gaming using a camera to recognize the player’s motion
Decathlete game
![Page 13: Computing Motion from Images](https://reader035.vdocument.in/reader035/viewer/2022062410/568159c9550346895dc71b77/html5/thumbnails/13.jpg)
Stockman MSU Fall 2009 13
Decathlete game
Cheap camera replaces usual mouse for input
Running speed and jumping of the avatar is controlled by detected motion of the player’s hands.
![Page 14: Computing Motion from Images](https://reader035.vdocument.in/reader035/viewer/2022062410/568159c9550346895dc71b77/html5/thumbnails/14.jpg)
Stockman MSU Fall 2009 14
Motion detection input device
Running (hands)Jumping (hands)
![Page 15: Computing Motion from Images](https://reader035.vdocument.in/reader035/viewer/2022062410/568159c9550346895dc71b77/html5/thumbnails/15.jpg)
Stockman MSU Fall 2009 15
Motion analysis controls hurdling event (console)
•Top left shows video frame of player
•Middle left shows motion vectors from multiple frames
•Center shows jumping patterns
![Page 16: Computing Motion from Images](https://reader035.vdocument.in/reader035/viewer/2022062410/568159c9550346895dc71b77/html5/thumbnails/16.jpg)
Stockman MSU Fall 2009 16
Related work
Motion sensed by crude cameras Person dances/gestures in space System maps movement into
music Creative environment? Good exercise room?
![Page 17: Computing Motion from Images](https://reader035.vdocument.in/reader035/viewer/2022062410/568159c9550346895dc71b77/html5/thumbnails/17.jpg)
Stockman MSU Fall 2009 17
Computing motion vectors from corresponding “points”
High energy neighborhoods are used to define points for
matching
![Page 18: Computing Motion from Images](https://reader035.vdocument.in/reader035/viewer/2022062410/568159c9550346895dc71b77/html5/thumbnails/18.jpg)
Stockman MSU Fall 2009 18
Match points between frames
Such large motions are unusual. Most systems track small motions.
![Page 19: Computing Motion from Images](https://reader035.vdocument.in/reader035/viewer/2022062410/568159c9550346895dc71b77/html5/thumbnails/19.jpg)
Stockman MSU Fall 2009 19
Requirements for interest points
Match small neighborhood to small neighborhood. The previous “scene” contains several highly textured neighborhoods.
![Page 20: Computing Motion from Images](https://reader035.vdocument.in/reader035/viewer/2022062410/568159c9550346895dc71b77/html5/thumbnails/20.jpg)
Stockman MSU Fall 2009 20
Interest = minimum directional variance
Used by Hans Moravec in his robot stereo vision system.
Interest points were used for stereo matching.
![Page 21: Computing Motion from Images](https://reader035.vdocument.in/reader035/viewer/2022062410/568159c9550346895dc71b77/html5/thumbnails/21.jpg)
Stockman MSU Fall 2009 21
Detecting interest points in I1
![Page 22: Computing Motion from Images](https://reader035.vdocument.in/reader035/viewer/2022062410/568159c9550346895dc71b77/html5/thumbnails/22.jpg)
Stockman MSU Fall 2009 22
Match points from I1 in I2
![Page 23: Computing Motion from Images](https://reader035.vdocument.in/reader035/viewer/2022062410/568159c9550346895dc71b77/html5/thumbnails/23.jpg)
Stockman MSU Fall 2009 23
Search for best match of point P1 in nearby window of I2
For both motion and stereo, we have some constraints on where to search for a matching interest point.
![Page 24: Computing Motion from Images](https://reader035.vdocument.in/reader035/viewer/2022062410/568159c9550346895dc71b77/html5/thumbnails/24.jpg)
Stockman MSU Fall 2009 24
Motion vectors clustered to show 3 coherent regions
All motion vectors are clustered into 3 groups of similar vectors showing motion of 3 independent objects. (Dina Eldin)
Motion coherence: points of same object tend to move in the same way
![Page 25: Computing Motion from Images](https://reader035.vdocument.in/reader035/viewer/2022062410/568159c9550346895dc71b77/html5/thumbnails/25.jpg)
Stockman MSU Fall 2009 25
Two frames of aerial imagery
Video frame N and N+1 shows slight movement: most pixels are same, just in different locations.
![Page 26: Computing Motion from Images](https://reader035.vdocument.in/reader035/viewer/2022062410/568159c9550346895dc71b77/html5/thumbnails/26.jpg)
Stockman MSU Fall 2009 26
Can code frame N+d with displacments relative to frame N
for each 16 x 16 block in the 2nd image
find a closely matching block in the 1st image
replace the 16x16 intensities by the location in the 1st image (dX, dY)
256 bytes replaced by 2 bytes! (If blocks differ too much, encode the
differences to be added.)
![Page 27: Computing Motion from Images](https://reader035.vdocument.in/reader035/viewer/2022062410/568159c9550346895dc71b77/html5/thumbnails/27.jpg)
Stockman MSU Fall 2009 27
Frame approximation
Left is original video frame N+1. Right is set of best image blocks taken from frame N. (Work of Dina Eldin)
![Page 28: Computing Motion from Images](https://reader035.vdocument.in/reader035/viewer/2022062410/568159c9550346895dc71b77/html5/thumbnails/28.jpg)
Stockman MSU Fall 2009 28
Best matching blocks between video frames N+1 to N (motion vectors)
The bulk of the vectors show the true motion of the airplane taking the pictures. The long vectors are incorrect motion vectors, but they do work well for compression of image I2!
Best matches from 2nd to first image shown as vectors overlaid on the 2nd image. (Work by Dina Eldin.)
![Page 29: Computing Motion from Images](https://reader035.vdocument.in/reader035/viewer/2022062410/568159c9550346895dc71b77/html5/thumbnails/29.jpg)
Stockman MSU Fall 2009 29
Motion coherence provides redundancy for compression
MPEG “motion compensation” represents motion of 16x16 pixels blocks, NOT objects
![Page 30: Computing Motion from Images](https://reader035.vdocument.in/reader035/viewer/2022062410/568159c9550346895dc71b77/html5/thumbnails/30.jpg)
Stockman MSU Fall 2009 30
MPEG represents blocks that move by the motion vector
![Page 31: Computing Motion from Images](https://reader035.vdocument.in/reader035/viewer/2022062410/568159c9550346895dc71b77/html5/thumbnails/31.jpg)
Stockman MSU Fall 2009 31
MPEG has ‘I’, ‘P’, and ‘B’ frames
![Page 32: Computing Motion from Images](https://reader035.vdocument.in/reader035/viewer/2022062410/568159c9550346895dc71b77/html5/thumbnails/32.jpg)
Stockman MSU Fall 2009 32
Computing Image Flow
![Page 33: Computing Motion from Images](https://reader035.vdocument.in/reader035/viewer/2022062410/568159c9550346895dc71b77/html5/thumbnails/33.jpg)
Stockman MSU Fall 2009 33
![Page 34: Computing Motion from Images](https://reader035.vdocument.in/reader035/viewer/2022062410/568159c9550346895dc71b77/html5/thumbnails/34.jpg)
Stockman MSU Fall 2009 34
Assumptions
![Page 35: Computing Motion from Images](https://reader035.vdocument.in/reader035/viewer/2022062410/568159c9550346895dc71b77/html5/thumbnails/35.jpg)
Stockman MSU Fall 2009 35
IMAGE
FLOW
EQUATION
1 of 2
![Page 36: Computing Motion from Images](https://reader035.vdocument.in/reader035/viewer/2022062410/568159c9550346895dc71b77/html5/thumbnails/36.jpg)
Stockman MSU Fall 2009 36
Image flow equation 2 of 2
![Page 37: Computing Motion from Images](https://reader035.vdocument.in/reader035/viewer/2022062410/568159c9550346895dc71b77/html5/thumbnails/37.jpg)
Stockman MSU Fall 2009 37
Aperture problem
![Page 38: Computing Motion from Images](https://reader035.vdocument.in/reader035/viewer/2022062410/568159c9550346895dc71b77/html5/thumbnails/38.jpg)
Stockman MSU Fall 2009 38
Solving flow by propagation of constraints
![Page 39: Computing Motion from Images](https://reader035.vdocument.in/reader035/viewer/2022062410/568159c9550346895dc71b77/html5/thumbnails/39.jpg)
Stockman MSU Fall 2009 39
Info at corner constrains the flow along both edges
Solve constraints using contraint propagation or differential equation with boundary conditions.
![Page 40: Computing Motion from Images](https://reader035.vdocument.in/reader035/viewer/2022062410/568159c9550346895dc71b77/html5/thumbnails/40.jpg)
Stockman MSU Fall 2009 40
Tracking several objects
Use assumptions of physics to compute multiple smooth
paths.(work of Sethi and R. Jain)
![Page 41: Computing Motion from Images](https://reader035.vdocument.in/reader035/viewer/2022062410/568159c9550346895dc71b77/html5/thumbnails/41.jpg)
Stockman MSU Fall 2009 41
![Page 42: Computing Motion from Images](https://reader035.vdocument.in/reader035/viewer/2022062410/568159c9550346895dc71b77/html5/thumbnails/42.jpg)
Stockman MSU Fall 2009 42
Tracking in images over time
![Page 43: Computing Motion from Images](https://reader035.vdocument.in/reader035/viewer/2022062410/568159c9550346895dc71b77/html5/thumbnails/43.jpg)
Stockman MSU Fall 2009 43
General constraints from physics
![Page 44: Computing Motion from Images](https://reader035.vdocument.in/reader035/viewer/2022062410/568159c9550346895dc71b77/html5/thumbnails/44.jpg)
Stockman MSU Fall 2009 44
Other possible constraints
Background statistics stable Object color/texture/shape might
change slowly over frames Might have knowledge of objects
under survielance Objects appear/disappear at
boundary of the frame
![Page 45: Computing Motion from Images](https://reader035.vdocument.in/reader035/viewer/2022062410/568159c9550346895dc71b77/html5/thumbnails/45.jpg)
Stockman MSU Fall 2009 45
![Page 46: Computing Motion from Images](https://reader035.vdocument.in/reader035/viewer/2022062410/568159c9550346895dc71b77/html5/thumbnails/46.jpg)
Stockman MSU Fall 2009 46
Sethi-Jain algorithm
![Page 47: Computing Motion from Images](https://reader035.vdocument.in/reader035/viewer/2022062410/568159c9550346895dc71b77/html5/thumbnails/47.jpg)
Stockman MSU Fall 2009 47
![Page 48: Computing Motion from Images](https://reader035.vdocument.in/reader035/viewer/2022062410/568159c9550346895dc71b77/html5/thumbnails/48.jpg)
Stockman MSU Fall 2009 48
Total smoothness of m paths
![Page 49: Computing Motion from Images](https://reader035.vdocument.in/reader035/viewer/2022062410/568159c9550346895dc71b77/html5/thumbnails/49.jpg)
Stockman MSU Fall 2009 49
Greedy exchange algorithm
![Page 50: Computing Motion from Images](https://reader035.vdocument.in/reader035/viewer/2022062410/568159c9550346895dc71b77/html5/thumbnails/50.jpg)
Stockman MSU Fall 2009 50
Example data structure
Total smoothness for trajectories of Figure 9.14
![Page 51: Computing Motion from Images](https://reader035.vdocument.in/reader035/viewer/2022062410/568159c9550346895dc71b77/html5/thumbnails/51.jpg)
Stockman MSU Fall 2009 51
Example of domain specific tracking (Vera Bakic)
Tracking eyes and nose of PC user. System presents menu (top). User moves face to position cursor to a particular box (choice). System tracks face movement and moves cursor accordingly: user gets into feedback-control loop.
![Page 52: Computing Motion from Images](https://reader035.vdocument.in/reader035/viewer/2022062410/568159c9550346895dc71b77/html5/thumbnails/52.jpg)
Stockman MSU Fall 2009 52
Segmentation of videos/movies
Segment into scenes, shots, specific actions, etc.
![Page 53: Computing Motion from Images](https://reader035.vdocument.in/reader035/viewer/2022062410/568159c9550346895dc71b77/html5/thumbnails/53.jpg)
Stockman MSU Fall 2009 53
Types of changes in videos
![Page 54: Computing Motion from Images](https://reader035.vdocument.in/reader035/viewer/2022062410/568159c9550346895dc71b77/html5/thumbnails/54.jpg)
Stockman MSU Fall 2009 54
Anchor person scene at left
Street scene for news story
Scene break
From Zhang et al 1993
How do we compute the scene change?
![Page 55: Computing Motion from Images](https://reader035.vdocument.in/reader035/viewer/2022062410/568159c9550346895dc71b77/html5/thumbnails/55.jpg)
Stockman MSU Fall 2009 55
Histograms of frames across the scene change
Histograms at left are from anchor person frames, while histogram at bottom right is from the street frame.
![Page 56: Computing Motion from Images](https://reader035.vdocument.in/reader035/viewer/2022062410/568159c9550346895dc71b77/html5/thumbnails/56.jpg)
Stockman MSU Fall 2009 56
Heuristics for ignoring zooms
![Page 57: Computing Motion from Images](https://reader035.vdocument.in/reader035/viewer/2022062410/568159c9550346895dc71b77/html5/thumbnails/57.jpg)
Stockman MSU Fall 2009 57
American sign language example
![Page 58: Computing Motion from Images](https://reader035.vdocument.in/reader035/viewer/2022062410/568159c9550346895dc71b77/html5/thumbnails/58.jpg)
Stockman MSU Fall 2009 58
Example from Yang and Ahuja
![Page 59: Computing Motion from Images](https://reader035.vdocument.in/reader035/viewer/2022062410/568159c9550346895dc71b77/html5/thumbnails/59.jpg)
Stockman MSU Fall 2009 59
![Page 60: Computing Motion from Images](https://reader035.vdocument.in/reader035/viewer/2022062410/568159c9550346895dc71b77/html5/thumbnails/60.jpg)
Stockman MSU Fall 2009 60