1. 1. MOTION FLOW SEGMENTATION & ANALYSIS Advanced Topics in Video Surveillance Yusuf Ziya UZUN
2. 2. MOTIVATION DARPA Visual Media Reasoning (VMR)
3. 3. WHAT IS VIDEO It is simply Sequence Of Images, literally: Group of Pictures(GOP) I-Frame : Intra-coded frame P-Frame : Predicted frame B-Frame : Bi-directional predicted frame B-Frame < P-Frame < I-Frame
4. 4. WHAT IS MOTION Motion: displacement, direction, velocity, acceleration, time and speed Motion Vector: Projection vector of motion from 3d to 2d Motion field: 3d motions projected to 2d images; dependency on depth Motion Flow: Ideally equals to Optical Flow http://en.wikipedia.org/wiki/Motion_(physics) http://en.wikipedia.org/wiki/Motion_estimation
5. 5. WHAT IS OPTICAL FLOW Optical Flow: distribution of the apparent velocities of objects in an image Brightness, color Zoom out Zoom in Pan right to left http://www.mathworks.com/discovery/optical-flow.html
6. 6. HOW TO ESTIMATE FLOW Why estimate motions? Track object behavior, alignment, stabilization. How to estimate pixel motion from image H to image I? Color constancy Looks same Brightness constancy Grayscale images Small motions Not to move far https://courses.cs.washington.edu/courses/csep576/05wi/lectures/motion.pdf
7. 7. SOME OF SEGMENTATION METHODS Background Subtraction Gaussian Distribution (PDF) Shot Boundary Detection Find Good Key-Frames (I-Frame) Feature Detection / Extraction Sobel Filter, SIFT Motion Segmentation Clustering of Motion Sets Clustering K-means Graph-based Segmentation Grouping, Cutting Superpixels Distance, graph, clustering
8. 8. PROBLEM The ground truth does not exist: the desired results always depend on the user requirements and specifications. Even for a fixed image, there may be more than one "best" segmentation because the criteria defining the quality of a segmentation are application dependent. -Pierre Soille http://cvl.ice.cycu.edu.tw/meeting/2008.09.23.pdf
9. 9. BACKGROUND SUBTRACTION Easy to implement, pretty fast, simply filtering One threshold for image (constant for every frame, no time dependency) What about Lighting changes, repetitive motions from clutter and long-term scene changes? Adaptive background mixture model http://en.wikipedia.org/wiki/Background_subtraction http://www.cs.utexas.edu/~grauman/courses/fall2009/slides/lecture9_background.pdf
10. 10. SHOT BOUNDARY DETECTION Hard cut, fade, dissolve How it helps? Storyline, time based localization Searching show me all films where there's a scene with a lion in it. Temporal Segmentation FeatureType of Edit Hard Cuts Fades Dissolve Color Histogram Differences X Edge Change Ratio X X X Standard Deviation of Pixel Intensities X Contrast X http://www.vis.uky.edu/~cheung/courses/ee639_fall04/readings/spie99.pdf http://en.wikipedia.org/wiki/Shot_transition_detection
11. 11. FEATURE DETECTION / EXTRACTION Interesting parts of image Corners, edges, blobs Feature Candidates Scale Invariant Feature Transform (SIFT) Speeded Up Robust Features (SURF) What is the relation between Motion Segmentation and Feature Extraction? We need to find good track points to create better segments!
12. 12. MOTION SEGMENTATION Seperate moving objects from background by using motion vectors(optical flow) Just split image N pieces. Trajectory segmentation, Local, Global Horn and Schunck, Kanade-Lucas-Tomasi(KLT) A local constraint to solve the aperture problem. Aperture, Barber-pole (Motion vs Optical) Closer Objects Have Bigger Velocity? http://en.wikipedia.org/wiki/Barberpole_illusion
13. 13. CLUSTERING Motion vector clustering for better segmentation and tracking
14. 14. GRAPH-BASED SEGMENTATION Affinity between pixels: Color distance Weighted with gradients Take into account optical flow From per pixel classifiers, etc. How to connect? Direct predecessor Displaced along optical flow t - 1 t t + 1 http://www.videosegmentation.com/
15. 15. APPLICATIONS Surveillance cameras Self driving cars (Autonomous) Estimating 3D structures Recognizing events and activities Facial expression recognition Video compression
16. 16. EXAMPLE: AUTONOMOUS CAR Sebastian Thrun, Google Self Driving Ca
17. 17. REFERENCES Motion Segmentation: a Review, Luca ZAPPELLA aXavier LLADa and Joaquim SALVI a Extracting representative motion flows for effective video retrieval, Zhe Zhao Bin Cui Gao Cong Zi Huang Heng Tao Shen The Computation of Optical Flow, S S Beauchemin and J L Barron A Database and Evaluation Methodology for Optical Flow, Simon Baker Daniel Scharstein J.P. Lewis Stefan Roth Michael J. Black Richard Szeliski Online Motion Segmentation using Dynamic Label Propagation Ali Elqursh, Ahmed Elgammal Performance of Optical Flow Techniques JL Barron, DJ Fleet and SS Beauchemin Background Subtraction Birgi Tamersoy Motion Estimation Motion and optical flow Optical flow (motion vector) computation, Nilesh Ghubade Optical Flow Estimation D.J. Fleet & A.D. Jepson, 2005 OPTICAL FLOW USING COLOR INFORMATION: PRELIMINARY RESULTS Kelson R. T. Aires, Andre M. Santana, Adelardo A. D. Medeiros Effcient Graph-Based Image Segmentation Pedro F. Felzenszwalb Graph-Based Hierarchical Video Segmentation Matthias Grundmann, Vivek Kwatra, Mei Han, Daniel Castro, Irfan Essa SLIC Superpixels* Radhakrishna Achanta, Appu Shaji, Kevin Smith, Aurelien Lucchi, Pascal Fua, and Sabine Susstrunk Comparison of Automatic Shot Boundary Detection Algorithms Rainer Lienhart
18. 18. THANK YOU