multi-resolution motion singularities for soccer video ...kblanc/documents/singletsslides.pdf ·...

07/21/20173rd International Workshop on Computer Vision in Sports (CVsports) at CVPR 2017

SingletsMulti-resolution Motion Singularities

for Soccer Video AbstractionKaty Blanc, Diane Lingrand, Frederic Precioso

I3S Laboratory, Sophia Antipolis, France

1

Katy Blanc Diane Lingrand Frederic Precioso

Overview


2

VIDEO & MOTIONSINGULARITIES

& SINGLETS

SOCCER SALIENT MOMENTS

Video Analysis


3

Burst of video content production

new sources of videosbig databases : Youtube 8M [1]much analyzed types : meeting/conferences, movies, news and sports

Diverse applications :

browsing in database, automatic video surveillance, driverless car, …

Exponential amount of information:

a match of soccer of HDTV → 324000 images of 1920×1080 pixels each

Collaboration with Wildmoka, themselves in collaboration with l’INA and BeIN.

Related works: Video description


4

Handcrafted features: Stip [3], iDT [4]

Deep learning representations [5]

Modelling human motion: [2]

Related works : sport abstraction

5

Clue Detection to detect highlights: ground color, jersey color, shot segmentation and view classification

Goals, attacks and other events usinglogo and score appearances and goal mouth position [7] Zawbaa et al.

Line marks positions and shot detection [6] Ye et al.

Face and skin detection, whistle detector and user specifications [8] Raventos et al.


Overview


6


& SINGLETS


Inspiration: fluid movement

7

Inspired from the work of Druon et al. [9] and the further work of Kihl et al. [10]


Optical Flow Approximation


8

Optical Flow = discrete bivariable vector field

Polynomial subspace and the Legendre Basis

𝐹 ∶ 𝛺 → ℝ2

𝑥1 , 𝑥2 → 𝑈 𝑥1 , 𝑥2 , 𝑉 𝑥1 , 𝑥2

with 𝛺 = −1, 1 2

𝑃𝐾,𝐿 𝑥1, 𝑥2 =

𝑘=0

𝐾

𝑙=0

𝐿

𝛾𝑘,𝑙 . 𝑥1𝑘𝑥2

𝑙

with 𝐾 + 𝐿 < 𝐷

Projection on the Legendre Basis

𝑈 = 𝑢0,0𝑃0,0 + 𝑢0,1𝑃0,1 + 𝑢1,0𝑃1,0𝑉 = 𝑣0,0𝑃0,0 + 𝑣0,1𝑃0,1 + 𝑣1,0𝑃1,0

Polynomial projection


9

Original Flow F

= (

Flow U Flow V Projection

𝑈𝑉

=0 0.02

3.91 0.𝑥1𝑥2

+0.69

−0.006

) ≃,

𝑈𝑉

= 1.38 + 0 + 0.007 + 0.023 + 0 − 4.47

−0.01 + 4.53 + 0.01 + 0 + 0 − 7𝐸−5

Approximation and coefficient analysis

10

From a simple analysis example to production, scenarization, event sementization


𝑈𝑉

= 1.38

0

Frame number

Coefficient value

𝑢0,0 horizontal global displacement𝑣0,0 vertical global displacement

𝑢0,0 horizontal global position

𝑣0,0 vertical global position

counterattack

Singularity


11

6 types of singularities

Then on the canonical basis𝑈𝑉

= 𝐴.𝑥1𝑥2

+ 𝑏 with 𝐴 ∈ 𝑀2,2 and 𝑏 ∈ ℝ2

λ1 and λ2 the Eigenvalues of A

𝑈 = 𝑢0,0𝑃0,0 + 𝑢0,1𝑃0,1 + 𝑢1,0𝑃1,0𝑉 = 𝑣0,0𝑃0,0 + 𝑣0,1𝑃0,1 + 𝑣1,0𝑃1,0

First projection on the Legendre basis

∆ 𝐴 = 𝑡𝑟 𝐴 2 − 4. det 𝐴 ,

Singularities extraction


12

From a multi-resolution analysis of the optical flow … 𝑈𝑉

= 𝐴.𝑥1𝑥2

+ 𝑏

Singlets: match singularities during time


13

From singularity on optical flow frame to tracks of singularities: Singlets

Overview


14


& SINGLETS


Application on soccer abstraction


15

Our database

Zoom

Slow Motion

Global Excitement

Soccer Saliant Moment

Soccer Summarization: Our database


16

Germany vs Portugal Nigeria vs Argentina France vs Honduras Switzerland vs France

4-0 2-3 3-0 2-5

© http://www.fifa.com/worldcup/matches/

Lack of standard benchmark for comparison sake…

http://www.fifa.com/worldcup/matches/


17

Soccer Summarization: Our database

Zoom detection


18

Zoom motion is a pure star node singularity.Zoom combined with a translation is an improper node

Conditions: there is a singularity it is a star or improper node : ∆ 𝐴 = 0 time consistent : last for one second

Positives eigenvalues -> zoom out

Negatives eigenvalues -> zoom in

Determine the zoom direction center

Zoom detection


19

• ∆ 𝐴 = 0 ↔ ∆ 𝐴 ≤ 0.2

• Threshold on an average |∆ 𝐴 | on a second

set to 0,2

• Comparison

- Global motion estimation [6,11]

- Duan et al. method [12] :Two histograms, one on magnitude one on angles to detect diagonal pattern (DL)

Slow Motion Detection


20

How to differentiate a fast motion that has been artificially slowed down from a slow motion?



21

Slow MotionFast Motion

Global excitement


22

Spatial histogram of 3x3 per frameSum spatial histograms on 10 framesThreshold set on 1500 to select the most agitated moments

Soccer Saliant Moment


23

Within 30 seconds:- at least two zoom direction changes- an activity peaks higher than 1500 in the farthest view- a slow motion replay in a close up view

Example of Summarization


24

Zooms Agitation Slow Motion

Extension to other sports


25

Set and Trained on Soccer and Tested on HandballAll hyperparameters, parameters and SVM were trained and set on soccer videos and we test ourframework on 5 minutes of Qatar Handball World Championship final without any adjustement or

retraining.

Conclusion


26

Optical Flow Approximation

Possibility to use :

- different basis- different degree- other space than the polynomial one

Singlets

Track singularities

Length histograms

Compute a description like for IDT

Singularity extraction

Vanishing point

6 types for the affine approximation

Motion description

Global distribution

Sport Video Abstraction

Zoom detection


Global excitement

Because there is not only soccer in life


27

Abstraction of concert

Facial Emotion

Action recognition


28

3rd International Workshop on Computer Vision in Sports: CVSports

References


29

[1] S. Abu-El-Haija, N. Kothari, J. Lee, P. Natsev, G. Toderici, B. Varadarajan, and S. ijayanarasimhan. Youtube-8m: A large-scalevideo classification benchmark. CoRR, abs/1609.08675, 2016.

[2] Gorelick, Lena, et al. "Actions as space-time shapes." IEEE transactions on pattern analysis and machine intelligence 29.12 (2007): 2247-2253.

[3] I. Laptev. On space-time interest points. Int. J. Comput. Vision, 64(2-3):107–123, Sept. 2005.

[4] H. Wang, A. Kläser, C. Schmid, and C.-L. Liu. Action Recognition by Dense Trajectories. In IEEE Conference on Computer Vision & Pattern Recognition, pages 3169–3176, Colorado Springs, United States, June 2011.

[5] D. Tran, L. Bourdev, R. Fergus, L. Torresani, and M. Paluri. Learning spatiotemporal features with 3d convolutional net-works. In roceedings of the IEEE International Conference on Computer Vision, pages 4489–4497, 2015.

[6] Q. Ye, Q. Huang, W. Gao, and S. Jiang. Exciting event detection in broadcast soccer video with mid-level description and incremental learning. In Proceedings of the 13th annual ACM international conference on Multimedia, 2005.

[7] H. M. Zawbaa, N. El-Bendary, A. E. Hassanien, and T. Kim. Event detection based approach for soccer video summarization using machine learning. International Journal of Multimedia and Ubiquitous Engineering, 2012

[8] A. Raventos, R. Quijada, L. Torres, and F. Tarres. Automatic summarization of soccer highlights using audio-visual descriptors. arXiv preprint arXiv:1411.6496, 2014.

[9] Druon, Martin. Modélisation du mouvement par polynômes orthogonaux: application à l'étude d'écoulements fluides. Diss. Université de Poitiers, 2009.

[10] O. Kihl, B. Tremblais, and B. Augereau. Multivariate orthogonal polynomials to extract singular points. In IEEE International Conference on Image Processing 2008. ICIP 2008 San Diego, CA, United States, Oct. 2008.

[11] X. Qian. Global Motion Estimation and Its Applications. INTECH Open Access Publisher, 2012.

[12] L.-Y. Duan, M. Xu, Q. Tian, C.-S. Xu, and J. S. Jin. A unified framework for semantic shot classification in sports video. IEEE Transactions on Multimedia, 2005

Optical Flow Extraction


30

Gunnar Farneback MethodPixel Neighborhood Approximation

Approximation translation

Relations

Speed vector estimation

Polynomial Space Projection


31

Scalar product

Legendre basis with 𝑤 𝑥1, 𝑥2 = 1 and Ω = −1,1 2

Polynomial degree 𝐷 ⇒ 𝑛𝑑 =(𝐷+1)(𝐷+2)

2polynomials in the basis and so 𝑛𝑑

coefficients

Flot optique blurring (decreasing with the degree)

Multi-Resolution Singularity


32

One sliding windows -> possibly one singularity

The same singularity can be at different size and aprroximately the sameposition: We keep the one with less angular deviation from the original flow.

𝑑𝑒𝑣 =

𝜔∈Ω

1

2sin 𝜃 𝜔 − 𝜃′ 𝜔

With 𝜃 𝜔 the angle of the motion vector at the pixel position 𝜔 for the originalf flow and 𝜃′ 𝜔 for the polynomial approximation

multi-resolution motion singularities for soccer video ...kblanc/documents/singletsslides.pdf ·...

Documents