player highlighting and team classification in broadcast ...yuhuang/papers/wocc09slide.pdf ·...

Player Highlighting and Team Classification in Broadcast Soccer Videos for the Next Generation TV

Yu HuangMultimedia Content Networking Lab

Core Network Research Dept., Huawei Technology (USA )400 Somerset Corporate Blvd., Bridgewater, NJ08807

WOCC’09, NJ, USA

Yu Huang, Huawei Technologies

Outline

• Background• Player Detection• Player Tracking• Use Case 1: Object Highlighting• Use Case 2: Team Classification• Demo of Use Cases • Conclusion


Background• The Next Generation TV (such as IPTV, Interactive TV and

Mobile TV etc.) offers appealing services:– Enhancement;– Personalization;– Interactivity.

• TV contents require to be analyzed and annotated.– Different levels: objects, scenes and events;– Different domains: spatial, temporal and luminance.

• Sports game (such as soccer) is a good application scenario to depict the viability of the next generation TV:

– Players are important clickable objects in the scene with linked rich media information for viewers;

– Scenes/events can be nonlinearly skimmed /browsed with summarization.

• Player localization/team classification provide a nice bridge for object level interactivity in sports programs (work focus).

(To be continued)


Background• Player Detection in Soccer Game Video

– Segmentation-based detection:• Motion segmentation;

– A special case: mosaic.

• Background modeling (static)-based;• Playfield model-based.

– Statistically learning-based detection:• SVMs;• Adaboost.

• Open questions:– Crowded;– Occlusion;– Body orientation;– Limb deformation.

(To be continued)


Background• Player Tracking in Soccer Game Video

– Top-down: Filtering and Data Association• Kalman filter, Particle Filter, PDAF;

– Bottom-up: Target Representation and Localization• Template matching, Motion tracking;• Histogram matching, Kernel-based (Mean-shift);

• Contour-based tracking (snake or level set).

• Challenging issues:– Fast abrupt motion (even motion blur);– Size variation;– Shape change/viewing angle;– Occlusion.

(To be continued)


Background• Main points:

– Segmentation-based player detection: • Gaussian model-based playfield model and updated with

filtered data from dominant color detection;

– Kernel-based tracking: • Incorporate the foreground map from playfield-based

segmentation into the mean shift-based iteration for tracking.

– Use cases:• Object highlighting;• Team Classification.


Player Detection

• Playfield model– Initialize a single Gaussian model in each

component (RGB) with labeled data;

– Update with filtered data based on the dominant color detection:

• Dominant (Gaussian) mode in color component histograms;

• Check it with the learned Gaussian and update the model.

(To be continued)

BGRixNxp iii ,,),,;()( == σµ


Player Detection

Mean:

Labeled data Dominant color detection

(To be continued)

meanh⋅+⋅−= ρµρµ )1('

cov)1(' q⋅+⋅−= ρσρσVariance:


Player Detection

(To be continued)

Playfield Pixel Detection

Connected Component Analysis

Size Filter

Interior Filling

XO

R

Connected Component Analysis

PlayfieldExtraction

ObjectDetection

Morphological filter (dilation, erosion)

Shape Filter (size, roundness, eccentricity)


Player Detection


Player Tracking

Initialization from User Interaction

Foreground Map from Playfield Model-based

Segmentation

Histogram Acquisition

Mean Shift iteration

Histogram Update

Next frame

Initial frame

Location Scale

Lost?N

Y

Player Detection

Player Tracking


Tracking Initialization

• Player (OOI, i.e. object of interest) Clicking;

• Build the target model (weighted histogram) as

∑ =−−= kn

i iiiqut uxbxkxFCq1

*2**, ])([)())(1( δ

Kronecker delta functionnormalization factorforeground map


Foreground Map Generation from Playfield-based Segmentation

Foreground mask

BGRixNxp iii ,,),,;()( == σµ

),;(),;(),;(),( BBGGRR bNgNrNvuF σµσµσµ ⋅⋅=

<−<−<−

=otherwiseT

tbANDtgANDtrvuG BBGGRR

,

if ,1),(

σµσµσµ

Playfield model

Playfield map

Simplified Foreground

Map

Gaussian


Mean Shift Iteration for Tracking

Initialize with the location and scale in the last frame

Mean Shift iteration Mean shift Iteration

original scale

original scale+10%

original scale-10%

Mean Shift Iteration

Discriminant Similarity Discriminant Similarity Discriminant Similarity

Choose the maximal similarity


Mean Shift Iteration with Foreground Map

Calculate candidate normalized histogram ∑=

−−−= hn

i iiipu uxbhxykxFCyp1

2

00 ])([)/)ˆ(())(1()ˆ( δ

Mean Shift vector

∑

∑

=

=

−−

−−

=h

h

n

i

iii

n

i

iiii

h

xygxFw

h

xygxFwx

y

1

2

0

1

2

0

1

)ˆ

())(1(

)ˆ

())(1(ˆ

ε≤− 10 ˆˆ yy

Y

10 ˆˆ yy =

N

Weighted by the foreground map!

g(x)=k’(x)


Discriminant Similarity Function• Choose the “Center-Context” window;

– The inward window size wxh while the outward window size 3wx3h.

• Build histogram from the surrounding background of the target;

• Similarity measure weighted by the discrimination metric between the

target and its surrounding background.

== otherwise

yp

ypif

v

ub

ub

u ,)(

0)( ,0

,

,

ϑ

)(ypb

)(yp f ∑ =−−= hn

i iiufuf uxbhxykvCyp1

2

, ])([)/)(()(' δCandidate

Target

Similarity function

.)('')](','[max1 ,,∑ =

= m

u ufutft ypqypqρ

∑ =−= kn

i iiuqut uxbxkvCq1

*2*, ])([)(' δ


The Target Model Update

• Given the estimated location yt and scale st at the current frame;

• Run mean shift iteration again to disturb the estimated location yt , where

the initial target histogram q0 (u) takes place of the current target

histogram qt (u).

• The update strategy act conservatively like

– If the new estimated location yt’ in the second iteration does not diverge too

far from yt, , i.e. , update the current target histogram with the

candidate histogram at location yt’ and scale st as qt+1 (u) = p(yt’ ), and

update the estimated location as yt’ ;

– Otherwise, keep the old target histogram qt+1 (u) = qt (u) and the estimated

location in the first iteration yt .

ε≤− tt yy'


Case 1: Object Highlighting

• Ellipse or rectangle window for the object of interests;

• Dimming the other areas except the object window:– Decrease exponentially the Y component in YUV format.


Case 2: Team Classification• Label the Players/Referees with Team A, Team B and

Referee:– Goalies for both teams are seldom present in the videos, so

temporally not considered.

• Defined Feature for Classification:– a multi-histogram representation (bi-histograms for jersey and

shorts regions).

(To be continued)


Case 2: Team Classification• Target histogram acquisition for each class:

– For each class, click on multiple samples in the video frames;

– Each object blob is separated into upper (jersey region) and lower (shorts region) parts;

– Collect all pixels for jersey and shorts regions, then build corresponding normalized color histograms respectively;

• Team Classification when the isolated player being clicked:– Acquire its bi-histogram feature as well;– Identify its class by bi-histogram matching.


Bi-Histogram Matching• Search the best cutting line to separate the object (player/referee) blob into two

parts vertically;

• The maximized objective function (Bhattacharyya distanceBhattacharyya distance) is to realize the most discrimination between jersey and shorts regions.

• Bi-histogram matching for team classification

∑ =−= m

u uudownupper qpqp1

0.1],[ρ

Normalized Color Histogram

Search range

Cutting line

∑∑ ==⋅−+⋅= m

u uiu

m

u uiuiii

qpqpqqpp1 221 112121 )1()],(),,[(max λλρ


Demo of Use Cases

• GUI implemented with MS Visual Studio VC++;• User clicks the object by the mouse on the played video;• Real-time segmentation, identification and tracking:• Use Case 1: Object Highlighting;• Use Case 2: Team Classification.

– Flagging of Players/Referees;– For instance:

• Germany ------------- Spain ------------- Referee.


Demo Video: Object Highlighting


Demo Video: Team Classification


Conclusions• Gaussian model-based playfield model + dominant color

detection for player detection; – A Semi-supervised method.

• Modified mean shift-based with soft constraints from the foreground map for player tracking;– Reduce the risk of tracking failure for fast-moving players.

• Scale adaptation based on discrimination between the target and its surrounding background in tracking;– avoid the “shrinkage” problem;

• Target histogram updating in a conservative way in tracking;– alleviate drifting problems.

• Two use cases for viability of interactive services:– “Object highlighting”;– “Team classification”: bi-histogram matching.


Acknowledgement

• Collaborators at Huawei Technologies:– Dr. Hongbing LI;– Dr. Jun TIAN;– Dr. Heather YU.


Thanks for your attention!

player highlighting and team classification in broadcast ...yuhuang/papers/wocc09slide.pdf ·...

Documents