trinity college dublin pixelgt: a new ground truth specification for video surveillance dr. kenneth...

Trinity College Dublin

PixelGT: A new Ground Truth specification for video surveillance

Dr. Kenneth Dawson-Howe,Graphics, Vision and Visualisation Group (GV2),

School of Computer Science & Statistics, Trinity College Dublin


Contents

Introduction Review Specification Annotation Conclusions

IntroductionReviewSpecificationAnnotationConclusions


Available video sequences

Video sequences with no ground truth

Video sequences with moving object pixel masks

Video sequences with moving object bounding boxes

Video sequences with labeled events



Available video sequencesIntroductionReviewSpecificationAnnotationConclusions


Video sequences with moving object pixel masks

Video sequences with moving object and moving shadow pixel masks.

Artificial video sequences with moving object pixel masks.

Video sequences with labeled pixels.


Cam

Vid

dat

aset

VS

SN

dat

aset

PE

TS

/LIM

U d

atas

etA

TO

N d

atas

et


Video sequences with moving object bounding boxes


ET

ISE

O d

atas

et

CA

VIA

R d

atas

et


Video sequences with labeled events


PE

TS

200

6 da

tase

t


Motivation

Existing ground truth requires a choice: Binary pixel masks (with no specific object

identification)OR

Object identification (with bounding boxes only and no shadows).

Relatively little pixel accurate ground truth is available.

No existing means of dealing with transparency and reflections.



Ground truth Specifcation

Pixel based ground truth

Bit organisation

Meta data

Gradient profiles

Bounding boxes & events



Pixel based ground truth

Object class (8 bits)

Object instance (9 bits)

Shadow level (4 bits)

Transparency level (3 bits)



Bit organisationIntroductionReviewSpecificationAnnotationConclusions

L = Label classI = IDS = ShadowT = Transparency1 = LSB8 = MSB


Meta data

Based on CVML

Includes: File names Bit organisation Meaning of object labels Gradient profile



Gradient profiles

Width of edges?

Required for assessing performance?



Bounding boxes and events

Bounding boxes for moving objects Implicitedly included Can be explicitly specified Aids comparison of techniques

Events Some implicitedly included Can be explicitly specified Aids comparison of techniques



Creation of ground truth

Difficult to label all pixels in all frames Short videos? Not all frames / pixels?

Method Background annotation Object and shadow boundary specification Object class and ID annotation Transparency, Meta-data, Bounding boxes, Events

and Gradient Profile specification Propagation of ground truth



Specifying object class & IDIntroductionReviewSpecificationAnnotationConclusions


Background annotation

For static cameras Annotate all background pixels Facilitates automatic event detection



Specifying object and shadow boundaries

Identify boundary edges of objects / shadows Once shadow boundary is complete shadow level is

computed automatically.



Specifying object class & ID

Once boundary is complete Select object class and (optionally) ID Click on the object



Transparency, Meta-data, Bounding boxes, Events and Gradient Profile specification

Transparency not yet addressed

Meta data – specified in CVML

Bounding boxes & events – extract automatically from pixel based ground truth

Gradient profile – extract automatically from each image



Propagation of ground truth

Propagation only required for moving objects

Working with every frame results in very little movement between frames (0-5 pixels) Search for the closest similar contour in the new

frame where the pixels on the object side of the contour appear similar…

Need to identify & correct1. when new objects enter the scene.2. when objects reappear from behind occlusions.3. when internal holes appear within an object.4. when errors occur in the propagation



The positive spin…

PixelGT is a new form of ground truth which will facilitate Pixel-level assessment of trackers Semi-automatic generation of object location,

bounding boxes and event descriptions. Comparison of trackers which assess performance

using different types of ground truth



Questions

Do we really need this detailed ground truth ?

Should we change anything about the specification ?

Other questions ?


trinity college dublin pixelgt: a new ground truth specification for video surveillance dr. kenneth...

Documents

trinity college dublin

trinity college dublin

boxes video sequences

ground truth video sequences

artificial video sequences

new ground truth specification

msb slide

ground truth object