7/26/2019

1

1

Data Management for Video Analytics

Brandon Haynes, Maureen Daum, Amrita Mazumdar, Magdalena Balazinska, Luis Ceze, & Alvin Cheung

2

Video data is everywhere.

3

Existing systems treat video data like it’s the

20th century

4

Hei

ght

in p

ixel

sWidth in pixels

5

Many recent video applications

require jointly querying multiple cameras,

reasoning about position and orientation, or

querying complex metadata.

6

1 2

3 4

5 6

7/26/2019

2

7

SELECT Metadata.Location FROM Cameras

WHERE ‘James Bridle’ IN Metadata.Name

8

A Video Data Management Benchmark

Visual RoadA Database System for

Virtual & Augmented Reality Video Applications

Optimized storage and retrieval of video data

Video

File SystemMetastoreEfficient querying of rich

video content

(Maureen Daum)

Data Management for Video Analytics Modern

LightDBA Database System for

Virtual & Augmented Reality Video Applications

How would you write this application?12

#include <libavformat/avformat.h>

20th century Imperative Code LightDB Query

source = Scan(“kittens”)detection = source.Map(𝑑𝑒𝑡𝑒𝑐𝑡) result = Union(source, detection)result.Save(“output.mp4”)

7 8

9 10

11 12

7/26/2019

3

13

source = Scan(“kittens”)detection = source.Map(𝑑𝑒𝑡𝑒𝑐𝑡) result = Union(source, detection)result.Save(“output.mp4”)

LightDB Query:

14

Logical Plan

Scan

Map(Grayscale)

EncodeHEVC

Scan

Map

Union

Save

source = Scan(“kittens”)detection = source.Map(𝑑𝑒𝑡𝑒𝑐𝑡) result = Union(source, detection)result.Save(“output.mp4”)

15

GPUOverlay

GPUOverlay

ScanSingleFile

SaveSingleFile

GPUMap GPUMap GPUEncode GPUEncode

GPUTee GPUTee GPUToCPU

GPUToCPU

GPUQueuedDecode

Homomorphic Union

Physical Plan

0

15

30

45

Timelapse Venice Coaster Cats

FPS

LightDB Ffmpeg Scanner OpenCV

(Light Field)

Object Overlay AR Application

Brandon Haynes bhaynes@cs.washington.edulightdb.uwdb.io

Key Features:• Data management system for VR & AR video applications• Unified data model for panoramic (360°) and light field video• Declarative queries with automatic optimization

Key Contributions:• ∼ Τ1 10 lines of code• Up to 4×performance for real-world workloads• Reduced client bandwidth & power requirements Scan(“LEGOS”)

.Map(GRAYSCALE)

.Store(“GRAYLEGOS”)

A Database System for Virtual & Augmented Reality

Video Applications

A Video Data Management Benchmark

Visual RoadOptimized storage and retrieval of video data

Video

File SystemMetastoreEfficient querying of rich

video content

(Maureen Daum)

Data Management for Video Analytics Modern

13 14

15 16

17 18

7/26/2019

4

19

The performance of video systems is evaluated

like it’s still the 20th century

20

Video System # Distinct Videos

LightDB (2018) 4

Chameleon (2018) 5

BlazeIt (2018) 6

NoScope (2017) 7

Focus (2018) 14

DeepLens (2019) ∼16

Scanner (2018) >100 (only 14 joined)

# Distinct videos performance tested by system

21

UA-DETRAC Dataset (2015)10 hours, 24 LocationsVideo by Wen et al.

Manually annotated with 1,210,000 hand-

drawn boxes!

22

Type Benchmarks Test Data

Video (Visual Road) Ad Hoc

OLTP (TPC-H) Synthetic

OLAP (SSB, DWEB) Synthetic

Streaming (Linear Road, DTDW) Synthetic

NoSQL (YCSB) Synthetic

Graph (LDBC) Synthetic

Privacy (SDV) Synthetic

Synthetic

23

Benchmark Queries & Verifier

Video Dataset Generator

Hyperparameters: scale, resolution,

duration, seed Configure

GenerateSynthetic Dataset

Commercial Gaming Engine

Autonomous Driving Simulator

Video Data Management System

Execute

Q1

Q2

Q2b

Q2c

Q2d Q

3

Q4

Q5

Q6a

Q6b Q

7

Q8

Q9

Q10

Verify

PregeneratedDatasets

24

19 20

21 22

23 24

7/26/2019

5

25 26

Automatic ground truth generation

27

Traffic Camera 𝑉𝑗 : Result for query instance 𝑖:

(4 × 𝑆𝑐𝑎𝑙𝑒 query instances)

⋮

Query 7: Object Detection

28

0

1

2

Q2(a)

0

8

16

Q2(d)

0

6

12

Q2(b)

0

8

16

Q3

0

1

2

Q1

0

36

72

Q2(c)

Scale Factor

0

1

2

0 2 4 6 8

Q5

0

2

4

0 2 4 6 8

Q6(a)

0

4

8

0 2 4 6 8

Q4

`

29

visualroad.uwdb.ioKey Features:

• Video data management benchmark• Synthetic dataset generation• Unlimited scale, resolution, duration, and overlap• Extensible suite of computer vision, VR, and microbenchmarks• Applicable to both general and specialized video DBMSs

A Database System for Virtual & Augmented Reality

Video Applications

A Video Data Management Benchmark

Visual RoadOptimized storage and retrieval of video data

Video

File SystemMetastoreEfficient querying of rich

video content

(Maureen Daum)

Data Management for Video Analytics Modern

25 26

27 28

29 30

7/26/2019

6

31

visualroad.uwdb.io

Key Features:• Video data management benchmark• Synthetic dataset generation• Unlimited scale, resolution, duration• Extensible suite of computer vision, VR, and

microbenchmark queries

Key Features:

• DBMS for VR & AR video applications• Unified data model• Declarative queries• Automatic optimization

lightdb.uwdb.io

A Database System for Virtual & Augmented Reality

Video Applications

A Video Data Management Benchmark

Visual RoadOptimized storage and retrieval of video data

Video

File SystemMetastoreEfficient querying of rich

video content

(Maureen Daum)

Data Management for Video Analytics Modern

Towards Efficient Querying of Rich Video Content

Maureen Daum

Motivation• We want to enable rich, content-based queries over video data

• Existing systems optimize running object detection over videos• As a result, they focus on simple queries only

• We want to use this metadata to enable more complex queries

Metadata

• Object labels

• Weather conditions

• Descriptions

• License plate numbers

• Aggregates

35

SELECT pixels FROM video WHERE dog

36

31 32

33 34

35 36

7/26/2019

7

Overlay the dog pixels from video 1 onto the background in video 2

37

Select sequences of frames that contain increasing numbers of cats

38

Metadata Model

t

y

x

Label Volume

Dog 𝑡0 , 𝑡1 , 𝑥0 , 𝑥1 , [𝑦0 , 𝑦1]

39

Executing Queries

40

• Videos are stored in a compressed format

• Encoding and decoding are expensive operators

0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%

Select frames with bicycles

Select frames with buses

Select frames with people

Draw boxes over people

Percent of Query Time Spent in Encode or Decode

Encode and decode

SELECT frames FROM video WHERE dog

This Photo by Unknown Author is licensed under CC BY-SA 41

SELECT frames FROM video WHERE dog

This Photo by Unknown Author is licensed under CC BY-SA 42

37 38

39 40

41 42

7/26/2019

8

SELECT frames FROM video WHERE dog

This Photo by Unknown Author is licensed under CC BY-SA 43

SELECT frames FROM video WHERE dog

This Photo by Unknown Author is licensed under CC BY-SA 44

SELECT frames FROM video WHERE dog

This Photo by Unknown Author is licensed under CC BY-SA 45

SELECT frames FROM video WHERE dog

This Photo by Unknown Author is licensed under CC BY-SA 46

SELECT frames FROM video WHERE dog

This Photo by Unknown Author is licensed under CC BY-SA 47

SELECT frames FROM video WHERE dog

This Photo by Unknown Author is licensed under CC BY-SA 48

43 44

45 46

47 48

7/26/2019

9

SELECT frames FROM video WHERE dog

This Photo by Unknown Author is licensed under CC BY-SA 49

SELECT frames FROM video WHERE dog

This Photo by Unknown Author is licensed under CC BY-SA 50

This Photo by Unknown Author is licensed under CC BY-SA 51

SELECT frames FROM video WHERE dog

Random access point

Encoded as a delta

This Photo by Unknown Author is licensed under CC BY-SA 52

SELECT frames FROM video WHERE dog

Random access point

Encoded as a delta

This Photo by Unknown Author is licensed under CC BY-SA 53

SELECT frames FROM video WHERE dog

Random access point

Encoded as a delta

This Photo by Unknown Author is licensed under CC BY-SA 54

SELECT frames FROM video WHERE dog

Random access point

Encoded as a delta

49 50

51 52

53 54

7/26/2019

10

This Photo by Unknown Author is licensed under CC BY-SA 55

SELECT frames FROM video WHERE dog

Random access point

Encoded as a delta

This Photo by Unknown Author is licensed under CC BY-SA 56

SELECT frames FROM video WHERE dog

Random access point

Encoded as a delta

57

SELECT pixels FROM video WHERE dog

58

SELECT pixels FROM video WHERE dog

59

SELECT pixels FROM video WHERE dog

60

SELECT pixels FROM video WHERE dog

55 56

57 58

59 60

7/26/2019

11

61

SELECT pixels FROM video WHERE dog

62

SELECT pixels FROM video WHERE dog

Add Metadata

Putting Things Together

Similar to DB cracking, incrementally partition the video and add indices

Original Video

Execute Query

Update Layout

63

Ongoing Work

• Measure the effectiveness of selection optimization techniques

• Investigate optimization techniques for more compute-heavy queries

• Determine how to effectively layout videos with a lot of metadata• Possibly store multiple versions of a video with different layouts

64

Conclusion

• Deep learning opens the door to rich queries over video data

• Videos are large and slow to process

• Database techniques can accelerate such queries• Partitioning

• Indexing

• Incremental physical tuning

• Indexing must be balanced with maintaining reasonable storage sizes

61 62

63 64

65