video quality assessment and comparative evaluation of peer-to-peer video streaming systems aditya...

Video Quality Assessment and Video Quality Assessment and Comparative Evaluation of Comparative Evaluation of

Peer-to-Peer Video Streaming Systems Peer-to-Peer Video Streaming Systems

Aditya MavlankarAditya Mavlankar Pierpaolo Baccichet Pierpaolo Baccichet

Bernd GirodBernd Girod

Stanford UniversityStanford UniversityStanford CA, USAStanford CA, USA

Sachin AgarwalSachin AgarwalJatinder Pal SinghJatinder Pal Singh

Deutsche Telekom A.G., Deutsche Telekom A.G., LaboratoriesLaboratories

Berlin, GermanyBerlin, Germany

Mavlankar et al.: Comparative Evaluation of Peer-to-Peer Video Streaming Systems Jun. 25, 2008 2

Outline

Introduction to P2P live video streaming Prior work on system performance assessment Test-bed setup Performance of tested systems


P2P Live Video Streaming

Extension of P2P file-sharing Low-cost and scalable delivery mechanism Several deployed commercial implementations today Increasing content / channels available


Related Work on Performance Assessment

Networking related metrics, e.g. bandwidth usage, packet loss, continuity index, etc.– CoolStreaming [Zhang et al., 2005]: PlanetLab– PPLive [Hei et al., 2006]: packet sniffing and crawling– SopCast [Sentinelli et al., 2007]: “watching”, PlanetLab– . . .

No video PSNR results No repeatable test conditions– Network conditions– Encoded video characteristics– Peer behavior

No fair head-to-head comparison of different systems


Test-Bed Setup

TU Munich, Germany (3)[Emulated HS Broadband]

Stanford, CA (22)[Emulated HS Broadband]

Test centerBerlin, Germany

Server 1, 2

Berlin, Germany (15)[Emulated HS Broadband]

128 X 2192 X 2576 X 21024 X 2

3072 X 3

576 X 161024 X 52048 X 1

576 X 91024 X 52048 X 1

52 Mbps

ISP

DatacenterErfurt, Germany

Internet

Berlin, Germany (8)[Real HS Broadband]

PLR, delay, jitter and bandwidth measured for representative real connections and emulated using NISTNet traffic shaper


Encoded Video Stream

La Dolce Vita (Fellini, 1960) 24 fps, 352x240 pixels H.264/AVC video codec, 400 kbit/sec CBR bitstream,

42 dB PSNR I B B P B B P B B P . . . (I frame every second) H.264 bitstream wrapped in Microsoft ASF container, if

required by tested system Last frame error concealment


Peer Churn Model

30-minute simulation run During each 6-minute time-slot– Peer on with probability 0.9– Peer off with probability 0.1– Peer can switch off for the rest of the run with

probability 0.05 During last 5 minutes, peer off with probability 0.5


Representative Results

Tested systems– System A: Tree-based, push approach– System B: Mesh-based, data-driven or pull approach

Emulation runs– Run 1: with traffic shaping (using NISTNet)– Run 2: without traffic shaping

Same realization of peer On-Off model for all runs


0 10 20 30 40 50 6010

15

20

25

30

35

40

45

Pre-roll delay [sec]

Avg

. PS

NR

acr

oss

all c

lient

s [d

B]

Tree, w/ traffic shapingTree, w/o traffic shapingMesh, w/ traffic shapingMesh, w/o traffic shaping

Pre-Roll Delay

about 30 sec enough for System A (tree-based)

about 60 sec enough for System B (mesh-based)


PSNR Drop (w/ traffic shaping)

0 20 400

1

2

3

4

5

6

Client ID

Avg

. dro

p in

PS

NR

[dB

]

0 20 400

1

2

3

4

5

6

Client ID

Avg

. dro

p in

PS

NR

[dB

]

System A (tree-based) System B (mesh-based)

¼X sdhsdj sdf hdsf j sdf j dsf sdf sdf kj sf j sdkj f skd¼

32


0 20 400

1

2

3

4

5

6

Client ID

Avg

. dro

p in

PS

NR

[dB

]

0 20 400

1

2

3

4

5

6

Client ID

Avg

. dro

p in

PS

NR

[dB

]

PSNR Drop (w/o traffic shaping)

System A (tree-based) System B (mesh-based)32


Statistics of Frame Freezes

Frames frozen (as percentage of total frames to be displayed)

Average no. of distinct frame-freeze events per client in 30 min.


Run 1 4.2% 2.0%

Run 2 3.0% 0.2%


Run 1 64 23

Run 2 40 2


Statistics of Frame Freezes (cont.)

0 20 40 60 80 100 1200.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

Length of frame freeze [number of frames]

cdf

System A, Run 1System A, Run 2System B, Run 1System B, Run 2

Long frame freezes more likely with System B (mesh-based)

System A (tree-based) employs content-aware prioritization


No. of Peers Failing to Decode a Frame

0 10 20 300

10

20

30

40

Time [minutes]

No.

of p

eers

faili

ng to

dec

ode

the

fram

e

0 10 20 300

10

20

30

40

Time [minutes]

No.

of p

eers

faili

ng to

dec

ode

the

fram

eSystem A (tree-based), Run 1 System B (mesh-based), Run 1


Redundancy (bytes received in excess of required video stream bytes)– System A (tree-based): 6% in both runs– System B (mesh-based): 35% and 20% in Runs 1 (w/

traffic shaping) and 2 (w/o traffic shaping) respectively For both Systems, peer receives on average less than 10%

of its data directly from the server; slightly more for Run 2 of System B

System A (tree-based): Sustained downloads from lower number of parent peers

Redundancy, Server Load and Parent-Peer Analysis


Summary

Proposed methodology allows measuring video PSNR, buffering time, frame-freeze statistics, peers failing to decode a frame, etc. beyond network usage, packet loss, etc.

Test conditions chosen by analyzing real-world conditions and experiments are repeatable

Tested three commercial-grade P2P video streaming systems Room for improvement in current systems:

– Long buffering time (10s of seconds)– Display freezes for more than 100 frames

Tested tree-based system outperforms mesh-based system:– Redundancy– Buffering time

Thank you!Thank you!http://www.stanford.edu/~maditya/publication.htmlhttp://www.stanford.edu/~maditya/publication.html

Related:Related:

[Agarwal, [Agarwal, et alet al., TRIDENTCOM 2008]., TRIDENTCOM 2008]

video quality assessment and comparative evaluation of peer-to-peer video streaming systems aditya...

Documents