1 developing a predictive model for internet video quality-of-experience athula balachandran, vyas...

1

Developing a Predictive Model for Internet Video Quality-of-Experience

Athula Balachandran, Vyas Sekar,Aditya Akella, Srinivasan Seshan,

Ion Stoica, Hui Zhang

2

QoE $$$

CDN

UsersContentProviders

$$$Better QualityVideo

HigherEngagement

The QoE model

Diagram courtesy: Prof. Ramesh Sitaraman, IMC 2012

3

Adapting video bitrates quicker

Picking the best server

Why do we need a QoE model?

Comparing CDNs

4

Traditional Video Quality Metrics

Objective Scores (e.g., Peak Signal to Noise Ratio)

Subjective Scores(e.g., Mean Opinion Score)

Does not capture new effects (e.g., buffering, switching

bitrates)

User studies not representative of “in-the-wild” experience

5

Internet Video is a new ball game

Objective Scores (e.g., Peak Signal to Noise Ratio)

Subjective Scores(e.g., Mean Opinion

Score)

Engagement(e.g., fraction of video viewed)

Quality metrics

6

Commonly used Quality MetricsJoin Time

Average Bitrate

Buffering ratio

Rate of buffering

Rate of switching

7

Which metric should we use?

Objective Scores (e.g., Peak Signal to Noise Ratio)

Subjective Scores(e.g., Mean Opinion

Score)

Engagement(e.g., fraction of video viewed)

Quality metrics Buffering Ratio, Average bitrate?

Today:QualitativeSingle-metric

8

Unified and Quantitative QoE Model

Objective Scores (e.g., Peak Signal to Noise Ratio)

Subjective Scores(e.g., Mean Opinion

Score)

Engagement(e.g., fraction of video viewed)

Quality metrics Buffering Ratio, Average bitrate?

ƒ (Buffering Ratio, Average bitrate,…)

9

Outline

• What makes this hard?

• Our approach

• Conclusion

Complex Engagement-to-metric RelationshipsEn

gage

men

t

Quality Metric

10

Non-monotonic

E

ngag

emen

t

Average bitrate

En

gage

men

t

Rate of switching

Threshold

[Dobrian et al. Sigcomm 2011]

Ideal Scenario

11

Complex Metric Interdependencies

Join

Tim

e

Average bitrate

Average bitrate

Rate

of b

uffer

ing

Join Time Bitrate

Buffering ratio

Rate of buffering

Rate of switching

Confounding Factors

12

Confounding Factors can affect:1) Engagement

Live and Video on Demand (VOD) sessions have different viewing patterns.

CD

F (

% o

f use

rs)

Engagement

Type of VideoLive

VOD

Confounding Factors

13

Confounding Factors can affect:1) Engagement2) Quality Metrics

Type of VideoLive

VOD

CD

F (

% o

f use

rs)

Join Time

Live and Video on Demand (VOD) sessions had different join time distribution.

Confounding Factors

14

Confounding Factors can affect:1) Engagement2) Quality Metrics3) Quality Metric

Engagement

Type of Video

ConnectivityDSL/Cable

Wireless (3G/4G)

Eng

agem

ent

Rate of buffering

Users onwireless connectivity weremore tolerant to rate of buffering.

Confounding Factors

15

Type of Video

Connectivity

Device

Location

Popularity

Time of day

Day of week

Need systematic approach to identify and incorporate confounding factors

16

Summary of Challenges

1. Capture complex engagement-to-metric relationships and metric-to-metric dependencies.

2. Identify confounding factors3. Incorporate confounding factors

17

Outline

• What makes this hard?

• Our approach

• Conclusion

18

Challenge 1: Capture complex relationships

19

Cast as a Learning Problem

MACHINE LEARNING

Engagement Quality Metrics

QoE Model

Decision Trees performed the best.Accuracy of 40% for predicting within a 10% bucket.

20

Challenge 2: Identify the confounding factors

21

Test Potential Factors

Engagement

Confounding Factors

Quality Metrics

22

Test Potential Factors

Test 1: Relative Information Gain

Engagement

Confounding Factors

Quality Metrics

23

Test Potential Factors

Test 1: Relative Information GainTest 2: Decision Tree StructureTest 3: Tolerance Level

Engagement

Confounding Factors

Quality Metrics

24

Identifying Key Confounding Factors

Factor Relative Information Gain

Decision TreeStructure

Tolerance Level

Type of video ✓ ✓ ✓Popularity ✗ ✗ ✗Location ✗ ✗ ✗Device ✗ ✓ ✓Connectivity ✗ ✗ ✓Time of day ✗ ✗ ✓Day of week ✗ ✗ ✗

25

Identifying Key Confounding Factors

Factor Relative Information Gain

Decision TreeStructure

Tolerance Level

Type of video ✓ ✓ ✓Popularity ✗ ✗ ✗Location ✗ ✗ ✗Device ✗ ✓ ✓Connectivity ✗ ✗ ✓Time of day ✗ ✗ ✓Day of week ✗ ✗ ✗

26

Challenge 3: Incorporate the confounding factors

Refine the Model

27

MACHINE LEARNING

EngagementQuality Metrics

QoE Model

Confounding Factors

Adding as a feature Splitting the dataConfounding Factors 1e.g., Live, Mobile

ML

EngmntQualityMetrics

Model 2

ML

EngmntQualityMetrics

Model 1

ML

EngmntQualityMetrics

Model 3

Confounding Factors 2e.g., VOD, Mobile

Confounding Factors 3e.g., VOD, TV

QoE Model

28

Comparing Candidate Solutions

Final Model: Collection of decision treesFinal Accuracy- 70% (c.f. 40%) for 10% buckets

29

Summary of Our Approach

1. Capture complex engagement-to-metric relationships and metric-to-metric dependencies

Use Machine Learning

2. Identify confounding factors Tests 3. Incorporate confounding factors

Split

30

Evaluation: Benefit of the QoE Model

Preliminary results show that using QoE model to select bitrate leads to 20% improvement in engagement

31

Conclusions• Internet Video needs a unified and quantitative QoE model

• What makes this hard?– Complex relationships– Confounding factors (e.g., type of video, device)

• Developing a model– ML + refinements => Collection of decision trees

• Preliminary evaluation shows that using the QoE model can lead to 20% improvement in engagement

• What’s missing?– Coverage over confounding factors– Evolution of the metric with time

32

EXTRA SLIDES

34

Choice of ML Algorithm Matters

ML algorithm must be expressive enough to handle the complex relationships and interdependencies

• Classify engagement into uniform classes• Accuracy = # of accurate predictions/ # of cases