1 developing a predictive model for internet video quality-of-experience athula balachandran, vyas...
TRANSCRIPT
1
Developing a Predictive Model for Internet Video Quality-of-Experience
Athula Balachandran, Vyas Sekar,Aditya Akella, Srinivasan Seshan,
Ion Stoica, Hui Zhang
2
QoE $$$
CDN
UsersContentProviders
$$$Better QualityVideo
HigherEngagement
The QoE model
Diagram courtesy: Prof. Ramesh Sitaraman, IMC 2012
3
Adapting video bitrates quicker
Picking the best server
Why do we need a QoE model?
Comparing CDNs
4
Traditional Video Quality Metrics
Objective Scores (e.g., Peak Signal to Noise Ratio)
Subjective Scores(e.g., Mean Opinion Score)
Does not capture new effects (e.g., buffering, switching
bitrates)
User studies not representative of “in-the-wild” experience
5
Internet Video is a new ball game
Objective Scores (e.g., Peak Signal to Noise Ratio)
Subjective Scores(e.g., Mean Opinion
Score)
Engagement(e.g., fraction of video viewed)
Quality metrics
6
Commonly used Quality MetricsJoin Time
Average Bitrate
Buffering ratio
Rate of buffering
Rate of switching
7
Which metric should we use?
Objective Scores (e.g., Peak Signal to Noise Ratio)
Subjective Scores(e.g., Mean Opinion
Score)
Engagement(e.g., fraction of video viewed)
Quality metrics Buffering Ratio, Average bitrate?
Today:QualitativeSingle-metric
8
Unified and Quantitative QoE Model
Objective Scores (e.g., Peak Signal to Noise Ratio)
Subjective Scores(e.g., Mean Opinion
Score)
Engagement(e.g., fraction of video viewed)
Quality metrics Buffering Ratio, Average bitrate?
ƒ (Buffering Ratio, Average bitrate,…)
9
Outline
• What makes this hard?
• Our approach
• Conclusion
Complex Engagement-to-metric RelationshipsEn
gage
men
t
Quality Metric
10
Non-monotonic
E
ngag
emen
t
Average bitrate
En
gage
men
t
Rate of switching
Threshold
[Dobrian et al. Sigcomm 2011]
Ideal Scenario
11
Complex Metric Interdependencies
Join
Tim
e
Average bitrate
Average bitrate
Rate
of b
uffer
ing
Join Time Bitrate
Buffering ratio
Rate of buffering
Rate of switching
Confounding Factors
12
Confounding Factors can affect:1) Engagement
Live and Video on Demand (VOD) sessions have different viewing patterns.
CD
F (
% o
f use
rs)
Engagement
Type of VideoLive
VOD
Confounding Factors
13
Confounding Factors can affect:1) Engagement2) Quality Metrics
Type of VideoLive
VOD
CD
F (
% o
f use
rs)
Join Time
Live and Video on Demand (VOD) sessions had different join time distribution.
Confounding Factors
14
Confounding Factors can affect:1) Engagement2) Quality Metrics3) Quality Metric
Engagement
Type of Video
ConnectivityDSL/Cable
Wireless (3G/4G)
Eng
agem
ent
Rate of buffering
Users onwireless connectivity weremore tolerant to rate of buffering.
Confounding Factors
15
Type of Video
Connectivity
Device
Location
Popularity
Time of day
Day of week
Need systematic approach to identify and incorporate confounding factors
16
Summary of Challenges
1. Capture complex engagement-to-metric relationships and metric-to-metric dependencies.
2. Identify confounding factors3. Incorporate confounding factors
17
Outline
• What makes this hard?
• Our approach
• Conclusion
18
Challenge 1: Capture complex relationships
19
Cast as a Learning Problem
MACHINE LEARNING
Engagement Quality Metrics
QoE Model
Decision Trees performed the best.Accuracy of 40% for predicting within a 10% bucket.
20
Challenge 2: Identify the confounding factors
21
Test Potential Factors
Engagement
Confounding Factors
Quality Metrics
22
Test Potential Factors
Test 1: Relative Information Gain
Engagement
Confounding Factors
Quality Metrics
23
Test Potential Factors
Test 1: Relative Information GainTest 2: Decision Tree StructureTest 3: Tolerance Level
Engagement
Confounding Factors
Quality Metrics
24
Identifying Key Confounding Factors
Factor Relative Information Gain
Decision TreeStructure
Tolerance Level
Type of video ✓ ✓ ✓Popularity ✗ ✗ ✗Location ✗ ✗ ✗Device ✗ ✓ ✓Connectivity ✗ ✗ ✓Time of day ✗ ✗ ✓Day of week ✗ ✗ ✗
25
Identifying Key Confounding Factors
Factor Relative Information Gain
Decision TreeStructure
Tolerance Level
Type of video ✓ ✓ ✓Popularity ✗ ✗ ✗Location ✗ ✗ ✗Device ✗ ✓ ✓Connectivity ✗ ✗ ✓Time of day ✗ ✗ ✓Day of week ✗ ✗ ✗
26
Challenge 3: Incorporate the confounding factors
Refine the Model
27
MACHINE LEARNING
EngagementQuality Metrics
QoE Model
Confounding Factors
Adding as a feature Splitting the dataConfounding Factors 1e.g., Live, Mobile
ML
EngmntQualityMetrics
Model 2
ML
EngmntQualityMetrics
Model 1
ML
EngmntQualityMetrics
Model 3
Confounding Factors 2e.g., VOD, Mobile
Confounding Factors 3e.g., VOD, TV
QoE Model
28
Comparing Candidate Solutions
Final Model: Collection of decision treesFinal Accuracy- 70% (c.f. 40%) for 10% buckets
29
Summary of Our Approach
1. Capture complex engagement-to-metric relationships and metric-to-metric dependencies
Use Machine Learning
2. Identify confounding factors Tests 3. Incorporate confounding factors
Split
30
Evaluation: Benefit of the QoE Model
Preliminary results show that using QoE model to select bitrate leads to 20% improvement in engagement
31
Conclusions• Internet Video needs a unified and quantitative QoE model
• What makes this hard?– Complex relationships– Confounding factors (e.g., type of video, device)
• Developing a model– ML + refinements => Collection of decision trees
• Preliminary evaluation shows that using the QoE model can lead to 20% improvement in engagement
• What’s missing?– Coverage over confounding factors– Evolution of the metric with time
32
EXTRA SLIDES
34
Choice of ML Algorithm Matters
ML algorithm must be expressive enough to handle the complex relationships and interdependencies
• Classify engagement into uniform classes• Accuracy = # of accurate predictions/ # of cases