T.Gy. Intrernetes médiakommunikáció. 2009.05.

13.

1

Internetes médiakommunikáció

Az Internetes Médiakommunikáció minőségének megítélése, mérése, a

rendszer tervezési alapjai

Takács György 12. előadás2009. 05. 13.

What is QoE?

The overall acceptability of an application or service, as perceived subjectively by the end user.

NOTES1 Quality of Experience includes the complete end-

to-end system effects (client, terminal, network, services infrastructure, etc).

2 Overall acceptability may be influenced by user expectations and context. ~From ITU-T Recommendation P.10/G.100~

Factors affecting QoE of IPTV services

Why QoE evaluation?

Recent survey indicates that…. • 84% of IPTV providers think video quality monitoring is

critical or very important• 77% say video quality is a main reason for customer

churn• 78% say video quality problems are a main reason for

high volumes of customer service calls~From http://telephonyonline.com/iptv/news/iptv-video-

quality-0219/~

There surely is a market need for QoE evaluation methods/tools!

Quality monitoring as well as quality design

Conventional telephone services over PSTN are relatively stable after a call is connected.

• In PSTN services, service providers established telephone systems

• with static quality design, then, monitor/manage the “availability” and “functionality”

IP networks (core/access/home-NWs) and IP-based terminals are time-variant systems.

• In IPTV services, media quality varies depending on coding performance at H/E, traffic conditions in networks, and terminal performance

What to measure?Pre-transmission domain• Encode quality• System failure (blackout, frame freezing etc.)Network domain• • Packet loss• • Delay• • Delay jitterPost-transmission domain• Error recovery performance such as FEC (Forward Error

Correction)• Decode quality (including PLC: Packet Loss Concealment)• D/A conversion performance• Color management performance• Display performance

When to measure?

QoE measurement is necessary in the following phases:

Before the service is developed:• Individual quality factors (coding distortion etc.)• Quality planningAfter the service is developed, but before the service is

delivered:• Off-service quality checkAfter the service is delivered:• In-service quality monitoring at user end• In-service network mid-point monitoring• On-site quality check (on claim)

Where to measure?

How to measure?

• Subjective quality assessment is time consuming and expensive.

• It cannot be applied to on-site and/or real-time measurement, which is essential in quality monitoring.

• Objective quality assessment, which estimates QoE by subjective testing, is the solution!

Overview of objective quality assessment models

From the viewpoint of input information into a model, objective quality assessment models can be categorized as follows:

• Media-layer models• Parametric packet-layer models• Bitstream-layer models• Parametric planning models• Hybrid models

Media-layer models

Current standards of media-layer models (video)

• ITU-T standardized Rec. J.144, which recommends four fullreference models for Standard Definition TV, in 2004.– Coding distortion only (no transmission errors such as

packet loss)

• VQEG finalized its evaluation on full-reference and reducedreference models applicable to QCIF ~ VGA. The draft Recommendation has been consented at ITU-T SG9 as Rec. J.247, which includes four models, and Rec. J.246, respectively in May 2008.– Packet-loss degradation as well as coding distortion is in

the scope of Recommendation.

• ITU-T SG9 considered that current no-reference technologies seemed to be premature in terms of quality-estimation accuracy.

Parametric packet-layer models

• Only uses packet-header information.• Light-weight implementation.• Can be applied even if payload is encrypted.

Bitstream-layer models

• Use up to coded bitstream (payload), but not decoder output.

• Can take into account the content dependence of the impact of packet loss.

Standardization activities for packet-layer andbitstream-layer models

Parametric packet-layer models• ITU-T SG12 Q.14 & VQEG• • P.NAMS (non-intrusive parametric model

for the assessment of performance of multimedia streaming)

Bitstream-layer models• ITU-T SG12 Q.14 & VQEG• • P.NBAMS (non-intrusive parametric

bitstream model for the assessment of performance of multimedia streaming)

Parametric planning models

• Use quality planning parameters as input.• ITU-T Recommendation G.1070 for videophone applications.• ITU-T SG12 Q.13 is working on G.OMVAS (opinion model for video

and audio streaming applications).

Present state of multimedia qualityassessment methods

Future trends of quality assessmentresearch

• Multimodality

• Multiparty

• Wideband

• Examples of NTT’s studies on multimedia quality assessment for audiovisual communication services

Relevant ITU-T/R Recommendationson Multimedia Quality Assessment

• Multimedia quality assessment is at a reasonably advanced stage.

Existing Subjective Multimedia QualityAssessment Methods (P.911)

Multimedia quality is assessed in a similar way to individual audio/video qualities.

• Absolute Category Rating (ACR)• Degradation Category Rating (DCR)• Pair Comparison Method (PC)• Single Stimulus Continuous Quality• Evaluation (SSCQE)Assessment paying attention to the cross-modal

influences is important.• Interactions between differing quality levels in

different modalities

Existing Subjective Multimedia QualityAssessment Methods (P.920)

Several category judgment scales are used to evaluate multimedia quality.

• Overall audiovisual quality• Individual audio/video qualities• Effort needed to interrupt• Communication difficulty• Acceptability of communicationCommunication quality depends on tasks used in

conversational test.Assessment considering interactivity and usability

is also important.

Existing Concept Model of ObjectiveMultimedia Quality Evaluation (J.148)

Basic components of an objective multimedia quality model are defined.

Examples of Multimedia QualityEvaluation Model

Framework for ConductingQuality Assessment Research

Key Words for Future Trends ofMultimedia Quality Assessment (1/3)

Multimodality

• Combination of multiple media such as audio, video, text, graphics, fax, and telephony in the communication of information

• Key points of quality assessment

√ Intermedia synchronization

√ Intermedia quality balance

Key Words for Future Trends ofMultimedia Quality Assessment (2/3)

• Multiparty

• Communication style extending from 1:1 to N:N (e.g., instant messaging, teleconferencing, and distributed collaboration services)

• Key points of quality assessment

√ Interdestination synchronization

√ Interdestination quality balance

Key Words for Future Trends ofMultimedia Quality Assessment (3/3)

• Wideband• Telecommunications applications having more

bandwidth available for higher-quality multimodal services

• Key points of quality assessment

√ Psychological factors: We need to assess the richness of high-quality services not only on a one-dimensional scale, like MOS, but also on a multi-dimensional scale.

Examples of NTT’s Studies on QualityAssessment for “Multimodality” (1/3)

Examples of NTT’s Studies on QualityAssessment for “Multimodality” (2/3)

Examples of NTT’s Studies on QualityAssessment for “Multimodality” (3/3)

Examples of NTT’s Studies on QualityAssessment for “Multiparty” (1/2)

Quality imbalance is one of the multiparty quality degradation factors.

Examples of NTT’s Studies on QualityAssessment for “Multiparty” (2/2)

• Overall quality is strongly affected by the inferior quality at another point.

• Other-point audiovisual quality depends on conversation task or roles.

Examples of NTT’s Studies on QualityAssessment for “Wideband” (1/3)

• Interactive multimodal quality can be evaluated using a multi-dimensional scale of psychological factors.

Examples of NTT’s Studies on QualityAssessment for “Wideband” (2/3)

Psychological factors were extracted by using the semantic differential (SD) technique and factor analysis.

• Subject’s impression of an audiovisual communication service was evaluated on the basis of 25 pairs of bipolar adjectives on a seven-grade comparison scale.

Examples of NTT’s Studies on QualityAssessment for “Wideband” (3/3)

• Multimedia quality was formulated as a function of two psychological factors expressing an aesthetic feeling and a feeling of activity.