Chapter 43. Video Streaming Quality Index VSQI

NT13-22100 ver 1.0 UM-457

43. Video Streaming Quality Index VSQI

The information element VSQI (Video Streaming Quality Index) estimates the viewer-perceived video and audio quality achieved during video streaming sessions. This chapter takes a look the VSQI algorithm.

See also the document Video Streaming Quality Measurement with VSQI which is included in the TEMS Investigation documentation package.

Compare chapter 44 on MTQI.

43.1. General Properties of VSQILike SQI (chapter 41), VSQI is a no-reference method which judges the quality of the received signal on its own merits, without knowledge of the original.

The kind of subjective test which VSQI strives to imitate is one where viewers are instructed to assess both video and audio and combine their perception of each into an overall multimedia quality score.

The output from the VSQI algorithm is expressed as a value between 1 and 5, conforming to the MOS (Mean Opinion Score) scale which is frequently used in subjective quality tests. The unit for VSQI is called MOS-VSQI.

43.2. What VSQI Is Based OnThe VSQI score is based on the following non-perceptual input:

1 The quality of the encoded (compressed) signal prior to transmission. This quality is straightforwardly a function of the video and audio codecs used, and their bit rates. The information actually used by the VSQI algorithm is the video codec type and the total (video + audio) bit rate. The clean quality has been computed in advance for the codecs listed in section 43.4.1.

TEMS Investigation 15.2 Users Manual

UM-458

2 The amount of initial delay and the subsequent interruptions during playback of the video sequence: that is, the time required for initial buffering and the incidence of rebuffering.

3 The amount of packet loss at the application level (i.e. in the video streaming client).

43.3. What VSQI Does Not ConsiderVSQI does not directly consider the signal presented to the human viewer; that is, no analysis of perceptual input is performed to detect specific visible artifacts. The transferred video is not analyzed frame by frame in any way. Thanks to the monitoring of packet loss (item no. 2 in section 43.2 above), however, even slight problems with blockiness, jitter, and so on will still be noticed by the algorithm and affect the VSQI score.

43.4. Static and Dynamic VSQITwo versions of the VSQI algorithm have been devised: one static and one dynamic version.

Static VSQI is presented in the event Streaming Quality VSQI. It does not appear as an information element. Dynamic VSQI, on the other hand, is contained in the information element Streaming VSQI.

43.4.1. Static VSQIThe static version of VSQI takes an entire streamed video clip as input and assigns a single quality score to it.

Input parameters to the static version of VSQI are as follows:

Video codec used (H.263, H.264, or MPEG4)

Total bit rate (video + audio)

Duration of initial buffering

Number of rebuffering periods

Duration of rebuffering periods

Amount of packet loss

With some degree of simplification, we may describe the calculation of static VSQI with the following formula:

Chapter 43. Video Streaming Quality Index VSQI

NT13-22100 ver 1.0 UM-459

Here, VSQIclean is the clean value obtained for the clip prior to transmission. This score is determined by the quality of the encoding, which is in turn dependent on the choice of codecs and bit rate.

The size of the buffering penalty depends on the time taken for initial buffering, the time spent rebuffering, and the number of rebuffering events.

The size of the packet loss penalty is determined as follows. A running packet loss average over the last 4 s is computed approximately every second, and the values thus obtained are weighted and summed to yield an appropriate overall measure of the packet loss. The latter is then translated into a deduction from the VSQI score.

The static VSQI algorithm has been fine-tuned for clips of around 30 s and should therefore in practical use be applied to clips of similar duration. The video sequences must not be too short because of how the buffering works: each instance of rebuffering takes several seconds to complete, and moreover if the clip is short enough it will have been buffered in its entirety before the replay starts, so that no rebuffering will ever occur. For clips considerably longer than 30 s, on the other hand, disturbances towards the end will be more harshly penalized by viewers than those occurring early on, simply because the late ones are remembered more vividly. Therefore, since the current VSQI algorithm does not take into account such memory effects, it would probably perform slightly worse for long clips. (The dynamic version of VSQI naturally is not affected by this limitation.)

43.4.2. Dynamic (Realtime) VSQIThe dynamic or realtime version of VSQI estimates the quality of a streaming video clip as perceived by viewers at a moment in time. It is updated regularly at intervals of the order of 1 s while the video clip is playing. Each VSQI output value is dependent on the recent history of the streaming session (i.e. recent packet loss levels and possible recent buffering events).

The design of dynamic VSQI is based on the following:

Previous research suggesting approximate times taken for the perceived quality to drop to MOS-VSQI 1 (during buffering) and to rise to the highest attainable VSQI (during normal replay)

Modeling of the impact of packet loss on perceived quality

TEMS Investigation 15.2 Users Manual

UM-460

Tailoring of mathematical functions for expressing viewer annoyance/satisfaction as a function of time (in each of the states that are possible during replay)

Codec and bit rate parameters as in the static version

The graph below shows in rough outline the different ways in which dynamic VSQI can evolve during the replay of a streaming video clip. The best achievable quality, i.e. the ceiling in the graph, is dependent on the codec/bit rate combination but is also affected by the amount of packet loss. In this example the packet loss is assumed to be constant so that the influence of buffering can be clearly discerned.

1 The user tolerates (and might even expect) a certain amount of initial delay; but the longer the buffering drags on, the more the user loses patience.

2 Once the replay gets going, the perceived quality picks up again and soon approaches the highest achievable level.

3 If rebuffering occurs, VSQI deteriorates rapidly. Rebuffering events are much less tolerated by viewers than initial buffering, especially if repeated; VSQI captures the latter by making the slope of the curve steeper for each new rebuffering event.

4 After the replay has recommenced, VSQI recovers reasonably quickly, but not infrequently from a rock bottom level.

Chapter 44. Mobile TV Quality Index MTQI

NT13-22100 ver 1.0 UM-461

44. Mobile TV Quality Index MTQI

MTQI (Mobile TV Quality Index) is a refinement of the video part of the VSQI quality measure (see chapter 43). Unlike VSQI, MTQI does not include an assessment of audio quality.

The MTQI algorithm can be concisely characterized as follows.

Algorithm components:

Modeling of clean quality

Modeling of packet loss

Modeling of corruption duration (total duration of corrupted frames)

Buffering with and buffering without skipping are distinguished. Buffering with skipping means that frames are skipped in connection with buffering; no skipping means that every frame is replayed.

Supported video codecs: H.263, H.264, REAL, MPEG4

Supported video formats: QCIF, QVGA

TEMS Investigation 15.2 Users Manual

UM-462

45. VQmon Video and Audio Quality Metrics

This chapter deals with certain aspects of VQmon video and audio perceptual quality metrics, which are computed in the course of HTTP streaming. The metrics are listed in Information Elements and Events, section 3.8: Streaming IEs; for the service testing setup, see section 12.16.6.4 of the present document.

The VQmon algorithms have been developed by Telchemy, and the information that follows is taken from Telchemy documentation.

45.1. VQmon Mean Opinion Scores (MOS)VQmon provides a set of Mean Opinion Scores (MOS) estimating the quality of each video and audio stream as perceived by end-users. Each MOS value ranges from 1 to 5, where 1 represents the verdict Unacceptable and 5 means Excellent:

MOS-V: Video MOS, considering the effects of the video codec, frame rate, packet loss distribution, and group-of-pictures structure on video quality.

MOS-A: Audio MOS, considering the effects of the audio codec, bit rate, sample rate, and packet loss on viewing quality.

MOS-AV: AudioVideo MOS, considering the effects of both picture and audio quality as well as audiovideo synchronization on the overall user experience.

45.1.1. Absolute and Relative MOS-VWhen comparing MOS values, it is important to consider that some types of video inherently produce a higher level of quality than others. Relying solely on absolute MOS values can be misleading when comparing dissimilar types of video service, as viewers tend to form expectations of quality based in part on the perceived capabilities of the medium.

Chapter 45. VQmon Video and Audio Quality Metrics

NT13-22100 ver 1.0 UM-463

For example, a video viewed on a handset with a small screen might receive an absolute MOS of 3.1 when little or no quality degradation is evident, while for a handset with a larger screen, the same MOS value might suggest that some noticeable impairments were present.

To facilitate quality comparisons between different video service types, VQmon provides both Absolute and Relative MOS-V:

Absolute MOS-V considers the impact of frame resolution, frame rate, codec, compression level, transmission impairments, and frame loss concealment on video quality.

Relative MOS-V considers the impact of all of the factors used to determine Absolute MOS-V except frame resolution, producing a MOS relative to the ideal for the current video format.

All VQmon MOS scores are reported as separate instantaneous, minimum, maximum, and average values, the last three spanning the current streaming session.

45.2. Video Service Transmission Quality (VSTQ)The VQmon metric Video Service Transmission Quality (VSTQ) is a codec-independent score that measures the ability of the network to reliably transport video. VSTQ is expressed in the range 050.