9th World Telecommunication/ICT Indicators

Meeting (WTIM-11) Mauritius, 7 - 9 December 2011

Contribution to WTIM-11 session Document C/16-E 8 December 2011

English

SOURCE: LIRNEasia

TITLE: Approaches to Measuring Broadband Quality of Service Experience (QoSE)

Approaches to Measuring Broadband Quality of Service

Experience (QoSE) Shazna Zuhyle

World Telecommunication / ICT Indicators Meeting

7-9 December 2011, Mauritius

This work was carried out with the aid of a grant from the International Development Research Centre, Canada and UKaid from the Department for International Development, UK.

Defining broadband access

Modes of Access

Wireless / Mobile

Network

Fixed Network

Mobile Nomadic

LIRNEasia tests QoSE for Fixed and Nomadic use

The user is stationary

Cell handover not required

Cell handover required

2

Defining test parameters Parameters Definitions Method Threshold /

Benchmarks

Download The rate at which data is received from a server Min. file size 1MB; Max. time 3 mins.

-

Upload The rate at which data is sent to a server 2 MB file -

Latency (RTT; Round Trip Time)

The time taken for a packet to reach its destination and back

The average of 10 pings (each ping provides 3 sets of results)

< 300 ms (e.g. India, Singapore)

Jitter The variation of latency (RTT) < 50 ms

Packet loss The fraction of packets that fail to reach its destination

< 3% (e.g. India, Singapore)

Availability The number of times the broadband link can be accessed within a given time frame = (1-F/T) x 100% [where T = total number of attempts; F = number of failed attempts]

> 98% (e.g. India)

3

It’s not just about download speeds

+++ Highly relevant; ++ Very relevant; + Relevant; - Irrelevant

Service Download (kbps)

Upload (kbps)

Latency (Round Trip Time, RTT) (ms)

Jitter (ms)

Packet Loss (%)

Browsing (Text) ++ - ++ - -

Browsing (Media) +++ - ++ + +

Downloading +++ - - - -

Transactions - - ++ + -

Streaming media +++ - ++ ++ ++

VOIP + + +++ +++ +++

Games + + +++ ++ ++

RTT has implications on client-server interactive systems

- Jitter adds to the ‘noise’ of the transmission

- Packet Loss affects streaming media

Source: Gonsalves, T.A & Bharadwaj, A. (2009) 4

Multiple domains; multiple times

4. Averages of multiple readings for each time slot 5. Varying server locations

- ISP domain - International domain - National domain

1. 6 Parameters: Download, Upload, RTT, Jitter, Packet Loss & Availability 2. Multiple days: during the week and on weekends 3. At six time slots: 0800, 1100, 1500, 1800, 2000, 2300 hrs (3 readings per slot)

Diagnostics can be used to identify potential bottle-necks in the

network. Value for money across packages can be compared.

Consumer protection can be enhanced based on data 5

Identifying bottlenecks in the network using the results

0

500

1000

1500

2000

2500

800 1100 1500 1800 2000 2300 Time

Download from ISP Domain

Qubee (1 Mbps) Dhaka, BD Banglalion (512 kbps) Dhaka, BD

Airtel (256 kbps) Bangalore, IN BSNL (256 kbps) Bangalore, IN

Airtel (2 Mbps) Chennai, IN BSNL (256 kbps) Chennai, IN

Airtel ( 2 Mbps) Mumbai, IN MTNL (320 kbps) Mumbai, IN

Airtel (2 Mbps) New Delhi, IN MTNL (512 kbps) New Delhi, IN

SLT (2 Mbps) Colombo, LK Telkom Speed (512 kbps) Jakarta, ID

800 1100 1500 1800 2000 2300 Time

Download from International Domain

The problem is in the International connectivity

Source: September 2011, LIRNEasia 6

The trouble with advertising theoretical speeds

7

0

5

10

15

20

25

30

35

40

45

0800 1100 1500 1800 2000 2300

Del

iver

ed v

s. A

dve

rtis

ed (

%)

Time

Dialog (upto 3.6 Mbps) Colombo, LK Mobitel (upto 1 Mbps) Colombo, LK Three (upto 3 Mbps) Jakarta, ID

Globe Tattoo (upto 3.6 Mbps) Manila, PH Smart (upto 3.6 Mbps) Manila, PH

Source: September 2011, LIRNEasia

Value for money

Source: September 2011, LIRNEasia 8

0

20

40

60

80

100

120

140

0800 1100 1500 1800 2000 2300

Do

wn

load

(kb

ps)

per

do

llar

Time

Qubee (1 Mbps) Dhaka, BD Banglalion (512 kbps) Dhaka, BD Airtel (256 kbps) Bangalore, IN

BSNL (256 kbps) Bangalore, IN Airtel (2 Mbps) Chennai, IN BSNL (256 kbps) Chennai, IN

Airtel ( 2 Mbps) Mumbai, IN MTNL (320 kbps) Mumbai, IN Airtel (2 Mbps) New Delhi, IN

MTNL (512 kbps) New Delhi, IN SLT (2 Mbps) Colombo, LK Telkom Speed (512 kbps) Jakarta, ID

Possible ways to monitor quality

Self regulation by operators

Direct monitoring by

regulators

User satisfaction

surveys

Demand-side (user) testing

Intrusiveness on network

+ +++/+ - +

Regulator participation

++/+/- +++ Depends on survey

-

Operator participation

+++ +++ Depends on survey

-

User participation

- - +++ +++

Subjectivity of results

++/+/- + +++ +

9

+++ High ++ Medium + low - none

Implementing QoSE indicators

• Ideally co-ordinated by regulators

• Follow one methodology so that results are comparative

• Undisclosed locations (not from the office)

• Publicly available results consumers can make informed decisions

• Policies can be based on diagnostics

10

Regulator’s efforts in South Asia

Regulator

Co

nsu

ltan

cy?

Met

ho

d

Pu

blis

he

d?

Re

po

rts

Pu

blis

he

d?

Test

fr

eq

ue

ncy

Parameters

Availability Throughput Latency (RTT)

Jitter Packet Loss

ATRA (AF) The indicators page is inaccessible – www.atra.gov.af

BTRC (BD) - - - - -

BICMA - - - - - - - - -

TRAI (IN) - Quarterly - -

CAM (MV) Website cannot be found

NTA - - - - - -

PTA - -

TRC (LK) - - Monthly - - - 11

Next Steps for LIRNEasia

• The software to be revamped

• Offered to regulators at no cost – just ask

LIRNEasia’s QoSE Diagnostic Tool (Beta):

www.broadbandasia.info

12

Key Takeaways

• There are other metrics apart from download speed that affect a user’s broadband experience

• These metrics (such as RTT, Jitter and Packet Loss) become more important when one goes beyond simply browsing the web

• Diagnostics should ideally be coordinated by regulators; not from a known location

• Results should be made public to benefit consumers

13

Thank You

More Info: www.lirneasia.net

Keywords: Benchmarks, ICT Indicators, Broadband QoSE

Contact: shazna@lirneasia.net

This work was carried out with the aid of a grant from the International Development Research Centre, Canada and UKaid from the Department for International Development, UK.