kpi definition and computation - dingli.com documents/white pap… · radio network unavailability...

KPI Definition and Computation

Table of Contents

1 KPI OVERVIEW ..................................................................................................................... 4

2 KPI DEFINITION .................................................................................................................... 4

3 KPI DIAGRAM........................................................................................................................ 6

3.1 CS Service .............................................................................................................. 6

3.1.1 Voice / Video Telephony ............................................................................... 6

3.2 PS Service .............................................................................................................. 7

3.2.1 FTP / HTTP / Email ...................................................................................... 7

3.2.2 WAP ............................................................................................................. 8

3.2.3 MMS ............................................................................................................. 9

3.2.4 Streaming ................................................................................................... 11

4 KPI DEFINITION ................................................................................................................... 11

4.1 Telephony ............................................................................................................. 12

4.1.1 Telephony Service Non-Accessibility [%] ................................................... 12

4.1.2 Call Setup Time [s] ..................................................................................... 12

4.1.3 Telephony Speech Quality on Sample Basis [MOS] .................................. 13

4.1.4 Telephony Cut-off Call Ratio [%] ................................................................ 13

4.2.1 VT Service Non-Accessibility [%] ............................................................... 13

4.2.2 VT Setup Failure Ratio [%] ........................................................................ 14

4.2.3 VT Service Access Time ............................................................................ 14

4.2.4 VT Setup Time [s] ....................................................................................... 15

4.2.5 VT Cut-off Call Ratio [%] ............................................................................ 15

4.2.6 Link Layer mean BLER [%] ........................................................................ 15

4.2.7 PEVQ [VMOS] ............................................................................................ 15

4.3.1 Ping Success Ratio [100%] ........................................................................ 16

4.3.2 Ping Round Trip Time [ms] ......................................................................... 16

4.4.1 Attach Failure Ratio [%].............................................................................. 16

4.4.2 Attach Setup Time [ms] .............................................................................. 17

4.4.3 PDP Context Activation Failure Ratio [%] .................................................. 17

4.4.4 PDP Context Activation Time [ms] ............................................................. 17

4.4.5 FTP Download Signalling Flow .................................................................. 18

4.4.6 FTP Download KPIs --- Method A .............................................................. 19

4.4.7 FTP Download KPIs --- Method B.............................................................. 19

4.4.8 FTP Download Signalling Flow .................................................................. 21

4.4.9 FTP Upload KPIs --- Method A .................................................................. 22

4.4.10 FTP Upload KPIs --- Method B ................................................................ 22



4.5.3 PDP Activation Failure Ratio [%] ................................................................ 23

4.5.4 PDP Activation Time [ms] ........................................................................... 23

4.5.5 HTTP Signalling Flow ................................................................................. 24

4.5.6 HTTP KPIs---Methods A............................................................................. 25

4.5.7 HTTP KPIs---Methods B ............................................................................ 25





4.6.5 E-mail Download Signalling Flow .............................................................. 27

4.6.6 E-Mail Download KPIs---Method A ............................................................ 28

4.6.7 E-Mail Download KPIs---Method B ............................................................ 28

4.6.8 E-Mail Upload Signalling Flow ................................................................... 29

4.6.9 E-Mail Upload KPIs---Method A ................................................................. 30

4.6.10 E-Mail Upload KPIs---Method B ............................................................... 30





4.7.5 MMS Signalling Flow.................................................................................. 32

4.7.6 MMS KPIs .................................................................................................. 33





4.8.5 WAP Signalling Flow .................................................................................. 36

4.8.6 WAP KPIs---Method A ................................................................................ 37

4.8.7 WAP KPIs ---Method B............................................................................... 37

1 KPI Overview

With the introduction of new networks services such as MMS, Email, HTTP, FTP,

Video Streaming, the requirement for RF network testing is increasingly critical. Compared

to traditional troubleshooting system, end-to-end KPI testing system is an initiative

solution which allows operator inspecting network from the perspective of end users, and

finishing troubleshooting before receiving user reports.

This document describes the KPI definition and algorithm adopted by Dingli's Pilot

Pioneer, the drive test solution, for services including voice services, video telephony, ping,

FTP, HTTP, Email, MMS, WAP.

2 KPI Definition

According to ITU specification, three aspects should be measured in order to evaluate

network performance from the perspective of the end-user perception.

1. Accessibility

Accessibility reflects user tolerance to services under certain conditions. For instance,

in order to access data service, user must perform a series of operations, such as

attaching network, activating PDP, accessing IP service while PDP is activated, activating

WAP session (for WAP and MMS), etc.

2. Retentivity

Retentivity is the ability of maintaining the service being accessed within a certain

period. The reliability of a service or session means that in a mature network, user does

not need to reproduce an operation, such as to re-activate PDP in data network.

3. Integrity

Integrity indicates that service is kept in good quality without interference after access

to service. Integrity of KPI reveals tremendous changes of network condition after a

service is accessed, which might fall from "very good" to "unacceptable".

Based on principles above, DingLi product measures end-to-end user perception

based on network key indicators (KPI) as follows:

Radio network unavailability

Attach failure rate

Attach setup time

Network non-accessibility

Setup time

Mean data rate

Data transfer cut-off ratio

IP service failure rate

IP service access time

3 KPI Diagram

3.1 CS Service

3.1.1 Voice / Video Telephony

3.2 PS Service

3.2.1 FTP / HTTP / Email

3.2.2 WAP

3.2.3 MMS

3.2.4 Streaming

4 KPI Definition

Currently two main views about the best way to reflect the user's experience for these

services are in place. One preferring the payload throughput philosophy and the other

preferring the transaction throughput philosophy:

- Method A defines trigger points which are as independent as possible from the

service used, therefore representing a more generic view (payload throughput).

- Method B defines trigger points on application layer, therefore representing a more

service oriented view (transaction throughput).

Kindly notice the mutual influence between KPI failure ratio and KPI time. A

parameter is not calculated unless the service setup attempt is successful. For instance,

KPI non-access ratio and setup time are mutually affected. If the duration for statistics is

set to be short, the mean setup time would be low and non-access ratio would be high

accordingly. And vice versa.

4.1 Telephony

For voice service, KPI is measured for MOC phone, while no KPI for MTC phone.

4.1.1 Telephony Service Non-Accessibility [%]

The probability that the end-user cannot access the mobile telephony service when

requested if it is offered by display of the network indicator on the UE.

unsuccessful call attempt

Telephony Service Non - Accessibility [%] = × 100

all call attempts

Start: The RRC Connection Request/Channel Request message is sent. It is possible that

more than one such message per call attempt is sent. Only the first one should be taken

into account for the calculation.

Stop: The Connect message is received from MSC. Stop trigger point is not reached for

an unsuccessful call.

4.1.2 Call Setup Time [s]

Call setup time describes the time period between sending call attempt and

connection established.

Call Setup Time[s] = Tconnection established —Tcall attempt

Start: RRC Connection Request / Channel Request message is sent. It is possible that




an unsuccessful call.

4.1.3 Telephony Speech Quality on Sample Basis [MOS]

An indicator representing the quantification of the end-to-end speech transmission

quality of the mobile telephony service. The score is measured based on PESQ algorithm.

Telephony Speech Quality on Sample Basis [MOS] =

sum of the PESQ score of all the sampling points

total number of PESQ sampling points

Start: The Connect message is received

Stop: The Disconnect message is sent.

4.1.4 Telephony Cut-off Call Ratio [%]

The probability that a successful call attempt is ended by a cause other than the

intentional termination by A- or B-party.

Unintentionally terminated telephony calls

Telephony Cut-off Call Ratio[%] = × 100

All successful telephony call attempts

Start: the Connect message is received.

Stop: the Disconnect message is sent.

4.2 Video Telephony

4.2.1 VT Service Non-Accessibility [%]

Probability that the end-user cannot access the service when requested while it is

offered by network indication on the mobile equipment.

unsuccessful video telephony call access attempts

VT Service Non-Accessibility [%] = × 100

All video telephony call access attempts





an unsuccessful call attempt.

4.2.2 VT Setup Failure Ratio [%]

Probability of receiving no video packets after VT call attempt.

VT setup failures

VT Setup Failure Ratio [%] = × 100

All VT call attempts

Start: RRC Connection Request / Channel Request message is sent. It is possible that more

than one such message per call attempt is sent. Only the first one should be taken into account for the

calculation.



4.2.3 VT Service Access Time

Time between pushing send button after input of MSISDN and receipt of alerting at MO

side.

VT Service Access Time [s] = Talerting tone- Tpush send button






4.2.4 VT Setup Time [s]

The elapsed time from the MT call acceptance indicated at MO side until audio and

video output starts at both sides.

VT Setup Time [s] =T audio/video start - T MO sees call acceptance

Start: RRC Connection Request / Channel Request message is sent.

Stop: Start of reception of audio and video data at both sides

4.2.5 VT Cut-off Call Ratio [%]

Probability that a successful service access is ended by a cause other than the

intentional termination of the user (calling or called party).

Video telephony dropped calls

VT Cut - off Call Ratio [%] = × 100

all successful video telephony service access attempts

Start: the Connect message is sent.

Stop: the Connect message is received.

4.2.6 Link Layer mean BLER [%]

The mean value of BLER on RLC layer

total BLER value (RLC layer)

BLER on RLC layer [%] = × 100

number of sampling points (RLC layer)

4.2.7 PEVQ [VMOS]

An objective method for end-to-end speech quality assessment of video telephony. By

simulating human visual system, it can detect abnormalities of VT signal and quantify KPI.

4.3 Ping Service

4.3.1 Ping Success Ratio [100%]

Probability of receiving ICMP echo reply from a destination after ICMP echo request is

sent.

ICMP echo reply is received by sender from destination

Ping Success Rate [%] = × 100

ICMP echo request from source

Start: ICMP echo request sent.

Stop: ICMP echo reply received by the sender.

4.3.2 Ping Round Trip Time [ms]

The round trip time is the time required for a packet to travel from a source to a

destination and back. It is used to measure the delay on a network at a given time. For this

measurement the service must already be established.

Ping Round Trip Time[ms] =T packet received - T packet sent

Start: ICMP echo request is sent.

Stop: ICMP echo reply is received by the sender

4.4 FTP service

4.4.1 Attach Failure Ratio [%]

The attach failure ratio describes the probability that a subscriber cannot attach to the

PS network

unsuccessful attach attempts

Attach Failure Ratio [%] = × 100

all attach attempts

Start: The "ATTACH REQUEST" message is sent by the UE.

Stop: The "ATTACH ACCEPT"/”Reject” message is received by the UE

4.4.2 Attach Setup Time [ms]

The attach setup time describes the time period needed to attach to the PS network.

Attach Setup Time [ms] = Tattach complete - Tattach request

Start: The "ATTACH REQUEST" message is sent by the UE.

Stop: The "ATTACH ACCEPT/Reject" message is received by the UE.

4.4.3 PDP Context Activation Failure Ratio [%]

PDP context cannot be activated probability. It is the attempt to the ratio of the total

number of unsuccessful attempts PDP context with the PDP context.

unsuccessful PDP context activation attempts

PDP Context Activation Failure Ratio[%] = × 100

all PDP context activation attempts

Start: The "PDP Context Activation Request" message is sent by the UE.

Stop: The "PDP Context Activation Accept" message is received by the UE

4.4.4 PDP Context Activation Time [ms]

The PDP context activation time describes the time period needed for activating the

PDP context.

PDP Context Activation Time [s] = T PDP context activation accept –T PDP context activation request

Start: The "PDP Context Activation Request" message is sent by UE.

Stop: the "PDP Context Activation Accept" message is received by UE.

4.4.5 FTP Download Signalling Flow

Trigger Point Corresponding Event

1 Attach Request message sent by UE

2 Attach Accept message received by UE

3 PDP Context Activation Request message sent by UE

4 PDP Context Activation Accept message received by UE

9 Connect server

10 Connection setup

11 RETR command sent

12 First packet received

13 Last packet received

4.4.6 FTP Download KPIs --- Method A

4.4.6.1 Download Mean Data Rate [kbit/s]

This parameter describes the average data transfer rate measured throughout the

entire connect time to the service.

User data transferred[KB]×8

Download Mean Data Rate [kbit/s] =

t data transfer complete –t data transfer start

Start: Reception of the first data packet containing content.

Stop: Reception of the last data packet containing content.

4.4.6.2 FTP Download Cut-off Ratio [%]

The proportion of incomplete data transfers and data transfers that were started

successfully.

incomplete data transfers

FTP Download Cut-off Ratio [%] = × 100

successfully started data transfers

Start: Reception of the first data packet containing content.

Stop: Reception of the last data packet containing content.

4.4.7 FTP Download KPIs --- Method B

4.4.7.1 Download Mean Data Rate [kbit/s]



user data transferred[KB]×8

Download Mean Data Rate [kbit/s] =

T last packet received –T RETR command sent

Start: RETR command is sent.

Stop: the receipt of the last packet.

4.4.7.2 FTP Download Cut-off Ratio [%]


successfully.


FTP Download Cut-off Ratio [%] = × 100


Start: RETR command is sent


4.4.8 FTP Download Signalling Flow






9 Connect server

10 Connection setup

11 STOR command sent


13 Last packet received. Confirmed by server

4.4.9 FTP Upload KPIs --- Method A

4.4.9.1 Upload Mean Data Rate [kbit/s]



User data transferred [KB]×8

Upload Mean Data Rate [kbit/s] =

T last packet received –T first packet sent

Start: the sending of the first packet.


4.4.9.2 FTP Upload Cut-off Ratio [%]


successfully.


FTP Upload Cut-off Ratio [%] = × 100


Start: the sending of the first packet;


4.4.10 FTP Upload KPIs --- Method B

4.4.10.1Upload Mean Data Rate [kbit/s]



User data transferred[KB]×8

Upload Mean Data Rate [kbit/s] =

T last packet received –T STOR command sent

Start: STOR command is sent.


4.4.10.2 FTP Upload Cut-off Ratio [%]


successfully.


FTP Upload Cut-off Ratio [%] = × 100


Start: STOR command is sent


4.5 HTTP


Kindly refer to relevant sections of FTP.



4.5.3 PDP Activation Failure Ratio [%]


4.5.4 PDP Activation Time [ms]


4.5.5 HTTP Signalling Flow






9 Connect to HTTP

10 “Start Transfer” command issued


13 Last packet received.

4.5.6 HTTP KPIs---Methods A

4.5.6.1 HTTP Mean Data Rate [kbit/s]

The average data transfer rate measured throughout the entire connect time to the

service.

HTTP file size [KB] × 8

HTTP Mean Data Rate [kbit/s] =

T last packet received –T first packet sent

Start: the receipt of the first packet.


4.5.6.2 HTTP Data Transfer Cut-off Ratio [%]


successfully.


HTTP Data Transfer Cut - off Ratio [%] = × 100


Start: the receipt of the first packet.


4.5.7 HTTP KPIs---Methods B

4.5.7.1 HTTP Mean Data Rate [kbit/s]


service.

HTTP file size [KB] × 8

HTTP Mean Data Rate [kbit/s] =

T last packet received –T transfer command issued

Start: The transfer command is issued


4.5.7.2 HTTP Data Transfer Cut-off Ratio [%]


successfully.


HTTP Data Transfer Cut-off Ratio [%] = × 100


Start: The transfer command is issued


4.6 Email









4.6.5 E-mail Download Signalling Flow






9 Connect server

12 RETR command sent; Download start



4.6.6 E-Mail Download KPIs---Method A

4.6.6.1 Mean Transfer Rate [kbit/s]


email service.

Email file size [KB] × 8

Mean Transfer Rate [kbit/s] =

T last packet received –T first packet received

Start: The first data packet is received.

Stop: The last data packet is received.

4.6.6.2 Email Data Transfer Cut-off Ratio [%]

The proportion of unsuccessful data transfers and data transfers that were started

successfully.

unsuccessful data transfers

Email Download Cut-off Ratio [%] = × 100

all successful started data transfers

Start: RETR command is sent. Email download starts

Stop: Last packet is received.

4.6.7 E-Mail Download KPIs---Method B



e-mail service.


E-mail Mean Data Rate[kbit/s] =

Tlast packet received – TRETR command issued



4.6.7.2 Email Download Cutoff Ratio [%]


successfully.


Email Download Cut-off Ratio [%] = × 100




4.6.8 E-Mail Upload Signalling Flow






9 Connect to server

12 Mail command issued

13 First packet sent

14 Last packet sent. Acknowledged by server

4.6.9 E-Mail Upload KPIs---Method A


The average data transfer rate measured throughout the entire connect time to the email

service.



T last packet sent –T first packet sent

Start: The first data packet is sent.

Stop: The last data packet is sent.

4.6.9.2 Email Data Transfer Cut-off Ratio [%]


successfully.


Email Upload Cut-off Ratio [%] = × 100


Start: The first data packet is sent.

Stop: Last packet is sent. Acknowledged by server.

4.6.10 E-Mail Upload KPIs---Method B



e-mail service.


E-mail Mean Data Rate[kbit/s] =

T last packet sent –T RETR command isued

Start: Mail command is issued.

Stop: Last packet is sent. Acknowledged by server.

4.6.10.2 Email Upload Cut-off Ratio [%]


successfully.


Email Upload Cut-off Ratio [%] = × 100


Start: Mail command is issued.

Stop: Last packet is received. Acknowledged by server.

4.7 MMS









4.7.5 MMS Signaling Flow


1 MMS Send attempt

2 Access WAP gateway attempt

3 Access WAP gateway success

4 MMS sent

5 MMS Notification received

6 MMS Receive attempt

7 Access WAP gateway attempt

8 Access WAP gateway success

9 MMS received

4.7.6 MMS KPIs

4.7.6.1 MMS Send Failure Ratio (MO) [%]

The probability that a MMS-message cannot be send by the subscriber, although he

has requested to do so by pushing the "send button"

unsuccessful MMS send attempts

(failed to access MMS_m-send.conf)

MMS Send Failure Ratio [%] = × 100

all MMS send attempts

Start: Attach Request message is sent.

Stop: MMS is sent (MMS m-send confirm)

4.7.6.2 MMS Retrieval Failure Ratio (MT) [%]

The probability that the MMS-message cannot be downloaded by the MT mobile,

which received a MMS notification before.

unsuccessful MMS delivery attempts

MMS Delivery Failure Ratio [%] = × 100

all MMS delivery attempts

Start: WAP Gateways is accessed (WSP/HTTP get request)

Stop: MMS is received (MMS_m-Notify Resp.ind).

4.7.6.3 MMS Send Time (MO) [s]

The time elapsing from pushing the send button after the editing of a MMS-message

to the completion of the data transfer.

MMS Send Time[s] = TMMS Sent complete – TPDP activation request initiated

Start: PDP Context Request message is sent.

Stop: MMS is sent (MMS m-send confirm)

4.7.6.4 MMS Retrieval Time (MT) [s]

The time elapsing between the WAP Get Request and the completion of the

download of the MMS.

MMS Delivery Time [s] =TMMS from MMSC complete – Tini WAP get request

Start: WAP Gateway is accessed (WSP / HTTP get request)

Stop: MMS is received completely (MMS_m-Notify Resp.ind).

4.7.6.5 MMS Notification Failure Ratio [%]

The probability that the Multimedia Messaging Service (MMS) is not able to deliver

the Notification of a MMS-message to the b-parties mobile.

failed MMS - notifications

MMS Notification Failure Ratio [%] = × 100

successfully submitted MMS

Start: MMS is sent completely (MMS m-send confirm)

Stop: MMS notification message is received (MMS_M_Notification_Ind).

4.7.6.6 MMS Notification Time [s]

The time elapsing from the complete submission of the Multimedia-Message to the

MMSC to the reception of the Notification (MT).

MMS Notification Time [s] = TrecNotif - TMMSsubmit

Start: MMS is sent completely. (MMS m-send confirm)

Stop: MMS notification message is received. (MMS_M_Notification_Ind).

4.7.6.7 MMS End-to-End Failure Ratio [%]

The probability that the Multimedia Messaging Service (MMS) is not able to deliver a

MMS-message after the "send button" has been pushed or the MO party has not received

an acknowledgement of the successful transmission from the MMSC.

unsuccessfully delivered MMS -messages

MMS End - to - End Failure Ratio [%] = × 100

all MMS send attempts

Start: PDP Context Request message is sent by UE.

Stop: the MMS is received completely (MMS_M_NotifyResp_Ind)

4.7.6.8 MMS End-to-End Delivery Time [s]

The time elapsing from pushing of the "send button" to the reception of the MMS by

the b-parties mobile.

MMS End-to- End Delivery Time [s] = TMMS rec – Tsend attempt

Start: PDP Context Request message is sent by UE.

Stop: MMS is received completely (MMS_M_NotifyResp_Ind)

4.8 WAP









4.8.5 WAP Signalling Flow






9 Access gateway attempt

10 Access gateway succeed/failed

11 Transfer command issued



4.8.6 WAP KPIs---Method A



WAP service.

WAP file size [KB] × 8


T last packet received –T first packet received


Stop: The last data packet is received.

4.8.6.2 WAP Data Transfer Cut-off Ratio [%]


successfully.


WAP Cut-off Ratio [%] = × 100



Stop: Last packet is received. Acknowledged by server.

4.8.7 WAP KPIs ---Method B



WAP service.

WAP file size [KB] × 8

Mean Data Rate[kbit/s] =

T last packet received–T transfer command issued

Start: Transfer command is issued. (HTTP Get or WSP Get)


4.8.7.2 WAP Data Transfer Cut-off Ratio [%]


successfully.


WAP Cut-off Ratio [%] = × 100


Start: Transfer command is issued. (HTTP Get or WSP Get)


kpi definition and computation - dingli.com documents/white pap… · radio network unavailability...

Documents