definition kpis edge doc

Definition of GPRS/EDGE End-to-End

KPIs

Abstract

The document provides the definition of GPRS/EDGE end-to-end Key Performance

Indicators (KPIs), which make a description of the end user’s perception of the service

performance possible.

• Definition of GPRS/EDGE End-to-End KPIs • • Introduction •

Table of Contents

1 I�TRODUCTIO� 4

1.1 SCOPE 4 1.2 STRUCTURE 4 1.3 REVISION HISTORY 5

2 REMARKS O� THE �ATURE OF THE KPIS 6

3 TERMI�OLOGY USED FOR CATEGORIZI�G THE KPIS 6

3.1 ACCESSIBILITY 6 3.2 RETAINABILITY 7 3.3 INTEGRITY 7 3.4 DISTINCTION BETWEEN KPIS AND PERFORMANCE INDICATORS (PIS) 7

4 SERVICES 9

5 DEFI�ITIO�S OF KPIS 11

5.1 CATEGORIZATION OF THE KPIS 11 5.2 ACCESSIBILITY KPIS 12

5.2.1 GPRS Unavailability 12 5.2.2 GPRS Attach Failure Ratio 13 5.2.3 GPRS PDP Context Activation Failure Ratio 15 5.2.4 Secondary PDP Context Activation Failure Ratio 17 5.2.5 GPRS Service Access Failure Ratio (for each service) 19 5.2.6 GPRS Accessibility Failure Ratio (for each service) 24

5.3 RETAINABILITY KPIS 26 5.3.1 GPRS PDP Context Cut-off Ratio 26 5.3.2 GPRS Data Transfer Cut-off Ratio 27 5.3.3 Cell Update Failure Ratio 29 5.3.4 Intra SGS, RAU Failure Ratio 31 5.3.5 Inter SGS, RAU Failure Ratio 32 5.3.6 PS Paging Failure Ratio 34

5.4 INTEGRITY KPIS 35 5.4.1 GPRS Attach Setup Time 35 5.4.2 GPRS PDP Context Activation Delay (per AP,) 36 5.4.3 GPRS Service Access Time (for each service) 37 5.4.4 GPRS Access Time (for each service) 41 5.4.5 GPRS Mean User Data Rate 42 5.4.6 Round Trip Time 44 5.4.7 Client-Server Interaction Time with Background Traffic 46 5.4.8 Packet Loss Ratio 47 5.4.9 Cell Update Time 49 5.4.10 Intra SGS, RAU Time 50 5.4.11 Inter SGS, RAU Time 52

6 OVERVIEW OF THE KPIS 53

7 SUMMARY 54

8 TERMI�OLOGY 56

8.1 ABBREVIATIONS 56 8.2 GLOSSARY 57

9 REFERE�CES 58

10 APPE�DIX 59

• Introduction • • Definition of GPRS/EDGE End-to-End KPIs •

10.1 STATISTICAL GUIDELINES 59

• Definition of GPRS/EDGE End-to-End KPIs • • Introduction •

1 Introduction

1.1 Scope

One aspect of quality when it comes to GPRS/EDGE is the performance perceived by

the end-user. Key Performance Indicators are those indicators that have big impact on

the end-user’s perception of the service performance. Therefore the KPI definitions

make a description of the end user’s perception of the performance possible.

Additionally they enable operators to compare GPRS/EDGE performance between their

own networks and with the performance of competitors’ networks.

Purpose of this document is to define GPRS/EDGE end-to-end Key Performance

Indicators (KPIs). In order to ensure comparable results when measuring KPIs the

definition of each KPI has to outline what should trigger the start of the measurement

and the stop of the measurement. This start and stop trigger can be certain signaling

messages or a specific data packet. Therefore the definitions include these trigger points

that are to be used when the KPIs are measured in a live network or in a lab

environment.

The document does not:

• specify test cases that should be run in order to measure the end-to-end performance,

• give benchmark figures for the different KPIs,

• describe parameter settings for an optimized GPRS/EDGE network.

The above-mentioned items are covered in the Generic Test Description for

GPRS/EDGE End-to-End Performance Testing (see [1]), and in the Ericsson GPRS

End-to-End Performance Guidebook (see [3]).

1.2 Structure

The document has the following structure:

• Nature of the KPIs

• Terminology used for categorizing the KPIs

• Definitions of the KPIs

• Appendix: Statistical Guidelines

• Introduction • • Definition of GPRS/EDGE End-to-End KPIs •

1.3 Revision History

• Definition of GPRS/EDGE End-to-End KPIs • • Remarks on the Nature of the KPIs •

2 Remarks on the Nature of the KPIs Main focus of the KPI definitions is that the KPIs shall reflect the end-user’s perception

of the service performance. The KPIs shall be measurable either with active or passive

tests.

Active Measurements

Measuring with active tests means that the person who performs the tests actively

generates traffic and only the performance for this generated traffic is measured. These

measurements are typically limited to a certain period of time, the conditions of the

system under test are known and detailed measurements and investigations are possible.

Test cases for performing active measurements are defined in the Generic Test

Description for GPRS/EDGE End-to-End Performance Testing [1].

Passive Measurements

Measuring with passive tests means that the performance of traffic generated by all

subscribers is continuously measured via counters in any node of the GPRS/EDGE

network (refer to Measurements of GRPS End-to-End KPIs with Counters [6])1 or with

specific tools for passive measurements. Detailed conditions are often unknown, e.g. cell

load, number of used TS etc.

3 Terminology Used for Categorizing the KPIs Terminology for grouping the KPI definitions follows the international ITU-T standards

(E.800), in order to facilitate alignment of KPI definitions between Ericsson and

vendors. Since standard ITU-T E.800 is valid for telephone network and ISDN only, the

categorization of the GPRS/EDGE e2e KPIs definitions borrows the terminology used in

the standard. This means that the KPI definitions are not standard-compliant, but that

concepts deployed in the standard are transferred onto the grouping of the KPIs.

According to ITU Recommendation E.800 [4] Quality of Service is: The collective

effect of service performances, which determine the degree of satisfaction of a user of

the service.

3.1 Accessibility

The ability of the user to obtain a service within specified tolerances and other given

conditions.

The accessibility of the service does include a chain of operations the user must be able

to execute:

• the user must be able to access GPRS/EDGE as such, meaning he must be able to do

a GPRS attach and a PDP context activation, and

• he must be able to access an IP service after he has an activated PDP context.

1 General remarks on measurement of KPIs with counters: The counters give information about the success ratios,

but not about the failure cause code.

• Terminology Used for Categorizing the KPIs • • Definition of GPRS/EDGE End-to-End KPIs •

The accessibility KPIs are the most important set of KPIs regarding IP services, since

the user is first of all interested in the fact if he is able to access the GPRS/EDGE

network and the IP services or not. Accessibility KPIs are defined in chapter 5.2.

3.2 Retainability

Retainability indicates the ability of the user to keep a service – once it was accessed -

under given conditions for a requested period of time.

The retainability of a service or session does include that the user does not have to

perform any additional manual operations that would not be necessary under stable

network conditions, like for example manual re-activation of the PDP context.

Retainability KPIs are defined in chapter 5.3.

3.3 Integrity

Integrity indicates the degree to which a service is provided without major interferences,

once it was accessed.

Integrity KPIs show the performance of successful service attempts. Even if a service

was accessed successfully, the user’s perception of the performance may vary between

very good and unacceptably bad. Integrity KPIs are defined in chapter 5.4.

3.4 Distinction between KPIs and Performance Indicators (PIs)

Some of the performance indicators are rated to be the primary performance indicators,

because they reflect the end-user’s perception of the service performance in an

outstanding manner. Therefore they are categorized as Key Performance Indicators,

while the performance indicators that contribute to the measurement of a KPI are

categorized as Performance Indicator (PI).

This categorization into KPIs and PIs is deployed in cases in which such a distinction

can be clearly drawn, i.e.:

Accessibility

� KPI ACC 050 GPRS Accessibility Failure Ratio [%]

PI ACC 010 GPRS Unavailability [%]

PI ACC 020 GPRS Attach Failure Ratio [%]

PI ACC 030 GPRS PDP Context Activation Failure Ratio [%]

PI ACC 040 GPRS Service Access Failure Ratio [%]

Retainability

� KPI RET 020 Data Transfer Cut-off Ratio

PI RET 010 PDP Context Cut-off Ratio

• Definition of GPRS/EDGE End-to-End KPIs • • Terminology Used for Categorizing the KPIs •

Integrity

� KPI I�T 040 GPRS Access Time [s]

PI I�T 010 GPRS Attach Setup Time [ms]

PI I�T 020 GPRS PDP Context Activation Delay [ms]

PI I�T 030 GPRS Service Access Time [s]

• Services • • Definition of GPRS/EDGE End-to-End KPIs •

4 Services The following services should be taken into consideration when defining KPIs and

measuring them with specific test cases:

• GPRS access service

• WAP

• E-Mail

• Web browsing/HTTP

• FTP

• MMS

• Streaming

When measuring KPIs related to the above mentioned services in a live network file and

e-mail sizes cannot be influenced, so that passive tests of KPIs will measure any file and

e-mail size that appears in a live network.

But when measuring the KPIs related to services with active tests the following file and

e-mail sizes should be chosen:

WAP: Fixed WAP pages to be downloaded have to be defined.

WAP browser to be used has to be defined.

E-Mail: Send e-mail: 120 Kbytes zip-file (e-Mail with attachment)

Receive e-mail: 240 Kbytes zip-file (e-Mail with attachment)

Send e-mail: 5 Kbytes plain text

Receive e-mail: 5 Kbytes plain text

No use of Java or Flash

Web/HTTP

:

Fixed web pages to be downloaded have to be defined, e.g.

www.ericsson.com

Internet browser to be used has to be defined.

Version of the internet browser has to be defined.

FTP: Uplink: 100 Kbytes

Downlink: 200 Kbytes

• Definition of GPRS/EDGE End-to-End KPIs • • Services •

Timeouts2 for Data Transfer per Service:

The timeout values summarized in Table 1 are only applicable for active measurements

of KPIs related to services.

The timeout values for e-mail download comprise only the download of the data content

(e-mail text, or e-mail text plus attachment) but not the access procedure preceding the

actual download of the data content, see chapter 5.2.4.

HTTP

File Size 40kB 120kB 40kB 240kB 120kB 5kB

5kB text +

120 kB

attachment

240kB 5kB

5kB text +

240 kB

attachment

frozen

page

Timeout 64s 192s 32s 192s 192s 28s 200s 192s 24s 196s

Data Transfer (content)

FTP UL FTP DL E-Mail UL E-Mail DL

Table 1: Timeout values for data transfer

Remark

Purely calculated timeout values for uploading 5kB and downloading 5kB would result

in 8s and 4s respectively. But the actual timeout values being applied while measuring

service-related KPIs with small files will be higher, because of the slow start behavior of

TCP or TCP timeouts which increase the delay to several seconds.

Therefore the timeouts are calculated as follows:

20s + <download size> / UL performance [5kbps]

20s + <download size> /DL performance [10kbps]

2 Timeouts are calculated assuming DL/UL via one timeslot (worst-case scenario). Example: FTP DL 240kB,

240 kB * 8 = 1920 Kbits, 1920 Kbits/10 kbps = 192 seconds

• Definitions of KPIs • • Definition of GPRS/EDGE End-to-End KPIs •

5 Definitions of KPIs

5.1 Categorization of the KPIs

The KPI definitions are divided into three categories as shown below:

QoS KPIs

Accessibility Retainability Integrity

GPRS/UMTS unavailability

Attach failure ratio

PDP context activation failure ratio

Service access failure ratio

e2e accessibility failure ratio

PDP context cut-off ratio

Data transfer cut-off ratio

Cell Update Failure Ratio

Intra SGSN RAU Failure Ratio

Inter SGSN RAU Failure Ratio

SGSN Paging Failure Ratio

Attach setup time

PDP context activation delay

Service access time

e2e access time

Mean user data rate

Round Trip Time

Packet Loss Ratio

Cell Update Time

Intra SGSN RAU Time

Inter SGSN RAU Time

QoS KPIs

AccessibilityAccessibility Retainability Integrity

GPRS/UMTS unavailability

Attach failure ratio

PDP context activation failure ratio

Service access failure ratio

e2e accessibility failure ratio

PDP context cut-off ratio

Data transfer cut-off ratio

Cell Update Failure Ratio

Intra SGSN RAU Failure Ratio

Inter SGSN RAU Failure Ratio

SGSN Paging Failure Ratio

Attach setup time

PDP context activation delay

Service access time

e2e access time

Mean user data rate

Round Trip Time

Packet Loss Ratio

Cell Update Time

Intra SGSN RAU Time

Inter SGSN RAU Time

• Definition of GPRS/EDGE End-to-End KPIs • • Definitions of KPIs •

5.2 Accessibility KPIs

5.2.1 GPRS Unavailability

TAG: ACC 010

Definition:

The PI GPRS/EDGE Unavailability [%] denotes the probability that GPRS/EDGE is not

active in the cell used by a subscriber.

Trigger Point(s):

Check if the GPRS/EDGE-specific signaling (SI13) exists on cell selection.

Check if the signaling message can be read out and if the required signaling exists on

BCCH or on PBCCH.

%10013#

13#[%] ∗=

SIreadtoattemptstotal

SIreadtoattemptsulunsuccessflityUnavailabiGPRS

Remark:

If this PI is measured for example during drive tests information is provided about the

GPRS/EDGE coverage on network level and coverage problems in special.

Since this PI can only be measured in live networks the PI is measured for a random set

of cells in the live network, since the GPRS/EDGE availability is checked before any

test case can be executed.

Active Measurement:

PI not covered in the Generic Test Description for GPRS/EDGE End-to-End

Performance Testing.

Passive Measurement:

There is no means of passive measurement (counter or Moniq tool) for this PI.


5.2.2 GPRS Attach Failure Ratio

TAG: ACC 020

Definition:

The PI GPRS Attach Failure Ratio [%] denotes the probability that a subscriber cannot

attach to the GPRS/EDGE network.

The timeout interval for the attach procedure is 30 seconds, meaning: if the attach

procedure was not completed after 30 seconds it is considered as failure. Note that the

timeout value of 30 seconds is no system internal value.

%100#

#[%] ∗=

attemptsattachGPRSoftotal

attemptsattachGPRSulunsuccessfRatioFailureAttachGPRS

Trigger Point(s):

Check if the whole GPRS attach procedure from sending the Attach Request and

receiving the Attach Accept message was executed successfully (Signaling messages 1

and 8 in Figure 1).

7d. Cancel Location Ack

7c. Cancel Location

7b. Update Location

7g. Update Location Ack

7e. Insert Subscriber Data

7f. Insert Subscriber Data Ack

6d. Insert Subscriber Data

6c. Cancel Location Ack

6b. Cancel Location

3. Identity Response

2. Identification Response

2. Identification Request

1. Attach Request

5. IMEI Check

3. Identity Request

4. Authentication

6a. Update Location

7a. Location Update Request

7h. Location Update Accept

6f. Update Location Ack

6e. Insert Subscriber Data Ack

MS BSS new SGSN old SGSN GGSN HLR EIR

old

MSC/VLR

new

MSC/VLR

9. Attach Complete

8. Attach Accept

10. TMSI Reallocation Complete

Figure 1: Signalling Messages during Attach Procedure


Remarks:

It might occur that the MS sends more than one Attach Request towards the SGSN,

since retries are necessary. Maximum four retries are possible and the guard time

between two requests is 15 seconds. These retries should not have impact on the Attach

Failure Ratio, since only one GPRS/EDGE Attach Request message should be counted

in the calculation.

Prerequisites:

GPRS/EDGE has to be active in the cell used by a subscriber (PI ACC 010)

Active Measurement:

This PI can be measured by executing the following test case described in detail in the

Generic Test Description for GPRS/EDGE End-to-End Performance Testing [1]:

GPRS Attach, TAG: ST 010.


This PI can be measured with a counter in the /// SGSN.


5.2.3 GPRS PDP Context Activation Failure Ratio

TAG: ACC 030

Definition:

The PI GPRS PDP Context Activation Failure Ratio [%] denotes the probability that a

subscriber cannot activate a PDP context.

The timeout interval for the PDP context activation procedure is 30 seconds, meaning: if

the activation procedure was not completed after 30 seconds it is considered as failure.

Note that the timeout value of 30 seconds is no system internal value.

%100#

#[%] ∗=

attemptsactivationcontextPDPoftotal

attemptsactivationcontextPDPulunsuccessfRatioFailureActivationContextPDP

Trigger Point(s):

Check if the PDP context activation procedure from sending the Activate PDP Context

Request and receiving the Activate PDP Context Accept message was executed

successfully. (Signaling messages 1 and 7 in Figure 2.)

2G-GGSN

7. Activate PDP Context Accept

5. Create PDP Context Response

5. Create PDP Context Request

1. Activate PDP Context Request

2G-SGSN BSS

2. Security Functions

MS

6. BSS Packet Flow Context Procedures

4. Invoke Trace

Figure 2: Signaling Messages during the PDP Context Activation Procedure

Remarks:

It might occur that the MS sends more than one PDP Context Activation Request

towards the SGSN, since retries are necessary. Maximum four retries are possible and

the guard time between two requests is 30 seconds. These retries should not have impact

on the Activation Failure Ratio, since only one GPRS PDP Context Activation Request

message should be counted in the calculation.


PDP Context Activations that failed, because they were done towards an undefined APN

should not be included in the calculation.

Prerequisites:

The MS is already GPRS attached. (PI ACC 020)

Active Measurement:



PDP Context Activation, TAG: ST020.


This PI can be measured with a counter in the /// SGSN-G 3.0.


5.2.4 Secondary PDP Context Activation Failure Ratio

TAG: ACC 035

Definition:

The PI Secondary PDP Context Activation Failure Ratio [%] denotes the probability that

a subscriber cannot activate a Secondary PDP context.

The timeout interval for the Secondary PDP context activation procedure is 30 seconds,

meaning: if the activation procedure was not completed after 30 seconds it is considered

as failure. Note that the timeout value of 30 seconds is no system internal value.

%100#

#[%] ∗=

attemptsactivationcontextPDPoftotal

attemptsactivationcontextPDPulunsuccessfRatioFailureActivationContextPDP

Trigger Point(s):

Check if the PDP context activation procedure from sending the Activate Secondary

PDP Context Request and receiving the Activate Secondary PDP Context Accept

message was executed successfully. (Signaling messages 1 and 7 in Figure 2 and Figure

4.)

2G-GGSN

7. Activate Secondary PDP Context Accept



1. Activate Secondary PDP Context Request

2G-SGSN BSS


MS

5. BSS Packet Flow Context Procedures

Figure 3: Secondary PDP Context Activation Procedure for GPRS


3G-GGSN

7. Activate PDP Context Accept



1. Activate Secondary PDP Context Request

3G-SGSN UTRAN MS

4. Radio Access Bearer Setup

6. Update PDP Context Response

6. Update PDP Context Request

Figure 4: Secondary PDP Context Activation Procedure for UMTS

Remarks:

It might occur that the MS sends more than one PDP Context Activation Request

towards the SGSN, since retries are necessary. Maximum four retries are possible and

the guard time between two requests is 30 seconds. These retries should not have impact

on the Activation Failure Ratio, since only one GPRS PDP Context Activation Request

message should be counted in the calculation.

Prerequisites:

The MS is already GPRS attached and PDP context activated. (PI ACC 020 & PI

ACC 030)

Active Measurement:



PDP Context Activation, TAG: ST020.




5.2.5 GPRS Service Access Failure Ratio (for each service)

TAG: ACC 040

Definition:

The PI GPRS Service Access Failure Ratio [%] denotes the probability that a subscriber

cannot access the service successfully, meaning that the data transfer of the content

could not be started.3

%100#

#[%] ∗=

attemptsaccessserviceoftotal

attemptsaccessserviceulunsuccessfRatioFailureAccessService

The trigger points as well as the timeout values have to be defined per service for PI

ACC 040.

Trigger Point(s):

The trigger points are defined per service:

FTP (UL)

Start trigger: MS sends its port (signaling message 2 in Figure 5).

Stop trigger: Sending of the first data packet (message 11 in Figure 5).

Figure 5: FTP UL trace

Note: The actual start trigger is the TCP SYN for port 21. If this SYN is not send the

port is not opened and the service could not be accessed. This signalling

message is not visible in the above log since the port was opened much earlier.

3 Content denotes different things depending on the service that is accessed. In case of an FTP session content is a

file, in the case of an http session a web page and content of an E-Mail session is the text of the mail and so on.


FTP (DL)

Start trigger: MS sends its port (signaling message 1 in Figure 6).

Stop trigger: Reception of the first data packet (message 10 in Figure 6).

Figure 6: FTP DL trace

Note: The actual start trigger is the TCP SYN for port 21. If this SYN is not send the

port is not opened and the service could not be accessed. This signalling

message is not visible in the above log since the port was opened much earlier.

E-Mail POP3 (Receiving)

Start trigger: TCP SYN sent by the MS (signaling message 2 in Figure 7).

Stop trigger: Reception of the first data packet containing mail content by the MS

(message 18 in Figure 7)

Figure 7: E-Mail POP3 trace


E-Mail SMTP (Sending)


Stop trigger: Sending of the first data packet containing mail content (message 15 in

Figure 8).

Figure 8: E-Mail SMTP trace

HTTP


Stop trigger: Reception of the first data packet and OK for http 1.1 or 1.0 (message 7 in

Figure 9).

Figure 9: HTTP trace

Service Access Timeouts:

The timeout values summarized in Table 2 are only applicable for active measurements

of the PI Service Access Failure Ratio.

FTP E-Mail HTTP

Timeout 30s 60s10s per

attempt

Service Access

Table 2: Service Access Timeouts


These timeout values are no system internal values but are expected to mirror the user

behavior. If – for example - an ftp file-transfer did not start after 30 seconds the end-user

will stop this transaction. Similarly, if the end-user does not perceive that the loading of

a web page started after 10 seconds he will try to reload the page. But if these reload

attempts are not successful he will stop accessing the web page.

Remarks:

The above-mentioned trigger points for the different services are defined on a rather

technical level and not from a total end-user’s point of view. An end-user would

consider a service access as successful if the complete data transfer was successful.

Measuring the PI Service Access Failure Ratio related to the transfer of the complete

data content would mean that the measured ratio and as well the service access delay

would depend on the file size.

Prerequisites:

The MS is already GPRS attached. (PI ACC 020) and a PDP Context is activated (PI

ACC 030)

Active Measurement:

This PI can be measured by executing the following test cases described in detail in the


D)S Lookup, TAG: SET 010,

FTP UL/DL, TAG: SET 020,

WAP Traffic, TAG: SET 030,

WWW Traffic, TAG: SET 040,

E-mail Traffic, TAG: SET 050.


This PI can be measured with the Moniq tool:

• WAP Connection Establishment Success Ratio (5.2.5.1)

• Web connection failure reasons: aborts, protocol error, congestion

• POP3, Mail Server Connection Success Ratio (5.2.5.2)

• Domain Name Service Query Success Ratio (5.2.5.3)

5.2.5.1 WAP Connection Establishment Success Ratio

TAG: ACC WAPconnect

The number of successful connection establishments divided by the total number of

connection attempts.


5.2.5.2 POP3, Mail Server Connection Success Ratio

TAG: ACC POP3connect

The number of successful connection establishments divided by the total number of

connection attempts.

5.2.5.3 Domain Name Service Query Success Ratio

TAG: ACC DNS

The number of successful domain name (DNS) resolutions divided by the total number

of DNS requests. A successful resolution means that the server has found the name and

has responded to the request with a valid name.


5.2.6 GPRS Accessibility Failure Ratio (for each service)

TAG: ACC 050

Definition:

The KPI GPRS e2e Accessibility Failure Ratio [%] denotes the probability that a

subscriber cannot access the service successfully, meaning that the data transfer of the

content could not be started due to a failure that occurred within the complete chain of

service access:

GPRS Availability � Attach � PDP Context Activation � Service Access

( ) ( ) ( ) ( )[ ]04010301020101011[%] ACCPIACCPIACCPIACCPIRatioFailureAccess −∗−∗−∗−−=

Trigger Point(s):

Trigger points and timeout values that are defined for the involved PIs:

• GPRS Unavailability (PI ACC 010)

• GPRS Attach Failure Ratio (PI ACC 020)

• GPRS PDP Context Activation Failure Ratio (PI ACC 030), and

• GPRS Service Access Failure Ratio (for each service) (PI ACC 040)

are valid for measuring GPRS e2e Accessibility failure ratio. See also Figure 15:

Overview of the KPIs.

Example:

Assuming 1000 measurements and the following measured PIs ACC 010, ACC 020,

ACC 030 and ACC 040:

GPRS Unavailability (PI ACC 010): 0%

� GPRS was available all the time � 1000 samples remain for the attach procedure

GPRS Attach Failure Ratio (PI ACC 020): 30%

� results in 700 samples for PDP context activation

GPRS PDP Context Activation Failure Ratio (PI ACC 030): 15%

� results in 595 samples for the service access, and

(PI ACC 040): 10%

� 535.5 service access attempts out of 1000 were successful and in 464.5 cases the

service could not be access.


( ) ( ) ( ) ( )[ ]

( ) ( ) ( ) ( )[ ]

( )[ ]

[ ]

%45.46

%100*5355.01

%100*9.0*85.0*7.0*11

%100*1.01̀*15.01*3.01*011

06010401020101011

[%]

=

−=

−=

−−−−−=

−∗−∗−∗−−= ACCPIACCPIACCPIACCPI

RatioFailureAccess

Active Measurement:

KPI is not covered in the Generic Test Description for GPRS/EDGE End-to-End



This KPI can be measured with the Moniq tool:

• Web/FTP/E-mail service interrupts

Note: The Web and SMTP success ratios are measured by dividing the successful

service completions with the total number of service attempts (including those

cases where the service access was already unsuccessful). The reason for this

small difference is that the service access and data transfer are not separated in

Moniq for these services.


5.3 Retainability KPIs

5.3.1 GPRS PDP Context Cut-off Ratio

TAG: RET 010

Definition:

The PI GPRS PDP Context Cut-off Ratio [%] denotes the probability that a PDP context

is deactivated without being initiated by the MS.

%100#

#[%] ∗=

contextsPDPactivatedtotal

lossescontextPDPRatioOffCutContextPDP

Trigger Point(s):

Different trigger points for a PDP Context Deactivation not initiated by the MS are

possible: SGSN failure or GGSN failure on which the PDP context will be deactivated

by the SGSN or GGSN respectively.

Expiry of the Mobile )ot Reachable Timer leads to PDP Context Deactivation and is

therefore another reason for PDP context cut-off.

Prerequisites:


ACC 030)

Active Measurement:

PI not covered in the Generic Test Description for GPRS/EDGE End-to-End



This PI can be measured with the help of counters in the /// GGSN.


5.3.2 GPRS Data Transfer Cut-off Ratio

TAG: RET 020

Definition:

The KPI GPRS Data Transfer Cut-off Ratio [%] denotes the probability that a data

transfer is incomplete after a timeout period or the transfer is aborted completely.

%100#

#[%] ∗=

transfersdatatotal

transfersdataincompleteRatioOffCutTransferData

Trigger Point(s):

1. Transfer abort � file transfer is not complete.

2. PDP context cut-off (PI RET 010) � file transfer is not complete.

3. FTP: Timeout counter of 90 seconds is triggered if no data reaches the receiver

during 90 seconds � file transfer is not complete.

Remarks:

In case of a data transfer cut-off it should be determined how much data was transmitted

before this cut-off.

Active Measurement:

This KPI can be measured by executing the following test case described in detail in the


Round Trip Time, TAG: TT 020,







• WAP Object Download Success Ratio (5.3.2.1)

• Web Object Download Success Ratio (5.3.2.2)

• FTP Download Success Ratio (5.3.2.3)

• POP3, E-mail Download Success Ratio (5.3.2.4)

• SMTP, E-mail Send Success Ratio (5.3.2.5)


5.3.2.1 WAP Object Download Success Ratio

TAG: RET WAPobject

The number of successfully finished and acknowledged object downloads divided by all

download attempts. Connection-mode WAP is measured. Precondition: established

WAP session.

5.3.2.2 Web Object Download Success Ratio

TAG: RET WEB

The number of successfully finished HTTP object downloads divided by all HTTP

download attempts.

5.3.2.3 FTP Download Success Ratio

TAG: RET FTP

The number of successfully finished file downloads divided by the number of all FTP

download attempts. Condition: connection to FTP server was successful.

5.3.2.4 POP3, E-mail Download Success Ratio

TAG: RET POP3download

The number of successfully finished mail downloads divided by all e-mail download

attempts. Condition: connection to the server was successful, and a request to download

(at least) one mail was initiated. This last condition is important since many sessions just

check the mailbox and do not attempt any mail download.

5.3.2.5 SMTP, E-mail Send Success Ratio

TAG: RET SMTPdownload

The number of successfully sent e-mails divided by the total number of attempts using

the SMTP protocol. Web-based e-mail services are not considered. (Note: all common e-

mail applications Netscape, Outlook, and mobile phone integrated clients use the SMTP

protocol.)


5.3.3 Cell Update Failure Ratio

TAG: RET 030

Definition:

The GPRS/EDGE KPI Cell Update Failure Ratio [%] denotes the probability that a Cell

Update procedure fails.

%100#

#[%] ∗=

attemptsUpdateCelloftotal

attemptsUpdateCellulunsuccessfRatioFailureUpdateCell

Note: Since a cell update can only be observed in READY state the cell update failure

ratio can be measured best during an ongoing FTP session.

Trigger Point(s):

The cell update procedure failed when the data transfer is not continued (sending or

receiving) after the MS entered the new cell.

MS BSS SGSN

FLUSH-LL-ACK

UL-UNITDATA (old cell)

DL-UNITDATA (old cell), TCP retransmission!

FLUSH-LL (old BVCI and new BVCI)

DL-UNITDATA (new cell)

DL-UNITDATA (old cell)

DL-UNITDATA (old cell)

UL-UNITDATA (new cell)

Figure 10: Signalling diagram for cell update during DL FTP

Prerequisites:


ACC 030). Two different cells belonging to the same RA configured with GPRS/EDGE

capability.


Active Measurement:



Cell Update during Downlink FTP, TAG: MT 010.

This KPI can be measured by executing the UDP based test solution: Transport Tests, IP

layer outage as described in detail in Proposed Extensions to GPRS e2e performance

Testing [2].


There is no means of passive measurement (counter or Moniq tool) for this KPI.


5.3.4 Intra SGSN RAU Failure Ratio

TAG: RET 040

Definition:

The KPI Intra SGSN RAU Failure Ratio [%] denotes the probability that an Intra SGSN

RAU procedure fails.

%100#

#[%] ∗=

attemptsRAUSGS�Intraoftotal

attemptsRAUSGS�IntraulunsuccessfRatioFailureRAUSGS�Intra

Trigger Point(s):

Check if the SGSN RAU procedure from sending the Routing Area Update Request and

receiving the Routing Area Update Accept message was executed successfully.

(Signaling messages 1 and 3 in Figure 11.)

1. Routeing Area Update Request

3. Routeing Area Update Accept


MS BSS SGSN

4. Routeing Area Update Complete C1

Figure 11: Intra SGSN RAU Procedure

Prerequisites:

The MS is already GPRS attached (PI ACC 020) and a PDP Context is activated (PI

ACC 030). Two different cells belonging to different RAs served by the same SGSN

configured with GPRS/EDGE capability.

Active Measurement:



Intra SGS) RA Update during Downlink FTP, TAG: MT 020


This KPI can be measured with counters in the /// SGSN.


5.3.5 Inter SGSN RAU Failure Ratio

TAG: RET 050

Definition:

The KPI Inter SGSN RAU Failure Ratio [%] denotes the probability that an Inter SGSN

RAU procedure fails.

%100#

#[%] ∗=

attemptsRAUSGS�Interoftotal

attemptsRAUSGS�InterulunsuccessfRatioFailureRAUSGS�Inter

Trigger Point(s):

Check if the SGSN RAU procedure from sending the Routing Area Update Request and

receiving the Routing Area Update Accept message was executed successfully.

(Signaling messages 1 and 11 in Figure 12.)

MS BSS new SGSN HLR GGSN old SGSN

2. SGSN Context Response



2. SGSN Context Request

6. Update PDP Context Request

6. Update PDP Context Response

7. Update Location

10. Update Location Ack

11. Routeing Area Update Accept

8. Cancel Location

8. Cancel Location Ack

9. Insert Subscriber Data Ack

9. Insert Subscriber Data

12. Routeing Area Update Complete

5. Forward Packets

4. SGSN Context Acknowledge

Figure 12: Inter SGSN RAU Procedure

Prerequisites:



ACC 030). Two different cells belonging to different RAs served by the different

SGSNs configured with GPRS/EDGE capability.

Active Measurement:



Inter SGS) RA Update during Downlink FTP, TAG: MT 030.




5.3.6 PS Paging Failure Ratio

TAG: RET 060

Definition:

The KPI PS Paging Failure Ratio [%] denotes the probability that an SGSN Paging

procedure fails.

%100#

#[%] ∗=

attemptsPagingSGS oftotal

attemptsPagingSGS ulunsuccessfRatioFailurePagingPSSGS

Trigger Point(s):

The network shall initiate the paging procedure for GPRS services using P-TMSI when

GMM signaling messages or user data is pending to be sent to the MS while the Mobile

Reachable timer is running. The network may page only GPRS MSs, which are GMM-

REGISTERED and identified by a local P-TMSI. The SGSN requests the RR sublayer to

start paging. Upon reception of a paging indication, the MS shall respond to the paging

with any LLC frame. [7]

Check if the Paging for GPRS Services procedure from sending the Paging Request by

the SGSN and receiving any LLC frame send by the mobile was executed successfully.

Prerequisites:


ACC 030).

Active Measurement:

KPI not covered in the Generic Test Description for GPRS/EDGE End-to-End





5.4 Integrity KPIs

5.4.1 GPRS Attach Setup Time

TAG: INT 010

Definition:

The PI GPRS Attach Setup Time [ms] denotes the length of the time period it takes to

attach to the GPRS/EDGE network.

Note: Difference between an attach of a known subscriber and an unknown subscriber

will be reflected in the length of the time period indicating the Attach Setup

Time. In case of an attach of an unknown subscriber (meaning that the SGSN

has changed since the GPRS detach, or if it is the very first attach of the mobile

to the network), the SGSN contacts the HLR in order to receive the subscriber

data.

The attach setup time of an unknown subscriber will be slightly longer than the

one of a known subscriber.

Trigger Point(s):

Measurement of the time [ms] from sending the Attach Request and receiving the Attach

Accept message, see Figure 1.

Remarks:

While determining the average Attach Setup Time only successful attach attempts are

included in the calculations. Failed attach attempts should not be part of the calculation

denoting a timeout period of 30 seconds. Note that the timeout value of 30 seconds is no

system internal value.

Prerequisites:

GPRS/EDGE has to be active in the cell used by a subscriber (PI ACC 010)

Active Measurement:



GPRS Attach, TAG: ST 010.




5.4.2 GPRS PDP Context Activation Delay (per APN)

TAG: INT 020

Definition:

The PI GPRS PDP Context Activation Delay Time [ms] denotes the length of the time

period it takes to activate a PDP context.

Trigger Point(s):

Measurement of the time [ms] from sending the Activate PDP Context Request and

receiving the Activate PDP Context Accept message, see Figure 2.

Remarks:

While determining the average PDP Context Activation Delay only successful activation

attempts are included in the calculations. Failed activation attempts should not be part of

the calculation denoting a timeout period of 30 seconds. Note that the timeout value of

30 seconds is no system internal value.

The PDP Context Activation Delay should be determined per APN.

Prerequisites:

The MS is already GPRS attached. (PI ACC 020).

Active Measurement:



PDP Context Activation, TAG: ST 020




5.4.3 GPRS Service Access Time (for each service)

TAG: INT 030

Definition:

The PI GPRS Service Access Time [s] denotes the length of the time period it takes to

access a service successfully.

Trigger Point(s):

Measurement of the time [s] according to the trigger points defined for PI ACC 040 (see

Figure 5 - Figure 9).

Remarks:

While determining the Access Time for a specific service only successful access

attempts are included in the calculations. Failed access attempts should not be part of the

calculation.

The Access Time should be determined per service, since the actual access time depends

strongly on the nature of the service.

For e-mail and ftp the transmission rate is of higher interest than the actual service

access time.

Remarks:

The above-mentioned trigger points for the different services are defined on a rather

technical level and not from a total end-user’s point of view. An end-user would be

rather interested in the time needed for transferring the complete data content.

Measuring the PI Service Access Time related to the transfer of the complete data

content would mean that the service access delay would depend on the file size.

Prerequisites:


ACC 030)

Active Measurement:



D)S Lookup, TAG: SET 010,







This PI Service Access Time can be measured with the Moniq tool:

• WAP Connection Setup Delay (5.4.3.1)

• Web Server Response Time (5.4.3.2)

• POP3, Mail Server Connection Time (5.4.3.3)

Measuring the time for transmitting the data content starting after the service access

itself (see Figure 13) reflects the end-user’s perspective and can be measured with the

Moniq tool:

• WAP Object Download Delay (5.4.3.4)

• Web small object download time (5.4.3.5)

• POP3, Mail Download Time (5.4.3.6)

• SMTP, Mail Sending Time (5.4.3.7)

• Domain Name Service Response Delay (5.4.3.8)

Service Access Time

Connection established/

Start of data transferMobile requests

service

Data Transfer Time

Data transfer complete

Service Access Time

Connection established/

Start of data transferMobile requests

service

Data Transfer Time

Data transfer complete

Figure 13: Service Access & Data Transfer

5.4.3.1 WAP Connection Setup Delay

TAG: INT WAPconnect

The average delay between the setup-packet from the mobile is received at the monitor

until the mobile successfully acknowledges the server response packet and the ACK

packet passes the monitor. The measured delay is the full setup round-trip time and it

includes the following delay components: Monitor � WAP Server � Mobile �

Monitor.

Precondition:

Successful connection setup


5.4.3.2 Web Server Response Time

TAG: INT WEBresponse

The average time until the server starts to transmit data measured from the first packet

from the mobile. This measure includes the time it takes to set up a connection over the

GPRS/EDGE and wired Internet, the client to deliver the request to the server, the server

to process the request, and to respond to it. The condition is that only successful

response is measured, but the content of the response may be either success or failure.

5.4.3.3 POP3, Mail Server Connection Time

TAG: INT POP3connect

The average time it takes to connect to the POP3 e-mail server. The connection time

includes the time to establish the TCP connection over the GPRS/EDGE and Internet

path, the server to send a greeting message, and the time to log into the server.

Condition: successfully established connections are measured.

PIs denoting the time for transmitting the data content starting after the service access

itself that can be measured with the Moniq tool:

5.4.3.4 WAP Object Download Delay

TAG: INT WAPdownload

The average delay between the request packet from the mobile until the server response

is successfully acked by the mobile.

Precondition:

Successfully finished download

5.4.3.5 Web Small Object Download Time (9-11kbyte)

TAG: INT WEBsmall

The average time it takes to download a small HTTP object. Only small (9-11 Kbytes)

files are measured. Due to the TCP protocol, small objects are not able to utilize the full

GPRS/EDGE throughput (timeslot capacity). The download time of such small objects

depends more on the round-trip time

Remarks:

To have comparable results between different measurements and operators, only objects

that fall into a narrow range are measured. The chosen range of 9-11 kbytes was chosen

because the median of Web downloads falls in this range. Condition: object download

was successful.


5.4.3.6 POP3, Mail Download Time

TAG: INT POP3download

This KPI defines the average time to successfully download one or more e-mails from

the POP3 e-mail server. The whole time is measured including server greeting,

authentication, and time to download of all mails and quit. Condition: mail download

was successful.

5.4.3.7 SMTP, Mail Sending Time

TAG: INT SMTPsend

This KPI defines the average time for the mobile to send an e-mail using the SMTP

protocol. Condition: e-mail sending was successful.

5.4.3.8 Domain Name Service Response Delay

TAG: INT DNSdelay

This KPI defines the average delay for the DNS server to respond to a DNS request,

counting retransmissions. The time is calculated between the request packet seen in the

uplink direction and the response packet seen in the downlink direction at the monitor.

Condition: resolution was successful.


5.4.4 GPRS Access Time (for each service)

TAG: INT 040

Definition:

The KPI GPRS e2e Access Time [s] denotes the length of the time period it takes to

access a service successfully, measuring the time period for the complete chain:

GPRS Availability � Attach � PDP Context Activation � Service Access

030020010][ I�TPII�TPII�TPIsTimeAccess ++=

Trigger Point(s):

Trigger points and timeout values that are defined for the involved KPIs:

• GPRS Attach Setup Time (PI I�T 010)

• GPRS PDP Context Activation Delay (per APN) (PI I�T 020), and

• GPRS Service Access Time (for each service) (PI I�T 030)

are valid for measuring GPRS e2e Access Time.

Remarks:

The lower the timeout values are for the different Access KPIs the higher the failure

ratio is. At the same time a lower timeout value goes along with a lower access time,

since the timeout value is the upper boundary for the measured time period it takes to

perform a certain procedure, i.e. attach, PDP Context activation or service access.

Active Measurement:

KPI not covered in the Generic Test Description for GPRS/EDGE End-to-End




Attach Setup

Time

PDP Context

Activation

Time

Service Access

Time

Attach Setup

Time

PDP Context

Activation

Time

Service Access

Time


5.4.5 GPRS Mean User Data Rate

TAG: INT 050

Definition:

The KPI GPRS Mean User Data Rate [kbits/s] denotes the average data rate in kbits/s.

)()(

8)(]/[

sTriggerStartofTimesTriggerStopofTime

kByteSizeFileskbitsRateDataMean

−

∗=

The trigger points have to be defined per service for KPI I�T 050.

Trigger Point(s):

The average throughput is measured from opening the data connection to the end of the

successful transfer of content (file, e-mail or web page).

The trigger points are defined per service (FTP DL/UL, E-Mail DL/UL, HTTP) in the

same way as they are defined for GPRS Data Transfer Cut-off Ratio (PI RET 020).

Refer to chapter 5.3.2.

Prerequisites:

The MS is already GPRS attached (PI ACC 020), a PDP Context is activated (PI ACC

030) and a service was accessed successfully (PI ACC 040).

Active Measurement:



Round Trip, Time, TAG: TT 020,

Controlled PDCH Sharing, TAG: TT 030,

Cell Update during DL FTP, TAG: MT 010,

Intra SGS) RA Update during DL FTP, TAG: MT 020,

Inter SGS) RA Update during DL FTP, TAG:MT 030,




This KPI can be measured by executing the UDP based test solution: Transport Tests,

RAW throughput as described in detail in Proposed Extensions to GPRS e2e

performance Testing [2].



This KPI can be measured for Web and FTP throughput in DL direction with the Moniq

tool:

• FTP Download Rate (5.4.5.1)

• Web Large Object Download Rate (larger than 50kbyte) (5.4.5.2)

• End-to-End Achievable Throughput (5.4.5.3)

• TCP Goodput/Throughput Ratio (5.4.5.4)

5.4.5.1 FTP Download Rate

TAG: INT FTPrate

The average rate is the size of the downloaded file divided by the time it takes to

download it. Only large files (> 50 kbyte) are measured, when the available end-to-end

path capacity dominates the measure and not the file size.

Precondition:

File download was successful

5.4.5.2 Web Large Object Download Rate (larger than 50kbyte)

TAG: INT WEBlarge

The average rate is the size of the object divided by the time it takes to successfully

download the object. This measure is also called goodput. Only large objects are

measured, when the available end-to-end path capacity dominates the measure and not

the object size. Condition: object download was successful.

5.4.5.3 End-to-End Achievable Throughput

TAG: INT THRHPUT

This KPI measures the average achievable IP layer throughput end-to-end. This KPI is

measured by tracing those mobiles that generate one or more parallel TCP downloads

saturating the downlink channel. In an optimal GPRS/EDGE network this KPI shows the

capacity of the available GPRS/EDGE data channel.

5.4.5.4 TCP Goodput/Throughput Ratio

TAG: INT THRHPUT/GPUT

The ratio of user data payload in TCP connections (Web, e-mail, FTP, etc.) divided by

the total IP layer data transmitted to the mobile terminal counted at the monitoring point.

The ratio depends on the protocol overheads (TCP/IP) and the retransmissions. 100% is

not possible; there is an upper limit even in an optimal network. The distance from this

optimal value has to be evaluated.


5.4.6 Round Trip Time

TAG: INT 060

Definition:

The KPI Round Trip Time (RTT) [s] denotes the length of the time period it takes for a

data packet to be sent from the MS and being returned to the sender.

Note: The ping program is a suitable method for measuring RTT.

Either a DNS server or an external Internet server can be pinged by the MS.

Trigger Point(s):

Since the ping program uses the ICMP echo request and echo reply messages (not using

the transport layer TCP or UDP) the start trigger is the ICMP echo request message and

the stop trigger the ICMP echo reply message.

Remarks:

The ping size has influence on the RTT, therefore measuring the KPI Round Trip Time

should be done with different ping sizes.

Furthermore the delay introduced by the Internet and the response time of the server

influence the RTT.

Prerequisites:


ACC 030).

Active Measurement:



Round Trip Time, TAG: TT 010.



• End-to-End Round Trip Delay (5.4.6.1)

• Client-side Round Trip Delay (5.4.6.2)

• Server-side Delay (5.4.6.3)

Remarks:

Packet delay is measured for TCP protocol exchanges, when the mobile client initiates a

TCP setup by sending a SYN packet. The server responds with a SYN/ACK packet,

which is finally acknowledged by the client with an ACK packet. Thus the TCP setup

includes a full end-to-end round-trip between the client and the server. The monitor

calculates the times between these packets, thus it can separate the delay for client and

server-side components.


5.4.6.1 End-to-End Round Trip Delay

TAG: INT DELAYe2e

The average time measured on the full client-server-client round-trip path. It is usually

dominated by the GPRS access delay (client-side delay).

Remarks:



which is finally acknowledged by the client with an ACK packet.

5.4.6.2 Client-side Round Trip Delay

TAG: INT DELAYgprs

The average time measured between the SYN/ACK and the ACK packets. This measure

includes the time over the GPRS channel and the client processing time.

Remarks:




5.4.6.3 Server-side Delay

TAG: INT DELAYinternet

The average time measured between the SYN and SYN/ACK packets. This measure

includes the delay over the core and Internet to the server, and the processing time in the

server. The propagation delay and the buffer congestion in the wired path of the

connections dominate this measure. The propagation delay is usually proportional to the

geographical distance (propagation time at speed of light). The processing time in

servers is usually negligible.

Remarks:





5.4.7 Client-Server Interaction Time with Background Traffic

TAG: INT 065

Definition:

The KPI Client Server Interaction Time denotes the time [s] it takes to send a small data

packet to the server and receive it back (RTT) during an ongoing DL TCP data transfer

(FTP) in the background.

Remarks:

The typical use case for this KPI is a user that downloads his emails or a big file from

the Internet in the background and starts at the same time a web session. For the web

session an HTTP GET will be sent to the web server and the server will respond with an

HTTP OK and start to send the content of the web page to the user. The user expects an

answer to his request of the web page in a reasonable time.

Note: If a big advertised TCP Window Size is used and the system buffers a lot of data,

it might happen that the data from the background traffic delays the response from the

web server to the user.

Trigger Point(s):

Since the ping program uses the ICMP echo request and echo reply messages (not using

the transport layer TCP or UDP) the start trigger is the ICMP echo request message and

the stop trigger the ICMP echo reply message.

Prerequisites:

The MS is already GPRS attached (PI ACC 020), a PDP Context is activated (PI ACC

030) and background traffic (FTP DL data transfer) is ongoing.

Active Measurement:



Round Trip Time during Background Traffic, TAG: TT 025.


5.4.8 Packet Loss Ratio

TAG: INT 070

Definition:

The KPI Packet Loss Ratio [%] denotes the rate of data packets, which are lost

regardless where they are lost in the network.

Note: The ping program is a suitable method for measuring the packet loss rate.

Trigger Point(s):

The start trigger is the ICMP echo request message and the stop trigger the ICMP echo

reply message.

The ping program should be used with different packet sizes, since sometimes only

packets of a special size are lost: The following packet sizes should be used: 32 byte,

548 byte and 1472 byte.

A ping command with a length of 548 byte would result in a total IP packet size of 576

byte (548 byte + 28 byte ICMP header). For the case of 1472 byte ICMP data the total IP

packet size would be 1500 byte. 576 and 1500 byte are typical IP packet sizes when

using TCP.

Remarks:

The Packet Loss ratio has big impact on the application performance.

Prerequisites:


ACC 030).

Active Measurement:



Round Trip Time, TAG: TT 010,

TCP/IP Transfer, TAG: TT 020,

WAP Traffic, TAG: SET 030.



• WAP Retransmission Ratio (5.4.8.1)

• Downlink End-to-End Packet Loss (5.4.8.2)

• Server to Monitor Loss (5.4.8.3)

• Monitor to Client Loss (5.4.8.4)


5.4.8.1 WAP Retransmission Ratio

TAG: INT WAPretransmission

The average number of packet retransmissions seen during WAP object downloads.

Retransmissions greatly affect the delay of page downloads. All object downloads are

measured regardless of their eventual result (success or failure).

5.4.8.2 Downlink End-to-End Packet Loss

TAG: INT LOSSe2e

The ratio of TCP packets lost downlink, between the server and the client end-to-end.

Typically the GPRS/EDGE channel access loss dominates this measure.

5.4.8.3 Server to Monitor Loss

TAG: INT LOSSinternet

The ratio of TCP (Web, e-mail, FTP, etc.) packets lost between the Internet server and

the monitoring point in the downlink direction. This KPI indicates the level of

congestion over the wired path from the server up to the monitor.

5.4.8.4 Monitor to Client Loss

TAG: INT LOSSgprs

The ratio of TCP (Web, e-mail, FTP, etc.) packets lost between the monitoring point and

the mobile terminal in the downlink direction. This measure characterizes the GPRS

access part of the path (GSN, PCU buffers and RLC/LLC loss).


5.4.9 Cell Update Time

TAG: INT 080

Definition:

The GPRS KPI Cell Update Time [s] denotes the length of the time period a cell update

takes.

Note: Since a cell update can only be observed in READY state the duration of a cell

update can be measured best during an ongoing FTP session.

Trigger Point(s):

Start measuring the time interval when the last IP frame is sent/received in the old cell

and stop measuring when the first IP frame is sent/received in the new cell by the MS.

Remarks:

In live networks parameters like the moving speed, the cell configuration and the MS

have big influence on the time needed for the cell update.

Prerequisites:


ACC 030). Two different cells belonging to the same RA configured with GPRS/EDGE

capability.

Active Measurement:



Cell Update during DL FTP, TAG: MT 010



Testing [2].




5.4.10 Intra SGSN RAU Time

TAG: INT 090

Definition:

The KPI Intra SGSN RAU Time [s] denotes the length of the time period an Intra SGSN

RAU takes.

Note: The duration of an Intra SGSN RAU can be measured best during an ongoing

FTP session.

In most of the case the RA belongs to a new LA, so that the MS performs a

LAU as well.

Trigger Point(s):



In case the new cell belongs to a new LA, the MS enters dedicated mode and sends a

Suspend message to the BSS. The BSS sends the Suspend message to the SGSN. The

SGSN sends a Suspend Ack to the BSS, which will discard all LLC frames stored for

the MS. After the MS has performed the LA update it will send a RA update in the new

cell to resume the transfer, see Figure 14. This will be done in case that the BSS does

not support the resume procedure. (Origin: [5]).

2. Suspend


1. Dedicated Mode

MS BSS SGSN MSC/VLR

3. Suspend

4. Resume

5. Channel Release

3. Suspend Ack

4. Resume Ack

Figure 14: Suspend/Resume Procedure

Remarks:

The continuation of the DL transmission after the RAU depends on the number of

buffered frames in the SGSN:

• If there is still data buffered in the SGSN the transfer will continue immediately

after the RAU complete.


• If no more data is stored in the SGSN the continuation of the transfer depends on the

application. � For an FTP DL transfer a timeout will occur on TCP layer and the

transmission continues afterwards.


have big influence on the time needed for the Intra SGSN RAU.

Prerequisites:


ACC 030). Two different cells belonging to different RAs served by the same SGSN

configured with GPRS/EDGE capability.

Active Measurement:



Intra SGS) RA Update during DL FTP, TAG: MT 020



Testing [2].




5.4.11 Inter SGSN RAU Time

TAG: INT 100

Definition:

The KPI Inter SGSN RAU Time [s] denotes the length of the time period an Inter SGSN

RAU takes.

Note: The duration of an Inter SGSN RAU can be measured best during an ongoing

FTP session.

In most of the case the RA belongs to a new LA, so that the MS performs a

LAU as well.

Trigger Point(s):



Remarks:


have big influence on the time needed for the Intra SGSN RAU.

Prerequisites:


ACC 030). Two different cells belonging to different RAs served by the different

SGSNs configured with GPRS/EDGE capability.

Active Measurement:



Inter SGS) RA Update during DL FTP, TAG: MT 030



• Overview of the KPIs • • Definition of GPRS/EDGE End-to-End KPIs •

6 Overview of the KPIs When measuring KPIs related in a live network file and e-mail sizes as well as timeout

values cannot be influenced, so that passive tests of KPIs will measure any file and e-

mail size that appears in a live network as well as system internal timeout values.

But when measuring the KPIs with active tests the following timeout values should be

chosen:

Figure 15: Overview of the KPIs

ACC 010

GRPS/UMTS Unavailability

KPIs:

Accessibility

Retainability

Integrity

max. 30s max. 30s max. 30s max. 30s

time out time outtime out time out

time out time out

service dependant service dependant

Data (content) transfer

INT 010

Attach Setup Time

INT 020

PDP Context Activation Time

INT 030

Service Access Time

ACC 020

Attach Failure Ratio

ACC 030

PDP Context Activation

Failure Ratio

ACC 040

Service Access Failure

Ratio

RET 010

PDP Context Cut-off

Ratio

RET 020

Data Transfer Cut-off

Ratio

INT 050

Mean User Data Rate

PDP Context

Deactivation

Wait for (max.) 30s, if PDP

Context in not deactivated

� reset phoneDetach

Wait for (max.) 30s, if

MS is not detached � reset phone

Start trigger: Attach request

Stop trigger: Attach Accept

Start trigger: PDP Context Activation request

Stop trigger: PDP Context Activation Accept

INT 070

Packet Loss Ratio

INT 060

Round Trip Time

ACC 010

GRPS/UMTS Unavailability

KPIs:

Accessibility

Retainability

Integrity

max. 30s max. 30s max. 30s max. 30s

time out time outtime out time out

time out time out

service dependant service dependant

Data (content) transfer

INT 010

Attach Setup Time

INT 020

PDP Context Activation Time

INT 030

Service Access Time

ACC 020

Attach Failure Ratio

ACC 030

PDP Context Activation

Failure Ratio

ACC 040

Service Access Failure

Ratio

RET 010

PDP Context Cut-off

Ratio

RET 020

Data Transfer Cut-off

Ratio

INT 050

Mean User Data Rate

PDP Context

Deactivation

Wait for (max.) 30s, if PDP

Context in not deactivated

� reset phoneDetach

Wait for (max.) 30s, if

MS is not detached � reset phone

Start trigger: Attach request

Stop trigger: Attach Accept

Start trigger: PDP Context Activation request

Stop trigger: PDP Context Activation Accept

INT 070

Packet Loss Ratio

INT 060

Round Trip Time

• Definition of GPRS/EDGE End-to-End KPIs • • Summary •

7 Summary In this chapter all KPIs are listed together with their tag and an indication whether the

can be measured actively or passively.

It is very important to use this chapter only for quick reference but not as ultimate input

when looking for a specific KPI and its definition. Detailed information on each KPI is

provided in chapters 5.2, 5.3, and 5.4.

Figure 16: Overview: KPIs and Test Measurements

KPI

Cat e

gor y

Ti tl e

Tag

Passi v

e M

easur e

ment

Act i

ve M

easur e

ment

Accessibility KPIs: GPRS/UMTS Unavailability ACC 010 - -

Attach Failure Ratio ACC 020 counter GTD: ST 010

PDP Context Activation Failure Ratio ACC 030 counter GTD: ST 020

Secondary PDP Context Activation Failure Ratio ACC 035 - GTD: ST 020

Service Acces Failure Ratio ACC 040 Moniq tool

GTD: SET 010, SET 020, SET 030,

SET 040, SET 050

GPRS Accessibility Failure Ratio ACC 050 Moniq tool -

Retainability KPIs: PDP Context Cut-off Ratio RET 010 - -

Data Transfer Cut-off Ratio RET 020 Moniq tool

GTD: TT 020, SET 020, SET 030,

SET 040, SET 050

Cell Update Failure Ratio RET 030 - GTD: MT 010

Intra SGSN RAU Failure Ratio RET 040 counter GTD: MT 020

Inter SGSN RAU Failure Ratio RET 050 counter GTD: MT 030

PS Paging Failure Ratio RET 060 counter -

Integrity KPIs: Attach Setup Time INT 010 - GTD: ST 010

PDP Context Activation Delay INT 020 - GTD: ST 020

Service Access Time INT 030 Moniq tool

GTD: SET 010, SET 020, SET 030,

SET 040, SET 050

End-to-End Access Time INT 040 - -

Mean User Data Rate INT 050 Moniq tool

GTD: TT 020, TT 030, MT 010, MT

020, MT 030, SET 020, SET 040,

SET 050, Extension GTD: RAW

throughput

Round Trip Time INT 060 Moniq tool GTD: TT 010

Packet Loss Ratio INT 070 Moniq tool GTD: TT 010, TT 020 & SET 030

Cell Update Time INT 080 -

GTD: MT 010, Extension GTD: IP

layer outage

Intra SGSN RAU Time INT 090 -

GTD: MT 020, Extension GTD: IP

layer outage

Inter SGSN RAU Time INT 100 - GTD: MT 030

• Summary • • Definition of GPRS/EDGE End-to-End KPIs •

KPI

Cat e

gor y

Ti t

l e

Tag

Passi v

e M

easur e

ment

Accessibility KPIs: Service Acces Failure Ratio ACC 040 Moniq tool

WAP Connection Establishment Success Ratio ACC WAPconnect Moniq tool

POP3, Mail Server Connection Success Ratio ACC POP3connect Moniq tool

Domain Name Service Query Success Ratio ACC DNS Moniq tool

Retainability KPIs: Data Transfer Cut-off Ratio RET 020 Moniq tool

WAP Object Download Success Ratio RET WAPobject Moniq tool

Web Object Download Success Ratio RET WEB Moniq tool

FTP Download Success Ratio RET FTP Moniq tool

POP3, E-mail Download Success Ratio RET POP3download Moniq tool

SMTP, E-mail Send Success Ratio RET SMTPdownload Moniq tool

Integrity KPIs: Service Access Time INT 030 Moniq tool

WAP Connection Set-up Delay INT WAPconnect Moniq tool

Web Server Response Time INT WEBresponse Moniq tool

POP3, Mail Server Connection Time INT POP3connect Moniq tool

WAP Object Downlaod Delay INT WAPdownload Moniq tool

Web Small Object Download Time INT Websmall Moniq tool

POP3, E-mail Download Time INT POP3download Moniq tool

SMTP, E-mail Sending Time INT SMTPsend Moniq tool

Domain Name Service Response Delay INT DNSdelay Moniq tool

Mean User Data Rate INT 050 Moniq tool

FTP Download Rate INT FTPrate Moniq tool

Web Large Object Download Rate INT WEBlarge Moniq tool

End-to-end Achievable Throughput INT THRHPUT Moniq tool

TCP Goodput/Throughput Ratio INT THRHPUT/GPUT Moniq tool

Round Trip Time INT 060 Moniq tool

End-to-end Round Trip Delay INT DELAYe2e Moniq tool

Client-side Round Trip Delay INT DELAYgprs Moniq tool

Server-side Delay INT DELAYinternet Moniq tool

Packet Loss Ratio INT 070 Moniq tool

WAP Retransmission Ratio INT WAPretransmission Moniq tool

Downlink End-to-End Packet Loss INT LOSSe2e Moniq tool

Server to Monitor Loss INT LOSSinternet Moniq tool

Monitor to Client Loss INT LOSSgprs Moniq tool

Figure 17: GPRS KPIs Based on Passive Measurements with Moniq

• Definition of GPRS/EDGE End-to-End KPIs • • Terminology •

8 Terminology

8.1 Abbreviations

BCCH Broadcast Control Channel

DL Downlink

e2e End-to-End

FTP File Transfer Protocol

GMM GPRS Mobility Management

ICMP Internet Control Message Protocol

KPI Key Performance Indicator

LA Location Area

LAU Location Area Update

LLC Logical Link Control

PBCCH Packet Broadcast Control Channel

PDCH Packet Data Channel

PDP Packet Data Protocol

PI Performance Indicator

P-TMSI Packet Temporary Mobile Subscription Identity

RA Routing Area

RAU Routing Area Update

RR Radio Resource

RTT Round Trip Time

UL Uplink

WAP Wireless Application Protocol

• Terminology • • Definition of GPRS/EDGE End-to-End KPIs •

8.2 Glossary

This chapter provides definitions of technical and other terms used in the document.

Accessibility

The ability of the user to obtain a service within specified tolerances and other given

conditions.

The accessibility of the service does include a chain of operations the user must be able

to execute:

• the user must be able to access GPRS/EDGE as such, meaning he must be able to do

a GPRS attach and a PDP context activation, and

• he must be able to access an IP service after he has an activated PDP context.

Integrity

Integrity indicates the degree to which a service is provided without major interferences,

once it was accessed.

Integrity KPIs show the performance of successful service attempts. Even if a service

was accessed successfully, the user’s perception of the performance may vary between

very good and unacceptably bad.

Retainability

Retainability indicates the ability of the user to keep a service – once it was accessed -

under given conditions for a requested period of time.

The retainability of a service or session does include that the user does not have to

perform any additional manual operations that would not be necessary under stable

network conditions, like for example manual re-activation of the PDP context.

Performance Indicator

Performance Indicators are those indicators that have impact on the end-user’s

perception of the service performance.

Key Performance Indicator

Key Performance Indicators are those performance indicators that are rate to be primary

performance Indicators reflecting the end-user’s perception of the service performance

in an outstanding manner.

Goodput & Throughput

Goodput is the user data payload in TCP connections (Web, e-mail, FTP, etc.), in

contrast to the total IP layer data transmitted to the mobile terminal. Throughput

includes the protocol overhead (TCP/IP) and the retransmissions while the goodput is

the raw data the end-user wants to receive or transmit.

• Definition of GPRS/EDGE End-to-End KPIs • • References •

9 References [1] Generic Test Description for GPRS/EDGE End-to-End Performance Testing in

a Live Network, EED/S/G-01:135 Uen, Revision C.

[2] Proposed Extensions to GPRS e2e performance Testing, (ETH).

[3] Ericsson GPRS End-to-End Performance Guidebook, EED/S/G-01:134 Uen

[4] ITU-T Recommendation E.800: Terms and Definitions Related to Quality of

Service and Network Performance Including Dependability (08/94)

[5] GPRS Service Description Stage 2 (Release 1999), 3GPP TS 23.060 V3.7.0

[6] Measurements of GPRS End-to-End KPIs with Counters, EED/S/G-02:044

[7] Mobile radio interface layer 3 specification, 3GPP TS 24.008

• Appendix • • Definition of GPRS/EDGE End-to-End KPIs •

10 Appendix

10.1 Statistical Guidelines

When analyzing loss and success ratio KPIs the proportional frequency of a specific

event is estimated from measurement data. Calculating the uncertainty of a measurement

the correct sample size can be derived.

Formula (1) represents the Gaussian approximation of the binomial distribution

considering a 95% confidence interval:

n

ff )1(**96.1

−

(1)

with:

f = measured proportional frequency of the KPI one is interested in.

n = length of the sample

p = unknown proportional frequency of the KPI one is interested in

Thus, the corresponding proportional frequency p falls in the confidence interval (CI)

with 95% probability:

−+

−−

n

fff

n

fff

)1(**96.1,

)1(**96.1 (2)

Example:

Assumption: Measured f = 20% for the GPRS Attach Ratio (KPI xx) with a sample size

of 200 measurements. Using formula (1) shows that the uncertainty of the measurement

is 5.54%. Therefore: with a 95%-probability the proportional frequency p is included in

the interval [14.46: 25.54]

definition kpis edge doc

Documents