Benchmarked Key Performance Indicators
Crisphine J. Ogongo,Engineer/Compliance
Communications Commission of Kenya
ITU-T Workshop on Delivering Good Quality Telecommunication Service
in a Safe Environment in Africa (Nairobi, Kenya, 26th July 2010)
Nairobi - Kenya 26 - 27 July 2010
Content
ITU/ETSI StandardsQoS Regulation And Comparison With Int’l StdsConclusionQuestions
Nairobi - Kenya 26 - 27 July 2010
ITU-T/ETSI STANDARDS
Nairobi - Kenya 26 - 27 July 2010
Definition
Quality of service (QoS): The collective effect of service performances, which determine the degree of satisfaction of a user of the service (ITU-T E.800)
Nairobi - Kenya 26 - 27 July 2010
What QoS Really Is!Network Performance Parameters translated to;
system design, configuration, operation and maintenance
Quality of service parameter influenced by; Statistics e.g «Call Block Rate»
User/customer requirementsIndividual experience e.g «inaccessibility»-
User opinion/requirements feed back in to the network planning process to alter planned performance and/or practical operational standards
Nairobi - Kenya 26 - 27 July 2010
ITU-T QOS Parameters with Targets
Call Set Up Time/Post Dialing DelayCall Release DelayAnswer Signal DelayEnd to End BlockingHandover Success Rate/Unsuccessful HandoverSpeech Quality Multimedia QoS
Nairobi - Kenya 26 - 27 July 2010
Call Set up timeCall type F-M M-F M-M Definition
Authentication/ciphering
0.0 2.5 2.5
Time interval between the end of dialing by the user and the reception by him of the appropriate tone or recorded announcement, or the abandon of the call without tone.
Paging/alerting 4.0 0.0 4.0
Routing number transfer 2.0 0.0 2.0
Post-selection delay
Local connection 3.0 3.0 3.0
Toll connection 5.0 5.0 5.0
International connection 8.0 8.0 8.0
TOTAL PLMN PSTN
F-M M-F M-M Normal Load High Load
Mean 95% Mean 95%
Local connection 9 5.5 11.5 3 s 6 s 4.5 s 9 s
Toll connection 11 7.5 13.5 5s 8 s 7.5 s 12 s
International connection 14 10.5 16.5 8 s 11 s 12 s 16.5 s
Nairobi - Kenya 26 - 27 July 2010
Answer Signal Delay
PLMN Connection Definition
Call type F-M M-F M-M Time interval between the establishment of a connection between calling and called users, and the detection of an answersignal at the originating exchange.
Local connection 1.0 1.0 1.25
Toll connection 1.75 1.75 2.0
International connection 2.75 2.75 3.0
PSTN
Normal Load High Load
Mean 95% Mean 95%
Local connection 0.75 sec 1.5 sec 1.0 sec 2.0 sec
Toll connection 1.5 sec 3.0 sec 2.0 sec 4.0 sec
International connection 2.0 sec 5.0 sec 3.3 sec 6.5 sec
Nairobi - Kenya 26 - 27 July 2010
Call Release Delay
Call release delay (secs)
Connection Type
Call type F-M, M-F, M-M PSTN
Calling, or called, party clears
1.0 sec 0.4 – 1sec
Call release delay is defined as the time interval from the instant the first bit of the DISCONNECT message is passed by the user terminal which terminated the call to the access signaling system, until the last bit of the RELEASE message is received by the same terminal (indicating that the terminals can initiate/receive a new call).
Definition
Nairobi - Kenya 26 - 27 July 2010
End to End Blocking
1990 Improvement
PLMNOn radio Channel 5-10% 1%
PLMN to Fixed 1% 0.5%
Normal Load High Load
PSTNLocal 2% 3%
Toll Connection 3% 4.5%
Int’l Connection 5% 7.5%
Definition: - The probability that any call attempt will be unsuccessful due to a lack of network resources
Nairobi - Kenya 26 - 27 July 2010
Unsuccessful Handovers
Call type F-M, M-F, M-M
Probability of unsuccessful land cellular handover
0.5%
Is the probability that a handover attempt fails because of lack of radio resources in the target cell, or because of a lack of free resources for establishing the new network connection. The failure condition is based either on a specified time interval since the handover request was first issued or on a threshold on signal strength
Nairobi - Kenya 26 - 27 July 2010
Speech Quality
PESQ: - Perceptual evaluation of speech quality ITU-T P.862 An objective method for end-to-end speech quality assessment of narrow-band telephone networks and speech codecs PESQ compares an original signal X(t) with a degraded signal Y(t) that is the result of passing X(t) through a communications system. The output of PESQ is a prediction of the perceived quality that would be given to Y(t) by subjects in a subjective listening test(6).Provides Speech Quality raw values -0.5 to 4.5
Nairobi - Kenya 26 - 27 July 2010
Speech Quality
Subjects in
Listening Test
Original Signal X(t)Perceived
Speech quality
PESQ: - Perceptual evaluation of speech quality ITU-T P.862 An objective method for end-to-end speech quality assessment of narrow-band telephone networks and speech codecs
Nairobi - Kenya 26 - 27 July 2010
Speech Quality cont’dITU-T P.862.1 - Mapping function for transforming P.862 raw result scores to MOS
The mapping ensures a domain rescaling from –0.5 ... 4.5 to 1.02 ... 4.56
P.862.1_F1
0
0.5
1
1.5
2
2.5
3
3.5
4
4.5
5
– 1 0 1 2 3 4 5P.862
Map
ped
P.86
2
Nairobi - Kenya 26 - 27 July 2010
Speech Quality Cont’d
Speech quality depends on Transmission Rating Factor RThe R‑value is a measure of a quality perception to be expected by the average user when communicating via the connection under consideration
5060708090100T1211030-99
linear quality scale
high quality
mediumquality
low quality
Area notrecommended
OverallRating "R"
Nairobi - Kenya 26 - 27 July 2010
Speech Quality Cont’d
R-value rangeEquivalent MOS Values
Speech transmission quality category
User satisfaction
90 R < 100 5 Excellent Very satisfied80 R < 90 4 Good Satisfied70 R < 80 3 Fair Some users dissatisfied60 R < 70 2 Bad Many users dissatisfied50 R < 60 1 Poor Nearly all users dissatisfiedConversion of R‑values into MOS
MOS = MOS = 1+0.035R + R(R-60)(100-R)7.10⁻⁶
Transmission Rating Factor conversion to MOS equivalent
Nairobi - Kenya 26 - 27 July 2010
MOS User Satisfaction Levels
R-Value
MOS Value
Comment Country
R ≤ 0 1 Nearly all users dissatisfied
R = 55 2.835 Many users dissatisfied Kenya
R = 60 3.1 Some users dissatisfied Nigeria/Kenya in 3yrs
R = 70 3.597 Some users dissatisfied
R = 80 4.024 Satisfied
R ≥ 100
4.5 Very satisfied Satisfied
MOS = MOS = 1+0.035R + R(R-60)(100-R)7.10⁻⁶
Typical values used in
Nairobi - Kenya 26 - 27 July 2010
Data/Multimedia Parameters Critical to Users
Delay variationsOccurs at the transport layer in packetized data systems due to the inherent variable arrival times of individual packets
Solved through buffering
Delaythe time taken to establish a particular service from the initial user request and the time to receive specific information once the service is established.
Packet LossOf packets or bits during transmissionIncludes coding degradation
Nairobi - Kenya 26 - 27 July 2010
Multimedia (Audio and Video) QoS Parameters and targets
KPI
1-way delay
Delay Variation
Information Loss (Packet Loss Ratio-PLR)
Audio Conversational Voice <150 ms < 1 ms < 3%
Audio Voice Messaging < 1 s playback
< 1 ms < 3%
High Quality Streaming Audio < 10 s << 1 ms < 1%
Videophone (2-way) < 150 ms < 1%
Videophone (1-way) < 10 s < 1%
Nairobi - Kenya 26 - 27 July 2010
Multimedia QoS Parameters
KPI Symmetry T-Values Delay
Web-browsing– HTML
1-way ~10 KB P< 2 s /pageA < 4 s /page
No delay variation or information Loss
KeyP-preferredA-AcceptableT-typical
Bulk data transfer/retrieval
1-way 10 KB-10 MB P < 15 sA < 60 s
Transaction services – high priority e.g. e‑commerce
2-way < 10 KB P < 2 s A< 4 s
Command/control 2-way ~ 1 KB < 250 ms
Still image 1-way < 100 KB P < 15 sA < 60 s
Interactive games 2-way < 1 KB < 200 ms
Telnet 2-way < 1 KB < 200 ms
E-mail (server access) 1-way < 10 KB P< 2 s A< 4 s
E-mail (server to server transfer)
1-way < 10 KB Can be severalminutes
Nairobi - Kenya 26 - 27 July 2010
QOS REGULATION IN AFRICA AND COMPARISON WITH INTERNATIONAL
STANDARDS
Case studiesKenyaNigeriaUgandaTanzania
Nairobi - Kenya 26 - 27 July 2010
Status of Country QoS Regulation
Nigeria Tanzania Uganda Kenya
QoS regulated √ √ √ √
QoS Parameters and Targets √ √ √ √
Regulations x √ x x
Guidelines √ x √ X
Licenses x x x √
Objective parameters √ √ √ √
Subjective Parameters √ √ √ √
PSTN Network x √ √ √
Cellular Mobile Network √ √ √ √
Internet x √ √ √
Data (Leased Lines) x √ √ x
Nairobi - Kenya 26 - 27 July 2010
Objective KPIs and Targets Benchmarks
KPIITU Targets
Kenyan Targets
Nigeria Tanzania Uganda
Completed Call (%) - 90 90 >99 99
Speech Quality (MOS)%age with Good SQ
4.0 ≤ MOS ≥ 4.5
95%>2.7 & 3.1 in 3 yrs
98 >2.0 MOS
>95 95
Call Drop Rate (%) - 2 2 <3 2
Call Block Rate (%) 1 10 - - 2
Call Set Up Time (s) varied 13.5 < 10 -
Call Set Up Success Rate (%)
90 90 - -
Handover Success Rate (%) 99.5% 85 90 - -
Call release delay 1s - - 2s -
Multimedia (IP) QoS Varied ITU - - -
Nairobi - Kenya 26 - 27 July 2010
Subjective QoS parametersAccount complaint rateAccount complaint resolution timeDisconnection complaint rateDisconnection complaint resolution time Miscellaneous complaint rate Miscellaneous complaint resolution time Fault report rate Fault repair timeBilling Accuracy Service supply time Call centre answer success ratio Call centre answer timeComplaint resolution
Nairobi - Kenya 26 - 27 July 2010
Conclusion
ITU-T recommendations have covered many QoS issues but very few targets have been specifiedWhere ITU-T targets are specified, they should be held as minimum thresholds not negotiable downwardsMany regulators monitor QoS parametersTargets adopted by regulators vary widely
Nairobi - Kenya 26 - 27 July 2010
Questions
How many regulators monitor QoS performance of providers?What are the KPI’s monitored?How comprehensive are KPIs in addressing customers requirements?How do the targets compare with international standards or best practices?Is there need for African Regulators to benchmark and adopt similar and adequate parameters and targets?
Nairobi - Kenya 26 - 27 July 2010
Thank You
Nairobi - Kenya 26 - 27 July 2010