straightforward | trusted | intelligenta2.mndcdn.com/image/upload/oy3yvacoqjbf6bzvr2wk.pdf ·...
TRANSCRIPT
STRAIGHTFORWARD | TRUSTED | INTELLIGENT
Competitive Network Benchmark for Telenor Sverige AB Final Report, Data
Mikko Valtonen, Sampsa Laamanen | 27 February 2013
©Omnitele Ltd. 2012 2 Competitive Network Benchmark for Telenor Sverige AB
table of contents
1. Executive summary
2. Omnitele in brief
3. Introduction
4. Methodology
5. Results
6. Annexes
©Omnitele Ltd. 2012 3 Competitive Network Benchmark for Telenor Sverige AB
executive summary 1/2
Scope: Telenor, Telia
Tests executed in three different scenarios:
− Data 3G: iPhone 5
− Data 3G/4G: Samsung Galaxy Note 2
− Data 3G/4G: Huawei E3276 USB Modem
Test campaign was extensive:
− 79 cities
− 550 test locations
− 5 000+ test cases per operator
Telenor contracted Omnitele to benchmark the end-user QoS of data services in Sweden in Q1/2013
©Omnitele Ltd. 2012 4 Competitive Network Benchmark for Telenor Sverige AB
Telenor has higher average bitrate across all scenarios
Telia scores higher in Total File Transfer Success Rate
Combining both factors (bitrate and file transfer success rate), Telenor performs better in 67% of the measured locations
executive summary 2/2
©Omnitele Ltd. 2012 5 Competitive Network Benchmark for Telenor Sverige AB
table of contents
1. Executive summary
2. Omnitele in brief
3. Introduction
4. Methodology
5. Results
6. Annexes
Straightforward | Trusted | Intelligent
Omnitele in brief
Omnitele was founded in 1988 to set up
the first GSM operator and network in
the world.
Omnitele has assisted operators in
1,000+ projects in 80+ countries.
Our services range from Technology
Strategy to Network Design and
Network Quality Improvement for
Telecom operations
“Since the beginning of DNA, Omnitele has played a critical role in developing an efficient model for radio network planning and its outsourcing.”
Riitta Tiuraniemi, Managing Director, DNA
The headquarter situates in Helsinki,
Finland. We have local presence in the
Netherlands and the Hashemite Kingdom
of Jordan.
The company is owned by Finnish telecom
operator groups and is independent of all
international operators and telecom
equipment vendors.
Straightforward | Trusted | Intelligent
Bridging the gap between technology & business
Technology Strategy
Evaluating, selecting and managing licenses,
technologies and vendors
Network Design
Turnkey network planning services for
mobile and broadcasting networks
Network Quality
Network auditing, optimisation, quality
benchmarking and performance
management services
“Due to excellent work by Omnitele we achieved our goal to be best in quality”
Dave Newbold Chief Operations & Technology Officer at JT
Straightforward | Trusted | Intelligent
1,000+ projects in 80+ countries
©Omnitele Ltd. 2012 9 Competitive Network Benchmark for Telenor Sverige AB
table of contents
1. Executive summary
2. Omnitele in brief
3. Introduction
4. Methodology
5. Results
6. Annexes
©Omnitele Ltd. 2012 10 Competitive Network Benchmark for Telenor Sverige AB
introduction | scope of work
OBJECTIVE
To evaluate and benchmark the end-user QoS of Swedish cellular networks Telenor and Telia under different data scenarios
SCOPE COVERAGE
Measurements in all cities with population > 15 000
Minimum 2 locations per city, 50 test samples per city, 5 test samples per location
Locations chosen independently by Omnitele to cover places with high subscriber density (airports, hotels, restaurants, shopping malls, railway/bus terminals, business districts...)
Alltogether 79 cities, 550 test locations, 5000+ test samples per scenario for each operator
Data Scenarios
FTP download with iPhone5 (3G)
FTP download with Galaxy Note2 LTE (3G/4G)
FTP download with Huawei E3276 LTE USB Modem (3G/4G)
©Omnitele Ltd. 2012 11 Competitive Network Benchmark for Telenor Sverige AB
introduction | project tasks in practice
VAST TEST CAMPAIGN
79 cities
550 locations
250+ testing hours
30 000+ FTP transfers
Tests executed Jan 15 – Feb 7
Test Scope in Practice
Test Execution Totals
Benchmarked Operators 2
Number of cities 79
Number of test locations 550
Calendar days of testing 23
Aggregates Totals
Successful file transfer tests 30 188
File transfer hours 252
Total GBytes transfered 2 271
Average bitrate Mbit/s 20.1
©Omnitele Ltd. 2012 12 Competitive Network Benchmark for Telenor Sverige AB
table of contents
1. Executive summary
2. Omnitele in brief
3. Introduction
4. Methodology
5. Results
6. Annexes
©Omnitele Ltd. 2012 13 Competitive Network Benchmark for Telenor Sverige AB
Nemo Outdoor Measurement PC
methodology | system overview
GPS
Data Test Terminals*
ftp.sunet.se
DATA
DC to AC Inverter
Apple iPhone5
Samsung Galaxy
Note2 LTE
Huawei E3276 USB Modems
*Terminal model details in annex
Nemo Outdoor is an industry standard laptop-based drive test tool supporting over 200 terminals and scanning receivers from various vendors and all major network technologies. More information from www.anite.com
©Omnitele Ltd. 2012 14 Competitive Network Benchmark for Telenor Sverige AB
All data scenarios in series, one scenario at time
Data test case: N x 30 sec FTP download*
methodology | test cycle overview
Telenor
Telenor
SCENARIO 1.3
Huawei E3276 USB Modem
3G/4G
Telenor
SCENARIO 1.1
Apple iPhone5
3G
SCENARIO 1.2
Samsung Galaxy Note2 LTE
3G/4G
Telia
Telia
Telia
*N = location specific value ranging from 5…25, so that in each city minimum 50 samples are collected.
©Omnitele Ltd. 2012 15 Competitive Network Benchmark for Telenor Sverige AB
methodology | test procedures
TESTING TIMES REPRESENT TYPICAL END-USER BEHAVIOUR:
• Test days: Testing only between Monday and Saturday.
• Test hours: Testing not conducted during night hours: 00.00 – 06.00
• Saturdays: Measurements only between 10:00 – 00:00. No Saturday measurements in business or university areas.
• Sundays: No testing on Sundays.
METHODOLOGY
• Commercial state of the art smartphones and modems used to benchmark best available end-user QoS offered by the operators.
• Locations chosen independently in blind test fashion by Omnitele to cover places with high population density (airports, hotels, restaurants, shopping malls, railway/bus terminals, business districts...)
• All tests executed in stationary state inside a car. No mobility testing.
• Measurement devices always in same place, on the car dashboard
• Detailed results saved for each test separately. Failures recorded as well.
Coherent test procedure strictly followed. Methodology defined to capture true end-user behaviour. Measurements conducted where and when real subscribers are using data services.
©Omnitele Ltd. 2012 16 Competitive Network Benchmark for Telenor Sverige AB
table of contents
1. Executive summary
2. Omnitele in brief
3. Introduction
4. Methodology
5. Results
6. Annexes
©Omnitele Ltd. 2012 17 Competitive Network Benchmark for Telenor Sverige AB
Per Location Telenor Telia Total Locations
Total 1104 546 1650
Apple iPhone5 338 212 550
Samsung Galaxy Note2 LTE 436 114 550
Huawei E3276 330 220 550
66.9 %
61.5 %
79.3 %
60.0 %
33.1 %
38.5 %
20.7 %
40.0 %
0% 20% 40% 60% 80% 100%
Total Average
Apple iPhone5
Samsung Galaxy Note2 LTE
Huawei E3276
Score Board: Telenor vs Telia (Bitrate*TFTSR)
Telenor
Telia
data | scoreboard per location
Scoreboard metric: Average bitrate per file transfer * Total File Transfer Success Rate
- - - -
Total Average
Taking into account Average Bitrate and Total File Transfer Success Rate, Telenor scores higher in 67% of locations.
©Omnitele Ltd. 2012 18 Competitive Network Benchmark for Telenor Sverige AB
data | scoreboard per area
Telenor 68 %
Telia 32 %
Central-Eastern
Telenor 71 %
Telia 29 %
Western
Telenor 72 %
Telia 28 %
Stockholm Metropolitan
Telenor 65 %
Telia 35 %
Southern
Telenor 49 %
Telia 51 %
Northern
Notes: - Scoreboard metric: Average bitrate per file
transfer * Total File Transfer Success Rate - The total number of tests is 1650. In each
of the 550 test sites 3 different data scenarios were evaluated.
- List of cities per area provided as annex
Area Telenor Telia Total tests
Western Sweden 215 88 303
Southern Sweden 198 108 306
Central-Eastern Sweden 208 98 306
Stockholm Metropolitan Area 388 152 540
Northern Sweden 95 100 195
Total 1104 546 1650
Considering the different geographical areas of Sweden, Telenor scores higher in 4 out of 5 areas.
©Omnitele Ltd. 2012 19 Competitive Network Benchmark for Telenor Sverige AB
Telenor bitrate is higher than Telia across all measured scenarios
From international perspective the bitrates are very high and enable superior end-user experience
data | average bitrate per file transfer
10.4 28.2 32.4 8.6 15.2 25.5 0
5
10
15
20
25
30
35
40
Apple iPhone5 Samsung Galaxy Note2 LTE Huawei E3276
Mb
it/
s
Average Bitrate per File Transfer
Telenor
Telia
©Omnitele Ltd. 2012 20 Competitive Network Benchmark for Telenor Sverige AB
data | peak bitrates
PEAK RATES GENERALLY VERY HIGH,
Peak bitrate is the practical maximum bitrate an end-user can experience.
In 3G scenario both operators have almost equally high peak bitrates, Telenor scoring highest with 31.3 Mbit/s – very close to DC-HSDPA practical maximum.
In 4G scenarios Telenor is also better than Telia, following the same trend observed in average bitrates.
Although Note2 and Modem are both 4G devices, the latter one has higher performance as it has better signal processing and it is optimised for data.
31.3 81.7 103.1 26.3 66.3 71.3 0
20
40
60
80
100
120
Apple iPhone5 Samsung Galaxy Note2 LTE Huawei E3276
Mb
it/
s
Peak Bitrate
Telenor
Telia
©Omnitele Ltd. 2012 21 Competitive Network Benchmark for Telenor Sverige AB
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
Den
sit
y
Mbit/s
Bitrate per File Transfer, Apple iPhone5 (Cumulative Distribution)
Telenor
Telia
data | iPhone5 bitrate distribution (3G)
The cumulative distribution shows that Telenor has more samples in the higher end of the scale.
Samples above 10 Mbit/s: Telenor 49% Telia 43%
©Omnitele Ltd. 2012 22 Competitive Network Benchmark for Telenor Sverige AB
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
Den
sit
y
Mbit/s
Bitrate per File Transfer, Samsung Galaxy Note2 LTE (Cumulative Distribution)
Telenor
Telia
data | note2 bitrate distribution (3G/4G)
The cumulative distribution shows that Telenor has more samples in the higher end of the scale.
Samples above 30 Mbit/s: Telenor 48% Telia 12%
©Omnitele Ltd. 2012 23 Competitive Network Benchmark for Telenor Sverige AB
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
Den
sit
y
Mbit/s
Bitrate per File Transfer, USB Modem (Cumulative Distribution)
Telenor
Telia
data | usb modem bitrate distribution (3G/4G)
The cumulative distribution shows that Telenor has more samples in the higher end of the scale.
Samples above 30 Mbit/s: Telenor 54% Telia 37%
Bitrate per File Transfer, Huawei E3276 (Cumulative Distribution)
©Omnitele Ltd. 2012 24 Competitive Network Benchmark for Telenor Sverige AB
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
Total File Transfer Success Rate
Telenor
Telia
data | total file transfer success rate
From end-user perspective both operators perform well.
Telia is better with 99.6%, Telenor follows close behind with 98.9%
Total File Transfer Success Rate takes into account all file transfer errors in end-to-end fashion: “No network access”, “FTP server connect failed”, “File transfer setup failed”, “FTP connection dropped”. Anything above 99% threshold value can be considered extremely good.
Significantly better than Telenor
No statistically significant difference
Significantly worse than Telenor
KPI Telenor Tele2 Telia Tre
Total File Transfer Success Rate 98.9 % 99.6 % 99.6 % 98.5 %
Difference to telenor
Ranking 3 1 2 4
KPI Telenor Tele2 Telia Tre
Total File Transfer Success Rate 98.9 % 99.6 % 99.6 % 98.5 %
Difference to telenor
Ranking 3 1 2 4
©Omnitele Ltd. 2012 25 Competitive Network Benchmark for Telenor Sverige AB
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
Apple iPhone5 Samsung Galaxy
Note2 LTE
Huawei E3276
Total File Transfer Success Rate
Telenor
Telia
Significantly better than Telenor
No statistically significant difference
Significantly worse than Telenor
data | total file transfer success rate per scenario
Telia has the highest total file transfer success rate in all scenarios
The difference is statistically significant only in USB modem scenario
Both operators score very high in all scenarios meeting industry standards and typical end-user expectations
Telenor Telia
Apple iPhone5 98.6 % 99.4 %
Samsung Galaxy Note2 LTE 99.2 % 99.6 %
Huawei E3276 99.0 % 99.9 %
Total File Transfer Success Rate
©Omnitele Ltd. 2012 26 Competitive Network Benchmark for Telenor Sverige AB
table of contents
List of Annexes
1. Key Performance Indicators, Data
2. Confidence interval calculation
3. Test terminal details
4. Measurement scripts
5. 79 cities, 550 locations
6. Cities and Areas
©Omnitele Ltd. 2012 27 Competitive Network Benchmark for Telenor Sverige AB
annexes | key performance indicators, data
KPI Description Unit
Average Bitrate per File Transfer
Average application layer throughput per each downlink file transfer according to ETSI TS 102 250-2 specification: Total_Bytes_Received * 8 / (T2 – T1), Where T2 = time of closing the FTP connection T1 = time of sending the file download request to FTP server
Bit/s
Average Bitrate per City
Average over all Average Bitrate per File Transfer –samples within a city: SUM[Average_Bitrate_per_File_Transfer] / COUNT[Average_Bitrateper_File_Transfer]
Bit/s
Average Bitrate per Location
Average over all Average Bitrate per File Transfer –samples within a measurement location: SUM[Average_Bitrate_per_File_Transfer] / COUNT[Average_Bitrate_per_File_Transfer]
Bit/s
Network Accessibility
According to ETSI TS 102 250-2 specification: The probability that an end-user can access the network and gets IP connectivity. Formula: [#Packet_Attach_Success / #Packet_Attach_attempts]*
[#PDP_context_activation_success / #PDP_context_activation_attempts]
%
FTP Server Connect Success Rate
The probability that an end-user can setup the connection to the FTP server, provided that Network Access exists. Formula: #Successfull_FTP_server_connects / #FTP_server_connection_attempts
Pre-conditions: PDP context activated
%
File Transfer Setup Success Rate
The probability that an end-user can start file transfer, provided that FTP server connectivity exists. Formula: #Successfull_File_Transfer_Setups / #File_Transfer_Attempts
Pre-conditions: FTP server connectivity established
%
File Transfer Completion Rate
The probability that an end-user can complete transfer, provided that file transfer setup was successfull. Formula: #Successfully_Completed_File_Transfers / #Successfull_File_Transfer_Setups
Pre-conditions: Successful FTP file transfer setup
%
Total File Transfer Success Rate
The overall probability that an end-user can successfully make a file transfer taking into account all possible failures in end-to-end fashion. Formula: Network_Accessibility * FTP_Server_Connect_Success_Rate *
File_Transfer_Setup_Success_Rate * File_Transfer_Completion_Rate
%
©Omnitele Ltd. 2012 28 Competitive Network Benchmark for Telenor Sverige AB
annexes | confidence interval calculation
RATIONALE
In order to analyse the statistical reliability of the results, Omnitele applies confidence interval analysis on the calculated KPIs. Confidence Interval can be considered as the error margin of the reported results.
The error margins are visible in the graphs of this report in the error bars, see example figure below. In case two competing operators have differing mean values, but overlapping error bars, the observed difference in average value is not statistically significant. If the error bars don’t overlap, the difference is statistically significant.
CALCULATION
The calculated confidence interval CI is based on (two-tailed) confidence level of 95%. That is, with 95% probability the true population mean is within the sample mean +/- CI.
The CI is calculated as 1.96 x SE, where
SE is equal to the standard error for the sample mean, and
1.96 is the .975 quantile of the normal distribution (CL = 0.95 α = 0.05 1- α/2 = 0.975, Norm.Inv[0.975] = 1.96)
Standard Error SE of the sample mean, is defined as s / 𝒏, where
s is the sample standard deviation (i.e., the sample-based estimate of the standard deviation of the population), and
n is the size (number of observations) of the sample.
95%
96%
97%
98%
99%
100%
KPI1 KPI2
Operator 1
Operator 2
Statistically significant difference
Statistically insignificant difference
©Omnitele Ltd. 2012 29 Competitive Network Benchmark for Telenor Sverige AB
annexes | test terminal details
Small screen 3G iPhone5
OS Version IOS 6.0.2 (10A551)
Modem Firmware 1.01.00
Sourced from Finland (Gigantti, Stockman)
Small screen 3G/4G Samsung Galaxy Note 2 LTE
Model GT-N7105
OS Version Android 4.1.1 Jelly Bean
Sourced from Sweden (ElGiganten, Phone House)
Large screen 3G/4G, Telenor Huawei E3276
Firmware version 21.260.03.00.00
Connection manager 23.009.05.00.03
Sourced from Sweden (Telenor)
Large screen 3G/4G, Telia Huawei E3276
Firmware version 21.192.03.01.07
Connection manager 21.005.18.00.03
Sourced from Sweden (Telia store)
Small Screen 3G, Small Screen 3G/4G:
Same individual terminal used for all operators
Devices sourced with blind test approach
Large Screen 3G/4G
Each operator tested with operator specific terminal
Flagship USB modems used
Latest firmware update applied
©Omnitele Ltd. 2012 30 Competitive Network Benchmark for Telenor Sverige AB
annexes | measurement scripts
1. Connect to FTP server: ftp.sunet.se
2. Start file download
3. Wait 30 seconds
4. Stop file download
5. Disconnect FTP
6. Wait 10 seconds
7. Go to 1
©Omnitele Ltd. 2012 31 Competitive Network Benchmark for Telenor Sverige AB
annexes | 79 cities, 550 locations
City Locations Tests/OP City Locations Tests/OP City Locations Tests/OP
Åkersberga 3 51 Kiruna 2 50 Piteå 3 51
Alingsås 3 51 Köpings 2 50 Sandviken 3 51
Ängelholm 3 51 Kristianstad 4 52 Skellefteå 4 52
Boden 2 50 Kristinehamns 2 50 Skövde 4 52
Boo 3 51 Kungälvs 3 51 Södertälje 7 56
Borås 7 56 Kungsbacka 2 50 Stockholm 138 690
Borlänge 5 50 Landskrona 4 52 Sundsvall 6 54
Enköping 3 51 Lerum 2 50 Täby 7 56
Eskilstuna 7 56 Lidingo 4 52 Trelleborg 3 51
Eslov 2 50 Lidköping 3 51 Trollhättan 5 50
Falkenberg 3 51 Linköping 11 55 Tumba 4 52
Falköping 2 50 Ljungby 2 50 Uddevalla 4 52
Falun 4 52 Luleå 5 50 Umeå 8 56
Gävle 8 56 Lund 9 54 Upplands 4 52
Göteborg 55 275 Malmo 29 145 Uppsala 15 75
Halmstad 6 54 Mariestad 2 50 Vallentuna 3 51
Härnösand 2 50 Märsta 3 51 Vänersborgs 3 51
Hassleholm 2 50 Mölnlycke 2 50 Varberg 3 51
Helsingborg 10 50 Motala 3 51 Värnamo 2 50
Hudiksvall 2 50 Nässjö 2 50 Västerås 12 60
Jönköping 9 54 Norrköping 9 54 Västerhaninge 2 50
Kalmar 4 52 Norrtälje 2 50 Vastervik 3 51
Karlshamn 2 50 Nyköping 3 51 Vaxjo 7 56
Karlskoga 3 51 Örebro 11 55 Visby 3 51
Karlskrona 4 52 Örnsköldsvik 3 51 Ystad 2 50
Karlstad 7 56 Oskarshamn 2 50
Katrineholm 3 51 Östersund 5 50
Aggregates
Total Towns/Cities 79
Total test locations 550
Total tests/operator/scenario 5067
©Omnitele Ltd. 2012 32 Competitive Network Benchmark for Telenor Sverige AB
annexes | cities and areas
City per Area division of scoreboard calculation is summarised below
Western Sweden Southern Sweden Central-Eastern
Sweden Stockholm
Metropolitan Area Northern Sweden
Alingsås Ängelholm Eskilstuna Åkersberga Boden
Borås Eslöv Falköping Boo Borlänge
Göteborg Falkenberg Karlskoga Lidingö Enköping
Jönköping Halmstad Karlstad Märsta Falun
Kungälv Hässleholm Katrineholm Norrtälje Gävle
Kungsbacka Helsingborg Köping Södertälje Härnösand
Lerum Kalmar Kristinehamn Stockholm Hudiksvall
Ljungby Karlshamn Lidköping Täby Kiruna
Mölnlycke Karlskrona Linköping Tumba Luleå
Nässjö Kristianstad Mariestad UpplandsVäsby Örnsköldsvik
Trollhättan Landskrona Motala Vallentuna Östersund
Uddevalla Lund Norrköping Västerhaninge Piteå
Vänersborg Malmö Nyköping Sandviken
Värnamo Oskarshamn Örebro Skellefteå
Trelleborg Skövde Sundsvall
Varberg Uppsala Umeå
Västervik Västerås
Växjö Visby
Ystad
Straightforward | Trusted | Intelligent
Want to know more?
Visit www.omnitele.com for news,
references, and events and check out our
blog for the latest Omnitele information:
Read our Blog
Find out about events
Check the latest news
View customer references
Read market commentaries