Public

Public

2020 Benchmarking Framework

Overview PresentationDocument Version V1.0, April 2020 7-Apr-20

Public

Preface. The umlaut Benchmarking Framework is based on a unified measurement method for true international comparability combined with the umlaut-Score approach.

2020 umlaut scoring framework 2

Key Features:⚫ Unified Methodology and Score Scheme, globally used in a wide range of activities, both public and non-public

→ International comparability

⚫ User-centric scoring and test cases plus crowd-based assessment

→ end customer relevance

⚫ Full drill-down capability

→ supports engineering and optimization work

⚫ Yearly methodology evolution in dialogue with network operators

→ Transparency, up to date

⚫ Common basis for public benchmarks and independent network certification, rooted in ETSI TR 103 559 recommendations

→ ideal PR tool

⚫ 5G capable setup

→ supporting network evolution

7-Apr-20

Public

Rationale. Measuring performance means collecting many different performance indicators – how to stay focused and easily communicate results?

2020 umlaut scoring framework 3

Measuring Network Performance⚫ Historically, benchmarking results look at a vast multitude of

indicators, easily missing the important aspects

The umlaut Score as a management Tool⚫ The umlaut score aggregates the complete set of KPIs into a

single score figure, preserving the drilldown capability

⚫ It features implicit weighting of environment, services and KPIs and comparability on national and international level

Distracting details

Tons of (K)PIs

What isimportant?

unique performance

profile

Benchmark

History

Learning from best-

in-class

Manageable targets

Priorities

0 1000

7-Apr-20

Public

umlaut Score Principles-Building Blocks. The umlaut score delivers a robust scale to assess network quality and performance. It is coupled with a unified measurement methodology for true international comparability.

2020 umlaut scoring framework 4

umlaut-Score: 1000 PtsOverallScore

Modular Score Approach

NT Data NT Voice UET

City Town Road All

QualifierKPIs

Differen-tiator KPIs

CrowdKPIs

Unified Measurement Methodology

- Measurement Systems- Test cases and Settings- Device capabilities- Geographical Mix- KPIs, post processing- Mapping functions

ServiceModules

ConsistentReporting

- C-level Views- Mgmt. Level

Breakdowns- Drilldown to KPI-Level- Cross-Project Reports

Area Level

KPILevel

7-Apr-20

Public

umlaut Score Principles-Building Blocks ii. For drive and walk testing, each service session is characterized by a Qualifier KPI and one or several Differentiator KPIs.

2020 umlaut scoring framework 5

Qualifier KPIs

Model minimum requirements that have to be met to make the experience of the mobile network satisfactory.

Examples:Voice: Percentage of successful and sustainable callsData: Percentage of sessions faster than 500kbps

Qualifiers are worth:55% of all the points

Differen-tiator KPIs

Differentiator KPIs describe additional aspects of those services that fulfil the qualifier criteria

Examples:Voice: (Avg. & P90) Call Setup Time, (Avg. & P10)

Speech Quality, Data availabilityBrowsing: Avg. Session Duration5MB File: Avg. Session Duration, 10 and 90 Percentile of

Throughput distribution

Differentiators are worth:45% of all the points

For every Service usage, a simple Success / Failure Distinction is made

7-Apr-20

Public

Concept Overview. Characterized by (i) defined weighting between environments and services (ii) fixed maximum score (iii) modular extensions (iv) qualifiers and differentiators.

2020 umlaut scoring framework 6

PropertiesKPIs are grouped into⚫ Qualifiers – achievement of min. requirements⚫ Differentiators – meeting state-of the art targetsPoints in both groups are determined by mapping functions for each individual KPI, yearly updatedThe Baseline umlaut-Score covers a default combination of different environments:⚫ Drive test Voice and Data⚫ Cities, Towns, RoadsAnd is optionally extended by⚫ Walk test⚫ Indoor city, public transport, commuter rail, railways⚫ Crowd-based scoreThe core umlaut-Score covers a default combination of services. Services may be added or dropped from the score on a yearly basis

Advantages⚫ Fixed maximum score allows historic comparison: „A network with 900

points in 2020 has the same performance in relation to the available state of the art as a network with the same points in 2014“

⚫ Fixed categories make the scoring invariant to the detailed campaign planning and the number of samples collected in each category

⚫ Threshold values and mapping functions for qualifiers are stable for min. 1 year (unless change technically required)

Requirements⚫ Threshold values and mapping functions for differentiators still require

regular adaptation to state of the art → yearly update process in place

⚫ Measurements satisfying the scoring requirements have to address a minimum set of scenarios and services → addressed by umlaut‘s best practice approach

7-Apr-20

Public

Test and crowd assessment methodology.Test cases and modules need to be included in the campaign scope.

7

Core RequirementsThe red and blue sets of test cases represent the Network Test.The green modules represent the User Experience Test (UET) component based on crowdsourced data.

M2M Voice

4G preferred to

4G preferred UE

Controlled Background Traffic

Web Browsing (static and live)

Reference web page download, HTTP and HTTPS

Popular live web pages download,

HTTP, HTTPS

HTTP/S File Up-/Download

Fixed Size

Fixed Size HTTP, HTTPS File download

Fixed Size HTTP, HTTPS File upload

YouTube

APP

YouTube Video download &

playout

Video-on-Demand (VOD),

Live Stream

HTTP File Up-/Download

Data stream

Fixed Duration multi socket download

Fixed Duration multi socket upload

BroadbandCoverage

Geographic:Broadband Coverage

Excellence (based on quality of

coverage metric)

User-Centric:

Time on Broadband

Latency

/Interactivity

RTTbased capability

for:

Gaming

OTT Voice

Data Speed

Download speed metrics based on

passive observation of real user traffic

Capability for:

Basic Internet

HD Video

UHD Video

Network Test: Customer Experience 4G and 5GNetwork Test:

Capability 4G and 5GCrowd: User Experience Assessment

2020 umlaut scoring framework 7-Apr-20

Public

Score relation and coexistence of different measurement scopes.In different activities performed by umlaut, different score models will coexist, but maximum scores per scenario are harmonized.

2020 umlaut scoring framework 8

480360 360

120 120

160 160 160

160 160100

60

800

600

200

200 200 200 200 200

1000

0

100

200

300

400

500

600

700

800

900

1000

Baseline umlaut-Score 2020 +Crowd

Incl. City Walktest 2020 +Crowd

Incl. City/Train Walktest 2020 +Crowd

Smallcountry Drive 2020 Small Country Drive + Walk2020

Crowd Only 2020

Cities Drivetest Cities Walktest Towns Drivetest Roads Drivetest Train Walktest Overall Drivetest Overall Walktest Crowd

7-Apr-20

Public

In different activities performed by umlaut, different score models will coexist, but maximum scores per scenario are harmonized.

2020 umlaut scoring framework 9

Overview of measurement scenarios with different geographical modules

⚫ All models share 1000 point maximum⚫ 800 achievable points for Network Testing (NT, drive and walk testing)⚫ 200 achievable points for User Experience Testing (UET, crowdsourcing component)

⚫ In case of non-availability of UET, NT points are scaled up to 1000 max⚫ In case of UET only, UET points are scaled up to 1000 max⚫ In all NT categories, the voice / data split 40 / 60 (not shown)⚫ In all NT categories, the weight of individual services and KPIs is the same

Score relation and coexistence of different measurement scopes.

7-Apr-20

Public

Public Benchmarks, Network Certification and Claim Support.The umlaut benchmarking activities with public visibility can be based on dedicated testing and crowd based assessment.

2020 umlaut scoring framework 10

Description of Approachumlaut activities with public visibility are:⚫ Public benchmarks executed in cooperation with a publishing partner⚫ Independent network certification and claim support activities, commissioned by one or more 3rd parties (e.g. mobile network operator,

network vendor, etc.)

The setup, scope and timeline⚫ for any certification and claim support activity is independently chosen and decided by umlaut⚫ for any public benchmarking activity is independently chosen and decided by the publisher and umlaut

The available components for any activity are⚫ Dedicated Network Testing activity for Smartphone voice and data service, consisting of drive test and optional walktest⚫ Crowd based User Experience assessment of coverage, data speeds and latencyPer default, network certification activities are a combination of dedicated testing activity + crowd based assessment

Exceptions may apply in markets where crowd panel size/distribution are not meeting umlaut requirements. The decision will be taken independently by umlaut. In such cases, all score categories can be proportionally expanded to reach a 1000 point maximum score.

7-Apr-20

Public

www.umlaut.com

2020 umlaut score framework concept 114/7/2020

Public2020 umlaut score framework concept 12

Disclaimer

This document and all information contained herein is the sole property of umlaut.No intellectual property rights are granted by the delivery of this document or the disclosure of its content. This document shall not be reproduced or disclosed to a third party without the express written consent of umlaut. This document and its content shall not be used for any purpose other than that for which it is supplied.

4/7/2020