5g benchmarking optimization and - mobile wireless testing€¦ · 5/12/2019 · subscription...

Arnd Sibila

Rohde & Schwarz Mobile Network Testing

5G BENCHMARKING AND OPTIMIZATION

Rohde & Schwarz

MOBILE NETWORK LIFECYCLE

Dec 2019 5G optimization and benchmarking2

Quality Benchmarking and

Network Optimization

are closely connected

Ideally, benchmarking data

should be used for optimization

Optimization actions should be

proven by benchmarking

Rohde & Schwarz

CONTENTS

Dec 2019 5G optimization and benchmarking

ı Market drivers and scoring motivation

ı Benchmarking methodology using NPS

ı Basis for network optimization

ı Example: Using benchmarking data for

optimization

ı Is Network Optimization in 5G different?

3

Rohde & Schwarz

MOTIVATION: WHY QUALITY OF EXPERIENCE INTELLIGENCE?

5G optimization and benchmarking4

Subscription penetration Q3 2019 (percent of population)

Nearly no people without subscriptions in most regions

New subscribers have to be gained from competing operators

Operators have to differentiate in provided Network Performance and Quality of Experience (QoE)

Perceived QoE of applications determines end user satisfaction and operators’ success

Applications!

Dec 2019

Source: Ericsson Mobility Report (Nov 2019)

Rohde & Schwarz

THE VALUE OF BENCHMARKING

Accurate and reliable benchmarking data has a lot of value:

► Insights about Quality of Experience of networks (user-perceived quality)

► Compare the own network quality with competitors’ networks

Some points have to be considered though:

► User-perceived QoE test most used service types

► Data collection: accurate and reproducible (temperature, shielding, …)

► Use devices that support ALL technologies of network under test

► Data from all layers (Application Layer down to spectrum information).

Needed for a deep dive can trigger optimization activities or defend against legal claims

Clear differentiation to e.g. crowdsourcing

5G optimization and benchmarking5 Dec 2019

Rohde & Schwarz

CURRENT SITUATION OF NETWORK SCORES

► Operator pays a service provider for a network scoring campaign or crowdsourcing data that promise

a certain result

► There are many different network scores in the market.

A few examples:

Connect Score / P3 Score

Chip Score

OneScore from GWS

RootMetrics score

OpenSignal score

Huawei/CMCC NEI

Ookla

…

► Different methodologies (often not disclosed) proprietary methods


POOR GOOD

All these existing scores are not comparable

ETSI STQ ratified and published TR in August 2019

Scored AT&T video streaming

Dec 2019

Rohde & Schwarz

THE VALUE OF A COMPARABLE NETWORK SCORE

► For Operator Marketing: to create marketing claims

“Best Network”, “Fastest Network”, “Best for Streaming services”,..

► For the Operator CTO: as a management tool

“I want my team to improve the network quality by n % over next m month!”

► For Infrastructure vendors: as a competitive advantage

“The networks using our infrastructure perform better than xy”

► For Service Companies: to generate more business

“We can help you to win next years public benchmark”


Rohde & Schwarz

CONTENTS






optimization


8

Rohde & Schwarz 5G optimization and benchmarking9

NETWORK PERFORMANCE SCORE

Represents the user-perceived QoE

Standardized (ETSI STQ TR 103 559) and accepted

Vendor independent, objective methodology:

► Reference benchmarking metric

► Holistic network view across competitors

► Target setting for improvements

► Framework for a QoE-based

investment prioritization

Dec 2019

Rohde & Schwarz

NETWORK PERFORMANCE SCORE: DIFFERENT LEVELS OF AGGREGATION

One score

per country

Regional aggregation

QoE Score for each service class based on

voice and data KPIs

Level 3: Weight and aggregate all regions to a

countrywide score (the “CxO level”)

Level 2: Weight and aggregate QoE of all

service classes per region

Level 1: Evaluate QoE for all service classes

(different dimensions of a service class)


Methodology currently in standardization in ETSI STQ (TR 103 559 draft available)

“Best practices for robust network QoS benchmark testing and scoring”

Rohde & Schwarz

Use existing KPIs for QoE evaluation:

► Service availability

Do I have access to the service at all?

Do I stop waiting because of too long waiting times?

► Waiting for ‘action’ (task being started and/or completed)

How is the accepted duration (patience) for a normal ‘web task’,

getting a call connected or seeing the video starting.

► How is the quality of the media (e.g. video, voice, pictures,…)

Is the quality how I expect it?

WHAT DRIVES QOE OF A SERVICE? WHAT ARE DIMENSIONS OF IT?

QoE

of a service

One

score per country


Score of QoE of each service class based on

voice and data KPIs


Rohde & Schwarz

LEVEL 1: HOW TO EVALUATE QOE OF SERVICE CLASSES?

12

How to rate KPIs different dimensions to each other? Simple example for telephony

► (Only) what can be perceived, should have an impact on scoring

► Normalize to a common scale (0 … 1000)

► Weight and Aggregation

One

score per country



voice and data KPIs


Rohde & Schwarz

LEVEL 1: EVALUATE QOE OF SERVICE CLASSES?

13

►Perception and normalization to a common scaleOne

score per country



voice and data KPIs

5G optimization and benchmarking

no benefit!

Target!

technical improvement

Perception ~ f(technical improvement)

Dec 2019

Rohde & Schwarz

DOES DATA THROUGHPUT DRIVE THE PERCEPTION?NO! …BUT TIME DOES!

Data Throughput

Saturation

This dominates the ‚Throughput KPI‘ average But there is no difference in perception!

Data throughput/Mbps Data throughput/Mbps

Duration of a task


One

score per country



voice and data KPIs

Rohde & Schwarz

► Video on YouTube Video resolution

WHAT DRIVES PERCEPTION IN VIDEO QUALITY?

MOS

Video resolution as KPI is not sufficient to rate the video quality

Video MOS according to ITU-T J.343.1 rates the quality of the video on the screen

Viewing distance is important


One

score per country



voice and data KPIs

Rohde & Schwarz

HOW TO TAKE INTO ACCOUNT THE SATURATION IN PERCEPTION?

16 5G optimization and benchmarking

0%

100%

Low

threshold

‘bad’

High

threshold

‘good’

0%

100%

80% 100%

Streaming Success Ratio

0%

100%

3 4.5

Average Video MOS J343.1

0%

100%

1 Mbps 100 Mbps

Average Mean Data Rate

0%

100%

4.5 s 12 s

Average Call Setup Time

Some examples

DL

Dec 2019

One

score per country



voice and data KPIs

Rohde & Schwarz

NETWORK PERFORMANCE SCORE – LEVEL 2:WEIGHT AND AGGREGATE ALL SERVICE CLASSES PER REGION


One

score per country



voice and data KPIs

QoE of service classes Weight Aggregation

Rohde & Schwarz

THE NETWORK PERFORMANCE SCORE – LEVEL 3:THE COUNTRY-WIDE LEVEL


One

score per country



voice and data KPIs

The Network Performance Score rates the overall QoE of a network by combining

the fulfillment of main KPIs from different services.

Rohde & Schwarz

CONTENTS






optimization


19

Rohde & Schwarz

THE NETWORK PERFORMANCE SCORE:AGGREGATION AND DRILL DOWN


Aggregation:

ı Regional weighting (acc. to traffic, population, …)

ı Benchmarking different operators

ı Comparison of technology and (infra) vendors

ı Inter-market comparison (compare countries)

Drill down:

ı Critical region

ı Critical service class

ı Critical dimension of service class

Basis for network optimization

+

Rohde & Schwarz

NETWORK OPTIMIZATION: FROM LABOUR-INTENSIVE WORK TO AUTOMATED ACTIONS


ı Accurate data collection in a comparable way is essential

ı Sophisticated data analytics with increasing automation required

ı Operators have to continuously measure and improve network performance and end-user QoE

ı New reliability and latency measurements to be defined

Traditional way What has changed? The new way

ı Capacity and data rates

ı Capacity upgrades lasted

longer

ı Customer complaints drove

optimization

ı Manual trouble shooting and

network optimization

ı Mobile data explosion due to

video

ı Tougher competition

ı 5G adds flexibility and

complexity

ı New 5G use cases require

reliability and low latency

ı Perceived QoE is focus

ı ETSI harmonized scoring

methodology for quality

ı Deep insights into network

performance and trend

analysis provide automated

action recommendations

Network Performance Score is the ETSI harmonized QoE scoring and benchmarking method

Dec 2019

Rohde & Schwarz

NETWORK PERFORMANCE SCORE


All measurement details are available for further analysis and drill-down

Dec 2019

The famous ONE figure per

countrywide network

performance (measured from

end users’ perspective QoE)

► Voice score (40% weight)

► Data score (60% weight)

Rohde & Schwarz

QOE CRITERIA AND KPIS (EXAMPLE: VOICE SERVICE)


10th percentile: additional points for low number of very bad samples /

90th percentile: additional points for high number of very good samples

Dec 2019

Service Availability (high weight)

Waiting for action

Quality of the media

Voice KPIs

Shows points including complete weighting

and the potential to max achievable points

Rohde & Schwarz

QOE CRITERIA AND KPIS (EXAMPLE: DATA SERVICE)


Data KPIs

Higher weight to

ı Browsing Services

ı and Video services

Higher weight to availability compared

to waiting time and media quality

More to come in the demonstration

Rohde & Schwarz

CONTENTS






optimization


25

Multi Country Benchmarking

in Africa to Improve

Customer Experience

Group Chief Technology Officer

Navindran Naidoo

Market Capitalization

Largest Market

Share in

Countries

million subscribers236

$28bn

Extensive

quality

infrastructure: 23countries across

Africa and the Middle East

14Customers

28M

MTN Leading Operator in Africa

Iran

Benin

Afghanistan

Guinea Conakry

Ivory

Coast

Nigeria

South

Africa

Liberia

Sudan

Ghana Uganda

Zambia

Rwanda

Swaziland

South

Sudan

Syria

Yemen

Guinea Bissau

Namibia

Congo

Brazzaville

Kenya

Cameroon

Botswana

ISP businesses: Namibia, Botswana, Kenya

27

Rationale for MTN standardising Benchmarking

Moving from a Network KPI centric assessment to Customer Centric Approach

2017 – Network KPI Focus

7 OPCOS #1 Network

NPS as at Dec 201712

2018 – Customer Centric Approach

OPCOS #1 Network NPS as at Aug 2019

Assessment conducted by

MS Vendor

Different methodologies

per vendor.

Vendor feature specific

recommendations

KPI focus methodology

and scores

Moving from a Network KPI centric assessment to Customer Centric Approach

Ensure one consistent version of the truth from a

standardised methodology

Independent and regulatory/industry aligned

assessment

Benchmark MTN network experience against

competitors as well as European markets.

Customer centric outcomes used to improve

customer experience

28

South Africa Results and Actions

Overall Benchmarking

Ranking

Network NPS :

2018 1st Round Results

Overall Benchmarking

Ranking:

Aug 2018

Network NPS :

2018 2nd Round Results

Sample Actions implemented.

• Improving call retainability (CDR in rural areas (roads) where the ratio of

dropped calls are higher)

• Implement AMR-WB 23.85 Kbit/s speech coding for UMTS

• Troubleshoot the UMTS RRC performance issues in Johannesburg

• Extend the usage of 256 QAM, 4x4 MIMO and 3CCA to compensate for the higher

spectrum utilization by competitor

29

Rohde & Schwarz

CONTENTS






optimization


30

Rohde & Schwarz

ISOLATED SINGLE 5G NR SITE (3 SECTORS // 3 X 7 BEAMS)


► Sharp split of individual beams per cell► Clear 3 sectors with different PCIs,

with some reflections / misalignments

Rohde & Schwarz

NUMBER OF MEMBERS PER TOPN POOL (5G NR & LTE)


► Number of members per TopN pool based on dynamic range of scanner

► LTE live network // isolated single 5G NR site with 3 cells

5G NR

LTE #1

LTE #2

LTE #3

Rohde & Schwarz Dec 2019 5G optimization and benchmarking33

5G NR – SINR VS RSRP (SCANNER MEASUREMENTS)

► SINR vs RSRP similar to LTE

► Gap due to SINR degradation?

Rohde & Schwarz

5G NR – SINR VS RSRP (SCANNER & UE MEASUREMENTS)


► Max SS-SINR depends on implementation

Scanner: 30 dB

Samsung S10 (EU): 25 dB

► Very good correlation between scanner and

UE measurements

► Scanner much more sensitive compared to UE

► Min SS-RSRP // SS-SINR

Scanner: -140 dBm // -15 dB

UE: -125 dBm // -5 dB

Rohde & Schwarz

DRILL DOWN INTO SPECIFIC CELLS (PCI)


► Drill down into a specific PCI offers the view on anomalies in the SINR vs RSRP plot

► E.g. degraded SINR values (blue circles) identified to same PCI (green)

► E.g. better SINR values (red circle) identified for PCI (blue)

Rohde & Schwarz

DRILL DOWN INTO INDIVIDUAL SSB INDEX PER CELL (PCI)


► Further drill down to SSB indices (color coded) of a specific PCI offers additional insight

about anomalies in the SINR vs RSRP plot

► In this example no obvious anomalies visible

Rohde & Schwarz

CONCLUSION


Rohde & Schwarz MNT is committed to support the industry with network test

solutions from early trial phase to network optimization and benchmarking

Rohde & Schwarz MNT implemented ETSI TR methodology in test solutions

Network Performance Score connects Benchmarking with Optimization

ETSI STQ ratified and published TR 103 559 in August 2019

“Best practices for robust network QoS benchmark testing and scoring”

SINR vs RSRP correlation plots are a powerful instrument to investigate interference

or network load caused performance degradations Drill down possible

Quality Benchmarking important for operator success

https://www.rohde-schwarz.com/MNT-5G

https://blog.mobile-network-testing.com/

Good correlation between network scanner and mobile measurements

Both devices are needed for network optimization tasks

https://www.rohde-schwarz.com/MNT-5G

https://blog.mobile-network-testing.com/

NETWORK PERFORMANCE SCORE:The fastest way from QoE centric

data to actionable insights

Questions? .

www.rohde-schwarz.com/nps .www.rohde-schwarz.com/smart .

http://www.rohde-schwarz.com/nps

http://www.rohde-schwarz.com/smart

Rohde & Schwarz

MNT SOLUTION PORTFOLIO


Data Analytics

Data Collection

Hardware options

5g benchmarking optimization and - mobile wireless testing€¦ · 5/12/2019 · subscription...

Documents