Evolution of New Feature Verification in 3G

NetworksMichael Monoghan,

LM Ericsson Ltd,.Ireland

Europe’s Premier Software Testing Event

Stockholmsmässan, Sweden

WWW.EUROSTARCONFERENCES.COM

“Testing For Real, Testing For Now”

Evolution of new feature

verification in 3G Networks

Eurostar

Stockholm, December 2009

Michael Monaghan, LM Ericsson (Ireland) Ltd

Short overview of LM Ericsson

Founded 1876, Stockholm, Sweden

Presence in 175 countries

> 75,000 employees

~24,000 patents

> 40% of all mobile traffic passes through Ericsson networks

Ericsson supports networks which serve > 1 billion subscribers

Business Units– Networks

– Multimedia

– Global Services

– Sony Ericsson

In this presentation, we will visit the 3G Mobile Network Development area– 3G refers to 3rd Generation, a family of standards for wireless communication

– A glossary of terms appears at the end of this presentation

Feature Verification in 3G Networks

The scenario

– Very complex systems

– Very complex features

– Very difficult to recreate real world environment

– Many levels of test from block test to complex network test

– Many tools needed

The aim

– To verify that features behave according to requirements

– To make a quality statement about new features, so a decision can

be made about whether to release them

– To satisfy Time to Market and cost goals for the feature

Feature Verification in 3G Networks

Elementmanagement

OSS-RC

RNC

RBS

RXI

EM

Operations Support System

Radio & Core

Radio Network Controller

Radio Base Station

Radio Network Integrator

Element Manager

EM

WCDMA: Wideband Code Division Multiple Access, an air interface standard

HSPA EvolutionUplink

20-40 Mbps

12 Mbps

5.8 Mbps

1.4 Mbps

0.4 Mbps

2 ms TTI

16QAM

Downlink

3.6 Mbps

14 Mbps

21 Mbps 28 Mbps

42 Mbps

80-160 Mbps

15 codes

2x2 MIMO64QAM

Multi Carrier

Multi Carrier

64QAM + MIMO

Higher Modulation

Combinations

Multi Carrier

Refer to glossary for explanation of terms...

Major Advances in Verification of

3G Network Features

Evolution from Function Based to Feature Based

Verification

More generic development leads to verification savings

Evolution of Feature Verification

Looking to the Future

Summary

Advances in RNC Feature

Verification

2004 2005 2006 2007 2008 2009 2010

Generic

Design/Test

Strategy

FT to FV

MoveFunction

Test

Parallel

TestingFuture

FT: Function Test

FV: Feature Verification

Function Test

RNC verification was done from a functional

viewpoint

Several functional teams triggered the same test

case and then checked one part of the result

A test case most often consists of– Setting up a particular type of call (Control Plane)

– Sending data (User Plane)

– Moving the phone (Mobility)

Function

Test

2004 2005

Function TestFunction

Test

2004 2005

Feature1

Feature2

Feature3

Feature4

Feature5

Mobility Team Control Plane Team User Plane Team

Mobility

Mobility

Mobility

Mobility

Control Plane

Control Plane

Control Plane

Control Plane

User Plane

User Plane

User Plane

Function Test

Small area of expertise for each engineer

Little feature overview knowledge in verification

Led to rigid test organisations and verification

projects

Function

Test

2004 2005

Feature Based Verification

Verification based on Use Cases

Verify features end to end

Verify each Use Case in the most suitable

environment

Many challenges had to be overcome

FT to FV

Move

2006 2007

Feature Verification

Feature1

Feature2

Feature3

Feature4

Feature5

Mobility Team Control Plane Team User Plane Team

Mobility

Mobility

Mobility

Mobility

Control Plane

Control Plane

Control Plane

Control Plane

User Plane

User Plane

User Plane

FT to FV

Move

2006 2007

Feature Based Verification

Feature Verification has led to significant cost savings

Very responsive and flexible test organisation

An enabler for a feature based release strategy

FT to FV

Move

2006 2007

Feature Based Verification

Feature Verification, an evaluation:

Feature #TCs Cost per

Test Case

(hours)

Total

Cost

(hours)

#TCs Cost per

Test

Case

Total

Cost

(hours)

Feature1 168 9 1512 34 8-20 (Avg

14)

476

69%

saving!!

FT to FV

Move

2006 2007

Function Test Feature Verification

RAB (Radio Access Bearer)

Concept

Speech

SpeechData

Data

Speech

plus data

Speech

plus data

Generic Design/

Test

Strategy

2005 2006

Generic Design/Test Strategy

Market pressure for many radio connection combinations

– e.g. different interactive packet RAB rates and combinations

Development began to be based on generic code units which made it easier to introduce new RAB rates and combinations

In P6 the number of RAB combinations exploded : (13 in P3, 18 in P4, 39 in P5), > 70 in P6, Not possible to verify all transitions

CS Streaming

UDI (CS64)PS Streaming+Int 8/8

SP0

SP64

UDI+Int. 8/8

Int. 64/128

Int. 64/384

64/64

Int. FACH

Speech

SRB

Other int. RAB state

SRB

Int. 128/128

Int. 128/384

128/64

Int. 384/128

Int. 384/384

384/64

HS

SPHS

Generic Design/

Test

Strategy

2005 2006

Generic Test Strategy

Test strategy was to verify the generic code once

and then run a greatly reduced test scope on new

RABs/RAB Combinations going forward

Reduced scope testing based on RAB type

groups

2005 2006

Generic Design/

Test

Strategy

Generic Test Strategy Savings

Feature Pre-Generic

(including all

error and

exception

TCs)

Generic

Test

Principle 1

(Removal of

error and

exception

TC)

Generic

Test

Principle 2

(Rab type

groups)

Generic

Test

Principle 3

(Some

functions

moved

outside FV)

Generic Test

Principle 4

(Functional

based

changes

costs

removed)

Feature2 ~ 196 TC

~ Exe cost

1764 hours

96 TC (-100)

~ Exe cost

864 hours

88 TC (-8)

~ Exe cost

792 hours

58 TC (-30)

~ Exe cost

522 hours

43 TC (-15)

~ Exe cost

387 hours

79% saving!

2005 2006

Generic Design/

Test

Strategy

Generic Test Strategy Effectiveness

Feature #TCs run in RNC

Feature

Verification

Faults found

by RNC

Feature

Verification

Faults found

by IODT

(Mobile

phone

Verification)

Faults

found by

RAN I&V

Slipthrough

analysis to

RAN I&V

Feature3 102 34 8

Analysis:

Same faults

would have

been found in

RNC but

testing was

done in

parallel

2

Actual TRs

on feature

2

Analysis: 1

due to tools

issues, 1 due

to intro of

fault after test

case had

been run.

Generic test is very effective when applied correctly

2005 2006

Generic Design/

Test

Strategy

More Demanding Environment

2 releases per year

Market demand for many and more complex features (Mobile Broadband)

Continuous improvements in Time To Market required

Need for continuous improvements in operational excellence

Need to streamline verification further:– Introduce more parallel testing

– Minimize repetition in testing at all levels

– Maximise test effectiveness

Parallel

Testing

2007 2008

Parallel Testing

RNC node test work more closely with higher layer test

organisations

Form a RAN Level Joint Test team with participation

from network nodes (RNC, RBS) and RAN

Make Phased Deliveries for key features

Split scope of feature test between RAN

and nodes

Parallel

Testing

2007 2008

Node/Subsystem I&V Integration

Subsystem I&V is a very useful, low cost and under-utilised test environment

Was Best Effort/Happy Test

Integration of test activities means node level TCs are passed in a Subsystem I&V environment

Improves verification lead times at RNC node level

Reduces repetition between Subsystem and RNC test documentation and execution

Need to minimise quality risk to RNC and Subsystem

Node/

Subsystem

I&V

Merge

2008 2009

Maximise Test Effectiveness

Merge of RNC node feature verification with RAN feature verification– Pass RAN level test cases in a subsystem verification environment

– Remove RNC node verification as a layer of verification

Eurostar conference– Ericsson verification problems are industry-wide

– We compare favourably with other companies in many respects

– However there are some methodologies we should study more

Risk Based Test Case Selection– Each test case ranked by likelihood and impact

– Trial was inconclusive

Estimate number of remaining faults– Tools exist which predict number of latent faults in a feature

– Too much emphasis now on “slip-through”

Preventive Testing

Future

2009 2010

Summary

We have come a long way

Time pressure will continue to mount

Borders between layers of the development

process are being broken down

Summary

2004 2010

Contact Details

Michael Monaghan, LM Ericsson (Ireland) Ltd

Michael.A.Monaghan@ericsson.com

Glossary

EM - Element Manager– A simplified management interface for RNC and RBS nodes

FT - Function Test– a test strategy where new features are verified by verifying the functions which realise the

features one by one

FV - Feature Verification– a test strategy where new features are verified as a unit

HSPA - High Speed Packet Access– a collection of protocols which provide high bandwidth and improved spectral efficiency in 3G

networks

Interfaces – The communication channel and protocol between network nodes– Iu - interface between the RNC and the core network

– Iub - interface between the RNC and the Radio Base Station

– Iur - interface between 2 RNC nodes

– Mun – interface between other management systems and OSS-RC

– Mur - operator management system interface towards RNC

– Mut - operator management system interface towards RXI

– Mub - interface from operator management system towards RBS

– Uu - interface between the RBS and the Ue

Glossary (contd...) I&V - Integration and Verification

– a generic term in Ericsson to refer to all verification activities

MIMO - Multiple-input and multiple-output– use of multiple antennas at both the transmitter and receiver to improve communication

performance

Multi-Carrier– a method of allowing a mobile device listen to two HSPA data streams from different cells

simultaneously in downlink, thus facilitating higher bandwidth

OSS-RC – Operation Support System for Radio and Core– a network management node

P6 – Project or Phase 6– Ericsson RAN product releases are numbered P2...Pn

RAB - Radio Access Bearer– a UMTS service providing an application using the UMTS network with the ability to send and

receive data over the UMTS network with a specific Quality of Service (QoS).

RAN - Radio Access Network– part of a mobile telecommunications system which implements the radio access technology,

so the mobile equipment can communicate with the backbone networks

Glossary (contd...) RBS - Radio Base Station

– a 3G network node containing radio frequency transmitter(s) and receiver(s) used to communicate directly with the User Equipment

RNC - Radio Network Controller– a network node in a 3G network which controls the base stations which are connected to it.

It also carries out other functions including radio resource management and mobility management

RXI – An Access Network Aggregator– an ATM switch supporting handling of various network topologies and technologies

TC - Test Case– For the purposes of this presentation, a test case generally involves establish a specific type

of phone call (voice and/or data)

Ue - User Equipment– any device used directly by an end-user to communicate. It can be a hand-held telephone, a

card in a laptop computer, or any other device. It connects to the base station

WCDMA - Wideband Code Division Multiple Access– an air interface standard used in 3G networks

16/64 QAM -16/64 Quadrature amplitude modulation– methods of coding data for transmission over the air

Glossary (contd...)

2MS TTI - 2MS Transmission Time Interval

– Transmission Time Interval refers to the length of an independently decodable radio

transmission

3G - 3rd Generation

– a family of standards for wireless communications. Main characteristics are wide range of

services, high throughput and spectral efficiency