transforming mobile handset testing

8/8/2019 Transforming Mobile Handset Testing

1/12

Transforming MobileHandset Testing

How Accentures new ROSA Testing Framework deliversHigh Performance by reducing costs, enhancing qualityand protecting brand reputations.


2/12

2

Whats driving these challenges? With

mobile technology maturing and openingup to third-party developers, handset

platforms are incorporating a new

generation of applications. No longer

simply a talk and text device, this

intelligent companion is now enabling

fully-fledged entertainment capabilities,

financial services and enterprise

mobility.

Given the growing list of specifications

associated with these mobile devices,

the demands on product testing are

rapidly increasing as well. To achieve

high performance, manufacturers,

carriers and other stakeholders

are challenged to produce high-

quality products that meet the rising

expectations of their customers without

missing critical timelines or allowing

testing costs to explode. Accenture

research confirms this trend as

application testing costs are likely to rise

from 2 percent of overall testing costs

to as much as 30 percent under present

circumstances.

To smartly and strategically address

the challenges around cost, qualityand reputation, companies will need

to explore new ways of managing and

executing product testing. Existing

testing approaches tend to be

fragmented, costly and labor intensive.

Recognizing these challenges, Accenture

has developed a comprehensive new

testing framework and approach, which

is known as ROSA*. This holistic and

systematic strategy is designed to help

product development and validation

organizations cut overall testing costs

by 30 to 50 percent and improve time to

market by as much as 20 percent while

increasing the volume of release testing

by as much as 25 percent.

Such gains will soon be critical to

success. If companies are to generate

profitable and sustained growth in

todays hypercompetitive mobile arena,

theyll need to ensure their products

cost-effectively meet the increasing

rigors and expectations of thisdemanding global market.

Market Drivers: ThreeKey Trends in MobileCommunications

The communications world is rapidly

beginning to embrace fourth generation

(4G) technologies, while opening up

new mobile device innovations through

open source applications and platform

foundations such as Symbian and

Android. This trend is expected to

offer users a wider range of advanced

services while achieving greater network

capacity. What are the market drivers

encouraging handset manufacturers and

network operators to rethink how they

are approaching product testing?

Industry challenges in testing todays

devices are linked to three key trends:

unprecedented application growth,

global product launches and the

uncompromising demands of customers.

Unprecedented Application Growth.

Convergence leads to applications

that can no longer be tested in asimple and easily controlled network

environment. Applications, for instance,

Mobile handset manufacturers and network operators are under severe pressure

to rethink their approaches to product testing. As mobile communications

technology becomes increasingly sophisticated and complex, current testing

strategies leave them vulnerable to spiraling costs, diminishing product quality

and threats to their brand reputations.

*Testing by Remote access, Offshore testing, Simulatednetwork testing, Automation


3/12

3

are now enabled by connectivity to

financial services, content providers,points of sale and location based

services. They also impose new security

requirements. All these capabilities and

specifications add complexity to the test

environmentraising the bar associated

with assuring and validating necessary

performance levels.

The application could be a gaming

device using Java and 3D hardware

accelerators. It could be a health service

that integrates content from various

sources, provides useful tips on wellness,

analyzes the users health profile and

measures key health indicators via

Bluetooth. It could be a Web 2.0 service

enabling social networking and access

to enterprise-based applications. While

the possibilities are staggering, the

challenge of cost-effectively testing

these proliferating applications to ensure

they perform to spec are eclipsing

the existing test capabilities of most

companies.

Global Product Launches.

Whereas manufacturers may have had

the luxury of introducing new mobile

handsets region by region in the past,

they are now expected to launch them

worldwidemandating a global launchstrategy. Considering that managing the

economics and efficiencies of a global

product launch is very different from

what has been experienced in the past,

most manufacturers and operators view

this as a new challenge.

What does it mean for manufacturers?

Network configurations and behavior

are not necessarily similar across

geographies. Local operators launch

services that are relevant and unique to

the local market. This not only includes

language support features but also

services that are used by operators as

differentiators. They may not be able

to identify many performance failures

until after the launch has occurred and

the product hits the market. What does

it mean for operators? Their reputations

are on the line as they roll out services

that are expected to meet certain

performance levels.

Unfortunately, manufacturers andoperators will have far less time and

opportunity to test their complex

offeringsregion by regionin an era of

global product launches. If they remain

wedded to existing testing approaches,

their vulnerabilities will quickly becomeevident to their customers.

The Uncompromising Demands

of Customers.

Unsurprisingly, customer expectations

remain exceedingly high in terms of

their experience with these products.

Indeed, user experience has become

a key driver in the success of product

launches. As underlying technologies

and platforms are commoditized,

manufacturers begin to focus on user

experience and user interfaces to

differentiate their handsets.

The integration of user interfaces on

multiple platforms further increases

testing challenges. As handsets grow to

support business-to-consumer, business-

to-business and business-to-employee

applications, users expect performance

to match what theyve experienced with

their laptops and personal computers.

They expect typical applications like

Skype, YouTube, Adobe Flash to beavailable at similar performance levels

on their devices. They also want the

ability to download media content

and perform actions on content such


4/12

4

environment. However, their specific

testing challenges differ slightly.

Testing Challenges Facing

Manufacturers:

Handset manufactures spend millions

of dollars testing a device from

development to product launch. This

testing comes in two phases:

1. Primary testing (or hygiene

testing) involves taking a handset

through various certifications that

are typically performed or provided

by accredited test houses. This

includes tests related to Global

Certification Forum (GCF) and

Personal Communication Service Type

Certification Review Board (PTCRB),

Federal Communication Commission

(FCC) marking, electromagnetic

compatibility testing (EMC), RadioFrequency (RF) Performance,

Bluetooth and WiFi, and Microwave

Communications. While primary

testing is demanding, the cost, quality

and schedules for such testing are

relatively predictable. They bring in

low to moderate challenges with

growing device complexity.

as editing, sorting and sending. And,

finally, they expect their handhelds to

have an intuitive design that is highly

user friendly. Clearly, performance

expectations are immenseand growing.

As convergence, innovation and market

pressures continue to drive new device

specifications and higher customerexpectations, testing strategies must

inevitably be transformed.

In previous years, testing for mobile

communications devices was far more

deterministic. What needed to be

tested was a lot more well defined.

The complexity of testing was most

often attributed to communication

technologies that essentially were

standards driven.

In the sweep of a decade, the industry

has shifted from voice quality as a

differentiator to an era defined by

the introduction of exotic third-party

applications and content. In this new

environment, application testing is

becoming an enormous challenge and,

for some, a barrier to continued growth.

As open-source platforms invite a wider

set of application developers into the

market, standardization has become

extremely difficult. Device testing is no

longer a deterministic endeavor. Indeed,

application testing costs grew from 3

percent of overall testing costs in 2004

to ~20 percent of overall costs in 2008.

This number is expected to increase over

the next two years to more than 30

percent of overall costs.

Smart and strategic testing is

now essential if companies are to

reduce these costs, deliver expected

quality levels and defend their

brand reputations in an unforgiving

market. These are the issues that both

manufacturers and network operators

must now confront.

Business Challenges:What Manufacturers andOperators Now Face

Handset manufacturers and network

operators both face significant testing

hurdles in todays fast-moving market


5/12


6/12

6

Consider the value of the following four

elements in the ROSA framework:

Remote Testing.

Through this method, devices are tested

within a particular geography under a

communications carrier. However, tests

are conducted from a remote location

by accessing the keys of the device

over the Internet and monitoring the

responses on the device screen of the

testers computer. The advantages of

this approach are multifold:

Atestercantestalargenumberofdevice features in multiple locationssimultaneously without traveling tothe location. Virtual teams across theAccenture Global Delivery Networktest devices and report results,helping to reduce costs significantly.

Scriptsarecreatedeasilybyplaybackof a user scenario that is performedonce on the device. Quality andconsistency of testing can beconfirmed as a result.

Testexecutioncanbeautomatedby

a single command that runs a testlibrary at a predetermined time andlogs the results in a log file. Comparedto manual testing, the volume of teststhat can be executed in a given timeis far larger.

Withminimumeffort,testscriptscan

be ported from one device to another,which helps confirm overall advantageover time.

Offshore Testing.

A large part of a handset test

specification does not require a live

network. These are network independent

tests that can be run on the device

manually. This type of test can be moved

to locations where tester availability and

costs are low.

However, offshore solutions can also

be defined in terms of having access tofield testers within any given geographic

area. Service providers with a global

network of talent and testing experience

may have the ability to conduct tests in

various geographical locations, sparing

the manufacturer or carrier the costs of

flying experts to the field location and

setting up relevant field tests.

Simulated Network Testing.

This method of testing is used for

new feature testing when network

infrastructure is not yet ready fornetwork tests. Using a simulated

network, features can be tested. The

method provides a network condition

under which a reasonable number

of defects can be identified. The

advantages of such testing include:

Easydebuggingenvironmentfortheproduct development team to localize,analyze and fix defects as well asverify the defect resolution.

Mechanismtoreproduceadefectobserved in the field that canotherwise not be reproduced in thereal network environment.

Softwarestabilityensuredbybuildinga test library and continuouslyenhancing it with test cases that

verify defect resolutions.Automation.

It is essential to fully leverage

automation when designing a test

framework for handset testing.

Whatever the means of testing,

automation dramatically increases the

quality of testing results. It enables

testers to run thousands of test

cases unmanned. It enables testers to

execute a test library, allowing them

to conclude that a defect resolution

has taken care of all possible scenarios

in which a defect can potentially

occur. Such results are impossible to

achieve through manual testing. Testing

automation, in other words, is a key

factor in confirming product stability.

Figure 1: ROSA Framework Drives Significant Cost Reduction

15-25%Down

35-45%Down

40-50%Down

PresentCost

FirstYear

SecondYear

ThirdYear

Graduate Cost Reduction from 15-50%over 3 year period

Estimated Quality andProductivity Benefits

Volume Time

Test Cases

Estimated Cost Reduction


7/12

77


8/12

8

Process Transformation

In a world of proliferating features,

there is no sensible and cost-effective

way to manage the testing challenge

using conventional, feature-driven

testing approaches. Testing teams

would merely grow in a linear fashion,

adding significant costs while slowingdown the testing operation.

To confirm success with the ROSA

approach, companies must begin to

organize around skill sets and test

categories instead of product features.

In order to redesign the testing process

(see figure 2), it is first necessary to

classify a handset test specification into

the following categories:

Test from Remote (TfR): All tests that

can be performed by remote access.Feature or functionality tests can all betested by remote access while mobilitytests and tests that verify performancequality may not be possible by remote.

No Network Required (NoN/W): Alltests that can be performed standalonewithout needing to connect to anetwork. Most tests for the phonebook,

media player and similar features fallunder this category.

Test by Simulation (TbS): All teststhat can be performed in a simulatedenvironment. Most features that can betested over an ideal network conditionand do not require complex interfaces tovarious services fall under this category.

Tester on Field (ToF): All tests thatmust be performed in the field. Thisassumes the tests must be performed bya tester physically operating the devicein a real network condition.

All test cases and test plans can be

coordinated using these test categories

or methodsan option that radically

simplifies testing approaches for

manufacturers and operators.

Organizing for ChangeTypically, product-testing teams are

organized by function or feature,

working across multiple handset models.

The ongoing addition of features and

functions merely increases the number

of teams and resources necessary to

meet certain performance benchmarks.

Companies, however, can enable a

significant increase in efficiencies

and cost optimization by reorganizing

teams based on the ROSA framework.

By organizing around ROSA testing

categories, they gain the ability to

group all testing activities within testing

categories (see figure 3).

Reorganizing teams in this fashion helps

drive testing productivity by enabling

the focused training and professional

development of teams. For instance:

When no network is required,standalone test teams requirevery little knowledge of deviceinternals. Training can be focusedon relevant features, which enablesthe standardization of techniquesfor coverage, failure and concurrentbehavior.

Remote testing requires training oncapabilities in scripting, automation,defect identification and reporting.Under this approach, tool licensingcan be optimized and knowledge ofmultiple testing tools can be impartedefficiently, which may require higher-degree technical skills as a tester thatcan be developed in a conventionalmanner.

Figure 2: ROSA Framework Drives Savings in Key Testing Categories

TfRPossible to access and test device fromRemote location(Test from Remote)

NoN/WStandalone Testing

(No Network Required)

TbSBest Tested in controlled/Simulatedenvironment(Test by Simulation)

ToFTesting on Field inevitable(Test on Field)

20%

35%

20%

25%

Device Feature NoN/W ToF TfR TbSActive Idle Screen 20% 20% 50% 10%Analog TV 90% 10%Applications 10% 30% 40% 20%Browser 10% 10% 30% 50%Camera 50% 10% 10% 30%Bluetooth 30% 30% 30% 10%Content Download 10% 30% 40% 20%Data Synchronization 60% 10% 10% 20%Digital Rights Management 10% 20% 60% 10%Digital TV 10% 70% 20% 0%Dual Keyboard 40% 10% 10% 40%Dual Screen 60% 5% 25% 10%Dual SIM 0% 30% 60% 10%Email related functions 10% 30% 60%

File Formats 70% 20% 10%Flash RSS 30% 30% 40%HTTP 10% 10% 10% 70%Language 25% 10% 25% 40%Local Connectivity 70% 10% 10% 10%Location Based Service 10% 20% 60% 10%Mail 20% 20% 60%Media Player 70% 10% 10% 10%Messaging 10% 20% 30% 40%Navigate condition 25% 10% 15% 50%News Casting 10% 60% 30% 0%Provisioning 20% 30% 50%Radio 90% 10%Ringtones 50%RSS Reader 30% 40% 30%Settings 90% 10%Screaming 10% 40% 50%Subscreen 50% 20% 30%

Test Editors 60% 10% 10%Utilities 30% 30% 20% 20%

Video Telephony 20% 10% 70%

Figures are representative of the categorization as an average of the various device test specifications across organizations that

Accenture has worked with.


9/12

9

Simulated network testing wouldrequire a strong understanding ofnetwork behavior as well as theability to use low-level scriptinglanguages, work with USB and otherconnectivities on the device, andautomate testing. Team membersmust have the experience and skills to

debug logs on the network side andreproduce user scenarios observed inthe field to enhance defect resolutionsignificantly.

Field testing will require skills innetwork maps, managing devicesand simulations. Testers mustreport defects in an intelligent andinformative way to developers at thehome office and reproduce typicalscenarios that are critical but notrepeatable. Such skills can be builtthrough a structured approach, andknowledge transfer can be managedefficiently through appropriatetraining and project management.

While these principles of organizationaland process redesign can be appliedwithin a product development andvalidation organization, manufacturersand network operators are increasinglylooking to outside specialists for severalkey reasons.

First, specialist service providers can

rationalize their investments in tools,

knowledge and experience across a wide

customer base. Second, their focused

commitment to such solutions enables

them to continually develop new skills

and experienceremaining on the

leading edge of testing solutions. And,

finally, they can offer global testing

coverage by remotely testing products

and by deploying personnel in regions

worldwide.

Accentures Assets andExperience

Accenture offers an example of the

types of capabilities and expertise that

must now be deployed to deliver high

performance in terms of handset testing.Test Libraries. Accenture has devicetest libraries with more than 6,000test scenarios based on experiencewith carrier test specifications, GCFfield trials guidelines and in-house testdevelopment.

Simulator Labs. The firms ThirdGeneration (3G) Network Simulatorenables testing in a controlled networkenvironment. Early detection ofdefects, feature testing and release testautomation are all handled throughthese labs.

Global Remote Test Labs. Accenturehosts remote test labs across its GlobalDelivery Network and client locations,which enables benchmarking and helpsto optimize device testing in terms oftime and cost.

Solution Enablers. Accenture has madean array of strategic investments inits partnerships and alliances, whichprovides know-how on commercialtest infrastructures and knowledge ofstandards and test specifications.

Such capabilitieswhich manufacturers

and operators should expect from theirtesting solution providersrepresent

the foundations of an advanced testing

infrastructure. By deploying and

leveraging such assets, companies can

cost effectively master the complexities

and challenges posed by the new

generations of mobile devices.

Mail

Browser

Device Management

Feature n

Handsets

Functions

A B C D

Handsets

Model 1

Model 2

Model 3

Model 4

Test Teams

NoN/W ToF TfR TbS

Figure 3: Organizing Around testing Catergories


10/12


11/12

11

So it goes with the companies confronted with testing todays mobile handsets.Expectations are greater than ever. Costs threaten to spiral out of control.

Quality is in danger of diminishing. And should you lose your grip, consumers

will turn on you like an angry tiger.


12/12

Copyright 2010 Accenture

All rights reserved.

Accenture, its logo, andHigh Performance Deliveredare trademarks of Accenture.

About AccentureAccenture is a global managementconsulting, technology servicesand outsourcing company, withmore than 176,000 people servingclients in more than 120 countries.Combining unparalleled experience,comprehensive capabilities acrossall industries and business functions,and extensive research on the worldsmost successful companies, Accenturecollaborates with clients to help thembecome high-performance businessesand governments. The companygenerated net revenues of US$21.58billion for the fiscal year ended Aug.31, 2009. Its home page iswww.accenture.com.

Contact us

For more information on achievinghigh performance through innovationandtestinggotowww.accenture.com/productsoftware or contact your localrepresentative about how Accenturecan deliver high performance throughproduct software and testing solutions.

Please contact:Risto ValtakariExecutive DirectorProduct Software SolutionsTel: +358 205725 508

Contributed byVenugopal RamakrishnanAccenture Product Software Solutions

transforming mobile handset testing

Documents