stressing a lte device with typical user behaviors

Stressing a LTE device with

Typical User Behaviors

April 16, 2013

Presented by:

Artun Kutchuk

Vice President, Business Development and Strategy, w2bi, Inc.

[email protected], +1.858.361.4870

© 2013 w2bi, inc.

Leading Provider of end-to-end automated device testing software solutions for LTE, UMTS/GSM, and CDMA handsets

to Tier 1 service providers and device manufacturers

Who is w2bi?

� Incorporated in 1997

� HQ in South Plainfield, NJ and R&D centers in NJ and India

� Sales presence in US, Canada, UK, Italy, Korea, Taiwan, and China

2013 North America LTE Forum 2

OEMs and ODMs

Mobile Operators OS and Chip ProvidersIndependent Labs

© 2013 w2bi, inc.

Smartphone Industry Challenges

� $6+ Billion/yr industry cost for device returns with No

Fault Found (NFF) or software Issue (67% of returns)*

� Cost of Quality (CoQ)�Impact on return rate �Impact

on margin

� Common Software Issues on returned devices:

• Device reset

• Application crashes

• Error messages

• Device lockup

• Launch failure

• Blank screen


Functionality

Application

OperatingSystem

Firmware

Key Reasons

*IWPC Device Return White Paper, Oct 2011

© 2013 w2bi, inc.

Mobile Technology Evolution & Diversity


Diverse Device Types

(HW/OS)

Diverse Services

Diverse Networks

Diverse Applications

This image cannot currently be displayed.



• 2012 - 70% mobile data traffic growth globally (885 peta bytes/month)*• 51% mobile video• 14% data on 4G (.9% of

connections = 60 million)• 18% of handsets =

smartphones

• 2017 - 45% data on 4G (10% of connections = 992 million)

*Cisco VNI: Global Mobile Data Traffic Forecast Update, 2012–2017, Feb 2013

© 2013 w2bi, inc.

Opportunities to assess impact on user QoE with cross layer variation

Current Test Methodologies

� App developers focused on software functionalities rather than

the device complexities or network impact

� Most of the apps are being tested on clean device with device

emulators or on static live networks

� Device testing for RF & Signalling conformance, data

throughput, SMS, core services, etc.

� Standards based testing focused on perceived user Quality of

Service (QoS) with no device variability

� Operator tests focused on their specific network and service

capabilities


R&D TestsLab

CertificationInteroperabi

lity TestField Test

Returned Device Test

Industry Certification

© 2013 w2bi, inc.

Impact of LTE Introduction on User Experience


� LTE advancements are heightening user expectations of

service quality and performance

� Device technology evolution is broadening

the types of mobile applications

� On the device cross layer

interactions are increasing

� What is the impact of small cells,

band combinations, VoLTE, Carrier

Aggregation, multi media collaboration, etc. under realistic

usage scenarios on device and user experience?

Opportunity to stress device and applications with realistic scenarios to discover software related issues early!

© 2013 w2bi, inc.

What is Device Stress Test?

� Real and practical user behavior scenarios executed

repetitively under different conditions:• Varying device memory and CPU loads, and battery levels

• Background and foreground device applications

• Live network (4G/3G, Wi-Fi)

• Emulated network

• Varying mobility and RF conditions, e.g. fading,

handoffs, carrier aggregation (CA), multiple bands

• Structured (i.e. predefined user behavior) or

unstructured (i.e. random)

� Capturing and verifying:• Failures (i.e. applications, device reset, etc.)

• Battery impact

• Network impact (i.e. data transfer, signaling)

• Quality of Experience (QoE) – responsiveness, buffering,

latency, throughput, jitter, etc.


© 2013 w2bi, inc.

Typical Stress Test Cycle

Choose Test Type

•Record a Test Plan

•Specific Sequence

•Randomly choose a sequence

Edit Test Sequence

•Add a setup script

•Edit specific test plan

•Edit order of plans

Add System Stress

•CPU/Memory Hogging

•Data Streaming, I/O

•RF, iRAT stress, etc.

Playback Setup

•Privacy sweep

•Capture existing configuration

•Load starting application and content configuration

Playback Execution

•Execute test plans

•Capture failure logs

•Capture KPIs

Playback Iteration

•Reset device to starting configuration

•Execute Playback

•Repeat X times•Repeat Y Devices

Re

co

rdP

lay

An

aly

ze

Run Basic Analytics

•MTBF/MTFF Reporting

•KPI Isolation

•CPU, Memory, Temp

•Batt. Level, Data Usage•RSSI, Processes

•KPI Comparison

Run Advanced Analytics

•Trending of failures

•By OS version

•By manufacturer

•By SKU•By hardware profile

•Isolation of specific failure trend

Proof of Correction

•Deep testing on isolated issue

•Work with OEM/ISV

•Verify isolation correction


© 2013 w2bi, inc.

Value of stress test during device lifecycle


Pre-Deployment Testing Post-Deployment Testing

R&DLab

CertificationPost-Market

ReturnsField

CertificationIn-Market

Triage

• Mobile Chipsets

• Mobile OS• Device

Integration• Mobile Apps

• Standards• Operator

Specific

• Interoperability• Operator

Specific• Core Services

& Applications

• Vertical Industry or Enterprise Specific

• Point-of-Sale Diagnostics

• On device self Diagnostics

• Device field data collection

• Repair CenterDiagnostics

Stress Test Emulated Emulated Live Live Emulated

Relative Duration

Network

© 2013 w2bi, inc.

What type of results should we expect?

� Rapid identification of device or application failures

under typical usage scenarios

� Impact of applications on the battery, CPU, memory

and network (e.g. data traffic, signalling, etc.)

� Identification of rogue applications on customer

commercial devices

� Comparison of device and network measurements

� Benchmarking of devices across: • Diverse networks and mobility conditions

• Varying device loads

• Different operating systems and versions

� Detailed device logs on failure conditions, or in

significant device or network changes


© 2013 w2bi, inc.

DUT automation

Live or Emulated

MHL / HDMI

stream

Typical Device Stress Test Setup

R&S VTEvideo

analysis

R&S CMW500LTE/HSPANetwork

under fadingconditions

w2bi QuikStress:DUT Application

Control,Benchmarking & Failure Analysis


EmulatorControl

© 2013 w2bi, inc.

Improve device quality and device return rate with NFF


� Adapt end user behavior scenarios in the overall smart device

test strategy

� Embrace the end-user device feedback into early part of

testing and certification process

� Take device application neutral testing approach to reduce

cycle time and integration

� Introduce stress tests at integration, field test, certification

and device logistics

phases

stressing a lte device with typical user behaviors

Technology

lte device

device stress test

device reset

device lifecycle

iwpc device

device emulators

device complexities

clean device