addressing mobile app testing challenges

© Utopia Solutions

Addressing Mobile App Testing Challenges

Lee Barnes, CTOUtopia Solutions

© Utopia Solutions

Objectives

Understand Mobile App Testing ChallengesUnderstand Mobile App Testing Challenges

Learn Ways to Adapt Learn Ways to Adapt

Discuss What the Future May BringDiscuss What the Future May Bring

© Utopia Solutions

What’s Different About the Mobile Environment?

© Utopia Solutions4

Multiple Platforms

AndroidApple iOSBlackberryWindowsSymbian

Source: comScore Reports August 2012 Smartphone Platform Market Share

Smartphone Platform Market Share


Multiple Devices


Multiple Carriers


Mobile TestingIceberg

App Functionality

Mobile Environment

Diversity Increases Test Burden


How Do We Adapt?

Analyze

And

Prioritize!


Identifying Target Test Configurations - Analyze This!

BusinessObjectivesBusinessObjectives

• Application value to the business

• Risk of application failure• User demographics

AppFunctionality &Technology

AppFunctionality &Technology

• Deployment platforms & versions

• App interaction with device

• Integrated systems

User ProfileUser Profile

• Prevalent devices• Browser preference• Concurrent usage / load

profile

• Target test configurations

• Mobile specific test conditions

• Test prioritization• Performance test

objectives


Fragmentation – Android & iOS

Version Distribution (%)1.5 0.11.6 0.42.1 3.42.2 12.9

2.3 - 2.3.2 0.32.3.3 - 2.3.7 55.5

3.1 0.43.2 1.5

4.0.3 - 4.0.4 23.74.1 1.8

Version Distribution (%)4.0 0.14.1 0.34.2 1.44.3 4.15.0 3.25.1 28.86.0 62.1

Based on Android devices accessing Google Play in the 14-day period prior to Oct 1, 2012.

Based on iOS devices using Audiobooks app as of Oct 29, 2012

80%Coverage


Platform

Supported Platforms

User Demographics

OS

Supported OS’s

User Demographic

Device

App Functionality

Device Integration

User Demographics

Test Configuration Selection Example – Native Apps


Prioritize Test Configurations

OS

Device

DeviceWeighting iOS 3 iOS4 Android

2.2Android

2.3Blackberry

5.0Blackberry

6.0

OS Weighting 7 9 8 6 5 4

iPhone 3GS 7 49 63 N/A N/A N/A N/A

iPhone 4 9 63 81 N/A N/A N/A N/A

HTC Thunderbolt 8 N/A N/A 64 48 N/A N/A

MotorolaAtrix 4G 6 N/A N/A 48 36 N/A N/A

Blackberry9700 6 N/A N/A N/A N/A 30 24

Primary Configuration Secondary Configuration


What Mobile Test Conditions Should We Consider?

InterruptInterrupt

• Incoming Call• Incoming SMS• Low battery warning• Alarm alert

NetworkNetwork

• Carrier network(s)• Varying network

speeds• Wi-Fi network

DeviceDevice

• Screen orientation• GPS• Camera• Battery drain

• Power off• Battery discharge

• Accelerometer input• Keyboard slide• Handset key mappings• Screen type/size

• Network loss• Network transition


Mobile Test Case Selection

Type of App

App Usage

App / Device Integration


Mobile Test Case Matrix


How Do We Work?

Factor Traditional App Testing

Test Platform Desktop / notebook

User Input Keyboard / mouse

Collaboration Screen sharing

Results Verification Screen capture / movie

Testing Utilities? Yes

Automated Execution? Yes

Performance Testing? Yes

System Monitoring? Yes

Factor Traditional App Testing Mobile App Testing

Test Platform Desktop / notebook Mobile device

User Input Keyboard / mouse Touch screen / Roller Ball

Collaboration Screen sharing ?

Results Verification Screen capture / movie ?

Testing Utilities? Yes ?

Automated Execution? Yes ?

Performance Testing? Yes ?

System Monitoring? Yes ?


Manual Test Execution – Device Interaction

Traditional AppsTraditional Apps Mobile AppsMobile Apps


Collaboration & Results Verification

Traditional AppsTraditional Apps


Mobile Manual Testing – Device Interaction

MobileCloudMobileCloud

Remote Device

Control*

Remote Device

Control*

* Currently requires devices to bejail broke or rooted unless accessingvia development console


Mobile Testing Utilities

Testing Need Utilities

Location Based Services

• GPS / location spoofing apps• Full scale hardware/software solutions

Varying NetworkConditions

• Network condition emulators• Bandwidth throttling

Mobile Web Sites• Mobile browser simulators• Markup checkers• Client- and server-side debuggers


What Do We Need to Know About Mobile Technology?

Platforms Android iOS Blackberry Windows WebOS

Devices Various Apple Blackberry Various HP / Palm

Dev. Language Java Objective C Java C# / C++ / VB HTML / CSS / JavaScript

Source Model Open Closed Closed Closed Closed

Multi-tasking Yes Limited(iOS 4+) Yes Limited Yes

Standard Browser WebkitBrowser Safari Blackberry

BrowserVaries with

Device / CarrierWebOSBrowser

Networks GSM CDMA LTE HSPA+ WiMAX

Carrier(s) AT&T, T-Mobile

Verizon, Sprint Verizon AT&T,

T-Mobile Sprint

Standard 3G 3G 4G 4G 4G

Typical Avg Speed 1769 / 739 kbps

848 / 506 kbps

6.44 / 5.0mbps

2.48 / 1.05 mbps

2.15 / .081 mbps

Simultaneous Voice & Data Yes No Yes Yes Yes


Understand Common Mobile Issues

InterruptResponseInterruptResponse

• Inappropriate response after interrupt condition• Lack of graceful shut down / recovery• Erratic behavior when “back grounded” for

extended periods

PerformancePerformance• Developing beyond technology limits• Ignoring back end system load

UsabilityUsability• Inconsistent user interactions• Confusing icons / user interface objects


Test Automation Success Criteria

ReliableReliable• Issue detection and recovery• Accurate verification• Unattended execution

MaintainableMaintainable• Minimum sensitivity to application and

test case changes• Test cases separate from automation

code

ScalableScalable• Test coverage expanded efficiently• Automated test cases created by non-

technical resources


WIFI / Carrier NetworksWIFI / Carrier Networks

Mobile Test Automation – Ideal Scenario

AutomationFrameworkAutomationFramework

Test CasesTest Cases

Backend SystemBackend System

Goal: Single set of test cases executing across a diverse set of real devices

Reality: This is a rapidly evolving space with many providers and approaches.

Research carefully and look beyond the demo!


Mobile Test Automation Tool Categories

Native Platform Frameworks

Multi-Platform ToolsVisual Based Object Based

• Provided by mobile platform vendor as part of SDK

• Typically interact with the application at the UI object level

• Interacts with devices using visual methods (text and image recognition)

• Interacts directly with application UI objects using native methods and properties

• Object based interaction with app UI controls

• Greatest degree of support for native UI objects

• Control of device settings

• Supports multiple platforms

• Ability to execute testsacross multiple devices

• Interact with entire device

• Supports multiple platforms• Ability to execute tests

across multiple platforms• Interacts with apps at the

object level for greater reliability

• Limited to a single platform• Requires code level access

to the app• May be limited to simulator

(e.g. Blackberry)

• Generally less reliable• No access to object

properties• May require jail-breaking /

rooting

• Interaction is limited to appunder test

• Requires app to be instrumented

• Android UI Automator• iOS UI Automation

• SeeTest (Experitest)• eggPlant (TestPlant)• Perfecto Mobile• Zap-Fix (Zap Tech.)• Trust (Mobile Labs)

• TouchTest (SOASTA)• M-eux Test (Jamo Solutions)

Des

crip

tion

Pros

Con

sEx

ampl

e


3D Screens &Gestures

3D Screens &Gestures

“Serious” Apps That Control Hardware / Gather Information

in Dangerous and Critical Situations

“Serious” Apps That Control Hardware / Gather Information

in Dangerous and Critical Situations

Future of Mobile Testing


Key Points

• Gain foundational mobile knowledge • Understand unique mobile testing challenges• Analyze and prioritize based on app context• Use available tools and utilities to adapt• Mobile test automation is currently immature, but evolving

rapidly• Technology is moving incredibly fast – keep pace with

what matters to you


Questions…

…. and Answers!

Direct future questions to:Lee BarnesFounder and CTOUtopia Solutions, Inc.

Email: [email protected]: twitter.com/USI_LeeBarnesLinkedIn: linkedin.com/in/leebarnesBlog: www.RaiseYourMQ.com