Download - Addressing Mobile App Testing Challenges
© Utopia Solutions
Addressing Mobile App Testing Challenges
Lee Barnes, CTOUtopia Solutions
© Utopia Solutions
Objectives
Understand Mobile App Testing ChallengesUnderstand Mobile App Testing Challenges
Learn Ways to Adapt Learn Ways to Adapt
Discuss What the Future May BringDiscuss What the Future May Bring
© Utopia Solutions
What’s Different About the Mobile Environment?
© Utopia Solutions4
Multiple Platforms
AndroidApple iOSBlackberryWindowsSymbian
Source: comScore Reports August 2012 Smartphone Platform Market Share
Smartphone Platform Market Share
© Utopia Solutions5
Multiple Devices
© Utopia Solutions6
Multiple Carriers
© Utopia Solutions7
Mobile TestingIceberg
App Functionality
Mobile Environment
Diversity Increases Test Burden
© Utopia Solutions8
How Do We Adapt?
Analyze
And
Prioritize!
© Utopia Solutions9
Identifying Target Test Configurations - Analyze This!
BusinessObjectivesBusinessObjectives
• Application value to the business
• Risk of application failure• User demographics
AppFunctionality &Technology
AppFunctionality &Technology
• Deployment platforms & versions
• App interaction with device
• Integrated systems
User ProfileUser Profile
• Prevalent devices• Browser preference• Concurrent usage / load
profile
• Target test configurations
• Mobile specific test conditions
• Test prioritization• Performance test
objectives
© Utopia Solutions10
Fragmentation – Android & iOS
Version Distribution (%)1.5 0.11.6 0.42.1 3.42.2 12.9
2.3 - 2.3.2 0.32.3.3 - 2.3.7 55.5
3.1 0.43.2 1.5
4.0.3 - 4.0.4 23.74.1 1.8
Version Distribution (%)4.0 0.14.1 0.34.2 1.44.3 4.15.0 3.25.1 28.86.0 62.1
Based on Android devices accessing Google Play in the 14-day period prior to Oct 1, 2012.
Based on iOS devices using Audiobooks app as of Oct 29, 2012
80%Coverage
© Utopia Solutions11
Platform
Supported Platforms
User Demographics
OS
Supported OS’s
User Demographic
Device
App Functionality
Device Integration
User Demographics
Test Configuration Selection Example – Native Apps
© Utopia Solutions12
Prioritize Test Configurations
OS
Device
DeviceWeighting iOS 3 iOS4 Android
2.2Android
2.3Blackberry
5.0Blackberry
6.0
OS Weighting 7 9 8 6 5 4
iPhone 3GS 7 49 63 N/A N/A N/A N/A
iPhone 4 9 63 81 N/A N/A N/A N/A
HTC Thunderbolt 8 N/A N/A 64 48 N/A N/A
MotorolaAtrix 4G 6 N/A N/A 48 36 N/A N/A
Blackberry9700 6 N/A N/A N/A N/A 30 24
Primary Configuration Secondary Configuration
© Utopia Solutions13
What Mobile Test Conditions Should We Consider?
InterruptInterrupt
• Incoming Call• Incoming SMS• Low battery warning• Alarm alert
NetworkNetwork
• Carrier network(s)• Varying network
speeds• Wi-Fi network
DeviceDevice
• Screen orientation• GPS• Camera• Battery drain
• Power off• Battery discharge
• Accelerometer input• Keyboard slide• Handset key mappings• Screen type/size
• Network loss• Network transition
© Utopia Solutions14
Mobile Test Case Selection
Type of App
App Usage
App / Device Integration
© Utopia Solutions15
Mobile Test Case Matrix
© Utopia Solutions16
How Do We Work?
Factor Traditional App Testing
Test Platform Desktop / notebook
User Input Keyboard / mouse
Collaboration Screen sharing
Results Verification Screen capture / movie
Testing Utilities? Yes
Automated Execution? Yes
Performance Testing? Yes
System Monitoring? Yes
Factor Traditional App Testing Mobile App Testing
Test Platform Desktop / notebook Mobile device
User Input Keyboard / mouse Touch screen / Roller Ball
Collaboration Screen sharing ?
Results Verification Screen capture / movie ?
Testing Utilities? Yes ?
Automated Execution? Yes ?
Performance Testing? Yes ?
System Monitoring? Yes ?
© Utopia Solutions17
Manual Test Execution – Device Interaction
Traditional AppsTraditional Apps Mobile AppsMobile Apps
© Utopia Solutions18
Collaboration & Results Verification
Traditional AppsTraditional Apps
© Utopia Solutions19
Mobile Manual Testing – Device Interaction
MobileCloudMobileCloud
Remote Device
Control*
Remote Device
Control*
* Currently requires devices to bejail broke or rooted unless accessingvia development console
© Utopia Solutions20
Mobile Testing Utilities
Testing Need Utilities
Location Based Services
• GPS / location spoofing apps• Full scale hardware/software solutions
Varying NetworkConditions
• Network condition emulators• Bandwidth throttling
Mobile Web Sites• Mobile browser simulators• Markup checkers• Client- and server-side debuggers
© Utopia Solutions21
What Do We Need to Know About Mobile Technology?
Platforms Android iOS Blackberry Windows WebOS
Devices Various Apple Blackberry Various HP / Palm
Dev. Language Java Objective C Java C# / C++ / VB HTML / CSS / JavaScript
Source Model Open Closed Closed Closed Closed
Multi-tasking Yes Limited(iOS 4+) Yes Limited Yes
Standard Browser WebkitBrowser Safari Blackberry
BrowserVaries with
Device / CarrierWebOSBrowser
Networks GSM CDMA LTE HSPA+ WiMAX
Carrier(s) AT&T, T-Mobile
Verizon, Sprint Verizon AT&T,
T-Mobile Sprint
Standard 3G 3G 4G 4G 4G
Typical Avg Speed 1769 / 739 kbps
848 / 506 kbps
6.44 / 5.0mbps
2.48 / 1.05 mbps
2.15 / .081 mbps
Simultaneous Voice & Data Yes No Yes Yes Yes
© Utopia Solutions22
Understand Common Mobile Issues
InterruptResponseInterruptResponse
• Inappropriate response after interrupt condition• Lack of graceful shut down / recovery• Erratic behavior when “back grounded” for
extended periods
PerformancePerformance• Developing beyond technology limits• Ignoring back end system load
UsabilityUsability• Inconsistent user interactions• Confusing icons / user interface objects
© Utopia Solutions23
Test Automation Success Criteria
ReliableReliable• Issue detection and recovery• Accurate verification• Unattended execution
MaintainableMaintainable• Minimum sensitivity to application and
test case changes• Test cases separate from automation
code
ScalableScalable• Test coverage expanded efficiently• Automated test cases created by non-
technical resources
© Utopia Solutions24
WIFI / Carrier NetworksWIFI / Carrier Networks
Mobile Test Automation – Ideal Scenario
AutomationFrameworkAutomationFramework
Test CasesTest Cases
Backend SystemBackend System
Goal: Single set of test cases executing across a diverse set of real devices
Reality: This is a rapidly evolving space with many providers and approaches.
Research carefully and look beyond the demo!
© Utopia Solutions25
Mobile Test Automation Tool Categories
Native Platform Frameworks
Multi-Platform ToolsVisual Based Object Based
• Provided by mobile platform vendor as part of SDK
• Typically interact with the application at the UI object level
• Interacts with devices using visual methods (text and image recognition)
• Interacts directly with application UI objects using native methods and properties
• Object based interaction with app UI controls
• Greatest degree of support for native UI objects
• Control of device settings
• Supports multiple platforms
• Ability to execute testsacross multiple devices
• Interact with entire device
• Supports multiple platforms• Ability to execute tests
across multiple platforms• Interacts with apps at the
object level for greater reliability
• Limited to a single platform• Requires code level access
to the app• May be limited to simulator
(e.g. Blackberry)
• Generally less reliable• No access to object
properties• May require jail-breaking /
rooting
• Interaction is limited to appunder test
• Requires app to be instrumented
• Android UI Automator• iOS UI Automation
• SeeTest (Experitest)• eggPlant (TestPlant)• Perfecto Mobile• Zap-Fix (Zap Tech.)• Trust (Mobile Labs)
• TouchTest (SOASTA)• M-eux Test (Jamo Solutions)
Des
crip
tion
Pros
Con
sEx
ampl
e
© Utopia Solutions26
3D Screens &Gestures
3D Screens &Gestures
“Serious” Apps That Control Hardware / Gather Information
in Dangerous and Critical Situations
“Serious” Apps That Control Hardware / Gather Information
in Dangerous and Critical Situations
Future of Mobile Testing
© Utopia Solutions27
Key Points
• Gain foundational mobile knowledge • Understand unique mobile testing challenges• Analyze and prioritize based on app context• Use available tools and utilities to adapt• Mobile test automation is currently immature, but evolving
rapidly• Technology is moving incredibly fast – keep pace with
what matters to you
© Utopia Solutions28
Questions…
…. and Answers!
Direct future questions to:Lee BarnesFounder and CTOUtopia Solutions, Inc.
Email: [email protected]: twitter.com/USI_LeeBarnesLinkedIn: linkedin.com/in/leebarnesBlog: www.RaiseYourMQ.com