Mobile Test Strategy

Shankar GargSenior Consultant - Testing

• Scope of Mobile Testing

• Why Quality is important

• Challenges in Mobile Testing

• Best Practices for Mobile Test Strategy

• Mobile Testing Tools

• Conclusion

3/10/14

Scope of Mobile Testing is:

• Web Apps : Web applications opened on mobile browsers ( chrome, safari, opera etc.)

• Native Apps : application program that has been developed for use on a particular platform or device.( android, iOS etc.)

• Hybrid Apps : Native Apps + Web Apps

3/10/14

• Back in the day, one bad customer experience would spread via word-of-mouth and reach few Customers.

• Thanks to Social Media platforms like Twitter , Facebook etc. however, that same word-of-mouth now travels at the speed of light and reaches million users.

• The other major difference is App Store ratings. Imagine if a poor review were stamped directly on the packing of a product.

• These Social Media Ratings and Likes can prove to be a make or break for the brand and product.

3/10/14

Why Quality is important

CHALLENGES

• Hardware - Device Variation

• Software - OS Variation

• Network – Vendor and Bandwidth Variation

• Interrupts – Alerts and Notifications

• Waterfall Software Development Process

• Updates – Quicker Release Cycles

• Data Security

3/10/14

Hardware - Device Variation

Mobile application testing is difficult because of :

• Various Manufacturers ( Samsung, Apple etc. )

• Various Screen Sizes ( Phone and Tablet etc.)

• Keypad – (Virtual or hardware)

• Variations in RAM, CPU, OS utilization etc.

• Touch Screen Responsiveness.

• No Guarantee that application tested on one device will run successfully run on other device also.

3/10/14

3/10/14

Hardware Fragmentation in Mobile

https://www.comscore.com/Insights/Press_Releases/2014/4/comScore_Reports_February_2014_US_Smartphone_Subscriber_Market_Share

3/10/14

Screen Fragmentation in Android

Source: https://developer.android.com/about/dashboards/index.html?utm_source=ausdroid.net

3/10/14

Vendor Fragmentation in Android

http://itcompanyegypt.com/android-fragmentation-samsung-is-dominant-but-will-that-always-be-the-case-forbes/

Software - OS Variation

Mobile application testing is difficult because of :

• Various OS ( iOS, android, Windows etc. )

• Various Versions of OS ( iOS 4 to iOS 7.1, android 4.2.2 to android 4.4 etc.)

• OS like iOS does not allow the device to be down graded to lower iOS once upgraded to higher iOS.

3/10/14

3/10/14

OS Fragmentation in Android

Source: https://developer.android.com/about/dashboards/index.html?utm_source=ausdroid.net

3/10/14

OS Fragmentation in iOS

Source: http://chitika.com/insights/2013/ios-distribution-update

Network

• Network Carrier ( Airtel , Vodafone etc.)

• Bandwidth ( 2G, 3G, Broadband etc.)

• GSM, GPRS, CDMA and LTE?

• No Network, Airplane Mode

• Network Connectivity (low, high etc.)

• Geo-graphical location of the target Audience.

• Replicate diverse locations?

• Test GPS-based Operations?

3/10/14

3/10/14

Network Speed vs App function

Interrupts

Mobile Application behavior is affected by:

• Alerts and Notifications from other apps installed on the machine.

• Very difficult to simulate

• While accessing internet if a calls comes than Internet connection is interrupted. (2G)

3/10/14

Waterfall

Waterfall Software Process is

a big challenge for mobile development.

• By the time the design the app is freezed, new version of iOS is released.

• By the time development is finished, new line of Smartphones is released.

• In the examples mentioned above it would be very difficult for the app developed in waterfall model to handle these changes.

3/10/14

Updates

• The ease of upgrading apps over the air combined with increased user expectations about quicker releases (both for bug fixes and new features) result in frequent application releases.

• App are also updated frequently to handle multiple major and minor OS updates.

• Test teams are continuously tasked continuously tasked with testing entire applications faster and more often.

3/10/14

Data Security

• Mobile Apps have access to more personal Data as compared to Web Sites

• Users expect their personal data to remain personal.

• Users expect their apps to be 100% free of viruses, malware and other threats.

3/10/14

BEST PRACTICES

Best Practices for Test Strategy

• Select Target Devices

• Emulators vs. Real Devices

• Cloud Computing

• Network Options

• Mobile Testing Tool

• Test Automation

• Agile Testing

• Security

• App Store Guidelines and International Standards

3/10/14

Select Target Devices

Points to keep in mind:

• OS Version

• Android, iOS, windows etc.

• Which all versions are supported ( don’t forget the minimum version).

• Smartphone or tablet

• Display Density

• Low, high and High Definition

• Misc.

• CPU, RAM and Memory etc.

3/10/14

Emulators vs. Real Devices

3/10/14

Advantages Disadvantages

Developers can verify certain functionality that is not specific to any device, carrier or operating system.

Touch Responsiveness can not be tested on Emulators. Ex. Tap and Gestures etc.

Usage of Emulators is very Cost –effective in early stages of the development

Actual user behavior can not be tested with emulators. Ex. ( Nokia Advertisement :- person jumping off bridge with phone)

To familiarize the team with basic features of a phone.

Camera and Mic can not be tested with Emulators. Ex. Instagram.

Access to multiple phones at no cost at all.

Emulators and Real Devices

3/10/14

Emulator Emulator + Device

Device

1 Used in early development/Demos

Distribute features across device and Emulators

Used onlyfor network dependent test cases

2 Basic Functionality Testing

Network Connectivity

3 Can be used in automation testing

Stable UI Features

Cloud Computing

Cloud Computing is a web based environment where testers can deploy, test and automate their mobile applications on real devices or emulators via internet.

3/10/14

User provides the input to the Web Interface

via internet

Server receives the request and sends it to the

device/emulator

Mobile Devices resolves the user

request and sends the

response to server which in terms send the

response to user

Cloud Computing

3/10/14

Advantages Disadvantages

1 Per Hour Rate/ Usage Based

Internet Connectivity Issues

2 Device logs are available Network latency

3 Large number of devices are available

Latest Device availabilityIssues

4 Tests can be run in parallel on many devices

Over Crowding issues ( device waitlisted)

5 Build Integration Automation Test Failure

6 Incoming call and SMS facility also available

7 Automation test executionis stored as video

Cloud Computing - Examples

3/10/14

Network

• Points to keep in mind for selecting real network:

• Geo-graphical

• Economical

• Real Cellular Network Testing should be done on some of the selected devices and selected operators.

• User Experience - basements

3/10/14

Agile TestingAgile Testing Approach automatically

handles most of the problems faced in

Mobile Testing:

• Consider the example where a new iOS has been introduced, then changes for that iOS can be included as early as next sprint onwards.

• Agile allows the developers to change course on a moment’s notice.

3/10/14

Data Security

Make sure to include this in your testing scope:

• Does your mobile app store saves information properly?

• What about data deletion –or worse – unintended data deletion?

• App does not allow unauthorized access to personal information.

3/10/14

App Store GuidelinesApp Store Guidelines Instead of being the last mile in Testing, should be followed through the project life cycyle

• Using Apple’s graphics in your own apps

• Icons

• Don’t use private APIs

• Avoid copying core functionality of the OS’s pre-installed apps

• If you’re collecting user information, be sure to note it somewhere in your app

• No Offensive Language

3/10/14

User Reviews on App Stores

3/10/14

3/10/14

Mobile Testing Tools

mobiReady

Android Developer

Stat Counter

Instrumentation

Crash Reports

Responsive

BrowserStack

Console Logs

Appium

Robotium

ConclusionDespite of all the challenges mentioned if we follow these best practices then we can ensure a cost effective and Comprehensive mobile testing process.

• Follow App Store Guidelines and Standards

• Target Device Selection ( using a mix of real devices and emulators)

• Network Throttling

• Use of Cloud Computing Solutions

• Taking Care of Data Security

• Appropriate Testing and Automation Tool Selection

• Be Agile and Follow Agile

• Maximizing Automation wherever possible

• Applying traditional best practices and methods of Desktop Testing to mobile Testing

3/10/14

picture by -bast-

Q & A

References

• http://developer.android.com/about/dashboards/index.html

• https://code.google.com/p/robotium/ https://developer.apple.com/library/mac/samplecode/UnitTests/Introduction/Intro.html

• https://developer.apple.com/library/ios/documentation/IDEs/Conceptual/iOS_Simulator_Guide/TestingontheiOSSimulator/TestingontheiOSSimulator.html

3/10/14