T10 Concurrent Class
10/3/2013 11:15:00 AM
"Mobile Testing Success: Real
World Strategies and
Techniques"
Presented by:
Clinton Sprauve
HP
Brought to you by:
340 Corporate Way, Suite 300, Orange Park, FL 32073
888-268-8770 ∙ 904-278-0524 ∙ [email protected] ∙ www.sqe.com
Clint Sprauve
Hewlett-Packard
Clint Sprauve is a senior solutions architect for HP Software. Previously, Clint was the director
of product strategy for the Borland Solutions Division of Micro Focus, where he created
Borland’s mobile strategy for functional test automation. He has more than twenty years of
experience in the software development and quality assurance industry. Clint has served as the
senior product marketing manager for the Silk Testing Solutions at Borland Software and Segue
Software, and served as a senior technical sales engineer for both companies.
9/19/2013
1
© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
Mobile TestingSuccessReal World Strategies & Techniques
Clint Sprauve
Solutions Architect & Software Innovation
Coach
© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
What You’ll Learn Today
Current State of the Mobile Industry
Mobile Opportunities and Challenges
Mobile Performance
Mobile Success – Case Studies
9/19/2013
2
© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.3
Android – Device Fragmentation
According to Google’s own numbers, only 37.9
percent of Android devices are running Jelly Bean,
the latest version of Android. That number is only
slightly higher than that of Android 2.3
Gingerbread, which still commands a 34.1 percent
share in spite of being released in 2010.
source: http://venturebeat.com/2013/07/30/android-fragmention-2013/
© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.4
Android – Device Fragmentation
source: http://venturebeat.com/2013/07/30/android-fragmention-2013/
“The availability of cheap Android phones (rarely
running the most recent version) means that
[Android phones] have a much greater global
reach than iOS, so app developers have a wider
audience to build for…”
9/19/2013
3
© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.5
Why Fragment the OS?
source: http://venturebeat.com/2013/07/30/android-fragmention-2013/
• At it’s core, it’s about choice
• Smartphone manufacturers can
sell a variety of phones
• Different configurations
• Different prices
• Different consumer market
segments
© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.6
Android vs. iOS
9/19/2013
4
© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.7
Mobile creates business disruptions
Location-aware, frictionless-
payment taxis & private cars
Accelerated check deposit
processing from image
Check in, change and monitor
flights
© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.8
Business challenges
The competition is going mobile
Accelerated time to market
Improving customer interaction
Transforming and streamlining processes
New sales channels
9/19/2013
5
© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.9
IT challenges
Chaos and change
New/unknown technology
Accelerate delivery
Complexity
Performance
Risk
Mobile drives IT to face a new reality
© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.10
Mobile success and failure is very visible
Very poor app
ranking in Google
play
Competitors have
4 star apps
Where is your app?
April 2012
9/19/2013
6
© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.11
Could better mobile testing have prevented this?
A very public launch
With very public
quality issues led to …
public apologies and
executive turn over
© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.12
Why is mobile testing so difficult?
1. User expectations are exceedingly high… it must work right the first time
- Incredibly competitive market
2. Real devices are critical
- Test results can be dependent on device, OS, and network provider
3. Speed and velocity is essential
- Pace of change in mobile platforms and devices is incredibly fast
- Mobile development projects must be agile in fast and frequent sprints
4. Mobile is not an isolated effort.
- Must be integrated in normal deliver processes
- Full lifecycle approach leads to quality outcomes
9/19/2013
7
© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.13
Cloud based service
Shared, remote access
Real smartphones
Real Tablets
Real mobile service
Device agnostic
Flexible and robust
Enterprise visibility
Short learning curve
Development
Functional
Performance
Monitoring
Five dimensions of mobile testing
Key considerations for mobile testing success
Test device
management
Reality Automation ALM
integration
End to end
lifecycle
Secure, centralized
© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.14
Is this your mobile test lab?Real devices, but how will it scale?
• Automation
• Security
• Location
• Change mgt
• Governance
• Risk exposure
• Cost
9/19/2013
8
© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.15
Mobile Performance
Mobile applications are everywhere
Impacting:
• Front and back-end economics
• Loss of revenue
• Loss of customer
• Available anytime, anywhere
• Significant business opportunity
• Additional sales channel
• Analyst influence
Why mobile performance has become a competitive differentiator
© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.16
What happens when customers “struggle” on a
mobile device? (TeaLeaf Industries, 2011)
43% Abandon the
transaction, try later
on computer16% Become more
likely to prefer a
competitor14% Log a
complaint with
customer service12% Go directly to a
competitor
You
Your
Competitor
9/19/2013
9
© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.17
Network Virtualization – Reduce Mobile Risk
Capacity > 500+ Users
Capacity < 200 Users
By not considering actual network conditions, it can be a trap!
Without real
network conditionsReal network conditions
VS
Performance test results in a false sense of security.
With real network conditions; Test results accurate and reliable.
© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.18
HP/Shunra customer survey:
•Network Virtualization eliminated >15 performance related production incidents per year
• ROI < 90 Days
ROI: Value of Network Virtualization
5
3.7
0
1
2
3
4
5
6
Results Before vs. After Network Virtualization*Source: 2011 Shunra for HP Customer Survey
Relative Costs to Fix a Problem (very expensive to fix in production)
(Steve McConnell’s “Code Complete”, pp. 960 - Microsoft Press)
# of Performance Related
Production Incidents Per Month
9/19/2013
10
© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
Functional Testing
Object Recognition Types
© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.20
Object Recognition Types
� Native Recognition - recognize objects by their native id and work with the actual
components within the application (i.e., text box, text field, button)
� Image Recognition - captures what is seen on the screen of the device. It captures the
pictures and images and uses complex algorithms to recognize objects while
executing the test
� Optical Character Recognition (OCR) - allows the tester to extract the text from
screen (also known as “screen scrapping”)
9/19/2013
11
© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.21
Difficult with customer objects
Hard to keep up with OS Updates
Flexible and Robust
Adjust to UI Changes quicker
Easier to maintain
Faster test execution
More coding knowledge required
Multi-OS Updates hinder automation activities
Native Object Recognition
Advantages Disadvantages
© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.22
Slow Execution
Difficult to maintain
OS Independent Portable
Easier to buildWorks with what you see on the screen
Increased test maintenance
Reuse is extremely difficult
Image Recognition
Advantages Disadvantages
9/19/2013
12
© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.23
Slow Execution
Not always dealing with text
OS Independent Portable
Easier to buildWorks with what you see on the screen
Increased test maintenance
Reuse is extremely difficult
Optical Character Recognition (OCR)
Advantages Disadvantages
Difficult to maintain
© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.24
Large US Airline
Case Study
Challenges: Struggled with automation of mobile testing, leveraging existing QA
processes, tools & knowhow
Engagement
• Started with a private cloud & function test automation
• Added complementary public cloud - device coverage.
• Performance plans
Currently: Extremely fast ramp up, Running regression, functional, automation,
offshore testing team.
9/19/2013
13
© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.25
Large US Financial Services Firm
Case Study
Challenges: Handset procurement and selection, regression testing around new
versions and new devices, utilizing offshore resources for domestic US networks.
Engagement
• Started with private cloud and functional test based automation
• Added complementary public cloud - device coverage.
• Extended and doubled the capacity recently
• Performance testing plans
Currently: Running regression, functional automation, hundreds of scripts, constant
execution, dedicated team.
© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
Summary
• Take time to understand your company’s
mobile strategy
• Review your team’s skill set
• Performance is as important as design
• Seek help from mobile vendors
• Mobile Testing is complex, but you can be
successful
9/19/2013
14
© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.27
Additional Resources
HP Mobile
Testing
HP Performance
Center
HP LoadRunnerHP Unified
Functional Testing
Resource URL
Mobile Testing @ HP .com www.hp.com/go/mobiletesting
Blog and Community www.hp.com/go/thefutureoftesting
© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
Thank you