four best practices for performance testing mobile apps soasta and utopia

© 2014 SOASTA. All rights reserved. March 4, 2015 1CONFIDENTIAL – Not for Distribution

65

43

21

© 2014 SOASTA. All rights reserved. March 4, 2015 2CONFIDENTIAL – Not for Distribution

Lee Barnes – Founder and CTO of Utopia Solutions

Tom Chavez– SOASTA Sr. Product Evangelist

© 2014 SOASTA. All rights reserved. March 4, 2015 3CONFIDENTIAL – Not for Distribution

o Device/OS diversity

o Network variability

o Device performance

o User location and volume

© Utopia Solutions

Best Practices for Performance Testing Mobile Apps

Lee Barnes, CTO

Utopia Solutions

© Utopia Solutions

Apology

I don’t believe in “best practices”…

I do believe in guidelines and better approaches for a given situation

© Utopia Solutions6

Key Questions

How is mobile different?

How can I adapt?

Why is performance of mobile important?

© Utopia Solutions7

Why Should I Care?

M-Commerce

66%OF

SHOPPERSABANDO

NTRANSACTIO

NS

5 OUT OF 10BECAUSE OF

POORPERFORMAN

CE

1DELA

Y

SECOND 7% DROP

INCONVERSIO

NS

equals a

2013 Mobile Commerce Insights study conducted by Jumio

© Utopia Solutions8

Still Not Convinced?

2013-2014 World Quality Report

© Utopia Solutions9

Evolution of Performance Testing

1990 2000 2010

2-TierClient Server Web Mobile

Challenges

State of the Practice

Testing Large Loads

Complex / Integrated

Systems

ProprietaryProtocols

Skilled Resources

In-Lab Testing

ToolMaturation

Open SourceTools

Specialized Resources

Cloud-based Testing

© Utopia Solutions

Mobile Challenges & Adaptations

© Utopia Solutions11

Mobile Challenges – User Access

Native App Mobile SiteFull / ResponsiveSite

Your users have options…

© Utopia Solutions12

Assess the User and Load Profile

1. Determine the user profile

• Business processes• Native app, mobile site, full site• Preferred browser• Network conditions• Geographic location

2. Determine the load profile

• Volume of users by business process and location

• Volume of users by access type• Frequency / throughput

Incorporate constant feedback from application logs and other monitoring solutions

© Utopia Solutions13

Mobile Challenges - Network

Mobile User Connected User

3G 4G InATunnelG

Network bandwidth and quality affect both end user experience and system load

© Utopia Solutions14

Assess Impact of Network Variability

1. Assess baseline performance(single user)

Observe UX with real devices behind network virtualization solutions

2. Assess performance under load

Put system under virtual user load

Observe UX with real devices behind network virtualization solutions

NV

Virtual Users

NV

Real

Devices

© Utopia Solutions15

Network Virtualization Tools

Single User (Device and/or Emulator)• Apple Network Link Conditioner• Android Emulator• Charles Proxy

Multiple Virtual Users• Charles Proxy

© Utopia Solutions16

Mobile Challenges – User Location and Volume

A high volume of global users is difficult to reproduce in a lab…

© Utopia Solutions17

Utilize Cloud-based Testing Solutions

• Achieve production level loads• Test entire infrastructure (vs.

behind the firewall)

• Distribute load geographically• Assess user impact on real

devices

© Utopia Solutions18

Mobile Challenges – Device Performance

Measuring System / Network Metrics is Not Enough…

Memory

CPU

Battery Drain

Graphics

I / O

© Utopia Solutions19

Measuring Device Performance

Critical device metrics:• CPU• Memory• Battery• Storage

Mobile development platforms• Android DDMS• iOS Instruments

Mobile test automation tools• TouchTest (SOASTA)Mobile lab / cloud solutions• deviceConnect (Mobile Labs)• Remote TestKit (NTT Resonant)

Capture metrics while exercising app

Include appropriate devices / configurations

Incorporate automation for consistency

Device Performance Measurement Solutions

© Utopia Solutions20

Summary

Key takeaways…

• Mobile users are not the same as connected users

• Performance is more than backend / network

• Push performance analysis to the left in the development cycle

© 2014 SOASTA. All rights reserved. March 4, 2015 21CONFIDENTIAL – Not for Distribution

SOASTA PRODUCT DEMO

© 2014 SOASTA. All rights reserved. March 4, 2015 22CONFIDENTIAL – Not for Distribution

o CI and Mobile Lab Scorecard

• 2 days

• Evaluate People, Process, Products, CI

• TouchTest and CI POC with customer tests

• Same mobile automation engineers that have implemented

mobile test automation for Fortune 5 and Fortune 10 retailers

© 2014 SOASTA. All rights reserved. March 4, 2015 23CONFIDENTIAL – Not for Distribution

UTOPIA

o Lee Barnes

o Founder and CTO

o Utopia Solutions, Inc.

o Email: lee.barnes@utopiasolutions.com

o Twitter: twitter.com/USI_LeeBarnes

o LinkedIn: linkedin.com/in/leebarnes

o Blog: www.RaiseYourMQ.com

SOASTA

o Tom Chavez

o Sr. Product Evangelist

o SOASTA

o Email: tchavez@soasta.com

o Twitter: @TomChavez

o LinkedIn: linkedin.com/in/tomchavez