mobile qa metrics risks and automation presentation by yuval golan
TRANSCRIPT
Mobile Testing challenges
Automation
Mobile Automation Meetup , February 2015
Yuval Golan , Erez Nachum , Tali Gilon , Vonage Mobile team
www.vonage.com 2
www.vonage.com 3
Mobile is “Taking over the world”It’s a multi platform age with mass adoption.
Where is your Mobile Testing,mobile CI and Mobile automation?
Complexity Scope Execute Loop back CI Automation
www.vonage.com 4
Why is it so simpleto QA mobile apps?
Complexity Scope Execute Loop back CI Automation
www.vonage.com 5
Complexity
• More than 15K Android devices (last year 4k..)
• Different drivers
• Hundreds of carriers
• Bandwidth & Connectivity
• Tens of OS
• Resolution (LDPI , MDPI…)
• Access rights
• It is actually a phone .. itvibrate , Peripherals (BT ,earpiece), airplane mode , notifications…
Environment
Device
Code
Complexity Scope Execute Loop back CI Automation
www.vonage.com 6
Scope
• Functional
• Usability
• Load
• security
• Localization
• Bugs we wish to resolve
• Network Impairments
• Battery
• Store / purchase
• Disconnections / interrupts (alarm clock ; calls)
Complexity Scope Execute Loop back CI Automation
www.vonage.com 7
Execute
• Module
• Level
• Code coverage
• Recent bugs found / fix
• Recent commits / changes
Complexity Scope Execute Loop back CI Automation
www.vonage.com 8
How to execute
• Manual
• Automate
• Crowd
• Perfecto / device any..
Complexity Scope Execute Loop back CI Automation
www.vonage.com 9
Loopbacks: CI and field feedback
CI Continuous IntegrationPre Commit
QA in most organizations
Field Feedback
Complexity Scope ExecuteLoop back CI Automation
www.vonage.com 10
Metrics - Crashes
47% of theapps crashes1% of their use
32% of theapps crashes2% of their use
Complexity Scope ExecuteLoop back CI Automation
www.vonage.com 11
Metrics , Crashes November 2013 – November 2014
Complexity Scope ExecuteLoop back CI Automation
www.vonage.com 12
Be alert listen to the customers and market
• OS (7.1 no audio)
• IPV6 / IPV4 (crash)
Complexity Scope ExecuteLoop back CI Automation
www.vonage.com 13
Metrics – Star Ranks / user reviews
Overall Store Rank by Version
Complexity Scope ExecuteLoop back CI Automation
www.vonage.com 14
Mobile CI for Commit
Developer commit some code to the
repository
Jenkins (CI) get notified and starts
compilation
On success “Static analysis” and post
commit Code review running
Deploy an environment Then automated “Commit tests”
When commit is approved it is included in next QA build , the latest stable build will
run on nightly regression
New code version is distributed to QA/BETA
Complexity Scope Execute Loop back CI Automation
www.vonage.com 15
E2E (Seetest) Per commit (KIF + Espresso)
Automation dev investment
Write test once for both platforms
Different code per platform, duplicate tests
Native actions V (Some IOS issues) X
Scalability License required Open source (dev use it)
Automation project location
Automation repository Part of Dev repository
Execution time 1m-2m (per E2E test, 2 devices)
0.2m-0.3m (1 device only)
Reports Built in Requires development
Number of mobile clients in use
1+ (we use 2) 1
Support V X
Automation 3rd party Tools that we use
Complexity Scope Execute Loop back CI Automation
www.vonage.com 16
Vonage Automation Framework Features• Reporting system (using Jenkins abilities + Junit reports )
• Smart investigation tools (search for specific exceptions and can connects them to existing Jira tickets, saves investigation time)
• Web Client (Vonage Mobile code compiled on Mac/PC, simulates 2nd device )
• Automation Test Apps (in order to simulate native changes in native address book ,native settings…)
• Retry mechanism (differentiates between one timers to 100% reproducible bugs)
• Network traffic investigation tools (Http request- response, SIP flows in calls , uses app logs)
• Private BE environment (Http request- response, SIP flows in calls, uses app logs)
Our Framework
Complexity Scope Execute Loop back CI Automation
www.vonage.com 17
Nightly run
2 tests fail because of an Amazon issue on uploading large video files.
Good morning ☺
Complexity Scope Execute Loop back CI Automation
www.vonage.com 18
Web client
No need in secondary devices for manual/automated tests, web client behaves like a real client and can eveninitiate events to your App.
Complexity Scope Execute Loop back CI Automation
www.vonage.com 19
UI – system with Keywords
anyone can
write tests using
it!
Complexity Scope Execute Loop back CI Automation
www.vonage.com 20
Module tests
Country Manager module tests we inject inputto the module's functions and compare the outputto the expected results.
Complexity Scope Execute Loop back CI Automation
www.vonage.com 21
DDT – Data Driven TestSend message setup- ~15 lines
Json file with scenarios-
Data Driven testing
One function + one “Jason” =
~1K tests
Complexity Scope Execute Loop back CI Automation
www.vonage.com 22
What’s next?
•Coverage , coverage , coverage
•Appium
•Appurify
•Qtest
What’s Next ?
Complexity Scope Execute Loop back CI Automation
www.vonage.com 23
