my 10 mobile automation questions
TRANSCRIPT
My 10 Mobile Automation
Questions
Hello!I’m Ran Ben Aharon
Automation & Tools lead @ EverythingMehttp://ranbena.com@ranbena
Track Record▣ Netcraft▣ Mobile Web dev▣ Firefox OS▣ HTML5IL + FEDs
I LUV▣ Coding (JS + Python)▣ Refactoring▣ Pull Requesting▣ Solving problems with Tools
<3
FIXED A TYPO
NOW APP CRASHES IN PRODUCTION
RELEASE DAY
SCARED SHITLESS
DREAMED OF MAKING MOBILE APPS
NOW I CAN’T SLEEP
‘’In order to move fast, you need good brakes.
Joey Simhon EverythingMe
Auto-Tools
aka Visibility
CommunityShare, Discuss, Support, Network
Doing Mobile Automation?
55%
Attempted
Yep
17%Not yet28%
Your answers
10 QuestionsI needed to answer in order to begin
1.Why is it so hard
to automate mobile?
Y u so hard
▣ No widely used solution▣ Hard to get advice▣ Not everything fits▣ Requires lots of attention▣ Devices fragmentation
24 devicesDistinct devices
8 sizesDistinct screen dimensions
70% usersLatest OS adoption rate
18,796 devicesDistinct devices (Aug ‘14)
150 sizesDistinct screen dimensions
2.5% usersLatest OS adoption rate
via OpenSignal
2.How is it so
different from desktop software
automation?
Different from desktop
▣ Not yet mature▣ No common open source solution▣ Device fragmentation▣ Users interact differently▣ Device resources are limited▣ App store limitations▣ Rapid deployment
3.Is there no solid
open source solution out
there?Why did we have to go and build our own?
No solid open source solution
▣ Maturity▣ Single platform*▣ Single dev language*▣ No one-stop-shop
CI > Testing fw > UI simulation fw > Visibility
▣ Each app has it’s unique needs□ None have ALL features□ Concentrate on UI testing
* Excluding Appium
4.How do the big
boys do it?
The big boys
EbayCalabash
Mobile automation at Ebay Slides
MozillaRobocop for AndroidMarionette for Firefox OS
Automated testing for Firefox OS
SoundCloudRobotium for AndroidFrank for iOS
FacebookSelendroid for Android
YelpKIF for iOSEspresso for Android
ZapposKIF for iOS
5.Must test on real
devices?Are emulators and simulators ok too?
Emulator▣ Cheap▣ Fast▣ Multiple on 1 comp▣ Sharable▣ Mock battery,
sensors, network▣ No hw fatigue▣ Low maintenance
Real or Emulator?
Real device▣ Reliable results▣ Any model available▣ No need for
resource emulation
6.Device labs on
premise or cloud service?
Pros▣ Focus on most
relevant devices▣ No dependency▣ No queue▣ Real time▣ Fast results*
On premise
Pros
▣ Wide range of devices▣ Scale▣ Zero maintenance▣ Real locations*▣ Real cell networks*▣ Your test fw + CI▣ Built-in stress testing*▣ Manual testing*
Device cloud
To name a few
Appthwack
Appurify
Experitest
Keynote
Perfectomobile
SauceLabs
TestDroid
...
LessPainful
Google I/O ‘14
opendevicelab.com
7.Open source or
commercial service?
Pros▣ Free▣ Community▣ Your code
Open source
To name a few▣ Appium▣ Calabash▣ Espresso▣ Frank▣ KIF▣ Robotium▣ Selendroid▣ MonkeyTalk▣ ...
Android▣ UiAutomator
API 16+ (Appium)
▣ UiAutomationAPI 18+ (Robotium)
▣ InstrumentationAPI 1+ (Espresso, Selendroid, Calabash)
User Action Simulation & Introspection
iOS▣ UiAutomation
(Appium, Zucchini)
▣ UiSpec (Kif, Frank)
Magneto - Command your Droids
Magneto
Pros▣ Working solution▣ Robust reports▣ Script recording▣ Device cloud
integration*
Proprietary solutions
To name a few▣ Experitest▣ Jamo▣ Keynote▣ Ranorex▣ SOASTA▣ TenKod▣ ...
8.Who writes test
scenarios? Should you?
▣ Automation engineer?▣ QA team?▣ Developer?▣ Product?
Who writes tests?
9.Where and when should tests run?
Is there a widely adopted methodology?
Release Flow Stages
Dev
Review
MergeRelease
Spec RC
Release Flow Stages
Dev
Review
MergeRelease
Spec RC
Build Build Build Build
10.How do you
maintain result trust?
▣ Unreliable results▣ Not part of release flow▣ No visibility▣ Cumbersome tools
Distrust and disengagement
▣ Reliable results▣ Integral part of release flow▣ Full visibility▣ Delightful tools
Trust and cooperation
UI Testing. Got it. Anything else?
Much much more.UI testing is just one aspect out of many.
This is where it gets even more interesting!
Much more than UI testing
▣ Broken connectivity simulation▣ Offline mode▣ Excessive resource usage▣ AB testing / feature switches▣ Non visual events assertions▣ Jank detection▣ API mocks▣ Stats testing
▣ OTA updates▣ Stress testing▣ Image recognition▣ Crash detection▣ Purchase/advert flow▣ Sensors▣ OS event simulation
More info
▣ GTAC - 2014 vids▣ SmashingMagazine article▣ TestDroid blog
