Cross-platform test automation using Appium

Cross-platform test automation using AppiumJatin Bhasin @jatin_bhasin

1

Who am I?Tester / hat-jugglerAgilistLearnerTravellerWork(ed) for:

Good evening allBefore I begin I would like to thank Ady, Vikas and everyone else involved in organising the meetup. You guys are doing a brilliant job.About me I work as a QA lead for the mobile team at MYOB since last 1 year and was on their flagship windows product prior to that. Before joining MYOB I was working with TWHaving worked for these companies has given me opportunities to work on a variety of platforms and tech stacks but I sincerely believe there is still heaps to learnI love travelling and exploring new places.2

Who are you?

So that was a bit about me, now I would like to know a bit about you all. Who here has worked on mobile projectHave you used some sort of automation toolGreat, so we do have a good mix here people who have worked on some of these areas and some who are keen to learn more3

Mobile whats the big deal?

So whats the big deal about mobile4

By 2018, more than 50% of users will use a tablet or smartphone first for all online activities

- GartnerFuture

A recent survey by Gartner indicates that by 2018 more than 50% of the users will use a mobile device first in order to access the internet.Well I would say mobile is not the future anymore in fact it is the present now.5

Desktop vs Mobile

source: Morgan Stanley Research

Another study by Morgan Stanley suggests that early last year the number of mobile users have already surpassed the total number of desktop users globally.Therefore, as a result - a lot of companies and products are now moving to the mobile first strategy.They want to make sure whatever you are able to do on their web site you should be able to do on your mobile too.I dont remember when was the last time I logged in to facebook on my laptop, neither have I logged in to my banks website for quite some time6

What is an appNativeWebHybrid

SO far I have used the word apps quite a few times. Now let us understand what does an app mean?Primarily there are 3 kinds of mobile apps Native, Web and Hybrid

Native - A native app is an app developed specifically for the targeted phones operating system, for example iOS or Android.The user experience, interaction with the phones technology & performance of these apps is unrivaled. There is a considerable dev and testing effort involved because you are dealing with 2 entirely different codebases. Key words you may hear when explaining the development environment are Objective-C, Swift, Java. This app is available in the app store.

Responsive Mobile web or often referred to as responsive is nothing but a web page that renders to a wide range of devices. Imagine opening Google or yahoo on your phones browser and you will notice that despite so much of content and such small screen of your mobile the content doesnt get distorted. So these are not available on app store and rather you simply open them on your browser.

Hybrid - A hybrid app is a mix between web and native. Technologies such as Phonegap and Cordova or Titanium allows developers to take web technology and wrap it in a native shell. This is done so the app can take advantage of the phones hardware and features but can also be ported across platform. These are again downloadable from the app store so from a normal users perspective they dont see much of a difference.7

Our Apps

Talking about our apps we have 2 apps PayDirect and OnTheGo which are available for both android and iOSThese are all native apps. So technically there are 4 apps to be taken care of.8

PayDirect & OnTheGo

PayDirect allows our customers to pair the card reader via bluetooth and then take credit card payments. They can swipe, insert or tap the card in order to do so.OnTheGo is primarily meant for raising invoices. Its being used by a lot of tradees or small businesses who have to visit different places.So imagine a plumber coming to your place who aftre doing his job can straight away raise an invoice and get his money right then. Its gaining a lot of traction in our small and medium enterprise business division.Apart from creating invoices, you can also create contacts or get in touch with our customer care.There is a but of feature overlap in these invoices.

We are still adding more features and typically do a release every 2 weeks. So you can imagine we always have to be on our toes.

15% of SME businesses are using these apps now and growing 10% compounded monthly

9

Mobile testing challengesRatings and reviewsDevice and OS distributionTime to marketComplexityAutomation

Big challenges on small devices. Mobile apps pose new challenges to testing:Rating and Reviews. Bad quality may cause trouble to the reputation of the company and brand. In the era of millions of charts, lists and reviews there hasnt been a case when a company released a low-quality app and people didint find out about it. The influence of bad reviews can be drastic.Device and OS distribution. Mobile devices change and improve with unprecedented speed. Annually hundreds on new devices appear on the market. Different screen sizes, form-factors and the variety of OS often cause problems to apps.Urgency. You need to be fast, extremely fast to move with the times. Teams should create design, write code, test and release an application within a very short period of time. Traditional manual testing doesnt meet the speed requirements of mobile world.Complexity. All the mobile devices are becoming more and more elaborate, and are based on more and more advanced technologies. Location recognition, Wi-Fi, real-time events, popups all this makes both manual and automated testing way more complicated.

10

Mobile Automation what??

I would say automation could be considered both challenge and a solution for all these other challenges.Because the time to market is so short you cant really rely on manually testing all the flows in the app when doing a release.So its best to automate your test cases. But please while doing so be congnizant of the basics such as testing pyramid.

11

Mobile Automation Tools

There are a plenty of automation tools available in market. I have put a few of those here on this slide.Some of these are platform specific whereas others support cross platform automation.

Robotium is an Android test automation framework that fully supports native and hybrid applications. Robotium makes it easy to write powerful and robust automatic black-box UI tests for Android applications. With the support of Robotium, test case developers can write function, system and user acceptance test scenarios, spanning multiple Android activities.UIautomator, by Google, provides an efficient way to test UIs. It creates automated functional test cases that can be executed against apps on real Android devices and emulators. It includes a viewer, which is a GUI tool to scan and analyze the UI components of an Android app.Espresso, by Google, is a pretty new test automation framework that got open-sourced just last year, making it available for developers and testers to hammer out their UIs. Espresso has an API that is small, predictable, easy to learn and built on top of the Android instrumentation framework. You can quickly write concise and reliable Android UI tests with it.Calabash is a cross-platform test automation framework for Android and iOS native and hybrid applications. Calabashs easy-to-understand syntax enables even non-technical people to create and execute automated acceptance tests for apps on both of these mobile platforms.

12

Test Stack (until recently)

iOSAndroidNativeTeam CityObjective C / SwiftJava / GroovyUIAutomation, BwokenRobotiumKiwi, KIFRoboSpock

Not very long ago our test stack used to look something like this.For iOS we were using UIAutomation for test automation.For android were had all our tests automated using RobotiumWe were also using Kiwi and KIF for Unit and Integration testing in iOSFor Android unit testing we use Robospock

Apart from the apps we also have a server component which handles most of the business logic. The server is ruby code base and we use cucumber for automating server acceptance tests

So even though from business perspective these are more or less identical apps, we were compelled to maintain 2 entirely different test suites.13

What were we looking for

So we knew there was scope of improvement and we started jotting down what were the things we were looking for14

Cross-platform automation

First and foremost we wanted to use a cross platform automation tool that could allow us to run the same test against both android and ios.This will mean Reusable Code And Less maintenance effort15

Native, hybrid and web

While as of now all our apps are native, our tech roadmap is quite diverse and there is a potential that we may create hyrbrid or web apps in future. So wanted to take our tech roadmap into consideration and invest in a tool that could automate these other types as well.16

Scalability

As new devices and operating system versions are released every day we want to go for a tool which is up to date with all these developments. And also the one which could allow to increase our coverage as and when required.17

On demand cloud testing

As a next step to scalability we wanted to look for a tool which is compatible with a cloud based testing service provider.There are cloud based testing service providers that allow you to run your tests on an array of devices and operating systems and then generate a detailed report.18

Freedom to choose the programming language

So far we have had our test written in tools which only supported a specific scripting language. Which means if a test fails or there is any issue only the people who have worked on that language will be able to investigate. We wanted a tool that could allow us to chose a language of our choice.19

BDD support

We definitely wanted to add BDD layer to our tests. So that these tests could be written even by the non technical stake holders of the project such as product owner.Also the BDD tests act as a living documentation.20

Tools comparisonMonkeyTalkCalabashAppiumAndroidiOSMobile Web (Only Android)Scripting languageMonkeyScriptRubyAlmost anyCostFree (Acquired recently)FreeFreeGithub stars91079 / 9702598Github forks4277 / 4511744Github ActivityLowHighVery HighTest unaltered appBDD Integration (Cucumber) (Many options)

Keeping all those motivations in mind, we started looking for cross platform automation tools and narrowed down to these.MonketTalk lost early in the race because of various reasons. In fact it has been acquired by Orcale recently so its future remains uncertain.Calabash is definitely good and I would say as popular as AppiumThe github stats vary slightly.

However, Appiums USP is that it doesnt require you to alter the app because the appium server utilizes the platform specific tools in order to gain control of your device. I will talk about its architecture in a bit more detail soon.

Calabash is undergoing a revamp and will come up as 2.0 wherein they are re-writing the API from scratch so there could be some major changes coming up21

Why Appium

So why did we chose appiumSupports both Android and iOSBecause it takes advantage of seleniums json wire protocol, all the languages that can be used to write selenium tests can be used to write appium tests too.Supports native, web and hybridIt allows you to run your tests in an emulator or on a real deviceBeing selenium based, it follows a well formulated API so if you have ever worked on selenium before, picking up Appium becomes quite easy.

Appium uses 2 technologies: the Selenium API (that is, a subset of Selenium commands that are relevant for mobile testing), and the WebDriver JSON Wire Protocol. The Selenium/WebDriver combo is a de-facto standard in end-to-end system testing and is well-known to many developers and testers. 22

Why Appium

BDD

Appium is sponsored by Saucelabs and therefore it seemlessly integrates with their cloud offering. However, it is compatible with most of the other players in the market.It does support BDD which was another key motivation for us.There is a strong community support. They have their own discussion forum and even the issues raised on github get responded to pretty quickly.As I said before, it allows you to Test the same app that you would submit to app store you do not need to compile a third party framework into your apps code that needs to be stripped out later. Avoids the risk of introducing any unforeseen bugs. Test the same app as you would release.And last but not the least, it is open source so its free and you can fork and create your own version if you want.

23

Appium Architecture

WebDriver Scripts

1. Commands are sent in form of JSON via HTTP2. Appium server invokes platform specific tools to execute commands3. Client sends back response to the Appium server4. Appium server logs the result

So how does Appium do all this? Lets try to understand what happens behind the scenes.Appium basically is an HTTP server written using nodeJS that exposes a REST APIProvides common front end API to the totally different back endsIt is almost the same as the Selenium server that gets HTTP requests from Selenium client libraries.Step 1 commands are sent in form of JSON via HTTPStep 2 Appium invokes platform specific tools to execute the commandsStep 3 Client sends back response to the appium serverWhich is then logged on the console

When I say platform specific Appium uses UIAutomator which is capable of controlling the entire device on Android and execute these commandsSimilarly, for iOS it uses Instruments which is again provided by apple and is capable of controlling and executing commands on an ios device

At the webscripts level, it uses desiredcapabilities the same way you can do it for dowing cross browser automation with selenium.24

Appium app

Appart from running the appium server from the command line, you can also launch it via UI. The Appium app is capable of doing pretty much everything that you may want to do with command line, so you dont need to worry about Node.

Clicking on the search icon here takes you to the appium inspector25

Appium Inspector

The inspector enables you to check out the hierarchy of your app. This can come in handy when writing tests.equivalent of selenium IDEAlso supports basic record and playback26

Adding BDD to the mix

We also added a BDD layer to our appium tests, in order to do this we were using cucumber to write scenariosThe step definitions were written using RubyAnd we are following page object pattern.

The Page Object pattern represents the screens of your app as a series of objects. You can separate out your testing code from the app screens. It also provides you the ability to reuse a screen in multiple tests. So if there is a UI change required, you dont need to modify all the tests rather just update the page (screen)27

Example: Scenario

Here is a small example of a test scenario written using cucumber

All I am trying to do here is Launch the appLogin using valid credentialsOnce I successfully login, I should be taken to the setup passcode screen28

Example: Step definitions

Here is the step definition for this scenario written in rubyAs you can see we have tried to write minimal code here and rather are calling our screens whenever we want to do a particular operation on those screens29

Example: Pages

AndroidiOS

Here is how a sample screen would look likeThey are both inheriting a common base classThey are a few differences in these screensThe title for Android screen is MYOB where as for iOS it is loginSimilarly, we are using IDs for android and name for iOS

Its a good idea to keep pages separate for both the platforms so that change in one app doesnt impact the tests for the other platform30

Appium Test Demo

Now I thought because login is such a quick scenario it will hardly take a few seconds to execute and so I have chosen a slightly longer feature for the purpose of demo

So in this example here I am testing the dashboard feature

GO TO NEXT SLIDE

=======================

So as you can see as part of these scenarios I drill into various areas of the app and assert that I am actually there

GO TO DEMO31

Test Stack (now)

iOSAndroidNativeTeam CityAppium, Cucumber, RspecObjective C / SwiftJava / GroovyUIAutomation, BwokenRobotiumKiwi, KIFRoboSpock

So after all these recent changes, our test stack has grown up a bit.We have this another layer of acceptance tests written using Appium and Cucumber32

Appium Limitations1 iOS test per Mac OS instruments limitationLimited support for Android < 4.1Some gestures not supportedWindows Mobile not supportedImage comparison not supported

Appium is No silver bullet. There are some limitations.For example you can only run 1 iOS test at a time. This is also because of the fact that Appium uses apples instruments framework under the hood and you can have only one instance of instruments running at a time. For android you need to start multiple appium servers with different flagsIt has limited support for Android versions lower than 4.1. There is a work around available by using Selendroid.Appium extends the WebDriver spec with extra commands to support mobile gestures such as long tap and shake. However, not all mobile gestures are supported out of the box and you may have to write your own methods in order to make some gestures work. Some people find this annoying.As of now it supports Android, iOS and even Firfox OS but doesnt support Windows MobileAppium doesnt have the ability to do image comparison. So if you are after something like that, tool like Sikuli may better suit your needs.

Fortunately, none of these limitations have been showstopper for us. But its good to be aware of both pros and cons before zeroing down on a particular tool.33

The road aheadMigrate tests to AppiumReview existing testsUse real devices

As I said we have just started with Appium for last few months and there is still a long road ahead of us in order to achieve NIRAVANA1st of all we need to migrate all our existing tests to Appium. Because we have a large number of tests written in the other tools which are absolutely valuable till we port them all into appium, we nee to have both the test frameworks in our pipelines.We are making sure all new tests are being written in Appium.

Also, as and when we add new tests to appium we take a momemnt to review the existing tests to see if something got covered and we can get rid of those tests.

We have been running our tests on simulators and emulators in the pipelines. However, its always best to run your tests on a real device. Appium does support that. In order to achieve this in a scalable manner we are evaluating cloud service offerings to see which one suits us the best.

So there are more milestones for us to achieve but Appium certainly looks promising and seems to be the step in right direction.34

Questions

linkedin.com/in/jatinbhasin@jatin_bhasin

I hope I ddnt bore you enough, feel free to ask questions now or get in touch via linkedin and I will be happy to help

Q. Can you test web view applications with appiu? A. Yes you can. Tests can be run in context of a webview, pretty much execute selenium tests on the mobile web viewQ. How scripts handle crashes? A. Scripts handle crashes by ending the session and you get an error code35

Referenceshttp://appium.io/https://docs.saucelabs.com/tutorials/appium/https://github.com/appium/appiumhttps://discuss.appium.io/https://github.com/appium/appium/blob/master/ROADMAP.md