![Page 1: Building a small-scale multiplatform automated software](https://reader034.vdocument.in/reader034/viewer/2022051315/627aa34690f87f21751f20d8/html5/thumbnails/1.jpg)
Building a small-scalemultiplatform automatedsoftware testing facility
Maxim Mozgovoy
The University of Aizu
![Page 2: Building a small-scale multiplatform automated software](https://reader034.vdocument.in/reader034/viewer/2022051315/627aa34690f87f21751f20d8/html5/thumbnails/2.jpg)
Primary area: human-like AI for computer games and simulations.
“AI that needs to possess other qualities rather than being good”
Also I have strong interest in practical software developmentand previous industrial experience
Hello!
2
![Page 3: Building a small-scale multiplatform automated software](https://reader034.vdocument.in/reader034/viewer/2022051315/627aa34690f87f21751f20d8/html5/thumbnails/3.jpg)
My friend’s idea
Your AI
3
Ourpeople
Greatgame
Get rich
![Page 4: Building a small-scale multiplatform automated software](https://reader034.vdocument.in/reader034/viewer/2022051315/627aa34690f87f21751f20d8/html5/thumbnails/4.jpg)
We made it (to some extent!)
4
A cool casual tennis game
Released on Win, Mac,Android, and iOSAround 1mln downloadsReally good reviews
Released laterthan hopedProfits aren’t stellar yet
![Page 5: Building a small-scale multiplatform automated software](https://reader034.vdocument.in/reader034/viewer/2022051315/627aa34690f87f21751f20d8/html5/thumbnails/5.jpg)
“
Nothing works better than justimproving your product.
— Joel Spolsky
5
![Page 6: Building a small-scale multiplatform automated software](https://reader034.vdocument.in/reader034/viewer/2022051315/627aa34690f87f21751f20d8/html5/thumbnails/6.jpg)
What’s special about game projects?
Users won’t tolerate bugs [1]
Negative reviews & crashescause downranking [2]
Low-rank apps haveno chance on the market
6
Frequent complaints %
Functional error 26.68
Feature request 15.13
App crash 10.51
Network problem 7.39
![Page 7: Building a small-scale multiplatform automated software](https://reader034.vdocument.in/reader034/viewer/2022051315/627aa34690f87f21751f20d8/html5/thumbnails/7.jpg)
What’s special about game projects?
They are quite complex (really):
Frontend/backend.
User analytics.
Billing/transactions.
Integration with numerous 3rd-party systems(social media, advertisements, online profiles)
7
![Page 8: Building a small-scale multiplatform automated software](https://reader034.vdocument.in/reader034/viewer/2022051315/627aa34690f87f21751f20d8/html5/thumbnails/8.jpg)
What’s special about game projects?
They rely on (unstable) 3rd-party libraries and tools
They have to be updated regularly
8
![Page 9: Building a small-scale multiplatform automated software](https://reader034.vdocument.in/reader034/viewer/2022051315/627aa34690f87f21751f20d8/html5/thumbnails/9.jpg)
What’s special about game projects?
They must run on diversehardware and software platforms
They are prone to issues that are hard to testautomatically (graphics, sound, animation)
GUI and animation is deeply integrated into a game
9
![Page 10: Building a small-scale multiplatform automated software](https://reader034.vdocument.in/reader034/viewer/2022051315/627aa34690f87f21751f20d8/html5/thumbnails/10.jpg)
What’s specialabout game projects?
They contain a largeproportion of code withhigh cost of unit testing [3]
10
unit testingcosts
unit testingbenefits
![Page 11: Building a small-scale multiplatform automated software](https://reader034.vdocument.in/reader034/viewer/2022051315/627aa34690f87f21751f20d8/html5/thumbnails/11.jpg)
Our approach
Primary emphasis onextensive testing of the complete game
Synergy of diverse tools
11
![Page 12: Building a small-scale multiplatform automated software](https://reader034.vdocument.in/reader034/viewer/2022051315/627aa34690f87f21751f20d8/html5/thumbnails/12.jpg)
Tool #1: Firebase Crashlytics
Embed Crashlytics reporting service into the app
12
Crashes are automatically reported to usvia Internet (along with stack traces)
Identified devices with insufficient RAM for the game
![Page 13: Building a small-scale multiplatform automated software](https://reader034.vdocument.in/reader034/viewer/2022051315/627aa34690f87f21751f20d8/html5/thumbnails/13.jpg)
Tool #2: Autobugs and Manual Bugs
Use soft (reporting) asserts and manual reporting tools
13
Report failed non-fatal asserts automatically.Give the users and testers tools to report easily.
Got numerous bug reports(wrong physics, animation, GUI flaws, etc.)
![Page 14: Building a small-scale multiplatform automated software](https://reader034.vdocument.in/reader034/viewer/2022051315/627aa34690f87f21751f20d8/html5/thumbnails/14.jpg)
Tool #3: Automated smoke testing (autotests)
Ensures that the most important subsystems work
by preforming various simple test scenarios
14
The most cost effective method for identifyingand fixing defects in software [after code reviews]
— Microsoft
“
![Page 15: Building a small-scale multiplatform automated software](https://reader034.vdocument.in/reader034/viewer/2022051315/627aa34690f87f21751f20d8/html5/thumbnails/15.jpg)
Our smoke tests
We invested a lot of effortinto building our ownsmoke-testing facilities,and now we find them essentialfor subsequent planned projects
15
![Page 16: Building a small-scale multiplatform automated software](https://reader034.vdocument.in/reader034/viewer/2022051315/627aa34690f87f21751f20d8/html5/thumbnails/16.jpg)
Our pipeline
16
Repository
Buildmachine(several
builds daily)
Ready game(5 platforms)
![Page 17: Building a small-scale multiplatform automated software](https://reader034.vdocument.in/reader034/viewer/2022051315/627aa34690f87f21751f20d8/html5/thumbnails/17.jpg)
Basic levels of QA
The game can be compiled (ensured by the build machine).
The game doesn’t crash on startup(it might due to fatal bugs or incorrect partial build)
The game can connect to our backend server.
Early detection is crucial: we need to knowwhich changeset in our repo causes problems.
17
![Page 18: Building a small-scale multiplatform automated software](https://reader034.vdocument.in/reader034/viewer/2022051315/627aa34690f87f21751f20d8/html5/thumbnails/18.jpg)
“
Let’s just test that the game doesn’tcrash and is able to go online
— Our project manager
18
![Page 19: Building a small-scale multiplatform automated software](https://reader034.vdocument.in/reader034/viewer/2022051315/627aa34690f87f21751f20d8/html5/thumbnails/19.jpg)
One year later
Every build goes through sixtest scenarios; each scenariotakes 30-60 min to complete
19
![Page 20: Building a small-scale multiplatform automated software](https://reader034.vdocument.in/reader034/viewer/2022051315/627aa34690f87f21751f20d8/html5/thumbnails/20.jpg)
Points to consider
The game has no “hidden interface”: the testing systemrelies on the ordinary user-end UI.
Note the synergy: automated tests also generate autobugsand crash events, detectable by our other tools!
Autotests are run on three Android, three iOSand one Windows devices (macOS is planned).
Autotests also report FPS and memory consumption;They can be run for several hours as stress tests.
20
![Page 21: Building a small-scale multiplatform automated software](https://reader034.vdocument.in/reader034/viewer/2022051315/627aa34690f87f21751f20d8/html5/thumbnails/21.jpg)
Example scenario
21
Other scenarios:
Pass tutorial.
Customize player,upgrade skills.
Link a Facebookaccount
![Page 22: Building a small-scale multiplatform automated software](https://reader034.vdocument.in/reader034/viewer/2022051315/627aa34690f87f21751f20d8/html5/thumbnails/22.jpg)
Farming and scripting
How to achieve it?
You’ll need to write test scripts.
You’ll need to have a device farm for running tests.
22
![Page 23: Building a small-scale multiplatform automated software](https://reader034.vdocument.in/reader034/viewer/2022051315/627aa34690f87f21751f20d8/html5/thumbnails/23.jpg)
The easy way
Use existing device farms offered by Amazon, Bitbar, etc.
Pros: easy setup, thousands of devices.
Cons: quite expensive (~15 USD per minute per device)(in our case it translates into ~250 USD daily),device choice is still limited.
Notes: you’ll have to rely on platform-supplied scripting(before writing scripts one must choose the platform).
23
![Page 24: Building a small-scale multiplatform automated software](https://reader034.vdocument.in/reader034/viewer/2022051315/627aa34690f87f21751f20d8/html5/thumbnails/24.jpg)
Our way
Let’s build a device farm ourselves!
24
![Page 25: Building a small-scale multiplatform automated software](https://reader034.vdocument.in/reader034/viewer/2022051315/627aa34690f87f21751f20d8/html5/thumbnails/25.jpg)
Own farm: pros and cons
Flexible: we can choose any devices we need.
Inexpensive (in the long run).
Limited to few specific devices.
Requires regular maintenance.
25
![Page 26: Building a small-scale multiplatform automated software](https://reader034.vdocument.in/reader034/viewer/2022051315/627aa34690f87f21751f20d8/html5/thumbnails/26.jpg)
Minimal setup
Logically, there are three components involved:
A device executing the test scripts (runner).
A device interfacing with the target platform (server).
A target device running our game.
26
testrunner
testserver
targetdevice
![Page 27: Building a small-scale multiplatform automated software](https://reader034.vdocument.in/reader034/viewer/2022051315/627aa34690f87f21751f20d8/html5/thumbnails/27.jpg)
Software setup
Testing capabilities are available on all major platforms. Nosoftware is necessary, but some configuration is required.
27
Android/iOS/Win/Mac
![Page 28: Building a small-scale multiplatform automated software](https://reader034.vdocument.in/reader034/viewer/2022051315/627aa34690f87f21751f20d8/html5/thumbnails/28.jpg)
Software setup
Test server must run a 3rd-party testing framework.We chose open-source Appium (https://appium.io).
28
AppiumAndroid/iOS/
Win/Mac
![Page 29: Building a small-scale multiplatform automated software](https://reader034.vdocument.in/reader034/viewer/2022051315/627aa34690f87f21751f20d8/html5/thumbnails/29.jpg)
Software setup
Test runner executes scripts written using a conventionalprogramming language supported by the framework.
29
AppiumAndroid/iOS/
Win/MacPython +Appium
libs
![Page 30: Building a small-scale multiplatform automated software](https://reader034.vdocument.in/reader034/viewer/2022051315/627aa34690f87f21751f20d8/html5/thumbnails/30.jpg)
How test scripts look like?
Technically, they consist of code like this:
e = appium.find_element_by_class_name('android.widget.EditText')
e.send_keys("hello") # type “hello” into the EditText control
ok = appium.find_element_by_class_name('android.widget.Button')
ok.click() # click the first button on the screen
30
![Page 31: Building a small-scale multiplatform automated software](https://reader034.vdocument.in/reader034/viewer/2022051315/627aa34690f87f21751f20d8/html5/thumbnails/31.jpg)
How test scripts look like?
In our game most GUI elements are drawn on the screensurface and thus not considered “UI” by the operating system.
Thus, we use image recognition:
ib_loc = find_image("input_box.png") # fail test if not found
click_location(ib_loc)
ok_loc = find_image("ok_button.png") # fail test if not found
click_location(ok_loc)31
![Page 32: Building a small-scale multiplatform automated software](https://reader034.vdocument.in/reader034/viewer/2022051315/627aa34690f87f21751f20d8/html5/thumbnails/32.jpg)
How test scripts look like?
32
input_box.png
ok_button.png
![Page 33: Building a small-scale multiplatform automated software](https://reader034.vdocument.in/reader034/viewer/2022051315/627aa34690f87f21751f20d8/html5/thumbnails/33.jpg)
How test scripts look like?
Our test scripts are integrated into the whole build process:
A test script checks whether a new build is available.
If yes, this version is tested with a set of scenarios.
Results are summarized and published as an HTML report.
The process is repeated.
HTML reports are used by the testers to check visual glitchesand understand why certain tests fail. 33
![Page 34: Building a small-scale multiplatform automated software](https://reader034.vdocument.in/reader034/viewer/2022051315/627aa34690f87f21751f20d8/html5/thumbnails/34.jpg)
Our pipeline
34
Repository
Buildmachine
Readygame
TestscriptsHTML
reports
![Page 35: Building a small-scale multiplatform automated software](https://reader034.vdocument.in/reader034/viewer/2022051315/627aa34690f87f21751f20d8/html5/thumbnails/35.jpg)
Complete hardware setup
35
3 Androiddevices
Test runner:
Zotac ZBox MI(Windows)
Test server 1:
Zotac ZBox E(Windows)
Test server 2:
Mac Mini
3 iOSdevices
![Page 36: Building a small-scale multiplatform automated software](https://reader034.vdocument.in/reader034/viewer/2022051315/627aa34690f87f21751f20d8/html5/thumbnails/36.jpg)
Complete hardware setup
36
Servers
Mobiledevices
![Page 37: Building a small-scale multiplatform automated software](https://reader034.vdocument.in/reader034/viewer/2022051315/627aa34690f87f21751f20d8/html5/thumbnails/37.jpg)
Tips, tricks, and notes
Farming isn’t easy!There are numerous pitfalls…
37
![Page 38: Building a small-scale multiplatform automated software](https://reader034.vdocument.in/reader034/viewer/2022051315/627aa34690f87f21751f20d8/html5/thumbnails/38.jpg)
Choosing test devices
When choosing devices, we tried to focus onhardware diversity and get some low-level models.
Sometimes code fails on certain hardware(it happens, e.g., on different GPU chips)
Developers usually have reasonable hardware,so autotests help to make sure the game isstill runs smoothly on low-end devices.
38
![Page 39: Building a small-scale multiplatform automated software](https://reader034.vdocument.in/reader034/viewer/2022051315/627aa34690f87f21751f20d8/html5/thumbnails/39.jpg)
Choosing USB hubs
Mobile devices are connected to a server via USB hubs.
Devices should get power through the hubs(otherwise they will simply discharge).
Surprisingly, it is very difficult to find a hub,able to charge several attached devices fast enough!
39
![Page 40: Building a small-scale multiplatform automated software](https://reader034.vdocument.in/reader034/viewer/2022051315/627aa34690f87f21751f20d8/html5/thumbnails/40.jpg)
Quirks of particular devices
40
Installs the app after several attempts.
Doesn’t unlock the screen.
Doesn’t want to charge from a USB hub.
Asks for regular updates, blocking tests (iOS)
![Page 41: Building a small-scale multiplatform automated software](https://reader034.vdocument.in/reader034/viewer/2022051315/627aa34690f87f21751f20d8/html5/thumbnails/41.jpg)
It is still worth the effort
The safety net feeling we have now brought peace to our lives41
![Page 42: Building a small-scale multiplatform automated software](https://reader034.vdocument.in/reader034/viewer/2022051315/627aa34690f87f21751f20d8/html5/thumbnails/42.jpg)
Contact me:[email protected]
Thanks!
42
[1] A. Moscaritolo (2017). Google Play Now Favoring 'High-Quality Apps’. PC Magazine,https://www.pcmag.com/news/355375/google-play-now-favoring-high-quality-apps[2] H. Khalid et al. (2015) What Do Mobile App Users Complain About? Software,vol. 32, pp. 70-77[3] S. Sanderson (2009). Selective Unit Testing – Costs and Benefits.https://blog.stevensanderson.com/2009/11/04/selective-unit-testing-costs-and-benefits/