introduction -...
TRANSCRIPT
Share this eBook!
INTRODUCTIONBuilding a million-download mobile game is not one-day easy job. Not only do you
need to outperform your direct competitors on sales and marketing, but also you
should provide an ACE quality mobile game to your frontier colleagues to win over the
gameplayers.
By understanding the differences between regular app testing and mobile game testing,
you have taken the first step. Once you gain the familiarity in the key areas in mobile game
testing, you are on the right track. But there is much more.
With several years providing the service and support to one of the most successful
mobile gaming developers, we have been gaining rich experience in this area. And we
would like to share with you our knowledge regarding the fundamentals of mobile gaming
development and testing.
With this ebook you will learn three critical fundamentals in mobile gaming development
- Architecture and Infrastructure, Agile Process in Development, and Value of Testing.
Share this eBook!
CHAPTER 1
Architecture and InfrastructureAspects of Mobile Game Testing
Share this eBook!
O ne of the most common problems in mobile game testing – similar to app testing
– is how to access all possible devices that your game users are using. As said,
despite thousands of unique models already out there for Android (and with all possible
iOS variants with different OS versions) only few hundred of them are really relevant
in the big picture. And naturally, this keeps changing all the time and new devices are
introduced by tens of major OEMs every quarter.
ARCHITECTURE AND INFRASTRUCTURE
Many of today’s mobile games are based on open source or commercial game engines,
such as Unreal, Unity3D, CryEngine, Construct, PlayCanvas, Cocos2D, and many others!
Furthermore, these game engines provide great tools and frameworks to get you game
quickly built, with all those nice graphical features.
Looking this from the traditional software point of view, this is like a “tool – app –
middleware” offering that these products will provide you, as well as compiling your game
to specific platform (many of these products are cross-platform and enable you instantly
to compile the game for Android, iOS and other platforms as well). For the platform part,
let’s use Android as an example: an “open” operating system with a stack of software
Share this eBook!
components which can be roughly divided into four layers: Application, Application
framework (manager layer for content, resource, package etc.), Libraries (e.g. OpenGL ES
software, Fonts, WebKit, SGL etc.) and the Linux kernel (drivers for graphics, audio and
keypad, power management etc.). In addition, platform here includes the actual hardware
running whatever chipset on it, with GPU and displays of different resolutions.
After that your mobile game needs to communicate with back-end servers using your
own or Google/Apple/other services via WiFi, radio or some other type of communication
channel. The need for testing the connectivity between your mobile game and back-end
services is nowadays very important. For example, ads are shown via connectivity and
without those running properly on your device, you are taking an extra risk in front of
your core audience – mobile gamers. Sure, they would probably like it better without ads,
but from the monetization point of view, you surely want to test this thoroughly before
submitting your app to app stores – and make sure you are not losing revenue because
of faulty implementation.
Share this eBook!
As said, testing connectivity should be one of the key things to do when building mobile
games. Test automation can bring you a lot benefits for automating the connectivity
testing, and scaling the infrastructure from one user to hundreds of thousands of users.
HERMETIC ENVIRONMENT AND TESTING
In most cases, today’s mobile games have a server-client interaction, some requiring a login,
uploading of data (e.g. game results/progress) and downloading of data (e.g. ads). When
adopting these kinds of services to be part of your games, developers need to ensure that all
their changes done in code, to either server side or client side do not break the functionality
of the service. Furthermore, this type of environment can be also set up on a local machine/
server where critical connectivity and interaction between user and server can be tested. In
fact, testing for some of these types of use cases could be done efficiently with a hermetic
environment – however, not jeopardizing the integrity of your game.
Hermetic testing can be seen and done as easily as switching your device to flight mode.
Nothing gets sent out and nothing gets in.
Share this eBook!
COMPATIBILITY AND PORTING – TESTING
When your mobile game is getting developed, you should always test it on the real platform
that end-users are using as well. This means that you must test your game on real devices.
Emulators won’t simply provide you as authentic environment as it would be required to
get full understanding of real behavior in hands of real users. What makes a difference
between using an emulator or real hardware? Typically, the lack of real hardware slows
down and forces developers to use emulators, but this can shrunk into 4 categories why
games should be only tested on real devices:
1) SOFTWARE. It’s just not only about vanilla platform software that makes the difference,
but nowadays OEMs customize their software a lot: UI layers, customized legacy software,
only certain widely used apps are pre-integrated with the device (e.g. Twitter, Facebook).
2) HARDWARE. There are LOTS of differences in chipsets, GPUs, even connectivity
hardware, displays, sensors, and memory. This is very hard to emulate appropriately –
and even it would get as close to real world scenario – all of that is executed on PC x86
hardware, and not ARM-based hardware.
3) USER EXPERIENCE. This is heavily related to performance. The graphics performance
of the system makes the user experience, either good or bad. And there is an absolute
must to get mobile game tested on real hardware.
4) NETWORK. The system can experience network issues and slow network is not possible
to be tested on emulation. In terms of network configuration, emulators run on the PC,
connect to the LAN and access the internet via your corporate firewall. Using real handsets,
the network is connected to the radio interface and from there to the internet.
Share this eBook!
Whatever platform (e.g. native mobile platforms, web etc.) you are building your mobile
game on, remember that only real devices give you real results. Furthermore, the depth of
testing (e.g. how many devices are used, how many different configuration those cover)
is important as you don’t want to minimize your target audience who have potential to
play your game – make you money!
SMOKE AND REFERENCE TESTING
When the mobile game is in the state that is testable in the context of 1) game itself, 2)
platform-compiled and 3) connects with back-end services (or just platform APIs) the
full-blown reference testing can give great results and understanding of how well game
does. But even much earlier than that, it is possible to start doing smoke testing and using
even hundreds of real Android/iOS devices for that. When bugs are revealed and fixed
earlier, it will save you a lot of time (and money) and gets you earlier to publish your game
Share this eBook!
for gamers. In addition to regular smoke and reference testing, it’s a good idea to include
real performance and battery drain tests as part of your testing pattern.
The reference testing gives you great understanding of how your game does, but it doesn’t
cover everything. For example, huge number of users can quickly take down the back-
end, slow down the game-play and even worse, force users to stop playing the game.
STRESS, LOAD, SPIKE, SOAK – PERFORMANCE TESTING
Good performance is very closely related to good user experience. User wants to see
constant progress with the game, do the smooth game-playing, graphics performance
needs to be up to snuff and work across all different mobile devices (phones, tablets etc.)
the very same. In addition, it is important to understand how well back-end(s) can handle
massive number of users and significant load.
Share this eBook!
There are lots of different ways to build back-end testing and scale up the performance
so that simulates the huge load for your service. One easy and very straightforward way
to do this is to use device farm on cloud to get all of those devices to do the same thing
for your service. It literally takes minutes to set it up and get results back instantly.
Share this eBook!
CHAPTER 2
The Agile Process for Mobile Game Development and Testing
Share this eBook!
T hings that get done manually are hardly agile. Manual testing is not an exception – but
it will significantly slow down the delivery if each and every change or new feature
add-on have to be manually tested. So, how do the most successful game development
teams do their game development and testing?
They do Agile – Developers always concentrate on the delivery of new features,
enhancements, and bug fixes. The use of test automation is in an epicenter of how
developers already build their software and especially how this process can be significantly
boosted, with automated testing included in the process and using real devices – the
same ones that gamers use.
Why Testing Must Be Included in Day-to-Day Process?
The common approach, even today, with mobile game testing is that all of that important QA
and testing work gets done at very late of the development process. Typically, companies
are even dedicating specific time for this when QA, testers and beta testers do their work
post-production. While testing is very important, nearly all of it can be included as part
Share this eBook!
of the agile development process and every aspect of the game can be tested during
the development – and during the production. Some certain foundations can be even
tested (e.g. frameworks, platform setup) during pre-production, but the most important
thing in order to improve the efficiency is to seamlessly integrate testing as part of the
production phase.
The basic definition of continuous quality and delivery would be simply delivering the
better games faster and do the majority of the testing while actual development is ongoing.
In any cases, continuous integration, delivery and testing can significantly improve the
outcome of your mobile games and result as a more robust end-product.
Continuous testing using real devices can save up to 60% of overall development and
testing time when testing is done as an integral part of the development. This has a direct
impact on staffing, costs and faster time-to-market. And when things get fixed earlier,
debugging time can go from months to hours.
Share this eBook!
When testing is not part of the actual development: In the first sprint you do the regular
things and get the development going. Before the testing of even smaller parts or standalone
features gets done, you simply don’t have visibility of the actual robustness of your mobile
game. At some point, these bugs will become obvious and then stirs up again your sprint
procedures and causes an halt of the overall development. In this example, developers
spend much more time on firefighting issue that was part of the code for weeks or even
months. Even the worse example would be that the bug gets revealed when the game out
for users – e.g. as an user feedback or by using some other post-release analytic systems.
Developers should always spend more time on new features, incremental improvements,
than fire fighting the old bugs and problems. Things with bugs escalate quickly when
noticed later on, which will have a direct impact on developers daily tasks.
HOW TO ADD “EXTRA DAYS” FOR AGILE DEVELOPMENT?
Effective mobile game testing derives from a well-structured and systematic approach, use of
test automation framework(s) and real mobile devices, and seamless integration with the agile
development process. By adding an ‘extra time’ for the process – e.g. nights, when typically
development work is not going on – you can easily dedicate this time for test automation.
Written test scripts and automatic pre-processing of test data (logs, screenshots, results) will
provide a report of how well the recent regression worked, what were the exact problems
and even suggestions on how to fix those issues will be part of the deal with test automation.
After the package (APK or IPA) is built, it can be automatically send for the first test – the smoke
test. The smoke test can be also seen as a compatibility test when it will be executed on all
possible device variants. Also, each regression can be tested with the existing test cases/scripts
for certain specific function. For example, if you mobile game relies on back-end service, this
connection can be after each build regression tested with automated scripts. In addition, many
of aspects can be instantly automated: localization (testing different languages), different types
Share this eBook!
of performance tests (load, spike, stress and others are typically tests when game contains
more ‘beef’ and are truly stressing the hardware), connectivity and hermetic testing (isolation
of game from network, other services etc.).
Instead of fixing bugs later in the following sprints, this approach focus on fixing bugs
instantly after the regression. Developers will be notified about their enhancements, new
source code/build problems during the next day, and will have a chance to fix those
problems even early in the ongoing sprint.
Example of tests that can be done during the development
Smoke testing with automatic test exerciser. Automatic test exercisers provide a great
way to smoke-test games. They need no specific tests but focus is in testing the user
interface logic (e.g. opening menus, clicking buttons, doing swipes, multi-gestures etc.).
Automatic test exercisers provide the least exact results, but those are very powerful in
terms of providing quick feedback on each iteration/regression of your mobile game.
Share this eBook!
Regression testing after every code change. Regression testing can be used to ensure
that the quality of mobile game is still good after any features, add-ons or changes in
game. Each feature of mobile game should be tested on all available platforms before
development of the next feature. Thus, regression testing is a part of testing life cycle and
very important element in effort to build more robust games.
Connectivity testing. E.g most of the mobile games today have a server-client interaction,
requiring a login, uploading of data (e.g. results) and downloading of data (e.g. data,
images). If you’re developing these kinds of services around your game, developers need
to ensure all their changes are done in code, to either server side or client side, do not
break the functionality of the service.
Performance testing and all variants of it are important for mobile game testing. Sluggish and
badly performing games can ensure that game won’t be successful, gets bad ratings and
won’t make its creator name. Test automation can help to understand the real performance
Share this eBook!
and many other aspects of game’s performance (e.g. battery consumption). For example,
developers can create patterns to see how graphics assets gets loaded and used – and based
on that information to optimize handling of those.
Localization and language settings testing becomes highly important when your game is
targeted for the global markets. The word ‘global’ means your game needs to be ‘local’ for
everyone. When your game titles, texts and content needs to be translated and tested with
devices in multiple languages, these types of tests can be
easily performed automatically (with help of cloud-based
device access and test automation) to change the language
of devices, and even do it again with the same test runs.
The problems layouts (e.g. fonts behaving bad) can be
easily picked from screenshots. Real mobile devices can
be also easily switch languages so all possible languages
are testable with the exact same game content.
Functional testing is probably the most common method in mobile game testing. Typically
functional testing is associated with manual testing and playing ‘game through’. However,
to be agile, all functional testing should be automated. Functional testing – with help of test
automation frameworks – requires some basic understanding of programming and setting
up environment for testing. Automated functional testing can reveal issues related to user
interface (and graphics), stability, game flow/mechanism, and integration of graphics assets.
MORE ADVANCED TEST METHODS FOR MOBILE GAMES
Today, one of the hottest method for mobile game testing with lots of graphics (e.g. OpenGL ES)
content is the use of image recognition. It can solve the well-known issue of instrumentation
not being able to recognize UI elements based on IDs, descriptions and other typical handles
that work with majority of test automation frameworks.
Share this eBook!
CHAPTER 3
The Value of Testing in Mobile Game Development Process
Share this eBook!
Crossing the Mobile Game Development Chasm
T he fact is that mobile game testing is very different compared to the regular app
testing. Much of the mobile game testing is still done manually and that doesn’t
provide an efficient approach to get everything properly covered. In fact, effective mobile
game testing should derive from a well-structured and systematic approach, use of test
automation framework(s) and seamless integration with your agile process.
Share this eBook!
Furthermore, many of test automation frameworks have been originally built for native
apps (or web apps/elements) and one of the biggest misconceptions among mobile game
developers has been that test automation frameworks do not comply to game testing
– where certain native UI elements, their IDs, characteristics etc. cannot be right away
identified. However, this isn’t really the case and many of those frameworks provide
excellent ways to testing mobile games, e.g.using image recognition.
When it comes to testing,
the simplistic view of it is
that bugs are identified
and documented so that
developers can remove
those bugs. Therefore some
metrics traditionally used
for software testing – such
as number of bugs found
and number of test cases
created – are great but do not show the value-add of testing and because of this, QA is
hardly seen as a productive and well-spend effort in organizations. However, due to the
capability of instant feedback from gamers through App Markets, concerns about user
retention have changed the way in which serious mobile game developers think about
testing. Bad ratings and feedback lead to low number of downloads and eventually causing
the game to be a rogue investment.
Does crash reporting help then? Absolutely yes, but that only provides you the information
that your mobile game doesn’t work, crashes or users experience something else bad
– and that’s already too late. Your game has been exposed to users, negative feedback
is publicly available and fixing an issue doesn’t always help. In some cases, bad ratings/
reputation can even prevent getting next titles approved, or for sure not getting that title
in front of hundreds of millions of gamers.
THE THREE FACTORS FOR IMPROVED VALUE
The value-add of testing brought into development process can be also seen through
effectiveness, speed up development (improved productivity) and possibility to release
your title sooner (time-to-market). All these ‘improvements’ can directly be measured
in the terms of value:
Effectiveness of your QA on finding and development on fixing the defects. This is the
no. #1 cost driver in today’s mobile app and game development. As said, direct costs of
defects have a direct effect on your bottom line, like the actual cost of fixing the defects
from testing process, cost of fixing and verifying crash reports coming from the market,
and lost customer acquisition costs (CPI) and lost lifetime value of the customers (LTV,
affecting top-line).
Share this eBook!
Effectiveness very oftenly is easy to be neglected as a cost driver, but when you put
your figures honestly on the table, you will find that late bug-fixing is consuming a lot of
developers’ time as well result in delaying the release, which then equals to lost revenue
and lost customers.
Time-to-market cost directly affecting your top-line. Time is a crucial attribute not only
against competition, but also to gain revenue as soon as possible. When you launch your
game, every day counts as you start generating revenue earlier. The important metrics are
monthly/daily acquired users (MAU/DAU) as well as average revenue per (daily active)
user (ARPU, ARPDAU). Remember that the delays with publishing aren’t coming because of
testing – if your development process accommodates true agile development and testing.
Many game developers do not calculate the impact of delays, but frankly, it is quite easy to
calculate impact of each day of delay to your top-line. Manual testing – even with mobile
games – will delay the launch even more. The solution is the agile, test automation driven
development with continuous integration, continuous testing and quality assurance on
real end user devices, all the time during development.
Productivity of your QA. This is the most obvious one affecting your bottom line. The easiest
way to further analyze the productivity is to compare the costs of manual testing efforts to
cost on test automation tooling & automation costs. While this gives you only the financial
cost estimates, it is also important to consider other aspects: the common approach with
mobile game testing, unfortunately even still today, is that all of that important QA and
testing work gets done at latest phases of the development process.
Share this eBook!
HOW TO ADD VALUE TO GAME DEVELOPMENT PROCESS?
Effectiveness. The easiest way to tackle the challenges related to effectiveness is to integrate
game development and testing process into cohesive agile development process. This
enables you to automate huge number of different types of tests, using real mobile devices
(all sorts of those) and check code after every build (daily/nightly/weekly builds). The
effectiveness has three following attributes and advantages:
1. Instant feedback on defects. This is widely documented benefit of agile practices
like continuous integration and testing. Mobile
development brings the additional dimension with
automating the process on real devices your end
users are using. From developers point of view,
the instant feedback on defects increases the
productivity dramatically, since you still have the
code you recently touched fresh in your mind. Who
remembers what you had for lunch 2 weeks ago vs.
today?
2. Effectiveness in defect finding prior to the
release will lead to fewer field failures. According
to Crittercism’s research, the gaming apps are the
least robust and have the highest crash rate of any
app types. Let’s take an example: a mobile game
with 50,000 monthly active users make 2,200 crash
reports every month. If you have 3 games in the market, that would be 6,600 crashes per
month. If the analysis-fix-verification takes around 1h, the cost is huge. Not because of
engineering costs, but cost to the reputation, given-up users and potentially lost revenue.
Share this eBook!
Proactive testing on real mobile devices before the release would have improved and
made the game more stable – and therefore the development effectiveness would have
been better by factor of 2-4. Post-launch monitoring is naturally still crucial, but you better
not be in reactive mode what come to defect finding, or it will cost you a lot.
3. Lost CPI and LTV. You know how much each install costs you, and you probably are
aware of the LTV of your users. If your game fails after a release it is highly likely that you
will lose both the CPI as well as LTV of that user/all users using the device model. It is
highly recommended to plug in your CPI + LTV for each crash coming from the field to
understand the impact of the failures.
Being proactive, and using true Agile development process
with continuous game testing approach on real
mobile devices, will lead to higher top-line
and highly improved bottom line as well as
happier developers: Every developer wants
to work on the new features rather than
fix existing features in fire-fighting mode
based on the crashes coming from the
market. Remember! Manual testing is
not agile, and it is not an answer for
those mentioned aspects. You do not
get instant feedback. If you want to
test your game manually on top 50-100
devices for example, the cost of doing
that bi-weekly or even monthly is high, and
practically impossible. You should do it daily
or nightly.
Share this eBook!
Time-to-Market. Shorter time-to-market drives up your top-line. The earlier you launch,
the earlier you start generating revenues. When you release your new game – or old game
with new monetized features – 2 weeks earlier than previously using agile combined with
automated mobile testing, that is 14 days shaved off from every release cycle. If you do
6 releases per year (every 2 months), it is 84 days saved annually.
The maths for lost revenue is really simple:
Suppose 100,000 daily average users (DAU)
Suppose $0.05 average revenue per daily active user (ARPDAU)
14 days (x) 6 release/year (x) 100 000 (x) $0.05 = $420 000/year/game more to your
top line.
If you have a portfolio of 5 games, total would be around $2M from faster time-to-market.
You can achieve all this by shaving off 2 weeks in every release cycle. Not all games are
directly revenue generating, so replace ARPDAU with your specific value for each mobile
user.
Again, time-to-market is improved with integrated development and testing process.
This is the case for all software development, but in mobile app dev it is the only way to
do development efficiently. There is a huge number of end-user platforms (e.g. different
devices with different hardware and software setups), that your games should support
and ever increasing complexity in games that need to be tested on those devices in every
release.
You can automate huge amount of testing, and do it on real devices after every new code
checking and/or nightly builds. This will shorten your development-testing cycle, improve
defect-fixing times and result in improved top line.
Faster time-to-market also keeps your competition at bay. Your competitors will not
get a chance to promote their offerings over yours, which is always a danger when your
Share this eBook!
release starts to slip. You can release more often with more confidence in the quality of
your releases.
Productivity. The quality assurance (QA) is a critical part of the process when creating
and maintaining successful mobile games for hundreds of millions of gamers. One of
the metrics in this game creation process is naturally QA’s productivity – the success to
efficiently find and filter out the problems in games before they land on hands of users.
This type of productivity of quality assurance can be measured by various metrics, for
example:
Bugs found rate – how efficiently the QA team finds bugs during the testing process per
time used on testing
Post launch issue rate – how many issues are found after the go-live release
Customer satisfaction rate – for example complaints to customer support, issues reported
to app stores, overall ratings of the games
Quality of deliverables and issue documentation – how easy it is for developers to
understand the issue and make a fix based on the documentation
The number of new test cases added per period of time
The challenges unique to mobile game development & QA are device diversity, different
operating system versions, hardware configurations and OEM/carrier customizations.
Those aspects increase the complexity of QA and result in longer development cycles
(time-to-market), higher fields crash rates (lower customer satisfaction and uninstalls
hurting your top line) and higher customer support costs (more customer complaints) and
inefficient development team (dealing with customer issues vs. developing new features).
In order to improve the QA productivity, teams need to have access to real devices – the
very same ones that end-users use.
Share this eBook!
Developers and QA teams need an access to those devices exactly when they face an
issue during development, they are fixing post-launch issues or get customer complaints.
The earlier issues get found, the lower the cost of fixing them. To maximize bug-finding
efficiency, you can automate testing, and do it on real devices after every new code
checking and/or nightly builds. This will shorten your development-testing cycle, improve
defect-fixing times and result in improved bottom line.
When using the cloud based service for mobile development and QA, the performance
of your team will improve a lot: The cost is fraction of the manual testing, developers are
happier (instant feedback on new code, no need to deal with purchasing when acquiring
new devices), product owner is happier (higher customer loyalty, higher LTV, faster time-
to-market), customer service is happier (less customer complaints) and most important of
all, your customers are happier as they do not get frustrated by “not working on my device”.
Share this eBook!
There are several practical ways to add value to existing mobile game development
process. If you focus on mentioned three items and follow the agile process with continuous
testing, you’ll very quickly gain benefits and be able to build more robust games. Here is
a check-list to move forward with better quality:
1. Consider automating most of the testing, choose any of the excellent open source
testing frameworks available for the most popular mobile platforms
2. Avoid totally unnecessary vendor lock-in by using standard languages (Java, dotnet,
perl, php, python, ruby), there is a pool of millions of developers using these languages
to tap into
3. Automate as much as possible of the testing to improve the organizational agility (even
make developers responsible for creating automated tests)
4. Use continuous integration and continuous testing, testing is a continuous process in
agile and will deliver you results 24/7
5. Integrate with your test management systems to improve the collaboration and
transparency throughout the organization. Tools must enable a free flow of collaboration
6. Instant accessibility. The development teams do not have time to engage with long
sales cycles, procurement processes for each tool and device. Choose a service with
rapid provisioning without waiting for hardware, procurement or internal processes.
7. Finally, the most important one: Tools matter, but how you use them matters more.
Tools make a difference, but how you use them will ultimately determine your level
of success.
Share this eBook!
ConclusionB uilding a successful mobile game
starts with understanding how
intuitive and great gameplay can be
enabled for gamers. Making sure that
users understand that they are fully
capable to use all great features (and
graphical stuff) you’ve built in your
mobile game makes a difference. The
usability, user interfaces, graphics assets,
OpenGL ES content and implementation
are actually those that form the playing
experience. And eventually, when
gameplay is great and gamers stick
with it, you’ll see that as an increased
potential to monetize that enthusiasm.
By implementing the essential infrastructure and architecture, you will be able to make
sure that you are testing for the real environment and real gameplay. Then adopting agile
process in your mobile game development and testing would bring the greatest value
to your daily tasks. In the end the value of agile testing is far more than making your life
easier - enabling your company to deliver the continuous quality of app and generate
continuous revenues.
Share this eBook!
Thank you for reading this
e-book!
To learn how agile process can help you boost your mobile game development and testing. Check out Testdroid Cloud.
Upload your app and run the test on thousands of devices -
You’ll know the agility we can bring to you.
Drop us a line about your thoughts at [email protected] and we will provide you with
the expertise and everything we know to help you make the right decision.