Methods of Video Game Testing by Chris Walden 08002952

Interface and Platform Development MEng Computer Games Design

1 Introduction With the vast advances made in video game production, the practice of video game testing has evolved along with it. What was initially a job that programmers would do themselves after writing new code has become its own job due to the complexity of current games. While most developers will keep dedicated QA staff in house and hire temporary employees near crunch time, some developers will opt to hire an external company to find errors. However, even with leaps and bounds in ensuring that the QA tests and testers are rigorous, bugs and glitches will still slip through the net. Games will frequently make use of online services to bring players patches when these issues come to light, but there needs to be a better way of finding these problems before the game goes up for sale.

2 Discussion So how is it that companies can strengthen the testing process? Bugs are often brought to light by players talking online in forums, or by video game journalists reporting on the matter, if the bug or glitch is significant enough. This can bring a lot of negative press along with it, especially if the problem relates to whether or not you can complete the game. One such bug was seen recently in Fallout: New Vegas (2010), where some players experienced getting stuck in buildings (Ashcraft, 2010). The game would not allow them to exit the building and continue the game, therefore rendering any progress up to this point void. If players hadn't kept extra saves they would lose their entire progress, as the game automatically saves upon entering a building. This could happen anywhere in the game, with players losing upwards of thirty hours of play time. Most of these issues, including severe "save corruption issues" were discovered just a day after the games release (Ashcraft, 2010). A month after the release of The Legend of Zelda: Twilight Princess (2006), many news sources warned players to "beware of a save-game glitch that makes the game impossible to finish" (Jackson, 2007). Unlike games such as Fallout: New Vegas that are released on platforms that have the ability to download and apply patches to games post-release, Twilight Princess did not have this option. The glitch only came into effect if the player saved and quit the game in a particular one-off area, so most people will never encounter it. However, a large number of people did encounter it, a game breaking issue that older games would resolve my actively replacing the game cartridge or disc. Such an example was MineStorm (1982) on the Vectrex, which would encounter a bug that wouldn't allow progression past level 13. Players that filed a complaint with the developer "would be sent a bug-free version of the game (Minestorm/II) free of charge" (Goldberg, 2009). While Twilight Princess was in the public eye, no similar offer was made. Not all post-release issues are the fault of bugs and glitches, as some can be attributed to design oversights. One such example is present in Final Fantasy X (2001) during a boss fight with a monster called 'Overdrive Sin'. The boss monster is flying and cannot be attacked via physical contact, so only ranged attacks and magic are effective (Roberts, 2003). However, due to the customisation options included in the game and being able to favour some characters over others, the magic and range proficient characters may not be strong enough to beat this boss. There is also a save point before this boss, meaning that if an unsuspecting player were to save here and not have an adequate team, the game is essentially rendered unbeatable. There are also similar examples in both of the Baten Kaitos games (2003, 2006).

Considering the speed in which bugs and glitches are found once a game has been released to the general public, it is obvious that testing methods need to evolve further. To be able to do this, it is necessary to dissect the current, widely used techniques used in video game testing. The majority of game developers will have staff dedicated to game testing, otherwise known as members of quality assurance (Gupta, 2011). They will play the game from the earliest of builds, listing any and all issues with the game for the programmers and designers to look at. Game effecting bugs, lockouts and other glitches are the main problems that are documented at this stage, but it also includes design choices. This could be related to in-game textures, level layouts or even item placement. When the game reaches the latter stages of development, the studio will either hire temporary testers to wring out any remaining issues, or hire an independent company to take the code and play it privately (ChaYoWo, 2011). There are numerous issues with this approach. For example, testers that see the same game so often will start to become used to certain features, which will contribute to a tester not seeing some glaring issues. This will largely be resolved when the temporary testers are hired, but this can set back release dates. If the release schedule is strict due to publisher obligations, some lesser bugs may be resolved post-release in order to save time. There is also minimal contact with the intended audience. While a game cannot cater for everybody, it is the gamers and fans alike that will spot important issues post-release. Even a small team of testers comprising of fans of the genre of game they are testing may not necessarily pick up all the issues. An increasingly popular method of testing is holding an open beta. Many popular first-person shooters, such as Battlefield and Call of Duty, will use these to smooth out game play issues, find bugs and also promote the game (Sterling, 20111). The input of thousands of gamers is certainly useful information, but as is expected, most gamers will participate to play the game. There needs to be a structure in order to gauge as much data as possible from those playing the game. Of course, some gamers will talk about the game in forums and other places online, but only a small portion will. As it is, players are not required to write feedback, so this is an area worth looking into. It is understandable that players will not want to work through a survey, so offering them rewards for filling these in is a simple way to tempt them into doing so. Closed-beta testing is sometimes employed, most commonly in massively multiplayer online games. Developers such as NCSoft have done this when developing new expansions in order to have experienced players give much required feedback (NCSoft, 2007). Clans of high-level players will be offered the chance to play early builds and criticise design features in order to make the game more appealing. It's a method that caters to the existing user base, as well as help improve the game for the new players an expansion will bring in. While the developers aren't obliged to follow a players advice, it can be important in case something that may seem trivial is changed that the fan base themselves find valuable. In some instances, developers will look at bringing regular players into their offices in order to get a hands on opinion and critical analysis of their new game. A popular method of doing this is by offering week-long work experience sessions. The developer Lionhead uses this regularly as a cost-free way of getting extra input on a title, as well as having them work in QA (Lionhead, 2011). More recently, users have been called in post-release in order to resolve issues that are a little too complex. An example of this occurred following the release of Uncharted 3 (2011), where some returning players complained about the gun mechanics. A select few of those finding this an issue were invited to the Naughty Dog

studios in order to find a resolution to the problem. According to Patrick Klepek, "Naughty Dog didn’t have a policy for bringing fans to the studio, but that may change" (2011).

3 Creating an effective pre-release testing structure Looking at the previous examples of missed bugs and glitches, it is certain that public involvement in testing is a potential way forward. Of course, the few errors that are encountered post-release are found due to the sheer number of players. Thousands upon thousands will play a title at release, but it only takes one to find a bug and speak out about it to make it public. While having the user base find a glitch post-release presents a resolvable situation, having the developer fix it pre-release is hugely favourable. Methodology for this already exists. The ever-popular 'open beta' events that many games receive are a great way of wringing out issues that remain, whether this is unintentional errors or simply design based issues like level layout or balancing (Yaster, 2011). The problem with this is getting the users to actively give feedback. No one is willing to fill out a questionnaire after a heated fire fight in a popular first-person shooter. By adding simple benefits to users that do indeed deliver feedback, a situation that benefits both parties is created. For example, if it were indeed a popular, upcoming first-person shooter that was being tested, the game would display a short four to six character code upon completion of a round. If the player then enters this into a promotional website and fills out a short but to-the-point questionnaire, this player will receive an in-game title or piece of apparel for his or her character that will be visible to others during gameplay. This simple idea of symbology has been tested and proven with many other games. One of these is Halo 3 (2007), where good players would be awarded special 'recon' armour, which signified their prowess to other players (Truta, 2007). Team Fortress 2 (2008) is another game that does this. By completing particular objectives, unlocking achievements or even purchasing certain games, players will get exclusive 'hats' that they can dress their avatar as (Devore, 2010). It's a popular system that other players can use to gauge what the player behind the character is like. Offering an exclusive item of this description in return for vital feedback is a good way to secure a large number of applicants. Another significant downfall of beta testing is that it is rather limited, as most will only look at the multiplayer aspects of a game. Most games receive downloadable demos, but these only show off a specifically polished portion of the game, and again, there is no feedback for the developers. While giving out longer demos is both threatening to sales (why buy a game when you can download most of it for free?) and would entail large download times, there is another solution to the problem. In order to generate both active feedback and excitement for an upcoming title, the OnLive platform can be explored. OnLive is a cloud gaming service that streams a game to your computer, which bypasses the need for a PC to have the correct specifications to run a game. Button inputs are transmitted to the computers owned by OnLive, which is executed and represented in video with no visible lag (OnLive, 2011). By getting players to complete levels on this service, it ensures the user doesn't need to download anything large, prevents users from keeping a near complete game that could prevent them purchasing it later, and it also promotes the game by allowing players to experience it before the official release. OnLive already operates using a timed-demo system, which can be used to prevent players completing the game entirely before release (Hornshaw, 2011). By having a flagging system, players who encounter problems in the game can simply raise the alarm, where the developers can watch

the streams and receive instantaneous feedback. Testing incentives can also be offered as a way of attracting players to the OnLive platform, as well as being able to test and later play a game. Google+ is a social medium that can be utilised when receiving feedback for work-in-progress games, whether this is before or after the release of a playable model. Using the conference call system, developers can set up a means in which to talk to fans about the upcoming game (Mitchell, 2011). It can be used as a way of pre-empting issues with in-progress games, catering the game to an audience or simply a hands-on method of receiving feedback after playing a demo build. Being able to talk to a member of the development team directly is itself a means to lure players into giving feedback, as the knowledge that your word has influenced the game is a powerful tool. It's a method that helps receive more feedback during testing, as well as acting as a promotional tool. Another method of utilising the in-house testing team is to create a rotational plan. By splitting the team into two groups, with one working on the latest build and the other working with the community testing, it becomes more effective for them to spot errors. There is always the risk that a game becomes too familiar when testing, so it becomes harder to spot the errors. By rotating, testing is still being carried out, though by 'refreshing' one team with a week or so out of hands-on testing, they can come back to the latest build with a fresh look and be able to offer different advice. It is also possible to keep a few members on in-house testing permanently as a way of checking that the rotational members advice is sound and valid.

4 Creating an effective post-release testing structure Monitoring forums, including those outside of the developers own moderated forums, can be an extremely worthwhile method to gain feedback and bug reports. When players encounter a bug in a game, the chances are that they will then turn to a popular forum, such as GameFAQs, to see if anyone else has encountered their issue and found a workaround (GameFAQs, 2011). Many members will then create a series of posts describing their problem and how they encountered, either to see if other posters have encountered the same thing and rectified the issue, or to warn others of a problem. There are three benefits to checking these: 1) Gaining an absolutely voluntary bug report; 2) Easing player frustration by mentioning that the bug is being worked on; 3) Gaining player support by associating with the community. As a bonus to this, communicating to online news sources about upcoming patches, with a list of changes, allows players to know exactly what is getting changed and fixed. In the case of Mortal Kombat (2011), players retorted to their 'blind updates', which would install and alter in-game stats without the player being aware (Holmes, 2011). In the case of Bethesda, they hold a policy of not fixing popular 'fun' bugs if the community wishes they stayed (Sterling, 20112). By having games send information back to the developers, early problems can be spotted and resolved quickly, thus avoiding a consumer backlash. With key statistics such as time spent immobile, enemy spawns and genre specific features, it can be possible to determine what multiple players are experiencing. If there are correlations between the data sets, then the testers and programmers can look into what is causing it, and if necessary, fix it.

Analysing reviews of the game in question is also a good way to discern problems. Major complaints can be resolved if they aren't major design decisions, and if they are, releasing a fix via additional levels in downloadable content could be the answer to drawing more people to the game. These errors, regardless of if they are acted upon during development for the game at hand, can be recorded and used in the development of the studios next game or sequel. Judging the audience reaction to the review is also important, as it helps show if the problems are an isolated case or not.

5 Conclusions After exploring different testing processes, both existing and theoretical, it comes to light that audience participation is paramount to ironing out bugs and glitches in game. The balance will of course come from trying to catch them before the release itself. Bugs and glitches that make it past the dedicated testers are discovered due to such a large player base, with probability dictating that sooner or later a player will come across an error, should there be one to find. By opening up testing to the general public in a monitored and secure fashion, more glitches and bugs will be discovered, the developers can remove them before the public release and the testers get to enjoy having hands-on access to an unreleased game, as well as gaining promotional benefits.

6 Further Work When observing testing techniques that involve the target audience, it is definitely evident that there are plenty of marketing opportunities. Not only can a player help with the active development of the game by pruning it of bugs and glitches, but they can then feel more of a reason to buy the game upon release. Marketing and community specialists could be sought out to maximise the benefits found in external testing sessions.

7 Bibliography

a) Video Games DICE. (2011). Battlefield 3. [DISC] XBOX360. California: Electronic Arts. Infinity Ward. (2011). Call of Duty: Modern Warfare 3. [DISC] XBOX360. California: Activision. Obsidian. (2010). Fallout: New Vegas. [DISC] XBOX360. Maryland: Bethesda Softworks. Square. (2001). Final Fantasy X. [DISC] Playstation 2. Tokyo: Square. General Consumer Electric. (1982). MineStorm. [CART] Vectrex. Unknown: General Consumer Electric. Nintendo EAD. (2006). The Legend of Zelda: Twilight Princess. [DISC] Wii. Kyoto: Nintendo.

b) Websites Ashcraft, B. (2010). Fallout: New Vegas Has Bus Issues [Online]. Available from: http://kotaku.com/5668552/fallout-new-vegas-has-bug-issues [Accessed: 26 November 2011] ChaYoWo (2011). Game Testing [Online]. Available from: http://www.chayowogames.com/game_testing.html [Accessed: 08 December 2011] Devore, J. (2010). Breaking news on the Team Fortress 2 hat front [Online]. Available from: http://www.destructoid.com/breaking-news-on-the-team-fortress-2-hat-front-182364.phtml [Accessed: 08 December 2011] GameFAQs. (2011). Weird glitch in chapter 5 [Online]. Available from: http://www.gamefaqs.com/boards/989995-corpse-party/61254737 [Accessed: 08 December 2011] Goldberg, M. (2009). Vectrex - 1982-1984 [Online]. Available from: http://classicgaming.gamespy.com/View.php?view=ConsoleMuseum.Detail&id=20&game=12 [Accessed: 8 December 2011] Gupta, N. (2011). Hire dedicated software testers [Online]. Available from: http://technology.ezinemark.com/hire-dedicated-software-testers-7d2ed0b6bd13.html [Accessed: 8 December 2011] Holmes, J. (2011). Do you want developers 'fixing' your games after release? [Online]. Available from: http://www.destructoid.com/do-you-want-developers-fixing-your-games-after-release--199271.phtml [Accessed: 26 November 2011] Hornshaw, P. (2011). OnLive Demo Lets You Play the First 30 Minutes of Red Faction: Armageddon [Online]. Available from: http://www.gamefront.com/onlive-demo-lets-you-play-the-first-30-minutes-of-red-faction-armageddon/ [Accessed: 8 December 2011] Jackson, M. (2007). Critical glitch plagues Twilight Princess [Online]. Available from: http://www.computerandvideogames.com/155673/critical-glitch-plagues-twilight-princess/ [Accessed: 26 November 2011] Klepeck, P. (2011). How Smart Uncharted Fans Changed Naughty Dogs Game [Online]. Available from: http://www.giantbomb.com/news/how-smart-uncharted-fans-changed-naughty-dogs-game/3816/ [Accessed: 26 November 2011] Lionhead. (2011). Students [Online]. Available from: http://lionhead.com/Students.aspx [Accessed: 8 December 2011] Mitchell, J. (2011). Google+ Hangouts Are Now Free Conference Calls Too[Online]. Available from: http://www.readwriteweb.com/archives/google_hangouts_are_now_free_conference_calls_too.php [Accessed: 8 December 2011]

NCSoft. (2007). Closed Beta Testing for Richard Garriott’s Tabula Rasa™ Starts [Online]. Available from: http://us.ncsoft.com/en/news/press-releases/closed-beta-tes.html [Accessed: 8 December 2011] OnLive. (2011). ONLIVE TECHNICAL FAQ [Online]. Available from: http://www.onlive.com/support/performance [Accessed: 8 December 2011] Roberts, B. (2003). How to defeat Sin in Final Fantasy X [Online]. Available from: http://www.ehow.com/how_6140876_defeat-sin-final-fantasy-x.html [Accessed: 8 December 2011] 1Sterling, J. (2011). Battlefield 3 beta arrives September 29 [Online]. Available from: http://www.destructoid.com/battlefield-3-beta-arrives-september-29-211864.phtml [Accessed: 8 December 2011] 2Sterling, J. (2011). Skyrim will not fix bugs... if they're fun [Online]. Available from: http://www.destructoid.com/skyrim-will-not-fix-bugs-if-they-re-fun-210386.phtml [Accessed: 8 December 2011] Truta, F. (2007). Bungie Awards Halo 3 Recon Armors if Your Game is Funny [Online]. Available from: http://news.softpedia.com/news/Bungie-Awards-Halo-3-Recon-Armors-if-Your-Game-is-Funny-69429.shtml [Accessed: 8 December 2011] Yaster. (2011). Bug and Glitch Reports Thread [Online]. Available from: http://community.killzone.com/t5/Killzone-3-Open-Beta-Feedback/Bug-amp-Glitch-Reports-Thread/td-p/620645 [Accessed: 8 December 2011]