test automation challenges in the gaming industry
DESCRIPTION
Gaming is a multibillion-dollar industry, and good testing is critical to any game’s success. Game testing has traditionally been black-box through the client—a method clearly insufficient with increasingly more complex software incorporating 3D physics, thousands of linked and interacting assets, large databases, and client-server architecture. Automation is an obvious answer, but how do you automate when the user interface is an immersive virtual environment, the data is as vital a part of the software as the code (and actually more likely to create bugs), and the games themselves are often built specifically to prevent automation? Brett Roark describes how Blizzard Entertainment is meeting this challenge by using automation to tame complex asset pipelines and building custom tools that make each tester more efficient. Take away a deeper understanding of the unique complexities of modern game testing, see why they require fresh and creative solutions, and consider how these solutions might apply to non-game testing.TRANSCRIPT
![Page 1: Test Automation Challenges in the Gaming Industry](https://reader033.vdocument.in/reader033/viewer/2022052507/558207e2d8b42aa9498b4f7a/html5/thumbnails/1.jpg)
T6 Concurrent Class
10/3/2013 9:45:00 AM
"Test Automation Challenges in
the Gaming Industry"
Presented by:
Brett Roark
Blizzard Entertainment
Brought to you by:
340 Corporate Way, Suite 300, Orange Park, FL 32073
888-268-8770 ∙ 904-278-0524 ∙ [email protected] ∙ www.sqe.com
![Page 2: Test Automation Challenges in the Gaming Industry](https://reader033.vdocument.in/reader033/viewer/2022052507/558207e2d8b42aa9498b4f7a/html5/thumbnails/2.jpg)
Brett Roark
Blizzard Entertainment
A gaming industry professional with more than fifteen years as a tester, developer, and
producer, Brett Roark has shipped multiple AAA titles as well as numerous casual games.
Before joining Blizzard Entertainment in 2010, Brett had spent more than a decade at Microsoft
as a software design engineer in test. While working on Windows 7, he also taught game design
and production at DigiPen.
![Page 3: Test Automation Challenges in the Gaming Industry](https://reader033.vdocument.in/reader033/viewer/2022052507/558207e2d8b42aa9498b4f7a/html5/thumbnails/3.jpg)
9/6/2013
1
Test Automation Challenges
In The Gaming Industry
©2013 Blizzard Entertainment, Inc. All rights reserved.
![Page 4: Test Automation Challenges in the Gaming Industry](https://reader033.vdocument.in/reader033/viewer/2022052507/558207e2d8b42aa9498b4f7a/html5/thumbnails/4.jpg)
9/6/2013
2
Brett Roark
• Senior Producer, Test
Automation
• 15 years game industry
experience
– Tester
– Developer
– Producer
Agenda
• Introduction
• Why testing games is
challenging
• Which test functions
should be automated
• How Blizzard built an
automation test team
![Page 5: Test Automation Challenges in the Gaming Industry](https://reader033.vdocument.in/reader033/viewer/2022052507/558207e2d8b42aa9498b4f7a/html5/thumbnails/5.jpg)
9/6/2013
3
Blizzard Product Lines
![Page 6: Test Automation Challenges in the Gaming Industry](https://reader033.vdocument.in/reader033/viewer/2022052507/558207e2d8b42aa9498b4f7a/html5/thumbnails/6.jpg)
9/6/2013
4
Core Values
Blizzard “polish” doesn’t just
refer to our gameplay
experiences, but to every
aspect of our jobs. Approach
each task carefully and
seriously. Seek honest feedback
from your peers and use it to
improve the quality of your
work. At the end of the day, no
one will remember if the game
was late – only that it was great.
![Page 7: Test Automation Challenges in the Gaming Industry](https://reader033.vdocument.in/reader033/viewer/2022052507/558207e2d8b42aa9498b4f7a/html5/thumbnails/7.jpg)
9/6/2013
5
Why Testing Games is
Challenging
Software Complexity
• Large development teams
• Long maintenance periods
• Shared technologies
• Computer science problems
– Distributed computing
– 32/64 bit conversions
– Complex math for physics and 3D
graphics
– Artificial intelligence
![Page 8: Test Automation Challenges in the Gaming Industry](https://reader033.vdocument.in/reader033/viewer/2022052507/558207e2d8b42aa9498b4f7a/html5/thumbnails/8.jpg)
9/6/2013
6
Software Complexity
• 3D environments
• Audio and cinematics
• Localization
• Allowing user interface
customization
Multiplayer Interactions
• Adds complexity similar to multi-
threaded vs. single-threaded
programming
• Consistency of experience
• Latency sensitivity
![Page 9: Test Automation Challenges in the Gaming Industry](https://reader033.vdocument.in/reader033/viewer/2022052507/558207e2d8b42aa9498b4f7a/html5/thumbnails/9.jpg)
9/6/2013
7
Data
• Games often data heavy
• Designers solve problems by using
game systems in unexpected ways
• Data source control difficulties
Which Test Functions
Should Be Automated?
![Page 10: Test Automation Challenges in the Gaming Industry](https://reader033.vdocument.in/reader033/viewer/2022052507/558207e2d8b42aa9498b4f7a/html5/thumbnails/10.jpg)
9/6/2013
8
Benefits of Automation
• Increase coverage to
mitigate risk
• Improve process
efficiency
• Find bugs earlier
Content that’s Difficult to
Automate
• Usability
• Graphic style
• Lore and story
• Game balance
• Difficulty
• FUN!
![Page 11: Test Automation Challenges in the Gaming Industry](https://reader033.vdocument.in/reader033/viewer/2022052507/558207e2d8b42aa9498b4f7a/html5/thumbnails/11.jpg)
9/6/2013
9
Areas to Consider
• Clients
• Servers
• Data
• Tools
Client Automation
• Challenges
– Frequent user interface changes
– 3D environments
– Automating around anti-cheat code
• Useful targets
– Client performance
– Compatibility suites
– Regression for expansions
![Page 12: Test Automation Challenges in the Gaming Industry](https://reader033.vdocument.in/reader033/viewer/2022052507/558207e2d8b42aa9498b4f7a/html5/thumbnails/12.jpg)
9/6/2013
10
Server Testing
• Functionality and APIs
• Error injection
• Stress testing
– Performance under stress
– User experience under stress
– Fault tolerance and recovery
Data Validation
• Content pipelines
• Data diffs
• Delivery and patching
• Graphics systems
– Shaders
– Lighting
– Hardware driver updates
![Page 13: Test Automation Challenges in the Gaming Industry](https://reader033.vdocument.in/reader033/viewer/2022052507/558207e2d8b42aa9498b4f7a/html5/thumbnails/13.jpg)
9/6/2013
11
Tools
• Help make QA and/or development
processes more efficient
• Examples:
– Quest tracker
– Arena teams
– In-game bug reporting
How Blizzard Built an
Automation Team
![Page 14: Test Automation Challenges in the Gaming Industry](https://reader033.vdocument.in/reader033/viewer/2022052507/558207e2d8b42aa9498b4f7a/html5/thumbnails/14.jpg)
9/6/2013
12
Blizzard Structure
• Multiple development teams
• Shared platform and technologies
through Battle.net
• QA is a separate division that
supports all development teams
Automation Team Structure
Each automation team consists of:
– An engineering lead
– A producer
– Automation engineers
![Page 15: Test Automation Challenges in the Gaming Industry](https://reader033.vdocument.in/reader033/viewer/2022052507/558207e2d8b42aa9498b4f7a/html5/thumbnails/15.jpg)
9/6/2013
13
Building Momentum
• Hired talented engineers who are
a cultural fit
• Started with Battle.net
• Expanded to products in early
development
• Built tools to increase efficiency
Talent Development
• Internal training team
• Robust onboarding program
• Multi-tiered training programs
– Leadership
– Project Management
– Testing
• Professional certifications
– ISTQB Foundation and Advanced Levels
– Scrum Master/Product Owner
![Page 16: Test Automation Challenges in the Gaming Industry](https://reader033.vdocument.in/reader033/viewer/2022052507/558207e2d8b42aa9498b4f7a/html5/thumbnails/16.jpg)
9/6/2013
14
Summary
• Games have become incredibly complex
• Some content is difficult to automate
• Identify key areas for automation
• Build momentum with initial successes
• Invest in your people
Questions?