Software Testing TrendsA High Level Overview

Purpose

• This document outlines the two major trends in software testing on a very high level

• Far from everything is covered, and this document is simplified to quickly give a broad understanding of what is being discussed in the software testing world

Software Testing Trends Overview

Context-Driven Testing

Microsoft & Google

Context-Driven Testing Principles is Action [10]

• Testing groups exist to provide testing-related services. They do not run the development project; they serve the project.

• Testing is done on behalf of stakeholders in the service of developing, qualifying, debugging, investigating, or selling a product. Entirely different testing strategies could be appropriate for these different objectives.

• It is entirely proper for different test groups to have different missions. A core practice in the service of one mission might be irrelevant or counter-productive in the service of another.

• Metrics that are not valid are dangerous.• The essential value of any test case lies in its ability to provide information (i.e. to

reduce uncertainty).• All oracles are fallible. Even if the product appears to pass your test, it might well

have failed it in ways that you (or the automated test program) were not monitoring.• Automated testing is not automatic manual testing: it’s nonsensical to talk about

automated tests as if they were automated human testing.• Different types of defects will be revealed by different types of tests–tests should

become more challenging or should focus on different risks as the program becomes more stable.

• Test artifacts are worthwhile to the degree that they satisfy their stakeholders’ relevant requirements.

Context-Driven Trends [1,2,8,9,10]

• Testing as a craft– The tester role is put in focus– Methods and practices that support the individual tester in performing better

testing are advocated

• Tests and Checks– Differentiation between checks and tests, where checks work as regression tests,

and tests explore new grounds– Automated tests which are executed are checks and the general view is that

they are mostly overvalued and cannot replace real testing

• Unnecessary artefacts– Test cases, test plans and other test artefacts should be kept to a minimum to

maximize the time a tester spends on actual testing

• KPI and quality assessments– KPI are mostly considered to be of low value, since quality is very hard to

quantify– The quality assessment of the individual tester is put in focus

• Session Based Exploratory Testing– Session based exploratory testing combines the above bullets into a usable

method which is advocated in combination

Testing at Google [5]

“This brings us to the current chapter in test which I call Testing 1.5. This chapter is being written by computer scientists, applied scientists, engineers, developers, statisticians, and many other disciplines. These people come together in the Software Engineer in Test (SET) and Test Engineer (TE) roles at Google. SET/TEs focus on; developing software faster, building it better the first time, testing it in depth, releasing it quicker, and making sure it works in all environments. We often put deep test focus on Security, Reliability and Performance. I sometimes think of the SET/TE’s as risk assessors whose role is to figure out the probability of finding a bug, and then working to reduce that probability. Super interesting computer science problems where we take a solid engineering approach, rather than a process oriented / manual / people intensive based approach. We always look to scale with machines wherever possible.”

Microsoft & Google Trends [3,4,5,6,7]

• Focus on testing as an activity in the development process– Testing is done by many different roles; test engineer, software developer in test, software

developer– Testing as an integrated part of the process, designing software with testability and automation

in mind

• Focus on good automation – Design good automated test cases, don’t try to automate manual test cases– Understand the costs of automation and use automation correctly

• It is not a numbers game– Don’t hire a lot of testers– Maximize the power of the testers you have– Tool support for as many activities as possible to optimize tester performance to spend as

much time as possible on actual testing– Integrated defect reporting, easy risk-management tools, low maintenance test plans, focus

on exploratory testing for manual testing, etc.

• Involving users as early as possible in an organized way– Live User testing as soon as possible

• Continuous integration systems– Use of smart tools and cloud computing infrastructure in the continuous integration system

makes it fast and reliable

Summary

• Two trends with many things in common

• The major differences are – Focus on the tester role vs. focus on testing activity independent of role– Focus on manual exploratory testing vs. focus on maximizing usage of test

automation, starting early with testability built into code– More focus on the individual tester´s qualitative report vs. Continuous

integration system with more quantitative reports – Customer s as requirement stakeholders vs. Customers as live user testers

• The major similarities are– Testing requires qualified professionals – Reduce the time spent on useless artefacts, either by tool support, or by

removing them completely– Mindless automation is not valuable– Customer involvement is important– Everything is risk-based!

Reference

[1] Michael Bolton [Context-Driven Testing]http://www.developsense.com/blog/[2] James Bach [Context-Driven Testing]http://www.satisfice.com/blog/[3] BJ Rollison [Microsoft]http://www.testingmentor.com/imtesty/[4] James Whittaker [Microsoft]http://blogs.msdn.com/b/jw_on_tech/[5] Google Testing Blog [Google]http://googletesting.blogspot.se/[6] Alan Page [Microsoft]http://angryweasel.com/blog/[7] Expert Testers [Microsoft]http://experttesters.com/[8] Jonathan Kohl [Mobile Testing]http://www.kohl.ca/blog/[9] Elizabeth Hendriksson [Agile Testing]http://testobsessed.com/[10] Cem Kaner [Context-Driven Testing]http://context-driven-testing.com/