reinventing testing devops

Reinventing Software Testing for DevOps

and Modern Application Delivery

By Wayne Ariola

v

Tricentis | Reinventing Software Testing | 1

www.tricentis.com © 2018 Tricentis GmbH. All Rights Reserved

Think back just 5 years ago. In 2014…

• The seminal DevOps book—Gene Kim’s The Phoenix Project—was one year old

• Gartner predicted that 25% of Global 2000 enterprises would adopt DevOps to some extent by 20161

• "Continuous Testing” just started appearing in industry publications and conferences2

• Many of today’s popular test frameworks were brand new—or not yet released

• The term “microservices” was just entering our lexicon

• QC/UFT and ALM were still sold by HP (not even HPE yet)

• Only 30% of enterprise software testing was performed fully “in house”3

• There was no GDPR restricting the use of production data for software testing

• Packaged apps were typically updated on an annual or semi-annual basis and modern platforms like

SAP S/4HANA and Salesforce Lightning hadn’t even been announced

Times have changed—a lot. If the way that you’re testing hasn’t already transformed dramatically, it will soon.

And the pace and scope of disruption will continue to escalate throughout the foreseeable future.

It’s becoming increasingly clear that the path forward requires testing to deliver fast, continuous insight into

an application’s business risk and “release readiness.” However, the scope of what’s needed to achieve that

varies broadly. It might be a matter of correlating data across your best-of-breed test automation toolset to

deliver actionable, business-focused insights on whether a given release is ready to progress into production.

You might need to transform people, processes, and technologies to speed up a decades-old manual testing

approach. Or, perhaps you need to extend your testing practices to cover all the web/mobile UIs, APIs,

mainframes, packaged apps, and other technologies involved in modern business transactions.

This paper explains what’s required across every testing team, explores some of the primary challenges to

delivering the expected level of insight, then provides a quick introduction to how 1600+ organizations

overcome those challenges with Tricentis.

1 Gartner, “Market Trends: DevOps — Not a Market, but a Tool-Centric Philosophy That Supports a Continuous Delivery Value Chain"

2 Alan Zeichick, “Forget ‘Continuous Integration’—the buzzword is now ‘Continuous Testing’,” SD Times

3 Capgemini, Sogeti, HP, “World Quality Report 2014-2015”

v



Continuous Testing: Does the Release Candidate Have an Acceptable Level

of Risk?

“Digital transformation” has elevated software delivery to a CXO conversation—and that includes software

testing. Today’s executives face relentless pressure to deliver innovative software faster than competitors. On

the one hand, testing is all-too-often the roadblock that stands between highly-accelerated Dev processes

and highly-automated Ops-driven delivery processes. But on the other hand, testing is essential for ensuring

that the release doesn’t place the business at risk—undermining the very “customer experience” that digital

transformation is dedicating to enhancing.

How can businesses achieve the optimal balance between speed and risk to deliver engaging customer

experiences faster than competitors? Enter Continuous Testing. Continuous Testing is the process of

executing automated tests as part of the software delivery pipeline in order to obtain feedback on the

business risks associated with a software release as rapidly as possible.

Continuous Testing really boils down to providing the right feedback to the right stakeholder at the right time.

For decades, testing was traditionally deferred until the end of the cycle. At that point, testers would provide

all sorts of important feedback…but nobody really wanted to hear it then. It was too late, and there was little

the team could feasibly do, except delay the release. With Continuous Testing, the focus is on providing

actionable feedback to people who really care about it and when they are truly prepared to act on it.

Contrary to popular belief, Continuous Testing can happen at any point in the application delivery lifecycle. At

some point, the concept of Continuous Testing was inappropriately conflated with the trend of “Shift Left.” To

deliver the right feedback to the right stakeholder at the right time, Continuous Testing needs to occur

throughout the software delivery lifecycle—and even beyond that to production (e.g., monitoring information

from production and feeding that back from the quality perspective). Just as the name indicates, Continuous

Testing involves testing continuously. Simply starting and finishing testing earlier is not, by definition,

Continuous Testing.

Of course, testing continuously in a way that provides the right feedback to the right stakeholders at the right

time is no easy feat. The short answer to why it’s so challenging is that the time available to test is constantly

decreasing while the complexity of what we need to test is increasing. But it’s more than that. To better

understand why “reinventing testing” is now so essential, let’s take a quick look at some of the core forces

influencing modern application delivery.

Decentralization of Architectures and Teams

System architectures have oscillated from a centralized to a much more decentralized set of integrated

systems. From the monolithic mainframe applications of the past, we’ve evolved into a set of integrated

systems that are becoming significantly more distributed. Cloud-native applications, the advent of

microservices which require an extreme degree of orchestration—this is all part of the decentralization trend

in system architectures.

v



The Development team structure has traditionally followed the system architecture trends. Development

team structure went from large centralized waterfall type teams to much smaller and more decentralized

teams.

v



As Development teams decentralize and grow more fragmented, it becomes increasingly difficult to maintain

control over epics and deliverables. Just imagine: instead of unified groups working on monolithic

applications, we shifted to many small autonomous groups each working on smaller parts of a highly-

distributed system. How do you ensure a coordinated and compelling user experience across the

overarching application?

Now, teams are oscillating back a bit: they’re not necessarily looking for ways to centralize the Dev team

structure, but rather to collaborate more effectively within the decentralized Dev team structure.

Then there’s the Test team structure—following the same general pattern, but lagging a bit behind.

The Test team structure has moved from a highly-centralized group responsible for all testing, to testing

centers of excellence, to digital testing centers of excellence, to testers being embedded within Agile Dev

team structures. Now, Test teams (like Dev teams) are shifting toward more of a mixed model that’s largely

decentralized, but has centralized components available to test end-to-end interactions.

v



Proliferation of Tools

One impact of this team decentralization is that different teams select different tools to drive their quality

processes. That selection of independent tools might fulfill the needs of individual teams—but what happens

when you need to make a fast (often automated) release decision regarding the broader application, which

probably involves multiple teams and toolsets?

Such a decision requires an accurate assessment about how the latest changes impact the user experience.

However, the data required to understand that change impact is scattered across different teams and tools—

open source, custom built, and commercial.

Highly Interdependent Applications

Next, let’s shift to the extreme interdependency that inevitably stems from decentralized system

architectures. Obviously, this interdependency is going to be a much more pressing pain for a QA team

responsible for end-to-end business transactions that involve SAP, legacy apps, mainframes, web UIs, APIs,

etc. than it is for a DevTest team building microservices for cloud-native apps. For the former, you can’t just

grab an open source tool, play around with it, and succeed. You need to understand all the different

technologies well enough to automate them, you need a way to feed secure, reliable compliant data into (and

across) the technologies, and you need on-demand access (real or simulated) to all the different components

associated with each of those transactions.

However, implementing test automation is only the start. Assume you have a series of transactions that might

involve Applications A through E—and you have automated regression tests built out for these core

transactions. Now imagine all the times that one of the many associated application components changes.

For each change, you need to review and potentially update all of the associated regression tests.

It’s just a matter of time before the results become ridden with false positives and the test suite fails to detect

critical failures or negative change impacts. As the application components continue to evolve, technical debt

accumulates and teams feel that an oppressive amount of work is required just to catch up (let alone keep

up). At this point, there’s certainly a temptation to fall back on manual testing. Yet, no matter how many

people you throw at the problem, there’s simply no way that manual testing can deliver the right feedback to

the right stakeholder at the right time in a modern delivery process.

v



Decentralization of Performance Testing

Finally, let’s look at one more impact of decentralized architectures and teams: decentralization of

performance testing. This shift has naturally led to the proliferation and expansion of APIs, web services, and

microservices. A performance issue in any of these distributed components could have a ripple effect across

the entire application. And now that new functionality is being released weekly, daily, or hourly, each

decentralized team needs instant insight into whether their incremental changes negatively impact

performance.

In-depth load testing by performance testing specialists remains critical—but it doesn’t provide the level of

fast, on-demand load testing that’s critical for Agile and DevOps. Developers and testers need a way to

expose critical performance issues before new functionality progresses through the delivery pipeline. To

achieve this, they must be able to easily create load tests that provide fast feedback on the functionality

they’re evolving. Beyond that, they must also be able to execute and scale those load tests as needed—

without the exorbitant costs and efforts traditionally required to establish, configure, and maintain a

performance test lab.

Tricentis, the Cloud’s #1 Continuous Testing Platform

Every team is different. Some might be focused on automating traditionally manual processes while others

might be wrestling with the orchestration and correlation of all the various test automation tools they’ve

come to master. The challenge is getting to the point where you can report on whether an overarching

application or project involving all these different teams—with different cadences, architectures, tool stacks,

structures, and challenges—has an acceptable level of risk.

That’s why Tricentis provides a Continuous Testing platform which is optimized for all the various challenges

that modern QA and DevTest teams face. Start by tackling your most pressing pains, then easily expand as

expectations shift and new challenges emerge.

Tricentis helps enterprise testing teams:

• Expose change impacts in minutes with advanced, resilient test automation optimized for 150+

technologies

• Modernize testing across SAP and packaged apps with the most comprehensive testing solution for

the intelligent enterprise

• Accelerate release cycles by orchestrating and scaling testing efforts across your teams, projects,

applications, and tools (including open source)

• Reduce risks with centralized visibility/traceability, predictive analytics, and “release readiness”

dashboards

v



Tricentis features several core components that can be applied individually or in concert across your teams

and projects:

If you’re intrigued and would like to learn more about the platform and its specific components, we

encourage you to read Gartner’s independent analysis, get a free trial, or sign up for a quick demo.

https://www.tricentis.com/resources/gartner-critical-capabilities-for-software-test-automation/

https://www.tricentis.com/software-testing-tool-trial-demo/

https://www.tricentis.com/software-testing-tool-trial-demo/

reinventing testing devops

Documents