best practices blueprint for growing and large scale continuous testing at the speed of devops

Large Scale DevOps CT

DevOps Summit New York June 2015

Hello everyone, my name is Marc Hornbeek. I have been involved in DevOps long before it was a

word. For over 35 years my career has centered on product development tools, process

automation and shaping organizations for efficient product change management which is, in my

opinion the heart of what DevOps is all about. My particular interest is optimization and

automation of testing.

Today I am Senior Solutions Architect at Spirent Communications and I represent Spirent’s

CLEAR DevOps solution for our global customers. Spirent is a global Test and Measurement

solutions provider for network systems. We help enterprises with their testing and test

automation challenges.

This presentation provides a blueprint for continuous testing for growing and large scale

DevOps environments.

I hope you will seen something here of value for your own situation whether your own DevOps is

large scale or not.



You will see in this talk that continuous testing for large scale DevOps environments presents

challenges and pitfalls that can be avoided if you have a blueprint to follow.

As a company with multiple business units that have employed DevOps and as a provider of

DevOps solutions for our customers we learned a lot about DevOps practices. We developed

opinions about “Best Practices” for DevOps and even developed a Best Practices Assessment

Tool that we used to calibrate the DevOps level of other Spirent Business Units and our

customers have also found this tool useful.

One thing we discovered early on is that the continuous testing aspects of DevOps is especially

challenging because DevOps is all about speed while testing is all about thoroughness of testing.

Getting thorough testing, at speed is tricky. This is the area that this talk emphasizes most.

The best practices and the Assessment tool is available to any organization that would like to

use them.



So what constitutes large scale DevOps anyway? Large scale indicates the complexity of the

DevOps infrastructure, which tends to correlate to the complexity of the product that is being

developed.

To qualify as large scale I set some numbers to give something to compare to.

500+ developers multiple dev sites

1MLOC+, 100+ features

40+ build targets

complex interdependencies

200+ topologies

10K+ TCs

80+ deployment packages (physical and virtual)

1,280,000,000 test possibilities

If your own DevOps environment varies I expect many of the practices in this presentation will

still be of some interest.



Both sides of this equation are uncomfortable.

The VP needs to improve product quality, release frequency, process visibility,

reduce corrective work and reduce costs at the same time. He knows DevOps

can help but is depending on his team to figure out how to scale.

The DevOps manager has experience with smaller scale DevOps but is

uncertain where to get large scale DevOps best practices, and proven DevOps

expertise.

This is true at any scale but is especially true for large scale DevOps.



DevOps is all about SPEED. Fast process cycles, Seamless integrations. Fast

delivery pipelines.

Organizations looking to implement large scale DevOps environments already

have DevOps at some smaller scale.

To scale up DevOps requires all of the process components to accelerate to

very quick cycles and still get all the work done.

But how can thorough testing be accomplished in quicker and quicker cycles?



Done right, CT will accelerate finding important defects early in the development schedule no

matter the scale.

Doing CT right means building the entire test environment for the desired integration and delivery

speeds.

Problems found by testing must be handled in real time and resolved continuously.

As the rabbit surely knows, stopping the continuous processes has negative consequences to

getting to the finish line on time.

Story: *** The major packet switch vendor I worked for initially took 12 months to deliver each

major release using protocol test suites that each took 2 weeks to run and analyze. We

automated and scaled horizontally the protocol testing so test cycles were reduced to hours,

deployment times were reduced to 6 weeks -- a nice 800% improvement. But we didn’t stop

there. The bottleneck was then test analysis times. When the test analysis cycles were

automated the deployment time reduced to 2 weeks – a whopping 2600% improvement, largely

due to implementing continuous testing principles !***



The point is, speed of test execution, alone, is not a sufficient solution.

Millions of tests can not be practically be accomplished in quick cycles.

Using tradition automated testing methods, large scale CT automation will only rapidly

accumulate massive test results that pile up and cause release delays even though many

of the failures are not important.



As the scale of continuous testing ramps up indeed the test cycles must be

faster but to complement that a more sophisticated view of what is

comprehensive testing becomes essential.

And the results also must be handled in a more sophisticated way.

While at smaller scales blasting through large test suites and blasting out

copious test results may be tolerable but at larger scale this does not work.

The work “Relevant” becomes critical.

Only run the most relevant tests.

Only report the relevant results.

To the relevant consumers.

At the relevant time.



At large scale it is essential to have a clear and mature view of quality assurance.

True quality is determined by the dependability of a product when used, rather than a simple

measure of defects. CT is much more than a measurement tool that points out defects early.

Because it is an integral part of all DevOps processes CT is an integral part of product

development, deployment and even post-deployment maintenance.

By integrating test processes deeply into DevOps and focussing CT processes on relevant cases

and results, CT ensures costly quality problems are avoided such as:

• Re-work

• Unsatisfied customers

• Non-compliance

• Bad Press

• Litigation

Quality is a product of development and operations. There is no such thing as a QA department.

Story: *** I recall being surprised myself when I was charged with a major quality study that

clearly demonstrated a nearly exponential inverse relationship between the # of independent

testers in an organization and the rate of customer found detects. It turned out that organizations

with fewer independent testers were doing much more thorough testing using testing tools

integrated directly into the development organization, and this continuous testing was driving

quality so high it was getting more and more difficult to measure quality improvements.

Exceptional quality due to integrated continuous testing was the primary reason the network

system I was involved with became the dominant system throughout the world, and I became the


DevOps Summit New York June 2015 Page ‹#›

youngest international industry quality representative in the history of the company.***


DevOps Summit New York June 2015 Page 10

*** Run this slide as animated ***

This slide is a quick overview of DevOps tools infrastructures and applied to all scale

levels.. DevOps has been described as tool-centric philosophy and in fact it is

really four types of processes orchestrated by different classes of tools that are

executed using physical, virtual and hybrid environments.

1) The first process, called CI ,starts after the software completes a Pre-Flight test prior

to integration to trunk. CI tools orchestrate the software changes, run static and unit

tests, perform basic functional tests and produce build packages.

2) The CT process used tools which orchestrate the test environment including the setup

and configuration of the system-under-test and test and lab management function and

run functional and system tests.

3) The CD process used tools which manage deliverable package approvals, promotions

and publish releases.

4) The CM process uses tools run analytics used to determine code rework or promotion

decisions.

CI, CT, CD and CM tools work in concert to make DevOps work. It is interesting that

although CI and CD are most talked about, CT and CM are vital to DevOps success.

At large scale DevOps C, if not implemented efficiently becomes a serious



bottleneck and impediment to accomplishing business and technical

goals.



In this presentation I will define five CT best practices that must be mastered for

large scale DevOps.

Pre-Flight tactics to ensure the input to DevOps is prepared for large scale

DevOps performance

Test Cycles partitioned in a modular way to keep up with the rapid large scale

DevOps cycles

CT Ready Tools requirements that are essential for scaling DevOps

Acceleration tactics for DevOps CT at high scale

Relevant Results Analytics suitable for high scale DevOps

Although theses are presented in a serial fashion, the implementation may

require changes in multiple areas at once to keep the system in balance as the

DevOps system is scaled up.



Before developers integrate their changes into the integration or trunk branch

due diligence to prevent integration failures is especially important for large

scale DevOps.

All Developers need to have CT training and special skills. A CT architect must

be assigned to ensure CT guidelines for automated test practices are in place

and developers are trained and test peer reviews are in place.

The designs of the software need to be designed for CT.

For example all software functions must be testable at the system level. Unit

level testing is not enough to ensure system performance when integrated.

Automated test cases and evident test results need to be available at the time

code changes are committed to trunk.

Metrics and thresholds must be in place determine whether code changes may

be accepted into trunk. For example more complex code changes or changes to

high risk and error-prone code areas require more automated tests to be

committed together with the code changes.



At small scale it may be sufficient to have a small number of static test suites to

support CI and product regression.

This doesn’t work with large scale DevOps. The total number of tests are either

too large to be executed in the CI cycle times, or the coverage is too low to be

relevant.

The test suites must be broken up into specific cycles that can be triggered more

intelligently depending on the code change and delivery pipeline conditions.

Multiple targeted test suites for CI, Integration, regression and system test

phases will be required and a mechanism to trigger these dynamically when

relevant is essential.

Ideally the test suites can be created from a pool of tests dynamically in

accordance with the relevance of the code changes.



Many existing tools are not CT-ready and can not accomplish best practices

DevOps and won’t scale to work in large scale environments.

This is a high level check-list of the most critical features that are needed in a

CT-ready toolkit.

Restful APIs are essential because CT tools must be able to integrate easily with

everything else and each other.

Features that allow all test activities to be cached and pipelined assure tests are

not waiting around for some resource to be available.

Test tools must be able to run virtualized for horizontal scaling yet optimized for

execution on physical servers when time-critical critical performance testing is

required.

Results reporting should be separated from execution and aggregated vertically

across as many test agents as needed to meet test cycle time SLAs.

Test topologies and configurations need to be pre-defined and pipelined ahead

of the tests.

The test tools need to easy to program all types of tests in a program agnostic

approach that is efficient to capture and create complex tests of all types.

Finally the tools need to be able to scale from simple unit tests to very large



scale systems tests that requires many test agents.



In my experience only the most advanced CT environments are orchestrating

their test topologies but the optimizations afforded by this is outstanding so

amazes me that more are not doing this. Perhaps they are not aware of tools

and workflows that implement this practice?

Do you know the difference between automation and orchestration? I am

referring to the automated management of test topologies and configuration data

system under test and test tools as required for each test case

Best practices test orchestration tools support physical, virtual and hybrid test

resources and their interconnections equally and the entire lab is operated in a

self-service fashion.

This includes the ability to inventory, catalogue, invoke, pre-load, implement and

re-configure systems and networked test topologies in concert with pipe-lined

test schedules.



Since the underlying mechanism that drives success in DevOps is speed of the

integration cycles, it should be clear that faster CT is essential for large scale

DevOps.

Here is a check-list of practices that accelerate the tests, test schedules and

test resources….

Use the most powerful computing resources you can afford and orchestrate

them optimally for the types of tests they are performing

Design the tests to focus on one verdict and get to the highest priority verdict

quickly.

Pre-load and pre-configure the next test while the current is running

Run as many test agents in parallel as you can afford.

Configure the entire system to remove bottlenecks and wait times.

Set thresholds for failures and immediately release resources when triggered.

Aggregate results on separate fast severs.

One thing to keep in mind though….watch out for essential cases that can be

compromised by continuous testing if not integrated properly. For example “

how to deal with long duration testing?, and how to integrate manual testing?

into DevOps also need to be fully considered.



As the scale of CT increases it becomes essential to dynamically configure test

schedules.

There are several techniques that are employed:

1. Assign test cases to source code group changes

2. Assign test cases to build image changes

3. Assign test cases according to changed source code test results.

The third approach is the most effective because it works at the system level, is

statistically relevant and can be used to speed up results analysis also.



CT analytics is more than just reporting pass/fail results.

Best practices CT environments use statistical methods to control the CT

process and also the CI and CD processes.

As DevOps itself scales up, and accelerates the number of test results scales up

rapidly to many thousands of results per cycle. If all of the results need to be

processed manually the value of automated testing is wasted on laborious test

analysis.

So in best practices test analytics are not only descriptive they are predictive –

they automatically keep track of test results trends correlated to software

changes for each case and assign priorities, diagnostics and probable solutions

rather than simply listing the pass/fail results.

I refer you to some papers I wrote on the subject of automated change-based

test selection and results analysis.



CT is more than simply running a smoke test when a build is done. There are

multiple levels of testing invoked as each software change progresses through

the DevOps system. Each level needs to be measured and actions need to be

taken for failed cases according to pre-determined workflows .

Firstly CT system environment stability must be accomplished and constantly

measured or the entire test environment will be a liability rather than a strategic

asset.

Pre-Flight testing metrics determine if the software does not meet its test pass

rate and prevents the code from being integrated to trunk.

Static analysis, unit tests and basic integration tests determine if the results of a

build are good enough for more extensive integration testing, system testing

and regression testing..

The metrics strategy sets thresholds for each test phase that determine whether

failures are significant enough to revert the changes or it is necessary to stop

the DevOps cycles long enough to fix the problems.



The picture says it all. The “handy” dapper business man on the tree obviously decided

he can save some time and money by doing this “simple task” himself on his lunch break

rather than hiring a professional tree trimmer. He appears to have the right tool and the

personal know-how to operate the tool to accomplish an intended result of cutting the

unwanted tree limb, but he doesn’t seem to fully understand all of the best practices so

he is about to experience a serious unintended consequence.

My advise is to be wary when your team tells you they “don’t need help with CT” I am

sure they are smart but before you decide to “do-CT-yourself ask yourself:

• Does your team have intimate knowledge of CT best practices?

• Can you afford the cost of not getting it right the first time?

• Can your organization afford the delays that inevitably occur when they get interrupted

by higher priority product escalations.

• Can you afford the ongoing cost of tools integrations with new tools and maintenance

and enhancements that will surely occur in the future?

Remember those IDC Fortune 1000 report…..last line on chart … 80% fail ! My advice

is think more than twice before you decide to “do-CT-yourself”!



** Animated slide **

Spirent’s CLEAR DevOps Solution is a comprehensive toolkit that embodies all of the

best practices and offers professional services to get it integrated into any customer tool

chain.

At the heart of the solution is a test orchestration system that has all of the features

needed to orchestrate not only topologies and configurations of the system-under-test

and any vendor test tools suitable for both small and large scale DevOps.

Spirent offers wide range of powerful test tools for conformance, performance,

network, security, feature and security testing that play well with Spirent orchestration

tools.

The entire suite of test tools can be operated in physical, virtualized or hybrid lab

environments as needed to scale tests as horizontally as needed for faster test cycles

and as fast as possible to overdrive the world’s faster performance tests.

Through plugins with Jenkins for instance the tools can be easily integrated with any CI,

CD and CM tools that can also play with CI orchestration toolkits.

The solution provides custom dashboards and integrates with ALM systems for

advanced analytics as needed for CT to not get bogged down at the analysis phase.

Finally the solution includes proven expertise and professional services scalable as

needed to ensure the entire system is assessed, implemented according to best

practices and seamlessly integrated into existing customer environments without

disrupting existing releases.



So…by now I have explained many of the practices for large scale DevOps CT.

It is not possible to cover the entire spectrum of possibilities in this one

presentation but hopefully you have gained some new ideas and are curious to

learn more.



We developed a DevOps Best Practices Assessment Tool which reflects our

own experiences, application to other Spirent Business Units and research into

other company experiences.

The tool enables and organization to obtain a measurement for their practices

levels, and a GAP score between best practices and their current practices.

The GAP measurement takes into account the customer’s perspective of

importance for each practice in the Assessment tool.



Continuous Testing Practices is only one part of the assessment.

The tool covers seven DevOps best practice categories including pre-Flight, CI,

CT, CM, CD, System and Team practices.

If you are interested to obtain a copy of this tool feel free to email me at the

address on this slide.



Ok so let me recap the thesis of my talk which hopefully has been made clear by

now.

While it should not be a secret, for some reason many organizations, even

many scholarly publications, have not fully recognized the strategic nature of

getting Continuous Testing right in order for DevOps to accomplish it’s primary

success goals of innovation, quality, time to market, and ROI.

There is much more to CT than just running some tests after a build. A deep

and broad understanding of CT best practices is vital to putting together a

winning DevOps infrastructure.

So we recommend that you don’t fall out of the tree by trying to “do-CT-yourself”.

Indeed it can be penny wise and pound foolish not to engage with experienced

professionals.



I conclude my talk with a call to action for you to thoroughly understand DevOps

best practices, and the GAP between your current practices and best practices.

The DevOps Best Practices Assessment tool that can help you determine your

current practices level and GAP. It is a straight forward spreadsheet your team

can fill out themselves or we would be happy to walk you through it.

We also offer free of charge DevOps Solutions Blueprint whitepaper at

www.spirent.com/solutions/devops or stop by the Spirent booth for a free copy.

Thank-you for your attention and I wish you the best in your DevOps efforts!

http://www.spirent.com/solutions/devops



Thank-you.

If you have any questions feel free to contact me.

best practices blueprint for growing and large scale continuous testing at the speed of devops

Technology