Continuous Testing

At the Speed of DevOps Marc Hornbeek

Sr. Solutions Architect

2Spirent Communications www.spirent.com/solutions/devops

IDC December 2014

Fortune 1000 Survey

Situation

Average cost of failure: >$500K

Application development waste: 25%

Solution

83% using or evaluating DevOps

21.4 % investing in Testing/QA tools

Cautions

Failure rate when using current tools: 80%

3Spirent Communications www.spirent.com/solutions/devops

SVM

Build(s)SUT

Commit

Commands

And

Responses

Source Code

P4, Git, SVN, etc.

Deliver

Dev, CI, QA Labs

Physical, virtual, hybrid environments

Test and Lab Management

Checkout

Images

Test

Info

Results data

Logs

Response

Info

Test I/P Test O/P

Images

AnalyticsDashboards,

ALM, PE

Not Ready Ready

CI/CD Orchestration

Tools

Contin

uous In

tegra

tion C

I

Continuous Test CT

Conti

nuous

Delivery

C

D

Continuous Change Management CCMPre-Flight

Development

Artifact RepositoryImages, tests, configs, logs, results

Software

Changes

4Spirent Communications www.spirent.com/solutions/devops

5Spirent Communications www.spirent.com/solutions/devops

Continuous Testing means… Innovation

Test strategy requires critical thinking which enables innovation and efficient CT which reduces wasted corrective work, frees time for

innovation.

6Spirent Communications www.spirent.com/solutions/devops

Continuous Testing means… Quality

CT is more than just quality measurement, it is an active part of quality development, verification and deployment.

7Spirent Communications www.spirent.com/solutions/devops

Continuous Testing means… Time-to-market

Build CT for speed, and don’t stop!

8Spirent Communications www.spirent.com/solutions/devops

Continuous Testing means… Return-On-Investment

Penny wise,

pound foolish

CT is a major factor affecting Return-On Investment

9Spirent Communications www.spirent.com/solutions/devops

Continuous Testing Best Practices

10Spirent Communications www.spirent.com/solutions/devops

Continuous Testing Best Practice #1 – Team and Culture

Architect

CT Skills

Test Design Reviews

Workflow Analysis

Collaboration

Training

11Spirent Communications www.spirent.com/solutions/devops

Continuous Testing Best Practice #2 – CT-Ready Tools

Restful APIs

Cache and pipeline

Virtual and physical

Vertical aggregation

Test topologies

Program agnostic

Large scale

12Spirent Communications www.spirent.com/solutions/devops

Continuous Testing Best Practice #3 – Tools Integration

Plug-in

CI Tools

CT Tools

Plug-in

CD Tools

Plug-in

CM Tools

Restful APIs

Restful APIs

Documented

Reusable

DevOps ready

Regression tests

13Spirent Communications www.spirent.com/solutions/devops

Continuous Testing Best Practice #4 – Stability and Metrics

Pre-Flight

Static analysis

Unit tests

Integration

Feature tests

System tests

Regression

STOP

REVERT

ALLOW

14Spirent Communications www.spirent.com/solutions/devops

Continuous Testing Best Practice #5 – CT Acceleration

Powerful servers

Test design

Pre-load

Pre-configure

Scale horizontally

No waiting!

Thresholds

Aggregate results

15Spirent Communications www.spirent.com/solutions/devops

Continuous Testing Best Practice #6 – CT Analytics

Test schedules

Test selection

Resources

Test phases

CT

Controls

CI, CD

Controls

Code reverts

Promotions

Releases

16Spirent Communications www.spirent.com/solutions/devops

Continuous Testing Best Practice #7 – Optimize CT Orchestration

Orchestrating both physical and virtual test

resources is important !

Physical

Virtual

Hybrid

Catalogue

Invoke

Pipeline

17Spirent Communications www.spirent.com/solutions/devops

Continuous Testing Requires Expert Solutions

Knowledge of CT best practices

Cost of not getting it right

Delays due to higher Ops priorities

Ongoing cost of tool integrations,

maintenance and enhancements

80% failure rate when using current toolsIDC Fortune 1000 survey, December 2014

18Spirent Communications www.spirent.com/solutions/devops

Spirent CLEAR DevOps Solution

Continuous

Deployment

(CD)

Continuous

Integration

(CI)

Plu

g-i

ns

Plu

g-i

ns

SUTTools

Lab Management

Physical, Virtual, Hybrid Lab

Analytics ALM

6. Expertise and professional services

1. Test orchestration & lab management

2. Comprehensive suite of test tools

3. Physical, virtual and mixed hybrid labs

4. CI/CT/CD/CCM tools integration (EVCI)

5. CT analytics, ALM integration

Orchestration

Continuous Test (CT)

CCM

19Spirent Communications www.spirent.com/solutions/devops

Implementation Strategies – Non-Disruptive Measured Progress

Macro-Phases

1. Assessment to

determine

bottlenecks

2. Proof of concept

3. Horizontal

integration

4. Vertical

deployment

Mini-Phases

1. Team & Culture

2. Tools integrations

3. Stabilize, measure

4. OptimizationsMacro-Phases

Mini-Phases

InitialChanging

Optimized

Micro-Phases

1. Change a little

2. Test

3. Deploy

20Spirent Communications www.spirent.com/solutions/devops

Real World Example – Network Equipment Manufacturer

Metric Before After

Major release (mo.) 6 3

Minor release (wks.) 4 2

# Features 113 150

Defects 1260 10

Integrations / day 0.5 100

Tests / day 0.3 10

Automated tests 5% 85%

The primary contributing factor was CT !

21Spirent Communications www.spirent.com/solutions/devops

Summary

The hidden secret of DevOps

success is CT

DevOps success will only be

accomplished if CT is

approached in accordance with

best practices

Don't fall into the “do-CT-

yourself” trap!

22Spirent Communications www.spirent.com/solutions/devops

spirent.com/solutions/devops