no devops without continuous testing

No DevOps Without

Continuous Testing

Arthur Hicken - Parasoft

1

Your PresenterArthur Hicken is Chief Evangelist at

Parasoft where he has been involved in

automating various software development

and testing practices for over 20 years.

He has worked on projects including

cybersecurity, database development, the

software development lifecycle, web

publishing and monitoring, and integration

with legacy systems and maintains the IoT

Hall-of-Shame http://bit.ly/iotshame and

SQLi Hall-of-Shame http://bit.ly/sqlishame

Follow him @codecurmudgeon

Blog: http://codecurmudgeon.com

Web: http://parasoft.com

http://bit.ly/iotshame

http://bit.ly/sqlishame

https://twitter.com/zeichick

http://codecurmudgeon.com/

http://parasoft.com/

Agenda

• What the shift from automated to

continuous means

• How disruption requires changes to how

we test software

• Addressing gaps between Dev and Ops

• Technologies that enable Continuous

www.iist.org 3www.psqtconference.com

Continuous

4www.psqtconference.com

• Build, Integration, Testing, Release,

Delivery, Deployment

• Builds on DevOps movement

• Enables DevOps reality

• High dependency on automation:

– processes

– assessment

– decisions

DevOps

“DevOps is the practice of operations and

development engineers participating

together in the entire service lifecycle, from

design through the development

process to production support.”

– The Agile Admin

• No more throwing code over the wall

• Design for deployment


DevOps Pipeline


Automated vs Continuous

• Just being automated isn’t continuous

• Scheduling / frequency

• Intermittent manual dependencies

• Decisions points

– Is it done?

– Is it good?

– Is it safe?

– Can I push it out the door?


Complexity a Barrier to Testing

3rd Party System

Evolving ComponentMainframe

Scheduled Access

API Economy

Pieces/Parts come from everywhere

Definitions changing (upgrades, new versions)

Not always available

Can’t test with real data without causing problems


We must Re-invent Test

• The penalty for faulty software is

increasing

– Brand erosion

– All industries at risk for total transformation

• Technology is faster and more distributed

– Containers

– Microservices

Fixed Fixed

Impact of Test in the Evolving

SDLC

Plan Develop Build Test Release

Test as a time-boxed event is the prevailing method

“Are we done testing?”

Impact of Test in Evolving SDLC

Plan Develop

Bu

ildTe

st

Release

Bu

ildTe

st

Bu

ildTe

st

Bu

ildTe

st

Bu

ildTe

st

Bu

ildTe

st

Bu

ildTe

st

Bu

ildTe

st

Bu

ildTe

st

Bu

ildTe

st

Bu

ildTest

“Does the release candidate have an

acceptable level of risk?”

Gaps between Dev and Ops

What is “done”?

Fully testing with external dependencies

Full environments and all their variants


Speed of Deployments

Rapid deployments

Deliver features and bug fixes

Deployments outpace ability to test

Too many changes

Since we didn’t test enough, Devs are mired in bug fixes

Glitches and outages can happen


QA and Testing

• Cannot keep up with the rate of change

• Quality lags behind

• Do we even know what to test?

• Complex technologies including message

protocols and services


Enabling Technologies

• Stubs

• Service Virtualization

• API testing

• Test data management

• Environment management

• Self-service test environments


How to do it better ?

• Give absolute priority to automated tests

• Invest time into designing interfaces (API)

• Use API (service) testing tools to cover

interfaces

• Measure the quality of the test


• Use stubs and mocks to improve:

– Facilitate test automation

– Increase code penetration


• Use service virtualization to improve

testing automation

– Isolate at the message layer

– Simulate functional scenarios and

performance conditions

How to do it better?

Test management

•Environments

•Data

•Dependencies


• The Challenge

– Performance issues found late in the SDLC resulting in large rework

and cost

– Middleware component able to handle large load but available

downstream dependent systems were unable to provide throughput

needed

Reduced testing time by 27-days for a major

commercial bank

Capacity Constrained staged environments

Database

Service

Mainframe

Application

1000 tps

Database

Servic

e

Mainfram

e

Applicatio

n

Reduced testing time by 27-days for a major

commercial bank

1000 tps

Virtual

Asset

Virtual

Asset

Virtual

Asset

Virtual

Asset

• The Solution

– Identified key use-cases and emulate the behavior of dependent

systems. Asset performance setup to match production response values

– API and Load testing tools used to validate system under test. Virtual

assets changed to model different scenarios

• The Business Benefit

– Able to predict performances issues and fix before full Perf testing.

– Able to run different scenarios not easily achieved with production

environment (excessive response times).

– Setup for ‘performance test’ reduced to near zero

– The Challenge

• Multiple teams using the same test database

• Teams not respecting data integrity & others test data records

• Regression tests consistently failing. Takes >1 hour to determine

that it was due to “data changes”.

• “Real problems” were getting lost in the noise

Shared

Databas

e

Eliminated 83% of the configuration time for a major

telecom company

Test data management for complex transactions

✔

✖

✖

Shared

Database

– The Solution

• Setup Virtual Assets to model the SQL queries and use API

testing tool to manage automated nightly regressions against

both virtual assets and live systems

– The Business Benefit

• Test teams able to focus on ‘real regressions’ and separate out

data integrity issues from functional test failure

Eliminated 83% of the configuration time for a major

telecom company

Virtual

Asset

Virtual

Asset

Virtual

Asset

✔

✔

✔

Reduced wait time for test team by 60% for a major media

conglomerate

• The Challenge

– Large agile development effort to adopt Service Oriented Architecture

(SOA)

– High risk project but the Test team “stuck waiting for the first build”

– Development of functionality was not easy to coordinate as different

teams had different schedules; not all finished at the same time

Agile/Parallel development limited by system dependencies

Iteration

Iteration

Iteration

Team A

Team B

Team C

Iteration Iteration

Iteration Iteration Iteration

Iteration Iteration

Current

Development/Testing

dependencies

Reduced wait time for test team by 60% for a

major media conglomerate• The Solution

– Use descriptions of the new services (WSDL, XSD, example JSON

payloads) to build Virtual assets prototyping the new functionality.

– Test team builds tests with against the prototypes with API testing tool

and the independent development tests use the prototypes to perform

early stage Integration Testing

• The Business Benefits

– Met business goals and timelines, were able to test functionality “as

soon as” it was available. Practiced TDD against prototypes to get a

head-start on ‘full system testing’

Iteration

Iteration

Iteration

Team A

Team B

Team C

Iteration Iteration

Iteration Iteration Iteration

Iteration Iteration

✔✖✔

✖✔✖

Extended testing hours for a stock trading broker

• The Challenge

– Leverages 3rd party (out sourced) system for trade exchanges

– Many teams need access, so time is scheduled into 4 hour time

windows, and could only test during market hours

Capacity Constrained staged environments

Database

Servic

e

Mainfram

e

Applicatio

n

Extended testing hours for a stock trading broker

• The Solution

– Use Test environment manager tool to setup ‘Test Silos’ where each

team can do 24x7 testing against virtual assets

Database

Mainfram

e

Applicatio

nEnvironmentManager

Conclusions

Emulates dependencies for the Test Environment

• Reduces complexity for early stage testing

• Increases predictability

On-Demand “Test Anytime, Anywhere”

• Automated Provisioning for different use-cases

• Automated Test Data Management/Simulation

Reduce time to configure for test

Realistic reusable assets

Reduce hardware costs and system access fees

Q&A

• Web– http://www.parasoft.com

• Blog– http://alm.parasoft.com

– http://codecurmudgeon.com

•Social

–Facebook: https://www.facebook.com/parasoftcorporation

–Twitter: @Parasoft @CodeCurmudgeon

–LinkedIn: http://www.linkedin.com/company/parasoft

http://www.parasoft.com/

http://alm.parasoft.com/

http://codecurmudgeon.com/

https://www.facebook.com/parasoftcorporation

http://www.linkedin.com/company/parasoft

no devops without continuous testing

Software