Agile, DevOps & Test

Background (me)

• 10 years process control & automation

• too many years Project & Programme management

• 10 years Release, Deployment, Implementation

• quick fact – don’t like ice cream

Synopsis

In no particular order, let’s look at

• the disconnect between traditional SQA & Agile approaches

• how DevOps can be perceived as the ‘mature Agile’ model

Wikipedia says

“Software quality assurance (SQA) consists of a means of monitoring the software engineering processes and methods used to ensure quality. The methods by which this is accomplished are many and varied, and may include ensuring conformance to one or more standards, such as ISO 9000 or a model such as CMMI.

SQA encompasses the entire software development process, which includes processes such as requirements definition, software design, coding, source code control, code reviews, software configuration management, testing, release management, and product integration. SQA is organized into goals, commitments, abilities, activities, measurements, and verifications.”

And the agile manifesto says…

• customer satisfaction by early and continuous delivery of useful software

• welcome changing requirements, even late in development

• working software is delivered frequently (weeks rather than months)

• close, daily cooperation between business people and developers

• projects are built around motivated individuals, who should be trusted

• face-to-face conversation is the best form of communication (co-location)

• working software is the principal measure of progress

• sustainable development, able to maintain a constant pace

• continuous attention to technical excellence and good design

• simplicity—the art of maximizing the amount of work not done—is essential

• self-organising teams

• regular adaptation to changing circumstance

And the agile manifesto says…

• customer satisfaction by early and continuous delivery of useful software

• welcome changing requirements, even late in development

• working software is delivered frequently (weeks rather than months)

• close, daily cooperation between business people and developers

• projects are built around motivated individuals, who should be trusted

• face-to-face conversation is the best form of communication (co-location)

• working software is the principal measure of progress

• sustainable development, able to maintain a constant pace

• continuous attention to technical excellence and good design

• simplicity—the art of maximizing the amount of work not done—is essential

• self-organising teams

• regular adaptation to changing circumstance

ASSURANCE? GOVERNANCE? OPERATIONS?

WHAT’S THAT?

And people think DevOps is…..

of course…………..

Back to earth….

DevOps is about

• bridging the gap between software development and operations

• collaboration – we all want an easy life, right?

• creating capabilities – do it first then help others to do it

• lean delivery – optimum automation

• rapid feedback – are we doing what we said?

• continuous innovation – it’s never good enough

DevOps IS NOT just about technology

• DevOps is about how you use technology to improve process, change culture and drive organisational change

DevOps Enables……

DevOps enables• agile ways of working (but works with Waterfall too!!)

• consistent delivery of code

• process visibility

• software configuration management

• lean governance

By applying• clearly defined principles

• lean ways of working

• automation

• continuous innovation

DevOps Principles

• everything in source control – have a defined branch/merge strategy

• everything as code

• everything is uniquely identified

• build binaries once

• control key environments

• just enough governance and process – delivered through automation

• Security, Test & Quality built in

• path to Production is defined and managed

• we care about process and principles as much as tools and technology

• toolset must be fully integrated

• measure everything

• feedback fast- fail fast, fail cheap!

• automate everything – minimise human intervention – CD is the end game!

Engineering pre-requisites

• understand your stories before you start coding

• don’t bite off more than you can chew

• developers own the code

• design for automation – build, test, deploy

• have a defined branch/merge strategy

• no tests = no code –GIVE ME TEST AUTOMATION!!

• don’t break the build – execute unit & integration tests

• make comments meaningful – reference JIRA tasks

• build once, deploy often – only deploy from artefact store

Then we can build….

And we all live happily ever after

A brief history…..

• Formed in 2012 as part of MCFP

• .com Delivery Function supporting

• ~ 30 Product Teams

• 11 AT&T environments

• ~ 400 AWS instances

• 35 .com components

• Defined & Implemented Group DevOps model

What we’ve achieved in .com

• DevOps toolset & automation provides the only mechanism to build & deliver code to controlled environments

• >600,000 builds with security & quality scanning built in

• >1,400 automated build/deploy jobs

• >20,000 automated deployments

• >4,000 self-serve deployments

• 2 continuous delivery microservices – lost count of the number of deploys!

When we started

• Security were sceptical

When we started

• Security were sceptical

• Engineers accumulated piles of tech debt

• ‘no time to write unit tests…..’

• ‘we’ve got loads of manual testers paid for…………’

• ‘…..and build/deploy engineers’

• Config management – what config management?

• ‘I need to fix code in the environment…..’

• ‘…don’t have time to do test automation….’

• ‘…but I need to know that I’ve got defects immediately so I can fix ‘em’

• ‘…..and the release note tells you everything you need to know…’

Yup, really..

• Security were sceptical

• Engineers accumulated piles of tech debt

• ‘no time to write unit tests…..’

• ‘we’ve got loads of manual testers paid for…………’

• ‘…..and build/deploy engineers’

• Config management – what config management?

• ‘I need to fix code in the environment…..’

• ‘…don’t have time to do test automation….’

• ‘…but I need to know that I’ve got defects immediately so I can fix ‘em’

• ‘…..and the release note tells you everything you need to know…’

• Senior Management didn’t get it

The sell

Build a PoC – AWS is awesome for this

• Make Security your best friend – we hooked HP Fortify into the build process

• Automate the build – show engineers the reduction in effort

• Build in quality scans, run unit tests (??!!??) – rapid feedback

• Automate the deploy – bring the engineers along!

Senior Management don’t get it, so demonstrate

• significantly reduced deployment times

| Jun 2012 (Shopping, no automation) – 1.5hrs build, 14hrs deploy

| Mar 2013 (Shopping, some automation) – build 30 mins, 4hrs deploy with outage

They bought in!

| FYI - Mar 2015 (Shopping full automation) – 15 mins build, 2 hours deploy, zero downtime

Then scale…

• build to your principles – there will be battles, don’t compromise!

• work with the Engineering teams – use tangible benefits to change their mind

• raise need for test automation – need de-prioritised

• demonstrate lean governance through tooling

• answer the who, what, when, how, why questions

• demonstrate ‘build once, deploy often

• metrics identify bottlenecks – in our case, lack of test automation

• raise need for test automation – get a few nods but no action

• give the teams ‘self serve’ capabilities

• demonstrate Continuous Delivery PoC

• raise the need for test automation

And watch the rush…

BDDTDD

Embedded TestersDevs in Test

What’s our test coverage?

What tools?

In honesty, we don’t mind. We just want the experts to give us robust automated test suites that we can hook into our delivery pipe and distribute the results

Auto PT

Baseline

And watch the rush…

BDDTDD

Embedded TestersDevs in Test

What’s our test coverage?

What tools?Auto PT

Baseline

You can tell which developer?

Corp Governance want to look at your SCM process…..

Automated Change process…..?

Not forgetting…

Not forgetting… TEST DATA

Tools deliver scm

JenkinsCI tool

ZabbixAWS monitoring

SonarCode analysis

NexusArtefact/binary management

AnsibleOrchestration/

CM

FortufySecurity scanning

SVNCode Repo

GITLabCode Repo

MongoDBNoSQL DBMS

ApimonAPI test &

monitoring

ElasticSearchLog search tool

KibanaDashboard creation

FluentDLog management

LDAPCloud UAM

OracleDBMS

CentOSLinux distribution

Windows OS ChefCM tool

AntBuild tool

MavenBuild tool

DockerContainer tool

Packer Cloud gold image

creation

VagrantVirtual Dev Env

Creation

GITHubCode repo

QTPTest tool

HPSVService

virtualisation

SeleniumWeb browser test

automation

CucumberBDD Test dev tool

AtomIDE

SourceTreeGIT mgt tool

EclipseIDE

GulpFE build tool

NPMFE build tool

GruntFE build tool

SauceLabsVirtual browser test

Dashing.ioDashboard creation

CloudabilityAWS utilisation

viewer

JIRATask/bug tracking

ConfluenceWiki

FisheyeCode viewer

JIRA captureCapture

screenshots

JIRA AgileAgile boards plug-in

JIRA EnhancerJIRA additional

features

JIRA Agile CardsTicket printing

Intelligent ReportsJIRA reporting

GliffyDrawing plug-in

CruciblePeer code review

JIRA SuiteAdditional features

JIRA ZephyrTest plug-in

AWS SESEmail send service

AWS IAMIdentity access

AWS CloudformationInfra deploy tool

AWS Route 53DNS mgt

AWS CloudwatchMonitoring

AWS EC2VM set-up service

AWS CloudfrontCDN service

AWS S3Elastic storage

AWS VPCVirtual private

cloud tool

AWS RDSManaged RDBMS

AWS tool

£££’s

Open Source

Key

WE LIKE TOOLS REALLY…

BTW We built this….

Summary

DevOps can be considered ‘Mature Agile’

• we care about the full SDLC (cradle to grave)

• we want to deliver light touch governance through our tooling – actually we like the challenge

• we want to drive best practice through automation

• get a kick out of demonstrating the ‘art of the possible’

And we really do get the need for maximum automated testing available as early in the lifecycle as possible

• And we shout about it. Loud.

Thank you and goodnight!

www.QualiTestGroup.com

Thank you!