21/05/15

1

Continuous Delivery &

The Changing Role of the Tester

Dr Kelvin Ross ANZTB 2015, Auckland 22 May 2015

2

Disruptive Technology

21/05/15

2

3

Disruptive Technology

4

Disruptive Technology

21/05/15

3

5

Disruptive Technology

CLOUD SOCIAL

BIG DATA MOBILE

DISRUPTIVE FORCES

21/05/15

4

Biggest Disruptors The Last 5 Years? Agile

Offshoring Virtualisation

8

Disruptive Forces

Customer Testing

DevOps

Big Data

21/05/15

5

10

Startup Corporate

“You can't just ask customers what they want and then try to give that to them. By the time you get it built, they'll want something new.”

Steve Jobs

21/05/15

6

12

14

Scientific Method for Customer Experiments

to Guide Software Development

21/05/15

7

15

FAIL FAST

16

21/05/15

8

17

3 – 6 Months

18

1 – 2 weeks

21/05/15

9

19

“You can't just ask customers what they want and then try to give that to them. By the time you get it built, they'll want something new.”

Steve Jobs

SPEED is the new black

20

21/05/15

10

21

DEV OPS

21/05/15

11

24

No Silos

25

CI CD CD

Continuous Integration

Continuous Deployment

Continuous Delivery

21/05/15

12

Source: Principles and Practices in Continuous Deployment Mike Brittain,Engineering Director, Etsy @mikebrittain mikebrittain.com/talks

Rapid Releases

Source: http://readwrite.com/2014/01/01/three-reasons-your-startup-needs-devops-or-else

21/05/15

13

CD Platform

Source Control

Build System

Pipeline Management

& Visualization

Artifact/Package

Repository

Automated Configuration Management

(IaC)

Test infrastructure

Automated Deployment

System

Source: Continuous Delivery at Scale John Esser, Director Engineering Productivity, Ancestry.com ALM Forum Seattle, 2014

Git Flow

Version & Configuration Management

21/05/15

14

30

Infrastructure

Application Code

Static Data

Test Data Version Everything

31

Data Versioning

Infrastructure/Platform Versioning

21/05/15

15

32

Cloud Computing

New Strategies •  Duplicate test

environments

•  Copy of production

•  Grid based test automation

•  Blue/Green Deployments

•  Resilience testing

33

21/05/15

16

34

Elastic Computing

Time

Which is better?

Source: Principles and Practices in Continuous Deployment Mike Brittain,Engineering Director, Etsy @mikebrittain mikebrittain.com/talks

Release Pipeline

21/05/15

17

36

UNIT

INTEGRATION

SYSTEM

•  Fast •  Isolated •  Aligned with Code

•  Slower •  UI focused •  Later in the processes

•  Focused on linkages •  Harnesses, Mocks and

Stubs

Testing Pyramid – Right Way

Source: Steve Neely, sneely@rallydev.com @stenvoon http://bit.ly/cdpaper ALM Forum Seattle, 2014

Release Pipeline

21/05/15

18

Source: Advanced Topics in Continuous Deployment Mike Brittain,Engineering Director, Etsy @mikebrittain mikebrittain.com/talks

Deployment Gating

Source: Principles and Practices in Continuous Deployment Mike Brittain,Engineering Director, Etsy @mikebrittain mikebrittain.com/talks

Production Testing

21/05/15

19

40

Source: Principles and Practices in Continuous Deployment Mike Brittain,Engineering Director, Etsy @mikebrittain mikebrittain.com/talks

Failure Monitoring

41

FAIL FAST

21/05/15

20

42

FAIL FAST MTBF ✗

43

FAIL FAST MTTR ✓

21/05/15

21

45

Customer Testing

Secured $80m in VC funding

Testing In The Wild Applause

21/05/15

22

47

Elastic Resourcing

Time

Which is better?

48

Customer Testing

21/05/15

23

49

A / B Tests

Source: Advanced Topics in Continuous Deployment Mike Brittain,Engineering Director, Etsy @mikebrittain mikebrittain.com/talks E.g.“Google Analytics Content Experiments”

50

Feature Toggles

21/05/15

24

51

Source: Advanced Topics in Continuous Deployment Mike Brittain,Engineering Director, Etsy @mikebrittain mikebrittain.com/talks

Feature Toggles

52

Source: Advanced Topics in Continuous Deployment Mike Brittain,Engineering Director, Etsy @mikebrittain mikebrittain.com/talks

Feature Toggles Progressive Delivery

To Minimise Risk

21/05/15

25

53

Can We Skip Some Pre-Prod

Testing?

Early Customer Testing

55

BIG DATA Volume Velocity Variety

21/05/15

26

56

How To Test Prediction

57

Big Data Challenges Staging test data sets

Test data privacy Variation of test data sets Production data variability

Controllability & Observability

Algorithm Evaluation Heuristic Testing

Exploratory Testing

A/B Testing Customer Sampling &

Monitoring

21/05/15

27

New QA Skills

CI / CD Process

Data Analytics

Test Environment Provisioning

Production Monitoring Feature Architecture & Deployment Planning

Technical Influence

Automation

Experimentation

Canary Releasing

21/05/15

28

Blue Green Deployments

Feature Toggles

21/05/15

29

Feature Flippers

64

A / B Tests

Source: Advanced Topics in Continuous Deployment Mike Brittain,Engineering Director, Etsy @mikebrittain mikebrittain.com/talks E.g.“Google Analytics Content Experiments”

21/05/15

30

Dark Launch

21/05/15

31

68 In 5 Years???

21/05/15

32

69

Skill Changes

70

DevOps Teams Embedded, Self Organising, Functional

Business Tester Automated Tester DevOps Tester

Non-Technical Functional Tester

21/05/15

33

71

Reduced Test Management

Coordinate Resourcing Practice Support

Consultants Coaches Governance

Test Managers Test Leads

72

•  Crowd Test Coordination •  Customer Test Coordination •  Feature Deployment Planning •  User Experience and Failure

Monitoring

21/05/15

34

73

Q&A