Renaissance Hotel 1 West Wacker Drive Chicago IL 60601

Speaker(s): Ken Robertson & Thomas Janik When: April 21, 2016

Company: American Family Insurance Time: 3:30 pm - 4:30 pm

CHICAGO April 18th — April 22th

DevOps: A Testers Best Friend

DEVOPS;

A TESTER’S BEST FRIEND

Press ESC to continue

TOM JANIK – AMERICAN FAMILY INSURANCE

DEVOPS MANAGER – RELEASE MANAGEMENT

KEN ROBERTSON – AMERICAN FAMILY INSURANCE

SQA TEST MANAGER – SPECIALIZED TESTING

OBJECTIVE OF SESSION:

WALK THROUGH THE AMERICAN FAMILY JOURNEY TO BECOMING A

COMPANY THAT OPERATES IN A CULTURE BASED IN DEVOPS

DEVOPS;

A TESTER’S BEST FRIEND

Deploys done by development teams, could happen at any time

No stability requirements, no triage of deployment / environment issues

4 year development cycle using Amgile SDLC (selective Agile)

Development “Leg” (flow), SIT “Leg”, 2 month effort in Release “Leg”

IN THE BEGINNING: RELEASE FLOW

DEV INT

DEVELOPMENT LEG

DEV INT QA

DEV INT QA PERF PROD

SIT LEG

RELEASE LEG

Manual testing

Testing led by development teams, lacked integration coordination

2 year SIT cycle with multiple code elevations

No formalized automation or performance testing

No shared test data, test scenarios, or documented financial testing

Limited communication between development and testing – throw it over the wall mentality

IN THE BEGINNING: SIT TESTING

Development and Unit Testing

Functional Testing

SIT

Elevation 1

Test Data Creation

Elevation 2 Elevation … n

TDC

LOCAL/DEV

DEV/INT

QA TDC

DEVELOPMENT LEG 1

DEV INT

DEV INT QA PERF PROD

RELEASE LEG

DEVELOPMENT LEG 2

DEV INT

DEVELOPMENT LEG 3

DEV INT

QA

QA

QA

D: Release Coordination, Automation, Performance, and Manual Testing Teams

O: Change Management, Build and Deploy, Software Configuration Management,

Infrastructure Coordination Teams

Specific deployment windows, DENT environments, Floating QA environment

Branching and merging strategies begin, targeted deploys to PERF (testing left)

3 parallel development (“Legs”) streams, Monthly maintenance release “Leg”

FORMING: THE DEVOPS REVOLUTION

Press ESC to continue

Introduction of DENT and QA HIST tiers

Point to point integration testing coordinated by development teams

Application integration coordination being introduced

6 month SIT duration – 3 months financial infrastructure testing and 3 months for financials

Documented financial testing in collaboration with controllers

FORMING: SIT TESTING CYCLESDevelopment and Unit Testing

Functional Testing

AIT (Application Integration Testing)

Phase 1 Phase 2 Phase 3

SIT (System Integration Testing)

Cycle 1 Cycle 2 Cycle n...

Test Data Creation

LOCAL/DEV

DEV/INT/DENT

QA HIST

QA

INT

ACTIVITY: DEVOPS

WHAT’S IMPORTANT TO YOU?

Using the index card in front of you write down 3 things that are most important to you

about DevOps. (1:00 Minute)

Gather in groups of #-# at one of the posters on the wall. (1:00 Minute)

Come up with something common between everyone in the group and write that on

top of the poster as your group name. (1:00 Minute)

Come up with one common thing from everyone’s index cards and draw (no words) that one common thing on the rest on the space on the poster. (1:00 Minute)

STORMING:

A CULTURE CHANGE TO DEVOPS

DevOps is the practice of operations and development engineers participating

together in the entire service lifecycle: from design, through the development

process, and then into production support

Specifically DevOps is characterized by:

Operations staff making use many of the same techniques as developers for their

systems work

Developers incorporating more operational concerns in their planning and coding (designing for operationalization)

DevOps can be a team, a department, an effort, a movement, but most importantly it

is a culture!

KEY TENETS OF DEVOPS

Theory of Constraints (“The Goal”)

The 3 Ways (“The Phoenix Project”):

Systems thinking

Amplified feedback loops

Continuous experimentation and learning

Lean principles of manufacturing (“TPS: Toyota Production System”)

People, then process, then tools (Men, Methods, Machines, Measurements)

Collaborative development (Scrum, Agile, Kanban)

Environment provisioning, quicker access to new code for testing

Continuous deployment, integration, testing, delivery, feedback, and improvement

Plan

Architect

DevTest

Release

Operate

FEEDBACK

THE NEW DEVOPS CULTURE

Release Management

Release Automation

Environment Management

Application Reliability

DevOps

Automation Testing

SpecialtyTesting

Data Team

ServicesTeam

Quality Assurance

Lean

Pri

nci

ple

sR

emo

val o

f Te

chni

cal D

ebt

Rem

ova

l of

Proc

ess

Deb

tSt

anda

rdiz

atio

nV

alu

e St

ream

M

appi

ng

The Automated Delivery Pipeline (DevOps)

Continuous Integration

Version Build Automation Basic Test (incl.

static code quality, security, performance, and code coverage)

Package Deploy to Software

Asset Repository Governance Checks Feedback

Mechanism Infrastructure

Integrations

Automated Testing

Layered Testing (GUI, API, Backend)

Full System Testing (Functional, UX Verification, Regression, Security, Performance, Compatibility, Acceptance)

Environment Sanity Tests Application Resiliency

Testing (Chaos Monkey, etc.)

Non-Functional Sanity Tests

Infrastructure Resiliency Testing

Environment Management

Baselined / Modelled Environments

Test Data Propagation

Automatic Configuration

Infrastructure-as-Code

Automation On-Premises/

Cloud/Hybrid Virtualization Loosely Coupled

Architectures

Production Deployment

Deployment Phased Rollout Rollback Plans APM & Analytics Communication

Feedback Mechanisms

Dashboards/Scorecards

Trends Reports Notifications ALDM Monitor/Measure

Everything Continuous Process

Improvement Common Metrics Measuring Quality Measuring

Efficiency Visible Status

Deployment Orchestration

Automated Deployment

Development Integration Acceptance Performance Production On-Demand

Environment Provisioning

Architecture, Design & Development

DeveloperCommit

Application Instrumentation

Design for OperationsIncl. Graceful Degradation

Microservices

Branching by Abstraction /Toggling

Loosely Coupled Architectures

Culture (Foundational)

Dev in Ops

Collaboration / Break Down Silos

Trust

Collective Ownership

Never Done

Continuous Improvement

Openness & Visibility

Continuous Experimentation

Blameless Postmortem

Ops in Dev Common Goals based on Customer Satisfaction

Planning (Enterprise)

Small Batches

Increased Frequency

Agile Project Tracking

Minimal Viable Product

Build Quality Into the Process

Speed to Defect Resolution

Lean

Pri

nci

ple

sR

emo

val o

f Te

chni

cal D

ebt

Rem

ova

l of

Proc

ess

Deb

tSt

anda

rdiz

atio

nV

alu

e St

ream

M

appi

ng

The Automated Delivery Pipeline (DevOps)

Continuous Integration

Version Build Automation Basic Test (incl.

static code quality, security, performance, and code coverage)

Package Deploy to Software

Asset Repository Governance Checks Feedback

Mechanism Infrastructure

Integrations

Automated Testing

Layered Testing (GUI, API, Backend)

Full System Testing (Functional, UX Verification, Regression, Security, Performance, Compatibility, Acceptance)

Environment Sanity Tests Application Resiliency

Testing (Chaos Monkey, etc.)

Non-Functional Sanity Tests

Infrastructure Resiliency Testing

Environment Management

Baselined / Modelled Environments

Test Data Propagation

Automatic Configuration

Infrastructure-as-Code

Automation On-Premises/

Cloud/Hybrid Virtualization Loosely Coupled

Architectures

Production Deployment

Deployment Phased Rollout Rollback Plans APM & Analytics Communication

Feedback Mechanisms

Dashboards/Scorecards

Trends Reports Notifications ALDM Monitor/Measure

Everything Continuous Process

Improvement Common Metrics Measuring Quality Measuring

Efficiency Visible Status

Deployment Orchestration

Automated Deployment

Development Integration Acceptance Performance Production On-Demand

Environment Provisioning

Architecture, Design & Development

DeveloperCommit

Application Instrumentation

Design for OperationsIncl. Graceful Degradation

Microservices

Branching by Abstraction /Toggling

Loosely Coupled Architectures

Culture (Foundational)

Dev in Ops

Collaboration / Break Down Silos

Trust

Collective Ownership

Never Done

Continuous Improvement

Openness & Visibility

Continuous Experimentation

Blameless Postmortem

Ops in Dev Common Goals based on Customer Satisfaction

Planning (Enterprise)

Small Batches

Increased Frequency

Agile Project Tracking

Minimal Viable Product

Build Quality Into the Process

Speed to Defect Resolution

ENVIRONMENT MANAGEMENT

The Automated Delivery Pipeline (DevOps)

Continuous Integration

Version Build Automation Basic Test (incl.

static code quality, security, performance, and code coverage)

Package Deploy to Software

Asset Repository Governance Checks Feedback

Mechanism Infrastructure

Integrations

Automated Testing

Layered Testing (GUI, API, Backend)

Full System Testing (Functional, UX Verification, Regression, Security, Performance, Compatibility, Acceptance)

Environment Sanity Tests Application Resiliency

Testing (Chaos Monkey, etc.)

Non-Functional Sanity Tests

Infrastructure Resiliency Testing

Environment Management

Baselined / Modelled Environments

Test Data Propagation

Automatic Configuration

Infrastructure-as-Code

Automation On-Premises/

Cloud/Hybrid Virtualization Loosely Coupled

Architectures

Production Deployment

Deployment Phased Rollout Rollback Plans APM & Analytics Communication

Feedback Mechanisms

Dashboards/Scorecards

Trends Reports Notifications ALDM Monitor/Measure

Everything Continuous Process

Improvement Common Metrics Measuring Quality Measuring

Efficiency Visible Status

Deployment Orchestration

Automated Deployment

Development Integration Acceptance Performance Production On-Demand

Environment Provisioning

NORMING: WHO DOES WHAT?

RELEASE AUTOMATIONRELEASE MANAGEMENTQUALITY ASSURANCEAPPLICATION RELIABILITY

Release management, Better defined schedules

2 week release activity cycle, Bi-monthly maintenance releases

Overlapping release cycles, Same branching strategy as development legs

NORMING

DEV INT QA

DEV INT QA

DEV INT QA

DEV INT QA

Prod

Release

P2

Final CodeFreeze

NORMING: QUARTERLY RELEASES

DEV INT QAm

Summer

6 Months

6 Months

2 Weeks2

CTP Code Freeze

FunctionalCode Freeze

Development and Functional Testing

Data Creation

SIT Testing

Performance Testing

Summer

Fall

Winter

Spring

DEVELOPMENT LEG 1

DEVELOPMENT LEG 2

Spring

m m m m m M m m m m m M m m m m m M m m m m m Mm m m m m m

NORMING: FULL RELEASE SCHEDULE

RELEASE LEG

M

Summer

m

Spring

6 Months

2

Program level test lead for coordination of functional and integration testing

Phased / cycle approach to functional integration and SIT testing

10 week SIT duration with data creation critical path constraint removed

Introduction of forced month ends for financial testing

NORMING: TESTINGDevelopment and Unit Testing

Functional Testing

Functional Integration Testing

Phase 1 Phase 2 Phase 3

SIT

Cycle 1 Cycle 3

LOCAL/DEV

DEV/INT/DENT

QA HIST

QA

INT

TDC

Cycle 2

Continuous integration builds

Gated deployments based on testing results

Automated unit testing

Automated smoke testing

Automatic rollback for failed deploys

Environment stability

Quicker, higher quality code delivery

Ability to feature toggle and/or deploy dormant code

Automated deployment notification and release notes

Better merging strategies ease regression testing requirements

Exposure into development process

TESTING BENFITS FROM DEVOPS

Maven for configuration

GIT for code management

CA Harvest for build and deploy

JUnit / GUnit for continuous integration testing

Jenkins for automated deployments

CA Dev Test (formerly Lisa) for automated testing

Word for requirements (moving toward TFS)

Excel for testing plan (moving toward TFS)

Excel release plans

Puppet for IaaS

CA Service Desk for change management and configuration item repository

Artifactory for Enterprise Data Management Solution (EDMS)

NORMING: TOOLS

THANK YOUTOOL APPENDIX FOLLOWS

QUESTIONS?

APPENDIX A: TOOL RESEARCH

http://www.devopsbookmarks.com/

https://xebialabs.com/periodic-table-of-devops-tools/

Tool Category Trial Site

AccuRev (Micro Focus Software) Configuration Management No borland.com

Agile Software Factory (Grid Dynamics) Application Release Automation No griddynamics.com

Amazon ECS Container Management No aws.amazon.com

Ansible (Red Hat) Configuration Management, Application Release Automation Yes ansible.com

Apache Ant Build Management Yes ant.apache.org

Name Description

Ansible  An agent less orchestration engine that can automate application and systems.

Artifactory  JFrog’s open source binary repository management product with support for Apache Maven artifacts.

Asgard  A netflix web interface tool for AWS Cloud Management and application deployment.

AWS  A cloud computing platform offering ready to use web services which are loosely coupled.

Awsbox  A featherweight DYI PaaS on top of awscloud EC2 for deploying node apps.