Continuous Testing: A Key to DevOps Success

Sujay Honnamane

Test Director

Cognizant Technology Solutions

1

© 2015 Cognizant

2

Continuous Delivery is critical for an enterprise

Dev & Test Teams

Operation Teams Gap

Evolving Customer

Expectations Faster Time to Market

Need for higher quality

Changing Business Requirements

Business Agility, performance, Availability and Stability

Leads to Continuous Delivery Challenges…

Lack of collaboration

Infrequent, manual, error prone

deployments Long QA feedback Loop

Mobile Cloud Social Big Data Connected

Systems

Emerging Technologies

DevOps Deploy

Test Continuous Delivery with DevOps

Let’s face it: • Applications that companies deal with

actually contain very complicated business logic implementations than what Facebooks, Twitters and Netflix/es are

• Years of legacy – these applications are out there for more than decade!

• There are patterns adopted by Development, Operations and QA and organization’s expectations revolve around it!

• Agile revolutionized development to the extent – Operations remained in silo and QA remained bottleneck

Image source : https://paimeiitguy.wordpress.com/2008/07/28/a-beacon-of-hope-for-the-keep-it-simple-stupid-way-of-working/

Reality Check – Simplicity is highly overrated by Unicorns!

3

• Traditional Ops – putting out fires, deal with recurring failures & urgent fixes • Test Environment overhead– Each environment is unique by zillion different ways • Several thousands of interdependent regression cases running through UI layer for

several days • Longer test cycles & manual handoffs – much slower feedback • Loosely defined test data management strategy • Downstream/Upstream dependencies • Silos within silos inhibiting cross functional leverage

Image source: http://legacyportfolio.co.uk/blog/2012/06/08/what-are-the-challenges-facing-corporate-occupiers-today/

Challenges for horses today

4

Spot the opportunities & Correct them – One at a time

5

OPS

What in the world?

Dev QA Product owner

• Build failure rate > 60% • OPS : DEV problem , DEV: OPS problem • Avg. time for build deployment > 2 days

Our teams..Chaos – First DevOps Meeting

6

There is a better way! And QA can help!!

Oh Really?

Who cares? Deployment is painful task, these folks won’t ever get it!

QA Architect

QA’s first attempt to break the silos

OPS

Dev

Product owner

7

Someone is making sense!!

No kidding! We work hard to make everyone’s life easy

Product owner

Hmm! We are listening!

A start..constructive discussion – QA gets Dev in alignment

OPS

Dev

8

• QA has greater insights to both Development and Operations side of the worlds

• Deployment delays hurts QA more than any one

• Continuous Integration is first step in right direction – QA and Dev collaboration can make it happen

Can QA be the DevOps enabler? Yes, we can !

QA

9

DevQA

With group of motivated individuals from both QA and Dev

Goal Statement: “Predictable builds, deployments and quality assurance procedure that delivers the working software from developer’s IDE to the hands of business users with click of button.”

And, DevQA journey began!

Image source: http://encontrosdesign.info/how-team-building-exercises-can-help-your-business/

10

WE, the people: Collaborative – Working together to build innovative solutions Open minded - Experiment,fail,learn and Adapt Tool agnostic – solution driven than driven by tool Fearless – Makes problem visible One Team spirit – Combined ownership of problems, no blame Love coding – ruby, python, ant whatever comes our way

DevQA team’s attitude

Image source : http://voices.whitman.syr.edu/student-life/undergraduate/teamwork-makes-the-dream-work

11

SVN Server

Automation Team

Jenkins Server

Slave Agents

Dev / QA Environments

Stakeholders

Continuous Integration – Dev / QA Environments

Updated Automation

Script

Triggers Build Verification Tests Health Checks Smoke Tests Intelligent regression

Re

sult

s

Code Deployments in Dev / QA environments

Accessing Environments

Build verification Results to stakeholders through email

Dev code: New User Stories /

Defect Fixes – Code Commit

Polling the Updated Code

Updated Automation

Script

Dev Team

Automated Build Creation

Automated Security Scan

Unit Test

Smoke Results

Critical Scenario Results

Regression Results

Static Code analysis

Build Quality In

Code Coverage analysis

Instant feedback on

quality

Virtualized services

Production data subset

12

© 2014 Cognizant

Continuous Integration - Critical Success Factors

Inte

gra

te,

Bu

ild

,D

ep

loy

an

d V

ali

da

te

1 2 3 Continuous Testing

4 Service Virtualization 5 Environments on Demand

“Shift Left” QA and Dev synchronization

13

13

Existing full regression running across three different tools took us 50hrs to complete, we needed to build something intelligent that validates relevant stuff for in shortest possible time! Rest of the regression runs over the weekend!

It addresses most common challenges QA organization face today: Legacy automation built on commercial tools over the period of time runs

slow Often, automation is built over altogether in different tools to cover for UI,

APIs, Mobile coverage etc. Business influence over regression definition often push testing team to

build long running tests

Intelligent Regression Model – The key success factor

14

Intelligent regression model – Key considerations

Image source: http://www.signaturelandscaping.ca/

• Dynamic regression scope for each build based on type of code change - Automatically generated release notes feeds in to regression job

• Test cases are traceable to the code files • Intelligent framework capable of turning test scenario

ON/OFF based on application change • Risk based approach with close to zero risk • Centralized repository of code-to- test case traceability

15

DevOps Transformation – QA Levers

Automated Test design through intelligent model based framework Faster feedback through

• Well defined heartbeat tests & deployment verification test • Massive parallelization of tests powered by efficient load

balancing of tests between nodes • Shift from UI based tests to Non-UI based tests • Intelligent test suite

Reduced testing dependency

• Service virtualization • Adoption to more isolated tests over End to end tests • Test Data Management

16

Conclusion: This is Dev Ops !

17

THANK YOU

18