Continuous Testing What QA means for DevOps

Eran SherCo-founder & CEO of SeaLights

Nolio (Acquired by CA Technologies)Mercury Interactive (Acquired by HP)

Conduct (Acquired by Mercury)

The State of QA & DevOps

“managing and implementing quality and test measures represent a challenge for teams trying to increase productivity (or release velocity) with acceptable quality”WORLD QUALITY REPORT 2016-2017

IncreasingRelease speed

Decreasing Release quality

Continuous Delivery is becoming the de-facto standard

The Industry Reaction

The Industry Reaction

Organizational People Skills Tools Practices Communication

Now that the hysteria has passed

What QA Means for DevOps

Continuous IntegrationHow we build software

Continuous DeploymentHow we deploy software

Continuous TestingHow we test software

Continuous Delivery

Continuous Testing

Increase quality while increasing speed

Step 1

Understand what is causing quality to decrease when speed is increased

Understanding Test Quality

How do you measure test quality?

Quality Holes

How do you stop untested code changes from reaching production?

Centralized Quality Dashboard

How do you collect quality metrics across environments and tools?

Step 2

Find/ develop a solution for the identified root cause problems

Uncover Code Coverage

Unit Integration Functional UI API Exploratory

? ? ? ? ?

Identify Quality Holes

Scan every build and analyze code changes

Track every test and analyze footprints

Build Quality Dashboard

Quality report per build Collect statuses and coverage across tools, environments and silos

Track executed tests

Step 3

Integrate solutions into CI/CD

Case Study

Our Technical Stack:1. Node.js, Java, and Python

2. Microservices architecture, Mongo DB

3. CI: Jenkins

4. SCM: Github

5. Test automation frameworks: Mocha, TestNG, JUnit, and Py.test

6. Infrastructure: AWS

7. CD: homegrown

Case Study

Our CI/CD Workflow1. Pull request

2. Build + unit test

3. Verify that all tests passed, no decrease in coverage, and no quality holes

4. Integration build

5. Setup and deploy build to integration lab (AWS)

6. Run integration tests

7. Verify that all tests passed, no decrease in coverage, and no quality holes

8. Ready for production » Deploy

Copyright © 2016 Sealights. All rights reserved.

Q&A

Start testing with the lights on

Copyright © 2016 Sealights. All rights reserved.