Continuous Testing for CTOs: How Zenefits Deploys 3x Day

Laks SriniCTO & Co-Founder

Russ Smith CTO & Co-Founder

#continuoustesting

Agenda

● Why do Continuous testing?

● Types of testing

● How do you integrate testing into dev flow?

● How did Zenefits approach this need for QA?

#continuoustesting

Challenges with Testing Today

● Manual testing is an important aspect of comprehensive testing, but it is usually done in huge chunks and takes forever

● Automated testing is brittle, not comprehensive● Most in-house QA teams have very limited bandwidth

https://www.capgemini.com/events/world-quality-report-2015-16-pre-order-form

#continuoustesting

Test Continuously (Shift Left)

Test as early & often as possible for Faster Shipping, Better Quality

● Unit tests ● UI tests (automated vs. manual)

Production monitoring + Rollbacks

#continuoustesting

How does continuous testing fit into the continuous delivery pipeline?

Code Development

Continuous Integration

Staging Production

ContinuousDeployment

#continuoustesting

How does continuous testing fit into the continuous delivery pipeline?

Code Development

Continuous Integration

Staging Production

ContinuousDeployment

#continuoustesting

Test Strategies in Development

Code Development

Staging Production

Usability Testing

Functional Testing

Regression Testing

GUI Testing

Cross Browser Testing

#continuoustesting

Test Strategies in Staging

Code Development

Staging Production

Exploratory

End to end

Usability

Functional

Regression

GUI

Cross Browser Testing

Database Testing

#continuoustesting

Test Strategies in Production

Code Development

Staging Production

Sanity / Smoke Tests

Automated Testing

#continuoustesting

How do you begin testing continuously?

● Identify how comprehensive are your current testing processes?

● Incorporate smaller tests, throughout your development pipeline

● Continuous Integration (CI) - automate regression suites

● Automation alone can be brittle and flaky

Zenefits was able to scale theircustomer base to 10,000+

without an in-house QA teamby supplementing their testingautomation suites with manual

testing.

#continuoustesting

Challenge:

Maintain product quality while growing exponentially.

There was no room for failure.

#continuoustesting

Why did Zenefits approach Continuous Testing?

#continuoustesting

Why did you choose Rainforest?

#continuoustesting

What is the Hub & Spoke model?Unique challenges with this

#continuoustesting

Challenges in SOA?

#continuoustesting

Scalability

#continuoustesting

Focus on Product

#continuoustesting

Ah-ha! moments

How has continuous testing has improved Zenefits?

● Automate the execution of manual testing● Firing off manual tests is just as easy as automated tests, part of the same

process● Now we can confirm results of automated tests and test things in bulk

with a human.● Overcome brittle/flaky automated tests with safety net● Deployment happened when it was ready

Questions?

Laks SriniCTO & Co-Founder

Russ Smith CTO & Co-Founder

Thank you!