drive continuous delivery with continuous testing
TRANSCRIPT
1
Keith Puzey
CA Technologies
Drive Continuous Delivery With Continuous Testing
Titus Fortner
Sauce Labs
August 2017
© 2017 CA. Confidential. All rights reserved. 2
Continuous Testing
How do you build better apps, faster?
© 2017 CA. Confidential. All rights reserved.
Why Many Companies Think They’re Agile…
They moved some Dev projects from waterfall to agile
They’re having daily standups
They have a scrum master
Product owner is part of the team
They are all talking and walking agile…
And are talking about Continuous Delivery
BUT…
© 2017 CA. Confidential. All rights reserved.
QA is STILL a Bottleneck… Even in DevOps Shops
http://www.computing.co.uk/digital_assets/634fe325-aa28-41d5-8676-855b06567fe2/CTG-DevOps-Review-2017.pdf
…of delays were occurring at
the Test/QA stage of the
cycle.
“Where are the main hold-ups in the software production process?”
32%16% 22%
63%
21% 23% 30%
63%A 2017 survey of self-
proclaimed DevOps
practitioners found that …
4
© 2017 CA. Confidential. All rights reserved.
Why Quality is So Hard to Achieve?
Clear evidence that you are unlikely to enjoy
the full benefits of continuous delivery unless you
modernize your testing tools and
practices.
75% highlight continuous testing as critical or
important
76% see Continuous Testing as critical or
highly desirable to ‘shift-left’ testing activity from
downstream back into development
Traditional testing tools
are often too complex
and expensive to use
Adoption of
continuous testing is
much easier with
the right tools
Continuous testing
requires a major
shift in mindset
and culture
agree or strongly agree
63%agree or strongly agree
79%agree or strongly agree
72%
“Continuous Testing as a Digital Business Enabler”, 2017
“
5
© 2017 CA. Confidential. All rights reserved.
“Continuous Testing as a Digital Business Enabler”, 2017
Tangible Evidence of the incremental value continuous testing represents for “Leaders”
more likely to have succeeded in
left-shifting testing activity to
requirements phase
more confident in quality of
output
more confident in speed
of delivery
more likely to experience
rapid revenue growth
more likely to deliver 10x faster iterations
more likely to reduce defects
by 50%+
2.5x 1.5x 1.9x
2.6x 2.4x
3.9x
SP
EE
D
QU
AL
ITY
RO
I
6
© 2017 CA. Confidential. All rights reserved.
What is Continuous Testing?First need to understand what it’s not!
It’s not… Test Automation
It’s not… Continuous Integration
It’s not… Unit Testing
It’s not… done only in the Test environment
It’s not… a QA-led activity
7
© 2017 CA. Confidential. All rights reserved.
What is Continuous Testing?
The practice of testing
across every activity in the SDLC to uncover and fix
unexpected behaviors as soon as they are injected
© 2017 CA. Confidential. All rights reserved.
Why are Traditional Testing Activities a Bottleneck?
Enable Interfaces
(real or virtual)
Create Detailed Test Cases
ExecuteTest Cases
Identify Interfaces (internal and 3P)
Automate Tests (usually Regression)
Create Defects Fix Defects
Define an Idea or Customer Needs
Write Requirements
Create Requirements
Traceability Matrix
Read Requirements
Manual Processes
Find/Create Test Data
Testing hasn’t changed in the past 30 years
Identify Test Data Requirements
© 2017 CA. Confidential. All rights reserved.
But Continuous Testing Changes the Game…
Enable Interfaces
(real or virtual)
Create Detailed Test Cases
ExecuteTest Cases
Identify Interfaces (internal and 3P)
Automate Tests (usually Regression)
Create Defects Fix Defects
Define an Idea or Customer Needs
MODELRequirements
Create Requirements
Traceability Matrix
Read Requirements
Find/Create Test Data
Identify Test Data Requirements
NOW ALL AUTOMATED
© 2017 CA. Confidential. All rights reserved.
End-to-End Integrated Continuous Testing from CA
Automatically generate
test automation scripts
from requirements1
2Simulate test
environment
3Access test data on-
demand & ensure PII
compliance
4Start multi-layer /
backend “requests
& response” testing
prior to UI
5Democratize
performance testing
Integrate open source
testing tools 6
7Ensure comprehensive,
cloud-based API testing
8Built-in automated
application security
testing
9Promote artifacts when tests
pass by orchestrating &
automating the pipeline
10
Harness application insight
across the SDLC to
improve user experience
Continuous
Delivery
Director
© 2017 CA. Confidential. All rights reserved.
WHAT IS SAUCE LABS?The world’s largest cloud-based platform for running automated tests
• Co-founded by Jason Huggins, the creator of Selenium; Selenium is the
de facto standard for front-end test automation (Gartner)
• One platform that spans desktop, simulators, emulators and real devices.
• Run tests across over 900 browser/OS/platform combinations & 1,000’s of
real mobile devices
• Enterprise-grade security trusted by thousands of customers
• Optimized for Continuous Testing and CI/CD
© 2017 CA. Confidential. All rights reserved.
ACCELERATE YOUR PIPELINE
Faster Releases = Competitive Advantage
• Test on every build; integrate with your CI Server* and pipeline orchestration
• Run tests in parallel to speed releases
• Test more scenarios per unit of time
• Catch bugs earlier in the development cycle & fix faster
A C H I E V E C O N T I N U O U S D E L I V E R Y W I T H A U T O M AT E D T E S T I N G
*Sauce supports: Jenkins, Bamboo, Travis, Circle CI, MSFT VSTS,
Serial Testing (9 minutes)
Chrome 1 Chrome 2 Chrome 3
FF 2 FF 2 FF 3
IE 1 IE 2 IE 3
Parallel Testing (1 minute)
Chrome 1
Chrome 2
Chrome 3
FF 1
FF 2
FF 3
IE 1
IE 2
IE 3
© 2017 CA. Confidential. All rights reserved.
Quality Code = Quality Experience
• Catch more bugs with increased coverage
• Higher quality keeps users from abandoning
sites and apps
• Increased coverage broadens your addressable
market
• Expand testing to GA / Beta & Dev releases
• Videos, screenshots and logs help speed bug
resolution
IMPROVE QUALITY THROUGH COVERAGEE X PA N D Y O U R T E S T C O V E R A G E W I T H O U T M O R E O V E R H E A D
Increased Quality
© 2017 CA. Confidential. All rights reserved.
IMPROVE EFFICIENCY
Less Time Testing = More Time Innovating
• Enable team to focus on high value activities
instead of infrastructure
• Eliminate time spent on supporting &
maintaining an internal test grid
• Improve productivity of Devs & QE
• Increase confidence in test results
Fewer false failures
Video debugging
R E D U C E C O S T S A N D S P E E D T I M E T O VA L U E
Infrastructure
Costs
False
Failure
Ongoing
Maintenance