devops summit 2015 presentation: continuous testing at the speed of devops
Post on 15-Jul-2015
3.647 Views
Preview:
TRANSCRIPT
2Spirent Communications www.spirent.com/solutions/devops
IDC December 2014
Fortune 1000 Survey
Situation
Average cost of failure: >$500K
Application development waste: 25%
Solution
83% using or evaluating DevOps
21.4 % investing in Testing/QA tools
Cautions
Failure rate when using current tools: 80%
3Spirent Communications www.spirent.com/solutions/devops
SVM
Build(s)SUT
Commit
Commands
And
Responses
Source Code
P4, Git, SVN, etc.
Deliver
Dev, CI, QA Labs
Physical, virtual, hybrid environments
Test and Lab Management
Checkout
Images
Test
Info
Results data
Logs
Response
Info
Test I/P Test O/P
Images
AnalyticsDashboards,
ALM, PE
Not Ready Ready
CI/CD Orchestration
Tools
Contin
uous In
tegra
tion C
I
Continuous Test CT
Conti
nuous
Delivery
C
D
Continuous Change Management CCMPre-Flight
Development
Artifact RepositoryImages, tests, configs, logs, results
Software
Changes
5Spirent Communications www.spirent.com/solutions/devops
Continuous Testing means… Innovation
Test strategy requires critical thinking which enables innovation and efficient CT which reduces wasted corrective work, frees time for
innovation.
6Spirent Communications www.spirent.com/solutions/devops
Continuous Testing means… Quality
CT is more than just quality measurement, it is an active part of quality development, verification and deployment.
7Spirent Communications www.spirent.com/solutions/devops
Continuous Testing means… Time-to-market
Build CT for speed, and don’t stop!
8Spirent Communications www.spirent.com/solutions/devops
Continuous Testing means… Return-On-Investment
Penny wise,
pound foolish
CT is a major factor affecting Return-On Investment
10Spirent Communications www.spirent.com/solutions/devops
Continuous Testing Best Practice #1 – Team and Culture
Architect
CT Skills
Test Design Reviews
Workflow Analysis
Collaboration
Training
11Spirent Communications www.spirent.com/solutions/devops
Continuous Testing Best Practice #2 – CT-Ready Tools
Restful APIs
Cache and pipeline
Virtual and physical
Vertical aggregation
Test topologies
Program agnostic
Large scale
12Spirent Communications www.spirent.com/solutions/devops
Continuous Testing Best Practice #3 – Tools Integration
Plug-in
CI Tools
CT Tools
Plug-in
CD Tools
Plug-in
CM Tools
Restful APIs
Restful APIs
Documented
Reusable
DevOps ready
Regression tests
13Spirent Communications www.spirent.com/solutions/devops
Continuous Testing Best Practice #4 – Stability and Metrics
Pre-Flight
Static analysis
Unit tests
Integration
Feature tests
System tests
Regression
STOP
REVERT
ALLOW
14Spirent Communications www.spirent.com/solutions/devops
Continuous Testing Best Practice #5 – CT Acceleration
Powerful servers
Test design
Pre-load
Pre-configure
Scale horizontally
No waiting!
Thresholds
Aggregate results
15Spirent Communications www.spirent.com/solutions/devops
Continuous Testing Best Practice #6 – CT Analytics
Test schedules
Test selection
Resources
Test phases
CT
Controls
CI, CD
Controls
Code reverts
Promotions
Releases
16Spirent Communications www.spirent.com/solutions/devops
Continuous Testing Best Practice #7 – Optimize CT Orchestration
Orchestrating both physical and virtual test
resources is important !
Physical
Virtual
Hybrid
Catalogue
Invoke
Pipeline
17Spirent Communications www.spirent.com/solutions/devops
Continuous Testing Requires Expert Solutions
Knowledge of CT best practices
Cost of not getting it right
Delays due to higher Ops priorities
Ongoing cost of tool integrations,
maintenance and enhancements
80% failure rate when using current toolsIDC Fortune 1000 survey, December 2014
18Spirent Communications www.spirent.com/solutions/devops
Spirent CLEAR DevOps Solution
Continuous
Deployment
(CD)
Continuous
Integration
(CI)
Plu
g-i
ns
Plu
g-i
ns
SUTTools
Lab Management
Physical, Virtual, Hybrid Lab
Analytics ALM
6. Expertise and professional services
1. Test orchestration & lab management
2. Comprehensive suite of test tools
3. Physical, virtual and mixed hybrid labs
4. CI/CT/CD/CCM tools integration (EVCI)
5. CT analytics, ALM integration
Orchestration
Continuous Test (CT)
CCM
19Spirent Communications www.spirent.com/solutions/devops
Implementation Strategies – Non-Disruptive Measured Progress
Macro-Phases
1. Assessment to
determine
bottlenecks
2. Proof of concept
3. Horizontal
integration
4. Vertical
deployment
Mini-Phases
1. Team & Culture
2. Tools integrations
3. Stabilize, measure
4. OptimizationsMacro-Phases
Mini-Phases
InitialChanging
Optimized
Micro-Phases
1. Change a little
2. Test
3. Deploy
20Spirent Communications www.spirent.com/solutions/devops
Real World Example – Network Equipment Manufacturer
Metric Before After
Major release (mo.) 6 3
Minor release (wks.) 4 2
# Features 113 150
Defects 1260 10
Integrations / day 0.5 100
Tests / day 0.3 10
Automated tests 5% 85%
The primary contributing factor was CT !
21Spirent Communications www.spirent.com/solutions/devops
Summary
The hidden secret of DevOps
success is CT
DevOps success will only be
accomplished if CT is
approached in accordance with
best practices
Don't fall into the “do-CT-
yourself” trap!
22Spirent Communications www.spirent.com/solutions/devops
spirent.com/solutions/devops
top related