devops 2017 conf: evolving from automated to continuous
TRANSCRIPT
Evolving from Automated to Continuous Testing for Agile and DevOps
DevOps Remote Conference 2017Arthur “Code Curmudgeon” Hicken
IntroArthur Hicken is Chief Evangelist at Parasoft where he has been involved in automating various software development and testing practices for over 20 years. He has worked on projects including cybersecurity, database development, the software development lifecycle, web publishing and monitoring, and integration with legacy systems and maintains the IoT Hall-of-Shame http://bit.ly/iotshame
Follow him @codecurmudgeonBlog: http://codecurmudgeon.comWeb: http://parasoft.com
2© 2017 Parasoft Corporation Perfecting Software
Agenda
Today’s challenges
What IS Continuous?
Barriers
Best practices
Role of automation
Role of static analysis!?
3© 2017 Parasoft CorporationPerfecting Software
Normal Situation
• Pressure to deliver software quickly• Pressure to reduce defect rate• Late stage “bug-hunting”• No time for error prevention• Quality can be rushed• Security is forgotten
4© 2017 Parasoft Corporation Perfecting Software
How to you know a build is successful?
• When it compiles?• When all unit-tests have run?• When the right coverage goal is achieved?• When the right % of test failures occurs?• When it’s deployed?
5© 2017 Parasoft Corporation Perfecting Software
Continuous
• Build, Integration, Testing, Release, Delivery, Deployment• Builds on and enables DevOps• High dependency on automation:
• Processes• Assessment• Decisions
6© 2017 Parasoft Corporation Perfecting Software
Feature Release
Business Stakeholder Developer Customer Support
SoftwareFeature
Continuous measurements mean continuous refinement of the process
Real-time feedback from
Objective assessment and go/no go Defects are eliminated at the point
that they are easiest to fix
Quality gates: Organizations can automatically promote software through quality gates when business expectations have been met
7© 2017 Parasoft Corporation Perfecting Software
Continuous Testing is Beyond Automation
Challenges of Continuous
• Tests must produce binary decision go/no-go• Reuse unit test and functional tests from dev to QA• High level of automation• Requires disciplined mature process• Testing must automatically answer
• Is it stable• Will it do what it’s supposed to do
8© 2017 Parasoft Corporation Perfecting Software
Importance of Testing
• How do you know when you're done? • How do you know if a fix for a minor bug broke a major function of the
system? • How can the system evolve into something more than is currently
envisioned? • Testing, both unit and functional, needs to be an integrated part of the
development process.
9© 2017 Parasoft Corporation Perfecting Software
Continuous Testing re-positions the question from “are you done testing?” to “is the level of risk understood and accepted?”
ContinuousTesting
RiskAssessment
PolicyAnalysis
RequirementsTraceability
TestEnvironment
Access
TestOptimization
AdvancedAnalysis
Ensure access to complete test environments
Automate defect prevention andPolicy measurement
Expand test coverage and measure test effectiveness
Define actionablepractices
Connect functional with non-functional requirements
Process improvementopportunities
10© 2017 Parasoft Corporation Perfecting Software
Elements of Continuous Testing
Prerequisites
Tools
• Version control• Build server• Deployment server• CI tools• Automation tools (test, etc)
Process
• Commit/update often (each change)• Always create tests (pass and fail)• Test regularly• Run regularly
11© 2017 Parasoft Corporation Perfecting Software
Continuous requires:Proper infrastructure
A binary definition of “done”
Reliable quality gates
Extreme automation
12© 2017 Parasoft Corporation Perfecting Software
Infrastructure
• Real requirements system• Flexible developer level project/scrum management system• Automated build/testing system• Data collection that can answer
• Is it done?• When will it BE done?• Is it good enough to release
13© 2017 Parasoft Corporation Perfecting Software
Defining Done
• Are requirements coded?• Do tests exist for the requirements and code?• Are the tests passing
• To your satisfaction (often not 100%)
• Will the tests be enough?
14© 2017 Parasoft Corporation Perfecting Software
Defining Quality
• Consistent• Repeatable• Meaningful
Automated objective measure
• Unit tests• Functional tests• Regressions tests• Static analysis
Made up of
15© 2017 Parasoft Corporation Perfecting Software
Effective Continuous Testing
• Avoid manual end-to-end testing• Invest more into automated tests• Follow testing pyramid rules• Measure:
• test results• test effectiveness
• Determine:• Risk
16© 2017 Parasoft Corporation Perfecting Software
How to do it better ?
• Give absolute priority to automated tests• Invest time into designing interfaces (API)• Use API (service) testing tools to cover interfaces • Measure the quality of the test
17© 2017 Parasoft Corporation Perfecting Software
How Can Static Analysis Help?
Coding standards
Quality gate (bug detection)
Prevention
18© 2017 Parasoft Corporation Perfecting Software
Bug Gates
• Binary Decisions• No “bugs”• No “critical” static analysis findings
19© 2017 Parasoft Corporation Perfecting Software
Quotable Quotes
• An ounce of prevention is worth a pound of cure. – Benjamin Franklin• Cease dependence on inspection to achieve quality. Eliminate the need for
inspection on a mass basis by building quality into the product in the first place. – Deming
• Simply finding bugs with static analysis isn’t enough.• Avoid risky behavior
20© 2017 Parasoft Corporation Perfecting Software
Static Analysis Continuous Feedback Loop
Identify Error
Isolate Root Cause
FixNew Rule to Prevent
Monitor
§ Code review
§ Regression
§ QA
§ Field bugs
§ Post-mortem
21© 2017 Parasoft Corporation Perfecting Software
Main Points for Today
• We must define “all” quality expectations upfront
• We must change the primary goal of testing
• We must accept that testing is only a task in the quality process
• We must be able to measure trust of the process
• We must automate breaking the application
22© 2017 Parasoft Corporation Perfecting Software
Speeding up the conveyor belt does not yield better results…
Modern DevOps must embrace systematic process improvement—Focused on testing
DevOps is Continuous Improvement
23© 2017 Parasoft Corporation Perfecting Software
How to Achieve Quality @ Speed• Business risks drive quality activities• Teams collaborate on risk definition• All team members trained on risks• Acceptance criteria visible and
measured• DevTest team activities prioritized per
risk definition• Results of quality activities are visible
and translated for all levels of business
24© 2017 Parasoft Corporation Perfecting Software
Why ContinuousTesting?
Systematic, centralized decision making to factor business risk into SDLC
Safety net to allow developers to bring innovations to market faster
…and automation to break the application
A feedback system for better trade-off decisions between release scope, time
and quality
Continuous Testing Mitigates Business Risk
25© 2017 Parasoft Corporation Perfecting Software
Enabling Technologies
Stubs
ServiceVirtualization
APItesting
Testdatamanagement
Environmentmanagement
Self-servicetestenvironments
26© 2017 Parasoft Corporation Perfecting Software
How to do it better?
• Use service virtualization to improve testing automation• Isolate at the message layer• Simulate functional scenarios and performance conditions
27© 2017 Parasoft Corporation Perfecting Software
The Test Environment ChallengeTest environment access is outside the control of development and test leaving gaps in the process
Parallel development delays… Need simple, realistic access to dependent components…
Too much time waiting for access…Need reliable test data
Need a realistic testenvironment easy to maintain
Scheduling
Configuration
Access Limits
DependentApplications
Staged Assets
3rd Party Assets Virtual Environments
HyperVisor
App App App
IT Operations
28© 2017 Parasoft Corporation Perfecting Software
• The Challenge• Multiple teams using the same test database• Teams not respecting data integrity & others test data records• Regression tests consistently failing. Takes >1 hour to determine
that it was due to “data changes”. • “Real problems” were getting lost in the noise
SharedDatabase
Eliminated 83% of configuration time for a major telecom company
Test data management for complex transactions
✔
✖
✖29© 2017 Parasoft Corporation
Perfecting Software
• The Solution• Setup Virtual Assets to model the SQL queries and use API
testing tool to manage automated nightly regressions against both virtual assets and live systems
• The Business Benefit• Test teams able to focus on ‘real regressions’ and separate out
data integrity issues from functional test failure
Virtual Asset
Virtual Asset
Virtual Asset
✔
✔
✔
30© 2017 Parasoft Corporation Perfecting Software
Eliminated 83% of configuration time for a major telecom company
Summary
• Continuous * requires mature infrastructure and process• Quality gates must produce a binary answer• Automation is everything• Policy is critical, because automation is critical• Static analysis gets you to the gate
31© 2017 Parasoft Corporation Perfecting Software