devops tactical adoption theory: continuous testing
TRANSCRIPT
DevOps Tactical Adoption Theory: Continuous Testing
Berk Dülger – Consultant, Account Manager
TEST CONSULTING TEST OUTSOURCING TEST TRAINING
Learning Organizations
Chapter I
Learning Organizations
Learning organization is one in which people at all
levels, individually and collectively, are
continually increasing their capacity to produce
results they really care about.
Peter Senge
Learning Organizations’ Five Discipline
Systems thinking: Ability to see whole
Personal mastery: The process of life long learning
Shared vision: A vision that shared & committed by
everybody
Mental model: Generalized & assumed view of how the world
works
Team learning: The process of developing the ability to
create the desired results as a team
Systems Thinking
State of Agile/DevOps
AdoptionChapter II
Lean/Agile Adoption in Europe
Global DevOps Rates
Principles and PracticesChapter III
Principles and Practices
Lean is the basis of AgileLean tells you to optimize the end-to-end process which creates value for your customer from the initial idea to collecting cash. Lean principles focus on flow more than anything else: bottlenecks in the process must be removed and wasteful activities need to be identified and avoided.
DevOps is not a goal, but a process of
continuous improvement
Sustainable success requires both bottom-up practices
and top-down management support
DevOps Practices
Practices should address problems like,• Manual Efforts
• Long feedback times
• Long MTTR
• Too much downtime
• Lead time
• Unrepetable work
…
Automation in DevOps
What can be automated?• Dev environments
• Builds on pull requests and merge• Static code analysis• Code style checking
• Dynamic code analysis• Verification
• Archiving artifacts• Deployment
Tactical Adoption TheoryChapter IV
Tactical DevOps Adoption
DevOps Tactical Adoption Theory tries to make the transition process as smooth as possible. It hypothesis each step towards DevOps maturity should bring a visible business value empowering management and team commitment for the next step.
The idea here, it is not required to add the tools/processes to stack from sequential beginning to end, but seeking benefit.
Tactical DevOps Adoption
The reason behind the theory is to encourage practitioners to apply each step one-by-one and then having the benefits in projects. Consequently, each step is tested in terms of utility and proved method validity for the further steps.
Large-Scale Adoption
• Begin with an end in mind• Start with a pilot project
• Make someone/unit responsible• Evangelize, build communities
• Gain executive buy-in• Make people believe
• Drive tool standardization• Automate, automate, automate: Build, Test, Deploy
• Demostrate the value!
Either two ways;
Choose to improve all categories for single project
Or, choose one category to improve across all projects (i.e. Testing)
Continuous Testing
Chapter V
Testing in DevOps
There is no DevOps without Continuous Testing
Test Automation Pyramid
Business Facing
Technology Facing
Testing in DevOps
Chec
k fo
r exp
ecte
dIt’s not ideal to automate everything
Find the unexpected
Unit
Integration
UI
High
Medium
Low
Low
Medium
High
Medium
Long / High
Short / Low
Test Type
Business Logic Coverage
Code Coverage Execution Time / Costs
Testing in DevOps
Unit Testing aims to test small chunks of your code in isolation from the rest of the world.
UI Testing, different name of system testing, where you test the entire system together to ensure it does what it is supposed to do under real life conditions. (Unless by UI testing you mean usability / look & feel etc. testing)
Testing in DevOps
You need both of these in most of projects, but at different times: unit testing during development (ideally from the very beginning, TDD!!!), and UI testing
later, once you actually have some complete end-to-end functionality.
If you already have a system running, but no tests, practically you have legacy code. Start to get the best test coverage achievable with the least
effort first, which means high level functional tests.
Adding unit tests is needed too, but it takes much more effort and starts to pay back later.
Continuous Testing Anti-Patterns
Long and slow deployment pipelines
Test Data Management is not a big deal
We can skip non-functional tests
Can be done by anyone
Don’t need to refactor/maintain automated tests
Long and Slow Deployment Pipeline Anti-Pattern
Tips:• If next stage(Automated Acceptance Tests) takes a significant amount of time (e.g.
More than 30 minutes), embed a small subset of them into commit stage. So, feedback interval will be decreased to act fast on major incidents
• Run tests in parallel (TestNG for Java and MbUnit for .Net might be good choices)• Focus on multi-threading for race conditions• Design atomic scenarios
Long and Slow Deployment Pipeline Anti-Pattern
Tip: Prefer wide and shallow architecture rather than deep and narrow.
Test Data Management is not a Big Deal Anti-Pattern
Four Design Techniques for Successful Test Automation Data Management
A typical maturity level of data management for test automation process is outlined here;
1. Fully Integrated Test Data2. Partially Independent Test Data3. Storing Test Data in an External Source4. Dynamic Test Data Management (Micro Services, GUI ?)
Non-Functional Testing Anti-Pattern
Tips:
• Select most business critical cases (Either
widely used or critical for a business)
• Test against a production replica environment,
for example staging (As much as possible)
• Do care about data. Effects computational cost
• Focus on subject matter practices (Anything!)
• Use automated-acceptance tests with counters
(As a first step maybe)
Can be Done by Anyone Anti-Pattern
Reasons Behind The Idea
• Test automation is a development activity (Performance,
Security Testing etc. as well )
• Convincing people to have a carreer in the field
• Positioning the personnel and task in the right place
• …
May prefer a different job title, like ‘Software Development Engineer in Test’ (SDET)
Automation Maintanance is not Required Anti-Pattern
Automation code is passive, meaning effected by
any change in product code.
Even with a perfect automation architecture, many
times it is not that possible, you will need to
redesign against living product.
Sounds like Software Gardening!
Continuous Testing – E-commerce Case Study
Inflection Point
2-3 Test Cases per Man/DayNearly No Maintance Effort
3-5 Test Cases per Man/DayLess Maintance Effort (%20)
2-3 Test Cases per Man/DayModerate Maintance Effort (%70)
3 Test Cases per Man/DayModerate Maintance Effort (%50)
~1 Test Cases per Man/DayHeavy Maintance Effort (%90)
Maximum number of test cases~350 – 500 depending of SUT
Based on metrics from 14 consultancy projects
QA Intelligence Survey 2015
Kristian Karl – TestIstanbul 2016
Mike Cohn Test Automation Pyramid
Google Search Trends - DevOps
searchsoftwarequality.techtarget.com/tip/Use-Agile-software-testing-principles-to-plan-your-tests
blog.martinfenner.org/images/Agile-vs-iterative-flow.jpg
slideshare.net/IBMDevOpsforEnterpriseSystems/lessons-learned-from-large-scale-adoption-of-devops-fori-bm-z-systems-software
slideshare.net/ThoughtWorks/when-enterprise-meets-devops/15
PRIORITIZE_PILLAR_OF_PRACTICES15ESSENTIALCollaborationBuild_for
slideshare.net/SkeltonThatcher/continuous-delivery-antipatterns-from-the-wild-matthew-skelton-continuous-lifecycle-london-2016
confengine.com/agile-india-2016/proposal/1680/how-to-explore-the-learning-organization-within-the-agile-organization
References
