devops tactical adoption theory: continuous testing

Post on 13-Apr-2017

318 Views

Category:

Software

2 Downloads

Preview:

Click to see full reader

TRANSCRIPT

DevOps Tactical Adoption Theory: Continuous Testing

Berk Dülger – Consultant, Account Manager

TEST CONSULTING TEST OUTSOURCING TEST TRAINING

Learning Organizations

Chapter I

Learning Organizations

Learning organization is one in which people at all

levels, individually and collectively, are

continually increasing their capacity to produce

results they really care about.

Peter Senge

Learning Organizations’ Five Discipline

Systems thinking: Ability to see whole

Personal mastery: The process of life long learning

Shared vision: A vision that shared & committed by

everybody

Mental model: Generalized & assumed view of how the world

works

Team learning: The process of developing  the ability to

create the desired results as a team

Systems Thinking

State of Agile/DevOps

AdoptionChapter II

Lean/Agile Adoption in Europe

Global DevOps Rates

Principles and PracticesChapter III

Principles and Practices

Lean is the basis of AgileLean tells you to optimize the end-to-end process which creates value for your customer from the initial idea to collecting cash. Lean principles focus on flow more than anything else: bottlenecks in the process must be removed and wasteful activities need to be identified and avoided.

DevOps is not a goal, but a process of

continuous improvement

Sustainable success requires both bottom-up practices

and top-down management support

DevOps Practices

Practices should address problems like,• Manual Efforts

• Long feedback times

• Long MTTR

• Too much downtime

• Lead time

• Unrepetable work

Automation in DevOps

What can be automated?• Dev environments

• Builds on pull requests and merge• Static code analysis• Code style checking

• Dynamic code analysis• Verification

• Archiving artifacts• Deployment

Tactical Adoption TheoryChapter IV

Tactical DevOps Adoption

DevOps Tactical Adoption Theory tries to make the transition process as smooth as possible. It hypothesis each step towards DevOps maturity should bring a visible business value empowering management and team commitment for the next step.

The idea here, it is not required to add the tools/processes to stack from sequential beginning to end, but seeking benefit.

Tactical DevOps Adoption

The reason behind the theory is to encourage practitioners to apply each step one-by-one and then having the benefits in projects. Consequently, each step is tested in terms of utility and proved method validity for the further steps.

Large-Scale Adoption

• Begin with an end in mind• Start with a pilot project

• Make someone/unit responsible• Evangelize, build communities

• Gain executive buy-in• Make people believe

• Drive tool standardization• Automate, automate, automate: Build, Test, Deploy

• Demostrate the value!

Either two ways;

Choose to improve all categories for single project

Or, choose one category to improve across all projects (i.e. Testing)

Continuous Testing

Chapter V

Testing in DevOps

There is no DevOps without Continuous Testing

Test Automation Pyramid

Business Facing

Technology Facing

Testing in DevOps

Chec

k fo

r exp

ecte

dIt’s not ideal to automate everything

Find the unexpected

Unit

Integration

UI

High

Medium

Low

Low

Medium

High

Medium

Long / High

Short / Low

Test Type

Business Logic Coverage

Code Coverage Execution Time / Costs

Testing in DevOps

Unit Testing aims to test small chunks of your code in isolation from the rest of the world.

UI Testing, different name of system testing, where you test the entire system together to ensure it does what it is supposed to do under real life conditions. (Unless by UI testing you mean usability / look & feel etc. testing)

Testing in DevOps

You need both of these in most of projects, but at different times: unit testing during development (ideally from the very beginning, TDD!!!), and UI testing

later, once you actually have some complete end-to-end functionality.

If you already have a system running, but no tests, practically you have legacy code. Start to get the best test coverage achievable with the least

effort first, which means high level functional tests.

Adding unit tests is needed too, but it takes much more effort and starts to pay back later.

Continuous Testing Anti-Patterns

Long and slow deployment pipelines

Test Data Management is not a big deal

We can skip non-functional tests

Can be done by anyone

Don’t need to refactor/maintain automated tests

Long and Slow Deployment Pipeline Anti-Pattern

Tips:• If next stage(Automated Acceptance Tests) takes a significant amount of time (e.g.

More than 30 minutes), embed a small subset of them into commit stage. So, feedback interval will be decreased to act fast on major incidents

• Run tests in parallel (TestNG for Java and MbUnit for .Net might be good choices)• Focus on multi-threading for race conditions• Design atomic scenarios

Long and Slow Deployment Pipeline Anti-Pattern

Tip: Prefer wide and shallow architecture rather than deep and narrow.

Test Data Management is not a Big Deal Anti-Pattern

Four Design Techniques for Successful Test Automation Data Management

A typical maturity level of data management for test automation process is outlined here;

1. Fully Integrated Test Data2. Partially Independent Test Data3. Storing Test Data in an External Source4. Dynamic Test Data Management (Micro Services, GUI ?)

Non-Functional Testing Anti-Pattern

Tips:

• Select most business critical cases (Either

widely used or critical for a business)

• Test against a production replica environment,

for example staging (As much as possible)

• Do care about data. Effects computational cost

• Focus on subject matter practices (Anything!)

• Use automated-acceptance tests with counters

(As a first step maybe)

Can be Done by Anyone Anti-Pattern

Reasons Behind The Idea

• Test automation is a development activity (Performance,

Security Testing etc. as well )

• Convincing people to have a carreer in the field

• Positioning the personnel and task in the right place

• …

May prefer a different job title, like ‘Software Development Engineer in Test’ (SDET)

Automation Maintanance is not Required Anti-Pattern

Automation code is passive, meaning effected by

any change in product code.

Even with a perfect automation architecture, many

times it is not that possible, you will need to

redesign against living product.

Sounds like Software Gardening!

Continuous Testing – E-commerce Case Study

Inflection Point

2-3 Test Cases per Man/DayNearly No Maintance Effort

3-5 Test Cases per Man/DayLess Maintance Effort (%20)

2-3 Test Cases per Man/DayModerate Maintance Effort (%70)

3 Test Cases per Man/DayModerate Maintance Effort (%50)

~1 Test Cases per Man/DayHeavy Maintance Effort (%90)

Maximum number of test cases~350 – 500 depending of SUT

Based on metrics from 14 consultancy projects

QA Intelligence Survey 2015

Kristian Karl – TestIstanbul 2016

Mike Cohn Test Automation Pyramid

Google Search Trends - DevOps

searchsoftwarequality.techtarget.com/tip/Use-Agile-software-testing-principles-to-plan-your-tests

blog.martinfenner.org/images/Agile-vs-iterative-flow.jpg

slideshare.net/IBMDevOpsforEnterpriseSystems/lessons-learned-from-large-scale-adoption-of-devops-fori-bm-z-systems-software

slideshare.net/ThoughtWorks/when-enterprise-meets-devops/15

PRIORITIZE_PILLAR_OF_PRACTICES15ESSENTIALCollaborationBuild_for

slideshare.net/SkeltonThatcher/continuous-delivery-antipatterns-from-the-wild-matthew-skelton-continuous-lifecycle-london-2016

confengine.com/agile-india-2016/proposal/1680/how-to-explore-the-learning-organization-within-the-agile-organization

References

Thank You

Berk Dülgerberk.dulger@keytorc.com

https://tr.linkedin.com/in/berkdulger

top related