TESTING IN THE NEW AGE OF DEVOPS

Moataz Mahmoud Ahmed

Junior QC developer at xWare

AGENDA

• Introduction to DevOps.

• Glossary.

• Continuous testing.

• The DevOps lifecycle.

• Where does QA fit in DevOps.

• Test-Driven Development (TDD).

• References.

Introduction

to DevOps

INTRODUCTION TO DEVOPS.• DevOps is a clipped compound of

"software DEVelopment" and "information

technology OPerationS“.

• DevOps is a term used to refer to a set of

practices that emphasize the collaboration and

communication of both software

developers and information technology (IT)

professionals while automating the process

of software delivery (deployment) and

infrastructure changes.

INTRODUCTION TO DEVOPS CONTD. • Because DevOps is a cultural shift and collaboration between development, operations

and testing, there is no single "DevOps tool": it is rather a set which is called "DevOps

toolchain", consisting of multiple tools.

• Generally, DevOps tools fit into one or more of these categories, which is reflective of

key aspects of the software development and delivery process:

1. Code — Code development and review, version control tools, code merging.

2. Build — Continuous integration tools, build status.

3. Test — Continuous testing tools that provide feedback on business risks.

4. Package — Artifact repository, application pre-deployment staging.

5. Release — Change management, release approvals, release automation.

6. Configure — Infrastructure configuration and management, Infrastructure as Code tools.

7. Monitor — Applications performance monitoring, end–user experience.

INTRODUCTION TO DEVOPS CONTD.

• There are many problems DevOps is trying to solve which are:

1. Shift Operations or IT tasks earlier in the process.

2. To leverage a bunch of newer tools and technologies to automate operations tasks, like provisioning and migrating code, leveraging these tools.

3. Have everyone on the product team communicate and collaborate much more and earlier.

4. To have the product be ready to be deployed—in a consistent state of readiness so that the business can decide when new functionality goes to customers. And ultimately, not be held back by Dev. or Ops being unstable and not prepared to go with whatever is read.

• The specific goals of DevOps span the entire delivery pipeline. They include improved deployment frequency, which can lead to:

1. Faster time to market.

• It means faster time to recovery (in the event of a new release crashing or otherwise

disabling the current system).

• It saves your time to work on other project though you can gain more profit.

2. Lower failure rate of new releases.

3. Shortened lead time between fixes.

4. Better quality of the final products.

5. As a result, gain customer satisfaction.

INTRODUCTION TO DEVOPS CONTD.

Glossary

INFRASTRUCTURE AS CODE (IAC)• It is the process of managing and provisioning computing infrastructure

(processes, bare-metal servers, virtual servers, etc.) and their configuration through machine processable definition files, rather than physical hardware configuration or the use of interactive configuration tools.

• The definition files may be in a version control system. This has been achieved previously through either scripts or declarative definitions, rather than manual processes, but developments as specifically titled “IaC” are now focused on the declarative approaches. Infrastructure as Code approaches have become increasingly widespread with the adoption of cloud computing and Infrastructure as a Service (IaaS). IaC supports IaaS, but should not be confused with it.

INFRASTRUCTURE AS A SERVICE(IAAS)

• It is a standardized, highly automated offering, where compute resources, complemented by storage and networking capabilities are owned and hosted by a service provider and offered to customers on-demand. Customers are able to self-provision this infrastructure, using a Web-based graphical user interface that serves as an IT operations management console for the overall environment. API access to the infrastructure may also be offered as an option.

ENVIRONMENT AS A SERVICE(EAAS)

• Environment-as-a-Service, often referred to as IaaS Plus, extends the traditional IaaS ecosystem into the application development space. Here advanced automation is layered on top of the existing IaaS instance to not only configure servers for a particular application, but also to deploy and test all the other components needed to run a given application.

CONTINUOUS DEVELOPMENT• It’s a software development practice in which every code change

goes through the entire pipeline and is put into production, automatically, resulting in many production deployments every day.

• With Continuous Delivery your software is always release-ready, yet the timing of when to push it into production is a business decision, and so the final deployment is a manual step.

• Continuous Deployment mandates Continuous Delivery, but the opposite is not required.

CONTINUOUS INTEGRATION• It’s a software engineering practice in which the changes made by

developers to working copies of code are added to the mainline code base

on a frequent basis, and immediately tested.

• The goal is to provide rapid feedback so that, if a defect is introduced into

the mainline, it can be identified quickly and corrected as soon as possible.

• In the end, well implemented CI reduces the cost of software development

and helps speed time to market.

• What’s this step is doing is it integrates every single changed component at

this build in the whole system.

CONTINUOUS DELIVERY• Is a software engineering approach in which teams produce software in short

cycles, ensuring that the software can be reliably released at any time.

• It aims at building, testing, and releasing software faster and more frequently.

• What’s this step is doing is confirming that the integrated components in the

previous stage are ready to be tested now. This is done by running unit tests,

collecting the release notes and other documents, versioning the new build,

and some other activities.

• There are many well known tools like Docker, Jenkins, Puppet, and Vagrant

are often used and frequently referenced in DevOps tooling.

CONTINUOUS TESTING• It is the process of executing automated tests as part of the software

delivery pipeline to obtain immediate feedback on the business risks

associated with a software release candidate.

• Because CI detects deficiencies early on in development, defects are

typically smaller, less complex, and easier to resolve.

• Once the integration is done, they would do unit tests on the integrated

code. They may run other tests such as white box security tests, code

performance tests, etc. All needed tests are running here.

• In the next section of this presentation, there are much more details

about continuous testing.

CONTINUOUS MONITORING• In Production, the Ops team manages and ensures that the application is

performing as desired and the environment is stable via Continuous Monitoring.

While the Ops teams have their own tools to monitor their environments and

systems, they need to ensure that the applications are performing at optimal

levels – down to levels lower than system monitoring tools would allow.

• This requires that Ops teams use tools that can monitor application performance

and issues.

• It may also require that they work with Dev. to build self monitoring or analytics

gathering capabilities right into the applications being built. This would allow for

true end-to-end monitoring – continuously.

Continuous

Testing

CONTINUOUS TESTING CONTD.

• When I think of Continuous Testing I

think of the Lean principle, quality

at every step. When developers

commit new code, test it! When the

product gets integrated, test it!

When the product gets moved to

any new environment, like the test

environment, staging environment

or production, test it!

CONTINUOUS TESTING CONTD.• There are three important parts here:

1. The build delivered by Development.

2. The deployment on infrastructure handled by OPS/IT.

3. The customers who use the apps/system.

• Test teams must:

- Have strong knowledge of these three parts.

- Know the goals of testing at each part.

- Be able to design great tests for each part.

- Have the ability to automate these tests.

CONTINUOUS TESTING CONTD.• Tests can be designed and built for successful continuous integration first, and expand that

to continuous testing by knowing the right things to do, staying focused on a few key

points and most importantly, automating as smartly as possible.

• DevOps for Testers is focusing on continuous testing.

• When people describe DevOps as a “shift left,” one reference is to running performance

tests early and not waiting until the end of development.

• This “shift left” will uncover bugs earlier which will save time and cost for your company.

• This is an important example because it illustrates the many pieces of DevOps.

• The production environment can be virtualized and spun up at any moment with a VM or

in the cloud and have full performance tests run early in development.

CONTINUOUS TESTING CONTD.• Your team has to be great at Agile. Scrum butts and Agile Falls will fail in

DevOps.

• A quick, Continuous Testing task overview is:

1. Start with an automated smoke test. Move these into CI build process tool.

2. Build bigger regression suites. Move these into CI build process tool.

3. Grow in levels of awesomeness of CI; Run smoke and/or regression on multiple VMs.

4. Easy and effective reporting back to the organization.

5. Use containers and/or virtualization for data and full production like environments.

6. Distribute automated tests into different suites with varying goals on different

environments. Use VMs for various environments to grow automation, coverage, speed,

monitoring and feedback to the team.

The DevOps

Lifecycle

THE DEVOPS LIFECYCLE.In order for DevOps to be successful, Development, Operations and the Test team must align their duties.

THE DEVOPS LIFECYCLE CONTD.

Finally, here is the full DevOps lifecycle.

Where does QA

fit in DevOps?

WHERE DOES QZ FIT IN DEVOPS?• DevOps QA is about preventing defects, not finding them:

• QA takes a critical role in this organizational structure because they have the visibility and

the directive to push code out when it is working, and roll it back

when it is not.

• This has several implications:

• QA owns continuous improvement and quality tracking across the entire development cycle.

• Test are code.

• Anything that can be automated, should be automated.

• Testers don’t just find bugs. They look for any opportunity to improve repeatability and

predictability of them.

• Beyond functional testing: automation for load testing, stress testing, and performance

testing.

WHAT YOU NEED TO CONSIDER BEFORE YOU JUMP TO CONTINUOUS TESTING…

1. Consider the platforms and devices.

• What are the platforms and devices that each of your products run over?

2. Consider the environment.

• Consider the environment that will be in the customer side and try to virtualize it.

3. Consider the tool.

• There are many available tools. You should consider a tool that’s scalable, and can be run on a variety

of machines and environments.

4. Consider the number of tests.

• It’s better to cover more scenarios in less number of tests to save time in recording and running them.

HOW TO MAKE THE JUMP TO DEVOPS.1. Automate more, automate faster.

• You need to automate not only your testing, but other tasks that were done by other teams. And yes,

automate faster to save the cost and raise the quality.

2. Build a solid framework.

• You will need a framework with reusable and low maintenance functions.

3. Choose effective methods.

• Automating traditional test scripts doesn’t work anymore. Consider Action based testing, test-driven

development, and A/B testing.

• Once you have efficient Agile practices, good low-maintenance test automation and Continuous

Integration processes in place, then Continuous Testing and DevOps is the next step in achieving hyper

efficiency.

Test-Driven

Development

TEST-DRIVEN DEVELOPMENT (TDD)• Test-driven development (TDD) is a software development

process that relies on the repetition of a very short development

cycle: requirements are turned into very specific test cases, then the

software is improved to pass the new tests, ONLY.

• The most important concept here is that testing team starts its work

before the development. And it’s the true meaning of “shift left” in

continuous testing.

• This approach prevents software development that allows software to

be added which is not proven to meet requirements.

TEST-DRIVEN DEVELOPMENT LIFECYCLE1. Add a test: Each new feature begins with that the developer writes a test that defines a function or

improvements of a function, which should be very concise.

2. Run this test: At the first run, this test will fail due to there are no code to make it pass! So now it’s

the time to write the code to pass this failed test.

3. Write the code: The main target at this step is to write code which passes the test. No matter how

elegant or efficient it’s. Just pass the test and check this step as completed.

4. Re-run the previously failed test: Now we will re-run the failed test. If it’s passed so you have

done. If not, keep looping between steps #3 and #4 until it’s done.

5. Refactor code: It’s the time now to refactor your code now. The previously written code passed

the test successfully, but it may not be the most efficient code.

TEST-DRIVEN DEVELOPMENT LIFECYCLE CONTD.

6. Add new tests: You can now move on and write new test for a new feature or requirement.

7. Run the new tests: This test will fail expectedly, so you move on to write some code to

make it pass. If these new tests pass from the first run, so there is a design problem here!

This means that you built the features which make it pass in the previous step and this is

out of the right flow.

8. Write more code: Write code to pass the new tests.

9. Run all the tests including the new ones: The new tests should pass now. If not keep

writing code until it pass. All the old tests should pass also, it’s for the sake of integration.

If not, stop working and fix the failed tests.

10. Refactor code.

Repeat…

Traditional development model Test driven development model

NOTES ABOUT TDD• The size of the steps should always be small, with as few as 1 to 10 edits

between each test run.

• If new code does not rapidly satisfy a new test, or other tests fail unexpectedly,

the programmer should undo or revert in preference to excessive debugging.

• Continuous integration helps here by providing revertible checkpoints.

• When using external libraries, there is some reason to believe that the library is

buggy or is not sufficiently feature-complete to serve all the needs of the

software under development.

• This process is also called Red-Green factorization. Red refers to the failed tests,

green refers to the passed ones, and factorization refers to the process of

refactoring the code each time it passes the tests.

NOTES ABOUT TDD CONTD.• Some tend to see TDD as a game of ping pong (or table tennis). The game is

very fast. They tend not to spend more than a minute on either side of the table

(test and implementation). Write a short test and run it (ping), write the

implementation and run all tests (pong), write another test (ping), write

implementation of that test (pong), refactor and confirm that all tests are

passing (score), repeat. Ping, pong, ping, pong, ping, pong, score, serve again.

Do not try to make the perfect code. Instead, try to keep the ball rolling until

you think that the time is right to score (refactor).

• Very useful tool when working in the TDD fashion are watchers. They are

frameworks or tools that are executed before we start working and are

watching for any change in the code. When such a change is detected, all the

tests are run.

NOTES ABOUT TDD CONTD.

• Only trust those tests which you saw in both cases (fail and pass).

• The most important part of TDD is the middle D. You let the

tests drive you. The tests tell you what to do, what to do next,

when you are done. They tell you what the API is going to be, what

the design is.

• This is important: TDD is not about writing tests first. There are

plenty of projects that write tests first but don't practice TDD.

Writing tests first is simply a prerequisite for being able to let the

tests drive the development.

USE TESTS AS DOCUMENTATION • Another very useful side effect of TDD (and well structured tests in general) is

documentation.

• In most cases, it is much easier to find out what the code does by looking at

tests than the implementation itself.

• Additional benefit that other types of documentation cannot provide is that

tests are never outdated.

• If there is any discrepancy between tests and the implementation code, tests

fail. Failed tests mean inaccurate documentation.

• If your tests aren’t that so clear or your product have much details which are

needed to be translated from code to words, you can use tests as main

headlines for the documentation.

REFERENCES• Test-Driven Development article: (Wikipedia).

• DevOps article: (Wikipedia).

• TDD testers.

• Technology Conversation

• Intro to TDD

• JavaScript testing:

• Tests as documentation:

• Testing in the new age of DevOps.