programutvecklingsmetodik: testing fundamentals › ~tddc88 › theory › 30testing...

Peter Arvidson, Linköping 2016-09-23

Programutvecklingsmetodik:

Testing fundamentals

Copyright © 2016 Accenture All rights reserved. 2

Agenda

About me and Accenture

What is testing and why is it important?

The V-model concept

Important test concepts/aspects

Test approach and methods

Risk-based testing

Testing as part of agile methodology


• Accenture is a leading global professional services company, providing a broad range of services and solutions in strategy, consulting, digital, technology and operations

• Combining unmatched experience and specialized skills across more than 40 industries and all business functions – underpinned by the world’s largest delivery network – Accenture works at the intersection of business and technology to help clients improve their performance and create sustainable value for their stakeholders

• We help clients become high-performance businesses. In other words, we help them become more successful in every part of their businesses, from strategic planning to day-to-day operations

• With more than 375 000 people serving clients in more than 120 countries, Accenture drives innovation to improve the way the world works and lives

• We are approx. 4 850 employees in the Nordic region, with 950 in Sweden

• Our Swedish head office is in Stockholm with additional offices in Gothenburgand Malmö

Accenture in brief


What differentiates Accenture from competitors?

Strategy ManagementConsulting

Technologystrategy

SystemsIntegration /

Implementation

ITOutsourcing

Accenture

IBM

McKinsey & Company

BCG

Capgemini

Tieto

BusinessProcess

Outsourcing

CGI

Deloitte

Connecta

TCS

HCL

Cognizant

The sheer scale of capabilities and client engagements

differentiates what we do from most other companies.


Originally from Eksjö, Småland

Master in Informatics (Systemvetenskap) and Bachelor in

Business Administration from LiU (1998-2004)

Worked 12+ years for Accenture with Technology Consulting and

Systems Integration – mostly within the Financial Services sector

(Banking and Insurance)

Normally working as Programme or Project manager

About me

Peter ArvidsonManager, Financial Services

Accenture

Stockholm, Sweden

Mail: [email protected]

Tel: +46 730513293


• An essential part of the software development life cycle

• A structured way of validating that requirements and specifications are properly implemented in a solution and proves if a solution meets a customer’s functional, technical, operational, and maintenance expectation

• Testing is a critical means for reducing software delivery risks

• A traditional testing approach show that the system (don’t forget negative scenarios):

• Does what it should

• Doesn’t do what it shouldn’t

What is testing?

Validation – Doing the right thing (according to need)

•Checks that output deliverables satisfy the requirements specified in a previous stage’s input deliverables

•Ensures that the work product is in scope, contributes to the intended benefits, and does not have undesirable side effects

• Is performed by inspecting, simulating, or prototyping

Verification – Doing it the right way (according to specifications)

•Checks that a deliverable is correctly derived and internally consistent

•Checks that output and the process conform to the standards

• Is performed by inspecting and reviewing

Testing – Right things working right

•Checks that a specification/need is properly implemented

• Is performed by executing the code


• Successful testing can save a

lot of costs, both during

implementation and by

avoiding costly errors in

production

• Testing have many different

purposes

• Stage containment an

important aspect of testing

Importance of testing

Detect and resolve errors

as soon as possible

Verify that system meets

client requirements

Ensure client operations are

functioning upon

deployment of the application

Improve Quality and

thus building confidence

Train users and pass

ownership to them

Purpose of testing


• Ideally errors/defects should be found before moving to the next stage

• Errors – problems found in the stage in which they were created

• Defects – problems found in a later stage

• With Stage Containment:

• More errors than defects

• Cost and effort for fixing problems is minimized

• Without Stage Containment:

• Increased risk of finding defects, also in production

• Fixes become more expensive and difficult since there is often a need to go back several stages to fix the error/defect

Testing to ensure stage containment


• Software Testing is not just a single activity, it consists of series of activities (stages) carried out methodologically to help certify the software product

• The V-Model is a proven, industry standard framework that defines the standard development life cycle

• The V-Model requires that each major deliverable is verified and validated in an attempt to identify problems as early as possible and ensure that specifications are complete, correct, and adhere to relevant standards

• Testing ensures that the specifications are properly and correctly implemented and that the solution meets the business and performance requirements

• The V-Model saves time and money in development while increasing the result quality and the delivery reliability

• Adhering to the V-Model leads the way to a substantially decreasing the number of errors found in production after each release

• Quality is designed into the system rather than tested into the system

The V-model concept


• Deployment Test. Tests the production environment’s readiness to handle the new system. Consists of verifying that all components of the system are collated and can be correctly deployed to the production environment in the time required and that the system is correctly installed and configured in the production environment

• Acceptance Test. OAT verifies that the correct functionality, architecture, and procedures are defined and implemented for maintenance and operations and UAT ensures that the users and stakeholders are satisfied with the solution. Testing should be done on production like environment.

• Product Test (System Test). Tests that all business requirements are met by the system. For systems with multiple applications this can be divided into Application (testing a specific application) and Integration (testing end-to-end functionality across applications)

• Performance Test. Ensures that the system is capable of operating at the load levels specified by the performance requirements and any agreed upon Service Level Agreements (SLAs)

• Assembly Test. Tests the interactions of related application components to ensure their proper function when integrated

• Unit Test. Tests each individual unit of the application

The V-Model


• Entry/Exit Criteria

• Sets of conditions that must be satisfied before entering or exiting a project stage

• Activities in one stage must be completed before moving on to the next stage. It is critical that the exit criteria defined for the preceding stage have been met; however, in cases where this is not possible, it is essential that this shortfall be correctly documented and understood by the recipient stage, and that necessary action plans be devised to resolve the identified issues.

• Regression Testing

• Ensures that when changes are introduced to a system (as a result of defect fixes or enhancements) they do not adversely affect the correct (or original) functionality of the system

• Design and execute test procedures that test (and retest) both the affected components and all other areas affected by the change

• Pass Strategy

• Ensures that test execution schedules and the associated test effort are for test execution, it is easy to overlook the fact that defects are raised during test execution/error fixing

• Usual a three pass strategy is used, executing test scripts for every test cycle in a test stage an average of three times.

Important test concepts/aspects (1/2)


Important test concepts/aspects (2/2)

• Defect Management

• Tracks and manages the discovery, resolution, and retest of system defects identified during test execution. Specifically, this process involves recording, reviewing, and prioritizing defects; assigning defects to developers for fixing; and assigning testers to retest fixes

• Defects are normally categorized according to severity and priority:• Severity: How critical the defect is for the business

• Priority: The importance of when the defect has to be fixed (less critical defects can block testing and might therefore be of higher priority to fix)

• Testing Metrics

• Testing metrics measure a project’s progress. This information assists management by providing predefined data on which progress is monitored, decisions are made, and actions are taken.

• During the project planning and analysis stages, a project team defines metrics to use across that project

• Testing documentation

• Test Approach – details the overall test strategy for all test stages

• Test Scenarios – high-level descriptions of functional and technical areas to be tested

• Test Conditions and Expected Results – define the tests to be performed

• Requirements Traceability Matrix – specification of where all requirements are covered in terms of design, development, test


• Testing should generally stop when you are confident that the system works correctly, however it can be hard to identify the right moment since it depends on the risks for the system:

• Risk of finding critical defects which will impact the business in a significant way

• Risk of incurring costs when finding defects (both related to the defect impact and the cost of fixing the defect)

• Risk of losing credibility and market share

• Incorrect assumptions about when to stop testing

• It’s never enough• Testing too much can be very costly, much more costly than the risk of finding production defects

• When you have tested what you planned• Plan might be incorrect and things might change during the testing which needs to be taken into account

• When your customer/user is happy• Often the customer/user is not fully happy until everything is tested and there are no open defects, however in

most situations this means too much testing has been done

• Normally a system should go into production with open defects and some defects should never be fixed

When to stop testing?


Manual vs. Automated testing

Manual testing

• Any new application must be manually tested before its testing can be automated. Manual testing requires more effort, but is necessary to check automation feasibility.

• Manual Testing does not require knowledge of any testing tool. One of the Software Testing Fundamental is "100% Automation is not possible". This makes Manual Testing imperative.

• The goal of Manual Testing is to ensure that the application is error free and it is working in conformance to the specified functional requirements. Test Suites or cases, are designed during the testing phase and should have 100% test coverage.

Automated testing

• Automation Testing means using an automation tool to execute your test case suite. The automation software can also enter test data into the system being tested, compare expected and actual results and generate detailed test reports.

• Test Automation demands considerable investments of money and resources. Successive development cycles will require execution of same test suite repeatedly. Using a test automation tool it's possible to record this test suite and re-play it as required. Once the test suite is automated, no human intervention is required (except for maintaining the automated test suite)

• Goal of Automation is to reduce number of test cases to be run manually, and not eliminate manual testing all together.


Testing methods

Black box testing

• Testing without having any knowledge of the interior workings of the application

• The tester is oblivious to the system architecture and does not have access to the source code

• Typically, while performing a black-box test, a tester will interact with the system's user interface by providing inputs and examining outputs without knowing how and where the inputs are worked upon

White box testing

• The detailed investigation of internal logic and structure of the code

• In order to perform white-box testing on an application, a tester needs to know the internal workings of the code

• The tester needs to have a look inside the source code and find out which unit/chunk of the code is behaving inappropriately

Grey box testing

• To test the application with having a limited knowledge of the internal workings of an application

• In software testing, the phrase the more you know, the better carries a lot of weight while testing an application. Mastering the domain of a system always gives the tester an edge over someone with limited domain knowledge

• Unlike black-box testing, where the tester only tests the application's user interface; in grey-box testing, the tester has access to design documents and the database. Having this knowledge, a tester can prepare better test data and test scenarios while making a test plan


• It is easy to overlook testing non functional requirements but this is very important since these are normally important for the system to work properly but also for the ability to be able to maintain and operate the system

• Non functional requirements include:

Non functional testing

Scalability Performance Recoverability

Security and confidentiality

Maintainability Operability

Availability Usability


Risk-based testing

• In general - It is the process of assessing threats, taking steps to mitigate risk to an acceptable level and maintain or reduce that level of risk.

• In a project context - It is the process of identifying, analyzing, and addressing risks proactively throughout the project lifecycle.

What is Risk Management?

• Prioritized testing - Project testing activities focus on the most important tests that will mitigate the highest amount of risk.

• Balance effectiveness and efficiency - The goal of Risk-Based Testing is to increase test effectiveness while potentially resulting in fewer tests thus reducing cost.

How do Risk Management and Testing

relate?

• Increased speed to market

• Enables reduced testing time by judiciously planning testing with emphasis on optimizing test resources and efforts

• Reduced cost

• Reduces overall cost of fixing defects by identifying poorly written requirements early in the development lifecycle

• Provides objective approach to determine required testing scope and to avoid unnecessary test effort. Helps in identifying test coverage gaps during test planning and test preparation.

• Focus on quality

• Focuses project testing activities on the most important tests that will mitigate the highest amount of risk.

• Better risk management

• Provides vehicle for business, development and test teams to jointly align on assumptions that drive the test approach and planning process.

• Creates a risk baseline to enable management to accurately assess change controls not only from impact to effort but also impact to project risk.

Benefits of Risk-based

testing


• During each sprint the development team executes a set of stories that were committed to during sprint planning

• Each story is divided into a set of tasks that include the activities of designing, coding, testing, and documenting the functionality specified by the story

• Author tests (before development) is the process of creating various automated and manual test cases that exercise both the newly produced code and the existing code for any possible regressions

• Acceptance Criteria could include User Story Acceptance Success Criteria or anything else that serves as criteria to demonstrate when a component of functionality has been implemented correctly and verifies against a declared need

• Unit Tests, however, often do not have written acceptance criteria since at such a low-level of testing that the tests themselves are often the acceptance criteria

• During this activity it is determined which types of tests are most appropriate for the given functionality to be tested (from Unit Tests to Acceptance Tests)

• The adoption of a Test Driven Development (TDD) approach is highly recommended

• TDD stipulates that an automated unit test that tests for the requirements of the code be written before each aspect of code itself

• Adherence to TDD is a method of software design – not just testing – since TDD forces focus on the requirements and how they can be tested for – before you write the code –to avoid adapting the tests to how the code is written

• A key benefit to TDD is that you are to write only enough code such that your test passes – thus reducing the amount of unused and unnecessary code

• During test execution the test results are evaluated to identify defects to be fixed and potential enhancements to be added to the solution backlog

Testing as part of agile methodology

programutvecklingsmetodik: testing fundamentals › ~tddc88 › theory › 30testing...

Documents