building qa automation using agile - synerzip€¦ · introduction • topic: building qa...

Building QA

Automation

Using AgileOctober 2, 2009

Introduction

• Topic: Building QA Automation Using Agile

Automated testing is a key requirement for effective Agile development teams. Automated regression test suites allow cost-effective regression testing with high coverage.

But, these suites tend to be large software themselves, with quality and maintainability requirements higher than the application under test (AUT). Therefore, automation suites need to be built using a disciplined software development approach, such as Agile.

However, the AUT itself is changing and this makes Agile approach an even more suitable development approach for building automated regression test suites

In this webinar we will discuss how to effectively build automated test suites using Agile approach

www.synerzip.com © SQS Software Quality Systems AG |

• Presenter: Gireendra (Giri) Kasmalkar

Giri is MD & CEO of SQS India. Giri founded VeriSoft, one of India’s leading independent testing companies. VeriSoft was acquired in July 2008 by SQS Software Quality Systems AG, the global leader in independent testing.

Giri did his BS in Mechanical Engineering from IIT Mumbai (India) and MS from Univ. of South Carolina.

Contents

Regression Automation Recap

The Problem

The Solution

Methodology

Standard Lifecycle

Agile Lifecycle

Sample Sprint

In Conclusion...

Questions?


Regression Automation Recap


Automation Suite

For Regression Testing

Faster

Higher Coverage

Better Process Adherence

Expected Attributes

Robust

Data Driven

Maintainable

Ultimate Goal

Should reduce total cost of regression testing and risks for the AUT


Automation Suite Structure

ApplicationUnderTest

CSV XML EXCEL DB

Utilities

TestResults

TestScript

Test Data Store

ObjectMap

Test Script

Base state

Action

Verification

Report


Automation Framework

For more robust, data-driven and maintainable automation

Can handle more complex functionality, alternate work-flows

Tracking and reproducing functionality is easier

Higher productivity

Reused (pre-tested) utilities and wrappers

Organization-wide usage

Across applications

Across tools

Typical frameworks

Data-driven

Test Script Modularity Framework

Test Library Architecture

Keyword driven or Table driven Framework

Hybrid Test Automation Framework


The Problem


The Problem

Large and critical software

Need formal methodologies

Need frameworks

Quality requirements higher

than AUT, esp.

Robustness

Maintainability

Dependence on

Tools – limits on automation

AUT – which keeps on changing

The peculiar nature of automation suites


The Solution


Overview

Premises

AUT will not be static

New features

Bugs

Regression testing is for the stable parts of AUT

These may keep on changing

Solution

Agile paradigm of incremental delivery

Broad plan for the entire automation, but detailed plan for each sprint

Benefits

Can be nimble about which test cases to add to automation

The use of automated regression starts much earlier than traditional methodology

Automation suite always in sync with AUT

The Solution


Track Record

Customers

Products

Enterprise Apps

Tools

Commercial tools – QTP, Silktest, Rational

Open Source – Selenium, AutoIT

Projects

2 people x 3 months to 14 people x 18 months

Iteration – 2 weeks to 4 weeks

Paper

Presented and published at QAI International Testing Conference, Pune, 2004

The Solution


Methodology


Standard Automation Life Cycle

Tool Evaluation

Comparative evaluation of different test automation tools

Object recognition, ease of learning, ease of use, capability of the scripting language

Requirement Analysis

To study the existing test cases to estimate the percentage of automation.

To determine requirement coverage of existing test cases and improve it

Design

Scripts

Data preparation for base states, input and master output

Utilities

Methodology


Standard Automation Life Cycle (contd)

Development

Recording

Programming for robustness (exception handling), data-drivenness and as per coding guidelines

Development and documentation of utilities

Coding for reports – logs, messages and test reports

Testing

Unit testing

Batch testing

Deployment

Creating configuration files

Use Test Management tool

Methodology


An

aly

sis

Co

de

Test

Imp

lem

en

t

An

aly

sis

Co

de

Test

Agile – Overview

3 6 9 12

36

912

Methodology


Objective: Select tool, establish the process, proof of concept, estimate percentage of automation, estimate efforts for the subsequent automation.

Deliverables

Deploy automation for the POC scenarios

Detailed plan for development and maintenance phases

Duration – 3-4 weeks

Pilot / Planning phase Development phase Maintenance phase

Objective: Automating the existing test scenarios to the estimated percentage

Deliverable: Bi-weekly deployment of automated test scenarios, execution of updated automation suite and test report

Objective: Maintain automation suite with enhancements with each release

Deliverable: Updated automation suite, execution of the updated automation suite and test report.

Methodology

Agile Automation Lifecycle


Methodology

Activities in Pilot / Planning Phase

1. Application understanding

2. Review of test cases

3. Prioritization of modules and their test cases for automation based on stability

4. Tool selection

5. Feasibility study to determine percentage of automation possible

6. Design of automation suite – directory structure, utilities

7. Pilot implementation of automation for a module

8. Creating a broad plan for subsequent development and maintenance phases.

Assumption: Scenario where product development is underway but test automation has not started.


Methodology

Activities in Development Phase

For each iteration (bi-weekly)

1. Selection of modules and test cases based on stability and criticality

2. Determination of backlog from previous iteration and customer feedback

3. Execution of automation suite – defect reporting and determination of maintenance requirements

4. Effort estimation and detailed planning for the iteration

5. Design, development, testing of utilities / framework elements

6. Design, development, testing and deployment as per typical SDLC – the latest release of the AUT is used for testing and deployment.

Iterations repeated till backlog of manual test cases automated to expected coverage


Tips:

Automation should lag behind manual testing by a few iterations

Test cases chosen for automation are not the ones newly added in the iteration, but the ones that have become stable.

A smaller team can be used as the time-lines are longer in this phase.

Methodology

Activities in Maintenance Phase

For each iteration (each release of AUT)

1. Selection of modules and test cases based on stability and criticality

2. Determination of backlog from previous iteration and customer feedback

3. Execution of automation suite – defect reporting and determination of maintenance requirements

4. Effort estimation and detailed planning for the iteration

5. Design, development, testing of utilities / framework elements

6. Design, development, testing and deployment as per typical SDLC – the latest release of the AUT is used for testing and deployment.


Sample sprint time-table

Methodology

No Activity Deliverable Day By

1 Release of AUT to automation team 1 Release Team

2 Selection of test cases for automation Product Owner

3 Preparation of test scenario and datasheets Test case and datasheet documentsCoverage of automation metric

1 Product Owner

4 Effort estimation for backlog from earlier cycle 1 Automation Team

5 Effort estimation for designed test cases 2 Automation Team

6 Run the existing automation suite to determine maintenance requirement and effort estimation

2 Automation Team

7 Defect Reporting 2 Automation Lead off-shore

8 Development work on backlog 1-2 Automation Team

9 Deciding on scope for the cycle Coverage of implementation metric 3 Automation Lead off-shore

10 Development of Automation scripts Updated automation suite 3-9 Automation Team

11 Deployment (adding to the existing suite), execution

Performance metricsRework requirements

10 Automation Lead off-shore

Acceptance 10 Product owner


In Conclusion...

Using Agile ensures success of Automation

Incremental delivery means

Early usage

Continuous sync with AUT

Concrete metrics of progress of the project

Can be used for any development methodology used by AUT

Using Agile facilitates out-sourcing / off-shoring of Automation

(Otherwise) Automation always takes a back-seat when release is near

Clear division of labour

Manual QA team working on release can focus on new features

Automation team can focus on regression of much larger number of old features (as regression is only for the stable parts, anyway)

Clear visibility of progress and success of Automation


Questions?


1. Software development partner for small/mid-sized technology companies

Exclusive focus on small/mid-sized technology companies

Deep experience in full SDLC – design, dev, QA/testing, deployment

Technology and industry domain agnostic

2. Dedicated team of high caliber software professionals

Seamlessly extends your local team, offering full transparency

NOT just “staff augmentation”, but provide full mgmt oversight

3. Actually reduces risk of development/delivery

Experienced team - uses appropriate level of engineering discipline

Practices Agile development – responsive, yet disciplined

4. Reduces cost – dual-shore team, 50% cost advantage

5. Offers long term flexibility – allows (facilitates) taking offshore team captive


Synerzip Clients – Representative

List


http://www.itko.com/

http://www.yagnaiq.com/index.htm

http://www.clearorbit.com/index.php?sess_id=b0a39bdaf17f0f7c5cd34f8cc3d49d10

http://www.zafesoft.com/index.asp

http://www.hallsettlementsystems.com/Default.aspx

http://www.fd2s.com/home.php

http://www.repliqa.com/

http://www.broadlane.com/index.html

http://www.neofirma.com/Default.aspx

http://www.sevayatra.com/index.html

At a glance: SQS is the leader in independent software testing

and quality management services.

More than 27 years of prosperous operations

Operations in 14 countries globally

Over 5,000 successfully completed projects

The customer base includes36 FTSE-100 companies, half ofthe DAX 30 companies and nearly a third of the STOXX-50 companies

The SQS philosophy is to increase success and efficiency of IT projects by using efficient testing solutions

SQS Software Quality Systems

SQS is listed on the AIM London

»«

The global leader in independent software

testing and quality management services -

majority of its business in Europe

Financial Times, 21 August 2007


SQS India Overview

Leading independent testing company in India

11+ year track record of off-shore services

150+ testing consultants

300+ projects for more than 100 customers globally

Captive off-shore centers, IT companies

Game companies, product companies

Acquired by SQS in July 2008

Primary destination for scalable off-shore

Mature offshore delivery process(SQS Global Offshore Delivery Model)

QA Partnership with Synerzip

Last 5 years

Automation and Load testing projects

Commercial as well as freeware tools

ISO 9001:2000 certifiedfor the last 5 years

ISO27001 in progress

About SQS


Thank You!

Please contact us for free consultation

Hemant Elhence

[email protected]

469.322.0349


building qa automation using agile - synerzip€¦ · introduction • topic: building qa...

Documents