Successful AutomationKDT as a Test Language

Aviram ShottenAP Program ManagerQualiTest

Today’s Menu

• Brief introduction to KDT• Practical KDT – how KDT needs to be implemented –

3 years of experience tells…

• KDT supporting tool – the AutomationPlanner®

• Measuring KDT – RoI and other metrics

• Thumb rules for KDT implementation

• Bad Humor

Introduction to KDT

• KDT is 3rd generation automation approach (after record & replay and functional decomposition)

• Previous approaches had very little success due to the following reasons:– Automation experts required to be also good functional

testers

– Automation became a bottleneck on the way to increase coverage

– Double (manual and automated) maintenance of test cases

– Test management (test leader) didn’t necessarily knew how to manage the automation experts

Introduction to KDT

• In order to better mitigate the drawbacks of automation that were mentioned before, KDT suggests the following:

– Automation experts will develop KDT infrastructure for the functional testers

– Functional testers will use infrastructure for test cases implementation

– Functional testers will mainly investigate failures and incidents rather than execute the test cases

Introduction to KDT

Automation Discipline

Testing Discipline

Developing infrastructure:

• Mapping AUT components

• Imp. components operations

• Imp. system operations

• Imp. business functions

Test engineering tasks:

• Test planning

• Test implementation

• Definition of commonly used sequences for test cases

• Test execution

• Debrief and report

• Continuous improvement

Requirements-KDT

• We, as functional testers, requires from a KDT based solution:

– Simplicity - test cases can be defined without development skills

– Functionality – maintain the option to perform any functional UI operation

– Readability – test cases and test reports will continue to be reviewed by other stakeholders

– Maintenance – test cases will be maintained by functional tester and not by automation experts

Generic KDT Suite

Experience led us to build 4 components to support a generic KDT implementation process among test teams:

– allowing easy test cases

implementation for the functional testers

–executes the test batches

KDT Engine – component that converts KDT test cases to executable format

- integrates the KDT tests to the

standard test process in HP Quality Center®

AP Architecture

Logical Layer

Infrastructure Layer

Testing Tool

AP ENGINE

VBS

Or

SUT

A Test Case

Step #ActionExpected Result

1Execute the “MyApplic” application Log in window appears

2Enter user “JohnB” in username field and “123456” in the password field and press “OK”

Error window appear

3Enter user “JohnB” in username field and “QA2010” in the password field and press “OK”

Login Successful, application’s main window appear

Traditional Test Case

KDT Syntax

• Keywords in use:

– Component Function (CF) – perform a specific operation on a given UI component – click, set value, verify exist etc.

– Service Function (SF) – scripts that enables assisting methods for the test implementation - wait X seconds, write to report etc.

– Business Function (BF) – execute a certain functional operation, that is hard\ non effective to implement by component\ service function

– Sequence (Seq) – a set of keywords (CF, SF, BF, or another Seq) that can be called from different test cases with different parameters.

Test Cases Syntax

Step #

Step Type

ComponentOperationValue

1SF-RunCommand“C:\Program Files\MyApplic.exe”

2CFUserName (Edit)Set“JohnB”

3CFPassword (Edit)Set“123456”

4CFOK (Button)Click-

5CFLoginErrorDialogueVerifyExist-

6CFUserName (Edit)Set“JohnB”

7CFPassword (Edit)Set“QA2010”

8CFOK (Button)Click-

9CFWelcomeWindowVerifyExist-

KDT Test Case

Test Cases Syntax

Step #Step TypeComponentOperationValue

1SF-RunCommand“C:\Program Files\MyApplic.exe”

2SEQLogin-“JohnB”, “123456”

3CFLoginErrorDialogueVerifyExist-

4SEQLogin-“JohnB”, “QA2010”

5CFWelcomeWindowVerifyExist-

KDT Test Case - Optimized

KDT Test Design

Traditional description KDT syntax

Test applicable data

KW type – CF, BF or Sequence

KW specific ID Steps parameters

KDT - Metrics

• Common RoI calculations are commonly defined as:– RoI (Return on Investment)= BENEFIT/COST

• Automation Cost = Price Of HW + Price of SW + Development Cost + Maintenance Cost + Execution Cost

• Manual Testing Cost = Development Cost + Maintenance Cost + Execution Cost

RoI = manual testing cost - automation cost

automation cost

• We suggest otherwise…

Time to Test – where KDT covered 35%

Defect Detection

Breakeven Calculation

• KDT test case implementation takes 3 times more than traditional\ manual TC

• This “extra effort” can be replaced with 3 manual executions of a given TC

• In KDT era, test execution time is not a matter of constraint. A given TC can now be executed numerously with no effort invested

Rules of Thumb for KDT Imp.

• SW\ system under test should be:– Has to be at least medium size dev. effort (12000 dev.

Hours)– Has already began development and it is possible to execute

test on– At least 50% of the TC can be done by a certain UI (the SUT

doesn’t necessarily has UI, can be simulators, etc.)

• Test team\ organization should consider:– KDT is simple yet requires good and talented functional

testers– All beginnings are hard

Conclusion

• Where thumb rules apply, KDT can be implemented with much better chance to succeed

• Where traditional automation applies KDT can increase coverage, effectiveness and reduce cost of automation

• We propose a risk free KDT imp. process using generic tools debugged in many projects of many kinds

• KDT improves the “quality of life” of the functional testers - less tedious execution, more planning and analysis

Personal Experience

• Life is now better for me as a functional tester

• Defects are being unveiled during the KDT test case implementation much earlier than in traditional TC definition

• New options are now available for us i.e 24 hours scenario of functional capabilities

• Assets of the KDT diffused to other teams such as SW integrators build tests, developer unit tests, algorithms test etc

10x!avirams@qualitest.co.il

The future of KDT testers...