combinatorial test design - software means engineering · "combinatorial test design" …...

Combinatorial Test Design Juergen Heymann August 2013

©  2012 SAP AG. All rights reserved. 2

The Problem of Testing


"Combinatorial Test Design"

… is a 'black box' test technique

views function of software from the outside

combines several test design techniques

to deal well with the problem of 'all combinations'


Step 1: Finding the Use Cases / Test Cases

� Ask "What can I do here?" ('user action') � Each action triggers a behavior / function

–  'Enter' / action-buttons but also menu items, tool bar clicks, …

– API: call of a service / function / method / … � Each action / 'use case' becomes a test

case

p1,p2,p3

What can I do here?

p4

F8

menu1

tool3

Assume you have a certain scope of functionality to test…

Step 1. Find 'use cases' / test cases

This should give you a complete list of possible actions / test cases that cover the test object – and have to be tested (test = setup-action-verify)


Step 2: Find the parameters of the test / action

Differentiate between Ø  Dummy Data: just needed to run tests, not varied

in testing, e.g. customer address, article name Ø  Test Model Relevant Parameters: affect the

behavior of code under test (CUT)

Test Relevant Parameters can be found in Ø  Direct Input e.g. parameters in API, on screen Ø  Application Configuration Ø  Object Data in DB

Step 2. Find parameters of the test case

Find parameters

Exec

This gives you the set of parameters that are relevant for this test case


Parameter Oriented Test Design

These 4 test design methods helps us to cover the parameter space in a smart way to find more defects with less effort!

The methods focus on functional correctness.


1: Identify Equivalence Classes Example: Insurance Handling


2: Analyze Boundary Values … to be considered in edge cases


Handling Multiple Parameters


3: Decision Tables Example: Account Opening


But …

Many functions/test cases have more than 4 parameters

Let’s focus on a method that supports this case very well …


4: All-Pairs Testing


Use All-Pairs Testing … to test big parameter spaces efficiently


An All-Pairs Example


All-Pairs Tool Support

You cannot generate 'minimal pair coverage' by hand! �  We use the PICT engine from Microsoft + XLS as wrapper for convenience

Tool DEMO


A Case Example: Combining all Techniques EPOS Mix&Match

All these fields are irrelevant for model (à 'other test data')

Single field errors need to be tested, e.g. date in the future etc. BUT: Error checks are outside of All-Pairs model scope

First set of relevant parameters: MMCriteria: Unique, Match, Any CompleteMM: 0,1 ProratedMM: 0,1 AllowRetMM: 0,1


Combining all techniques 2

EligibilityRule1: ItemKeyExisting, NoDiscToAll, AllItems, Cosmetics, VenderSelection, UnitChargeDiscount Operation1: Greater|GreaterEQ, EQ, LessEQ|Less # aliasing : we consider Greater|GreaterEQ 'almost the same' / equivalent; they are iterated but generate no addtl. pair coverage Qty : 2 # Qty: states how many items of the same type (item key) need to be bought before the special price / … applies. # Qty=2 covers the case that the program identifies at least 2 separate items to belong to the 'special' (as 'condition items'). # You need the proper test data to 'trigger' this quantity Discount1: DollarOff, PctOff, SellPrice # Amount: ignored for model, just use any test data; error checks: amount off > price, pct off > 100, Sell price > orig price ApplyTo1: Item, Group # constraint: LowestPriceItem only used with EligibilityRules; item/group only with item LowestPriceItems1: 0,1 # count semantics? Count1: 0, 1, 10

ItemKey: must exist, but is single-field error condition è ignore in model


All-Pairs Summary

All-Pairs in Practice � Some advanced modeling concepts needed: Conditionals, Alias Groups � Models generate 'strange cases' where one sometimes must think hard

what is supposed to happen

ROI � On average, All-Pairs found 4x the defects-per-test-effort compared to

'manual test design'

Links � Overview of AllPairs / Pairwise Tools (incl. PICT):

http://www.pairwise.org/tools.asp

Thank you

Contact information: Juergen Heymann [email protected]


Decision Tables 5 Redundancy and Inconsistency

Decision Tables can be Redundant and/or Inconsistent Redundant := More than 1 rule defined for the same set of condition values Inconsistent := Inconsistent actions defined for the same set of condition values �  Needs more than 1 rule defined for same set of condition values

Redundancy: # of actually defined cases is greater

than 'Full case count'

Tool shows redundancy: both rules cover n,y,n case Inconsistency:

Inconsistent actions specified for n,y,n

If you avoid redundancy (tool shows this) you avoid inconsistency

ECP

State

BVA DT

AP


All-Pairs Testing Exercise 2: Test FINDSTR (a command line tool)

Develop All-Pairs test model for FINDSTR but ignore regular expressions for 'strings' to be searched for

Searches for strings in files. FINDSTR [/B] [/E] [/L] [/R] [/S] [/I] [/X] [/V] [/N] [/M] [/O] [/P] [/F:file] [/C:string] [/G:file] [/D:dir list] [/A:color attributes] [/OFF[LINE]] strings [[drive:][path]filename[ ...]] /B Matches pattern if at the beginning of a line. /E Matches pattern if at the end of a line. /L Uses search strings literally. /R Uses search strings as regular expressions. /S Searches for matching files in the current directory and all subdirectories. /I Specifies that the search is not to be case-sensitive. /X Prints lines that match exactly. /V Prints only lines that do not contain a match. /N Prints the line number before each line that matches. /M Prints only the filename if a file contains a match. /O Prints character offset before each matching line. /P Skip files with non-printable characters. /OFF[LINE] Do not skip files with offline attribute set. /A:attr Specifies color attribute with two hex digits. See "color /?" /F:file Reads file list from the specified file(/ stands for console). /C:string Uses specified string as a literal search string. /G:file Gets search strings from the specified file(/ stands for console). /D:dir Search a semicolon delimited list of directories strings Text to be searched for. [drive:][path]filename Specifies a file or files to search.

ECP

State

BVA DT

AP


All-Pairs Exercise 2: FINDSTR (Windows) Commented solution (Microsoft)

All-pairs model for FINDSTR parameters Note: regular expression arguments (for strings to be found) not shown here Trailing commas do not generate 'empty' value

This is a sample solution from Microsoft


All-Pairs Exercise 2: FINDSTR (Windows) Approach2: Strict outside view

All-pairs model for FINDSTR parameters Note: We do consider FINDSTR strictly from the outside, i.e. we model whatever the user can enter as options – not what we think the software should do Only when options are truly mutually exclusive (you get error message), we put them as alternatives in the model (à parameter with alternative values, not constraints)

This is a sample solution from us (SAP)


© 2013 SAP AG. All rights reserved.

No part of this publication may be reproduced or transmitted in any form or for any purpose without the express permission of SAP AG. The information contained herein may be changed without prior notice.

Some software products marketed by SAP AG and its distributors contain proprietary software components of other software vendors.

Microsoft, Windows, Excel, Outlook, PowerPoint, Silverlight, and Visual Studio are registered trademarks of Microsoft Corporation.

IBM, DB2, DB2 Universal Database, System i, System i5, System p, System p5, System x, System z, System z10, z10, z/VM, z/OS, OS/390, zEnterprise, PowerVM, Power Architecture, Power Systems, POWER7, POWER6+, POWER6, POWER, PowerHA, pureScale, PowerPC, BladeCenter, System Storage, Storwize, XIV, GPFS, HACMP, RETAIN, DB2 Connect, RACF, Redbooks, OS/2, AIX, Intelligent Miner, WebSphere, Tivoli, Informix, and Smarter Planet are trademarks or registered trademarks of IBM Corporation.

Linux is the registered trademark of Linus Torvalds in the United States and other countries.

Adobe, the Adobe logo, Acrobat, PostScript, and Reader are trademarks or registered trademarks of Adobe Systems Incorporated in the United States and other countries.

Oracle and Java are registered trademarks of Oracle and its affiliates.

UNIX, X/Open, OSF/1, and Motif are registered trademarks of the Open Group.

Citrix, ICA, Program Neighborhood, MetaFrame, WinFrame, VideoFrame, and MultiWin are trademarks or registered trademarks of Citrix Systems Inc.

HTML, XML, XHTML, and W3C are trademarks or registered trademarks of W3C®, World Wide Web Consortium, Massachusetts Institute of Technology.

Apple, App Store, iBooks, iPad, iPhone, iPhoto, iPod, iTunes, Multi-Touch, Objective-C, Retina, Safari, Siri, and Xcode are trademarks or registered trademarks of Apple Inc.

IOS is a registered trademark of Cisco Systems Inc.

RIM, BlackBerry, BBM, BlackBerry Curve, BlackBerry Bold, BlackBerry Pearl, BlackBerry Torch, BlackBerry Storm, BlackBerry Storm2, BlackBerry PlayBook, and BlackBerry App World are trademarks or registered trademarks of Research in Motion Limited.

Google App Engine, Google Apps, Google Checkout, Google Data API, Google Maps, Google Mobile Ads, Google Mobile Updater, Google Mobile, Google Store, Google Sync, Google Updater, Google Voice, Google Mail, Gmail, YouTube, Dalvik and Android are trademarks or registered trademarks of Google Inc.

INTERMEC is a registered trademark of Intermec Technologies Corporation.

Wi-Fi is a registered trademark of Wi-Fi Alliance.

Bluetooth is a registered trademark of Bluetooth SIG Inc.

Motorola is a registered trademark of Motorola Trademark Holdings LLC.

Computop is a registered trademark of Computop Wirtschaftsinformatik GmbH.

SAP, R/3, SAP NetWeaver, Duet, PartnerEdge, ByDesign, SAP BusinessObjects Explorer, StreamWork, SAP HANA, and other SAP products and services mentioned herein as well as their respective logos are trademarks or registered trademarks of SAP AG in Germany and other countries.

Business Objects and the Business Objects logo, BusinessObjects, Crystal Reports, Crystal Decisions, Web Intelligence, Xcelsius, and other Business Objects products and services mentioned herein as well as their respective logos are trademarks or registered trademarks of Business Objects Software Ltd. Business Objects is an SAP company.

Sybase and Adaptive Server, iAnywhere, Sybase 365, SQL Anywhere, and other Sybase products and services mentioned herein as well as their respective logos are trademarks or registered trademarks of Sybase Inc. Sybase is an SAP company.

Crossgate, m@gic EDDY, B2B 360°, and B2B 360° Services are registered trademarks of Crossgate AG in Germany and other countries. Crossgate is an SAP company.

All other product and service names mentioned are the trademarks of their respective companies. Data contained in this document serves informational purposes only. National product specifications may vary.

The information in this document is proprietary to SAP. No part of this document may be reproduced, copied, or transmitted in any form or for any purpose without the express prior written permission of SAP AG.


© 2013 SAP AG. Alle Rechte vorbehalten.

Weitergabe und Vervielfältigung dieser Publikation oder von Teilen daraus sind, zu welchem Zweck und in welcher Form auch immer, ohne die ausdrückliche schriftliche Genehmigung durch SAP AG nicht gestattet. In dieser Publikation enthaltene Informationen können ohne vorherige Ankündigung geändert werden.

Die von SAP AG oder deren Vertriebsfirmen angebotenen Softwareprodukte können Softwarekomponenten auch anderer Softwarehersteller enthalten. Microsoft, Windows, Excel, Outlook, PowerPoint, Silverlight und Visual Studio sind eingetragene Marken der Microsoft Corporation. IBM, DB2, DB2 Universal Database, System i, System i5, System p, System p5, System x, System z, System z10, z10, z/VM, z/OS, OS/390, zEnterprise, PowerVM, Power Architecture, Power Systems, POWER7, POWER6+, POWER6, POWER, PowerHA, pureScale, PowerPC, BladeCenter, System Storage, Storwize, XIV, GPFS, HACMP, RETAIN, DB2 Connect, RACF, Redbooks, OS/2, AIX, Intelligent Miner, WebSphere, Tivoli, Informix und Smarter Planet sind Marken oder eingetragene Marken der IBM Corporation.

Linux ist eine eingetragene Marke von Linus Torvalds in den USA und anderen Ländern.

Adobe, das Adobe-Logo, Acrobat, PostScript und Reader sind Marken oder eingetragene Marken von Adobe Systems Incorporated in den USA und/oder anderen Ländern.

Oracle und Java sind eingetragene Marken von Oracle und/oder ihrer Tochtergesellschaften. UNIX, X/Open, OSF/1 und Motif sind eingetragene Marken der Open Group.

Citrix, ICA, Program Neighborhood, MetaFrame, WinFrame, VideoFrame und MultiWin sind Marken oder eingetragene Marken von Citrix Systems, Inc. HTML, XML, XHTML und W3C sind Marken oder eingetragene Marken des W3C®, World Wide Web Consortium, Massachusetts Institute of Technology. Apple, App Store, iBooks, iPad, iPhone, iPhoto, iPod, iTunes, Multi-Touch, Objective-C, Retina, Safari, Siri und Xcode sind Marken oder eingetragene Marken der Apple Inc.

IOS ist eine eingetragene Marke von Cisco Systems Inc. RIM, BlackBerry, BBM, BlackBerry Curve, BlackBerry Bold, BlackBerry Pearl, BlackBerry Torch, BlackBerry Storm, BlackBerry Storm2, BlackBerry PlayBook und BlackBerry App World sind Marken oder eingetragene Marken von Research in Motion Limited.

Google App Engine, Google Apps, Google Checkout, Google Data API, Google Maps, Google Mobile Ads, Google Mobile Updater, Google Mobile, Google Store, Google Sync, Google Updater, Google Voice, Google Mail, Gmail, YouTube, Dalvik und Android sind Marken oder eingetragene Marken von Google Inc.

INTERMEC ist eine eingetragene Marke der Intermec Technologies Corporation.

Wi-Fi ist eine eingetragene Marke der Wi-Fi Alliance. Bluetooth ist eine eingetragene Marke von Bluetooth SIG Inc.

Motorola ist eine eingetragene Marke von Motorola Trademark Holdings, LLC. Computop ist eine eingetragene Marke der Computop Wirtschaftsinformatik GmbH.

SAP, R/3, SAP NetWeaver, Duet, PartnerEdge, ByDesign, SAP BusinessObjects Explorer, StreamWork, SAP HANA und weitere im Text erwähnte SAP-Produkte und -Dienst-leistungen sowie die entsprechenden Logos sind Marken oder eingetragene Marken der SAP AG in Deutschland und anderen Ländern. Business Objects und das Business-Objects-Logo, BusinessObjects, Crystal Reports, Crystal Decisions, Web Intelligence, Xcelsius und andere im Text erwähnte Business-Objects-Produkte und -Dienstleistungen sowie die entsprechenden Logos sind Marken oder eingetragene Marken der Business Objects Software Ltd. Business Objects ist ein Unternehmen der SAP AG. Sybase und Adaptive Server, iAnywhere, Sybase 365, SQL Anywhere und weitere im Text erwähnte Sybase-Produkte und -Dienstleistungen sowie die entsprechenden Logos sind Marken oder eingetragene Marken der Sybase Inc. Sybase ist ein Unternehmen der SAP AG.

Crossgate, m@gic EDDY, B2B 360°, B2B 360° Services sind eingetragene Marken der Crossgate AG in Deutschland und anderen Ländern. Crossgate ist ein Unternehmen der SAP AG.

Alle anderen Namen von Produkten und Dienstleistungen sind Marken der jeweiligen Firmen. Die Angaben im Text sind unverbindlich und dienen lediglich zu Informations-zwecken. Produkte können länderspezifische Unterschiede aufweisen. Die in dieser Publikation enthaltene Information ist Eigentum der SAP. Weitergabe und Vervielfältigung dieser Publikation oder von Teilen daraus sind, zu welchem Zweck und in welcher Form auch immer, nur mit ausdrücklicher schriftlicher Genehmigung durch SAP AG gestattet.

combinatorial test design - software means engineering · "combinatorial test design" …...

Documents