Advanced Test Design

Methods

Sharon Elgarat 01/July/2013

• What is a Perfect world?• Designer in a Perfect world

• Testing in a Perfect world• Design down to Earth• Business is King• Simplicity &Complexity• The Data factor• Risk based Design• Next Generation

Advanced Design Methods

Sharon Elgarat

What is a Perfect World?

Business People never change their mind

Methodologies always assume the world is perfect:

Documents are always clear and complete

Executing TCs people read every word

Documents are always updated

The solution addresses all needs

Communications flow is immediate

We review to present more then to approve

Design can be automatically created from Requirement documents

Feel

Designer in a Perfect world

Usability

Look

The Tester is the first user to experience the system

No Amount of Requirements definitions and pre-planned Automatic tests could ever replace the inputs coming from a User who looks at a screen

While the defects are considered Low Severity they are the ones which Prove SI value How are the Icons, Logos, Fonts, Details representation shown on a screen? Is information presented to the customer understandable when trying to perform

an action or overcome a mistake? How high is the Density of information presented to the Eye at any given moment?

Even in a Perfect world:Smart Manual Design can guide a Tester to investigate efficiently the solution

• Where is the Designer’s Value?

• Aspects of a Smart Design• Design from Start to Finish

• Testing in a Perfect world• Design down to Earth• Business is King• Simplicity & Complexity• The Data factor• Risk based Design• Next Generation

Advanced Design Methods

Sharon Elgarat

Where is the Designer’s value? The world is far from perfect!!! And should never strive to be!!!

Overall goal of the Designer:Plan tests which will produce the confidence needed to take decisions, and can be executed within given timeframe.

Secondary Crucial Targets:

• Smart and Efficient

• Be Clear and Realistic

• Guide the Executer to think

• Presentable Design

Creative

Aspects of a Smart Design• All effort invested in planning/design gives value to the execution.

• Read documents while extracting points for design• Give the executor a single design document• Collect all non execution information to the strategy• Edit all Reused materials to fit your project/stream• Better to keep an activity Empty then detail unclear steps

• Inputs from multiple sources were used early in the design process• Use Document Center to get background information• Contact the Developer even before you start reading• Generate a quick presentable list of your flows• Share for Review High level design before emotionally

Investing in Detail design

• Dependencies were put only in places where they improve the quality of the test• Placing two functions in a flow adds a risk of missing one of them• Limiting to use specific data my delay execution of the test• Adding an external system validation forces integrated env• Running batch activities requires alignment

Aspects of a Smart Design• Tests added only in places they increase the coverage of the scope

• End the flow when it’s value ends• Avoid copy/past of the same validations• Every flow has a clear business purpose• Manage a data coverage table to be sure all is used efficiently

• Tests added in to a calendar have both Technical & Business reasoning for grouping• Start Design by Defining the scope of each calendar• Never fear from too small calendars• Split Calendars based on Business scope• Split Calendars based on large Batch dependency• Look for Common Functions based on Business Process

• Risk calculated by the actions the user will do and not what system allows• Add tests based on Risk to the Business• Prioritize based on Customer inputs and Usability• Concentrate on realistic Business Use cases• Always keep in mind the Customer’s Master target

Aspects of a Smart Design• Retesting full flow must be simple and feasible within reasonable timeline

• Flows with multiple targets are harder to retest• Flows with time dependency should be very focused• Flows length must fit the timetable provided• Consider possible WA options during the design

• The execution of the Designed tests feet the timeline and resources of the project• Prioritize so if limitations arise change would be quick• Estimate at least 15% of execution as added Ad-Hoc• Design the amount of tests to leave contingency• Order the tests based on Priority• Make sure Value output is reached at all points of execution

• The Designed TCs please the Customer• Start Design after understanding the high level customer’s goal• Search for inputs the customer gave earlier on• During Review collect all customer inputs• When revising include all Customer inputs even at expense of

other tests you feel are important

Quality

Time

ResourcesScope

Design from Start to Finish1. Gather Inputs:

DC Docs Talk to Dev Previous AL Customer perception

2. Analyze Requirements to Extract Use Cases:Describe the User experience

3.1. Breakdown the Detailed Activities:

3.2. Share your view:

Review and UpdateBuild Activity Library

4. Design Full flows:Describe highlights Map data Requirement Coverage

5. Detailed Design:Build Calendar in ATS Export to QC Generate Report in ATS

Winning Design

• Business Process at all Design levels

• Customer Centered – what makes your Account Special

• Testing in a Perfect world• Design down to Earth• Business is King• Simplicity & Complexity• The Data factor• Risk based Design• Next Generation

Advanced Design Methods

Sharon Elgarat

Business Process at all Design Levels

Business is King

Business Use CasesExample: A Subscriber Migrate a SOC with Multiple subscribers, Add Multi SIM, Buy Data Pass, Usage query on customer with multiple subscribers, free and counted traffic

Concept: A high, high level description of the flow, focused completely on the functions tested , addressing information in business language only.

Purpose: Allows The designer to extract quick valuable Testing output which can be reviewed by Business Units to provide their perception of the testing scope. Gaining feedback at an early stage allows the designer to incorporate the remarks and the spirit of the Business outlook as the designer still did not have time to get attached to his own concepts of what’s important to test.

Method: Start from the Business Requirements, gather the functions which need to be tested in to a diagram detailing the flow within

all known business processes (COP, CHQ, PLC, MRL, CCS, BDA, CP, CDNP, ATC, MPR). Break down the diagram to it’s different flows and describe use cases for every Top Risk permutation of the flow. Each Requirement must be reflected in at least one Use Case, Requirements with conditions or parameters will require

multiple use cases to cover them. Information form multiple requirements can be gathered to one Use case but make sure to do so only if it gives value to your

test Every Business Use case must detail 3 sections:

Data origin – New, Converted/Migrated or just Select – if selection has no added value, create new to be sure the tester will reach the important functions BP functions – combine between BPs if there is specific value to it, if not remain within the BP End point validation – always use the user’s real method of validating (online query or a next action)

Business Process at all Design LevelsBusiness Process Layer for Activity Library

Example:

Concept: An Added layer of Activities in the AL which joins activities from different applications to describe the full business process. It doesn’t contain end steps, but links to the original activities in their Application tree. The BPAL layer keeps the order of activities using sequences

Purpose: Provide designers with a clear view of each business process without a need to lookup in each application. Allow reuse of the same activity for different business processes. Quick modification of BP level customizations when the base activities remain the same

Method: Once the Application level Activities are created open under the main AL directory a Directory named ‘_BP’. Under it place directories for each BP. Under each BP create sequenced directories for each section in the BP.Create Sequenced TCs for each Activity in each section. If there is more then one option for an activity, place the same Sequence. Advance the sequence based on the steps in the process. Optional steps must be marked.Use the ‘Call to Test’ to link to the original activity – so each activity in the BPAL contains just one link Business is King

CRMCustomer Activities

Billing Arrangement ActivitiesSubscriber Activities

Customer CreationIndividual Customer CreationBusiness Customer CreationCorporate Customer Creation

Customer MaintenanceDescription Expected Results

1 Open CRM Interaction overview shows

2 Click Menu Create create Customer

Create Customer screen opens

3 Populate first name

Field accepts minimal 2 chars

… ……………………. ……………………..

OMS_Provide Order

Change OrderSuspend Order

001_Start Order002_SOAP – Select Order Action & Product003_NIA – Negotiate Installation Address004_NPC – Negotiate Product Configuration005_NCD – Negotiate charge Distribution………

COP01 Customer

011_Individual Customer Creation011_Business Customer Creation011_Corporate Customer Creation

02 Financial Account & Billing Arrangement

021_Create FA and BA - CC

Description

1 Go to Individual Customer Creation

03 Order

031_Start Order032_SOAP – Select Order Action & Product033_NIA – Negotiate Installation Address034_NPC – Negotiate Product Configuration035_NCD – Negotiate charge Distribution………

04 DB Verifications

ATC

CHQBDA

Business Process at all Design LevelsBusiness Process Oriented Testing flowsExample:

Concept: A high level description of a flow representation of the Business Use cases, done either in word or excel and is later also used as base for test results documentation. Provides the reader complete understanding on ‘What it is we Test’ leaving out only details related to ‘How we test’.

Purpose: Presents to Reviewers and the Executer all the details needed to understand our testing plan and the method in which we cover our scope.

Method: Breakdown the Use Cases in to detailed flows, each use case must be represented by one scenario. From the Development HLDs extract details related to the exact sub steps of each action and the many parameters which must be covered to insure a function works. Detail in the bullets of the flows the following types of information: The System being used The function executed in the test The values which need to be selected to perform the test – focus on values which are needed for generation of

the intended flow. A validation which can be done as a direct result of the action performed (no added action needed) – validations

which require an added action should be detailed as a separate bullet

Business is King

• Advantages of each approach

• Balance Point

• Testing in a Perfect world• Design down to Earth• Business is King• Simplicity & Complexity• The Data factor• Risk based Design• Next Generation

Advanced Design Methods

Sharon Elgarat

Bill

Advantages of each ApproachThe Business Process Oriented Approach:

Examples:

When to Use:

Strong Points:

Order Use Change Use Bill Complain UseChange

Best use when expecting unstable behavior and high rate of Critical issues. Types of Activities for which this approach works well for are:

Green Field project Conversion Project New Business Line on existing System Large Customization with dramatic CRs Defect Retesting portion of a Maintenance Delivery

Allows to see and reflect to the customer clear results for each function – to facilitate difficult decision making.

Enables Defect retesting based on Full flow – to insure recreation of all conditions contributing to the previously defective flow

Permits running all flows in parallel giving full system indication of quality even if some functions have critical issues

Allows Focusing and re-running Specific Business Processes at higher rate then others

/ / /

Re-Bill

Advantages of each ApproachThe Full length, all in one Approach:

Example:

When to Use:

Strong Points:

Order Use Complain Change Use Bill Complain UseChange

Best use when expecting flows to pass without generating damaged data – meaning expected errors are on screen or flow levels, meaning each individual action will work but the flow will not end with the expected result (for example event is rated but with wrong rate)

Types of Activities for which this approach works well for are: Regression for stable System Large number of Small GUI CRs touching the same Business Process

Allows to see a real, full, user experience of the system Makes full use of each Resource, reusing the data created by one action as base for the next thus

minimizing the number of un-effective test cases for data creation. Insures Higher confidence defects related to complex behavior between seemly unrelated function

are identified during testing Provides the ultimate validation each action works by proving the data can be used for added

activities without error

Balance Point

Fidgeted Unstable

Simplicity Complexity

Accurate results User Experience

Make sure flows can be executed within the time given for Execution of a test round Make sure the scenario name reflects the tests in the scenario – so managers who read the

report will be able to understand what works and doesn’t work Make sure all parameters related to a function are covered, including data types which

require a previous action to be done to generate the data When there are two Options to generate the same end state of a test, choose the shorter

option. Combining unrelated tests which don’t give real value to each other must be avoided Join to one Bullet actions and validations when the validation is a direct result of the action Hint to it when expecting the user to perform look and feel general validations. A validation should be added only if it’s giving a real new input – avoid Copy/Paste

• Present Coverage• Allow Flexibility for Change• New & Existing data conflict

• Testing in a Perfect world• Design down to Earth• Business is King• Simplicity & Complexity• The Data factor• Risk based Design• Next Generation

Advanced Design Methods

Sharon Elgarat

Present CoverageImplementation & Applicative data Coverage can be presented in two levels: Full Coverage – for parameters which require a small exact specific list (Examples: Rate/Allowance PITs, Customer Type) Cover by Groups – for parameters with large range of data, grouping by Rules or by

Type (Examples: Service filters grouped by Event Type, Customer Address Grouped by Country)

Coverage MatrixBuilt to show in one view the data types covered during testing. Focused on the Parameters being tested.Constructed from the Parameters encountered during the flow of each Business Process.Recommended Structure:

*Update Parameter names and values and add the covering Use cases during the design

Allow Flexibility for ChangeReal Data is always unstable:• Customer Security Policies often prevent from publishing Real Implementation early• Production Implementation is a living, changing element for any Company striving to be #1• Last Minute Regulations can cause unexpected change in Implementation• Delay in Timeline introduces new, unexpected Implementation and Applicative data in to play• Applicative Data is forever changing both in values and in Types• The Go live date determines different types of data reconciliation during conversion projects.

Methods to ease coping with change:Place Final values only in places the specific value generates the flowUse Indexes from Gathering Tables instead of final values when it is not absolutely certain the list if finalAfter reaching full coverage of a Parameter allow Random selection during executionAt any Delay in timeline approach BPT team to find if any updates were made in the implementation gatheringRefresh Data from Production at least once during any version which exceeds 3 monthsFor Migration flows Prepare Extra Designed flow for scenarios which depends on Go live date, keep the designed scenario out of the delivered document, as safety for delay option

The New & Existing Data Conflict

Why Should we Create our own Data: Designer is in Full Control of the Data used No Dependency on Stable Data from Production/Previous Synchronized in all Systems Allows Use of Test Cards for physical device tests Fresh

Why Should we Use Production Data: Test is Meant to Test Production data Conversion Quality Defects Benefit from Real Full Customer situations Quality Defects from local language data Quality Defects from long existing history data Shortcut for long process data creation Real

How to Chose: Using Existing Data must have a reason – if there is no reason go with New If a testing target was achieved by a different scenario using existing, use new When testing Conversion New data can still be used to test converted functions Plan the scenarios which require existing data first, and then move to the functions

which can be tested on newly created data

• All Design is Risk Based• Prioritization Factors

• Testing in a Perfect world• Design down to Earth• Business is King• Simplicity & Complexity• The Data factor• Risk based Design• Next Generation

Advanced Design Methods

Sharon Elgarat

All Design is Risk basedIt is Impossible to do every possible test

The Customer Lifecycle is a never ending Story

A single order can go to Loops of Back , Next and Amend

A large group of small parameters generate a huge testing matrix

Execution Time is limited and LD is not always supported

Choice introduces Risk Critical Tests or entire functions might be missing from the design

Focus could be placed on un important elements

Impossible situations could be designed when trying to minimize

Tests could go far from the customer’s main use cases

Covering matrixes might not leave time for creativity

TC

Prioritization Factors

1. Main Business Target related Use Cases2. Creative Use Cases combining multiple related functions3. Use Cases targeting race conditions of data flow between systems4. Use Cases targeting race conditions of data flow dependent on time5. Use Cases aimed for Edge particular populations

Follow the Business Use Cases as Top Priority:

What if we can’t cover all values of a Parameter?1st Priority: Most Commonly Used in Production2ed Priority: Most likely to assist in introducing Complexity to the flow3rd Priority: Reproduces a known former DefectSurely Avoid:

• Hasn’t got even 1 existing case in Production• Has been declared by Business as out of scope of the Function tested

How to make sure our focus is ‘Acceptance Test’?1. When checking coverage matrix avoid adding too many tests on Unit test level

validations just to complete the coverage2. When designing a Test always keep the User in Mind, ask yourself if the

created test is something that can happen to a User

• Testing in a Perfect world• Design down to Earth• Business is King• Simplicity & Complexity• The Data factor• Risk based Design• Next Generation

Advanced Design Methods

Sharon Elgarat

Are you Tired of Testing?

Tired of writing design no one ever Reads?

Frustrated with Unclear descriptions from your designers?

Don’t want to spend so much time on ReadingWhen you should be executing?

Come See the Revolution

The Revolution: Visual tests, comparing ‘apples’ to ‘apples’

Expected Result:The Design by Mockup View

Actual Result:The System view

Passed

Passed

Failed

No Run

Visual Testing Method Design Process

In the Tool:Gathering GenericActivity LibraryOf Mockups

Out of the Tool:Designer Maps the Scope & Selects the Use Cases

In the Tool:Designer createsfor each use casea mockup flow in the tool with testsDefined at each mockup

* Phase 2 will provide a solution within the tool for Mapping of the scope and mathematically selecting Use cases

In the Tool:AutomaticallyExport TCs toQuality CenterKeeping a link to the tests in the Tool

Visual Testing Method Execution

The Executer places the Tool’s View on one screen and the System on the other

The tool describes visually the tests to perform and allows to Pass or Fail a test within the tool

Passed

Failed

No Run

No Run

*

PassedFailed

No Run

No Run

No Run

* Fully Integrated with QC

Phase 2 – Shooting over the MoonBusiness Process Integration within the Tool

Enhancement 1: Generic Sub Flows as Elements of BP Activity library

Enhancement 2: Diagram flow description of the full BP tree

Enhancement 3: Automatic Pairwise Selection of Optimized Use Cases

Thank YouSharon Elgarat