Testing Strategy & ExecutionThe Key to Implementation Success

By: Nick Lagen nlagen@hitachiconsulting.com

Agenda

We will answer the following questions…

• Why do we need a disciplined test strategy?• What is this disciplined test strategy?• Who does what?• What are the testing milestones?• How can we accomplish this?

…to effectively manage project test activities.

Why do we need a disciplined test strategy?• To effectively manage test activities

– Manage formal test sign-offs and organizational buy-in– Obtain real-time visibility to testing progress– Reduce project cost and risks– Communicate progress to the Steering Committee

• To maximize team productivity– Clearly define the test cases and scenarios– State concise and factual test expectations and results– Optimize training material content

What is this disciplined test strategy?

Know where you are going!

Define and Integrate your test components

Functional Team ABAP Team Basis Team Functional andABAP Team

Configuration Configuration DocumentsDocuments

Bus. Process Bus. Process Procedures Procedures

(BPPs)(BPPs)

4.1 Process 4.1 Process Owner Owner

Acceptance Acceptance TestsTests

Business Business Process Process MappingMapping

Functional Functional SpecificationsSpecifications

1.1 Bus. 1.1 Bus. Process Process

Procedure Procedure Unit TestsUnit Tests

1.2 BPP 1.2 BPP Scenario Scenario

TestsTests

1.3.1 Business 1.3.1 Business Integration Integration

Test 1Test 1

1.3.2 Business 1.3.2 Business Integration Integration

Test 2Test 2

3.2 SAP 3.2 SAP System System

Stress TestStress Test

4.2 4.2 Authorization Authorization

TestTest

5.1 Go Live5.1 Go Live

2.2 Custom ABAP 2.2 Custom ABAP Functional TestsFunctional Tests

2.3 Custom ABAP 2.3 Custom ABAP Prototype TestsPrototype Tests

2.4 Custom ABAP 2.4 Custom ABAP System System

Integration TestsIntegration Tests

2.5 Data 2.5 Data Migration Migration

TestsTests

2.1 ABAP 2.1 ABAP Code Unit Code Unit

TestsTests

ABAP CodingABAP Coding

Technical Technical SpecificationsSpecifications

3.1 3.1 InfrastructureInfrastructure

TestTest

4.2 4.2 Authorization Authorization

TestTest

Define your SAP business process validation approachFunctional Testing

1.1) Business Process Procedures – Unit Test• Test individual SAP transactions defined as the execution

of a single menu path or transaction code (e.g.: create a sales order).

1.2) Business Process Procedures – Scenario Testing• Test baseline business processes consisting of linked

native SAP transaction codes (e.g.: order to cash).• No custom development objects are tested.

1.3) Integration Testing (1.3.1 and 1.3.2)• Test linked SAP transactions, both native SAP transaction

codes and ABAP programs (e.g.: end-to-end).• The objective is to validate the business will operate as

expected after go live.

Define your custom programs validation approachDevelopment Testing

2.1) ABAP Code Unit Tests• The programmer tests their ABAP program

2.2) Custom ABAP Functional Tests• The ABAP program is delivered to the functional owner for

testing.

2.3) Custom ABAP Functional Prototype Tests• The baseline business processes are tested with their

requisite ABAP programs.

2.4) Custom ABAP System Integration Tests• Customized system interfaces are tested while connected

to external systems (Legacy, etc.).

2.5) Data Migration Tests• Test custom legacy data extraction programs, mappings to

SAP fields, and custom SAP ABAP loading programs

Define your Basis system validation approachSystem Testing

3.1) Infrastructure Testing• Hardware, software, network, location testing, etc.• The test facility computers, printers, software, and

network are proven to work for project testing

3.2) SAP System Stress Testing• The expected transactions are tested to determine the

ability of the SAP system to withstand expected real-world transaction volumes and frequencies.

Define what else will be validatedOther Tests

4.1) Process Owner Acceptance Tests• Demonstrate key scenarios to achieve approvals

4.2) Authorization Testing• Validate defined security profiles provide end-users access to

required transactions.

4.3) Regression Tests• Previously executed tests in any of the test components are

retested if significant changes occur to related SAP configuration settings, ABAP programs, or Basis parameters.

5.1) Live SAP System• “What you don’t test during the project, you will test after go

live”

Now write your formal test strategy document• Describe the detailed activities for each test component

– Entrance Criteria • Specifies the preceding conditions required to perform the test

– Test Procedures• Specifies the details on how the test is to be performed

– Exit Criteria • Specifies the success threshold for test completion and the test outputs

– Testing Roles• Defines who does what and when

– Test Cases Development• Defines what specifically will be tested and when (if applicable)

– Systems Used• Defines the required IT systems

– Infrastructure Required (desks, etc.)

Define test success criteria:Elaborating on exit criteria

• High priority “must have” test items– The project cannot go live until these items are tested

and signed off.

• Medium priority “want to have” test items– The project can go live but will encounter some issues if

these are not tested and signed off.

• Low priority “nice to have” test items– The project can go live without any related issues.– These test items are typically deferred to a post-

implementation effort.

Who does what?

Define and manage the team that will deliver

the test results!

Define your test organization

• There are several test organization options to chose from:– Option 1: Highly centralized– Option 2: Moderately centralized– Option 3: Decentralized– Option 4: Highly decentralized– Option 5: Chaos

• Several variables affect how the test organization is established. These include:– Project budget and schedule– Team member skill levels– Sponsor and project team politics

The Test Lead coordinates key Realization Phase activities

Option 1 – Highly Centralized• A dedicated full-time Test Lead is

responsible for managing and coordinating the key non-Basis Realization Phase activities

• Pros– Single test owner– No project management test burden– Structured– Minimized risk

• Cons– Dedicated test FTE– Maximum cost option

• Use of this approach– $4M+ fee projects– Large teams – Hierarchical teams– Complex processes

COLeadFI

Lead

SDLead

BWLead

ABAPLead

CRMLead

APOLead

TestLead

MMLead

PPLead

The Test Lead coordinates unit and integration testing

Option 2 – Moderately Centralized• The strongest functional lead is tasked with the

responsibility for managing and coordinating the unit and integration tests. ABAP unit testing is independently managed by the project manager.

• Pros– Single test owner– Low project management burden– Structured– Marginal risk

• Cons– Burdens 1 functional team – High cost

• Use of this approach– $1M to $4M fee projects– Moderate size teams– Organic project hierarchy– Complex processes

COLead

SDLead

BWLead

ABAPLead

CRMLead

APOLead

FI andTest

Leads

MMLead

PPLead

The Project Manager oversees testing

Option 3 – Decentralized • The project manager actively manages

test details and provides task-level direction to team leads.

• Pros– Moderate cost– Single test owner– Organic structure

• Cons– Burden’s project management– Some loss of control– Moderate risk– Potential for missed tasks

• Use of this approach– $250K to $1M fee projects– Moderate complex processes– Strong project management

Project Mgr.

SD Lead

CRM Lead

BW Lead

APO Lead

CO Lead

FI Lead

PP Lead

MM Lead

ABAP Lead

Functional team leads work together to manage testing

Option 4 – Highly Decentralized• The team leads organically coalesce to

self-manage the project testing activities. • Pros

– Marginal cost– Minimal team burden

• Cons– Unstructured leadership– Minimal controls– Poor reporting– High risk

• Use of this approach– Up to $250K fee projects– Small teams– Collaborative culture– Simple processes

ABAPLead

MMLead

CRMLead

SDLead

BWLead

APOLead

COLeadFI

Lead

PPLead

No defined or assigned test roles

Option 5 – Chaos • The project has no defined testing structure.

Team members are left to their own devices to validate the system satisfies stated business requirements.

• Pros– No cost– No team burden

• Cons– Maximum risk – Missed deliverables– Quality issues

• Use of this approach (if any)– Very simple mySAP projects– 1-5 processes– None to minimum ABAP coding CO

LeadFI

Lead

SDLead

BWLead

ABAPLead

CRMLead

APOLead

MMLead

PPLead

SAP Project Test organization options matrix

Organizational Model Pros Cons Use Impact of not using or using improperly

Highly Centralized

(Dedicated Test Lead)

• Single test owner• No proj. mgt. burden• Structured• Minimized risk

• Dedicated test FTE• Maximum cost

• Above $4M project fees• Large teams • Hierarchical teams• Complex processes

• Poor tracking• Poor reporting• Ineffective teams• Missed milestone dates

Moderately Centralized

(Matrixed Test Lead)

• Single test owner• Low proj. mgt. burden• Structured• Marginal risk

• Burdens 1 fct. team • High cost• Possible task misses

• $1M to $4M project fees• Moderate size teams• Organic proj. hierarchy• Complex processes

• Coordination issues• Poor tracking• Poor reporting• Missed milestone dates

Decentralized

(PM is the Test Lead)

• Moderate cost• Single test owner• Organic structure

• Burden’s project mgt.• Some loss of control• Moderate risk• Missed tasks

• $250K to $1M project fees• Moderate complex

processes• Strong project mgt.

• Ineffective teams• Leadership issues• Poor controls• Missed milestone dates

Highly Decentralized

(Silos Test Mgt.)

• Marginal cost• Minimal team burden

• Unstructured leaders• Minimal controls• Poor reporting• High risk

• Up to $250K project fees• Small teams• Collaborative culture• Simple processes

• Chaos (see below)

Chaos

(No Test Leadership)

• No project testing cost• No team burden

• Maximum risk • Missed deliverables• Quality issues

• Simple mySAP projects• 1-5 processes• Minimal ABAP coding

• Avoid this under most

circumstances

Real-world examples

Project Characteristics

Project A Project B Project C Project D

Test organizationHighly Centralized(Dedicated Test Lead)

Highly Centralized(Dedicated Test Lead)

Moderate Centralized(Matrixed Test Lead)

Moderate Centralized(Matrixed Test Lead)

Project Complexity & Rollouts

Complex Fortune 100 global rollout

Complex regional multi-country rollout

Complex 4 location rollout in US & Canada

Moderate complex 3 country rollout

Plan Project Duration 2+ years 13 months 14 months 9 months

Project Fees Cost, ($) $21 million $5 million $7 million $1.8 million

Project Size, (team) 90+ core and consults 50+ core and consults 65+ core and consults 23 core and consults

Project Culture Rigid & hierarchical Flexible & organic Collaborative & organic Collaborative & organic

How did it turn out?

• Formal test strategy• Disciplined test mgt.• Daily progress report• Executive oversight

• No formal test strategy• Did not define scenarios properly• Poor status reporting

• Formal test strategy• Executive oversight• Test Lead overwhelmed

• Formal test strategy• Loose test mgt.• Executive oversight• Weekly progress report

Outcome

• On-time• On-budget• Good controls• Transparent communications• Good go-live

• 3 month schedule slip• Budget over-run• Reduced quality• Increased risk • Recovered and had a good go-live

• 2 month schedule slip• Budget over-run• Recovered and had a good go-live

• On-time• Project was on-budget• Some go-live issues• Some support costs

Real-world examples

Project Characteristics

Project E Project F Project G Project H

Test organizationDecentralized

(PM is the Test Lead)

Decentralized(PM is the Test Lead)

Highly Decentralized(Silos Test Mgt.)

Chaos(No Test Leadership)

Project Complexity & Rollouts

Moderately complex 1 location project

Simple 1 location project Simple 1 location project Simple 1 location project

Plan Project Duration 6 months 7 months 4 months 4 months

Project Fees Cost, ($) $650 K $750 K $400 K $225 K

Project Size, (team) 10 core and consults 12 core and consults 10 core and consults 5 core and consults

Project Culture Collaborative & organic Independent Rigid & uncooperative Collaborative & organic

How did it turn out?

• Formal test strategy• Disciplined test mgt.• Daily progress report• Executive oversight

• No formal test strategy but strong PM• Weekly test reporting• Marginal Executive oversight

• Formal test strategy• Team overburdened • Teamwork issues• Marginal Executive or PM oversight

• No formal test strategy• No test reporting• No Executive or PM oversight

Outcome

• On-time• On-budget• Low risk• Good controls• Transparent communications

• On-time• Project was on-budget• Moderate scenario validation• Some go-live issues• Some support costs

• On-time• Project was on-budget• Minimal scenario validation• Minor go-live issues• No support costs

• On-time• Project was on-budget• No scenario validation• Major go-live issues• Major support costs

What are the testing milestones?

Clearly define your objectives before starting!

Testing Milestone ChecklistPart IPreparation Phase Define formal and informal test procedures Complete and approve your test strategy Define, approve, and communicate the project testing roles

to the team

Blueprint Phase Create and approve the Business Process Master List

(BPML) from the ASAP Q&A database Create and approve the Development List (DL) from the

ASAP Q&A database

Testing Milestone ChecklistPart IIRealization Phase Define the test scenarios that will validate the system

satisfies the business requirements Assign test cases to the scenarios Define the detailed testing success criteria Team members are regularly updating progress on:

– Configuration– ABAP – Business Process Procedure (BPP) writing– BPP testing– Integration 1 & 2 testing

Perform end user acceptance test Perform executive End-to-End demonstration Sign-off and close the Realization Phase

Testing Milestone ChecklistPart IIIFinal Preparation Phase Validate end user authorizations work as expected. A

realistic time is to do this is concurrently with end user training

Perform system stress and volume tests Perform regression testing to validate that system changes

have not impacted the results of earlier tests

How do we accomplish this?

Your testing procedure and tools are

critical to your success!

Follow this test flow procedure

Define Test Scenarios

Business Process Master List

ApproveScenarios

Type Or Upload Into Scenario

Table

Development List

Manual Process List

Type Or Upload Into Test Case

Table

Assign Sequenced Test Cases To All The Scenarios

Generate Daily Test Schedule

Issue Scenarios To Testers At Start Of Day

Collect Scenarios From Testers At

End Of Day

Testers Perform Assigned Tests

Update Test Tool

Issue Daily Testing Status

Report to Mgmt.Test Feedback & Adjustments

Print Scenarios For Test

Execution

ApproveScenarios

With Detailed

Test Cases

Track & Complete

Unit Testing

Define & Finalize Success Criteria

Use a testing tool to manage Realization Phase testing

Test Manager is one such tool

Manage BPPs, scenarios,

reports, and tool administration from the main

menu

Create or load Scenarios

Maintain scenario approvals

Maintain scenario status

Scenario Pre-Test Approval

Obtain scenario pre-test approvals

Approve the list of scenarios, by

owner, before the first cycle of BPPs are completed

Create or load test cases

Load the BPML and DL into your

test tool

Define and link the BPP and Configuration

document filenames

Maintain the various Unit Test

statuses

Track BPP Unit Test progress

Config. Status - Week Ending w/Status Meeting

0%10%20%30%40%50%60%70%80%90%

100%

Week8/10

Week8/17

Week8/24

Week8/31

Week9/7

Week9/14

Sta

tus

Per

cen

tag

es n/a

Completed

Not Started

In-Process

Issues

BPP Status - Week Ending w/Status Meeting

0%10%20%30%40%50%60%70%80%90%

100%

Week8/10

Week8/17

Week8/24

Week8/31

Week9/7

Week9/14

Sta

tus

Per

cen

tag

es n/a

Completed

Not Started

In-Process

Issues

Unit Test Status - Week Ending w/Status Meeting

0%10%20%30%40%50%60%70%80%90%

100%

Week8/10

Week8/17

Week8/24

Week8/31

Week9/7

Week9/14

Sta

tus

Per

cen

tag

es n/a

Completed

Not Started

In-Process

Issues

Overall Status

0%10%20%30%40%50%60%70%80%90%

100%

Week8/10

Week8/17

Week8/24

Week8/31

Week9/7

Week9/14

Sta

tus

Per

cen

tag

es n/a

Completed

Not Started

In-Process

Issues

Define test case assignments and sequencing to scenarios• Scenarios consist of

discrete manual steps, SAP transaction codes, ABAP programs linked together in a sequence that mimics how the business process actually works

• Defining scenario details is critical to the project’s success

FunctionalUnit TestReports &Finalized

BPPs

DevelopmentObject UnitTest Reports& Finalized

BPPs

Manual

SAP

Code

IntegratedScenario

Define testing success criteria

• Integrated scenarios contain more test case permutations that can be realistically tested during the project.

• Balance the test success criteria with the project resources.

***

***

***

***

Universe of saleable goods

Universe ofcustomers

***

***

**

***

Universe ofvendors

*

Universe of raw materials

***

***

**

***

Universe ofCustomers

*

Distribution options

Step i Step i+1 Step i+j

Assign and sequence test cases to scenarios

First, choose scenario test

steps from the pool of manual

steps, SAP transaction

codes, and ABAP programs

Then, assign & sequence test

steps to a scenario

Scenario Content

Obtain scenario pre-test sign-offs

Approve the list of test steps in a given scenario

Perform scenario tests

Input test step transaction

results into your testing tool

Enter testing errors and resolution

Test a single BPP (step) in a given scenario

Sign-off completed scenario tests

Staple all tested output documents to the

scenario sign-off sheet

Scenario Post-Test Approval

Conclude testing with a formal

scenario sign-off

+

Invoice

PurchaseOrder

Bar Code Label

CheckForm

Track and report scenario test progress

Conclusion

• Your test strategy is dependent on:– Project size, budget, and schedule– Project discipline– Team skill level and motivation– Team culture and politics

• Strike a balance between these variables in creating and executing your test strategy to effectively manage project test activities, increase implementation quality, and reduce project cost.

• Don’t forget that you need a testing tool

Thank you for attending!Please remember to complete and return your evaluation form following this session.

Session Code: [2806]