e-book...proper end-to-end testing essentially implies that we are talking about a bpml (business...

©KTern

1

E-Book – What you should know about SAP S/4HANA Testing Automation

Table of Contents Executive Summary ..................................................................................................................................... 2

What is SAP S/4HANA Testing? ................................................................................................................... 2

What is the SAP S/4HANA Test Lifecycle? ................................................................................................... 2

Problems in SAP S/4HANA Testing .............................................................................................................. 5

A Devil in Disguise – Testing Failures in SAP S/4HANA projects .................................................................. 6

5 Steps for S/4HANA Testing Automation ................................................................................................... 7

Nice To Haves along with Testing Automation ........................................................................................... 9

KTern Features for S/4HANA Testing Automation .................................................................................... 10

8.1 Test Case Identification ..................................................................................................................... 10

8.2 Test Case Execution ........................................................................................................................... 10

8.3 Test Case Reporting ........................................................................................................................... 14

8.4 VVV Scoring (Value, Volume and Velocity Analysis) .......................................................................... 14

8.5 Sign Off Wizard .................................................................................................................................. 14

Next Steps .................................................................................................................................................. 15

About KTern ............................................................................................................................................... 15

©KTern

2


Executive Summary

SAP S/4HANA is an intelligent ERP powered by SAP HANA and SAP Leonardo Platform. SAP S/4HANA provides solutions with tendrils and impacts every nook and corner of its enterprise records and processes. Hence, testing is mandatory and remains critical for every single SAP ECC to SAP S/4HANA migration project.

Enterprises must ensure high quality end-to-end business processes even though they are often committed to continuous change. This results in problems surrounding timely releases and business process quality. In addition, test managers face tremendous pressure to reduce time-to-market cycles. With more and more departments utilizing IT and undergoing “digital transformation”, it is becoming difficult to standardize test management and accelerate test execution. However, comprehensive testing across business-critical applications involving change and process is always needed. Testing should never be a bottleneck for releasing change in any enterprise. Enterprises are now slowly moving to test automation to ensure continuous delivery of change and quality.

However, to meet quality standards when implementing continuous change, enterprises need a single tool which can not only ensure test automation, but also take care of test management, test execution, test tracking, and test collaboration. After all, choosing the right testing solution can remove the problems involved in testing, ensure stress-free testing for test managers, and help enterprises stay ahead of the curve. So, what is the right testing solution for your enterprise? You’ll get the answer at the end of this E-Book.

What is SAP S/4HANA Testing?

SAP S/4HANA testing is the process of verifying the functioning of SAP S/4HANA with an intent to find out

whether the ERP system meets the specified requirements and ensure that the system is defect free. Testing

is essentially a quality assurance process that ensures the correct operation of SAP S/4HANA before the

project implementation Go-Live. Also, testing is not just restricted to the project implementation and project

go-live phase. SAP S/4HANA is a new-generation ERP with regular updates and new releases. So, testing is

necessary at each upgrade and therefore, customers must realize that SAP S/4HANA Testing is a never-

ending process.

What is the SAP S/4HANA Test Lifecycle?

SAP recommends Activate methodology for S/4HANA Projects. Each phase of Activate has its own set of buzz words associated with it. For Discover, it is Value Scoping and Impact Study. Similarly, for Prepare it is Project Team Preparation and for Explore the words associated are Cutover Preparation and Design of the system. Coming to the Realize phase, the keyword associated is Testing, Testing, and only Testing.

As per the SAP Activate Methodology, the test lifecycle consists of three phases:

1. Test Planning 2. Test Preparation and Execution 3. Test Validation

©KTern

3


Image 1: Testing Life Cycle as per SAP Activate

Test Planning

To minimize the number of issues during or after Go-Live, it is critical to manage the quality of the solution. As part of any scheduled maintenance event, it is necessary to consider and plan the testing cycles required to mitigate production support issues. In this stage the following key elements of the test planning must be documented:

1. Project Testing Objectives & Assumptions

➢ Unit Testing is complete before Integration Testing ➢ Unit Testing is only required for delta functionality

2. Test Scope 3. Types of Testing

➢ Unit Testing ➢ Business Process (String) Testing ➢ Integration testing ➢ Data Conversion Testing ➢ User Acceptance Testing

4. Testing Approach

➢ Description on how different test types relate to each other, e.g. how a successful unit test is a prerequisite for doing a string test or how a migration test results might lead into a pre-requisite for a user acceptance testing

5. Testing Deliverables 6. Test processes per phase, test environment, test tools

©KTern

4


7. Test Automation 8. Testing Tools - which tools will be used to perform different tests (e.g. KTern, SAP Solution

Manager) 9. Defect Management

➢ Description of how defects will be documented (e.g. Test Workbench in SAP Solution Manager)

10. Roles and Responsibilities

➢ Description of required test roles and responsibilities ➢ Test lead and responsibilities of individual project team members related to testing

Test Preparation and Execution

As determined in the evaluation of the existing test materials and documented within the testing plan, additional assets may need to be developed to support the execution of the testing cycles. Develop the missing test materials and test scripts in accordance with the detailed test plan.

Test Preparation

Within in each implemented solution scope, the following steps need to be executed:

1. Extend best-practice test cases 2. Develop delta process test cases 3. Finalize integration and user-acceptance test cases and plan 4. Prepare approval procedure 5. Prepare tool adaption and delta user acceptance test training 6. Prepare a test environment with the required test data as defined in the activity Test Planning. Once

the tests have been planned and test data is available on the test systems, testing can begin.

Test Execution

The typical process for the Realize phase is as follows:

• Software developers perform unit tests in the DEV system. Depending on the type and scope of the test cycle, various functional tests are performed.

• Manual testers are provided with the tester handout document and receive details regarding their test package by e-mail.

• Automated tests are scheduled or started directly.

• Every test that is executed is logged and documented with test notes and a test status is set manually or automatically.

• If the system responds in an unexpected way during manual testing, for example, if an error message appears, the tester records the incident in the corresponding ITSM system, attaching screenshots, log messages, and so on. Usually, this also must be done manually even for automated tests.

• The incident is sent to the persons responsible for analysis and categorization of defects, who then corrects the defect in the development system.

• The correction is transported to the test system according to the existing arrangements and timelines, where it is then retested.

©KTern

5


Given the complexity and heterogeneity of modern software solutions, SAP recommends performing Integration Validation, especially for important business processes. This involves gathering and evaluating a substantial amount of data from the software applications that are active while a given business process is being executed. This type of validation allows you to identify hidden warnings and error messages that frequently occur at the interfaces between applications. Furthermore, the operations team should monitor the testing system like it was a production system. This will help gain early visibility and hands-on experience for possible production issues which might arise in the future.

If large-scale changes are made or new software solutions are implemented, load tests should be performed before these are used in production. These tests simulate a situation in which the expected load (known number of users and background load in a load-peak situation) is simulated. In doing so, the system behavior while handling large data volumes can be inspected. Throughout the entire test cycle, test coordinators must monitor the test status and progress, as well as the processing status of incidents that have been reported. The quality of test data and test scripts directly affect the stability of the production system following the Go-Live of the change event. Consider an array of representative variances when preparing for the execution of the regression test cycle. It is important to execute realistic data sets that represent production operations of critical business processes.

Test Validation

The steps involved in the Test Validation Phase are:

1. Prepare User Acceptance Test plan: Update the existing integration test cases, end-to-end customer business process scenarios, based on the learnings from previous test phase. UA test plans and test case documentations are stored in Solution Manager.

2. Prepare and document User Acceptance Test Case #1 – n: The purpose of this task is to document the UA test case outlined in the UA test plan. This activity also contains aligned setup of relevant test data that will be commonly used.

3. Execute User Acceptance Test Case #1 – n: Perform the test according to previously defined plan. During the test all issues must be logged and documented in the system for traceability purposes.

4. Perform defect resolution for User Acceptance Test: Resolve any issue identified during testing. It is crucial that these issues are re-tested by the users that reported them (or designated re-testers) and that they are confirmed.

5. Obtain User Acceptance Test Sign Off: Obtain customer approval (sign-off) of the User Acceptance Test.

Testing is a time-consuming process, but make sure that you do not cut corners. Quality time spent on testing can go a long way in saving you from unforeseen glitches.

Problems in SAP S/4HANA Testing

In today’s digital world, SAP S/4HANA testing remains a great area of concern for both the business users and the business decision makers. The major difficulties encountered by the key persons during SAP S/4HANA testing are –

➢ Poor documentation of testers ➢ Low degree of test automation ➢ Maintaining test audits ➢ Cost benefit analysis of testing

©KTern

6


A Devil in Disguise – Testing Failures in SAP S/4HANA projects

SAP Landscapes are complex and business processes are coupled in, out, around, and beyond the standard functionalities. The disruption that S/4HANA brings technically, functionally, and business process wise to these landscapes is huge. This disruption requires careful compensation in the form of testing. However, there are times when testing proves to be a huge devil in disguise. These are the times when there are significant testing failures. Let’s look at some of the major reasons behind SAP S/4HANA testing failures and how they can be avoided.

Not understanding the “As-Is” and “To-Be” Landscape Architectures

Why should someone even think about landscape architectures while considering the move to S/4HANA? It is important to understand the current to-be process. SAP ECC to SAP S/4HANA migration is not just a technical upgrade. In fact, it relies a lot on data and processes. For all we know, the simplifications from S/4HANA cut across everything, right from field changes to process simplifications. Therefore, a vision of the landscape architecture design becomes inevitable during the test requirement phase.

Lack of knowledge about processes or the process itself

An organization requires a clearly defined Business Process Documentation to start the transformation project. But, most of the times, organizations miss out on this critical factor. They do not undertake this study before starting the project. This could pose a serious threat to the transformation project. Without the business process understanding, it would be nearly impossible to move into S/4HANA with so many simplifications.

Improper Scope Definition between testing and defect management teams

Proper end-to-end testing essentially implies that we are talking about a BPML (Business Process Master List) which has been running a business for many years. An average sized enterprise may have around 1000+ processes, each having multiple test cases based on their data. When a Responsibility Assignment Matrix (RACI Matrix) is drawn on the test management, it would make the system integrator accountable. The Project Health deteriorates severely during testing because of the involvement of the test engineering teams as well as business end users. So, a clear scope definition on the test items as well as the KPI Metrics is required. If not a plan, then at least a testing strategy is needed.

Piled up Conversations and Collaboration messages

The introduction of external business and testing teams brings along hundreds of email communications and issue logs every single day. A pile-up of communication data means improper test and defect management which would reduce the project health as well as the productivity of the consultants. A conversation pertaining to a test case log or an issue log can information like the criticality of the business process to be tested, type of defects, type of issues, status of defects, status of issues, aging of issues, and more.

Improper identification of stakeholders and criticality of businesses

A transformation project’s testing is not synonymous with 100% coverage of the BPML items. It could just cover the items that are getting impacted and are critical to the business. This way, the project team can focus on what’s necessary and get the test cycle completed quickly. This may give rise to two more questions:

©KTern

7


Question 1 - “Who knows what gets impacted when the move to S/4HANA is made?”

Question 2 - “Who can estimate the criticality of the business processes?”

Thus, identifying the key stakeholders and the criticality of the business is very essential for the success of the testing phase.

Improper execution of plans and phase signoffs

When we talk about test plans and cycles, it involves what must be executed, when it must be executed, how it must be executed, who must execute it, the entrance criteria definitions, and the exit criteria definitions. A lot of effort will be saved with the help of these information. Proper execution of test cycles will have a direct impact on the proper execution of project plans, phase closures and phase signoffs.

Improper Monitoring and Reporting Methods

With huge data being generated every day across multiple workstreams, reporting would be a challenge with data accumulated in multiple places and across spreadsheets. There are several chances to miss out on critical impact items and ending up with overhead issues in the long run.

As you can see, it’s easy to lose your way during SAP S/4HANA testing. Therefore, it becomes clearer as to why enterprises need a single tool which can not only ensure test automation, but also take care of test management, test execution, test tracking, and test collaboration.

5 Steps for S/4HANA Testing Automation

Step 1 – Identify critical business processes

Identifying the most critical business processes to be tested is the first step in scoping any test plan. This

requires business understanding and current data on business processes – derived from SAP systems and

delivered with actionable insights. This essentially means hard data from which we can infer the business

processes that are most critical or used most often. In addition, multiple variants exist in the way each

business process is fulfilled by different users. For scoping, statistics of business processes must include all

usage variants and the respective usage frequency.

For the sake of an example, let’s take the following scenarios. Generating a report comparing delivery times

and other parameters of different suppliers may not be test-worthy as it is only executed once a year when

the suppliers’ agreements are up for renewal. Other processes like issuing a discount for example, might

have variants, but only two of them are commonly used and therefore are worth testing.

You may now realize that the main scoping challenge is focusing effort and resources on business processes

that must be tested instead of wasting them on irrelevant or outdated processes. Unfortunately, the usual

test automation solutions do not address this challenge – mainly because they are unable to map the raw

business processes of SAP. Some organizations use excel or other spreadsheet software to keep track of

their business process data – but this is the absolute worst idea to follow for your testing strategy as they

must be manually maintained and act as counterproductive measure to automation.

Take-Away

A tool that can automatically generate business process data and present them as insights for test case

©KTern

8


creation, test execution, and test reporting is needed to identify the most business-critical processes to test.

This would help to save time, effort, and the number of resources needed.

Step 2 – Choose the right business processes to automate

Enterprise and especially testing teams should realize it is easy to get carried away. It is easy to get into a

rush and try to automate all business-critical processes. Take a chill pill. Findings state that most

organizations automate only 30-40% of the business-critical processes. This is why only 1/5th of the

regression test cases consist of critical business processes.

But how do you know which test cases to automate? What factors must one consider? Is there anything

such as useless automation? Is there a chance automation is causing more harm than good?

Businesses change often. This is the status-quo of the digital world. For example, a simple software purchase

in an organization might need to go through multiple approvals along the length and breadth of the

company depending on company policies. In addition, policy changes could change rapidly based on

management changes and many other factors. Automating and maintaining test cases for changes which

happen too often is not feasible.

Likewise, some business processes can have multiple roots and dependencies. For example, the shipping

vendor location may depend on geographic destinations, order cost, order channels, payment method,

taxation, and available shipping partners. Some dependencies may in turn represent many other business

processes. Creating and maintaining automated test cases does not make sense in this case either.

So then, what is a good example of an ideal test case for automation, you ask? Test cases for automation

should be lengthy, repetitive, and effort-intensive business processes. “Order creation” is one good example.

Order to cash, Procure to Pay, Record to Report, Plan to Produce, Plan to Maintain, Fund Management are

all good example for ideal test cases for automation.

Take-Away

When deciding on the test cases to automate, it is important to only automate the lengthy, static, labor-

intensive test cases. Test cases which are dynamic or have multiple dependencies should be kept manual. A

tool (like KTern) which utilizes Machine Learning and RPA (at KTern we call it i-RPA) can help automate test

cases for the relevant business processes and eliminate the need for manually engineered test cases.

Step 3 – Capture the knowledge of key users

SAP is vast. SAP is old. SAP is filled with knowledge for those who search for them. Experience and all the

“dog-years” worth of SAP knowledge are highly important. Testing acumen is important. But, so is business

process know-how. In fact, Capgemini World Quality Report lists ‘business process understanding’ as the key

skill for Testing and Agile Quality Analysis according to 64% of the IT leaders whom they surveyed.

Identifying the right users, contacting them, and obtaining their application knowledge is a laborious activity

and is time and effort intensive. The key users prioritize their individual business objectives and it could be

hard to get the right inputs from them. Incentives could help them to devote time and get their input of

recommended test scripts. Alternatively, a functional analyst maybe needed to record and monitor usage

data to pick the key users and get their inputs. If the employees are present in multiple locations across the

globe, then the inputs of key users in each location is needed since there could be process variants from one

location to another.

It might be clear why capturing and maintaining business knowledge is difficult. Some organizations use

©KTern

9


Excel sheets or other spreadsheet software – but this is not recommended. It is time consuming to edit,

share, and keep the sheets up-to-date from different key users across the globe.

Take-Away

Identifying the right key users to prepare test scripts, following up with them to get the information,

updating test scripts, and confirming the validity of test scripts are all difficult processes – especially since

the production system is constantly changing. The right tool (like KTern) will capture the right test cases

along with the key users “know-hows” to resemble production and usage patterns. Provide instant

information and real-time insights by removing batch jobs and manual effort.

Step 4 – Preparing test cases which incorporate business process variants

Business processes are not constant. They vary from department to department, function to function, or

location to location. For example, a business user in Italy may complete the same business process

differently from that completed by a business user in the United States. This may be attributed to different

locations filling the system fields using different time and date formats, different metric systems, or other

factors. This is where the coverage of automated test cases plas a major role. The automated test cases

should cover the frequently used variants of the relevant business cases.

Take-Away

The right test automation tool will generate test cases which incorporate the different variants of relevant

business processes based on actual usage and up-to-date data from production.

Step 5 – Test what must be tested

Perhaps the most important part of automated testing is test case maintenance. Maintaining an accurate

test case repository can result in critical savings in test case management. It is therefore important to

maintain updated data sources which are current on usage patterns and variants. A dynamic record of the

change statistics of business processes helps to eliminate costs and produce meaningful savings for test case

maintenance.

However, this seems to be easier said than done. Studies and surveys indicate that the biggest challenge for

testing leaders is to maintain and update their test case library to resemble the scenarios in production.

Take-Away

The right test automation tool will help maintain an updated test repository and captures the business

knowledge of the relevant test cases in order to maintain a near replica of production.

Nice To Haves along with Testing Automation

Test automation is just the first step. But, a good testing tool will be able to offer much more. Some of the

key features which are nice to have along with test automation are workflow management, defect

management, and an easy-to-use UI and UX to accelerate and increase user adoption.

Workflow Management

Workflow management is a really good nice to have. Coupled with other process orchestration activities it

can reduce a significant amount (up to 35%) of the testing effort needed. Some of the features which can

help are sequencing of manual tests, handover of testing tasks between key users, progress monitoring, and

©KTern

10


a central defect management center to speed up the complete test execution. Additional features which can

help are “Time to Test Notifications” when a user has to test within a business process, “Retest

Notifications” to testers for defect resolutions and “Close Notifications” when retesting is a success.

Key users can benefit a lot from the features of workflow management. They can execute their testing steps

without any dependencies – even if the business process requires multiple steps and multiple testers.

Defect Management

Automation can also help with managing defects and resolving them. A good defect management tool will

provide a central center for monitoring defects, their resolution, and a view of the business processes

affected by the defects. In addition, the right defect management tool and features can help to

automatically identify all the other tests which could be affected by the defect and warn the testers and the

test manager until the main defect is resolved.

User Adoption

People are always busy. The same applies for the key business users. They could feel it to be a severe

learning process to try and adapt to yet another new tool. It is therefore important to have an intuitive user

experience to promote user adoption. The application must be simple as logging into your email inbox, and

require training as minimal as adjusting to a new mobile phone.

KTern Features for S/4HANA Testing Automation

8.1 Test Case Identification

KTern enables test scripts to be loaded by connecting KTern with an ECC system and running “Check Test

Case Recommendations”. The Results are loaded in a hierarchical manner as Plan, Scenario, and Test case.

Each Plan can have “n” scenarios under it and in turn contain “n” test cases. Recommendations are provided

based on Process Configurations and Customizations enabled in the system. Test Cases are recommended

based on documents generated and business criticality.

8.2 Test Case Execution

The typical process for the Realize phase is as follows:

• Software developers perform unit tests in the DEV system. Depending on the type and scope of the test cycle, various functional tests are performed.

• Manual testers are provided with the tester handout document and receive details regarding their test package by e-mail.

• Automated tests are scheduled or started directly.

• Every test that is executed is logged and documented with test notes and a test status is set manually or automatically.

• If a system responds in an unexpected way during manual testing, for example, if an error message appears, the tester records the incident in the corresponding ITSM system, attaching screenshots, log messages, and so on. Usually, this also must be done manually even for automated tests.

• The incident is sent to the persons responsible for analysis and categorization of defects, who then corrects the defect in the development system.

• The correction is transported to the test system according to the existing arrangements and timelines, where it is then retested.

©KTern

12


In Image: Test Lab

In Image: Auto Test Console

KTern has an inbuilt test management framework to automate and manage test executions and allows multiple stakeholders to be involved in the test cases. Test Execution involves multiple executions in different systems during different times of the project. KTern enables users to collaborate and record discussions against each test item. The User Management framework can be used to invite business users, test engineers, IT SMEs, and consultants to be part of the project. In addition, Assignment, Workflows, and Notifications are enabled by the KTern Management framework.

©KTern

13


In Image: Defect Repository

The Test Suite Framework would facilitate creating test scripts, recording test scripts, and recording test

executions against a test case. Defects generated from test cases are managed using the Defects Repository

to help manage Test Script Documentation, Test Execution Logs, and Defects created. KTern and its

intelligent RPA (i-RPA) framework help execute test cases on the fly through the application layer. This helps

reduce the effort and time needed to unit test all test cases. In addition, the KTern i-RPA framework

facilitates “record and play” to help users record test cases and execute them for multiple data sets. Defects

for failed test cases are auto-created by KTern based on message text type received.

In Image: Test Case Reporting

©KTern

14


8.3 Test Case Reporting

KTern and its versatile analytical framework allow test case reporting in a robust manner. Test cases are

sliced and diced based on different KPI models and provides an accurate track on the amount of test cases,

executions failed, test cycle progress, defect status, resource utilizations, test plan coverage etc.

8.4 VVV Scoring (Value, Volume and Velocity Analysis)

VVV Scoring covers the Volume, Value, and Velocity Analysis of the business per transaction usage and helps

decide the critical business processes which must be tested. As you may recall, identifying the most critical

business processes to be tested is the first step in scoping any test plan. This requires business

understanding and current data on business processes – derived from SAP systems and delivered with

actionable insights. KTern is a tool that can automatically generate business process data and present them

as insights for test case creation, test execution, and test reporting is needed to identify the most business-

critical processes to test. This would help to save time, effort, and the number of resources needed.

In Image: VVV Scoring

8.5 Sign Off Wizard

Sign Off Wizard provides a hierarchical view of the Test Cycle Approvals with business stakeholders. It

provides an unimagined level of transparency to the testing process and makes it easy to complete phase

closures and signoffs during the testing phase.

©KTern

15


In Image: Test Suite Sign-Off Wizard

Next Steps

▪ Visit our website.

▪ View other whitepapers and other resources.

▪ Get Started with KTern and explore the free KTern Demo or Sign-Up for your free Trial.

▪ Get in touch with us so that we can set up a call and discuss further.

About KTern

KTern is an SAP Certified product to automate and manage SAP S/4HANA migrations from start to finish. With 12 inbuilt modules, the AI & RPA driven "S/4HANAPEDIA" Engine, and the vast knowledge base of SAP and SAP Partners, KTern will reduce up to 1/3rd of the total migration cost and timeline. Given below are some of the salient features of KTern across the different stages of an S/4HANA Transition. Before the Transition

• Automated assessment to analyze technical, functional, & business impact of moving to S/4HANA

• Auto-recommendations pertaining to testing strategy, landscape & infra, and data archiving

• Auto-generation of an S/4HANA conversion blueprint incorporating the right plan, the right effort,

the right cost, the right system sizing, the right project management, and the right quality

During the Transition

• Custom process refinement using preset automation rulesets for custom code remediation

• Business Process Automation Testing based on ECC system usage and inbuilt datasets

• Automated Data Validation based on the migrated data from ECC to S/4HANA

After the Transition

• Provision of a repository of test cases, issues, and bugs for faster and safer future upgrades

• Readily available and accessible data for audit requirements

• UX transformation through Fiori App Recommendations

https://ktern.com/?&utm_source=content&utm_medium=ebook&utm_content=s4hanatestingautomation

https://ktern.com/get-started/?&utm_source=content&utm_medium=ebook&utm_content=s4hanatestingautomation

https://ktern.com/company/contact/?&utm_source=content&utm_medium=ebook&utm_content=s4hanatestingautomation

e-book...proper end-to-end testing essentially implies that we are talking about a bpml (business...

Documents