automation healthcheck report
TRANSCRIPT
-
8/11/2019 Automation HealthCheck Report
1/26
SQS Group Limited
VodafoneAutomation HealthCheckReport
Darryl Kennedy6 th February 2013
-
8/11/2019 Automation HealthCheck Report
2/26
Executive Summary
Background
Vodafone have an existing test automation solution created using QC 10 and QTP 11 The automated regression testing solution includes both ECC (GUI) and SRM (Portal) The existing solution has been in existence since its creation in December 2011 The existing solution is maintained in Pune by VISPL Automation Team
Test execution and reporting in carried out in Pune by VISPL Automation Team
Purpose of the Automation HealthCheck
In order to maximise the benefits of automation, SQS have been asked to undertake anindependent review (HealthCheck) of the current test automation solution to ensure that
Industry best practice has been employed
The solution is extensible and provides the necessary portability and robustness
The test assets created are re useable and easily maintainable
Tests execute unattended, results and evidence are clear and easily interpretable
SQS have been asked to provide any recommendations for improvement along withsupporting benefits of implementing any recommendations
-
8/11/2019 Automation HealthCheck Report
3/26
Work Carried out
SQS provided Vodafone with a test automation specialist who Reviewed a cross section of the QC and QTP assets available
Assessed the approach to automation and the automation framework deployed
Assessed the versatility, robustness and extensibility of the automation framework
Assessed the approach to script development including test data Assessed error / exception triggers and recovery
Assessed automation synchronisation and execution performance
Assessed ability to execute unattended and the existing infrastructure
High Level HealthCheck Finding
The QC and QTP solution adheres to certain design principles necessary for amaintainable component driven framework but lacks necessary robustness
The existing solution performs reasonably but essential attributes are missing or poorlyimplemented which are necessary for enterprise strength automation
Important strategic changes must be implemented if the true benefits of test automation areto be realised before significantly extending the scale of the existing solution
Executive Summary
-
8/11/2019 Automation HealthCheck Report
4/26
Automation Solution Summary A good technical basis for an automated testing solution incorporating some featuresprovided by the HP Tool set. Consideration has been made to ensure the framework isextensible and maintainable with minimal effort.Some fundamental characteristics of test automation have been introduced such as sharedobject repositories, data abstraction and the use of external function libraries to provide amodular design.
Web and SAP automation present a number of challenges and although innovation hasbeen applied in the framework to provide solutions above what QC and QTP provide out ofthe box some basic principles have not been adhered to:
Modular design of the solution and use of external data and function library files
Custom HTML reporting easily interpretable by non technical users
XML Configuration files for multiple environment execution & Country specific test dataLacks an overall framework engine with the appropriate flow controlSynchronisation and error handling needs a lot of work so able to run unattended
No validation included into the functions which make up the Test Scenarios
The VISPL team were knowledgeable of their subject and extremely collaborative.
Executive Summary
-
8/11/2019 Automation HealthCheck Report
5/26
Recommendation Priority & Implementation ComplexityThe table below summarises all the recommendations made in this report into the priorityfor implementation.
The table below summarises all the recommendations made in this report into thecomplexity of the implementation.
Executive Summary
Implementation Priority Total Recommendations
High 36
Medium 13
Low 1
Implementation Complexity Total Recommendations
High 12
Medium 10
Low 28
-
8/11/2019 Automation HealthCheck Report
6/26
-
8/11/2019 Automation HealthCheck Report
7/26
The HealthCheck focussed upon
Adherence to test automation best practice
Ease and skill requirements of execution and maintenance
Future proofing
The HealthCheck did not
Test existing automation (any issues identified during the HealthCheck will be reported)
Review the automated scripts for test coverage, accuracy to reflect business processesor ability to reflect business testing needs
Review the scope or prioritisation of test scenarios being automated
Estimate the cost to run and maintain the scripts
Basis for the HealthCheck
The findings are based on virtual meetings that took place on 4 th and 5 th February
Thank you to the VISPL Automation Team for the help and support they provided
Raghavendar Bandaru, Ganesh Kaushik and Nagesh Battula
Automation HealthCheck Scope
-
8/11/2019 Automation HealthCheck Report
8/26
-
8/11/2019 Automation HealthCheck Report
9/26
FrameworkComponent Criteria Rating Comments
Audit Trail
NamingConventions
AmberNaming conventions exist for Test Scenarios and Functions whichare adhered to. Additional detail should be applied to make it easierfor end users to interpret these easier i.e. SAP T Code.
UniqueIdentifiers Amber
Test Scenarios and functions are distinctly named but do not containa unique identifier.
Mapping toRequirements
AmberNo requirements exist in QC. A matrix of requirements ismaintained by the functional test team using Excel. This makestraceability time consuming.
Mapping toManual Tests Red
Manual and technical assets are duplicated in QC so traceability isdifficult and time consuming. Extra actions must be performed bythe end user to see the manual steps. Manual and technical assetsmay get out of synchronisation with each other.
Coverage Requirements Red No requirements exist in QC so there is no technical solutionavailable to ascertain coverage of the technical assets.
Test Automation Score Card
-
8/11/2019 Automation HealthCheck Report
10/26
Day 1
Initiation meeting with Tom Sewell to agree scope of the HeathCheck Review (SQS & VISPL) of the QC and QTP framework implementation Review (SQS & VISPL) of the Scenarios, Objects, Reporting and Error Handling First day wrap up meeting with Tom Sewell
Day 2
Review (SQS & VISPL) of the Data, Config. File, Reporting, Execution & Development Execution (SQS) of Portal and SAP Test Scenario Initial Report Creation Walkthrough of the findings report structure with Tom Sewell
Day 3
Report Finalised Assessment of the automation Recommendations and examples of findings Issues identified during the HealthCheck
Walkthrough of findings (Vodafone & SQS)
Schedule of Activities
-
8/11/2019 Automation HealthCheck Report
11/26
In order to conduct the HealthCheck, SQS requested access to the following
Access to all test assets
Access to the system under test and existing manual test scripts Access to working end-to-end automation test scripts
Access to a sample of custom built drivers for the various automation tools
Access to sample reports generated
A demonstration of the test execution:
Against the applications under review
Using manual run
Via build triggered run if applicable
Availability of the VISPL Automation Team
Environment support during demonstration and test execution of the solution Reasonable turn around times on queries raised whether against the VISPL
Automation Team or external (for example a manual test or solution design query).
All of the above was provided as requested by the VISPL Automation Team.
Pre-requisites and Support Criteria
-
8/11/2019 Automation HealthCheck Report
12/26
-
8/11/2019 Automation HealthCheck Report
13/26
Recommendations
Recommendation Guidelines
Each of the recommendations made has a unique identifier
Each of the recommendations is clearly defined
Each of the recommendations is supported by a benefit
Each of the recommendations has an implementation priority assigned
Each of the recommendations has an implementation complexity assigned
Ordering of Recommendations in Table format
The unique identifier does not reflect an implementation order or importance
Many of the recommendations are linked and must be considered as a whole
-
8/11/2019 Automation HealthCheck Report
14/26
Recommendations for Quality Center
ID Recommendation Benefit Priority Complexity
QC01 Implement the usage of the Detail fieldin QC for each of the assets storedProvides the end user with a clearoverview of the purpose of the test High Low
QC02 Implement the usage of the Status fieldin QC for each of the assets stored
Creates a development workflow andimproves visibility of the overallprogress
High Low
QC03Implement the usage of the DesignSteps in QC for each of the assetsstored
Ensures the manual tests assets andthe technical test assets do not vergeapart from each other. Provides thetechnical team a clear and easy way ofworking
High Low
QC04 Store the requirements in QC whichare currently tracked using Excel
Provides dynamic traceability ofprogress and enables tracking ofissues found
High Low
QC05
Move the Test Resources outside ofQC to enable the implementation of anenterprise strength version controlsolution (unless the upgraded QCversion control provides the necessaryfunctionality)
QC provides a basic level of versioncontrol but not sufficient fordevelopment of automated test suiteson an enterprise level with manypeople working concurrently
Medium High
-
8/11/2019 Automation HealthCheck Report
15/26
Recommendations for QTP Scripts
ID Recommendation Benefit Priority Complexity
SC01 Naming Convention with uniqueidentifiers applied to all assetsSimplifies the traceability of all assetsand improves the reporting in QC High Low
SC02 Detailed headers and script commentsto be implemented to all assetsImproves the ease of use of thetechnical assets for collaboration Medium Low
SC03Code repeated in each test whichshould be a framework function i.e. EnvVariables
Simplifies and speeds up maintenance
activities required on all testsMedium Low
SC04
All tests must be suitable forenvironment portability. Load XMLenvironment variables from a centralfile dynamically
Supersedes XML loaded manually toevery test which is a slow andcumbersome process prone to error
High Low
SC05 Start up and teardown functions to beapplied to all assets. Allows for quick changes to be madeglobally to all test assets High Low
SC06Break away from functions and createQC assets for each businesscomponent
Will introduce overall framework andflow control and ability to execute testsfrom a failure point
Medium High
SC07
Use the QTP object to dynamically loadrun-time setting at the beginning ofeach test. Screen prints Never/Alwaysand global object timeouts
Applies consistency to every test caserunning in the suite and removes theneed to execute VBS manually
High Low
-
8/11/2019 Automation HealthCheck Report
16/26
Recommendations for QTP Objects
ID Recommendation Benefit Priority Complexity
OR01 Remove version numbers from currentOR in useRemoves the requirement to manuallyassociate this with every test using it High Low
OR02 Changes the object description, dont just use the SAP generated nameMakes scripts readable, intuitive andallows other users to understand them High Low
OR03 Object Windows must be distinct,merge _2 and _3 as appropriateKeeps OR tidy and reduces thenumber of windows to maintain High Medium
OR04 Objects must be distinct, merge _2 and _3 as appropriate
Keeps OR tidy and reduces thenumber of objects to maintain
High Medium
OR05Object property and value pairs mustbe improved and regular expressionsapplied where appropriate
Clearer object definition and reducesfails during execution Medium Medium
OR06 ORs for SAP and Web should bebroken up into smaller filesReduces the change of corruption andallows concurrent development Medium Low
OR07 Portal OR cleaned up to remove manyinstances of Browser objectKeeps OR tidy and reduces thenumber of objects to maintain High Medium
OR08 Remove all local objects once mergedinto the shared ORPrevents tests failing on shared andlocal conflicts High Low
OR09 Store shared OR in a central locationand not on a C:\
Easily accessible by all developers High Low
-
8/11/2019 Automation HealthCheck Report
17/26
Recommendations for QTP Functions
ID Recommendation Benefit Priority Complexity
FL01
Large single files over 12,000 lines ofcode must be broken down into smallerlibraries of technical functions and QCassets for business components
Collaborative development isimpossible and contention exists.Using QC assets will improve thetraceability and maintainability
High High
FL02 Remove the version numbers from thenames of function librariesReduces the manual overhead ofcontinually associating to every test High Low
FL03Library files and individual functionsmust have detailed headers whichinclude modification history
Provides a level of self documentingsolution and improves collaboration High Low
FL04Naming conventions must be improvedand include SAP TCode whereappropriate
Improved script readability and ease ofdebugging when errors occur High Medium
FL05 A Version Control solution must beadopted, taking code out of QC ifnecessary
Improves concurrent development,
traceability, multiple environmentexecution and reversion wherenecessary
High High
FL06Create traceability with the manualassets stored in QC by creatingtechnical assets in the same QC entity
Improves the overall knowledge of theteam. Keeps manual and technicalassets in sync and improvestraceability
High High
-
8/11/2019 Automation HealthCheck Report
18/26
Recommendations for QTP Functions
ID Recommendation Benefit Priority Complexity
FL07Dynamic synchronisation must beapplied to all business components andall hard coded waits removed
Currently there is limited or none. Thestandard QTP object synchronisationpoints are not robust enough forunattended execution and porting thescripts across different testenvironments
High Medium
FL08 Login functions must handle multipleOPCO logins for all authorisation levelsThese should be stored in an externalfile Medium Low
FL09
An overall flow control must beimplemented into every businesscomponent. Currently QTP simplyexecutes linear with no checks toensure events are happening asexpected to allow control to pass back
to the calling test for error reporting andhandling
This will prevent tests from failingunexpectedly and provide a muchgreater visibility of what caused thefailure in the QTP execution report
High High
FL10Evaluate all reporter message toensure failing steps are reportedcorrectly.
Found an example of code which failsbut report message contained a pass.
An If / Else statement was completelybypassed as SAP was on the incorrectscreen
High Low
-
8/11/2019 Automation HealthCheck Report
19/26
Recommendations for QTP Functions
ID Recommendation Benefit Priority Complexity
FL11
Remove the dependency on the QTPRecovery Manager. This should beused as a catch all not an everyinstance solution. These should behandled in the code itself
Much greater control of the errorhandling and provides far more detailin the QTP execution report as to thecause of the failure. Can then alsoincorporate screen captures
High High
FL12 Validation and checkpoints must beapplied to all business components
This provides detailed feedback as to
why tests failed and allows tests to runin batches unattended without theneed for constant monitoring
High Medium
FL13Error detection, error reporting, errorhandling and recovery must be appliedto all business components.
QTP Run settings updated with a VBSfail to suggest Test Stops on error, thisis only part of the necessary errorhandling. Tests must also resetconditions i.e. close browser and then
continue to the next test cleanly
High High
FL14 An overarching function is missing tohandle and recover from failure
Inbuilt QTP setting used to capturescreenshot on failure. This is typicallyunreliable and should only beincorporated in the code where failureshave been anticipated to occur
High Medium
-
8/11/2019 Automation HealthCheck Report
20/26
Recommendations for QTP Functions
ID Recommendation Benefit Priority Complexity
FL15Improve the robustness of the businesscomponents by adding additionalvalidation to the objects before use.
Creating overload functions in QTPcould include this and many otherchecks i.e. Enabled, Visible etc. Asthey are overloaded no change to theend user so scripts can still berecorded in QTP.
High High
FL16Function to create HTML reportrequires lots of additional code in thefunctions and should be included in theoverloaded functions.
This will simplify the script creation andensure consistency throughout overbusiness component
Medium High
FL17 HTML creation functions must alsoupload to QC RUN table. Reduce manual overheads High High
FL18HTML functions for automatedreporting must be included inoverarching function to handle errors
Ensures these are created anduploaded during test runs whichencounter errors
High Low
-
8/11/2019 Automation HealthCheck Report
21/26
Recommendations for Test Data
ID Recommendation Benefit Priority Complexity
TD01OPCO User Accounts are stored on acentral file governed by environmentand country
Easy for the end user to maintain andprovides versatility for multipleenvironment execution
Medium Low
TD02
Different tabs for OPCOs should beamalgamated to single sheet and QTPfunction created to select appropriate
row
Reduces overall maintenance as thereare less Excel sheets to manage Medium Medium
TD03 Needs to have a unique identifier foreach Excel sheet which map to the test
Improves visibility and traceability High Low
TD04 Additional iterations of data should beincorporated into the Excel data sheetsto test more data permutations
Incorporating additional data setsprovides greater coverage easily withapplying a unique identifier to the datarow and mapping to test name.Current constraint in the frameworkdoes not allow this
Medium Low
-
8/11/2019 Automation HealthCheck Report
22/26
Recommendations for Test Execution
ID Recommendation Benefit Priority Complexity
TE01
6 to 10 dedicated QTP servers hostedvirtually so accessible from anywhere
load balanced using QC host groupsinstead of the 4 machines currently inuse
Utilises more of the 20 QTP licensesavailable and provides greaterflexibility from the 4 machines usedcurrently which are also the day to daymachines of the team. Tests can thenbe developed and executedconcurrently
Removes the dependency that theuser must be present to monitor theexecution
Removed the dependency of theDeveloper running their own script
High Low
TE02 Hardcoded C:\ path in the HTMLreports must be removed
This must be addressed to allow manymachines to run unattended and seeresults from anywhere
High Low
TE03
HTML report is uploaded dynamicallyto QC in the same way manual testsdo, these should be uploaded to theRUN table / Execution history
Ensures results can be accessed fromanywhere and not just the machinethey were executed on. Provides clearand concise audit trail
High Medium
-
8/11/2019 Automation HealthCheck Report
23/26
Recommendations for General Adoption
ID Recommendation Benefit Priority Complexity
GA01 QTP Patching level should be broughtup to the latest versionIssues already identified by HP will beaddressed Low Low
GA02Development work items must bevisible to everyone so its clear who isworking on what changes
Greater visibility of the changesplanned and progress tracked againsteach. Allows work to be decided on apriority basis
Medium Low
GA03 Solution documentation must becreated for the framework
Allows up skilling of anyone whoneeds to develop, maintain or executetests
Medium Low
GA04Solid Version Control solution requiredwhich may include moving outside ofQC
Unable to develop and execute testsconcurrently so different versions ofQTP code base must be maintained.QTP Environments in General Optionsalong with dynamic library loading inQTP 11 allow the developer to quicklychange environments and versions
High High
-
8/11/2019 Automation HealthCheck Report
24/26
Implementation Plan
-
8/11/2019 Automation HealthCheck Report
25/26
Implementation Plan To be followed as a pilot phase
Implementation Area Detailed Description
Build a Core Framework
1) Object Identification Mechanism to be streamlined
2) Object Repository to be re-structured
3) Custom functions to be built to replace QTP inbuilt methods
4) Incorporate Error handling, Object Validation, Screen Capture for evidence, Object syncand Checkpoints in the scripts
5) Reduce the large Function Library by moving the "Application Specific" functions tomodular QTP scripts
Test Plan
1) Convert the Application Specific functions along with the new framework into individual,modular, robust QTP scripts
2) Include the design step for each T-code - for better readability and review by Non-Technical/ Business Users
Test Lab
1) Update the scenario details, description, Objective, Pre-requisite
2) Build the end-to-end scenario by calling the modular QTP scripts into Test Lab andsetting the execution flow
3) Incorporate custom reporting to generate a report at the end of each scenario execution
4) Datasheet consolidation
Scheduling/ Host Management
1) Get the physical assets in place (20 Concurrent user license for QTP available)
2) Prepare a scheduling strategy to run the scenarios overnight where possible to bettermanage the execution timelines
-
8/11/2019 Automation HealthCheck Report
26/26
SQS Group Limited
Darryl KennedyPrincipal ConsultantEmail : [email protected] Tel: 07950 128 060
SQS Group Limited
82 King Street,Manchester, M2 4WQ
Internet: www.sqs-uk.com Internet: www.sqs-group.com
mailto:[email protected]://www.sqs-uk.com/http://www.sqs-group.com/http://www.sqs-group.com/http://www.sqs-group.com/http://www.sqs-group.com/http://www.sqs-uk.com/http://www.sqs-uk.com/http://www.sqs-uk.com/mailto:[email protected]