Automated Test Execution and Reporting(ATER) Pluggable

Solution using JIRA

Banani Ghosh, Senior Software Engineer Tanvi Dharmarha, Quality Engineering Manager

Adobe Systems

Abstract Test Automation is an extremely important step in the Agile world to control and manage the execution of test cases and reporting the result of the automation run. Several teams in several projects across various companies create test automation frameworks to automate regression activities. While giving major focus on building automation frameworks and automating workflows, managing automation runs and updating execution results for each run often gets little or no attention. This results in loss of tracking testing activity and makes automation efforts somewhat redundant, as engineers are unsure which test cases passed, failed or need re-execution. Sometimes though the results are recorded, the reports generated becomes large enough to be shared across the team over mail or all the relevant information cannot be attached in the custom reports which are used for debugging, thus increasing manual effort and failing the purpose of automation.

We have created a solution that bridges this gap of maintaining test case execution and reporting execution activity by managing the test automation. Most organizations use JIRA as a platform for test case management and bug reporting and as such, we have implemented our solution that integrates seamlessly with JIRA by picking up tests cases, creating test runs for desired set of test cases for the supported environments, locales, platforms, build etc., triggers automation and finally updates results of automation for each test run. It is a pluggable solution on top of any automation framework written by testing teams irrespective of programming language or tools used, as our solution will create test runs and simply trigger the already built automation, removing the hassle of rebuilding reporting frameworks, thus saving time to spend on automating more challenging workflows.

Through this paper, we will deep dive into the implementation and usability of the solution to manage test case automation and reporting. Also, we shall elicit configurational changes needed at project’s end to plug this seamlessly by providing a walk-through on the functioning of the plugin’s publicly exposed APIs that syncs an automation framework with the JIRA Zephyr Rest servers.

Keywords: Automation Testing, JIRA, Zephyr, Reporting

Introduction

In the present-day scenario, there is need for accelerated software development. Automated software testing is a process through which thorough and quick testing is done to roll out developed functionality to market.

The Test Automation Process Life Cycle consists of several steps which starts from Finalizing the test plan, selection and customization of tools for developing test scripts till analysing the automation results thus validating the Quality of a build. It has been observed that, in this process of Automated Test Lifecycle, while building the test utilities frameworks and developing testing scripts scrips obtains the major focus, on the other hand, the following phases including the execution and reporting phase gets faded away, not due to unawareness, but due to limited time and resource crunch.

Testing teams, traditionally use third party software tools to manage the execution and tracking of their automated test cases. The major disadvantage is that they mostly use separate tools for execution and reporting. Also, some software tools require large amount of coding and configuration for integration. Shortage of skilled resources, for tool specific code language, and runtimes also becomes an inhibiter to adoption of a preferred software tool compromising quality. License cost is another challenge while tool selection and use.

Our present solution is an effective Testcase Execution and Reporting(ATER) software that integrates with JIRA making life easier for both testers and developers. Different automation systems coded in different languages are treated as a black box and our solution simply plugs to them to manage their execution and reporting without much ado.

Solution

ATER is a Java-based plugin which reduces the time and effort of integrating third party complex software’s into the automation tool. This saves the redundancy of re-writing customizable reporting frameworks, thus making the optimized use of the licensed project management tools like JIRA.

With the introduction of ATER in the automation frameworks, whenever Quality Engineers find any errors in the automation tool, instead of jumping straight into debugging the scripts, one can open the JIRA dashboard to observe issues related to the test scenario or any relevant files like snapshots or screen recordings, which could not be shared with the customized reports earlier, due to space and resource crunch. If the test has failed for the first time by observing the execution history, then one can be detected easily as a new bug and raised with team for a fix. In case of existing failures, ATER adds test results to the linked issues as well, thus reducing manual efforts and eventually the watchers will be informed through JIRA. Thus, the entire Automated Test Life Cycle is tightened by this open source ATER plugin.

The essential features of the ATER plugin as follows:

• User creates an Execution Cycle. The execution cycle shall be unique for every test runs.

package com.adobe.JIRAUtil;public void createExecCycle (final String description){

String cycleName = test + "_"+ Common.getCurrentDate();

String desc = "Regression Test Cycle for " + release + " release"; }

• User defines the type of environment, project, build and version

cycleInfo=createExecutionCycle (account, cycleName, env, pID, vID); {

"build" : account, "environment" : env, "projectId": pID, "versionId”: vID

}

• User provides the overall summary and description, the estimated start and end date of the execution cycle

cycleInfo=createExecutionCycle(desc);{

"description", desc, "startDate", sDate (), "endDate", endDate ()

}

• User will have options to add set of test cases to the execution cycle or may clone from an existing cycle

cycleInfo=createExecutionCycle (cycleName, clonedID){

“clonedCycleId", clonedID, "name", cycleName,

} [e.g.: clonedID = "38152”]

• Users can define comments in failure cases including the stack traces (if any)• The status of the execution of the test case (Pass/Fail/etc.) can be set after the test executions

{"status": testResult.getExecStatus() "comment": testResult.getComment() "bugList”: testResult.getIssueList()

}

• Any other files or snapshots can be attached to the test case execution• The work flow can be provided which describes the progress.

I. System Architecture

As illustrated in the Figure 1, below, our solution uses open source technologies like Selenium, Java, JAX-RS, etc. to create the bridge between the automation framework and the execution and reporting problem. The application under test (AUT) can be a web-based application running on different browsers like Mozilla Firefox, Google Chrome, Safari, Internet Explorer or any Desktop applications on installed on any platforms. During automation, the test run results are enqueued to this plugin. These results are intended to be updated in JIRA which helps to track them during the automated testing life cycle.

(fig 1. System Architecture diagram)

The plugin through a connection pipeline, connects to one of the Zephyr[3] Rest Servers (here, ZAPI), which interacts with the project configurations in JIRA. For each of the automated workflows, a set of test cases with the results are passed to the ATER plugin which handles the REST calls between the Zephyr server and Jira REST client seamlessly. The plugin makes sure to acknowledge the AUT with the relevant response for Zephyr Server if required for custom reporting.

Maintains a parallel updating process to minimize the execution time of the Automated framework.

(fig 2. Relational diagram for JIRA & Zephyr)

II. TESTCASE EXECUTION

JIRA's REST API is used by the developers who need to interact with JIRA in a procedural and programmatic manner. This could fulfil the development department who want to integrate their own software or other Atlassian applications with JIRA.

(fig 3. Producer Module & ATER framework initialize)

At ATER framework init ():

• The plugin collects the credentials from the Automated framework.• It then sends a request to the JIRA enterprise host server to open a connection pipeline with Basic

Authentication.• The plugin then creates a producer-consumer[4] model to create and update the test execution results from the

client automation framework to Jira servers

(fig 4. Consumer Module: Interaction with JIRA Zephyr Rest Server)

After init (), the plugin initializes the following:

• an execution client instance.• The routine creates a connection to the JIRA host server of the Company.• This routine also spawns two threads,

o a test case producer ando a consumer executor that updates the test case results.

• Once the consumer thread is spawned, a mapping is created between the JIRA test case ids with the internalexecution ids used to track test case keys in Jira.

tcMap = jConn.createIssueExecMap();

public void createIssueExecMap() {

JSONObject executions = execClient.get("/rest/zapi/latest/execution", payload);

JSONArray issueList = (JSONArray) respObj.get("executions"); Iterator<?> itr1 = issueList.iterator();

while (itr1.hasNext()) { if(pair.getKey().equals("ID")){

id = (Integer) pair.getValue(); }

if(pair.getKey().equals( "IssueKey")) key = (String) pair.getValue();

tcMap.put(key, id); }

III. Test Result Generator (Producer Module)

In the automation frameworks used, Users create a Test case producer object. It will consist of:

• the status of the execution,• relevant comments,• list of issues if any related to the test case.

Finally, the driving suite consists of a concurrent hash map to store the producer objects. The test case objects are added to the list by the execution Client of the plugin. The client then enqueues the test case objects.

public void produce (List<Map<String, TestCase>> testCaseList) {

try { il = testCaseList.iterator(); while (il.hasNext()) {

getProducer().enQueue(il.next()); }

IV. Test Case Updater (Consumer Module)

At every interval, the consumer thread keeps polling from the queue. This routine extracts the execution id of the corresponding test case key from the internal mapping created by the consumer init () routine.

Then the Jira Rest API for the test case execution is called by the executor routine and the data members of the test case object are passed as the payload to the Jira Zephyr Rest Server.

public void run() { Map<String,TestCase> testCaseMap = queue.take();

Integer execID =execClient.getTcMap() .getTestCaseKey());

updateTestCase(execID, testCaseMap .get(key)); sleep(5000); }

private void updateTestCase(Integer executionId, TestCase testResult) {

execute = "/rest/zapi/latest/" + "execution/" + executionId + "/execute"

resp = getJiraClient().getRestClient() .put(execute, payload);

}

Advantages of ATER: Example Use Case

Imagine a common scenario, several new features included in an urgent major release of a product. Due to the intricacies and huge complexities of the features, multiple patches released everyday. The time to production is minimal and Quality Engineers need to assure the entire build in every patch and simultaneously monitor the test case results each time over mails becomes cumbersome. In Adobe, too, similar scenario aroused with few teams: -

Crash Reporter: Responsible for analyzing Adobe software crasheso web-based automation developed using Selenium, Java and TestNG

App streaming: Responsible for verifying streamed Adobe apps on browsero web-based image comparison Automation using use SikuliX, Java and testNG

Thor team: Responsible for handling installers of Adobe Software’s.o Platform-based automation developed on Python and Ruby on Rails.

All the teams were struggling with the execution and mapping of daily patch results, inspite of highly efficient automated workflows, due to the absence of a strong reporting system. All QEs blocked with quality assurance of various features and tracking of daily build results made it even difficult to cater some of the corner yet major issues in the new features. Redundant efforts & resource crunch would have occurred if the teams invested in developing a strong reporting system at such a crucial hour. ATER plugin came to the rescue. Due to its REST architecture, it was integrated easily by teams in few hours. Post integration, the linking of the test cases in JIRA to their automation scripts using just the Testcase Keys was completed in less than 3 days for all the teams, inspite of having hundreds of automated workflows. Thus, improved quality of the major release as automation ran multiple times in a day and all the stability reports of the test cases were found documented in one place in JIRA which was accessible across team members, with all the relevant information handy at the time of fix.

Conclusion

In the current competitive world, technicalities can find many find various software and tools which is growing rapidly. Designing and development of software and tools should provide user friendly environment and provide user desirable features in cost effective manner. The automation tool is developed with multiple features as a test case management system. The organizations make use of test case management system to track the execution of each test case in JIRA after running the tests, presently which is performed either manually or using separate software’s. It is helpful, if the Automation framework contains this pluggable solution. In this paper, there is explanation of the integration of test case management system with automation suite. Future work includes development and integration of User interface to take the ease of plugging this to the next level.

References & Appendix [1] http://www.seleniumhq.org/

[2] http://watir.com/

[3] https://www.getzephyr.com/products/zephyr-for-jira

[4] https://en.wikipedia.org/wiki/Producer%E2%80%93consumer_problem

Author Biography Banani Ghosh is working with Adobe Systems as Senior Software Engineer having 2years of experience in manual, automated and API Testing. She has been working as a quality engineer for Adobe Crash Reporting System. She holds an engineering degree in Electronics and Electrical. Prior to Adobe she has worked with Aricent Technologies in Telecom domain being responsible in developing and maintaining several Security Gateway APIs and tools. Banani has few paper publications to her credit.

Tanvi Dharmarha is working with Adobe Systems as Quality Engineering Manager and has over 10 years of experience in manual, automated and API testing. She owns the quality engineering for Adobe Crash Reporting System. Tanvi has several paper publications to her credit. She holds an engineering degree in Information Technology and is also a certified Stanford Advanced Project Manager.

THANK YOU!