dv regression testing with professional jmeter · test tool from apache foundation, is being used...

1

ProfessionalServices

www.tibco.com

GlobalHeadquarters 3303HillviewAvenue PaloAlto,CA94304

Tel:+1650-846-1000 +1800-420-8450 Fax:+1650-846-1005

TIBCOSoftwareempowersexecutives,developers,andbusinessuserswithFastDatasolutionsthatmaketherightdataavailableinrealtimeforfasteranswers,betterdecisions,andsmarteraction.Overthepast15years,thousandsofbusinessesacrosstheglobehavereliedonTIBCOtechnologytointegratetheirapplicationsandecosystems,analyzetheirdata,andcreatereal-timesolutions.LearnhowTIBCOturnsdata—bigorsmall—intodifferentiationatwww.tibco.com.

DV Regression Testing with JMeter

Project Name

Release 1.1

Date March 2018

Primary Author

Tony Young

Document Owner

Tony Young

Client

Document Location

Purpose



Revision History Version Date Author Comments

1.0 November 2016 Tony Young Initial version

1.1 March 2018 Tony Young Updated with TIBCO branding

Approvals This document requires the following approvals. Signed approval forms are filed in the project files.

Name Signature Title Company Date of Issue Version

Distribution This document has been distributed to:

Name Title Company Date of Issue Version

Related Documents This document is related to:

Document File Name Author



Copyright Notice COPYRIGHT© TIBCO Software Inc. This document is unpublished and the foregoing notice is affixed to protect TIBCO Software Inc. in the event of inadvertent publication. All rights reserved. No part of this document may be reproduced in any form, including photocopying or transmission electronically to any computer, without prior written consent of TIBCO Software Inc. The information contained in this document is confidential and proprietary to TIBCO Software Inc. and may not be used or disclosed except as expressly authorized in writing by TIBCO Software Inc. Copyright protection includes material generated from our software programs displayed on the screen, such as icons, screen displays, and the like.

Trademarks All brand and product names are trademarks or registered trademarks of their respective holders and are hereby acknowledged. Technologies described herein are either covered by existing patents or patent applications are in progress.

Confidentiality The information in this document is subject to change without notice. This document contains information that is confidential and proprietary to TIBCO Software Inc. and its affiliates and may not be copied, published, or disclosed to others, or used for any purposes other than review, without written authorization of an officer of TIBCO Software Inc. Submission of this document does not represent a commitment to implement any portion of this specification in the products of the submitters.

Content Warranty The information in this document is subject to change without notice. THIS DOCUMENT IS PROVIDED "AS IS" AND TIBCO MAKES NO WARRANTY, EXPRESS, IMPLIED, OR STATUTORY, INCLUDING BUT NOT LIMITED TO ALL WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE. TIBCO Software Inc. shall not be liable for errors contained herein or for incidental or consequential damages in connection with the furnishing, performance or use of this material.

Export This document and related technical data, are subject to U.S. export control laws, including without limitation the U.S. Export Administration Act and its associated regulations, and may be subject to export or import regulations of other countries. You agree not to export or re-export this document in any form in violation of the applicable export or import laws of the United States or any foreign jurisdiction.

For more information, please contact:

TIBCO Software Inc. 3303 Hillview Avenue Palo Alto, CA 94304 USA



Table of Contents 1 Introduction ........................................................................................................................................ 6

2 RT Setup and Configuration .............................................................................................................. 7

3 Functional Testing .............................................................................................................................. 8

3.1 JMeter Test Plan ................................................................................................................................ 8

3.2 Test Case File .................................................................................................................................. 10

3.3 Reviewing Test Results ................................................................................................................... 10

4 Conclusion ....................................................................................................................................... 14



1 Introduction

Generally, regression testing (RT) is part of every development cycle. This document specifically targets regression testing that should occur during the CIS 7.0.x upgrade project.

The scope of this special RT activity was identified as follows:

- Functional testing: result sets returned by all queries executed against virtual databases pre- and post-upgrade, should be identical.

- Basic performance testing: response time of the data sources should be “acceptable” (for example, not more than 10% slower than before upgrade).



2 RT Setup and Configuration

Regression testing is performed by comparing results of querying two different TDV servers. JMeter, an open source test tool from Apache Foundation, is being used to execute the queries and assess the results. A conceptual diagram of the testing environment is presented on the diagram below.



3 Functional Testing

Functional testing looks to validate the content (results) of the queries executed against virtual databases pre- and post-upgrade. A side-by-side comparison should reveal no differences.

In addition, the response time of the refactored data sources should be “acceptable”, for example not to exceed current response time plus 10%.

3.1 JMeter Test Plan Functional testing is performed by executing -Regression-Test-Plan (Fig. below), contained in JMeter script file JDBC_Regression_CSV.jmx. Test plan can be executed via JMeter GUI, or from a command line.

Essentially, this plan executes the same set of queries against two data sources, identified by JDBC configuration elements “Baseline Config” and “Target Config”, and compares the results. A script reads SQL text for the queries to be executed at runtime from an external.csv file. In the script, the default location of the query file and its name is determined by the user-defined variable DEFAULT_QUERY_FILE (above). To process a different query file, one can either edit its name right on the Test Plan GUI screen, or override the default by using option -J QUERY_FILE=file_name when script is executed via command-line. Details of a query file format are discussed in the next section.

The summary of test plan’s functional elements, together with their brief description is presented in Table 2.



Element name Element type Description

Regression Test Plan Test Plan Test Plan configuration settings. Specifies default config file location, and location of TDV JDBC driver JAR file.

Baseline Config JDBC Connection Configuration

Defines configuration of a connection pool used to execute queries against the baseline data source. TDV server host name and port, and authorized user credentials should be supplied.

Target Config JDBC Connection Configuration

Defines configuration of a connection pool used to execute queries against the target (refactored) data source. TDV server host name and port, and authorized user credentials should be supplied.

Test Execution Thread Group A simple container that groups together queries executed during the test. Defines the number of threads (virtual users) running concurrently; values > 1 are useful for load testing, for a clean regression test we just set it to 1.

CSV Data Set Config CSV Data Set Config

Element that reads lines from a query file, and splits them into variables. File name is determined by a property QUERY_FILE (defined on a command line), or, in the absence of it, by the DEFAULT_QUERY_FILE variable. More about query file structure – in the next section.

BASE_${Test Plan}_${Test Name}

JDBC Request Runs a query specified by user defined variable ${Test Query}, against baseline data source. Variables ${Test Plan}, ${Test Name} and ${Test Query} are assigned values from corresponding columns of a query file.

Assert Error Response Assertion

Evaluate the response to see whether query was successful. Currently, only asserts the presence of word “Exception” in the response text.

Baseline Metrics BeanShell Postprocessor

Calculates the digest of a response received from the baseline data source. This digest will be compared against the digest of a response from target. Also, stores the response time of a query, and the offset duration, which is the maximum time a query may take before being considered to have run for too long.

TARGET__${Test Plan}_${Test Name}

JDBC Request Runs a query specified by user defined variable ${Test Query}, against target data source.

Assert Error Response Assertion

Evaluate the response to see whether query was successful. Currently, only asserts the presence of word “Exception” in the response text.

Assert Duration Duration Assertion

Compares response time of a query against target data source with that of the baseline.

Assert Target MD5 MD5Hex Assertion

Compares MD5 digest of a response from the target data source with that of the baseline.

Summary Report Summary Report

Summary of the results of all queries executed by the test plan.



Assertion Results Assertion Results

List of all assertions with their success/failed status.

View Results Tree View Results Tree

A tree of all responses including query text and response data. Should be turned off (disabled) upon regression test reaching certain stability level.

jp@gc... Response Graphs

Visualization means to represent essential parameters of the results. Can be turned off or removed if not needed.

3.2 Test Case File Query file processed by the above test plan is a .csv file. It should have valid SQL queries defined in the column labeled “Test Query”. During run-time, a script reads the file one record at a time and executes those queries, first against baseline, and then against target environments.

Two other required columns are “Test Name” and “Test Plan”. They are used to uniquely label each executed test case, see Reviewing Test Results section further. There may be more columns in .csv, for the purposes of QA automation for example, but they will be treated as purely informational and won’t be used by the script.

3.3 Reviewing Test Results Summary Report screen presents the list of all executed queries. It provides basic metrics: min-max-avg response time, bytes received and throughput, and error percentage. Test cases are labeled as BASE_ (or TARGET_) ${TEST_PLAN}_${TEST_NAME} where the user variable values TEST_PLAN and TEST_NAME are taken from corresponding columns of a Query Config file.



In the Results Tree View (if enabled), we can review text of the SQL queries and the results of their execution.

A couple of notes:



- MD5Hex is a non-commutative function, so two result will be evaluated as equal only when both the order and the content of each row are identical

- To avoid running out of memory when processing large result sets, we may want to limit the number of rows by using FETCH X ROWS ONLY.

jp@gc components provide graphical representation of various test metrics.



4 Conclusion

This document provides technical details of the JMeter test plans that were developed to perform functional regression testing after an upgrade from TDV 6.2 to TDV 7.0.2. It briefly describes the process of creating and modifying test cases for each RT phase as well