performance testing

Performance Testing By Jaya and Anji

Table of ContentsTable of Contents ............................................................................... 1

Performance Testing Life Cycle .......................................................... 2

Scenario Design ................................................................................. 4

Scripting ............................................................................................ 6

Recording ................................................................................................................................................. 6

Customization .......................................................................................................................................... 7

Page Validation ...................................................................................................................................... 10

Test Execution ................................................................................. 11

Runtime Settings and other options ...................................................................................................... 13

Analysis ........................................................................................... 14

Analyzing the test results and finding bottlenecks ................................................................................ 14

Bottlenecks ............................................................................................................................................ 16

Most common mistakes done during application set-up ....................................................................... 17

Conclusion ....................................................................................... 19

Confidential Property – Please don’t share to others except our friends.


Performance Testing Life Cycle

Let us discuss the performance testing life Cycle of a typical service based companies. The

phases of performance Testing are as below:

1. RFP (Request for Proposal)

2. SoW(Statement of Work)

3. Discovery

4. Planning

5. Automation

6. Smoke Tests

7. Final Tests

8. Analysis and Report

9. Re –test if needed

1. RFP: The clients, who are looking for performance testing their products, will

provide the organization a brief introduction about their application and requests for

the proposal of performance testing. The sales team will analyze it and prepares a

Statement of Work based on the client calls and Email Communication that is

happened during this phase.

2. SoW: Based on the information provided by client, the sales team will prepare a

Statement of Work which highlights the scope of the project, the tool, the user load,

the kinds of tests, the total number of scenarios(sometimes they are also called as

use cases) etc

3. Discovery: Once the SoW is signed, the Team lead will initiate the discovery

process. In this phase, we will get more clarity on what is to be done, the application

architecture, the exact scenarios to be automated, the think times, the transaction

mix or the user mix, the SLA’s if any. This phase is proceeded either by the

Email/phone communication between the client and the team. The team will be

introduced to the team and their responsibilities are also identified. At the client end,

the point of contact will be identified. Usually, the duration of this phase is 3-4 days.

At the end of this phase, the team lead will send an Email setting up the

expectations to the client.



4. Planning: This phase is to prepare a plan, to calculate the resources required, the

infrastructure required etc for each phase. A performance test plan will be delivered

to the client at the end of this phase.

5. Automation: The automation for the identified scenarios will be completed in this

phase

6. Smoke Tests: Validate the automation scripts and test environment for few

users (5 -10 users)

7. Final Tests: Once all the above phases are completed successfully, we will go for

the actual tests (can be load, stress, endurance and what ever)

8. Analysis and Report: Once the tests are completed, the client side and server

side (if in scope) metrics are analyzed for bottlenecks (if any). A detailed test report

will be delivered at the end of this phase.

9. Re –tests: Re – test if the client requests

For further drill down of each phase, continue reading



Scenario Design

We will have multiple discussions with client and will understand the application

architecture, server configuration and his application.

Then we will identify the business scenarios based on the following parameters. (Most of the

cases, the business scenarios are identified by the client, if not we will help them in

finalizing the scenarios).

1. The scenario that is being used by most of the users.

2. The scenario which generates more revenue to the client.

3. The scenario which is critical to the application.

After finalizing the scenarios, we will have a walkthrough of the scenarios to make sure that

the functionality of the scenarios is fine (Functionality means whether the screens and

expected output is coming or not, not end to end functionality).

Then, we will decide the number of users and types of performance testing.

The types of performance testing are

1. Load Testing: The test execution with a specified user load under given conditions.

Major focus will be on system stability, response times and success and failures.

2. Stress Testing: The test execution where we continuously increase the load on the

application until it crashes. Major focus will be on the threshold limit of users and

response time is not at all considered.

3. Endurance Testing: The test execution where we load the application continuously

for longer durations like days or months to identify bottlenecks like memory leaks.

4. Spike Testing: The test execution where the application is loaded with varied user

load suddenly (sudden increase and decrease) to check the system behavior.

5. Volume Testing: The test execution where we send the continuous data to the db

to check whether it can handle that much volume of data at a time.

Once we decide all these things, we have to decide other things like think time, ramp up,

duration, server monitoring and analysis.



Then we prepare a test plan with detail description of our activities and deliverables of this

project.

• The test plan consists of the following.

• Application Overview

• Application Architecture

• Scenarios in detail (step by step description)

• User load

• Test schedule

• Server monitoring (metrics)

• Deliverables (test plan , automation scripts, Analysis, reports)

• Assumptions, Limitations

• Entry criteria & Exit criteria



Scripting

Recording

Once we get the sign off from the client, we will start with scripting.

Since ours is a web application, we record with web (HTTP/HTML) protocol.

Then we have recording mode options

Html based script:

This is the general recording mode we select where the data will be captured at the HTML

level and appears in the script in form of GET & POST requests.

Here we have two advanced options

a. web_link, web_submit_form: This will capture the data in terms of link and form names.

(In other words, it is page level recording and it is contextual recording which means each

request is dependent on the previous request)

b. web_url, web_submit_data: This will capture the data in terms of link & form data. (In

other words it is network level recording; it is also called as context less recording, each

request is independent of the other requests)

URL based script:

This is generally used for component based testing.

Each and every request (js, css, img, gif) will be captured as a separate request and hence

the script will be of huge size. This is the reason we always prefer the HTML mode

recording. The purpose of having the URL based recording is, if the component



communicates with the server occasionally, it may not be recorded by HTML (for example in

case of applets). In such situations, we should go for URL mode.

Once the record button is pressed in the VUGen, all information related to the business

scenario will be saved to a log file called Recording Log. Based on the recording settings

that we choose, the script will be created and the entries are logged to Generation Log.

Customization

Correlation:

What? Capturing the dynamic data that is being generated by the server is called

correlation.

Why? To simulate the real user actions and real sessions, if not correlated, the scripts may

fail

How?

It can be done in two ways.

Automatic correlation:

In LoadRunner auto correlation can be done at two phases:

Enable Auto Correlation: In the recording options Correlation Enable auto

correlation. LoadRunner maintains a set of predefined correlation rules called

autocorrelation library. If any dynamic values from these rules are found, LoadRunner

automatically inserts the web_reg_save_param() function into the script.

Scan for Correlation: Once the script is recorded, it can be replayed and then we can do a

scan for correlations (shortcut is F8). Once we replay the script, a Replay Log will be

created. When we do a scan for correlation, VUGen compares the recording log and the

replay log and highlights the differences between them. We can click on Correlate button on

the right bottom of the log window to correlate the differences.



But Auto correlation cannot ensure that we have captured all the dynamic values (especially

when may miss user specific data). This is the reason; we cannot depend Only on Auto

Correlation.

Manual correlation: To ensure that our script is fully correlated, we need to go for manual

correlation as well. In manual correlation, we have to identify the dynamic data generated

by the server and then correlate them using the function web_reg_save_param().

We can identify the dynamic values by recording the script three times:

1. Record with user 1(business flow)

2. Record with user 1(same steps as above)

3. Record with user 2(same steps as above but with different input values)

Compare the first two scripts using windiff or any other tools which highlights the session

specific dynamic values (for example: session id, ordernum, PNR number etc).

Compare first and third scripts, this will highlight the user specific dynamic values (for ex

already registered credit card info, his membership details etc).

If we found any differences, using generation log we can get the left and right boundaries of

the dynamic value. We use web_reg_save_param function for correlation.

The parameters for this function are

Parameter name

LB (left boundary): the source data on left side of the dynamic value.

RB (right boundary): the source data on right side of the dynamic value.

ORD (ordinance): the number of occurrence of the value between same LB& RB

Search (body/all/head)

LAST: end of the function.

Parameterization:



What? Sending different user inputs for different users

Why? To simulate the real user actions, to ensure that the results are not fetched from

cache, to stress the db.

How?

Select the value that is to be replaced with a parameter Right click Replace with a

parameter Parameter Name, Parameter type

LoadRunner have various parameter types, but reading the values from a data file is most

commonly used. The others are random number, date and time, Table etc

For data file type, the next row from the file can be fetched in three ways.

• Sequential: takes the data sequentially

• Unique: data will be used only once, we need to assign the block size for each VUser.

For ex: we have 1000 values in the file and we are doing 100 user test, we can

assign 10 values for each VUser. The other users will not be able to access these 10

values.

• Random: data will be picked randomly

The below table explains in detail about these options:

# of rows of a table is: 2000

Block size for unique: 10

All the three users are executed simultaneously:

UserFirst Iteration Second Iteration

Sequential Random Unique Sequential Random UniqueVUser1 1 67 1 2 176 2VUser2 1 74 11 2 288 12VUser3 1 89 21 2 786 22

In the above case, please note that though we are selecting sequential all the three VUsers

are getting the same value. This is due to the fact that the “Script is executed per user,



and the data file is a part of the script”. If this is not accepted as per the customer’s

requirement, then we need to go for Unique.

Page Validation

What? To verify a static text of a particular web page

Why? When executed for multiple users, it is not possible to verify whether all the pages

are successful or not using the “Browser Display”. To overcome this, we can verify for a

static text of a particular page in the response of the request. If the static text is found, it is

considered as success else it is considered as a failure.

How? The function web_reg_find () can be used to check whether we got the expected

page by giving the static text of the expected page.

Once the scripting is done, we do a smoke test with limited number of users to make sure

that scripts & application is working fine.



Test Execution

Once scripting is ready, we will go for test execution.

The below are the activities performed before we start the actual test:

1. Configure the scenario

2. Configure the monitoring

3. Verify the run time settings and other important settings

Configure Scenario

As a best practice, we need to create the scenarios well in advance for the final tests. There

are two types of Scenarios (do not confuse this from the business scenario, in LoadRunner

scenario represents the test set-up)

Goal Oriented Scenario: If the customers requirement is a specific goal in terms of Hits

per Second or pages per second or transactions per second, this option would be used to

configure the goal. The user load would be automatically updated based on the goal (We do

not need to increase the load manually or we do not need to set-up the ramp-up patterns).

Manual Scenario: This is the typical scenario that is used in the day to day life of a

performance engineer. The main objective here is the User Load. The ramp-up, duration

and ramp-down are to be specified manually for this option. The Ramp-up patterns can be

specified for the overall scenario (set of groups) or for the individual group (each individual

script). The options are Schedule by Group and Schedule by Scenario.

Schedule by Group: If the customers’ objective is to execute a group after the completion

of another group or after 15 minutes of the first group etc, then this option is set up. This is

also used when each group has its own ramp-up, ramp-down patterns

We will use this when there is an interdependency of scripts.

Ex: scenario1 is raising a request for shift change

Scenario2 is accepting the request by some other user.



In this case, script 2 can be executed only after script1 is executed at least for some time or

completely.

Schedule by Scenario: If the customers’ objective to pump the user load at a specific rate

irrespective of the groups, then this option is used.

In LoadRunner Scenario, We have two tabs

Design tab: the schedule for test execution can be set here

Run tab: To start the test, monitor the results, monitor the servers.

Design tab:

Here, we will set the range of user load and run the tests.

Other settings in design tab are

1. Load generators (we can add different load generators by giving the LG installed m/c ip's)

2. Add Vusers (we can add the users during run time using this option)

3. Add Groups: we can add the scripts

4. Runtime settings: we can modify the run time settings of a script for a test.

5. Detail: details of a script

6. View Script: it will open the script in vugen.

Run Tab:

Here we run the scenario by giving path for storing the results.

This tab will shows you the details like passed transactions, failed transactions, hits/sec and

errors

This tab also shows you the min, max and avg. response times of the transactions of a

script.



We can also monitor the servers by selecting windows/unix resources.

Configure the monitoring

If the scope includes monitoring the servers as well, it can be done in two ways.

1. Monitoring from LoadRunner

2. Executing the utilities available on the machines

Monitoring from LoadRunner: LoadRunner has the facility to monitor almost all the

servers (for the hardware and application specific counters). To monitor the hardware

resources the steps are as below:

1. Go to the Ran tab of the LoadRunner scenario

2. Click on monitors Select Windows/Unix Monitoring (based on the OS, of the

servers)

3. On the right a graph will be shown Right click Add Measurements

4. Provide the IP address of the server and select the counters to be monitored

To monitor the application specific counters, it requires configuring some settings at the

application end.

Monitoring using the other tools: A document was prepared and shared to the people.

Runtime Settings and other options

Before starting the test ensure the following:

1. The result settings are updated

2. The Log settings of run time settings are appropriate

3. The connection timeout, receive timeout, and step download timeout are configured

4. The think times are proper

5. The run logic is correct



Analysis

Once the test is completed, the results can be viewed by clicking the analyze results option.

It launches the Analysis component of the LoadRunner which is used to analyze the test

results.

The default analysis summary contains the details like the test duration, total throughput,

average throughput, total hits and average hits per second, passed transactions, and failed

transactions, the min, max, avg and 90% of the response times. Filters can be applied on

the summary to get the statistics for a specified duration or for a specified load generator or

specified transaction and percentile requested.

The other section of the Analysis is the graphs section, the defaults are: transaction

summary, throughput, hits per second, avg transaction response time etc;

New graphs can be added, merged with the existing graphs and filters can be applied on

any graphs.

These statistics and graphs will be helpful in analyzing the trends, bottlenecks.

We can generate reports either in html/word report and can add our suggestions for the

graphs and share them with the client.

Analyzing the test results and finding bottlenecks

This section gives a brief idea on how to start the analysis of the test results. Once the test

is completed, the test results should be divided into three sections:

1. Till the normal and expected behavior

2. From where the response times, started increasing exponentially and started seeing

errors

3. From where the errors increased drastically to till the end of the test.

Note down the user loads at all these three levels.

The next classification would be based on the requests:



1. Identify the requests that are having high response times

2. Identify the requests having more failed transactions

Next classification should be based on the request type whether the request is GET or POST.

All the POSTs send the request to the database. If the high response times or the failed

rounds are observed for the POSTs, it could be possibly due to the database issues.

Next classification should be on kind of errors, whether we are seeing 500 internal server

errors, 503 service unavailable errors or step download timeout errors or page validation

errors. If most of the errors are step-download timeout errors that indicate a possible

network issues.

If most of the errors are due to 500 or 503, it could be due to the application issues.

If a particular request is taking very long time and the response times are high, it can be

analyzed using the Web Page Diagnostics graph. This graph provides the information at

1. Different layers – time to taken to establish, time taken in the SSL, time taken to

send the request, time taken to receive the first byte (Time to First buffer an

important counter that should be considered while analyzing for network issues) and

the receive time. If the receive time is relatively high compared to others, it could be

a network issue. If no network issues are observed and most of the time was spent

during time to first buffer, then the focus should be changed to the resource level.

2. Resource level statistics – Web Page diagnostics also provides the statistics at the

resource level. If the request with high response time is downloading 15 resources,

the time taken for each resource would be provided in the graph. If the static

resources like images, css and java scripts are taking more time, it could be due to

the Web server issue. If none of them are the reason, then it would be finally the

blame on the application server/db server.

Please note all the analysis that has been done is at the client end to find out the

bottlenecks. If servers are also monitored, the bottleneck can be pinpointed whether it was

due to the hardware resources or the configuration issues on Web/App/DB server or the due

to bad programming or due to poor SQL or a table that is not indexed properly.



Bottlenecks

Here I am mentioning three bottlenecks which I have faced in my projects.

Bottleneck1:

In .net application, Gen0 should collect all objects during garbage collection.

1/10th of GC0 should be collected at GC1 and 1/10th of GC1 should be at GC2

In my application GC1 has collected around 1/3rd of GC0.

So, I inferred that garbage collection has not done properly.

I suggested them to do application profiling for the scenario and find out the objects that

were not properly garbage collected.

Application profiling tools are

JProbe – Java

Ant, CLR -- .net

These profilers will show us all the objects that were being created during the scenario and

also show the garbage collection details of these objects.

With this, we can find how the garbage collection has happened.

Bottleneck2:

In .net application I found the sp’s that were causing deadlocks at the DB level.

I have done SQL Server profiling to identify this.

SQL Profiling gives you the details of sp’s that were being run during test execution, time

taken by them and show us if any sp is creating deadlock to other SP.

Also, we check the server logs for deadlocks.

With this I inferred that deadlocks are more for this particular transaction and you need to

resolve them.



Bottleneck3:

In Java application, I found that for a particular scenario the CPU has hit 95% and was there

for some part of time.

I found that this is because of limited number of threads available in the tomcat server and

since the requests were increasing, they started to be in queue which has affected the CPU

processing and ultimately it hit the CPU to peak 95%.

I suggested them to increase the number of server threads so that there will less number of

threads in queue and CPU peak utilization will be reduce. If it stills peaking, I advised them

to increase the CPU RAM.

Most common mistakes done during application set-up

Added to the above mentioned bottlenecks, the below are the common mistakes that are

done at while setting up the environment:

1. JVM Settings: JVM may not be configured properly to use the optimized heap

settings. They are left to the defaults which is 26 MB of heap. This may not be

sufficient at the high user load. This should be configured based on the OS and

application load. This can be set to the maximum of 3GB on a 32 bit OS and easily

till 8GB on a 64 bit OS. To monitor garbage collection details JSTAT can be used. If

the Full GC is happening very frequently that is probably due to the low heap

settings. If the heap settings are good and even then if the full GC is happening that

indicates a possible memory leak in the application.

2. Lock the resources: This is a common feature that comes with the UNIX based OS

where the administrator can lock the resources. If the RAM of the machine is 16GB,

the administrator can control the system to use only 4GB and same is the case with

the number of CPUs. If the administrator has locked the resources and the tester is

not aware of it, the memory utilization graph will show 100% where the actual

memory used is just 25% and the rest 75% is locked.

3. Poor configuration settings: The application might not be configured to support

the target user load especially in terms of the threads and JDBC connections at the

application level and memory settings at the DB layer. One has to ensure that these

settings are optimal for the test.



4. Missing Indexes on the table: On the database, indexes may not be created

properly for each table which results in high response time at the db to fetch the

results. This can be found using the execution plans for the tables



Conclusion

Hope, this document helps in understanding the end to end performance testing using Load

Runner. But please do not only on this document; you need to do your research for more

information.


performance testing - a brief introduction

Documents

performance testingby jaya

real user actions

application specific counters

high response time

run time settings

garbage collection

network issues