automating performance testing

8/8/2019 Automating Performance Testing

1/12

AUTOMATING PERFORMANCE TESTING

FOR PREDICTING SYSTEM BEHAVIORAND PERFORMANCE


2/12

Abstract.2

What.is.Performance.Testing?.3

Why.Automate.Performance.Testing?.4

The.Automated.Performance.Testing.Process. 4

. The.Design.Phase.5

. The.Build.Phase.6

. The.Execute.Phase.6

. The.Analyze,.Diagnose,.and..

. Tune.Iterative.Phases.6

Who.Should.Be.Involved.in.an.Effective..

Performance.Test?.7

Why.Mercury.LoadRunner?.8

. Simplifying.Script.Creation.with.Mercury..

. LoadRunner.Click.and.Script.9

. Using.Mercury.Diagnostics.to.Resolve..

. Performance.Problems.9

. New.Features.in.Mercury.LoadRunner. 10

Summary.11

TABLE OF CONTENTS

ABSTRACT

Successul businesses rely on specialized sotware applications to drive eiciency and productivity gains

across the enterprise. These sotware solutions can provide a more eective medium or collaboration

and inormation sharing and have become the primary channel both or business-critical inormation

sharing and transaction processing o all kinds. Todays sotware applicationsrom email to CRM to

transaction processingrun the business.

While sotware development technologies have changed and matured tremendously over the last ew

years, the complexity o modern applications has also exploded. Applications may utilize hundreds o

separate components to do work once done with paper or by hand. This complexity directly correlates

to more potential points o ailure in a business process and more diiculty in isolating the root cause

o a perormance problem.

Moreover, sotware applications dont work like a car. They dont have permanent parts that are only

replaced when they wear out. Whether they are designed to deliver competitive advantage or to respond

to changing business conditions, sotware applications are evolving weekly, monthly, and annually. This

stream o changes introduces yet another set o risks that companies have to manage.

This incredible rate o change and the explosion o sotware complexity introduce tremendous risk into

the sotware development process. Rigorous testing is the most common strategy to both quantiy and

reduce this risk to the business. The question or developers, QA teams, and management alike is how to

accurately and thoroughly validate system perormance beore going livewithout breaking the IT budget.


3/12

By automating perormance testing, companies can veriy that applications meet the needs o the business beore going live so there are no

deployment surprises, quantiy the impact o changes on the end-user experience, and pinpoint ailing components or rapid resolution. Yet the

prospect o automating perormance testing raises new issues. Beore undertaking an automated testing program, enterprises need to thoroughly

understand:

What is perormance testing? What should it accomplish?

Why should it be automated?

What are the right processes or perormance testing?

Design

Build

Execute

Analyze, Diagnose, and Tune

Who should be involved in a good perormance test to ensure success?

Who needs to see the results and how can results be reported to best quantiy ROI?

Which eatures are essential when comparing automated load testing solutions?

Mercury oerings support more than 90 percent o the Fortune 500 and more than 73 percent o all load-testing initiatives. Through its 14 years

o load-testing experience and with considerable customer input and eedback, Mercury has accumulated signiicant expertise in the proper

methods o automating the perormance testing process.

This paper presents a brie overview o the beneits o automating perormance testing and covers how best to approach perormance testing.

It also summarizes the key unctions and beneits o Mercury LoadRunner, the industry-standard automated load-testing solution that is part o

Mercury Perormance Center.

WhAT IS PErFOrmANCE TESTINg?

Perormance testing is the only way to accurately test the end-to-end perormance o a system prior to going live. Perormance testing solutions

should be able to:

Emulate hundreds or thousands o users interacting with the system using minimal hardware

Measure end-user response times

Repeat load in a consistent way

Monitor system components under load

Provide robust analysis and reporting engines

Eective automated testing solutions typically use our major components to build and run tests. These include:

A Virtual User Generator to capture end-user business processes into automated scripts

A Controller to organize, drive, manage, and monitor the load

Load Generators to run the virtual users during execution

An Analysis Engine to view, dissect, and compare results


4/12

Why AuTOmATE PErFOrmANCE TESTINg?

Automated perormance testing is a discipline that leverages people, processes, and technology to reduce the risks o application deployment,

upgrades, or patch deployments. At its core, automated perormance testing is about applying production workloads to pre-deployment systems

while simultaneously measuring system perormance and end-user experience. A well-constructed perormance test should be able to answers

questions like:

Does the application respond quickly enough or the intended users?

Will the application handle the expected user load and beyond?

Will the application handle the number o transactions required by the business?

Is the application stable under expected and unexpected user loads?

Will users will have a positive experience (i.e., ast response time) on go-live day?

By answering these questions, automated perormance testing solutions quantiy the impact o a change in business terms. This in turn clariies

the risks o deployment. An eective automated perormance testing process can help the enterprise make more inormed release decisions

and prevent system downtime and availability problems.

ThE AuTOmATEd PErFOrmANCE TESTINg PrOCESS

Figure1:Thefourphasesofaneffectiveautomatedperformancetestingprocess.

Organizations that have successully implemented automated perormance testing have done so by breaking the process into discrete phases.

While speciic implementations may dier, perormance testing can be broadly described as having our phasesDesign, Build, Execute, and

Diagnose/Tune. Each o these phases has speciic tasks involving dierent roles that should be completed beore moving to the next phase. At

the highest level, the our phases can be described as ollows:

The desin Pase involves dening the business processes to be tested, the business process mix o an average or peak production hour,

and the overall user and response time goals or the system.

The Bil Pase involves setting up and conguring the test system and inrastructure, and using the automated perormance testing

solution to build both test scripts and load scenarios.

The Execte Pase consists o running the load scenarios and measuring system perormance.

TheAnalze, dianose, an Tne Iteatie Pases go beyond measuring system perormance and take load testing to another level.

Here, the ocus is on pinpointing problems to help resolve them rapidly and tuning system parameters to maximize perormance in an

ongoing ashion.

The ollowing sections will drill down another level to examine the tasks that are necessary to make each separate phase o the automated

perormance testing process successul.

DESIGN

BUILD EXECUTEDIAGNOSE

AND TUNE

GATHER

REQUIREMENTS

DESIGN TEST

STRATEGY

DEFINE BUSINESS

PROCESS

DEFINE SYSTEM

WORKLOAD

SET UP TEST

ENVIRONMENT

RECORD TEST

SCRIPTS

CREATE TEST

SCENARIOS

BENCHMARK

TESTING

PERFORMANCE

TESTING

SCALABILITY

TESTING

REPORT

GENERATION

DIAGNOSE

BOTTLENECKS

TUNE

CONFIGURATIONS

QUANTIFY

IMPROVEMENTS


5/12

Te desin Pase

The Design Phase is the primary time where the perormance testing team works with the line o business (LOB) mangers to gather

perormance requirements. Requirements can be thought o as alling into our bucketsbusiness, technical, system, and team requirements.

Business requirements are generally gathered by meeting with subject-matter experts (SMEs). These may be business analysts or end users. A

comprehensive set o business requirements exists when the ollowing are in place:

An application oeiew: A demo o system usage is created to allow the perormance team to understand at a high level how theapplication is used.

A bsiness pocess list: A list o the key business processes is generated to refect the activities that end users perorm on the system.

Bsiness pocess ows: Word documents are created to detail the exact steps/screens o each business process.

A tansaction list: A list o the key activities within the business process that need to be measured under load is complied, such as Login

or Transer Funds.

Bsiness pocess iaas: Business process fowcharts are created to illustrate branching conditions in the business process fows.

Technical requirements can be gathered by meeting with the system administrators and database administrators (DBAs). These individuals may

be part o the enterprises development group, operations division, or both. A comprehensive set o technical requirements exists when the

ollowing are completed:

An enionent walkto: A walkthrough is conducted rom the systems or inrastructure team on the testing architecture.

Sstes scope eetin: A meeting is held to discuss and agree what pieces o the system will be stubbed out or excluded rom the test

process.

Poction iaa: A diagram o the production inrastructure is created to fag deltas that may impact perormance during the migration

rom QA to production.

Last but not least, system requirements must be gathered. These are the high-level goals or the system that govern the pass/ail status o the

load-testing process. These generally are agreed upon by working with the project manager rom the LOB. System requirements include answers

to the ollowing:

How many users must the system support at normal and peak periods?

How many transactions per second must the system be able to process?

What are the minimum and maximum acceptable response times or all business-critical transactions?

How do the user communities connect to the system?

What are the system workloads experienced in production? What are the transaction mixes?

Team requirements are the last issue to be ironed out beore moving to the build phase. This consists o determining the right perormance

team members who will participate in the upcoming load test. Initially, this may be determined automatically (e.g., when there is a team o just

one person). However, i perormance testing becomes part o a Center o Excellence (CoE), resource allocations and internal logistics should

be handled in the Design Phase.

Gathering a complete set o business, technical, system, and team requirements up ront lays the oundation or an eective and successul

load test.


6/12

Te Bil Pase

The Build Phase consists o turning the business processes and workloads identiied in the Design Phase into automated components that can

be leveraged to drive repeatable, realistic load. This can be broken into two areas o ocus: automation setup and environment setup. Automation

setup consists o a set o sequential tasks done typically by a perormance engineer:

1. Sciptin: Record the documented business processes into automated scripts

. Tansactions: Insert timers to produce the logical timings desired by the business

. Paaeteization: Replace all input data, such as login IDs and passwords, with a pool so each virtual user accesses the application using

unique data

. Scenaios: Create production workloads by assigning groups o users dierent scripts, connectivity, and user behavior

. monitos: Decide which servers or machines to monitor under load

Environment setup consists o assembling the hardware, sotware, and data required to execute a successul, realistic load test. This may involve

working with the Systems, DBA, Operations, and Business teams. The end result o the Build Phase is a set o automated assets that can be

executed at will on an available, conigured environment.

Te Execte PaseAmong those new to perormance testing, there is oten a misconception that execution is a single event. In act, it is a multi-step process

consisting o several types o perormance tests. Each type o test provides inormation necessary to understanding the business risk o releasing

the application. The dierent types o load tests include:

1. Baseline tests veriy that the system and its surrounding environments unction within reasonable technical parameters. Perormance tests

are run with only ve to 10 users to baseline end-user transaction perormance. These tests should be executed at the start and end o the

perormance testing process to measure absolute response-time improvement.

. Peoance tests simulate a load on an environment to determine the optimal and maximum number o users the system can handle.

These tests should emulate average and peak hour production usage. They should apply real-world user behaviors such as think time,

modem emulation, and multiple browser types or maximum accuracy. All monitors and diagnostics should run or maximum visibility into

system degradation and bottlenecking.

. Bencak tests are designed to measure and compare the perormance o each machine type, environment, or build o the application in

an ideal situation. These are run ater the system has been scalability tested to understand the perormance impact o dierent architectures.

. Soak tests are designed to run the system or a long period o time under load and examine how well the system perorms.

. Peak tests are designed to simulate peak load on the system or a period o time to ensure that the application and the underlying

hardware can handle high load or reasonable periods o time.

Te Analze, dianose, an Tne Iteatie Pases

Ater the Design, Build, and Execution phases o a load test are completed, the project advances to the Analysis, Diagnosis, and Tuning Phases.

These unctions are ongoing, and conducted iteratively. The load testing solution should provide a comprehensive view o end-user, system-,

and code-level perormance data, and identiy the likely causes o system slowdown. This enables users to determine i perormance goals havebeen met and i not, why not and who owns the problem.


7/12

How to Determine the ROI of Performance Testing

The ROI o a good perormance testing solution consists o two components:

risk itiation ensures that a project goes live with the right system scalability and perormance. Risk mitigation is classic perormance

testing. Most users o automated perormance testing are able to report back inormation to the development/project teams that provides

clear, quantiable inormation on how well the system will scale and perorm in production.

Peoance optiization quantiably improves the perormance o the system as measured by improved end-user response time or by

reducing the overall required hardware inrastructure.

How to Optimize Performance

During and ater a perormance test, a wealth o inormation suraces that can improve system perormance. Key inormation can be uncovered

during monitoring, analyzing, tuning, and during diagnostics.

1. monitoin: Monitoring during a perormance test shows what is happening at each tier o the inrastructure, oering more clarity about how

the database server, web server, application server, or individual application or processes are perorming during a test. This inormation may

quickly surace valuable inormation, or example that the CPU at the application server is being pegged at 100 percent with 200 users, well

short o the goal o 300 users. This would point to either requiring more application server capacity or optimizing the application itsel.

. Analsis: Ater a load test is completed, the user can correlate metricssuch as virtual users against CPU or application server CPU to webserverto uncover additional inormation about application behavior.

. Tnin: Many companies conduct automated perormance testing beore, during, and ater application deployment. Some automated

perormance testing solutions can systematically identiy, isolate, and resolve inrastructure perormance bottlenecks by modiying the

system conguration settings. By iterating through the process o resolving inrastructure bottlenecks, users can establish the optimal or

golden conguration or go-live.

. dianostics: An eective perormance testing solution should provide perormance engineers with a single, unied view o how individual

tiers, components, and SQL statements impact the overall perormance o a business process under load conditions. Perormance

engineers will be able to see all components touched by an end-user transaction and determine how long each component takes and

how many times it is called. With this inormation, project and QA managers can ocus resources to improve the end-user experience by

targeting the most signicant web, application, and database server bottlenecks.

WhO ShOuLd BE INvOLvEd IN AN EFFECTIvE PErFOrmANCE TEST?

A successul perormance testing project requires the contributions rom a variety o individuals. Some o the roles that should be included in a

testing program include:

Poject anae: Coordinates multiple perormance projects, manages testing schedules, acquires necessary hardware and/or sotware,

and handles resource and unding issues

Bsiness analst: Accountable or review and sign-o o the perormance o the system rom a business standpoint; assists in the

development o the transaction mix and the expected timings or perormance testing

Peoance anae: Accountable or coordinating the eorts o the perormance support team, acting as the point o contact or the

group; responsible or managing the day-to-day activities o the perormance eort

Peoance testes: Responsible or the creation and execution o the automated tests and the gathering o the test results

Application acitect: A recipient o the inormation rom diagnostics and analysis o a load test to optimize application perormance or to

solve perormance bugs

Inastcte specialists (dBAs, netwok ainistatos, sste acitects): Recipients o the inormation rom tuning and analysis o

a load test to optimize system perormance or to solve perormance bugs


8/12

Why mErCury LOAdruNNEr?

Mercury has over 77 percent o the perormance testing market share (IDC, Worldwide Distributed Automated Software Quality Tools Market

Analysis 2005). Mercury LoadRunner prevents costly perormance problems in production by detecting bottlenecks beore a new system or

upgrade is deployed. Enterprises can veriy that new or upgraded applications will deliver intended business outcomes beore go-live, preventing

over-spending on hardware and inrastructure. It is the industry-standard load testing solution or predicting system behavior and perormance,

and the only integrated load testing, tuning, and diagnostics solution in the market today.

With Mercury LoadRunner web testing sotware, enterprises can measure end-to-end perormance, diagnose application and system bottlenecks,

and tune or better perormanceall rom a single point o control. It supports a wide range o enterprise environments, including web services,

J2EE, and .NET.

The ollowing section describes key capabilities o Mercury LoadRunner that make it the industry leader in this space:

1. On-ean poction wokloas: Mercury LoadRunner is able to drive hundreds and thousands o virtual users executing dierent

business processes to emulate the production conditions a deployed application will ace. This helps uncover perormance and scalability

bottlenecks that would otherwise surace in production beore going live. It also helps minimize production downtime and poor perormance,

and meet service-level and uptime requirements.

. Entepise enionent sppot: Mercury oers the most extensive testing environment with support or all o the industrys leading

protocols and platorms. Mercury LoadRunner now supports over 60 protocolsmore than any other vendor. This includes web, J2EE, .NET,

XML, SAP, Siebel, Oracle, PeopleSot, wireless, Citrix, and client/server applications. As the types o applications being deployed change

rom client/server to web to Java, the same tool can be used or perormance testing. It oers one consistent tool and one set o employee

skills even i applications change over time, as well as lower TCO.

. Entepise onitoin sppot: Mercury LoadRunners non-intrusive, real-time perormance monitors provide detailed metrics on all parts

o the system under test. This includes web servers, application servers, databases, ERP and CRM systems, rewalls, and load balancers.

Mercury LoadRunner can identiy hardware limitations and sotware conguration issues that might otherwise go undetected.

. dianostics: Mercury LoadRunner is the only perormance testing solution that can trace, time, and troubleshoot individual application

components under load. Users can now drill down rom a slow end-user transaction to the bottlenecked method or SQL statement that

is causing the slowdown. This granularity o results data ensures that every load test provides development with actionable results, thus

reducing the cost and time required to optimize J2EE, Siebel, and Oracle deployments.

. Atoatic analsis: Mercury LoadRunners AutoCorrelation Wizard automatically takes all the monitoring and diagnostics data and

provides the top-ve causes o perormance degradation. As a result, Mercury LoadRunner has the industrys deepest and broadest

analysis. Converting perormance testing results into actionable, precise data to development dramatically reduces time to resolution and

allows or more testing cycles. This is a critical piece in ensuring that a high-quality application goes into production.

. Ease o se: Mercury LoadRunner is built rom the ground up or QA users. It provides a visual scripting language, Data and autocorrelation

wizards, and ActiveScreen technology to make scripting and running o load tests as easy as possible. This results in short ramp-up time,

immediate ROI, and perormance testing within two weeks o training.

. TboLoa: Mercury LoadRunner requires minimal CPU and memory resources per virtual user or maximal scalability on minimal hardware.

This ensures that there is little or no hidden hardware costs in implementation.

. unife sciptin enine: Mercury LoadRunner has the same scripting engine as Mercurys Application Management oerings. This

ensures lower training costs, lower scripting costs, and lower total cost o ownership o Mercury solutions.


9/12

Sipliin Scipt Ceation wit mec Loarnne Click an Scipt

The majority o Mercury LoadRunner users test web-based applications. These users typically spend 70 percent o their time scripting tests.

Until now, all scripts were recorded using low-level HTTP protocol to capture web applications or perormance testing. But HTTP scripts are long,

time-consuming to create, and hard to interpret and maintain.

Mercury LoadRunner Click and Script provides a new approach or easy generation o scripts or web load tests. It is based on a script describing

user-level actions and is similar to the GUI level scripts in Mercury QuickTest Proessional

. Mercury LoadRunner Click and Script Technologyenables users to record scripts at a higher presentation layer, making the scripting process much aster and easier. Click and Script Technology

automatically captures the most valuable scripting inormation to create intuitive and sel-explanatory scripts, expressed in user-actions

termssuch as pressing a button, illing an edit ield, etc. It also executes client-side JavaScript codejust like the browser doeswhich virtually

eliminates the need or correlations. It dramatically reduces scripting time and provides much more succinct inormation that is easier to interpret

and understand.

Mercury LoadRunner Click and Script makes it much easier to read testing scripts, reducing the time required to develop scripts by as much

as 80 percent. These scripts are also much easier to maintain, since anyone can look at the scripts and quickly see what is going on in each

statement. Mercury LoadRunner Click and Script Technology allows customers to ocus on more testing cycles, more applications, and more

analysis, as well as ocus on providing their lines o business (LOBs) with accurate and meaningul perormance results.

Figure2:CycletimeefficiencyimprovementsusingMercuryLoadRunnerClickandScripttechnology.

usin mec dianostics to resole Peoance Pobles

Mercury Diagnostics extends Mercury LoadRunner and Mercury Perormance Center to address the unique challenges o testing complicated

J2EE and ERP/CRM applications across the application liecycle. Mercury Diagnostics isolates application perormance problems and reduces

the mean time to resolution (MTTR) o application perormance bottlenecks. It provides actionable inormation to resolve perormance problems.

Mercury Diagnostics provides the ability to:

Find and solve more problems earlier in the liecycle

Achieve a higher quality bar by nding the most common application problems beore they go live

Collect concrete data to support decisions around the go-live o an application

Manage and monitor applications ater they have gone live with role-based visibility to quickly solve problems

PROJECT TIMELINE

CLICK & SCRIPT

SCRIPT

DEVELOPMENT

SCRIPT

DEVELOPMENT

RUN

TEST

RUN

TEST

ADVANCED

ANALYSISFIX

CLICK & SCRIPT

SCRIPT

DEVELOPMENT

RUN

TEST

ADVANCED

ANALYSISFIX

ANALYSIS FIX

MERCURY

LOAD

TESTING

REDUCE

SCRIPTING

TIME


10/12

1 0

During a perormance test, Mercury Diagnostics traces J2EE, .NET, or ERP/CRM business processes rom the client-side across all tiers o

the inrastructure. The modules then break down each transaction response time into time spent in the various tiers and within individual

components. Perormance testing teams gain:

An intuitive, easy-to-use view o how individual tiers, components, memory, and SQL statements impact overall perormance o a business

process under load conditions. During or ater a load test, testers can not only point out to the application team that the application is not

scaling, but also provide actionable data to them.

The ability to eectively triage and nd problems with business context, enabling teams to ocus on problems impacting business

processes. The ability to more easily nd components relevant to a specic business process under test. Because J2EE and ERP/CRM

applications potentially use thousands o components, this can be a challenge. Mercury Diagnostics automatically detects which

components are active when a given transaction is executed and collects data on them or analysis. Components untouched by the

business process are ltered out, enabling the team to ocus on getting the job done, not conguring the system.

New Feates in mec Loarnne

Mercury LoadRunner has recently added a wide variety o new eatures or load testing. The ollowing sections describe only a ew o the new

options available in Mercury LoadRunner. (For complete inormation on the latest Mercury LoadRunner eatures, visit the Mercury website at

www.mercury.com.)

Web Services and SOA

Service-oriented architecture (SOA) in general and web services in speciic are redeining the architectures enterprises are using to provide

LOBs with applications that matter to their bottom line. Perormance testing o this management layer is essential or ensuring proper application

perormance in the ever-changing world o shared services and components.

Mercury LoadRunner now includes broad support or web services and SOA testing, such as:

An SOA Script Generator wizard or selecting testing aspects o SOA environments and creating scripts based on these selections; tests

include boundary testing, interoperability, and security-related issues.

The ability to create test scripts to emulate servers by analyzing server trac capture les

An importing tool to import WSDLs rom les, URLs, UDDI registries, and Mercury Quality Center, plus the ability to automatically update

WSDLs rom their source

Checkpoints to determine i the service perormed correctly

The ability to congure the transport level or messages: HTTP/S or JMS

XML parameter support, providing the ability to replace an entire XML structure with a single parameter

Asynchronous messaging with WS Addressing support

Security enhancements with SAML and token support with message signatures and encryption

Full integration with Mercury Quality Center, including the ability to arrange services and tests in a repository and create test plans, set

requirements, and track deects or web services


11/12

1 1

Microsoft .NET

Mercury LoadRunner has also added enhanced support or Microsot .NET, including:

Support or Framework 2.0

Support or .NET Remoting, ADO calls, and custom libraries

Support or C# and provides improved Visual Studio 2005 integration

New recording options or managing and creating custom lters in customized environments

Mercury LoadRunner Click and Script

Mercury LoadRunner Click and Script now provides:

Support or applications in all languages

Support or recording on Windows 2003

New limited applet support

Enhanced API or verications (check points) and or easier scripting

Additional enhancements, especially or AJAX applications, recording engine, and replay snapshots

SummAry

Todays businesses cannot gamble on the perormance o their mission-critical applications. By automating perormance testing, enterprises can

prevent costly perormance problems in production by detecting bottlenecks beore new systems or upgrades are deployed.

Mercury LoadRunner is the industry-standard load testing solution or predicting system behavior and perormance, and the only integrated

load testing, service testing, and diagnostics solution in the market today. With Mercury LoadRunner web testing sotware, enterprises can

measure end-to-end perormance, execute asynchronous perormance testing, and diagnose application and system bottlenecks or better

perormance.


12/12

Mercury is the global leader in business technology optimization (BTO). We are committed to helping customers optimize the business outcome of IT.

WWW.MERCURY.COM

2006.Mercury.Interactive.Corporation.Patents.pending.All.rights.reserved.Mercury.Interactive,.Mercury,.the.Mercury.logo,.Mercury.Performance.Center,.Mercury.Quality.Center,.Mercury.Diagnostics,.Mercury.LoadRunner,.and.Mercury.

QuickTest.Professional.are.trademarks.or.registered.trademarks.of.Mercury.Interactive.Corporation.in.the.United.States.and/or.other.foreign.countries.All.other.company,.brand,.and.product.names.are.marks.of.their.respective.holder.

WP-1083-1106

automating performance testing

Documents