session # at22 rational user conference 2002 author note: to edit session # go to: view/master/title...
Post on 28-Dec-2015
214 Views
Preview:
TRANSCRIPT
Session # AT22
Rational User Conference 2002Rational User Conference 2002
©199
8, 1
999,
200
0, 2
001,
200
2 Ra
tiona
l Sof
twar
e - A
ll rig
hts
rese
rved
Intermediate and Advanced Tips and Tricks for Performance Testing with Rational Suite
TestStudio
Intermediate and Advanced Tips and Tricks for Performance Testing with Rational Suite
TestStudio
Scott SilverstoneJeff Robbins
Rational Software
Scott SilverstoneJeff Robbins
Rational Software
Your PresentersYour Presenters Scott Silverstone, Rational Marketing Engineer
Jeff Robbins, Rational Product Manager
Special thanks to presentation reviewers
Kent Siefkes
Dawn Haynes
Scott Silverstone, Rational Marketing Engineer
Jeff Robbins, Rational Product Manager
Special thanks to presentation reviewers
Kent Siefkes
Dawn Haynes
What Does A Performance Tester Do?What Does A Performance Tester Do?
Execute/Monitor Test
Evaluate Results
Manage Test Data
Create Test Suite
Plan Test
Design Test
Tester
(from Actors)
Create Test Scripts
Getting StartedGetting Started
Execute/Monitor Test
Evaluate Results
Manage Test Data
Create Test Suite
Plan Test
Design Test
Tester
(from Actors)
Create Test Scripts
Performance Test Planning & DesignPerformance Test Planning & Design Several other presentations cover Planning & Design
AT03 Basic Tips and Tricks for Performance Testing AT32 Modeling the Real World for Load Testing Web Sites RTL09 Introduction to Functional and Performance Testing
See www.rational.com/ruc for other presentations
Due to coverage in these presentations, this presentation will not cover planning & design
Several other presentations cover Planning & Design AT03 Basic Tips and Tricks for Performance Testing AT32 Modeling the Real World for Load Testing Web Sites RTL09 Introduction to Functional and Performance Testing
See www.rational.com/ruc for other presentations
Due to coverage in these presentations, this presentation will not cover planning & design
Digging InDigging In
Execute/Monitor Test
Evaluate Results
Manage Test Data
Create Test Suite
Plan Test
Design Test
Tester
(from Actors)
Create Test Scripts
Creating Test Scripts: Hint OutlineCreating Test Scripts: Hint Outline
Use of “non-traditional” performance test scripts
Use of INFO SERVER for monitoring
VU language trick to conserve hardware
Use of “non-traditional” performance test scripts
Use of INFO SERVER for monitoring
VU language trick to conserve hardware
Creating Test Scripts: Hint OutlineCreating Test Scripts: Hint Outline
Use of “non-traditional” performance test scripts
Use of INFO SERVER for monitoring
VU language trick to conserve hardware
Use of “non-traditional” performance test scripts
Use of INFO SERVER for monitoring
VU language trick to conserve hardware
Use Of “Non-Traditional” Test ScriptsUse Of “Non-Traditional” Test Scripts You are not limited to using VU language scripts
Other scripting languages are supported Java Command line (compiled executables,shell scripts, perl scripts, etc) Visual Basic
Presentations on Java & Visual Basic scripting: AT28 - Designing Scalable Performance Test Scripts for Java Environments AT30 - Custom Load Generation with Rational TestManager VM14 - Model Based Performance Testing of EJB and DCOM/COM+ Components
Let’s talk about Command Line scripts in more detail
You are not limited to using VU language scripts
Other scripting languages are supported Java Command line (compiled executables,shell scripts, perl scripts, etc) Visual Basic
Presentations on Java & Visual Basic scripting: AT28 - Designing Scalable Performance Test Scripts for Java Environments AT30 - Custom Load Generation with Rational TestManager VM14 - Model Based Performance Testing of EJB and DCOM/COM+ Components
Let’s talk about Command Line scripts in more detail
Command Line Test Scripts – Why Use? Command Line Test Scripts – Why Use? Command Line Test Script Type
Run a setup script at the very start of the test Run a cleanup script at the very end of the test Execute any command in system PATH Simply put, run anything that can be run from the command line
New Types based on Command Line Execution Adapter Use perl scripts as your load test scripts Execute shell scripts throughout the run Use binaries compiled from C, Ada or Smalltalk as load test scripts Launch preVue tests as part of a larger load test under TestManager
Creating a new type allows more control over execution conditions than simply using the command line test script type
Command Line Test Script Type Run a setup script at the very start of the test Run a cleanup script at the very end of the test Execute any command in system PATH Simply put, run anything that can be run from the command line
New Types based on Command Line Execution Adapter Use perl scripts as your load test scripts Execute shell scripts throughout the run Use binaries compiled from C, Ada or Smalltalk as load test scripts Launch preVue tests as part of a larger load test under TestManager
Creating a new type allows more control over execution conditions than simply using the command line test script type
Using The Command Line Test Script Type (3)Using The Command Line Test Script Type (3) Resulting Suite: Resulting Suite:
QuestionsQuestions What is the name of the project we are using?
What is the name of our suite?
What are the name of our user groups?
What is the name of the project we are using?
What is the name of our suite?
What are the name of our user groups?
Creating Test Scripts: Hint OutlineCreating Test Scripts: Hint Outline
Use of “non-traditional” performance test scripts
Use of INFO SERVER for monitoring
VU language trick to conserve hardware
Use of “non-traditional” performance test scripts
Use of INFO SERVER for monitoring
VU language trick to conserve hardware
Use Of INFO SERVER For Monitoring (1)Use Of INFO SERVER For Monitoring (1) As discussed in AT03 (Basic Tips and Tricks for Performance
Testing), monitoring of system resources on the local computer and agents is done automatically
Monitoring of computers other than the local computer and agents requires use of the INFO SERVER declaration
Robot automatically inserts the INFO SERVER declaration for the database server into SQL scripts
Robot does not automatically insert INFO SERVER into other types of scripts (Tuxedo, HTTP, Jolt, CORBA, DCOM)
As discussed in AT03 (Basic Tips and Tricks for Performance Testing), monitoring of system resources on the local computer and agents is done automatically
Monitoring of computers other than the local computer and agents requires use of the INFO SERVER declaration
Robot automatically inserts the INFO SERVER declaration for the database server into SQL scripts
Robot does not automatically insert INFO SERVER into other types of scripts (Tuxedo, HTTP, Jolt, CORBA, DCOM)
Use Of INFO SERVER For Monitoring (2)Use Of INFO SERVER For Monitoring (2) To add servers to monitor, add the INFO SERVER declaration to a
script. Which script, you ask? One approach is to have a set of re-usable scripts to initialize
monitoring for each computer of interest These scripts can be inserted in any suites you like
Alternatively, the INFO SERVER declarations can be added directly to the scripts where the server access occurs
To add servers to monitor, add the INFO SERVER declaration to a script. Which script, you ask?
One approach is to have a set of re-usable scripts to initialize monitoring for each computer of interest These scripts can be inserted in any suites you like
Alternatively, the INFO SERVER declarations can be added directly to the scripts where the server access occurs
Use Of INFO SERVER For Monitoring (3)Use Of INFO SERVER For Monitoring (3) During Suite Execution, activate the computer view: During Suite Execution, activate the computer view:
Right-click cells to add/remove stats from graph
Right-click cells to add/remove stats from graph
Creating Test Scripts: Hint OutlineCreating Test Scripts: Hint Outline
Use of “non-traditional” performance test scripts
Use of INFO SERVER for monitoring
VU language trick to conserve hardware
Use of “non-traditional” performance test scripts
Use of INFO SERVER for monitoring
VU language trick to conserve hardware
VU Language Trick To Conserve HardwareVU Language Trick To Conserve Hardware If you don’t need to match on the _response or log all
the data (or only need the first part of the _response ):
Set Max_nrecv_saved to a low number just big enough, such as 1 or 100
• SQL: limits # of rows saved• HTTP, socket: limits # of bytes saved
Applies to all receive commands, not just the “nrecv” flavor
Can be used selectively, such as around large SQL queries, or web pages with lots of images...
If you don’t need to match on the _response or log all the data (or only need the first part of the _response ):
Set Max_nrecv_saved to a low number just big enough, such as 1 or 100
• SQL: limits # of rows saved• HTTP, socket: limits # of bytes saved
Applies to all receive commands, not just the “nrecv” flavor
Can be used selectively, such as around large SQL queries, or web pages with lots of images...
Example: Max_nrecv_savedExample: Max_nrecv_saved Using Max_nrecv_saved to save only the first row of a
large SQL query: Using Max_nrecv_saved to save only the first row of a
large SQL query:
sqlexec ["exec big query"] "select * from books"; push Max_nrecv_saved = 1; sqlnrecv ["retrieve big query"] ALL_ROWS; pop Max_nrecv_saved;
sqlexec ["exec big query"] "select * from books"; push Max_nrecv_saved = 1; sqlnrecv ["retrieve big query"] ALL_ROWS; pop Max_nrecv_saved;
Digging Up Data…Digging Up Data…
Execute/Monitor Test
Evaluate Results
Manage Test Data
Create Test Suite
Plan Test
Design Test
Tester
(from Actors)
Create Test Scripts
Manage Test Data: Hint OutlineManage Test Data: Hint Outline
Streamlining transfer of datapools to agents
All about the Authentication Datapool What it is Why to modify it How to modify it
Streamlining transfer of datapools to agents
All about the Authentication Datapool What it is Why to modify it How to modify it
Manage Test Data: Hint OutlineManage Test Data: Hint Outline
Streamlining transfer of datapools to agents
All about the Authentication Datapool What it is Why to modify it How to modify it
Streamlining transfer of datapools to agents
All about the Authentication Datapool What it is Why to modify it How to modify it
Streamlining Transfer Of Datapools (1)Streamlining Transfer Of Datapools (1)
Datapools are automatically copied to Agents at the start of a test run to minimize network traffic during script execution
Huge datapools may cause a long startup delay at beginning of a run when they are copied to agents
Control with system env variable on MasterRT_MASTER_DISABLE_MANDATORY_DP_XFER = 1
Datapools are automatically copied to Agents at the start of a test run to minimize network traffic during script execution
Huge datapools may cause a long startup delay at beginning of a run when they are copied to agents
Control with system env variable on MasterRT_MASTER_DISABLE_MANDATORY_DP_XFER = 1
Streamlining Transfer Of Datapools (2)Streamlining Transfer Of Datapools (2) This turns off automatic copying of
datapools to agents Note: Rational TestManager & Rational
Robot must be restarted to read the new value of this variable
Caution!!! Updates to datapools won’t be used
unless the datapool file is refreshed The best way to refresh the datapool is
to set RT_MASTER_DISABLE_MANDATORY_DP_XFER to 0, restart Rational TestManager and Rational Robot, then run a test. The updated datapools will then be copied to the agents.
This turns off automatic copying of datapools to agents
Note: Rational TestManager & Rational Robot must be restarted to read the new value of this variable
Caution!!! Updates to datapools won’t be used
unless the datapool file is refreshed The best way to refresh the datapool is
to set RT_MASTER_DISABLE_MANDATORY_DP_XFER to 0, restart Rational TestManager and Rational Robot, then run a test. The updated datapools will then be copied to the agents.
Manage Test Data: Hint OutlineManage Test Data: Hint Outline
Streamlining transfer of datapools to agents
All about the Authentication Datapool What it is Why to modify it How to modify it
Streamlining transfer of datapools to agents
All about the Authentication Datapool What it is Why to modify it How to modify it
What Is The Authentication Datapool?What Is The Authentication Datapool? Answer:
A unique datapool that collects secure login information during Rational Robot script recordings
Rational Robot generates entries into this file as the connection is opened during secure login sessions
Contains information about username, password, connection type (ex. SQL) and connection state (Active or Passive)
For secure login purposes ONLY, not a general-use datapool file
Answer: A unique datapool that collects secure login information
during Rational Robot script recordings Rational Robot generates entries into this file as the
connection is opened during secure login sessions Contains information about username, password, connection
type (ex. SQL) and connection state (Active or Passive) For secure login purposes ONLY, not a
general-use datapool file
Why Modify The Authentication Datapool?Why Modify The Authentication Datapool? Why?
If the password is encrypted during recording• The client/server security doesn’t allow a virtual tester to log into the
database multiple times simultaneously• Common problem when recording scripts
against secure SQL & Oracle systems• Passwords have been changed
since the initial recording• Stores original input during recording of the
secure login so data can be referencedduring later test runs
Why? If the password is encrypted during recording
• The client/server security doesn’t allow a virtual tester to log into the database multiple times simultaneously
• Common problem when recording scripts against secure SQL & Oracle systems
• Passwords have been changed since the initial recording
• Stores original input during recording of thesecure login so data can be referencedduring later test runs
Task: Authenticating logon information
Example Situation: Passwords are encrypted during the initial recording of the test run – Rational Suite TestStudio can’t play back the results properly because all virtual users after the first “Mary Smith” are locked out
Task: Authenticating logon information
Example Situation: Passwords are encrypted during the initial recording of the test run – Rational Suite TestStudio can’t play back the results properly because all virtual users after the first “Mary Smith” are locked out
How To Modify The Authentication DatapoolHow To Modify The Authentication Datapool
How To Modify The Authentication DatapoolHow To Modify The Authentication Datapool What is inside the script
#include <VU.h>{sparky = sqlconnect("sparky", "scott", LOOKUP_PWD,
"sparky", "oracle7.3");{ INFO SERVER "sparky"="192.168.241.101"; } /*1*/
The VU script uses the “LOOKUP_PWD” constant instead of the actual password.At playback, the authentication datapool issearched.
What is inside the script
#include <VU.h>{sparky = sqlconnect("sparky", "scott", LOOKUP_PWD,
"sparky", "oracle7.3");{ INFO SERVER "sparky"="192.168.241.101"; } /*1*/
The VU script uses the “LOOKUP_PWD” constant instead of the actual password.At playback, the authentication datapool issearched.
How To Modify The Authentication DatapoolHow To Modify The Authentication Datapool
In Rational TestManager
1) Open the Datapool manager
2) Select RTAuthentication (the Authentication DataPool)
3) Edit values accordingly
How To Modify The Authentication DatapoolHow To Modify The Authentication Datapool Tips:
Use realistic data• Experiment with different values/character lengths• Emulate patterns from current usage
Don’t delete or rename the Authentication Datapool (RTAuthentication)
Don’t modify the values of the preset columns
Tips: Use realistic data
• Experiment with different values/character lengths• Emulate patterns from current usage
Don’t delete or rename the Authentication Datapool (RTAuthentication)
Don’t modify the values of the preset columns
Digging InDigging In
Execute/Monitor Test
Evaluate Results
Manage Test Data
Create Test Suite
Plan Test
Design Test
Tester
(from Actors)
Create Test Scripts
Hints In Creating Test Suites: OutlineHints In Creating Test Suites: Outline Use of Transactor Distributions Use of Transactor Distributions
Use Of Transactor DistributionsUse Of Transactor Distributions Basic transactor tips were covered in
AT03 Basic Tips and Tricks for Performance Testing
Quick Review: Transactors control the rate at which Rational TestManager starts scenarios (groups of scripts)
As such, transactors control the arrival rate of work to the system under test
Here, we’ll be discussing transactor distributions When to use each one What each one “looks like”
Basic transactor tips were covered in AT03 Basic Tips and Tricks for Performance Testing
Quick Review: Transactors control the rate at which Rational TestManager starts scenarios (groups of scripts)
As such, transactors control the arrival rate of work to the system under test
Here, we’ll be discussing transactor distributions When to use each one What each one “looks like”
Constant Distribution – When To UseConstant Distribution – When To Use Use the constant distribution when input to the system under
test occurs at an exact and unvarying rate
Customer situation: Insurance company with 20 automated document scanners At start of day, scanners are each initialized 1 second apart All forms to be scanned are of the same length Each scanner takes exactly 20 seconds to scan one of these forms Scanned data is added to a customer record in a database
We would like to emulate the 20 scanners in order to verify that the database can handle the workload
Use the constant distribution when input to the system under test occurs at an exact and unvarying rate
Customer situation: Insurance company with 20 automated document scanners At start of day, scanners are each initialized 1 second apart All forms to be scanned are of the same length Each scanner takes exactly 20 seconds to scan one of these forms Scanned data is added to a customer record in a database
We would like to emulate the 20 scanners in order to verify that the database can handle the workload
Constant Distribution – How To Do ItConstant Distribution – How To Do It
Contains SQL statements that add customer info to database
Contains SQL statements that add customer info to database
Simulatesstaggered initializationof scanners
Simulatesstaggered initializationof scanners
Constant Distribution – What It Looks LikeConstant Distribution – What It Looks Like If the database processes & verifies addition of each
customer record in exactly one second: If the database processes & verifies addition of each
customer record in exactly one second:
No overlaps, no gaps!
No overlaps, no gaps!
Uniform Distribution – When To UseUniform Distribution – When To Use Use the uniform distribution when input to the system under test
occurs at a known average rate, but the time between each input is not constant
Customer situation: Insurance company with 20 automated document scanners At start of day, scanners are each initialized one second apart Forms to be scanned are of varying length Each form requires a different amount of time to scan
• Average scan time is 20 seconds After a scanner scans a form, it submits a customer record for addition to
a database
We would like to emulate the 20 scanners in order to verify that the database can handle the workload
Use the uniform distribution when input to the system under test occurs at a known average rate, but the time between each input is not constant
Customer situation: Insurance company with 20 automated document scanners At start of day, scanners are each initialized one second apart Forms to be scanned are of varying length Each form requires a different amount of time to scan
• Average scan time is 20 seconds After a scanner scans a form, it submits a customer record for addition to
a database
We would like to emulate the 20 scanners in order to verify that the database can handle the workload
Uniform Distribution – How To Do ItUniform Distribution – How To Do It
Contains SQL statements that add customer record to database
Contains SQL statements that add customer record to database
Simulatesstaggered initializationof scanners
Simulatesstaggered initializationof scanners
Uniform Distribution – What It Looks LikeUniform Distribution – What It Looks Like Assuming the database processes & verifies addition of
each customer record in exactly one second: Assuming the database processes & verifies addition of
each customer record in exactly one second:
Some overlaps, a few gaps!
Some overlaps, a few gaps!
Negative Exponential Distribution – When To UseNegative Exponential Distribution – When To Use Use the negative exponential distribution when input to the system
under test occurs at a variable rate
Customer situation: Insurance company with 20 customer service representatives (CSRs) CSRs enter customer information into application running on PCs This application, just like the automated scanners, adds customer information
as record in a database Incoming customer calls are assigned to CSRs in the order in which the queue
for that CSR was initialized Because of timing of incoming calls, as well as length of each call, the time
between the submission of customer data to the database varies
We would like to emulate the 20 customer service representatives & their desktop PCs in order to verify that the database can handle the workload
Use the negative exponential distribution when input to the system under test occurs at a variable rate
Customer situation: Insurance company with 20 customer service representatives (CSRs) CSRs enter customer information into application running on PCs This application, just like the automated scanners, adds customer information
as record in a database Incoming customer calls are assigned to CSRs in the order in which the queue
for that CSR was initialized Because of timing of incoming calls, as well as length of each call, the time
between the submission of customer data to the database varies
We would like to emulate the 20 customer service representatives & their desktop PCs in order to verify that the database can handle the workload
Negative Exponential Distribution – How To Do ItNegative Exponential Distribution – How To Do It
Contains SQL statements that add customer record to database
Contains SQL statements that add customer record to database
Simulatesstaggered Initialization of work queues
Simulatesstaggered Initialization of work queues
Negative Exp Distribution – What It Looks LikeNegative Exp Distribution – What It Looks Like Assuming the database processes & verifies addition of
each customer record in exactly one second: Assuming the database processes & verifies addition of
each customer record in exactly one second:
Many overlaps, more gaps!
Many overlaps, more gaps!
QuestionsQuestions What if the database response time increases with number of concurrent updates? In the previous response examples, the database response time for the addition of a customer record was always one second, no matter how many requests were being served concurrently
However, server response times often increase as the number of updates being concurrently handled increases, especially at higher user loads
If database response time = 1 second * number of updates currently in progress
What response times will users experience if they arrive in a constant distribution? negative exponential distribution?
What if the database response time increases with number of concurrent updates? In the previous response examples, the database response time for the addition of a customer record was always one second, no matter how many requests were being served concurrently
However, server response times often increase as the number of updates being concurrently handled increases, especially at higher user loads
If database response time = 1 second * number of updates currently in progress
What response times will users experience if they arrive in a constant distribution? negative exponential distribution?
Answer: Constant DistributionAnswer: Constant Distribution No difference from before No difference from before
Answer: Negative Exponential DistributionAnswer: Negative Exponential Distribution
A drastic difference!!! A drastic difference!!!
Digging InDigging In
Execute/Monitor Test
Evaluate Results
Manage Test Data
Create Test Suite
Plan Test
Design Test
Tester
(from Actors)
Create Test Scripts
Introduction To cwDBMON Introduction To cwDBMON cwDBMON is a user-
contributed tool
Monitors database(s) during test execution
Provides integration into Rational TestManager reports for results evaluation
cwDBMON is a user-contributed tool
Monitors database(s) during test execution
Provides integration into Rational TestManager reports for results evaluation
Using cwDBMON (2)Using cwDBMON (2) Test-time monitoring of the database Test-time monitoring of the database
Using cwDBMON (3)Using cwDBMON (3) Integrating DB parameters into Rational
TestManager Integrating DB parameters into Rational
TestManager
Using Of cwDBMON (4)Using Of cwDBMON (4) Response VS Time graph with Oracle <Latch Efficiency> parameter
Response VS Time graph with Oracle <Latch Efficiency> parameter
Oracle Latch Efficiency
Oracle Latch Efficiency
Response timesResponse times
Downloading cwDBMON Downloading cwDBMON Remember, this is a user-contributed tool
Not part of any Rational product
Not supported by Rational Technical Support
But the user community provides help
Download instructions posted atftp://exchange.rational.com/exchange/outgoing/ruc_2002_at_22/cwdbmon.txt
Remember, this is a user-contributed tool
Not part of any Rational product
Not supported by Rational Technical Support
But the user community provides help
Download instructions posted atftp://exchange.rational.com/exchange/outgoing/ruc_2002_at_22/cwdbmon.txt
A Quick Word On EvaluationsA Quick Word On Evaluations In the Comments section:
Please indicate your level of performance testing experience• Beginning, Intermediate, Advanced
The 3 tips you’ve found most useful Format for next year
• Same as this year (separate sessions for Beginning & Intermediate/Advanced)
OR• One unified session for Beginning , Intermediate and AdvancedOR• Other suggestions
– Things you would like to see– Things you didn’t need to see
Any other comments are greatly appreciated
In the Comments section: Please indicate your level of performance testing experience
• Beginning, Intermediate, Advanced The 3 tips you’ve found most useful Format for next year
• Same as this year (separate sessions for Beginning & Intermediate/Advanced)
OR• One unified session for Beginning , Intermediate and AdvancedOR• Other suggestions
– Things you would like to see– Things you didn’t need to see
Any other comments are greatly appreciated
Questions?Questions?
Rational User Conference 2002Rational User Conference 2002
©199
8, 1
999,
200
0, 2
001,
200
2 Ra
tiona
l Sof
twar
e - A
ll rig
hts
rese
rved
Scott Silverstonessilverstone@rational.com
Jeff Robbinsjr@rational.com
Scott Silverstonessilverstone@rational.com
Jeff Robbinsjr@rational.com
Thank You!Thank You!
This presentation will be posted by tomorrow at: http://www.rational.com/ruc
This presentation will be posted by tomorrow at: http://www.rational.com/ruc
Rational User Conference 2002Rational User Conference 2002
©199
8, 1
999,
200
0, 2
001,
200
2 Ra
tiona
l Sof
twar
e - A
ll rig
hts
rese
rved
top related