a national resource working in the public interest © the mitre corporation. all rights...
TRANSCRIPT
A National Resource Working in the Public Interest © The MITRE Corporation. All rights reserved.04/19/23
MITRE Performance Testing: Load Testing With Usage Analysis
The MITRE CI&T Performance Test Team
February 2009
A National Resource Working in the Public Interest © The MITRE Corporation. All rights reserved.
Page 2
04/19/23
Acknowledgements
This briefing was prepared by adopting material from the Fundamentals of LoadRunner 8.1 training manual
The final usage analysis method and model described in this briefing was pioneered by the MITRE CI&T Performance Test Team.
Initial methods and models were originated by Henry Amistadi and evolved with the input of Chris Alexander, Betty Kelley and Aimee Bechtle.
A National Resource Working in the Public Interest © The MITRE Corporation. All rights reserved.
Page 3
04/19/23
Purpose & Goal
Provide background on our team and load testing at MITRE
Understand how the Project Lead can support load testing throughout the performance testing process
Introduce our Usage Analysis methods so the Project Lead
1. Understands how we target a realistic load in load testing
2. Has confidence that the right amount of testing is being performed on a project
Our goal is always to provide high quality, timely performance tests that meet the needs of the customer and the performance test team
A National Resource Working in the Public Interest © The MITRE Corporation. All rights reserved.
Our Team
Comprised of 3 Performance Test Engineers and 1 Team Lead
Instantiated in 2001 as the Automated Test Team
Invested in Mercury (now HP):
– LoadRunner for Load Testing
– QuickTest Pro for Functional Testing
– Quality Center/TestDirector for requirements and test case management
In the beginning customers suspicious of the reality and accuracy of our testing
In 2005 stopped performing automated functional testing to focus on our core competency, performance testing
Have matured from single application and environment testing to multi-application and multi-environment testing
Preparing for a large-scale, enterprise load test for 2009-2010
Now, high level of confidence in our service
Page 4
04/19/23
A National Resource Working in the Public Interest © The MITRE Corporation. All rights reserved.
Page 5
04/19/23
Background: What is Load?
What is Load? Traffic!!!!!
– Transactions distributed over time, expressed as rates
Transaction Per hour (TPH)
Transactions Per minute (TPM)
Transactions Per second (TPS)
Why Load Test? To assess how well your application is directing traffic
– Find the problems before your customers do
– Prevent poor response times, instability and unreliability (Road Rage!)
– The longer you wait to test the more costly the problem may become
A National Resource Working in the Public Interest © The MITRE Corporation. All rights reserved.
Page 6
04/19/23
Background: When to Use Performance Testing When should I consider load testing? How can I use you throughout the project?
Contact performance test team during the project planning process and consider their incorporation into the plan
iSTEP Phase Examples/Activities
Requirements COTS Evaluation, Usage Analysis
Design Prototyping, architecture and server evaluation
Implementation Early assessment of application response times, Support capacity planning
Test Determining if the system is reliable enough for production
Post Rollout Isolating performance bottlenecks and degradation in performance after production. Regression testing.
Load testing can be used throughout the product’s life
A National Resource Working in the Public Interest © The MITRE Corporation. All rights reserved.
Page 7
04/19/23
Background: The Process
Plan LoadTest
AnalyzeResults
ExecuteTests
Create Scenarios
Create Scripts
Tweak System or Test
Based on Analysis
§ Desc Goals & Objectives
§ Select Load Test Types
§ Analyze Application:
§ Document Plan
§ Define steps§ Create
Scripts
§ Create scenarios based on goals and targets
§ Execute scenarios:
PeakStressOther
1 2 3 4 5
TestTransaction
Rate
§ Analyze Data and pinpoint bottlenecks
§ Compare Test Transaction Rate to Target
Target Transaction RatesEnvironmentUsage
§ Supply Goals, Obj, Schedule
§ Initiate Data Collection
§ Select targets§ Review PlanP
roje
ct L
ead
/D
evel
op
er
Fu
nct
’l E
xper
t
§ Develop use cases as input into scripts
§ Provide Data
§ Supply transaction logs
Sys
an
d D
BA
dm
ins
Per
form
ance
Exp
ert
§ Provide Steps & Sample Data
§ Review scripts
§ Provide Steps & Sample Data
§ Review scripts
§ Prepare Test Environment
§ Monitor resources
§ Review results
§ Review results
§ Review Scenario design
§ Prioritize & schedule tests
§ Change settings (if applicable)
§ Change application
A National Resource Working in the Public Interest © The MITRE Corporation. All rights reserved.
Page 8
04/19/23
Define Goals & Objectives
Begin working with performance test team upon completion of planning process
Start by discussing high-level goals & objectives.
And what are these objectives?
Objective For Example…
Measure Application Response Time
How long does it take to complete a business transaction?
Support Configuration Tuning Which configuration provides the best performance level?
Regression Does a new version of the software adversely affect response time?
Assess Reliability How stable is the system under a heavy work load?
Capacity Planning Can the application support a growth in usage?
Identify Bottlenecks What is the cause of degradation in performance?
Evaluate Products What is the best product (COTS, hardware, etc.)?
Plan LoadTest
1
A National Resource Working in the Public Interest © The MITRE Corporation. All rights reserved.
Page 9
04/19/23
Select Load Test Types
What are the types of load tests?
Type Description
Typical or Peak Load Determine whether system handles real world loads
Stress Load Find the system’s breaking point. Test at 1.5, 2, 2.5, 3
times peak.
Exploratory or Goal Meet predetermined criteria or “What ifs?”
Architecture Verify architectural design, like failover, load
balancing
Extreme/Volume Load Check stability of system under extended periods of
high load
Plan LoadTest
1
A National Resource Working in the Public Interest © The MITRE Corporation. All rights reserved.
Page 10
04/19/23
Analyze System - Environment
Need to understand the logical and physical test environment to make the proper tool, protocol selection and to know
our constraints
What is the historical CPU utilization on these machines?
What other apps are on these machines?
Plan LoadTest
1
A National Resource Working in the Public Interest © The MITRE Corporation. All rights reserved.
Page 11
04/19/23
Analyze System – Usage Analysis
Usage analysis is appropriate when:
– Plan is to run a Typical, Peak or Stress test to meet the agreed upon objectives
– There’s an existing system to collect information from
Usage Analysis is the process of calculating load targets from log file data for the relevant URLS
– AKA Workload Analysis or Log File Analysis
– We employ a statistical evaluation of data collected
Results in a Transaction’s:
– Target Typical & Peak TPH
– Target Typical & Peak TPM
– Target Typical & Peak TPS
Plan LoadTest
1
A National Resource Working in the Public Interest © The MITRE Corporation. All rights reserved.
Page 12
04/19/23
Analyze System – Usage Analysis Plan Load
Test
1
Identify Business
Trans.Convert TPS
Select Targets
Analyze Fine Grain
Analyze Coarse Grain
§ Select key business processes to be analyzed and tested
§ Select the days to be analyzed
§ Collect log files for these days
§ Parse and grep log file data
§ Aggregate data by hr, min and sec
§ Create cumulative freq. distrib.
§ Review cumulative freq. distrib. With PL
§ Select target typical and peak TPHs and TPMs
A B C D E
§ Convert TPS from TPM using Total Time Distrib. or
§ Convert TPS from TPM using Active Time Distrib.
A National Resource Working in the Public Interest © The MITRE Corporation. All rights reserved.
Page 13
04/19/23
Usage Analysis – Business Transactions
“A business transaction is a set of user steps performed within an application to accomplish a business goal”
For example:
– Logon to Home Page
– Search
– Update profile
Focus on the business processes that are
– Most commonly used
– Business Critical
– Resource Intensive
We need the URLs that complete these business processes. These are the target URLs
Also, what are your expected target response times for these processes under typical load? Peak load?
Identify Business
Trans.
A
A National Resource Working in the Public Interest © The MITRE Corporation. All rights reserved.
Page 14
04/19/23
Usage Analysis – Coarse Grain Purpose is to identify Typical Peak Usage
Patterns
Use WebTrends
We identify the Typical Peak by answering questions such as:
– Is this application used cyclical or quarterly?
– Are some months or days of the weeks higher use than others?
– Is this application heavily used on a daily basis?
– Is this application heavily used throughout the day or only at specific hours?
– What are the highest use days and hours?
Analyze Coarse Grain
B
A National Resource Working in the Public Interest © The MITRE Corporation. All rights reserved.
Page 15
04/19/23
Usage Analysis – Coarse Grain Example
Corporate Portal:
– Looked at weekdays from Oct – Dec 2006
– Compared daily and hourly page views in WebTrends for portal/dt and amserver/UI/Login
– Ranked days in terms of daily totals
– Excluded two busiest days because of abnormal patterns
– Selected next 8 busiest days for further analysis
MyMII (Sun Portal) Daily Requests for portal/dtWEBTRENDS DATA
0
20000
40000
60000
80000
100000
120000
140000
160000
180000
200000
18-Sep-06 25-Sep-06 2-Oct-06 9-Oct-06 16-Oct-06 23-Oct-06 30-Oct-06 6-Nov-06 13-Nov-06 20-Nov-06 27-Nov-06 4-Dec-06 11-Dec-06 18-Dec-06
Requests Per Day
Analyze Coarse Grain
B
A National Resource Working in the Public Interest © The MITRE Corporation. All rights reserved.
Page 16
04/19/23
Usage Analysis – Fine Grain
Once the Typical Peak timeframe is determined, log files from that period are collected for more detailed analysis.
Relevant fields are isolated for inclusion, such as:
– Requesting URL (as agreed upon with the project team), Response Code, Date and Time request completed, Referring URL, User information
Analyze Fine Grain
C
A National Resource Working in the Public Interest © The MITRE Corporation. All rights reserved.
Page 17
04/19/23
The data is aggregated by time frame and frequency analysis is performed. For an example please see: Usage Frequency Analysis Example
The transaction rates are placed in a cumulative frequency distribution
Work Hours Cumulative FrequencyHTTP 200 (/portal/dt)
TPH TPM TPS
Mean (Ave) 4499 75 2
Median (50) 5103 81 2
60.0 5214 86 2
70.0 5381 91 2
80.0 5532 96 3
90.0 5778 104 3
95.0 5896 110 4
97.5 5988 118 5
99.0 6075 131 5
100.0 6075 309 26
Usage Analysis – Fine GrainAnalyze Fine Grain
C
A National Resource Working in the Public Interest © The MITRE Corporation. All rights reserved.
Page 18
04/19/23
Usage Analysis – Select Targets
Select your Typical and Peak targets
We recommend using the TPH and TPM rates at the Median, 50th Percentile, as the Typical Transaction Rates
We recommend using the TPH and TPM rates at the 99th percentile as Peak Transaction Rates
The TPS in this analysis is a reference point.
This index is a satisfaction index:
– What percentage of the time are you willing to have users be satisfied/unsatisfied?
– If you design a system to handle 131 TPMs your users would be satisfied with response time performance 99% of the time
Work Hours Cumulative FrequencyHTTP 200 (/portal/dt)
TPH TPM TPS
Mean (Ave) 4499 75 2
Median (50) 5103 81 2
60.0 5214 86 2
70.0 5381 91 2
80.0 5532 96 3
90.0 5778 104 3
95.0 5896 110 4
97.5 5988 118 5
99.0 6075 131 5
100.0 6075 309 26
Select Targets
D
A National Resource Working in the Public Interest © The MITRE Corporation. All rights reserved.
Page 19
04/19/23
Usage Analysis – Convert TPS
We can choose from two methods
– Method 1: Total Time Distribution (How?)
Assumes transactions are evenly distributed and every minute or second is active
Represents the lowest transaction rate in terms of load distribution
– Method 2: Active Time Distribution (How?)
Accounts for the wave patterns of transaction activity
Assumes that not every minute and second is active
The active and inactive times are calculated
Method 1 Total Time Distribution
0
0.5
1
1.5
2
2.5
9:00 9:01 9:02 9:03 9:04 9:05
Time Sample
TP
S actual trans
avg trans
Method 2 Active Time Distribution
0
0.5
1
1.5
2
2.5
9:00 9:01 9:02 9:03 9:04 9:05
Time Sample
TP
S actual trans
avg trans
Purpose is to know the target TPS, and at what time intervals, the transactions are going to occur in the test
Convert TPS
E
A National Resource Working in the Public Interest © The MITRE Corporation. All rights reserved.
Page 20
04/19/23
Pulling It All Together
At the conclusion of the usage analysis we meet with the Project Lead and review the plan.
Ideal if conducted during development, prior to integration testing
Review Business Processes and Target Transactions Rates
Target Transactions Rates Example: “The system shall sustain a load home page transaction time of 3 seconds or less during typical hours at a rate of 2.02 transactions per second, and 3-5 seconds maximum during peak hours at a rate of 3.3 transactions per second.”
Transaction Typical Targets Peak Targets
TPH TPM TPS PRT* TPH TPM TPS PRT*
Portal Home Page 5103 81 2.02 <3s 6075 131 3.3 3-5s
* PRT = Preferred Response Time
Plan LoadTest
1
A National Resource Working in the Public Interest © The MITRE Corporation. All rights reserved.
Page 21
04/19/23
Create Scripts
Scripts contain the recorded steps, test data, and think times that define the transactions
Functional test cases, use cases, design documents and training manuals are a good source for this information
A sample of the detail we need:
Step Data
Click link for “My Insurance Plan Summary”
Logon Username
Password
Click on the “Life” link under Type of Benefit
Click on the Edit button under Covered Beneficiaries Section
Enter New Primary allocation box for first beneficiary 100
Click “Save”
Click “Ok”
A real example: Detailed Steps
Create Scripts
2
A National Resource Working in the Public Interest © The MITRE Corporation. All rights reserved.
Page 22
04/19/23
Create Scenarios
Scenarios define the overall load test execution. The instructions.
Elements of a scenario are:
• Scripts
• Vusers: Emulate the steps of real users. Steps are recorded into a script.
• Ramp Up Rate: The rate at which vusers are suddenly or gradually introduced into the test
• Think Times: Emulates how long the user pauses when:
• Executing a transaction or business process. I.e. pauses between clicks or keystrokes
• Between transactions (iterations), between starting the script over
• Test Duration: The length the test will run. 1 hour, 4 hours, 8 hours. Varies by test type.
Create Scenarios
3
A National Resource Working in the Public Interest © The MITRE Corporation. All rights reserved.
Page 23
04/19/23
Favorites
Home Page(Default Config)
Project Services
RSS
HomePage (Configured)
Center Portlet
Typical Day 1x(Home Default Config)
ExploratoryEdits
Typical Day 3x(Home Config)
Typical Day 1x95% Home (Default)
+ 5% Edits
Typical Day 1x(Home Config)
ExploratoryProject Services
Script
Test Run2007-03-07-D
Scenarios Test Runs
Test Run2007-03-04-A
Test Run2007-03-09-F
Test Run2007-03-09-H
Test Run2007-03-09-I
Peak Second Bursts +1x(Home Default Config)
Test Run2007-03-12-N
Test Run2007-03-12-L
Home Page(Default Config)
Bursts
Pulling it All TogetherExecuteTests
4
A National Resource Working in the Public Interest © The MITRE Corporation. All rights reserved.
Page 24
04/19/23
Prepare Test Environment
– Account Setup
– Data Setup
Plan for Multiple test runs
– Debug
– Baseline
– Full execution
Tests will be run in priority order as we agreed upon
Execute tests as a team!!!
– Systems and Database administrators should monitor the performance of each tier of the application
– Ideally a network administrator could monitor the network
Execute TestsExecuteTests
4
A National Resource Working in the Public Interest © The MITRE Corporation. All rights reserved.
Page 25
04/19/23
Analyze Results
At the end of the test run(s) an analysis file is produced that reports:
• Running Vusers
• Duration
- Response Times By Transaction:
o Average & Maximum
o Percentiles
• Test Transaction Rates
- Test Total Transactions (Passed/Failed)
- Test Transactions Per Second
Example analysis file: Sample LoadRunner Analysis File
The Target Transaction Rates from the Usage Analysis are compared to the Test Transaction Rates. If they don’t reconcile then…
AnalyzeResults
5
A National Resource Working in the Public Interest © The MITRE Corporation. All rights reserved.
Page 26
04/19/23
Tweak The Load Test or The System
The test will need to be updated and/or rerun if:
1. The test goals have not been met then change the script or scenario
– Test Transaction Rates are not similar to Target Transaction Rates
For an hourly test the Total Transactions should be close to the Target TPH
The TPM and TPS rates should meet or exceed the TPM or TPS from the usage analysis
Increase vusers and decrease think time to increase load
2. Performance has not met expectations (i.e. poor response time or CPU utilization)
– Change Code– Change System Configuration– Change Architecture
This is an iterative process which takes time. This needs to be planned for.
Tweak System or Test
Based on Analysis
A National Resource Working in the Public Interest © The MITRE Corporation. All rights reserved.
Page 27
04/19/23
Finally…
Roll out and repeat!!– When the load test and performance goals are met you are good
to go
– Contact performance test team when application changes are made that affect performance:
Significant GUI or Code changes
Changes in the architecture or environment
Performance degradation in production
Adding more users
A National Resource Working in the Public Interest © The MITRE Corporation. All rights reserved.
Page 28
04/19/23
In Conclusion…
Partner with the peformance test team:
– Engage team from the beginning – during planning
– Have goals and objectives about what you want your performance test to accomplish
– Provide:
Architecture diagrams
Transactions and steps to be scripted
Test data
Webtrends and log file data
Server permissions and DBA/Sysadmin support
– Select Targets
– Review the test plan, prioritize and schedule the tests
– Plan enough time for an iterative testing process
– Keep team informed of schedule or resource changes
A National Resource Working in the Public Interest © The MITRE Corporation. All rights reserved.
Page 29
04/19/23
Background
Detailed Calculation Examples
A National Resource Working in the Public Interest © The MITRE Corporation. All rights reserved.
Page 30
04/19/23
Method 1: Even Distribution - Example
Evenly divide the TPH and TPM transaction rate by 60
TPM = TPH/60
TPS = TPM/3600
For example:
TPM = 6075/60
–TPS = 131/60
101.3
2.2 (We would use this one)
% TPH TPM TPS
A National Resource Working in the Public Interest © The MITRE Corporation. All rights reserved.
Page 31
04/19/23
Method 2: Active Time Distribution
Calculate Active Time from log as the percentage of time requests are written to the log
We calculate the Active Time per hour of day in terms of:
– Active Minutes per Hour
– Active Seconds per Minute
For example:
– At 9 o’clock every minute in the hour had at least 1 transaction, but every second did not.
– Only 46.7 seconds of the minute were active
Active Time
hour mins / hour secs / min
6 60.0 19.0
7 60.0 34.1
8 60.0 45.4
9 60.0 46.7
10 60.0 46.0
11 60.0 44.3
12 60.0 42.5
13 60.0 44.5
14 60.0 45.3
15 60.0 44.7
16 59.3 40.2
17 60.0 28.8
Average 59.9 40.1
A National Resource Working in the Public Interest © The MITRE Corporation. All rights reserved.
Page 32
04/19/23
Method 2: Active Time Distribution - Example
Evenly divide the TPH transaction rate by Average Active Time
– TPM = TPH/Active Minutes in a Hour
– TPS = TPM/Active Seconds in a Minute For example:
TPM = 6075/59.9
TPS = 131/40.1
101.4
3.3
4b
% TPH TPM TPS
A National Resource Working in the Public Interest © The MITRE Corporation. All rights reserved.
Page 33
04/19/23
Add Inactive Time--beginning point for determining think times or pacing
– Percentage of time when no requests are completed or written to the logs or the time between completed
requests
– It includes the processing time of the second request
We calculate the Inactive Time per hour of the day in terms of:
– Inactive Minutes per Hour
– Inactive Seconds per Minute
Inactive Time needs to be sprinkled evenly between the Active Times to avoid creating a
compressed hour
10-second Snapshot
120-second Snapshot
Method 2: Active Time Distribution – Example (con’t)