performance engineering case study v1.0
TRANSCRIPT
![Page 2: Performance Engineering Case Study V1.0](https://reader033.vdocument.in/reader033/viewer/2022060120/5592c01e1a28ab0d418b4774/html5/thumbnails/2.jpg)
Partners Testing ProductsPre-Sales, Sales, & Post Sales Support
2
Requirement Management Solution
Optimal Trace
Raven Flow
Functional Testing Tools
TestPartner
QA Director
Silk Test
Performance Testing Tools
QALoad
Web Load
Silk Performer
Development and Debugging Tools
DevPartner Studio
DevPartner Java
Bounds checker
Security checker
IT Service Management Products
Vantage
Vantage for Network Performance
Vantage for Java/.NET Performance
Vantage for Server Performance
![Page 3: Performance Engineering Case Study V1.0](https://reader033.vdocument.in/reader033/viewer/2022060120/5592c01e1a28ab0d418b4774/html5/thumbnails/3.jpg)
IonIdea -Testing Services Offerings
3
Ion - Virtual Testing Team
Functional Testing Automation Services
Performance Testing Services
Software Testing Automation Consulting
Scripted Functional Testing Services
Software Testing outsourced
Training – Test Automation
![Page 4: Performance Engineering Case Study V1.0](https://reader033.vdocument.in/reader033/viewer/2022060120/5592c01e1a28ab0d418b4774/html5/thumbnails/4.jpg)
Delivering High Performing ApplicationsWhat’s Needed: Load Testing with Infrastructure Monitoring
4
Virtual Users
Web Servers Application Servers
DBMS
Mainframe
Combining load testing with infrastructure monitoring:
• Provides realistic emulation of transaction and traffic usage patterns to:
• Ensure servers will scale to meet current and future
needs
• Correlate important performance relationships
between all components of the application
infrastructure
![Page 5: Performance Engineering Case Study V1.0](https://reader033.vdocument.in/reader033/viewer/2022060120/5592c01e1a28ab0d418b4774/html5/thumbnails/5.jpg)
Delivering High Performing ApplicationsWhat’s Needed: End-to-End Analysis of All Application Components
5
Virtual Users
Web Servers Application ServersDBMS
Mainframe
JSP-ASP
Method
Method
Method
Method
Method
URL JSP-ASP
Method
Method
Method
Web Service
Web Service
JDBC-ODBC
MQ
End-to-end performance analysis:
• Provides the depth of analysis necessary to pinpoint performance
problems
![Page 7: Performance Engineering Case Study V1.0](https://reader033.vdocument.in/reader033/viewer/2022060120/5592c01e1a28ab0d418b4774/html5/thumbnails/7.jpg)
7
Why Load Testing is Not Enough
NOT ENOUGH INFORMATION
• Only delivers general response time, throughput or server metrics
• Does not identify where bottlenecks are, across environment or inside application
• Doesn’t get to the root cause of the problem
• Leads to finger-pointing
NOT TIMELY
• You have to wait until after load testing to understand whether you have a problem
RESULT…
• Missed delivery dates
• Poor-quality applications
• High-end resources involved in resolving problems and waiting until the end of testing
• Costly/unnecessary infrastructure changes to fix problems
![Page 8: Performance Engineering Case Study V1.0](https://reader033.vdocument.in/reader033/viewer/2022060120/5592c01e1a28ab0d418b4774/html5/thumbnails/8.jpg)
8
PREDICTION:
• Predict performance under varying conditions
• Identify impact of network on application from multiple locations
• Pinpoint bottlenecks across application tiers
• Fix code prior to conducting load testing
TROUBLESHOOTING:
• Deeper analysis during load test
• Pinpoint Java and .NET performance problems
• Perform fewer application retests
What Sets Performance Triage apart?
![Page 9: Performance Engineering Case Study V1.0](https://reader033.vdocument.in/reader033/viewer/2022060120/5592c01e1a28ab0d418b4774/html5/thumbnails/9.jpg)
9
Load Testing only…
PT Prediction & Troubleshooting…
Performance Triage (PT)
![Page 10: Performance Engineering Case Study V1.0](https://reader033.vdocument.in/reader033/viewer/2022060120/5592c01e1a28ab0d418b4774/html5/thumbnails/10.jpg)
10
Re-active Leverage profiling to deal with performance issues before
load testing
Helps to make best use of limited testing timeframes
Performance Triage – Two mixes
Pro-active Well structured approach to performance testing to prevent
the ‘hair on fire’ scenarios
![Page 12: Performance Engineering Case Study V1.0](https://reader033.vdocument.in/reader033/viewer/2022060120/5592c01e1a28ab0d418b4774/html5/thumbnails/12.jpg)
12
Setting the scene• Online banking arm of large corporate finance house
• Urgent requirement to validate existing infrastructure capacity and to investigate capacity to handle further growth
• Limited time to execute
PT in Action - Case StudyIntroduction
![Page 13: Performance Engineering Case Study V1.0](https://reader033.vdocument.in/reader033/viewer/2022060120/5592c01e1a28ab0d418b4774/html5/thumbnails/13.jpg)
13
PT in Action - Case StudyPerformance Goals
• Response Time SLA• Login / Signon less than 10 seconds
• All other page response times less than 8 seconds
• All transaction response times less than 20 seconds
• Based on broadband bandwidth
• Concurrency SLA• Support 600 concurrent end users
• Server Utilization SLA• Server utilization < 50% measured as CPU, Memory and Disk I/O
![Page 14: Performance Engineering Case Study V1.0](https://reader033.vdocument.in/reader033/viewer/2022060120/5592c01e1a28ab0d418b4774/html5/thumbnails/14.jpg)
14
PT in Action – Case StudyLoad testing alone won’t identify hidden problems
Bank Data Centre
Web Servers
App Servers
DB Servers
External Users
Internal Users
WAN Sensitivity
Bad SQL
Slow Methods
Insufficient Capacity
Contention Issues
![Page 15: Performance Engineering Case Study V1.0](https://reader033.vdocument.in/reader033/viewer/2022060120/5592c01e1a28ab0d418b4774/html5/thumbnails/15.jpg)
15
• Enrollment
• Signon
• View account summary
• View account details
• Intra FI transfer
• View check images
PT in Action - Case Study
Identify and profile key transactions
Profile
WAN Sensitivity Bad SQL
![Page 16: Performance Engineering Case Study V1.0](https://reader033.vdocument.in/reader033/viewer/2022060120/5592c01e1a28ab0d418b4774/html5/thumbnails/16.jpg)
16
PT in Action – Case Study
Profile – Predicting WAN sensitivity
Increase in response time of 11 seconds when connecting over T1 link with 50ms latency
![Page 17: Performance Engineering Case Study V1.0](https://reader033.vdocument.in/reader033/viewer/2022060120/5592c01e1a28ab0d418b4774/html5/thumbnails/17.jpg)
17
PT in Action – Case Study
Profile – Bad SQL performance
SQL call taking in excess of 13 seconds to complete
![Page 18: Performance Engineering Case Study V1.0](https://reader033.vdocument.in/reader033/viewer/2022060120/5592c01e1a28ab0d418b4774/html5/thumbnails/18.jpg)
18
PT in Action – Case StudyProfiling identifies hidden problems
Bank Data Centre
Web Servers
App Servers
DB Servers
External Users
Internal Users
WAN Sensitivity
Bad SQL
Slow Methods
Insufficient Capacity
Contention Issues
![Page 19: Performance Engineering Case Study V1.0](https://reader033.vdocument.in/reader033/viewer/2022060120/5592c01e1a28ab0d418b4774/html5/thumbnails/19.jpg)
19
PT in Action – Case Study
Load Testing
Load Test
Contention Issues
Insufficient Capacity
• Performance goals• Login / Signon less than 10 seconds
• All other page response times less than 8 seconds
• All transaction response times less than 20 seconds
• Support 600 concurrent end users
• Server utilization < 50% measured as CPU, Memory and Disk I/O
Slow Methods
![Page 20: Performance Engineering Case Study V1.0](https://reader033.vdocument.in/reader033/viewer/2022060120/5592c01e1a28ab0d418b4774/html5/thumbnails/20.jpg)
20
PT in Action – Case StudyLoad Testing – Transaction performance
Concurrent users
Elapsed timeTransaction response time
Concurrent SLA: 600 users
Response time SLA: < 20 seconds
Transaction performance exceeds response time SLA
FAIL!! Response Time
![Page 21: Performance Engineering Case Study V1.0](https://reader033.vdocument.in/reader033/viewer/2022060120/5592c01e1a28ab0d418b4774/html5/thumbnails/21.jpg)
21
PT in Action – Case StudyLoad Testing – Server performance
Concurrent users
Elapsed time
Web server CPU %
Concurrent SLA: 600 users
Server CPU SLA: < 50 %
Web server CPU utilization breaches SLA
FAIL!! Insufficient Capacity
![Page 22: Performance Engineering Case Study V1.0](https://reader033.vdocument.in/reader033/viewer/2022060120/5592c01e1a28ab0d418b4774/html5/thumbnails/22.jpg)
22
PT in Action – Case StudyLoad Testing – Application server
Analysis inside the JVM and CLR
Slow performing method impacting response time and web server CPU
FAIL!! Slow Method
![Page 23: Performance Engineering Case Study V1.0](https://reader033.vdocument.in/reader033/viewer/2022060120/5592c01e1a28ab0d418b4774/html5/thumbnails/23.jpg)
23
PT in Action – Case StudyLoad Testing – Application server
Analysis inside the JVM and CLR
Memory not being released
FAIL!! Memory Leak
![Page 24: Performance Engineering Case Study V1.0](https://reader033.vdocument.in/reader033/viewer/2022060120/5592c01e1a28ab0d418b4774/html5/thumbnails/24.jpg)
24
PT in Action – Case StudyLoad Testing – Transaction performance
Concurrent users
Elapsed timeTransaction response time
Concurrent SLA: 600 users
Response time SLA: < 20 seconds
Transaction performance remains below response time SLA
SUCCESS!!
![Page 25: Performance Engineering Case Study V1.0](https://reader033.vdocument.in/reader033/viewer/2022060120/5592c01e1a28ab0d418b4774/html5/thumbnails/25.jpg)
25
PT in Action – Case StudyLoad Testing – Server performance
Concurrent users
Elapsed time
Web server CPU %
Concurrent SLA: 600 users
Server CPU SLA: < 50 %
Web server CPU utilization remains below SLA
SUCCESS!!
![Page 26: Performance Engineering Case Study V1.0](https://reader033.vdocument.in/reader033/viewer/2022060120/5592c01e1a28ab0d418b4774/html5/thumbnails/26.jpg)
26
PT in Action – Case Study
Profiling + Load Testing = All problems identified
Bank Data Centre
Web Servers
App Servers
DB Servers
External Users
Internal Users
WAN Sensitivity
Bad SQL
Slow Methods
Insufficient Capacity
Contention Issues
![Page 27: Performance Engineering Case Study V1.0](https://reader033.vdocument.in/reader033/viewer/2022060120/5592c01e1a28ab0d418b4774/html5/thumbnails/27.jpg)
27
PT in Action – Case StudyDeliverables
PASS!! • Response Time SLA• Login / Signon less than 10 seconds
• All other page response times less than 8 seconds
• All transaction response times less than 20 seconds
• Based on broadband bandwidth
• Concurrency SLA• Support 600 concurrent end users
• Server Utilization SLA• Server utilization < 50% measured as CPU, Memory and Disk I/O
PASS!!
PASS!!
![Page 28: Performance Engineering Case Study V1.0](https://reader033.vdocument.in/reader033/viewer/2022060120/5592c01e1a28ab0d418b4774/html5/thumbnails/28.jpg)
28
PT – ‘Pro-active’ remix
PT – Becoming pro-activeAvoiding the ‘Hair on Fire’ scenario
![Page 29: Performance Engineering Case Study V1.0](https://reader033.vdocument.in/reader033/viewer/2022060120/5592c01e1a28ab0d418b4774/html5/thumbnails/29.jpg)
29
PT ‘Pro-active remix’ – Mission Statement
• Performance considerations should be factored into the application lifecycle at the earliest opportunity
![Page 31: Performance Engineering Case Study V1.0](https://reader033.vdocument.in/reader033/viewer/2022060120/5592c01e1a28ab0d418b4774/html5/thumbnails/31.jpg)
31
PT ‘Pro-active remix’ – Requirements capturePerformance goals
• Response Time SLA• Login / Signon less than 10 seconds
• All other page response times less than 8 seconds
• Transaction response time less than 20 seconds
• Based on broadband bandwidth
• Application Concurrency SLA• 600 concurrent end users
• Server Resource Utilization SLA• Less than 50% measured as CPU, Memory, Network Utilization and Disk I/O
![Page 32: Performance Engineering Case Study V1.0](https://reader033.vdocument.in/reader033/viewer/2022060120/5592c01e1a28ab0d418b4774/html5/thumbnails/32.jpg)
32
PT ‘Pro-active remix’ – Requirements captureTest environment
Compare production to test system and identify differences
![Page 33: Performance Engineering Case Study V1.0](https://reader033.vdocument.in/reader033/viewer/2022060120/5592c01e1a28ab0d418b4774/html5/thumbnails/33.jpg)
33
• Enrollment
• Signon
• View account summary
• View account details
• Intra FI transfer
• View check images
• Copy check
• Bill Payment Transaction
• Check reorder
PT ‘Pro-active remix’ – Requirements captureIdentify key transactions
![Page 34: Performance Engineering Case Study V1.0](https://reader033.vdocument.in/reader033/viewer/2022060120/5592c01e1a28ab0d418b4774/html5/thumbnails/34.jpg)
34
PT ‘Pro-active remix’ – Requirements captureTransaction detail + Data
View Account Summary
• Home Page <- Target URL
• Logon Page <- Login credentials
• Verification Page
• Challenge Page
• Retrieving default account summary
• Retrieving maximum amount of account history (90 days max)
• Logoff
![Page 35: Performance Engineering Case Study V1.0](https://reader033.vdocument.in/reader033/viewer/2022060120/5592c01e1a28ab0d418b4774/html5/thumbnails/35.jpg)
35
PT ‘Pro-active remix’ – Requirements captureIdentify data requirements
• A minimum of 100 sets of unique end user login credentials
• Test database to be a recent cut of live database
• Each consumer banking logon account should contain:
• 2-3 different accounts. (If multiple hosts are accessed during a logon, additional considerations may be warranted).
• Several items of history in one of the accounts – generally a direct draw/checking account. Best results are obtained when this transaction history contains 20 or more items per month distributed over 90 days.
![Page 36: Performance Engineering Case Study V1.0](https://reader033.vdocument.in/reader033/viewer/2022060120/5592c01e1a28ab0d418b4774/html5/thumbnails/36.jpg)
36
PT ‘Pro-active remix’ – Requirements captureVirtual user allocation
Allocation of virtual users to transactions
![Page 37: Performance Engineering Case Study V1.0](https://reader033.vdocument.in/reader033/viewer/2022060120/5592c01e1a28ab0d418b4774/html5/thumbnails/37.jpg)
37
PT ‘Pro-active remix’ – Requirements captureSummary
Critical requirements identified
Performance goals ( SLA’s )
Validation of performance test environment
Data requirements identified
Key transactions identified
Virtual user allocation
![Page 40: Performance Engineering Case Study V1.0](https://reader033.vdocument.in/reader033/viewer/2022060120/5592c01e1a28ab0d418b4774/html5/thumbnails/40.jpg)
© 2008 Compuware Corporation — All Rights Reserved
IonIdea38 – 40 EPIP White Field
Bangalore - 560066
Mohammed Ibrahim, Manager – Testing Practice
Works - +91-80-66581577
Mobile - +91-9986650636
Cell(US)- (703) 268-2920
E-mail: [email protected]
www.ionidea.com
Contact