Download - Multiple Dimensions of Load Testing
![Page 1: Multiple Dimensions of Load Testing](https://reader031.vdocument.in/reader031/viewer/2022022415/5a64b6f37f8b9adf788b4681/html5/thumbnails/1.jpg)
Multiple Dimensions of Load Testing
Alexander Podelko [email protected]
alexanderpodelko.com/blog
@apodelko
Performance & Capacity
2015 by CMG
November 2, 2015
![Page 2: Multiple Dimensions of Load Testing](https://reader031.vdocument.in/reader031/viewer/2022022415/5a64b6f37f8b9adf788b4681/html5/thumbnails/2.jpg)
Agenda
• Load Testing
• Five [New] Load Testing Dimensions
-Environment
-Load Generation
-Testing Approach
-Life-Cycle Integration
-Feedback and Analysis
2
Disclaimer: The views expressed here are my personal views only and do not necessarily represent those of my
current or previous employers. All brands and trademarks mentioned are the property of their owners.
![Page 3: Multiple Dimensions of Load Testing](https://reader031.vdocument.in/reader031/viewer/2022022415/5a64b6f37f8b9adf788b4681/html5/thumbnails/3.jpg)
Load Testing
Applying multi-user synthetic load to the system
– Load testing
– Performance testing
– Stress testing
– Scalability testing
– Volume testing
– Reliability testing
– Concurrency testing
– Realistic Testing
– Endurance testing
– Longevity testing
– Soak testing
– Stability testing
3
![Page 4: Multiple Dimensions of Load Testing](https://reader031.vdocument.in/reader031/viewer/2022022415/5a64b6f37f8b9adf788b4681/html5/thumbnails/4.jpg)
Load Testing Process
Collect Requirements
Define Load
Run Tests
Analyze Results
Done
Modify System
Goals are met
Goals are not met
4
Traditional
View
![Page 5: Multiple Dimensions of Load Testing](https://reader031.vdocument.in/reader031/viewer/2022022415/5a64b6f37f8b9adf788b4681/html5/thumbnails/5.jpg)
The Stereotype
• Load / Performance Testing is:– Last moment before deployment– Last step in the waterfall process– Protocol level record-and-playback– Large corporations– Expensive tools requiring special skills– Lab environment– Scale-down environment – …
5
![Page 6: Multiple Dimensions of Load Testing](https://reader031.vdocument.in/reader031/viewer/2022022415/5a64b6f37f8b9adf788b4681/html5/thumbnails/6.jpg)
Load Testing
• Traditional load testing is not enough anymore
• New industry trends change a lot
– Cloud
– Continuous Integration / Delivery / Deployment
– DevOps
– Agile
• Some even say that load testing is not needed anymore
– Due to other ways to mitigate performance risk
6
![Page 7: Multiple Dimensions of Load Testing](https://reader031.vdocument.in/reader031/viewer/2022022415/5a64b6f37f8b9adf788b4681/html5/thumbnails/7.jpg)
Performance Risk Mitigation• Single-user performance engineering
– Profiling, WPO, single-user performance
• Software Performance Engineering– Modeling, Performance Patterns
• Instrumentation / APM / Monitoring– Production system insights
• Capacity Planning/Management– Resources Allocation
• Continuous Integration / Deployment– Ability to deploy and remove changes quickly
7
![Page 8: Multiple Dimensions of Load Testing](https://reader031.vdocument.in/reader031/viewer/2022022415/5a64b6f37f8b9adf788b4681/html5/thumbnails/8.jpg)
But all of them don’t replace load testing:
Load testing complements them in several important ways !
8
![Page 9: Multiple Dimensions of Load Testing](https://reader031.vdocument.in/reader031/viewer/2022022415/5a64b6f37f8b9adf788b4681/html5/thumbnails/9.jpg)
Can System Handle Peak Load?
• You can’t know without testing:
9
![Page 10: Multiple Dimensions of Load Testing](https://reader031.vdocument.in/reader031/viewer/2022022415/5a64b6f37f8b9adf788b4681/html5/thumbnails/10.jpg)
Verify Multi-User Performance
• Single-user improvement may lead to multi-user performance degradation
10
![Page 11: Multiple Dimensions of Load Testing](https://reader031.vdocument.in/reader031/viewer/2022022415/5a64b6f37f8b9adf788b4681/html5/thumbnails/11.jpg)
What Else Load Testing Adds
• Performance optimization
– Apply exactly the same load
– See if the change makes a difference
• Debugging/verification of multi-user issues
• Testing self-regulation functionality– Such as auto-scaling or changing the level of
service depending on load
11
![Page 12: Multiple Dimensions of Load Testing](https://reader031.vdocument.in/reader031/viewer/2022022415/5a64b6f37f8b9adf788b4681/html5/thumbnails/12.jpg)
So What Is Going On?
• I believe that load testing is here to stay, but should fully embrace the change
– Not one-time, to become dynamic
• Many things that were practically given became a hard choice of a continuum of options (dimension vs. point)
– Environment, Load Generation, Testing Approach, Life-Cycle Integration, Feedback and Analysis
12
![Page 13: Multiple Dimensions of Load Testing](https://reader031.vdocument.in/reader031/viewer/2022022415/5a64b6f37f8b9adf788b4681/html5/thumbnails/13.jpg)
Agenda
• Load Testing
• Five [New] Load Testing Dimensions
-Environment
-Load Generation
-Testing Approach
-Life-Cycle Integration
-Feedback and Analysis
13
![Page 14: Multiple Dimensions of Load Testing](https://reader031.vdocument.in/reader031/viewer/2022022415/5a64b6f37f8b9adf788b4681/html5/thumbnails/14.jpg)
Load Testing Process
Collect Requirements
Define Load
Run Tests
Analyze Results
Done
Modify System
Goals are met
Goals are not met
14
Environment ?
![Page 15: Multiple Dimensions of Load Testing](https://reader031.vdocument.in/reader031/viewer/2022022415/5a64b6f37f8b9adf788b4681/html5/thumbnails/15.jpg)
Deployment
• Lab vs. Service (SaaS) vs. Cloud (IaaS)
– For both the system and load generators
• Test vs. Production
• No best solution, depends on your goals / system
15
![Page 16: Multiple Dimensions of Load Testing](https://reader031.vdocument.in/reader031/viewer/2022022415/5a64b6f37f8b9adf788b4681/html5/thumbnails/16.jpg)
Scenarios
• System validation for high load– Outside load (service or cloud), production system– Wider scope, lower repeatability
• Performance optimization / troubleshooting– Isolated lab environment– Limited scope, high repeatability
• Testing in Cloud– Lowering costs (in case of periodic tests)– Limited scope, low repeatability
16
![Page 17: Multiple Dimensions of Load Testing](https://reader031.vdocument.in/reader031/viewer/2022022415/5a64b6f37f8b9adf788b4681/html5/thumbnails/17.jpg)
Find Your Way
• If performance risk is high it may be a combination of environments, e.g.
– Outside tests against the production environment to test for max load
– Lab for performance optimization / troubleshooting
– Limited performance environments to be used as part of continuous integration
17
![Page 18: Multiple Dimensions of Load Testing](https://reader031.vdocument.in/reader031/viewer/2022022415/5a64b6f37f8b9adf788b4681/html5/thumbnails/18.jpg)
Scaling
• Becomes critical as you get to a large number of virtual users
• The number of supported users per unit of computing power may differ drastically
– Depending on tool, protocol, scenario, system…
• If you need deploy it on a large number of machines automation would be helpful
18
![Page 19: Multiple Dimensions of Load Testing](https://reader031.vdocument.in/reader031/viewer/2022022415/5a64b6f37f8b9adf788b4681/html5/thumbnails/19.jpg)
Agenda
• Load Testing
• Five [New] Load Testing Dimensions
-Environment
-Load Generation
-Testing Approach
-Life-Cycle Integration
-Feedback and Analysis
19
![Page 20: Multiple Dimensions of Load Testing](https://reader031.vdocument.in/reader031/viewer/2022022415/5a64b6f37f8b9adf788b4681/html5/thumbnails/20.jpg)
Load Testing Process
Collect Requirements
Define Load
Run Tests
Analyze Results
Done
Modify System
Goals are met
Goals are not met
20
Create Test Assets
![Page 21: Multiple Dimensions of Load Testing](https://reader031.vdocument.in/reader031/viewer/2022022415/5a64b6f37f8b9adf788b4681/html5/thumbnails/21.jpg)
Record and Playback: Protocol Level
Load Testing Tool
Virtual Users
ServerLoad Generator
Application
Network
21
![Page 22: Multiple Dimensions of Load Testing](https://reader031.vdocument.in/reader031/viewer/2022022415/5a64b6f37f8b9adf788b4681/html5/thumbnails/22.jpg)
Considerations
• Usually doesn't work for testing components
• Each tool support a limited number of technologies (protocols)
• Some technologies are very time-consuming
• Workload validity in case of sophisticated logic on the client side is not guaranteed
22
![Page 23: Multiple Dimensions of Load Testing](https://reader031.vdocument.in/reader031/viewer/2022022415/5a64b6f37f8b9adf788b4681/html5/thumbnails/23.jpg)
Record and Playback: UI Level
23
Load Testing Tool
Virtual
Users
ServerLoad Generator
Application
NetworkBrowsers
![Page 24: Multiple Dimensions of Load Testing](https://reader031.vdocument.in/reader031/viewer/2022022415/5a64b6f37f8b9adf788b4681/html5/thumbnails/24.jpg)
Considerations
• Scalability
– Still require more resources
• Supported technologies
• Timing accuracy
• Playback accuracy
– For example, for HtmlUnit
24
![Page 25: Multiple Dimensions of Load Testing](https://reader031.vdocument.in/reader031/viewer/2022022415/5a64b6f37f8b9adf788b4681/html5/thumbnails/25.jpg)
Programming
Load Testing Tool App.
Virtual Users
ServerLoad Generator
Application
NetworkAPI
25
![Page 26: Multiple Dimensions of Load Testing](https://reader031.vdocument.in/reader031/viewer/2022022415/5a64b6f37f8b9adf788b4681/html5/thumbnails/26.jpg)
Considerations
• Requires programming / access to APIs
• Tool support
– Extensibility
– Language support
• May require more resources
• Environment may need to be set
26
![Page 27: Multiple Dimensions of Load Testing](https://reader031.vdocument.in/reader031/viewer/2022022415/5a64b6f37f8b9adf788b4681/html5/thumbnails/27.jpg)
Agenda
• Load Testing
• Five [New] Load Testing Dimensions
-Environment
-Load Generation
-Testing Approach
-Life-Cycle Integration
-Feedback and Analysis
27
![Page 28: Multiple Dimensions of Load Testing](https://reader031.vdocument.in/reader031/viewer/2022022415/5a64b6f37f8b9adf788b4681/html5/thumbnails/28.jpg)
Load Testing Process
Collect Requirements
Define Load
Run Tests
Analyze Results
Done
Modify System
Goals are met
Goals are not met
28
Sounds as running a
fixed set of tests,
doesn’t it?
![Page 29: Multiple Dimensions of Load Testing](https://reader031.vdocument.in/reader031/viewer/2022022415/5a64b6f37f8b9adf788b4681/html5/thumbnails/29.jpg)
Mentality Change
• Making performance everyone’s job
• Late record/playback performance testing -> Early Performance Engineering
• System-level requirements -> Component-level requirements
• Record/playback approach -> Programming to generate load/create stubs
• "Black Box" -> "Grey Box”
![Page 30: Multiple Dimensions of Load Testing](https://reader031.vdocument.in/reader031/viewer/2022022415/5a64b6f37f8b9adf788b4681/html5/thumbnails/30.jpg)
Performance Testing• Usually is not separated from:
– Tuning
• System should be properly tuned
– Troubleshooting / Diagnostics
• Problems should be diagnosed further to the point when it is clear how to handle them
– Capacity Planning / Sizing
• "Pure" performance testing is rare
– Regression testing ?
![Page 31: Multiple Dimensions of Load Testing](https://reader031.vdocument.in/reader031/viewer/2022022415/5a64b6f37f8b9adf788b4681/html5/thumbnails/31.jpg)
Exploratory Testing
• Rather alien for performance testing, but probably more relevant than for functional testing
• We learn about system’s performance as we start to run test– Only guesses for new systems
• Rather a performance engineering process bringing the system to the proper state than just testing
31
![Page 32: Multiple Dimensions of Load Testing](https://reader031.vdocument.in/reader031/viewer/2022022415/5a64b6f37f8b9adf788b4681/html5/thumbnails/32.jpg)
Test Approach Dimension
NewWell-known
System
Testing Approach
Exploratory / Agile
Automated /Regression
Traditional
![Page 33: Multiple Dimensions of Load Testing](https://reader031.vdocument.in/reader031/viewer/2022022415/5a64b6f37f8b9adf788b4681/html5/thumbnails/33.jpg)
Test Management
• Managing test execution / collecting results
• Virtual users coordination, e.g.:
– Synchronization points
– Data exchange
– Sophisticated scheduling
• Environment simulation, e.g.:
– Browser Simulation
– Network simulation (including mobile)
– IP spoofing
33
![Page 34: Multiple Dimensions of Load Testing](https://reader031.vdocument.in/reader031/viewer/2022022415/5a64b6f37f8b9adf788b4681/html5/thumbnails/34.jpg)
Agenda
• Load Testing
• Five [New] Load Testing Dimensions
-Environment
-Load Generation
-Testing Approach
-Life-Cycle Integration
-Feedback and Analysis
34
![Page 35: Multiple Dimensions of Load Testing](https://reader031.vdocument.in/reader031/viewer/2022022415/5a64b6f37f8b9adf788b4681/html5/thumbnails/35.jpg)
Load Testing Process
Collect Requirements
Define Load
Run Tests
Analyze Results
Done
Modify System
Goals are met
Goals are not met
35
No integration points
at all !
![Page 36: Multiple Dimensions of Load Testing](https://reader031.vdocument.in/reader031/viewer/2022022415/5a64b6f37f8b9adf788b4681/html5/thumbnails/36.jpg)
Agile Support
• Agile / CI support becoming the main theme
• Integration with Continuous Integration Servers
– Jenkins, Hudson, etc.
– Several tools announced integration recently
– Making a part of automatic build process
• Automation support
• Easiness to use
• Support of newest technologies
36
![Page 37: Multiple Dimensions of Load Testing](https://reader031.vdocument.in/reader031/viewer/2022022415/5a64b6f37f8b9adf788b4681/html5/thumbnails/37.jpg)
Automation: Difficulties
• Complicated setups
• Long list of possible issues
• Complex results (no pass/fail)
• Not easy to compare two result sets
• Changing Interfaces
• Tests may be long
![Page 38: Multiple Dimensions of Load Testing](https://reader031.vdocument.in/reader031/viewer/2022022415/5a64b6f37f8b9adf788b4681/html5/thumbnails/38.jpg)
Automation: Considerations
• You need know system well enough to make meaningful automation
• If system is new, overheads are too high
– So almost no automation in traditional environments
• If the same system is tested again and again
– It makes sense to invest in setting up automation
• Automated interfaces should be stable enough
– APIs are usually more stable on early stages
![Page 39: Multiple Dimensions of Load Testing](https://reader031.vdocument.in/reader031/viewer/2022022415/5a64b6f37f8b9adf788b4681/html5/thumbnails/39.jpg)
Tool Support
• Not much tool support was until recently
• Some vendors claimed that their load testing tool better fits agile processes
– Often it meant that the tool is a little easier to use
• Was difficult to find what is available
– Ability to automate: command line, API, data access
– Ability to extend scripts
– Supported technologies
![Page 40: Multiple Dimensions of Load Testing](https://reader031.vdocument.in/reader031/viewer/2022022415/5a64b6f37f8b9adf788b4681/html5/thumbnails/40.jpg)
Tool Support: Recent Developments
• Recently agile support became the main theme
– A lot of new developments
• Integration with Continuous Integration Servers
– Several tools announced integration recently
• Cloud integration
• Support of newest technologies
![Page 41: Multiple Dimensions of Load Testing](https://reader031.vdocument.in/reader031/viewer/2022022415/5a64b6f37f8b9adf788b4681/html5/thumbnails/41.jpg)
Agenda
• Load Testing
• Five [New] Load Testing Dimensions
-Environment
-Load Generation
-Testing Approach
-Life-Cycle Integration
-Feedback and Analysis
41
![Page 42: Multiple Dimensions of Load Testing](https://reader031.vdocument.in/reader031/viewer/2022022415/5a64b6f37f8b9adf788b4681/html5/thumbnails/42.jpg)
Load Testing Process
Collect Requirements
Define Load
Run Tests
Analyze Results
Done
Modify System
Goals are met
Goals are not met
42
Isn’t so simple
anymore
![Page 43: Multiple Dimensions of Load Testing](https://reader031.vdocument.in/reader031/viewer/2022022415/5a64b6f37f8b9adf788b4681/html5/thumbnails/43.jpg)
Reporting and Analysis
• Good integrated reporting and analysis greatly increases efficiency
– Getting all data in one place and synchronized
– Integration of monitoring data is a great help
• Weak spot of many open source tools
43
![Page 44: Multiple Dimensions of Load Testing](https://reader031.vdocument.in/reader031/viewer/2022022415/5a64b6f37f8b9adf788b4681/html5/thumbnails/44.jpg)
Monitoring
• System level
• Application level (APM)
– AppDynamics, New Relic, Dynatrace, etc.
– Many tools have integration
• Integration allows analyze monitoring data together with test results
44
![Page 45: Multiple Dimensions of Load Testing](https://reader031.vdocument.in/reader031/viewer/2022022415/5a64b6f37f8b9adf788b4681/html5/thumbnails/45.jpg)
The Main Change in Monitoring
• Configuration becomes dynamic, changing on the fly
• Auto scaling, auto provisioning, etc.
– Challenge to monitor all moving parts
– Challenge to compare results of dynamic configurations
– Shift to application monitoring
45
![Page 46: Multiple Dimensions of Load Testing](https://reader031.vdocument.in/reader031/viewer/2022022415/5a64b6f37f8b9adf788b4681/html5/thumbnails/46.jpg)
The Main Change in Analysis
• Not only comparison with the requirements
• Many different forms of analysis depending on the tests
– Adjusting to configuration / type of the test
• Component testing
– Automatic analysis / alerting
• Continuous Integration / Delivery / Deployment
– Input for tuning / optimization / sizing
46
![Page 47: Multiple Dimensions of Load Testing](https://reader031.vdocument.in/reader031/viewer/2022022415/5a64b6f37f8b9adf788b4681/html5/thumbnails/47.jpg)
Summary
• The industry is rapidly changing – performance testing should change too
– Fully embrace agile
• Five [new] dimensions introduced by the changes
– Environment, Load Generation, Testing Approach, Life-Cycle Integration, Feedback and Analysis
• Good tools help, but there is no best tool – it depends on your needs
47