load testing with visual studio and azure - andrew siemer
TRANSCRIPT
Performance testing with Visual Studio & Azure
How to test your application using the tools you already know!
Andrew Siemer | Clear Measure
@asiemer
Andrew Siemerhttp://about.me/andrewsiemer
ASP InsiderMS v-TSP (Azure)
Azure Advisor ProgramFather of 6. Jack of all trades, master of some.
We are hiring!!!
Introduction
Agenda
• What is performance testing?
• How can Visual Studio help?
• How can Azure make load testing easy?
What is performance
testing?
What is it?
• A type of testing intended to determine• Responsiveness
• Throughput
• Reliability
• Scalability
…under a given workload
Why would I care?
• Are you ready to go to production?
• Can you handle the expected load?
• Do you have edge case bugs?
• How much traffic causes your app to tip over?
• Do you need to optimize something?
• Have new features caused the app to support less traffic?
• Do you have enough hardware to support known load?
• What causes your application to fail spectacularly?
Three types of performance tests
• Performance testing
• Load testing
• Stress testing
Performance testing
• Speed
• Scalability
• Stability
• Achieves expected performance in general use case
• Can be run often
• Cheap to run with low volume
Load testing
• Expected performance remains steady under production load
• Assumes everything is normal
• Does the network handle this load
• Can the database handle it
• Is the application still meeting SLA and usable
Stress testing
• Pushing the application beyond expected limits
• Identifies ceilings in capacity
• Tests low memory, disk space limitations, or dead server
• Helps see how and when an application will fail
• What happens when the network gets clogged
Many other types of tests
• Capacity (system capacity meets business volume)
• Component (component meeting expectations)
• Endurance (is performance consistent over time)
• Investigation (performance trending over time)
• Smoke (build ready for perf testing)
• Spike (temporarily exceeds expected load)
• Unit (segment of code reasonably efficient)
• Validation (faster or slower)
Baselines
• A baseline tells you where the app is now
• Allows you to see change over time• Target to demonstrates improvements
• Baselines can be created for• System
• Component
• Application
Benchmarking
• The comparison of a current test with a baseline• Looking for changes in the results
• Or a comparison against industry standards• Are my commerce pages as fast as the industry expects
When to shift from testing to tuning?
• Once testing has found unacceptable results
• When results are acceptable, but servers are running extra hot
• When an SLA is breached that we know we need to fix• Especially when it impacts a large test surface
Test planning
Before we test
• Define acceptance criteria
• Write down key scenarios
• Create workload model
• Target load levels
• Determine metrics to capture
• Design tests
Acceptance criteria
• Write down your objectives• Response time – page load speed
• Throughput - number of concurrent users
• Resource utilization – processor, memory, disk I/O, network I/O
• Maximum user load
• Business metrics – number of orders to handle
User scenarios
• “Happy paths” through your application
• Touches many components in your system
• Commonly used paths through the application
• Highest business value paths through the application
User scenarios - examples
• Log in
• Browse catalog
• Add to cart
• Check out
• Register
• Search
• Faceted navigation
User scenarios – specific example
• For a given scenario – determine activities in the test
• The scenario “add to cart” may include these activities• View home page
• View dirt bike category page
• Choose grips
• Add a grip to the cart
• Go to the cart
• Validate that the item is in the cart
User scenarios – most common/intensive
• Be sure to include highest use scenarios
• And include most resource intensive scenarios
• Look at your existing applications log files for top use case
Define the workload
• Know and attempt to simulate existing traffic patterns
• Understand user delay, or “think” time• A load test should not be a denial of service attack!
• Plan scenarios around average session times• Not too short or too long
• Not every scenario can be a new user, returning user, or either• Plan around the reality of your application
Target load levels
• Load levels are applied to “workload”
• Understand business volume as it relates to your objectives• Common load, vs. a big marketing push, or black Friday
• Key scenarios
• Distribution of work
• Average session times
Define what metrics to capture - business
• Too many metrics can make the results hard to read
• Ask questions related to performance that have specific answers• How many orders are place per minute
• What is the response time of the cart page
• Identify what metrics to capture based on your questions
• Looks for lower level metrics that help answer your questions
• Reevaluate this often – applications change – so should the metrics
Define what metrics to capture - application
• Network – hardware, switches, routers, gateways, load balancers
• System – disk, processor, memory, network
• Platform – the host of your app: IIS, worker role, web role, VM
• Application – perf counters, instrumentation, file locks, db locks, queue
Design tests
• Using your thought so far design specific tests to meet your needs
• Don’t change the tests because it is hard to write it as designed
• Test expected data as well as unexpected data (form validation, bad credit card, etc.)
• Be sure to include think time
• Best tests data is collected from production data (where applicable)
• Think about spiders, batch processes, while real users are browsing
• Don’t over simplify your tests!
How can Visual Studio help?
Visual Studio 2013 Ultimate
• VS feature matrix: http://goo.gl/3VtY01
• Ultimate gets you web load and performance testing
• Test Pro doesn’t get you this functionality!
Web performance tests
The beginnings of our load testing!
The test project
Recording a test
Recording a test – uh oh!
• The first time you try to record, IE likely won’t work
• Basically, you need to• Disable Enhanced Protection Mode
• Enable Web Test Recorder
• Go here for full steps to fix this: http://goo.gl/LcFNSJ
Recording a test – uh oh!
Recording a test
Recording a test
Recording a test
Verbose recording…likely too much!
Let’s clean up the recording
Pruned tests
Recorded tests cleaned up
Run the tests locally
Run the tests locally
Modifying test run settings
Making dynamic tests
Way more useful!
Once you start seeing repetition
Consider investigating patterns in the tests
• Look for patterns in the requests
• Create a data source with variable data• Sku’s
• Makes
• Models
• Year
• Dynamically create a wide set of tests
Adding a data source
• Once tests need to cover more…
• Or you need to create representative load…
• Types of data source• Database – “select * from tablename”
• CSV
• XML
Create a shippable datasource
Add the data source to your project
Point at the file then magic
Choose the table to include
And then we have a data source
Update recording to use data source
Update recording to use data source
If the test only runs once…
View the dynamic results
Conditional & loop steps
• Context parameter exists (test context)
• Cookie exists
• Cookie value comparisons
• Last request outcome• Pass or fail
• Last response code • http response codes
• Number comparison
• String comparison
• Probability rules• Return true some times
• Loops
Plug ins
• Web test plug ins
• Web test request plugins
• Should be in a separate assembly
• Allows you to run code pre or post a test/request execution
• Microsoft.VisualStudio.TestTools.WebTesting• WebTestPlugin
• WebTestRequestPlugin
Load tests
Web tests to the max!!!
• Use the web test scenarios you create
• But put the workload, load, metrics, browser mix, and other points around them
Load test wizard
• Running the load test wizard makes life easy!
When matching transaction percentages
When matching percentage of users per test
When matching pace of user per test
When matching pace of user per test
Choose to tests to include
Set distribution for each test
Define network mix
Define browser mix
Capture computer metrics
Configure the run frequency
Ready to run!
Errors?
But waitthere’s more!
How can Azure let you go big?
Your laptop can’t tip over production!
• Local tests are great to capture big pain points
• Or to capture simple content missing errors
• Or to run as a smoke test
Test farm in Azure is magic!
• With Visual Studio Online• And Azure
• We can go big!
Open local test settings
Choose to run in VSO
Connect to your VSO account
Choose your team project
And run the test!
Ok, that rocks, HOW MUCH?
• Based on “Virtual Users Minutes” – VUM
• Azure calculator for load testing: • http://goo.gl/XBrkY1
• $.0004/VUM for 20,001-2M VUM
• $.0002/VUM for 2,000,001-10M VUM
• $.0001/VUM for usage above 10M VUM/MO
