Understanding VSTS Load Testing Framework

Understanding the Visual Studio Team SystemLoad Testing Framework

Introduction................................................................................................................................................2Overview....................................................................................................................................................2Test Client..................................................................................................................................................3Load Test...................................................................................................................................................4

Scenario..................................................................................................................................................4Counter Sets...........................................................................................................................................5Run Settings...........................................................................................................................................5

Test Execution Environment......................................................................................................................6Test Controller & Test Agent.................................................................................................................6Run Configurations................................................................................................................................6Test Rig..................................................................................................................................................7

Test Management Repository....................................................................................................................8References................................................................................................................................................10

© 2006 Chris Birmele, Microsoft Corporation 1

Understanding VSTS Load Testing Framework

Introduction

The purpose of this paper is to provide an overview of the Load Testing Framework that is part of Microsoft’s Visual Studio Team System.

This paper is not a step-by-step tool guide - refer to the comprehensive Visual Studio Team System Online Help, neither is it a best practices guide - consult with Testing Consultants for this kind of advice.

Overview

Before delving into the details, let’s start with some basics.

The purpose of a load test is to understand a system’s behaviour when used by a large number of concurrent users. Is the system still reliable when 500 users access it simultaneously? What about response times? How many users can the system handle? This and many other questions can be answered by performing load tests.

One way to find out is to invite all your friends and family members and have them use the system; the other is to use a software testing solution like Visual Studio Team System. A much cheaper and more reliable alternative!

In order to perform a load test on a system using software tools:

We need a test client (a user interface and/or API’s) to define tests we want to run against the system, to launch tests and monitor and analyse results.

We need to create load tests that accurately simulate the behaviour of a large number of users.

We need a test execution environment responsible for running tests and collecting data for analysis.

We need a test management repository to store test results and manage test asset versions.

We need a system to test!

© 2006 Chris Birmele, Microsoft Corporation 2Figure 1

Test Client

System Under Test

TestsTestsTestsTestsTestsTestsTestsTests

create

run Test Execution Environment

results

Visual Studio Team System

Test AssetManagementRepository

manage

Test Results

Test Runs

Tests

Visual Studio Command LineTest Client

Test Manager Interface

Test Assets Management

Repository(Team Foundation Server)

Test ExecutionEnvironment

(Test Controller)

.NET Remoting .NET Remoting

Visual Studio Command LineTest Client

Test Manager Interface

Test Assets Management

Repository(Team Foundation Server)

Test ExecutionEnvironment

(Test Controller)

.NET Remoting .NET Remoting

Understanding VSTS Load Testing Framework

Test Client

Visual Studio Team System1 provides two test clients:

A graphical user interface A command line interface (not covered in this paper!)

The test client allows us to create and manage tests, launch tests and monitor and analyse test results.

The two interfaces communicate via the Test Manager Interface (a managed API) with other components of the testing framework, namely the Test Execution Environment and the Test Management Repository.

The graphical user interface in Visual Studio Team System consists of a Test Manager view allowing you to create and manage tests, a Test Run View to see current and past test runs as well as a Test Results view (Figure 3).

1 I use the term Visual Studio Team System in this paper, which is the name used to describe an entire product family. The testing capabilities described in this paper are really only part of two products:

Visual Studio Team Suite Visual Studio Team Edition for Testers

Check this for an overview of the various product version: http://msdn.microsoft.com/vstudio/teamsystem/products/compare/default.aspx

© 2006 Chris Birmele, Microsoft Corporation 3

Figure 2

Figure 3

Scenarios Counter Sets

Load Test

Run SettingsRun SettingsRun SettingsCounter SetsCounter SetsScenariosScenariosScenarios Counter Sets

Load Test

Run SettingsRun SettingsRun SettingsCounter SetsCounter SetsScenariosScenarios

Understanding VSTS Load Testing Framework

Load Test

Designing a load test to simulate the behaviour of a large number of users is not an easy task. The challenge is to understand how users will use the system, what data we need to capture, where and how we collect the required data and what does the data mean.

In Visual Studio Team System this information is captured in a load test. A load test consists of the following parts:

Scenarios specifying the tests to run to simulate a specific user interaction.

Counter Sets specifying the data a test run will collect and their thresholds.

Run Settings specifying the computers that will collect the data and other runtime parameters such as the test duration and the data collection sampling rate.

Scenario

A scenario captures all the information required to simulate a specific user interaction with the system as realistically as possible:

Test Mix describes the mix of tests that constitute a test scenario.

Browser Mix describes the browser mix simulated during execution.

Network Mix describes the network mix simulated during execution.

Load Pattern describes how test instances are generated during execution.

© 2006 Chris Birmele, Microsoft Corporation 4

Figure 4

Scenario

Test Mix Browser Mix Network Mix Load Pattern

• Web Test• Unit Test

• IE• Netscape• …

• LAN• WAN• …

• Constant• Stepped• Goal Seeking

Scenario

Test Mix Browser Mix Network Mix Load Pattern

• Web Test• Unit Test

• IE• Netscape• …

• LAN• WAN• …

• Constant• Stepped• Goal Seeking

Figure 5

Understanding VSTS Load Testing Framework

Counter Sets

A counter set specifies the data collected during the load test execution. Visual Studio Team System provides a large number of counter sets that group individual counters around technologies.

Counters can have thresholds associated with them to indicate values that need to be reported as warnings or as errors during the execution of a load test.

Run Settings

The run settings specify important runtime parameters such as:

The load test duration The data collection sampling rate The maximum number of errors to report The computers that collect the data (includes test controllers, agents and any other computer

you want to collect data from)

A Load Test can be associated with one or more run settings, but only one can be active at anyone time.

Figure 6 is an example of how scenarios, counter sets and run settings appear in Visual Studio Team System:

© 2006 Chris Birmele, Microsoft Corporation 5

Scenarios

Counter Sets

Run Settings

Run Settings Properties

Figure 6

Understanding VSTS Load Testing Framework

© 2006 Chris Birmele, Microsoft Corporation 6

Understanding VSTS Load Testing Framework

Test Execution EnvironmentThe test execution environment simulates concurrent users by executing a number of test instances – one for each simulated user – against the system under test.

At the heart of the test execution environment is the test controller and test agents:

Test Controllers coordinate test agents, collect test results and communicate with test clients.

Test Agents execute test code, collect performance statistics and return the information to the controller.

Test Controller & Test Agent

A test client initiates a load test run by communicating to the test controller the test(s) to execute.

The controller then delegates the task of executing a test instance to one or more agents.

The test agent executes the test code, collects performance data and returns it to the controller, which logs it in a data store.

The controller communicates performance data back to the test client for analysis.

The interactions between test client, test controller and test agents are governed by the Run Settings and the Run Configurations settings.

Run Configurations

Run Settings are specific to load tests; Run Configurations define more generic settings applicable to most test types (Figure 9):

Test run naming conventions

© 2006 Chris Birmele, Microsoft Corporation 7

Test Execution Environment

Controller

Agent(s)

System under Test

Test Management Interface

Test Client

.NET Remoting

Agent(s)Agent(s)

Test Execution Environment

Controller

Agent(s)

System under Test

Test Management Interface

Test Client

.NET Remoting

Agent(s)Agent(s)

Figure 7

Controller Service

Test Controller

Test ResultsStore

Agent Service

Agent Process

Test Agent

Test Code

1. d

eplo

ys

2. s

tarts

3. ru

ns

4. re

sults

•Test Code•Tests•Test Results

Test Client

Controller Service

Test Controller

Test ResultsStore

Agent Service

Agent Process

Test Agent

Test Code

1. d

eplo

ys

2. s

tarts

3. ru

ns

4. re

sults

•Test Code•Tests•Test Results

Test Client

Figure 8

Understanding VSTS Load Testing Framework

Whether tests are run locally or on a remote Test Rig Assemblies that need to be instrumented to collect Code Coverage information Additional files and directories to be deployed for tests to run successfully (the test code is

automatically deployed) Generic setup and cleanup scripts that must be executed as part of a test run Test abortion and test failure timeouts Web Test specific settings that need to be persisted across multiple test runs

A Visual Studio Team System solution can have multiple Run Configurations, but only one can be active at anyone time (Figure 9).

Test Rig

Tests - load tests and others - run either locally or on a remote Test Rig.

For a local load test run, all required components are located on the test client machine (i.e. Visual Studio Team System).

Generating test instances to simulate virtual users requires memory and CPU cycles. There is a limit of how many test instances can be generated on a single computer.

If you need to simulate a large number of test instances that can not be generated from a single computer you need to set up a Test Rig. A Test Rig consists of one test controller and a collection of test agents that generate the desired user load (Figure 12).

© 2006 Chris Birmele, Microsoft Corporation 8

Figure 10

Figure 9

System Under Test

Test ClientTest Agent

Test Controller

System Under Test

Test ClientTest Agent

Test Controller

Figure 11

Understanding VSTS Load Testing Framework

In this scenario, the test client communicates with a remote test controller to initiate a test run.

The test controller collaborates with all registered test agents each of which executes test instances against the system under test.

The Run Configurations settings determine how the test controller distributes the test instances among the test agents.

Test controllers and test agents have default counter sets assigned, determining the performance data to collect.

Visual Studio Team System automatically maps these counter sets to the test controller and test agents that make up the test rig.

Test Management Repository

One of the most important components of Visual Studio Team System is the Team Foundation Server or TFS.

It is beyond the scope of this paper to describe all features of TFS, but suffice to say that here it serves as the Test Management Repository (see Figure 13).

The test assets, i.e. the tests (load test as well as other tests) and the Run Configurations are stored in the version control system of TFS. They can be viewed, compared, branched and merged like any other file.

The results of individual test runs are stored in a separate database. The summary of these runs is recorded in the data warehouse.

Test runs can be associated with any work items. For example a failed test can be associated with a new defect, or a successful test can be associated with a Quality of Service requirement to prove the requirement has been satisfied.

© 2006 Chris Birmele, Microsoft Corporation 9

System Under Test

Test Client

Test Agent

Test Controller

Test Agent Test Agent

Test Rig

System Under Test

Test Client

Test Agent

Test Controller

Test Agent Test Agent

Test Rig

Source Control

Build / TestResults

Work Items

Data Warehouse

Team Foundation Server

Test Management Interface

Test Client

Web Service

Source Control

Build / TestResults

Work Items

Data Warehouse

Team Foundation Server

Test Management Interface

Test Client

Web Service

Figure 13

Figure 12

Understanding VSTS Load Testing Framework

Figure 14 shows how these scenarios appear in Visual Studio Team System.

© 2006 Chris Birmele, Microsoft Corporation 10

TeamFoundation

Server

Test Asset Version Control

Work Item Associations

Test Run Summaries

TeamFoundation

Server

Test Asset Version Control

Work Item Associations

Test Run Summaries

Figure 14

Understanding VSTS Load Testing Framework

References

The content of this paper is based on a variety of information sources, such as blogs, Microsoft presentations, the Visual Studio Team System online help and others.

To the best of my knowledge the explanations given in this document are correct, but I would appreciate any corrections and constructive suggestions of how this paper could be improved.

Sydney, June 06

Chris BirmeleDeveloper Tools Technical SpecialistAustralia & New Zealand

cbirmele@microsoft.com

© 2006 Chris Birmele, Microsoft Corporation 11