1

Automatic Load Test Verification Using Control Charts

Thanh Nguyen, Bram Adams, ZhenMing Jiang, Ahmed E. HassanQueen’s University, Kingston, Canada

Mohamed Nasser, Parminder FloraResearch in Motion, Waterloo, Canada

2

Load testing: Ensure the software performance

With 2000 concurrent users, a GET request should be responded within 2 seconds

3

Development

FunctionalTesting

LoadTesting

4

Importance of software performance:More users

Load testing is regression based

5

Apply operational load e.g. 2000 requests/min

Existing version e.g. version 1.2

New release e.g. version 1.3

BaselinePerformance counters

TargetPerformance counters

Compare the counters to detect performance

regressionCode changes

Automatede.g. HP LoadRunner,MS Visual Studio, …

Still a manual process

6

Challenge in load testing: Lots of data

There are thousands of counters with millions of samples

Frontend BackendClient

Client

Client

Frontend

Client

Frontend

Client

Backend

Backend

Load balancer

CPU%User CPU%Privileged CPU%IO data bytes/secIO read bytes/sec

IO write bytes/secPage faults/secPool paged bytesPrivate bytesVirtual bytes

7

Manufacturing process also requires automatic verification

8

Control charts example

0 5 10 15 20 25710

720

730

740

750

760

770

780

Milk Filled (ml)

CL

LCL

UCL

Violation ratio: 0% 20% 20% 60%

9

Using control charts to verify load test results

BaselinePerformance counters

TargetPerformance counters

Determine the UCL, CL, LCL

0 5 10 15 20 25710720730740750760770780

Performance counter

10

Using control charts to verify load test results

BaselinePerformance counters

TargetPerformance counters

Determine the UCL, CL, LCL

0 5 10 15 20 25710720730740750760770780

Performance counter

Violation ratio

11

BaselinePerformance counters

TargetPerformance counters

TargetPerformance counters

0 5 10 15 20 25710720730740750760770780

Performance counter

BaselinePerformance counters

0 5 10 15 20 25680

700

720

740

760

780

Performance counter

Low violation ratio

High violation ratio

12

We conduct two case studies

• Enterprise system:– Large and complex– Multi-tiers

• Dell DVD store– Small and simple– 3-tiers

13

Type Description

Front end problems

CPU hog a) 50% and b) 75% of CPU on the front end server

Back end problems

CPU hog a) 50% and b) 75% of CPU on one of the back end servers

Busy waiting Busy waiting in the front end server code

Query Extra query to the database back end

NormalPerformance counters

ProblemsPerformance counters

Type Description

Normal Runs of existing version without performance problems

RQ1: Can control charts detect a problematic run?

14Enterprise’s Response Time

Normal Front 50% Front 75% Back 50% Back 75%0

10

20

30

40

50

60

70

80

90

100

Violation ratio

15DVD Store’s Response Time

Normal Front 50% Front 75% Back 50% Back 75% Busy Query0

10

20

30

40

50

60

70

80

90

100

Violation Ratio

16

RQ2: Can control charts identify the problematic component?

Frontend BackendClient

Client

Client

Frontend

Client

Frontend

Client

Backend

Backend

Load balancer

High violation in frontend’s counters

High violation in backend’s counters

17

Backend 50% stress

F B B B B F O B F O0%

10%20%30%40%50%60%70%80%90%

100%

Violation ratioBaseline

Target

18

Filter out unstable countersBaseline

Target

F F F F B B B B B B0%

10%20%30%40%50%60%70%80%90%

100%

Violation ratio

These counters are always out-of-control We filter then out

19

Backend 50% stress

B B B B B O0%

10%20%30%40%50%60%70%80%90%

100%

Violation ratioBaseline

Target

20

Frontend 50% stress

F F F F F F O0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

Violation ratioBaseline

Target

21