performance testing data visualisation

Copyright © 2014 Equinox Limited

Performance TestingData Visualisation

Mining the Nuggets

Richard LeekeNZTester ConferenceAugust 2014


Agenda

• Background and Context• Why Data Visualisation?• Relevance to Performance Testing• Case Studies Galore


Equinox IT is New Zealand’s leading independent IT consultancy, delivering software development, consulting and training services.

Established in 1995, we inject fresh thinking to solve tough business problems.

Equinox IT has over 60 staff based in Wellington and Auckland.

About Equinox IT

www.equinox.co.nz


About me

• Owner and Director, Equinox IT• Principal Consultant, Performance

Intelligence Practice• Regular speaker & content owner

at ‘invite only’ WOPR conferences• Tableau Zen Master (eqit.co/tzm)


Background and Context

• Equinox IT – Performance Intelligence Practice• Touchstones

• Not just about response times - focus on:• Stability• Capacity• Response Time

• Realism in all activities• Workload, script behaviour, correctness, errors, infrastructure

• Measurement at sufficient level of granularity• Visualisation:

• Diagnosis of issues• Communicating results

• Decision making based on confidence in the results


Data Visualisation

• Visually representing data – a picture paints 1000 words• Workload• Response Times• Error Rates• Resource Utilisation• Diagnostics

• Appropriate Representation for Purpose• Aggregate for summary reporting• Detail for diagnosis

• Inappropriate representation leads to wrong conclusion• Use of Business Intelligence techniques

• Slice and Dice• Filter• Drill Down


The Danger of Extrapolation

xkcd.com


Evolution of our use of Visualisation

• Excel• Ubiquitous, easy to distribute results• Limited data volume

• Performance testing tools• Built in, easy access to tool’s own results• Often hard to combine other measurements, often slow, often only summarised, sometimes

downright statistically wrong

• Home grown tools• Allows highly specialised analysis• Too much effort to maintain and enhance

• BI tools• Generally our preferred approach


What to Visualise

• Workload• Model the required workload• Was the correct workload applied with realistic time distribution?

• Resource Utilisation• System Under Test capacity• Was the test constrained by resource exhaustion on the test drivers?

• Error Rates• Did the test actually work?

• Response Times• Was it fast enough?

• Diagnostics• Why not?


Communicating the Results

• Story telling not just diagnosis• Team engagement through evidence• Communicate the complex in a simple way• Engender team collaboration• Business impact

• Project “burn” cost of delays in resolving issues• Hours to solve problems in ineffective war rooms• Buy too much/too little hardware• Cancel project if problems not resolved


Case Studies


Case Study #1How Colour Helps


Case Study #2Pinpointing a Database Lock• Original Excel View• Data Visualised with Tableau


Case Study #2Pinpointing a Database Lock


Case Study #3Impact of Resource Monitoring


Case Study #4“Aerial View” of Test Outcome• Live Visualisation with Tableau


Case Study #4“Aerial View” of Test Outcome


Case Study #5Identifying which Tier has a Problem• Live Visualisation with “Scatter Brain”


Case Study #5Identifying which Tier has a Problem


Case Study #6Hunting an Obscure Virtualisation Issue• Live Visualisation with Tableau


Case Study #6Hunting an Obscure Virtualisation Issue


Case Study #7Cyclic Pattern of CPU Usage

0

5

10

15

20

25

30

35

40

0 60 120

180

240

300

360

420

480

540

600

660

720

780

840

900

960

1020

1080

1140

1200

1260

1320

1380

1440

monitoringother

Average of tot_cpu

time

proc_set


Case Study #7CPU Usage per Process

0

0.01

0.02

0.03

0.04

0.05

0.06

0.07

0.08

0.09

0 60 120

180

240

300

360

420

480

540

600

660

720

780

840

900

960

1020

1080

1140

1200

1260

1320

1380

svprd - bash/1svprd - enm/5svprd - ero/5svprd - ert/5svprd - in_adaptor/3svprd - in_adaptor/4svprd - oracle/1svprd - perl/1svprd - perl/3svprd - prstat/1svprd - tremox/4svprd - trerate/4svprd - trerodb/4svprd - trerwdb/4svprd - WSH/4

Sum of CPU

time

USERNAMEPROCESS/NLWP


Case Study #7Cycle Time Changed by Server Reboot

Total

0

100

200

300

400

500

600

700

800

900

0 300 600 900 1200 1500 1800 2100 2400 2700 3000 3300 3600 3900

Total

Sum of tot_cpu

time


Case Study #7Bursts of Requests Every 10 ms

18:4

9:30

18:5

0:00

18:5

0:30

18:5

1:00

18:5

1:30

18:5

2:00

18:5

2:30

18:5

3:00

18:5

3:30

18:5

4:00

18:5

4:30

18:5

5:00

18:5

5:30

18:5

6:00

18:5

6:30

18:5

7:00

18:5

7:30

18:5

8:00

18:5

8:30

18:5

9:00

18:5

9:30

19:0

0:00

19:0

0:30

19:0

1:00

19:0

1:30

19:0

2:00

19:0

2:30

DEBIT - request_ms - 01 0-1_ms




PROFILE - request_ms - 03 2-3_ms



RESERVE - request_ms - 02 1-2_ms



300-400200-300100-2000-100

Sum of count

bucket_time

mox_operationstats_typebin


Case Study #7Responses also in Bursts

18:4

9:30

18:5

0:10

18:5

0:50

18:5

1:30

18:5

2:10

18:5

2:50

18:5

3:30

18:5

4:10

18:5

4:50

18:5

5:30

18:5

6:10

18:5

6:50

18:5

7:30

18:5

8:10

18:5

8:50

18:5

9:30

19:0

0:10

19:0

0:50

19:0

1:30

19:0

2:10

19:0

2:50

DEBIT - response_ms - 01 0-1_ms




PROFILE - response_ms - 03 2-3_ms



RESERVE - response_ms - 02 1-2_ms



400-500300-400200-300100-2000-100

Sum of count

bucket_time

mox_operationstats_typebin


Summary

• Collect as much detailed data as you can• Invaluable for problem identification and diagnosis• Often helps “tell the story” convincingly• You won’t know exactly what you need till after

you have run the test• Choose the appropriate visualisation

• Exploration vs Story Telling


Resources

• Equinox blog posts• Calculating Percentiles with Tableau (eqit.co/b)• Visualising Lots of Data (eqit.co/c)

http://blog.equinox.co.nz/blog/Lists/Posts/Post.aspx?ID=160






Questions?


Injecting fresh thinking to solve tough business problems.

performance testing data visualisation

Data & Analytics