performance testing data visualisation
Post on 14-Apr-2017
199 Views
Preview:
TRANSCRIPT
Copyright © 2014 Equinox Limited
Performance TestingData Visualisation
Mining the Nuggets
Richard LeekeNZTester ConferenceAugust 2014
Copyright © 2014 Equinox Limited
Agenda
• Background and Context• Why Data Visualisation?• Relevance to Performance Testing• Case Studies Galore
Copyright © 2014 Equinox Limited
Equinox IT is New Zealand’s leading independent IT consultancy, delivering software development, consulting and training services.
Established in 1995, we inject fresh thinking to solve tough business problems.
Equinox IT has over 60 staff based in Wellington and Auckland.
About Equinox IT
www.equinox.co.nz
Copyright © 2014 Equinox Limited
About me
• Owner and Director, Equinox IT• Principal Consultant, Performance
Intelligence Practice• Regular speaker & content owner
at ‘invite only’ WOPR conferences• Tableau Zen Master (eqit.co/tzm)
Copyright © 2014 Equinox Limited
Background and Context
• Equinox IT – Performance Intelligence Practice• Touchstones
• Not just about response times - focus on:• Stability• Capacity• Response Time
• Realism in all activities• Workload, script behaviour, correctness, errors, infrastructure
• Measurement at sufficient level of granularity• Visualisation:
• Diagnosis of issues• Communicating results
• Decision making based on confidence in the results
Copyright © 2014 Equinox Limited
Data Visualisation
• Visually representing data – a picture paints 1000 words• Workload• Response Times• Error Rates• Resource Utilisation• Diagnostics
• Appropriate Representation for Purpose• Aggregate for summary reporting• Detail for diagnosis
• Inappropriate representation leads to wrong conclusion• Use of Business Intelligence techniques
• Slice and Dice• Filter• Drill Down
Copyright © 2014 Equinox Limited
The Danger of Extrapolation
xkcd.com
Copyright © 2014 Equinox Limited
Evolution of our use of Visualisation
• Excel• Ubiquitous, easy to distribute results• Limited data volume
• Performance testing tools• Built in, easy access to tool’s own results• Often hard to combine other measurements, often slow, often only summarised, sometimes
downright statistically wrong
• Home grown tools• Allows highly specialised analysis• Too much effort to maintain and enhance
• BI tools• Generally our preferred approach
Copyright © 2014 Equinox Limited
What to Visualise
• Workload• Model the required workload• Was the correct workload applied with realistic time distribution?
• Resource Utilisation• System Under Test capacity• Was the test constrained by resource exhaustion on the test drivers?
• Error Rates• Did the test actually work?
• Response Times• Was it fast enough?
• Diagnostics• Why not?
Copyright © 2014 Equinox Limited
Communicating the Results
• Story telling not just diagnosis• Team engagement through evidence• Communicate the complex in a simple way• Engender team collaboration• Business impact
• Project “burn” cost of delays in resolving issues• Hours to solve problems in ineffective war rooms• Buy too much/too little hardware• Cancel project if problems not resolved
Copyright © 2014 Equinox Limited
Case Studies
Copyright © 2014 Equinox Limited
Case Study #1How Colour Helps
Copyright © 2014 Equinox Limited
Case Study #1How Colour Helps
Copyright © 2014 Equinox Limited
Case Study #1How Colour Helps
Copyright © 2014 Equinox Limited
Case Study #1How Colour Helps
Copyright © 2014 Equinox Limited
Case Study #2Pinpointing a Database Lock• Original Excel View• Data Visualised with Tableau
Copyright © 2014 Equinox Limited
Case Study #2Pinpointing a Database Lock
Copyright © 2014 Equinox Limited
Case Study #3Impact of Resource Monitoring
Copyright © 2014 Equinox Limited
Case Study #3Impact of Resource Monitoring
Copyright © 2014 Equinox Limited
Case Study #3Impact of Resource Monitoring
Copyright © 2014 Equinox Limited
Case Study #4“Aerial View” of Test Outcome• Live Visualisation with Tableau
Copyright © 2014 Equinox Limited
Case Study #4“Aerial View” of Test Outcome
Copyright © 2014 Equinox Limited
Case Study #4“Aerial View” of Test Outcome
Copyright © 2014 Equinox Limited
Case Study #4“Aerial View” of Test Outcome
Copyright © 2014 Equinox Limited
Case Study #5Identifying which Tier has a Problem• Live Visualisation with “Scatter Brain”
Copyright © 2014 Equinox Limited
Case Study #5Identifying which Tier has a Problem
Copyright © 2014 Equinox Limited
Case Study #5Identifying which Tier has a Problem
Copyright © 2014 Equinox Limited
Case Study #6Hunting an Obscure Virtualisation Issue• Live Visualisation with Tableau
Copyright © 2014 Equinox Limited
Case Study #6Hunting an Obscure Virtualisation Issue
Copyright © 2014 Equinox Limited
Case Study #6Hunting an Obscure Virtualisation Issue
Copyright © 2014 Equinox Limited
Case Study #6Hunting an Obscure Virtualisation Issue
Copyright © 2014 Equinox Limited
Case Study #6Hunting an Obscure Virtualisation Issue
Copyright © 2014 Equinox Limited
Case Study #6Hunting an Obscure Virtualisation Issue
Copyright © 2014 Equinox Limited
Case Study #6Hunting an Obscure Virtualisation Issue
Copyright © 2014 Equinox Limited
Case Study #7Cyclic Pattern of CPU Usage
0
5
10
15
20
25
30
35
40
0 60 120
180
240
300
360
420
480
540
600
660
720
780
840
900
960
1020
1080
1140
1200
1260
1320
1380
1440
monitoringother
Average of tot_cpu
time
proc_set
Copyright © 2014 Equinox Limited
Case Study #7CPU Usage per Process
0
0.01
0.02
0.03
0.04
0.05
0.06
0.07
0.08
0.09
0 60 120
180
240
300
360
420
480
540
600
660
720
780
840
900
960
1020
1080
1140
1200
1260
1320
1380
svprd - bash/1svprd - enm/5svprd - ero/5svprd - ert/5svprd - in_adaptor/3svprd - in_adaptor/4svprd - oracle/1svprd - perl/1svprd - perl/3svprd - prstat/1svprd - tremox/4svprd - trerate/4svprd - trerodb/4svprd - trerwdb/4svprd - WSH/4
Sum of CPU
time
USERNAMEPROCESS/NLWP
Copyright © 2014 Equinox Limited
Case Study #7Cycle Time Changed by Server Reboot
Total
0
100
200
300
400
500
600
700
800
900
0 300 600 900 1200 1500 1800 2100 2400 2700 3000 3300 3600 3900
Total
Sum of tot_cpu
time
Copyright © 2014 Equinox Limited
Case Study #7Bursts of Requests Every 10 ms
18:4
9:30
18:5
0:00
18:5
0:30
18:5
1:00
18:5
1:30
18:5
2:00
18:5
2:30
18:5
3:00
18:5
3:30
18:5
4:00
18:5
4:30
18:5
5:00
18:5
5:30
18:5
6:00
18:5
6:30
18:5
7:00
18:5
7:30
18:5
8:00
18:5
8:30
18:5
9:00
18:5
9:30
19:0
0:00
19:0
0:30
19:0
1:00
19:0
1:30
19:0
2:00
19:0
2:30
DEBIT - request_ms - 01 0-1_ms
DEBIT - request_ms - 04 3-4_ms
DEBIT - request_ms - 07 6-7_ms
DEBIT - request_ms - 10 9-10_ms
PROFILE - request_ms - 03 2-3_ms
PROFILE - request_ms - 06 5-6_ms
PROFILE - request_ms - 09 8-9_ms
RESERVE - request_ms - 02 1-2_ms
RESERVE - request_ms - 05 4-5_ms
RESERVE - request_ms - 08 7-8_ms
300-400200-300100-2000-100
Sum of count
bucket_time
mox_operationstats_typebin
Copyright © 2014 Equinox Limited
Case Study #7Responses also in Bursts
18:4
9:30
18:5
0:10
18:5
0:50
18:5
1:30
18:5
2:10
18:5
2:50
18:5
3:30
18:5
4:10
18:5
4:50
18:5
5:30
18:5
6:10
18:5
6:50
18:5
7:30
18:5
8:10
18:5
8:50
18:5
9:30
19:0
0:10
19:0
0:50
19:0
1:30
19:0
2:10
19:0
2:50
DEBIT - response_ms - 01 0-1_ms
DEBIT - response_ms - 04 3-4_ms
DEBIT - response_ms - 07 6-7_ms
DEBIT - response_ms - 10 9-10_ms
PROFILE - response_ms - 03 2-3_ms
PROFILE - response_ms - 06 5-6_ms
PROFILE - response_ms - 09 8-9_ms
RESERVE - response_ms - 02 1-2_ms
RESERVE - response_ms - 05 4-5_ms
RESERVE - response_ms - 08 7-8_ms
400-500300-400200-300100-2000-100
Sum of count
bucket_time
mox_operationstats_typebin
Copyright © 2014 Equinox Limited
Summary
• Collect as much detailed data as you can• Invaluable for problem identification and diagnosis• Often helps “tell the story” convincingly• You won’t know exactly what you need till after
you have run the test• Choose the appropriate visualisation
• Exploration vs Story Telling
Copyright © 2014 Equinox Limited
Resources
• Equinox blog posts• Calculating Percentiles with Tableau (eqit.co/b)• Visualising Lots of Data (eqit.co/c)
Copyright © 2014 Equinox Limited
Questions?
Copyright © 2014 Equinox Limited
Injecting fresh thinking to solve tough business problems.
top related