abc of teradata system performance analysis

38
ABC of Teradata System Performance Analysis Shaheryar Iqbal [email protected] GCC Pakistan. Date created: Feb 20, 2009. Last Updated: Oct 12, 2009.

Upload: ajaypulavarti

Post on 12-Mar-2015

303 views

Category:

Documents


11 download

TRANSCRIPT

Page 1: ABC of Teradata System Performance Analysis

ABC of Teradata System Performance Analysis

Shaheryar [email protected]

GCC Pakistan.Date created: Feb 20, 2009.Last Updated: Oct 12, 2009.

Page 2: ABC of Teradata System Performance Analysis

2 > 10 April 2023

Agenda

1) Co-existing systems parallel efficiency

2) CPU Reports1) CPU Utilization2) CPU Node Hours available3) Percent Shift Busy4) OS % of CPU5) CPU Utilization among Vprocs6) Nodes Parallel Efficiency7) AMPs Parallel Efficiency

3) Memory Reports1) Memory availbilty2) Mem allocation Failures

4) Disk I/O Reports1) Disk busy %2) I/O Wait 3) Measuring I/O4) Disk Read Writes5) Full cylinder Reads6) Logical vs. Physical Reads7) Mini Cyl Packs

5) Buddy Backup Effectiveness

6) AWT Usage

7) Host Utilities Report

8) Charting Teradata Manager Reports

Page 3: ABC of Teradata System Performance Analysis

3 > 10 April 2023

Feed back

______________________________________________ From: Clark, Dave Sent: Saturday, March 07, 2009 12:14 AMTo: Iqbal, ShaheryarSubject: ABC of Teradata System Performance Analysis

Shaheryar-

I have approved your presentation for general viewing. Thank-you very much for your effort in developing this. The information is very useful.

-dave.clark(858)485-2177

• http://pc01.teradata.com/CKS

Page 4: ABC of Teradata System Performance Analysis

4 > 10 April 2023

1) Co-existing systems parallel efficiency

Page 5: ABC of Teradata System Performance Analysis

5 > 10 April 2023

Co-existing systems parallel efficiencyCalculation & Reconfiguration

Before Reconfiguration:

Unusable Node Capacity

14.23 – 13.25 = 0.98;

Parallel Efficiency = 93.09

After Reconfiguration:

Unusable Node Capacity

14.23 -14.20 = 0.03;

Parallel Efficiency = 99.80

Gain in Co-exiting systems parallel efficiency is the source of the biggest gain in performance.

Page 6: ABC of Teradata System Performance Analysis

6 > 10 April 2023

• 2) CPU Reports

Page 7: ABC of Teradata System Performance Analysis

7 > 10 April 2023

CPU Utilization Chart

Generated with Higa Macro ResPMATotal

Higa Macro ResPmaTotal

Page 8: ABC of Teradata System Performance Analysis

8 > 10 April 2023

Absolute CPU Node Hours available

•ABS Avail Hours: Time when CPU was available to process a job but there was nothing to process•This graph gives a ratio between total CPU available to Max CPU used on daily basis

Higa Macro ResPmaThreeShifts

Page 9: ABC of Teradata System Performance Analysis

9 > 10 April 2023

CPU Utilization: Percent Shift Busy

Shift Time 00-07Mon-Fri

Shift Time 07-17Mon-Fri

Shift Time 17-24Mon-Fri

Shift Time 00-24Sat-Sun

•From previous slide same graph shown in Percent shift busy

Higa Macro ResPmaThreeShifts

Page 10: ABC of Teradata System Performance Analysis

10 > 10 April 2023

OS % of CPU Higa Macro ResPmasec

Extremely degraded system performance

•CPU 100 % Busy•OS% of CPU less than 20%•For more than 30 minutes

In-Efficient Use of CPU

Page 11: ABC of Teradata System Performance Analysis

11 > 10 April 2023

CPU Utilization among Vprocs Example #1 Higa Macro ResSvprCPUs

A normal breakdown of the total CPU utilization among the AMP, PE and node vprocs

Page 12: ABC of Teradata System Performance Analysis

12 > 10 April 2023

CPU Utilization among Vprocs Example #2 Higa Macro ResSvprCPUs

A very high CPU usage by PE. (Non optimized ) TPUMP sessions one of the reasons

PE consuming 40% of Node CPU, works at their max capacity. And have tendency to become bottleneck.

Page 13: ABC of Teradata System Performance Analysis

13 > 10 April 2023

Nodes Parallel Efficiency – Example 1 Higa Macro ResPmaTotal

Extreme case of Node skew

•Node Skew = Max Node CPU – Avg Node CPU

Page 14: ABC of Teradata System Performance Analysis

14 > 10 April 2023

Nodes Parallel Efficiency – Example 2 Higa Macro ResPmaTotal

Node Skew = Max Node CPU – Avg Node CPU

High Parallel Efficiency among Nodes

Page 15: ABC of Teradata System Performance Analysis

15 > 10 April 2023

AMPs Parallel Efficiency – Example 1

High parallel Efficiency is present among AMPs

Higa Macro ResSvprCPUs

Page 16: ABC of Teradata System Performance Analysis

16 > 10 April 2023

AMPs Parallel Efficiency – Example 2 Higa Macro ResSvprCPUs

Parallel Efficiency not good among AMPs at times

Page 17: ABC of Teradata System Performance Analysis

17 > 10 April 2023

AMPs Parallel Efficiency – Example 3 Higa Macro ResSvprCPUs

Extreme case of skew among AMPs

Page 18: ABC of Teradata System Performance Analysis

18 > 10 April 2023

• 3) Memory Reports

Page 19: ABC of Teradata System Performance Analysis

19 > 10 April 2023

Memory Utilization

Average & Minimum Free Memory Available

0

200

400

600

800

1 000

1 200

12

/18

Sa

t 0

0:1

0

12

/20

Mo

n 2

3:1

0

12

/23

Th

u 2

1:5

0

12

/26

Su

n 2

0:3

0

12

/29

We

d 1

9:1

0

01

/01

Sa

t 1

7:5

0

01

/04

Tu

e 1

6:3

0

01

/07

Fri

15

:10

01

/10

Mo

n 1

3:5

0

01

/13

Th

u 1

2:3

0

01

/16

Su

n 1

1:1

0

12

/19

Su

n 1

7:2

0

12

/22

We

d 1

6:0

0

12

/25

Sa

t 1

4:4

0

12

/28

Tu

e 1

3:2

0

12

/31

Fri

12

:00

01

/03

Mo

n 1

0:4

0

01

/06

Th

u 0

9:2

0

01

/09

Su

n 0

8:0

0

01

/12

We

d 0

6:4

0

01

/15

Sa

t 0

5:2

0

12

/18

Sa

t 1

1:1

0

12

/21

Tu

e 1

0:1

0

12

/24

Fri

08

:50

12

/27

Mo

n 0

7:3

0

12

/30

Th

u 0

6:1

0

01

/02

Su

n 0

4:5

0

01

/05

We

d 0

3:3

0

01

/08

Sa

t 0

2:1

0

01

/11

Tu

e 0

0:5

0

01

/13

Th

u 2

3:3

0

01

/16

Su

n 2

2:1

0

Mem Free ------

Mem Free ------

Coexistence View

One of the node groupsis experiencing

low memory condition

• Can FSGcache adjustment help?

Higa Macro ResPma

Page 20: ABC of Teradata System Performance Analysis

20 > 10 April 2023

Memory Utilization

There are low memory or depletions present, as marked with circles. But there frequency of occurrence is not alarming.

•Available Free memory less than 100 MB and 40 MB are termed as “Memory depletion” and “system panic” states respectively.

Higa Macro ResPma

Page 21: ABC of Teradata System Performance Analysis

21 > 10 April 2023

Paging & Memory Allocation Fails

Mem Alloc Fails occurs when FREE MEM reaches to zero.

Negligible Mem Alloc Failures occurred in last month.

Higa Macro ResPma

Page 22: ABC of Teradata System Performance Analysis

22 > 10 April 2023

• 4) Disk IO Reports

Page 23: ABC of Teradata System Performance Analysis

23 > 10 April 2023

DISK IO: Disk % Busy

System is not IO bound: Disk doesn't remain 100% busy.

Higa Macro ResSldvNode

•Disk Busy time is the amount of time in which there is at least one I/O request outstanding

Page 24: ABC of Teradata System Performance Analysis

24 > 10 April 2023

I/O Wait% Example #1

Average Cpu Busy vs. I/O Wait

0

20

40

60

80

100

120

03

/01

We

d 0

0:1

0

03

/01

We

d 0

9:5

0

03

/01

We

d 1

9:3

0

03

/02

Th

u 0

5:1

0

03

/02

Th

u 1

4:5

0

03

/03

Fri

00

:30

03

/03

Fri

10

:10

03

/03

Fri

19

:50

03

/04

Sa

t 0

5:3

0

03

/04

Sa

t 1

5:1

0

03

/05

Su

n 0

0:5

0

03

/05

Su

n 1

0:3

0

03

/05

Su

n 2

0:1

0

03

/06

Mo

n 0

5:5

0

03

/06

Mo

n 1

5:3

0

03

/07

Tu

e 0

1:1

0

03

/07

Tu

e 1

0:5

0

03

/07

Tu

e 2

0:3

0

03

/08

We

d 0

6:1

0

03

/08

We

d 1

5:5

0

03

/09

Th

u 0

1:3

0

03

/09

Th

u 1

1:1

0

03

/09

Th

u 2

0:5

0

03

/10

Fri

06

:30

03

/10

Fri

16

:10

03

/11

Sa

t 0

1:5

0

03

/11

Sa

t 1

1:5

0

03

/11

Sa

t 2

1:3

0

03

/12

Su

n 0

7:1

0

03

/12

Su

n 1

6:5

0

03

/13

Mo

n 0

2:3

0

Chart Type: Stacked Area

Wait % ----

CPU bsy ---

• Average CPU + I/O Wait = System Busy• If Avg CPU + I/O Wait = 100%, desirable ratio is >= 90% avg CPU to >= 10% I/O Wait

Ratio is well within target range

Higa Macro ResPmaTotal

Page 25: ABC of Teradata System Performance Analysis

25 > 10 April 2023

I/O Wait% Example #2

Average Cpu Busy vs. I/O Wait

0

20

40

60

80

100

120

1/9

Fri

0:00

1/9

Fri

8:50

1/9

Fri

17:4

0

1/10

Sat

2:3

0

1/10

Sat

11:

20

1/10

Sat

20:

10

1/11

Sun

6:2

0

1/11

Sun

15:

10

1/12

Mon

0:0

0

1/12

Mon

8:5

0

1/12

Mon

17:

40

1/13

Tue

2:3

0

1/13

Tue

11:

20

1/13

Tue

20:

10

1/14

Wed

5:0

0

1/14

Wed

13:

50

1/14

Wed

22:

40

1/15

Thu

7:3

0

1/15

Thu

16:

20

1/16

Fri

1:10

1/16

Fri

10:0

0

1/16

Fri

18:5

0

1/17

Sat

3:4

0

1/17

Sat

12:

30

1/17

Sat

21:

20

1/18

Sun

6:1

0

1/18

Sun

15:

00

1/19

Mon

0:1

0

1/19

Mon

12:

30

1/19

Mon

21:

30

1/20

Tue

6:2

0

1/20

Tue

15:

10

1/21

Wed

0:0

0

1/21

Wed

8:5

0

1/21

Wed

17:

40

1/22

Thu

2:3

0

1/22

Thu

11:

20

1/22

Thu

20:

10

1/23

Fri

5:00

1/23

Fri

13:5

0

Chart Type: Stacked Area

I/O Wait %

Avg CPU bsy

• Wait I/O should be 10% or less for configuration optimally balanced for power and throughput

#1 40%-60% Wait I/O shown here

Higa Macro ResPmaTotal

Page 26: ABC of Teradata System Performance Analysis

26 > 10 April 2023

Disk Read KByte and Write KByte

0

50 000

100 000

150 000

200 000

250 000

300 00006

/15

Thu

06:

20

06/1

5 T

hu 0

6:20

06/1

5 T

hu 0

6:20

06/1

5 T

hu 0

6:20

06/1

5 T

hu 0

6:20

06/1

5 T

hu 0

6:30

06/1

5 T

hu 0

6:30

06/1

5 T

hu 0

6:30

06/1

5 T

hu 0

6:30

06/1

5 T

hu 0

6:30

06/1

5 T

hu 0

6:40

06/1

5 T

hu 0

6:40

06/1

5 T

hu 0

6:40

06/1

5 T

hu 0

6:40

06/1

5 T

hu 0

6:40

06/1

5 T

hu 0

6:50

06/1

5 T

hu 0

6:50

06/1

5 T

hu 0

6:50

06/1

5 T

hu 0

6:50

06/1

5 T

hu 0

6:50

Chart Type: Stacked Area

Disk WrKB /Sec

Disk RdKB /Sec

Measuring I/O

On the system, differences in throughputwere viewable with disk read/writes Actual rated bandwidth for the configuration

Higa Macro ResPmabyNode

Page 27: ABC of Teradata System Performance Analysis

27 > 10 April 2023

Disk Position Reads, Pre-Reads and Writes

0

2 000 000

4 000 000

6 000 000

8 000 000

10 000 000

12 000 000

09/0

3 S

at 00:0

0

09/0

4 S

un 0

1:1

0

09/0

5 M

on 0

2:2

0

09/0

6 T

ue 0

3:3

0

09/0

7 W

ed 0

4:4

0

09/0

8 T

hu 0

5:5

0

09/0

9 F

ri 0

7:0

0

09/1

0 S

at 08:1

0

09/1

1 S

un 0

9:2

0

09/1

2 M

on 1

0:3

0

09/1

3 T

ue 1

3:0

0

09/1

4 W

ed 1

4:1

0

09/1

5 T

hu 1

5:2

0

09/1

6 F

ri 1

6:5

0

09/1

7 S

at 18:0

0

09/1

8 S

un 1

9:1

0

09/1

9 M

on 2

0:2

0

09/2

0 T

ue 2

1:3

0

09/2

1 W

ed 2

3:3

0

09/2

3 F

ri 0

0:4

0

09/2

4 S

at 01:5

0

09/2

5 S

un 0

3:0

0

09/2

6 M

on 0

4:1

0

09/2

7 T

ue 0

5:2

0

09/2

8 W

ed 0

6:3

0

09/2

9 T

hu 0

7:4

0

09/3

0 F

ri 0

8:5

0

10/0

1 S

at 10:0

0

10/0

2 S

un 1

1:1

0

Chart Type: Stacked Area

Total DB Wrts

Total Pre Rds

Total Position Rds

Total Disk Reads/Writes

Total Pre-Reads are large proportion of total reads, so this system may be a good candidate for raising

Full Cylinder Read slots

Next step is to look at FCR denied cache. If this is high, and there is wait I/O often,

then this system is a candidate for higher FCR slots.

Higa Macro ResPmaTotal

Page 28: ABC of Teradata System Performance Analysis

28 > 10 April 2023

Full Cylinder Reads Example #1 Higa Macro ResFullCylReadTotal

Less FCR requests with high

successful rate

Page 29: ABC of Teradata System Performance Analysis

29 > 10 April 2023

Full Cylinder Reads Example #2 Higa Macro ResFullCylReadTotal

High FCR requests with moderate successful rate

Page 30: ABC of Teradata System Performance Analysis

30 > 10 April 2023

Logical vs. Physical Reads Higa Macro SvprReadTotal

The more that the logical reads exceeds physical reads, the better use of memory as a cache

Page 31: ABC of Teradata System Performance Analysis

31 > 10 April 2023

Mini-cyl-packs Higa Macro ResCylPackTotal

Mini Cyl Pack occurs when there are only ten Free cylinders left on any AMP resulting in degraded system performance.

Page 32: ABC of Teradata System Performance Analysis

32 > 10 April 2023

5) AWT Usage Higa Macro ResSvprQLenAvg

ByVproc

Potential indicator of “FLOW Control” state

Message queue length > 20

Page 33: ABC of Teradata System Performance Analysis

33 > 10 April 2023

6) Buddy Backup Effectiveness Higa Macro ResPmaBkupHour

Total

When there are many complete and few partial blocks, sent to the buddy, then the buddy backup should be turned off

Page 34: ABC of Teradata System Performance Analysis

34 > 10 April 2023

7) Host Utilities Traffic Higa Macro reshostTotalHour

Total read/write traffic per Hour, physical MBytes transferred between Host and Node.

how much data is read or written by utility

Page 35: ABC of Teradata System Performance Analysis

35 > 10 April 2023

• 8) Charting Teradata Manager Reports

Use Teradata Manager to record

Active Session detail data

Generate Custom reports using pivot

tables

Page 36: ABC of Teradata System Performance Analysis

36 > 10 April 2023

Charting Teradata Manager Reports:CPU utilized per User

CPU utilized per User

Page 37: ABC of Teradata System Performance Analysis

37 > 10 April 2023

Charting Teradata Manager Reports:CPU utilization Per Group

CPU utilized per Performance Group

Page 38: ABC of Teradata System Performance Analysis

38 > 10 April 2023

Questions

The only bad questionis the questionnever asked