1e06

8/3/2019 1E06

http://slidepdf.com/reader/full/1e06 1/115

8/3/2019 1E06


2April 7 th 2003Performance Troubleshooting.ppt page 2 © 2003

26 th DECUS Symposium

www.decus.de

Using the right tools toTroubleshooting

Performance Problemson

Tru64 UNIX®

Jan Mark HolzerSenior Member of Technical Staff

[email protected]

Enterprise UNIX® Divison

8/3/2019 1E06




www.decus.de

agendaconcepts

tools to troubleshoot

resources

8/3/2019 1E06


page 4Performance Troubleshooting.ppt © 2003


www.decus.de

Concepts

8/3/2019 1E06




www.decus.de

Concepts

§ follow the 5 P’s rule !!!

8/3/2019 1E06




www.decus.de

Concepts

§ follow the 5 P’s rule !!!

Proper PlanningPrevents

Piss PoorPerformance

26 th DECUSS i

8/3/2019 1E06

http://slidepdf.com/reader/full/1e06 7/115page 7Performance Troubleshooting.ppt © 2003


www.decus.de

a good tool selection method?

A: i = random(0, length(ToolList)-1);

execute ToolList[i];

if does not work, roll back;

else commit;

goto A;

method based on Cary Millsap’s Oracle performance method

26 th DECUSS i

8/3/2019 1E06



www.decus.de

a good tool selection method?

A: i = random(0, length(ToolList)-1);

execute ToolList[i];

if does not work, roll back;

else commit;

goto A;

method based on Cary Millsap’s Oracle performance method

26 th DECUSSymposium

8/3/2019 1E06

http://slidepdf.com/reader/full/1e06 9/115April 7 th 2003Performance Troubleshooting.ppt © 2003


www.decus.de

selecting the right tools

§ use the right tool for the problem – understand the problem

– isolate area of ‘concern’

§ often the obvious or commonly used tool might notbe the right one

§ use lightweight tools to avoid introduction ofadditional overhead


8/3/2019 1E06



www.decus.de

troubleshooting concepts

§ understand Application Characteristics

– how many processes?

– how much IO? What are the patterns (bursts, orover time)?

– how much memory is used?

– what kind of networking resources are needed?

(how many sockets, …)


8/3/2019 1E06



www.decus.de

concepts (cont)

§ determine Performance Requirements

– number of simultaneous accessors

§ mail, nfs, database, telnet

– ideal response time

§ test Fulfillment of Requirements

– benchmark or simulator


8/3/2019 1E06


26 t DECUS Symposium

www.decus.de

concepts (cont)

§ understand capabilities and limitations of targetenvironment

– read product specs – isolate the ‘bottlenecks’ (ie. limiting factors for

performance) in the current configuration

§ DO NOT BELIEVE propaganda material (especially

from our friends west of the border J) – draw/build map of storage

subsystem/configuration

§ can use sys_check output to some degree


8/3/2019 1E06


26 DECUS Symposium

www.decus.de

concepts (cont)

§ read up

– “System Configuration and Tuning Guide”

– application specific guide – whitepapers

– other people’s experiences


8/3/2019 1E06


26 DECUS Symposium

www.decus.de

Toolshttp://www.tru64.org/~jmh/


8/3/2019 1E06


26 DECUS Symposium

www.decus.de

Troubleshooting Guidelines

§ tools for troubleshooting and collecting

– CSC data collection§

sys_check (unicensus) – performance data collection§ syd, top, monitor, ps, collect, vmubc, Performance

Manager, iostat, vmstat, volstat,advfsstat

– other tools§ lsof, netconfig, kdbx scripts, truss, trace

– kernel/application data collection§ kprofile, uprofile, iprobe, DCPI, kdbx,

cda,om,atom,spike,lockinfo


8/3/2019 1E06


y p

www.decus.de

Reconfiguration(cont)

§ to see the maxium value for a given tunable

# sysconfig –Q proc maxusers

§ all subsystems now online as man pages

# man sys_attrs_SubSystemName

§ don’t have to reboot the system for modified tunable parameters toalter system operation.

§ since all parameters are range checked based on the existing tuningand configuration, the “only” harm that can occur is performancedegradation.

§ 2 ways to reconfigure tunable parameters: – the CDE/dx kerneltuner slide bars.

# sysconfig -r <subsystem> <parameter> = <value>

# sysconfig -r vm vm_page_free_target = 1024


8/3/2019 1E06



y p

www.decus.de

hardwareinformation

hwmgr

dsfmgr

emxmgr


8/3/2019 1E06


y p

www.decus.de

emxmgr

§ emxmgr

– Replaced in 5.1B with “hwmgr –view topology”

– Useful to obtain FC topolgy information – Can look at

§ Adapter mappings

§ FC Port Ids

§ Ports logged in


8/3/2019 1E06

http://slidepdf.com/reader/full/1e06 19/115page 19Performance Troubleshooting.ppt © 2003www.decus.de

Hardware Management and multi-pathinginformation

§ V5.x introduced new device naming and nativemulti-pathing

§ Automatic configuration and monitoring§ Today there’s no single utility to view per

path/device statistics

– hwmgr(8) and collect(8) are the best options so far

– Sometimes emxmgr(8) can be useful too


8/3/2019 1E06


hwmgr the Swiss Army Knife

§ To view hardware topology

# hwmgr –view hierarchy

§ To find EMX controllers

# hwmgr –view hierarchy | grep –E “qbb|emx”

or in 5.1B

# hwmgr –view topology

§Display “stale” path information# hwmgr –show scsi –full

– “in kernel” view

# hwmgr –get attr current | egrep “dev_base_name|path_state”


8/3/2019 1E06


hwmgr the Swiss Army Knife

§ Path usage

# hwmgr –get attr current | egrep

“dev_base_name|path_state|path_xfer”§ “Cross RAD” I/O information

# hwmgr –get attr current | egrep\ “dev_base_name|cross_rad|path_xfer”

§Hardware Database “inconsistencies”# hwmgr –show component –inconsistency (-full)


8/3/2019 1E06


hwmgr –get attr current / Cross RAD I/Os


8/3/2019 1E06


23April 7 th 2003Performance Troubleshooting.ppt page 23 © 2003www.decus.de

systeminformation

sys_check


8/3/2019 1E06


page 24Performance Troubleshooting.ppt © 2003www.decus.de

How to run sys_check for Tru64

§ Install kit (part of mandatory OS install V4.0Eand later, or via setld(8) for internet kit)

§ Log in as root, choose one of: ( /usr/sbin in path)

sys_check > file.html ( STD HTML output)

sys_check -perf > file.html

sys_check -all > file.html (Includes security info)

sys_check -frames ( Frames output)§ Typically runs in 10-30 minutes, HTML file is

between .5 - 2MB in size

§View file with browser


8/3/2019 1E06


April 7 th 2003Performance Troubleshooting.ppt © 2003www.decus.de

What does sys_check NOT do?

§ Does not:

– gather every possible system parameter or operating

statistic

– check / warn for every possible configuration oroperational problem

– gather core files, or other binary files needed wheninvestigating system crashes (by default)

– gather information on every possible layered product

(but it does do quite a few)

– mail or otherwise transmit any data off the system.


8/3/2019 1E06



Some components sys_check gathers

§ Hardware (Boot information, firmware revs)

§ OS Software (setld, Patches)

§ Kernel (VM, configuration parameters)

§ File systems (VFS, AdvFS, UFS, NFS, CDFS)§ Storage (disks, tapes, HSZs, SWXCR, LSM,

Prestoserve, Storage Map)

§ Network (TCP/IP, routing, DECnet, LAT)

§

Pathworks/ASDU, Oracle, SAP, Sybase, Informix,Baan, Samba, Netscape, BMC

§ Clusters (TCR Available, Production Server, V5.0)

§ Printers

8/3/2019 1E06


Frames sys_check output( No args)

8/3/2019 1E06



F h k ( N )

8/3/2019 1E06




8/3/2019 1E06



System Monitoring Utilities

Your trial period is over

Please register now

OK


8/3/2019 1E06


31April 7 th 2003Performance Troubleshooting.ppt page 31 © 2003www.decus.de

process

&system

monitoring

syd

monitor

top

vmubc

collect

vmstat

cda/crash


8/3/2019 1E06



syd, top, monitor

§ display of important system statistics

§ sorts process in various ways (CPU used)

– dxproctuner part of the CDE base Sysman suite§ many options for sorting output

§ signals

§ updates


8/3/2019 1E06



syd


8/3/2019 1E06



swapon -s


8/3/2019 1E06



vmubc V5.1


8/3/2019 1E06



collect

§ Home grown tool based on variousbenchmarking experiences ...

– collects everything you ever wanted to sample

– collects to stdout or binary file for later analysis – PerlTK based GUI (collgui)

– cfilt script for custom output (Excel spreadsheetetc...)

– Select subsystems, sample intervals, processes,commands, totals, etc...

– Looking for feedback/input from YOU


8/3/2019 1E06



Collect Dataflow Diagram


8/3/2019 1E06



collect -h


collect sn

8/3/2019 1E06



collect -sn


ll T

8/3/2019 1E06



collect -T


ll d T

8/3/2019 1E06



collect –sd -T


8/3/2019 1E06



collect –sd –Ddsk? -T


8/3/2019 1E06



collect –sd –Ddsk3 -T


8/3/2019 1E06



collect –sp –n10 –S


8/3/2019 1E06



collect –sp –P Csetitathome

26th

DECUS Symposium

ll 7 S

8/3/2019 1E06



collect –scp –n7 -S

26th

DECUS Symposium

collect sc

8/3/2019 1E06



collect -sc

26th

DECUS Symposium

collect sd / BusyTime

8/3/2019 1E06



collect –sd / BusyTime

26th

DECUS Symposium

collect sd / BusyTime

8/3/2019 1E06



collect –sd / BusyTime

26th

DECUS Symposium

8/3/2019 1E06



collgui

26th

DECUS Symposium

collg i

8/3/2019 1E06



§ collgui will then draw both expressions on a graph

collguisupports multiple concurrent graphs

26th

DECUS Symposium

8/3/2019 1E06



vmstat -P

26th

DECUS Symposium

8/3/2019 1E06



vmstat –P(2)

26th

DECUS Symposium

vmstat –R/–r RAD#

8/3/2019 1E06



vmstat –R/–r RAD#

26th

DECUS Symposium

8/3/2019 1E06


57

April 7 th 2003Performance Troubleshooting.ppt page 57 © 2003www.decus.de

filesystemmonitoringvolstat

advfsstat

26th

DECUS Symposium

8/3/2019 1E06



Example: I/O Statistics for a LSM Volume

# volstat -vpsd OPERATIONS BLOCKS AVG TIME(ms)TYP NAME READ WRITE READ WRITE READ WRITEdm dsk1 130395 260827 8382362 16758335 18.8 85.5dm dsk2 130650 261303 8387544 16796853 18.0 78.3dm dsk3 130653 261128 8396046 16776460 18.1 51.8dm dsk4 0 69637 0 905281 0.0 8.1

dm dsk261 130804 260827 8419760 16758335 20.0 99.8dm dsk266 130277 261303 8367152 16796853 18.0 89.8dm dsk267 130594 261128 8378784 16776460 18.4 86.1vol v1 393216 393216 50331648 50331648 22.6 155.6pl v1-01 196609 393216 25165952 50331648 22.2 98.8sd dsk1-01 130395 260827 8382362 16758335 18.8 85.5sd dsk2-01 130650 261303 8387544 16796853 18.0 78.3

sd dsk3-01 130653 261128 8396046 16776460 18.1 51.8pl v1-02 196607 393216 25165696 50331648 23.0 120.0sd dsk261-01 130804 260827 8419760 16758335 20.0 99.8sd dsk266-01 130277 261303 8367152 16796853 18.0 89.8sd dsk267-01 130594 261128 8378784 16776460 18.4 86.1pl v1-03 0 69637 0 905281 0.0 8.1sd dsk4-01 0 69637 0 905281 0.0 8.1

26th

DECUS Symposium

8/3/2019 1E06



AdvFS I/O queues in V4.0x

SynchronousI/O Request

Asynchronous

I/O Request

Blocking Queue

Device

Queue

Lazy Queue

WaitQueue

ReadyQueue

Consolqueue

Disk

Reads

Metadata

26th

DECUS Symposium

8/3/2019 1E06



AdvFS I/O queues in V5.x

SynchronousI/O Request

Asynchronous

I/O Request

Blocking Queue

Flush Queue

UBC Request Queue

Device

Queue

Lazy Queue

WaitQueue

SmoothSync

Queue

ReadyQueue

Consolqueue

Disk

Sync. Writes

(fsync)

VM Requests

Reads

Metadata

26th

DECUS Symposium

d f

8/3/2019 1E06



advfsstat

# advfsstat -i 5 -v 3 test_domain

vol1 vol2

rd wr rg arg wg awg blk ubcr flsh wlz sms rlz con dev rd wr rg arg wg awg blk ubcr flsh wlz sms rlz con dev

190 0 190 256 0 0 2K 0 0 0 0 0 0 0 0 178 0 0 177 254 0 0 2K 2K 2K 0 0 2K

181 0 181 256 0 0 2K 0 0 0 0 0 0 0 0 166 0 0 166 250 0 0 2K 2K 2K 0 0 2K

120 0 120 256 0 0 1K 0 0 0 0 0 0 0 0 206 0 0 204 252 0 0 6K 1K 1K 0 0 3K

rd number of reads on volume blk blocking queue requests on volume

wr number of writes on volume flsh flush queue requests on volume

rg number of consolidated reads on volume ubcr UBC queue requests on volume

arg average number of blocks per consolidated wlz wait lazy queue requests on volume

read on volume sms smooth sync queue requests

wg number of consolidated writes on volume rlz ready lazy queue requests on volume

awg average number of blocks per consolidated write on con consol queue requests on volume

volume dev device queue requests on volume

26th

DECUS Symposium

8/3/2019 1E06



advfsstat - reads and writes

§ rd,wr

– total number of io requests handed to the device(consolidated + nonconsolidated)

§ rg,wg

– number of requests that were consolidated

§ arg,awg – average consolidation size (limited by the transfer size of

the device).

§ failure to consolidate may mean

– disk fragmentation – random access patterns

– thread competition

26th

DECUS Symposium

8/3/2019 1E06



advfsstat – I/O queues (priority order)

§blocking queue – all Reads and AdvFS metadata writes

– high count may mean lots of reads

§ UBC request queue

– VM flush requests – high count may mean memory contention

§ flush queue

– synchronous writes (fsync)

– bursts. Should not be continually high§ wait queue

– hold up metadata (write ahead logging)

– high values may mean a device problem

26th

DECUS Symposium

8/3/2019 1E06



advfsstat – I/O queues (priority order)

§ smooth sync queue

– ages buffers for reuse.

– low counts may mean aging too low, low write activity andvice versa.

§ ready queue

– Buffers are sorted here. – High counts may device problem.

§ consol queue

– Prevents starvation behavior difficult to predict

§ device queue – Size determined by device

– Constant value means I/O bound.

26

th

DECUS Symposium

8/3/2019 1E06



lsof

§ List open files

§ List open sockets

§ List processes owning sockets/files

§ Automatically kill processes etc...

§ fuser(8) light weight tool part of the base operatingsystem


lsof -I :23 :6000 etc…

8/3/2019 1E06



lsof I :23 :6000 etc…


lsof /usr

8/3/2019 1E06



so /us


lslk

8/3/2019 1E06




8/3/2019 1E06



sched_stat

§ scheduler statistics

– per CPU info

– interrupt distribution

– load balancing

– documented in Tru64 5.1B


8/3/2019 1E06



Tru64 V5.1 Distributed Interrupts

cpu user nice system idle widle | scalls intr csw tbsyc

-----+-------------------------------+-----------------------------------------

0 | 53.7 0.0 41.1 0.0 5.1 | 30129 75505 109576 1545

1 | 76.2 0.0 19.1 0.0 4.8 | 27322 0 46620 1395

2 | 75.9 0.0 19.3 0.0 4.8 | 27168 0 46117 1557

3 | 75.4 0.0 19.9 0.0 4.7 | 29494 0 46646 1484

4 | 56.3 0.0 42.0 0.0 1.7 | 28844 85065 100670 1571

5 | 76.8 0.0 21.4 0.0 1.9 | 34241 0 38533 1563

6 | 76.8 0.0 21.6 0.0 1.6 | 34484 0 38261 1573

7 | 76.7 0.0 21.3 0.0 1.9 | 35580 0 38650 1573

8 | 55.7 0.0 40.4 0.0 3.9 | 29341 70907 100808 1559

9 | 75.9 0.0 20.3 0.0 3.8 | 27306 0 39599 1451

10 | 74.0 0.0 22.2 0.0 3.7 | 35458 0 39223 1067

11 | 74.3 0.0 21.9 0.0 3.9 | 34119 0 39825 982

12 | 57.0 0.0 41.8 0.0 1.2 | 36373 91000 101645 157413 | 77.7 0.0 21.1 0.0 1.2 | 49068 0 36888 1572

14 | 78.0 0.0 20.8 0.0 1.2 | 45700 0 36600 1572

15 | 77.1 0.0 21.7 0.0 1.2 | 43264 0 37136 1579


sched stat -u

8/3/2019 1E06



_


sched_stat –u sleep 10

8/3/2019 1E06



_ p


8/3/2019 1E06



Kernel Profiling Tools


8/3/2019 1E06



Profiling Tools

§Used to perform kernel and application levelsurgery

– lockinfo command to troubleshoot “hot” locks

– kprofile to profile kernel

– (D)CPI to profile kernel and application


lockinfo sort misses sleep 20

8/3/2019 1E06



lockinfo –sort=misses sleep 20# lockinfo –sort=misses sleep 20hostname: rmcabs05locktype: MCS Locks

lockmode: 2 (SMP default)processors: 32start time: Sat Dec 9 16:08:27 2000end time:Sat Dec 9 16:08:47 2000command: sleep 20

tries reads trmax misses percent sleeps waitmax waitsummisses seconds seconds

cnode.c_statelock (M)2464194 0 0 1397731 56.7 0 0.00006 19.98978processor.runq.lock (M)

1241862 0 0 6971 0.6 0 0.00002 0.04308BfAccessFreeLock (M)

16506 0 0 204 1.2 0 0.00001 0.00103lock.l_lock (M)

2483207 0 0 114 0.0 0 0.00002 0.00113msgQT.mutex (M)

11105 0 0 82 0.7 0 0.00001 0.00041pag.runq.lock (M)

3950 0 0 60 1.5 0 0.00001 0.00028kmembuckets.kb_lock (M)

58781 0 0 29 0.0 0 0.00001 0.00012vn_free_lock (M)

6303 0 0 22 0.3 0 0.00001 0.00012


lockinfo –class=cnode c statelock sleep 20

8/3/2019 1E06



lockinfo –class=cnode.c_statelock sleep 20

# lockinfo –class=cnode.c_statelock sleep 20

hostname: rmcabs05locktype: MCS Lockslockmode: 2 (SMP default)processors: 32start time: Sat Dec 9 16:16:21 2000end time:Sat Dec 9 16:16:41 2000command: sleep 20

Locks asserted by PC for lock class: cnode.c_statelock

count misses remote caller : line # return : line #------------------------------------------------------------------------------------------------------------------------------2119093 1669769 449307 cfsspec_write: 8738 vn_write: 1372

1053 0 912 cfsspec_read: 8581 vn_read: 1242700 0 473 cfspol_touchfile: 392 cfstok_req_one: 1441

388 0 224 cfs_pfscacheread: 1852 cfs_read: 956356 0 192 cfs_read: 938 vn_read: 1242262 0 124 cfs_access: 3469 cfs_lookup: 4430232 0 102 cfs_inactive: 4037 vrele: 2603

73 0 53 cfspol_touchfile: 392 cfs_getattr: 272160 0 0 cfs_access: 3570 cfs_lookup: 443052 0 32 cfs_getattr: 2705 vn_stat: 1411


lockinfo sleep 10

8/3/2019 1E06




lockinfo –class=fsContext_mutexsleep 10

8/3/2019 1E06




8/3/2019 1E06



kprofile

§ Enabling kernel profiling – Load driver dynamically, must be unloaded BEFORE using

DCPI# sysconfig –c pfm

§ To start profiling use# kprofile –each -k /vmunix

– to profile individual commands just add the command to thecommand line

# kprofile –k /vmunix rm

§ Creates a kmon.out in your current directory, – Analyze kprofile data using prof(8)# prof /vmunix kmon.out


8/3/2019 1E06



DCPI

§ Continuous Profiling Infrastructure

– designed for continuous use on productionsystems

– intended for programmers and optimization tools

– based on statistical sampling

§ Advantages

– transparent, complete, efficient

– produces accurate fine-grained information

§ execution frequencies

§ stall times & explanations


8/3/2019 1E06



User Space Daemon

§ User-Space Daemon

– extracts raw samples from driver

– associates samples with load-files

– updates disk-based profiles for load-files§ Finding Load-Files from <PID, PC>

– exec hook for statically linked load-files

§ Profiles

– text header + compact binary samples


8/3/2019 1E06



Performance of Data Collection

§ Time

– 1-3% total overhead for most workloads

– often less than variation from run to run

§ Space

– 512 KB kernel memory per processor

– 2-10 MB resident for daemon

– 10 MB disk after one month of profiling onheavily used timeshared 4-processor

machine§ Non-intrusive enough to be run for many

hours on production systems, e.g.


8/3/2019 1E06



Example: Getting the Big Picture

Total samples for event type cycles = 6095201

cycles % cum% load file

2257103 37.03% 37.03% /usr/shlib/X11/lib_dec_ffb_ev5.so1658462 27.21% 64.24% /vmunix928318 15.23% 79.47% /usr/shlib/X11/libmi.so

650299 10.67% 90.14% /usr/shlib/X11/libos.so

cycles % cum% procedure load file

2064143 33.87% 33.87% ffb8ZeroPolyArc /usr/shlib/X11/lib_dec_ffb_ev5.so517464 8.49% 42.35% ReadRequestFromClient /usr/shlib/X11/libos.so305072 5.01% 47.36% miCreateETandAET /usr/shlib/X11/libmi.so

271158 4.45% 51.81% miZeroArcSetup /usr/shlib/X11/libmi.so245450 4.03% 55.84% bcopy /vmunix209835 3.44% 59.28% Dispatch /usr/shlib/X11/libdix.so186413 3.06% 62.34% ffb8FillPolygon /usr/shlib/X11/lib_dec_ffb_ev5.so170723 2.80% 65.14% in_checksum /vmunix161326 2.65% 67.78% miInsertEdgeInET /usr/shlib/X11/libmi.so133768 2.19% 69.98% miX1Y1X2Y2InRegion /usr/shlib/X11/libmi.so


8/3/2019 1E06



DCPI Install/Scan phase

§ Grab DCPI from public website and install using “INSTALL” script

http://www.tru64unix.compaq.com/dcpi

§ Can actually create multiple scans for different directories

# dcpiscan –R /YourLocalDirectoryTree > local.map

# dcpiscan [-R] $ORACLE_HOME/bin > oracle.map§ Start data collection: (db is the directory you created for the data

collection)

# dcpid –m local.map -m oracle.map db

§ Get a new epoch ( this means that the collection is stopped and

restarted to a new set of collection data)# dcpiepoch

§ Stop the collection:

# dcpiquit


8/3/2019 1E06



DCPI - Analyze the data

# dcpiprof –iColumn Total Period (for events)------ ----- ------cycles 230954 63488imiss 179284 4096

The numbers given below are the number of samples for eachlisted event type or, for the ratio of two event types, theratio of the number of samples for the two event types.===========================================================cycles % cum% imiss % image224842 97.35% 97.35% 178556 99.59% /vmunix

3055 1.32% 98.68% 90 0.05% /var/subsys/lat.mod

1884 0.82% 99.49% 455 0.25% /oracle/8.1.7/bin/oracle234 0.10% 99.59% 60 0.03% /bin/dcpid202 0.09% 99.68% 4 0.00% /oracle/8.1.7/bin/ogms176 0.08% 99.76% 15 0.01% /usr/local/bin/top147 0.06% 99.82% 32 0. 02% /usr/shlib/libc.so


DCPI

8/3/2019 1E06




# dcpiprof /vmunix

Column Total Period (for events)------ ----- ------cycles 224842 63488imiss 178556 4096

The numbers given below are the number of samples for eachlisted event type or, for the ratio of two event types, theratio of the number of samples for the two event types.

===========================================================cycles % cum% imiss % procedure image125138 55.66% 55.66% 67659 37.89% idle_thread /vmunix51395 22.86% 78.51% 107879 60.42% gh_zero_memor /vmunix36844 16.39% 94.90% 77 0.04% vm_page_teste /vmunix

794 0.35% 95.25% 7 0.00% pmap_zero_page /vmunix663 0.29% 95.55% 71 0.04% table /vmunix660 0.29% 95.84% 10 0.01% reg_save /vmunix

441 0.20% 96.04% 5 0.00% find_bfap /vmunix426 0.19% 96.23% 32 0.02% nofault_bcopy /vmunix341 0.15% 96.38% 114 0.06% syscall /vmunix312 0.14% 96.52% 165 0.09% hardclock /vmunix268 0.12% 96.64% 70 0.04% _ Xsyscall /vmunix


DCPI

8/3/2019 1E06




# dcpiprof -m /map3.local /oracle/8.1.7/bin/oracle

Column Total Period (for events)------ ----- ------cycles 1884 63488imiss 455 4096

The numbers given below are the number of samples for eachlisted event type or, for the ratio of two event types, theratio of the number of samples for the two event types.===========================================================

cycles % cum% imiss % procedure image244 12.95% 12.95% 38 8.35% _OtsZero /oracle/8.1.7/bin/oracle136 7.22% 20.17% 14 3.08% kslwte /oracle/8.1.7/bin/oracle125 6.63% 26.80% 41 9.01% mux_wait /oracle/8.1.7/bin/oracle97 5.15% 31.95% 27 5.93% kslgetl /oracle/8.1.7/bin/oracle96 5.10% 37.05% 29 6.37% ksliwat /oracle/8.1.7/bin/oracle78 4.14% 41.19% 38 8.35% skgxpwait /oracle/8.1.7/bin/oracle77 4.09% 45.28% 11 2.42% kclpbi /oracle/8.1.7/bin/oracle76 4.03% 49.31% 12 2.64% ksbcti /oracle/8.1.7/bin/oracle55 2.92% 52.23% 14 3.08% skgxpte /oracle/8.1.7/bin/oracle51 2.71% 54.94% 3 0.66% wire_need_send /oracle/8.1.7/bin/oracle47 2.49% 57.43% 22 4.84% ksarcv /oracle/8.1.7/bin/oracle46 2.44% 59.87% 6 1.32% kjatwal /oracle/8.1.7/bin/oracle40 2.12% 62.00% 23 5.05% kslfre /oracle/8.1.7/bin/oracle40 2.12% 64.12% 9 1.98% ksimpoll /oracle/8.1.7/bin/oracle37 1.96% 66.08% 15 3.30% ksbabs /oracle/8.1.7/bin/oracle


8/3/2019 1E06



Phoenix

§Graphical User Interface to DCPI

– Makes analysis much easier

– Installed using setld(8)

– Run using phoenix command

– Report Generation

– Selection of events to profile using the GUI


M i S

8/3/2019 1E06



Main Screen


E h

8/3/2019 1E06



Epoch


E h D t il d

8/3/2019 1E06



Epoch Detailed


E h i

8/3/2019 1E06



Epoch vmunix


8/3/2019 1E06



Phoenix Example


8/3/2019 1E06


April 7 th 2003Performance Troubleshooting.ppt page 94 © 2003www.decus.de

recommended

Tru64 UNIX® settings

for ORACLE®

vm

ipc

procrt

vfs

advfsrdg


8/3/2019 1E06



“vm” subsystem Tru64 UNIX® settings

Virtual Memory Subsystem

– new_wire_method

§ Should be set to ‘0’ on systems where

§ ORACLE8 or ORACLE 8i/9i installed

§ DirectIO used for filesystem I/O

§ SSM memory is in use (ssm_threshold at default)

– ubc_maxpercent

§ Should be set between 35% - 80%

§ Usual rule of thumb would be somewhere between 50%-75%

§ Setting it too low may cause contention in the UBC

§ Setting it too high might cause memory being “wasted” to UBC


8/3/2019 1E06



“vm” Subsystem Tru64 SettingsVirtual Memory Subsystem

– vm_ubcseqstartpercent

§ Should be left at default in V5.1 (50%)

§ Most likely being modified in V4.0x stream

§ Algorithm changed in V5.1 to % of ubc_maxpercent

§ In V4.0x stream % of managed pages

– vm_swap_eager

§ Can be set to ‘0’ or ‘1

§ Database server can potentially be run with ‘lazy’ swap

– rad_gh_regions

§ Should be used on GS80/160/320 system for optimal performance

§ Used to enable the use of “Large Pages” (4MB)

§ Less tbmiss’es


8/3/2019 1E06



“ipc” Subsystem Tru64 Settings

Inter Process Communication Subsystem

– shm_min§ Should be left at default value

§ 9i Release Notes might be misleading

§ shm_min instead of shm_mni – shm_max

§ Should be set to 2GB – 8MB (2139095040)

§ V5.1 allows larger value but not proven/tested

– ssm_threshold

§ If rad_gh_regions or gh_chunks in use should be set to ‘0’

§ Enables/Disable use of SSM (Segmented Shared Memory)Memory

§ Default is SSM memory enabled (ie. set to 8MB)


8/3/2019 1E06



“ipc” Subsystem Tru64 Settings

Inter Process Communication Subsystem

– shm_mni§ Should be set to 256

§ Maximum number of shared memory regions which can active at

any given time§ If large SGA/Share Memory usage may want to increase to 1024

– shm_seg

§ Should be set to 128

§ Number of shared memory regions a single process can be

attached to§ For large scale environment may want to increase to 256 or 512


8/3/2019 1E06



“proc” Subsystem Tru64 Settings

Process Subsystem – (max_)per_proc_stack_size§ <= 512MB (actual limit is 3GB)§ Do not set value to >3GB , might cause SGA/stack corruption

– (max_)per_process_data_size§ <= available physical memory (~75% of PHYSMEM)§ Be careful to not over commit memory

– (max_)per_process_address_size§ <= available physical memory (~75% of PHYSMEM)§

Be careful to not over commit memory – As of ORACLE 9i the {max_}per_proc_’mumble’ parametersshould be at least set to the value of pga_aggregate_target


8/3/2019 1E06


page100

Performance Troubleshooting.ppt © 2003www.decus.de

“inet” Subsystem Tru64 Settings

Internet Subsystem

– udp_sendspace


§ Important for ORACLE/OPS operations using UDP as the IPC

protocol – udp_rcvspace


§ Important for ORACLE/OPS operations using UDP as the IPCprotocol

http://www.tru64unix.compaq.com/docs/best_practices/BP_INTUNING/TITLE.HTM

http://www.tru64unix.compaq.com/docs/best_practices/BP_GIGABIT/TITLE.HTM


8/3/2019 1E06


page101


“rt” Subsystem Tru64 Settings

Realtime Subsystem

– aio_task_max_num

§ Should be set to at least 2048

§ Only used if DirectIO/AIO is in use

§ db_block_size is a good starting point – or multiples of it (4096,8192,etc…)

§ Be careful in setting this parameter as you might cause

I/O bottlenecks


8/3/2019 1E06


page102


“vfs” Subsystem Tru64 Settings

Virtual Filesystem Subsystem

– fifo_do_adaptive


§ Can provide significant performance increase (+20%)

§ Important tuning feature if BEQ protocol is used


8/3/2019 1E06


page103


“advfs” Subsystem Tru64 Settings

Advanced Filesystem Subsystem

– AdvfsSync_MmapPages


§ Can provide significant performance increase

§ If read-only mmap()’d files are in use almost a must

§ Avoids premature flushing of mmap()’d pages


8/3/2019 1E06


page104


“rdg” Subsystem Tru64 Settings

Reliable Datagram Subsystem

– max_objs

§ Should be set to at least 5 x the # of Oracle processes pernode

§ At least to the larger of 10240or§ The number of Oracle processes multiplied by 70.

– msg_size

§ Should be set to or greater than the maximum value of the

DB_BLOCK_SIZE parameter for the database.§ Recommended value of 32768 because Oracle 9i supports

different block sizes for each tablespace.


8/3/2019 1E06


page105


“rdg” Subsystem Tru64 Settings

Reliable Datagram Subsystem (cont)

– max_async_req§ Should be set to at least 100

§ Note: A value of 256 might provide better performance. – max_sessions§ Should be at least the number of Oracle processes

plus 2

– rdg_max_auto_msg_wires§ Must be set to 0


8/3/2019 1E06


page106


m e th o d o l o g i e s

Come on! It can‘t gowrong every time...


resources (documentation)

8/3/2019 1E06



resources (documentation)

§ documentation – Tru64 V5.1 Documentation

§ http://www.tru64unix.compaq.com/faqs/publications/pub_page/V51_DOCS/V51_DOCLIST.HTM

– TruCluster V5.1 Documentation

§ http://www.tru64unix.compaq.com/faqs/publications/cluster_doc/cluster_51/TCR51_DOC.HTM

– NUMA Overview

§ http://www.tru64unix.compaq.com/faqs/publications/base_do

c/DOCUMENTATION/V51_HTML/NUMA/TITLE.HTM


Resources (Documentation)

8/3/2019 1E06



Resources (Documentation)

§ Documentation – Tru64/TruCluster Best Practice Documentation

§ http://www.tru64unix.compaq.com/faqs/publications/best_practices/


Online Resources

8/3/2019 1E06


page109


Online Resources

§ AdvFS Direct IO – Tru64 UNIX 5.1 doc set – http://www.tru64unix.compaq.com/faqs/publication

s/pub_page/doc_list.html

– Direct IO (Synch): “Programmer's Guide” section

10.2.2

– Direct IO (Asynch): “Realtime Programmer'sGuide” section 7.2


Online Resources (cont)

8/3/2019 1E06


page110



§CFS Direct IO – TruCluster Server 5.1

– http://www.tru64unix.compaq.com/faqs/publications/cluster_doc/cluster_51/TCR51_DOC.H

TM – Technical Overview section 2.2


Online Resources

8/3/2019 1E06


page111


Online Resources

§ Tru64 UNIX NUMA Technical Overview – http://www.tru64unix.compaq.com/faqs/publications/b

ase_doc/DOCUMENTATION/V51_HTML/NUMA/TITLE.H

TM

§

Tru64 UNIX V5.1 Base Operating System Documentation – http://www.tru64unix.compaq.com/faqs/publications/pub_page/V51_DOCS/V51_DOCLIST.HTM

§ TruCluster V5.1 Documentation

– http://www.tru64unix.compaq.com/faqs/publications/cluster_doc/cluster_51/TCR51_DOC.HTM



8/3/2019 1E06


page112



§TruCluster ORACLE Best Practice Documentations

§ Configuring a single instance ORACLE 8i installation onTruCluster V5.1

– http://www.tru64unix.compaq.com/faqs/publications/best_practices/BP_TCR_ORA_SS/TITLE.HTM

§ Configuring an ORACLE/OPS 8i installation onTruCluster V5.1

– http://www.tru64unix.compaq.com/faqs/publications/b

est_practices/BP_TCR_ORA_OPS817/TITLE.HTM



8/3/2019 1E06


page113



§ORACLE 8i Website – http://www.oracle.com/ip/deploy/database/8i/index.htm

l?ee.html

§ORACLE Technet 8i Website

– http://technet.oracle.com/products/oracle8i/

§ORACLE Magazine – http://www.oramag.com

§ORACLE Performance Website (like sys_check) – http://www.oraperf.com


Resources (Tools)

8/3/2019 1E06



Resources (Tools)

§tools repository – http://janix.zk3.dec.com/pub/tools/

– http://www.tru64.org/~jmh/

§wildfire tools

– http://janix.zk3.dec.com/pub/tools/wf_tools/

§ Brad Nichols’ tools page

– http://www.zk3.dec.com/~bnichols/

8/3/2019 1E06


1e06

Documents