rac tuning sig 2004

8/3/2019 Rac Tuning Sig 2004

1/160

A TUSC Presentation 1

Performance Monitoring &

Tuning for RACRich Niemiec, TUSC

Special Thanks: Janet Bacon, Mike Ault, Madhu Tumma, Janet Dahmen,

Randy Swanson, Rick Stark , Sohan DeMel Erik Peterson and Kirk McGowan

Oracle RAC SIG 2004


2/160


Audience Knowledge

Goals

Overview of RAC & RAC Tuning

Target RAC tips that are most useful

Non-Goals

Learn ALL aspects of RAC


3/160


Overview

RAC Overview

Tuning the RAC Cluster Interconnect

Monitoring the RAC Workload

Monitoring RAC specific contention

Whats new in 10g

Summary


4/160

Overview of Oracle9iRAC

Many instances of Oracle running on many nodes

All instances share a single physical database

and have common data & control files

Each instance has its own log files and rollbacksegments (UNDO Tablespace)

All instances can simultaneously executetransactions against the single database

Caches are synchronized using Oracles Global

Cache Management technology (Cache Fusion)


5/160

Oracle9i Database Clusters

Start small, grow incrementally

Scalable AND highly available

NO downtime to add servers and disk


6/160


Protect from SERVER failures

Server 1Instance A

Server 2Instance A

Real Application Clusters

SERVER failure - your database remains available

DatabaseA


7/160A TUSC Presentation 7

1 Node 2 Nodes 4 Nodes 5 Nodes 6 Nodes0

1,000

2,000

3,000

4,000

5,000

6,000

7,000

1 Node 2 Nodes 4 Nodes 5 Nodes 6 Nodes

# Users

2,296*

4,368*

Oracle11i E-Business Suite Benchmark

E-Business Suite Scalabilitywith Oracle9iRAC

Running on HP Computers *Audited

84%Scalability

1,288

5,433

6,496



Analysis of Performance Issues

Normal database Tuning and Monitoring

RAC Cluster Interconnect Performance

Monitoring Workload Monitoring RAC specific contention



Normal Database Tuning and Monitoring

Prior to tuning RAC specific operations, each

instance should be tuned separately.

APPLICATION Tuning

DATABASE Tuning

OS Tuning

You can begin tuning RAC




Your primary tuning efforts after tuning each

instance individually should focus on the processes

that communicate through the cluster interconnect.

Global Services Directory Processes

GESGlobal Enqueue Services

GCSGlobal Cache Services


11/160



Global Cache Services (GCS) Waits Indicates how efficiently data is being transferred over the cluster

interconnect.

The critical RAC related waits are:

global cache busy A wait event that occurs whenever a session hasto wait for an ongoing operation on the resource

to complete.

buffer busy global cache A wait event that is signaled when a process hasto wait for a block to become available because

another process is obtaining a resource for this

block.

buffer busy global CR Waits on a consistent read via the global cache.


12/160



How:

1) Query V$SESSION_WAIT to determine whether

or not any sessions are experiencing RAC relatedwaits.

2) Identify the objects that are causing contention for

these sessions.

3) Modify the object to reduce contention.


13/160



1) Query v$session_wait to determine whether or not any

sessions are experiencing RAC related waits.

SELECT inst_id,

event,

p1 FILE_NUMBER,

p2 BLOCK_NUMBER,

WAIT_TIME

FROM v$session_waitWHERE event in ('buffer busy global cr',

'global cache busy',

'buffer busy global cache');


14/160



The output from this query should look something like this:

INST_ID EVENT FILE_NUMBER BLOCK_NUMBER WAIT_TIME

------- ----------------------------------- ----------- ------------ ----------

1 global cache busy 9 150 15

2 global cache busy 9 150 10

The block number and file numberwill indicate the object the

requesting instance is waiting for.


15/160



2) Identify objects that are causing contention for these

sessions by identifying the object that corresponds to the

file and block for each file_number/block_number

combination returned:

SELECT owner,

segment_name,

segment_type

FROM dba_extents

WHERE file_id = 9

AND 150 between block_id AND block_id+blocks-1;


16/160



The output will be similar to:

OWNER SEGMENT_NAME SEGMENT_TYPE

---------- ---------------------------- ---------------SYSTEM MOD_TEST_IND INDEX


17/160



3) Modify the object to reduce the chances for

application contention.

Reduce the number of rows per block

Adjust the block size to a smaller block sizeModify INITRANS and FREELISTS


18/160



CR Block Transfer Time

Block contention can be measured by using block

transfer time, calculated by:

global cache cr block receive timeglobal cache cr blocks received

Accumulated round-trip time for allrequests for consistent read blocks.

Total number of consistent read blockssuccessfully received from another instance.


19/160




Use a self-join query on GV$SYSSTAT to compute this ratio:

COLUMN "AVG RECEIVE TIME (ms)" FORMAT 9999999.9

COLUMN inst_id format 9999

PROMPT GCS CR BLOCKSSELECT b1.inst_id, b2.value "RECEIVED",

b1.value "RECEIVE TIME",

((b1.value / b2.value) * 10) "AVG RECEIVE TIME (ms)"

FROM gv$sysstat b1, gv$sysstat b2

WHERE b1.name = 'global cache cr block receive time'

AND b2.name = 'global cache cr blocks received'AND b1.inst_id = b2.inst_id;

INST_ID RECEIVED RECEIVE TIME AVG RECEIVE TIME (ms)

------- ---------- ------------ ---------------------

1 2791 3287 11.8

2 3760 7482 19.9


20/160




Problem Indicators:

High Transfer Time

One node showing excessive transfer time

Use OS commands to verify cluster interconnects

are functioning correctly.


21/160



CR Block Service Time

Measure the latency of service times.

Service time is comprised of consistent read build

time, log flush time, and send time.

global cache cr blocks

served

Number of requests for a CR block served by

LMS

global cache cr block

build time

The time that the LMS process requires to

create a CR block on the holding instance.


flush time

Time waited for a log flush when a CR request

is served (part of the serve time)


send time

Time required by LMS to initiate a send of the

CR block.


22/160




Query GV$SYSSTAT to determine average service times byinstance:

SELECT a.inst_id "Instance",

(a.value+b.value+c.value)/decode(d.value,0,1, d.value) "LMS ServiceTime"

FROM gv$sysstat A,

gv$sysstat B,

gv$sysstat C,

gv$sysstat D

WHERE A.name = 'global cache cr block build time'

AND B.name = 'global cache cr block flush time'

AND C.name = 'global cache cr block send time'

AND D.name = 'global cache cr blocks served'

AND B.inst_id = A.inst_id

AND C.inst_id = A.inst_id

AND D.inst_id = A.inst_id

ORDERBY a.inst_id;


23/160




Result:

Instance LMS Service Time

--------- ----------------

1 1.07933923

2 .636687318

Instance 2 is on theslower node.

Instance 1 is on afaster node and servesblocks faster.

The service time TO

Instance 2 is shorter!


24/160




Query GV$SYSSTAT to drill-down into service time for individualcomponents:

SELECT A.inst_id "Instance", (A.value/D.value) "Consistent Read Build",

(B.value/D.value) "Log Flush Wait",(C.value/D.value) "Send Time

FROM GV$SYSSTAT A, GV$SYSSTAT B,

GV$SYSSTAT C, GV$SYSSTAT DWHERE A.name = 'global cache cr block build time'

AND B.name = 'global cache cr block flush time'

AND C.name = 'global cache cr block send time'

AND D.name = 'global cache cr blocks served'

AND B.inst_id=a.inst_id

AND C.inst_id=a.inst_id

AND D.inst_id=a.inst_id

ORDERBY A.inst_id;

Instance Consistent Read Build Log Flush Wait Send Time

--------- --------------------- -------------- ----------

1 .00737234 1.05059755 .02203942

2 .04645529 .51214820 .07844674


25/160



OS Troubleshooting Commands

Monitor cluster interconnects using OS commands

to find:

Large number of processes in the run state waiting for cpu or

scheduling

Platform specific OS parameter settings that affect IPC buffering or

process scheduling

Slow, busy, faulty interconnects. Look for dropped packets,retransmits, CRC errors.

Ensure you have a private network

Ensure inter-instance traffic is not routed through a public network


26/160



OS Troubleshooting Commands

Useful OS commands in a Sun Solaris environment

to aid in your research are:

$ netstat l

$ netstat s

$ sar c

$ sar q

$ vmstat


27/160



Global Performance Views for RAC

Global Dynamic Performance view names for Real

Application Clusters are prefixed with GV$.

GV$SYSSTAT, GV$DML_MISC, GV$SYSTEM_EVENT,

GV$SESSION_WAIT, GV$SYSTEM_EVENT contain

numerous statistics that are of interest to the DBA when

tuning RAC.

The CLASS column of GV$SYSSTAT tells you the type of

statistic. RAC related statistics are in classes 8, 32 and 40.


28/160



Your Analysis Toolkit

RACDIAG.SQL is an Oracle-provided script that can also

help with troubleshooting.

RACDIAG.SQL can be downloaded from Metalink or the

TUSC website,www.tusc.com.

STATSPACK combined with the queries illustrated in this

presentation will provide you with the tools you need to

effectively address RAC system performance issues.
http://www.tusc.com/http://www.tusc.com/


29/160



Undesirable Statistics

As mentioned, the view GV$SYSSTAT will contain

statistics that indicate the performance of your RAC

system.

The following statistics should always be as near to

zero as possible:

global cache blocks

lost

Block losses during transfers. May indicate

network problems.

global cache blocks

corrupt

Blocks that were corrupted during transfer.

High values indicate an IPC, network, or

hardware problem.


30/160




SELECT A.VALUE "GC BLOCKS LOST 1",

B.VALUE "GC BLOCKS CORRUPT 1",

C.VALUE "GC BLOCKS LOST 2",

D.VALUE "GC BLOCKS CORRUPT 2"

FROM GV$SYSSTAT A, GV$SYSSTAT B, GV$SYSSTAT C, GV$SYSSTAT D

WHERE A.INST_ID=1 AND A.NAME='global cache blocks lost'

AND B.INST_ID=1 AND B.NAME='global cache blocks corrupt'

AND C.INST_ID=2 AND C.NAME='global cache blocks lost'

AND D.INST_ID=2 AND D.NAME='global cache blocks corrupt'

GC BLOCKS LOST 1 GC BLOCKS CORRUPT 1 GC BLOCKS LOST 2 GC BLOCKS CORRUPT 2

---------------- ------------------- ---------------- -------------------

0 0 652 0Instance 2 isexperiencing some lostblocks.


31/160




Take a closer look to see what is causing the lost blocks:

SELECT A.INST_ID "INSTANCE", A.VALUE "GC BLOCKS LOST",

B.VALUE "GC CUR BLOCKS SERVED",

C.VALUE "GC CR BLOCKS SERVED",A.VALUE/(B.VALUE+C.VALUE) RATIO

FROM GV$SYSSTAT A, GV$SYSSTAT B, GV$SYSSTAT C

WHERE A.NAME='global cache blocks lost' AND

B.NAME='global cache current blocks served' AND

C.NAME='global cache cr blocks served' and

B.INST_ID=a.inst_id AND

C.INST_ID = a.inst_id;

Instance gc blocks lost gc cur blocks served gc cr blocks served RATIO

---------- -------------- -------------------- ------------------- ----------

1 0 3923 2734 0

2 652 3008 4380 .088251218

In this case the databaseInstance 1 takes 22 seconds

to perform a series of tests,

Instance 2 takes 25 minutes.

T i h RAC Cl I


32/160




The TCP receive and send buffers on Instance 2

were set at 64K

This is a 8k block size instance with adb_file_multiblock_read_count of 16. This was

causing excessive network traffic since the system

was using full table scans resulting in a read of

128K. In addition the actual TCP buffer area was set to a

small number.

T i h RAC Cl I


33/160



Current Block Transfer Statistics

In addition to monitoring consistent read blocks,

we also need to be concerned with processing

current mode blocks.

Calculate the average receive time for current

mode blocks:

Global cache current block receive time

Global cache current blocks received

Accumulated round trip time for allrequests for current blocks

Number of current blocks received from the holdinginstance over the interconnect

T i h RAC Cl I


34/160



Current Block Transfer Statistics

Query the GV$SYSSTAT view to obtain this ratio for eachinstance:

COLUMN "AVG RECEIVE TIME (ms)" format 9999999.9

COLUMN inst_id FORMAT 9999

PROMPT GCS CURRENT BLOCKSSELECT b1.inst_id,

b2.value "RECEIVED",

b1.value "RECEIVE TIME",

((b1.value / b2.value) * 10) "AVG RECEIVE TIME (ms)"

FROM gv$sysstat b1, gv$sysstat b2

WHERE b1.name = 'global cache current block receive time'

AND b2.name = 'global cache current blocks received'

AND b1.inst_id = b2.inst_id;

INST_ID RECEIVED RECEIVE TIME AVG RECEIVE TIME (ms)

------ ---------- ------------ ---------------------

1 22694 68999 30.4

2 23931 42090 17.6

T i th RAC Cl t I t t


35/160



Current Block Service Time Statistics

Service time for current blocks is comprised of pin time, log

flush time, and send time.

global cache current

blocks served

The number of current blocks shipped to the

requesting instance over the interconnect.

global cache block pin

time

The time it takes to pin the current block before

shipping it to the requesting instance. Pinning is

necessary to disallow changes to the block while it

is prepared to be shipped to another instance.

global cache block flushtime

The time it takes to flush the changes to a block todisk (forced log flush), before the block is shipped

to the requesting instance.

global cache block send

time

The time it takes to send the current block to the

requesting instance over the interconnect.



36/160




To calculate current block service time, queryGV$SYSSTAT:


(a.value+b.value+c.value)/d.value "Current Blk Service Time"FROM GV$SYSSTAT A,

GV$SYSSTAT B,

GV$SYSSTAT C,

GV$SYSSTAT D

WHERE A.name = 'global cache current block pin time'

AND B.name = 'global cache current block flush time'

AND C.name = 'global cache current block send time'

AND D.name = 'global cache current blocks served'AND B.inst_id = A.inst_id

AND C.inst_id = A.inst_id

AND D.inst_id = A.inst_id

ORDER

BY a.inst_id;



37/160




The output from the query may look like this:

Instance Current Blk Service Time

--------- ------------------------

1 1.18461603

2 1.63126376

Instance 2 requires38% more time to

service current blocksthan Instance 1.

Drill-down to service time components will help identify the

cause of the problem.



38/160




SELECT A.inst_id "Instance",

(A.value/D.value) "Current Block Pin",

(B.value/D.value) "Log Flush Wait",

(C.value/D.value) "Send Time"

FROM GV$SYSSTAT A,

GV$SYSSTAT B,

GV$SYSSTAT C,GV$SYSSTAT D

WHERE A.name = 'global cache current block build time'

AND B.name = 'global cache current block flush time'

AND C.name = 'global cache current block send time'

AND D.name = 'global cache current blocks served'

AND B.inst_id=a.inst_id AND

AND C.inst_id=a.inst_id AND

AND D.inst_id=a.inst_idORDER

BY A.inst_id;

Instance Current Block Pin Log Flush Wait Send Time

--------- ----------------- -------------- ----------

1 .69366887 .472058762 .018196236

2 1.07740715 .480549199 .072346418

High pin times could indicate

problems at the IO interface level.



39/160



Global Cache Convert and Get Times

A final set of statistics can be useful in identifying

RAC interconnect performance issues.

global cache converttime

The accumulated time that all sessions require to

perform global conversion on GCS resources

global cache converts Resource converts on buffer cache blocks.Incremented whenever GCS resources are

converted from Null to Exclusive, shared to

Exclusive, or Null to Shared.

global cache get time The accumulated time of all sessions needed toopen a GCS resource for a local buffer.

Global cache gets The number of buffer gets that result in opening anew resource with the GCS.

GCS: Global CacheServices. Aprocess that

communicatesthrough the cluster

interconnect



40/160




To measure these times, query the GV$SYSSTAT view:SELECT A.inst_id "Instance", A.value/B.value "Avg Cache Conv. Time",

C.value/D.value "Avg Cache Get Time", E.value "GC Convert Timeouts"


GV$SYSSTAT C, GV$SYSSTAT D,

GV$SYSSTAT

WHERE A.name = 'global cache convert time'AND B.name = 'global cache converts'

AND c.name = 'global cache get time'

AND D.name = 'global cache gets'

AND E.name = 'global cache convert timeouts'

AND B.inst_id = A.inst_id and

AND C.inst_id = A.inst_id and

AND D.inst_id = A.inst_id and

AND E.inst_id = A.inst_idORDER

BY A.inst_id;

Instance Avg Cache Conv. Time Avg Cache Get Time GC Convert Timeouts

--------- -------------------- ------------------ -------------------

1 1.85812072 .981296356 0

2 1.65947528 .627444273 0

Neither convert time is excessive.Excessive would be > 10-20 ms. Instance

1 has higher convert times because it isgetting and converting from instance 2,

which is running on slower CPUs



41/160




Use the GV$SYSTEM_EVENT view to review

TIME_WAITED statistics for various GCS events if

the get or convert times become significant.

STATSPACK can be used for this to view these

events.



42/160




Interpreting the convertandgetstatistics

High convert times Instances swapping a lot of

blocks over the interconnect

Large values or rapid increases in gets,

converts or average times

GCS contention

High latencies for resource operations Excessive system loads

Non-zero GC converts Timeouts System contention or

congestion. Could indicate

serious performance problems.



43/160



Other Wait Events

If these RAC-specific wait events show up in theTOP-5 wait events on the STATSPACK report,you need to determine the cause of the waits:

global cache open s

global cache open x

global cache null to s

global cache null to x global cache cr request

Global Cache Service Utilization for Logical Reads



44/160



Other Wait Events

Event:

global cache open s

global cache open x

Description:

A session has to wait for receiving permission forshared(s) or exclusive(x) access to the requestedresource

Wait duration should be short.

Wait followed by a read from disk.

Blocks requested are not cached in any instance.

Action:

Not much can be done

When associated with high totals or high per-transaction wait time, data

blocks are not cached.

Will cause sub-optimal buffer cache hit ratios

Consider preloading heavily used tables



45/160



Other Wait Events

Event:

global cache null to s

global cache null to x

Description:

Happens when two instances exchange the

same block back and forth over the network.

These events will consume a greater

proportion of total wait time if one instance

requests cached data blocks from other

instances.

Action:

Reduce the number of rows per block to eliminate the need for block

swapping between instances.



46/160



Other Wait Events

Event:

global cache cr request

Description:

Happens when an instance requests a CR

data block and the block to be transferred

hasnt arrived at the requesting instance.

Action:

Examine the cluster interconnects for possible problems.

Modify objects to reduce possibility of contention.



47/160


Examine the Cluster Statistics page of your STATSPACKreport when:

Global cache waits constitute a large proportion of the wait time listedon the first page of your STATSPACK report.

Response times or throughput does not conform to your service level

requirements.

STATSPACK report should be taken during heavy RACworkloads.


High GCS Time Per Request

AND



48/160




Causes of a high GCS time per request are:

Contention for blocks

System Load

Network issues



49/160




Contention for blocks

You already know this:

Modify the object to reduce the chances forapplication contention.

Reduce the number of rows per block

Adjust the block size to a smaller block sizeModify INITRANS and FREELISTS



50/160




System Load

If processes are queuing for the CPU, raise the priority

of the GCS processes (LMSn

) to have priority over otherprocesses to lower GCS times.

Reduce the load on the server by reducing the numberof processes on the database server.

Increase capacity by adding CPUs to the server

Add nodes to the cluster database



51/160




Network issues

OS logs and OS stats will indicate if a network link is

congested.

Ensure packets are being routed through the privateinterconnect, not the public network.



52/160



Using STATSPACK Reports

The STATSPACK report shows statistics ONLY

for the node or instance on which it was run.

Run statspack.snap procedure andspreport.sql script on each node you want tomonitor to compare to other instances.



53/160




Top 5 Timed Events

Top 5 Timed Events

~~~~~~~~~~~~~~~~~~ % TotalEvent Waits Time (s) Ela Time

-------------------------------------------- ------------ ----------- --------

global cache cr request 820 154 72.50

CPU time 54 25.34

global cache null to x 478 1 .52

control file sequential read 600 1 .52

control file parallel write 141 1 .28

-------------------------------------------------------------

Exceeds CPU time,therefore needsinvestigation.CPU time (processing

time) should be thepredominant event

Transfer times are excessive from other instances in this cluster to

this instance.

Could be due to network problems or buffer cache sizing issues.



54/160




Network changes were made

An index was added

STATSPACK report now looks like this:

Top 5 Timed Events

~~~~~~~~~~~~~~~~~~ % Total

Event Waits Time (s) Ela Time

-------------------------------------------- ------------ ----------- --------

CPU time 99 64.87global cache null to x 1,655 28 18.43

enqueue 46 8 5.12

global cache busy 104 7 4.73

DFS lock handle 38 2 1.64

CPU time is now thepredominant event



55/160




Workload characteristics for this instance:Cluster Statistics for DB: DB2 Instance: INST1 Snaps: 105 -106

Global Cache Service - Workload Characteristics

-----------------------------------------------

Ave global cache get time (ms): 3.1

Ave global cache convert time (ms): 3.2

Ave build time for CR block (ms): 0.2

Ave flush time for CR block (ms): 0.0

Ave send time for CR block (ms): 1.0

Ave time to process CR block request (ms): 1.3

Ave receive time for CR block (ms): 17.2

Ave pin time for current block (ms): 0.2

Ave flush time for current block (ms): 0.0

Ave send time for current block (ms): 0.9

Ave time to process current block request (ms): 1.1

Ave receive time for current block (ms): 3.1

Global cache hit ratio: 1.7

Ratio of current block defers: 0.0

% of messages sent for buffer gets: 1.4

% of remote buffer gets: 1.1

Ratio of I/O for coherence: 8.7

Ratio of local vs remote work: 0.6

Ratio of fusion vs physical writes: 1.0

Snaps: 25 -26

8.2

16.5

1.5

6.0

0.9

8.5

18.3

13.7

3.9

0.8

18.4

17.4

2.5

0.2

2.2

1.6

2.9

0.5

0.0

After network and indexchanges.



56/160




Global Enqueue Services (GES) control the inter-

instance locks in Oracle 9i RAC.

The STATSPACK report contains a special section

for these statistics.

Global Enqueue Service Statistics

---------------------------------

Ave global lock get time (ms): 0.9

Ave global lock convert time (ms): 1.3

Ratio of global lock gets vs global lock releases: 1.1



57/160




Guidelines for GES Statistics:

All times should be < 15ms

Ratio of global lock gets vs global lock releases should

be near 1.0

High values could indicate possible network or

memory problems

Could also be caused by application locking issues

May need to review the enqueue section of

STATSPACK report for further analysis.



58/160




GCS AND GES messaging

Watch for excessive queue times (>20-30ms)

GCS and GES Messaging statistics

--------------------------------

Ave message sent queue time (ms): 1.8

Ave message sent queue time on ksxp (ms): 2.6

Ave message received queue time (ms): 1.2

Ave GCS message process time (ms): 1.2

Ave GES message process time (ms): 0.2% of direct sent messages: 58.4

% of indirect sent messages: 4.9

% of flow controlled messages: 36.7



59/160




The GES Statistics section of the STATSPACK report showsgcs blocked convertstatistics.

GES Statistics for DB: DB2 Instance: INST2 Snaps: 25 -26

Statistic Total per Second per Trans

--------------------------------- ---------------- ------------ ------------

dynamically allocated gcs resource 0 0.0 0.0

dynamically allocated gcs shadows 0 0.0 0.0

flow control messages received 0 0.0 0.0

flow control messages sent 0 0.0 0.0

gcs ast xid 0 0.0 0.0

gcs blocked converts 904 2.1 150.7gcs blocked cr converts 1,284 2.9 214.0

gcs compatible basts 0 0.0 0.0

gcs compatible cr basts (global) 0 0.0 0.0

gcs compatible cr basts (local) 75 0.2 12.5

:

:



60/160




Statistic:

gcs blocked converts

gcs blocked cr converts

Description:

Instance requested a block from another

instance and was unable to obtain the

conversion of the block.

Indicates users on different instances need

to access the same blocks.

Action:Prevent users on different instances from needing access to the same blocks.

Ensure sufficient freelists (not an issue if using automated freelists)

Reduce block contention through freelists, initrans.

Limit rows per block.



61/160




Waits are a key tuning indicator:

Wait Events for DB: DB2 Instance: INST2 Snaps: 25 -26

-> s - second

-> cs - centisecond - 100th of a second-> ms - millisecond - 1000th of a second

-> us - microsecond - 1000000th of a second

-> ordered by wait time desc, waits desc (idle events last)

Event Waits Timeouts Time (s) (ms) /txn

---------------------------- ------------ ---------- ---------- ------ --------

global cache cr request 820 113 154 188 136.7

global cache null to x 478 1 1 2 79.7control file sequential read 600 0 1 2 100.0

control file parallel write 141 0 1 4 23.5

enqueue 29 0 1 18 4.8

library cache lock 215 0 0 2 35.8

db file sequential read 28 0 0 7 4.7

LGWR wait for redo copy 31 16 0 4 5.2

:

::

Our predominate wait.Network issues

Indicates potential problemswith the IO subsystem ifsevere. (not severe here,total wait time is < 1 sec)



62/160


u g t e C C uste te co ect


The Library Cache Activity report shows statistics regarding the GES.

Watch for GES Invalid Requests and GES Invalidations.

Could indicate insufficient sizing of the shared pool resulting in GEScontention.

Library Cache Activity for DB: DB2 Instance: INST2 Snaps: 25 -26

->"Pct Misses" should be very low

GES Lock GES Pin GES Pin GES Inval GES Invali-

Namespace Requests Requests Releases Requests dations

--------------- ------------ ------------ ------------ ----------- -----------

BODY 1 0 0 0 0CLUSTER 4 0 0 0 0

INDEX 84 0 0 0 0

SQL AREA 0 0 0 0 0

TABLE/PROCEDURE 617 192 0 77 0

TRIGGER 0 0 0 0 0



63/160


g


Single-instance tuning should be performed before

attempting to tune the processes that communicate

via the cluster interconnect.

The STATSPACK report contains valuable statistics

that can help you identify SQL-related problems,

memory problems, IO problems, latch contentionand enqueue issues.



64/160


g

GES Monitoring

Oracle makes a Global Cache Service request

whenever a user accesses a buffer cache to read or

modify a data block and the block is not in the local

cache.

RATIOS can be used to give an indication of how

hard your Global Services Directory processes areworking.

and the

crowd gasps!



65/160


g

GES Monitoring

To estimate the use of the GCS relative to the

number of logical reads:

global cache gets + global cache converts + global cache

cr blocks rcvd + global cache current blocks rcvd

consistent gets + db block gets

Sum of GCS requests

Number of logical reads



66/160


This information can be found by querying theGV$SYSSTAT view


(A.VALUE+B.VALUE+C.VALUE+D.VALUE)/(E.VALUE+F.VALUE) "GLOBAL CACHE HIT RATIO"


GV$SYSSTAT C, GV$SYSSTAT D,

GV$SYSSTAT E, GV$SYSSTAT FWHERE A.NAME='global cache gets'

AND B.NAME='global cache converts'

AND C.NAME='global cache cr blocks received'

AND D.NAME='global cache current blocks received'

AND E.NAME='consistent gets'

AND F.NAME='db block gets'

AND B.INST_ID=A.INST_ID AND C.INST_ID=A.INST_ID

AND D.INST_ID=A.INST_ID AND E.INST_ID=A.INST_IDAND F.INST_ID=A.INST_ID;

Instance GLOBAL CACHE HIT RATIO

---------- ----------------------

1 .02403656

2 .014798887

g

GES Monitoring

SINCE INSTANCE

STARTUP



67/160


g

GES Monitoring

Some blocks, those frequently requested by local and remote

users, will be hot.

If a block is hot, its transfer is delayed for a few milliseconds

to allow the local users to complete their work. The following ratio provides a rough estimate of how

prevalent this is:

A ratio of higher than 0.3 indicates that you have some

pretty hot blocks in your database.

global cache defers

global cache current blocks served



68/160


g

GES Monitoring

To find the blocks involved in busy waits, query the columns NAME,KIND, FORCED_READS, FORCED_WRITES.

Select INST_ID "Instance", name, kind,

sum(forced_reads) "Forced Reads",

sum(forced_writes) "Forced Writes"FROM gv$cache_transfer

WHERE owner# != 0

GROUP BY inst_id, name, kind

ORDER BY 1,4 desc,2;

Instance NAME KIND Forced Reads Forced Writes

--------- -------------------- ---------- ------------ -------------1 MOD_TEST_IND INDEX 308 0

1 TEST2 TABLE 64 0

1 AQ$_QUEUE_TABLES TABLE 5 0

2 TEST2 TABLE 473 0

2 MOD_TEST_IND INDEX 221 0



69/160


g

GES Monitoring

If you discover a problem, you may be able to

alleviate contention by:

Reducing hot spots or spreading the accesses to index

blocks or data blocks.

Use Oracle has or range partitions wherever applicable,

just as you would in single-instance Oracle databases

Reduce concurrency on the object by implementing

resource management or load balancing.

Decrease the rate of modifications on the object (use

fewer database processes).



70/160


g

GES Monitoring

Fusion writes occur when a block previouslychanged by another instance needs to be written todisk in response to a checkpoint or cache aging.

Oracle sends a message to notify the other instancethat a fusion write will be performed to move thedata block to disk.

Fusion writes do not require an additional write to

disk. Fusion writes are a subset of all physical writes

incurred by an instance.



71/160


GES Monitoring

The following ratio shows the proportion of writes thatOracle manages with fusion writes:

SELECT A.inst_id "Instance",

A.VALUE/B.VALUE "Cache Fusion Writes Ratio"

FROM GV$SYSSTAT A,

GV$SYSSTAT B

WHERE A.name='DBWR fusion writes'

AND B.name='physical writes'

AND B.inst_id=a.inst_id

ORDERBY A.INST_ID;

Instance Cache Fusion Writes Ratio

--------- -------------------------

1 .216290958

2 .131862042

DBWR fusion writes

physical writes



72/160


GES Monitoring

A high large value for Cache Fusion Writes ratio

may indicate:

Insufficiently sized caches

Insufficient checkpoints

Large numbers of buffers written due to cache

replacement or checkpointing.



73/160


CACHE_TRANSFER views

The V$XXX_CACHE_TRANSFER views show the types of

blocks that use the cluster interconnect in a RAC

environment.



74/160



V$CACHE_TRANSFER Shows the types and classes of blocks thatOracle transfers over the cluster interconnect

on aper-object basis.

V$CLASS_CACHE_TRANSFER Use this view to identify the class of blocksthat experience cache transfers on aper-class

of object basis.

V$FILE_CACHE_TRANSFER Use this view to monitor the blockstransferredper file. The FILE_NUMBER

column identifies the datafile that contained

the blocks transferred.

V$TEMP_CACHE_TRANSFER Use this view to monitor the transfer of

temporary tablespace blocks. Has the same

structure as V$FILE_CACHE_TRANSFER.



75/160



FILE# NUMBER

BLOCK# NUMBER

CLASS# NUMBER

STATUS VARCHAR2(5)

XNC NUMBER

FORCED_READS NUMBER

FORCED_WRITES NUMBER

NAME VARCHAR2(30)

PARTITION_NAME VARCHAR2(30)

KIND VARCHAR2(15)

OWNER# NUMBER

GC_ELEMENT_ADDR RAW(4)

GC_ELEMENT_NAME NUMBER

V$CACHE_TRANSFER FORCED_READS andFORCED_WRITES column are used to

determine which types of objects your

RAC instances are sharing.

Values in FORCED_WRITES columnprovide counts of how often a certain

block type experiences a transfer out of a

local buffer cache due to a request for

current version by another instance.

The NAME column shows the name of

the object containing blocks being

transferred.



76/160



FILE_NUMBER NUMBER

X_2_NULL NUMBER

X_2_NULL_FORCED_WRITE NUMBER

X_2_NULL_FORCED_STALE NUMBERX_2_S NUMBER

X_2_S_FORCED_WRITE NUMBER

S_2_NULL NUMBER

S_2_NULL_FORCED_STALE NUMBER

RBR NUMBER

RBR_FORCED_WRITE NUMBER

RBR_FORCED_STALE NUMBER

NULL_2_X NUMBER

S_2_X NUMBER

NULL_2_S NUMBER

CR_TRANSFERS NUMBER

CUR_TRANSFERS NUMBER

V$FILE_CACHE_TRANSFER

FILE_NUMBER shows the file

numbers of datafiles that are source of

blocks transferred.

Use this view to show the block

transfers per file.



77/160



The shared disk architecture of Oracle 9i virtually

eliminates forced disk writes, but

V$CACHE_TRANSFER and

V$FILE_CACHE_TRANSFER may still show thenumber of block mode conversions per block class

or object, however, the values in

FORCED_WRITES column will be zero.



78/160


Monitoring the GES Processes

To monitor the Global

Enqueue Service Processes,

use the

GV$ENQUEUE_STAT view. INST_ID NUMBEREQ_TYPE VARCHAR2(2)

TOTAL_REQ# NUMBER

TOTAL_WAIT NUMBER

SUCC_REQ# NUMBER

FAILED_REQ# NUMBER

CUM_WAIT_TIME NUMBER

GV$ENQUEUE_STAT



79/160



Retrieve all of the enqueues with a total_wait# value greaterthan zero:

SELECT *

FROM gv$enqueue_statWHERE total_wait# > 0

ORDER BY inst_id, cum_wait_time desc;

INST_ID EQ TOTAL_REQ# TOTAL_WAIT# SUCC_REQ# FAILED_REQ# CUM_WAIT_TIME

---------- -- ---------- ----------- ---------- ----------- -------------

1 TX 31928 26 31928 0 293303

1 PS 995 571 994 1 556581 TA 1067 874 1067 0 10466

1 TD 974 974 974 0 2980

1 DR 176 176 176 0 406

1 US 190 189 190 0 404

1 PI 47 27 47 0 104

:

:



80/160



Oracle says enqueues of interest in the RAC

environment are:SQ Enqueue Indicates there is contention for sequences. Increase the cache size

of sequences using ALTER SEQUENCES. Also, when creatingsequences, the NOORDER keyword should be used to avoid forced

ordering of queued sequence values.

TX Enqueue Application-related row locking usually causes problems here. RACprocessing can magnify the effect of TX enqueue waits. Index block

splits can cause TX enqueue waits. Some TX enqueue performance

issues can be resolved by setting the value of INITRANS parameterfor a table or index (have heard to set equal to the number of CPUs

per node multiplied by the number of nodes in a cluster multiplied

by 0.75). I prefer setting INITTRANS to the number of concurrent

processes issuing DML against the block.

Enterprise Manager 9i


81/160


Enterprise Manager 9i


82/160


Events


83/160


Events

Top 5 Timed Events

~~~~~~~~~~~~~~~~~~

% Total

Event Waits Time (s) Ela Time

--------------------------- ------------ ----------- --------db file sequential read 399,394,399 2,562,115 52.26

CPU time 960,825 19.60

buffer busy waits 122,302,412 540,757 11.03

PL/SQL lock timer 4,077 243,056 4.96

log file switch 188,701 187,648 3.83

(checkpoint incomplete)

Statspack - Top Wait EventsThi t l k f


84/160


Things to look for

Wait Problem Potential Fix

Sequential Read Indicates many index readstune thecode (especially joins)

Scattered Read Indicates many full table scanstunethe code; cache small tables

Free Buffer Increase the DB_CACHE_SIZE;shorten the checkpoint; tune the code

Buffer Busy Segment HeaderAdd freelists orfreelist groups (Use ASSM)

Buffer Busy Data BlockSeparate hot data; use

reverse key indexes; smaller blocks



85/160


Things to look for


Buffer Busy Data Block (cont.)Increase initransand/or maxtrans

Buffer Busy Undo HeaderAdd rollback segmentsor areas

Buffer Busy Undo blockCommit more (not toomuch) Larger rollback segments/areas.

Latch Free Investigate the detail (Covered later)

Enqueue - ST Use LMTs or pre-allocate large extents

Enqueue - HW Pre-allocate extents above high water

mark



86/160


Things to look for


EnqueueTX Increase initrans and/or maxtrans on(transaction) the table or index

Enqueue - TM Index foreign keys; Check application(trans. mgmt.) locking of tables

Log Buffer Space Increase the Log Buffer; Faster disksfor the Redo Logs

Log File Switch Archive destination slow or full; Addmore or larger Redo Logs

Log file sync Commit more records at a time; Faster

Redo Log disks; Raw devices

Statspack - Latch WaitsThi t l k f


87/160


Things to look for

Latch Problem Potential Fix

Library Cache Use bind variables; adjust theshared_pool_size

Shared Pool Use bind variables; adjust theshared_pool_size

Row cache objects Increase the Shared Pool

Redo allocation Minimize redo generation and

avoid unnecessary commitsRedo copy Increase the

_log_simultaneous_copies

Locally ManagedTablespaces LMT


88/160


Tablespaces- LMT

Manage space locally using bitmaps

Benefits no tablespace fragmentation issues

better performance handling on large segments lower contention for central resources, e.g., no ST

enqueue contention

fewer recursive calls

Implementation specify EXTENT MANAGEMENT LOCAL clause

during tablespace creation

in-place migration

Automatic Segment SpaceM t (ASSM)


89/160


Management (ASSM)

Automatic intra-object space managementBenefits Eliminates Rollback Segment Management

simplified administration (no more FREELISTS, FREELIST

GROUPS, PCTUSED) improved space utilization & concurrency

enhanced performance

Can set undo retention time If set to longest running query time, no more Snapshot

too old!

Implementation specify SEGMENT SPACE MANAGEMENT AUTO

clause during tablespace creation

Enterprise Manager 10g for the Grid


90/160


Host andHardware

te p se a age 0g o t e G d

Database

Oracle9iAS

Storage

Network andLoad Balancer

Applications

AdministrationMonitoring

ProvisioningSecurity

EnterpriseManager

Whats New in EM 10g


91/160


What s New in EM 10g

Area EM 9i EM 10g

Oracle Database

Oracle9iAS

Oracle Collab Suite

Oracle eBusinessSuite

Operating System

Storage

Network

SLB (Nortel, F5)

Hardware

Admin and

Monitoring

Application PerformanceManagement

Real performance for All YourUsers, All Your Pages, All theTime

Application Availability Synthetic transactions

End-to-end tracing

Enterprise ConfigurationManagement

Rapid installation

Deployment Provisioning

Upgrade

Automated patching

Key EM10G functionality

End to End Solution


92/160


All Target Types

System and applicationavailability Set up of target-specific

metrics/thresholds Configuration data collection Performance data collection Comprehensive monitoring Alerts Email and Paging notifications Blackouts User-defined jobs System and application

response time measurements Policy violation reporting Clone Oracle Home

Patch search/download

ASM Disk group admin (rebalance) Startup/Shutdown Disk group usage/status I/O performance

Add/Remove disks

DB

General health assessment Bad SQL identification Top SQL identification SQL recommendations Tuning advisories Performance trending Backup

Restore Security vulnerability ID Create/remove

Physical/Logical Standby Standby health assessment Standby switchover/failover

RAC Cluster cache coherency

monitoring Failover events Discovery of RAC topology Startup/shutdown Relocate services Failover jobs

iAS Config changes across cluster

(OC4J, Apache) Create/Manage cluster Deploy app to cluster Reconfigure a farm Add/Remove node from cluster

Clone (mid-tiers) Patch

OCS IMAP, SMTP End-2-end service

monitoring Files End-2-end service

monitoring Files document analysis: count,size, format

Files user analysis: number,quota consumed

Host/Host Clusters

Top processes identification

Monitor All Targets


93/160


Monitor All Targets

Monitor

Targets

are they

up?

Target

Alerts

View it Wireless


94/160


View it Wireless

Target

Alerts

Application Performance Issues


95/160


Whois the problemimpacting?

Whatis the businessimpact?

Wherein the infrastructureis the problem?

Whoseproblem is it? Whyis the problem

occurring?

Internet

Database

Application Server

Applications

OS

Storage

Network

Application

Developer

DBA

Data CenterSupport

System

Administrator

Network Service

Provider

pp

Middle Tier PerformanceAnalysis


96/160


Analysis

Application server and back-end problem

diagnostics

Middle tier processing time and load breakoutsFind slowest URLs and get number of hits

Top servlets / JSPs by requests /processing

timeTuning recommendations

Monitor App Servers


97/160


Monitor App. Servers

Monitor

Web

Apps.

Web

App.

Alerts

Choose a Host


98/160


Choose a Host

Select

Targets

tab

Pick a

Host

App. Server Response Time


99/160


App. Server Response Time

App

ServerResponse

Time &

Info.

Choose

App.

Server

App. Server Performance


100/160


App. Server Performance

Memory

CPU

WebCache

Hits

View the Web Application


101/160



Choose

Web

Apps.

Choose

the App.

To

Monitor



102/160



Page

Response

Time

Detailed

Info.

Click

Alert

for this



103/160



Page

Response

Time

Detail

URL Response Times


104/160


URL Response Times

Web

Page

Response

Time all

URLs

Middle Tier Performance


105/160


Servlet EJB JDBC

Full URL processing call stack analysis

Middle tier processing time and load breakouts

Tracing down to the SQL statement level

internet

Middle Tier

JSP

dd

Middle Tier Performance


106/160


dd e e e o a ce

Web

Page

Detail

Response

Time all

URLs

Splits

Time

into

Parts

Host Performance


107/160


Host Performance

Each

Piece of

the Web

App.

Choose

the Host

Host Performance


108/160


Host Performance

I/O

Memory

CPU


109/160

Host Performance


110/160


Side by

Side

Compare

Managing Groups


111/160


g g p

Managed from a single-view

Monitoring and automatedoperations

Logical modeling of sets ofsystems

Applications, Clusters, othersets

Leveraged by all services

Jobs, Policies, Membership-based

inheritance

Applications

Sets of Systems

Automated Group Operations


112/160


p p

All Target Types Config. change tracking Configuration inventory Diff configurations Search configurations Behavior inheritance (Jobs) Create/manage groups

OS command jobs SQL Script jobs Aggregate Metrics Alert Rollups Set blackouts Set monitoring levels Set target properties

Set thresholds Installation hardening Security alerts and patches Discovery

Host/Host Clusters Add/Remove node from OS

cluster

ASM Disk group admin (rebalance) Startup/Shutdown Disk group usage/status I/O performance Add/Remove disks

DB Analyze Backup Export Startup/Shutdown Configuration Advise Clone, Patch

RAC Spfile changes across

instances Start/Stop/Relocate services Startup/Shutdown Cluster cache coherency Monitoring rollups

Add/Remove Instance

iAS Config changes across cluster

(OC4J, Apache) Create/Manage cluster Deploy app to cluster Reconfigure a farm Add/Remove node from cluster

Clone (mid-tiers) Patch

OCS Custom grouping Home pages for EMAIL and IM

WebApp End-2-End availability End-2-End monitoring End-2-End tracing

EM Agent deployment

Database Performance


113/160


Pick

Items for

the Group

Create a

Prod DBs

Group



114/160


Check

Alerts &

Policy

Issues

View the

Prod DBs

Group



115/160


Alert

History

Waits

AdviceClick

Here for

Issues



116/160


Alert

Issues

Major

Waits



117/160


Fix

Critical

Oracle Standardization


118/160


Policy Management Rule definitions

Violation detection

Corrective action

Security policies Software installation

hardening

Excess services/ports

Excess user privileges

Configuration policies Best practices

Base images

Performance polices Thresholds

Policy

Automated Best Practices -Database


119/160


1. Insufficient Number of Control Files

2. Insufficient Redo Log Size

3. Insufficient Number of Redo Logs

4. Use of Unlimited Autoextension

5. Use of Non-Standard Init. Parameters

6. Recovery Area Location Not Set

7. Autobackup of Control File is not Enabled

8. SYSTEM TS Used as User Default TS

9. Segment with Extent Growth Policy Violation10.Tablespace Containing Mixed Segment Types

11. Not Using Locally Managed Tablespaces

12. SYSTEM TS Contains Non-System Data Seg

13. Users with Permanent TS as Temporary TS

14. Insufficient Recovery Area Size

15. Force Logging Disabled

16. Not Using Spfile

17. Rollback in SYSTEM Tablespace

18. Not Using Undo Space Management

19. Non-uniform Default Extent Size


120/160



121/160


Monitor

Database

We have a

CPU

issue!


122/160



123/160


Monitor

for a

period of

time

Check the

Top SQL



124/160


Drill

down to

show the

Top SQL

Target

Alerts



125/160


Top SQL

CheckExecution

History or

Other

Options

Use Automatic DatabaseDiagnostics Monitor (ADDM)


126/160


agnost cs Mon to (A M)

Monitor

Database

Check the

Diagnostic

Summary

Target

Alerts



127/160


g ( )

Check a

snapshot

Check the

findings



128/160


Finding

Details

Suggests!



129/160


I can do itfor you !

SQL TuningAdvisor

DBAHigh-Load

SQL

ADDM

SQLWorkload

g ( )

ADDMs Architecture


130/160


SQLAdvisor

High-loadSQL

IO / CPUissues

RAC issues

Automatic Diagnostic Engine

Snapshots in

Automatic Workload

Repository

Automatic Diagnostic Engine

SystemSizing

Advice

Network +DB config

Advice

Uses Time & Wait Model datafrom Workload Repository

Classification Tree is based on

decades of Oracle

performance tuning expertise

Time based analysis

Recommends solutions or next

steps

Runs proactively & manually

Automatic Workload Repository


131/160


Like a better statspack!

Repository of performance information Base statistics

SQL statistics Metrics

ACTIVE SESSION HISTORY

Workload data is captured every 30 minutes or

manually and saved for 7 days by defaultSelf-manages its space requirements

Provides the basis for improved performancediagnostic facilities

Automatic Tuning Optimizer (ATO)


132/160


Add MissingIndexes

Modify SQLConstructs

Create a SQLProfile

Automatic Tuning Optimizer

SQL StructureAnalysis

Access Path

Analysis

SQLProfiling

StatisticsAnalysis

Gather Missing orStale Statistics

DBA

SQL Tuning

Recommendations

SQL TuningAdvisor

Automatic Tuning Optimizer (ATO)


133/160


It is the query optimizer running in tuning mode

Uses same plan generation process but performsadditional steps that require lot more time

It performs verification steps To validate statistics and its own estimates

Uses dynamic sampling and partial executions

It performs exploratory steps

To investigate the use of new indexes that couldprovide significant speed-up

To analyze SQL constructs that led to expensiveplan operators

ADDM SQL Tuning Advisor


134/160


Drill into

Top SQL

for worst

timeperiod

Get Help!

Top SQL; Use SQL Tuning Sets


135/160


Tuning

Sets

Tune the

Top SQL

set.

Using SQL Tuning Sets


136/160


SQL

Tuning

Sets Info

How long

to work

on this



137/160


Tuning

Results

for this

job

Suggests

using aprofile

For this

SQL



138/160


Detailed

info

Improve

Factor and

click to

Use



139/160


Confirmed

to use

suggestion

Detail

Using SQL Tuning Sets (STS)


140/160


Motivation

Enable user to tune a custom set of SQL statements

It is a new object in Oracle10g for capturing SQL

workload It stores SQL statements along with

Execution context: parsing user, bind values, etc.

Execution statistics: buffer gets, CPU time, elapsetime, number of executions, etc.

It is created from a SQL source

Sources: AWR, cursor cache, user-defined SQL

workload another STS

Business Transaction Monitoring- Set up Beacons!


141/160


Monitor critical onlinebusiness processes

Scope and quantify impactof performance problems onall user communities

Isolate network vs serverrelated delays

Transaction profiling

Alerts and notifications

Web Application

internet

Transaction MonitoringRecorder


142/160


Rapid deployment and immediate value

Simple, automatic, no scripting

Beacons automatically synchronized to play

transactions at specific intervals

Enter URLand navigate

through

transaction

SaveTransaction

DownloadRecorder

(once)

Start recorderwhich opensnew browser

window

1 2 3 4


143/160

Availability Monitoring Topology


144/160


Oracle9iAS

OracleDatabase

Oracle9iAS

Oracle

E

-BusSuite

End Users Apps and

Mid-Tier ServersDatabase Hosts Storage

3rdP

arty

AppServer

Network

Beacon running availabilitytransaction


ChicagoSales Office

TorontoSales Office

TokyoSales Office



Headquarters

Enterprise Manager for theGrid; Many more Options!


145/160


; y p


146/160

Questions and Answers


147/160


For More Information


148/160

A TUSC Presentation148

www.tusc.com

Oracle9i Performance

Tuning Tips &Techniques; Richard

J. Niemiec; Oracle

Press (May 2003)

If you are going through hell, keep going- Churchill

References


149/160


Oracle9i Performance Tuning Tips & Techniques, RichNiemiec

The Self-managing Database: Automatic PerformanceDiagnosis; Karl Dias & Mark Ramacher, OracleCorporation

EM Grid Control 10g;otn.oracle.com, Oracle Corporation

Oracle Enterprise Manager 10g: Making the Grid a

Reality; Jay Rossiter, Oracle CorporationThe Self-Managing Database: Guided Application and

SQL Tuning;Benoit Dageville, Oracle CorporationThe New Enterprise Manager: End to End Performance

Management of Oracle; Julie Wong & Arsalan Farooq

References


150/160


Oracle Database 10g Performance Overview; Herv

Lejeune, Oracle Corporation

Oracle 10g; Penny Avril,, Oracle Corporation

Forrester Reports, Inc., TechStrategy Research, April

2002, Organic IT

Internals of Real Application Cluster, Madhu Tumma,

Credit Suisse First BostonOracle9i RAC; Real Application Clusters Configuration

and Internals, Mike Ault & Madhu Tumma

References


151/160


Oracle Tuning Presentation, Oracle Corporationwww.tusc.com, www.oracle.com, www.ixora.com,

www.laoug.org, www.ioug.org, technet.oracle.com

Oracle PL/SQL Tips and Techniques, Joseph P. Trezzo;Oracle Press

Oracle9i Web Development, Bradley D. Brown; OraclePress

Special thanks to Steve Adams, Mike Ault, Brad Brown,Don Burleson, Kevin Gilpin, Herve Lejeune, RandySwanson and Joe Trezzo.

References


152/160


Oracle Database 10g Automated Features , MikeAult, TUSC

Oracle Database 10gNew Features, Mike Ault,

Daniel Liu, Madhu Tumma, RampantTechnical Press, 2003, www.rampant.cc

Oracle Database 10g - The World's First Self-

Managing, Grid-Ready Database Arrives, KelliWiseth, Oracle Technology Network, 2003,otn.oracle.com

References


153/160


Oracle 10g documentation

Oracle 9i RAC class & instructors comments

Oracle 9i Concepts manual

http://geocities.com/pulliamrick/Tips for Tuning Oracle9i RAC on Linux, Kurt

Engeleiter, Van Okamura, Oracle

Leveraging Oracle9i RAC on Intel-based servers tobuild an Adaptive Architecture, Stephen White,Cap Gemini Ernst & Young, Dr Don Mowbray,Oracle, Werner Schueler, Intel

References


154/160


Running YOUR Applications on Real ApplicationClusters (RAC); RAC Deployment Best Practices,Kirk McGowan, Oracle Corporation

The Present, The Future but not Science Fiction; RealApplication Clusters Development, Angelo Pruscino,Oracle

Building the Adaptive Enterprise; Adaptive

Architecture and Oracle, Malcolm Carnegie, CapGemini Ernst & Young

Internals of Real Application Cluster, Madhu Tumma,Credit Suisse First Boston

Creatin Business Pros erit in a Challen in

References


155/160


Real Application Clusters, Real Customers RealResults, Erik Peterson, Technical Manager, RAC,Oracle Corp.

Deploying a Highly Manageable Oracle9i RealApplications Database, Bill Kehoe, Oracle

Getting the most out of your database, AndyMendelsohn, SVP Server Technologies, Oracle

CorporationOracle9iAS Clusters: Solutions for Scalability and

Availability, Chet Fryjoff, Product Manager, OracleCorporation

Oracle RAC and Linux in the real enter rise Mark


156/160

References


157/160


Oracle 10g; Penny Avril, Principal Database ProductManager, Server Technologies, Oracle Corporation

To Infinity and Beyond, Brad Brown, TUSC

Forrester Reports, Inc., TechStrategy Research, April2002, Organic IT

Internals of Real Application Cluster, Madhu Tumma,Credit Suisse First Boston

Oracle9i RAC; Real Application Clusters Configurationand Internals, Mike Ault & Madhu Tumma

Oracle9i Performance Tuning Tips & Techniques,

Richard J Niemiec

References


158/160

The Traits of the Uncommon Leader; U.S. Marine CorpsManual

60 Minute Manager - Joe Trezzo, TUSC

Uncommon Leaders; TUSC, 1989Whats next for IT?; Larry Geisel, Netscape

The Miracle of Motivation; George Shinn

Gods little devotional book for leaders

7 Habits of Highly Effective People; Steven CoveyThe Laws of Leadership - John Maxwell

The making of a leader - Frank Damazio

Bullet Proof Manager Seminars, Krestcom Productions, Inc.

TUSC Services
http://www.motivateus.com/


159/160


Oracle Technical Solutions Full-Life Cycle Development Projects

Enterprise Architecture

Database Services

Oracle Application Solutions Oracle Applications Implementations/Upgrades

Oracle Applications Tuning

Managed Services 24x7x365 Remote Monitoring & Management

Functional & Technical Support

Training & Mentoring

Oracle Authorized Reseller

Copyright Information


160/160

Neither TUSC, Oracle nor the authors

guarantee this document to be error-free.

Please provide comments/questions to

[email protected].

TUSC 2004. This document cannot be

reproduced without expressed written consent

from an officer of TUSC

rac tuning sig 2004

Documents