best practices oracle cluster ware session 355 ppt

Best Practices of Oracle

10g/11g Clusterware:

Configuration, Administration

and Troubleshooting

Kai Yu

Dell Inc

About Author:

Kai Yu

• Senior System Consultant in Dell Oracle Solutions

Engineering lab: kai_yu@dell.com, 512-725-0046

• Oracle Database/Applications DBA since 1995

• IOUG Collaborate 09 Committee Member and RAC SIG

US Event Chair

• Frequent Presenter and Author for IOUG SELECT

Journal, Dell Power Solutions, OOW06/07/08 and

Collaborate 08/09, RAC SIG Web seminar

Agenda

Oracle Clusterware Architecture

Shared Storage Configuration

Network Configuration

Managing Oracle Clusterware

Oracle Clusterware Troubleshooting

QA

Oracle Clusterware Architecture

Introduction to Oracle Clusterware– Clusterware role: manage cluster resources

– Components: OCR, voting disk, Interconnect,

Clusterware processes

Oracle Clusterware Architecture

Oracle clusterware components– Voting disk stores cluster membership, 3 copies

– OCR stores information about clusterware resources ,

multiplexed OCR for high availability

– Clusterware processes:

• CRS manages cluster resourcesroot 9204 1 0 06:03 ? 00:00:00 /bin/sh /etc/init.d/init.crsd run

root 10058 9204 0 06:03 ? 00:01:23 /crs/product/11.1.0/crs/bin/crsd.bin reboot

• CSSD manages the node membership through heartbeats and votingdisk, notify the membership

status changes:root 9198 1 0 06:03 ? 00:00:22 /bin/sh /etc/init.d/init.cssd fatal

root 10095 9198 0 06:03 ? 00:00:00 /bin/sh /etc/init.d/init.cssd oprocd

root 10114 9198 0 06:03 ? 00:00:00 /bin/sh /etc/init.d/init.cssd oclsomon

root 10151 9198 0 06:03 ? 00:00:00 /bin/sh /etc/init.d/init.cssd daemon

oracle 10566 10151 0 06:03 ? 00:00:40 /crs/product//11.1.0/crs/bin/ocssd.bin

• Event manager(EVM) publishes events through ONS;

communicate between CRS and CSS

Oracle Clusterware Architecture

root 9196 1 0 06:03 ? 00:00:00 /bin/sh /etc/init.d/init.evmd run

root 10059 9196 0 06:03 ? 00:00:00 /bin/su -l oracle -c sh -c 'ulimit -c

unlimited; cd /crs /product/11.1.0/crs/log/kblade2/evmd;

exec /crs/product/11.1.0/crs/bin/evmd

oracle 10060 10059 0 06:03 ? 00:00:02 /crs/product/11.1.0/crs/bin/evmd.bin

OPROCD processor monitor monitors the cluster

. Locked in memory

. Introduced in 10.2.0.4 , replace hangcheck timer

. Reboot cluster if failedroot 9198 1 0 06:03 ? 00:00:22 /bin/sh /etc/init.d/init.cssd fatal

root 10095 9198 0 06:03 ? 00:00:00 /bin/sh /etc/init.d/init.cssd oprocd

root 10465 10095 0 06:03 ? 00:00:00 /crs/product/11.1.0/crs/bin/oprocd run -t 1000 -m 500 -f

. Other processes:

RACG: oracle 12039 1 0 06:06 ? 00:00:00 /opt/oracle/product/11.1.0/asm/bin/racgimon

daemon ora.kblade2.ASM2.asm

oracle 12125 1 0 06:06 ? 00:00:06 /opt/oracle/product/11.1.0/db_1/bin/racgimon startd

test1db

ONS: Oracle Notification Services oracle 12063 1 0 06:06 ? 00:00:00 /crs/oracle/product/11.1.0/crs/opmn/bin/ons -d

oracle 12064 12063 0 06:06 ? 00:00:00 /crs/oracle/product/11.1.0/crs/opmn/bin/ons -d

Oracle Clusterware Architecture

Hardware Configuration of Oracle Clusterware– Servers, shared storage, interconnect

– Two interconnect switches for redundant interconnects

– Butterfly connections to shared storage:

Servers <-> IO Switches <->SAN storage

Shared Storage Configuration

Storage Requirement:– Shared storage for OCR and voting disk

– Types: block devices, RAW devices, OCFS/OCFS2, NFS on certified NAS (Oracle Storage Compatibility Program list)

– HA requirement for the shared storage

Physical connections to shared SAN storage

– Fully Redundant active-active IO paths: for HA and IO Load balancing

FC storage: dual HBAs and dual FC switches: each server has two independent paths to both storage processors

Shared Storage Configuration

Fully redundant IO paths for iSCSI storage

Multiple NIC cards of each server; two Gigabit Ethernet

switches. On each storage control module, one network

interface connects to one switch and other two network

interfaces connects to other switch

Shared Storage Configuration

Multipath Devices of the Shared Storage– Multipathing device driver to combine multiple IO paths

– Linux native Device Mapper (DM) , or storage vendor such as EMC PowerPath driver

– Example of configuring Linux Device Mapper (DM)

• Verify : rpm –qa | grep device-mapper

• Find the unique SCSI ID of the device: $/sbin/scsi_id -gus /block/sdb

36090a028e093fc906099540639aa2149

$/sbin/scsi_id -gus /block/sde

36090a028e093fc90609954063g9aa2149

• Configure multipathing in /etc/multipath.confmultipath {

wwid 36090a028e093fc906099540639aa2149 #<---- for sdb and sde

alias votingdisk1

}

……

• service multipathd restart

Shared Storage Configuration

• Verify the multipath devices: multipath -ll

ls -lt /dev/mapper/* brw-rw---- 1 root disk 253, 8 Feb 18 02:02 /dev/mapper/votingdisk1

– EMC PowerPath driver for EMC storage:

• Install EMC PowerPath and Naviagent software:rpm –ivh EMCpower.LINUX-5.1.2.00.00-021.rhel5.x86_64.rpm

rpm –ivh naviagentcli-6.24.2.5.0-1.noarch.rpm

• Start naviagent agent and PowerPath daemons:service naviagent start , service PowerPath start

• Verify EMC pseudo devices in /proc/partitions:120 32 419430400 emcpowerc

• powermt display dev=emcpowercPseudo name=emcpowerc

==============================================================================

---------------- Host --------------- - Stor - -- I/O Path - -- Stats ---

### HW Path I/O Paths Interf. Mode State Q-IOs Errors

==============================================================================

2 lpfc sdc SP B1 active alive 0 0

2 lpfc sdh SP A0 active alive 0 0

Shared Storage Configuration

Block Devices vs Raw Devices– For RHEL4: use raw devices for 10g, block devices for 11g

– For RHEL 5: raw devices are depreciated:

• 11g clusterware: use block devices:

set the proper ownerships and permissions in /etc/rc.local file

chown root:oinstall /dev/mapper/ocr*

chmod 0640 /dev/mapper/ocr*

chown oracle:oinstall /dev/mapper/voting*

chmod 0640 /dev/mapper/voting*

• Two options for 10g RAC

a. Use 11g clusterware for 10g RAC

b. Map block devices to raw devices by udev rules:/etc/udev/rules.d/65-rasw.rules:

ACTION=="add", KERNEL=="emcpowera1", RUN+="/bin/raw /dev/raw/raw1 %N"

/etc/udev/rules.d/89-raw_permissions.rules:

KERNEL=="raw1",OWNER="root", GROUP="oinstall", MODE="640"

c. Start udev: /sbin/udev

Network Configuration

Public IP and Virtual IP configuration– Virtual IP for fast database connection failover

• Avoid a up to 10 minutes TCP/IP timeout

• Automatically failover to other node$srvctl status nodeapps -n kblade1

VIP is running on node: kblade1

GSD is running on node: kblade1

Listener is running on node: kblade1

ONS daemon is running on node: kblade1

When kblade1 fails, kblade1-vip is failed over to kblade2:$ srvctl status nodeapps -n kblade1

VIP is running on node: kblade2

GSD is not running on node: kblade1

Listener is not running on node: kblade1

ONS daemon is not running on node: kblade1

$ ping kblade1-vip

PING 155.16.9.171 (155.16.9.171) 56(84) bytes of data.

From 155.16.0.1 icmp_seq=9 Destination Host Unreachable

From 155.16.0.1 icmp_seq=9 Destination Host Unreachable

….. (waiting for 2 seconds before being failed over)

64 bytes from 155.16.9.171: icmp_seq=32 ttl=64 time=2257 ms

64 bytes from 155.16.9.171: icmp_seq=33 ttl=64 time=1258 ms

Network Configuration

Private Interconnection Configuration– Fully Redundant Ethernet Interconnects:

Two NIC cards, two non-roundtable interconnect switches

– NIC teaming to bond two network interfaces for failoverifcfg-eth1: ifcfg-eth2: ifcfg-bond0:DEVICE=eth1 DEVICE=eth2

USERCTL=no USERCTL=no IPADDR=192.168.9.52

ONBOOT=yes ONBOOT=yes NETMASK=255.255.255.0

MASTER=bond0 MASTER=bond0 ONBOOT=yes

SLAVE=yes SLAVE=yes BOOTPROTO=none

BOOTPROTO=none BOOTPROTO=none USERCTL=no

TYPE=Ethernet TYPE=Ethernet

Add: alias bond0 bonding

options bonding miimon=100 mode=1 in /etc/modproble.conf

Network Configuration

Private Interconnection Configuration– Configuration best practices from Oracle. Refer to [7]

• Set UDP send/receive buffers to max

• Use the same interconnect for both Oracle clusterware and Oracle RAC communication

• NIC settings for interconnect: 1. Define control : rx=on, tx=off

2. Ensure NIC names/slots order identical on all nodes:

3. Configure interconnect NICs on fastest PCI bus

4. Compatible switch settings:

802.3ad on NICs = 802.3ad on switch ports

MTU=9000 on NICs = MTU=9000 on switch ports

• Recommended Linux kernel configuration for networking:

11gR1 10gR2

net.core.rmem_default = 4194304 262144

net.core.rmem_max = 4194304 262144 net.core.wmem_default = 262144 262144 net.core.wmem_max = 262144 262144

• Network Heartbeat Misscount: 60 secs for 10g, 30 secs for 11g

• Hangcheck timer value:

modprobe hangcheck-timer hangcheck_tick=1

hangcheck_margin=10 hangcheck_reboot=1

Managing Oracle clusterware

Managing voting disk– Locate voting disk: crsctl query votedisk css

– Backup/restore using dd command for example:

dd if=/dev/mapper/votingdisk1p1 of=/backup/vd bs=4096

dd if=/backup/vd of=/dev/mapper/votingdisk1p1 bs=4096

– Adding/removing voting disk:

crsctl add css votedisk /dev/mapper/votingdisk3p1 –force

crsctl delete css votedisk /dev/mapper/votingdisk2p1 –force

Managing OCR– Three tools: OCRCONFIG, OCRDUMP and OCRCHECK

– Add mirror: ocrconfig –replace ocrmirror /dev/mapper/ocr2p1

– replace OCR: ocrconfig –replace ocr /u01/ocr/ocr1

– OCR backup: Clusterware automatically backs up OCR

show backup: ocrconfig –showbackup

– Restore OCR: ocrconfig –restore file_name

– Manual backup: ocrconfig –manualbackup

– OCR export/import:

ocrconfig –export /home/oracle/ocr_export

ocrconfig –import /home/oracle/ocr_export

cluvfy comp ocr

Managing Oracle clusterware

Extending Cluster by Cloning Oracle Clusterware – Task:

• Existing cluster: k52950-3-n1 and k52950-3-n2

• Add new node: k52950-3-n3

– Step 1: Pre-requisite tasks on new node k52950-3-n3 :

OS install, share storage for OCR and voting disk

Network configuration: public, private and VIP

– Step 2:

• Create CRS home backup on the source node k52950-3-n1

• Copy CRS home backup to the new node

• Shutdown CRS in the source node

• Remove all the log files and trace files from the backup

• Create oracle inventory in the new node

• Set ownership for oracle inventory

• Run preupdate.sh in the new node.

– Step3: Run CRS clone process in the new node:

Managing Oracle clusterware

•

Managing Oracle clusterware

•

Managing Oracle clusterware

Execute ./root.sh as root on new node k52950-3-n3 as instructed

Step 4: Run addNode.sh on the source node k52950-3-n1:

•

Managing Oracle clusterware

Start CRS on node1 k52950-3-n1 and

execute rootaddnode.sh on the source node k52950-3-n1

Managing Oracle clusterware

Execute root.sh on the new node:

•

Restart CRS on node 2: k52950-3-n2:

[root@k52950-3-n2 bin]# ./crsctl start crs

Clusterware Troubleshooting

Split Brain Condition and IO Fencing Mechanism• Split brain condition: a node failure partitions the cluster into

multiple sub-clusters without knowledge of the existence of others.

• Consequence: data collision and corruption

• IO fencing: fencing the failed node off from all the IOs: STOMITH

• Node eviction: pick a cluster node as victim to reboot.

Always keep the largest cluster possible up, evicted other nodes

two nodes: keep the lowest number node up and evict other

• Two CSS heartbeats and misscounts to detect node eviction

• Network heartbeat (NHB) : cross the private interconnect

establish and confirm valid node membership

CSS misscount:

the maximal # of seconds

complete heartbeat

without trigging a node

eviction

• Disk heartbeat : between the cluster node and voting disk

CSS misscount, the default is 200 seconds for 10.2.0.1 and up

•

Clusterware Troubleshooting

Node Eviction Diagnosis Case Study – Random node evicted in a 11-node 10g cluster on Linux:

/var/log/messages: Jul 23 11:15:23 racdb7 logger: Oracle clsomon failed with fatal status 12.

Jul 23 11:15:23 racdb7 logger: Oracle CSSD failure 134.

Jul 23 11:15:23 racdb7 logger: Oracle CRS failure. Rebooting for cluster integrity

OCSSD log: $CRS_HOME/log/<hostname>/cssd/ocssd.log file[ CSSD]2008-07-23 11:14:49.150 [1199618400] >WARNING:

clssnmPollingThread: node racdb7 (7) at 50% heartbeat fatal, eviction in 29.720 seconds

..

clssnmPollingThread: node racdb7 (7) at 90% heartbeat fatal, eviction in 0.550 seconds

…

[ CSSD]2008-07-23 11:15:19.079 [1220598112] >TRACE:

clssnmDoSyncUpdate: Terminating node 7, racdb7, misstime(60200) state(3)

– Root cause analysis:

• A network heartbeat failure triggered a node eviction on node 7

• Private IP node not pingable right before the node eviction

• Public and private shared a single physical switch

– Solution : Use two dedicated switches for interconnect

– Result: no more node eviction after the switch change

•

Clusterware Troubleshooting

CRS Reboots Troubleshooting Procedure – OCSSD, OPROCD and OCLSOMON monitoring processes

• Detect certain conditions of impacting data integrity

• Kill clusterware and trigger CRS reboot

• Leave critical errors in the related log files

– Troubleshooting Methods:• Review syslog file /var/log/messages in Linux

• OCSSD log ocssd.log node eviction due to internal health

error, interconnect and membership error.

• PROCD log : /etc/oracle/oprocd/<host>.oprocd.log node

eviction due to hardware and driver freeze

• OCLSOMON log oclsmon.log for rebooting due to

hangs/scheduling issues.

– Troubleshooting Flowchart

Clusterware Troubleshooting

Clusterware Troubleshooting

RAC Diagnostic Tools:– Diagwait:

• Delay the node reboot for a short time to write all

diagnostic messages to the logs.

• Doesn’t increase of probability of data corruption

• Setup steps: shutdown CRS

crsctl set css diagwait 13 –force

restart CRS

– Oracle problem detections tool(IPD/OS)• Monitor and record resources degradation and failure

related to oracle clusterware and Oracle RAC issue

• Historical mode goes back to the time before node eviction

• Linux x86 kernel 2.6.9 up.

– RAC-RDDT and Oswatcher

• Collect information leading up to the time of reboot.

• From OS utilities: netstat, iostat, vmstat

• Start: ./startOSW.sh 60 10

• Metalink #301138.1, #301137.1

References

[1] Oracle 10g Grid Real Applications Clusters Oracle 10g Grid Computing

with RAC, Mike Ault, Madhu Tumma

[2] Oracle Metalink Note: # 294430.1, CSS Timeout Computation in Oracle

Clusterware

[3] Oracle Metalink Note: # 265769.1, Troubleshooting CRS Reboots

[4] Oracle Clusterware Administration and Deployment Guide 11g Release 1

(11.1) September 2007

[5] Deploying Oracle Database 11g R1 Enterprise Edition Real Application

Clusters with Red Hat Enterprise Linux 5.1 and Oracle Enterprise

Linux 5.1 On Dell PowerEdge Servers, Dell/EMC Storage, Kai Yu

http://www.dell.com/downloads/global/solutions/11gr1_ee_rac_on_rhel5_1__and_

OEL.pdf?c=us&cs=555&l=en&s=biz

[6] Dell | Oracle Supported configurations: Oracle Enterprise Linux 5.2

Oracle 11g Enterprise Edition Deployment Guide:

http://www.dell.com/content/topics/global.aspx/alliances/en/oracle_11g_oracle_ent

_linux_4_1?c=us&cs=555&l=en&s=biz

[7] Oracle Real Applications Clusters Internals, Oracle Openworld 2008

presentation #298713, Barb Lundhild

[8] Looking Under the hood at Oracle Clusterware, Oracle Openworld 2008

presentation #299963, Murali Vallath

Q/A

IOUG Session Survey

Session Title:

Best Practices of Oracle 10g/11g Clusterware:

Configuration, Administration and Troubleshooting

Session Speaker:

Kai Yu

Session Number:

355