2512680 cookbook 11grac r1 asm aix5l san storage installation guide
Post on 30-May-2018
219 Views
Preview:
TRANSCRIPT
-
8/14/2019 2512680 Cookbook 11grac r1 Asm Aix5l San Storage Installation Guide
1/393
DRAFT
COOKBOOK
I ns ta l l a t i on G u i deOracle 11g RAC Release 1with Oracle AutomaticStorage Management (ASM)on IBM System p
-
8/14/2019 2512680 Cookbook 11grac r1 Asm Aix5l San Storage Installation Guide
2/393
This document is based on our experiences.
This is not an official (Oracle or IBM) documentation.
This document will be constantly updated
and were open to any add-on
or feedback from your own experiences,on same or different storage solution !!!
Document history :
Version Date Update Who Validated by
1.0 January 2008 - Creation Frederic MichiaraThierry Plumeau
Alain RoyPaul Bramy
April 2008 - Review Dider WojciechowskiPaul Bramy
C t ib t
-
8/14/2019 2512680 Cookbook 11grac r1 Asm Aix5l San Storage Installation Guide
3/393
1 The aim of this document ............................................... ......................... 62 About Oracle clusterware ................................................. ....................... 8
3 About Oracle Automatic storage management ........................................ 154 About Real Application Cluster ............................................................. .. 175 About 11G RAC / ASM on AIX .......................................................... ........ 196 Whats new with 11g RAC implementation on AIX ................................... 207 infrastructure requirements .................................... .............................. 20
7.1 General Requirements ..................................................................................................... ...............227.1.1 About Servers and processors ................................................................................. ................227.1.2 About RAC on IBM System p ............................................................................ ........................23
7.1.3 About Network ........................................................................................................... ..............247.1.4 About SAN Storage ................................................................................................. .................267.1.5 Proposed infrastructure with 2 servers ................................................................... .................287.1.6 What do we protect ? ....................................................................................................... ........297.1.7 About IBM Advanced Power Virtualization and RAC .................................................... .............30
7.2 Cookbook infrastructure ...................................................................................... ...........................357.2.1 IBM Sytem p servers .................................................................................................... ............367.2.2 Operating System ............................................................................................................ ........447.2.3 Multi-pathing and ASM ............................................................................ ................................45
7.2.4 IBM storage and multi-pathing ........................................................................ ........................467.2.5 EMC storage and multi-pathing ........................................................................... ....................537.2.6 HITACHI storage and multi-pathing ........................................................... ..............................557.2.7 Others, StorageTek, HP EVA storage and multi-pathing ....................................................... ....57
8 Specific cOnsiderations for RAC/ASM setup with HACMP installed ........... 599 Installation steps ........................................................ .......................... 6210 Preparing the system ............................................... ........................... 64
10.1 Network configuration ........................................................................................ ..........................65
10 1 1 Define Networks layout Public Virtual and Private Hostnames 65
-
8/14/2019 2512680 Cookbook 11grac r1 Asm Aix5l San Storage Installation Guide
4/393
11.2.9 Formating the virtual devices (zeroing) .................................................................... ...........12211.3 ASM disks .................................................................................................... ...............................123
11.3.1 Required LUNs ......................................................................................... ...........................12511.3.2 How to Identify if a LUN is used or not ? ........................................................... ...................12511.3.3 Register LUNs at AIX level.............................................................................................. .....12711.3.4 Identify LUNs and corresponding hdisk on each node ...................................... ..................12911.3.5 Removing reserve lock policy on hdisks from each node ................................. ...................13211.3.6 Identify Major and Minor number of hdisk on each node .................................................... .13411.3.7 Create Unique Virtual Device to access same LUN from each node ....................................13611.3.8 Set Ownership / Permissions on Virtual Devices ........................................................ ..........13811.3.9 Formating the virtual devices (zeroing) .................................................................... ...........141
11.3.10 Removing assigned PVID on hdisk ................................................................................. ....14211.4 Checking Shared Devices ..................................................................................... ......................14311.5 Recommandations, hints and tips ................................................................ ..............................145
11.5.1 OCR / Voting disks .................................................................................. .............................14511.5.2 ASM disks ................................................................................................ ............................148
12 Oracle Clusterware (CRS) Installation ................................................ 15112.1 Cluster VeriFication utility ............................................................................................ ...............152
12.1.1 Understanding and Using Cluster Verification Utility ................................................... ........15212.1.2 Using CVU to Determine if Installation Prerequisites are Complete ............................ .........152
12.2 Installation ....................................................................................................................... ...........16312.3 Post Installation operations ............................................................................. ...............................4
12.3.1 Update the Clusterware unix user .profile ........................................................ .......................412.3.2 Verify parameter CSS misscount .................................................................. ...........................512.3.3 Cluster Ready Services Health Check ....................................................................... ...............612.3.4 Adding enhanced crsstat script ....................................................................... .......................912.3.5 Interconnect Network configuration Checkup ................................................................ ........1112.3.6 Oracle CLuster Registry content Check and Backup .............................................. ................12
12.4 Some usefull commands ................................................................................. .............................19
12 5 Accessing CRS logs 21
-
8/14/2019 2512680 Cookbook 11grac r1 Asm Aix5l San Storage Installation Guide
5/393
15.7 About DBCONSOLE ........................................................................................... ..........................16715.7.1 Checking DB CONSOLE ........................................................................................... .............16715.7.2 Moving from dbconsole to Grid Control.......................................................... .....................171
16 asm advanced tools ............................................................. .............. 17216.1 ftp and http access .................................................................................................. ...................172
17 Some usefull commands ............................. ....................................... 17317.1 Oracle CLuster Registry content Check and Backup ................................................. ..................174
18 Appendix A : Oracle / IBM technical documents ................................... 17519 Appendix B : Oracle technical notes .................................................. .. 176
19.1 CRS and 10g Real Application Clusters ................................................................................... ....17619.2 About RAC ....................................................................................................................... ...........186
19.3 About CRS .......................................................................................................................... ........18619.4 About VIP ............................................................................................................. ......................18719.5 About manual database cration ................................................................................................ .18719.6 About Grid Control .................................................................................................. ...................18719.7 About TAF .................................................................................................... ..............................18719.8 About Adding/Removing Node ..................................................................................... ..............18719.9 About ASM .......................................................................................................... .......................18719.10 Metalink note to use in case of problem with CRS .................................................... ..............188
20 Appendix C : Usefull commands ...................................................... .... 18921 Appendix D : Empty tables to use for installatioN ................................ 19921.1 Network document to ease your installation ......................................................... .....................19921.2 Steps Checkout .................................................................................................................. .........20121.3 Disks document to ease disks preparation in your implementation .............................. .............202
22 Documents, BOOKS to look at .. ........................................................ .. 203
-
8/14/2019 2512680 Cookbook 11grac r1 Asm Aix5l San Storage Installation Guide
6/393
1 THE AIM OF THIS DOCUMENT
This document is written to provide help installingOracle11gReal Application Clusters (11.1) release 1 withOracle Automated Storage Management on IBM System p
and i servers with AIX.
Within the Oracle/IBM Joint Solutions Center, we are receiving a lot of request about RAC and ASM, this isthe reason why we decided to deliver the first 11gRAC / AIX cookbook with Oracle ASM (AutomatedStorage Management). AIX6 is not covered as 11gRAC is not yet supported with AIX6, but certificationshould be available soon. Check on metalink for update, the cookbook will be updated with AIX6 whenofficial certification will be released.
We will describe step by step the architecture Oracle CRS (Cluster Ready Service) on raw disks and database on ASM(Automatic Storage Management).
For the architecture using Oracle 11gRAC with IBM GPFS as cluster files system, cookbook named InstallationGuideOracle 11gRAC Release 1 and IBM GPFS on IBM eServer System p and i running AIX with SAN Storage will be released soon.
M t li k (htt // t li k l / t li k/ l l/ l2 i t t )
http://metalink.oracle.com/metalink/plsql/ml2_gui.startuphttp://metalink.oracle.com/metalink/plsql/ml2_gui.startuphttp://metalink.oracle.com/metalink/plsql/ml2_gui.startup -
8/14/2019 2512680 Cookbook 11grac r1 Asm Aix5l San Storage Installation Guide
7/393
The information contained in this paper resulted from :
-Oracle and IBM documentations
-Workshop experiences done in the Oracle/IBM Joint Solutions Center
-Benchmarks and POC implementations for customers performed by PSSC Montpellier
-This documentation is a joint effort from Oracle and IBM specialists.
Please also refer to Oracle & IBM online documentation for more information :
http://docs.oracle.com
o Oracle Database 11gRelease 1 (11.1) Documentationhttp://www.oracle.com/technology/documentation/database.html
http://tahiti.oracle.com
Oracle RAC home page : http://www.oracle.com/database/rac_home.html
For more information IBM System p : http://www-03.ibm.com/systems/p/
Your comments are important for us, and we thanks the ones who send us their feedback about previousrelease, and about how this document did help them in their implementation. We want our technical papers tobe as helpful as possible.
Please send us your comments about this document to the Oracle/IBM Joint Solutions Center.
U il dd O h b
http://docs.oracle.com/http://www.oracle.com/technology/documentation/database.htmlhttp://www.oracle.com/database/rac_home.htmlhttp://www-03.ibm.com/systems/p/http://docs.oracle.com/http://www.oracle.com/technology/documentation/database.htmlhttp://www.oracle.com/database/rac_home.htmlhttp://www-03.ibm.com/systems/p/ -
8/14/2019 2512680 Cookbook 11grac r1 Asm Aix5l San Storage Installation Guide
8/393
2 ABOUT ORACLE CLUSTERWARE
Extract from : Oracle Database 2 Day + Real Application Clusters Guide 11gRelease 1 (11.1)Part Number B28252-02http://www.oracle.com/pls/db111/homepage?remark=tahiti
Oracle Real Application Clusters (Oracle RAC) uses Oracle Clusterware as the infrastructure that binds togethermultiple nodes that then operate as a single server. Oracle Clusterware is a portable cluster management solution thatis integrated with Oracle Database. In an Oracle RAC environment, Oracle Clusterware monitors all Oraclecomponents (such as instances and listeners). If a failure occurs, Oracle Clusterware automatically attempts to restartthe failed component and also redirects operations to a surviving component.
Oracle Clusterware includes a high availability framework for managing any application that runs on your cluster.Oracle Clusterware manages applications to ensure they start when the system starts. Oracle Clusterware alsomonitors the applications to make sure that they are always available. For example, if an application process fails, thenOracle Clusterware attempts to restart the process based on scripts that you customize. If a node in the cluster fails,then youcan program application processes that typically run on the failed node to restart on another node in thecluster.
Oracle Clusterware includes two important components: the voting disk and the OCR. The voting disk is a file thatmanages information about node membership, and the OCR is a file that manages cluster and Oracle RAC database
configuration information.
The Oracle Clusterware installation process creates the voting disk and the OCR on shared storage. If you select theoption for normal redundant copies during the installation process, then Oracle Clusterware automatically maintainsredundant copies of these files to prevent the files from becoming single points of failure. The normal redundancyfeature also eliminates the need for third-party storage redundancy solutions. When you use normal redundancy,Oracle Clusterware automatically maintains two copies of the OCR file and three copies of the voting disk file.
Y ill l b t th ti f th O l l t h t b ild th l t d th t t f
http://www.oracle.com/pls/db111/homepage?remark=tahitihttp://www.oracle.com/pls/db111/homepage?remark=tahiti -
8/14/2019 2512680 Cookbook 11grac r1 Asm Aix5l San Storage Installation Guide
9/393
Oracle Clusterware, Group Membership and Heartbeats
Cluster needs to know who is a member at all times
Oracle Clusterware has 2 heartbeats
Network heartbeat
If a node does not send a heartbeat forMissCount (time in seconds), then node isevicted from cluster.
Disk heartbeat
If disk heartbeat is not updated in I/Otimeout, then node is evicted from cluster
Oracle Clusterware, OCR and Voting disks
Oracle Cluster Registry contains allinformation on the cluster.
Since 10gRAC R2 OCR can bemirrored
Voting is the cluster heartbeat.
Since 10gRAC R2 voting can have
multiple copies in odd number formajority.
11gRAC/ASM/AIX oraclibm@fr.ibm.com 9 of 393
-
8/14/2019 2512680 Cookbook 11grac r1 Asm Aix5l San Storage Installation Guide
10/393
Oracle Clusterware and its virtual IPs
- 1 -
Each node participating to the Oracle Cluster has :
1 Static IP address (10.3.25.81)
1 Virtual IP address, each virtual address(10.3.25.181) has its home node
- 2 -
If one node fails :
Static IP (10.3.25.81) from failed node wil notbe anymore reachable
Virtual IP (10.3.25.181) from failed node willswitch to one of the remaining node, and willremain reachable
- 3 -
When failed node return to normal operation :
Static IP (10.3.25.81) will be back andreachable
Virtual IP (10.3.25.181) from failed node,hosted on one of the remaining nodes willswitch back to its home node
11gRAC/ASM/AIX oraclibm@fr.ibm.com 10 of 393
-
8/14/2019 2512680 Cookbook 11grac r1 Asm Aix5l San Storage Installation Guide
11/393
Protecting Single Instance Database and Thrid application tier thru Oracle Clusterware
- 1 -
Using Oacle clusterware, we can have Cluster Databases, butalso non cluster database. In our exemple, we have one singleinstance database on one node.
Each node has its own node VIP, and we can create anapplication VIP (APPS VIP) on wich different resources aredependents :
ASM Single instance,
HR Single Database Instance,
Listener
HR application tier
/apps and /oracle local files systems
- 2 -
If first node where APPS VIP is hosted failed THEN :
Node VIP (10.3.25.81) will switch to an other node
APPS VIP (10.3.25.200) will switch to a preffered node (if configured, inj our caseon the third node. If third node was not available, APPS VIP will switch to one available node).
11gRAC/ASM/AIX oraclibm@fr.ibm.com 11 of 393
-
8/14/2019 2512680 Cookbook 11grac r1 Asm Aix5l San Storage Installation Guide
12/393
- 3 -
APPS VIPS has switched to preferred 3rd node. THEN, All resourcesdependant of APPS VIP will be restarted on third node.
- 4 -
When node 1 come back to normal operation, node VIP (10.3.25.81) will switch back
automatically to its home node, meaning first node. BUT APPS VIPS will not switchback !
- 5 -
To switch back APPS VIP and its dependant resources to first node, administrator wil l have to relocate the APPS VIP to first node thru clusterware commands.
Sending command on cluster events thru Oracle Clusterware
11gRAC/ASM/AIX oraclibm@fr.ibm.com 12 of 393
-
8/14/2019 2512680 Cookbook 11grac r1 Asm Aix5l San Storage Installation Guide
13/393
- 1 -
With Oracle clusterware, all cluster events(node, database, instance, etc )information (up, down, etc ) can beobtained and used to associate an action.
In our example case, we have :
3 nodes, with 3 LPARs each (RAC, APPS,TEST/DEV)
IBM HMC to administrate the IBM POWERSystem p and LPARs (cores, memory)
In normal operation, all LPARs are usingtheir defined cores and memory, butcould also use micro-partitionningbetween LPARs.
- 2 -
In our example case, if first node fails, THEN :
Node VIP (10.3.25.81) from first node will switchon one available node.
Oracle clusterware will have a FAN (FastApplication Notification) Event : node1 down
A unix shell script (Call out script) will analyze theinformation and send a request to the IBM HMC.
o Request to remove resources from
TEST/DEV LPARs from second and thirdnode.
o Request to assign new resources to APPSand RAC LPARs.
- 3 -
THEN, the HMC will execute the requests, and willresize tle LPARs as requested on each availablenodes.
11gRAC/ASM/AIX oraclibm@fr.ibm.com 13 of 393
-
8/14/2019 2512680 Cookbook 11grac r1 Asm Aix5l San Storage Installation Guide
14/393
For more information :
On http://otn.oracle.comhttp://www.oracle.com/technology/products/database/clusterware/index.html
Oracle Clusterware
Oracle Clusterware is a portable cluster software that allows clustering of single servers so that they cooperate as a
single system. Oracle Clusterware also provides the required infrastructure for Oracle Real Application Clusters (RAC).In addition Oracle Clusterware enables the protection of any Oracle application or any other kind of application within acluster. In any case Oracle Clusterware is the intelligence in those systems that ensures the required cooperationbetween the cluster nodes.
Oracle Clusterware Oracle Clusterware Whitepaper (PDF)
Oracle Clusterware Technical Articles Using standard NFS to support a third voting disk on an Extended Distance
cluster configuration on Linux, AIX, HP-UX, or Solaris (PDF) February 2008 Oracle Homes in an Oracle Real Application Clusters Environment (PDF) February 2008
Using Oracle Clusterware to protect any kind of application Using Oracle Clusterware to Protect 3rd Party Applications (PDF) February 2008
Using Oracle Clusterware to Protect 3rd Party Applications (PDF) February 2008
Using Oracle Clusterware to Protect Oracle Application Server (PDF) November 2005
U i O l Cl t t P t t O l D t b 10 ith
http://otn.oracle.com/http://www.oracle.com/technology/products/database/clusterware/index.htmlhttp://www.oracle.com/technology/products/database/clusterware/pdf/TWP_Clusterware_11g.pdfhttp://www.oracle.com/technology/products/database/clustering/pdf/thirdvoteonnfs.pdfhttp://www.oracle.com/technology/products/database/clustering/pdf/thirdvoteonnfs.pdfhttp://www.oracle.com/technology/products/database/clustering/pdf/oh_rac.pdfhttp://www.oracle.com/technology/products/database/clusterware/pdf/TWP-Oracle-Clusterware-3rd-party.pdfhttp://www.oracle.com/technology/products/database/clusterware/pdf/TWP-Oracle-Clusterware-3rd-party.pdfhttp://www.oracle.com/technology/products/database/clusterware/pdf/Using-Clusterware-to-protect-Appserver.pdfhttp://www.oracle.com/technology/products/database/clusterware/pdf/EM-DB-Failover.pdfhttp://otn.oracle.com/http://www.oracle.com/technology/products/database/clusterware/index.htmlhttp://www.oracle.com/technology/products/database/clusterware/pdf/TWP_Clusterware_11g.pdfhttp://www.oracle.com/technology/products/database/clustering/pdf/thirdvoteonnfs.pdfhttp://www.oracle.com/technology/products/database/clustering/pdf/thirdvoteonnfs.pdfhttp://www.oracle.com/technology/products/database/clustering/pdf/oh_rac.pdfhttp://www.oracle.com/technology/products/database/clusterware/pdf/TWP-Oracle-Clusterware-3rd-party.pdfhttp://www.oracle.com/technology/products/database/clusterware/pdf/TWP-Oracle-Clusterware-3rd-party.pdfhttp://www.oracle.com/technology/products/database/clusterware/pdf/Using-Clusterware-to-protect-Appserver.pdfhttp://www.oracle.com/technology/products/database/clusterware/pdf/EM-DB-Failover.pdf -
8/14/2019 2512680 Cookbook 11grac r1 Asm Aix5l San Storage Installation Guide
15/393
3 ABOUT ORACLE AUTOMATIC STORAGE MANAGEMENT
Extract from : Oracle Database 2 Day + Real Application Clusters Guide 11gRelease 1 (11.1)Part Number B28252-02http://www.oracle.com/pls/db111/homepage?remark=tahiti
With Oracle RAC, each instance must have access to the datafiles and recovery files for the Oracle RAC database.Using Automatic Storage Management (ASM) is an easy way to satisfy this requirement.
ASM is an integrated, high-performance database file system and disk manager. ASM is based on the principle that thedatabase should manage storage instead of requiring an administrator to do it. ASM eliminates the need for you todirectly manage potentially thousands of Oracle database files.
ASM groups the disks in your storage system into one or more disk groups. You manage a small set of disk groups andASM automates the placement of the database files within those disk groups.
ASM provides the following benefits:
StripingASM spreads data evenly across all disks in a disk group to optimize performance and utilization.This even distribution of database files eliminates the need for regular monitoring and I/O performance tuning.
MirroringASM can increase data availability by optionally mirroring any file. ASM mirrors at the file level,unlike operating system mirroring, which mirrors at the disk level. Mirroring means keeping redundant copies,or mirrored copies, of each extent of the file, to help avoid data loss caused by disk failures. The mirrored copyof each file extent is always kept on a different disk from the original copy. If a disk fails, ASM can continue toaccess affected files by accessing mirrored copies on the surviving disks in the disk group.
Online storage reconfiguration and dynamic rebalancingASM permits you to add or remove disks fromyour disk storage system while the database is operating. When you add a disk to a disk group, ASMautomatically redistributes the data so that it is evenly spread across all disks in the disk group, including thenew disk. The process of redistributing data so that it is also spread across the newly added disks is known as
b l i It i d i th b k d d ith i i l i t t d t b f
http://www.oracle.com/pls/db111/homepage?remark=tahitihttp://www.oracle.com/pls/db111/homepage?remark=tahiti -
8/14/2019 2512680 Cookbook 11grac r1 Asm Aix5l San Storage Installation Guide
16/393
For more information :
On http://otn.oracle.com
http://www.oracle.com/technology/products/database/asm/index.html
Automatic Storage Management
Automatic Storage Management (ASM) is a feature in Oracle Database 10g/11g that provides the databaseadministrator with a simple storage management interface that is consistent across all server and storage platforms. Asa vertically integrated file system and volume manager, purpose-built for Oracle database files, ASM provides theperformance of async I/O with the easy management of a file system. ASM provides capability that saves the DBAstime and provides flexibility to manage a dynamic database environment with increased efficiency.
What's New for ASM in Oracle Database 11g
11g ASM New Features Technical White PaperOracle Database 11g ASM new features technical white paper
ASM Overview and Technical Papers ASM and Multipathing Best Practices and Information Matrix (PDF)
Multipathing software and ASM information matrix and best practices guide Oracle Database 10gRelease 2 ASM - New Features White Paper(PDF)
This paper discusses the enhancements to ASM which are new in Release 2 of Oracle Database 10g. ASM Overview and Technical Best Practices White Paper(PDF)
This paper discusses the basics of ASM such as the steps to add disks, create a diskgroup, and createa database within ASM, emphasizing best practices.
T k th G k O t f D t b IO T i Whit P (PDF)
http://otn.oracle.com/http://www.oracle.com/technology/products/database/asm/index.htmlhttp://www.oracle.com/technology/products/database/asm/pdf/11gr1%20asm%20new%20features%20wp%2005-2007.pdfhttp://www.oracle.com/technology/products/database/asm/pdf/asm%20and%20multipathing%20best%20practices%20info%20matrix%203-5-08.pdfhttp://www.oracle.com/technology/products/database/asm/pdf/asm%20r2%20new%20features.pdfhttp://www.oracle.com/technology/products/database/asm/pdf/asm%20r2%20new%20features.pdfhttp://www.oracle.com/technology/products/database/asm/pdf/asm%20r2%20new%20features.pdfhttp://www.oracle.com/technology/products/database/asm/pdf/asm_10gr2_bestpractices%2009-07.pdfhttp://www.oracle.com/pls/wocprod/url/ITEM/0A94991092E93070E040149032B17CE1http://otn.oracle.com/http://www.oracle.com/technology/products/database/asm/index.htmlhttp://www.oracle.com/technology/products/database/asm/pdf/11gr1%20asm%20new%20features%20wp%2005-2007.pdfhttp://www.oracle.com/technology/products/database/asm/pdf/asm%20and%20multipathing%20best%20practices%20info%20matrix%203-5-08.pdfhttp://www.oracle.com/technology/products/database/asm/pdf/asm%20r2%20new%20features.pdfhttp://www.oracle.com/technology/products/database/asm/pdf/asm_10gr2_bestpractices%2009-07.pdfhttp://www.oracle.com/pls/wocprod/url/ITEM/0A94991092E93070E040149032B17CE1 -
8/14/2019 2512680 Cookbook 11grac r1 Asm Aix5l San Storage Installation Guide
17/393
4 ABOUT REAL APPLICATION CLUSTER
Extract from : Oracle Database 2 Day + Real Application Clusters Guide 11gRelease 1 (11.1)Part Number B28252-02http://www.oracle.com/pls/db111/homepage?remark=tahiti
Oracle RAC extends Oracle Database so that you can store, update, and efficiently retrieve data using multipledatabase instances on different servers at the same time. Oracle RAC provides the software that facilitates serversworking together in what is called a cluster. The data files that make up the database must reside on shared storagethat is accessible from all servers that are part of the cluster. Each server in the cluster runs the Oracle RAC software.
An Oracle Database database has a one-to-one relationship between datafiles and the instance. An Oracle RACdatabase, however, has a one-to-many relationship between datafiles and instances. In an Oracle RAC database,multiple instances access a single set of database files. The instances can be on different servers, referred to as hostsor nodes. The combined processing power of the multiple servers provides greater availability, throughput, andscalability than is available from a single server.
Each database instance in an Oracle RAC database uses its own memory structures and background processes.Oracle RAC uses Cache Fusion to synchronize the data stored in the buffer cache of each database instance. CacheFusion moves current data blocks (which reside in memory) between database instances, rather than having onedatabase instance write the data blocks to disk and requiring another database instance to reread the data blocks from
disk. When a data block located in the buffer cache of one instance is required by another instance, Cache Fusiontransfers the data block directly between the instances using the interconnect, enabling the Oracle RAC database toaccess and modify data as if the data resided in a single buffer cache.
Oracle RAC is also a key component for implementing the Oracle enterprise grid computing architecture. Having ult ipledatabase instances accessing a single set of datafiles prevents the server from being a single point of failure. Anypackaged or custom application that ran well on a Oracle Database will perform well on Oracle RAC without requiringcode changes.
You will learn more about the operation of the Oracle RAC database in a cluster, how to build the cluster, and the
t t f O l RAC d t b i th ti f thi id
http://www.oracle.com/pls/db111/homepage?remark=tahitihttp://www.oracle.com/pls/db111/homepage?remark=tahiti -
8/14/2019 2512680 Cookbook 11grac r1 Asm Aix5l San Storage Installation Guide
18/393
For more information :
On http://otn.oracle.com
http://www.oracle.com/technology/products/database/clustering/index.html
Oracle Real Application Clusters
Oracle Real Application Clusters (RAC) is an option to the award-winning Oracle Database Enterprise Edition. Oracle
RAC is a cluster database with a shared cache architecture that overcomes the limitations of traditional shared-nothingand shared-disk approaches to provide highly scalable and available database solutions for all your businessapplications.Oracle Database Standard Edition includes Real Application Clusters support for higher levels of system uptime.
Real Application Clusters 11g Oracle Real Application Clusters Datasheet (PDF) July 2007
Oracle Real Application Clusters 11gTechnical Overview(PDF) July 2007
Workload Management with Oracle Real Application Clusters (FAN, FCF, Load Balancing)(PDF) July
2007 NEW Oracle Homes in an Oracle Real Application Clusters Environment(PDF) January 2008
UPDATED Using standard NFS to support a third voting disk on an Extended Distance clusterconfiguration on Linux,AIX, HP, or Solaris (PDF) February 2008
http://otn.oracle.com/http://www.oracle.com/technology/products/database/clustering/index.htmlhttp://www.oracle.com/technology/products/database/clustering/pdf/ds_rac11g.pdfhttp://www.oracle.com/technology/products/database/clustering/pdf/twp_rac11g.pdfhttp://www.oracle.com/technology/products/database/clustering/pdf/twp_rac11g.pdfhttp://www.oracle.com/technology/products/database/clustering/pdf/twp_rac11g.pdfhttp://www.oracle.com/technology/products/database/clustering/pdf/awmrac11g.pdfhttp://www.oracle.com/technology/products/database/clustering/pdf/oh_rac.pdfhttp://www.oracle.com/technology/products/database/clustering/pdf/thirdvoteonnfs.pdfhttp://www.oracle.com/technology/products/database/clustering/pdf/thirdvoteonnfs.pdfhttp://otn.oracle.com/http://www.oracle.com/technology/products/database/clustering/index.htmlhttp://www.oracle.com/technology/products/database/clustering/pdf/ds_rac11g.pdfhttp://www.oracle.com/technology/products/database/clustering/pdf/twp_rac11g.pdfhttp://www.oracle.com/technology/products/database/clustering/pdf/awmrac11g.pdfhttp://www.oracle.com/technology/products/database/clustering/pdf/oh_rac.pdfhttp://www.oracle.com/technology/products/database/clustering/pdf/thirdvoteonnfs.pdfhttp://www.oracle.com/technology/products/database/clustering/pdf/thirdvoteonnfs.pdf -
8/14/2019 2512680 Cookbook 11grac r1 Asm Aix5l San Storage Installation Guide
19/393
5 ABOUT 11G RAC / ASM ON AIX
11gRAC with OCR (Oracle Cluster Registry) disk(s), and Voting (Heartbeat) disk(s) on raw disks, and databaseon Oracle ASM (Automated Storage Management).
-
8/14/2019 2512680 Cookbook 11grac r1 Asm Aix5l San Storage Installation Guide
20/393
6 WHATS NEW WITH 11G RAC IMPLEMENTATION ON AIX
With Oracle 1 1g RAC on IBM POWER System p, i running AIX :
HACMP is not any more necessary as clusterware software since10gRAC Release 1.
Oracle provide with 11gRAC its own clusterware , named as Oracle
Clusterware, or CRS (Oracle Cluster Ready Service). Oracle Clusterware can cohabitate with HACMP under some
conditions.
With 11gRAC, Oracle Clusterware is mandatory as clusterware, evenso other vendors clusterware software are installed on the samesystems. You MUST check that third party clusterware can cohabitate
with Oracle clusterware.Subject: Using Oracle Clusterware with Vendor Clusterware FAQ
Doc ID:Note:332257.1
With Oracle Clusterware 11g Release 1 :
https://metalink.oracle.com/help/usaeng/Search/search.html#filehttps://metalink.oracle.com/help/usaeng/Search/search.html#filehttps://metalink.oracle.com/help/usaeng/Search/search.html#file -
8/14/2019 2512680 Cookbook 11grac r1 Asm Aix5l San Storage Installation Guide
21/393
Oracle Real Application Cluster intend to provide :
High Availability of users services to maintain continuity of the business
Scale-in within each node, adding resources when possible, as your business grow
Scale out when scale in is not anymore possible, responding to your business grow
Workload balancing and affinity
Etc
-
8/14/2019 2512680 Cookbook 11grac r1 Asm Aix5l San Storage Installation Guide
22/393
7.1 General Requirements
This chapter will list the general requirements to look at for an Oracle RAC implementation on IBM system p.
Topics covered are :
Servers and processors
Network
Storage
Storage attachements
VIO Server
7.1.1 About Servers and processors
-
8/14/2019 2512680 Cookbook 11grac r1 Asm Aix5l San Storage Installation Guide
23/393
7.1.2 About RAC on IBM System p
Oracle Real Application cluster could be implemented on LPARs/DLPARs from Separated physical IBM System pservers
For production ortesting to achieve :
High Availability
(protecting loss ofa physical server)
Scalability (addingcores and memory,or adding newserver)
Database
Workload affinityacross servers
OR implemented between LPARs, DLPARs from a same physical IBM System p server
-
8/14/2019 2512680 Cookbook 11grac r1 Asm Aix5l San Storage Installation Guide
24/393
7.1.3 About Network
-
8/14/2019 2512680 Cookbook 11grac r1 Asm Aix5l San Storage Installation Guide
25/393
For PRIVATE Network (Cluster/RAC Interconnect) !!!Gigabit switch is mandatory for production
implementation, even for only 2 nodes architecture.
(Cross-over cable can be used only for test purpose, and its not supported by Oracle Support,please read RAC FAQ on http://metalink.oracle.com).
A second gigabit ethernet interconnect, with a different network mask, can be setup for securitypurposes or performance issues.
Network cards for public network must have same name on each participating node in the RAC cluster(For example en0 on all nodes).
Network cards for Interconnect Network (Private) must have same Name on each participating Node in the
RAC cluster (For example en1 on all nodes).1 virtual IP per node must be reserved, and not used on the network prior to Oracle clusterwareinstallation. Dont set IP allias at AIX level, Oracle clusterware will take charge of it.
http://metalink.oracle.com/http://metalink.oracle.com/ -
8/14/2019 2512680 Cookbook 11grac r1 Asm Aix5l San Storage Installation Guide
26/393
7.1.4 About SAN Storage
When implementing RAC, you must be carrefull on the SAN storage to use. The SAN Storage must becapable thru its drivers of read/write concurrency at same time from any member of the RAC cluster, whichmeans that reserve_policy attribute from disks (hdisk, hdiskpower, dlmfdrv, etc ) discovered must be ableto be set to no_reserve or no_lock values.
-
8/14/2019 2512680 Cookbook 11grac r1 Asm Aix5l San Storage Installation Guide
27/393
-
8/14/2019 2512680 Cookbook 11grac r1 Asm Aix5l San Storage Installation Guide
28/393
7.1.5 Proposed infrastructure with 2 servers
Example of infrastructure as it should be with 2 servers, for network and storage.
-
8/14/2019 2512680 Cookbook 11grac r1 Asm Aix5l San Storage Installation Guide
29/393
7.1.6 What do we protect ?
Oracle RAC protect against node failures, the infrastructure should cover a maximum of failure case as :
For the storage
loss of 1 HBA on one node
loss of 1 SAN switch
For the Network
loss of 1 interconnect Network card on one node
loss of 1 Interconnect Network switch
-
8/14/2019 2512680 Cookbook 11grac r1 Asm Aix5l San Storage Installation Guide
30/393
7.1.7 About IBM Advanced Power Virtualization and RAC
Extract from Certify : RAC for Unix On IBM AIX based Systems (RAC only) (27/02/2008, check for last update)https://metalink.oracle.com/metalink/plsql/f?p=140:1:12020374767153922780:::::
Oracle products are certif ied for AIX5L on all servers that IBM supports with AIX5L. This includes IBM System iand System p models that use POWER5 and POWER6 processors. Please visit IBM's website at this URL formore information on AIX5L support for System i details.
IBM power based systems that support AIX5L includes machines branded as RS6000, pSeries, iSeries, Systemp and System i.
The minimum AIX levels for POWER 6 based models are AIX5L 5.2 TL10 and AIX5L 5.3 TL06
AIX5L certifications include AIX5L versions 5.2 and 5.3. 5.1 was desupported on 16-JUN-2006.
o Customers should review MetaLink Note 282036.1
64-bit hardware is required to run Real Application Clusters.
AIX 64-bit kernel is required for 10gR2 RAC. AIX 32-bit kernel and AIX 64-bit kernel are supported with 9.2 and10g.
Extract from RAC Technologies Matrix for UNIX Platforms (11/04/2008, check for last update)http://www.oracle.com/technology/products/database/clustering/certify/tech_generic_unix_new.html
PlatformTechnology
CategoryTechnology Notes
https://metalink.oracle.com/metalink/plsql/f?p=140:1:12020374767153922780:::::http://www-03.ibm.com/systems/i/os/aix/https://metalink.oracle.com/metalink/plsql/ml2_documents.showDocument?p_database_id=NOT&p_id=282036.1http://www.oracle.com/technology/products/database/clustering/certify/tech_generic_unix_new.htmlhttps://metalink.oracle.com/metalink/plsql/f?p=140:1:12020374767153922780:::::http://www-03.ibm.com/systems/i/os/aix/https://metalink.oracle.com/metalink/plsql/ml2_documents.showDocument?p_database_id=NOT&p_id=282036.1http://www.oracle.com/technology/products/database/clustering/certify/tech_generic_unix_new.html -
8/14/2019 2512680 Cookbook 11grac r1 Asm Aix5l San Storage Installation Guide
31/393
7.1.7.1 Network and VIO Server
Implementing Etherchannel links thru VIO servers for public and private network between nodes :
-
8/14/2019 2512680 Cookbook 11grac r1 Asm Aix5l San Storage Installation Guide
32/393
-
8/14/2019 2512680 Cookbook 11grac r1 Asm Aix5l San Storage Installation Guide
33/393
7.1.7.2 Storage and VIO Server
Following implementation is now supported !! !
-
8/14/2019 2512680 Cookbook 11grac r1 Asm Aix5l San Storage Installation Guide
34/393
Following implementation also supported :
-
8/14/2019 2512680 Cookbook 11grac r1 Asm Aix5l San Storage Installation Guide
35/393
7.2 Cookbook infrastructure
For our infrastructure, we used a cluster which is composed of three partitions (IBM LPAR) on an IBM systemp 570 using AIX 5L.
BUT in the real world, to achieve true high availability its necessary to have at least two IBM Systems p / i servers asshown bellow :
-
8/14/2019 2512680 Cookbook 11grac r1 Asm Aix5l San Storage Installation Guide
36/393
7.2.1 IBM Sytem p servers
This is the IBM SytempServer we used forour installation
http://www-03.ibm.com/servers/eserver/pseries/hardware/highend/
http://www-03.ibm.com/systems/p/
??????
http://www-03.ibm.com/servers/eserver/pseries/hardware/highend/http://www-03.ibm.com/systems/p/http://www-03.ibm.com/servers/eserver/pseries/hardware/highend/http://www-03.ibm.com/systems/p/ -
8/14/2019 2512680 Cookbook 11grac r1 Asm Aix5l San Storage Installation Guide
37/393
Commands to print the config for IBM System p on AIX5L (node1) :
-
8/14/2019 2512680 Cookbook 11grac r1 Asm Aix5l San Storage Installation Guide
38/393
{node1:root}/ # prtconf
System Model: IBM,9117-MMAMachine Serial Number: 651A260Processor Type: PowerPC_POWER6Number Of Processors: 1Processor Clock Speed: 3504 MHzCPU Type: 64-bitKernel Type: 64-bitLPAR Info: 1 JSC-11g-node1Memory Size: 3072 MBGood Memory Size: 3072 MBPlatform Firmware level: EM310_048
Firmware Version: IBM,EM310_048Console Login: enableAuto Restart: trueFull Core: false
Network InformationHost Name: node1IP Address: 10.3.25.81Sub Netmask: 255.255.255.0Gateway: 10.3.25.254
Name Server:Domain Name:
Paging Space InformationTotal Paging Space: 512MBPercent Used: 7%
Volume Groups Information==============================================================================rootvg:
-
8/14/2019 2512680 Cookbook 11grac r1 Asm Aix5l San Storage Installation Guide
39/393
Commands to print the config for IBM System p on AIX5L (node2) :
-
8/14/2019 2512680 Cookbook 11grac r1 Asm Aix5l San Storage Installation Guide
40/393
{node2:root}/ # prtconf
System Model: IBM,9117-MMAMachine Serial Number: 651A260Processor Type: PowerPC_POWER6Number Of Processors: 1Processor Clock Speed: 3504 MHzCPU Type: 64-bitKernel Type: 64-bitLPAR Info: 2 JSC-11g-node2Memory Size: 3072 MBGood Memory Size: 3072 MBPlatform Firmware level: EM310_048
Firmware Version: IBM,EM310_048Console Login: enableAuto Restart: trueFull Core: false
Network InformationHost Name: node2IP Address: 10.3.25.82Sub Netmask: 255.255.255.0Gateway: 10.3.25.254
Name Server:Domain Name:
Paging Space InformationTotal Paging Space: 512MBPercent Used: 2%
Volume Groups Information==============================================================================rootvg:
-
8/14/2019 2512680 Cookbook 11grac r1 Asm Aix5l San Storage Installation Guide
41/393
-
8/14/2019 2512680 Cookbook 11grac r1 Asm Aix5l San Storage Installation Guide
42/393
Command to get information on the LPAR :
{node1:root}/ # lparstat -iNode Name : node1Partition Name : JSC-11g-node1Partition Number : 1Type : Shared-SMTMode : UncappedEntitled Capacity : 0.50Partition Group-ID : 32769Shared Pool ID : 0
Online Virtual CPUs : 1Maximum Virtual CPUs : 1Minimum Virtual CPUs : 1Online Memory : 3072 MBMaximum Memory : 6144 MBMinimum Memory : 1024 MBVariable Capacity Weight : 128Minimum Capacity : 0.10Maximum Capacity : 1.00
Capacity Increment : 0.01Maximum Physical CPUs in system : 16Active Physical CPUs in system : 16Active CPUs in Pool : 16Shared Physical CPUs in system : -Maximum Capacity of Pool : -Entitled Capacity of Pool : -Unallocated Capacity : 0.00Physical CPU Percentage : 50.00%Unallocated Weight : 0
-
8/14/2019 2512680 Cookbook 11grac r1 Asm Aix5l San Storage Installation Guide
43/393
-
8/14/2019 2512680 Cookbook 11grac r1 Asm Aix5l San Storage Installation Guide
44/393
7.2.2 Operating System
Operating system must be installed the same way on each LPAR, with the same maintenance level, same APARand FILESETS level.
Check PREPARING THE SYSTEM chapter for Operating System requirements on AIX5L
-
8/14/2019 2512680 Cookbook 11grac r1 Asm Aix5l San Storage Installation Guide
45/393
7.2.3 Multi-pathing and ASM
Please check Metalink note Oracle ASM and Multi-Pathing Technologies Doc ID: Note:294869.1
Note, that Oracle Corporation does not certify ASM against multipathing utilities. The MP util ities listed below are onesthat known working solutions. As we do more testing, additional MP utilities will be listed here, thus, this document is anactive document.
Multi-pathing allow SAN access failover, and load balancing accros SAN Fiber Channel attachements.
OSPlatfor
m
Multi-pathingtool
ASM DeviceUsage
Notes
AIX EMC PowerPath Use raw partitionsthru the pseudodevice/dev/rhdiskpowerx
IBM SDD (Vpath)
NOTSUPPORTED forRAC/ASM onAIX !!!
/dev/vpathx As of this writing, SDD-AIX is known to cause discovery anddevice handling problems for ASM, and thus is not viablesolution.ASM needs to access disks/vpath thru non root user, which is notallowed by SDD as for today 27 March 2007.See SDDPCM section below for an alternative solution to SDDfor AIX
IBM SDDPCM Use /dev/rhdiskxdevice
You must install SDDPCM filesets and enable SDDPCM..SDDPCM cannot co-exist w/ SDD.
https://metalink.oracle.com/help/usaeng/Search/search.html#filehttps://metalink.oracle.com/help/usaeng/Search/search.html#file -
8/14/2019 2512680 Cookbook 11grac r1 Asm Aix5l San Storage Installation Guide
46/393
7.2.4 IBM storage and multi-pathing
With IBM, please refer to IBM to confirm which IBM storage is supported with RAC, if not specified in our document.
IBM TotalStorage products for IBM Sytem p
IBM DS4000, DS6000 and DS8000 series are supported with 10gRAC.
IBM Storage DS300 and DS400 are not, and will not be supported with 10gRAC.As for today March 27, 2007 IBM Storage DS3200 and DS3400 are not yetsupported with 10gRAC.
IBM System Storage and TotalStorage products
http://www-03.ibm.com/servers/storage/product/products_pseries.html
??????
There are 2 cases when using IBM storage :
IBM MPIO (Multi-Path I/O).MPIO driver is supported with IBM Total Storage ESS, DS6000 and DS8000 series onlyAnd with IBM SVC (SAN Volume Controler).
IBM RDAC (Redundant Disk Array Controller) for IBM Total Storage DS4000.RDAC driver is supported with IBM Total Storage DS4000 series only, and former FasTt.
http://www-03.ibm.com/servers/storage/product/products_pseries.htmlhttp://www-03.ibm.com/servers/storage/product/products_pseries.html -
8/14/2019 2512680 Cookbook 11grac r1 Asm Aix5l San Storage Installation Guide
47/393
7.2.4.1 IBM MPIO (Multi Path I/O) Setup Procedure
AIX Packages needed to installon all nodes :
devices.sddpcm.53.2.1.0.7.bffdevices.sddpcm.53.rtedevices.fcp.disk.ibm.mpio.rte
devices.fcp.disk.ibm.mpio.rte download page :http://www-
1.ibm.com/support/docview.wss?rs=540&context=ST52G7&dc=D410&q1=host+scripts&uid=ssg1S4000203&loc=en_US&cs=utf-8&lang=en
MPIO for AIX 5.3 download page :http://www-1.ibm.com/support/docview.wss?uid=ssg1S4000201
On node 1,and node 2
Installingthe filesets :
smitty install
Install and Update SoftwareInstall Software
* INPUT device / directory for software[/mydir_with_my_filesets]
SOFTWARE to install []Press F4
Select devices.fcp.disk.ibm.mpio
Install Software
http://www-1.ibm.com/support/docview.wss?rs=540&context=ST52G7&dc=D410&q1=host+scripts&uid=ssg1S4000203&loc=en_US&cs=utf-8&lang=enhttp://www-1.ibm.com/support/docview.wss?rs=540&context=ST52G7&dc=D410&q1=host+scripts&uid=ssg1S4000203&loc=en_US&cs=utf-8&lang=enhttp://www-1.ibm.com/support/docview.wss?rs=540&context=ST52G7&dc=D410&q1=host+scripts&uid=ssg1S4000203&loc=en_US&cs=utf-8&lang=enhttp://www-1.ibm.com/support/docview.wss?uid=ssg1S4000201http://www-1.ibm.com/support/docview.wss?rs=540&context=ST52G7&dc=D410&q1=host+scripts&uid=ssg1S4000203&loc=en_US&cs=utf-8&lang=enhttp://www-1.ibm.com/support/docview.wss?rs=540&context=ST52G7&dc=D410&q1=host+scripts&uid=ssg1S4000203&loc=en_US&cs=utf-8&lang=enhttp://www-1.ibm.com/support/docview.wss?rs=540&context=ST52G7&dc=D410&q1=host+scripts&uid=ssg1S4000203&loc=en_US&cs=utf-8&lang=enhttp://www-1.ibm.com/support/docview.wss?uid=ssg1S4000201 -
8/14/2019 2512680 Cookbook 11grac r1 Asm Aix5l San Storage Installation Guide
48/393
Check the insatllationsucced and the
installation summarymessage :
Installation Summary--------------------Name Level Part Event Result-------------------------------------------------------------------------------devices.fcp.disk.ibm.mpio.r 1.0.0.0 USR APPLY SUCCESS
Install devices.sddpcm.53
Select :
devices.sddpcm.53
ALL + 2.1.0.0 IBM SDD PCM for AIX V53 + 2.1.0.7 IBM SDD PCM for AIX V53
Install Software
Type or select values in entry fields.Press Enter AFTER making all desired changes.
* INPUT device / directory for software .* SOFTWARE to install [devices.sddpcm.53
> +PREVIEW only? (install operation will NOT occur) no
+COMMIT software updates? yes
+SAVE replaced files? no
+AUTOMATICALLY install requisite software? yes
+
-
8/14/2019 2512680 Cookbook 11grac r1 Asm Aix5l San Storage Installation Guide
49/393
Commandsto know AIX WWPN
{node1:root}/ #pcmpath query wwpn
Adapter Name PortWWNfscsi0 10000000C935A7E7fscsi1 10000000C93A1BF3
Commands
to check disks :
{node1:root}/ # lsdev -Cc disk -t 2107
hdisk2 Available 0A-08-02 IBM MPIO FC 2107hdisk3 Available 0A-08-02 IBM MPIO FC 2107hdisk4 Available 0A-08-02 IBM MPIO FC 2107hdisk5 Available 0A-08-02 IBM MPIO FC 2107hdisk6 Available 0A-08-02 IBM MPIO FC 2107hdisk7 Available 0A-08-02 IBM MPIO FC 2107hdisk8 Available 0A-08-02 IBM MPIO FC 2107hdisk9 Available 0A-08-02 IBM MPIO FC 2107hdisk10 Available 0A-08-02 IBM MPIO FC 2107hdisk11 Available 0A-08-02 IBM MPIO FC 2107
Commands
to check disks :
{node1:root}/ #pcmpath query device
DEV#: 2 DEVICE NAME: hdisk2 TYPE: 2107900 ALGORITHM: Load BalanceSERIAL: 75271812000==========================================================================Path# Adapter/Path Name State Mode Select Errors
0 fscsi0/path0 CLOSE NORMAL 0 01 fscsi1/path1 CLOSE NORMAL 0 0
DEV#: 3 DEVICE NAME: hdisk3 TYPE: 2107900 ALGORITHM: Load BalanceSERIAL: 75271812001
-
8/14/2019 2512680 Cookbook 11grac r1 Asm Aix5l San Storage Installation Guide
50/393
7.2.4.2 IBM AIX RDAC (FCP.ARRAY filesets) Setup Procedure
This ONLY apply to use of of DS4000 storage series. NOT to DS6000, DS8000 and ES800.
RDAC is installed by default on AIX5L.Each node must have 2 HBA cards, for multi-pathing. With ONLY 1 HBA per node, it will works but path to SAN will notbe protected. THEN in production, 2 HBA per node must be used.
All AIX hosts in your storage subsystem must have the RDAC multipath driver installed.In a single server environment, AIX allows load sharing (also called load balancing). You can set the load balancingparameter to yes. In case of heavy workload onone path the driver will move other LUNs to the controller with less
workload and, if theworkload reduces back to the preferred controller. Problem that can occur is disk thrashing.Thatmeans that the driver moves the LUN back and forth from one controller to the other. As a result the controller is moreoccupied by moving disks around than servicing I/O. Therecommendation is to NOT load balance on an AIX system.The performance increase isminimal (or performance could actually get worse).
RDAC (fcp.array filesets) for AIX support round-robin load-balancing
Setting the attributes of the RDAC driver for AIXThe AIX RDAC driver files are not included on the DS4000 installation CD.
Either install them from the AIX Operating Systems CD, if the correct version is included, or download them from thefollowing Web site: http://techsupport.services.ibm.com/server/fixesorhttp://www-304.ibm.com/jct01004c/systems/support/
Commands
to check thatnecessary filesets
{node1:root}/ # lslpp -L devices.fcp.disk.array.rteFileset Level State Type Description (Uninstaller)----------------------------------------------------------------------------devices.fcp.disk.array.rte
5.3.0.52 A F FC SCSI RAIDiant Array DeviceSupport Software
http://techsupport.services.ibm.com/server/fixeshttp://www-304.ibm.com/jct01004c/systems/support/http://techsupport.services.ibm.com/server/fixeshttp://www-304.ibm.com/jct01004c/systems/support/ -
8/14/2019 2512680 Cookbook 11grac r1 Asm Aix5l San Storage Installation Guide
51/393
Commands
to check the RDAC configuration and HBA path to hdisk
On node1
{node1:root}/ # fget_config -v -A
---dar0---
User array name = 'DS4000_JSC'dac0 ACTIVE dac3 ACTIVE
Disk DAC LUN Logical Drivehdisk1 dac0 0 G8_spfilehdisk2 dac3 1 G8_OCR1hdisk3 dac0 2 G8_OCR2hdisk4 dac3 3 G8_Vote1hdisk5 dac0 4 G8_Vote2
hdisk6 dac3 5 G8_Vote3hdisk7 dac0 6 G8_Data1hdisk8 dac3 7 G8_Data2hdisk9 dac0 8 G8_Data3hdisk10 dac3 9 G8_Data4hdisk11 dac0 10 G8_Data5hdisk12 dac3 11 G8_Data6hdisk13 dac0 12 G8_tie{node1:root}/ #
On node2
{node2:root}/ # fget_config -v -A
---dar0---
User array name = 'DS4000_JSC'
dac0 ACTIVE dac3 ACTIVE
Disk DAC LUN Logical Drivehdisk0 dac0 0 G8_spfilehdisk1 dac3 1 G8_OCR1hdisk2 dac0 2 G8_OCR2hdisk3 dac3 3 G8_Vote1hdisk4 dac0 4 G8_Vote2hdisk5 dac3 5 G8_Vote3
hdisk6 dac0 6 G8_Data1hdisk7 dac3 7 G8_Data2hdisk8 dac0 8 G8_Data3hdisk9 dac3 9 G8_Data4hdisk10 dac0 10 G8_Data5hdisk12 dac3 11 G8_Data6hdisk13 dac0 12 G8_tie{node2:root}/ #
-
8/14/2019 2512680 Cookbook 11grac r1 Asm Aix5l San Storage Installation Guide
52/393
Fcs0 is one of the HBA.
Commands
To see the HBA fiberchannel statistics :
{node1:root}/ # fcstat fcs0
FIBRE CHANNEL STATISTICS REPORT: fcs0
Device Type: FC Adapter (df1080f9)Serial Number: 1F41709923Option ROM Version: 02E01871Firmware Version: H1D1.81X1World Wide Node Name: 0x20000000C93F8E29World Wide Port Name: 0x10000000C93F8E29
FC-4 TYPES:Supported:0x0000012000000000000000000000000000000000000000000000000000000000
Active:0x0000010000000000000000000000000000000000000000000000000000000000Class of Service: 3Port Speed (supported): 2 GBITPort Speed (running): 2 GBITPort FC ID: 0x650B00Port Type: Fabric
Seconds Since Last Reset: 2795
Transmit Statistics Receive Statistics------------------- ------------------
Frames: 41615 96207Words: 1537024 12497408
LIP Count: 0NOS Count: 0Error Frames: 0
-
8/14/2019 2512680 Cookbook 11grac r1 Asm Aix5l San Storage Installation Guide
53/393
7.2.5 EMC storage and multi-pathing
With EMC, please refer to Hitachi to see which EMC storage is supported with RAC.
There are 2 cases when using EMC storage :
case 1: Luns provided by the EMC storage with IBM MPIO installed as multi-pathing driver.
Disks (LUNs) will be seen as hdisk at AIX level using lspv command.
On node 1 {node1:root}/ # lspvhdisk0 00ced22cf79098ff rootvg activehdisk1 none Nonehdisk2 none Nonehdisk3 none Nonehdisk4 none None
Then for disks to be used for ASM, and on all nodes :
1. Install MPIO on all nodes, attach the LUN to each node, dicover LUNs with cfgmgr.
2. Identify hdisk names on each nodes, for a given LUN ID.
3. remove PVID from hdisk and change the reserve policy to no reserve using :
chdev -l hdisk -a pv=clear
chdev -l hdisk -a reserve_policy=no_reserve
4. set ownerchip to oracle:dba to the /dev/rhdisk
5. set read/write permissions to 660 to the /dev/rhdisk
-
8/14/2019 2512680 Cookbook 11grac r1 Asm Aix5l San Storage Installation Guide
54/393
7.2.5.1 EMC PowerPath Setup Procedure
See PowerPath for AIX version 4.3 Installation & Administration Guide, P/N 300-001-683 for details
On node 1, and node 2
1. Install EMC ODM drivers and necessary filesets 5.2.0.1 from ftp://ftp.emc.com/pub/elab/aix/ODM_DEFINITIONS/EMC.AIX.5.2.0.1.tar.Z
install using smit install
2. remove any existing devices attached to the EMC{node1:root}/ # rmdev dl hdiskX
3. run /usr/lpp/EMC/Symmetrix/bin/emc_cfgmgr to detect devices
4. Install PowerPath version 4.3.0 minimum using smit install
5. register PowerPath{node1:root}/ # emcpreg install
6. initialize PowerPath devices{node1:root}/ # powermt config
7. verify that all PowerPath devices are named consistently across all cluster nodes{node1:root}/ # /usr/lpp/EMC/Symmetrix/bin/inq.aix64 | grep hdiskpower
compare results. Consistent naming is not required for ASM devices, but LUNs usedfor the OCR and VOTE functions must have the same device names on all rac systems.
-
8/14/2019 2512680 Cookbook 11grac r1 Asm Aix5l San Storage Installation Guide
55/393
7.2.6 HITACHI storage and multi-pathing
With Hitachi, please refer to Hitachi to see which HDS storage is supported with RAC.There are 3 cases when using Hitachi HDS storage :
case 1: Luns provided by the HDS storage with IBM MPIO installed as multi-pathing driver.
*** NOT SUPPORTED with all HDS storage, check with Hitachi ***
Disks (LUNs) will be seen as hdisk at AIX level using lspv command.
On node 1 {node1:root}/ # lspvhdisk0 00ced22cf79098ff rootvg activehdisk1 none Nonehdisk2 none Nonehdisk3 none Nonehdisk4 none None
Then for disks to be used for ASM, and on all nodes :
1. Install MPIO on all nodes, attach the LUN to each node, dicover LUNs with cfgmgr.
2. Identify hdisk names on each nodes, for a given LUN ID.
3. remove PVID from hdisk and change the reserve policy to no reserve using :
chdev -l hdisk -a pv=clear
-
8/14/2019 2512680 Cookbook 11grac r1 Asm Aix5l San Storage Installation Guide
56/393
Case 3 : Luns provided by the HDS storage with HDLM as multi-pathing driver.
Disks will be seen as dlmfdrv at AIX level using lspv command, and part of a HDLM VG(volume groups).
On node 1 {node1:root}/ # lspvdlmfdrv0 00ced22cf79098ff rootvg activedlmfdrv1 none vg_asm activedlmfdrv2 none vg_asm activedlmfdrv3 none vg_asm activedlmfdrv4 none vg_asm activedlmfdrv5 none vg_asm active
or
{node1:root}/ # lspvdlmfdrv0 00ced22cf79098ff rootvg activedlmfdrv1 none vg_ocr_disk1 activedlmfdrv2 none vg_voting_disk1 activedlmfdrv3 none vg_asm_disk1 activedlmfdrv4 none vg_asm_disk2 activedlmfdrv5 none vg_asm_disk3 active
Then for disks to be used for ASM, and on all nodes :
1. Install HDLM on all nodes, attach the LUN to each node, dicover LUNs with cfgmgr.
2. Turn off reserve locking on all nodes dlnkmgr set rsv on 0 s
3. Create VGs (Volume Groups) and LVs (Logical Volumes)
2 options :
-
8/14/2019 2512680 Cookbook 11grac r1 Asm Aix5l San Storage Installation Guide
57/393
dlmvaryoffvg vg_asm
On node 2
6) Identify which dlmfdrv correspond to dlmfdrv1 fromnode1.
dlmfdrv1 on node1 dlmfdrv0 on node2
7) import the volume groups on node2
This will copy the vg/lv configuration that was made onnode1
dlmimportvg -V 101 -y vg_asm dlmfdrv0
7) enable the volume groups on node2
dlmvaryonvg vg_asm
8) this will ensure the vg will not get varyon'd atboot
dlmchvg -a n vg_asm
On node 1
9) enable the volume groups on node1
dlmvaryonvg vg_asm
Check for document :
Hitachi Dynamic Link Manager (HDLM) for IBM AIXSystems Users Guide:
chmod 660 /dev/rlv_voting_disk1chmod 660 /dev/rlv_asm_disk1chmod 660 /dev/rlv_asm_disk2chmod 660 /dev/rlv_asm_disk3
5) disable the volume groups
dlmvaryoffvg vg_ocr1dlmvaryoffvg vg_vot1dlmvaryoffvg vg_asm1dlmvaryoffvg vg_asm2dlmvaryoffvg vg_asm3
On node 2
6) Identify which dlmfdrv correspond to dlmfdrv1 fromnode1, and so on with other dlmfdrv
dlmfdrv1 on node1 dlmfdrv0 on node2dlmfdrv2 on node1 dlmfdrv1 on node2dlmfdrv3 on node1 dlmfdrv2 on node2dlmfdrv4 on node1 dlmfdrv3 on node2dlmfdrv5 on node1 dlmfdrv4 on node2
7) import the volume groups on node2
This will copy the vg/lv configuration that was made onnode1
dlmimportvg -V 101 -y vg_ocr1 dlmfdrv0dlmimportvg -V 101 -y vg_vot1 dlmfdrv1dlmimportvg -V 101 -y vg_asm1 dlmfdrv2dlmimportvg -V 101 -y vg_asm2 dlmfdrv3dlmimportvg -V 101 -y vg_asm3 dlmfdrv4
http://sys-admin.net/uploads/SAN/hldm_admin_guide.pdfhttp://sys-admin.net/uploads/SAN/hldm_admin_guide.pdfhttp://sys-admin.net/uploads/SAN/hldm_admin_guide.pdfhttp://sys-admin.net/uploads/SAN/hldm_admin_guide.pdfhttp://sys-admin.net/uploads/SAN/hldm_admin_guide.pdf -
8/14/2019 2512680 Cookbook 11grac r1 Asm Aix5l San Storage Installation Guide
58/393
-
8/14/2019 2512680 Cookbook 11grac r1 Asm Aix5l San Storage Installation Guide
59/393
8 SPECIFIC CONSIDERATIONS FOR RAC/ASM SETUP WITH HACMPINSTALLED
Oracle Clusterware does not requires HACMP to work, but some customers may need to have HACMPinstalled on the RAC node cluster to protect third party products, or ressources. Oracle clusterware couldreplace HACMP for most operations as cold failover for 11g, 10g and 9i single database, applications servers or anyapplications.
Please check following documents for details onhttp://www.oracle.com/technology/products/database/clustering/index.html :
Comparing Oracle Real Applicaton Clusters to Failover Clusters for Oracle Database(PDF) December 2006 Workload Management with Oracle Real Application Clusters (FAN, FCF, Load Balancing) (PDF) May 2005
Using Oracle Clusterware to Protect 3rd Party Applications (PDF) May 2005
Using Oracle Clusterware to Protect a Single Instance Oracle Database (PDF) November 2006
Using Oracle Clusterware to protect Oracle Application Server (PDF) November 2005
How to Build End to End Recovery and Workload Balancing for Your Applications 10g Release 1(PDF) Dec2004
Oracle Database 10gServices (PDF) Nov 2003
Oracle Database 10gRelease 2 Best Practices: Optimizing Availability During Unplanned Outages UsingOracle Clusterware and RACNew!
HOWEVER, if customer still need to have HACMP, Oracle Clusterware can cohabitate with HACMP.
Please check for last status of certification on Oracle Metalink (certify)
Extract from metalink in date of March 3rd, 2008.
http://www.oracle.com/technology/products/database/clustering/index.htmlhttp://www.oracle.com/technology/products/database/clustering/pdf/failover_cluster.pdfhttp://www.oracle.com/technology/products/database/clustering/pdf/twpracwkldmgmt.pdfhttp://www.oracle.com/technology/products/database/clustering/pdf/twp_oracleclusterware3rdparty%5B1%5D.pdfhttp://www.oracle.com/technology/products/database/clustering/pdf/ocsingleinstance.pdfhttp://www.oracle.com/technology/products/database/clustering/pdf/Using_Oracle_Clusterware_to_protect_Oracle_Application_Server.pdfhttp://www.oracle.com/technology/products/database/clustering/pdf/oowfan.pdfhttp://www.oracle.com/technology/products/database/clustering/pdf/twp_rac_services_10gr1_112503.pdfhttp://www.oracle.com/technology/products/database/clustering/pdf/twp_rac_services_10gr1_112503.pdfhttp://www.oracle.com/technology/products/database/clustering/pdf/twp_rac_services_10gr1_112503.pdfhttp://www.oracle.com/technology/deploy/availability/pdf/MAA_WP_10gR2_FastRecoveryOracleClusterwareandRAC.pdfhttp://www.oracle.com/technology/deploy/availability/pdf/MAA_WP_10gR2_FastRecoveryOracleClusterwareandRAC.pdfhttp://www.oracle.com/technology/deploy/availability/pdf/MAA_WP_10gR2_FastRecoveryOracleClusterwareandRAC.pdfhttp://www.oracle.com/technology/deploy/availability/pdf/MAA_WP_10gR2_FastRecoveryOracleClusterwareandRAC.pdfhttp://www.oracle.com/technology/products/database/clustering/index.htmlhttp://www.oracle.com/technology/products/database/clustering/pdf/failover_cluster.pdfhttp://www.oracle.com/technology/products/database/clustering/pdf/twpracwkldmgmt.pdfhttp://www.oracle.com/technology/products/database/clustering/pdf/twp_oracleclusterware3rdparty%5B1%5D.pdfhttp://www.oracle.com/technology/products/database/clustering/pdf/ocsingleinstance.pdfhttp://www.oracle.com/technology/products/database/clustering/pdf/Using_Oracle_Clusterware_to_protect_Oracle_Application_Server.pdfhttp://www.oracle.com/technology/products/database/clustering/pdf/oowfan.pdfhttp://www.oracle.com/technology/products/database/clustering/pdf/twp_rac_services_10gr1_112503.pdfhttp://www.oracle.com/technology/deploy/availability/pdf/MAA_WP_10gR2_FastRecoveryOracleClusterwareandRAC.pdfhttp://www.oracle.com/technology/deploy/availability/pdf/MAA_WP_10gR2_FastRecoveryOracleClusterwareandRAC.pdf -
8/14/2019 2512680 Cookbook 11grac r1 Asm Aix5l San Storage Installation Guide
60/393
Detailed information for HACMP 5.3 and 11gRAC :
Certify - Additional Info RAC for Unix Version 11gR1 64-bit On IBM AIX based Systems (RAC only)
Operating System: IBM AIX based Systems (RAC only) Version 5.3RAC for Unix Version 11gR1 64-bitHACMP Version 5.3Status: Certified
Product Version Note:None available for this product.
Certification Note:
Use of HACMP 5.3 requires minimum service levels of the following:
o AIX5L 5.3 TL 6 or later, specifically bos.rte.lvm must be at least 5.3.0.60
o HACMP V5.3 with PTF5, cluster.es.clvm installed and ifix for APAR IZ01809
o RSCT (rsct.basic.rte) version 2.4.7.3 and ifix for APAR IZ01838
Detailed information for HACMP 5.4.1 and 11gRAC :
Certify - Additional Info RAC for Unix Version 11gR1 64-bit On IBM AIX based Systems (RAC only)
Operating System: IBM AIX based Systems (RAC only) Version 5.3RAC for Unix Version 11gR1 64-bitHACMP Version 5.4.1Status: Certified
Product Version Note:
-
8/14/2019 2512680 Cookbook 11grac r1 Asm Aix5l San Storage Installation Guide
61/393
Following rules have to be applied :
1. HACMP must not take-over/failover the Oracle Clusterware ressources (VIP, database, etc )
2. HACMP VIP must not be configured on IP from Public node name used by RAC (hostname), or OracleClusterware VIP
3. With 11g, its not necessary to declare the RAC interconnect in HACMP
4. Its not mandatory to declare hdisks used for ASM in HACMP as logical volumes (LV) from Volume Groups(VG). In this case follow the cookbook to prepare the disks for OCR, Voting and ASM disks.
5. If the choice is to declare hdisks used by ASM in HACMP Volume Groups, THEN youll have to prepare thedisks for OCR, Voting, ASM spfile and ASM disks as describe in official Oracle document available onhttp://tahiti.oracle.com
Please check :
Oracle Real Application Clusters Installation Guide11g Release 1 (11.1) for Linux and UNIXPart Number B28264-03http://download.oracle.com/docs/cd/B28359_01/install.111/b28264/toc.htm
And Metalink note 404474.1
Status of Certification of Oracle Clusterware with HACMP 5.3 & 5.4STATUS of IBM HACMP 5.3, 5.4 Certifications with Oracle RAC 10g5.4 Certifications with Oracle RAC 10g What do you need to do? Configuring HACMP 5.3 orHACMP 5.4.1,and 10gR2 with Multi-Node Disk Heartbeat (MNDHB)
Even so this note is written for 10gRAC R2, its also applicable to 11g RAC R1.
http://tahiti.oracle.com/http://download.oracle.com/docs/cd/B28359_01/install.111/b28264/toc.htmhttps://metalink.oracle.com/metalink/plsql/ml2_documents.showDocument?p_database_id=NOT&p_id=404474.1http://tahiti.oracle.com/http://download.oracle.com/docs/cd/B28359_01/install.111/b28264/toc.htmhttps://metalink.oracle.com/metalink/plsql/ml2_documents.showDocument?p_database_id=NOT&p_id=404474.1 -
8/14/2019 2512680 Cookbook 11grac r1 Asm Aix5l San Storage Installation Guide
62/393
9 INSTALLATION STEPS
Prioir to install and use Oracle 11g Real Application Cluster, you must :
1. Prepare the infrastructurea. Hardwareb. Storagec. Networkd. San and Network connectivitye. Operating system
2. Prepare the systems
a. Defining Network Layout, Public, Virtual andPrivate hostnames
b. AIX Operating system level, required APARsand filsets
c. System Requirements (Swap, temp, memory,internal disks)
d. Users And Groupse. Kernel and Shell Limitsf. User equivalencesg. Time Server Synchronization
-
8/14/2019 2512680 Cookbook 11grac r1 Asm Aix5l San Storage Installation Guide
63/393
When all that is done !!!, you can process the installation in the following order :
1. Install Oracle 11g Clusterwarea. Apply necessary patchset and patchesb. Check that Oracle clusterware is working
properly
2. Install Oracle 11g Automated Storage Managementa. Apply necessary patchset and patches
b. Create default nodes listenersc. Create ASM instances on each noded. Configure ASM instances Local and remote
listeners if requirede. Change necessary ASM instances parameters
(process, etc )f. Create ASM diskgroup(s)
3. Inst all Oracle 11g Real Application Cluster or/and
Oracle 10g Real Application clustera. Apply necessary patchset and patches
-
8/14/2019 2512680 Cookbook 11grac r1 Asm Aix5l San Storage Installation Guide
64/393
10 PREPARING THE SYSTEM
Preparing the system is to be done on all servers which are planned to participate in the Oracle Cluster.All servers MUST be as clones, with ONLY different HOSTNAME and IP addresses !!!
IMPORTANT !!!
For ALL servers, you MUST apply the Oracle pre-requisites, those prerequisites are notoptional, BUT MANDATORY !!!
For some parameters as tuning settings, values specified are the minimum required, andmigh be increase depending of your needs !!!
PLEASE check Oracle documentation for last update, and MOSTLY :Oracle METALINK Note
Preparing the system is about :
Defining Network Layout, Public, Virtual and Private hostnameso Hostname and RAC Public Node nameo Network interface identificationo Default Gateway
-
8/14/2019 2512680 Cookbook 11grac r1 Asm Aix5l San Storage Installation Guide
65/393
10.1 Network configuration
10.1.1 Define Networks layout, Public, Virtual and Private Hostnames
We need 2 differents networks, with associated network interfaces on each node participating to the RAC cluster :
Public Network to be used as Users network or reserved network for Application and Databases servers.
Private Network to be used as a Reserved network for Oracle clusterware, and RAC.
MANDATORY : Network interfaces must have same name, same subnet and same usage !!!
For each node, We have to define :
-
8/14/2019 2512680 Cookbook 11grac r1 Asm Aix5l San Storage Installation Guide
66/393
Let identify hostnames, and network interfaces
Please make a table as follow to have a clear view of your RAC network architecture :PUBLIC NODE NAME MUST BE NAME RETURNED BY hostname AIX COMMAND
Public Hostname VIP Hostname
(Virtual IP)
Private Hostname
(RAC Interconnect)
Not Used
en ? (Public Network) en ? (Private Network) en ?
NodeName
IP NodeName
IP NodeName
IP Node Name IP
-
8/14/2019 2512680 Cookbook 11grac r1 Asm Aix5l San Storage Installation Guide
67/393
For second server ...
{node2:root}/ # hostnamenode2
{node2:root}/ #ping node2PING node2: (10.3.25.82): 56 data bytes64 bytes from 10.3.25.82: icmp_seq=0 ttl=255 time=0 ms64 bytes from 10.3.25.82: icmp_seq=1 ttl=255 time=0 ms64 bytes from 10.3.25.82: icmp_seq=2 ttl=255 time=0 ms^C
----node2 PING Statistics----3 packets transmitted, 3 packets received, 0% packet lossround-trip min/avg/max = 0/0/0 ms{node2:root}/ #
Other method to check the default hostname :
{node1:root}/crs/11.1.0/bin # lsattr -El inet0authm 65536 Authentication Methods Truebootup_option no Use BSD-style Network Configuration Truegateway Gateway Truehostname node1 Host Name Truerout6 IPv6 Route Trueroute net,-hopcount,0,,0,10.3.25.254 Route True{node1:root}/crs/11.1.0/bin #
-
8/14/2019 2512680 Cookbook 11grac r1 Asm Aix5l San Storage Installation Guide
68/393
Oracle clusterware VIPs IP adress and corresponding nodes names must not be used on the network prior toOracle Clusterware installation. Dont make any AIX allias on the public network interface, the clusterware installationwill do it. Just reserve 1 VIP and its hostname per RAC node.
Public Hostname VIP Hostname
(Virtual IP)
Private Hostname
(RAC Interconnect)
Not Used
en ? (Public Network) en ? (Private Network) en ?
NodeName
IP NodeName
IP NodeName
IP Node Name IP
node1 10.3.25.81 node1-vip 10.3.25.181
node2 10. 3.25.82 node2-vip 10. 3.25.182
Oracle Clusterware VIPs IP and corresponding nodes names can be declared in the DNS, or at minimum in the localhosts file.
10.1.2 Identify Network Interfaces cards
-
8/14/2019 2512680 Cookbook 11grac r1 Asm Aix5l San Storage Installation Guide
69/393
As root, Issue the following shell to get necessary information from network interfaces on each node :
Result from node1 :
{node1:root}/ # for i in en0 en1 en2do
echo $ifor attribut in netaddr netmask broadcast
statedo
lsattr -El $i -a $attributdone
doneen0netaddr 10.3.25.81 Internet Address Truenetmask 255.255.255.0 Subnet Mask Truebroadcast Broadcast Address Truestate up Current Interface Status Trueen1netaddr 10.10.25.81 Internet Address True
netmask 255.255.255.0 Subnet Mask Truebroadcast Broadcast Address Truestate up Current Interface Status Trueen2netaddr 20.20.25.81 Internet Address Truenetmask 255.255.255.0 Subnet Mask Truebroadcast Broadcast Address Truestate up Current Interface Status True{node1:root}/ #
Result from node2 :
{node2:root}/ # for i in en0 en1 en2do
echo $ifor attribut in netaddr netmask broadcast
statedo
lsattr -El $i -a $attributdone
doneen0netaddr 10.3.25.82 Internet Address Truenetmask 255.255.255.0 Subnet Mask Truebroadcast Broadcast Address Truestate up Current Interface Status Trueen1netaddr 10.10.25.82 Internet Address True
netmask 255.255.255.0 Subnet Mask Truebroadcast Broadcast Address Truestate up Current Interface Status Trueen2netaddr 20.20.25.82 Internet Address Truenetmask 255.255.255.0 Subnet Mask Truebroadcast Broadcast Address Truestate up Current Interface Status True{node2:root}/ #
-
8/14/2019 2512680 Cookbook 11grac r1 Asm Aix5l San Storage Installation Guide
70/393
Now, we have identified the following network interfaces :
en0 is set to be the Public Network Interface on all nodes.
en0 is set to be the VIP Network Interface on all nodes.
en1 is set to be the Private Network Interface, also named as RAC Interconnect on all nodes.
en2 is set as not used
THEN, the table looks as follow :
Public Hostname VIP Hostname
(Virtual IP)
Private Hostname
(RAC Interconnect)
Not Used
en0 (Public Network) en1 (Private Network) en2
-
8/14/2019 2512680 Cookbook 11grac r1 Asm Aix5l San Storage Installation Guide
71/393
THEN, we will get the following table with our system :
Public VIP RAC Interconnect (PrivateNetwork)
Not Used
en0 en1 en2
NodeName
IP NodeName
IP NodeName
IP NodeName
IP
node1 10.3.25.81 node1-vip 10.3.25.181 node1-rac 10.10.25.81
node2 10. 3.25.82 node2-vip 10. 3.25.182 node1-rac 10.10.25.82
Within our infrastructure for the cookbook, we have the following layout :
-
8/14/2019 2512680 Cookbook 11grac r1 Asm Aix5l San Storage Installation Guide
72/393
10.1.4 Defining Default gateway on public network interface
-
8/14/2019 2512680 Cookbook 11grac r1 Asm Aix5l San Storage Installation Guide
73/393
may have if you have already a default gateway set on a different network interface, and multiple networkinterfaces not used for Oracle clusterware and RAC purpose.
First check if default gateway is set :Use netstat -r On both nodes.
{node1:root}/ # netstat -rRouting tablesDestination Gateway Flags Refs Use If Exp Groups
Route Tree for Protocol Family 2 (Internet):
default 10.3.25.254 UG 1 73348 en0 - - =>default 9.212.131.254 UG 0 435 en0 - -10.3.25.0 node1 UHSb 0 0 en0 - - =>10.3.25/24 node1 U 11 6113590 en0 - -node1 loopback UGHS 37 1034401 lo0 - -node1-vip loopback UGHS 8 80831 lo0 - -10.3.25.255 node1 UHSb 0 4 en0 - -10.10.25.0 node1-rac UHSb 0 0 en1 - - =>10.10.25/24 node1-rac U 25 350557 en1 - -
node1-rac loopback UGHS 16 481 lo0 - -10.10.25.255 node1-rac UHSb 0 4 en1 - -20.20.25.0 node1-rac-b UHSb 0 0 en2 - - =>20.20.25/24 node1-rac-b U 16 176379 en2 - -node1-rac-b loopback UGHS 5 392 lo0 - -20.20.25.255 node1-rac-b UHSb 0 4 en2 - -127/8 loopback U 49 187105 lo0 - -
Route Tree for Protocol Family 24 (Internet v6):
-
8/14/2019 2512680 Cookbook 11grac r1 Asm Aix5l San Storage Installation Guide
74/393
10.1.5 Configure Network Tuning Parameters
Verify that the network tuning parameters shown in the following table are set to the values shown or higher values.The procedure following the table describes how to verify and set the values.
Network TuningParameter Recommended minimum Value on all nodes
ipqmaxlen 512
rfc1323 1
sb_max 1310720
tcp_recvspace 65536
tcp_sendspace 65536
udp_recvspace 655360
Note: The recommended value of this parameter is 10 times the value of the udp_sendspaceparameter. The value must be less than the value of the sb_max parameter.
udp_sendspace 65536
Note: This value is suitable for a default database installation. For production databases, theminimum value for this parameter is 4 KB plus the value of the database DB_BLOCK_SIZEinitialization parameter mult iplied by the value of the DB_MULTIBLOCK_READ_COUNTinitialization parameter:
(DB_BLOCK_SIZE * DB_MULTIBLOCK_READ_COUNT) + 4 KB
To check values
-
8/14/2019 2512680 Cookbook 11grac r1 Asm Aix5l San Storage Installation Guide
75/393
To change thecurrent values torequired ones, ifnecessary,
follow these steps :
1. If you must change the value of any parameter, enter the following command todetermine whether the system is running in compatibility mode:
# /usr/sbin/lsattr -E -l sys0 -a pre520tune
If the system is running in compatibility mode, the output is similar to thefollowing, showing that the value of the pre520tune attribute is enable:
pre520tune enable Pre-520 tuning compatibility mode True
By default, with AIX5L, compatibility mode is set to false !!!Change it to true ONLY if necessary !!!
** if you want to enable the compatibility mode, issue the following command :# chdev -l sys0 -a pre520tune=enable
2. If the system is running in compatibility mode,
THEN
follow these steps to change the parametervalues:
Enter commands similar to the following tochange the value of each parameter:
# /usr/sbin/no oparameter_name=value
For example:
ELSE
# /usr/sbin/lsattr -E -l sys0 -a pre520tune
Will give the following :
pre520tune disable Pre-520 tuningcompatibility mode True
The system is not running in compatibility mode, entercommands similar to the following to change the
-
8/14/2019 2512680 Cookbook 11grac r1 Asm Aix5l San Storage Installation Guide
76/393
10.2 AIX Operating system level, required APARs and filsets
-
8/14/2019 2512680 Cookbook 11grac r1 Asm Aix5l San Storage Installation Guide
77/393
10.2.1 Filesets Requirements for 11g RAC R1 / ASM (NO HACMP)
AIX filesets required on ALL nodes for 11gRAC Release 1 implementation withASM !!!
Check that therequired filsets areinstalled on thesystem.
(Note: If the PTF isnot downloadable,customers shouldrequest an efixthrough AIXcustomer support.)
AIX 5L version 5.3,Maintenance Level 6 or later
AIX6 version ...Maintenance Level ... or later
Filesets
bos.adt.base bos.adt.lib
bos.adt.libm
bos.perf.libperfstat
bos.perf.perfstat
bos.perf.proctools
rsct.basic.rte
rsct.compat.clients.rte
xlC.aix50.rte 8.0.0.5
xlC.rte 8.0.0.5
Specific Filesets
For EMC Symmetrix : EMC.Symmetrix.aix.rte.5.2.0.1 or higher
For EMC CLARiiON : EMC.CLAR11ON.fcp.rte.5.2.0.1 or
higher
NOT YETSUPPORTEDCheck Metalink and certify for last update oncertification status.
-
8/14/2019 2512680 Cookbook 11grac r1 Asm Aix5l San Storage Installation Guide
78/393
10.2.2 APARs Requirements for 11g RAC R1 / ASM (NO HACMP)
Check that the required APARs are installed on the system.
AIX Patches (APAR) required on ALL nodes for 11gRAC R1 implementation withASM !!!
(Note: If the PTF is not downloadable, customers should request an efix through AIX customer support.)
AIX 5L version 5.3,Maintenance Level 6 or later
AIX6 version ...Maintenance Level ... or later
APARs
IY89080
IY92037
IY94343
IZ01060 or efix for IZ01060
IZ03260 or efix for IZ03260
JDK : (Not mandatory for the installation), thes APAR arerequired only if you are using the associated JDK version
IY58350 Patch for SDK 1.3.1.16 (32-bit)
IY63533 Patch for SDK 1.4.2.1 (64-bit)
IY65305 Patch for SDK 1.4.2.2 (32-bit)
IBM JDK 1.5.0.06 (IA64 - mixed mode) is installed
NOT YETSUPPORTEDCheck Metalink and certify for last update on certificationstatus.
-
8/14/2019 2512680 Cookbook 11grac r1 Asm Aix5l San Storage Installation Guide
79/393
10.3 System Requirements (Swap, temp, memory, internal disks)
Requirements to meet on ALL nodes !!!
RAM >= 512 MB minimum. Command to check the physical memory : lsattr El sys0 a realmem
{node1:root}/ # lsattr -El sys0 -a realmemrealmem 3145728 Amount of usable physical memory in Kbytes False{node1:root}/ #
{node2:root}/home # lsattr -El sys0 -a realmemrealmem 3145728 Amount of usable physical memory in Kbytes False{node2:root}/home #
Internal disk >= 12 GB for the oracle code (CRS_HOME, ASM_HOME, ORACLE_HOME)
This part will be detailed in chapter Required local disks (Oracle Clusterware, ASM and RAC software) !!!
{node1:root}/ # df -kFilesystem 1024-blocks Free %Used Iused %Iused Mounted on/dev/hd4 262144 207816 21% 13591 23% //dev/hd2 4718592 2697520 43% 46201 7% /usr/dev/hd9var 262144 233768 11% 565 2% /var/dev/hd3 1310720 1248576 5% 255 1% /tmp/dev/hd1 262144 261760 1% 5 1% /home
-
8/14/2019 2512680 Cookbook 11grac r1 Asm Aix5l San Storage Installation Guide
80/393
Paging space = 2 x RAM, with a minimum of 400 MB and a maximum of 2 GB. To check the paging space configured : lsps a
{node1:root}/ # lsps -aPage Space Physical Volume Volume Group Size %Used Active Auto Typehd6 hdisk0 rootvg 512MB 7 yes yes lv{node1:root}/ #
{node2:root}/home # lsps -aPage Space Physical Volume Volume Group Size %Used Active Auto Typehd6 hdisk1 rootvg 512MB 2 yes yes lv{node2:root}/home #
Temporary Disk Space : The Oracle Universal Installer requires up to 400 MB of free space in the /tmp directory. To check the free temporary space available: df k /tmp
{node1:root}/ # df -k /tmp
Filesystem 1024-blocks Free %Used Iused %Iused Mounted on/dev/hd3 1310720 1248576 5% 255 1% /tmp{node1:root}/ #
{node2:root}/home # df -k /tmpFilesystem 1024-blocks Free %Used Iused %Iused Mounted on/dev/hd3 2097152 29636 99% 4812 35% /tmp{node2:root}/home #
-
8/14/2019 2512680 Cookbook 11grac r1 Asm Aix5l San Storage Installation Guide
81/393
10.4 Users And Groups
2 options possibles
OPTION 1 OPTION 2
1 user for each installation, for example :
unix crs user for CRS_HOME installation
unix asm user for ASM_HOME installation
unix rdbms user for ORACLE_HOMEinstallation
1 user for all installations,for example oracle unix user for :
CRS_HOME installation
ASM_HOME installation
ORACLE_HOME installation
-
8/14/2019 2512680 Cookbook 11grac r1 Asm Aix5l San Storage Installation Guide
82/393
This setup has to be done on all the nodes of the cluster.
Default user home for crs, asm and rdbms users MUST be in /home, otherwise you may have troublewith ssh setup (user equivalence).
-
8/14/2019 2512680 Cookbook 11grac r1 Asm Aix5l San Storage Installation Guide
83/393
The crs, asm and rdbms users must have oinstallas primary group, dba as
top related