DBA,Clustering

IINSTALLINGNSTALLINGRACFRACFROMROMTTHEHEGGROUNDROUNDUUPP

DanNorris,dannorris(at)dannorris.com,DanNorris.com

INTRODUCTIONANDTERMINOLOGY ThispaperoffersarecipeforcreaUngaRACenvironment.TheinstallaUonguidescontainmostoftheinformaUononinstallaUon,andafewinternetresources(citedinthereferencessecUon)providethenecessaryextrastepsyouneedtocompletethesetup.

Tosupportthediscussionthatfollows,thesedefiniUonsmaybehelpful:

Term DefiniKon

Database thecontrolfiles,datafiles,andonlineredologs

Instance thesharedmemoryandbackgroundprocessesthatoperateonadatabase

ClusterwaresoEwarethatmanagesclustermembershipandmonitorsthenodesandnetworksinacluster

StorageAreaNetwork(SAN)

astorageenvironmentwheremulUpleserverscanuUlizeasinglestoragearray;astoragenetworkiscommonlyimplementedusingfiberchanneltechnology

LocalStoragediskspacethatisavailabletoexactlyonenodeinacluster;thisstoragemaybepartofaSANormaybedirect‐aVachedtotheserver

SharedStoragediskspacethatisavailabletomorethanonenodeinaclusteratthesameUme;thisstorageiscommonlypartofaSAN

RawDevice thecharacter(unbuffered)specialdevicepresentedbytheoperaUngsystem

ClusterFilesystemaspecialtypeoffilesystemthatcanbeaccessedbymulUpleclusternodesatthesameUme

OracleAutomaUcStorageManagement(ASM)

soEwarethatwillmanagedisksdirectlyandprovidevolumemanagementfuncUonslikestripingandmirroring;thisfeaturewasnewwithOracleDatabase10g

Single‐instanceDatabase

thetradiUonal"standalone"databaseconfiguraUonusedbymostOracleenvironmentstoday

MulU‐instanceDatabase

asingledatabaseservicedbymorethanoneinstance

InthesecUonsthatfollow,youwilllearn:

AnarchitecturaloverviewofRAC

SystemPrerequisitesandcheckingthemthoroughly

ConfiguraUonandinstallaUonprocesses

Challengesandcommonmistakes

ThispaperisintendedtobeusedbyOracleDBAsthathaveatleastaliVlebitofLinuxknowledgeandhaveinstalledsomeversionofLinuxatleastonce.RACexperienceisalsohelpful,butmostDBAswillprobablybeabletocompletetheinstallaUonwithouthavingdoneitbefore.Thisdocumentisjustarecipeonhowtoinstalltheproducts,nothow

Paper4881

DBA,Clustering

tousethem.Ifyou’renewtoRACandneedamorebasicintroducUontoRAC,checkoutthe“RACForBeginners:TheBasics”whitepaperandpresentaUonfoundonlineathVp://www.dannorris.com/professional‐acUviUes/underthePresentaUonssecUon.

ORACLERACARCHITECTUREOVERVIEW TheobviousdifferencesbetweenRACandatradiUonalsingle‐instancedatabaseareapparentonseveraldifferentlevels.ThesecUonsthatfollowdescribethesedifferencesinmoredetail.

SYSTEMARCHITECTURE

ThediagramaboveshowsasimpleRACclusterarchitecture.AminimalclusterconsistsofmulUpleservers,aprivatenetwork,apublicnetwork,andsharedstorage.MostDBAsarefamiliarwiththesingle‐instanceconfiguraUoninwhichstorageisnotsharedbetweennodesandallaccesstothedatabasecomesfromthesingleinstance.

Sharingthesamestorageposesseveralchallenges.First,theoperaUngsystemmustbeabletohandleaccessesfromothernodesatthesameUmeasitaccessesthestorage.IntheearlydaysofOracleParallelServer(OPS),thiswasaddressedbyusingrawdevices.TheoperaUngsystemswereaccustomedtofilesystemsthatsupportedaccessfromexactlyonehost.Whenonlyonehostwasinvolved,thehostcouldmanagealllockingandsynchronizaUonfromasingleoperaUngsystemkernel.WithmulUplehosts,thefilesystemaccessmustbecoordinatedbetweenallhostssharingthefilesystem.Thisintroducessomelatencyand,unUlfairlyrecently,cluster‐widefilesystemswereveryuncommon.Today,severalvendorsofferclusterfilesystemsthataresupportedbyOracleRAC.OraclealsooffersitsOCFSfilesystemforLinuxandWindowswithreportedplanstomakeitavailableformoreplaTormsinthefuture.

Sharingadatabasealsocreateschallenges.Beingabletohandlethecross‐hostlockingrequiredtoensuredatabaseintegrityisthefirstandmostimportantchallenge.Previously,thatchallengewasaddressedbyexchangingblocksthatrequiredserializaUonviadisk.ThiswasknownasablockpingandwasamajorlimiUngfactorforOPS.WithCacheFusiontechnologyintroducedinOracle9i,theblockpingwasputontheendangeredspecieslistandwasonlyrequiredforasmall,infrequentlyseensetofcircumstances.

TheaddiUonalhardwarerequiredforaRACclusterusuallyconsistsofafewaddiUonalnetworkstosupporttheprivateinterconnect.TheprivateinterconnectcanbemadeupofmulUplephysicalnetworkinterfaces.ItiscommontouUlizetwoormoregigabitEthernetinterfacesfortheinterconnect.Inpastversions,expensiveandproprietaryinterconnecttechnologieswereused.TheseproprietaryinterconnectswererequiredbecausegigabitEthernetcreatedtoomuchlatencyanddidnothavethebandwidthnecessarytosupportsomeenvironments.WithadvancementsinEthernettechnologyandtheabilitytouUlizemulUpleinterconnects,gigabitEthernetisthemostcommoninterconnectinusetoday.InfinibandisarelaUvelynewtechnologythatwasrecently(10.2.0.3)cerUfiedforuseasanOracleprivateinterconnect.InfinibandpromisestobeasignificantplayerintheRACenvironmentbecauseofitsimpressivetechnicalspecificaUons.

Finally,beforewemoveon,let’sreviewsomepointsaboutdatabasesandinstances.OracleRACis,bydefiniUon,amulU‐instancedatabase.Thereisonedatabase.That’so‐n‐e,asin1,uno.That(one)databaseresidesonsharedstorageandisaccessedconcurrently(that’satthesameUme)bymulUpleinstancesthatrunonseparatehosts.Readthatsentence

Paper488

Shared Storage

Local Storage

Local Storage

RAC1 RAC2

2

DBA,Clustering

again—itiscriUcalthatyouunderstandwhatyou’rebuildingandthatlastsentencesumsitallup.Databaseandinstancearedefinedinthetableabove.It’sagoodideatoreviewtheirmeaningssinceoEenUmesin“normal”Oracleenvironments,thetermsdatabaseandinstancetendtobeusedinterchangeably.That’sfine,butitwillbenefityoutotrytousethosewordsproperlyintheRACenvironmentcontextwhileyou’relearninghowtheenvironmentworks.

AswemoveontotheclusterimplementaUonprocesses,keepinmindtheterminologyand“bigpicture”ofwhatwe’retryingtobuild.Forme,italwayshelpstorefertotheoverallarchitecturedesignwe’retryingtocreatefrequentlyduringcomplexinstallaUons.

TheimplementaUontopicscoveredintheremainderofthisdocumentare:

Systemrequirements

Prerequisitechecking

ConfiguraUonandinstallaUonprocesses

Challengesandcommonmistakes

SYSTEMREQUIREMENTSFORRAC ThefollowingsecUonsdescribesomeoftheconfiguraUon.

SUGGESTEDMINIMUMSFORCPU,RAM,DISK

IfyoureallyhavepaUenceandcanwaitforextendedperiodsfortheinstallaUonstocomplete,youcangenerallycompletetheinstallaUonsandsuccessfullybuildaclusterwithverymodesthardware.However,ifyouputanystressonsuchasystem,itislikelytoproduceundesirableresultsassomeCPUisrequiredtomaintainclusterheartbeatsandoverallclusterhealth.

Resource DocumentedMinimums SuggestedMinimums

CPU nonedocumented 2Ghz(forx86)

Memory 1Gb 2Gb

DiskSpace 3.5Gb(binaries) atleast5Gb

SwapSpace1.5xRAMifunder2Gb;sameasRAMif2‐8Gb;0.75xRAMif>8Gb

followdocumentedminimums

PLANNING ThemostimportantpartofanyRACimplementaUonistheplanningphase.PlanningisrequiredforstorageconfiguraUon,networkconfiguraUon,databaseconfiguraUon,services,andORACLE_HOMElocaUon.EachoftheseareashasseveralopUons,eachwithitsownadvantagesanddisadvantages.CarefulplanningwillmakeforasmoothimplementaUon.

STORAGE

Storageplanninginvolvestwomaincomponents:accesspathplanningandLUNconfiguraUonplanning.

Whenplanningaccesspaths,you'llneedtoconsiderifyou'reaccessingthestorageviafibrechannel,iSCSIorNFS.Ineachcase,considerthatthestorageaccesspathmaybeasinglepointoffailure(SPOF)ifitisn'tconfiguredwithredundancy.Foreachstorageaccesspathtype,themethodformakingthepathredundantmaybedifferent.SinceNFSusesTCP/IP,redundantNICsusingNICbonding(onLinux,otherplaTormsmaynotcallit“bonding,”buthavesimilarmethods)maybesufficient.Withfibrechannel,you'lllikelyneedmulUpathingsoEwaredrivers.OnLinux5,therearemulUpathdriversbuilt‐in.OnalmostallplaTorms,enterprisestoragevendorsprovidetheirownmulUpathsoEware(forexample,EMCPowerPathandHPSecurePath).

Paper4883

DBA,Clustering

Ofcourse,themoreobviousstorageplanningtaskisdeterminingwhatandhowstorageshouldbepresentedtothehostOSes.WhiletherearemanygoodsourcesforinformaUontoday,mostpeoplefollowOracle'sStripeAndMirrorEverything(SAME)methodologywhenconsideringtheirstorageconfiguraUon.Withlargestoragearrays,itiscommontoseeasmallnumberoflargeRAIDsetsthatare“carved”upintoindividualLUNs.Inthiscase,itdoesn'tusuallyprovideanybenefittopresentmanysmallLUNsthatallresideonthesameRAIDset.Plus,withmoreLUNs,thereismoremanagement.So,myrecommendaUonistouseasmallnumberofLUNs(usually1or2)incaseswheretheLUNsallresideonthesameRAIDsetanyway.TheonlyexcepUontothisiswhentherearemanydatabasesonthecluster.Inthatcase,IprefertohaveamaximumofonedatabaseperLUNsothatwecanmeasureI/OattheOS(andpossiblyatthearraytoo)separatelyforeachdatabase.ThisisimportantwhentryingtodeterminewhichdatabaseiscausingapotenUalperformanceissue.

NETWORK

OneoftheconfiguraUonareasIoEenseeneglectedisnetworkredundancy.Whileitistruethatnetworkinterfacecards(NICs)don'tfailveryoEen(certainlynotasoEenasdisks),theycanfailandtheyusuallydo—attheworstpossibleUme.MyexperiencehasbeenthatNICsusuallydon'tfailallatonceeither,makingtheirgradualdegradaUondifficulttodetectandtroubleshoot.

AllNICsneedtobeconfiguredwitharedundantbackup.Insomecases,youmaybeabletodobeVerthananacUve/passiveconfiguraUon(withonlyoneNICofeachpairacUve)andconfigureacUve/acUve(alsosomeUmescalled“trunking”)NICswherethebandwidthofallNICsareaggregated.Ineithercase,whenaNICfailureisdetected,thesurvivingNICinthepairshouldtakeoverimmediatelyandseamlessly.

SomeUmes,usuallyduetohardwarelimitaUonsinbladechassis,siteschoosetoaccepttheriskofbasicallyhavingacompletenodefailureifaNIConthatnodefails.

ItisbesttoconfiguretheNICswithredundancy(bondingortrunking)beforeyoustarttheOracleClusterwareinstallaUon.

TheotherpartofnetworkplanningischoosingtheproperIPaddressesthatwillbeused.Foreachnodeinthecluster,you'llneedatleastthreeIPaddresses:public(IsomeUmescallthisthe“base”IPaddressofthenode),private(interconnect),andvirtual(VIP,fromthesamenetworkasthepublicnetwork).

Inthispaper,wewillshowhowtobondnetworkinterfacesusingbuilt‐infuncUonalityonLinux.TrunkingusuallyrequiresspecialsoEwareandhardwareconfiguraUonandisbeyondthescopeofthispaper.

DATABASE

PlanningthedatabaseisprobablyoneoftheeasiestpartsformostDBAssinceitisaboutthesameasplanningnon‐RACdatabaseimplementaUons.Planningforthedatabaseusuallyjustinvolvesplanningtablespacelayouts,opUonalcomponentstoinstallduringcreaUon,blocksizeandcharactersetchoices,andchoosinginstanceanddatabasenames.

TheonlydifferencefordatabaseplanningforRACcomparedtosingle‐instanceisthatnamingwillinvolvemakinganamingprefixthatwillbeusedforallinstances.Forexample,ifyouchooseaprefixof“dan”,thentheinstanceswillbenamed“dan1”,“dan2”,“dan3”,andsoon.Stayingwiththisnamingformatofprefix+numberisthedefaultexpectedbythedatabaseconfiguraUonassistant(dbca)andisalsothemostcommondeployment.However,technically,youcoulduse“fred”and“barney”fortwoinstancenamesofthesamedatabase—itjustendsupbeingmoreconfusingthanthenormalnamingmethod.Typically,usinga“scheme”(likecartooncharacters)doesn'tscalewellandifyoudoendupwithalargerclusterthanmembersofthegroupyouused,thenyouhavetostartmakinguprandomnames,therebybreakingfromthescheme.

SERVICES

OracleServicesareanimportantpartofanyRACclusterastheyprovidethebestmethodformanagingthecluster'sworkload.IfyouarenotyetfamiliarwithServices,youshouldgeteducated.AnexcellentstarUngpointismyfriend

Paper4884

DBA,Clustering

JeremySchneider'swhitepaperonthetopic:“UnleasingOracleServices:AComprehensiveReviewof'Services'inOracleDatabases”availableathVp://www.ardentperf.com/pub/services‐schneider07.pdf

Asyouknow,Servicescanbecreatedtohandleanytypeofgrouping.SomesitesprefertomakeservicesforindividualapplicaUons(ifthedatabasehasmulUpleapplicaUonsthataccessit),typesofusers(HR,Payroll,Purchasing,etc.),applicaUontype(batchprocessing,OLTP,BI/DW),orotherspecialapplicaUons.IfyouusetheDatabaseResourceManager,youcanalsoaVacharesourceprofiletoaservice.IfyouuseOracle10gorhigher,eachinstancealsotracksstaUsUcsonaper‐servicebasis(v$service_stats,v$service_wait_class,etc.)thatcanhelpyoudeterminewhichserviceisconsumingthelargestshareofresourcesonthesystem.

YourdeploymentofServicesshouldbesomethingyouplanaheadofUmeanddiscusswiththebusiness.AstheDBA,oneofyourrolesistotakeadvantageoffeaturesofferedbythedatabaseandmapthemtobusinessneeds.Forexample,ifyouanUcipatethatthebusinessmaywanttoknowwhetheritwouldhelptomovetheHRuserstoanotherdatabase(ormaybejusttotheirowninstance),thenyoushouldconsidercreaUngaservicefortheHRuserstoprovidethatdatalater.Servicesprovideworkloadmanagementbygivingyouaneasywaytoisolateaservicetoaspecificinstanceorsub‐groupofinstancesinthecluster.Theworkloadmanagementfeaturesarebeingenhancedwitheachrelease,soitislikelythatmorenewfeatureswillbenefitthosethathavealreadyimplementedandusedServiceswisely.

SHAREDVS.LOCALORACLE_HOMES

Thistopicisincludedinyourplanningasanawarenessthatitispossible,withsomeconfiguraUons,tocreateasingle,sharedORACLE_HOMEforyourcluster'ssoEware.You'llsUllneedseparateORACLE_HOMEsforOracleClusterwareandOracleRDBMSbinaries(andsuggestaseparateoneforOracleASMbinariestoo,ifyouuseASM).Ifyouhaveasupportedclusterfilesystem,youcansharethebinariesamongallnodesandavoidhavingmulUpleORACLE_HOMEstomaintain.ThereisanexcellentwhitepaperprovidedbyOracleonthistopic(seeitathVp://www.oracle.com/technology/products/database/clustering/pdf/oh_rac.pdf).TheshortsummaryofthatwhitepaperisthatlocalORACLE_HOMEsaresuggestedforsmallclusters(I'dsaylessthan5nodes).Ifyouhavelargerclusters,IwouldconsidermulUplesharedORACLE_HOMEs,butIwouldneverrecommendasingle,cluster‐wideORACLE_HOMEasitcreatesapotenUalsinglepointoffailure.IunderstandtheargumentthatmaintainingmulUpleORACLE_HOMEsismorework,buttheinstallersimplifiesmuchofthemaintenanceprocessandyoualsohavetheeasyabilitytoperformrollingupgradesandpatches(forthosepatchesthatsupportrollingupgrades).

MyrecommendaUonistouselocalORACLE_HOMEsonlocalstorageoneachnodeofyourcluster.

PREPAREOPERATINGSYSTEM ThisprocessismostlyderivedfromagreatarUclewriVenbyTimHallatOracle‐Base.ThearUclecanbefoundonlineathVp://www.oracle‐base.com/arUcles/11g/OracleDB11gR1RACInstallaUonOnOEL5UsingVMware.php(nottheeasiestURLtoremember).YoucanalsofinditbygoingtohVp://www.oracle‐base.com/andusingthe“OracleRACArUcles”linkintheleE‐handmenu.ThankstoTimforgraciouslyallowingmetousetheinformaUoninthatarUcleforthispaperaswell.Onelastnote—TimalsowroteandhostsarUclesonbuildingthesamesetupusingthepossiblecombinaUonsof10g,11g,Linux,andWindows.

IwillforegothemyriadscreenshotsofeachinstallaUonstepasTim’sarUclecoversthatnicely.Instead,I’lldescribeeachstepandtrytokeepyouoncoursetocompletetheinstallaUontoendupwithasuccessfulcluster.Youdon’tneedanythingotherthanthisdocumenttoguideyou,butreadingTim’sarUclemaybehelpful.

INSTALLOS

WiththeCDinplace,bootthesoon‐to‐benode1.Oncebooted,followthesestepstoinstalltheOS:

Installasyouwouldnormallyinstallanyserver.FordetailedinstrucUonsonserverinstallaUon,TimHallhasanotherfinearUclepostedathVp://www.oracle‐base.com/arUcles/linux/OracleEnterpriseLinux5InstallaUon.phpthatwalksthroughtheinstallaUonstep‐by‐step.

Whenprompted,choosethefollowingpackagegroups:

Paper4885

DBA,Clustering

GNOMEDesktopEnvironment

Editors

GraphicalInternet

Text‐basedInternet

DevelopmentLibraries

DevelopmentTools

ServerConfiguraUonTools

AdministraUonTools

BaseSystem

SystemTools

XWindowSystem

UnderBaseSystem‐>SystemTools‐>OpUonalpackages,makesuretoselect“oracle‐validated”.YoucanopUonallychoosetoinstalloracleasmpackagesandocfs2packagesfromthatsamelistofopUonalpackages.

InmysampleinstallaUon,IusedthefollowinginformaUonforhostidenUficaUon:

hostname:node1.dannorris.net

IPAddresseth0:192.168.78.51(publicaddress)

DefaultGatewayeth0:192.168.78.1(publicaddress)

IPAddresseth1:192.168.150.51(privateaddress)

DefaultGatewayeth1:none

Disablethebuilt‐infirewallandalsodisabletheSELinuxsubsystem.Disablingthesecomponentswillmakeiteasiertoconfigureandusethesystem.

Installnode2inasimilarway(makeexactlythesamechoicesduringinstallaUon)andentertheappropriatehostnameandIPaddressesforit.

WiththeinstallaUoncompleteandpackagesinstalled,wecanmoveontoconfiguringtheOracleprerequisites.

POST‐INSTALLATIONCONFIGURATION

TheLinuxprerequisitesforinstallingOracleRACareallthesameonesforinstallingasingleinstanceenvironment,plusafewaddiUonalrequirements.Thefollowingstepsarethe“cheatsheet”forconfiguringthenecessarypartsoftheOStopreparefortheOracleinstallaUons.

Notethatifyouchosetheoracle-validatedpackageduringtheinstallaUonprocess,mostofthesechangesshouldalreadybemadeforyou.So,firstconfirmthatthechangesaren'talreadymadebeforeaddingtoormodifyingthefilesmenUoned.Thesechecksmustbedoneonbothnodes.

Whilenotanofficialprerequisite,Iusuallychangethedefaultrunlevelfrom5to3inordertodisableXwindowsontheconsole.ThissavessomeRAMontheserverandXwindowsisrarelyneededanyway.Tochangethis,modifytheinitdefaultseWnginthe/etc/inittabfilefrom5to3.Then,asroot,run“init q; telinit 3”tomakethechangeeffecUveimmediately.IfyoupreferaGUIconsole,don’tmakethischange—itisopUonal,butsinceitfreesupmemory,performancemayimprovebyfollowingthisrecommendaUon.

Modifythe/etc/hostsfiletolookliketheonebelow(replacethewholefilewiththesecontents):127.0.0.1 localhost.localdomain localhost192.168.78.120 node1.dannorris.net node1192.168.78.121 node2.dannorris.net node2172.16.100.120 node1-priv.dannorris.net node1-priv172.16.100.121 node2-priv.dannorris.net node2-priv

Paper4886

DBA,Clustering

192.168.78.130 node1-vip.dannorris.net node1-vip192.168.78.131 node2-vip.dannorris.net node2-vip

Addthefollowinglinestothe/etc/sysctl.conffile(NotethatonOEL5,manyoftheparametersthatyouhadtoaddinpreviousversionsarenowinthefilebydefault):kernel.shmmni = 4096# semaphores: semmsl, semmns, semopm, semmnikernel.sem = 250 32000 100 128net.ipv4.ip_local_port_range = 1024 65000net.core.rmem_default=4194304net.core.rmem_max=4194304net.core.wmem_default=262144net.core.wmem_max=262144

Oncethe/etc/sysctl.conffileismodified,forcethenewparametersintoeffectimmediatelywith“sysctl –p”

Addthefollowinglinesto/etc/security/limits.conf:@dba soft nproc 2047@dba hard nproc 16384@dba soft nofile 1024@dba hard nofile 65536

Addthefollowinglinestothe/etc/pam.d/loginfile:session required /lib/security/pam_limits.sosession required pam_limits.so

Addthenecessarygroupsandusertothesystemwiththefollowingcommands:groupadd -g 500 oinstallgroupadd -g 501 dbagroupadd -g 502 asmadminuseradd -u 500 -g oinstall -G dba,asmadmin -s /bin/bash oraclepasswd oraclemkdir -p /u01/app/oracle /u01/crs/oraclechown -R oracle:dba /u01

Setdefaultshellvariablesbyaddingthefollowingtothe/home/oracle/.bash_profilefile:ORACLE_BASE=/u01/app/oracle; export ORACLE_BASEPATH=/usr/sbin:$PATH; export PATH

if [ $USER = "oracle" ]; then if [ $SHELL = "/bin/ksh" ]; then ulimit -p 16384 ulimit -n 65536 else ulimit -u 16384 -n 65536 fifi

Configurethehangcheck‐Umerkernelmodulebyaddingthefollowinglineto/etc/rc.d/rc.local:modprobe hangcheck-timer hangcheck_tick=1 hangcheck_margin=10 hangcheck_reboot=1

SeeMetalinkNote726833.1formoreinformaUononhangcheck‐UmerandrequirementsforRAC.

CONFIGURENICBONDING

ThefirststeptoconfiguringNICbondingisseWngupthekernelparametersforthebondinginterfaces.Theseparameterswillcreatethepseudodevicesforthebondedinterfaces.ThehardestpartofpairingtheNICsisfiguringoutwhichtwoshouldbepaired.Usually,theeasiestwaytodeterminewhichonesareonwhichnetworkistousetheMACaddresses.YournetworkandsystemadministratorsshouldbeabletoassistwiththedeterminaUons.

Inourcase,we'llassumethatwehave4NICs(2forpublic,2forprivate).We'regoingtopaireth0andeth2forthepublicnetwork(bond0)andeth1andeth3fortheprivatenetwork(bond1).Tocreatethebonds,firstchangethe/etc/modprobe.conffiletoaddtheselines(don'trebootnow,we'lldothatlater):

alias bond0 bondingalias bond1 bonding

Paper4887

DBA,Clustering

options bond0 mode=active-backup miimon=100 max-bonds=2options bond1 mode=active-backup miimon=100 max-bonds=2

Next,createanewfileonnode1named/etc/sysconfig/network‐scripts/ifcfg‐bond0withthefollowingcontents:DEVICE=bond0 IPADDR=192.168.78.120 NETMASK=255.255.255.0 ONBOOT=yes BOOTPROTO=none USERCTL=no MII_NOT_SUPPORTED=yes

Andanothernewfileonnode1named/etc/sysconfig/network-scripts/ifcfg-bond1withthefollowingcontents:

DEVICE=bond1IPADDR=172.16.100.120 NETMASK=255.255.255.0 ONBOOT=yes BOOTPROTO=none USERCTL=no MII_NOT_SUPPORTED=yes

Createthesesametwofilesonnode2,butreplacetheIPaddressesappropriatelysothatthelastoctetis121ineachfile.

Onnode1,youneedtomodifythecontentsof/etc/sysconfig/network-scripts/ifcfg-eth0.Inthisfile,youshouldkeeptheexisUngHWADDRline,butreplacetherestofthelinessothatthefilelookslikethis:

DEVICE=eth0 USERCTL=no ONBOOT=yes MASTER=bond0 SLAVE=yes BOOTPROTO=none MII_NOT_SUPPORTED=yes HWADDR=<value-from-existing-file> Dothesamethingfor/etc/sysconfig/network-scripts/ifcfg-eth2,keepingitsHWADDRlineand

modifyingtheDEVICE=linetoreadeth2(allotherlinesshouldmatchabove).

Similarly,weneedtomodifythe/etc/sysconfig/network-scripts/ifcfg-eth1and/etc/sysconfig/network-scripts/ifcfg-eth3files,butinthosefiles,we'llsetMASTER=bond1.

Finally,withallconfiguraUonchangesinplace,ensurethatyouhaveaccesstotheserver'sconsole(incasethechangesdon'tworkwell,networkconnecUvitymaybedisabled).Onceyoudo,itissafetoreboottheserverand,hopefully,ifallchangesaredoneproperly,you'llbeabletoreconnectaEerthereboot.Then,youshouldseeaconfiguraUonlikethis(thisissampleoutputfromnode1):

[root@node1 ~]# ifconfig -abond0 Link encap:Ethernet HWaddr 00:0C:29:FC:5F:65 inet addr:192.168.78.120 Bcast:192.168.78.255 Mask:255.255.255.0 inet6 addr: fe80::20c:29ff:fefc:5f65/64 Scope:Link UP BROADCAST RUNNING MASTER MULTICAST MTU:1500 Metric:1 RX packets:127 errors:127 dropped:0 overruns:0 frame:0 TX packets:123 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:17143 (16.7 KiB) TX bytes:19870 (19.4 KiB)

bond1 Link encap:Ethernet HWaddr 00:0C:29:FC:5F:6F inet addr:172.16.100.120 Bcast:172.16.100.255 Mask:255.255.255.0

Paper4888

DBA,Clustering

inet6 addr: fe80::20c:29ff:fefc:5f6f/64 Scope:Link UP BROADCAST RUNNING MASTER MULTICAST MTU:1500 Metric:1 RX packets:80 errors:77 dropped:0 overruns:0 frame:0 TX packets:111 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:11408 (11.1 KiB) TX bytes:15874 (15.5 KiB)

eth0 Link encap:Ethernet HWaddr 00:0C:29:FC:5F:65 inet6 addr: fe80::20c:29ff:fefc:5f65/64 Scope:Link UP BROADCAST RUNNING SLAVE MULTICAST MTU:1500 Metric:1 RX packets:122 errors:122 dropped:0 overruns:0 frame:0 TX packets:119 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:14015 (13.6 KiB) TX bytes:18422 (17.9 KiB) Interrupt:67 Base address:0x2024

eth1 Link encap:Ethernet HWaddr 00:0C:29:FC:5F:6F inet6 addr: fe80::20c:29ff:fefc:5f6f/64 Scope:Link UP BROADCAST RUNNING SLAVE MULTICAST MTU:1500 Metric:1 RX packets:56 errors:53 dropped:0 overruns:0 frame:0 TX packets:90 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:7014 (6.8 KiB) TX bytes:11712 (11.4 KiB) Interrupt:67 Base address:0x20a4

eth2 Link encap:Ethernet HWaddr 00:0C:29:FC:5F:65 inet6 addr: fe80::20c:29ff:fefc:5f65/64 Scope:Link UP BROADCAST RUNNING SLAVE MULTICAST MTU:1500 Metric:1 RX packets:23 errors:23 dropped:0 overruns:0 frame:0 TX packets:22 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:4316 (4.2 KiB) TX bytes:4396 (4.2 KiB) Interrupt:51 Base address:0x2424

eth3 Link encap:Ethernet HWaddr 00:0C:29:FC:5F:6F inet6 addr: fe80::20c:29ff:fefc:5f6f/64 Scope:Link UP BROADCAST RUNNING SLAVE MULTICAST MTU:1500 Metric:1 RX packets:24 errors:24 dropped:0 overruns:0 frame:0 TX packets:21 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:4394 (4.2 KiB) TX bytes:4162 (4.0 KiB)

Interrupt:59 Base address:0x24a4

Youcanquerythestatusofthebondsbylookingatthecontentsof/proc/net/bonding/bond0(orbond1).Undernormalcircumstances,here'swhatthebond0statusinformaUonmightlooklike:[root@node1 ~]# cat /proc/net/bonding/bond0Ethernet Channel Bonding Driver: v3.2.4 (January 28, 2008)

Bonding Mode: fault-tolerance (active-backup)Primary Slave: NoneCurrently Active Slave: eth0MII Status: upMII Polling Interval (ms): 100Up Delay (ms): 0Down Delay (ms): 0

Slave Interface: eth0

Paper4889

DBA,Clustering

MII Status: upLink Failure Count: 0Permanent HW addr: 00:0c:29:fc:5f:65

Slave Interface: eth2MII Status: upLink Failure Count: 0Permanent HW addr: 00:0c:29:fc:5f:79[root@node1 ~]#

CONFIGURESHAREDSTORAGE

Next,weneedtoconfigurethesharedstoragethatwewillusetostoretheRACdatabase.TheprocessforconfiguringstoragewillvarydependingonwhichtypeofstorageaVachment(fibrechannel,iSCSI,NFS,etc.)youuse.YoushouldabsolutelyconfigureredundantpathstoyourstoragenomaVerwhichstoragetypeyouuse.

Becauseofthevariedmethodsforconfiguringredundantpaths,we'llleavethattopicforthesystemandstorageadministratorstoaddress.Oncethestorageisaccessibletoallhostsinthecluster,itneedstobeconfiguredforusewithRAC.Forthiscluster,wewilluseASM(withoutASMLib),soourfirststepistoparUUonthedisks.Ittypicallydoesn'tmaVerwhichpathtothediskyouuseinordertoparUUonit.However,itmakesahugedifferencewhichpathyouusetoaccessitfornormalread/writeaccess,sopaycloseaVenUontothepathsusedforeach.

YoucanparUUonthediskswiththesecommandsforeachdisk(/dev/sdbthrough/dev/sdf):fdisk /dev/sdb,thenenter“n”,“p”,“1”,<return>,<return>,and“w”.ThatwillcreateasingleparUUonthatisthesizeofthewholediskoneachdisk.

OnLinux5,theproperwaytosetprivilegesonthediskdevicefilesistomodifyfilesin/etc/udev/rules.d.SeeMetalink465001.1,564580.1,and605828.1,asapplicable,forinstrucUonsontheproperconfiguraUonfordiskdevicefiles.

Seethenoteslaterinthisdocument(onpage16,under“SharedStorageMisconfiguraUon”)aboutusingudevtoconfiguredeviceaccess.udevisabeVerandmoresupportablemethodandthenotesmenUonedinthatsecUonwillprovideinstrucUonsonhowtoconfigureanduseudevonyoursystem.

TherearenorawdevicestocreateonOEL5,sowe’redoneconfiguringoursharedstorageatthispoint.

WhiletherearewaystorunaseriesofcommandstorefreshthecachedparUUontablesforeachdiskineachhost,itisprobablyeasiesttorebootallnodesatthispointtomakethingseasier.WhenmakingalargenumberofconfiguraUonchanges,itisalsohelpfultorebootthenodesperiodicallyduringtheconfiguraUonstepstoensurethattheywillbootwithouterror.

ESTABLISHHOSTEQUIVALENCY

Ournextstepswillestablishhostequivalencyfortheoracleusersothattheoracleaccountisfreetologintotheremotenodewithoutbeingpromptedforapassword.Inourcase,we’renotasconcernedwithsecurityaswearewithkeepingtheconfiguraUonverysimple,sowewon’tprotectourprivatekeywithapassword(notabestpracUceforproducUon).

NotethatoncetheinstallaUonsarecompleted,thereisnorequirementforhostequivalency(otherthanconvenienceformaintenanceoperaUons),soyoumaydisablethehostequivalencyestablishedinthissecUononceallinstallaUonsarecomplete.

Toestablishhostequivalency,followthesesteps:

Logintonode1astheoracleuser.Run“ssh-keygen –t dsa”tocreateanSSHkey.Specifyapassphraseforthekeyasyouwish.Youcanalsoleaveitblankandavoidthessh‐agentandssh‐addcommandscomingupinafewsteps.However,aprivatekeyfilewithoutapassphraseisadangeroussecuritypracUceandleavestheaccountopentocertainvulnerabiliUes,soIrecommendusingapassphraseforallkeyfiles.

Paper48810

DBA,Clustering

AEerthekeyiscreated,run“cp ~/.ssh/id_dsa.pub ~/.ssh/authorized_keys”toprepareusforthehostequivalencyrequiredlaterfortheinstallaUons.Alsorunthesecommandsastheoracleuser“eval `ssh-agent`; ssh-add”andthenenterthepassphraseyoucreatedforyourSSHkey.ItisimportantthatyourunthesecommandsfromthesameshellenvironmentwhereyouwillulUmatelyruntheOracleInstallerfrom.

Alsodothisastheoracleuseronnode1:scp -rp ~/.ssh node2:.(thepreceedingperiodispartofthecommand)ThiswillcopytheenUre.ssh directoryovertonode2sothat,mostimportantly,theauthorized_keysfileisinplaceandwecanrunremotecommandsonnode2withoutbeingpromptedforapassword.

Foreachhostinthecluster,sshtothathosttoconnectbyrunningthefollowingcommandsinsuccessionfromtheoracleaccount.Notethatyouwillbepromptedtoaccepteachhost’skey;respond“yes”whenprompted.Youshouldn'tbepromptedtoenterapasswordforanyoftheseconnecUons.ssh node1 datessh node1-priv datessh node1.dannorris.net datessh node1-priv.dannorris.net datessh node2 datessh node2-priv datessh node2.dannorris.net datessh node2-priv.dannorris.net date

Atthispoint,we’vedonealltheOSpreconfiguraUonandhaveallthenecessaryitemsinplacetobegininstallingsoEware.Ifyou’recauUousandwanttosavesomeUmeintheeventofacatastrophe,thisisagoodpointtoshutdownthevirtualmachinesandtakeacopyofthemforbackuppurposes.

CONFIRMPREREQUISITESWITHTOOLS:CLUVFYANDRDA MostRACimplementaUonsthatseemdifficultareusuallyplaguedwithchallengesbecauseofmissingprerequisites.ItissomeUmesdifficulttocheckallprerequisitesaccuratelyforRACclustersandthat'swhyOraclehasbuiltafewtoolstohelpwiththeprocess.TheClusterVerificaUonUUlity(a.k.a.CVUorcluvfy)andRemoteDiagnosUcAssistant(RDA)helpautomatealltheprerequisitechecksnecessaryforRACclusters.OEen,theyincorporateupdatedrequirementsthataren'tinthestandarddocumentaUonaswell.ItishighlyrecommendedtoruneachoftheseuUliUesoneachnodeinthecluster.

CLUSTERVERIFICATIONUTILITY(CVU)

TheCVUisavailablefromOTNandisalsoavailableontheOracleClusterwareinstallaUonmedia.IrecommenddownloadingthelatestversionfromOTNathVp://www.oracle.com/technology/products/database/clustering/cvu/cvu_download_homepage.htmlsinceitisoEenupdatedaEertheOracleClusterwaremediaiscreated.

TheCVUhasmanycommand‐lineopUonsasitprovidesdifferentchecksthatcanberunatvariousstagesoftheinstallaUon(pre‐Clusterwareinstall,pre‐DBinstall,post‐DBinstall,etc.).Inourcase,wewanttorunthepre‐ClusterwareinstallaUonchecks.

ToruntheCVUchecks,gotothelocaUonwhereyouunpackedtheCVUdownloadandrunitastheoracleuser.Beforerunningit,ensurethatyouhavestartedthessh‐agentandloadedyourprivatekeyfileintotheagentsothatyouhavehostequivalencywiththeotherclusternodes.Here'showtorunCVU:cd /path/to/cvu/unzip/directory./cluvfy stage –pre crsinst –n node1,node2 –r 11gR1

Theoutputyou'llseeisaseriesoftests.Eachtestwillhaveastatusreportedassuccessfulorfailed.

NotethatonLinuxx86‐64,youmayseesomeofthepackageslistedtwice.Whileitlookslikethesearethesamepackages,itiscommonforoneofthetwocheckstofailwhiletheothersucceeds.Thesechecksareforthesamepackage,butonecheckisforthe32‐bitversionandtheotherforthe64‐bitversion.The32‐bitversionisn'tinstalledby

Paper48811

DBA,Clustering

default.ThedocumentaUongivesalistofthepackageprerequisitesincludingthepackagesthatneedtobeinstalledwithboththe64‐bitand32‐bitversions.Tolistthepackagesandtheirarchitecturestogether,usethisrpmcommand:rpm -qa –queryformat="%{NAME}-%{V}-%{R}.%{ARCH}\n"

Thisallowsyoutoseemoreeasilywhichpackagesaremissing.

Here'sasampleoutputfromtheCVUonmytestLinux32‐bitcluster(notethatmyswapspaceandmemoryrequirementsarenotmet).[oracle@node1 ~]$ /mnt/runcluvfy.sh stage -pre crsinst -n node1,node2 -r 11gR1

Performing pre-checks for cluster services setup

Checking node reachability...Node reachability check passed from node "node1".

Checking user equivalence...User equivalence check passed for user "oracle".

Checking administrative privileges...User existence check passed for "oracle".Group existence check passed for "oinstall".Membership check for user "oracle" in group "oinstall" [as Primary] passed.

WARNING: The effective group id "dba" differs from the primary group id "oinstall" of user "oracle".

Administrative privileges check passed.

Checking node connectivity...

Node connectivity check passed for subnet "192.168.78.0" with node(s) node2,node1.Node connectivity check passed for subnet "172.16.100.0" with node(s) node2,node1.

Interfaces found on subnet "192.168.78.0" that are likely candidates for VIP:node2 bond0:192.168.78.121node1 bond0:192.168.78.120

Interfaces found on subnet "172.16.100.0" that are likely candidates for a private interconnect:node2 bond1:172.16.100.121node1 bond1:172.16.100.120

Node connectivity check passed.

Checking system requirements for 'crs'...Total memory check failed.Check failed on nodes: node2,node1Free disk space check passed.Swap space check failed.Check failed on nodes: node2,node1System architecture check passed.Kernel version check passed.

Paper48812

DBA,Clustering

Package existence check passed for "make-3.81".Package existence check passed for "binutils-2.17.50.0.6".Package existence check passed for "gcc-4.1.1".Package existence check passed for "libaio-0.3.106".Package existence check passed for "libaio-devel-0.3.106".Package existence check passed for "libstdc++-4.1.1".Package existence check passed for "elfutils-libelf-devel-0.125".Package existence check passed for "sysstat-7.0.0".Package existence check passed for "compat-libstdc++-33-3.2.3".Package existence check passed for "libgcc-4.1.1".Package existence check passed for "libstdc++-devel-4.1.1".Package existence check passed for "unixODBC-2.2.11".Package existence check passed for "unixODBC-devel-2.2.11".Package existence check passed for "glibc-2.5-12".Group existence check passed for "dba".Group existence check passed for "oinstall".User existence check passed for "nobody".

System requirement failed for 'crs'

Pre-check for cluster services setup was unsuccessful on all the nodes. [oracle@node1 ~]$

REMOTEDIAGNOSTICASSISTANT:HARDWARECOMPATIBILITYVERIFICATIONENGINE(RDAHCVE)

TheRDAisavailablefromMetalinkvianote314422.1andhasversionsavailableforeachplaTormsupportedbyRAC.ToruntheRDAinHCVEmode,downloadtheappropriateversionfromMetalink,placeitonyourproposeddatabaseserversandrunit(astheoracleuserisfine—norootprivilegesneeded)usingthiscommand:cd /path/to/rda/unzip/directory./rda.sh -T hcve

You'llbepromptedwithamenuofRDAprerequisitecheckers.IrecommendmulUplerunsfirstcheckingfortheregulardatabaseversionyou'reinstallingandasecondrunwhenyoushouldchoosetheRACchecks.

Onceyoumakeyourchoice,you'llbepromptedwithafewaddiUonalquesUonsandthenofferedasummaryreportofthefindings.You'remostinterestedinthethingsmarkedasFAILED.

Here'sasamplerunformytestenvironment:[oracle@node1 rda]$ ./rda.sh -T hcveProcessing HCVE tests ...Available Pre-Installation Rule Sets: 1. Oracle Database 10g R1 (10.1.0) PreInstall (Linux-x86) 2. Oracle Database 10g R1 (10.1.0) PreInstall (Linux AMD64) 3. Oracle Database 10g R1 (10.1.0) PreInstall (IA-64 Linux) 4. Oracle Database 10g R2 (10.2.0) PreInstall (Linux AMD64) 5. Oracle Database 10g R2 (10.2.0) PreInstall (IA-64 Linux) 6. Oracle Database 10g R2 (10.2.0) PreInstall (Linux-x86) 7. Oracle Database 11g R1 (11.1.0) PreInstall (Linux AMD64) 8. Oracle Database 11g R1 (11.1.0) PreInstall (Linux-x86) 9. Oracle Application Server 10g (9.0.4) PreInstall (Linux) 10. Oracle Application Server 10g R2 (10.1.2) PreInstall (Linux) 11. Oracle Application Server 10g R3 (10.1.3) PreInstall (Linux AMD64) 12. Oracle Application Server 10g R3 (10.1.3) PreInstall (IA-64 Linux) 13. Oracle Application Server 10g R3 (10.1.3) PreInstall (Linux-x86) 14. Oracle Portal PreInstall (Generic)Available Post-Installation Rule Sets: 15. Oracle Portal PostInstall (generic) 16. RAC 10G DB and OS Best Practices (Linux) 17. Data Guard PostInstall (Generic)

Paper48813

DBA,Clustering

Enter the HCVE rule set numberHit 'Return' to accept the default (1)> 8

Enter value for < Planned ORACLE_HOME location or if set >> /u01/app/oracle/product/11.1.0/db_1

Enter value for < JDK Home >> /u01/app/oracle/product/11.1.0/db_1/jdk

Test "Oracle Database 11g R1 (11.1.0) PreInstall (Linux-x86)" executed at Thu Mar 12 23:54:01 2009

Test Results~~~~~~~~~~~~

ID NAME RESULT VALUE====== ==================== ======= ==========================================A00010 OS Certified? PASSED AdequateA00020 User in /etc/passwd? PASSED userOKA00040 Group in /etc/group? PASSED GroupOKA00050 Input ORACLE_HOME RECORD /u01/app/oracle/product/11.1.0/db_1A00060 ORACLE_HOME Valid? PASSED OHexistsA00070 O_H Permissions OK? PASSED CorrectPermsA00080 oraInventory Permiss PASSED oraInventoryOKA00090 Got ld,nm,ar,make? PASSED ld_nm_ar_make_foundA00100 Umask Set to 022? PASSED UmaskOKA00120 Limit Processes PASSED AdequateA00130 Limit Descriptors PASSED AdequateA00140 LDLIBRARYPATH Unset? PASSED UnSetA00180 JAVA_HOME Unset? PASSED UnSetA00190 JDK Home RECORD /u01/app/oracle/product/11.1.0/db_1/jdkA00200 JDK Version PASSED AdequateA00210 Other O_Hs in PATH? PASSED NoneFoundA00220 Other OUI Up? PASSED NoOtherOUIA00230 /tmp Adequate? PASSED TempSpaceOKA00240 Disk Space OK? PASSED DiskSpaceOKA00250 Swap (in MB) RECORD 1023A00260 RAM (in MB) FAILED 1011A00270 Swap OK? FAILED NAA00280 Network PASSED ConnectedA00290 IP Address RECORD 192.168.78.120A00300 Domain Name RECORD dannorris.netA00310 DNS Lookup FAILED nslookup host.domainA00320 /etc/hosts Format PASSED Contains host.domainA00330 Kernel Parameters OK FAILED NAA00380 Tainted Kernel? PASSED NotVerifiableA00400 ip_local_port_range PASSED ip_local_port_rangeOKA00460 Asianux2 RPMs OK? PASSED NotAsianuxA00470 Asianux3 RPMs OK? PASSED NotAsianuxA00480 EL4 RPMs OK? PASSED NotEL4A00490 EL5 RPMs OK? PASSED EL5rpmsOKA00530 RHEL4 RPMs OK? PASSED NotRedHatA00540 RHEL5 RPMs OK? PASSED NotRedHatA00570 SUSE SLES10 RPMs OK? PASSED NotSuSEResult file: /home/oracle/rda/output/RDA_HCVE_A200DB11R1_lnx_res.htm[oracle@node1 rda]$

Paper48814

DBA,Clustering

ORACLEPRODUCTINSTALLATIONS

INSTALLCLUSTERWARE

Finally,wecanstartinstallingsomesoEware.FollowthesestepstoinstallOracleClusterwareonbothnodes.

MakeaOracleClusterware11ginstallaUonCD(orthedownloadoftheCDcontents)availabletothenode1virtualmachine.

Ifyou’vefollowedalongcarefullytothispoint,youwon’thaveagraphicalconsole.Ifthat’strue,thenyou’llprobablywanttostartVNCinordertoperformthegraphicalinstallaUons.TostartVNC,whileloggedintonode1astheoracleuser,createthefollowingfilenamed$HOME/.vnc/xstartup(youmighthavetocreatethe$HOME/.vncdirectoryfirst):xsetroot -solid greygnome-session &

Then,youcanstarttheVNCserverbyrunning“vncserver”fromthecommandpromptastheoracleuser.Notethatthevncserversessionwillinherityourenvironment,soitisbesttonothaveORACLE_HOMEorORACLE_SIDset(andORACLE_HOME/binshouldn'tbeinyourPATHeither).ThefirstUmeyourunvncserver,you’llbepromptedtosetapasswordforyourvncserverdesktop.

OnceVNCisrunning,connecttotheserverwiththeaVNCviewer.Then,ensureyourenvironmenthastheORACLE_BASEenvironmentvariablesetto/u01/app/oracle.

AEerconfirmingtheproperenvironmentseWngs,runthesecommands:cd <download_location>/clusterware; ./runInstaller

OntheSpecifyInventoryDirectoryscreen,takethedefaults(/u01/app/oraInventoryandoinstall).

OntheSpecifyHomeDetailsscreen,takethedefaultsofOraCrs11g_homeand/u01/app/11.1.0/crs

Allprerequisitechecksshouldsucceed.

OntheSpecifyClusterConfiguraUonscreen,thelocalnode(node1)seWngsshouldalreadybedisplayed.AddtheinformaUonfornode2tothescreenusingtheappropriatenamesforPublic,PrivateandVirtual.

OntheSpecifyNetworkInterfaceUsagescreen,seteth0tobePublicandleaveeth1asPrivate.

OntheSpecifyOCRLocaUonscreen,chooseyourdesiredredundancylevel(IpreferNormal)andentertheappropriatemulUpathedpseudodevicefile(s)fortheOCRlocaUon.

OntheSpecifyVoUngDiskLocaUonscreen,chooseyourdesiredredundancylevel(IpreferNormal)andentertheappropriatemulUpathedpseudodevicefile(s)forthevoUngdisklocaUon.

OntheSummaryscreen,ensurethattheClusterNodeslistincludesbothnodesandthenclickInstall.

OncetheinstallaUonsfinish,runtherootscriptsasprompted.Remembertorunthemserially—notconcurrently.

AEertherootscriptsarerun,clickOKandthenthefinalassistantsrun.Allassistantsshouldrunwithoutissueorerror.Somewarningsintheroot.shoutputarenormal,though.

INSTALLASMANDRDBMSSOFTWARE

WerecommendinstallingASMinaseparateORACLE_HOMEdirectoryfromtheRDBMSsoEware.TheASMsoEwareisjustthesameastheRDBMSsoEwareexceptthatinsteadofconfiguraUonadatabase,youconfigureittorunASM.So,repeatthestepsbelowtwicesothatyouhavetwoseparateORACLE_HOMEswithidenUcalsoEwareinthem.

HerearethestepstoinstallandconfigureASMandRDBMS.

Onceagain,we’lluseVNCtoperformthisinstallaUonfromthenode1node.MaketheRDBMS11gsoEwareCDsordownloadsavailableonnode1(eitherlocallyorviaanNFSmount).We’llassumethatthesoEwareisavailableat/mnt/cdrom/db11gr1-linux/databaseinourcase.

Paper48815

DBA,Clustering

WithintheVNCsessionthatisrunningastheoracleuseraccount(ensureORACLE_BASE issetintheenvironment),runtheinstallerbyrunningthesecommands:“cd /mnt/cdrom/db11gr1-linux/database ; ./runInstaller”

OntheSelectInstallaUonTypescreen,chooseEnterpriseEdiUonorStandardEdiUon,asappropriate.

OntheInstallLocaUonscreenleavethedefaultsofOracleBase:/u01/app/oracle;Name:whatevernameyoulike;Path:/u01/app/oracle/product/11.1.0/asm_1 (for ASM) or/u01/app/oracle/product/11.1.0/db_1 (for RDBMS)

OntheSpecifyHardwareClusterInstallaUonModescreen,chooseClusterInstallaUonandclick“SelectAll”tochooseallnodes.

Allprerequisitechecksshouldbesuccessful.

OntheSelectConfiguraUonOpUonscreen,chooseinstallsoEwareonly.

OntheOracleConfiguraUonManagerRegistraUonscreen,clicknexttobypassregistraUon.

ClickInstallonthesummaryscreentostarttheinstallaUon.Gogetsomecaffeineandwait.

OncetheinstallaUonfinishes,you’llbepromptedtoruntheroot.shscriptoneachnode.RunthemsequenUally(eithernodefirst).Takethedefaultforthelocalbindirectoryaspromptedbytheroot.shscript.

Atthisstage,theASMinstancesshouldberunningoneachnode(youshouldseelotsofasm_XXXX_+ASM? processesrunningoneachnode.The“crs_stat –t”commandshouldalsoshowASMandlistenerresourcesconfiguredforeachnode.

INSTALLPATCHES

NowthatyouhavetheORACLE_HOMEsinstalled,youneedtoapplypatches.ThebasicpatchsetsforeachoftheORACLE_HOMEs(Clusterware,ASM,andRDBMS)arethesamepatchset(in10gR2and11gR1,anyway)‐‐theRDBMSpatchset.FollowthepatchREADMEinstrucUonstoinstallthepatchoneachoftheORACLE_HOMEsinthisorder:Clusterware,ASM,RDBMS.

YoushouldalsoconsiderinstallingthelatestCriUcalPatchUpdate(seehVp://www.oracle.com/technology/deploy/security/alerts.htm)and,ifusing10gR2,alsoconsiderinstallingthelatestClusterwarepatchbundle(seeML405820.1).ItisrecommendedtoinstallthelatestCriUcalPatchUpdatepatchesastheypatchseveresecuritybugsthatmayleaveyourenvironmentvulnerableifnotpatched.TheCPUpatcheshavetheirowninstallaUoninstrucUonsandproceduresthatshouldbefollowedcarefully.

CONFIGUREASM

Beforeyoucancreateadatabaseisthisnewlypatchedenvironment,youfirstneedtoconfigurethestorageyou'lluseforthedatabase.ToconfigureASM,usetheDatabaseConfiguraUonAssistant.

StartVNC,justasbefore.

ConnecttoVNCandsetyourenvironmenttotheASMORACLE_HOMEintheterminalwindowinsidetheVNCsession.Withitsetproperly,starttheDatabaseConfiguraUonAssistant(dbca&)attheprompt.

Onthefirstscreen,chooseConfigureAutomaUcStorageManagement.

OntheConfigureAutomaUcStorageManagementscreen,youmayfirsthaveto“ChangeDiskDiscoveryPath”to“/dev/sd*”(orwhatevertheappropriatepaVerntoidenUfythemulUpathingpseudodevices)inordertoseeyourcandidatedrives.Then,createoneExternalRedundancydiskgroupnamedDATAcheckingtheboxesnexttothedrivesthatshouldbeinthatgroup.BesurethatthedevicesyouseelistedareonlythemulUpathingdevicesandnoneofthedirect,non‐mulUpatheddevices.

Usingthisinterface,createthedesireddiskgroupconfiguraUonwhichyouhavealreadyplannedout.

Paper48816

DBA,Clustering

CONFIGURENEWDATABASE

NextonthetasklististoconfigureaRACdatabase.We’llusethequickandeasyDBCAtocreatethenewdatabase.Forthis,we’llusetheVNCconnecUononceagainsinceDBCAisaGUItool.ConnecttoaVNCsessionononeofthetwoserversandfollowthestepsheretocreateadatabaseinstance.

Setyourenvironmentvariablesasshownhere:export ORACLE_HOME=/u01/app/oracle/product/11.1.0/db_1export PATH=$ORACLE_HOME/bin:$PATH

ThenruntheDBCAtoolbyentering:dbca &

Ontheopeningscreen,choosedatabasetype“OracleRealApplicaUonClustersdatabase”

ChoosetheoperaUon“CreateaDatabase”

OnthenodeselecUonscreen,chooseSelectAll

Onthetemplatesscreen(step3of19),chooseGeneralPurposeorTransacUonProcessing

FortheGlobalDatabaseNameenter“RAC.world”andlettheSIDPrefixdefaultaccordingly(defaultshouldbeRAC)

OntheconfiguraUonassistantscreen(step5of18),choosethedefaultof“ConfiguraUonEnterpriseManager”andyoudonotneedtoenablealertnoUficaUonsnordailydiskbackups

Forsimplicity(insteadofsecurity),choose“UsetheSameAdministraUvePasswordforAllAccounts”onstep6of18andenterapassword.

OnthestorageselecUonscreen,chooseAutomaUcStorageManagement(ASM)

SelecttheDATAdiskgroupandclicknextonthediskgroupselecUonscreen.

UseOracle‐ManagedFilesontheDatabaseFileLocaUonsscreen(step9of17)

NoneedtoconfigureaflashrecoveryareaorarchivingatthisUme

Youmayinstallthesampleschemasifyouwouldlike.

IchosetousethedefaultandsuggestedtypicalmemoryconfiguraUon.Youcanusewhateveryou’dlike,butensureaminimumSGAsizeof400Mb.

OntheCharacterSettab,IprefertouseUnicode.

Keeptheenhanced11gsecurityseWngsonstep13of16

ItisrecommendedtoleavetheautomaUcmaintenancetasksenabled

TherearenorequiredchangesontheDatabaseStoragescreen

Onthefinalscreen,clickFinish,thenclickOKonthefinalsummary.Wait.

OncetheDBCAiscomplete,itwilldisplaythesummaryconfirmaUonscreen.MakeanoteoftheURLfortheDBControlwebsite.

WhenyouclickExitonthesummaryscreen,theinstancesarestarted.Ittakesawhilebeforeanymessagesaredisplayed,sobepaUent.

Alldone!RACdatabaseiscreated,servicescreatedandeverythingshouldberunning.

CHALLENGESANDCOMMONMISTAKES

FIREWALLSETTINGSINGUEST(LINUX)OS

IfyoufailtodisabletheLinuxfirewallorleaveSELinuxinanyseWngotherthandisabled,youwilllikelyencountersecurity‐relatedissuesandwarnings.TheclusterwareinstallaUonmayhaveerrorsduetonetworkportsnotbeingavailableoropenforcommunicaUon.

Paper48817

DBA,Clustering

SHAREDSTORAGEMISCONFIGURATION

OneofthecommonsetupissuesisconfiguraUonofsharedstorage.OnLinuxservers,devicesarenotpersistentlyboundtothesamedevicename.Thatis,Linuxmaydecidetomake/dev/sdb into/dev/sdc onsubsequentreboots,basedonanumberoffactors.Asthisisaknownissue,thereisalsoaknown,stable,acceptedwaytoaddresstheissue.WithRACclusters,thisisespeciallydaunUng,sincethedevicenotonlyneedstobethesamenameonthatnodeeachUme,butitneedstobethesamenameonallnodesinthecluster.

ThesoluUonusedformostrecentLinuxversionsistouseasubsystemcalled“udev.”Theudevsubsystemcreatesalayerofdevicenamesontopoftherealdevices.ThenamesusedbyudevarecontrolledbyrulesandallmappingsaredoneusingdeviceidenUfiersuniquetoeachdeviceinordertobecertainthattherightdevicesareidenUfiedfromeachnode.ThisconfiguraUondocumentdidn’tuseudev,butinsteadsetprivilegesdirectlyusingcommandsin/etc/rc.local(notanappropriateconfiguraUonforaproducUonenvironment).

Ifyoudonotconfigureudevordevicemapper,theclusterverificaUonuUlitymayreportsuccess,butaEerrebooUngthenodes(especiallywhenaddingorreconfiguringstorage),youmayfindthattheclusternolongercomesupbecausethedevicenamesallchangedduringtherebootcycle.

Tolearnmoreabouthowtoconfigureudev,seethefollowingnotesonMetalink:

443996.1–HowtomaprawdeviceonRHEL5andOEL5

465001.1–Configuringrawdevices(singlepath)forOracleClusterware10gRelease2(10.2.0)onRHEL5/OEL5

564580.1–Configuringrawdevices(mulUpath)forOracleClusterware10gRelease2(10.2.0)onRHEL5/OEL5

605828.1–ConfiguringmulUpathdevicesforOracleClusterware11g(11.1.0)onRHEL5/OEL5

414897.1–Howtosetup/etc/udev/rules.dforOCRandVoUngforRAConSLES10/RHEL5andEL5

371814.1–UsingudevwithOracleArchitecture(RAC&ASM)–RedHat4.0

456239.1–UnderstandingDevice‐mapperinLinux2.6Kernel

357472.1–Configuringdevice‐mapperforCRS/ASM

465001.1–ConfiguringrawdevicesforOracle10gR2(10.2.0)ClusterwareonRedHat/EnterpriseLinux5(RH/EL5)

NETWORKINGSETUPISSUES

ItissomeUmesdifficulttoconfigurethemulUplenetworksrequiredforaRACdeployment.Usuallythat’sbecauseit’shardtoknowwhichphysicalportontheservercorrespondstoeth0,eth1,etc.Toaddressthis,IfinditbesttoconfigureonenetworkinterfaceataUmeinsteadofconnecUngthemallandthentryingtosortthingsout.Ifnecessary,evengoingasfarasphysicallydisconnecUngthewirecanhelp.That’sbecausemostEthernetdriverswillreportwhenthelinkisupordownontheconsolebuffer(whichyoucanseebyrunning“dmesg”onaserver).BydisconnecUngandreconnecUngawire,youshouldbeabletoseemessagesindicaUngwhichphysicalinterfacelostitslinkandthatshouldtellyouwhichphysicalportcorrespondstothatphysicalinterface.

It’salsowisetoincludetheHWADDRinthe/etc/sysconfig/network-scripts/ifcfg-ethXfiletoensurethatLinuxdoesn’trenumbertheinterfaces(whichcanhappenwhenhardwareconfiguraUonschange).Bylockingthatdevicetoaphysicalhardwareaddress(MACaddress),youensurethatitwillalwaysremain“Ued”together.

ToestablishredundancyfornetworkconnecUons,youcanfindmoreinformaUonaboutusingandconfiguringnetworkbondingonLinuxfromthesearUclesonMetalink:

456055.1–HowtocreatetwodifferentbondingconfiguraUonsinOEL4

298891.1–ConfiguringLinuxfortheOracle10gVIPorprivateinterconnectusingbondingdriver

434375.1–LinuxEthernetbondingdriver

Paper48818

DBA,Clustering

LINUXINSTALLATIONCHALLENGES

Ifyouhaven’tinstalledLinuxbefore,youmayfearthatitwillbecomplexordaunUng.ManypeoplefindthatLinuxinstallaUonisnotnearlyasdifficultastheythoughtitmayhavebeen.

ThemostcommonissueswithLinuxinstallaUonusuallyinvolvediskparUUoning.SometrytosizetheparUUonstoosmallsoastomakethebestuseofspaceandotherstrytocreateseparateparUUonsformanydifferentareasofthefilesystemtree.Forthe“normal”installaUonsIperform,IusuallyhaveparUUonsfor/(root),swap,/tmpand/u01(forOraclesoEware).Withdisksizesbeingquitelargethesedays,Iusuallysizethingsthisway:

Purpose Size Notes

/ 15Gb

swap >=2GbMoreifRAMsizeislarger,buttypicallynolargerthan8Gb

/tmp 2GbIfthereisalotofspaceavailable,Imaymakethis4Gb,butnolargerthanthat.

/u01 therestEverythingelseisallocatedhere—minimumof20Gb,usuallymuchlarger.

LINUXUSEREQUIVALENCY ConfiguringuserequivalencyonLinux(oranyUNIX‐basedplaTorm)istheprocessofseWnguptheuseraccountsontwoormorehostssothattheywon’trequireapasswordtologinand/orruncommandsoneachother’shosts.Formany,thisconfiguraUonischallengingandtheydon’tunderstandthereasonswhytheirconfiguraUondoesn’tworksincetheydon’tknowhowitreallyworks.

TherearequiteafewreferencesonlinetofindhelpwiththisconfiguraUonandgenerally,ifyoufollowtherightsteps,evenifyoudon’treallyunderstandhowitworks,you’llendupwithaworkingconfiguraUon.Thatsaid,it’salwaysbeVertounderstandhowitworksandespeciallysointhiscasesinceithastodowithsystemsecurityandmessinguptheserverconfiguraUonmayresultinalesssecuresystem.Forourpurposesinthisdocument(geWngaworkingRACconfiguraUon),followingthestepsforuserequivalencyshownhereinissufficientanddoesnotcompromisesystemsecurity.

hVp://www.ardentperf.com/2007/03/27/dbca‐slow‐check‐host‐equivalence/

372795.1–HowtoconfigureSSHforuserequivalence

300548.1–HowtoconfigureSSHforaRACinstallaUon

REFERENCES TimHall’sarUcleoninstalling11gRAConVMWareusingOracleEnterpriseLinux5athVp://www.oracle‐

base.com/arUcles/11g/OracleDB11gR1RACInstallaUonOnOEL5UsingVMware.php

OracleMetalinknotes(specificnotescitedelsewhereinthisdocument)

JeremySchneider,“UnleasingOracleServices:AComprehensiveReviewof“Services”inOracleDatabases”,hVp://www.ardentperf.com/pub/services‐schneider07.pdf

Metalink567730.1–ChangesinOracleClusterwareonLinuxwiththe10.2.0.4Patchset

Metalink284752.1–10gRAC:StepsToIncreaseCSSMisscount,REbooWmeandDiskUmeout

Metalink294430.1–CSSTimeoutComputaUoninOracleClusterware

Metalink395878.1–Heartbeat/VoUng/QuorumRelatedTimeoutConfiguraUonforLinux,OCFS2,RACStacktoavoidunnecessarynodefencing,panic,andreboot

Paper48819

DBA,Clustering

FROMTHELAWYERS TheinformaUoncontainedhereinshouldbedeemedreliablebutnotguaranteed.TheauthorhasmadeeveryaVempttoprovidecurrentandaccurateinformaUon.IfyouhaveanycommentsorsuggesUons,pleasecontacttheauthoratdannorris(at)dannorris.com.

Paper48820