Download - 2009_488_norris_ppr-20090316
DBA,Clustering
IINSTALLINGNSTALLINGRACFRACFROMROMTTHEHEGGROUNDROUNDUUPP
DanNorris,dannorris(at)dannorris.com,DanNorris.com
INTRODUCTIONANDTERMINOLOGY ThispaperoffersarecipeforcreaUngaRACenvironment.TheinstallaUonguidescontainmostoftheinformaUononinstallaUon,andafewinternetresources(citedinthereferencessecUon)providethenecessaryextrastepsyouneedtocompletethesetup.
Tosupportthediscussionthatfollows,thesedefiniUonsmaybehelpful:
Term DefiniKon
Database thecontrolfiles,datafiles,andonlineredologs
Instance thesharedmemoryandbackgroundprocessesthatoperateonadatabase
ClusterwaresoEwarethatmanagesclustermembershipandmonitorsthenodesandnetworksinacluster
StorageAreaNetwork(SAN)
astorageenvironmentwheremulUpleserverscanuUlizeasinglestoragearray;astoragenetworkiscommonlyimplementedusingfiberchanneltechnology
LocalStoragediskspacethatisavailabletoexactlyonenodeinacluster;thisstoragemaybepartofaSANormaybedirect‐aVachedtotheserver
SharedStoragediskspacethatisavailabletomorethanonenodeinaclusteratthesameUme;thisstorageiscommonlypartofaSAN
RawDevice thecharacter(unbuffered)specialdevicepresentedbytheoperaUngsystem
ClusterFilesystemaspecialtypeoffilesystemthatcanbeaccessedbymulUpleclusternodesatthesameUme
OracleAutomaUcStorageManagement(ASM)
soEwarethatwillmanagedisksdirectlyandprovidevolumemanagementfuncUonslikestripingandmirroring;thisfeaturewasnewwithOracleDatabase10g
Single‐instanceDatabase
thetradiUonal"standalone"databaseconfiguraUonusedbymostOracleenvironmentstoday
MulU‐instanceDatabase
asingledatabaseservicedbymorethanoneinstance
InthesecUonsthatfollow,youwilllearn:
AnarchitecturaloverviewofRAC
SystemPrerequisitesandcheckingthemthoroughly
ConfiguraUonandinstallaUonprocesses
Challengesandcommonmistakes
ThispaperisintendedtobeusedbyOracleDBAsthathaveatleastaliVlebitofLinuxknowledgeandhaveinstalledsomeversionofLinuxatleastonce.RACexperienceisalsohelpful,butmostDBAswillprobablybeabletocompletetheinstallaUonwithouthavingdoneitbefore.Thisdocumentisjustarecipeonhowtoinstalltheproducts,nothow
Paper4881
DBA,Clustering
tousethem.Ifyou’renewtoRACandneedamorebasicintroducUontoRAC,checkoutthe“RACForBeginners:TheBasics”whitepaperandpresentaUonfoundonlineathVp://www.dannorris.com/professional‐acUviUes/underthePresentaUonssecUon.
ORACLERACARCHITECTUREOVERVIEW TheobviousdifferencesbetweenRACandatradiUonalsingle‐instancedatabaseareapparentonseveraldifferentlevels.ThesecUonsthatfollowdescribethesedifferencesinmoredetail.
SYSTEMARCHITECTURE
ThediagramaboveshowsasimpleRACclusterarchitecture.AminimalclusterconsistsofmulUpleservers,aprivatenetwork,apublicnetwork,andsharedstorage.MostDBAsarefamiliarwiththesingle‐instanceconfiguraUoninwhichstorageisnotsharedbetweennodesandallaccesstothedatabasecomesfromthesingleinstance.
Sharingthesamestorageposesseveralchallenges.First,theoperaUngsystemmustbeabletohandleaccessesfromothernodesatthesameUmeasitaccessesthestorage.IntheearlydaysofOracleParallelServer(OPS),thiswasaddressedbyusingrawdevices.TheoperaUngsystemswereaccustomedtofilesystemsthatsupportedaccessfromexactlyonehost.Whenonlyonehostwasinvolved,thehostcouldmanagealllockingandsynchronizaUonfromasingleoperaUngsystemkernel.WithmulUplehosts,thefilesystemaccessmustbecoordinatedbetweenallhostssharingthefilesystem.Thisintroducessomelatencyand,unUlfairlyrecently,cluster‐widefilesystemswereveryuncommon.Today,severalvendorsofferclusterfilesystemsthataresupportedbyOracleRAC.OraclealsooffersitsOCFSfilesystemforLinuxandWindowswithreportedplanstomakeitavailableformoreplaTormsinthefuture.
Sharingadatabasealsocreateschallenges.Beingabletohandlethecross‐hostlockingrequiredtoensuredatabaseintegrityisthefirstandmostimportantchallenge.Previously,thatchallengewasaddressedbyexchangingblocksthatrequiredserializaUonviadisk.ThiswasknownasablockpingandwasamajorlimiUngfactorforOPS.WithCacheFusiontechnologyintroducedinOracle9i,theblockpingwasputontheendangeredspecieslistandwasonlyrequiredforasmall,infrequentlyseensetofcircumstances.
TheaddiUonalhardwarerequiredforaRACclusterusuallyconsistsofafewaddiUonalnetworkstosupporttheprivateinterconnect.TheprivateinterconnectcanbemadeupofmulUplephysicalnetworkinterfaces.ItiscommontouUlizetwoormoregigabitEthernetinterfacesfortheinterconnect.Inpastversions,expensiveandproprietaryinterconnecttechnologieswereused.TheseproprietaryinterconnectswererequiredbecausegigabitEthernetcreatedtoomuchlatencyanddidnothavethebandwidthnecessarytosupportsomeenvironments.WithadvancementsinEthernettechnologyandtheabilitytouUlizemulUpleinterconnects,gigabitEthernetisthemostcommoninterconnectinusetoday.InfinibandisarelaUvelynewtechnologythatwasrecently(10.2.0.3)cerUfiedforuseasanOracleprivateinterconnect.InfinibandpromisestobeasignificantplayerintheRACenvironmentbecauseofitsimpressivetechnicalspecificaUons.
Finally,beforewemoveon,let’sreviewsomepointsaboutdatabasesandinstances.OracleRACis,bydefiniUon,amulU‐instancedatabase.Thereisonedatabase.That’so‐n‐e,asin1,uno.That(one)databaseresidesonsharedstorageandisaccessedconcurrently(that’satthesameUme)bymulUpleinstancesthatrunonseparatehosts.Readthatsentence
Paper488
Shared Storage
Local Storage
Local Storage
RAC1 RAC2
2
DBA,Clustering
again—itiscriUcalthatyouunderstandwhatyou’rebuildingandthatlastsentencesumsitallup.Databaseandinstancearedefinedinthetableabove.It’sagoodideatoreviewtheirmeaningssinceoEenUmesin“normal”Oracleenvironments,thetermsdatabaseandinstancetendtobeusedinterchangeably.That’sfine,butitwillbenefityoutotrytousethosewordsproperlyintheRACenvironmentcontextwhileyou’relearninghowtheenvironmentworks.
AswemoveontotheclusterimplementaUonprocesses,keepinmindtheterminologyand“bigpicture”ofwhatwe’retryingtobuild.Forme,italwayshelpstorefertotheoverallarchitecturedesignwe’retryingtocreatefrequentlyduringcomplexinstallaUons.
TheimplementaUontopicscoveredintheremainderofthisdocumentare:
Systemrequirements
Prerequisitechecking
ConfiguraUonandinstallaUonprocesses
Challengesandcommonmistakes
SYSTEMREQUIREMENTSFORRAC ThefollowingsecUonsdescribesomeoftheconfiguraUon.
SUGGESTEDMINIMUMSFORCPU,RAM,DISK
IfyoureallyhavepaUenceandcanwaitforextendedperiodsfortheinstallaUonstocomplete,youcangenerallycompletetheinstallaUonsandsuccessfullybuildaclusterwithverymodesthardware.However,ifyouputanystressonsuchasystem,itislikelytoproduceundesirableresultsassomeCPUisrequiredtomaintainclusterheartbeatsandoverallclusterhealth.
Resource DocumentedMinimums SuggestedMinimums
CPU nonedocumented 2Ghz(forx86)
Memory 1Gb 2Gb
DiskSpace 3.5Gb(binaries) atleast5Gb
SwapSpace1.5xRAMifunder2Gb;sameasRAMif2‐8Gb;0.75xRAMif>8Gb
followdocumentedminimums
PLANNING ThemostimportantpartofanyRACimplementaUonistheplanningphase.PlanningisrequiredforstorageconfiguraUon,networkconfiguraUon,databaseconfiguraUon,services,andORACLE_HOMElocaUon.EachoftheseareashasseveralopUons,eachwithitsownadvantagesanddisadvantages.CarefulplanningwillmakeforasmoothimplementaUon.
STORAGE
Storageplanninginvolvestwomaincomponents:accesspathplanningandLUNconfiguraUonplanning.
Whenplanningaccesspaths,you'llneedtoconsiderifyou'reaccessingthestorageviafibrechannel,iSCSIorNFS.Ineachcase,considerthatthestorageaccesspathmaybeasinglepointoffailure(SPOF)ifitisn'tconfiguredwithredundancy.Foreachstorageaccesspathtype,themethodformakingthepathredundantmaybedifferent.SinceNFSusesTCP/IP,redundantNICsusingNICbonding(onLinux,otherplaTormsmaynotcallit“bonding,”buthavesimilarmethods)maybesufficient.Withfibrechannel,you'lllikelyneedmulUpathingsoEwaredrivers.OnLinux5,therearemulUpathdriversbuilt‐in.OnalmostallplaTorms,enterprisestoragevendorsprovidetheirownmulUpathsoEware(forexample,EMCPowerPathandHPSecurePath).
Paper4883
DBA,Clustering
Ofcourse,themoreobviousstorageplanningtaskisdeterminingwhatandhowstorageshouldbepresentedtothehostOSes.WhiletherearemanygoodsourcesforinformaUontoday,mostpeoplefollowOracle'sStripeAndMirrorEverything(SAME)methodologywhenconsideringtheirstorageconfiguraUon.Withlargestoragearrays,itiscommontoseeasmallnumberoflargeRAIDsetsthatare“carved”upintoindividualLUNs.Inthiscase,itdoesn'tusuallyprovideanybenefittopresentmanysmallLUNsthatallresideonthesameRAIDset.Plus,withmoreLUNs,thereismoremanagement.So,myrecommendaUonistouseasmallnumberofLUNs(usually1or2)incaseswheretheLUNsallresideonthesameRAIDsetanyway.TheonlyexcepUontothisiswhentherearemanydatabasesonthecluster.Inthatcase,IprefertohaveamaximumofonedatabaseperLUNsothatwecanmeasureI/OattheOS(andpossiblyatthearraytoo)separatelyforeachdatabase.ThisisimportantwhentryingtodeterminewhichdatabaseiscausingapotenUalperformanceissue.
NETWORK
OneoftheconfiguraUonareasIoEenseeneglectedisnetworkredundancy.Whileitistruethatnetworkinterfacecards(NICs)don'tfailveryoEen(certainlynotasoEenasdisks),theycanfailandtheyusuallydo—attheworstpossibleUme.MyexperiencehasbeenthatNICsusuallydon'tfailallatonceeither,makingtheirgradualdegradaUondifficulttodetectandtroubleshoot.
AllNICsneedtobeconfiguredwitharedundantbackup.Insomecases,youmaybeabletodobeVerthananacUve/passiveconfiguraUon(withonlyoneNICofeachpairacUve)andconfigureacUve/acUve(alsosomeUmescalled“trunking”)NICswherethebandwidthofallNICsareaggregated.Ineithercase,whenaNICfailureisdetected,thesurvivingNICinthepairshouldtakeoverimmediatelyandseamlessly.
SomeUmes,usuallyduetohardwarelimitaUonsinbladechassis,siteschoosetoaccepttheriskofbasicallyhavingacompletenodefailureifaNIConthatnodefails.
ItisbesttoconfiguretheNICswithredundancy(bondingortrunking)beforeyoustarttheOracleClusterwareinstallaUon.
TheotherpartofnetworkplanningischoosingtheproperIPaddressesthatwillbeused.Foreachnodeinthecluster,you'llneedatleastthreeIPaddresses:public(IsomeUmescallthisthe“base”IPaddressofthenode),private(interconnect),andvirtual(VIP,fromthesamenetworkasthepublicnetwork).
Inthispaper,wewillshowhowtobondnetworkinterfacesusingbuilt‐infuncUonalityonLinux.TrunkingusuallyrequiresspecialsoEwareandhardwareconfiguraUonandisbeyondthescopeofthispaper.
DATABASE
PlanningthedatabaseisprobablyoneoftheeasiestpartsformostDBAssinceitisaboutthesameasplanningnon‐RACdatabaseimplementaUons.Planningforthedatabaseusuallyjustinvolvesplanningtablespacelayouts,opUonalcomponentstoinstallduringcreaUon,blocksizeandcharactersetchoices,andchoosinginstanceanddatabasenames.
TheonlydifferencefordatabaseplanningforRACcomparedtosingle‐instanceisthatnamingwillinvolvemakinganamingprefixthatwillbeusedforallinstances.Forexample,ifyouchooseaprefixof“dan”,thentheinstanceswillbenamed“dan1”,“dan2”,“dan3”,andsoon.Stayingwiththisnamingformatofprefix+numberisthedefaultexpectedbythedatabaseconfiguraUonassistant(dbca)andisalsothemostcommondeployment.However,technically,youcoulduse“fred”and“barney”fortwoinstancenamesofthesamedatabase—itjustendsupbeingmoreconfusingthanthenormalnamingmethod.Typically,usinga“scheme”(likecartooncharacters)doesn'tscalewellandifyoudoendupwithalargerclusterthanmembersofthegroupyouused,thenyouhavetostartmakinguprandomnames,therebybreakingfromthescheme.
SERVICES
OracleServicesareanimportantpartofanyRACclusterastheyprovidethebestmethodformanagingthecluster'sworkload.IfyouarenotyetfamiliarwithServices,youshouldgeteducated.AnexcellentstarUngpointismyfriend
Paper4884
DBA,Clustering
JeremySchneider'swhitepaperonthetopic:“UnleasingOracleServices:AComprehensiveReviewof'Services'inOracleDatabases”availableathVp://www.ardentperf.com/pub/services‐schneider07.pdf
Asyouknow,Servicescanbecreatedtohandleanytypeofgrouping.SomesitesprefertomakeservicesforindividualapplicaUons(ifthedatabasehasmulUpleapplicaUonsthataccessit),typesofusers(HR,Payroll,Purchasing,etc.),applicaUontype(batchprocessing,OLTP,BI/DW),orotherspecialapplicaUons.IfyouusetheDatabaseResourceManager,youcanalsoaVacharesourceprofiletoaservice.IfyouuseOracle10gorhigher,eachinstancealsotracksstaUsUcsonaper‐servicebasis(v$service_stats,v$service_wait_class,etc.)thatcanhelpyoudeterminewhichserviceisconsumingthelargestshareofresourcesonthesystem.
YourdeploymentofServicesshouldbesomethingyouplanaheadofUmeanddiscusswiththebusiness.AstheDBA,oneofyourrolesistotakeadvantageoffeaturesofferedbythedatabaseandmapthemtobusinessneeds.Forexample,ifyouanUcipatethatthebusinessmaywanttoknowwhetheritwouldhelptomovetheHRuserstoanotherdatabase(ormaybejusttotheirowninstance),thenyoushouldconsidercreaUngaservicefortheHRuserstoprovidethatdatalater.Servicesprovideworkloadmanagementbygivingyouaneasywaytoisolateaservicetoaspecificinstanceorsub‐groupofinstancesinthecluster.Theworkloadmanagementfeaturesarebeingenhancedwitheachrelease,soitislikelythatmorenewfeatureswillbenefitthosethathavealreadyimplementedandusedServiceswisely.
SHAREDVS.LOCALORACLE_HOMES
Thistopicisincludedinyourplanningasanawarenessthatitispossible,withsomeconfiguraUons,tocreateasingle,sharedORACLE_HOMEforyourcluster'ssoEware.You'llsUllneedseparateORACLE_HOMEsforOracleClusterwareandOracleRDBMSbinaries(andsuggestaseparateoneforOracleASMbinariestoo,ifyouuseASM).Ifyouhaveasupportedclusterfilesystem,youcansharethebinariesamongallnodesandavoidhavingmulUpleORACLE_HOMEstomaintain.ThereisanexcellentwhitepaperprovidedbyOracleonthistopic(seeitathVp://www.oracle.com/technology/products/database/clustering/pdf/oh_rac.pdf).TheshortsummaryofthatwhitepaperisthatlocalORACLE_HOMEsaresuggestedforsmallclusters(I'dsaylessthan5nodes).Ifyouhavelargerclusters,IwouldconsidermulUplesharedORACLE_HOMEs,butIwouldneverrecommendasingle,cluster‐wideORACLE_HOMEasitcreatesapotenUalsinglepointoffailure.IunderstandtheargumentthatmaintainingmulUpleORACLE_HOMEsismorework,buttheinstallersimplifiesmuchofthemaintenanceprocessandyoualsohavetheeasyabilitytoperformrollingupgradesandpatches(forthosepatchesthatsupportrollingupgrades).
MyrecommendaUonistouselocalORACLE_HOMEsonlocalstorageoneachnodeofyourcluster.
PREPAREOPERATINGSYSTEM ThisprocessismostlyderivedfromagreatarUclewriVenbyTimHallatOracle‐Base.ThearUclecanbefoundonlineathVp://www.oracle‐base.com/arUcles/11g/OracleDB11gR1RACInstallaUonOnOEL5UsingVMware.php(nottheeasiestURLtoremember).YoucanalsofinditbygoingtohVp://www.oracle‐base.com/andusingthe“OracleRACArUcles”linkintheleE‐handmenu.ThankstoTimforgraciouslyallowingmetousetheinformaUoninthatarUcleforthispaperaswell.Onelastnote—TimalsowroteandhostsarUclesonbuildingthesamesetupusingthepossiblecombinaUonsof10g,11g,Linux,andWindows.
IwillforegothemyriadscreenshotsofeachinstallaUonstepasTim’sarUclecoversthatnicely.Instead,I’lldescribeeachstepandtrytokeepyouoncoursetocompletetheinstallaUontoendupwithasuccessfulcluster.Youdon’tneedanythingotherthanthisdocumenttoguideyou,butreadingTim’sarUclemaybehelpful.
INSTALLOS
WiththeCDinplace,bootthesoon‐to‐benode1.Oncebooted,followthesestepstoinstalltheOS:
Installasyouwouldnormallyinstallanyserver.FordetailedinstrucUonsonserverinstallaUon,TimHallhasanotherfinearUclepostedathVp://www.oracle‐base.com/arUcles/linux/OracleEnterpriseLinux5InstallaUon.phpthatwalksthroughtheinstallaUonstep‐by‐step.
Whenprompted,choosethefollowingpackagegroups:
Paper4885
DBA,Clustering
GNOMEDesktopEnvironment
Editors
GraphicalInternet
Text‐basedInternet
DevelopmentLibraries
DevelopmentTools
ServerConfiguraUonTools
AdministraUonTools
BaseSystem
SystemTools
XWindowSystem
UnderBaseSystem‐>SystemTools‐>OpUonalpackages,makesuretoselect“oracle‐validated”.YoucanopUonallychoosetoinstalloracleasmpackagesandocfs2packagesfromthatsamelistofopUonalpackages.
InmysampleinstallaUon,IusedthefollowinginformaUonforhostidenUficaUon:
hostname:node1.dannorris.net
IPAddresseth0:192.168.78.51(publicaddress)
DefaultGatewayeth0:192.168.78.1(publicaddress)
IPAddresseth1:192.168.150.51(privateaddress)
DefaultGatewayeth1:none
Disablethebuilt‐infirewallandalsodisabletheSELinuxsubsystem.Disablingthesecomponentswillmakeiteasiertoconfigureandusethesystem.
Installnode2inasimilarway(makeexactlythesamechoicesduringinstallaUon)andentertheappropriatehostnameandIPaddressesforit.
WiththeinstallaUoncompleteandpackagesinstalled,wecanmoveontoconfiguringtheOracleprerequisites.
POST‐INSTALLATIONCONFIGURATION
TheLinuxprerequisitesforinstallingOracleRACareallthesameonesforinstallingasingleinstanceenvironment,plusafewaddiUonalrequirements.Thefollowingstepsarethe“cheatsheet”forconfiguringthenecessarypartsoftheOStopreparefortheOracleinstallaUons.
Notethatifyouchosetheoracle-validatedpackageduringtheinstallaUonprocess,mostofthesechangesshouldalreadybemadeforyou.So,firstconfirmthatthechangesaren'talreadymadebeforeaddingtoormodifyingthefilesmenUoned.Thesechecksmustbedoneonbothnodes.
Whilenotanofficialprerequisite,Iusuallychangethedefaultrunlevelfrom5to3inordertodisableXwindowsontheconsole.ThissavessomeRAMontheserverandXwindowsisrarelyneededanyway.Tochangethis,modifytheinitdefaultseWnginthe/etc/inittabfilefrom5to3.Then,asroot,run“init q; telinit 3”tomakethechangeeffecUveimmediately.IfyoupreferaGUIconsole,don’tmakethischange—itisopUonal,butsinceitfreesupmemory,performancemayimprovebyfollowingthisrecommendaUon.
Modifythe/etc/hostsfiletolookliketheonebelow(replacethewholefilewiththesecontents):127.0.0.1 localhost.localdomain localhost192.168.78.120 node1.dannorris.net node1192.168.78.121 node2.dannorris.net node2172.16.100.120 node1-priv.dannorris.net node1-priv172.16.100.121 node2-priv.dannorris.net node2-priv
Paper4886
DBA,Clustering
192.168.78.130 node1-vip.dannorris.net node1-vip192.168.78.131 node2-vip.dannorris.net node2-vip
Addthefollowinglinestothe/etc/sysctl.conffile(NotethatonOEL5,manyoftheparametersthatyouhadtoaddinpreviousversionsarenowinthefilebydefault):kernel.shmmni = 4096# semaphores: semmsl, semmns, semopm, semmnikernel.sem = 250 32000 100 128net.ipv4.ip_local_port_range = 1024 65000net.core.rmem_default=4194304net.core.rmem_max=4194304net.core.wmem_default=262144net.core.wmem_max=262144
Oncethe/etc/sysctl.conffileismodified,forcethenewparametersintoeffectimmediatelywith“sysctl –p”
Addthefollowinglinesto/etc/security/limits.conf:@dba soft nproc 2047@dba hard nproc 16384@dba soft nofile 1024@dba hard nofile 65536
Addthefollowinglinestothe/etc/pam.d/loginfile:session required /lib/security/pam_limits.sosession required pam_limits.so
Addthenecessarygroupsandusertothesystemwiththefollowingcommands:groupadd -g 500 oinstallgroupadd -g 501 dbagroupadd -g 502 asmadminuseradd -u 500 -g oinstall -G dba,asmadmin -s /bin/bash oraclepasswd oraclemkdir -p /u01/app/oracle /u01/crs/oraclechown -R oracle:dba /u01
Setdefaultshellvariablesbyaddingthefollowingtothe/home/oracle/.bash_profilefile:ORACLE_BASE=/u01/app/oracle; export ORACLE_BASEPATH=/usr/sbin:$PATH; export PATH
if [ $USER = "oracle" ]; then if [ $SHELL = "/bin/ksh" ]; then ulimit -p 16384 ulimit -n 65536 else ulimit -u 16384 -n 65536 fifi
Configurethehangcheck‐Umerkernelmodulebyaddingthefollowinglineto/etc/rc.d/rc.local:modprobe hangcheck-timer hangcheck_tick=1 hangcheck_margin=10 hangcheck_reboot=1
SeeMetalinkNote726833.1formoreinformaUononhangcheck‐UmerandrequirementsforRAC.
CONFIGURENICBONDING
ThefirststeptoconfiguringNICbondingisseWngupthekernelparametersforthebondinginterfaces.Theseparameterswillcreatethepseudodevicesforthebondedinterfaces.ThehardestpartofpairingtheNICsisfiguringoutwhichtwoshouldbepaired.Usually,theeasiestwaytodeterminewhichonesareonwhichnetworkistousetheMACaddresses.YournetworkandsystemadministratorsshouldbeabletoassistwiththedeterminaUons.
Inourcase,we'llassumethatwehave4NICs(2forpublic,2forprivate).We'regoingtopaireth0andeth2forthepublicnetwork(bond0)andeth1andeth3fortheprivatenetwork(bond1).Tocreatethebonds,firstchangethe/etc/modprobe.conffiletoaddtheselines(don'trebootnow,we'lldothatlater):
alias bond0 bondingalias bond1 bonding
Paper4887
DBA,Clustering
options bond0 mode=active-backup miimon=100 max-bonds=2options bond1 mode=active-backup miimon=100 max-bonds=2
Next,createanewfileonnode1named/etc/sysconfig/network‐scripts/ifcfg‐bond0withthefollowingcontents:DEVICE=bond0 IPADDR=192.168.78.120 NETMASK=255.255.255.0 ONBOOT=yes BOOTPROTO=none USERCTL=no MII_NOT_SUPPORTED=yes
Andanothernewfileonnode1named/etc/sysconfig/network-scripts/ifcfg-bond1withthefollowingcontents:
DEVICE=bond1IPADDR=172.16.100.120 NETMASK=255.255.255.0 ONBOOT=yes BOOTPROTO=none USERCTL=no MII_NOT_SUPPORTED=yes
Createthesesametwofilesonnode2,butreplacetheIPaddressesappropriatelysothatthelastoctetis121ineachfile.
Onnode1,youneedtomodifythecontentsof/etc/sysconfig/network-scripts/ifcfg-eth0.Inthisfile,youshouldkeeptheexisUngHWADDRline,butreplacetherestofthelinessothatthefilelookslikethis:
DEVICE=eth0 USERCTL=no ONBOOT=yes MASTER=bond0 SLAVE=yes BOOTPROTO=none MII_NOT_SUPPORTED=yes HWADDR=<value-from-existing-file> Dothesamethingfor/etc/sysconfig/network-scripts/ifcfg-eth2,keepingitsHWADDRlineand
modifyingtheDEVICE=linetoreadeth2(allotherlinesshouldmatchabove).
Similarly,weneedtomodifythe/etc/sysconfig/network-scripts/ifcfg-eth1and/etc/sysconfig/network-scripts/ifcfg-eth3files,butinthosefiles,we'llsetMASTER=bond1.
Finally,withallconfiguraUonchangesinplace,ensurethatyouhaveaccesstotheserver'sconsole(incasethechangesdon'tworkwell,networkconnecUvitymaybedisabled).Onceyoudo,itissafetoreboottheserverand,hopefully,ifallchangesaredoneproperly,you'llbeabletoreconnectaEerthereboot.Then,youshouldseeaconfiguraUonlikethis(thisissampleoutputfromnode1):
[root@node1 ~]# ifconfig -abond0 Link encap:Ethernet HWaddr 00:0C:29:FC:5F:65 inet addr:192.168.78.120 Bcast:192.168.78.255 Mask:255.255.255.0 inet6 addr: fe80::20c:29ff:fefc:5f65/64 Scope:Link UP BROADCAST RUNNING MASTER MULTICAST MTU:1500 Metric:1 RX packets:127 errors:127 dropped:0 overruns:0 frame:0 TX packets:123 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:17143 (16.7 KiB) TX bytes:19870 (19.4 KiB)
bond1 Link encap:Ethernet HWaddr 00:0C:29:FC:5F:6F inet addr:172.16.100.120 Bcast:172.16.100.255 Mask:255.255.255.0
Paper4888
DBA,Clustering
inet6 addr: fe80::20c:29ff:fefc:5f6f/64 Scope:Link UP BROADCAST RUNNING MASTER MULTICAST MTU:1500 Metric:1 RX packets:80 errors:77 dropped:0 overruns:0 frame:0 TX packets:111 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:11408 (11.1 KiB) TX bytes:15874 (15.5 KiB)
eth0 Link encap:Ethernet HWaddr 00:0C:29:FC:5F:65 inet6 addr: fe80::20c:29ff:fefc:5f65/64 Scope:Link UP BROADCAST RUNNING SLAVE MULTICAST MTU:1500 Metric:1 RX packets:122 errors:122 dropped:0 overruns:0 frame:0 TX packets:119 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:14015 (13.6 KiB) TX bytes:18422 (17.9 KiB) Interrupt:67 Base address:0x2024
eth1 Link encap:Ethernet HWaddr 00:0C:29:FC:5F:6F inet6 addr: fe80::20c:29ff:fefc:5f6f/64 Scope:Link UP BROADCAST RUNNING SLAVE MULTICAST MTU:1500 Metric:1 RX packets:56 errors:53 dropped:0 overruns:0 frame:0 TX packets:90 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:7014 (6.8 KiB) TX bytes:11712 (11.4 KiB) Interrupt:67 Base address:0x20a4
eth2 Link encap:Ethernet HWaddr 00:0C:29:FC:5F:65 inet6 addr: fe80::20c:29ff:fefc:5f65/64 Scope:Link UP BROADCAST RUNNING SLAVE MULTICAST MTU:1500 Metric:1 RX packets:23 errors:23 dropped:0 overruns:0 frame:0 TX packets:22 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:4316 (4.2 KiB) TX bytes:4396 (4.2 KiB) Interrupt:51 Base address:0x2424
eth3 Link encap:Ethernet HWaddr 00:0C:29:FC:5F:6F inet6 addr: fe80::20c:29ff:fefc:5f6f/64 Scope:Link UP BROADCAST RUNNING SLAVE MULTICAST MTU:1500 Metric:1 RX packets:24 errors:24 dropped:0 overruns:0 frame:0 TX packets:21 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:4394 (4.2 KiB) TX bytes:4162 (4.0 KiB)
Interrupt:59 Base address:0x24a4
Youcanquerythestatusofthebondsbylookingatthecontentsof/proc/net/bonding/bond0(orbond1).Undernormalcircumstances,here'swhatthebond0statusinformaUonmightlooklike:[root@node1 ~]# cat /proc/net/bonding/bond0Ethernet Channel Bonding Driver: v3.2.4 (January 28, 2008)
Bonding Mode: fault-tolerance (active-backup)Primary Slave: NoneCurrently Active Slave: eth0MII Status: upMII Polling Interval (ms): 100Up Delay (ms): 0Down Delay (ms): 0
Slave Interface: eth0
Paper4889
DBA,Clustering
MII Status: upLink Failure Count: 0Permanent HW addr: 00:0c:29:fc:5f:65
Slave Interface: eth2MII Status: upLink Failure Count: 0Permanent HW addr: 00:0c:29:fc:5f:79[root@node1 ~]#
CONFIGURESHAREDSTORAGE
Next,weneedtoconfigurethesharedstoragethatwewillusetostoretheRACdatabase.TheprocessforconfiguringstoragewillvarydependingonwhichtypeofstorageaVachment(fibrechannel,iSCSI,NFS,etc.)youuse.YoushouldabsolutelyconfigureredundantpathstoyourstoragenomaVerwhichstoragetypeyouuse.
Becauseofthevariedmethodsforconfiguringredundantpaths,we'llleavethattopicforthesystemandstorageadministratorstoaddress.Oncethestorageisaccessibletoallhostsinthecluster,itneedstobeconfiguredforusewithRAC.Forthiscluster,wewilluseASM(withoutASMLib),soourfirststepistoparUUonthedisks.Ittypicallydoesn'tmaVerwhichpathtothediskyouuseinordertoparUUonit.However,itmakesahugedifferencewhichpathyouusetoaccessitfornormalread/writeaccess,sopaycloseaVenUontothepathsusedforeach.
YoucanparUUonthediskswiththesecommandsforeachdisk(/dev/sdbthrough/dev/sdf):fdisk /dev/sdb,thenenter“n”,“p”,“1”,<return>,<return>,and“w”.ThatwillcreateasingleparUUonthatisthesizeofthewholediskoneachdisk.
OnLinux5,theproperwaytosetprivilegesonthediskdevicefilesistomodifyfilesin/etc/udev/rules.d.SeeMetalink465001.1,564580.1,and605828.1,asapplicable,forinstrucUonsontheproperconfiguraUonfordiskdevicefiles.
Seethenoteslaterinthisdocument(onpage16,under“SharedStorageMisconfiguraUon”)aboutusingudevtoconfiguredeviceaccess.udevisabeVerandmoresupportablemethodandthenotesmenUonedinthatsecUonwillprovideinstrucUonsonhowtoconfigureanduseudevonyoursystem.
TherearenorawdevicestocreateonOEL5,sowe’redoneconfiguringoursharedstorageatthispoint.
WhiletherearewaystorunaseriesofcommandstorefreshthecachedparUUontablesforeachdiskineachhost,itisprobablyeasiesttorebootallnodesatthispointtomakethingseasier.WhenmakingalargenumberofconfiguraUonchanges,itisalsohelpfultorebootthenodesperiodicallyduringtheconfiguraUonstepstoensurethattheywillbootwithouterror.
ESTABLISHHOSTEQUIVALENCY
Ournextstepswillestablishhostequivalencyfortheoracleusersothattheoracleaccountisfreetologintotheremotenodewithoutbeingpromptedforapassword.Inourcase,we’renotasconcernedwithsecurityaswearewithkeepingtheconfiguraUonverysimple,sowewon’tprotectourprivatekeywithapassword(notabestpracUceforproducUon).
NotethatoncetheinstallaUonsarecompleted,thereisnorequirementforhostequivalency(otherthanconvenienceformaintenanceoperaUons),soyoumaydisablethehostequivalencyestablishedinthissecUononceallinstallaUonsarecomplete.
Toestablishhostequivalency,followthesesteps:
Logintonode1astheoracleuser.Run“ssh-keygen –t dsa”tocreateanSSHkey.Specifyapassphraseforthekeyasyouwish.Youcanalsoleaveitblankandavoidthessh‐agentandssh‐addcommandscomingupinafewsteps.However,aprivatekeyfilewithoutapassphraseisadangeroussecuritypracUceandleavestheaccountopentocertainvulnerabiliUes,soIrecommendusingapassphraseforallkeyfiles.
Paper48810
DBA,Clustering
AEerthekeyiscreated,run“cp ~/.ssh/id_dsa.pub ~/.ssh/authorized_keys”toprepareusforthehostequivalencyrequiredlaterfortheinstallaUons.Alsorunthesecommandsastheoracleuser“eval `ssh-agent`; ssh-add”andthenenterthepassphraseyoucreatedforyourSSHkey.ItisimportantthatyourunthesecommandsfromthesameshellenvironmentwhereyouwillulUmatelyruntheOracleInstallerfrom.
Alsodothisastheoracleuseronnode1:scp -rp ~/.ssh node2:.(thepreceedingperiodispartofthecommand)ThiswillcopytheenUre.ssh directoryovertonode2sothat,mostimportantly,theauthorized_keysfileisinplaceandwecanrunremotecommandsonnode2withoutbeingpromptedforapassword.
Foreachhostinthecluster,sshtothathosttoconnectbyrunningthefollowingcommandsinsuccessionfromtheoracleaccount.Notethatyouwillbepromptedtoaccepteachhost’skey;respond“yes”whenprompted.Youshouldn'tbepromptedtoenterapasswordforanyoftheseconnecUons.ssh node1 datessh node1-priv datessh node1.dannorris.net datessh node1-priv.dannorris.net datessh node2 datessh node2-priv datessh node2.dannorris.net datessh node2-priv.dannorris.net date
Atthispoint,we’vedonealltheOSpreconfiguraUonandhaveallthenecessaryitemsinplacetobegininstallingsoEware.Ifyou’recauUousandwanttosavesomeUmeintheeventofacatastrophe,thisisagoodpointtoshutdownthevirtualmachinesandtakeacopyofthemforbackuppurposes.
CONFIRMPREREQUISITESWITHTOOLS:CLUVFYANDRDA MostRACimplementaUonsthatseemdifficultareusuallyplaguedwithchallengesbecauseofmissingprerequisites.ItissomeUmesdifficulttocheckallprerequisitesaccuratelyforRACclustersandthat'swhyOraclehasbuiltafewtoolstohelpwiththeprocess.TheClusterVerificaUonUUlity(a.k.a.CVUorcluvfy)andRemoteDiagnosUcAssistant(RDA)helpautomatealltheprerequisitechecksnecessaryforRACclusters.OEen,theyincorporateupdatedrequirementsthataren'tinthestandarddocumentaUonaswell.ItishighlyrecommendedtoruneachoftheseuUliUesoneachnodeinthecluster.
CLUSTERVERIFICATIONUTILITY(CVU)
TheCVUisavailablefromOTNandisalsoavailableontheOracleClusterwareinstallaUonmedia.IrecommenddownloadingthelatestversionfromOTNathVp://www.oracle.com/technology/products/database/clustering/cvu/cvu_download_homepage.htmlsinceitisoEenupdatedaEertheOracleClusterwaremediaiscreated.
TheCVUhasmanycommand‐lineopUonsasitprovidesdifferentchecksthatcanberunatvariousstagesoftheinstallaUon(pre‐Clusterwareinstall,pre‐DBinstall,post‐DBinstall,etc.).Inourcase,wewanttorunthepre‐ClusterwareinstallaUonchecks.
ToruntheCVUchecks,gotothelocaUonwhereyouunpackedtheCVUdownloadandrunitastheoracleuser.Beforerunningit,ensurethatyouhavestartedthessh‐agentandloadedyourprivatekeyfileintotheagentsothatyouhavehostequivalencywiththeotherclusternodes.Here'showtorunCVU:cd /path/to/cvu/unzip/directory./cluvfy stage –pre crsinst –n node1,node2 –r 11gR1
Theoutputyou'llseeisaseriesoftests.Eachtestwillhaveastatusreportedassuccessfulorfailed.
NotethatonLinuxx86‐64,youmayseesomeofthepackageslistedtwice.Whileitlookslikethesearethesamepackages,itiscommonforoneofthetwocheckstofailwhiletheothersucceeds.Thesechecksareforthesamepackage,butonecheckisforthe32‐bitversionandtheotherforthe64‐bitversion.The32‐bitversionisn'tinstalledby
Paper48811
DBA,Clustering
default.ThedocumentaUongivesalistofthepackageprerequisitesincludingthepackagesthatneedtobeinstalledwithboththe64‐bitand32‐bitversions.Tolistthepackagesandtheirarchitecturestogether,usethisrpmcommand:rpm -qa –queryformat="%{NAME}-%{V}-%{R}.%{ARCH}\n"
Thisallowsyoutoseemoreeasilywhichpackagesaremissing.
Here'sasampleoutputfromtheCVUonmytestLinux32‐bitcluster(notethatmyswapspaceandmemoryrequirementsarenotmet).[oracle@node1 ~]$ /mnt/runcluvfy.sh stage -pre crsinst -n node1,node2 -r 11gR1
Performing pre-checks for cluster services setup
Checking node reachability...Node reachability check passed from node "node1".
Checking user equivalence...User equivalence check passed for user "oracle".
Checking administrative privileges...User existence check passed for "oracle".Group existence check passed for "oinstall".Membership check for user "oracle" in group "oinstall" [as Primary] passed.
WARNING: The effective group id "dba" differs from the primary group id "oinstall" of user "oracle".
Administrative privileges check passed.
Checking node connectivity...
Node connectivity check passed for subnet "192.168.78.0" with node(s) node2,node1.Node connectivity check passed for subnet "172.16.100.0" with node(s) node2,node1.
Interfaces found on subnet "192.168.78.0" that are likely candidates for VIP:node2 bond0:192.168.78.121node1 bond0:192.168.78.120
Interfaces found on subnet "172.16.100.0" that are likely candidates for a private interconnect:node2 bond1:172.16.100.121node1 bond1:172.16.100.120
Node connectivity check passed.
Checking system requirements for 'crs'...Total memory check failed.Check failed on nodes: node2,node1Free disk space check passed.Swap space check failed.Check failed on nodes: node2,node1System architecture check passed.Kernel version check passed.
Paper48812
DBA,Clustering
Package existence check passed for "make-3.81".Package existence check passed for "binutils-2.17.50.0.6".Package existence check passed for "gcc-4.1.1".Package existence check passed for "libaio-0.3.106".Package existence check passed for "libaio-devel-0.3.106".Package existence check passed for "libstdc++-4.1.1".Package existence check passed for "elfutils-libelf-devel-0.125".Package existence check passed for "sysstat-7.0.0".Package existence check passed for "compat-libstdc++-33-3.2.3".Package existence check passed for "libgcc-4.1.1".Package existence check passed for "libstdc++-devel-4.1.1".Package existence check passed for "unixODBC-2.2.11".Package existence check passed for "unixODBC-devel-2.2.11".Package existence check passed for "glibc-2.5-12".Group existence check passed for "dba".Group existence check passed for "oinstall".User existence check passed for "nobody".
System requirement failed for 'crs'
Pre-check for cluster services setup was unsuccessful on all the nodes. [oracle@node1 ~]$
REMOTEDIAGNOSTICASSISTANT:HARDWARECOMPATIBILITYVERIFICATIONENGINE(RDAHCVE)
TheRDAisavailablefromMetalinkvianote314422.1andhasversionsavailableforeachplaTormsupportedbyRAC.ToruntheRDAinHCVEmode,downloadtheappropriateversionfromMetalink,placeitonyourproposeddatabaseserversandrunit(astheoracleuserisfine—norootprivilegesneeded)usingthiscommand:cd /path/to/rda/unzip/directory./rda.sh -T hcve
You'llbepromptedwithamenuofRDAprerequisitecheckers.IrecommendmulUplerunsfirstcheckingfortheregulardatabaseversionyou'reinstallingandasecondrunwhenyoushouldchoosetheRACchecks.
Onceyoumakeyourchoice,you'llbepromptedwithafewaddiUonalquesUonsandthenofferedasummaryreportofthefindings.You'remostinterestedinthethingsmarkedasFAILED.
Here'sasamplerunformytestenvironment:[oracle@node1 rda]$ ./rda.sh -T hcveProcessing HCVE tests ...Available Pre-Installation Rule Sets: 1. Oracle Database 10g R1 (10.1.0) PreInstall (Linux-x86) 2. Oracle Database 10g R1 (10.1.0) PreInstall (Linux AMD64) 3. Oracle Database 10g R1 (10.1.0) PreInstall (IA-64 Linux) 4. Oracle Database 10g R2 (10.2.0) PreInstall (Linux AMD64) 5. Oracle Database 10g R2 (10.2.0) PreInstall (IA-64 Linux) 6. Oracle Database 10g R2 (10.2.0) PreInstall (Linux-x86) 7. Oracle Database 11g R1 (11.1.0) PreInstall (Linux AMD64) 8. Oracle Database 11g R1 (11.1.0) PreInstall (Linux-x86) 9. Oracle Application Server 10g (9.0.4) PreInstall (Linux) 10. Oracle Application Server 10g R2 (10.1.2) PreInstall (Linux) 11. Oracle Application Server 10g R3 (10.1.3) PreInstall (Linux AMD64) 12. Oracle Application Server 10g R3 (10.1.3) PreInstall (IA-64 Linux) 13. Oracle Application Server 10g R3 (10.1.3) PreInstall (Linux-x86) 14. Oracle Portal PreInstall (Generic)Available Post-Installation Rule Sets: 15. Oracle Portal PostInstall (generic) 16. RAC 10G DB and OS Best Practices (Linux) 17. Data Guard PostInstall (Generic)
Paper48813
DBA,Clustering
Enter the HCVE rule set numberHit 'Return' to accept the default (1)> 8
Enter value for < Planned ORACLE_HOME location or if set >> /u01/app/oracle/product/11.1.0/db_1
Enter value for < JDK Home >> /u01/app/oracle/product/11.1.0/db_1/jdk
Test "Oracle Database 11g R1 (11.1.0) PreInstall (Linux-x86)" executed at Thu Mar 12 23:54:01 2009
Test Results~~~~~~~~~~~~
ID NAME RESULT VALUE====== ==================== ======= ==========================================A00010 OS Certified? PASSED AdequateA00020 User in /etc/passwd? PASSED userOKA00040 Group in /etc/group? PASSED GroupOKA00050 Input ORACLE_HOME RECORD /u01/app/oracle/product/11.1.0/db_1A00060 ORACLE_HOME Valid? PASSED OHexistsA00070 O_H Permissions OK? PASSED CorrectPermsA00080 oraInventory Permiss PASSED oraInventoryOKA00090 Got ld,nm,ar,make? PASSED ld_nm_ar_make_foundA00100 Umask Set to 022? PASSED UmaskOKA00120 Limit Processes PASSED AdequateA00130 Limit Descriptors PASSED AdequateA00140 LDLIBRARYPATH Unset? PASSED UnSetA00180 JAVA_HOME Unset? PASSED UnSetA00190 JDK Home RECORD /u01/app/oracle/product/11.1.0/db_1/jdkA00200 JDK Version PASSED AdequateA00210 Other O_Hs in PATH? PASSED NoneFoundA00220 Other OUI Up? PASSED NoOtherOUIA00230 /tmp Adequate? PASSED TempSpaceOKA00240 Disk Space OK? PASSED DiskSpaceOKA00250 Swap (in MB) RECORD 1023A00260 RAM (in MB) FAILED 1011A00270 Swap OK? FAILED NAA00280 Network PASSED ConnectedA00290 IP Address RECORD 192.168.78.120A00300 Domain Name RECORD dannorris.netA00310 DNS Lookup FAILED nslookup host.domainA00320 /etc/hosts Format PASSED Contains host.domainA00330 Kernel Parameters OK FAILED NAA00380 Tainted Kernel? PASSED NotVerifiableA00400 ip_local_port_range PASSED ip_local_port_rangeOKA00460 Asianux2 RPMs OK? PASSED NotAsianuxA00470 Asianux3 RPMs OK? PASSED NotAsianuxA00480 EL4 RPMs OK? PASSED NotEL4A00490 EL5 RPMs OK? PASSED EL5rpmsOKA00530 RHEL4 RPMs OK? PASSED NotRedHatA00540 RHEL5 RPMs OK? PASSED NotRedHatA00570 SUSE SLES10 RPMs OK? PASSED NotSuSEResult file: /home/oracle/rda/output/RDA_HCVE_A200DB11R1_lnx_res.htm[oracle@node1 rda]$
Paper48814
DBA,Clustering
ORACLEPRODUCTINSTALLATIONS
INSTALLCLUSTERWARE
Finally,wecanstartinstallingsomesoEware.FollowthesestepstoinstallOracleClusterwareonbothnodes.
MakeaOracleClusterware11ginstallaUonCD(orthedownloadoftheCDcontents)availabletothenode1virtualmachine.
Ifyou’vefollowedalongcarefullytothispoint,youwon’thaveagraphicalconsole.Ifthat’strue,thenyou’llprobablywanttostartVNCinordertoperformthegraphicalinstallaUons.TostartVNC,whileloggedintonode1astheoracleuser,createthefollowingfilenamed$HOME/.vnc/xstartup(youmighthavetocreatethe$HOME/.vncdirectoryfirst):xsetroot -solid greygnome-session &
Then,youcanstarttheVNCserverbyrunning“vncserver”fromthecommandpromptastheoracleuser.Notethatthevncserversessionwillinherityourenvironment,soitisbesttonothaveORACLE_HOMEorORACLE_SIDset(andORACLE_HOME/binshouldn'tbeinyourPATHeither).ThefirstUmeyourunvncserver,you’llbepromptedtosetapasswordforyourvncserverdesktop.
OnceVNCisrunning,connecttotheserverwiththeaVNCviewer.Then,ensureyourenvironmenthastheORACLE_BASEenvironmentvariablesetto/u01/app/oracle.
AEerconfirmingtheproperenvironmentseWngs,runthesecommands:cd <download_location>/clusterware; ./runInstaller
OntheSpecifyInventoryDirectoryscreen,takethedefaults(/u01/app/oraInventoryandoinstall).
OntheSpecifyHomeDetailsscreen,takethedefaultsofOraCrs11g_homeand/u01/app/11.1.0/crs
Allprerequisitechecksshouldsucceed.
OntheSpecifyClusterConfiguraUonscreen,thelocalnode(node1)seWngsshouldalreadybedisplayed.AddtheinformaUonfornode2tothescreenusingtheappropriatenamesforPublic,PrivateandVirtual.
OntheSpecifyNetworkInterfaceUsagescreen,seteth0tobePublicandleaveeth1asPrivate.
OntheSpecifyOCRLocaUonscreen,chooseyourdesiredredundancylevel(IpreferNormal)andentertheappropriatemulUpathedpseudodevicefile(s)fortheOCRlocaUon.
OntheSpecifyVoUngDiskLocaUonscreen,chooseyourdesiredredundancylevel(IpreferNormal)andentertheappropriatemulUpathedpseudodevicefile(s)forthevoUngdisklocaUon.
OntheSummaryscreen,ensurethattheClusterNodeslistincludesbothnodesandthenclickInstall.
OncetheinstallaUonsfinish,runtherootscriptsasprompted.Remembertorunthemserially—notconcurrently.
AEertherootscriptsarerun,clickOKandthenthefinalassistantsrun.Allassistantsshouldrunwithoutissueorerror.Somewarningsintheroot.shoutputarenormal,though.
INSTALLASMANDRDBMSSOFTWARE
WerecommendinstallingASMinaseparateORACLE_HOMEdirectoryfromtheRDBMSsoEware.TheASMsoEwareisjustthesameastheRDBMSsoEwareexceptthatinsteadofconfiguraUonadatabase,youconfigureittorunASM.So,repeatthestepsbelowtwicesothatyouhavetwoseparateORACLE_HOMEswithidenUcalsoEwareinthem.
HerearethestepstoinstallandconfigureASMandRDBMS.
Onceagain,we’lluseVNCtoperformthisinstallaUonfromthenode1node.MaketheRDBMS11gsoEwareCDsordownloadsavailableonnode1(eitherlocallyorviaanNFSmount).We’llassumethatthesoEwareisavailableat/mnt/cdrom/db11gr1-linux/databaseinourcase.
Paper48815
DBA,Clustering
WithintheVNCsessionthatisrunningastheoracleuseraccount(ensureORACLE_BASE issetintheenvironment),runtheinstallerbyrunningthesecommands:“cd /mnt/cdrom/db11gr1-linux/database ; ./runInstaller”
OntheSelectInstallaUonTypescreen,chooseEnterpriseEdiUonorStandardEdiUon,asappropriate.
OntheInstallLocaUonscreenleavethedefaultsofOracleBase:/u01/app/oracle;Name:whatevernameyoulike;Path:/u01/app/oracle/product/11.1.0/asm_1 (for ASM) or/u01/app/oracle/product/11.1.0/db_1 (for RDBMS)
OntheSpecifyHardwareClusterInstallaUonModescreen,chooseClusterInstallaUonandclick“SelectAll”tochooseallnodes.
Allprerequisitechecksshouldbesuccessful.
OntheSelectConfiguraUonOpUonscreen,chooseinstallsoEwareonly.
OntheOracleConfiguraUonManagerRegistraUonscreen,clicknexttobypassregistraUon.
ClickInstallonthesummaryscreentostarttheinstallaUon.Gogetsomecaffeineandwait.
OncetheinstallaUonfinishes,you’llbepromptedtoruntheroot.shscriptoneachnode.RunthemsequenUally(eithernodefirst).Takethedefaultforthelocalbindirectoryaspromptedbytheroot.shscript.
Atthisstage,theASMinstancesshouldberunningoneachnode(youshouldseelotsofasm_XXXX_+ASM? processesrunningoneachnode.The“crs_stat –t”commandshouldalsoshowASMandlistenerresourcesconfiguredforeachnode.
INSTALLPATCHES
NowthatyouhavetheORACLE_HOMEsinstalled,youneedtoapplypatches.ThebasicpatchsetsforeachoftheORACLE_HOMEs(Clusterware,ASM,andRDBMS)arethesamepatchset(in10gR2and11gR1,anyway)‐‐theRDBMSpatchset.FollowthepatchREADMEinstrucUonstoinstallthepatchoneachoftheORACLE_HOMEsinthisorder:Clusterware,ASM,RDBMS.
YoushouldalsoconsiderinstallingthelatestCriUcalPatchUpdate(seehVp://www.oracle.com/technology/deploy/security/alerts.htm)and,ifusing10gR2,alsoconsiderinstallingthelatestClusterwarepatchbundle(seeML405820.1).ItisrecommendedtoinstallthelatestCriUcalPatchUpdatepatchesastheypatchseveresecuritybugsthatmayleaveyourenvironmentvulnerableifnotpatched.TheCPUpatcheshavetheirowninstallaUoninstrucUonsandproceduresthatshouldbefollowedcarefully.
CONFIGUREASM
Beforeyoucancreateadatabaseisthisnewlypatchedenvironment,youfirstneedtoconfigurethestorageyou'lluseforthedatabase.ToconfigureASM,usetheDatabaseConfiguraUonAssistant.
StartVNC,justasbefore.
ConnecttoVNCandsetyourenvironmenttotheASMORACLE_HOMEintheterminalwindowinsidetheVNCsession.Withitsetproperly,starttheDatabaseConfiguraUonAssistant(dbca&)attheprompt.
Onthefirstscreen,chooseConfigureAutomaUcStorageManagement.
OntheConfigureAutomaUcStorageManagementscreen,youmayfirsthaveto“ChangeDiskDiscoveryPath”to“/dev/sd*”(orwhatevertheappropriatepaVerntoidenUfythemulUpathingpseudodevices)inordertoseeyourcandidatedrives.Then,createoneExternalRedundancydiskgroupnamedDATAcheckingtheboxesnexttothedrivesthatshouldbeinthatgroup.BesurethatthedevicesyouseelistedareonlythemulUpathingdevicesandnoneofthedirect,non‐mulUpatheddevices.
Usingthisinterface,createthedesireddiskgroupconfiguraUonwhichyouhavealreadyplannedout.
Paper48816
DBA,Clustering
CONFIGURENEWDATABASE
NextonthetasklististoconfigureaRACdatabase.We’llusethequickandeasyDBCAtocreatethenewdatabase.Forthis,we’llusetheVNCconnecUononceagainsinceDBCAisaGUItool.ConnecttoaVNCsessionononeofthetwoserversandfollowthestepsheretocreateadatabaseinstance.
Setyourenvironmentvariablesasshownhere:export ORACLE_HOME=/u01/app/oracle/product/11.1.0/db_1export PATH=$ORACLE_HOME/bin:$PATH
ThenruntheDBCAtoolbyentering:dbca &
Ontheopeningscreen,choosedatabasetype“OracleRealApplicaUonClustersdatabase”
ChoosetheoperaUon“CreateaDatabase”
OnthenodeselecUonscreen,chooseSelectAll
Onthetemplatesscreen(step3of19),chooseGeneralPurposeorTransacUonProcessing
FortheGlobalDatabaseNameenter“RAC.world”andlettheSIDPrefixdefaultaccordingly(defaultshouldbeRAC)
OntheconfiguraUonassistantscreen(step5of18),choosethedefaultof“ConfiguraUonEnterpriseManager”andyoudonotneedtoenablealertnoUficaUonsnordailydiskbackups
Forsimplicity(insteadofsecurity),choose“UsetheSameAdministraUvePasswordforAllAccounts”onstep6of18andenterapassword.
OnthestorageselecUonscreen,chooseAutomaUcStorageManagement(ASM)
SelecttheDATAdiskgroupandclicknextonthediskgroupselecUonscreen.
UseOracle‐ManagedFilesontheDatabaseFileLocaUonsscreen(step9of17)
NoneedtoconfigureaflashrecoveryareaorarchivingatthisUme
Youmayinstallthesampleschemasifyouwouldlike.
IchosetousethedefaultandsuggestedtypicalmemoryconfiguraUon.Youcanusewhateveryou’dlike,butensureaminimumSGAsizeof400Mb.
OntheCharacterSettab,IprefertouseUnicode.
Keeptheenhanced11gsecurityseWngsonstep13of16
ItisrecommendedtoleavetheautomaUcmaintenancetasksenabled
TherearenorequiredchangesontheDatabaseStoragescreen
Onthefinalscreen,clickFinish,thenclickOKonthefinalsummary.Wait.
OncetheDBCAiscomplete,itwilldisplaythesummaryconfirmaUonscreen.MakeanoteoftheURLfortheDBControlwebsite.
WhenyouclickExitonthesummaryscreen,theinstancesarestarted.Ittakesawhilebeforeanymessagesaredisplayed,sobepaUent.
Alldone!RACdatabaseiscreated,servicescreatedandeverythingshouldberunning.
CHALLENGESANDCOMMONMISTAKES
FIREWALLSETTINGSINGUEST(LINUX)OS
IfyoufailtodisabletheLinuxfirewallorleaveSELinuxinanyseWngotherthandisabled,youwilllikelyencountersecurity‐relatedissuesandwarnings.TheclusterwareinstallaUonmayhaveerrorsduetonetworkportsnotbeingavailableoropenforcommunicaUon.
Paper48817
DBA,Clustering
SHAREDSTORAGEMISCONFIGURATION
OneofthecommonsetupissuesisconfiguraUonofsharedstorage.OnLinuxservers,devicesarenotpersistentlyboundtothesamedevicename.Thatis,Linuxmaydecidetomake/dev/sdb into/dev/sdc onsubsequentreboots,basedonanumberoffactors.Asthisisaknownissue,thereisalsoaknown,stable,acceptedwaytoaddresstheissue.WithRACclusters,thisisespeciallydaunUng,sincethedevicenotonlyneedstobethesamenameonthatnodeeachUme,butitneedstobethesamenameonallnodesinthecluster.
ThesoluUonusedformostrecentLinuxversionsistouseasubsystemcalled“udev.”Theudevsubsystemcreatesalayerofdevicenamesontopoftherealdevices.ThenamesusedbyudevarecontrolledbyrulesandallmappingsaredoneusingdeviceidenUfiersuniquetoeachdeviceinordertobecertainthattherightdevicesareidenUfiedfromeachnode.ThisconfiguraUondocumentdidn’tuseudev,butinsteadsetprivilegesdirectlyusingcommandsin/etc/rc.local(notanappropriateconfiguraUonforaproducUonenvironment).
Ifyoudonotconfigureudevordevicemapper,theclusterverificaUonuUlitymayreportsuccess,butaEerrebooUngthenodes(especiallywhenaddingorreconfiguringstorage),youmayfindthattheclusternolongercomesupbecausethedevicenamesallchangedduringtherebootcycle.
Tolearnmoreabouthowtoconfigureudev,seethefollowingnotesonMetalink:
443996.1–HowtomaprawdeviceonRHEL5andOEL5
465001.1–Configuringrawdevices(singlepath)forOracleClusterware10gRelease2(10.2.0)onRHEL5/OEL5
564580.1–Configuringrawdevices(mulUpath)forOracleClusterware10gRelease2(10.2.0)onRHEL5/OEL5
605828.1–ConfiguringmulUpathdevicesforOracleClusterware11g(11.1.0)onRHEL5/OEL5
414897.1–Howtosetup/etc/udev/rules.dforOCRandVoUngforRAConSLES10/RHEL5andEL5
371814.1–UsingudevwithOracleArchitecture(RAC&ASM)–RedHat4.0
456239.1–UnderstandingDevice‐mapperinLinux2.6Kernel
357472.1–Configuringdevice‐mapperforCRS/ASM
465001.1–ConfiguringrawdevicesforOracle10gR2(10.2.0)ClusterwareonRedHat/EnterpriseLinux5(RH/EL5)
NETWORKINGSETUPISSUES
ItissomeUmesdifficulttoconfigurethemulUplenetworksrequiredforaRACdeployment.Usuallythat’sbecauseit’shardtoknowwhichphysicalportontheservercorrespondstoeth0,eth1,etc.Toaddressthis,IfinditbesttoconfigureonenetworkinterfaceataUmeinsteadofconnecUngthemallandthentryingtosortthingsout.Ifnecessary,evengoingasfarasphysicallydisconnecUngthewirecanhelp.That’sbecausemostEthernetdriverswillreportwhenthelinkisupordownontheconsolebuffer(whichyoucanseebyrunning“dmesg”onaserver).BydisconnecUngandreconnecUngawire,youshouldbeabletoseemessagesindicaUngwhichphysicalinterfacelostitslinkandthatshouldtellyouwhichphysicalportcorrespondstothatphysicalinterface.
It’salsowisetoincludetheHWADDRinthe/etc/sysconfig/network-scripts/ifcfg-ethXfiletoensurethatLinuxdoesn’trenumbertheinterfaces(whichcanhappenwhenhardwareconfiguraUonschange).Bylockingthatdevicetoaphysicalhardwareaddress(MACaddress),youensurethatitwillalwaysremain“Ued”together.
ToestablishredundancyfornetworkconnecUons,youcanfindmoreinformaUonaboutusingandconfiguringnetworkbondingonLinuxfromthesearUclesonMetalink:
456055.1–HowtocreatetwodifferentbondingconfiguraUonsinOEL4
298891.1–ConfiguringLinuxfortheOracle10gVIPorprivateinterconnectusingbondingdriver
434375.1–LinuxEthernetbondingdriver
Paper48818
DBA,Clustering
LINUXINSTALLATIONCHALLENGES
Ifyouhaven’tinstalledLinuxbefore,youmayfearthatitwillbecomplexordaunUng.ManypeoplefindthatLinuxinstallaUonisnotnearlyasdifficultastheythoughtitmayhavebeen.
ThemostcommonissueswithLinuxinstallaUonusuallyinvolvediskparUUoning.SometrytosizetheparUUonstoosmallsoastomakethebestuseofspaceandotherstrytocreateseparateparUUonsformanydifferentareasofthefilesystemtree.Forthe“normal”installaUonsIperform,IusuallyhaveparUUonsfor/(root),swap,/tmpand/u01(forOraclesoEware).Withdisksizesbeingquitelargethesedays,Iusuallysizethingsthisway:
Purpose Size Notes
/ 15Gb
swap >=2GbMoreifRAMsizeislarger,buttypicallynolargerthan8Gb
/tmp 2GbIfthereisalotofspaceavailable,Imaymakethis4Gb,butnolargerthanthat.
/u01 therestEverythingelseisallocatedhere—minimumof20Gb,usuallymuchlarger.
LINUXUSEREQUIVALENCY ConfiguringuserequivalencyonLinux(oranyUNIX‐basedplaTorm)istheprocessofseWnguptheuseraccountsontwoormorehostssothattheywon’trequireapasswordtologinand/orruncommandsoneachother’shosts.Formany,thisconfiguraUonischallengingandtheydon’tunderstandthereasonswhytheirconfiguraUondoesn’tworksincetheydon’tknowhowitreallyworks.
TherearequiteafewreferencesonlinetofindhelpwiththisconfiguraUonandgenerally,ifyoufollowtherightsteps,evenifyoudon’treallyunderstandhowitworks,you’llendupwithaworkingconfiguraUon.Thatsaid,it’salwaysbeVertounderstandhowitworksandespeciallysointhiscasesinceithastodowithsystemsecurityandmessinguptheserverconfiguraUonmayresultinalesssecuresystem.Forourpurposesinthisdocument(geWngaworkingRACconfiguraUon),followingthestepsforuserequivalencyshownhereinissufficientanddoesnotcompromisesystemsecurity.
hVp://www.ardentperf.com/2007/03/27/dbca‐slow‐check‐host‐equivalence/
372795.1–HowtoconfigureSSHforuserequivalence
300548.1–HowtoconfigureSSHforaRACinstallaUon
REFERENCES TimHall’sarUcleoninstalling11gRAConVMWareusingOracleEnterpriseLinux5athVp://www.oracle‐
base.com/arUcles/11g/OracleDB11gR1RACInstallaUonOnOEL5UsingVMware.php
OracleMetalinknotes(specificnotescitedelsewhereinthisdocument)
JeremySchneider,“UnleasingOracleServices:AComprehensiveReviewof“Services”inOracleDatabases”,hVp://www.ardentperf.com/pub/services‐schneider07.pdf
Metalink567730.1–ChangesinOracleClusterwareonLinuxwiththe10.2.0.4Patchset
Metalink284752.1–10gRAC:StepsToIncreaseCSSMisscount,REbooWmeandDiskUmeout
Metalink294430.1–CSSTimeoutComputaUoninOracleClusterware
Metalink395878.1–Heartbeat/VoUng/QuorumRelatedTimeoutConfiguraUonforLinux,OCFS2,RACStacktoavoidunnecessarynodefencing,panic,andreboot
Paper48819
DBA,Clustering
FROMTHELAWYERS TheinformaUoncontainedhereinshouldbedeemedreliablebutnotguaranteed.TheauthorhasmadeeveryaVempttoprovidecurrentandaccurateinformaUon.IfyouhaveanycommentsorsuggesUons,pleasecontacttheauthoratdannorris(at)dannorris.com.
Paper48820