exadata and ovm best practices - oracle.com · –change memory only when recommended by oracle....

Copyright©2018, Oracleand/oritsaffiliates.Allrightsreserved.|

OracleExadataandOVMBestPractices

December2018


TopicsCovered

2

UseCases

Exadata OVMSoftwareRequirements

Exadata IsolationConsiderations

Exadata OVMSizingandPrerequisites

Exadata OVMDeploymentOverview

Exadata OVMAdministration andOperationalLifeCycle

Migration,HA,Backup/Restore, Upgrading/Patching

Monitoring, ResourceManagement


ExadataVirtualMachinesHigh-PerformanceVirtualizedDatabasePlatform

• VMsprovideCPU,memory,OS,andsysadminisolation forconsolidatedworkloads– Hosting,cloud,crossdepartmentconsolidation, test/dev,non-database orthirdpartyapplications

• ExadataVMsdelivernearrawhardwareperformance– I/Osgodirectlytohigh-speed InfiniBandbypassinghypervisor

• CombinewithExadatanetworkandI/Oprioritizationtoachieveuniquefullstackisolation

• TrustedPartitionsallowlicensingbyvirtualmachine

3

NoAdditional Cost

X7-2,X6-2,X5-2,X4-2,X3-2,X2-2

DB11.2andhigher

FINANCE

SUPPLYCHAIN

SALES


ExadataConsolidationOptions

4

VM

Database12cMultitenant

ManyDBsinoneServer

VirtualMachines

DedicatedDBServers

• VMshavegoodIsolationbutpoorefficiencyandhighmanagement– VMshaveseparateOS,memory, CPUs, andpatching– Isolation without need totrustDBA, SystemAdmin

• DatabaseconsolidationinasingleOSishighlyefficientbutlessisolated– DBResourcemanager isolation adds nooverhead– Resources canbesharedmuch moredynamically– But,must trustadmins toconfigure systems correctly

• BeststrategyistocombineVMswithdatabasenativeconsolidation–Multiple trustedDBsorPluggableDBsinaVM– FewVMsperserver tolimit overhead offragmentingCPUs/memory/patching etc.

VM VMMoreIso

latio

n

MoreEfficient


DatabaseServer:BareMetal/PhysicalversusOVMSoftwareArchitectureComparison

OVMDatabaseServer

dom0

Exadata (Linux,Xen,fw)

domU-3

Exadata (Linux)

OracleGI/DBhomesdomU-2

Exadata (Linux)

OracleGI/DBhomesdomU-1

Exadata (Linux)

OracleGI/DBhomes

BareMetal/PhysicalDatabaseServer

Exadata (Linux,fw)

OracleGI/DBhomes

NochangetoStorageGrid,Networking,or Other

5


MajorDifferencesBetweenPhysicalandOVM• Detailsexpandedthroughoutremainingslides

6

Topic HowOVM differs fromPhysical

Hardwaresupport 2-socket only

Cluster config Systemhas oneormoreVMclusters, eachwithownGI/RAC/DB install

Exadata storageconfig Separategriddisks/DATA/RECO foreachVMcluster; Bydefault noDBFSdisk group

Dbnode disk config VMfilesystem sizes aresmall;GI/DBseparatefilesystems

Software updates Dbnodes require separatedom0 (Linux+fw) anddomU (Linux) patchmgr updates

Exachk Runonce fordom0/cells/ibswitches, runonce foreach VMcluster

Enterprise Manager EM+Exadata plugin +Virtualization Infrastructure plugin


ExadataVMUsage

• Primaryfocusedonconsolidationandisolation• CanonlyruncertifiedOracleLinuxversions–Windows,RedHat,andotherguestoperatingsystemsarenotsupported

• Canvirtualizeotherlightweightproducts– E.g.Lightweightapps,management tools,ETLtools,securitytools,etc.

• Not recommendedforheavyweightapplications– E.g.E-business SuiteorSAPapplication tier– InsteaduseExalogic,Supercluster, orPrivateCloudAppliance

7


Exadata OVMRequirements• Hardware– 2-socketdatabase serverssupported (X2-2andlater)

• Software– Recommend latest Exadata 18.1software• Suppliedsoftware(updatewithpatchmgr - seeMOS888828.1)– domU anddom0 runsameUEKkernel asphysical (e.g.4.1.12-94.8.4.el6uek (ueknano) for18.1.7.0.0)– domU runs sameOracleLinux (OL) asphysical (e.g.OL6.9for18.1.7.0.0)– dom0 runsOracleVMServer (OVS) 3.x(e.g.OVS 3.4.4for18.1.7.0.0)

– GridInfrastructure/Database• Recommend18c,12.2.0.1,or12.1.0.2withlatestquarterlyupdate• Supported 18c,12.2.0.1,12.1.0.2,or11.2.0.4

8


Exadata SecurityIsolationRecommendations• EachVMRACclusterhasitsownExadatagriddisksandASMDiskGroups– SettingUpOracleASM-ScopedSecurityonOracleExadataStorageServers

• 802.1QVLANTagging forClientandManagementEthernetNetworks– Dbnodes configuredw/OEDAduringdeployment (requirespre-deployment switchconfig)

– Orconfiguremanuallypost-deployment• Clientnetwork - MOS2018550.1 Managementnetwork- MOS2090345.1

• InfiniBandPartitioningwithPKEYsforExadataPrivateNetwork– OSandInfiniBandswitches configuredw/OEDAduringdeployment

• StorageServeradministrationisolationthroughExaCLI

9


ExadataOVMSizingRecommendations• UseReferenceArchitectureSizingTooltodetermineCPUs,memory,diskspaceneededbyeachdatabase– Sizingevaluation shouldbedonepriortodeployment sinceOEDAwilldeployyourdesiredVMconfiguration inanautomated andsimplemanner.

– Changescanbemadepostdeployment, butrequiresmanymoresteps– SizingapproachdoesnotreallychangeexceptforaccommodatingDOM0,andadditional systemresourcesperVM

– Sizingtoolcurrentlydoesnotsizevirtualsystems– Consider dom0memoryandCPUusage insizing

10


MemorySizingRecommendations• Cannotover-provisionphysicalmemory– SumofallVM+dom0memoryusedcannotexceedphysicalmemory

• dom0memorysizing– Recommendleavingasdefaultof8GB– ChangememoryonlywhenrecommendedbyOracle.ChangerequiresrestartingDom0andDomUs

• VMmemorysizing– Minimum16GBperVMtosupport starterdatabase,plusOS,Java,GI/ASM,etc.• IncreaseVMmemoryforlargerdatabaseoradditionaldatabases(DBprocesses,PGA,SGA)

– Maximum720GBforsingleVM– VMmemorycannot bechangedonline(noballooning),memoryresizerequiresreboot– OEDAVMtemplatedefaults(Adjustableatconfig time)• Small– 16GB;Medium– 32GB;Large– 64GB

11


CPUSizingRecommendations• CPUover-provisioning ispossible– ButworkloadperformanceconflictscanariseifallVMsbecomefullyactive– Dom0allocated2cores(4vCPUs)

• MinimumperVMis1core(2vCPUs)– 1vCPU==1hyper-thread;1core==2hyper-threads==2vCPUs

• MaximumperVMperDBServerisnumberofcoresminus 2fordom0– E.g.:forX7-2,maximumperVMperDBServeris46cores(48totalminus2fordom0)

• NumberofvCPUsassigned toaVMcanbechangedonline• OEDAVMtemplate defaults (Adjustableatconfig time)– Small– 2cores(4vCPUs);Medium– 4cores(8vCPUs);Large– 8cores(16vCPUs)

12


LocalDiskSizingRecommendations• TotallocaldiskspaceforVMsis1.6TB(X4andlater),3.7TBwithdiskexpansionkit• DiskspaceusedperVMatdeploymentdependsonVMsizeselectedinOEDA– Small190GB;Medium210GB;Large230GB (selectable butnot adjustable inOEDA)• 70GBsystem(rootsys1/sys2,swap)• 100GBsoftwarehomes(50GBGIhome,50GBDBhome)• User/u01– Smalltemplate20GB;Mediumtemplate40GB;Largetemplate60GB

– Actualallocated space fordomU diskimages initially muchlowerduetosparsenessandshareable reflinks,butwillgrowwithdomU useassharedspacediverges andbecomes lesssparse• Over-provisioningdiskmaycauseunpredictableout-of-spaceerrorsinsideVMsifdom0spaceisexhausted• RestoringVMbackupwillreduce(mayeliminate)spacesavings(i.e.relyingonover-provisioningmayprevent fullVMrestore)• Longlived/prodVMsshouldbudgetforfullspaceallocation(assumenobenefitfromsparsenessandshareablereflinks)• Shortlivedtest/devVMscanassume100GBallocation

• DomU localspacecanbeextendedafterinitialdeploymentbyaddinglocaldiskimages– Additionally,domU spacecanbeextendedwithsharedstorage(e.g.ACFS,DBFS,externalNFS)foruser/app files– AvoidsharedstorageforOracle/Linuxbinaries/config files.Access/networkissuesmaycausesystemcrashorhang.

13


DatabaseServerDiskExpansion

• 2socketdatabaseservershave8diskbays,only4arepopulatedoutofthefactory• VirtualMachinesneedmorestorageonthedatabaseservers• X7-2,X6-2, andX5-2databaseserversnowsupport8x600GBHDDs– Onlytwosupportedconfigurations: 4drivesor8drives– Serverswillshipwithonly4drivesoutofthefactory,customerscanadd4moreharddrivesinthefield

• Minimumsoftwareversion– ExadataStorageSoftware12.1.2.3.0

14


Exadata StorageRecommendation• DATA/RECOsizeforinitialVMclustersshouldconsiderfutureVMadditions– Usingallspace initiallywillrequireshrinkingexisting DATA/RECO beforeaddingnew

• SpreadDATA/RECOforeachVMclusteracrossalldisksonallcells– BydefaultnoDBFSdiskgroup

• EnableASM-ScopedSecuritytolimitgriddiskaccess

15

VM Cluster Clusternodes Griddisks(DATA/RECO forall clustersonalldisksinallcells)

clu1 db01vm01db02vm01

DATAC1_CD_{00..11}_cel01 RECOC1_CD_{00..11}_cel01DATAC1_CD_{00..11}_cel02 RECOC1_CD_{00..11}_cel02DATAC1_CD_{00..11}_cel03 RECOC1_CD_{00..11}_cel03

clu2 db01vm02db02vm02

DATAC2_CD_{00..11}_cel01 RECOC2_CD_{00..11}_cel01DATAC2_CD_{00..11}_cel02 RECOC2_CD_{00..11}_cel02DATAC2_CD_{00..11}_cel03 RECOC2_CD_{00..11}_cel03

Copyright©2018, Oracleand/oritsaffiliates.Allrightsreserved.| 16

Hardware X2-2 X3-2 X4-2 X5-2 X6-2 X7-2

VMs

MaxVMsper databaseserver 8

Mem

ory Physicalpernode(default/max)

72GB144GB

256GB512GB

256GB512GB

256GB768GB

256GB1.5TB

384GB1.5TB

MinperdomU 16GBmin+additionalDBsorappmemory

MaxforDatabaseServer(excludingDom0) 96GB 464GB 720GB

OEDAtemplatedefaults Small– 16GB;Medium– 32GB;Large– 64GB(adjustableatconfig time)

CPU*

Cores* pernode 12 16 24 36 44 48

MinperVM 1core(2vCPUs)

MaxperVM Coresminus2 (dom0assigned2cores/4vCPUs)

OEDAtemplatedefaults Small– 2cores;Medium– 4cores;Large– 8cores(adjustableatconfig time)

Disk

TotalusablediskpernodeforalldomUs 700GB 1.6TB 1.6TB(3.7TBw/DBStorageExpansionKit)

UseddiskperdomU atdeployment(basedonOEDAtemplates)

Small190GB;Medium210GB;Large230GB(not adjustableatconfig time)

Actualallocated spacefordomU diskimagesinitially muchlowerduetosparsenessandshareablereflinks,butwillgrowwithdomUuseassharedspacedivergesandbecomeslesssparse,hencebudgetforthesevalueswhensizing.

*1core=1OCPU=2hyper-threads=2vCPUs

DeploymentSpecificationsandLimits


DeploymentOverview• OEDAistheonlytoolthatshouldbeusedtocreateVMsonExadata1. CreateconfigurationwithOEDAConfigurationTool2. PreparecustomerenvironmentforOEDAdeployment– ConfigureDNS,configureswitches forVLANs(ifnecessary)

3. PrepareExadatasystemforOEDAdeployment– switch_to_ovm.sh; reclaimdisks.sh; applyElasticConfig.sh

4. DeploysystemwithOEDADeploymentToolNote:OSVLANconfig canbedonebyOEDAorpost deployment (MOS2018550.1)

17


OEDAConfigurationToolAdvancedNetworkConfiguration

• EthernetNetwork802.1QVLANTagging– ForOVMdefineVM-specific VLANIDsinClusterconfigurationpages lateron

– Ethernet switches (customerandCisco)musthaveVLANtagconfigurationdonebeforeOEDAdeployment

• InfiniBandNetworkPartitioningwithPKEYS

18


OEDAConfigurationToolIdentifyNodes

• ScreentodecideOVMorPhysical– AllOVM– AllPhysical– SomeOVM,somephysical

19


OEDAConfigurationToolDefineClusters

• Decide– NumberofVMclusters tocreate– Dbnodes andCells thatwillmakeupthoseVMclusters• Recommendusingallcells

• Whatisa“VMcluster?”– 1ormoreuserdomainsondifferentdatabase serversrunningOracleGI/RAC,eachaccessing thesamesharedExadatastoragemanagedbyASM.

20


OEDAConfigurationToolClusterConfiguration

• EachVMclusterhasitsownconfigurationpage– VMsize(memory,CPU)– Exadata softwareversion– Networkingconfig– OSusersandgroups– GI/DBversionandlocation– Starterdatabase config– ASMdiskgroupconfig

21


OEDAConfigurationTool

• VirtualGuestsize– DefineCPUandMemoryconfiguredduringdeployment

– Adjustbychangingdefaults– 1vCPU==1hyper-thread– 1core==2hyper-threads==2vCPUs

– /u01“localdisk”size isfixed• Small20GB;Medium40GB;Large60GB• GI/DBhomesareseparatefs(not/u01)

22

ClusterConfiguration


OEDAConfigurationToolClusterConfiguration

• GridinfrastructureinstalledineachVM(griddisks“owned”byaVMcluster)– Cluster1- DATAC1/RECOC1acrossallcells– Cluster2- DATAC2/RECOC2acrossallcells– Considerfutureclusterswhensizing– DBFSnotconfigured– ASM-ScopedSecuritypermitsaclustertoaccessonlyitsowngriddisks.AvailablewithAdvancedbutton.

23


OEDAConfigurationToolClusterAdvancedNetworkConfiguration

• EthernetVLANIDandIPdetails– Toseparate EthernettrafficacrossVMs,usedistinct VLANIDandIPinfoforeachcluster

• InfiniBandPKEYandIPdetails– TypicallyjustuseOEDAdefaults– ComputeClusternetworkfordbnode-to-dbnode RACtraffic.SeparatesIBtrafficbyusingdistinctClusterPKEYandIPsubnet foreachcluster.

– Storagenetworkfordbnode-to-cellorcell-to-celltraffic- samePKEY/subnet forallclusters

24


OEDAConfigurationToolReviewandEdit

• ThispagelistsallnetworkdetailsforeachVMguest inallVMclusters

25


OEDAConfigurationToolInstallationTemplate

• VerifypropersettingsforallVMclustersinInstallationTemplatesotheenvironmentcanproperlyconfiguredbeforedeployment(DNS,switches,VLANs,etc.).

26


OEDAConfigurationToolNetworkRequirements

27

Component Domain Network Example hostname

Databaseservers

dom0(one per databaseserver)

Mgmteth0 dm01dbadm01

MgmtILOM dm01dbadm01-ilom

domU(oneormoreperdatabase server)

Mgmteth0 dm01dbadm01vm01

Client bondeth0 dm01client01vm01

ClientVIP dm01client01vm01-vip

Client SCAN dm01vm01-scan

Privateib dm01dbadm01vm01-priv1

Storageservers(same asphysical)

Mgmteth0 dm01celadm01

MgmtILOM dm01celadm01-ilom

Privateib dm01celadm01-priv1

Switches(same asphysical) Mgmt eth0 dm01sw-*


ExadataOVMBasicMaintenance• RefertoExadata DatabaseMaintenanceGuide:Managing OracleVMDomainsonOracleExadata DatabaseMachine– ShowRunningDomains,Monitoring, Startup,Shutdown– DisablingUserDomainAutomatic Start–ModifyMemory,CPU,localdiskspace inauserdomain– Remove/Create RACVMCluster– ExpandOracleRACVMcluster– CreateUserDomainwithoutGridInfrastructure(e.g.AppVM)–MovingaUserDomaintoaDifferentDatabaseServer– Deleting aUserDomainfromanOracleRACVMCluster– Runningexachk

28


ExadataOVMBasicMaintenance– BackingUpandRestoringOracleDatabases onOracleVMUserDomains– CreatingOracleVMOracleRACClusters– CreatingOracleVMwithoutGIandDatabaseforApps– AddorDropOracleRACnodes inOracleVM– Expanding /EXAVMIMAGESonUserDomainsafterDatabase ServerDiskExpansion– Implementing Tagged VLANInterfaces– Implementing InfiniBand Partitioning acrossOVMRACClustersonOracleExadata– BackinguptheManagement Domain(dom0)andUserDomains (domU)inanOracleVirtualServerDeployment

–MigratingaBareMetalOracleRACClustertoanOVMRACCluster

29


OEDACLItoPerformMaintenanceOperations• OEDAcommandlineoedacli (DocID2293678.1)– FirstreleasedwithOEDAAugust2017– alwaysusemostrecenttoget latest/greatest– Relies onoriginalOEDAconfiguration/XML fileusedfordeployment

• Supportedpost-deploymentoperationswithVMs(asofSeptember2017)– Add/Remove node– Add/Remove database– Add/Remove database home– Add/Remove storagecell– ResizeASMdiskgroup– UpgradeOracleClusterware 12.1.0.2to12.2.0.1

30


ExadataOVMMigration• Dynamicoronlinemethodtochangephysicaltovirtual– DataGuardorbackupscanbeusedtomovedatabases – minimumdowntime– Convertonenodeorsubset ofnodestovirtualatatime

• MigratinganexistingphysicalExadataracktousevirtualrequires– Backingupexisting databases, redeploying existingHWwithOEDAandthenRestoringDatabases

– Duplicating thedatabases toexisting Exadata OVMconfiguration– Ifmovingfromsourcetoanewtarget,standardExadata migrationpracticesstillapply.RefertoBestPracticesforMigratingtoExadataDatabaseMachine

31


Exadata OVMMigration• Dynamicoronlinemethodtochangephysicaltovirtualusinganyoftheproceduresbelow– MigratetoOVMRACclusterusingtheexistingbaremetalOracleRACclusterwithzerodowntime– MigratetoOVMRACclusterbycreatinganewOVMRACclusterwithminimaldowntime– MigratetoOVMRACclusterusingOracleDataGuardwithminimaldowntime– MigratetoOVMRACclusterusingRMANbackupandrestorewithcompletedowntime

• Forrequirementsanddetailedsteps,refertoMyOracleSupportnote2099488.1:Migration ofaBaremetalRACclustertoanOVMRACclusteronExadata

32


Backup/RestoreofVirtualizedEnvironment• Dom0– Standardbackup/restorepracticestoexternal• DomU – TwoMethods– BackupfromDom0:SnapshottheVMimageandbackupsnapshotexternally– BackupfromDomU:StandardOSbackup/restorepracticesapply– Ifover-provisioning localdiskspace- RestoringVMbackupwillreduce(mayeliminate)spacesavings(i.e.relyingonover-provisioningmaypreventfullVMrestore)

• Databasebackups/restore:StandardExadataMAAbackup/restorepracticesstillapplywithExadata orwithZFSStorage• Refer toExadataMaintenanceGuide:– BackinguptheManagementDomain(dom0)andUserDomains(domU)inanOracleVirtualServerDeployment

– RecoveringinanOracleVirtualServerDeployment

33


UpdatingSoftwareComponenttoupdate

Method

Storageservers Same asphysical - runpatchmgr fromanyserverwithssh access toallcells, oruse StorageServer Cloud Scale SoftwareUpdate feature(startingin 18.1).

InfiniBandswitches

Same asphysical - runpatchmgr fromdom0with ssh access toallswitches.

Databaseserver –dom0

Runpatchmgr fromanyserverwith ssh access toalldom0s. Dom0update upgrades databaseserver firmware. Dom0 reboot requires restartof alllocal domUs. DomU software not updatedduring dom0 update. Dom0/domU donot havetorunsame version, although specific updateordering mayberequired (see 888828.1).

Databaseserver –domU

Runpatchmgr fromanyserverwith ssh access toalldomUs. Typically done onaper-VM clusterbasis (e.g.vm01on allnodes, then vm02, etc.),orupdate allVMson aserver beforemoving tonext.

Grid Infrastructure/Database

Standard upgrade andpatchingmethods apply, maintained onaper-VM clusterscope. GI/DBhomes should bemounted disk images, like initial deployment. 12.2upgradeMOS2111010.1.

34


HealthChecksandMonitoring• Exachk runsinDom0andDomU (cells andIBswitches checksrunwithDom0)– Runinonedom0foralldom0s,cells,switches– RuninonedomU ofeach VMclusterforalldomUs,GI/DBofthatcluster

• EMMonitoringsupport(MOS1967701.1)– RequiresEMFramework12.1.0.4(recommend12.1.0.5),Exadata Plugin12.1.0.6,VIPlugin12.1.0.1.

• Exawatcher runsinDom0andDomU• Database/GImonitoringpracticesstillapply• Considerations– Dom0-specificutilities(xmtop)– Dom0isnotsizedtoaccommodateEMorcustomagents– OracleVMManagernot supportedonExadata

35


• IncludesVMs/DB/GI/ASM

VMprovisioning onVirtualized Exadatainvolves reliable,

automated, &scheduledmassdeployment ofRAC

Cluster

• IncludingDB/GI/ASM

Create /delete RACCluster

• IncludesDB/GI/ASM

Scaleup /down RACCluster byadding or

removing VMs

36

EMSupportforExadataVirtualizationProvisioning

IncreaseOperationalEfficiencybyDeployingRACClusterFasteronVirtualizedExadata


Exadata MAA/HA• ExadataMAAfailure/repairpracticesstillapplicable.RefertoMAABestPracticesforOracleExadataDatabaseMachine• OVMLiveMigrationisnotsupported– useRACtomoveworkloadsbetweennodes

37


ResourceManagement• Exadata ResourceManagementpracticesstillapply– Exadata IOandflashresourcemanagement areallapplicable anduseful

• WithinVMsandwithinacluster,databaseresourcemanagementpracticesstillapply– CPUcountstillneeds tobesetatthedatabase instance level formultiple databases inaVM.Recommended min=2

• Nolocaldiskresourcemanagementandprioritization– IOintensiveworkloadsshouldnotuse localdisks– ForhigherIOperformanceandbandwidth,useACFSorDBFSonExadataorNFS.

38

exadata and ovm best practices - oracle.com · –change memory only when recommended by oracle....

Documents