oracle maximum availability best practices: oracle ... · –if you are 11.2 use the standard rman...

Copyright©2017, Oracleand/oritsaffiliates.Allrightsreserved.|

OracleMaximumAvailabilityBestPractices:OracleDatabase12cBestPracticesandTechniques

MichaelSmithConsultingMemberofTechnicalStaffOracleDatabaseMAA

October03,2017


SafeHarborStatementThefollowingisintendedtooutlineourgeneralproductdirection.Itisintendedforinformationpurposesonly,andmaynotbeincorporatedintoanycontract.Itisnotacommitmenttodeliveranymaterial,code,orfunctionality,andshouldnotberelieduponinmakingpurchasingdecisions.Thedevelopment,release,andtimingofanyfeaturesorfunctionalitydescribedforOracle’sproductsremainsatthesolediscretionofOracle.

3


ProgramAgenda

OracleMAA

MAABestPractices

LookingForward

1

2

3

4


Percentageofallunplannedoutagesduetopeopleandprocessissues80%

Averagecostofdowntimeperhour

$10M

Averageresolutiontimeperoutage

5

$350K

Source:Ponemon Institute,KPMGTechRiskRadar,ForresterResearch

ImpactofDatabaseDowntime

Averagecostofunplanneddatacenteroutageordisaster

3hours


AlignsOracleSolutionswithCustomerBusinessObjectives

6

OracleMAAReferenceArchitecture

www.oracle.com/goto/maa

ZeroDataLossDRtotheCloudUseCaseZeroDowntime

RACZeroDataLossBackuptotheCloudUseCase

Prod/Departmental

BusinessCritical

Dev,Test,Prod

MissionCritical

BackupandRecovery

Bronze +

ZeroDowntimeHighAvailability

Silver+

ZeroDataLossHAandDR

GOLD

BRONZESILVER

PLATINUM

ZeroDowntimeGoldenGateCloudSvc.

Gold+

ZeroDowntimeMaintenance/Migration

Local&RemoteBackups

Bronze +

Active/ActiveDatabaseClustering+Backup&Recovery

Silver+

RemoteReplicationwithnear-Zerodataloss,fasterfailover

Gold+AdvancedCapabilitiesforZeroApplicationOutagesandZeroDataLoss


ProgramAgenda

OracleMAA

MAABestPractices

LookingForward

1

2

3

7


MAABestPractices

• Wellknown,proven,andtested• EngineeredintoallOracleproducts

–Software–EngineeredSystems–OracleCloud

• Validatedwitheverynewrelease• DocumentedinMAAwhitepapersandkeyOracleSupport(MOS)notes

AlwaysMovingForward

8

Copyright©2017, Oracleand/oritsaffiliates.Allrightsreserved.| 9

AdditionalResources

www.oracle.com/goto/maa www.oracle.com/goto/ha


MAABestPractices

10

ActiveDataGuard


BestDisasterProtection,Real-timeFailover,HighROIActiveDataGuard

PrimaryDatabaseOpenRead-Write

ActiveStandbyDatabaseOpenRead-Only

PrimaryOracleInstance

DatabaseFiles

StandbyOracleInstance

DatabaseFiles

Oracle-awareReplicationContinuousOracleDataValidation


DataGuardConfigurationBestPractices

• Benefit:FastestandeasiestDataGuarddeploymentforyourenvironment• NewmasterMOSnotethatdirectsyoutothebestwaytodeployaDataGuardstandby

• MyOracleSupportNote2275154.1– Ifyouare11.2usethestandardRMANDUPLICATEmethod– Ifyouare12.1.0.2orhigherthenuseRMANrestorefromservicemethod– Ifyou12.2.0.1orhigherandsingleinstanceuseDBCAmethod

DataGuardCreationMadeEasy

12



• Benefit:LearntopushASYNCperformancetoprovidenearzerodataloss• Howtoaccuratelydeterminetransportlag• DiagnosingandtuningASYNCtransport

– Networkperformance– Identifybottlenecks

• DiagnosingreasonsforASYNCtransportlag– UsingAWRtoassesspeakredoratecanbemisleadingduetoaveragesbringdowntherateoverlongerperiodoftime

– Examinethetimespentineachlogtodeterminethepeakredorateonafinerlevel

BestPracticesfor11g/12c– ASYNCTransport

13

MAABestPracticePaper:DataGuardAsynchronousRedo

Transport



• Benefit:Achievezerodatalosswithminimalperformanceimpact• DeepdiveonSYNCperformancetuning

– Testresultsthatillustrateperformancegainswhenusingbestpractices– Forexample,properonlinelogfilesizeswithalargebankingcustomerimprovedperformanceby30%

– FrequentlogswitchesforceacheckpointonthestandbywhichresultsinincreasedI/Otherebyaffectingperformance

– Singlememberstandbyredologplacedonfaststorage

BestPracticesfor11g/12c– SYNCTransport

14

MAABestPracticePaper:DataGuardSynchronousRedo

Transport



• Benefit:AchievebestperformanceforreducedRTO• Howtotunewithexamplesforvariousscenarios

– Tuningusingtopfivewaitevents– Testresultsthatillustrateperformancegainswhenusingbestpractices

• Newinstallationandusageinstructionsforstandbystatspack– RACconsiderationsdocumentedinthepaper– Usingin-memoryASHreportwhenyoucan’tinstallstandbystatspack

• Newupdatecomingsoonthatincludesmultiinstanceredoapply

BestPracticesfor11g/12c– RedoApply

15

MAABestPracticePaper:DataGuard

RedoApply


12.2ActiveDataGuardEnhancements

• In-memoryDBonActive DataGuard– CreateIMCtablesandcolumnsforanalyticsonActiveDataGuard– Canpopulatewithdifferentdatathanproductiondatabase

• Parallel,multi-instancerecovery– Standbyrecovery- utilizesCPUandIOacrossallnodesofRACstandby– StandbywillkeepupwitheventhelargestprimaryRACcluster

Performance

MonthIn-Memory

YearIn-Memory

Standby

16


12.2Multi-InstanceRedoApply• Benefit:ScaleredoapplyandkeepRTOlow• Parallel,multi-instancerecovery:standbywillkeepup

– Standbyrecovery- utilizesCPUandIOacrossallnodesofRACstandby– SomeofourOLTPworkloadtestsonExadata showgreatscalability

17

0200400600800

100012001400

1Instance 2Instances 4Instances 8Instances

OLTPWorkloadStandbyApplyRate

MB/sec


ActiveDataGuardEnhancements

• Benefit:Moreintuitivetuning• SupportsDiagnosticPack

– TakeAWRsnapshots fromstandby(s),storeonprimary– CreateAWRreports,andAWRExtract&LoadofStandbyAWRSnapshots– RunADDMonprimaryusingAWRsnapshotsfromstandby

• SupportsTuningPackandSQLPlanAnalyzer– TuneStandbySQLworkload(onstandby)

• CreateandexecuteSQLTuningtasksonADG(reportswrittentoprimaryoverdblinks)

– OffloadtuningofprimarySQLworkloadtostandby• CreateremoteSQLPerformanceAnalyzertrialstooffloadtestworkloads

– CreateSQLprofilesonprimarythatcanbeappliedonstandby

Manageabilityin12.2

18



• Benefit:Reduceddowntimeforbothunplannedandplanned• Discussiononhowswitchovershavechangedfor12c

– Singlecommand– Newvalidateoptionpriortoswitchover

• Howtoassessyourroletransitiontimingsandwherethetimeisbeingspent

BestPracticefor11g/12c– RoleTransition

19

MAABestPracticePaper:DataGuardSwitchoverand

Failover


12.2ActiveDataGuardEnhancementsFasterFailover

• Read-onlysessionsconnectedtoActiveDataGuardwill– Remainconnectedduringthefailover/switchover

– Becomeread/writeafterActiveDataGuardbecomestheprimary

• Dramaticallyreducesbrownoutduetoreconnectstormafterfailover– AnotherreasontouseActiveDataGuard(insteadofDataGuard)

Read/Write Read

Read/Write

Primary

Standby

Standby

20



• Benefit:Decreaseddowntime• DetaileddiscussionofhowDataGuardhandlesvariousoutages– Testresultsthatshowoutagerepairtimes

– Howtoperformthetestandwhattoexpect

• Beingupdatedwith12.2improvements

BestPracticefor11g/12c– OutageRepair

21

MAABestPracticePaper:Automatic

ResolutionofOutagestoRestoreZeroData

LossProtection


MAABestPractices

22

ApplicationContinuity


End-to-endApplicationAvailability

• Benefit:Seamlessapplicationfailover• Replayin-flightworkonrecoverableerrors• Maskshardware,software,network,storageerrorsandoutages

• 12.1supportsJava(JDBC-Thin,UCP,WebLogic Server,3rd PartyJavaapplicationservers)

• 12.2addssupportforOCI,ODP.NETunmanaged,JDBCThinonXA,Tuxedo,SQL*Plus

• Acchk (partofOrachk)ReplayCoverageAnalysis–identifies%ofSQLcallsthatare[not]coveredbyautomaticreplay

ApplicationContinuityforUnplannedOutage

ApplicationServer

DBServersorDataCenters

1. DatabaseRequest

5.Response

3.Errors,Timeouts

2.DB Calls 4.ApplicationContinuity

23


End-to-endApplicationAvailability

• WhenpatchingDB,wanttomigrateexistingconnectionstoanotherDBinstance• Workswellin12.1withOracleconnectionpools

– WebLogic ActiveGridLink,UCP,OCI,ODP.NETmanagedandunmanaged,OCISessionPool,Tuxedo),and3rd partyAppServersusingUCP:IBMWebSphere,ApacheTomcat,NECWebOTX

– Newworkdirectedtonewconnections;existingsessionsreleasedwhenreturnedtopools

• 12.2makesthisworkevenwhenOracleconnectionpoolsarenot used– JDBCdriverautomaticallyreclaimsconnectionsatsafeplaces

• Alsoavailablewithotherdrivers(OCI,ODP.NET,…)

– Worksagainst11.2and12.1DB• 12.2makesthiseasierbysupportingrelocateorstopofall servicesassociatedwithaPDB,instance,node,orDB

MaskingPlannedDBOutage

ApplicationServer

DBServersorDataCenters

srvctl relocateservice

24


ApplicationResiliencyAttackingtheRootofProblem

• ApplicationsarenotwrittentohandleDBfailures(e.g.,killingbatchjobsbeforepatching)

• DBMS_DISRUPT PL/SQLpackageprovidesanAPItocause“controlled”terminationof:– SessionsandServices– Canselectsomeorallservices,jobs,instances.Alsospecifyfailurefrequency

• Ensuresthatapplicationsarecodedtohandlethistypeoffailure

25


MAABestPractices

26

Multitenant


ArchitectureforconsolidatingdatabasesandsimplifyingoperationsOracleMultitenant

GL OE

AP Self-containedPDBforeachapplication• Portability(viapluggability)• Rapidprovisioning(viaclones)• Applicationsrununchanged• PDBupgradesviaplug/unplug

Sharedmemoryandbackgroundprocesses• Moreapplicationsperserver

CommonoperationsperformedatCDBlevel• Managemanyasone(upgrade,backups,HA)• Granularcontrolwhenappropriate• SimpleDR

PDBs

Root

CDB

MAABestPracticePaper:Database

ConsolidationwithOracleMultitenant

MAAandMultitenant• Solutionsforplanned/unplannedoutages


SinglePDBFailover/Relocation• LossofsinglePDBatprimary

• AllotherPDBsoperatingnormally• StandbycopyofPDBpristine

• Currentsolutions?• FailoverentireCDB?• Restore/RecoverPDB?

28


SinglePDBFailover/Relocation• Usestandbycopyto“unplug”PDB(MOS2088201.1)

• Createmanifestwithdbms_pdb.describe• PluginPDB’sstandbyfilesintoco-locatedCDBwithNOCOPY(quick)

• MigrateServicestonewCDB• Brokerenhancedtoautomateprocess(12.2)

• Sourcecanbeprimaryorstandby• AutomationwillnotcreatethedestinationCDB

29


NormalRuntime

PDB1 PDB2CDB1

Read-WriteCDB1StandbyRead- OnlyDataGuard

CDB2Read-Write

PDB4

PDB2 PDB3PDB1 PDB3

30


Unplug/plugPDB2fromCDB1standbytoCDB2andfailoverapplicationconnectionsPDB2Outage

PDB1

CDB1Read-Write

CDB1StandbyRead- OnlyDataGuard

CDB2Read-Write

PDB4

PDB2PDB1 PDB2 PDB3 PDB3

PDB2

31


OutageEnhancementsin12.2• Per-PDBflashback

• EnableFlashbackatCDBlevel• AllPDBshaveflashbackenabled• RestorePointsbyPDB

• PDBRelocate• Relocatewithnearzerodowntime• NoApplicationOutage

32


MAABestPractices

33

GoldenGate


FlexibleLogicalReplicationOracleGoldenGate

• Zero-downtimemaintenanceandmigrations• Active-Activehighavailability• Heterogeneousreplication,datadistributionandintegration

Source&TargetOracle&Non-OracleDatabase(s)

Target&SourceOracle&Non-OracleDatabase(s)Bi-directional

LAN / WAN / InternetOver TCP/IP

34

Note:MAAforGoldenGate Microservices ArchitectureiscurrentlyWIP


OracleGoldenGate

• TransparentRoleTransitionsinDataGuardConfigurations– NomanualinterventionisrequiredwithFSFOandDGBroker

• Configurationmakesuseof:• OracleGridInfrastructureBundledAgent(XAG)• DBFSforsharedGoldenGate files(trailsandcheckpointfiles)• Rolebasedservices• IntegratedExtract(withHANDLEDLFAILOVERoptionforASYNCDG)• IntegratedReplicat isrecommended

MAABestPractices

35

MAABestPracticePaper:TransparentRoleTransitionswith

DataGuardandOracleGoldenGate


SampleDeployment

36

Observer

PrimaryDatabase StandbyDatabase

RedoTransport(SYNCorASYNC)

IntegratedExtract LogMining

Server

TrailandotherOGGFilesInDBFS

RedoTransport

OCIConnection

FileI/OWarehouse

BidirectionalGoldenGate Replication


SampleDeployment– PostRoleTransition

37

Observer

(OLD)PrimaryDatabase (NEW)PrimaryDatabase

RedoTransport(SYNCorASYNC)

IntegratedExtractLogMining

Server

Trail/Checkpoint/BRFilesInDBFS

LogMiningServer

RedoTransport

OCIConnection

FileI/O

Warehouse

BidirectionalGoldenGate Replication


OracleGoldenGate

• ConfiguredatabaseSTREAMS_POOL_SIZE– (#ofintegratedGGprocesses*1GB)+25%headroom

• Usetheautomaticheartbeattabletomonitorend-to-endlatency– IntroducedinGoldenGate release12.2.0.1

• ForintegratedExtract/Replicat installandrunStreamsPerformanceAdvisor(SPADV)– Showsprocesspercentagesplitbetweenidle,busyandwaiting(flowcontrol)

• UseGoldenGate IntegratedExtractandReplicat PerformanceDiagnosticCollector(MOSnote2262988.1)– Gathersrequireddatafordiagnosingperformanceissuesbyasinglescript.

MAAPerformanceBestPractices

38

MAABestPracticePaper:OracleGoldenGate

PerformanceBestPractices


MAABestPractices

39

Sharding


• Horizontalpartitioningofdataacrossafarmofindependentdatabasesusingashardingkey(i.e.account_id)

• SomeWeb-scaleOLTPapplicationsusedatabase sharding– Avoidscalabilityoravailabilityedgecasesofasinglelargesystemimagedatabase

– Willingtocustomizedatamodelandapplicationstoenabletransactionstobeautomaticallyroutedtotherightshard

• NoSQL databasescanshard,nowOracleDatabaseNativecanshardwithoutlosingalltheadvanceddatabasetechnologies

40

OracleSharding– ADatabaseArchitecturePatternOnegiantdatabasepartitionedinto

manysmalldatabases(shards)


OracleDatabaseSharding– Benefits

41

LinearScalability

… …

FaultTolerant

Nosharedhardwareorsoftwaretoisolatefaults.Shardsmayrun

differentOraclereleases.

GeographicDistribution

Userdefineddataplacementforperformance,availability,DRortomeetregulatoryrequirements.

Addshardsonlinetoscaletransactionsandconcurrentusers.

Onlinerebalance.


DeploymentofaSystem-ManagedSDBwithDataGuard

42

Clients

DataGuardFast-StartFailover

RegionAvailabilty_Domain1

ShardCatalogshardcat_stdby

ShardDirectorshdir3,4

RegionAvailability_Domain2

Shardgroupshgrp2

Shardgroupshgrp1

…

…

ShardDirectorshdir1,2

ShardCatalogshardcat

ConnectionPools

ConnectionPools

Primaries

HAStandbys


OracleSharding– MAAOutageTesting

• OutageofShardCataloghasnoeffectonapplicationperformance

• ShardKeysarecachedwithinthesharddirectors

• MAABestPracticeistoprotectcatalogwithDataGuardMaximumAvailability

MAABestPracticePaper:OracleShardingMAABestPractices



• Outageofsharddirectorsdoesnotaffectarunningconnectionpool

• Connectionpoolcachesrangeofshardkeys/shards

• MAAbestpracticetohave3sharddirectorsperregion



0

500

1000

1500

2000

2500

3000

3500

4000

4500

18:12:50

18:12:53

18:12:56

18:12:59

18:13:02

18:13:05

18:13:08

18:13:11

18:13:14

18:13:17

18:13:20

18:13:23

18:13:26

18:13:29

18:13:32

18:13:35

18:13:38

18:13:41

18:13:44

18:13:47

18:13:50

18:13:53

Tran

sactionsPerSecon

d

FailoverPerformance

Read/Write

ReadOnly

• Failoverofanindividualsharddoesnotaffectapplicationperformanceforremainingshards

• Fastfailoverforbothread/writeandreadonlyconnections

• GenericMAAbestpracticesapplyforsharded environments


MAABestPractices

46

OracleCloud


MAAEvolutionfromOn-PremisestoAutonomous

On-Premises

On-PremisesExadata

ExadataCloud

AutonomousDatabase

47

• Architecture• DatabaseManagement(Tooling)• Configuration,Tuning• LifecycleOperations(Tooling)• ApplicationPerformance

• ChoosingtheSLApolicy• Applicationperformance

• InfrastructureManagement

• Architecture• DatabaseManagement• Configuration,Tuning• Lifecycleoperations• ApplicationPerformance• Infrastructure

Management• Architecture• Configuration,Tuning• DatabaseManagement• LifecycleOperations• ApplicationPerformance

• Blueprints• Feedbackto

products&features

• Blueprints• Exadata isthebest

integratedMAADBplatform

• OracleownsandmanagesthebestintegratedMAADBplatform

• Cloudautomationforprovisioningandlifecycleoperations

• OracleownsandmanagesInfrastructure

• Policydrivendeployments

• MAAIntegratedcloud• FullyautomatedSelf-

Driving,Self-Securing,Self-RepairingDatabase

CustomerOracle


MAAAutomationintheCloud

• MAAmadeeasy• SimpleUI/CLI/RESTinterfaces• Databasesareprovisionedwithoptimalparameterconfigurations

48

Database Deployment MadeEasy

Primary

AD#1

Standby

AD#2SILVER

(HA)

Primary

Region

#1

Standby

Region

#2GO

LD(D

R)

BRONZE

SingleInstance

DB Backup Service

RACSILVER

(HA)


AutonomousDatabaseCloud

49

SLAs:99.995Uptime+RPO=seconds

Exadata,RAC,ActiveDataGuardIntegrated

MAAConfigurationBestPractices

MAALifeCycleOperationsandBestPractices

OngoingMAACloudValidationsandOptimizations

ZERO DatabaseorSystemAdministration

FocusonApplicationDesign,TuningandBusiness

BENEFIT:MaximizeAvailabilityandReduceUserErrors

Lifecycle

Migration Notify

Optimize Manage


ZeroDataLosswithMinimalPerformanceImpact

0

500

1000

1500

2000

2500

1 14 27 40 53 66 79 92 105

118

131

144

157

170

183

196

209

222

235

248

261

274

287

300

Tran

sactionsPerSecon

d

SYNCPerformanceImpact

async

fastsync

sync

ASYNC FASTSYNC SYNC

RedoRate(MB/sec) 14.93 14.48 14.39

BlockChanges/sec(KB/sec) 96.92 94.3 93.86

Txn Rate 2082 2025 2018

%DifferencefromASYNC N/A 97% 97%

50

ExaCS Primary/Standbyindifferentregions


DatabaseFailoverwithMinimalDowntime

0

500

1000

1500

2000

2500

3000

3500

4000

4500

18:12:50

18:12:53

18:12:56

18:12:59

18:13:02

18:13:05

18:13:08

18:13:11

18:13:14

18:13:17

18:13:20

18:13:23

18:13:26

18:13:29

18:13:32

18:13:35

18:13:38

18:13:41

18:13:44

18:13:47

18:13:50

18:13:53

Tran

sactionsPerSecon

d

FailoverPerformance

Read/Write

ReadOnly

FSFO

Thresho

ld-6Second

s

Database

Failover-

8Second

s

• SwingbenchOLTPapplicationperformingmixtureofinserts,updates,anddeletes

• Applicationredorateof15MB/sec

• FastStartfailoverinMaximumAvailabilitymode,FSFOthresholdconfiguredfor6seconds

• Databasefailovertimeof8seconds

51



PlannedMaintenancewithMinimalDowntime

• SwingbenchOLTPapplicationperformingmixtureofinserts,updates,anddeletes

• Applicationredorateof15MB/sec

• Applicationoutageof12secondsduringtheswitchoverprocess

• Totalswitchovertimeofapproximately40seconds

0

500

1000

1500

2000

2500

3000

3500

4000

4500

18:45:50

18:45:53

18:45:56

18:45:59

18:46:02

18:46:05

18:46:08

18:46:11

18:46:14

18:46:17

18:46:20

18:46:23

18:46:26

18:46:29

18:46:32

18:46:35

18:46:38

18:46:41

18:46:44

18:46:47

18:46:50

18:46:53

Tran

sactionsPerSecon

d

SwitchoverPerformance

ReadWrite

ReadOnly

52



AdditionalResources

www.oracle.com/goto/maa www.oracle.com/goto/ha


Q&A


SafeHarborStatementTheprecedingisintendedtooutlineourgeneralproductdirection.Itisintendedforinformationpurposesonly,andmaynotbeincorporatedintoanycontract.Itisnotacommitmenttodeliveranymaterial,code,orfunctionality,andshouldnotberelieduponinmakingpurchasingdecisions.Thedevelopment,release,andtimingofanyfeaturesorfunctionalitydescribedforOracle’sproductsremainsatthesolediscretionofOracle.

55

oracle maximum availability best practices: oracle ... · –if you are 11.2 use the standard rman...

Documents