zdlra deep dive
TRANSCRIPT
BASLE BERN BRUGG DÜSSELDORF FRANKFURT A.M. FREIBURG I.BR. GENEVA
HAMBURG COPENHAGEN LAUSANNE MUNICH STUTTGART VIENNA ZURICH
ZDLRA – Deep Dive
Daniele Massimi – Principal Consultant
BE-IMS
Agenda
ZDLRA - in Action2 21.06.2017
1. ZDLRA – Functionality under the hood
2. Installation and Configuration from bottom up
3. Have a look inside – (just particularities)
4. From protected Databases to backups
5. Backup and Recovery Capabilities
6. Recovery Appliance Views and Utilities
7. Conclusion
Who am I
ZDLRA - in Action3 21.06.2017
Daniele Massimi, Trivadis AG (CH, Bern)Principal Consultant [email protected]
Seit 2012 bei Trivadis IMS (Infrastructure Service Management)
Oracle DBA seit 2000 (Exadata > 6 Jahre)
Infrastruktur-Architekturen, Operations & Administration, Upgrades and Migration, High Availability, Backup & Recovery, Virtualizierung
Engineered Systems (Exadata, ODA, ZDLRA, PCA)
Trainer (Oracle Architectur and Internal, 12c New Features, Exadata)
Oracle CertificationsOracle Certified Master (11g)Oracle Certified Professional (8i – 12c)Oracle Certified Expert (RAC)Oracle Implementation Specialist (Exadata, OVM)
Our company.
© Trivadis – The Company4 21.06.2017
Trivadis is a market leader in IT consulting, system integration, solution engineering
and the provision of IT services focusing on and
technologies
in Switzerland, Germany, Austria and Denmark. We offer our services in the following
strategic business fields:
Trivadis Services takes over the interactive operation of your IT systems.
O P E R A T I O N
ZDLRA - in Action5 21.06.2017
ZDLRA
Functionality under the Hood
Common problems with Oracle backups
ZDLRA - in Action6 21.06.2017
RPO depends on archive backup frequency (many minutes or hours)
Backup window and nightly batches compete for resources
High network bandwidth usage for full backups
Low-speed backups and restores
Backup and restore success ratio penalized by shared infrastructure
Critical databases require Data Guard for transaction safety
– Additional storage space, license and computation required
Expiration of backup set on ML: need to crosscheck and delete expired
Lack of backup validation
…
Recovery Appliance Architecture
ZDLRA - in Action7 21.06.2017
Source: Oracle
Minimal Backup Overhead
8 21.06.2017
Traditional Backup
– Read/Write for Disk/Tape Backup
– Deduplication
– Compression
– Validation
– Deletion
– Merging
All on database host
Degrade performance
Recovery Appliance
– Offloading operations
– Delta Push
– Delta Store
Source: Oracle
Delta Push
ZDLRA - in Action9 21.06.2017
Delta Push is a highly optimized form of source-
side optimization
– Through RMAN block change tracking
– No reading of unchanged data
Two operations on the protected Database
– Incremental "forever" backup
– Real time redo transport
One-time full backup as prerequirement
Afterwards Incremental forever backup
Validate incoming Backups against corrupted
physical blocks
Compress the Backups using special block-level
Algorithm
Backuped Database
Changed Data
Less data, less I/O, less network traffic
Source: Oracle
Eliminate data loss – Real Time redo transport
ZDLRA - in Action10 21.06.2017
Real Time redo transport
– Data Guard like but asynchronous
– Changes out of Logbuffer transfered
– Validate and writes to a stage area
Redolog switch on database
– RA converts the staged redo data to a
compressed archived redolog backup
– Writes this archlog backup to catalog
– Ready for future restores
Explicit Archivelog backup no more necessary
Reduces RPO to (near) zero
Source: Oracle
Delta Store – the “brains” of the Recovery Appliance
ZDLRA - in Action11 21.06.2017
Backup
Totality of all Database Backup in a
Storage Location
validates the incoming blocks
compresses, indexes and stores
deduplicate, less storage usage
High number of virtual full backups
Higher recover window
Delta Store
Day NIncr
Day 1 Virtual Full
Day N Virtual Full
Day 1Incr
Day 0Full
Source: Oracle
Delta Store – efficient restore/recover
ZDLRA - in Action12 21.06.2017
Restore
Creates a Virtual Full Database
Backups
No need of transfer and apply
incremental backups during restore
operations
Roll forward with restored redologs
Uses Exadata performance and
scalability
Delta Store
Day 0Full
Day 1Incr
Day NIncr
Day ‘N’ Full Backup
Source: Oracle
Delta Pools
ZDLRA - in Action13 21.06.2017
Chunks inside the Delta Store
Backups will be indexed and stored inside the Delta Pools
Set of datafiles blocks from which RA constucts Virtual Full Backups
Each backuped datafile will have his own delta Pool
Automated delta pool space management
– Protection Policy will be inherited to database retention policy
– Delete automatically expired or obseleted Backups
– Periodically reorganising for performance improving
– Delta pool optimization when old blocks are deleted and new incremental Backup
arriving
Source: Oracle
Delta Push – how it works (1/2)
ZDLRA - in Action14 21.06.2017
Clients address the HTTP Servlet on ZDLRA
RMAN ships incremental Backup to Staging Area
Data will be validated and Metadata will be stored on RAC Database
Data Blocks of Backupsets will be indexed and stored on Delta Store
Source: Oracle
Delta Push – how it works (2/2)
ZDLRA - in Action15 21.06.2017
Optionally Real Time Redo Transport could be activated
ZDLRA will be using Data Guard Technology (RFS on RA)
Validated Redo Blocks will be stored on Delta Store
Archive Logs will be generated whenever a Log Switch happen on Prod DB
Optionally you can replicate or copy to a remote ZDLRA or Tape
Source: Oracle
Restore – how it works
ZDLRA - in Action16 21.06.2017
Clients address the HTTP Servlet on ZDLRA
ZDLRA re-assemble and validate "virtual Backupsets"
Data Blocks will be shipped back to Client
Source: Oracle
Policy-Based Database Protection as a Service
ZDLRA Workshop - Introduction17 21.06.2017
Standardized
protection policies
– Recovery window
– Retention
– Replication
Gold Policy – Customer Critical
Disk: 45 daysTape: 90 days
Silver Policy – Internal Critical
Disk: 30 daysTape: 45 days
Bronze Policy - Test/Dev
Disk: 15 daysTape: 30 days
Protection Policies Attributes
ZDLRA Workshop - Protection Policies18 21.06.2017
Storage Location Recovery Appliance storage location for storing backups
Recovery Window Disk Tape recovery window goal for the protected database
Recovery Window Tape Tape recovery window goal for the protected database
Max Retention Policy maximum length of time that the Recovery Appliance retains
backups for databases that use this retention policy
Guaranteed Copy guaranteed copy setting, which determines whether backups
protected by this policy must be copied to tape or replicated before being considered
for deletion
Polling Policy optional backup polling policy that determines whether Recovery
Appliance polls a storage location for backups or deletion
Unprotected Window maximum acceptable difference between the current time
and the latest time that the database can be restored
Recovery Window – Index Efficiency (1/2)
ZDLRA - in Action19 21.06.2017
Only new version of Data Blocks will be backed up (inc level 1)
Recovery windows controls deletion policy of the blocks
Main scope is to maintain the restorability within the recovery windows
Old and no more needed blocks will be deleted
If enough space is available the Recovery Windows can be overachieved
Source: Oracle
Recovery Window – Index Efficiency (2/2)
ZDLRA - in Action20 21.06.2017
Once the old blocks was deleted, we will have a defragmentation within the
storage
This situation will provide random I/O instead of sequential I/O Performance
degradation
Intermittently the RA will automatically reorganize the blocks
Implicitly all touched blocks will be phisical and logical checked on his integrity
Source: Oracle
Storage Locations
ZDLRA - in Action21 21.06.2017
Main Storage Location are the Container within the ASM Diskgroup
It is possible to have more than one Storage Location in ASM
Backup Polling Locations are Storage Location outside from ZDLRA (e.g. NFS Mount)
• Backups are placed directly to this location without interacting with ZDLRA
• With Polling Policies you can define how often and where
is the polling directory
• Once all requirement meets e.g. backup related to a
protected database a copy to the RA will be created
• After copying process the protected database will be delete
the backup Automatically from polling directorySource: Oracle
ZDLRA - in Action22 21.06.2017
Installation and Configuration
from bottom up
Installation Steps
ZDLRA - in Action23 21.06.2017
Create the ZDLRA configuration using
the latest OEDA
Start the Re-Imaging and Setup from the first
Computing Node (like an Exadata Setup)
[root@zdldbadm01 linux-x64]# ./install.sh -cf ./WorkDir/zdl-zdl.xml –r 1-16
Installation Steps
ZDLRA - in Action24 21.06.2017
Recovery Appliance specific Installation Steps
[root@zdldbadm01 install]# ./ra_install.pl --help
ra_install.pl - Recovery Appliance Installer
You must be logged in as root to run this command.
All steps should be run successfully for install.
Usage: ./ra_install.pl --help | --step=STEP_NUMBER {--install|--uninstall} [--
stdout]
Options:
--help: displays this help message
--step=number: Which step to run
--stdout: Print all messages to stdout rather then the log file
--install: Installs the step [Default]
--uninstall: Uninstalls the step
Step Numbers:
Step 1 - Validation and Configuration Prep
Step 2 - OS Setup
Step 3 - Oracle User Setup
Step 4 - DBFS Setup
Step 5 - Tape Backup configuration
Step 6 - ZDLRA DB Backup Setup
Step 7 - Enable ZDLRA Services
Configuration Steps
ZDLRA - in Action25 21.06.2017
Deployment of the Agents on the compute Nodes
Recovery Appliance Discovery
Installation of the Backup Module on any Clients
[oracle@odax30 zdlra]$ java -jar ra_install.jar -dbUser ra_orcl02 -dbPass
manager -host zdlingest-scan -port 1521 -serviceName zdlra -walletDir
/u01/app/oracle/admin/orcl02/xdb_wallet -libDir $ORACLE_HOME/lib
Recovery Appliance Install Tool, build 2015-11-03
Recovery Appliance credentials are valid.
Recovery Appliance wallet created in directory
/u01/app/oracle/admin/orcl02/xdb_wallet.
Recovery Appliance initialization file
/u01/app/oracle/product/12.1.0.2/dbhome_1/dbs/raorcl02.ora created.
Downloading Recovery Appliance Software Library from file ra_linux64.zip.
Downloaded 27189305 bytes in 5 seconds. Transfer rate was 5437861 bytes/second.
Download complete.
[oracle@odax30 zdlra]$
ZDLRA - in Action26 21.06.2017
Have a look inside
(just particularities)
On the computing Nodes
ZDLRA - in Action27 21.06.2017
Metadata Database
– RAC Database (also used for Recovery Catalog (VPC))
– Non Container Database
– DBFS Mount-Points for Backup and Replication purpose
– DBFS added as Cluster Resources
– HTTP Servlet inside the database will be started for communication between the
clients and the ZDLRA (dbms_ra.startup_recovery_appliance)
oracle@zdldbadm01:~/ [+ASM1] df -h | grep dbfs
957M 120K 956M 1% /dbfs_repdbfs
[email protected]:/ 300G 23G 278G 8% /dbfs_obdbfs
SQL> exec dbms_ra.startup_recovery_appliance;
PL/SQL procedure successfully completed.
On the computing Nodes
ZDLRA - in Action28 21.06.2017
– Three new FS for internally purposes are provided
[oracle@zdldbadm01 ~]$ df -h
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/VGExaDb-LVDbSys1
30G 15G 14G 54% /
tmpfs 253G 53M 253G 1% /dev/shm
/dev/sda1 504M 72M 407M 15% /boot
/dev/mapper/VGExaDb-LVDbOra1
99G 59G 36G 63% /u01
/dev/mapper/VGExaDb-LVRADump
296G 191M 281G 1% /radump Dump destination for internal Stuff
/dev/mapper/VGExaDb-LVRABackup
99G 351M 94G 1% /rabackup
/dev/mapper/VGExaDb-LVOBIndex
296G 191M 281G 1% /obindex
957M 120K 956M 1% /dbfs_repdbfs
[email protected]:/ 300G 4.6M 300G 1% /dbfs_obdbfs
On the computing Nodes
ZDLRA - in Action29 21.06.2017
Backup of Metadata Database
– Done using Export of RASYS Schema every 2 Hours
– Backup Location on DBFS FS: /dbfs_obdbfs/OSB/ra_export
– Scheduling via "anacron"[root@zdldbadm01 ~]# cat /etc/anacrontab
SHELL=/bin/sh
PATH=/sbin:/bin:/usr/sbin:/usr/bin
MAILTO=root
# the maximal random delay added to the base delay of the jobs
#RANDOM_DELAY=45
# the jobs will be started during the following hours only
START_HOURS_RANGE=3-22
#period in days delay in minutes job-identifier command
1 65 cron.daily nice run-parts /etc/cron.daily
7 70 cron.weekly nice run-parts /etc/cron.weekly
30 75 cron.monthly nice run-parts /etc/cron.monthly
[root@zdldbadm01 cron.hourly]# ls -l /etc/cron.hourly/
total 4
-rwx------ 1 root root 409 Nov 10 2015 0anacron
lrwxrwxrwx 1 root root 46 Aug 25 16:20 ra_export.pl -> /opt/oracle.RecoveryAppliance/bin/ra_export.pl
On the computing Nodes
ZDLRA - in Action30 21.06.2017
ASM Configuration
– Two Diskgroups DELTA (for Backups), CATALOG (Metadatas)
– New special sub-Directory CONTAINER for storing Container Files which
contains Delta Pools and Delta Stores
– Each Container File as a size of 2 TB ASMCMD [+delta/zdlra] > ls -l
Type Redund Striped Time Sys Name
Y ARCHIVELOG/
Y CONTAINER/
Y DATAFILE/
Y TEMPFILE/
ASMCMD [+delta/zdlra/CONTAINER] > ls -s
Block_Size Blocks Bytes Space Name
512 4294959104 2199019061248 4398059094016 con.365.920823681
512 4294959104 2199019061248 4398059094016 con.366.920823687
512 4294959104 2199019061248 4398059094016 con.367.920823699
...
On the cell Nodes
ZDLRA - in Action31 21.06.2017
Fortunatelly nothing special ☺
Same appearance as in Exadata Storage Cell
During the Installation the "Write Back" Option will be enabled automatically for the
Flashcache
Little difference on the Grid Disks compared with the Exadata, only 10 Grid Disks
instead of 12 for the CATALOG ASM Disk Group
ZDLRA - in Action32 21.06.2017
From protected Databases
to backups
Protected Databases
ZDLRA - in Action33 21.06.2017
Once the Environment meets all the requirements we can add the databases to the
ZDLRA
The requirement are:
• Network connectivity
• Backup Module is installed for every Oracle Home
• Definition of Protection Policies are created (Retention Time)
Block Change Tracking not mandatory, but recommended !!!
Two Methods are possible:
• Enterprise Manager
• Manually DBMS_RA PL/SQL Package
Prerequisite for Adding Protected Database
(EM and manually)
ZDLRA - in Action34 21.06.2017
Create an VPD User on the Metadata Database
Grant the Create Session privilege
SQL> create user ra_orcl03 identified by <pwd>;
SQL> grant create session to ra_orcl03;
Adding Protected Database with EM
ZDLRA - in Action35 21.06.2017
Add Protected Database on Recovery Appliance Page
– Choose the Protection Policy
Adding Protected Database with EM
ZDLRA - in Action36 21.06.2017
Configuring Backup Settings– Register database
– Add RMAN configuration (configure...)
– Enabling Redo Transport
(add parameter to spfile and restart DB)
Scheduling Protected Database with EM
ZDLRA - in Action37 21.06.2017
Protected Database is ready for backing up
– We need a Schedule
Scheduling Protected Database with EM
ZDLRA - in Action38 21.06.2017
Configure the repeating interval for
the Incremental-Forever Backups
Configure the Archive Log Backups and
deletion rule
Scheduling Protected Database with EM
ZDLRA - in Action39 21.06.2017
Also useful as Script Generator here the Script is not modifiable !!!
Scheduling Protected Database with EM
ZDLRA - in Action40 21.06.2017
Output of an Backup Schedule on EM13c
Adding Protected Database manually
ZDLRA - in Action41 21.06.2017
Add Database for the protected database
Grant access to Virtual Private Catalog
Configure the RMAN Channel for ZDLRA
begin
dbms_ra.add_db (
db_unique_name => 'orcl03',
protection_policy_name => 'bronze',
reserved_space => '250G');
end;
begin
dbms_ra.grant_db_access (
db_unique_name => 'orcl03',
username => 'ra_orcl03');
end;
CONFIGURE CHANNEL DEVICE TYPE 'SBT_TAPE' FORMAT '%d_%U' PARMS
"SBT_LIBRARY=/u01/app/oracle/product/12.1.0.2/dbhome_1/lib/libra.so,
ENV=(RA_WALLET='location=file:/u01/app/oracle/product/12.1.0.2/dbhome_1/dbs/ra_wallet
credential_alias=zdlingest-scan:1521/zdlra:dedicated')";
Adding Protected Database manually
ZDLRA - in Action42 21.06.2017
Enable Real-Time Redo Transport
Restart the Database
ALTER SYSTEM SET REDO_TRANSPORT_USER=ra_orcl03 SCOPE=BOTH;
[oracle@odax30 ~]$ srvctl stop database -db orcl03
[oracle@odax30 ~]$ srvctl start database -db orcl03
Execute Backup manually
ZDLRA - in Action43 21.06.2017
Backups works like in traditional way
Only "forever" incremental level 1 backups should be schedule
Level 0 Backup are of course always possible
It works also with old fashion syntax soft link from libra.so to libobk.so is
needed !!!
Scheduling is also free selectable, EM is recommended !
[oracle@odax30 ~]$ rman target / catalog ra_orcl02/manager@zdlingest-scan:1521/zdlra
...
connected to target database: ORCL02 (DBID=3592015831)
connected to recovery catalog database
RMAN> backup incremental level 1 database;
run {
allocate channel c1 type sbt_tape;
backup incremental level 1 database plus archivelog delete input;
}
...
ORA-27211: Failed to load Media Management Library
ZDLRA - in Action44 21.06.2017
Copy to Tape
Purpose of Copy to Tape
ZDLRA - in Action45 21.06.2017
Tape Infrastructure provides a second or third copy of your backups
Tape Infrastructure could be located on different Data Center DR capability !
Tape backups are "cheaper" for long-term Storage
High efficiency is given in combination with ZDLRA
Oracle Secure Backup just pre-installed on ZDLRA
Setting up Copy to Tape
ZDLRA - in Action46 21.06.2017
Configuring Media Management Library• Library Name
• Number of Drives
• Number of Restore Drives
• Media Manager Location (libobk.so)
Setting up Copy to Tape Template
ZDLRA - in Action47 21.06.2017
Template helps to define specific attributes
• Backup types
• Database or Protection Policies
• Recovery Windows will be inherited
• Number of copies
• Runtime Window
• Schedule
Executing Copy to Tape
ZDLRA - in Action48 21.06.2017
With the scheduling will be iniatiate the copy to tape process
backupset of defined database or protection policy will be copied to tape
Long Term Backup should be done with copy_backup procedure
Restores of backupset from tape will be retrieved transparently !!!
ZDLRA - in Action49 21.06.2017
Recovery Capabilities
General statement about Recoveries of protected
Databases
ZDLRA - in Action50 21.06.2017
Recoveries of protected databases can be done with EM or manually
With EM several Recovery-Scenarios are possible
All Recovery scenarios with EM are full automated
(e.g. Creation of auxiliary Databases, etc.)
EM generates RMAN Scripts for each Scenario
with the possibility of adaption
Possibility of selection between full and point-in-time
recoveries
Duplicate Database (for standby) need some preparation
(e.g. FS-Structure (admin tree))
Protected Database Recovery with EM
ZDLRA - in Action51 21.06.2017
From database Page under Availability Backup & Recovery Perform Recovery
Protected Database Recovery with EM
ZDLRA - in Action52 21.06.2017
Choose the needed Recovery Scenario
... You can choose an other restore
location if needed...
Protected Database Recovery with EM
ZDLRA - in Action53 21.06.2017
A schedule will be created
... And you can adapt the script if needed and submit the job...
Protected Database Recovery with EM
ZDLRA - in Action54 21.06.2017
The Job can be monitored by clicking the "View Job" button...
Protected Database Recovery manually
ZDLRA - in Action55 21.06.2017
Restore can be done also manually
– of course db must be mounted before...
– RMAN Client should be configured otherwise parameter are needed (e.g. ENV=...)
Or a point-in-time Recovery
run {
restore database;
recover database;
alter database open;
}
run {
set until time '08.09.2016 08:45:00';
restore database;
recover database;
alter database open resetlogs;
}
Real Time Redo Transport – what happens really ?
ZDLRA - in Action56 21.06.2017
We tested what happens really when the protected database crash…
▪ We did a backup of a protected database with enabled real time redo transport
▪ then we shutdown the database inconsistently (abort)… before we checked the actual SCN
▪ the ZDLRA figure out the crash of the protected database and catalog the redo stream from the
staging area as archivelog with the latest SCN he know… that will be latest consistent SCN for
restore and recover the database
RMAN> backup incremental level 1 database plus archivelog delete input;
SQL> select current_scn from v$database;
CURRENT_SCN
-----------
28933241
SQL> shutdown abort
RMAN> list backup of archivelog all;
.
.
1 26 28933197 18-APR-17 28933202 18-APR-17
1 27 28933202 18-APR-17 28933232 18-APR-17
1 28 28933232 18-APR-17 28933246 18-APR-17
ZDLRA - in Action57 21.06.2017
Recovery Appliance Views and
Utilities
Some RA Views
ZDLRA - in Action58 21.06.2017
It is also possible to read the Recovery Appliance Information over Views from Recover
Appliace Metadata Database
View Name Description
RA_DATABASE Information about Protected Databases
RA_DB_ACCESS Describes User Accounts that have access to which protected database
RA_PROTECTION_POLICY Protection Policies defined on the Recovery Appliance
RA_DATABASE_STORAGE_USAGE Storage usage for each protected database
RA_RESTORE_RANGE Describe Restore Range for each protected database
RA_INCIDENT_LOG Describe Recovery Appliance Incidents
RA_REPLICATION_SERVER Replication Server Information (if used)
rastat.pl Utility
ZDLRA - in Action59 21.06.2017
rastat Utility help to evaluate the performance of Recovery Appliance
– NETBACKUP/NETRESTORE
• Measure the network performance
– ASMREAD/ASMWRITE
• Measure the ASM Disk I/O Performance
– CONTAINERREAD/CONTAINERWRITE/CONTAINERALLOC
• Measure the Container Disk I/O Performance
[oracle@zdldbadm01 ~]$ perl /opt/oracle.RecoveryAppliance/client/rastat.pl --test=CONTAINERWRITE --
filesize=2048M --rasys=rasys/manager@zdlingest-scan:1521/zdlra --diskgroup=/:DELTA
RECOVERY APPLIANCE WRITE IO TEST TO CONTAINER FILES
Disk Group: /:DELTA
2048 MB, .75s write IO time, .51s CPU time, 2724.80 MB/s, 68.51% CPU usage
PL/SQL procedure successfully completed.
dbms_ra Package
ZDLRA - in Action60 21.06.2017
dbms_ra provides administration function from inside the RA Database
Ca. 50 Procedure for different types of administration
– Start/Stop of Recovery Appliance
– Add/Delete Protected Databases
– Grants Access to specific User to enable to backup and restore
– Add/Update Protection Policies
– Manage Tape Configuration (if available)
– Add Repliaction Server (if available)
But, the utilization of Enterprise Manager is recommended !!!
ZDLRA - in Action61 21.06.2017
Conclusion
Conclusion
ZDLRA - in Action62 21.06.2017
ZDLRA worked in our Tests as
expected
Very good implementation of well
known Backup and Recovery
Technology on Engineered System
RPO is (near) zero with Real Time
Redo Transport
Even RTO ist much better then
existing systems
Very simple Management with EM
Risk reduced, cost reduced,
qualitiy improved
ZDLRA will find it’s
customer !
Questions and Answers...
Daniele Massimi – Principal Consultant
Tel. +41 58 459 50 92
21.06.2017 ZDLRA - in Action63