oracle rman design best practices with data domain...

Dedu

plic

atio

nSt

orag

e

w w w . d a t a d o m a i n . c o m - 2 0 0 7

WHITE PAPER

Oracle RMAN Design Best Practices with Data Domain

2Oracle RMAN Design: Best Practices with Data Domain

D A T A D O M A I N I C o n t e n t s

Contents

IntroductIon . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3

BasIc concepts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3

addItIonal concepts . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3

executIve summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4

oracle Background . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4

termInology . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5

types of Backups . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5

rman defInItIons . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5

asm . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8

data domaIn product Background . . . . . . . . . . . . . . . . 8

models . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8

software releases . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8

deployment optIons . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8

How It works . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8

advantages of data domaIn

In an oracle envIronment . . . . . . . . . . . . . . . . . . . . . . 8

IntegratIon . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9

plannIng . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9

Important rman optIons . . . . . . . . . . . . . . . . . . . . . . . 10

oracle recommendatIons . . . . . . . . . . . . . . . . . . . . . . . . 12

purpose . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12

QuestIons and answers . . . . . . . . . . . . . . . . . . . . . . . . 12

summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13

appendIx a – trouBlesHootIng . . . . . . . . . . . . . . . . . . . . 14

usIng oracle enterprIse manager . . . . . . . . . . . . . . 14

from tHe command lIne . . . . . . . . . . . . . . . . . . . . . . . 19

AcknowledgementsMuch of the information contained herein is taken from copyrighted Oracle documentation that is publicly available. All rights by Oracle to any of this documentation is, of course, retained and acknowledged.


IntroductionThis technical note gives information on the configuration, setup and tuning of Oracle RMAN 9.x/10.x with Data Domain deduplica-tion storage systems. It is expected that the reader of this paper has a passing knowledge of both Oracle management practices as well as familiarity with backup and recovery techniques for Oracle. A working knowledge of UNIX/Linux and/or Microsoft Windows and the networking components of each is also required. Finally a basic understanding of the setup and management of a Data Domain System is also required.

This perhaps seems like a great deal of background required but will actually serve more as a shortcut to understand some of the shorthand used in this paper. Some theory and background will be presented, but generally only to further explain the recommendations provided.

Oracle has now released the next genera-tion database, 11g. As acceptance of this new technology grows, this document will be updated and revised with additional information as necessary. For now, it is recommended that 11g users follow the advice for 10g.

Basic ConceptsA Data Domain System is an appliance which is typically used as a target for backup or archived data. The characteristics that make it good for this include:

• Supportforanyconventionalnearlineapplicationthroughgeneralized support for Network Attached Storage (NAS) interfaces over Ethernet, a Virtual Tape Library (VTL) interface option over Fibre Channel, and product-specific interfaces such as NetBackup OpenStorage;

• High-speed,inlinededuplicationusingsmall,variable-sizedsequences to identify redundancies;

• IntegrateddataprotectiontechnologiessuchasRAID-6,post-backup data verification, and periodic validation checks of existing data sets;

• Deduplicatedreplicationtoautomatedisasterrecovery(DR).

While a Data Domain appliance offers NAS interfaces, its use as a general purpose file system is generally not recommended because it has not been tuned for small, random-access I/O requests typical of database transactions; it is better tuned for applications that read or write a whole file at a time. On a Windows network, the Data Domain system presents as a network file server accessible via Microsoft’s Common Internet Filesystem (CIFS) protocol. To a UNIX or Linux network, the Data Domain system presents as a network file server accessible via NFS. A single Data Domain system can present both interfaces simultaneously.

Oracle’s Recovery Manager (RMAN) is a built-in tool that allows the Database Administrator (DBA) to easily backup and recover the data in an Oracle database. RMAN handles the coordination required to ensure that transaction integrity is preserved and sufficient information is maintained to recover the database to any appropriate point. RMAN can create backupsets that comprise as much or as little recovery information as the DBA requires but usu-ally includes information from the database datafiles, controlfiles, and redo logfiles. By default RMAN is set up to backup database files to a tape, a disk, a Windows UNC path or UNIX NFS mount path. There are tuning parameters that can be applied for optimum performance and compression.

Figure 1 - Example of Traditional Backup Environment

Additional conceptsOther useful concepts to be aware of prior to getting too far into this paper deal with the details of accessing a Microsoft Windows shared drive, either by mapping a drive a referencing a Universal Naming Convention (UNC) name and setting up and accessing NFS mounts on UNIX/Linux boxes. In either case, sufficient information should be provided during setup to ensure that RMAN has access to the neces-sary path when it runs. Frequently, user credentials are forgotten during this step so ensure they are correct prior to starting RMAN.

Much the same applies to NFS for a UNIX/Linux environment. Appropriate edits to the /etc/fstab or /etc/vfstab files will be required

Finally, credential checking also occurs on the Data Domain system. Ensure that these are setup appropriately.

Example of Traditional Backup Environment

Clients OracleServer

PrimaryStorage

Backup/MediaServer

StagingDisk

TapeLibrary

Offsite Storage

WAN


Executive SummaryTo save reading for technicians already well briefed on both Oracle and Data Domain, here is a summary of the suggested best practices. Explanations and reasoning for these are included further on.

PArAmETEr or oPTion SETTing

Type of backup Full

“FiLESPErSET” 1

“mAXoPEnFiLES” 1

Parallelism or number of Channels

As many as the system can bear. Usually ranges between 3 and 5.

“ConTroFiLE AUToBACKUP”

On

Encryption No

Compression (“AS ComPrESSED” clause)

No

nFS moUnT oPTionS SETTing

Sun Solaris: rw,noac,hard,rsize=32768,wsize=32768, llock

Linux and other UniX: rw,noac,hard,rsize=32768,wsize=32768, nolock

Access to mount point: Restrict access to a mount point to only the UNIX machines actually running RMAN and/or any standby, recovery systems.

replication Yes, use Data Domain system to replicate backupsets to remote DR site

miSCELLAnEoUS oPTionS SETTing

Use of multiple Data Domain systems to backup very large databases

Yes, but be sure to carefully configure channels to ensure the same datafiles are always backed up to the same Data Domain system.

multiplex archivelogs to the Data Domain system

Multiple copies of archivelogs provide added protection. Specifying the Data Domain system as an optional destination increases protection and gets the logs to the system even sooner than waiting until the RMAN backup runs.

Figure 2 - Summary of Best Practice Settings

Figure 3 - rmAn Environment with Data Domain

RMAN Environment with Data Domain

Clients OracleServer

BackupOracleServer

PrimaryStorage

Archiveto Tape

OnsiteRetentionStorage

WAN

OffsiteDisaster Recovery

Storage

Oracle BackgroundOracle environments range from the small, experimental installations to the large production environments that run multi-billion dollar corporations. Because of this large difference in scale, a single recommendation is seldom applicable across all environments. However,somebasicbestpracticesformaintainingareliableOracledatabase exist1.

run in archivelog modeIn this mode, Oracle does not delete the redo logfiles that track changes to the database. Instead it “archives” these files to secondary (and sometimes tertiary) locations when Oracle performs its periodic “logswitch”. Without this history of archived logfiles, recovery of an Oracle database to a point prior to an error is significantly constrained.

Keep the production oracle data on extremely reliable storageEnough has been written about various means for storing Oracle data on SAN or NAS storage that little more need be said here.

Spread the oracle data across many disk devicesSimilar to the previous item, it is important to follow Oracle’s and/or the storage vendor’s documentation for optimal storage of Oracle database files.

Perform full oracle backups on a regular basis.HavingmultiplerecoveryoptionsisimportanttotheOracleDBAasthe size and importance of an Oracle database grows. Starting in Oracle 10g, Oracle introduced the Flashback Recovery Area (FRA). This is a special storage area set aside to hold previous copies of the Oracle data to ease and simplify some recovery operations. For the purpose of this paper, we will focus more on the use of RMAN to create backup information outside the FRA that can be used for recovery.It is important that all DBA’s develop a practice for doing regular backups, the base of which is a full backup. Periodic practice recoveries are also recommended to ensure all the pieces, processes, and skills are ready and available for the inevitable day when a critical recovery is needed. We will get into the various types and levels of Oracle backups later in this paper.

Keep archivelogs and/or incremental backups in between each full backupKeeping a history of archivelogs is critical to maximizing recovery opportunities. A backupset consisting of a full database backup along with all the subsequent archivelogs allows that DBA to recover the database to any point desired. Incremental backups also provide in-betweenrecoveryopportunities.However,manyDBA’seschewthe use of incrementals in favor of nightly fulls if possible since the recovery involving an incremental is a bit more complicated and time consuming. One benefit of storing to deduplicated storage is that when doing nightly fulls, the new storage consumed is typically less than an incremental would require.

1 many useful documents exist on the oracle website, too many to be cited here . However, one especially useful document is the rman Best practices which is included verbatim later in this document .


BackupsetA backupset is a set of one-or-more files that is written by Oracle as the output of a backup operation. The DBA can characterize a backupset in a number of different ways through different naming facilities as well as a separate tagging facility. What is important to remember is that the files in a backupset can hold interleaved data from a number of different Oracle data files unless an image backup is being performed. In this latter case, RMAN backs up each Oracle datafile to a separate file within the backupset.

Figure 4 - Example: rmAn “list backup” command

rmAn ParametersSeveral options exist with RMAN that every DBA should be aware of. These effect how RMAN behaves when performing its backup duties. Some of the most important are outlined here because they affect how RMAN will interact with a Data Domain deduplication storage system. For more complete description of these options, refer to Oracle’s documentation.

Keep one or more copies of the data on disk in separate locationsUnfortunately we live in an uncertain world, and keeping only a single set of backup files runs the risk that something could happen to make it unavailable or unusable. Therefore it is highly recom-mended that the backup practice arranges to keep multiple copies of the backup files in various locations.

Use RMAN to help organize the various backups and simplify possible recoveryTo simplify all these different operations, copies, and locations of files, the use of RMAN is highly recommended. It assists the DBA in keeping track of the state of all these various backup images as well as their locations. It also assists in making the decision that particular backup information has become obsolete and can be safely retired. Finally, it makes recovery for the DBA much simpler since RMAN keeps track of all the files required for certain recovery operations the DBA might require.

Terminology

Types of backupsRMAN supports several different kinds of backups.

• Afullbackup,asitsnameimplies,isabackupoftheentiredatabase.

• Apartialbackupcanbeofasingleormultipleportionsofthedatabase but not necessarily everything.

• AnincrementalbackupusesafacilitywithinOracletotrackchanges to the database that occur between backups to backup only those specific pieces of the database that have been altered.

• Thefinaltypeofbackupisanimagebackupwhichmakesprecisecopies of the Oracle database files but the result looks just like a copy of the files when viewed from the operating system.

In each of these backups RMAN processes the data to be backed up and consolidates it into one or more files called a backupset. We will spend some time talking about backupsets in just a little while.

RMAN definitionsOracle RMAN has a set of definitions and parameters that are unique to it and must be understood by a DBA (and a reader of this paper) to get very far with RMAN.


FILESPERSET [=] integer

When used with commands that create backupsets, specifies the maximum number of files to include in each backupset created. By default, RMAN divides files among backupsets in order to make optimal use of channel resources. The number of files to be backed up is divided by the number of channels. If the resultislessthan64,thenitisthenumberoffilesplacedineachbackupset.Otherwise,64files will be placed in each backupset.

maxsetsizeThis parameter defines the largest size a single file within the backupset can reach.

MAXSETSIZE [ CLEAR | TO [sizeSpec | UNLIMITED ]

Specifies the maximum size of each backup set created on a channel. By default MAXSETSIZE is set to UNLIMITED, meaning that it is unconstrained.

CompressionRMAN provides two types of compression. First, and mostly invisible to the DBA is unused-block elimination which was introduced with Oracle 10g. If RMAN encounters a data block in a file being backed up that is not in use, it will not send it to the backup stream at all unless a certain set of conditions exists (see Oracle’s RMAN docu-mentation for details). The second type of compression tells RMAN to apply a compression algorithm (BZIP2) to the data being written to the backupset. This results in less disk space being used at the cost of significantly greater CPU consumption during the backup operation.

To quote from Oracle’s own documentation:

note: If you are backing up to tape and your tape device performs its own compression, you should not use both RMAN backup-set compression and the media manager vendor’s compres-sion. In most instances you will get better results using the media manager’s compression. See the discussion of tuning RMAN’s tape backup performance in Oracle Database Backup and Recovery Advanced User’s Guide for details.

We will discuss settings for compression later in this paper.

EncryptionRMAN provides for backupsets to be encrypted to increase security. Any backupset can be encrypted except “image” backupsets. Automatic encryption requires the configuration and setup of key management functions such as provided by the Oracle Encryption Wallet.

Channels and parallelismRMAN backs up Oracle datafiles using a series of separate processes (i.e., separate running programs and/or threads) which all run in par-allel. Typically, while one process is waiting on I/O to complete for a given set of reads/writes, another process can be performing similar tasks against a separate set of files. By keeping these “channels” running simultaneously but against data stored on different storage

Some RMAN parameters are global and persistent, others apply to only specific tasks. A listing of the global parameters can be generated by the “show all” RMAN command:

RMAN> show all;

RMAN configuration parameters are:CONFIGURE RETENTION POLICY TO REDUNDANCY 1; # defaultCONFIGURE BACKUP OPTIMIZATION OFF; # defaultCONFIGURE DEFAULT DEVICE TYPE TO DISK; # defaultCONFIGURE CONTROLFILE AUTOBACKUP ON;

CONFIGURE CONTROLFILE AUTOBACKUP FORMAT FOR DEVICE TYPE DISK TO ‘%F’; # default

CONFIGURE DEVICE TYPE DISK PARALLELISM 3 BACKUP TYPE TO BACKUPSET;

CONFIGURE DATAFILE BACKUP COPIES FOR DEVICE TYPE DISK TO 1; # default

CONFIGURE ARCHIVELOG BACKUP COPIES FOR DEVICE TYPE DISK TO 1; # default

CONFIGURE CHANNEL DEVICE TYPE DISK FORMAT ‘/dd580a/oracle/%U’;

CONFIGURE CHANNEL 1 DEVICE TYPE DISK MAXOPENFILES 1 FORMAT ‘/dd580a/backup/oracle/backup_%U’;

CONFIGURE CHANNEL 2 DEVICE TYPE DISK MAXOPENFILES 1 FORMAT

‘/dd580a/backup/oracle/backup_%U’;CONFIGURE MAXSETSIZE TO UNLIMITED;CONFIGURE ENCRYPTION FOR DATABASE OFF; # defaultCONFIGURE ENCRYPTION ALGORITHM ‘AES128’; # defaultCONFIGURE ARCHIVELOG DELETION POLICY TO NONE; # default

CONFIGURE SNAPSHOT CONTROLFILE NAME TO ‘/home/oracle/oracle/product/10.2.0/db_1/dbs/snapcf_orcl.f’; # default

Figure 5 - Example: rmAn “show all” command

maxopenfilesThis parameter controls the number of Oracle datafiles that RMAN can have open for reading at any one time.

MAXOPENFILES = integer

Controls the maximum number of input files that a BACKUP command can have open atanygiventime(thedefaultis8).Usethisparameter to prevent “Too many open files” error messages when backing up a large number of files into a single backup set.

An important effect of this parameter is on the amount of system memory required by RMAN while a backup is occurring. To minimize the number of I/O buffers that RMAN allocates it is important to keep MAXOPENFILES to the smallest value possible while still allowing good performance2.

FiLESPErSETThis option, a parameter on the “begin backup” RMAN command, tells RMAN how many Oracle data files it can combine into a single backup file. In other words, it controls the multiplexing of Oracle data into the backupsets.

2 see oracle metalink note: 73354 .1 RMAN: I/O Slaves and Memory Usage


devices, the overall RMAN backup can complete in less time than if the entire process ran sequentially against a single file at a time.

This example explicitly parallelizes a backup operation by specifying to Oracle Enterprise Manager that multiple parallel processes should be used:

Figure 6 - oEm Backup Settings

ByspecifyingtoOEMthatParallelismshouldbe4and(onadiffer-ent screen) that FILESPERSET should be set to 1, the resulting script that OEM will run looks like:

Figure 7 - oEm example script


described will apply equally well to all backup scenarios and recom-mendations in this paper hold for all presently available (and currently planned) versions of Data Domain Operating System (DD OS).

Deployment optionsA given Data Domain system is usually deployed in one of two modes; either as a Network Attached Storage (NAS) device or as a VTL. We earlier stated we were not going to cover the VTL deployment. This is because RMAN does not have the capability to talk directly to a VTL; it relies on an intermediate backup application to provide that function.

Therefore this paper focuses on the use of a Data Domain dedupli-cation storage system with RMAN when RMAN is using the system directly as either an NFS device or a CIFS device. Both of these options involve attaching the Data Domain system to an Ethernet, giving it an appropriate IP address on the network, and configuring the rest of the network information as required to deploy a typical file server for the given NFS/CIFS environment. It is presumed the reader is already familiar with the procedures for accomplishing this.

How it worksThe Data Domain system runs a proprietary file system that accepts backup data from its network interfaces and writes this data to disk. In the process, it examines the incoming datastreams and eliminates the data segments that have been seen before and writes small segment indicators in their place. It also applies an LZ-type (or optionally GZ) compression algorithm to the data just before writing it to disk.

These characteristics, and several others, make the Data Domain system an ideal target for Oracle RMAN.

For further understanding on these mechanisms, please refer to the SISL white paper on the Data Domain Web site at http://www.datadomain.com/resources/701300000008qvV_lp.html.

Advantages of Data Domain in an Oracle environmentBy eliminating redundant data segments, the Data Domain system allows many more backups to be stored and managed than would normally be possible for a traditional storage server. While completely new data has to be written to disk whenever discovered, the variable-length segment deduplication capability of the Data Domain system makes finding identical segments extremely efficient.

For example, a full backup of an Oracle database will be stored ontheDataDomainsysteminapproximately40%ofthespaceit originally occupied. Why? First, even in a full database backup, some data segments are repeated and eliminated. Not many, but let’ssay20%.ThedataisthencompressedusinganLZ-compressionalgorithm to further compress the data which usually gets about 2Xcompression(or50%spacereduction).Sothesetwonumbersgenerallyyieldapproximately60%reduction.

The next time a full backup is performed, a much higher percentage ofthedatahasalreadybeenseen;oftenwellover90%dependingon the database. So even though the DBA requested a full database backup, the Data Domain system only has to write a small fraction of the whole database to disk this time. This effect accumulates

The DBA can either define a global setting for parallelism and allow RMAN to balance the I/O load as best it can, or the DBA can manually define these channels to get even better performance than Oracle is able to obtain on its own.

Figure 8 - Channel Allocation

overall effectsSome of these parameters have significant effects on how well Oracle RMAN behaves when backing up to a Data Domain system. This includes the use of the Data Domain system as a direct target for RMAN (either using an NFS mount or a CIFS share) or if the files created are subsequently copied to the Data Domain system using a backup application or a operating system utilities (e.g., “cp” or “dd” for UNIX/Linux or “copy” for Windows). These effects, and combinations thereof, will be discussed in a later section.

ASMOracle’s Advanced Storage Manager (ASM) was introduced in Oracle 9i and further enhanced and extended in 10g. For the purposes of this paper, ASM actually is involved very little except in related areas of managing I/O to the Oracle datafiles as efficiently as possible. However,sinceASMisadatabaseuntoitself,itisimportantforaDBA to keep in mind the need to maintain effective backup and protection of the information required by ASM to provide its service to the rest of the Oracle environment.

Data Domain Product Background

ModelsData Domain systems are available in a variety of system configura-tions. Fortunately, these configurations generally all run the same software (with the restriction that newer systems sometimes have to run newer software) so different Data Domain systems models have little or no effect on the operational characteristics within an RMAN environment.

Software ReleasesMuch the same can be said of the software releases for the line of Data Domain systems. This software has been designed to be agnostic to the data being stored on it; all data presented is effec-tively deduplicated, compressed, and protected. The characteristics

Channel Allocation

channel ch1RecoveryManager

ServerSession

ServerSession channel ch1


Database size 500GB Combined size of all Oracle datafiles and logfiles

rate of change 5%perday Involves100%changeoflogfilesand10%changetodatabasefiles3.

Daily backup size 500GB + 25GB Datafiles + archivelogs

retention period 3 weeks

Figure 9 - Sizing considerations

With5%changerateeachday,multipliedby2toaccountforthechange being in both the datafiles and the archive logfiles (perhaps a bit more in UNDO depending on configuration), the amount of new data presented is about 350GB/week. Assume the combination ofde-duplicationandlocal(LZ)compressionprovides4x(orabout12.5GB of actual new data needing to be stored), then this example with full daily backups would look like:

DAy oF wEEK

CUmULATivE AmoUnT

BACKED UP

CUmULATivE STorAgE

ConSUmEDovErALL

ComPrESSion

Sunday 500GB 125GB 4X

monday Sunday + 550GB = 1050GB

137.5GB 7.5X

Tuesday Monday + 550GB =1600GB

150GB 10.5X

wednesday 2150GB 162.5GB 13.2X

Thursday 2700GB 175GB 15.4X

Friday 3250GB 187.5GB 17.3X

Saturday 3800GB 200GB 19X

Sunday # 2 4350GB 212.5GB 20.4X

Sunday # 3 8200GB 300GB 27.3X

Sunday # 4 12,050GB 387.5GB 31.1X

Figure 10 - Cumulative storage example

In this example, the customer now has three full weeks of backups kept on disk with each image being a fully recoverable image. Over 12TB of data has been sent to the Data Domain system but due to the effects of deduplication and compression, the actual space consumedisheldtolessthan400GB.Thisexamplealsoassumesthe database in question does not grow in size which is unlikely, but a reasonable rate of growth would only change the numbers by a small percentage.The space used will start to level out as the reten-tion period for backups is reached and RMAN starts letting them expire. This point is determined by the DBA with various retention settings within RMAN.

over time so that the compression ratio continues to improve until a plateau is reached where the new data being written is nearly the same as the amount of old data being retired. A better example is presented later.

The integration of the Data Domain system into an Oracle/RMAN environment is quite seamless since the system presents as either a NFS or CIFS shared storage server. Oracle/RMAN already supports and documents this type of installation for effective RMAN storage. The ability of the Data Domain system to store several weeks or months of full Oracle backups provides the DBA a backup and recovery environment with great flexibility and protection while occupying a minimal amount of space and using a minimal amount of resources.

For all critical Oracle environments, it has become a best practice to replicate the production Oracle data to a secondary recovery location. The DBA has many options to choose from including tools built-in to Oracle itself, solutions offered by enterprise storage providers, as well as third party solutions. The Data Domain system also offers replication facilities which are extremely easy to deploy. The primary advantage of Data Domain system replication is the fact that the data being moved is all de-duplicated and compressed prior to moving. Also, as the RMAN backup process proceeds, each file in the backupset is rapidly replicated to the remote site as soon as it is closed allowing the overall “time-to-DR” to be minimized since much of the replication takes place even while the RMAN backup process is still active. Allowing the Data Domain system to provide this service reduces CPU and network requirements that would be required if either Oracle itself or the enterprise storage were to perform similar service. And, of course, the replicated image occupies the same minimal footprint as the primary backup image, keeping overall costs to a minimum.

Integration

PlanningPlanning to use a Data Domain system as a target for an RMAN backup is relatively straightforward since the system just appears as diskstorage.However,considerationforspacerequired,networkthroughput, replication effects, recovery operations, etc. must be taken into account to ensure that the entire “Oracle + Oracle/RMAN + operating system + network + Data Domain” system fulfills the customer’s requirements for backup windows, recovery time objectives, desired retention, and support for disaster recovery.

CapacityHowmuchspaceisrequired?Thisisobviouslyanimportantquestion to answer when some of the databases being backed up are growing to multiple terabytes in size. A detailed analysis of this question can be developed by a knowledgeable Data Domain System Engineer who can apply extensive profiles and models developed through thousands of real-world deployments. But a simple approach might go something like this.

3 of course, oracle “housekeeping” will create a great deal of change to the logfiles without necessarily making any changes to the database, so these numbers are unlikely to be terribly precise .


Multiplexing of backup data was introduced to address the performance issues affecting backup to tape. It was important that the backup process keep the tape drive fed with data so as to avoid “shoe shining” where the tape would have to stop and backup when it had to wait for more data to be presented. Backing up to a disk device does not suffer this problem so multiplexing has minimal value when disk is used.

Important RMAN options

Type of backup: Full, Incremental, ImageBest Practice recommendation:It is generally recognized that Oracle recovery processes work best against full backups. A full backup has all of the most recent information and is fastest to recover.

Incremental backups were developed because some databases are just too large to make full backups every night. With deduplication technologies removing redundant copies of database segments, thisisnowlessofanissue.However,afullbackuptoaDataDomain system does still involve moving all the data over a network connection. Some databases may simply be too large to send over a network in an acceptable period of time. If this is the case, incre-mental nightly backups are the only available solution with periodic full backups when the opportunity exists; usually weekends.

If these are still not acceptable, Oracle presents other techniques which involve storing data on mirrored disks which can be separated and backups made from the mirror. Oracle also offers standby database solutions where the secondary database can be frozen and backups made from it. These advanced solutions are beyond the scopeofthispaper.However,aDataDomainsystemstillmakesanexcellent backup repository for even these extremely large databases.

FILESPERSETFILESPERSET is the option Oracle RMAN uses to control multiplexing of the data files into the backupsets.

Best Practice recommendation:Specify FILESPERSET = 1 or use image copy when backing up to the Data Domain deduplication storage system.

As described earlier, FILESPERSET controls how many datafiles are written to a particular file within the backupset. Since the files involved are read in parallel and asynchronously, the blocks written to the backupset are likely to be in slightly different order each time.

What does this mean when using a Data Domain deduplication storage system?

While this does not affect the ability to recover the data from the backupset, it does reduce the ability for the Data Domain system to effectively de-duplicate the incoming data. Identical data to previous backups may come in but not be recognized as already-written because it is aligned slightly differently or is in a slightly different order.

The space used will start to level out as the retention period for backups is reached and RMAN starts letting them expire. This point is determined by the DBA with various retention settings within RMAN.

Of course, the best and most accurate way to make a determination on the required storage and the expected compression effects would be to do a pilot using real customer data. Because the deployment of the Data Domain system is so simple, and additional copies of RMAN backups can be scripted directly into RMAN, a real-world test is relatively simple and non-disruptive.

NetworkingHowfastcanthedatamove?Obviously,thefasterthenetworkandthe greater the number of network paths, the faster data can move. However,thereareotherchokepointsinthecommunicationpaththat must be considered. Many CPU’s cannot pump data out on a network at full network bandwidths. Some network environments impose several switch hops between the RMAN server and the backup device. Finally, a given model of Data Domain system has a limit as to how many streams and how much data can be written into it at any given instant. These last numbers are available from any Data Domain system engineer, but currently range between 50-220MB/sec and 5-20 streams depending on model. As a useful datapoint, a single gigabit Ethernet can move about 125 MB/s, so to see more aggregate performance on a Data Domain system that is rated faster, more than one link and write stream would be needed simultaneously.

SecurityBackup security to a Data Domain system is usually maintained by controlling access to the share points on the Data Domain system. By limiting access to specific directories on the Data Domain system, the DBA can ensure access control to the backup files. Further controls by setting UID/GID credentials (in a UNIX environment) or SID credentials (for Windows) can add additional control.

note: Full SID control in a Windows environment is not yet provided inaDataDomainsystemasofDDOSVersion4.3.Butevenwith the existing limitations, access to the Oracle backup data can be restricted to only the Oracle machines themselves and the SID under which RMAN runs so the limitation should not be an issue.

Best Practice recommendation:It is recommended that the shares (both NFS and CIFS) accessed directly by Oracle RMAN be restricted to only the servers actually running Oracle RMAN and any servers that might be used for a potential recovery in the future.

Optimizing DeduplicationWhen dealing with any deduplication technology, a few key factors need to be kept in mind. Most importantly is the fact that multiplex-ing of backup datastreams is detrimental to deduplication. This is because all deduplication techniques rely on being able to quickly identify repeated data patterns. When backup data is multiplexed, there is the probability that the data presented to the backup device is in slightly different patterns each time. This will cause degradation in the deduplication performance.


Multiplexing Archivelog DestinationsBest Practice recommendation:Including the Data Domain system as a destination for log archiving provides two advantages. Primarily, a second copy of the archivelogs provides additional protection in the event that the primary copy is damaged or lost. Since the archivelogs will eventually be backed up to the Data Domain system as part of an RMAN full backup, this simply means that the files are getting to the Data Domain system sooner than when RMAN runs. The files are identical (even though RMAN renames them) as long as the FILESPERSET=1 guideline mentioned earlier is followed so the Data Domain system will detect the common data sequences and deduplicate the second copy so that little or no additional disk space is consumed on the Data Domain system. Additional protection without additional disk space consumption, something to consider taking advantage of.

Marking this second destination to Oracle as “OPTIONAL” (as opposed to “MANDATORY”) will allow Oracle to continue running in the event that the Data Domain system is unavailable for service or other duties.

The changes required to achieve this setup are well documented in various Oracle administration manuals.

Very Large Database ConsiderationsBest Practice recommendation:In the case where the database being backed up is consuming tens or hundreds of terabytes, it will be necessary to utilize multiple Data Domain systems to provide backup resources. This is easy to do, but some planning of the deployment is required.

Keep in mind that the Data Domain system can only deduplicate data that it has seen before and has something to compare to. This implies that compression is maximized when the same data is presented to the Data Domain system repeatedly so it has a baseline and can store only the new data presented.

If a large database, composed of possibly thousands of datafiles, is backed up to a series of Data Domain systems, RMAN (if left to its own defaults) will try to spread the backup load across the various targets. Since this is somewhat random and therefore non-determin-istic, RMAN might well send some datafiles to Data Domain system 1 on the first backup but to Data Domain system 2 on a subsequent backup. Obviously this is not optimal.

Remembering the earlier discussion of RMAN channels, we utilize channels to provide the necessary controls. In general, we would define one-or-more channels to each Data Domain system and then constrain each channel to backing up specific datafiles.

MaxopenfilesBest Practice recommendation:Specify MAXOPENFILES = 1 for each channel defined. This will ensure that each RMAN channel only reads from a single file at any one time. Setting FILESPERSET=1 should implicitly cause MAXOPENFILES to be one, but setting it explicitly guarantees there is no doubt.

Parallelism (aka, Channels)Best Practice recommendation:Define additional channels to increase the number of parallel backup processes running. With FILESPERSET=1 (specified per backup job) and MAXOPENFILES=1 (specified per channel) combin-ing to limit the multiplexing of data, specifying additional channels and/or degrees of parallelism will allow RMAN to keep more data moving into the backupsets.

The optimal number of channels will be determined by the available CPU and memory resources as well as I/O capacity. The rate at which a Data Domain system can receive data is dependent on model, and connectivity as well as the ability of the host OS to push data over the network.

EncryptionBest Practice recommendation:EncryptionofbackupsetsissupportedbyOracleRMAN.However,like multiplexing, encryption changes the data patterns presented to the backup device. Unless the customer is certain that the encryp-tion chosen will always encrypt the exact same data in exactly the same way each time, it is recommended that RMAN encryption NOT be used.

CompressionBest Practice recommendation:As previously mentioned, RMAN provides two styles of compression. The unused block compression is the default for Oracle 10g and can only be disabled by doing an image (COPY) backup. Generally, most customers have been seeing acceptable deduplication when doing “normal” full backups.

To invoke the BZIP2 compression of RMAN, the DBA specifies “AS COMPRESSED” on the BACKUP command. This style of compres-sion is similar to the hardware compression facility provided by most modern tape backup facilities. Since this is done in software, it uses scarce CPU and memory resources during the backup that can be used for other purposes.

Given that the Data Domain deduplication storage system imple-ments a similar compression facility AFTER the incoming data has been deduplicated, it is more efficient to allow the system to perform the compression.

So the Best Practice is to NOT specify AS COMPRESSED on the BACKUP command.


Questions and Answers

1. Turn on block checking.REASON: The aim is to detect, very early the presence of corrupt blocks in the database. This has a slight performance overhead, but Checksums allow Oracle to detect early corruption caused by underlying disk, storage system, or I/O system problems.

SQL> alter system set db_block_checking = true scope=both;

2. Turn on block tracking when using rmAn backups (if running 10g)

REASON: This will allow RMAN to backup only those blocks that have changed since the last full backup, which will reduce the time taken to back up, as less blocks will be backed up.

SQL> alter database enable block change tracking using file ‘/u01/oradata/ora1/change_tracking.f’;

3. Duplex log groups and members and have more than one archive log dest.

REASON: If an archivelog is corrupted or lost, by having multiple copies in multiple locations, the other logs will still be available and could be used.

If an online log is deleted or becomes corrupt, you will have another member that can be used to recover if required.

SQL> alter system set log_archive_dest_2=’location=/new/location/archive2’ scope=both;

SQL> alter database add logfile member ‘/new/location/redo21.log’ to group 1;

4. when backing up the database use the ‘check logical’ parameter

REASON: This will cause RMAN to check for logical corruption within a block as well as the normal head/tail checksumming. This is the best way to ensure that you will get a good backup.

RMAN> backup check logical database plus archivelog delete input;

5. Test your backup.REASON: This will do everything except actually restore the data-base. This is the best method to determine if your backup is good and usable before being in a situation where it is critical and issues exist.

RMAN> restore validate database;

6. Have each datafile in a single backup pieceREASON: When doing a partial restore RMAN must read through the entire piece to get the datafile/archivelog requested. The smaller the backup piece the quicker the restore can complete. This is especially relevent with tape backups of large databases or where the restore is only on individual / few files.

RMAN> backup database FILESPERSET 1 plus archivelog delete input;

An example might look something like this:

RUN{CONFIGURE CONTROLFILE AUTOBACKUP ON;ALLOCATE CHANNEL ddr1 DEVICE TYPE DISK FORMAT ‘/ddmnt/DDR-a/backup/oraback’;

ALLOCATE CHANNEL ddr2 DEVICE TYPE DISK FORMAT ‘/ddmnt/DDR-b/backup/oraback’;

ALLOCATE CHANNEL ddr3 DEVICE TYPE DISK FORMAT ‘/ddmnt/DDR-c/backup/oraback’;

BACKUP(DATAFILE 1,2,3 FILESPERSET 1 CHANNEL ddr1)(DATAFILE 4,5 FILESPERSET 1 CHANNEL ddr2)(ARCHIVELOG ALL FILESPERSET 1 CHANNEL ddr3 INCLUDE CURRENT CONTROLFILE);

}

This example also shows how the degree of parallelism can be controlled manually. This will run three simultaneous tasks backing up files to three different Data Domain systems.

Oracle RecommendationsFor completion, we include a copy of an FAQ from Oracle’s MetaLink website. We sincerely hope they will not mind:

Subject: Top 10 Backup and Recovery best practices. DocID: Note:388422.1 Type: FAQ LastRevisionDate:04-OCT-2007 Status: PUBLISHED

Purpose

Top 10 Backup and Recovery best practices.This document assumes that you are doing the Backup and Recovery basics.

•RunninginArchivelogmode

•Multiplexingthecontrolfile

•Takingregularbackups

•Periodicallydoingacompleterestoretotestyourprocedures.

Channel Allocation to Speci�c Data�les

channel ddr3channel ddr1 channel ddr2

Datafile1

Datafile2

Datafile3

Datafile4

Datafile5

Archivedredo logs

Backupset1

DDR-A DDR-B DDR-C

Backupset2

Backupset3

Backupset4

Backupset5

Backupset6


SeeNote443814.1ManagingmultiplearchivelogdestinationswithRMAN for details.

SummaryA Data Domain System makes an excellent target for Oracle/RMAN backups because

1. A Data Domain system integrates easily and seamlessly into existing Oracle environments

2. Allows the DBA to retain online a greater number of full backup images thereby optimizing recovery options while occupying a minimal footprint in the datacenter

3. Greatly reduces the dependence on tape

7. maintain your rmAn catalog/controlfileREASON: Choose your retention policy carefully. Make sure that it compliments your tape subsystem retention policy, requirements for backup recovery strategy. If not using a catalog, ensure that your controlfile record keep time instance parameter matches your retention policy.

SQL> alter system set control_file_record_keep_time=21 scope=both;

This will keep 21 days of backup records.

Run regular catalog maintenance.

REASON: Delete obsolete will remove backups that are outside your retention policy. If obsolete backups are not deleted, the catalog will continue to grow until performance becomes an issue.

RMAN> delete obsolete;

REASON: crosschecking will check that the catalog/controlfile matches the physical backups. If a backup is missing, it will set the piece to ‘EXPIRED’ so when a restore is started, that it will not be eligible, and an earlier backup will be used. To remove the expired backups from the catalog/controlfile use the delete expired command.

RMAN> crosscheck backup;RMAN> delete expired backup;

8. Prepare for loss of controlfiles.

set autobackup on

REASON: This will ensure that you always have an up to date controlfile available that has been taken at the end of the current backup not during.

RMAN> configure controlfile autobackup on;

keep your backup logs

REASON: The backup log contains parameters for your tape access, locations on controlfile backups that can be utilised if complete loss occurs.

9. Test your recoveryREASON: During a recovery situation this will let you know how the recovery will go without actually doing it, and can avoid having to restore source datafiles again.

SQL> recover database test;

10. Do not specify ‘delete all input’ when backing up archivelogsREASON: Delete all input’ will backup from one destination then delete both copies of the archivelog where as ‘delete input’ will backup from one location and then delete what has been backed up. The next backup will back up those from location 2 as well as new logs from location 1, then delete all that are backed up. This means that you will have the archivelogs since the last backup available on disk in location 2 (as well as backed up once) and two copies backup up prior to the previous backup.


Appendix A – TroubleshootingThere have been situations where getting the “FILESPERSET” option set correctly has been a problem. The following displays are presented to assist in confirming the proper behavior of RMAN.

Using Oracle Enterprise ManagerIf access to the Oracle Enterprise Manager (OEM) is available, the following screenshots provide the clue. In particular, we are looking for the backup report that shows how many datafiles have been placed into a backup set. If the setting is in effect, RMAN will create a backup set (one backup file) for each datafile and archived log in the database.

Within OEM, navigate to the Maintenance -> Backup Reports screen. This screen should look something like this:

As you can see, there are three backup jobs that completed recently. For this example, the third one (started at 2:07pm) will show a backup WITHOUTFILESPERSET=1,thesecondone(startedat2:17pm)willshowabackupWITHFILESPERSET=1,andthenewestone(startedat2:31pm) will show an Image (or Copy) backup with FILESPERSET=1.


Example: “FILESPERSET” not set to “1”Selecting the 2:07pm job shows:

We have scrolled down a bit and only show the most important part of the display.

Note that the report shows:

Input Summary- Datafile: Files backed up: 5- Archived Log: Files backed up: 1- Control file: Files backed up: 1- SPFILE: Files backed up: 1Output Summary- Total Backup Sets: 3

Without “FILESPERSET = 1”, Oracle will combine multiple database files into a handful of backup files. In particular, all the datafiles can be combined into one-or-more backup files, all the archivelog files can similarly combined, and the controlfile can be combined with the parameter (SPFILE) file.

When FILESPERSET is set to one, each datafile and archivelog are backed up to separate files. In this case, we see that RMAN only created 3 Backup Sets. What RMAN has done is combined all the Datafiles into a single backup set / file, all the archived logfiles (in this case only one) into a second backup set / file, and the Control file into a third.


Further down the screen we see:

note: Notice the column “Compression Ratio”. Even though we did not specify to RMAN to use compression, some compression is still done. Oracle describes this as “unused block elimination” which basically means that empty space within the Oracle data files is not sent to backup. This should not be confused with either the compression effect observed on the Data Domain nor compression that would be seen if RMAN performed its own optional compression.

Toward the end of the report, we see the listing of the actual files RMAN created with their (rather obscure) filenames. These are the files actually written to the Data Domain system.

So that ends our example of a backup performed without setting FILESPERSET and simply going with the Oracle default.


Example: “FILESPERSET = 1”In this example, we specify “FILESPERSET=1” for the backup steps for both the datafiles and the archivelogs. If the backup is configured with OEM, this can be set simply by filling in the field shown below and setting it to ‘1’. This screen appears when performing a “Customized” backup using OEM.

This results in a script being generated that looks like:


OK, if we run this script, how do the results vary. Below are the same three screen shots shown earlier with the first backup.


RMAN> list backup;

List of Backup Sets===================

BS Key Type LV Size Device Type Elapsed Time Completion Time------- ---- -- ---------- ----------- ------------ ---------------2 Full 619.50M DISK 00:01:42 01-NOV-07 BP Key: 2 Status: AVAILABLE Compressed: NO Tag: BACKUP_ORCL_000006_110107020658 Piece Name: /dd580a/oracle/09ivv8cf_1_1 List of Datafiles in backup set 2 File LV Type Ckp SCN Ckp Time Name ---- -- ---- ---------- --------- ---- 1 Full 969073 01-NOV-07 /home/oracle/oracle/product/10.2.0/oradata/orcl/system01.dbf 2 Full 969073 01-NOV-07 /home/oracle/oracle/product/10.2.0/oradata/orcl/undotbs01.dbf 3 Full 969073 01-NOV-07 /home/oracle/oracle/product/10.2.0/oradata/orcl/sysaux01.dbf 4 Full 969073 01-NOV-07 /home/oracle/oracle/product/10.2.0/oradata/orcl/users01.dbf 5 Full 969073 01-NOV-07 /home/oracle/oracle/product/10.2.0/oradata/orcl/example01.dbf

BS Key Type LV Size Device Type Elapsed Time Completion Time------- ---- -- ---------- ----------- ------------ ---------------3 Full 6.80M DISK 00:00:05 01-NOV-07 BP Key: 3 Status: AVAILABLE Compressed: NO Tag: BACKUP_ORCL_000006_110107020658 Piece Name: /dd580a/oracle/0aivv8fo_1_1 Control File Included: Ckp SCN: 969266 Ckp time: 01-NOV-07 SPFILE Included: Modification time: 01-NOV-07

BS Key Size Device Type Elapsed Time Completion Time------- ---------- ----------- ------------ ---------------4 32.42M DISK 00:00:12 01-NOV-07 BP Key: 4 Status: AVAILABLE Compressed: NO Tag: BACKUP_ORCL_000006_110107020658 Piece Name: /dd580a/oracle/0bivv8gb_1_1 Thrd Seq Low SCN Low Time Next SCN Next Time ---- ------- ---------- --------- ---------- --------- 1 33 937224 01-NOV-07 969323 01-NOV-07

BS Key Type LV Size Device Type Elapsed Time Completion Time------- ---- -- ---------- ----------- ------------ ---------------5 Full 373.12M DISK 00:01:42 01-NOV-07 BP Key: 5 Status: AVAILABLE Compressed: NO Tag: BACKUP_ORCL_000007_110107021744 Piece Name: /dd580a/oracle/0civv90n_1_1 List of Datafiles in backup set 5 File LV Type Ckp SCN Ckp Time Name ---- -- ---- ---------- --------- ---- 1 Full 970468 01-NOV-07 /home/oracle/oracle/product/10.2.0/oradata/orcl/system01.dbf

BS Key Type LV Size Device Type Elapsed Time Completion Time------- ---- -- ---------- ----------- ------------ ---------------6 Full 26.17M DISK 00:00:05 01-NOV-07 BP Key: 6 Status: AVAILABLE Compressed: NO Tag: BACKUP_ORCL_000007_110107021744 Piece Name: /dd580a/oracle/0divv940_1_1 List of Datafiles in backup set 6 File LV Type Ckp SCN Ckp Time Name ---- -- ---- ---------- --------- ---- 2 Full 970533 01-NOV-07 /home/oracle/oracle/product/10.2.0/oradata/orcl/undotbs01.dbf

From the command lineIf Oracle Enterprise manager is not available, the required information can also be gained from the command line. The RMAN “list backup” command is used for this. An example is shown below.


BS Key Type LV Size Device Type Elapsed Time Completion Time------- ---- -- ---------- ----------- ------------ ---------------7 Full 164.66M DISK 00:00:38 01-NOV-07 BP Key: 7 Status: AVAILABLE Compressed: NO Tag: BACKUP_ORCL_000007_110107021744 Piece Name: /dd580a/oracle/0eivv947_1_1 List of Datafiles in backup set 7 File LV Type Ckp SCN Ckp Time Name ---- -- ---- ---------- --------- ---- 3 Full 970561 01-NOV-07 /home/oracle/oracle/product/10.2.0/oradata/orcl/sysaux01.dbf

BS Key Type LV Size Device Type Elapsed Time Completion Time------- ---- -- ---------- ----------- ------------ ---------------8 Full 56.38M DISK 00:00:13 01-NOV-07 BP Key: 8 Status: AVAILABLE Compressed: NO Tag: BACKUP_ORCL_000007_110107021744 Piece Name: /dd580a/oracle/0fivv95k_1_1 List of Datafiles in backup set 8 File LV Type Ckp SCN Ckp Time Name ---- -- ---- ---------- --------- ---- 5 Full 970615 01-NOV-07 /home/oracle/oracle/product/10.2.0/oradata/orcl/example01.dbf

BS Key Type LV Size Device Type Elapsed Time Completion Time------- ---- -- ---------- ----------- ------------ ---------------9 Full 1.90M DISK 00:00:01 01-NOV-07 BP Key: 9 Status: AVAILABLE Compressed: NO Tag: BACKUP_ORCL_000007_110107021744 Piece Name: /dd580a/oracle/0givv963_1_1 List of Datafiles in backup set 9 File LV Type Ckp SCN Ckp Time Name ---- -- ---- ---------- --------- ---- 4 Full 970652 01-NOV-07 /home/oracle/oracle/product/10.2.0/oradata/orcl/users01.dbf

BS Key Type LV Size Device Type Elapsed Time Completion Time------- ---- -- ---------- ----------- ------------ ---------------10 Full 6.77M DISK 00:00:02 01-NOV-07 BP Key: 10 Status: AVAILABLE Compressed: NO Tag: BACKUP_ORCL_000007_110107021744 Piece Name: /dd580a/oracle/0hivv964_1_1 Control File Included: Ckp SCN: 970664 Ckp time: 01-NOV-07

BS Key Type LV Size Device Type Elapsed Time Completion Time------- ---- -- ---------- ----------- ------------ ---------------11 Full 80.00K DISK 00:00:01 01-NOV-07 BP Key: 11 Status: AVAILABLE Compressed: NO Tag: BACKUP_ORCL_000007_110107021744 Piece Name: /dd580a/oracle/0iivv966_1_1 SPFILE Included: Modification time: 01-NOV-07

BS Key Size Device Type Elapsed Time Completion Time------- ---------- ----------- ------------ ---------------12 2.00M DISK 00:00:02 01-NOV-07 BP Key: 12 Status: AVAILABLE Compressed: NO Tag: BACKUP_ORCL_000007_110107021744 Piece Name: /dd580a/oracle/0jivv96a_1_1

List of Archived Logs in backup set 12 Thrd Seq Low SCN Low Time Next SCN Next Time ---- ------- ---------- --------- ---------- --------- 1 34 969323 01-NOV-07 970724 01-NOV-07

BS Key Type LV Size Device Type Elapsed Time Completion Time------- ---- -- ---------- ----------- ------------ ---------------13 Full 80.00K DISK 00:00:01 01-NOV-07 BP Key: 13 Status: AVAILABLE Compressed: NO Tag: BACKUP_ORCL_000008_110107023121 Piece Name: /dd580a/oracle/0qivva1h_1_1 SPFILE Included: Modification time: 01-NOV-07

Copyright©2008DataDomain,Inc.Allrightsreserved.Specificationssubjecttochangewithoutnotice.DataDomain,theDataDomainlogoandGlobal Compression are trademarks or registered trademarks of Data Domain, Inc. All other trademarks used or mentioned herein belong to their respective owners.

2421 mission college Blvd .

santa clara, ca 95054

866-we-ddupe

sales@datadomain .com

wp-oBp-0108

Oracle RMAN Design Best Practices with Data Domain

oracle rman design best practices with data domain...

Documents