ukoug06 refreshes

7/27/2019 UKOUG06 Refreshes

1/47

Refreshing Test Databases Using Oracle Transportable Tablespaces and Checkpoints

Alex Gorbachev

The Pythian Group

2006 November

Special thanks to Nicolae Hachi

http://www.pythian.com/documents/UKOUG06-Refreshes.ppt


2/47

Agenda

Who am I?Refresh scenarios and challenges

Transportable Tablespaces and their roleRename tablespace in pre-10gVeritas Storage Checkpoints

AlternativesIs Flashback an option?

ZFS Clones example


3/47

Who am I?

Senior Oracle DBA at The Pythian GroupWorking with Oracle since 97 Pythian is a leading DBA outsourcing

provider remote DBA 24x7Offices worldwide Canada, US, UK, India,Australia

Im blogging at www.pythian.com/blogs


4/47

Low Test Systems Capacity

Test systems do not earn moneyShortage of

StorageMemoryCPU

Network capacity


5/47

Instant Refresh

Minutes, not hours!HourlyFull production size


6/47

Multiple Independent Clones

Each client need its own test system Temporary test environments (migrations)Development teams dont want to messwith each other QA testsMassive parallel testingWhat-If testing


7/47

Transportable Tablespaces

Designed to facilitate fast data movementsPlug datafiles directly into another database

Export/Import from 8i and aboveData Pump starting in 10gTTS covers:

Datafiles and tablespacesTables and indexes

Constraints, grants, triggers No sequences, synonyms, PL/SQL procedures, typesand etc.


8/47

TTS Limitations

Same character set Non-supported objects

Materialized viewsFBIAQ 8.0 multi-consumer queues

Cant rename tablespace to import twice 10g can remap and rename

Same platform10g cross-platform with RMAN

Compatibility considerations between releases


9/47

TTS Move Procedure

Generate TTS setCheck that it self-containedPlace tablespaces in read only ( !)Export Metadata

Copy TTS set (datafiles and metadata)Copy done => tablespaces can be put back in readwrite in the source database

Import TTS to the target databaseSet read write

Outage in 9i

Too long

No re-import after read write

9i cant import twice same name


10/47

Life Scenario Requirements

Up to 1 TB production databasesAbsolutely no production downtimeWeekly refreshes from the same capture

Re-capture periodically

15 minutes downtime for refreshUp to 7 independent test environmentsLimited test environment capacity


11/47

Capture

Restore into auxiliary DBPre-processing

CustomizationCompressionRemove sensitive info

Remove DB linksDisable FK

Prepare TTS set + capture other objects


12/47

Test System

Bundle test environments in one DBFlip-flop mechanism to avoid downtimeduring copyPlug tablespace in several times?

10g rename or remap in Data Pump9i prepare several copies during capture9i use unsupported remap method


13/47

Pre- 10g Remap

Manually update dump file bvi, bvedit, bsedReplace tablespace names with new ones

Same length ( !)Careful dont replace unwanted strings

Tablespace names in datafile header are not checked onimport (only DBID, file#)After import, datafiles are read-only so headers areignoredDuring read-write headers are flushed with new infoincluding new tablespace name


14/47

600 GB

system

600 GB

600 GB

600 GB

600 GB

600 GB

600 GB

TTS Refresh

PreparationsCopy to inactive destinationsLong but no impact

Refresh 15 min.Drop exiting tablespacesImport TTS from alternativelocationCreate/import other objectsEnable FK Make tablespace read-write

Preparations

300 GB

Datafiles part of the DB

Datafiles TTS set

Datafiles read only


15/47

Results So Far (TTS + Rename)

Zero production impactOne DB many testenvironments

Reduced memory andCPU requirements whiledelivering several testenvironments

Refresh in 15 minutes

Storage group is reallyunhappy

Test systems use7*2+1 of production size15 * 300 = 4.5 TB

New requirements fromQA group

Another 3 environmentsRefresh on demand several times per day,every hour


16/47

VxFS Storage Checkpoints

Filesystem technologyComparable volume based technology exists

Checkpoint visible as an exact image of the base filesystemCreated in a matter of secondsFor a database its a separate filesystemon a different mount-point


17/47

Checkpoints Are Writable

Base filesystem is writable No impact on checkpoints

Checkpoints are writable No impact on base filesystem No impact on other checkpoints

Changes to existing data consumeadditional spaceCopy-on-write technology


18/47

Checkpoints Are Persistent

Checkpoint information is stored on media Not in memory

Checkpoints survive everything just likenormal FS

Unmount / mountServer rebootSystem crashes

Do not protect from media corruptions ( !)


19/47

Storage Efficiency

Checkpoint consumes practically no spacewhen created

Additional space consumedChanges on the base filesystemChanges on the checkpointCopy-on-write

Base filesystem and all checkpoints sharethe same free space pool


20/47

Multiple Checkpoints

Unlimited number of checkpointsHierarchical checkpoints

Base

Ckpt1 Ckpt2 Ckpt3

Ckpt3-1 Ckpt3-2 Ckpt3-3

Ckpt3-2-1


21/47

Creating Checkpoint

Master list contains aninode of every fileInode stores a list of

descriptors pointing to thelocation of files data Ordinary extentdescriptors

Checkpoint is just another

inode listOverlay extentdescriptors

Master list

allocated extent free space pool

checkpoint


22/47

Copy-On-Write

Write to checkpointVxFS detects an overlaydescriptor

Allocates new oneWrites new content tothe new extentUpdates descriptor incheckpoint to ordinary,

pointing to the newextentUpdating the sameextent doesnt causemetadata changes

Master list


checkpoint


23/47

Copy-On-Write

Write to base filesystemWrite to an extentchanged in checkpoint

Write to shared extentisnt straightforward Allocate new extentCopy old imageChange overlaydescriptor in checkpointto ordinary referencingthe new extentPerform the write

Master list


checkpoint


24/47

Storage Checkpoints Recap

Checkpoints are writable

Multiple checkpoints

Storage efficiency

Persistency


25/47

600 GB

system

600 GB

600 GB

600 GB

600 GB

600 GB

600 GB

TTS + Checkpoints Refresh

300 GB


Datafiles TTS set

Datafiles read only


26/47

Datafiles in checkpoint part of the DB

Datafiles TTS set

Datafiles in checkpoint read only


Datafiles TTS set

Datafiles read only

system


Only one big filesystem Create checkpoint

Import TTS + other objects + enable FK Set read-writeRepeat for othersCheckpoint changes gradually consume

space

/base /ckpt1

/ckpt2

/ckpt3

/ckpt4

/ckpt5

/ckpt6

/ckpt7


27/47

/base


Datafiles TTS set


system


How much space?300 GB original base10% changes (30 GB) x 7 times = 210 GB

What about fresh capture?+300 GB for overwriting the base

810 MB

/base /ckpt1

/ckpt2

/ckpt3

/ckpt4

/ckpt5

/ckpt6

/ckpt7Overwrite too costlyFirst delete base filesCreate new baseRecreate checkpoints

during next refresh


28/47


Datafiles TTS set


system

TTS + Checkpoints QA Request

What about 3 QA test systems?Short living 1% changes (3 GB)+ 40MB

+ 3 test environments+ another 3 test environments

/base /ckpt1

/ckpt2

/ckpt3

/ckpt4

/ckpt5

/ckpt6

/ckpt7

/cqa2

/cqa1

/cqa3

/cqa5

/cqa4

/cqa6


29/47

Refresh with Checkpoints Recap

1. Drop old tablespaces on checkpoint2. Remove checkpoint

Careful with umount check that no critical processes have inodes open on checkpoint

3. Create fresh checkpoint4. Import tablespaces + others + FK + RW


30/47

Hints

Removing the last checkpoint from cleaned up basecan take long be patientSolaris, AIX, and Linux VxFS 3.4+

HP-UX VxFS 3.5+Recommend at least VxFS 4.0+Use ODM or at least direct IOTest, test, test

Several bugs on HP-UXCheckpoint corruption under heavy load with ODMAutoextend problem without ODM but with direct IORead write bug


31/47

Alternatives to Veritas?

Flashback DatabaseEvery testenvironment requiresseparate DB

No flashback tablespace

Storage inefficient Might be longer toflashback

NetAppFlexClone technology

OEMed by IBM ZFS

Solaris10 Update2Open SolarisBeing ported to Linuxand FreeBSD


32/47

ZFS Experience (1)

Create ZFS base # zpool create pythian c1t1d0 c1t2d0# zfs create pythian/orabase# zfs set recordsize=8k pythian/orabase# zfs set atime=off pythian/orabase# mkdir /z1# zfs set mountpoint=/z1 pythian/orabase# chown oracle:dba /z1


33/47

ZFS Experience (2)

Create base tablespaces SQL> create user pythian identified by "$E%&V#$V%$";SQL> grant resource to pythian;

SQL> create tablespace pythian_datadatafile '/z1/pythian_data01.dbf' size 128M,datafile '/z1/pythian_data02.dbf' size 128M,datafile '/z1/pythian_data03.dbf' size 128M;

SQL> create tablespace pythian_idxdatafile '/z1/pythian_idx01.dbf' size 128M,datafile '/z1/pythian_idx02.dbf' size 128M;


34/47

ZFS Experience (3)

Create base schema SQL> create table pythian.obj tablespace pythian_data

as select oo.object_id o1, o.*from dba_objects o, dba_objects oowhere rownum alter table pythian.obj add primary key(o1,object_id) using index tablespace pythian_idx;

SQL> create index pythian.obj_idx1 on pythian.obj(owner,object_id) tablespace pythian_idx;


35/47

ZFS Experience (4)

Export TTSSQL> BEGIN

SYS.DBMS_TTS.TRANSPORT_SET_CHECK(ts_list => 'PYTHIAN_DATA,PYTHIAN_IDX'

);END;/SQL> SELECT * FROM SYS.TRANSPORT_SET_VIOLATIONS;SQL> ALTER TABLESPACE PYTHIAN_DATA READ ONLY;SQL> ALTER TABLESPACE PYTHIAN_IDX READ ONLY;

$ expdp dumpfile=pythian.dmptransport_tablespaces=PYTHIAN_DATA,PYTHIAN_IDX


36/47

ZFS Experience (5)

Prepare test schemasSQL> create user pythian01 identified by e4!d6^hsk"; SQL> grant resource to pythian01;SQL> create user pythian02 identified by 0d(hs grant resource to pythian03;

And mount points# mkdir /z1c1 /z1c2 /z1c3


37/47

ZFS Experience (6)

Clone it# zfs snapshot pythian/orabase@snap# zfs clone pythian/orabase@snap pythian/oraclone1# zfs clone pythian/orabase@snap pythian/oraclone2# zfs clone pythian/orabase@snap pythian/oraclone3# zfs set mountpoint=/z1c1 pythian/oraclone1# zfs set mountpoint=/z1c2 pythian/oraclone2# zfs set mountpoint=/z1c3 pythian/oraclone3


38/47

ZFS Experience (7)

Space usage right after cloning# zfs listNAME USED AVAIL REFER MOUNTPOINTpythian 647M 1.31G 24.5K /pythianpythian/orabase 647M 1.31G 647M /z1pythian/orabase@s1 0 - 647M -pythian/oraclone1 0 1.31G 647M /z1c1pythian/oraclone2 0 1.31G 647M /z1c2pythian/oraclone3 0 1.31G 647M /z1c3


39/47

ZFS Experience (8)

Create import par fileUSERID='/ as sysdba'DUMPFILE=pythian.dmpTRANSPORT_DATAFILES = (

/z1c1/pythian_data01.dbf,/z1c1/pythian_data02.dbf,/z1c1/pythian_data03.dbf,/z1c1/pythian_idx01.dbf,/z1c1/pythian_idx02.dbf)REMAP_SCHEMA=PYTHIAN:PYTHIAN01REMAP_TABLESPACE=PYTHIAN_DATA:PYTHIAN_DATA01REMAP_TABLESPACE=PYTHIAN_IDX:PYTHIAN_IDX01


40/47

ZFS Experience (9)

Import TTS impdp parfile=imp_z1c1.par

Set tablespaces read-writeALTER TABLESPACE PYTHIAN_DATA01 READ WRITE;ALTER TABLESPACE PYTHIAN_IDX01 READ WRITE;


41/47

ZFS Experience (10)

Space usage after the first import # zfs listNAME USED AVAIL REFER MOUNTPOINTpythian 647M 1.31G 24.5K /pythianpythian/orabase 647M 1.31G 647M /z1pythian/orabase@s1 0 - 647M -pythian/oraclone1 58.5K 1.31G 647M /z1c1pythian/oraclone2 0 1.31G 647M /z1c2pythian/oraclone3 0 1.31G 647M /z1c3


42/47

ZFS Experience (11)

Repeat import for other clones # zfs listNAME USED AVAIL REFER MOUNTPOINTpythian 648M 1.30G 24.5K /pythianpythian/orabase 647M 1.30G 647M /z1pythian/orabase@s1 0 - 647M -pythian/oraclone1 204K 1.30G 647M /z1c1pythian/oraclone2 204K 1.30G 647M /z1c2pythian/oraclone3 204K 1.30G 647M /z1c3


43/47

ZFS Experience (12)

Delete first 400 thousand rows # zfs listNAME USED AVAIL REFER MOUNTPOINTpythian 812M 1.14G 24.5K /pythianpythian/orabase 647M 1.14G 647M /z1pythian/orabase@s1 0 - 647M -pythian/oraclone1 164M 1.14G 647M /z1c1pythian/oraclone2 204K 1.14G 647M /z1c2pythian/oraclone3 204K 1.14G 647M /z1c3

10% is only 64.7 MB so why 164MB?Index leaves are all over the place


44/47

ZFS Experience (13)

Lets drop and recreate the table SQL> drop table pythian01.obj;SQL> create table pythian01.obj as select ;

Now the bad news Solaris virtual machine crashed on CTAS

but the good news Clones survived just fineOracle started after reset and CTAS worked


45/47

Whats Next?

Doesnt work with ASM Comparable volume technology exists, but

Cloned disks in conflict with original disks No rename disk groupBinary patching of disk header is not trivial

Play with hierarchical checkpointsPromoting checkpoints

Merge changes to parent


46/47

References

Symantec Yellow Book Using LocalCopy Services

VERITAS Storage Checkpoints AN UNDOBUTTON FOR USERS AND ADMINISTRATORS NetApp FlexClone Homepage IBM Redbooks Paper An Introduction toFlexClone Volumes Open Solaris ZFS Community
http://eval.veritas.com/mktginfo/enterprise/yellowbooks/using_local_copy_services_03_2006.en-us.pdfhttp://eval.veritas.com/mktginfo/enterprise/yellowbooks/using_local_copy_services_03_2006.en-us.pdfhttp://www4.symantec.com/Vrt/offer?a_id=9102http://www4.symantec.com/Vrt/offer?a_id=9102http://www4.symantec.com/Vrt/offer?a_id=9102http://www.netapp.com/products/software/flexclone.htmlhttp://www.redbooks.ibm.com/redpapers/pdfs/redp4133.pdfhttp://www.redbooks.ibm.com/redpapers/pdfs/redp4133.pdfhttp://www.opensolaris.org/os/community/zfs/http://www.opensolaris.org/os/community/zfs/http://www.opensolaris.org/os/community/zfs/http://www.opensolaris.org/os/community/zfs/http://www.redbooks.ibm.com/redpapers/pdfs/redp4133.pdfhttp://www.redbooks.ibm.com/redpapers/pdfs/redp4133.pdfhttp://www.netapp.com/products/software/flexclone.htmlhttp://www4.symantec.com/Vrt/offer?a_id=9102http://www4.symantec.com/Vrt/offer?a_id=9102http://www4.symantec.com/Vrt/offer?a_id=9102http://eval.veritas.com/mktginfo/enterprise/yellowbooks/using_local_copy_services_03_2006.en-us.pdfhttp://eval.veritas.com/mktginfo/enterprise/yellowbooks/using_local_copy_services_03_2006.en-us.pdf


47/47

Q & A

Thank you!

[email protected]

Alex Gorbachev

The Pythian Group
mailto:[email protected]:[email protected]

ukoug06 refreshes

Documents