migrate 10tb to exadata tips and tricks (presentation)
DESCRIPTION
Migrate 10 TB to Exadata -- Tips and Tricks Presentation at IOUG COLLABORATE 14, April 2014 Las VegasTRANSCRIPT
AMIN ADATIAKNOWTECH SOLUTIONS INC.
WORKING WITHORACLE SINCE 1985EXADATA SINCE 2011
FOCUSCUSTOM APPLICATION
DEVELOPMENT
Migrate to Exadata X3Tips and Tricks
Level 3, San Polo 3403, Session 326Thu, Apr 10, 2014 (12:15 PM - 12:45 PM)
Contact Details
eMail => [email protected] Twitter => aminadatiaLinkedIn
Data Architecture Professionals Oracle Exadata (IOUG SIG) Oracle Exadata Users
Oracle User Groups IOUG ODTUG
Agenda
Background Source and Target Environments
Migration ConstraintsData Migration Techniques
Approach Non Partitioned Tables Partitioned Tables
Without LOB Columns With LOB Columns Sub-Partition by Hash Tables
Oracle Text Indexes without using TTSOracle Label Security without Custom Label Tags
Environments
SOURCE Oracle 10.2.0.4
AIX 2 Node RAC 48 CPU per Node
TargetOracle 11.2.0.3
Exadata X3 8 Node RAC 32 CPU per Node RAM
PatchsetCellPatch
Downtime Constraints
Users mostly query for the past 240 PartitionsSome queries span 1200 PartitionsPlan for
480 Partitions loaded before users allowed access and ETL loading to resume
Estimated time was 5 hours and so 6 hours was allocated for cold migration
Not much prospect to drop indexes and recreate ETL was done for 3 days prior to migration Having NO INDEXES resulted in zero ETL At the end of the downtime => ETL would start
Source AIX Environment
Tablespaces => 450 (9.8 TB Disk Used)Table Partitions distributed across all TablespacesTables (for Migration)
Partitioned => 33 (~ 2500 Partitions/Table) Non Partitioned => 65 (2 Large Number
of Rows) Oracle Text Indexes => 4 Tables with LOBs => 4 (CLOB and BLOB)
Oracle Label Security => ~ 280 LabelsNetwork Link => 1 MBit and 10 MBit
Target X3 Environment
Tablespaces => Same as SourceOracle Label Security
Create Policy Create Labels and Tags Map New Tags to Old Tags
Oracle Text Define Preferences
Schema Definitions Export from Source with Rows = N Import into Target
Target X3 Environment
Going from Test (X2) to Production (X3)DataPump had a bug => Extremely Slow Revert to using 10g EXP
Virtual Columns Not RecognizedSchema Definitions (3)
Export from X2 with (Rows = N) Create Oracle Text Preferences Create OLS Policy Import into X3 Recreate Objects with Virtual Columns
Non Partitioned Tables
Non Partitioned Tables … 1
All commands Issued from Exadata
INSERT INTO <Table_Name>SELECT /*+ PARALLEL (T1,p_Degreee) */FROM <Source_Table>@DbLink_Pipe T1
PX Elapsed Time 12 57 minutes 24 35 minutes
WITH NO LOB COLUMNSWITH LOB COLUMNSSUB-PARTITION BY HASH AND LOB COLUMN
Partitioned Tables
Partitioned Tables Outcome
Hou
rs
Table
WITH NO LOB COLUMNS
Partitioned Tables (No LOBs)
Partitioned Table (No LOBs) ..1
v_Step := 'Create => '||v_Q_Table||' => '||p_Table_Name;
EXECUTE IMMEDIATE 'create table '||v_Q_Table||chr(10)||'TABLESPACE '||v_Tablespace_Name||chr(10)||'NOLOGGING'||chr(10)||'as select'||chr(10)||' /*+'||chr(10)||' parallel (a,'||p_Parallel_Source||')'||chr(10)||' no_index ('||p_No_Index_Hint||')'||chr(10)||' */'||chr(10)||' * from '||p_Table_Name||'@'||p_Data_Source||' a'||chr(10)||'where a.PartKey||chr(10)||’ BETWEEN p_Start_Number - '||p_Offset_Start||chr(10)||' AND p_Start_Number - '||p_Offset_End;
Partitioned Table (No LOBs) ..2
v_Step := 'Gather Stats => '||v_Q_Table;
v_Step := ‘Exchange Working Table for Partition'; If RECORDS > 0
Insert Data from Q_Table into Working Table based on Partition_Key.
v_Step := 'Create Matching Partition Indexes';
Create Index on Working Table Exchange Working Table with Partition
Partitioned Table (No LOBs) .. 3
EXECUTE IMMEDIATE 'ALTER TABLE '||p_Table_Name ||chr(10)||' EXCHANGE PARTITION‘ ||v_Partition_Name ||chr(10)||' WITH TABLE ' ||v_Working_Table ||chr(10)||' INCLUDING INDEXES' ||chr(10)||' WITHOUT VALIDATION' ||chr(10)||' UPDATE GLOBAL INDEXES';
Partitioned Table (No LOBs) … 4
Parallel at Source => 2,4,8,12,24depending on the amount of datahow many jobs were still running
Parallel at Target => usually 8After 4 hours Resources on AIX were diverted to deal to Tables with LOB Columns
TABLES WITH LOB COLUMNS
Partition Tables (LOBs)
Partitioned Table (LOBs) ..1
Parallel at Source => 1,2,4,8,16,24,32Eventually found out that PX = 1 was
bestBy the time we figured out the solution
only one table was left to doSubmitted 45 jobs on each source node
using parallel = 1 at source Each job took about 15 minutes
Partitioned Table (LOBs) ..2
Partitioned Table (LOBs) ..3
v_Begin INTEGER := 210;BEGIN FOR I in 1..45 LOOP v_Start := v_Begin + (I - 1); v_End := v_Start; v_Job_Name := 'M_B1_'||v_Start||'_'||v_End;
v_Job_Action := 'BEGIN' ||chr(10) ||'MIGRATION.DIRECT_EXCHANGE(''‘ ||p_Partition_LOB_Table ||'''',12,''a'',1,8,' ||v_Start||','||v_End||','''||v_Source||''');' ||chr(10)||'END;';
Partitioned Table (LOBs) ..4
BEGIN DBMS_SCHEDULER.CREATE_JOB ( job_name => v_Job_Name ,job_type => 'PLSQL_BLOCK' ,job_action => v_Job_Action ,start_date => SYSTIMESTAMP ,enabled => TRUE );
DBMS_SCHEDULER.SET_ATTRIBUTE ( name => v_Job_Name ,attribute => 'instance_stickiness' ,value => FALSE );
END;
END LOOP;
Partition Exchange
Partition Exchange .. 2
Worked fine except for continuous ETL … TABLE gets LOCKED for the EXCHANGE If the duration was “small” (less than 2
minutes) then there was not a noticeable impact on the ETL process
32 SubPartition Exchange approach did not work well
Migration Summary
Object Type Estimate
Partitions
Completed/Total
Actual
Partitions within
6 Hour Downtime
Actual Time
Non Partitioned Tables (65) 30 minutes 12 minutes Partitioned Tables
Non LOB Columns (30)480/2500 26 Tables
completed1 Table 8802 Tables 4401 Table 200
8 Hours11 Hours20 Hours
Partitioned Tables
CLOB Column Tables (2 Tables)1 Table 480/25001 Table 40/1800
1000160
11 Hours15 Hours
Sub Partitioned (32) Table
BLOB Column28/1200 16 8 Days!!
Sub Partition Exchange
Sub Partition Exchange .. 1
Table with Range Partition on two columns 32 Sub Partitions by Hash on third column
Getting the Hash Value and finding the subpartition seemed to slow down the exchange
Simple insert from Exchange Table also had the same kind of issue. Worse than Sub Partition Exchange
Insert using Sorted by (ORA_HASH(HashKey,31,0)+1) significantly improved performance
Sub Partition Exchange .. 2
INSERT INTO HASH_LOB_TABLESELECT /*+ NO_INDEX(A) */ * FROM Q_TABLE AORDER BY A.PartKey_Keep,(ORA_HASH(A.<hashkey>,31,0) + 1);
Sub Partition Exchange … 3Time per Record
Not much good for Simultaneous ETL
Sub Partition Exchange … 4Relative Insert Time/ Record
WITHOUT TRANSPORTABLE TABLESPACES
Oracle Text Index
Oracle Text Environment
Tables with Text Indexes Multi-lingual Documents stored on File System (2
Tables) Generated XML from Data (2 Tables)
Transportable Tablespaces Method for migrating was estimated to take 20 days
New data could not be loaded until TTS was completed
Document Indexing
Using High Degree Parallel (32,64,96) One Partition Data could be indexed in about 30 – 60
minutes This was less than the time it took to move the files to
the new file systemIndexing routine checked the status of the
file moveIndexing Started when File move was
completedFor Highlighting Search Items, the Text
Query goes against the Text Document stored on the file system and so the files needed to be moved
Sync Indexing
Using CTX_USER_PENDING to get the Partition to be SYNCed did not work well => 18 minutes to return Partition Names
Write a View => Less than 1 second CTXSYS.DR$PENDING (Parallel 2) CTXSYS.DR$INDEX CTXSYS.DR$INDEX_PARTITION (Parallel 2)
Loop through Partitions found “PENDING”Job runs every 10 secondsParallel Degree usually
Text 8 (used 16,24,32,64) XML 16 (used 4,8,16,24,32,64,96)
Optimize Indexing
SYNC_INDEX is with Parallel (4,8,16,32,64,96)OPTIMIZE_INDEX – Parallel 1
SYNC_INDEX get Locked out CTX_DDL.LOCK_NOWAIT did not resolve the problem
Optimize done via DBMS_SCHEDULER JobWrote View to exclude Partitions from
SYNC_INDEX which were part of the Optimize Index Job
Optimize done on Previous Partitions and it takes about 3 Hours to gain 60% - 85% tokens reduction
Impact on Switch Log Frequency (+ 100/hour)
XML Document Indexing
Using High Degree Parallel (16,32,64) One Partition Data could be indexed in about 30
minutes This was about the time it took to generate the XML XML generation could only be done once the
underlying tables had the data migratedBuild XML routine checked for all the data
tables for the day had been migratedIndexing routine checked the status of the
XML BuildIndexing Started when XML Build was
completed
WITHOUT CUSTOM LABEL TAGS
Oracle Label Security
Setup OLS Metadata
Create Label PolicyCreate Labels from SourceDefine Mapping between OLD => NEW TagsUse CASE/DECODE to convert during
Migration of Table WHEN <Label_Tag> = Old Value THEN <New-
Label_Tag> ELSE <not_matched_tag>
Issues => Updating Label_Tag after activating OLS on the Table
was painstakingly slow
Questions??
ENJOY THE REST OF THE CONFERENCE
Thank you!!