migrate 10tb to exadata tips and tricks (presentation)

AMIN ADATIAKNOWTECH SOLUTIONS INC.

WORKING WITHORACLE SINCE 1985EXADATA SINCE 2011

FOCUSCUSTOM APPLICATION

DEVELOPMENT

Migrate to Exadata X3Tips and Tricks

Level 3, San Polo 3403, Session 326Thu, Apr 10, 2014 (12:15 PM - 12:45 PM)

Contact Details

eMail => amin@knowtech.ca Twitter => aminadatiaLinkedIn

Data Architecture Professionals Oracle Exadata (IOUG SIG) Oracle Exadata Users

Oracle User Groups IOUG ODTUG

Agenda

Background Source and Target Environments

Migration ConstraintsData Migration Techniques

Approach Non Partitioned Tables Partitioned Tables

Without LOB Columns With LOB Columns Sub-Partition by Hash Tables

Oracle Text Indexes without using TTSOracle Label Security without Custom Label Tags

Environments

SOURCE Oracle 10.2.0.4

AIX 2 Node RAC 48 CPU per Node

TargetOracle 11.2.0.3

Exadata X3 8 Node RAC 32 CPU per Node RAM

PatchsetCellPatch

Downtime Constraints

Users mostly query for the past 240 PartitionsSome queries span 1200 PartitionsPlan for

480 Partitions loaded before users allowed access and ETL loading to resume

Estimated time was 5 hours and so 6 hours was allocated for cold migration

Not much prospect to drop indexes and recreate ETL was done for 3 days prior to migration Having NO INDEXES resulted in zero ETL At the end of the downtime => ETL would start

Source AIX Environment

Tablespaces => 450 (9.8 TB Disk Used)Table Partitions distributed across all TablespacesTables (for Migration)

Partitioned => 33 (~ 2500 Partitions/Table) Non Partitioned => 65 (2 Large Number

of Rows) Oracle Text Indexes => 4 Tables with LOBs => 4 (CLOB and BLOB)

Oracle Label Security => ~ 280 LabelsNetwork Link => 1 MBit and 10 MBit

Target X3 Environment

Tablespaces => Same as SourceOracle Label Security

Create Policy Create Labels and Tags Map New Tags to Old Tags

Oracle Text Define Preferences

Schema Definitions Export from Source with Rows = N Import into Target

Target X3 Environment

Going from Test (X2) to Production (X3)DataPump had a bug => Extremely Slow Revert to using 10g EXP

Virtual Columns Not RecognizedSchema Definitions (3)

Export from X2 with (Rows = N) Create Oracle Text Preferences Create OLS Policy Import into X3 Recreate Objects with Virtual Columns

Non Partitioned Tables

Non Partitioned Tables … 1

All commands Issued from Exadata

INSERT INTO <Table_Name>SELECT /*+ PARALLEL (T1,p_Degreee) */FROM <Source_Table>@DbLink_Pipe T1

PX Elapsed Time 12 57 minutes 24 35 minutes

WITH NO LOB COLUMNSWITH LOB COLUMNSSUB-PARTITION BY HASH AND LOB COLUMN

Partitioned Tables

Partitioned Tables Outcome

Hou

rs

Table

WITH NO LOB COLUMNS

Partitioned Tables (No LOBs)

Partitioned Table (No LOBs) ..1

v_Step := 'Create => '||v_Q_Table||' => '||p_Table_Name;

EXECUTE IMMEDIATE 'create table '||v_Q_Table||chr(10)||'TABLESPACE '||v_Tablespace_Name||chr(10)||'NOLOGGING'||chr(10)||'as select'||chr(10)||' /*+'||chr(10)||' parallel (a,'||p_Parallel_Source||')'||chr(10)||' no_index ('||p_No_Index_Hint||')'||chr(10)||' */'||chr(10)||' * from '||p_Table_Name||'@'||p_Data_Source||' a'||chr(10)||'where a.PartKey||chr(10)||’ BETWEEN p_Start_Number - '||p_Offset_Start||chr(10)||' AND p_Start_Number - '||p_Offset_End;

Partitioned Table (No LOBs) ..2

v_Step := 'Gather Stats => '||v_Q_Table;

v_Step := ‘Exchange Working Table for Partition'; If RECORDS > 0

Insert Data from Q_Table into Working Table based on Partition_Key.

v_Step := 'Create Matching Partition Indexes';

Create Index on Working Table Exchange Working Table with Partition

Partitioned Table (No LOBs) .. 3

EXECUTE IMMEDIATE 'ALTER TABLE '||p_Table_Name ||chr(10)||' EXCHANGE PARTITION‘ ||v_Partition_Name ||chr(10)||' WITH TABLE ' ||v_Working_Table ||chr(10)||' INCLUDING INDEXES' ||chr(10)||' WITHOUT VALIDATION' ||chr(10)||' UPDATE GLOBAL INDEXES';

Partitioned Table (No LOBs) … 4

Parallel at Source => 2,4,8,12,24depending on the amount of datahow many jobs were still running

Parallel at Target => usually 8After 4 hours Resources on AIX were diverted to deal to Tables with LOB Columns

TABLES WITH LOB COLUMNS

Partition Tables (LOBs)

Partitioned Table (LOBs) ..1

Parallel at Source => 1,2,4,8,16,24,32Eventually found out that PX = 1 was

bestBy the time we figured out the solution

only one table was left to doSubmitted 45 jobs on each source node

using parallel = 1 at source Each job took about 15 minutes

Partitioned Table (LOBs) ..2

Partitioned Table (LOBs) ..3

v_Begin INTEGER := 210;BEGIN FOR I in 1..45 LOOP v_Start := v_Begin + (I - 1); v_End := v_Start; v_Job_Name := 'M_B1_'||v_Start||'_'||v_End;

v_Job_Action := 'BEGIN' ||chr(10) ||'MIGRATION.DIRECT_EXCHANGE(''‘ ||p_Partition_LOB_Table ||'''',12,''a'',1,8,' ||v_Start||','||v_End||','''||v_Source||''');' ||chr(10)||'END;';

Partitioned Table (LOBs) ..4

BEGIN DBMS_SCHEDULER.CREATE_JOB ( job_name => v_Job_Name ,job_type => 'PLSQL_BLOCK' ,job_action => v_Job_Action ,start_date => SYSTIMESTAMP ,enabled => TRUE );

DBMS_SCHEDULER.SET_ATTRIBUTE ( name => v_Job_Name ,attribute => 'instance_stickiness' ,value => FALSE );

END;

END LOOP;

Partition Exchange

Partition Exchange .. 2

Worked fine except for continuous ETL … TABLE gets LOCKED for the EXCHANGE If the duration was “small” (less than 2

minutes) then there was not a noticeable impact on the ETL process

32 SubPartition Exchange approach did not work well

Migration Summary

Object Type Estimate

Partitions

Completed/Total

Actual

Partitions within

6 Hour Downtime

Actual Time

Non Partitioned Tables (65) 30 minutes 12 minutes  Partitioned Tables

Non LOB Columns (30)480/2500 26 Tables

completed1 Table 8802 Tables 4401 Table 200

 8 Hours11 Hours20 Hours

Partitioned Tables

CLOB Column Tables (2 Tables)1 Table 480/25001 Table 40/1800

1000160

11 Hours15 Hours

Sub Partitioned (32) Table

BLOB Column28/1200 16 8 Days!!

Sub Partition Exchange

Sub Partition Exchange .. 1

Table with Range Partition on two columns 32 Sub Partitions by Hash on third column

Getting the Hash Value and finding the subpartition seemed to slow down the exchange

Simple insert from Exchange Table also had the same kind of issue. Worse than Sub Partition Exchange

Insert using Sorted by (ORA_HASH(HashKey,31,0)+1) significantly improved performance

Sub Partition Exchange .. 2

INSERT INTO HASH_LOB_TABLESELECT /*+ NO_INDEX(A) */ * FROM Q_TABLE AORDER BY A.PartKey_Keep,(ORA_HASH(A.<hashkey>,31,0) + 1);

Sub Partition Exchange … 3Time per Record

Not much good for Simultaneous ETL

Sub Partition Exchange … 4Relative Insert Time/ Record

WITHOUT TRANSPORTABLE TABLESPACES

Oracle Text Index

Oracle Text Environment

Tables with Text Indexes Multi-lingual Documents stored on File System (2

Tables) Generated XML from Data (2 Tables)

Transportable Tablespaces Method for migrating was estimated to take 20 days

New data could not be loaded until TTS was completed

Document Indexing

Using High Degree Parallel (32,64,96) One Partition Data could be indexed in about 30 – 60

minutes This was less than the time it took to move the files to

the new file systemIndexing routine checked the status of the

file moveIndexing Started when File move was

completedFor Highlighting Search Items, the Text

Query goes against the Text Document stored on the file system and so the files needed to be moved

Sync Indexing

Using CTX_USER_PENDING to get the Partition to be SYNCed did not work well => 18 minutes to return Partition Names

Write a View => Less than 1 second CTXSYS.DR$PENDING (Parallel 2) CTXSYS.DR$INDEX CTXSYS.DR$INDEX_PARTITION (Parallel 2)

Loop through Partitions found “PENDING”Job runs every 10 secondsParallel Degree usually

Text 8 (used 16,24,32,64) XML 16 (used 4,8,16,24,32,64,96)

Optimize Indexing

SYNC_INDEX is with Parallel (4,8,16,32,64,96)OPTIMIZE_INDEX – Parallel 1

SYNC_INDEX get Locked out CTX_DDL.LOCK_NOWAIT did not resolve the problem

Optimize done via DBMS_SCHEDULER JobWrote View to exclude Partitions from

SYNC_INDEX which were part of the Optimize Index Job

Optimize done on Previous Partitions and it takes about 3 Hours to gain 60% - 85% tokens reduction

Impact on Switch Log Frequency (+ 100/hour)

XML Document Indexing

Using High Degree Parallel (16,32,64) One Partition Data could be indexed in about 30

minutes This was about the time it took to generate the XML XML generation could only be done once the

underlying tables had the data migratedBuild XML routine checked for all the data

tables for the day had been migratedIndexing routine checked the status of the

XML BuildIndexing Started when XML Build was

completed

WITHOUT CUSTOM LABEL TAGS

Oracle Label Security

Setup OLS Metadata

Create Label PolicyCreate Labels from SourceDefine Mapping between OLD => NEW TagsUse CASE/DECODE to convert during

Migration of Table WHEN <Label_Tag> = Old Value THEN <New-

Label_Tag> ELSE <not_matched_tag>

Issues => Updating Label_Tag after activating OLS on the Table

was painstakingly slow

Questions??

ENJOY THE REST OF THE CONFERENCE

Thank you!!