database migration assistant for unicode (dmu)

Database Migration Assistant for Unicode (DMU)A Real Customer Case

Ludovico CaldaraOracle ACE DirectorTrivadis AG

About Ludovico Caldara

Database Migration Assistant for Unicode2 03/05/2023

■ 17 Years DBA (Not Only Oracle)

▪ I do it everywhere (even Windows)

■ RAC ATTACK Ninja & co-writer

■ President, SOUG & ITOUG Board

■ OCP (11g, 12c, MySQL) & OCE

■ Italian living in Switzerland

■ http://www.ludovicocaldara.net

■ @ludodba

■ ludovicocaldara

Database Migration Assistant for Unicode3 03/05/2023

Our company.Trivadis is a market leader in IT consulting, system integration, solution engineeringand the provision of IT services focusing on and technologies in Switzerland, Germany, Austria and Denmark.We offer our services in the following strategic business fields:

Trivadis Services takes over the interactive operation of your IT systems.

O P E R A T I O N

Database Migration Assistant for Unicode4 03/05/2023

COPENHAGEN

MUNICH

LAUSANNEBERN

ZURICHBRUGG

GENEVA

HAMBURG

DÜSSELDORF

FRANKFURT

STUTTGART

FREIBURG

BASEL

VIENNA

With over 600 specialists and IT experts in your region.

14 Trivadis branches and more than600 employees

200 Service Level Agreements

Over 4,000 training participants

Research and development budget:CHF 5.0 / EUR 4 million

Financially self-supporting and sustainably profitable

Experience from more than 1,900 projects per year at over 800customers

Database Migration Assistant for Unicode5 03/05/2023

Conversion to Unicode

Database Migration Assistant for Unicode6 03/05/2023

Why

Single-byte character sets cannot represent all the characters

Nowadays it is no more acceptable

Well-known character sets:

– US7ASCII

– WE8ISO8859P1(5)

– WE8MSWIN1252

Database Migration Assistant for Unicode7 03/05/2023

Issues with Unicode

Variable length per character (1-4 bytes, increase in data size)

Different representation for the non-US7ASCII characters

Data conversion

Database Migration Assistant for Unicode8 03/05/2023

Conversion to Unicode is Serious!

Be careful, you might corrupt your data!

Database Migration Assistant for Unicode9 03/05/2023

DMU

Database Migration Assistant for Unicode10 03/05/2023

DMU identity card

Born in 2011

Last version: Check MOS Patch 21138450 (currently 2.1)

Java-based GUI tool

Replaces csscan and csalter (they are now desupported)

End-to-end migration process

Database Migration Assistant for Unicode11 03/05/2023

Database Migration Assistant for Unicode12 03/05/2023

Conversion Flow

AnalyzePrepare Fix / Clean Convert Validate

Database Migration Assistant for Unicode13 03/05/2023

Prepare

Database Migration Assistant for Unicode14 03/05/2023

Prepare

Do it on a non-production first!

– Several times, reiterate to be sure the conversion will work

– Use a physical copy of your production

Compile invalid objects ( @?/rdbms/admin/utlrp.sql )

Purge the dba_recyblebin

List invalid objects

– create table pre_dmu_invalid as select owner, object_name, object_type from dba_objects where status != 'VALID' ;

Database Migration Assistant for Unicode15 03/05/2023

Prepare (cont.)Install the DMU support package SYS.DBMS_DUMA_INTERNAL

– SQL> @?/rdbms/admin/prvtdumi.plbLibrary created.Package created.No errors.Package body created.No errors.

Create a separate tablespace for the DMU repository

– CREATE TABLESPACE DMU_REPO DATAFILE SIZE 100M EUTOEXTEND ONNEXT 100M MAXSIZE 30g;

Extend the Undo Tablespace (it requires a lot of space)

– alter database datafile '/data/orcl/undotbs01.dbf' autoextend on maxsize 30G;

Database Migration Assistant for Unicode16 03/05/2023

Prepare (cont.)

Save the size of the segments before the migration

– CREATE TABLE dmu_size_before AS SELECT tablespace_name, owner, segment_type, segment_name, bytes, extents FROM dba_segments WHERE tablespace_name NOT IN ('UNDOTBS','TEMP') ;

Why? Bigger character representation

– Migrated rows

– Chained rows

Database Migration Assistant for Unicode17 03/05/2023

Analyze

Database Migration Assistant for Unicode18 03/05/2023

Launch DMU

Download and unzip dmu under $ORACLE_HOME/dmu-$version

Launch it:

– export DISPLAY=192.178.56.1:0.0export JAVA_HOME=$ORACLE_HOME/jdkcd /ccv/app/oracle/product/12.1.0.2/dmu-2.1/dmush dmu.sh

Create the connection to the database

Database Migration Assistant for Unicode19 03/05/2023

Repository installation

Database Migration Assistant for Unicode20 03/05/2023

Launch the scan

Database Migration Assistant for Unicode21 03/05/2023

Run the Scan Wizard

Increase according to your server

Database Migration Assistant for Unicode22 03/05/2023

Scan – Table properties

Database Migration Assistant for Unicode23 03/05/2023

Scan – Table properties

Database Migration Assistant for Unicode24 03/05/2023

Fix / Clean

Database Migration Assistant for Unicode25 03/05/2023

All the blocking issues must be solved

BLOCKING

Database Migration Assistant for Unicode26 03/05/2023

Find the issues in the scan report

Database Migration Assistant for Unicode27 03/05/2023

Need conversion

Database Migration Assistant for Unicode28 03/05/2023

Need conversion

Application Data will be converted during the conversion phase

– No action for you here

Dictionary data needing conversion must be fixed!

– Check on My Oracle Support!

– Tips For and Known Issues With The Database Migration Assistant for Unicode (DMU) Tool version 2.1 (Doc ID 2018250.1)

Database Migration Assistant for Unicode29 03/05/2023

Dictionary Cleansing for my Customer Case

Cleansing REG$

– truncate table reg$;

Cleansing AWR tables -> recreate AWR      SQL> @?/rdbms/admin/catnoawr.sql   SQL> @?/rdbms/admin/catawr.sql   SQL> execute dbms_swrf_internal.register_local_dbid;

Cleansing scheduler logs   exec dbms_scheduler.purge_log;

Cleansing SYS.WRP$_REPORTS* tables (SQL Monitor and R-T ADDM reports) truncate table wrp$_reports ;   truncate table wrp$_reports_details ;   truncate table wrp$_reports_time_bands ;

Database Migration Assistant for Unicode30 03/05/2023

Dictionary Cleansing for my Customer Case (Cont.)

Cleansing source$

– Fixed the source code manually

Cleansing col$

– A table column name was containing a special character… OMG…

Scan the dictionary again to make sure that everything has been fixed

Database Migration Assistant for Unicode31 03/05/2023

Over Column Limit

Database Migration Assistant for Unicode32 03/05/2023

Fixing Over Column Limit – By Column

Database Migration Assistant for Unicode33 03/05/2023

Fixing Over Column Limit – Bulk Cleansing

Database Migration Assistant for Unicode34 03/05/2023

Invalid Representation

Database Migration Assistant for Unicode35 03/05/2023

Fixing invalid representations – Find the source!

Why there is invalid data?

– Is it because of non-existing characters in current set?

– Or there is a bad configured client application?

– Or there is a bad configured ETL job?

Database Migration Assistant for Unicode36 03/05/2023

Fixing invalid representations – Cleansing

Database Migration Assistant for Unicode37 03/05/2023

Fixing invalid representations – Ignoring

Database Migration Assistant for Unicode38 03/05/2023

Fixing invalid representations – Ignoring

Use it as the last step in your cleansing phase!

Database Migration Assistant for Unicode39 03/05/2023

Other Fixes

Enable Row Movement on your tables to enable CTAS conversion

– Much faster than standard UPDATE

– No migrated rows after the upgrade

Put your application tablespaces in AUTOEXTEND

– Space increase due to UNICODE conversion

– Space used by Create Table As Select conversion method

Database Migration Assistant for Unicode40 03/05/2023

Convert

Database Migration Assistant for Unicode41 03/05/2023

Convert

BEFORECLEANSING

AFTERCLEANSING

Database Migration Assistant for Unicode42 03/05/2023

Convert (Cont.)

You cannot have connected session to the database

– Alter system enable restricted session;

You may want to ignore other warnings

Database Migration Assistant for Unicode43 03/05/2023

Convert (Cont.)

My customer had a few issues

– Tables locked by the DMU itself (retry the operation)

– DMU client disconnecting (client on the DB server via VNC is the best solution)

– Tablespaces running out of space (set them in autoextend before converting)

Database Migration Assistant for Unicode44 03/05/2023

Convert (Cont.)

Restart the database after the conversion

Database Migration Assistant for Unicode45 03/05/2023

Validate

Database Migration Assistant for Unicode46 03/05/2023

Install the repository in Validation mode

Database Migration Assistant for Unicode47 03/05/2023

Scan in validation mode

Database Migration Assistant for Unicode48 03/05/2023

Timing

Database Migration Assistant for Unicode49 03/05/2023

A Customer’s Database

500 Application Tables

250 Gb of data segments

Original Charset: WE8ISO8859P1

Not critical in terms of downtime

Very critical in terms of data correctness

Database Migration Assistant for Unicode50 03/05/2023

Timing

Scan: 10-12 minutes

Conversion: 6-10 hours (depending on parameters)

– Sometimes the elapsed time changes without reason

Validation scan: 8-10 minutes

Allows additional time for backups

Database Migration Assistant for Unicode51 03/05/2023

Speeding Up

Make sure that big tables are converted with CTAS

Huge tables? Consider:

– Expdp

– Truncate

– Convert

– Impdp

– (Don’t forget to validate the data manually)

Database Migration Assistant for Unicode52 03/05/2023

Total migration project (for one application)

May 2016: First test on a fake copy

June 2016: Migration of the Test environment

June – July 2016: Data validation and certification of the application for Unicode

July 2016: Migration of the pre-production

August 2016: Refresh prod-test and 2nd migration of the test environment

August 2016: Migration of the Quality environment

September 2016: Migration of the Production

– Downtime reserved from Friday evening to Saturday morning

Database Migration Assistant for Unicode53 03/05/2023

Alternative methods

Database Migration Assistant for Unicode54 03/05/2023

Alternative methods

Export/Import with Data Pump

– Bad: No check for conversion issues

Oracle GoldenGate

– Bad: No check for conversion issues + GoldenGate cost

DMU + GoldenGate -> Near Zero Downtime

– Bad: GoldenGate cost,

– But very efficient!

Questions?

03/05/2023 Database Migration Assistant for Unicode55

Ludovico CaldaraACE Director, Senior Consultant

ludovico.caldara@trivadis.com