DB2 Migrations: Planning, Preparing and Executing Migration to DB2 LUW James Sobieski Fourth Millennium Technologies

Session Code: C20 Fri, May 08, 2015 (09:15 AM - 10:15 AM) | Platform: DB2 LUW

Agenda • Reasons for Migrations

• Types of Migrations

• Platform changing

• DB2 Architecture changing

• Endianness Changing

• Configuration Changing

• Big-Bang vs Multi-Step migrations techniques

• Use cases for Backup/Restore, HPU, Q-Rep, CDC, Federation, DMT/MTK

• Migration tips and techniques

• Managing the migration – how to track success and progress

• Dealing with Indexes, Referential Integrity, Generated Columns

• Dealing with Stored Procedures, Functions, Triggers

• Migration Examples

2

Reasons for Database Migrations

• Outgrow current hardware

• New Architecture provides needed functionality

• Need better performance

• Costs – SW/HW/Support

• New Applications and Merged DB designs

• What Other Reasons have you Migrated a Database ?

3

Types of Database Migrations

• Platform Changes

• Upgrade of hardware

• Architecture Changes

• Endianess - Intel to Power (& Vice-versa)

• Mainframe to LUW (& Vice-versa)

• Clustered to Non-clustered (& Vice-versa)

• Change of Database Product

• Oracle to DB2 (never Vice-versa)

4

Challenges: Database Migrations often:

• Migrations can require downtime

• Migrations can be man-power consuming

• Changing platforms can cause performance issues

• New Procedures and Techniques

5

Challenge: Avoiding Evil Downtime

6

Challenges: Migration Planning

7

• Budget • How will this migration affect my

budget this year? Over 3 years?

• Will this take advantage of

existing equipment and licenses

we own?

• Technical • Can it be done?

• What transformations and

processes will be required?

• How can risk be minimized

throughout the process?

• Operational • Will this fit in our schedule?

• How will testing happen?

• Does this migration meet our

business requirements?

• How well will it work?

• Skills and Processes • What skills will be required to

operate the new system?

• What internal processes will need

to change?

• How will this migration affect us?

Implication: A Plan Must Evolve

• Must identify

• Business Objective & Technical Objectives

• Technical Constraints

• Identify Potential Techniques

• Obtain Appropriate Assistance and Advice

• Analyze Each Potential Technique

• Plan for the NW & Client SW changes required

• Develop Cost B&P estimates

• Evaluate Alternatives

• Test Solutions (if needed)

• Select Solution

• Implement – Strong PM required

• Evaluate Implementation

8

9

Technical Solutions Development

Migration Techniques and Effort Required

• Simplest • None ( Processor only swap)

• Disk/SAN copies to new hardware

• Moderate • Backup and Restore

• Complex • Changes in HW architecture

• Changes in Software (DB Product)

• Changes in DB architecture

• Non cluster to cluster and vice versa

• Changes in cluster configuration

• Least/ No Downtime Allowed

10

Migration Options • Big-Bang vs “Texas-Two-Step” migrations techniques

• Consider in-house tools and skills and use cases for those tools/Techniques

• Disk Replication

• Backup/Restore

• CDC and Q-Rep

• Federation and Load from cursor

• DMT/MTK

• HPU

• Export / Unload

• Data Studio

• db2move

11

Tools/Techniques

12

Technique Manpower Speed Downtime Architecture Change Endieness

Product Change

Feature/Tool Cost

Disk Replication Small Fast Small No No No Disk Feature

Backup /Restore Small Medium Medium No No No None

CDC and Q-Rep High Medium

None to small Some Yes Yes Sometimes

Federation and Load from cursor Medium Medium

Small to Large Yes Yes Some None

DMT/MTK Medium Slow Medium Yes Yes Yes None

HPU Small Fast Medium Yes Yes Yes High

Export / Unload Medium Slow High Yes Yes Yes None

Data Studio Medium Slow High Yes Yes Yes None

Db2move Medium Slow Medium Yes Yes Yes None

Testing and Validation

• Define TESTING Success Criteria

• Schema Migration • DB Platform Change

• DB2 version upgrades

• Version Feature Exploitation

• Data Migration • Row Counts and/or Lower Level validation

• Workload Validation • Test Key Workloads prior to Go Live

• ETL and Exports

• Reporting (Cognos, Business Objects)

• Ad Hoc

• Parallel Run Required ?

13

14

Manual Migrations (Table by Table)

Manual Migrations – Table by Table

15

• Table by Table migrations are most flexible but most effort

• Must not forget non-table objects (SPs, etc)

• Multiple techniques • Export/Load

• HPU

• Federated Load from cursor

• Tools like MTK, DMT, db2move and Data Studio

• Flexibility is big plus • Change Physical Characteristics (Tablespaces, Pagesize, etc)

• Can include schema changes

• Simplicity drives success

Manual Migrations

16

• Managing the migration is the hardest part

• Must manage “Drift” during Testing

• Must track table migrations with AUTOMATION

• Record (Learn) as much as you can in test migrations • Time (seconds to migrate each table)

• Rows Per Second per Table

• Bytes Per Second per Table

• CPU, IO and NW usage during testing (allows you to scale up more concurrent data moves)

• Plan the Testing in Detail

The Need for Speed

17

• Speeding up Migrations • Move Static Data Ahead of Time

• Focus on longest running table copies

• First estimates of longest are often wrong

• Concurrency is key

• LOAD is Target Based CPU bound

• Alternative copy techniques • Range partition by partition

• DB partition by partition

• HPU from backup images

Migration Tips and Techniques

• DRIFT: Consider and Plan to Track Source Side changes (“drift” from the initial conversion) while testing is ongoing

• Schema Changes

• DBA scripts

• Application

• ETL changes

18

Select tabschema,tabname,type ,CREATE_TIME,ALTER_TIME from syscat.tables where (date(CREATE_TIME) > '10/1/2014' OR date(ALTER_TIME) > '10/1/2014') order by tabschema,tabname

Migration Tips and Techniques

• Dealing with Indexes

• Fastest to Create Indexes during Load commands

• Idea: To allow testers access to the data faster, do not create all indexes during load. Only create primary keys or unique indexes to ensure no duplication. Create secondary indexes while testers are validating data

• Unique indexes on Range Partitioned Tables cause locking if loading by DB Partition

• Referential Integrity

• Wait and run SET INTEGRITY FOR <table> ALL IMMEDIATE UNCHECKED after loads

• Generated and Identity Columns

• Requires LOAD command option

• Must reset max values (db2look generates

• Dealing with Stored Procedures, Functions, Triggers

• External SPs require that the executable code be moved as well

• Cross Platform usually requires rewrites to some degree

• Rebind after migration

19

Identity, Generated and Rowchangetimestamp Columns

• Generated Columns require different options on LOAD command. • Identity columns

• Requires LOAD command option • Must reset max values: alter table TAB1 alter column C1 RESTART WITH < max

+ 1 > • Query SYSCAT.COLUMNS to identify tables with Identity, Rowchangetimestamp and

Generated Columns • Use MODIFIED BY GENERATEDOVERIDE , IDENTITYOVERRIDE or

ROWCHANGETIMESTAMP as needed per table

20

select tabschema,tabname ,case when generated = ‘ ‘ then ‘N” else generated end as generated ,identity ,rowchangetimestamp from syscat.columns c group by tabschema,tabname

Migration Tips and Techniques

• Schema Validation Important

• There can be legitimate differences in some migrations

• Federated DBs allow queries between catalogs

• Data Studio and other 3rd Party tools provide Schema comparisons

21

High Performance Unload (HPU) Features:

22

• Unload DB2 data to flat files, tapes, or named pipes

• Supports multiple formats for output files

• Can perform compete system migration

• HPU’s column function supports conversion to wide range of data types

HPU:

23

• Highly threaded to exploit all available system processors

• Complete system migration by unloading, transferring, and loading data on target hosts

• Flexible partition support; can create files on each remote host rather than one file on one system

• Reads TS container files directly; does not displace data in bufferpools

• Bypasses database manager

• Can unload data from tablespace containers or backup files

High Performance Unload

24

• In DPF environment, you can unload table data on all partitions with single execution. Can unload data to a single output file or multiple files, either on the local nodes or on the current nodes

• HPU executed with db2hpu command, either by command line processing or specifying a control file, which contains all of the desired options.

• Database, format, partition, table, catalog block, unload block

• Unload processed by DB2 method • NO – if the order cannot be processed directly by HPU, it fill fail

• YES – If SELECT statement is too complex for HPU, it will be handled by DB2

• FORCE- DB2 Must be used to extract the rows (cannot be used unloading from backup)

25

Migration Experiences

Migration Example 1: Intel BCU to Power DPF

• Challenge: Intel BCU to DPF on Power

• Intel to AIX

• 45TB of Data (DB2 9.7 Static) 112TB uncompressed data

• 56 Database Partitions

• 2,900+ Tables

• Applications and ETL must be tested

• Downtime target 60 hours but must allow R/O access during

migration

• No Static Data could be identified

• Solutions Considered

• Texas Two-Step Migration could not be used

• Solution Chosen

• Federated Load using a “Temp Server”

26

Solution Design Considerations

• Must change Endianness Change

• Table by Table Migration Required

• HPU not owned (cost issue)

• Replication considered, but ruled out

• Complex number of tables

• Previously experienced problems

• Large number of transactions ( TBs/week)

• Simplify Migration

• No product version change (Stay at 9.7)

• No architecture change (keep same number of DB Partitions)

• Automation of table copies must be implemented

• Scripted federated load selected

27

Solution Design:

28

- “Temp Server” configured with 8 Servers (OS Images) in DB2 Cluster. Each Server has:

1. 8 Servers clustered using DPF

2. 10Gb connection to DB2 FCM Switch

3. 10Gb Ethernet Card connecting to power AIX Server

- Power Server 3 LPARs (1 admin and 2 Data )

4. Power Server connects to Temp Server with 8 x 10Gb paths

5. Power Server Multiple LPARs on 2 physical servers

Intel Temp Server

Power Server

DB Space

DB Space

AIX DB2

Intel DB2

1

3

SAN

2 10Gb

DB2 FCM Switch

4 8 x

10Gb

5

29

1. Backup Taken on BCU of Source DB

2. Restore BCU DB Backup to SAN Disk on Temp Server.

3. After Backup Completes, Log Shipping begins to Temp Server

4. Log Shipping brings Temp Server DB consistent with BCU DB;

FREEZE or Go Into Read-Only on Prod for Go-live migration

5. Power DB2 populated with Load from Cursor & Partition to Partition Table copies

6. FMTMT Validates that AIX DB and Intel DB data is equivalent

BCU (intel)

Intel Temp Server

Power Server

Log Shipping

DB Space

DB Space

Log Space

AIX DB2

Intel DB2

backup

3

1

2

4

5

6

SAN

restore 4 to 8 x 10Gb

Solution Implemented:

• Implemented with Migration Tool

• Create DB2 tables pre-populated with source and target

database and tables, cardinality, fpages and statistics collection

columns

• Databases Federated

• Load scripts:

• Full Table load from cursor

• Single DB Partition of table load

• Each script tracks progress in DB2 tables

• Small tables loaded with single full table load from cursor

• Large tables loaded by loading all DBPs in parallel

30

Solution Implemented:

• Sophisticated Set of Tables to track progress and Scripts

to implement Actions

• All parameter driven and SQL based reporting

• Schedules, Executes and Reports SQL codes for:

• Data Copies (Loads)

• Index Builds

• Primary Key additions

• Constraints additions

• Runstats executions

• Reorg executions

31

Actual “Back-of-Napkin” Stream-Processor Design:

Identify next table to load

Shell Count of Src table

Shell Load to Physical

Node to run

Post Process

Update Status

Repeat!

32

Partition by Partition Loads

33

Solution Implemented:

• “Stream Servers” used to run multiple table-loads

concurrently

• Each Stream server :

• Processes the next Table in the queue (a db2 table)

• Run multiple Stream Servers to load multiple tables concurrently

• Stream Servers wait for last partition before initiating next table

• Developed Partition Stream-Servers to load One DB

Partition of a table and move onto another partition of

same or different table

• Adjusted Load_Stats table for future runs

34

Results

35

OBJECT_TYPE DB1_Dev DB1_Test DB2_Dev DB2_Test DB2_Prod

foreign_keys 259 553 499 969

fucntions 151 140 90 79

indexes 4,548 5,245 3,174 12,124 6,233

tables 4,540 4,623 4,400 15,596 4,114

primary_keys 3,155 3,301 2,702 11,257

procedures 643 706 732 1,366

table_constraints 4 53 25 11

rows_migrated 8,132,028,629 11,612,428,211 8,772,769,798 13,788,006,093 152,703,121,053

load_jobs 4,396 4,976 4,409 58,794 225,086

Minutes 398 1,271

Rows/Minute 34,643,231 120,106,277

~42TB compressed data in 22 hours

Lessons Learned:

• Check Table Compression after migration, Reorg if need

• Build indexes, Primary Keys, Foreign Keys, Constraints

after table loads – minimize locking on catalog when

multi-DB-partitions load concurrently

• Capture VMSTAT or SAR output, monitor real time and

add stream processors if CPU is available

• Heaviest CPU consumption is on Target Machine

• With DPF, you must compare PMAPS. If different, load

source PMAP into Target before creating objects

36

Lessons Learned 2:

• Adjusted Load_Stats table after Rehearsal Runs

• Enable FCM Parallelism

• Enable FETCH Parallelism

• Increase SHEAPTHRES_SHR, SORTHEAP and

UTIL_HEAP_SZ

• Table Ordering matters

• Use rehearsals to identify longest running migrations

• Use “streams” to increase Parallelism

• ssh load scripts to physical machine where target DBP resides

• Target Machine consumes much more CPU than source

• Monitor Target system CPU to drive utilization

37

Lessons Learned 2:

• Avoid recreating tables just prior to Cut-over migration

• Use KEEP DICTIONARY – speeds data load

• Keep Statistics collected during rehearsal runs

• Range Partition tables with large number of partitions

and DB Partitions take a long time to create

• Where feasible, use ALTER table to propagate Schema

changes rather than DROP/CREATE

38

Migration Example 2: DPF to Non-DPF

• Challenge:

• BCU (DPF) to non-DPF migration

• Intel to Power Architecture change

• Applications and ETL needed to run in parallel for 2 weeks

• Downtime allowed 12 hours

• Solution

• “Texas Two-Step” Migration

• Migrate data with last change timestamp older than 3 weeks

• Cutover weekend, catchup of last 3 weeks changes

• Rehearsed Migration and Parallel ETL Run

• Discovered Surrogate Key generated with different values on

each system – caused by lack of result set ordering in ETL query

processing

39

Migration Example 3: Architecture Changing

• Challenge: 12 Partition DPF to BLU (underway)

• Number of Partitions Changes to 1

• Row organized to Column Organized

• Power to Power

• Downtime allowed - none

• ETL timing biggest conversion effort 100s of complex exports to

feed BI data marts

• All Update/Insert/Delete in Nightly ETL only

• Solution

• Backup Restore not option due to DPF to Non-DPF and BLU

• Table by Table copy only choice

• Load time not an issue as ETL load files can accumulate during cut-

over

• Parallel ETL run for 2 to 4 weeks to synch systems

40

Migration Example 3:

• Findings to date:

• Currently some restrictions in BLU

• LOB and XML data not supported in COLUMN tables

• HADR ROS not supported

• NOT LOGGED INITIALLY not supported

• NO Enforced RI

• No column constraints

• Some SQL Syntaxes work better than others

• SUBSTR on Column data forces CTQ

• OLAP functions (eg Row_number) force CTQ

• Left Outer Join better than NOT Exists/Correlated Subselect

• Compression on large tables EXCELLENT

41

Findings to date:

42

TABNAME COMP TORG CARD FPAGES Savings PGSZ EXTSZ

Table1_COL X C 50,000,000 33782 13.6% 32768 4

Table1_8_16_ADAPTIVE R R 50,000,000 113088 48.1% 8192 16

Table1_8_16_STATIC R R 50,000,000 231056 100.0% 8192 16

~70% of extracts faster than DPF

15% similar

15% drastically slower

Rewrite of 7 sample slow queries has them faster than DPF now

James Sobieski Fourth Millennium Technologies james@fmtusa.com

Session: C20 DB2 Migrations: Planning, Preparing and Executing Migrations to DB2 LUW

Please fill out your session

evaluation before leaving!