10g data pump

8/2/2019 10g Data Pump

1/47

Oracle 10g Data Pump

Say good bye to exp and imp (or not)!2006/02/23

Simon Pane


2/47

2

Agenda

Data Pump Overview

Using Data Pump

Demonstration

Data Pump Test Cases


3/47

3

Data Pump Overview


4/47

4

What is Data Pump?

A replacement of the traditional export/import utilities?

The evolution of the traditional export/import utilities?

A completely new 10g utility serving a similar yetslightly different purpose?


5/47

5

Other Options for Moving Data

Traditional Export and Import Pros

Easy to use most DBAs have years of experience using theseutilities

Versatile various options available; can specify what to include

Platform independent

Serial output

Cons

Comparatively slow

Can be network intensive

Non-interruptible / resumable

Limited filtering options (for example, can exclude just VIEWS)

Limited remapping options (i.e. from one tablespace to another)


6/47

6

Other Options for Moving Data

Transportable Tablespaces Pros

Undoubtedly the fastest way to move data

Can use the traditional exp/imp or Data Pump to move meta-data

Cross-platform support if the platform byte-order is the same

Cons

Tablespaces must be made read-only

Not selective (must move the entire tablespace)

Flashback is not possible (tablespace is read only when copied)

No physical reorganization is performed

Datafile sizes remain constant

Must use RMAN to convert the datafile if migrating to a platform witha different byte-order (checkV$TRANSPORTABLE_PLATFORM)


7/477

Other Options Used Less Frequently

Extraction to a flat file and loading using SQL Loader Direct copy using database links (SQL Plus COPY

command)

Oracle Streams

3rd Party data ETL or reorg tools


8/478

Top 10 Reasons to Love DataPump

10. Similar look and feel to the old exp/imp

9. Can filter on the full range of object types

8. Can re-map datafiles and or tablespaces on import

7. Estimates the export file size (space needed)

6. Parallelizable

5. Significantly faster than the traditional exp/imp

4. PL/SQL Interface programmable

3. A file is not actually required - can import through anetwork link

2. Track in v$session_longops

1. Resumable (interruptible and restartable)


9/47

9

Top 10 Reasons Not to Love Data Pump

10. Still generates redo (unlike direct path inserts)

9. Aggregation of exported data is not possible (sort only)

8. Performance on the server

7. Harder to tell what its doing at any given time

6. No equivalent to the STATISTICS option5. Cannot be used with sequential media such as tapes and

pipes (not read/written serially)

4. Only accesses files on the server, never the client

3. Oracle directories are required in the DB to access the files

2. Does not support COMMIT on imp or CONSISTENT on exp

1. If constraints are violated on import, the load is discontinued


10/47

10

Operation Fundamentals

Export/Import These utilities would basically connect to the Oracle database

via Oracle NET and run queries or DDL/DML

Processing of returned results and I/O operations were done onthe client

Data Pump

The executables call PL/SQL APIs

Therefore processing is done on the database server

This can be an advantageor a disadvantagedepending on thesituation

Self-Tuning: no longer need to use BUFFERor RECORDSET


11/47

11

Export Operation

Network

Export

File(s)

exp.exe

OracleDatabase


12/47

12

Data Pump Export Operation

Network

ExportFile(s)

expdp.exe

OracleDatabase


13/47

13

Key Differences

Dump and log files are on the server, not the client Must have a DIRECTORY created in the Oracle

database for I/O

Permissions for the useridconnecting to the instance, not theschemas being exported or imported

Canceling the client process does not stop the job

Doesnt automatically overwrite dump file if it alreadyexists returns an error instead

Parameters (command line) are reported in the log file

Exported objects order based on table size(descending) instead of alphabetically


14/47

14

Multiple Interfaces

1. Command line utilitiesexpdb

andimpdb

Similar to the familiar exp and imp in usage

Use HELP=Y for a list of commands

Oracle documentation provides a comparison table to exp/imp

2. Enterprise Manager3. PL/SQL

Can be used independently but is difficult

All of these call the DBMS_DATAPUMP API

Uses Oracle Advanced Queuing

Uses DBMS_METADATA


15/47

15

Unload Mechanisms

Data Pump automatically chooses to unload data eitherusing:

Direct path

External Tables (new driver called ORACLE_DATAPUMP)

Same External Tables mechanism that was introduced inOracle9i

When will it use External tables:

When parallelism can be used

When the table contains a complex data type or structure thatprevents direct path unloads

A lot of tables fall under this situation see Oracle documentation for acomplete list

It doesnt really matter to us which method is used


16/47

16

Multiple Processes

Master Control Process Spawns worker processes

Populates the master control table and log file

The master control table can be queried to track the jobsprocess

At the end of an export, the master control table is written to thedump file and dropped from the database

Worker Processes

Performs the loading/unloading

Number of processes depends on the degree of parallelism(the PARALLEL option)


17/47

17

Detaching and Re-Attaching

Issuing Ctrl-C

from the data pump import will

detach

Import is running on the server so it will continue

Brings you into interactive-command mode

To re-attach, run impdp with theATTACH= option Example: impdp userid=system/oracle attach=JOB_01

Brings you back into interactive-command mode


18/47

18

New Views

DBA_DATAPUMP_JOBSand

USER_DATABASE_JOBS Identify all jobs regardless of their state

Identify any master tables not associated with an active job

DBA_DATAPUMP_SESSIONS Identify user sessions that are attached to a job

Data pump sessions populate v$session_longops

Documentation says that it is 100% accurate for imports buttesting proves otherwise!!!


19/47

19

Security Considerations

Still uses theEXP_FULL_DATABASE

andIMP_FULL_DATABASE

A privileged user will have these two roles

A privileged user can:

Export/import objects owned by other schemas

Export non-schema objects (metadata)

Attach to, monitor, and control jobs initiated by others

Perform schema, datafile, and tablespace remapping

Similar to the traditional export/import

Supports label security

If exporting user has the EXEMPT ACCESS POLICY role


20/47

20

Object Statistics

From Oracle documentation regarding data pumpexports:

A parameter comparable to STATISTICS is not needed.Statistics are always saved for tables.

From Oracle documentation regarding data pumpimports:

A parameter comparable to STATISTICS is not needed. If thesource table has statistics, they are imported.


21/47

21

Other Random Points

Can still use a parameter file and thePARFILE

command line option

Fully supports Automatic Storage Management (ASM)

Can still flashback to a specified time or SCN

Can still extract (or backup) DDL (meta data) Using the SQLFILE option instead of the traditionalINDEXFILE or SHOWoptions

Full support of LOBS


22/47

22

Using Data Pump


23/47

23

Oracle Directory Objects

Must first create an Oracle directory object and give theuser who will be performing the Data Pump activitiespermission to use it (or rely on defaults):

SQL> create or replace directory dpump_demo as 'C:\temp';

Directory created.

SQL> grant read,write on directory dpump_demo to simon;

Grant succeeded.

SQL>


24/47

24

Key Data Pump Export Parameters

CONTENT={ALL | DATA_ONLY | METADATA_ONLY}

DIRECTORY=directory_object (default=DATA_PUMP_DIR)

DUMPFILE=[directory_object:]file_name[,]

ESTIMATE={BLOCKS | STATISTICS}

ESTIMATE_ONLY={Y | N}

EXCLUDE=object_type[:name_clause] [,]

FILESIZE=integer[B | K | M |G] (default=unlimited)

FLASHBACK_SCN=scn_value

FLASHBACK_TIME=TO_TIMESTAMP(time_value) FULL={Y | N}

INCLUDE=object_type[:name_clause] [,]

JOBNAME=jobname_string


25/47

25

Key Data Pump Export Parameters

LOGFILE=[directory_object:]file_name

NOLOGFILE={Y | N}

PARALLEL=integer(default=1)

QUERY=[schema.][table_name:]query_clause

SCHEMAS=schema_name[,]

TABLES=[schema_name.]table_name[:partition_name] [,]

TABLESPACES=tablespace_name[,]


26/47

26

Data Pump Export Parameter Samples

Multiple dump files using a substitution variable (%U):

DUMPFILE=DP_DIR1:SCOTT_20060223_%U.dmp

Excluding indexes that start with EMP:

EXCLUDE=INDEX:LIKE EMP%

Excluding the SCOTT schema from a FULL export:

EXCLUDE=SCHEMA:=SCOTT

Mimicking the traditional CONSITENT parameter:

FLASHBACK_TIME=TO_TIMESTAMP

Exporting only TABLES, FUNCTIONS and VIEWS: INCLUDE=TABLE,FUNCTION,VIEW

Using a query clause

QUERY=emp:WHERE salary > 100000


27/47

27

Key Data Pump Import Parameters

CONTENT={ALL | DATA_ONLY | METADATA_ONLY}

DIRECTORY=directory_object (default=DATA_PUMP_DIR)

DUMPFILE=[directory_object:]file_name[,]

EXCLUDE=object_type[:name_clause] [,]

FULL={Y | N}

INCLUDE=object_type[:name_clause] [,]

JOBNAME=jobname_string

LOGFILE=[directory_object:]file_name

NOLOGFILE={Y | N} PARALLEL=integer(default=1)

QUERY=[schema.][table_name:]query_clause


28/47

28

Key Data Pump Import Parameters

REMAP_DATAFILE=source_datafile:target_datafile

REMAP_SCHEMA=source_schema:target_schema

REMAP_TABLESPACE=source_tablespace:target_tablespace

REUSE_DATAFILES={Y | N}

SCHEMAS=schema_name[,]

SKIP_UNUSABLE_INDEXES={Y | N}

SQLFILE=[directory_object:]file_name

TABLE_EXISTS_ACTION={SKIP|APPEND|TRUNCATE|REPLACE}

TABLES=[schema_name.]table_name[:partition_name] [,] TABLESPACES=tablespace_name[,]


29/47

29

Interactive Mode Commands

ADD_FILE=[directory_object:]file_name[,]

CONTINUE_CLIENT

EXIT_CLIENT

FILESIZE=number

KILL_JOB

PARALLEL=integer

START_JOB

STATUS

STOP_JOB


30/47

30

Demonstration

S S


31/47

31

Exporting and Importing Sample Schemas

expdp system/oracle@ORA1020dumpfile=scott.dmp schemas=scott

impdp system/oracle@ORA1020

dumpfile=scott.dmp schemas=SCOTTremap_schema=SCOTT:LARRY

expdb system/oracle@ORA1020

dumpfile=larry.dmp schemas=larry

SELECT * FROM DBA_DATAPUMP_JOBS;

U i I t ti M d


32/47

32

Using Interactive Mode

Ctrl-C to detach from the current export

Export> status

Export> stop_job

expdp system/oracle@ORA1020attach=SYS_EXPORT_SCHEMA_01

Export> start_job

Export> exit_client


33/47

33

Data Pump Test Cases

T t S i #1


34/47

34

Test Scenario #1

Generated sample SIEBEL data

Brushed off the dust on some old SIEBEL data populationscripts (circa 07/2000)

Designed for SIEBEL 6 on Oracle 8.1.6

Actual data is not important

Schema objects

Tables: 218 (many empty tables)

Indexes: 1180 (SIEBEL is a heavily indexed application)

Schema size (from DBA_SEGMENTS) Tables: 1255MB

Indexes: 148MB

E t P f T t C it i


35/47

35

Export Performance Test Criteria

All work performed on laptop C681

SGA remains constant

SGA_TARGET=0

SGA_MAX_SIZE=256MB

BUFFER CACHE=152MB SHARED_POOL=60MB

Mostly default parameters

Not monitoring CPU utilization

Performed 4 runs

Disregarded results from 1st run and averaged the other 3

E t S i t


36/47

36

Export Scripts

EXP

sqlplus -s system/oracle @timestamp.sql > exp.log

exp.exe userid=system/oracle@ORA1020 file=SIEBEL.dmplog=SIEBEL.log owner='SIEBEL'

sqlplus -s system/oracle @timestamp.sql >> exp.log

EXPDP

erase SIEBEL.dmp

sqlplus -s system/oracle @timestamp.sql > expdp.log

expdp.exe userid=system/oracle@ORA1020dumpfile=SIEBEL.dmp logfile=SIEBEL.logschemas='SIEBEL' directory=test_dir

sqlplus -s system/oracle @timestamp.sql >> expdp.log

E t P f T t R lt


37/47

37

Export Performance Test Results

exp expdp

Average ExportTime

6:02 3:18

Estimated File Size N/A 1.201 GB

Actual File Size 965 MB 621MB

I t S i t


38/47

38

Import Scripts

IMP


imp.exe userid=system/oracle@ORA1020 file=SIEBEL.dmplog=SIEBEL.log fromuser='SIEBEL' touser='SCOTT'commit=y


IMPDP


impdp.exe userid=system/oracle@ORA1020dumpfile=SIEBEL.dmp logfile=SIEBEL.logschemas='SIEBEL' directory=test_dirremap_schema=SIEBEL:SCOTTremap_tablespace=TOOLS:SCOTT_DATA


I t P f T t R lt


39/47

39

Import Performance Test Results

imp impdb

Average Import Time 27:07

Average Import Time (no indexes) 25:19

Average Import Time (no rows) 27:27

Database was inARCHIVELOG mode

Destination tablespace and archived log destinationwere both on ASM drives

Machine performance was degraded much more byimpdb import

No import tuning performed (only COMMIT=Y)

Test Scenario #2


40/47

40

Test Scenario #2

Data taken from an actual CGI internal application

Schema objects

Tables: 22

Indexes: 26

Schema size (from DBA_SEGMENTS) Tables: 300MB

Indexes: 101MB

Export Scripts


41/47

41

Export Scripts

EXP


exp.exe userid=system/oracle@ORA1020 file=SCOTT.dmplog=SCOTT.log owner='SCOTT'


EXPDP

erase SIEBEL.dmp


expdp.exe userid=system/oracle@ORA1020dumpfile=SCOTT.dmp logfile=SCOTT.log schemas='SCOTT'directory=test_dir




42/47

42


exp expdp

Average ExportTime

1:24 1:32

Estimated File Size N/A 290MB

Actual File Size 261MB 233MB

Import Scripts


43/47

43

Import Scripts

IMP


imp.exe userid=system/oracle@ORA1020 file=SCOTT.dmplog=SCOTT.log fromuser='SCOTT' touser='LARRY' commit=y


IMPDP


impdp.exe userid=system/oracle@ORA1020

dumpfile=SCOTT.dmp logfile=SCOTT.log schemas='SCOTT'directory=test_dir remap_schema=SCOTT:LARRYremap_tablespace=SCOTT_DATA:LARRY_DATA




44/47

44


imp impdb

Average Import Time 5:48 2:26

Database was inNOARCHIVELOG mode

Destination tablespace and archived log destinationwere both on ASM drives

No import tuning performed (only COMMIT=Y)


45/47

45

Conclusions

Conclusions


46/47

46

Conclusions

Data Pump is an exciting new Oracle 10g tool thatprovides many benefits over the traditional export andimport utilities

Whether to use Data Pump, Transportable Tablespaces,or even the traditional exp/imp will depend on the situation

Since the command line interface is easy to use and sosimilar to the traditional exp/imp, DBAs and developersshould spend the time to learn how to use it

Final thought: since Data Pump dump files and traditionalexport dump files are not compatible/interchangeable,should a new file extension be used??? (.dmp vs .dpd)

The End


47/47

The End

Comments, Questions ???

10g data pump

Documents