6-1

11Oracle Data IntegratorData Quality (Integrity Control)

6-2

Objectives

• Know the different types of data quality business rules ODI manages.

• Be able to enforce data quality with ODI.• Understand how to create constraints on datastores.

After completing this lesson, you will:

6-3

When to Enforce Data Quality?

• The IS can be broken into 3 sub-systems• Source application(s)• Data integration process(es)• Target application(s)

• Data Quality should be managed in all three sub-systems

• ODI provides the solution for enforcing quality in all three.

6-4

Data Quality Business Rules

• Defined by designers and business analysts• Stored in the Metadata repository• May be applied to application data• Defined in two ways:

• Automatically retrieved with other metadata• Rules defined in the databases• Obtained by reverse-engineering

• Manually entered by designers• User-defined rules

6-5

From Business Rules to Constraints

• De-duplication rules• Primary Keys• Alternate Keys• Unique Indexes

• Reference rules• Simple: column A = column B• Complex: column A = function(column B, column C)

• Validation rules• Mandatory Columns• Conditions

6-6

Source

CORRECTIONSFile

ORDERS

LINES

Target

SALES

Overview of the Data Quality System

Integration Process

Error Recycling is performed- by Interfaces

Static Control is started- by Interfaces after integration- by Packages- manually

Static Control is started- Automatically (scheduled)- manually

Errors

Flow Control is started- by Interfaces during execution

Errors

6-7

Constraints in ODI

• Mandatory Columns• Keys

• Primary Keys• Alternate Keys• Indexes

• References• Simple: column A = column B• Complex: column A = function(column B)

• Conditions

6-8

1. Double-click the column in the Models view.

2. Select the Control tab.

3. Check the Mandatory option.

4. Select when the constraint should be checked (Flow/Static).

Mandatory Columns

6-9

1. Select the Constraints node under the datastore.

2. Right-click, select Insert Key.3. Fill in the Name.4. Select the Key or Index Type5. Go to the Columns tab6. Add/remove columns from the

key.

Keys

6-10

1. Go to the Control tab.2. Select whether the key is

Defined in the Database, and is Active

3. Select when the constraint must be checked (Flow/Static).

4. Click the Check button to perform a synchronous check of the key.

Checking Existing Data with a New Key

Number of duplicate rows

6-11

1. Select the Constraints node under the datastore

2. Right-click, select Insert Reference3. Fill in the Name4. Select the reference type

• User Reference• Complex Reference

5. Select a Parent Model and Table• Set the model and table to

<undefined> to manually enter the catalog, schema and table name.

Creating a Reference

6-12

1. Go to the Columns tab

2. Click the Add button

3. Select the column from the Foreign Key table.

4. Select the corresponding column from the Primary Key table.

5. Repeat for all column pairs in the reference.

Creating a User Reference

6-13

1. Go to the Expression tab

2. Set the Alias for the Primary Key table.

3. Code the Expression• Prefix with the tables aliases• Use the

Expression Editor.

Creating a Complex Reference

6-14

1. Go to the Control tab.

2. Choose when the constraint should be checked (Flow/Static).

3. Click the Check button to immediately check the reference.

• Not possible for heterogeneous references.

Checking Existing Data with a New Reference

6-15

1. Right-click Constraints node, select Insert Condition

2. Fill in the Name.3. Select ODI Condition type.4. Edit the condition clause

• Use the Expression Editor

5. Type in the error message for the condition.

Creating a Condition

6-16

1. Go to the Control tab

2. Select when the constraint must be checked (Flow/Static).

3. Click the Check button to perform a synchronous check of the condition.

Checking Existing Data with a New Condition

6-17

How to Enforce Data Quality in an Interface

The general process:1. Enable Static/Flow Control2. Set the options3. Select the Constraints to

enforce• Table constraints• Not null columns

4. Review the erroneous records

6-18

How to Enable Static/Flow Control

1. Go to the interface’s Flow tab. 2. Select the target datastore.

The IKM properties panel appears.

3. Set the FLOW_CONTROL and/or STATIC_CONTROL IKM options to “Yes”.

4. Set the RECYCLE_ERRORS to “Yes”, if you want to recycle errors from previous runs

6-19

How to Set the Options

1. Select the interface’s Controls tab.

2. Select a CKM.3. Set up the CKM Options.4. Set the Maximum Number of

Errors Allowed.• Leave blank to allow an

unlimited number of errors. • To specify a percentage of the

total number of integrated records, check the % option.

6-20

How to Select Which Constraints to Enforce

For flow control:• For most constraints:

1. Select the interface’s Controls tab.

2. For each constraint you wish to enforce, select Yes.

• For Not Null constraints:1. Select the interface’s Diagram tab. 2. Select the Target datastore column that

you wish to check for nulls.3. In the column properties panel, select

Check Not Null.

6-21

How to Review Erroneous Records

First, execute your interface.To see the number of records:1. Select the Execution tab.2. Find the most recent execution.

• The No. of Errors encountered by the interface is displayed.

To see which records were rejected:1. Select the target datastore in

the Models view.2. Right-click > Control > Errors… 3. Review the erroneous rows.