hello

Information Steward 4.0 Information Steward 4.0

Cleansing Package Builder

Venkata Ramana Paidi

© 2

012

Uto

pia,

Inc

. A

ll R

ight

s R

eser

ved.

1. Overview of Cleansing Package Builder/Positioning2. Targeted Personas3. Impact of Cleansing Package Builder on Data Cleanse4. Cleansing Package Builder Roles, Components, and Architecture 5. Cleansing Package Builder Requirements6. Additional Cleansing Packages with Data Services 4.07. Cleansing Package Builder Workflow8. Explore Cleansing Package Screen9. Create a new Cleansing Package Wizard - Design Mode10.Edit Existing Cleansing packages11.Use Advanced Mode12.Publish a Cleansing Package13.Export and Import from LCM

Agenda

© 2

012

Uto

pia,

Inc

. A

ll R

ight

s R

eser

ved.

Positioning: What is Cleansing Package Builder?

A tool that creates a Cleansing Package that Data Services’ Data Cleanse transform uses parses, standardizes and cleanse business data

Such as account numbers, product codes, product descriptions, purchase dates, part numbers, SKUs, and so on.

Provides user interface that allows Data Steward to visualize how their data is parsed and standardized, and evaluate the impact of their customized changes

Provides ability to read and write Unicode data

Creates a Cleansing Package that will be used In Data Services’ Data Cleanse transform that will parse, standardize and cleanse party data such as names, firms, titles, emails, phone numbers, SSNs, and dates

Allows Data Stewards to customize standard forms based on the company’s data and standards

Note: Cleansing Package Builder is delivered as part of Data Services, but is installed as a component of Information Steward

© 2

012

Uto

pia,

Inc

. A

ll R

ight

s R

eser

ved.

Positioning: Why Was Cleansing Package Builder Created?

The primary drivers behind the creation of Cleansing Package Builder were: Empower the data steward/subject matter expert to develop a data cleansing

solution Allow the user to

Easily and quickly develop new data cleansing solutions for data domains SAP does not provide out of the box

Product data, for example: Pharmaceutical data Financial data

Customize the cleansing packages SAP delivers to our customers The data steward provides insight about how the data should be classified simply based on

the desired output Cleansing Package Builder automatically creates the data dictionary, rules, and patterns that

make up a cleansing package, which is then consumed by Data Services Data Cleanse transforms

© 2

012

Uto

pia,

Inc

. A

ll R

ight

s R

eser

ved.

Positioning Cleansing Package Builder: Business Example

Input Data

Glove ultra grip profit 2.3 large black synthetic leather elastic with Velcro Mechanix Wear

Parsed Output

Product Category Glove

Size Large

Material Synthetic Leather

Trademark Pro-Fit 2.3 Series

Cuff Style Elastic Velcro

Palm Type Ultra-Grip

Color Black

Vendor Mechanix Wear

Standard Description Glove – Synthetic Leather, Black, size: Large, Cuff Style: Elastic Velcro, Ultra-Grip, Mechanix Wear

© 2

012

Uto

pia,

Inc

. A

ll R

ight

s R

eser

ved.

Target Personas

Cleansing Package Builder is designed to target the following personas: Data Stewards

Act as conduits between IT and the business portion of a company with both decision support and operational help

Subject-Matter / Domain Experts Line of Business wants a friendly environment to collaborate with IT Know what the data should look like

Business users have different expectations for user experience than IT Business users do not want to learn programming or scripting languages Business users want direct comparison of “before” and “after” data

Previous Cleansing Package Developers Advanced mode

© 2

012

Uto

pia,

Inc

. A

ll R

ight

s R

eser

ved.

Cleansing Package Builder Customer Benefits

The major customer benefits of Cleansing Package Builder are: Business-Oriented: Intuitive point-and-click and drag-and-drop user experience;

no rules or languages to master Data Agnostic: Setup to cleanse both product/operational and party data Ease of Use: Wizard generates default starting points of attributes, data

standards and corresponding rules Results Driven: Fine tune through an iterative process based on actual output

© 2

012

Uto

pia,

Inc

. A

ll R

ight

s R

eser

ved.

Data Cleanse: Removal of Dictionary Menu Option

Removal of Dictionary menu option in Data Services (DS) Designer: Search Creating or deleting a dictionary Adding, editing, and deleting an Entry, Output, or Classification

© 2

012

Uto

pia,

Inc

. A

ll R

ight

s R

eser

ved.

Data Cleanse: Transform – Options Tab

Data Cleanse Transform - Options tab:Reference files and parsing dictionary all combined into one parameter

Cleansing package will include: Dictionary data information/Parsing Dictionary Reference files (rule, email, international phone, social security file and user defined pattern files)

Data Services 3.2: Data Services 4.0:

© 2

012

Uto

pia,

Inc

. A

ll R

ight

s R

eser

ved.

Data Cleanse: Removal of Data Cleanse Tab in the View Data

Removal of Data Cleanse tab in the View Data from Writer transform

© 2

012

Uto

pia,

Inc

. A

ll R

ight

s R

eser

ved.

BDB

Cleansing Package Builder Integration with Data Services Engine

Data Cleanse / Universal Data

Cleanse

Information Steward

Target Data

CMS Repository

Data Services

FRS

BDB: Berkley Database (used Data Cleanse) DQ: Data QualityBI: Business Intelligence FRS: File Repository ServerCMS: Central Management Server IS: Information Steward CPB: Cleansing Package Builder UDC: Universal Data Cleanse DC: Data Cleanse

Source Data

Sample

Includes rules,

reference data, output

fields

Includes rules,


fields

SQLite BDB

BI Platform

Cleansing PackageBuilder

Sample Data

BDB

© 2

012

Uto

pia,

Inc

. A

ll R

ight

s R

eser

ved.

BDB

Cleansing Package Builder Integration with Data Services Engine

Data Cleanse / Universal Data

Cleanse

Information Steward

Target Data

CMS Repository

Data Services

FRS

Source Data

Sample

Includes rules,


fields

Includes rules,


fields

SQLite BDB

BI Platform

Cleansing PackageBuilder

Sample Data

BDB

DC verifier will query the BI Platform to get the list of published Cleansing Package names (BOE InfoObjects).During runtime of the DS job DC will download the required BDB files (delta or full) from FRS:

DC queries user specified publish CP name to retrieve version, number of files, and so on Checks the file system to see if the BDB file is already downloaded or not

If the BDB file does not exist, it will download full BDB file (LE or BE depending on the OS)If the BDB file exist, then it will compare the published version string and download the delta file to synch

© 2

012

Uto

pia,

Inc

. A

ll R

ight

s R

eser

ved.

Cleansing Package Builder Integration with Information Steward

Information Steward has a Cleansing Package Builder tab

© 2

012

Uto

pia,

Inc

. A

ll R

ight

s R

eser

ved.

Setting Up CPB Users in CMC

The Administrator can: Set up Cleansing Package Builder users in Central Management Console (CMC) Reassign a cleansing package to another user in CMC Delete a cleansing package in CMC Run Cleansing Package Builder Have all permissions and rights of a Cleansing Package Builder user

The Cleansing Package Builder user can: Create new cleansing packages Publish their own (private) cleansing package Create a copy (save as) of their own (private) cleansing packages Browse and import published cleansing packages Rename and delete their private cleansing packages

© 2

012

Uto

pia,

Inc

. A

ll R

ight

s R

eser

ved.

Cleansing Package Builder Requirements

Cleansing Package Builder 4.0 requirements include (following IS requirements): Browser and version

Internet Explorer 7 and 8 with Flash Player 9.0 or 10.0 Support platforms and versions

Windows Server 2003 SP1 64-bit, SP2 64-bit, R2 64-bit (SP2) Windows Server 2008 SP1 64-bit, SP2 64-bit, R2 64-bit Solaris 10 (SPARC) 64-bit AIX 5.3 (p-series) 64-bit, AIX 6.1 (p-series) 64-bit HP-Itanium v11.31 64-bit Linux (RedHat 5) 64-bit Linux (Suse 10) 64-bit, (Suse 11) 64-bit

Supported web services and versions SAP NW 7.2 WebLogic 9.2, 10 and 10.3 WebSphere 6.1 and 7 Tomcat 6.0 JBoss 4.2.3 and 5.0 WACS (aka Bobcat, BOBJ’s Tomcat)

© 2

012

Uto

pia,

Inc

. A

ll R

ight

s R

eser

ved.

Cleansing Package Builder Workflow: Set Up Data Cleanse Job in Data Services

Open Data Cleanse transform in Data Services Options tab

Select Published CP from dropdown box Contains data Contains rules

© 2

012

Uto

pia,

Inc

. A

ll R

ight

s R

eser

ved.

Cleansing Package Builder Workflow: Review Standardized Output

Run Data Cleanse job in Data Services View output data

© 2

012

Uto

pia,

Inc

. A

ll R

ight

s R

eser

ved.

Maintain Cleansing Package Verify output data Change CP

Continually tweak and update CP Import more sample data

Re-publish CP Run Data Cleanse Job

Build or refine a cleansing packageBuild or refine a

cleansing package

Publish the cleansing package


DATA SERVICESDATA SERVICES Create a job that includes the Data Cleanse transform

Create a job that includes the Data Cleanse transform

Configure the transform option to refer to the cleansing package

Configure the transform option to refer to the cleansing package

Run the job to cleanse your data

Run the job to cleanse your data

As necessary, refine the cleansing package

As necessary, refine the cleansing package

INFORMATION STEWARD

CLEANSING PACKAGEBUILDER

INFORMATION STEWARD

CLEANSING PACKAGEBUILDER

Cleansing Package Builder Workflow: Maintain Cleansing Packages

Build or refine a cleansing packageBuild or refine a

cleansing package



© 2

012

Uto

pia,

Inc

. A

ll R

ight

s R

eser

ved.

Cleansing Package Task Screen: New Cleansing Package

Create a New Cleansing Package Custom Cleansing Package

Wizard Sample input file Parsing Strategy Suggested Attributes Quick Start

Person and Firm Cleansing Package Wizard

Name, Description, Japanese Data, and Normalized Data

© 2

012

Uto

pia,

Inc

. A

ll R

ight

s R

eser

ved.

Cleansing Package Task Screen: Open an Existing Cleansing Package

Open an existing Cleansing Package My Cleansing Packages

Opens in Design mode Edit Cleansing Package Design mode Advanced mode

© 2

012

Uto

pia,

Inc

. A

ll R

ight

s R

eser

ved.

Cleansing Package Task Screen: Save As

Save As My Cleansing Package

Make a copy

© 2

012

Uto

pia,

Inc

. A

ll R

ight

s R

eser

ved.

Cleansing Package Task Screen: Publish

Publish a Cleansing Package My Cleansing Package

Publish your Cleansing Package Moves to Published Cleansing Packages Cleansing Package available for Data Cleanse

© 2

012

Uto

pia,

Inc

. A

ll R

ight

s R

eser

ved.

Cleansing Package Task Screen: Rename

Rename a Cleansing Package My Cleansing Package

Change name

© 2

012

Uto

pia,

Inc

. A

ll R

ight

s R

eser

ved.

Cleansing Package Task Screen: Delete

Delete a Cleansing Package My Cleansing Package

© 2

012

Uto

pia,

Inc

. A

ll R

ight

s R

eser

ved.

Cleansing Package Task Screen: Browse

Browse a Cleansing Package Published Cleansing Package

© 2

012

Uto

pia,

Inc

. A

ll R

ight

s R

eser

ved.

Cleansing Package Task Screen: Import

Import a Cleansing Package Published Cleansing Package

© 2

012

Uto

pia,

Inc

. A

ll R

ight

s R

eser

ved.

Cleansing Package Task Screen: Migrate Data Cleanse 3.2 Dictionary and Rule Files

Import Data Cleanse 3.2 Dictionary and Rule Files More menu option

© 2

012

Uto

pia,

Inc

. A

ll R

ight

s R

eser

ved.

Get Data Services ATL Published Cleansing Packages

Cleansing Package Task Screen: Generate Data Cleanse ATL

© 2

012

Uto

pia,

Inc

. A

ll R

ight

s R

eser

ved.

Cleansing Package Task Screen: Package Details

Hover on Cleansing Package name Cleansing Package details

My Cleansing Packages Published Cleansing Packages

© 2

012

Uto

pia,

Inc

. A

ll R

ight

s R

eser

ved.

Create a New Cleansing Package: Process

To create a new Cleansing Package: Custom Cleansing Process - Wizard six-step process

1. Import sample data2. Define sample data3. Select rows to analyze4. Determine which parsing strategy to use5. Select any out of the box suggestions that are provided based on your sample data6. Assign additional Attributes, Standard Forms and/or Variations to a category

© 2

012

Uto

pia,

Inc

. A

ll R

ight

s R

eser

ved.

Create a New Cleansing Package: Import Sample Data

Custom Package Step 1 of 6 (Name and Data)

Saves the data entered in a normalized form. There are full-width and half-width Latin characters and the normalized form will be saved.

Enables the Japanese parsing engine

Cleansing Package name needs to be an unique name of letters, numbers or underscore.

Select language to return suggested attributes, standard forms and variations to help build cleansing package.

© 2

012

Uto

pia,

Inc

. A

ll R

ight

s R

eser

ved.

Create a New Cleansing Package: Define Sample Data Definition

Custom Package Step 2 of 6 (Sample Definition)

© 2

012

Uto

pia,

Inc

. A

ll R

ight

s R

eser

ved.

Create a New Cleansing Package: Select Rows from Sample Data

Custom Package Step 3 of 6 (Select Rows)

© 2

012

Uto

pia,

Inc

. A

ll R

ight

s R

eser

ved.

Create a New Cleansing Package: Determine Which Parsing Strategy to Use

Custom Package Step 4 of 6 (Parsing Strategy)

© 2

012

Uto

pia,

Inc

. A

ll R

ight

s R

eser

ved.

Create a New Cleansing Package: Assign Attributes to a Category

Custom Package Step 5 of 6 (Suggested Attributes)

© 2

012

Uto

pia,

Inc

. A

ll R

ight

s R

eser

ved.

Create a New Cleansing Package: Define Attributes, Standard Forms, and Variations

Custom Package Step 6 of 6 (Suggested Attributes)

© 2

012

Uto

pia,

Inc

. A

ll R

ight

s R

eser

ved.

Create a New Cleansing Package: Person and Firm

Person and Firm Cleansing Package

© 2

012

Uto

pia,

Inc

. A

ll R

ight

s R

eser

ved.

Edit an Existing Cleansing Package: Options

To edit an existing Custom Cleansing Package in Design mode:1. Add Attributes2. Add values to Standard Forms and Variations3. Use suggested Standard Forms and Variations4. View records affected by Last User Action tab5. Use Search/Filter Panel tab6. Define Context7. Resolve Conflict8. Add additional rows to sample input9. Delete a row from the Input pane10. Define Format for Category

© 2

012

Uto

pia,

Inc

. A

ll R

ight

s R

eser

ved.

Edit an Existing Cleansing Package: Add Attributes

Custom Cleansing Package – Design mode Add

Unique Attribute name

© 2

012

Uto

pia,

Inc

. A

ll R

ight

s R

eser

ved.

Edit an Existing Cleansing Package: Add Values to Standard Forms and Variations

Custom Cleansing Package – Design Screen Add

Drag and Drop Import list Manually add

© 2

012

Uto

pia,

Inc

. A

ll R

ight

s R

eser

ved.

Edit an Existing Cleansing Package: Use Suggested Standard Forms and Variations

Custom Cleansing Package – Design Screen Use Suggestion list

© 2

012

Uto

pia,

Inc

. A

ll R

ight

s R

eser

ved.

Edit an Existing Cleansing Package: View Records Affected by Last User Action

Custom Cleansing Package – Design Screen Last User Action tab

© 2

012

Uto

pia,

Inc

. A

ll R

ight

s R

eser

ved.

Edit an Existing Cleansing Package: Use Search/Filter Panel

Custom Cleansing Package – Design Screen Search/Filter panel

© 2

012

Uto

pia,

Inc

. A

ll R

ight

s R

eser

ved.

Edit an Existing Cleansing Package: Define Context

Custom Cleansing Package – Design mode Define context based on sample data

© 2

012

Uto

pia,

Inc

. A

ll R

ight

s R

eser

ved.

Define format for context

Edit an Existing Cleansing Package: Define Context Format

© 2

012

Uto

pia,

Inc

. A

ll R

ight

s R

eser

ved.

Edit an Existing Cleansing Package: Resolve Conflict – Generating a Conflict

Generate Conflict

© 2

012

Uto

pia,

Inc

. A

ll R

ight

s R

eser

ved.

Edit an Existing Cleansing Package: Resolve Conflict - Wizard

Custom Cleansing Package – Design Screen Resolve Conflict

© 2

012

Uto

pia,

Inc

. A

ll R

ight

s R

eser

ved.

Edit an Existing Cleansing Package: Resolve Conflict – Conflict Resolution

Custom Cleansing Package – Design Screen Resolve Conflict

© 2

012

Uto

pia,

Inc

. A

ll R

ight

s R

eser

ved.

Edit an Existing Cleansing Package: Add Additional Sample Rows

Custom Cleansing Package – Design Screen Add More Sample Rows

© 2

012

Uto

pia,

Inc

. A

ll R

ight

s R

eser

ved.

Delete an Row from the Input Sample Records

Custom Cleansing Package – Design Screen Delete a Input Row

© 2

012

Uto

pia,

Inc

. A

ll R

ight

s R

eser

ved.

Edit an Existing Cleansing Package: Define Category Format

Format Category

© 2

012

Uto

pia,

Inc

. A

ll R

ight

s R

eser

ved.

Edit an Existing Cleansing Package: Define Category Format

Format Category Drag and Drop Attributes Remove Attributes Change Order Add Text

© 2

012

Uto

pia,

Inc

. A

ll R

ight

s R

eser

ved.

Edit an Existing Cleansing Package: View Category Format

View Category Format

© 2

012

Uto

pia,

Inc

. A

ll R

ight

s R

eser

ved.

Advanced Mode

Advanced mode enables you to: Search for values Manage Classifications and Entries Edit Rules Edit Reference Data Note:

Modifying or creating a Person and Firm Cleansing Package will automatically open in Advanced mode, there is no Design mode option with Person and Firm Cleansing Package. Adding/modifying/deleting any data in Advanced mode will not automatically generate or change any rule files.

© 2

012

Uto

pia,

Inc

. A

ll R

ight

s R

eser

ved.

Advanced Mode: Search for Values

Cleansing Package – Advanced mode Search

© 2

012

Uto

pia,

Inc

. A

ll R

ight

s R

eser

ved.

Advanced Mode: Manage Classifications and Entries

Cleansing Package – Advanced mode Manage Classifications and Entries

© 2

012

Uto

pia,

Inc

. A

ll R

ight

s R

eser

ved.

Advanced Mode: Rules Files

Cleansing Package – Advanced mode Add/Edit/Delete Rule

Auto Generated Rules Custom Rules

© 2

012

Uto

pia,

Inc

. A

ll R

ight

s R

eser

ved.

Advanced Mode: Rule Options

Cleansing Package – Advanced mode Rule - Options

Rename Rule Edit Description View History Create Copy Delete

Note: Modifying any of the rules, should only be done by an expert user. Changing the rules will affect how data is parsed and

standardized.

© 2

012

Uto

pia,

Inc

. A

ll R

ight

s R

eser

ved.

Advanced Mode: Rules History

Cleansing Package – Advanced mode Rule - View History

Revert Rule Pattern Definition Revert Rule Action

© 2

012

Uto

pia,

Inc

. A

ll R

ight

s R

eser

ved.

Advanced Mode: Edit Reference Data

Cleansing Package – Advanced mode Add/Edit/Delete Reference Data

Phone, Email, User-defined

© 2

012

Uto

pia,

Inc

. A

ll R

ight

s R

eser

ved.

Advanced Mode: Edit Reference Data

Cleansing Package – Advanced mode Social Security Reference Data

Import

© 2

012

Uto

pia,

Inc

. A

ll R

ight

s R

eser

ved.

Publish a Cleansing Package: Process

To publish a Cleansing Package:1. Navigate to the Project Task screen2. Select Cleansing Package (under My Cleansing Package)3. Click Publish4. Enter Published Cleansing Package name5. View published Cleansing Package in Data Services

© 2

012

Uto

pia,

Inc

. A

ll R

ight

s R

eser

ved.

Creating a Data Cleanse Transform

Create a Data Cleanse ATL that can be used in Data Services Publish Cleansing Package

Get Data Services ATL from More menu option Copy data in the text box and save with a .atl file extension

The atl will be a base atl based on the Cleansing Package settings including: Cleansing Package name Japanese engine enabled Whitespace only

© 2

012

Uto

pia,

Inc

. A

ll R

ight

s R

eser

ved.

Life Cycle Management Console

AMERICAS | EUROPE | MIDDLE EAST | ASIA PACIFIC | INDIA

hello

Documents

cleansing

existing cleansing

user action

view category

firm cleansing

define category

data cleanse

suggested