informatica profiling and discovery - 10.0 - profiling and ... library/1/0855... · the profiling...

Profiling and Data Discovery Transition from

Version 9.6.X to Version 10.0

© 1993-2015 Informatica LLC. No part of this document may be reproduced or transmitted in any form, by any means (electronic, photocopying, recording or otherwise) without prior consent of Informatica LLC. All other company and product names may be trade names or trademarks of their respective owners and/or copyrighted materials of such owners.

AbstractThe Profiling and Data Discovery Transition document is written for data analysts and developers to understand the new features that have been introduced in 10.0 and how the UI and features differ in the two releases.

Supported Versions• Profiling and Discovery 10.0

Table of ContentsCreate a Profile. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3

Profiles and Profile Results Overview in the Analyst Tool. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3

Create Profile Overview. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3

Create an Enterprise Discovery Profile. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6

Create a Column Profile with Data Domain Discovery. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8

Create a Data Domain Discovery Profile. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10

Add or Choose Data Objects Overview. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12

Data Object Overview in Version 9.6.X. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12

Data Objects Overview in Version 10.0. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13

Profiling Options. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16

Profiling Options in Version 9.6.X. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16

Profile and Data Discovery Options Overview in Version 10.0. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16

Summary of Column Profile Results Overview. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19

Column Profile Results Overview in Version 9.6.X. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19

Column Profile Results Overview in Version 10.0. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19

View Column Profile Results. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21

Column Profile Results Overview. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22

General Values Overview. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23

Data Types Overview. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24

Patterns Overview. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26

View Properties Overview. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28

Values Overview. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29

Drilldown Overview. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32

2

Data Domain Overview. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33

Create a Profile

Profiles and Profile Results Overview in the Analyst ToolIn Version 10.0, you can use various options to create different types of profiles, and view the profile results in summary view and detailed view in the Analyst tool.

In the following sections, you can view the changes in the Analyst tool from Version 9.6.X to Version 10.0 when you create a profile and view the profile results.

Create Profile OverviewYou can create column profiles, data domain discovery profile, and enterprise discovery profiles.

Creating a Column Profile in Version 9.6.X

1. Click New > Data Object Profile in the header area.

The profile wizard appears.

2. In the profile wizard, select Column profiling.

3

3. Continue with the steps in the profile wizard to create a column profile.

Creating a Column Profile in Version 10.0

You can use various options to create a column profile.

The following sections explain the different options that you can use to create a column profile:Option 1

1. Click New > Profile in the header area.


2. In the profile wizard, select Single source in the profile wizard.


Option 2

1. Click Open in the header area.

The Library workspace appears.

4

2. In the Library workspace, right-click on the data object, and click Create Profile.


5


Create an Enterprise Discovery ProfileYou can create enterprise discovery profiles in Version 9.6.X and Version 10.0.

Creating an Enterprise Discovery Profile in Version 9.6.X

1. Click New > Enterprise Discovery Profile in the header area.

The New Enterprise Discovery Profile wizard appears.

2. Continue with the steps in the profile wizard to create an enterprise discovery profile.

Creating an Enterprise Discovery Profile in Version 10.0



2. In the profile wizard, select Enterprise Discovery in the profile wizard.

6

3. Continue with the steps in the profile wizard to create an enterprise discovery profile.

Create a Column Profile with Data Domain DiscoveryYou can create column profile with data domain discovery in Version 9.6.X and Version 10.0.

Creating a Column Profile with Data Domain Discovery in Version 9.6.X



2. In the profile wizard, select Column profiling and Data domain discovery.

8

3. Continue with the steps in the profile wizard to create a data domain discovery profile.

Creating a Column Profile with Data Domain Discovery in Version 10.0



2. In the profile wizard, select Single source. Click Next.

3. Enter information as needed in the Specify General Properties and Select Source screens. Click Next.

4. In the Specify Settings screen, select Data Domain Discovery.

9

5. Click Save and Run to save and run the profile, or click Save and Finish to create the profile.

Create a Data Domain Discovery ProfileYou can create data domain discovery profile in Version 9.6.X and Version 10.0.

Creating a Data Domain Discovery Profile in Version 9.6.X



2. In the profile wizard, select Data domain discovery.

10

3. Continue with the steps in the profile wizard to create a data domain discovery profile.

Creating a Data Domain Discovery Profile in Version 10.0




3. Enter information as needed in the Specify General Properties and Select Source screens. Click Next.

4. In the Specify Settings screen, select only Data Domain Discovery.

11


Add or Choose Data Objects OverviewYou can choose an existing data object to create a column profile in Version 9.6.X. You can choose data objects or import data objects to create a column profile in Version 10.0.

Data Object Overview in Version 9.6.XYou can choose an existing data object to create a column profile.

Choosing a Data Object

1. Create a physical data object with the Flat File Data Object or Table Data Object option in the header area.

2. Create a column profile with the New > Data Object Profile option.

3. In the Step 1 of 6 screen, choose Column profiling, Data domain discovery, or Column profiling and Data domain discovery option . Click Next.

4. In the Step 2 of 6 screen, select a data object. Click Next.

The following image displays the Step 2 of 6 screen where you can select a data object.


12

6. Click Save to create the profile, or click Save & Run to save and run the profile.

Data Objects Overview in Version 10.0You can choose an existing data object, or choose a connection, schema, table, or view to import a data object to create a column profile.

Choosing a Data Object

1. Create a physical data object with the Flat File Data Object or Table Data Object option in the header area.

2. Create a column profile with the New > Profile> option.


4. In the Specify General Properties screen, enter a name and an optional description for the profile. In the Location field, select the project or folder where you want to create the profile. Click Next.

5. In the Select Source screen, click Choose.

13

The Choose Data Object dialog box appears.

6. In the Choose Data Object dialog box, select a data object. Click OK.

1. Projects pane. Choose a data object from the projects and folders.

2. Data Objects pane. Choose a data object.

3. Properties pane. View the data object properties.

4. Data Preview pane. View the data in the data object.

7. Enter information in the Specify Settings screen and Specify Rules and Filters screen as needed.


Importing a Data Object

1. Create a column profile with the New > Profile option.

14

2. Select Single source in the profile wizard. Click Next.

3. In the Specify General Properties screen, enter a name and an optional description for the profile. In the Location field, select the project or folder where you want to create the profile. Click Next.

4. In the Select Source screen, click New.

1. Schema, views, and synonyms. Select the Show only default schema option to view the schema. Select Show views and synonyms option to display views and synonyms.

2. Select a data object in the Data Object pane.

The New Data Object dialog box appears.

5. In the New Data Object dialog box, select a connection, schema, table, or view, select a location, and create a folder to import the data object. Click OK.

6. Enter information in the Specify Settings screen and Specify Rules and Filters screen as needed.

15


Profiling OptionsYou can choose various profiling options for a column profile in Version 9.6.X and Version 10.0.

Profiling Options in Version 9.6.XYou can choose the sampling options, drilldown options, and data domain discovery options in the profile wizard.

Profiling Options

You can choose the sampling options, drilldown options, and data domain discovery options in the various screens in the profile wizard.

You can choose the profiling options in the following screens for various profiles:

• Column profile. Choose the profiling options in Step 4 of 6 and Step 6 of 6 screens.

• Column profile with data domain discovery. Choose the profiling options in Step 4 of 7, Step 5 of 7, Step 7 of 7 screens.

• Data domain discovery profile. Choose the profiling options in Step 4 of 5 and Step 5 of 5 screens.

Profile and Data Discovery Options Overview in Version 10.0You can choose the sampling options, drilldown options, and data domain discovery options in the Specify Settings screen in the profile wizard. In Version 10.0, you can create an SQL filter along with the existing types of filters, and create, apply, edit, or delete rules in the profile wizard.

Sampling, Drilldown, and Data Domain Discovery Options

You can choose one of the sampling options in the Run profile on pane. You can enable or disable drilldown, and choose drilldown options in the Drilldown pane. You can choose the data domain discovery options in the Data Domain pane. All the panes are available in the Specify Settings screen in the profile wizard.

The following image displays the sampling, drilldown, and data domain discovery options for a column profile in the Specify Settings screen in the profile wizard:

16

1. Run profile on pane. Select one of the sampling options for the profile.

2. Drilldown pane. Select a drilldown option or disable the drilldown option for the profile.

3. Data Domain pane. Select data domains along with the options for the profile.

SQL Filter

In Version 10.0, you can create SQL filters in the profile wizard.

SQL Filter




3. Enter information as needed in the Specify General Properties, Select Source, and Specify Settings screens. Click Next.

4. In the Specify Rules and Filters screen, click Actions > New Filter.

The New Filter dialog box appears.

5. Click SQL.

The following image displays the SQL filter pane where you can enter a SQL statement to create a filter:

17

The SQL filter pane appears.


Rules

In Version 10.0, you can create rules or apply rules in the profile wizard.

Creating, Applying, Editing, or Deleting a Rule

1. Create a column profile with the New > Profile option.

2. Select Single source in the profile wizard. Click Next.

3. Enter details into the Specify General Properties, Select Source, and Specify Settings screens. Click Next.

4. In the Select Rules and Filters screen, you can perform the following actions:

• Select Actions > Create Rule to create an expression rule.

• Select Actions > Apply Existing Rule to apply a predefined rule.

• Select a rule and click Actions > Edit Rule to edit a rule.

• Select a rule and click Actions > Delete Rule to delete a rule.

The following image displays the various options available in the profile wizard to create, apply, edit, or delete a rule:

18


Summary of Column Profile Results OverviewYou can view column profile results in different screens in Version 9.6.X and Version 10.0.

Column Profile Results Overview in Version 9.6.XYou can view profile results in various screens to understand and analyze the content, structure, and quality of data.

The following image displays the profile results:

1. Profile name. Displays the name of the profile.

2. Screens. Displays the various screens that you can use to view the profile results, data domain results, data in the profile, and properties of the profile.

3. Column results pane. Displays the profile results for the column.

4. Panes. Displays the values, patterns, data types, and statistics for a column.

5. Drill down pane. Displays the drilldown results.

Column Profile Results Overview in Version 10.0View profile results to understand and analyze the content, structure, and quality of data. You can view all the columns and rules in a profile in summary view. You can view the properties of a column or rule in detail in the detailed view.

You can view the profile results under the Discovery workspace. The view header displays the type of profile, the number of columns in the profile, number of rules in the profile, sampling data, and date and time of creation.

In summary view, you can view the properties of each column as a value, horizontal bar chart, or as a percentage. You can view column properties, such as null, unique, non-unique values, patterns, data types, and data domains. You can view the profile results in summary view based on the default filters.

In detailed view, you can view null, unique, and non-unique values, inferred data types, inferred data domains, inferred patterns, values, business terms, and preview the data in panes.

You can view the profile properties when you click Actions > Profile Properties.

19

Summary View

You can view the profile results for a column profile in summary view.

You can view the profile results in summary view:

• When you click Save and Finish in the profile wizard

• When you choose Save and Run in the profile wizard, you can open and run the profile in the Library workspace.

The following image displays the profile results in summary view:

1. Summary view. Displays a summary of profile results for a column profile.

2. Summary view header. Displays the profile run number, number of columns participating in the profile, number of rules in the profile, number of rows participating in the profile, and the date and time of the profile run.

3. Profile Statistics. Displays the general values, statistics, data types, data domain, and business terms for the columns in the profile. For more information about the profile statistics, you can hover the mouse over the statistics to display the tooltip.

4. Icons. Displays the profile edit icon, add tag icon, add comment icon, and the Actions menu icon.

5. Actions menu. Displays options to run and edit the profile, detect outliers in the profile, add the profile or column to a scorecard, choose a profile run to view the profile results, compare two profile runs, export data to a file, verify the inference results, show comments and tags, and view profile properties.

6. Column with associated rule. Displays a column that has an associated rule. You can apply the rule in the profile wizard when you create or edit the profile.

7. Rules. Displays the rule that is associated with the profile.

8. Outliers. Displays the number of pattern outliers and value frequency outliers in the columns in the profile results. By default, the Analyst tool does not determine outliers in the profile results and the tool displays N/A. You can run the outlier to view the outlier results.

20

Detailed View

You can view the column specific results for a column profile in detailed view.

When you click a column in the summary view, the detailed view of the column profile results appear.

The following image displays the column profile results in detailed view:

1. Overview link. Click the overview link to display the summary view of the profile results.

2. Detailed view. Displays the results for a column in detail.

3. Outlier dropdown list. Displays the outlier dropdown list. You can highlight the outliers, disable outlier, or filter the results to display only outlier values in the column.

4. Slider. Displays the slider in the upper panel of the Values pane. You can move the slider to view the values within the slider in the lower panel of the Values pane.

5. Sparkline. You can view the increase and decrease of the general values in every consecutive profile run as a sparkline. A sparkline displays the variation in a general value across the last five consecutive profile runs in a line chart. The sparkline displays the latest profile run in orange. You can view specific results when you hover the mouse over the sparkline.

6. Outlier values. Displays the outlier values in the column. You can view the outlier values as vertical bars with orange highlight.

7. Statistics. Displays the various statistics for the values in the column.

8. Data preview. Displays the drilldown results and the drilldown column in a highlight.

View Column Profile ResultsYou can view column profile results for a column profile when you run a profile.

The following sections explain the profile results in Version 9.X and Version 10.0 in detail.

21

Column Profile Results OverviewYou can view the profile results in various screens to understand and analyze the content, structure, and quality of data.

Viewing Column Profile Results in Version 9.6.X



2. Click the profile in the Assets > Profiles pane.

3. In the column results pane, you can view the column names, general values, statistics, data types of the columns, and the data and time of profile run.

The following image displays the results screen for the profile:

Viewing Column Profile Results in Version 10.0




The summary view appears.

3. In the summary view, you can view the summary view header, default filters, column names, general values, statistics, data types of the columns, and business terms.

The following image displays the column results in summary view for the profile:

22

General Values OverviewYou can view the general values in the column profile results. Null values, percentage of null values, unique values, percentage of unique values, non-unique values, and percentage of non-unique values are called as general values.

Viewing General Values in Version 9.6.X




3. You can view the general values in the profile results.

The following image displays the general values in the profile results:

Viewing General Values in Version 10.0




3. You can view the general values for all the columns in the profile in summary view.

The following image displays the general values in summary view:

23

4. Click a column in summary view to view the general values for the column. You can view the number of rows with null values, unique values, and non-unique values in different colors. You can view the values in percentages. You can view the increase and decrease of the general values in every consecutive profile run as a sparkline. You can view specific results when you hover the mouse over the sparkline.

The following image displays the general values for the column in General pane in detailed view:

Data Types OverviewYou can view documented and inferred data types along with any data conflicts in the profile results.

Viewing Data Types in Version 9.6.X




24

3. You can view the documented data types, inferred data types, and percentage of inferred data types in the profile results.

The following image displays the data types in the column results pane in the profile results:

The following image displays the data types in the Data Types pane in the profile results:

Viewing Data Types in Version 10.0




3. You can view the documented and inferred data types for all the columns in the profile in summary view.

The following image displays the data types in summary view:

25

4. Click a column in summary view to view the data types for the column. You can view the inferred data types for the column. The frequency of the data types in a column appears as a horizontal bar chart and in percentages. You can drill-down, approve, reject, or reset the selected inferred data type.


Patterns OverviewYou can view patterns for column values.

Viewing Patterns in Version 9.6.X




26

3. You can view the patterns in the profile results.

The following image displays the patterns in the column results pane in the profile results:

Viewing Patterns in Version 10.0




3. You can view the patterns for all the columns in the profile in summary view.

The following image displays the patterns in summary view:

27

4. Click a column in summary view to view the patterns for the column. You can view the patterns for the column values. The frequency in which the patterns appear in a column appears as a horizontal bar chart and in percentages. You can drill-down, add a pattern to a reference table, or create a data domain with the selected pattern.

The following image displays the general values for the column in Patterns pane in detailed view:

View Properties OverviewYou can view the properties for a profile.

Viewing Profile Properties in Version 9.6.X




The profile results appear.

3. Click Properties to view the profile properties for the profile.

The following image displays the profile properties for the profile.

Viewing Profile Properties in Version 10.0




28

The profile results appear in summary view.

3. Click Actions > Profile Properties in summary view or detailed view to view the profile properties.

The following image displays the profile properties for the profile:

Values OverviewYou can view the values in a column in the profile results.

Viewing Values and Statistics in Version 9.6.X




3. You can view the values in the Values pane in the profile results.

The following image displays the values in the profile results:

29

4. You can view the statistics in the Statistics pane in the profile results

The following image displays the statistics in the profile results:

Viewing Values in Version 10.0




3. You can view the values and statistics for all the columns in the profile in summary view.

The following image displays the values and statistics in summary view:

31

4. Click a column in summary view to view the values and statistics for the column. You can viewall the values in the column in a graphical representation along with the frequency, length, and percentage. You can drill-down on each value. You can add the value to a reference table, create a value frequency rule, and create a data domain.


Drilldown OverviewYou can view the properties for a profile.

Viewing Drilldown Results in Version 9.6.X





32

3. Select a value, pattern, or data type. Click Actions > Drilldown.

The following image displays the drilldown results in the Drilldown pane:

Viewing Drilldown Results in Version 10.0





3. In summary view, click a column.

The profile results for the column appears in detailed view.

4. In the detailed view, right-click on a value in the Values pane, and select Drilldown.

The following image displays the drilldown results and the drilldown column in a highlight in the Data Preview pane:

Data Domain OverviewYou can view the properties for a profile.

Viewing Data Domains in Version 9.6.X





33

3. Click Data Domain Discovery screen.

The following image displays the Data Domain Discovery screen in the profile results:

Viewing Data Domains in Version 10.0





3. You can view the data domain values for all the columns in the profile in summary view.

The following image displays the data domains in summary view:

34

4. Click a column in summary view to view the profile results for the column.

5. You can view the names of the data domains associated with the column or rule and the percentages of conformance in the Data Domain pane.

The following image displays the Data Domain pane in detailed view:

AuthorsLavanya SSenior Technical Writer

35

informatica profiling and discovery - 10.0 - profiling and ... library/1/0855... · the profiling...

Documents