ibm infosphere stewardship center for iis dqec

© 2015 IBM Corporation

IBM InfoSphere Information Server Stewardship Center and Data Quality Exception Console

2 © 2015 IBM Corporation

Agenda

IBM Data StewardShip Center

Data Quality Exception Console

Questions & Réponses


Stewardship Center


Governance Officers

Data Stewards (IT and LOB)

Data Analysts/SMEs

Web

Mobile

Social Collaboration

DashboardWorkflow &

Rules

Provide a single collaborative environment for business users to

define and monitor compliance with data quality policies and to

manage data quality issues to resolution

Addressing the needs of various roles across the governance organization

Key capabilities delivered within customizable and collaborative interfaces

IBM InfoSphere Stewardship Center


InfoSphere Stewardship Center: Event Notification

with Workflow and Remediation

Read File

Errors

Business Term

Updates

Notifications/Email

Exception

Fixes

Quality Exception

NotificationDuplicate

Resolution

Policy

Changes

Suspect

Duplicates

Catalog Updates


Data StewardStewie Silvers

Governance Council Member

Gary Grayson

Data Steward Manager

Martin McGregor

Meets the Needs of Critical Business Roles

Traditional Data Steward and LOB users

collaborate to remediate record violations,

improving data quality.

Data Steward Manager responsible for overseeing a team of subject matters experts.

Data Quality Champion and LOB leader, collaborate defining data policies and rules for the business in addition to monitoring policy compliance


Stewardship Center Performance Dashboard

The Stewardship Center provides out of the box Dashboards:– Team Performance and Process Performance


Stewardship Center takes advantage of IBM BPM

Enterprise Processes

Services

Information Server Services

CustomerPatientVendorProductAccount

IBM BPM & Decisions

BPM Standard

OnboardingRegistrationParty updates

SalesMarketingDelivery

Information

Governance and

Data Quality

Stewardship

Information Server and IBM Business

Process Management come together to

help you manage your Governance

and Data Quality processes:

Information Server delivers real time data

exceptions and events to be governed using

a repeatable business process

BPM provides orchestration of the steps

required to improve data quality and increase

common understanding

Business Process Management is a discipline that leverages software and

services to provide total visibility into your organization. Discover, document,

automate, and continuously improve business processes to increase efficiency

and reduce costs.


Go Mobile, Get Social, and Govern

Mobile

Access

Social Collaboration

Governance

Workflow

Reporting


What is the Stewardship Center?

The integration of IBM BPM Standard Edition and Information Server– ‘Stewardship Center’ is a customized home screen for IBM BPM Process Portal

IBM provides a set of sample BPM applications to demonstrate the

basic capabilities– Exception Management

– Information Governance Catalog Governance Rule workflow

– Information Governance Catalog Business Term emails

This section will focus on the Exception Management

sample application– Information Governance Catalog applications have already been discussed


InfoSphere Stewardship Center

Single Inbox and Remediation Portal

MDM tasks

DataStage/

QualityStage

tasks

Information

Analyzer

tasks

Information

Governance

Catalog tasks


Start

Exception list

submit

fixes

reject

approve

Approval

request list

Apply fix request list

escalate

apply fix

Data Steward


Data Source Owner

End

Out Of the Box Exception Remediation Sample

Endresolve

User=silvers

Name=Stewie Silvers

User=mcgregor

Name=Martin McGregor

User=williams

Name=Chad Williams

Exception


A View in IBM Process Designer


Workflow – Data Steward

Sign in as Data Steward ‘silvers’

See ‘Open Tasks’ new exceptions

Sample application as defined

the due date as date run + 24

hours.

PoC/Demo Tip:

Switch between ‘Open Tasks’

and ‘Completed Tasks’ to

refresh the screen

PoC/Demo Tip: Carefully read to see the exception source


Workflow – Data Steward (continued)

Claim the task



Work with the task

PoC/Demo Tip:

‘Submit’ after changes

‘Escalate’ if you don’t know what changes to make



Work with the task (continued)

View the BPM

Process Diagram



Work with the task (continued)– See the exception description

You can drill directly to the Information

Governance Catalog to see the Data Rule

The link is AUTOMATIC and illustrates integration of our suite!



Work with the task (continued)– See the exception description definition in IGC

The Data Steward can find what they need with a few clicks!

…and continue to drill into

the Data Rule Definition



Work with the task (continued)– Edit and change individual records

The sample application allows any data to be changed.

It also shows how it is possible to have a ‘read only’ flag

to prevent data change.

The sample application only allows

editing of one exception at a time. Stewardship Center supports EDIT!



Work with the task (continued)– Edit and change individual records (continued)



Work with the task (continued)– View changed records

Changed records are highlighted in light blue.

Changed data is highlighted in dark blue.

Data

Steward can

VIEW

changes!



Work with the task (continued)– Changing multiple records with Export

• Also can be used to share data with others



Work with the task (continued)– Changing multiple records with Export (continued)

A csv file is downloaded to the client

Every column output by the rule will have a ‘_new’ column




You can use Microsoft™ Excel® to edit the file

PoC Tip:

Be very careful when opening .csv files in Excel.

Any data with leading zeroes will have those

leading zeroes remove. For example, the valid US

Postal Code 02201 will become the invalid US

Postal Code of 2201.

Excel will also corrupt timestamps: 2015-01-25

09:40:13.0 becomes 40:13.0.




Then ‘Import’

Your Firefox ‘import’ screen will be blank

PoC/Demo Tip:

Import does not currently work with

Internet Explorer. Use Firefox instead.




You will need to change the number of ‘Rows per page’ to force a

refresh


But Where Is The Fixed Data Stored?

The fixed data is stored in the providers repositoryDataStage and QualityStage Exceptions Stage = Exceptions Stage Database


But Where Is The Fixed Data Stored? (continued)

DataStage and QualityStage Exceptions Stage– In the same output table as the exceptions

• Reminder: Get the table name from the job run:


But Where Is The Fixed Data Stored? (continued)

DataStage and QualityStage Exceptions Stage (continued)– In the same output table as the exceptions (continued)

– The data is defined as a BLOB

The data is JSON stored in binary format, but CAN

be parsed by the Hierarchical Stage…

This is the view from IBM Data Studio


Back to the Workflow



Data Steward can do one of two things:– Submit Fixes

• When changes are done

– Escalate Task• Ask the Data Steward Manager to review unknown exceptions

Updates shown in the DQEC


Workflow – Data Steward Manager

When the Data Steward selects ‘Escalate’– Resolve – the task is complete

– DQEC is updated


Workflow – Data Steward Manager

(continued)

When the Data Steward selects ‘Submit Fixes’

– Approve – goes on to Data Source Owner

– Reject – goes back to Data Steward• Any fixes are also discarded

– Either one updates the Data Quality Exception Console


Workflow – Data Source Owner

When the Data Source Owner select ‘Approve’– Apply Fixes

• Once changes done (however they are done)

– Data Quality Exception Console is updated


Start

Exception list

submit

fixes

reject

approve

Approval

request list

Apply fix request list

escalate

apply fix

Data Steward


Data Source Owner

End

Out Of the Box Exception Remediation Sample - Done

Endresolve

Exception


InfoSphere Stewardship Center

CollaborationMonitoring

Inbox

Data Quality Exception ConsoleCommon

Event

Framework

Information Analyzer

MDM Server

DataStage/QualityStage

Exception Stage

Information Governance

Catalog

Information Governance Dashboard

API

(Information Analyzer,

DataStage/QualityStage

Exceptions Stage)

Source

Systems

WorkflowsWorkflows

1a 1b

2a

2b

1c

1d

3

1a-d

2a-b

3

Information Analyzer and DataStage/QualityStage

Exception Stage exception events and notifications

Information Governance Catalog events and notifications

MDM integration

InfoSphere Stewardship Center Architecture

Event Notifier

Event Registry

Event Service


Subscriptions

Subscription Manager is on the Information Server Launchpad


Subscriptions (continued) PoC/Demo Tip:

Name your subscriptions:

Asset Type + Application + Other Information

For example: Business Term Basic Email Notification Create

Email is available for all Event Types. Application will only

be listed if there is a registered Callback.

All subscriptions allow filtering.

Filters must be a single item.

ACTION=CREATE or

ACTION=MODIFY would require

two subscriptions.


Subscriptions (continued)

Notification Type ‘Email’ available to all event types– Make sure to set up email provider (the smtp server) as per this link

– The email will go to the email address entered for the user

– You can send a copy to any other user

PoC/Demo Tip:

Subscription Manager Email only supports ‘no security’ (port

25), or ‘SSL’ security (port 465). TLS (port 587) is not

supported. Hotmail, Live, and Outlook all use TLS.

http://www-01.ibm.com/support/knowledgecenter/



Notification Type ‘Email’ available to all event types (continued)– At GA, the email will look better for Remediation events than for IGC events

• Remediation DataStage/QualityStage emails:

• Information Analyzer emails:

• IGC emails (from creation of a new Category):

Work is underway to make the emails customizable.



Note the Event Types are not sorted


Notifications

Notifications are published every time a subscribed action is

performed– Notifications are ONLY generated for subscribed actions

You can list and delete notifications based on time ranges– -beginDate “YYYY-MM-DD hh:mm:ss”

– -endDate “YYYY-MM-DD hh:mm:ss”

– You can specific just:• beginDate (everything from a specific time forward)

• endDate (everything up to a specific time)


Thank You

End of Section 3


Agenda

Stewardship Center Overview

Brief BPM Overview

High Level Architecture

Information Governance

Catalog– Basic Event Notification

– Sample IBM BPM Applications

Labs Part 1

Data Quality Remediation– Data Quality Exception Console

– Stewardship Center

Labs Part 2

Common Event Framework– Event Management Architecture

– Event Providers

– Event Types

– Callbacks

– Subscriptions

– Notifications

– Events

– Task Scheduling

Labs Part 3

Appendices– Delete from Data Quality

Exception Console

– Delete from Stewardship Center

Objective: Clearly Explain Stewardship Center Architecture,

Capabilities and Competitive Advantages


Data Quality Execption Console


Data Quality Exception Console

Replaces the Data Quality Console– Data Quality Console still available today

– Data Quality Console still fully functional

– Will be removed completely in a future release

Is the first step to managing exceptions in the Stewardship Center– Exceptions ‘sets’ must be selected to be ‘managed’ for the the out-of-the-box

application

The Data Quality Exception Console allows only critical exceptions

to be managed in the Stewardship Center.


Data Quality Exception Console (DQEC)

Start from the Launchpad


Data Quality Exception Console (continued)

Filters to limit what is displayedExceptions

Browser based Link to Information

Governance Dashboard

Sort on any column


Filtering

Many Filter options

Combining filters is an ‘and’– Example: Contain string ‘val’ and produced by

Information Analyzer

Filters are predefined

Filters can not be saved (yet)


What Does The Data Quality Exception

Console Show?

Information Analyzer– Output from Data Rule run

– Output from Rule Set run

This option does not matter.

Whatever you output will go to the

Data Quality Exception Console.

PoC Tip:

Choose wisely!

These are the columns seen

in the Stewardship Center

PoC Tip:

DO NOT use spaces in

your Data Rules Names!



Console Show? (continued)

DataStage and QualityStage– Output from Exceptions Stage only

• Not from the One-Source Match Exceptions nor the Two-Source Match Exceptions

Tip: Always include the ‘ruleID’

from the Data Rules Stage so

you know which Data Rule

Definition caused the exception



Console Show? (continued)

Information Analyzer Data Rule Definition and Rule Set Definition

‘Test’ runs will also show

System assigned Name

Poc/Demo Tip:

Delete these test runs when

testing is completed.


Displaying Exception Details

Status will be updated by

the Stewardship Center!!!!!!!

Role based actions

For example: ‘Administrators’ can

only ‘Delete’

The Data Quality Exception Console is integrated!


Viewing Records

More details to come…

Demo Tip:

Do not use the

browser ‘back’

button. Click

here instead.


‘Edit’ and ‘Delete’

Edit to change the exception priority

– Not to edit the data (that can be done in the Stewardship Center!)– Not reflected in Stewardship Center– Is for this exception only, not all instances of this exception


‘Edit’ and ‘Delete’ (continue)

Delete to remove the exception

– Also removes the output table• Does not matter what type – all are deleted


But What About ‘Manage’ and ‘Monitor’?

All exceptions will be visible in the DQEC

Data Steward Manager decides how to handle the exceptions with:– Manage = Which exceptions are to be managed in the Stewardship Center

– Monitor = Which exceptions are to be monitored in the DQEC


But What About ‘Manage’ and ‘Monitor’? (continued)

Selecting ‘Manage’ will make this exception visible in the

Stewardship Center– AND all future exceptions with the same name


Questions & Réponses

ibm infosphere stewardship center for iis dqec

Documents

data quality processes

ibm bpm process portal

data quality policies

data policies

data quality issues

data quality champion

business processes

business users