collaborative metadata management with david loshin

Breaking Down the Fence: Using Business-IT Collaboration to Foster

Consistent Enterprise Metadata

David Loshin Knowledge Integrity, Inc.

www.knowledge-integrity.com

1 © 2014 Knowledge Integrity, Inc. www.knowledge-integrity.com

(301) 754-6350

Islands of Data?

p  Different business functions have their own views of necessary data p  Requirements are based on each function’s particular needs p  Data subsystems represent each function’s particular views

© 2014 Knowledge Integrity, Inc. www.knowledge-integrity.com

(301) 754-6350

2

Finance

Financial Data

Sales

Sales Data

Marketing

Marketing Data

Service

Service Data

H/R

H/R Data

Transactions

Transaction Data

Compliance

Compliance Data

Cross-Function Data Sharing

p  Cross functional applications requires data sharing and exchange p  Data concepts may exist in different forms in different applications


(301) 754-6350

3

Finance

Financial Data

Sales

Sales Data

Marketing

Marketing Data

Service

Service Data

H/R

H/R Data

Transactions

Transaction Data

Compliance

Compliance Data

The Most Dangerous Question in Data Management

p  Hundreds have posed it… p  Thousands have considered it… p  Tens of thousands of hours have been invested in answering… p  The most dangerous question in data management:


(301) 754-6350

4

Ambiguity in Contextual Definitions

p  Example: Uses of the business term “customer”

p  These are potentially conflicting definitions p  Correspondingly, the representations and underlying meanings may

differ p  Identify business terms and provide a means for

n  Identifying the authoritative definitions n  Resolving similar concepts n  Differentiating distinct concepts


(301) 754-6350

5

Sales: Someone who pays for our products or services

Support: Someone who has a license for use of our product

The Challenge of Meaning in Different Contexts


(301) 754-6350

6

p  “Employee hire date”

p  Any selected definition changes the meaning of the other uses! p  Any use may be dependent on contextual meaning

The date the employer offered the position

The date the prospect accepted the offer

The first date the employee came to work

The date the employee first provided services for pay

The date the rehired employee resumed providing services for pay

Understanding the Data Lifecycle

p  Data element use is considered only at design and implementation


(301) 754-6350

7

Conception

Definition

Design

Implementation

Deployment

Utilization

Maintenance

Repurposing

Retention

Retirement

p  A more mature approach looks at data repurposing

p  Ultimately, all lifecycle events have implications to structure and semantics

Evolution of the Business Metadata Glossary

p  Many sources of entity concepts and business terms may conflict with each other

p  The data steward must facilitate the collection and documentation of business terms

p  The data steward must also prepare for harmonization of terms


(301) 754-6350

8

Policies

System Docs

Processes

Models

Standards

Applications

Business Rules

Profiling

Etc.

Entity Concepts

Business Terms

Definition Contextual Meaning

… …




Documenting Metadata: Business Term Definitions

p  Goal: n  Identify key business terms n  Find authoritative definitions n  Document definition(s) and

use(s)

p  Establish a template to capture: n  Business term n  Type of term (noun, verb,

etc.) n  Definition n  Authority n  Usage


(301) 754-6350

9

Business Term ID

Term Type Definition Authority Usage

Biii-HQLA1

High-quality liquid asset

Noun, Entity

“Assets that can be immediately converted into case at little or no loss of value.”

“Basel III: International framework for liquidity risk measurement, standards and monitoring ” http://www.bis.org/publ/bcbs188.pdf

Standard for Liquidity Coverage Ratio (LCR) in “Basel III: International framework for liquidity risk measurement, standards and monitoring ” http://www.bis.org/publ/bcbs188.pdf

Evolving a Community of Interest

p  Different stakeholders have a vested interest in variance in n  Naming n  Definition n  Use

of common business terms and entity concepts p  Each stakeholder may register

his/her interest in the status of particular metadata objects


(301) 754-6350

10 10 10


… …




Business Term

Defining Business Glossary Terms


(301) 754-6350

11

Business Term

Use Definition Data Element Concept

Data Element

Data Element

Data Element Concept

Data Element

Data Element

Data Element

Use Definition

Conceptual Domain

Value Domain

Conceptual Domain

Value Domain

Conceptual Domain

Value Domain

Conceptual Domain

Value Domain

Conceptual Domain

Value Domain

Identifying Lineage of Common Business Terms


(301) 754-6350

12

Business Term

Use Definition Data Element Concept

Data Element

Data Element

Data Element Concept

Data Element

Data Element

Data Element

Use Definition

Conceptual Domain

Value Domain

Conceptual Domain

Value Domain

Conceptual Domain

Value Domain

Conceptual Domain

Value Domain

Conceptual Domain

Value Domain

Data Element

Type

First VARCHAR(25)

Middle VARCHAR(25)

Last VARCHAR(30)

SocialSec CHAR(9)

Data Element

Type

FirstName VARCHAR(35)

LastName VARCHAR(40)

SSN CHAR(11)

Telephone VARCHAR(20)

Centrally Managing a Business Term Glossary

p  Common business terms are likely to have many variant definitions

p  All definitions can be captured in a shared, central repository

p  Variances can be resolved in two ways: n  Consolidation of

meaning n  Differentiation into

distinct concepts


(301) 754-6350

13

Data Element

Type

First VARCHAR(25)

Middle VARCHAR(25)

Last VARCHAR(30)

SocialSec CHAR(9)

Data Element

Type

FirstName VARCHAR(35)

LastName VARCHAR(40)

SSN CHAR(11)

Telephone VARCHAR(20)

Social Security Number Identifier

Unique number assigned by Social

Security Administration

Centralizing Metadata Across the Organization

p  Logging business terms and their definitions in a centralized repository provides a single source of information about the chain of definition

p  As data is shared across functions and among business processes, interested parties can review the variety of definitions


(301) 754-6350

14

Collaborative Metadata Management

p  The repository becomes the “one-stop shop” for information about n  Business terms n  Definitions n  Conceptual domains n  Value domains n  Data element concepts n  Data elements n  Entities

p  A collaborative platform creates an environment for logging a “vested interest” in the metadata


(301) 754-6350

15

Metadata

Harmonization through Collaboration

p  The collaborative platform exposes the variances that may impact semantic consistency downstream

p  Harmonization (via resolution or differentiation) can be facilitated by the virtual workgroup

p  Other potentially interested parties can be invited to join the conversation


(301) 754-6350

16

Social Security Number

Identifier Unique number assigned by Social Security Administration

Authentication Last four digits of number

assigned by the Social Security Administration

Governance through Collaboration

p  A collaborative platform allows interested individuals to track those business terms, data element concepts, and reference metadata relevant to their business processes

p  A comprehensive directory provides visibility to all related concepts

p  Data policy changes can be governed through the collaborative platform


(301) 754-6350

17

Social Security Number

Identifier Unique number assigned by Social Security Administration

Authentication Last four digits of number

assigned by the Social Security Administration

Identifier Unique number assigned by the company Customer ID

Encouraging a Culture of Semantic Harmony

p  Small variance in definitions in isolated functions become magnified when data is shared across functions

p  Establish a level playing ground by n  Instituting a common business term glossary n  Harmonizing business term definitions n  Unifying shared reference data into conceptual domains and

corresponding value domains n  Socializing use of shared metadata n  Establishing standards for future development n  Integrate methods for monitoring compliance with standards


(301) 754-6350

18

Collaborative Metadata: Considerations

p  Process n  Are there defined practices, disciplines, and training to ensure socialization

and integration? p  Governance

n  Does the metadata platform incorporate compliance with defined data policies for oversight of shared metadata?

p  Comprehensiveness n  Can the shared platform capture all aspects of lineage, relationships,

business glossary, definitions, etc.? p  Ease of Use

n  The system must be easy to use by consumers who will range from being technically proficient IT staff to informed business users

p  Maintenance n  Is the organization willing to support the ongoing investment in

maintaining the collaborative resource?


(301) 754-6350

19

Check Out These Resources!

p  www.knowledge-integrity.com p  www.dataqualitybook.com p  If you have questions,

comments, or suggestions, please contact me David Loshin 301-754-6350 [email protected]


(301) 754-6350

20

collaborative metadata management with david loshin

Technology

data repurposing p

data steward

islands of data

exchange p data concepts

data management p hundreds

p thousands

business term customer

p different business