oracle enterprise metadata management (oemm) · pdf fileibm datastage ibm discovery ibm...

Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |

Oracle Enterprise Metadata Management (OEMM)

Hüseyin Celebi

Cloud Integration Specialist - EMEA


Safe Harbor Statement

The following is intended to outline our general product direction. It is intended for information purposes only, and may not be incorporated into any contract. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions. The development, release, and timing of any features or functionality described for Oracle’s products remains at the sole discretion of Oracle.

2


OEMM Overview

Oracle Confidential – Internal/Restricted/Highly Restricted 3


What do we mean by Data Governance and why do we need to do it?

4

“Data governance encompasses the people, processes, and information technology required to create a

consistent and proper handling of an organization's data across the business enterprise.” [Wikipedia]

Where did these

numbers come from?Do we trust what this

graph is telling us?

How confident are we

in this data?


OEMM and Data Governance

Oracle Big Data Governance 5

Data Governance

Metadata Management

Business Glossary

Data Profiling

Data Cleansing

Data Archiving

Data Privacy

PEOPLE

PROCESS TECHNOLOGY

…people and process

“Poor data quality is the

primary reason for 40% of

all business initiatives

failing to achieve their

targeted benefits”

Gartner, Measuring

Business Value of Data

Quality, 2011

“More than 50 percent of

data warehouse projects

will have limited

acceptance, or will be

outright failures, as a

result of a lack of attention

to data quality issues”

Gartner PR 2005

(http://www.gartner.com/new

sroom/id/492112)

…supported by tools and

capabilities

…in partnership with the business

OEMM


Value of Enterprise Metadata Management

6

ETL

BIDashboards

App

ETL

ETL

How was sales figure calculated?

How do I organize my DW and

Reports

What reports use the mainframe

data? Sys Admin

Executive

BI Developer

Where did this data

come from?

Application User

What will happen if I change this

table?

CDC

Data Reservoir

Data Steward

Can I trust the sources of this

customer data?

ETL

Developer

Solves significant pain points for wide variety of business consumers and technical staff

I want to design an experiment to measure the

success of a signup page. What data do I have?

Data Scientist

GG

Which reports use this

customer data?Enterprise

Architect

http://hadoop.apache.org/


It’s all about…

7

…in your data


Oracle Enterprise Metadata Management (OEMM)

8

• Metadata Management – horizontal and semantic data lineage for all data sources

• Business Glossary – simple tools to catalog, link and collaborate on business terms

✓ Business Data Catalog

✓ Report to Source Lineage

✓ Impact Analysis

✓ Audit, Versioning & Diff Reports

✓ Social/Collaboration Features

✓ Annotations and Tagging

✓ Comprehensive Harvesting

✓ 3rd Party BI Metadata

✓ 3rd Party ETL Metadata

✓ 3rd Party DB Metadata

✓ 3rd Party Modeling Tools

✓ Big Data Metadata

✓ Metadata Standards


Business User Understanding

✓ Lineage Analysis for reports

✓ Semantic Analysis

✓ Browsing/Searching/organizing of metadata across the enterprise

✓ Business Term maintenance and understanding it’s usage

Simplifying Enterprise Complexities

✓ Understanding past implementations by various teams on various platforms

✓ Fills the knowledge gap due to lack of documentation & employee turn-around

✓ Liberates metadata from its platform silos and consolidates for centralized access

Metadata Management Aids In…

Addressing the Data Mess

✓ Identifies duplicate metadata/data flows

✓ Checks for semantic consistency in data flows

✓ Analyzes metadata for detecting errors /standards

Providing Structure to Change Management

✓ Assists in application migration efforts by understanding old application metadata

✓ Assists ETL & BI development by metadata discovery of source & target systems

✓ Provides metadata standardization across the enterprise

✓ Keeps track of metadata changes over time

9

Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | 10

Data Lineage in Three Crucial Varieties

Vertical Lineage▪ For the Data Stewards▪ Links a business friendly set of

terms to the IT metadata and operational assets

▪ Links Business Glossary, Taxonomy, Ontology, Conceptual Models to Technical Metadata (Columns, Files, Objects etc)

Horizontal Column Level ▪ For the Architects▪ Links the data fields from Business

Intelligence Dashboards or Reports back to the Source DB/App Columns

▪ Schemas, BI View Layers, ETL Transformations, Calculations, etc.

Business Lineage▪ For the Business Users▪ Links a business friendly set of

Glossary terms to the BI reporting models (eg; Semantic Models, Universe, etc.)

▪ Links Business Glossary, Taxonomy, Ontology, Conceptual Models to Reporting Fields

BIZTERM

BIZTERM

REPORTFIELD

DBCOLS REPORT

FIELDSOURCE

COLS


Customer Initiatives & Project Drivers

Realtime BI and Data Warehousing

Cloud Integration

Big Data Reservoir

All Data Integration Use Cases

Oracle Enterprise Metadata Mgmnt

Oracle Enterprise Metadata MgmntOracle Enterprise Metadata Mgmnt

11


High Level OverviewMetadata sources , data flow and semantic links

•Business Glossary•Metadata harvested into models

•Databases•Data Modeling•ETL•Business Intelligence•Big Data•… many more

•Data flow•Semantic flow

Oracle Confidential, under Non-Disclosure 12


Business GlossaryBusiness Glossary is a collection of business terms and relationship definitions across enterprise. It provides a common understanding of terms to employees and business partners.

The Business Glossary can be used to document• Business Terms• Relationship between Business Terms• Business Rules • Domains • Data Entities logical/physical • Attributes across IT collaboratively. • Relationship of Terms with physical data assets



Business Glossary

Business Term

•Description•Properties•Custom Attribute value (see “Line Of Business”)•Documentation•Relationship with other terms•Comments



Business Glossary

Business Rules and Policies

• Defined in Business Terms• Linked to supported document for further details• Work flow from draft to approval• Linked to physical assets


Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | Oracle Confidential, under Non-Disclosure 16

Glossary Workflow

•Drafting•Reviewing•Approving•Publishing

Business Glossary


Harvesting Models


Harvesting bridges and Metadata Stitching

• 150+ metadata sources• One click harvest• Fully automated metadata stitching


Searching for Metadata

Easy to use search for finding metadata of interest

• Tables/Columns• Attributes• Data Models• Reports

•Search UI is customizable •Reporting and analysis after finding metadata



Browsing Metadata


Browsing is another way for finding metadata of interest

• Tables/Columns• Attributes• Data Models• Reports

•Reporting and analysis after finding metadata


Ad-hoc reporting of metadata

Excel like grid view for

•Listing•Sorting•Filtering•Updating•Sharing•Exporting





Filtering result set

Add/remove filters on•Metadata attributes•Custom attributes•Labels



Documenting metadata

• Any number of custom attributes• Update metadata for documentation• Share metadata



Browsing Data Modeling Diagram

Creating and Browsing diagrams

•Diagram harvested from Modeling tools•Diagrams can be created if it doesn’t exist. Full feature editor available.•Fully interactive diagram•Viewing tools for navigating complex diagrams•Lineage and Impact analysis can be initiated from the diagram



Social Collaboration for Stewardship Teams


Collaboration

•Discussion Threads•Labeling•Sharing•Bookmarking•eMailing


Lineage Analysis

Lineage analysis from everywhere

•Search results•Browsing list•Data Model diagrams

Lineage Results shown as•Textual overview•Graphical overview•Export for further analysis

Ability to trim lineage results by excluding interim details.



Lineage Analysis


Lineage analysis from everywhere


Lineage Results shown as•Textual overview•Graphical overview•Export for further analysis

Ability to trim lineage results by excluding interim details.


Lineage Analysis


Drilling into Lineage Analysis

•Source/target attributes•Transformations•Intermediate attributes•Control flows•Drill through for more details


Lineage Analysis


Tracing Lineage Analysis

•Source to target data flow•Transformations overview•Drill into for more details


Impact Analysis


Impact analysis from everywhere


Impact Results shown as•Textual overview•Graphical overview•Export for further analysis

Ability to trim Impact results by excluding interim details.


Semantic Analysis


Semantic analysis from Glossary

• Find linked physical assets • Graphical view of semantic relationship• Semantic distance• Drill into for more details


Semantic Analysis


Semantic analysis from search results



Versioning with Model Comparisons


Easily group and compare data models from similar databases to determine crucial differences


Open Source, Standards and Third Party (150+ sources)


▪ Apache HDFS

▪ Apache Hive Metastore

▪ Apache Hive Server

▪ Borland

▪ CA Component Modeler

▪ CA COOL (previously Sterling)

▪ CA Erwin

▪ CA Gen

▪ Cloudera Enterprise

▪ Cloudera Impala

▪ Database (JDBC)

▪ DataStax

▪ Embarcadero

▪ EMC ProActivity

▪ GentleWare

▪ Google BigQuery

▪ Grandite

▪ Hadapt Hive

▪ Hortonworks Hive

▪ IBM Cognos

▪ IBM DB2

▪ IBM Data Architect

▪ IBM DataStage

▪ IBM Discovery

▪ IBM Federation Server

▪ IBM Information Server

▪ IBM Lotus Notes

▪ IBM Netezza

▪ IBM Rational Rose

▪ IBM Rational Architect

▪ IBM Telelogic

▪ Informatica Developer

▪ Informatica PowerCenter

▪ MapR Hadoop Hive

▪ MicroFocus

▪ Microsoft Office Excel

▪ Microsoft Visio

▪ Microsoft SQL Server

▪ Microsoft SSAS

▪ Microsoft SSIS

▪ Microsoft SSRS

▪ Microsoft Visual Studio

▪ Microstrategy

▪ Magic Draw

▪ OMG CWM Standard

▪ OMG UML Standard

▪ Oracle BI Answers

▪ Oracle BI Enterprise Edition

▪ Oracle BI Server

▪ Oracle DAC

▪ Oracle Data Integrator

▪ Oracle Data Modeler

▪ Oracle Database

▪ Oracle Designer

▪ Oracle Hyperion EIS

▪ Oracle Warehouse Builder

▪ Pivotal Greenplum

▪ PostgreSQL

▪ SAP BO Crystal Reports

▪ SAP BO Designer

▪ SAP BO Desktop Intelligence

▪ SAP BO Information Design Tool

▪ SAP BO Repository

▪ SAP Sybase PowerDesigner

▪ SAP Sybase ASE Database

▪ SAS OLAP Server (via ODBO)

▪ Select SE

▪ Sparx Architect

▪ Tableau

▪ Talend

▪ Teradata

▪ Tigris

▪ Visible

▪ W3C XSD Schema

Metadata Harvesting (Glossary, Lineage & Impact Analysis)Key Standards

Oracle Confidential 36

oracle enterprise metadata management (oemm) · pdf fileibm datastage ibm discovery ibm...

Documents