CSE 590DB: Database SeminarCSE 590DB: Database Seminar

Autumn 2002: Meta Data Autumn 2002: Meta Data ManagementManagement

Phil BernsteinPhil Bernstein

Microsoft ResearchMicrosoft Research

Course AdministrationCourse Administration Course creditCourse credit

1 credit for reading the papers and showing 1 credit for reading the papers and showing upup

2 credits for leading a discussion2 credits for leading a discussion I’m giving today’s lectureI’m giving today’s lecture

So I get 2 creditsSo I get 2 credits Next week, we have a visitor Next week, we have a visitor

Wang-Chiew TanWang-Chiew Tan Send me mail to sign up for later weeksSend me mail to sign up for later weeks

What’s Meta Data? What’s Meta Data? DB defns, form defns, documents, interface defns, DB defns, form defns, documents, interface defns,

source code, help text, executables, icons, source code, help text, executables, icons, makefiles, ...makefiles, ...

SpecSpec

Table DefnsTable Defns

Interface DefnsInterface Defns

ArchitectureArchitecture

View DefnsView Defns

ER DiagramER DiagramCustomer

Order

ScheduledDelivery

Product

Salesperson

FormsForms

BillCustomer

UpdateMarketing

Inventory

AuthorizeCredit

OrderEntry

ScheduleDelivery

Business Business ProcessProcess

Emp.Sal < Emp.Mgr.Sal

Business RulesBusiness Rules

The Schemas are the DataThe Schemas are the Data Plus relationships between the schemasPlus relationships between the schemas

Depends-on, generated-from, authored-by, Depends-on, generated-from, authored-by, ……

This isn’t like data-processing data that This isn’t like data-processing data that you store in relational databasesyou store in relational databases It’s more like the content of a SQL catalogIt’s more like the content of a SQL catalog

It’s inherently heterogeneous & It’s inherently heterogeneous & distributeddistributed

SQL doesn’t help a lot to manipulate itSQL doesn’t help a lot to manipulate it

What’s Meta Data What’s Meta Data Management?Management?

Generic mechanisms to help store, Generic mechanisms to help store, search, and manipulate meta data.search, and manipulate meta data.

Main componentsMain components A database engineA database engine Information Models (i.e. meta models)Information Models (i.e. meta models)

Schemas whose instances are meta data (i.e. Schemas whose instances are meta data (i.e. models)models)

ToolsTools Generic – for any meta data applicationGeneric – for any meta data application For Vertical Applications – usually design-time, For Vertical Applications – usually design-time,

but can be run-timebut can be run-time

Meta Data Manager ArchitectureMeta Data Manager Architecture

select allselect all

custempdept

dnodna Bill

Customer

UpdateMarketing

Inventory

AuthorizeCredit

OrderEntry

ScheduleDelivery

Customer

Order

ScheduledDelivery

Product

Salesperson

SpecSpec

TablesTables

CodeCode

ArchitectureArchitecture

ViewsViews

ERDERDCustomer

Order

ScheduledDelivery

Product

Salesperson

FormsForms

DatabaseDatabase

Database SystemDatabase System

Repository Mgror OO DBMS• Objects, propertiesObjects, properties• Rich relationshipsRich relationships• ExtensibilityExtensibility• VersioningVersioning• ConfigurationsConfigurations

Information Model• Predefined types

Generic ToolsGeneric Tools• BrowserBrowser• Scripting languageScripting language• Data translators Data translators (import/export) (import/export)• Model editorModel editor• Model mergeModel merge• Component mgrComponent mgr

Typical Usage PatternTypical Usage Pattern

Scanner (importer)

Object-OrientedStructure

Meta Data Source Database catalog, language introspection, modeling tool, prog. environment, …

Meta DataApplication

Database design, impact analysis,data translation, data integration,view integration, message mapping, data whse loading, data migration, ….

Generator (exporter)

Code SQL DDL, interface defns, XSLT, Java, …

Meta TerminologyMeta Terminology

meta-meta-meta data = meta-meta-model

Definition of “object”

meta-meta data = meta-model

Definition of “Table”

meta data = model

Definition of (schema for) the Employee Table

data Employee Table

Vertical Applications IVertical Applications I Database designDatabase design

Map ER model to SQL schemaMap ER model to SQL schema Reverse engineer SQL schema to ER modelReverse engineer SQL schema to ER model

DB Application developmentDB Application development Map SQL schema to default formMap SQL schema to default form Map business rule to SQL constraints and Map business rule to SQL constraints and

form validation codeform validation code Manage dependencies between code and Manage dependencies between code and

schemas and formsschemas and forms

Vertical Applications IIVertical Applications II Data translationData translation

Import source and target schemasImport source and target schemas Build a mapping between themBuild a mapping between them Data translator interprets the mappingData translator interprets the mapping

Schema integrationSchema integration Merge data sources into a global schema Merge data sources into a global schema

View integrationView integration Define use-case scenarioDefine use-case scenario Identify views for each use-caseIdentify views for each use-case Integrate views into a conceptual Integrate views into a conceptual

schemaschema

Vertical Applications IIIVertical Applications III Message MappingMessage Mapping

Map messages from one format to Map messages from one format to anotheranother

Data WarehousingData Warehousing Import schemas of data sourcesImport schemas of data sources Identify overlapping attributes, etc.Identify overlapping attributes, etc. Build data cleaning scriptsBuild data cleaning scripts Build data transformation scriptsBuild data transformation scripts Enable data lineage tracingEnable data lineage tracing

Vertical Applications IVVertical Applications IV

Data MigrationData Migration Import a schema and its modified versionImport a schema and its modified version Build a mapping between themBuild a mapping between them Data migration tool inteprets the mappingData migration tool inteprets the mapping

Scientific data management Scientific data management Merge schemas from related experimentsMerge schemas from related experiments Manage transformations of experimental dataManage transformations of experimental data Track evolution of schemas and Track evolution of schemas and

transformationstransformations

Vertical Applications VVertical Applications V Information Resource ManagementInformation Resource Management

Inventory control of schemas and apps Inventory control of schemas and apps for large enterprisesfor large enterprises

Import & browse schemas and Import & browse schemas and interfacesinterfaces

Impact analysisImpact analysis Semantic query processingSemantic query processing

Pose queries against conceptual modelPose queries against conceptual model Automatically map the query to DB Automatically map the query to DB

schemasschemas

Vertical Applications VIVertical Applications VI Integrated CASEIntegrated CASE

Model the application in UMLModel the application in UML Translate the UML to interfaces, schemas, Translate the UML to interfaces, schemas,

etc.etc. Tools for consistency checking, test Tools for consistency checking, test

generation, impact analysis, ….generation, impact analysis, …. Integrating customized applicationsIntegrating customized applications Workflow design and managementWorkflow design and management Document managementDocument management Application configuration managementApplication configuration management

Course TopicsCourse Topics Meta data applicationsMeta data applications

Data translation [Popa et al]Data translation [Popa et al] Data integration [Spaccapietra & Parent]Data integration [Spaccapietra & Parent]

Schema managementSchema management Meta meta models [Noy et al]Meta meta models [Noy et al] Properties of mappings [Hull]Properties of mappings [Hull]

Meta data mechanismsMeta data mechanisms Lineage tracing [Buneman et al.] (Wang-Chiew Lineage tracing [Buneman et al.] (Wang-Chiew

Tan)Tan) Mapping generation [Atzeni & Torlone]Mapping generation [Atzeni & Torlone] Schema merging [Buneman, Davidson, Kosky]Schema merging [Buneman, Davidson, Kosky] Model Management [Bernstein]Model Management [Bernstein]

Model ManagementModel Management Model – a complex information structure, e.g., XML Model – a complex information structure, e.g., XML

schema, SQL schema, .NET type def, UML model.schema, SQL schema, .NET type def, UML model. Mapping – a transformation from one model into anotherMapping – a transformation from one model into another

Map between two XML schemas Map between two XML schemas Map SQL schema to XML schema Map SQL schema to XML schema Map data source to data warehouse Map data source to data warehouse Map classes to data source defnsMap classes to data source defns

Model Management AlgebraModel Management Algebra Match (Match (MM11, , MM22, , mapmap)) Merge (Merge (MM11, , MM22, , mapmap, , MM33)) Compose (Compose (mapmap11, , mapmap22, , mapmap33))

Copy, Delete, UpdateCopy, Delete, Update Diff, Apply, ModelGenDiff, Apply, ModelGen Select, EnumerateSelect, Enumerate

Implementation VisionImplementation Vision

MatchMerge

ComposeCopy

Apply …

Model-DrivenUI Generator

ModelManager

Object-OrientedRepository

SQLDBMS

BillCustomer

UpdateMarketing

Inventory

AuthorizeCredit

OrderEntry

ScheduleDelivery

Customer

Order

ScheduledDelivery

Product

Salesperson

select allselect all

custempdept

dnodna

Generic ToolsGeneric Tools• BrowserBrowser• Import/exportImport/export• ScriptingScripting

• EditorsEditors• CatalogsCatalogs

OperationSpeciali-zations

InferencingEngine

Web pointersWeb pointers

//www.research.microsoft.com/~philbe //www.research.microsoft.com/~philbe

//research.microsoft.com/db/ModelMgt/ //research.microsoft.com/db/ModelMgt/