moving towards enterprise data architecture
DESCRIPTION
Presented at the 2011 Teradata Partners User Conference in San Diego, CATRANSCRIPT
Moving towards Enterprise Data Architecture
Kevin Messer, Enterprise Data Architect
Standard Register
• Standard Register (HQ: Dayton, Ohio; NYSE: SR) helps the world‟s leading organizations build and protect their reputations through expert management and precise execution of critical communications
> Secure, confidential print
> Industrial labels and logos
> Fraud prevention solutions
> MSRA-resistant armbands
> www.standardregister.com
• Kevin Messer (Presenter)
> SR Enterprise Data Architect
> Prior: Lockheed, Accenture, NCR
> www.linkedin.com/in/kevinmesser
About . . .
Moving towards Enterprise Data Architecture 2
> Multichannel marketing
> Print outsourcing
> eLearning
> NFC-enabled pokenHEALTH
Background: Improved Data Management
Moving towards Enterprise Data Architecture 3
• New initiatives
> Complex integrations
> Heterogeneous source systems
> How to avoid data duplication?
> Reference sources for data quality?
• Systems documentation challenges
> Multiple repositories
> Variety of documentation tools
> Some documentation ≠ data
> Budget and resource constraints
• Options?
Master Data Management?
Moving towards Enterprise Data Architecture 4
• Brief survey of products > Informatica, Microsoft, Oracle
> Software AG, Talend, Teradata
> Etc, etc, etc.
• Many Challenges > Large upfront cost (>$1M)
> Hidden costs - Internal staffing, training
- Process and culture changes
- How to measure Return on Investment and Total Costs?
- Where to start?
• Lower cost alternatives? > Start small, grow scope incrementally
> Every plan needs an architecture . . .
Simple Master Data Architecture
Moving towards Enterprise Data Architecture 5
Enterprise Data Data structured for reporting and
analysis, based on Reference,
Master, BU data
Business Unit Data Business Unit variations of Master
Data, used for specific markets
and transactions
Master Data Data consistently used by the
business: customer; product
Reference Data External data used by reference:
zip codes; area codes; state; city;
country
Metadata Data about data: source, formats,
business rules
This simple
diagram for
Master Data
Architecture is
easy for people to
remember and
was the basis for
the next steps,
starting with
metadata. . .
Teradata Meta Data Services (MDS)
Moving towards Enterprise Data Architecture 6
• Part of Teradata Utility Pack
• Already licensed, so essentially “free” > No client or seat licenses
• Very low-cost implementation > Windows client utility for
administration
> Microsoft IIS Server (we used laptops)
> Web browser interface (MetaSurf) for users
• Did I mention it was FREE?
Teradata MDS Architecture
Moving towards Enterprise Data Architecture 7
• MDS Repository is a Teradata database
• Versions 13 to V2R6.0
• API supports C++, Microsoft COM, XML, Java, and Excel
• ERwin® Data Modeler r7 includes a wizard
• CWM XMI for Metadata Interchange using other tools
• Supports storing metadata from other DBMS
Teradata
MDS
Repository
MDS Utilities Custom
Applications BI Tools
XML Bridge
MDS Engine
MDS APIs
Auto
DIM
Update
Diagram from “Discover Teradata Meta Data Services” by
Rudy Ezquerro, Copyright 2007 by Teradata Corporation.
Building a “Master Data Catalog”
Moving towards Enterprise Data Architecture 8
• Started “proof of concept” effort in early 2011
• Objectives:
> Use MDS as a central “catalog” of all database metadata
- Capture 100% of Teradata database schemas
- Capture 100% of SQL-compliant non-Teradata sources
- Explore approaches for non-SQL databases
> Evaluate use of MDS as central catalog/portal for both technical and business process documentation
• The slides that follow describe the steps we took and the outcomes
First Steps: Teradata Metadata
Moving towards Enterprise Data Architecture 9
• Capturing from Teradata
> We used QA databases
> Using MetaManager, connect to Teradata
> Select the databases to load
• This approach worked without difficulty
Second Step: Importing non-Teradata Metadata
Moving towards Enterprise Data Architecture 10
• Our original plan:
> Using ERwin, reverse-engineer the databases
> Export schemas from Erwin as XML (CWM XMI) files
> Use MetaBridge to import the XMI
• Software compatibility issues between Erwin and our version of MDS emerged, forcing a different approach
Alternate approach to non-Teradata Metadata
Moving towards Enterprise Data Architecture 11
• The alternate approach used Excel
> Using the MDS API Interfaces
• Microsoft Excel Macros are included with MDS Development Kit sample files
• Macros include loading descriptions for:
> Databases, tables, columns, views
> Business attributes, entities, subject areas
> Macros can be modified
• We worked with tools created by Dan Riehle
Examples
Moving towards Enterprise Data Architecture 12
• Oracle
> Created a query joining All_Tab_Columns and all_tables system views
• SQL Server
> Used INFORMATION_Schema.columns system view
• Supra II (mainframe)
> Flatten the DDL export
• Export the DDL information to Excel format
Important Tip
Moving towards Enterprise Data Architecture 13
• MetaSurf (web browser viewer of MDS) only displays Teradata data types
• When exporting DDL from non-Teradata databases, data types must be converted
• Example
> VARCHAR converted to a code of „CV‟ to display*
*Note: Teradata column type abbreviations are found in the Teradata RDBMS Data Dictionary manual.
MetaSurf Examples
Moving towards Enterprise Data Architecture 14
• Example views of metadata from an Oracle database
Lessons Learned about Teradata MDS
Moving towards Enterprise Data Architecture 15
• MDS is a powerful, flexible tool for > Quickly capturing database technical information
> Providing this database information centrally to - Database Administrators, ETL and Interface Programmers,
Support Staff
> Searching across multiple databases for similar tables, columns, attributes, etc
• MDS is also extensible, allowing it to serve as a portal to information related to databases > Data Model diagrams
> Business Process diagrams and descriptions
> Links to documentation stored in other repositories like SharePoint, Subversion, or the internet
• MDS provides a low-cost way to incrementally implement Enterprise Data Architecture and Master Data Management
Next Steps
Moving towards Enterprise Data Architecture 16
• Plans include:
> Implementing MDS in production
- Staffing
- Support
- Governance
> Position MDS as the company‟s Master Data Catalog
- Single site for documentation about data and systems
- Content to include links to documents in other repositories
Questions?
Moving towards Enterprise Data Architecture 17
?
Kevin Messer
Dan Riehle [email protected]
www.getreals.com