effective capture of metadata using ca e rwin data modeler 09232010
DESCRIPTION
TRANSCRIPT
![Page 1: Effective capture of metadata using ca e rwin data modeler 09232010](https://reader034.vdocument.in/reader034/viewer/2022051609/547b5c58b479595e098b4dd9/html5/thumbnails/1.jpg)
Effective Capture of Metadata Using CA ERwin Data ModelerMetadata –Data gets meaning.
![Page 2: Effective capture of metadata using ca e rwin data modeler 09232010](https://reader034.vdocument.in/reader034/viewer/2022051609/547b5c58b479595e098b4dd9/html5/thumbnails/2.jpg)
PAGE 2
Abstract
• In any data centric environment either Data warehouse or an OLTP data environment ,vital information anybody looks for is the purpose and content of the tables and columns. Its metadata of the data, provides more insight about the structure. In many organization, the lack of metadata has lead to redundant definition of table and columns , ignorance of real capability of your data centric system, inability to define standards and build the knowledge layer for the business. In the case of data warehouse its vital to capture the source and transformation rules along with dimensional model as it will help fixing incorrect mappings early in the life cycle, effective communication to ETL team and store the ETL rules close to the data model. Without metadata ,it will lead to individual's interpretation of data just like blind folded touching the elephant. In this webinar we will discuss about the various flexible features provided by CA ERwin Data Modeler for data warehouse and relational model.
![Page 3: Effective capture of metadata using ca e rwin data modeler 09232010](https://reader034.vdocument.in/reader034/viewer/2022051609/547b5c58b479595e098b4dd9/html5/thumbnails/3.jpg)
PAGE 3
Speaker Bio
• Sampath Kumar brings 11 years of experience in implementing small, medium and large scale data centric environments (both relational and data warehouse ) using CA ERwin Modeling suite of products. He is currently working for Infosys Technologies Limited as Technology Architect in their DW/BI practice group. Prior to that he was working with American Express Credit Cards as Sr System Analyst for their Worldwide Risk Information Management group. In all his experience he has worked extensively in various database products ,BI tools ,data modeling and related products offered by CA such as CA ERwin Data Modeler, CA ERwin Model Validator,CA ERwin Model Manager and CA ERwin Data Profiler.
![Page 4: Effective capture of metadata using ca e rwin data modeler 09232010](https://reader034.vdocument.in/reader034/viewer/2022051609/547b5c58b479595e098b4dd9/html5/thumbnails/4.jpg)
PAGE 4
Agenda
• Not going to focus on the known fundamentals and data jargons
• Effective capture of metadata in Data warehouse environment
– Case study using Customer_Dim
• Other Flexible Options to capture metadata into data model
– Using an example.
![Page 5: Effective capture of metadata using ca e rwin data modeler 09232010](https://reader034.vdocument.in/reader034/viewer/2022051609/547b5c58b479595e098b4dd9/html5/thumbnails/5.jpg)
Effective capture of metadata in Data warehouse environmentCase study using Customer_Dim
![Page 6: Effective capture of metadata using ca e rwin data modeler 09232010](https://reader034.vdocument.in/reader034/viewer/2022051609/547b5c58b479595e098b4dd9/html5/thumbnails/6.jpg)
PAGE 6
Problem Statement
In any data warehouse development project, some of the major challenges include
• Effective capture of metadata information in data model such as data source ,transformation, enrichment and data synchronization rules etc.
• Keeping data model in synch with changing ETL rules and vice versa i.e. keeping ETL rules close to DW Data model (blueprint of your DW data)
• Early identification of incorrect ETL mappings in the complete lifecycle.
![Page 7: Effective capture of metadata using ca e rwin data modeler 09232010](https://reader034.vdocument.in/reader034/viewer/2022051609/547b5c58b479595e098b4dd9/html5/thumbnails/7.jpg)
PAGE 7
Problem Statement contd…
• Effective communication of captured metadata information by data modeler to other teams such as ETL
![Page 8: Effective capture of metadata using ca e rwin data modeler 09232010](https://reader034.vdocument.in/reader034/viewer/2022051609/547b5c58b479595e098b4dd9/html5/thumbnails/8.jpg)
PAGE 8
Background
3 Important pieces of information:
• Source of data
• Transformation rules-The method in which the data is getting extracted, transformed and loaded
• Frequency: The frequency and timing of data warehouse updates.
![Page 9: Effective capture of metadata using ca e rwin data modeler 09232010](https://reader034.vdocument.in/reader034/viewer/2022051609/547b5c58b479595e098b4dd9/html5/thumbnails/9.jpg)
PAGE 9
CA ERwin Features
CA ERwin Data Modeler supports the following salient features to capture the metadata information effectively.
• Data warehouse Sources Dialog
• Columns Editor
• Data Movement Rules Editor
![Page 10: Effective capture of metadata using ca e rwin data modeler 09232010](https://reader034.vdocument.in/reader034/viewer/2022051609/547b5c58b479595e098b4dd9/html5/thumbnails/10.jpg)
PAGE 10
Customer_Dim
Snapshot
– customer_SKID
– snapshot_Begin_Date
– snapshot_End_Date
– current_ind
• Basic Information
– Customer name
– Customer Date of Birth
– Driving License
• Address
– Mailing Address
– Physical Address
Communication
Email Address
Phone
Fax
Segmentation
Shopping
Behavior
![Page 11: Effective capture of metadata using ca e rwin data modeler 09232010](https://reader034.vdocument.in/reader034/viewer/2022051609/547b5c58b479595e098b4dd9/html5/thumbnails/11.jpg)
PAGE 11
Capturing Data Source
![Page 12: Effective capture of metadata using ca e rwin data modeler 09232010](https://reader034.vdocument.in/reader034/viewer/2022051609/547b5c58b479595e098b4dd9/html5/thumbnails/12.jpg)
PAGE 12
Creating Customer_Dim
![Page 13: Effective capture of metadata using ca e rwin data modeler 09232010](https://reader034.vdocument.in/reader034/viewer/2022051609/547b5c58b479595e098b4dd9/html5/thumbnails/13.jpg)
PAGE 13
Make it Dimensional Model
![Page 14: Effective capture of metadata using ca e rwin data modeler 09232010](https://reader034.vdocument.in/reader034/viewer/2022051609/547b5c58b479595e098b4dd9/html5/thumbnails/14.jpg)
PAGE 14
Make it Dimensional Model…
![Page 15: Effective capture of metadata using ca e rwin data modeler 09232010](https://reader034.vdocument.in/reader034/viewer/2022051609/547b5c58b479595e098b4dd9/html5/thumbnails/15.jpg)
PAGE 15
Data Source Enabled…
![Page 16: Effective capture of metadata using ca e rwin data modeler 09232010](https://reader034.vdocument.in/reader034/viewer/2022051609/547b5c58b479595e098b4dd9/html5/thumbnails/16.jpg)
PAGE 16
Data warehouse Source Selector
![Page 17: Effective capture of metadata using ca e rwin data modeler 09232010](https://reader034.vdocument.in/reader034/viewer/2022051609/547b5c58b479595e098b4dd9/html5/thumbnails/17.jpg)
PAGE 17
Data warehouse Source
![Page 18: Effective capture of metadata using ca e rwin data modeler 09232010](https://reader034.vdocument.in/reader034/viewer/2022051609/547b5c58b479595e098b4dd9/html5/thumbnails/18.jpg)
PAGE 18
Data warehouse Sources
The “Import other” provides three options to import the table structure
• Flat File
• Database/Script
• Model Manager
![Page 19: Effective capture of metadata using ca e rwin data modeler 09232010](https://reader034.vdocument.in/reader034/viewer/2022051609/547b5c58b479595e098b4dd9/html5/thumbnails/19.jpg)
PAGE 19
Importing table from CA ERwin® Model Manager
Customer
customer_id
customer_first_name
customer_last_name
dob
driving_license_nbr
driving_license_state
Customer_Address
customer_id (FK)
mailing_address_line1
mailing_address_line2
mailing_city
mailing_state
mailing_county
mailing_country
physical_address_line1
physical_address_line2
physical_county
physical_city
physical_state
physical_country
Customer_Segmentation
customer_id (FK)
behavioral_segment_nbr (FK)
shopping_segment_nbr (FK)
Behavioral_Segment
behavioral_segment_nbr
behavioral_segment_name
Shopping_Segment
shopping_segment_nbr
shopping_segment_name
Opens Model Mart Library
![Page 20: Effective capture of metadata using ca e rwin data modeler 09232010](https://reader034.vdocument.in/reader034/viewer/2022051609/547b5c58b479595e098b4dd9/html5/thumbnails/20.jpg)
PAGE 20
Import source tables
![Page 21: Effective capture of metadata using ca e rwin data modeler 09232010](https://reader034.vdocument.in/reader034/viewer/2022051609/547b5c58b479595e098b4dd9/html5/thumbnails/21.jpg)
PAGE 21
Source Tables Populated
![Page 22: Effective capture of metadata using ca e rwin data modeler 09232010](https://reader034.vdocument.in/reader034/viewer/2022051609/547b5c58b479595e098b4dd9/html5/thumbnails/22.jpg)
PAGE 22
Data warehouse Source Selector.Multiple
sources can be added
Transformation and business rules can be added here.
![Page 23: Effective capture of metadata using ca e rwin data modeler 09232010](https://reader034.vdocument.in/reader034/viewer/2022051609/547b5c58b479595e098b4dd9/html5/thumbnails/23.jpg)
PAGE 23
Source as Flat File
![Page 24: Effective capture of metadata using ca e rwin data modeler 09232010](https://reader034.vdocument.in/reader034/viewer/2022051609/547b5c58b479595e098b4dd9/html5/thumbnails/24.jpg)
PAGE 24
ETL Mapping Template-using Data Browser
![Page 25: Effective capture of metadata using ca e rwin data modeler 09232010](https://reader034.vdocument.in/reader034/viewer/2022051609/547b5c58b479595e098b4dd9/html5/thumbnails/25.jpg)
PAGE 25
Report Template Builder
![Page 26: Effective capture of metadata using ca e rwin data modeler 09232010](https://reader034.vdocument.in/reader034/viewer/2022051609/547b5c58b479595e098b4dd9/html5/thumbnails/26.jpg)
PAGE 26
Customize the Report
![Page 27: Effective capture of metadata using ca e rwin data modeler 09232010](https://reader034.vdocument.in/reader034/viewer/2022051609/547b5c58b479595e098b4dd9/html5/thumbnails/27.jpg)
PAGE 27
Generate the Metadata Report
![Page 28: Effective capture of metadata using ca e rwin data modeler 09232010](https://reader034.vdocument.in/reader034/viewer/2022051609/547b5c58b479595e098b4dd9/html5/thumbnails/28.jpg)
PAGE 28
ETL Mapping Sheet
![Page 29: Effective capture of metadata using ca e rwin data modeler 09232010](https://reader034.vdocument.in/reader034/viewer/2022051609/547b5c58b479595e098b4dd9/html5/thumbnails/29.jpg)
PAGE 29
Data Movement Rules
![Page 30: Effective capture of metadata using ca e rwin data modeler 09232010](https://reader034.vdocument.in/reader034/viewer/2022051609/547b5c58b479595e098b4dd9/html5/thumbnails/30.jpg)
PAGE 30
Documenting the rule
![Page 31: Effective capture of metadata using ca e rwin data modeler 09232010](https://reader034.vdocument.in/reader034/viewer/2022051609/547b5c58b479595e098b4dd9/html5/thumbnails/31.jpg)
PAGE 31
Attaching table to rule.
![Page 32: Effective capture of metadata using ca e rwin data modeler 09232010](https://reader034.vdocument.in/reader034/viewer/2022051609/547b5c58b479595e098b4dd9/html5/thumbnails/32.jpg)
PAGE 32
Export as Report
![Page 33: Effective capture of metadata using ca e rwin data modeler 09232010](https://reader034.vdocument.in/reader034/viewer/2022051609/547b5c58b479595e098b4dd9/html5/thumbnails/33.jpg)
Other Flexible Options to capture metadata into data modelUsing simple example
![Page 34: Effective capture of metadata using ca e rwin data modeler 09232010](https://reader034.vdocument.in/reader034/viewer/2022051609/547b5c58b479595e098b4dd9/html5/thumbnails/34.jpg)
Import from MS Excel
Using simple example
![Page 35: Effective capture of metadata using ca e rwin data modeler 09232010](https://reader034.vdocument.in/reader034/viewer/2022051609/547b5c58b479595e098b4dd9/html5/thumbnails/35.jpg)
PAGE 35
Metadata Capture from MS Excel
• When it would be useful
– Import the definitions available already into data model
– Import the definitions from business stakeholders for key columns to avoid wrong interpretation.
• Step 1: Store the model locally in the hard disk
• Step 2: Use the excel sheet “Import Definitions” or VBA macro provided by CA .
• Step 3 :Import the metadata into the model by running VBA code.
![Page 36: Effective capture of metadata using ca e rwin data modeler 09232010](https://reader034.vdocument.in/reader034/viewer/2022051609/547b5c58b479595e098b4dd9/html5/thumbnails/36.jpg)
PAGE 36
Person Table
![Page 37: Effective capture of metadata using ca e rwin data modeler 09232010](https://reader034.vdocument.in/reader034/viewer/2022051609/547b5c58b479595e098b4dd9/html5/thumbnails/37.jpg)
PAGE 37
Import Definitions
![Page 38: Effective capture of metadata using ca e rwin data modeler 09232010](https://reader034.vdocument.in/reader034/viewer/2022051609/547b5c58b479595e098b4dd9/html5/thumbnails/38.jpg)
PAGE 38
In this format
![Page 39: Effective capture of metadata using ca e rwin data modeler 09232010](https://reader034.vdocument.in/reader034/viewer/2022051609/547b5c58b479595e098b4dd9/html5/thumbnails/39.jpg)
PAGE 39
Final Step
Open the first sheet and click on “Update Entity Defns” which will update the definitions written for that particular table into the data model. Similarly click on the “Update Attribute Defns” which will update the attribute definitions.
Note:
• Keep the data model closed otherwise you will get error that it’s open.
• Make sure table and column names are exactly same as in the data model.
• It’s not only for business people but also for the data modelers who can enter the definitions in MS Excel and get the approval from the business or data management team, then it can uploaded separately using this utility.
![Page 40: Effective capture of metadata using ca e rwin data modeler 09232010](https://reader034.vdocument.in/reader034/viewer/2022051609/547b5c58b479595e098b4dd9/html5/thumbnails/40.jpg)
Capture metadata in Data Browser.
![Page 41: Effective capture of metadata using ca e rwin data modeler 09232010](https://reader034.vdocument.in/reader034/viewer/2022051609/547b5c58b479595e098b4dd9/html5/thumbnails/41.jpg)
PAGE 41
Metadata Capture using Data Browser
![Page 42: Effective capture of metadata using ca e rwin data modeler 09232010](https://reader034.vdocument.in/reader034/viewer/2022051609/547b5c58b479595e098b4dd9/html5/thumbnails/42.jpg)
PAGE 42
Conclusion
• The metadata information such as “Data Source”, “Transformations rules” and “Data Movement rules” are very important for any Data warehousing efforts and it’s very critical to capture the correct information.
• Metadata from data management standpoint , reduces considerable amount of time while consolidating the attributes or entities or databases during acquisition or merger.
• Knowing the importance of metadata for the data model ,CA ERwin has provided these flexible options which can be leveraged to make the data model & data more meaningful.
![Page 43: Effective capture of metadata using ca e rwin data modeler 09232010](https://reader034.vdocument.in/reader034/viewer/2022051609/547b5c58b479595e098b4dd9/html5/thumbnails/43.jpg)
PAGE 43
Questions?
In case of any additional questions you can reach me in