creating data marts from cobol files (isam to rdbms)

Creating Data Marts from COBOL Files (ISAM to RDBMS)

Creating Data Marts from COBOL FilesDoug Evans / Chris Martin

GDT 2008 International User ConferenceAugust 10 – 13 Lake Las Vegas, Nevada

GDT ETL

What is ETL? Extract Transform Load

Extract data from your legacy system. Transform (Convert or cleanse). Load the data into the relational database, data warehouse,

data mart or Operational Data Store. Why GDT-ETL?

Built for Legacy Systems. Native access to ISAM data. Transformation from COBOL data types to more general

types. Allows you to focus on where the data is going not how you

get access to it.



GDT ETL

Data flow for a system running GDT MF file system to a relational database

MF FilesMF Files

GDT

RDMSGDT-ETLGDT-ETL

RDMS



GDT ETL

ETL Features Command line tool and Graphical User Interface

(ETLPortal) Comprehensive error logging and process

monitoring. Will work with existing enterprise scheduling

facilities.

ETL Requirements Windows

GDT V4.1 or greater .NET Framework 2.0



PLUS

PLUS

FILE DEFINITION

FILE COBOL Select statement

Multi 01 RecordDefinitions

-------------------------------------------------------------------

SQL1GEN

Creates GDTSQL01.DLL

Using the ORDERS fileSQL1GEN process

SQLCONV.PAR



SQL1GEN process

GDT process that generates a File Access Component (GDTSQL01.DLL) To be used by GDT ETL to generically access all

Micro Focus ISAM files Configuration

SQLCONV.PAR FILE DEFINITION(S) fd’s FILE SELECT STATEMENTS MULTIPLE 01 RECORD DEFINITION PARAMETER FILES

Requires GDT V4.1 GDTSQL01.DLL contains all necessary file info.

Key information, data types, record types, fields etc..



The ETL processGDT_ETL

GDTSQL01DLL

OrderInfo.

File Access Component – Found via the GDTSYS.PAR SQL Setting

ETLCONFIG.XML – Configuration for ETL to find / access FAC and DB



GDT_ETLCommand Line Control

[-v | -version] Displays the programs build version. No other commands will

be executed after this one.

[-c | --createobj] <outputfile> Will create the necessary database objects. <outputfile> is

the flat file that will contain all of the SQL that would have been used to create the objects. Default is no value which will load everything into the database

[-l | --loaddata] <reportfreq> Will load the data into the database. <reportfreq> is an integer

representing how often GDT-ETL will report the status of the load. [DEFAULT <reportfreq> = 0]




[-r | --record] <recordname> Will only execute the given commands [-lDdcf] when it finds

the record that matches the name given in <recordname>. If used with [-i] it will limit the command for the <recordname> found in <ISAMFile>.

[-i | --isamfile] <ISAMfile> Will only execute the given commands [-lDdcf] when it finds

the ISAM file that matches <ISAMfile> Can be used to limit the [-r] to a specific record.

[-D | --delobj] Will delete all objects in the database. Limit using [-ri]

commands. Do not use at the same time as [-d]. [-d | --deldata]

Will delete all data in the database NOT the objects. Limit using the [-ri] commands. Do not use at the same time as [-D].




[-f | --filelayout] Will display the layout of the file(s) in the file access

component. [-t | --dbtype] Will display the DB type defined in the XML config

file. [-T | --testload]

Will test GDT_ETL to make sure it will process the file access component properly before attempting any action against the database.

[-? | --help] Short help usage message will be displayed by [-?],

verbose use [--help]



Data is

in th

e data

base via the E

TL Load

Results of the ETL process



ETL in a nutshellHear from the Chief Engineer

Build the tables and the relationships in SQL Each record denotes a new table. Each sub record denotes a new table with a 1 to

many relationship with the parent record. Nested OCCURS within a file produces more relationships.

Each field produces a field in a SQL table and ETL maps COBOL data types to the SQL Servers data types.



ETL in a nutshell Hear from the Chief Engineer

Optional Modify the table schema and stored procedures to

suit your needs. Build transformations as needed.




How does ETL build database schemas? When Generating a table schema GDT-ETL makes

the following assumptions. Each file consists of 1 or more record types. Each record in a file consists of 1 or more fields and it may

contain hierarchy of sub records as denoted by the OCCURS clause.

Each field has a location in a record as well as an associated COBOL data type.

Group name and redefines are ignored.




Create the database schema and stored procedures. Build the tables and relationships Three basic data type transformations:

PIC X(NN) to char(NN) PIC 9(NN) to INT PIC S9(NN)V9(NN) to double

Setup default stored procedures for each table: For each table there are 3 separate stored procedures

Insert Update Delete

The tables can be modified but the stored procedures must reflect any schema modifications.

The store procedures can be modified but the parameters must stay the same.




Build the default database objects Setup the ETL Configuration.

Set database properties. Reference the location of the file access component.

Build the database objects.




What value do transformations add? Allows for cleansing of data before it enters the

database. Determine if it is accurate. Avoid data corruption.

Create more interesting data transformation Example: Date time




Use Data Integration tools BusinessObjects Data Integrator.

Develop the transformations by hand. Modify the table layout if desired. Modify the default stored procedures to setup desired

transformations.



ETL Reference Guides

Installation, Configuration and Operations Guide

ETL Portal User Guide Command Line HELP ETL Portal HELP



ETLPortal

Setup for SQL1GEN process. All of the pieces that we talked about when processing via the Command Interface.



ETLPortal

Selection of Files via the SQLCONV.PAR file. By selecting the Disabled boxprevents that file from being processed into the GDTSQL01.DLL. (like commenting

it out in the SQLCONV.PAR file).



ETLPortal



ETLPortalRunning the SQL1GEN Process



ETLPortal

ETL Command Selection.



ETLPortal

Create Objects is selected. This will create the tables and stored procedures.Filtering by File and / or Record is possible.

Force ILLDEC recovery is checked to allow ETL to recover from Illegal Decimal.



ETLPortal

ETL create is successful!



ETLPortal

Viewing the TABLES produced under SQL Server Manager.



ETLPortal

Viewing the Stored Procedures produced under SQL Server Manager.



ETLPortal

Load Data is selected. This will load the tables by processing the stored procedures.



ETLPortal

ETL Load is successful!



ETLPortal

ETL output window shows issues with the data.



ETLPortal

Viewing the Table contents under SQL Server Manager



ETLPortal

A look at the Insert stored procedure for the PRODUCT_REC table.As is from the creation by ETL.



ETLPortal

A look at the Update stored procedure for the PRODUCT_REC table.As is from the creation by ETL.



ETLPortal

A look at the Delete stored procedure for the PRODUCT_REC table.As is from the creation by ETL.



GDT-ETL

THE DEMONSTRATION! and Questions?

creating data marts from cobol files (isam to rdbms)

Documents