Data Mapper Manual

L i n k T e c h n i c a l S e r v i c e s

w w w . l i n k t e c h n i c a l . c o m

j s h a e n i n g @ l i n k t e c h n i c a l . c o m

2 4 8 - 7 5 6 - 0 0 8 9

Version 2.0

1 | P a g e

Contents Overview ................................................................................................................................................................................. 3

Data Mapper Dashboard ......................................................................................................................................................... 4

Data Map Screen ..................................................................................................................................................................... 5

Column Level Functionality ..................................................................................................................................................... 6

Source Data Columns .......................................................................................................................................................... 6

Transformation ................................................................................................................................................................... 7

ETL Code .............................................................................................................................................................................. 8

Unit Test .............................................................................................................................................................................. 9

Changes ............................................................................................................................................................................. 10

Questions .......................................................................................................................................................................... 11

Table Info .............................................................................................................................................................................. 12

Group Level Functionality ..................................................................................................................................................... 13

Source SQL ........................................................................................................................................................................ 13

Cursor Field Info ................................................................................................................................................................ 15

Compile ............................................................................................................................................................................. 16

REPORT MASTER ................................................................................................................................................................... 17

Report Selector ................................................................................................................................................................. 18

DM Manual ....................................................................................................................................................................... 19

Maintenance ......................................................................................................................................................................... 20

Import Meta Data ............................................................................................................................................................. 21

Add DDL to Data Mapper .............................................................................................................................................. 21

Import Table and Column Info ...................................................................................................................................... 22

Clone Target Tables ....................................................................................................................................................... 23

Manage Projects ............................................................................................................................................................... 24

Manage Groups ................................................................................................................................................................. 25

Assign Groups.................................................................................................................................................................... 26

Manage Connections ........................................................................................................................................................ 27

Manage Reports ................................................................................................................................................................ 28

Manage PROC Builder ....................................................................................................................................................... 28

Manage Status Code ......................................................................................................................................................... 29

Manage XREFS ................................................................................................................................................................... 30

Compile All Procs .............................................................................................................................................................. 31

2 | P a g e

3 | P a g e

Data Mapper

Overview The primary objective of the Data Mapper tool (DM) is to facilitate communications between members of the

Conversion team. A recent upgrade to the system provides the ability for the DM to generate the Extract Transform

Load (ETL) code. The primary actors in this process are the Data Analyst, ETL Developer, and the QA tester. The mapping

process normally begins with the data analyst determining the technical requirements for each of the migrated data

elements. Each extracted element has a source location. It then needs to be transformed and loaded into the target

tables.

The next beneficiary of the system would be the ETL developer who will get the specifications for the transformation of

data from the Data Mapper. The ETL developer has the option of writing the code using Data Mapper or using the tool

of their choice. Once they have written the necessary code, they will update the status of the mapping which will advise

the QA Tester that their code is ready for testing.

The QA Tester will read the same transformation that the ETL developer reads, and will write a SQL statement that will

return the expected result from their perspective. We now have the capability to test the individual mappings.

In addition to the above functions, the Data Mapper tool can be used to test entire groups of data which we use for

regression and parallel testing tasks.

4 | P a g e

Data Mapper Dashboard This screen provides a quick view of a couple of key metrics along with recent user activity. To return to this screen click

the Data Mapper logo.

5 | P a g e

Data Map Screen The web based interface includes a tree view that allows the user to easily find the target table/column they will be

mapping.

At the top of the tree view are two dropdown lists that will filter by the Project and then the Status of the targeted field.

Data is organized into three levels. Group Info, Table Info, and Column Info. Additional functionality that is not target

related is organized under the Reports and Maintenance menus on the left. DM has Context related sub menus. For

instance, at the Column Info level, the user can select from the submenu the appropriate item as needed.

6 | P a g e

Column Level Functionality

Source Data Columns Provide the Data Analyst the ability to define the source data fields used in the transformation. The top screen is a grid

that shows the existing items. The screen below is a Form used to add/edit items. Table and Column fields are auto

completed which means once you start typing all matching items will appear in a drop down list. Once you have added

the Source columns click Paste Transformation to copy these values to the transformation text. If additional instructions

beyond a direct move of the data is required you can edit as needed.

7 | P a g e

Transformation Transformation includes the instructions provided by the Data Analyst. The table/column references should already be

there as a result of clicking the Past Transformation as noted previously. Status is used to apprise other members of the

team of the current state of the mapping. The status at this point should be marked as mapped. A comments field is for

additional information.

8 | P a g e

ETL Code Used to enter the T-SQL code necessary to transform the data. Sequence numbers control the order in which the ETL

Code is executed in the stored procedure. The System will automatically check the syntax and identify any problems as

shown on the screen below. Keyboard shortcuts are provided to quickly call up the Target Field @x and the Source

Fields @s as shown in the screen shot below.

For cursor based workflows the syntax shoud be Set @TargetVariable = @SourceVariable.

For set based workflows the syntax only requires SourceVariable along with any required T-SQL like a Case statement.

9 | P a g e

Unit Test The Business Analyst or Developer adds the expected results. The Key Fields are used to compare the test record with

the actual record generated by the ETL code. Records additions or edits as shown on the screen below.

10 | P a g e

Changes Any changes made to the transformation instructions or ETL Code get logged as shown on this tab.

Logging solves the problem of “Who changed the mapping?” question.

11 | P a g e

Questions If there is a need for additional information regarding a target field the user can ask a question using the form below.

The person responding will receive an Email as shown below.

12 | P a g e

Table Info On this screen, the Business Analyst would define any record selection criteria for returning data from the table.

There is a section for comments as shown below.

When your cursor type is ‘Set Based’ the Post Load SQL will execute after the table insert.

13 | P a g e

Group Level Functionality

Source SQL Prior to this step in the process the Data Analyst should have completed the Source Data Columns process. These steps

identify all of the source tables/fields that will be needed for the Workflow. We have a couple of reports as shown

below that will help identify the tables/columns needed.

The developer will use this information to determine the appropriate syntax to join the tables in their query along with

the fields to include in their Select clause.

For Set based queries there is a requirement to include a literal (—FromPoint—)before the beginning of the from clause

as shown in the sample below.

14 | P a g e

15 | P a g e

Cursor Field Info

The elements referenced in the SQL select are stored and shown below. This data is used to create the Source variables

that the ETL can reference. Additional variables, if needed, by clicking Add New Record.

16 | P a g e

Compile This function is for the Developer to compile the Stored Procedure that DM generated. The Download File button allows

the developer to save a copy of the procedure on the Clients machine for execution as part of their conversion load.

17 | P a g e

REPORT MASTER This section of the menu allows you to select Report Master for a list of out of the box reports and DM Manual which is a

PDF of this manual.

18 | P a g e

Report Selector The user can select from a list of standard reports that have some prebuilt filter criteria. The developer can also add

reports by selecting MANAGE REPORTS in the Maintenance Menu.

19 | P a g e

DM Manual

The user can download a copy of the manual in a PDF format.

20 | P a g e

Maintenance

In addition to the Group/Table/Column functionality Data Mapper provides the following functionality to support

importing data and other routine support type tasks.

21 | P a g e

Import Meta Data

Add DDL to Data Mapper

Source and Target data are unique in each conversion. Data structures (Meta Data) from the client’s system are

imported into DM so they can be available for the creation of transformation rules, code, test cases, etc.

The import is a two step process. The first step is outside of the DM application. Using SQL Server Management Studio

select the database, right mouse click and select Generate and Publish Scripts. A file will be generated to create the

tables in Azure. There is a limitation in Azure that you can only connect to one database at a time. We get around this

limitation by having the schema name include both the database and schema with a $ separating the two. A required

edit of the DDL generated replaces the schema name with a combination of database$schema name as shown below.

If this is the first time you are loading the tables, you will also need to add the new schema to the database.

CREATE TABLE [dbo].[ccst_activity](

becomes CREATE TABLE [CCDB#dbo].[ccst_activity](

22 | P a g e

Import Table and Column Info

The user will need to need to import the tables/columns into the Data Mapper Table_Info and Column_Info tables. If

the Table has already been imported, you will only need to select the Column_Info type to add any new columns.

To facilitate this, we begin by selecting TABLE_INFO from the Type drop down list. Complete the items as shown in the

screen shot above and select Preview Data. The Preview Box contains the tables to be added. If the list is what you

expected then click the Import Data button to load the new Table

The next step is to add the column information. Select COLUMN INFO from the Type drop down list. Click the Prev Data

button and review list of columns added. If this is not the initial load you may want to copy the contents into the

Windows clipboard and paste the values into an email or document for reviewing the conversion impact of these new

fields. If you are OK with the values shown then click the Import Data button to load the values into the Column_Info

table.

Please note that client is never loaded into the DM cloud platform.

23 | P a g e

Clone Target Tables

It is not unusual for a generic table target table like contact data to have from different source references. For instance,

the Claimant contact record may be in one file, and Insured contact information may be in another table. Each of these

records has a unique set of attributes. Shown below is an example of this:

cc_contact_clmt.Name = claimant.Name

cc_contact_insured.Name = insured.Name

When importing a cloned table into Data Mapper, the system retains the Base Table Name, so the generated code will

insert the data into the Base Table. However it will use mapping information from the cloned table.

The first step in the process is select TABLE_INFO CLONE from the Type drop down list. Enter the base table and the

cloned table name as shown above. Click Preview data and review the results. If for some reason you incorrectly

spelled the base table name the Preview text box would be empty. Click the Import Data to create the Cloned Target

table.

The process for importing cloned columns is very similar to importing regular columns. Change the Type from

TABLE_INFO_CLONE to COLUMN_INFO_CLONE click

24 | P a g e

Manage Projects If there is more than one project that DM is managing they can be segmented using the screen below. Mappings are

filtered at the project level. Adding a project called ‘In Progress’ may be useful to avoid your code being part of the next

build.

25 | P a g e

Manage Groups Stored procedures are built at the group level. It is quite common that one group procedure could insert rows into

multiple tables.

26 | P a g e

Assign Groups Each target table is assigned a Group. Once a target table has a group it will appear on the Tree View.

27 | P a g e

Manage Connections Each data source is managed by the screen below.

28 | P a g e

Manage Reports Management Studio is used to create new reports. Once you confirm the data retrieved is correct, you can add the

report as shown below.

The label and value fields allow you to have up to 4 variables to filter your data.

Manage PROC Builder This feature should only be used by site admins to customize the way the stored procedure is built.

29 | P a g e

Manage Status Code Status Codes provide the ability to generate reports by each category. Consult with your project manager before

changing any of these codes.

30 | P a g e

Manage XREFS

A routine part of any conversion is converting the clients coding of values in their source system to the appropriate

Target system Code.

If there are a lot of source values for a category it will be easier to populate the Conv_Xref table via SQL versus keying

the records manually. To accomplish this we will create a SQL statement that will generate the Insert SQL DML

necessary to add the records. You will typically run the initial query against the database that contains the source data

records. The query that this creates will then be run in the AZURE Environment.

Step one insert Source Values First

Select distinct 'Insert into [Data_Mapping].[Conv_XRef] (Category_Code, Source_Value, Source_Description) Values (''PersonType'', ' + '''' + PersonType + ''',' + '''' + PersonType + ''');' from [AdventureWorks2014$Person].[Person]

A second SQL statement is used to populate the drop down list for the potential target values. The example shown

below is a manually created insert script that will also be run in Azure. If there is a cross reference table in the target

system then you could write a query (similar to the one above) to generate the target values.

Insert into Data_Mapping.Conv_Xref_Target values ('PhoneType', 'W', 'Work Phone') Insert into Data_Mapping.Conv_Xref_Target values ('PhoneType', 'C', 'Cell Phone') Insert into Data_Mapping.Conv_Xref_Target values ('PhoneType', 'H', 'Home Phone')

The Business Analyst familiar with the source data would use the screen below to complete the source to target

mapping.

31 | P a g e

Compile All Procs This facility will provide the ability to compile all of the procedures into one file that can be downloaded and run on the

Client Site.

The text box shown below is used to display any errors in the compilation process. If there are no errors the drop

procedure statements are generated so they can be copied and executed before loading the recently generated

procedure.