master data on the fly - ms

SAP COMMUNITY NETWORK scn.sap.com

© 2012 SAP AG 1

Master Data on the fly - SAP

Planning and Consolidations,

version for the Microsoft Platform

Applies to:

SAP Planning and Consolidations, version for the Microsoft platform 7.5, and higher.

SAP Planning and Consolidations, version for the Microsoft platform 10, EPM Add-In, SP3 and higher.

For more information, visit the Enterprise Performance Management homepage.

Summary

This paper will describe how to create a SSIS package to automatically create master data based on the transactional data values. This paper is based on an actual business requirement and has been published with the permission and consent of the customer.

Authors: Daniel Jacinto and Pervendren Naidoo

Company: SAP Africa and Allied Electronics Corporation Limited (Altron)

Created on: August 1, 2012

Version: 0.02

Author Bio

Daniel Jacinto is a senior technical consultant with SAP South Africa in the Business Analytics division and is currently involved with a wide range of projects and solutions in SAP Consulting. His current areas of focus are working with SAP HANA and Planning and Consolidation Solutions

Pervendren Naidoo is a senior technical specialist at Altron Management Services in South Africa. He has over 3 years’ experience with BPC MS and SAP Business Objects BI Solutions. He has over 8 years’ experience with various Microsoft products and comes from a networking back office background. His current areas of focus are Business Objects BI and BI Mobile platforms

http://scn.sap.com/community/epm

Master Data on the fly - SAP Planning and Consolidations, version for the Microsoft Platform


© 2012 SAP AG 2

Table of Contents

Executive Summary ............................................................................................................................................ 3

Business Requirements and Summary ........................................................................................................... 3

Technical Overview ......................................................................................................................................... 5

Package Overview .......................................................................................................................................... 5

Process Overview ........................................................................................................................................... 6

Package Information ....................................................................................................................................... 7

Process Flow: Update Master Data process flow ............................................................................................. 12

Reference Material ........................................................................................................................................... 30

Related Content ................................................................................................................................................ 32

Copyright........................................................................................................................................................... 33



© 2012 SAP AG 3

Executive Summary

This paper will detail how to create master data on the fly in SAP Business Objects Planning and Consolidations, version for Microsoft. It details how this package was created to meet an actual business requirement at a particular client. The requirement was to do Human Capital planning based on several external HR payroll data sources, previously this was done manually and involved several hours of manual manipulation of external data files to meet this requirement. This package has automated and streamlined this process.

The package was built using an ETL framework which will automatically add and update the master data of the Human Capital planning application / model based on the HR transactional data from the different external data sources from the different subsidiaries.

This document is technical in nature and is assumed that the reader has an understanding of the concepts and technologies; there has been no attempt to simplify the content of the document.

Some key highlights of this package:

Automate Master Data updates and additions

Automated Data Import of HR transactional data

Automatic updating of Dimension Member workbook

Built in Data Validations

Business Requirements and Summary

Allied Electronics Corporation Limited (Altron) is invested in telecommunications, power electronics and multimedia and IT. With revenues of over US$2.8 billion, it employs more than 13,000 employees in over 150 subsidiaries. Altron implemented SAP BusinessObjects™ solutions to consolidate financial information from its subsidiaries, gaining deeper, more accurate insight enterprise wide and the ability to better manage assets and risk.

Altron is using HR payroll data from its subsidiaries to perform human capital planning in a Sustainability Environment/ AppSet. Due to the federated nature of the Altron’s business there is no single HR system throughout the group’s subsidiaries. There are several HR systems spanning from ORACLE, SAP and other 3

rd party HR systems, which represents a major challenge as the structure and the information stored in the

respective subsidiaries HR systems differs and is non-standardized.

In terms of productivity, several hours were lost in simply formatting and ensuring that the import data files are correct and contained the necessary master data values to ensure that the data import runs without any error. This brought about a requirement to standardize and automate the way in which the HR data was brought into SAP Business Objects Planning and Consolidation which would improve productivity and reduce the amount of manual effort needed to import the data into the system.

An ETL framework was built in order to cater for any new automation or ETL requirements, part of this framework involved the creation of a staging database in which any ETL related activities would take place in the staging database and not in the Environment / Appset database. This approach also lessened the security requirements as access to the Environment / Appset database could be restricted to object access with limited rights.

The package is a standard SAP BusinessObjects Planning and Consolidation, version for Microsoft Example IMPORT From SQL package that was modified to meet the business requirements. The core logic of the package is built in Microsoft SQL Server Stored procedures which get executed from within the SSIS framework.



© 2012 SAP AG 4

Figure 1: High level overview of the process

The core principle of the package is that it will import data from a staging database table, in which all of the subsidiaries have inserted there relevant HR transactional data into. When the package runs, it will import the data and perform basic checks to see if there are any transactional values that have no corresponding master data values. If there are no corresponding master data values, it will then automatically update the dimension master data and attempt to reimport the data with the updated dimension master data.

SAP HR

3rd

Party HR

ORACLE

HR

Subsidiary HR Systems

Staging DB

Environment DB

SAP BPC System



© 2012 SAP AG 5

Technical Overview

This package is built as an SSIS package and is executed from within the SAP BusinessObjects Planning and Consolidations, version for Microsoft Data Manager Framework. The original package was taken from the pre-built examples folder and modified accordingly to meet business requirements. It contains standard BPC tasks which are delivered as part of the installation.

The core logic of the package is built using Microsoft SQL Server Stored Procedures; they perform the logical checks to see if the transactional data has corresponding master data values. The package was built on Microsoft SQL Server 2008 R2 on a Windows 2008 R2 platform. It has been tested on SAP BusinessObjects Planning and Consolidations 7.5, version for Microsoft and BusinessObjects Planning and Consolidations 10, version for Microsoft platform.

Package Overview

At a high level, the package gets the records from the staging database table in which the HR transactional data has been inserted into and then checks to see if there are any values that don’t have corresponding master data values. If there are values or new ‘master data’ records, it will flag the records and automatically append the dimension with the new values. Once it has updated the dimension it will then re-import the records into SAP BPC.

Figure 2 displays the package map with all of the defined tasks. All ‘bpc_’ tasks are predefined BPC tasks and perform the activities within the BPC context.

Figure 2: Package Map



© 2012 SAP AG 6

Process Overview

The package will execute the Execute SQL task ‘sp_BuildData’ which will identify if there are any new import records. Once it has identified that there has been new transactional data, it will then check to see if there are any values that don’t have corresponding master data values. Based on the output of the Execute SQL task ‘sp_BuildData’ it will branch off the different package flows within the package. The tasks outlined in red are the ‘Update Dimension or Rebuild Master Data’ process flow, this process will update the dimension master data. The tasks outlined in blue as the ‘Import Data’ process flow, this process will simply import the data using a defined transformation and conversion file in SAP Business Planning and Consolidations, version for Microsoft.

Figure 3: Package process flow map

Note: The high level package flow is: First we check to see if there are any transactional values that don’t have a corresponding master data value. If there are any records that don’t have corresponding master data, flag them, and then perform an automated update of the dimension to append the flagged records (new master data values). Once the dimension is updated with the new values re-import the transactional data.



© 2012 SAP AG 7

Package Information

This section will detail the process flow of the package. Each task will be highlighted and detailed with rationale behind the task.

Figure 4: Package process flow map with detailed task information

Task name: sp_BuildData | Task type: [Execute SQL Task]

This Step executes the sp_BuildPayrollIMportData stored procedure with an OUTPUT parameter. The stored procedure drops the temp tables created during the last package run, then the data is checked to see if there are new records to be added or if the data will imported without new data. Based on the output parameter it will then trigger the process flow will be executed within the package.



© 2012 SAP AG 8

Figure 5: sp_BuildData execute SQL task

Code Block – SQL Stored Procedure { sp_BuildPayrollImportData }

USE [SustainabilityStaging]

GO

/****** Object: StoredProcedure [dbo].[sp_BuildPayrollImportData] Script Date:

05/30/2012 12:13:15 ******/

SET ANSI_NULLS ON

GO

SET QUOTED_IDENTIFIER ON

GO

ALTER proc [dbo].[sp_BuildPayrollImportData]

@timeid nvarchar(10),

@rcode int output

as

begin

/*

-- This Section shows the Validation Check Codes used when deciding which route will

be taken in the package

Validation Checks:

Return Code Descriptions and Logic:

0 = Success and no master data needs to be added

1 = Master Data needs to be added and that will trigger the update / add master

data chain of tasks in the data manager package

*/

declare @strSQL nvarchar(MAX)



© 2012 SAP AG 9

--we set our timeid value accordingly

set @timeid = (select [timeid] from Sustainability.dbo.mbrTime

where [ID] = @timeid)

--This section will check if the temp table exists, if the table does exist then the

table will be dropped.

if object_id('tempDB..##tmpIDnum') is not null

begin

drop table ##tmpIDnum

end

--This Section checks the Staging Database for the tmpMDPayrollImport table. If the

table exists, the table is dropped.

if object_id('tmptblMDPayrollImport') is not null

begin

drop table SustainabilityStaging.dbo.tmptblMDPayrollImport

end

--We build the temp tables with the ID‟s of all the existing records in the BPC

application Dimension MBR file.

select distinct([id]) into ##tmpIDnum from sustainability.dbo.mbrHC_employees

--we need to flag our invalid records which have no ID or Passport numbers to be

input into the tblImportPayrollInvalidData Table

if exists(select [IDNumber], [PassportNumber] from

SustainabilityStaging.dbo.tblImportPayrollData

where ([IDNumber] is null or LEN([IDNumber]) = 0 )and ([PassportNumber] is null or

LEN([PassportNumber]) = 0 ))

begin

begin transaction trn_MoveInvalidRec

insert into SustainabilityStaging.dbo.tblImportPayrollInvalidData

([FirstName],[LastName],[CompanyName],[PersonnelArea],[PersonnelSubarea],[PersNo],

[Ethnicorigin],[Gender],[BirthDate],[IDNumber],[PassportNumber],[Nationality],[Disabi

lity],[OccupationLevel],[HireDate],[EmployeeSubgroup],[EmploymentIndicator],[Employee

Cnt])

select

[FirstName],[LastName],[CompanyName],[PersonnelArea],[PersonnelSubarea],[PersNo],[Eth

nicorigin],[Gender],[BirthDate],[IDNumber],[PassportNumber],[Nationality],[Disability

],[OccupationLevel],[HireDate],[EmployeeSubgroup],[EmploymentIndicator],[EmployeeCnt]

FROM [SustainabilityStaging].[dbo].[tblImportPayrollData]

where ([IDNumber] is null or LEN([IDNumber]) = 0 )and ([PassportNumber] is

null or LEN([PassportNumber]) = 0 )

commit transaction trn_MoveInvalidRec

delete from [SustainabilityStaging].[dbo].[tblImportPayrollData]

where ([IDNumber] is null or LEN([IDNumber]) = 0 )and ([PassportNumber] is null or

LEN([PassportNumber]) = 0 )

end

--we need to flag our records which are blank for OCCUPATIONLEVEL as this is Master

Data added to the mbr%Dim% table.

if exists(select [OccupationLevel] from

SustainabilityStaging.dbo.tblImportPayrollData



© 2012 SAP AG 10

where ([OccupationLevel] is null or LEN([OccupationLevel]) = 0 ))

begin

begin transaction trn_MoveInvalidRec

insert into SustainabilityStaging.dbo.tblImportPayrollInvalidData

([FirstName],[LastName],[CompanyName],[PersonnelArea],[PersonnelSubarea],[PersNo],

[Ethnicorigin],[Gender],[BirthDate],[IDNumber],[PassportNumber],[Nationality],[Disabi

lity],[OccupationLevel],[HireDate],[EmployeeSubgroup],[EmploymentIndicator],[Employee

Cnt])

select

[FirstName],[LastName],[CompanyName],[PersonnelArea],[PersonnelSubarea],[PersNo],[Eth

nicorigin],[Gender],[BirthDate],[IDNumber],[PassportNumber],[Nationality],[Disability

],[OccupationLevel],[HireDate],[EmployeeSubgroup],[EmploymentIndicator],[EmployeeCnt]

FROM [SustainabilityStaging].[dbo].[tblImportPayrollData]

where ([OccupationLevel] is null or LEN([OccupationLevel]) = 0 )

commit transaction trn_MoveInvalidRec

delete from [SustainabilityStaging].[dbo].[tblImportPayrollData]

where ([OccupationLevel] is null or LEN([OccupationLevel]) = 0 )

end

--We now do a validation check on the records we received from the source systems to

the records we just created in the temp table above. We build a new

tmptblMDPayrollImport table we dropped earlier so we don‟t have any duplicates in our

table to be imported. Any new records are then added to the tmptblMDPayrollImport

table.

select * into SustainabilityStaging.dbo.tmptblMDPayrollImport from

SustainabilityStaging.dbo.tblImportPayrollData where

([IDNumber] is null or [IDNumber] not in (select [id] from ##tmpIDnum)) and

([PassportNumber] is null or [PassportNumber] not in (select [id] from

##tmpIDnum))

--A Validation code is then generated. If there are records added to the

tmptblPayrollImport table the Validation code is set to „1‟. This will also start

the Package on the route definded on the Right shown in the Full Package Landscape.

if (select count(*) from SustainabilityStaging.dbo.tmptblMDPayrollImport) <> 0

begin

--we need to update master data

set @rcode = 1;

end

else

-- If there are no records in the above table then the Left Route in the Pacakge

Landscape is taken.

-- A check to find out if the tmptblTDPayrollImport table exists, if it does the

table is dropped.

-- A dynamic SQL statement(strSQL) is setup to create the tmptblTDPayrollImport

again.

begin

set @strSQL = '--we need to check if the table exists

if not exists (select * from sys.objects where object_id =

object_id(N''[dbo].[tmptblTDPayrollImport]'') and type in (N''U''))

begin

create table [dbo].[tmptblTDPayrollImport](



© 2012 SAP AG 11

[IDNumber] [varchar](50) null,

[Gender] [varchar](50) null,

[Time] [varchar](8) not null,

[EmployeeCnt] [int] not null,

[CompanyName] [varchar](255) not null,

[OccupationLevel] [varchar](150) not null,

[Ethnicorigin] [varchar] (50) not null

) on [PRIMARY]

end

else

begin

truncate table [dbo].[tmptblTDPayrollImport]

end '

exec(@strSQL)

--We run another SQL String to import the data from the tblImportPayrollData table

into the tmptblTDpayrollImport

if @timeid <> ''

begin

set @strSQL = 'set ansi_nulls off

Insert into [dbo].[tmptblTDPayrollImport]

([IDNumber],[Gender],[Time],[EmployeeCnt],[CompanyName],

[OccupationLevel],[Ethnicorigin])

select (case [IDNumber] when '' '' then [PassportNumber]

when null then [PassportNumber]

when '''' then [PassportNumber]

else [IDNumber] end) as [IDNumber],[Gender],''' + @timeid + ''' as [Time],1 as

[EmployeeCnt],[CompanyName],[OccupationLevel],[Ethnicorigin] from

SustainabilityStaging.dbo.tblImportPayrollData'

exec(@strSQL)

end

set @rcode = 0;

end

end

Based on the value of @rcode it will determine the process flow path. If the @rcode = 1, it will trigger the update master data process flow. If the @rcode = 0 it will trigger the import data process flow. In order to achieve this functionality attached to the execute SQL task is an expression constraint which will based on the output of the stored procedure will determine which process flow to execute. In order to achieve the different process flows, a conditional constraint was used with an expression.



© 2012 SAP AG 12

Process Flow: Update Master Data process flow

Based on the output value of the sp_BuildPayrollImportData stored procedure, the return code ‘@rcode = 1’, should trigger the Update Master Data process flow, which will the Update Master Data of the HC_Employees dimension.

Figure 6: Update Master Data process flow

Task name: sp_UpdateLogStatus | Task type: [Execute SQL Task]

This task will insert a record into the tblStatus table, stating that the ‘Update Master Data’ process flow is being executed because there is invalid or new master data records. The table tblStatus contains event records which will aid in troubleshooting and provide a mechanism of auditability.



© 2012 SAP AG 13

Figure 7: sp_UpdateLogStatus task

Task name: sp_BuildTempMDTables | Task type: [Execute SQL Task]

This task will create the tables needed for the BPC Admin Task which is used to update the dimension. It will create two tables’ tbltmpMDHC_Employees and tblShadowMDC_Employees. The table tbltmpMDHC_Employees will be used for the BPC Admin Task to update the dimension HC_Employees. The table tblShadowMDHC_Employees will be a backup of the mbrHC_Employees table and in the event any error, it will rebuild the dimension from the tblShadowMDHC_Employees table.

Figure 7: sp_BuildTempMDTables task

Task name: df_BuildMD | Task type: [Data Flow Task]

This data flow task will execute an OLE command. This is similar to a SQL Cursor in which, it applies an OLE command to each record in the recordset. You could use a SQL cursor to apply the same logic and principles, but the OLE command is faster and consumes fewer resources than an SQL Cursor.

This is a data flow task that includes two steps outlined below and is the heart of the Update Master Data process flow.



© 2012 SAP AG 14

Figure 7: df_BuildMD dataflow task

In the dataflow task there are two tasks:

src_tmptblMDPayrollImport – OLE DB Source This is the source of the master data. It contains the contents of the mbr<DimensionName> table inclusive with the newly flagged records. The newly flagged records were flagged in the sp_BuildPayrollImportData stored procedure. Figure 7: src_tbltblMDPayrollImport OLE DB Source task

Figure 7 represents the task that connects to the database through an OLEDB Source Connection which maps to the tmptblMDPayrollImport table in the Staging Database.

sp_BuildPayrollMasterData – OLE DB Command This OLE DB command will execute the sp_BuildPayrollMasterData stored procedure. This stored procedure will apply the Insert/Update methodology for master data. If the master data value exists, it will update the record, if it doesn’t exist, it will then insert a new value. Figure 8: spBuildPayrollMasterData OLE DB command parameter mapping



© 2012 SAP AG 15

Figure 8 displays the column mapping from the OLE DB source [src_tmptblMDPayrollImport] to the input parameters for the SQL stored procedure [sp_BuildPayrollMasterData]

Figure 9: spBuildPayrollMasterData OLE DB command statement

Figure 9 displays the statement that the OLE DB command sp_BuildPayrollMasterData executes. It executes the SQL Stored procedure sp_BuildPayrollMasterData

Code Block – SQL Stored Procedure { sp_BuildPayrollMasterData }


GO

/****** Object: StoredProcedure [dbo].[sp_BuildPayrollMasterData] Script Date:

05/30/2012 12:13:15 ******/


GO

CREATE PROC [dbo].[sp_BuildPayrollMasterData]

@idnumber nvarchar(255),

@Firstname nvarchar(255),

@LastName nvarchar(255),

@EthnicOrgin nvarchar(255),

@Nationality nvarchar(255),

@Gender nvarchar(255),

@Disability nvarchar(255),



© 2012 SAP AG 16

@Birthdate nvarchar(255),

@TempPerm nvarchar(255),

@CompanyName nvarchar(255),

@HireDate nvarchar(255),

@OccupationLevel nvarchar(255),

@PersNumber nvarchar(255),

@PassportNumber nvarchar(255)

as

begin

declare @seqid numeric(8,0)

--Set the SEQID(the „counter‟ for the amount of members in the dimension) to the last

one found and then one is added for the new record.

set @seqid = (select top 1 seq from Sustainability.dbo.tbltmpMDC_Employees

order by seq desc) + 1

--Data validation is done to check if the records exist, if not the record is added

to the shadow table. This is done in “Step 2” above.

if not exists(select [id] from Sustainability.dbo.tbltmpMDHC_Employees where [id] =

@idnumber)

begin

--the user doesnt exists and so the record will need to be inserted

insert into [Sustainability].[dbo].[tbltmpMDC_Employees]

([ID],[NEWID],[EVDESCRIPTION],[PARENTH1],[ACCTYPE],[SCALING],[RATETYPE],[EthnicOrigin

],[Nationality],[Gender],[Disability],[IDNumber],[Birthdate],[TempPerm],[CompanyName]

,[HireDate],[OccupationLevel],[PersNumber],[PassportNumber],[CALC],[DIMCALC],[ISBASEM

EM],[HIR])

values (@idnumber,'',@FirstName + ' ' + @LastName,„Unallocated',

'EXP','','',@EthnicOrgin,@Nationality,@Gender,@Disability, @IDnumber,

Birthdate,@TempPerm,@CompanyName,@HireDate,@OccupationLevel,@PersNumber,@PassportNumb

er, 'N','N','Y','H1' )

end

end

Task name: bpc_AdminMakeDim | Task type: [BPC task]

This is a BPC standard task; it is the Admin_Task which is responsible for several BPC actions and processes. This task will create the dimension from SQL table after the addition or updates of the master data records. If this process is successful the sp_RebuildData Execute SQL Task is run. If the process fails the bpc_AdminMakeDimPlanB BPC task is run to rebuild the dimension from the shadow table. This is a form of a ‘rollback’. This logic was done in order to ensure that the dimension is always in a consistent state.



© 2012 SAP AG 17

Figure 10: bpc_AdminMakeDim BPC task

This task will create the dimension from the tbltmpMDHC_Employees table. One of the limitations of this task is that it will not see any tables outside of the AppSet / Environment DB. So in the task sp_BuildTempMDTables it will create the tables in the AppSet / Environment DB in order for the bpc_AdminMakeDim BPC task to use as a source for creating or building the dimension.

Task name: Replace_HC_Employees | Task type: [File System Task]

The File System Task is used to make a copy of the existing Excel dimension file and replace the file with a template we created. We do this in order to keep the Dim<DimensionName> and Dimension Excel files in sync. This is done to prevent the scenario in which the dimension will be updated, but in the event of a user processing the dimension from the member sheet it will overwrite and cause the dimension to be out of sync.

Figure 11: Replace_HC_Employees File System Task



© 2012 SAP AG 18

Task name: sp_DumpToExcel | Task type: [Execute SQL Task]

This SQL Task executes a Stored Procedure that uses the OPENROWSET command to export all the contents of the mbr<DimensionName> table to the Excel Template file which was copied by the Replace_HC_Employees step. This step will ensure that the dimension file and the mbr<DimensionName> table are in sync. This is done to prevent the scenario in which the dimension will be updated, but in the event of a user processing the dimension from the member sheet it will overwrite and cause the dimension to be out of sync.

Figure 11: sp_DumpToExcel Execute SQL task

Code Block – SQL Stored Procedure { sp_Excel}

SET ANSI_NULLS ON

GO


GO

CREATE Procedure [dbo].[SP_Excel]

As

Begin

insert into OPENROWSET('Microsoft.ACE.OLEDB.12.0','Excel 12.0 XML;

HDR=YES;

Database=\\<ServerName>\d$\BPC\Data\Webfolders\Sustainability\ADMINAPP\hc_employe

es.xlsx',

'Select

ID,EVDESCRIPTION,NEWID,PARENTH1,ACCTYPE,SCALING,RATETYPE,ETHNICORIGIN,NATIONALITY,GE



© 2012 SAP AG 19

NDER,DISABILITY,IDNUMBER,

BIRTHDATE,TEMPPERM,COMPANYNAME,HIREDATE,OCCUPATIONLEVEL,PERSNUMBER,PASSPORTNUMBER

,EMPLDATE from [Members$]')

Select

ID,EVDESCRIPTION,NEWID,PARENTH1,ACCTYPE,SCALING,RATETYPE,ETHNICORIGIN,NATIONALITY,GE

NDER,DISABILITY,IDNUMBER,

BIRTHDATE,TEMPPERM,COMPANYNAME,HIREDATE,OCCUPATIONLEVEL,PERSNUMBER,PASSPORTNUMBER

,EMPLDATE from sustainability.dbo.mbrHC_Employees

End

GO

Task name: sp_ReBuildPayrollImportData | Task type: [Execute SQL Task]

This SQL Task executes a Stored Procedure sp_ReBuildPayrollImportData which will populate the tmptblTDPayrollImport table which will be used to import the transactional data.

Figure 12: sp_ReBuildPayrollImportData Execute SQL task

Code Block – SQL Stored Procedure { sp_ReBuildPayrollImportData }


GO

/****** Object: StoredProcedure [dbo].[sp_ReBuildPayrollImportData] Script Date:

05/30/2012 13:25:11 ******/

SET ANSI_NULLS ON

GO


GO

ALTER proc [dbo].[sp_ReBuildPayrollImportData]

@timeid nvarchar(10)



© 2012 SAP AG 20

as begin

set ANSI_NULLS OFF

--we set our timeid value accordingly

set @timeid = (select [timeid] from Sustainability.dbo.mbrTime

where [ID] = @timeid)

declare @strSQL nvarchar(MAX)

-- We check to see if the tmptblTDPayrollImport still exists, if not then the table

is created. If the table does exist, all records in the table will be truncated.

set @strSQL =

if not exists (select * from sys.objects where object_id =

object_id(N''[dbo].[tmptblTDPayrollImport]'') and type in (N''U''))

begin

Create table [dbo].[tmptblTDPayrollImport](

[IDNumber] [varchar](50) null,

[Gender] [varchar](50) null,

[Time] [varchar](8) not null,

[EmployeeCnt] [int] not null,

[CompanyName] [varchar](255) not null,

[OccupationLevel] [varchar](150) not null,

[Ethnicorigin] [varchar] (50) not null

) on [PRIMARY]

end

else

begin

truncate table [dbo].[tmptblTDPayrollImport]

end '

-- we then reimport the records from the tblImportPayrollData into the newly created

table.

exec(@strSQL)

if @timeid <> ''

Begin

set @strSQL = 'set ansi_nulls off

insert into [dbo].[tmptblTDPayrollImport]

([IDNumber],[Gender],[Time],[EmployeeCnt],[CompanyName], [OccupationLevel],

[Ethnicorigin])

select (case [IDNumber] when null then [PassportNumber]

when '' '' then [PassportNumber]

when '''' then [PassportNumber]

else [IDNumber]end) as [IDNumber],

[Gender],''' + @timeid + ''' as

[Time],1,[CompanyName],[OccupationLevel],[Ethnicorigin] from

SustainabilityStaging.dbo.tblImportPayrollData'

exec(@strSQL)

end

--Execute Dynamic String to change "," character so BPC import does not fail with

"Head Dimension count and data column count are not matched".

Begin

Set @strSQL = 'UPDATE SustainabilityStaging.dbo.tmptblTDPayrollImport



© 2012 SAP AG 21

SET occupationlevel = REPLACE(LTRIM(RTRIM(occupationlevel)), '','', ''_'')'

exec(@strSQL)

End

end

Rebuild Dimension: {Plan B} process flow

This process will trigger on the event if there is any error on bpc_AdminMakeDim BPC task. In the event of an error it means that the building of a dimension has failed and has resulted in an error. There is an ‘OnError’ constraint and it will trigger the bpc_AdminMakeDimPlanB BPC task which will build the dimension using the shadow table. This shadow table was created in the step sp_BuildTempMDTables

Figure 13: Rebuild Dimension Process Flow

Note: The values for the bpc_AdminMakeDimPlanB BPC task are defined at runtime. The values are passed through the dynamic script. Please make reference to the Dynamic Script for additional details regarding the values being passed to the BPC tasks.



© 2012 SAP AG 22

Task name: bpc_AdminMakeDimPlanB | Task type: [BPC Task]

This task will rebuild the dimension HC_Employees from the tblShadowMDHC_Employees table. This task will execute if there is any errors in the task bpc_AdminMakeDim.

Figure 14: bpc_AdminMakeDimPlanB BPC task

Task name: sp_UpdateLogError | Task type: [Execute SQL Task]


Figure 15: sp_UpdateLogError Execute SQL Task



© 2012 SAP AG 23

Process Flow: Import Data process flow

Based on the output value of the sp_BuildPayrollImportData stored procedure, the return code ‘@rcode = 0’, should trigger the Import data process flow, which will the import the transactional data.

Note: Although the package is called IMPORT From SQL, it still takes the contents of the SQL table and dumps it to a text file, which will then be used for the bpc_ConvertData and bpc_LoadNProcess tasks. These tasks are the same tasks used as the standard IMPORT package. Although there have been significant improvements to the way that BPC loads data into the fact tables in version 10, if you are importing large amounts of data, you run the risk of the process failing. It is quite common to for the bpc_LoadNProcess task to fail when importing millions of records. If you are importing large amounts of records then try to break them into smaller amounts or batch them.

Figure 16: Import Data process flow

Task name: df_DumpData | Task type: [Data Flow Task]

This data flow task will dump the data from the tbltmpTDPayrollImport table which will be used for the BPC Load and Process task to import the data into BPC. This data flow tasks has two tasks: tmptblTDPaylrollImport and TDImportPayroll respectively.



© 2012 SAP AG 24

Figure 17: df_DumpData dataflow task

This is a data flow task that includes two steps outlined below.

Task name: tmptblTDPaylrollImport | Task type: [OLE DB Source]

This task will take the contents of the tmptblTDPayrollImport table in the staging database which contains the transactional data which needs to be imported into SAP BPC.

Figure 18: OLE DB Source task

Task name: TDImportPayroll | Task type: [Flat File Destination]

This task will take the contents of the tmptblTDPayrollImport table which contains the transactional data which needs to be imported into SAP BPC and dumps it into a flat file which will be used by the BPC task Load and Process.



© 2012 SAP AG 25

Figure 19: Flat File Destination task

Figure 20: Flat File Mapping

.

The mapping section defines the column mappings which will be used to generate the flat file that will be used for importing into SAP BPC. The Load and Process task is not able to take a SQL table as a source; it still dumps it to a flat file and then loads it into SAP BPC.

These column mappings have a corresponding mapping entry in the transformation file.

Task name: bpc_ConvertData | Task type: [BPC Task]

This task will take the contents of the flat file in which the data flow task df_DumpData generated and then performs conversions and transformations which are defined in the conversion and transformation file respectively.



© 2012 SAP AG 26

Figure 21: BPC Task – bpc_ConvertData

Note: The parameter values for the BPC tasks are taken from the Data Manager Dynamic Script. The values are generated at runtime and are not stored as values in the task.

Task name: bpc_LoadNProcess | Task type: [BPC Task]

This task will take the contents of the flat file in which the data flow task df_DumpData generated and then import it into the SAP BPC Environment / AppSet database fact table. This is the actual task that imports the transactional data into SAP Business Planning and Consolidations, version for Microsoft.

Figure 22: BPC Task – bpc_LoadNProcess

Note: The parameter values for the BPC tasks are taken from the Data Manager Dynamic Script. The values are generated at runtime and are not stored as values in the task.



© 2012 SAP AG 27

Task name: sp_UpdateLog | Task type: [Execute SQL Task]


This task will update the tblStatusCode table in the Staging Database with a ‘0’ code meaning that the import of the data was a success and that no new data records were added.

Figure 23: sp_UpdateLog Execute SQL Task

Error checking process flow

This process will trigger a notification task to notify the administrators that there were invalid records or that there were errors in processing of the package.

Figure 24: Error checking process flow



© 2012 SAP AG 28

Task name: check_InvalidRecords | Task type: [Execute SQL Task]

This task will check the table tblImportPayrollInvalidData for any records, if there are any records it means that there were errors in processing the transactional data



© 2012 SAP AG 29

Figure 25: check_InvalidRecords Execute SQL task

Task name: SendMailTask | Task type: [SMTP Task]

This task will send an email to the desired recipients to notify them that there were some invalid records or that there are errors in processing the package. This is meant for informational reporting purposes and to ensure that there is a level proactive fault notification.

Figure 26: SendMailTask SMTP Task



© 2012 SAP AG 30

Reference Material

Package Dynamic Script

DEBUG(ON)

PROMPT(TEXT,%TIMEVAL%,"Please enter the correct time period to post the data

to..",,"")

PROMPT(TRANSFORMATION,%TRANSFORMATION%,"Transformation file:",,,Import.xls)

PROMPT(RADIOBUTTON,%CLEARDATA%,"Select the method for importing the data from the

source file to the destination database",0,{"Merge data values (Imports all records,

leaving all remaining records in the destination intact)","Replace && clear

data values (Clears the data values for any existing records that mirror each

entity/category/time combination defined in the source, then imports the source

records)"},{"0","1"})

PROMPT(RADIOBUTTON,%RUNLOGIC%,"Select whether to run default logic for stored values

after importing",1,{"Yes","No"},{"1","0"})

PROMPT(RADIOBUTTON,%PROCESSCUBE%,"Select whether to run process cube after

importing",1,{"Yes","No"},{"1","0"})

PROMPT(RADIOBUTTON,%CHECKLCK%,"Select whether to check work status settings when

running logic.",1,{"Yes, check for work status settings before running logic","No,

do not check work status settings"},{"1","0"})

PROMPT(RADIOBUTTON,%PROCESS%,"Perform a full process the application after creating

the dimension",1,{"Yes","No"},{"1","0"})

PROMPT(RADIOBUTTON,%AVAILABLE%,"Make system available after creating the

dimension",1,{"Yes","No"},{"1","0"})

PROMPT(RADIOBUTTON,%SECURITY%,"Process security after creating the

dimension",1,{"Yes","No"},{"1","0"})

PROMPT(MESSAGE,"When this package is running, end user functionality will be

affected (i.e. data cannot be sent to the database or incorrect numbers can be

retrieved from the database while this package is running)")

INFO(%FILE%,%TEMPPATH%%RANDOMFILE%)

INFO(%TEMPFILE%,%TEMPPATH%%RANDOMFILE%)

INFO(%STAGINGDB%,SustainabilityStaging)

INFO(%DIMENSION%,HC_Employees)

INFO(%SRCTBL%,tbltmpMDC_Employees)

INFO(%SHADOWSRCETBL%,tblShadowMDC_Employees)

CONNECTION(FileConn,ConnectionString,%FILE%)

CONNECTION(SQLServerConn,ConnectionString,Data

Source=%SQLSERVER%;Provider=SQLNCLI10.1;Integrated Security=SSPI;Auto

Translate=False;)

CONNECTION(SQLServerConn,InitialCatalog,%STAGINGDB%)

CONNECTION(SQLServerConn,ServerName,%SQLSERVER%)

TASK(sp_BuildData,SqlStatementSource,execute

SustainabilityStaging.dbo.sp_BuildPayrollImportData '%TIMEVAL%',? OUTPUT)

TASK(sp_RebuildData,SqlStatementSource,execute

SustainabilityStaging.dbo.sp_ReBuildPayrollImportData '%TIMEVAL%')

TASK(bpc_ConvertData,INPUTFILE,%FILE%)

TASK(bpc_ConvertData,OUTPUTFILE,%TEMPFILE%)

TASK(bpc_ConvertData,CONVERSIONFILE,%TRANSFORMATION%)

TASK(bpc_ConvertData,STRAPPSET,%APPSET%)

TASK(bpc_ConvertData,STRAPP,%APP%)

TASK(bpc_ConvertData,STRUSERNAME,%USER%)

TASK(bpc_LoadNProcess,APPSET,%APPSET%)

TASK(bpc_LoadNProcess,APP,%APP%)

TASK(bpc_LoadNProcess,USER,%USER%)



© 2012 SAP AG 31

TASK(bpc_LoadNProcess,DATATRANSFERMODE,1)

TASK(bpc_LoadNProcess,CLEARDATA,%CLEARDATA%)

TASK(bpc_LoadNProcess,PROCESSCUBE,%PROCESSCUBE%)

TASK(bpc_LoadNProcess,PROCESSMODE,2)

TASK(bpc_LoadNProcess,FILE,%TEMPFILE%)

TASK(bpc_AdminMakeDim,DESCRIPTION,bpc_AdminMakeDim)

TASK(bpc_AdminMakeDim,APPSET,%APPSET%)

TASK(bpc_AdminMakeDim,APP,%APP%)

TASK(bpc_AdminMakeDim,USERID,%USER%)

TASK(bpc_AdminMakeDim,PROCESSMODE,4)

TASK(bpc_AdminMakeDim,DIMENSIONNAME,%DIMENSION%)

TASK(bpc_AdminMakeDim,SOURCETABLE,%SRCTBL%)

TASK(bpc_AdminMakeDim,BAPPLICATIONPROCESS,%PROCESS%)

TASK(bpc_AdminMakeDim,BTAKESYSTEMAVAILABLE,%AVAILABLE%)

TASK(bpc_AdminMakeDim,BSECURITYPROCESS,%SECURITY%)

TASK(bpc_AdminMakeDimPlanB,DESCRIPTION,bpc_AdminMakeDimPlanB)

TASK(bpc_AdminMakeDimPlanB,APPSET,%APPSET%)

TASK(bpc_AdminMakeDimPlanB,APP,%APP%)

TASK(bpc_AdminMakeDimPlanB,USERID,%USER%)

TASK(bpc_AdminMakeDimPlanB,PROCESSMODE,4)

TASK(bpc_AdminMakeDimPlanB,DIMENSIONNAME,%DIMENSION%)

TASK(bpc_AdminMakeDimPlanB,SOURCETABLE,%SHADOWSRCETBL%)

TASK(bpc_AdminMakeDimPlanB,BAPPLICATIONPROCESS,%PROCESS%)

TASK(bpc_AdminMakeDimPlanB,BTAKESYSTEMAVAILABLE,%AVAILABLE%)

TASK(bpc_AdminMakeDimPlanB,BSECURITYPROCESS,%SECURITY%)



© 2012 SAP AG 32

Related Content

Reference SQL Code

http://db.tt/RkMZD35y

SCN forum discussions for Updating of Master Data

http://scn.sap.com/thread/1688034


http://scn.sap.com/message/13442330#13442330






How to Build SSIS Packages for Business Planning and Consolidation 5.0

http://scn.sap.com/docs/DOC-4131

Microsoft SQL Server Integration Services Tutorials

http://msdn.microsoft.com/en-us/library/ms169917(v=sql.105).aspx

http://www.accelebrate.com/sql_training/ssis_2008_tutorial.htm

http://db.tt/RkMZD35y



http://scn.sap.com/message/13442330#13442330






http://scn.sap.com/docs/DOC-4131

http://msdn.microsoft.com/en-us/library/ms169917(v=sql.105).aspx

http://www.accelebrate.com/sql_training/ssis_2008_tutorial.htm



© 2012 SAP AG 33

Copyright

© Copyright 2012 SAP AG. All rights reserved.

No part of this publication may be reproduced or transmitted in any form or for any purpose without the express permission of SAP AG. The information contained herein may be changed without prior notice.

Some software products marketed by SAP AG and its distributors contain proprietary software components of other software vendors.

Microsoft, Windows, Excel, Outlook, and PowerPoint are registered trademarks of Microsoft Corporation.

IBM, DB2, DB2 Universal Database, System i, System i5, System p, System p5, System x, System z, System z10, System z9, z10, z9, iSeries, pSeries, xSeries, zSeries, eServer, z/VM, z/OS, i5/OS, S/390, OS/390, OS/400, AS/400, S/390 Parallel Enterprise Server, PowerVM, Power Architecture, POWER6+, POWER6, POWER5+, POWER5, POWER, OpenPower, PowerPC, BatchPipes, BladeCenter, System Storage, GPFS, HACMP, RETAIN, DB2 Connect, RACF, Redbooks, OS/2, Parallel Sysplex, MVS/ESA, AIX, Intelligent Miner, WebSphere, Netfinity, Tivoli and Informix are trademarks or registered trademarks of IBM Corporation.

Linux is the registered trademark of Linus Torvalds in the U.S. and other countries.

Adobe, the Adobe logo, Acrobat, PostScript, and Reader are either trademarks or registered trademarks of Adobe Systems Incorporated in the United States and/or other countries.

Oracle is a registered trademark of Oracle Corporation.

UNIX, X/Open, OSF/1, and Motif are registered trademarks of the Open Group.

Citrix, ICA, Program Neighborhood, MetaFrame, WinFrame, VideoFrame, and MultiWin are trademarks or registered trademarks of Citrix Systems, Inc.

HTML, XML, XHTML and W3C are trademarks or registered trademarks of W3C®, World Wide Web Consortium, Massachusetts Institute of Technology.

Java is a registered trademark of Oracle Corporation.

JavaScript is a registered trademark of Oracle Corporation, used under license for technology invented and implemented by Netscape.

SAP, R/3, SAP NetWeaver, Duet, PartnerEdge, ByDesign, SAP Business ByDesign, and other SAP products and services mentioned herein as well as their respective logos are trademarks or registered trademarks of SAP AG in Germany and other countries.

Business Objects and the Business Objects logo, BusinessObjects, Crystal Reports, Crystal Decisions, Web Intelligence, Xcelsius, and other Business Objects products and services mentioned herein as well as their respective logos are trademarks or registered trademarks of Business Objects S.A. in the United States and in other countries. Business Objects is an SAP company.

All other product and service names mentioned are the trademarks of their respective companies. Data contained in this document serves informational purposes only. National product specifications may vary.

These materials are subject to change without notice. These materials are provided by SAP AG and its affiliated companies ("SAP Group") for informational purposes only, without representation or warranty of any kind, and SAP Group shall not be liable for errors or omissions with respect to the materials. The only warranties for SAP Group products and services are those that are set forth in the express warranty statements accompanying such products and services, if any. Nothing herein should be construed as constituting an additional warranty.

master data on the fly - ms

Documents