workbench guide - sap help portal · pdf filesap data services document version: 4.2 support...

PUBLIC

SAP Data ServicesDocument Version: 4.2 Support Package 8 (14.2.8.0) – 2017-01-18

Workbench Guide

Content

1 Introduction. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51.1 Welcome to SAP Data Services. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .5

Welcome. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5Documentation set for SAP Data Services. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5Accessing documentation from the Web. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .7SAP information resources. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7

1.2 Overview of this guide. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8About this guide. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9Who should read this guide. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9

1.3 Naming Conventions. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9

2 Workbench User Interface. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 122.1 Logging into the Workbench. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 122.2 Workbench window. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12

Workspace editors. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 132.3 Menu bar. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .14

File menu. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14Edit menu. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .14Tools menu. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14Validate menu. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .15Window menu. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .15Help menu. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16Replication job editor. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .16

2.4 Toolbar. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 172.5 Project explorer. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 182.6 Data Flow editor. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .18

Data Flow editor options. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19Additional Data Flow options. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20

2.7 Datastore editor. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .212.8 File Format editor. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22

General options. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23Data File(s) options. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .25Delimiters options. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27Default Format options. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28Input/Output options. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30Custom Transfer options. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31Locale options. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32

2 P U B L I CWorkbench Guide

Content

Error Handling options. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33Fields work area. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35

2.9 Target file properties. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36General options. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36Data file options. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38Delimiters options. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39Default Format options. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40Input/Output options. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41Custom Transfer options. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .42Locale options. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43

2.10 Monitoring editor. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .432.11 Data Services preferences. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44

Data Flow detections. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45Data Flow editor. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47Data preview. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47Datastore editor. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48Expression editor. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48Expression macros. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49File Format editor . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 50Project explorer. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51

2.12 General preferences. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 522.13 Language preferences. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52

3 Database Migration. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 533.1 Migrating data and schema information. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .53

Creating a Workbench project. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 54Using the quick replication wizard . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 54Create datastores using the datastore editor. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 56Using the replication job editor. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .57Using the Data Flow editor. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .63Converting replication tables to data flows. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 63Expression macros . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 64Using the File Format editor. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .64Deploying and executing the replication job. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 65

3.2 Delta load migration. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .67Delta load jobs in the Workbench. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 67Delta load options for tables. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 68Delta load options for SAP applications. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 68

4 Using an SAP Information Steward Data Cleansing Solution. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 704.1 Viewing data cleansing solutions. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .70

Deleting a data cleansing solution. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 71

Workbench GuideContent P U B L I C 3

4.2 Configuring a data flow. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 71Using the Basic Cleanse transform . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 72

4.3 Modifying the reference data location. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 734.4 Modifying the reports substitution variables. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .734.5 Validating a data flow. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 744.6 Deploying a data flow. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 744.7 Executing a data flow. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 744.8 Viewing data. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 754.9 Modifying a Workbench job in Data Services Designer. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .75

5 Additional Information. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 765.1 Supported data migration sources and targets. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .765.2 Using the expression editor. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 77

Supported operators. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .77Include functions in expressions. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 78

5.3 Managing Workbench project information. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 88Creating a Workbench project. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 88Exporting resources and preferences. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 89Importing resources and preferences. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .89


Content

1 Introduction

1.1 Welcome to SAP Data Services

1.1.1 Welcome

SAP Data Services delivers a single enterprise-class solution for data integration, data quality, data profiling, and text data processing that allows you to integrate, transform, improve, and deliver trusted data to critical business processes. It provides one development UI, metadata repository, data connectivity layer, run-time environment, and management console—enabling IT organizations to lower total cost of ownership and accelerate time to value. With SAP Data Services, IT organizations can maximize operational efficiency with a single solution to improve data quality and gain access to heterogeneous sources and applications.

1.1.2 Documentation set for SAP Data Services

Become familiar with all the pieces of documentation that relate to your SAP Data Services product.

The latest Data Services documentation can be found on the SAP Help Portal.

Table 1:

Document What this document provides

Adapter SDK Guide Information about installing, configuring, and running the Data Services Adapter SDK .

Administrator Guide Information about administrative tasks such as monitoring, lifecycle management, security, and so on.

Configuration Guide for CTS+

System landscape information and detailed steps for CTS+ configuration.

Customer Issues Fixed Information about customer issues fixed in this release.

NoteIn some releases, this information is displayed in the Release Notes.

Designer Guide Information about how to use Data Services Designer.

Documentation Map Information about available Data Services books, languages, and locations.

Installation Guide for UNIX Information about and procedures for installing Data Services in a UNIX environment.

Installation Guide for Windows

Information about and procedures for installing Data Services in a Windows environment.

Integrator Guide Information for third-party developers to access Data Services functionality using web services and APIs.

Workbench GuideIntroduction P U B L I C 5

http://help.sap.com/bods


Management Console Guide Information about how to use Data Services Administrator and Data Services Metadata Reports.

Master Guide Information about the application, its components and scenarios for planning and designing your system landscape. Information about SAP Information Steward is also provided in this guide.

Performance Optimization Guide

Information about how to improve the performance of Data Services.

Reference Guide Detailed reference material for Data Services Designer.

Release Notes Important information you need before installing and deploying this version of Data Services.

Sizing Guide Guidelines and recommendations on hardware requirements and software considerations for your implementation.

Technical Manuals A compiled, searchable, “master” PDF of core Data Services books:

● Administrator Guide● Configuration Guide for CTS+● Designer Guide ● Reference Guide● Management Console Guide● Performance Optimization Guide● Integrator Guide● Supplement for Adapters● Supplement for Google BigQuery● Supplement for J.D. Edwards● Supplement for Oracle Applications● Supplement for PeopleSoft ● Supplement for SAP ● Supplement for Siebel● Workbench Guide

Text Data Processing Extraction Customization Guide

Information about building dictionaries and extraction rules to create your own extraction patterns to use with Text Data Processing transforms.

Text Data Processing Language Reference Guide

Information about the linguistic analysis and extraction processing features that the Text Data Processing component provides, as well as a reference section for each language supported.

Tutorial A step-by-step introduction to using Data Services.

Upgrade Guide Information to help you upgrade from previous releases of Data Services and release-specific product behavior changes from earlier versions of Data Services to the latest release.

What's New Highlights of new key features in this SAP Data Services release. This document is not updated for support package or patch releases.

Workbench Guide Provides users with information about how to use the Workbench to migrate data and database schema information between different database systems.

In addition, you may need to refer to several Supplemental Guides.

Table 2:


Supplement for Adapters Information about how to install, configure, and use Data Services adapters.


Introduction


Supplement for Google BigQuery Information about interfaces between Data Services and Google BigQuery.

Supplement for J.D. Edwards Information about interfaces between Data Services and J.D. Edwards World and J.D. Edwards OneWorld.

Supplement for Oracle Applications Information about the interface between Data Services and Oracle Applications.

Supplement for PeopleSoft Information about interfaces between Data Services and PeopleSoft.

Supplement for SAP Information about interfaces between Data Services, SAP Applications, and SAP NetWeaver BW.

Supplement for Siebel Information about the interface between Data Services and Siebel.

We also include these manuals for information about SAP BusinessObjects Information platform services.

Table 3:


Information platform services Administrator Guide Information for administrators who are responsible for configuring, managing, and maintaining an Information platform services installation.

Information platform services Installation Guide for UNIX Installation procedures for SAP BusinessObjects Information platform services on a UNIX environment.

Information platform services Installation Guide for Windows Installation procedures for SAP BusinessObjects Information platform services on a Windows environment.

1.1.3 Accessing documentation from the Web

You can access the complete documentation set for SAP Data Services from the SAP Business Users Support site.

To do this, go to http://help.sap.com/bods.

You can view the PDFs online or save them to your computer.

1.1.4 SAP information resources

A list of information resource links.

A global network of SAP technology experts provides customer support, education, and consulting to ensure maximum information management benefit to your business.

Useful addresses at a glance:



Table 4:

Address Content

Customer Support, Consulting, and Education services

http://service.sap.com/

Information about SAP Business User Support programs, as well as links to technical articles, downloads, and online discussions.

Product documentation

http://help.sap.com/bods/

SAP product documentation.

SAP Data Services tutorial

http://help.sap.com/businessobject/product_guides/sbods42/en/ds_42_tutorial_en.pdf

Introduces core features, concepts and techniques to extract, transform, and load batch data from flat-file and relational database sources for use in a data warehouse.

SAP Data Services Community Network

http://scn.sap.com/community/data-services

Get online and timely information about SAP Data Services, including forums, tips and tricks, additional downloads, samples, and much more. All content is to and from the community, so feel free to join in and contact us if you have a submission.

EIM Wiki page on SCN

http://wiki.sdn.sap.com/wiki/display/EIM/EIM+Home

The means with which to contribute content, post comments, and organize information in a hierarchical manner to so that information is easy to find.

Product Availability Matrix (PAM)

https://apps.support.sap.com/sap/support/pam

Information about supported platforms for SAP Data Services with a search function to quickly find information related to your platform.

Blueprints

https://wiki.scn.sap.com/wiki/display/EIM/SAP+Data+Services+Blueprints

Blueprints for you to download and modify to fit your needs. Each blueprint contains the necessary SAP Data Services project, jobs, data flows, file formats, sample data, template tables, and custom functions to run the data flows in your environment with only a few modifications.

1.2 Overview of this guide

Welcome to the Workbench Guide. The Data Services Workbench provides a graphical user interface (GUI) development environment in which you define data application logic to migrate data and database schema information between different databases in a data warehousing environment.

Key features of the Workbench include:

● Browsing table metadata and data.● Selecting individual source tables for migration.● Specifying the order in which the source tables should be migrated.● Adjusting the table schema in detail. For example, adding or removing columns, defining constraints,

partitions, indexes, and so on.● Specifying filters and simple projection expressions.● Specifying source and target table options such as array fetch size and bulk-loading options.● Executing a replication job as an initial load or delta load.● Modeling and consuming a data quality job.


Introduction

http://help.sap.com/disclaimer?site=http://service.sap.com/




http://help.sap.com/disclaimer?site=http://scn.sap.com/community/data-services

http://help.sap.com/disclaimer?site=http://wiki.sdn.sap.com/wiki/display/EIM/EIM+Home

http://help.sap.com/disclaimer?site=https://apps.support.sap.com/sap/support/pam

http://help.sap.com/disclaimer?site=https://wiki.scn.sap.com/wiki/display/EIM/SAP+Data+Services+Blueprints

http://help.sap.com/disclaimer?site=https://wiki.scn.sap.com/wiki/display/EIM/SAP+Data+Services+Blueprints

1.2.1 About this guide

This guide contains two kinds of information:

● Conceptual information that helps you understand the Data Services Workbench and how it works.● Procedural information that explains in a step-by-step manner how to accomplish a task.

You will find this guide most useful:

● While you are learning about the product.● While you are performing tasks in the design and early testing phases of your data-movement projects.● As a general source of information during any phase of your projects.

1.2.2 Who should read this guide

This and other Data Services product documentation assumes the following:

● You are an application developer, consultant, or database administrator working on data warehousing.● You understand your source data systems and RDMBS.● You understand your organization's data needs.● You are familiar with SQL (Structured Query Language).● You are familiar with Data Services installation environments — Microsoft Windows or UNIX.

1.3 Naming Conventions

In this documentation, the following naming conventions apply:

Terminology

● “Data Services system” refers to “SAP Data Services”.● “BI platform” refers to “SAP BusinessObjects BI platform”.

NoteThe BI platform components required by Data Services may also be provided by SAP BusinessObjects Information platform services (IPS).

● “CMC” refers to the Central Management Console provided by the BI or IPS platform.● “CMS” refers to the Central Management Server provided by the BI or IPS platform.


Variables

Table 5:

Variables Description

<INSTALL_DIR> The installation directory for the SAP software.

Default location:

● For Windows: C:\Program Files (x86)\SAP BusinessObjects● For UNIX: $HOME/sap businessobjects

<BIP_INSTALL_DIR>

The root directory of the BI or IPS platform.

Default location:

● For Windows: <INSTALL_DIR>\SAP BusinessObjects Enterprise XI 4.0● For UNIX:<INSTALL_DIR>/enterprise_xi40

NoteThese paths are the same for both the SAP BusinessObjects BI platform and SAP BusinessObjects Information platform services.

<LINK_DIR> The root directory of the Data Services system.

Default location:

● All platforms<INSTALL_DIR>/Data Services

This system environment variable is created automatically during installation.

<DS_COMMON_DIR> The common configuration directory for the Data Services system.

Default location:

● Windows (Vista and newer)ALLUSERSPROFILE\SAP BusinessObjects\Data Services

● Windows (Older versions)ALLUSERSPROFILE\Application Data\SAP BusinessObjects\Data Services

● UNIX systems (for compatibility)<LINK_DIR>

This system environment variable is created automatically during installation.

NoteStarting with Data Services 4.2 SP6, users can designate a different default location for DS_COMMON_DIR. If you cannot find the DS_COMMON_DIR in the listed default location above, ask your System Administrator to find out where the default location is for DS_COMMON_DIR.


Introduction

Variables Description

<DS_USER_DIR> The user-specific configuration directory for the Data Services system.

Default location:

● Windows (Vista and newer)USERPROFILE\AppData\Local\SAP BusinessObjects\Data Services

● Windows (Older versions)USERPROFILE\Local Settings\Application Data\SAP BusinessObjects\Data Services

This user environment variable is created automatically during installation.

NoteThis variable is used only for Data Services client applications on Windows, such as the Designer. <DS_USER_DIR> is not used on UNIX platforms.


2 Workbench User Interface

This section provides basic information about the Workbench user interface.

2.1 Logging into the Workbench

Before you can use Workbench to create, edit, and execute replication jobs, you must log in as a user who is defined in the Central Management Server (CMS).

1. Launch the Workbench.

The Workbench can be accessed from the Windows Start Menu: All Programs SAP Data Services <x.x>Data Services Workbench .

2. Enter your user credentials for the CMS.

Option Description

System Server name (and optionally the port number) for the CMS.

User name User name for logging into the CMS.

Password User password for logging into the CMS.

Authentication mode Authentication type used by the CMS.

3. Click Logon.The software attempts to connect to the CMS using the specified information. If the authentication is successful, the Workbench window opens.

2.2 Workbench window

By default, the Workbench user interface consists of several primary elements.

The Workbench user interface contains a menu bar, toolbar, and a project explorer. The Workbench project explorer contains the current project, and all jobs and datastores that are a part of it.

The workspace is the area of the Workbench window where you define, display, and modify objects. When you create a new object, the Workbench automatically opens a new editor tab in the workspace. The workspace can also display the data of a source table and the job status and execution logs.

Related Information


Workbench User Interface

Replication job editor [page 16]Datastore editor [page 21]Data Flow editor [page 18]Monitoring editor [page 43]File Format editor [page 22]

2.2.1 Workspace editors

The workspace area of the Workbench contains tabs for the editors you currently have open.

The following table lists the editors that may appear in the workspace.

Table 6:

Editor Description

Replication job Display and edit replication job options such as:

● Target table schema● Projections, expressions, and filters● Replication groups

The name of the open replication job is indicated in the tab's title. If the replication job contains unsaved data, it is indicated with an asterisk (*).

Data Flow Create data flows and include them in replication jobs. The supported tranforms are:

● Query● Case● Merge● Row Generation● MAP Operation● SQL

Reference existing data flows from the replication job by dragging and dropping the data flows from the Explorer. Additionally, verify all data flows and specify whether the data flow is for initial load, delta load, or both.

Datastore Display and edit datastore properties, browse and import metadata, and display and edit table properties. The name of the open datastore is indicated in the tab's title. If the datastore contains unsaved data, it is indicated with an asterisk (*).

File Format Set properties for file format templates and source and target file formats.

Monitoring Display your replication jobs' status, execution progress, and any applicable logs. Trace, monitor, and error logs are available.

Workbench GuideWorkbench User Interface P U B L I C 13

2.3 Menu bar

This section contains a brief description of the Workbench menus.

2.3.1 File menu

The File menu contains standard Windows as well as software-specific options.

Table 7:

Option Description

New Define a new folder, project, datastore, data flow, or replication job.

Open File Open an existing Workbench file.

Close Close the active Workbench editor.

Close All Close all open editors.

Save Save the object in the active editor.

Save All Save all objects in open editors.

Exit Exit the Workbench.

2.3.2 Edit menu

The Edit menu contains standard Windows options.

Table 8:

Option Description

Undo Undo the last operation.

Redo Redo the last undone operation.

Cut Cut the selected objects or text and place it on the clipboard.

Copy Copy the selected objects or text to the clipboard.

Paste Paste the contents of the clipboard into the active editor or text box.

Delete Delete the selected objects or text.

Select All Select all objects in the active editor or all text in the active text box.

2.3.3 Tools menu

The Tools menu contains replication job editor and datastore editor options.



Table 9:

Option Description

Open monitoring editor Open the job monitoring editor.

Quick replication wizard Start the quick replication wizard.

Export to ATL Export a selected replication job to the local file system.

Deploy Deploy a replication job or datastore definition to a Data Services repository.

Execute Deploy a replication job to a Data Services repository and execute the job.

NoteThis option is available only for the replication job editor.

2.3.4 Validate menu

The Validate menu contains replication job editor and datastore editor validation options.

Table 10:

Option Description

Validate Validate the active Workbench replication job for errors.

NoteThe Validate option is available only for the replication job editor.

Show DDL Display the data definition language (DDL) for the active Workbench object.

NoteThe Show DDL option is available only for the replication job editor and only applies when the replication job editor is active.

Show ATL Display the ATL for the active Workbench object.

Note

Do not use this option to create a file for import into Data Services. Use the Tools

Export Replication Job option to export a replication job.

2.3.5 Window menu

The Window menu provides software-specific window options.


Table 11:

Option Description

Data Cleansing Solutions Display the Data Cleansing Solutions tab in the workspace area.

Error Log Display the Error Log tab in the workspace area.

Problems Display the Problems tab in the workspace area.

Project Explorer Display the project explorer.

Reference Analysis Display a reference impact analysis for the active Workbench object in the workspace area.

Other Display other window panes.

Reset Perspective Reset the Workbench interface perspective to its default settings.

Preferences Display the Preferences window.

2.3.6 Help menu

The Help menu provides standard help options.

Table 12:

Option Description

Welcome Display the Workbench welcome screen.

Help Display the SAP Data Services Workbench Guide.

About Display information about the software including the version of the Workbench and copyright information.

Key Assist Display information about keyboard shortcuts available in the Workbench.

2.3.7 Replication job editor

The Workbench replication job editor opens in the workspace when you create a new replication job or open an existing replication job from the project explorer.

The replication job editor consists of several areas:

● Project explorerDisplays the name of the source datastore and all imported tables.

● Target datastore paneDisplays the name of the target datastore and all mapped tables and columns. In this pane, mapped objects may be grouped into replication groups.

● Replication job configuration paneWhen you select an object such as a datastore table or column in the replication job editor, its configuration options are displayed in this pane.The option groups that are available depend on the type of object that is selected:



Table 13:

Object Available options

Project explorer Properties

Replication job Properties

Variables

Scripts

Replication group Properties

Target table Properties

Columns

Foreign Keys

Indexes

Options

Filter data

DDL Options

Delta load

Column Properties

You can adjust the size of each pane by dragging the borders between them. In addition, you can hide or reveal the replication job configuration pane by clicking the arrows in the center of its border.

Related Information

Using the replication job editor [page 57]

2.4 Toolbar

In addition to many of the standard Windows tools, the software provides application specific tools, including:

Table 14:

Icon Tool Description

New Define a new folder, project, datastore, or replication job.

Save Save the object in the active editor.

Cut Cut the selected objects or text and place it on the clipboard.

Copy Copy the selected objects or text to the clipboard.


Icon Tool Description

Paste Paste the contents of the clipboard into the active editor or text box.

Delete Delete the selected objects or text.

Open monitoring editor Open the job monitoring editor.

Validate Validate the active Workbench resource for errors.

Show DDL Display the data definition language (DDL) for the active Workbench resource.

Deploy ATL Deploy the ATL for the active replication job to the Data Services repository.

Execute Deploy the ATL for the active replication job to the Data Services repository, and execute the replication job.

Quick replication wizard Start the quick replication wizard to easily define source and target datastores and create a basic replication job.

2.5 Project explorer

The project explorer contains the projects currently defined in the user workspace. Each Workbench project is represented as a folder containing all datastores and replication jobs associated with that project.

When you create new projects, datastores, data flows, file formats, and replication jobs, they are displayed in the project explorer. You can also use the project explorer to import and export Workbench project information.

2.6 Data Flow editor

The Workbench Data Flow editor enables you to create data flows and include them in the replication job. The supported tranforms are:

● Query● Case● Merge● Row Generation● MAP Operation● SQL

You can reference existing data flows from the replication job by dragging and dropping the data flows from the Explorer. You can verify all data flows and specify whether the data flow is for initial load, delta load, or both.

You can also convert replication tables to data flows by highlighting multiple replication tables, right clicking the selection, and selecting Convert Data. You have the option to generate a single data flow or one data flow per replicated table.



2.6.1 Data Flow editor options

The Workbench Data Flow editor includes the Data Flow diagram, a list of input schemas for the selected transform, a list of available transforms, and a tabbed area containing the properties of the object currently selected in the Data Flow diagram.

Table 15:

Object Available options

Data Flow diagram Composed of a graphical area and a palette of elements you can drop on the graphical area.

Data Flow diagram toolbar Includes the following display options:

● Auto arrange● Show transform metadata● Save as image● Toggle snap to grid on/off● Toggle highlighting of related elements● Toggle lineage display of selected field● Toggle descriptions on/off● Connection style● Fit to window● Reset zoom● Zoom slider

Transforms Displays all available transforms, which you can drag and drop onto the Data Flow diagram.

Input Schemas panel Displays all the available input schemas/fields that the user can drag and drop to the properties area. Every field that is being used in the current property tab is marked with a black triangle. Some fields are marked with a key icon indicating they are part of the primary key.

Object properties Displays the properties of the object that is currently selected in the Data Flow diagram.

The following options contain the same properties as in the Data Services Designer:

● Table Reader properties● File Reader● Table Loader● Merge● Case● SQL● Row Generation● Map Operation● Date Generation● Effective Date

The Template Table Loader uses the same database specific loader options as Data Services Designer as well as some extra options:

● Use NVARCHAR for VARCHAR● Drop and re-create table● Double quote names● Table Type (for HANA only)


Use data file as source or target

You can select and drop data from the Project Explorer onto the Data Flow diagram and choose to use it as a source or target.

Validation in the Data Flow editor

Objects containing errors are underlined in red and a tooltip shows the list of errors/warnings for an element.

2.6.2 Additional Data Flow options

The Workbench Data flow editor includes several options that are not available in the Designer Data Flow editor.

The following options are only available in the Workbench Data Flow editor.

Table 16:

Options Description

Template file loader Located in the Transforms panel, the Template File loader enables you to add data to a file without having to create a file format. Drop the Template file loader onto the Data Flow diagram and set the template file properties. You must enter the fully qualified file name(s).

Detect Mappings Located on the Mappings tab, the Detect mappings option assists you during target to source mapping by detecting source fields with similar names. The dialog provides a list of proposed candidate mappings for you to view and confirm. Every proposed candidate field has a matching score (from 0 to 100), and proposals are displayed from the highest score to the lowest to allow you to see the most relevant proposals first. Only the proposals that are confirmed are applied.

Detect Joins Join detection combines both foreign key relationships analysis (if the source has foreign keys) and name, data type matching. Every join that is detected is assigned a detection score and shown in a confirmation dialog. A join found using foreign key detection will always have a score of 100, and a detection using name will have a score between 0 and 99 depending on the similarity of the fields that are proposed as a join.



Options Description

Create Query transform The Data Flow feature allows you to right-click and select to add a new query transform to the data flow. Workbench will create the query transform object, connect it to both selected sources, connect it to the loader, run the detect joins, and detect mapping.

The Data Flow detections page under Preference gives you the flexibility to be more restrictive for the minimum score and allows you to choose whether to always confirm the detections or not.

Related Information

Data Flow detections [page 45]

2.7 Datastore editor

The Workbench datastore editor opens in the workspace when you create a new datastore or open an existing datastore from the project explorer or replication job editor.

The datastore editor consists of these areas:

● The navigation pane at left contains the Configurations, Tables, and Views tabs. If applicable, the number of those objects contained in the datastore is indicated in parentheses. For SAP datastores, the navigation pane also displays the imported SAP extractors.When the datastore configuration is active, you can edit the datastore properties and settings. When the datastore tables, extractors, or views are active, you can import or re-import tables, extractors, and views, view data, and compare imported metadata with the metadata in the database.

● The object's properties pane displays tabs for the properties and options for the object currently selected in the navigation pane. The option tabs that are available depend on the type of object that is selected:

Table 17:

Object Available option tabs

Tables Properties

Columns

Foreign Keys

Indexes

Partitions

Attributes


Object Available option tabs

Column Properties

Attributes

You can adjust the size of each pane by dragging the borders between them. In addition, you can hide or reveal the navigation pane by clicking the arrow in the center of its border.

Related Information

Creating a datastore [page 56]

2.8 File Format editor

A file format describes the structure of an ASCII file. A file format consists of multiple properties. You set the properties in the File Format editor.

The work area on the left side of the File Format editor lists file format properties that are not field specific. The tables in this section list all of the options grouped by type. These options are filtered by the mode you are using.

Related Information

Using the File Format editor [page 64]General options [page 23]Data File(s) options [page 25]Delimiters options [page 27]Default Format options [page 28]Input/Output options [page 30]Custom Transfer options [page 31]Locale options [page 32]Error Handling options [page 33]Fields work area [page 35]



2.8.1 General options

The following table contains the General options in the File Format editor.

Table 18:

Option Possible values Description Mode

File format Delimited The format of the data in the text file. Available properties change based on the selected file format type.

New, Edit

Adaptable file reading Yes, No Indicates whether the schema of a delimited file format is adaptable or fixed.

● Yes indicates that the schema is adaptable. The actual file can contain fewer or more columns than indicated by the file format. If a row contains fewer columns than expected, the software loads null values into the columns missing data. If a row contains more columns than expected, the software ignores the additional data.

● No indicates that the schema is fixed. The software requires the number of columns in each row to match the number of columns specified in the file format. The default is No.

If you select Yes, you must ensure that the selected column and row delimiters do not appear inside the actual data.

New, Edit, Source

Custom transfer Yes, No Enables the software to use a third-party file transfer program. Displays additional Custom transfer program options in the File Format Editor below the Input/Output properties.

All

Skip error handling Yes, No Selecting Yes disables the Error Handling section in the Format editor. The default is No.

New, Edit, Source



Number of processing threads

Integer greater than 0, {none}, {default}

Specifies the number of threads for parallel processing. Parallel processing can improve performance by maximizing CPU usage on the Job Server computer. For example, if you have four CPUs, enter 4 for this option. For more information, see “File multi-threading” in the Performance Optimization Guide.

For jobs that process USPS certification tests, the value should be set to {none}.

All

Related Information

Using the File Format editor [page 64]Data File(s) options [page 25]Delimiters options [page 27]Default Format options [page 28]Input/Output options [page 30]Custom Transfer options [page 31]Locale options [page 32]Error Handling options [page 33]Fields work area [page 35]File Format editor [page 22]



2.8.2 Data File(s) options

The following table contains the Data File(s) options in the File Format editor.

Table 19:


File location Local, Job Server During design, indicates the files are located on the local machine or on the machine that runs the Job Server. If you select Job Server, you must enter the absolute path to files. Remember that UNIX systems are case-sensitive.

During execution, all files must be located on the Job Server machine that executes the job. If you use different files to design your job, change the file specified (through the Root directory and File properties) before you execute the job.

All

Root folder Path name for the file or blank The directory where the file is located.

For added flexibility, you can enter a variable for this option.

If you enter a directory name, then enter only the file name for the File property.

If you leave the root directory blank, then enter a file name that includes the full path name in the File property.

New, Edit



File name(s) File name(s), file name(s) including full path name, or blank

In new and edit modes, specifies an existing file on which you base the file format description. Data from this file appears in the Column Attributes area. In these modes, you can leave this property blank.

In source and target modes, specify the location of the actual file for this source or target. In these modes, you cannot leave this property blank. For added flexibility, you can enter:

● A variable that is set to a particular file with full path name. Use variables to specify file names that you cannot otherwise enter, such as file names that contain multi-byte characters.

● A list of files, separated by commas, or a file name containing a wildcard. In this case, the software reads all these files as a single source.See “Reading multiple files at one time” in the Designer Guide.

All

Related Information

Using the File Format editor [page 64]General options [page 23]Delimiters options [page 27]Default Format options [page 28]Input/Output options [page 30]Custom Transfer options [page 31]Locale options [page 32]Error Handling options [page 33]Fields work area [page 35]File Format editor [page 22]



2.8.3 Delimiters options

The following table contains the Delimiters options in the File Format editor.

Table 20:


Column delimiter Tab, Semicolon, Comma, Space, or any character sequence

For delimited file formats, the character sequence that indicates the end of one column and the beginning of the next.

New, Edit

Row delimiter {new line}, {Windows new line}, {Unix new line}, or any character sequence

A character sequence that indicates where one row of data ends and the next begins.

New, Edit

Row delimiter within text string

Character, Row delimiter Defines how the row delimiter is interpreted within a text string.

● Character: The specified row delimiter is treated as a character within the text string.

● Row delimiter: The specified row delimiter is interpreted and defines rows within the text string.

New, Edit

Text delimiter Single quotation marks ('), double quotation marks ("), or {none}

Denotes the start and end of a text string. All characters (including those specified as column delimiters) between the first and second occurrence of this character is a single text string. The treatment of the row characters is defined by the Row within text string setting.

New, Edit

NoteData in columns cannot include the column or row delimiter, unless you also specify a text delimiter. For example, if you specify a comma as your column delimiter, none of the data in the file can contain commas. However, if you specify a comma as the column delimiter and a single quote as the text delimiter, commas are allowed in strings in the data.

You can use any ASCII characters (including non-printing characters) for column and row delimiters.

You can specify an ASCII character by entering a forward slash (/) followed by the decimal representation of the character. For example, to use Y umlaut (Ÿ) as a delimiter, enter /255 in the delimiter property box.


Related Information

Using the File Format editor [page 64]General options [page 23]Data File(s) options [page 25]Default Format options [page 28]Input/Output options [page 30]Custom Transfer options [page 31]Locale options [page 32]Error Handling options [page 33]Fields work area [page 35]File Format editor [page 22]

2.8.4 Default Format options

The following table contains the Default Format options in the File Format editor.

Table 21:


Escape character Any character sequence, or {none}

A special character sequence that causes the software to ignore the normal column delimiter. Characters following the escape character sequence are never used as column delimiters.

For example, suppose you specify a forward slash as the escape character and a comma as the column delimiter. Then, you must have a forward slash to have a comma appear inside a field.

New, Edit

NULL indicator {none} or any other character sequence

A special character sequence that the software interprets as NULL data.

The software ignores any NULL indicator specified in the file format for blob columns.

New, Edit




Ignore row marker(s) {none} or any other character sequence

A character sequence, which when found at the beginning of rows, causes the software to ignore the row when reading the file or automatically creating metadata. To enter multiple character sequences, separate each with a semi-colon. To include a semi-colon or backslash as a marking character, precede with a backslash.

New, Edit, Source

Date format yyyy.mm.dd

or other combinations

The date format for reading or writing date values to and from the file.

New, Edit

Time format hh24:mi:ss


The time format for reading or writing time values to and from the file.

New, Edit

Date-Time format yyyy.mm.dd hh24:mi:ss


The datetime format for reading or writing datetime values to and from the file.

New, Edit

Related Information

Using the File Format editor [page 64]General options [page 23]Data File(s) options [page 25]Delimiters options [page 27]Input/Output options [page 30]Custom Transfer options [page 31]Locale options [page 32]Error Handling options [page 33]Fields work area [page 35]File Format editor [page 22]


2.8.5 Input/Output options

The following table contains the Input/Output options in the File Format editor.

Table 22:


Style Headers or BOF/EOF The format of the start and end of the file. Available properties in the Input/Output property group may change, based on this selection.

New, Edit

Skipped rows Integer For file formats using Headers style, the number of rows skipped when reading the file. Specify a non-zero value when the file includes comments or other non-data information.

New, Edit

Skip row header Yes, No For file formats using the Headers style, indicates whether the first row of data in the file contains the column names and should be skipped when reading the file. The software uses this property in addition to the Skipped rows property.

When you select Yes, the software does not read data from the first row, and uses data in the first row to determine the file's column names.

New, Edit

Write row header Yes, No For file formats using Headers style, indicates whether to write column names in the first row of the output file.

New, Edit

Write BOM Yes, No For file formats using UTF-8 and UTF-16, determines the writing of BOM characters into the file. Choose Yes if you want to include BOM characters into a UTF-8 or UTF-16 file in which byte order is not otherwise defined. For a UTF-16 file, the software assumes the file to be UTF-16be, unless BOM characters are added by this property.

New, Edit




Beginning string in a file Any character sequence, including a blank space, an empty string, or {none}

For file formats using BOF/EOF style, the string that marks the start of data in the file.

New, Edit

Ending string in a file Any character sequence, including a blank space, an empty string, or {none}

For file formats using BOF/EOF style, the string that marks the end of data in the file.

New, Edit

Related Information

Using the File Format editor [page 64]General options [page 23]Data File(s) options [page 25]Delimiters options [page 27]Default Format options [page 28]Custom Transfer options [page 31]Locale options [page 32]Error Handling options [page 33]Fields work area [page 35]File Format editor [page 22]

2.8.6 Custom Transfer options

The following table contains the Custom Transfer options in the File Format editor.

Table 23:


Program executable File name (Required) The name of the custom transfer program or its initialization script. For example: MyProgram.exe or MyProgram.cmd.

New, Edit

User name Any character sequence, including a blank space, an empty string, or {none}

(Optional) Log in ID for the server to which the custom transfer program connects. You may want to allow a custom program user to enter their user name when they enter their password in the software.

New, Edit



Password Any character sequence, including a blank space, an empty string, or {none}

(optional) Password for the server to which the custom transfer program connects. Passwords entered into this option are encrypted.

New, Edit

Arguments Any character sequence, including a blank space, an empty string, or {none)

(Optional) You can create arguments in your custom transfer program and then specifically flag them from within the software using this box. For example, you might have security or compression mechanisms in your program. You can also link connection data to your transfer program's flags.

New, Edit

Related Information

Using the File Format editor [page 64]General options [page 23]Data File(s) options [page 25]Delimiters options [page 27]Default Format options [page 28]Input/Output options [page 30]Locale options [page 32]Error Handling options [page 33]Fields work area [page 35]

2.8.7 Locale options

The following table contains the Locale options in the File Format editor.

Table 24:


Language The three-letter language abbreviations specified in the ISO 639-2/T standard

Specifies the human language (for example, Korean, Japanese, or English) in which data is stored or processed. Select from the displayed list.

New, Edit




Code Page The list of supported code pages. See Supported Locales and Encodings.

Specifies the sequence of bits that defines a character. For example, the Japanese code page contains ASCII, Greek, Cyrillic, and Japanese characters, thereby supporting the English, Greek, Russian, and Japanese languages.

New, Edit

Related Information

Using the File Format editor [page 64]General options [page 23]Data File(s) options [page 25]Delimiters options [page 27]Default Format options [page 28]Input/Output options [page 30]Custom Transfer options [page 31]Error Handling options [page 33]Fields work area [page 35]File Format editor [page 22]

2.8.8 Error Handling options

The following table contains the Error Handling options in the File Format editor

Table 25:


Log data conversion warnings Yes, No Determines whether to include data-type conversion warnings in the error log. The default is Yes.

New, Edit, Source

Log row format warnings Yes, No Determines whether to include row-format warnings in the error log. The default is Yes.

New, Edit, Source

Maximum warnings to log Integer greater than 0 or {no limit}

If Log data conversion warnings or Log row format warnings is enabled, you can limit how many warnings the software logs. The default is {no limit}.

New, Edit, Source



Capture data conversion errors

Yes, No Determines whether to capture data-type conversion errors when processing a flat-file source. The default is No.

New, Edit, Source

Capture row format errors Yes, No Determines whether to capture row-format errors when processing a flat-file source. The default is Yes.

New, Edit, Source

Accepted bad records in file Integer greater than 0 or {no limit}

Specify the number of bad records in a file that will be accepted.

Write error rows to file Yes, No Determines whether to write invalid rows to an error file. The default is No.

New, Edit, Source

Error file root directory Directory path or blank If Write error rows to file is enabled, type the root directory in which to save the error file. For added flexibility, you can enter a variable for this option.

If you type a directory path here, then only enter the file name in the Error file name property.

If you leave Error file root directory blank, then type a full path and file name in the Error file name property.

New, Edit, Source

Error file name File name, file name including full path name, or blank

If Write error rows to file is enabled, type the name of the file in which to record the invalid rows.

For added flexibility, you can enter a variable that is set to a particular file with full path name. Use variables to specify file names that you cannot otherwise enter such as file names that contain multi-byte characters.

New, Edit, Source

Related Information

Using the File Format editor [page 64]



General options [page 23]Data File(s) options [page 25]Delimiters options [page 27]Default Format options [page 28]Input/Output options [page 30]Custom Transfer options [page 31]Locale options [page 32]Fields work area [page 35]File Format editor [page 22]

2.8.9 Fields work area

The Fields work area in the File Format editor contains properties about the fields in the file format.

Table 26:

Property Possible values Description

Field name Any sequence of letters or numbers, not including blank spaces

A name that identifies data in this column. If your file format uses the Headers style and you select Yes for the Write row header property, the software writes the field names in the target file.

Data type blob, date, datetime, decimal, double, integer, interval, long, numeric, real, time, varchar

The data type of values in this column. The long data type is not available in fixed-width formats.

Format {none} For all data types other than varchar, specifies the format for this particular field. You can use this property to overwrite the default format. For example, if one date field is different than others, you can specify the different format here.

Nullable Yes, No Indicates whether the the field will accept NULL values.

Description Any sequence of letters or numbers, not including blank spaces

Text that that specifies the type of data in a column. Typically use the field name, or a name similar to the field name. For example, if your field name is LastName, you may want to name the content type Family_Name.

If you delete a file format template from the object library, you must also delete all file sources and targets that are based on that file format template.


Related Information

Using the File Format editor [page 64]General options [page 23]Data File(s) options [page 25]Delimiters options [page 27]Default Format options [page 28]Input/Output options [page 30]Custom Transfer options [page 31]Locale options [page 32]Error Handling options [page 33]File Format editor [page 22]

2.9 Target file properties

The tables in this section list all options that are available on the Target file properties tab in the Replication Job editor. The tables are grouped by type.

Related Information

General options [page 36]Data file options [page 38]Delimiters options [page 39]Default Format options [page 40]Input/Output options [page 41]Custom Transfer options [page 42]Locale options [page 43]

2.9.1 General options

The following General options are available on the Target file properties tab in the Replication Job editor.

Table 27:

Option Possible values Description

File format Delimited The format of the data in the text file. Available properties change based on the selected file format type.




Adaptable Schema Yes, No Indicates whether the schema of a delimited file format is adaptable or fixed.

● Yes indicates that the schema is adaptable. The actual file can contain fewer or more columns than indicated by the file format. If a row contains fewer columns than expected, the software loads null values into the columns missing data. If a row contains more columns than expected, the software ignores the additional data.

● No indicates that the schema is fixed. The software requires the number of columns in each row to match the number of columns specified in the file format.The default is No. If you select Yes, you must ensure that the selected column and row delimiters do not appear inside the actual data.

Custom transfer program Yes, No Enables the software to use a third-party file transfer program. Displays additional Custom transfer program options in the File Format Editor below the Input/Output properties.

Parallel process threads Integer greater than 0, {none}, {default} Specifies the number of threads for parallel processing. Parallel processing can improve performance by maximizing CPU usage on the Job Server computer. For example, if you have four CPUs, enter 4 for this option. For more information, see “File multi-threading” in the Performance Optimization Guide.

For jobs that process USPS certification tests, the value should be set to {none}.

Related Information

Target file properties [page 36]Data file options [page 38]Delimiters options [page 39]Default Format options [page 40]Input/Output options [page 41]Custom Transfer options [page 42]Locale options [page 43]


2.9.2 Data file options

The following Data file options are available on the Target file properties tab in the Replication Job editor.

Table 28:


Location Local, Job Server During design, indicates the files are located on the local machine or on the machine that runs the Job Server. If you select Job Server, you must enter the absolute path to files. Remember that UNIX systems are case-sensitive.

During execution, all files must be located on the Job Server machine that executes the job. If you use different files to design your job, change the file specified (through the Root directory and File properties) before you execute the job.

Root directory Path name for the file The parent directory for the specified file name. If the directory name is entered, the file will be generated there and the file name format will be: <jobname_tablename>.<extension>. If the directory is blank, the file will be generated under <DS_COMMON_DIR>/workspace/.

File Extension File extension The file extension to be applied to all loader files in the replication job.

Related Information

Target file properties [page 36]General options [page 36]Delimiters options [page 39]Default Format options [page 40]Input/Output options [page 41]Custom Transfer options [page 42]Locale options [page 43]



2.9.3 Delimiters options

The following Delimiters options are available on the Target file properties tab in the Replication Job editor.

Table 29:


Column Tab, Semicolon, Comma, Space, or any character sequence

For delimited file formats, the character sequence that indicates the end of one column and the beginning of the next.

Row {new line}, {Windows new line}, {Unix new line}, or any character sequence

A character sequence that indicates where one row of data ends and the next begins.

Row within text string Character, Row delimiter Defines how the row delimiter is interpreted within a text string.

● Character: The specified row delimiter is treated as a character within the text string.

● Row delimiter: The specified row delimiter is interpreted and defines rows within the text string.

Text Single quotation marks ('), double quotation marks ("), or {none}

Denotes the start and end of a text string. All characters (including those specified as column delimiters) between the first and second occurrence of this character is a single text string. The treatment of the row characters is defined by the Row within text string setting.

NoteData in columns cannot include the column or row delimiter, unless you also specify a text delimiter. For example, if you specify a comma as your column delimiter, none of the data in the file can contain commas. However, if you specify a comma as the column delimiter and a single quote as the text delimiter, commas are allowed in strings in the data.

You can use any ASCII characters (including non-printing characters) for column and row delimiters.

You can specify an ASCII character by entering a forward slash (/) followed by the decimal representation of the character. For example, to use Y umlaut (Ÿ) as a delimiter, enter /255 in the delimiter property box.

Related Information

Target file properties [page 36]General options [page 36]Data file options [page 38]Default Format options [page 40]


Input/Output options [page 41]Custom Transfer options [page 42]Locale options [page 43]

2.9.4 Default Format options

The following Default Format options are available on the Target file properties tab in the Replication Job editor.

Table 30:


Escape character Any character sequence, or {none} A special character sequence that causes the software to ignore the normal column delimiter. Characters following the escape character sequence are never used as column delimiters.

For example, suppose you specify a forward slash as the escape character and a comma as the column delimiter. Then, you must have a forward slash to have a comma appear inside a field.

NULL indicator {none} or any other character sequence A special character sequence that the software interprets as NULL data.

The software ignores any NULL indicator specified in the file format for blob columns.

Ignore row marker(s) {none} or any other character sequence A character sequence, which when found at the beginning of rows, causes the software to ignore the row when reading the file or automatically creating metadata. To enter multiple character sequences, separate each with a semi-colon. To include a semi-colon or backslash as a marking character, precede with a backslash.

Date format yyyy.mm.dd


The date format for reading or writing date values to and from the file.

Time format hh24:mi:ss


The time format for reading or writing time values to and from the file.

Date-Time format yyyy.mm.dd hh24:mi:ss


The datetime format for reading or writing datetime values to and from the file.

Validate decimal data Yes, No Perform decimal validation and formatting on output.



Related Information

Target file properties [page 36]General options [page 36]Data file options [page 38]Delimiters options [page 39]Input/Output options [page 41]Custom Transfer options [page 42]Locale options [page 43]

2.9.5 Input/Output options

The following Input/Output options are available on the Target file properties tab in the Replication Job editor.

Table 31:


Style Headers or BOF/EOF The format of the start and end of the file. Available properties in the Input/Output property group may change, based on this selection.

Skipped rows Integer For file formats using Headers style, the number of rows skipped when reading the file. Specify a non-zero value when the file includes comments or other non-data information.

Write row header Yes, No For file formats using Headers style, indicates whether to write column names in the first row of the output file.

Write BOM Yes, No For file formats using UTF-8 and UTF-16, determines the writing of BOM characters into the file. Choose Yes if you want to include BOM characters into a UTF-8 or UTF-16 file in which byte order is not otherwise defined. For a UTF-16 file, the software assumes the file to be UTF-16be, unless BOM characters are added by this property.

BOF marker Any character sequence, including a blank space, an empty string, or {none}

For file formats using BOF/EOF style, the string that marks the start of data in the file.

EOF marker Any character sequence, including a blank space, an empty string, or {none}

For file formats using BOF/EOF style, the string that marks the end of data in the file.


Related Information

Target file properties [page 36]General options [page 36]Data file options [page 38]Delimiters options [page 39]Default Format options [page 40]Custom Transfer options [page 42]Locale options [page 43]

2.9.6 Custom Transfer options

The following Custom Transfer options are available on the Target file properties tab in the Replication Job editor.

Table 32:


Custom transfer program executable File name (Required) The name of the custom transfer program or its initialization script. For example: MyProgram.exe or MyProgram.cmd.

Custom transfer user name Any character sequence, including a blank space, an empty string, or {none}

(Optional) Log in ID for the server to which the custom transfer program connects. You may want to allow a custom program user to enter their user name when they enter their password in the software.

Custom transfer password Any character sequence, including a blank space, an empty string, or {none}

(optional) Password for the server to which the custom transfer program connects. Passwords entered into this option are encrypted.

Custom transfer arguments Any character sequence, including a blank space, an empty string, or {none)

(Optional) You can create arguments in your custom transfer program and then specifically flag them from within the software using this box. For example, you might have security or compression mechanisms in your program. You can also link connection data to your transfer program's flags.

Related Information

Target file properties [page 36]General options [page 36]Data file options [page 38]



Delimiters options [page 39]Default Format options [page 40]Input/Output options [page 41]Locale options [page 43]

2.9.7 Locale options

The following Locale options are available on the Target file properties tab in the Replication Job editor.

Table 33:


Language The three-letter language abbreviations specified in the ISO 639-2/T standard

Specifies the human language (for example, Korean, Japanese, or English) in which data is stored or processed. Select from the displayed list.

Code Page The list of supported code pages. See Supported Locales and Encodings.

Specifies the sequence of bits that defines a character. For example, the Japanese code page contains ASCII, Greek, Cyrillic, and Japanese characters, thereby supporting the English, Greek, Russian, and Japanese languages.

Related Information

Target file properties [page 36]General options [page 36]Data file options [page 38]Delimiters options [page 39]Default Format options [page 40]Input/Output options [page 41]Custom Transfer options [page 42]

2.10 Monitoring editor

The Workbench Monitoring editor opens in the workspace when you click the icon in the toolbar or select it from the Tools menu.

The monitoring editor consists of several areas:


Table 34:

Area Description

Job selection toolbar Displays the Data Services jobs available in the repository to which the Workbench is connected and allows you to execute them.

NoteThe monitoring editor displays all standard Data Services jobs in the repository, not Workbench-specific replication jobs.

Execution history pane Displays the execution history for the selected job. Each line indicates the current status, execution date, and duration of a single job execution instance.

Execution history dashboard pane Displays a graphical representation of the history for the selected job. You can change the type of chart by choosing from the drop-down box in the upper-right corner of the pane.

Execution details pane Displays the details for the individual job execution instance selected in the execution history pane. You can view the trace, monitor, and error logs for the execution instance, as well as the execution statistics.

You can adjust the size of each pane by dragging the borders between them. In addition, you can hide or reveal the dashboard and details panes by clicking the arrows in the center of their borders.

2.11 Data Services preferences

The Data Services Preferences options allow you to customize several areas of Data Services Workbench.

Open Data Services preferences by clicking Window Preferences... and expanding the Data Services node.

From the main Data Services page, specify a number in the Max errors to show in tooltips option.

Other options under the Data Services node in Preferences include:

● Data Flow detections● Data Flow editor● Data preview● Datastore editor● Expression editor● Expression macros● File Format editor● Project explorer

Related Information

Data Flow detections [page 45]



Data Flow editor [page 47]Data preview [page 47]Datastore editor [page 48]Expression editor [page 48]Expression macros [page 49]File Format editor [page 50]Project explorer [page 51]

2.11.1 Data Flow detections

The Data Flow detections options allow you to customize the way detection algorithms work. For example, you can set the minimum scores to display in the Mappings detection and Joins detection groups.

To open, click Window Preferences... , and expand Data Services Data Flow detections .

Related Information

Mappings detection options [page 45]Joins detection [page 46]

2.11.1.1 Mappings detection options

The following table describes the Mappings detection group of options in the Data Flow detections preferences dialog.

Table 35:

Option Description

Minimum score threshold Select the minimum mapping score to display. Scores at or above this setting display in the Mapping detection dialog.

Enable mapping detection in create Query context menu Check to enable mapping detection when creating a query transform by the following methods:

● Select multiple source elements from a dataflow and right-click to open the context menu

● Select a loader and right-click to open the context menu

Minimum score threshold Select the minimum mapping score to display. Scores at or above this setting display in the Mapping detection dialog. This option applies only to the Enable Mapping Detection option in the create query context menu.


Option Description

Always confirm mappings Select to always confirm mappings when creating a new Query transform. This option applies to the Enable Mapping Detection option in the create query context menu.

Related Information

Data Flow detections [page 45]Joins detection [page 46]

2.11.1.2 Joins detection

The following table describes the Data Flow Mappings detection options in the Data Flow detections preferences dialog.

Table 36:

Option Description

Minimum score threshold Select the minimum join score to display. Scores at or above this setting display in the join detection dialog.

Enable join detection in create Query context menu Check to enable join detection when creating a query transform by the following methods:

● Select multiple source elements from a dataflow and right-click to open the context menu

● Select a loader and right-click to open the context menu

Minimum score threshold Select the minimum mapping score to display. Scores at or above this setting display in the Mapping detection dialog.

Always confirm joins Select to always confirm joins when creating a new query transform. This option applies to the Enable Join detection option in the create query context menu.

Related Information

Data Flow detections [page 45]Mappings detection options [page 45]



2.11.2 Data Flow editor

Data Flow preferences allow you to customize various settings of the Data Flow editor, including the way that the Auto-arrange option arranges the elements in the Data Flow diagram (for example, minimizing spacing between elements).

Table 37:

Option Description

Top Margin Specify the top margin spacing for the data flow diagram.

Left Margin Specify the left margin spacing for the data flow diagram.

Horizontal Margin Specify the horizontal margin spacing for the data flow diagram.

Vertical Margin Specify the vertical margin spacing for the data flow diagram.

Horizontal Alignment Specify the horizontal alignment for the data flow diagram:

● Left Aligned● Center Aligned● Right Aligned

Automatically put names in edit mode when an element is dropped in the diagram

Select to put the field names in edit mode when they are dropped into the diagram.

2.11.3 Data preview

Set Data preview preferences so that you can customize the way Data preview opens and you can change the maximum number of rows and kilobytes to retrieve.

To open, click Window Preferences... , and expand Data Services Data preview .

Table 38:

Option Description

Max Rows to retrieve from Tables Specify the maximum number of rows to retrieve from tables.

Max Size (in KB) to retrieve from Files Specify the maximum number of kilobytes to retrieve from files.

Data Preview opening mode Specify the viewing method:

● In an editor● In a view● In a dialog


2.11.4 Datastore editor

The Datastore editor preferences page provides options displaying tables and columns, and to detect the number of rows when you import tables.

To open, click Window Preferences... , and expand Data Services Datastore editor .

Table 39:

Option Description

Automatically detect row count when importing tables (might be slower)

Select to have the software display the row count in the table properties and project explorer dialogs when you import table metadata.

Tables:

Prefix tables with owner Check to prefix tables with owner.

Show column count Check to show column count.

Show row count Check to show row count.

Show row size Check to show row size.

Show description Check to show the table description.

Columns:

Show datatype Check to show the column datatype.

Show description Check to show the column description.

2.11.5 Expression editor

Customize the Expression editor by setting the Content assist and Syntax coloring options.

Content assist options

Table 40:

Option Description

Insert single proposals automatically Select this option to automatically insert single proposals.

Insert schema qualified column names Select this option to schema qualified column names.

Enable auto activation Select this option to enable automatic activation.

Activation delay (ms) Enter the activation delay in milliseconds. Default value is 200.



Option Description

Activation triggers Select the activation triggers you want to include:

'.' to propose qualified names

'$' to propose variables

'[' to propose substitution parameters

Syntax coloring

Use the syntax coloring options to set the color and fonts of the expression editor syntax.

2.11.6 Expression macros

These options allow you to manage the expression macros (creation, edition, and deletion) and import and export macros from this screen.

Expression macro listing

Table 41:

Button/option Description

New folder Create a new folder.

New macro Create a new expression macro.

Delete Delete selected expression macro(s).

Move up Move selected macro up.

Move down Move selected macro down.

General tab

Table 42:

Option Description

Description Describes the use for selected function.

Expression Enter the expression macro you want to add.


Conditions tab

Table 43:

Option Description

Restrict to following datatypes Check to only apply the expression macro to select datatypes. Select the datatypes from the dropdown list.

Restrict to following mapping types Check to apply the expression macro to select mapping types. Select mapping types from the dropdown list.

Restrict to the following content types Check to only apply expression macro to select content types. Select the content types from the dropdown list.

Restrict to the following field name pattern Check to restrict a specific field name pattern. Select the field name pattern from the dropdown list.

Other options/buttons

Table 44:

Option/button Description

Import Import the expression macros from file.

Export Export the expression macros to a file.

Restore Defaults Restores the expression macros back to the defaults included with Data Services Workbench.

2.11.7 File Format editor

The File Format editor options allow you to specify the number of kilobytes to scan during Data Preview and the detection algorithms of files.

Table 45:

Option Description

Maximum file size to use for the detect fields Specify the number of kilobytes to scan during the detection algorithms of files.

Maximum file size to use for Data Preview Specify the number of kilobytes to scan during data preview.

2.11.7.1 Detection settings

The Data Type detection options affect the data types detected when creating a file format from an existing file and detecting data types. If a field's values match one of the listed formats, Workbench will assign the



corresponding data type to the field: Date, Time or DateTime. Workbench attempts to detect the formats in the order they appear in the list for each format.

Table 46:

Option Descriptioin

DateTime Formats Add, edit, delete, or reorder DateTime formats.

Date Formats Add, edit, delete, or reorder Date formats.

Time Formats Add, edit, delete, or reorder Time formats.

Locales Add, edit, delete, or reorder Locales.

The configured locales can affect the detection of decimal data types if thousands-separators or decimal-separators are used. For example, 10.237.329,95 will not be recognized as a decimal (10,2) unless a locale that is configured with appropriate decimal characters (German) is included in the list.

Null Values Add, edit, delete, or reorder null values.

2.11.8 Project explorer

These options allow you to fine-tune the way Data Services Workbench project files are displayed in the project explorer. For example, you can specify how to group the tables of the Datastore file.

Table 47: General

Option Description

Show file extensions Check to show file extensions.

Table 48: Data stores

Option Description

Show table count Check to show the table count.

Show database count Check to show the database type.

Show description Check to show the datastore description.

Table 49: Tables

Option Description

Show columns count Check to show the number of columns.

Show table type Check to show the table type.

Show row count Check to show the table row count.

Show description Check to show the description.

show row size Check to show the row size.


Table 50: Tree structure

Option Description

Group table by type Check to group tables by type.

Group table by owner Check to group tables by owner.

Enable table chunking Check to enable table chunking.

Chunk size If Enable table chunking is selected, enter the number of tables.

Table 51: Columns

Option Description

Show data type Check to show the data type.

Show description Check to show the description.

2.12 General preferences

The Preferences section allows you to set your preferences for the following:

● General● Appearance● Content Types● Editors● Keys● Perspectives● Workspace

2.13 Language preferences

Enter the product languages you want to use in Workbench. To apply language changes you must restart Workbench.



3 Database Migration

Migrating data and schema information between different database systems is a complex task that can take days or even months depending on the type of database systems, the complexity of the database schema, and the size of the data set being migrated. In addition, incompatibilities between the source and target database types could require manual schema and data corrections.

The Data Services Workbench simplifies this database migration process considerably. In the Workbench, you provide connection information for the source and target databases and select the tables that you want to migrate. The Workbench automatically creates Data Services jobs, work flows, and data flows and imports them into a Data Services repository.

After you have created the replication jobs, you can use the Workbench to execute the jobs and monitor their progress. If required, you can also edit the generated data flows in the Designer, and use other Data Services tools to schedule and execute the jobs.

3.1 Migrating data and schema information

The process of migrating data and schema information from one database to another with the Workbench generally consists of a few basic steps:

1. Create a Workbench project in the project explorer.2. Create a source datastore definition.3. Create a target datastore definition.4. Create a replication job using your source and target datastores.

1. Select the source objects that you want to migrate.2. Group the target objects and specify the order of migration.3. Adjust the target schema as required.4. Specify the replication behavior for the job.

5. Deploy and execute the replication job.

You can choose to perform the datastore and replication job definition steps manually with the standard individual datastore and replication job editors, or you can complete the process using the quick replication wizard. If you use the replication wizard, you can modify your replication job by using the standard editors after the wizard completes.

Related Information

Project explorer [page 18]Datastore editor [page 21]Replication job editor [page 16]

Workbench GuideDatabase Migration P U B L I C 53

3.1.1 Creating a Workbench project

Create a Workbench project using the new project wizard. Start the wizard in one of the following ways:

● Right-click inside the project explorer and choose New Project .

● Choose New Project from the File menu.

● Click the New toolbar icon. Choose General Project and click Next.

With the new project wizard open, follow these steps:

1. Specify the name for the new project and click Next.

NoteThe project name can contain only alphanumeric characters and the underscore character.

2. If you want to save the files for the project to a different location, uncheck Use default location and specify a location for the project files.

3. Select the Data Services repository.4. Click Finish to create the project.

The new project is displayed in the project explorer.

You can now add Workbench objects such as datastores and replication jobs to your project.

Related Information

Creating a datastore [page 56]Using the replication job editor [page 57]

3.1.2 Using the quick replication wizard

You can use the quick replication wizard to easily migrate source data to one of the supported target databases or files.

1. Click the Quick replication wizard icon in the toolbar.The quick replication wizard opens.

2. Specify the name for the new replication project and click Next.


The source datastore configuration window opens.3. Specify the settings for the source datastore and click Next.

a. Select the datastore type.The available configuration settings for the selected database or application type display.


Database Migration

b. Set the datastore configuration settings as required for your database or application.

NoteRequired settings are marked with an asterisk (*), and required settings that are missing a value are highlighted in red in the Property name column. Settings that have changed from the default value are listed in bold.

c. Click Test connection to verify that the connection settings are valid.If the connection test fails, correct the datastore connection settings and re-test the connection.

NoteThe connection test is between the database or application and the Central Management Console (CMC) host, not the Workbench host. Ensure that the connection settings between the CMC and datastore database or application are valid.

The source object selection window opens.4. Select the objects that you want to replicate to the target datastore or files and click Next.

The configure target settings window opens.5. Specify the settings for the target datastore or files and click Next.

a. Select a target type of Files or Datastoreb. If the target type is Datastore, select the datastore type.

The available configuration settings for the selected database or application type are displayed.c. Set the configuration settings as required for your database, files, or application.


d. When you select datastore as the target type, cick Test connection to verify that the connection settings are valid.If the connection test fails, correct the datastore connection settings and re-test the connection.

The repository and job server selection window opens.6. If you want to execute the replication job without making any further changes, select Execute replication job

now.a. In the left column, select the Data Services repository to use.

The jobs generated by the Workbench will be stored in this repository. By default, the first repository in the list is selected.

b. In the right column, select the Data Services Job Server to use to execute the replication job.By default, the Job Server associated with the selected repository is used.

7. Click Finish to save the replication job and and execute it with the selected settings, if applicable.If you chose to execute the replication job, the monitoring editor opens.

The Workbench saves the replication job and datastore definitions to the local file system. If you chose to execute the replication job now, it also creates Data Services jobs and datastores for the replication job, imports them to the selected repository, and executes them using the selected Job Server.


3.1.3 Create datastores using the datastore editor

The datastore editor provides a powerful interface that allows you to fine-tune your datastore definition.

Use the datastore editor to import database tables, insert or modify columns, keys, indexes, and partitions, and to modify attributes, for example.

The datastores defined and used in the Workbench are standard Data Services datastores. For more information about Data Services datastores, see the Reference Guide.

Related Information

Datastore editor [page 21]Creating a datastore [page 56]

3.1.3.1 Creating a datastore

Create and configure a datastore.

If you have already created a datastore, open the applicable project in the Project Explorer and double-click the datastore name.

If you want to create a new datastore definition, start the new datastore wizard in one of the following ways:

● Right-click your project in the project explorer and choose New Datastore .

● Select your project in the project explorer and choose New Datastore from the File menu.

● Select your project in the project explorer and click the New toolbar icon. Choose Data ServicesDatastore and click Next.

The New Datastore dialog box opens.

1. Specify the name for the new datastore and optionally enter a description. Click Next.

NoteThe datastore name can contain only alphanumeric characters and the underscore character.

The Configuration parameters dialog box opens.2. Select a Datastore type from the drop-down list.3. Select a Database type from the drop-down list.

The available configuration settings for the selected database or application type display.4. Complete the datastore configuration settings as required for your database or application by double-clicking

in the property's Value row and entering the applicable values.


Database Migration


5. Click Test connection to verify that the connection settings are valid.If the connection test fails, correct the datastore connection settings and re-test the connection.

6. Click Next.The object selection window opens.

7. Select the objects to import to the datastore definition and click Finish.

The new datastore opens in the datastore editor.

You can now use the datastore editor to further customize your datastore definition. For example, you can select the Table or View tabs (at left) and modify properties, columns, or keys, or open the Configurations tab to add additional configurations to your datastore.

Related Information

Supported data migration sources and targets [page 76]Datastore editor [page 21]Create datastores using the datastore editor [page 56]

3.1.4 Using the replication job editor

The replication job editor provides a powerful interface that allows you to fine-tune your replication job, including filtering datastore tables, customizing target replication groups, modifying table and column mappings, and so on.

NoteThe replication jobs defined and used in the Workbench are defined in Workbench-specific metadata. When you deploy the replication job to a Data Services repository, the Workbench uses this metadata to create standard Data Services objects such as job, work flows, and data flows. For more information about Data Services objects, see the Reference Guide.

If you have already created a replication job, open it in the replication job editor by double-clicking the replication job name in the project explorer.

If you want to create a new replication job, start the new replication job wizard in one of the following ways:

● Right-click your project in the project explorer and choose New Replication Job .

● Select your project in the project explorer and choose New Replication Job from the File menu.

● Select your project in the project explorer and click the New toolbar icon. Choose Data ServicesReplication job and click Next.

The New Replication Job dialog box opens.


1. Select the parent project or folder and click Next.2. Enter a name and description for the new replication job and click Finish.

NoteThe replication job name can contain only alphanumeric characters and the underscore character.

The new replication job is opened in the replication job editor.3. Select a Replication Target Type from the drop-down list. Options include:

○ Datastore○ Files

4. Do one of the following:

○ If you selected Datastore, click ... for Target Datastore and select the source datastore.

A list of available datastores is displayed. If you have not already created the datastores, you can define the datastores later.

○ If you selected Files, double-click in the Value column for Root directory in Target File Properties and enter the root directory. You can click ... to browse for the root directory.

You can now use the replication job editor to further customize your replication job definition. For example, you can change the replication behavior, modify the target schema, customize column mappings, and so on.

Related Information

Replication job editor [page 16]

3.1.4.1 Modifying replication job properties

When you open a replication job or select the root replication job node in the target datastore pane, the configuration area displays several tabs that you can use to configure the way that the Workbench processes the replication job:

Table 52:

Tab Description

Properties Replication target type of Datastore: Displays the replication target type, name, owner, replication behavior and description for the replication job.

Replication target type of Files: Displays the replication target type, name, description and target file properties.

Variables Displays information about the variables available for use in the replication job.

You can add or delete variables, as well as alter variable properties such as data type, length, precision, scale, and default value.


Database Migration

Tab Description

Scripts Displays the pre-load and post-load scripts for the replication job.

You can modify the scripts directly in the text box or by clicking Edit to launch the Workbench expression editor. You can also validate the scripts.

Related Information

Using the expression editor [page 77]

3.1.4.1.1 Replication behavior

When you deploy and execute a Workbench replication job, the Workbench automatically generates the ETL metadata for the job and deploys it to a Data Services repository. The metadata that is generated depends on the replication behavior setting for the replication job.

The following replication behavior options are available:

● Schema and dataIn the schema and data scenario, the Workbench assumes that the tables do not exist in the target database and need to be created. As a result, the ETL job metadata that is generated includes the following objects:○ Source and target datastores○ Source and target tables○ A Data Services job that contains both the script to create the target tables, as well as the work flows and

data flows that move data from the source tables to the target tables.● Data only

In the data-only scenario, the Workbench assumes that the tables already exist in the target database. As a result, the ETL job metadata that is generated includes the following objects:○ Source and target datastores○ Source and target tables○ A Data Services job that contains work flows and data flows that move data from the source tables to the

target tables.

NoteSchema and data is the default replication behavior.


3.1.4.2 Adjusting the target schema

3.1.4.2.1 Modifying target table properties

When you select a table in the target datastore pane, the configuration area displays several tabs that you can use to configure the way that the Workbench processes that table:

Table 53:

Tab Description

Properties Displays the table name, owner, and description and provides the ability to view the data in the source table.

Columns Displays information about the target table columns.

After the table has been imported, you can insert and delete columns from the target table. You can also change column names, data types, descriptions, and so on, as well as alter the column mappings with the expression editor.

Foreign Keys Displays information about the table's primary and foreign keys, if any.

After the keys have been imported, you can add, delete, and modify them.

Indexes Displays information about the table's indexes. The primary index is listed first and followed by any secondary indexes.

You can add or remove indexes, as well as add or remove columns from existing indexes.

Options Displays options that Data Services uses to read the source table and load the target table.

For more information about the available source and target options, see the Reference Guide.

NoteOnly options used by the Workbench are displayed.

Filter data Allows you to filter the rows of the source table.

You can use basic filter editor or the expression editor to modify the filter.

DDL Options Allows you to specify the behavior of DDL generation.

For example, whether to create foreign keys or indexes, or create tables organized by row or column.

Delta load Displays options that modify the delta load behavior for a table.

For more information about the available delta load options, see Delta load migration [page 67].

3.1.4.2.2 Grouping target tables

Many data warehouse tables have referential integrity constraints such as primary key and foreign key relationships. The primary key table needs to be loaded before the foreign key table if there is a relationship between two tables.


Database Migration

You can avoid referential integrity issues by using the replication job editor to group tables. For example, you can put the primary key tables in the first group and foreign key tables in the second group. In another scenario, you could put dimension tables in the first group and fact tables in the second group.

By default, tables are added to the Default_Group that is created in a new replication job. However, there is no limit to the number of replication groups supported in a Workbench replication job.

To group tables in the replication job editor:

1. Create a new replication group in the target schema.a. Right-click on the replication job root node in the target schema pane and choose New replication group.

A new replication group is added to the target schema.b. Assign a meaningful name to the replication group by highlighting the new replication group and clicking it

again.2. Assign tables to the new group by dragging them from the source pane to the replication group.

NoteA table may belong to only one group in the target schema. You can create multiple target tables from a single source table, but each target table must be named uniquely.

3. Use the up and down arrows in the target schema to order the replication groups as needed for your requirements. When the replication job is executed, the groups will be migrated starting from the top of the list.You can also rearrange replication groups by dragging them within the target schema.

TipTo automatically order the tables in the target schema by their dependencies, click Auto order.

NoteReplication groups can be sequential or parallel. In a sequential group, the data flows generated by the Workbench will be executed in sequential order. In a parallel group, the data flows are executed in parallel. To change the group type, right-click on the group and choose Parallel or Sequential from the Change replication group type menu.

3.1.4.2.3 Changing column data types

Data Services supports conversion to and from its internal data types and performs data type conversions when it imports metadata from external sources or targets into the repository, as well as when it loads data into an external table. When the Workbench creates target tables, the software converts from internal data types to the data types used by the target database.

For more information about internal data type conversions in Data Services, see the Reference Guide.

The Workbench allows you to overwrite the default data conversion rules. To change the data type for a column:

1. Expand the table containing the column in the target schema pane.2. Select the target column.


The column details are displayed in the configuration pane.3. Select the new data type for the column from the Data type drop-down box.

If required for the new data type, set the Length, Precision, and Scale for the column.

3.1.4.3 Filtering datastore tables

In a replication job, you might want to migrate only a subset of the rows from a particular source table to your target datastore.

To migrate a subset of table rows, use a filter on the datastore table in the replication job editor:

1. Select the table that you want to filter in the target schema pane.2. Click the Filter data tab in the configuration pane.3. For basic filtering, use the table filters configuration window.

a. Select Basic and click Edit filter.The table filters configuration window is displayed.

b. Drag the columns you want to use to filter from the columns pane to the filters pane.The columns are displayed in the filters panel and grouped together with a logical AND or OR grouping operator.

c. For each column, define the filter logic by specifying the comparison operator and value.

TipYou can create more complex, nested filter expressions by dragging the columns on top of each other. The nesting layout is displayed in a graphical preview.

d. Change the grouping logic as required by double-clicking the grouping operator.e. To remove a filter, click the column or grouping operator to highlight the filter and click the Remove

selected filter button.f. When you are satisfied with the filter, click OK.

The filter is displayed in the Filter data tab.4. For more advanced filtering, use the Workbench expression editor to define the filter.

a. Select Advanced and click Edit filter.The expression editor is displayed.

b. Complete your filtering expression and click OK.The filter expression is displayed in the Filter data tab.

c. To validate your filter expression, click Validate.

Related Information

Using the expression editor [page 77]


Database Migration

3.1.5 Using the Data Flow editor

The Data Flow editor enables you to create data flows and include them in the replication job. You also have the ability to convert replication tables to data flows. Highlight multiple replication tables, right-click the selection, and select convert data. You have the option to generate a single data flow or one data flow per replicated table.

If you have already created a data flow, open it in the Data Flow editor by clicking the data flow name in the project explorer.

To create a new data flow, do one of the following:

● Select your project in the project explorer and right-click your data flow in the project explorer and choose New Data flow .

● Select your project in the project explorer and choose New Data Flow from the File menu.

● Select your project in the project explorer and click the New toolbar icon. Choose Data Services Data Flow and click Next.

1. Specify the name for the new data flow and click Next.

NoteThe data flow name can contain only alphanumeric characters and the underscore character.

2. Enter the description for the new data flow and click Finish.

The new data flow is opened in the Data Flow editor.

Related Information

Data Flow editor [page 18]Expression macros [page 64]Data Flow editor [page 47]

3.1.6 Converting replication tables to data flows

To convert a replication table to a data flow:

1. Highlight the replication tables you want to convert.2. Right-click the selection.3. Select one of the following:

○ Generate a single data flow○ One data flow per replicated table


3.1.7 Expression macros

You have the option to apply expression macros to multiple fields at one time when working in the Query transform of a data flow. You can access the Workbench Preferences options from the Query transform. This enables you to create new expression macros and apply them to the fields as needed.

3.1.7.1 Creating an expression macro

To create a new expression macro, access the Preferences Expression macros options one of the following ways:

● From the Output Schema tab, right-click a field and choose Apply expression macro Manage expression macros .

● Choose Preferences from the Window menu and select the Expression Macros option.

1. Click the New macro button and enter the macro name.2. From the General tab, enter a Description for the macro.3. From the General tab, enter the Expression.4. From the Conditions tab, select any conditions you want to apply to the expression macro.

Related Information

Expression macros [page 49]

3.1.7.2 Applying an expression macro to multiple fields

To apply an expression macro to multiple fields at one time:

1. Select the fields you want to include.2. Right-click the highlighted fields and select Apply expression macros.3. Select one of the expressions from the list or select Manage expression macros to add a new expression

macro.

3.1.8 Using the File Format editor

The File Format editor allows you to set properties for file format templates and source and target file formats.

If you have already created a file format, open it in the File Format editor by clicking the file format name in the project explorer.

If you want to create a new file format, start the File Format wizard in one of the following ways:


Database Migration

● Select your project in the project explorer, right-click and choose New Flat File Format .

● Select your project in the project explorer and click the New toolbar icon. Choose Data Services Flat File Format and click Next.

3.1.8.1 Manually create a new file format

To manually create a file format with default properties:

1. Select Manually create a new File Format and click Next:2. Specify the name and description for the new file format and click Finish.

NoteThe name can contain only alphanumeric and underscore characters.

The new file format is opened in the File Format editor.

Now you can further define the properties for the file format templates using the the File Format editor.

3.1.8.2 Automatically import file formats

This option enables you to import multiple files with different formats at the same time and generate multiple file formats. You have the option to auto-detect or specify the file formats.

To automatically import file formats from existing files:

1. Select Automatically import File Formats from existing files and click Next:2. Under File location, select Local or Job Server and click Add files.3. Select the file(s) you want to add and click Next.4. Do one of the following:

○ Click Next to set the detection options.○ Click Finish to run auto-detection.

5. If you chose to set detection options, set the detection options and click Finish.

The new file format is opened in the File Format editor.

Now you can use the File Format editor to further define the properties for the file format templates.

3.1.9 Deploying and executing the replication job

When you are satisfied with your replication job configuration, you can deploy it to a Data Services repository and execute it.

1. Double-click the replication job in the project explorer to open it in the replication job editor.


2. Click Validate to verify that the replication job is free of errors.3. If you want to deploy the job but not execute it, click Deploy.

a. Select the repository where you want to deploy the replication job and click OK.A progress bar indicating the deployment progress is displayed.

A message indicating the success or failure of the deployment is displayed. If any errors occur during deployment, the message includes any details available.

4. If you want to deploy and execute the job immediately, click Execute.a. Select the repository where you want to deploy the replication job and click Next.

The job execution parameters screen is displayed.b. Specify whether to execute the job as an initial load or delta load.c. Specify the job server and other execution options to use when executing the job in the Execution options

tab.d. If you want to include additional information in the trace log, change the values in the Trace tab to Yes as

required.e. Click Finish to deploy and execute the job.

A progress bar indicating the deployment progress is displayed. If you want to halt the deployment, click the Cancel Operation button next to the progress bar.

The monitoring editor for the job is displayed.

Related Information

Monitoring editor [page 43]

3.1.9.1 Error recovery

If an error occurs while your replication job is being executed, only some rows may be inserted into a target table.

There are several methods that you can use to prevent duplicate rows from being inserted into the target table, including:

● Data flow designYou can design the replication job to completely replace the target table each time the job is executed. You might choose to use this method if the number of rows changed in the target table is a significant percentage of the target table. You can also use techniques such as bulk loading options to improve the overall performance of the replication job.

● Auto-correct load target table optionYou can use the auto-correct load option to check the target table for existing rows before new rows are added to the table. However, this option can needlessly slow jobs that are not executed in recovery mode. You might choose to use this method if the number of rows changed in the target table is a relatively small percentage of the target table.

● Pre-load SQL commandsYou can use pre-load SQL commands to remove partial database updates that have occurred during an incomplete job execution. Typically, the pre-load SQL command deletes rows based on a variable set before the partial insertion began.


Database Migration

3.2 Delta load migration

Delta load jobs move rows that have been added or modified since the last time the job was executed. There are many reasons to implement a delta load job, but the most common is to reduce the time the loading process takes. Instead of loading millions of rows each time the job is run, you can process only the few that have changed. Another reason could be to maintain historical data; you might want to keep the old data in your data warehouse and add the current state so that you can see the changes over time.

3.2.1 Delta load jobs in the Workbench

When you create a replication job that has a table configured with a delta load method, the Workbench automatically creates two sets of metadata: a first load job and a delta load job. The first load job performs a full replication, with any user-defined filters applied, from the source datastore to the target datastore. The delta load job can be used to capture only changed data after the first load.

The Workbench creates two scripts within the delta load job: the pre-load script and the post-load script. These scripts keep track of the time each job is executed so that the next execution can continue from where the last one ended. You can customize the pre-load and post-load scripts in the Scripts property tab for the replication job.

To help keep track of the delta job executions, the Workbench maintains an internal table in the target datastore that is created after the successful execution of the first load job. Because of this, the full replication job must be successfully executed at least once before the delta load job can be executed.

NoteBy default, all target tables within the Workbench delta load job are created with the Auto correct load option enabled. For more information about this option, see the “Target tables” section in the Reference Guide.

NoteThe delta load mode is disabled in the replication job execution if the selected target type is Files.

Delta job execution

During execution, the delta job first determines the last time that it was executed, if ever. It uses the last execution time in a filter within the data flows to select only those records that are new. After the data flows have completed successfully, the delta job updates the internal metadata with the current time, so that any future delta execution starts from that time.


3.2.2 Delta load options for tables

For each table, the Workbench supports several delta load options:

NoteThe delta load option will be disabled when a target type of Files is selected. In this case, only the No delta load option will be enabled.

● No delta loadThe Workbench skips this table in the delta load.

● Reload the full tableThe Workbench performs a complete refresh of the table.

● Use timestamp columnThe Workbench uses a timestamp column or columns from the table to perform the delta load. Data Services uses the timestamp from each load to extract only new data from after the last execution.If a table has more than one datetime or timestamp column, you must choose the column to use for the delta load process. If a table does not contain a datetime or timestamp column, this option is disabled. If you use a column that contains only the date, you can also choose to use an additional column that specifies the time.

● Use CDCThe Workbench uses the changed data capture (CDC) functionality provided by a supported SAP application to extract only the data that has changed.If the datastore does not support CDC functionality, this option is disabled.

● Use custom filterThe Workbench uses a user-defined filter to determine the start time used to perform the delta load.

TipIn the Workbench, the delta load job works best with tables that use a timestamp column as the primary key.

To change the delta load method for a table, right-click the table in the target schema pane and choose the delta load method under Change delta load support. You can also select the table in the target schema pane and choose the delta load method in the Delta load tab of the replication job editor.

3.2.3 Delta load options for SAP applications

For applications that support changed data capture (CDC) functionality, the Workbench automatically extracts changed data for the delta load.

You can use the Workbench-generated variable $END_TIME to specify the time frame for which to extract changed data:

● If the value of $END_TIME is not specified, the Workbench uses the current system time for its value. Each delta load will extract new data from the last execution to the current date-time.

● If you want to continue from a previous execution, retrieve the timestamp of the execution you want to recover from your application and assign it to $END_TIME in the replication job pre-load script.


Database Migration

NoteThe Workbench treats applications that do not support CDC as tables for the delta load.


4 Using an SAP Information Steward Data Cleansing Solution

The Workbench Data Cleansing Solution feature allows a technical ETL developer with no data quality expertise to create a data flow containing a data cleansing solution and validate the results in a production environment.

First, a data steward uses the Data Cleansing Advisor feature of SAP Information Steward to create and publish a data cleansing solution. The data cleansing solution is stored in the SAP BusinessObjects Business Intelligence platform's Central Management Server (CMS). In the Workbench, the ETL developer then models the data cleansing solution as a Data Cleansing Solution transform within the context of a data flow, and maps the transform's input and output schema to production sources and targets, and validates the results in the production environment. Finally, the ETL developer generates and deploys the data flow as ATL to the Data Services repository for execution in the Data Services engine.

For more information about input and output fields and options, see the SAP Data Services Reference Guide. For more information about Data Cleansing Advisor, see the SAP Information Steward User Guide.

4.1 Viewing data cleansing solutions

You can create multiple data flows that use a data cleansing solution. Each data flow can contain one or more unique instances of the Data Cleansing Solution transform that references the data cleansing solution.

The Data Cleansing Solution transform references the latest published version of the data cleansing solution from Data Cleansing Advisor. If a data cleansing solution is updated and republished in Data Cleansing Advisor, Workbench automatically uses the updated solution, and you must redeploy the data flow to the Data Services repository.

1. Select Window Data Cleansing Solutions . The Data Cleansing Solutions tab opens in the workspace and displays the data cleansing solutions that have been published in the Data Quality Advisor.The Published column contains the date and time that the solution was published. Workbench displays only the latest active version of a data cleansing solution published in the Data Cleansing Advisor. Icons display the status of the data cleansing solution.

Table 54:

Icon Description

Indicates a new data cleansing solution that has not been used.

Indicates a current data cleansing solution that has been used in one or more data flows.

Indicates an updated data cleansing solution that has been republished since it was used.

2. To refresh the list of data cleansing solution, click the refresh icon ( ).3. Click a column header to sort the data cleansing solutions by the content in that column.4. To view more details about and identify the appropriate data cleansing solution, double-click a solution.


Using an SAP Information Steward Data Cleansing Solution

The Data Cleansing Solution Details window opens and displays information about the solution such as the publication date and time, cleansing and matching settings, and the connection information and input schema that were used to create the data cleansing solution in Information Steward.

5. To open all of the data flows that contain a data cleansing solution, right-click the solution and select Open Containing Data Flows from the menu.

4.1.1 Deleting a data cleansing solution

You can delete a data cleansing solution in Workbench only if its corresponding project has first been deleted in the Data Cleansing Advisor in Information Steward.

To delete a data cleansing solution, right-click the solution and select Delete from the menu.

4.2 Configuring a data flow

Before you start modeling, a data cleansing solution must have already been created in SAP Information Steward and published to the CMS, and you should create a project, datastore, and data flow in the Workbench.

1. Open or create a data flow. The palette of elements on the right side of the window contains readers, loaders, and Platform and Data Integrator transforms.

2. Add a reader and loader to the data flow, as well as any other transforms required to manipulate the data.

3. Select Window Data Cleansing Solutions .The Data Cleansing Solutions tab opens in the workspace and displays the data cleansing solutions that have been published in the Data Cleansing Advisor.

4. Select a data cleansing solution and drag it into the graphical area of the Data Flow editor. A Data Cleansing Solution transform is placed in the data flow that represents the data quality transforms, such as Data Cleanse, Global Address Cleanse, and Match. The underlying transforms included in the Data Cleansing Solution transform depend on the data cleansing solution recommendations that were chosen in Information Steward.The object properties panel opens below the data flow graphical area with three tabs.

Table 55:

Property tab Description

Input Displays a list of all input fields provided by the data cleansing solution that must be mapped in order for the data cleansing solution to be executed in Data Services. The specific input fields vary depending on the configuration of the data cleansing solution. The Description column displays a text description of the input field based on the content type that was identified by Data Cleansing Advisor.

Properties Displays the Data Services options that you can set for the Data Cleansing Solution transform. Data quality configuration options (the business rules of cleansing and matching) are not editable within the Workbench. If an option setting must be modified, this is done in the Data Cleansing Advisor feature in SAP Information Steward, and the data cleansing solution must be republished.

Workbench GuideUsing an SAP Information Steward Data Cleansing Solution P U B L I C 71

Property tab Description

Output Displays a list of all available output fields from the data cleansing solution. The available output fields may vary depending on the configuration of the data cleansing solution.

5. To perform basic cleansing on your data to prepare it for more accurate cleansing and matching results, add a Basic Cleanse transform to the data flow and select the appropriate options. Typically, this basic cleansing is performed in Information Steward rather than Data Services Workbench.

6. Connect the Data Cleansing Solution transform to the reader and loader or other transforms in the data flow.7. To automatically map fields from the reader to the Data Cleansing Solution transform, select the Input tab

and click the Detect Mappings button.The Detect Mapping window opens and displays the field names candidates for mapping.

8. To confirm all of the suggested field mappings, click the Confirm All button; otherwise, select individual field names and click Confirm or Reject as appropriate. Click OK.

9. To manually map fields, select a field in the Input Schema and drag it to the appropriate row in the Input tab.10. In the Output tab, select the field names that you want to output from the Available Output Fields and drag

them into the Selected Fields list. If you want to output passthrough fields containing the original data before it was processed, select them in the Input Schema and drag them into the Selected Fields list. Use the Up and Down buttons to reorder the fields as you want to output them.

For information about transform options and input and output fields, see the SAP Data Services Reference Guide.

Related Information

Using the Basic Cleanse transform [page 72]

4.2.1 Using the Basic Cleanse transform

In Information Steward, the Data Cleansing Advisor lets the data steward set up textual changes (transformations) to apply to specified columns in the data. Transformations set up in the wizard can help prepare the data for more accurate cleansing and matching results. When the data cleansing solution is published and then used in a data flow in Workbench, the transformations are represented in the Data Cleansing Solution transform, and the options are not editable.

Although basic cleansing is typically set up by the data steward in Information Steward, in Workbench, you can also add a Basic Cleanse transform, a specialized type of Query transform, to your data flow to perform the same cleansing on your data.

When the data flow is converted to ATL and deployed to the Data Services repository, the Basic Cleanse transform is converted to a Query transform.

Table 56:

Option Description

Remove control characters

Removes control characters from the selected column.



Option Description

Remove double quotes Removes double quotation marks (") that appear anywhere in the selected column.

Remove leading spaces Removes spaces before the text in the selected column.

Remove single quotes Removes single quotation marks (') that appear anywhere in the selected column.

Remove trailing spaces Remove spaces after the text in the selected column.

Replace null values with <default>

Enter the text that you want to take the place of empty fields in the selected column.

Search and replace In the Replace box, enter the text that you want to delete from the selected column. In the with box, enter the text that you want to take the place of the deleted text. Click Add.

Repeat these steps for each string of text that you want to replace.

Use the up and down arrow buttons if you need to reorder search-and-replace values.

4.3 Modifying the reference data location

The location of the reference data is always set to the substitution parameter, $$RefFilesAddressCleanse, and cannot be modified within Workbench. The substitution parameter is set to the default location of the reference data during the Data Services installation, and Workbench automatically finds the reference data if you haven't modified the default.

To use a different reference data location to run a Data Cleansing Solution transform within Workbench, use the Data Services Designer to log into the repository that you want to modify, and edit the substitution parameter, $$RefFilesAddressCleanse, to point to the desired reference file directory. All Data Cleansing Solution transforms in Workbench will use this new location when executing a data flow for the repository.

4.4 Modifying the reports substitution variables

Enabling and disabling of data quality reports is set by the substitution parameters, $$ReportsAddressCleanse and $$ReportsMatch, and cannot be modified within Workbench. This allows reporting to be controlled at runtime within Data Services rather than being set within Workbench.

To enable or disable data quality reports for Workbench, use the Data Services Designer to log into the repository that you want to modify, and edit the substitution parameters, $$ReportsAddressCleanse and $$ReportsMatch.


4.5 Validating a data flow

To validate a data flow, click the Validate button. The status bar displays the validation status and the number of errors and warnings, if any.

Workbench performs the following validation on the Data Cleansing Solution transform:

● The Data Cleansing Solution can be retrieved from the SAP BusinessObjects Business Intelligence platform's Central Management Server (CMS).

● All input fields specified by the Data Cleansing Solution are mapped.● All output fields in the Data Cleansing Solution transform output schema have a unique field name.● At least one output field is specified in the Data Cleansing Solution transform.

If a validation error is found, the error is displayed in the Problems tab. Double-click an error to open the relevant Property tab in the Data Flow editor.

4.6 Deploying a data flow

Deploying a data flow publishes it to the Data Services repository.

The Data Cleansing Solution transform references the latest published version of the data cleansing solution from Data Cleansing Advisor. If a data cleansing solution is updated and republished in Data Cleansing Advisor, Workbench automatically uses the updated solution, and you must redeploy the data flow to the Data Services repository.

1. With the data flow open in the Data Flow editor, click the Deploy button. If you haven't saved the data flow, a prompt opens asking if you want to save the changes. Click Yes to save the changes.The Choose Repository window opens.

2. Select the Data Services repository that you want to deploy to, and click OK. When deployment is complete, an information window opens stating that the data flow was successfully deployed on the repository.

3. Click Close.

Deploying a data flow generates ATL that represents the elements in the data flow and can be executed by Data Services. After you deploy the job to the Data Services repository, you can open, modify, and execute it as a new, separate job instance in the Data Services Designer. However, any changes that you make in Data Services Designer are not saved to the Workbench instance of the job.

4.7 Executing a data flow

The execution process validates a data flow, generates ATL, deploys it to a Data Services repository, and executes the data flow.

1. With the data flow open in the Data Flow editor, click the Execute button.If you haven't saved the data flow, a prompt opens asking if you want to save the changes.



2. Click Yes to save the changes.The Execute job wizard opens.

3. In Step 1 of the Execute Job wizard, select the Data Services repository that you want to use and click Next.The wizard proceeds to the next step.

4. In Step 2 of the Execute Job wizard, select the appropriate execution parameters and click Finish.The job executes, and the monitor editor opens and displays the job status, execution progress, and any applicable logs. Trace, monitor, and error logs are available. When the job is completed, the job status is displayed at the top.

4.8 Viewing data

To view the data before and after it is processed, you can view data in both the reader and loader transforms.

1. In the graphical area of the Data Flow editor, select the reader or loader.2. In the Properties tab for the selected loader, click the View Data button. A data grid opens and displays the

first 200 records.3. To view more data rows, edit the Max Rows field and click the refresh button.

4.9 Modifying a Workbench job in Data Services Designer

Workbench allows limited changes to transform options and fields in the data flow. In Data Services Designer, you can modify a data flow that was created in Workbench more extensively; however, any changes that you make in the Designer cannot be exported to or viewed in the Workbench. In the Designer, you see the individual transforms (Data Cleanse, Global Address Cleanse, and Match) that were represented by the Data Cleansing Solution transform in Workbench. You can change options, input fields, and output fields for Data Cleanse and Global Address Cleanse. You cannot modify Match settings. You can edit connection between all of the transforms on the canvas as necessary.

1. In the Local Object Library of the Data Services Designer, click the Jobs tab, and expand Batch Jobs to display the Workbench data cleansing solution job.

2. Double-click the job. The job opens on the canvas and contains a work flow.3. Double-click the work flow. The work flow opens and contains a data flow.4. Double-click the data flow. The data flow opens. Rather than displaying the single Data Cleansing Solution

transform, the data flow contains the data quality transforms that the Data Cleansing Solution transform represents, such as Data Cleanse, Global Address Cleanse, and Match. The data quality transforms used depends on the configuration of the data cleansing solution created in Information Steward.

5. Modify and execute the job as necessary.

Any changes that you make in Data Services Designer are not saved to the Workbench instance of the job.


5 Additional Information

5.1 Supported data migration sources and targets

Supported sources

The Workbench supports the following database and application types as migration sources:

● Data Services-supported databases○ Attunity Connector○ DB2○ SAP HANA○ Informix○ Microsoft SQL Server○ MySQL○ Netezza○ ODBC○ Oracle○ SQL Anywhere○ SAP ASE○ Sybase IQ○ Teradata

● SAP Applications● SAP BW Source

NoteFor more information about the available options for each supported datastore source type, see the Reference Guide.

Supported targets

The Workbench supports the following database types as migration targets:

● SAP HANA● DB2● Netezza● Oracle● Sybase IQ● Teradata


Additional Information

● SAP ASE● SQL Anywhere● Microsoft SQL Server

NoteFor more information about the available options for each supported datastore target type, see the Reference Guide.

5.2 Using the expression editor

The expression editor allows you to use variables, functions, and basic logic structures in many areas of a replication job. For example, you can use the expression editor to customize the mapping of a target column or the replication job's pre-load or post-load scripts.

5.2.1 Supported operators

This section describes the operators that can be used in Workbench expressions and filters.

Table 57:

Operator Description

+ Addition

- Subtraction

* Multiplication

/ Division

= Comparison, equal to

< Comparison, less than

<= Comparison, less than or equal to

> Comparison, greater than

>= Comparison, greater than or equal to

!= Comparison, not equal to

|| Concatenate

% Modulus. Returns the remainder when one number is divided by another.

AND Logical AND

OR Logical OR

NOT Logical NOT

Workbench GuideAdditional Information P U B L I C 77

Operator Description

IS NULL Comparison, is a NULL value

IS NOT NULL Comparison, is not a NULL value

5.2.2 Include functions in expressions

In Data Services, functions take input values and produce a return value. Input values can be parameters passed into a data flow, values from a column of data, or variables defined inside a script. This section discusses functions and how to include them in a Workbench replication job.

5.2.2.1 Built-in functions

This section describes each built-in function available in Data Services.

The tables in this section list the names and descriptions of functions, grouped by each function's category in the Workbench expression editor.

For a complete listing of all built-in functions and their syntax, parameters, and return values, see “Functions and Procedures, Descriptions of built-in functions” in the Reference Guide.

Related Information

Aggregate functions [page 79]Conversion functions [page 79]Cryptographic functions [page 80]Database functions [page 80]Date functions [page 81]Environment functions [page 82]Lookup functions [page 82]Math functions [page 83]Miscellaneous functions [page 83]SAP functions [page 85]String functions [page 85]System functions [page 87]User functions [page 87]Validation functions [page 87]



5.2.2.1.1 Aggregate functions

List and descriptions of aggregate functions that are built into Workbench.

Table 58:

Function Description

avg Calculates the average of a given set of values.

count Counts the number of values in a table column.

count_distinct Count the number of distinct non-null values in a table column.

max Returns the maximum value from a list.

min Returns the minimum value from a list.

sum Calculates the sum of a given set of values.

Related Information

Built-in functions [page 78]

5.2.2.1.2 Conversion functions

List and descriptions of conversion functions that are built into Workbench.

Table 59:


cast Returns a value in the cast data type.

extract_from_xml Extracts XML directly from single column in a database table, and converts it into its internal nested relational data model (NRDM).

interval_to_char Returns a string representation of the interval.

julian_to_date Converts a Julian value to a date.

load_to_xml Generates XML text from NRDM and loads it into a single database column (Assumes the database supports XML text in its columns).

long_to_varchar Converts a data type from long to varchar.

num_to_interval Converts a numeric value to an interval.

to_char Converts a date or numeric type to a string.

to_date Converts a string to a date.

to_decimal Converts a varchar to a decimal.

varchar_to_long Converts a data type from varchar to long.


Related Information


5.2.2.1.3 Cryptographic functions

List and descriptions of aggregate functions that are built into Workbench.

Table 60:


decrypt_aes Decrypts the input string using the user-specified passphrase and key length using the AES algorithm.

encrypt_aes Encrypts the input string using the user-specified passphrase and key length using the AES algorithm.

Related Information


5.2.2.1.4 Database functions

List and descriptions of database functions that are built into Workbench.

Table 61:


key_generation Generates keys for the specified table, after determining the appropriate starting value.

sql Runs a SQL operation in the specified database.

total_rows Returns the number of rows in a particular table in a datastore.

Related Information




5.2.2.1.5 Date functions

List and descriptions of date functions that are built into Workbench.

Table 62:


add_months Adds a given number of months to a date.

concat_date_time Returns a datetime from separate date and time inputs.

date_diff Returns the difference between two dates or times.

date_part Extracts a component of a given date.

day_in_month Determines the day in the month on which the given date falls.

day_in_week Determines the day in the week on which the given date falls.

day_in_year Determines the day in the year on which the given date falls.

fiscal_day Converts a given date into an integer value representing a day in a fiscal year.

isweekend Indicates that a date corresponds to Saturday or Sunday.

julian Converts a date to its integer Julian value, the number of days between the start of the Julian calendar and the date.

last_date Returns the last date of the month for a given date.

month Determines the month in which the given date falls.

quarter Determines the quarter in which the given date falls.

sysdate Returns the current date as listed by the Job Server's operating system.

systime Returns the current time as listed by the Job Server's operating system.

week_in_month Determines the week in the month in which the given date falls.

week_in_year Determines the week in the year in which the given date falls.

year Determines the year in which the given date falls.

Related Information



5.2.2.1.6 Environment functions

List and descriptions of environment functions that are built into Workbench.

Table 63:


get_env Returns a value for the specified environmental variable.

get_error_filename Returns the full path and file name for the error log.

get_monitor_filename Returns the full path and file name for the monitor log.

get_trace_filename Returns the full path and file name for the trace log.

is_set_env Verifies if the specified environment variable is set.

set_env Sets an environmental variable temporarily to a specified value.

Related Information


5.2.2.1.7 Lookup functions

List and descriptions of lookup functions that are built into Workbench.

Table 64:


lookup Finds a value in one table or file based on values in a second table or file.

lookup_ext Finds data from a database table, flat file, or memory datastore table.

Related Information




5.2.2.1.8 Math functions

List and descriptions of math functions that are built into Workbench.

Table 65:


abs Returns the absolute value of an input number.

ceil Returns the smallest integer value greater than or equal to an input number.

floor Returns the largest integer value less than or equal to an input number.

ln Returns the natural logarithm of the given numeric expression.

log Returns the base-10 logarithm of the given numeric expression.

mod Returns the remainder when one number is divided by another.

power Returns the value of the give expression to the specified power.

rand Returns a random number between 0 and 1.

rand_ext Returns a random number between 0 and 1.

round Rounds a given number to the specified precision.

sqrt Returns the square root of the given expression.

trunc Truncates a given number to the specified precision.

Related Information


5.2.2.1.9 Miscellaneous functions

List and descriptions of miscellaneous functions that are built into Workbench.

Table 66:


base64_decode Returns the source data after decoding the base64-encoded input.

base64_encode Returns the base64-encoded data in the engine locale character set.

current_configuration Returns the name of the datastore configuration in use at runtime.

current_system_configuration

Returns the name of the system configuration in use at runtime. If no system configuration is defined, returns a NULL value.

dataflow_name Returns the data flow name in which this call exists. If the call is not in a data flow, returns NULL.

datastore_field_value Retrieves the value of a specified datastore field.



db_database_name Returns the database name of the datastore configuration in use at runtime.

db_owner Returns the real owner name for the datastore configuration that is in use at runtime.

db_type Returns the database type of the datastore configuration in use at runtime.

db_version Returns the database version of the datastore configuration in use at runtime.

decode Returns an expression based on the first condition in the specified list that evaluates to TRUE.

file_exists Checks to see if a given file or directory exists.

gen_row_num Returns an integer value beginning with 1 then incremented sequentially by 1 for each additional call. This function can be used to generate a column of row IDs.

gen_row_num_by_group Returns group row number of the record.

gen_uuid Returns a unique varchar string identifier.

get_domain_description Returns the description of a value when given the domain name and the value.

get_file_attribute Returns date created, date modified, or size (in bytes) of a physical file.

greatest Returns greatest of the list of one or more expressions.

host_name Returns the name of the computer on which the job is executing.

ifthenelse Allows conditional logic in mapping and selection operations.

is_group_changed Returns 1 if the group is changed, 0 otherwise.

isempty Indicates if a nested table contains data.

job_name Returns the name of the job in which the call to this function exists.

job_run_id Retrieves the job run ID for the current job execution.

least Returns the least in a list of one ormore expressions.

nvl Replaces NULL values.

previous_row_value Returns the column value of previous row.

pushdown_sql Allows you to create dynamic WHERE clauses.

raise_exception Calling this function causes an exception to be generated.

raise_exception_ext Same as raise_exception, but takes a second parameter for an exit code.

repository_name Returns a database connection string and owner name. For example: beq-local.DBUser. This is the ID for the repository from which the job is run.

sleep Suspends the execution of the data flow or work flow from where it is called.

system_user_name Returns the user name used to log into the Job Server's operating system.

table_attribute Retrieves the value of a specified table attribute.

truncate_table Allows you to explicitly expunge data from a memory table.

wait_for_file Returns the existing files that match the input file pattern.

workflow_name Returns the work flow in which this call exists. Returns the name of the inner most work flow in cases where several work flows enclose this function call. If no work flow is found, returns job name.



Related Information


5.2.2.1.10 SAP functions

List and descriptions of SAP functions that are built into Workbench.

Table 67:


sap_openhub_processchain_execute

Starts the process chain that extracts data from an SAP NetWeaver Business Warehouse(BW) and loads the extracted data into an Open Hub Destination table.

sap_openhub_set_read_status

Sends the read status for the Open Hub table to SAP NetWeaver BW.

Related Information


5.2.2.1.11 String functions

List and descriptions of string functions that are built into Workbench.

Table 68:


ascii Returns the decimal value of the first character for the given string using ASCII character set. If the character passed is not a valid ASCII character, -1 is returned.

chr Get character representation of provided ASCII value.

double_metaphone Encodes the input string usin the Double Metaphone algorithm and returns a string.

index Returns the index of a given word in a string.

init_cap Changes the characters in a string to title case.

length Returns the number of characters in a given string.

literal Returns an input constant expression without interpolation. Allows you to assign a pattern to a variable without interpolation.

lower Changes the characters in a string to lowercase.

lpad Pads a string with characters froma specified pattern.

lpad_ext Pads a string with logical characters from a specified pattern.



ltrim Removes specified characters from the start of a string.

ltrim_blanks Removes blank characters from the start of a string.

ltrim_blanks_ext Removes blank and control characters from the start of a string.

match_pattern Matches whole input strings to simple patterns supported by Data Services. This function does not match substrings.

match_regex Matches whole input strings to the pattern that you specify with regular expressions (regular expressions based on the POSIX standard) and flags. This function does not match substrings.

match_simple

print Prints the given string to the trace log.

replace_substr Returns a string where every occurrence of a given search string in the input is substituted by the given replacement string.

replace_substr_ext Takes an input string, replaces specified occurrences of a specified sub-string with a specified replacement and returns the result. You can also use this function to search for hexadecimal or reference characters.

rpad Pads a string with characters from a given pattern.

rpad_ext Pads a string with logical characters from a given pattern.

rtrim Removes given characters from the end of a string.

rtrim_blanks Removes blank characters from the end of a string.

rtrim_blanks_ext Removes blank and control characters from the end of a string.

search_replace Searches input parameters and replaces by matching criteria and values specified by search table.

soundex Encodes the input string using the Soundex algorithm and returns a string. Use when you want to push down the function to the database level.

substr Returns a specific portion of a string starting at a given point in the string.

upper Changes the characters in a string to uppercase.

word Returns one word out of a string.

word_ext Returns the word identified by its position in a delimited string.

Related Information




5.2.2.1.12 System functions

List and descriptions of system functions that are built into Workbench.

Table 69:


exec Sends a command to the operating system for execution.

mail_to Sends the specified e-mail message.

smtp_to Sends the specified e-mail message using the SMTP protocol.

Related Information


5.2.2.1.13 User functions

Create your own functions

You can create your own functions by writing script functions in SAP Data Services scripting language using the Designer smart editor.

For more information about user-defined custom functions, see “Functions and Procedures, Custom functions” in the Reference Guide.

Related Information


5.2.2.1.14 Validation functions

List and descriptions of validation functions that are built into Workbench.

Table 70:


is_valid_date Indicates if an expression can be converted into a valid date value.

is_valid_datetime Indicates if an expression can be converted into a valid datetime value.

is_valid_decimal Indicates if an expression can be converted into a valid decimal value.



is_valid_double Indicates if an expression can be converted into a valid double value.

is_valid_int Indicates if an expression can be converted into a valid integer value.

is_valid_real Indicates if an expression can be converted into a valid real value.

is_valid_time Indicates if an expression can be converted into a valid time value.

Related Information


5.3 Managing Workbench project information

5.3.1 Creating a Workbench project

Create a Workbench project using the new project wizard. Start the wizard in one of the following ways:

● Right-click inside the project explorer and choose New Project .

● Choose New Project from the File menu.

● Click the New toolbar icon. Choose General Project and click Next.

With the new project wizard open, follow these steps:

1. Specify the name for the new project and click Next.


2. If you want to save the files for the project to a different location, uncheck Use default location and specify a location for the project files.

3. Select the Data Services repository.4. Click Finish to create the project.

The new project is displayed in the project explorer.

You can now add Workbench objects such as datastores and replication jobs to your project.

Related Information

Creating a datastore [page 56]Using the replication job editor [page 57]



5.3.2 Exporting resources and preferences

Use the Export wizard to export Workbench folders, resources, and user preferences.

1. Right-click within the project explorer pane and choose Export.The Export wizard appears.

2. Select the type of export that you want to perform and click Next.

○ Archive FileExports selected Workbench resources to a .zip or .tar archive on the local file system.

○ File SystemExports selected Workbench resources directly to a location on the local file system.

○ PreferencesExports Workbench configuration preferences to a location on the local file system.

3. If you are exporting to an archive file or a location on the file system, select the resources to export and set the export options as desired.a. Select the folders and resources that you want to export.

Folders appear in the left pane and resources appear in the right pane.

TipFolders with a grayed-out selection box are not explicitly selected, but contain objects that will be exported.

b. Click Browse to specify the location where the archive file or resource files will be created.c. If you are exporting to an archive file, select the type of archive file to create.d. Select the directory structure mode.

Table 71:

Option Description

Create directory structure for files Creates the directory structure for all selected resources, even if no folders are explicitly selected.

Create only selected directories Creates the directory structure only for folders that are explicitly selected.

4. If you are exporting preferences, select the preferences to export and set the export options as desired.a. Select the preferences to export, or select Export all to export all Workbench preferences.b. Click Browse to specify the location where the preference file will be exported.

5. Click Finish.The selected resources or preferences are exported to the specified archive file or file system location.

5.3.3 Importing resources and preferences

Use the Import wizard to import Workbench folders, resources, and user preferences.

1. Right-click within the project explorer pane and choose Import.The Import wizard appears.


2. Select the type of export that you want to perform and click Next.

○ Archive FileImports selected Workbench resources from a .zip or .tar archive on the local file system.

○ File SystemImports selected Workbench resource files from a location on the local file system.

○ PreferencesImports Workbench configuration preferences from a file on the local file system.

3. If you are importing from an archive file or a location on the file system, select the resources to import and set the import options as desired.a. Click Browse to specify the location of the archive file or resource files.b. Select the folders and resources that you want to import.

Folders appear in the left pane and resources appear in the right pane.

TipFolders with a grayed-out selection box are not explicitly selected, but contain objects that will be imported.

c. Click Browse to specify the folder where the imported resources will be stored.4. If you are importing preferences, select the preferences to import.

a. Click Browse to specify the preference file to import.b. Select the preferences to import, or select Import all to import all Workbench preferences.

5. Click Finish.The selected resources or preferences are imported to the Workbench.



Important Disclaimers and Legal Information

Coding SamplesAny software coding and/or code lines / strings ("Code") included in this documentation are only examples and are not intended to be used in a productive system environment. The Code is only intended to better explain and visualize the syntax and phrasing rules of certain coding. SAP does not warrant the correctness and completeness of the Code given herein, and SAP shall not be liable for errors or damages caused by the usage of the Code, unless damages were caused by SAP intentionally or by SAP's gross negligence.

AccessibilityThe information contained in the SAP documentation represents SAP's current view of accessibility criteria as of the date of publication; it is in no way intended to be a binding guideline on how to ensure accessibility of software products. SAP in particular disclaims any liability in relation to this document. This disclaimer, however, does not apply in cases of willful misconduct or gross negligence of SAP. Furthermore, this document does not result in any direct or indirect contractual obligations of SAP.

Gender-Neutral LanguageAs far as possible, SAP documentation is gender neutral. Depending on the context, the reader is addressed directly with "you", or a gender-neutral noun (such as "sales person" or "working days") is used. If when referring to members of both sexes, however, the third-person singular cannot be avoided or a gender-neutral noun does not exist, SAP reserves the right to use the masculine form of the noun and pronoun. This is to ensure that the documentation remains comprehensible.

Internet HyperlinksThe SAP documentation may contain hyperlinks to the Internet. These hyperlinks are intended to serve as a hint about where to find related information. SAP does not warrant the availability and correctness of this related information or the ability of this information to serve a particular purpose. SAP shall not be liable for any damages caused by the use of related information unless damages have been caused by SAP's gross negligence or willful misconduct. All links are categorized for transparency (see: http://help.sap.com/disclaimer).

Workbench GuideImportant Disclaimers and Legal Information P U B L I C 91

http://help.sap.com/disclaimer/

go.sap.com/registration/contact.html

© 2016 SAP SE or an SAP affiliate company. All rights reserved.No part of this publication may be reproduced or transmitted in any form or for any purpose without the express permission of SAP SE or an SAP affiliate company. The information contained herein may be changed without prior notice.Some software products marketed by SAP SE and its distributors contain proprietary software components of other software vendors. National product specifications may vary.These materials are provided by SAP SE or an SAP affiliate company for informational purposes only, without representation or warranty of any kind, and SAP or its affiliated companies shall not be liable for errors or omissions with respect to the materials. The only warranties for SAP or SAP affiliate company products and services are those that are set forth in the express warranty statements accompanying such products and services, if any. Nothing herein should be construed as constituting an additional warranty.SAP and other SAP products and services mentioned herein as well as their respective logos are trademarks or registered trademarks of SAP SE (or an SAP affiliate company) in Germany and other countries. All other product and service names mentioned are the trademarks of their respective companies.Please see http://www.sap.com/corporate-en/legal/copyright/index.epx for additional trademark information and notices.

https://go.sap.com/registration/contact.html

https://go.sap.com/registration/contact.html

http://www.sap.com/corporate-en/legal/copyright/index.epx

http://www.sap.com/corporate-en/legal/copyright/index.epx

workbench guide - sap help portal · pdf filesap data services document version: 4.2 support...

Documents