websphere transformation extender - ibm · pdf filewebsphere datastage. 6 ibm software group |...

1

IBM FSS

© 2006 IBM Corporation

Kimberly DrouotWebSphere Transformation Extender Technical Lead – SW [email protected]

WebSphere Transformation ExtenderWebSphere’s Universal Transformation Engine for the Enterprise

IBM Software Group | WebSphere software

What Does IBM WebSphere TX Do?

It takes any kinds of data from their native forms

ANY Data

Processes them together, natively, with NO CODE

And outputs them into their native target formatsANY Data

Many-to-many Integration

Ex. - Hierarchical Data , Binary Data, Packed Data, Tabular Data, Relational Data, Nested Structures, Mixed-Type Data, and on and on…

2


Transformation Vision - Simplified

�Transform Anything

�Transform Anywhere

�Everyone Transforms


The Need for Transformation

� "Organizations pursuing service-oriented integration involving document-oriented, proprietary, legacy or modern XML-oriented data standards… should consider products capable of automating complex data transformation as an important building block of their integration architecture."

�Massimo Pezzini, VP and Distinguished Analyst, Gartner

� “Two of the biggest and most significant challenges of dealing with data are transformation and semantics.”

�Zapthink

� “There are many integration styles and challenges, each with their own unique demands; what is consistent across every one of them is the need for data transformation and mediation.”

�IBM

“Transformation is the hard, ugly part of integrati on”-Ted Friedman, Research VP, Gartner

3


Resources and IT Assets (insurance company)

…Legacy Claims

Database

Home Claims System

Business Partner Claims System

AutoClaims System

LifeInsurance

Claims

The Challenge and TransformationThe Challenge and Transformation

1. Each system has its own data definitions, business rules and interfaces

2. Providing granularity to application’s business fu nctions requires ability to maintain contextual usage and data meaning across systems

3. Interactions between applications expose data and business rule anomalies which result in high maintenance costs and inflexibility to change

4. The Validation and Use Rules must align with the Transformation needs and execute in the same unit of work with Transformation

5. Ever increasing integration logic points with duplicate or slightly modified templatesrob from anticipated benefits and efficiencies

“ About 35 percent of an enterprise’s software budget is spent on maintaining the multitude of point-to-poin t application links already in place. — Gartner Group


Real World Transformation DemandsDemand a Broad Range of Capabilities ProcessingProcessing

Easy/Simple

Hard/Complex

Almost all projects that require these today, do so via programming, even if they are using middleware as the platform (ie Java, C++, etc)

Manyto

Many

Manyto

Any

AnyTo

ManyAny to Any

Interdependent Data

Only WTX can provide all of these……AND DOES SO WITHOUT CODING ! .

4


CASH RECONCILATION

10029847 - $100,000,000.0013948589 - $679,495,094.9813950967 - $588,345,058.0013950968 - $000,000,000.0014001321 - $098,957,038.12

AAA: HT4459AAA: B33566AAA: C4058GAAB: 948409AAB: 874931

What is the problem of Transformation

BINARY00011110010010011010010100100100100100100000111101010010101011001001010010010010010001010010100100101010101010001001110001010001001000100100100100100100101010010010101001000100100100100100100111000101001010101010101010010010101001001000100101001010000101010101000100101000100101001010100101010100101011001010000000000011111001001001010111001001001010101001010101101111

ãÖ@Ô…™ƒ�£–™z@Ã–

•¥…™£@”…@£–@ÅÂÃÄÉÃ

DUNS 0123 F046000INV for 2 months 12079912 718-339-1700I43989D-2 120799999-b 003000010000DUNS 4445 P55590 120799499QR 000004004000

OH,257*IN,142*MI,154*WI,80*MT,5*ID,8*WY,3*CO,21*NM,8*AZ,15*UT,13*NV,4*MN,48*MO,67*ND,9

*SD,9*KS,27

2984782q, 93847920, 3438084

34890PrefectFord

2984782q, 93847920, 3438084

34890PrefectFord

2984782q, 93847920, 3438084

34890PrefectFord

2984782q, 93847920, 3438084

34890PrefectFord

2984782q, 93847920, 3438084

34890PrefectFord

2984782q, 93847920, 3438084

34890PrefectFord

Extended_Features

PKGModelMake01 TP-API-CB.03 TP-AGAPI-CB.

05 TP-AGAPI-REQUEST PIC X(40).88 TP-AGAPI-INITIALIZE-REQUEST

VALUE 'INITIALIZE-MAPPING'.88 TP-AGAPI-PERFORM-MAPPING

VALUE 'PERFORM-MAPPING'.88 TP-AGAPI-FINISH-MAPPING

VALUE 'FINISH-MAPPING'.05 TP-AGAPI-VERSION PIC X(04).

88 TP-AGAPI-VERSION-VALID VALUES ARE '0100' '0200'.88 TP-AGAPI-VERSION-0100 VALUE '0100'.88 TP-AGAPI-VERSION-0200 VALUE '0200'.

05 TP-AGAPI-RESPONSE.10 TP-AGAPI-RESPONSE-CODE PIC 9(04) COMP.

88 TP-AGAPI-ALL-OKAY VALUE 0.88 TP-AGAPI-REQUEST-ERROR VALUE 1.88 TP-AGAPI-INITIALIZE-ERROR VALUE 2.88 TP-AGAPI-MAP-ERROR VALUE 3.88 TP-AGAPI-FINISH-ERROR VALUE 4.88 TP-AGAPI-UNKNOWN-LOOP-ID VALUE 5.88 TP-AGAPI-NO-ALGORITHM VALUE 6.88 TP-AGAPI-NO-PARTNER VALUES 7 15.88 TP-AGAPI-NO-APPLICATION VALUE 8.88 TP-AGAPI-ALGORITHM-IO-ERROR VALUE 9.88 TP-AGAPI-FATAL-GATEWAY-ERROR VALUE 10.88 TP-AGAPI-GATEWAY-WRITE-ERROR VALUE 11.88 TP-AGAPI-PARTNER-IO-ERROR VALUE 12.88 TP-AGAPI-BAD-VERSION VALUE 13.88 TP-AGAPI-BAD-NUMERIC-TYPE VALUE 14.88 TP-AGAPI-NO-ALIAS VALUE 15.88 TP-AGAPI-ACCESS-ERROR VALUE 16.

10 TP-AGAPI-RESPONSE-MESSAGE PIC X(80).05 TP-AGAPI-APPLICATION-ID PIC X(10).

0301

TABLE COPYBOOK

STREAM PROPRIETARY EBCDIC

Processing Information with all the business rules and usage mechanisms in tact…

<MSG 19934749><ACCT BAL RPT>

<ACCTNUM><“14001321”><ENDBALANCE><” $098,957,038.12”><DEBIT><“103048382$394,394.00”><DEBIT><“103048383$001,293.65”><CREDIT><“987463921$928,943.67”>

</ACCTNUM>

</MSG>

413 msgs

27 files

DDA Application Updates


What Makes WebSphere TX so Different?

�Code-Free Design and Deployment

� In-Process Data Validation

�High Throughput Execution of Complex Transformations and Enhancements

�One Engine, Multiple Deployment Options

5


Examples of Complex Transformation Challenges Addressed by WebSphere TX

� Data Enhancement�Lookups

�Data Logic and Routing

�Data Validation�Context Based Data and Usage Rules

� Many to Many Transformation�Single-Transaction, interdependent data sets, conversions and logic

�Mixed Data and Source/Target Types

�Dependent Result Sets, Nested Structure Dependencies

� Complex Data Transformation�Nested, Semi-structured And Hierachical Data Types�Dependent Inputs And Outputs

�Binary, Packed, EBCDIC, ASCII, Mixed Character Data


One Engine, Multiple Deployment Options

� WebSphere TX is the one transformation engine that can be reusedacross all applications and infrastructures in the enterprise �Deploy the same WebSphere transformation asset across:

� ESB

� SOA

� Web Services

� SOAP

� EJB

� HTTP

� Event driven

� Batch

� Java

� C/C++

� COBOL

� Unix

� Linux

� Windows

� z/OS

– Batch

– CICS

– IMS

– USS

� 30+ Adapters

� ETL

� Data quality

� Enterprise Applications

� Application Servers

� WebSphere Message Broker

� WebSphere ESB

� WebSphere Process Server

� WebSphere Partner Gateway

� WebSphere Application Server

� WebSphere DataStage

6


WebSphere TX is further enhanced by Vertical Product Packs

Financial Services Health Care

�SWIFTNet�SWIFTNet Funds�SEPA*�Many Services Based Solutions

�FIX�FedWire, CHIPs�NACHA�BAI, BAI2�AL3�ACORD�Etc.

�HIPAA�NCPDP�HL7�Many Services Based Solutions

�HealthCare Hub�Payment Processing�Clearing House Processing

CPG, Manufacturing

�ERP/CRM Integration�EDI (ANSI X12)�TRADACOMS�ODETTE�EDIFACT�Many Services Based Solutions

Embedded Software

�Integration to the Enterprise as an embedded service�Telecom�Enterprise Apps�Shop Floor Systems, etc…


WebSphere TX –A Long And Storied Road

Jan 1994

V1.0

June 1995

V1.1

Nov 1995

V1.2

June 1996

V1.3

Aug 1997

V1.4

Dec 1999

V2.1

June 2000

V5.0

Sept 2001

V6.0

Jan 2002

V6.5

Jan 2003

V6.7

June 2004

V7.5

July 2005

V8.0

…

V…

Proven

Reliable

Mature

Innovative

Q3 2006

V8.1

7


Global Financial Institutions rely on WebSphere Transformation Extender

� Fortune 1000 Coverage� 9 of Top 10 Commercial Banks� 6 of Top 15 Diversified Financials

� 4 of Top 12 Life and Health Insurance Companies

� 8 of Top 14 Property and Casualty Insurance

� Significant Impact on Capital Markets:� $3 Trillion of transactions flow through

WebSphere TX daily� 40% of London securities transactions are

powered by WebSphere TX

� WebSphere TX for STP:� 100 international banks

� 70 investment managers� 40 broker dealers� 10 global custodians

� Major Stock Exchanges� European Clearing Houses


MANUFACTURING, RETAIL & DISTRIBUTIONFINANCIAL SERVICES

HEALTHCARE / INSURANCE

WebSphere TX is widely used in all industries

8


� Transform, validate, and enrich any document, message or complex data

� Deliver trustworthy information for critical business initiatives

� Meet regulatory compliance requirements

� Codeless development; universal reuse and deployment

WebSphere TX: Universal Transformation & Reuse

“Make Advanced ESB More Advanced”

� Embedded� Standalone � Event driven

MAP TRANSFORM

Advanced ESBESB

J2EE

App. ServerProcess Server

Standalone System z (native)

Windows, UNIX, Linux

Multi-In Multi-Out


For batch environments(as a standalone engine)

For WebSphere Process Server

For WebSphere ESB

WebSphere TX

WebSphere TX

MAP

TRANSFORM

Unifying Enterprise Data Transformation

Unifying Enterprise Data Transformation

WebSphere TX: A Unified Transformation Environment across the WebSphere portfolio

� ONE Transformation Engine for the Enterprise!�Powerful transform

capabilities without coding

�Natively handles any data type

�Solves really hard transformation problems

� Complements existing portfolio of products

� Delivers new sales opportunities

For WebSphere Message Broker

For 3 rd Party Integration Products(as an embeddable engine)

For WebSphere Partner Gateway

9

®

IBM Software Group


Developing a WebSphere TX Solution


Adapter Type Tree

The Components of a WebSphere TX Solution

� A WebSphere TX solution is composed of the following:

Rules Type Tree Adapter

MappingSource Destination

� In other words…�An adapter is a technical connector which can be

“plugged” into a source or destination - without parsing!�A Type Tree is a graphical representation of meta-data�Rules define how a Source structure is transformed to

a Destination structure� Together these components form a WebSphere TX Map

10


Adapter Type Tree

Adapter Type Tree

Adapter Type Tree

� However…many to many mapping is also possible:

Rules

MappingSources Destinations

� A Map can perform many tasks :� Validation� Enrichment, � Content Based Routing� etc.

Type Tree

Type Tree

Adapter

Adapter

SOURCE FILETRANSFORMED MESSAGE

ON OUTPUT

CROSS-REFERENCE

TABLE

DATABASE UPDATE

The Components of a WebSphere TX Solution


Orchestrating Transformations

11


Developing a WebSphere TX Solution

� We usually talk about three simple steps:

�Describe the data...

� Data that is handled by IBM WebSphere TX, are described in "Type Trees”

that define the data structure and semantics.

�Transform the data...

� Data is mapped between the source and destination using drag-and-drop.

�Deploy the and execute the transformation

� Multiple deployment models

� Batch

� Event Driven

� Tightly-coupled (through APIs)

� Loosely-coupled (RMI)


WebSphere TX Design Environment

• Flows• Events• Monitoring

Describe the Data

Interfaces Resources

Define Transformation Rules

Deploy and Manage

• Properties• Structure• Validation Rules

• Structure• Formats• Semantics

• Communication• Adapters

Type Designer

Database InterfaceDesigner

Map Designer

MessageBroker

Launcher

Batch / JCLIMS CICS

12


Describe the Data...

� IBM WebSphere TX Type Trees� The Type Trees defines the structure of the data to be handled in IBM WebSphere TX.� This XML structure is then represented as an IBM WebSphere TX Type Tree.


WebSphere TX Design Environment

• Flows• Events• Monitoring

Describe the Data

Interfaces Resources

Define Transformation Rules

Deploy and Manage

• Properties• Structure• Validation Rules

• Structure• Formats• Semantics

• Communication• Adapters

Type Designer

Database InterfaceDesigner

Map Designer

MessageBroker

Launcher

Batch / JCLIMS CICS

13


Map Designer� Create maps to:

�Validate data from data sources

�Identify data targets

�Specify data transformation logic (rules)


Identifying Data Sources and Targets

Identify type tree and type tree components that define

data target

Identify type tree and type tree components that define

data target

Identify type of data and location of data

Identify type of data and location of data

14


WebSphere TX Adapters

WebSphere TX

Design Studio

Communication✔CICS *✔COM✔CORBA✔E-MAIL *✔FTP/FTPS *✔HTTP/HTTPS✔JAVA Class✔JCA Gateway✔SOAP✔SOCKET✔VAN

ApplicationsSiebel- BIM, EIM, OI COM

✔SAP R/3 & BW- ALE, BDC, DMI, DXOB- BW, EDI, BAPI

✔PeopleSoft- Message Agent API, - Open Query, EDI

✔QualityStage

Security✔S/MIME✔OpenPGP✔SSL Module

Database✔DB2 *✔Informix✔ODBC *✔OLE-DB✔Oracle *✔Microsoft SQL Server✔Sybase✔HP NonStop SQL/MX & SQL/MP

Messaging✔BEA MessageQ & Tuxedo✔Candle Roma✔IBM WebSphere MQ *

(MQSeries)✔JMS✔Microsoft MSMQ✔Oracle AQ✔TIBCO Rendezvous

Utility✔Archive (ZIP/TAR)✔Base 64✔Batch✔File *✔GZIP/ZLIB✔JNDI✔LDAP✔MIME✔Quoted-Printable✔Shell Script✔Sink *✔SNMP✔XML *

* Adapters supported on z/OS, varies by execution method


Step 1 – Define the data structures

� Pre-defined Importers for automatic generation

� XML Schema, DTDs

� COBOL Copybooks

� Database (Tables, Vues, Procédures Stockées)

� 3rd Party Applications

� SAP: BAPI, Idoc, DXOB, BDC

� PeopleSoft 7: Business Document, Message Agent

� PeopleSoft 8: Application Messaging,

Component Interface

� Siebel: COM Business Object, EAI, EIM

� WSDL (Web Services)

� EJB API

� Java Class

� Java Messaging Service

� Others: Tuxedo, Flat File, PL1, …

• Pre-defined libraries (Packs)

– EDIFACT

– SWIFT

– SEPA*

– ACORD

– HIPAA

– HL7

* En cours

15


Built-in Validation (Restrictions)

Items have extensive validation options

• Enumerations• Exclusions

(with substitution)• Patterns• Ranges


Additional Built-in Validation

� Component Rules allow cross-field validation

� Incorporates semantic validation directly into the parsing process

16


Additional Built-in Validation

� Component Rules support aggregate functions

� Use component rules to specify conditions (business rules) that must be met for a particular component to be valid


Step 2 - Creating transformation rules

• Drag and Drop

• No Procedural Coding!!

• Easily represent complex business rules

Pre-defined

Mapping

Functions

17


Calling external functions and programs


Rejecting Errors

18


Step 3 – Deploy and execute the transformation map� Option “Build” creates a

compiled map

�For the execution server

�Automatically accounts for “byte-order” and code page differences


Z/OS

Design Studio

Map

Importer

Importer

XML Schema

COBOL Copybook

000520****************************************************************

000530 01 TP-API-CB.

000540 03 TP-AGAPI-CB.

000550

000560 05 TP-AGAPI-REQUEST PIC X(40).

000570 88 TP-AGAPI-INITIALIZE-REQUEST

000580 VALUE 'INITIALIZE-MAPPING'.

000590 88 TP-AGAPI-PERFORM-MAPPING

000600 VALUE 'PERFORM-MAPPING'.

000610 88 TP-AGAPI-FINISH-MAPPING

000620 VALUE 'FINISH-MAPPING'.

000640 05 TP-AGAPI-VERSION PIC X(04).

000650 88 TP-AGAPI-VERSION-VALID VALUES ARE '0100'

000660 '0200'.

000670 88 TP-AGAPI-VERSION-0100 VALUE '0100'.

000680 88 TP-AGAPI-VERSION-0200 VALUE '0200'.

Windows

WTX Adapters

Deploy

Multiple Execution Methods

� API

� Batch / JCL

� Event-Driven / ESB� WebSphere Message Broker

� WebSphere ESB

� WebSphere Process Server

� WebSphere Partner Gateway

� WebSphere Application Server

� WebSphere DataStage, etc.

Example – Developing a WebSphere TX Map

1. Define the Data Structures

3. Deploy and execute

2. Define the transformation rules

Serveur WTX

(moteur)

19


Server A

WebSphere TX General Architecture

WTX Core EngineWTX Adapters

WTX Compiled Maps (.mmc)

WebSphere TX Server

MapDesigner

Unit Test Engine

Design Studio

Type Designer

Integration Flow Designer Importaters Database

Interface Designer

®

IBM Software Group


WTX for z/OS

20


� Transformation and execution on z/OSWTX for z

�APIs & Interfaces for: Cobol, Java, C, EJB

�For use with WAS, WPS, Cobol programs, etc.

WebSphere Transformation Extender(embedded edition)

�WTX Parser

�WTX Map Node

WebSphere Transformation Extender for Message Broker USS Edition

Execution MethodProduct Offering

�Event-driven, multi-threaded, multi-process transformation engine

WebSphere Transformation Extenderwith Launcher USS Edition

�Batch / JCL�CICS

�IMS TM

�IMS/DC

WebSphere Transformation Extenderwith Command Server


WTX for Z/OS - Batch� « Pure » Batch execution mode invoked through JCL

� SMP/E installation

� Supports the following sources/targets natively�Files (QSAM, VSAM)

�DB2

�FTP

�MQ

� EXIT function allows existing COBOL and C programs to be called from a WTX Map

// SET WTXLIB=L333703.WTX.REL8.XPL.LOAD <= WTX LOAD LIB

//DSTX EXEC PGM=DSTX,REGION=0M,// PARM='SEPA2APP /IF1 INPTSEPA’'//STEPLIB DD DISP=SHR,DSN=&WTXLIB

21


WTX for Z/OS – Batch - continued� Administration through standard z tools (ISPF, etc.)

�No need to learn a new monitoring tool

� No permanent active processes�WTX is only active when called by the JCL

�When map transformation endsWTX is no longer active

� Integrates seamlessly into existingproduction procedures�Does not require changes to customer

job and system administration processes


WTX for Z/OS - CICS� Transactional execution mode through a CICS transactio n

� Runs natively in CICS – inline and batch modes supported

� SMP/E installation

� Supports the following sources/targets natively

� VSAM Files

� key-sequenced

� entry-sequenced

� relative-record data sets

� DB2

� MQ

� Transient data queues

� Temporary storage queues

� Storage buffers (COMMAREA)

� EXIT function allows existing COBOL and C programs to becalled from a WTX Map

CICSAppli

WTXCICS

Z/OS

WTX Map

Guaranteed

Transactio

nCICS

22


WTX for Z/OS – CICS - continued

� WTX z/OS CICS can be invoked:

�From a clear terminal by using the DSTX transaction

� i.e. DSTX MYMAP –IF1 INPUTF

�From a user program by starting the DSTX transaction in the EXEC CICS START command

�From a trigger transient data queue by initiating the DSTX transaction

�By calling the DSTXCICS program through an EXEC CICS LINK

� Administration through standard z tools (CEBR, etc.)

� Integrates seamlessly into existing production procedures


WTX for Z/OS – IMS TM & IMS/DC� Allows WTX Maps to be programmatically executed through IMS

� Relies on WTX Platform APIs and high-level language programs

� C/C++ and COBOL programs can invoke WTX through the IMS execution option

� WTX Maps participate in the IMS transaction

� EXIT function allows existing COBOL and C programs to be calledfrom a WTX Map

Cobol/C Application

Z/OS

IMS

WTX

23

®

IBM Software Group


WTX for WebSphere Message Broker


WebSphere Message Broker

XSLTtransformation

Compute ESQL

transformation

WebSphere TX

Compute Java

transformation

Mappingtransformation

“The most powerful Any-to-Any

Broker on the Market”Industry Packs

(SWIFT, EDI, HIPAA) optional

UniversalTransformation

WebSphere Transformation Extender for WMB USS Edition

� Embed WebSphere TX transformations within Message Broker Message Flows

� Interoperates with Message Broker V5.x or 6.0

� Fully compatible with Message Broker transaction control

� Takes full advantage of Message Broker scalability

� Full support for multiple sources and multiple targets within a single transaction scope

� Installs in minutes

� WebSphere TX is an additional Node on the Message Broker design palette

24


WTX for WebSphere Message Broker Components

� Parser�Transforms bit streams to Message Trees (MQInput)

and Message Trees to bit streams (MQOutput)

�Transforms Message Trees to Message Trees (RCD)

� Plug-in node�Transforms anything to anything

�Icon installs into Message Flow palette

�Easy configuration


WTX for WebSphere Message Broker Plug-In Node

Any to Any TransformationsEDI TransformationsSWIFT Transformations

The Plugin node can be used in message flow like any other node

25


Chief Value Differentiators of WebSphere TX to alternative transformation methods

� Powerful Transform capabilities with No Coding�Lowers cost to implement, maintain and re-use�High-Throughput, Complex Transformations�Cost per transaction very low

� Natively Handles Any Data Type�Never resort to “flattening” or “re-structuring data”� Data integrity and meaning are never lost

� In Process Data Validation�Greatly reduces development efforts and implementation costs

� Solve Really Hard problems, in less time, with one common design method


Merci

Grazie

Gracias

Obrigado

Danke

Japanese

French

Russian

German

Italian

Spanish

Brazilian Portuguese

Arabic

Traditional Chinese

Simplified Chinese

Thai

websphere transformation extender - ibm · pdf filewebsphere datastage. 6 ibm software group |...

Documents