®

IBM Software Group

© 2005 IBM Corporation

November, 2006

SOA and WebServices Enablement of WebSphereTransformation Extender Maps Using WebSphereMessage Broker

Gerrit_detroyer@be.ibm.com

WebSphere Software

2

Agenda

�WebSphere Message Broker highlights

�WebSphere Transformation Extender (TX) Product Overview

�Developing a Solution Using TX

�WebSphere TX for Message Broker

�Use case: WebService enablement of a TX map using WebSphere Message Broker

®

IBM Software Group

© 2005 IBM Corporation

November, 2006

WebSphere Message Broker Highlights

WebSphere Software

4

Apps &

Info Assets

Business Innovation & Optimization Services

Development

Services

Interaction Services Process Services Information Services

Partner Services Business App Services Access Services

Integrated

environment

for design

and creation

of solution

assets

Manage and

secure

services,

applications

&

resources

Facilitates better decision-making with

real-time business information

Enables collaboration

between people,

processes & information

Orchestrate and

automate business

processes

Manages diverse data

and content in a

unified manner

Connect with trading

partners

Build on a robust,

scaleable, and secure

services environment

Facilitates interactions

with existing information

and application assets

ESBFacilitates communication between services

IT Service

Management

Infrastructure Services

Optimizes throughput, availability

and performance

WebSphere Process Server

WebSphere PortalWebSphere Information

Server

WebSphere Application Server

WebSphere Partner Gateway

WebSphere Adapters

WebSphere XD

WebSphere Message Broker

WebSphereTransformation

Extender for Message Broker

IBM SOA reference Architecture

ENTERPRISE SERVICE BUS

WebSphere Software

5

What is an Enterprise Service Bus?

An Enterprise Service Bus (ESB) is a flexible connectivity infrastructure for integrating applications and services.

An ESB powers your SOA by reducing the number, size, and complexity of interfaces.

Shape = Protocol

Color = Data type

An ESB performs the following

between requestor and service

• CONVERTING transport protocols between requestor

and service

• HANDLING business events from disparate sources

• TRANSFORMING message formats between requestor

and service

• ROUTING messages between services

WebSphere Software

6

WebSphere ESB and WebSphere Message Broker

ESB:

WebSphere ESB

Advanced ESB:

WebSphere Message Broker

Customers face a range of ESB requirements. As a result, any given

project might require an ESB or an Advanced ESB… OR BOTH.

Web Services connectivityand data transformation

Web Services

JMS

WebSphere MQ

HTTP

XML

WebSphere Adapters

JMS

TIBCO Rendezvous®

Web Services

Weblogic JMS®

WebSphere MQ

TIBCO EMS JMS®

HTTP

SonicMQ JMS®

Biztalk®

MQe Tuxedo®

Real-time IP

FTPMulticast

MQTT

XML

COBOLCopybook

EDI-X.12

ACORD

EDI-FACT

ebXML

AL3

HIPAA HL7

SWIFT FIX Custom Formats

Word/Excel/PDF

Plus the following:

WebSphereAdapters

Universal connectivity and data transformation

WebSphere Software

7

WebSphere Message Broker Overview

Message comesIn from

application/service client

ParseTo tree structureRoute + Transform

Parse

Message reachesDestination(s)

WebSphere Software

8

Message Formats in WebSphere Message Broker

� XML (with or without namespace support)

� Fixed-length Formats (COBOL, C, …)

� Tag Delimited Formats (CSV…)

� SAP iDOCs

� Binary

� EDI and industry standards

�HIPAA

�EDIFACT

�HL7

�ACORD

�SWIFT

�EDI-X12

�…

WebSphere Software

9

Message Broker Techniques For Transforming Messages

� Scripting via ESQL

� Scripting via Java

� Visual Drag & Drop Mapping

� Invocation of a Java Method

� XSLT for XML to XML

� WTX node (see later)

� Transactional database interactions

WebSphere Software

10

Message Sets

- Tell the broker how to parse a bitstring to an internal tree structure and vice versa

- Decouple the technical format from the business meaning of the data

- Allow parsing of all kinds of formats, far beyond XML

WebSphere Software

11

Message Sets (Ctd)

WebSphere Software

12

Creating the flows…

WebSphere Software

13

Message Flow overview: some I/O nodes

WebSphere Software

14

Some Non-Functional characteristics� Transactionality

�XA compliant resource coördinator

�z/OS: distributed transactions including message flow and a CICS transaction

� Performance & Scaleability�WebSphere Message Broker is designed to deliver performance…

� Highly performing processes

� possibility to create multiple Execution Groups (Processes) per broker

� possibility to process flows in Multiple parallel threads

� possibility to place multiple brokers in parallel

�Automatically scales to the # of CPUs available

� DRP & HASeveral scenario’s are possible:

� Parallel Sysplex (z/OS)

� HACMP

� MQ Clustering / Http loadbalancing

®

IBM Software Group

© 2005 IBM Corporation

November, 2006

WebSphere TX Product Overview

WebSphere Software

16

A little history on Transformation Extender…

� 1983, TSI International

� Industry’s strongest transformation engine

� Strong MQSeries Partner

� Ascential acquired Mercator in 2003

� Renamed to DataStage TX

� IBM Acquired Ascential (2005)

� Renamed to WebSphere Transformation Extender

� Universal advanced transformation engine throughout the WebSphere portfolio

1983

2003

2005-2006

WebSphere Software

17

WebSphere TX

Input Card(s) Output Card(s)

MAP

TRANSFORM

Adapters

WebSphere Transformation Extender: A Powerful and Portable Transformation Environment

Native O/S

Embedded On IBM & Others

Executeson:

Through Enterprise Service

Bus

WebSphere Message Broker

WebSphere TXLauncher

Extending Process Server

Extending ESBWebSphere ESB

WebSphere ProcessServer

Available Now

Available Now

Available Now

WebSphere Software

18

WebSphere Transformation Extender is a strategic solution for our customers

� On average, over US$ 800B* in transactions flows through WebSphere TX on a daily basis�4 Major Stock exchanges

�9 of 10 Major US Financial Firms

�75% of Blue Cross / Blue Shield claims

�6 of 10 Major Insurance Firms

�3 of 3 Top US Pharmacy Benefits Providers

�6 of 6 Top Global Banks

�6 of 10 Top US Banks, 4 of 5 UK Top UK Banks

�Major Global and National Telecoms

�Over 95% of all US Department of Defense logistic transactions

�and on and on and on….

* $800B is based on available, tangible measurements. Actual estimates are in excess of $2.7T /day

WebSphere Software

19

MANUFACTURING, RETAIL & DISTRIBUTION

FINANCIAL SERVICESHEALTHCARE /

INSURANCE

WebSphere TX has significant Vertical Marketshare

®

IBM Software Group

© 2005 IBM Corporation

November, 2006

Developing A Solution Using WebSphere TX

WebSphere Software

21

RuntimeMap

Designer

� Map Data Structures

� Defines Transformations

� Attach Adapters

Input Card(s) Output Card(s)

MAP

TRANSFORM

Type Designer

� Define Data

Structure

R

Tree

Importer

Database

Interface

Designer

Tree

Maker

Predefined

Trees

Application

Templates

Map

Source

Compiled

Map

The IBM WTX Integration Environment

DEPLOY

� Standalone

� Embedded

� WebSphere MB

WebSphere Software

22

Developing a WTX Solution

� We usually talk about three simple steps:

�Describe the data...

� Data that is handled by IBM WebSphere TX, are described in "Type Trees”

that define the data structure and semantics.

�Transform the data...

� Data is mapped between the source and destination using drag-and-drop.

�Deploy the transformation

Multiple deployment models:

Batch

Standalone Event Driven TX (installed base)

Tightly-coupled (MB Plug-in, Java API)

Loosely-coupled (RMI)

WebSphere Software

23

Describe the Data...

� IBM WebSphere TX’s Type Trees� The Type Trees defines the structure of the data to be handled in IBM WebSphere TX.

� This XML structure is then translated into an IBM WebSphere TX Type Tree.

WebSphere Software

24

Integration: Metadata� Automatic data definitions: Importers

� XML DTDs, Schema

� COBOL Copybooks

� Database Structures

� Application Definitions

� SAP: BAPI, Idoc, DXOB, BDC

� PeopleSoft 7: Business Document, Message Agent

� PeopleSoft 8: Application Messaging, Component

Interface

� Siebel: COM Business Object, EAI, EIM

� WSDL (Web Services)

� EJB API

� Java Class

� Java Messaging Service

� Other: Tuxedo, text files

• Built-in data definitions

– EDI, HL7, SWIFT,

ACORD, …

– All definitions are

modifiable through GUI

WebSphere Software

25

WTX Industry Accelerator Packs

WTX (any of the server types)

WTX Industry PACK’s

EDI

EDIFACT

TRADACOMS

ODETTE

SWIFT Net FIN

SWIFT Net Funds

*SEPA

*ACORD

*NACHA

HIPAA

HL7

NCPDP

* Available today as an ISSW offering

WebSphere Software

26

WTX (any of the server types)

WTX Enterprise Application PACK’s

SAP R/3

SAP XI

SiebelPeopleSoft

WTX Enterprise Application Accelerator Packs

Includes BAPI, ALE and IDOC integration capabilitiesRuns inside SAP App Server

Includes BAPI, ALE and IDOC integration capabilities

WebSphere Software

27

Built-in Validation (Restrictions)

Items have extensive validation options

• Enumerations• Exclusions (with substitution)

• Patterns• Ranges

WebSphere Software

28

Additional Built-in Validation

� Component Rules allow cross-field validation

� Incorporates semantic validation directly into the parsing process

WebSphere Software

29

Additional Built-in Validation

� Component Rules support aggregate functions

WebSphere Software

30

Transform the Data...

� The Type Trees for input and output are added to IBM WebSphere TX

Map Designer.

� For each input and output, or “card”, an adapter and a data source are defined.

� IBM WebSphere TX can handle several inputs to one or more outputs, and vice-versa.

� The mapping is a simple drag and drop from the input source to the output target.

� For each field in the output card, rules can be added--for example, mathematical formulas, conversions, etc. More than 150 different rules, similar to MS Excel formulas, are available through a self-describing wizard.

WebSphere Software

31

Sample mapping in IBM WebSphere TX’s Map Designer � A sample mapping in Map Designer:

WebSphere Software

32

• Drag and Drop

• No Procedural Coding!!

• Easily represent complex business rules

Enter transformation rules

WebSphere Software

33

Deploying and running the maps

� IBM WebSphere Transformation Extender

The core transformation engine is embedded within C, Cobol, Java applications though

the APIs which come with the product.

� IBM WebSphere Transformation Extender with Command Server

Suitable for scheduling and executing batch file transformation from command line, Unix

script and JCL Integration

� IBM WebSphere Transformation Extender for Message Broker

TX maps can be embedded in WebSphere Message Broker flows.

� IBM WebSphere Transformation Extender for ... (ESB, Process Server, Partner

Gateway)

®

IBM Software Group

© 2005 IBM Corporation

November, 2006

WebSphere Transformation Extender For Message Broker

WebSphere Software

35

WMB Plug-in components

� Parser

�Transforms bit streams to Message Trees and Message Trees to bit streams

�WebSphere TX map can be used as a plugin parser in WebSphere Message Broker

�Used in MQInput node (Bitstream to tree structure) and MQOutput node (tree structure to bitstream)

�New Message Domain (DSTX/DTX)

� Plug-in node

�Transforms anything to anything

�Broker sends BLOB to DSTX and gets BLOB with map result back

� Icon installs into Message Flow palette

�Easy configuration

�Can make use of TX adapters

WebSphere Software

36

�Can be used to create a tree structure out of complex data structures.

� Tree structure can be accessed by ESQL or Java code

�Alternative for Custom written plug-in parsers

� TX map instead of coding c!

�Uses TX map which converts Complex data to XML (used by MQInput node

�Uses TX map which converts XML to Complex data (used by MQOutputnode)

WebSphere TX as a parser for WebSphere Message Broker

WebSphere Software

37

WebSphere TX as a parser for WebSphere Message Broker

Call TX map for parsing

WebSphere Software

38

WebSphere TX Plugin parser configuration

<MessageTypes>

<Type Name=“ZORDRS_01"

InputMapPath=“edi_xml.mmc" InputMapCard="1" OutputMapPath="xml_edi.mmc" OutputMapCard="1" />

</MessageTypes>

Message Type

� Create a Type Element in dstxwmqi.ini where the Name matches the Message Type (broker \bin directory)

� For MQInput nodes, InputMapPath attribute points to WebSphere TX map that transforms a bit stream to a Message Tree

� For MQOutput nodes, use OutputMapPath to point to WebSphere TXmap that transforms a Message Tree to a bit stream

DsTXmaps

WebSphere Software

39

Message Broker MQInput node to enable DsTXparser

Message Type

Plug-in name� Open properties of MQInput node

� Type DSTX intoMessage Domain

(not in dropdown yet)

� Message Type references type name in dstxwmqi.ini

WebSphere Software

40

WMB Parser Use Cases

� Convert any input stream to Message Broker logical tree

� Reuse existing ESQL

� Mix and match DSTX and standard parsers

� No need for custom written (c coded) Parsers

� Content Based Routing of very complex data structures

� TX map can be used as a plugin parser at the input node. It converts the complex data structure to the internal Message Broker logical tree.

�Message Broker performs the Content Based Routing

� TX map parses the logical tree back to Complex Data Structures

� This is just the beginning....

WebSphere Software

41

�Can be used to call any TX map from within a Message Broker flow

�Transaction Control of message broker is maintained

�Message Broker passes BLOB to TX map

�Plugin can pass BLOB back to message broker, or can parse to any other format: XML, MRM, ...

TX plugin node for WebSphere Message Broker

Call TX map for parsing

WebSphere Software

42

� Plugin node Installs into standard message flow pallete

�Plugin node can be used in message flow like any other node

TX plugin node for WebSphere Message Broker

WebSphere Software

43

WebSphere TX Plug-in Node Configuration

DSTX Map Name

� Open properties of Plug-in node

� Enter Map name and OutputCard

�Optionally, enter Output Domain, Set, Type, Format for MRM parsing�If these are empty, a BLOB is propagated

WebSphere Software

44

WMB Plug-in Node Use Cases

� Mapping between complex message formats� Message Broker native model (MRM) supports C, COBOL, XML and many tagged or string-delimited formats

� WTX adds support for custom formats that are hard to support with the MRM

� WTX EDI libraries are more comprehensive

� Rich Industry Support via WebSphere TX is fully leveragable in Message Broker

� Complex and Proprietary Data Types handled natively in TX

� Support for advanced transformation requirements� Multiple data sources with interdependencies

� Very efficient processing of large data records or messages

� Advanced data validation without complex coding

� Leveraging Message Broker’s Clustering capabilities for scalability� Deployment to multiple execution groups, brokers, machines

� Use of the TX adaptersTX offers it’s own set of adapters. Some of these adapters offer functionality which is not offered by any of the WBI adapters: (ZIP, …)

� File handling of very large files (up to 4 GB)TX is very fast in translating large files. The location of a file to be translated can be passed through by Message Broker to TX. TX finds the file and performs the translation which

� WebService enablement of existing TX maps

®

IBM Software Group

© 2005 IBM Corporation

November, 2006

Deploying WebSphere TX In An SOA

WebSphere Software

46

WebSphere TX can leverage the webservices capabilities of WebSphere Message Broker

WebSphere TX: powerful

Any-to-any mapping

WebSphere Message

Broker: powerful SOA

capabilities

•WSDL generation from Message Set

•Out-of-the- box SOAP message sets

•SOAP/JMS support •SOAP/HTTP(s) support •Webservice mode•WebService Client mode•...

WebSphere Software

47

WebSphere Message Broker: powerful SOA

capabilities

•WSDL generation from Message set

• Out-of-the- box SOAP message sets• SOAP/JMS •SOAP/HTTP•Webservice mode •WebService Client mode•...

WebSphere TX can leverage the webservices capabilities of WebSphere Message Broker

WebSphere TX: powerful

Any-to-any mapping

WebSphere Software

48

How to enable a DsTX map for WebServices using WMB example for SOAP/http

HTTPRequestInitial Processing:

-SOAP de-enveloping

-parse as BLOB

DSTX map

Final Processing:

-Parse as XML

-SOAP Enveloping

HTTPResponse

Web

Services

Client

SOAP/http

request

SOAP/http

response

WebSphere Software

49

WebService Enablement of DsTX maps: Howto

1. Read input

2. De-envelope and Parse content as BLOB

3. Pass BLOB to DsTX plugin

4. Parse output of Plugin as required and envelope

5. Reply

WebSphere Software

50

How to enable a DsTX map for WebServices using WMB - CASE

� A DsTX map (Fixed Format to XML) needs to be webservice enabled

� The map takes in non-xml data � needs encapsulation in XML to fit in a SOAP message

� If the map ends OK, the resulting XML data need to be passed back in the SOAP response

� If the map fails, a SOAP Fault needs to be returned, containing the Map error code thrown by DsTX

� Transport required is SOAP/http

� the map is designed to receive CRLF as a delimiter, but clients can send LF and we need to cope with that

WebSphere Software

51

The flow layout

Web Services Client

Strip SOAP

Parse asBLOB

MAPenvelope

reply

Read

WebSphere Software

52

What makes WebSphere TX Different?

One Engine,One Engine,

Multiple Multiple

Deployment Deployment

OptionsOptions

Ontological DataOntological Data

Model forModel for

All Data TypesAll Data Types

High High --throughputthroughput

Execution of ComplexExecution of Complex

Transforms andTransforms and

EnhancementsEnhancements

InIn--Process Process

Data ValidationData Validation CodeCode--FreeFree

Design andDesign and

DeploymentDeployment

WebSphere TXWebSphere TX

WebSphere Software

53

What value does WebSphere TX bring to WebSphere Message Broker users?

� Support for additional, complex message formats�Message Broker native model (MRM) supports C, COBOL, XML and many tagged or string-delimited formats

�WebSphere TX adds support for custom formats that are hard to support with the MRM

�WebSphere TX EDI libraries are more comprehensive

�Rich Industry Support via WebSphere TX is fully leveragable in Message Broker

�Complex and Proprietary Data Types handled natively in TX

� Support for advanced transformation requirements�Multiple data sources with interdependencies

�Very efficient processing of large data records or messages

�Advanced data validation without complex coding

� Support for TX adapters

� Large Data Size Processing�Messages/Files up to 4GB in size per transform

WebSphere Software

54

What value does WebSphere Message Broker bring to WebSphere TX users?� Delivers a powerful event processing framework for the transformation engine

�Extensible and open “message flow” framework integrates transformation with routing, validation, correlation and other rich event/message processing

�Complex Event Processing capabilities allow maps to be triggered by configurable patterns of event sequences

� Exploits the Enterprise Service Bus for effective data delivery and web service enablement

�Integrated environment for all aspects of transactional data movement in the enterprise

�Provides a complete application integration solution

�Leverages robust file transfer and messaging capabilities

�Allows easy inter-operation with ESB-enabled adapters and services

� Provides a robust, scalable environment for transformation services

�Ensures high availability and failover

�Exploits WMB clustering capabilities

WebSphere Software

55

Q & A

WebSphere Software

56

Enroll for the free Proof-Of-Technology!

� When ?

�06-12-2006 to 08-12-2006 08:30-17:00

� Where?

� IBM Forum Brussels

� Content?

�TX Lectures and hands-on lab exercises

�Usage of TX with and without WebSphere Message Broker

� Where to enroll?

�http://www-5.ibm.com/be/events/en/agenda.html

or

�http://www-5.ibm.com/be/events/en/middleware_appdev.html#362