Demystifying ESB patterns

Colin Gniel and David Arnold, November 2009

Agenda

• ESB – A Recap

• SOA Reference Architecture

• IBM ESB offerings

• IBM ESB Positioning in the SOA Market

– Best fit, Hybrid and “Federated ESB”

• What is a Best Fit ESB

• What is a Hybrid ESB

• What is a “Federated ESB”

Assets Layer of disjointed, poorly understood enterprise assets, preventing reuse

Architectures Tightly-coupled architectures hindering IT flexibility

Skills IT skills shortage and silos limiting staff productivity and mobility

Processes and toolsDuplicate processes, tools and infrastructures limiting collaboration

Investments Increasing maintenance costs limiting flexibility for new investments

E01-EDI

Data Warehouse(Interfaces to and from the

Data Warehouse are not

displayed on this diagram)

G02 - GeneralLedger

A05 - AP

S01 - SalesCorrections

I01 POReceiving

I03 Return toVendor

I06 WarehouseManagement

M a inframe apps - B lue

PC/NT apps - G reen

Unix apps - Y ellow

3rd party interface - Orange

Lines: Colors have no special meaning.They are to help make the diagram easier toread.For More Information: See the databasecontaining information about eachapplication: Application V4.mdb

S06 - Credit App

P15 EES EmployeeChange Notice

OTHER APPS - PC

AP - Collections/Credit

TM - Credit Card DB

ACCTS REC APPS - PC

990COR

Bad Debt

Benefical Fees

Beneficial Reconcil

JEAXF

JEBFA

JEBKA

JEDVA

JESOA

JEVSA

JEVSF

NSF

TeleCredit Fees

INVENTORY CONTROL APPS - PC

Code Alarm

Debit Receivings

Devo Sales

Display Inventory

In Home

Junkouts

Merchandise W ithdrawl

Promo Credits

RTV Accrual

Shrink

AP Research - Inv Cntrl

AP Research-Addl Rpts

Book to Perpetual Inventory

Close Out Reporting

Computer Intelligence Data

Count Corrections

Cross Ref for VCB Dnlds

Damage W rite Off

Debit Receivings

DFI Vendor Database

Display Inventory Reconcil

Display Inventory Reporting

INVENTORY CONTROL APPS - PCDPI/CPI

IC Batching

Inventory Adj/Count Correct

Inventory Control Reports

Inventory Levels

Inventory Roll

Merchandise W ithdrawl

Open Receivings

PI Count Results

PI Time Results from Inv

Price Protection

Sales Flash Reporting

Shrink Reporting

SKU Gross Margin

SKU Shrink Level Detail

USM

VCB Downloads

Journal Entry Tool Kit

Scorecard - HR

L02-ResourceScheduling(Campbell)

P09 - P17Cyborg

M02 - Millennium

M03 - Millennuim 3.0

Banks - ACH and Pos toPay

Cobra

B01 - StockStatus

S03-Polling

P14 On-line NewHire Entry

CTS

Plan Administrators(401K, PCS, Life,

Unicare, SolomonSmith Barney)

D01 Post LoadBilling

I04 HomeDeliveries

I02 -Transfers

Arthur Planning

I07 PurchaseOrder

I12 EntertainmentSoftware

I05Inventory Info

E13E3 Interface

S04 - Sales Posting

V01-Price ManagementSystem

I10 Cycle PhysicalInventory

I55 SKUInformation

K02Customer Repair

TrackingI35 Early Warning

System

B02 MerchandiseAnalysis

I13- AutoReplenishment

U18 - CTO

Intercept

I09 Cycle Counts

E02-EmployeePurchase

Texlon 3.5

ACH

Stock Options

I17 Customer PerceivedIn-Stock

U16-Texlon

SiteSeer

C02 - CapitalProjects

F06 - FixedAssets

US Bank ReconFile

Star Repair

EDICoordinator

Mesa Data

NEW Soundscan

NPD Group

AIG W arranty Guard

Resumix

Optika

Store BudgetReporting

P16 - Tally Sheet

Cash Receipts/Credit

S05 - HouseCharges

Ad Expense

L01-PromoAnalysis

V02-PriceMarketingSupport

BMP - Busperformance Mngt

StoreScorecard

I11 PriceTesting

Valley Media

P09Bonus/HR

I15 Hand ScanApps

Roadshow

POS

S08 - VertexSalesTax

A04 - CustRefund Chks

Equifax

ICMS Credit

CellularRollover

S09 - DigitalSatelliteSystem

NPD,SoundScan

Sterling VANMailbox (Value)

I18SKU Rep

X92-X96Host to AS400

Communication

S02 -Layaways

Washington,RGIS,

Ntl Bus Systems

V04-SignSystem

I14 Count CorrectionsNARM

P01-EmployeeMasterfile

I06 - CustomerOrder

FrickCo

UAR - Universal AccountReconcilliation

DepositoryBanks

S07 - CellPhones

S11 - ISPTracking

AAS

Fringe PO

Cash Over/Short

L60 MDFCoop SKU Selection

Tool

SKUPerformance

SupplierCompliance

1

I35 - CEI

ASIS

Misc Accounting/Finance Apps - PC/NTCOBA (Corp office Budget Assistant)PCBS(Profit Center Budget System)

Merchandising Budget

AIMSMerch Mngr Approval

Batch ForcastingAd Measurement

AIMS Admin

AIMSReportingAd

Launcher

V03- MktReactions

SpecSource

CTO2.Bestbuy.com

RebateTransfer

SignSystem

CopyWriter'sWorkspace

ELTPowerSuite

StoreMonitor

AIS Calendar

Stores & Mrkts

Due Dates

Smart Plus

InsertionsOrders

BudgetAnalysis Tool

Print CostingInvoice App

AIS Reports

BroadcastFilter

Smart PlusLauncher

GeneralMaintenance

Printer PO

PrinterMaintenance

VendorMaintenance

Vendor Setup

Connect 3

Connect 3Reports

Connect 3PDF Transfe

Spec SourceSKU Tracking

S20-SalesPolling

Prodigy

PSP

In-HomeRepair

WarrantyBillingSystem

Process Servers(Imaging)

Prepared by Michelle Mills

Page 1 of 2

Actual application architecture of an IBM customer

J2EE

Linux

Palm / PDABlackberry

SAP / Oracle

VB / C++

PackagedApps

Improving Business Agility and the challenge of integrating existing applications

System i

System z

CICS / IMS

Web 2.0

Softwarearcheology

“Legacy modernisation is morphing into a strategic function. IT can't afford to toss away reliable application transactions indiscriminately.“

-- Phil Murphy, Forrester Research, April 2007

Outsourced vendor

Packaged app providerAcquired dev team

Testing team

Core dev team

Regional SI

LOB Leaders

Web team

Architecture team

Java/ . Net

Enterprise Service Bus

An ESB enables flexible SOA connectivity for integrating business applications and processes as services

Converts between different transport protocols

ESB promotes Agile ConnectivityThe Enterprise Service Bus (ESB)

Matches & routes communications between services

Connects everything to everything

Distributes Business events

Transforms between different data formats

Communications Layer

ESBService Enrichment,

Composition, Mediation,Connection

• Match & Route communications between services

• Converts between transport protocols• Transforms between data formats• Identifies and distributes bus events

Communications Layer(Messaging)

• Connect everything in your SOA• Leverage existing skills and assets• Support business growth and agility• Deliver fast ROI

A Cross-sectional view of the ESB

SOA Foundation Reference Architecture

Business Application

Services

Strategy and Planning Services

Enterprise Service Bus

Access Services

Partner Services

Process Services

Information Services

Interaction Services

Business Services and Events

Lifecycle Services

Asset and Registry Services

Development Services

Management Services

Infrastructure Services

Enables collaboration between

people, processes & information

Business-driven Enterprise Architecture and Standards

Manages diverse data and

content in a unified manner

Integrated

environment for

design and

creation of solution

assets

Connect with trading partners Build on a robust, scaleable, and

secure services environment

Facilitate interactions with

existing information and

application assets

Manage and

secure services,

applications &

resources

Optimizes throughput, availability and utilization

Orchestrate and automate

business processes

Supports the specification of enterprise business

solutions through business architecture

WebSphere DatapowerWebSphere MB

WebSphere MQ

WebSphere ESBWSRR

ESB builds upon the Communications Layer

Only WebSphere delivers the most complete ESB solution

– Adds “awareness” of what the messages contain on the backbone

– Adds message mediation, message transformation and content-based routing

– Underpins BPM and Application Integration in an SOA

WebSphere MQ family Embedded Messaging

WebSphere DataPowerIntegration Appliance

Purpose-built hardware ESB

with simplified deployment

and hardened security

WebSphereMessage Broker

ESB for universal connectivity and

transformation in heterogeneous

IT environments

WebSphere ESBBuilt on WebSphere

Application Server for an

integrated SOA platform

Communications Layer

IBM ESB Positioning within the SOA Market

Enterprise Service Bus

An architectural pattern within Service Oriented

Architecture rather than a product.

In nearly all cases a recommended best practice.

Best Fit ESB

Hybrid ESB

“Federated ESB”

Best Fit ESB

• What is a Best Fit ESB

– Single technology (product)

– Maybe multiple instance for scaling and availability

– Often some compromise required

• Non-functionals: throughput, response time

• Functionals: extension through coding required

– Runtime registry interaction not necessarily required

• When a “Best Fit” ESB

– Size of organisation

– Skills availability

– Scope of requirements

– Single Service Domain

Best Fit – WebSphere Message BrokerWhen to Consider?

• You are an existing WebSphere Message Broker user

Leverage existing WMB skills, while migrating to 7.0

• You have extensive heterogeneous infrastructures including

both standard and non-standards-based

Applications

Protocols

Data formats

• You have extensive MQ skills and infrastructure

• You are using Industry formats such as SWIFT, EDI, ACORD

• You need extensive pre-built mediation support

• You are implementing more complex messaging and integration patterns

• You have non-XML or complex transformation needs

• Reliability and extensive transactional support are key requirements

• You need high performance with horizontal and vertical scaling

• You use WebSphere Application Server

– Your team has skills with WAS Administration and Java coding

• You are now or planning on developing business

processes using WebSphere Process Server

– Common tooling, programming model, and runtime

• You are integrating with ISV business applications hosted

on WAS or 3rd party solutions which extend and support

WAS

• You want to mediate between Web services and existing

systems using JMS and WebSphere JCA Adapters

• Reliability and extensive transactional support are key

requirements

• You want to minimize your server investment by co-

hosting WebSphere services and ESB in one application

server

Best Fit – WebSphere ESBWhen to Consider?

• Ease of use is a pre-dominant consideration

– Simple experience of drop-in installation and admin-based

configuration with no or minimal development required

• You are transforming between XML-and-XML or XML-

and-any other format

• You are using XML-based or WS-Security extensively

• Your interaction patterns are relatively simple

• You require use of advanced Web services standards

• Your mediation requirements are met by the existing DP

mediations and minimal extensibility is needed

• You need to minimize message latency when adding an

ESB layer

• You are doing extensive XML processing combined with

high performance requirements

• Your ESB must be in production very quickly

Best Fit – WebSphere DatapowerWhen to Consider?

Hybrid ESB

• What is a Hybrid ESB

– Also termed, “Multi-node” or “Distributed” ESB

– Connectivity centric patterns

– Engineered for each customer to provide the optimal SOA value and footprint – powerful but small.

– Focused on connectivity and runtime optimization

– All about “feeds and speeds”

– Multiple technologies (products) to build an ESB

– Little compromise on Functional or non-Functional requirements

– Runtime registry recommended

• When a Hybrid ESB

– Single Service Domain

– Broad range of functional requirements

– Challenging non-functional requirements

Technology1 Technology3

ESB

Technology2

1414

Total

Spend

Requests /

Second

Custom ESB

Other Vendor ESB

WebSphere Message Broker

Hybrid ESB (with

appliance

Hybrid Bus(with appliance) enables cost effective high performance

1

2

• DataPower: Extreme Performance,

processing of standard messages at the

highest possible rates, low resource

consumption

• Ease of configuration and deployment

• Perimeter Security

• Message Broker: Universal Connectivity,

higher value ESB functions requiring greater

flexibility and lightweight orchestration

• For proprietary and packaged application

integration requiring hosted adapters or

custom extensions

Hybrid ESB and B2B Gateway ServicesKey

Portal Cluster – Mobile Users

Mobility Extensions,

Portal Server

Web Content Services

Mobility Extensions

Portal Server

Web Content Services

Load Balancing WebSphere Partner Gateway

Load Balancing WMB Instances

Backend Systems

BRAVO WISNET Other

External Information

Providers

ReutersRace

CalendarOther

Load Balancing Datapower XI50s

WMQ

External Information

Providers

SkySocial

NetworkingOther

Security

Services

Mediation

Services

Legacy

B2B

Gateway

Services

Complex

Mediation and

Service

Composition

Industry

Support

Services

Protocol

Switching

Services

Pe

rsis

ten

ce

Complex

Integration

Patterns

Integration Layer

Gateway

Services

Integration

B2B

Integration/B2B

Portal Cluster – Web Users

Mobility Extensions,

Portal Server

Web Content Services

Mobility Extensions

Portal Server

Web Content Services

“Federated ESB” – Federation of Services

• What is a “Federated ESB”

– Multiple ESBs working together

Maybe a combination of Hybrid and Best Fit ESBs

– Focused on non-runtime optimization, governance, system management, monitoring and security.

– Runtime registry integral part of the ESB

– Pre-requisites

Federated security

Federated systems management

Federated Governance

• When a “Federated ESB”

– Multiple Service Domains

• Example: Domain per Business Unit

• Domain per Geography

– Shared Services across business units

– Existing ESB technologies in business units

– Existing registries in business units

Service Domain Service Domain

Service

Federation

Service Domain

ESB 1

ESB 2

ESB 3

ESBs providing the Federated Connectivity to enable the Federation of Services across

multiple Service Domains in an Enterprise.

Analyst Views 2007 – the Future is Federated

ESB Adoption Trends - Gartner

“80% of large companies will have ESBs or similar SOA infrastructure products from three or more vendors in 2009. (0.7 probability)”

“…half of all large companies will apply a systematic, federated approach to managing their disparate SOA domains and ESBs in 2009. (0.7 probability)”

– Roy Schulte, Gartner, “Succeeding With Multiple SOA Service Domains and Disparate ESBs”, May 2007

ESB Adoption Trends - Forrester

“Some of Forrester's most advanced clients got on the ESB bandwagon years ago and today find that they have an embarrassment of riches, with multiple ESBs. This outcome is almost inevitable for larger enterprises; no single ESB today can satisfy all requirements equally well across the whole of a large enterprise.”

– Larry Fulton, Forrester, “Shaping Your Middleware Strategy to Benefit from ESBs, May 2007

The Future is Here!

“FederatedESB”

ServiceSecurity Service

ManagementService

Governance

Service Federation – going beyond connectivity Which Links Together Multiple, Disparate Domains

A “Federated ESB” is an enterprise-spanning connectivity infrastructure of multiple ESBs working together to extend

service reuse across as well as within domains.

Domain

Domain

Domain

Federated Connectivity Patterns

• Peer to Peer (direct routing)

– Peers share services directly

– All domains are service providers and consumers

• Parent/child (direct routing)

– Children do not share services

– Parent is a provider of services

– Parent is a consumer of child services

– Children are consumers of parent services

• Brokered (centralized routing)

– Children share services via the brokering domain

– Parent and children are both providers and consumers of services

1 2

2

B

1

2

P

1

P P P

21 21 21

3

• Most, if not all, ESB architectures will include a Registry and Repository

– WSRR implementation maybe beyond first deployment phase

• Non-Runtime Considerations

– Lifecycle management (ex. Conformance) in ESB

– Service Cataloging

• Runtime

– Need runtime lookup of runtime service endpoint locations for dynamic routing

– Need to use service metadata at runtime –policy driven ESB

– Use real-time information about service use in the ESB provided via WSRR using ITCAM for SOA

• Key to Service Federation

When To Add A Service Registry to an ESBWhen to Consider WSRR?

Role of the registry in ESB Patterns

Best Fit Hybrid Federated

Development Recommended Highly

Recommended

Essential

Runtime Optional Recommended Highly

Recommended

Federation model (Service Connectivity Management)

Atom/APP based protocol used

to add, update, remove resources

Federation

Console

DomainFederation

Connectivity

Provider

Service

Group

Registry

Proxy

Service

Endpoints

SFM

Coordinator

• WSRR v7.0 includes product in support of this model

– Injection of service proxies into buses

– Projection of service proxy meta data into the appropriate

registry/registries

• Supported Registries

• WSRR,UDDI

• Support ESB products

– WMB and WESB (DP to come next year)

Service Domains, Service Providers and proxies

Proxies in consumer and/or provider domain allow:

Control points for policy enforcement

Identity mapping between trust domains

A’A’’ AA’

A’ AA’’

Provider domainConsumer domain

Service Registry Service Registry

ESBESB

Delegation Delegation