operationalizing soa – lessons learned

October 11th 2006 © Arun Majumdar and parts by permission, Andre Leclerc

Operationalizing SOA – Lessons Learned

Arun Majumdar(Cutter Consortium / VivoMind Intelligence Inc. )

Take Home Msg. : Multi-Level MDA & FOL


Agenda

• SOA has 3 basic Stages for Operationalization– re-engineering, data types (Ontology), process workflows

• Apply Incremental development and Semantics– Data model, information transfer model, – Transformation model– workflow model

• Lesson Learned: SSOA Meta-Model– An aspect-oriented architectural approach

Real World Case Studies


Case Studies

• A large US Gov. Consulting Firm (80 offices across USA, UK, Japan)

• Large TELCO (one of the big 5)

• Big Pharma. Company – Semantic SOA – 200 Subsidiaries & many product lines

Key Focus: Lessons Learned


A Large Consulting Company

SOA for Workflow Automation


Desires• Wanted to have a new, better, faster,

Contracting, Billing and Work delivery system.

• Wanted to reduce or remove manual low level menial tasks in the corporate business processes

• Use new technology for a web-based service façade.

• Wanted to have hot-swappable services with new unforeseen capabilities add-ons

• Remove Vendor Lock-Ins (Oracle, Siebel, BEA etc… ) and other grand dreams and desires …

Desire: To improve the bottom line


Client’s Situation #*%&!

Gap: How to get there from here


Client Situation – How it was

Situation: Processes, Activities, Workflows


Automation Tools: Semantic ETML

Strategy: Extract Legacy, Transform & Model


From Legacy to CG’s To Services• Focus on the transfer/transformation

mechanisms.

• Focus on Business Semantics and Ontologies using automation tools.

• We built our own tools using Popkin, VBA, C-code, Prolog (which we called “LegacyWorks”, “Intellitext” for parsing data to Conceptual Graphs ) : – http://www.cutter.com/architecture/fulltext/reports/

2002/09/index.html

Lesson: A Business Ontology


CG Transfer Architecture

CICS CMSQL

System

System

System

Observation: Businesses Do Lots of Transfers


CG Transformation Architecture

Drill Down: Transfers with Transformations


CG Business Process Architecture

Fusion: A System of Transfers and Transforms


Lessons

• Need good tool-integration platforms for automation. • Model and develop adaptors and wrappers using

Meta-Data (MD) • Choose a communications model to replicate the

business processes via MD.• Abstract away all the specific low level functions in

the system modules and focus on:– Transfers– Transformations

• Develop these into “Services”

Building: A Service Oriented Architecture


A Large Telco

SOA for Order Configuration Management


Desires

• Wanted to have a new, better, faster, Order Management System.

• Wanted to avoid Manual Knowledge Extraction (error prone, costly process)

• Use Semantic Web style technology for CLEC/ILEC/B2B/B2C … all interfaces

• Wanted to have hot-swappable services configurations with new unforeseen capabilities add-ons

• Remove Vendor Lock-Ins (Oracle, Siebel, BEA etc… ) and other grand dreams and desires …

Desire: Improve the bottom line


Desires

Desire: An Elegant Dream!


Client Situation

KnowledgeExtraction

AnalyzeExisting

Requirements

GenerateWorkflows Code

ASR (Access Service Request) Order Type Partitioning

ASR Type BusinessProcess and Rules

BuildWorkflowModels

Workflowand Data Models

The initial BEA WLI (WebLogic Integration) test required 2-3 months for just one simple (New Connect) workflow; but, there are hundreds of workflows, hence years of work!This approach requires developers to have expert level knowledge of BEA’s WLI; not possible in the timeframe of the project so hire experts from BEA; but Telco training has yet to begin. Limited programming resources and domain expertise mismatches cause delays.

Situation: Processes and Workflows


Reality

ASOG

WFMain Frame

NOS Common Services

TIRKSAAISDS

AAIScore

LW

NDM

CSG

UOM

Billing Common Services

TIRKSTIRKSTIRKS

CABS

Fatal ASRs are returned to the Customer throughthe same channel ASRs were received.

DBDataset

AWDWork Administration, Monitoring

and Distribution

AFMretrieves received

ASRs

AFM passesMechspec

files to AVS

AFMreceivesvalidated

ASRs

TELCOSpecific

BusinessRules

AFMrequestsVerizonSpecific

Validations


ASRs

Billing

AFMrequests

BillingValidations


ASRs

Facility

AFMrequestsFacilities

Validations


ASRs

ClarificationNotice

TELCO - Wholesale ASR Processes V1.0

FatalError

SoftError

Manual

No No

YesYes

Yes

Work Drivers

Soft Errors are returned to the Customer through thesame channel ASRs were received. (Clarification Form)

ClarificationNotice Generated

UI

Work Assignment

Distribute to Rep

If necessary, Repcreatesclarificationrequest orupdates andvalidates ASR

A

UI

Rep entersandvalidatesASR

TransactionalOrders

& Status

Order isreceived

Access Flow-through Manager (AFM) Workflow and Flow Through

AVS (VALIDATION)No

Address

AFMrequestsAddress

Validations


ASRs

RequestCreation

RequestData

AcceptClient Order

FormatRequest

StoreRequest

ValidateRequestSyntax

ValidateRequest

Semantics

Verify OrderData

ValidateOrder

FeasabilityB

DB+

JRULES

Fatal / ClarificationWork

AssignmentEdit

Master OMS Database

The OMS database is partitioned forspeed and throughput. The high-levelpartitions include:- Pre-production- In-process Production- Post-production (completed) orders.

Reality: Telephony is very complicated!


Reality

SOP

TIRKSAAISDS

BillingCommonServices

Facility

No

No

YesA

Service OrderNumber

Assignment

RequestNet (critical dates &facility reservation)

TELCO - Wholesale ASR Processes V1.0

AFM requestscritical dates.

IOF/OSPverify

facilities

Facility

Yes

IOF/OSPCreatesCriticalDates

IOF/OSPCreatesECCD

AFM receivescritical dates

Customer is sent CR indicating a lack of facilitieswithin 72 hours

Service OrderCreation

Customer is sent CN (FOC) specifying critical datesand circuit POI details

AWAS

NOS Common Services

TIRKSAAISDS

NSDB WFAC

End to End Status Auto-Completion

CompletionSet Archive Flag

Rep receives ASRsrequiring downstreamattention and escalations

Service OrderAssignment

Routing

Access Flow-through Manager (AFM) Workflow and Flow Through

CID Creation

Circuit ID

EscalationManagement

Pre-definedand adhocreporting

Reporting

CompletionSet Archive Flag

ReportingEscalationand

monitoring

CriticalDates

Assignment

FacilitiesReservation

Status EscalationCompletion

TIRKSTIRKSTIRKS

CABS

GenerateServiceOrder

Verify Order FeasibilityGenerateServiceOrder

InformClient of

Order ResultB

Firm OrderCommitment

GenerateBilling

Transaction

Implementation /Provisioning

ProactiveNotification

Customer is sent proactive notification indicatingchange of status

StatusTracking

Implementation &Provisioning of the order

SOPSOP

Master Order ManagementSystem Database

The OMS database is partitioned for speed andthroughput. The high-level partitions include:- Pre-production- In-process Production- Post-production (completed) orders.

Reality: This picture was developed in Visio™ !


Step 1: Work with Real DataORACLE PL/SQL ITEM UML MODEL ITEM WLI W/FLOW COMPONENT

procedure;PROCEDURE…END activity workflow step

statement block;BEGIN…END; activity workflow stepFOR LOOP …END activity workflow stepnext statement control flow step linkCALL control flow step link"implied by context" object flow underlying object

IF (condition) THEN … ENDIF decision decisioncondition of if statement decision condition

true branch of decisionbranch of decision (true condition) true branch of decision

false branch of decisionbranch of decision (false condition) false branch of decision

statement;block of statement;SP call activity action>> not directly supported<< send event in-coming event>> not directly supported<< receive event out-going eventSP call activity interface callSP call activity web service callSP call activity java class method call>> not directly supported<< branch from synch point parallel fork>> not directly supported<< branch to a synch point fork join>> not directly supported<< synch point synch pointbegin of SP start activity start activityRETURN; end of SP end activity end activityinterface object actor connector to system

EXCEPTION…END activity triggered by exception Exception block

GOTObranch of unconditional decision

branch of unconditional decision

1. Start with specific, measurable, in-use “data” itself: i.e.: Oracle DBMS’s

2. Refactor into from the vendor language, PL/SQL to a standards based language (UML).

3. Rebuild as Workflows.

Inventory the Data: Data Semantics are key


Step 2: Scan Source CodePACKAGE PK_LWIRE AS/******************************************************************************* Release : DEC03 Project Name : CAFE Module Name : Livewire Purpose : This package validates the Address in Livewire Tables Used : Global Variables : G_Lwire_mode Called from : Procedures Invoked : Nil Parameters Used : Modification History : -------------------------------------------------------------------------------- . . .G_Lwire_Mode VARCHAR2 ( 1 ); G_Ld3_changed VARCHAR2(1) := 'N'; PROCEDURE SP_LW_REQUEST_INIT; PROCEDURE SP_LW_REQUEST_MAN; PROCEDURE SP_LW_REQUEST_AUTO; PROCEDURE SP_LW_VALIDATE_ASR( P_Asr_Id IN NUMBER, P_Req_Type IN VARCHAR2, P_Ret_Code OUT NUMBER); PROCEDURE SP_LW_CHECK_OVERRIDE (

P_Asr_id IN NUMBER,

P_Refnum IN VARCHAR2,

P_Loc_ind IN VARCHAR2,

P_Ovr_flag OUT VARCHAR2 ); PROCEDURE SP_LW_VALIDATE_ADDR (

P_Asr_id IN NUMBER ,

P_Refnum IN VARCHAR2,

P_Loc_ind IN VARCHAR2,

P_Req_type IN VARCHAR2,

P_Ret_code OUT NUMBER ); /* Nov03:C383700 - Procedure added for supp candidacy check*/ PROCEDURE SP_SUPP_CAND ( P_Asr_Id IN NUMBER, P_refnum IN VARCHAR2, P_Loc_ind IN VARCHAR2, P_supp_Cand OUT VARCHAR2); . . .

Code snippetfrom the OraclepackagePK_LWIRE

(entire packageis severalthousand lines of code.)

Automation: without tools, its too hard to do!


Step 3: Code to Prolog (Horn Clauses)[PACKAGE,PK_LWIRE,AS,,,,G_Lwire_Mode,VARCHAR2,(,1,),(;),,G_Ld3_changed,VARCHAR2,(,1,),:,=,',N,',(;),,PROCEDURE,SP_LW_REQUEST_INIT,(;),,PROCEDURE,SP_LW_REQUEST_MAN,(;),,PROCEDURE,SP_LW_REQUEST_AUTO,(;),,,PROCEDURE,SP_LW_VALIDATE_ASR,(,,P_Asr_Id,IN,NUMBER,(,),,P_Req_Type,IN,VARCHAR2,(,),,P_Ret_Code,OUT,NUMBER,),(;),,,PROCEDURE,SP_LW_CHECK_OVERRIDE,(,,P_Asr_id,IN,NUMBER,(,),,P_Refnum,IN,VARCHAR2,(,),,P_Loc_ind,IN,VARCHAR2,(,),,P_Ovr_flag,OUT,VARCHAR2,),(;),,,PROCEDURE,SP_LW_VALIDATE_ADDR,(,,P_Asr_id,IN,NUMBER,(,),,P_Refnum,IN,VARCHAR2,(,),,P_Loc_ind,IN,VARCHAR2,(,),,P_Req_type,IN,VARCHAR2,(,),,P_Ret_code,OUT,NUMBER,),(;),,,,,PROCEDURE,SP_SUPP_CAND,(,,P_Asr_Id,IN,NUMBER,(,),,P_refnum,IN,VARCHAR2,(,),,P_Loc_ind,IN,VARCHAR2,(,),,P_supp_Cand,OUT,VARCHAR2,),(;),,,PROCEDURE,SP_LW_CHECK_OTC,(,,P_Asr_id,IN,NUMBER,(,),,P_Refnum,IN,VARCHAR2,(,),,P_Loc_ind,IN,VARCHAR2,(,),,P_Req_type,IN,VARCHAR2,(,),,P_Otc,OUT,VARCHAR2,(,),,P_Ret_code,OUT,NUMBER,),(;),,ETC . . .

struct_code(package,[plsql, ,END,PK_LWIRE,(;),,PACKAGE,BODY,PK_LWIRE,AS,,PROCEDURE,SP_LW_REQUEST_INIT,IS,,CURSOR,Asr_Crsr,IS,,SELECT,A,.,Asr_ID,Asr_Id,(,),,A,.,Act,Act,(,),,SUBSTR,(,A,.,Reqtyp,(,),1,(,),1,),Reqtyp,(,),,B,.,Status,Status,(,),,A,.,Supp_Pending,Sup_Flag,,FROM,Cta_Asr,A,(,),Cta_Asr_Status,B,,WHERE,A,.,Asr_Id,=,B,.,Asr_Id,,AND,B,.,Flowmode,=,',I,',,AND,B,.,Status,IN,(,4040,(,),7050,),(;),,,L_Asr_Id,NUMBER,(,13,),(;),,L_Ret_Code,NUMBER,(,3,),:,=,0,(;),,L_Sqlerr,VARCHAR2,(,80,),:,=,NULL,(;),,M_Sqlerrm,VARCHAR2,(,80,),(;),,M_Sqlcode,NUMBER,(;),,M_Cafe_ref,VARCHAR2,(,100,),(;),,M_Err_Pt,NUMBER,(;),,L_actl_count,NUMBER,:,=,0,(;),BEGIN,,M_Err_Pt,:,=,10,(;),,BEGIN,,M_Err_Pt,:,=,20,(;),,SELECT,DECODE,(,Lw_Mode,(,),0,(,),',N,',(,),1,(,),',Y,',(,),2,(,),',D,',(,),',N,',),,INTO,G_Lwire_Mode,,FROM,Ctr_Livewire_Control,(;),,EXCEPTION,,WHEN,No_Data_Found,THEN,,L_Sqlerr,:,=,SUBSTR,(,Sqlerrm,(,),1,(,),50,),(;),,INSERT,INTO,Cta_Intf_Log,(,Module,(,),Req_Id,(,),Status,(,),Message,),,VALUES,(,',L,',(,),0,(,),',1,',(,),',No,Data,Found,In,Ctr_Livewire_Control,',),(;),,G_Lwire_Mode,:,=,',N,',(;),,WHEN,Others,THEN,,M_Sqlcode,:,=,Sqlcode,(;),,M_sqlerrm,:,=,SUBSTR,(,Sqlerrm,(,),1,(,),80,),(;),,M_Cafe_Ref,:,=,',CONTROL,TYPE,',|,|,',LWIRE_MOD,',(;),,SP_INS_SYS_MESSAGES,(,',PK_LWIRE,.,SP_LW_REQUEST_INIT,',(,),M_Cafe_Ref,(,),,M_sqlcode,(,),M_sqlerrm,(,),',Y,',(,),0,(,),M_Err_Pt,),(;),,L_Sqlerr,:,=,SUBSTR,(,Sqlerrm,(,),1,(,),50,),(;),,INSERT,INTO,Cta_Intf_Log,(,Module,(,),Req_Id,(,),Status,(,),Message,),,VALUES,(,',L,',(,),0,(,),',1,',(,),',Ctr_Livewire_Control_Error,',|,|,L_Sqlerr,),(;),,G_Lwire_Mode,:,=,',N,',(;),,END,(;),,M_Err_Pt,:,=,30,(;),,FOR,Asr_Rec,IN,Asr_Crsr,,LOOP,ETC . . .]).

Scanner outputshowing proceduretokens

The Main Idea: It’s not about Prolog, but LOGIC


Step 4: FOL to XMI<?xml version="1.0" encoding="windows-1252"?><XMI xmi.version="1.1" xmlns:UML="omg.org/UML1.3" timestamp="01/23/04 11:14:13"><XMI.header><XMI.documentation><XMI.exporter>Enterprise Architect</XMI.exporter><XMI.exporterVersion>2.5</XMI.exporterVersion></XMI.documentation></XMI.header><XMI.content><UML:Model name="EA Model" xmi.id="MX_EAID_6DCAB6E5_74DB_4e03_AAA8_11417163F24B"><UML:Namespace.ownedElement><UML:Class name="EARootClass" xmi.id="EAID_11111111_5487_4080_A7F4_41526CB0AA00" isRoot="true" isLeaf="false" isAbstract="false"/><UML:Package name="Activities" xmi.id="EAPK_6DCAB6E5_74DB_4e03_AAA8_11417163F24B" isRoot="true" isLeaf="false" isAbstract="false" visibility="public"><UML:ModelElement.taggedValue><UML:TaggedValue tag="created" value="2001-03-05 00:00:00"/><UML:TaggedValue tag="modified" value="2001-03-05 00:00:00"/><UML:TaggedValue tag="iscontrolled" value="FALSE"/><UML:TaggedValue tag="lastloaddate" value="2004-01-22 20:46:45"/><UML:TaggedValue tag="lastsavedate" value="2004-01-22 20:46:45"/><UML:TaggedValue tag="isprotected" value="FALSE"/><UML:TaggedValue tag="usedtd" value="FALSE"/><UML:TaggedValue tag="logxml" value="FALSE"/><UML:TaggedValue tag="batchsave" value="0"/><UML:TaggedValue tag="batchload" value="0"/><UML:TaggedValue tag="phase" value="1.0"/><UML:TaggedValue tag="status" value="Proposed"/><UML:TaggedValue tag="complexity" value="1"/></UML:ModelElement.taggedValue><UML:Namespace.ownedElement><UML:ActivityModel xmi.id="EAID_6DCAB6E5_74DB_4e03_AAA8_11417163F24B_ActivityModel" context="EAPK_6DCAB6E5_74DB_4e03_AAA8_11417163F24B" name="ActivityModel" visibility="public"><UML:StateMachine.transitions/><UML:StateMachine.top><UML:CompositeState xmi.id="EAID_6DCAB6E5_74DB_4e03_AAA8_11417163F24B_Activity_Top" name="{top}"><UML:CompositeState.substate>name> SP_LW_RESPONSE_PARSE<UML:ActionState name="SP_LW_RESPONSE_PARSE" xmi.id="EAID_C7946CD7_A379_4a6f_A353_56FE1875609E" visibility="public" namespace="EAPK_6DCAB6E5_74DB_4e03_AAA8_11417163F24B"><UML:ModelElement.taggedValue>

ETC . . .

Partial XMIoutput for package SP_LW_RESPONSE_PARSE

This new representation is used for importing into a tool (Enterprise Architect™)

FOL: First Order Logic – Vocabulary & Terms


Step 5: Import/Generate XMI to UML

SP_LW_RESPONSE_PARSE SP_LW_LOC_DSG_INSP_LW_RETREIVE_DEST_BBID

SP_LW_INSERT_PREM_REQ

SP_LW_RESPONSE_PROCESS

SP_LW_IREP_REQUEST_FORMAT

Importing Model from XML***** Root Package: Activities *****Add State: SP_LW_RESPONSE_PARSEAdd State: SP_LW_LOC_DSG_INAdd State: SP_LW_RETREIVE_DEST_BBIDAdd State: SP_LW_INSERT_PREM_REQAdd State: SP_LW_RESPONSE_PROCESSAdd State: SP_LW_IREP_REQUEST_FORMAT***** Resolving Classifiers ********** Resolving Ownerss ********** Resolve Package Links ********** Add Extensions *********** Add Diagrams ********** Resolving DiagramIDs ********** Resolving Association Classes ********** Resolving Parameter Types ********** Resolving Attribute Types ********** Resolving Note links *****Import Complete!

The importcreates the nodes and the diagraminto theUML repository(package bypackage)

Transformations: Use Logic to generate XMI


Outcome

• Need good tools based on LOGIC (Common Logic is in ISO process)

• Develop Models and Meta-Data using FOL (now Common Logic) and languages (for now, MOF, UML)

• Focus on understanding the Workflows of processes:– Their Transfers of Data/Information etc..– The Transformations they accomplish

• Develop these into “Services”

Outcome: Use FOL principles to guide SOA


Big-Pharma

SSOA for Knowledge Based Interoperability


Desire• To provide timely knowledge from sources to

clients (physicians, hospitals, labs…etc) within federal mandates

• To save lives as a direct consequence of network performance in delivering knowledge

• To provide information assurance (IA) and Multi-Level Security (MLS)

• To provide high-availability, high-performance, optimized Quality Of Service

• To provide end-user personalized experience

Desire: Save Human Lives & Bottom Lines


Scenario – Synchronized Training

Physical Network - Heterogeneity

videovideo

video

video

Breers - Belgium

Raritan-N.J

Sydney - Australia

Channel Type: Publish (TrainingVideo)Subscribers

(Client Users)

ROUTERS

Failover Capacity

Pipieline-1

Video Pipelines

Filled

Failover Capacity Pipeline-2

Problem: Space & Time, Content Synchronicity


Client Situation #*!?

Personalization

ServerBrowser

XHTML?

Client?

text

form

CGI

WebServer

Other App. Servers – XML MessagingCore Objects

Router

WorkFlowTransact

Sd fdsa as dfsaasdf asd asd asd

asdfasd asd tasdf asfd as asdf Sd f

dsa as dfsaasdf asd asd asd

asdfasd asd tasdf asfd as asdf

















Sd fdsa as dfsaasdf asd asd asd asdfasd asd ta





ROUTERS

Legacy

CORBA / RMI …

TCP/IP

JAVA, .NET, DCOM…???

What?

App.Server

How Can it All Work in

synch?

ASP

XML

HTML

Situation: Spaghetti-Code and Systems


Effort : Cross-Cutting Concerns

Xlink - ChannelsXlink allows elements to create and describe links using XML syntax to describe simple unidirectional hyperlinks of regular HTML; Additionally, the syntax supports Channels & Pipelines to Services (via a pointer reference mechanism).

XML: MetaData

XSL - Form Generation

XML: Templates

(Rules, Processes)

Browser BehavioursHTML-GUI/ODBC/HTTP

A DTD would be defined for each big pharma Trusted Partner, Affiliate, Company and this would include the corporate ontology and taxonomy.

XML: DTD Types &

Taxonomy

XML - Flow Logic (Browser Engine)

Personalization Info

Pipeline #1

Transactions

Processor

Pipeline #3Pipeline #2

DB Remote

Top-Level: Connect Aspects / Ontologies


Communication Model : Aspects

XML

ROUTER

ACCESSOR

VIEWER

Portlet

Browser

Information

text

image

audio

video

forms

Streaming AudioStreaming Video

The Channel Pipeline gets channels with specified security access and data-transfer capacity to the Router component of the Browser Portlet (object). This permits complete structural dynamism to be determined on a performance/availability/failover basis --- the individual allocated channels are used by the Accessor Component to search or acquire information, while the Viewer Component uses its special channels for streaming data displays.

The App.Server provides these components

Channel Types

Channel Pipeline

Pass the different channel types along a special channel type called the Channel Pipeline

Personalize

Channels can be Public, Broadcast, Private & Secured

Broker

Node

Mid-Level: Aspects Model = Channels


AccessDelivery

WorkflowsOntologies Agents

Content-Containers Hardware-Network

SSOA NODESERVICES

Workflow QOSQOI

Resources Single Sign-onFunction

•TAXONOMIES•HIERARCHY TYPES•SYSTEM TYPES•CAPABILITIES

•TASKS•RULES•PROCESSES•ORHESTRATION

•HUMANS•ACTORS•SYSTEMS•ENTERPRISES

•DATA•INFORMATION•KNOWLEDGE•INTELLIGENCE

•WIRELESS / PDA•ROUTERS•COMPUTERS•CONTROLLERS

An SSOA Meta-Model Emerges!

IA

REF: http://www.cutter.com/content/architecture/fulltext/reports/2006/01/index.html

MOF MDA


Outcome

• Smart JAVA node-based server nodes• Channels with priorities to Brokers that connect

clients to servers (using SOAP Objects/XML)• Channel monitors provide metrics (for IA, QOS)• Channel profiles deliver categories of

personalized experience (physician vs. nurse)• Currently implementable with current tools, and

able to accommodate future evolution

Take Home : Multi-Level MDA via FOL methods


Thank You

operationalizing soa – lessons learned

Documents