Table of Contents1. Asset Bounds....................................................................................................................................3

2. Asset Introduction...........................................................................................................................3

3. Advantages of BizTalk..........................................................................................................................4

4. Integration Process..............................................................................................................................5

4.1 File or Web Service......................................................................................................................6

5. Adapter and Integration Solutions.......................................................................................................7

6. BIZTALK Best Practices.........................................................................................................................8

a. Design Best Practices.......................................................................................................................8

b. Schema Creation Best Practices.......................................................................................................9

c. Map Creation Best Practices..........................................................................................................10

d. Orchestration Best Practices..........................................................................................................11

e. Best Practices for Performance.....................................................................................................15

1

Integration Reusable Assets - Systems Integration Approach for MES

Date 27/1/2014

Owner Vinod kumar kodatham

Description This approach will help to change the traditional approach of point-to-point communication in Manufacturing Execution Systems (MES) to using BizTalk server as a middleware to Integrate several systems. Multiple channels of several systems can be plugged into the centralized BizTalk server. Enterprise Service Bus (BizTalk server) with its inbuilt BAM and ESB capabilities will help especially for reporting Application tracking and application failures in respective BAM and ESB portals respectively. These are totally reusable and isolated which can be used by any application across the industry which uses BizTalk as the middle ware platform

OBJECT OVERVIEW

Technical NameIntegration Approach for MES.

Description

The asset will provide high level design considerations which enable seam less integration cross platform and systems using BizTalk framework which developers can modify as per the requirement.

2

1. Asset Bounds

This asset can be used for the several simple to complex Integration scenarios that needs to scale from low to high volumes with low latency and need of fault tolerance that has got redundancy, scalable and message retry features and governance over the solution. The conceptual architecture of a solution is done by even taking into account non-functional requirements like performance, availability, extensibility, resilience, robustness, and scalability and ensure they will be properly addressed during the technical design phase. This gives you out of the box features to develop/customize all these functionalities. The asset is independent of Source and Target System(s)/ Application.

2. Asset Introduction

This document describes the idea of systems Integration using a centralized BizTalk application i.e. using BizTalk framework as an Integration platform for integrating systems (process in MES System) like POM, RIM, IOM, RCM, QOM, SAP and PCS (Process Control Systems). BizTalk framework orchestrates the integration process by maintaining the performance, robustness and availability. This framework will describe the design decision made and development task required to build the BizTalk application’s components.

Any Business scenario which operates with integration has the above typical interface.

1. Business has a need to transfer information from one application to a different application or one system to a different system that are residing in a single or different multiple servers. These applications can be either in a same server or a different servers

2. These applications can be either an application server which uses Webservices or a database server. Now this source data needs to be migrated to the target system meeting the business requirements which is called as transformations.

3. In order to integrate these two different systems so that they can communicate each other business requires a Middleware i.e. BizTalk server.

There are lots of Middleware technologies available in market but if you need to go for the best then we say its BizTalk server as it is one of the most successful product in the market. It’s a Microsoft product. The benefits of this tool are mention in the very next section.

3

3. Advantages of BizTalkFigure 2: Orchestrated Data transfer Process

While planning for the integration scenario BizTalk takes care about tasks like Gathering information, Defining naming conventions, planning team development, setting up and working with source control.

Advantages of using BizTalk are.

1. Recoverable Interchange. In BizTalk, an Interchange can contain two or more messages, such as a batch. Recoverable Interchange was introduced in BizTalk Server 2006, only messages will be suspended if they fail validation, and when corrected they can be resumed after the error.

2. Failed message routing. This functionality is available to enable failed messages to be subscribed by orchestration and send ports. When used appropriately, failed message routing will be used to notify users about the failed messages or to handle error properly and repairing the messages if required.

3. Lowest total cost of ownership (TCO). BizTalk Server reduces the cost and complexity of managing business processes and its automation with a single, unified solution for Enterprise Application Integration (EAI), Business-to-Business integration (B2Bi), and Business Process Management (BPM).

4. BizTalk Integrated management and development tools will enhance productivity.5. Using TAPI it also supports Computer Telephony Interface (CTI).6. Powerful, familiar tools to design, develop, deploy, will enable to manage those

processes easily

4

4. Integration Process

Figure 3: A typical Integration Scenario

In this scenario, information will be transferred from multiple source systems like Enterprise Resource Planning (ERP), Line of Business (LOB) systems or any other third party vendor systems to Target system (ERP, LOB or a third party vendor). Messages inside BizTalk are through the XML documents and defined with the XML schemas in XSD standard. Maps are implemented with the XSLT standard. Orchestrations or mappings are implemented to meet the business needs while transferring data from source to target. Schemas, maps, pipelines and orchestrations are created visually using graphical tools within Microsoft Visual Studio. The additional functionality can be delivered by .NET assemblies that can be called from existing modules—including, for instance, orchestrations, maps, pipelines, business rules.

BizTalk allows creating 'send' and 'receive' port through which external applications can exchange messages .The ports can be configured to use a set of transport protocols and application adapters that are shipped with BizTalk Server. With these adapters, BizTalk can exchange messages over multiple channels such as files, MSMQ, HTTP (Hypertext Transfer Protocol), SOAP, IBM MQ, etc. Application adapters, which can directly interact with packaged applications like SAP, Oracle Apps, etc., are also available.

5

4.1 File or Web Service

Data is retrieved from a third party web service or a file based where the business rules are applied in and will transform the data to the target system. There are different ways to expose and consume Webservices from BizTalk Server.

BizTalk can understand both positional/delimited and XML message formats. Biztalk can transform messages from one format to another.

Since BizTalk has HTTP and SOAP Adapters, it can interact with web-based applications, especially web services. The BizTalk Server interacts with services in the following two ways:

1. BizTalk process orchestrations can be exposed as Web services. In such a case, external clients can invoke the orchestration by making SOAP/HTTP calls.

2. BizTalk can consume Web services by using the SOAP Adapter.

6

5. Adapter and Integration Solutions

Adapter for BizTalk Server seamlessly integrates BizTalk with enterprise databases, applications, transactions, business packages, EDI, and more.

Adapters allow connecting Microsoft BizTalk server to Enterprise Information Systems, including homegrown applications, databases like DB2, legacy data stores, transaction systems, e-business formats such as EDI, HIPAA and HL7, and packaged applications.

The Adapter Offering for BizTalk Server:

1. Provides easy administration and configuration through existing BizTalk Server Administration Console

2. Complete, Proven, High-Performance Adapter Set3. Relational and legacy database systems4. Transaction processors5. Custom applications6. Object models like COM objects, Enterprise JavaBeans, and others7. Messaging environments like IBM MQSeries,Oracle AQ,TIBCO EMS Q8. Terminal-based systems like mainframes and AS/400s9. Packaged applications such as Enterprise Resource Planning (ERP) systems, Supply

Chain Management (SCM) systems, and Customer Relationship Management (CRM) systems

10. Electronic Data Interchange (EDI) systems11. e-Business XML formats such as SWIFT, FIX, HIPAA, ebXML, BOD, and cXML

7

Each adapter masks the complexities of its underlying source, so developers can use industry-standard XML, SQL, or Stored Procedures to work with all types of information. Also, each adapter is highly optimized for its target, so developers don't require specialized knowledge to take full advantage of special features and syntax for unusual data structures, applications, and transactions. All adapters provide standard metadata about available functions and fields, return values, error codes, and more.

6. BIZTALK Best Practices

Some of the BizTalk design best practices to be followed are.

a. Design Best PracticesCategory Technique Reference

Design Group the projects as per artifacts type.

It is a best to group the

projects as per the kind of

artifacts it contains. Schemas

are broken into Internal, plus

either External, Import and

Export, or even separated by

adapter: SQL, Oracle,

etc. Orchestrations or

Processes should be in one

project.  Avoid using

Transforms in Orchestrations. 

Put maps in the Ports.

Pipelines, if you need them, in

a separate project.

Maps or Transforms in a

project.

Design Promote only those properties that are required for routing or tracking. Use distinguished property for expression shape usage.

Performance

More processing required to find the promoted properties from the message. These properties are loaded in the memory, hence having a lot of them can put the system under extreme loads. Also, they are

8

written to the database for tracking taking up disk space and more CPU cycles.

 

b. Schema Creation Best Practices

Category Technique Reference

Schemas Always Use Separate Internal and External Schemas

Always transform messages

into canonical schema—

regardless of how simple the

schema is or who the source

is. This can reduce the number

of maps you need. Suppose

you need to map three types of

inbound messages to three

types of outbound messages

(perhaps not today, but maybe

in the future). Applying this

technique, you create a map to

your canonical schema for

each inbound schema and then

a map from your canonical

schema to each outbound

schema. That’s only six maps

to build and maintain, not

nine.

Schemas Always validate xml message against the standard schema.

N/A

 

9

c. Map Creation Best Practices

Category Technique Reference

Transformation Use Transformation on Receive and Send Ports when you have multiple partners sending multiple message formats and you need to convert these messages to a canonical format.

Business reason. Embed the map in the schedule and the partner changes the format, not only do you have to rebuild the map, you have to rebuild the schedule to use the new version of the map.

Transformation Use transformation in your orchestration schedule When you need to generate a new message in the schedule and use the modified (mapped) contents of an existing message as the base. When you want to map multiple parts of a message into one outbound message (this type of mapping cannot be done in receive / send port). There are performance gains which come from doing mappings in receive ports sometimes, but they are mostly around how many persisted messages your scenario generates and it is a bit complicated to explain.

Performance.

Transformation Pure XSLT is much faster than XSLT which uses inline script or referenced assemblies.

Performance

Use xslt as much as possible in

mapping.

Transformation Use external XSLT or Maintainability.

10

serializable classes in case of complex maps with 1000+ connections.

Transformation Do not use .NET objects directly inside the map. Use external assemblies instead.

Maintainability and Performance.

Transformation To minimize memory consumption, we recommend that you put any map that uses these functoids into a small assembly. Then, reference that assembly.

Performance.

The

System.Policy.Security.Evidence

object is often used in transforms

and can consume a lot of

memory. Whenever a map

contains a scripting functoid that

uses inline C# (or any other inline

language), the assembly is created

in memory. The

System.Policy.Security.Evidence

object uses the object of the actual

calling assembly. This situation

creates a rooted object that is not

deleted until the BizTalk service

is restarted.

d. Orchestration Best Practices

Category Technique Reference

Orchestration Eliminate orchestrations for messaging only patterns

When possible minimize the use of orchestrations to increase overall throughput and reduce the latency of business processes. If there is

11

no need to run long running transactions and there is no need to invoke several systems for each request, then consider eliminating orchestrations and moving business logic to Receive and Send Ports to reduce the total amount of roundtrips to the BizTalkMsgBoxDb and decrease the latency due to database access. In this case, implement custom pipelines and reuse helper classes that were previously invoked from orchestrations. Use orchestrations only when strictly needed to implement design patterns as Scatter and Gather or Convoys.

Orchestration Always Use Multi-Part Message Types

If your logical port and Receive/Send shape are using your multi-part message type, and you can easily change the underlying schema in the future without having to disconnect the port from the receive shape.

Orchestration Optimize orchestration latency by reducing the number of persistence points.

Best practice

Orchestration Minimize orchestration complexity to improve performance

Best practice

Orchestration Use the Call Orchestration shape versus the Start Orchestration shape to improve performance

Best practice

Orchestration Use XmlReader with XLANGMessage versus using XmlReader with

Performance

12

XmlDocument to improve orchestration performance

Orchestration minimizing the use of logical ports bound to physical ports

Performance

Orchestration to extract or set properties used with business logic in an orchestration

If you are using a map to extract or set properties used with business logic in an orchestration, use distinguished fields or promoted properties. This practice should be followed because when extracting or setting values with a map the document is loaded into memory however when using distinguished fields or promoted properties, the orchestration engine accesses the message context and does not load the document into memory.

Orchestration aggregate several fields into one field

If you are using a map to aggregate several fields into one field, use distinguished fields or promoted properties with an orchestration variable to accumulate the result set.

Orchestration Use Orchestration Design Patterns when required

Performance and maintainability

Orchestration Make appropriate use of .NET classes in orchestrations to maximize performance

Performance and maintainability

Orchestration Orchestration Exception Handler Blocks

When using Orchestration exception Handler blocks, include all orchestration shapes in one or multiple scopes (non transactional scopes whenever possible) and create at least the following exception handler blocks:

13

An exception handler block for handling a generic System.Exception error.An exception handler block for handling a general exception in order to catch and handle possible unmanaged errors, such as COM exceptions.

Orchestration Always Try to Design Orchestrations with Direct-Bound Ports

You’ve got three Direct-

Bound port types: Message

Box Routing, Self-Correlating,

and Orchestration-to-

Orchestration (also called

Partner Ports). Direct-Bound

ports are self-configured,

without sacrificing flexibility

or performance. A handy

benefit of Direct-Bound ports

is that neither the developer

nor the administrator needs to

create corresponding physical

ports in BizTalk Explorer or

BizTalk Administration

Console. So Direct-Bound

ports yield orchestrations that

are more self-contained, and

therefore more reusable and

easier to redeploy

independently.

Orchestration Publish Your External Schema, Not Your Orchestration

It provides loosely coupled

mechanism for exposing the

services. This buys you more

freedom to change your

orchestration without breaking

14

the caller

e. Best Practices for Performance

Category Technique Reference

Performance Optimize the BizTalk Registry for Web Services.

If your BizTalk orchestration is published as a Web service, you’ll definitely want to tweak some of the default ASP.NET parameters used by BizTalk. Below are the recommended settings.

 Windows Registry Editor version 5.00

 [HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\BTSSvc$BTSHOST\CLR Hosting]

“MaxIOThreads”=dword:00000064

“MaxWorkerThreads”=dword:00000064

“MinIOThreads”=dword:00000019

“MinWorkerThreads”=dword:00000019

 Please refer http://msdn2.microsoft.com/en-us/library/aa561380.aspx for more details.

Performance Configure Parameters Used for Low Latency Performance Tuning

Please refer below link for more information.

 http://msdn.microsoft.com/en-us/library/aa475435.aspx

Performance Create separate hosts for orchestration, Receive and

Creating separate hosts allows BizTalk to implement throttle mechanism and

15

Send Ports help it to load balance the incoming messages.

High Availability Cluster the resources for high availability.

Clustering of MSDTC, SSO, Disk and BizTalk for High availability.

http://www.microsoft.com/downloads/details.aspx?displaylang=en&FamilyID=eb437722-2828-4cbb-84c3-17556b4df26b

High Availability Use database mirroring for BizTalk Server databases high availability.

Database mirroring of BizTalk databases helps to achieve the High Availability for Database servers and helps to recover from Failover scenarios.

Appendix

Reference used for Best Practices:  

http://social.technet.microsoft.com/wiki/contents/articles/5210.biztalk-server-2010-orchestration-best-practices.aspx

16