tim de borger principal solution consultant may 18 th, 2007 tuning the esb how to make the bus drive...

Tim De BorgerPrincipal Solution Consultant

May 18th, 2007

Tuning the ESB

How to make the Bus drive faster

© 2007 Progress Software Corporation2

Agenda

Quick overview Current tools What are we measuring Message Influences (what is Flow Control?) Simulation environment Scaling in different ways Know what you are doing Hands on testing The Simple Calculation Conclusion


Quick overview

Performance Tuning• Art of tuning

– Change only 1 parameter at a time– Take your time for it– Always check the same things (even reboot/restart after each

run)– Know what you are measuring

• Check resources– Disk– Memory– CPU– Network

• Have a plan– What are the requirements– Multiply requirements by factor X (Stress Situation)– When to stop


Current tools

Testing SonicMQ• Sonic Test Harness

– Sends in a lot of messages– Receives a lot of messages– Difficult to control– Focused on message throughput (Volume)

Testing the ESB • No tool is provided (yet)• A lot of external influences• No Controlled Environment


What are we measuring (or trying to)

We need to be able to understand the time a message is spending in our system

External influences are very important for performance• Longer runs for external DB calls

• WebServices

We need to define the latency of the message on the System (not on A Single Queue or Topic)


Messaging Influences

Message Scenario 1• Running 1 SonicMQ Broker• Using 1 Topic• Running 1 Publisher (Fast)• Running 1 Subscriber (Fast)

Everything is working just fine Message Scenario 2

• Adding another Subscriber !!!• Unknown behavior of Subscriber =>

DANGER!!!!!


Flow Control Demo

Point to Point

SenderSenderSenderSenderPotentialPotentialReceiverReceiver

PotentialPotentialReceiverReceiver



QueueQueueQueueQueue

Flow Control in PtP:• If the Maximum Size of a Queue is reached, Flow Control

is kicking in

• Sender seems to be hanging but is only waiting for space in the Queue

• Flow Control can be switched off and replaced by an Exception in your code.


Flow Control Demo

In Pub/Sub, the working of Flow Control is more difficult to predict• Each Subscriber gets a Buffer (Outgoing Buffer Size)• Sonic will try to only have 1 copy of the message in memory• When the buffer is reaching a threshold, Flow Control is issued.• Threshold depends on message priority

Publish and Subscribe

PublisherPublisherPublisherPublisher TopicTopicTopicTopic

SubscriberSubscriberSubscriberSubscriber



Outgoing Buffer

Flow Control Demo

Publish and Subscribe

PublisherPublisherPublisherPublisher TopicTopicTopicTopic



Pub/Sub

TEXT

Outgoing Buffer

Only a pointer is inserted in the Buffer but the calculated size is the message size

Ack

Ack

Danger !!!Slow Subscriber may result in the following situation

TEXT

TEXTTEXTTEXT

TEXTTEXTTEXT

TEXT TEXT

FLOW CONTROL


Conclusion of the test

We can have a fully working system that runs without any problems

Adding 1 Client can have dramatic impact• 1 Slow SubScriber will make the Publisher Slow

• Badly written Client(s) can bring any message Broker to its knees

Where to look and debug if this is in an ESB Service?• More important: HOW TO SOLVE THIS?


Using the ESB to Simulate the ESB

We developed the Simulation Tool in order to:

• Have a controlled environment to investigate the impact of introducing an ESB.

• Have an environment to understand the working and concepts of the ESB

– Difference between a Process and loosely linked services– What is the impact of Intra-Container messaging– How to develop a deployment scheme and calculate the number of

listeners. (slow 1st Service, Slow last Service, …)

• Have the possibility to trace the message through the system

– Time-Stamp– Endpoints report in the message body

• Easy deployable model to test ESB over multiple containers

– Impact on Quality of Service– Testing with a DRA Model

We developed it with the ESB itself.


Concept

How to Simulate• Test Client that sends an incoming message• Test Client that receives the outcome of the ESB (REPLY-

TO)• Sender puts a time-stamp• Receiver gets the timestamp and calculates the message

latency in the ESB System.• ESB ServiceType for Simulation

What to Simulate• CPU• External influences• Message growth


Simulation environment

External Influences

Processing Power

Message Growth


Scaling in different ways

When looking at the ESB, we can scale in 3 different ways

Horizontal: Moving from 1 listener to X listeners• Differences between Process and Service

• Intra-container messaging

Vertical: Moving from 1 JVM to Y JVM’s• Quality of Service might forbid this!

Diagonal: Moving over Z multiple machines


XYZ Scaling

X-scaling: Multiple ListenersY-scaling: Multiple JVMsZ-scaling: Multiple Machines


Know what you are doing

There are limitations:• The Obvious:

– Disk traffic– CPU– Memory

• The not so Obvious:– Flow Control– Your JVM:

Thread handling Memory size Resources given by the O/S SonicMQ (move to a cluster etc)


The Files

Files are available:• Java code for the Service Type • esbstyp definition for the Service type• Properties definition for the Service Type• Java Client/Receiver

Installing the Service Type• Put the jar file in the Classpath of the ESB

Container• Import the esbstyp file• Copy the properties file to the right place


Setup

Create a new ESB container Create some test endpoints if you want or use

Sample Queues/Endpoints that exist Create a couple of the Benchmark Services Deploy them into a container Add the container to an MF Container Start the container(s) so the environment is

running.


Configuration

Create 3 Services from the ESB Benchmarking Type and define the needed Endpoints on them.

Create a Process that uses the 3 services

• Make the waiting times 500, 1000, 2000 ms for the 3 entries

Set REPLY-TO on the process


Hands on testing: Running Test

In the Benchmarking\bin directory find the bench.bat file.

Set the path right for your SonicMQ installation as it calls SonicMQ.bat from the samples

It is a modified Talk application so make changes to the parameters for connection

Added 3 parameters to give on the Bench.bat command• Number of messages to send• Sleep time in ms between those messages• Size of the initial message to be sent in• Bench.bat 20 2000 2000


Hands on testing

First go:• 1 ESB Container• Make the 2 services each sleep for 500 ms.• Run Bench 100 1000 1024• Results?• Run Bench 100 100 1024• Results?• Why? Solutions?

Play further with different setups.• Bench 100 250 1024• Bench 100 100 1024• Bench 100 50 1024• Bench 100 1 1024


Performance Behavior

1 ESB Container 1 Listener

0

10000

20000

30000

40000

50000

60000

1 9 17 25 33 41 49 57 65 73 81 89 97

Message Sequence

Mes

sag

e L

aten

cy 100:1000:1024

100:500:1024

100:250:1024

100:100:1024

100:50:1024

Messages are handled without Delay by the running ESB Service. This results in a flat latency line

Messages are handled with a very small Delay by the running ESB Service. This results in a very slow increasing latency line

In all other cases, the ESB cannot cope with the publishing rate of the client and the message latency will grow. The faster the publishing rate, the steeper the curve.


The Simple Calculation

There is a calculation that can be made in order to find out how many Listeners your Service Needs (This calculation is needed for every Service in the Process):• Maximum time for the message to go through the

Service (Latency) = TSL (2 sec)• Time to handle the Load = THL (300 sec)• Number of messages in the Load = NM (1000

msg)• Formula to give the number of handlers:

– ((NM / THL) * TSL) (6.67) For this setting, you should make sure that the

number of messages that can be treated in parallel should be 7 or 8.


Conclusion

This tool can be used for testing/simulating:• DRA environments

• Deployments of ESB Components over multiple containers

• Investigate and test the differences between an ESB Process or linking the ESB Services together

This tool should not be used for Stress-Testing the SonicMQ environment.


QuestionsQuestions


CONNECT EVERYTHING. WORKING ON ACHIEVE ANYTHING.

www.sonicsoftware.com

tim de borger principal solution consultant may 18 th, 2007 tuning the esb how to make the bus drive...

Documents

progress software corporation

flow control demoflow

working of flow control

flow control demopublish

queueflow control

message influences

flow control demoin

message priority