201207 - esb01 - wmb monitoring auditing record and replay

IBM WebSphere Message BrokerMessage Monitoring, Auditing, Record and Replay

Tim KimberWebSphere Message Broker Development

1

WebSphere Message Broker DevelopmentIBM Hursley Park, UK

Agenda

� Overview of Monitoring

� Monitoring support in WebSphere Message Broker

� Configuring monitoring events� Configuring monitoring events

� Record and Replay in WebSphere Message Broker V8

Agenda





3

Business critical applications and in-flight data

� Business applications rely on information exchange– Application to application communication represents business interactions– E.g. Web to Order system, Warehouse to Distribution, Payments to Accounts

� Business data passes through the ESB– Message Broker has full visibility of the data content in any format– Uniquely positioned to report on the status of the business data as it is being – Uniquely positioned to report on the status of the business data as it is being

processed– Can emit events containing relevant data as well as the success of failure of

transactions– Supports Business Activity Monitoring (BAM), auditing or data capture for

potential later replay

What is Business Activity Monitoring?

� Ability to monitor your business performance

– Giving a real-time view of what is happening in your business

– Identifying problems in your business processes

– Opportunity to improve your business processes and business competitivenessbusiness competitiveness

� A way of making the business more transparent

– Allows evidence-based decision making

– “X-Ray for business processes”

� WebSphere Business Monitor– A BAM software product that

subscribes to business events and displays the event information in a dashboard.

5

Business Activity Monitoring

� Why?– Decision-makers need Key

Performance Indicators (KPIs)

� Where?– Best source for KPIs are the

applications which run the business

– The ESB has visibility of all – The ESB has visibility of all of those applications

� How?– A Monitor Model describes

how to process events and how to collect the information in dashboards.

– Applications can send the monitoring events which are collected by Business Monitor.

6

Agenda





Monitoring support in Message Broker

� Events are published to an MQ topic– To allow multiple subscribers – To allow each subscriber to choose the level of granularity

• Domain / Broker / Execution group / Message flow

� Event format is XML (published schema)– Designed to be compatible with CBE– Allows message broker to integrate with other monitoring applications– Allows entire message to be captured and logged to a database for audit purposes

� Events can be forwarded to monitoring tools, e.g. WebSphere Business Monitor– A WBM message driven bean is provided with the monitoring sample– The bean is hosted on a WebSphere Application Server and wraps the message broker

event with a Common Base Event wrapper and then submits it to the CEI server.

8

Monitoring Events

� A message flow can emit two kinds of events:

� Transaction events– Each input node in a message flow contains three events sources (in addition to terminal

events):• transaction.Start• transaction.End• transaction.Rollback• transaction.Rollback

� Terminal events– Any terminal in a message flow can be an event source. – If the event source is active, it emits an event each time a message passes through the

terminal.

� All events are optional, and fully configurable

� Events can contain simple or complex data from message payload

Event Configuration

� Events are configured in two ways:

� Via the message flow editor– Excellent support in message flow editor via Monitoring panel on all nodes.

� Via the command line– Using a monitoring profile configurable service. – Activate monitoring for the flow using

• mqsichangeflowmonitoring -c • mqsichangeflowmonitoring -c – Enable event sources using

• mqsichangeflowmonitoring -i– Ensure that the event sources are enabled using:

• mqsireportflowmonitoring

� The use of a monitoring profile configurable service overrides the monitoring properties of a message flow.

� The monitoring profile can be used to customize events without re-deploying the message flow.

Notes� Terminal events

– Event is only emitted if the message passes through the terminal in a forward direction

– Events for error conditions should be configured on the nodes attached to the failure/catch terminals of the input node(s)

� Configuration– Purposely designed to be administered without the toolkit

• So command-line / monitoring profile can do anything that the toolkit can dodo

• …and toolkit config can be exported as monitoring profile to ease the transition ( see later slide re: mqsireportflowmonitoring )

11

Monitoring event format details

EventData

MessageFlowData

EventIdentity

EventSequence

EventCorrelation

broker

executionGroup

EventPointData localTransactionIdparentTransactionIdglobalTransactionId

creationTime

counter

eventName

brokerName / uuid

executionGroupName / uuid

SimpleContent

ComplexContent

messageFlow

node

ApplicationData

BitstreamData EncodingBitstream

flowName / uuid

nodeNameterminalName

namevaluedataType

name

Notes� EventPointData/EventData

– Provides the 3 main items of data for a monitoring application• Identity, Correlation, Sequence

� EventPointData/MessageFlowData– Should be self-explanatory – no surprises here

� ApplicationData– Can come from message headers or body– Automatically included as XML, even if source message was non-XML– If the XPath/ESQL returned a simple element, it is placed in simpleContent, else

it goes into complexContentit goes into complexContent

� BitstreamData– More details in later slides

Agenda





Configuring monitoring events using the toolkit

Note: terminals are coloured ‘red’ on this slide to highlight those terminals with monitoring events defined.

Notes

� Note the Monitoring page in the Properties view– The canvas of the message flow has been clicked, so it is displaying

configured event sources for the message flow • not all event sources

– Clicking an individual node would show event sources for that node

� How many potential event sources are there in this flow?– 15 - 3 terminal events on each node(though the compute node could have

more if more output terminals are added) + the 3 transaction events on the input node )

� Highlight the Event Source Address column� Highlight the Event Source Address column– ESA is used to address an event source from the command line, or from a

monitoring profile. It will be unique within a message flow, provided that the flow does not contain duplicate node names.

� Terminals highlighted in red have events– Not displayed like this in the editor!

Configuring monitoring events using the toolkit These event sources are internal to the input node. They are not located on a

terminal


Configuring monitoring events using the toolkit


Configuring: Adding an event to a node

Configuring: Adding an event to a node

Select the event source. Input nodes include the special ‘transaction’ event sources

Configuring: Customizing an event

The event name can be a literal value, or can be literal value, or can be extracted from the message payload using an expression

Configuring: Adding a filter to an event

XPath expression to indicate if this event should be emitted or not

Configuring: Adding a filter to an event (2)

� Expression evaluates to – True – event emitted– False – event not emitted

� Evaluated at runtime

� Set on event source definition

� Expression can reference fields from anywhere in the message assembly

� XPath expression builder support available

� Event filter appears on Monitoring summary table

Notes:

� Event Filter section in the Monitoring tab (V7 onwards)

� Enables user to filter out events which do not match a business rule. So events can be filtered at Message Broker rather than emitting to Business Monitor and filtering there - which will help performance

� The event filter can be set to a numeric, boolean or string XPath expression which will evaluate to boolean true or false. Typically the result of an a = 'b' or x > 'y' type test

– If the expression evaluates to true then events are emitted– If the expression evaluates to false then events are not emitted

� The default setting is true()

Customizing an event – bitstream data

Click here to add part or all of the bitstream

to the event


Include headers, body or entire

bitstream


Encode bitstream as hexBinary, base64 or

CData

Notes� CData format for bitstream

– Not safe unless you know that the XML is free from invalid characters• NB: CData does not protect you from invalid XML characters

– So not usually safe for use with ‘All’• Because that will include headers which may contain binary data.

– Recommendation: Use CData encoding with care, and only with content set to ‘Body’

When to include bitstream data

� Bitstream data for monitoring– Not expected to be used in standard BAM scenarios.

� Bitstream data for auditing– Events can be captured and written to a database– Often required by compliance bodies

� Bitstream data for resubmission– Used by the ‘Record & Replay’ facility in WMB V8

• More later

� For V7 and earlier, WMBTM offering can provide capture/repair/resubmit based on these monitoring events

Notes: Command Line: mqsichangeflowmonitoring

� mqsichangeflowmonitoring – activate monitoring for the specified message flow(s)– set name of monitoring profile to use for the message flow(s)– enable and disable individual event sources in a message flow– Multiple event sources can be modified in a single command invocation– Multiple event sources can be modified in a single command invocation– No need to edit message flow and redeploy

Notes: Command Line: mqsireportflowmonitoring

� mqsireportflowmonitoring – Reports whether monitoring is active, and name of monitoring profile– report all configured event sources for a single message flow– report all available event sources in a single message flow– Export the current monitoring properties as a monitoring profile.– Export the current monitoring properties as a monitoring profile.– If monitoring profile is in use, registry contents are written to file– If node properties are in use, XML is constructed from them– Tip: Use this to easily construct a monitoring profile, rather than hand-crafting

it in a schema editor.

Notes: Example output from mqsireportflowmonitoring

Example output from mqsireportflowmonitoring command with –n option

BIP8911I: Monitoring settings for flow 'TotalPurchaseOrderFlow'

in execution group 'EventsEmitter.1' - State?: active, ProfileName: ''.

BIP8912I: Event: 'InputOrder.transaction.Start', Event name: 'InputOrder.Trans

ctionStart', Configured?: yes, State?: enabled

BIP8912I: Event: 'InputOrder.transaction.End', Event name: 'InputOrder.TransacBIP8912I: Event: 'InputOrder.transaction.End', Event name: 'InputOrder.Transac

ionEnd', Configured?: yes, State?: enabled

BIP8912I: Event: 'InputOrder.transaction.Rollback', Event name: 'InputOrder.Tr

nsactionRollback', Configured?: yes, State?: enabled

BIP8912I: Event: 'GoldOrderTotal.terminal.in', Event name: 'GoldOrderTotal.InT

rminal', Configured?: yes, State?: enabled

BIP8912I: Event: 'RegularOrderTotal.terminal.in', Event name: 'RegularOrderTot

l.InTerminal', Configured?: yes, State?: enabled

BIP8071I: Successful command completion.

Agenda





�Data Recording – provides view of in-flight data for transfers in MB connected network

�Replay Information – to target applications for

Message Broker Version 8 Record and Replay

� New Message Broker feature to enable record and replay in-flight data

� Universal support for file, messaging, web service, ERP & other data

� Browser tooling for access to end users, auditors and administrators

� Multiple data formats (binary, text, XML)

�Replay Information – to target applications for resubmission or redelivery processing

�Multiple sources and targets – WMB handles messages, databases, SAP, JMS and more…

�Broad Range of Data Formats – Handles binary, text and XML message formats

�Web based tooling – graphical facilities for record, edit and replay; no need to program

�Secure – sensitive information can be encrypted on the wire as necessary

�Source and Target Independent - information providers and consumers do not require change

�Simple Deployment – Integrates with existing infrastructure

Record and ReplayWebSphere Message Broker

……

Message Database JMS SAP

replayrecord

High level lifecycle

� A TRIGGER occurs that causes message details to be enqueued

� The broker will RECORD the message in a database

� Web user can QUERY saved messages

1

2

3 � Web user can QUERY saved messages

� Web user can REPLAY messages to the same or a different source

3

4

Data source(ODBC)

2

1 Message Flow(s)

WMB V8 Broker

Web User

HTTP/ REST

3

4

Business Activity Monitoring – message emission

� WMB’s existing Business Monitoring framework provides a way of capturing messages

– Configure event emission at design time or operationally

– Customisable payload allows you to capture entire bit-streams or only the message elements you require

– Messages are emitted to a queue

� The ‘Record’ part of this feature builds on this BAM framework

1

BAM framework

Message record

� Configure the broker to listen for these emitted messages

– Execution groups will asynchronously copy the messages to a target data source

– Execution group properties (via configurable service):

2

Monitoring Topic Target data source$SYS/#/FLOW1 AUDITDB$SYS/#/FLOW2 AUDITDB… …

� Scalable solution with flexible recording from multiple execution groups to multiple datasources

� Design allows for future implementation of tracking and auditing of messages that are not sent through WMB (e.g. XI50, custom applications)

… …

Querying messages� Connect to the broker from a web browser to list

available messages

� Sophisticated filters to narrow down to just the messages you are interested in

– e.g. by transaction, by message ID, by time

� Fully secured access (including encryption) using existing facilities (e.g. HTTPS)

– Design allows for future role based access to transactions and destinations.

� Uses a new public REST API (more on this later)

3

� Uses a new public REST API (more on this later)– This allows you to develop your own web

interfaces, for custom branding and advanced query/edit scenarios

Replaying messages

� Web UI provides an option to submit selected message(s) to an MQ queue– for reprocessing in a message flow (either the same or different)– for retransmission directly to the destination application

� Destinations predefined by administrator– Named set of MQ queues

4

� User selects message(s) and the target destination, and the broker will redeliver automatically.

� Remember that not all of the message payload may have been captured– If you replay messages back to the original point of capture, or to a target application

make sure that you record the entire bitstream!

Replaying messages to an application

� Replay messages directly to an application – Request redelivery from the web UI or custom REST application– The target application can request redelivery direct from the broker– Intermediate message flow not required!

4

WMBHTTP/ REST

Destination Name

Destination Type

Destination parameters

MYDEST MQ MYTARGETAPP.INPUTQ

… …

“Deliver MSGID ‘x’ to destination ‘MYDEST’WMB

REST

Replaying messages to a message flow

� Replay to the same or a different node– Define available entry points operationally

� Replay scenarios may require a different path through the message flow

– You may need to write your message flow differently to cope with replay

4

Destination Name

Destination Type

Destination parameters

MYDEST2 MQ myeg / myflow / ‘Replay Input’ node

… …

Flexible system design

� Supports full multiplicity of all components– Can configure multiple recorders, re-players, viewers– Highly scalable– Minimal performance impact

� Event emission from v7 brokers supported

� Different datasources to support different QoS

Broker 1 RecordingBroker 1

Broker 2

Broker nViewingBroker

RecordingBroker

REST

Events

New Web Interface

� In V7, the broker includes a HTTP listener for remote administration of service proxies (service federation management)

� V8 has extended this listener to include general broker administration capabilities

– Including providing the interface for Record

WMB V8 Broker

HTTP/ REST– Including providing the interface for Record

and Replay

� Accessed through a zero-footprint general purpose administration web UI

– Initially provided as a product sample– Built on Record and Replay web UI

� The interface this web UI uses will be made an external product interface and use standard technology throughout

– e.g. REST – Complements existing CMP interface

Web User

REST

The REST Administration Interface

� The web browser communicates with the broker through a series of REST verbs– The URI identifies the object being manipulated– Options on the URI describe parameters addressing the object (e.g. filters)– HTTP payload provides details of the object– Lists of objects returned

WMB V8 Broker

http/s port

Web User

GET /admin/eg/MYEGNAME

POST /admin/messages/MYDATASOURCE/MSGID

Summary

� Message Broker has built-in support for Business Activity Monitoring– Designed for WebSphere Business Monitor / CBE integration– Highly configurable– Can be administered without the toolkit– Also supports audit, capture/replay scenarios

� WMB Record and Replay addresses significant recurring problemsWMB Record and Replay addresses significant recurring problems– Create or populate System of Record– Repair malformed data for resubmission without original data source– Replay historical data to target systems for reprocessing

� Feature Rich– Record in-flight data: messages, files, services…– Query, View & Edit captured data – Replay data on demand

� REST API for zero-footprint administration

Copyright and Trademarks

© IBM Corporation 2012. All Rights Reserved.

IBM, the IBM logo, ibm.com are trademarks or registered trademarks of International Business Machines Corp.,

registered in many jurisdictions worldwide. Other product and service names might be trademarks of IBM or other companies. service names might be trademarks of IBM or other companies.

A current list of IBM trademarks is available on the Web at “Copyright and trademark information” at

www.ibm.com/legal/copytrade.shtml.

201207 - esb01 - wmb monitoring auditing record and replay

Documents

monitoring

websphere

destination

configuring

event bitstream

event sources

monitoring

business monitor