ibm websphere mq for z/os - the inside story

A9: WebSphere MQ for z/OS The Inside Story

Damon Cross, Advisory Software [email protected]

© 2014 IBM Corporation © 2014 IBM Corporation

Please NoteIBM’s statements regarding its plans, directions, and intent are subject to change or withdrawal without notice at IBM’s sole discretion. Information regarding potential future products is intended to outline our general product direction and it should not be relied on in making a purchasing decision.

The information mentioned regarding potential future products is not a commitment, promise, or legal obligation to deliver any material, code or functionality. Information about potential future products may not be incorporated into any contract. The development, release, and timing of any future features or functionality described for our products remains at our sole discretion

Performance is based on measurements and projections using standard IBM benchmarks in a controlled environment. The actual throughput or performance that any user will experience will vary depending upon many factors, including considerations such as the amount of multiprogramming in the user’s job stream, the I/O configuration, the storage configuration, and the workload processed. Therefore, no assurance can be given that an individual user will achieve results similar to those stated here.

© 2014 IBM Corporation

Agenda

• Transactions• Queue managers• Resource managers• Scenario walk-through• Summary


Transactions

An attempt to change the state− Either fully completes or is fully restored to the original state− Like a contract− Atomicity, Consistency, Isolation, Durability

Business is all about transactions− e.g. Banking, Order processing, Inventory management

Transactional messaging− Once and only once delivery of messages

Coordination is essential for Extended Transactions− Transactions that update multiple resources


Agenda



The queue manager – delivering messages

APPLICATION

EXEC SQL QUERY

EXEC SQL INSERT

DATABASE

Read

Write

APPLICATION

MQGET

MQPUT

Read / Write

Write

MQ

DB/2 IMSCICS

z/OS

CLIENT

CLIENT

CLIENTMQI

QUEUE


Core technology – logging and buffering

...C3 A1B4

Conveyor Belt Log

A1

C3

B4

Pigeon Hole Page set

112

2

3

45

67

8

9

10

11

C3 A1B4

?


Accessing the MQI - stubs and adapters

APPLICATION

MQI STUB

RMI TRUE

CICS REGION

APPLICATION

MQI STUB

ADAPTER

BATCH / TSO

PROCESS

MQI STUB

ADAPTER

CHANNEL INITIATOR

APPLICATION

MQI STUB

IMS REGION (BMP/FP/MPP)

CONTROL

ESAF EXITS

QUEUE MANAGER

PROGRAM CALLS (QRPL)


Agenda



Building blocks – resource managers

CONNECTION

MANAGER

RECOVERY

MANAGER

LOG

MANAGER

MESSAGE

MANAGER

DATA

MANAGER

BUFFER

MANAGER

LOCK

MANAGER

CF

MANAGER

COMMIT / BACKOUT UR

MQI

CKPT

CKPT

REDO / UNDO

URRELEASE

MQOPEN


Handling applications – Connection Manager

APPLICATION

MQCONN

MQOPEN

MQPUT

…

TRANSACTION COORDINATOR

THREAD

EOT

EOMPHASE 1

PHASE 2

CONTROL BLOCKS

CONNECTION MANAGER

ADDRESS SPACE BOUNDARY

OTHER RESOURCE MANAGERS

DATA

LOGS

CONTROL BLOCKS

CONTROL BLOCKS


Supporting the MQI & MQSC – Message Manager

MESSAGE MANAGER

APPLICATIONCOMMAND, RUNTIME & GROUP SERVERS

MQSC CONSOLE

QSG

MQOPEN

MQCLOSE

MQGET

MQPUT

MQSET

MQINQ

MQSUB

OBJECTS

SYSTEM.CLUSTER.*

MQOPEN

BASE

VALIDATION, CONSISTENCY CHECKING, NAME RESOLUTION, TRIGGERING, GET-WAIT PROCESSING, PUT-TO-WAITING-GETTER


Managing messages and objects – Data Manager

112

23

4567

8910

11

SCAVENGER

LOG

REDO

UNDO

MESSAGES

OBJECTS

QUEUE ATTRIBUTES

SUB-QUEUE MESSAGE POINTERS

SPACE GROUP

SPACE GROUP

SPACE GROUP

PAGE SET 0

PAGE SETS 1--99


Improving performance – Buffer Manager

PAGE SET

BUFFER

PAGES

READ PAGE SET

DEFINE BUFFPOOL(bpid) BUFFERS(nnnn)

DEFINE PSID(psid) BUFFPOOL(bpid)

STEAL LRU

DIRTY

DIRTY

DIRTY

NO FORCECHECKPOINT,

STEAL, WRITE AHEAD

OLDESTWRITE PAGESET

FORCE DIRTY PAGES > 2 CHECKPOINTSCHECKPOINT


Local message queue storage

HEAD

START SCAN

TAIL

SUB-QUEUE (x 20)

PAGE SET 0

MSG 1 (DELETED)

MSG 2 (≤ 4 KB) MQMD + DATA

MSG 3 (≤ 4 MB)

MQMD + DATA NEXT DATA

MSG 4 (≤ 100 MB)

MQMD + DATA NEXT DATA NEXT DATA NEXT DATA

PAGE SET n


Local message queue storage

• Pages are generally deallocated as messages removed from queues.

•• Asynchronous deallocation is performed by the scavenger task

– Acquires exclusive locks on pages for all messages to be deleted.

•• Pure text pages don't suffer contention and can always be

deleted during MQGET processing.•• Page allocation for small (<4KB) messages changed in V6.•• Objects are stored much like messages on pageset 0.


Handling transactions – Recovery Manager

TRANSACTION STATES

112

2

3

4567

8

9

1011 112

2

3

4567

8

9

1011 112

2

3

4567

8

9

1011 112

2

3

4567

8

9

1011 112

2

3

4567

8

9

1011

1-PHASE COMMIT

1-PHASE BACKOUT

2-PHASE COMMIT

BEGIN

BEGIN

BEGIN

IN FLIGHT

IN FLIGHT

IN FLIGHT

IN COMMIT

IN BACKOUT

IN DOUBT

COMMITTED

BACKED OUT

IN COMMIT COMMITTED

LOGLOGLOAD (zPARM)

CHECKPOINT

LOG

PAGESET

RESTART

RECOVERY

/


Providing logging interfaces – Log Manager

• Log read and write functions

• Log shunting

• Multiple active log data sets and archive

• Archive inventory management

• Duplexed for reliability

• “Bootstrap” file – End of log location– Archive inventory– CSQJU004

• Various Utilities – CSQ1LOGP, CSQ4LOGS


Examining transactions – a log print 00000000D569 URID(00000000D569) RM(RECOVERY) TYPE( START UR ) **** 00640024 00200001 03000000 0000D569 00000000 D545 0000 00240000 0000D000 00000000 00000700 00000000 00000000 00000000 0000D6C4 0020 D6E6C4C1 4040B5B4 8FA08793 02864040 40404040 4040C2C1 E3C3C840 4040D6C4 0040 D6E6C4C1 40400000 00000000 0000

00000000D5CD URID(00000000D569) RM(DATA) LRID(00000000.00000E01) TYPE( UNDO REDO ) SUBTYPE( DECREMENT BY ) **** 002A0064 0600000F C9000000 0000D569 00000000 D569 0000 00000000 00000E01 00040326 00000001 00000001

00000000D5F7 URID(00000000D569) RM(DATA) LRID(00000001.00000201) TYPE( UNDO REDO ) SUBTYPE( DELETE ) **** 0026002A 06000008 C9000000 0000D569 00000000 D5CD 0000 00000001 00000201 00000000 00000E01

00000000D61D URID(00000000D569) RM(RECOVERY) TYPE( START COMMIT1 ) **** 007C0026 00200002 03000000 0000D569 00000000 D5F7 0000 00240000 0000D000 00000000 00000700 00000000 00000000 00000000 00004040 0020 40404040 40400000 00000000 00000000 00000000 00000000 00000000 00000000 0040 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 0060 00000000 0000

00000000D699 URID(00000000D569) RM(RECOVERY) TYPE( PHASE 1 TO 2 ) **** 0024007C 0020000C 03000000 0000D569 00000000 D61D 0000 00240000 0000D000 00000000 0000

00000000D6BD URID(00000000D569) RM(RECOVERY) TYPE( END COMMIT2 ) **** 00240024 00200010 03000000 0000D569 00000000 D699 0000 00240000 0000D000 00000000 0000

00000000D569 URID(00000000D569)

TYPE( END COMMIT2 )

RM(RECOVERY)

RM(DATA)SUBTYPE( DECREMENT BY )

SUBTYPE( DELETE )

LRID(00000000.00000E01)

LRID(00000001.00000201)

TYPE( START UR )

TYPE( UNDO REDO )

00000000D699 URID(00000000D569) RM(RECOVERY) TYPE( PHASE 1 TO 2 ) **** 0024007C 0020000C 03000000 0000D569 00000000 D61D0000 00240000 0000D000 00000000 0000


Concurrency and isolation – Lock Manager

MQOO_INPUT_SHARED

MQOO_INPUT_EXCLUSIVE

A

B

C

A B C

PAGE (shared)

MESSAGE (exclusive)

MESSAGE (exclusive)

112

2

3

4567

8

9

1011 112

2

3

4567

8

9

1011 112

2

3

4567

8

9

1011 112

2

3

4567

8

9

1011

REQUEST

COMMIT

ALLOCATION

MQOPEN

vs

DELETE QLOCAL

API Request

Transaction

Lifetime


Managing shared messages – CF Manager

QM1

CF MANAGER

LIST STRUCTURE

QM2

CF MANAGER

QM3

CF MANAGER

IXL*

DEFINE QLOCAL(...)

QSGDISP(SHARED)

CFSTRUCT(...)


Shared message queue storage

SQ1 LH

SQ2 LH

SQM1 LH

LIST STRUCTURE

COMMITTED PUTS

UNCOMMITTED GETS

UNCOMMITTED PUTS QUEUE DETAILS

EXPIRED MESSAGES

MQCMIT

MQCMIT

MQGET

(MQGET)

MQPUTMQGET

KEYS

DATA

State Priority Time Queue Manager

MsgId / CorrelationId

READ

WRITE

MOVE

DELETE

MONITOR


Agenda



Scenario: MQPUT to a triggered queueAPPLICATION MESSAGE MGR DATA MGR BUFFER MGR RECOVERY MGR LOG MGR LOCK MGR

MQOPEN

ACQUIRE LOCK

LOCATE QUEUE (HASH TABLE)

SECURITY CHECK

RESOLVE NAME

GET HANDLE

MQPUT

USE HANDLE

LOCATE NEW PAGE (FOR MESSAGE)

BUFFER PAGE

START UR LOG RECORDS

LOG RECORDS

CHECK TRIGGER RULES

MQCMIT

FORCE LOG

RELEASE LOCKS


Scenario: MQGET from a queueAPPLICATION MESSAGE MGR DATA MGR BUFFER MGR RECOVERY MGR LOG MGR LOCK MGR

MQOPEN

ACQUIRE LOCK

LOCATE QUEUE (HASH TABLE)

SECURITY CHECK

RESOLVE NAME

GET HANDLE

MQGET

USE HANDLE

FIND MESSAGE (INDEX / NEXT)

BUFFER PAGE

START UR LOG RECORDS

LOG RECORDS

MQCMIT

FORCE LOG

RELEASE LOCKS


Summary• Delivers transactional messaging

– Enables robust business applications• Complex, but well organised

– Adapters– Address spaces– Resource managers

• Designed for throughput, availability and scalability– Logging– Buffering– Locking– Communications

Questions?



For Additional Information

IBM Training http://www.ibm.com/training

IBM WebSphere http://www.ibm.com/software/websphere/ http://www.ibm.com/software/products/ibm-mq

IBM developerWorks http://www.ibm.com/developerworks/websphere https://www.ibm.com/developerworks/community/blogs/messaging

WebSphere forums and community http://www.ibm.com/developerworks/websphere/community/


IBM MQ Sessions this week

10:30 - 12:00 13:15 - 14:15 14:30 - 15:30 16:00 - 17:00 17:15 - 18:15

Tuesday

Opening General Session- IBM Digital Experience and WebSphere Technical University

Session A31: IBM MQ CHLAUTH rules – with MQ V8 updates

Speaker: Morag HughsonRoom 02

Session A4: WebSphere MQ for z/OS: Performance and Accounting

Speaker: Alexander RossRoom 8

Session I26: DataPower-MQ Connectivity Deep Dive (Theory)

Speaker: Robin WileyRoom 27

Session Z1: WebSphere MQ for z/OS V8: Latest Features Deep Dive

Speaker: Damon CrossRoom 6

9:00 - 10:00 10:30 - 11:30 11:45 - 12:45 14:00 - 15:00 15:15 - 16:15 16:45 - 17:45

Wednesday

Session Z5: WebSphere MQ for z/OS: Security


Session A21: What's New in IBM Messaging


Session C7: Messaging in the Cloud with IBM MQ Light and IBM Bluemix

Speaker: Rob NicholsonRoom 27

Session A17: Managing work-loads, scaling and availability with IBM MQ clusters

Speaker: David WareRoom 6

Lab IL5: DataPower-MQ Connectivity Deep Dive (Hands-On)

Speaker: Robin WileyRoom 7b

Session A9: WebSphere MQ for z/OS: The Inside Story


Thursday

Session A35: How to Develop Responsive Applications with IBM MQ Light

Speaker: Rob NicholsonRoom 27

Session A22: New IBM MQ V8 Security Features


Session A3: WebSphere MQ for z/OS: Shared Queues

Speaker: Alex RossRoom 6

Session A18: Using Publish /Subscribe with IBM MQ


Friday

Lab AL6: Developing a First Application with IBM WebSphere MQ Light

Speakers: Robert Nicholson, Alex RossRoom 7b

Session A16: Using IBM MQ Pub/Sub in an MQ network



Thank you!

ibm websphere mq for z/os - the inside story

Technology

important resource managers

queue manager implementation

simple transaction

potential future products

summary transactions

transaction look like

important considerations

future features