ibm impact session ed addison nuts and bolts ws

© 2014 IBM Corporation

ACI-1787 - Debugging CICS Web Services in a Mobile AgeEd Addison – STSM L2 Support

Please Note

IBM’s statements regarding its plans, directions, and intent are subject to change or withdrawal without notice at IBM’s sole discretion.

Information regarding potential future products is intended to outline our general product direction and it should not be relied on in making a purchasing decision.

The information mentioned regarding potential future products is not a commitment, promise, or legal obligation to deliver any material, code or functionality. Information about potential future products may not be incorporated into any contract. The development, release, and timing of any future features or functionality described for our products remains at our sole discretion.

Performance is based on measurements and projections using standard IBM benchmarks in a controlled environment. The actual throughput or performance that any user will experience will vary depending upon many factors, including considerations such as the amount of multiprogramming in the user’s job stream, the I/O configuration, the storage configuration, and the workload processed. Therefore, no assurance can be given that an individual user will achieve results similar to those stated here.

2

CICS TS Feature Pack for Mobile Extensions V1.0

A fully supported addition to base CICS installed using SMP/E

Separately orderable from Shopz (PID 5655-Y48)

Common to CICS TS 4.2 and 5.1

Requires enabling APARs PM82511 (4.2) and PM82519 (5.1)

Most of the functionality is actually provided by the APARs to allow code reuse

3

CICS TS Feature Pack for Mobile Extensions V1.0 Capabilities

Expose existing CICS applications as JSON web services (bottom-up approach)

Create new RESTful interfaces to CICS applications (top-down approach)

Call existing JSON web services from your CICS applications

Create Java applications in Liberty applications using JAX-RS

4

Feature Pack Contents

USS - <pathprefix>/usr/lpp/cicsts/mobilefp• Shell scripts for DFHJS2LS and DFHLS2JS• json_schema

– JSON schema for IVP

• Pipelines– Sample pipeline conifg file

• Wsbind

– WSBIND file for IVP

Datasets• SDFHMOBI - DFHJS2LS and DFHLS2JS JCL procedures• SDFHMOBL – IVP load module• SDFHMOBS – Source, language structure and CSD defintions

for IVP

5

What is JSON

Javascript Object Notation

A lightweight, text-based data interchange format

• Typically half the size of an XML payload

Natively supported in Javascript

Commonly used in mobile applications

6

Producing JSON in CICS

Support for JSON web services is based closely on CICS support for SOAP web services so the development and deployment steps are very similar.

Decide whether to start with a language structure (bottom-up approach) or from a JSON schema (top-down approach)

Run the JSON assistant batch job, supplying the appropriate artifacts. This generates a WSBIND file.

Define and install a PIPELINE resource in CICS with an appropriate pipeline configuration file for JSON.

Copy the WSBIND file to the PIPELINE's web services directory, and perform a PIPELINE SCAN to generate the WEBSERVICE and URIMAP resources.

Your JSON web service is ready to process requests.

7

Differences Between JSON and SOAP

The content of a SOAP message is XML data, whereas a JSON message contains JSON data. JSON and XML are different encoding mechanisms for describing structured data. JSON tends to be a more efficient encoding mechanism, so a typical JSON message will be smaller than the equivalent XML message.

JSON is easy to integrate in JavaScript applications, but XML isn't. This makes JSON a preferred data format with many mobile application developers.

SOAP provides a mechanism to add Headers to a message, and a family of specifications for qualities of service (such as security configuration, and distributed transactions). JSON does not provide this mechanism, it instead relies on the services of the underlying HTTP network protocol. This results in fewer options for securing and configuring a workload.

SOAP web services have an explicit error format involving SOAP Fault messages. There's no equivalent for JSON.

8

Similarities Between JSON and SOAP

The CICS implementation of JSON is derived from the SOAP architecture, and shares many of the concepts and artifacts.

Both involve offline utility programs that assist with mapping application data to and from the external data representation. For SOAP there is DFHLS2WS and DFHWS2LS, for JSON there is DFHLS2JS and DFHJS2LS.

The deployment mechanism for both technologies involve a PIPELINE resource, a WEBSERVICE resource, and a URIMAP resource.

9

10

Problem One: Disabled Pipeline

11

Explorer View of Disabled Pipeline DFH0MOBI

Messages Received

ICH408I USER(USASSC9 ) GROUP(TSOUSER ) NAME(ADDISON, EDWARD A. ()

/u/ephoppe/IYNX43/PIPELINE/DFH0MOBI/jsonjavaprovider.xml

CL(DIRSRCH ) FID(00000001000000010000000000000000)

INSUFFICIENT AUTHORITY

DFHPI0705 E IYNX43 CICSUSER PIPELINE DFH0MOBI encountered an error writing the

configuration to the derived shelf /u/ephoppe/IYNX43/PIPELINE/DFH0MOBI/.

The response code from the HFS write was X'0000006F' and the reason code was

X'EF786015'.

DFHPI0709 E IYNX43 CICSUSER PIPELINE DFH0MOBI resolution failed because the XML configuration file cannot be

copied to the derived shelf.

12

Explorer view of Pipeline Definition

13

Note: Shelf Directory called out in message DFHPI0705 is /u/ephoppe

TSO BPXMTEXT

14

TSO BPXMTEXT Output

15

Change Pipeline Shelf Directory

16

Enabled Pipeline

17

Problem Two: DFHPI0735 Web Service Not Found

18

CEDA Install of DFH0MOBI Pipeline and URIMAP

ENTER COMMANDS NAME TYPE GROUP LAST CHANGE DFH0MOBI PROGRAM DFH0MOBI 16/08/13 21:02:54 DFH0MOBI URIMAP DFH0MOBI *n INSTALL SUCCESSFUL DFH0MOBI PIPELINE DFH0MOBI *n INSTALL SUCCESSFUL DFH0MOBI WEBSERVICE DFH0MOBI 16/08/13 21:57:54 SYSID=NX43 APPLID=IYNX43 RESULTS: 1 TO 4 OF 4 TIME: 16.24.41 DATE: 10/04/14 PF 1 HELP 2 SIG 3 END 4 TOP 5 BOT 6 CRSR 7 SBH 8 SFH 9 MSG 10 SB 11 SF 12 CNCL

19

Messages During dfh0mobi Web Service Creation

DFHRD0124 IYNX43 IYCNTC49 CICSUSER CEDA INSTALL PIPELINE(DFH0MOBI)

DFHPI0703 IYNX43 CICSUSER PIPELINE DFH0MOBI is about to scan the WSDIR directory.

DFHPI0901 IYNX43 CICSUSER New WEBSERVICE dfh0mobi is being created during a scan against PIPELINE

DFH0MOBI.

DFHPI0910 IYNX43 CICSUSER WEBSERVICE dfh0mobi within PIPELINE DFH0MOBI has been created.

DFHPI0915 IYNX43 CICSUSER WEBSERVICE dfh0mobi is now INSERVICE and is ready for use.

DFHPI0704 IYNX43 CICSUSER PIPELINE DFH0MOBI Implicit scan has completed. Number of wsbind files found in the

WSDIR directory: 000001. Number of successful WEBSERVICE creates: 000001.

Number of failed WEBSERVICE creates: 000000.

20

Explorer Display of Web Service

21

Attempt to run Mobile Feature Pack IVP

22

Message from MSGUSR

DFHPI0735 E IYNX43 Error 'com.ibm.cics.domains.DomainException:

Class: Dfhpiwrj, function: INQUIRE_WEBSERVICE,

response: EXCEPTION, reason: WEBSERVICE_NOT_FOUND'

occurred whilst configuring WEBSERVICE DFH0MOBI

within a JVMSERVER.

23

DFHPI0735 Web Service not found – What we know so far

Client downloaded Mobile Feature Pack

Client CEDA installed PIPELINE and URIMAP DFH0MOBI

Pipeline scan built Web Service dfh0mobi

CICS installed Web Service dfh0mobi and put it in-service

Attempt to run the Mobile IVP resulted in DFHPI0735 WEBSERVICE _NOT_FOUND

24

CEMT Inquire of Web Service dfh0mobi

I WEBSERVICE STATUS: RESULTS - OVERTYPE TO MODIFY Webs(dfh0mobi ) Pip(DFH0MOBI) Ins Pro(DFH0MOBI) SYSID=NX43 APPLID=IYNX43 RESPONSE: NORMAL TIME: 22.16.58 DATE: 02/04/14 PF 1 HELP 3 END 5 VAR 7 SBH 8 SFH 9 MSG 10 SB 11 SF

25

CEDA view PIPELINE for DFH0MOBI

26

OBJECT CHARACTERISTICS CICS RELEASE = 0680 CEDA View PIpeline( DFH0MOBI ) PIpeline : DFH0MOBI Group : DFH0MOBI DEScription : IVP PIPELINE FOR MOBILE FEATURE PACK STatus : Enabled Enabled | Disabled Respwait : Deft Default | 0-9999 COnfigfile : /u/usassc9/ephoppe/jsonjavaprovider.xml (Mixed Case) : : : : SHelf : /u/usassc9 (Mixed Case) : : : : Wsdir : /u/usassc9/mobilewsbind (Mixed Case) : SYSID=NX43 APPLID=IYNX43

WSBind file in OMVS

NOTE: This will create a lowercase dfh0mobi Web Service when the Pipeline Scan is completed. This is due to the name of the bind file in z/FS.

27

CEDA View URIMAP for DFH0MOBI

OBJECT CHARACTERISTICS CICS RELEASE = 0680 CEDA View Urimap( DFH0MOBI ) Urimap : DFH0MOBI Group : DFH0MOBI DEScription : IVP URL FOR MOBILE FEATURE PACK STatus : Enabled Enabled | Disabled USAge : Pipeline Server | Client | Pipeline | Atom | Jvmserver UNIVERSAL RESOURCE IDENTIFIER SCheme : HTTP HTTP | HTTPS POrt : No No | 1-65535 HOST : * (Mixed Case) : PAth : /DFH0MOBI (Mixed Case) : : ASSOCIATED CICS RESOURCES TCpipservice : ANalyzer : No No | Yes COnverter : TRansaction : CPIH PRogram : PIpeline : DFH0MOBI Webservice : DFH0MOBI (Mixed Case) ATomservice :

28

DFHPI0735 Web Service not found – What we know so far






Reviewing the URIMAP for DFHM0BI it has the Web Service coded as DFH0MOBI (Upper Case)

Reason for lower case dfh0mobi was due to DFH0MOBI PIPELINE SCAN finding dfh0mobi.wsbind in wsdir /u/usassc9/mobilewsbind

29

CEDA Install of DFH0MOBI WEBSERVICE

ENTER COMMANDS NAME TYPE GROUP LAST CHANGE DFH0MOBI PROGRAM DFH0MOBI 16/08/13 21:02:54 DFH0MOBI URIMAP DFH0MOBI 16/08/13 21:02:54 DFH0MOBI PIPELINE DFH0MOBI 31/03/14 20:21:30 DFH0MOBI WEBSERVICE DFH0MOBI *n INSTALL SUCCESSFUL SYSID=NX43 APPLID=IYNX43 RESULTS: 1 TO 4 OF 4 TIME: 16.24.41 DATE: 10/04/14 PF 1 HELP 2 SIG 3 END 4 TOP 5 BOT 6 CRSR 7 SBH 8 SFH 9 MSG 10 SB 11 SF 12 CNCL

30

Explorer Display of Web Services

31

Successful Execution of DFH0MOBI

32

Mobile Feature Pack run from Android Phone

33

DFHPI0735 Web Service not found – What we know






Reviewing the URIMAP for DFHM0BI it has the Web Service coded as DFH0MOBI (Upper Case)

Reason for lower case dfh0mobi was due to DFH0MOBI PIPELINE SCAN finding dfh0mobi.wsbind in wsdir /u/usassc9/mobilewsbind

Directions from Infocenter on installing Mobile Feature Pack:• Install the sample application: Edit the sample PIPELINE DFH0MOBI in group DFH0MOBI. Ensure that the

path to the configuration file, /pathprefix/usr/lpp/cicsts/mobilefp/samples/pipelines/jsonjavaprovider.xml, is correct for your installation.

• Edit the sample WEBSERVICE DFH0MOBI in group DFH0MOBI. Ensure that the path to the wsbind file, /pathprefix/usr/lpp/cicsts/mobilefp/samples/wsbind/dfh0mobi.wsbind, is correct for your installation.

• Install group DFH0MOBI (PROGRAM, PIPELINE, WEBSERVICE, and URIMAP).

34

35

Problem Three: JVM Server Hang

JVM Server Hang

Customer called Support Center indicating access to their JVM Server for Mobile Web Service stalled

36

IPCS Primary Menu

------------------- IPCS PRIMARY OPTION MENU ---------------------------------

OPTION ===>

0 DEFAULTS - Specify default dump and options

1 BROWSE - Browse dump data set

2 ANALYSIS - Analyze dump contents

3 UTILITY - Perform utility functions

4 INVENTORY - Inventory of problem data

5 SUBMIT - Submit problem analysis job to batch

6 COMMAND - Enter subcommand, CLIST or REXX exec

T TUTORIAL - Learn how to use the IPCS dialog

X EXIT - Terminate using log and list defaults

Enter END command to terminate IPCS dialog

F1=HELP F2=SPLIT F3=END F4=RETURN F5=RFIND F6=MORE F7=UP

F8=DOWN F9=SWAP F10=LEFT F11=RIGHT F12=CURSOR

Ed Addison 37

IPCS Default Menu

Ed Addison 38

------------------------- IPCS Default Values --------------------------------- Command ===> You may change any of the defaults listed below. The defaults shown before any changes are LOCAL. Change scope to GLOBAL to display global defaults. Scope ==> BOTH (LOCAL, GLOBAL, or BOTH) If you change the Source default, IPCS will display the current default Address Space for the new source and will ignore any data entered in the Address Space field. Source ==> DSNAME('USASSC1.JVMHANG.DUMP') Address Space ==> ASID(X'0044') Message Routing ==> NOPRINT TERMINAL Message Control ==> FLAG(WARNING) NOCONFIRM VERIFY Display Content ==> MACHINE REMARK REQUEST NOSTORAGE SYMBOL Press ENTER to update defaults. Use the END command to exit without an update.

IPCS Primary Menu


OPTION ===>

0 DEFAULTS - Specify default dump and options

1 BROWSE - Browse dump data set

2 ANALYSIS - Analyze dump contents

3 UTILITY - Perform utility functions

4 INVENTORY - Inventory of problem data

5 SUBMIT - Submit problem analysis job to batch

6 COMMAND - Enter subcommand, CLIST or REXX exec

T TUTORIAL - Learn how to use the IPCS dialog





Ed Addison 39

Problem Three - ST SYS

SYSTEM STATUS: Nucleus member name: IEANUC01 Sysplex name: SYSPLEX1 TIME OF DAY CLOCK: CCFC281A C867F408 04/10/2014 15:57:46.700415 local TIME OF DAY CLOCK: CCFC1AB1 8E27F408 04/10/2014 14:57:46.700415 GMT Program Producing Dump: SVCDUMP Program Requesting Dump: DFHKETCB Incident token: SYSPLEX1 MV23 04/10/2014 14:57:45.943332 GMT

Ed Addison 4141

Note: You want to know the exact time the dump was taken. You also want to know the difference between CICS and GMT time.

Some CICS Verbx Outputs are in GMT time and some are in Local.

42

CICS Verbexit options Keyword Functional area AI = 0|2 Autoinstall Model ManagerAP = 0|1|2|3 Application DomainAPS=<TASKID=nnnnn>AU = 0|2 CICS affinities utilityBA = 0|1|2|3 Business application managerBR = 0|1|2|3 3270 bridge CC = 0|2 CICS catalog domainCP = 0|2 Common Programming InterfaceCQ = 0|1|2 Auto install model managerCSA=0|2 CICS Common System AreaDB2=0|1|2|3 The CICS DB2 interfaceDD = 0|1|2|3 Directory DomainDH = 0|1|2|3 Document handling domainDLI = 0|2 CICS DL/I InterfaceDM = 0|1|2|3 Domain ManagerDP = 0|1|2|3 Debug Profiles managerDS = 0|1|2|3 Dispatcher DomainDU = 0|2 Dump DomainEC = 0|1|2|3 Event Capture domainEJ = 0|1 Enterprise JAVAEM = 0|1|2|3 Event manager domain for BTSEP = 0|1|2|3 Event Processing domainFCP=0|2 File Control ProgramFT = 0|1|2|3 CICS WEB InterfaceICP = 0|2 Interval Control ProgramIE = 0|1|2|3 IP ECI DomainII = 0|1|2|3 IIOPIND = 0|1|2|3 Page number indexes for outputIS = 0|1|2|3 IP Interconnectivity domainJCP=0|2 Journal Control Program KE = 0|1|2|3 CICS KernelLD = 0|1|2|3 Loader DomainLG = 0|1|2|3 Logger DomainLM = 0|1|2|3 Lock Manager domainME = 0|2 Message domainML = 0|1|2|3 Markup Language domainMN = 0|1|2|3 Monitoring domainMQ = 0|1|2|3 CICS-MQ interface

Keyword Functional area

MRO=0|2 CICS Multi-Region OperationNQ = 0|1|2|3 Enqueue ManagerOT = 0|1|2|3 Object Transaction DomainPA = 0|2 Parameter manager domainPCP=0|2 Program Control ProgramPCT=0|2 Program Control TablePG = 0|1|2|3 Program Manager DomainPI = 0|1|2|3 Pipeline DomainPR = 0|2 Partner Resource managementPT = 0|1|2|3 Partner DomainRD = 0|2 Resource definition managerRL = 0|1|2|3 Resource Lifecycle domainRM = 0|2 Recovery ManagementRS = 0|1|2|3 Region Status domainRX = 0|1|2|3 Recoverable EXCI domainRZ = 0|1|2|3 Request StreamsSH = 0|1 Scheduler services domain for BTSSJ = 0|1|2|3 JVM DomainSM = 0|1|2|3 Storage Manager domainSO = 0|1|2|3 Sockets domainSSA=0|2 Static Storage AreasST = 0|1|2|3 Statistics domainSZ = 0|1 Front End Programming InterfaceTCP= 0|1|2|3Terminal Control ProgramTDP= 0|1|2|3Transient Data ProgramTI = 0|1|2|3 Timer domainTMP=0|2 Table Manager ProgramTR = 0|1|2|3 Trace domainTRS=<trace selection parameters>TS = 0|1|2|3 Temporary Storage ProgramUEH=0|2 User Exit HandlerUS = 0|1|2|3 User DomainWB = 0|1|2 The web interfaceW2 = 0|1|2|3 Web 2.0 domainXM = 0|1|2|3 The transaction managerXRF=0|2 The extended recovery facilityXS = 0|1 Security Domain

Ed Addison 42

Problem Three - VERBX DFHPD680 ‘DS=3’DS_TOKEN KE_TASK T S F P TT RESOURCE RESOURCE_NAME W TIME OF TIMEOUT DTA AD ATTACHER MD SUSPAREA XM_TXN_TOKEN TYPE SUSPEND DUE (DSTSK) TOKEN 0D020005 14F65800 N S N N - USERWAIT WorkWait M 18:02:24.930 - 1A8E7200 XM 15028700 QR 14F70728 150287000000044C 0D060003 14F74800 N S N N - USERWAIT EMSTATUS M 14:53:34.060 - 1A8E7500 XM 15028B00 QR 14F7F728 15028B000000046C 0B080199 14F47100 N S P N - ICWAIT S 14:53:19.944 - 1A8E3680 XM 166AC300 QR 1A8E3680 166AC3000006902C 0D0A02C9 14F65100 N S P N - JVMTHRED DFH$AXIS S 14:53:34.992 - 1A8E7800 XM 166AC700 QR 1A8E7800 166AC7000006908C 0600A279 0A394080 N S P N - ENQUEUE FCDSRECD C 19:53:35.267 - 0E747080 XM 0B371340 QR 0A4FEC85 0B3713400011613C 0D080375 14F38800 N S P N - ICWAIT S 14:53:34.865 - 1A8E7680 XM 166AC900 QR 1A8E7680 166AC9000006907C 0604C013 0A359780 N S P N - ENQUEUE FCDSRECD C 19:51:10.696 - 0E747380 XM 0970A960 QR 0A4FEC85 0970A9600011204C 0606B575 0A3B1400 N S P N - ENQUEUE FCDSRECD C 19:44:57.451 - 0E747500 XM 0970A1B8 QR 0B348CEC 0970A1B80010048C 0B120167 14F56100 N S P N - ICWAIT S 14:53:34.559 - 1A8E3E00 XM 166AC500 QR 1A8E3E00 166AC5000006904C 06080CC5 0A3B1080 N S P N - ENQUEUE FCDSRECD S 19:44:53.971 - 0E747680 XM 0970A340 QR 0E747680 0970A3400010033C

==DS: TASKS USING OPEN TCBS SUMMARY DS_TOKEN KE_TASK XM_TXN_TOKEN TCB_ID DS_TCB MVS_TCB 0B080199 14F47100 166AC3000006902C T8000 1395B500 00000000 0B120167 14F56100 166AC5000006904C T800I 1395B000 00000000 0D080375 14F38800 166AC9000006907C T800J 13890E00 00000000

Note: ST SYS showed dump was taken at 14:57:45.943332 Task waiting in JVMTHRED suspend for over four minutes – Needs a thread to JVM server DFH$AXIS Majority of user tasks are in ICWAIT All owners of T8 TCBs are in ICWAIT

JVMTHRED: The thread limit for the JVM server has been reached, as specified in the THREADLIMIT attribute of the JVMSERVER resource. The JVM server must wait until a thread becomes available before starting another task. To reduce the frequency of waits for a particular JVM server, increase the value of the THREADLIMIT attribute for that JVMSERVER resource.

Ed Addison 4343

T8 TCB Thread Management

CICS uses the open transaction environment (OTE) to run JVM server work. Each task runs as a thread in the JVM server and is attached using a T8 TCB.

When CICS enables a JVM server, the JVM server runs on a Language Environment process thread. This thread is a child of the TP TCB. Every CICS task is attached to a thread in the JVM using a T8 TCB.

• You can control how many T8 TCBs are available to the JVM server by setting the THREADLIMIT attribute on the JVMSERVER resource.

The T8 TCBs that are created for the JVM server exist in a virtual pool and cannot be reused by another JVM server running in the same CICS region.

• The maximum number of T8 TCBs that can exist in a CICS region across all JVM servers is 2000

• The maximum for a specific JVM server is 256.

44

CEMT Display of JVM Server

45

I JVMSERVER(DFH$AXIS) STATUS: RESULTS - OVERTYPE TO MODIFY Jvm(DFH$AXIS) Ena Prf(DFHJVMAX) Ler(DFHAXRO ) Threadc(003) Threadl( 003 ) Cur(7758784) SYSID=NX43 APPLID=IYNX43 RESPONSE: NORMAL TIME: 13.58.41 DATE: 11/04/14 PF 1 HELP 3 END 5 VAR 7 SBH 8 SFH 9 MSG 10 SB 11 SF

JVM Server Hang – What We Know So Far

Client called support indicating access to the JVM Server for Mobile Web Services stalled

Dump was taken at 14:57:45.943332 GMT

Task 06908 was in a wait for DFH$AXIS JVM Server Thread for four minutes

CEMT Display of JVM Server showed three threads defined and all threads used

Which tasks own the JVM Server Threads?

46

Problem Three - DFHPD680 ‘SJ=3’

47

SJ JSR fields JVMSERVER Name : DFH$AXIS JVM Process ID : 050303E8 LE Runtime Options : DFHAXRO JVM Profile : DFHJVMAX State : enabled resolved

GC Policy : -Xgcpolicy:gencon Initial Heap Size : 00000000 01000000 Max Heap Size : 00000000 10000000 Current Heap Size : 00000000 0098B808 Peak Heap Size : 00000000 00AC8980 IPT TCB Token : 13890C00 00000001 Thread Limit : 00000003 Current Threads : 00000003 Thread HWM : 00000003 Thread Wait Count : 00000001 Thread Number : 00000001 TCB Token : 1395B500 00000001 Thread Number : 00000002 TCB Token : 1395B000 0000000B Thread Number : 00000003 TCB Token : 13890E00 00000009

Task Token : 0B080199 Task Token : 0B120167 Task Token : 0D080375

NOTE: Take Task Token back to DS=3 Output to find transactions owning the threads

Problem Three - VERBX DFHPD680 ‘DS=3’

DS_TOKEN KE_TASK T S F P TT RESOURCE RESOURCE_NAME W TIME OF TIMEOUT DTA AD ATTACHER MD SUSPAREA XM_TXN_TOKEN TYPE SUSPEND DUE (DSTSK) TOKEN 0D020005 14F65800 N S N N - USERWAIT WorkWait M 18:02:24.930 - 1A8E7200 XM 15028700 QR 14F70728 150287000000044C 0D060003 14F74800 N S N N - USERWAIT EMSTATUS M 14:53:34.060 - 1A8E7500 XM 15028B00 QR 14F7F728 15028B000000046C 0B080199 14F47100 N S P N - ICWAIT S 14:53:19.944 - 1A8E3680 XM 166AC300 QR 1A8E3680 166AC3000006902C 0D0A02C9 14F65100 N S P N - JVMTHRED DFH$AXIS S 14:53:34.992 - 1A8E7800 XM 166AC700 QR 1A8E7800 166AC7000006908C 0600A279 0A394080 N S P N - ENQUEUE FCDSRECD C 19:53:35.267 - 0E747080 XM 0B371340 QR 0A4FEC85 0B3713400011613C 0D080375 14F38800 N S P N - ICWAIT S 14:53:34.865 - 1A8E7680 XM 166AC900 QR 1A8E7680 166AC9000006907C 0604C013 0A359780 N S P N - ENQUEUE FCDSRECD C 19:51:10.696 - 0E747380 XM 0970A960 QR 0A4FEC85 0970A9600011204C 0606B575 0A3B1400 N S P N - ENQUEUE FCDSRECD C 19:44:57.451 - 0E747500 XM 0970A1B8 QR 0B348CEC 0970A1B80010048C 0B120167 14F56100 N S P N - ICWAIT S 14:53:34.559 - 1A8E3E00 XM 166AC500 QR 1A8E3E00 166AC5000006904C 06080CC5 0A3B1080 N S P N - ENQUEUE FCDSRECD S 19:44:53.971 - 0E747680 XM 0970A340 QR 0E747680 0970A3400010033C

==DS: TASKS USING OPEN TCBS SUMMARY DS_TOKEN KE_TASK XM_TXN_TOKEN TCB_ID DS_TCB MVS_TCB 0B080199 14F47100 166AC3000006902C T8000 1395B500 00000000 0B120167 14F56100 166AC5000006904C T800I 1395B000 00000000 0D080375 14F38800 166AC9000006907C T800J 13890E00 00000000

Task Token 0B080199 is associated with transaction number 6902Task Token 0B120167 is associated with transaction number 6904Task Token 0D080375 is associated with transaction number 6907

Note: ST SYS showed dump was taken at 14:57:45.943332 Task waiting in JVMTHRED suspend for over four minutes – Needs a thread to JVM server DFH$AXIS Majority of user tasks are in ICWAIT and FCDSRECD Enqueue All owners of T8 TCBs are in ICWAIT

JVMTHRED: The thread limit for the JVM server has been reached, as specified in the THREADLIMIT attribute of the JVMSERVER resource. The JVM server must wait until a thread becomes available before starting another task. To reduce the frequency of waits for a particular JVM server, increase the value of the THREADLIMIT attribute for that JVMSERVER resource.

Ed Addison 4848

FIND 0B080199

Problem Three – VERBX DFHPD680 ‘WB’

49

==WB: WEB DOMAIN WEBREQUEST BLOCK SUMMARY

Task No Service Program Userid Clientaddress HTTP 32K+ Recv Complete API Send Analyzer _______________________________________________________________________________________________ 06904 EXMPPORT DFH0MOBI CICSUSER 9.27.47.174 Y N Y N DFHWBAAX 06908 EXMPPORT DFH0MOBI CICSUSER 9.27.46.169 Y N Y N DFHWBAAX 06907 EXMPPORT DFH0MOBI 9.27.46.236 Y N Y N DFHWBAAX 06902 EXMPPORT DFH0MOBI CICSUSER 9.27.47.95 Y N Y N DFHWBAAX

Problem Three - VERBX DFHPD680 ‘KE=3’

50

===KE: Kernel Domain KE_TASK Summary KE_NUM KE_TASK STATUS TCA_ADDR TRAN_# TRANSID DS_TASK KE_KTCB ERROR 0001 137F5000 KTCB Step 00000000 00000000 138390380002 137F5680 KTCB QR 00000000 1383DE00 1383C2000003 13804000 KTCB RO 00000000 1383DF00 1383B1680004 13804680 KTCB FO 00000000 13884F00 1383A0D000EF 14F38800 Not Running 13A9F700 06907 CPIH 1A8E7680 1383C20000F0 14F47100 Not Running 13A9E700 06902 CPIH 1A8E3680 1383C20000F1 14F47800 Not Running 13A9D100 06883 CEDA 19E03080 1383C20000F2 14F56100 Not Running 13A9E100 06904 CPIH 1A8E3E00 1383C20000F3 14F56800 ***Running** 13A9D700 06910 CEMT 1A8E6800 1383C20000F4 14F65100 Not Running 13A9F100 06908 CPIH 1A8E7800 1383C20000F5 14F65800 Not Running 13A9C100 00044 COI0 1A8E7200 1383C200

Transaction 06902: TRANSID CPIH: TCA Address: 13A9E700 – Owns JVM Thread Transaction 06904: TRANSID CPIH: TCA Address: 13A9E100 – Owns JVM Thread Transaction 06907: TRANSID CPIH: TCA Address: 13A9F700 – Owns JVM ThreadTransaction 06908: TRANSID CPIH: TCA Address: 13A9F100 – Waiting for JVM Thread

FIND 6902

JVM Server Hang – What We Know So Far





Transactions 06902, 06904 and 06907 own the JVM Server Threads and are in ICWAIT

Who put the tasks in an ICWAIT?

51

Finding Information About the Request

Ed Addison 52

TCA.06902 13A9E700 Task Control Area (User Area) 0000 13A9E800 00000000 00000000 00041048 00000000 14DB5C90 00000000 00000000 *.zY...................*.......* 0020 00000000 0006902C 00000000 00000000 00000000 00000000 008B2000 15120048 *..............................* 0040 95241692 0008ACBC 00000000 95241528 14E9CDE0 00000000 20000000 00000000 *n..k........n....Z.\..........* 0060 C4C6F0F0 F2F6F1C1 0000500C 00000000 00000000 00000000 00000000 00000000 *DF00261A..&...................*

SYSEIB.06902 13A9EA80 System EXEC Interface Block -0008 5CE2E8E2 C5C9C240 * *SYSEIB * 0000 0155319C 0114100F C3D7C9C8 0006902C 00000000 00000000 00000010 04000000 *........CPIH..................* 0020 00000000 00000000 00000000 00000000 00000040 40404040 40404000 00000000 *................. .....* 0040 00000000 00000000 00000000 00000000 00000000 00 *................... *

EIUS.06902 15120008 EXEC Interface User Structure 0000 00E86EC4 C6C8C5C9 E4E24040 40404040 15120668 00000000 151268C0 00000000 *.Y>DFHEIUS ...........{..* 0020 00000000 00000000 00000000 00000000 00000000 15120100 00000000 1512BB28 *..............................* 0040 00000000 00000000 151265E0 9409DCC6 00540F20 1409DDF8 14DFB840 14DFBA14 *...........\m..F.......8... ..* 0060 140A1388 14DFB040 00041048 14F47100 13A9E988 166AC300 140A1409 14DFB898 *...h... .....4...zZh..C......q* 0080 153EECA8 13A9E700 00000000 00000000 00000000 00000000 00000000 00000000 *...y.zX.......................* 00A0 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 *..............................* 00C0 00000000 00000000 00000000 00000000 00000000 00000000 1512003C 15120040 *............................. * 00E0 00000000 00000000 *........ * EIB.06902 15120100 EXEC Interface Block -0010 00656EC4 C6C8C1D7 6DC4C6C8 C5C9C25C * ..>DFHAP_DFHEIB** 0000 0155319C 0114100F C3D7C9C8 0006902C 00000000 00000000 00000034 16000000 *........CPIH..................* 0020 00000000 00000000 00000000 00000000 00000040 40404040 40404000 00000000 *................... .....* 0040 00000000 00000000 00000000 00000000 00000000 00 *................... * *

DFHPD680 ‘AP=3’

Application Register Savearea

EIB Function DELAY

If the EIB function code in the (User) EIB is different than the function code in the System EIB, then typically the currentEXEC CICS request has not yet completed.

The function code in the System EIB is the most current request. The current EXEC CICS request above is an EXEC CICS DELAY. The previous request was a PUT CONTAINER

EIB Function PUT CONTAINER

IPCS Primary Menu


OPTION ===>

********************

0 DEFAULTS - Specify default dump and options * USERID - USASSC1

1 BROWSE - Browse dump data set * DATE - 06/29/13

2 ANALYSIS - Analyze dump contents * JULIAN -

3 UTILITY - Perform utility functions * TIME - 16:46

4 INVENTORY - Inventory of problem data * PREFIX - USASSC1

5 SUBMIT - Submit problem analysis job to batch * TERMINAL- 3278

6 COMMAND - Enter subcommand, CLIST or REXX exec * PF KEYS - 24

T TUTORIAL - Learn how to use the IPCS dialog ********************





Ed Addison 53

Browse Mode – Register Savearea

Ed Addison 54

ASID(X'0043') Command ===> L 1512BB28

1512BB28 00104001 1512B980 | .. ..... |1512BB30 1512BCD0 96718612 00000000 1512CE54 | ...}o.f......... |1512BB40 1512BC28 1512CEC0 1512D010 16718060 | .......{..}....- |1512BB50 14DFB840 14DFB7F8 00000000 1512CF40 | ... ...8....... |1512BB60 1512CD18 16718154 16718314 1512AC50 | ......a...c....& |1512BB70 00000000 1512BCD0 00000000 00000000 | .......}........ |

Note: Registers 14 through 12 are stored in the Application Savearea at offset X’C’ when a CICS call is issued. Register 14 will point to where the CICS call was issued from. Register 1 will contain the parameters when the CICS call was issued.

Browse Mode – Register 14

Ed Addison 55

ASID(X'0043') ADDRESS(166100BE.) STORAGE --------------------------------------------Command ===> L 16718612 16718612 58C0 90E840F0 2008D210 D108A071 | .{.Y 0..K.J... |16718620 4130D108 5030D100 9680D100 4110D100 | ..J.&.J.o.J...J. |16718630 58F0A000 4100913C 58C09080 0DEF58C0 | .0....j..{.....{ |16718640 90E840F0 200847F0 B34C9120 D08447E0 | .Y 0...0.<j.}d.\ |16718650 B34C5830 905C58F0 30F44110 A1920DEF | .<...*.0.4..~k.. |16718660 5830915C 5B30C000 5030915C 91289054 | ..j*$.{.&.j*j... |16718670 4770B372 48F02008 58D0D004 58E0D00C | .....0...}}..\}. |

Note: Register 14 will point to where the CICS call was issued from. Backing up from Register 14 will tell you the module that made the call.

16718000 C4C6C8E8 C9F6F8F0 E3F0021C 001758F0 | DFHYI680T0.....0 |16718010 F0D058F0 F01458F0 F00C58FF 000C07FF | 0}.00..00....... |16718020 5CC6C9D3 D3C9D55C 47F0F028 00C3C5C5 | *FILLIN*.00..CEE |16718030 000001A8 00000014 47F0F001 98CEAC00 | ...y.....00.q... |16718040 167180DE 00000000 00000000 00000000 | ................ |16718050 00104001 00000008 C4C6C8F0 D4D6C2C9 | .. .....DFH0MOBI |

Backup from R14 to start of module

Browse Mode – Register 1

Ed Addison 56

ASID(X'0043') ADDRESS(1406B0.) STORAGE --------------------------------------------------- Command ===> L 1512BC28 1512BC28 1512BC60 1512D010 | ...-.... |1512BC30 1512BC80 1512BC88 1512BC90 1512BC98 | .......h.......q |

Note: Register 1 will point to the parameters when CICS was called.

ASID(X'0043') ADDRESS(1406B0.) STORAGE --------------------------------------------------- Command ===> L 1512BC60 1512BC60 10048010 07000420 80F0F0F0 F4F24040 | .........00042 |

ASID(X'0043') ADDRESS(16610148.) STORAGE --------------------------------------------------- Command ===> L 1512D010 1512D010 00000005 00000000 00000000 00000000 | ................ |

Parameters

First Parameter (EIB Function Code 1004 DELAY)

Second Parameter (Five Minutes)

JVM Server Hang – What We Know





Transactions 06902, 06904 and 06907 own the JVM Server Threads and are in ICWAIT

Program DFH0MOB1 issued EXEC CICS DELAY for 5 minutes

57

Summary

Pipeline Disabled

Web Service Not Found

JVM Server Hang

58

Useful Debugging Tools

Postman REST client plugin for Chrome• Send HTTP requests and headers

• Formats JSON responses

• Allows saving requests

• https://chrome.google.com/webstore/detail/postman-rest-client/fdmmgilgnpjigdojojpjoooidkmcomcm?hl=en

Online JSON validator• If JSON fails to validate in CICS, this tool will highlight the area of error

• http://jsonlint.com/

Online schema validator• Validate schema syntax – if schema is not accepted or produces strange results in

DFHJ2LS

• Validate input message against schema – check request structure if transformation errors occur

• http://json-schema-validator.herokuapp.com/

59

http://jsonlint.com/

Social media: Facebook, YouTube, Twitter, Aggregator

Visit the WebSphere and CICS Support Facebook page

• https://www.facebook.com/WebSphereSupport

Check out our CICS videos on YouTube

• http://www.youtube.com/user/CICSfluff

Follow CICS Support on Twitter

• http://www.ibm.com/support/docview.wss?uid=swg21384915 or https://twitter.com/IBM_CICS

See multiple channels by using the Social Media Aggregator

• https://www.ibm.com/social/aggregator/cics

Subscribe to the Quarterly Support Newsletter for CICS Transaction Server and CICS Tools

• http://www.ibm.com/support/docview.wss?uid=swg27006900 (on-line version)

See the Blog Post for CICS

• https://www.ibm.com/developerworks/community/blogs/aimsupport/tags/cics

60

http://www.youtube.com/user/CICSfluff

https://twitter.com/IBM_CICS

https://www.ibm.com/social/aggregator/cics

Software Services and Support Zone

61

Visit us in the Solution EXPOibm.com/software/expertise

Site features:

Client successes Specialized practices Services catalog Solution brochures Practitioner stories Video library Social connections Consultant profiles and more

Technologyto propel you

Expertiseto help you

Resourcesto educate you

Successesto assure you

Peopleto guide you

Questions?

Thank You

Legal Disclaimer

• © IBM Corporation 2014. All Rights Reserved.• The information contained in this publication is provided for informational purposes only. While efforts were made to verify the completeness and accuracy of the information contained

in this publication, it is provided AS IS without warranty of any kind, express or implied. In addition, this information is based on IBM’s current product plans and strategy, which are subject to change by IBM without notice. IBM shall not be responsible for any damages arising out of the use of, or otherwise related to, this publication or any other materials. Nothing contained in this publication is intended to, nor shall have the effect of, creating any warranties or representations from IBM or its suppliers or licensors, or altering the terms and conditions of the applicable license agreement governing the use of IBM software.

• References in this presentation to IBM products, programs, or services do not imply that they will be available in all countries in which IBM operates. Product release dates and/or capabilities referenced in this presentation may change at any time at IBM’s sole discretion based on market opportunities or other factors, and are not intended to be a commitment to future product or feature availability in any way. Nothing contained in these materials is intended to, nor shall have the effect of, stating or implying that any activities undertaken by you will result in any specific sales, revenue growth or other results.

• If the text contains performance statistics or references to benchmarks, insert the following language; otherwise delete:Performance is based on measurements and projections using standard IBM benchmarks in a controlled environment. The actual throughput or performance that any user will experience will vary depending upon many factors, including considerations such as the amount of multiprogramming in the user's job stream, the I/O configuration, the storage configuration, and the workload processed. Therefore, no assurance can be given that an individual user will achieve results similar to those stated here.

• If the text includes any customer examples, please confirm we have prior written approval from such customer and insert the following language; otherwise delete:All customer examples described are presented as illustrations of how those customers have used IBM products and the results they may have achieved. Actual environmental costs and performance characteristics may vary by customer.

• Please review text for proper trademark attribution of IBM products. At first use, each product name must be the full name and include appropriate trademark symbols (e.g., IBM Lotus® Sametime® Unyte™). Subsequent references can drop “IBM” but should include the proper branding (e.g., Lotus Sametime Gateway, or WebSphere Application Server). Please refer to http://www.ibm.com/legal/copytrade.shtml for guidance on which trademarks require the ® or ™ symbol. Do not use abbreviations for IBM product names in your presentation. All product names must be used as adjectives rather than nouns. Please list all of the trademarks that you use in your presentation as follows; delete any not included in your presentation. IBM, the IBM logo, Lotus, Lotus Notes, Notes, Domino, Quickr, Sametime, WebSphere, UC2, PartnerWorld and Lotusphere are trademarks of International Business Machines Corporation in the United States, other countries, or both. Unyte is a trademark of WebDialogs, Inc., in the United States, other countries, or both.

• If you reference Adobe® in the text, please mark the first use and include the following; otherwise delete:Adobe, the Adobe logo, PostScript, and the PostScript logo are either registered trademarks or trademarks of Adobe Systems Incorporated in the United States, and/or other countries.

• If you reference Java™ in the text, please mark the first use and include the following; otherwise delete:Java and all Java-based trademarks are trademarks of Sun Microsystems, Inc. in the United States, other countries, or both.

• If you reference Microsoft® and/or Windows® in the text, please mark the first use and include the following, as applicable; otherwise delete:Microsoft and Windows are trademarks of Microsoft Corporation in the United States, other countries, or both.

• If you reference Intel® and/or any of the following Intel products in the text, please mark the first use and include those that you use as follows; otherwise delete:Intel, Intel Centrino, Celeron, Intel Xeon, Intel SpeedStep, Itanium, and Pentium are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States and other countries.

• If you reference UNIX® in the text, please mark the first use and include the following; otherwise delete:UNIX is a registered trademark of The Open Group in the United States and other countries.

• If you reference Linux® in your presentation, please mark the first use and include the following; otherwise delete:Linux is a registered trademark of Linus Torvalds in the United States, other countries, or both. Other company, product, or service names may be trademarks or service marks of others.

• If the text/graphics include screenshots, no actual IBM employee names may be used (even your own), if your screenshots include fictitious company names (e.g., Renovations, Zeta Bank, Acme) please update and insert the following; otherwise delete: All references to [insert fictitious company name] refer to a fictitious company and are used for illustration purposes only.

64

http://www.ibm.com/legal/copytrade.shtml

ibm impact session ed addison nuts and bolts ws

Technology

json data

dfhls2js json

cics implementation

existing json web services

schema json schema

typical json message

cics web services

soap web services