ibm bpm problem determination
TRANSCRIPT
![Page 1: Ibm bpm problem determination](https://reader034.vdocument.in/reader034/viewer/2022050802/55cd2a43bb61ebca558b469e/html5/thumbnails/1.jpg)
IBM BPM Problem Determination
http://www.coutureconsulting.com/
![Page 2: Ibm bpm problem determination](https://reader034.vdocument.in/reader034/viewer/2022050802/55cd2a43bb61ebca558b469e/html5/thumbnails/2.jpg)
Overview
![Page 3: Ibm bpm problem determination](https://reader034.vdocument.in/reader034/viewer/2022050802/55cd2a43bb61ebca558b469e/html5/thumbnails/3.jpg)
Websphere application server architecture
![Page 4: Ibm bpm problem determination](https://reader034.vdocument.in/reader034/viewer/2022050802/55cd2a43bb61ebca558b469e/html5/thumbnails/4.jpg)
Network deployment concepts
![Page 5: Ibm bpm problem determination](https://reader034.vdocument.in/reader034/viewer/2022050802/55cd2a43bb61ebca558b469e/html5/thumbnails/5.jpg)
Runtime Flow
![Page 6: Ibm bpm problem determination](https://reader034.vdocument.in/reader034/viewer/2022050802/55cd2a43bb61ebca558b469e/html5/thumbnails/6.jpg)
Administration flow
![Page 7: Ibm bpm problem determination](https://reader034.vdocument.in/reader034/viewer/2022050802/55cd2a43bb61ebca558b469e/html5/thumbnails/7.jpg)
Clustered application servers
![Page 8: Ibm bpm problem determination](https://reader034.vdocument.in/reader034/viewer/2022050802/55cd2a43bb61ebca558b469e/html5/thumbnails/8.jpg)
IBM BPM key databases● CMNDB
o BPEL instances● BPMDB
o BPD instances● PDWDB● The CMNDB and BPMDB are the runtime databases, which store
process instances information for BPD and BPEL● The default configuration when creating the BPE container through the
administrative console is CMNDB, but can be changed
More details: https://www.evernote.com/shard/s210/sh/70704404-0d24-42f0-9e1a-6231855b02af/1a3fd21649db5656ed4163f45d0b5948
![Page 9: Ibm bpm problem determination](https://reader034.vdocument.in/reader034/viewer/2022050802/55cd2a43bb61ebca558b469e/html5/thumbnails/9.jpg)
Tuning method
![Page 10: Ibm bpm problem determination](https://reader034.vdocument.in/reader034/viewer/2022050802/55cd2a43bb61ebca558b469e/html5/thumbnails/10.jpg)
Tuning considerations
![Page 11: Ibm bpm problem determination](https://reader034.vdocument.in/reader034/viewer/2022050802/55cd2a43bb61ebca558b469e/html5/thumbnails/11.jpg)
Problem determinationProblem analysis
1. review the log file (SystemOut.log, ffdc logs, jvm logs)
2. check product version, search message code with the product version
3. ibm support assistant, forum, bpm community , tech notes, ...
Reference link: http://www-01.ibm.com/support/docview.wss?uid=swg21569731
![Page 12: Ibm bpm problem determination](https://reader034.vdocument.in/reader034/viewer/2022050802/55cd2a43bb61ebca558b469e/html5/thumbnails/12.jpg)
Log files and locations
![Page 13: Ibm bpm problem determination](https://reader034.vdocument.in/reader034/viewer/2022050802/55cd2a43bb61ebca558b469e/html5/thumbnails/13.jpg)
Log or trace file format
![Page 14: Ibm bpm problem determination](https://reader034.vdocument.in/reader034/viewer/2022050802/55cd2a43bb61ebca558b469e/html5/thumbnails/14.jpg)
SystemOut.log
- Useful data such as software levels for the WebSphere Application Server binary files and the IBM Business Process Manager binary files
- A mix of informational, warning, and error messages
![Page 15: Ibm bpm problem determination](https://reader034.vdocument.in/reader034/viewer/2022050802/55cd2a43bb61ebca558b469e/html5/thumbnails/15.jpg)
SystemOut.log examples
![Page 16: Ibm bpm problem determination](https://reader034.vdocument.in/reader034/viewer/2022050802/55cd2a43bb61ebca558b469e/html5/thumbnails/16.jpg)
FFDC (first-failure data capture)
file naming convention: <server name>_<thread ID>_yy.MM.dd_HH.mm.ss_<unique ID>.txt
and contains information that is relative to the value of the ffdcRun.properties Level property value
![Page 17: Ibm bpm problem determination](https://reader034.vdocument.in/reader034/viewer/2022050802/55cd2a43bb61ebca558b469e/html5/thumbnails/17.jpg)
ffdc in logs folder example
![Page 18: Ibm bpm problem determination](https://reader034.vdocument.in/reader034/viewer/2022050802/55cd2a43bb61ebca558b469e/html5/thumbnails/18.jpg)
ffdc content example
![Page 19: Ibm bpm problem determination](https://reader034.vdocument.in/reader034/viewer/2022050802/55cd2a43bb61ebca558b469e/html5/thumbnails/19.jpg)
High Performance Extensible Logging (HPEL)• Performance
– Log and trace events are stored as binary data so performance is substantiallyfaster than default log and trace framework (basic mode)– Less impact to systems when trace is enabled• Extensions– HPEL includes information about which application each log or trace record isfrom– When used with cross-component trace, HPEL includes information about whichrequest each log or trace record is from– Developers can add their own log or trace record extensions– HPEL logViewer command enables filtering logs or trace data by extensionname and value• Filtering– HPEL logViewer command helps you filter logs and trace data by date, time,level, and thread ID
![Page 20: Ibm bpm problem determination](https://reader034.vdocument.in/reader034/viewer/2022050802/55cd2a43bb61ebca558b469e/html5/thumbnails/20.jpg)
High Performance Extensible Logging (HPEL)
HPEL on developerworks:http://www.ibm.com/developerworks/websphere/techjournal/1208_bourne/1208_bourne.html
![Page 21: Ibm bpm problem determination](https://reader034.vdocument.in/reader034/viewer/2022050802/55cd2a43bb61ebca558b469e/html5/thumbnails/21.jpg)
Problem diagnostic tips
- understanding the system architecture- timestamp in the log files (eg:
ttps://docs.google.com/spreadsheets/d/1c65eHb8mZTBeva2HOvmyuHv5ksl7UiEW-MLq3LTgzXE/pubhtml )
- count occurrence times of error message- guest in which context may cause the errors
![Page 22: Ibm bpm problem determination](https://reader034.vdocument.in/reader034/viewer/2022050802/55cd2a43bb61ebca558b469e/html5/thumbnails/22.jpg)
Kind of data should be collect for performance-related problem
- Thread dumps- Heap dumps- Java gc logging
http://www-01.ibm.com/support/docview.wss?uid=swg21114927- Event manager and process monitor- Custom solution code logging- Instrumentation logging
https://bp3.zendesk.com/hc/en-us/articles/202144583-Instrumentation-Monitor-page-in-IBM-BPM-the-missing-manual-part-I
)
![Page 23: Ibm bpm problem determination](https://reader034.vdocument.in/reader034/viewer/2022050802/55cd2a43bb61ebca558b469e/html5/thumbnails/23.jpg)
Troubleshooting JVM problems
Problem Determination - collect log files
1. verbose gc log, heap memory dump2. java core files
![Page 24: Ibm bpm problem determination](https://reader034.vdocument.in/reader034/viewer/2022050802/55cd2a43bb61ebca558b469e/html5/thumbnails/24.jpg)
Troubleshooting JVM problemsSymptoms analysis1. server crashes
- application server stop or exit unexpectedly1. hung process
- verify that app server process is running- threads might be deadlocked- looping in code
1. out of memory- errors and exceptions that are logged without process exit- at times, can process in unexpected process exit
4. performance degradation- app server might crash and the nodeagent restarts it- check to see whether the process ID is continually changing
![Page 25: Ibm bpm problem determination](https://reader034.vdocument.in/reader034/viewer/2022050802/55cd2a43bb61ebca558b469e/html5/thumbnails/25.jpg)
Troubleshooting JVM problems
garbage collection logs
● verbosegc is written to native_stderr.log (ibm jdk), native_stdout.log (sun solaris, hp-ux java package)
● to enable verbosegc log (http://www-01.ibm.com/support/docview.wss?uid=swg21114927 )
● tool to visualize http://www.ibm.com/developerworks/java/jdk/tools/gcmv/
![Page 26: Ibm bpm problem determination](https://reader034.vdocument.in/reader034/viewer/2022050802/55cd2a43bb61ebca558b469e/html5/thumbnails/26.jpg)
Troubleshooting JVM problemsheapdumps and javacores (javadumps, threaddumps)
heapdumps● heapdumps is dump of all the live objects that are on the java heap● heapdumps file (phd portable heap dump) - OutOfMemory error located in <profile root> dir
javacore ( javadump, threadump) ● all about java thread present, classes loaded, heap statistics● javacore is generated automatically by event of java crash or OutOfMemory error● manually:
o - kill -3 on unix or wsadmino cmd>wsadmin
wasdmin>set jvm [$AdminControl completeObjectName type=JVM,process=%1,*] > %FILENAME%wsadmin>$AdminControl invoke $jvm dumpThreads >> %FILENAME%location
● <profile root> for ibm jdk ● native_sdout.log solaris and hp-ux native_stdout.log
![Page 27: Ibm bpm problem determination](https://reader034.vdocument.in/reader034/viewer/2022050802/55cd2a43bb61ebca558b469e/html5/thumbnails/27.jpg)
Troubleshooting JVM problems
root profile location
![Page 28: Ibm bpm problem determination](https://reader034.vdocument.in/reader034/viewer/2022050802/55cd2a43bb61ebca558b469e/html5/thumbnails/28.jpg)
Troubleshooting JVM problemsmemory and heap analysis● memory analyzer tool● memory dump diagnostic for java● ibm heapanalyzer
verbosegc log analysis● ibm garbage collection and memory visualizer● ibm pattern modeling and analysis tool for java garbage collector
javacore analysis● ibm thread and monitor dump analyzer for java
![Page 29: Ibm bpm problem determination](https://reader034.vdocument.in/reader034/viewer/2022050802/55cd2a43bb61ebca558b469e/html5/thumbnails/29.jpg)
Troubleshooting JVM problems
jvm documentshttp://www.ibm.com/developerworks/java/jdk/docs.html
demo: java -jar /home/longnguyen/ccvietnam/workspace/costo/utils/ga456.jar
![Page 30: Ibm bpm problem determination](https://reader034.vdocument.in/reader034/viewer/2022050802/55cd2a43bb61ebca558b469e/html5/thumbnails/30.jpg)
Problem prevention and best practices
- set up monitoring and detect problems- create and maintain system architecture
diagram- establish baselines (system load time,
response time)- define a diagnostic collection plan- create maintenance plan
![Page 31: Ibm bpm problem determination](https://reader034.vdocument.in/reader034/viewer/2022050802/55cd2a43bb61ebca558b469e/html5/thumbnails/31.jpg)
Resources:course materials https://www.evernote.com/shard/s210/sh/8b02a558-639a-4e54-a610-3f247d8849c4/3ca31d18ebfe78df8e1106db4f6f623fibm knowledge center for trouble shooting http://www-01.ibm.com/support/knowledgecenter/SS964W/com.ibm.wbpm.admin.doc/topics/welcome_wps_tro.htmlibm waithttps://wait.ibm.com/ibm developer works
http://www-01.ibm.com/support/knowledgecenter/SSFPJS_8.5.0/ditamaps/ic-homepage-bpm.html
analyze java thread dumps
http://architects.dzone.com/articles/how-analyze-java-thread-dumps