royal cyber inc
TRANSCRIPT
Tuning WebSphere Application Server ND 7.0
Royal Cyber Inc.
Application server stops responding• Server crash• Hung process• Out of memory condition
Performance degradation• Check if the process ID is continually changing
o Indicates the application server is probably crashing and being restartedCore filesJavacore filesVerboseGC logsHeap dump
JVM related problems
The tuning process is iterative in nature. To find the optimal configuration several tests and evaluation should be performed
Tuning Methodology
IBM Monitoring and Diagnostic Tools for Java - Garbage Collection and Memory Visualizer
JVM Analysis Tools
IBM Pattern Modeling and Analysis Tool for Java Garbage Collector
JVM Analysis Tools
WebSphere process hang detection stepsa) Once a hang is suspected, obtain a thread dump or javacoreb) For a typical hang, collect three dumps at a few minutes interval
To see if anything is moving within the process (but slowly)
a) Examine the thread dumps with Thread Analyzer or by handHang detection toolsThreadMonitor architecture was created to monitor thread pools withinThreadAnalyzerGUI-based tool Gathers and analyzes thread dumps from a WebSphere Application ServerProvides recommendations based on analysis
Troubleshooting server hang issues
ThreadAnalyzer
ThreadAnalyzer: Analysis
ThreadAnalyzer: Multiple dump analysis
ThreadAnalyzer: Overall thread analysis
SymptomsProcess terminated with Java exception or native signal
Usual causesOut of memory exceptionCall stack overflowUnexpected exception (for example, out of disk space) Optimizer failure (for example, JiT)Bad JNI call or library problemSegmentation violations while executing native code
How to troubleshoot crashes
Dump Analyzer: Analysis completed
Java virtual machine error Not enough memory to allocate an object can be caused by the following:The Java heap is too smallMemory is available in the heap, but it is fragmented (for JDK 1.4.2 and earlier)Memory leak in the Java codeNot enough space in the native memory
How to troubleshoot Out Of Memory error?
Administrative Console TPV Graph
PMAT: Summary page
PMAT: Analysis and recommendations
PMAT: Chart view
Each task performed by the server runs on a thread obtained from one of WebSphere Application
Server’s many thread pools
Three of the most commonly used (and tuned) thread pools within the application server are:
• Web container: Used when requests come in over HTTP
• Default: Used when requests come in for a message driven bean or if a particular transport
chain has not been defined to a specific thread pool.
• ORB: Used when remote requests come in over RMI/IIOP for an enterprise bean from an
EJB application client, remote EJB interface, or another application server
Thread pool size
A good practice is to use 5 threads per server CPU core for the default thread pool10 threads per server CPU for the ORB and Web container thread pools. For a machine with up to 4 CPUs, the default settings are usually a good start for most applications
There could be situations where the thread pool size might need to be increased to account for slow I/O or long running back-end connections
Default Thread Pool Setting
The IBM Tivoli® Performance Viewer is a tool embedded in the administrative console that lets you view the PMI (Performance Monitoring Infrastructure) data associated with almost any server component.The viewer provides advice to help tune systems for optimal performance and recommends alternatives to inefficient settings.
Monitoring and Tuning of Thread Pools
Each time an application attempts to access a back-end store (such as a database), it requires
resources to create, maintain, and release a connection to that data store.
PMI Metrics before tuning connection pool
Monitoring and Tuning Connection Pool
PMI Metrics after tuning connection pool
Monitoring and Tuning Connection Pool
• Data source statement cache size specifies the number of prepared JDBC statements that can be
cached per connection.
• Review the application code (or an SQL trace gathered from the database or database driver) for
all unique prepared statements, and ensure the cache size is larger than that value.
• Iteratively increase the cache size and run the application under peak steady state load until the
PMI metrics report no more cache discards.
Monitoring and Tuning Data source statement cache size
• PMI metrics after tuning the data source statement cache
Monitoring and Tuning Data source statement cache size
Five steps for all messaging issues• Check that the ME is in a started state
Debug steps for Websphere Default Messaging
Check SystemOut.log for the specific exception we are trying to debug.
Check the FFDCs for further details on the specific exception we are trying to debug.
Check the configuration.
Trace and engage IBM support.