websphere technical university: top websphere problem determination features

A6: Top IBM WebSphere Application Server Problem Determination

Features

Chris BaileySTSM, IBM Runtime Monitoring and Diagnostics

© 2014 IBM Corporation

Please Note

IBM’s statements regarding its plans, directions, and intent are subject to change or withdrawal without notice at IBM’s sole discretion. Information regarding potential future products is intended to outline our general product direction and it should not be relied on in making a purchasing decision.

The information mentioned regarding potential future products is not a commitment, promise, or legal obligation to deliver any material, code or functionality. Information about potential future products may not be incorporated into any contract. The development, release, and timing of any future features or functionality described for our products remains at our sole discretion

Performance is based on measurements and projections using standard IBM benchmarks in a controlled environment. The actual throughput or performance that any user will experience will vary depending upon many factors, including considerations such as the amount of multiprogramming in the user’s job stream, the I/O configuration, the storage configuration, and the workload processed. Therefore, no assurance can be given that an individual user will achieve results similar to those stated here.

2

3

Introduction to the Speaker

Chris BaileySTSM, IBM Runtime Monitoring and Diagnostics Architect14 years experience developing and deploying Java SDKs 1 years experience developing and deploying Node.js SDKs

Recent work focus:Java and Node.js monitoring and diagnosticsJava and Node.js integration into the cloudHighly resilient and scalable deploymentsRequirements gathering

Contact Details:[email protected]://www.linkedin.com/in/chrisbaileyibmhttp://www.slideshare.net/cnbailey/@Chris__Bailey

mailto:[email protected]

http://www.linkedin.com/in/chrisbaileyibm

http://www.slideshare.net/cnbailey/

4

Problem Reduction

1. Eliminate Problemsremove the source of problems so that they no longer occur. For example, redesign a component to eliminate the need to configure something that customers have trouble with.

2. Improve Serviceabilitymake it easier for customers or support to service problems when they occur. For example, provide a way for customers to quickly check to see if there are any configuration problems.

EliminateProblems

ImproveServiceability

Goal: Reduce total problem hours

5

Serviceability Features

• Hang detection• First Failure Data Capture• High Performance Extensible Logging• Cross Component Trace• Memory Leak Detection and Prevention• Timed Operations• …

Serviceability Features

• Hang detection• First Failure Data Capture• High Performance Extensible Logging• Cross Component Trace• Memory Leak Detection and Prevention• Timed Operations• …

Component Improvements

• Security• Systems Management• Install• Plugin• Java™ and JDK• WebServices• ...

Component Improvements

• Security• Systems Management• Install• Plugin• Java™ and JDK• WebServices• ...

Problem Determination Tools

• IBM Support Assistant Team Server (with Automated Analysis)

• Garbage Collection and Memory Visualizer

• Memory Analyzer (with WAS extensions)

• Java Health Center• Thread and Monitor Dump Analyzer• Performance Tuning Toolkit• ...

Problem Determination Tools

• IBM Support Assistant Team Server (with Automated Analysis)

• Garbage Collection and Memory Visualizer

• Memory Analyzer (with WAS extensions)

• Java Health Center• Thread and Monitor Dump Analyzer• Performance Tuning Toolkit• ...

Problem reduction is driven from multiple directions

Component improvements reduce problem determination time for key components

Serviceability features simplify problem determination across the product

Problem determination tools make it easier to analyze complex issues

Problem Reduction

6

WAS V5.0.xFFDCCollector SummaryConfig ValidationConnection Manager ServiceabilityHang Detection in the Runtime

WAS V5.1.xSession data crossover assist

Socket Connect Timeouts

WAS v6.0.xJSR 47 LoggingCommon Base EventsFFDC AspectsFFDC Capability for z/OSClass Loader Viewer

WAS V6.1.x (*)Diagnostic ProvidersUnique Message IDs

WAS V7.0.x (*)FFDC ImprovementsDiagnostic Tooling Framework for Java (DTFJ)

OngoingServiceability Defects Process

to report and address specific serviceability issues encountered

in the field

(*) Several tooling-related projects were delivered as WAS Line Items in WAS 6.1 and 7.0 (F)=Full Profile (L)=Liberty Profile

WAS V8.5.x(F) Cross Component TraceISA DC integration(F) HPEL enhanced filtering(F) Memory leak detection and protection(L) Liberty Serviceability (logs, dumps, FFDC)

WAS V8.0.x High Performance Extensible Logging (HPEL)

Trigger dumps from consoleSensitive log and trace guard

JVM Serviceability Enhancements

WAS V8.5.5(L) Binary Logging

(L) Timed Operations(L) Sensitive Log and Trace GuardWAS Runtime Features

7

Liberty

© 2014 IBM Corporation8

Liberty Problem Determination

Server dump Command

Server javadump Command

TimedOperations Feature

Binary Logging


Server Dump Command

The server dump command is a utility you can run from the bin directory to dump and zip up important state information from the application server itself.

The server dump command collects:– Log files– Configuration information– Details of deployed applications

If the server is running when the command executes, it also collects:– State of each OSGi bundle in the server– Wiring information for each OSGi bundle in the server– Component list managed by the Service Component Runtime (SCR) environment– Detailed information of each component from SCR– Configuration administration data of each OSGi bundle– Information about registered OSGi services– Runtime environment settings such as Java™ virtual machine (JVM), heap size,

operating system, thread information, and network status

wlp\bin> server dump


Server JavaDump Command

The server javadump command is a utility you can run from the bin directory to collect javacores, heap dumps, and system cores.

– Collects a javacore from a running server. Javacores are helpful for diagnosing hangs, 100% CPU conditions, and deadlocks. Use tools such as the Thread and Monitor Dump Analyzer available in the IBM Support Assistant to analyze javacores.

– Collects a heap dump from a running server. Heap dumps are helpful for diagnosing memory leaks. Use tools such as Memory Analyzer available in the IBM Support Assistant to analyze heap dumps.

– Collects a system core from a running server. System cores are helpful for diagnosing memory leaks, or exploring specific data in your heap address space. Use tools such as Memory Analyzer, or Interactive Diagnostic Data Explorer available in the IBM Support Assistant to analyze system cores.

wlp\bin> server javadump

wlp\bin> server javadump –-include=heap

wlp\bin> server javadump --include=system


Timed Operations

Timed operations is a new feature that tracks the duration of JDBC operations running in the application server

–Logs a warning when operations take more or less time to execute than expected

–Periodically creates a report in the application server log, detailing which operations took longest to execute

– When running the server dump command, it will generate a report containing information about all operations it has tracked

–The information listed in these reports can be used to decide if anything has an unusual behavior

–Recommended for production environments to track slowdowns when they first occur


Timed Operations

Sample logged message

Sample automatically generated report in the log

[3/14/13 14:01:25:960 CDT] 00000025 TimedOperatio W TRAS0080W: Operation websphere.datasource.execute:jdbc/social:INSERT INTO DB2INST1.CommentTab (comment, extension, object, time, userid, mentionsid) VALUES (?, ?, ?, ?, ?, ?) took 16.613 ms to complete, which was longer than the expected duration of 10.859 ms based on past observations.


Timed Operations

Sample introspection file

The description of this introspectable service: Information about the timed operations, grouped by type, and sorted within each group by expected duration---------------------------------------------------------------Timed operations for: websphere.datasource.execute---------------------------------------------------------------Expected duration (ms) Standard deviation (ms) Timed operation 1.204297 0.929251 websphere.datasource.execute:jdbc/TradeDS:update orderejb set holdingID = ? where orderid = ? 1.107012 1.127067 websphere.datasource.execute:jdbc/TradeDS:delete from orderejb 0.286977 0.163669 websphere.datasource.execute:jdbc/TradeDS:update orderejb set holdingid=null where holdingid = ? 0.211317 0.083466 websphere.datasource.execute:jdbc/TradeDS:delete from accountejb 0.209597 0.243975 websphere.datasource.execute:jdbc/TradeDS:update orderejb set orderstatus = ? where orderid = ? 0.201933 0.059850 websphere.datasource.execute:jdbc/TradeDS:select * from orderejb o where o.orderstatus = 'closed' 0.144153 0.052846 websphere.datasource.execute:jdbc/TradeDS:delete from holdingejb where holdingID = ? 0.143007 0.037623 websphere.datasource.execute:jdbc/TradeDS:update orderejb set holdingID = ? where orderid = ? 0.139108 0.042636 websphere.datasource.execute:jdbc/TradeDS:update accountejb set lastLogin=?, where userID=? 0.137844 0.125086 websphere.datasource.execute:jdbc/TradeDS:delete from orderejb where orderStatus='cancelled' 0.128342 0.042986 websphere.datasource.execute:jdbc/TradeDS:delete from accountprofileejb 0.114781 0.029487 websphere.datasource.execute:jdbc/TradeDS:update accountejb set balance = ? where account = ? 0.028868 0.128856 websphere.datasource.execute:jdbc/TradeDS:select * from orderejb o where o.orderid = ? 0.021680 0.008652 websphere.datasource.execute:jdbc/TradeDS:select * from quoteejb q where q.symbol=?


Enabling Timed Operations

To enable timed operations add the following feature to your server.xml file:

To disable timed operations remove this line from your server.xml file

A server restart is not required to enable or disable timed operations

<feature>timedOperations-1.0</feature>


Configuring Timed Operations

To disable the generation of the report to the logs

To change the frequency of the report, for example to once every 12 hours

To log a warning when the total number of timed operations reaches the specified value

To enable automatic cleanup of least recently used timed operation data from memory (freeing up space as needed for new operations to be tracked), add the following to your server.env file:

<timedOperation reportFrequency="12"/>

<timedOperation enableReport="false"/>

<timedOperation maxNumberTimedOperations="10000"/>

com.ibm.timedOperations.autoCleanup=true

16

Liberty and Full Profile


Problem Determination Capabilities in Both Liberty and Full Profile

Binary Logging (Liberty)

High Performance Extensible Logging (Full Profile)


Binary logging / High performance extensible logging

High speed log and trace handling for WebSphere Application Server

–Available in full profile since WAS 8.0–Referred to as High Performance Extensible Logging (HPEL)

–Available in Liberty since WAS 8.5.5 - Referred to as binary logging


Binary logging / High performance extensible logging

Benefits

–Greatly improves speed of logging and tracing

–Enables you to add new fields to log and trace records

–Helps you filter through logs and trace with a powerful command line tool

–Provides a common solution for all supported platforms

–Includes an API to read from binary repositories programmatically

Notes

– Stores log and trace data in a binary format

– Does not require any change to your application code

– Able to work with log analytics tools


Runtime Logs – Full Profile

Full Profile - default logs trace System.out System.err

SystemOut.log

SystemErr.log

trace.log

activity.log1

Full Profile - HPEL logs trace System.out System.err

logdata/

tracedata/

TextLog2 31activity.log is deprecated and can be disabled.2TextLog is redundant and can be disabled if you don't need a plain text log3Inclusion of trace content in TextLog is optional and not recommended for high performance


Runtime Logs – Liberty

Liberty - default logs trace System.out System.err

messages.log

trace.log

console.log1

Liberty – binary logging logs trace System.out System.err

logdata/

tracedata/

console.log1 1console.log is redundant and can be disabled if you don't need it


Viewing logs and trace

Viewing logs and trace– [Liberty] binaryLog command is used to view content from logdata / tracedata– [Full Profile] logViewer command is used to view content from logdata /

tracedata– displays repository content in plain text– provides options to filter the displayed content and to choose the output format– can be used to monitor (tail) the repositories in real time


Binary Logging Performance (Liberty)

0 2 4 6 8 10 12 14 16 180

100000

200000

300000

400000

500000

600000

700000

Liberty Trace Performance

Default

HPEL

threads

eve

nts

pe

r s

eco

nd

0 2 4 6 8 10 12 14 16 180

100000

200000

300000

400000

500000

600000

700000

Liberty Logging Performance

Default (with console.log)

Default (no console.log)

HPEL (with console.log)

HPEL (no console.log)

threads

eve

nts

pe

r s

eco

nd

Performance– Measured using java.util.logging– Much faster than logging to messages.log when console.log was disabled– Much faster than tracing to trace.log– Performance gains are very dependent on log / trace content

Use binary logging to reduce performance impact of logs and traces on your production servers

– console.log should be disabled when superior logging performance is required (note that this is not a factor in trace performance) <logging consoleLogLevel="OFF" />


HPEL Performance (Full Profile) Performance

– Measured using java.util.logging– Much faster than logging to SystemOut.log when TextLog and JMX Notification are

disabled– Much faster than tracing to trace.log– Performance gains are very dependent on log / trace content

Use HPEL to reduce performance impact of logs and traces on your production servers

0 2 4 6 8 10 12 14 16 180

100000

200000

300000

400000

500000

600000

700000

800000

Full Profile Logging Performance

Default

Default (no JMX Notification)

HPEL

HPEL (no JMX Notification, no TextLog)

threads

eve

nts

pe

r s

eco

nd

0 2 4 6 8 10 12 14 16 180

100000

200000

300000

400000

500000

600000

700000

Full Profile Trace Performance

Default

HPEL

threads

eve

nts

pe

r s

eco

nd


Enabling binary logging (Liberty)

To enable binary logging add the following line to your bootstrap.properties file in your server root directory:

To disable binary logging remove the above line from your bootstrap.properties file

A server restart is required to enable or disable binary logging

websphere.log.provider=binaryLogging-1.0


Enabling HPEL (Full Profile)

A server restart is required to enable or disable binary logging

Administrative Console

1. Expand Troubleshooting2. Select Logs and trace3. Select server14. Select Change log and trace mode5. Click Switch to HPEL Mode6. Save your configuration7. Restart your server


Configuring binary logging (Liberty) Binary log and trace settings are set in child elements of the logging element in the

server.xml

Settings for the console log, trace specification, and log directory are inherited from the logging element

<logging logDirectory="${server.output.dir}/logs" traceSpecification="*=info" consoleLogLevel="AUDIT">

<binaryLog purgeMaxSize="50"/> <binaryTrace purgeMaxSize="50"/></logging>


Configuring HPEL (Full Profile)

Administrators can independently configure (as an immediate runtime change or a persistent config change):• HPEL Log• HPEL Trace• HPEL Text log

Log detail level setting is the same as previous releases.


Viewing logs and trace (Liberty ) The binaryLog command is in the wlp/bin directory

To view all binary logs and trace

To view all warnings and errors

To view all warnings and errors from thread 33 from the latest run of the server

To tail the content of the logs and trace as they are written by the server

To see all of the available options

binaryLog view serverName

binarylog view serverName --minLevel=WARNING

binarylog view serverName –-includeThread=33 --includeInstance=”latest”

binarylog help view

binarylog view serverName --monitor


Viewing logs and trace (Full profile) The logViewer command is in the profile bin directory


LogViewer | LogViewer -monitor



To view all warnings and errors

LogViewer -minLevel WARNING




LogViewer -minLevel WARNING -latestInstance




LogViewer -startDate "05/31/12 22:10:40:000 EDT" -stopDate "05/31/12 22:10:58:000 EDT" -thread 91 -minLevel info


Viewing logs and trace (Full profile)

Administrative Console

1. Expand Troubleshooting2. Select Logs and trace3. Select server14. Select View HPEL logs and trace


Copying log and trace (Liberty)

The binaryLog command lets you make new binary copies of your data

Any of the filtering options can be used when making a new copy

To create a new binary copy of your log and trace data with only messages that contain the string “hello”

To view all log and trace data stored in this new copy


binarylog copy serverName myOutputDirectory --includeMessage="*hello*"

binarylog view myOutputDirectory

binarylog help copy


Copying log and trace (Full profile)

The logViewer command lets you make new binary copies of your data

Any of the filtering options can be used when making a new copy

To create a new binary copy of your log and trace data with only messages that contain the string “hello”

To view all log and trace data stored in this new copy


logViewer -extractToNewRepository myOutputDirectory -message "*hello*"

logviewer -repositoryDir myOutputDirectory

logViewer -help


Extensibility

Developers can add key-value pairs to log and trace records using the LogRecordContext API

– For example, in an application hosting multiple stores, you could add the storeId to each log and trace record in a servlet filter

– Key-value pairs appear in log output when the advanced format is used (selectable when running the [Liberty] binaryLog / [Full Profile] logViewer command)

– Key-value pairs can be used as filter parameters in [Liberty] binaryLog / [Full Profile] logViewer command

– [Full Profile] appName extension is provided automatically, and other extensions are provided for thing like cross component trace, and SIP

– Liberty:

– Full Profile:

[11/26/13 8:15:13:495 EST] 00000074 I UOW= source=com.mymall.ServletX class=com.mymall.ServletX method=doGet org= prod= component= thread=[WebContainer : 0] storeId=[WidgetsPlus] Welcome to Widget's Plus!

binarylog view serverName –-includeExtension=”storeId=WidgetsPlus” --format=advanced

logViewer -includeExtensions “appName=MyMall” ”storeId=WidgetsPlus” -format advanced


Using High Performance Extensible Logging / Binary Logging with SplunkTM

Step 1 - Install Splunk app for WAS

Step 2 - Configure Splunk forwarder

– Option #1 (simplest) - Read from TextLog files

– Add monitor stanza to Splunk inputs.conf file to read from TextLog (use index=websphere and sourcetype=WebSphere:SystemOutErrLog)

– Option #2 (any environment) - Read from logViewer -monitor script

– Configure Splunk to read output from logViewer -monitor

– Set up a no-arg script to run logViewer -monitor from profile bin

– Add script stanza to Splunk inputs.conf file to run the above created script (use index=websphere sourcetype=WebSphere:SystemOutErrLog interval=-1)

– Add -Xrs to Java command invocation in WAS\bin\logViewer script (for graceful shutdown when Splunk stops)



Option #1 (simplest) - Read from TextLog files

inputs.conf

[monitor://D:\WAS\profiles\AppSrv01\logs\server1] whitelist=Text.*\.log$ crcSalt = <SOURCE> disabled = false followTail = 0 index = websphere sourcetype = WebSphere:SystemOutErrLog



Option #2 (any environment) - Read from logViewer -monitor script

inputs.conf [script://$SPLUNK_HOME\etc\system\bin\runLogViewer.cmd] disabled = false interval = -1 index = websphere sourcetype = WebSphere:SystemOutErrLog

runLogViewer.cmd call D:\WAS\profiles\AppSrv01\bin\logViewer -monitor

WAS\bin\logViewer.cmd change: %JAVA_EXE% ... com.ibm.ws.logging.hpel.viewer.LogViewer %* to: %JAVA_EXE% -Xrs ... com.ibm.ws.logging.hpel.viewer.LogViewer %*

41

Full Profile


Full Profile Problem Determination

IBM Support Assistant Data Collector

Cross Component Trace

TimedOperations Feature

Memory Leak Detection, Prevention, and Correction



The isadc command is a console utility you can run from your profile bin directory (starting from WAS 8.0.0.6) to collect diagnostic information from your application server when you are experiencing problems.

The IBM Support Assistant data collector executes scripts which follow the same steps as detailed in MustGather documents produced by IBM L2 support teams for common problems.

Use of the IBM Support Assistant Data Collector can speed up problem determination by ensuring you have the right artifacts to work with support teams

The isadc command must be run on the system on which you are experiencing problems. Many of the scripts require the server to be running, and in the problem state.

The resultant data collected can be sent to IBM via secure file transfer, or kept for your own purposes.

AppServer\bin> isadc



The IBM Support Assistant data collector is menu driven – you can choose from dozens of possible problem symptoms to gather problem-specific artifacts.

Responses to menu choices and prompts can be recorded to a response file as follows:

Response files are in plain text and can be reused for future invocations of the tool:

The tool provides a way to securely transfer data to IBM ECuRep, if desired.

AppServer\bin> isadc -record response.txt

AppServer\bin> isadc response.txt



Console (text) Interface isadc.bat or isadc.sh

in WAS_HOME/bin

Silent response capability

Collection Archive Transfer to IBM for

analysis with a PMR


Cross Component Trace (XCT) is a log/trace correlation technology.

XCT enables you to determine which log/trace entries are part of each request.

XCT can be used in any of three different modes:1)Request ID mode2)Request ID and correlation log record mode3)Request ID, correlation log record, and data snapshot mode

XCT works best in combination with High Performance Extensible Logging (HPEL).• HPEL stores XCT Request IDs• HPEL can filter log/trace records by XCT Request ID



IBM WebSphere Cross Component Trace Logviewer can be used to view files augmented with correlation log records.

• Available for the IBM Support Assistant• Can load multiple files simultaneously• Can show flat / hierarchical views

Cross Component Trace Logviewer


The panel below is where XCT is enabled on the administrative console• The panel can be found here: WebSphere Application Servers > SERVER_NAME >

Change log detail levels

Enabling XCT through Administrative Console


Correlation log records look like this example:

[4/23/12 13:54:44:509 IST] 0000008e XCT I BEGIN AAADx/itMDz-AAAAAAAAAAA 00000000000-cccccccccc2 HTTPCF(InboundRequest /JMSApp/LocalMessageSend RemoteAddress(127.0.0.1) RequestContext(2082603117))

<Date>: The date and time when the log record was generated

<Thread_ID>: The thread which generated this message

<XCT_Logger_Name>: The XCT logger name is XCT. This logger is used to identify the XCT Records in the Log file

<Message_Type>: Type of the log message

<XCT_STATE>: Each XCT Record has a State, it can be BEGIN, END

<XCT_ID>: A Unique ID generated for correlating the XCT Records

<XCT_PARENT_ID>: The XCT_ID of the parent XCT context

<XCT_MESSAGE>: The XCT message contains the information about the XCT record; this can contain some Associations and Annotations

Understanding XCT Correlation Log Records


The XCT requestID is added to all log and trace records associated with HTTP requests. The requestID can only be seen when using the HPEL logViewer command-line tool with

the advanced format

Identifying all log/trace entries that are part of the same HTTP request

logViewer -thread 91 -minlevel info -format advanced


The XCT requestID is added to all log and trace records associated with HTTP requests.

– The requestID can only be seen when using the HPEL logViewer command-line tool with the advanced format

Identifying all log/trace entries that are part of the same HTTP request

logViewer -includeExtensions requestID=AAAKLwUVkuH-AAAAAAAAAAC


The scenarios following this slide use the IBM WebSphere Cross Component Trace Logviewer – available as a tool add-on for the IBM Support Assistant

Tool used to examine XCT entries in a log

Logs can be loaded from multiple servers and they are stitched together

IBM WebSphere Cross Component Trace Logviewer


Customers have discovered several Classloader and ThreadLocal leaks in WebSphere Application Server and their own applications

Prior to this feature, WebSphere Application Server does not contain an application level that provides top down memory leak detection and protection

Customers want to increase application uptime without cycling the server

Frequent redeployments of the application result in OOM errors

Existing (v7 & up) PM39870: Improved classloader leak detection.

NEW in V8.5 (Not in Liberty Profile)

Prevention – Code to proactively fix suspect application classloader leak patterns

Detection - Recognize Application triggered classloader leaks & provide diagnostics

Fixing – Leverage existing JDK APIs and reflection to remedy classloader leaks.

Enabled by setting the setting JVM custom properties

Memory leak detection, prevention and correction


Class Unloading 101

A Java Object has a reference to its Class object.getClass()

A Class has a reference to its ClassLoader class.getClassloader()

A ClassLoader has a reference to every Class it has loaded classloader.findLoadedClass()

Java classes are loaded per-class, but unloaded per-classloader

Class unloading can be denied because of references to ClassLoader, Class or Object– References can come from anywhere:

• Other Objects• Other Llasses/ClassLoaders• Thread stacks• Thread variables• JNI global references• Finalizer queue entries

55 © 2010 IBM Corporation

Unwanted reference to application classloader – example 1

■ Here, we have a CompoundClassLoader which is kept alive because a thread named “Keep-Alive-Timer” has its contextClassLoader set to it

■ “Keep-Alive-Timer” is a daemon thread– Spawned by the classlibraries– Daemon threads live until the JVM ends

■ Threads inherit contextclassloader– From their parent

■ This is a Java classlibrary bug– Being raised with Oracle– Fix is simple:

• thread.setContextClassLoader(null);


Unwanted reference to a class which was loaded by the application classloader – example 2

■ Here, a CompoundClassLoader is kept alive because a class it loaded – org.richfaces.model.selection.ClientSelection – has been stored inside a HashMap in the system class java.beans.PropertyEditorManager

■ Because this is a system class, it has javadoc!

■ Looks like a RichFaces bug...– https://jira.jboss.org/browse/RF-7911

• “OutOfMemory when redeploying - ClientSelection not unregistered from PropertyEditorManager”


Common leaks

■ ThreadLocal problems– Custom class extending ThreadLocal?

■ Threads' contextClassLoaders– Daemon threads started by a servlet– Careless use of java.util.Timer– Daemon threads started by 3rd party libraries, shared between two applications

■ Bean introspection– If you introspect (call getBeanInfo()) on a Bean loaded by the app classloader, you must

call flushfromCaches(beanClass) on app shutdown

■ JMX MBeans and NotificationListeners– Must be unregistered when the application stops


Example BAD code

public class MyCounter { private int count = 0;

public void increment() { count++; }

public int getCount() { return count; }}

public class MyThreadLocal extends ThreadLocal<MyCounter> {}

public class LeakingServlet extends HttpServlet { private static MyThreadLocal myThreadLocal = new MyThreadLocal();

protected void doGet(HttpServletRequest request, HttpServletResponse response) throws ServletException, IOException {

MyCounter counter = myThreadLocal.get(); if (counter == null) { counter = new MyCounter(); myThreadLocal.set(counter); }

response.getWriter().println( "The current thread served this servlet " + counter.getCount() + " times"); counter.increment(); }}


Detection, prevention and action

Detection: Issue warnings when a memory leak is detected

Prevention is on by default and applies only to JRE triggered leaks.

Action: Take proactive action to fix memory leaks.

Actions have reasonable defaults and are configured on a case-by-case

Pseudo code for clearing leaksprotected void com.ibm.ws.classloader.clearReferences()

{ if(ENABLE_CLEAR_REFERENCES_JDBC) clearReferencesJdbc(); if(ENABLE_CLEAR_REFERENCES_THREADS) clearReferencesThreads();

if(ENABLE_CLEAR_REFERENCES_THREADLOCALS) clearReferencesThreadLocals(); if(ENABLE_CLEAR_REFERENCES_RMI_TARGETS) clearReferencesRmiTargets(); if(ENABLE_CLEAR_REFERENCES_STATICS) clearReferencesStaticFinal();

}

60


Leak detection messages

CWMML0015E: The web application [WasSwat#WasSwatWeb.war] created a ThreadLocal with key of type [test.memleak.MyThreadLocal] (value [test.memleak.MyThreadLocal@216c691]) and a value of type [test.memleak.MyCounter] (value [test.memleak.MyCounter@21942ff]) but failed to remove it when the web application was stopped.

CWMML0010E: The web application [LeakApp#leak.war] appears to have started a thread named [Thread-73] but has failed to stop it.

CWMML0011E: The web application [LeakApp#leak.war] appears to have started a TimerThread named [leaked-thread] via the java.util.Timer API but has failed to stop it.

CWMML0024W: About to interrupt thread [leakingThread] which is currently executing

CWMML0026I: ClassLoader memory leak is fixed. Clearing leak References succeeded for LeakApp#leak.war.

61


High Performance Extensible Logging

Using High Performance Extensible Logging to troubleshoot applications

http://pic.dhe.ibm.com/infocenter/wasinfo/v8r5/index.jsp?topic=%2Fcom.ibm.websphere.nd.multiplatform.doc%2Fae%2Fttrb_usinghpel.html

High Performance Extensible Logging (HPEL) http://publib.boulder.ibm.com/infocenter/ieduasst/v1r1m0/index.jsp?topic=/com.ibm.iea.was_v8/was/8.0/ProblemDetermination/WASv8_HPEL/player.html

System administration in WebSphere Application Server V8.5, Part 3: High Performance Extensible Logging (HPEL)

http://www.ibm.com/developerworks/websphere/techjournal/1208_bourne/1208_bourne.html


WebSphere Application Server v8.5 Cross Component Trace

http://webspherecommunity.blogspot.ca/2012/07/websphere-application-server-v85-cross.html

Use Cross Component Trace (XCT) request Ids to see which log entries were generated by each request

http://www.youtube.com/watch?v=oXsfDoiNb5c

Use the IBM WebSphere Cross Component Trace Logviewer to view WebSphere Application Server log files

http://www.youtube.com/watch?v=qorwRZh4DgQ

High Performance Extensible Logging and Cross Component Trace Tech Video

https://www.youtube.com/watch?v=FKKfUdNux70

IBM Training: IBM WebSphere Application Server V8 Problem Determination

http://www-304.ibm.com/jct03001c/services/learning/ites.wss/us/en?pageType=course_description&courseCode=WU582

Useful Links – Runtime Capabilities


Memory Leak Detection and Prevention

Configuring the memory leak policy http://pic.dhe.ibm.com/infocenter/wasinfo/v8r5/topic/com.ibm.websphere.nd.doc/ae/ttrb_configmemleak.html

Memory leaks in Java Platform, Enterprise Edition applications

http://pic.dhe.ibm.com/infocenter/wasinfo/v8r5/index.jsp?topic=%2Fcom.ibm.websphere.nd.doc%2Fae%2Fctrb_memleakdetection.html

Binary Logging

Liberty Profile: Binary Logging http://pic.dhe.ibm.com/infocenter/wasinfo/v8r5/topic/com.ibm.websphere.wlp.express.doc/ae/cwlp_HPELOverview.html

Binary logging for Liberty profile http://publib.boulder.ibm.com/infocenter/ieduasst/v1r1m0/topic/com.ibm.iea.was_v8/was/8.5.5.0/content/WASV855_Binary_Logging/player.html?dmuid=20130807101913462846



Timed Operations

Timed operations for Liberty http://publib.boulder.ibm.com/infocenter/ieduasst/v1r1m0/topic/com.ibm.iea.was_v8/was/8.5.5.0/content/WASV855_Serviceability_TimedOperations/player.html?dmuid=20130815082326115285

Lab: Liberty profile timed operations for JDBC http://publib.boulder.ibm.com/infocenter/ieduasst/v1r1m0/topic/com.ibm.iea.was_v8/was/8.5.5.0/content/WASv855_Liberty_Timed_Operations_Lab.pdf?dmuid=20130807102148029081

Files for lab: http://publib.boulder.ibm.com/infocenter/ieduasst/v1r1m0/topic/com.ibm.iea.was_v8/was/8.5.5.0/content/WASv855Labs_libtops.zip?dmuid=20130807102201566285

Sensitive Log and Trace Guard

Using sensitive log and trace guard http://pic.dhe.ibm.com/infocenter/wasinfo/v8r5/topic/com.ibm.websphere.express.doc/ae/ctrb_sensitivelogtrace.html


http://publib.boulder.ibm.com/infocenter/ieduasst/v1r1m0/topic/com.ibm.iea.was_v8/was/8.5.5.0/content/WASv855Labs_libtops.zip?dmuid=20130807102201566285

65

IBM Support Assistant v5.0


ISA 5.0 Team Server

•Server-based model

•Install once - shared by many team members via browser

•Web 2.0 browser interface

•Remote execution of PD tools

•Off-load analysis processing

•Collaboration on PD

•Case Management

•Tool Management

•Single-user option available

IBM Support Assistant 5.0

Available at www.ibm.com/software/support/isa

BrowserBrowser

BrowserBrowser

ISATeam Server


IBM Support Assistant 5.0 – Deployment options

67

Team ServerSingle installMultiple end usersLeverages resources of ISA server systemShared investigation

StandaloneSingle userLocal installUser administered

ibm.com(ISA and

Tool updates)

ISA Team Server

ibm.com(ISA and

Tool updates)

ISA Team Server (local)


Automation of Data Upload to ISA 5

■ ISA Provide a REST interface for some of its functions:

■ Case creation:curl --user user1:user1password -X POST -H Content-Type:application/x-www-form-urlencoded -v -o /home/dumps/curl.log --url http://myISA5TeamServer.mydomain.net/rest/1/tickets -d "summary={case summary}&description={case description}"

■ File upload:curl --user user1:user1password -X POST -v -k -o /home/dumps/curl.log -F file=@'{}' --url

http://myISA5TeamServer.mydomain.net/rest/1/files/${caseno}/file_upload?path=/${caseno}/

■ Can be combined with IBM JDKs “-Xdump” options to run automatically:-Xdump:tool:events=systhrow,filter=java/lang/OutOfMemoryError,exec="DumpUpload.sh %pid”


Tools – Toolbox

•Catalog of available tools•Learn about and discover tools•Launch•Tool Help•Execution History


Files – Launch tools

Suggested tools based on file type

Automatically pass files to a tool All tools available from the

“Other…” menu


Available Problem Determination Tools in ISA 5.0 Team Server

Tool Name Problem Areas Type

Classloader Analyzer Java™, Runtime Desktop

Database Connection Pool Analyzer WebSphere, Runtime Desktop

FileNet Optical Storage And Retrieval (OSAR) Cable Tool Desktop

Garbage Collection and Memory Visualizer (GCMV) Java, Performance, Memory Desktop, Report

Health Center Java, Performance Desktop

HeapAnalyzer Java, Memory Desktop

Interactive Diagnostic Data Explorer (IDDE) Java, Memory Desktop+Server/Web

Memory Analyzer (MAT) Java, Memory Desktop, Report, Web

Pattern Modeling and Analysis Tool (PMAT) Java, Performance, Memory Desktop, Report

Portal Log Analyzer WebSphere, Configuration Report

Processor Time Analysis Tool for Linux Java, Performance Desktop

Profile Port Checker WebSphere, Configuation Report

Thread and Monitor Dump Analyzer (TMDA) Java, Runtime Desktop, Report

Trace and Request Analyzer for WebSphere Application Server WebSphere, Runtime Desktop

Web Server Plug-in Analyzer for WebSphere Application Server WebSphere, Perfomance Desktop

WebSphere Application Server Configuration Visualizer WebSphere, Configuration Report

WebSphere Cross Component Trace Logviewer WebSphere, Runtime Desktop


Additional Resources

ISA YouTube playlist

@ISA_Tools

Requests for Enhancement

72

Contact Information

Need support or have questions about Team Server?

Visit our forum:http://ibm.biz/ISA-Forum

IBM Support Assistant web pagehttp://www.ibm.com/software/support/isa

http://tiny.cc/ISA5PL

https://twitter.com/ISA_Tools

http://ibm.biz/ISA-RFE

http://ibm.biz/ISA-Forum

http://www.ibm.com/software/support/isa


IBM Support Assistant

IBM Support Assistant http://www.ibm.com/software/support/isa/

WAIT

IBM Whole-system Analysis of Idle Time (WAIT) http://wait.ibm.com

Useful Links – Problem Determination Tools


IBM Training

IBM WebSphere Application Server V8.5.5 Problem Determination

http://www-304.ibm.com/jct03001c/services/learning/ites.wss/ca/en?pageType=course_description&courseCode=WA591CE

WebSphere Support Technical Exchanges

Support Technical Exchanges http://www.ibm.com/support/entry/portal/Scheduled_tech_exchanges/Software/WebSphere/WebSphere_brand_support_(general)

IBM Education Assistant

IBM Education Assistant http://www.ibm.com/software/info/education/assistant/

Problem Determination Practices

RedPaper – Increasing Resiliency for IBM WebSphere Application Server Deployments

http://www.redbooks.ibm.com/redpapers/pdfs/redp5033.pdf

Useful Links – Knowledge and Education

75

Questions?


Your feedback is valuable - please complete your session or lab evaluation!

Session number[A6]

Provide your evaluations by:

Evaluation forms:

Fill out a form at the end of each session

Paper forms are located in each of the session or lab rooms

Place the completed form in the tray as you exit the room- Or –

Complete the session survey on Event Connect Portal: ibmeventconnect.eu/euxdx

Select Sessions, then Session Finder, and complete the survey

Visit WASdev.net for downloads and other resources

WASdev.net

https://wasdev.net/


For Additional Information

IBM Traininghttp://www.ibm.com/training

IBM WebSpherehttp://www-01.ibm.com/software/be/websphere/

IBM developerWorkswww.ibm.com/developerworks/websphere/websphere2.html

WebSphere forums and communitywww.ibm.com/developerworks/websphere/community/

78

http://www.ibm.com/training

http://www-01.ibm.com/software/be/websphere/



http://www.ibm.com/developerworks/websphere/websphere2.html

http://www.ibm.com/developerworks/websphere/community/

http://www.ibm.com/developerworks/websphere/community/

websphere technical university: top websphere problem determination features

Technology

server wiring information

server component list

thread information

ibm runtime monitoring

osgi bundle information

application server

wlpbin server

problem reduction1