sutol - a hitchhiker’s guide to troubleshooting ibm connections

8th Sutol Conference, November 2016

A Hitchhiker’s Guide to

Troubleshooting IBM Connections

Sharon Bellamy James

& Roberto Boccadoro


A Hitchhiker’s Guide to

Troubleshooting IBM Connections


… and no the answer is not

42

Sharon Bellamy James

• IBM WebSphere since 1999 &

Connections since 2 / 2008

• Many years experience with -

migrations, installations, administration,

integration, customization

& DOCUMENTATION!!

• Tiny bit of a Star wars & Disney fan

• Charity fundraising Cosplayer/Costumer

• Part of the team that brings you ICON UK

5


Joined Lotus in 1994, we were selling then, Notes was V3 and was sold as development platform. Remained in IBM since Sept. 2012 in Technical Sales. During those years I have worked with almost all the products Lotus/IBM has created in the collaboration space. I am now a consultant IT Specialist/IT Architect and my actual focus is mostly IBM Connections, IBM Docs, IBM Sametime.

I have 10 IBM Technical certifications and I am an

Outside work, I am a husband and father of 4 (2 humans, 2 cats) and an amateur astronomer.

@robboc59

blog http://rob59blog.wordpress.com/

[email protected]

Roberto Boccadoro


Overview

• Different types of logs, where are they located?

• How to read system out logs

• Applying traces

• Fiddler traces

• A bunch of other useful stuff


Connecting Connections – there are many

integration pointsBasic WebSphere Architecture plus:

Plugins

Search

Mobile

Portal

Cognos

FileNet

IBM Docs

Mail Integration

TAM / SiteMinder

SPNEGO / SAML


A Jigsaw of many parts


Know your Topology• Is the installation:

Small – one server (jvms)

Medium – at least 3 servers (jvms)

Large – one server (jvm) for each Connections

application


Small Deployment• One single node

– Deployment manager

– Nodeagent / Connections WebSphere server

deployed on the

same machine

• All applications are deployed to one JVM

(WebSphere Server)

• DB2, TDI & HTTP can also be deployed to

the same machine or another machine

• Add additional

– Mail integration, Cognos, IBM Docs,

CCM/Filenet etc….


Medium Deployment

• One or more nodes

• Applications are spread over at least 3 JVMS

(WebSphere servers)

• TDI, DB2 and HTTP usually on additional

machine(s)

• Potential for load balanced HTTP servers

• Plus additional

– IBM Docs, Cognos, CCM/Filenet,

Mail integration, SSO – Siteminder/TAM/SPNEGO,


Large Deployment

• Generally used for 100’s of thousands of users

• Multiple nodes

– Each application is deployed to its own JVM / WAS server

• Load balanced multiple HTTP servers

• Enterprise sized clustered/fail over DBs

• Clustered LDAP

• Plus additional

– IBM Docs, Cognos, CCM/Filenet, Mail integration,

– SSO – Siteminder/TAM/SPNEGO


Everything has a log

But which one do you need?• WAS System out - First place to start

• HTTP Access / Error

– 404’s, other access issues

• HTTP Plugin

– SSL issues, HTTP to WAS connection issues

• TDI - Sync issues

• DB2 - Diag, admin logs

Associated Software

• IBM Docs – WAS logs

• CCM/Filenet – WAS logs / P8 Logs

• LDAP – LDAP logs / domino / AD etc.

• Cognos – WAS logs


Different Types of Logs, Where are They?Because Connections connects toeverything, and because Connections is acollection of applications that run on theWebSphere platform, there are a variety oflogs in a variety of locations:

-WebSphere-Connections Install-Connections Migration-HTTP server-TDI-Congnos-FileNet-wsadmin-Database-Mobile-Fiddler


Where are the logs? WebSphere

• AppServer simply distinguishes the

WebSphere Application Server from

other WebSphere deployments that may

be under the WebSphere folder:

- The first layer of logs only pertain to the

high level install and are seldom used

- Most all of Connections log activity

pertains to the logs within the profiles

directory.



• Again, most all of our log activity

pertains to the logs within the

profiles directory.

-For example-AppSrv01 – Profile for Connections server

-AppSrv02 – Profile for Cognos server

-Dmgr01 – Profile for Deployment Manager



- AppSrv01 – The Profile

- iccluster_server1 – The actual Connectionsserver JVM.

• Systemerr – System level logs that cansometimes have key information, but not usually

• Systemout – WebSphere application server log– this is what we use 95% of the time.

• Traces – If captured, would also be located here

• Multiply these logs by 2, 3, 4 or more forclustered environments!


Where are the logs? Connections install and

CR updatesC:\IBM\Connections\logs

While Connections runs on WebSphere, there are

additional resources that are wrapped around

WebSphere. This data is stored in the

\ibm\connections folder

It includes such things as:• Indexing• Customization folders• Files - attachments• Logs:

– 5.0 - Install / CR updates– 4.5 - Install

NOTE: As of Connections 5.0:CRs are installed using the IBM InstallationManager.iFixes are still installed using the connections update installer


Where are the logs? Connections install V5.5

• Connections 5.5 installation

log

• C:\IBM\Connections\install.log

• /opt/IBM/Connections/install.log

• The install log contains warning,

errors and

success messages and details of the

installation


Where are the logs? iFixes• C:\IBM\Connections\version\log

• /opt/IBM/Connections/version/log

• Each iFix has its own log with the

info, error or warning messages


Where are the logs? Current release info

• To find the current release level

you can browse these files:

• C:\IBM\Connections\version

• /opt/IBM/Connections/version


Where are the logs? TDI (TDISOL)• TDISOL logs are in the TDISOL\logs directory

• C:\IBM\TDI\V7.1.1\tdisol\logs

/opt/IBM/TDI/V7.1.1/tdisol/logs

• Or wherever you put TDISOL (same for

external users EXTTDISOL etc.)

• What is TDISOL?

• The TDI Solution directory. This holds all

of the scripts and configuration

that TDI requires to populate the profiles

DB and

keep it in sync with your LDAP directory.


Where are the logs? DB2 / Database

DB wizard logs:In the *home* directory of the user running the

wizards – typically

Windows :

C:\Users\Administrator\lcWizard\log\dbWizard

Linux: /home/db2inst1/lcWizard/log/dbWizard

DB2DIAG LOG: Windows:

C:\ProgramData\IBM\DB2\DB2COPY1\DB2\DAIG

0000

Linux:

/home/db2inst1/sqllib/db2dump

Asked for by IBM for debugging issues


Where are the logs? HTTP Server

C:\IBM\HTTPServer\logs

/opt/IBM/HTTPServer/logs• As all Connections application to

application interactions flow through the

HTTP server, there are times when these

logs will help to determine routing failures.

access.log – logs all access via the http

server

error.log – logs errors for the http server


Where are the logs? HTTP ServerThese logs can be useful for tracking

calls to various applications.

Remember, every application to application

is actually a request to the HTTP server to

find out the location of the application.

-Connections uses LotusConnections-

Config.xml to get the URL for the

application, then contacts the HTTP

server to get to that location via the

WebSphere plugin

-The HTTP return code is second from the

end on the right


Where are the logs? Filenet / P8C:\IBM\WebSphere\AppServer\profiles\App

Srv01\FileNet\CL_connections_server\

/opt/IBM/WebSphere/AppServer/profiles/Ap

pSrv01/FileNet/CL_connections_server/

• p8_server_error.log

p8_server_trace.log

pesvr_system.log

pesvr_trace.log


Where are the logs? wsadminThe WebSphere administrative (wsadmin) scriptingprogram is a non-graphical command interpreter environment enabling you to run administrativeoperations in a scripting language. Connectionsuses this tool to perform various maintenance andupdate functions.

Wsadmin logs: (typically not logged, theinformation just displays in the command window)

wsadmin>execfile("communitiesAdmin.py")WASX7015E: Exception running command:"execfile("communitiesAdmin.py")"; exceptioninformation:com.ibm.bsf.BSFException: exception from Jython:Traceback (innermost last): File "<input>", line 1, in ?IOError: File not found - communitiesAdmin.py (No suchfile or directory) wsadmin>

In this case, the wsadmin command was run from theAppserver rather than from the Dmgr and therefore the filewas not present.

Wsadmin tracing:Tracing operations using the wsadmin scripting tool(this will help to capture wsadmin errors, not functionalerrors)

http://pic.dhe.ibm.com/infocenter/wasinfo/v7r0/index.jsp?topic=%2Fcom.ibm.websphere.base.doc%2Finfo%2Faes%2Fae%2Ftxml_traceadmincontrol.html

Right before you enter the communities.Admincommand, run the following (bolded) command to addtracing:wsadmin>AdminControl.trace('com.ibm.*=all')

wsadmin>execfile("communitiesAdmin.py")

wsadmin>CommunitiesService.moveCommunityToSubcommunity................

This will create "wsadmin.traceout" log under the\IBM\Websphere\Appserver\profiles\AppSrv01\logs

http://pic.dhe.ibm.com/infocenter/wasinfo/v7r0/index.jsp?topic=/com.ibm.websphere.base.doc/info/aes/ae/txml_traceadmincontrol.html


Switching log language to englishOpen ISC Open Servers > Application Servers.

Then choose the server where you want to

modify the language for the log.

Click Java and Process Management > Process

definition in section Server Infrastructure,

Then Additional Properties > Java Virtual

Machine

In the Generic JVM Arguments box type:

-Duser.language=en -Duser.region=US *

Click OK, and Save, Logout from ISC

Restart the server to apply the changes of the

JVM.

*NOTE: depending on the machine region it

sometimes will not take the –Duser.region=US

ensure the machine LOCALE is set to US


Making sense of the SystemOut log - structureDisplays:

WebSphere release

WebSphere server name

PID – Process ID

Operating System

File location of the server

Tracing applied

Check to see if this is the startup of the server that you want!

Does not display:

Release of Connections This is because Connections is a

collection of applications that run on WebSphere server.

We can however find this from the systemout.log that

includes the application startups. Stay tuned.


Making sense of the SystemOut log


How to Read and Analyse the Systemout Log

Searching:

- Get counts of errors

- See historical trends of errors

- “ E “ error

- “ W “ warning

Best trouble-shooting tip I learned last year, notice the spaces before and after the E

Will still need to review the log – but this can help rise to “10,000 feet” and see the context.


Making sense of the SystemOut log – stack trace

Tools:A decent text editor – notepad++, editpad etc. for windows, gedit for linux etc.You may need to access multiple logs in a clustered environment

Find the most recent error and work backwards

Make sure you are looking at what you think you are looking at! (Some logs have months of data, others have minutes)

Systemout is not always at the beginning of a reboot, check to see "application started“ in

systemout.

From Notepad++ you can “find all in current document” and see all applications started in a

systemout.log:


WebSphere Tracing

There is little information about reading stack traces from a support perspective and

unfortunately healthy servers may also have stack traces, so it can be hard to interpret

which ones are serious and which are not.

The first line will tell you key information about the error

The lines in-between are all pointing back to what called them

The last time is the ultimate source

Look for the “caused by” line if available


WebSphere TracingLog on to the WebSphere Application Server

(WAS) integrated solutions console using an

administrator IDGo to -- Troubleshooting -> Logs and Trace -> Logging

and Tracing -> and the server you wish to add tracing to

Diagnostic Trace Server -> Change Log Detail Levels ->

Add the following string and click Apply -> OK

*=info:com.ibm.lconn.moderation.*=all: (sample trace)

This is dynamic

It will not let you save a bad trace

In a multi-node system, collect the logs from one node at a

time

• i.e. only have one node running to test / collect logs

• Makes trouble-shooting much easier

• IBM will normally request this if you have a PMR open


Fiddler Tracing

IBM support will almost always ask for a fiddler trace - http://www.fiddler2.com/fiddler2/

Ensure SSL traffic is decrypted : Tools - Fiddler Options - HTTPS tab - then select both

"Capture HTTPS CONNECTS" and "Decrypt HTTPS traffic” and accept the cert warning

Clearing the trace is easy – just use Ctrl + x

Shut down other apps/browsers that add extraneous data to the trace

Note: that Fiddler time is in your local browser’s time – Make a note of local time tests

are run and server time

Save the trace file (.saz) – send these to IBM with the systemout.logs along with time

stamps and trace info.

http://www.fiddler2.com/fiddler2/


Fiddler Tracing


Business as usual – Useful Stuff


Useful Stuff – WAS versionInfoFrom the WAS_HOME\bin (i.e

/opt/IBM/WebSphere/AppServer/bin)

run the versioninfo command

versionInfo.bat /sh

– this command will list the top level WAS

version

and major fix/feature pack

versionInfo.bat /sh

–maintenancePackages – this command

will list WAS version,

fix/feature pack and any iFixes installed


Useful Stuff – Connections Version Info

From the Connections install_home\updateinstaller directory run the following commands

updateSilet.bat/sh –fix –installDir <connections_home>

i.e.

./updateSilent.bat/sh –fix –installDir /opt/IBM/Connections

This command will list the installed version of connections and any fixes applied


Useful Stuff - Change WAS log sizeDone via the community scripts (see

http://scripting101.org) or via the ISC

Set for EACH server including nodeagent

and dmgr

Troubleshooting > Logs and trace > JVM

Logs

Default it 1mb – useless

Change to at least 10 and keep 20 historical

Don’t forget to archive them !!


Useful Stuff - Check SIB Queue Depth

Use the following path in ISC to see if there is a backlog in the messaging bus: (keep this

handy!)

Service Integration > Buses > ConnectionsBus >Destinations > connections.events >

Publication points >[email protected] [Runtime] >

Subscriptions

If any of the queues have a backlog it is possible to

refresh them. If they backlog up that indicates a more

serious issue. A server restart can sometimes kick

the queue into life – check SystemOut for log

messages relating to the SIBs

mailto:[email protected]


Useful Stuff - WAS Ports / Apps & Context Roots

Where is my app running?

Check app and manage modules

which cluster app is deployed to

Check ports via


Useful Stuff - Clear WAS TempShut down all WAS processes : Connectionsserver(s), the DMGR, and the nodeagent.

Clear the temp/cache folders:IBM\WebSphere\AppServer\profiles\AppSrv01\tempIBM\WebSphere\AppServer\profiles\AppSrv01\wstempIBM\WebSphere\AppServer\profiles\Dmgr01\tempIBM\WebSphere\AppServer\profiles\Dmgr01\wstemp

Stop and start the HTTP server – this kills off any stale web threads / sessionsRestart DMGR ,node agent and Connections servers

Clear the cache on the browser before you test (EVERYTHING)

Must be done every time a fix is applied to clear issues with UI – also good housekeeping

Useful Stuff - mtail

Mtail http://ophilipp.free.fr/op_tail.htm

When you run Connections a a Windows service, it will tell you it is started but in fact this is not

true. It takes a while between the service appears to be running and Connections is really

started.

Use Mtail to open the Connections_server SystemOut.log, when you see “server

Connections_server open for e-business” then Connections is running.

Tip – in the Mtail window filter for the word “open”

Also baretail is a free GUI tool for tailing on windows

http://ophilipp.free.fr/op_tail.htm


http://rob59blog.wordpress.com

robertoboccadoro

@robboc59

Any Questions?

http://socialshazza.com

dilftechnical

@socialshazza

sutol - a hitchhiker’s guide to troubleshooting ibm connections

Technology