connected 2015 - ibm notes traveler daily business

BP104:IBM Notes TravelerDaily Business –Administration, Monitoring and Support

René Winkelmeyer, midpoints GmbH

Agenda

Administration basics

Monitoring

Tuning

Troubleshooting

About me

midpoints GmbH

IBM Advanced Business PartnerIBM Design Partner (Notes Domino, Mobile, Verse)

Apple Enterprise Developer and MDM Group MemberSamsung Enterprise Alliance Partner

Services- Enterprise Mobility Service- Mobile Device and Application Management- IBM Notes Traveler and IBM Mobile Connect

René WinkelmeyerHead of Development

About me

Reach out any time

Skype / Twitter / LinkedIn => muenzpraeger

Webhttp://blog.winkelmeyer.comhttp://www.midpoints.de

[email protected]@midpoints.de

OpenNTF

File NavigatorGeneric NSF View Widget for IBM Connections

René WinkelmeyerHead of Development

Credits

Credits to the guy who I‘ve worked on Traveler and IBM Mobile Connect stuff in the last years.

Blog

http://www.netzgoetter.net

Mail

[email protected]

Detlev PöttgenManaging Director

Legal

The following product names will be used in this session:

DB2®

Domino®

Java™

Microsoft SQL Server®

Notes®

Agenda


Monitoring

Tuning

Troubleshooting

Administration Basics – High Availability

IBM Notes Traveler allows two operating modes:

Standalone Traveler Server

– LotusTraveler.nsf

– Local Java Derby Database

High Availability (HA) Traveler Server Pool

– Traveler-“Cluster“

– Remote RDBMS (IBM DB2 or M$ SQL)

Administration Basics – Big Picture Traveler

SQL

Backend

Server

SQL

Backend

Server

Traveler Server

HTTP

TASK

TravelerOSGi

SERVLET

Traveler

TASKSSL

Domino

Directory

Server

Config LotusTraveler

Default

Settings

Derby

Rel.-DB

/ntsdb

Notes

notes.ini

Domino

& HTTP

Security

SQL


Benefits of a HA environment

– Better scaling (Standalone should operate to max. 2.500 devices)

– Failover capabilites, no downtime of Traveler services

– Servers can be updated any time (there‘s never a good time for downtime...)


Domino Mail

Domino Mail

Domino Mail

TravelerServer

HTTPS

Notes

TravelerServer

DB2 / SQL Server

DB2 / SQL Server

DB2/SQL

HTTP(S)

IBM Notes Traveler

Service Pool

Load Balancer

IP-Sprayer

Reverse Proxy

(i.e. IBM Mobile Connect)

DB2 / SQL

Backend

Server in HA

IBM Notes Traveler – High Availability

Two or more Domino servers with installed Traveler addon work in a so called „Traveler Service Pool“

All pool members use the same state database

The state database is centrally organized (IBM DB2 or M$ SQL)

IBM Notes Traveler – High Availability

What‘s a Pool?

IBM Notes Traveler – Traveler Service Pool

Every user can be served by every Pool member server

All Pool member servers are equal (in terms of service)

Travelers own Availabiltiy Index (AI) is used for internal Load Balancing using TCP ports 50125/50126

Each user is bound to a single server within the pool for his Master Monitoring Sesssion (MM or User Session)

This Master Monitor Server is responsible for mail database synchronisation

IBM Notes Traveler – HA Load Balancing

tell traveler HADR show

Domino ID Host IP:SrvrPort,SrvltPort Alive Server Servlet Last HB AI Users Devices

L1/NETWORK 330 s1.network.com 10.3.1.1:50125,50126 true true true 2014-08-26 96 2315 1179



L4/NETWORK 580 s4.network.com 10.3.1.4:50125,50126 true true false 2014-08-26 100 0 346


AI = Traveler internal Availability Index (AI)

Users = Master Monitor Session / one per user

Devices = HTTP Sessions (Devices last seen) / one session per device

Note: L4/NETWORK has been restarted

IBM Notes Traveler – HA Load Balancing

• Traveler Availability Index (AI)

• Master Monitor Server (MM) per user

• User Load Balancing Bias

+ 10 Bias for local server

+ 20 Bias for current MM

• Load Balancing algorithm

• AI calculated per server

• Chooses the highest AI (incl. Bias) and defines that server as MM

• All devices (of this user) are routed to the current MM

• The MM isn‘t allowed to re-balance within 10 minutes

HTTP –Task

Servlet

HTTP –Task

Servlet

Traveler –Task

AI = 75

Traveler –Task

AI = 80

Server 1 Server 2

AI 75 + BIAS 10 + BIAS 20 > AI 80

AI 75 + BIAS 20 > AI 80 + BIAS 10

TCP 50125

Administration Basics – Webfrontend

Starting with 8.5.3. UP1 Traveler administration is done by using a XPages application

Administration Basics

The webfrontend is unfortunately not sufficient – you‘ll often need the Domino server console for troubleshooting


Current user status – tell traveler user <user name>

tell traveler user Detlev Poettgen

CN=Detlev Poettgen/O=midpoints does not have sufficient access rights to the database

mail/dpoettge.nsf.

…

tell traveler user [email protected]

IBM Notes Traveler has validated that it can access the database mail/rwinkelm.nsf.

Encrypting, decrypting and signing messages are not enabled because the Notes ID is not in

the mail file or the ID vault.

……


IBM Notes Traveler has validated that it can access the database mail/rwinkelm.nsf.

Encrypting, decrypting and signing messages are not enabled because the Notes ID is not in the mail file or the ID

vault.

Canonical Name: CN=Rene Winkelmeyer/O=midpoints

Internet Address: [email protected]

Master Server: Traveler01/srv/midpoints-trav, version 9

Master Server Locked: Jan 21, 2015 11:05 AM, type=Soft

Home Mail Server: CN=mail01/OU=srv/O=midpoints

Home Mail File: mail/rwinkelm.nsf

Current Mail Server: CN=mail01/OU=srv/O=midpoints Release 9.0

Current Mail File: mail/rwinkelm.nsf

Mail File Replicas: [CN=mail02/OU=srv/O=midpoints, mail/ms.nsf], [CN=mail01/OU=srv/O=midpoints,

mail/rwinkelm.nsf]

ACL for Rene Winkelmeyer/midpoints: Access=Editor Capabilities=create,update,read,delete,copy Missing

Capabilities=none

ACL for Traveler01/srv/midpoints-trav: Access=Manager Capabilities=create,update,read,delete,copy Missing

Capabilities=none

Notes ID: Mail File does not contain the Notes ID.

Auto Sync User State: Monitoring disabled

Last Prime Sync: Monday, Jan, 2015 2:31:11 PM CEST

Banned Documents: 0…

mailto:[email protected]


Devices:

Device ID: ApplC38JCFABDTWG

Device Description: ApplC38JCFABDTWG

Security Policy Status: No policy

Security State: Clear

Approval State: Not required

Last Sync: Never

Auto Sync Device State: Inactive

Device offline time: Monday, Jan 20, 2015 2:39:42 PM CEST

Auto Sync Connection State: Disconnected

Auto Sync Applications to Synchronize: folder, mail, calendar, contact, serviceability, security

Auto Sync Change Flags: folder:add, mail:add(4:add), serviceability:configGet/configSet


Pipe command output to file – tell traveler –f <filepath> user <user name>

Watch out:

Command leaves zero byte validation file .sem in the directory which doesn‘t get cleaned up automatically.

tell traveler –f /tmp/user.txt user rwinkelm

Output for command ‘-f /tmp/user.txt show rwinkelm’ can be found at /tmp/user.txt.

[root@incinerate tmp]# ls -lrt

insgesamt 3240

-rw-rw-r--. 1 domino domino 424 20. Jan 13:19 user.txt

-rw-rw-r--. 1 domino domino 0 20. Jan 13:19 user.txt.sem


Get SQL content via Domino console – tell traveler sql „<QUERY>“

tell traveler sql "SELECT HOSTNAME FROM TS_GLOBAL“

[04683:00036-3357230848] Command 'SELECT HOSTNAME FROM TS_GLOBAL' was

completed successfully.

[04683:00036-3357230848] HOSTNAME |

[04683:00036-3357230848] incinerate.midpoints.net |

[04683:00036-3357230848] hellfire.midpoints.net |

[04683:00036-3357230848] aryastark.midpoints.net |

Agenda


Monitoring

Tuning

Troubleshooting

Monitoring – Server-Task

Notes Traveler is part of the „Server Tasks“ within Domino Administrator (added in 2013, check if your domadmin.nsf is updated)

Monitoring

Traveler server status – tell traveler status

tell traveler status

The IBM Notes Traveler task has been running since Tue May 14 12:31:09 BST 2014.

The IBM Notes Traveler availability index is currently 100 while servicing 431 users.

The last successful device sync was on Sat Jan 22 15:05:15 BST 2015.

The overall status of IBM Notes Traveler is Green.

You‘ll find the default statistic values for yellow/red statuses in an IBM wiki

– http://www-10.lotus.com/ldd/dominowiki.nsf/dx/Status_command_considerations_and_examples_LNT853

Monitoring – Statistics

Issuing tell traveler stat show on the Domino console brings you all Traveler statistics

– http://www-01.ibm.com/support/knowledgecenter/#!/SSYRPW_9.0.1/statfields.dita


Statistics are also available using Domino Administrator. Some are very useful – some have only an informational level.


Analyzing connection issues and latencies

tell traveler stat show

[0A8C:0068-0924] Availability.Index.060-070 = 1

[0A8C:0068-0924] Availability.Index.090-100 = 48

[0A8C:0068-0924] Availability.Index.Current = 100

….

[0A8C:0068-0924] DCA.DB_CLOSE = 54

[0A8C:0068-0924] DCA.DB_OPEN = 41

[0A8C:0068-0924] DCA.DB_OPEN.Time.Histogram..000-001 = 37

[0A8C:0068-0924] DCA.DB_OPEN.Time.Histogram.CN=Traveler01/OU=srv/O=midpoints-trav.000-001 = 4

….


CPU.Pct.070-080 CPU.Pct.080-090 CPU.Pct.090-100

– Will be set when the CPU usage is within the defined range (i. e. 70-80% of the first parameter)

– Important parameter as Traveler performance can be affected heavily if CPU usage is 78% or more.


DCA.C.DB_OPEN.Time.Histogram.<server>.<bucket>

– A „bucket“ defines the time in seconds which was needed to open a mail file.

– This statistic helps to analyze latencies and connection issues to remote mail servers.

Monitoring – End-To-End

All shown mechanisms for analyzing Traveler health and status are missing some key elements:

– Is Traveler available from external and can a device synchronize?

– Are all involved components available (i. e. Internet connectivity, Firewall, Load Balancer, Reverse Proxy, Network to mail servers, mail server itself etc.)

An administrator needs to know issues before the CxO calls during BBQ.


Check if Traveler infrastructure works from external

– HTTP(S) request to /traveler?action=getStatus

– HTTP response code 200 mean: Traveler is available

Validations

– Traveler access (Load Balancer, Proxy, authentication)

– HTTP task active

– Traveler task active

– Mail server reachable

– Traveler database (SQL backend) available


Traveler Mail Delivery Confirmation Message

– Send a mail to a specific mail account (which is set on a device)

– Device fetches mail via Traveler

– The recipient will receive a confirmation mail if the mail has been delivered to the device

– If the recipient doesn‘t get the confirmation mail – Houston, we have a problem!



– Available since Traveler 9.0.0.1 IF2

– Must be explicitly activated via notes.ini

NTS_MAIL_DELIVERY_CONFIRMATION=true

– Allowed senders can be explicitly set

NTS_MAIL_DELIVERY_CONFIRMATION_SENDERS=MonitorMail1/Company



– Mail-Subject must begin with <$Confirm>

– Additional keywords/combinations:<$Confirm,RemoveOnDelivery><$Confirm,SuppressSaveInSentItems><$Confirm,RemoveOnDelivery,SuppressSaveInSentItems>

– http://www-10.lotus.com/ldd/dominowiki.nsf/dx/Using_Mail_delivery_confirmation_messages_with_IBM_Notes_Traveler

Monitoring – Port checks

80 / 443 - HTTP/HTTPS

50125 - CommunicationServlet => Traveler taskTraveler task => Traveler task

50126 - CommunicationTraveler task => Servlet

Port checks should always be combined with further/other checks.

– Port 50125 may i. e. answer but the Traveler task may not reach the SQL backend

Monitoring – Know your devices

Knowing your devices (os, os version type etc) is critical.

Agenda


Monitoring

Tuning

Troubleshooting

Tuning

Always watch the amount of devices in your infrastructure and update your configuration accordingly.

Important parameters are i. e.

– CPU and RAM

– HTTP threads

– Memory cache

– Maximum memory size

– Address cache

– Request size

Tuning

Assign CPU and RAM more then needed. 64bit is always recommended.

Tuning

HTTP threads => 1,2 * number of devices per server(default: 100 32bit / 400 64bit)

Watch out: RAM for all threads will be allocated a HTTP task start

Tuning

Maximum cached users:Number of users per server

Cached user expiration level:Recommendation: 28.800 sec (8h)

Tuning

Maximum Memory Size: varies depending of the user/device numbers

Should be minimum 1.024 MB (rule of thumb: always ¼ of available memory)

Tuning

Memory – tell traveler memCPU and Memory (MB) Usage History

Date CPU Pct Java Mem C Mem Avl Indx # Users # Errors # DB Conn

2015-01-02 15:01:53 BST 0.01 56 1307 100 5 1 0

2015-01-02 15:16:53 BST 0.01 44 1307 100 5 1 0

2015-01-02 15:31:53 BST 0.01 73 1307 100 5 1 0

2015-01-02 15:46:53 BST 0.01 39 1306 100 5 1 0

2015-01-02 16:01:53 BST 0.01 53 1306 100 5 1 0

2015-01-02 16:16:53 BST 0.01 66 1307 100 5 1 0

2015-01-02 16:31:53 BST 0.01 87 1307 100 5 1 0

…

Current Memory Usage

Java Memory Usage

Max Total 1024 MB

Current Total 96 MB

Free 940 MB (92 percent of Max Total)

Allocated 84 MB (8 percent of Max Total)

C Memory Usage

Allocated 1293 MB (33 percent of Total Physical)

Current Usage

Java 84 MB

C 1293 MB

Tuning

Traveler performance varies depending on multiple factors like mail database size and sync filter settings.

You‘ll see that in the size of the Derby / SQL database

– 850 User, 500 MB quota, no filter => 1 GB

– 850 User, no quota (Ø 2 GB), no filter => 5 GB

– 2.000 User, no quota (Ø 1,5 GB), filter ”1 year” => 4 GB

Largest seen Derby database: 17 GB (1.000 users, no filter)

U * (D * (700 * V + 4000)) = Recommended DB space available in bytes

Tuning

Usage – tell traveler dbusage

tell traveler dbusage

IBM Notes Traveler Database Statistics

Accounts: 11

Devices: 23

Total device documents: 35307

Device documents synced: 12124

Device documents filtered: 23183

Domino documents: 11056

Highest Total Usage Documents Percentage

------------------------------------------------------------------------------------------------

Rene Winkelmeyer/midpoints 7229 65.39

Michael Schloemp/midpoints/de 1471 13.30

Detlev Poettgen/midpoints 1302 11.78

Michael Ingendoh/midpoints 819 7.41

Benjamin Gaisser/midpoints/de 235 2.13

Tuning

Usage – tell traveler dbusage

Mail documents: 14321

Highest Mail usage Documents Percentage EMail filter

--------------------------------------------------------------------------------------------------------------------

Rene Winkelmeyer/midpoints 6773 72.66 unlimited

Michael Schloemp/midpoints/de 1254 13.45 unlimited

Detlev Poettgen/midpoints 743 7.97 30 days

Michael Ingendoh/midpoints 399 4.28 14 days

Benjamin Gaisser/midpoints/de 152 1.63 30 days

Calendar documents: 10085

…

Tuning

Tuning

Standalone (Derby)

– Frequent defragmentation

HA (IBM DB2 / M$ SQL)

– Frequent Runstats for table and index status

– Index ReOrg => TALK TO YOUR RDBMS-ADMIN

– Translog check (size and storage)

Tuning – DON‘T DO THIS AT HOME

Get all Traveler configuration parameters – tell traveler config

ADMINP_POLL_INTERVAL = 0

DEBUG_OUTFILE = "”

LOGFILE_DIR = "”

NTS_64_BIT = false (default = true)

NTS_ACCESS_ALLOW = "*:TravelerUsers" (default = "")

NTS_ACCESS_DENY = "”

NTS_ACCESS_ENABLED = true

….

NTS_FONT_CONVERSION = 0

NTS_FORCE_GC_MEMORY_LEVEL = 5

NTS_FORCE_OUTBOX_MAIL_TO_DRAFTS = false

NTS_FORCE_START = false

NTS_HOST_IP_ADDR = "11.11.11.100" (default = "")

NTS_HTTP_HEADERS_RESPONSE_X_IBM_TRAVELER_HOST = ""

NTS_IGNORE_TIMEZONE_ERROR = false

NTS_INSTALLATION_TYPE = ON_PREMISE

NTS_INSTALL_INSTANCE = ""

NTS_INTERNAL_BATCH_UPDATES = false

…

Tuning – DON‘T DO THIS AT HOME

More then 500 mostly undocumented parameters.

You should change them only if you they are documented or IBM support tells you to use them

– NTS_AUTOSTART_HTTP

– NTS_PUSH_APNS_SERVER

– NTS_ROUTE_LOCAL_BIAS

– NTS_STATUS_DATA_DIR_FREE_GIGABYTES_RED

Agenda


Monitoring

Tuning

Troubleshooting

Troubleshooting – Device setup

Can the device connect to the Traveler server?

Can the device open the Traveler website (/traveler)?

Can the user authenticate himself (wrong password)?

Is Internet Lockout active (you should use it if you don‘t have a secure reverse proxy in front of Traveler)?

Is the user allowed to use the Traveler server?


Can the Traveler server connect to the user‘s mail server?

Is the Traveler server allowed to connect to the user‘s mail server?

Is a cross-certificate for the Traveler server missing (if Traveler is hosted in another domain)?

Has Traveler Manager access rights (incl. Delete) for the mail database?

Has the user Editor access rights (incl. Delete) for the mail database?


Is „Replication of Unread Marks“ set in the mail database properties?

Has the mail database quota been reached (Traveler creates/uses two profile documents in the mail database)?

Troubleshooting – Logs

Central Log Directory

IBM_TECHNICAL_SUPPORT/traveler/logs

Enable Logging per User

tell traveler log adduser finest <username>

tell traveler log removeuser <username>

Dump user information

tell traveler dump <username>

Troubleshooting – Logs

Collect Information for a PMR and upload

tell traveler pmr <pmr_number>

If you cannot do that use the following

tell traveler systemdump

tell traveler log collect

Check IBM_TECHNICAL_SUPPORT\traveler\logs\<timestamp>

Engage Online

SocialBiz User Group socialbizug.org

– Join the epicenter of Notes and Collaboration user groups

Social Business Insights blog ibm.com/blogs/socialbusiness

– Read and engage with our bloggers

Follow us on Twitter

– @IBMConnect and @IBMSocialBiz

LinkedIn http://bit.ly/SBComm

– Participate in the IBM Social Business group on LinkedIn

Facebook https://www.facebook.com/IBMConnected

– Like IBM Social Business on Facebook

https://reg.socialbizug.org/wispubs/socbizwidgets.nsf/homepage.xsp

http://www.ibm.com/blogs/socialbusiness

https://twitter.com/IBMConnect

https://twitter.com/IBMSocialBiz

http://bit.ly/SBComm

https://www.facebook.com/IBMConnected

Notices and Disclaimers

Copyright © 2015 by International Business Machines Corporation (IBM). No part of this document may be reproduced or transmitted in any form without written permission from IBM.

U.S. Government Users Restricted Rights - Use, duplication or disclosure restricted by GSA ADP Schedule Contract with IBM.

Information in these presentations (including information relating to products that have not yet been announced by IBM) has been reviewed for accuracy as of the date of initial publication and could include

unintentional technical or typographical errors. IBM shall have no responsibility to update this information. THIS DOCUMENT IS DISTRIBUTED "AS IS" WITHOUT ANY WARRANTY, EITHER EXPRESS OR IMPLIED.

IN NO EVENT SHALL IBM BE LIABLE FOR ANY DAMAGE ARISING FROM THE USE OF THIS INFORMATION, INCLUDING BUT NOT LIMITED TO, LOSS OF DATA, BUSINESS INTERRUPTION, LOSS OF

PROFIT OR LOSS OF OPPORTUNITY. IBM products and services are warranted according to the terms and conditions of the agreements under which they are provided.

Any statements regarding IBM's future direction, intent or product plans are subject to change or withdrawal without notice.

Performance data contained herein was generally obtained in a controlled, isolated environments. Customer examples are presented as illustrations of how those customers have used IBM products and the results

they may have achieved. Actual performance, cost, savings or other results in other operating environments may vary.

References in this document to IBM products, programs, or services does not imply that IBM intends to make such products, programs or services available in all countries in which IBM operates or does business.

Workshops, sessions and associated materials may have been prepared by independent session speakers, and do not necessarily reflect the views of IBM. All materials and discussions are provided for informational

purposes only, and are neither intended to, nor shall constitute legal or other guidance or advice to any individual participant or their specific situation.

It is the customer’s responsibility to insure its own compliance with legal requirements and to obtain advice of competent legal counsel as to the identification and interpretation of any relevant laws and regulatory

requirements that may affect the customer’s business and any actions the customer may need to take to comply with such laws. IBM does not provide legal advice or represent or warrant that its services or products

will ensure that the customer is in compliance with any law.

Information concerning non-IBM products was obtained from the suppliers of those products, their published announcements or other publicly available sources. IBM has not tested those products in connection with

this publication and cannot confirm the accuracy of performance, compatibility or any other claims related to non-IBM products. Questions on the capabilities of non-IBM products should be addressed to the suppliers

of those products. IBM does not warrant the quality of any third-party products, or the ability of any such third-party products to interoperate with IBM’s products. IBM EXPRESSLY DISCLAIMS ALL WARRANTIES,

EXPRESSED OR IMPLIED, INCLUDING BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE.

The provision of the information contained herein is not intended to, and does not, grant any right or license under any IBM patents, copyrights, trademarks or other intellectual property right.

IBM, the IBM logo, ibm.com, BrassRing®, Connections™, Domino®, Global Business Services®, Global Technology Services®, SmartCloud®, Social Business®, Kenexa®, Notes®, PartnerWorld®, Prove It!®,

PureSystems®, Sametime®, Verse™, Watson™, WebSphere®, Worklight®, are trademarks of International Business Machines Corporation, registered in many jurisdictions worldwide. Other product and service

names might be trademarks of IBM or other companies. A current list of IBM trademarks is available on the Web at "Copyright and trademark information" at: www.ibm.com/legal/copytrade.shtml.

connected 2015 - ibm notes traveler daily business

Mobile

traveler hadr

dbntsdb notes notes

organized ibm db2

domino servers

installed traveler addon

single server

pool member servers

pool members