ibm traveler management, security and performance
TRANSCRIPT
Traveler Management, Security and Performance
Gabriella Davis - Technical DirectorThe Turtle [email protected]
Who Am I?
Adminofallthingsandespeciallyquitecomplicatedthingswherethefunis
Workingwithsecurity,healthchecks,singlesignon,designanddeploymentofDomino,ST,Connec>onsandthingsthattheytalkto
Stubbornandrelentlessproblemsolver
LivesinLondonabouthalfofthe>[email protected]:gabturtle
AwardedthefirstIBMLife>meAchievementAwardforCollabora>onSolu>ons
Traveler Behaviour
Standalone
Domino Traveler Server
User’s Mail Server
User Defined Behaviour
HTTP Task
Traveler Task
JDBC Derby DB notes.ini
DominoServer
Security
names.nsf(default traveler policy settings)
LotusTraveler.nsf(traveler policy
and device settings)
server activity server configuration traveler behaviour
Domino Directories for lookups
User Mail Db
TravelerProfile Doc
device
High Availability
Domino Traveler Server User’s Mail Server
User Defined Behaviour
HTTP Task
Traveler Task
JDBC Derby DB notes.ini
DominoServer
Security
names.nsf(default traveler policy settings)
LotusTraveler.nsf(traveler policy
and device settings)
server configuration traveler behaviourDomino
Directories for lookups
User Mail Db
TravelerProfile Doc
device
SQL or DB2 Server
load balancer or DNS round robin
server activity
To ensure not all information is synced each time, the database contains state data which tracks what
has already synced and when
High Availability Behaviour❖ Servers in a HA pool must be accessed using the same
URL
❖ A round robin DNS or load balancer must sit in front of the servers
❖ Users can be forcibly bound to a specific server in the HA Pool using the Traveler “bind” command
❖ useful when debugging
❖ Tell Traveler Available On/Off
Verse
❖Available as a mobile application for iOS and Android
❖on Android it is the only Mobile application. Traveler is no longer available
❖Verse can connect to both your cloud based mail and an on premises Traveler server
❖Connections integration and photos are only available for cloud based mail
❖Functionality for Verse when connecting to an on premises server is not the same as that for Cloud connections.
Understanding Threads
Server Thread❖ There is a SINGLE thread on the Traveler server that scans target servers
❖ Target servers are servers that house users utilizing the Traveler service❖ Traveler issues a call to each server in turn
❖ NSFGetChangedDB❖ Lists all changed databases since last scan
❖ Very fast/efficient request❖ Traveler is served the list of all changed databases❖ Parses list and keeps changed databases that it is interested in
❖ I.e., mail files❖ Passes the list of changed mail files to the Prime Sync Thread❖ By default, Traveler will scan the same server at a minimum of three seconds
Prime Sync Thread❖ Scans the target mail files
❖ One prime sync can work with one mail file at a time
❖ Identifies what has changed in the mail file
❖ I.e., what is out of sync
❖ Passes to device sync thread/worker thread
❖ By default, there are 200 prime sync threads on a Traveler server
❖ Should never need to be manually changed
❖ notes.ini setting NTS_THREADS_PRIMESYNC requires more memory
❖ Once complete, passed to device sync thread/worker thread
Device Sync Thread/Worker Thread❖ The thread that does the work
❖ Sends changed data to device
❖ Retrieves changed data from device
❖ Touchpoint thread between mail file and device
❖ Default of 5,000 device threads
❖ Worker thread is for internal Traveler communication
❖ Default of 5,000 worker threads
Traveler Threads
Scaling Traveler servers is about CPU and number of threads, not users. A user with multiple devices
or who is set to maintain a large amount of historical data creates a higher demand
Architectural Decisions
Single Domino Domain❖ Same Domain
❖ Single Directory
❖ Low admin overhead
ORG DOMAIN
Traveler Server 1
Traveler Server 2
NAMES.NSFall users and servers
Mail Server 1 Mail Server 2 Mail Server 3
Separate Traveler Domain❖ Requires
Directory Assistance
❖ More secure
❖ Can easily support multiple domains
❖ Manage isolated HTTP passwords
TRAVELER DOMAIN
Traveler Server 1
Traveler Server 2
NAMES.NSFempty of users and
mail servers
Directory AssistanceOrgA domainOrgB Domain
ORGA DOMAIN
Mail Server 1
Mail Server 2
Mail Server 3
NAMES.NSFcontains users and
mail servers
ORGB DOMAIN
Domino Clustering
❖ Traveler servers support clustering of Domino mail servers
❖ failing over to most available for handling mail
❖ Little value in clustering Traveler servers themselves at a Domino level
❖ Traveler servers don’t hold much data worth clustering
Sizing❖ IBM “a server with 8 cores and 16GB RAM is estimated to support 2000
devices”
❖ 4 cores and 8GB RAM can support up to 1000 devices
❖ Multiple servers configured in a Connection Pool can handle more devices
❖ A 32bit server is not supported for High Availability
❖ Estimating the size of the enterprise DB requires understanding of how many documents and how much history is being maintained
❖ Limitations include the number of HTTP threads which is controlled in the server document
❖ Assume each device consumes 1.2 threads
Single Server❖ Mail is retrieved from the inbox of each user on their
mail server or cluster mate
❖ Sent mail is delivered directly into the mail.box of the user’s mail server
❖ Directory lookups are performed on user’s mail server by default
❖ State data is held in a derby database on the local disk of the Traveler server /traveler/ntsdb
High Availability❖ Traveler behaviour remains the same
❖ State data is moved from a local derby database to an enterprise database (SQL or DB2)
❖ All Traveler servers in a Connections pool use the same enterprise database
❖ Users are directed to a random server in the Connections pool either via a load balancer or round robin DNS
Moving to HA & Back❖ Enabling a server for HA
❖ Create and grant rights to the Enterprise DB
❖ /traveler/cfg/db/TravelerSQL.zip contains the DDL and manual scripts
❖ travelerUtil db set url=jdbc:db2://dew.turtleweb.com:50000/TRAVELER user=LNTUSER pw=xxxxxx
❖ Migrating data can take an hour or more, during which Traveler access to the server will be denied
❖ Disabling a server for HA will cause a prime sync on every device as no state data will exist
Configuring Directory Assistance For Lookups
❖Traveler will perform lookups against any directories defined on a user’s mail server
❖You can configure the Traveler server to use itself for lookups and build a custom Directory Assistance definition to be used entirely by mobile users
❖Set the notes.ini value NTS_TRAVELER_AS_LOOKUP_SERVER=true
❖Traveler can even honour reader fields in contact documents
Lookups on the Traveler Server❖Pros
❖Faster / more efficient as the databases should be local and the server can use the same directory cache for every Traveler user
❖Ability to custom design a directory to be used by mobile users vs the one used by full mail or application users
❖Ease of troubleshooting, one place to look
❖Avoids having to perform directory lookups on remote and possibly WAN connected servers
❖Multiple Domains
Lookups on the Traveler Server❖Cons
❖Increases load on Traveler server that is now also performing directory lookups
❖If directory lookups fail, they fail for all Traveler users, not just those on a specific server
❖Single point of failure
❖Potentially large consolidated directory lookups, unless extensive and performance impacting reader fields are used
Authentication❖ Devices store credentials in the Traveler profile
❖ That includes the password
❖ Password expiry / resets will cause Traveler on the devices to fail until the user manually updates their password
❖ Using an AD password via Directory Assistance would require the AD password to never change
❖ Here’s where we can do something clever with Domino HTTP passwords
Authentication Workaround❖ Set up Traveler servers in their own domain
❖ Replicate in the names.nsf from the Org domain but don’t allow the HTTP field to replicate
❖ Set a different password in the Traveler replica that is never changed
❖ If that’s too high risk, instead of using a replica of names.nsf from the Org domain in Directory Assistance, use a copy
❖ have an agent keep the copy up to date
❖ no possibility of passwords “crossing the void” and breaking anything
Traveler Management
Traveler Health❖On the Traveler server console type
❖Tell traveler status
❖The status that prints to the console will highlight any problems with the Traveler environment
❖Green means healthy
❖Yellow means there are problems that could affect some aspect of the Traveler service
❖Red means there are failures within the Traveler service
❖If status is reported as “Green” there is no further information provided
❖Yellow or Red status also show the source of the problem
❖[0FB0:0009-1608] tell traveler status
❖[11D0:0DB8-0CAC] The Lotus Notes Traveler task has been running since Tue Dec 12 23:36:23 GMT 2015.
❖[11D0:0DB8-0CAC] The last successful device sync was on Mon May 16 10:14:45:19 GMT 2016.
❖[11D0:0DB8-0CAC] The overall status of Lotus Notes Traveler is Green.
Traveler Health❖The statuses are triggered dynamically but can be overridden in notes.ini
❖http://ibm.co/1TDOsex for all available thresholds
❖Health is tracked on system resources for example
❖CPU over 70% usage flags as “Yellow”. Over 90% usage flags as “Red”
❖Java and Native memory usage. Over 85% flags as “Yellow”. Over 95% flags as “Red”
❖Once the Traveler server decides its system resources are overloaded, it will put itself into a “constraint” state where it will refuse to perform any new sync activities but will continue to try and complete the existing syncs it is working with
❖Both system memory and database connection thresholds will trigger constraints
Traveler Health - Threads
❖Health is also tracked on thread usage
❖Prime Sync or Device Sync threads that have been running a long time
❖503 errors mean that the server is out of available threads and is too busy
❖HTTP Thread count exceeding 80% of available will flag as “Yellow”
Traveler Health Over Time❖Tell Traveler Mem
Tell Traveler Status❖The server command “Tell Traveler Stat Show” also gives us more data
❖constrained.count - how many times, if ever, Traveler entered the constrained state
❖primesync.count.current - the number of currently running prime syncs
❖primesync. .inqueue - the number of currently queued requests awaiting a thread
❖push.users.total - the number of users registered for push syncing
❖push.devices.total is the same but for number of devices which may be 2x number of users
❖push.users.online - the number of users online receiving push data now
❖push.devices.http - the number of devices registered for HTTP push
❖also push.devices.activesync, push.devices.sms etc
Useful Traveller Tell Commands❖ bind showall - where are users assigned to servers
❖ “bind” users to specific servers
❖ DBMaint - database maintenance scheduling
❖ HADR show
❖ displays the status of all servers in the pool
❖ or the status of the standalone server
❖ Tell Traveler Help
Tell Traveler Show “User”
Tell Traveler Threads❖Threads total: 6
❖Threads available: 5
❖Threads busy: 1
❖Busy Thread Counts (Name: Current / Peak / Max) --
❖DS: 0 / 3 / 5000
❖PS: 0 / 5 / 100
❖Worker: 0 / 3 / 5000
❖TC: 1 / 1 / 50
❖Alarm: 0 / 5 / 20
❖HTTP: 4 / 9 / 100
Managing Users❖Tell Traveler Push Status User
❖Tell Traveler Push Status “Tim Davis”
❖Displays the status of that user and all their devices
❖Tell Traveler Reset Device User
❖Tell Traveler Delete Device User
❖both will take * for “any device” as a option
❖forces the device to rebuild all the Traveler data again
❖“Delete” also removes all personal preferences / data
❖These are both at the “fixup” level of solutions and so should only be used if other troubleshooting fails
What is SMS Mode?❖A standard mobile device configured to use Traveler uses HTTP(S) to
maintain a connection to the Traveler server to keep the device in sync
❖only works for Android, Windows Mobile and Nokia
❖This constant HTTP activity on a device can sometimes cause problems
❖battery can drain quickly if using 3G or 4G
❖some users might not have good or reliable data connections
❖An alternate option for those with unlimited SMS contacts is to use SMS to maintain the Traveler connection for auto syncing
❖The downside to this method would mainly be related to the cost of SMS messaging on the user’s phone plan
Enabling SMS Mode❖On your Windows, Android or Nokia device
❖Go into Traveler configuration “Lotus Notes Traveler”
❖Turn on Auto Sync
❖Enter your SMS email address (phone number + carrier domain eg)
❖Select “SMS Notifications” to “ON” or “Enable SMS Notifications”
SMS Mode Behaviour❖Once SMS Mode is enabled, the device no longer attempts to maintain a permanent connection to the Traveler server
❖The Traveler server instead sends an SMS update to the device when it needs syncing
❖The SMS message is sent silently and won’t trigger SMS notifications
❖Since the sync request comes from the Traveler server, it will only be initiated by server side changes
❖If SMS messages aren’t getting through it’s possible your carrier is blocking it because the from and to addresses are set as the same. To fix this you can set a custom “From” address for all SMS messages
❖Edit notes.ini
Traveler Security
Policies❖ Traveler policies can be applied as part of the Domino Directory or in
LotusTraveler.nsf
❖ If explicit directory policies exist in the names.nsf those override the Traveler server policies
❖ Traveler server policies are stored in the LotusTraveler.nsf and apply to all users connecting to that server
❖ In a multi domain environment, using LotusTraveler.nsf ensures consistency in applying policies
❖ It also ensures policies can be centrally created in a HA environment
TLS❖ Ensure you are using the latest Domino version
❖ Separating Traveler into its own domain helps with that
❖ Use TLS / SSL for configuring HTTP
❖ Disable_SSLV3=1
❖ Restrict less secure ciphers
❖ If you use BYOD security your options may be limited by what the oldest devices support
❖ Create 4096 certificates using OpenSSL to deploy on the Traveler servers or the load balancer
❖ Encrypt traffic between the LB and the Traveler servers
Traveler Performance
Cause of Problems: 1❖Security and supported certificates on older devices
❖Unable to find a route to user’s mail servers
❖Watch LocalDomainServer and OtherDomainServers
❖Unable to access user’s mail file
❖Unable to deposit mail in server mail.box on mail server
❖User authentication / password expiries or resets
❖Hostname - FQDN
❖If the server cannot resolve to it’s own hostname
❖Out of memory issues
❖Especially on 32bit systems
❖Ensure JVM memory set in advance and Domino memory minimised - turn off what you don’t need
Cause of Problems: 2❖HTTP threads
❖Tell traveler status
❖Tell http debug thread all
❖HTTP threading is not dynamic
❖Server pre-allocates memory
❖Ensure you have enough threads for all devices
❖There is such a thing as “too many threads”
❖Traveler server threads are dynamic
❖200 prime sync threads by default
❖5000 messaging threads
❖dynamically maintained. Can be carefully overridden in notes.ini
Server Resources -- What Does Traveler Care About?
❖Memory
❖More memory means more users, more concurrent devices, more threads made available but it’s only an option if you run at 64bit. At 32bit you will always be limited to 3GB available to Domino
❖Processor
❖Much of the activity on Traveler is in memory and retrieved from other servers but the JDBC Connection from the Traveler task to the derby database uses CPU.
❖Disabling any unwanted / unnecessary services and Domino tasks will help
❖If you have multiple mail servers used within your Traveler setup then the server to server connection between the Traveler and Mail servers will also require CPU
❖Deploy in production with 4 processor cores if Mail servers aren’t on the local network
Server Resources -- What Does Traveler Care About?
❖Disk
❖Separate disk for transaction logs
❖Domino works best with fast disks, so Traveler does to. Especially in a standalone configuration since there’s a lot of read and write activity to the derby data outside of Domino
❖Network and Connectivity
❖Mail servers which are constantly being probed by the Traveler servers should be on a LAN not WAN connection
❖Consider regional Traveler servers if your mail infrastructure is distributed
❖Be wary or clustered mail servers with cluster mates on remote networks
Disk Fragmentation❖The Derby database will get fragmented
❖Although this isn’t a problem for Domino, it will become a performance problem for Traveler
❖If the Derby data gets corrupted or deleted then every device will resync everything since it contains state information about last sync time as well as folder and document ids to identify what has synced
❖tell traveler quit
❖tell http quit
❖load traveler -defrag
❖Similarly in a HA environment, you need to monitor the DB2 or SQL health and ensure it is maintained
❖The DBMaint commands enable you to schedule maintenance
Logging❖Traveler output logs configured in server document
❖Written to IBM Technical Support directory
❖Command based logging creates date stamp sub directory
❖Lists all data
❖We can modify what is logged
❖Default 50MB limit on activity log file before new one created
❖Very detailed
❖You can modify
❖Be careful.. you can log the text body
❖Very malleable
❖You can increase logging for particular users
❖You can dump all stats to logs easily
Server side logging control❖Data is written to
❖..\data\ibm technical support\traveler
❖Default is informational
❖Can change via console or server doc
❖Tell traveler log level <level>
❖You can increase logging per user
❖Tell traveler log adduser <level> <username>
Extending logging❖List field types logged
❖Tell traveler log fields <fieldinitials>
❖S=Subject, B=Body, L=Location, A=Address, P=Phone
❖*=show all fields
❖blank=hide all fields
❖Do you want to log body text?
❖Dump all logs into date stamp directory
❖Tell traveler log collect
❖copies traveler configuration data and current logs to a zip file
