®®

Adminblast 2009. 60 tips in 60 minutes!Paul Mooney| Senior Architect | Bluewave Technology

BP101

®®

Adminblast 2009. 60 tips in 60 minutes!Paul Mooney| Senior Architect | Bluewave Technology

BP101

®®

Adminblast 2009. 60 tips in 60 minutes!Paul Mooney| Senior Architect | Bluewave Technology

BP101

5

About This Presentation� Provides you with a brain dump of useful tips

� Presentation is self-contained There is plenty to cover All the information you need to implement these tips is in this presentation

� There are over 60 tips Although some “tip pages” have more than one tip!

� If you have any trouble with a tip, email me at: pmooney@pmooney.net

6

Where Did I Get These Tips?� From my life as a consultant …

� Forums www.notes.net www-142.ibm.com/software/sw-lotus/support

� Blogs Just too many to mention! www.planetlotus.org

� Show 'n Tell Thursday Tips donated by bloggers All categorized and aggregated for convenience

– www.rhs.com/web/blog/gregate/v3test.nsf

7

Right... what about last year?� Adminblast 2008� Available on pmooney.net resources page

� This year, most tips are new� The more crucial ones from last year are still here

� Download last year’s to get more

� No need to take notes in this session

� Everything is in the slides that you need

� Just watch� We go a little deeper this year (more notes.ini goodness)

AGENT Tips

8

9

Tip #1: Agent Manager Commands� To see the schedule of agents to be run on the server

Type “Tell amgr show schedule” on the console

� To force an agent to run from the console tell amgr run “database.nsf” 'agent name'

– Note: Both the double quotes and single quotes are required for this to work

10

Tip #2: Agent Manager and CPU’s� For the record...

� The agent manager (amgr task) does not span CPU’s

� Multiple tasks can span CPU’s Different executives operate on different CPU’s

� BUT>>>> An agent can only use ONE CPU If you have one, really long/harsh agent.. this is an issue

� Faster CPU’s = good Even at the cost of multiple CPU’s Depends on your environment!

� Break your agent down to call other agents? Not always possible

11

Tip #3: Stop agents creating indexesEver seen this on your console?

"Warning: Agent is performing full text operations on database '<name>' which is not full text indexed. This is extremely inefficient."

This a BAD thing

FT_FLY_INDEX_OFF=1

Will prevent the agent from creating the index

Forces the agent to return with error “db not indexed”

BUT, it saves your server from unnecessary work

WEB Server Tips

12

13

Tip #4: Limiting the Availability of DWA Tabs� Domino Web Access (DWA) interface loads all tabs by default

Welcome Page, Mail, Calendar, To Do, Contacts, Notebook

� What if you want to limit the tabs? For instance, only have users see the Mail and Contacts tabs

� Starting in 7.0, the ini variable, iNotes_WA_Areas, allows enabling and disabling of tabs

14

Tip #4: Limiting the Availability of DWA Tabs (cont.)� iNotes_WA_Areas=1,1,1,1,1,1

Where 0 = disable and 1 = enable and the tabs are in order of Welcome Page, Mail, Calendar, To Do, Contacts, Notebook

� Example: iNotes_WA_Areas=011010 Only Mail, Calendar, and Contacts are visible

15

Tip #5: Troubleshooting HTTP Configuration� Issue a Tell command at the server console

Tell HTTP Dump Config

� Dumps HTTP configuration to HTTPCFG.txt in: x\Lotus\Domino\data\IBM_TECHNICAL_SUPPORT

� Excellent method of getting full configuration of HTTP stack and includes: Rules Port settings

� Don’t forget “Tell HTTP refresh” Rereads HTTP config without restart

16

Tip #6: HTTP and Internet Passwords� HTTP password changes are cached on the server

� Can confuse users At some point, both their old and new passwords work!

� HTTP_PWD_CHANGE_CACHE_HOURS=0 Set in the notes.ini file Prevents both passwords from functioning The moment the user changes his/her password, it is the only active password

17

Tip #7: HTTP Web Agents� Agents initiated by an HTTP application are NOT managed by the AMgr task

They are managed by the HTTP task This is useful to know in order to debug major HTTP peaks

18

Tip #8: HTTP Web Agent Timeouts� Troublesome HTTP Web Agents can hurt HTTP performance

Reporting Agents are an example For example, if the user clicks a button that generates a large report over a browser, this could

affect the HTTP stack (consuming resources)

� Set HTTP Agents to timeout after x seconds Default is not set Poorly configured HTTP triggered agents could “hurt” your server

19

Tip #9: Restricting Web Access � The database property, “Don’t allow URL open,” ensures a database will not

be opened from a Web client Restricts: http://Host/Database.nsf/* commands

– Sends 500 Not Authorized error

20

Tip #10: Restricting Web Access (other way)� Certain databases, you just don’t want people to browse to

names.nsf for example

� Create a rule redirecting all requests to that URL to a HTML file

� You don’t need Internet sites enabled but i’m assuming you do

� Go to the internet sites view

� Create a global web settings document

� Give it a name and save it

21

Tip #10: Restricting Web Access (other way)� Open the global settings document and create a rule

� Point all requests to the database to a html file you have written..

22

Tip #11: Make a busy HTTP server “serve” faster� Domino HTTP Server uses round robin method to serve requests

Each request is added to a thread sequentially Even if the thread is already busy

� Example - one person is clicking on a ?OpenAgent URL string That thread is busy but STILL gets requests dumped to it in turn

� HTTPQueueMethod=2 Goes to the “old way” A pool of requests is created HTTP Threads take a request when they are free

� Typically a excellent performance enhancement on servers

23

Tip #12: Setting iNotesLite as the default� iNotesLite is the new black...

� Fantastic, fast interface Not full functionality, but more then enough to work

� Want it to be the default for all users?

� iNotes_WA_DefaultUI=dwa_lite 8.0.1 and higher only

24

Tip #13: Make SSL sites run faster� Creating SSL keys for encryption for each session is processor expensive

Key generation is kinda hard ya know!

� Domino retains some of the calculation cache for the last 50 sessions To speed up the key generation

� Enter SSL_RESUMABLE_SESSIONS=0 Domino retains unlimited amount of key caches

� Substantial improvement in performance NO EFFECT ON SECURITY

CLUSTER Tips

25

26

Tip #14: Clustering Over Private Ports???� Over the past 10 years, I have been recommending to cluster traffic over

private ports

� Exactly how to do that is in 2008 Adminblast presentation pmooney.net site - resources page

� I’ve change my mind

� Good data centre/resilient NICs(teaming) / Resilient switches No need to do it anymore

� Exception could be vmware You may want a dedicated NIC

27

Tip #15: Cluster auxiliary port (ND8)� What if the private NIC fails?

� You should have standard replication as a backup to cluster replication anyway

For a few reasons!

� Server_Cluster_Auxiliary_Port=* Tells the cluster engine to failover to alternative port if default port is unavailable

28

Tip #16: Cluster Work Queues� Cluster performance is dependent on the work queue depth

Type “Sh stat replica” on the console Look for cluster work queue depth statistics

� Work queue depth is critical to cluster performance

29

Tip #17: Customising Client Errors� When a server fails with a database open, a user gets this message, known

as Error 0807

� Or, a user might get this message, known as Error 0A02

30

Tip #17: Customizing Client Errors (cont.) You can change the error messages by adding these lines to your NOTES.INI

file Err_0807=Your e-mail server is no longer responding. However, you may be able to switch to a

backup server and continue working. To do this, you must close your mail file and re-open it. Err_0A02=Your e-mail server is no longer responding. However, you may be able to switch to a

backup server and continue working. To do this, you must close your mail file and re-open it.

� Go to www.pmooney.net and click on resources for a free tool!

31

Tip #18: Making Server Availability Index work!� The Server Availability Index is a mystery

If you want to understand it, go to/download the clustering session with Kathleen McGivney and myself

(bring a calculator)

� If you want it to cheat.. Type in SH AI on the Domino 7 console (or higher) It will tell you what to set the SERVER_TRANSINFO_RANGE= value to in the ini file Availability index will be more accurate now

� You have to check this frequently

� Base your calculations when server is busy

32

Tip #19: Clustered servers in connection docs� Notes clients are cluster aware

� So are Domino servers!

� If you want full resilience in replication/routing

� When creating a connection document Destination server field Put in the cluster name of the server you want to connect to!

PERFORMANCE Tips

33

34

Tip #20: Port Compression� Enable network port compression!

� This is especially good for server-to-server communication Must be enabled on both sides

� Up to 60% compression of data

35

Tip #21: Anti-Virus Software on Domino Servers� OS platform AV software

Exclude your TMP directory Exclude your data directory Exclude your Txn directory Exclude your view rebuild directory

� These folders constantly change

� Performance of your server can be hampered by the OS AV software

� Use Domino AV software for the Domino engine Alternate your AV products between the platform and

Domino engine!

36

Tip #22: Its all about the disks...� Domino is very reliant on disk performance

� Know your arrays!

� Ensure if you are on a SAN that you have dedicated LUN and disks for Domino data directory

Make the SAN administrator promise!

� Domino is disk read intensive More so then disk write intensive

37

Tip #23: Performance and Transactional Logging� Transactional Log files should NOT run on the same RAID array as the data

drive

� Separate disks Local if possible (Raid 1) If on a Storage Area Network (SAN), use local drives for the Txn logs/view rebuild directory

38

Tip #24: View Indexes and Performance� View index rebuild directory

Directory used by update/updall tasks for rebuilding indexes Usually the windows TEMP directory

� Set path — separate array/disk if you can View_ReBuild_Dir=f:\views Huge performance boost

39

Tip #25: Performance Enhancement� PleaseDearGodLetTheNotes8ClientRunFaster=1 ;)

Myth … Does not do anything

– But sometimes it makes administrators feel better …

� 8.02 is faster..

40

Tip #26: Make Notes 8 starndard run faster� Close Notes fully (even restart before doing this)

� Open this directory NotesProgramDirectory}\framework\rcp\eclipse\plugins\com.ibm.rcp.j2se.{Version}

� Look for the file jvm.properties

� Look for this line.. vmarg.Xmx=-Xmx256m

� Change the 256 to approx 1/2 your RAM available

� More memory used but significantly faster

41

Tip #27: Database Icon management� Get familiar with database redirects!

Moving a database Deleting a database Just create Redirect tool

� Very simple to use from the admin client

� You can point different people to differentservers!

� These WILL become a part of your everyday admin life

42

Tip #28: Check your buffer pool� The NSF buffer pool is automatically calculated by the Domino server

Based on 32/64 bit, amount of physical memory

� The Notes Indexing Facility operates in the buffer Then writes/reads to disk if it needs to

� Check how much of the buffer is in use Show Stat Database.database.b*

� Over 95% is good Much of the information needed is in memory

� 99.9% is bad... Too many requests may be writing to disks Server needs more buffer

� NSF_Buffer_Pool_Size_MB= to increase amount Sametime server needs a small buffer This works better for partitioned servers

43

Tip #29: Prevent simple searching� Simple searching hurts a domino server

Searches a database that does not have an FTI CPU intensive

� You can prevent simple searching now

� Database properties

44

Tip #30: Do you need unread marks?� All to many times

� Poor performing application servers

� All databases have unread marks maintained/replicating

� Is it needed? Does the application benefit from it? Do the users benefit from it?

� Turn it off!

45

Tip #31: Separate View updates from FTIs� If you have many applications

� Busy update task View updates suffer as Full Text Updates are still in operation

� Separate the FTI update from the view update FTI updates happen in their own memory thread Performance improves

� Update_Fulltext_Thread=1

46

Tip #32: Take Full text indexing out of Domino memory pool

� You can take the FTI thread out of the limited Domino update pool

� ftg_use_sys_memory=1 Full text thread now gets memory from the OS pool Frees up domino system memory

Mail Tips

47

48

Tip #33: Remove MIME conversions from log� Have you ever seen this on your console?

� Message converted to MIME format....

� Annoying isn’t it? Happens all the time

� You only want to know when it doesn’t work

� Converter_log_level=10 Only errors logged

49

Tip #34: Pasting images into mail� From 8.5 on onwards, images pasted into emails are compressed!

� Huge saving on disk space!

� Did you know, a notes.ini setting has been available since 6.5.4?

� OptimizeImagePasteSize=1

50

Tip #35: Delayed email notifications� Domino is used by really big companies

Still, typically no more then 6 hops MAX to get mail from sender to recipient

� Server failure causes NDR mail to sender

� What if its not your fault? Comms line down Other server down?

� What if the message is pending, waiting to get to the recipient

� You can warn the sender that the message is delayed!

� Configuration document

�

51

Tip #36: Change Attachment Handling on Client� Notes client gives options with attachments

Users get confused Too many options!

� Remove the options! AttachmentActionDefault=1 Double-click on attachment opens attachment by default

52

Tip #37: Configure the ND8 Out-of-Office (OOO) Agent� ND8 OOO is a router service

Not driven by the agent manager

� Many advantages Immediate replies to mail messages You can set the OOO for less than a day

� Configuration is easy Configuration document

– Router SMTP tab– Advanced– Controls

Set OOO type to Service

� Important... Enable it on both servers in a cluster at the same time

53

Tip #38: See Who Has OOO Enabled� Two specific ways

Type in “TELL ROUTER O” on the console

� Admin client Files tab Column on right hand side

54

Tip #39: Inbox maintenance feature � Excellent improvement on mail server performance

Especially large mail farms

� Administration process controls the task Defaults to Saturday at 1am

� Configured in the server document Administration Process section

� Choosing users From the server document OR from a mail policy setting

55

Tip #40: Speed up mail between different DNN’s� By Default mail routed between servers in the same DNN can be multi

threaded The router can open many connections to other servers Mail routes quicker

� If the servers are in different DNN’s A single thread is created Messages go through “single file”

� With RouterAllowConcurrentXferToAll=1 Multi-threaded connections between servers in different DNN’s or domains are now permitted Yay!

56

Tip #41: Simple one.. fix long URL’s in emails� Sometimes people using “other” email systems have horrible times

interpeting long URL links sent by Lotus Notes Half the link is URL Half is not User has to copy paste User gets confused/angry At this point, we are supposed to care I think

� Tell your users to encapsulate the URL with “”

� Outlook will then display the URL correctly

57

Tip #42: Quick Blackberry tip!� Disclosure... I love this one!

� Want to ensure an email gets to a blackberry? I never got that message....blah blah blah!

� At the START of your subject place <$Confirm> in the line

� The unit will send you a confirmation receipt

� There are other tags you can use... <$Confirm,RemoveOnDelivery> is an example

SECURITY Tips

58

59

Tip #43: Secure Ports� By default, all ports for tasks in Domino are enabled

LDAP, POP3, IMAP, HTTP, etc.

� By loading the task that supports one of these ports, the port is instantly active

Could happen by accident

� Disable the ports in the Server documents for tasks you do not use

60

Tip #44: Remove .mdm Files� Old modem (.mdm) files still exist on servers in the MDM directory

� Remove the files unless you need them

� Possible security risk Cell phone connected to port at back of server Simple .mdm file used to accept connections Port-enabled – admin guy unaware …

61

� The ID files for servers are usually created without a password If captured, this can compromise security

� Place a password on the server.id file Restart server command does not require password to be entered – the password is cached Automatic fault recovery does not require password

� Don’t hate me for saying this!

Tip #45: Secure the ID Files of Your Servers

62

Tip #46: Protecting Certifier IDs� Single password on a certifier is not enough

� Place multiple passwords on the ID file Too many times I have seen the password set to “lotusnotes”

� Alternatively implement and use the CA process

GENERAL Tips

63

64

Tip #47: Checking Policies on Clients Want to check what policies have been applied to the client locally?

Open the users’ local address book Hold CTRL + SHIFT and choose View Goto from the menu Open the $Policies view

For more information on policies and troubleshooting, go here: www.pmooney.net/blogsphe.nsf/d6plinks/PMOY-6M326J#comments

65

Tip #48: Checking Server Connectivity Tool� JPING

Trace testing for Domino servers www-10.lotus.com/ldd/sandbox.nsf/ecc552f1ab6e46e4852568a90055c4cd/

222294a30361b72888256aa20083873e?OpenDocument&Highlight=0,connect

� Always very useful to test ports and DNS Similar to traceroute and ping for Domino!

66

Tip #49: Get Domino to Use Text Files� You can write a “batch file” of commands

� Save it into the domino directory E.g., batch1.txt

� Run batch file on console “< batch1.txt”

67

Tip #50: Diagnostic Collection� You can configure the diagnostic collection feature on servers

Crash information is sent to a mail-in database!

� Build the fault reports database Set as mail-in database

� All .nsd and console.log information will be mailed to the database Fault Analyzer in ND7 Collates and displays information relevant to the crash(es)

68

Tip #51: Collecting All Database Information from the Catalog

� CATALOG.NSF database records all database information that has the following fields checked in database properties

69

Tip #52: Collecting All Database Information from the Catalog (cont.)

� In fact … the catalog.nsf holds ALL database and template information!� Hold down CTRL + SHIFT when opening a database� Select $ReplicaID view� All databases/templates listed

Regardless of the db property checked Very, very useful for audits

70

Tip #53: Fixing Bookmark.nsf Corruption� In the case of local design element corruption, you don’t always have to

delete bookmark.nsf! For example, users get errors when opening databases from the server Removing the cache and bookmarks resolves it It really annoys users!

� Open the bookmark.nsf database� Hold CTRL + SHIFT and click View Go To

Select (ByURL) view Select all the documents Delete them

71

Tip #54: Unread Marks in Replica Databases� Sometimes … just sometimes … go out of sync …

If you need to sync the unread marks Hidden tool

� Unstack the replicas on the workspace� Hold CTRL and select the two databases� Hold CTRL + SHIFT and select View Unread Marks Exchange Unread

Marks

72

Tip #55: Restart Server Delay� Restart server command defaults to a 10-second delay

� Sometimes this is too short Running additional tasks

– Sametime– BlackBerry enterprise server

� Make it longer! SERVER_RESTART_DELAY=n

– N = value in seconds

73

Tip #56: Want to list all database information?� Sh Dir -xml >mydata.xml

Writes an xml file to the domino directory All data in there Open in browser/import into spreadsheet

� Or

� Julian Robichaux has a free tool Will export all information from the “Files” tab to a csv file foryou Its free! http://www.nsftools.com/blog/blog-08-2008.htm#08-22-08

74

Tip #57: Get rid of all users with manager access to mail files

� Very common problem User registered with manager access to mail file

� User can now go wild...(and they will) Delete file Encrypt file Screw with design..

� Free tool!

� Checks for owner field in mail file and resets ACL to whatever you want

� Runs on server

� pmooney.net/resources page

75

Tip #58: How can I tell what changed in the server doc?

� Did you know that the Domino server records changes to the server document automatically?

� Every time you make a change, a new date/time stamped file is stored in the IBM TECHNICAL SUPPORT directory on the server

� DXL Files.. open them in a browser or import into another product

� Nice troubleshooting tip!

76

Tip #59: When upgrading - copy templates� Did you know that the system databases have had the same replica id for

years? pubnames.ntf admin4.ntf most of the others

� It is a GOOD idea to create a new database copy of the templates before you plan start a domain upgrade

Prevents other copies of templates causing replication issues for you

77

Tip #60: Get Domino to Create Text Files!� To get a list of mail files on the server

Ones that the server executing this command is the home server

� Type in the following on the console Nconvert -I mailfiles.ind

� Mailfile.ind file written All files listed

� Use this in console commands! E.g., nconvert -u mailfiles.ind * mail8.ntf

78

Tip #61: RSS readers and Lotus Notes client� The Lotus Notes client RSS reader...

No comment..

� One of its issues is you CANNOT import OPML files into the reader Unlike just about every other RSS reader

� There is a file on your client called presetfeeds.opml x:\Notes\framework\rcp\deploy\extras directory

� Close Notes

� Replace that file with yours

� Restart Notes Subscribe to pre-sets

� Done

79

Tip #62: Standardise Notes deployment� Power of the workstation is the issue with Notes 8 standard

Too slow? Too old?

� Nightmare of deploying different packages of client software

� Solution Deploy the standard (full fat) version of Notes 8 On machines too slow

– UseBasicNotes=1 This makes the basic (low fat) version load

� Use a policy to deploy this!

80

Tip #63: Release Notes

�Read them

�They make your life easier!

�1/2 hour reading and you are up to date

81

Tip #64: Your service

�Your service is key

�Protect it

82

Tip #65: Hidden secret of the geeks� We answer questions for beer ;)

� See you at the bar!

83

Thank you...

pmooney@pmooney.net (geekmail)

pmooney.net (geeksite)

www.twitter.com/pmooneynet (geektweets)

bleedyellow.com (geekchat)

paul_mooney@hotmail.com (geekmsn)

pmooney.net (skype) (geekcall)

www.bluewave.ie (worksite)

paul.mooney@bluewave.ie (workmail)

paul.mooney@besystems.eu (moreworkmail)