bp401 -- admin zero to hero

8/6/2019 BP401 -- Admin Zero to Hero

1/67

BP 401 - Admin Zero to Heroin 60 Minutes

The question is no longer, "How can we?"The question now is, "How should we?"

Andrew Pollack, PresidentNorthern Collaborative Technologies


2/67

Language Note

I realize that for some of you, English is not your primarylanguage, and for others, my accent is not the same as yours.

If you are having trouble understanding me during this talk,please raise your hand and I will try to slow down and speakmore clearly.

Thank you.


3/67

Wireless Devices

Wireless device noises are rude in any language. Please take amoment to turn off any of the following:

Cell Phones

Scheduler Devices

Pagers

Alarm Clocks

Pacemaker low-battery warning alarms

Anything else you are carrying on or about your personwhich may make noise during this

presentation.


4/67

About this Presentation

A "best practices" session is different This is not a list of product features.

This is a practical 'field guide' of which ones to use, and why.

Focused on What and Why, pointers to resources for how.

Designed for re-use These are not empty bullet points.

The details you need are in this text.

The Goal of this Presentation Provide an overview of what you should be thinking about as an

administrator

Provide a trail map for finding out more, and implementing the ones youfind of value

Help you start thinking in terms of the big picture rather than beingconstantly swamped by the details


5/67

Agenda

Who am I to be telling youanything?

The Scenario Setup

Server Stability Management

Security Management

Mail Management

Database Management

Client Software Management

End User Support


6/67

Who am I To Tell You Anything?

Andrew Pollack President, Northern Collaborative Technologies

2003 IBM Lotus Beacon Award Winner

1999 Lotus Beacon Award Finalist

Administrator & Developer since version 2.0

Member of the Penumbra Group

Firefighter Cumberland, Maine!

Lieutenant of Engine 1, Ladder 7, Heavy Rescue, RIT, Special Operations

In firefighting, just like Server Administration it's all in the planning

Why We're Here

To learn and grow as human beings

The question has changed, now it isn't "How Can we," it's "How Should We"

Also, I'm here because it makes the phone ring more


7/67

A Typical Environment

Three OfficesSoutheast The Home Office

Mid Sized, easy to get to, excellentnet connection

Southwest A Production Facility

Mid Sized, easy to get to

Northeast R&D

Small Office

Terrible Airport Access

Heavy Ground Traffic

Weather & Power Issues

Expensive Travel Costs

Then theres you

The new DominoAdministrator


8/67

ServerLoad & Hardware Choices


9/67

Clustering vs. Giant Boxes

Benchmarks are just statistics, and we know how much weshould trust those.

Would you really put 12,000 users on one server? 20,000?More?

Domino clusters do not shared any hardware or part of thesame operating system. They are fully redundant.

Balance the load across all the servers in the cluster, but makesure that if one goes down, the others can handle the loadwithout crashing.

A performance drop is acceptable for a brief period in most shops.


10/67

Clusters Provide High Availability, Low Cost


11/67

Domino Clustering is REALLY Easy

Put databases on both servers Make sure they replicate, and have proper access

Select the servers in the directory

Click "Add to Cluster"


12/67

Considering PeakLoads

We think of number of users dont do that.

Think number of concurrent users.

If you run three shifts, and only one shift is active at a time, you may beable to use smaller hardware.

Think total disk space.

Disk usage is critical on the server, even if it isn't in use it costs the serverresources to keep indexes and run checks.

In some customer sites, mailbox size dictates server count because ofdrive space limitations and the cost of massive storage networks.

For more information about clustering

JMP102 An Introduction To All Things IBM Lotus Domino Clustering --Gabriella Davis


13/67

Software Version Management


14/67

Operating System Choice

Which operating system is the best? Avoid politics, religion, and operating system preference discussions at the

dinner table

Either choose an OS that your staff knows well, or send them to school

All operating systems need to be patched and updated. Keeping up withthese is required for stability

Make a choice that is not unique in your company

Test, Test, Test

Watch out for case sensitivity when moving off Win32 Debugging can be very difficult because the initial hit to a resource is case

sensitive, but once the object is in the cache, it may not be.

BP403 Best Practices: IBM Lotus Domino for Linux -- Daniel Nashed


15/67

Remote Server Administration

No matter what tool you use, always use encryption

Many tasks you might think you need remote control softwarefor, can be done with the Web Administration Tool and the LotusDomino Administration Client

Editing the NOTES.INI on the server

Starting and Stopping Windows Services

Use the Server Controller and Java Console These can restart even crashed servers remotely

Start the server with "jc"

Stat the console from the Notes program directory "jconsole.exe"


16/67

Remote Control Software

Make sure it is set to lock the console automatically if yourconnection drops

Make sure it requires encryption for connections

Keep up with the vendors patches and updates for the serverside

Security patches could be critical

These ports are scanned constantly


17/67

ADMINP is your best friend

Properly configured, this will do a lot of the hardest and mosttedious work for you

Distribution of new databases to multiple servers

User move, add, or change requests

This becomes more and more important with each new version

of the IBM Lotus Domino server

Each server should have a replica of the "ADMIN4.NSF" fromthe administration server

For more information ID113 Maximize the Power of AdminP in IBM Lotus Domino -- Kathleen

McGivney, Susan Bulloch


18/67

Local Staff

Nothing is better than local staff

Before doing any kind of remote access work, compile a list of local contactstaff with phone numbers and availability

Have someone check the cdrom trays you do not want to reboot to a

setup disk

Nothing is worse than local staff

Control access to the sever

More on this topic when we talk security


19/67

Monitoring and Event Handling

Use Events Be the First to Know

Easy to set up

Know about problems before your phone rings

Fix problems before the boss calls you

Make sure to log them, so he knows what you do

Event notices make great justification tools for new servers!

For more information

BP407 What are Your Servers Trying to Tell You Now: The (Even) EasierRoute to IBM Lotus Domino Reporting & Logging -- Gabriella Davis


20/67

Heres what I use

http://www.cpscom.com/gprod/ipn.htm

Power-off Recycle Devices

When all else fails, sometimes you need to power cycle amachine from 3000 miles away

Inexpensive power modules can be commanded to recyclepower with a 5 second power down pause

Controlled through serial port

Include "watchdog" software

Many devices on the market Some include remote shell access

Some include Web browser control


21/67

Developer Management

Sir, please step away fromthat Designer Client.


22/67

Deployment Policies

These are a good thing, and you should have some.

Questions to answer with your deployment policies:

Who decides when a database has been tested enough? Who will be called when a problem is reported?

Do you have a contact number for this developer?

How will you know when the database is no longer in use?


23/67

More Deployment Policy Questions

How big is the database expected to get? What servers does it need to be on?

Is external replication required?

How volatile is the access control going to be?

What kinds of agent code will be running at the server?

Server side java agents? Agents that call COM objects?

File System Access? ODBC or Connector LSX Use?

API Calls?


24/67

Do Not Modify the Domino Directory

Nothing impacts performance more than changes to the DominoDirectory

There are two critical view indexes in the Domino Directory $ServerAccess

$Users

If the indexer is busy doing other things in that database, theseupdates will take longer

If these indexes are not up to date, authentication and accessrights may not be granted to users


25/67

ava Agents Must Be Tested at Full Scale

Multi-threading is so powerful, you can shoot yourself in bothfeet at once

Very easy and common mistakes in Java agents can killproduction servers easily

Unlike LotusScript, when writing Java agents programmers must call

"recycle()" on every object you instantiate, or their parent document

In test, it is frequently possible to get away with simply recycling the"session" object when the agent terminates

In production, this kills servers when the agent handles a large number of

documents in a loop, among other things

Yes, I know this from bitter experience


26/67

Restricted vs. Unrestricted Agents

Unrestricted agents can do to things outside the scope of theagent itself

Access the operating system

Access files on the server important ones

Reboot or shutdown the server

If someone needs to run an unrestricted agent, you need tounderstand why


27/67

Security Management


28/67

The Five Pillars ofSecurity

Physical Server Security Operating System / File System Security

Lotus Domino Server Access

Certificates & Cross Certification

Public / Private Key Certification

Cross Certification

Server Access Settings

Database Access The ACL

Document Access Reader Names


29/67

Notes Client Side Security

Guard Your Certifier Dealing with a compromised certifier

Assume Users have Designer It's easy to get

Obscurity is not Security

Encrypt Workstation Data Escrow ID Files

Preventing Workstation Copies Third Party Tool: dotNSF Tools noCopy www.dotNSF.com

Client to Server Communication Encryption


30/67

Browser Access Security

Obscurity is not security! This is the #1 issue on Web sites

URL Hacking

NoteID Crawling

Common Word Crawling

/database.nsf/knownViewName/

SSL Preventing Man in the Middle Attacks

Creating an SSL Key Ring

Obtaining an SSL Certificate

An authority unto yourself Are you trusted?

Buying an SSL Certificate

Deploying an SSL Key Ring to Domino


31/67

Securing the Other Protocols

Understand your ports If your server faces the internet, put a firewall in front of it

Many of the server tasks listen on a port, understand them or dont loadthem. Particularly, LDAP and SMTP can give away a lot of valuableinformation if improperly configured

If you dont need a protocol, shut it down

If nothing is listening on a port, that port is secure. Well, mostly.


32/67

Password Guessing isnt Just Browsers!

User's "Internet" passwords are frequently less complex thantheir Notes ID Passwords Use the tools to enforce complexity

It is now very common for hackers to "Name Guess" via POP3,SMTP, and even "Harvest" names from Web sites, e-mailaddresses, and open LDAP ports

Once a name is guessed or harvested, POP3 or other protocolsare used to guess passwords

With a name and password, spammers can use your serverusing an authenticated username


33/67

Mail Management

This is probably why many of you

came here in the first place.


34/67

Notes Mail Routing

Servers on the same Notes Named Network

Should be able to find each other "by name" without connection documents with TCPIP, this would be DNS

Servers on the same "named" network route mail automatically; no connectiondocument is needed

This is a "least cost" indicator to Domino's routing cost matrix

Use this to your advantage Set up your named networks to reflect your network's faster and slower links.

Put only servers that have excellent connectivity on the same "Named Network"


35/67

Connection Documents

Connection documents tell servers which are not on the same"Notes Named Network" how to find each other

They're also used for replication, but we'll get to that later


36/67

Internet Mail Routing -- Turning offSMTPinside the Network

If you turn off the SMTP Inbound Listener, local Windows clientswhich have been infected with a virus, worm, Trojan horse, orspy-ware application cannot send mail through your servers.

This also eliminates accidental or deliberate use of your internal servers for

spam routing. Even if you require password access for SMTP mail sending, password

guessing is now quite common.

If you disable SMTP Outbound on your servers, it will force themail to route through your single gateway. In many cases this isa more secure method and provides greater traffic control onyour network.


37/67

Using aSingle Internet Mail Gateway

Server Documents (all but the server that will route smtp):

Set "SMTP Listener" to Disabled Set "Routing Tasks" to "Mail Routing" but not "SMTP Mail Routing"

Create a "Foreign SMTP Domain" Domain Document Route *.* to "OurFakeName"

Create a Connection Document

Type: SMTP Source Server: The domino server with smtp

Destination Server: MAKE UP a name

Destination Domain: "OurFakeName"

Routing Task: SMTP Mail Routing

This method means you dont even need TCPIP as a protocolon your other Domino servers, because the routing all happensusing Notes RPC protocols to the one server with SMTPcapability.


38/67

Single Internet Mail Gateway --What Really happens?

All the servers where SMTP Mail Routing is not a task, look fora route to send the mail.

These servers see that *.* goes to the domain "OurFakeName" That's the SMTP Domain Document's Job

The router task on the servers see that one Domino server has

a connection to the "OurFakeName" domain so they route themessages to that server

That's the connection document's job

The server which is SMTP Mail Routing Enabled receives themail in its INBOX and knows how to send SMTP mail directly,

so it does.


39/67

Standardizing on a Mail Template

Beware of Customized Templates

Prevents Update & Bug Fix

Look at the update lists in each point release and note how many related tosmall fixes in the mail templates.

Serious Performance Issues

More views means more view indexing work for the server.

Limiting Design Access to Mail Files People are most likely to make "quick" (untested) updates to the design of their

mail file, considering it their own problem if they cause a problem. These peoplecan take down your server.

If you want additional features, look for "Packaged" alternativemail templates which are properly supported.

openNTF.org has a very popular one, for example.


40/67

Managing Mail File Size SCOS

Single Copy Object Store has been a feature for many years.

It DRASTICALLY reduces disk usage by keeping one copy ofeach file no matter how many different people have it in theirmail files.

It's significantly better than it was, and with "TransactionLogging" and Domino clustering can be much more reliable thanever before.

It's still a single point of failure if you do have a problem,everyone is affected by the problem.


41/67

Managing Mail File Size (continued)

Take Advantage of Archiving

Archiving can be easily set up and managed through policies

Put Archives on different server, they're less frequently accessed andhave different load characteristics

Impose Realistic Limits with Quotas


42/67

Managing Unwanted Mail

Don't be a Relay

In the "Configuration" document for your server not the Server document,on the "Router/SMTP:Restrictions And Controls:SMTP Inbound Controls"Tab

Deny messages from the following internet hosts to be sent to externalinternet domains:(* means all) Set to "*"

This is the Default on all recent Domino versions

Hold Undeliverable Mail

Don't send bounce messages Frequently, the mail never even originatedon your site and you're only adding to the problem

Fighting unwanted mail is much more complex than this BP405 Controlling Spam Mail In Your Organization

BOF509 Keeping Up with the Spammers with IBM Lotus Notes andDomino


43/67

Don't Give Away Address Information

Verify that local domain recipients exist in the Domino Directory:

Pros:

Stops inbound SMTP messages send with dictionary style drops and nameguesses from clogging your router

Can make your site less attractive to spammers who get credit for "delivered"messages accepted by your server

Cons:

Makes it easy for spammers to test for valid names on your server

Consider using this if you have another tool that can detectmultiple failed attempts from the same source and ban thosesources at the firewall.


44/67

Other Message Filtering Considerations

Using Black Lists (aka Real-time Black Hole or RBL) Many "black lists" exist that you can use

(e.g. bl.spamcop.net; sbl-xbl.spamhaus.org)

Not 100% accurate

Read the lists website to understand their criteria for listing

Using White Lists (aka "Known Good" addresses) Most mail you get, is from people you've communicated with already

New to version 7 of Lotus Domino, but part of several 3rd party tools forsome time


45/67

Mail Filtering Tools

Third Party Tools User-Interactive Products like spamJam can be excellent because each

user decides individually what's wanted and what's not

Appliance Solutions can be inexpensive and effective, but less user-specific

My Recommendations

spamJam because users really like being able to interact with it

Barracuda for simplicity and price, this device works very well

ASSP Open source proxy, good but scale is uncertain


46/67

Signed Mail

Signed mail to Notes users Your Public Key

Use "Files-Security-User Security" to get it or copy it from your Domino Directoryperson document

Signed Mail to Internet users X.509 Certificates The modern standard for authentication

Self Certifying

If you create your own certificate authority, everyone will always have todecide accept it as trusted

Excellent alternative for internal company use

Buying Certificates or Certification Rights

Free Certification Network


47/67

Importing Your X.509 Certificate

If you obtain a personal x.509 certificate, you can import it intoyour person document in the Domino Directory

Open your Person Document

Select "Actions Import Internet Certificates"

Once this is done, you can "sign" mail to be sent to users withInternet addresses


48/67

Verifying Signed Mail

From Notes Users The Lotus Notes Public Key

You must have their public key in your address book

Verifying Signed Mail from Internet Users

Accepting a Cross Certificate

Do this the first time you get signed mail from a user

Call the user, make sure its them sending the message


49/67

Adding aSender's Public Key to Your PersonalAddress Book

While viewing, use "Tools Add sender to address book" Advanced tab, check to add "x.509 certificate"


50/67

Mail Encryption

The Recipients Public Key is required

The Public Key is used to create a one-way cipher that can onlybe read with the private key and only the user has the privatekey, it's in their Notes ID file (or other file if a non-Notes user)


51/67

Obtaining a Recipient's Public Key

Notes Mail users in your domain already have it in their"Person" document in the Domino Directory.

Notes Mail users in other domains must send it to you. They cancopy it from their record in their Domino directory, or use the

options in "Files Security User Security" to get it.

Users can also simply send you a "Signed" document, and youcan "Cross Certify" them when you receive the mail. (You'll beprompted.)


52/67

Adding aSender's Public Key to Your PersonalAddress Book

While viewing, use "Tools Add sender to address book" Advanced tab, check to add "x.509 certificate"


53/67

Database Management


54/67

Deployment Policies

Limit Designer & Manager Access On the fly changes cause most problems

Use Database Access Groups to Delegate Control Create Groups that a database owner can manage

Example: "SalesTools.NSF Editors"

Set the database owner to be the owner of that group


55/67

The Connection Document for Replication

A connection document is required for replication even on the

same "Notes Named Network"A common error on the connection document is not changing the

schedule to work around the clock. Default is 8am-10pm.

Keep in mind that following replication, the indexer may be verybusy. Consider having replication occur prior to the start of the

normal business day.


56/67

Database Deployment Policies

Track Database Usage & Ownership

Every Database must have an Owner

Every Database must have a Review Date

Remove Outdated or Unused Databases

Even unused databases can load the server

Old data represents a security, accuracy, and legal risk


57/67

Replication Topologies

Avoid "Everyone Replicates with Everyone" Map Network Choke Points


58/67

Creating a Redundant Hub & Spoke

Two distinct local area networks or wellconnected individual networks

One high bandwidth connection between thetwo clustered hubs

Reduces traffic across the expensive long haulnetwork


59/67

Client Software Management


60/67

Common Policy Settings

Use policies to define ECL (Execution Control List) settings

Use policies to make sure users have the right replicas on thelocal workstations

Policies in version 7 can be much more rigidly enforced


61/67

Client Version Update Rollout

Excellent for ROI No more touching the desktop

Reduces support due to version/template incompatibility

BP404 Best Practices in IBM Lotus Notes Client Deployment --Steve Sterka, David Via

ID117 IBM Lotus Notes Deployment Made Easy -- Jeff Mitchell,John Paganetti


62/67

Handling UserSupport


63/67

Delegating Admin Roles Safely?

Version 6.x added granularity to "Administrator" access

Allows you to delegate specific areas of responsibility withoutgiving complete control to junior administrators.

Using the administrator task, you can allow area managers toregister users without giving them a certifier.


64/67

Admin Roles in Version 6.x

Full Access administrators Able to leap tall ACLs; impervious to Reader-Names

Administrators Use all the power of the administrator tool, but subject to database and

document controls

Database Administrators Manage databases, but not the server itself

Full Remote Console Administrators / View-only Administrators

System Administrators

No database controls, but plenty of server setup access

Restricted System Administrators

Restricted System Commands


65/67

Limit Use of Full Access Administration

Full Access Administration should only be used rarely, when aneed to override ACL or ReaderNames is required.

Grant this only to specific ID files. Make the administrator switchto this ID file when needed.

Create an "Event" notification to notify management any time

this level of access is granted. Use encryption on database you dont want full access

administrators to read.


66/67

In summary

It's no longer a question of whether or not something can bedone, it's a question of which is the best way to do it and why.

This presentation serves as a guideline, not a bible.

This has been a high to medium high level look at the featuresyou should be using, with pointers to where to find moredetailed information.


67/67

Thank you for playing!Were all Lotus professionals here, please ask your questions

so others can here the answers. You may also contact me

directly if you like.Please fill out your evaluations

The latest copy of this presentation will also be available atmy website: http://www.thenorth.com

For those playing the home game, direct questions & comments to:

Andrew Pollack

[email protected]

http://www.thenorth.com