ccd
TRANSCRIPT
Compact, Compress, De-DuplicateUlrich Krause
ILUG, 10. – 12. November 2010,
Belfast, Ireland
www.eknori.de
About: Ulrich Krause
Lotus Notes / Domino Administrator & Developer since 1993
OpenNTF project chef
– !!HELP!!
– Cebulon
– TriggerHappy
German Notes Forum (http://www.atnotes.de)
Blog http://www.eknori.de
www.eknori.de
The situation
More and bigger documents and attachments
More expensive storage is needed
The average size of an application increases
Duplicate attachments ( n times )
Duration of standard tasks increases
More time for Backup
Network traffic increases
I / O and performance issues
www.eknori.de
The aim
Leverage existing storage
Reduce size of applications
Avoid attachment redundancies
Speed-up standard tasks
Reduce backup time
Reduce network traffic
Reduce disc I / O
www.eknori.de
Application model ( simplified )
Design
Documents
– Text
– Attachments
– Embedded objects
Design
Text
Attachments
www.eknori.de
Compact Secrets
DEBUG_ENABLE_COMPACT_8_5=1
– W nn - Only compact databases which haven't been compacted
successfully in the last nn days.
– w - Exclude system databases (e.g., log.nsf, names.nsf).
– X nn - Limit each database compaction to no more than nn minutes.
(Applies to in-place compaction only)
– x nn - Limit total compaction time to nn minutes.
– # nn - Execute up to nn compactions in parallel using different threads.
• Don‟t use more than 8 threads in parallel
– ODS performs a copy-style compact only on databases with an ODS
smaller than the ODS provided with this parameter.
www.eknori.de
Compact Secrets (Sample)
Load compact -# 8 mail\
Compacts all applications in the mail subdir in 8 parallel
Threads
www.eknori.de
Compact via application properties
Compacting a database does not
necessarily reduce its size
If the Notes database requires
structural changes (typically a change of
ODS [on-disk structure]). then a
copy-style compaction will be performed (equivalent to -L).
If the Notes database is logged (Transactional Logging
enabled), then a simple in-place compaction will be
performed (equivalent to -b).
If the Notes database is not logged (Transactional Logging
disabled), an in-place with file size reduction compaction will
be performed (equivalent to -B).
www.eknori.de
Copy Style Compaction
The compact -C (copy-style compaction) (1) creates a new
empty application container on the same server with the
same replica ID and a .tmp suffix, (2) copies all of the
documents from the original application to the new application
container.
After all documents have been populated in the new
application, (3) the old .nsf application is deleted and the (4)
.tmp file is renamed to the original filename.
www.eknori.de
Defrag
File fragmentation saps Domino server I/O performance
Even a slightly fragmented file can cause a 50% decrease in
non-cached read performance
Load compact –C does not allocate space needed before
compacting the application
Applications are fragmented during copy-style compaction
Full Text Indexes are fragmented too
Even NLO files in DAOS repository are fragmented over the
time
Use a tool to defrag your files speed up your Domino server !
www.eknori.de
Defrag
DefragNSF
• Adam Osborne
• http://www.preemptive.com.au/defrag
Domino Defrag
• Andrew Luder, OpenNTF
• http://www.openntf.org/Projects/pmt.nsf/Pr
ojectLookup/DominoDefrag
Both products use
the Microsoft
defragmentation
API
www.eknori.de
Defrag
Sample: Lotus Software KnowledgeBase ( 5 * compact –c )
Before
– 68 fragments
– 7550 ms
After
– 1 fragments
– 125 ms
www.eknori.de
Defrag
Domino Defrag uses IBM Lotus Notes C API
Open Source
FREE
Well documented
32/64 Bit support
New in Version 3: Free Space Defragmentation
– Uses jkdefragcmd.exe (http://kessels.com/jkdefrag/)
There is also a professional support plan available
www.eknori.de
Design Note Compression
www.eknori.de
Design Note Compression
New feature as of Domino 8
Needs ODS 48 or above
– Create_R8_Databases=1
– or Create_R85_Databases=1
– load compact -c
– load compact -c –n
Check “Compress database design”
Reduces size of design by 55 – 60 %
Reduces disc I / O
www.eknori.de
Design Note Compression
Design
Text
Attachments
Design
Text
Attachments
www.eknori.de
Document Body Compression
www.eknori.de
Document Body Compression
New feature as of Domino 8.0.1
Reduces size of Rich Text up to 60%
Needs ODS 48 or above
load compact -c –v
Check „Compress document data“
www.eknori.de
Document Body Compression
Design
Text
Attachments
Design
Text
Attachments
www.eknori.de
LZ1 Attachment Compression
www.eknori.de
LZ1 Attachment Compression
As of release 6
Replaces Huffmann algorithm
NOT active when creating a new
applications.
Recompress using
• load compact -c –ZU
• In 6.5.6 also set
DEBUG_ENABLE_LZ1_REPAIR=1
To avoid problems with JAVA applets
in <= 8.5 set
• DEBUG_ENABLE_LZ1_HOST_TYPES=0
• fixed in 8.5.1
www.eknori.de
LZ1 Attachment Compression
There have been problems reported that the fulltext search
does not work as expected using LZ1 compression.
To revert back to Huffmann use
– Load compact –c –ZD
– In 6.5.6 also set DEBUG_ENABLE_LZ1_REPAIR=1
www.eknori.de
LZ1 Attachment Compression
Design
Text
Attachments
Design
Text
Attachments
www.eknori.de
DAOS Estimator
www.eknori.de
DAOS Estimator
Free tool to plan your DAOS rollout
• Download: http://www-
01.ibm.com/support/docview.wss?rs=463&uid=swg24021920
www.eknori.de
DAOS Estimator (notes.ini)
DAOS Estimator by default has a large gap in attachment
size evaluation between 64k and 1 MB
– DAOSEST_BUCKETS=16,64,128,256,512,768,1024,2048,3072,4096
– The above line sets to:
– 16K, 64K, 128K, 256K, 512K, 768K, 1MB, 2MB, 3MB and 4MB
Source: Lotus Software Knowledge Base
• http://www-
01.ibm.com/support/docview.wss?rs=0&q1=1418102&uid=swg21418102
www.eknori.de
LZ1 != HUFFMAN
Db1 – LZ1 enabled
Db2 – LZ1 disabled
Result is 2 .NLO Files !!
www.eknori.de
D A O S
www.eknori.de
DAOS
Domino Attachment Object Service
Attachments are no longer stored in application
Attachments are stored as NLO (Notes Large Object )
Document contains only a ticket
Separates static and dynamic data
Only one copy of a bit-like attachment
Transparent to users and 3rd-Party Products
www.eknori.de
Prerequisites
Domino 8.5 ( ODS 51)
SHARED MAIL OFF
Transaction protocol
Create DAOS Repository
Activate DAOS on server
Activate DAOS on application
www.eknori.de
Switch OFF Shared Mail
Serverdocument – Shared Mail
Field Shared Mail = None
„Load object unlink shared.nsf“
www.eknori.de
Transaction Protocol
Server document – Transactional Logging
Required for DAOS – no discussion!!
Use a separate disk drive.
Create log file folder
Choose method
• Circular
» 4GB limit
• Archive
» No limit, backup will delete logs
• Linear
» Like circular, NO size limit
CREATE_R85_LOG=1
Restart server
www.eknori.de
DAOS Repository
Use separate disk drive
Can be SAN, NAS
Each server has it„s own repository
– Max.1.000 subdirectories
– Max 40.000 .NLO / subdirectory
– = 40.000.000 unique .NLOs
– Daos.cfg ( in Domino Data Directory)
– Daoscat.nsf
.NLO files are encrypted with server.id
– DAOS_ENCRYPT_NLO=0
Include repository in backup!!
www.eknori.de
DAOS Repository
How do you change the number of DAOS NLO's per
directory?
DAOS_MAX_FILES_PER_SUBCONTAINER=40000
Requires a server restart
Will only affect the current DAOS subdirectory (older
subdirectories are not changed).
This will also affect the maximum number of files in the DAOS
store which is currently this values multiplied by 1000 (this
value is not configurable currently).
• 1.000 * DAOS_MAX_FILES_PER_SUBCONTAINER = MAX_NLO
www.eknori.de
Activate DAOS on server
Goto Serverdocument – DAOS
Set Minimum size of an object
• recommended 64kB (= DEFAULT in 8.5.1 )
• Evaluate with DAOS Estimator
Set DAOS base path
• Put is outside the data path!
Set Delete objects in repository
• Depends on your backup strategy
• 0 Days = immediate delete after the refcount = 0
• Manually from console: TELL DAOSMGR PRUNE 0
Restart the server
show server
www.eknori.de
Activate DAOS on applications
Application properties „propeller
hat“ for single application
Use Admin client for multiple
applications
Valid for new attachments only!
To move existing attachments into the
DAOS repository
– Load compact –C –DAOS ON <Path\NSFs>
www.eknori.de
Bug Alert !!!
load compact -c –DAOS ON mail (w/o trailing backslash)
enables DAOS on ALL applications, not only on those in mail
subdirectory !!
Documented Lotus Software Knowledge Base #1410926
SPR AHOE7XHM9J (fixed in 8.5.2)
www.eknori.de
The Result
Design
Text
Tickets
Design
Text
Attachments
DAOS Repository
www.eknori.de
Transaction Protocol Trap
load compact –c –DAOS ON
NO attachment in DAOS repository
tell daosmgr dbsummary count = 0
load compact -C -T -DAOS on
www.eknori.de
DAOS and replication
DAOS is a server feature!
On a client replica, all attachments are stored in the
application.
Replicas on a NON-DAOS server store attachments in the
application
Each server in a cluster has it„s own DAOS repository.
www.eknori.de
DAOS and replication (cont.)
Client Server (as of 8.5.1)
– „known” attachments are no longer transferred to server = „smart
replication“
• Forward mail
• Reply with history
Server Server (as of 8.5.1)
– “known” attachments are no longer replicated to target server
8 new statistics
– show stat DAOS
www.eknori.de
DAOS and replication (cont.)
Domino 8.5
Domino 8.5.1
WARNING!!: ADMINP_ACCELERATED_REPLICA_OVERRIDE=4 !!
www.eknori.de
DAOS and replication (last slide )
www.eknori.de
DAOS and mail.box
Remove RM_NO_LOG_OBJECTS_IN_MAILBOX from
notes.ini.
Enable DAOS on all mail.box files
Enable DAOS on all mail files
Increases performance
www.eknori.de
DAOS and Encryption
Increases number of NLO files!
– User A writes an encrypted mail with attachment to User B
– User B writes an encrypted mail with same attachment to User A
– Results in 4 NLO Files in repository
Does not save space but separates static from dynamic data
www.eknori.de
DAOS and Quotas
Technote: 1405456
Depends on the quota enforcement setting that is in place
Check space used in file when adding a note
– Acts on the physical size of the application (without attachments)
– Possibly a problem when switching DAOS off!!
• 64GB limit
Check file size when extending the file
Check file size when adding a note
– Both acts on the logical size of the application (with attachments)
www.eknori.de
Keep daoscat.nsf in sync
DON„T use DDM !!
Use program document
Tell daosmgr resync
SPR# DROO7ZNPZ8
Added a new DAOSMGR resync switch
to perform a quick resync.
This new switch updates the DIT and
DOI but does not run through the
applications or updates the NLO
refcounts. ( as of 8.5.1 IF1 )
www.eknori.de
„Needs Resync“
DDM points to the first application that causes the DAOS
catalog to go into „Needs Resync“ state.
If you cannot find a cause for the issue set
debug_threadid=1
console_log_enabled=1
daos_logging=<path/daos.txt>STATE_CHANGE
Many improvements in 8.5.2 that reduces the number of state
changes (SPR# RCOR83NLQK)
www.eknori.de
DAOS resync time window feature
The following two notes.ini parameters control when resync
will be allowed to run
– DAOS_RESYNC_START_TIME=12:00:00 AM
– DAOS_RESYNC_STOP_TIME=04:00:00 AM
Note that these parameters do not launch resync
automatically; they just control whether or not resync may run
at a given time, and also tell resync when to stop.
A program document could be created to launch resync just
after midnight. If the catalog is SYNCHRONIZED, then the
resync will not take place. However, if a resync is needed,
then the resync process will begin. When the end time is
reached, the resync threads will finish up the databases they
are processing and exit.
www.eknori.de
DAOS OFF
What happens when DAOS is disabled / switched OFF?
In application properties
– No documents with attachments in the application
– Application has documents with attachments
• Status in Admin Client changes to „Read Only“
• DAOS still works for already stored attachments but no new attachments are
written to the repository for the specific application.
In server document
– Same as 1
www.eknori.de
DAOS and missing NLO
NLO has been deleted on the OS level
TELL DAOSMGR LISTNLO MISSING <application.nsf>
Output to <datadir> by default
www.eknori.de
Daosmgr in Domino 8.5.2
daosmgr options have been enhanced in Domino 8.5.2
You can now do a search for missing NLO on all applications
in a subdirectory
– tell daosmgr listnlo missing mail
www.eknori.de
DAOS and missing application
NSF has been deleted on the OS level
Message on server startup
• Informational - The DAOS catalog is not synchronized. Deletions will be
postponed. Please run 'tell daosmgr resync' at the next convenient
opportunity to re-synchronize.
Has the application been deleted by accident?
– NO
• TELL DAOSMGR RESYNC oder RESYNC FORCE
– YES
• Restore from backup
• Create a new replica from a cluster mate
Same behaviour when moving an application (on OS level)
www.eknori.de
Can we save another 50%
If we disable encryption, then we could put NLO for all
servers into a single repository …
Yes, in theory
It„s only experimental
Does not work because
each server has its own
daoscat.nsf.
Would cause problems
with pruning
www.eknori.de
DAOS and Nagios ( Statistics )
Catalog.Status
Repository.Size
Plugin
www.eknori.de
DAOS amd Statistics
8 new statistics (8.5.1)
– show stat DAOS
8.5.1
Debug_NSF_Show_Allstats=1
www.eknori.de
DAOS and Statistics
Windows:@echo off
setlocal
for /f "tokens=*" %%a in ('dir/w/s %1 ^|findstr "Datei(en)"') do (
set info=%%a
)
echo %info%
Linux:#!/bin/sh
REPOSITORY=/local/daos
OUTFILE=/local/daos.txt
fCount=`find $REPOSITORY -type f | wc –l`
fSize=`du -ksb $REPOSITORY`
IFS=”/”
array=($fSize)
fDate=$(date +”%d-%m-%Y”)
echo $fDate / $fCount / ${array[0]}>> $OUTFILE
Use XML
www.eknori.de
DAOS and daos.cfg
Contains information on file count and file size of .NLO files
Created on server startup when it has been deleted
Updated on server shutdown
Do not backup or restore!!
www.eknori.de
DAOS and daoscat.nsf
Contains index and refcount of NLO
Contains index of all applications using DAOS
Does NOT contain any view
Re-Created on re-sync
Created on server startup when it has been deleted
Do not backup or restore!!
www.eknori.de
Admin Client ( Files tab )
New columns regarding DAOS
– Logical size
– Physical size
– DAOS status
– DAOS count
– DAOS size
New columns regarding compression
www.eknori.de
Backup & Restore
No change in backing up NSF files.
Additional backup job for NLO files
NLO files do not require any special agent for backup
IMPORTANT: the backup sequence
– server is up during backup
• NSF before NLO
– server is down during backup
• it‟s your choice
IBM Technote # 1358548 - DAOS Backup and Restore
www.eknori.de
Resources
DAOS Wiki
• http://www-
10.lotus.com/ldd/dominowiki.nsf/archive?openview&title=DAOS&type=cat&c
at=null&tag=DAOS
DAOS Backup and Restore
• http://www-10.lotus.com/ldd/dominowiki.nsf/dx/daosbackup-and-restore
DAOS Estimator
• http://www.ibm.com/support/docview.wss?rs=463&uid=swg24021920
DAOS Technical Overview
• http://www.nelotus.org/A55CBA/nelotus.nsf/87fde291d7608ada852564c9006
eeba5/b6523e7f4b15585b862575b5004d3488/$FILE/DAOSTechnicalOvervi
ew_NELotus.pdf
