Download - Practical Tips for Novell Cluster Services
Novell Corporate Presentation Template 2009 - blue
Practical Tips for
Novell Cluster Services
Mark RobinsonCTO Linux, [email protected]
Tim HeywoodCTO, [email protected]
Agenda
Introduction
Cluster Services in OES2
Our Environment
AutoYaST
Cluster Build Methodology
Creating Resources
Cluster Management
Troubleshooting
Introduction
Introduction
Mark RobinsonLinux Geek
Working with SUSE since 1998
Working with OES since OES1 Beta 5
CLP, CLE, NCE ES, CNI, etc.
Ex-SysOp
Tim HeywoodWorking with Novell since ????
Working with OES since OES1 Beta 5
CNE, MCNE, CNI(ish)
Novell Knowledge Partner (SysOp)
Introduction
NDS8 Network Design and Support Ltd.Platinum Consulting Partner
Based in Edinburgh, work worldwide
Specialities:Linux
Workgroup
SRM
Cluster Services in OES2
Cluster Services in OES2
New features are Linux only
New from OES2 FCS on:Resource monitoring
XEN virtualization support
x86_64 platform supportIncluding mixed 32/64 bit node support
Dynamic Storage Technology
What's new in SP1/2?
Major rewrite of cluster code for SP2Removed NetWare translation layer
Much faster
Much lower system load
Typical load average of 0.2!
New/improved clustering for:iFolder 3
AFP
NCP virtual server for POSIX filesystem resources
Types of Clusters
Traditional clusterServers (nodes)
ResourcesNSS
GroupWise
iPrint
XEN clusterDom0 hosts (nodes)
XEN guests (DomU) resources
Each resource is a server in its own right
Live migration with para-virtualised DomU
XEN Cluster Architecture
AutoYaSTOCFS2 LUNDomU Files
Cluster Node
Xen Dom0
Cluster Node
Xen Dom0
Cluster Node
Xen Dom0
Resource
DomU
Linux
iPrint
Resource
DomU
Linux
iPrint
Resource
DomU
Linux
iFolder
Resource
DomU
Linux
GroupWise
Resource
DomU
NetWare
pCounter
Live Migrate
Live Migrate
Our Environment
Our Environment
VMware Workstation based
VMware shared disk as an alternative to iSCSI
Virtual MachinesResource Server
Node 1 (built, in the cluster)
Node 2 (to be joined to cluster)
Node 3 (to be built)
SUSE Linux Enterprise Server 10 SP3/OES2 SP2 combined iso
Explain about building combined iso
VMware Setup
Create disks standalone
Add config to node vmx files
disk.locking = "false"diskLib.dataCacheMaxSize = "0"
scsi1.present = "TRUE"scsi1.sharedBus = "none"scsi1.virtualDev = "lsilogic"scsi1.pciSlotNumber = "35"
scsi1:0.present = "TRUE"scsi1:0.fileName = "cluster-lun0.vmdk"scsi1:0.mode = "independent-persistent"scsi1:0.redo = ""
scsi1:1.present = "TRUE"scsi1:1.fileName = "cluster-lun1.vmdk"scsi1:1.mode = "independent-persistent"scsi1:1.redo = ""
vmware-vdiskmanager -c -s 100mb -a lsilogic -t 2 cluster-lun0.vmdkvmware-vdiskmanager -c -s 1gb -a lsilogic -t 2 cluster-lun1.vmdk
Discuss flags for vmware-vdiskmanager especially -t
Disk types:0 : single growable virtual disk1 : growable virtual disk split in 2GB files2 : preallocated virtual disk3 : preallocated virtual disk split in 2GB files4 : preallocated ESX-type virtual disk5 : compressed disk optimized for streaming
Discuss path to SAN virtual disks
Mention different VMware versions
Our Environment
Resource Server
(Tree master, iManager,Installation services,AutoYaST, SMT)
Storage
VMware Shared Disks
OES2 SP2 Nodes
Our Environment
AutoYaST Third Node
Resource Server
(Tree master, iManager,Installation services,AutoYaST, SMT)
Storage
VMware Shared Disks
OES2 SP2 Nodes
Tell audience that the autoyast build may take a while we'll concentrate on existing two nodes for most of the demo.
AutoYaST
Why AutoYaST?
Repeatable (exactly)No human element
XML forms part of Documentation
Drink coffee (or suitable non-caffeinated beverage) while server builds itself!
Multiple simultaneous buildsStagger by at least 15 minutes
Easy to expand cluster with new nodes
Helps with DR
Why AutoYaST?
What will AutoYaST do?Disk partitioning
Software patterns
Network configuration (including VLAN, bonding etc)
OES serviceseDirectory new or existing tree
NSS
NCS
Security lockdown
Scripts/Complete config files to do the rest
At the end of the install we will migrate an NSS resource to the new node with no additional configuration!
AutoYaST New Cluster
cn=admin.o=novell novell cn=cluster,ou=resources,o=novell 10.0.0.100 New Cluster node IP,LDAP server IP 636 nodename Later sdx sdy
Offer copy of autoyast profiles used in demos.
AutoYaST Existing Cluster
cn=admin.o=novell novell cn=cluster,ou=resources,o=novell Existing Cluster node IP,LDAP server IP 636 nodename Later
Demo
AutoYaST
Cluster Build Methodology
Cluster Build Methodology
Start with a Resource ServeriManager
Network Installation Server HTTP or NFS
AutoYaST repository (can be password protected on HTTP)
SMT for patching
Magic PiXiEs server
Cluster Build Methodology
Create a template nodeNodeZ
Use it toCreate AutoYaST template using Clone this system...
Test the shared disk
Create the cluster
Create the SBD
Creating resources
This build will NOT be part of final production cluster
Cluster Build Methodology
Copy this XML for additional nodesModifyserver name
IP address(es)
Use diffuse to compare XML files
Build the other nodes Use the XML created above
If not 100% right, whack it, modify XML and start build again
Remember these are now commodity items
Whack NodeZ and rebuild to complete the system
Cluster Build Methodology
Implement NIC bondingNIC driver independent
7 different methods some require switch support
Link state vs arp monitoring blades often cannot lose local link!
Configurable with AutoYaST
Implement Multipath (MPIO)Very simple to configure mainly autodetect
Wide range of SAN support
Friendly LUN naming
Configuration file can be used (put in place with autoYaST)
Creating Resources
File Sharing Resources
An NSS poolUse iManager
Will end up as Primary for DST pair
Another NSS poolUse NSSMU (just because we can)
Will end up as Shadow for DST pair
Combine them into one resourceDelete resource for shadow
Modify load script for primary
Talk about creating the two pools and why.
DEMO: Create pool1/vol1Create pool1_shd/vol1_shd
File Sharing Resources
POSIX filesystem based resource with NCPEasier than Samba to access files
Can be used for iPrint, DHCP etc
Use evmsgui to create and format the volume
Create the resource in iManager
Script to create NCP virtual server
Talk about creating the two pools and why.
DEMO: Create pool1/vol1Create pool1_shd/vol1_shd
File Sharing Resources
Add resource monitoring
Add NFS accessLUM enablement of target users
NSS/POSIX rights
exportfs in load script rather then /etc/exports on nodes
Use fsid=x for NSS
Talk about creating the two pools and why.
DEMO: Create pool1/vol1Create pool1_shd/vol1_shd
NFS access
FPC1
FPC2
FPC3
FPC4
FPC5
SHARED1Virtual Server
SHARED1Volume
eDirectory
fpc.server.novell
NFSaccess
MisUID: 1010
OracleUID: 60003
Iface
UID: 1012
Mis-dweeb
UID: 1004
LUM
NSS Rights
MisUID: 1010
OracleUID:
60003
Iface
UID: 1012
Dweeb
MisUID: 1010
OracleUID:
60003
Iface
UID: 1012
Gromit
MisUID: 1010
OracleUID:
60003
Iface
UID: 1012
Wallace
NFS
Talk about creating the two pools and why.
DEMO: Create pool1/vol1Create pool1_shd/vol1_shd
iPrint
Create iPrint on NSS
Run iprint_nss_relocate on each node with volume in place
NB: only one iPrint resource may run on a node
Need to accept certificates in iManager for each node
Talk about creating the two pools and why.
DEMO: Create pool1/vol1Create pool1_shd/vol1_shd
iFolder
Create iFolder on POSIX/mnt/cluster/ifolder
Run /opt/novell/ifolder3/bin/ifolder_cluster_setup on each nodeCopy /etc/sysconfig/novell/ifldr3_2_sp2 to nodes first
NB: Only one iFolder resource may run on a node
Talk about creating the two pools and why.
DEMO: Create pool1/vol1Create pool1_shd/vol1_shd
DNS
DNS must be on NSS as NCP server required for eDirectory integration
Check NCP:NCPServer objects
LUM user required for NSS rights
Talk about creating the two pools and why.
DEMO: Create pool1/vol1Create pool1_shd/vol1_shd
DHCP
Create DHCP on NSS
Leases file on NSS volume
Log file on NSS volumeSyslog-ng configuration
Logrotate configuration
Default AppArmor configuration will not allow logging to here!
Talk about creating the two pools and why.
DEMO: Create pool1/vol1Create pool1_shd/vol1_shd
GroupWise
Create PO on NSS
Set namespace in load script/opt=ns=long
Disable atime/diratime on volumeOpen nsscon
Run /noatime=volname
Talk about creating the two pools and why.
DEMO: Create pool1/vol1Create pool1_shd/vol1_shd
OCFS2 Shared Storage
Shared disk! Multi-mount, read/write with distributed lock management
/etc/ocfs2/cluster.conf automagically created by NCS
Fstab mounting uses /etc/init.d/ocfs2 service
Talk about creating the two pools and why.
DEMO: Create pool1/vol1Create pool1_shd/vol1_shd
Cluster Management
Cluster Management
iManagerThe ONLY way to create/delete/edit resources
View event log
Cluster commandSame as NetWare
No cvb rebuild or device scan. These are not required on Linux as EVMS does it for you
Console One
Talk about creating the two pools and why.
DEMO: Create pool1/vol1Create pool1_shd/vol1_shd
The cluster Command
The usual suspectscluster online/offline/migrate
cluster join/leave
cluster status/resources/view/info
More interestingcluster stats display check heartbeat/SBD ticks
cluster pools check NSS pools and location
cluster set modify heartbeat etc
cluster exec potentially very dangerous
Lots of BCC commands
Talk about creating the two pools and why.
DEMO: Create pool1/vol1Create pool1_shd/vol1_shd
Troubleshooting
Useful Linux Tools
ip command manage TCP/IP on Linuxip addess show/add/del
ip route show
ethtool NIC settings
cat /proc/net/bonding/bondX
netstat network communication statusCheck which ports services are listening on
Check IP based connections to node
nmap network/port scanner
multipath -ll
Script logs are rolling logs now show all operations on this node for a particular resource
LUN Identification
Which LUN is which?
lsscsi shows LUN ID numbers
ls -l /dev/disk/by-idscsi-360a980004334616f6b5a55572d625550
Need to find ID on SAN. (Netapp uses ASCII!)
Multipathing will show the ID as the multipath nameUse friendly naming
Script logs are rolling logs now show all operations on this node for a particular resource
Useful Tools
sbdutil create/check/modify the SBDsbdutil -f to find the SBD
sbdutil -v to view the current state of the SBD
/opt/novell/ncs/bin/ncs-configd.py-init option to pull down load scripts, fix node names etc
cifsPool.py to fix CIFS attributes (TID #7005192)
OES2 NCS Master Reference TID, FAQ and Troubleshooting TID #7001433
NSA Novell Support AdvisorMany patterns for NCS
Script logs are rolling logs now show all operations on this node for a particular resource
File Locations
Cluster configuration file/etc/opt/novell/ncs/clstrlib.conf
Load/Unload scripts/var/run/ncs (run from here)
/var/opt/novell/ncs
Load script output logs/var/opt/novell/log/ncs
System Log/var/log/messages
Script logs are rolling logs now show all operations on this node for a particular resource
File Locations
Admin filesystem virtual filesystem for NCS management/admin/Novell/Cluster
Proc filesystem virtual filesystem for Linux/NCS management/proc/ncs
Cluster event logiManager
/admin/Novell/Cluster/EventLog.xml
Script logs are rolling logs now show all operations on this node for a particular resource
/proc/ncs Magic
Enable serious debugging!echo -n "TRACE ON" > /proc/ncs/vll
echo -n "TRACE SBD ON" > /proc/ncs/vll
echo -n "TRACE GIPC ON" > /proc/ncs/vll
echo -n "TRACE MCAST ON" > /proc/ncs/vll
echo -n "TRACE CVB ON" > /proc/ncs/cluster
Can be made permanent by editing /opt/novell/ncs/bin/ldncs
Find the SBDcat /proc/ncs/sbdlib
Script logs are rolling logs now show all operations on this node for a particular resource
AdminFS Magic
Two type of file in /admin/Novell/Cluster*.xml contain cluster/state information
*.cmd write then read files for issuing cluster commands
Script logs are rolling logs now show all operations on this node for a particular resource
Known issues
EVMS issue with no NSSUnpatched SP2 nodes without NSS cannot load SBD kernel moduled
iFolder shutdown scriptDoesn't shut down components if names have been changed during configuration
IP address problemUnpatched nodes can allow duplicate IP addresses on network
Resources in NDS sync stateCheck replica rings/referrals
Check case of cluster DN in clstrlib.conf
Script logs are rolling logs now show all operations on this node for a particular resource
www.nds8.co.uk
Unpublished Work of Novell, Inc. All Rights Reserved.This work is an unpublished work and contains confidential, proprietary, and trade secret information of Novell, Inc. Access to this work is restricted to Novell employees who have a need to know to perform tasks within the scope of their assignments. No part of this work may be practiced, performed, copied, distributed, revised, modified, translated, abridged, condensed, expanded, collected, or adapted without the prior written consent of Novell, Inc. Any use or exploitation of this work without authorization could subject the perpetrator to criminal and civil liability.
General DisclaimerThis document is not to be construed as a
promise by any participating company to develop, deliver, or market
a product. It is not a commitment to deliver any material, code, or
functionality, and should not be relied upon in making purchasing
decisions. Novell, Inc. makes no representations or warranties with
respect to the contents
of this document, and specifically disclaims any express or implied
warranties of merchantability or fitness for any particular
purpose. The development, release, and timing of features or
functionality described for Novell products remains at the sole
discretion of Novell. Further, Novell, Inc. reserves the right to
revise this document and to make changes to its content, at any
time, without obligation to notify any person or entity of such
revisions or changes. All Novell marks referenced in this
presentation are trademarks or registered trademarks of Novell,
Inc. in the United States and other countries. All third-party
trademarks are the property of their respective owners.
Click to enter the title (44pt)
Second line or subtitle (22pt)
Presenter Name (16pt)Presenter Title (14pt)
Company/email (14pt)
Click to Edit Section Break Text (32pt)
Right Justified
piece in master that I can't get rid of
Click to edit the title text format
Novell, Inc. All rights reserved.
Click to edit the title text format
Click to edit the outline text formatSecond Outline LevelThird Outline LevelFourth Outline Level