linuxkongress2010.gfs2ocfs2.talk
DESCRIPTION
Some technical information on GFS2 and OCFS2 (autumn/fall 2010)TRANSCRIPT
![Page 1: Linuxkongress2010.gfs2ocfs2.talk](https://reader036.vdocument.in/reader036/viewer/2022070317/5562edc4d8b42a213b8b4de4/html5/thumbnails/1.jpg)
Divide and conquer – Shared disk cluster file systems shipped with the
Linux kernel
Udo Seidel
![Page 2: Linuxkongress2010.gfs2ocfs2.talk](https://reader036.vdocument.in/reader036/viewer/2022070317/5562edc4d8b42a213b8b4de4/html5/thumbnails/2.jpg)
Shared file systems
● Multiple server access same data● Different approaches
● Network based, e.g. NFS, CIFS● Clustered
– Shared disk, e.g. CXFS, CFS, GFS(2), OCFS2– Distributed parallel, e.g. Lustre, Ceph
![Page 3: Linuxkongress2010.gfs2ocfs2.talk](https://reader036.vdocument.in/reader036/viewer/2022070317/5562edc4d8b42a213b8b4de4/html5/thumbnails/3.jpg)
History
● GFS(2)● First version in the mid 90’s● Started on IRIX, later ported to Linux● Commercial background: Sistina and RedHat● Part of Vanilla Linux kernel since 2.6.19
● OCFS2● OCFS1 for database files only● First version in 2005● Part of Vanilla Linux kernel since 2.6.16
![Page 4: Linuxkongress2010.gfs2ocfs2.talk](https://reader036.vdocument.in/reader036/viewer/2022070317/5562edc4d8b42a213b8b4de4/html5/thumbnails/4.jpg)
Features/Challenges/More
● As much as possible similar to local file systems● Internal setup● management
● Cluster awareness● Data integrity● Allocation
![Page 5: Linuxkongress2010.gfs2ocfs2.talk](https://reader036.vdocument.in/reader036/viewer/2022070317/5562edc4d8b42a213b8b4de4/html5/thumbnails/5.jpg)
Framework
● Bridges the gap between one-node and cluster● 3 main components
● Cluster-ware● Locking● Fencing
![Page 6: Linuxkongress2010.gfs2ocfs2.talk](https://reader036.vdocument.in/reader036/viewer/2022070317/5562edc4d8b42a213b8b4de4/html5/thumbnails/6.jpg)
Framework GFS2 (I)
● Cluster-ware of general purpose● More flexible● More options/functions● More complexity● Configuration files in XML
● Locking uses cluster framework too● systemconfigcluster OR Conga OR vi
& scp
![Page 7: Linuxkongress2010.gfs2ocfs2.talk](https://reader036.vdocument.in/reader036/viewer/2022070317/5562edc4d8b42a213b8b4de4/html5/thumbnails/7.jpg)
Framework GFS2 (II)
# cat /etc/cluster/cluster.conf
<?xml version="1.0" ?>
<cluster config_version="3" name="gfs2">
<fence_daemon post_fail_delay="0" post_join_delay="3"/>
<clusternodes>
<clusternode name="node0" nodeid="1" votes="1">
<fence/>
</clusternode>
<clusternode name="node1" nodeid="2" votes="1">
...
</cluster>
#
![Page 8: Linuxkongress2010.gfs2ocfs2.talk](https://reader036.vdocument.in/reader036/viewer/2022070317/5562edc4d8b42a213b8b4de4/html5/thumbnails/8.jpg)
Framework OCFS2 (I)
● Cluster-ware just for OCFS2● Less flexible● Less options/functions● Less complexity● Configuration file in ASCII
● Locking uses cluster framework too● ocfs2console OR vi & scp
![Page 9: Linuxkongress2010.gfs2ocfs2.talk](https://reader036.vdocument.in/reader036/viewer/2022070317/5562edc4d8b42a213b8b4de4/html5/thumbnails/9.jpg)
Framework OCFS2 (II)
# cat /etc/ocfs2/cluster.conf
node:
ip_port = 7777
ip_address = 192.168.0.1
number = 0
name = node0
cluster = ocfs2
...
cluster:
node_count = 2
#
![Page 10: Linuxkongress2010.gfs2ocfs2.talk](https://reader036.vdocument.in/reader036/viewer/2022070317/5562edc4d8b42a213b8b4de4/html5/thumbnails/10.jpg)
Locking
● Distributed Lock Manager (DLM)● Based on VMS-DLM● Lock modes
● Exclusive Lock (EX)● Protected Read (PR)● No Lock (NL)● Concurrent Write Lock (CW) – GFS2 only● Concurrent Read Lock (CR) – GFS2 only● Protected Write (PR) – GFS2 only
![Page 11: Linuxkongress2010.gfs2ocfs2.talk](https://reader036.vdocument.in/reader036/viewer/2022070317/5562edc4d8b42a213b8b4de4/html5/thumbnails/11.jpg)
Locking - Compatibility
Existing Lock
Requested Lock NL CR CW PR PW EX
NL Yes Yes Yes Yes Yes Yes
CR Yes Yes Yes Yes Yes No
CW Yes Yes Yes No No No
PR Yes Yes No Yes No No
PW Yes Yes No No No No
EX Yes No No No No No
![Page 12: Linuxkongress2010.gfs2ocfs2.talk](https://reader036.vdocument.in/reader036/viewer/2022070317/5562edc4d8b42a213b8b4de4/html5/thumbnails/12.jpg)
Fencing
● Separation of host and storage● Power Fencing
– Power switch, e.g. APC– Server side, e.g. IPMI, iLO– Useful in other scenarios– Post-mortem more difficult
● I/O fencing– SAN switch, e.g. Brocade, Qlogic– Possible to investigate “unhealthy” server
![Page 13: Linuxkongress2010.gfs2ocfs2.talk](https://reader036.vdocument.in/reader036/viewer/2022070317/5562edc4d8b42a213b8b4de4/html5/thumbnails/13.jpg)
Fencing - GFS2
● Both fencing methods● Part of cluster configuration● Cascading possible
![Page 14: Linuxkongress2010.gfs2ocfs2.talk](https://reader036.vdocument.in/reader036/viewer/2022070317/5562edc4d8b42a213b8b4de4/html5/thumbnails/14.jpg)
Fencing - OCFS2
● Only power fencing● Only self fencing
![Page 15: Linuxkongress2010.gfs2ocfs2.talk](https://reader036.vdocument.in/reader036/viewer/2022070317/5562edc4d8b42a213b8b4de4/html5/thumbnails/15.jpg)
GFS2 – Internals (I)
● Superblock● Starts at block 128● Expected data + cluster information● Pointers to master and root directory
● Resource groups● Comparable to cylinder groups of traditional Unix
file system● Allocatable from different cluster nodes -> locking
granularity
![Page 16: Linuxkongress2010.gfs2ocfs2.talk](https://reader036.vdocument.in/reader036/viewer/2022070317/5562edc4d8b42a213b8b4de4/html5/thumbnails/16.jpg)
GFS2 – Internals (II)
● Master directory● Contains meta-data, e.g journal index, quota, ...● Not visible for ls and Co.
● File system unique and cluster node specific files
● Journaling file system● One journal per cluster node ● Each journal accessible by all nodes (recovery)
![Page 17: Linuxkongress2010.gfs2ocfs2.talk](https://reader036.vdocument.in/reader036/viewer/2022070317/5562edc4d8b42a213b8b4de4/html5/thumbnails/17.jpg)
GFS2 – Internals (III)
● Inode/Dinode● Usual information, e.g. owner, mode, time stamp● Pointers to blocks: either data or pointer● Only one level of indirection● “stuffing”
● Directory management via Extendible Hashing● Meta file statfs
● statfs()
● Tuning via sysfs
![Page 18: Linuxkongress2010.gfs2ocfs2.talk](https://reader036.vdocument.in/reader036/viewer/2022070317/5562edc4d8b42a213b8b4de4/html5/thumbnails/18.jpg)
GFS2 – Internals (IV)
● Meta files● jindex directory containing the journals
– journalX
● rindex Resource group index
● quota● per_node directory containing node specific files
![Page 19: Linuxkongress2010.gfs2ocfs2.talk](https://reader036.vdocument.in/reader036/viewer/2022070317/5562edc4d8b42a213b8b4de4/html5/thumbnails/19.jpg)
GFS2 – what else
● Extended attributes xattr● ACL’s● Local mode = one node access
![Page 20: Linuxkongress2010.gfs2ocfs2.talk](https://reader036.vdocument.in/reader036/viewer/2022070317/5562edc4d8b42a213b8b4de4/html5/thumbnails/20.jpg)
OCFS2 – Internals (I)
● Superblock● Starts at block 3 (1+2 for OCFS1)● Expected data + cluster information● Pointers to master and root directory● Up to 6 backups
– at pre-defined offset– at 2^n Gbyte, n=0,2,4,6,8,10
● Cluster groups● Comparable to cylinder groups of traditional Unix
file system
![Page 21: Linuxkongress2010.gfs2ocfs2.talk](https://reader036.vdocument.in/reader036/viewer/2022070317/5562edc4d8b42a213b8b4de4/html5/thumbnails/21.jpg)
OCFS2 – Internals (II)
● Master or system directory● Contains meta-data, e.g journal index, quota, ...● Not visible for ls and Co.● File system unique and cluster node specific files
● Journaling file system● One journal per cluster node ● Each journal accessible by all nodes (recovery)
![Page 22: Linuxkongress2010.gfs2ocfs2.talk](https://reader036.vdocument.in/reader036/viewer/2022070317/5562edc4d8b42a213b8b4de4/html5/thumbnails/22.jpg)
OCFS2 – Internals (III)
● Inode● Usual information, e.g. owner, mode, time stamp● Pointers to blocks: either data or pointer● Only one level of indirection
● global_inode_alloc● Global meta data file ● inode_alloc node specific counterpart
● slot_map● Global meta data file● Active cluster nodes
![Page 23: Linuxkongress2010.gfs2ocfs2.talk](https://reader036.vdocument.in/reader036/viewer/2022070317/5562edc4d8b42a213b8b4de4/html5/thumbnails/23.jpg)
OCFS2 – Internals (IV)
● orphan_dir● Local meta data file● Cluster aware deletion of files in use
● truncate_log● Local meta data file● Deletion cache
![Page 24: Linuxkongress2010.gfs2ocfs2.talk](https://reader036.vdocument.in/reader036/viewer/2022070317/5562edc4d8b42a213b8b4de4/html5/thumbnails/24.jpg)
OCFS2 – what else
● Two versions: 1.2 and 1.4● Mount compatible● Framework not network compatible● New features disabled per default
● For 1.4:● Extended attributes xattr● Inode based snapshotting● preallocation
![Page 25: Linuxkongress2010.gfs2ocfs2.talk](https://reader036.vdocument.in/reader036/viewer/2022070317/5562edc4d8b42a213b8b4de4/html5/thumbnails/25.jpg)
File system management
● Known/expected tools + cluster details● mkfs● mount/umount● fsck
● File system specific tools● gfs2_XXXX● tunefs.ocfs2, debugfs.ocfs2
![Page 26: Linuxkongress2010.gfs2ocfs2.talk](https://reader036.vdocument.in/reader036/viewer/2022070317/5562edc4d8b42a213b8b4de4/html5/thumbnails/26.jpg)
GFS2 management (I)
● File system creation needs additional information● Cluster name● Unique file system identifier (string)● Optional:
– Locking mode to be used– number of journals
● Tuning by changing default size for journals, resource groups, ...
![Page 27: Linuxkongress2010.gfs2ocfs2.talk](https://reader036.vdocument.in/reader036/viewer/2022070317/5562edc4d8b42a213b8b4de4/html5/thumbnails/27.jpg)
GFS2 management (II)
● Mount/umount● No real syntax surprise● First node checks all journals ● Enabling ACL, quota, single node mode
![Page 28: Linuxkongress2010.gfs2ocfs2.talk](https://reader036.vdocument.in/reader036/viewer/2022070317/5562edc4d8b42a213b8b4de4/html5/thumbnails/28.jpg)
GFS2 management (III)
● File system check● Journal recovery of node X by node Y● Done by one node● file system offline anywhere else● Known phases
– Journals– Meta data– References: data blocks, inodes
![Page 29: Linuxkongress2010.gfs2ocfs2.talk](https://reader036.vdocument.in/reader036/viewer/2022070317/5562edc4d8b42a213b8b4de4/html5/thumbnails/29.jpg)
GFS2 tuning (I)
● gfs2_tool● Most powerful
– Display superblock– Change superblock settings (locking mode, cluster name)– List meta data– freeze/unfreeze file system– Special attributes, e.g. appendonly, noatime
● Requires file system online (mostly)
![Page 30: Linuxkongress2010.gfs2ocfs2.talk](https://reader036.vdocument.in/reader036/viewer/2022070317/5562edc4d8b42a213b8b4de4/html5/thumbnails/30.jpg)
GFS2 tuning (II)
● gfs2_edit● Logical extension of gfs2_tool● More details, e.g. node-specific meta data, block
level
● gfs2_jadd● Different sizes possible● No deletion possible● Can cause data space shortage
![Page 31: Linuxkongress2010.gfs2ocfs2.talk](https://reader036.vdocument.in/reader036/viewer/2022070317/5562edc4d8b42a213b8b4de4/html5/thumbnails/31.jpg)
GFS2 tuning (III)
● gfs2_grow● Needs space in meta directory● Online only● No shrinking
![Page 32: Linuxkongress2010.gfs2ocfs2.talk](https://reader036.vdocument.in/reader036/viewer/2022070317/5562edc4d8b42a213b8b4de4/html5/thumbnails/32.jpg)
OCFS2 management (I)
● File system creation● no additional information needed● Tuning by optional parameters
● Mount/umount● No real syntax surprise● First node checks all journals ● Enabling ACL, quota, single node mode
![Page 33: Linuxkongress2010.gfs2ocfs2.talk](https://reader036.vdocument.in/reader036/viewer/2022070317/5562edc4d8b42a213b8b4de4/html5/thumbnails/33.jpg)
OCFS2 management (II)
● File system check● Journal recovery of node X by node Y● Done by one node● file system offline anywhere else● Fixed offset of superblock backup handy● Known phases
– Journals– Meta data– References: data blocks, inodes
![Page 34: Linuxkongress2010.gfs2ocfs2.talk](https://reader036.vdocument.in/reader036/viewer/2022070317/5562edc4d8b42a213b8b4de4/html5/thumbnails/34.jpg)
OCFS2 tuning (I)
● tunefs.ocfs2● Display/change file system label● Display/change number of journals● Change journal setup, e.g.size● Grow file system (no shrinking)● Create backup of superblock● Display/enable/disable specific file system features
– Sparse files– “stuffed” inodes
![Page 35: Linuxkongress2010.gfs2ocfs2.talk](https://reader036.vdocument.in/reader036/viewer/2022070317/5562edc4d8b42a213b8b4de4/html5/thumbnails/35.jpg)
OCFS2 tuning (II)
● debugfs.ocfs2● Display file system settings, e.g. superblock● Display inode information● Access meta data files
![Page 36: Linuxkongress2010.gfs2ocfs2.talk](https://reader036.vdocument.in/reader036/viewer/2022070317/5562edc4d8b42a213b8b4de4/html5/thumbnails/36.jpg)
Volume manager
● Necessary to handle more than one LUN/partition
● Cluster-aware● Bridge feature gap, e.g. volume based
snapshotting● CLVM ● EVMS – OCFS2 only
![Page 37: Linuxkongress2010.gfs2ocfs2.talk](https://reader036.vdocument.in/reader036/viewer/2022070317/5562edc4d8b42a213b8b4de4/html5/thumbnails/37.jpg)
Key data - comparisonGFS2 OCFS2
Maximum # of cluster nodes Supported 16 (theoretical: 256) 256
journaling Yes Yes
Cluster-less/local mode Yes Yes
Maximum file system size 25 TB (theoretical: 8 EB) 16 TB (theoretical: 4 EB)
Maximum file size 25 TB (theoretical: 8 EB) 16 TB (theoretical: 4 EB)
POSIX ACL Yes Yes
Grow-able Yes/online only Yes/online and offline
Shrinkable No No
Quota Yes Yes
O_DIRECT On file level Yes
Extended attributes Yes Yes
Maximum file name length 255 255
File system snapshots No No
![Page 38: Linuxkongress2010.gfs2ocfs2.talk](https://reader036.vdocument.in/reader036/viewer/2022070317/5562edc4d8b42a213b8b4de4/html5/thumbnails/38.jpg)
Summary
● GFS2 longer history than OCFS2● OCFS2 setup simpler and easier to maintain● GFS2 setup more flexible and powerful● OCFS2 getting close to GFS2● Dependence on choice of Linux vendor
![Page 39: Linuxkongress2010.gfs2ocfs2.talk](https://reader036.vdocument.in/reader036/viewer/2022070317/5562edc4d8b42a213b8b4de4/html5/thumbnails/39.jpg)
References
http://sourceware.org/cluster/gfs/
http://www.redhat.com/gfs/
http://oss.oracle.com/projects/ocfs2/
http://sources.redhat.com/cluster/wiki/
http://sourceware.org/lvm2/
http://evms.sourceforge.net/
![Page 40: Linuxkongress2010.gfs2ocfs2.talk](https://reader036.vdocument.in/reader036/viewer/2022070317/5562edc4d8b42a213b8b4de4/html5/thumbnails/40.jpg)
Thank you!