how to build a scalable storage system with oss at tlug meeting 2008/09/13
DESCRIPTION
TRANSCRIPT
![Page 1: How To Build A Scalable Storage System with OSS at TLUG Meeting 2008/09/13](https://reader033.vdocument.in/reader033/viewer/2022051322/5464659ab4af9fda3f8b456e/html5/thumbnails/1.jpg)
TLUG Meeting 2008/09/13
Gosuke Miyashita
![Page 2: How To Build A Scalable Storage System with OSS at TLUG Meeting 2008/09/13](https://reader033.vdocument.in/reader033/viewer/2022051322/5464659ab4af9fda3f8b456e/html5/thumbnails/2.jpg)
My company
paperboy&co.Web hosting, blog, ec hosting and so on for
indivisualsAbout 1,000 Linux serversMany single servers ...
![Page 3: How To Build A Scalable Storage System with OSS at TLUG Meeting 2008/09/13](https://reader033.vdocument.in/reader033/viewer/2022051322/5464659ab4af9fda3f8b456e/html5/thumbnails/3.jpg)
My goal of a scalable storage system Storage system for a web hosting service
High resource availabilityFlexible I/O distributionEasy to extend Mountable by multiple hostsNo SPoFWith OSSWithout expensive hardwares
Now I’m trying technologies for these purposes
![Page 4: How To Build A Scalable Storage System with OSS at TLUG Meeting 2008/09/13](https://reader033.vdocument.in/reader033/viewer/2022051322/5464659ab4af9fda3f8b456e/html5/thumbnails/4.jpg)
![Page 5: How To Build A Scalable Storage System with OSS at TLUG Meeting 2008/09/13](https://reader033.vdocument.in/reader033/viewer/2022051322/5464659ab4af9fda3f8b456e/html5/thumbnails/5.jpg)
cman CLVM GFS2 GNBD DRBD DM-MP
Technologies
![Page 6: How To Build A Scalable Storage System with OSS at TLUG Meeting 2008/09/13](https://reader033.vdocument.in/reader033/viewer/2022051322/5464659ab4af9fda3f8b456e/html5/thumbnails/6.jpg)
cman
Cluster Manager A component of Red Hat Cluster Suit Membership management Messaging among cluster nodes Needed for CLVM and GFS2
![Page 7: How To Build A Scalable Storage System with OSS at TLUG Meeting 2008/09/13](https://reader033.vdocument.in/reader033/viewer/2022051322/5464659ab4af9fda3f8b456e/html5/thumbnails/7.jpg)
CLVM
Cluster Logical Volume Manager Cluster-wide version of LVM2 Automatically share LVM2 metadata
among all cluster nodes So logical volumes with CLVM available
to all cluster nodes
![Page 8: How To Build A Scalable Storage System with OSS at TLUG Meeting 2008/09/13](https://reader033.vdocument.in/reader033/viewer/2022051322/5464659ab4af9fda3f8b456e/html5/thumbnails/8.jpg)
CLVM
Logical volumeon shared storage
LVM2Metadata
clvmd
LVM2Metadata
clvmd
LVM2Metadata
clvmd
clvmd distributes metadata among cluster nodes
Logical volumes presented to each cluster node
![Page 9: How To Build A Scalable Storage System with OSS at TLUG Meeting 2008/09/13](https://reader033.vdocument.in/reader033/viewer/2022051322/5464659ab4af9fda3f8b456e/html5/thumbnails/9.jpg)
GNBD
Global Network Block Device Provides block-device access over
TCP/IP Similar to iSCSI Advantage over iSCSI is built-in fencing
![Page 10: How To Build A Scalable Storage System with OSS at TLUG Meeting 2008/09/13](https://reader033.vdocument.in/reader033/viewer/2022051322/5464659ab4af9fda3f8b456e/html5/thumbnails/10.jpg)
GNBD
GNBD client
GNBD client
GNBD client
GNBD Server
Exported block device
TCP/IP network
![Page 11: How To Build A Scalable Storage System with OSS at TLUG Meeting 2008/09/13](https://reader033.vdocument.in/reader033/viewer/2022051322/5464659ab4af9fda3f8b456e/html5/thumbnails/11.jpg)
GFS2
Global File System 2 One of cluster-aware file systems Multiple nodes can simultaneously
access this filesystem Uses DLM(Distributed Lock Manager) of
cman to maintain file system integrity OCFS is another cluster-aware file
system
![Page 12: How To Build A Scalable Storage System with OSS at TLUG Meeting 2008/09/13](https://reader033.vdocument.in/reader033/viewer/2022051322/5464659ab4af9fda3f8b456e/html5/thumbnails/12.jpg)
GFS2
GNBD Server
GFS2
GNBD client
cman
GNBD client
cman
GNBD client
cman
These nodes can access to the GFS2 file system simultaneously
![Page 13: How To Build A Scalable Storage System with OSS at TLUG Meeting 2008/09/13](https://reader033.vdocument.in/reader033/viewer/2022051322/5464659ab4af9fda3f8b456e/html5/thumbnails/13.jpg)
DRBD
Distributed Replicated Block Device RAID1 over a network Mirrors a whole block device over
TCP/IP Available Active/Active with cluster file
systems
![Page 14: How To Build A Scalable Storage System with OSS at TLUG Meeting 2008/09/13](https://reader033.vdocument.in/reader033/viewer/2022051322/5464659ab4af9fda3f8b456e/html5/thumbnails/14.jpg)
DRBD
Server
Block Device
Server
Block Device
Replication
![Page 15: How To Build A Scalable Storage System with OSS at TLUG Meeting 2008/09/13](https://reader033.vdocument.in/reader033/viewer/2022051322/5464659ab4af9fda3f8b456e/html5/thumbnails/15.jpg)
DM-MP
Device-Mapper Multipath Bundles I/O paths to one virtual I/O path Can choose active/passive or
active/active
![Page 16: How To Build A Scalable Storage System with OSS at TLUG Meeting 2008/09/13](https://reader033.vdocument.in/reader033/viewer/2022051322/5464659ab4af9fda3f8b456e/html5/thumbnails/16.jpg)
DM-MP with SAN storage
Node
HBA1 HBA2
SAN swtich 1 SAN swtich 2
Storage
CNTRLR1 CNTRLR2
/dev/sda1 /dev/sdb1
Seen as one device
/dev/mapper/mpath0
active/passiveor
active/active
![Page 17: How To Build A Scalable Storage System with OSS at TLUG Meeting 2008/09/13](https://reader033.vdocument.in/reader033/viewer/2022051322/5464659ab4af9fda3f8b456e/html5/thumbnails/17.jpg)
![Page 18: How To Build A Scalable Storage System with OSS at TLUG Meeting 2008/09/13](https://reader033.vdocument.in/reader033/viewer/2022051322/5464659ab4af9fda3f8b456e/html5/thumbnails/18.jpg)
A scalable storage system
cmanGNBD
cmanGNBD
/dev/VG0/LV0 (CLVM)
GNBD
Server
GFS2
GNBD
Server
GFS2
Replication(DRBD)
/dev/mapper/mpath0(DM-MP)/dev/gnbd0 /dev/gnbd1
mount /dev/VG0/LV0 /mnt
GNBD
Server
GFS2
GNBD
Server
GFS2
Replication(DRBD)
/dev/mapper/mpath1(DM-MP)/dev/gnbd2 /dev/gnbd3
![Page 19: How To Build A Scalable Storage System with OSS at TLUG Meeting 2008/09/13](https://reader033.vdocument.in/reader033/viewer/2022051322/5464659ab4af9fda3f8b456e/html5/thumbnails/19.jpg)
How to extend
cmanGNBD
cmanGNBD
/dev/VG0/LV0 (CLVM)
mount /dev/VG0/LV0 /mnt
GNBD
Server
GFS2
GNBD
Server
GFS2
/dev/mapper/mpath0
/dev/gnbd0 /dev/gnbd1GNB
DServe
rGFS
2
GNBD
Server
GFS2
/dev/mapper/mpath1
/dev/gnbd2 /dev/gnbd3GNB
DServe
rGFS
2
GNBD
Server
GFS2
/dev/mapper/mpath2
/dev/gnbd4 /dev/gnbd5
![Page 20: How To Build A Scalable Storage System with OSS at TLUG Meeting 2008/09/13](https://reader033.vdocument.in/reader033/viewer/2022051322/5464659ab4af9fda3f8b456e/html5/thumbnails/20.jpg)
![Page 21: How To Build A Scalable Storage System with OSS at TLUG Meeting 2008/09/13](https://reader033.vdocument.in/reader033/viewer/2022051322/5464659ab4af9fda3f8b456e/html5/thumbnails/21.jpg)
I wonder ...
Many components cause troubles? How about overhead and performance? How about stability? More better way? How about other than Red Hat Linux?
![Page 22: How To Build A Scalable Storage System with OSS at TLUG Meeting 2008/09/13](https://reader033.vdocument.in/reader033/viewer/2022051322/5464659ab4af9fda3f8b456e/html5/thumbnails/22.jpg)