gluster cloud night in tokyo 2013 -- tips for getting started

Agenda

!  About me !  Tips 1: Installation !  Tips 2: NFS Workloads !  Tips 3: Logging !  Conclusion

About Me

About me

!  Work at as a ... !  Programmer

!  (Most recently)

! Also Interested in

!  Software Engineer ! Cloud Computing ! Big Data ! Something new technologies ! Supporting GlusterFS/Red Hat Storage Introduction

with Red Hat K.K.

About me

! GlusterFS since 2007 (v1.3.7) !  for my internet crawler at first.

!  Love Gluster because of the ... !  Potential !  Performance !  Code !  Community

!  Introduced or introducing it into ... !  Printer and scanner solution (field trial) !  Email services !  File storage services (WebDAV, NFS) !  Backup services !  Shared storage platform

!  A board member of the Gluster Community

Getting started today!

See http://www.gluster.org/community/documentation/index.php/QuickStart

Tips 1 Installation

Tips 1: Installation

!   Question 1: Have you installed GlusterFS?

!   Question 1: Have you installed GlusterFS? !   Question 2: Did you find it easy?

!   Question 1: Have you installed GlusterFS? !   Question 2: Did you find it easy? !   Question 3: Did you do it on each single node?

!   Question 1: Have you installed GlusterFS? !   Question 2: Did you find it easy? !   Question 3: Did you do it on each single node? !   Question 4: Don't you think you should take easier way?

!   Question 1: Have you installed GlusterFS? !   Question 2: Did you find it easy? !   Question 3: Did you do it on each single node? !   Question 4: Don't you think you should take easier way? !   Question 5: Do you use deployment automation tools?

Development team says that hand-to-hand construction of systems is inefficient so they want to show an installation way with a deployment tool in the documents for installation.

Operational team says that they never want to learn the usage of deployment tools and never trust them so they don't accept such a document and they will install the system by hand work.

https://twitter.com/keithseahus/status/398453753352691714

My offer is that what development team should do is only copy the commands that the deployment tool write on your stdout and paste them into the documents for operational team.

Deployment Automation

peer 1 peer 2 3 4 N

A node for automated deployment

deployment automation

peer 1 peer 2 3 4 N

glusterfs

peer 1 peer 2 3 4 N

glusterfs

peer 1 peer 2 3 4 N

glusterfs

I use this mainly in my projects written in Ruby so am used to it.

Capistrano

$ gem install capistrano $ cap install mkdir -p config/deploy create config/deploy.rb create config/deploy/staging.rb create config/deploy/production.rb mkdir -p lib/capistrano/tasks Capified $ tree . . ├── Capfile ├── config │ ├── deploy │ │ ├── production.rb │ │ └── staging.rb │ └── deploy.rb └── lib └── capistrano └── tasks

capistrano-glusterfs

$ tree capistrano-glusterfs capistrano-glusterfs ├── Capfile ├── Gemfile ├── Gemfile.lock ├── LICENSE ├── README.md ├── config │ ├── deploy │ │ ├── production.rb │ │ ├── production.rb.example │ │ ├── staging.rb │ │ └── staging.rb.example │ ├── deploy.rb │ └── deploy.rb.example └── lib └── capistrano └── tasks ├── gluster.cap ├── glusterd.cap └── glusterfs.cap

$ tree capistrano-glusterfs capistrano-glusterfs ├── Capfile ├── Gemfile ├── Gemfile.lock ├── LICENSE ├── README.md ├── config │ ├── deploy │ │ ├── production.rb ... settings for production environment │ │ ├── production.rb.example │ │ ├── staging.rb ... settings for staging environment │ │ └── staging.rb.example │ ├── deploy.rb ... global settings and main sequences │ └── deploy.rb.example └── lib └── capistrano └── tasks ├── gluster.cap ... tasks to execute gluster command remotely ├── glusterd.cap ... tasks to start/stop all remote glusterd └── glusterfs.cap ... tasks for deploying GlusterFS

$ cat capistrano-glusterfs/config/deploy/staging.rb set :stage, :staging set :branch, "master" server '192.168.200.100', roles: %w(glusterfs glusterd gluster), user: 'deploy' server '192.168.200.101', roles: %w(glusterfs glusterd), user: 'deploy' server '192.168.200.102', roles: %w(glusterfs glusterd), user: 'deploy' server '192.168.200.103', roles: %w(glusterfs glusterd), user: 'deploy' server '192.168.200.104', roles: %w(glusterfs glusterd), user: 'deploy' server '192.168.200.105', roles: %w(glusterfs glusterd), user: 'deploy' server '192.168.200.106', roles: %w(glusterfs glusterd), user: 'deploy' server '192.168.200.107', roles: %w(glusterfs glusterd), user: 'deploy' server '192.168.200.108', roles: %w(glusterfs glusterd), user: 'deploy' server '192.168.200.109', roles: %w(glusterfs glusterd), user: 'deploy'

$ bundle exec cap -vT | grep -i "glusterfs:" cap glusterfs:build # Starting installation cap glusterfs:compile # Compile GlusterFS cap glusterfs:configuration # GlusterFS related configurations cap glusterfs:dependancies # Install GlusterFS dependancies cap glusterfs:make_install # Install GlusterFS cap glusterfs:preparation # Prepare installing GlusterFS $ bundle exec cap -vT | grep -i "glusterd:" cap glusterd:restart # Restart glusterd cap glusterd:start # Start glusterd cap glusterd:status # Get the status of glusterd cap glusterd:stop # Stop glusterd $ bundle exec cap -vT | grep -i "gluster:" cap gluster:peer:help # Help command for peer cap gluster:peer:status # list status of peers cap gluster:pool:list # list all the nodes in the pool cap gluster:volume:help # display help for the volume command cap gluster:volume:info # list information of all volumes cap gluster:volume:list # list all volumes in cluster cap gluster:volume:status # display status of all volume(s)/brick

http://www.capistranorb.com/documentation/getting-started/flow/

prepare and install dependancies before starting

build GlusterFS from source and install it

after published

task :preparation do .... execute "sudo -i yum install -y wget git yum-priorities" .... end

task :dependancies do .... dependancies = %w[ openssh-clients make rpm-build bison flex automake libtool ncurses-devel readline-devel openssl-devel libxml2-devel libibverbs-devel libacl-devel libattr-devel python-devel python-setuptools lvm2-devel systemtap-sdt-devel libaio-devel xfsprogs ]

task :compile do on roles(:glusterfs), in: :sequence, wait: 5 do execute "cd #{DIR}; sh ./autogen.sh" execute "cd #{DIR}; sh ./configure \ --prefix=#{INSTALL_PATH} \ --enable-bd-xlator \ --enable-fusermount \ --enable-systemtap \ --enable-debug" execute "cd #{DIR}; make" end end

task :make_install do on roles(:glusterfs), in: :sequence, wait: 5 do execute "cd #{DIR}; sudo make install" symlink = "/usr/local/glusterfs" execute "sudo -i rm -f #{symlink}" if test("[ -L #{symlink} ]") execute "sudo -i ln -snf #{INSTALL_PATH} #{symlink}" end end

$ git clone git@github.com:keithseahus/capistrano-glusterfs.git $ cd capistrano-glusterfs $ bundle exec cap staging deploy

$ git clone git@github.com:keithseahus/capistrano-glusterfs.git $ cd capistrano-glusterfs $ bundle exec cap staging deploy DEBUG [e5bdd4a7] Running /usr/bin/env /bin/egrep '/bin' ~/.bashrc > /dev/null; exit $? on 192.168.200.101 DEBUG [e5bdd4a7] Command: /bin/egrep '/bin' ~/.bashrc > /dev/null; exit $? DEBUG [e5bdd4a7] Finished in 0.096 seconds with exit status 0 (successful). INFO [a54cf65d] Running /usr/bin/env sudo -i yum install -y wget git yum-priorities on 192.168.200.101 DEBUG [a54cf65d] Command: sudo -i yum install -y wget git yum-priorities DEBUG [a54cf65d] Loaded plugins: fastestmirror, priorities DEBUG [a54cf65d] Loading mirror speeds from cached hostfile epel/metalink | 6.4 kB 00:00 DEBUG [a54cf65d] * base: ftp.tsukuba.wide.ad.jp DEBUG [a54cf65d] * epel: ftp.kddilabs.jp DEBUG [a54cf65d] * extras: ftp.tsukuba.wide.ad.jp DEBUG [a54cf65d] * remi: rpms.famillecollet.com DEBUG [a54cf65d] * remi-php55: rpms.famillecollet.com DEBUG [a54cf65d] * remi-test: rpms.famillecollet.com DEBUG [a54cf65d] * rpmforge: ftp.kddilabs.jp DEBUG [a54cf65d] * updates: mirrors.tuna.tsinghua.edu.cn ...

... DEBUG [80062be9] GlusterFS configure summary DEBUG [80062be9] =========================== DEBUG [80062be9] FUSE client : yes DEBUG [80062be9] Infiniband verbs : no DEBUG [80062be9] epoll IO multiplex : yes DEBUG [80062be9] argp-standalone : no DEBUG [80062be9] fusermount : yes DEBUG [80062be9] readline : yes DEBUG [80062be9] georeplication : yes DEBUG [80062be9] Linux-AIO : yes DEBUG [80062be9] Enable Debug : yes DEBUG [80062be9] systemtap : yes DEBUG [80062be9] Block Device backend : yes DEBUG [80062be9] glupy : yes DEBUG [80062be9] Use syslog : yes DEBUG [80062be9] XML output : yes DEBUG [80062be9] QEMU Block formats : no ... INFO [a9e8de7e] Finished in 0.015 seconds with exit status 0 (successful).

$ bundle exec cap staging gluster:pool:list INFO [201946fe] Running /usr/bin/env sudo -i gluster pool list on 192.168.200.101 DEBUG [201946fe] Command: sudo -i gluster pool list DEBUG [201946fe] UUID Hostname State DEBUG [201946fe] d185fa19-d1a1-489c-a653-4510122f5278 localhost Connected INFO [201946fe] Finished in 0.294 seconds with exit status 0 (successful). $ bundle exec cap staging glusterd:status INFO [cb482772] Running /usr/bin/env sudo -i /sbin/service glusterd status on 192.168.200.101 DEBUG [cb482772] Command: sudo -i /sbin/service glusterd status INFO [cb482772] Finished in 0.151 seconds with exit status 0 (successful). DEBUG [cb482772] glusterd (pid 25855) is running... INFO [cb482772] Finished in 0.151 seconds with exit status 0 (successful).

$ bundle exec cap staging gluster:volume:help INFO [4628fd38] Running /usr/bin/env sudo -i gluster volume help on 192.168.200.101 DEBUG [4628fd38] Command: sudo -i gluster volume help DEBUG [4628fd38] volume info [all|<VOLNAME>] - list information of all volumes DEBUG [4628fd38] volume create <NEW-VOLNAME> [stripe <COUNT>] [replica <COUNT>] [device vg] [transport <tcp|rdma|tcp,rdma>] <NEW-BRICK> ... [force] - create a new volume of specified type with mentioned bricks DEBUG [4628fd38] volume delete <VOLNAME> - delete volume specified by <VOLNAME> DEBUG [4628fd38] volume start <VOLNAME> [force] - start volume specified by <VOLNAME> DEBUG [4628fd38] volume stop <VOLNAME> [force] - stop volume specified by <VOLNAME> DEBUG [4628fd38] volume add-brick <VOLNAME> [<stripe|replica> <COUNT>] <NEW-BRICK> ... [force] - add brick to volume <VOLNAME> DEBUG [4628fd38] volume remove-brick <VOLNAME> [replica <COUNT>] <BRICK> ... {start|stop|status|commit|force} - remove brick from volume <VOLNAME> DEBUG [4628fd38] volume rebalance <VOLNAME> [fix-layout] {start|stop|status} [force] - rebalance operations DEBUG [4628fd38] volume replace-brick <VOLNAME> <BRICK> <NEW-BRICK> {start [force]|pause|abort|status|commit [force]} - replace-brick operations DEBUG [4628fd38] volume set <VOLNAME> <KEY> <VALUE> - set options for volume <VOLNAME> ...

capistrano-glusterfs is available here: https://github.com/keithseahus/capistrano-glusterfs

Tips 2 NFS Workloads

Tips 2: NFS Workloads

!   Question 1: Do you mount GlusterFS in NFS?

!   Question 1: Do you mount GlusterFS in NFS? !   Question 2: Which is your main workload, write or read?

peer 1

glusterfsd

brick brick brick brick

glusterd glustershd glusterfs

peer 2

d shd fs

b b b b

peer 1

glusterfsd

peer 2

d shd fs

b b b b

TCP connection in gluster protocol

peer 1

glusterfsd

peer 2

d shd fs

b b b b

NFS Client

kernel

vfs/mnt/nfs

user app

page cache

peer 1

glusterfsd

peer 2

d shd fs

b b b b

NFS Client

kernel

vfs/mnt/nfs

user app

page cache

TCP connection in NFS v3

peer 1

glusterfsd

peer 2

d shd fs

b b b b

NFS Client

kernel

vfs/mnt/nfs

user app

page cache

read access to a file

peer 1

glusterfsd

peer 2

d shd fs

b b b b

NFS Client

kernel

vfs/mnt/nfs

user app

page cache

performance/io-cache

peer 1

glusterfsd

peer 2

d shd fs

b b b b

NFS Client

kernel

vfs/mnt/nfs

user app

page cache cachefilesd

/var/cache/fscacheactivate cachefilesd

peer 1

glusterfsd

peer 2

d shd fs

b b b b

NFS Client

kernel

vfs/mnt/nfs

user app

/var/cache/fscache

read files from its own RAM!

# NFS_SERVER=192.168.200.101 # MOUNT_POINT=/mnt/nfs

# NFS_SERVER=192.168.200.101 # MOUNT_POINT=/mnt/nfs # yum install -y cachefilesd

# NFS_SERVER=192.168.200.101 # MOUNT_POINT=/mnt/nfs # yum install -y cachefilesd # if test `grep 'SELINUX=disabled' /etc/selinux/config > /dev/null; echo $?` -ne 0; then sed -i 's/^secctx/#secctx/g' /etc/cachefilesd.conf fi

# NFS_SERVER=192.168.200.101 # MOUNT_POINT=/mnt/nfs # yum install -y cachefilesd # if test `grep 'SELINUX=disabled' /etc/selinux/config > /dev/null; echo $?` -ne 0; then sed -i 's/^secctx/#secctx/g' /etc/cachefilesd.conf fi # chkconfig cachefilesd on # service cachefilesd start

# NFS_SERVER=192.168.200.101 # MOUNT_POINT=/mnt/nfs # yum install -y cachefilesd # if test `grep 'SELINUX=disabled' /etc/selinux/config > /dev/null; echo $?` -ne 0; then sed -i 's/^secctx/#secctx/g' /etc/cachefilesd.conf fi # chkconfig cachefilesd on # service cachefilesd start # mount -t nfs -o fsc,vers=3 $NFS_SERVER $MOUNT_POINT

# free -m root@dev-centos:/root total used free shared buffers cached Mem: 996 926 69 0 32 579 -/+ buffers/cache: 314 682 Swap: 2015 0 2015 # ls /var/cache/fscache/cache/@4a/I03nfs/@7f/Jc0000000000000cG0I00/@8d/J1100000000000QFi24tY1EH900000000000M000000g00000100Kb000wFe000jt000oG3000000040000g000000000/ @01 @0d @13 @1b @22 @35 @40 @50 @58 @61 @6e @78 @87 @a7 @b3 @b9 @d1 @d7 @e6 @06 @10 @17 @1e @25 @3c @4d @52 @5d @63 @6f @81 @97 @a9 @b6 @c1 @d3 @dc @f1 @0a @11 @1a @21 @2b @3d @4f @57 @5f @66 @73 @82 @9a @ae @b7 @cc @d4 @de @fe # du -h /var/cache/fscache/cache/@4a/I03nfs/@7f/Jc0000000000000cG0I00/@8d/J1100000000000QFi24tY1EH900000000000M000000g00000100Kb000wFe000jt000oG3000000040000g000000000/ 186M /var/cache/fscache/cache/@4a/I03nfs/@7f/Jc0000000000000cG0I00/@8d/J1100000000000QFi24tY1EH900000000000M000000g00000100Kb000wFe000jt000oG3000000040000g000000000/

nodes: VirtualBox CentOS 6.4 x86_64 4 vCPUs, Core i7 2.4GHz 2GB RAM Striped SSDs

postmark 1.51 set size 1048576 10485760 set number 500 set transactions 1000 set location /mnt/nfs/rep2 set subdirectories 100 set read 8192 set write 8192 set buffering true

MB/sec (read)postmark 1.51 set size 1048576 10485760 set number 500 set transactions 1000 set location /mnt/nfs/rep2 set subdirectories 100 set read 8192 set write 8192 set buffering true

↓Oops...

Okay... but this is still a hypothesis. We need your feedback!

Tips 3 Logging

Tips 3: Logging

!   Question 1: Do you monitor GlusterFS logs?

Tips 3: Logging

!   Question 1: Do you monitor GlusterFS logs? !   Question 2: Do you utilize the logs?

Tips 3: Logging

!   Question 1: Do you monitor GlusterFS logs? !   Question 2: Do you utilize the logs? !   Question 3: Do you use a log collector or aggregator?

Tips 3: Logging

I pick it up because: •  easy to install •  easy to configure •  make unstructured logs structured •  collect logs from multiple resources •  output aggregated and structured

logs into a target as you prefer •  easy to create a module •  high performance •  well maintained

Tips 3: Logging

peer 1

glusterfsd

peer 2

d shd fs

b b b b

NFS Client

kernel

vfs/mnt/nfs

user app

/var/cache/fscache

Tips 3: Logging

peer 1

glusterfsd

peer 2

d shd fs

b b b b

NFS Client

kernel

vfs/mnt/nfs

user app

/var/cache/fscache

Log server

Tips 3: Logging

peer 1

glusterfsd

peer 2

d shd fs

b b b b

NFS Client

kernel

vfs/mnt/nfs

user app

/var/cache/fscache

Log server

Tips 3: Logging

peer 1

glusterfsd

peer 2

d shd fs

b b b b

NFS Client

kernel

vfs/mnt/nfs

user app

/var/cache/fscache

Log server

Tips 3: Logging

peer 1

glusterfsd

peer 2

d shd fs

b b b b

NFS Client

kernel

vfs/mnt/nfs

user app

/var/cache/fscache

Log server

Tips 3: Logging

peer 1

glusterfsd

peer 2

d shd fs

b b b b

NFS Client

kernel

vfs/mnt/nfs

user app

/var/cache/fscache

Log server

Tips 3: Logging/etc/td-agent/td-agent.conf (on the log server)

<source> type forward port 24224 bind 0.0.0.0 </source> <match filtered.glusterd.**> type file path /home/keith/var/log/fluent/glusterd </match>

Today I will get the td-agent output the logs into a file.

Tips 3: Logging

<source> type tail path /var/log/glusterfs/usr-local-glusterfs-3.4.0beta2-etc-glusterfs-glusterd.vol.log pos_file /var/log/td-agent/usr-local-glusterfs-3.4.0beta2-etc-glusterfs-glusterd.vol.log.pos tag glusterd format /^(?:\[(?<date_ymd>[0-9]{4}-[01][0-9]-[0-3][0-9]) (?<time_hms>[0-2][0-9]\:[0-5][0-9]\:[0-6][0-9])\.(?<time_usec>[0-9]{6})\]) (?<log_level>[TDINWECA]) (?:\[(?<source_file_name>[^\[\]:/ ]*):(?<source_line>[0-9]*):(?<function_name>[^\[\]:/ ]*)\]) (?<component>[^\[\]:/ ]*)\: *(?<message>.*)$/ rotate_wait 5 </source> <match glusterd.**> type hostname key_name host add_prefix filtered </match> <match filtered.glusterd.**> type forward send_timeout 60s recover_wait 10s heartbeat_interval 1s phi_threshold 8 hard_timeout 60s <server> …

/etc/td-agent/td-agent.conf (on the GlusterFS peers)

Too complex to input and maintain!

Tips 3: Logging

You can specify log levels to forward.

Tips 3: Logging

You can specify each field name to output in JSON.

Comment out if you don't need the fields.

Tips 3: Logging

Parse error should be treated as a failure of fluent-plugin-glusterfs.

Tips 3: Logging

[root@glusterfs-unstable-01 plugin]# tail -3 /var/log/glusterfs/usr-local-glusterfs-3.4.0beta2-etc-glusterfs-glusterd.vol.log [2013-07-07 15:52:43.778961] I [socket.c:3495:socket_init] 0-management: using system polling thread [2013-07-07 15:52:43.779397] W [socket.c:514:__socket_rwv] 0-management: readv failed (No data available) [2013-07-07 15:52:43.779668] I [socket.c:2236:socket_event_handler] 0-transport: disconnecting now

[root@dev-centos ~]# grep glusterfs-unstable-01 /home/keith/var/log/fluent/glusterd.20130708.b4e0ed6e32655d35f | tail -3 2013-07-08T00:52:43+09:00 filtered.glusterd {"date_ymd":"2013-07-07","time_hms":"15:52:43","time_usec":"778961","log_level":"I","source_file_name":"socket.c","source_line":"3495","function_name":"socket_init","component":"0-management","message":"using system polling thread","host":"glusterfs-unstable-01.localdomain"} 2013-07-08T00:52:43+09:00 filtered.glusterd {"date_ymd":"2013-07-07","time_hms":"15:52:43","time_usec":"779397","log_level":"W","source_file_name":"socket.c","source_line":"514","function_name":"__socket_rwv","component":"0-management","message":"readv failed (No data available)","host":"glusterfs-unstable-01.localdomain"} 2013-07-08T00:52:43+09:00 filtered.glusterd {"date_ymd":"2013-07-07","time_hms":"15:52:43","time_usec":"779668","log_level":"I","source_file_name":"socket.c","source_line":"2236","function_name":"socket_event_handler","component":"0-transport","message":"disconnecting now","host":"glusterfs-unstable-01.localdomain"}

It makes the logs structured in JSON.

Tips 3: Logging

The logs on each peer could be output in a single log file.

Tips 3: Logging

Also available on Gluster Forge: https://forge.gluster.org/fluent-plugin-glusterfs

Tips 3: Logging

More detailed explanation in Japanese is here!: http://www.nttpc.co.jp/company/r_and_d/technology/gluster_fs_fluentd.html

Conclusion

!  Deployment automation tools are useful to install and operate GlusterFS on multiple nodes.

!  With cachefilesd, NFS can might be more efficient under heavy read workload.

!  Aggregating GlusterFS logs is easy enough and you can utilize them.

gluster cloud night in tokyo 2013 -- tips for getting started

nttpc communications

installation copyright

installation tips

glusterfsglusterfscopyright

installationcopyright

capistranocopyright

gluster community copyright

logging conclusioncopyright

Technology

gluster fs architecture_&_roadmap_atin_punemeetup_2015

gluster can do...

red hat gluster storage advanced features information · pdf...

red hat gluster storage advanced features information ·...

gluster and kubernetes

gluster fs architecture_future_directions_tlv

red hat gluster storage advanced features...

gluster object storage

gluster hadoop compatible storage

object storage with gluster

20160401 gluster-roadmap

clusters with gluster fs

gluster ovirt integration_gluster_meetup_pune_2015

gluster containers!

demystifying gluster

openstack tokyo meeup - gluster storage day

red hat gluster storage 3...gluster storage 3.4.0 and is the...

gluster fs buero20_presentation

red hat gluster storage 3€¦ · red hat gluster storage...

gluster for sysadmins