introduction to irods christine staiger surfsara amsterdam, 8 th may 2014 10 th oct 2014
TRANSCRIPT
![Page 1: Introduction to iRODS Christine Staiger SURFsara Amsterdam, 8 th May 2014 10 th Oct 2014](https://reader036.vdocument.in/reader036/viewer/2022062313/56649cab5503460f9496c029/html5/thumbnails/1.jpg)
Introduction to iRODS
Christine StaigerSURFsara
Amsterdam, 8th May 2014
10th Oct 2014
![Page 2: Introduction to iRODS Christine Staiger SURFsara Amsterdam, 8 th May 2014 10 th Oct 2014](https://reader036.vdocument.in/reader036/viewer/2022062313/56649cab5503460f9496c029/html5/thumbnails/2.jpg)
iRODS Concepts
The Basics
① Installation
② Tools for administration
③ Adding data and metadata
Contents
![Page 3: Introduction to iRODS Christine Staiger SURFsara Amsterdam, 8 th May 2014 10 th Oct 2014](https://reader036.vdocument.in/reader036/viewer/2022062313/56649cab5503460f9496c029/html5/thumbnails/3.jpg)
3
iRODS Concepts
![Page 4: Introduction to iRODS Christine Staiger SURFsara Amsterdam, 8 th May 2014 10 th Oct 2014](https://reader036.vdocument.in/reader036/viewer/2022062313/56649cab5503460f9496c029/html5/thumbnails/4.jpg)
4
What is iRODS?
![Page 5: Introduction to iRODS Christine Staiger SURFsara Amsterdam, 8 th May 2014 10 th Oct 2014](https://reader036.vdocument.in/reader036/viewer/2022062313/56649cab5503460f9496c029/html5/thumbnails/5.jpg)
5
What does iRODS provide?• Storage virtualization of different disk and tape
storage systems
• A logical namespace across storage locations
• A rule engine to automate data management according to defined policies
• Various client interfaces
![Page 6: Introduction to iRODS Christine Staiger SURFsara Amsterdam, 8 th May 2014 10 th Oct 2014](https://reader036.vdocument.in/reader036/viewer/2022062313/56649cab5503460f9496c029/html5/thumbnails/6.jpg)
6
iRODS Components
![Page 7: Introduction to iRODS Christine Staiger SURFsara Amsterdam, 8 th May 2014 10 th Oct 2014](https://reader036.vdocument.in/reader036/viewer/2022062313/56649cab5503460f9496c029/html5/thumbnails/7.jpg)
7
iRODS Rule Engine
![Page 8: Introduction to iRODS Christine Staiger SURFsara Amsterdam, 8 th May 2014 10 th Oct 2014](https://reader036.vdocument.in/reader036/viewer/2022062313/56649cab5503460f9496c029/html5/thumbnails/8.jpg)
8
The users view:iRODS Data Collections
Collection0
DataObject
DataObject
DataObjectData
Object A
Collection1
DataObject
DataObjectData
Object X
…/Collection0/…/Collection0/DataObject A-B…/Collection0/Collection1/…/Collection0/Collection1/DataObject X-Z
Logical Name Space (iCAT) independent from physical resources
• Similar to Unix Directory and Files• ichmod to set user/group ACLs• Core.re: msiSetDefaultResc(Resource)• .irodsEnv: irodsDefResource=Resource
![Page 9: Introduction to iRODS Christine Staiger SURFsara Amsterdam, 8 th May 2014 10 th Oct 2014](https://reader036.vdocument.in/reader036/viewer/2022062313/56649cab5503460f9496c029/html5/thumbnails/9.jpg)
9
In the background:iRODS Resources
• (Storage) Resource is a Software or Hardware system that stores data
• 3 Resource classes:
HighLatencyLow
Latency
ArchiveResource
Cache Resource
CompoundResource
POSIX like
IO calls
![Page 10: Introduction to iRODS Christine Staiger SURFsara Amsterdam, 8 th May 2014 10 th Oct 2014](https://reader036.vdocument.in/reader036/viewer/2022062313/56649cab5503460f9496c029/html5/thumbnails/10.jpg)
iRODS Resource Groups
10
LowLatency
Cache Resource
POSIX like
IO calls
Resource Group
LowLatency
Cache Resource
LowLatency
Cache Resource
@Server A @Server B @Server C
![Page 11: Introduction to iRODS Christine Staiger SURFsara Amsterdam, 8 th May 2014 10 th Oct 2014](https://reader036.vdocument.in/reader036/viewer/2022062313/56649cab5503460f9496c029/html5/thumbnails/11.jpg)
11
Hands on The Basics
![Page 12: Introduction to iRODS Christine Staiger SURFsara Amsterdam, 8 th May 2014 10 th Oct 2014](https://reader036.vdocument.in/reader036/viewer/2022062313/56649cab5503460f9496c029/html5/thumbnails/12.jpg)
Demo Data GridsVirtual machines: ssh user@IP
IP adress User Password
145.100.58.144 irods1 irods104
145.100.58.147 irods2 irods204
145.100.58.148 irods3 irods304
145.100.58.150 irods4 irods404
145.100.58.155 irods6 irods604
145.100.58.157 irods7 irods704
145.100.58.158 irods8 irods804
145.100.58.160 irods9 irods904
145.100.58.161 irods10 irods1004
![Page 13: Introduction to iRODS Christine Staiger SURFsara Amsterdam, 8 th May 2014 10 th Oct 2014](https://reader036.vdocument.in/reader036/viewer/2022062313/56649cab5503460f9496c029/html5/thumbnails/13.jpg)
Installation and configuration• Execute ./irodssetup in the folder iRODS
• Export the icommandsexport PATH=$PATH:/home/user/iRODS/clients/icommands/bin
• ~/iRODS/config/irods.config
# Database configuration$DATABASE_TYPE = 'postgres';…$DATABASE_HOST = 'localhost';$DATABASE_PORT = '5432';$DATABASE_ADMIN_PASSWORD = 'irods2';$DATABASE_ADMIN_NAME = ’alice';
# iRODS configurationIRODS_HOME = '/home/alice/iRODS';$IRODS_PORT = '1247’;…$IRODS_ADMIN_PASSWORD = ’alice';$IRODS_ICAT_HOST = '';
![Page 14: Introduction to iRODS Christine Staiger SURFsara Amsterdam, 8 th May 2014 10 th Oct 2014](https://reader036.vdocument.in/reader036/viewer/2022062313/56649cab5503460f9496c029/html5/thumbnails/14.jpg)
.irodsEnv file # iRODS personal configuration file.## This file was automatically created during iRODS installation.# Created Mon Jun 18 13:13:27 2012## iRODS server host name:irodsHost ’surfsara-cloud'# iRODS server port number:irodsPort 1247# Default storage resource name:irodsDefResource 'demoResc'# Home directory in iRODS:irodsHome '/alicesZone/home/alice'# Current directory in iRODS:irodsCwd '/alicesZone/home/alice'# Account name:irodsUserName ’alice'# Zone:irodsZone ‘alicesZone'
The .irodsEnvdetermines the zone and the user for the icommands-client.You can only use onesuch file at a time.
![Page 15: Introduction to iRODS Christine Staiger SURFsara Amsterdam, 8 th May 2014 10 th Oct 2014](https://reader036.vdocument.in/reader036/viewer/2022062313/56649cab5503460f9496c029/html5/thumbnails/15.jpg)
iRODS i-commands
Documentation: https://wiki.irods.org/index.php/icommands
(we don’t provide a exhaustive list of i-commands)
![Page 16: Introduction to iRODS Christine Staiger SURFsara Amsterdam, 8 th May 2014 10 th Oct 2014](https://reader036.vdocument.in/reader036/viewer/2022062313/56649cab5503460f9496c029/html5/thumbnails/16.jpg)
16
iRODS control• Execute ./irodsctl <arg> in the folder iRODS
• Restarting, starting and stopping the iRODS and iCAT server with• <arg> = restart, start, stop
• Restarting, starting and stopping only the iRODS server with• <arg> = irestart, istart, istop
• Initialise an account and log in with iinit
![Page 17: Introduction to iRODS Christine Staiger SURFsara Amsterdam, 8 th May 2014 10 th Oct 2014](https://reader036.vdocument.in/reader036/viewer/2022062313/56649cab5503460f9496c029/html5/thumbnails/17.jpg)
The icommands client• icommands are automatically installed when you install the iRODS
server
• Users must install the client software to connect to an iRODS server
• Categories:• Informational• UNIX and FTP like• Metadata• Functional
![Page 18: Introduction to iRODS Christine Staiger SURFsara Amsterdam, 8 th May 2014 10 th Oct 2014](https://reader036.vdocument.in/reader036/viewer/2022062313/56649cab5503460f9496c029/html5/thumbnails/18.jpg)
ihelp• ihelp Lists all commands
• ihelp -h Usage : ihelp [-ah] [icommand] Display i-commands synopsis or a particular i-command help text Options are: -h this help -a print the help text for all the i-commands
Run with no options to display a synopsis of the i-commands
iRODS Version 3.3.1 February 2014 ihelp
![Page 19: Introduction to iRODS Christine Staiger SURFsara Amsterdam, 8 th May 2014 10 th Oct 2014](https://reader036.vdocument.in/reader036/viewer/2022062313/56649cab5503460f9496c029/html5/thumbnails/19.jpg)
ienv• Lists the iRODS environment variables.
• Equivalent to “iinit -l”
• Usage : ienv [-h]
NOTICE: Release Version = rods3.3.1, API Version = dNOTICE: irodsHost=surfsara-cloudNOTICE: irodsPort=1247NOTICE: irodsDefResource=demoRescNOTICE: irodsHome=/alicesZone/home/aliceNOTICE: irodsCwd=/alicesZone/home/aliceNOTICE: irodsUserName=aliceNOTICE: irodsZone=alicesZone
![Page 20: Introduction to iRODS Christine Staiger SURFsara Amsterdam, 8 th May 2014 10 th Oct 2014](https://reader036.vdocument.in/reader036/viewer/2022062313/56649cab5503460f9496c029/html5/thumbnails/20.jpg)
iuserinfo• Show information about your iRODS user account or the entered
user
• Usage: iuserinfo [-vVh] [user]
name: aliceid: 10007type: rodsadminzone: alicesZoneinfo: comment: create time: 01396887567: 2014-04-07.18:19:27modify time: 01396887567: 2014-04-07.18:19:27member of group: alicemember of group: public
![Page 21: Introduction to iRODS Christine Staiger SURFsara Amsterdam, 8 th May 2014 10 th Oct 2014](https://reader036.vdocument.in/reader036/viewer/2022062313/56649cab5503460f9496c029/html5/thumbnails/21.jpg)
ilsresc• ilsresc lists iRODS resources and resource-groups
• Usage: ilsresc [-lvVhA] [Name]
alice@sara-cloud:~$ ilsrescdemoResc
alice@sara-cloud:~$ ilsresc –l demoRescresource name: demoResc resc id: 10010 zone: alicesZonetype: unix file system class: archive location: surfsara-cloud vault: /home/alice/iRODS/Vault …
![Page 22: Introduction to iRODS Christine Staiger SURFsara Amsterdam, 8 th May 2014 10 th Oct 2014](https://reader036.vdocument.in/reader036/viewer/2022062313/56649cab5503460f9496c029/html5/thumbnails/22.jpg)
Some commands• Unix-like:
ils, ipwd, icd, imkdir, irm, icp ichmod, ipasswd, irsync, ichksum, imv
• FTP-like:iinit, iput, iget, iexit
• Test: Create folder in the logical name space ipwd
imkdir NewFolderils NewFolderirm –r NewFolder
![Page 23: Introduction to iRODS Christine Staiger SURFsara Amsterdam, 8 th May 2014 10 th Oct 2014](https://reader036.vdocument.in/reader036/viewer/2022062313/56649cab5503460f9496c029/html5/thumbnails/23.jpg)
User and Data administration• Change to admin mode: iadmin• Quit admin mode: q
• Make/delete a user: mkuser, rmuser, moduser (iadmin) mkuser user1 rodsuser (iadmin) moduser user1 password *****
• Parameters and their values: iadmin lt, iadmin lt user_type
• iadmin mkresc, iadmin rmresc
• Create new resource: mkdir Data (in your home, not in the iRODS environment) iadmin mkresc NewDataResc “unix file system” cache 145.100.57.139 “/home/alice/Data”
![Page 24: Introduction to iRODS Christine Staiger SURFsara Amsterdam, 8 th May 2014 10 th Oct 2014](https://reader036.vdocument.in/reader036/viewer/2022062313/56649cab5503460f9496c029/html5/thumbnails/24.jpg)
Put and replicate data
• Add the file /home/login/put1.txt to resource iput -v put1.txt ils -l or ils -L iput -h
• ichksum put1.txt (checksum of copy in Vault) md5sum put1.txt (checksum of file in /home/login)
• iput -R NewDataResc put2.txt ils -L
• Replicate data: irepl -R NewDataResc put1.txt
• Trim number of copies: itrim -S demoResc -N 1 put1.txt
![Page 25: Introduction to iRODS Christine Staiger SURFsara Amsterdam, 8 th May 2014 10 th Oct 2014](https://reader036.vdocument.in/reader036/viewer/2022062313/56649cab5503460f9496c029/html5/thumbnails/25.jpg)
Access Control Lists• Change permission for a user: ichmod read user1 put1.txt List access: ils -A
• Change permissions for group “public”: Make a directory in the iRODS environment: imkdir temp Make directory readable for “public”: ichmod -r read public temp
• ils -A /bobsZone/home/bob/temp: ACL - bob#bobsZone:own g:public#bobsZone:read object Inheritance - Disabled
![Page 26: Introduction to iRODS Christine Staiger SURFsara Amsterdam, 8 th May 2014 10 th Oct 2014](https://reader036.vdocument.in/reader036/viewer/2022062313/56649cab5503460f9496c029/html5/thumbnails/26.jpg)
iRODS Metadata• Commands: imeta, iquest, idbo
• Add metadata: imeta add -d put1.txt “Date” “08.10.2013” imeta add -d put1.txt “AnotherDate” “04.04.2014” imeta ls -d put1.txt
• Remove metadata: imeta rm -d put1.txt “Date” “08.10.2013”
• Querying the iCAT: iquest "SELECT DATA_NAME,DATA_CHECKSUM WHERE
DATA_RESC_NAME like ’%Resc%’” DATA_NAME = put1.txt DATA_CHECKSUM = c16d7d3488677a3348b12eb82795d28c
![Page 27: Introduction to iRODS Christine Staiger SURFsara Amsterdam, 8 th May 2014 10 th Oct 2014](https://reader036.vdocument.in/reader036/viewer/2022062313/56649cab5503460f9496c029/html5/thumbnails/27.jpg)
Federation between data grids• List all existing zones: ils /• Zone A acknowledges Zone B and vice-cersa
iadmin mkzone B remote Host:Port• Zone A adds remote user from Zone B: iadmin mkuser user#B rodsuser
• Example: Alice and Bob:Alice acknowledges Bob’s zone:
iadmin mkzone bobsZone remote 145.100.57.74:1247 Bob acknowledges Alice’s zone:
iadmin mkzone alicesZone remote 145.100.57.66:1247 Alice creates remote user Bob:
iadmin mkuser bob#bobsZone rodsuser Now Bob can list and alter files in Alice’s zone: ils /alicesZone
![Page 28: Introduction to iRODS Christine Staiger SURFsara Amsterdam, 8 th May 2014 10 th Oct 2014](https://reader036.vdocument.in/reader036/viewer/2022062313/56649cab5503460f9496c029/html5/thumbnails/28.jpg)
Data Replication between grids• Bob copies a file from his zone to his account in Alice’s zone irsync -R demoResc i:/bobsZone/home/bob/put1.txt i:/alicesZone/home/bob#bobsZone/put1.txt -R specifies the target resource
• Bob copies a file from his home directory to Alice’s zone irsync -R demoResc put3.txt i:/alicesZone/home/bob#bobsZone/put3.txt
• Bob copies a file from Alice’s zone to his home directory mkdir test irsync i:/alicesZone/home/bob#bobsZone/put3.txt test/put3.txt
• Bob deletes a file in Alice’s zone irm /alicesZone/home/bob#bobsZone/put3.txt
![Page 29: Introduction to iRODS Christine Staiger SURFsara Amsterdam, 8 th May 2014 10 th Oct 2014](https://reader036.vdocument.in/reader036/viewer/2022062313/56649cab5503460f9496c029/html5/thumbnails/29.jpg)
Thank you !
![Page 30: Introduction to iRODS Christine Staiger SURFsara Amsterdam, 8 th May 2014 10 th Oct 2014](https://reader036.vdocument.in/reader036/viewer/2022062313/56649cab5503460f9496c029/html5/thumbnails/30.jpg)
Appendix
![Page 31: Introduction to iRODS Christine Staiger SURFsara Amsterdam, 8 th May 2014 10 th Oct 2014](https://reader036.vdocument.in/reader036/viewer/2022062313/56649cab5503460f9496c029/html5/thumbnails/31.jpg)
Trouble shooting
ERROR: putUtil: put error for /alicesZone/home/bob#bobsZone/put1.txt, status = -9000 status = -9000 SYS EXCEED CONNECT CNT
Host configuration problem on the server.Solution: Edit iRODS/server/config/irodsHost
localhost you.rIP.XX.XX surfsara-cloudlocalhost 127.0.0.1
./irodsctl restart