iRODS Tutorial

II. Data Grid Administration

Leesa Brieger

2

iRODS Tutorial Preview

I. iRODS Getting Started – Unix client (icommands)– Usage

II. iRODS Data Grid Administration– Installing server and iCAT– Changing default settings– Setting up users– Installing resource server and adding new resources

to a data grid/zone– Federating with other grids/zones, remote users– Microservices and rules for policy implementation

and enforcement

3

II. iRODS Data Grid Administration

4

iRODS Info• Main page: http://irods.org

• Github repositories: https://github.com/irods/ (Request a guest account to get access.)

• Github tickets (issues): https://github.com/irods/irods/issues?state=open

• Chat list: irods-chat@irods.org

• The original iRODS wiki: https://wiki.irods.org

• The iRODS Manual: http://irods.org/dev/wp-content/uploads/2014/03/irods-manual-4.0.0.pdf

5

iRODS Books

Available from Amazon

• The integrated Rule-Oriented Data System (iRODS) Micro-service Workbookhttp://www.amazon.com/dp/1466469129

• An updated microservices reference book is coming out soon.

6

iRODS Download• Download link for binary packages from the iRODS main page

http://irods.org/download/

• Open source, BSD license

• Binary packages – rpm and deb – available for CentOS, Red Hat, Debian, Ubuntu, SUSE. Planned: Mac OSX, Windows– See http://irods.org/2014/03/irods-4-0-0-released/ – 4.0 tested platforms: CentOS 5 & 6, SuSE 11 & 12, Ubuntu 10 & 12

• iRODS source is available via GitHubhttps://github.com/irods/irods

• Other iRODS packages and plug-ins are available via GitHub athttps://github.com/irods

7

iRODS iCAT-Enabled Server (IES)

• Installation – we’ll do Ubuntu and CentOS.

• This is the heart of the data grid.

• Other resources, on other platforms can be added to the data grid.

• We will use PostgreSQL as the database for the iCAT for our installations; with database plug-ins, other DBs can also be used.

8

iRODS Data Grid Installation - Ubuntu 12.04

• First, set up PostgreSQL> sudo apt-get install postgresql-9.3

This creates a Unix service account “postgres”, the DB superuser.

• As user postgres, create a DB role/user “irods”, with rights to create DBs in the Postgres instance

> sudo su – postgres> createuser -d -e -l -P irods

Enter password for new role: ******* Enter it again: ******* CREATE ROLE irods PASSWORD

'md50061c043f3987cc21363299b9b1c0c3f' NOSUPERUSER CREATEDB NOCREATEROLE INHERIT LOGIN;

> sudo exit

continued…

Enter the “irods” DB pw here; this will be usedas the iCAT admin pw.

Installing the iCAT-enabled Server

9

• Once “irods” is a user/role for the database, then install iRODS• Download the iRODS packages

– irods-icat-4.0.0-64bit.deb – irods-database-plugin-postgres-1.0.deb

• Install the packages> sudo dpkg -i irods-icat-4.0.0-64bit.deb irods-database-plugin-postgres-1.0.deb> sudo apt-get -f install

• Open the ports for iRODS client and server contact and parallel transfers

> sudo iptables -A INPUT -m state --state NEW -m tcp -p tcp --dport 1247 –j ACCEPT> sudo iptables -A INPUT -m state --state NEW -m tcp -p tcp --dport 20000:20199 -j ACCEPT> sudo iptables -A INPUT -m state --state NEW -m udp -p udp –dport 20000:20199 –j ACCEPT

continued…

iRODS Installation - Ubuntu 12.04Installing the iCAT-enabled Server

This creates an “irods”service account.

10

• As user irods, run the setup script> sudo su – irods (Home directory for this user is /var/lib/irods)> ./packaging/setup_database.sh

This creates the iCAT database, using Postgresrole ”irods”, and starts

up the data grid, with iRODSadmin user “rods”

iRODS Installation - Ubuntu 12.04Installing the iCAT-enabled Server

11

Running the setup script setup_database.sh carries out these actions…• Prompts for config input

– hostname (localhost)– port #– DB user name and pw (user “irods” and pw as specified in the postgres setup)

• Creates an iRODS admin user “rods” with pw “rods” (This is an iRODS account, not Unix or postgres.)

• Creates configuration files in /etc/irods/

• Creates the iCAT and brings up the data grid• Creates the environment file for iRODS user “rods”

• Names the data grid (zone) and its resource with default names– tempZone– demoResc (and default physical path /var/lib/irods/iRODS/Vault)

iRODS Installation – Ubuntu 12.04Installing the iCAT-enabled Server

Change this password with

“ipasswd”

(/var/lib/irods/.irods/.irodsEnv)

12

# iRODS personal configuration file.# This file was automatically created during iRODS installation.# iRODS server host name:irodsHost 'tutorub.renci.org'# iRODS server port number:irodsPort 1247# Default storage resource name:irodsDefResource 'demoResc'# Home directory in iRODS:irodsHome '/tempZone/home/rods'# Current directory in iRODS:irodsCwd '/tempZone/home/rods'# Account name:irodsUserName 'rods'# Zone:irodsZone 'tempZone’…

iRODS Environment File: ~/.irods/.irodsEnv

13

• This file contains the environment of the grid you want to contact.

• You can use multiple environment files to choose from among many grids (only one at a time has the name .irodsEnv) – This is only possible for Unix accounts that are not the “irods” service

account.

• NEVER change the .irodsEnv file in the “irods” service account running a grid

.irodsEnv

14

iRODS Data Grid Installation – CentOS 6

• Download the packages– RPM_INSTALLATION_HOWTO.txt– irods-icat-4.0.0-64bit-centos6.rpm– irods-database-plugin-postgres-1.0-centos6.rpm

• Get and install the packages that iRODS depends on> sudo yum install postgresql postgresql-server unixODBC

perl authd> sudo yum install postgresql-odbc

(This installs PostgreSQL that will be used for the iCAT.)

• Start the Postgres server and initialize database tables > sudo /sbin/service postgresql initdb

Initializing database: [ OK ]> sudo /sbin/service postgresql start

Starting postgresql service:

continued…

Installing the iCAT-enabled Server

Read this first

15

iRODS Data Grid Installation – CentOS 6

• As user postgres, create role/user irods with “create DB” privileges

> sudo su – postgres> createuser -d -e -l -P irods

Enter password for new role: ******* Enter it again: ******* CREATE ROLE irods PASSWORD

'md50061c043f3987cc21363299b9b1c0c3f' NOSUPERUSER CREATEDB NOCREATEROLE INHERIT LOGIN;

> exit

• Modify authd config file for xinetd.d (/etc/xinetd.d/auth) to remove the command line argument “-E”. Change [ server_args = -t60 --xerror --os -E ] to [ server_args = -t60 --xerror --os ]

continued…

Installing the iCAT-enabled Server

16

iRODS Data Grid Installation – CentOS 6

• Set the proper runlevel for authd: > sudo /sbin/chkconfig --level=3 auth on

• Restart xinetd > sudo /etc/init.d/xinetd restart

• Open your firewall, if necessary, to listen for iRODS and allow parallel transfers. Add the following to your /etc/sysconfig/iptables: -A INPUT -m state --state NEW -m tcp -p tcp --dport 1247 -j ACCEPT -A INPUT -m state --state NEW -m tcp -p tcp --dport 20000:20199 -j ACCEPT -A INPUT -m state --state NEW -m udp -p udp --dport 20000:20199 -j ACCEPT

• Restart the firewall: > sudo service iptables restart

continued…

Installing the iCAT-enabled Server

17

iRODS Data Grid Installation – CentOS 6Installing the iCAT-enabled Server

• Download the iRODS package and install> sudo rpm -i irods-icat-4.0.0-64bit-centos6.rpm

• As Unix user irods, run the database setup script:> sudo su – irods (Home directory for this user is /var/lib/irods)> ./packaging/setup_database.sh

• Respond to script promptsDB server's hostname: localhost (<--- "localhost" here for a local DB)port: 5432DB name: ICATusername: irodspw: *******

Enter your iCAT DBadmin pw here.

This creates Unixservice

account “irods”.

18

Running the setup script setup_database.sh carries out these actions…• Prompts for config input

– hostname (localhost)– port #– DB user name and pw (user “irods” and pw as specified in the postgres setup

• Creates an iRODS admin user “rods” with pw “rods” (This is an iRODS account, not Unix or postgres.)

• Creates configuration files in /etc/irods/

• Creates the iCAT and brings up the data grid• Creates the environment file for iRODS user “rods”

• Names the data grid (zone) and its resource with default names– tempZone– demoResc (and default physical path /var/lib/irods/iRODS/Vault)

iRODS Installation – CentOS 6Installing the iCAT-enabled Server

Change this password with

“ipasswd”

(/var/lib/irods/.irods/.irodsEnv)

19

Installing the iCAT-Enabled Server (IES) with a Remote iCAT PostgreSQL Database

On the database host machine:1) Update postgresql.conf2) Update pg_hba.conf3) Restart postgres4) Configure and restart iptables

On the iRODS server host machine:1) New setup script is necessary – will be supported in 4.0.32) During the configuration setup, give the hostname of the

remote database machine

20

Installing the IES with a Remote iCAT

• Location:– Ubuntu: /etc/postgresql/9.3/main– CentOS: /var/lib/pgsql/data

• Change required: – Update listen_addresses to contain the hostname (FQDN) or IP

address of the postgres host

1) postgresql.conf

21

2) pg_hba.conf• Location:

– Ubuntu: /etc/postgresql/9.3/main– CentOS: /var/lib/pgsql/data

• Change required: – Add a line to permit the ICAT server to connect using md5

authentication: host all all <hostname> md5

Installing the IES with a Remote iCAT

22

3) Restart postgres

• Ubuntu: sudo /etc/init.d/postgres restart

• CentOS: sudo /etc/init.d/postgres-9.3 restart

Installing the IES with a Remote iCAT

23

4) iptables• Ubuntu

– Install iptables-persistent: sudo apt-get install iptables-persistent

– Edit /etc/iptables/rules.v4– Add (before the COMMIT line): -A INPUT -m state --state NEW -m tcp -p tcp --dport 5432 -j ACCEPT– Restart iptables:

/etc/init.d/iptables restart

• CentOS– Edit /etc/sysconfig/iptables– Add this line:

-A INPUT -m state --state NEW -m tcp -p tcp --dport 5432 -j ACCEPT– Restart iptables:

service iptables restart

Installing the IES with a Remote iCAT

24

Server side: 1) Get the setup script that supports this (4.0.3)

2) During the configuration setup, give the hostname of the remote DB

Installing the IES with a Remote iCAT

25

# iRODS personal configuration file.# This file was automatically created during iRODS installation.# iRODS server host name:irodsHost 'tutoricat.renci.org'# iRODS server port number:irodsPort 1247# Default storage resource name:irodsDefResource 'demoResc'# Home directory in iRODS:irodsHome '/tempZone/home/rods'# Current directory in iRODS:irodsCwd '/tempZone/home/rods'# Account name:irodsUserName 'rods'# Zone:irodsZone 'tempZone’…

iRODS Environment File: ~/.irods/.irodsEnv

26

iRODS Post-Install• Configuration parameters saved in /etc/irods/irods.config and

/etc/irods/server.config• Install logs are in /var/lib/irods/iRODS/installLogs/irods_setup.log

• Server logs are in directory /var/lib/irods/iRODS/server/log/

• Rule file: /etc/irods/core.re contains rules that are triggered by actions in the data grid

• Environment file /var/lib/irods/.irods/.irodsEnv is created automatically for service account “irods”

• Copy this file into ~/.irods/.irodsEnv wherever you want admin access to the data grid

• Copy this file and modify it for all other users’ access to the data grid

27

iadmin – administrative functions• h for help, “h <command>” for help with that command

• q to exit

• Add new users, modify passwords, add new resources, federate to remote zones, create resource groups,…– mkresc/rmresc– mkuser/rmuser, moduser (modify passwords)– mkzone/rmzone, modzone (for federation)

• Information on users, resources, tokens, etc– lt (el-tee), “lt <token type>” for listing of that token’s possible

values– lu, lr, lz,…

28

iRODS control

./irodsctl– start– stop– restart

./irodsctl –h gives documentation on using irodsctl

Other options have been deprecated – iRODS no longer controls starting and stopping the iCAT DB

Start/stop/restart the iRODS server

29

iRODS Installation – Change Defaults• Modify default zone name (tempZone) • Modify default resource path (/var/lib/irods/iRODS/Vault/)• Modify default resource name (demoResc)• Alternative: Leave demoResc alone (name and path) and

instead, introduce a separate, new resource on the same host; just stop using demoResc

Examples1. Change zone name from “tempZone” to “ubgrid”2. Change default resource path to “/opt/irodsVault/”3. Change default resource name from “demoResc” to “ubresc1”4. Introduce new resource “ubresc2” on this server

30

iRODS Installation – Change Defaults1. Modify zone name (data grid name tempZone)

– If you’re going to do this, it’s a good idea (though not absolutely necessary) to do it before you set up other users

> iadminiadmin> modzone tempZone name ubgrid

– Once done, must edit the ~/.irods/.irodsEnv files of all users and change the zone name wherever it appears in that file (3 places)

– Must also edit any rules (in /etc/irods/core.re) that might contain the zone name

– Edit /etc/irods/irods.config: $ZONE_NAME = 'tempZone'; |-> $ZONE_NAME = ’ubgrid’

– Restart the data grid:> ./irodsctl restart

31

iRODS Installation – Change Defaults2. Modify resource path (/var/lib/irods/iRODS/Vault)

– Again, best to do this before there are other users and much data in the data grid

> iadmin iadmin> lr demoResc

resc_id: 10010resc_name: demoResczone_name: ubgridresc_type_name: unixfilesystemresc_net: tutorub.renci.orgresc_def_path: /var/lib/irods/iRODS/Vault…

iadmin> modresc demoResc path /opt/irodsVaultLevel 0: Previous resource path: /var/lib/irods/iRODS/Vault

32

iRODS Installation – Change Defaults2. Modify resource path (continued…)

> iadmin iadmin> lr demoResc

resc_id: 10010resc_name: demoResczone_name: ubgridresc_type_name: unixfilesystemresc_net: tutorub.renci.orgresc_def_path: /opt/irodsVault…

If there are data objects in the resource and the physical path names need to be changed:

> iadmin iadmin> modrescdatapaths demoResc /var/lib/irods/iRODS/Vault/ /opt/irodsVault/

“iadmin> help modrescdatapaths” for help

33

iRODS Installation – Change Defaults

3. Modify resource name (demoResc)– Best to do this before there are other users on the data grid> iadmin iadmin> lr

bundleRescdemoResc

iadmin> modresc demoResc name ubresc1

iadmin> lrbundleRescubresc1

– Edit the .irodsEnv file to rename default resource if necessary: change irodsDefResource 'demoResc’

to irodsDefResource ’ubresc1'

34

iRODS Installation – Change Defaults

3. Modify resource name (continued)– Must change the resource name used in any /etc/irods/core.re rules

that use the default name– acSetRescSchemeForCreate

{msiSetDefaultResc("demoResc","null"); }– acSetRescSchemeForRepl

{msiSetDefaultResc("demoResc","null"); }

change to– acSetRescSchemeForCreate

{msiSetDefaultResc(”ubresc1","null"); }– acSetRescSchemeForRepl {msiSetDefaultResc(”ubresc1","null"); }

35

iRODS Installation – Change Defaults

4. Introduce new resource on the same host> iadmin iadmin> lrbundleRescubresc1

iadmin> mkresc ubresc2 unixfilesystem tutorub.renci.org:/opt/irodsVault2

Creating resource:Name: "ubresc2"Type: "unixfilesystem"Host: "tutorub.renci.org"Path: "/opt/irodsVault2"Context: "”

iadmin> lrbundleRescubresc1ubresc2

Create the physical pathfirst and make sure Unixuser “irods” has write

permission on it

“iadmin> lt” is useful in choosingparameters to the mkresc command

36

Changing Default Environment

> more .irodsEnv# iRODS server host name:irodsHost 'tutorub.renci.org'# iRODS server port number:irodsPort 1247# Default storage resource name:irodsDefResource ’ubresc1'# Home directory in iRODS:irodsHome '/ubgrid/home/rods'# Current directory in iRODS:irodsCwd '/ubgrid/home/rods'# Account name:irodsUserName 'rods'# Zone:irodsZone 'ubgrid’ …

> ilsrescubresc1ubresc2

The Ubuntu Example

Default data grid name (zone name) was

changed to ubgrid;Default demoResc was

renamed to ubresc1 and its physical path was redefined;new resource ubresc2 on the same host was added (new

physical path).

Edit .irodsEnv file to accurately reflect

the changed environment.

37

Changing Default Environment

> more .irodsEnv# iRODS server host name:irodsHost 'tutoricat.renci.org'# iRODS server port number:irodsPort 1247# Default storage resource name:irodsDefResource ’resc1'# Home directory in iRODS:irodsHome '/centosgrid/home/rods'# Current directory in iRODS:irodsCwd '/centosgrid/home/rods'# Account name:irodsUserName 'rods'# Zone:irodsZone 'centosgrid’ …

> ilsrescresc1demoRescstateRescbundleResc

The CentOS Example

Default data grid name (zone name) was changed to centosgrid;default demoResc was

left as-is; new resources resc1 and stateResc on the

same host were added (new physical paths).bundleResc is createdwhen ibun is used.

Edit .irodsEnv file to accurately reflect

the changed environment.

38

Setting Up New Users

iadmin>lt user_type– rodsgroup – rodsadmin – rodsuser – domainadmin – groupadmin – storageadmin – rodscurators

• Use iadmin

• Two steps: mkuser and moduser (for a password)iadmin> mkuser user1 rodsuseriadmin> moduser user1 password *****

• Use iadmin to see what user types are possibleiadmin> lt

zone_typeuser_typedata_typeresc_typeaction_typerulexec_typeaccess_typeobject_typecoll_mapauth_scheme_type

iadmin> lt user_type

rodsgrouprodsadminrodsuserdomainadmingroupadminstorageadminrodscurators

Token List

Possible values

of token“user_type”

39

User Naming Convention

An iRODS user account is identified by the user name and the data grid/zone name:

– admin user on centosgrid: rods#centosgrid

– admin user on ubgrid: rods#ubgrid

– one of the rodsuser accounts on ubgrid: guest#ubgrid

40

Resource Server Installation on CentOS

• Install a non-IES iRODS server in order to add a resource to an existing data grid

• Host that will be running the iRODS server for this resource: tutorres.renci.org

1. Download irods-resource-4.0.0-64bit.deb from the download site2. Install the packages that iRODS depends on

> sudo yum install unixODBC perl authd> sudo yum install postgresql-odbc

3. PostgreSQL is set up on the IES, the iCAT-enabled server; don’t need it here on the non-IES resource server

continued …

Installing a non-IES Resource

41

Resource Server Installation on CentOS

4. Modify /etc/xinetd.d/auth locally to remove the -E from a command line argument for auth:

change [ server_args = -t60 --xerror --os -E ] to [ server_args = -t60 --xerror --os ]

5. Set the proper runlevel for authd> sudo /sbin/chkconfig --level=3 auth on

6. Restart xinetd > sudo /etc/init.d/xinetd restart

continued …

Installing a non-IES Resource

42

Resource Server Installation on CentOS

7. Open your firewall, if necessary– Add the following lines to your /etc/sysconfig/iptables

-A INPUT -m state --state NEW -m tcp -p tcp --dport 1247 -j ACCEPT -A INPUT -m state --state NEW -m tcp -p tcp --dport 20000:20199 -j ACCEPT -A INPUT -m state --state NEW -m udp -p udp --dport 20000:20199 -j ACCEPT

– Restart the firewall > sudo service iptables restart

8. Install the iRODS RPM > rpm -i irods-resource-4.0.0-64bit-centos6.rpm

9. In case of missing FUSE libraries, install those> sudo yum install fuse-libs and then install the RPM (repeat step 8)

continued …

Installing a non-IES Resource

43

Resource Server Installation on CentOS

10. As user irods, run the resource setup script> sudo su – irods> ./packaging/setup_resource.shAnswer the prompts for username and pw for iRODS admin user.> exit

Running this script will – create /etc/irods/server.config, /etc/irods/irods.config, and /etc/irods/core.re– connect to the iCAT-enabled server (the IES) running the data grid– create the /var/lib/irods/.irods/.irodsEnv file (for local service account irods)– bring up the iRODS resource server– add info to the log file /var/lib/irods/iRODS/server/log– add a resource to the existing data grid

continued …

Installing a non-IES Resource

44

Resource Server Installation on CentOS• Default resource name and location:

– <hostname>Resource– /var/lib/irods/iRODS/<hostname>ResourceVault

Example• Take a look at the new resource

> iadmin lr bundleResc resc1 stateResc demoResc tutorresResource

> iadmin lr tutorresResourceresc_id: 10020resc_name: tutorresResourcezone_name: centosgridresc_type_name: unixfilesystemresc_net: tutorres.renci.orgresc_def_path: /var/lib/irods/iRODS/tutorresResourceVault…

My resource host is

tutorres.renci.org,thus the defaultresource name

and path.See a listing of the data grid resources

See more info aboutthe new resource, in

particular the physical path

Installing a non-IES Resource

45

Resource Server Installation on CentOS

• Change resource name (before any data ingestion)> iadmin iadmin> modresc tutorresResource name resc2

• Change the physical path; my example path will be /opt/irodsVaultUnix: create the directory and set permissions for user “irods”– create the directory: > sudo mkdir /opt/irodsVault– set Unix user “irods” as owner: > sudo chown irods irodsVault– set group to “irods” for this directory: > sudo chgrp irods irodsVaultiRODS:– modify path: iadmin> modresc resc2 path /opt/irodsVaultcontinued …

Installing a non-IES Resource

46

Resource Server Installation on CentOS

Now take a look at the resources available on data grid centosgrid

> ilsresc bundleResc resc1 demoResc resc2 stateResc

> iadmin iadmin> lr resc2

resc_id: 10020resc_name: resc2zone_name: centosgridresc_type_name: unixfilesystemresc_net: tutorres.renci.orgresc_def_path: /opt/irodsVault …

Installing a non-IES Resource

From anywhere that you have your ~/.irods/.irodsEnv file and the icommands client, you have access to all the

resources of your data grid.

47

Resource Server Installation on Ubuntu

In this example, we’ll install another non-IES resource server on an Ubuntu host, to add yet another resource to centosgrid.

1. Download irods-resource-4.0.0-64bit.deb from the download site2. Go to the directory containing the .deb package and do dpkg...

> sudo dpkg -i irods-resource-4.0.0-64bit.deb

3. Use apt-get to install with all dependencies satisfied> sudo apt-get -f install

continued …

This creates theUnix account “irods”,

locally

Installing a non-IES Resource

48

Resource Server Installation on Ubuntu

4. As Unix user “irods” run the setup script> sudo su – irods> ./packaging/setup_resource.sh

5. Answer config questions about existing data grid this server is joining:

iCAT server's hostname or IP address: tutoricat.renci.orgiCAT server's port [1247]: iCAT server's ZoneName: centosgridiRODS admin username [rods]: rods

continued…

/var/lib/irods is the irods home directory

Installing a non-IES Resource

49

Resource Server Installation on Ubuntu

Running the setup script (./packaging/setup_resource.sh) will:– prompt for irods admin user and pw

– create /etc/irods/server.config

– connect to the iCAT-enabled server (the IES) running the data grid

– create the /var/lib/irods/.irods/.irodsEnv file (for Unix user irods)

– bring up the iRODS resource server

– add info to the log file /var/lib/irods/iRODS/server/log

continued…

Installing a non-IES Resource

50

Resource Server Installation on Ubuntu• Default resource name and location:

– <hostname>Resource– /var/lib/irods/iRODS/<hostname>ResourceVault

Example• Take a look at the new resource

> iadmin lr bundleResc resc1 stateResc demoResc resc2 tutorResource

> iadmin lr tutorResourceresc_id: 10020resc_name: tutorresResourcezone_name: centosgridresc_type_name: unixfilesystemresc_net: tutor.renci.orgresc_def_path: /var/lib/irods/iRODS/tutorResourceVault…

Resource host:tutor.renci.org

See a listing of the data grid resources

See more info aboutthe new resource, in

particular the physical path

Installing a non-IES Resource

bundleResc is created when ibun is used

(see the Getting Started

tutorial)

51

Resource Server Installation on Ubuntu

Change resource name (before any data ingestion)> iadmin iadmin> modresc tutorResource name resc3

> ilsresc bundleResc resc3 resc1 demoResc stateResc resc2

> iadmin iadmin> lr resc3

resc_id: 10023resc_name: resc3zone_name: centosgridresc_type_name: unixfilesystemresc_net: tutor.renci.orgresc_def_path:

/var/lib/irods/iRODS/tutorResourceVault

Installing a non-IES Resource

52

Resource Server Installation on Ubuntu

Change resource path(before any data ingestion)First create the directory and set permissions for Unix user “irods”

> sudo mkdir /opt/irodsVault> sudo chown irods irodsVault> sudo chgrp irods irodsVault> iadmin iadmin> modresc resc3 path /opt/irodsVault

iadmin> lr bundleResc demoResc stateResc resc1 resc2 resc3

iadmin>lr resc3 resc_id: 10023 resc_name: resc3 zone_name: centosgrid resc_type_name: unixfilesystem resc_net: tutor.renci.org resc_def_path: /opt/irodsVault …

Installing a non-IES Resource

53

Uninstalling the Resource Server

• To uninstall an iRODS resource server on Ubuntu> sudo dpkg --purge irods-resource

This deletes all files, including config files, and removes service account “irods”

• On CentOS> sudo rpm -e irods-resource

This will not remove the irods service account, so do> sudo /usr/sbin/userdel irods

54

Strict ACL Policy – acAclPolicy rule

• In /etc/irods/core.re, the rule acAclPolicy sets policy for strict access.

• Default setting is STRICT: acAclPolicy {msiAclPolicy("STRICT"); }- This disallows perusal of collections by users without read permission.

• By commenting out that rule and uncommenting the acAclPolicy rule that does NOT call the microservice to set the access control to strict, one can change the default behavior: acAclPolicy { } #acAclPolicy {msiAclPolicy("STRICT"); }

• This will allow free perusal of all collections by all users. Users will still not be able to view or download files to which they don’t have read permission

55

Federation Between Data GridsExample: Federate the centosgrid and ubgrid data grids

1. centosgrid acknowledges ubgrid> iadmin mkzone ubgrid remote tutorub.renci.org:1247

2. ubgrid acknowledges centosgrid> iadmin mkzone centosgrid remote tutoricat.renci.org:1247

3. Server-to-server authentication (the necessity for this can be disabled)In /etc/irods/server.config on each grid, define LocalZoneSID and then RemoteZoneSID.

continued…

56

Federation Between Data GridsExample: Federate the centosgrid and ubgrid data grids

3. Server-to-server authentication. Example: In /etc/irods/server.config

on centosgrid:LocalZoneSID terraRemoteZoneSID ubgrid-firma

In /etc/irods/server.configon ubgrid:LocalZoneSID firmaRemoteZoneSID centosgrid-terra

See the iRODS manual for info on scrambling these SIDs.

NB: These zone SIDs must now be set on ALL remote resourceson a zone to preserve server-to-server authentication

between the resources and their own data grids.

57

Federation Between Data Grids

Add remote users. First, see each grid’s usersubgrid:> iadmin lu rods#ubgrid

4. centosgrid adds remote user: > iadmin mkuser rods#ubgrid rodsuser

5. ubgrid adds remote user: > iadmin mkuser guest#centosgrid rodsuser

centosgrid:> iadmin lu

rods#centosgrid leesa#centosgrid guest#centosgrid

NB: rods#ubgrid is not

an admin accounton centosgrid

58

Federation Between Data GridsNow see each grid’s users…

ubgrid:> iadmin lu rods#ubgrid guest#centosgrid

centosgrid:> iadmin lu

rods#centosgrid leesa#centosgrid guest#centosgrid rods#ubgrid

59

Federation Between Data GridsUsers with accounts on remote, federated data grids: • authenticate to their “home” data grid and then inherit access to the

remote grid • have home collections on the remote data grid and ownership (write)

permissions there• can be granted read/write access to other collections just like any user• NB: the remote home collection is of the form:

/<remote_zone>/home/<user_name>#<local_zone>– So to do an “icd” or to write to your remote collection, remember to

append your local zone name to the collection name

– Example: User “guest” logs on to his home zone, centosgrid, and does an “iput” to his collection on ubgrid:

iput –R ubresc1 testfile /ubgrid/home/guest#centosgrid/testfile

60

Federation Between Data Grids

• To write to a remote federated data grid, users must know the resources available on the remote grid:

From centosgrid, to see the resources on ubgrid: > ilsresc –z ubgrid ubresc1 ubresc2

• User guest#centosgrid now wants to write to his home collection on ubgrid– User guest does a simple iinit and authenticates to his home data grid,

centosgrid> icd /ubgrid/home/guest#centosgrid> iput –R ubresc1 myfile> ils -L/ubgrid/home/guest#centosgrid: guest 0 ubresc1 1883 2014-05-13.21:31 & myfile ubresc1 generic /opt/irodsVault/home/guest#centosgrid/myfile

Writing across data grids

61

Removing/deleting Data or Resources

• “irm /zone/home/user/file1” moves file1 to /zone/trash/user/file1Not physically removed from disk

• “irm –f /zone/home/user/file1” physically deletes file1

• When administrator wants to remove a resource, it must be empty– If files are in the trash directory, resource is not empty

• To delete old users’ files for removing a resource– Admin user can use ichmod –M in admin mode, take ownership of

files, and delete them physically

62

Administrative Rights

• -M option for some icommands

• Some rules and queries are restricted to admin users

• Strict ACL exceptions for admins

63

ireg – the administrator’s side

Get data into iRODS without making an additional copy or moving it

Example: Directory /projects/state-data contains state LiDAR data that we now want in an iRODS repository… without moving or copying it

1. /projects/state-data is mounted on the iRODS server host

2. Data admin sets up existing directory as an iRODS resource:> iadmin mkresc stateResc unixfilesystem tutorres.renci.org:/projects/state-data

3. Register existing data into iRODS iCAT> ireg -C -R stateResc /projects/state-data /centosgrid/home/state-data

Register incoming files rigorously OR modify a directory only through iRODS once it has been registered to keep the iCAT consistent with the directory.

(-f option for picking up unregistered files)

64

S3 Resources – Cloud Management

See https://www.irods.org/index.php/S3_Resource

1. Set up an Amazon S3 resource– http://aws.amazon.com/s3/– You will need both the Access Key ID and the Secret Access Key

2. Download and build the libs3 library:

http://libs3.ischo.com.s3.amazonaws.com/index.html

65

S3 Resources – Cloud Management

3. Edit iRODS/config/config.mk– Uncomment the line: AMAZON_S3=1– Define the s3 libraries header directories, for example:

S3_LIB_DIR=/usr/libS3_HDR_DIR=/usr/include

4. Add path to the S3 library to the LD_LIBRARY_PATH environment variable:

set LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:/usr/lib

5. Rebuild the server./irodsctl istop./irodssetup (or gmake and then ./irodsctl istart)

66

6. Set up authentication to your Amazon resource– In server/config, use the file s3Auth.template as the template for the

s3Auth file– cp s3Auth.template s3Auth – Edit s3Auth as indicated in template file: add S3_ACCESS_KEY_ID

and S3_SECRET_ACCESS_KEY that you got from Amazon

7. Create an S3 compound resource> iadmin> mkresc amazonResc s3 compound irods00.lab.nsc.liu.se /snicVault> atrg s3Group s3Resc create resource group> atrg s3Group comp523Resc add resource of class “cache”

Configuring an Amazon S3 Resource

Path in s3storage

67

Cloud Resource• Admin creates the S3 resource - see the S3 resource and

group: iadmin>lr

bundleReschttpRescsnicRescamazonResc

iadmin>lrgs3GrouphttpGroup

• Any user can ingest and access data to the S3 resource(unless your own policy forbids it)> iput -f -K -R s3Resc irods-intro.pptx

• Cloud data is now managed by iRODS

> ilsreschttpRescsnicRescamazonRescs3Group (resource group)httpGroup (resource group)

68

Using the Cloud Resource

iput -R amazonResc iplant-file > ils

/snicZone/home/leesa: iplant-file C- /snicZone/home/leesa/slides

> ils -L/snicZone/home/leesa: leesa 0 snicResc 14 2012-09-24.19:49 & iplant-

file /home/irods7800/Vault/home/leesa/iplant-file s3Group leesa 1 amazonResc 14 2012-09-24.19:49 &

iplant-file /snicVault/home/leesa/iplant-file s3Group C- /snicZone/home/leesa/slides

69

Using the Cloud Resource

70

• Drivers support connections to external data

• These microservice plug-ins come with iRODS – two microservice drivers for each protocol (get & put)

• Instantiated through a compound resource

• Symbolic links implemented for http, slink, and Z39.50

• Admin users can implement new drivers: see How to Create a New MSO Type athttp://wiki.irods.org/index.php/How_to_Create_a_New_MSO_Type

MSO: Microservice ObjectsSupporting realizable objects

71

Symbolic Links to an http Source – the admin’s side

• Requires a compound coordinating resource with two child resources - one cache and one archive

• Set up the compound resourceiadmin> mkresc msoResc compound

• Set up the archive resource, of type msoiadmin> mkresc httpResc mso tutorub.renci.org:/fakeVault

• We will use an existing resource as the cache child resource, so now set up the parent-child relationships

iadmin> addchildtoresc msoResc ubresc2 cacheiadmin> addchildtoresc msoResc httpResc archive

Hostname is

necessary here;

physical path is

a dummy argument.

No Hostname for coordinating resource

72

Symbolic Links to an http Source – the administrator’s side• iadmin>lr msoResc• resc_id: 10035• resc_name: msoResc• zone_name: ubgrid• resc_type_name: compound• resc_net: EMPTY_RESC_HOST• resc_def_path: EMPTY_RESC_PATH• free_space: • free_space_ts 1969-12-31.19:00:00• resc_info: • r_comment: • resc_status: • create_ts 2014-05-22.16:58:35• modify_ts 2014-05-22.17:00:58• resc_children: ubresc2{cache};httpResc{archive}• resc_context: • resc_parent: • resc_objcount: 0

73

Symbolic Links to an http Source – the administrator’s side• By default, path checking is on, disallowing a rodsuser from putting

realizable objects into the mso resource. The user’s side:> ireg -D mso -R httpResc "//http://people.renci.org/~leesa/irods4.0/irods4.0-intro.pptx" /ubgrid/home/leesa/slides/irods4.0-intro.pptx

ERROR: regUtil: reg error for /ubgrid/home/leesa/slides/irods4.0-intro.pptx, status = -129000 status = -129000 PATH_REG_NOT_ALLOWED• By setting

• [ADD info here]

74

Symbolic Links to an http Source – the user’s side• User’s side:

> ireg -D mso -R httpResc "//http://people.renci.org/~leesa/irods4.0/irods4.0-intro.pptx" /ubgrid/home/leesa/slides/irods4.0-intro.pptx

ERROR: regUtil: reg error for /ubgrid/home/leesa/slides/irods4.0-intro.pptx, status = -129000 status = -129000 PATH_REG_NOT_ALLOWED• Admin side:

> ireg -D mso -R httpResc "//http://people.renci.org/~leesa/irods4.0/irods4.0-intro.pptx" /ubgrid/home/rods/slides/irods4.0-intro.pptx> ils slides/ubgrid/home/rods/slides: irods4.0-intro.pptx