Integration of the ICAT APIat

Institut Laue-Langevin (ILL)(Grenoble, France)

Tuesday, August 25, 2009 Jean-François Perrin

ICAT developper Workshop (25/26 August 2009)

Agenda

• Overview (ILL - Project)

• Data(file) import (ILL data import into ICAT)

• User Interface (web-based user Frontend)

• Conclusions (Problems, Questions)

IT Service ILL

• Besides two other IT services at ILL: – SCI (Instrument Control) and CS (Computing for science)

• Responsibilities (groups): – Infrastructure (network and phone)– DB – Usersupport– Developpement – Administration

• ICAT Team (Developpement group): – Holger Gebhard– Najor Cruz-Cruz (since July)

ILL-ICAT-Project

• Current state at ILL (IDA application): – text and script based system, – user must know how to retrieve date (not a search)

• 3 steps to integrate ICAT at ILL– replace existing (IDA) functionalities– public, more comfortable access for a larger group of users– Extend meta-data base and search functionalities

• Practical point of viewInstallation and exploration data-import GUI (Web App)

Data Import 1• General:

– Since 1973 all ILL data are available online– about 40 active instruments (60 overall) – Filesystem organised by cycle – instrument

stored on NFS (sometimes CIFS)– Total number of files: 29.145.595 (148 cycles)– Average size: 229 kb max ~ 40 mb (compressed)

• ILL formats: – Historical ILL Formats (ascii) – Nexus (2 Instruments)– edf (proprietary binary format)

• Impacts for ICAT integration at ILL relevant parameters are specific to (groups) of instrument(s). Need for expert advice

Data Import 2

Example: Import first cycle of 2006 for instrument d22 No relation to proposals IDA replacement !

Analyse directory levelAnalyse file level

Map files to dataset

Integrate found•Data files•Data sets •Investigationsinto Database

…/061/d22/

015673.Z

…

021354.Z

…/061/d22/

015673.Z…

015713.Z

….

020967.Z… 021354.Z

Read parameters and metadata

One specific parser class per (group) of instrument(s)

Data Import 3

• DATASET

• DATAFILE• DATAFILE_PARAMETER

• INVESTIGATION

• ICAT_AUTHORIZATION: – (user_id = any, role=downloader)

• PARAMETER– (ex. wavelength, pressure, voltage)

•DATAFILE_FORMAT•ILL Nexus, ILL ASCII

•INSTRUMENT (# 110, 61 activ)

•FACILITY_CYCLE (#142)

•INVESTIGATION_TYPE

•DATASET_STATUS (complete)•DATASET_TYPE(experiment_raw)

•USER_ROLES (some testusers)

Used (filled) tables:

ILL User Interface

Available in a stand-alone application and/or integrated into intra-/internet

As described in documentation: Compile WSDL togenerate remote proxy and classes for business delegates

Call remote proxy methods to interact with ICAT API

Feedback

• Tests and developments done so far worked fine for us

• ICAT Installation: straightforward – Database Scripts, ICAT API– Test with a dummy prototype

• Certificate handling and user authentication during installation

• No performance critical tests

ILL’s open issues

ICAT integration: • Mapping between data files and proposal data

no proper notion of "ICAT investigation" at ILL • Specimen data (and others) are stored with proposal• Control of data format and parameter variety

General• ILL Data Policy

– data ownership – delay until public availability

• Unification of data file formats– Competence and responsibility– Complexity

Some open issues

• Search by parameter in ICAT

• Future prospect: Opening ICAT’s Authentication mechanism to ILL’s SSO/LDAP

• Future prospect: Search between ICAT facilities repositories

Thanks

Comments?Questions?

Technical envorinment

• Hardware and Database– Virtual Server (Suse SLES 10.2) for ICAT– Oracle 10g (Cluster RAC) as DB

• Development Environment:– Java 1.6, – Glassfish 1.4, – Ant 1.7.1

• ICAT version: 3.3.3

Question details searchable parameters

Parameter table:‘Wavelength’ , ‘Å’, ‘Amstrong’ , Y, Y, N, Y, Y, ‘Desc Wavelenght’, Y, 999, GEBHARD, 31/07/09 02:20, 31/07/09 02:20,

‘NAJOR’, Y, N

Also: Voltage and Pressure

Join to tables: data_file_parameter and data_set_parameter

Known searches: • SearchByAdvanced• SearchByKeywords• SearchByRunNumber• SearchByUserId• SearchByUserSurName• SearchDatasetsBySample

Advanced Search Fields private java.lang.String datafileName; private java.util.Calendar dateRangeEnd; private java.util.Calendar dateRangeStart; private java.lang.String experimentNumber; private java.lang.Long grantId; private java.lang.String[] instruments; private java.lang.String investigationAbstract; private uk.icat3.client.InvestigationInclude investigationInclude; private java.lang.String investigationName; private java.lang.String investigationType; private java.lang.String[] investigators; private java.lang.String[] keywords; private java.lang.Double runEnd; private java.lang.Double runStart; private java.lang.String sampleName; private java.lang.String visitId;

ILL-ICAT-Project 2

From the practical point of view• ICAT installation and exploration

• Data Import (File metadata to ICAT, ~ 10 of 40 instruments)

• Webbased User Interface (Simple version, planned for end of the year)

• Extension of metadata and userinterfaceAutomated, continious data import (all intruments, all cycles)

Data File exampleRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRR 068762 0 4 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA 80 1 Inst User L.C. Date Time D19 tf sam 14-May-09 10:16:11 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA 80 1 Title Scantypebackgr om-mono IIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIII 31 4 nvers ntype kctrl manip nbang nkmes npdone jcode ipara ianal imode itgv iregul ivolt naxe npstart ilast1 isa flgkif ih ik nbsqs nb_det nbdata icdesc1 icdesc2 icdesc3 icdesc4 icdesc5 icdesc6

4 2 4 8 1 21 21 1 1 0 0 0 1 0 1 0 0 0 0 0 0 0 1 163840 8 0 0 0 0 0FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF 50 10 H (Hmin) K (Kmin) L (Lmin) phi chi omega 2theta (gamma) psi ub(1,1) ub(1,2) ub(1,3) ub(2,1) ub(2,2) ub(2,3) ub(3,1) ub(3,2) ub(3,3) wavelength dmonochromator danalyser energy Hmax Kmax Lmax DeltaH-0.10000000E+01 0.50000000E+00 0.50000000E+00 -0.49880001E+02 0.15721400E+03 0.23701000E+02 0.64004000E+02 0.00000000E+00 0.12238280E+00 0.53997743E-01 -0.64825200E-02 -0.53893240E-01 0.12252440E+00 0.37250000E-02 0.69433160E-02 -0.74291900E-03 0.14317410E+00 0.24220001E+01 0.10000000E+01 0.10000000E+01 0.00000000E+00 0.00000000E+00 0.00000000E+00 0.00000000E+00 0.00000000E+00