practical on special jobs and wmproxy

FP6−2004−Infrastructures−6-SSA-026409

www.eu-eela.org

E-infrastructure shared between Europe and Latin America

Practical on special jobsand WMProxyGILDA TEAMINFNEELA tutorialMadrid 24 February 2006

Madrid, EELATutorial, 24 February 2006 2FP6−2004−Infrastructures−6-SSA-026409

E-infrastructure shared between Europe and Latin America

Outline of exercises

• MPI on gLite

• DAG job

• Job Collection

• Parametric jobs

Madrid, EELATutorial, 24 February 2006 3FP6−2004−Infrastructures−6-SSA-026409

E-infrastructure shared between Europe and Latin America

Create the following “mpiglite.jdl” file on your home.

[ Type = "Job";

JobType = "MPICH";

Executable = “cpi";

NodeNumber = 2;

StdOutput = “cpi.out"; StdError = “cpi.err";

InputSandbox = {"cpi"};

OutputSandbox = {“cip.err",“cip.out"};

Requirements = Member("GLITE-1.4", other.GlueHostApplicationSoftwareRunTimeEnvironment);

RetryCount = 0;

]

MPI exercise

Madrid, EELATutorial, 24 February 2006 4FP6−2004−Infrastructures−6-SSA-026409

E-infrastructure shared between Europe and Latin America

MPI submission

[glite-tutor] /home/vardizzo > glite-job-submit -o id mpi_glite.jdl

Selected Virtual Organisation name (from proxy certificate extension): gildaConnecting to host glite-rb.ct.infn.it, port 7772Logging to host glite-rb.ct.infn.it, port 9002

========== glite-job-submit Success ========================== The job has been successfully submitted to the Network Server. Use glite-job-status command to check job current status. Your job identifier

is:

- https://glite-rb.ct.infn.it:9000/bsrbbzbcXZWSzU3iUYlm6g

The job identifier has been saved in the following file: /home/vardizzo/id==========================================================

Madrid, EELATutorial, 24 February 2006 5FP6−2004−Infrastructures−6-SSA-026409

E-infrastructure shared between Europe and Latin America

MPI status and output

Query the status of the job using the following command:

[glite-tutor] /home/vardizzo > glite-job-status -i id

…………………………………………….

When the status of the job is “DONE”, you can retrieve output with

the following command:

[glite-tutor] /home/vardizzo > glite-job-output -i id

……………………………………………

Madrid, EELATutorial, 24 February 2006 6FP6−2004−Infrastructures−6-SSA-026409

E-infrastructure shared between Europe and Latin America

WMProxy : submission & monitoring

• In order to submit job with WMProxy, it’s mandatatory credentials delegation

• The submission/monitoring commands are slightly different, but the most of the “old” options are supported

glite-wms-job-delegate-proxy -d del_ID_01

glite-wms-job-submit -d del_ID_01 collection.jdl

glite-wms-job-output \

https://glite-rb.ct.infn.it:9000/LHIIGaCVdl7Olm

sz0jpI_g

Madrid, EELATutorial, 24 February 2006 7FP6−2004−Infrastructures−6-SSA-026409

E-infrastructure shared between Europe and Latin America

DAG job

• DAG job is a set of jobs where the input, output, or execution of one or more jobs depends on one or more other ones

• Dependencies are represented through Directed Acyclic Graphs, where the nodes are graphs, and the edges identify the dependencies

nodeA

nodeB nodeC NodeF

nodeD

Madrid, EELATutorial, 24 February 2006 8FP6−2004−Infrastructures−6-SSA-026409

E-infrastructure shared between Europe and Latin America

DAG jdl[ type = "dag"; max_nodes_running = 4; nodes = [ nodeA = [ file ="nodes/nodeA.jdl" ; ]; nodeB = [ file ="nodes/nodeB.jdl" ; ]; nodeC = [ file ="nodes/nodeC.jdl" ; ]; nodeF = [ file ="nodes/nodeF.jdl"; ]; dependencies = { {nodeA, nodeB}, {nodeA, nodeC}, { {nodeB,nodeC}, nodeF } } ];;]

Node description could be done also

here, instead of using separate file

Madrid, EELATutorial, 24 February 2006 9FP6−2004−Infrastructures−6-SSA-026409

E-infrastructure shared between Europe and Latin America

Job Collection

• Job collection is a set of independent jobs that user wants to submit and monitor as a single request

• Jobs of a collection are submitted as DAG nodes, without dependencies

• JDL is a list of classad, which describes the subjobs

[ Type = "collection"; VirtualOrganisation = “gilda";

nodes = { [ <job descr 1 >], [ <job descr 2 >], …};

]

Madrid, EELATutorial, 24 February 2006 10

FP6−2004−Infrastructures−6-SSA-026409

E-infrastructure shared between Europe and Latin America

gridftp basic use

• Upload TO a gridftp server

• Download FROM a gridftp server

globus-url-copy file:<local absolute path> gsiftp://<gsiftp server>/<remote absolute path>

Ex: globus-url-copy file:/home/madrid01/file.txt gsiftp://glite-rb.ct.infn.it/tmp/file.txt

globus-url-copy gsiftp://<gsiftp server>/<remote absolute path>

file:<local absolute path>

Ex: globus-url-copy gsiftp://glite-rb.ct.infn.it/tmp/file.txt file:/home/madrid01/file.txt

Madrid, EELATutorial, 24 February 2006 11

FP6−2004−Infrastructures−6-SSA-026409

E-infrastructure shared between Europe and Latin America

Job collection examples

[ Type = "collection"; InputSandbox = {"start_hostname.sh"}; RetryCount = 0; nodes={ [ Executable = "/bin/sh"; StdOutput = "host.out"; StdError = "host.err"; InputSandbox = root.InputSandbox; OutputSandbox = {"host.err","host.out"}; OutputSandboxURI={"gsiftp://glite-rb.ct.infn.it:2811/tmp/host.out",

"host.err"}; Arguments = "start_hostname.sh"; ],[ Executable = "/bin/sh"; StdOutput = "test.out"; StdError = "test.err"; InputSandbox={"starter.sh","gsiftp://glite-rb.ct.infn.it:2811/tmp/

t01.txt"}; OutputSandbox = {"test.err","test.out"}; Arguments = "starter.sh"; ],[ file = "hostname.jdl"; ] };]

Madrid, EELATutorial, 24 February 2006 12

FP6−2004−Infrastructures−6-SSA-026409

E-infrastructure shared between Europe and Latin America

Parametric Job

• A parametric job is a job where one or more of its attributes are parametrized

• Value of attributes vary according to parameter

• Job monitoring / managing is always done through an unique jobID, as if the job was single (see submission of collection

[ JobType = "Parametric"; Executable = “/bin/echo";

Arguments = “PARAM”; #InputSandbox =

“input_PARAM_.txt"; StdOutput =

"myoutput_PARAM_.txt"; StdError =

"myerror_PARAM_.txt"; Parameters = 2500; ParameterStep = 100; ParameterStart = 1000;

OutputSandbox = {“myoutput_PARAM_.txt”}; ]

Madrid, EELATutorial, 24 February 2006 13

FP6−2004−Infrastructures−6-SSA-026409

E-infrastructure shared between Europe and Latin America

Parametric job / 2

• Parameter can be also a list of string• InputSandbox (if present) has to be coherent with

parameters[ui-test] /home/giorgio/param > cat param2.jdl[ JobType = "Parametric";

Executable = “/bin/cat"; Arguments = “input_PARAM_.txt”;

InputSandbox = "input_PARAM_.txt";

StdOutput = "myoutput_PARAM_.txt";

StdError = "myerror_PARAM_.txt"; Parameters = {earth,moon,mars}; OutputSandbox =

{“myoutput_PARAM_.txt”};][ui-test] /home/giorgio/param > ls

inputEARTH.txt inputMARS.txt inputMOON.txt param2.jdl

Madrid, EELATutorial, 24 February 2006 14

FP6−2004−Infrastructures−6-SSA-026409

E-infrastructure shared between Europe and Latin America

References

• JDL attributes specification for WM proxy– https://edms.cern.ch/document/590869/1

• WMProxy quickstart– http://egee-jra1-wm.mi.infn.it/egee-jra1-wm/wmproxy_client_quickstart.shtml

• WMS user guides– https://edms.cern.ch/document/572489/1