practical on special jobs and wmproxy
DESCRIPTION
Practical on special jobs and WMProxy. GILDA TEAM INFN EELA tutorial Madrid 24 February 2006. Outline of exercises. MPI on gLite DAG job Job Collection Parametric jobs. MPI exercise. Create the following “ mpiglite.jdl ” file on your home. [Type = "Job"; JobType = "MPICH"; - PowerPoint PPT PresentationTRANSCRIPT
FP6−2004−Infrastructures−6-SSA-026409
www.eu-eela.org
E-infrastructure shared between Europe and Latin America
Practical on special jobsand WMProxyGILDA TEAMINFNEELA tutorialMadrid 24 February 2006
Madrid, EELATutorial, 24 February 2006 2FP6−2004−Infrastructures−6-SSA-026409
E-infrastructure shared between Europe and Latin America
Outline of exercises
• MPI on gLite
• DAG job
• Job Collection
• Parametric jobs
Madrid, EELATutorial, 24 February 2006 3FP6−2004−Infrastructures−6-SSA-026409
E-infrastructure shared between Europe and Latin America
Create the following “mpiglite.jdl” file on your home.
[ Type = "Job";
JobType = "MPICH";
Executable = “cpi";
NodeNumber = 2;
StdOutput = “cpi.out"; StdError = “cpi.err";
InputSandbox = {"cpi"};
OutputSandbox = {“cip.err",“cip.out"};
Requirements = Member("GLITE-1.4", other.GlueHostApplicationSoftwareRunTimeEnvironment);
RetryCount = 0;
]
MPI exercise
Madrid, EELATutorial, 24 February 2006 4FP6−2004−Infrastructures−6-SSA-026409
E-infrastructure shared between Europe and Latin America
MPI submission
[glite-tutor] /home/vardizzo > glite-job-submit -o id mpi_glite.jdl
Selected Virtual Organisation name (from proxy certificate extension): gildaConnecting to host glite-rb.ct.infn.it, port 7772Logging to host glite-rb.ct.infn.it, port 9002
========== glite-job-submit Success ========================== The job has been successfully submitted to the Network Server. Use glite-job-status command to check job current status. Your job identifier
is:
- https://glite-rb.ct.infn.it:9000/bsrbbzbcXZWSzU3iUYlm6g
The job identifier has been saved in the following file: /home/vardizzo/id==========================================================
Madrid, EELATutorial, 24 February 2006 5FP6−2004−Infrastructures−6-SSA-026409
E-infrastructure shared between Europe and Latin America
MPI status and output
Query the status of the job using the following command:
[glite-tutor] /home/vardizzo > glite-job-status -i id
…………………………………………….
When the status of the job is “DONE”, you can retrieve output with
the following command:
[glite-tutor] /home/vardizzo > glite-job-output -i id
……………………………………………
Madrid, EELATutorial, 24 February 2006 6FP6−2004−Infrastructures−6-SSA-026409
E-infrastructure shared between Europe and Latin America
WMProxy : submission & monitoring
• In order to submit job with WMProxy, it’s mandatatory credentials delegation
• The submission/monitoring commands are slightly different, but the most of the “old” options are supported
glite-wms-job-delegate-proxy -d del_ID_01
glite-wms-job-submit -d del_ID_01 collection.jdl
glite-wms-job-output \
https://glite-rb.ct.infn.it:9000/LHIIGaCVdl7Olm
sz0jpI_g
Madrid, EELATutorial, 24 February 2006 7FP6−2004−Infrastructures−6-SSA-026409
E-infrastructure shared between Europe and Latin America
DAG job
• DAG job is a set of jobs where the input, output, or execution of one or more jobs depends on one or more other ones
• Dependencies are represented through Directed Acyclic Graphs, where the nodes are graphs, and the edges identify the dependencies
nodeA
nodeB nodeC NodeF
nodeD
Madrid, EELATutorial, 24 February 2006 8FP6−2004−Infrastructures−6-SSA-026409
E-infrastructure shared between Europe and Latin America
DAG jdl[ type = "dag"; max_nodes_running = 4; nodes = [ nodeA = [ file ="nodes/nodeA.jdl" ; ]; nodeB = [ file ="nodes/nodeB.jdl" ; ]; nodeC = [ file ="nodes/nodeC.jdl" ; ]; nodeF = [ file ="nodes/nodeF.jdl"; ]; dependencies = { {nodeA, nodeB}, {nodeA, nodeC}, { {nodeB,nodeC}, nodeF } } ];;]
Node description could be done also
here, instead of using separate file
Madrid, EELATutorial, 24 February 2006 9FP6−2004−Infrastructures−6-SSA-026409
E-infrastructure shared between Europe and Latin America
Job Collection
• Job collection is a set of independent jobs that user wants to submit and monitor as a single request
• Jobs of a collection are submitted as DAG nodes, without dependencies
• JDL is a list of classad, which describes the subjobs
[ Type = "collection"; VirtualOrganisation = “gilda";
nodes = { [ <job descr 1 >], [ <job descr 2 >], …};
]
Madrid, EELATutorial, 24 February 2006 10
FP6−2004−Infrastructures−6-SSA-026409
E-infrastructure shared between Europe and Latin America
gridftp basic use
• Upload TO a gridftp server
• Download FROM a gridftp server
globus-url-copy file:<local absolute path> gsiftp://<gsiftp server>/<remote absolute path>
Ex: globus-url-copy file:/home/madrid01/file.txt gsiftp://glite-rb.ct.infn.it/tmp/file.txt
globus-url-copy gsiftp://<gsiftp server>/<remote absolute path>
file:<local absolute path>
Ex: globus-url-copy gsiftp://glite-rb.ct.infn.it/tmp/file.txt file:/home/madrid01/file.txt
Madrid, EELATutorial, 24 February 2006 11
FP6−2004−Infrastructures−6-SSA-026409
E-infrastructure shared between Europe and Latin America
Job collection examples
[ Type = "collection"; InputSandbox = {"start_hostname.sh"}; RetryCount = 0; nodes={ [ Executable = "/bin/sh"; StdOutput = "host.out"; StdError = "host.err"; InputSandbox = root.InputSandbox; OutputSandbox = {"host.err","host.out"}; OutputSandboxURI={"gsiftp://glite-rb.ct.infn.it:2811/tmp/host.out",
"host.err"}; Arguments = "start_hostname.sh"; ],[ Executable = "/bin/sh"; StdOutput = "test.out"; StdError = "test.err"; InputSandbox={"starter.sh","gsiftp://glite-rb.ct.infn.it:2811/tmp/
t01.txt"}; OutputSandbox = {"test.err","test.out"}; Arguments = "starter.sh"; ],[ file = "hostname.jdl"; ] };]
Madrid, EELATutorial, 24 February 2006 12
FP6−2004−Infrastructures−6-SSA-026409
E-infrastructure shared between Europe and Latin America
Parametric Job
• A parametric job is a job where one or more of its attributes are parametrized
• Value of attributes vary according to parameter
• Job monitoring / managing is always done through an unique jobID, as if the job was single (see submission of collection
[ JobType = "Parametric"; Executable = “/bin/echo";
Arguments = “PARAM”; #InputSandbox =
“input_PARAM_.txt"; StdOutput =
"myoutput_PARAM_.txt"; StdError =
"myerror_PARAM_.txt"; Parameters = 2500; ParameterStep = 100; ParameterStart = 1000;
OutputSandbox = {“myoutput_PARAM_.txt”}; ]
Madrid, EELATutorial, 24 February 2006 13
FP6−2004−Infrastructures−6-SSA-026409
E-infrastructure shared between Europe and Latin America
Parametric job / 2
• Parameter can be also a list of string• InputSandbox (if present) has to be coherent with
parameters[ui-test] /home/giorgio/param > cat param2.jdl[ JobType = "Parametric";
Executable = “/bin/cat"; Arguments = “input_PARAM_.txt”;
InputSandbox = "input_PARAM_.txt";
StdOutput = "myoutput_PARAM_.txt";
StdError = "myerror_PARAM_.txt"; Parameters = {earth,moon,mars}; OutputSandbox =
{“myoutput_PARAM_.txt”};][ui-test] /home/giorgio/param > ls
inputEARTH.txt inputMARS.txt inputMOON.txt param2.jdl
Madrid, EELATutorial, 24 February 2006 14
FP6−2004−Infrastructures−6-SSA-026409
E-infrastructure shared between Europe and Latin America
References
• JDL attributes specification for WM proxy– https://edms.cern.ch/document/590869/1
• WMProxy quickstart– http://egee-jra1-wm.mi.infn.it/egee-jra1-wm/wmproxy_client_quickstart.shtml
• WMS user guides– https://edms.cern.ch/document/572489/1