egee-ii infso-ri-031688 enabling grids for e-science command line grid programming spiros spirou...
DESCRIPTION
Enabling Grids for E-sciencE EGEE-II INFSO-RI Roadmap Application definition Environment configuration Information discovery Data management Program definition Program execution SummaryTRANSCRIPT
![Page 1: EGEE-II INFSO-RI-031688 Enabling Grids for E-sciencE Command Line Grid Programming Spiros Spirou Greek Application Support Team NCSR “Demokritos”](https://reader035.vdocument.in/reader035/viewer/2022062317/5a4d1b237f8b9ab059995f9f/html5/thumbnails/1.jpg)
EGEE-II INFSO-RI-031688
Enabling Grids for E-sciencE
www.eu-egee.org
Command Line Grid Programming
Spiros SpirouGreek Application Support TeamNCSR “Demokritos”
![Page 2: EGEE-II INFSO-RI-031688 Enabling Grids for E-sciencE Command Line Grid Programming Spiros Spirou Greek Application Support Team NCSR “Demokritos”](https://reader035.vdocument.in/reader035/viewer/2022062317/5a4d1b237f8b9ab059995f9f/html5/thumbnails/2.jpg)
Enabling Grids for E-sciencE
EGEE-II INFSO-RI-031688
Overview
• Application definition• Environment configuration• Information discovery• Data management• Program definition• Program execution• Summary
![Page 3: EGEE-II INFSO-RI-031688 Enabling Grids for E-sciencE Command Line Grid Programming Spiros Spirou Greek Application Support Team NCSR “Demokritos”](https://reader035.vdocument.in/reader035/viewer/2022062317/5a4d1b237f8b9ab059995f9f/html5/thumbnails/3.jpg)
Enabling Grids for E-sciencE
EGEE-II INFSO-RI-031688
Roadmap
• Application definition• Environment configuration• Information discovery• Data management• Program definition• Program execution• Summary
![Page 4: EGEE-II INFSO-RI-031688 Enabling Grids for E-sciencE Command Line Grid Programming Spiros Spirou Greek Application Support Team NCSR “Demokritos”](https://reader035.vdocument.in/reader035/viewer/2022062317/5a4d1b237f8b9ab059995f9f/html5/thumbnails/4.jpg)
Enabling Grids for E-sciencE
EGEE-II INFSO-RI-031688
Application definition
originalimage
compressionalgorithm
compressedimage
Image Compression
![Page 5: EGEE-II INFSO-RI-031688 Enabling Grids for E-sciencE Command Line Grid Programming Spiros Spirou Greek Application Support Team NCSR “Demokritos”](https://reader035.vdocument.in/reader035/viewer/2022062317/5a4d1b237f8b9ab059995f9f/html5/thumbnails/5.jpg)
Enabling Grids for E-sciencE
EGEE-II INFSO-RI-031688
Application parallelization
datadecomposition
parallelprocessing
datacomposition
![Page 6: EGEE-II INFSO-RI-031688 Enabling Grids for E-sciencE Command Line Grid Programming Spiros Spirou Greek Application Support Team NCSR “Demokritos”](https://reader035.vdocument.in/reader035/viewer/2022062317/5a4d1b237f8b9ab059995f9f/html5/thumbnails/6.jpg)
Enabling Grids for E-sciencE
EGEE-II INFSO-RI-031688
Programming prerequisites
UI accountVO membership
1 2 3
CA certificate
You must already have…
![Page 7: EGEE-II INFSO-RI-031688 Enabling Grids for E-sciencE Command Line Grid Programming Spiros Spirou Greek Application Support Team NCSR “Demokritos”](https://reader035.vdocument.in/reader035/viewer/2022062317/5a4d1b237f8b9ab059995f9f/html5/thumbnails/7.jpg)
Enabling Grids for E-sciencE
EGEE-II INFSO-RI-031688
Roadmap
• Application definition• Environment configuration• Information discovery• Data management• Program definition• Program execution• Summary
![Page 8: EGEE-II INFSO-RI-031688 Enabling Grids for E-sciencE Command Line Grid Programming Spiros Spirou Greek Application Support Team NCSR “Demokritos”](https://reader035.vdocument.in/reader035/viewer/2022062317/5a4d1b237f8b9ab059995f9f/html5/thumbnails/8.jpg)
Enabling Grids for E-sciencE
EGEE-II INFSO-RI-031688
Virtual Organization Membership Service (VOMS)
Authentication
UI Grid
certificate proxycertificate
create(w/ password)
service 1
service 2
authenticate
VO, Groups, Roles
![Page 9: EGEE-II INFSO-RI-031688 Enabling Grids for E-sciencE Command Line Grid Programming Spiros Spirou Greek Application Support Team NCSR “Demokritos”](https://reader035.vdocument.in/reader035/viewer/2022062317/5a4d1b237f8b9ab059995f9f/html5/thumbnails/9.jpg)
Enabling Grids for E-sciencE
EGEE-II INFSO-RI-031688
VOMS commands
Command(voms-proxy-)
Description
init Creates VOMS proxy
info Prints VOMS proxy information
destroy Destroys VOMS proxy
list Lists VOMS server attributes
- gLite 3.0 User’s Guide- VOMS Core Services User’s Guide
![Page 10: EGEE-II INFSO-RI-031688 Enabling Grids for E-sciencE Command Line Grid Programming Spiros Spirou Greek Application Support Team NCSR “Demokritos”](https://reader035.vdocument.in/reader035/viewer/2022062317/5a4d1b237f8b9ab059995f9f/html5/thumbnails/10.jpg)
Enabling Grids for E-sciencE
EGEE-II INFSO-RI-031688
VOMS example
Contact VOMS server and create proxy certificate> voms-proxy-init –voms=see
Check proxy certificate> voms-proxy-info -all
![Page 11: EGEE-II INFSO-RI-031688 Enabling Grids for E-sciencE Command Line Grid Programming Spiros Spirou Greek Application Support Team NCSR “Demokritos”](https://reader035.vdocument.in/reader035/viewer/2022062317/5a4d1b237f8b9ab059995f9f/html5/thumbnails/11.jpg)
Enabling Grids for E-sciencE
EGEE-II INFSO-RI-031688
Roadmap
• Application definition• Environment configuration• Information discovery• Data management• Program definition• Program execution• Summary
![Page 12: EGEE-II INFSO-RI-031688 Enabling Grids for E-sciencE Command Line Grid Programming Spiros Spirou Greek Application Support Team NCSR “Demokritos”](https://reader035.vdocument.in/reader035/viewer/2022062317/5a4d1b237f8b9ab059995f9f/html5/thumbnails/12.jpg)
Enabling Grids for E-sciencE
EGEE-II INFSO-RI-031688
Berkeley DatabaseInformation Index
server (BDII)
Information service
UI Grid
query
user
ComputingElement (CE)
StorageElement (SE)
information
![Page 13: EGEE-II INFSO-RI-031688 Enabling Grids for E-sciencE Command Line Grid Programming Spiros Spirou Greek Application Support Team NCSR “Demokritos”](https://reader035.vdocument.in/reader035/viewer/2022062317/5a4d1b237f8b9ab059995f9f/html5/thumbnails/13.jpg)
Enabling Grids for E-sciencE
EGEE-II INFSO-RI-031688
Information service command
Command Description
lcg-infosites Prints Grid service information
Attributes
--vo VO name (mandatory)
--is BDII server name
![Page 14: EGEE-II INFSO-RI-031688 Enabling Grids for E-sciencE Command Line Grid Programming Spiros Spirou Greek Application Support Team NCSR “Demokritos”](https://reader035.vdocument.in/reader035/viewer/2022062317/5a4d1b237f8b9ab059995f9f/html5/thumbnails/14.jpg)
Enabling Grids for E-sciencE
EGEE-II INFSO-RI-031688
lcg-infosites options
Options Description
ce Prints CE names
se Prints SE names
closeSE Prints CE and “close” SE names
all Prints CE and SE names
lfc Prints File Catalog (LFC) name
- gLite 3.0 User’s Guide- lcg-infosites -help
![Page 15: EGEE-II INFSO-RI-031688 Enabling Grids for E-sciencE Command Line Grid Programming Spiros Spirou Greek Application Support Team NCSR “Demokritos”](https://reader035.vdocument.in/reader035/viewer/2022062317/5a4d1b237f8b9ab059995f9f/html5/thumbnails/15.jpg)
Enabling Grids for E-sciencE
EGEE-II INFSO-RI-031688
lcg-infosites example
Get CE and SE names for VO> lcg-infosites --vo see all
Get LFC name for VO> lcg-infosites --vo see lfc
![Page 16: EGEE-II INFSO-RI-031688 Enabling Grids for E-sciencE Command Line Grid Programming Spiros Spirou Greek Application Support Team NCSR “Demokritos”](https://reader035.vdocument.in/reader035/viewer/2022062317/5a4d1b237f8b9ab059995f9f/html5/thumbnails/16.jpg)
Enabling Grids for E-sciencE
EGEE-II INFSO-RI-031688
Roadmap
• Application definition• Environment configuration• Information discovery• Data management• Program definition• Program execution• Summary
![Page 17: EGEE-II INFSO-RI-031688 Enabling Grids for E-sciencE Command Line Grid Programming Spiros Spirou Greek Application Support Team NCSR “Demokritos”](https://reader035.vdocument.in/reader035/viewer/2022062317/5a4d1b237f8b9ab059995f9f/html5/thumbnails/17.jpg)
Enabling Grids for E-sciencE
EGEE-II INFSO-RI-031688
Grid file(ref: LFN)
Grid files
UI Grid
file
SE 1
File Catalog(LFC)
store
register
SE 2
store
register
![Page 18: EGEE-II INFSO-RI-031688 Enabling Grids for E-sciencE Command Line Grid Programming Spiros Spirou Greek Application Support Team NCSR “Demokritos”](https://reader035.vdocument.in/reader035/viewer/2022062317/5a4d1b237f8b9ab059995f9f/html5/thumbnails/18.jpg)
Enabling Grids for E-sciencE
EGEE-II INFSO-RI-031688
lcg-utils
Command(lcg-)
Description
cp Copies file from Grid to UI
cr Copies file from UI to Grid
rep Creates replica of file
lr Lists replicas of file
del Deletes file (and replicas)
- gLite 3.0 User’s Guide- lcg-utils man pages
![Page 19: EGEE-II INFSO-RI-031688 Enabling Grids for E-sciencE Command Line Grid Programming Spiros Spirou Greek Application Support Team NCSR “Demokritos”](https://reader035.vdocument.in/reader035/viewer/2022062317/5a4d1b237f8b9ab059995f9f/html5/thumbnails/19.jpg)
Enabling Grids for E-sciencE
EGEE-II INFSO-RI-031688
lcg-utils example
Upload data (sub-images) to Grid> lcg-cr --vo see -l lfn:/grid/see/subimg1
file:/users/johndoe/subimg1> lcg-cr --vo see -l lfn:/grid/see/subimg2
file:/users/johndoe/subimg2[…]
Locate data (sub-image1)> lcg-lr --vo see lfn:/grid/see/subimg1
Replicate data (sub-image1) to specific SE> lcg-rep --vo see -d se01.ariagni.hellasgrid.gr
lfn: /grid/see/subimg1
![Page 20: EGEE-II INFSO-RI-031688 Enabling Grids for E-sciencE Command Line Grid Programming Spiros Spirou Greek Application Support Team NCSR “Demokritos”](https://reader035.vdocument.in/reader035/viewer/2022062317/5a4d1b237f8b9ab059995f9f/html5/thumbnails/20.jpg)
Enabling Grids for E-sciencE
EGEE-II INFSO-RI-031688
Roadmap
• Application definition• Environment configuration• Information discovery• Data management• Program definition• Program execution• Summary
![Page 21: EGEE-II INFSO-RI-031688 Enabling Grids for E-sciencE Command Line Grid Programming Spiros Spirou Greek Application Support Team NCSR “Demokritos”](https://reader035.vdocument.in/reader035/viewer/2022062317/5a4d1b237f8b9ab059995f9f/html5/thumbnails/21.jpg)
Enabling Grids for E-sciencE
EGEE-II INFSO-RI-031688
Job Description Language (JDL)
Grid jobs
originalsub-image
compressionalgorithm
compressedsub-image
input data program output data
![Page 22: EGEE-II INFSO-RI-031688 Enabling Grids for E-sciencE Command Line Grid Programming Spiros Spirou Greek Application Support Team NCSR “Demokritos”](https://reader035.vdocument.in/reader035/viewer/2022062317/5a4d1b237f8b9ab059995f9f/html5/thumbnails/22.jpg)
Enabling Grids for E-sciencE
EGEE-II INFSO-RI-031688
JDL attributes (1/2)
Attribute Description
Executable Program to run on Grid
Arguments Arguments of program
InputSandbox Files to copy from UI to Grid
StdOutput File to save program stdout
StdError File to save program stderr
OutputSandbox Files to copy from Grid to UI
![Page 23: EGEE-II INFSO-RI-031688 Enabling Grids for E-sciencE Command Line Grid Programming Spiros Spirou Greek Application Support Team NCSR “Demokritos”](https://reader035.vdocument.in/reader035/viewer/2022062317/5a4d1b237f8b9ab059995f9f/html5/thumbnails/23.jpg)
Enabling Grids for E-sciencE
EGEE-II INFSO-RI-031688
JDL attributes (2/2)
Attribute Description
DataRequirements
InputData Grid files needed by program
DataCatalogType Type of catalog referencing Grid files
DataAccessProtocol Protocol for accessing Grid files
Requirements Program resource requirements
- gLite 3.0 User’s Guide- JDL Attributes Specification
![Page 24: EGEE-II INFSO-RI-031688 Enabling Grids for E-sciencE Command Line Grid Programming Spiros Spirou Greek Application Support Team NCSR “Demokritos”](https://reader035.vdocument.in/reader035/viewer/2022062317/5a4d1b237f8b9ab059995f9f/html5/thumbnails/24.jpg)
Enabling Grids for E-sciencE
EGEE-II INFSO-RI-031688
JDL Example
[Executable = “compress.sh”;Arguments = “subimg1”;InputSandbox = {“file:///users/johndoe/compress.sh”};StdOutput = “std.out”;StdError = “std.err”;OutputSandbox = {“std.out”, “std.err”, subimg1”};DataRequirements = {[InputData = {“lfn:/grid/cms/subimg1”}; DataCatalogType = “DLI”;]};DataAccessProtocol = {“gsiftp”};
]
![Page 25: EGEE-II INFSO-RI-031688 Enabling Grids for E-sciencE Command Line Grid Programming Spiros Spirou Greek Application Support Team NCSR “Demokritos”](https://reader035.vdocument.in/reader035/viewer/2022062317/5a4d1b237f8b9ab059995f9f/html5/thumbnails/25.jpg)
Enabling Grids for E-sciencE
EGEE-II INFSO-RI-031688
Roadmap
• Application definition• Environment configuration• Information discovery• Data management• Program definition• Program execution• Summary
![Page 26: EGEE-II INFSO-RI-031688 Enabling Grids for E-sciencE Command Line Grid Programming Spiros Spirou Greek Application Support Team NCSR “Demokritos”](https://reader035.vdocument.in/reader035/viewer/2022062317/5a4d1b237f8b9ab059995f9f/html5/thumbnails/26.jpg)
Enabling Grids for E-sciencE
EGEE-II INFSO-RI-031688
Job execution
WorkloadManagement
System (WMS)(WMProxy interface)
UI Grid
job
user
SE
CE
LFC
VOMS
BDII
![Page 27: EGEE-II INFSO-RI-031688 Enabling Grids for E-sciencE Command Line Grid Programming Spiros Spirou Greek Application Support Team NCSR “Demokritos”](https://reader035.vdocument.in/reader035/viewer/2022062317/5a4d1b237f8b9ab059995f9f/html5/thumbnails/27.jpg)
Enabling Grids for E-sciencE
EGEE-II INFSO-RI-031688
Job management commands
Command(glite-wms-job-)
Description
delegate-proxy Delegates VOMS proxy to WMProxy
submit Submits a job
cancel Cancels a submitted job
output Copies OutputSandbox to UI
glite-job-status Prints submitted job status
- gLite 3.0 User’s Guide- WMProxy Service User’s Guide
![Page 28: EGEE-II INFSO-RI-031688 Enabling Grids for E-sciencE Command Line Grid Programming Spiros Spirou Greek Application Support Team NCSR “Demokritos”](https://reader035.vdocument.in/reader035/viewer/2022062317/5a4d1b237f8b9ab059995f9f/html5/thumbnails/28.jpg)
Enabling Grids for E-sciencE
EGEE-II INFSO-RI-031688
Job management example
Delegate proxy and set delegation id > glite-wms-job-delegate-proxy -d del_id
Submit job and save job id> glite-wms-job-submit -d del_id -o job_id compress.jdl
Check job status> glite-job-status -i job_id
Get job output when done> glite-wms-job-output -i job_id
![Page 29: EGEE-II INFSO-RI-031688 Enabling Grids for E-sciencE Command Line Grid Programming Spiros Spirou Greek Application Support Team NCSR “Demokritos”](https://reader035.vdocument.in/reader035/viewer/2022062317/5a4d1b237f8b9ab059995f9f/html5/thumbnails/29.jpg)
Enabling Grids for E-sciencE
EGEE-II INFSO-RI-031688
Roadmap
• Application definition• Environment configuration• Information discovery• Data management• Program definition• Program execution• Summary
![Page 30: EGEE-II INFSO-RI-031688 Enabling Grids for E-sciencE Command Line Grid Programming Spiros Spirou Greek Application Support Team NCSR “Demokritos”](https://reader035.vdocument.in/reader035/viewer/2022062317/5a4d1b237f8b9ab059995f9f/html5/thumbnails/30.jpg)
Enabling Grids for E-sciencE
EGEE-II INFSO-RI-031688
Summary
• Parallelize application
• Authenticate with VOMS
• Get information with lcg-infosites
• Upload data with lcg-utils
• Describe application with JDL
• Execute application with glite-wms-job-*
The Grid is fast, simple, and free. So, use it!
![Page 31: EGEE-II INFSO-RI-031688 Enabling Grids for E-sciencE Command Line Grid Programming Spiros Spirou Greek Application Support Team NCSR “Demokritos”](https://reader035.vdocument.in/reader035/viewer/2022062317/5a4d1b237f8b9ab059995f9f/html5/thumbnails/31.jpg)
EGEE-II INFSO-RI-031688
Enabling Grids for E-sciencE
www.eu-egee.org
Questions?
Spiros SpirouGreek Application Support TeamNCSR “Demokritos”