gangatasks automating of job submission wit athena/athenamc in ganga sw week, 24.10.2007 johannes...

GangaTasksGangaTasksAutomating of job Automating of job

submission wit submission wit Athena/AthenaMC in GangaAthena/AthenaMC in Ganga

sw weeksw week, 24.10.2007, 24.10.2007

johannes ebke, johannes ebke, tariq mahmoudtariq mahmoud

tariq mahmoud atlas sw week, cern, 24102007

automating of job submission in Athena/AthenaMC

outline

I. introduction, concept of tasksII. starting and submitting a taskIII. automating of job submissionIV. job interconnectionV. conclusion

tariq mahmoud atlas sw week, cern, 24102007

automating of job submission in Athena/AthenaMC I. motivation

aim: automate job submission.job chaining (interconnection)

the user gives few parameters …

and gets RESULTS

tariq mahmoud atlas sw week, cern, 24102007

automating of job submission in Athena/AthenaMC I. concept

taskfew parameters: task parameters: float, name, number of events etc.

process parameters: datasets, process name, geometry etc.

input/output specifications Athena version

AbstractJobgenerated automatically: name, task, status, necessary, prerequisites, run limit, number of attempts and attempt jobs, …

AbstractJobAbstractJob

jobjob

job Ganga jobs

tariq mahmoud atlas sw week, cern, 24102007

automating of job submission in Athena/AthenaMC I. GangaTasks

MCTask

MCJobMCJobMCJob

AnaTask

AnaJob

Task

AnaJobAnaJob

repository/tools.data

TaskList(GangaObject)

GPI.tasks

tariq mahmoud atlas sw week, cern, 24102007

automating of job submission in Athena/AthenaMC II: starting and submitting a new

task

running numbering on all tasks created by the same user

if the task is 'new', make some settings permanent and set the status to 'running‘

tariq mahmoud atlas sw week, cern, 24102007

automating of job submission in Athena/AthenaMC III: automating

Task() AnaTask()AnaJob()

II: starting and submitting a new task

auto submitting info resubmitting

tariq mahmoud atlas sw week, cern, 24102007

automating of job submission in Athena/AthenaMC III: automating

s=AnaTask()s.name="PythiaZmumu"

s.float= 10s.application_option_file=‘$HOME/athena/…./run/AnalysisSkeleton_topOptions.py's.inputdata_dataset= "trig1_misal1_csc11.005145.PythiaZmumu.recon.AOD.v12000601_tid005999"s.requirements_sites=['LRZ', 'FZK', 'LYON‘]s.outputdata_outputdata=[“AnaPythiaZmumu.root"]s.submit()

tariq mahmoud atlas sw week, cern, 24102007

automating of job submission in Athena/AthenaMC III: info and more

tariq mahmoud atlas sw week, cern, 24102007

automating of job submission in Athena/AthenaMC III: resubmitting

tariq mahmoud atlas sw week, cern, 24102007

automating of job submission in Athena/AthenaMC IV: jobs interconnections

Task() MCTask()MCJob()MCJob()MCJob()

II: starting and submitting a new task III:

• auto submitting• info• resubmitting

chaining task steps interconnection,

prerequisites

automating of job submission in Athena/AthenaMC

IV: chaining task stepsmc.total_events = 200;mc.events_per_job={"evgen": 50, "simul": 5, "recon":10}

1 3 5 7 9

1 3 5 7 9

1 3 5 7 9

1 3 5 7 9

1 3 5 972 4 6 108

1 3 5 972 4 6 108

1 3 5 972 4 6 108

1 3 5 972 4 6 108

2 3 41

tariq mahmoud atlas sw week, cern, 24102007

evgen:i-0

simul:i-j

recon:i-k

mc.total_events = 200 mc.events_per_job= {"evgen": 50, "simul": 5, "recon":10} mc.float=10

AbstractJob.prerequisites()

tariq mahmoud atlas sw week, cern, 24102007

automating of job submission in Athena/AthenaMC IV: interconnections,

prerequisites (1)

Task.float: does NOT apply for subjobs

tariq mahmoud atlas sw week, cern, 24102007

automating of job submission in Athena/AthenaMC

AnaTask: optimise sites

CE= ‘lcg-lrz-ce.lrz-muenchen.de:2119/jobmanager-sge-atlas’requirements_sites= ['LRZ', 'FZK', 'LYON']optimize_sites=Falseinputdata_match_ce_all =Trueinputdata_min_num_files=100sites_optimized=[]

tariq mahmoud atlas sw week, cern, 24102007

automating of job submission in Athena/AthenaMC

V. conclusion

GangaTask package: automates job management (small scale production system)works with Ganga Athena and Ganga AthenaMC.wiki in progress: https://twiki.cern.ch/twiki/bin/view/Atlas/GangaTasks

To do: 1. gather the largest number of files possible of an

in-complete dataset.2. test everything on Lxplus “and elsewhere”.3. soon available in one of the next in Ganga

releases

the user gives

few parameters

… and gets

RESULTS