![Page 1: High Volume Batch Submission System for Earthquake Engineering ( Batchsubmit )](https://reader035.vdocument.in/reader035/viewer/2022070503/5681638d550346895dd48116/html5/thumbnails/1.jpg)
High Volume Batch Submission System
for Earthquake Engineering (Batchsubmit)
ByAnup Mohan([email protected])
Thomas HackerGregory Rodgers
![Page 2: High Volume Batch Submission System for Earthquake Engineering ( Batchsubmit )](https://reader035.vdocument.in/reader035/viewer/2022070503/5681638d550346895dd48116/html5/thumbnails/2.jpg)
Outline
Overview of NEES Need for Batchsubmit Batchsubmit Features Batchsubmit Examples Batchsubmit Architecture Project Warehouse Lessons Learned
![Page 3: High Volume Batch Submission System for Earthquake Engineering ( Batchsubmit )](https://reader035.vdocument.in/reader035/viewer/2022070503/5681638d550346895dd48116/html5/thumbnails/3.jpg)
NSF Network for Earthquake Engineering Simulation (NEES)
Safer buildings and civil infrastructure are needed to reduce damage and loss from earthquakes and tsunamis
To facilitate research to improve seismic design of buildings and civil infrastructure, the National Science Foundation established NEES
NEES Objectives Develop a national, multi-user, research
infrastructure to support research and innovation in earthquake and tsunami loss reduction
Create an educated workforce in hazard mitigation Conduct broader outreach and lifelong learning
activities
![Page 4: High Volume Batch Submission System for Earthquake Engineering ( Batchsubmit )](https://reader035.vdocument.in/reader035/viewer/2022070503/5681638d550346895dd48116/html5/thumbnails/4.jpg)
Vision for NEES
Facilitate access to the world's best integrated network of state-of-the art physical simulation facilities
Build a cyber-enabled community that shares ideas, data, and computational tools and models.
Promote education and training for the next generation of researchers and practitioners.
Cultivate partnerships with other organizations to disseminate research results, leverage cyberinfrastructure, and reduce risk by transferring results into practice.
![Page 5: High Volume Batch Submission System for Earthquake Engineering ( Batchsubmit )](https://reader035.vdocument.in/reader035/viewer/2022070503/5681638d550346895dd48116/html5/thumbnails/5.jpg)
NEES Research Facilities
NEES has a broad set of experimental facilities Each type of equipment produces unique data Located at 14 sites across the United States
Shake Table, Tsunami Wave Basin Large-Scale Testing Facilities Centrifuge, Field and Mobile Facilities Large-Displacement Facility Cyberinfrastructure
![Page 6: High Volume Batch Submission System for Earthquake Engineering ( Batchsubmit )](https://reader035.vdocument.in/reader035/viewer/2022070503/5681638d550346895dd48116/html5/thumbnails/6.jpg)
University of CaliforniaSanta Barbara
University of CaliforniaSan Diego
University of CaliforniaLos Angeles
University of CaliforniaDavis
Lehigh University
Rensselaer Polytechnic Institute0
Cornell University
University of Buffalo
University of MinnesotaUniversity of Illinois- Urbana
Oregon State University
University of CaliforniaBerkeley
University of NevadaReno
University of TexasAustin
https://www.nees.org
![Page 7: High Volume Batch Submission System for Earthquake Engineering ( Batchsubmit )](https://reader035.vdocument.in/reader035/viewer/2022070503/5681638d550346895dd48116/html5/thumbnails/7.jpg)
NEEShub (www.nees.org)
![Page 8: High Volume Batch Submission System for Earthquake Engineering ( Batchsubmit )](https://reader035.vdocument.in/reader035/viewer/2022070503/5681638d550346895dd48116/html5/thumbnails/8.jpg)
21st Century Science
Theory and Experiment Computational Simulation
Third leg of science Past 50 years or so
Data Fourth “leg” of science Researchers are flooded with data Tremendous quantity and multiple scales of data Difficult to collect, store, and manage How can we distill meaningful knowledge from data?
![Page 9: High Volume Batch Submission System for Earthquake Engineering ( Batchsubmit )](https://reader035.vdocument.in/reader035/viewer/2022070503/5681638d550346895dd48116/html5/thumbnails/9.jpg)
Data is the 4th Paradigm
Producing an avalanche of high resolution digital data All (or most) of the data needs to be accessible over a
long period of time Much of the data is not reproducible
Example – NEES project Structure or sample destroyed through testing Very expensive to rebuild for more tests
![Page 10: High Volume Batch Submission System for Earthquake Engineering ( Batchsubmit )](https://reader035.vdocument.in/reader035/viewer/2022070503/5681638d550346895dd48116/html5/thumbnails/10.jpg)
Need for High Performance Computing(HPC) Simulation
Earthquake engineering problems: Involves intense computation on large volumes of data Will take days to complete simulation on normal computers
Earthquake engineering problems on an HPC resource takes hours to complete
Using HPC resources for solving computationally intensive problems results in huge time savings
Different HPC resources include: XSEDE: Kraken, Stampede Purdue: Hansen, Carter Open Science Grid (OSG)
![Page 11: High Volume Batch Submission System for Earthquake Engineering ( Batchsubmit )](https://reader035.vdocument.in/reader035/viewer/2022070503/5681638d550346895dd48116/html5/thumbnails/11.jpg)
Why Batchsubmit?
Different steps of job submission on an HPC resource includes: Login to the remote venue and copy all input files Ensure that the simulation program is available on the remote
venue and copy program to the venue if necessary Prepare venue specific script file specifying parameters for job
submission Submit the job to the correct job class and wait for the result Copy the results to a local system for analysis and visualization
![Page 12: High Volume Batch Submission System for Earthquake Engineering ( Batchsubmit )](https://reader035.vdocument.in/reader035/viewer/2022070503/5681638d550346895dd48116/html5/thumbnails/12.jpg)
Why Batchsubmit?
Repeating this process for multiple simulations is tedious and time consuming
Batchsubmit automatically performs all the steps for submitting a job in an HPC resource
User can submit jobs using batchsubmit to multiple venues Batchsubmit will intimate the user when job finishes execution
and all results will be available in user job directory Batchsubmit makes job submission easier and results in huge
time savings
![Page 13: High Volume Batch Submission System for Earthquake Engineering ( Batchsubmit )](https://reader035.vdocument.in/reader035/viewer/2022070503/5681638d550346895dd48116/html5/thumbnails/13.jpg)
Batchsubmit Features
Simple command line interface and GUI interface Easy to do parallel processing Asynchronous job submission Select HPC resource(venue) to run the job Send executables to the venue, if required
![Page 14: High Volume Batch Submission System for Earthquake Engineering ( Batchsubmit )](https://reader035.vdocument.in/reader035/viewer/2022070503/5681638d550346895dd48116/html5/thumbnails/14.jpg)
Batchsubmit Features
Automatic retrieval of results Email notification once results are available Monitor job status Monitor queue traffic at different venues Cancel a job
![Page 15: High Volume Batch Submission System for Earthquake Engineering ( Batchsubmit )](https://reader035.vdocument.in/reader035/viewer/2022070503/5681638d550346895dd48116/html5/thumbnails/15.jpg)
Batchsubmit Examples
batchsubmit date batchsubmit --venue carter --ncpus 16
OpenSeesMP /apps/opensees/NEEShubExamples/SmallMP/Example.tcl
batchsubmit --venue stampede --ncpus 64 --appdir /apps/share64/opensees/stampede --rcopyindir OpenSeesMP /LargeMP/Example.tcl
![Page 16: High Volume Batch Submission System for Earthquake Engineering ( Batchsubmit )](https://reader035.vdocument.in/reader035/viewer/2022070503/5681638d550346895dd48116/html5/thumbnails/16.jpg)
Architecture
![Page 17: High Volume Batch Submission System for Earthquake Engineering ( Batchsubmit )](https://reader035.vdocument.in/reader035/viewer/2022070503/5681638d550346895dd48116/html5/thumbnails/17.jpg)
Architecture: Workspace
![Page 18: High Volume Batch Submission System for Earthquake Engineering ( Batchsubmit )](https://reader035.vdocument.in/reader035/viewer/2022070503/5681638d550346895dd48116/html5/thumbnails/18.jpg)
Architecture: Batchsubmit Client
JOB STATUS = PRESUBMIT
![Page 19: High Volume Batch Submission System for Earthquake Engineering ( Batchsubmit )](https://reader035.vdocument.in/reader035/viewer/2022070503/5681638d550346895dd48116/html5/thumbnails/19.jpg)
Architecture: Batchsubmit
Server
JOB STATUS = SUBMIT
![Page 20: High Volume Batch Submission System for Earthquake Engineering ( Batchsubmit )](https://reader035.vdocument.in/reader035/viewer/2022070503/5681638d550346895dd48116/html5/thumbnails/20.jpg)
Architecture: HPC Resource
JOB STATUS = COMPLETE
![Page 21: High Volume Batch Submission System for Earthquake Engineering ( Batchsubmit )](https://reader035.vdocument.in/reader035/viewer/2022070503/5681638d550346895dd48116/html5/thumbnails/21.jpg)
Architecture: Batchsubmit
Server
JOB STATUS = COMPLETE
![Page 22: High Volume Batch Submission System for Earthquake Engineering ( Batchsubmit )](https://reader035.vdocument.in/reader035/viewer/2022070503/5681638d550346895dd48116/html5/thumbnails/22.jpg)
Architecture: Workspace
JOB STATUS = COMPLETE
![Page 23: High Volume Batch Submission System for Earthquake Engineering ( Batchsubmit )](https://reader035.vdocument.in/reader035/viewer/2022070503/5681638d550346895dd48116/html5/thumbnails/23.jpg)
Accessing Batchsubmit
Workspace Tool Linux Desktop in the browser
OpenSees Laboratory Parallel Job Submission mode uses Batchsubmit
![Page 24: High Volume Batch Submission System for Earthquake Engineering ( Batchsubmit )](https://reader035.vdocument.in/reader035/viewer/2022070503/5681638d550346895dd48116/html5/thumbnails/24.jpg)
Project Warehouse
NEES online data repository Allows researchers to upload, archive, and disseminate data from
their physical, cyber, and hybrid experiments and simulations. Provides user with tools to organize data into projects,
experiments, trials, and results. The data in the Project Warehouse can be cited and shared with
researchers across the world, or only within a research group. Facility to curate the data in the repository to ensure the
long-term preservation of valuable NEES data.
![Page 25: High Volume Batch Submission System for Earthquake Engineering ( Batchsubmit )](https://reader035.vdocument.in/reader035/viewer/2022070503/5681638d550346895dd48116/html5/thumbnails/25.jpg)
Usage StatisticsYear Month Submitted Number
of Jobs2012 January 1453
February 1525March 1466April 1191May 92461June 18519July 1849August 4553September 3341October 6352November 2712December 3484
2013 January 5260February 2898
![Page 26: High Volume Batch Submission System for Earthquake Engineering ( Batchsubmit )](https://reader035.vdocument.in/reader035/viewer/2022070503/5681638d550346895dd48116/html5/thumbnails/26.jpg)
Lessons Learned
Need for community gateway Selection of correct venue Specifying correct walltime limit Not exceeding the venue specific parameter
limits Need for data backup
![Page 27: High Volume Batch Submission System for Earthquake Engineering ( Batchsubmit )](https://reader035.vdocument.in/reader035/viewer/2022070503/5681638d550346895dd48116/html5/thumbnails/27.jpg)
Lessons Learned
Asynchronous method, hence email notifications are necessary
Listing available venues Showing how busy are the queues of different
venues Need for proper documentation and tutorials
![Page 28: High Volume Batch Submission System for Earthquake Engineering ( Batchsubmit )](https://reader035.vdocument.in/reader035/viewer/2022070503/5681638d550346895dd48116/html5/thumbnails/28.jpg)
Summary
Batchsubmit makes parallel job submission easy for the user
Batchsubmit is Asynchronous – does not require the session to be alive till job finishes execution
Batchsubmit provides easy access to multiple supercomputers.
Batchsubmit system could be easily extended for use by other science and engineering domains.