the janus computing environment · software support rc expertise less general unsupported software...
TRANSCRIPT
![Page 1: The JANUS Computing Environment · Software support RC expertise less general Unsupported software Installation Consulting Advice on installing your software and any dependancies](https://reader033.vdocument.in/reader033/viewer/2022042910/5f3e40a40378bb00dc64f221/html5/thumbnails/1.jpg)
Research ComputingUNIVERSITY OF COLORADO
The JANUS Computing Environment
Monte [email protected]
Thursday, June 21, 12
![Page 2: The JANUS Computing Environment · Software support RC expertise less general Unsupported software Installation Consulting Advice on installing your software and any dependancies](https://reader033.vdocument.in/reader033/viewer/2022042910/5f3e40a40378bb00dc64f221/html5/thumbnails/2.jpg)
What is JANUS?November, 2011
1,368 Compute nodes
16,416 processors
~ 20 GB of available space
~ 800 TB of storage
2.8Ghz Intel Westmere
TFLOPS is a rate of execution, trillions of floating point operations per second
Thursday, June 21, 12
![Page 3: The JANUS Computing Environment · Software support RC expertise less general Unsupported software Installation Consulting Advice on installing your software and any dependancies](https://reader033.vdocument.in/reader033/viewer/2022042910/5f3e40a40378bb00dc64f221/html5/thumbnails/3.jpg)
NUMA Architecture
Resource Management and
“queues”
Parallel file systems
Different architectures
Explicit environment
Lots of ways to do something...
Thursday, June 21, 12
![Page 4: The JANUS Computing Environment · Software support RC expertise less general Unsupported software Installation Consulting Advice on installing your software and any dependancies](https://reader033.vdocument.in/reader033/viewer/2022042910/5f3e40a40378bb00dc64f221/html5/thumbnails/4.jpg)
Online resourceswww.rc.colorado.edu
Thursday, June 21, 12
![Page 5: The JANUS Computing Environment · Software support RC expertise less general Unsupported software Installation Consulting Advice on installing your software and any dependancies](https://reader033.vdocument.in/reader033/viewer/2022042910/5f3e40a40378bb00dc64f221/html5/thumbnails/5.jpg)
OverviewAccess
Login, file system, data transfer
Software
Supported software, dotkits, building software
Resource Management
Queues, Moab, and Torque
Running Jobs
Single-core, load-balanced, MPI, OpenMP
Questions
Thursday, June 21, 12
![Page 6: The JANUS Computing Environment · Software support RC expertise less general Unsupported software Installation Consulting Advice on installing your software and any dependancies](https://reader033.vdocument.in/reader033/viewer/2022042910/5f3e40a40378bb00dc64f221/html5/thumbnails/6.jpg)
Access
Thursday, June 21, 12
![Page 7: The JANUS Computing Environment · Software support RC expertise less general Unsupported software Installation Consulting Advice on installing your software and any dependancies](https://reader033.vdocument.in/reader033/viewer/2022042910/5f3e40a40378bb00dc64f221/html5/thumbnails/7.jpg)
Login Proceduressh <username>@login.rc.colorado.edu
Password: Yubikeys or Cryptocards
Thursday, June 21, 12
![Page 8: The JANUS Computing Environment · Software support RC expertise less general Unsupported software Installation Consulting Advice on installing your software and any dependancies](https://reader033.vdocument.in/reader033/viewer/2022042910/5f3e40a40378bb00dc64f221/html5/thumbnails/8.jpg)
RC FilesystemHome directory
/home/<user_name>
2 Gb, Network File System (NFS)
Project space
/projects/<user_name>
250 Gb, NFS
Scratch space
/lustre/janus_scratch/<user_name>
No quota, no backup
Lustre file system
Build software here
Run software here
Thursday, June 21, 12
![Page 9: The JANUS Computing Environment · Software support RC expertise less general Unsupported software Installation Consulting Advice on installing your software and any dependancies](https://reader033.vdocument.in/reader033/viewer/2022042910/5f3e40a40378bb00dc64f221/html5/thumbnails/9.jpg)
SnapshotDid you accidentally remove a file or directory?
$HOME/.snapshot/hourly.[0-12]$HOME/.snapshot/nightly.[0-6]$HOME/.snapshot/weekly.[0-7]
Example
rm $HOME/bugreport.csh cp $HOME/.snapshot/weekly.0/bugreport.csh $HOME
Where?
$HOME/.snapshot/projects/<user_name>/.snapshot
Thursday, June 21, 12
![Page 10: The JANUS Computing Environment · Software support RC expertise less general Unsupported software Installation Consulting Advice on installing your software and any dependancies](https://reader033.vdocument.in/reader033/viewer/2022042910/5f3e40a40378bb00dc64f221/html5/thumbnails/10.jpg)
LustreScalable, POSIX-compliant parallel file system designed for large, distributed-memory systems
Object Storage Targets (OST)
Store user file data
Object Storage Servers (OSS)
Control I/O access and handling network request
Metadata Target (MDT)
Stores filenames, directories, permissions and file layout
Metadata Server (MDS)
Assigns storage locations associated with each file in order to direct file I/O requests to the correct set of OST
Thursday, June 21, 12
![Page 11: The JANUS Computing Environment · Software support RC expertise less general Unsupported software Installation Consulting Advice on installing your software and any dependancies](https://reader033.vdocument.in/reader033/viewer/2022042910/5f3e40a40378bb00dc64f221/html5/thumbnails/11.jpg)
IB
MDS MDT
OSS OST
Metadata server (MDS) and target (MDT)
Object storage server (OSS) and target (OST)
Thursday, June 21, 12
![Page 12: The JANUS Computing Environment · Software support RC expertise less general Unsupported software Installation Consulting Advice on installing your software and any dependancies](https://reader033.vdocument.in/reader033/viewer/2022042910/5f3e40a40378bb00dc64f221/html5/thumbnails/12.jpg)
IB
MDT
OSS OST
File Access
MDS
Compute node requests storage location
Compute node then interacts directly with OST
Thursday, June 21, 12
![Page 13: The JANUS Computing Environment · Software support RC expertise less general Unsupported software Installation Consulting Advice on installing your software and any dependancies](https://reader033.vdocument.in/reader033/viewer/2022042910/5f3e40a40378bb00dc64f221/html5/thumbnails/13.jpg)
StripingFile - contiguous sequence of bytes
Key feature: Lustre file system can distribute these segments multiple OSTs using a technique called file striping.
A file is said to be striped when its contiguous sequence of bytes is separated into small chunks, or stripes, so that read and write operations can access multiple OSTs concurrently.
/file
/file
Thursday, June 21, 12
![Page 14: The JANUS Computing Environment · Software support RC expertise less general Unsupported software Installation Consulting Advice on installing your software and any dependancies](https://reader033.vdocument.in/reader033/viewer/2022042910/5f3e40a40378bb00dc64f221/html5/thumbnails/14.jpg)
File I/O
/file /file1 /file2 /filen
Serial File-per-process
Shared file
/file
Collective Buffering: Not currently supported on JANUS
Thursday, June 21, 12
![Page 15: The JANUS Computing Environment · Software support RC expertise less general Unsupported software Installation Consulting Advice on installing your software and any dependancies](https://reader033.vdocument.in/reader033/viewer/2022042910/5f3e40a40378bb00dc64f221/html5/thumbnails/15.jpg)
Single processor
stripe count
writ
e sp
eed
(Mb/
s)
0
200
400
600
800
●
● ● ●●
●
●
●
●
●●
●
●
●
1 2 4 8 15 30 60
Transfer size● 1 mb● 32 mb
Thursday, June 21, 12
![Page 16: The JANUS Computing Environment · Software support RC expertise less general Unsupported software Installation Consulting Advice on installing your software and any dependancies](https://reader033.vdocument.in/reader033/viewer/2022042910/5f3e40a40378bb00dc64f221/html5/thumbnails/16.jpg)
File per processor
processors (files)
writ
e sp
eed
(Mb/
s)
0
2000
4000
6000
8000
10000
12000
●●
● ●●
●
●
●
●
●
●
●
1 2 4 8 16 32 64 128 256 512 1024 2048
Thursday, June 21, 12
![Page 17: The JANUS Computing Environment · Software support RC expertise less general Unsupported software Installation Consulting Advice on installing your software and any dependancies](https://reader033.vdocument.in/reader033/viewer/2022042910/5f3e40a40378bb00dc64f221/html5/thumbnails/17.jpg)
Shared-file with striping
processors (files)
writ
e sp
eed
(Mb/
s)
1000
2000
3000
4000
5000
6000
7000
● ● ● ● ●
●
●
●
●
●
1 2 4 8 16 32 64 128 256 1024
Thursday, June 21, 12
![Page 18: The JANUS Computing Environment · Software support RC expertise less general Unsupported software Installation Consulting Advice on installing your software and any dependancies](https://reader033.vdocument.in/reader033/viewer/2022042910/5f3e40a40378bb00dc64f221/html5/thumbnails/18.jpg)
Examples
bash-janus> mkdir temp_dir
bash-janus> lfs setstripe -c 3 temp_dir
bash-janus> touch temp_dir/temp_file
bash-janus> lfs getstripe temp_dir
temp_dirstripe_count: 3 stripe_size: 33554432 stripe_offset: -1
temp_dir/temp_filelmm_stripe_count: 3lmm_stripe_size: 33554432lmm_stripe_offset: 18 obdidx objid objid group 18 12787913 0xc320c9 0 7 12863377 0xc44791 0 23 12496893 0xbeaffd 0
Thursday, June 21, 12
![Page 19: The JANUS Computing Environment · Software support RC expertise less general Unsupported software Installation Consulting Advice on installing your software and any dependancies](https://reader033.vdocument.in/reader033/viewer/2022042910/5f3e40a40378bb00dc64f221/html5/thumbnails/19.jpg)
Data transferhttps://www.rc.colorado.edu/crcdocs/file-transfer
Grid FTP
GridFTP is a high-performance, secure, reliable data transfer protocol optimized for high-bandwidth wide-area networks
Globus Online
Large file transfers with “drag and drop archiving” to move data between its long-time archival storage and compute systems
Utilities
scp, sftp, rsync
Good for small files
Thursday, June 21, 12
![Page 20: The JANUS Computing Environment · Software support RC expertise less general Unsupported software Installation Consulting Advice on installing your software and any dependancies](https://reader033.vdocument.in/reader033/viewer/2022042910/5f3e40a40378bb00dc64f221/html5/thumbnails/20.jpg)
Access tipsControl Sockets
One-time passwords make multiple terminal sessions and file transer painful.
mkdir -p ~/.ssh/socketscat >> ~/.ssh/config << EOFHost login.rc* ControlMaster auto ControlPath ~/.ssh/sockets/%r@%h:%pEOF
Mount Drive
http://macfusionapp.org/
Symbolic links
/project, /scratch
Thursday, June 21, 12
![Page 21: The JANUS Computing Environment · Software support RC expertise less general Unsupported software Installation Consulting Advice on installing your software and any dependancies](https://reader033.vdocument.in/reader033/viewer/2022042910/5f3e40a40378bb00dc64f221/html5/thumbnails/21.jpg)
Software
Thursday, June 21, 12
![Page 22: The JANUS Computing Environment · Software support RC expertise less general Unsupported software Installation Consulting Advice on installing your software and any dependancies](https://reader033.vdocument.in/reader033/viewer/2022042910/5f3e40a40378bb00dc64f221/html5/thumbnails/22.jpg)
Software support
RC expertise
less general
Unsupported software
Installation
Consulting
Advice on installing your software
and any dependancies
Supported software
select state-of-the-art software
Installation, verification, and training
user
exp
ertis
e
Thursday, June 21, 12
![Page 23: The JANUS Computing Environment · Software support RC expertise less general Unsupported software Installation Consulting Advice on installing your software and any dependancies](https://reader033.vdocument.in/reader033/viewer/2022042910/5f3e40a40378bb00dc64f221/html5/thumbnails/23.jpg)
Environment
To run an executable, you need to know where it is.
/opt/openmpi/1.4.4/bin/mpicxx
/opt/mpitch2/1.5a2/bin/mpicxx
Which one does the command which mpicxx use?
PATH
What about libraries?
/opt/openmpi/1.4.4/lib/libmpi.so
/opt/mpitch2/1.5a2/lib/libmpi.so
LD_LIBRARY_PATH
Thursday, June 21, 12
![Page 24: The JANUS Computing Environment · Software support RC expertise less general Unsupported software Installation Consulting Advice on installing your software and any dependancies](https://reader033.vdocument.in/reader033/viewer/2022042910/5f3e40a40378bb00dc64f221/html5/thumbnails/24.jpg)
DotkitManages your environmental variables
use list packages in use
use -a list hidden packages in use
use <package_name> add a package to environment
unuse <package_name> remove package from environment
use -la list available packages
use -la <term> list packages that contain <term>
Thursday, June 21, 12
![Page 25: The JANUS Computing Environment · Software support RC expertise less general Unsupported software Installation Consulting Advice on installing your software and any dependancies](https://reader033.vdocument.in/reader033/viewer/2022042910/5f3e40a40378bb00dc64f221/html5/thumbnails/25.jpg)
Examplesuse NCAR-Parallel-Intel
bash-janus> echo $PATH/curc/tools/free/redhat_5_x86_64/parallel-netcdf-1.2.0_openmpi-1.4.5_intel-12.1.4/bin/curc/tools/free/redhat_5_x86_64/openmpi-1.4.5_intel-12.1.4/bin/curc/tools/free/redhat_5_x86_64/torque-2.5.8/bin/curc/tools/free/redhat_5_x86_64/netcdf-4.1.3_intel-12.1.4_hdf-4.2.6_hdf5-1.8.8_openmpi-1.4.5/bin/curc/tools/free/redhat_5_x86_64/hdf5-1.8.8_openmpi-1.4.5_intel-12.1.4/bin/curc/tools/nonfree/redhat_5_x86_64/intel-12.1.4/composer_xe_2011_sp1.10.319/bin/intel64/curc/tools/free/redhat_5_x86_64/sun_jdk-1.6.0_23-x86_64/bin/curc/tools/free/redhat_5_x86_64/hdf-4.2.6_ics-2012.0.032/bin/curc/tools/free/redhat_5_x86_64/szip-2.1/bin/curc/tools/nonfree/redhat_5_x86_64/moab-6.1.5/bin
Thursday, June 21, 12
![Page 26: The JANUS Computing Environment · Software support RC expertise less general Unsupported software Installation Consulting Advice on installing your software and any dependancies](https://reader033.vdocument.in/reader033/viewer/2022042910/5f3e40a40378bb00dc64f221/html5/thumbnails/26.jpg)
Building SoftwareI need the Boost C++ library for my software. Where should I build this?
/home/molu8455/projects/software/boost/1.49.0
Build on a compute node (e.g. qsub -I)
Ideas
Consider sharing this with your group.
How about your own dotkit?
Thursday, June 21, 12
![Page 27: The JANUS Computing Environment · Software support RC expertise less general Unsupported software Installation Consulting Advice on installing your software and any dependancies](https://reader033.vdocument.in/reader033/viewer/2022042910/5f3e40a40378bb00dc64f221/html5/thumbnails/27.jpg)
Build your own dotkitcat $HOME/.kits/TeachingHPC.dk
#c Teaching HPC#d This contains the libraries I use for teaching HPC:#d .openmpi-1.4.3_gcc-4.5.2_torque-2.5.8_ib #d .hdf5-1.8.6
# Dependenciesdk_op -q .torque-2.5.8dk_op -q .openmpi-1.4.3_gcc-4.5.2_torque-2.5.8_ib dk_op -q .hdf5-1.8.6
# Variablesdk_alter HDF5_DIR /curc/tools/free/redhat_5_x86_64/hdf5-1.8.6dk_alter BOOST_ROOT /home/molu8455/projects/software/boost/1.49.0
dk_alter LD_LIBRARY_PATH /home/molu8455/projects/software/boost/1.49.0/lib
Thursday, June 21, 12
![Page 28: The JANUS Computing Environment · Software support RC expertise less general Unsupported software Installation Consulting Advice on installing your software and any dependancies](https://reader033.vdocument.in/reader033/viewer/2022042910/5f3e40a40378bb00dc64f221/html5/thumbnails/28.jpg)
Resource Management
Thursday, June 21, 12
![Page 29: The JANUS Computing Environment · Software support RC expertise less general Unsupported software Installation Consulting Advice on installing your software and any dependancies](https://reader033.vdocument.in/reader033/viewer/2022042910/5f3e40a40378bb00dc64f221/html5/thumbnails/29.jpg)
Scheduling
1
6
4
3
5
7
2Time
Node
s
Thursday, June 21, 12
![Page 30: The JANUS Computing Environment · Software support RC expertise less general Unsupported software Installation Consulting Advice on installing your software and any dependancies](https://reader033.vdocument.in/reader033/viewer/2022042910/5f3e40a40378bb00dc64f221/html5/thumbnails/30.jpg)
Scheduling
1
6
4
3 572
Time
Node
s
Thursday, June 21, 12
![Page 31: The JANUS Computing Environment · Software support RC expertise less general Unsupported software Installation Consulting Advice on installing your software and any dependancies](https://reader033.vdocument.in/reader033/viewer/2022042910/5f3e40a40378bb00dc64f221/html5/thumbnails/31.jpg)
Moab and TorqueMoab
Brains of the operation
Comes up with the “schedule”
Torque
Reports information to Moab
Receives direction from Moab
Handles users requests
Provide job query facilities
Thursday, June 21, 12
![Page 32: The JANUS Computing Environment · Software support RC expertise less general Unsupported software Installation Consulting Advice on installing your software and any dependancies](https://reader033.vdocument.in/reader033/viewer/2022042910/5f3e40a40378bb00dc64f221/html5/thumbnails/32.jpg)
Commands
showq -u <username> Show jobs in the queue
canceljob <job_id> or ALL Cancel your job(s)
checkjob <job_id> Information about your job
qsub submit jobs
showstart <job_id> When will your job start?
showq -u <username> Show jobs in the queue
Thursday, June 21, 12
![Page 33: The JANUS Computing Environment · Software support RC expertise less general Unsupported software Installation Consulting Advice on installing your software and any dependancies](https://reader033.vdocument.in/reader033/viewer/2022042910/5f3e40a40378bb00dc64f221/html5/thumbnails/33.jpg)
qsubRequest a resource for your job
1) batch or 2) interactive
Makes environmental variables available to your job
PBS_O_*PBS_O_WORKDIRPBS_NODEFILE
Options
-q <queue_name>-l <resource_list>-I interactive-N <name>-e <error_path>-o <output_path>-j <join_path>
Thursday, June 21, 12
![Page 34: The JANUS Computing Environment · Software support RC expertise less general Unsupported software Installation Consulting Advice on installing your software and any dependancies](https://reader033.vdocument.in/reader033/viewer/2022042910/5f3e40a40378bb00dc64f221/html5/thumbnails/34.jpg)
Queues
Name Nodes Max Time Node Sharing
janus-debug 1-480 1 hour
janus-short 1-480 4 hours
janus-long 1-80 7 days
janus-small 1-20 1 day
janus-normal 21-80 1 day
janus-wide 81-480 1 day
Thursday, June 21, 12
![Page 35: The JANUS Computing Environment · Software support RC expertise less general Unsupported software Installation Consulting Advice on installing your software and any dependancies](https://reader033.vdocument.in/reader033/viewer/2022042910/5f3e40a40378bb00dc64f221/html5/thumbnails/35.jpg)
Running Jobs
Thursday, June 21, 12
![Page 36: The JANUS Computing Environment · Software support RC expertise less general Unsupported software Installation Consulting Advice on installing your software and any dependancies](https://reader033.vdocument.in/reader033/viewer/2022042910/5f3e40a40378bb00dc64f221/html5/thumbnails/36.jpg)
ProcessHow many processors do I need?
Approximately how long will this take?
showstart 1024@30:00showstart 16@16:00:00
Which queue best fits this criteria?
2
4
Node
s
Time
Name Nodes Max Time Node Sharing
janus-debug 1-480 1 hour
janus-short 1-480 4 hours
janus-long 1-80 7 days
janus-small 1-20 1 day
janus-normal 21-80 1 day
janus-wide 81-480 1 day
Thursday, June 21, 12
![Page 37: The JANUS Computing Environment · Software support RC expertise less general Unsupported software Installation Consulting Advice on installing your software and any dependancies](https://reader033.vdocument.in/reader033/viewer/2022042910/5f3e40a40378bb00dc64f221/html5/thumbnails/37.jpg)
Serial Jobs#!/bin/bash
#PBS -N example_1#PBS -q janus-debug#PBS -l walltime=00:05:00#PBS -l nodes=1:ppn=1#PBS -e errfile#PBS -o outfile
cd $PBS_O_WORKDIR
# run trial 1 of the simulator./simulator 1 > sim.1
Thursday, June 21, 12
![Page 38: The JANUS Computing Environment · Software support RC expertise less general Unsupported software Installation Consulting Advice on installing your software and any dependancies](https://reader033.vdocument.in/reader033/viewer/2022042910/5f3e40a40378bb00dc64f221/html5/thumbnails/38.jpg)
Pack the node#!/bin/bash
#PBS -N example_2#PBS -q janus-debug#PBS -l walltime=0:00:30, nodes=1:ppn=12
cd $PBS_O_WORKDIR
./simulator 1 > sim.1 &
./simulator 2 > sim.2 &
./simulator 3 > sim.3 &
./simulator 4 > sim.4 &
./simulator 5 > sim.5 &
./simulator 6 > sim.6 &
./simulator 7 > sim.7 &
./simulator 8 > sim.8 &
./simulator 9 > sim.9 &
./simulator 10 > sim.10 &
./simulator 11 > sim.11 &
./simulator 12 > sim.12 &
wait
Thursday, June 21, 12
![Page 39: The JANUS Computing Environment · Software support RC expertise less general Unsupported software Installation Consulting Advice on installing your software and any dependancies](https://reader033.vdocument.in/reader033/viewer/2022042910/5f3e40a40378bb00dc64f221/html5/thumbnails/39.jpg)
Multi-node serial jobs?Consider using our load-balancing tool.
https://www.rc.colorado.edu/tutorials/loadbalance
#!/bin/bash#PBS -N example_1#PBS -q janus-debug#PBS -l walltime=00:05:00#PBS -l nodes=2:ppn=12
cd $PBS_O_WORKDIR
. /curc/tools/utils/dkinitreuse LoadBalance
mpirun load_balance -f cmd_lines
./simulator 1 > sim.1
./simulator 2 > sim.2
./simulator 3 > sim.3
./simulator 4 > sim.4
./simulator 5 > sim.5
./simulator 6 > sim.6
./simulator 7 > sim.7
./simulator 8 > sim.8
./simulator 9 > sim.9
./simulator 10 > sim.10
...
./simulator 2000 > sim.2000
Thursday, June 21, 12
![Page 40: The JANUS Computing Environment · Software support RC expertise less general Unsupported software Installation Consulting Advice on installing your software and any dependancies](https://reader033.vdocument.in/reader033/viewer/2022042910/5f3e40a40378bb00dc64f221/html5/thumbnails/40.jpg)
MPI#!/bin/bash
#PBS -N example_4#PBS -q janus-debug#PBS -l walltime=0:10:00#PBS -l nodes=3:ppn=12
cd $PBS_O_WORKDIRresuse .openmpi-1.4.5_intel-12.1.4
# run trial 1 of the simulatormpirun -np 36 ./simulator mpirun ./simulator
Thursday, June 21, 12
![Page 41: The JANUS Computing Environment · Software support RC expertise less general Unsupported software Installation Consulting Advice on installing your software and any dependancies](https://reader033.vdocument.in/reader033/viewer/2022042910/5f3e40a40378bb00dc64f221/html5/thumbnails/41.jpg)
Non-Uniform Memory Access (NUMA)Each socket has a dedicated memory area for high speed access
Also has an interconnect to other sockets for slower access to the other sockets' memory
memory memory
memory controlmemory control
Thursday, June 21, 12
![Page 42: The JANUS Computing Environment · Software support RC expertise less general Unsupported software Installation Consulting Advice on installing your software and any dependancies](https://reader033.vdocument.in/reader033/viewer/2022042910/5f3e40a40378bb00dc64f221/html5/thumbnails/42.jpg)
MPI OpenMP / High Memory#!/bin/bash
#PBS -N example_5#PBS -q janus-debug#PBS -l walltime=0:10:00#PBS -l nodes=3:ppn=12
cd $PBS_O_WORKDIR. /curc/tools/utils/dkinitresuse .openmpi-1.4.5_intel-12.1.4
export OMP_NUM_THREADS=12mpirun --bind-to-core --bynode --npernode 1 ./simulator
export OMP_NUM_THREADS=6mpirun --bind-to-socket --bysocket --npersocket 1 ./simulator
Thursday, June 21, 12
![Page 43: The JANUS Computing Environment · Software support RC expertise less general Unsupported software Installation Consulting Advice on installing your software and any dependancies](https://reader033.vdocument.in/reader033/viewer/2022042910/5f3e40a40378bb00dc64f221/html5/thumbnails/43.jpg)
SummaryAccess
Use control sockets for login
Filesystem
Build software in /projects/<username>
Run your jobs in /lustre/janus_scratch/<user_name>
Recover files with .snapshot
Consider striping when using shared-file access.
Data Transfer
Large files: Globus Online, Grid FTP
Smaller files: sftp, scp
Thursday, June 21, 12
![Page 44: The JANUS Computing Environment · Software support RC expertise less general Unsupported software Installation Consulting Advice on installing your software and any dependancies](https://reader033.vdocument.in/reader033/viewer/2022042910/5f3e40a40378bb00dc64f221/html5/thumbnails/44.jpg)
Software
Build on compute node.
Manage environment with your own dotkits.
Resource Management
Familiarize yourself with the queues
When you have choices... showstart
Running Jobs
Request what you need and manage with LoadBalance
OpenMP: be aware of NUMA
Limit the number of processes per node for hybrid and high memory
Thursday, June 21, 12
![Page 45: The JANUS Computing Environment · Software support RC expertise less general Unsupported software Installation Consulting Advice on installing your software and any dependancies](https://reader033.vdocument.in/reader033/viewer/2022042910/5f3e40a40378bb00dc64f221/html5/thumbnails/45.jpg)
Questions?
Thursday, June 21, 12
![Page 46: The JANUS Computing Environment · Software support RC expertise less general Unsupported software Installation Consulting Advice on installing your software and any dependancies](https://reader033.vdocument.in/reader033/viewer/2022042910/5f3e40a40378bb00dc64f221/html5/thumbnails/46.jpg)
Collective bufferingAt large core counts, I/O performance can be hindered by:
MDS contention (file-per-process)
file system contention (shared-file)
Use a subset of application processes to perform I/O.
limits the number of files (file-per-process)
limits the number of processes accessing file system resources (shared-file).
Offloads work from the file system to the application
A subset of processors write - reducing contention
Thursday, June 21, 12