ultra users guide v1.2

7/29/2019 Ultra Users Guide v1.2

1/35

HPCC USERS GUIDEVersion 1.2

July 2012

IITS (Research Support)Singapore Management University


2/35

Revision History

Version 1.0 (27 June 2012):- Modified HPCC users guide after creating the new Ultra HPC

clusterVersion 1.1 (3 July 2012):

- Modified HPCC users guide after changing the job queueselection option in the HPC job submission script

Version 1.2 (25 July 2012):

- Modified HPCC users guide after adding new job queue selectionoption in the HPC job submission script


3/35

Table of Contents

Introduction ..................................................................................... 4

HARDWARE ....................................................................................................... 4

SOFTWARE ....................................................................................................... 4HOW TO USE THIS GUIDE ................................................................................... 5

Chapter 1:Accessing HPC Cluster ....................................................... 6

1.1. Download Secure Shell Program ................................................................. 6

1.2. Connect to Ultra on SSH ............................................................................ 61.3. File Transfers to / from your Home Directory on Ultra .................................... 7

Chapter 2:Job Scheduler: Univa Grid Engine ....................................... 8

2.1. Grid Engine Commands ............................................................................. 8

2.2 Job Logs: Error and Output Streams .......................................................... 10

Chapter 3:Software Job Submission ................................................. 11

3.1. Overview ............................................................................................... 11

3.2. MATLAB ................................................................................................. 163.3. MATLAB Compiler .................................................................................... 173.4. GAUSS .................................................................................................. 183.5. R .......................................................................................................... 20

3 6 STATA SE 21


4/35

Introduction

HARDWARE

The High Performance Computing (HPC) Cluster is available in SMU for researchers needing

computational resources to run numerical analysis and simulations on the Linux plat form. Managed

by the Integrated IT Services (IITS) Research Support, the cluster comprises of:-

a. 1 x Intel Xeon CPU X5460 3.16GHz 8 cores processors compute server with 16GB acting as

the frontend login host

b. 8 x Intel Xeon CPU X5460 3.16GHz 8 cores processors compute server with 32GB RAM per

server

c. 2 x Intel Xeon CPU E5620 2.40GHz 80 cores processors compute server with 512GB RAM

per server

d. 2 x Intel Xeon CPU X7542 2.67GHz 24 cores processors compute server with 128GB RAM

per server

The servers are connected with a 10 Gigabit Ethernet switch on a private network for high speed low


5/35

HOW TO USE THIS GUIDE

This guide details the information required for you to use the HPC cluster.

Chapter 1 (Accessing HPC Cluster) tells you how you can connect to the cluster from inside and

outside of the SMU network.

Chapter 2 (Job Scheduler: Univa Grid Engine) touches on the three Linux commands that you need

to know to run and manage jobs in the cluster.

Chapter 3 (Software Job Submission) provides you with information required to run jobs for the

specific software in the cluster.

The appendixes provide quick referrals or refreshers when youre using the cluster:-

Appendix A (Linux Commands for home directory administration) shows some Linux commands for

self administration of your home folder.


6/35

Chapter 1.

Accessing HPC Cluster

Please apply for a HPC cluster account with IITS via email ([email protected]) if you do

not have an account. Otherwise you risk locking up the password to your SMU account if you

follow the steps shown in this guide.

The Ultra cluster is a shared resource. The current user disk quota is set at 12GB (soft limit) and

15GB (hard limit). Please ensure that you have sufficient disk space before running large data jobs.

If you need additional disk space, please email us [email protected].

To login to the cluster, you will need a secure shell program or any Linux-based OS. Suggestedsecure shell programs include:-

1.1. Secure Shell Program

1. Putty

http://www.google.com.sg/search?hl=en&q=putty&meta=
mailto:[email protected]:[email protected]:[email protected]:[email protected]:[email protected]:[email protected]://www.google.com.sg/search?hl=en&q=putty&metahttp://www.google.com.sg/search?hl=en&q=putty&metamailto:[email protected]:[email protected]


7/35

1.2.1 Using Putty.exe

Save the file putty.exe to your computer and double-click to run it. You will see the interface below:-

Figure 1.1: Putty Settings


8/35

Figure 1.3: WinSCP Settings

Fill in the hostname, user name, password and choose the SFTP protocol as detailed in Figure 1.3

above. Once logged in, you can proceed to transfer files between the 2 displayed windows (one of the

windows will be your desktop while the other will be your home directory in Ultra).

Chapter 2.

Job Scheduler: Univa Grid Engine


9/35

This command is used to submit a job description file (which describes the job you wish to run) for job

queuing and execution. More information on specific software job description file will be provided in

the later chapter which discusses the software in the cluster.

Syntax:

qsub [ job script ]

[ job script ] Job description file to submit

Example:

To submit a job: qsub jobscript.sh

2.1.2. qstat (to query job status)

This command is used to query the status of submitted jobs and queues. Use this command after you

submit a job to check whether the job is running.

Syntax:

qstat [ -f ] [ -u userid ] [ -s {r|p|s|z} ] [ -help ]

[ -f ] Optional. Display the full list of compute nodes and jobs running or queued

[ -u userid ] Optional. Display jobs belonging to the userid

[ -s {r|p|s|z} ] Optional. Display jobs in the specified state of r (running), p (pending), s(suspended) orz (zombie). Can be used in combination, example -s rpsz.

[ h l ] O ti l Di l i k h l


10/35

2.2 Job Logs: Error and Output Streams

For every job submitted, the screen output and errors will be dumped into a file named after your job

submission script in the following format:

output log: job_name.sh.ojob_id

If the job did not run as expected and encountered errors, the file might log some information to aid

troubleshooting.


11/35

Chapter 3.Software Job Submission

The following are some of the computational codes that can be run in the Ultra cluster:

1. MATLAB codes

2. MATLAB-compiled binaries

3. Gauss-compiled pseudo-code

4. R

5. STATA SE

6. ILOG CPLEX

7. C/C++ and FORTRAN codes (uses the Intel, Pathscale or GNU compiler)

8. Java Codes

Code samples and instructions to run the codes are available in users home directory under the

subfolder ultra-samples.Please con tact IITS (email:[email protected])for assistance for

sof tware not l is ted above.
mailto:[email protected]:[email protected]:[email protected]:[email protected]


12/35

You need to create a job submission script for every job that you intend to submit. This job submission

script contains the Univa Grid Engine switches (parameters) and software specific instructions to run

your code. This section shows you how to write a job submission script.

A job submission script consists of 2 parts:

1. Univa Grid Engine (GE) switches

2. Software specific instruction

Example of a job submission script:

#! bin/bash

#$ -j y#$ -cwd#$ -m e#$ -M [email protected]

#$ -q short.q

./sim1

Grid Engine switches:

GE switches

Software specifics


13/35

Software Launching Instructions

C / C++ / FORTRANcompiled binary

./[compiled filename] > [ output file ]

example:

./sim1 > out.txt

Gauss pseudocode gsrun b [ gauss filename ]

example:

gsrun -b ols.e.gcg

ILOG CPLEX cplex < [ cplex script ] > [ output file ]

example:

cplex < sim1 > result.txt

JAVA . /etc/profile.d/java.sh

java [ java classname ]

example:

. /etc/profile.d/java.sh

java testrun > results.txt

MATLAB matlab -r '[ filename ]'

example (the.m f i le extension m ust be om it ted):


14/35

1. Name the script with a file extension .sh to indicate that the file is a shell script

2. Start the script with the line: #! /bin/bash

3. Add the Grid Engine switches

4. Add the software specific instructions

The following is an example of a job submission script named mysim1.sh which is used to run a 64-

bit MATLAB compiled binary file named sim1:

Example job submission script sendjob1.sh:

#! /bin/bash

#$ -j y

#$ -cwd#$ -m e

#$ [email protected]

#$ -q short.q
mailto:[email protected]:[email protected]:[email protected]


15/35

Example output:

job-ID prior name user state submit/start at queue slots ja-task-ID

-----------------------------------------------------------------------------------------------------------------------------------------------------------------

110 0.55500 sendjob1f. kenyamada r 06/28/2012 18:57:50 express.q@c3-1 1



Possible states are:-

- r: running

- qw: queuing

- t: transfer

- Eqw: error queuing

Alternatively, run the command jobwatch which displays per second update of qstat. To break out

of jobwatch, press Ctrl-C.

Note: If your job state is Eqw, you need to delete the job. Then check the error and output file for

hints on why the job could not run. You may also want to double-check your job-script for typo

mistakes. Then you can resubmit the job again.


16/35

3.2. MATLAB

You can run MATLAB codes directly in the cluster. However, you are encouraged to compile MATLAB

codes instead as compiled MATLAB jobs do not utilize network licenses. More information on

MATLAB compilation can be found in the next section.

3.2.1 MATLAB Codes

MATLAB codes are interpreted by the software and are platform independent. In most cases,

MATLAB codes can be ported between Windows and Linux platforms. However, note that there is a

file path difference between the codes on Windows and Linux. On Windows, file paths are written like:

D:\run1\compute.m

while on the Linux platform, it is written as:

/home/[ your userid ]/run1/compute.m

I f you use such f i le path references in you r code, you have to change i t accordingly w hen you

port the codes over to the cluster .


17/35

3.3. MATLAB Compiler

The MATLAB Compiler allows MATLAB codes to be compiled into binary (executable) form that can

be run across the same platform using the MATLAB Run-Time Library. Running a MATLAB

executable does not require any MATLAB network licenses.

To compile MATLAB codes, scripts must be first be marked as functions. This usually involves putting

in a header line like function main. This command then compiles the code inside MATLAB:

>> mcc mv simulation1.m

Using the above example, a few related files will be created in the folder:

1. simulation (this is the compiled binary)

2. files with extension .c (these can be removed)

3. simulation.prj (this can be removed)

4. simulation.log (this can be removed)

5. run_simulation.sh (this can be removed)

3.3.1. Compiled MATLAB Job Submission Script


18/35

3.4. GAUSS

There are no GAUSS licenses available on the compute nodes. Instead, the compute nodes are

installed with the GAUSS Run-Time Module that can run GAUSS-compiled pseudo-codes. On the

cluster, the front-end node has the following GAUSS packages:

1. Constrained Maximum Likelihood add-on

2. Constrained Optimization add-on

3. GAUSS Run-Time module

3.4.1 Gauss Codes

GAUSS codes are interpreted by the software and are platform independent. In most cases, GAUSS

codes can be ported between Windows and Linux platforms. However, note that there is a file path

difference between the codes on Windows and Linux. On Windows, file paths are written like:

D:\run1\compute.m

while on the Linux platform, it is written as:


19/35

In the following example, compute.e.gcg is the GAUSS pseudo-code file that we want to run. We

create a job submission script sendjob1.sh to describe compute.e.gcg as the Gauss pseudo-code

file to be run:


#! /bin/bash

#$ -j y

#$ -cwd

#$ -m e


#$ -q short.q

gsrun -b compute.e.gcg

The following command submits the job submission script:

qsub sendjob1.sh


20/35

3.5. R

R is available on all nodes in the cluster.

3.5.1. R Job Submission Script

Just like Matlab and Gauss, a job submission script has to be created in order to submit a job. In this

example, simulation.R is the R code that needs to be submitted. A job submission script

sendjob1.sh can be created to point simulation.R as an R code to be run:


#! /bin/bash

#$ -j y#$ -cwd

#$ -m e

#$ -M [email protected]

#$ -q short.q

R -q --vanilla < simulation.R


21/35

3.6. STATA SE

STATA SE is available only on the Intel EM64T and AMD64 nodes in the HPC Cluster.

3.6.1. STATA Job Submission Script

Just like all other jobs, a job submission script has to be created first in order to submit a STATA job.

In this example, sortdata.do is the STATA code that needs to be submitted and run. A job

submission script sendjob1.sh can be created to describe sortdata.do as the code to be run:


#! /bin/bash

#$ -j y

#$ -cwd

#$ -m e


#$ -q short.q


22/35

3.7. ILOG CPLEX

ILOG CPLEX is available only on the 64-bit nodes in the HPC Cluster.

3.7.1. ILOG CPLEX Job Submission Script

Submitting an ILOG CPLEX job is slightly different from the other software in the HPC Cluster. There

are 3 components instead of the usual 2 components (software job and submission script). Submitting

an ILOG CPLEX job requires 3 files:

a. CPLEX problem formulation

b. Script file containing CPLEX commands

c. Cluster submission script containing submission characteristics of job

The following illustrates an example (also found in the sub-folder Ultra-samples/ilog) for submitting a

CPLEX job:

(a) create the CPLEX problem:

(example file name problem.lp)

------------------------------

maximize

x1 + 2 x2 + 3 x3


23/35

(example file name cplex-submit.sh)

------------------------------

#! /bin/bash

#$ -j y

#$ -cwd

#$ -m e

#$ -M [email protected]

#$ -q short.q

cplex < cplex-script

------------------------------


qsub cplex-submit.sh


24/35

3.8. Intel C / C++ / FORTRAN

The cluster front-end node is installed with the Intel Compiler Suite (C/C++ and FORTRAN compilers,

Math Kernel Library, Performance Primitives and Vtune Analyzer) as well as the GNU compilers. The

compute nodes only have the Intel run-time libraries and the GNU compilers.

The Intel compilers have been optimized for the Intel platform (here 64-bit architecture) while the GNU

compilers are generalized x86 open-source compilers. For this cluster, it is recommended that you

use the Intel compilers unless you prefer the GNU compilers.

3.8.1 C / C++ Compiler

To run C/C++ codes on the 32-bit platform of the cluster, the codes must first be compiled into binary

form. To compile a file:

icc static [ source file name ] -o [ output file name ]

If the code is compiled successfully, it can then be dispatched for running by creating and submitting a

job submission file. See section below.

3.8.2. FORTRAN Compiler


25/35

#$ -cwd

#$ -m e


#$ -q short.q

./run_c_code


qsub sendjob1.sh


26/35

3.9. IMSL Numerical Library (32-bit )

The IMSL Numerical Library is a 32-bit numerical library and is available on the 64-bit nodes.

3.9.1. IMSL FORTRAN Job Submission Script

The example below is taken from the sub-folder Ultra-samples/imsl in the home directory ofall user.

The file README in the sub-folder also explains how you can compile and run the job. In this

example, lin_eig_self_ex4.f90 is the FORTRAN code that uses IMSL functions and needs to be

compiled. A job submission script called imsl-submit.sh can be created to include lin_eig_self_ex4

as the FORTRAN code to be run:

2 methods are suggested below to compile the FORTRAN code that uses IMSL functions:

(a) First, setup the environment for the IMSL libraries:

. /opt/vni/CTT6.0/ctt/bin/cttsetup.sh

Next, type the following command line in a single continuous line (case-sensitive):

$F90 $F90FLAGS -o lin_eig_self_ex4 lin_eig_self_ex4.f90

$LINK_F90_STATIC -lpthread

The above command line will compile the code lin_eig_self_ex4.f90 into an output binary file


27/35

#$ -q short.q

./lin_eig_self_ex4 > output.txt


qsub imsl-submit.sh

When the job is completed without any problems, an output file named output.txt will be created.

Check the output file. If it shows Example 4 of LIN_EIG_SELF is correct, then the job ran

successfully.


28/35

3.10. JAVA

The Java SDK is available across all nodes in the cluster.

3.10.1. JAVA Compiler

You can compile your Java codes using the Java compiler installed on the frontend node. You can

test out the example code in the Ultra-samples/java folder in your home directory.

After the code is compiled successfully, you can submit the job using a job submission script:


#! /bin/bash

#$ -j y

#$ -cwd

#$ -m e


#$ -q short.q


29/35

3.11. Quick Summary of Cluster Software

QUICK SUMMARY OF THE SOFTWARE

Software Commands (case sensitive)

MATLAB matlab nodisplay

Gauss tgauss (or) gauss

Gauss Run-time gsrun

R R

STATA SE stata (or) stata-se

ILOG CPLEX cplex

Java Compiler javac

Java Runtime java

Using IMSL (32-bit) $F90 $F90FLAGS -o $LINK_F90_STATIC lpthread

Intel C Compiler(64-bit) /opt/intel/cce/10.1.018/bin/icc -o

Intel C++ Compiler(64-bit) /opt/intel/cce/10.1.018/bin/icpc -o

Intel Fortran Compiler(64-bit) /opt/intel/fce/10.1.018/bin/ifort -o

Intel C Compiler(32-bit) /opt/intel/cc/10.1.018/bin/icc -o


30/35

Appendix A.

Linux Commands for Self Administration

A.1. Disk Quota

A.1.1. Check your quota usage using the command quota -s

Example output:

---------------------------------------------------------------------------

[legolas@Ultra legolas]$ quota s

Disk quotas for user legolas (uid 502):

Filesystem blocks quota limit grace files quota limit grace

/dev/hda3 6016M 7813M 9766M 101 0 0

---------------------------------------------------------------------------

Column 2 blocks shows your current quota usage.

Column 3 quota shows the soft limit quota.


31/35

[ file or dir name ] : Wildcards (*,?) can be used here to filter

filenames to display.

To create a folder: mkdir [ new foldername ]

To remove a file: rm [ file to delete ]

To remove a folder: rm -rf [ folder to delete ]

To rename a file: mv [ source file name ] [ target file name ]

To change directory: cd [ foldername ]

To move out of folder: cd ..

To move into some folder: cd [ absolute path from root directory ]

To copy a file: cp [ source file ] [ target location ]

To copy a folder: cp -r [ source folder ] [ target location ]

Path ofpresent working directory: pwd


32/35

A.3.2. Permissions Handling

Let r/w/x represent numbers 4/2/1 respectively. To turn on the necessary permissions, add the

numbers respectively. Example, r-x is counted as 5.

To set the permission for owner/group/others, use the above number and link them together.

Example, rwxr-x--- is interpreted as 750.

After calculating the numbers from the above 2 steps, type the command to complete the permissions

change:

chmod 755 [ filename ]

This is useful if you want to share a particular folder with a collaborator (contact IITS to create a

common group for both you and the collaborator).

A.4. Miscellaneous Commands

To finding a file, type:

locate [ filename ]


33/35

Appendix B.

The vi Text Editor

The vi text editor is one of the text editors available (others include pico, emacs) in the cluster for

editing text files. To use vi from the command line, type either of the following commands:

vi [filename] (opens up the vi editor. Type :help inside to get extensive help)vi +n [filename] (starts editing at line numbern)

VI uses a set of key codes to indicate commands. The following table shows the various commonly

used commands (case-sensit ive):

File Operations Keys

Write to file (save) :wWrite to file and Quit (save and quit) :wq

Write to another filename file :w file

Quit :q

Quit and discard changes :q!

Quit, saving changes if any :x

Text Editing Keys


34/35

IITS, Singapore Management University Page 34 of 35

Appendix C.

Cluster Software Availability List

COMPILERS

Software Cmd / Location Documentation Man Pages

GNU C/C++ Compiler

GNU F77 Compiler

gcc / g++

g77

Official Software:

http://www.gnu.org/software/gcc/

Type:

man gcc

man g77

MATHEMATICAL AND STATISTICAL


GAUSS

- Constrained Max. Likelihood

- Constrained Optimization

- Run-Time Module

tgauss

gauss

gsrun

Official Software:

http://www.aptech.comType:

gauss -h

gsrun -h

MATLAB matlab Official Software:http://www.mathworks.com

Type:

matlab -h
http://www.gnu.org/software/gcc/http://www.gnu.org/software/gcc/http://www.aptech.com/http://www.aptech.com/http://www.mathworks.com/http://www.mathworks.com/http://www.mathworks.com/http://www.aptech.com/http://www.gnu.org/software/gcc/


35/35

IITS, Singapore Management University Page 35 of 35

R R Official Software:

http://www.r-project.orgType:

R h

STATA SE stata Official Software Website:

http://www.stata.comType:

stata -h

ILOG CPLEX cplex Official Software Website:

http://www.ilog.comType:

cplex

MATHEMATICAL LIBRARY ROUTINES (FOR USE WITH FORTRAN COMPILERS)


IMSL Fortran NumericalLibraries (32-bit version) $F90 $F90FLAGS -o example example.f90$LINK_F90_STATIC lpthread Official Software:http://www.vni.comNone
http://www.r-project.org/http://www.r-project.org/http://www.stata.com/http://www.stata.com/http://www.ilog.com/http://www.ilog.com/http://www.vni.com/http://www.vni.com/http://www.vni.com/http://www.ilog.com/http://www.stata.com/http://www.r-project.org/

ultra users guide v1.2

Documents