fokke dijkstra, bob dröge high performance computing and … · 2015-10-02 · 24-9-2012 using the...

93
24-9-2012 Using the Millipede cluster - I Fokke Dijkstra, Bob Dröge High Performance Computing and Visualisation group Donald Smits Centre for Information Technology

Upload: others

Post on 29-Mar-2020

2 views

Category:

Documents


0 download

TRANSCRIPT

Page 1: Fokke Dijkstra, Bob Dröge High Performance Computing and … · 2015-10-02 · 24-9-2012 Using the Millipede cluster - I Fokke Dijkstra, Bob Dröge High Performance Computing and

24-9-2012

Using the Millipede cluster - IFokke Dijkstra, Bob Dröge

High Performance Computing and Visualisation group

Donald Smits Centre for Information Technology

Page 2: Fokke Dijkstra, Bob Dröge High Performance Computing and … · 2015-10-02 · 24-9-2012 Using the Millipede cluster - I Fokke Dijkstra, Bob Dröge High Performance Computing and

24-9-2012

General introduction

› Course aimed at beginners

• No knowledge about clusters or Linux necessary

• Examples based on UWP Windows installation

• Some pointers for Linux or Mac OS X users

› Topics – Part I

• Connecting to the cluster

• Using the Linux command line

• Editing files

Page 3: Fokke Dijkstra, Bob Dröge High Performance Computing and … · 2015-10-02 · 24-9-2012 Using the Millipede cluster - I Fokke Dijkstra, Bob Dröge High Performance Computing and

24-9-2012

› How do I get onto the system?

Page 4: Fokke Dijkstra, Bob Dröge High Performance Computing and … · 2015-10-02 · 24-9-2012 Using the Millipede cluster - I Fokke Dijkstra, Bob Dröge High Performance Computing and

24-9-2012

Millipede cluster

userfront-end node

internet fast network

Cluster nodes

CIT

Office somewhere

Page 5: Fokke Dijkstra, Bob Dröge High Performance Computing and … · 2015-10-02 · 24-9-2012 Using the Millipede cluster - I Fokke Dijkstra, Bob Dröge High Performance Computing and

24-9-2012

SSH

› SSH protocol used to connect to the cluster

• Standard interface for Unix systems

• Encrypted network traffic

› Software for Windows called PuTTY:• http://www.chiark.greenend.org.uk/~sgtatham/putty/

• Open source software, freely available

• Already installed on UWP

› Use terminal window under Linux or Mac OS X

› Normally university account id used (p- or s-number)

› Hostname millipede.service.rug.nl

Page 6: Fokke Dijkstra, Bob Dröge High Performance Computing and … · 2015-10-02 · 24-9-2012 Using the Millipede cluster - I Fokke Dijkstra, Bob Dröge High Performance Computing and

24-9-2012

PuTTY start window

Page 7: Fokke Dijkstra, Bob Dröge High Performance Computing and … · 2015-10-02 · 24-9-2012 Using the Millipede cluster - I Fokke Dijkstra, Bob Dröge High Performance Computing and

24-9-2012

PuTTY host key check

• This happens when you connect for the first time

• Identity of system not known by PuTTY client

• May also happen after an update on the login node

Page 8: Fokke Dijkstra, Bob Dröge High Performance Computing and … · 2015-10-02 · 24-9-2012 Using the Millipede cluster - I Fokke Dijkstra, Bob Dröge High Performance Computing and

24-9-2012

PuTTY login

Page 9: Fokke Dijkstra, Bob Dröge High Performance Computing and … · 2015-10-02 · 24-9-2012 Using the Millipede cluster - I Fokke Dijkstra, Bob Dröge High Performance Computing and

24-9-2012

› Help, where is my data?

Page 10: Fokke Dijkstra, Bob Dröge High Performance Computing and … · 2015-10-02 · 24-9-2012 Using the Millipede cluster - I Fokke Dijkstra, Bob Dröge High Performance Computing and

24-9-2012

File organisation

› In windows directory structure separated by \

› Drive letter included

C:\Users\Peter\My Documents\testfile.txt

› In Linux directories separated by /

› Drive letters not used, everything appears as a directory name/home/peter/My Documents/testfile.txt

› Note that the use of “spaces” is difficult on the command line

› Use quotes if you really need to use them

”/home/peter/My Documents/testfile.txt”

› Case-sensitive: Myfile.txt ≠ MyFile.txt

Page 11: Fokke Dijkstra, Bob Dröge High Performance Computing and … · 2015-10-02 · 24-9-2012 Using the Millipede cluster - I Fokke Dijkstra, Bob Dröge High Performance Computing and

24-9-2012

A Linux system concept: the home directory

› A unique home directory for each user.

› Every user has one.

› The ssh session (putty terminal) starts in this directory.

› On millipede looks like:/home/username, e.g./home/p742111

Page 12: Fokke Dijkstra, Bob Dröge High Performance Computing and … · 2015-10-02 · 24-9-2012 Using the Millipede cluster - I Fokke Dijkstra, Bob Dröge High Performance Computing and

24-9-2012

File transfer clients

› Windows

• WinSCP is a file transfer program for Windows

• http://winscp.net

• Open source, freely available

• Not pre-installed in UWP

› Linux

• Command line scp or sftp

• Nautilus file manager, Konqueror (fish://), FileZilla

› Mac OS X (not tested)

• Command line scp or sftp

• CyberDuck (Open source), FileZilla (Open source), Fetch (Free academic license)

Page 13: Fokke Dijkstra, Bob Dröge High Performance Computing and … · 2015-10-02 · 24-9-2012 Using the Millipede cluster - I Fokke Dijkstra, Bob Dröge High Performance Computing and

24-9-2012

Copying files: WinSCP

Page 14: Fokke Dijkstra, Bob Dröge High Performance Computing and … · 2015-10-02 · 24-9-2012 Using the Millipede cluster - I Fokke Dijkstra, Bob Dröge High Performance Computing and

24-9-2012

WinSCP host key check

Page 15: Fokke Dijkstra, Bob Dröge High Performance Computing and … · 2015-10-02 · 24-9-2012 Using the Millipede cluster - I Fokke Dijkstra, Bob Dröge High Performance Computing and

24-9-2012

WinSCP file operations: copy, rename, move, delete

Local filesMillipede

Page 16: Fokke Dijkstra, Bob Dröge High Performance Computing and … · 2015-10-02 · 24-9-2012 Using the Millipede cluster - I Fokke Dijkstra, Bob Dröge High Performance Computing and

24-9-2012

› But it runs Linux! Introducing the Linux command line

Page 17: Fokke Dijkstra, Bob Dröge High Performance Computing and … · 2015-10-02 · 24-9-2012 Using the Millipede cluster - I Fokke Dijkstra, Bob Dröge High Performance Computing and

24-9-2012

Basic commands: ls

› ls lists files:$ ls

Job1_script.sh Task1.in.mat Task1.state.mat Task2.jobout.mat

matlab_metadata.mat Task1.jobout.mat Task2.common.mat Task2.out.mat

Task1.common.mat Task1.out.mat Task2.in.mat Task2.state.mat

$ ls -l

total 32

-rw-rw-r-- 1 peter peter 1354 Jan 11 10:10 Job1_script.sh

-rw-rw-r-- 1 peter peter 587 Jan 11 10:10 matlab_metadata.mat

-rw-rw-r-- 1 peter peter 0 Jan 11 10:10 Task1.common.mat

-rw-rw-r-- 1 peter peter 690 Jan 11 10:10 Task1.in.mat

-rw-rw-r-- 1 peter peter 0 Jan 11 10:10 Task1.jobout.mat

-rw-rw-r-- 1 peter peter 889 Jan 11 10:10 Task1.out.mat

-rw-rw-r-- 1 peter peter 7 Jan 12 09:47 Task1.state.mat

-rw-rw-r-- 1 peter peter 0 Jan 11 10:10 Task2.common.mat

-rw-rw-r-- 1 peter peter 690 Jan 11 10:10 Task2.in.mat

-rw-rw-r-- 1 peter peter 0 Jan 11 10:10 Task2.jobout.mat

-rw-rw-r-- 1 peter peter 889 Jan 11 10:10 Task2.out.mat

-rw-rw-r-- 1 peter peter 7 Jan 12 09:47 Task2.state.mat

Page 18: Fokke Dijkstra, Bob Dröge High Performance Computing and … · 2015-10-02 · 24-9-2012 Using the Millipede cluster - I Fokke Dijkstra, Bob Dröge High Performance Computing and

24-9-2012

Basic commands: cd

› cd changes to a directory$ ls

Readme.txt Job1 Job2 Job3

$ cd Job1

$ ls

Job1_script.sh Task1.in.mat Task1.state.mat Task2.jobout.mat

matlab_metadata.mat Task1.jobout.mat Task2.common.mat Task2.out.mat

Task1.common.mat Task1.out.mat Task2.in.mat Task2.state.mat

› “..” points to directory above

$ cd ..

$ ls

Readme.txt Job1 Job2 Job3

› “.” points to current directory$ ./myprog

$ cp ../AnotherProgram .

Page 19: Fokke Dijkstra, Bob Dröge High Performance Computing and … · 2015-10-02 · 24-9-2012 Using the Millipede cluster - I Fokke Dijkstra, Bob Dröge High Performance Computing and

24-9-2012

Basic commands: mkdir/rmdir

› mkdir creates a directory:$ ls

Job1_script.sh Task1.in.mat

$ mkdir testdir

$ ls

Job1_script.sh Task1.in.mat testdir

› rmdir removes a directory (must be empty):

$ rmdir testdir

$ ls

Job1_script.sh Task1.in.mat

Page 20: Fokke Dijkstra, Bob Dröge High Performance Computing and … · 2015-10-02 · 24-9-2012 Using the Millipede cluster - I Fokke Dijkstra, Bob Dröge High Performance Computing and

24-9-2012

Basic commands: cp

› cp copies files (and/or directories):$ ls

Job1_script.sh Task1.in.mat Task1.state.mat Task2.jobout.mat

matlab_metadata.mat Task1.jobout.mat Task2.common.mat Task2.out.mat

Task1.common.mat Task1.out.mat Task2.in.mat Task2.state.mat

› Copy single file$ cp Task1.state.mat Task3.state.mat

$ ls

Job1_script.sh Task1.in.mat Task1.state.mat Task2.jobout.mat

matlab_metadata.mat Task1.jobout.mat Task2.common.mat Task2.out.mat

Task1.common.mat Task1.out.mat Task2.in.mat Task2.state.mat

Task3.state.mat

› Copy multiple files$ cp *.mat ../newdirectory

$ cp -r mydir newdir

Page 21: Fokke Dijkstra, Bob Dröge High Performance Computing and … · 2015-10-02 · 24-9-2012 Using the Millipede cluster - I Fokke Dijkstra, Bob Dröge High Performance Computing and

24-9-2012

Basic commands: rm

› rm removes files (and/or directories):$ ls

Job1_script.sh Task1.in.mat Task1.state.mat Task2.jobout.mat

matlab_metadata.mat Task1.jobout.mat Task2.common.mat Task2.out.mat

Task1.common.mat Task1.out.mat Task2.in.mat Task2.state.mat

› Remove single file$ rm Task1.state.mat

› Remove multiple files$ rm *.mat

$ rm Task?.common.mat

$ rm *

everything in current directory is removed. Make sure you know your current directory! To avoid this go a directory up, then remove the desired directory

$ cd ..

$ rm dirname/*

› Remove a directory with all files and subdirectories$ rm -r mydir

Page 22: Fokke Dijkstra, Bob Dröge High Performance Computing and … · 2015-10-02 · 24-9-2012 Using the Millipede cluster - I Fokke Dijkstra, Bob Dröge High Performance Computing and

24-9-2012

Basic commands: mv

› mv moves/renames files:$ ls

Job1_script.sh Task1.in.mat Task1.state.mat Task2.jobout.mat

$ mv Job1_script.sh jobscript.sh

$ ls

jobscript.sh Task1.in.mat Task1.state.mat Task2.jobout.mat

› Can move a whole directory to another place

$ mv Job1Directory ../jobDirectory

Page 23: Fokke Dijkstra, Bob Dröge High Performance Computing and … · 2015-10-02 · 24-9-2012 Using the Millipede cluster - I Fokke Dijkstra, Bob Dröge High Performance Computing and

24-9-2012

Useful tricks

› <Tab> does auto completion of file names$ ls

Job1_script.sh Task1.in.mat Task1.state.mat Task2.jobout.mat

$ mv J<Tab>

results in

$ mv Job1_script.sh

› Arrows keys ( ) can be used to scroll through list of previous commands

› Ctrl-A or Home takes your cursor to the beginning of the commandline

› Ctrl-E or End takes your cursor to the end of the command line

› Ctrl-R enables search through previous commands.

› Ctrl-C interrupts the current program

› Ctrl-D logs you out (ends the session)

Page 24: Fokke Dijkstra, Bob Dröge High Performance Computing and … · 2015-10-02 · 24-9-2012 Using the Millipede cluster - I Fokke Dijkstra, Bob Dröge High Performance Computing and

24-9-2012

Other useful commands

› pwd: prints current directory$ pwd

/home/peter/testdir

› echo: puts some text on the screen$ echo Hello world!

Hello world!

› less: Look at the contents of a file (exit using q)$ less my_output_file

› man: Get information about a command (exit using q)$ man ls

› copy-paste: select the piece of text of interest

• Ctrl+Insert – copy the text

• Shift+Insert - paste the copied text

Page 25: Fokke Dijkstra, Bob Dröge High Performance Computing and … · 2015-10-02 · 24-9-2012 Using the Millipede cluster - I Fokke Dijkstra, Bob Dröge High Performance Computing and

24-9-2012

Environment variables

› Linux scripts make extensive use of environment variables

› Prefixed by $

› Expanded when the command is run

› Examples

$ cd $HOME

Will change into home directory $HOME is replaced with e.g. /home/f204524

$ echo $USER

Will display the user name $USER is replaced by e.g f204524

› If ambiguity can arise use ${VARIABLE}

Page 26: Fokke Dijkstra, Bob Dröge High Performance Computing and … · 2015-10-02 · 24-9-2012 Using the Millipede cluster - I Fokke Dijkstra, Bob Dröge High Performance Computing and

24-9-2012

Redirecting in- and output

› Input and/or output of programs can be redirected to another program or to a file

› Send output to file using “>”$ ls -l > files.txt

› Read input from a file using “<“$ myprog < inputfile > outputfile

› Pipe output from a command into another command$ qstat | less

› Append output to a file usign “>>”$ ls –l >> files.txt

Page 27: Fokke Dijkstra, Bob Dröge High Performance Computing and … · 2015-10-02 · 24-9-2012 Using the Millipede cluster - I Fokke Dijkstra, Bob Dröge High Performance Computing and

24-9-2012

› Do I really have to know what ctrl-x ctrl-s or :wq does? Editing files

Page 28: Fokke Dijkstra, Bob Dröge High Performance Computing and … · 2015-10-02 · 24-9-2012 Using the Millipede cluster - I Fokke Dijkstra, Bob Dröge High Performance Computing and

24-9-2012

Editing files through WinSCP

Page 29: Fokke Dijkstra, Bob Dröge High Performance Computing and … · 2015-10-02 · 24-9-2012 Using the Millipede cluster - I Fokke Dijkstra, Bob Dröge High Performance Computing and

24-9-2012

Nano editor

$ nano testjob

Page 30: Fokke Dijkstra, Bob Dröge High Performance Computing and … · 2015-10-02 · 24-9-2012 Using the Millipede cluster - I Fokke Dijkstra, Bob Dröge High Performance Computing and

24-9-2012

Linux and windows text

› Windows uses two characters (carriage return and linefeed) to end a line

› Linux and Mac OSX only use the linefeed character to end a line

› Older Macs only use the carriage return character instead

› This may sometimes give you problems

› Tools dos2unix & unix2dos & mac2unix can convert text if necessary

Page 31: Fokke Dijkstra, Bob Dröge High Performance Computing and … · 2015-10-02 · 24-9-2012 Using the Millipede cluster - I Fokke Dijkstra, Bob Dröge High Performance Computing and

Full Linux course available

Topics:› History and philosophy behind Linux› Installing and configuring Ubuntu, installing software› The desktop environment› Controlling processes › The Linux filesystem and the commands available› Using bash (the Bourne-Again Shell)› Connecting programs together› Basic scripting

http://www.rug.nl/cit/onderwijs/cursus/linux

Page 32: Fokke Dijkstra, Bob Dröge High Performance Computing and … · 2015-10-02 · 24-9-2012 Using the Millipede cluster - I Fokke Dijkstra, Bob Dröge High Performance Computing and

24-9-2012

Using the Millipede cluster - IIFokke Dijkstra, Bob Dröge, Ana Oprescu

High Performance Computing and Visualisation group

Donald Smits Centre for Information Technology

Page 33: Fokke Dijkstra, Bob Dröge High Performance Computing and … · 2015-10-02 · 24-9-2012 Using the Millipede cluster - I Fokke Dijkstra, Bob Dröge High Performance Computing and

24-9-2012

General introduction

› Course aimed at beginners

• This part assumes knowledge about the Linux command line, file transfers and editing files

› Topics – Part II

• What is a cluster

• Cluster storage

• Module environment

• Submitting jobs

Page 34: Fokke Dijkstra, Bob Dröge High Performance Computing and … · 2015-10-02 · 24-9-2012 Using the Millipede cluster - I Fokke Dijkstra, Bob Dröge High Performance Computing and

24-9-2012

HPC/V

› High performance computing and visualisation group

› HPC Facilities

• Millipede cluster

• IBM BlueGene/P

• Grid cluster

› Visualisation

• Cave

• Theatre

• Scientific data

• Virtual reality

Page 35: Fokke Dijkstra, Bob Dröge High Performance Computing and … · 2015-10-02 · 24-9-2012 Using the Millipede cluster - I Fokke Dijkstra, Bob Dröge High Performance Computing and

24-9-2012

› What is a computer cluster?

Page 36: Fokke Dijkstra, Bob Dröge High Performance Computing and … · 2015-10-02 · 24-9-2012 Using the Millipede cluster - I Fokke Dijkstra, Bob Dröge High Performance Computing and

24-9-2012

Computer cluster

› A cluster is a collection of computers connected by a network

› A single front-end

› Lots of computers in the background for running tasks

› 1994 first cluster of commodity PCs at NASA

› Millipede cluster looks quite different

› Most clusters run the Linux operating system

Cluster at Michigan Technical University

Page 37: Fokke Dijkstra, Bob Dröge High Performance Computing and … · 2015-10-02 · 24-9-2012 Using the Millipede cluster - I Fokke Dijkstra, Bob Dröge High Performance Computing and

24-9-2012

Cluster in more detail

userfront-end node

internet fast network

Cluster nodes

CIT

Office somewhere

Page 38: Fokke Dijkstra, Bob Dröge High Performance Computing and … · 2015-10-02 · 24-9-2012 Using the Millipede cluster - I Fokke Dijkstra, Bob Dröge High Performance Computing and

24-9-2012

How does it work?

internet

sshscp

Page 39: Fokke Dijkstra, Bob Dröge High Performance Computing and … · 2015-10-02 · 24-9-2012 Using the Millipede cluster - I Fokke Dijkstra, Bob Dröge High Performance Computing and

24-9-2012

Why a command line?

› Easier over the network

› Scripts are easier to run many times

› Instead of:

› One can use a small program

Page 40: Fokke Dijkstra, Bob Dröge High Performance Computing and … · 2015-10-02 · 24-9-2012 Using the Millipede cluster - I Fokke Dijkstra, Bob Dröge High Performance Computing and

24-9-2012

SSH

› SSH protocol used to connect to the cluster

• Standard interface for Unix systems

• Encrypted network traffic

› Software for Windows called PuTTY:• http://www.chiark.greenend.org.uk/~sgtatham/putty/

• Open source software, freely available

• Already installed on UWP

› Use terminal window under Linux or Mac OS X

› Normally university account id used (p- or s-number)› Hostname millipede.service.rug.nl

Page 41: Fokke Dijkstra, Bob Dröge High Performance Computing and … · 2015-10-02 · 24-9-2012 Using the Millipede cluster - I Fokke Dijkstra, Bob Dröge High Performance Computing and

24-9-2012

Millipede overview

› 4 node types

• 1 login node

• 12 cores, 24 GB memory

• 231 standard nodes

• 12 cores, 24 GB memory

• 16 large nodes

• 24 cores, 128 GB memory

• 1 special SMP node

• 64 cores, 478 GB memory

› Network

• 20 Gbit/s Infiniband network

› 3 File systems

Standard modern desktop PC

• 4 cores

• 8 GB memory

Page 42: Fokke Dijkstra, Bob Dröge High Performance Computing and … · 2015-10-02 · 24-9-2012 Using the Millipede cluster - I Fokke Dijkstra, Bob Dröge High Performance Computing and

24-9-2012

Accessing storage on UWP

› Cluster storage /home and /data can be reached on UWP

• Documentation available• http://www.rug.nl/cit/hpcv/publications/docs/millipede_man/cl

uster-UWP.pdf

• Example: Start -> Run

Page 43: Fokke Dijkstra, Bob Dröge High Performance Computing and … · 2015-10-02 · 24-9-2012 Using the Millipede cluster - I Fokke Dijkstra, Bob Dröge High Performance Computing and

24-9-2012

File transfer clients

› Windows

• WinSCP is a file transfer program for Windows

• http://winscp.net

• Open source, freely available

› Linux

• Command line scp or sftp

• Nautilus file manager, Konqueror (fish://), FileZilla

› Mac OS X (not tested)

• Command line scp or sftp

• CyberDuck (Open source), FileZilla (Open source), Fetch (Free academic license)

Page 44: Fokke Dijkstra, Bob Dröge High Performance Computing and … · 2015-10-02 · 24-9-2012 Using the Millipede cluster - I Fokke Dijkstra, Bob Dröge High Performance Computing and

24-9-2012

/home file system

› Each user has a directory within /home

• E.g /home/f205229

› 2 * 7 TB in total (mirrored)

› Backed up

› Default quota of 10 GB per user

• Prevents system from uncontrolled filling up

• Contact us if you need more space

• Use “quota” command to see the status

› Shared over NFS to all nodes

• Limited performance

Page 45: Fokke Dijkstra, Bob Dröge High Performance Computing and … · 2015-10-02 · 24-9-2012 Using the Millipede cluster - I Fokke Dijkstra, Bob Dröge High Performance Computing and

24-9-2012

/data & /data/scratch

› 110 TB file system based on Fraunhofer Parallel File System (FhGFS)

› Faster than /home for very large files

› Currently slow for small file access

› /data

• No active quota, please limit yourself to 200 GB

• Contact us if you need more space

› /data/scratch

• Temporary storage

• May be cleaned by system administrators

Page 46: Fokke Dijkstra, Bob Dröge High Performance Computing and … · 2015-10-02 · 24-9-2012 Using the Millipede cluster - I Fokke Dijkstra, Bob Dröge High Performance Computing and

24-9-2012

/local

› Local disk on all of the nodes

› Only reachable on that specific node

› Needs to be cleaned up within the job at the end

› $TMPDIR points to a directory that is created per job on the master node of the job

• Automatically removed after the job has finished

Page 47: Fokke Dijkstra, Bob Dröge High Performance Computing and … · 2015-10-02 · 24-9-2012 Using the Millipede cluster - I Fokke Dijkstra, Bob Dröge High Performance Computing and

24-9-2012

› What can it do for me?

Page 48: Fokke Dijkstra, Bob Dröge High Performance Computing and … · 2015-10-02 · 24-9-2012 Using the Millipede cluster - I Fokke Dijkstra, Bob Dröge High Performance Computing and

24-9-2012

For whom?

› Account available for University staff & students

• Undergraduate students through supervisor/teacher

• Provide contact details and short description of planned use

› Access through ssh protocol

• millipede.service.rug.nl

Page 49: Fokke Dijkstra, Bob Dröge High Performance Computing and … · 2015-10-02 · 24-9-2012 Using the Millipede cluster - I Fokke Dijkstra, Bob Dröge High Performance Computing and

24-9-2012

What can it be used for?

› Applications must be able to run under Linux

• Compile the application yourself

• Preinstalled applications

• matlab, R, gromacs, …

• Run anywhere languages

• Java, Python, Perl, ….

• Some programs can be run under a windows emulator

• License must be available

Page 50: Fokke Dijkstra, Bob Dröge High Performance Computing and … · 2015-10-02 · 24-9-2012 Using the Millipede cluster - I Fokke Dijkstra, Bob Dröge High Performance Computing and

24-9-2012

No direct user interaction

› Applications must run without user intervention

• You don’t know when the application will start

• Makes running multiple times difficult

› No graphical user interface

• If no user interaction is necessary the output can be sent to a dummy display

• Use case: saving matlab and R plots

› Input and output through files

Page 51: Fokke Dijkstra, Bob Dröge High Performance Computing and … · 2015-10-02 · 24-9-2012 Using the Millipede cluster - I Fokke Dijkstra, Bob Dröge High Performance Computing and

Licenses

› If you want to use commercial software a license may be necessary

› Provided by the university

• May be installed system wide

• Problem with Linux versions or number of licenses

› Provided by yourself

• Install software in home directory

› Other cases may be more difficult

24-9-2012

Page 52: Fokke Dijkstra, Bob Dröge High Performance Computing and … · 2015-10-02 · 24-9-2012 Using the Millipede cluster - I Fokke Dijkstra, Bob Dröge High Performance Computing and

24-9-2012

Use cases

› Application areas that can be used:• Data analysis using Matlab or R• Molecular dynamics using Gromacs• fMRI image analysis• Quantum chemistry• High energy physics using root, geant4• Run your own software in Fortran, C, C++

› General scenarios:• Long running calculations• Parallel calculations• Many calculations

Page 53: Fokke Dijkstra, Bob Dröge High Performance Computing and … · 2015-10-02 · 24-9-2012 Using the Millipede cluster - I Fokke Dijkstra, Bob Dröge High Performance Computing and

24-9-2012

› Where is my (insert favourite software package)?Available software, using the module system

Page 54: Fokke Dijkstra, Bob Dröge High Performance Computing and … · 2015-10-02 · 24-9-2012 Using the Millipede cluster - I Fokke Dijkstra, Bob Dröge High Performance Computing and

24-9-2012

Modules environment

› Software organised through modules system

› Environment set for software when module is loaded

› Useful commands:

• module avail

• module list

• module add

• module remove

• module initadd

Page 55: Fokke Dijkstra, Bob Dröge High Performance Computing and … · 2015-10-02 · 24-9-2012 Using the Millipede cluster - I Fokke Dijkstra, Bob Dröge High Performance Computing and

24-9-2012

Module examples

$ module avail

---------------------------- /cm/local/modulefiles -----

cluster-tools/5.0 freeipmi/0.7.11 shared

cmd module-info use.own

dot null version

---------------------------- /cm/shared/modulefiles ----

R/2.10.1 hdf5/1.8.5

R-tcltk/2.10.1 hpl/2.0

acml/gcc/64/4.3.0 intel/compiler/32/11.

acml/gcc/mp/64/4.3.0 intel/compiler/64/11.

....

$ icc

-bash: icc: command not found

$ module add intel/compiler/64

$ icc --version

icc (ICC) 11.1 20090630

Copyright (C) 1985-2009 Intel Corporation. All rights reserved.

$ module remove intel

Page 56: Fokke Dijkstra, Bob Dröge High Performance Computing and … · 2015-10-02 · 24-9-2012 Using the Millipede cluster - I Fokke Dijkstra, Bob Dröge High Performance Computing and

24-9-2012

Useful software

› Compilers for Fortran, C and C++

• GNU compilers

• Intel compilers

› Other languages

• Java, Python, Perl

› MPI libraries

• OpenMPI, MVAPICH2

› Scheduler software – torque, maui

› Matlab and R

Page 57: Fokke Dijkstra, Bob Dröge High Performance Computing and … · 2015-10-02 · 24-9-2012 Using the Millipede cluster - I Fokke Dijkstra, Bob Dröge High Performance Computing and

24-9-2012

› But I only see a single computer!

Page 58: Fokke Dijkstra, Bob Dröge High Performance Computing and … · 2015-10-02 · 24-9-2012 Using the Millipede cluster - I Fokke Dijkstra, Bob Dröge High Performance Computing and

24-9-2012

Login node

› Front-end node

› Used for access to the cluster

• Login

• Data transfers

• Job submission

• Editing & Compiling programs

• (Very) small tests

• Hardware same as standard nodes

Page 59: Fokke Dijkstra, Bob Dröge High Performance Computing and … · 2015-10-02 · 24-9-2012 Using the Millipede cluster - I Fokke Dijkstra, Bob Dröge High Performance Computing and

24-9-2012

Scheduling system (1)

› Users write job descriptions

› Scheduler takes job description and finds matching resource

› Scheduler tries to make optimal use of the resources

› If no resources are available, jobs have to wait in a queue

› Priority in the queue determined by usage of system in the recent past

Page 60: Fokke Dijkstra, Bob Dröge High Performance Computing and … · 2015-10-02 · 24-9-2012 Using the Millipede cluster - I Fokke Dijkstra, Bob Dröge High Performance Computing and

24-9-2012

≤ 1 day ≤ 3 days ≤ 10 days

Standardnodes

nodes nodesmedium nodeslong

Big nodes(“quads”)

quads quadsmedium quadslong

Special SMP node

smp smpmedium smplong

≤ 30 minutes (only for testing!)

Standard nodes

short

Scheduling system (2)

Page 61: Fokke Dijkstra, Bob Dröge High Performance Computing and … · 2015-10-02 · 24-9-2012 Using the Millipede cluster - I Fokke Dijkstra, Bob Dröge High Performance Computing and

24-9-2012

Multiple jobs per node

› Nodes have at least 12 cores

› Scheduling full nodes to users that only use a single core is a waste of resources

› Users can still request a complete node if necessary

› Jobs on a single node will influence each other

• Disk space (space and bandwidth)

• Memory (space and bandwidth)

• Can sometimes be problematic

Page 62: Fokke Dijkstra, Bob Dröge High Performance Computing and … · 2015-10-02 · 24-9-2012 Using the Millipede cluster - I Fokke Dijkstra, Bob Dröge High Performance Computing and

24-9-2012

Node types

› 231 12 core nodes

• 2 x 6 core AMD Opteron @ 2.6 GHz

• 24 GB memory

• Bulk of the cluster capacity

› 16 16 core nodes

• 4 x 6 core AMD Opteron @ 2.6 GHz

› Reachable through scheduler

› Default multiple jobs per node!

Infiniband switch

Page 63: Fokke Dijkstra, Bob Dröge High Performance Computing and … · 2015-10-02 · 24-9-2012 Using the Millipede cluster - I Fokke Dijkstra, Bob Dröge High Performance Computing and

24-9-2012

SMP node

› Single 64 core node

• Combination of 8 nodes into a SMP node using ScaleMP

• 8 * 2 * 4 core Intel Xeon E5520 “Nehalem” @ 2.3 GHz

• 4 * 48 + 4 * 96 GB memory = 576 GB (- ~ 90 GB used for ScaleMP, including vSMP cache)

› Reachable through scheduler

› Large shared memory jobs

• Large memory

• More cores

› Multiple jobs on the node

Page 64: Fokke Dijkstra, Bob Dröge High Performance Computing and … · 2015-10-02 · 24-9-2012 Using the Millipede cluster - I Fokke Dijkstra, Bob Dröge High Performance Computing and

24-9-2012

› What is a job script?

Page 65: Fokke Dijkstra, Bob Dröge High Performance Computing and … · 2015-10-02 · 24-9-2012 Using the Millipede cluster - I Fokke Dijkstra, Bob Dröge High Performance Computing and

24-9-2012

Job script

› Job script tells the system what you want to do

• List of steps, small program

› Includes requirements needed to be able to run it

• Amount of memory necessary

• How many cores or nodes you need

• Time you will need before it is finished

Page 66: Fokke Dijkstra, Bob Dröge High Performance Computing and … · 2015-10-02 · 24-9-2012 Using the Millipede cluster - I Fokke Dijkstra, Bob Dröge High Performance Computing and

24-9-2012

Job requirements

› Can be put in job script using#PBS

› Requirements specified using -l option:

• Wall clock time

• #PBS -l walltime=hh:mm:ss

• Number of nodes and cores

• #PBS -l nodes=n:ppn=m

› Specific node types using -q option:

• #PBS -q quads

Page 67: Fokke Dijkstra, Bob Dröge High Performance Computing and … · 2015-10-02 · 24-9-2012 Using the Millipede cluster - I Fokke Dijkstra, Bob Dröge High Performance Computing and

24-9-2012

Memory requirements

› Memory requirements can be specified using#PBS -l mem=nGBn is the amount of memory you need

› If you need a lot of memory and a few cores• Consider using the quad nodes• Consider requesting the number of cores

corresponding to the amount of memory› 2 GB / core on standard nodes› 5.3 GB /core on quad nodes› Currently not everyone gives a memory requirement

• Scheduling may be wrong

Page 68: Fokke Dijkstra, Bob Dröge High Performance Computing and … · 2015-10-02 · 24-9-2012 Using the Millipede cluster - I Fokke Dijkstra, Bob Dröge High Performance Computing and

24-9-2012

Job properties

› Also using #PBS lines or on the command line

› Name of the job

• #PBS -N name

› Combine standard output and error into a single file

• #PBS -j option

• oe : Merge into standard output file

• eo: Merge into standard error file

› Sending e-mails

• #PBS -m options

• Options is a combination of n (no), b (begin), e (end) and/or a (aborted)

• Mail can be problematic in case of Millipede problems

Page 69: Fokke Dijkstra, Bob Dröge High Performance Computing and … · 2015-10-02 · 24-9-2012 Using the Millipede cluster - I Fokke Dijkstra, Bob Dröge High Performance Computing and

24-9-2012

Job script itself

› Contains Linux commands

• cd, mkdir, etc.

› Sample script:

#PBS -l nodes=1:ppn=2#PBS -l walltime=00:30:00#PBS -N testjob

cd $PBS_O_WORKDIR./myprog

Page 70: Fokke Dijkstra, Bob Dröge High Performance Computing and … · 2015-10-02 · 24-9-2012 Using the Millipede cluster - I Fokke Dijkstra, Bob Dröge High Performance Computing and

24-9-2012

Useful environment variables

$HOME: your home directory

$USER: your user name

$TMPDIR: Temporary directory created for your job. Removed after your job has finished

$PBS_O_WORKDIR: Directory from which the job was submitted

$PBS_NODEFILE: Points to a file with a list of the allocated nodes (one entry for each core on the node)

$PBS_JOBID: Id of job, useful for creating unique files or directories for a job

Page 71: Fokke Dijkstra, Bob Dröge High Performance Computing and … · 2015-10-02 · 24-9-2012 Using the Millipede cluster - I Fokke Dijkstra, Bob Dröge High Performance Computing and

24-9-2012

Submitting jobs

› qsub jobscript

$ qsub testjob1870565.millipede.cm.cluster

Job id

Page 72: Fokke Dijkstra, Bob Dröge High Performance Computing and … · 2015-10-02 · 24-9-2012 Using the Millipede cluster - I Fokke Dijkstra, Bob Dröge High Performance Computing and

24-9-2012

Checking job status (1)

› qstat

$ qstat Job id Name User Time Use S Queue------------------------- ---------------- --------------- -------- - -----1820058.millipede Dimeric-4 p552330 21054:10 R nodeslong 1820631.millipede SM16-DISORDER p552330 5204:21: R nodeslong 1824996.millipede Complex p982535 26127:01 R quadslong 1832677.millipede DEMIX p834345 2710:47: R nodeslong 1835800.millipede BIG-DUPC-3 p552330 18479:20 R nodeslong 1835801.millipede BIG-DUPC-4 p552330 17856:18 R nodeslong 1835802.millipede BIG-DUPC-5 p552330 17798:52 R nodeslong 1843168.millipede aaTHLmem_45 s8656343 1438:06: R nodeslong 1843186.millipede walp-dupc-nochol p539241 2729:18: R nodeslong 1846698.millipede DEMIXTC p834345 2525:40: R nodeslong 1846798.millipede RAFT-DIGAL p834345 2516:58: R nodeslong 1847258.millipede testjob peter 0 Q nodes 1847358.millipede 120cpus p523301 11484:12 R nodeslong 1849156.millipede genome p735434 93:30:29 R nodeslong 1849216.millipede myjob p724263 105:51:1 R nodeslong 1849376.millipede 120cpus p523301 11037:28 R nodeslong 1849433.millipede PMF0 p633535 1103:37: R nodeslong 1849434.millipede PMF1 p633535 1103:29: R nodeslong .........

Page 73: Fokke Dijkstra, Bob Dröge High Performance Computing and … · 2015-10-02 · 24-9-2012 Using the Millipede cluster - I Fokke Dijkstra, Bob Dröge High Performance Computing and

24-9-2012

Checking job status (2)

$ qstat -u peter

millipede.cm.cluster: Req'd Req'd Elap

Job ID Username Queue Jobname SessID NDS TSK Memory Time S Time-------------------- -------- -------- ---------------- ------ ----- --- ------ ----- - -----1847258.milliped peter nodes testjob -- 1 1 -- 01:00 Q --

Status:

Q: queued (waiting)

R: running

E: finishing

H: halted (waiting for another job to finish)

Page 74: Fokke Dijkstra, Bob Dröge High Performance Computing and … · 2015-10-02 · 24-9-2012 Using the Millipede cluster - I Fokke Dijkstra, Bob Dröge High Performance Computing and

24-9-2012

Checking job status (3)

› showq (can also use -u option)$ showqACTIVE JOBS--------------------JOBNAME USERNAME STATE PROC REMAINING STARTTIME

1869946 dijkstra Running 2 00:02:12 Mon Jun 27 13:10:521869948 dijkstra Running 2 00:04:00 Mon Jun 27 13:12:401869949 dijkstra Running 2 00:05:18 Mon Jun 27 13:13:581869951 dijkstra Running 2 00:05:18 Mon Jun 27 13:13:581869952 dijkstra Running 2 00:05:18 Mon Jun 27 13:13:581869953 dijkstra Running 2 00:05:18 Mon Jun 27 13:13:581869969 dijkstra Running 2 00:07:23 Mon Jun 27 13:16:031869970 dijkstra Running 2 00:07:23 Mon Jun 27 13:16:031870063 dijkstra Running 2 00:53:54 Mon Jun 27 14:02:341870064 dijkstra Running 2 00:53:54 Mon Jun 27 14:02:341870065 dijkstra Running 2 00:53:54 Mon Jun 27 14:02:341870066 dijkstra Running 2 00:54:28 Mon Jun 27 14:03:081870361 dijkstra Running 2 1:12:54 Mon Jun 27 14:21:341870362 dijkstra Running 2 1:13:36 Mon Jun 27 14:22:161870363 dijkstra Running 2 1:13:36 Mon Jun 27 14:22:161870386 dijkstra Running 2 1:13:36 Mon Jun 27 14:22:161870406 dijkstra Running 2 1:16:36 Mon Jun 27 14:25:16......

Page 75: Fokke Dijkstra, Bob Dröge High Performance Computing and … · 2015-10-02 · 24-9-2012 Using the Millipede cluster - I Fokke Dijkstra, Bob Dröge High Performance Computing and

24-9-2012

Checking job status (4)

...

...1867510 p539241 Running 24 9:16:45:00 Mon Jun 27 07:54:321867515 p539241 Running 24 9:16:45:21 Mon Jun 27 07:54:531869709 s7104947 Running 12 9:19:46:57 Mon Jun 27 10:56:29

281 Active Jobs 2748 of 3208 Processors Active (85.66%)224 of 247 Nodes Active (90.69%)

IDLE JOBS----------------------JOBNAME USERNAME STATE PROC WCLIMIT QUEUETIME

1850797 p552330 Idle 120 10:00:00:00 Fri Jun 24 16:58:271868042 p552330 Idle 24 10:00:00:00 Mon Jun 27 08:41:181868046 p552330 Idle 24 10:00:00:00 Mon Jun 27 08:41:311868049 p552330 Idle 24 10:00:00:00 Mon Jun 27 08:41:431868052 p552330 Idle 24 10:00:00:00 Mon Jun 27 08:41:541868055 p552330 Idle 24 10:00:00:00 Mon Jun 27 08:42:051868059 p552330 Idle 24 10:00:00:00 Mon Jun 27 08:42:211868063 p552330 Idle 24 10:00:00:00 Mon Jun 27 08:42:341868066 p552330 Idle 24 10:00:00:00 Mon Jun 27 08:42:451868069 p552330 Idle 24 10:00:00:00 Mon Jun 27 08:42:551868072 p552330 Idle 24 10:00:00:00 Mon Jun 27 08:43:061868074 p552330 Idle 24 10:00:00:00 Mon Jun 27 08:43:161868078 p552330 Idle 24 10:00:00:00 Mon Jun 27 08:43:27......

Page 76: Fokke Dijkstra, Bob Dröge High Performance Computing and … · 2015-10-02 · 24-9-2012 Using the Millipede cluster - I Fokke Dijkstra, Bob Dröge High Performance Computing and

24-9-2012

Checking the results

› If the job disappears from qstat it has finished

› Output files written to same directory as from which the job was submitted

› jobname.ojobnum and jobname.ejobnum filese.g. testjob.o1873505 and testjob.e1873505

› .o file contains everything written to standard output

› .e file contains everything written to standard error

Page 77: Fokke Dijkstra, Bob Dröge High Performance Computing and … · 2015-10-02 · 24-9-2012 Using the Millipede cluster - I Fokke Dijkstra, Bob Dröge High Performance Computing and

24-9-2012

Oops, I did not want to run that!

› qdel

$ qsub testjob1870679.millipede.cm.cluster$ qstat -u peter

millipede.cm.cluster: Req'd Req'd Elap

Job ID Username Queue Jobname SessID NDS TSK Memory Time S Time-------------------- -------- -------- ---------------- ------ ----- --- ------ ----- - -----1847258.milliped peter nodes testjob -- 1 1 -- 01:00 Q --1870679.milliped peter short test -- 1 4 500mb 00:20 Q --$ qdel 1870679$

Page 78: Fokke Dijkstra, Bob Dröge High Performance Computing and … · 2015-10-02 · 24-9-2012 Using the Millipede cluster - I Fokke Dijkstra, Bob Dröge High Performance Computing and

24-9-2012

A real example

› A job script that runs R code:

#PBS -N R_job#PBS -l walltime=00:01:00#PBS -l nodes=1:ppn=1#PBS -l mem=1GB#PBS -q short

cd $PBS_O_WORKDIRmodule load R/2.15.0Rscript myscript.r

Page 79: Fokke Dijkstra, Bob Dröge High Performance Computing and … · 2015-10-02 · 24-9-2012 Using the Millipede cluster - I Fokke Dijkstra, Bob Dröge High Performance Computing and

24-9-2012

Another real example

› A job script that runs Matlab code:

#PBS -N matlab_job#PBS –l nodes=1:ppn=1#PBS –l walltime=00:00:30#PBS -j oe#PBS –q short

cd $PBS_O_WORKDIRmodule load matlabmatlab -nodisplay -r mycode

Code in file mycode.m

Page 80: Fokke Dijkstra, Bob Dröge High Performance Computing and … · 2015-10-02 · 24-9-2012 Using the Millipede cluster - I Fokke Dijkstra, Bob Dröge High Performance Computing and

24-9-2012

Remarks on Matlab

› Matlab needs a license to be able to run

› Number of concurrent users is limited

› Matlab code can be compiled using Matlab compiler

• Running the compiled code does not require a license

› Parallel computing toolbox supported

• Can use up to 8 cores

• Parallel Matlab code cannot be compiled

Page 81: Fokke Dijkstra, Bob Dröge High Performance Computing and … · 2015-10-02 · 24-9-2012 Using the Millipede cluster - I Fokke Dijkstra, Bob Dröge High Performance Computing and

24-9-2012

Graphical output

Page 82: Fokke Dijkstra, Bob Dröge High Performance Computing and … · 2015-10-02 · 24-9-2012 Using the Millipede cluster - I Fokke Dijkstra, Bob Dröge High Performance Computing and

24-9-2012

Running applications with a GUI

› Only on the login node!

› Needs fast network connection

• May work very slow at home

› Linux uses X protocol for displaying graphics

› X graphics can be shown on another display

› Needs X server software on your desktop

• Xming (Free version available), easier to usehttp://www.straightrunning.com/XmingNotes/

• X-deep/32 (UWP)

Page 83: Fokke Dijkstra, Bob Dröge High Performance Computing and … · 2015-10-02 · 24-9-2012 Using the Millipede cluster - I Fokke Dijkstra, Bob Dröge High Performance Computing and

24-9-2012

X-Deep/32 (1)

› Start: X-Deep/32 X-server

off

Page 84: Fokke Dijkstra, Bob Dröge High Performance Computing and … · 2015-10-02 · 24-9-2012 Using the Millipede cluster - I Fokke Dijkstra, Bob Dröge High Performance Computing and

24-9-2012

X-Deep/32 (2)

Select X-Deep/32 icon

options

Page 85: Fokke Dijkstra, Bob Dröge High Performance Computing and … · 2015-10-02 · 24-9-2012 Using the Millipede cluster - I Fokke Dijkstra, Bob Dröge High Performance Computing and

24-9-2012

X-Deep/32 (3)

off

Page 86: Fokke Dijkstra, Bob Dröge High Performance Computing and … · 2015-10-02 · 24-9-2012 Using the Millipede cluster - I Fokke Dijkstra, Bob Dröge High Performance Computing and

24-9-2012

Xming

Keep blocking since we will be using

connections through ssh

Page 87: Fokke Dijkstra, Bob Dröge High Performance Computing and … · 2015-10-02 · 24-9-2012 Using the Millipede cluster - I Fokke Dijkstra, Bob Dröge High Performance Computing and

24-9-2012

PuTTY X connections

• Only necessary when you want to display programs with a graphical user interface on your local desktop

• You will need an X server for that

Page 88: Fokke Dijkstra, Bob Dröge High Performance Computing and … · 2015-10-02 · 24-9-2012 Using the Millipede cluster - I Fokke Dijkstra, Bob Dröge High Performance Computing and

Example: Matlab

› To start Matlab on the login node:module load matlabmatlab

24-9-2012

› Can only be used for simple tests, preparing jobs, viewing output, etcetera

Page 89: Fokke Dijkstra, Bob Dröge High Performance Computing and … · 2015-10-02 · 24-9-2012 Using the Millipede cluster - I Fokke Dijkstra, Bob Dröge High Performance Computing and

24-9-2012

› Help!

Page 90: Fokke Dijkstra, Bob Dröge High Performance Computing and … · 2015-10-02 · 24-9-2012 Using the Millipede cluster - I Fokke Dijkstra, Bob Dröge High Performance Computing and

24-9-2012

Support

› Support through CIT central service desk

• Phone: 3232

• E-mail: [email protected]

› Documentation online• http://www.rug.nl/cit/hpcv/publications/docs/millipede

› Comments and questions are always welcome

Page 91: Fokke Dijkstra, Bob Dröge High Performance Computing and … · 2015-10-02 · 24-9-2012 Using the Millipede cluster - I Fokke Dijkstra, Bob Dröge High Performance Computing and

24-9-2012

› I want to know more!

Page 92: Fokke Dijkstra, Bob Dröge High Performance Computing and … · 2015-10-02 · 24-9-2012 Using the Millipede cluster - I Fokke Dijkstra, Bob Dröge High Performance Computing and

24-9-2012

Useful documents

› Introduction to Linux by Machteld Garrels: http://tldp.org/LDP/intro-linux/html/index.html

› Bash shell guide by Machteld Garrels:http://tldp.org/LDP/Bash-Beginners-Guide/html/index.html

› Linux guide: http://www.tuxfiles.org

› Linux command line: http://linuxcommand.org/

Page 93: Fokke Dijkstra, Bob Dröge High Performance Computing and … · 2015-10-02 · 24-9-2012 Using the Millipede cluster - I Fokke Dijkstra, Bob Dröge High Performance Computing and

24-9-2012

Questions?