getting started with hpc on iceberg
DESCRIPTION
Getting Started with HPC On Iceberg. Michael Griffiths and Deniz Savas Corporate Information and Computing Services The University of Sheffield www.sheffield.ac.uk/wrgrid. Introduction to Iceberg Getting connected Basic Linux Using the Linux Operating System - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: Getting Started with HPC On Iceberg](https://reader036.vdocument.in/reader036/viewer/2022081515/56812bc8550346895d901663/html5/thumbnails/1.jpg)
Getting Started with HPCOn Iceberg
Michael Griffiths and Deniz Savas
Corporate Information and Computing Services
The University of Sheffield
www.sheffield.ac.uk/wrgrid
![Page 2: Getting Started with HPC On Iceberg](https://reader036.vdocument.in/reader036/viewer/2022081515/56812bc8550346895d901663/html5/thumbnails/2.jpg)
• Introduction to Iceberg– Getting connected– Basic Linux
• Using the Linux Operating System• Running Applications and Jobs on Iceberg• How to distribute work across the cluster• The N8 Facility• Getting Help
Outline
![Page 3: Getting Started with HPC On Iceberg](https://reader036.vdocument.in/reader036/viewer/2022081515/56812bc8550346895d901663/html5/thumbnails/3.jpg)
•Cluster Grid•Computing clusters ( e.g. iceberg )
•Enterprise Grid, Campus Grid, Intra-Grid
•Departmental clusters, servers and PC network
•Cloud, Utility Grid•Access resources over internet on demand
•Global Grid, Inter-grid•White Rose Grid, National Grid Service, Particle Physics Data Grid
Types of Grids
![Page 4: Getting Started with HPC On Iceberg](https://reader036.vdocument.in/reader036/viewer/2022081515/56812bc8550346895d901663/html5/thumbnails/4.jpg)
Iceberg Cluster
HEAD NODE1
Iceberg(1)
HEAD NODE2
Iceberg(2)
Worker node
Worker node
Worker node
Worker node
Worker node
Worker node
Worker node
Worker node
Worker node
Worker node
Worker node
Worker node
There are 232 worker machines
in the cluster
There are two head-nodes for the iceberg cluster
loginloginlogin
qsh,qsub,qrshqsh,qsub,qrsh
All workers share the same user filestore
![Page 5: Getting Started with HPC On Iceberg](https://reader036.vdocument.in/reader036/viewer/2022081515/56812bc8550346895d901663/html5/thumbnails/5.jpg)
iceberg cluster specificationsiceberg cluster specificationsAMD-based nodes containing;
• 96 nodes each with 4 cores and 16 GB of memory
• 31 nodes each with 8 cores and 32 GB of memory
• TOTAL AMD CORES = 632, TOTAL MEMORY = 2528 GBThe 8-core nodes are connected to each other via 16 GBits/sec infiniband for MPI jobsThe 4-core nodes are connected via the much slower "1 Gbits/sec" ethernet connections for MPI jobs. Scratch space on each node is 400 GBytes
Intel Westmere based nodes all infiniband connected, containing;
• 103 nodes each with 12 cores and 24 GB of memory ( i.e. 2 * 6-core Intel X5650 )
• 4 nodes each with 12 cores and 48 GB of memory
• 8 Nvidia Tesla Fermi M2070s GPU units for GPU programming
• TOTAL INTEL CPU CORES = 1152 , TOTAL MEMORY = 2400 GBScratch space on each node is 400 GBTotal GPU memory = 48 GBEach GPU unit is capable of about 1TFlop of single precision floating point performance, or 0.5TFlops at double precision. Hence yielding maximum GPU processing power of upto 8 TFlops in total.
![Page 6: Getting Started with HPC On Iceberg](https://reader036.vdocument.in/reader036/viewer/2022081515/56812bc8550346895d901663/html5/thumbnails/6.jpg)
Introduction to Linux on Iceberg
• What are UNIX and Linux?
• The Shell
• Getting Connected
• Basic UNIX/Linux Commands▬ Working with Directories▬ Editing files
• Help!
• Running Programs
• Exercise
![Page 7: Getting Started with HPC On Iceberg](https://reader036.vdocument.in/reader036/viewer/2022081515/56812bc8550346895d901663/html5/thumbnails/7.jpg)
What is UNIX/Linux ?
• Multi-Tasking O/S• Multi-User O/S• Available on a range of Computers
![Page 8: Getting Started with HPC On Iceberg](https://reader036.vdocument.in/reader036/viewer/2022081515/56812bc8550346895d901663/html5/thumbnails/8.jpg)
History of UNIX/Linux?.
• Unix operating system was developed around 1969 in the Bell Labs
• Originally written using C• Around 1990 Linus Torvalds of Helsinki University
started off a freely available academic version of Unix
• Linux is the Antidote to a microsoft dominated future
![Page 9: Getting Started with HPC On Iceberg](https://reader036.vdocument.in/reader036/viewer/2022081515/56812bc8550346895d901663/html5/thumbnails/9.jpg)
Which UNIX?• SunOS Sun Microsystems – now OpenIndiana,
OpenSolaris from Oracle• IRIX Silicon Graphics• HP-UX Hewlett Packard• Linux For IBM PC compatibles
There are a number of certification bodies with published standards and test suites to ensure quality of products. Such as;
• Posix: Portable Operating System Interface ( IEEE standard).
![Page 10: Getting Started with HPC On Iceberg](https://reader036.vdocument.in/reader036/viewer/2022081515/56812bc8550346895d901663/html5/thumbnails/10.jpg)
Linux and GNU• Linux is an implementation of Unix• Linux/Unix operating system is written in ‘C’• Linux is not part of the GNU project but uses the same licensing
agreements• Many of the linux utilities and tools are taken from the GNU
project. • There are many flavours of linux distributions. The mix of the
kernel (linux) with the utilities (GNU and other) and the installation procedure determine the flavour. Some of these are;– Fedora– SuSE– Redhat– Debian – Mandrake– Knoppix
![Page 11: Getting Started with HPC On Iceberg](https://reader036.vdocument.in/reader036/viewer/2022081515/56812bc8550346895d901663/html5/thumbnails/11.jpg)
UNIX Internals (Simplified)
• Kernel– System Internals
• Shell– Command Interpreter– Programming language
• File System• Process Management
![Page 12: Getting Started with HPC On Iceberg](https://reader036.vdocument.in/reader036/viewer/2022081515/56812bc8550346895d901663/html5/thumbnails/12.jpg)
UNIX Shells
• sh Bourne Shell (Original Shell) (Steven Bourne of AT&T)• bash Bourne Again Shell (GNU Improved Bourne Shell)• csh C-Shell (C-like Syntax)(Bill Joy of Univ. of California)• ksh Korn-Shell (Bourne+some C-shell)(David Korn of AT&T)• tcsh Turbo C-Shell (More User Friendly C-Shell).
You can switch from one shell to another by just typing the name of the shell. exit return you back to previous shell.
![Page 13: Getting Started with HPC On Iceberg](https://reader036.vdocument.in/reader036/viewer/2022081515/56812bc8550346895d901663/html5/thumbnails/13.jpg)
Getting an account
All staff and research students are entitled to use iceberg
For Registration See: http://www.shef.ac.uk/wrgrid/register
Staff can have an account by simply emailing [email protected]
![Page 14: Getting Started with HPC On Iceberg](https://reader036.vdocument.in/reader036/viewer/2022081515/56812bc8550346895d901663/html5/thumbnails/14.jpg)
passwords
• In normal linux environment the passwd command can be used to change the user passwords. However, because we manage passwords centrally this command will not work on iceberg.
• If you wish to change your iceberg password you will have to do this via a web interface at the following URL:
http://www.shef.ac.uk/cics/password
![Page 15: Getting Started with HPC On Iceberg](https://reader036.vdocument.in/reader036/viewer/2022081515/56812bc8550346895d901663/html5/thumbnails/15.jpg)
Access iceberg
Remote logging in
Terminal access is described at: http://www.shef.ac.uk/wrgrid/using/access
Recommended access is via any browser at: www.shef.ac.uk/wrgrid This uses Sun Global Desktop ( All platforms, Graphics-capable)
Also possible:• Using an X-Windows client ( MS Windows, Graphics-capable)
▬ Exceed 3D ▬ Cygwin
• Various ssh clients (MS Windows, Terminal-only )▬ putty, SSH
Note: ssh clients can also be used in combination with Exceed or Cygwin to enable graphics capability. Above web page describes how this can be achieved.
![Page 16: Getting Started with HPC On Iceberg](https://reader036.vdocument.in/reader036/viewer/2022081515/56812bc8550346895d901663/html5/thumbnails/16.jpg)
Access icebergfrom MAC or Linux platforms
• The web browser method of access ( as for Windows platforms) also works on these platforms.
• More customary and efficient method of access is by using the ssh command from a command-shell.Example: ssh –X [email protected]
Note1: -X parameter is needed to make sure that you can use the graphics or gui capabilities of the software on iceberg.Note2: Depending on the configuration of your workstation you may also have to issue the command : xhost + before the ssh command.
![Page 17: Getting Started with HPC On Iceberg](https://reader036.vdocument.in/reader036/viewer/2022081515/56812bc8550346895d901663/html5/thumbnails/17.jpg)
Basic X Concepts• X Server runs on local machine
– PC Exceed, Cygwin, Xming
– UNIX WorkstationIncluded in OS
– Apple Mac Exodus
• X Client runs on remote machine– Graphical Application
• xterm
• xcalc
• Modelling and visualisation packages etc.
![Page 18: Getting Started with HPC On Iceberg](https://reader036.vdocument.in/reader036/viewer/2022081515/56812bc8550346895d901663/html5/thumbnails/18.jpg)
Multiple ssh or xterm shells
There are no limits to the number of ssh or xterm windows one can start simultaneously by methods described in the previous slides. You may also start extra xterm windows from the host by simply typing xterm &
On iceberg we also have a local command named Xterm that starts up an xterm window with nicer to use parameters. On iceberg we strongly recommend that you use Qsh rather than xterm or Xterm command so as to make use of a free worker node. Qsh will act like Xterm but will make use of a worker node.
Typing exit will terminate an xterm or ssh session neatly.This will also close the xterm window but not the ssh window. ssh
windows can be closed via the file exit menu.
![Page 19: Getting Started with HPC On Iceberg](https://reader036.vdocument.in/reader036/viewer/2022081515/56812bc8550346895d901663/html5/thumbnails/19.jpg)
Start session on headnode
Start an interactive session on a worker
Help
Start Application on a Worker
Web browser method of access: Sun Global Desktop
![Page 20: Getting Started with HPC On Iceberg](https://reader036.vdocument.in/reader036/viewer/2022081515/56812bc8550346895d901663/html5/thumbnails/20.jpg)
Operating system and utilities
• Linux Operating System version is-
Scientific Linux, which is RedHat Enterprise 5 based
• Default Shell is- bash
• Available editors-Graphical Editors:
▬ gedit ( not available on headnode)
▬ emacs ▬ nedit
Text editors:▬ vi ▬ pico ( not available on headnode) ▬ nano
![Page 21: Getting Started with HPC On Iceberg](https://reader036.vdocument.in/reader036/viewer/2022081515/56812bc8550346895d901663/html5/thumbnails/21.jpg)
Login Environment
• Default shell is bash ( you can request to change your default shell ).
• On login into iceberg many environment variables are setup automatically to provide a consistent interface and access to software.
• Each user has a .bashrc file in their directory to help setup the user environment.
• Type set to get a list of all the environment variables.
• Change all variables that are in CAPITALS with extreme care.
• Modify/enhance your environment by adding commands to the end of your .bashrc file , such as alias commands. (Again do this with care! )
![Page 22: Getting Started with HPC On Iceberg](https://reader036.vdocument.in/reader036/viewer/2022081515/56812bc8550346895d901663/html5/thumbnails/22.jpg)
Some basic rules
• Unix is case sensitive.• Commands are in lower case.• Backspace and/or Del Keys correct typing errors.
If the terminal parameters are not correctly set; try Ctrl+H
• Ctrl+C Aborts a program or command.• You can use the arrow keys to recall previous
commands, optionally edit and execute them.
![Page 23: Getting Started with HPC On Iceberg](https://reader036.vdocument.in/reader036/viewer/2022081515/56812bc8550346895d901663/html5/thumbnails/23.jpg)
Key combinations in bash
Ctrl+A Move cursor to beginning of line
Ctrl+C <Break> Stop running program
Ctrl+D <Logout>
Ctrl+E Move cursor to end of line
Ctrl+H Delete character to the left
Ctrl+Z Suspend the program
Arrow left/right Move pointer left/right to insert chars etc.
Arrow up/down Recall previous commands so as to edit them andsubmit them by pressing ENTER
Tab
Tab tab
Command or filename completion. Auto-complete the command line (to save typing)
If tab does not work tab again to show possibilities
![Page 24: Getting Started with HPC On Iceberg](https://reader036.vdocument.in/reader036/viewer/2022081515/56812bc8550346895d901663/html5/thumbnails/24.jpg)
Format of Unix commands
• command [option ...] [filename ...]
eg: ls
ls -l tutorial
more tutorial
![Page 25: Getting Started with HPC On Iceberg](https://reader036.vdocument.in/reader036/viewer/2022081515/56812bc8550346895d901663/html5/thumbnails/25.jpg)
List Directory
• ls list directory• ls -l list directory in long format• ls -a list all (inc. hidden) files
-rw------ l course01 57 Oct 18 11:05 hello.c
Number ofbytes in file
AccessPermissions
Date and time last modified
![Page 26: Getting Started with HPC On Iceberg](https://reader036.vdocument.in/reader036/viewer/2022081515/56812bc8550346895d901663/html5/thumbnails/26.jpg)
Directory Structure
Home directory of user cs4un1 : /home/cs4un1
usr
/ (root)
home
cs4un2cs4un1
When you log in you are positioned in your home directory.The environment variable $HOME is also set to contain this directory name.
![Page 27: Getting Started with HPC On Iceberg](https://reader036.vdocument.in/reader036/viewer/2022081515/56812bc8550346895d901663/html5/thumbnails/27.jpg)
Working with Directories• pwd print working directory• cd change directory
cd move to home directory
cd .. move up one level
cd mydir move into a subdirectory
cd /var/adm move to an absolute directory
• mkdir directory_name create a new directory
• rmdir directory_name delete an empty directory
![Page 28: Getting Started with HPC On Iceberg](https://reader036.vdocument.in/reader036/viewer/2022081515/56812bc8550346895d901663/html5/thumbnails/28.jpg)
Filenames• Filenames can comprise of:
a-z, A-Z alphabetic characters
0-9 digits
.-_+ special characters
mon+tue_01.06-03-96
• Wildcards when referencing files* any character or sequence of characters
? any single character
![Page 29: Getting Started with HPC On Iceberg](https://reader036.vdocument.in/reader036/viewer/2022081515/56812bc8550346895d901663/html5/thumbnails/29.jpg)
Displaying contents of a text_file• more filename
This command will start listing the contents of filename on screen and pause after a screenfull of data. While pausing, use the following characters to control the output.
Spacebar next screenful
n Spacebar : next n lines
Enter next line
b back one screen
n b : back n screen’s full
q quit
? or h list commands
where n is a whole number
![Page 30: Getting Started with HPC On Iceberg](https://reader036.vdocument.in/reader036/viewer/2022081515/56812bc8550346895d901663/html5/thumbnails/30.jpg)
Displaying contents of a text_file… continued
• cat [options] filename [filename … ]
This command will output the contents of filename[s] to standard-output ( normally screen) without pausing.
Following options are useful;
-v display non-printing characters
-n display with lines numbered on the left
• tail [-n] filename
This command lists the last 10 lines of a text file.
If a number is specified (.eg. -20 ) lists the last n (i.e 20) lines
![Page 31: Getting Started with HPC On Iceberg](https://reader036.vdocument.in/reader036/viewer/2022081515/56812bc8550346895d901663/html5/thumbnails/31.jpg)
Copying files• Copy files (optionally directories)
cp fromfile tofile
Some of the useful options are:
-R or –r : Recursive copy ‘fromfile’ is a directory so the entire directory and its contents are copied.
e.g. cp –r mydir newdir
-p : preserve. Preserves all attributes of the file ,such as access rights and creation date.
• Copy and concatenate files by using cat
Cat command concatenates contents of list of files and directs the output to standard output (normally screen). When used with redirection ‘>’ it can be used to join files together.
e.g. cat file1 file2 file3 > new_big_file
![Page 32: Getting Started with HPC On Iceberg](https://reader036.vdocument.in/reader036/viewer/2022081515/56812bc8550346895d901663/html5/thumbnails/32.jpg)
Renaming and deleting files• mv :This command will move a file or directory to a new location. It can thus be
used to rename files/directories as well as change their locations in the global directory structure.
Syntax:mv source destination
Example:mv myfile mynewfile mv myfile subdirectory/myfilemv mysubdir mynewsubdir
• rm : This command will delete a file (optionally a directory if used with –r option).Syntax: rm object_to_deleteExample:
rm myfile rm –r mydirectory
![Page 33: Getting Started with HPC On Iceberg](https://reader036.vdocument.in/reader036/viewer/2022081515/56812bc8550346895d901663/html5/thumbnails/33.jpg)
Working with files
To copy a file: cp my_file my_new_fileTo move ( or rename ) a file : mv my_file my_new_fileTo delete a file : rm my_file To list the contents of a file : less file_name
To make a new directory( i.e. folder) : mkdir new_directory
To copy a file into another directory: cp my_file other_directoryTo move a file into another directory: mv my_file other_directoryTo copy a directory to another location: cp –R directory other_directory
To remove a directory and all its contents!!!: rm –R directory ( use with care )Wildcards : * means matching any sequence of characters.For example: cp *.dat my_directory
![Page 34: Getting Started with HPC On Iceberg](https://reader036.vdocument.in/reader036/viewer/2022081515/56812bc8550346895d901663/html5/thumbnails/34.jpg)
Problem Session
• Attempt problem 1-5 on the handout
![Page 35: Getting Started with HPC On Iceberg](https://reader036.vdocument.in/reader036/viewer/2022081515/56812bc8550346895d901663/html5/thumbnails/35.jpg)
Transferring files to/from iceberg
Summary of file transfer methods as well as links to downloadable tools for file transfers to iceberg are published at: http://www.sheffield.ac.uk/wrgrid/using/access
• Command line tools such as scp, sftp and gftp are available on most platforms.
• Can not use ftp ( non-secure ) to iceberg.• Graphical tools that transfer files by dragging and dropping files
between windows are available winscp, coreftp, filezilla, cyberduck
![Page 36: Getting Started with HPC On Iceberg](https://reader036.vdocument.in/reader036/viewer/2022081515/56812bc8550346895d901663/html5/thumbnails/36.jpg)
Pitfalls when transferring files
• ftp is not allowed to by iceberg. Only sftp is accepted.
• Do not use spaces ‘ ’ in filenames. Linux do not like it.
• Secure file transfer programs ‘sftp’ classify all files to be transferred as either ASCII_TEXT or BINARY.
• All SFTP clients attempt to detect the type of a file automatically before a transfer starts but also provide advanced options to manually declare the type of the file to be transferred.
• Wrong classification can cause problems particularly when transfers take place between different operating systems such as between Linux and Windows.
• If you are transferring ASCII_TEXT files to/from windows/Linux, to check that transfers worked correctly while on iceberg, type; cat –v journal_fileIf you see a ^M at the end of each line you are in trouble !!!CURE: dos2unix wrong_file on iceberg
![Page 37: Getting Started with HPC On Iceberg](https://reader036.vdocument.in/reader036/viewer/2022081515/56812bc8550346895d901663/html5/thumbnails/37.jpg)
Your filestore• Three areas of filestore always available on iceberg
• These areas are available from all headnode and worker nodes. 1. home directory:
• /home/username
• 5 GBytes allocations
• Permanent, secure, backed up area ( deleted files can be recovered )
2. data directory
• /data/username
• 50 GBytes of ollocation
• Not backed but mirrored on another server
3. /fastdata area
• /fastdata
• Much faster access from MPI jobs
• No storage limits but no backup, or mirroring
• Files older than 90 days gets deleted automatically
• Always make a directory under /fastdata and work there.
![Page 38: Getting Started with HPC On Iceberg](https://reader036.vdocument.in/reader036/viewer/2022081515/56812bc8550346895d901663/html5/thumbnails/38.jpg)
Scratch area ( only available during a job)
• Located at /scratch
• Used as temporary data storage on the current compute node alone.
• File I/O to /scratch is faster than to NFS mounted /home and /data areas
• File I/O to small files is faster than to /fastdata but … for large files /fastdata is faster than /scratch
• Data not visible to other worker nodes and expected to exist only during the duration of the job.
HOW TO USE SCRATCH:
• Create a directory using your username under /scratch on a worker and work from that directory
Example: mkdir /scratch/$USERcp mydata /scratch/$USER
cd /scratch/$USER
( then run your program )
![Page 39: Getting Started with HPC On Iceberg](https://reader036.vdocument.in/reader036/viewer/2022081515/56812bc8550346895d901663/html5/thumbnails/39.jpg)
Storage allocations• Storage allocations for each area are as follows:
– On /home 5 GBytes – On /data 50 GBytes– No limits on /fastdata
• Check your usage and allocation often to avoid exceeding the quota by typing quota
• If you exceed your quota, you get frozen and the only way out of it is by reducing your filestorage usage by deleting unwanted files via the RM command ( note this is in CAPITALS ).
• Requesting more storage:Email [email protected] to request for more storage.
• Excepting the /scratch areas on worker nodes, the view of the filestore is the same on every worker.
![Page 40: Getting Started with HPC On Iceberg](https://reader036.vdocument.in/reader036/viewer/2022081515/56812bc8550346895d901663/html5/thumbnails/40.jpg)
Running tasks on iceberg
• Two iceberg headnodes are gateways to the cluster of worker nodes.
• Headnodes’ main purpose is to allow access to the worker nodes but NOT to run cpu intensive programs.
• All cpu intensive computations must be performed on the worker nodes. This is achieved by;
▬ qsh command for interactive jobs and▬ qsub command for batch jobs.
• Once you log into iceberg, taking advantage of the power of a worker-node for interactive work is done simply by typing qsh and working in the new shell window that is opened. The next set of slides assume that you are already working on one of the worker nodes (qsh session).
![Page 41: Getting Started with HPC On Iceberg](https://reader036.vdocument.in/reader036/viewer/2022081515/56812bc8550346895d901663/html5/thumbnails/41.jpg)
Where on the cluster ?
• Most of the application packages, compilers and software libraries are only available on the worker_nodes.
• Iceberg headnodes are suitable for only light-weight jobs such as editing files.
• If you are on one of the headnodes, you will need to type qsh to get an interactive session to the worker nodes.
• How do you know where you are ? The command prompt will contain your userid@hostname Example: ch1abc@node-056 $
▬ If you are on one of the headnodes, hostname will be iceberg1 or iceberg2
▬ If you are on an amd-based worker-node, it will be amd-nodennwhere nn is the number of the amd-node.
▬ If you are on an intel-based worker-node it will be node-nnnwhere nnn is the number of the intel node.
You can always type echo $HOSTNAME to find out the name of the machine you are currently using.
![Page 42: Getting Started with HPC On Iceberg](https://reader036.vdocument.in/reader036/viewer/2022081515/56812bc8550346895d901663/html5/thumbnails/42.jpg)
Running programs on iceberg• Iceberg is the gateway to the cluster of worker nodes and the
only one where direct logging in is allowed. • Iceberg’s main purpose is to allow access to the worker
nodes but NOT to run cpu intensive programs. • All cpu intensive computations must be performed on the
worker nodes. This is achieved by the qsh command for the interactive jobs and qsub command for the batch jobs.
• Once you log into iceberg, taking advantage of the power of a worker node for interactive work is done simply by typing qsh and working in the new shell window that is opened. This what appears to be a trivial task has would in fact have queried all the worker nodes for you and started a session on the least loaded worker in the cluster.
• The next set of slides assume that you are already working on one of the worker nodes (qsh session).
![Page 43: Getting Started with HPC On Iceberg](https://reader036.vdocument.in/reader036/viewer/2022081515/56812bc8550346895d901663/html5/thumbnails/43.jpg)
Running programs• Two modes of operation foreground and background
– Foreground Interact with program via keyboard/screen
– Background No connection with keyboard/screen
Submit to backbround by Appending ‘&’
EG: myprog >& myfile &
The symbols ‘>&’ redirect output and any errors to the file myfile
Although the above method of running jobs on the background is feasible, the prio we recommend that you submit your background into the batch queue via the qsub command.
![Page 44: Getting Started with HPC On Iceberg](https://reader036.vdocument.in/reader036/viewer/2022081515/56812bc8550346895d901663/html5/thumbnails/44.jpg)
Redirection
Most unix commands are not aware of the source of their input or the destination of their output. They simply read/write from/to stdin/stdout. The shell takes care of these issues.
• Standard Input (default=>keyboard) • Standard Output (default=>screen)• Redirection symbols <,>,>> can be used to specify
files as the source/destination of the read/write operations to override the above defaults.
![Page 45: Getting Started with HPC On Iceberg](https://reader036.vdocument.in/reader036/viewer/2022081515/56812bc8550346895d901663/html5/thumbnails/45.jpg)
Redirection continued …
Most unix commands are not aware of the source of their input or the destination of their output. They simply read/write from/to stdin/stdout. The shell takes care of these issues.
• To redirect the output to a file use the ‘>‘ symbol.Example: ls -l > dirlist
• The ‘>‘ symbol should be used with care as it may over-write an existing file. ‘>>’ symbol can be used instead if the output should be appended to the end of an existing file rather than over-writing it. Example: ls -l >> logfile
• If nothing is directed to a file then a zero size file is created, or if the file already existed then the contents of the file is removed Example: > afile
• The file /dev/null is a special symbol to indicate a ‘black-hole’Example : ls –l > /dev/null
![Page 46: Getting Started with HPC On Iceberg](https://reader036.vdocument.in/reader036/viewer/2022081515/56812bc8550346895d901663/html5/thumbnails/46.jpg)
Redirection continued …
If any program expects any of it’s input from the standard-input-channel , i.e. the keyboard, it can also read the same information from a file by redirection.
• To read input from a file use the ‘<‘ symbol.Example: write cs1xyz < message.fil
Here any text input {write} program expects from the keyboard will be simply read from a file named message.fil.
Each end-of-line will be treated as an <ENTER> on the keyboard.
![Page 47: Getting Started with HPC On Iceberg](https://reader036.vdocument.in/reader036/viewer/2022081515/56812bc8550346895d901663/html5/thumbnails/47.jpg)
Piping
• Feeding the output of one command into the input of another command
• The symbol ‘|’ is called a pipecommand | command
• eg: ls -al | more
ls -la | grep Nov
![Page 48: Getting Started with HPC On Iceberg](https://reader036.vdocument.in/reader036/viewer/2022081515/56812bc8550346895d901663/html5/thumbnails/48.jpg)
Examples of re-direction and piping
• ls –l | grep ‘Jun’• ls –l | grep `Jun` > june_files• ls –l | grep `Jun` | cut –c 57-80 > june_files• cut –c 1-10 < test_files• aspell –l < message.txt > report.txt• grep fluent < news.dat • grep fluent < news.dat | cut –c 1-72 • (grep fluent < news.dat ) | cut –c 1-72 • (grep fluent < news.dat ) | cut –c 1-72 > fluent.news
![Page 49: Getting Started with HPC On Iceberg](https://reader036.vdocument.in/reader036/viewer/2022081515/56812bc8550346895d901663/html5/thumbnails/49.jpg)
Foreground Program Control
• Kill a program Ctrl C• Stop a program Ctrl Z
Note a stopped program still exists in the system
and hence can be re-started.
![Page 50: Getting Started with HPC On Iceberg](https://reader036.vdocument.in/reader036/viewer/2022081515/56812bc8550346895d901663/html5/thumbnails/50.jpg)
Program control within current shell• jobs Lists jobs (programs)• bg %job_id Place a job in the background• fg %job_id Return a job to the foreground• stop %job_id Stop a job• kill %job_id Kill a jobProcess_id can be used in place of %job_id for more
definitive way of identification.jobs[1] + Running time.sh > outstop %1[1] + Stopped (signal)time.sh > outbg %1[1] + time.sh > out &kill %1Terminated
![Page 51: Getting Started with HPC On Iceberg](https://reader036.vdocument.in/reader036/viewer/2022081515/56812bc8550346895d901663/html5/thumbnails/51.jpg)
Program control using ‘ps’ and ‘kill’• ps Report process status
ps -f -u username UID PID PPID C STIME TTY TIME CMD
username 24816 24585 0 16:23:04 pts/50 0:00 sleep 2
username 20169 19956 0 16:05:45 pts/50 0:01 -csh
username 24585 20169 0 16:35:07 pts/50 0:00 /bin/sh time.sh
• kill Terminate process
– Find its process ID (PID) using the ps command
– Kill the process using the kill commandkill 24585
– Sometimes kill on its own does not work so trykill -KILL 24585 or kill -9 pid
![Page 52: Getting Started with HPC On Iceberg](https://reader036.vdocument.in/reader036/viewer/2022081515/56812bc8550346895d901663/html5/thumbnails/52.jpg)
Repeating Previous Commands
• Operation– history List previous commands
– !! re-run last command
– !n re-run the nth command
– !str last command starting with str
eg: !vi
• Setup– Add the following to your .cshrc file
– set history=40
![Page 53: Getting Started with HPC On Iceberg](https://reader036.vdocument.in/reader036/viewer/2022081515/56812bc8550346895d901663/html5/thumbnails/53.jpg)
Searching in files
• grep string file
This command finds and prints out the lines in the file(s) containing the specified string
string = word or phrase
file = file or list of files (wild_card can be used)
Note: We strongly advise that the string is quoted.
Examples: grep ‘Green Man’ england.dat
grep ‘Zodiac’ t*.dat
grep ‘Zone[a-z]’ security.fil
![Page 54: Getting Started with HPC On Iceberg](https://reader036.vdocument.in/reader036/viewer/2022081515/56812bc8550346895d901663/html5/thumbnails/54.jpg)
Manual Pages and Info system
Man: Manual pages : Give text-based help on usage.
Manual pages are grouped into sections (1,2,3,4 ..).
There is usually one manual page per command which is
located in one of the directories defined by the MANPATH
environment variable.
To access the Manual page for a command just type;
• man command
To get a list of manual pages that contain a ‘word’ type;
• man – k topic
Info : Similar to man but can scroll with cursor keys and link to other information.
![Page 55: Getting Started with HPC On Iceberg](https://reader036.vdocument.in/reader036/viewer/2022081515/56812bc8550346895d901663/html5/thumbnails/55.jpg)
Further Help• Getting started with iceberg
– http://www.shef.ac.uk/wrgrid/iceberg• http://www.shef.ac.uk/wrgrid/using/introlinux• http://www.shef.ac.uk/wrgrid/using/introbash• GNU/Linux Command-Line Tools Summary
– http://en.tldp.org/LDP/GNU-Linux-Tools-Summary/html/index.html • Bash Guide for Beginners
– http://tldp.org/LDP/Bash-Beginners-Guide/html/index.html • Advanced Bash-Scripting Guide
– http://en.tldp.org/LDP/abs/html/index.html • The Linux Documentation Project
– http://en.tldp.org/
![Page 56: Getting Started with HPC On Iceberg](https://reader036.vdocument.in/reader036/viewer/2022081515/56812bc8550346895d901663/html5/thumbnails/56.jpg)
Problem Session
• Attempt problem 6-9 on the handout