Introduction to Unix

Advanced Computing Center for Research and Education

http://www.accre.vanderbilt.edu

@ACCREVandyFollow us on Twitter for important news and updates:

Why Unix?What comes to mind when you hear the word “Unix”?

Command line!

Unix is a programmer’s paradise Takes advantage of the keyboard and short commands (typically 2-4 letters in length) to efficiently manage files, folders, and so on Emphasizes performance Large user community (including most supercomputing centers) NOT designed to be a math editor

What is Unix?✦Unix is an operating system (un)like Windows✦Originally created in the late 1960’s at AT&T Bell Labs✦Designed to be a programmer’s OS✦Turned out to be a portable, multi-tasking, multi-user OS (we take those things for granted now, but they were revolutionary at the time!)

✦There are many different “flavors” of Unix...

Different “flavors” of Unix have different niches

✦ IBM’s AIX and HP’s HP-UX are heavily used in business✦Oracle’s (formerly Sun’s) Solaris once powered 70% of the web / e-mail servers on the Internet

✦Apple’s MacOS X is a user-friendly desktop✦Linux is very high performance and free, which makes it ideal for HPC clusters

Unix ArchitectureShell

Kernel (OS)

Physical

Devices

✦Only the Kernel (the operating system) can manipulate physical devices (disks, the network, printers, etc.)

✦Users interact with the kernel via a shell (command interpreter)

✦Two primary shells: 1) bash, 2) tcsh

Unix Philosophy✦Everything is a file. Executables, program source, and text documents are all files. Directories are files that list other files. Devices (disks, network, etc.) are files.

✦You really want to do what you said you want to do. Therefore Unix should do it as quickly and quietly as possible. I.e. No “Are you sure? (Y/N)” messages by default.

Unix Commands✦The format of Unix commands is command [ options ] [ arguments ]

✦ id is a command✦ -f is an option to the ps command✦example1 is an argument to the ls command (and -l is an option)

The most important command of all

✦The man command displays manual pages

✦Example at left is the output of man ls

✦Displays synopsis✦Gives description of each option and, if applicable, argument to the command

Command History / Editing

✦The shell maintains a history of the commands you have previously executed (1000 commands by default)

✦Up and down arrow keys scroll thru your command history

✦Left and right arrow keys scroll thru the command✦Edits can be made by adding, deleting, or replacing parts of the command

✦Pressing the enter / return key executes the command

Hierarchical Filesystem/

bin usr home scratchetc tmp

chmod

cp

date

grep

mv

rm

vi

nikkiannie codybin lib

bin docs src

libc.so

libgpfs.so

libjpeg.so

libstdc++.so

diff

find

gcc

id

make

perl

ssh

prog1.c

prog2.f77

prog3.cpp

myprog.sh

dothis.pl

dothat.py

Working With Directories✦pwd prints your present working directory

✦ cd changes directories (with no arguments always changes to your home directory)

✦mkdir makes a directory✦ rmdir removes a (empty) directory

✦ ls lists directories (and files)

Working With Files✦file tells you what type of file a file is

✦more displays the contents of a file

✦ cp copies files✦mv renames (moves) files

✦ rm removes files✦The -i option makes cp, mv, and rm “interactive”

Editing Files✦There are three text editors available...✦emacs - popular with programmers✦ nano (pico) - basic, easy to use editor✦ vi - the original Unix text editor; has steeper learning curve, but is the fastest of the three

✦Which should you use?✦Whichever one you like!

File Permissions✦All files / directories have one user (or owner - nikki in this example) and one group (guest) associated with them

✦There are three sets (user, group, other) of three permissions (read, write, execute)

✦Only the owner may change the owner, group, or the permissions

Read, Write, Execute

Read Write Execute

FilesYou can look at the file contents

You can modify the file contents

You can run the program

DirectoriesYou can ls the

directory

You can create and delete files

You can cd to the directory

Changing Permissions✦ chmod (change mode)✦alphabetic method - add or take away (r)ead, (w)rite, or e(x)ecute for the (u)ser, (g)roup, and / or (o)ther

✦ numeric method - read = 4, write = 2, execute = 1; total up what you want for the user, group, and other

Pattern Matching

✦ * - matches zero or more instances of any character✦ ? - matches one instance of any character✦ [abc] - matches any one character within the brackets; “a”, “b”, or “c” in this example

✦ [0-9] - matches any one character within the range defined within the brackets

✦ [A-z] - matches all letters, plus most punctuation characters, because this is an ASCII range (use “[A-Za-z]” instead to match only letters

Pattern Matching Examples

Aliases

✦bash syntax - alias rm=”rm -i”✦ tcsh syntax - alias rm ‘rm -i’✦Now when you type rm, the shell will automatically replace it with rm -i

✦alias ll=”ls -la”✦alias mroe=more✦alias cdmydir=”cd /some/big/long/path/that/is/hard/to/remember/much/less/type”

Shell Variables✦Many variables are set for you

✦Two types of variables: environment and local

✦ local variables are visible only to the current shell

✦environment variables are also visible to child shells and processes

✦env shows environment variables

Defining Shell Variables✦my_var=some_value - bash syntax for creating a local variable

✦export my_var - makes my_var an environment variable✦export my_var=some_value - creates an environment variable in one step

✦ set my_var some_value - tcsh syntax for creating a local variable

✦ setenv my_var some_value - tcsh syntax for creating an environment variable

Displaying Shell Variables

✦The echo command can be used to display the values of variables

✦When referencing a variable, precede the name with a $

✦This example shows the difference between local and environment variables

Shell Initialization Files✦Any aliases or variables you define on the command line are only in effect for the duration of that session

✦To make them permanent, simply add them to your .bashrc or .cshrc file as appropriate

Command Substitution✦Any command enclosed in grave accents (not single quotes) is executed first and then its output is substituted in on the command line

✦Command substitution can be used with other commands (as the first example shows) or to assign a value to a variable (as the second example shows)

Input / Output Redirection

✦Any shell has 3 filehandles open by default:✦stdin - standard input, defaults to keyboard, file descriptor 0

✦stdout - standard output, defaults to screen, file descriptor 1

✦stderr - standard error, defaults to screen, file descriptor 2

✦And yes, if you open a file, the first one you open would have file descriptor 3, etc.

Input / Output Redirection

✦ Input redirection - e-mail yourself a file: mailx Nikki.Husky@vanderbilt.edu < example1

✦Output redirection - myprogram > output.log✦Error redirection - myprogram 2> error.log✦Output and error redirection to different files - myprogram > output.log 2> error.log

✦Output and error redirection to the same file - myprogram > combined.log 2>&1

A Few Useful Filters

✦wc - word count✦grep - (g)et (r)egular (e)xpression and (p)rint (i.e. pattern matching)

✦ sort - sort alphabetically or numerically✦uniq - filter duplicate lines✦ cut - cut specific fields or columns✦ sed - stream editor (i.e. search and replace)✦awk - programming language for quick processing

Pipes and Filters

✦Pipes take the output of one command and make it the input to another command

✦Analogous to plumbing pipes✦ Filters are commands which can accept input from another command and also produce output

✦Syntax is command | filter_command✦Multiple pipes and filters can be strung together on one command line - command | filter_command1 | filter_command2 | filter_command3

Compressing / Archiving Files

✦ If you have a Windows client, use zip / unzip✦Example: zip mydata.zip ./mydatadir/*✦Then use scp to copy the data to your client and unzip it there: unzip mydata.zip

✦ If you have a Linux or Mac client, use tar and gzip✦Example: tar cvf mydata.tar ./mydatadir/*; gzip mydata.tar

✦Then use scp to copy the data to your client and uncompress / untar it there: gzip -d mydata.tar.gz; tar xvf mydata.tar

Getting Help from ACCRE✦ ACCRE website FAQ and Getting Started pages: http://

www.accre.vanderbilt.edu/support✦ ACCRE helpdesk: http://www.accre.vanderbilt.edu/support/contact/

submit_RT.php✦Tickets default to non-rush queue, which is for tickets about an issue which only impacts you and / or can wait until the next business day.

✦Rush tickets are for issues which would impact the cluster as a whole.

✦Rush queue pages ACCRE personnel (we normally only have staff on site during normal business hours), so please think before opening up a rush ticket at 3 AM!

✦ If you open a rush ticket - day or night - please plan to be available to work with ACCRE personnel to resolve the issue.

✦Once the issue (rush or non-rush) is resolved we would appreciate it if you would let us know.

Getting Help from ACCRE

Getting Help from ACCRE✦accre-forum mailing list✦ACCRE office hours: open a help desk ticket and mention that you would like to work personally with one of us in the office. The person most able to help you with your specific issue will contact you to schedule a date and time for your visit.