Shell introduction

More features and gory details in ‘man sh’—hard to follow forbeginners, but hopefully much better after these notes.

I stick to old, basic Bourne shell (author’s name was Bourne) ‘sh’in this course for more fundamental understanding.

Most systems default to Bourne Again shell (a pun) ‘bash’. Manymore features and cursor editing.

There are others, e.g., csh, tcsh, zsh, fish.

Your homework submissions and test/exam answers must workunder the very basic sh. You may use other nicer shells otherwise.

1 / 41

CommentsA comment begins with ‘#’ and extends until end of line. Can bewhole line or begin from middle of line.

ls -l # this is a comment

2 / 41

VariablesType is string.

Set value: var=abcNote: No space around ‘=’

Read value: $var or ${var}(If uninit: get empty string.)

Why ‘${var}’ syntax provided: Try this example:

v=xxxv0=yyyecho $v0 # yyyecho ${v}0 # xxx0

3 / 41

“Simple” Commands“simple command” = command name, arguments, optionally [file]redirection (next slide).

Example (without redirection): tr -d 123

Command name has 4 cases, not apparent from syntax:

I Shell built-in command, e.g., ‘cd’I Shell function (user-defined).I Shell alias (user-defined) (omitted, but dead simple).I Program name, e.g., ‘tr’.

4 / 41

[File] RedirectionExample: tr -d 123 < infile > outfile

tr

stdin stdout

infile outfile

‘>’ overwrites. Use ‘>>’ instead if you want to append.

Redirect stderr: 2>

Redirect both stdout and stderr to same file:command > file 2>&1

5 / 41

PipelineE.g., sort | uniq

sort

stdin stdout

uniq

stdin stdout

Operator precedence: Lower than redirection.

6 / 41

[Command] List—Sequential CompositionMultiple commands can be put on multiple lines (especially in shellscript files).

Or, a single line but separated by semicolons. Example:

cd B09 ; ls ; ls -l ; cd ..

Either way, known as “list” or “command list”, sequentiallyexecuted: wait for one to finish before running the next.

Operator precedence: Lower than ‘|’.

Example: ls ; sort file1 | uniqRun ‘ls’, then run ‘sort file1 | uniq’.

7 / 41

Success, Failure, Exit CodeMost commands declare success or failure:

I Most commands fail upon non-existent pathnames.I ‘grep’ fails if nothing matches at all.

Conveyed by “exit code”. 0 for success, non-0 for failure. Become“true” and “false” for logical operators and conditional branching.

In C, recall “int main(...)”, that return value is exit code!

$? is most recent command’s exit code.

Beware: ‘echo $?’ is also a command! And it succeeds. Exercise:What does it print if you run it twice consecutively?

8 / 41

Logical AND, OR, NOTmkdir foo && cp myfile fooSequential execution, but stop upon first “false”.

mkdir foo1 || mkdir foo2 || mkdir foo3Sequential execution, but stop upon first “true”.

(So they are short-circuiting.)

! mkdir fooLogical not: turn 0 to 1, non-0 to 0.

Operator precedence:

‘;’ lower than ‘&&’, ‘||’‘&&’, ‘||’ same precedence, lower than ‘!’‘!’ lower than ‘|’

9 / 41

GroupingWhen operator precedence doesn’t work for you, write

{ list ; }

for explicit grouping. (Recall “[command] list”.)

Example:{ grep foo file1 ; ls ; } > file2

Again, may use newline instead of ;

Easy to miss: after last command, before }, you need newline or ;

10 / 41

Operator Summary And PrecedenceFrom highest to lowest precednece:

description operatorgrouping {}redirection < > >>pipeline |not !and, or && ||command list ; newline

11 / 41

Conditional Branchingif list1 ; then

list2elif list3 ; then

list4elselist5

fi

Easy to miss:

I before “then”, need ; or newlineI “elif”, not “else if”

Exercise: “else if” is not wrong, but what is annoying about it?

12 / 41

While-Loopwhile list1 ; do

list2done

May use ‘break’ and ‘continue’.

Easy to miss: before “do”, need ; or newline

13 / 41

Test CommandsA whole suite of shell builtin “[ expression ]” commands to douseful tests and give you exit codes for booleans.

File tests:

I [ -e path ]: existsI [ -f path ]: exists and regular fileI [ -d path ]: exists and directoryI [ -r path ]: exists and readableI [ -w path ]: exists and writableI [ -x path ]: exists and executableI [ -nt path1 path2 ]: both exist and path1 is newerI [ -ot path1 path2 ]: both exist and path1 is older

Example usage: [ -d lab02 ] || echo sadface

14 / 41

Test CommandsString comparisons:

I [ s1 = s2 ]: two strings equalAlso ‘!=’, ‘<’, ‘>’

I [ -n string ]: string not emptyI [ -z string ]: string empty

Number comparisons (parsing strings to numbers):

I [ n1 -eq n2 ]: two integers equalAlso ‘-ne’, ‘-gt’, ‘-ge’, ‘-lt’, ‘-le’

15 / 41

Test CommandsLogical connectives, by example:

I [ $x -eq 5 -a $y -eq 6 ]: andI [ $x -eq 5 -o $y -eq 6 ]: orI [ ! -e path ]: not

Parentheses also supported.

16 / 41

Test Commands in if/whileif [ $x = $y ] ; then...

fi

while [ $x = $y ] ; do...

done

Easy to miss: Still need ; or newline, even though ] “ends thecondition”.

Recall the syntax isif list ; thenhere, “[ $x = $y ]” is that list.

17 / 41

For-loopfor var in word1 word2 ... ; do

listdone

In the loop, use ‘$var’ to read the variable.

for i in Albert Frank do ; doecho $imkdir $i

done

May use ‘break’ and ‘continue’ in the loop.

Easy to miss: Need ; or newline before do to mark end of words.Lest computer thinks your do is one of the words, like above.

18 / 41

Patterns (to match filenames)

I ‘*’ matches any string (but doesn’t cross directory boundaries)Example: ls a2/*.pyAll python files in directory a2

I ‘?’ matches one characterI ‘[ace]’ matches “a” or “c” or “e”I ‘[0-9]’ matches a digitI ‘[!0-9]’ matches a non-digit

Important: Shell expands pattern to multiple pathnames beforehanding to program. ‘ls’ never saw “a2/*py”; it saw “a2/foo.py”,“a2/bar.py”, etc.

Important: If no match, the pattern stays as itself.

For-loop may use pattern too:for i in *.py ; do echo $i ; done

19 / 41

Escaping And QuotingShell syntax is full of special-meaning characters like ‘*’, ‘[’, ‘$’, ‘#’,‘(’, ‘;’, even space. . . You have seen them in special notations.

What if you want those characters themselves? For example howto safely print “x* #” (2 spaces in between)?

I Use backslash ‘\’ to “escape”:echo x\*\ \ \#Note: So ‘\’ is also special! Use ‘\\’ for backslash itself.

I Use single quotes:echo ’x* #’

I Use double quotes:echo "x* #"Note: ‘$’ still special, for a good reason shown later.

Likewise when setting variables: v=’x* #’

20 / 41

Example Uses of Escaping And Quoting

I Filenames containing spaces.ls -l ’Harry Potter.pdf’

I Some operators in test commands:[ ! ’(’ aaa ’<’ abc -o aaa ’>’ abc ’)’ ]

I Regular expressions for ‘grep’.grep ’<[a-z]*>’ lab.html

21 / 41

Example Uses of Escaping And Quoting

I Filenames containing spaces.ls -l ’Harry Potter.pdf’

I Some operators in test commands:[ ! ’(’ aaa ’<’ abc -o aaa ’>’ abc ’)’ ]

I Regular expressions for ‘grep’.grep ’<[a-z]*>’ lab.html

21 / 41

Example Uses of Escaping And Quoting

I Filenames containing spaces.ls -l ’Harry Potter.pdf’

I Some operators in test commands:[ ! ’(’ aaa ’<’ abc -o aaa ’>’ abc ’)’ ]

I Regular expressions for ‘grep’.grep ’<[a-z]*>’ lab.html

21 / 41

Variables in Double QuotesHave a filename “Harry Potter.pdf” that contains a space.

Set this: f=’Harry Potter.pdf’

What happens when: ls -l $f

Explanation: ls -l Harry Potter.pdftwo filenames.

Solution: ls -l "$f"

This is why $ is still special in double quotes.

One more example:v=’x* #’echo "$v"

22 / 41

Variables in Double QuotesCommon mistake when checking whether $v is non-empty:[ -n $v ]

No! If $v is empty, shell sees [ -n ], which makes no sense.

Solution: [ -n "$v" ]

Exercise: Older generation used[ x != x$v ]When does it work? When does it break?

23 / 41

CaseConditional branching on pattern matching.case "$var" in

*.py)rm "$var";;

*.c | *.sh | myscript)echo w00t "$var";;

*)echo meh "$var"

esac

24 / 41

Shell ScriptsPut your commands in a file, call it “myscript” say. You can run itwith

sh myscript

More savvy users go one step further:

I Put as first line: #!/bin/shI Set executable flag on the file:chmod u+x myscript

I Run it with ./myscript

Next: How to access arguments? E.g.,‘./myscript foo’how can the script see that “foo”?

25 / 41

Positional ParametersIf I run your script as ‘./myscript foo bar xyz’:

I $# has 3, the number of argumentsI $1 has “foo”I $2 has “bar”I $3 has “xyz”I $* has “foo bar xyz”"$*" expands to one single word “foo bar xyz”

I $@ has “foo bar xyz”"$@" expands to 3 words “foo”, “bar”, “xyz”

26 / 41

shift

Shift positional parameters. E.g., starting from the previous slide,one shift causes:

I $# has 2I $1 has “bar”I $2 has “xyz”I $* has “bar xyz”"$*" expands to one single word “bar xyz”

I $@ has “bar xyz”"$@" expands to 2 words “bar”, “xyz”

Convenient for looping over arguments!

27 / 41

Small Example Script (Pg 1/2)#!/bin/shdryrun=verbose=while [ $# -gt 0 ]; do

case "$1" in-n)

dryrun=y;;

-v)verbose=y;;

*)break

esacshift

done

28 / 41

Small Example Script (Pg 2/2)for f in "$@" ; docase "$f" in*.py)[ -n "$verbose" ] && echo "deleting $f"[ -z "$dryrun" ] && rm "$f";;

*)[ -n "$verbose" ] && echo "not deleting $f"

esacdone

Code file: smallscript.sh

29 / 41

Small Example Script: ExplanationGo through arguments (meant to be filenames), delete those thatare Python files.

But if there are ‘-n’ and/or ‘-v’ at the beginning:-n means dry-run—don’t actually delete-v means verbose—say what is happening to each filename

Page 1 detects ‘-n’ and ‘-v’.

After that, ‘$@’ is left with the filenames.

Page 2 can use a for-loop over ‘$@’ to process each filename.

Exercise: Why didn’t I use a for-loop in page 1?

30 / 41

ExitMay use the command ‘exit’ to terminate the shell script and/orthe shell process.

Not required if your script just runs from start to finish normally.

But handy for:

Early termination (when inside loops, functions, etc.)

Controlling exit code, e.g., ‘exit 1’.

(Default exit code is whatever the last executed command gives.)

31 / 41

FunctionsExample function definition:

myfunction() {echo $1echo $@

}

Example function call:

myfunction foo bar xyz

Inside a function, positional parameters become functionarguments.

May return from function early, or specify exit code, with ‘return’or e.g., ‘return 1’.

(Default exit code is from the last executed command.)

32 / 41

Local Variables in FunctionsExample:

myfunction() {local x yx=1# y deliberately unchanged , see belowecho $x $y

}

Local as in: Does not change outer variables.

Initial values copied from caller.

y=foomyfunction

prints “1 foo”.

33 / 41

Dot command: Execute stuff in current shellThis reads commands from cmds.sh, executes them in currentshell:

. ./cmds.sh

The command name is a single dot “.”

Contrast: sh cmds.sh runs in newly spawned shell process.

Use case: If cmd.sh defines functions, set variables, or uses cd,then

I . ./cmds.sh does them in current shell.I sh cmds.sh does them in new shell process, which then

quits, much ado about nothing../cmd.sh ditto.

34 / 41

Here DocumentTo feed multi-line hardcoded text into stdin of a command:

cat << EOFHello I’m Albert.You can use variables tooE.g., \$x=$xEOF

The first time I said “EOF”, shell takes note. Second time, shellknows I’m marking the end.

“EOF” is not a keyword, you may choose another word, just don’tclash with your actual text!

35 / 41

Here Document: One More ThingIf you declare your end-marker in quotes:

cat << ’EOF’Hello I’m Albert.Now $x is $xEOF

cat << "EOF"Hello I’m Albert.And $x is still $xEOF

then $ is no longer special.

36 / 41

Command SubstitutionRun a command, capture its stdout, insert output data in-place:$(command)

or‘command‘

(both backticks, not single quotes)

The data is split into words.

for i in $(echo aaa bbb ccc) ; doecho "$i"

done

But if inside double-quotes, not splitted. (And other technicalitiesnot shown here.)

for i in "$(echo aaa bbb ccc)" ; doecho "$i"

done

37 / 41

Environment VariablesEvery process (shell or otherwise) has a collection of “environmentvariables”, as part of process state.

Names are strings, values are strings too. Convention: Names inall caps, e.g., PATH, HOME, TZ, LC_ALL (these are standard Unixones), CLASSPATH (specific to Java).

Initialized by copying from launcher (done by kernel): If p launchesq, q gets a copy of p’s. But independently changeable otherwise.

Program ‘printenv’ prints the environment variables you currentlyhave. It works because at startup it gets a copy of yours! Now itjust has to print what it has.

38 / 41

Environment Variables in ShellShell downplays difference between shell variables andenvironment variables. Only convention: shell variable names arein lowercase.

Both read by same syntax: $x, $LC_ALL

Both changeable by same syntax:x=CLC_ALL=C

Both erasable by same ‘unset’ command.

How to mark a variable as environment variable:export MYENVVAR=foo

or two commands:MYENVVAR=fooexport MYENVVAR

39 / 41

Environment Variables in ShellTo run a program but give it different environment variables(existing or new) without changing your own:

LC_ALL=C MYNEWENV=foo printenv

This is why the following two commands mean different things:

x=’foo bar’x=foo bar

40 / 41

Some Standard Environment VariablesHOME: Home directory.

TZ: User timezone preference. (Can be absent.)

PATH: Colon-separated list of directories. Searched when youlaunch a program, if program name does not contain any slash.

Example: AssumePATH=/usr/local/cms/jdk1.8.0_31/bin:/usr/bin:/bin

javac Foo.javafound in /usr/local/cms/jdk1.8.0_31/bin

printenvfound in /usr/bin

shfound in /bin

41 / 41