case study #2 - dcclblopes/aulas/ic/t10.pdf$ diff teste1.c teste2.c to find text patterns within...
TRANSCRIPT
Case Study #2
Bourne Again Shell (bash)an interpreted language
What is a Shell?
A program that takes keyboard commands and issues them to the operating system to
be executed on behalf of the user
Several types, e.g., C-Shell (csh, tcsh), Bourne Shell (sh), Bourne Again Shell (bash)
With a graphical interface, terminal emulators are used to issue commands, e.g., terminal,
xterm, konsole, gnome-terminal.
The files in a Unix system are organized hierarchically as directory tree called the file system
the root of the file system is the directory “/”
when a user logs in, the operating system sets the current directory to the user’s home directory, a sub-
tree of the file system where the user can freely operate on files
in the labs the home directory has the form/net/areas/homes/up2018xxxx
to move between directories we use the “cd” command with an absolute or a relative path
an absolute path gives the location of a file with respect to root directory, thus beginning always with “/”
$ cd /net/areas/homes/up2018xxx/IC
a relative path gives the location of a file with respect to the current directory
$ cd ../PI/
the symbols “.” and “..” stand for, respectively, the current directory and the parent directory
command “pwd” can be used to verify which is the current directory
$ pwd
command “tree” can be used to get a graphical representation of the file system
under the current directory
$ cd .. $ tree
we can use the command “ls” or “ls -l” to get extra information about files and directories
$ ls -l PI ; cd PI ; ls -l teste.c
the “-l” option provides extra information on the files, namely, permissions, size, owner, creation date
to find out the type of a given file we can use the command “file”
$ file teste.c teste.c: ASCII c program text
$ gcc teste.c -o teste $ file teste
teste: 64-bit executable x86_64
we can create new directories with the command “mkdir” and new (empty) files with the command “touch” or with
an editor like “emacs”
$ cd PI ; mkdir praticas ; cd praticas$ touch folha2.txt or $ emacs folha2.txt
to remove files and directories we can use the commands “rm” and “rmdir” (the latter only if the directory is empty)
$ cd .. ; rm -rf praticas (be very carefull!)or
$ rm -rf ./* ; cd .. ; rmdir praticas
we can change the permissions associated with files and directories using command “chmod”
chmod a+rwx filechmod ugo-x file
or in the octal notation
chmod 755 file (typical for executables)chmod 644 file (typical for text files)
the shell keeps a record of the commands we execute in a session
$ history
we can see the contents of files using multiple commands
$ cat teste.c$ more teste.c$ less teste.c$ head -n 100 teste.c$ tail -n 100 teste.c
to find files or directories with certain characteristics we can use the powerful command “find”
$ find . -name *.txt -print$ find . -type d -size +1k -print$ find . -name *.o -exec rm -f {}\;
also the command “locate”
$ locate fcup
to find out which version of a program we are running we can use the “which” command
$ which gcc ; which emacs ; which find
to identify differences between files we use “diff”
$ diff teste1.c teste2.c
to find text patterns within files we can use the “grep” and “egrep” commands
$ grep int test.c$ grep \”*\” teste.c$ grep \;$ teste.c
to select only certain parts of the input text we use “cut”
$ cut -d ‘=‘ -f 1 teste.c $ grep = teste.c | cut -d ‘=‘ -f 1
we can find/replace certain text patterns in files using the “sed” command
sed 's/Marco/Paulo/' file.txt(only first instance)sed 's/Marco/Paulo/4' file.txt(only 4th instance)sed ’s/Marco/Paulo/g' file.txt(all instances in line)sed ’s/^[ \t]*//‘ file.txt(remove spaces at beginning of each line)
we may see which processes are running in the computer with “ps”
$ ps (user processes only)$ ps -aux (all processes)
we can execute a command and still be able to use the shell by doing
$ emacs &
the processes thus created can be listed with the command “jobs”
$ jobs
each process has an unique system identifier that can be used to send it signals, namely the “commit suicide” signal
$ kill %1 (the number in the list given by jobs)
or
$ kill -9 23893 (the PID given by ps)
shell commands can be composed sequentially
$ cd PI ; gcc teste.c -o teste ; ./teste
or through pipes that link the output of the first to the input of the second
$ cat teste.c | grep int
$ ps -aux | grep ^up2018xxxx
we may redirect the input and/or output of commands using the “<“ and “>” operators, respectively
$ cat file1.txt file2.txt > all.txt
$ gcc calcPi.c -o calcPi$ ./calcPi > digitosPi.txt
$ gcc calcMedia.c -o calcMedia$ ./calcMedia < dados.txt
SHELL SCRIPTING
the following material is mostly based on the excellent online tutorial by Ryan Chadwick
https://ryanstutorials.net/bash-scripting-tutorial/
it is possible to include multiple shell commands in a file, with a little extra syntax, to
perform more complex processing
such a shell program is commonly called a (shell) script
shell scripts are a fundamental tool for the software developer and, even more, to the
system/network administrator
they allow many tedious tasks to be programmed once and automated
we will now see how to write such scripts
your very first Bash script
#!/bin/bash
echo Hello World!
write it in emacs, save to myscript.sh
the sequence #! in the beginning is known as the ‘shebang’
after it comes the path to the interpreter that will be used to process the remainder of the file
you can run it as follows
$ ./myscript.sh
you may need to change the file permissions
$ chmod 755 myscript.sh$ ./myscript.sh
scripts can have command line arguments
these can be accessed through the variables $1, $2, …
this script makes a copy of the file $1 into a file named $2
#!/bin/bashcp $1 $2# check that $1 and $2 are similarls -l $1ls -l $2
there are several special variables
$0 — the name of the script$1 to $9 — the first 9 arguments to the script$# — the number of arguments passed to the script$@ — all the arguments passed to the script$? — the exit status of the most recently run process$$ — the process ID of the script$USER — the username of the user running the script$HOSTNAME — the name of the machine running the script$SECONDS — the number of seconds since script started$RANDOM — different random number each reference $LINENO — current line number in the script
we can also define variables to hold temporary values
#!/bin/bashmyvariable=Helloanothervar=Fredecho $myvariable $anothervarechosampledir=/etcls $sampledir
we can assign constants and the values in other variables to variables
var1=3var2=“Hello!”var3=/etc/passwdvar4=$var3var5=$HOSTNAME
but also, the results of commands
var=$( ls /etc | wc -l )
note: no space on either side of the equals ( = ) sign !!
we can interactively ask users for input
#!/bin/bashecho Hello, who am I talking to?read varnameecho Nice to meet you $varname
we can format the request
#!/bin/bashread -p 'Username: ' uservarread -sp 'Password: ' passvarechoecho $uservar we have your login details
each process in a Unix system has a PID and associated input, output and error devices
STDIN — /dev/stdinSTDOUT — /dev/stdoutSTDERR — /dev/stderr
the STDIN, STDOUT and STDERR can be redirected with the operators |, > and <
cmd1|cmd2 — connect STDOUT of cmd1 to STDIN of cmd2cmd < file — file is used as STDIN of cmdcmd > file — file is used as STDOUT of cmd
using stdin to feed data to a bash script that cuts the 2 first fields of each line and sorts the resulting list of values
#!/bin/bash# basic sales reportecho Summary of the sales data:echo ==========================echocat /dev/stdin | cut -d ' ' -f 2,3 | sort
this can be executed as
$ ./summary.sh < sales_data.txt
or $ cat sales_data.txt | ./summary.sh
$ cat sales_data.txtFred apples 20 November 4Susy oranges 5 November 7Mark watermelons 12 November 10Terry peaches 7 November 15
$ cat sales_data.txt | ./summaryor$ ./summary < sales_data.txt
Summary of the sales data:==========================apples 20oranges 5peaches 7watermelons 12
we can also do arithmetic using let to assign the result to a variable
#!/bin/bashlet a=5+4echo $a # 9let "a = 5 + 4"echo $a # 9let a++echo $a # 10let "a = 4 * 5"echo $a # 20let "a = $1 + 30"echo $a # 30 + first argument
or using expr that evaluates and prints the result immediately
#!/bin/bashexpr 5 + 4 #9expr “5 + 4” #5 + 4 expr 5+4 # 5+4expr 5 \* $1 # $1 * 5expr 11 % 2 # 1a=$( expr 10 - 3 )echo $a # 7
sometimes it is very useful to know the size of a variable, i.e., the number of characters it
requires to be represented
#!/bin/basha='Hello World'echo ${#a} # 11b=4953echo ${#b} # 4
bash also features basic “conditional statements”
#!/bin/bash
if [ $1 -gt 100 ]then
echo Hey that\'s a large number.fi
#!/bin/bashif [ $1 -eq 0 ]thenecho You have a zero
elif [ $1 -gt 0 ]thenecho You have a positive
elseecho You have a negative
fi
full form of “conditional statement”
! EXPRESSION (expression is false)-n STRING (length of STRING is greater than zero)
-z STRING (the lengh of STRING is zero)STRING1 = STRING2STRING1 != STRING2
INTEGER1 -eq INTEGER2INTEGER1 -gt INTEGER2INTEGER1 -lt INTEGER2
-d FILE (FILE exists and is a directory)-e FILE (FILE exists)
-r FILE (FILE exists and is readable)-s FILE (FILE exists and is not empty)-w FILE (FILE exists and is writable)
-x FILE (FILE exists and is executable)
#!/bin/bashif [ -r $1 ] && [ -s $1 ] thenecho This file is useful.
fi
#!/bin/bashif [ $USER == ‘bob' ] || [ $USER == ‘andy' ]thenecho User can be trusted.
elseecho User can not be trusted.
fi
another example
#!/bin/bashcase $1 instart)echo starting;;stop)echo stoping;;restart)echo restarting;;*)echo don\'t know;;esac
basic “case statement”
#!/bin/bashcounter=1while [ $counter -le 10 ]do
echo $counter((counter++))
doneecho all done
basic “while loop”
#!/bin/bashcounter=1until [ $counter -eq 10 ]do
echo $counter((counter++))
doneecho all done
basic “until loop”
#!/bin/bash
until [ git pull &> /dev/null ]do echo Trying git server ... sleep 1done
echo Repository is pulled.
another example
#!/bin/bashnames='Stan Kyle Cartman'for name in $namesdo
echo $namedoneecho all done
basic “for loop”
#!/bin/bashfor value in {1..5}do
echo $valuedoneecho all done
“for loop” with range
#!/bin/bashfor file in $1/*.htmldo
cp $file $1/$( basename -s .html $file ).php
done
example: copy html files to php files with same name
#!/bin/bashfor file in $1/*do
if [ ! -r $file ]thenecho $file not readable 1>&2continue
ficp $file $1/backup/
done
example: backing up a set of readable files
#!/bin/bashfor file in $1/*do
used=$( df $1 | tail -1 | awk '{print $5}' | sed 's/%//' )if [ $used -gt 90 ]then
echo Low disk space 1>&2break
ficp $file $1/backup/
done
example: backing up a set of files if space is scarce
#!/bin/bashprint_something () {
echo Hello I am a function}print_somethingprint_something
functions
#!/bin/bashprint_something () {
echo Hello $1}print_something Marsprint_something Jupiter
passing arguments to functions
#!/bin/bashprint_something () {
echo Hello $1return 5
}print_something Marsprint_something Jupiterecho The return value is $?
returning values from functions