Download - 4 Tools and Commands
-
7/30/2019 4 Tools and Commands
1/69
Operating systems - Tools 1
Unix Operating System
Tools and commands
Politecnico di Torino
Dip. Automatica e Informatica
2010/2011
Original copyright from Giorgio Di Natale, Stefano Di Carlo and Fabio Maino
-
7/30/2019 4 Tools and Commands
2/69
Operating systems - Tools 2
Outline
Some commands find
grep
tar, gzip, gunzip Filters
sort
sed
awk
-
7/30/2019 4 Tools and Commands
3/69
Operating systems - Tools 3
File search
find [-opt]
some options:
-name patternWarning: use single quotes if you are usingregular expressions
-type [b c d l]
-
7/30/2019 4 Tools and Commands
4/69
Operating systems - Tools 4
Advanced use of find
It is possible to execute a command on allfound files specifying:
-exec command \;
In command, use \{} for the current file example:
find . -name core -exec rm \{} \;
-
7/30/2019 4 Tools and Commands
5/69
Operating systems - Tools 5
Advanced use of find
Search expressions can contain logicalconditions
Logical operations:
AND: conditions must be listed one after the other
OR: use -o
NOT: use !
-
7/30/2019 4 Tools and Commands
6/69
Operating systems - Tools 6
Advanced use of find
Use ( ) to build complex expressions
Example:find . \( -name core -o -size 10b \)
find . \( -name test -o ! \( -size 10c \) \)
\ is to protect ( from the shell expansion; 10cmeans 10 byte, while 10b means 10 blocks
-
7/30/2019 4 Tools and Commands
7/69
Operating systems - Tools 7
grep
In order to search a string in a set of files, it ispossible to use:
grep [-options] pattern files
-
7/30/2019 4 Tools and Commands
8/69
Operating systems - Tools 8
grep
Options:
-c counts lines which contain the pattern
-i case unsensitive
-l shows only names of the files containingthe pattern (usually also the line is shown)
-n shows the line number
-v shows only the lines not containing the
pattern
-
7/30/2019 4 Tools and Commands
9/69
Operating systems - Tools 9
Regular expressions with grep
Patterns in grep can be strings orregularexpressions
Some characters have a special meaning (if
there is not a preceding \) . any character
^ beginning of a line
$ end of a line
-
7/30/2019 4 Tools and Commands
10/69
Operating systems - Tools 10
Regular expressions with grep
* repetition (zero or more times)
+ repetition (one or more times)
[ ] any character among those in
parentheses [^ ] a character excluding those in
parentheses
\< beginning of the word
\> end of the word
-
7/30/2019 4 Tools and Commands
11/69
Operating systems - Tools 11
Recursive search in a diretcory
tree
In order to do a recursive search in adirectory tree:
find . -name * -exec grep pattern \{} \;
Please note that you should use and not `(using ` launches the shell)
-
7/30/2019 4 Tools and Commands
12/69
Operating systems - Tools 12
tar
tar [option] files
Stores and extracts files in a tarfile
-
7/30/2019 4 Tools and Commands
13/69
Operating systems - Tools 13
tar - archive creation
Options to create a tarfile:
-c creates a new tarfile
-f file name of the tarfile
-v verbose
Example:
tar -cvf /tmp/maino.tar /home/maino
-
7/30/2019 4 Tools and Commands
14/69
Operating systems - Tools 14
tar - archive extraction
Options to extract from a tarfile:
-x extracts files from tarfile
-t tests tarfile
-f file name of the tarfile
-v verbose
Examples:
tar -tvf /tmp/maino.tar
tar -xvf /tmp/maino.tar
-
7/30/2019 4 Tools and Commands
15/69
Operating systems - Tools 15
gzip
gzip [opt] file
compresses a file
options:
-1 fastest
-9 max compression
-
7/30/2019 4 Tools and Commands
16/69
Operating systems - Tools 16
gunzip
gunzip file
Decompresses a file
-
7/30/2019 4 Tools and Commands
17/69
Operating systems - Tools 17
Filters
A filteris a program receiving data from stdinand producing results on stdout
Filters are very useful with I/O redirection and
with pipes Examples:
more, less
head, tail
-
7/30/2019 4 Tools and Commands
18/69
Operating systems - Tools 18
Data sorting
sort [-options] [file ...]
options:
-b ignores beginning blanks
-d considers only blanks and alphanumericcharacters
-f case unsensitive
-
7/30/2019 4 Tools and Commands
19/69
Operating systems - Tools 19
Data sorting
sort [-options] [file ...]
-o file writes results on file (default is on stdout)
-r reverse ordering
-tcar field separator
Eg. sort k2.2,2.3 filename orders filename on afield based on characters from 2 to 3 of thesecond field of the file
-
7/30/2019 4 Tools and Commands
20/69
Operating systems - Tools 20
sed - Stream text EDitor
It is a filter for processing the file as givenby script options
sed [-n] script [file ] sed [-n][-e script][-f script_file][file ]
Filters stdin as specified in script
-n does not repeat stdin on stdout; prints only
what is requested from script
-
7/30/2019 4 Tools and Commands
21/69
Operating systems - Tools 21
sed - Stream text EDitor
Script syntax:
[address[,address]] function [args]
address: line number or a regular expression
function: command on the pattern match args: args of function
-
7/30/2019 4 Tools and Commands
22/69
Operating systems - Tools 22
sed functions
p prints current line
d deletes current line
q quits
y/orig/subs/
modifies orig in subs
-
7/30/2019 4 Tools and Commands
23/69
Operating systems - Tools 23
sed functions
s/regexp/replace/flags
replaces patterns satisfying regexpwith replace
flags: num replaces only num occurrences
g replaces all the occurrences
p prints the line if it has replaced something
-
7/30/2019 4 Tools and Commands
24/69
Operating systems - Tools 24
sed: examples
sed 1,3 d filename (deletes from line 1 to 3)
sed 3,$ d filename (deletes from line 3 to theend of file)
sed -n /^pluto/ p filename (prints only ifstarts with pluto, because of -n)
sed -f sedfile filename
-
7/30/2019 4 Tools and Commands
25/69
Operating systems - Tools 25
sed: esempi
sed 1,3 d filename
sed 3,$ d filename
sed -n /^pluto/ p filename
sed -f sedfile filename
1,1 {
s/^/Begin:/
s/$/ -- End/}
/\/\*.*\*\// d
-
7/30/2019 4 Tools and Commands
26/69
Operating systems - Tools 26
sed: esempi
sed 1,3 d filename
sed 3,$ d filename
sed -n /^pluto/ p filename
sed -f sedfile filename
1,1 {
s/^/Begin:/
s/$/ -- End/}
/\/\*.*\*\// d
One Two Three
A B C/* Remark */
-
7/30/2019 4 Tools and Commands
27/69
Operating systems - Tools 27
sed: esempi
sed 1,3 d filename
sed 3,$ d filename
sed -n /^pippo/ p filename
sed -f sedfile filename
1,1 { [only for the first line]
s/^/Begin:/
s/$/ -- End/}
/\/\*.*\*\// d [deletes remarks]
One Two Three
A B C/* Remarks */
Begin: One Two Three --End
A B C
-
7/30/2019 4 Tools and Commands
28/69
Operating systems - Tools 28
Awk
awk was invented in 1977 from
A. V.Aho
P. J. Weinberger
B. W. Kernighan It is a computer language based on string
pattern recognition
-
7/30/2019 4 Tools and Commands
29/69
Operating systems - Tools 29
Basic principle
For each line of the open file, patterns aresearched
When a match is found, an action is done
Syntax is similar to C language
-
7/30/2019 4 Tools and Commands
30/69
Operating systems - Tools 30
Input file
Input file is organized in records (file line)
A record is organized in fields (words of theline)
-
7/30/2019 4 Tools and Commands
31/69
Operating systems - Tools 31
Input file
Variables:
RS: record separator
FS: field separator (blank)
$0: whole record $1: first field
$n: n-th field
-
7/30/2019 4 Tools and Commands
32/69
Operating systems - Tools 32
Execution
It is possible to run awk both from commandline and from a script
Command line:
awk command < inputFile > outputFile
Script:
awk -f scriptFile < inputFile > outputFile
-
7/30/2019 4 Tools and Commands
33/69
Operating systems - Tools 33
Command structure
Each awk command is composed by apattern and by an action
pattern {action;}
Decides when action isexecuted
Can be made by one ormore instructions.
It is executed only ifpattern is true
-
7/30/2019 4 Tools and Commands
34/69
Operating systems - Tools 34
Command structure
If there is not a pattern, action is alwaysexecuted
If there is not an action, the current line isprinted
-
7/30/2019 4 Tools and Commands
35/69
Operating systems - Tools 35
Examples
awk $1==Ciao < inputFile
prints all lines in which the first word is Ciao
awk {print $1;} < inputFile
prints the first field of each line of the file
-
7/30/2019 4 Tools and Commands
36/69
Operating systems - Tools 36
Patterns
Patterns can be:
regular expressions
/ expr /
comparison operators$1 == Ciao
operators for intervals
( cond1, cond2 )
-
7/30/2019 4 Tools and Commands
37/69
Operating systems - Tools 37
Regular expressions
\ escape sequence
^ beginning of the line
$ end of the line
. one character
[abc] one of the characters
[a-z] sequence of characters
[^abc] all characters with the exception ofa, b, c
-
7/30/2019 4 Tools and Commands
38/69
Operating systems - Tools 38
Regular expressions
one|two can be one or two
* 0 or more times of the precedingcharacter
+ 1 or more times of the precedingcharacter
? [AB]? can be A or B or the null string
-
7/30/2019 4 Tools and Commands
39/69
Operating systems - Tools 39
Regular expressions
( ) combines regular expressions
Logi(c|cal) is satisfied by:
- Logic
- Logical
-
7/30/2019 4 Tools and Commands
40/69
Operating systems - Tools 40
Regular expressions -
Examples
/^((may)|(MAY)|(May))$/ { print May; }
/^[Tt]itle.*/ { print \nNew title.; }
-
7/30/2019 4 Tools and Commands
41/69
Operating systems - Tools 41
Comparison operators
== equal
< less than
> greater than = greater or equal
!= not equal
~ equal as a regular expression
!~ not equal as a regular expression
-
7/30/2019 4 Tools and Commands
42/69
Operating systems - Tools 42
Comparison operators
&& logical AND
|| logical OR
! logical NOT
-
7/30/2019 4 Tools and Commands
43/69
Operating systems - Tools 43
Comparison operators -
Examples
$1 == Bob { print Bob stuff; }
$1 !~ /[Mm]aggio/ { print Not may; }
($1 == Bob) && ($2 ~ /[mM]*xy?RR$/) {
print First field is Bob. Second is wrong;}
-
7/30/2019 4 Tools and Commands
44/69
Operating systems - Tools 44
Interval operators
cond1, cond2
Pattern is true only when first condition is true andremains true until second condition becomes true
It works only with nawk o gawk
-
7/30/2019 4 Tools and Commands
45/69
Operating systems - Tools 45
Interval operators
$1==1, $1==CIAO { print $2; }
Test ciao
1 number one
2 number two
CIAO End
3 number three
number
number
End
-
7/30/2019 4 Tools and Commands
46/69
Operating systems - Tools 46
Special patterns
BEGIN: run before opening input file
END: run after having read all input file
-
7/30/2019 4 Tools and Commands
47/69
Operating systems - Tools 47
Example
BEGIN { FS=:; }
$1 ~ /[0-9]/ { print $3; }
Test:ciao
1:number:one
2:number:two
CIAO:End3:number:three
one
two
three
-
7/30/2019 4 Tools and Commands
48/69
Operating systems - Tools 48
Actions
Variables
Strings
Arrays
Operators
If
Loop
Input
Functions Shell interaction
-
7/30/2019 4 Tools and Commands
49/69
Operating systems - Tools 49
Variables
Variables can be:
A field, with a $ ($0, $n)
Predefined User defined (declaration is not requested)
-
7/30/2019 4 Tools and Commands
50/69
Operating systems - Tools 50
Predefined variables
ENVIRON Environment variables (it is anarray). E.g.:
ENVIRON[PATH]
FS field separator (in input) IGNORECASE 0 means case sensitive,
1 no
NF number of fields
NR number of records alreadyread
-
7/30/2019 4 Tools and Commands
51/69
Operating systems - Tools 51
Examples
AA BB CC
print NF; 3
print $NF;CC
$NFprint $NF;
-
7/30/2019 4 Tools and Commands
52/69
Operating systems - Tools 52
String operators
Union:
Putting strings one after the other:
awk '{x=Hello"; y= everybody"; print x,y;} gives true
delete day[march]
-
7/30/2019 4 Tools and Commands
57/69
Operating systems - Tools 57
Multidimensional arrays
With numerical index:
V[5,3]
With symbolic index:
V[abc, def] With mixed index:
V[1999, january]
-
7/30/2019 4 Tools and Commands
58/69
Operating systems - Tools 58
Operators
x+y, x-y, x*y, x/y
x^y, x%y
++x, x++, --x, x--
sin(x), cos(x)
rand( ) (random number among 0 and 1)
systime( ) (seconds starting from January 1st1970)
-
7/30/2019 4 Tools and Commands
59/69
Operating systems - Tools 59
If
if (cond) {
trueinstruction
} else {
falseinstruction
}
condition ? trueinstruction : falseinstruction;
-
7/30/2019 4 Tools and Commands
60/69
Operating systems - Tools 60
Loop
do {
instruction
} while (condition)
for (init; cond; op) {instruction
}
for (i in array) {
instruction
}
-
7/30/2019 4 Tools and Commands
61/69
Operating systems - Tools 61
Loop (cont)
while (condition) {
instruction
}
-
7/30/2019 4 Tools and Commands
62/69
Operating systems - Tools 62
Advance input
exit
is like the end of the file (if END pattern exists, it isrun)
getline reads a new line and copies it in $0. Returns 1 if
the reading has been successful.
-
7/30/2019 4 Tools and Commands
63/69
Operating systems - Tools 63
Functions
Work only with nawk or gawk
function myFunc (parameters)
{
...return x;
}
-
7/30/2019 4 Tools and Commands
64/69
Operating systems - Tools 64
Shell interaction
system (shellCommand);
runs a shell and executes specified command;e.g. system(ls la)
using redirection print Ciao > filename.txt;
print myVar | more;
-
7/30/2019 4 Tools and Commands
65/69
Operating systems - Tools 65
Example
Bookmark file:
+|category
url1|description
url2|description...
+|University
http://www.polito.it|Politecnico di Torino
http://www.unito.it|Universit di Torino+|Music
http://www.mp3.com|Mp3 web site
-
7/30/2019 4 Tools and Commands
66/69
Operating systems - Tools 66
Example
My bookmarks
University
Politecnico di Torino
Universit di Torino
Music
Mp3 web site
-
7/30/2019 4 Tools and Commands
67/69
Operating systems - Tools 67
Header
BEGIN {
FS = |;
firstTime = 1;
printf ();printf (My bookmarks);
printf (\n\n);
}
-
7/30/2019 4 Tools and Commands
68/69
Operating systems - Tools 68
Title
$1 == + {
if (firstTime == 0) {
print ;
} else {firstTime = 0;
}
print $2
;} -
7/30/2019 4 Tools and Commands
69/69
Operating systems - Tools 69
URL and final part
$1 != + {
print $2 ;
}
END {
print
}