4 tools and commands

7/30/2019 4 Tools and Commands

1/69

Operating systems - Tools 1

Unix Operating System

Tools and commands

[email protected]

[email protected]

Politecnico di Torino

Dip. Automatica e Informatica

2010/2011

Original copyright from Giorgio Di Natale, Stefano Di Carlo and Fabio Maino


2/69


Outline

Some commands find

grep

tar, gzip, gunzip Filters

sort

sed

awk


3/69


File search

find [-opt]

some options:

-name patternWarning: use single quotes if you are usingregular expressions

-type [b c d l]


4/69


Advanced use of find

It is possible to execute a command on allfound files specifying:

-exec command \;

In command, use \{} for the current file example:

find . -name core -exec rm \{} \;


5/69



Search expressions can contain logicalconditions

Logical operations:

AND: conditions must be listed one after the other

OR: use -o

NOT: use !


6/69



Use ( ) to build complex expressions

Example:find . $ -name core -o -size 10b $

find . $ -name test -o ! \( -size 10c $ \)

\ is to protect ( from the shell expansion; 10cmeans 10 byte, while 10b means 10 blocks


7/69


grep

In order to search a string in a set of files, it ispossible to use:

grep [-options] pattern files


8/69


grep

Options:

-c counts lines which contain the pattern

-i case unsensitive

-l shows only names of the files containingthe pattern (usually also the line is shown)

-n shows the line number

-v shows only the lines not containing the

pattern


9/69


Regular expressions with grep

Patterns in grep can be strings orregularexpressions

Some characters have a special meaning (if

there is not a preceding \) . any character

^ beginning of a line

$ end of a line


10/69


Regular expressions with grep

* repetition (zero or more times)

+ repetition (one or more times)

[ ] any character among those in

parentheses [^ ] a character excluding those in

parentheses

\< beginning of the word

\> end of the word


11/69


Recursive search in a diretcory

tree

In order to do a recursive search in adirectory tree:

find . -name * -exec grep pattern \{} \;

Please note that you should use and not `(using ` launches the shell)


12/69


tar

tar [option] files

Stores and extracts files in a tarfile


13/69


tar - archive creation

Options to create a tarfile:

-c creates a new tarfile

-f file name of the tarfile

-v verbose

Example:

tar -cvf /tmp/maino.tar /home/maino


14/69


tar - archive extraction

Options to extract from a tarfile:

-x extracts files from tarfile

-t tests tarfile

-f file name of the tarfile

-v verbose

Examples:

tar -tvf /tmp/maino.tar

tar -xvf /tmp/maino.tar


15/69


gzip

gzip [opt] file

compresses a file

options:

-1 fastest

-9 max compression


16/69


gunzip

gunzip file

Decompresses a file


17/69


Filters

A filteris a program receiving data from stdinand producing results on stdout

Filters are very useful with I/O redirection and

with pipes Examples:

more, less

head, tail


18/69


Data sorting

sort [-options] [file ...]

options:

-b ignores beginning blanks

-d considers only blanks and alphanumericcharacters

-f case unsensitive


19/69


Data sorting

sort [-options] [file ...]

-o file writes results on file (default is on stdout)

-r reverse ordering

-tcar field separator

Eg. sort k2.2,2.3 filename orders filename on afield based on characters from 2 to 3 of thesecond field of the file


20/69


sed - Stream text EDitor

It is a filter for processing the file as givenby script options

sed [-n] script [file ] sed [-n][-e script][-f script_file][file ]

Filters stdin as specified in script

-n does not repeat stdin on stdout; prints only

what is requested from script


21/69


sed - Stream text EDitor

Script syntax:

[address[,address]] function [args]

address: line number or a regular expression

function: command on the pattern match args: args of function


22/69


sed functions

p prints current line

d deletes current line

q quits

y/orig/subs/

modifies orig in subs


23/69


sed functions

s/regexp/replace/flags

replaces patterns satisfying regexpwith replace

flags: num replaces only num occurrences

g replaces all the occurrences

p prints the line if it has replaced something


24/69


sed: examples

sed 1,3 d filename (deletes from line 1 to 3)

sed 3,$ d filename (deletes from line 3 to theend of file)

sed -n /^pluto/ p filename (prints only ifstarts with pluto, because of -n)

sed -f sedfile filename


25/69


sed: esempi

sed 1,3 d filename

sed 3,$ d filename

sed -n /^pluto/ p filename


1,1 {

s/^/Begin:/

s/$/ -- End/}

/\/\*.*\*\// d


26/69


sed: esempi

sed 1,3 d filename

sed 3,$ d filename

sed -n /^pluto/ p filename


1,1 {

s/^/Begin:/

s/$/ -- End/}

/\/\*.*\*\// d

One Two Three

A B C/* Remark */


27/69


sed: esempi

sed 1,3 d filename

sed 3,$ d filename

sed -n /^pippo/ p filename


1,1 { [only for the first line]

s/^/Begin:/

s/$/ -- End/}

/\/\*.*\*\// d [deletes remarks]

One Two Three

A B C/* Remarks */

Begin: One Two Three --End

A B C


28/69


Awk

awk was invented in 1977 from

A. V.Aho

P. J. Weinberger

B. W. Kernighan It is a computer language based on string

pattern recognition


29/69


Basic principle

For each line of the open file, patterns aresearched

When a match is found, an action is done

Syntax is similar to C language


30/69


Input file

Input file is organized in records (file line)

A record is organized in fields (words of theline)


31/69


Input file

Variables:

RS: record separator

FS: field separator (blank)

$0: whole record $1: first field

$n: n-th field


32/69


Execution

It is possible to run awk both from commandline and from a script

Command line:

awk command < inputFile > outputFile

Script:

awk -f scriptFile < inputFile > outputFile


33/69


Command structure

Each awk command is composed by apattern and by an action

pattern {action;}

Decides when action isexecuted

Can be made by one ormore instructions.

It is executed only ifpattern is true


34/69


Command structure

If there is not a pattern, action is alwaysexecuted

If there is not an action, the current line isprinted


35/69


Examples

awk $1==Ciao < inputFile

prints all lines in which the first word is Ciao

awk {print $1;} < inputFile

prints the first field of each line of the file


36/69


Patterns

Patterns can be:

regular expressions

/ expr /

comparison operators$1 == Ciao

operators for intervals

( cond1, cond2 )


37/69


Regular expressions

\ escape sequence

^ beginning of the line

$ end of the line

. one character

[abc] one of the characters

[a-z] sequence of characters

[^abc] all characters with the exception ofa, b, c


38/69


Regular expressions

one|two can be one or two

* 0 or more times of the precedingcharacter

+ 1 or more times of the precedingcharacter

? [AB]? can be A or B or the null string


39/69


Regular expressions

( ) combines regular expressions

Logi(c|cal) is satisfied by:

- Logic

- Logical


40/69


Regular expressions -

Examples

/^((may)|(MAY)|(May))$/ { print May; }

/^[Tt]itle.*/ { print \nNew title.; }


41/69


Comparison operators

== equal

< less than

> greater than = greater or equal

!= not equal

~ equal as a regular expression

!~ not equal as a regular expression


42/69


Comparison operators

&& logical AND

|| logical OR

! logical NOT


43/69


Comparison operators -

Examples

$1 == Bob { print Bob stuff; }

$1 !~ /[Mm]aggio/ { print Not may; }

($1 == Bob) && ($2 ~ /[mM]*xy?RR$/) {

print First field is Bob. Second is wrong;}


44/69


Interval operators

cond1, cond2

Pattern is true only when first condition is true andremains true until second condition becomes true

It works only with nawk o gawk


45/69


Interval operators

$1==1, $1==CIAO { print $2; }

Test ciao

1 number one

2 number two

CIAO End

3 number three

number

number

End


46/69


Special patterns

BEGIN: run before opening input file

END: run after having read all input file


47/69


Example

BEGIN { FS=:; }

$1 ~ /[0-9]/ { print $3; }

Test:ciao

1:number:one

2:number:two

CIAO:End3:number:three

one

two

three


48/69


Actions

Variables

Strings

Arrays

Operators

If

Loop

Input

Functions Shell interaction


49/69


Variables

Variables can be:

A field, with a $ ($0, $n)

Predefined User defined (declaration is not requested)


50/69


Predefined variables

ENVIRON Environment variables (it is anarray). E.g.:

ENVIRON[PATH]

FS field separator (in input) IGNORECASE 0 means case sensitive,

1 no

NF number of fields

NR number of records alreadyread


51/69


Examples

AA BB CC

print NF; 3

print $NF;CC

$NFprint $NF;


52/69


String operators

Union:

Putting strings one after the other:

awk '{x=Hello"; y= everybody"; print x,y;} gives true

delete day[march]


57/69


Multidimensional arrays

With numerical index:

V[5,3]

With symbolic index:

V[abc, def] With mixed index:

V[1999, january]


58/69


Operators

x+y, x-y, x*y, x/y

x^y, x%y

++x, x++, --x, x--

sin(x), cos(x)

rand( ) (random number among 0 and 1)

systime( ) (seconds starting from January 1st1970)


59/69


If

if (cond) {

trueinstruction

} else {

falseinstruction

}

condition ? trueinstruction : falseinstruction;


60/69


Loop

do {

instruction

} while (condition)

for (init; cond; op) {instruction

}

for (i in array) {

instruction

}


61/69


Loop (cont)

while (condition) {

instruction

}


62/69


Advance input

exit

is like the end of the file (if END pattern exists, it isrun)

getline reads a new line and copies it in $0. Returns 1 if

the reading has been successful.


63/69


Functions

Work only with nawk or gawk

function myFunc (parameters)

{

...return x;

}


64/69


Shell interaction

system (shellCommand);

runs a shell and executes specified command;e.g. system(ls la)

using redirection print Ciao > filename.txt;

print myVar | more;


66/69


Example

My bookmarks

University

Politecnico di Torino

Universit di Torino

Music

Mp3 web site


67/69


Header

BEGIN {

FS = |;

firstTime = 1;

printf ();printf (My bookmarks);

printf (\n\n);

}


68/69


Title

$1 == + {

if (firstTime == 0) {

print ;

} else {firstTime = 0;

}

print $2
;}


69/69


URL and final part

$1 != + {

print $2 ;

}

END {

print

}

4 tools and commands

Documents