Chapter 10

Advanced File Processing

Regular Expressions• A compact notation for representing patterns in strings• Used by many common Linux utilities such as:

– less/more– ed– vi– emacs– egrep– grep– fgrep– awk– sed– lex– perl– python

Table 10.1 Regular Expression Operators

Table 10.1 More Regular Expression Operators

Regular Expression Examplescat|dog matches "cat" or "dog“..at matches any three-character string

ending with "at", including "hat", "cat", and "bat".

[hc]at matches "hat" and "cat".[^b]at matches all strings matched by .at

except "bat".^[hc]at matches "hat" and "cat", but only at the

beginning of the string or line.[hc]at$ matches "hat" and "cat", but only at the

end of the string or line.

Regular Expression Examples

• (0|1)+• 1+0*• a*ba*ba*• [A-Za-z][A-Za-z0-9]*• (yaba|daba)+doo

Table 10.2 Examples of Regular Expressions for vi and Their Meaning

Table 10.3 Some Commonly Used vi Commands Illustrating the Use of Regular

Expressions

File Compression• compress• uncompress• gzip• gunzip• gzexe• zcat• zmore• uuencode• uudecode

Figure 10.1 The process of uuencoding and uudecoding

The Sort Utility

• Sorts lines of ASCII text• Options for:– Ascending/descending order– Case folding– Ignoring leading whitespace– Alpha/numeric– Multiple keys

sort - sort lines of text files• Syntax

sort [OPTION]... [FILE]...• Options

-b, --ignore-leading-blanks ignore leading blanks-d, --dictionary-order consider only blanks and alphanumeric

characters-f, --ignore-case fold lower case to upper case characters-i, --ignore-nonprinting consider only printable characters-n, --numeric-sort compare according to string numerical value-r, --reverse reverse the result of comparisons Other options:-k, --key=POS1[,POS2] start a key at POS1, end it at POS2 (origin 1)-t, --field-separator=SEP use SEP instead of non-blank to blank

transition

Sort examples

File donors.datBay Ching 500000 ChinaJack Arta 250000 IndonesiaCruella Lumper 725000 Malaysia

sort donors.datBay Ching 500000 ChinaCruella Lumper 725000 MalaysiaJack Arta 250000 Indonesia

Sort Example – by last name

sort +1 donors.dat Jack Arta 250000 Indonesia Bay Ching 500000 China Cruella Lumper 725000 Malaysia

Sort Example – by donation amount

sort +2 donors.dat Jack Arta 250000 Indonesia Bay Ching 500000 China Cruella Lumper 725000 Malaysia

sort +2 -n donors.dat Jack Arta 250000 Indonesia Bay Ching 500000 China Cruella Lumper 725000 Malaysia

sort +2 -n -r donors.dat Cruella Lumper 725000 Malaysia Bay Ching 500000 China Jack Arta 250000 Indonesia

Locating Files

• find• whereis• which

Find command

SyntaxFind directory-list expression

• Searches directory-list for files that match conditions in expression

• Can find files by name, owner, permission, size, age, links

Find Command Examples

find ~ -name *.gif -printfind / -name socket.h -printfind / -size+5000000c -printfind ~ -newer filetocompare

Searching File Contents

• grep• egrep• fgrep

Grep – Get Regular Expression & Print

Syntaxgrep [options] pattern [file-list]

• Searches files in file-list, prints lines which match the given pattern

• Options to fold case, print line #’s, filenames, non-matching lines

• fgrep – faster but dumb• egrep – slower but more powerful

Command History

A list of the recently used commands is maintained and controlled by the variables:– HISTSIZE– HISTFILE– HISTFILESIZE

Table 10.4 Commonly Used Event Designators for History Expansion

Table 10.5 Commonly Used Event Designators for History Expansion

Table 10.5 Commonly Used Event Designators for History Expansion

Table 10.6 Commonly Used Modifiers for History Expansion