Agenda

• Sed Utility - Advanced– Using Script-files / Example

• Awk Utility - Advanced– Using Script-files– Math calculations / Operators / Functions– Floating Point Decimal Calculations

• Sort Utility - Advance– Reverse Sort / Sort & Subsort by Fields

Using Script-Files

• As you recall from the previous lesson, sed and awk utilities can contain search and action commands within single or double quotes (often referred to as “start-stop” commands) which can be executed as a command line

• Unfortunately, executing a one-line sed or awk command may not allow for large quantities of separate categories or search and action for lines contained within a file.

Using Script-Files

• Therefore, for more complex manipulation of records contained within files involving the sed and awk utilities, it may be more efficient to have the utilities read search and instructor or action sequences from a script-file

• We will now demonstrate the use of script-files for the sed and awk utilities

Examples of sedUsing Script-Files

• Assume that you want to display contents of a file called “data” that matches several different searches and instructions:– Display all lines that contain a single character “#” at

the beginning of a comment line– Display all lines that contain three characters “###” at

the beginning of a comment line, but replace “###” with just “#”

– Quit the program at the comment line:“# This is the last comment”

Examples of sedUsing Script-Files

• You could create a script-file called “change” that could contain regular expressions and instructions

Contents of “change” script-file: /^#/ p /^###/ s/###/#/g w output_file /^# This is the last comment/ q

Examples of sedUsing Script-Files

• To execute this command, you would enter:

sed -nf change data

• Refer to my website for further examples of issuing sed commands with script-files

Using Script-Files with awk

• The concept is similar when using script-file with “awk” as it was for “sed”, except that the actions can be more complex.

• In fact, many awk script-files tend to resemble c programming code!

Examples of Using Script-Fileswith awk Utility

Examples:– Generate totals– Generate averages– Format report display– Perform statistical analysis / Create graphs– Change file formats– Perform Floating-Point Decimal Calculations

In fact, there are many programs that you can write in the script-file for awk!

Program File (Script)Basic Structure

The basic structure of a program file is a /pattern/ search followed by an {action}

Below are contents of search program-file: /chevy/ {print $1 $3} /ford/ {print $1 $3}

Command : awk -f search cars

Variables

• The following is a list of common variables that can be used with awk: $0 Current Record

$1 Field number in RecordNF Number of fields in recordNR Record number of current recordFS Input Field Separator (default space / tab)OFS Output Field Separator (default space)RS Input Record Separator (default new line)FILENAME Name of current input file

• Please check my OPS224 webpage for more variables and examples using variables

Functions

• The following is a list of common functions that can be used with awk: length(str)

int(num)index(str1,str2)substr(str,pos,len)tolower(str)toupper(str)

• Please check my OPS224 webpage for more variables and examples using functions

Operators

• The following is a list of common operators that can be used with awk:– +, -, *, /, % (add,subtract,mulitply,divide,modulus)

++, -- (Increment / Decrement variable)

+=, -= (Add/Sub expression to preceeding variable)

*=, /= (Mult/Div expression to preceeding variable)

%= (Modulus after dividing preceeding variable)

• Please check my OPS224 webpage for more variables and examples using operators

Control-Flow Statements

• You can also use control-flow statements to alter the flow of awk’s processing

• Refer to the next slide to see how the syntax for the if, else if else statement resembles c programming language!

Example

• Below are contents of p_range file: { if ($5 <= 5000)

$5 = "inexpensive"

else if ($5 > 5000 && $5 <10000)

$5 = "Please ask"

else if ($5 >= 10000)

$5 = "expensive” else

printf "%-10s %-8s 19%2d %5d %-12s\n",\

$1, $2, $3, $4, $5 }

Command: awk -f p_range cars

Mathematical Calculations Involving awk

• Mathematical calculations can be used with awk’s action statement involving field numbers ($n)

• Amounts can be stored in user-created variables, etc.

Floating-Point Decimal Calculations

• In system such as Phobos, floating-point decimal calculation and display is only available by using awk

• Since $n represents the nth argument or positional parameters, you can actually send arguments through awk to be processed as floating-point decimal (see next slide for an example)

Floating-Point Decimal Calculations - Example

# sends arguments to be processed via awk

echo $* | awk ' { result_1=$1+$2 result_2=$1-$2 result_3=$1*$2 result_4=$1/$2 }

Further Examples

• Refer to your instructor’s webpage for more examples involving the use of script-files with the awk utility

• For more examples, visit my OPS224 website under “Samples - Advanced awk” and “Samples - Floating Point Calculations”

Advanced Sort(Sorting within Fields)

Sort [options] [field specifier list] [file-list]

-r Sorts in reverse order

-k KeyDefinition Specifies a sort key.

-n Sorts numeric fields by arithmetic value. Alpha-numeric sorting (i.e. no “n” option) of a field containing any numeric character gives different results than a numeric sort.

Advanced Sort(Sorting within Fields)

• Sorting fields that contain text: sort -k2 filename (sorts records of filename by second field)

• Sorting fields that contain numbers; sort -kn5 filename (sorts records of filename by fifth field - if “n”

option was not indicated, would perform an alpha-numeric sort!)

Advanced Sort(Sorting within Fields)

• A sub-sort is a method to rearrange records or line in order by a secondary field if patterns in the primary field are identifcal (eg sub-sort by model for all chevy’s)

• sort -k1,2 filename (sorts by field 1 and sub-sort by field 2)