![Page 1: Filters and Utilities. Notes: This is a simple overview of the filtering capability Some of these commands are very powerful ▫Only showing some of the](https://reader035.vdocument.in/reader035/viewer/2022062720/56649f0e5503460f94c22485/html5/thumbnails/1.jpg)
Filters and Utilities
![Page 2: Filters and Utilities. Notes: This is a simple overview of the filtering capability Some of these commands are very powerful ▫Only showing some of the](https://reader035.vdocument.in/reader035/viewer/2022062720/56649f0e5503460f94c22485/html5/thumbnails/2.jpg)
Notes:
•This is a simple overview of the filtering capability
•Some of these commands are very powerful▫Only showing some of the basics of a few of
the commands
![Page 3: Filters and Utilities. Notes: This is a simple overview of the filtering capability Some of these commands are very powerful ▫Only showing some of the](https://reader035.vdocument.in/reader035/viewer/2022062720/56649f0e5503460f94c22485/html5/thumbnails/3.jpg)
Reminder:
•Grave accent▫AKA backtick or backquote▫Used for command substitution in bash and
other Linux utilities and languages▫Typical use:
put a command between a pair of ` the std out of the command is substituted
▫Example: #echo The date is:`date`!#The date is:Sun Mar 17 15L51:28 EDT 2013!
![Page 4: Filters and Utilities. Notes: This is a simple overview of the filtering capability Some of these commands are very powerful ▫Only showing some of the](https://reader035.vdocument.in/reader035/viewer/2022062720/56649f0e5503460f94c22485/html5/thumbnails/4.jpg)
What are Filters?
▫Use std in and std out Monitor the input Modify data as appropriate
Change Delete Move "as appropriate"
Send data to standard out
![Page 5: Filters and Utilities. Notes: This is a simple overview of the filtering capability Some of these commands are very powerful ▫Only showing some of the](https://reader035.vdocument.in/reader035/viewer/2022062720/56649f0e5503460f94c22485/html5/thumbnails/5.jpg)
Filter examples• Simple
▫ pr▫ cmp▫ diff▫ comm▫ head▫ tail▫ cut▫ paste▫ sort▫ uniq▫ tr
• Complex▫ grep▫ sed
• Filter/script▫ awk
![Page 6: Filters and Utilities. Notes: This is a simple overview of the filtering capability Some of these commands are very powerful ▫Only showing some of the](https://reader035.vdocument.in/reader035/viewer/2022062720/56649f0e5503460f94c22485/html5/thumbnails/6.jpg)
pr: Paginate Files• Prepare files for printing• Adds:
▫Headers▫Footers▫Formatted text
• Default adds 5 lines before and after text on page• Options:
▫Make columns▫Set page length▫Set page width▫Number lines in output
![Page 7: Filters and Utilities. Notes: This is a simple overview of the filtering capability Some of these commands are very powerful ▫Only showing some of the](https://reader035.vdocument.in/reader035/viewer/2022062720/56649f0e5503460f94c22485/html5/thumbnails/7.jpg)
cmp: Byte by Byte Compare
•Compares two files•Terminates on first delta
▫Echoes the location of first mismatch Usually reports line and character position
▫Returns: True if identical False otherwise
![Page 8: Filters and Utilities. Notes: This is a simple overview of the filtering capability Some of these commands are very powerful ▫Only showing some of the](https://reader035.vdocument.in/reader035/viewer/2022062720/56649f0e5503460f94c22485/html5/thumbnails/8.jpg)
comm: What Is Common between files•Compares files line by line
▫Requires sorted files to work properly•Returns 3 types of differently indented lines
▫ Lines unique to first file▫ Lines unique to second file▫ Lines common to both
• Output is “weird” in columns1st col is lines unique to 1st file
2nd col is lines unique to 2nd file3rd col is common lines
comm.sh in ~/ITIS3110/bashscriptscommbad.sh (with error)
![Page 9: Filters and Utilities. Notes: This is a simple overview of the filtering capability Some of these commands are very powerful ▫Only showing some of the](https://reader035.vdocument.in/reader035/viewer/2022062720/56649f0e5503460f94c22485/html5/thumbnails/9.jpg)
diff: "How to make files the same"•Details how to change one file to make it
the same as the other▫For deltas instructions of how to change
![Page 10: Filters and Utilities. Notes: This is a simple overview of the filtering capability Some of these commands are very powerful ▫Only showing some of the](https://reader035.vdocument.in/reader035/viewer/2022062720/56649f0e5503460f94c22485/html5/thumbnails/10.jpg)
head: Display beginning of file
•Show the first n lines of a file▫Default is 10▫Can change with –n x
•Example use:▫Want to re-edit the last file you edited:▫nano `ls –t | head –n 1`
ls –t: list by time head –n 1: list first entry Feed as a parameter to nano with the
backticks
![Page 11: Filters and Utilities. Notes: This is a simple overview of the filtering capability Some of these commands are very powerful ▫Only showing some of the](https://reader035.vdocument.in/reader035/viewer/2022062720/56649f0e5503460f94c22485/html5/thumbnails/11.jpg)
tail: Display end of file
•Show the last n lines of a file▫Default is 10▫Can change with –n x
•Options▫-f
Monitor the file as it grows Must terminate with <ctrl-C>
▫-c Do the last n chars instead of lines
![Page 12: Filters and Utilities. Notes: This is a simple overview of the filtering capability Some of these commands are very powerful ▫Only showing some of the](https://reader035.vdocument.in/reader035/viewer/2022062720/56649f0e5503460f94c22485/html5/thumbnails/12.jpg)
cut: Splitting a file vertically• Cuts a range out by:
▫Columns Good for fixed length entries -c range
-c1-4▫Fields
Good for delimited entries Tab is default
-d specifies delimiter -d/ set the / as the delimiter
-f specifies the fields to use -f1,4 specifies the first and fourth fields
![Page 13: Filters and Utilities. Notes: This is a simple overview of the filtering capability Some of these commands are very powerful ▫Only showing some of the](https://reader035.vdocument.in/reader035/viewer/2022062720/56649f0e5503460f94c22485/html5/thumbnails/13.jpg)
paste: Paste files vertically
•Paste two files together line by line•Can be used on a single file to join
multiple sequential lines together▫-s
Do serial on a single file▫-d
Separate joined element with the list of delimiters
![Page 14: Filters and Utilities. Notes: This is a simple overview of the filtering capability Some of these commands are very powerful ▫Only showing some of the](https://reader035.vdocument.in/reader035/viewer/2022062720/56649f0e5503460f94c22485/html5/thumbnails/14.jpg)
sort: Order files• Put files in order
▫ Default is ascending order on column 1 ASCII order
• Options:▫ -t
Define a delimiter▫ -k
Used with –t, which field to use Can have multiple keys
Use commas to separate ranges Use –k again to denote a new field
Can sort on columns in a field Use a dot to separate
▫ -n Treat a field as a number, not an ASCII character Remember the number 1 is different than the character "1"
▫ -u Remove repeated lines
![Page 15: Filters and Utilities. Notes: This is a simple overview of the filtering capability Some of these commands are very powerful ▫Only showing some of the](https://reader035.vdocument.in/reader035/viewer/2022062720/56649f0e5503460f94c22485/html5/thumbnails/15.jpg)
uniq: Locating identical lines
•Returns only unique lines▫Options:
-u Return only the non-repeated lines
-d Return only the repeated lines
▫But only one copy of each -c
Return the count of how many times each line is repeated
![Page 16: Filters and Utilities. Notes: This is a simple overview of the filtering capability Some of these commands are very powerful ▫Only showing some of the](https://reader035.vdocument.in/reader035/viewer/2022062720/56649f0e5503460f94c22485/html5/thumbnails/16.jpg)
tr: Translate characters• Changes one set of characters to another, default input is
the standard input• Example:
▫ #tr 'ab' 'cd'This is abnormalThis is cdnormclabsolutecdsoluteab a b ccd c d c^C Blue is std in Red is std out – bold is what changed
▫ Note: a c and b d, not ab cd▫ Note: ^D can be used to denote end of file to tr instead of
the shown ^C which stops the process tr
![Page 17: Filters and Utilities. Notes: This is a simple overview of the filtering capability Some of these commands are very powerful ▫Only showing some of the](https://reader035.vdocument.in/reader035/viewer/2022062720/56649f0e5503460f94c22485/html5/thumbnails/17.jpg)
tr: Translate characters• More examples:
▫Can be used to translate case for a file tr a-z A-Z <file1ortr '[a-z]' '[A-Z]' <file1 Takes the input from file1 with the < redirection Turns all lower case letters to upper case Output goes to std out
▫Get rid of characters tr –d [a-z] <file1
Gets rid of all lower case chars from file1 Again output is std out
▫Compressing repeated chars tr –s ' ' <file1
Changes repeated spaces to a single space
![Page 18: Filters and Utilities. Notes: This is a simple overview of the filtering capability Some of these commands are very powerful ▫Only showing some of the](https://reader035.vdocument.in/reader035/viewer/2022062720/56649f0e5503460f94c22485/html5/thumbnails/18.jpg)
Filters Using Regular Expressions
![Page 19: Filters and Utilities. Notes: This is a simple overview of the filtering capability Some of these commands are very powerful ▫Only showing some of the](https://reader035.vdocument.in/reader035/viewer/2022062720/56649f0e5503460f94c22485/html5/thumbnails/19.jpg)
Regular Expression Review
![Page 20: Filters and Utilities. Notes: This is a simple overview of the filtering capability Some of these commands are very powerful ▫Only showing some of the](https://reader035.vdocument.in/reader035/viewer/2022062720/56649f0e5503460f94c22485/html5/thumbnails/20.jpg)
Regular Expression
•A pattern to match strings of text which is:▫Concise▫Flexible
•Used by many programming languages and operating systems
![Page 21: Filters and Utilities. Notes: This is a simple overview of the filtering capability Some of these commands are very powerful ▫Only showing some of the](https://reader035.vdocument.in/reader035/viewer/2022062720/56649f0e5503460f94c22485/html5/thumbnails/21.jpg)
Regular Expressions
•BRE▫Basic Regular Expression
•ERE▫Extended Regular Expression
•IRE▫Interval Regular Expression
•TRE▫Tagged Regular Expression
![Page 22: Filters and Utilities. Notes: This is a simple overview of the filtering capability Some of these commands are very powerful ▫Only showing some of the](https://reader035.vdocument.in/reader035/viewer/2022062720/56649f0e5503460f94c22485/html5/thumbnails/22.jpg)
Character class
•Set of characters enclosed within square brackets [ ]▫Can be a list of single characters
[aD1] a, D, and the character 1 only
▫Can be a range of characters [a-zA-Z]
All the upper and lower case chars
▫Negate a class [^0-9]
Not the numeric chars 0-9
![Page 23: Filters and Utilities. Notes: This is a simple overview of the filtering capability Some of these commands are very powerful ▫Only showing some of the](https://reader035.vdocument.in/reader035/viewer/2022062720/56649f0e5503460f94c22485/html5/thumbnails/23.jpg)
Regular Expressions
•*▫Refers to the immediately proceeding
character▫Any number of repeated character(s)
0 or more Used with other patterns
[A*]▫Anything that matches 0 or more ‘A’s in a row
▫s*print will match sprint, ssprint, sssprint and print!
• Note: this is not related to the familiar wildcard *
![Page 24: Filters and Utilities. Notes: This is a simple overview of the filtering capability Some of these commands are very powerful ▫Only showing some of the](https://reader035.vdocument.in/reader035/viewer/2022062720/56649f0e5503460f94c22485/html5/thumbnails/24.jpg)
Regular Expressions• .
▫Any character Exactly one
▫S... with match Sort, Sxxx, … Any four char string starting with S
▫Note .* means 0 or more of any character•Pattern starting locations
▫^ Pattern starts at the beginning of a line
▫$ Pattern starts at the end of a line
![Page 25: Filters and Utilities. Notes: This is a simple overview of the filtering capability Some of these commands are very powerful ▫Only showing some of the](https://reader035.vdocument.in/reader035/viewer/2022062720/56649f0e5503460f94c22485/html5/thumbnails/25.jpg)
Extended Regular Expressions
•|▫Either one of a set▫[a|b]
Matches if an a or a b•( and )
▫Chars between the parenthesis and what is before or after
▫‘animaltype:(dog|cat)’ look for animaltype:dog or animaltype:cat
![Page 26: Filters and Utilities. Notes: This is a simple overview of the filtering capability Some of these commands are very powerful ▫Only showing some of the](https://reader035.vdocument.in/reader035/viewer/2022062720/56649f0e5503460f94c22485/html5/thumbnails/26.jpg)
Resume 9/8
![Page 27: Filters and Utilities. Notes: This is a simple overview of the filtering capability Some of these commands are very powerful ▫Only showing some of the](https://reader035.vdocument.in/reader035/viewer/2022062720/56649f0e5503460f94c22485/html5/thumbnails/27.jpg)
Advanced Filters
![Page 28: Filters and Utilities. Notes: This is a simple overview of the filtering capability Some of these commands are very powerful ▫Only showing some of the](https://reader035.vdocument.in/reader035/viewer/2022062720/56649f0e5503460f94c22485/html5/thumbnails/28.jpg)
grep
![Page 29: Filters and Utilities. Notes: This is a simple overview of the filtering capability Some of these commands are very powerful ▫Only showing some of the](https://reader035.vdocument.in/reader035/viewer/2022062720/56649f0e5503460f94c22485/html5/thumbnails/29.jpg)
grep – Search a pattern
•Searches for a pattern in a file▫grep options pattern filename(s)
std in is used if there is no filename Can also pipe data to grep
▫Notes: Pattern does not need be quoted if no
delimiters or special chars in it Can always use quotes to be safe
![Page 30: Filters and Utilities. Notes: This is a simple overview of the filtering capability Some of these commands are very powerful ▫Only showing some of the](https://reader035.vdocument.in/reader035/viewer/2022062720/56649f0e5503460f94c22485/html5/thumbnails/30.jpg)
grep - Options• -i
▫ Ignore case• -v
▫ Don’t display lines matching expression• -l
▫ Display filenames Useful when grepping multiple files
• -e▫ Useful when grepping for –
• -x▫ Match entire line
• -f file▫ Takes expression from a file
![Page 31: Filters and Utilities. Notes: This is a simple overview of the filtering capability Some of these commands are very powerful ▫Only showing some of the](https://reader035.vdocument.in/reader035/viewer/2022062720/56649f0e5503460f94c22485/html5/thumbnails/31.jpg)
grep - examples
•Examples:▫#grep 3 bigfile3file 3 text
▫#grep file bigfilefile 1 textfile 1 textfile 3 textfile 1 textfile 1 textfile 1 text
#cat bigfile3file 1 textfile 1 textfile 3 textfile 1 textfile 1 textfile 1 text
![Page 32: Filters and Utilities. Notes: This is a simple overview of the filtering capability Some of these commands are very powerful ▫Only showing some of the](https://reader035.vdocument.in/reader035/viewer/2022062720/56649f0e5503460f94c22485/html5/thumbnails/32.jpg)
sed
![Page 33: Filters and Utilities. Notes: This is a simple overview of the filtering capability Some of these commands are very powerful ▫Only showing some of the](https://reader035.vdocument.in/reader035/viewer/2022062720/56649f0e5503460f94c22485/html5/thumbnails/33.jpg)
sed – Streaming Editor• Edit a file(s) with a specified action
▫ sed options 'address action' file(s)• Basics:
▫ Take input from the file(s)▫ Performs the action on the file(s)▫ Sends output to std out
• Uses:▫ Select part(s) of a file
By line By content
▫ Edit a file e.g. create a template, then use sed to customize for a run
• Oddities▫ Usually need –n to get rid of unwanted duplicated lines
![Page 34: Filters and Utilities. Notes: This is a simple overview of the filtering capability Some of these commands are very powerful ▫Only showing some of the](https://reader035.vdocument.in/reader035/viewer/2022062720/56649f0e5503460f94c22485/html5/thumbnails/34.jpg)
sed – Line addressing• Select specific lines
▫ #sed '3q' tenline.fileLine 1Line 2Line 3 Selects the first 3 lines then quits
▫ #sed '$p' tenline.fileLast Line Prints last line
$ - last line p – print
▫ #sed '5,7p' tenline.fileLine 5Line 6Line 7 Prints lines 5 through 7
#cat tenline.fileLine 1Line 2Line 3Line 4Line 5Line 6Line 7Line 8Line 9Last Line
![Page 35: Filters and Utilities. Notes: This is a simple overview of the filtering capability Some of these commands are very powerful ▫Only showing some of the](https://reader035.vdocument.in/reader035/viewer/2022062720/56649f0e5503460f94c22485/html5/thumbnails/35.jpg)
sed – Line addressing• Select specific lines with ;
▫ #sed '1p;3p;$p' tenline.fileLine 1Line 3Last Line Prints line 1, 3 and the last line ($)
• ! Will negate operations▫ #sed '3,$!p' tenline.fileLine 1Line 2 Does not print line 3 through the end
• Notes:▫ By default sed will echo the input lines as well as the
selected lines get duplicated lines Use –n to not echo the input lines
![Page 36: Filters and Utilities. Notes: This is a simple overview of the filtering capability Some of these commands are very powerful ▫Only showing some of the](https://reader035.vdocument.in/reader035/viewer/2022062720/56649f0e5503460f94c22485/html5/thumbnails/36.jpg)
sed – Context addressing
•Use a pattern to identify lines to work with▫Use / to delimit the pattern
•Examples▫#sed –n '/2/p' tenline.fileLine 2 Find all lines with 2 in them and print
▫#sed –n '/^2/p' tenline.file Finds all lines that start with 2 and print ^ - starting the line
![Page 37: Filters and Utilities. Notes: This is a simple overview of the filtering capability Some of these commands are very powerful ▫Only showing some of the](https://reader035.vdocument.in/reader035/viewer/2022062720/56649f0e5503460f94c22485/html5/thumbnails/37.jpg)
sed – Writing selected lines to a file
•Can use w to write the selected lines to a file
•Example▫sed –n '/2/w twos.file' tenline.file
w instead of p puts the output to a file -n does not print duplicated
![Page 38: Filters and Utilities. Notes: This is a simple overview of the filtering capability Some of these commands are very powerful ▫Only showing some of the](https://reader035.vdocument.in/reader035/viewer/2022062720/56649f0e5503460f94c22485/html5/thumbnails/38.jpg)
sed – Text editing•Can edit the stream
▫i Insert
▫a Append
▫c Change
▫d Delete
▫s Substitute
![Page 39: Filters and Utilities. Notes: This is a simple overview of the filtering capability Some of these commands are very powerful ▫Only showing some of the](https://reader035.vdocument.in/reader035/viewer/2022062720/56649f0e5503460f94c22485/html5/thumbnails/39.jpg)
sed - editing•Example: inserting
▫#sed '1i\>#!/bin/bash\># using the bash shell>' test.sh > $$ Notes:
1i inserts text starting line 1 Need \ as a continuation character within the quotes Input is the code or text in test.sh Redirecting the output to $$ (temporary file) Ends up with the 2 new lines at the beginning in $$ Can further modify $$
![Page 40: Filters and Utilities. Notes: This is a simple overview of the filtering capability Some of these commands are very powerful ▫Only showing some of the](https://reader035.vdocument.in/reader035/viewer/2022062720/56649f0e5503460f94c22485/html5/thumbnails/40.jpg)
sed - editing
•Use s to indicate substitution•Example: substituting
▫sed 's/a/b/' file replaces a with b for the first instance on
each line▫sed 's/a/b/g' file
g (global) replaces a with b for all instances on each line