1

EE 355 Unit 16

Linux Utilities

Mark Redekopp

2

File Utilities

• chmod– Files and directories have permissions based on who is

trying to access it and what kind of access is desired

• Can specify as 3 octal digits:– chmod 644 * [change all files to have r/w user access and

read access for all others]

– chmod u+x test.txt [u,g,a][+/-][r,w,x]

Read Write Execute

User Yes/No Yes/No Yes/No

Group Yes/No Yes/No Yes/No

All Yes/No Yes/No Yes/No

3

Activity 1

• ssh into parallel05 or aludra.usc.edu

– $ ssh –Y username@parallel05.usc.edu

– $ ssh –Y username@aludra.usc.edu

• Use your USC password first, then try 10-digit ID

• Make a directory

– $ mkdir temp

• Attempt to access – [If aludra]: $ cat /home/scf-22/ee499mr/hi.txt

– [If Parallel05]: $ cat /home/parallel05/redekopp/rel/hi.txt

• Reattempt

4

File Utilities

• Copies from to/from current machine to another network machine over an SSH connection

• scp current_file dest_loc

– Current_file or dest_location can be on a remote machine which requires user@machine syntax before file location

• Examples:– scp username@aludra.usc.edu:temp/hi.txt .

– scp hi.txt username@aludra.usc.edu:temp/

5

Activity 2

• From Mac or Linux Terminal on your PC/VM, find a file on your local drive and scp it to your temp directory on aludra or parallel05.

6

Download

• Download sample files– $ wget http://ee.usc.edu/~redekopp/ee355/code/linux_tools.tar

– $ tar xvf linux_tools.tar

• Be sure you get these 3 files

– re.txt

– war_and_peace.txt

– linux_grades_comma.txt

7

Text Utilities

• more filename

– Prints the specified file one screen at a time

– ‘q’ to quit

• cat filename

– Prints the specified file to the screen

• grep ‘pattern’ filename(s)

– Find occurrences of pattern in filename

8

Text Utilities

• wc

– Count words or lines in a text file

– wc –l for number of lines

– wc –w for number of words

• sort

– Sorts a text file by a specified column of data

– -n (numeric string order) versus dictionary ordering

– -knum field number to sort by

– -tchar (field delimeter…defaults to whitespace)

• cut

– Extracts column of text from a text file

– -dchar indicate delimeter [TAB] by default

– -fnum indicates which field to extract

9

Redirection & Pipes

• '<' redirect contents of a file as input to program

– ./puzzle 3 30 1537 2 < input.txt

• '>' redirect program output to a file

– sort input.txt > sorted_text.txt

• '>&' redirect stderr to a file

– g++ -o test test.cpp >& compile.log

• '|' pipe output of first program to second

– grep "day" re.txt l wc -l

• 'l&' pipe stderr of first program to second

– g++ -o test test.cpp l& grep error

10

Activity 3

• Sort the data by Test 1 scores from smallest to largest

• Extract sorted Test 2 scores and write them to a file “test2.txt”

• Write a command (using pipes) to count how many students got a 100 on Test 3

11

Regular Expressions

• Used to find substrings occurrences within a larger string which is useful for many text processing applications

• Regular expressions can be thought of as a finite state machine that will receive the larger string as input and attempt to find matches of a target string

Target: day

Input: Today is Monday

Target State

d

^a^d

d a

^d

y

^y

12

Sample Regular Expressions

Example Description

cat Finds ‘cat’ pattern

[BCbc]at Finds ‘Bat’, ‘Cat’, ‘bat’, or ‘cat’ [matches one of the characters in the brackets followed by ‘at’]

x[A-Za-z0-9] Matches ‘x’ followed by a single letter/digit ([..] indicates one of the characters in the brackets)

x.y Matches ‘x’ followed by any single character followed by ‘y’

argh? Matches ‘arg’ or ‘argh’ (h can occur 0 or 1 time)

argh* Matches ‘arg’, ‘argh’, ‘arghh (h can occur 0 or more times)

argh+ Matches ‘argh’, ‘arghh’ (h can occur at least 1 or more times)

[a]\{2\} Find the letter ‘a’ repeated exactly twice

[^Cc]at Anything but ‘cat’ or ‘Cat’ (^ means not one of the indicated characters)

cat l dog Matches ‘cat’ or ‘dog’ [‘l’ indicates or]

13

More Text Utilities• grep pattern filename(s)

• egrep regex filename(s)

– Find occurrences of regular expression in filename

– Pattern can be any regular expression: "t[o]+" matches "to" or "too"

– https://www.digitalocean.com/community/tutorials/using-grep-regular-expressions-to-search-for-text-patterns-in-linux

• sed– Find and replace text from stdin or a file

– sed ‘s/find_pat/replace_pat/g’ input_file

• find_pat can be a regular expression

• replace_pat can use ‘&’ to use whatever pattern was found by the regexp and then add on to it

– sed ‘s/[Cc]at[^s]/&s/g’ input.txt

– Will make all occurrences of ‘Cat’ or ‘cat’ plural

14

Activity 4

• Use egrep to find how many times the word ‘the’ (or The) occurs in war_and_peace.txt

• Use sed to

– Find and replace ‘bat’ and/or ‘cat’ with ‘dog’ in re.txt and save to new.txt

– Replace ‘hi’ followed by 0 or more ‘s’ characters with that same pattern preceded by a ‘t’• hi becomes thi

• hiss becomes thiss

15

Environment variables

• Contain values that can be accessed by other programs that provide system and other info

– PATH

– LD_LIBRARY_PATH

• Set with export command

– export VARIABLE=VALUE

• Use with $VARIABLE in shell

• Use with $(VARIABLE) in Makefile

16

ANSWERS

17

Activity 3

• Sort the data by Test 1 scores from smallest to largest

– sort linux_grades_comma.txt –n –k2 –t,

• Extract sorted Test 2 scores and write them to a file “test2.txt”

– sort linux_grades_comma.txt -n -k3 -t, | cut -d, -f3 > test2.txt

• Write a command (using pipes) to count how many students got a 100 on Test 3

• cut linux_grades_comma.txt –d, -f4 | grep 100 | wc -l

18

Activity 4

• egrep [^A-Za-z0-9][Tt]he[^A-Za-z0-9] war_and_peace.txt

• sed ‘s/[bc]at/dog/g’ re.txt

• sed ‘s/his*/t&/g’ re.txt