linux intermediate

Linux IntermediateLinux Intermediate

ITS Research Computing CenterC. D. Poon, Ph.D.

Email: [email protected]

its.unc.edu 2

Linux Command Category

Stdout/Stdin/Stderr, Pipe and Redirection, Wildcards

Linux Command Review

Break

Tips and Tricks

Conclusion

Exercise

OutlineOutline

its.unc.edu 3



its.unc.edu 4

Communication

ssh scp

File/Directory Management

cat cd chmod cp ln ls mkdir more less mv pwd dirs rm head tail wc

Comparisons

diff



its.unc.edu 5

Searching

grep find locate

Archiving

compress uncompress gzip gunzip zcat tar

Text Processing

cut paste sort sed awk

Linux Command Category Cont’d Linux Command

Category Cont’d

its.unc.edu 6

System Status

chgrp chown date df du env who w uptime

Miscellaneous

bc cal clear man

Linux Command Category Cont’d Linux Command

Category Cont’d

its.unc.edu 7

Stdout/Stdin/StderrPipe and Redirection

Wildcards

Stdout/Stdin/StderrPipe and Redirection

Wildcards

its.unc.edu 8

Output from commands • usually written to the screen

• referred to as standard output (stdout)

Input for commands• usually come from the keyboard (if no

arguments are given

• referred to as standard input (stdin)

Error messages from processes• usually written to the screen

• referred to as standard error (stderr)

stdout stdin stderr stdout stdin stderr

its.unc.edu 9

Pipe (|): stdout of one command to stdin of another command

Output Redirection (>): stdout of a command to a file

Output Appending (>>): stdout of a command appending to a file

Input Redirection (<): stdin of a command

Error Redirection (& in tcsh/csh, 2> in ksh/bash): stderr of a command

Pipe and Redirection Pipe and Redirection

its.unc.edu 10

Multiple filenames can be specified using special pattern-matching characters. The rules are:

• ‘?’ matches any single character in that position in the filename

• ‘*’ matches zero or more characters in the filename.

• ‘[…]’ Characters enclosed in square brackets match any name that has one of those characters in that position

Note that the UNIX shell performs these expansions before the command is executed.

Wildcards Wildcards

its.unc.edu 11



its.unc.edu 12

Log on to remote machine

Examplesssh [email protected]

ssh emerald.isis.unc.edu –l cdpoon

ssh topsail

ssh –X cedar.isis.unc.edu

ssh –Y tarheelgrid.unc.edu

ssh ssh

its.unc.edu 13

Using ssh, login to Emerald, hostname emerald.isis.unc.edu

To start ssh using SecureCRT in Windows, do the following.• Start -> Programs -> Remote Services ->

SecureCRT

• Click the Quick Connect icon at the top.

• Hostname: emerald.isis.unc.edu

• Login with your ONYEN and password

ssh using SecureCRTin Windows

ssh using SecureCRTin Windows

its.unc.edu 14

Copy files and directories to and from remote computers

Examplesscp file1 topsail.isis.unc.edu:/ifs1/home/cdpoon/.scp zircon.its.unc.edu:/home/cdpoon/file2 .

scp –r dir1 emerald.isis.unc.edu:/netscr/cdpoon/.scp –r topsail.isis.unc.edu:/ifs1/scr/cdpoon/dir2 dir3

scp emerald:/netscr/cdpoon/f topsail:/ifs1/scr/cdpoon/.

scp scp

its.unc.edu 15

Read one or more files and print output to stdout

Examplescat file1cat file1 file2 file3 > file_allcat file4 >> file_all Append file4 to file_all

cat > file5 Create file at stdin, end with EOF (^D normally, use “stty –a” to find

out)

cat > file6 << STOP Create file at stdin, end with STOPHere Document

cat cat

its.unc.edu 16

Change directory, built-in shell command

Examplescd /afs/isis/home/c/d/cdpooncd ../../ Change directory to 2 levels up

cd .. Change directory to 1 level up

cd ~ Change directory to Home

cd Change directory to Home

cd – Change to previous directory

cd cd

its.unc.edu 17

Change the access mode of one or more files

Exampleschmod u+x file1chmod go-w file2

chmod u=rwx, g=rx, o=x file3chmod 751 file3 Same as above, 7=rwx, 5=rx, 1=x

chmod =r file4chmod 444 file4 Same as above, 4=r, 2=w, 1=x

chmod chmod

its.unc.edu 18

Copy a file/dir to another file/dir Examples

cp file1 file2 Copy to the same directory and change filename

cp file1 ../dir/file2 Copy to different directory and change filename

cp file1 ../dir/. Keep the same filename

cp –r dir1 dir2 Copy directory recursively

cp –r dir1 new_dir/dir2 Copy directory recursively to another directory

cp –p file3 file4 Preserve the modification time and permission modes

cp cp

its.unc.edu 19

Create links for file/dir and allow them to be accessed by different names

Examplesln file1 file2 Hard link for file

ln dir1 dir2 Hard link not allowed for directory

ln –s dir1 dir2 Symbolic/Soft link for directory, dir2 -> dir1

ln –s file3 file4 Symbolic/Soft link, file4 -> file3

ln –s dir/file5 file6 Symbolic/Soft link, file6 -> dir/file5

ln ln

its.unc.edu 20

List all files and directories in the current directory

Examples

ls

ls –a List files/directories starting with “.” too

ls –l Long listing

ls –lh List file sizes in human readable format

ls –F Flag filenames by appending / to directories, * to executables files,

and @ to symbolic links

ls ls

its.unc.edu 21

Create one of more directories

Examples

mkdir dir1

mkdir –p dir1/dir2/dir3

Create intervening parent directories if they don’t exist

Same as mkdir dir1; cd dir1; mkdir dir2; cd dir2; mkdir dir3; cd ../../

mkdir mkdir

its.unc.edu 22

Display files on a terminal, one screen at a time

Examples

more file1 Hit space bar for another page, q to quit

more –d file2 Display the prompt “Press space to continue, ‘q’ to quit

more –c file3 Page through the file by clearing each window instead of

scrolling

more more

its.unc.edu 23

Works like “more” but allows backward and forward movement

Examples

less file1 Hit space bar for another page, q to quit

Hit b to scroll backward one window

Hit /pattern to highlight “pattern” in the text

Hit Return to scroll one line at a time

less less

its.unc.edu 24

Move files and directories within the same machine and/or rename them

Examples

mv file1 dir1/file1 Move file1 to dir1, Same as mv file1 dir1/

mv file3 file4 Rename file3 to file4

mv dir1 dir2 Rename directory dir1 to dir2

mv dir3 dir4/dir5/dir6 Rename directory dir3 to dir6 and move to

dir4/dir5 directory

mv mv

its.unc.edu 25

Print the full pathname of the current directory

Examplespwd

dirs C shell built-in command, works like “pwd”

dirs –lPrint working directory in long listing

pwd dirs pwd dirs

its.unc.edu 26

Delete one or more files and directories

Delete empty directory with “rmdir”

Examplesrm file1

rm file* Remove all files with filename starting as “file”

rm –I file* Prompt for y (remove the file) or n (do not remove the file)

rm –r dir1 Delete directory “dir1” and its content

rm rm

its.unc.edu 27

Print first/last few lines of one or more files

Exampleshead file1 Print the first 10 lines of file “file1”

head –n100 file2 Print the first 100 lines of file “file2”

tail file* Print the last 10 lines of files with filename

starting as “file”

tail –f file3 Print the last 10 lines of file “file3” and follow file

as it grows

head tail head tail

its.unc.edu 28

Print a character, word, and line count for files

Examples

wc –c file1 Print character count for file “file1”

wc –l file2 Print line count for file “file2”

wc –w file3 Print word count for file “file3”

wc wc

its.unc.edu 29

Report lines that differ between file1 and file2

with file1 text flagged by < and file2 by >

Examples

diff file1 file2 Show difference between file1 and file2

diff diff

its.unc.edu 30

Search for lines that match a regular expression

Examples

grep abc file1 Print line(s) in file “file1” with “abc”

grep –i abc file2 Print line(s) in file “file2” with “abc” ignoring

uppercase and lowercase distinctions

grep grep

its.unc.edu 31

Find particular groups of files

Examples

find . –name temp Find file named “temp” in current

directory

find /etc –name ‘rc*’ Find file(s) in /etc directory with name

starting with “rc”

find /usr/share/man –type d –name ‘man*’

Find directories in /usr/share/man with name starting with “man”

find find

its.unc.edu 32

Find files with matching pattern in database prepared by updatedb

Database needed to be updated Examples

locate which Find files named with pattern “which” in the OS

locate –c which Count number of files named with pattern “which” in

the OS

locate –i which Find files named with pattern “which” in the OS ignoring case distinctions

locate locate

its.unc.edu 33

Reduce or expand the size of one or more files using adaptive Lempel-Ziv coding

Use uncompress to expand data Examples

compress file1 Reduce the size of file1 and create new file named file1.Z

compress –f file2 Force to reduce the size of file2 and create new file named file2.Z

uncompress file3.Z Expand file3.Z and restore file3

compress uncompress

compress uncompress

its.unc.edu 34

Reduce or expand the size of one or more files using Lempel-Ziv coding (LZ77)

Use gunzip to expand data Examples

gzip file1 Reduce the size of file1 and create new file named file1.gz

gzip –f file2 Force to reduce the size of file2 and create new file named file2.gz

gunzip file3.gz Expand file3.gz and restore file3

gzip gunzip gzip gunzip

its.unc.edu 35

Expand the size of one or more files created by compress or gunzip

List file contents to stdout without deleting the .Z or .gz file

Examples

zcat file1.Z Expand file1.Z and list the content of file1 in stdout

zcat file2.gz Expand file2.gz and list the content of file2 in stdout

zcat zcat

its.unc.edu 36

Archive files and directories Create a single file with extension .tar Examples

tar –cvf file123.tar file1 file2 file3Create archive file named file123.tar in

verbose mode with contents, file1, file2, and file3

tar –xvf file123.tarExpand file123.tar in verbose mode and

generate the original files and directories back

tar tar

its.unc.edu 37

Remove sections from each line of files

Examples

cut –d: -f1,5 /etc/passwd Use field delimiter “:” to

locate fields 1 and 5 from file /etc/passwd to extract usernames and real names

cut –k4 file1 Take column 4 out from each line of file1 and display in stdout

cut cut

its.unc.edu 38

Write to stdout consisting of sequentially corresponding lines of each given file

Examples$ cat file1 $paste file1 file2

1 1 a

2 2 b

$ cat file2 c

a

b

c

paste paste

its.unc.edu 39

Sort lines of text files Examples

sort –fd file1

Alphabetize lines (-d) in file1 and ignore lower and upper cases (-f)

sort –t: -k3 -n /etc/passwd

Take column 3 of file /etc/passwd separated by “:” and sort in arithmetic order

sort sort

its.unc.edu 40

Edit one or more files without user interaction using stream editor

Examplessed s/xx/yy/g file1

Substitude all occurrences of “xx” in file1 with “yy” and display on stdout

sed /abc/d file3 Delete all lines containing “abc” in file3

sed /efg/!d file4 Delete all lines not containing “efg” in file4

sed /BEGIN/,/END/s/XYZ/xyz/g file5

Substitute “XYZ” on lines between BEGIN and END with “xyz” in file5

sed sed

its.unc.edu 41

Process files by pattern-matching Examples

awk –F: ‘{print $1}’ /etc/passwdExtract the 1st field separated by “:” in /etc/passwd and print to

stdout

awk ‘/abcde/’ file1Print all lines containing “abcde” in file1

awk ‘/xyz/{++i}; END{print i}’ file2Find pattern “xyz” in file2 and count the number

awk ‘length <= 1’ file3

Display lines in file3 with only 1 or no character

awk awk

its.unc.edu 42

Change the group ownership of one or more files or directories

Exampleschgrp employee file1

Change group ownership to “employee” for file “file1”

chgrp –R student dir1Change group ownership to “student” for directory “dir1” including

subdirectories recursively

chgrp chgrp

its.unc.edu 43

Change the ownership of one or more files or directories

Exampleschown employee file1

Change ownership to “employee” for file “file1”

chown –R student dir1Change ownership to “student” for directory “dir1” including

subdirectories recursively

chown chown

its.unc.edu 44

Print the current date and time in certain format

Set the current date and time Examples

datePrint the current date and time

date +%DPrint the current date and time in mm/dd/yy format

date 1201160108Set the current date and time to Dec 01 4:01pm 2008

date –d friShow the date of the coming Friday

date date

its.unc.edu 45

Report the number of used and free disk block on all mounted file systems

Examplesdf

Print used and free disk block on all mounted file system

df -kPrint used and free disk block in kilobyte

df df

its.unc.edu 46

Print disk usage of directories and its subdirectories

Examplesdu dir1

Print disk usage in kilobyte of directory “dir1”

du –-block-size=1M dir2Print disk usage in megabyte of directory “dir2”

du du

its.unc.edu 47

Display the current environment variables or set new values

Examplesenv

Display all of the current environment variables

env env

its.unc.edu 48

Display information about the current status of the system

Exampleswho

Display the names of users currently logged in to the system

who –bReport information about the last reboot

who am IPrint the username of the invoking user

who who

its.unc.edu 49

Print summaries of system usage, currently logged-in users, and what they are doing

Examplesw

Print summaries of system usage, currently logged-in users

w –sDisplay in short form

w w

its.unc.edu 50

Print the current time, amount of time logged in, and the system load averages

Examplesuptime

Print a one line display of the current time, how long the system has been running, how many users are currently logged on, and the system load averages for the past 1, 5, 15 minutes

uptime uptime

its.unc.edu 51

Interactively perform arbitrary-precision arithmetic or convert numbers from one base to another, type “quit” to exit

Examplesbc Invoke bc

1+2 Evaluate an addition

5*6/7 Evaluate a multiplication and division

ibase=8 Change to octal input

20 Evaluate this octal number

16 Output decimal value

ibase=10 Change back to decimal input

bc bc

its.unc.edu 52

Print calendar of a month or all months in a year

Examplescal Print calendar of the current month

cal 12 2008Print calendar of December 2008

cal 2008 Print calendar of all months in 2008

cal -3 Display previous/current/next months

cal cal

its.unc.edu 53

Clear the terminal display and have the prompt locate at the top of the terminal window

Examples

clear Clean up the current terminal display

clear clear

its.unc.edu 54

Display information from the online reference manuals

Examples

man man Display the manual for the command “man”

man –k link compile Display commands related to linking and compiling using a keyword search

man man

its.unc.edu 55

BreakBreak

its.unc.edu 56

Tips and TricksTips and Tricks

its.unc.edu 57

Show files changed on a certain date in all directories

Examplels –l * | grep ‘Sep 26’

Show long listing of file(s) modified on Sep 26

ls –lt * | grep ‘Dec 18’ | awk ‘{print $9}’

Show only the filename(s) of file(s) modifed on Dec 18

Tips and Tricks #1 Tips and Tricks #1

its.unc.edu 58

Sort files and directories from smallest to biggest or the other way around

Exampledu –k –s * | sort –n

Sort files and directories from smallest to biggest

du –ks * | sort –nr

Sort files and directories from biggest to smallest


its.unc.edu 59

Change timestamp of a file

Exampletouch file1

If file “file1” does not exist, create it, if it does, change the

timestamp of it

touch –t 200801011200 file2

Change the time stamp of file “file2” to 1/1/2008 12:00


its.unc.edu 60

Find out what is using memory

Exampleps –ely | awk ‘{print $8,$13}’ | sort –k1 –nr | more


its.unc.edu 61

Remove the content of a file without eliminating it

Examplecat /dev/null > file1


its.unc.edu 62

Backup selective files in a directory

Examplels –a > backup.filelist

Create a file list

vi backup.filelist

Adjust file “backup.filelist” to leave only filenames of the files to be backup

tar –cvf archive.tar `cat backup.filelist`

Create tar archive “archive.tar”, use backtics in the “cat” command


its.unc.edu 63

Get screen shots

Examplexwd –out screen_shot.wd

Invoke X utility “xwd”, click on a window to save the image as “screen_shot.wd”

display screen_shot.wd

Use ImageMagick command “display” to view the image “screen_shot.wd”

Right click on the mouse to bring up menu, select “Save” to save the image to

other formats, such as jpg.


its.unc.edu 64

Sleep for certain time, then pop up a message

Example(sleep 60; xmessage –near One Minute Has Gone By) &


its.unc.edu 65

Count number of lines in a file

Examplecat /etc/passwd > temp; cat temp | wc –l; rm temp

wc –l /etc/passwd


its.unc.edu 67

Find name and version of Linux distribution, obtain kernel level

Exampleuname -a

head –n1 /etc/issue


its.unc.edu 68

Show system last reboot

Examplelast reboot | head –n1


its.unc.edu 69

Combine multiple text files into a single file

Examplecat file1 file2 file3 > file123

cat file1 file2 file3 >> old_file


its.unc.edu 70

Create man page in pdf format

Exampleman –t man | ps2pdf - > man.pdf

acroread man.pdf


its.unc.edu 71

Remove empty line(s) from a text file

Exampleawk ‘NF>0’ < file.txt

Print out the line(s) if the number of fields (NF) in a line in file

“file.txt” is greater than zero

awk ‘NF>0’ < file.txt > new_file.txt

Write out the line(s) to file “new_file.txt if the number of fields (NF)

in a line in file “file.txt” is greater than zero


its.unc.edu 72

ConclusionConclusion

its.unc.edu 73

Many ways to do a certain thing

Unlimited possibilities to combine commands with |, >, <, and >>

Even more powerful to put commands in shell script

Slightly different commands in different Linux distributions

Emphasized in System V, different in BSD

Conclusion Conclusion

its.unc.edu 74

ExerciseExercise

its.unc.edu 75

cd Change to home directory

cat /etc/passwd > temp Save the file /etc/passwd into a new file called “temp”

more temp | wc –l Find out the number of lines in file “temp”

awk ‘/home/’ temp Show line(s) with the word “home”

awk ‘/nologin/{++i}; END{print i}’ temp

Count number of word “nologin” in file “temp”

sed s/no/yes/g temp > temp1 Replace all “no” with “yes” in “temp” and save in “temp1”

awk ‘/nologin/{++i}; END{print i}’ temp1

Count number of work “nologin” in file “temp1”

awk ‘/yeslogin/{++i}; END{print i}’ temp1

Count number of work “yeslogin” in file “temp1”

grep –i daemon temp Display line(s) with word “daemon” regardless of case

more temp | grep –i daemon Display line(s) with word “daemon” regardless of case

sort –t: -k3 –n temp Sort numerically over the 3rd column in file “temp” separated by “:”

rm –i temp* Delete the file(s) with filename starting with “temp” interactively

Exercise Exercise

linux intermediate

Documents

allcat file4 file

commandoutput redirection

fileinput redirection

copy files

executables files

standard output stdoutinput

allappend file4

print output