04/24/23 Introduction to the Shell – Session 3 1

2.0.0.3 – Introduction to the Shell

2.0.0.3.3Introduction to the Shell – Session 3

· Job control· Start, stop and monitor running

processes· Monitoring disk usage

· Command line tools· grep, cut, sort, uniq, paste, fold,

tr, head, tail…· Command line Perl

· When command line tools don’t quite do it

04/24/23 Introduction to the Shell – Session 3 2

2.0.0.3 – Introduction to the Shell

Seeing Running Processes: top· See a constantly updated list of what’s running with

· Sorted by CPU use

· Also shows available memory and processors

· Processes can be controlled based on the PID (process ID)

top

Overall memory use

Overall CPU use

Process memory use

Process CPU use

I’m not running all that!Many processes are run by the OS (as user “root”) in the background. These processes generally have very low PIDs.

04/24/23 Introduction to the Shell – Session 3 3

2.0.0.3 – Introduction to the Shell

Seeing Running Processes: ps· See a complete list of what’s running with

· By default only shows what you are running (no root or other user’s processes)

· Can select which processes to show: eg.U by userp by PIDr only running processes

· See full list of everything running with ps auxw

ps

04/24/23 Introduction to the Shell – Session 3 4

2.0.0.3 – Introduction to the Shell

Seeing Running Jobs: jobs· See currently running jobs with

· These are processes that the current user is running in this terminal (does not show other user’s processes, or jobs started from other terminal windows)

· Useful for job control….

jobs

04/24/23 Introduction to the Shell – Session 3 5

2.0.0.3 – Introduction to the Shell

Putting Processes in the Background: &, bg and fg· What if you’re running a long job (eg. BLAST), but you want to use your prompt at the same time?

· Put the job in the background by starting it with <command> &

· Or, once the job is running, stop it with Cntl-Z (interrupts but does not kill), then use

· To put the job back in the foreground (and lose your prompt), use

· To specify which job to fg or bg, use the PID or %<job id>

fg

bg

Job ID

04/24/23 Introduction to the Shell – Session 3 6

2.0.0.3 – Introduction to the Shell

Ending Jobs: Cntl-C, kill· To end (without saving anything) a job currently running in the terminal, use Cntl-C

· To kill a job that is running in the background or from another terminal, use

· If it really won’t die, try kill -9 but only as a last resort!

Killing applicationsIf you are running applications (eg. word processors, web browsers) from the command line, do NOT kill them unless you cannot exit the application normally – this can corrupt files or cause application errors.

kill

04/24/23 Introduction to the Shell – Session 3 7

2.0.0.3 – Introduction to the Shell

Sharing the CPU: nice· Jobs are assigned priority (the lower the number, the higher the priority) which determines sharing of CPU

· Set the priority of your job lower when running intensive jobs so that others can use the CPU too with

· Or, for running jobs, with

nice

renice

Priority

Niced

04/24/23 Introduction to the Shell – Session 3 8

2.0.0.3 – Introduction to the Shell

Memory and Swap· Nice does not control memory use· When running memory-intensive jobs, avoid requiring swap (virtual memory – actually disk space) as this will slow all jobs on that computer

04/24/23 Introduction to the Shell – Session 3 9

2.0.0.3 – Introduction to the Shell

Checking Disk Location and Usage: df· To confirm that the disk you are on has enough space for you to create large files:

-h human-readable

df

04/24/23 Introduction to the Shell – Session 3 10

2.0.0.3 – Introduction to the Shell

Determining Disk Space Usage: du· To check total space taken by a directory structure, use

· Find out what is taking up all your disk space!

-h human-readable-c produce total

--max-depth only print totals for

this deep in directory tree

du

Saving spaceCut down on disk usage by using du and find to find large, old files, and use gzip to compress them.

04/24/23 Introduction to the Shell – Session 3 11

2.0.0.3 – Introduction to the Shell

Working with Text· Simple text processing is a large part of bioinformatics

· Converting between program formats· Data processing

· Command line tools are a fast and easy way to do a lot of text processing· Usually faster than Perl· Don’t require any coding!

04/24/23 Introduction to the Shell – Session 3 12

2.0.0.3 – Introduction to the Shell

Searching for Text: grep· Search for text in a file with

· Prints matching lines· Can do regular expression matching (slightly different from Perl)

· Useful options:-i case-insensitive-w text must appear as a “word” (surrounded by white space)

-v find non-matching lines-c count matching lines-B <#> print # lines before matching lines

-A <#> print # lines after matching lines

-f <file> find lines matching patterns listed in <file>

grep

04/24/23 Introduction to the Shell – Session 3 13

2.0.0.3 – Introduction to the Shell

Sorting Text: sort· Sort a file using

· Useful options:-f case-insensitive-g sort numerically (i.e. 11<100)-r reverse sort-k <#> sort on column #-u only print one copy of unique lines

sort

04/24/23 Introduction to the Shell – Session 3 14

2.0.0.3 – Introduction to the Shell

Working with Columns: cut and paste· Get particular columns from a file with

· Have to specify which field (column) to get· Useful options:-f <#> fields to output-d field delimiter

· Put columns back together with

cut

paste

04/24/23 Introduction to the Shell – Session 3 15

2.0.0.3 – Introduction to the Shell

Finding Unique or Repeated Lines: uniq· Determine unique lines with

· Collapses repeated lines

· Useful options:-i case insensitive-c count copies of each line-d print only duplicated lines-u print only unique lines-f <#> do not compare the first # fields

uniq

Pipe it!The UNIX text tools are particularly powerful in combination – use pipes to do a series of processing commands on your text.

04/24/23 Introduction to the Shell – Session 3 16

2.0.0.3 – Introduction to the Shell

Counting: wc· Count lines, words, and characters with

· Good for checking if files have correct number of records

· Useful in combination with other commands

wc

lines words

bytes (characters)

How many SAGE tags match more than one gene

04/24/23 Introduction to the Shell – Session 3 17

2.0.0.3 – Introduction to the Shell

Replacing Characters: tr· Replace characters with

· Like Perl’s tr///

· Useful options:-d delete-s replace repeats with single occurrence

-c use complement (replace everything EXCEPT what is specified)

tr

Input to trInstead of reading from a file, tr reads from standard input. If you want it to read from a file, redirect that file to standard input with “<“, or use cat and pipe the output to tr.

Format comma-separated list

Capitalize

04/24/23 Introduction to the Shell – Session 3 18

2.0.0.3 – Introduction to the Shell

Formatting Text: fold· To wrap long lines in a file, use

· Good for formatting before printing text files

· Useful options:-w width to wrap to-s break at spaces

fold

04/24/23 Introduction to the Shell – Session 3 19

2.0.0.3 – Introduction to the Shell

Getting Parts of Files: head and tail· Print the first 10 lines with

· Print the last 10 lines with

· Specify how many lines to get with head <#>eg. head -100 -> gets first 100 lines

· Good for making smaller “test” files

head

tail

Divide a file into three

Make a test file

04/24/23 Introduction to the Shell – Session 3 20

2.0.0.3 – Introduction to the Shell

Comparing files: diff· Find differences between lines in files with

· Can also compare directories

· Useful options:-b ignore differences in white space

-i ignore changes in case

diff

Confirm file copying

04/24/23 Introduction to the Shell – Session 3 21

2.0.0.3 – Introduction to the Shell

More advance processing: sed and awk· Simple programming languages· Designed for command-line use· Good for more advanced text processing

· But I prefer….

04/24/23 Introduction to the Shell – Session 3 22

2.0.0.3 – Introduction to the Shell

Command Line Perl· Perl can be run on a script, or directly on text input at the command line (a very short Perl script that does not have to be saved to disk)

· Input file with cat or <

· Options to use command line:-e find code on command line not in file

-n run command line script on each line of input

· Check out other Perl command line options with man perlrun

“What?” ‘Quotes?’When running Perl on the command line, use single quotes (‘) so that the shell does not try to interpret the Perl commands. Within these quotes, you can then use double quotes (“) for eg. print statements.

04/24/23 Introduction to the Shell – Session 3 23

2.0.0.3 – Introduction to the Shell

2.0.0.3.3Introduction to the Shell – Session 3

· Now you know….· How to control your processes· How to keep up on system usage· How to play with text on the

command line· How to run Perl from the

command line

04/24/23 Introduction to the Shell – Session 3 24

2.0.0.3 – Introduction to the Shell

2.0.0.3.3Further Readings

· Job control· Linux Cookbook Ch. 4.3· Learning the UNIX Operating System Ch. 6

· Command line goodies· Linux Cookbook Ch.13.1, 15.1-15.4

· Command line Perl· Linux Cookbook Ch. 13.5, 15.5