sphinx/rest memo
TRANSCRIPT
Sphinx/Rest Memo
Marc Zonzon
May 02, 2019
Contents
1 Introduction 3
2 Getting started 7
3 Files and Names 13
4 Getting Help 17
5 Files and Directories 21
6 Wildcards 25
7 Piping and redirecting 29
8 Finding files 33
9 Editors 37
10 Bash Scripts 41
11 Managing Processes 47
12 Working Remotely 53
Python Module Index 57
i
ii
Sphinx/Rest Memo
This tutorial should give you an overview of how to use linux from the command line. Its aim is to be brief enoughthat you can complete it quickly, but comprehensive enough that you can get on and work on the command line.
Each chapter has specific aims stated at the beginning, with a wrap up at the end list the fundamentals. Each sectionalso has excises that the reader can complete - these should test that you understand what you have learned, nottake up your time.
These notes will introduce you to using the command line interface in a Linux/Unix environment, and provide youwith the skills needed to login to a remote machine, navigate around the file system, create files and directories,write simple scripts, and get help from the system. This will be done across a series of chapter, each containing anoverview of the chapter, some key points to take away, and a set of exercise for you to check your understandingof the subject matter.
Contents:
Contents 1
Sphinx/Rest Memo
2 Contents
CHAPTER 1
Introduction
This tutorial is about using the command line in linux. The aim is to ensure you have the require skills to succes-fully navigate through linux, get to your files and begin automating your research.
At first the command line looks archaic, something from a by-gone age of computing. However it allows re-searchers to work faster, to automate their analyses, and more it is a more powerful and flexible way of using thetools we have. To borrow a quote: “This is the tool of a Researcher. Not as clumsy or as random as a GUI. Anelegant tool. . . for a more civilized age.”
With GUI (Graphical User Interface) the power of the underlying program is limitted by what has been put intothe GUI, and the designers idea of how to chain commands together. Accessing the command line directly allowsus to chain commands in the way that allows us to complete the tasks at hand.
Reproducability is another requirement of research; we can take notes as we click on buttons, but what if weaccidentally click on somethign else, or type the wrong input? Do we know for sure how we got here? With thecommand line we can review what we have done and be sure that we are running the same things each time.
To access this power and control, we need to start at the beginning. Learn how to navigate through the filesystemusing the command line, how to invoke the commands we need, find help to use them, learn how to chain themtogether, and how to script them so we can automate and repeat.
This chapter introduces the command line, the shell and the terminal. The following chapters will build on whatwe learn here.
1.1 Overview
In this chapter you will learn
• How to open a terminal.
• What the shell is.
• How to use the shell.
1.2 Opening a terminal
The first thing we will need to do is open a terminal. If you are using Linux or OSX, this should be simple. Ifyou are using windows, you will need to access a remote linux machine using a program called PuTTY. All three
3
Sphinx/Rest Memo
methods are described in their relevant subsections:
1.2.1 Linux
This varies from system to system, but a good bet is to try Applications -> System or Activities -> Show Appli-cations, and type terminal into the search box.
1.2.2 Mac OS
You should find the terminal under Applications -> Utilities.
1.2.3 Windows
On windows you will need two programs, PuTTY and Xming. You can find instructions on how to set these uphere
1.3 The Shell
The shell is a program that sits between the user (and keyboard) and the operating system. Users type commandsinto the shell and run them by pressing the return key. The shell takes text input from the keyboard and outputsinformation as text to the screen. Using this we can interact with the computer and run any programs we wish to.
Sometimes the word terminal is used interchangeably with shell. Terminals were originally hardware that wasused to enter and display information from computers, these days when people talk about terminals they usuallyare referring to terminal emulators that run on GUIs and give you access to the shell. Each terminal can run manydifferent kinds of shells, most these days run a single shell called BASH (Bourne Again SHell). This tutorial willassume that BASH is the shell being used.
The shell allows us to run any of the programs on the computer by typing there name. Many of these programs havestrange short names such as command{ls}, which runs the listings program that lists the contents of a directory.We shall learn more about these commands in the next chapter, but first a little look at the command line.
The shell takes input from the keyboard. You can type in commands, and they are not ran until you press the returnkey. You can use the left and right cursor keys to navigate the cursor forwards and backward along the line, deletetext or insert new text.
The shell contains a history of all the commands you have run in it. You can navigate this using the up and downcursor keys.
Tip: To search back through the history, press Ctrl + r, and then type part of the previouscommand. Pressing Ctrl + r again cycles through options.
1.4 Summary
1.4.1 Concepts
The shell is how you interact with the computer and give it commands to run. You can edit anything you type onthe command line using the left and right cursor keys. Using the up and down cursor keys allows you to accessthe history of previous commands.
4 Chapter 1. Introduction
Sphinx/Rest Memo
1.4.2 Commands
• Left and right arrow keys - navigate along text on the command line.
• Up and down arrow keys - browse through previously entered commands.
• Ctrl + r - Search back through previous commands.
1.5 Further Reading
• Linux in a nutshell, O’Reilly - Chapter 3, The UNIX Shell.
• Linux in a nutshell, O’Reilly - Chapter 4, bash.
• Linux Documentation Project - Chapter 2, Section 2, Absolute Basics
• Introduction to the command line
1.5. Further Reading 5
Sphinx/Rest Memo
6 Chapter 1. Introduction
CHAPTER 2
Getting started
In the previous chapters we looked at how we interact with the computer using the shell, which allows us to passcommands to the computer which are then run. We also looked at what files are in linux, and that directoriescontain files. These directories are stored in a common hierarchy. If we want to find our files, or run programs,then we will need to have an idea of what that hierarchy is, how to navigate around it, and what paths are. Sincewe want to run programs, we shall start looking at how to pass commands to the shell, and how to enable optionalextras in those commands.
2.1 Overview
In this chapter you will learn:
• how to find out where you are in the directory structure.
• how to view what files and directories are in our current directory.
• the difference between relative and absolute paths.
• how to move around the filesystem.
• how linux directory structure is layed how
• the differences between command, flag and arguments.
2.2 Where are we?
The first thing we will want to know is where we are in the system. We can find this out by using the “printworking directory” command, pwd (Like many commands in linux, its name is an abbreviation of what it does.The command lists the name of the directory that we are currently in; let’s give it a try:
1 $ pwd2 /home/rel
7
Sphinx/Rest Memo
2.3 What is in our current location?
Now that we know where we are, we will want to know what files we have in our directory. Again, there is acommand to do this, ls (it lists the directories contents). Lets have a look.
1 $ ls2 Documents Private Public shopping_list.txt
Whilst pwd was enter as a command by itself, ls is more powerful and has more functions. As such, it hasoptional arguments and options (called flags) that can be passed to it to achieve more.
ls [options] [location]
In this example square brackets, [], represent something that is optional. We can pass a location to ls and itwill list the contents of that location. Additionally we can pass options (in the form of flags) that will modify thebehaviour or output. There are some examples below.
1 $ ls2 Documents Public Private shopping_list.txt3 $ ls -l4 drwxr-xr-x. 14 longr longr 4096 Jun 3 11:16 Documents5 drwxr-x---. 10 longr longr 4096 Jul 13 09:54 Private6 drwxr-xr-x. 9 longr longr 4096 Jul 13 09:52 Public7 drwxrwxr-x. 3 longr longr 4096 Aug 12 14:03 test8 $ ls /usr9 bin games include lib lib64 libexec local sbin share src tmp
10 $ ls -l /usr11 dr-xr-xr-x. 2 root root 65536 Aug 22 15:08 bin12 drwxr-xr-x. 2 root root 4096 Feb 3 2016 games13 drwxr-xr-x. 39 root root 4096 Aug 12 09:46 include14 dr-xr-xr-x. 46 root root 4096 Aug 12 09:46 lib15 ...
Lets see what’s going on here:
• Line 1 - We run the ls command without any arguments, and it lists the contents of our current directoryon line 2.
• Line 3 - We run the ls command with the flag (-l) which changes the output to a long listing. This displaysextra information on the files. It has the following format: - d at the beginning indicates that it is a directory,- indicates it is a file. - The next 9 characters represent the permissions on the file. We shall read moreabout this in section ooooooo. - The next field is the number of links to the file or directory (do not worryabout this). - The third field is the owner of the file (in this case rel). - The forth field is the group the file ordirectory belongs to (in this case atlas). - The fifth field is the size. - Fields six to eight are the modificationtime. - The final field is the name of the file or directory.
• Line 8 - We run ls with the optional location /usr. This tells ls not to list the contents of our currentdirectory, but that of the location we have specified.
• Line 10 - We run ls again, but this time with the (-l) flag and an argument (/usr); this tells ls to printouta long listing of the contents of /usr.
• Line 15 - just indicates that we have cut off the output and not printed the full amount.
2.4 Paths
In the previous section we passed an argument to ls of /usr. This was a location of the filesystem. In linuxwhenever we pass a file name or a location to a command, we are passing it a path. A path describes how to get toa file or directory.
8 Chapter 2. Getting started
Sphinx/Rest Memo
2.4.1 Relative vs Absolute paths
There are two types of paths in linux, relative and absolute. We always use one of these when refereeing to a file.
In linux the file system has a defined hierarchy, the top (or root) of this hierarchy is called the root directory, andis represented by a single forward slash, /. This contains subdirectories, which may contains subdirectories andso on. Every file and directory on linux is located somewhere is here.
Absolute paths are paths that begin with a / and specify a file/directories position with relation to the rootdirectory.
Relative paths specify a files location relative to the current directory, and as such never begin with a /.
Lets look at a an example of both types:
1 $ pwd2 /home/rel3 $ ls Documents4 report.txt paper.txt presentation.pdf5 ...6 $ ls /home/rel/Documents7 report.txt paper.txt presentation.pdf8 ...
• line 1 - We use pwd to get our current location.
• line 3 - We run the ls command and give it the name of a directory in our current locations. This resultcould give a different output depending on where we are in the system. The directory may not exist, or wemay be in another directory that also contains a directory called Documents.
• line 6 - We run ls again, but this time we use an absolute instead of relive path. As this is an absolute pathit will provide the same output regardless of where we are on the system.
2.4.2 More paths
There are several short cuts that you mind find helpful with paths:
• ~ (tilde) - This is a short cut for your homer directory. In the examples above our home directory is /home/rel. We could therefore refer to the directory Documents with either /home/rel/Documentsor ~/Documents.
• . (dot) - This refers to the current directory. In the example above, when using a relative path we could useeither Documents or ./Documents.
• .. (double dot, or dot dot) - This refers to the parent directory. This can be used multiple times to keepgoing up the file hierarchy. The command ls ../../ would print out the contents of the root directory(assuming we are in /home/rel).
As you can now see, there are many ways to refer to directories and files. We can therefore use the one that ismost convenient or practical. Lets see a few examples:
1 $ pwd2 /home/rel3 $ ls4 Documents Private Public shopping_list.txt5 $ ls ~6 Documents Private Public shopping_list.txt7 $ ls Documents8 report.txt paper.txt presentation.pdf9 $ ls ./Documents
10 report.txt paper.txt presentation.pdf11 $ ls ..12 rel
(continues on next page)
2.4. Paths 9
Sphinx/Rest Memo
(continued from previous page)
13 $ ls /home14 rel15 $ ls ../..16 bin boot dev etc home lib var17 $ ls /18 bin boot dev etc home lib var
2.5 Moving around
To move around the system, we use a command to change directory, cd It takes the following format:
cd [location]
Given that the job of cd is to change to a new directory, it might seem strange that the location is optional. If cdis ran without an argument, it takes you back to your home directory.
The location we pass to cd is a path, and as such may be relative or absolute. It can also use any of the paths listedabove.
Tip: The change directory command also has another shortcut that we can use, cd -. This will take you to thedirectory you were in before you changed directory.
1 $ pwd2 /home/rel3 $ cd Public4 $ pwd5 /home/rel/Public6 $ cd /7 $ pwd8 /9 $ cd ~/Public
10 $ pwd11 /home/rel/Public12 $ cd ../../..13 $ pwd14 /15 $ cd16 $ pwd17 /home/rel18 $ cd -19 $ pwd20 /
Tip: Tab completion Typing out long paths can be tiring and time consuming. Sometimes you also forget whatthe name of a directory is in your paths, stopping to do an ls is annoying. This is where tab completion comes inuseful.
When you start typing a path in a terminal, pressing the tab key will result in the name of the file or directorybeing automatically completed for you. If nothing happens, this means there is no unique solution, and pressing ita second time will result in a list of possible completions.
10 Chapter 2. Getting started
Sphinx/Rest Memo
2.6 Summary
2.6.1 Concepts
• All files in linux are inside the root directory /
• Paths consist of one of more directories.
• Absolute paths begin a the root directory /.
• Relative paths do not begin with /.
• ., .., and ~; are alias’ that refer to the current directory, the directory above, and the home directory.
2.6.2 Commands
• pwd - prints the current working directory.
• ls - lists files and directories.
• cd - move between directories.
2.7 Exercise
• Use ls and cd to look and move around the system. Don’t forget to use relative and absolute paths.
• How many way are there to get to your home area? try all of them.
• Practise using tab completion
2.8 Further Reading
Introduction to Linux - Chapter 3. About files and the file system
2.6. Summary 11
Sphinx/Rest Memo
12 Chapter 2. Getting started
CHAPTER 3
Files and Names
Before we dive deeper into linux and how to use it, we need to pause to look at files and theirs names a littlelonger. This chapter will introduce you to a brief view of the linux file system and how files are named. I wouldadvise that you do not skip this section, as there are many things may trip you up later on if you have not graspedthe basics of how the files systems and files work.
3.1 Overview
In this chapter you will learn:
• what a file is in linux
• how to find out what type a file is.
• how to deal with special characters in file names.
• what constitutes a good file name.
3.2 Everything is a file
It is said of linux that “. . . everything is a file; if it is not a file, it is a process.”
This describes linux quite well, there are some special files, but these are still just files. Directories are just filesthat contain names of files, programs are files that the system reads and then runs, keyboards are files that thesystem reads from, screens are files, and so on.
3.2.1 Extensionless
You are probably used to seeing files with file extensions, the 2-4 characters after a full sop in the files name.Some operating systems such as windows require this extension to know what the file is. Linux is an extensionlesssystem, it does not need the file extension, and instead looks inside the file to see what type of file it is.
Using the path command we can find out what type of file something is.
file [path]
13
Sphinx/Rest Memo
Whilst Linux does not the extension, we humans prefer it. Sometimes it is useful to know what type of filesomething is. If we are compiling code, it is nice to have the source code and the executable with the same name,in which case an extension helps identify them. Some common ones are:
• file.exe - an executable or program.
• file.txt - plain text file.
• file.png, file.jpg, file.svg - an image file.
• file.cxx, file.hxx - c++ source and header files.
• file.tex - LaTeX file.
• file.py - python script.
• file.pdf - A PDF file. (Sometimes known as a Acrobat file)
3.3 Case sensitive
Another common source of problems on linux is file names. Linux is case sensitive, as such you can have twofiles with the same name, but one starts with a capital letter.
Warning: Commands and command line options are also case sensitive. ls is the correct command, LS isnot. Also many commands have flags with different cases -s and -S do very different things.
3.4 Spaces and Quotes
Having a space in a file name is allowed, but we have to be careful. On the command line we use spaces to separatecommands and arguments. If we have a space in a filename, and don’t do anything the command line will assumeit separates arguments.
1 $ ls2 Public Private My Files3 $ cd My Files4 bash: cd: My: No such file or directory
Here bash assumed the space separtated two arguments, My and Files. As such cd ignored the second argument(it only takes one) and tried to change into the directory My which does not exist. There are two ways to solve thislist below.
3.4.1 Escape Character
One option is to use an escape character; these are characters that are used before a special character to cancel outits special meaning. In bash the escape character is the backslash (\).
1 $ cd My\ Files2 $ pwd3 /home/user/My Files
We can use the escape character on any character that has special meanings in bash such as: $, ", ', and even \.
If you use tab completion, and the file is unique, then bash will auto-complete and include the escape character.
14 Chapter 3. Files and Names
Sphinx/Rest Memo
3.4.2 Quotes
The second option is to use quotes. This is useful if there are lots of special characters or spaces in the file nameand we do not want to escape them all individually.
1 $ cd "My Files"2 $ pwd3 /home/user/My Files
Important: Linux has two types of quotes, " (Double quotes) and ' (single quotes). These behave differently.
• Single quotes remove all special characters meanings, anything inside them is interpreted as regular text.
• Double quotes remove the special meaning of all characters expect $ and `. These will be expanded by theshall first (see Bash scripts and Variables).
3.5 Hidden files and directories
Linux has a simple way of hiding files and directories. Any file or directory whose name starts with a full stop (.)is hidden from normal view. To view these on the command line we need to pass the -a flag to ls.
1 $ ls2 Public Private3 $ ls -a Public4 . .. Public Private .bashrc bash_history
Passing the flag to ls results in the hidden files being displayed. In your home directory you will find manyhidden files such as .bashrc which hold configuration information for the programs you use. You will also seetwo more files, . and .., these are discussed in the section on Paths in Getting Started.
3.6 Summary
3.6.1 Concepts
• Everything is a file in linux, even the directories.
• Linux is extensionless, files do not need extensions for the system to know what they are.
• Linux is case sensitive.
• Spaces have a special meaning, so we need to escape or quote them in the command line.
3.6.2 Commands
file - tells you what type a file is. ls -a - displays hidden files.
3.7 Further Reading
Linux Documentation Project - Chapter 4, About files and the file system
3.5. Hidden files and directories 15
Sphinx/Rest Memo
16 Chapter 3. Files and Names
CHAPTER 4
Getting Help
There are many different programs on linux, all of which have many different options. Remembering all thesecommands is not easy. Fortunately there are manuals or help commands for nearly all the programs on linux.
4.1 Overview
In this chapter you will learn:
• how to use the manual pages and search them.
• how to search the manual pages.
• how to use the help flags.
4.2 Man Pages
The man pages are a series of pages or manuals that detail almost every program on the system and what it does(some programs are very extensive and have multiple man pages).
4.2.1 Viewing the man pages
You access the man pages by typing man, and then the command you want help with, for example:
$ man <command_you_want_to_use>
This will produce an out put similar to this:
LS(1) User Commands LS(1)
NAMEls - list directory contents
SYNOPSISls [OPTION]... [FILE]...
(continues on next page)
17
Sphinx/Rest Memo
(continued from previous page)
DESCRIPTIONList information about the FILEs (the current directory by default).Sort entries alphabetically if none of -cftuvSUX nor --sort is speci-fied.
Mandatory arguments to long options are mandatory for short optionstoo.
-a, --alldo not ignore entries starting with .
-A, --almost-alldo not list implied . and ..
4.2.2 Navigating the man pages
The man command displays the manual pages using the less viewer, this means we can use the same navigationcommands to look through the man pages.
To navigate up and down we can either uses the arrow keys to go up and down by one line at a time, or use thepage up and page down keys to do whole pages at a time.
More often than not you will want to search for a key phrase or word, this can be done by typing / followed by thekey word or phrase and pressing enter. Pressing n will move to the match of the pattern, and pressing shift+ n will go to the previous match.
4.2.3 Searching the man pages
Sometimes you will not know what the command is that you need, in this case you can search through the manpages. All man pages have a short description, and we can search for keywords or patterns in this. To search forman pages, we type:
$ man -k compressbunzip2 (1) - a block-sorting file compressor, v1.0.6bzcat (1) - decompresses files to stdoutbzcmp (1) - compare bzip2 compressed filesbzdiff (1) - compare bzip2 compressed filesbzegrep (1) - search possibly bzip2 compressed files for a regular e...
As you can see this prints out two columns: the first is the name of the man page, and the second is the shortdescription.
4.3 More help, and flags
Some commands lack a help file, or the man pages are not installed. In this case we can just type the commandwith a -h flag and press enter. This will display limited help about the command such as what arguments ittakes, and what flags it has.
There are two types of flags: short and long. Short flags are one character long and start with a -. These can bechained together like so:
$ ls -a -ddrwx------. 52 user user 12288 Aug 11 15:14 .drwxr-xr-x. 4 root root 4096 Feb 3 2016 ..$ ls -addrwx------. 52 user user 12288 Aug 11 15:14 .drwxr-xr-x. 4 root root 4096 Feb 3 2016 ..
18 Chapter 4. Getting Help
Sphinx/Rest Memo
Long flags start with two hyphens (--) and are whole words. Sometimes a command has a -h flag, and sometimesit has a --help flag, or even both (a good example if ls which uses -h for human readable units, so it has--help instead.
4.4 Summary
4.4.1 Concepts
• Man pages contain information on many different commands
• Some commands also have a help flag.
• Short flags are one letter long, start with a single hyphen (-), and can be merged together.
• Long commands are words, start with two hyphens (--), and cannot be merged.
4.4.2 Commands
• man <command> - to display the man page for a command.
• man -k <pattern> - to search the short description for a pattern.
• / <pattern> to search for a pattern whilst in a man page. - n - to move to the next pattern whilstsearching - shift + n - to move to the previous pattern whilst searching
4.5 Exercises
Using the man pages, find out how to modify ls to:
• Print file sizes.
• Print file sizes in a human readable format.
• Sort the files by size, largets first (you might want to have the size displayed so you can confirm this)
• Sort the files by size, smallest first.
4.6 Further Reading
Linux Documentation Project - Getting Help
4.4. Summary 19
Sphinx/Rest Memo
20 Chapter 4. Getting Help
CHAPTER 5
Files and Directories
We learn in the previous chapter how to list the contents of directories, and how the file system was structured.We also learnt how to navigate from one directory to another. Now we need to be able to create and delete thesefiles and directories ourselves; we also will want to modify these files and directories by moving, renaming, andcopying them. The final part will introduce the nano editor and how to use it to create text files.
5.1 Overview
In this chapter you will learn:
• how to create, copy, move and delete files and directories
• how to edit files.
5.2 Creating directories
The first thing we will probably want to do when starting a new project is to create a new directory. This is donesimply with the make directory command, mkdir. This take a single argument, the name of the directory wewish to create:
1 $ ls -F2 Public/ Private/ Documents/3 Thesis/ Research/ contacts.txt4 $ mkdir Projects5 $ ls -F6 Public/ Private/ Projects/7 Documents/ Thesis/ Research/8 contacts.txt
% Should we mention paths and mkdir -p If we wished, we could also provide a path which includes the name ofthe new directory.
1 $ ls -F2 Public/ Private/ Documents/3 Thesis/ Research/ contacts.txt
(continues on next page)
21
Sphinx/Rest Memo
(continued from previous page)
4 $ mkdir Projects/polonium/data5 mkdir: cannot create directory ‘Projects/polonium/data’: No such file or directory
The error is a little strange, however it is caused by the fact that we have a path that does not exists. WhilstProjects exists, Projects/polonium does not. We can use the -p flag, which creates the parent directoriesif they do not already exist.
1 $ mkdir -p Projects/polonium/data2 $ ls -F -a Projects3 . ..4 $ ls -F Projects5 polonium/6 $ ls -F Projects/polonium
5.3 Copying file and directories
At some point we may want to copy files or directories - perhaps to create a back up, or we are reusing part for anew project. Either way we will need cp, the copy command.
This takes the form:
cp [options] <source_path> <destination_path>
cp has a many options available and we shall look at some later, however it is still worth looking at the man pagesto see what else it can do.
1 $ ls2 report.txt3 $ cp report.txt backup_report.txt4 $ ls5 backup_report.txt report.txt
We mentioned earlier that <source_path> and <destination_path> are paths. We used relative paths in theexample above, but we could also use full paths, or other relative paths such as:
1 $ cp ~/report.txt /home/rel/backup_report.txt2 $ cp /home/rel/report.txt ../../backup/backup_report.txt3 $ cp report.txt /backup/.4 $ ls /backup5 backup_report.txt report.txt
Line 3 is the more interesting of these examples. Instead of specifying a full path that included a destination filename, we provided just a path to a directory. When we do this cp assumes we want to keep the name of the filethe same.
Tip: cp also accepts the format cp [options] <multiple_sources> <directory>. In this format,you can pass a whole list of files as arguments to cp, and as along as the final argument is a directory (or path toone) they will be copied (with their original names) into that directory.
This is especially useful when combined with wildcards which we will cover later.
Without passing any options to cp, it will not copy directories. To copy directories and their contents we need touse the -r flag which tells cp to recursively copy the directory and its contents.
1 $ ls2 backup_report.txt data report.txt3 $ cp data bk_data
(continues on next page)
22 Chapter 5. Files and Directories
Sphinx/Rest Memo
(continued from previous page)
4 cp: omitting directory 'data'5 $ cp -r data bk_data6 $ ls7 backup_report.txt bk_data data report.txt
Tip: Another useful flag is -p which preserves the owner, permissions and timestamps of the file (don’t worryabout these yet.)
5.4 Moving files and directories.
To move files (and directories) we use the move command, mv. This has the same syntax as the cp command,with the exception that the -r flag is not needed to move directories.
1 $ ls2 backup_report.txt bk_data data report.txt3 $ mv backup_report.txt data/4 $ ls5 bk_data data report.txt6 $ ls data/7 backup_report.txt
5.4.1 Renaming files and directories
Since mv requires a source and a destination, it can be used to rename files by moving them to the same locationwith a different name.
1 $ mv data/backup_report.txt old_report.txt2 $ ls3 old_report.txt bk_data data report.txt4 $ mv old_report.txt backup_report.txt5 $ ls6 backup_report.txt bk_data data report.txt
On line 1 we renamed the file and moved it in the same line. On line 4 we renamed the file. We can do the samewith directories without the need for flags.
5.5 Deleting files and directories
To delete files we use the remove command, rm. One very important thing to remember, is that unlike when usingthe GUI, there is no undo command when you delete a file on the command line, so be very careful. This syntaxis:
rm [options] <path_to_file>
As with the previous commands, rm accepts an absolute or relative file name. Lets see some examples:
1 $ ls2 backup_report.txt bk_data data report.txt3 $ rm backup_report.txt4 $ rm /home/rel/report.txt5 $ ls6 bk_data data
5.4. Moving files and directories. 23
Sphinx/Rest Memo
5.5.1 Deleting directories
When it comes to deleting directories we have two choices. Firstly their is the remove directories command(rmdir). This only works on empty directories; this is useful as it forces us to be careful and check what is in thedirectory first. The second option is to use rm. Like many commands, this has additional options. In this case (aswith cp) we want the recursive option, -r. This will delete a non-empty directory, so it is quicker than rm, butwe need to be more careful.
1 $ ls2 bk_data data3 $ rmdir bk_data4 rmdir: failed to remove 'bk_data': Directory not empty5 $ rm -r bk_data6 $ ls7 data
Notice how on line 4 we are warned that bk_data is not empty, but we are not warned when we use the rm -rcommand.
5.6 Summary
5.6.1 Commands
• mkdir - creates directories.
• cp - copies files/directories.
• mv - moves or renames files/directories
• rm - removes (deletes) files and directors.
• rmdir - removes (deletes) empty directors.
5.6.2 Concepts
• No undo. There is no undo when we delete files and directories on the command line.
5.7 Exercises
• Using the command touch, create a blank file.
• Rename the file.
• Create a copy of the file.
• Create a new directory.
• Move the file into the new directory.
• Delete the directory using rmdir.
– Why did this fail?
– Which two methods could you use to delete the directory?
5.8 Further Reading
Linux Documentation Project - Files and the Filesystem
24 Chapter 5. Files and Directories
CHAPTER 6
Wildcards
In previous sections we have used commands that act on a single file. Part of the power of the command line isbeing able to act on many files at once. Wildcards allow us to specify patterns instead of individual file names.
6.1 Overview
In this chapter you will learn:
• what globbing and wildcards are.
• how the shell expands wildcards.
• about the types of wildcards and how to use them.
6.2 What are wildcards?
Wildcards are characters with special meanings. They can represent letters or numbers, sets of characters or evenmany characters. When we put a wildcards in place of a filename, bash will expand these wildcards out to anymatching file name (this is known as file name expansion, or globbing.).
6.3 Wildcards
Bash only has few wildcards that it recognises, but these are very powerful.
• ? - represents a single character.
• * - represents zero or more characters.
• [] - represents a range or set of characters.
• {} - represents a comma separated list of terms.
• [!] - represents any character not in range or set.
• \ - this is used to escape any of the above characters so that you can search for them in a filename.
We use these like so:
25
Sphinx/Rest Memo
1 $ ls2 file1.jpg file2.jpg figure.png report.txt facts.txt3 $ ls f*4 file1.jpg file2.jpg figure.png facts.txt5 $ ls *.txt6 report.txt facts.txt
6.4 What’s going on?
When we execute commands such as ls *.txt bash expands this and replaces *.txt with any file or directoryin that matches the pattern (in this case, zero or more characters followed by ‘.txt’).
Caution: Unless we place an explicit path, or path separator (/) into our patterns, bash will not expand to filenames outside the current directory.
6.5 More examples
The best way to understand these is with examples.
6.5.1 Matching many, one and zero characters (? and *)
To match zero or many, or exactly one of any character, we use the * and ? wildcards.
1 $ ls2 abc abc1 abc2 abc3 xyz xyz1 xyz2 xyz33 $ ls abc*4 abc abc1 abc2 abc35 $ ls abc?6 abc1 abc2 abc3
Notice how * matches both abc and abc1, abc2, abc3, whilst ? does not match abc. This is because * is for zeroor more of any character whilst ? is for exactly one of any character. If we wanted to match against one or morecharacters, * would not work. However if we combined the two, ?* would match one of more characters.
6.5.2 Sets and ranges, []
Square brackets ([]) allow us to do two types of matches. We can match against a list of characters, or a rangeseparated by a hyphen -.
If we wanted to list all files that are three letters long, we could do.
1 $ ls [a-z][a-z][a-z]2 abc xyz3 $ ls [abcxyz][abcxyz][abcxyz]4 $ ls [a-y][a-y][a-y]5 abc6 $ ls [A-Z][A-Z][A-Z]7 $
On line 1 we gave bash three ranges to match. Each range will only match a single character, so three ranges wereneeded to match three characters. Also note how the last command did not match up against anything as linux iscase sensitive.
26 Chapter 6. Wildcards
Sphinx/Rest Memo
Lines 1 and 3 give the same output, but where as line 1 uses a range, line 3 uses sets. We could combine ranges inthe set. To match all letters and numbers, both upper and lowercase we would use [a-zA-Z0-9].
By placing an ! at the beginning of a set or range, we tell bash to match anything except the characters listed inthe set.
Warning: All special characters lose their meaning inside of square brackets ([]), with the exception of -which gains special meaning. i.e. [?*] will only match a question mark or an exclamation mark, these will notbe used a wildcards.
6.5.3 Lists, {}
Curly brackets {} are used to hold a comma separated list of terms that can be matched
1 $ ls {abc,xyz}12 abc1 xyz13 $ ls abc{1,3}4 abc1 abc3
6.6 Everyday examples
• cp filename{,.bak} - Create a backup of a file called “filename.bak” -
• cp *{,.bak} - Create a backup of all files in the local directory called “filename.bak” -
• cp *{.txt,.jpg} ~ - copy all files ending in .txt and .jpg to the home area.
• rm *~ - delete all emacs backups.
6.7 Summary
6.7.1 Concepts
• Bash will expand wildcards, and replace them with all matching files and directories in the current directory.
• This is known as globbing.
6.7.2 Commands
• ? - represents a single character.
• * - represents zero or more characters.
• [] - represents a range or set of characters.
• {} - represents a comma separated list of terms.
• [!] - represents any character not in range or set.
• \ - this is used to escape any of the above characters so that you can search for them in a filename.
6.6. Everyday examples 27
Sphinx/Rest Memo
6.8 Exercises
The following questions ask you to consider what globbing pattern would match the following groups of files.Feel free to create the files using the touch command, and then use ls and globbing to try and match.
• What pattern would match any jpeg, or png file (but not files with other extensions)?
• What pattern would match only files that started with two letters followed by a number?
• What pattern would match files that started with only a capital letter?
• Look around your home directory, and try matching files with a few different patterns.
6.9 Further Reading
Linux Documentation Project - Wildcards
28 Chapter 6. Wildcards
CHAPTER 7
Piping and redirecting
One of the great powers of the command line, and Linux, is the ability to seemlessly chain small commandstogether to do really useful things. In the previous chapters we have learnt how to use the command line to dovarious tasks, and run commands. In this chapter we look at how to save the output from those commands, andchain commands together.
7.1 Overview
In this chapter you will learn:
• what streams are.
• how to redirect stdout and stderr to a file.
• how to direct a file into a command using stdin.
• how to chain the input and output of commands together with pipes.
7.2 Introduction to stdin, stdout, and stderr
Every command that we run on the command line has three * streams* connected to it. These are:
• stdin(0) - Standard In(put), data fed into the program/command.
• stdout(1) - Standard Out(put), data received from the program/command. By default this prints out to thescreen.
• stderr(2) - Standard Error, for error messages generated by the program/command. This is also sent to thescreen by default.
We can redirect these streams into files, into each other, or into commands. Piping allows us to chain manycommands together, each using the output of the previous.
29
Sphinx/Rest Memo
7.3 Redirecting to a file
When we run a command it normally prints its output to the screen. Sometimes it is more convenient to save thisto a file for use later, or as a record. The greater than symbol, >, is used to tell the command line we want thestandard output (stdout) to be sent to a file instead of the screen. Here is an example
1 $ ls2 Public Private Documents3 Thesis Research contacts.txt4 $ ls > output.txt5 $ ls6 Public Private Documents7 Thesis Research contacts.txt8 output.txt9 $ cat output.txt
10 Public11 Private12 Documents13 Thesis14 Research15 contacts.txt
Let’s look at this line by line.
• Line 1 - We use the ls to show the contents of our current directory.
• Line 4 - Now we use the > to redirect the output to a file.
• Line 5 - Using ls to show the directory contents again, we can see a new file called output.txt hasbeen created.
• Line 8 - Using cat we look at the contents of the file.
7.3.1 Different outputs
You might have noticed that in the above example the formatting of the output when it was printed to screen, andwhen it was written to a file are different. This is because the program knows the size of the screen, and can formatthe output to fit. When we output to somewhere other than the screen, the safest thing is to place one entry oneach line.
7.3.2 Redirecting to an existing file
When we redirect to a file that does not exist, it will be created. However if we redirect to a file that does exist, itscontents will be overwritten.
1 $ cat output.txt2 Public3 Private4 Documents5 Thesis6 Research7 contacts.txt8 $ wc -l contacts.txt > output.txt9 $ cat output.txt
10 2 contacts.txt
if we wish the output to be appended to the file, we can use the double greater than operator, >>.
30 Chapter 7. Piping and redirecting
Sphinx/Rest Memo
1 $ cat output.txt2 2 contacts.txt3 $ ls >> output.txt4 $ cat output.txt5 2 contacts.txt6 Public7 Private8 Documents9 Thesis
10 Research11 contacts.txt
7.4 Redirecting standard error, stderr
We mentioned earlier that there are two output streams; standard out, and stand error. You will notice that thesehad numbers associated with them, which actually represent the stream. If we put a number infront of the redirectoperator (>) then it will redirect that stream to a file, without a number it defaults to stream 1, stdout.
1 $ ls contacts.txt mycontacts.txt2 ls: cannot access 'mycontacts.txt': No such file or directory3 contacts.txt4 $ ls contacts.txt mycontacts.txt 2> error.txt5 contacts.txt6 $ cat error.txt7 ls: cannot access 'mycontacts.txt': No such file or directory
We can also redirect both stdout and stderr to a file. We do this by directing stdout to a file, and then directingstderr to stdout. This is similar to redirecting to a file, however we need to put \& infront of 1 to identify it as astream, and not a file.
1 $ ls contacts.txt mycontacts.txt > output.txt 2>&12 $ cat output.txt3 ls: cannot access 'mycontacts.txt': No such file or directory4 contacts.txt
7.5 Redirecting from a file
Using the less than operator, <, we can send data to a program from a file. This reads in the data from a file andpassed it to the standard in, STDIN, stream of a program.
1 $ wc -l contacts.txt2 2 contacts.txt3 $ wc -l < contacts.txt4 2
Notice that whilst both commands use the contents of the file as input to the program, the output is different. Thisis because when we redirect the file to the input, the data is sent anonymously. This can be useful sometimes ifyou do not want ancillary data sent to another command.
7.6 Pipes
So far we have looked at how to send output to files, and read input from files. Lets instead look at how to senddata from one program to another. We do this using the pipe operator, |. This takes the output from the commandon the left and sends it as input to the command on the right.
7.4. Redirecting standard error, stderr 31
Sphinx/Rest Memo
1 $ cat contacts.txt | wc -l2 2 contacts.txt
We can chain as many of these as we want together. You may also want to combine pipes with a redirect so thatthe output can be saved.
1 $ cat contacts.txt | wc -l > words_contact.txt2 $ cat words_contact.txt3 2
7.7 Summary
7.7.1 Concepts
• Streams are flows of input or output data.
• There are two outputs, standard output, and standard error.
• Streams can be redirected to or from a file.
7.7.2 Commands
• > Save output to a file.
• >> Append output to a file.
• < Read input from a file.
• 2> Redirect error messages.
• 2>\&1 Redirect stderr to stdout.
• | Send output from one command as an input to another.
7.8 Exercises
1. Using ls and redirection, create a file listing all files and directories in your home directory.
2. Add to the file you created in the previous question, the contents of /. Do not overwrite your previous entry.
3. Using the command ls / /nonexistantdir, redirect standard out to one file, and stderr to another.
4. This time send them to the same file.
7.9 Further Reading
Linux Documentation Project - All about redirection.
32 Chapter 7. Piping and redirecting
CHAPTER 8
Finding files
At somepoint we will lose something, or know that there is something we want, but not know where it is. Linuxhas very powerful tools built into it to find files and even things in files. This chapter will give a brief overview ofhow to find files based on their names and other attributes (but not contents).
8.1 Overview
In this chapter you will learn:
• how to use the find command to search for files.
8.2 Finding files with find
In this section we introduce the find command for finding files on your system. This can search the filesystemfor any files and directories using many different tests to find the files.
Searching with find When we need to search for files the main command to use is find. Find has many options,however we shall focus on a few examples which will cover most of what you will need.
Suppose we want to find a file, we know that it is called data.txt, and know it is somewhere in our /homedirectory, but cannot remember where:
1 $ find . -name 'data.txt'2 ./Public/analysis/data.txt
Lets look at this in more detail.
• find - - this is the name of the command we want to use, it is then followed by a path and an expression.
• . - - this is the path, we have old find to look in the current directory. It will also look in any directoriesinside this one.
• -name 'data.txt' - - this is the expression, in this case we have told find to look for a file name thatmatches what is in the quotes.
The expression is a test with an optional argument, or series of tests and arguments. In the example above the testtells find to look at the name of the file (we could also use iname which looks at the name but ignores the case).
33
Sphinx/Rest Memo
We then provide a pattern (inside quotes) for find to match the name to. Whilst we provided a full name, we canuse wildcards as we would in bash. A quick reminder is listed(see chapter ref{wildcards} for more detail).
• * - - matches multiple characters.
• ? - - matches a single character.
• [] - - matches an characters between the brackets.
• [\^{}] - - matches an characters not between the brackets.
8.2.1 Searching by age
Perhaps we have some code that worked yesterday, but does not work today. In this case we could search for anyfile modified in the last 24 hours. We can do this using the test mtime:
1 $ find . -mtime -12 ./Public/analysis/analysis.sh3 ./Public/analysis/output.txt
The mtime takes an integer($n$) as its argument, this can be:
• -n - - less than n days ago.
• +n - - greater than n days ago.
• n - - exactly n days ago.
If we wanted finer control, we could use mmin which uses the same arguments, but uses minutes as its units.
8.2.2 Searching by size
Another example might be that we are running out of disk space and want to find the largest files over a certainsize in our home directory.
1 $ find ~ -size +500M -exec ls -sh {} \;2 ./Public/analysis/analysis.sh3 ./Public/analysis/output.txt
Lets look at this bit by bit:
• find - the program we are using
• ~ - the path we want to look in, in this case our home directory.
• -size +500M - the expression. This has two parts, the test (size) and the condition (+500M), greater than500 mb.
• -exec - This tells find that once it has found a match, in stead of printing it to the screen it should runcommand on it. In this case the command is ls -sh \{\}\\;, which causes find to print the file nameand size. {} is replaced with the file found, and we end exec with a command{;}, but as this meanssomething to bash we have to escape it with a \ - .
We could also combine tests to look for files bigger than 500mb and older than 5 days:
1 $ find ~ -mtime 5 -size +500M -exec ls -sh {} \\;
There are many more tests that can be used with find, a comprehensive list is given in the man pages.
34 Chapter 8. Finding files
Sphinx/Rest Memo
8.3 Finding files quickly with locate
The locate command is faster than find, however this comes at a price. Locate works by searching through adatabase which is usually built everynight. This means that is does not usually find newly created files, or thoseon external devices. To find a file you just type:
1 $ locate pattern
Pattern can be a file name, a pattern including globbing character. Locate matches against the full path a file name,so if you use globbing characters, but do not start the expression with a full path or wildcard, you will not matchanything.
8.4 Which
Sometimes, especially when writing a script it is useful to know where the command (such as ls) is actualylocated. This means you can then run the command with out any local modifications (you can configure your shellto add certain flags to a command by default, unless you then run the command from its full path, those flags willalways be included).
The which command will tell you the full path of any command on your system (it will also tell you which flagsare enabled by default).
The syntax is just which <command>.
8.5 Summary
8.5.1 Concepts
• Meta-data about files, such as access and modification times is searchable. We can use this to find files of acertain age, or access within the last n minutes.
8.5.2 Commands
• find path_to_search [options]
8.6 Exercises
• Find all files ending with .txt in your home directory.
• Find all files greater than 5Mb in size in your home directory.
• Find all files less than 10 minutes old in your home directory (use ls to verfiy their age).
• Find all files in your home directory that have not been accessed for 1 week.
• Have a play with locate and which.
8.7 Further Reading
Linux Documentation Project - Complex Commands, find
8.3. Finding files quickly with locate 35
Sphinx/Rest Memo
36 Chapter 8. Finding files
CHAPTER 9
Editors
At some point we are going to want to create files on the command line and fill them with text. To do this wewill need a text editor. There are many editors on linux, and many arguments. If you use linux with a GUI thereare many graphical editors that you can use (Kate and gEdit for example), however the command line editors aregenerally more powerful, and are very useful when working remotely. The main editors used are emacs and vim,whilst I might add tutorials for these later for now I will focus on nano as it is easier to use than either (for simpleeditting).
9.1 Overview
In this chapter you will learn:
• what nano is.
• what a text editor is.
• basic commands for nano.
9.2 Nano, a command line editor
nano is a text editor. This means it will display only text files, no font formatting, font sizes, styles or images. Italso means there is no mouse, everything is plain and has to be done with the keyboard. Whilst this seems like astep back, it is very useful for editing and has some benefits when it comes to writing as style is separated fromcontent.
To start nano we issue a single command, with an optional file name. If the file exists, it is opened for editing, ifnot it will be created when we save later.
nano <file>
One advantage of nano when learning an editor is that it displays the main commands that you need along thebottom of the screen.
GNU nano 2.5.3 New Buffer
(continues on next page)
37
Sphinx/Rest Memo
(continued from previous page)
^G Get Help ^O Write Out ^W Where Is ^K Cut Text ^J Justify ^C Cur Pos^X Exit ^R Read File ^\ Replace ^U Uncut Text ^T To Spell ^_ Go To Line
9.3 Saving and Exiting
The key combination for exiting nano is ^X - to do this press and hold Ctrl and then tap x (the case of the letteris not important).
Tip: Many interactive programs in linux make use of the modifier keys: The Control key, Ctrl; The alt or metakey, Alt; and the super or windows key Super. These are sometimes shortened and represented as: * Ctrl: C-or ^ * Alt/Meta: M- * Super/Windows, Super: S-
These can be stacked as either: * C-M-x, which means press and hold Ctrl and Alt, and then tap x; or * C-cM-x, which means press and hold Ctrl and tap c, and then press and hold Alt and tap `x.
Whilst nano will prompt us to save the file if we have not done already, we can do this manually. To Write Out, orsave the file press ^O. Nano will prompt for a file name; if the file already has a name, this will be pre-filled, butcan be changed. Press Enter to confirm.
• ^X - eXit nano. Nano prompts you to save if you have not already.
• ^O - write Out. This saves the file, with its current name
If you started a command and want to cancel it, or just got stuck in a menu, press ^C to cancel.
9.4 Navigating files
To navigate through text in nano we use the same keys as on the command line.
• Left and Right Arrow keys - move the cursor along the line.
• Up and Down Arrow keys - move cursor up and down one line at a time.
• PgUp and PgDn - move up and down one page at a time.
• Home and End - move to beginning and end of line.
9.5 Deleting, Cutting and Pasting text
To delete text in nano we use the del (delete) key and the backspace key. They work slightly differently toeach other. The del key will delete one character at a time to the right of the cursor. The backspace key willdelete one character at a time to the left of the cursor.
To delete more than one character at a time, we use the cut commands.
38 Chapter 9. Editors
Sphinx/Rest Memo
9.5.1 Cutting and pasting text
^K will cut text, but only whole lines at a time. ^U will then paste the text at the cursors current position.
To cut only specific sections of text, press ^^ (that is, hold Ctrl and Shift, and then press 6). This will markthe start of the text to cut, we can then move the cursor with with arrow keys to the end of the text we wish to cutand press ^K to cut it.
9.6 Undoing actions
If we make a mistake, we can undo this by press M-U. To undo an undo, that is, to redo something we press M-E.
9.7 Summary
9.7.1 Concepts
• Many text editors exits in linux.
• Command line text editors use the keyboard and do not use the mouse.
• Many linux programs make use of key combinations and modifier keys.
• ^ or C- means press and hold the Ctrl key, and then tap the next key shown.
• M- means press and hold the Alt or Meta key, and then tap the next key shown.
• S- means press and hold the Super or windows key, and then tap the next key shown.
9.7.2 Commands
• nano <filename> - open file.
• ^X - Exit nano
• ^O - Save file.
• ^K - Cut the whole line of text.
• ^U - Paste or uncut text.
• ^^ (C-^) - mark the beginning of text to cut, mark the end with ^K.
• M-U - undo a change.
• M-U - redo a change.
9.8 Exercises
• Use nano to create a files with a few lines of text.
• Using find and replace, make sure every sentance ends with the word penguin.
• Save the file.
• Move the first line to the bottom of the file.
• Exit nano.
9.6. Undoing actions 39
Sphinx/Rest Memo
9.9 Further Reading
Nano Manual
40 Chapter 9. Editors
CHAPTER 10
Bash Scripts
One of the beauties of using the command line is that we can automate the more tidious or repetative tasks. Wedo this by using scripts. These are exactly what they say they are, scripts for the command line to follow. Thischapter will introduce the idea of scripts, how to run them, and some of the features of bash we may want to usein them. Hopefully I will write a more comprehensive tutorial on bash scripts and the more advanced featuresat some point. Please let me know if you would be interested as this will help raise the priority of it in my everincreasing to-do list.
10.1 Overview
In this chapter you will learn:
• what a script is.
• how to run scripts.
• how to pass variables to a script.
• how to use conditionals.
• how to use loops.
10.2 What is a Bash Script?
Bash scripts are files that contain a series of commands for the shell to execute. Any command you run in theterminal can be ran in a bash shell, and any command you run in a script can be ran in the terminal.
Bash or shell scripts, can be used to automate many of the tasks we do on the command line, they can be also takeinputs to make them more generic and reusable.
10.3 Basics
Creating and running bash scripts (should) follow a common basic form. All scripts should start with a shebang(this is explained below), and then they should include some code and comments. To run, or execute, a bash script
41
Sphinx/Rest Memo
we need to make sure it has the correct permissions an then run in it in the correct way. All of this is detailedbelow, and an example script is given as well.
10.3.1 Example Script
1 #!/bin/bash2
3 # Print out hello world.4 echo "Hello World!"
10.3.2 Shebang!
In the example above, the first line begins with a shebang (#!). This is then followed by the path to an interpretorthat can read the code in the script. As we are writing a bash script, this is line points to the bash shell (/bin/bash). It could point to any interpretor, including other scripting languages such as python.
10.3.3 Code and Comments
After the shebang (and a blank line for readability) comes the code. Any commands that you can type into theterminal and run can be used in here. In the example above we use echo "Hello World!". The echocommand takes a string as input and print it to the command line. This script, when run, will just print helloWorld! out to the terminal.
It is good to have comments in our code to tell people what it will do. To write comments in bash, we just put ahash, #, at the beginning of the line.
10.3.4 Permissions
To run the code, the file must have executable permissions. This is looked at in detail in sec: perms. For ourbash scripts, running chmod 755 my_script.sh, “my_script.sh” is the name of your script should be fine.This will grant the user read, write, and execute permissions; and grant other users read and execute permissions.
10.3.5 What’s in a name?
Linux is an extentionless system. This means that the file extension (the 2-4 characters after the fullstop) do notcontrol how the file is treated. We do not need file names to end in .exe to be able to execute them, that is doneby permissions. Likewise we do not need scripts to end in .sh; However, giving files a name with an extensionhelps in knowing how to run them without opening them. By convention bash scripts end in .sh.
10.3.6 Running the code
To run the code we just need to type the following in a terminal:
1 ./my_script.sh
This will then (hopefully) run the code and print “Hello World!” to the terminal.
Important: Notice the ./ at the beginning of the text command? This tells the shell that the command is in ourcurrent directory. When we run commands, the shell searches our PATH. This contains a list of directories thathold executables that we can run. As our current directory is not in one of these, we need to explicitly tell the shellwhere it is.
42 Chapter 10. Bash Scripts
Sphinx/Rest Memo
10.4 Variables
Variables allow us to store a piece of data for use at a latter date, or part of the script. They can contain both staticvalues, or command argument to be evaluated. They are very useful: they can save us time by having one variableto edit that is then called later in the script, they can store data from a command, or they can allow us to pass dataand inputs to our scripts.
It is easy to create and call variables in bash. To create them we specify the variables name followed by an equalssign,‘‘=‘‘, and then the value we wish it to hold. To get the out put of the variable, we call it by its name, with adollar sign infront, $. Let’s look at our simple script again, this time with variables.
1 #!/bin/bash2
3 myString="Hello World!"4 # Print out hello world.5 echo $myString
10.4.1 Storing commands - backticks
We mention above that we can also use variables to store commands to be evaluated. To do this we use backticks.When we create a variable then anything inside backticks ` will be evaluated on the command line, that is thecommand will be processed and the output will be stored in the variable.
1 #!/bin/bash2
3 todaysDate=`date +%F`4 # Print out the date5 echo "Todays date is: " $todaysDate
10.4.2 Inbuilt variables - Command line arguments
Bash has many inbuilt variables that are set automatically. Some are only set when we are using scripts. These arethe ones that are most useful to us at the minute.
There are four variables set when we run a script:
• $0 - This returns the name of the script.
• $# - This returns the number of command line arguments given to the script.
• $1 - $9 - These return the 1st to 9th command line arguments.
• $* - This returns all the command line arguments.
These allow us to pass arguments to our scripts, and use them. Lets look at another example to see more:
1 #!/bin/bash2
3 echo "You just ran " $04 echo "You entered " $# " names."5 echo "The third name is " $36 echo "All the names are: " $*
We can then run this with arguments on the command line.
1 $ ./sith_lords.sh Vader Sidious Maul2 You just ran ./sith_lords.sh3 You entered 3 names.4 The third name is Maul5 All the names are: Vader Sidious Maul
10.4. Variables 43
Sphinx/Rest Memo
10.5 Conditionals
As our scripts progress and get more advanced, we will want to add some flow control to them. This helps to makethem more “intelligent”. We do not want our script to output the third name if there are only 2, or we do not wantto access a file that does not exist. To guard against this we can use if statements to test whether something is truebefore we run the code.
If statements take the following minimal form:
1 if [ <expression> ]2 then3 <action_if true>4 fi
All if‘ statements start with a test, this consists of an expression placed inside square brackets. On a new line wethen have the command then followed by the code to run if the test is passed - this can be multiple lines, it isterminated by fi.
We can also use the if-else conditional:
1 if [ <expression> ]2 then3 <action_if true>4 else5 <action_if false>6 fi
Let’s look at an example.
1 #!/bin/bash2
3 echo "You just ran " $04
5 # Check if there are less than 36 if [ $# -lt 3 ]7 then8 echo "You only entered " $#" names, this script requires at least 3".9 else
10 echo "You entered " $# " names."11 echo "The third name is " $312 echo "All the names are: " $*13 fi
Note: There are many comparison operators in bash such as the -lt that we used above which test if one valueis less than another. A comprehensive list can be found here.
10.6 Loops
One of the main reasons to use shell scripts is to automate boring and repetitive jobs. To help do this bash hasloops to aid in repeating jobs. The for loop allows us to repeat certain blocks of code as we count through asequence - this could be a list of files etc.
1 #!/bin/bash2
3 for planet in "Mercury Venus Earth Mars Jupiter Saturn Uranus Neptune Pluto;4 do5 echo $planet6 done
44 Chapter 10. Bash Scripts
Sphinx/Rest Memo
Here we provide the for loop with a list of planets. Each one is then assigned to the variable planet one byone, and the code after do is executed for each item in order.
Suppose we had an analysis code to run called “myAnalysis”. This takes one input, a data set and prints its outputto the screen. Using a for loop, and redirect we can save this data.
1 #!/bin/bash2
3 for dataset in `ls *.data`;4 do5 ./myAnalysis $dataset > $dataset.out6 done
Here we passed the loop the command `ls *.data`, this was expanded out to a list of files ending in “.data”.For each file in the list, the code ./myAnalysis $dataset > $dataset.out is ran. This runs over thedataset, and produces an output file of the same name as the dataset, but with “.out” on the end.
10.7 Summary
10.7.1 Concepts
• Scripts - these are a series of commands that can be run from a file.
• To run a script we need to give the full path of the script, this means it must start with ‘‘./‘‘
10.7.2 Commands
• mvVar="some value" - variables are created by specifying a variable name, followed by the equalssign and then the value.
• $myVar - variables are accessed by placing a $ in front of their name.
10.8 Exercise
• Write a simple script that takes your surname (family name) as input, and prints it to the screen with todaysdate.
• Write a simple script that loops over all the files in your home directory and prints them to the screen.
• Write a script that takes a single filename as input, and prints out its filename if it exists, and “Cannot findfile x” if it does not. You may wish to look at the if statments in further reading to find the correct check.
10.9 Further Reading
Developing good scripts - Bash Guide for Beginners
If statements - Bash Guide for Beginners
For Loops - Bash Guide for Beginners
10.7. Summary 45
Sphinx/Rest Memo
46 Chapter 10. Bash Scripts
CHAPTER 11
Managing Processes
Early we learnt that almost everything in Linux is a file, if it is not a file, then it is a process. Aprocesses carry outtasks on the computer; A program is code and instructions for the computer to run, when it runs these instructionsit is carrying out a process.
The command line gives us a lot of control over these processes. We can stop them, pause them, and even killthem. Being able to control these processes and what jobs are runnign gives us even better control of the system.Sometimes programs that we are running go wrong, or we want to pause them whilst we do something else. Thissection will look at how to control jobs and processes, and what they are.
11.1 Overview
In this chapter you will learn:
• what processes are.
• how to see what is running on your system.
• how to control processes.
• what jobs are, and how to control them.
11.2 What are processes
When we run programs on linux, each one takes up an amount of processing power and memory. Linux managesthese programs and grants them access to the resources on the system. When a program is running it is called aprocess (sometimes a daemon).
11.3 What is running
We can see what is currently running on our system by using the top command. This displays the process runningon the system ordered by percentage of the CPU that they are using (then can be ordered by other fields if we wish).Typing top into the terminal and pressing return and you should be presented with something that looks likethis:
47
Sphinx/Rest Memo
1 top - 12:57:54 up 3:45, 1 user, load average: 0.35, 0.23, 0.262 Tasks: 307 total, 1 running, 299 sleeping, 7 stopped, 0 zombie3 %Cpu(s): 1.8 us, 1.0 sy, 0.0 ni, 97.2 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st4 KiB Mem : 8066744 total, 1897928 free, 2048852 used, 4119964 buff/cache5 KiB Swap: 8388604 total, 8388604 free, 0 used. 5050556 avail Mem6
7 PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND8 2100 user 20 0 568740 120908 103240 S 4.3 1.5 3:26.42 Xorg9 3108 user 20 0 2371504 773616 176456 S 4.3 9.6 31:40.68 firefox
10 2214 user 20 0 2189232 166192 64576 S 3.6 2.1 3:29.37 gnome-shell11 2786 user 20 0 1246196 290288 87348 S 2.3 3.6 7:23.17 thunderbird12 4030 user 20 0 750548 41684 25452 S 2.3 0.5 0:42.35 gnome-term+
Lets look at the important bits in some more detail:
• line 2 - Tasks are just another name for processes. We can see that there are a few different states (running,sleeping, stopped, and zombie). Sleeping processes are ones that are waiting for an event to occur beforethey can run again. Stopped jobs are waiting to be told to run again. Zombies are processes that have beenkilled, but have not been cleaned up properly.
• line 4/5 - This lists how much memory there is in the system(total), how much is free to be used (free), andhow much is actually being used (used). line 5 is the same but for swap; this is memory that has been copiedto disk (usually when the OS runs out of physical memory. If too much swap is being used, the system willbe slow.
• lines 7-12 - This is a table of process. The import columns here are: PID, which is the unique id given toeach process; %CPU and %MEM, which is the percentage of CPU and memory being used by the process;TIME, which is how long the process has been running for; and COMMAND, which is the command (orprogram) which is running.
top will give a real time view of what is running on the system. This is useful for finding what is taking up allthe CPU and memory, but more often we will want to see a full list of everything that is running, not just the topprocesses. In which case, we use the command ps.
ps will show the programs currently running in that terminal. To display all running processes on the systemwe need to type ps aux. This will produce a lot of output. Usually we pipe it to grep so that we can findinformation on specific process.
11.4 Killing a process
Occasionally a program crashes, or it is taking too long and we want to kill it. In this case we use ps and grep tofind the PID of the process. Then we can kill the process with the kill command. Lets assume that firefox hasstopped working and we want to kill it.
1 $ ps aux | grep firefox2 user 3108 14.8 10.2 2396440 825772 tty2 Sl+ 09:17 40:29 /usr/lib64/firefox/
→˓firefox3 user 11283 0.0 0.0 118496 896 pts/1 S+ 13:49 0:00 grep --color
→˓firefox4 $ kill 31085 $
• line 1 - here we pipe ps aux to grep and search for firefox.
• line 2-3 - these are the lines that matched our grep request. Line 2 is our firefox process, whilst line 3 is thecommand we ran on line 1. There may be more than one instance of a program running on a system so becareful.
• line 4 - we pass the PID from the second column on line 2 to the kill command.
Using kill like this asks the process to shutdown nicely, if for some reasons the process is not able to shutdown,we can reissue the command with a -9 as a brute force method:
48 Chapter 11. Managing Processes
Sphinx/Rest Memo
1 $ kill -9 31082 $
Tip: If you are working on a linux machine locally, and the whole system freezes, and you cannot access aterminal, try pressing ctrl + alt + F2. This should bounce you to a new command line session. From hereyou can log in and kill the troublesome process.
Danger: Normally a user can only kill a process they own, one that either they started, or was started whenthe logged in. However, root can kill any and all processes.
11.5 Jobs
Jobs are process started interactively in the terminal. They can be displayed by typing jobs in the command line.Usually when we run programs in the terminal, we cannot use the terminal again until that process has finished -in this case the job is in the foreground. When a job runs in the background, it releases the terminal back to you,and then outputs its results when done. Lets look at this with the sleep command:
1 $ sleep 102 $
The above command runs for 10 seconds and then releases the terminal back. To run a command in the backgroundwe put an & after it:
1 $ sleep 10 &2 [1] 102443 $4 [1]+ Done sleep 105 $
Lets look at this line by line:
• line 1 - we enter the command, are then given the terminal back as soon as we press return.
• line 2 - This is the output from putting the job in the background. ‘‘[1]‘‘ is the number of the job, which isunique to this terminal only, and ‘‘10244‘‘ is the global PID which is unique to the system.
• line 3 - pressing return after 10 second wait, which gives the process time to finish.
• line 4 - the job has finished, and the terminal is notified. Again the job number and command are shown.
If we start running a command, and then wish to move it to the background we can do. First you have to pressctrl + z - this will pause the currently running process. We can then either bring it back to the foreground orsend it into the background to continue running.
1 $ sleep 302 ^Z3 [1]+ Stopped sleep 304 $ bg %15 [1]+ sleep 30 &6 $ fg %17 sleep 30
Lets look at the line by line:
• line 1 - we run the command sleep 30.
• line 2 - we press ctrl + z to pause the command and move it into the background.
11.5. Jobs 49
Sphinx/Rest Memo
• line 3 - the job id ([1]) is printed to the terminal along with the command.
• line 4 - we use the command bg %1, where 1 is the job id, and % tell the terminal this is a job id and not aprocess id.
• line 5 - the job id ([1]) is printed to the terminal along with the command followed by an & to tell us thatit is running in the background.
• line 6 - we use the command fg %1 similar to line 4 where we used the bg command, except that thisbrings the job to the foreground instead.
Tip: If no job id is passed to bg or fg they will default to the most recently executed process, that is the one withthe highest job id.
11.5.1 Killing jobs
Sometimes well will set a job running in the background To get a list of all the jobs running in a terminal we usethe jobs command. Then we can just use the kill command with the job id (don’t forget the % to show that it is ajob and not a PID).
1 $ sleep 30&2 [1] 124873 $ jobs4 [1]+ Running sleep 30 &5 $ kill %16 [1]+ Terminated sleep 307 $
11.6 Summary
11.6.1 Concepts
• All programs have a unique process (PID).
• All processes started in a terminal have a job id unique to that terminal.
• Jobs can run in the foreground or the background.
11.6.2 Commands
• top will show the top few processes ranked by CPU usage.
• ps aux will show all processes running on the system.
• jobs will list all jobs in the current terminal.
• kill <pid> or kill %<job id> will terminate a process based on process or job id.
• bg %<job id> and fg %<job id will move a program between background and foreground.
• pressing ctrl + z will send a foreground job to the background.
50 Chapter 11. Managing Processes
Sphinx/Rest Memo
11.7 Exercises
• What is the process using most resources on your system?
• Open nano, and then send it to the background.
• Bring it back to the foreground, and then send it back again.
• List processes running in your terminal, and terminate the one that is running nano.
11.8 Further Reading
Linux Documentation Project - Chapter 4, Processes
11.7. Exercises 51
Sphinx/Rest Memo
52 Chapter 11. Managing Processes
CHAPTER 12
Working Remotely
Over the previous chapters we have learnt how to run commands and programs on the command line, how toconrtol the jobs, edit files and bend it to our will. However all of this is on one machine (probably you local one).
At somepoint you will move beyond what your machine can do, and will want to run your code on more powerfulmachines, or pehaps your files are saved on a different machine and you want to access them. What ever thereason, you will need remote access to another linux machines, this is easy under Linux.
Linux has many built in tools to do this. ssh lets us access remote machines, scp and rsync let transfer files,and screen can be used to run commands in a remote environment.
12.1 Overview
In this chapter you will learn:
• how to use ssh to access remote machines.
• how to use scp to copy remote files.
• how and when to use rsync to copy remote files.
• how to use screen to work remotely.
12.2 SSH
This is the most basic command for working remotely. SSH stands for Secure SHell, and is used to connect to aremote machine. The syntax for ssh is:
$ ssh username@name_or_remote_machine
Here we supply the username we wish to login with on the remote system, and the full name of the remotemachines, such as ius.lancs.ac.uk.
This will then prompt for our password to login.
From here we will have a terminal to run any command we would do locally. Typing exit will kill the remotesession.
53
Sphinx/Rest Memo
If we want to display graphics from the remote machine we will need to enable “X Forwarding”. To do this weneed the -X flag with ssh:
$ ssh -X username@name_or_remote_machine
12.3 Transferring files
Inevitably, once we start working with remote machines, we will want to transfer files between them. This can bedone with both scp and rsync. scp is the more basic one, so we shall look at this first.
12.3.1 Secure Copy - scp
scp stands for Secure CoPy. It is the version of the cp command used to transfer files to and from remotemachines. It has a simple syntax, very similar to cp.
$ scp path_of_source path_of_destination
scp takes two arguments, the path of the source and destination files. These can be absolute or relative paths. Tocopy from or too a remote machine, the source or destination needs to be prefixed with the address of the machinefollowed by a colon (:).
$ scp ius.lancs.ac.uk:Public/myData .
$ scp Public/myData ius.lancs.ac.uk:Public/.
In the first line we copy the file myData from the directory Public on our remote machine to the currentdirectory on our local machine. In the second line we copy the file myData from the directory Public on ourlocal to the directory Public on the remote machine.
12.3.2 Synchronising files - rsync
RSync can be used to transfer files locally or remotely. It is more efficient than scp as it only transfers files ifthey were modified more recently on the source than the destination, in which case it only transfers the difference- this reduces the time taken to transfer files.
The syntax is:
rsync options source destination
The source and destination must be full paths. If one of them is a remote machine, this needs to be prefixed withthe address of the machine followed by a colon (:).
Some useful options are:
• -v - verbose, this prints out what files it is copying.
• -r - recursive, this will copy directories as well.
• -a - archive, this preserves symlinks, permissions and timestamps; and does recursive.
• -e ssh - this tells rsync to use ssh for the remote connections, this ensure that the connection is encryptedand secure.
• -z - this tells rsync to compress the files, this helps speed up the transfer at the expense of CPU power.
To transfer Public from the remote machine to the local we would do this: .. code-block:: bash
$ rsync -avz -e ssh ius.lancs.ac.uk:Public .
#nohup
54 Chapter 12. Working Remotely
Sphinx/Rest Memo
12.4 Screen
Screen can do many things, but it is most useful when you need to have long running programs on a remoteconnection, but do not want to leave a terminal open all that time, or your connection might end. screen allowsyou to create a new terminal session, and “detach” it, keeping your program alive even if you log out.
To use screen, ssh into a machine, and then start screen:
$ ssh [email protected]$ screen$
This will start a new instance of bash. You can then run any commands or scripts you want in there.
12.4.1 Detaching and reattaching screen sessions
To detach the screen, we use Ctrl+a d - if your connection ends, or your close the terminal, the screen sessionwill detach itself. To reattach the screen, restart the connection and type screen -r. If there are multiplescreens, you will get a warning:
1 $ screen -r2 There are several suitable screens on:3 16447.pts-2.dyn-191-235 (Detached)4 16381.pts-2.dyn-191-235 (Detached)5 Type "screen [-d] -r [pid.]tty.host" to resume one of them.6 $ screen -r 164477 $
We are then given a list of screens, we can reconnect by using the first 5 digits of the “screen” as an argument. Wecan also get a list of screens using screen -ls.
To kill a screen completely, type Ctrl+a k
12.5 Summary
12.5.1 Concepts
• ssh allows you to access a machine remotely
• scp and rsync allow us to transfer files, rsync only transfers files that are newer on the source.
• screen allows us to keep sessions alive.
12.5.2 Commands
Connect to remote host
• ssh user@remote_host
Transfer files
• scp source destination
• rsync -vaz -e ssh source destination
12.4. Screen 55
Sphinx/Rest Memo
Screen
• To start: screen
• To detach: Ctrl+a d
• To reattach: screen -r <screen_id>
• Get list of screen sessions: screen -ls
• To kill a session: Ctrl+a k
12.6 Exercises
• Try opening, detaching, reattaching and then killing at least 2 screen sessions.
• If you have access, copy a file to a remote machine, and then backagain.
12.7 Further Reading
Screen Examples
56 Chapter 12. Working Remotely
Python Module Index
llinux, ??
57
Sphinx/Rest Memo
58 Python Module Index
Index
Llinux (module), 1
59