linux for researchers
DESCRIPTION
Linux for Researchers. Dr Andrew Richards, Oxford e-Research Centre & Oxford Supercomputing Centre. Topics. What does the operating system do? Brief History of UNIX and Linux What is Linux ? Unix Philosophy Comparison to Windows Practical Exercises Further Information. - PowerPoint PPT PresentationTRANSCRIPT
1
Linux for Researchers
Dr Andrew Richards,Oxford e-Research Centre &Oxford Supercomputing Centre
2
Topics
• What does the operating system do?
• Brief History of UNIX and Linux
• What is Linux ?
• Unix Philosophy
• Comparison to Windows
• Practical Exercises
• Further Information
3
What does the operating system do?
• Manage the computer resources
• Interact with the user
• Run applications
4
Operating Systems?
5
Operating Systems
6
Brief history of UNIX and Linux
• A multi-user, multi-tasking operating system
• 1960s – MIT, AT&T, General Electric develop Multics (Multiplexing Information and Computing Services)
• 1970s – Unics (Uniplexed Information and Computing Service) – renamed to Unix
• 1972 – rewritten in C – resulted in more portable software to different hardware
• 1980 – AT&T licenses UNIX System III – consolidated differences into UNIX System V release 1
• 1983 – AT&T commercialise UNIX System V
– Academia continue to develop BSD Unix as an alternative
– BSD Unix main contributor of TCP/IP network code to the mainstream Unix kernel
• 1990– Open Software Foundation released OSF/1 – standard Unix implementation (Mach and BSD)
• 1991– Linus Torvalds starts Linux (originally Freax). Mixture of BSD and SYS-V
7
What is Linux ?
• Linux is an Operating System– Software loaded into your computer's memory when you switch it on
– Software which manages the hardware
• Linux is “Unix-like”– Unix is an operating system originally developed in the 1970s to run large, central computers
with many users
– Runs everything from phones to supercomputers today
• Features and Benefits– Stability
– Security
– Performance
– Price
– Flexibility & Open Source nature
8
Unix Philosophy
• Unix is based around commands which– Carry out one (or very few) simple tasks
– Carry tasks out very efficiently
– Carry tasks out very quietly
• Commands tend to be typed in a “shell”, the layer between you and the core Linux “kernel”
• More complex tasks are carried out by joining simple commands together:
9
Example
• grep – search for a pattern in a file• sort - sort into order• uniq – only show one copy of identical things
grep OSC *txt | sort | uniq > output.txt
• The above command could search thousands of files for lines containing “OSC” and save one copy of each to a new file (in seconds). What would a Windows user do?
10
More Unix Philosophy
• Treat everything as files and whenever possible, make files plain text and readable by anyone with anything – don't hide or obscure stuff unless it needs to be hidden
– Hierarchical File System
– Top level = /
– /bin
– /sbin
– /dev
– /etc
– /tmp
– …
• Assume that the user knows what they are doing– Comes with it's own perils!
11
Comparison to Windows
• Windows is just another Operating system• Windows is not just Desktop, Server and ‘HPC’ variants• Not Free, Not Open Source• Mostly interacted with through a GUI but there is a ‘shell’ -> ‘CMD’ or PowerShell
• Things to consider:– Installation
– Driver Support
– User Experience
– Development platform? Where will you develop and use your code?
• Read Wikipedia– http://en.wikipedia.org/wiki/Comparison_of_Windows_and_Linux
12
Practical Exercises (Information)
• Logging into a remote system– From a Unix machine (including Mac OS ) – use SSH or GSISSH to access grid resources
– From a Windows machine – use PuTTY (and Xming for X11 support) • http://www.chiark.greenend.org.uk/~sgtatham/putty/
• http://www.straightrunning.com/XmingNotes/• http://sourceforge.net/projects/xming/ (*** Use to Download ***)
• Conventions– Commands are typed in at a “prompt” which is represented by:
– prompt>
– Parts of a command you should replace (e.g. fill-in-your-name) are in itallics.
13
Remote Login
• ssh
prompt> ssh <username>@<hostname>
prompt> ssh [email protected]
prompt> ssh –X [email protected]
14
Practical Exercise: Where am I ?
• A Linux system is arranged in a tree like structure with files (leaves) and directories (branches).
• When you do something with files and directories on Linux (making a new file for example), if you do not specify where in the directory “tree” the file should be created, it defaults to your current “location”.
• This will show you your current location
• And with the ‘ls’ command will show the contents of this directory
prompt> pwd
prompt> ls
15
Practical Exercise: Create a Directory
prompt> mkdir training
This will create a new directory called training in your current location
prompt> ls
Now lets move from our home area to the training directory
prompt> pwdprompt> cd trainingprompt> pwdprompt> ls
No content should be listed as this is a new directory with no content!
16
Practical Exercise: Moving around the filesystem
prompt> pwd
To move up one level
prompt> cd ..prompt> pwd
You can also jump to other parts of the filesystemprompt> cd /usr/localprompt> pwdprompt> ls
Now keep trying to go up one directory level until you can't go any higher – you have reached the “root” of the file system – what is it called?
prompt> pwd
17
Practical Exercise: Moving to your home directory
prompt> pwd
To go back to your home directory, there are two short cuts:
prompt> cd
Or
prompt> cd ~
18
Practical Exercise: More moving around
• Pushd and Popd
prompt > pwd
prompt > pushd /tmp
prompt > pwd
prompt > popd
prompt > pwd
19
Practical Exercise: Question
What do you think will happen if you type
prompt> cd ~training
20
Practical Exercise: Permissions (1)
• All users have their own identity on a Linux system and in addition they belong to one or more groups.
• Linux (and Unix) attaches an ownership and set of permissions to every file (including directories) which define what you can and can't do to that file depending on who you are and which groups you are in
• How do we find out what permissions are in place?
prompt> ls –l
The ‘-l’ option is an example of ‘flag’ in Unix
21
Practical Exercise: Permissions (2)
Lets look at the permissions on a specific directory
prompt> ls –l /bin
The permissions of the files are listed in the first column of the output 1st character is the type of file Characters 2-4 are the read/write/execute permissions for the owner of the file Characters 5-7 are the read/write/execute permissions for the group the file belongs to Characters 8-10 are the read/write/execute permissions for everyone else.
-rwxr-xr-x 3 root root 3542 2010-01-20 15:39 zfgrep -rwxr-xr-x 1 root root 1498 2010-01-20 15:39 zforce -rwxr-xr-x 3 root root 3542 2010-01-20 15:39 zgrep -rwxr-xr-x 1 root root 74944 2006-06-16 14:59 zip
22
Practical Exercise: Permissions (3)
First character is “-” so it's a normal file (d means a directory) Characters 2-4 are “rwx” so the user can read and write and execute (run) this file as a
program 5-7 are “r-w” so any other user in the group “root” can read and run the file but not change it 8-10 is also “r-w” so any user in any group can read and run the file but not change it
Permissions can be changed with the commands chmod
prompt> chmod g-w filename
Most common problem on Unix systems is the error message ‘Permissions Denied’
Type User
Group
World Links Owner Owners group
Size Last modification time
Name
- rwx r-x r-x 3 root root 3542 2010-01-20 15:39 zgrep
23
Practical Exercise: HELP !
Help this is getting complicated !!!
All Unix / Linux commands come with their own manual
prompt> man ls
prompt> man chmod
The ‘man’ pages will give further information on ‘flags’ available and often give examples of how to use
Note: - Press ‘q’ to quit a man page
- typing man –k something will search for commands that relate to something
24
Practical Exercise: Wildcards (1)
• As you saw from the /bin directory, some areas contain lots of files.• Many Linux commands support the use of Wildcards
prompt> ls /bin
And then try
prompt> ls /bin/a*
“*” is a substitute for “anything from zero to N characters of any type”. So everything starting with a “a” is listed.
Note: If a directory starts with an “a” then all its files are listed even if they don’t start with an “a”
25
Practical Exercise: Wildcards (2)
Now try
prompt> ls /bin/A*
Is the result the same of different?
Now try
prompt> ls /bin/*a*
Differences ?
26
Practical Exercise: Wildcards (3)
Try the following:
prompt> ls /bin/*aprompt> ls /bin/[a-c]*prompt> ls /bin/[a-c]*f*prompt> ls /bin/*[0-9]*
What is happening in each of the different cases?
27
Practical Exercise: Wildcards (4)
• As well as “*” there is another wild card, “?”. • This means “anything which is is one character long”.
Try the following:
prompt> ls /bin/?a*prompt> ls /bin/*a?prompt> ls /bin/??a*
Can you explain what happens in each case ?
Make sure you understand the difference between “/bin/*a*” and “/bin/?a?”
28
Practical Exercise: More ls
ls and wildcards
prompt > ls
For Directories ?
prompt > ls –d */
prompt > ls –l | grep ^d
29
Practical Exercise: Keyboard shortcuts
Three shortcuts which will speed things up:
1. The up and down arrow keys – these enable you to go back and forth through your history of previous commands
2. The left and right arrow keys, allow you to move along a line so you can change things
3. Tab – this auto-completes command names and file names
30
Practical Exercise: Creating a file with a text editor
• Linux tries to use plain text format files as much as possible• This keeps files simple and readable by many different applications
prompt> cd ~/trainingprompt> lsprompt> nano myfile.txt
Type some text into the window that has appeared.
“The Oxford Supercomputing Centre is a complete High Performance Computing Service”
Now save the file (Ctrl-O) and exit (Ctrl-X).
prompt> lsYou should have a file called myfile.txt
31
Practical Exercise: Looking inside Files (1)
You can use a text editor to look inside a file without changing anything of course but there are quicker (and it turns out, more useful) ways of looking inside text files. To see what's inside a text file:
prompt> cat myfile.txt
(cat is short for concatenate). The problem with “cat” is that big files rapidly scroll off the top of the terminal screen.
prompt> less myfile.txt
A more powerful command is called “more” but that has been supplanted by an even more powerful command named “less”:
Remember!! For more information try ‘man less’
32
Practical Exercise: Looking inside Files (2)
There are even faster ways to look for words in text files.
Use “grep” to look for the word “OSC” on myfile.txt:
prompt> grep super myfile.txt
Note: this will search for ‘super’ exactly as it is written. Case sensitive!
Prompt> grep –i super myfile.txt
This will do the same search but ignore the case of the word
33
Practical Exercise: Copying, Moving and Deleting Files & Directories(1)
Warning: Linux does not warn about deletions
To copy a file, use “cp” and to move or rename it, use “mv”.
prompt> cp myfile.txt myjunk.txtprompt> lsprompt> mv myjunk.txt myold.txtprompt> ls
To delete a file we use the command “rm”.
prompt> rm myold.txt
prompt> ls
Q. What would ‘rm *’ do ?
34
Practical Exercise: Copying, Moving and Deleting Files & Directories(2)
Now make a new directory called “temporary” and then try to remove it using:
1. rm 2. rmdir
rm is for removing files, rmdir for empty directories.
35
Practical Exercise: Making a Shell Script
• Linux is made up from lots of very small commands which perform very specific jobs.
• It is often the case that you need to group lots of commands together into a “recipe” which you use over and over again.
• Such recipes are held in text files and are known as “scripts”.
• There are different flavours of scripts depending on which language they are written in and because we are making a script written with BASH-shell commands, this will be a “shell script”.
36
Practical Exercise: An Example Script
Our script will do two things:
1. List the contents of /etc2. Put the output into a new file called listing.txt
Use “nano” to enter the following script into a file called listing.sh:
# A comment. Other than #!, anything starting with # is ignored
#List /etc and REDIRECT the output to a file called listing.txt
ls /etc > listing.txt
and save the script as “listing.sh”.
There are two ways we can run this script. Method one involves sourcing the script (saying to the shell you're typing at, “please run this text file as a BASH script”.
prompt> . ./listing.sh
The “.” character on its own means “run this” file as a BASH script.
37
Practical Exercise: An Example Script continued…
The second way of doing this is a bit tidier. We place the strange looking line “#!/bin/bash” at the start of the script which forces the script to run as a BASH shell script every time.
Use nano to change the first line of the script so it looks like this
#!/bin/bash
# A comment. Other than #!, anything starting with # is ignored
#List /etc and REDIRECT the output to a file called listing.txt
ls /etc > listing.txt
Now try to run the file directly (so no “.” sign).
prompt> ./listing.sh
You should get ‘Permission Denied’
38
Practical Exercise: Fixing the script to run
• Use the chmod command to add executable permission to listing.sh executable and run it using the line above. Re-read the chmod man page
prompt> chmod a+x listing.shprompt> ./listing.sh
• Now looking inside your new file, “listing.txt” to check the script has worked. Note how “>” redirected the output from a command into a file. It is also possible to redirect files into commands using “<”. This is known as “redirection”
39
Practical Exercise: Other tools, finding tools, adding tools
• which <command>• e.g. which man
• whereis man
Package management (prompt > cat /etc/issue ) (prompt> uname –a )
• Apt (Debian)• apt-get install <package>• apt-cache search <string>
• Yum (Redhat)• yum search <string>• yum install <package>
• Others include yast on SuseLinux, up2date (older Redhat etc. )
40
But its not all Command Line
41
Further Information
• History of UNIX– http://www.levenez.com/unix/
• File system hierarchy standard– http://www.pathname.com/fhs/pub/fhs-2.3.html
• Vi cheat Sheet– http://www.lagmonster.org/docs/vi.html
• X– Xming (use sourceforge version
• FreeNX or NoMachine NX– http://www.nomachine.com/
– http://freenx.berlios.de/