www.ischool.drexel.edu info 320 server technology i week 9 unix/linux tools 1info 320 week 9

www.ischool.drexel.edu

INFO 320Server Technology I

Week 9

Unix/Linux tools

1INFO 320 week 9


Overview

• This wraps up the course with a handful of other tools and technologies of interest– Linux Utilities– RAID

2INFO 320 week 9


Linux Utilities

3INFO 320 week 9

From notes by Dr. Randy M. Kaplan, and (Petersen, 2009)


Linux Utilities

• We’ll look at a few more Unix applications of common interest for server administration– uniq, diff– lpr, cups– which, tar– ntp, cron– emacs

4INFO 320 week 9


uniq

• uniq removes duplicate lines in a file•uniq filename

• Example– uniq lastnames– Any duplicate names in the file named lastnames will be eliminated

– Make sure to sort the file first!

5INFO 320 week 9


diff

• diff compares the contents of one file with another file– diff filename1 filename2

• Example– diff colorsa colorsb– Compares the contents of colorsa with the

contents of colorb

6INFO 320 week 9


diff

• The diff utility bears some further examination

• Let’s assume that colorsa contains –– Red

– Blue

– Green

– Yellow

• Let’s assume that colorsb contains –– Red

– Blue

– Green

7INFO 320 week 9


diff

• The diff utility operates under the assumption the first file is to be converted into the second file

• diff output is a series of lines that define how the first file can be modified so that its content matches the second file– The output for colorsa and colorsb would be

4d3 which specifies that the 4th line of the colorsa file should be deleted in order that the colorsa file match colorsb

8INFO 320 week 9


lpr

• lpr prints a file•lpr filename

• Example– lpr areport– lpr –PofficePrinter areport (prints

the file areport on the printer named officePrinter)

9INFO 320 week 9


cups

• The Common Unix Printing System (CUPS) is a GNU print server application

• Cups is managed using a web-based tool at http://localhost:631– There you can see print jobs, manage

printers, etc.– But first you need to define the printers

to be managed

10INFO 320 week 9


cups

• It config file is /etc/cups/printers.conf– That contains the device file for each printer

and configuration entries for it– Devices can be identified by parallel name

(lp0, lp1, …), serial name (tty0, tty1), or URI

• Each device gets a print spool directory under /var/spool/cups/devicename

• The cups daemon is cupsd

11INFO 320 week 9


which

• which shows the location of the file related to the command (like whereis)

•which command

– Example•which info•/usr/local/bin/info

• Useful when a command is not doing what is expected

12INFO 320 week 9


tar

• tar is short for ‘tape archive’– Original purpose was to create and read

magnetic tape archives – Today it’s used to extract a directory with

subdirectories and files beneath it

• tar files are also frequently compressed– The extension of a tar file that has been

compressed will be .tar.gz or .tgz

13INFO 320 week 9


tar

• tar syntax is– tar options filename

• Example– tar –xvzf myTarfile.tgz– Decompress and restore (extract) all files in

the archive named myTarfile.tgz

14INFO 320 week 9


ntp

• The Network Time Protocol (ntp) coordinates time settings across your network

• It and its documentation (ntp-doc) are installed with apt-get or aptitude like most packages– Docs are likely under /usr/share/doc/ntp-doc/html/index.html

15INFO 320 week 9


ntp

• The ntp config file is /etc/ntp.conf• Start ntp with

– sudo /etc/init.d/ntp start

• Check status of ntp with– ntpq –p

• Or use an external ntp server with the ntpdate command (add to startup or cron)– sudo ntpdate ntp.ubuntu.com

16INFO 320 week 9


cron

• Addressed in more detail in connection with backups, cron allows you to schedule scripts to execute

• cron is started automatically from /etc/init.d on entering multi-user runlevels– crontab files can be in its spool area

(/var/spool/cron/crontabs) and more important (/etc/crontab) this is where you define cron jobs!

17INFO 320 week 9


cron

• cron logs its action to the syslog facility ’cron’, and logging may be controlled using the standard syslogd facility

• The heart of setting a service to run, is defining a crontab entry for it– You can specify when a file runs by minute,

hour, day of month, month, or day of week– Each of those five parameters correspond to

numbers or *s at the start of each entry18INFO 320 week 9


cron

• # use /bin/sh to run commands, no matter what /etc/passwd says

• SHELL=/bin/sh • # mail any output to ‘[email protected]’, no matter

whose crontab this is • [email protected] • # run five minutes after midnight, every day • 5 0 * * * $HOME/bin/daily.job >> $HOME/tmp/out 2>&1

19INFO 320 week 9


cron

• # • # run at 2:15pm on the first of every month -- output

mailed to bruce (above) • 15 14 1 * * $HOME/bin/monthly • 23 0-23/2 * * * echo "run 23 minutes after midn, 2am,

4am ..., everyday" • 5 4 * * sun echo "run at 5 after 4 every sunday"

20INFO 320 week 9


Editors

• Before there were word processors there were programs called editors

• Like a word processor, an editor is used to create text files and manipulate them

• The original use for a text editor was to create program files and modify them– A venerable text editor, designed for Lisp

programming, is Emacs

21INFO 320 week 9


Emacs

• Not developed in the UNIX environment– Therefore – not a special purpose utility

• More than a text editor– Built-in programming language, Lisp– Designed to do everything

• Emacs is a language sensitive editor with specific features for editing programming languages

22INFO 320 week 9


The big pattern!

• Notice the pattern for most Unix applications outside of the kernel– App names are all lower case, e.g. appname– Most apps can be installed with apt-get or aptitude, e.g. •sudo apt-get install appname

– Every app has a man page; man appname– Most apps have a text configuration file, often

something like /etc/appname.conf

23INFO 320 week 9


The big pattern!

– Many apps can be started with sudo /etc/init.d/appname start

– If the app needs a spool directory, it’s under /var/spool/appname

– The daemon for an app is usually appnamed – If there’s separate documentation for the app,

it’s probably under /usr/share/doc/appname

• See? UNIX is easy!

24INFO 320 week 9


RAID

25INFO 320 week 9

Mostly from (Bytepile, 2009) and (Rankin, 2009)Above image from www.fordcortina.net/pix/raid.jpg


RAID

• RAID (was Redundant Array of Inexpensive Disks, now often Independent) was developed circa 1988 by Patterson, Gibson & Katz; Berkeley CA– A RAID array is a collection of drives which

collectively act as a single storage system, which can tolerate the failure of a drive without losing data, and which can operate independently of each other.

26INFO 320 week 9


RAID

• RAID is defined by various levels a set of disks (a RAID array) is designed to achieve – RAID 0, RAID 5, etc.

• They differ mainly in terms of – Cost – Speed– Redundancy, and efficiency to do so– How many disks are needed at a minimum

27INFO 320 week 9


RAID

• RAID can be implemented in software alone, or with dedicated hardware (e.g. cards or built into a motherboard)– Hardware-based RAID is faster and more

expensive

• RAID typically requires all drives in an array to be identical (same make and model, not just same size and capacity)

28INFO 320 week 9


RAID 0

• RAID 0 is known as striping (watch spelling, it’s not stripping!)

• It offers no redundancy, but improves performance (speed) by reading alternately between disks– No speed improvement for writing data!

• Two or more drives can do RAID 0

29INFO 320 week 9


RAID 1

• RAID 1 is called mirroring

• It does nothing for performance, but provides redundancy by maintaining a duplicate copy of data– Hence there is 100% duplication of data, the

least efficient possible RAID– Using at least two disks in matched pairs, it

simply makes two copies of everything on separate disks

30INFO 320 week 9


RAID 0+1

• So if we do both mirroring and striping at once, we have RAID 0+1– RAID 0+1 is implemented as a mirrored array

whose segments are RAID 0 (striped) arrays

• Hence we need at least four disks, and get the read speed boost of striping, and the redundancy of mirroring

31INFO 320 week 9


RAID 3

• RAID 3 does striping, plus adds a parity drive; 3 drive minimum– The last drive has a parity bit from all of the

other drives, used to reconstruct one drive if it fails

• Very high read and write speeds, high efficiency, good for multimedia data, but expensive and complex

32INFO 320 week 9


RAID 5

• RAID 5 is similar to RAID 3, but the parity information is distributed across all the drives; 3 drive minimum– Hence if any one drive fails, it can be

recreated from the other drives

• High read rate and efficiency, medium write rate– Very commonly used for dB, web, file, and

other servers33INFO 320 week 9


RAID 10

• Notice that RAID 10 is not the same as RAID 0+1

• RAID 10 is a striped array whose segments are RAID 1 arrays (mirrors) instead of single drives– Minimum of 4 disks needed– Good when you need higher performance

than RAID 1; but very expensive

34INFO 320 week 9


RAID 10

• RAID 10 can handle multiple drive failures, as long as you don’t lose both halves of a mirrored pair

35INFO 320 week 9


RAID 50

• RAID 50 or 05 is striping two sets of RAID 5 arrays– Minimum of six

disks!– Better redundancy,

faster reads

36INFO 320 week 9


RAID and Ubuntu

• A RAID 5 array can not be used as the boot partition in Ubuntu – GRUB can’t read it– A boot partition can be RAID 1

• RAID is configured during the partitioning process of installation– Each partition is formatted to type ‘K raid’, and

they are joined into an array as a multidisk (MD) device, then partitioned to /, /boot, swap

37INFO 320 week 9


RAID and Ubuntu

• The main tools for administering RAID in Linux are /proc/mdstat and mdadm

• The former is a file with your current arrays– cat /proc/mdstat

• Linux recognizes RAID 0, 1, 4, 5, 6 and 10– This is software-only RAID

38INFO 320 week 9


RAID Capacity

• The usable capacity of common RAID levels is– RAID 0 = # Drives x Cap Drive– RAID 1 = (# Drives / 2) x Cap Drive– RAID 3 or 5 = (# Drives -1) x Cap Drive

• RAID 3 and 5 ‘lose’ one drive’s capacity for parity information from the other drives, so larger arrays are more efficient

39INFO 320 week 9

From http://www.kintronics.com/raidwpaper.htm


Other RAIDs

• There are other RAID approaches (RAID 2, 4, 6, 7, 53, 1E, etc.) but combinations of RAID 0, 1, and 5 are most common

• Other storage technologies include– JBOD– DAS vs. NAS – SAN

40INFO 320 week 9


JBOD

• JBOD is Just a Bunch Of Disks– RAID systems require identical drives– JBOD allows combining physical drives into

large logical volumes without requiring identical drives

– Cheap, easy, but no redundancy

41INFO 320 week 9


DAS vs. NAS

• DAS is direct-attached storage, a fancy name for disks which are part of a server

• NAS (network-attached storage) is 100% dedicated to serving files over a network– It’s one or more hard drives, with enough

computer attached to let it communicate directly over the network

– Simple and cheap

42INFO 320 week 9


SAN

• A Storage Area Network is a dedicated, high performance storage network – Transfers data among servers and storage

devices, separate from the local area network– Used when high speed large transfers are

common, often using fiber optic connections– Often does load balancing and has high

reliability– Often used for transaction processing

43INFO 320 week 9


SAN

44INFO 320 week 9

From http://www.rad-direct.com/Product-Whitepaper-IP-SAN-3.htm


Hot-swappable

• Finally, storage systems can have hot-swappable drives or not– Hot-swappable means they can be safely

removed from the system while it’s running

• Most RAID systems are hot-swappable, so that failed drives can be removed without reducing availability

45INFO 320 week 9


References

• Bytepile, 2009. “RAID Types - Classifications ” from http://bytepile.com/raid_class.php

• The RAID Advisory Board (RAB) does not seem to exist any longer

46INFO 320 week 9

www.ischool.drexel.edu info 320 server technology i week 9 unix/linux tools 1info 320 week 9

Documents

tar file

file diff output

file areport

diff lpr

device file

config file

colorsa file match colorsb

diff utility