CIS 620 Advanced Operating

Systems

Lecture 8 – Naming and File Systems

Prof. Timothy Arndt

BU 331

Names, Identifiers, And Addresses

• Naming was originally (and still is primarily, to some extent, today) applied to file systems Nowadays other entities are organized into

namespaces• Processes, IPCs, etc.

• Properties of a true identifier:• An identifier refers to at most one entity.• Each entity is referred to by at most one identifier.• An identifier always refers to the same entity

Flat Naming

• Non-hierarchical Can locate an entity using flat naming by

broadcasting in a LAN (e.g. ARP) In a point-to-point network (WAN) can use

multicasting What about flat naming for mobile entities?

• Can use the forwarding pointers approach

Forwarding Pointers

• The principle of forwarding pointers using (client stub, server stub) pairs.

Forwarding Pointers

• Redirecting a forwarding pointer by storing a shortcut in a client stub.

Forwarding Pointers

• Redirecting a forwarding pointer by storing a shortcut in a client stub.

Home-Based Approaches

• The principle of Mobile IP. A home location keeps track of the current location an entity

Distributed Hash Tables

• Mapping a name to a location in a DHT can take a long time if we take the naïve approach (just try each host around the logical ring until we find the name) Instead, we can use a finger table which

records locations of entities at logarithmically increasing distances

Distributed Hash TablesGeneral Mechanism

• Resolving key 26 from node 1 and key 12 from node 28 in a Chord system.

Name Spaces

An alternative to flat naming is structured naming leading to a name space

A general naming graph with a single root node.

Linking and Mounting

• The concept of a symbolic link explained in a naming graph.

Linking and Mounting

• Information required to mount a foreign name space in a distributed system• The name of an access protocol.• The name of the server.• The name of the mounting point in the foreign

name space.

Linking and Mounting

• Mounting remote name spaces through a specific access protocol.

Name Space Distribution

• An example partitioning of the DNS name space, including Internet-accessible files, into three layers.

Name Space Distribution

• A comparison between name servers for implementing nodes from a large-scale name space partitioned into a global layer, an administrational layer, and a managerial layer.

Name Resolution

• Discovering the location of a resource, given its name, is called name resolution Name resolution in hierarchical systems like

DNS uses iteration, recursion (and caching)

Implementation of Name Resolution

• The principle of iterative name resolution.

Implementation of Name Resolution

• The principle of recursive name resolution.

Implementation of Name Resolution

• Recursive name resolution of <nl, vu, cs, ftp>. Name servers cache intermediate results for subsequent lookups.

Example: The Domain Name System

• The comparison between recursive and iterative name resolution with respect to communication costs.

The DNS Name Space

• The most important types of resource records forming the contents of nodes in the DNS name space.

DNS Implementation

• An excerpt from the DNS database for the zone cs.vu.nl.

DNS Implementation

• An excerpt from the DNS database for the zone cs.vu.nl.

DNS Implementation

• Part of the description for the vu.nl domain which contains the cs.vu.nl domain.

The File System in UNIX

The UNIX file system is organized in a tree structure. The files on this tree are one of the following types:• An ordinary file• A directory file• A special file (representing an I/O device)• A link that points to another file• A socket that is used for interprocess

communication

Directories

A directory is a file whose content consists of directory entries for the file in the directory.• A directory entry contains the name of the file and a

pointer to the file. The pointer is an index called the i-number to a table known as the i-list.

• Each entry in the i-list is an i-node containing status and address information about a file or is free.

• The entire file system may contain several self-contained filesystems, each with its own i-list.

Directories

grail:/users/faculty/arndt> df/home0 (arthur.cba:/home0 ): 14736728 blocks -1

i-nodes/home1 (arthur.cba:/home1 ): 7846106 blocks -1

i-nodes/userspace (shamu.cba:/export/home0/userspace): 1327190 blocks -1 i-nodes/usr (/dev/vg00/lvol3 ): 65700 blocks 13437

i-nodes/tmp (/dev/vg00/lvol4 ): 219084 blocks 12016

i-nodes/users (/dev/vg00/lvol5 ): 245610 blocks 65662

i-nodes/mnt1 (/dev/vg00/lvol6 ): 129708 blocks 17093

i-nodes/client_server (/dev/dsk/c4d0s2 ): 512710 blocks 357634

i-nodes/ (/dev/vg00/lvol1 ): 23880 blocks 6769

i-nodes

grail:/users/faculty/arndt>

Special Files

UNIX represents I/O devices such as terminals, printers, tape, and disk drives as special files in the file system.• In this way, an application program can treat file and

device I/O in the same way.• Each special file is stored in the directory /dev.• A special file can be either a character special file or

a block special file. A character special file represents a character-oriented I/O

device. A block special represents a high-speed I/O device that

transfers data in blocks rather than bytes.

Links

A directory entry may be a pointer to another file. This is called a link. There are two kinds of links:• hard links• symbolic links

A hard link is an entry in a directory with a name and some other file’s i-number. The hard link is not distinguishable from the original file.• A file may have several links to it. The link must be

to an ordinary file in the same filesystem.

Links

A link is created to another file by using the ln command:ln fileln file linknameln file1 . . . Dir

By default, a hard link is created by ln. • It is not necessary to be the owner of a file to link to

the file. The command rm removes the directory entry of the file. The file is physically deleted when the last link to it is removed.

• The number of hard links to a file is kept as part of the status.

Symbolic Links

A symbolic link is a directory entry that contains the pathname of another file. The symbolic link can be used as an argument in a command.

The file pointed to can be removed and the link will remain.

A symbolic link can span filesystems and may point to a directory.

A symbolic link can be created using ln -s

File Access Control

• The file mode for a file as displayed by ls -l is, for example:-rw-rw-rw- The first position shows the file type: - for

normal, d for directory, l for symbolic link, c for character special file, b for block special file.

Positions 2-4 show read, write, and execute permission for the owner of the file. A - means no permission.

File Access Control

Positions 5-7 show the permissions for users in the file group.

Positions 8-10 show the permissions for other users.• To access a directory, you must have execute

permission for that directory. To list its contents, you must have read permission. Write permission is necessary to create or delete a file in a directory

• New files are created with a default protection. To change this default, use the command umask. This command uses three octal numbers.

File Status

A file status is kept in the i-node for each file in the UNIX files system. The file status include:

mode number of links owner, group size last access, last content change, last status change i-number device block size block count

File Mode

The file mode consists of 16 bits. We have already seen the low order 9 bits: the permissions. The four high bits specify the file type while the next three bits define the way in which an executable file is run.• Set userid: owner of a file changed to user during

execution.• Set groupid: sets group id during execution.• The stickybit keeps a shared binary in memory.

The mode of a file may be changed using the chmod command

File Mode

grail:/usr/bin> ls -l e* | more-r-xr-xr-t 6 bin bin 245760 Sep 7 1993 e-r-xr-xr-t 6 bin bin 245760 Sep 7 1993 edit-rwxr-sr-x 1 root bin 479232 Jun 17 1996 elm-r-xr-sr-x 1 bin mail 270336 Sep 7 1993 elm.orig-r-xr-xr-x 1 bin bin 32768 Jun 17 1996 elmalias-r-sr-xr-x 1 lp bin 20480 Oct 22 1993 enable-r-xr-xr-t 6 bin bin 245760 Sep 7 1993 ex-r-xr-xr-x 1 bin bin 16384 Sep 7 1993 expandgrail:/usr/bin>

Note the ‘s’ and ‘t’ above in place of ‘x’.

Groups

• A user can belong to one or more groups specified in the file /etc/group.

• Use the command ‘groups’ to see which groups you belong to.

• Each file is associated with a group owner. To change the group owner of a file use ‘chgrp’ This allows groups to work on the same files

(but it doesn’t coordinate their changes!).

File System Implementation

A UNIX file is an array of bytes stored in a number of data blocks in a specific filesystem.

Originally, blocks were 512 bytes. Some implementations now use 4K bytes per block. Since the final block may waste space if it is not full, it may be broken up into fragments of (e.g.) 1K bytes.

As said previously, information on files is stored in an i-node, where the i-nodes for a filesystem are stored in an i-list.

File System Implementation

The i-node also contains pointers to the data blocks of the file. An i-node contains 10 pointers to data blocks, a pointer to a data block containing pointers to data blocks, and a pointer to a data block containing pointers to data blocks containing pointers to data blocks (!). This allows for extremely large files.

Why use this arrangement rather than just a chain of data blocks?

Removable Filesystems

• One of the UNIX filesystems is the root filesystems. All of the other filesystems are removable. Removable filesystems are attached to the root

filesystem at some leaf node using the mount command: mount devfile directory [-r]

The command umount is used to unmount a removable filesystem.

Mount as read only

Super Block and Cylinder Groups

The super block of a filesystem records information about the filesystem:• Size of the filesystem• Block/fragment size• Length of the i-node list• A list of free blocks and length of the list• A list of free i-nodes and length of the list

The super block is stored at the beginning of a filesystem.

Super Block and Cylinder Groups

Contemporary UNIX system use an improved organization based on the idea of a cylinder group (a set of physically close cylinders on the disk).

I-nodes are distributed within each cylinder group rather than at the beginning of the filesystem. Therefore, i-nodes are close to their data blocks, and access time is improved. Super blocks are also duplicated in order to increase robustness.

Filesystem Table

Each filesystem is represented by its own block-type special file. The name of this file along with other information is kept in /etc/checklist or /etc/fstab:

/dev/vg00/lvol1 / hfs defaults 0 1 31484 564/dev/vg00/lvol2 swap ignore sw 0 0 16392 592/dev/vg00/lvol3 /usr hfs defaults 0 2 0/dev/vg00/lvol4 /tmp hfs defaults 0 2 13 16408/dev/vg00/lvol5 /users hfs quota,defaults 0 2 0/dev/vg00/lvol6 /mnt1 hfs rw,suid 0 3 0/dev/dsk/c4d0s2 /client_server hfs quota,defaults 0 2 0arthur.cba:/home0 /home0 nfs rw 0 2 0arthur.cba:/home1 /home1 nfs rw 0 2 0shamu.cba:/export/home0/userspace /userspace nfs rw 0 2 0

Filesystem Table

The currently mounted tables are kept in /etc/mnttab:

/dev/vg00/lvol1 / hfs defaults 0 1 917804700 1/dev/dsk/c4d0s2 /client_server hfs quota 0 2 917804709 1/dev/vg00/lvol6 /mnt1 hfs defaults 0 3 917804709 1/dev/vg00/lvol5 /users hfs quota 0 2 917804709 1/dev/vg00/lvol4 /tmp hfs defaults 0 2 917804709 1/dev/vg00/lvol3 /usr hfs defaults 0 2 917804709 1shamu.cba:/export/home0/userspace /userspace nfs rw 0 2

917804764 0arthur.cba:/home1 /home1 nfs rw 0 2 917804764 0arthur.cba:/home0 /home0 nfs rw 0 2 917804764 0

Network File System

Many UNIX systems allow file operations on remote filesystems stored on other computers on a LAN. This is done using NFS.

A machine on a LAN can function as file server (e.g. by having all users home directories on that machine).• NFS can work on different hardware and OSs (e.g.

different varieties of UNIX). A filesystem is made available remotely by

exporting it. It can then be mounted on a remote machine.

Linux File Types

• Besides regular files (-), directory files (d), block special files (b) and character special files (s) Linux also has the following file types Unix domain socket (s) Soft link (l) Named pipe (p)

File System Consistency

In order to improve filesystem performance, a pool of data blocks are kept in memory so that further references to them don’t require a disk access.

Changes made to these blocks are not immediately reflected on disk.

Periodically, they are written to disk. This arrangement is known as lazy write.

• DOS used careful write in which changes are written immediately to disk.

Powering down the computer without going through a shutdown sequence can cause unwritten data to be lost.

File System Consistency

The program fsck can be run when the system is rebooted to check for and repair file system inconsistency (compare Windows chkdisk program).

This loss of data is unacceptable for mission-critical database management systems.• Therefore these systems completely bypass the

UNIX filesystem, managing a raw device by themselves in order to ensure a consistent state of the database in case of system failure.

Problems

• The user types in the command “cat /user/home/me/sub/file1” - what are the steps involved in retrieving the data in the file?

• A removable hard drive formatted for Unix (i.e. Firewire or USB) is attached to a Unix system. What steps are taken in order to make the data on the drive available to application programs?

procfs

• procfs (the proc filesystem) is a special filesystem in Linux and other Unix-like OSs that presents information about processes and other kernel information in a a hierarchical file-like structure Makes it easier to access info about (e.g.)

processes Mounted at /proc

Windows Filesystems

Windows NT supported multiple file system types:• FAT (for compatibility with non-NT systems)• HPFS (the native OS/2 file system)• NTFS (the native NT file system type)

NTFS incorporates the normal features of other modern file systems.• Journaling• Enhanced file security• Lazy-write scheme• Mirrored master file table and boot sector

Journaling

Like AIX, NTFS uses techniques borrowed from real-time transaction processing to ensure file system structure integrity at all times.

Before any change is made to the file system, it is first recorded in a designated log file.

During a system boot, the file system structure is automatically made current with its log.

This doesn’t prevent data loss since only changes to the structure of the file system are logged, not changes to a file’s contents.

Lazy-write scheme

As in Linux, disk operations are performed to a cache in memory rather than directly to disk.

• This optimizes system I/O performance, but may result in file changes being lost in case of a power failure.

Windows also uses the lazy-write scheme on FAT file systems, unlike DOS.

The journaling technique allow for a quicker recovery than other lazy-write systems - chkdsk is not required to scan the disk in case of crash.

Master File Table

The equivalent of the i-list in NTFS is the Master File Table (MFT)• The first 16 records in the MFT contains metadata

files used to implement the file system: Master File Table: a list of all contents of the NTFS

volume MFT2: a mirror of the first three records of the MFT, used

to guarantee access to the MFT in case of a single-sector error

Log File: a list of transaction steps used for recoverability. Volume: the volume name, NTFS version, and other info. Attribute Def. Table: A table of attribute names, numbers,

and descriptions

File System Consistency

Root Filename Index: Root folder Cluster Bitmap: A representation of the clusters showing

which clusters (i.e. blocks) are in use. Partition Boot Sector: the bootstrap for the volume, if

bootable. Bad Cluster File: List of bad clusters. Quota table: Disk quota usage for each user on a volume.

Currently unused. Upcase Table: Used for converting lowercase characters to

the matching Unicode uppercase characters. The next 5 MFT records are reserved for future use.

MFT Records

A file in an NTFS file system is a collection of attributes stored in an MFT record (and possibly elsewhere on the disk). Attributes include the following:

Basic file information, such as the file’s DOS attributes, modification and access time, and number of hard links.

Filename: NTFS filenames may be up to 255 characters in length and may contain any letters, number or Unicode characters. A DOS-style filename may also be given.

Security attributes: the Windows access permissions. Data attribute: the data in the file. Additional indexing data for directories.

Advanced Filesystem Features

NTFS has optional transparent file compression.

On Windows, all local file systems are automatically mounted when the the system boots.

File system volumes (as defined with a unique drive letter) can occupy more than one disk partition.

Striped file systems may be created for higher I/O performance.

Advanced Filesystem Features

RAID filesystems are also available. RAID (Redundant Array of Inexpensive

Disks) is a set of six specific disk organizations that can be used to improve disk performance, or reliability, or both.• Most RAIDs consist of a RAID SCSI controller plus

a box of SCSI disks that appear to the OS as a single large disk. No software changes are required to use the RAID.

RAID

The data in a RAID are distributed over the drives, to allow parallel operation. Several schemes were devised by Patterson and are known as RAID level 0 through RAID level 5.

RAID level 0 stripes data over multiple drives in round robin fashion. In this way, a block of data can be read from multiple drives, working in parallel. This works best when data is requested in large chunks. The controller must coordinate the request and issue the individual requests to the disks.

RAID

RAID level 0 is less robust than using a single disk, since the failure of any disk leads to failure for the RAID.

RAID level 1 is a true RAID. It duplicates all the disks so that there are four primary and four backup disks. On a write, a strip is written twice. On a read, either copy can be read. Fault tolerance is excellent.

RAID

RAID levels 4 and 5 work with strips of data, with parity. RAID level 4 writes a parity strip (computed from the strips on each of the other disks) on an extra drive. If a drive crashes, the lost bytes can be recomputed from the parity drive.

RAID level 5 distributes the parity bits uniformly over all the disks so that the parity drive does not become a bottleneck. This makes recovery more complex, however. Level 5 RAID is supported on Windows Server only.

RAID

Other RAID levels have been defined as well RAID 6

• Identical to RAID 5, except it has a 2nd (distributed) parity block.

RAID 10• Contains a top-level RAID-0 array (or stripe set)

composed of two or more RAID-1 arrays (or mirrors).

Removable Disks

• FAT and NTFS file systems may be created on removable disks (USB external drives, USB flash drives, etc.).

• Such disks are removable in the normal manner when the FAT file system is used.

• For disks formatted with the NTFS system, must use ‘Safely Remove Hardware’ since write caching is used.

Network Filesystem Access

Files and directories can be shared across a network using three mechanisms:• Windows shares: the native facility for sharing disk

resources within a Microsoft network.• The Distributed File System: similar to NFS,

allows disk resources of different Windows computers to be combined into a single, network-accessible directory tree.

• NFS and SAMBA: third party resources allow the sharing of file systems resources between NT and UNIX systems.

File Types

There are several different NTFS file types:• Regular files• Directories• Hard links• Shortcuts (similar to symbolic links, but used in the

GUI environment)

Note that there are fewer file types than in Unix since I/O devices, sockets are implemented differently.

File Ownership and Permissions

• FAT file systems offer virtually nothing in the way of file and directory access controls. NTFS on the other hand, offers sophisticated controls. Every file in an NTFS file system has an owner

associated with it. There is no group owner. Access control lists (ACLs) are used to grant

and limit access to files and directories. ACLs are composed of name-permission pairs

known as access control entries (ACEs).

File Ownership and Permissions

Windows uses the following basic permissions: Read (R), Write (W), Execute (X), Delete (D), Change Permissions (P), Take Ownership (O). There are also named commonly used subsets of these permissions.

It is easier to assign permissions to groups, rather than individual users.• ACLs can be viewed using the security tab of the

file’s Properties dialog box.

Attribute-Based Naming

• In some applications it is useful to be able to lookup entities in terms of (attribute, value) pairs This is called attribute-based naming Implemented by directory services

• Full database approach is too costly• A popular hierarchical implementation is LDAP

Based on the OSI X.500 directory service

Hierarchical Implementations: LDAP

A simple example of an LDAP directory entry using LDAP naming conventions.

Hierarchical Implementations: LDAP

• Part of a directory information tree.

Hierarchical Implementations: LDAP

• Two directory entries having Host_Name as RDN.

Decentralized Approach

• We can also implement directory services in a P2P network, resulting in a decentralized implementation Each entity is described by a hierarchically

organized set of attributes known as an AVTree Hash each path in each tree and assign to DHT

node Query represented by AVTree

• Compute the hashes – send to nodes

Mapping to Distributed Hash Tables

• A general description of a resource. Its representation as an AVTree.

Mapping to Distributed Hash Tables

• The resource description of a query. Its representation as an AVTree.

Semantic Overlay Networks

• Maintaining a semantic overlay through gossiping.