Bonus Material B

File and Operating Systems

The two main software components of any computer platform are its operating system and its file system. In the beginning, computer systems had neither. Each entry into the processor was a program or a routine that informed every element of the computer what to do; each time the program wanted the element of the computer to do it. Given there were no graphical user interfaces, no keyboards, and not much more than a few toggle switches; the need for an operating system (OS) or a file system (FS) was not even apparent.

Today that has all changed. The OS is now the backbone of a computer, managing all of the software and hardware resources. Operating systems are responsible for everything from the control and allocation of memory to recognizing input from external devices and transmitting output to others. The OS also manages the files on physical memory, control systems, and peripherals such as printers and scanners.

We know the file system (sometimes written as one word "filesystem") mostly by the way we name files or by the way they are placed logically so we can retrieve them from disks, USB sticks, or from a network. File systems form that intermediate layer between block-oriented disk drives and the applications that remain on that drive.

KEY CHAPTER POINTS

Operating system and file system types that include conventional, time-sharing, real-time, embedded OSs, and mobile

Discussion of services, managers, and the layers of an operating system File system categories and types including local, distributed, clustered, and those

familiar to most PC users, as well as those not so familiar Flash, journaling, and traditional file systems and their applications to mobile and

media operations and users

Operating Systems

Most of us know of operating systems by their commercial names, e.g., Mac OS or Windows 7. Fundamentally, an operating system (OS) is a collection of software routines designed to run other programs on a computer. The software of an operating system controls, organizes, and accounts for the hardware blocks and turns those elements into a cohesive entity with powerful functionality.

The tasks that an OS must be capable of handling include processor management, memory and storage management, and device management. The OS provides for a common application interface (API) and a common user interface (UI). Operating systems, based upon the kind of applications they support, can be described in four overall dimensions.

Single user/single task—managing of the computer tasks for single users that are doing only one thing at a time

Single user/multitask—the more common place computer in the home, workplace, or laptop, such a fully-fledged "Windows-like" OS

Multiuser—a balanced system that can handle many users and in turn many tasks simultaneously, such as in the Unix OS

Real-time operating system (RTOS)—tasks that must be handled (executed) with precise timing, every time

OS Infancy

The first-generation electronic digital computing systems of 1940s had no operating systems. In that era, programs were entered in the form of one bit at a time on rows of mechanical switches physically mounted on “plug boards.” No assembly languages or programming languages were known, and operating systems were unheard of.

By the early 1950s, the earliest implementation of any form of an operating system emerged. The second generation of computers, those that included their only I/O on punch cards, began to see the introduction of a crude operating system. General Motors Research Laboratories implemented a first-generation operating system in early 1950s for their IBM 701, the first production computer from IBM and labeled as Defense Calculator. Systems of this generation generally processed one job at a time, called a single-stream batch processing system, because programs and data were submitted in groups or batches.

Third-generation computing systems of the 1960s took advantage of the computer's resources and processed (ran) several jobs simultaneously. Still referred to as batch processing systems, the designers needed to develop a concept for multiprogramming whereby several jobs could be held in main memory, and the processor would then switch from job to job as needed in order to advance multiple jobs, keeping the peripheral devices active. The operating system kept the CPU from sitting idle as each I/O process was in progress. This was accomplished by partitioning the host memory systems so that each job could process in each separate partition. The result was that wait cycles were reduced, and processing could continue as I/O processes were executed.

These third-generation computing systems employed an operating system that allowed for new features such as spooling which permitted simultaneous peripheral operations to occur concurrently or sequentially, and online. Spooling is essentially a buffering technique that matches peripheral I/O operations to the computer memory and processing functions. In this operation, a high-speed device such as a disk drive is interposed between a running program and a low-speed device involved with the program during an input/output operation. Instead of the program writing directly to a printer, the output data is written to a disk drive where it is spooled out to the printer when and as the printer is available to handle that data stream. Programs can be run to completion faster, and other programs can be initiated sooner. When the printer becomes available, the output data may be sent to that device and then printed.

This generation of computing systems also employed a variant of the multiprogramming technique called time-sharing in which each user has a directly connected (online) terminal console. In this new mode, which differs from the batching techniques, users are interacting with the computer in near real time. This operational mode requires that the computer system responds quickly to user requests to avoid a decrease in productivity. Time-sharing systems were developed to support the multiprogram mode whereby large numbers of simultaneous interactive users utilized the central processing units concurrently.

Operating systems entered the mainstream world of fourth-generation computer systems with

the introduction of the microprocessor (MPU), large scale integrated (LSI) circuits, and complex peripheral chip sets that spawned the evolution of the personal computer and the workstation age. The most noted of the modern operating systems, the one that dominated the personal computer scene, is MS-DOS (Microsoft Disk Operating System), written by Microsoft, Inc. for the IBM PC and those machines employing the Intel 80xx CPU and its successors, and UNIX, originally developed in 1969 by a group of AT&T employees at Bell Labs which became most prevalent on larger personal computers, servers, and workstations using the Motorola 6899 CPU family.

A Time-Sharing Operating System

One of the earliest operating systems, Multics (Multiplexed Information and Computing Service), became an influential guide to what would be referred to as a time-sharing operating system. The project, started in 1964, was originally a cooperative project led by MIT (by Fernando Corbató) along with General Electric and Bell Labs. Bell Labs dropped out the project in 1969, and in 1970 GE's computer business including Multics was taken over by Honeywell.

Multics implemented a single-level store for data access. This concept discarded the clear distinction between files (called segments in Multics) and process memory. The memory of a process consisted solely of segments (files) which were mapped into its address space. To read or write to them, the process simply used normal CPU instructions, and the operating system then managed the services to be sure that all the modifications were saved to disk.

Familiar Types

Besides being just a program that controls the execution of application programs and acts as the interface between the user of a computer, the OS is that interface between the computer hardware, including storage, memory, and peripherals, and the user. The user may be a human or another computer or a device that depends on the functions executed by a computer-like device.

The more notable operating systems are provided by those software vendors that have influenced and guided the development of the modern computer society. Recognized computer operating systems include the following:

UNIX (the OS whose source code and product implementation went to SCO and whose trademark UNIXWARE is owned by The Open Group)

Mach (the operating system kernel developed at Carnegie Mellon University derived from the Mac OS)

MS-DOS, MS-Windows, and Windows/NT (Microsoft Corporation's various OS platforms)

OS/2 and Warp (IBM's endeavor for the PS-2 era) Mac OS (Apple's "system software") OpenVMS (for VAX and AXP-Alpha CPUs) FreeVMS (Virtual Memory System) MVS (Multiple Virtual Storage) VM (Virtual Machine), as in the software implementation of machine that executes

programs as though it was a physical machine

Needless to say, the term operating system or simply OS is associated with many a different set of software, hardware, and even virtual machines. The OS, by itself, can be confusing even though it essentially remains in the background quietly performing the tasks that most users want nothing to know about.

Control

Controlling the computer involves software at several levels. In the same fashion, there are components of the operation system that must be differentiated, even though most of them are software based in nature. Of those elements that control the computer, it is the OS that is made up of kernel services, library services, and application-level services.

The core of the operating system is its kernel. A kernel is a control program that functions in privileged state, which is an execution context that allows all the hardware instructions to be executed. The kernel reacts and responds to interrupt from external devices and to service requests and traps from processes. The operating system uses processes to run applications, which are linked together with libraries to perform standard services. It is the kernel that supports the processes by providing a path to the peripheral devices.

In most cases, the kernel is a permanent resident of the computer which creates and terminates processes and then responds to their request for service. Operating systems are the resource managers whose main resource is the computer hardware in the form of processors, storage, input/output devices, communication devices, and data.

OS Functions

The operating system enables and manages many functions which are essential to the successful "operations" of a computer system. These functions include the user interface, the sharing of hardware among users, the sharing of data among the systems in the computer, and the prevention of interference among those systems.

The OS further facilitates the I/O functions, the recovery from errors, and the accounting of resources and usages. Its tasks facilitate parallel operations and prioritization, the organization and securing of the software and data, and the handling of network communications internally (on the buses of the computer) and externally over public or private connections.

At a deeper level, the OS is responsible for the allocation of each resource. This responsibility extends to memory management of the hardware and of the processor. The OS determines when an I/O device can be used by the applications or programs running on the computer. It controls access to and the use of the files, although a great deal of that is managed independently by the "file system" manager.

At a critical level, the OS determines just how much processor time is devoted to the execution of programs, services, and other functions which occur internally and peripherally to the core processor or processors. In a failover state, it is the OS (in conjunction with the records kept by the file system) that picks up all the pieces and quickly resumes its business.

Resource Manager

The OS is a resource manager that handles the movement, storage, and processing of data including the control of those functions. The OS functions in a similar way to that of ordinary software except the program(s) is executed by the processors. An operating system will frequently relinquish its control at which time it then depends upon the processor to recover that control.

The operating system's job is to direct the processor in the use of the system resources. In order for the processor to do accomplish its tasks, it must temporarily cease the execution of the operating system code and execute other programs. How this situation is managed is dependent upon the functionality of the core processor, and the relationship that it has at each level of program execution.

Operational Layers

File systems and volume managers are generic applications; that is, their services and the performance of those services are not necessarily optimized for any specific application. However, in media specific storage systems, both may be tailored to the kinds of specialized applications that are typical to move larger, contiguous files with specific characteristics.

When applied to a more generic set of applications, such as databases or transactional data that is continually in flux, the volume manager and file systems must be configured for a variety of services and data sets. The performance of the file system and the volume manager are characteristic parameters from which a user may choose one storage platform over another.

Volume Manager

Within the operating system is an intermediate layer located between the file system and standard small computer systems interface (SCSI) device drivers. It is the system whose basic function is to aggregate multiple hard disk drives (HDD) to form a large virtual disk drive that is visible only to the higher levels of the operating system.

A file system, which is discussed in detail during the second part of this chapter, in the traditional sense resides on single devices. When control of more than one device is necessary, a volume manager is employed. A volume manager can provide the same function as that of an intelligent disk subsystem or a RAID controller.

Embedded Operating Systems

An embedded system is an application-oriented special computer system. These systems are most often found in devices that are intrinsically designed to focus on a set of specific tasks, such as the control and instrumentation in an automobile, a GPS tracker, a mobile phone, or consumer electronics. Embedded systems must be scalable on both the software and hardware sides. As the applications that run on embedded systems get more complex, the need for an operating system and the support for its development become critical.

Linux Embedded

In most recent times, Linux has become quite popular on embedded systems. The modular nature of Linux and its fully functional OS allows it to be slimmed down through the removal

of unnecessary components, utilities, tools, and other software system services.

Windows CE and Windows Mobile

Another more formidable embedded OSs is Microsoft Windows CE, which was first introduced in the Handheld PC (H/PC) set of products in November 1996. Currently at Version 6, the developers at Microsoft are headstrong in completing the next iteration of what is still scheduled for Q1-2011; but has had several delays over the course of nearly 3 years of waiting. The OS is officially known as Windows Embedded Compact (or Windows Embedded CE post Version 6.0).

Windows Mobile is yet another operating system developed for smart phones and other mobile devices. And now, with Microsoft's latest endeavor, Windows Phone 7, it is based on the Windows Embedded Compact 7 core; but it is not clear what the underlying OS will be. Compact 7 brings Silverlight 3 for Windows Embedded, an embedded Internet Explorer, Adobe Flash 10.1 support, and has HD and MPEG-4 streaming support. A real-time operating system for “connected,” battery-powered and handheld devices is included in the tool set.

Real-Time Operating Systems

Functions or events that must be processed in an isochronous (time dependent) or instantaneous fashion are considered to occur in “real time.” The real-time operating system (RTOS) enables computing where system correctness not only depends on the correctness of logical result but also on the time of the delivery. Video and audio are the best examples of where the RTOS fits. Imagine that video to the home was randomly interrupted, lots of frames were skipped, or the audio was just placed wherever it felt like regardless of what the image was. Without a "live-time" RTOS, this is what users in the digital world would experience.

The RTOS should have features to support these critical requirements and must have predictable behavior to unpredictable external events. A true RTOS will be deterministic under any condition.

Hard and Soft RTOS

A real-time operating system that would not miss a deadline without catastrophic consequences is called a hard real-time system. Should the system be allowed to miss deadlines, and still recover, they are referred to as soft real-time systems. In actuality, the computer system does not require an operating system. A simple timer that uses a motor to advance a hub that activates a switch which opens or closes an electric circuit does not need an operating system. However, a timer that adjusts for daylight savings time that has a digital display, that knows the seasonal timing of sunrise and sunset, and that can be remotely called into from a cell phone or a computer does need an operating system, and it might need to be a real-time OS if it is coupled to life safety or mission critical applications.

Adaptability

A well-designed RTOS will allow for real-time applications to be easily designed and

expanded upon. Functionality should be added without a major set of changes to the software. The RTOS should allow for the splitting of application code into separate tasks that can be modularized without recompiling length strings of code sets. A preempty RTOS will handle all critical events in as quick and efficient time frame as possible.

OS Services

The operating system needs to be capable of providing a mix of services to the user, the applications, and the computer itself. These fundamental services need to include at least the following:

Editors and debuggers—a variety of facilities and service that assist the user, at almost a programmer's level, in the diagnosis of issues that might be affecting the application, memory, storage, or utilities.

Program execution—the tasks associated with the functional aspects of executing a program. The OS manages the instructions and data that must be loaded into main memory, I/O devices and files that must be initialized, and other resources that must be scheduled to perform.

Access to I/O devices—the specific instructions that control and make functional items such as data file exchanges between processor and disk storage, input from the keyboard, or output to a display.

Controlled access to files—dealing with the file format of the storage system and the interpretation of privileges or permissions necessary for users or devices that request access to data.

System access—accessibility to the system as a whole and to specific system resources. Access functions need to provide the protection for resources and data from unauthorized users. The OS must be able to resolve conflicts at the system level and at the I/O level and more.

Error detection and response—the ability to react and take a response that clears an error conditions with the least impact on running applications.

Accounting—the collection of usage statistics from various resources, the monitoring of performance parameters, the logging and classification of events, and the decision processes that can automatically improve overall system performance.

File Systems

A file system is generally created at the time the storage system is first initialized. The file system is necessary to create new files and folders (formerly called directories) which are used by applications and users to organize, categorize, and segregate various elements of the operating system, applications, and data. These folders are in turn managed by the file system.

The way in which we name the files, and where they are placed so that they can be logically retrieved is the underlying concept behind the file system. The most familiar files systems are those found in DOS, Windows, OS/2, Mac OS, and UNIX-based operating systems. Each of these OSs has file systems whereby files are hierarchically placed. Nearly all of these OSs use the "tree" structure, sometimes viewed as a "leaf and node" architecture. Files are placed in a directory (known as a folder in Windows and Mac OS) or subdirectory (subfolder in Windows) at the desired, hopefully logical place in the tree structure. The naming

conventions for folders, directories, subfolders, and subdirectories usually change depending upon the operating system.

File systems specify conventions for naming files. These conventions include the maximum number of characters in a name, which characters can be used, and in some systems, how long the file name suffix can be used. A file system also includes a format for specifying the path to a file through the structure of directories.

The Mac OS X (Version 10.5) uses the "home folder" as the location that stores the user's personal information, which is created for each user of the computer when that particular user's account is set up. The "home folder" is not typically named Home; but it usually is given the user’s short name as specified in the user account. The home folder is almost never changed once it is set and is usually subdivided into additional folders that make it easier and more logical to locate the various types of files (e.g., music, pictures, public or shared files, or sites).

The Windows OS (XT, Vista, and Windows 7) have a very similar architecture to the Mac OS; generally with the same concepts, but using slightly different names.

These elements are the basic components at the surface level of the file system; the parts that the users see the most and probably only ever need to directly interface with. Getting any deeper than the directory or folder level is more for the advanced user and becomes quite different for each operating system; therefore we will leave these discussions outthis book. However, there is a need to understand what is under the hood if you are a media professional or IT administrator who needs to mix or associate different file systems in order to set up storage or configure a cross platform architecture. The remainder of this chapter will cover the variances of some of the more known file systems.

File System Categories

Industry and users have different perceptions about what a file system is, and what it is perceived to do versus what it really does. For the purposes of this discussion, we’ll define a file system as that entity in a computer system that stores named data sets and attributes about those data sets so that they can be subsequently accessed and interpreted according to the needs of that system.

The attributes stored in the named data sets include data describing the originator and ownership, the access rights and date of the last access, and the physical location of those data sets. Like descriptive metadata, there may be other optional or advanced attributes that might include textual descriptions of the data, any migration or performance policies, and encryption keys.

It can be quite useful to break file systems into a group of categories. Examples include the following:

Local file systems—providing for the naming, data access, and attribute interpretation for only the services they provide and only for the local system they run.

Shared file systems—a means of sharing data between more than one system. Familiar examples include NFS (network file system) and CIFS (common Internet

file system). Most shared file systems will require that a local file supports the file server or network attached storage (NAS) appliance.

Special file systems—those with properties specific to applications or uses, but tend to store data in a conventional sense. An example is "tmpfs," which stands for "temporary file storage facility" (as in Unix or Unix-like OSs). Here the file system appears like a mounted file system, but the data is stored in volatile memory rather than persistent storage, such as a disk drive. RAM disks, which look like virtual disk drives but host a disk file system, could be a tmpf.

Media-specific file systems—those associated to the media on which they are typically found. This is not media, as in images or sound, but the physical media such as a DVD-RW or DVD+RW. A more familiar example is the universal disk format (UDF), which is the file system format found on most DVDs.

Over the next pages, we will look at "local file system" examples, which most readers will see, are probably the most familiar of the file systems which they routinely come into contact with.

Local File System

The names that stand out most in file systems where personal computers are used consist of the following:

File Allocation Table (FAT) New Technology File System (NTFS) Universal Disk Format (UDF)

The number of other file systems, including those for mobile systems, and those that are embedded, are beginning to grow as storage, and memory moves well outside the boundaries of spinning disks and CD-ROMs. To cover all of these file systems would be a journal unto itself, so this section will pick a few of the more recognizable file systems that are employed with or on the operating systems discussed in the first part of this chapter.

File Allocation Table

The FAT file system, developed during the period around 1976–1977, was first created for managing disks that supported the Intel 8086 microprocessor version of Microsoft Stand-Alone Disk BASIC, a BASIC interpreter with a built-in operating system. FAT is the file system supported by the Microsoft MS-DOS operating system.

FAT uses a table which centralizes the information that describes which areas of the storage media (drives) belong to files, those areas that are free or possibly unusable, and the location of each file as it is stored on the disk. Disk space is allocated to files in contiguous groups of hardware sectors called clusters (i.e., allocation units). Early disk drives had a relatively low number of clusters compared with today’s drives. Throughout the evolution of magnetic storage media, beginning with the flexible removable floppies and growing into comparatively huge fixed hard disk drives, the number of clusters that divided these storage areas on these drives grew significantly. This escalation required increasing the number of bits used to identify each cluster. Successive versions of the FAT format were created to address the number of table element in terms of the bits that described the table entries as in

the following.

FAT12

An initial version designed as a file system for floppy disks with cluster addresses limited to 12-bit values. This file system had a limited cluster count of 4078. The12-bit value is derived from 212 (= 4096), which ordinarily would be the number of clusters (or allocation units) that can be addressed. However, the actual cluster count is less than this, given that 000h and 001h are not used, and FF0h to FFFh are reserved (or used for other purposes), leaving the possible range of clusters between 002h and FEFh (decimal numbers 2–4079).

FAT16

[sty: Text flush] The second implement introduced in 1988 which became the primary file system for Microsoft MS-DOS 4.0 (disk operating system) up through Microsoft Windows 95. FAT16 would support disk drive capacities up to 2 Gbytes.

FAT32

This FAT system was introduced in 1996 for Windows OSR2 users and became the primary file system for consumer versions of Windows up through Windows ME. FAT32 supports disk drive capacities up to 8 Tbytes.

New Technology File System

Windows New Technology File System, the formal name which quickly became simply NTFS, was introduced by Microsoft in 1993 with the Windows NT 3.1 operating system. NTFS is the primary file system used in Microsoft's Windows NT, Windows 2000, Windows XP, Windows 2003, and Windows Vista/Windows 7 operating systems. NTFS supports hard drive capacities up to 256 Tbytes.

When Microsoft first designed NTFS, it already supported both FAT and HPFS. The company had determined that it needed a new operating system, which it would name Windows NT, so the Windows NT File System was engineered to be a more robust replacement for HPFS. NTSF, as a mature journaling file system, would become a superlative file system for fixed hard drives because it provided for the essentials of recoverability and availability.

Availability meant that should a service or system crash, it would recover because of how the data was stored in the journaling process. The file system would in a few seconds recover without having to run a CHKDSK as often as other systems without journaling.

NTFS provides recoverability because of the techniques employed that maintain, in the database-like logging structure, all the activities of the OS and other activities typical to a journaling file system, which commits metadata changes to the file system in transactions. As explained earlier, should a crash or a power failure occur without backup, the NTFS file system rolls back any uncommitted transactions so as to quickly return the file system to a stable, consistent state. Then it continues to execute instructions.

Other FAT File Systems

Beyond the mainstream file systems developed for the consumer and industrial workspace, other proprietary and nonpropriety file systems have been developed for applications including high-performance storage systems, real-time operating systems, and embedded systems.

Some of those modified FAT file systems for Windows include FATX designed for the Microsoft XBox gaming system's hard drive and memory cards. Another incompatible replacement for the FAT file system, called exFAT, was introduced for Windows Embedded CE 6.0 where its intended use was for flash drives. exFAT introduced a free space bitmap that increased faster space allocation and faster deletes, and supported for files up to 264 (= 18.435 Ebytes), and included much larger cluster sizes among other features.

Besides those file systems described previously, other examples of disk file systems include the following:

HFS (and HFS+)

Formally named the Hierarchical File System, the proprietary file system developed by Apple for those running the Mac OS; HFS Plus is the extended implementation also called the Mac OS Extended. HFS divides a volume into logical blocks of 512 bytes. Those logical blocks are grouped into allocation blocks which are comprised of one or more logical blocks, depending upon the total size of the volume. HFS is limited to 65,536 allocation blocks because it was originally set up using a 16-bit value to an address in those allocation blocks.

HPFS

The High-Performance File System that was developed for the OS/2 operating system initially created by IBM and Microsoft, and then later exclusively developed by IBM, and is no longer marketed by them. HPFS allows for mixed case file names, supports 255-character-long file names (FAT was limited to 8+3), stores files on a per-sector basis as opposed to using multisector clusters (this improves disk space utilization), and uses the B+ tree structure for directories.

B+ tree (B plus tree) uses a key identifier that allows for multilevel indexing which improves record insertion, removal, and retrieval. The alternative version, B-tree (B minus tree) is a structure generalized from the binary search tree which has its records stored in a leaf level of the tree. The binary search tree allows a node to have more than two children. B-tree is commonly used in databases and file systems and is optimized for those systems that read and write large blocks of data.

UFS and V7

This stands for the Unix file system, which is used by Unix and Unix-like platforms. UFS is a hierarchical file system, with the root of the file system as inode 2. UFS is the default Solaris file system.

Also known as the Berkeley Fast File System, the Berkeley Software Distribution (BSD),

sometimes just Berkeley Unix, the BSD Fast File System or the FFS,all these names are historically, yet distantly related to the original file system used by Unix known as Version 7 Unix (or Version 7, or V7). V7 was developed for the PDP-11 from Digital Equipment Corporation (also known as DEC). During the era of the minicomputer, V7 was the first "portable" version of Unix and was released in 1979. A public release of V7/x86 (for the x86 based IA-32 PC) has been from Nordier & Associates, with the majority of files in the V7/x86 distribution derived from UNIX Version 7 source code, and some of them from 32V UNIX source code.

UDF

Defined by the Optical Storage Technology Association (OSTA), the Universal Disk Format is a file system that is compliant with ISO-13346/ECMA-167, which is the successor to the CD-ROM file system (CDFS or ISO-9660). All the standard formats used in video recording, on DVD-like media, will use some version of the UDF file system.

Philips' DVD+VR format uses UDF 1.02 with an ISO 9660 bridge for DVD+R and DVD+RW.

The DVD Forum's DVD-VR format uses UDF 2.00 for DVD-R, DVD-RW and DVD-RAM.

Blu-ray Disc and the DVD Forum's HD-DVD will be using UDF 2.50 or UDF 2.60.

The UDF file system has become popular as the requirements for storing large files on cross-platform media whose fully rewriteable storage capabilities have exceeded several gigabytes. Those media include, for example, flash media (see Chapter Seven) and the 35 Gbytes and 70 Gbytes Iomega REV removable hard disks.

ZFS

Originally called the Zettabyte File System—a zettabyte (ZB) is 1000 Ebytes, with an exabyte equals to 1000 Pbytes—this file system was designed and implemented in the 2004 time frame by Sun Microsystems (now part of Oracle) and released for the Solaris around mid 2006. The main focus of ZFS is to protect the file against "silent corruption,” which designers claim that it is not currently possible with most of the other named file systems including XFS, JFS, or, NTFS.

Flash File Systems

] Designed for storing files on flash memory devices, flash file systems are becoming more prevalent especially as the number of mobile devices increases and the capacity of flash memories gets much larger. Flash file systems are optimized for the particular architectures of flash which are related to how data is written, read, and altered on that device.

Although a disk file system can be used on a flash device, this is suboptimal for several reasons:

Block erasure—flash memory blocks have to be explicitly erased before they can be rewritten to. The time required to erase these blocks can be significant; thus, it is advantageous to erase those unused blocks while the device is idle.

Random accessibility—a disk file system is optimized to avoid the latency associated with disk seeks (i.e., the activity of locating the blocks or data on the physical drive) whenever possible. Flash memory devices impose comparatively little to no impact on seek latency.

Wear leveling—when a single block in a flash memory device is continually overwritten, they will tend to wear out over time. The flash file system is designed such that it spreads out and writes evenly.

A log-structured file system, such as JFFS2, UBIFS, and YAFFS, will have many of the desirable properties for a flash file system.

JFFS2

This revision is the successor to the original Journaling Flash File System (JFFS) that was used only with NOR flash memory and was typically employed on the Linux operating system. JFFS enforced wear leveling by treating the flash device as a circular log. JFFS2 now supports NAND flash, whereas the original JFFS supported only NOR. NAND devices have a sequential I/O-interface and thus cannot be memory-mapped for reading. JFFS2 remedies this and other issues including hard links, compression, and better performance including support for larger flash devices in excess of 64–512 Mbytes.

The successor to JFFS2 is the Unsorted Block Image File System (UBIFS), designed when flash memory chips began to exceed 512 Mbytes. The indexing on flash with UBIFS is B+ tree, with a wandering algorithm, and indexing in memory uses Tree-Node Cache (TNC), which caches indexing nodes and may be shrunk should Linux virtual machine (VM) requires more memory. UBIFS allows for write-back operations, which perform caching at writing, as opposed to write through in JFFS2

YAFFS

As if three letter acronyms (TLAs) were not enough, this five-letter acronym is named Yet Another Flash File System and was designed by New Zealander Charles Manning for Aleph One. Of course, there is a YAFFS1 and a YAFFS2 designed because NAND flash memory evolved from the original 512-byte page (with 16-byte spare) to the new larger pages with 2048-byte and 64-byte spares. Still a log-structured file system, it is designed with data integrity and robustness in mind and is used on Linux, pSOS, eCos and various other "special purpose" OSs including WinCE.

The importance of these newer and relatively obscured file systems is in the development and rising popularity of mobile devices and other portable platforms. Most of these systems use embedded OSs, which rely on flash memory for storage. Ordinary file systems place undo wear on the flash chips, causing them to wear out prematurely. These file systems are built around the needs of wear leveling, the practice which prevents using the same memory space in the flash chip repetitively before exhausting other areas of open/free memory on the chip.

Traditional File Systems

Traditional file systems offer facilities, i.e., capabilities or services, that can create, move, and delete both files and directories. However, they also may lack the abilities to create supplemental links to a directory (called “hard links” or “parent links” in Unix and Unix-like

OS) or to create bidirectional links to files.

Traditional file systems may further offer facilities that can truncate, append to, create, move, delete, or in-place modify files, but they lack the facilities to prepend to or truncate from the beginning of a file and may further preclude the capability of arbitrary insertion into or deletion from a file.

Editing systems for media centric systems depend upon many of the functions that are outside the bounds of traditional operating systems. Because of the requirements that media activities present, such as the continual modification of files of enormous proportions, specialized considerations must be given to these types and sizes of files.

These sets of operations are highly asymmetric, and they lack the generality to be useful in unexpected contexts. In Unix, as an example, interprocess pipes have to be implemented outside of the file system because the pipes concept does not offer truncation from the beginning of files. Other similar concepts may be necessary, and are often proprietary, to the architectures of file systems and operating systems for media-focused activities.

Networked or Distributed File Systems

One of the two possible references to a "networked file system" is the familiar NFS, more of a protocol that allows a client computer to access files across a network. NFS is referenced throughout this chapter and other chapters in this book.

The other version makes reference to a distributed file system. In a more generalized fashion, NFS (the protocol) is the implementation of a distributed file system. In this later perspective, the explanation is quite similar. A distributed file system is any file system that allows access to and the sharing of files between two or more host computers (clients). As mainframe computer systems were experiencing their halcyon days, and file servers were emerging (ca. 1970s), leaders in the computer business saw the need to allow multiple users on multiple machines to interchange files and data among themselves. For a mainframe implementation, long before the personal computer, this was rather an obvious requirement especially since there was only a small subset of computer systems available versus users and/or terminals.

As more machines came into being, more users needed access and eventually the minicomputer began performing the same kinds of tasks as the mainframes but with less compute power. The minis also needed shared access so that they and their users could exchange files.

Files that are accessed over a network need to have the same core principles available as those accessed on a local drive. This is readily apparent when capturing a file over the Internet and copying it to the local drive. If another user from another machine has sharing abilities with your machine, the file on your machine should look and act exactly the same as the one looked at on the other machine, and both views should be the same as the one found originally on the Internet. Updates on a file made on one machine should not interfere with the access or updates in another location.

Distributed file systems need to have protective measures implemented such as access control lists (ACLs) that prevent unauthorized users from viewing, tampering, or destroying

files that are network accessed. Since the files may be location independent, this functionality is very important.

The file system needs to be secure. It may employ a file system client such as to mount and interact with directories and files that are located on a remote workstation or server. In this case, the client will most likely interact with this remote file system via SSH File Transfer Protocol, also called Secure File Transfer Protocol (SFTP). There are many acceptable means to prevent unauthorized or hostile access, most of which came from the work of the IETF, and employ security measures based more on networks than file systems.

Some of the recognized distributed file systems in use include the following:

Apple Filing Protocol (AFP) Server Message Block (SMB), also known as CIFS Amazon S3 NetWare Core Protocol (NCP), from Novell, used on Netware-based network

Fault Tolerance

A distributed fault-tolerant file system is a closely associated, dedicated, and distributed file system that is engineered for high availability (HA) and supports activities in a disconnected (offline) operation. These systems utilize synchronization methods (sometimes over HTTP) and are available on various operating systems and may further include encryption.

Clustered file systems and shared disk file systems are alternative names for what are considered distributed file systems.

File System Processes

Within each file system, a number of processes are available that control access to files, permissions, and other security-related functions. Each activity or event associated with the file system can be controlled through a set of permissions, assignments, or allocations. This information can be obtained from the metadata associated with the file attributes and a data rights and policies process called an access control list.

Access Control List

When referencing file systems, the access control list (ACL) is usually a data structure in the form of a table containing entities that describe and assign permissions which are attached to an object. The topics of object-based storage (OBS) and those devices that support objects, i.e., object-based storage devices (OSD), are discussed in greater depth in Chapter Eleven. The ACL specifies which system rights, processes, and what users are granted access to those objects. The objects may be programs, processes, or files. The ACL further delegates what operations are allowed to be performed on which objects.

ACL models may be applied to a set (i.e., a collection) of objects as well individual entities within the system.

Security is a key consideration in any ACL-based model. Of concern is how these access

control lists are edited, by which users and under which processes. The ACL drives specific access rights which show up as privileges or permissions for users, such as executing, writing to or reading from an object. ACL processes can be extensible to a digital rights management (DRM) system whereby any modification process that could jeopardize the integrity of the content, that is, the files which are the content.

ACLs will also be known as access control entries (ACE) in the operating systems for Microsoft Windows NT, OpenVMS, Unix-like systems (e.g., Linux, BSD, or Solaris), and Apple's Mac OS X.

Attributes of Metadata

Files have their own set of attributes that are used as a means of bookkeeping for both the individual files and the file system as a whole. The information which is in those attributes is called metadata. The metadata is typically organized so that it associates with the structure of how that information is identified, tracked, and cataloged within the file system. The data length of a file may be stored as the number of blocks allocated for the file or as an exact byte count. When the file was last modified may be stored as the file's timestamp. Other metadata attributes employed by some file systems include the creation time, the time it was last accessed, and the time that the file's metadata was changed.

Additional information, some of which early PC operating systems did not track, included the file's device type, its block, character, socket, subdirectory, or rights management data. The file may also catalog the owner's user-ID or group-ID and access permission settings such as whether the file is read-only, if it is an executable file or the security level associated with the file.

Advanced file systems carry information considered arbitrary, such as in XFS, some versions of UFS, and HFS+. This information is cataloged using a set of "extended file attributes." Such features are implemented in the kernels of Linux, FreeBSD, and Mac OS X operating systems, which enable metadata to be associated with the file at the file system level. The information could include, for example, the author of a document, the character encoding of a plain-text document, or even a data checksum for file integrity or recovery authentication.

Journaling File Systems

A journaling file system, in general principle, is a file system that tracks changes into a journal which can be thought of as a form of register or log. In this context, the file system's "journal" is usually a circular log placed into a dedicated area of the file system. This is where any changes are placed before they are committed to the main file system. Should a failure occur, such as a system crash or power interruption, the journal which has kept these potentially volatile files is utilized by the file system to quickly bring the system, and its files, back online with less likelihood of corruption.

The journaling file system is used in many product lines and on various operating systems. Because the file system uses a special area on the disk, referred to as the "journal," to write changes which occur during operation, this significantly reduces the time required for data updating while increasing file system reliability.

Modern versions of the journaling file system were developed for the OS/2 Warp Server (from IBM) for e-Business. IBM calls their implementation a "journaled file systems" (JFS) built on a 64-bit journaling file system architecture. The original IBM JFS was later ported to IBM AIX and Linux. The goal of the developers was to provide high performance, reliability, and scalability on symmetrical multiprocessing machines (SMP).

When a computer's hardware architecture employs two or more identical microprocessors that share the same main memory and are controlled by a single instance of the operating system, the computer will generally operate in a symmetrical mode. When multiple processors are connected using cross-bar switching, on-chip mesh networks or through buses, the control of the interconnections is critical to avoid bottlenecks. Scaling SMP systems is complicated when using buses or cross-bar switching because of the interconnections that must be made between memory, storage, and the associated processors.

USN Journal

The Update Sequence Number Journal (USN Journal) is a system management feature that records changes to all files, streams, and directories on the volume, as well as their various attributes and security settings.

USN Journal is a critical functionality of NTFS, and a feature that FAT/FAT32 does not provide, for ensuring that its internal complex data structures will remain constants during instances where the system crashes. These data structures are notably the volume allocation bitmap (data moves performed by the defragmentation API), modifications to Master File Table (MFT) records, updates to the shared security descriptors or to the boot sector as well as its local mirrors. The later structures are where the last USN transaction that would be committed on the volume is stored. In the event of a crash, the indices for the directories and security descriptors will remain consistent; thus, allowing for an easy rollback of those uncommitted changes to critical data structures once the volume is remounted.

In versions of Windows and Windows Server (2003 et al), the USN journal was extended to trace the state of other transactional operations on other parts of the NTFS file system. Such services, like that of the Volume Shadow Copy Service, will create shadow copies of system files with, for example, cloning (a full copy/split mirror) or copy-on-write (i.e., differential copy) semantics. Volume Snapshot Service (VSS) is another technology (included in Windows) that operates at the block level of the file system. VSS is similar in purpose, but different in functionality. Snap shotting is often found in intelligent disk storage systems.

Large Scale File Systems

Installing and configuring large-scale file systems can be extremely complex. Whether it is a clustered file system in front of SAN-attached storage or a number of traditional dual controller NAS systems glued together with virtualization software; time and money can be wasted trying to deploy legacy systems for large-scale multimedia and video applications.

Multimedia and Video Applications

Meeting the requirements of a mission critical system requires more than just duplicated or mirrored systems which are designed mostly to prevent a complete failure of a primary

system. In media and video applications, performance, capacity, and reliability become paramount in the design and deployment of servers and storage systems. Employing traditional, legacy storage architectures for these extremely demanding systems can be expensive, risky, and complex.

When the storage systems built for media or video applications utilize conventional approaches built around NAS and SAN (refer to Chapter Fifteen) storage systems, one may find there are limitations that limit either the capacity or the scalability of the storage platform. These limitations may present themselves at unexpected moments when, for example, the maximum number of addressable or attachable NAS-heads is realized; or the expansion of a SAN is curtailed because of port count that has been reached or the finite amount of addressable drives in a given volume is approached.

Some storage architectures also begin to suffer from performance (bandwidth) constraints as the number of drives or array components approaches some degree of physical capacity whereby the only alternative is a forklift upgrade that replaces some or all of the drives.

There continue to be improvements in addressing the capacity and performance capabilities of media-centric storage systems. These issues have in recent years been resolved by various storage system approaches that include new approaches in grid storage and file system handling.

Grid Architectures

When a storage system is built such that it utilizes a distributed architecture that spreads slices of the files into sets of replicated components, the system may be referred to as grid storage or as a grid architecture. When the file data is segmented into chunks and then broken further down into slices, some form of tracking mechanism must be employed so that the file elements can be reconstructed when read from the store.

Metadata is used to track the locations of file data. This metadata is usually managed by a secondary server. In addition to being the mechanism for reassembling the files, the metadata controller is allocated the task of organizing the data when a storage element (e.g., a disk drive or content store) fails, and it must be reconstructed using some flavor of a parity system similar to the way a RAID set is rebuilt.

Grid architectures are quite useful for optimizing performance and capacity scalability. When servers or storage components are added to a grid system, not only does storage capacity increase but also performance is increased as bandwidth and throughput are scaled up.

Grids and clusters are discussed in Chapter Fourteen.

Further Readings

ECMA-167. Volume and File Structure for Write-Once and Rewritable Media using NonSequential Recording for Information Interchange (1997, 3rd ed.). Specifies a format and associated system requirements for volume and boot block recognition, volume structure, file structure, and record structure for the interchange of information on media between users of information processing systems http://www.ecma-international.org/publications/Ecma-

167.htm

Universal Disk Format (UDF) specification, Revision 2.50 http://www.osta.org/specs/

ISO-13346 - the ISO number for the UDF file specification http://www.iso.org

The Filesystem Manager http://www.swd.de/documents/manuals/sysarch/fsys_en.html