294

OSI at Work

File Access and Serving in an OSI Environment

Wil l i am B L A C K Joint Network Team, c /o Rutherford Appleton Laboratory, Did- cot, Oxfordshire, United Kingdom 0Xll OQX (tel." + 44 235 44 5880; fax: + 44 235 44 5808; e.mail: w. Black@rl.ac.uk)

Abstract. FTAM is now a published ISO standard and prod- ucts using FTAM for simple file transfer are now being an- nounced by suppliers. This paper reviews how the use of FTAM might be extended to do the more complex file access and serving in an open environment and points out some difficulties which must be overcome if FTAM is to replace vendor specific and de facto protocols in this area. Some recommendations to RARE are given.

Keywords. OSI, File Access, File Server, FTAM, ISO 8571, Functional Standards, Document Types, Operational Require- ments.

William Black graduated in Mathe- matics and Astronomy from the Uni- versity of Glasgow in 1971 and con- tinued there to be awarded his Doctorate in numerical methods ap- plied to dynamical systems in 1974. He then joined the High Energy Physics group in the Department of Nuclear Physics in Oxford where he was involved in the Bubble Chamber film analysis project using computer

~ I controlled flying-spot scanners (PEPR).

Having, by the mid-seventies, become involved in the UK's Academic networking, he took over the technical leadership of the UK contribution to the Generalized Internetworking File Transfer gateway project (GIFT). He has represented the UK on RARE WG2 from its first meeting and recently became this group's deputy chairman.

In 1986 he joined the Joint Network Team as the section leader with responsibility for file transfer, job transfer, ISO upper layers, and the networking of UNIX, OS/2 and DOS systems within the UK Academic Community. In 1987 he took up the leadership of BSI's committee on FTAM and attends ISO FTAM meetings and EWOS functional standards meet- ings on behalf of the UK.

North-Holland Computer Networks and ISDN Systems 17 (1989) 294-299

1. Definitions

It is important, when discussing the handling of files in an OSI environment, to distinguish clearly among several methods of dealing with files across computer networks. Three forms of usage are de- fined here.

File Transfer involves the moving or copying of the whole file from one system to another. The file is read sequentially on the sender's system and written sequentially on the receiver's system. The network between passes the records or bytes from sender to receiver. If the file is large, then offline or spooling mechanisms are often invoked so that the user need not wait for completion of the transfer. Indeed it may be desired to defer transfer until an off-peak time when network tariffs are lower. Again, especially when the files are large and the transfer time large compared to the mean failure time of the network, restart and recovery mechanisms are highly desirable to prevent re- transmission of parts of the file when the network recovers.

File Access permits the initiating system to read or write selected, individual records on the file, which is held on the responding system. This reading or writing need not be sequential and could be random or indexed by a key field. The important distinction is that the whole file need not be moved or copied to the initiator's disk before individual records may be accessed. It is a useful mechanism when, for example, a large central database is held and made available to individual workstations whose local disks may even be too small to contain the complete database.

File Serving must not be confused with the previous mechanism. It is an extension of it. The term will be used to refer to the ability to manipulate files and complete sub-trees of the

W. Black / File Access and Serving in an OS1 Environment 295

directory structure held on the remote machine as if they were local. The complete File Access mech- anism must be available, but it also involves the ability to c rea te /de le te /manipula te remote direc- tory structures. It may also involve the swapping or paging of process images from the local mac- hine's memory to the remote machine's disk, as well as the ability to activate executable images held on the remote disk. This mechanism is seen at its most important where a cluster of disk-less workstations use the network to access all their files held on a common network file server. It can lead to very intense use of network bandwidth.

It is the aim of this paper to discuss some of the problems of File Access and File Serving over LANs and WANs in the context of open inter- working and vendor independence. Note, how- ever, that while it may be possible to invoke certain operations over a WAN, it may not be practical to do so (e.g., swapping and paging) due to the lower bandwidth of the WAN.

2. Standards

It has been said somewhat sceptically, that a given ISO Standard can be used to provide any required service and that a given service can be implemented using any ISO Standard. However, whilst it may be possible to provide a form of File Transfer using the Mail Standards, and a limited form of Record Access using Remote Database Access Standards, "the development of ISO 8571, File Transfer, Access and Managament (FTAM) was geared primarily to the solution of all the requirements in the previous section. It is as- sumed, therefore, that in an open and vendor independent environment the only eligible stan- dard is FTAM.

It is appropriate to consider the current state of the base standard and the corresponding func- tional standards (profiles).

The base standard ISO 8571, parts 1-4, was completed in 1988 and is available in published form from the ISO Central Secretariat and Na- tional Standards Bodies. Part 5, the Protocol Im- plementation Conformance Statement (PICS) is at the DIS stage and it is hoped that it will become IS around the end of 1989.

Functional standards are available from Europe (EWOS and C E N / C E N E L E C ) and the US (NIST,

formerly NBS). The European standard for File Transfer is ENV 41 204 ( A / l l l ) and is published by C E N / C E N E L E C . However, in the last year EWOS and NIST (and also AOW, which covers the Far East region) have been moving towards harmonization of their respective profiles. A / l l l has been revised in this light and also to align with the File Access standards which have recently been completed. The revised A / l l l was ratified by EWOS during April 1989 and is known as ED005. Standards for File Management (A/13), Transfer of flat files (A / l12 ) and Access to flat files (A/122) were also ratified by EWOS as ED001, ED006 and ED007 respectively. It is in- tended that these will proceed to C E N / C E N E - LEC for ratification as ENVs.

As part of the worldwide harmonization effort, the draft version of the International Standardized Profile (ISP) AFT11 is now being finalized by a group of 12 editors, 3 from each regional workshop. It is hoped that this will be resolved by the middle of 1989. At present it covers only File Transfer, but work will now begin to add further parts to cover File Access and Management.

Two addenda to ISO 8571 are being progressed within ISO. One covers "overlapped access" to obtain more efficiency from the underlying net- work by overlapping the acknowledgement of a previous data transfer with a new data transfer. The other is "filestore management" and provides a modelling of directories in the virtual filestore and also "group" operations and a new "group select" regime. This enables wild-card selection of groups of files for later individual select regimes. It is hoped that texts of both addenda will be agreed as DP status towards the end of May 1989, and proceed shortly thereafter for National Body ballot.

With respect to the subject of this present paper, it is clear that the way is prepared for File Access by the base standard and functional standard A/122. However to support Full File Serving, the Filestore Management addendum (and corre- sponding functional standards) will also be needed.

3. Existing Solutions

The concepts of File Access and File Serving are not new. There are many existing protocols which, to a greater or lesser extent, give the func-

296 W. B&ck / File Access and Serving in an 0 S 1 Environment

tionality required. Three wide-spread existing solutions are reviewed.

DECnet is available principally on Digital Equipment's VAX/VMS range of computers, al- though it is noted that implementations exist for others, e.g., DOS. It is designed to work over LANs and WANs and provides File Access and some aspects of File Serving (paging and swap- ping being the exceptions). It is, of course, a proprietary set of standards. However, DEC have announced their policy of migration to the ap- propriate ISO standards when they become availa- ble and it could be inferred that FTAM will replace the corresponding protocols within DEC- net in due course.

SUN's NFS has become a de facto standard and although principally designed for UNIX sys- tems, is also available for a wide range of others (e.g., DOS, VMS). Again, it is usable over LANs and WANs, but a study carried out for the Joint Network Team by Bath University Computing Service, advises against using NFS in multi-vendor or wide-area networks due to the problems of security and management involved. It is, however, quite suitable for contained clusters of UNIX machines sharing common files. The standard is widely available, but the change control is not in the public domain and would not, therefore, be classified as "open". It is implemented to run over an unconfirmed datagram service and not over any accepted ISO stacks at layers 6 and below. Indeed, due to there being no real presentation layer or concept of document types, it is only suitable for use with UNIX files and filestores.

Lastly, there are a large number of PC LANs in use, which use the NETBIOS mechanism to per- mit individual PCs in a LAN to access files on common file servers. It is restricted almost exclu- sively to DOS based machines (although inter- working with OS/2 is possible). It is again a de facto standard. It provides File Access and some File Serving. As with DECnet, swapping and pag- ing (which are not applicable to DOS) are not supported.

4. Model

It is appropriate to consider a general model for the incorporation of File Access (initially) and

File Serving (ultimately) into existing operating systems using FTAM.

Before doing so, it must be noted that the real end-user is not at all concerned with which proto- cols, if any, are involved in providing these services. If use is being made of a common network file server, then the optimum solution, when initialis- ing a workstation or logging on to a "home" system, is for the user's local filestore to be "merged" logically with any remote filestore sub-trees to which permission is granted. This linking could be built into appropriate initializa- tion files. Thus there is a "seamless" join between files held locally and those held remotely. How- ever, in a fully open environment, the user may wish to add (or remove) further sub-trees from other remote open systems at any time, and it is at this point only that the user becomes aware of the " join" between local and remote files. Suitable system commands should be available to effect these operations. Most users do not demand NFS or DECnet, but rather the functionality of File Access and Serving that these protocols give. Products based on FTAM must hide the user in the same way from the very existence of the underlying protocol.

Figure 1 shows a typical system structure with the user's application at the top. It might, depend- ing on the implementation, make "system service" calls to obtain the services of a common Record Management System. On the other hand, the RMS routines may be linked statically into the user's executable image and it is the RMS routines which use system service calls to pass control to the routines of the operating system kernel below. The term "system service" is usually implemented as a software interrupt mechanism, which takes the

System service?

System service?

Application

RMS

Kernel

Disk driver

Fig. l.

OPEN READ

Read Format d i r s cony

Read l o g i ca l disk c lu s t e r s

Read physica l disk blocks

W. Black / File Access and Serving in an OS1 Environment 297

execution path off into some privileged mode and returns to user mode on completion of the func- tion requested. In some systems, each high level language may have its own RMS module, and there may be no system-wide format for file types such as "Indexed".

For example, for a file OPEN operation, the RMS might access the appropriate tree of directo- ries, checking the user's access permissions etc., until the file header is located. For a READ data operation, format conversions may be done: ASCII character to floating point etc. At the kernel level the operations on the disk are usually in the form of actions (locate, read, write, delete) on logical clusters. These are independent of the physical block sizes on specific disk drives. It is the func- tion of the disk driver routines at a lower level in the kernel to map the logical addresses into physi- cal positioning information for the drive itself.

The aim in File Access and File Serving is to permit the user's application to remain un- changed, and as far as possible unaware, of the fact that the real disk is not on the local system. Imagine, therefore, that Fig. 1 is pulled apart by tension between the top and bottom layers and suppose that a break forms somewhere in the RMS layer. The " tear" between the two parts of the RMS module are "healed" by the use of a suitable networking protocol and Fig. 2 shows the networked form of Fig. 1. Here the RMS system uses the FTAM Service as defined in ISO 8571-3 to communicate to its remaining half on the re-

Application

RMS

FTAM

FTAM

RMS

Kernel

Disk driver

Work Stations

and

General Purpose

General Purpose

and

File Servers

Fig. 2.

mote system. Of course, the two FTAM layers linked in the diagram by a jagged line are in reality using layers of 1-6 of the ISO stack to communicate across the network.

The upper part of the diagram would be pre- sent in workstations and also in general purpose systems wishing to access the filestores on remote machines. The lower part of the diagram would be implemented in both general purpose systems and dedicated file servers.

It is important to remember the distinction between the Virtual Filestore and the protocol used to communicate between systems. Thus NFS, the protocol, should not be confused with the Virtual Filestore functionality of NFS. It seems quite feasible to the present author, that NFS, the protocol, could be replaced by FTAM, the pro- tocol, while maintaining the same appearance at the application level to the end-user. (The same argument applies equally well to DECnet and NETBIOS.)

5. Problems

However, life is never quite so simple! There are many problems that must be solved if this is to be made real. It is possible that new addenda will be needed to FTAM to overcome some of the problems.

Firstly, there is a need to agree on document types. Document types are used by FTAM so that the two ends can agree on the structure and semantics of files. Unfortunately in spite of the world-wide harmonization of the functional stan- dard mentioned above, these standards leave it optional as to whether some ten or so document types are supported by conforming implementa- tions. These include NBS-8 for indexed files, and NBS-10, which maps well onto the typical UNIX file as a sequence of individual bytes with no record structure. Indeed the latter uncovers a deep seated problem as to how and whether the simple file structure of UNIX and DOS can interwork with the Record Management Systems built into VAX/VMS and IBM operating systems. In the previous section the actual break in the RMS level was left slightly vague. Should it be at a low level where files are in essence simple sequences of bytes, or should it be nearer the level of the

298 W. Black / File Access and Serving in an OSI Environment

high-level programming language primitives, e.g., READ/WRITE record? In the author's view this is the most important problem to be solved before FTAM can support open interworking using files.

Secondly, FTAM and its underlying stack are, in many people's view, too heavyweight to be usable in LANs where high intensity traffic is present. It must be determined, therefore, how efficiently the FTAM and layers 5-6 can be pro- grammed and integrated into existing operating systems in the way described above. Buffer copy- ing must be minimized at all costs. It must be remembered in criticising FTAM that the func- tions included in the protocol (and those in the Session and Presentation below) are there for good reasons, viz., to enable interworking between di- verse systems. Implementors must prune the func- tionality to the minimum required for open inter- working.

Thirdly, designers must consider how much can be offloaded onto front-end processors or intelli- gent "networking" boards, where most of the un- derlying networking layers (1-6) and some of FTAM itself is offloaded from the main processor. This should be considered in the light of chips such as Intel's 80386 and 2-4 Megabytes of RAM memory being able to be packaged easily on a single board, which can deliver sufficient power to implement the above stacks.

One problem remains, however, in the PC world, namely, that of DOS's 640k limit. It would seem impossible to implement FTAM and sup- porting layers within DOS and still leave room for non-trivial applications to run. After all, the rea- son for providing File Access is for such applica- tions (databases, spreadsheets) to access their files across the network. Perhaps the only solution here is to provide gateways between NETBIOS LANs and FTAM.

Lastly, while File Access could be implemented now against the agreed international standards, File Serving must wait for the completion of the filestore management addendum. Indeed some thought must be given to whether the swapping and paging aspects should even be implemented using FTAM. These are closely coupled with the intimate details of how operating systems work and where required over networks, it is perhaps best to leave this to proprietary mechanisms in a fairly close coupled LAN environment.

6. What Should be Done?

What must RARE as a user community do to progress this work? Firstly, the question must be asked, "Does RARE believe in OSI?" If the answer is affirmative, then plans must be made to achieve the goal of using FTAM for File Access and Serving. Of course, it would be useless to go out now with Operational Requirements for FTAM based solutions. The present interims must be reviewed and a selection made as to which to accept until the above problems are solved and FTAM products are implemented. However, doing only this does not solve the problem. Manufac- turers will move only when user communities de- mand openess and vendor independence. The fol- lowing are suggestions as to what should be done as a minimum.

RARE must agree on common Operational Re- quirement texts, especially on the selection of document types to be upgraded in the functional standards from "optionally supported" to "mandatory". This is tied in very closely to where the break is made in the RMS module. Should RARE, for example, be pressing hard in areas such as POSIX for a common UNIX Indexed Sequential file format, which all UNIX applica- tions would use and which would be supported by a common UNIX-wide RMS system. This would clearly aid greatly the interworking between UNIX systems and VAX and IBM systems. The same argument might be applied to DOS and OS/2, where again, every database package probably uses its own ISAM format.

Some development projects are perhaps called for to provide gateways at the File Access and Serving level between the interims (DECnet, NFS and NETBIOS) for migration purposes, but also as permanent servers in the case of NETBIOS.

Manufacturers must be pressed for their plans for end systems and workstations. They should also be approached for the supply of FTAM based file servers, which can be purchased independently from the workstations in a LAN, and which can interwork with several operating systems on that LAN. Note that this requires a separate Oper- ational Requirement as there are also many non- protocol aspects associated with file servers (e.g., performance, caching, management of quotas, backup etc.)

W. Black / File Access and Serving in an OSI Environment 299

7. Conclusions

If you believe in OSI, then you will: Formulate a migration plan from your interim to

FTAM. Warn the suppliers that interims will not be accep-

table after a given date. Demand to know the suppliers' plans. Work with and support Working Group 2 to

develop common RARE Operational Require- ments texts and experimental services.

START NOW!