Case Study 1: AMOEBA

History of Amoeba Amoeba originated at the Vrije Universiteit, Amsterdam, The

Netherlands in 1981 as a research project in distributed and parallel computing. It was designed primarily by Andrew S. Tanenbaum and three of his Ph.D. students. By 1983, an initial prototype, Amoeba 1.0, was operational.

Starting in 1984, a second group was set up. This work used Amoeba 3.0, which was based on RPC. Using Amoeba 3.0, it was possible for clients in Tromso to access servers in Amsterdam transparently, and vice versa.

Research Goals The primary goal of the project was to build a

transparent distributed operating system. An important distinction between Amoeba and

most other distributed systems is that Amoeba has no concept of a “home machine”.

A secondary goal of Amoeba is to provide a testbed for doing distributed and parallel programming.

The Amoeba System Architecture

Processor pool

X-terminals

Fileserver Print server

The Amoeba Microkernel

Microkernel

Client

Process managementMemory managementCommunicationI/O

ServerThread

The Amoeba Servers Amoeba is based on the client-server

model. Probably the most important server is the

file server, known as the bullet server. Another important server is the directory

server, also known as the soap server.

Objects and Capabilities The basic unifying concept underlying all

the Amoeba servers and the services they provide is the object.

Each object is managed by a server process.

Objects are named and protected by capabilities.

A capability in Amoeba

Server port Object Rights Check

Bits 48 24 8 48

Capability When the server creates an object, it

generates a capability.

If a client wants to create a restricted capability, go through the following.

On subsequent operations, the client must present the capability to identify the object.

port object 1111 1111 C (random number)

Restricted capability

port object C1111 1111

port object 00000001 f(C XOR 00000001)

New rights mask

0000 0001

Exclusive OR

One-way function

Restricted capability

Capability

Standard Operations Age Perform a garbage collection cycle: starts a new garbage collection cycle to get rid of old objects

that are no longer accessible.

Copy Duplicate the object and return a capability for the copy: it is a shortcut that makes it possible to duplicate an object without actually transferring it. Without this operation, copying a file would require sending it over the network twice: from the server to the client and then back again.

Detroy Destroy the object and reclaim its storage: deletes the object

Getparams Get parameters associated with the server: allow the system administrator to read and write parameters that control server operation. For example, the algorithm used to choose processors can be selected using this mechanism.

Info Get an ASCII string briefly describing the object

Restrict Produce a new, restricted capability for the object

Setparams Set parameters associated with the server: same as Getparams

Status Get current status information from the server

Touch Pretend the object was just used: tells the server that the object touched is still in used.

Process Management in Amoeba A process is an object in Amoeba. When a

process is created, the parent process is given a capability for the child process. The child can be suspended, restarted, signaled, or destroyed.

Process management is handled at three different levels in Amoeba.

1. At the lowest level are the process servers, which are kernel threads running on every machine.

2. At the next level up we have a set of library procedures that provide a more convenient interface for user programs.

3. Finally, the simplest way to create a process is to use the run server, which does most of the work of determining where to run the new process.

Process Descriptor

Architecture = 386Capability for exit statusSegment descriptorsThread 1 PC1 SP1

Thread 2 PC2 SP2

Thead 3 PC 3 SP 3

Text Shared data 1 2 3

Private data

Stacks

PC1

PC2

PC3

Segments

SP1SP2

SP3

Library procedures exec: to do an RPC with the specified

process server asking it to run the process. getload: returns information about the CPU

speed, current load, and amount of memory free at the moment.

stun: to suspend a child process. Two kinds of stuns are supported: normal and emergency.

Threads When a process starts up, it has one thread. During execution, the process can create

additional threads, and existing threads can terminate.

Three methods are provided for threads to synchronize: signals, mutexes, and semaphores.

All threads are managed by the kernel.

Memory Management in Amoeba Amoeba has an extremely simple memory model. A process can have any number of segments it

wants to have, and they can be located wherever it wants in the process’ virtual address space.

Segments are not swapped or paged, so a process must be entirely memory resident to run.

Each segment is stored contiguously in memory.

Mapped segments

S

D

T

S DT

Process virtualaddress space Memory segments

Communication in Amoeba Amoeba supports two forms of

communication: RPC, using point-to-point message passing and group communication.

Remote Procedure Call RPC Primitives: 1. get_request – indicates a server’s

willingness to listen on a port.2. put_reply – done by a server when it has a

reply to send.3. trans – send a message from client to

server and wait for the reply.  

Group Communication in Amoeba

CreateGroup Create a new group and set its parameters

JoinGroup Make the caller a member of a group

LeaveGroup Remove the caller from a group

SendToGroup Reliably send a message to all members of a group

ReceiveFromGroup

Block until a message arrives from a group

ResetGroup Initiate recovery after a process crash

The Amoeba Reliable Broadcast Protocol 1. The user process traps to the kernel, passing it the

message.2. The kernel accepts the message and blocks the user

process.3. The kernel sends a point-to-point message to the

sequencer.4. When the sequencer gets the message, it allocates the next

available sequence number, puts the sequence number in a header field reserved for it, and broadcasts the message (and sequence number).

5. When the sending kernel sees the broadcast message, it unblocks the calling process to let it continue execution.

System Structure

A

SKernel

A

Kernel

A

Kernel

Sequencer enabled Sequencer disabled

Application programs

Broadcast network

An example

A

Last = 24

A

SLast=24

A

Last=24

A

Last=23

AM

M

M25

B

C

history

M25

M25

M25

Sequencer machine

M25

M25

Request for 24

buffered

Sender’s action for sending The sender sends a message to the sequencer and starts a timer: (a) the broadcast comes back before the timer runs out.

(normal case). the sender just stops the timer. (b) the broadcast has not come back before the timer expires.

(either the message or the broadcast has been lost). the sender retransmits the message. if the original message is lost, no harm is done. if the sender missed the broadcast, the sequencer will

detect the retransmission as a duplicate and tell the sender everything is all right.

(c ) The broadcast comes back before the timer expires, but it is the wrong broadcast. This occurs when two processes attempt to broadcast simultaneously.

If message A gets to the sequencer first, and is broadcast. A sees the broadcast and unblocks its application program. However, B sees A’s broadcast and realizes it has failed to go first. B will accept A’s broadcast and wait.

Sequencer’s action If a Request for Broadcast arrives: (a) check to see if the message is a

retransmission. If so, inform the sender that the broadcast has been done.

(b) if the message is new, assign the next sequence number to it, and increment the sequencer counter by 1.

The message and its identifier are stored in a history buffer, and the message is then broadcast.

Sender’s action for receiving When the sender receives a broadcast: (a) if the sequence number is 1 higher than

the most recent one (normal case). No broadcast has been missed.

(b) if the sequence number is more than 1 higher (a broadcast has been missed), the sender will send a message to the sequencer asking for the lost broadcast.

Management of the history buffer If the history buffer fills up, if the sequencer knows that

all machines have received broadcasts, say, 0 through 23, correctly, it can delete these from its history buffer. There are several mechanisms to discover this information:

(a) each Request for Broadcast message sent to the sequencer carriers a piggybacked acknowledgement, k, meaning that all broadcasts up to and including k have been correctly received.

(b) the sequencer can broadcast a Request for Status message asking the number of the highest broadcast received in sequence.

Two methods for doing reliable broadcasting

A S BAB S1

2 2

2

22

2 2

2

22

11

1

11

1. Message sent to the sequencer2. The sequencer broadcasts it

1. A broadcast M2. S broadcasts Accept

In method 1, each message appears in full on the network twice. Each user machine is interrupted only once.

In method 2, the full message appears only once on the network. Each machine is interrupted twice.

Summary of the protocol This protocol allows reliable broadcasting

to be done on an unreliable network in just over two messages per reliable broadcast. Each broadcast is indivisible, and all applications receive all messages in the same order, no matter how many are lost.

Fault tolerance The protocol is also fault tolerant. When a processor crashes, sooner or later, some

kernel will detect that the crashed machine are not being acknowledged. The kernel will mark the crashed processor as dead and initiates a recovery.

In phase 1, one process is elected as coordinator. In phase 2, the coordinator rebuilds the group and

brings all the other processes up to date.

The protocol is also fault tolerant

40 43 4441 40 X

40 43 4441 40 X

44 44 4444 44 X

0 1 2 3 4 5

coordinator coordinator Sequencer dies

coordinator Dead sequencer

new sequencer

0 1 2 3 4 5

0 1 2 3 4

(a)

(b)

(c )

(a) The sequencer crashes (b) A coordinator is selected (c) Recovery

How does the coordinator get any message it has missed if the sequencer has crashed?

The solution lies in the value of k. If k is 0 (non-fault tolerant), only the sequencer maintains a history buffer.

If k >0, k+1 machines maintain an up-to-date history buffer. If k machines crash, there is still one left to supply the coordinator with any messages it needs.

How to implement? In method 2, when the sequencer sees a message,

M, that was just broadcast, it does not immediately broadcast an Accept message. Instead, it waits until the k lowest-numbered kernels have acknowledged that they have seen and stored it. Now k+1 machines have stored M in their history buffers.

The Fast Local Internet Protocol Amoeba uses a custom protocol called

FLIP for actual message transmission. The protocol handles both RPC and group communication and is below them in the protocol hierarchy.

FLIP is a network layer protocol.

A

RPC Group

FLIP layer

The Bullet Server Create Create a new file; optionally commit it as

well

Read Read all or part of a specified file

Size Return the size of a specified file

Modify Overwrite n bytes of an uncommitted file

Insert Insert or append n bytes to an uncommitted file

Delete Delete n bytes from an uncommitted file

The Directory Server Create Create a new directory

Delete Delete a directory or an entry in a directory

Append Add a new directory entry to a specified directory

Replace Replace a single directory entry

Lookup Return the capability set corresponding to a specified name

Getmasks Return the rights masks for the specified entry

Chmod Change the rights bits in an existing directory entry

The Replication Server The Run Server The Boot Server The TCP/IP Server Disk server I/O server A time-of-day server A random number server Swiss Army Knife server