A Language-based Approach to Unifying Events and Threads

Peng LiUniversity of Pennsylvania

Steve ZdancewicUniversity of Pennsylvania

Abstract

This paper presents a language-based technique to unifytwo seemingly opposite programming models for build-ing massively concurrent network services: the event-driven model and the multithreaded model. The resultis a unified concurrency modelproviding both threadabstractions and event abstractions. Using this model,each component in an application can be implementedusing the appropriate abstraction, simplifying the designof complex, multithreaded systems software.

This paper shows how to implement the unified con-currency model in Haskell, a pure, lazy, functional pro-gramming language. It also demonstrates how to usethese techniques to build anapplication-levelthread li-brary with support for multiprocessing and asynchronousI/O mechanisms in Linux. The thread library is type-safe, is relatively simple to implement, and has goodperformance. Application-level threads are extremelylightweight (scaling to ten million threads) and ourscheduler, which is implemented as a modular and ex-tensible event-driven system, outperforms NPTL in I/Obenchmarks.

1 Introduction

Modern network services present software engineerswith a number of design challenges. Web servers, mul-tiplayer games, and Internet-scale data storage appli-cations must accommodate thousands of simultaneousclient connections. Such massively-concurrent programsare difficult to implement, especially when other require-ments, such as high performance and strong security,must also be met.

Events vs. threads: Two implementation strategiesfor building such inherently concurrent systems havebeen successful. Both the multithreaded and event-driven approaches have their proponents and detractors.The debate over which model is “better” has waged formany years, with little resolution. Ousterhout [17] has

argued that “threads are a bad idea (for most purposes),”citing the difficulties of ensuring proper synchronizationand debugging with thread-based approaches. A counterargument, by von Behren, Condit, and Brewer [23], ar-gues that “events are a bad idea (for high-concurrencyservers),” essentially because reasoning about controlflow in event-based systems is difficult and the appar-ent performance wins of the event-driven approach canbe completely recouped by careful engineering [22].

From the programmer’s perspective, both models areattractive. Sometimes it is desirable to think about thesystem in terms of threads (for example, to describe thesequence of events that occur when processing a client’srequests), and sometimes it is desirable to think about thesystem in terms of events and event handlers (for exam-ple, to simplify reasoning about asynchronous commu-nications and resource scheduling).

A unified concurrency model: This paper shows thatevents and threads can be unified in a single concur-rency model, allowing the programmer to design partsof the application as if she were using threads, wherethreads are the appropriate abstraction, and parts of thesystem using events, where they are more suitable. Sec-tion 2 gives some additional background about the multi-threaded and event-driven models and motivates the de-sign of our unified concurrency model.

In our model, for higher-level application code, theprogrammer can use use a multithreaded programmingstyle similar to C and Java, with familiar control-flow el-ements such as sequencing, functions calls, conditionalsand exceptions, and with user-defined system calls forI/O and thread control. Figure 1 shows two sample func-tions written in the multithreaded style.

For lower-level I/O code, the programmer can conve-niently use asynchronous OS interfaces such asepoll

and AIO in Linux. The thread scheduler has an ex-tensible, modular event-driven architecture in which theapplication-level threads can be seen as event handlers.

A language-based approach: Languages like C andC++ have historically been used to implement high-performance, concurrent software. However, they suf-fer from well known security and reliability problemsthat have prompted a move toward type-safe languageslike Java and C#. Their general-purpose threads pack-ages are typically quite heavyweight though, and none ofthese languages provide appropriate abstractions to sim-plify event-driving programming. Implementing scal-able thread or event systems is feasible using these lan-guages, but the results can be cumbersome to use. Tech-niques such as compiler transformations can addressthese problems to some extent [22], but even then the en-gineering challenges typically force the programmer tochoose between threads or events—they don’t get both.

Our case-study implementation of the unified concur-rency model, described in detail in Section 3, is writ-ten in the programming language Haskell [12]. Haskellis a pure, lazy, strongly-typed, functional languagewith many advanced language features, such astypeclasses, that make it very convenient to implementthe unified concurrency model. Our implementation isbased on techniques developed some time ago by KoenClaessen [8] in the programming languages researchcommunity.

Application-level threads: Using this language sup-port, we have built anapplication-levelthread library, inwhich the threaded code and thread scheduler are writ-ten insidethe application. Section 4 describes the threadlibrary, which uses both event-driven and multithreadedprogramming models, and shows how it can flexibly sup-port several synchronization mechanisms for interthreadcommunication and shared state.

Compared to traditional approaches (both multi-threaded and event-driven), our application-level threadlibrary has many advantages for building scalable net-work services:

• Flexibility: The programmer can choose to use theappropriate programming models for different partsof the system.

• Scalability: The implementation of application-level threads are extremely lightweight: it scales upto 10,000,000 threads on a modest test system.

• Parallelism: The application-level threads can exe-cute on multiple processors concurrently.

• Performance:The thread scheduler behaves like ahigh-performance event-driven system. It outper-forms equivalent C programs using NPTL in our I/Obenchmarks.

• Safety: The implementation of the thread libraryand the application programming interface are bothtype-safe.

� �−−send a file over a socketsend_file sock filename =

do { fd <- file_open filename;buf <- alloc_aligned_memory buffer_size;sys_catch (

copy_data fd sock buf 0) \exception -> do {

file_close fd;sys_throw exception;

} −−so the caller can catch it againfile_close fd;

}−−copy data from a file descriptor to a socket until EOFcopy_data fd sock buf offset =

do { num_read <- file_read fd offset buf;if num_read==0 then return () else

do { sock_send sock buf num_read;copy_data fd sock buf (offset+num_read);

}}

� �Figure 1: Example of multithreaded code in Haskell

Section 5 gives the results of our performance ex-periments and describes a simple web server we imple-mented as a case study. Our experience shows that theHaskell is a reasonable language for building scalablesystems software: it is expressive, succinct, efficient andtype-safe; it also interacts well with C libraries and APIs.

Summary of contributions:1. A unified programming model that combines event-

driven and multithreaded programming. This modelis suitable for building highly scalable, concurrentsystems software.

2. A Haskell implementation of the interfaces for theunified concurrency model, based on techniquesfrom the programming languages community.

3. An application-level thread library and accompany-ing experiments that demonstrate the feasibility andbenefits of this approach.

2 Unifying events and threads

This section gives some background on the multi-threaded and event-driven approaches and motivates thedesign of the unified concurrency model.

2.1 The thread–event dualityIn 1978, Lauer and Needham [15] argued that the multi-threaded and event-driven models are dual to each other.They describe a one-to-one mapping between the con-structs of each paradigm and suggest that the two ap-proaches should be equivalent in the sense that eithermodel can be made as efficient as the other. The dual-ity they presented looks like this:

Threads Eventsthread continuation ∼ event handler

scheduler ∼ event loopexported function ∼ event

procedure call ∼ send event / await reply

2

The Lauer-Needham duality suggests that despite theirlarge conceptual differences and the way people thinkabout programming in them, the multithreaded andevent-driven models are really the “same” underneath.Most existing approaches trade off threads for events orvice versa, choosing one model over the other. We pro-pose a different route: rather than using the duality tojustify choosing threads over events or vice versa (sinceeither choice can be made as efficient and scalable as theother), we see the duality as a strong indication that theprogrammer should be able to useboth models of con-currency in the same system. The duality thus suggeststhat we should look for natural ways to support switchingbetween the views as appropriate to the task at hand.

2.2 A comparison of events vs. threads

Programming: The primary advantage of the threadmodel is that the programmer can reason about the seriesof actions taken by a thread in the familiar way, just asfor a sequential program. This approach leads to a nat-ural programming style in which the control flow for asingle thread is made apparent by the program text, usingordinary language constructs like conditional statements,loops, exceptions, and function calls.

Event-driven programming, in contrast, is hard. Mostgeneral-purpose programming languages do not provideappropriate abstractions for programming with events.The control flow graph of an event-driven program hasto be decomposed to multiple event handlers and rep-resented as some form of state machines with explicitmessage passing or in continuation-passing style (CPS).Both representations are difficult to program with andreason about, as indicated by the name of Python’s popu-lar, event-driven networking framework, “Twisted” [21].

Performance: The multithreaded programming styledoes not come for free: In most operating systems, athread uses a reserved segment of stack address space,and the virtual memory space exhausts quickly on 32-bit systems. Thread scheduling and context switchingalso have significant performance overheads. However,such performance problems can be reduced by well engi-neered thread libraries and/or careful use of cooperativemultitasking—a recent example in this vein is Capric-cio [22], a user-level threads package specifically for usein building highly scalable network services.

The event-driven approach exposes the schedulingof interleaved computations explicitly to the program-mer, thereby permitting application-specific optimiza-tions that significantly improve performance. The eventhandlers typically perform only small amounts of workand usually need only small amounts of local storage.Compared to thread-based systems, event-driven systemscan have the minimal per-thread memory overheads and

Figure 2: Threads vs. events

context switching costs. Furthermore, by grouping sim-ilar events together, they can be batch-processed to im-prove code and data locality [14].

Flexibility and customizability: Most thread sys-tems provide an abstract yet rigid, synchronous program-ming interface and the implementation of the scheduler ismostly hidden from the programming interface. Hidingthe scheduler makes it inconvenient when the programrequires the use of asynchronous I/O interfaces not sup-ported by the thread library, especially those affecting thescheduling behavior. For example, if the I/O multiplex-ing of a user-level thread library is implemented usingthe portableselect interface, it is difficult to use an al-ternative high-performance interface likeepoll withoutmodifying the scheduler.

Event-driven systems are usually more flexible andcustomizable because the programmer has direct con-trol of resource management and direct access to asyn-chronous OS interfaces. Many high-performance I/Ointerfaces (such as asynchronous I/O,epoll and ker-nel event queues) provided by popular OSes are asyn-chronous or event-driven, because this programmingmodel corresponds more closely to the hardware inter-rupts. An event-driven system can directly take ad-vantage of such asynchronous, non-blocking interfaces,while using thread pools to perform synchronous, block-ing operations.

Another concern is that most user-level cooperativethread systems do not take advantages of multiple pro-cessors, and adding such support is often difficult. Event-driven systems can easily utilize multiple processors byprocessing independent events concurrently [26].

2.3 The unified concurrency model

One important reason that the user-level threads in sys-tems like Capriccio achieve good performance is thatthe thread scheduler is essentially an event-driven ap-plication that uses asynchronous I/O interfaces to makescheduling decisions. Although the performance of user-level threads packages can rival their event-driven coun-terparts, they are less flexible than event-driven systems,because the scheduler is mostly hidden from the pro-grammer and new event-based interfaces sources cannotbe easily added. An event-driven application such as the

3

Figure 3: The unified concurrency model

Flash web server [18], on the other hand, has flexibilitywhen choosing I/O interfaces, but there is no appropri-ate abstraction for generic multithreaded programming.This situation is illustrated in Figure 2.

Ideally, we would like to use anapplication-levelthread system, where the thread abstraction is pro-vided inside the application and the scheduler is a pro-grammable part of the application. We use the termapplication-levelthreads in contrast with the more gen-eral concept ofuser-levelthreads, which are typicallyimplemented in a library that hides the thread scheduler.The problem is how to provide appropriateabstractionwhen the scheduler is part of the application: the sched-uler is a complex piece of code, it heavily uses low-level,unsafe operations and internal data structures. Withoutappropriate abstraction, writing such an application re-quires almost as much work as implementing an user-level thread scheduler from scratch.

The key idea of this paper is to separate the low-level,internal representation of threads from the scheduler im-plementation. As illustrated in Figure 3, the goal is todesign a software library (as the box in the center) thatprovides two different abstractions for application-levelthreads: thethread view, which allows per-thread codebe written in the natural, imperative, multithreaded styleas shown previously in Figure 1, and theevent view,which allows the threads be passively manipulated by theunderlying scheduler in an abstract, type-safe way.

The scheduler is part of the application as in mostevent-driven systems. The programmer needs to imple-ment only high-level algorithms and data structures fordispatching events and scheduling threads. Figure 17(described later) shows three example event loops pro-grammed in this style; such code is concise and easilycustomizable for a given application. Reusable librarieswrap the low-level operating system calls, which helpskeep the scheduler code clean.

This dualized model gives the best of two worlds:the expressiveness of threads and the customizability ofevents. The next section shows how this design is imple-mented in Haskell.

3 Implementing the unified model

In the duality of threads and events, an atomic block ofinstructions in a thread continuation (also called adelim-ited continuation) corresponds to an event handler, andthe thread scheduler corresponds to the main event loop.The relationship between threads and events can be madeexplicit using acontinuation-passing style(CPS) trans-lation [3]. However, CPS translation is painful to dealwith in practice—programmingpurely in an event-driven(or message passing) style essentially amounts to doingCPS translation manually. The goal of the unified pro-gramming model is to hide the details of CPS translationfrom the programmer. As shown in Figure 3, in orderto hide the implementation details of the middle box ina software library, the language mechanism should pro-vide adequate abstraction for both thethread viewandtheevent viewinterfaces.

Thread view: Code for each thread is written in anatural, sequential style with support for most commoncontrol-flow primitives: branches, function calls, loops,exception handling, as shown in Figure 1. Thread con-trol and I/O can be implemented using a set of systemcalls configurable for each application. The key point ofthis abstraction is that the internal representation of thethreads should be completely hidden at this level and theprogrammer does not need to manage the details such ascontinuation passing.

Event view: The internal representation of threads canbe conveniently accessed from the scheduler. The eventloop (scheduler) can (1) examine a system call request(such as I/O) submitted from a thread, and (2) execute adelimited continuation of a thread in the same way as in-voking an event handler function, passing appropriate ar-guments as responses to the system call requests. This al-lows the main scheduler to play the “active” role and thethreads to play the “passive” role: the scheduler actively“pushes” the thread continuations to make progress. Thisactive programming model makes it easy to use control-flow primitives provided by the programming languageto express the scheduling algorithms.

Here we use the techniques developed by KoenClaessen [8] to present an simple, elegant andlightweight abstraction mechanism that uses the follow-ing language features (which will be further explainedbelow):

• Monadsprovide thethread abstractionby definingan imperative sub-language of Haskell with systemcalls and thread control primitives.

• Higher-order functionsprovide the internal repre-sentation of threads in continuation-passing style.

• Lazy data structuresprovide theevent abstraction,

4

which is a lazy tree that represents the trace of sys-tem calls generated by threads.

3.1 TracesTraces and system calls: A central concept in the uni-fied concurrency model is atrace, which is a tree struc-ture describing the sequence of system calls made bya thread. A trace may have branches because the cor-responding thread can usefork to spawn new threads.For example, executing the (recursive)server functionshown on the left in Figure 5 generates the infinite traceof system calls on the right.

� �server = do {

sys_call_1;fork client;server;

}

client = do {sys_call_2;

}

� �

� �SYS_CALL_1

|SYS_FORK

/ \SYS_CALL_2 SYS_CALL_1

/SYS_FORK

/ \SYS_CALL_2 SYS_CALL_1

... ...� �

Figure 5: Some threaded code (left) and its trace (right)

A run-time representation of a trace can be defined as atree using algebraic data types in Haskell. The definitionof the trace is essentially a set of system calls, as shownin Figure 6. Each system call in the multithreaded pro-gramming interface corresponds to exactly one type oftree node. For example, theSYS FORK node has two sub-traces, one for the continuation of the parent thread andone for the continuation of the child. Note that Haskell’stype system distinguishes code that may perform side ef-fects as shown in the type of aSYS NBIO node, whichcontains anIO computation that returns a trace.

� �−−A list of system calls used in the multithreaded programmingstyle:sys_nbio c −−Perform a nonblocking IO function csys_fork c −−Create a new thread running function csys_yield −−Switch to another threadsys_ret −−Terminate the current threadsys_epoll_wait fd event −−Block and wait for an epoll event on a... ... −−file descriptor

� �� �data Trace = −−Haskell data type for traces

SYS_NBIO (IO Trace)| SYS_FORK Trace Trace| SYS_YIELD Trace| SYS_RET| SYS_EPOLL_WAIT FD EPOLL_EVENT Trace| ... ...

� �

Figure 6: System calls and their corresponding traces

Lazy traces and thread control: We can think of thetrace as an output of the thread execution: as the threadruns, the nodes in the trace are generated. What makesthe trace interesting is that the computation islazy: acomputation is not performed until its result is used. Us-ing lazy evaluation, the consumer of a trace can control

the execution of a thread: whenever a node in the trace isaccessed, the thread runs to the system call that generatethe corresponding node, and the execution of that threadis suspended until the next node in the trace is accessed.In other words, the execution of threads can be controlledby traversing their traces.

Figure 4 shows how traces are used to control thethread execution. It shows a run-time snapshot of thesystem: the scheduler decides to resume the execu-tion of a thread, which is blocked on a system callsys epoll wait in thesock send function. The fol-lowing happens in a sequence:

1. The scheduler decides to run the thread until itreaches the next system call. It simply forces thecurrent node in the trace to be evaluated, by usingthecase expression to examine its value.

2. Because of lazy evaluation, the current node of thetrace has not been created yet, so the continuation ofthe thread is launched in order to compute the valueof the node.

3. The thread runs until it performs the next systemcall,sys nbio.

4. The thread is suspended again, because it has per-formed the necessary computation to create the newnode in the trace.

5. The value of the new node,SYS NBIO is availablein the scheduler. The scheduler then handles thissystem call by performing the non-blocking I/O op-eration and running the continuation of the thread.

Therefore, the lazy trace provides theevent abstrac-tion we need: it is an abstract interface that allows themain scheduler to play the “active” role and the threadsto play the “passive” role: the scheduler can use traces toactively “push” the thread continuations to execute. Eachnode in a trace is essentially a delimited continuation thatrepresents part of the thread execution.

The remaining problem is how to provide a mecha-nism that transforms multithreaded code into traces—how do we design a software module that provides boththe thread abstractionand theevent abstractionin Fig-ure 4? The answer is the CPS monad. The next two sec-tions introduces the concept ofmonadsand shows howthe CPS monad solves these problems.

3.2 Background: monadsHaskell has support for a mechanism calledmonads[24,11] that provide an abstract type of computations withside effects.Monad is a standard interface for program-ming with functional combinators, which can be used asa domain-specific language. By designing thread con-trol primitives as monadic combinators, the monad in-terface can be used as an abstractions for multithreaded

5

Figure 4: Thread execution through lazy evaluation (the steps are described in the text)

programming, because it provide a way of hiding the “in-ternal plumbing” needed to write programs in CPS style.

The basic idea behind monads can be explained bylooking at Haskell’sIO monad, which is provided aspart of its standard libraries. A value of typeIO α isan effectful computation that may perform some actionsas side-effects before yielding a value of typeα. IO

side effects include input and output, reading and writ-ing shared memory references and accessing many sys-tem utilities. For example, the functionhGetChar takesa handle to an I/O stream and returns an action that, whenexecuted, reads a character from the stream. The func-tionhPutChar is similar, but it takes a handle and a char-acter to output and produces no result (Haskell’s type()

is similar tovoid in C). These functions have the fol-lowing types:

hGetChar :: Handle -> IO CharhPutChar :: Handle -> Char -> IO ()

There are also two standard operations calledreturn

andbind that all monads must implement. For theIOmonad, these operations have types:

return :: α -> IO α

(>>=) :: IO α -> (α -> IO β) -> IO β

Thereturn combinator “lifts” an ordinary expressionof typeα into theIO monad by returning the trivial ac-tion that performs no side effects and yields the input asthe final answer. The infix combinator>>=, pronounced“bind”, sequences the actions in its arguments:e >>= f

returns a new action that first performs the I/O operationsin e and passes the result computed bye to the func-tion f, which may then produce subsequent actions. InHaskell there is support for the so called “do” notation,which provides more familiar looking syntactic sugar forbinding multiple actions in series. For example, the twoimplementations ofdouble in Figure 7 are equivalent—both read in a character and output it twice (here and

elsewhere we use Haskell’s syntax\x->e for an anony-mous function with argumentx and bodye):

� �double :: Handle -> IO ()double h =hGetChar h >>= ( \x ->hPutChar x >>= ( \_ ->hPutChar x ))

� �

� �double h =do { x <- hGetChar h;

hPutChar x;hPutChar x;

}� �

Figure 7: The “do” syntax of Haskell

The IO monad is but one example of a wide varietyof well-known monads, most of which are useful for en-capsulating side effects. Haskell has a mechanism calledtype classes that allows the “do” notation to be over-loaded for programmer-defined monads, and we exploitthis feature to give natural syntax to the threads imple-mented on top of continuation-passing mechanisms wedescribe next.

3.3 The CPS monadThe goal is to design a monad that provides athreadabstraction, so the programmer can write multithreadedcode using the overloaded “do”-syntax with a set of sys-tem calls. The implementation of this monad is tricky,but the details are hidden from the programmer (in thebox between the thread abstraction and the event abstrac-tion in Figure 3).

The monad encapsulates the side effect of a multi-threaded computation, that is, generating a trace. Thetricky part is that, if we simply represent a computationwith a data type that carries its result value and its trace,such a data type cannot be used as a monad, because themonads require that computations be sequentially com-posable in a meaningful way. Given two complete (pos-sibly infinite) traces, there is no meaningful way to com-pose them sequentially.

The solution is to represent computations in

6

continuation-passing style(CPS), where the finalresult of the computation is the trace. A computationof type a is thus represented as a function of type(a->Trace)->Trace that expects a continuation, itselfa function of type(a->Trace), and produces aTrace.This representation can be used to construct a monadM.The standard monadic operations (lifting and sequentialcomposition) are defined in Figure 8.

� �newtype M a = M ((a->Trace)->Trace)instance Monad M where

return x = M (\c -> c x)(M g)>>=f = M (\c -> g (\a -> let M h = f a in h c))

� �Figure 8: The CPS monadM

Given a computationM a wrapped in the above CPSmonad, we can access its trace by adding a “final contin-uation” to it, that is, adding a leaf nodeSYS RET to thetrace. The functionbuild trace in Figure 9 converts amonadic computation into a trace:

� �build_trace :: M a -> Tracebuild_trace (M f) = f (\c-> SYS_RET)

� �Figure 9: Converting monadic computation to a trace

In Figure 10, each system call is implemented as amonadic operation that creates a new node in the trace.The arguments of system calls are filled in to correspond-ing fields in the trace node. Since the code is internallyorganized in continuation-passing style, we are able tofill the trace pointers (fields of type “Trace”) with thecontinuation of the current computation (bound to thevariablec in the code).

� �sys_nbio f = M(\c->SYS_NBIO (do x<-f;return (c x)))sys_fork f = M(\c->SYS_FORK (build_trace mx) (c ())sys_yield = M(\c->SYS_YIELD (c ()))sys_ret = M(\c->SYS_RET)sys_epoll_wait fd event =

M (\c -> SYS_EPOLL_WAIT fd event (c ()))� �

Figure 10: Implementing system calls

For readers unfamiliar with monads in Haskell, it maybe difficult to follow the above code. Fortunately, thesebits of code encapsulateall of the “twisted” parts of theinternal plumbing in the CPS. The implementation of themonad can be put in a library, and the programmer needsto use only its interface. To write multithreaded code, theprogrammer simply uses the “do”-syntax and the systemcalls in Figure 10; to access the trace of a thread in theevent loop, one just applies the functionbuild trace

to it to get the lazy trace.

3.4 Programming with threadsUsing the monad abstraction and the system calls, theprogrammer can write code for each thread in a natu-ral, multithreaded programming style. The system callsprovide the basic non-blocking I/O primitives. The

sys nbio system call works with any non-blocking I/Ofunction. The threaded programming style makes it easyto hide the non-blocking semantics and provide higherlevel abstractions by using nested function calls. For ex-ample, a blockingsock accept can be implemented us-ing non-blockingaccept as shown in Figure 11.

� �sock_accept server_fd = do {

new_fd <- sys_nbio (accept server_fd);if new_fd > 0

then return new_fdelse do { sys_epoll_wait fd EPOLL_READ;

sock_accept server_fd;}

}� �

Figure 11: Wrapping non-blocking operations

Thesock accept function tries to accept a connec-tion by calling the non-blockingaccept function. If itsucceeds, the accepted connection is returned, otherwiseit waits for anEPOLL READ event on the server socket, in-dicating that more connections can be accepted. Figure 1also shows similar examples of multithreaded code.

3.5 Programming with events

A simple scheduler: Traces provide an abstract inter-face for writing thread schedulers: a scheduler is just atree traversal function. To make the technical presenta-tion simpler, suppose there are only three system calls:SYS NBIO, SYS FORK and SYS RET. Figure 13 showscode that implements a naive round-robin scheduler; ituses a task queue calledready queue and an event loopcalledworker main. The scheduler does the followingin each loop:

1. Fetch a trace from the queue.

2. Examine the current node of the trace. This stepcauses the corresponding user thread to execute un-til a system call is generated.

� �worker_main ready_queue = do {−− fetch a trace from the queuetrace <- readChan ready_queue;case trace of−−Nonblocking I/O operation: c has typeIO TraceSYS_NBIO c ->

do { −−Perform the I/O operation in c−−The result is cont, which has typeTracecont <- c;−−Add the continuation to the end of the ready queuewriteChan ready_queue cont;

}−−Fork: write both continuations to the end of the ready queueSYS_FORK c1 c2 ->

do { writeChan ready_queue c1;writeChan ready_queue c2;

}SYS_RET -> return (); −− thread terminated, forget it

worker_main ready_queue; −− recursion}

� �Figure 13: A round-robin scheduler for three sys. calls

7

Figure 12: The event-driven thread scheduler: event loops and task queues

� �sys_throw e −− raise an exception esys_catch f g −−execute computationf using the exception handlerg

data Trace =... −−The corresponding nodes in the trace:| SYS_THROW Exception| SYS_CATCH Trace (Exception->Trace) Trace

� �Figure 14: System calls for exceptions

3. Perform the requested system call

4. Write the child nodes (the continuations) of thetrace to the queue.

This scheduler can already run some simple userthreads, but it only executes one operation at a time andit does not support any I/O or synchronization primitives.

Advanced control flow: Because the threaded codeis internally structured in CPS, it is easy to implementcontrol-flow operations. For example, a scheduler cansupport exceptions by adding two new system calls asshown in Figure 14. The code in Figure 1 illustrates howexceptions are used by an application-level thread.

The tree traversal algorithm inworker main must bemodified so that when it sees aSYS CATCH node it pushesthe node onto a stack of exception handlers maintainedfor each thread. The scheduler then continues executionuntil it reaches either aSYS RET node, indicating nor-mal termination, or aSYS THROW node, indicating excep-tional termination. The scheduler then pops back to theSYS CATCH node and continues with either the normaltrace or exception handler, as appropriate.

Note that if an application does not need exceptions,the programmer can remove them to simplify the sched-uler code. Conversely, if more complex per-thread stateor fancier scheduling algorithms are required, this modelaccommodates them too. It is easy to customize thescheduler to the needs of the application.

4 An application-level thread library

This section shows how to scale up the event-driven de-sign described above to build a real thread library that

takes advantage of asynchronous I/O interfaces in Linux.The development version of the Glasgow Haskell Com-piler (GHC) [9] already supports efficient, lightweightuser-level threads, but the default GHC run-time libraryuses the portable (yet less scalable)select interfaceto multiplex I/O and it does not directly support non-blocking disk I/O. Our application-level thread libraryimplements its own I/O primitives using the ForeignFunction Interface (FFI); our thread scheduler uses onlyseveral native GHC threads internally, each mapped to aseperate OS thread by the GHC run-time library.

The rest of this section shows the detailed design.First we discuss how to accommodate multiple kernel-level threads to make better use of OS-level resources.Next, we consider asynchronous I/O and low-level event-driven interfaces. We then describe several synchroniza-tion mechanisms that can be used to provide interthreadcommunication and shared state.

4.1 Using multiple event loopsFigure 12 shows the architecture of the event-drivenscheduler in our full implementation. It consists of sev-eral event loops (likeworker file open) that processevents generated by aworker main loop. Each suchloop runs in its own kernel thread, repeatedly fetching atask from an input queue or waiting for an OS event andthen processing the task/event before putting the contin-uation of the task in the appropriate output queue. Thequeues are implemented using standard thread synchro-nization primitives supported by GHC.

To boost performance on multiprocessor machines,the scheduler runs multipleworker main event loopsin parallel so that multiple application-level threads canmake progress simultaneously. This setup is based on theassumption that all the non-blocking I/O operations sub-mitted bySYS NBIO are thread safe. Thread unsafe I/Ooperations are handled in this framework either by usinga separate queue and event loop to serialize such opera-tions, or by using mutexes in the application-level thread(such synchronization primitives are described below).

8

� �−−Block and wait for an epoll event on a file descriptorsys_epoll_wait fd event−−Submit AIO read requests, returning the number of bytes readsys_aio_read fd offset buffer−−Open a file, returning the file descriptorsys_file_open filename mode

data Trace =... −−The corresponding nodes in the trace:| SYS_EPOLL_WAIT FD EPOLL_EVENT Trace| SYS_AIO_READ FD Integer Buffer (Int -> Trace)| SYS_FILE_OPEN String OPEN_MODE (FD -> Trace)

� �Figure 15: System calls forepoll and AIO

This event-driven architecture is similar to that inSEDA [25], but our events are finer-grained: instead ofrequiring the programmermanuallydecompose a com-putation into stages and specify what stages can beperformed in parallel, this event-driven schedulerauto-maticallydecomposes a threaded computation into fine-grained segments separated by system calls. Haskell’stype system ensures that each segment is a purely func-tional computation without I/O, so such segments can besafely executed in parallel.

Most user-level thread libraries do not take advantageof multiple processors, primarily because synchroniza-tion is difficult due to shared state in their implementa-tions. Our event abstraction makes this task easier, be-cause it uses a strongly typed interface in which purecomputations and I/O operations are completely sepa-rated. In Figure 4, the five steps of lazy thread execu-tion are purely functional: they process an “effect-free”segment of the user thread execution. All I/O operations(including updates to shared memory) are submitted viasystem calls and are explicitly performed in the sched-uler; the programmer has complete control on what I/Ooperations to run in parallel.

4.2 Epoll and Asynchronous IO (AIO)The trace representation of threads makes it easy to takeadvantage of event-driven I/O interfaces. To supportepoll and AIO in Linux, the thread library provides aset of system calls and trace nodes shown in Figure 15.

A special system call for opening files is needed be-cause it is a blocking operation. Our scheduler uses aseparate OS thread to perform such operations. Here, weomit AIO writing to save space as it is similar to read-ing. To execute these system calls in the scheduler, theworker main event loop in Figure 13 is extended with afew branches in Figure 16.

For anepoll request, the scheduler calls a libraryfunctionepoll add to register an event handler with thesystemepoll device. The registered event contains areference toc, the child node that is the continuationof the application thread, so theepoll event loop cansend it toready queue when the event arrives. AIO

� �case trace of...SYS_EPOLL_WAIT fd event c -> epoll_add fd event cSYS_AIO_READ fd off buf f -> aio_read fd off buf fSYS_FILE_OPEN filename mode f ->

writeChan (file_open_queue sched) trace� �

Figure 16: Handling system calls forepoll and AIO

� �worker_epoll sched =

do { −−wait for some epoll eventsresults <- epoll_wait;−− for each thread object in the results,−−write it to the ready queue of the schedulermapM (writeChan (ready_queue sched)) results;worker_epoll sched;

} −− recursively calls itself and loopworker_aio sched =

do { −−wait for some AIO eventsresults <- aio_wait;mapM (writeChan (ready_queue sched) results;worker_aio sched;

}worker_file_opener sched =

do { −− fetch a task from corresponding scheduler queue−−each task has type (String, FD→ Trace)(SYS_FILE_OPEN filename mode c) <-

readChan (file_open_queue sched);−−perform a blocking callfd <- native_file_open filename mode;−−apply c to fd, and write it to the ready queuewriteChan (ready_queue sched) (c fd);worker_file_opener sched;

}� �Figure 17: Event loops forepoll, AIO and file opening

requests are handled similarly. For a file-open request,the scheduler moves the entire trace to a queue calledfile open queue, which is used to synchronize withanother event loop.

Under the hood, the library functions such asepoll add are just wrappers for their corresponding Clibrary functions implemented through the Haskell For-eign Function Interface (FFI).

Finally, three event loops process these events, asshown in Figure 17. Each event loop runs in a sepa-rate OS thread. They work in the same way: the loopwaits for some events from its event source, processesthe events and then puts the resulting traces (correspond-ing to application threads) back into theready queue.

An application programmer can easily add other eventsources to the thread library using the same procedure:

1. Add definitions for system calls and trace nodes.

2. Interpret the system calls inworker main and reg-ister the event handlers.

3. Add event loops to process the events.

4.3 Thread synchronizationOf course, application-level threads must be able to com-municate with each other. The thread library implemen-tation provides several synchronization options.

9

To begin, application-level threads use cooperativemultitasking: context switches may happen only at thesystem calls defined by the thread library interface. Astandard callsys yield allows a thread to explicitlyyield control. Also recall that thereturn operationcauses the scheduler to treat a pure (side-effect free)block of code as an atomic operation. For single proces-sor machines, this means that the programmer has com-plete control over the atomicity of blocks of code.

On multiprocessor machines, it is necessary to pre-vent conflicts due to multiple kernel threads. For non-blocking operations, the programmer can use softwaretransactional memory (STM) [10], which is supporteddirectly by GHC. Application threads useSYS NBIO tosubmit STM transactions as non-blockingIO operations.This approach supports communication via shared mem-ory data structures with thread-safety provided by thetransaction semantics.

For blocking operations, the GHC implementation ofSTM cannot be used because the it may block the ex-ecution of worker threads. Nevertheless, the devel-oper can add application-specific system calls to pro-vide synchronization primitives. As an example, thethread library provides mutexes via two system calls:sys mutex lock andsys mutex unlock. A mutex isimplemented as a memory reference that points to a pair(locked, queue) where locked indicates whetherthe mutex is locked, andqueue is a linked list of threadtraces blocking on this mutex. Just as for other eventsources (likesys file open), an event loop and a taskqueue are added1 to process mutex system calls: lock-ing a locked mutex adds the trace to the waiting queueinside the mutex; unlocking a mutex with a non-emptywaiting queue dispatches the next available trace to theready queue. This setup of queues and event loopsguarantees the thread safety of mutex operations. Othersynchronization primitives such as theMVar in Concur-rent Haskell [13] can also be similarly implemented.

Finally, because the thread library supports theepoll

interface, threads can also communicate efficiently usingpipes. This mechanism is suitable for writing programsin message-passing style on either single processor ormultiprocessor machines.

5 Experiments

The main goal of our tests was to determine whetherour Haskell implementation of the unified concurrencymodel could achieve acceptable performance for mas-sively concurrent applications like web servers and mul-tiplayer games. Such applications are typically bound

1As pointed out by Simon P. Jones, this is an overweighted design.The mutex can be implemented completely inside the main workerthreads, without using additional queues and event loops.

by network or disk I/O and often have many idle con-nections. A second goal of our tests was to determinehow our implementation performs on a multiprocessor,since one of the benefits of our approach is the ease withwhich our thread library can take advantage of softwaretransactional memory. Finally, we wanted to investigatepossible overheads of using Haskell itself: Haskell pro-grams allocate memory frequently and the runtime heav-ily depends on garbage collection. Also, our application-level threads are implemented using higher-order func-tions and lazy datastructures that we were concernedwould impose too many levels of indirection and leadto inefficient code.

We implemented a number of benchmarks designedto assess the performance of our thread library and theoverheads of using Haskell. For I/O benchmarks wetested against comparable C programs using the NativePOSIX Thread Library (NPTL), which is an efficient im-plementation of Linux kernel threads. We also built asimple web server as a case study in using application-level threads and compared it against Apache.

Software setup: The experiments used Linux (kernelversion 2.6.15) and the development snapshot of GHC6.5, which supports multiple processors and softwaretransactional memory. Except for the memory consump-tion test, we set the suggested size for GHC’s garbagecollected heap to be 100MB. The C versions of ourbenchmarks configured NPTL so that the stack size ofeach thread is limited to 32KB. This limitation allowsNPTL to scale up to 16K threads in our tests.

Machine setup: We used different machine config-urations to run different benchmarks: Those involvingdisk and networking I/O were run on a single-processorCeleron 1.2GHz machine with a 32KB L1 cache, a256KB L2 cache, 512MB RAM and a 7200RPM, 80GBEIDE disk with 8MB buffer. The multiprocessor exper-iments (which did not need disk or network I/O) usedan SMP machine with four Pentium-III 550MHz pro-cessors, a 512KB L2 cache, and 512MB RAM. Finally,the memory consumption benchmark used a dual Xeon3.20GHz uniprocessor with 2GB RAM and 1024KB L2Cache. Each experiment described below is labeled withIO, MP, or Mem to indicate which of the three machineconfigurations was used, respectively.

5.1 Benchmarks

Memory consumption (Mem): To measure the min-imal run-time state needed for each application-levelthread, we wrote a test program that launches ten mil-lion threads that just loop callingsys yield. For thisbenchmark we set the GHC’s suggested heap size forgarbage collection to be 1GB. Using the profiling infor-mation from the garbage collector, the live set of mem-

10

1 2 3 4 5Number of processors

0.5

1

1.5

2

2.5

3

3.5

4Execution

speed

Figure 18: Speedup using multiple processors

10 100 1000 10000 100000Number of working threads

0.525

0.55

0.575

0.6

0.625

0.65

0.675

throughputHMB�sL

1 10 100 1000 10000 100000

C�NPTL

Haskell

Figure 19: Disk head scheduling test

ory objects is as small as 480MB after major garbagecollections—each thread costs only 48 bytes in this test.

Multiprocessor computation (MP): We tested 1024application-level threads, each performing a CPU-intensive operation before yielding control. Figure 18shows the speedup as the number of kernel processorsincreases. In this test, each processor runs its own copyof worker main as a kernel thread and STM is used forsynchronization. A 3.65 times speed up is achieved when4 CPUs are used.

Disk performance (IO): Our test scheduler uses theLinux asynchronous I/O library, so it benefits from thekernel disk head scheduling algorithm just as the kernelthreads and other event-driven systems do. We ran thebenchmark used to assess Capriccio [22]: each threadrandomly reads a 4KB block from a 1GB file opened us-ing O DIRECT without caching. Each test reads a total of512MB data and the overall throughput is measured, av-eraged over 5 runs. Figure 19 compares the performanceof our thread library with NPTL. This test is disk-bound:the CPU utilization is1% for both programs when 16Kthreads are used. Our thread library outperforms NPTLwhen more than 100 threads are used. The throughput ofour thread library remains steady up to 64K threads—itperforms just like the ideal event-driven system.

100 1000 10000 100000Number of inactive threads

20

40

60

80

throughputHMB�sL

100 1000 10000 100000

C�NPTL

Haskell

Figure 20: FIFO pipe scalability (simulating idle net-work connections)

FIFO pipe performance—mostly idle threads (IO):Our event-driven scheduler uses the Linuxepoll inter-face for network I/O. To test its scalability, we wrote amultithreaded program to simulate network server appli-cations where most connections are idle. The programuses 128 pairs of active threads to send and receive dataover FIFO pipes. In each pair, one thread sends 32KBdata to the other thread, receives 32KB data from theother thread and repeats this conversation. The buffersize of each FIFO pipe is 4KB. In addition to these 256working threads, there are many idle threads in the pro-gram waiting forepoll events on idle FIFO pipes.

Each run transfers a total amount of 64GB data.The average throughput of 5 runs are used. Figure 20shows the overall FIFO pipe throughput as the numberof idle threads changes. This test is bound by CPUand memory performance. Both NPTL and our Haskellthreads demonstrated good scalability in this test, but thethroughput of Haskell is30% higher than NPTL. To fur-ther investigate this performance difference, we designedthe next benchmark.

FIFO pipe performance—no idle threads (IO):This benchmark is like the previous one, except that allthreads are actively sending/receiving data over FIFOpipes. Figure 21 plots the total throughput as the num-ber of active threads increases. The overall shape of twocurves appears to be determined by cache performance:each thread allocates a 32KB buffer, so the memory foot-print of the program quickly exceeds the CPU cache sizeas the number of threads increases.

The comparison between two curves is interesting:NPTL performs better when there are fewer than 32threads, but our thread library becomes faster as morethreads are used. We conjecture that this performancedifference is caused because our thread library needsfewer kernel context switches. Because the FIFO pipehas a buffer size of 4KB, the C program performsa context switch after each 4KB data is transfered.

11

1 10 100 1000 4000Number of communicating threads

25

50

75

100

125

150

175

200throughputHMB�sL

1 10 100 1000 4000

C�NPTL

Haskell

Figure 21: FIFO pipe performance (no idle threads)

The worker main event loop in our thread schedulergroups the data transfers from all application threads andbatches them. When there are plenty of application-level threads waiting for execution, the kernel thread run-ningworker main does not yield control until it is pre-empted by other kernel threads. Therefore, our threadlibrary causes many fewer kernel context switches whenthe concurrency level is high. Intuitively, our schedulerworks like a pipeline: it delivers the best performancewhen it is fully loaded.

STM and Haskell overhead (MP): This benchmarktests the overhead of using STM transactions in Haskell.In the C program, there is one kernel thread per proces-sor; each thread is a loop that locks a mutex, incrementsa shared integer, and then unlocks the mutex. Program1 is written in Haskell using one standard kernel threadper processor. Program 2 is written in Haskell usingour application-level thread library, again with one ker-nel thread per processor and one application-level threadfor each kernel thread. In Programs 1 and 2, each threadincrements the shared integer within an STM transaction.Figure 22 compares the throughput of these programs inmillions of increment operations per second.

The results show that when there is no concurrency,the combination of Haskell and its STM implementationis significantly slower than C. However, as the amountof contention for the shared state increases, the differ-ence diminishes significantly. With four processors, theC code is less than twice as fast as the Haskell code.Note that this is a worst-case scenario for software trans-actional memory: it assumes that, in the common case,writes from two concurrent transactions won’t conflict(causing one of the transactions to roll back), but in thistest every write to the shared memory is guaranteed toconflict. Also note that the difference in performance be-tween Program 1 and Program 2 in the case of one pro-cessor is caused by the overhead of using the CPS monadin our thread library. With multiple processors, the over-head can be shared among the CPUs and the difference

1 2 3 4Number of processors

0.2

0.4

0.6

0.8

1

1.2

Throughput

1 2 3 4

Haskell�STMH2L

Haskell�STMH1L

C�Mutex

Figure 22: STM synchronization overheads

becomes negligible.

Memory allocation and garbage collection: The im-plementation of our CPS monad requires memory alloca-tion on almost every line of code. Fortunately, GHC im-plements garbage collection efficiently. In the I/O testsin Figures 19, 20 and 21, garbage collection takes lessthan0.2% of the total program execution time.

5.2 Case study: A simple web server

To test our approach on a more realistic application, weimplemented a simple web server for static web pagesusing our thread library. We reused some HTTP parsingand manipulation modules from the Haskell Web Serverproject [16], so the main server consists of only 370 linesof multithreaded glue code. To take advantage of LinuxAIO, the web server application implements its owncaching. I/O errors are handled gracefully using excep-tions. Not only is the multithreaded programming stylenatural and elegant, but the event-driven architecture alsomakes the scheduler clean. The scheduler, including theCPS monad, system call implementations, event loopsand queues for AIO,epoll, mutexes, file-opening andexception handling (but not counting the wrapper inter-faces for C library functions), is only 220 lines of well-structured code. The scheduler is designed to be cus-tomized and tuned: the programmer can easily add moresystem I/O interfaces or implement application-specificscheduling algorithms to improve performance. The webserver and the thread scheduler are completely type-safe:debugging is made much easier because most low-levelprogramming errors are rejected at compile-time.

Figure 23 compares our simple web server to Apache2.0.55 for a disk-intensive load. We used the defaultApache configuration on Debian Linux except that weincreased the limit for concurrent connections. Using ourthread library, we implemented a multithreaded clientload generator in which each client thread repeatedly re-quests a file chosen at random from among 128K pos-

12

1 10 100 1000Number of concurrent connetions

1.25

1.5

1.75

2

2.25

2.5

2.75ThroughputHMB�sL

1 10 100 1000

Apache

Haskell

Figure 23: Web server under disk-intensive load

sible files available on the server; each file is 16KB insize. The server ran on the same machine used for the IObenchmarks, and the client machine communicated withthe server using a 100Mbps Ethernet connection. Ourweb server used a fixed cache size of 100MB. Beforeeach trial run we flushed the Linux kernel disk cache en-tirely and pre-loaded the directory cache into memory.The figure plots the overall throughput as a function ofthe number of client connections. On both servers, CPUutilization fluctuates between70% and 85% (which ismostly system time) when 1,024 concurrent connectionsare used. Our simple web server compares favorably toApache on this disk-bound workload.

For mostly-cached workloads (not shown in the fig-ure), the performance of our web server is also compara-ble to Apache. A future work is to test our web server onmore realistic workloads and implement more advancedscheduling algorithms, such asresource aware schedul-ing used in Capriccio [22].

6 Evaluation and discussion

We find the results of our experiments and our experi-ence with implementing a web server using the unifiedconcurrency model encouraging. Although our threadlibrary implementation in Haskell is slower than C interms of raw speed, it performs quite well in our testsand scales well in terms of the number of concurrentthreads it can handle. In this section we discuss someof the non-quantifiable aspects of programming with thisconcurrency model, and describe our experience with us-ing Haskell.

Programming experience: The primary advantage ofour approach is the simplified programming model it pro-vides: threads can be written in a natural sequential style,yet custom, event-driven schedulers can easily be definedby using the trace abstraction. As one example, our sim-ple web server uses a file cache whose state is sharedacross all threads handling client connections. Imple-menting the caching code, modifying the server to use

the cache, and debugging the implementation took undertwo hours. The cache implantation itself takes only 80lines of code.

To test the extensibility of the scheduler (and as partof an ongoing project), we built an application-levelTCP stack in Haskell and plugged it into our thread li-brary. The TCP stack adds only one more event loopto Figure 12; this loop is driven by packet I/O events,timer events, and user thread requests. These schedulerchanges can be made cleanly, without requiring an com-plete rewrite of the code. Having the TCP stack, thethread scheduler and web server in the same user levelapplication, the programmer has complete control of thenetworking code.

Using Haskell: Because Haskell is a pure, lazy, func-tional language, we were initially concerned about per-formance. However, in our experience Haskell pro-grams, while slower than C programs, are not orders ofmagnitude slower. The Computer Language ShootoutBenchmarks [20] give other anecdotal evidence corrob-orating this assessment: Haskell (GHC) performs wellon a wide range of tasks, and many Haskell programsperform better than their corresponding C or C++ pro-grams. When performance or OS libraries are needed,GHC provides good interoperability with C code via itsForeign Function Interface.

In exchange for performance, Haskell delivers manyfeatures that simplify program development, includinga very expressive static type system, type inference,lightweight closures, garbage collection, and convenientsyntax overloading. We heavily use these features in ourthread library implementation; it might be possible toimplement the unified concurrency model in a general-purpose language lacking some of these features, but theresults would likely be cumbersome to use. Neverthe-less, it is worth investigating how to apply our approachin more mainstream languages like Java.

7 Related work

We are not the first to address concurrency problems byusing language-based techniques. There are languagesspecifically designed for concurrent programming, suchas Concurrent ML (CML)[19] and Erlang [4], or forevent-driven programming such as Esterel [6]. Java andC# also provide some support for threads and synchro-nization. There are also domain-specific languages, suchas Flux [5], intended for building network services out ofexisting C libraries. Most of these approaches pick eitherthe multithreaded or event model. Of the ones mentionedabove, CML is closest to our work because it providesvery lightweight threads and an event primitive for con-structing new synchronization mechanisms, but its threadscheduler is still part of the language runtime.

13

The application-level thread library is motivated bytwo projects: SEDA [25] and Capriccio [22]. Ourgoal is to get the best parts from both projects: theevent-driven architecture of SEDA and the multithreadedprogramming style of Capriccio. Capriccio uses com-piler transformations to implement linked stack frames;our application-level threads uses first-class closures toachieve the same effect.

Besides SEDA [25], there are other high-performance,event-driven web servers, such as Flash [18]. Lau-rus and Parkes showed that event-driven systems canbenefit from batching similar operations in different re-quests to improve data and code locality [14]. How-ever, for complex applications, the problem of repre-senting control flow with events becomes challenging.There are libraries and tools designed to make event-driven programs easier by by structuring code in CPS,such as Python’s Twisted package [21] and C++’s Adap-tive Communication Environment (ACE) [1]. Adya etal. [2] present a hybrid approach to automate stack man-agement in C and C++ programming.

Multiprocessor support for user-level threads is a chal-lenging problem. Cilk [7] uses a work-stealing algorithmto map user-level threads to kernel threads. Event-drivensystems, in contrast, can more readily take advantageof multiple processors by processing independent eventsconcurrently [26]. A key challenge is how to determinewhether two pieces of code might interfere: our threadscheduler benefits from the strong type system of Haskelland the use of software transactional memory.

Our thread abstraction is inspired by Claessen’slightweight concurrency model, which also uses CPSmonads and lazy data structures [8]. This paper extendsClaessen’s work with more practical features such as ex-ception handling, inter-thread communication and I/Ointerfaces.

8 Conclusion

Events and threads should be combined into an unifiedprogramming model in general-purpose programminglanguages. With proper language support, application-level threads can be made extremely lightweight and easyto use. Our experiments demonstrate that this approach ispractical and our programming experience suggests thatthis is a very appealing way of writing scalable, mas-sively concurrent software.

References

[1] The ADAPTIVE Communication Environment: Object-OrientedNetwork Programming Components for Developing Clien-t/Server Applications. 11th and 12th Sun Users Group Confer-ence, December 1993 and June 1994.

[2] Atul Adya, Jon Howell, Marvin Theimer, William J. Bolosky, andJohn R. Douceur. Cooperative Task Management without Manual

Stack Management. InProceedings of the 2002 Usenix AnnualTechnical Conference, 2002.

[3] Andrew Appel. Compiling with Continuations. Cambridge Uni-versity Press, 1992.

[4] Joe Armstrong, Robert Virding, Claes Wikstrom, and MikeWilliams. Concurrent Programming in Erlang, Second Edition.Prentice-Hall, 1996.

[5] Emery Berger, Brendan Burns, Kevin Grimaldi, Alex Kostadinov,and Mark Corner. Flux: A Language for Programming High-Performance Servers. InProceedings of the 2006 Usenix AnnualTechnical Conference, 2006.

[6] Gerard Berry and Georges Gonthier. The Esterel SynchronousProgramming Language: Design, Semantics, Implementation.Science of Computer Programming, 19(2):87–152, 1992.

[7] Robert D. Blumofe, Christopher F. Joerg, Bradley C. Kuszmaul,Charles E. Leiserson, Keith H. Randall, and Yuli Zhou. Cilk:Anefficient multithreaded runtime system.Parallel and DistributedComputing, 37(1), August 1996.

[8] Koen Claessen. A poor man’s concurrency monad.Journal ofFunctional Programming, 9(3):313–323, 1999.

[9] The Glasgow Haskell Compiler. http://www.haskell.org/ghc.

[10] Tim Harris, Maurice Herlihy, Simon Marlow, and Simon Peyton-Jones. Composable Memory Transactions. InProceedings of theACM Symposium on Principles andPractice of Parallel Program-ming, to appear, Jun 2005.

[11] Simon L. Peyton Jones and Philip Wadler. Imperative Func-tional Programming. InConference record of the Twentieth An-nual ACM SIGPLAN-SIGACT Symposium on Principles of Pro-gramming Languages, Charleston, South Carolina, pages 71–84,1993.

[12] Simon Peyton Jones, editor.Haskell 98 Language and Libraries:the Revised Report. Cambridge University Press, 2003.

[13] Simon Peyton Jones, Andrew Gordon, and Sigbjorn Finne.Con-current Haskell. InPOPL ’96: The23

rd ACM SIGPLAN-SIGACT Symposium on Principles of Programming Languages,pages 295–308, St. Petersburg Beach, Florida, 21–24 1996.

[14] James R. Larus and Michael Parkes. Using cohort-schedulingto enhance server performance. InUSENIX Annual TechnicalConference, General Track, pages 103–114, 2002.

[15] H.C. Lauer and R.M. Needham. On the Duality of OperatingSys-tems Structures. InProceedings Second International Symposiumon Operating Systems. IRIA, October 1978.

[16] Simon Marlow. Developing a high-performance web server inConcurrent Haskell. Journal of Functional Programming, 12,2002.

[17] J. K. Outsterhout. Why Threads Are A Bad Idea (for most pur-poses). InPresentation given at the 1996 Usenix Annual Techni-cal Conference, 1996.

[18] Vivek S. Pai, Peter Druschel, and Willy Zwaenepoel. Flash: Anefficient and portable Web server. InProceedings of the USENIX1999 Annual Technical Conference, 1999.

[19] J. H. Reppy. Concurrent ML: Design, Application and Semantics.In P. E. Lauer, editor,Functional Programming, Concurrency,Simulation and Automated Reasoning, pages 165–198. Springer,Berlin, Heidelberg, 1993.

[20] The Computer Language Shootout Benchmarks.http://shootout.alioth.debian.org/.

[21] The Twisted Project. http://twistedmatrix.com/.

[22] R. von Behren, J. Condit, F. Zhou, G. Necula, and E. Brewer.Capriccio: Scalable threads for internet services. InProceed-ings of the Ninteenth Symposium on Operating System Principles(SOSP), October 2003.

14

[23] Rob von Behren, Jeremy Condit, and Eric Brewer. Why EventsAre A Bad Idea (for high-concurrency servers). InProceedings ofthe 10th Workshop on Hot Topics in Operating Systems (HotOSIX), May 2003.

[24] P. Wadler. Monads for functional programming. InProceedingsof the Marktoberdorf Summer School on Program Design Cal-culi, August 1992.

[25] Matt Welsh, David E. Culler, and Eric A. Brewer. SEDA: AnArchitecture for Well-Conditioned, Scalable Internet Services. InProceedings of the Symposium on Operating System Principles(SOSP), 2001.

[26] Nickolai Zeldovich, Alexander Yip, Frank Dabek, Robert Morris,David Mazieres, and Frans Kaashoek. Multiprocessor supportfor event-driven programs. InProceedings of the 2003 USENIXAnnual Technical Conference (USENIX ’03), San Antonio, Texas,June 2003.

15