Building Fast and Secure Web Services with OKWS

by

Maxwell Krohn

Submitted to the Department of Electrical Engineering and ComputerScience

in partial fulfillment of the requirements for the degree of

Master of Science in Computer Science and Engineering

at the

MASSACHUSETTS INSTITUTE OF TECHNOLOGY

September 2005

© Massachusetts Institute of Technology 2005. All rights reserved.

Author . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .Department of Electrical Engineering and Computer Science

September 2, 2005

Certified by . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .M. Frans Kaashoek

ProfessorThesis Supervisor

Accepted by. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .Arthur C. Smith

Chairman, Department Committee on Graduate Students

2

Building Fast and Secure Web Services with OKWSby

Maxwell Krohn

Submitted to the Department of Electrical Engineering and Computer Scienceon September 2, 2005, in partial fulfillment of the

requirements for the degree ofMaster of Science in Computer Science and Engineering

Abstract

OKWS is a Web server specialized for secure and fast delivery of dynamic content. Itprovides Web developers with a small set of tools powerful enough to build complex Web-based systems. Despite its emphasis on security, OKWS shows performance improvementscompared to popular systems: when servicing fully dynamic, non-disk-bound databaseworkloads, OKWS’s throughput and responsiveness exceed that of Apache 2 [3], Flash [42]and Haboob [76]. Experience with OKWS in a commercial deployment suggests it canreduce hardware and system management costs, while providing security guarantees absentin current systems. In the end, lessons gleaned from the OKWS project provide insight intohow operating systems might better facilitate secure application design.

Thesis Supervisor: M. Frans KaashoekTitle: Professor

3

4

Acknowledgments

I thank the following contributors to this thesis: David Mazieres, for his help throughoutthe OKWS project; Robert Morris and Russ Cox, for their help in debugging; MichaelWalfish, Eddie Kohler, and the anonymous Usenix 2004 reviewers for their help on theoriginal OKWS Usenix paper; Sarah Friedberg and Jeremy Stribling for proofreading; andthe programmers, writers and designers at OkCupid.com—Patrick Crosby, Jason Yung,Chris Coyne, Alfred Perlstein, Christian Rudder and Sam Yagan—for adopting and im-proving OKWS. The gzip work in Appendix A was done in collaboration with abhi shelat.Work done on making changes to the underlying operating system is done as part of the As-bestos Project, a collaboration with Petros Efstathopoulos, Steve VanDeBogart, Cliff Frey,David Ziegler, Eddie Kohler, David Mazieres, Frans Kaashoek and Robert Morris. Finally,I thank my advisor Frans Kaashoek.

This research was supported in part by the DARPA Composable High Assurance TrustedSystems program (BAA #01-24), under contract #N66001-01-1-8927.

5

6

Contents

1 Introduction 131.1 A Solution . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 131.2 Contributions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 141.3 Thesis Organization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14

2 Related work 15

3 A Brief Survey of Web Server Bugs 173.1 Apache Core and Standard Modules . . . . . . . . . . . . . . . . . . . . . 173.2 Scripting Extensions to Apache . . . . . . . . . . . . . . . . . . . . . . . . 18

4 Design 194.1 Four Practical Security Guidelines . . . . . . . . . . . . . . . . . . . . . . 194.2 OKWS Design . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 214.3 Process Isolation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22

5 Implementation 255.1 okld . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 255.2 okd . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 275.3 oklogd . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 285.4 pubd . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 285.5 Asynchronous Database Proxies . . . . . . . . . . . . . . . . . . . . . . . 295.6 Building A Web Service . . . . . . . . . . . . . . . . . . . . . . . . . . . 305.7 Code Size . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32

6 OkCupid.com: OKWS In Action 336.1 Separating Static From Dynamic . . . . . . . . . . . . . . . . . . . . . . . 336.2 Compression . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34

7 Python Support in OKWS 357.1 Approach . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 367.2 Implementation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36

7.2.1 Wrappers for the SFS Libraries . . . . . . . . . . . . . . . . . . . 367.2.2 OKWS Glue . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37

7.3 Launching Python Services and Security . . . . . . . . . . . . . . . . . . . 38

7

7.4 Jailing Python . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40

8 Performance Evaluation 418.1 Testing Method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 418.2 Experimental Setup . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 428.3 OKWS Process Pool Tests . . . . . . . . . . . . . . . . . . . . . . . . . . 428.4 Web Server Comparison . . . . . . . . . . . . . . . . . . . . . . . . . . . 43

8.4.1 Single-Service Workload . . . . . . . . . . . . . . . . . . . . . . . 448.4.2 Many-Service Workload . . . . . . . . . . . . . . . . . . . . . . . 45

9 Security Analysis 479.1 Security Benefits . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 479.2 Security Shortcomings . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48

10 Discussion and Directions for Future Research 5110.1 Restricting the Unix System Call Interface . . . . . . . . . . . . . . . . . . 5110.2 Alternatives to Unix . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52

10.2.1 Naming and Managing Capabilities . . . . . . . . . . . . . . . . . 5310.2.2 OKWS Under Unestos . . . . . . . . . . . . . . . . . . . . . . . . 53

10.3 Fine-Grained POLP with MAC . . . . . . . . . . . . . . . . . . . . . . . . 54

11 Conclusion 57

A Speeding Up gzip Compression for Dynamic Webservers 59A.1 The gzip Algorithm . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 59A.2 Cacheable Compression . . . . . . . . . . . . . . . . . . . . . . . . . . . . 60A.3 Related Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62

B Jailing Python for OKWS 63

8

List of Figures

4.1 An example server setup, in which Web clients download HTML data froma firewalled clusted, and then are redirected to a content distribution net-work (CDN) or external image servers for downloading graphics. . . . . . . 20

4.2 Block diagram of an OKWS site setup with three Web services (svc1, svc2, svc3)and two data sources (data1, data2), one of which (data2) is an OKWSdatabase proxy. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22

4.3 Dependency graphs for various Web server architectures. . . . . . . . . . . 23

5.1 A typical Database Proxy for a MySQL database, configured with fiveworker threads. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29

5.2 Example of database proxy code with MySQL wrapper library. In thiscase, the Web developer is loading SQL results directly into an RPC XDRstructure. Error conditions ignored for brevity. The user xdt t structureis defined in the protocol file in Figure 5.3. . . . . . . . . . . . . . . . . . 30

5.3 user.x: the RPC protocol specification file used in client/server examplesfor this section. The program has one RPC, GET USER, which takes asinput an unsigned user ID and outputs a user t structure. . . . . . . . 31

5.4 Fragment of a Web service programmed in OKWS. The remote client sup-plies the title and color of the page via standard CGI-style parameter pass-ing. The runtime templating system substitutes the user’s choice of colorfor the token COLOR in the template /body.html. The variable my svc t::resprepresents a buffer that collects the body of the HTTP response and thenis flushed to the client via output(). With the FilterCGI flag set,OKWS filters all dangerous metacharacters from the param associativearray. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32

7.1 A sample Python application using SFS libraries . . . . . . . . . . . . . . . 377.2 An example OKWS service implemented in Python. No exceptions are

caught for brevity. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39

8.1 PHP version of the null service . . . . . . . . . . . . . . . . . . . . . . . . 418.2 Throughputs achieved in the process pool test . . . . . . . . . . . . . . . . 428.3 Context switching in the process pool test . . . . . . . . . . . . . . . . . . 438.4 Throughputs for the single-service test . . . . . . . . . . . . . . . . . . . . 448.5 Median latencies in the single-service test . . . . . . . . . . . . . . . . . . 448.6 Client latencies for 1,600 concurrent clients in the single-service test . . . . 45

9

8.7 Throughputs for the many-service workload . . . . . . . . . . . . . . . . . 46

A.1 A simple fragment of PHP code. . . . . . . . . . . . . . . . . . . . . . . . 60A.2 A simple, incomplete cacheable LZ77 algorithm. Several important corner

cases are elided for clarity. The notation ||Mn|| gives the length of stringMn. 61

10

List of Tables

5.1 An example configuration of OKWS. The entries in the “run directory”column are relative to “chroot jails”. . . . . . . . . . . . . . . . . . . . . . 26

8.1 Average throughputs in connections per second . . . . . . . . . . . . . . . 46

11

12

Chapter 1

Introduction

Many of today’s dynamic Web sites are nothing more than HTTP interfaces to centralized,managed databases. As such they are gatekeepers, tasked with showing Alice only thesmall part of the database that pertains to her, while keeping Bob and Charlie’s private dataout of her reach. This task is critical and difficult in practice. One need only browse the ITheadlines to get a taste for the flagrant access control failures that plague commercial WebSites [47, 48]. Such attacks have yielded millions of e-mail addresses in some cases, andmillions of credit card accounts in others. Attacks against Web sites often exploit weak-nesses in popular Web servers or bugs in custom application-level logic. In theory, HTTP(or HTTPS) exposes narrow interfaces to remote clients and should not allow the litany ofattacks that have surfaced over the years. In practice, emphasis on rapid deployment andperformance often comes at the expense of security.

Consider the following example: Web servers typically provide Web programmerswith powerful and generic interfaces to underlying databases and rely on coarse-graineddatabase-level permission systems for access control. Web servers also tend to packagelogically separate programs into one address space. If a particular Web site serves its searchand newsletter-subscribe features from the same machine, a bug in the former might allowa malicious remote client to select all rows from a table of subscribers’ email addresses. Ingeneral, anything from a buffer overrun to an unexpected escape sequence can expose pri-vate data to an attacker. Moreover, few practical isolation schemes exist aside from runningdifferent services on different machines. A large Web site, serving thousands of differentfunctions, might only be as strong as it weakest service.

1.1 A Solution

To plug the many security holes that plague existing Web servers, and to limit the severityof unforeseen problems, we have designed, implemented and deployed OKWS, the OKWeb Server. Unlike typical Web servers, OKWS is specialized for dynamic content and iscurrently not well-suited to serving files from disk. It relies on existing Web servers, such asFlash [42] or Apache [3], to serve images and other static content. In deployment, we foundthat this separation of static and dynamic content is natural and, moreover, contributes tosecurity.

13

Supporting dynamic content on Unix necessitates a trade-off between security and per-formance. On the one hand, the most secure Web architecture on Unix is one in whichremote Web clients connect to a single dedicated server-side process. The user’s data onthe server, whether in memory or on disk, is only available to that one process. Thisarchitecture is impractical: storage constraints dictate that user data be aggregated intosearchable databases; the operating system cannot handle too many active processes, norcan it afford to fork processes as users connect. On the other hand, most Web servers in theliterature (such as Apache, Flash, and Haboob [76]) assign processes to users and servicesto maximize performance and without regard to security.

OKWS chooses a different point in the design space: it confines each service that aWeb server runs to a single process, so that if there are any security problems in one ser-vice, they will not spread to others. OKWS’s alignment of services and processes is animplementation of the natural principle of least privilege [54], beneficial in this case forboth performance and security.

1.2 Contributions

The main contributions of this thesis are the design and implementation of the OKWS, andan analysis of its security, performance, and usability. We believe OKWS has carved out anew niche in the crowded Web server design space, achieving security properties that otherWeb servers have not, while still outperforming all the servers we tested it against. Nomere academic exercise, OKWS is a real system currently used in at least one large-scalecommercial Web site; details about OKWS’s commercial deployment are discussed. Forbusinesses and academics alike, OKWS is freely available under an open source license.

Though OKWS goes to great lengths to squeeze the maximum set of security propertiesout of the Unix interface, in some respects, it still falls short. Experience with OKWSsuggests that writing secure Web servers is hard, and that the operating system shouldprovide better security primitives to secure application programmers. This thesis sketchesthe more promising experimental OS designs that would make OKWS simpler and moresecure, without sacrificing its current features, usability or performance.

1.3 Thesis Organization

The next two chapters examine popular Web technology, arguing that today’s Web servershave a bad track record in security and do not seem to be improving. Yet, the simple designprinciples in Chapter 4 would go a long way to closing many Web server vulnerabilities,and Chapter 5 discusses OKWS’s implementation of these principles. Chapters 6 and 7argue that the resulting system is practical for building large systems, and Chapter 8 an-alyzes its performance, showing that OKWS’s specialization for dynamic content helps itachieve better performance in simulated dynamic workloads than general purpose servers.Chapter 9 discusses the security achieved by the implementation, and Chapter 10 coversthe overarching lessons learned from two years of experience with OKWS and exploresnew ideas in operating systems inspired by the work.

14

Chapter 2

Related work

There is a wide range of related work in Web servers—both static and dynamic. Many Webservers do not specialize in dynamic content, and none to our knowledge have achievedOKWS’s security and performance properties.

Despite its problems, Apache is today’s most popular popular Web Server [3]. Apache’smany configuration options and modules allow Web programmers to extend its function-ality with a variety of different programming languages. However, neither 1.3.x’s multi-process architecture nor 2.0.x’s multi-threaded architecture is conducive to process iso-lation. Also, its extensibility and mushrooming code base make its security propertiesdifficult to reason about.

Highly-optimized event-based Web servers such as Flash [42] and Zeus [79] haveeclipsed Apache in terms of performance. While Flash in particular has a history of out-standing performance serving static content, our performance studies here indicate that itsarchitecture is less suitable for dynamic content. In terms of process isolation, one couldmost likely implement a similar separation of privileges in Flash as we have done withOKWS. However, as shown in Chapter 8, Flash’s architecture might not be well-suited fordynamic content.

FastCGI [20] is a standard for implementing long-lived CGI-like helper processes. Itallows separation of functionality along process boundaries but neither articulates a specificsecurity policy nor specifies the mechanics for maintaining process isolation in the face ofpartial server compromise. Also, FastCGI requires the leader process to relay messagesbetween the Web service and the remote client. OKWS passes file descriptors to avoid theoverhead associated with FastCGI’s relay technique.

The Haboob server studied here is one of many possible applications built on SEDA, anarchitecture for event-based network servers. In particular, SEDA uses serial event queuesto enforce fairness and graceful degradation under heavy load. Larger systems such asNinja [62] build on SEDA’s infrastructure to create clusters of Web servers with the sameappealing properties. However, performance measurements in Chapter 8 show that Haboobachieves between one fifth and one tenth the throughput of OKWS despite a strictly worsesecurity model.

Other work has used the SFS toolkit to build static Web Servers and Web proxies [78].Though the current OKWS architecture is well-suited for SMP machines, the adoption oflibasync-mp would allow for finer-grained sharing of a Web workload across many CPUs.

15

OKWS uses events but the same results are possible with an appropriate threads library.An expansive body of literature argues the merits of one scheme over the other, and mostrecently, Capriccio’s authors [63] argue that threads can achieve the same performance asevents in the context of Web servers, while providing programmers with a more intuitiveinterface. Other recent work suggests that threads and events can coexist [1]. Such tech-niques, if applied to OKWS, would simplify stack management for Web developers.

In addition to the PHP [44] scripting language investigated here, many other Web de-velopment environments are in widespread use. Zope [81], a Python-based platform, hasgained popularity due to its modularity and support for remote collaboration. CSE [25]allows developers to write Web services in C++ and uses some of the same sandboxingschemes we use here to achieve fault isolation. In more commercial settings, Java-basedsystems often favor thin Web servers, pushing more critical tasks to application serverssuch as JBoss [30] and IBM WebSphere [28]. Such systems limit a Web server’s accessto underlying databases in much the same way as OKWS’s database proxies. Most Javasystems, however, package all aspects of a system in one address space with many threads;our model for isolation would not extend to such a setting. Furthermore, our experimen-tal results indicate significant performance advantages of compiled C++ code over Javasystems.

Other work has proposed changes to underlying operating systems to make Web serversfast and more secure. The Exokernel operating system [31] allows its Cheetah Web serverto directly access the TCP/IP stack, in order to reduce buffer copies allow for more effectivecaching. The Denali isolation kernel [77] can isolate Web services by running them onseparate virtual machines.

16

Chapter 3

A Brief Survey of Web Server Bugs

To justify our approach to dynamic Web server design, we briefly analyze the weaknessesof popular software packages. The goal is to represent the range of bugs that have arisenin practice. Historically, attackers have exploited almost all aspects of conventional Webservers, from core components and scripting language extensions to the scripts them-selves. The conclusion we draw is that a better design—as opposed to a more correctimplementation—is required to get better security properties.

In our survey, we focus on the Apache [3] server due to its popularity, but the types ofproblems discussed are common to all similar Web servers, including IBM WebSphere [28],Microsoft IIS [38] and Zeus [79].

3.1 Apache Core and Standard Modules

There have been hundreds of major bugs in Apache’s core and in its standard modules.They fit into the following categories:

Unintended Data Disclosure. A class of bugs results from Apache delivering files overHTTP that are supposed to be private. For instance, a 2002 bug in Apache’s mod davreveals source code of user-written scripts [74]. A 2003 bug in a default installation ofApache Tomcat on Gentoo Linux publicly revealed authentication credentials [13]. An-other recent vulnerability in Apache Tomcat causes the server to return unprocessed sourcecode, as opposed to executing the source code and outputting the result [73]. Similarly,Tomcat’s improper handling of null bytes (%00) and escape sequences has allowed remoteattackers to view hidden directory contents [66].

A recent discovery of leaked file descriptors allows remote users to access sensitivelog information [12]. On Mac OS X operating systems, a local find-by-content indexingscheme creates a hidden yet world-readable file called .FBCIndex in each directory in-dexed. Versions of Apache released in 2002 expose this file to remote clients [72]. In allcases, attackers can use knowledge about local configuration and custom-written applica-tion code to mount more damaging attacks.

17

Buffer Overflows and Remote Code Execution. Buffer overflows in Apache and itsmodules are common. Unchecked boundary conditions found recently in mod aliasand mod rewrite regular expression code allow local attack [69]. In 2002, a commonApache deployment with OpenSSL had a critical bug in client key negotiation, allowingremote attackers to execute arbitrary code with the permissions of the Web server. Theattacking code downloads, compiles and executes a program that seeks to infect other ma-chines [65].

There have been less-sophisticated attacks that resulted in arbitrary remote code exe-cution. Some Windows versions of Apache execute commands in URLs that follow pipecharacters (‘|’). A remote attacker can therefore issue the command of his choosing froman unmodified Web browser [71]. On MS-DOS-based systems, Apache failed to filter outspecial device names, allowing carefully-crafted HTTP POST requests to execute arbitrarycode [75]. Other problems have occurred when site developers call Apache’s htdigestutility from within CGI scripts to manage HTTP user authentication [10].

Denial of Service Attacks. Aside from TCP/IP-based DoS attacks, Apache has been vul-nerable to a number of application-specific attacks. Apache versions released in 2003 failedto handle error conditions on certain “rarely used ports,” and would stop servicing incom-ing connections as a result [68]. Another 2003 release allowed local configuration errorsto result in infinite redirection loops [14]. In some versions of Apache, attackers couldexhaust Apache’s heap simply by sending a large sequence of linefeed characters [67].

3.2 Scripting Extensions to Apache

Apache’s security worsens considerably when compiled with popular modules that enabledynamically-generated content such as PHP [44]. In the past two years alone, at least 13buffer overruns have been found in the PHP core, some of which allowed attackers to re-motely execute arbitrary code [16,56]. In six other cases, faults in PHP allowed attackers tocircumvent its application level chroot-like environment, called “Safe Mode.” One vulner-ability exposed /etc/passwd via posix getpwnam [9]. Another allowed attackers towrite PHP scripts to the server and then remotely execute them; this bug persisted acrossmultiple releases of PHP intended as fixes [64].

Even if a correct implementation of PHP were possible, it would still provide Webprogrammers with ample opportunity to introduce their own vulnerabilities. A canonicalexample is that beginning PHP programmers fail to check for sequences such as “..” inuser input and therefore inadvertently allow remote access to sensitive files higher up inthe file system hierarchy (e.g., ../../../etc/passwd). Similarly, PHP scripts thatembed unescaped user input inside SQL queries present openings for “SQL Injection.” If aPHP programmer neglects to escape user input properly, a malicious user can turn a benignSELECT into a catastrophic DELETE.

The PHP manual does state that PHP scripts might be separated and run as differentusers to allow for privilege separation. In this case, however, PHP cannot run as an Apachemodule, and the system requires a new PHP process forked for every incoming connection.This isolation strategy compromises performance, and is seldom used in practice.

18

Chapter 4

Design

If we assume that bugs like the ones discussed in Chapter 3 are inevitable when building alarge system, the best remedy is to limit the effectiveness of attacks when they occur. Thissection presents four simple guidelines for protecting sensitive site data in the worst-casescenario, in which an adversary remotely gains control of a Web server and can executearbitrary commands with the Web server’s privileges. We also present OKWS’s design,which follows the four security guidelines without sacrificing performance.

Throughout, we assume a cluster of dynamic Web servers and database machines con-nected by a fast, firewalled LAN. Site data is cached at the Web servers and persistentlystored on the database machines. The site’s static data can be served from outside of thefirewall since static content servers typically do not require access to sensitive site data.This type of configuration, exemplified in Figure 4.1, is common for high-volume Websites. With this server setup, the primary security goals are to prevent intrusion into thefirewalled cluster and to prevent unauthorized access to site data.

4.1 Four Practical Security Guidelines

(1) Server processes should be chrooted. chrooting privileged server processes is acommonly-accepted though infrequently-applied Unix security technique.1 The rationalebehind chroot is that privileged processes, if compromised, would do less damage ifthey could not access sensitive parts of the file system. In particular, secure systems canuse chroot to hide local passwords, private keys, startup scripts, commonly-executed ex-ecutable images (such as the kernel) and local configuration files from privileged servers. Ifa privileged server cannot access these files due to chroot, then a compromised privilegedserver cannot learn their secrets or sully their integrity.

From a more technical perspective, OS-level chroot-jails ought to hide all setuid2

executables from the Web server, to prevent compromised processes from escalating priv-

1chroot is a system call, available only to the superuser, that changes an application’s idea of what thefile system root is. Calling chroot(dir) renders only the files and directories under dir accessible to theprocess [52].

2If a file is set to setuid, then any process that executes that file has its user ID set to the file’s owner [52].Typically, su is owned by the superuser and is marked setuid, allowing non-privileged users to execute itand become privileged.

19

Figure 4.1: An example server setup, in which Web clients download HTML data froma firewalled clusted, and then are redirected to a content distribution network (CDN) orexternal image servers for downloading graphics.

ileges (examples include the ptrace and bind attacks mentioned in [32]). Privilege escala-tion is possible without setuid executables but requires OS-level bugs or race conditions(such as a recently discovered flaw in Linux’s mremap system call [70]) that are typicallyrarer.

Moreover, developers should design their application around the chroot call as op-posed to blindly applying a chroot container ex-post facto. Consider the Apache exam-ple. Once chrooted, Apache still needs access to configuration files, source files, andbinaries that correspond to services the site administrator wants to make available; thesefiles, in turn, are valuable to attackers who can compromise the server. For instance, PHPscripts often include the username and plaintext password used to gain access to a MySQLdatabase. If an attacker can control Apache, he can read PHP scripts and wreak havoc bycontrolling the databases those PHP scripts access. OS-enforced policy ought to hide evenapplication-specific source, binary and configuration files from running Web servers.

(2) Server processes should run as unprivileged users. To do otherwise—to run a serverprocess as a privileged user— opens to the door to significant damage, even if the processis chrooted. A privileged, chrooted process can bind to a well-known network port. A priv-ileged process might also interfere with other system processes, especially those associatedwith the Web server, by tracing their system calls, sending them signals or tampering withtheir binaries so that they exhibit unintended behavior on future executions.

(3) Server processes should have the minimal set of database access privileges nec-essary to perform their task. Separate processes should not have access to each other’sdatabases. Moreover, if a Web server process requires only row-wise access to a table, anadversary who compromises it should not have the authority to perform operations over theentire table.

(4) A server architecture should separate independent functions into independent pro-cesses. An adversary who compromises a Web server can examine its in-memory data

20

structures, which might contain soft state used for user session management, or possiblysecret tokens that the Web server uses to authenticate itself to its database. With control of aWeb server process, an adversary might hijack an existing database connection or establisha new one with the authentication tokens it acquired. Though more unlikely, an attackermight also monitor and alter network traffic entering and exiting a compromised server.

The important security principle here is to limit the types of data that a single processcan access. Site designers should partition their global set of site data into small, self-contained subsets, and their Web server ought to align its process boundaries with thispartition.

If a Web server implements principles (1) through (4), and if there are no critical kernelbugs, an attacker cannot move from vulnerable to secure parts of the system. The samedefenses also protect careless Web programmers from their own mistakes. For example,if a server architecture denies a successful attacker access to /etc/passwd, then a pro-grammer cannot inadvertently expose this file to remote clients. Similarly, if a successfulattacker cannot arbitrarily access underlying databases, then even a broken Web script can-not enable SQL injection attacks.

4.2 OKWS Design

We designed OKWS with these four principles in mind. OKWS provides Web developerswith a set of libraries and helper processes so they can build Web services as independent,stand-alone processes, isolated almost entirely from the file system. The core librariesprovide basic functions of receiving HTTP requests, accessing data sources, composingan HTML-formatted response, responding to HTTP requests, and logging the results todisk. A process called OK launcher daemon, or okld, launches custom-built services andrelaunches them should they crash. A process called OK dispatcher, or okd, routes incom-ing requests to appropriate Web services based on URL. A helper process called pubdprovides Web services with limited read access to configuration files and HTML templatefiles stored on the local disk. Finally, a dedicated logger daemon called oklogd writes logentries to disk. Figure 4.2 summarizes these relationships.

This architecture allows custom-built Web services to meet our stated design goals:

(1) OKWS chroots all services to a remote jail directory. Within the jail, each processhas just enough access privileges to read shared libraries upon startup and to dumpcore upon abnormal termination. The services otherwise never access the file systemand lack the privileges to do so.

(2) Each service runs as a unique non-privileged user.

(3) OKWS interposes a structured RPC interface between the Web service and the databaseand uses a simple authentication mechanism to align the partition among databaseaccess methods with the partition among processes.

(4) Each Web service runs as a separate process. The next section justifies this choice.

21

Figure 4.2: Block diagram of an OKWS site setup with three Web services (svc1, svc2, svc3)and two data sources (data1, data2), one of which (data2) is an OKWS database proxy.

4.3 Process Isolation

Unlike the other three principles, the fourth, of process isolation, implies a security andperformance tradeoff since the most secure option—one Unix process per external user—would be problematic for performance. OKWS’s approach to this tradeoff is to assign oneUnix process per service.

Our approach is to view Web server architecture as a dependency graph, in which thenodes represent processes, services, users, and user state. An edge (a, b) denotes b’s depen-dence on a, meaning an attacker’s ability to compromise a implies an ability to compromiseb. The crucial design decision is thus how to establish dependencies between the more ab-stract notions of services, users and user states, and the more concrete notion of a process.

Let the set S represent a Web server’s constituent services, and assume each serviceaccesses a private pool of data. Two services are defined as distinct if they access disjointpools of data—whether in-memory soft state or database-resident hard state. A set of usersU interacts with these services, and the interaction between user uj and service si involvesa piece of state ti,j. If an attacker can compromise a service si, he can compromise stateti,j for all j; thus (si, ti,j) is a dependency for all j. Compromising state also compromisesthe corresponding user, so (ti,j, uj) is also a dependency.

Let P = {p1, . . . , pk} be a Web server’s pool of processes. The design decision of howto allocate processes reduces to where the nodes in P belong on the dependency graph. Inthe Apache architecture [3], each process pi in the process pool can perform the role ofany service sj. Thus, dependencies (pi, sj) exist for all j. For the Flash Web server [42],each process in P is associated with a particular service: for each pi, there exists sj such

22

u1 u2

t1,1 t2,1 t1,2 t2,2

s1 s2

p1 p2 p3

(a) Apache

u1 u2

t1,1 t2,1 t1,2 t2,2

s1 s2

p1 p2 p3 p4

(b) Flash

u1 u2

t1,1 t2,1 t1,2 t2,2

s1 s2

p1

(c) Haboob

u1 u2

t1,1 t2,1 t1,2 t2,2

s1 s2

p1 p2

(d) Strict

u1 u2

t1,1 t2,1 t1,2 t2,2

s1 s2

p1 p2

(e) OKWS

Figure 4.3: Dependency graphs for various Web server architectures.

that (pi, sj) is a dependency. The size of the process pool P is determined by the numberof concurrent active HTTP sessions; each process pi serves only one of these connections.Java-based systems like the Haboob Server [76] employ only one process; thus P = {p1},and dependencies (p1, sj) exist for all j.

Figures 4.3(a)-(c) depict graphs of Apache, Flash and Haboob hosting two services fortwo remote users. Assuming that the “dependence” relationship is transitive, and that anadversary can compromise p1, the shaded nodes in the graph show all other vulnerableentities.

This picture assumes that the process of p1 is equally vulnerable in the different archi-tectures and that all architectures succeed equally in isolating different processes from eachother. Neither of these assumptions is entirely true, and we will return to these issues inSection 9.2. What is clear from these graphs is that in the case of Flash, a compromise ofp1 does not affect states t2,1 and t2,2. For example, an attacker who gained access to ui’ssearch history (t1,i) cannot access the contents of his inbox (t2,i).

A more strict isolation strategy is shown in Figure 4.3(d). The architecture assigns aprocess pi to each user ui. If the attacker is a user ui, he should only be able to compromisehis own process pi, and will not have access to state belonging to other users uj. Theproblem with this approach is that it does not scale well on current operating systems. AWeb server would either need to fork a new process pi for each incoming HTTP requestor would have a large pool of mostly idle processes, one for each currently active user (ofwhich there might be tens of thousands).

OKWS does not implement the strict isolation strategy but instead associates a singleprocess with each individual service, shown in Figure 4.3(e). As a result OKWS achievesthe same isolation properties as Flash but with a process pool whose size is independentof the number of concurrent HTTP connections. OKWS thus requires significantly fewerprocesses than Flash under heavy load.

23

24

Chapter 5

Implementation

OKWS is a portable, event-based system, written in C++ with the SFS toolkit [37]. It hasbeen successfully tested on Linux and FreeBSD. In OKWS, the different helper processesand site-specific services shown in Figure 4.2 communicate among themselves with SFS’simplementation of Sun RPC [61]; they communicate with external Web clients via HTTP.Unlike other event-based servers [42, 76, 79], OKWS exposes the event architecture toWeb developers. We could most likely have achieved equivalent security and performanceresults through the use of cooperative user-level threading, or perhaps through a hybridscheme [1].

To use OKWS, an administrator installs the helper binaries (okld, okd, pubd and oklogd)to a standard directory such as /usr/local/sbin, and installs the site-specific servicesto a runtime jail directory, such as /var/okws/run. The administrator should allocatetwo new UID/GID pairs for okd and oklogd and should also reserve a contiguous user andgroup ID space for “anonymous” services. Finally, administrators can tweak the masterconfiguration file, /etc/okws config. Table 5.1 summarizes the runtime configurationof OKWS.

5.1 okld

The root process in the OKWS system is okld—the launcher daemon. This process nor-mally runs as superuser but can be run as a non-privileged user for testing or in othercases when the Web server need not bind to a privileged TCP port. When okld starts up, itreads the configuration file /etc/okws config to determine the locations of the OKWShelper processes, the anonymous user ID range, which directories to use as jail directories,and which services to launch. Next, okld launches the logging daemon (oklogd) and thedemultiplexing daemon (okd), and chroots into its runtime jail directory. It then launchesall site-specific Web services. The steps for launching a single service are:

1. okld requests a new Unix socket connection from oklogd.

2. okld opens 2 socket pairs; one for HTTP connection forwarding, and one for RPCcontrol messages.

3. okld calls fork.

25

process chroot jail run directory uid gid

okld /var/okws/run / root wheelpubd /var/okws/htdocs / www www

oklogd /var/okws/log / oklogd oklogdokd /var/okws/run / okd okd

svc1 /var/okws/run /cores/51001 51001 51001svc2 /var/okws/run /cores/51002 51002 51002svc3 /var/okws/run /cores/51003 51003 51003

Table 5.1: An example configuration of OKWS. The entries in the “run directory” columnare relative to “chroot jails”.

4. In the child address space, okld picks a fresh UID/GID pair (x.x), sets the new pro-cess’s group list to {x} and its UID to x. It then changes directories into /cores/x.

5. Still in the child address space, okld calls execve, launching the Web service. Thenew Web service process inherits three file descriptors: one for receiving forwardedHTTP connections, one for receiving RPC control messages, and one for RPC-basedrequest logging. Some configuration parameters in /etc/okws config are rele-vant to child services, and okld passes these to new children via the command line.

6. In the parent address space, okld sends the server side of the sockets opened in Step 2to okd.

Upon a service’s first launch, okld assigns it a group and user ID chosen arbitrarily fromthe given range (e.g., 51001-51080). The service gets those same user and group IDs insubsequent launches. It is important that no two services share a UID or GID, and okldensures this invariant. The service executables themselves are owned by root, belong to thegroup with the anonymous GID x chosen in Step 4 and are set to mode 0410.

These settings allow okld to call execve after setuid but disallow a service processfrom changing the mode of its corresponding binary. okld changes the ownerships andpermissions of service executables at launch if they are not appropriately set. The directoryused in Step 4 is the only one in the jailed file system to which the child service can write.If such a directory does not exist or has the wrong ownership or permissions, okld createsand configures it accordingly.

okld catches SIGCHLD when services die. Upon receiving a non-zero exit status, okldchanges the owner and mode of any core files left behind, rendering them inaccessible toother OKWS processes. If a service exits uncleanly too many times in a given interval, okldwill mark it broken and refuse to restart it. Otherwise, okld restarts dead services followingthe steps enumerated above.

okld runs as super-user but unlike other OKWS daemons does not drop its privileges.However, it has many fewer potential vulnerabilities in that it does not listen for any explicitmessages—RPC, HTTP or otherwise. Rather, it responds only to SIGCHLD signals.

26

5.2 okd

The okd process accepts incoming HTTP requests and demultiplexes them based on the“Request-URI” in their first lines. For example, the HTTP/1.1 standard [21] defines thefirst line of a GET request as:

GET /〈abs path〉?〈query〉 HTTP/1.1

Upon receiving such a request, okd looks up a Web service corresponding to abs pathin its dispatch table. If successful, okd forwards the remote client’s file descriptor to therequested service. If the lookup is successful but the service is marked “broken,” okd sendsan HTTP 500 error to the remote client. If the request did not match a known service,okd returns an HTTP 404 error. In typical settings, a small and fixed number of theseservices are available—on the order of 10. The set of available services is fixed once okdreads its configuration file at launch time. These restrictions would prove inconvenient fora traditional static Web server, because site maintainers frequently add, rename and deletestatic content. In our setting, we assume that site developers add, rename and delete Webservices infrequently.

Upon startup, okd reads the OKWS configuration file (/etc/okws config) to con-struct its dispatch table. okd inherits two file descriptors from okld: one is a socket thatconnects to oklogd, the other is a pipe across which okld will send it control messages. okduses its connection to the logger to log errors messages such as error 404 (file not found)or error 500 (internal server error). okd uses the pipe from okld to listen for informationabout Web services that are starting up. When okld starts up a service, it creates two socketpairs, one for control messages, and one for HTTP connections. okld gives one half ofthese connections to the service (see Section 5.1, Step 6). It sends the other half of thesesocket pairs to okd over the okld- to - okd pipe.

Thus, okd receives two file descriptors from okld for each service launched. okd usesone of these sockets to send service control messages. It uses the other socket to pass filedescriptors after successfully demultiplexing incoming HTTP requests.

okd typically is launched with supervisor privileges but drops them (via setuid) afterbinding to its listen port and chrooting into its designated runtime jail directory. For testingpurposes, okd can run as a non-privileged user, assuming it does not need to bind to aprivileged port (such as a TCP port greater than 1024). If the given configuration uses alocal pubd, then okd must also launch pubd before dropping its privileges.

In addition to listening for HTTP connections on port 80, okd listens for internal re-quests from an administration client. It services the two RPC calls: REPUB and RE-LAUNCH. A site maintainer should call the former to “activate” any changes she makesto HTML templates (see Section 5.4 for more details). Upon receiving a REPUB RPC, okdtriggers a simple update protocol that propagates updated templates.

A site maintainer should issue a RELAUNCH RPC after updating a service’s binary.Upon receiving a RELAUNCH RPC, okd simply sends an EOF to the relevant service onits control socket. When a Web service receives such an EOF, it finishes responding toall pending HTTP requests, flushes its logs, and then exits cleanly. The launcher daemon,okld, then catches the corresponding SIGCHLD and restarts the service.

27

5.3 oklogd

All services, along with okd, log their access and error activity to local files via oklogd—the logger daemon. Because these processes lack the privileges to write to the same log filedirectly, they instead send log updates over a local Unix domain socket. To reduce the totalnumber of messages, services send log updates in batches. Services flush their log buffersas they become full and at regularly-scheduled intervals.

Because only oklogd writes to the log file, we expect performance improvements overmulti-process architectures, in which atomic appends to log files can only be guaranteedthrough kernel-level synchronization or spin-locks. For a busy Webserver, we would ex-pect frequent contention over log file resources. But batched logging via oklogd has twoimportant downsides. First, logs are not locally ordered unless oklogd sorts and mergesbatches from different processes. To avoid the additional computation and data manipula-tion involved with a run-time sort, OKWS defers the responsibility of completely orderingthe log files to the offline tools.

Second, and more important, if a Web service crashes, its buffered batch of log entriesis lost. In the deployments we have seen, we do not anticipate the loss of a few seconds oflog entries to be a cause for concern. In our current implementation, the cause of the crashis discernible from the core dump that the service produces upon abnormal termination.In fact, for large HTTP POSTs that trigger server crashes, a core dump contains moreinformation about the offending request than a single HTTP log line. Future versions ofOKWS might ship log entries to oklogd via memory-mapped files (one for each service);in this case, the logger can access a service’s cached log entries after it has crashed.

For security, oklogd runs as an unprivileged user in its own chroot environment. Also,OKWS runs oklogd as a different user from the okd user. The combination of these twomethods ensures that a compromised okd or Web service cannot maliciously overwrite ortruncate log files; it would only have the ability to fill them with “noise”.

5.4 pubd

Dynamic Web pages often contain large sections of static HTML code. In OKWS, suchstatic blocks are called HTML “templates”; they are stored as regular files, can be shared bymultiple services and can include each other in a manner similar to Server Side Includes [4].

OKWS services do not read templates directly from the file system. Rather, uponstartup, the publishing daemon (pubd) parses and caches all required templates. It thenships parsed representations of the templates over RPC to other processes that requirethem. pubd runs as an unprivileged user, relegated to a jail directory that contains onlypublic HTML templates. As a security precaution, pubd never updates the files it serves,and administrators should set its entire chrooted directory tree read-only (perhaps, on thoseplatforms that support it, by mounting a read-only nullfs).

Administrators in a cluster setting might chose to run pubd locally on an NFS-mountedtemplate directory. pubd also runs as a network service so that a single pubd instance canserve a cluster of machines.

28

Figure 5.1: A typical Database Proxy for a MySQL database, configured with five workerthreads.

5.5 Asynchronous Database Proxies

OKWS’s library implementation is agnostic to particular flavors of threading libraries andcurrently allows the site developer to chose from one of three possibilities. For instance, asite developer would choose kernel threads when implementing a database proxy for inter-facing with an embedded database such as Berkeley DB and would choose cooperative userthreads for a socket-based database such as MySQL. The general architecture is reminis-cent of Flash’s helper processes [42], and “manual calling automatic” in Adya’s work [1].

A database proxy has one dispatcher thread, and a pool of worker threads. The dis-patcher thread receives incoming RPC requests and delegates it to one of its ready workerthreads, or queuing it if all threads are busy. Based on the given RPC, the worker threadqueries the database, perhaps blocking. Upon completion, it stores its response to a mem-ory region shared among threads, and sends a small “I am finished” message to the dispatchthread over a pipe. The dispatch thread then responds to the Web server’s RPC with theappropriate result. This configuration is shown in Figure 5.1.

Database proxies employ a static number of worker threads and do not expand theirthread pool. A site maintainer should tune the database proxy’s concurrency factor to findthe minimum number of threads that allow the database to perform at maximum through-put. The intent here is simply to overlap requests to the underlying data source so that itmight overlap its disk accesses and benefit from disk arm scheduling. An unnecessarilylarge thread pool might be bad for performance, since databases like MySQL often requiresignificant per-thread state. A reasonable choice is 5 threads per database disk.

For security, database proxies ought to run on the database machines themselves. Sucha configuration allows the site administrator to “lock down” a socket-based database server,

29

// can only occur at initialization timeq = mysql−>prepare ("SELECT age,name FROM tab WHERE id=?");

id = 1; // get ID from clientuser t u; 5

stmt = q−>execute (id); // might block!stmt−>fetch (&u.age, &u.name);reply (u);

Figure 5.2: Example of database proxy code with MySQL wrapper library. In this case, theWeb developer is loading SQL results directly into an RPC XDR structure. Error conditionsignored for brevity. The user xdt t structure is defined in the protocol file in Figure 5.3.

so that only local processes can execute arbitrary database commands. All other machinesin the cluster—such as the Web server machines—only see the structured, and thus re-stricted, RPC interface exposed by the database proxy. Database proxies employ a simplemechanism for authenticating Web services. After a Web service connects to a databaseproxy, it supplies a 20-byte authentication token in a login message. The database proxythen grants the Web service permission to access a set of RPCs based on the supplied au-thentication token.

The database proxy libraries provide the illusion of a standard asynchronous RPC dis-patch routine. Internally, these proxies are multi-threaded and can block; the library hidessynchronization and scheduling details. To facilitate development of OKWS database prox-ies, we wrapped MySQL’s standard C library in an interface more suitable for use withSFS’s libraries. We model our MySQL interface after the popular Perl DBI interface [43]and likewise transparently support both parsed and prepared SQL styles. Figure 5.2 showsa simple database proxy built with this library.

5.6 Building A Web Service

A Web developer creates a new Web service as follows:

1. Extends two OKWS generic classes: one that corresponds to a long-lived service(ok service t), and one that corresponds to a short-lived, individual HTTP re-quest (ok client t). Implements the initmethod of the former and the processmethod of the latter.

2. Runs the source file through OKWS’s preprocessor, which outputs C++ code.

3. Compiles this C++ code into an executable, and installs it in OKWS’s service jail.

4. Adds the new service to /etc/okws config.

5. Restarts OKWS to launch.

The resulting Web service is a single-threaded, event-driven process.

30

struct user t {string name<30>;int age;};

5

program USER PROG {version USER PROG 1 {

user tGET USER(unsigned) = 1;

} = 1; 10

} = 10000;

Figure 5.3: user.x: the RPC protocol specification file used in client/server examples forthis section. The program has one RPC, GET USER, which takes as input an unsigneduser ID and outputs a user t structure.

The OKWS core libraries handle the mundane mechanics of a service’s life cycle andits connections to OKWS helper processes. At the initialization stage, a Web service es-tablishes persistent connections to all needed databases. The connections last the lifetimeof the service and are automatically reopened in the case of abnormal termination. Alsoat initialization, a Web service obtains static HTML templates and local configuration pa-rameters from pubd. These data stay in memory until a message from okd over the RPCcontrol channel signals that the Web service should refetch. In implementing the initmethod, the Web developer need only specify which database connections, templates andconfiguration files he requires.

The process method specifies the actions required for incoming HTTP requests. Informulating replies, a Web service typically accesses cached soft-state (such as user sessioninformation), database-resident hard state (such as inbox contents), HTML templates, andconfiguration parameters. Because a Web service is implemented as a single-threaded pro-cess, it does not require synchronization mechanisms when accessing these data sources.Its accesses to a database on behalf of all users are pipelined through a single asynchronousRPC channel. Similarly, its accesses to cached data are guaranteed to be atomic and can beachieved with simple lightweight data structures, without locking. By comparison, otherpopular Web servers require some combination of mmap’ed files, spin-locks, IPC synchro-nization, and database connection pooling to achieve similar results.

Native Web service development in OKWS is with C++, with the same SFS eventlibrary that undergirds all OKWS helper processes and core libraries. (See Section 7 for adescription of higher-level language support.) To simplify memory management, OKWSexposes SFS’s reference-counted garbage collection scheme and high-level string library tothe Web programmer. OKWS also provides a C++ preprocessor that allows for Perl-style“heredocs” and simplified template inclusion. Figure 5.4 demonstrates these facilities.

Developers using the OKWS system develop a standalone server process for each dis-crete Web service they wish to deploy. In many cases, different services share many of thesame library routines, but should be separated into as many different processes as possiblewhile still allowing for effective caching. For instance, a Web site’s mail and search fea-

31

void my_srvc_t::process (){str color = param["color"];/*oprint (resp) <<EOF;

<html><head><title>${param["title"]}</title>

</head>EOF

include (resp, "/body.html",{ COLOR => ${color}});

o*/output (resp);

}

Figure 5.4: Fragment of a Web service programmed in OKWS. The remote client sup-plies the title and color of the page via standard CGI-style parameter passing. The runtimetemplating system substitutes the user’s choice of color for the token COLOR in the tem-plate /body.html. The variable my svc t::resp represents a buffer that collects thebody of the HTTP response and then is flushed to the client via output(). With theFilterCGI flag set, OKWS filters all dangerous metacharacters from the param asso-ciative array.

tures access and cache almost entirely disjoint pools of data—perhaps mesage headers inthe former case and the results to the most common search results in the latter case. Theymight, however, use the same codebase to effect a “look-and-feel” common to the wholeWebsite. They might also use local RPC servers to share small pieces of state, such as usersession information.

5.7 Code Size

Currently, OKWS is approximately 25,000 lines of C++, FLEX and YACC code. About9200 lines of code implement the HTTP templating libraries, 6200 lines implement ourasynchronous HTTP libraries, 3000 lines round out the helper processes, and 1700 linesimplement the database proxy libraries.

32

Chapter 6

OkCupid.com: OKWS In Action

The author and two other programmers built a commercial Web site using the OKWSsystem in six months [41]. We were assisted by two designers who knew little C++ butmade effective use of the HTML templating system. The application is Internet dating,and the site features a typical suite of services, including local matching, global matching,messaging, profile maintenance, site statistics, and picture browsing. Almost a millionusers have established accounts on the site, and at peak times, thousands of users maintainactive sessions. The current implementation uses 34 Web services and 12 database proxies.

For developers accustomed to higher level languages and direct programming seman-tics, SFS’s continuation-style semantics took some time to acquaint to. We also have beenfrustrated at times by OKWS’s long compile times (OKWS and SFS together make exten-sive use of C++ templating). Finally, certain types of programming—database accesses inparticular—take more lines of code in OKWS as they would in a system like PHP, some-times by factors of two or three.

Otherwise, we have found the system to be usable, stable and well-performing. Inthe absence of database bottlenecks or latency from serving advertisements, OKWS feelsvery responsive to the end user. Even those pages that require heavy iteration—like matchcomputations—load instantaneously. In terms of stability, we’ve gone weeks at a timewithout the core OKWS processes crashing, and when they do, a core dump is usuallyenough to reconstruct our error.

OkCupid.com’s Web cluster consists of four load balanced OKWS Web server ma-chines, two read-only cache servers, and two read-write database servers, all with dualPentium 4 processors. We use multiple OKWS machines only for redundancy; one ma-chine can handle peak loads (about 200 requests per second) at about 7% CPU utilization,even as it gzips most responses. A previous incarnation of this Web site required six Mod-Perl/Apache servers [39] to accommodate less traffic. It ultimately was abandoned dueprohibitive hardware and hosting expenses [59].

6.1 Separating Static From Dynamic

OKWS relies on other machines running standard Web servers to distribute static content.This means that all pages generated by OKWS should have only absolute links to external

33

static content (such as images and style sheets), and OKWS has no reason to support keep-alive connections [21]. The servers that host static content for OKWS, however, can enableHTTP keep-alive as usual.

Serving static and dynamic content from different machines is already a common tech-nique for performance reasons; administrators choose different hardware and software con-figurations for the two types of workloads. Moreover, static content service does not re-quire access to sensitive site data and can therefore happen outside of a firewalled cluster,or perhaps at a different hosting facility altogether. Indeed, some sites push static contentout to external distribution networks such as Akamai [2]. The OkCupid deployment hostsa cluster of OKWS and database machines at a local colocation facility, where hands-onhardware access and custom configuration is possible. OkCupid serves static content fromleased, dedicated servers at a remote facility where bandwidth is significantly cheaper.

6.2 Compression

To reduce bandwidth costs and to improve perceived site responsiveness, OKWS respondsto clients queries with gzip-encoded responses whenever possible. OKWS’s current ap-proach is to encode the static HTML portions of responses with the densest possible com-pression, and to encode dynamically-generated portions at lower compression levels. Inpractice, this means that pubd compresses HTML templates as it reads them off disk, andsends both plain and compressed versions to the OKWS services. Dynamic output fromOKWS services passes through a faster gzip filter, and the OKWS libraries perform the re-quired checksums and concatenation. A sample August 2005 trace shows that 82% of clientbrowsers support gzip encoding. Using level 2 and 9 compression for dynamic and static el-ements, respectively, OkCupid’s Web servers compress outgoing documents to about 37%of their original size. Over all Web requests, OKWS’s compression strategy gives a 48%savings in bandwidth. See Appendix A for more details and future work concerning com-pression.

34

Chapter 7

Python Support in OKWS

A recent addition to OKWS is support for Web services built in Python [50], a high levelinterpreted language favored by Web serving systems like Zope [81]. There are some im-portant advantages to writing Web applications in Python as opposed to C++: faster proto-typing, no compilation, and less required programming expertise are just a few. In practice,experienced programmers might write a Web site’s most CPU-intensive and frequently-accessed services in C++, but a team of junior programmers might implement less popularservices in an interpreted language like Python. Indeed, a Web server like OKWS needs toincorporate some higher level language support to be useful to commercial Web sites.

The choice of Python as an interpreted language was largely one of taste: Ruby and Perlare similarly popular among programmers, also have a large repository of useful librariesand would have provided the same core features.

Four important goals guided the introduction of Python into OKWS:

1. Minimal changes to the underlying OKWS code base. Because OKWS is rela-tively stable, production software, we did not want to introduce regressions into thesystem and therefore hoped to leave most of the current code intact.

2. Maximal reuse of exiting OKWS code. One possibility for a Python port would beto implement services as pure Python processes, reimplementing in Python the Unixprotocols that C++ services use on startup, and REPUB. However, this approachwould require any future changes to the OKWS protocol to be made in two places:the C++ classes and also the Python clone. We hoped to avoid this situation byreusing as much of the OKWS libraries as possible.

3. Decent performance. Though a slowdown is inevitable when moving from com-piled C/C++ code to an interpreted language such as Python, decent performanceshould still be possible.

4. Preservation of all security properties. Processes running OKWS services writtenin Python should be isolated from other Python and C++ OKWS services. Moreover,Python interpreters should be unavailable to compromised Web services within thejail.

35

7.1 Approach

A hybrid C++ and Python approach satisfies all of these goals. The rough idea is one newOKWS service written in C++ that acts as generic glue code. It negotiates with okd, pubd,oklogd and okld as normal but calls into a Python interpreter at the init and processmethods. Python Web developers then implement init and process in Python, and thegeneric glue service implements the mechanics of moving data into and out of the Pythonenvironment.

This approach satisfies Requirements 1 through 3 listed above. First, OKWS need notbe modified at all to accommodate the Python glue service, since it has the same interfaceinto the OKWS system as native C++ Web services. Second, any changes made to the ini-tialization or REPUB protocol would be automatically reflected in the Python glue serviceafter recompiling and relinking, since to OKWS, the Python glue service is indistinguish-able from regular C++ services. Finally, some of the more computationally-expensive partsof the HTTP request cycle — such as the core select loop or gzipping output — are stillhandled in the C++ libraries and therefore will still perform well. Naturally, the applicationlogic, which might be doing iterative tasks like formatting HTML tables, will be in Pythonand therefore slow when compared to compiled code.

A final detail to consider is database interaction: Python services need to access OKWSdatabase proxies over asynchronous RPC, much in the same way as native C++ services.One approach to asynchronous RPC proxies is to implement the RPC transport and XDRmarshalling/unmarshalling routines in Python. This reimplementation of a feature alreadyavailable in the SFS libraries would contradict implementation goals 1 and 2. It might alsobe slow.

But the hybrid or “glue” approach is also possible here. When a service needs to makea database call, it calls a Python method that calls into the SFS RPC libraries, registering auser-specified Python callback. An RPC compiler generates code to handle marshalling aPython-accessible data representation into the standard XDR wire representation, and theSFS libraries handle the details of sending the RPC as normal. When a response comesback from the database, the SFS libraries call into autogenerated unmarshalling routines tomake the data available to Python. The SFS libraries then call a standard C++ callback,which will in turn fire the Python callback that the Web service registered earlier, callingback into the Python interpreter.

7.2 Implementation

7.2.1 Wrappers for the SFS Libraries

The first implementation detail for OKWS Python support is the “glue” between the SFSlibraries and Python, so that Python programs can access SFS’s core event loop, its asyn-chronous network event dispatch, and its RPC libraries. Figure 7.1 shows a code samplethat uses the Python interface to the SFS libraries. This code sample is a client that cor-responds to the database server seen in Figure 5.2. It makes a connection to the databaseserver with sock.connect(), gets an SFS-style asynchronous TCP transport at line 18,

36

import sfs.coreimport sfs.arpcimport socketimport xdr.user as user

5

def cb (err, res):if err != 0:

print "RPC error: " + errelse:

print "Username: " + res.name + "; age: " + res.age 10

sfs.core.exit ()

port = 30888uid = 1

15

sock = socket.socket (socket.AF INET, socket.SOCK STREAM)sock.connect (("127.0.0.1", port))x = sfs.arpc.axprt stream (sock)client = sfs.arpc.aclnt (x, user.user prog 1 ())client.call (user.GET USER, uid, cb) 20

sfs.core.amain ()

Figure 7.1: A sample Python application using SFS libraries

gets an RPC client that will appropriate marshall and unmarshall data at line 19, makesthe RPC call with client.call(), and finally, calls into the SFS select loop at line 22.When the database has replied with a response, the callback cb() is called, with the firstargument being an error code, and the second argument the data sent back in response. Asline 10 shows, the data is now accessible as standard Python classes.

To run this code, the programmer first compiles user.x (as shown in Figure 5.3) witha new Python-aware RPC compiler. The RPC compiler outputs header and source codefiles in C++, which represent data structures, methods, and functions that will eventuallybe made accessible to the Python interpreter. It also outputs marshalling and unmarshallingroutines that convert the Python-compatible structures to and from wire representation.Compiling these source files with a standard C++ compiler, and linking them into a sharedobject yields a resource that can be imported into Python, as seen in line 4 of Figure 7.1.Now, the programmer is ready to run the given Python code with the standard Pythoninterpreter.

7.2.2 OKWS Glue

With an appropriate Python interface into the SFS libraries, the rest of the task is completedwith straightforward glue code. An example of how Python in OKWS is used is shownin Figure 7.2. Like the C++ interface, the Python interface requires the programmer tosubclass two virtual base classes: service and client. Under the covers, these twoclasses are actually Python wrappers around the C++ base classes ok service t and

37

ok client t, respectively. Within these two classes, the programmer fills in code forservice. init and client.process.

In this particular example, the programmer is using the service. init methodto acquire an RPC connection to a database on line 27. The client.process methodis, as usual, where the HTTP connection is actually handled. In this example, the pro-grammer checks the user’s request for the uid variable and then makes a database querywith that value as input. When the query returns, control is transfered to the callbackexample client.cb(), and the database’s response is pasted into the given template(/get user.html) by the publishing system. The zbuf object used to collect the re-sponse and send it out to the client via self.output() is just a wrapper around theOKWS smart gziper mentioned in Section 6.2.

To start this script, as in line 30, the programmer must furnish the underlying librarieswith the first command line argument sys.argv[1], which okld uses to pass config-uration parameters to the Web service. These options are parsed and acted upon in theservice class’s init ()method, which calls into the OKWS libraries. The launch()call on line 31 starts the standard service setup negotiation with okd. Finally, as in Fig-ure 7.1, the Python script calls sfs.core.amain() to immediately enter into the SFSselect loop.

7.3 Launching Python Services and Security

Recall from Section 5.1, that the OK launcher daemon (okld) starts each OKWS serviceprocess as its own anonymous user; furthermore, okld assigns each executable ownershipand permissions so that other services cannot read the file, and no service can write to it.The same ideas should apply to Python OKWS services, but the situation is complicatedby the fact that each Python service requires two files: the Python interpreter itself, and thescript to be executed. By goal 4, we must ensure that no OKWS Python scripts can readthe text of any other script, or of any C++ service, and moreover, that no C++ service canread the code of Python scripts, and that no C++ service can execute the Python interpreter.

If we consider all of these constraints, the more convenient solutions to jailing thePython environment are insufficient. For instance, one approach is to copy the Pythoninterpreter into the jail, owned by root and belonging to group pysvc, and set to mode0550. Then, all Python scripts in the jail are owned by unique anonymous users, and eachanonymous user is a member of pysvc. Python scripts are set to mode 0400 so that onlythe owner of the script can read it and execute it with the interpreter. The problem, ofcourse, is that a compromised Python script can change the mode of its script and write toit, so that any damage to the system can persist across reboots.

OKWS adopts a safer albeit clunkier approach:

1. For each Python service, okld picks a fresh UID/GID pair (x.x) as it does for C++services.

2. okld copies the Python interpreter (python) to a new interpreter python-x, changesits owner to root, its group to x, and it access mode to 0550.

38

import okws.objsfrom okws.client import clientfrom okws.service import serviceimport sfs.coreimport sfs.arpc 5

import xdr.user as user

class example client (client):

def cb (self, err, res): 10

buf = okws.objs.zbuf ()self.pub.include (buf, "/get_user.html",

{ "name" : res.name, "age" : res.age } )self.output (buf)

15

def process (self):uid = self.cgi.lookup (key="uid")self.service.db.call (user.GET USER, int (uid), self.cb)

class example service (service): 20

def make newclient (self):return example client ()

def init (self, servicename, parameters): 25

service. init (self, servicename, parameters)self.db = self.add db (port=30888, host="127.0.0.1",

prog=user.user prog 1 ())

svc = example service (sys.argv[0], sys.argv[1]) 30

svc.launch ()sfs.core.amain ()

Figure 7.2: An example OKWS service implemented in Python. No exceptions are caughtfor brevity.

39

3. okld changes the owner of the Python script to root, its group to x, and its accessmode to 0440.

4. okld sets its user ID to x after forking, but before executing the Python interpreterwith the required script.

Thus, OKWS’s approach in the case of Python scripts mimics its approach for C++ scriptsseen in Section 5.1. An unfortunate complication is that a fresh copy of the interpreter mustbe made for each script.

7.4 Jailing Python

Coercing the Python interpreter to run inside a chroot jail is an onerous task. All Pythonlibraries and the shared libraries they link against must be copied over. The Python inter-preter itself requires many shared libraries, some of which it dlopens and hence are notreported by running ldd on the executable. To locate all shared libraries Python needs,one must examine Python’s text segment while the interpreter is running, perhaps using thelsof utility [15]. Finally, up-to-date versions of SFS/OKWS Python glue and compiledPython XDR modules must also be copied into OKWS’s jail.

The Python 2.4 distribution and its required libraries consumes about 100 MB of diskspace in the jail on FreeBSD 5.4. The Python interpreter itself, copied over once per ser-vice, consumes about 1MB of disk space. Appendix B details the mundane mechanicsof running Python in a chroot environment. This code gives a good flavor of Unix’shostility toward secure application development.

40

Chapter 8

Performance Evaluation

OKWS design limits the process pool to a small and fixed size—one per service. To eval-uate, we examined OKWS’s performance as a function of the number of active serviceprocesses. We also present and test the claim that OKWS can achieve high throughputsrelative to other Web servers because of its smaller process pool and its specialization fordynamic content.

8.1 Testing Method

Performance testing on Web servers usually involves the SPECweb99 benchmark [60],but this benchmark is not well-suited for dynamic Web servers that disable Keep-Aliveconnections and redirect to other machines for static content. We therefore devised a simplebenchmark that better models serving dynamic content in real-world deployments, whichwe call the null service benchmark. For each of the platforms tested, we implemented anull service, which takes an integer input from a client, makes a database SELECT on thebasis of that input, and returns the result in a short HTML response (see Figure 8.1). Testclients make one request per connection: they connect to the server, supply a randomlychosen query, receive the server’s response, and then disconnect.

<html><head><title>Test Result</title></head><body><?

$db = mysql_pconnect("okdb.lcs.mit.edu");mysql_select_db("testdb", $db);$id = $HTTP_GET_VARS["id"];$qry = "SELECT x,y FROM tab WHERE x=$id";$result = mysql_query("$qry", $db);$myrow = mysql_fetch_row($result);print("QRY $id $myrow[0] $myrow[1]\n");

?></body></html>

Figure 8.1: PHP version of the null service

41

0

500

1000

1500

2000

2500

1 10 100

requ

ests

per

sec

ond

number of processes (log scale)

500 clients1000 clients2000 clients

Figure 8.2: Throughputs achieved in the process pool test

8.2 Experimental Setup

All Web servers tested use a large database table filled with sequential integer keys and their20-byte SHA-1 hashes [23]. We constrained our client to query only the first 1,000,000rows of this table, so that the database could store the entire dataset in memory. Thedatabase used was MySQL version 4.0.16.

All experiments used four FreeBSD 4.8 machines. The Web server and database ma-chines were uniprocessor 2.4GHz and 2.6GHz Pentium 4s respectively, each with 1GB ofRAM. Our two client machines ran Dual 3.0GHz Pentium 4s with 2GB of RAM. All ma-chines were connected via fast Ethernet, and there was no network congestion during ourexperiments. Ping times between the clients and the Web server measured around 250 µs,and ping times between the Web server and database machine measured about 150 µs.

The test clients uses the OKWS and SFS libraries. There was no resource strain on theclient machines during our tests.

8.3 OKWS Process Pool Tests

We experimentally validated OKWS’s frugal process allocation strategy by showing thatthe alternative—running many processes per service—performs worse. We thus config-ured OKWS to run a single C++ service as a variable number of processes, and collectedthroughput measurements (in requests per second) over the different configurations. Thetest client simulated either 500, 1,000 or 2,000 concurrent remote clients in the differentruns of the experiment.

Figure 8.2 summarizes the results of this experiment as the number of processes variedbetween 1 and 450. We attribute the general decline in performance to increased context-switching, as shown in Figure 8.3. In the single-process configuration, the operating systemmust switch between the null service and okd, the demultiplexing daemon. In this configu-ration, higher client concurrency implies fewer switches, since both okd and the null service

42

0

2000

4000

6000

8000

10000

1 10 100

cont

ext s

witc

hes

per

seco

nd

number of processes (log scale)

500 clients1000 clients2000 clients

Figure 8.3: Context switching in the process pool test

have more outstanding requests to service before calling sleep. This effect quickly disap-pears as the server distributes requests over more processes. As their numbers grow, eachprocess has, on average, fewer requests to service per unit of time, and therefore calls sleepsooner within its CPU slice.

The process pool test supports our hypothesis that a Web server will consume morecomputational resources as its process pool grows. Although the experiments completedwithout putting memory pressure on the operating system, memory is more scarce in realdeployments. The null service requires about 1.5MB of core memory, but our experienceshows real OKWS service processes have memory footprints of at least 4MB, and hencewe expect memory to limit server pool size. Moreover, in real deployments there is lessmemory to waste on code text, since in-memory caches on the Web services are crucial togood site performance and should be allowed to grow as big as possible.

8.4 Web Server Comparison

The other Web servers mentioned in Section 4.3—Haboob, Flash and Apache—are pri-marily intended for serving static Web pages. Because we have designed and tuned OKWSfor an entirely dynamic workload, we hypothesize that when servicing such workloads, itperforms better than its more general-purpose peers. The experiments in this section testthis hypothesis.

Haboob is Java-based, and we compiled and ran it with FreeBSD’s native JDK, version1.3. We tested Flash v0.1a, built with FD SETSIZE set high so that Flash reported anability to service 5116 simultaneous connections. Also tested was Apache version 2.0.47compiled with multi-threading support and running PHP version 4.3.3 as a dynamic sharedobject. We configured Apache to handle up to 2000 concurrent connections. OKWS ranin its standard configuration, with a one-to-one correspondence between processes andservices. All servers enabled HTTP access logging, except for Haboob, which does notsupport it.

43

200 500 1000 1500 20000

500

1000

1500

2000

2500

3000

3500

requ

ests

per

sec

ond

concurrent clients

OKWSFlashApache+PHPHaboob

Figure 8.4: Throughputs for the single-service test

200 500 1000 1500 20000

0.5

1

1.5

2

2.5

3

med

ian

late

ncy

in s

econ

ds

concurrent clients

OKWSFlashApache+PHPHaboob

Figure 8.5: Median latencies in the single-service test

8.4.1 Single-Service Workload

In the single-service workload, clients with negligible latency request a dynamically gen-erated response from the null service. This test entails the minimal number of service pro-cesses for OKWS and Flash and therefore should allow them to exhibit maximal through-put. By contrast, Apache and Haboob’s process pools do not vary in size with the numberof available services. We examined the throughput (Figure 8.4) and responsiveness (Fig-ure 8.5) of the four systems as client concurrency increased. Figure 8.6 shows the cumula-tive distribution of client latencies when 1,600 were active concurrently. Of the four Webservers tested, Haboob spent the most CPU time in user mode and performed the slowest.A likely cause is the sluggishness of Java 1.3’s memory management.

When serving a small number of concurrent clients, the Flash system outperforms theothers; however, its performance does not scale well. We attribute this degradation to

44

0.1 1 10 1000

0.2

0.4

0.6

0.8

1

perc

entil

e

latency in seconds (log scale)

OKWSFlashApache+PHPHaboob

Figure 8.6: Client latencies for 1,600 concurrent clients in the single-service test

Flash’s CGI model: because custom-written Flash helper processes have only one threadof control, each instantiation of a helper process can handle only one external client. Thus,Flash requires a separate helper process for each external client served. At high concur-rency levels, Flash ran a large number of processes (on the order of 2000), starving itself ofrequired OS resources. Flash also puts additional strain on the database, demanding one ac-tive connection per helper—thousands in total. A database pooling system might mitigatethis negative performance impact. Flash’s results were noisy in general, and we can bestexplain the observed non-monotonicity as inconsistent operating system (and database) be-havior under heavy strain.

Apache achieves 37% of OKWS’s throughput on average. Its process pool is bigger andhence requires more frequent context switching. When servicing 1,000 concurrent clients,Apache runs around 450 processes, and context switches about 7500 times a second. Wesuspect that Apache starts queuing requests unfairly above 1,000 concurrent connections,as suggested by the plateau in Figure 8.5 and the long tail in Figure 8.6.

Apache with PHP makes frequent calls to the sigprocmask system call to serializedatabase accesses among kernel threads within a process. In addition, Apache makesfrequent (and unnecessary) file system accesses, which though serviced from the buffercache still entail system call overhead. OKWS can achieve faster performance because ofa smaller process pool and fewer system calls.

8.4.2 Many-Service Workload

In attempt to model a more realistic workload, we investigated Web servers running moreservices, serving more data, as experienced by clients over the WAN. Null services on allplatforms were modified to send out an additional 3000 bytes of text with every reply (largerresponses would have saturated the Web server’s access link in some cases). We made tenuniquely-named copies of the new null service to model ten distinct services. Finally, theclient was modified to pause an average of 75 ms between establishing a connection and

45

500 1000 1500 20000

500

1000

1500

2000

2500

3000

3500

requ

ests

per

sec

ond

concurrent clients

OKWSFlashApache+PHPHaboob

Figure 8.7: Throughputs for the many-service workload

Haboob Apache Flash OKWS

1 Service 490 895 1,590 2,40110 Services 225 760 1,232 2,089

Change −54.0% −15.1% −22.5% −13.0%

Table 8.1: Average throughputs in connections per second

sending an HTTP request.Figure 8.7 shows the achieved throughputs, and Table 8.1 compares these results to

those results observed in the single-service workload. Haboob’s performance degradesmost notably, probably because the many-service workload demands more memory allo-cations. Flash’s throughput decreases by 23%. We observed that for this workload, Flashrequires even more service processes, and at times over 2,500 were running. When weswitched from the single-service to the many-service configuration, the number of OKWSservice processes increased from 1 to 10. The results from Figure 8.2 show this change haslittle impact on throughput.

46

Chapter 9

Security Analysis

9.1 Security Benefits

The security of OKWS is evaluated in terms of its implementation of the design goals inChapter 4.

Design Goal 1: Minimal Filesystem Privileges. An OKWS service has almost no access tothe file system when execution reaches site-specific service code. If compromised, a ser-vice has write access to its coredump directory and can read from OKWS shared libraries.Otherwise, it cannot access setuid executables, the binaries of other OKWS services, orcore dumps left behind by crashed OKWS processes. It cannot overwrite HTTP logs orHTML templates. Other OKWS services such as oklogd and pubd have more privileges,enabling them to write to and read from the file system, respectively. However, as OKWSmatures, these helpers should not present security risks since they do not run site-specificcode.

Design Goals 2 and 4: Separation of Privileges. Because OKWS runs logically separateprocesses under different user IDs, compromised processes (with the exception of okld) donot have the ability to kill or ptrace other running processes. Similarly, no process save forokld can bind to privileged ports.

OKWS is careful to separate the traditionally “buggy” aspects of Web servers from themost sensitive areas of the system. In particular, those processes that do the majority ofHTTP parsing (the OKWS services) have the fewest privileges. By the same logic, okld,which runs as superuser, does no message parsing; it responds only to signals. For the otherhelper processes, we believe the RPC communication channels to be less error-prone thanstandard HTTP messaging and unlikely to allow intruders to traverse process boundaries.

Process isolation also limits the scope of those DoS attacks that exploit bugs in site-specific logic. Since the operating system sets per-process limits on resources such asfile descriptors and memory, DoS vulnerabilities should not spread across process bound-aries. We could make stronger DoS guarantees by adapting “defensive programming” tech-niques [51]. Qie et al. suggest compiling rate-control mechanisms into network services,for dynamic prevention of DoS attacks. Their system is applicable within OKWS’s archi-

47

tecture, which relegates each service to a single address space. The same cannot be said forthose systems that spread equivalent functions across multiple address spaces.

Design Goal 3: Restricted Database Access. As described, all database access in OKWSis achieved through RPC channels, using independent authentication tokens. As a result,the attacker does not gain generic SQL Client access but instead gains access in the mannerspecified by an RPC protocol declaration. This is a stronger restriction than simple databasepermission systems alone can guarantee. For instance, on PHP systems, a particular ser-vice might only have SELECT permissions to a database’s USERS table. But with control ofthe PHP server, an attacker could still issue commands like SELECT * FROM USERS. WithOKWS, if the RPC protocol restricts access to row-wise queries and the keyspace of thetable is sparse, the attacker has significantly more difficulty “mining” the database.1

OKWS’s separation of code and privileges further limits attacks. If a service is com-promised, it can attempt a new connection to any remote RPC database proxy, but it willhave little success. Because the service has no access to source code, binaries, or ptracesof other services, it knows no authentication tokens aside from its own.

Finally, OKWS database libraries provide runtime checks to ensure that SQL queriescan be prepared only when a proxy starts up, and that all parameters passed to queries areappropriately escaped. This check insulates sloppy programmers from the “SQL injection”attacks mentioned in Section 3.2. We expect future versions of OKWS to enforce the sameinvariants at compile time.

9.2 Security Shortcomings

The current implementation of OKWS supports C++ for service development. OKWS pro-grammers should use the provided “safe” strings classes when generating HTML output,and they should use only auto-generated RPC stubs for network communication. however,OKWS does not prohibit programmers from using unsafe programming techniques andcan therefore be made susceptible to buffer overruns and stack-smashing attacks. OKWSprogrammers can choose Python services if they fear vulnerabilities commonly associatedwith lower-level languages.

However, higher-level languages are no cure-all for OKWS or any other Web server.Many high-performance implementations of Python, Perl, PHP and Java use lower-levellibraries written in C or C++. Errors in these libraries can make the programs that linkagainst them vulnerable. For example, a recent bug in java.util.zip pushes someJava Virtual Machine implementations [11] to crash or exhibit undefined behavior. A recentbug in zlib shows that many higher-level languages (like Python, Perl and PHP) aresimilarly vulnerable [8].

Another shortcoming of OKWS is that an adversary who compromises an OKWS ser-vice can gain access to in-memory state belonging to other users. Developers might protectagainst this attack by encrypting cache entries with a private key stored in an HTTP cookie

1Similar security properties are possible with a standard Web server and a database that supports storedprocedures, views, and roles.

48

on the client’s machine. Encryption cannot protect against an adversary who can compro-mise and passively monitor a Web server.

Finally, independent aspects of the system might be vulnerable due to a common bugin the core libraries.

49

50

Chapter 10

Discussion and Directions for FutureResearch

OKWS can be seen as a concerted effort to apply the principle of least privilege (POLP) to aUnix Web server. OKWS attempts to uphold POLP, but experience detailed here shows thisattempt is cumbersome and labor-intensive: following POLP feels more like an abuse of theoperating system’s interface than a judicious use of its features. As we draw lessons fromour experience with OKWS, we can ask if changes to the operating system could allow pro-grammers to hold fast to privilege separation without all of the unpleasant implementationdetails. Additionally, might better OS support address the security weaknesses discussed inthe previous chapter? We consider some changes to the Unix operating system—in orderof increasing severity—to answer these questions.

10.1 Restricting the Unix System Call Interface

One of the main difficulties with OKWS’s ad-hoc privilege separation is that startingwith a privileged process and subtracting privileges is more cumbersome and error-pronethan starting with a totally unprivileged process and adding privileges. Unix-like oper-ating systems in general favor the subtractive model, while capability-based operatingsystems [7, 58] favor the additive one. But Unix file descriptors are in fact capabilities.By hobbling system calls sufficiently—either through system call interposition [26, 49] orsmall kernel modifications—we can emulate those semantics of capability-based operatingsystems that enable privilege separation.

The idea is to allow calls that use already-opened file descriptors (such as read,write, and mmap), but shut off all “sensitive” system calls, including those that createnew capabilities (such as open), assign capabilities control of named resources (such asbind), and perform file system modifications, permissions changes, or IPC without ca-pabilities (such as chown, setuid, or ptrace). In OKWS, the launcher could applysuch a policy to the worker processes, which only require access to inherited or passed filedescriptors. The launcher could run without privilege, and would no longer navigate thesystem call sequence seen in Sections 5.1 and 7.3. By disabling all unneeded privileges,the operating system could enforce privilege separation by default.

51

A benefit of Unix’s capability-like system calls is that they are virtualizable. Processesare usually indifferent to whether a file descriptor is a regular file, a pipe to another pro-cess, or a TCP socket, since the same read and write calls work in all three cases. Inpractical terms, virtualization simplifies POLP-based application design. Splitting a sys-tem into multiple processes often involves substituting user-space helper applications forkernel services; for instance, OKWS services write log entries to the logger instead of aUnix file. With virtualizable system calls, user processes can mimic the kernel’s interface;programmers need not rewrite applications when they choose to reassign the kernel’s roleto a process.

More important, virtualizable system calls enable interposition. If an untrustworthyprocess asks for a sensitive capability, a skeptical operator can babysit it by handing it apipe to an interposer instead. The interposer allows harmless queries and rejects those thatinvolve sensitive information. If the kernel API is virtualizable, then the operator need notrecompile an untrustworthy process to interpose on it.

Unfortunately, most Unix system calls resist virtualization. Some system calls do notinvolve any capability-like objects; others use hard-wired capabilities hidden in the kernel,such as “current working directory” and “file system root”. Exact user-level emulation ofthese problematic calls—which include open and accept—is messy, if not impossible.

10.2 Alternatives to Unix

From above, we claim that a step in the right direction for secure applications is to hobblethe richness of the Unix interface to expose only capability-based system calls to appli-cations. Such a scheme would give OKWS a stronger assurance of isolation between itsprocesses but might not simplify implementation due to bad support for virtualization.

An alternative to Unix is a hypothetical OS dubbed “Unestos” [33]. In Unestos, in-teractions between a process and other parts of the system take the form of messages sentto devices. Devices include processes and system services as well as hardware drivers.Messages follow the outline “perform operation O on capability C, and send any reply tocapability R.” The kernel forwards this message to the device that originally issued C.

There are a small number of operation types, as in NFS [55] and Plan 9’s 9P [45]:LOOKUP, READ, WRITE, and so forth. This design aids virtualization. All of a process’sinteractions with the system—whether with the kernel or other user applications—take thesame form, explicitly involve capabilities, and shun implicit state. Consider, for example,the Unix call open("foo"). This call in Unestos would translate to a message that aprocess P sends to the file server device FS:

P → 〈CCWD, LOOKUP, "foo", CP 〉 → FS.

The first argument is a capability CCWD that identifies P ’s current working directory. Thesecond is the command to perform, the third represents the arguments, and the fourth is thecapability to which the file system should send its response. Since Unestos makes explicitthe CWD state hidden in the Unix system call, either the file server or a user processmasquerading as the file server can answer the message.

52

10.2.1 Naming and Managing Capabilities

When an Unestos process P1 launches a child process P2, it typically grants P2 a num-ber of capabilities, ranging from directories on the file system to opened network con-nections. How can P2 then access these capabilities? Traditional capability systems suchas EROS [58] favor global, persistent naming, but persistence has proven cumbersome tokernel and application designers [57].

Instead, we advocate a per-process, Unix-style namespace. Under Unestos, P1 makescapabilities available to P2 as files in P2’s namespace. Suppose P1’s namespace containsa tree of files and directories under /secret, and P1 wishes to grant P2 access to filesunder /secret/bob. As in Plan 9 [46], P1 can mount /secret/bob as the directory/home in P2’s namespace. Unlike in Plan 9, the state implicit in the per-process namespaceis handled at user level, and the kernel only sends and receives messages to and fromcapabilities. For example, when the process P2 opens a file under /home, the user levellibraries translate the directory /home to some capability C. The kernel sees a LOOKUPmessage on C.

10.2.2 OKWS Under Unestos

We now consider what OKWS might look like on Unestos. Similar to before, the applica-tion suite consists of a okld, okd and worker processes. Under Unestos, the logger processsimply enforces append-only access to a log file, and might be useful for many applications(much like syslogd on today’s systems). pubd is no longer needed.

okld starts each process with an empty namespace (and thus no capabilities), then aug-ments their namespaces as follows:

• In the logger’s namespace, mounts a logfile on /okws/log.

• In okd’s namespace, mounts TCP port 80 on /okws/listen. For each workerprocess i, makes a socket pair and connects one end to /okws/worker/i.

• In worker process i’s namespace, mounts the other end of the above socket pair to/okws/listen. Mounts a connection to the logger on /okws/log. Mounts aread-only capability to the root HTML directory on /www.

• In all namespaces, makes required shared libraries available under /lib.

The launcher then launches all processes as before.Under Unix, the launcher had to carefully construct jails, physically copying over files

and invoking custom helper applications like the publisher and logger to limit file systemaccess. Unestos, by contrast, lets the launcher expose capabilities to child processes atarbitrary points in their namespaces. Each child receives a synthetic file system (whichperhaps only exists in memory), perfectly suited to its task.

Moreover, all capabilities available to the Unestos OKWS processes are virtualizable.Workers accept connections on /okws/listen regardless of whether they originate fromthe kernel’s TCP stack or okd. Similarly, logging might be to a raw file or through a

53

logging process that enforces append-only behavior; worker processes are oblivious to thedifference.

In this hypothetical operating system, the interface exposed to applications feels likethe familiar Unix namespace (with added flexibility for unprivileged, fine-grained jails).In reality, an application’s system interactions are entirely defined by its capabilities, andUnestos behaves like a capability system for the purposes of security analysis.

10.3 Fine-Grained POLP with MAC

Though we believe Unestos is an improvement over the status quo, it still falls short ofenabling the high-level, end-to-end security policies we seek. Applications in Unestoscan only express security policies in terms of processes, but processes often access manydifferent types of data on behalf of different users. A security policy based on processesalone can therefore conflate data flows that ought to be handled separately. For example,OKWS on Unestos achieves the policy that data from a /change-pw process cannot flowto a corrupted /show-inbox process; but the policy says nothing about whether user U ’sdata within /show-inbox can flow to user V , meaning an attacker who compromises/show-inbox might be able to read an arbitrary user’s private e-mail.

Of course, a much better policy for OKWS would be that “only user U can access userU ’s private data”. We would like to separate users from one another, much as we separateservices in the current OKWS. Though a user session involves many different processes(such as okd, databases and worker processes), a policy for separating users should beachievable with a few stanzas of privileged code, as opposed to hidden authorization checksscattered throughout the system. This section extends Unestos to a new system, Asbestos,whose kernel uses flexible mandatory access control primitives to enforce richer end-to-end security policies. We are currently designing and building Asbestos as a full operatingsystem for x86 machines [19].

The Asbestos operating system proposes a decentralized, fine-grained version of MACto solve the security problems inherent in an OKWS-like system. Similar to traditionalMAC, Asbestos assigns devices on the system to compartments, which form a partially-orderable lattice. If device A sends device B a message, and they are in the same compart-ment, they remain so after delivery. If A’s compartment is strictly higher than B’s, thenreceiving a message from A pushes B into A’s compartment. If A and B’s compartmentsare incomparable, or A’s compartment is strictly less than B’s, then message delivery fails.With compartments, Asbestos tracks all devices that have accessed a given datum, whetherdirectly or via proxy.

We propose two important modifications to traditional MAC-based operating systems.First, decentralization [40]: processes can create their own compartments on the fly, sothat a Web server can associate each remote user with her own compartment. Second,compartments apply at the fine-grained level of individual memory pages, so that a singleprocess can act on behalf of mutually distrustful users without fear of leaking data amongthem. Taken together, these two modifications allow application designers to dynamicallypartition a process’s virtual address space into compartmentalized event-processes.

Under Asbestos, OKWS behaves as follows: okd peaks into user U ’s incoming TCP

54

connection, authorizing U based on session state or login information in the HTTP head-ers. If U is logging on for the first time, okd creates a compartment for U ; if U is returning,then okd reassigns U to its previous compartment. It then forwards U ’s connection tothe appropriate event-process of the appropriate worker. When handling U ’s request, theevent-process can access virtual memory pages and devices available to U ’s compartment;for instance, it might access session state cached on the worker process or a database pro-cess trusted to store data for all users. If the event-process errantly accesses data in V ’scompartment, the read or write will fail, since U and V occupy incomparable compart-ments.

Once a worker has finished serving U , it can restore its memory and register state to asaved checkpoint, and is then safe to enter a different event-process, and speak on behalfof a different user. Finally, since okd created the user compartments, it can sanction trusteddeclassifiers to traverse them. For example, it might authorize a trusted statistics collectorto comb all pages in a worker’s virtual address space, regardless of compartment.

OKWS implemented on Asbestos provides the end-to-end security properties that Website administrators care most about: that the system cannot leak data from one user toanother. Though site-specific service code is often in flux and touched by many developers,a correct Asbestos kernel guarantees that however flawed the service code is, large-scaledata theft is very unlikely.

55

56

Chapter 11

Conclusion

This thesis described in detail the design and implementation of the OKWS system. Theimportant results of this investigation are twofold. First, OKWS represents a significantimprovement over the status quo in terms of security and performance, and its commer-cial uses shows that the system is no mere academic exercise. The key performance im-provements in OKWS stem from its specialization for dynamic content. The key securityimprovements result from a design focused on separation and isolation, as opposed to anex-post facto application of Unix security techniques to an otherwise monolithic and privi-leged application. OKWS has pushed as much as possible on the Unix interface to achieveuseful security properties without sacrificing performance.

The second important result begins where OKWS falls short: a complex implementa-tion and less-than-perfect security guarantees. If OKWS represents a concerted effort tobuild a secure Unix application from scratch but still cannot provide needed end-to-endsecurity guarantees (such as keeping Alice from Bob’s data and vice-versa), perhaps it istime to look beyond Unix to an operating system whose API is friendlier to secure ap-plication developers. Using the lessons from OKWS, this thesis proposes a rough sketchof what that operating system (dubbed “Asbestos”) might look like: a system in whichprivileges are explicitly added, not assumed and then subtracted; in which all user-kerneland user-user interactions involve capabilities, or explicit rights; in which systems calls arevirtualizable, so that user-user communication resembles user-kernel communication; inwhich IPC is auditable; in which data can be tracked as it flows through the system.

Availability

Visit www.okws.org for documentation and source code, available under a GPL license.

57

58

Appendix A

Speeding Up gzip Compression forDynamic Webservers

Most currently active Web clients such as Mozilla/Firefox and Microsoft Internet Explorersupport HTTP/1.1 gzip compression [18, 22]. A Web server can therefore transparentlysend HTML output to a client compressed via gzip. Older clients are still supported, sincenewer clients advertise their ability to accept gzip-encoded transfers in their initial request.There are obvious advantages for both client and server; they both conserve bandwidth andthey both enjoy lower latency. Moreover, since network bandwidth is expensive relativeto computational power, a cost-effective Web server should compress its output wheneverpossible.

Compression of static HTML content on the server side is easy to imagine: the serversimply maintains two copies of each document — one compressed, and one expanded—and answers client requests appropriately. For dynamic HTML content, the state-of-the artis for the Web server to naıvely compress each response on-the-fly. Compression is notcheap in terms of computation, and in many cases, compressing each response on the flysignificantly curtails server throughput. However, given a simple observation—that evendynamically generated HTML documents are often composed primarily of static HTMLcomponents— there is an opportunity for a Web server to cache work spent on compres-sion, and to stitch several cached components together for each request. We develop thisidea in more detail by first describing the generic gzip algorithm, and then by proposing acacheable implementation thereof.

A.1 The gzip Algorithm

gzip [18] is a 2-layer compression scheme. The top layer uses LZ77-compression [80],that looks for repeated substrings within a document. When the compressor finds a secondinstance of a string, it inserts a “backpointer” that points back to the original instance. Ifit finds multiple earlier instances of the same string, it selects the instance that yields thelongest possible match — in general, this heuristic improves compression ratios. A require-ment of the standard gzip decompressors built into most browsers is that these backpointerscan point no more than 32K characters into the past. This requirement helps clients decom-

59

<html><head><title>Foo Page</title><head>

<body><? for ($i = 0; $i < 10; $i++)print "bar $i ";

?></body></html>

Figure A.1: A simple fragment of PHP code.

press with only a 32K memory window. If the compressor finds no earlier instances of astring, it simply outputs the character itself. At the bottom layer, gzip feeds all backpointersand non-matches (i.e., single characters) through a Huffman encoder. It tacks on a 32-bitCRC checksum, and the compression is complete.

A naıve implementation of gzip checks every character position against a number ofcharacter positions linear in the size of the scan window. The standard gzip tool uses thismethod, in concert with 3-character Rabin fingerprinting.1 Though aggressively optimized,the standard gzip implementation2 absorbs ASCII text at about 19MB/sec and outputs com-pressed data at about 4MB/sec on a 2.4 GHz Pentium IV. A call graph profile shows thatonly 10% of computational resources are dedicated to Huffman encoding, while LZ77computation occupies a large majority of the remaining cycles.

A.2 Cacheable Compression

An improvement on the standard gzip algorithm is cacheable gzip compression. The goal ofthis algorithm is to speed up the LZ77 phase of gzip compression, by performing potentiallyexpensive precomputations on the HTML fragments that are likely to recur. The first ideais that a Web server splits each outgoing Web response into a series of texts:

R = (S1,M1, S2,M2, . . .Mn, Sn)

A text Mi is a mutable text, and might be unique to a given HTML response. A text Siis a static text, and is likely to repeat in many HTML responses. Given enough access tothe internals of one’s Web server, one can readily tell which texts are static, and which aremutable, without expensive techniques such as “shingling.” For instance, a typical PHPscript [44] might look something like Figure A.1. The “static” text in this example is allof the HTML not within the special PHP tags (given by <?. . .?>). All of the other datais mutable. Thus, with sufficient integration with PHP’s internals, a compression filter can

1The same is true for the zlib library [24] used by most popular Web servers.2gzip-1.2.4, compiled with gcc-2.95.4

60

Cacheable-LZ77 on input (R = (S1,M1, . . .Mn, Sn))1 foreach X ∈ R do2 If X is static, then append X toW .3 TrimW so that its total length is less than 32K4 o← 15 while o ≤ ||X|| do6 〈j, o∗, l〉 ← LongestMatch(X, o,W)7 p← o− o∗ + ||Sn||+ ||Mn||+ · · ·+ ||Mj+1||+ ||Sj||8 Output backpointer 〈p, l〉9 o← o+ l10 done11 done

Figure A.2: A simple, incomplete cacheable LZ77 algorithm. Several important cornercases are elided for clarity. The notation ||Mn|| gives the length of string Mn.

receive input in the form given above — a series of demarcated static and mutable texts.It is now possible to formulate a rough approximation of the proposed cacheable com-

pression algorithm. Given any text — mutable or static — called X . Assume the existenceof a routine LongestMatch, that takes as input the text X , a starting offset o into that text,and window of textsW = (S1, . . . , Sm) to compare X against. The routine returns a triple〈j, o∗, l〉, with the property that Sj[o∗, . . . , o∗ + l − 1] = X[o, . . . , o + l − 1]. That is, thesubstring of text Sj, starting at offset o∗ of l characters long, exactly matches the given textX , starting at offset o, for l characters. Moreover, this l is maximal for the given contextwindowW .3 We assume for simplicity that LongestMatch(X, o, (X)) is handled properly— that is, all offsets o∗ output are bounded below o. This is a natural constraint, since gzipdoes not accommodate forward pointers.

A simple cacheable LZ77 algorithm is shown in Figure A.2, which makes use of theabstract algorithm LongestMatch just defined. Note that only static blocks are consideredas part of the compression windowW . Assuming that static content composes a majority ofdynamic HTML pages, this omission has a minimal effect on compression ratios. Also notethat Figure A.2 elides certain corner cases for clarity — such as offsets without matchesand comparing a block X against itself.

By far, the most expensive part of the Cacheable-LZ77 algorithm are the repeatedcalls to LongestMatch on line 6. Thus, the two important performance goals are to callLongestMatch as infrequently as possible (through cache and reuse of earlier return values)and to make LongestMatch as inexpensive as possible. In pursuit of the former goal, a realimplementation of Cacheable-LZ77 should cache the results of all calls to LongestMatch

for static content blocks Si. That is, when the first argument to LongestMatch is a static

3If two equally long matches are found, the string further right is preferred, since it reduces backpointerlength.

61

block Si, all of the arguments are likely to be seen again: (1) the set of all possible staticblocks S is assumed to be small for a given Web site (on the order of 1,000) and that allblocks are known to the Web server at boot time; (2) the context window W is a subsetof S, and only a small number of these subsets are used in practice; and (3) the offset ofollows a predictable pattern based on Si andW , since the LongestMatch computation isdeterministic. Furthermore, the output of LongestMatch does not depend on the orderingof texts within the compression windowW . Thus, Web servers can reorder static blocks inHTML responses, without diminishing their ability to cache compression.

The mutable texts Mi do not enjoy the same cacheability as the static texts Si, as theyrecur infrequently, if at all. For compression of mutable texts, servers have no choice but tocompute LongestMatch as efficiently as possible. Efficient LongestMatch computation viasuffix trees is the ultimate goal for achieving fast, cacheable compression. The challengeis to do so without hoarding memory, which is scarce on busy Web servers. We leave thischallenge to future work.

A.3 Related Work

There is a rich body of literature dedicated to suffix trees, and compressing their in-memoryrepresentations. Gusfield’s book [27] provides an overview of the state-of-the-art as of1997. Since then, Kurtz [34] has discovered an efficient implementation for single textsthat consumes 10m memory for ASCII texts on average. It is not readily apparent, how-ever, how to map these techniques to trees for multiple texts. Other scheme such as suffixarrays [36] and suffix binary search trees [29] trade-off lookup speed for memory savings.Suffix arrays do not meet our requirements, as there is not an efficient way to phase outthe compression windowW while traversing the suffix array. If specialized for cacheablegzip compression, suffix binary search trees would most likely incur the same overhead ofstoring the table D.

Another approach is to compress suffix trees into directed acyclic word graphs, knownas DAWGs [6]. The idea is that if a node x has a suffix link to a node y, and the subtreesof x and y are isomorphic, then there is no need to store both trees; rather, the suffix treecan contain an edge from x to y, breaking the tree property of the graph. In the context ofmultiple texts though, this compression technique is of little use — one will see very fewsuch subtree isomorphisms in practice.

Other work has addressed the gzip compression problem directly. Larsson discovereda construction for sliding window suffix trees with direct application to LZ77 compres-sion [35]. Later, Sadakane and Imai argued that this construct is not practical in terms ofspeed, memory usage, or compression ratios. Instead, they suggest hashing and sortingtechniques based on suffix arrays [53].

Another approach to the problem is to modify the HTTP standard and client browsers,to minimize the data sent over and active HTTP session [5, 17]. While most likely moreefficient in terms of compression and server resources than the gzip method discussed here,these approaches face obvious deployment barriers; our proposed scheme, by contrast, istransparent to clients.

62

Appendix B

Jailing Python for OKWS

What follows is the actual code needed to make Python run inside of OKWS’s jail. Thoughjailing Python is possible, it does not seem natural on the Unix platform.

"""PyOKWS Jail Utilities

Copy the system Python and all associated libraries into theOKWS runtime jail.""" 5

import sysimport osimport reimport os.path 10

import string

from okws.setup.config import ConfigParser

# for reading python configuration stuff, and also for copying 15

# files and stuff over from the global directories to inside the jail.from distutils.dir util import mkpath, copy treefrom distutils.file util import copy file, write filefrom distutils.sysconfig import get python lib

20

version = 0.1

def find files by ext (root, ext):"""Find files with the given extensions, recursively, starting withthe given directory root""" 25

from os import listdirfrom os.path import join, isdir, islink, splitextls = listdir (root)files = [ ]

30

# add the leading ’.’ if not already thereif ext[0] != ".":

ext = "." + ext;

for f in ls: 35

63

fullname = join (root, f)if isdir (fullname):

files += find files by ext (fullname, ext)elif (splitext (fullname))[1] == ext:

files.append (fullname) 40

passpass

return files

#find file by ext () 45

def ldd (file):"""Given a dynamically linked executable, collect a list of all.so files that it links to."""from os import popen 50

p = popen ("ldd " + file, "r")rxx = re.compile ("(\S+) => (\S+)")sofiles = [ ]line = p.readline ()while line != "": 55

m = rxx.search (line)if m:

sofiles.append (m.group (2))pass

line = p.readline () 60

passreturn sofiles

#ldd ()65

def my join (a,b):import os.paths = os.path.sep # ’/’ for mostly anything reasonable ;)if a[−1] == s:

if b[0] != s: 70

ret = a + belse:

ret = a + b[1:]else:

if b[0] == s: 75

ret = a + belse:

ret = a + s + breturn ret

80

## call lsof to figure out which dynamic libraries python loaded# dynamically. oh bother, what a pain.## XXX - this is probably platform specific. bummer. 85

#def lsof me ():

import osp = os.popen ("lsof", "r")

64

90

# the first line tells us the name of the columns (i hope. . .)line = p.readline ()cols = line.split ()

# make an index of column names 95

col names = {}for i in range (0, len (cols)):

col names[cols[i]] = ipass

100

ret = [ ]while line != "":

s = line.split ()if s[col names["COMMAND"]] == "python" and \

s[col names["FD"]] == "txt": 105

ret.append (s[col names["NAME"]])pass

line = p.readline ()pass

return ret 110

class Jailer:"""A class that takes care of moving Python’s world into a givenchroot jail. All specification of jails and options is given with 115

the initilization function."""

def init (self, jaildir=None, okws config=None, servicebin=None,verbose=False, dry run=False):

120

self.jaildir = jaildirself.okws config = okws configself.servicebin= servicebinself.verbose = verboseself.dry run = dry run 125

self. config ()return

# init ()130

def config (self):

attrs = [ "jaildir", "servicebin"]

for a in attrs: 135

if getattr (self, a) is None:

# reads an okws config file, taking the standard# one at /etc/okws config if given parameter# ’None’ 140

p = ConfigParser (self.okws config)( , self.cfg file nvtab) = p.parse ()break

65

pass145

# fill in missing vfor a in attrs:

if getattr (self, a) is None:setattr (self, a, self.cfg file nvtab[a])pass 150

passreturn

# config ()

def get lib dirs (self): 155

return [ get python lib (standard lib=True) ]# get lib dirs ()

def run (self):"""Given an initialized OKWS/Python Jailer, run it and actually 160

do the require FS operations."""dst = self.jaildirsrcs = self.get lib dirs ()sos = [ ]done = {} # keep track of those already done 165

# copy the trees and also find all .so files in the treesfor s in self.get lib dirs ():

if done.get (s):continue 170

else:done[s] = True

copy tree (src=s, dst=my join(dst, s),verbose=self.verbose, 175

dry run=self.dry run)sos += find files by ext (s, "so")pass

# keep track of all .so files on the system the python 180

# .so files link to, and watch out for duplicatessolibs = {}

for s in sos:for l in ldd (s): 185

solibs[l]= Truepass

# also, don’t forget about the python executable itself!for l in ldd (sys.executable): 190

solibs[l] = Truepass

# also, don’t forget about those libraries python loaded# dynamically that it didn’t tell the dynamic linker about! 195

for l in lsof me ():solibs[l] = True

66

pass

for s in solibs.keys (): 200

if not os.path.isfile (s):print "Skipping non-file (did lsof misreport?): " + s

else:(d,f) = (os.path.dirname (s), os.path.basename (s))mkpath (name=my join (dst,d), verbose=self.verbose, 205

dry run=self.dry run)copy file (src=s, dst=my join (dst, s),

verbose=self.verbose, dry run=self.dry run)

# copy over the python executable, too 210

d = os.path.dirname (sys.executable)jail bin dir = my join (dst, d)mkpath (name=jail bin dir, verbose=self.verbose)copy file (src=sys.executable, dst=jail bin dir, verbose=self.verbose)

# run () 215

# Run the jailer with default argumentsj = Jailer ()j.run ()

67

68

Bibliography

[1] Atul Adya, Jon Howell, Marvin Theimer, William J. Bolosky, and John R. Douceur.Cooperative task management without manual stack management or, event-drivenprogramming is not the opposite of threaded programming. In Proceedings of the2002 USENIX, Monterey, CA, June 2002. USENIX.

[2] Akamai Technologies, Inc. http://www.akamai.com.

[3] The Apache Software Foundation. http://www.apache.org.

[4] Apache Tutorial: Introduction to Server Side Includes.http://httpd.apache.org/docs/howto/ssi.html.

[5] Gaurav Banga, Fred Douglis, and Michael Rabinovich. Optimistic deltas for WWWlatency reduction. In 1997 USENIX Annual Technical Conference, pages 289–303,1997.

[6] A. Blumer, J. Blumer, D. Haussler, A. Ehrenfeucht, M.T. Chen, and J. Seiferas J.The smallest automaton recognizing the subwords of a text, 1985.

[7] Allen C. Bomberger, William S. Frantz, Ann C. Hardy, Norman Hardy, Charles R.Landau, and Jonathan S. Shapiro. The KeyKOS nanokernel architecture. In USENIXWorkshop on Microkernels and Other Kernel Architectures. USENIX, 1992.

[8] Bugtraq ID 14162. SecurityFocus.http://www.securityfocus.com/bid/14162/info.

[9] Bugtraq ID 4606. SecurityFocus.http://www.securityfocus.com/bid/4606/info/.

[10] Bugtraq ID 5993. SecurityFocus.http://www.securityfocus.com/bid/5993/info/.

[11] Bugtraq ID 7109. SecurityFocus.http://www.securityfocus.com/bid/7109/info/.

[12] Bugtraq ID 7255. SecurityFocus.http://www.securityfocus.com/bid/7255/info/.

[13] Bugtraq ID 7768. SecurityFocus.http://www.securityfocus.com/bid/7768/info/.

69

[14] Bugtraq ID 8138. SecurityFocus.http://www.securityfocus.com/bid/8138/info/.

[15] Gary Burd. Running python CGI in OpenBSD httpd chroot jail.http://burd.info/gary/2003/11/running-python-cgi-in-openbsd-httpd.html.

[16] CERT® Coordination Center. http://www.cert.org.

[17] Mun Choon Chan and Thomas Y. C. Woo. Cache-based compaction: A newtechnique for optimizing web transfer. In INFOCOM, pages 117–125, 1999.

[18] Peter Deutsch. Gzip File Format Specification Version 4.3. Internet NetworkWorking Group RFC 1952, 1996.

[19] Petros Efstathopoulos, Maxwell Krohn, Steve VanDeBogart, Cliff Frey, DavidZiegler, Eddie Kohler, David Mazieres, Frans Kaashoek, and Robert Morris. Labelsand event processes in the asbestos operating system. In Proceedings of the 20thSymposium on Operating Systems Principles, Brighton, UK, October 2005.

[20] Open Market. Fastcgi. http://www.fastcgi.com.

[21] R. Fielding, J. Gettys, J. Mogul, H. Frystyk, L. Masinter, P. Leach, andT. Berners-Lee. Hypertext Transfer Protocol — HTTP/1.1. Internet NetworkWorking Group RFC 2616, 1999.

[22] R. Fielding, J. Gettys, J. Mogul, H. Frystyk, L. Masinter, P. Leach, andT. Berners-Lee. Hypertext Transfer Protocol — HTTP/1.1. Internet NetworkWorking Group RFC 2616, 1999.

[23] FIPS 180-1. Secure Hash Standard. U.S. Department of Commerce/N.I.S.T.,National Technical Information Service, Springfield, VA, April 1995.

[24] Jean-Loup Gailly and Greg Roelofs. zlib, 1993. http://www.gzip.org/zlib.

[25] Thomas Gchwind and Benjamin A. Schmit. CSE — a C++ servlet environment forhigh-performance web applications. In Proceedings of the FREENIX Track: 2003USENIX Technical Conference, San Antonio, TX, 2003. USENIX.

[26] Ian Goldberg, David Wagner, Randi Thomas, and Eric A. Brewer. A secureenvironment for untrusted helper applications. In Proceedings of the 6th UsenixSecurity Symposium, San Jose, CA, USA, 1996.

[27] Dan Gusfield. Algorithms on Strings, Trees, and Sequences: Computer Science andComputational Biology. Cambridge University Press, 1997.

[28] IBM corporation. IBM websphere application server. http://www.ibm.com.

[29] Robert W. Irving and Lorna Love. The suffix binary search tree and suffix AVL tree.Journal of Discrete Algorithms, 1(5-6):387–408, 2003.

70

[30] JBoss Group. http://www.jboss.org.

[31] M. Frans Kaashoek, Dawson R. Engler, Gregory R. Ganger, Hector M. Briceno,Russell Hunt, David Mazieres, Thomas Pinckney, Robert Grimm, John Jannotti, andKenneth Mackenzie. Application performance and flexibility on exokernel systems.In Proceedings of the 16th ACM Symposium on Operating Systems Principles,Saint-Malo, France, October 1997. ACM.

[32] Samuel T. King and Peter M. Chen. Backtracking intrusions. In Proceedings of the19th ACM Symposium on Operating Systems Principles, Bolton Landing, NY,October 2003. ACM.

[33] Maxwell Krohn, Petros Efstathopoulos, Cliff Frey, Frans Kaashoek, Eddie Kohler,David Mazieres, Robert Morris, Michelle Osborne, Steve VanDeBogart, and DavidZiegler. Make least privilege a right (not a privilege). In Proceedings of the 10thWorkshop on Hot Topics in Operating Systems (to appear), Santa Fe, NM, June2005.

[34] Stefan Kurtz. Reducing the space requirement of suffix trees. Software Practice andExperience, 29(13):1149–1171, 1999.

[35] N. Jesper Larsson. Extended application of suffix trees to data compression. In J. A.Storer and M. Cohn, editors, Proceedings of the Data Compression Conference,pages 190–199, Snowbird, UT, 1996. IEEE Computer Society Press.

[36] Udi Manber and Gene Myers. Suffix arrays: A new method for on-line stringsearches. SIAM Journal of Computation, 22(5):935–948, 1993.

[37] David Mazieres. A toolkit for user-level file systems. In Proceedings of the 2001USENIX. USENIX, June 2001.

[38] Microsoft Corporation. IIS. http://www.microsoft.com/windowsserver2003/iis/default.mspx.

[39] mod perl. http://perl.apache.org.

[40] Andrew C. Myers and Barbara Liskov. A decentralized model for information flowcontrol. In Proceedings of the 16th ACM Symposium on Operating SystemsPrinciples, pages 129–142, Saint-Malo, France, October 1997. ACM.

[41] OkCupid.com. http://www.okcupid.com.

[42] Vivek S. Pai, Peter Druschel, and Willy Zwaenepoel. Flash: An efficient andportable Web server. In Proceedings of the 1999 USENIX, Monterey, CA, June1999. USENIX.

[43] Perl DBI. http://dbi.perl.org.

[44] PHP: Hypertext processor. http://www.php.net.

71

[45] Rob Pike, Dave Presotto, Sean Dorward, Bob Flandrena, Ken Thompson, HowardTrickey, and Phil Winterbottom. Plan 9 from Bell Labs. Computing Systems,8(3):221–254, Summer 1995.

[46] Rob Pike, Dave Presotto, Ken Thompson, Howard Trickey, and Phil Winterbottom.The use of name spaces in Plan 9. In Proceedings of the 5th ACM SIGOPSWorkshop, Mont Saint-Michel, 1992.

[47] Kevin Poulsen. Car shoppers’ credit details exposed in bulk. SecurityFocus,September 2003. http://www.securityfocus.com/news/7067.

[48] Kevin Poulsen. Ftc investigates petco.com security hole. SecurityFocus, December2003. http://www.securityfocus.com/news/7581.

[49] Niels Provos. Improving host security with system call policies. In 12th USENIXSecurity Symposium, pages 257–271, Washington, DC, August 2003.

[50] The Python Programming Language. http://www.python.org.

[51] Xiaohu Qie, Ruoming Pang, and Larry Peterson. Defensive programming: Using anannotation toolkit to build DoS-resistant software. In 5th Symposium on OperatingSystems Design and Implementation (OSDI ’02), Boston, MA, October 2002.USENIX.

[52] Dennis M. Ritchie and Ken Thompson. The UNIX time-sharing system.Communications of the ACM, 17(7):365–375, July 1974.

[53] Kunihiko Sadakane and Hiroshi Imai. Improving the speed of LZ77 compression byhashing and suffix sorting. TIEICE: IEICE Transactions on Communications/Electronics/Information and Systems, 2000.

[54] Jerome H. Saltzer and M. D. Schroeder. The protection of information in computersystems. In Proceedings of the IEEE, volume 63, 1975.

[55] Russel Sandberg, David Goldberg, Steve Kleiman, Dan Walsh, and Bob Lyon.Design and implementation of the Sun network filesystem. In Proceedings of theSummer 1985 USENIX, pages 119–130, Portland, OR, 1985. USENIX.

[56] SecurityFocus. http://www.securityfocus.com.

[57] Jonathan S. Shapiro, Michael Scott Doerrie, Eric Northup, Swaroop Sridhar, andMark Miller. Towards a verified, general-purpose operating system kernel. InGerwin Klein, editor, Proc. NICTA Formal Methods Workshop on Operating SystemsVerification, Sydney, Australia, 2004. NICTA Technical Report 0401005T-1,National ICT Australia.

[58] Jonathan S. Shapiro, Jonathan Smith, and David J. Farber. EROS: a fast capabilitysystem. In Proc. Symposium on Operating Systems Principles, pages 170–185, 1999.

72

[59] The SparkMatch service. Previously available at http://www.thespark.com.

[60] Standard performance evaluation corporation. the specweb99 benchmark.http://www.spec99.org/osg/web99/.

[61] R. Srinivasan. RPC: Remote procedure call protocol specification version 2. RFC1831, Network Working Group, August 1995.

[62] J. Robert van Berhen, Eric A. Brewer, Nikita Borisova, Michael Chenan Matt Welsh, Josh MacDonald, Jeremy Lau, Steve Gribble, and David Culler.Ninja: A framework for network services. In Proceedings of the 2002 USENIX,Monterey, CA, June 2002. USENIX.

[63] Rob von Behren, Jeremy Condit, Feng Zhou, George C. Necula, and Eric Brewer.Capriccio: scalable threads for internet services. In Proceedings of the 19th ACMSymposium on Operating Systems Principles, Bolton Landing, NY, October 2003.ACM.

[64] Vulnerability CAN-2001-1246. SecurityFocus.http://www.securityfocus.com/bid/2954/info/.

[65] Vulnerability CAN-2002-0656. SecurityFocus.http://www.securityfocus.com/bid/5363/info/.

[66] Vulnerability CAN-2003-0042. SecurityFocus.http://www.securityfocus.com/bid/6721/info/.

[67] Vulnerability CAN-2003-0132. SecurityFocus.http://www.securityfocus.com/bid/7254/info/.

[68] Vulnerability CAN-2003-0253.http://www.securityfocus.com/bid/8137/info/.

[69] Vulnerability CAN-2003-0542. SecurityFocus.http://www.securityfocus.com/bid/8911/info/.

[70] Vulnerability CAN-2004-0077. SecurityFocus.http://www.securityfocus.com/bid/9686/info.

[71] Vulnerability CVE-2002-0061. SecurityFocus.http://www.securityfocus.com/bid/4435/info/.

[72] Vulnerability Note VU117243. CERT.http://www.kb.cert.org/vuls/id/910713.

[73] Vulnerability Note VU208131. CERT.http://www.kb.cert.org/vuls/id/208131.

[74] Vulnerability Note VU91073. CERT.http://www.kb.cert.org/vuls/id/910713.

73

[75] Vulnerability Note VU979793. CERT.http://www.kb.cert.org/vuls/id/979793.

[76] Matt Welsh, David Culler, and Eric Brewer. SEDA: An architecture forwell-conditioned, scalable internet services. In Proceedings of the 18th ACMSymposium on Operating Systems Principles, Chateau Lake Louise, Banff, Canada,October 2001. ACM.

[77] Andrew Whitaker, Marianne Shaw, and Steve D. Gribble. Scale and performance inthe denali isolation kernel. In 5th Symposium on Operating Systems Design andImplementation (OSDI ’02), Boston, MA, October 2002. USENIX.

[78] Nickolai Zeldovich, Alexander Yip, Frank Dabek, Robert Morris, David Mazieres,and Frans Kaashoek. Multiprocessor support for event-driven programs. InProceedings of the 2003 USENIX, San Antonio, TX, June 2003. USENIX.

[79] Zeus Technology Limited. Zeus Web Server. http://www.zeus.co.uk.

[80] Jacob Ziv and Abraham Lempel. A universal algorithm for sequential datacompression. IEEE Transactions on Information Theory, 23(3):337–343, 1977.

[81] The Zope Corporation. http://www.zope.org.

74