pdl packet spring update - carnegie mellon · pdf filepdl packet spring update...

NE W S L ET T E R O N T HE PA RA L LE L D AT A L A BO RAT O RY • SPR IN G 2O O5

http://www.pdl.cmu.edu/

PDL Packet Spring Update

Comparison-Based File Server Verification

Tan, Wong, Strunk & Ganger

USENIX ‘05 Annual Technical Confer-ence, April 10-15, 2005. Anaheim, CA.

Comparison-based server verificationinvolves testing a server by comparing itsresponses to those of a reference server.An intermediary, called a “server Tee,”interposes between clients and the refer-ence server, synchronizes the system-under-test (SUT) to match the referenceserver’s state, duplicates each request forthe SUT, and compares each pair ofresponses to identify any discrepancies.The result is a detailed view into any dif-ferences in how the SUT satisfies the cli-ent-server protocol specification, whichcan be invaluable in debugging servers,achieving bug compatibility, and isolatingperformance differences. This paper intro-duces, develops, and illustrates the use ofcomparison-based server verification. Asa concrete example, it describes a NFSv3Tee and reports on its use in identifyinginteresting differences in several produc-tion NFS servers and in debugging a pro-

totype NFS server. These experiencesconfirm that comparison-based server ver-ification can be a useful tool for serverimplementors.

Challenges in Building a Two-Tiered Learning Architecture for Disk Layout

Salmon, Thereska, Soules, Strunk & Ganger

Carnegie Mellon University Parallel DataLab Technical Report CMU-PDL-04-109,August, 2004.

Choosing the correct settings for largesystems can be a daunting task. The per-formance of the system is often heavilydependent upon these settings, and the“correct” settings are often closely cou-pled with the workload. System designersusually resort to using a set of heuristicapproaches that are known to work well insome cases. However, hand-combiningthese heuristics is painstaking and fragile.We propose a two-tiered architecture thatmakes this combination transparent androbust, and describe an application of thearchitecture to the problem of disk layoutoptimization. This two-tiered architectureT H E P D L

P A C K E T

C O N S O R T I U M

M E M B E R S

Recent Publications ........................ 1

PDL News ...................................... 2

Proposals & Defenses..................... 3

EMC

EqualLogic

Hewlett-Packard

Hitachi

Hitachi Global Storage

IBM

Intel

Microsoft Research

Network Appliance

Oracle

Panasas

Seagate

Sun Microsystems

EDITOR

Joan Digney

CONTACT

Greg Ganger

PDL Director

Karen Lindenfelser

PDL Business Administrator

The Parallel Data Laboratory

Carnegie Mellon University

5000 Forbes Avenue

Pittsburgh, PA 15213-3891

TEL 412-268-6716

FAX 412-268-3010

… continued on pg. 2

S E L E C T E D R E C E N T P U B L I C A T I O N S

http://www.pdl.cmu.edu/Publications/

C O N T E N T S

Unmodified

Clients

SUT

Reference

Server

RPC

Request

Response

(reference's)

Relay

NFSPlugin

SynchronizationComparisonDuplication

Server Tee

Software architecture of an NFS Tee. To minimize potential impact on clients, we sepa-rate the relaying functionality from the other three primary Tee functions (which containthe vast majority of the code). One or more NFS plug-ins can be dynamically initiatedto compare a SUT to the reference server with which clients are interacting.

2 T H E P D L P A C K E T

March 2005Matthew Wachs awarded NDSEG Fellowship

Congratulations to Matthew Wachs,who has been selected to receive a2005-2006 National Defense Scienceand Engineering Graduate (NDSEG)Fellowship. This fellowship is spon-sored by the Department of Defensethrough the Air Force Office of Scien-tific Research (AFOSR), the Office ofNaval Research (ONR), the ArmyResearch Office (ARO), and the HighPerformance Computing Moderniza-tion Program, and is administered bythe American Society for EngineeringEducation (ASEE).

March 2005 Dawn Song Receives NSF CAREER AWARD

Dawn Song, assistant professor ofElectrical and Computer Engineeringand Computer Science, has received a

N a t i o n a l S c i en ce F o u n d a t i o nCAREER Award for her research pro-posal, “Toward Exterminating LargeScale Internet Attacks.” The award“recognizes and supports the earlycareer-development activities of thoseteacher-scholars who are most likelyto become the academic leaders of the21st century.”

–with info from CMU's 8 1/2 x 11 News

January 2005Anastassia Ailamaki selected as a Sloan Research Fellow

We are extremely happy to announcethe CMU 2005 winners of a SloanResearch Fellowship: Natassa Aila-maki, Karl Crary and Anupam Gupta.A Sloan Fellowship is a prestigiousaward intended to enhance the careersof the very best young faculty mem-bers in specified fields of science.Currently a total of 116 fellowshipsare awarded annually in seven fields:

chemistry, com-pu ta t iona l an dev o l u t i o n a r ymolecular biolo-gy, computer sci-ence, economics,ma t h e ma t i c s ,n e u r o s c i e n c e ,and physics.

See more aboutt h e S l o a n f e l l o w s h i p s a thttp://www.sloan.org/programs/sci-tech_fellowships.shtml

December 2004Outstanding Researchers

Professor David O'Hallaron andResearch Engineer Volkan Akcelikhave been awarded the OutstandingResearch Award for their work on theQuake Project by the College of Engi-neering in its CIT Faculty Awards for

P D L N E W S

consists of a set of independent heuris-tics, and an adaptive method of com-bining them. However, building such asystem has proved to be more difficultthan expected. Each heuristic dependsheavily on decisions from other heu-ristics, making it difficult to break theproblem into smaller pieces. Thispaper outlines our approaches andhow they have worked, discusses thebiggest challenges in building the sys-tem, and mentions additional possiblesolutions. Whether this problem issolvable is still open to debate, but theexperiences reported provide a cau-tionary tale; system policy automationis complex and difficult.

What If You Could Ask “What-if”? (Simplifying Large System Administration)

Thereska, Narayanan & Ganger

Carnegie Mellon University ParallelData Laboratory Technical ReportCMU-PDL-05-101, February 2005.

Today, management and tuning ques-tions are approached using if/then

… continued from pg. 1

R E C E N T P U B L I C A T I O N S

rules of thumb. This reactive approachrequires expertise regarding of systembehavior, making it difficult to dealwith unforeseen uses of a system’sresources and leading to systemunpredictability and large systemmanagement overheads. We propose a“what if” approach that allows interac-tive exploration of the effects of sys-tem changes, thus converting complextuning problem into simpler searchproblems. Through two concrete man-agement problems, automating systemupgrades and deciding on servicemigrations, we identify system designchanges that enable a system toanswer “what if” questions aboutitself.

On Hierarchical Routing in Doubling Metrics

Gupta, Maggs & Zhou

Carnegie Mellon University ParallelData Lab Technical Report CMU-PDL-04-106, December 2004.

We study the problem of routing indoubling metrics, and show how to

perform hierarchical routing in suchmetrics with small stretch and com-pact routing tables (i.e., with a smallamount of routing information storedat each vertex). We say that a metric(X,d) has doubling dimension dim(X)at most a if every set of diameter D

can be covered by 2α sets of diameterD/2. (A doubling metric is one whosedoubling dimension dim(X) is a con-stant.) For a connected graph G, whoseshortest path distances dG induce thedoubling metric (X, dG), we show howto perform (1 + τ)-stretch routing on Gfor any 0 < τ # 1 with routing tables of

size at most (α/τ)O(α)log∆logδ bits

with only (α/τ)O(α)log∆ entries, where∆ is the diameter of G and δ is themaximum degree of G. Hence thenumber of routing table entries is just

τ - -O(1)log∆ for doubling metrics.These results extend and improve onthose of Talwar (2004).

http://www.pdl.cmu.edu/News/



S P R I N G 2 0 0 5 3

QPipe: A Simultaneously Pipelined Relational Query Engine

Harizopoulos, Shkapenyuk & Ailamaki

SIGMOD 2005, June 14-16, 2005,Baltimore, Maryland, USA.

Relational DBMS typically executeconcurrent queries independently byinvoking a set of operator instancesfor each query. To exploit commondata retrievals and computation inconcurrent queries, researchers haveproposed a wealth of techniques, rang-ing from buffering disk pages to con-structing materialized views andoptimizing multiple queries. The ideasproposed, however, are inherently lim-ited by the query-centric philosophyof modern engine designs. Ideally, thequery engine should proactively coor-dinate s ame-opera tor executionamong concurrent queries, therebyexploiting common accesses to mem-ory and disks as well as commonintermediate result computation.

This paper introduces on-demandsimultaneous pipelining (OSP), a nov-el query evaluation paradigm for max-

imizing data and work sharing acrossconcurrent queries at execution time.OSP enables proactive, dynamic oper-ator sharing by pipelining the opera-t or ’s o u tp u t s im u l t a n eo u s ly tomultiple parent nodes. This paper alsointroduces QPipe, a new operator-cen-tric relational engine that effortlesslysupports OSP. Each relational operatoris encapsulated in a micro-engine

serving query tasks from a queue, nat-urally exploiting all data and worksharing opportunities. Evaluation ofQPipe built on top of BerkeleyDBshows that QPipe achieves a 2x speed-up over a commercial DBMS whenrunning a workload consisting ofTPC-H queries.




P R O P O S A L S & D E F E N S E S

DISSERTATION ABSTRACT:Cluster Scheduling for Explicitly-Speculative Tasks

David Petrou

Carnegie Mellon University, Dept.ECE Ph.D. Dissertation CMU-PDL-04-112. December 2004.

A process scheduler on a shared clus-ter, grid, or supercomputer that isinformed which submitted tasks arepossibly unneeded speculative taskscan use this knowledge to better sup-port increasingly prevalent user workhabits, lowering user-visible responset ime, lowering user costs , andincreasing resource provider revenue.

Large-scale computing often consistsof many speculative tasks (tasks thatmay be canceled) to test hypotheses,search for insights, and review poten-tially finished products. For example,speculative tasks are issued by bioin-

formaticists comparing dna sequenc-es , co m pu te r g rap h ic s a r t i s t srendering scenes, and computerresearchers studying caching. Thisbehavior—exploratory searches andparameter studies, made more com-mon by the cost effectiveness of clus-t e r co m pu t ing — on ex i s t ingschedulers without speculative tasksupport results in a mismatch of goalsand suboptimal scheduling. Userswish to reduce their time waiting forneeded task output and the amountthey will be charged for unneededspeculation, making it unclear to theuser how many speculative tasks theyshould submit.

This thesis introduces ‘batchactive’scheduling (combining batch andinteractive characteristics) to exploitthe inherent speculation in commonapp l i ca t ion scena r ios . Wi th abatchactive scheduler, users submit

explicitly labeled batches of specula-tive tasks exploring ambitious linesof inquiry, and users interactivelyrequest task outputs when these out-puts are found to be needed. Afterreceiving and considering an outputfor some time, a user decides whetherto request more outputs, cancel tasks,or disclose new speculative tasks.Users are encouraged to disclosemore computation because batchac-tive scheduling intelligently prioritiz-es among speculative and non-speculative tasks, providing goodwait-time-based metrics, and becausebatchactive scheduling employs anincentive pricing mechanism whichcharges for only requested task out-puts (i.e., unneeded speculative tasksare not charged), providing bettercost-based metrics for users. Theseaspects can lead to higher billed serv-


0

.2

.4

.6

1

TPC-H Queries

Q8 1Q

2 31Q

41Q

emit

de

zila

mro

N

100%

0%

1Q

9

Lineitem

Orders

Part

Other

Time breakdown for five TPC-H que-ries. Each component shows timespent reading a TPC-H table.

0

10

20

30

40

50

60

70

80

90

0 2 4 6 8 10 12Number of clients

)ru

oh/

seir

eu

q ( tu

ph

gu

orh

T

2 4 6 8 10 12

0

30

60

90

DBMS X

QPipe

Throughput for one to twelve concur-rent clients running TPC-H queries onDBMS X and QPipe.



P R O P O S A L S & D E F E N S E S

er utilization, encouraging batchac-tive adoption by resource providersorganized as either cost- or profit-centers.

Not all tasks are equal—only taskswhose outputs users eventually desirematter—leading me to introduce the‘visible response time’ metric whichaccrues for each task in a batch ofpotentially speculative tasks whenthe user needs its output, not whenthe entire batch was submitted, andthe batchactive pricing mechanism ofcharging for only needed tasks,which encourages users to disclosurefuture work while remaining resilientto abuse. I argue that the existence ofuser think times, user away periods,and server idle time makes batchac-tive scheduling applicable to today’ssystems.

I study the behavior of speculativeand non-speculative scheduling usinga highly-parameterizable discrete-event simulator of user and taskbehavior based on important applica-tion scenarios. I contribute this simu-lator to the community for furtherscheduling research.

For example, over a broad range ofrealistic simulated user behavior andtask characteristics, I show that undera batchact ive scheduler vis ibleresponse time is improved by at leasta factor of two for 20% of the Simu-lations. A batchactive schedulerwhich favors users who historicallyhave desired a greater fraction oftasks that they speculatively dis-closed provides additional perfor-mance and is resilient to a denial-of-service. Another result is that visibleresponse time can be improved whileincreasing the throughput of taskswhose outputs were Desired. Undersome situations, user costs decrease

while server revenue increases. Arelated result is that more users canbe supported and greater server reve-nue generated while achieving thesame mean visible response time. Acomparison against an impracticalbatchactive scheduler shows that theeasily implementable two tieredbatchactive schedulers, out of allbatchactive schedulers, provide mostof the potential performance gains.Finally, I demonstrate deploymentfeasibility by describing how to inte-grate a batchactive scheduler with apopular clustering system.

DISSERTATION ABSTRACT:Efficient, Scalable Consistency for Highly Fault-tolerant Storage

Garth Goodson

Carnegie Mellon University Ph.DDissertation. CMU-PDL-04-111,August 2004.

Fault-tolerant storage systems spreaddata redundantly across a set of stor-age-nodes in an effort to preserve andprovide access to data despite fail-ures. One difficulty created by thisarchitecture is the need for a consis-tent view, across storage-nodes, ofthe most recent update. Such consis-tency is made difficult by concurrentupdates, partial updates made by cli-ents that fail, and failures of storage-nodes.

This thesis demonstrates a novelapproach to achieving scalable, high-ly fault-tolerant storage systems byleveraging a set of efficient and scal-able, strong consistency protocolsenabled by storage-node versioning.Versions maintained by storage-nodes can be used to provide consis-tency, without the need for centralserialization, and despite concurren-cy. Since versions are maintained forevery update, even if a client failspart way through an update, concur-rency exists during an update, the lat-est complete version of the data-itembeing accessed still exists in the sys-tem—it does not get destroyed bysubsequent updates. Additionally,versioning enables the use of opti-mistic protocols.

This thesis develops a set of consis-tency protocols appropriate for con-

structing blockbased storage andmetadata services. The block-basedstorage protocol is made space effi-cient through the use of erasure codesand made scalable by offloadingwork from the storage-nodes to theclients. The metadata service is madescalable by avoiding the high costsassociated with agreement algorithmsand by utilizing threshold voting quo-rums. Fault-tolerance is achieved bydeveloping each protocol in a hybridstorage-node faultmodel (a mix ofByzantine and crash storage-nodescan be tolerated), capable of tolerat-ing crash or Byzantine clients, andutilizing asynchronous communica-tion.

THESIS PROPOSAL:Automating Attribute Assignment Using Context to Assist in Personal File Retrieval

Craig Soules

School of Computer Science, Carn-egie Mellon University, Pittsburgh,PA, Feb. 25, 2005.

Context information consists of bothhow a user perceives a piece of data,as well as how the user perceives theconnection between data. A recentuser study showed that most usersuse context information to locatetheir data during a search, howevermost organizational systems andsearch tools do not take this intoaccount. In my work, I proposeschemes to automatically gather con-text information from user activityand use that to generate file attributesfor use in organization and searchtools. By increasing the number ofavailable attributes, I believe thatsuch tools can be made more effec-tive.

Raja takes a break from Retreat ‘04 proceedings.

Brandon Salmon and Mike Mesnier discuss their research with Richard

New of HGST.

S P R I N G 2 0 0 5 5

On the Effectiveness of Rate Limiting Mechanisms

Wong, Bielski, Studer & Wang

CMU Parallel Data Lab TechnicalReport CMU-PDL-05-103, March 2005.

One class of worm defense techniquesthat received attention of late is to“rate limit” outbound traffic to containfast spreading worms. Several propos-

als of rate limiting techniques haveappeared in the literature, each with adifferent take on the impetus behindrate limiting. This paper presents anempirical analysis on different ratelimiting schemes using real traffic andattack traces from a sizable network.In the analysis we isolate and investi-gate the impact of the critical parame-ters for each scheme and seek tounderstand how these parameters

might be set in realistic network set-tings. Analysis shows that using DNS-based rate limiting has substantiallylower error rates than schemes basedon other traffic statistics. The empiri-cal analysis additionally brings to lighta number of issues with respect to ratelimiting in practice. We explore theimpact of these issues in the context ofgeneral worm containment.

2004–2005. The Quake project is jointeffort by the Dept of Civil and Envi-ronmental Engineering and the Schoolof Computer Science at Carnegie Mel-lon University. Its goal is to developthe capability for predicting, by com-puter simulation, the ground motion oflarge basins during strong earth-quakes, and to use this capability tostudy the seismic response of theGreater Los Angeles Basin. Moreinformation on the Quake project mayb e f o u n d a t h t t p : / / w ww -2.cs.cmu.edu/~quake/–with info from CMU's 8 1/2 x 11 News

November 2004Pittsburgh – The Epicenter of Storage Innovation A reception highlighting Pittsburgh asthe Epicenter of Storage Innovation isbeing held on Thursday, November11, as a part of Supercomputing 2004,hosted by Pittsburgh’s Storage Innova-tors. These innovators include theData Storage Systems Center at Carn-egie Mellon University, Intel ResearchPittsburgh, Network Appliance, Pana-sas, the Parallel Data Laboratory atCarnegie Mellon University, the Pitts-burgh Digital Greenhouse, the Pitts-burgh Supercomputing Center andSeagate Research.

SC04 in Pittsburgh marks the firstyear that storage will get explicit rec-ognition in high-performance comput-ing. Pittsburgh has long been a leaderin storage innovations that havespread out from here and left theirmark on the entire industry. Dean Jim

Morris of Carnegie Mellon University,former Dean of the School of Comput-er Science and current Dean of Carn-egie Mellon University West CoastCampus will provide a short history ofstorage innovation in Pittsburgh. Thishistory stretches from the days of theInformation Technology Center (ITC)in the early 80s, and the creation of theAndrew File System (AFS), which lat-er became a product from TransarcCorporation and then from IBM. AFSin turn inspired Coda, one basis for thedistributed storage work now continu-ing at Intel Labs–Pittsburgh; led toMulti-Resident AFS (MR-AFS), stillin production use at the PittsburghSupercomputing Center (PSC); andinspired the global file system devel-oped by Spinnaker Networks (nowNetwork Appliance, the market leaderin networked storage). The Data Stor-age Systems Center (DSSC) wasfounded in 1990 to study advancedmagnetic recording, inventing some ofthe basic technology and trainingmany of the technical innovators whoare still increasing the capacity of diskdrive storage at the amazing rate of50% per year. The DSSC providedearly support for the Parallel DataLaboratory (PDL), which today leadsthe academic community in researchinto storage systems and originatedthe technology for Network AttachedSecure Disks (NASD), a core compo-nent of the products currently beingdeveloped by Panasas. The DSSC alsoled directly to the founding of SeagateResearch, which provides central

R&D for the world’s largest disk drivemaker right here in Pittsburgh. ThePittsburgh Digital Greenhouse (PDG)supports commercialization and tech-nology transfer, including storagetechnology, in the greater Pittsburghregion.

–with info from SC04 technical program notes

October 2004CMU & PDL Host Posix Extensions Workshop

Carnegie Mellon University and theParallel Data Lab hosted a PosixExtensions Workshop on November 8.The goal for the workshop was toachieve a well accepted by industryPOSIX I/O API extension, or set ofextensions, to make the POSIX I/OAPI more friendly to HPC, clustering,parallelism, and high concurrencyapplications. The meeting was held inconjunction with SuperComputing2004 and covered the initial mechan-ics for how the POSIX API is to beextended, ideas for extensions, and theformation of a plan of attack, organi-zation, and mapping of next steps.

September 2004Spiros Papadimitriou Named a Siebel Scholar

Congratulations to Spiros Papadimitri-ou, who has been selected as a SiebelScholar, providing him with one yearof financial (tuition plus stipend) sup-port. These scholarships are fundedfrom an endowment set up by the Sie-bel Corporation.

P D L N E W S






compared to a Hilbert curve approach.For scans of the primary dimension,MultiMap and naive both providealmost two orders of magnitude higherthroughput than the Hilbert curveapproach.

Ursa Minor: Versatile Cluster-based Storage

Ganger, Abd-El-Malek, Cranor, Hendricks, Klosterman, Mesnier, Prasad, Salmon, Sambasivan, Sinnamohideen, Strunk, Thereska & Wylie

Carnegie Mellon University ParallelData Lab Technical Report CMU-PDL-05-104, April 2005.

No single data encoding scheme orfault model is right for all data. A ver-satile storage system allows these to bedata-specific, so that they can bematched to access patterns, reliabilityrequirements, and cost goals. UrsaMinor is a cluster-based storage systemthat allows data-specific selection andon-line changes to encoding schemesand fault models. Thus, different datatypes can share a scalable storage infra-structure and still enjoy customizedchoices, rather than suffering from“one size fits all.” Experiments withUrsa Minor show performance penal-ties as high as 2–3X for workloadsusing poorly-matched choices. Experi-ments also show that a single clustersupporting multiple workloads is muchmore efficient when the choices arespecialized rather than forced to use amiddle-of-the-road value.

Connections: Using Context to Enhance File Search

Soules & Ganger

Carnegie Mellon University ParallelData Lab Technical Report CMU-PDL-05-105, April 2005.

The continued growth of personal filesystems demands a shift from manualfile organization to effective on-demand search tools. Today’s bestsearch tools use content analysis tech-niques to provide targeted, rankedresults for user queries. However,these tools are missing a key way thatusers remember and search for theirdata: context. Context is the set of

Multimap: Preserving Disk Locality for Multidimensional Datasets

Shao, Schlosser, Papadomanolakis, Schindler, Ailamaki, Faloutsos & Ganger

Carnegie Mellon University ParallelData Lab Technical Report CMU-PDL-05-102, March 2005.

MultiMap is a new approach to map-ping multidimensional datasets to thelinear address space of storage sys-tems. MultiMap exploits modern diskcharacteristics to provide full stream-ing bandwidth for one (pr imary)dimension and maximally efficientnon-sequential access (i.e., minimalseek and no rotational latency) for theother dimensions. This is in contrast toexisting approaches, which eitherseverely penalize non-primary dimen-sions or fail to provide full streamingbandwidth for any dimension. Experi-mental evaluation of a prototypeimplementation demonstrates Multi-Map’s superior performance for rangeand beam queries. On average, Multi-Map reduces overall I/O time by over50% when compared to traditionalnaive layouts and by over 30% when

external events that a user associateswith a file’s use: the user’s currenttask, other files being accessed, thetime of day, etc. This paper presentsConnections, a search system thatcombines content analysis with con-text information using temporal locali-ty of fi le accesses. Through thiscombination, Connections improvesboth the false-negative rate (recall)and false-positive rate (precision) overcontent analysis alone. That is, byadding context information, our sys-tem finds more of the desired files andranks them more accurately.



Applications

Tracer

File System

Results Keywords

Context-enhanced Search

Content-based

SearchResults

Relation

Graph

User

Architecture of Connections. Bothapplications and the file system remainunchanged, as the only informationrequired by Connections can be gath-ered either by a transparent tracingmodule or directly from existing filesystem interfaces.

Applications

Tracer

File System

Results Keywords

Context-enhanced Search

Content-based

SearchResults

Relation

Graph

User

Architecture of Connections. Bothapplications and the file system remainunchanged, as the only informationrequired by Connections can be gath-ered either by a transparent tracingmodule or directly from existing filesystem interfaces.

Track0

Track1

Track2

0 1 2 3 4 5 6 7

8 9 10 11 12 13 14 15

16 17 18 19 20 21 22 23

Dim0

Dim1

S0 = T = 8

S1

= 3

Dim0

Dim2

Dim

1

S1 = D = 4

0 1 2 3 4 5 6 7

8 9 10 11 12 13 14 15

32

64

S1 = D = 4

S0 = T = 8

S2

= 3

Above, a 2-D dataset is mapped todisks. The first dimension is mapped tothe track; the second dimension ismapped to the sequences of the first-αneighbors. Below, a 2-D dataset ismapped to disks. Each surface is a 2-D structure as in the 2-D map. Thethird dimension is mapped to thesequences of 4th-α neighbors.

Andy Klosterman presents The Cli-ent/Metadata Interface of Ursa Majorat last year’s Retreat.

pdl packet spring update - carnegie mellon · pdf filepdl packet spring update...

Documents