e lving rpc fo sconsistency semantics: nfs vs afs n n fs n s tateless server n c lie n t ch e cks p...

Evolving RPC for Active StorageEvolving RPC for Active Storage

Muthian Sivathanu

Andrea C. Arpaci-DusseauRem

zi H. Arpaci-Dusseau

University of Wisconsin-M

adison

Architecture of the futuren

Everything is “active”n

Cheaper, faster processing powern

Example: “sm

art” disksn

Everything is “distributed”n

Network between processors and devicesn

Example: Network-attached storage

nNeed: Effective software paradigm

sn

Leverage power of active components

nBut rem

ains easy to use

Software systems of the future

nNeed: Tools to build “active”, “distributed” system

sn

Pragmatic: Easy for system

developers to usen

Powerful: Exploit active nature of systems

nActive system

s permit extensibility

nDownload code to device

nTailor to needs of applications/system

nSim

plicity, maintainability

•Provide prim

itives, allow clients to compose interface

nTraditional “distributed” system

s built w/ RPCn

Simple, easy-to-use com

munication paradigm

nBut not designed for “active” world

nBuild better distributed system

s w/ “active” components

Scriptable RPCn

SRPC: Paradigm for extensible distributed system

sn

Pragmatic: RPC-like developm

ent processn

Powerful: Exploit active components easily

nCase study: Active storagen

High Performance

•Efficient “com

position” of primitives

nRapid addition of new functionality

•Powerful: Advanced consistency sem

antics over NFS•

Simple: Substantial functionality in < 20 lines of code

nSim

plicity in design•

Obviate distributed locking, crash recovery

nCom

pelling paradigm for future system

s

Outline

nM

otivationn

Scriptable RPCn

Case Study: Active Storagen

Performance

nFunctionality

nSim

plicityn

Summ

ary

Scriptable RPC (SRPC)n

Evolve Remote Procedure Call (RPC)

nAugm

ent RPC interface of “server” with ascripting capability

ClientScript

Limited execution

context

Active Disk

Script Interpreter

Result

ß Prototype uses Tcl as the scripting language

SRPC : Key issuesn

Migration path

nEfficient execution of scripts

nSafety

Migration path

nM

ake transition to new paradigm less

intrusiven

Code to embed scripting into server

automatically generated

nExisting unm

odified clients co-exist withscripting clients

nDevelopm

ent process exactly the same

Development: RPC

Interfacedefinition

InterfaceIm

plementation

IDL

Compiler

Stubs

Native

Compiler

Distributed Service

Development: SRPC

Interfacedefinition

InterfaceIm

plementation

IDL

Compiler

Stubs

Native

Compiler

Distributed Service

Tcl Wrappers

Interpreter

Efficient execution of scriptsn

Hide script interpretation overheadn

Script cachingn

Exploit efficient Tcl bytecode representationn

Concurrencyn

Multiple interpreters run sim

ultaneouslyn

“Fast” standard library of primitives

nIm

plemented in C

Safetyn

Guard against m

isbehaving client scriptsn

Limited execution environm

ent: SafeTcln

Even while loops can be turned offn

Runtime type-checking

nPrevent illegal m

emory references

nAutom

atic tracking of locksn

Safe concurrent execution

Outline

nM

otivationn

Scriptable RPCn

Case Study: Active Storagen

Performance

nFunctionality

nSim

plicityn

Summ

ary

Case Study: Active storagen

Utilize CPU power at disks for client-specific processing

nPrevious approachesn

Demonstrate perform

ance benefitsn

But, require radically new architectures•

No migration path for existing services

nLim

ited class of applications•

Parallel database primitives

Evaluation environment

nPlatformn

P-III 550 MHz m

achines, 1GB m

em, 100 M

b/s netn

Linux kernel v2.2.19n

Case studies enhance ScFS using SRPC

Application

VFS

ScFS

ActiveD

iskN

FS - like protocolLinux

Kernel

Clien

t

ScFS: Performance enhancem

entsn

Combine dependent sequence of

operations into single scriptn

Reduction in network round-trips needed fora logical operation

•Benefit sensitive to network delay

•Significant savings over dialup, wide-area

•Even across overloaded “fast” networks

nReduction in total network traffic

nHelps overcom

e limitations in interface

Pathname lookup

Read (dir page “/”)

ClientD

isk

RPC

Lookup“/foo”

Pathname lookup

Find inodenum

ber

Read (dir page “/”)

Page Data

ClientD

isk

RPC

abc

21d

ef39

bar

52foo

40

Lookup“/foo”

Pathname lookup

Find inodenum

berPage D

ata

GetAttr (inode 40)

ClientD

isk

RPC

Lookup“/foo”

Read (dir page “/”)

Pathname lookup

Find inodenum

berPage D

ata

GetAttr (inode)

Attributes

ClientD

isk

RPC

Lookup“/foo”

Read (dir page “/”)

Pathname lookup

Find inodenum

berPage D

ata

GetAttr (inode)

Attributes

ClientD

iskClient

Disk

ExecScript (Read-G

etAttr)

Read page, find

inode number, get

attributes

RPC

SRPC

Lookup“/foo”

Lookup“/foo”

Read (dir page “/”)

Pathname lookup

Find inodenum

berPage D

ata

GetAttr (inode)

Attributes

ClientD

iskClient

Disk

ExecScript (Read-G

etAttr)

Read page, find

inode number, get

attributes

Attributes

RPC

SRPC

Lookup“/foo”

Lookup“/foo”

Read (dir page “/”)

Pathname lookup: Benefits

Pathname lookup: Benefits

Pathname lookup: Benefits

Pathname lookup: Benefits

Performance: Sum

mary

nExam

ples only illustrativen

Other “com

positions” possible too!n

Micro-benchm

arksn

Benefit due to reduced network roundtripsn

Macro-benchm

arksn

Postmark: 54%

less network trafficn

TPC-B: 96% less network traffic

nFacilitates working around m

inimal interfaces

Outline

nM

otivationn

Scriptable RPCn

Case Study: Active Storagen

Performance

nFunctionality

nSim

plicityn

Summ

ary

ScFS: Functionality enhancements

nIm

plement enhanced virtual protocols

over physical protocolsn

State can be added to stateless protocolsn

System provides prim

itives•

Clients compose them

into desired functionalityn

Examples

nAFS consistency sem

antics over NFSn

Sprite consistency semantics over NFS

Consistency semantics: NFS vs AFS

nNFSn

Stateless servern

Client checks periodically for updatesn

AFSn

Write-on-close sem

anticsn

Server tracks clients caching a file•

Notifies clients when modified file written

nRequires server-side state, participation

•Cannot im

plement using existing paradigm

s

Scripted AFS consistency

Client A

Client B

Scripted AFS consistency

Client A

Client B

ExecScript (AFS_OPEN

)

Open

Scripted AFS consistency

Client A

Client B

File Data

A

ExecScript (AFS_OPEN

)

Callback list

Cached file

Scripted AFS consistency

Client A

Client B

ExecScript (AFS_OPEN

)

Open

ACallback list

Cached file

Scripted AFS consistency

Client A

Client BFile D

ata

ABExecScript (AFS_O

PEN)

Callback list

Cached file

Scripted AFS consistency

Client A

Client BExecScript

(AFS_CLOSE_D

IRTY)

AB

File Close

Callback list

Cached file

Scripted AFS consistency

Client A

Client B

B

INVALID

ATE_CACHE

ExecScript

(AFS_CLOSE_D

IRTY)Callback list

Cached file

Functionality: Summ

aryn

SRPC: Powerfuln

Possible to add complex functionality

nEven those requiring augm

enting server staten

SRPC: Simple

nAFS consistency

•2 scripts, < 10 lines each

nSprite consistency

•3 scripts, < 20 lines each

nSim

ple base system, com

pact scripts to extend it

ScFS: Simplicity enhancem

ents

nAbility to group operations at servern

Simplifies im

plementation of atom

ic sets ofoperations

nO

ften, obviates need for distributed locks,distributed crash recovery

nExam

ple - concurrent directory updates

Concurrent directory updates

Client AClient B

abc

21d

ef39

Concurrent directory updates

Client AClient B

READab

c21

def

39

abc

21d

ef39

Create(“/foo”)

Concurrent directory updates

Client AClient B

READ

abc

21d

ef39

abc

21d

ef39

Create(“/foo”)Create(“/bar”)

abc

21d

ef39

Concurrent directory updates

Client AClient B

abc

21d

ef39

abc

21d

ef39

foo41

abc

21d

ef39

bar

52

Concurrent directory updates

Client AClient B

WRITE

abc

21d

ef39

foo41

abc

21d

ef39

foo41

abc

21d

ef39

bar

52

Concurrent directory updates

Client AClient B

WRITE

abc

21d

ef39

bar

52

abc

21d

ef39

foo41

abc

21d

ef39

bar

52

Concurrent directory updatesn

Non-scriptingn

Distributed locking, distributed crash recovery•

Clients acquire locks before read-modify-write

•Recover from

client failures while holding locksn

SRPCn

Script acquires in-mem

ory lock at servern

Just enforce mutual exclusion within single

address space

Summ

aryn

SRPCn

High Performance, rapid extensibility, sim

plicityn

Makes effective use of “active” architecture

nAll scripts less than 20 lines of coden

Some im

plement non-trivial functionality

nFewer lines of code =>n

Fewer bugs, more robust system

sn

Ease of building systems with active com

ponentsn

Don’t have a complex system

catering to allclient requirem

entsn

Provide primitives, enhance with com

pact scripts

Wisconsin N

etwork D

isks Group

http://ww

w.cs.w

isc.edu/wind

Questions ?