Download - NeST: Network Storage Flexible Commodity Storage Appliances

John BentCondor, WiND projects

UW-Madison

NeST: Network StorageFlexible Commodity Storage

Appliances

www.cs.wisc.edu/condorwww.cs.wisc.edu/condor/nest

New worlds, new promises

› The grid has transformed dist comp

› Multiple LANS have merged

› Confederations of auto admin domains

› Possibilities for resource and data sharing abound


New worlds, new problems

› Interoperability between domains

› Management of resources more important

› Discovery of resources more difficult


Enter NeST

› Traditional file servers have not evolved NeST is a 2nd gen file server

› Flexible storage appliance for the grid Provides local and remote access to data Easy management of storage resources

› User level sw turns machines into storage apps Deployable and portable


Research meets production

› NeST exists at an exciting intersection

› Freedom to pursue academic curiosities

› Opportunities to discover real user concerns


Very exciting intersection


Outline

› Introduction

› Interoperability

› Resource management

› Questions?


Multi-protocol support

› Interoperability between admin domains› NeST currently speaks

Grid FTP and FTP HTTP NFS (beta) Chirp

› Designed for integration of new protocols


Why not JBOS?

› Just a bunch of servers has limitations

› NeST advantages over JBOS: Single config and admin interface Optimizations across multiple protocols

• e.g. cache aware scheduling

Management and control of protocols• e.g. prefer local users to remote users


Dangerous migrations


Wasted migrations

› Wasted migrations are more expensive

› Distant resources are a mixed blessingGreat opportunitiesPossibility for great waste


Full migration support

› NeST makes storage schedulable and reservable

› NeST allows creation of dynamic users Too expensive to create all accounts everywhere Sharing is insecure and makes accounting

difficult

› NeST has lots Reservations of time and space


Three way matching

MachineMachine NeSTJob

JobAd

MachineAd

StorageAd

matc

h

Refers toNearestStorage.

Knows whereNearestStorage is.


Three way ClassAdsType = “job”TargetType = “machine”Cmd = “sim.exe”Owner = “thain”Requirements = (OpSys==“linux”) && NearestStorage.HasCMSData

Job ClassAd

Type = “machine”TargetType = “job”OpSys = “linux”Requirements = (Owner==“thain”)NearestStorage = ( Name = “turkey”) && (Type==“Storage”)

Machine ClassAd

Type = “storage”Name = “turkey.cs.wisc.edu”HasCMSData = trueCMSDataPath = /cmsdata”

Storage ClassAd


Conclusions

› User level sw turns machines into storage appliances

› Single admin interface across multiple protocols

› Participates in resource discovery› Management of space and

transfers


For more information

› http://www.cs.wisc.edu/condor/nest Version 0.9 now available (linux only, no NFS) Solaris and NFS coming soon Requests welcome

› People John Bent, Nick Leroy, Alain Roy, Joseph Stanley Miron Livny, Andrea Arpaci-Dusseau, Remzi Arpaci-Dusseau


Dispatcher

Transfer MgrConcurrencies

Storage Mgr

Control flow

Datal flow

Chirp FTP Grid ftp NFS

Common protocol layer

HTTP

Physical network layer

Physical storage layer

Design structure


Lot management overhead


NeST v JBOS, solo


NeST v JBOS, team


NeST proportional sharing

Download - NeST: Network Storage Flexible Commodity Storage Appliances

Top Related