network file systems victoria krafft cs 614 10/4/05
TRANSCRIPT
Network File Systems
Victoria Krafft
CS 614
10/4/05
General Idea
People move around
Machines may want to share data
Want a system with: No new interface for applications No need to copy all the data No space consuming version control
Network File Systems
Diagram from http://www.cs.binghamton.edu/~kang/cs552/note11.ppt
A Brief History
Network File System (NFS) developed in 1984 Simple client-server model Some problems
Andrew File System (AFS) developed in 1985 Better performance More caching client-side
SFS 1999 NFS can be run on untrusted networks
Lingering Issues
Central server is a major bottleneck
All choices still require lots of bandwidth
LANs getting faster & lower latency Remote memory faster than local disk ATM faster with more nodes sending data
Cooperative Caching
Michael D. Dahlin, Randolph Y. Wang, Thomas E. Anderson, and David A. Patterson in 1994
ATM, Myrinet provide faster, low-latency network
This makes remote memory 10-20x faster than disk
Want to get data from memory of other clients rather than server disk
Cooperative Caching
Data can be found in:1. Local memory
2. Server memory
3. Other client memory
4. Server disk
How should we distribute cache data?
Design DecisionsPrivate/Global Coop. Cache?
Coordinated Cache Entries?
Static/Dynamic Partition?
Block Location? Weighted LRU
Hash
N-Chance
Direct Client Cooperation
Greedy Forwarding
Cent. Coord
Private
Any Client
DynamicStatic
Coordination
Global
No Coordination
Fixed
Direct Client Cooperation
Active clients use idle client memory as a backing store
Simple
Don’t get info from other active clients
Greedy Forwarding
Each client manages its local cache greedily
Server stores contents of client caches
Still potentially large amounts of data duplication
No major cost for performance improvements
Centrally Coordinated Caching
Client cache split into two parts – local and global
N-Chance Forwarding
Clients prefer to cache singlets, blocks stored in only one client cache.Instead of discarding a singlet, set recirculation count to n and pass on to a random other client.
Sensitivity
Variation in Response Time with Client Cache Size
Variation in Response Time with Network Latency
Simulation results
Average read response time Sever load
Simulation results
Slowdown
Slowdown
Results
N-Chance forwarding close to best possible performance
Requires clients to trust each other
Requires fast network
Serverless NFS
Thomas E. Anderson, Michael D. Dahlin, Jeanna M. Neefe, David A. Patterson, Drew S. Roselli, and Randolph Y. Wang in 1995
Eliminates central server
Takes advantage of ATM and Myrinet
Starting points
RAID: Redundancy if nodes leave or fail
LFS: Recovery when system fails
Zebra: Combines LFS and RAID for distributed systems
Multiprocessor Cache Consistency: Invalidating stale cache info
To Eliminate Central Servers
Scaleable distributed metadata, which can be reconfigured after a failure
Scalable division into groups for efficient storage
Scalable log cleaning
How it works
Each machine has one or more roles:1. Client
2. Storage Server
3. Manager
4. Cleaner
Management split among metadata managers
Disks clustered into stripe groups for scalability
Cooperative caching among clients
xFS
xFS is a prototype of the serverless network file system
Lacks a couple features: Recovery not completed Doesn’t calculate or distribute new manager or
stripe group maps No distributed cleaner
File Read
File Write
Buffered into segments in local memory
Client commits to storage
Client notifies managers of modified blocks
Managers update index nodes & imaps
Periodically, managers log changes to stable storage
Distributing File Management
First Writer – management goes to whoever created the file
*does not include all local hits
Cleaning
Segment utilization maintained by segment writer
Segment utilization stored in s-files
Cleaning controlled by stripe group leader
Optimistic Concurrency control resolves cleaning / writing conflicts
Recovery
Several steps are O(N2), but can be run in parallel
Steps For Recovery
xFS Performance
Aggregate Bandwidth Writing 10MB files
Aggregate Bandwidth Reading 10MB files
NFS max with 2 clients
AFS max with 32 clients AFS max with
12 clients
NFS max with 2 clients
xFS Performance
Average time to complete the Andrew benchmark, varying the number of simultaneous clients
System Variables
Aggregate Large-Write Bandwidth with Different Storage Server Configurations
Variation in Average Small File Creation Speed with more Managers
Possible Problems
System relies on secure network between machines, and trusted kernels on distributed nodes
Testing done on Myrinet
Low-Bandwidth NFS
Want efficient remote access over slow or wide area networks
File systems better than CVS, copying all data over
Want close-to-open consistency
LBFS
Large client cache containing user’s working set of files
Don’t send all the data – reconstitute files from previous data, and only send changes
File indexing
Non-overlapping chunks between 2K and 64K
Broken up using 48 byte Rabin fingerprints
Identified by SHA-1 hash, indexing on first 64 bits
Stored in database, recomputed before use to avoid synchronization issues
Protocol
Based on NFS, added GETHASH, MKTMPFILE, TMPWRITE, CONDWRITE, COMMITTMPSecurity infrastructure from SFSWhole file cachingRetrieve from server on read unless valid copy in cacheWrite back to server when file closed
File Reads
File Writes
Implementation
LBFS server accesses file system as an NFS clientServer creates trash directory for temporary filesServer inefficient when files overwritten or truncated, which could be fixed by lower-level access.Client uses xfs driver
Evaluation
Bandwidth consumptionMuch higher bandwidth
for first build
Application Performance
Bandwidth and Round Trip Time
Conclusions
New technologies open up new possibilities for network file systems
Cost of increased traffic over Ethernet may cause problems for xFS, cooperative caching.