headline in arial bold 30pt making the linux nfs server suck faster greg banks file serving...
TRANSCRIPT
![Page 1: Headline in Arial Bold 30pt Making the Linux NFS Server Suck Faster Greg Banks File Serving Technologies, Silicon Graphics, Inc Making The Linux NFS Server](https://reader035.vdocument.in/reader035/viewer/2022062407/56649e175503460f94b02a8c/html5/thumbnails/1.jpg)
Headline in Arial Bold 30pt
Making the Linux NFS Server Suck Faster
Greg Banks <[email protected]>
File Serving Technologies,
Silicon Graphics, Inc
Making The Linux NFS Server Suck Faster
![Page 2: Headline in Arial Bold 30pt Making the Linux NFS Server Suck Faster Greg Banks File Serving Technologies, Silicon Graphics, Inc Making The Linux NFS Server](https://reader035.vdocument.in/reader035/viewer/2022062407/56649e175503460f94b02a8c/html5/thumbnails/2.jpg)
Slide 2Linux.conf.au Sydney Jan 2007
Overview
• Introduction
• Principles of Operation
• Performance Factors
• Performance Results
• Future Work
• Questions?
![Page 3: Headline in Arial Bold 30pt Making the Linux NFS Server Suck Faster Greg Banks File Serving Technologies, Silicon Graphics, Inc Making The Linux NFS Server](https://reader035.vdocument.in/reader035/viewer/2022062407/56649e175503460f94b02a8c/html5/thumbnails/3.jpg)
Slide 3Linux.conf.au Sydney Jan 2007
SGI
• SGI doesn't just make honking great compute servers
• also about storage hardware
• and storage software
• NAS Server Software
![Page 4: Headline in Arial Bold 30pt Making the Linux NFS Server Suck Faster Greg Banks File Serving Technologies, Silicon Graphics, Inc Making The Linux NFS Server](https://reader035.vdocument.in/reader035/viewer/2022062407/56649e175503460f94b02a8c/html5/thumbnails/4.jpg)
NAS
• NAS = Network Attached Storage
![Page 5: Headline in Arial Bold 30pt Making the Linux NFS Server Suck Faster Greg Banks File Serving Technologies, Silicon Graphics, Inc Making The Linux NFS Server](https://reader035.vdocument.in/reader035/viewer/2022062407/56649e175503460f94b02a8c/html5/thumbnails/5.jpg)
NAS
• your data on a RAID array
• attached to a special-purpose machine with network interfaces
![Page 6: Headline in Arial Bold 30pt Making the Linux NFS Server Suck Faster Greg Banks File Serving Technologies, Silicon Graphics, Inc Making The Linux NFS Server](https://reader035.vdocument.in/reader035/viewer/2022062407/56649e175503460f94b02a8c/html5/thumbnails/6.jpg)
NAS
• Access data over the network via file sharing protocols
• CIFS
• iSCSI
• FTP
• NFS
![Page 7: Headline in Arial Bold 30pt Making the Linux NFS Server Suck Faster Greg Banks File Serving Technologies, Silicon Graphics, Inc Making The Linux NFS Server](https://reader035.vdocument.in/reader035/viewer/2022062407/56649e175503460f94b02a8c/html5/thumbnails/7.jpg)
NAS
• NFS: a common solution for compute cluster storage
• freely available
• known administration
• no inherent node limit
• simpler than cluster filesystems (CXFS, Lustre)
![Page 8: Headline in Arial Bold 30pt Making the Linux NFS Server Suck Faster Greg Banks File Serving Technologies, Silicon Graphics, Inc Making The Linux NFS Server](https://reader035.vdocument.in/reader035/viewer/2022062407/56649e175503460f94b02a8c/html5/thumbnails/8.jpg)
Anatomy of a Compute Cluster
• today's HPC architecture of choice
• hordes (2000+) of Linux 2.4 or 2.6 clients
![Page 9: Headline in Arial Bold 30pt Making the Linux NFS Server Suck Faster Greg Banks File Serving Technologies, Silicon Graphics, Inc Making The Linux NFS Server](https://reader035.vdocument.in/reader035/viewer/2022062407/56649e175503460f94b02a8c/html5/thumbnails/9.jpg)
Anatomy of a Compute Cluster
• node: low bandwidth or IOPS
• 1 Gigabit Ethernet NIC
• server: large aggregate bandwidth or IOPS
• multiple Gigabit Ethernet NICs...2 to 8 or more
![Page 10: Headline in Arial Bold 30pt Making the Linux NFS Server Suck Faster Greg Banks File Serving Technologies, Silicon Graphics, Inc Making The Linux NFS Server](https://reader035.vdocument.in/reader035/viewer/2022062407/56649e175503460f94b02a8c/html5/thumbnails/10.jpg)
Anatomy of a Compute Cluster
• global namespace desirable
• sometimes, a single filesystem
• Ethernet bonding or RR-DNS
![Page 11: Headline in Arial Bold 30pt Making the Linux NFS Server Suck Faster Greg Banks File Serving Technologies, Silicon Graphics, Inc Making The Linux NFS Server](https://reader035.vdocument.in/reader035/viewer/2022062407/56649e175503460f94b02a8c/html5/thumbnails/11.jpg)
SGI's NAS Server
• SGI's approach: a single honking great server
• global namespace happens trivially
• large RAM fit
• shared data & metadata cache
• performance by scaling UP not OUT
![Page 12: Headline in Arial Bold 30pt Making the Linux NFS Server Suck Faster Greg Banks File Serving Technologies, Silicon Graphics, Inc Making The Linux NFS Server](https://reader035.vdocument.in/reader035/viewer/2022062407/56649e175503460f94b02a8c/html5/thumbnails/12.jpg)
SGI's NAS Server
• IA64 Linux NUMA machines (Altix)
• previous generation: MIPS Irix (Origin)
• small by SGI's standards (2 to 8 CPUs)
![Page 13: Headline in Arial Bold 30pt Making the Linux NFS Server Suck Faster Greg Banks File Serving Technologies, Silicon Graphics, Inc Making The Linux NFS Server](https://reader035.vdocument.in/reader035/viewer/2022062407/56649e175503460f94b02a8c/html5/thumbnails/13.jpg)
Building Block
• Altix A350 “brick”
• 2 Itanium CPUs
• 12 DIMM slots (4 – 24 GiB)
• lots of memory bandwidth
![Page 14: Headline in Arial Bold 30pt Making the Linux NFS Server Suck Faster Greg Banks File Serving Technologies, Silicon Graphics, Inc Making The Linux NFS Server](https://reader035.vdocument.in/reader035/viewer/2022062407/56649e175503460f94b02a8c/html5/thumbnails/14.jpg)
Building Block
• 4 x 64bit 133 MHz PCI-X slots
• 2 Gigabit Ethernets
• RAID attached
with FibreChannel
![Page 15: Headline in Arial Bold 30pt Making the Linux NFS Server Suck Faster Greg Banks File Serving Technologies, Silicon Graphics, Inc Making The Linux NFS Server](https://reader035.vdocument.in/reader035/viewer/2022062407/56649e175503460f94b02a8c/html5/thumbnails/15.jpg)
Building A Bigger Server
• Connect multiple bricks
with NUMALinkTM
up to 16 CPUs
![Page 16: Headline in Arial Bold 30pt Making the Linux NFS Server Suck Faster Greg Banks File Serving Technologies, Silicon Graphics, Inc Making The Linux NFS Server](https://reader035.vdocument.in/reader035/viewer/2022062407/56649e175503460f94b02a8c/html5/thumbnails/16.jpg)
NFS Sucks!
• Yeah, we all knew that
![Page 17: Headline in Arial Bold 30pt Making the Linux NFS Server Suck Faster Greg Banks File Serving Technologies, Silicon Graphics, Inc Making The Linux NFS Server](https://reader035.vdocument.in/reader035/viewer/2022062407/56649e175503460f94b02a8c/html5/thumbnails/17.jpg)
NFS Sucks!
• But really, on Altix it sucked sloooowly
• 2 x 1.4 GHz McKinley slower than
2 x 800 MHz MIPS
• 6 x Itanium -> 8 x Itanium
33% more power, 12% more NFS throughput
• With fixed # clients, more CPUs was slower!
• Simply did not scale; CPU limited
![Page 18: Headline in Arial Bold 30pt Making the Linux NFS Server Suck Faster Greg Banks File Serving Technologies, Silicon Graphics, Inc Making The Linux NFS Server](https://reader035.vdocument.in/reader035/viewer/2022062407/56649e175503460f94b02a8c/html5/thumbnails/18.jpg)
NFS Sucks!
• My mission...
make the Linux NFS server suck faster on NUMA
![Page 19: Headline in Arial Bold 30pt Making the Linux NFS Server Suck Faster Greg Banks File Serving Technologies, Silicon Graphics, Inc Making The Linux NFS Server](https://reader035.vdocument.in/reader035/viewer/2022062407/56649e175503460f94b02a8c/html5/thumbnails/19.jpg)
Bandwidth Test
• Throughput for streaming read, TCP, rsize=32K
4 6 80
200
400
600
800
1000
# CPUs, NICs, clients
Th
rou
gh
pu
t, M
iB/s Better
Theoretical Maximum
Before
![Page 20: Headline in Arial Bold 30pt Making the Linux NFS Server Suck Faster Greg Banks File Serving Technologies, Silicon Graphics, Inc Making The Linux NFS Server](https://reader035.vdocument.in/reader035/viewer/2022062407/56649e175503460f94b02a8c/html5/thumbnails/20.jpg)
Call Rate Test
• IOPS for in-memory rsync from simulated Linux 2.4 clients
0 5 10 15 20 25 30 35 40 45 500
10000
20000
30000
40000
# virtual clients
rsyn
c IO
PS Scheduler overload!
Clients cannot mount
![Page 21: Headline in Arial Bold 30pt Making the Linux NFS Server Suck Faster Greg Banks File Serving Technologies, Silicon Graphics, Inc Making The Linux NFS Server](https://reader035.vdocument.in/reader035/viewer/2022062407/56649e175503460f94b02a8c/html5/thumbnails/21.jpg)
Overview
• Introduction
• Principles of Operation
• Performance Factors
• Performance Results
• Future Work
• Questions?
![Page 22: Headline in Arial Bold 30pt Making the Linux NFS Server Suck Faster Greg Banks File Serving Technologies, Silicon Graphics, Inc Making The Linux NFS Server](https://reader035.vdocument.in/reader035/viewer/2022062407/56649e175503460f94b02a8c/html5/thumbnails/22.jpg)
Principles of Operation
• portmap
maps RPC program # -> TCP port
![Page 23: Headline in Arial Bold 30pt Making the Linux NFS Server Suck Faster Greg Banks File Serving Technologies, Silicon Graphics, Inc Making The Linux NFS Server](https://reader035.vdocument.in/reader035/viewer/2022062407/56649e175503460f94b02a8c/html5/thumbnails/23.jpg)
Principles of Operation
• rpc.mountd
• handles MOUNT call
• interprets /etc/exports
![Page 24: Headline in Arial Bold 30pt Making the Linux NFS Server Suck Faster Greg Banks File Serving Technologies, Silicon Graphics, Inc Making The Linux NFS Server](https://reader035.vdocument.in/reader035/viewer/2022062407/56649e175503460f94b02a8c/html5/thumbnails/24.jpg)
Principles of Operation
• kernel nfsd threads
• global pool
• little per-client state (< v4)
• threads handle calls
not clients
• “upcall”s to rpc.mountd
![Page 25: Headline in Arial Bold 30pt Making the Linux NFS Server Suck Faster Greg Banks File Serving Technologies, Silicon Graphics, Inc Making The Linux NFS Server](https://reader035.vdocument.in/reader035/viewer/2022062407/56649e175503460f94b02a8c/html5/thumbnails/25.jpg)
Kernel Data Structures
• struct svc_socket
• per UDP or TCP socket
![Page 26: Headline in Arial Bold 30pt Making the Linux NFS Server Suck Faster Greg Banks File Serving Technologies, Silicon Graphics, Inc Making The Linux NFS Server](https://reader035.vdocument.in/reader035/viewer/2022062407/56649e175503460f94b02a8c/html5/thumbnails/26.jpg)
Kernel Data Structures
• struct svc_serv
• effectively global
• pending socket list
• available threads list
• permanent sockets list (UDP, TCP rendezvous)
• temporary sockets (TCP connection)
![Page 27: Headline in Arial Bold 30pt Making the Linux NFS Server Suck Faster Greg Banks File Serving Technologies, Silicon Graphics, Inc Making The Linux NFS Server](https://reader035.vdocument.in/reader035/viewer/2022062407/56649e175503460f94b02a8c/html5/thumbnails/27.jpg)
Kernel Data Structures
• struct ip_map
• represents a client IP address
• sparse hashtable, populated on demand
![Page 28: Headline in Arial Bold 30pt Making the Linux NFS Server Suck Faster Greg Banks File Serving Technologies, Silicon Graphics, Inc Making The Linux NFS Server](https://reader035.vdocument.in/reader035/viewer/2022062407/56649e175503460f94b02a8c/html5/thumbnails/28.jpg)
Lifetime of an RPC service thread
• If no socket has pending data, block– normal idle condition
![Page 29: Headline in Arial Bold 30pt Making the Linux NFS Server Suck Faster Greg Banks File Serving Technologies, Silicon Graphics, Inc Making The Linux NFS Server](https://reader035.vdocument.in/reader035/viewer/2022062407/56649e175503460f94b02a8c/html5/thumbnails/29.jpg)
Lifetime of an RPC service thread
• If no socket has pending data, block– normal idle condition
• Take a pending socket from the (global) list
![Page 30: Headline in Arial Bold 30pt Making the Linux NFS Server Suck Faster Greg Banks File Serving Technologies, Silicon Graphics, Inc Making The Linux NFS Server](https://reader035.vdocument.in/reader035/viewer/2022062407/56649e175503460f94b02a8c/html5/thumbnails/30.jpg)
Lifetime of an RPC service thread
• If no socket has pending data, block– normal idle condition
• Take a pending socket from the (global) list
• Read an RPC call from the socket
![Page 31: Headline in Arial Bold 30pt Making the Linux NFS Server Suck Faster Greg Banks File Serving Technologies, Silicon Graphics, Inc Making The Linux NFS Server](https://reader035.vdocument.in/reader035/viewer/2022062407/56649e175503460f94b02a8c/html5/thumbnails/31.jpg)
Lifetime of an RPC service thread
• If no socket has pending data, block– normal idle condition
• Take a pending socket from the (global) list
• Read an RPC call from the socket
• Decode the call (protocol specific)
![Page 32: Headline in Arial Bold 30pt Making the Linux NFS Server Suck Faster Greg Banks File Serving Technologies, Silicon Graphics, Inc Making The Linux NFS Server](https://reader035.vdocument.in/reader035/viewer/2022062407/56649e175503460f94b02a8c/html5/thumbnails/32.jpg)
Lifetime of an RPC service thread
• If no socket has pending data, block– normal idle condition
• Take a pending socket from the (global) list
• Read an RPC call from the socket
• Decode the call (protocol specific)
• Dispatch the call (protocol specific)– actual I/O to fs happens here
![Page 33: Headline in Arial Bold 30pt Making the Linux NFS Server Suck Faster Greg Banks File Serving Technologies, Silicon Graphics, Inc Making The Linux NFS Server](https://reader035.vdocument.in/reader035/viewer/2022062407/56649e175503460f94b02a8c/html5/thumbnails/33.jpg)
Lifetime of an RPC service thread
• If no socket has pending data, block– normal idle condition
• Take a pending socket from the (global) list
• Read an RPC call from the socket
• Decode the call (protocol specific)
• Dispatch the call (protocol specific)– actual I/O to fs happens here
• Encode the reply (protocol specific)
![Page 34: Headline in Arial Bold 30pt Making the Linux NFS Server Suck Faster Greg Banks File Serving Technologies, Silicon Graphics, Inc Making The Linux NFS Server](https://reader035.vdocument.in/reader035/viewer/2022062407/56649e175503460f94b02a8c/html5/thumbnails/34.jpg)
Lifetime of an RPC service thread
• If no socket has pending data, block– normal idle condition
• Take a pending socket from the (global) list
• Read an RPC call from the socket
• Decode the call (protocol specific)
• Dispatch the call (protocol specific)– actual I/O to fs happens here
• Encode the reply (protocol specific)
• Send the reply on the socket
![Page 35: Headline in Arial Bold 30pt Making the Linux NFS Server Suck Faster Greg Banks File Serving Technologies, Silicon Graphics, Inc Making The Linux NFS Server](https://reader035.vdocument.in/reader035/viewer/2022062407/56649e175503460f94b02a8c/html5/thumbnails/35.jpg)
Overview
• Introduction
• Principles of Operation
• Performance Factors
• Performance Results
• Future Work
• Questions?
![Page 36: Headline in Arial Bold 30pt Making the Linux NFS Server Suck Faster Greg Banks File Serving Technologies, Silicon Graphics, Inc Making The Linux NFS Server](https://reader035.vdocument.in/reader035/viewer/2022062407/56649e175503460f94b02a8c/html5/thumbnails/36.jpg)
Performance Goals: What is Scaling?
• Scale workload linearly– from smallest model: 2 CPUs, 2 GigE NICs
– to largest model: 8 CPUs, 8 GigE NICs
![Page 37: Headline in Arial Bold 30pt Making the Linux NFS Server Suck Faster Greg Banks File Serving Technologies, Silicon Graphics, Inc Making The Linux NFS Server](https://reader035.vdocument.in/reader035/viewer/2022062407/56649e175503460f94b02a8c/html5/thumbnails/37.jpg)
Performance Goals: What is Scaling?
• Scale workload linearly– from smallest model: 2 CPUs, 2 GigE NICs
– to largest model: 8 CPUs, 8 GigE NICs
• Many clients: Handle 2000 distinct IP addresses
![Page 38: Headline in Arial Bold 30pt Making the Linux NFS Server Suck Faster Greg Banks File Serving Technologies, Silicon Graphics, Inc Making The Linux NFS Server](https://reader035.vdocument.in/reader035/viewer/2022062407/56649e175503460f94b02a8c/html5/thumbnails/38.jpg)
Performance Goals: What is Scaling?
• Scale workload linearly– from smallest model: 2 CPUs, 2 GigE NICs
– to largest model: 8 CPUs, 8 GigE NICs
• Many clients: Handle 2000 distinct IP addresses
• Bandwidth: fill those pipes!
![Page 39: Headline in Arial Bold 30pt Making the Linux NFS Server Suck Faster Greg Banks File Serving Technologies, Silicon Graphics, Inc Making The Linux NFS Server](https://reader035.vdocument.in/reader035/viewer/2022062407/56649e175503460f94b02a8c/html5/thumbnails/39.jpg)
Performance Goals: What is Scaling?
• Scale workload linearly– from smallest model: 2 CPUs, 2 GigE NICs
– to largest model: 8 CPUs, 8 GigE NICs
• Many clients: Handle 2000 distinct IP addresses
• Bandwidth: fill those pipes!
• Call rate: metadata-intensive workloads
![Page 40: Headline in Arial Bold 30pt Making the Linux NFS Server Suck Faster Greg Banks File Serving Technologies, Silicon Graphics, Inc Making The Linux NFS Server](https://reader035.vdocument.in/reader035/viewer/2022062407/56649e175503460f94b02a8c/html5/thumbnails/40.jpg)
Lock Contention & Hotspots
• spinlocks contended by multiple CPUs
• oprofile shows time spent in ia64_spinlock_contention.
![Page 41: Headline in Arial Bold 30pt Making the Linux NFS Server Suck Faster Greg Banks File Serving Technologies, Silicon Graphics, Inc Making The Linux NFS Server](https://reader035.vdocument.in/reader035/viewer/2022062407/56649e175503460f94b02a8c/html5/thumbnails/41.jpg)
Lock Contention & Hotspots
• on NUMA, don't even need to contend
• cache coherency latency for unowned cachelines
• off-node latency much worse than local
• “cacheline ping-pong”
![Page 42: Headline in Arial Bold 30pt Making the Linux NFS Server Suck Faster Greg Banks File Serving Technologies, Silicon Graphics, Inc Making The Linux NFS Server](https://reader035.vdocument.in/reader035/viewer/2022062407/56649e175503460f94b02a8c/html5/thumbnails/42.jpg)
Lock Contention & Hotspots
• affects data structures as well as locks
• kernel profile shows time spent in un-obvious places in functions
• lots of cross-node traffic in hardware stats
![Page 43: Headline in Arial Bold 30pt Making the Linux NFS Server Suck Faster Greg Banks File Serving Technologies, Silicon Graphics, Inc Making The Linux NFS Server](https://reader035.vdocument.in/reader035/viewer/2022062407/56649e175503460f94b02a8c/html5/thumbnails/43.jpg)
Some Hotspots
• sv_lock spinlock in struct svc_serv– guards global list of pending sockets, list of pending threads
• split off the hot parts into multiple svc_pools– one svc_pool per NUMA node
– sockets are attached to a pool for the lifetime of a call
– moved temp socket aging from main loop to a timer
![Page 44: Headline in Arial Bold 30pt Making the Linux NFS Server Suck Faster Greg Banks File Serving Technologies, Silicon Graphics, Inc Making The Linux NFS Server](https://reader035.vdocument.in/reader035/viewer/2022062407/56649e175503460f94b02a8c/html5/thumbnails/44.jpg)
Some Hotspots
• struct nfsdstats– global structure
• eliminated some of the less useful stats– fewer writes to this structure
![Page 45: Headline in Arial Bold 30pt Making the Linux NFS Server Suck Faster Greg Banks File Serving Technologies, Silicon Graphics, Inc Making The Linux NFS Server](https://reader035.vdocument.in/reader035/viewer/2022062407/56649e175503460f94b02a8c/html5/thumbnails/45.jpg)
Some Hotspots
• readahead params cache hash lock– global spinlock
– 1 lookup+insert, 1 modify per READ call
• split hash into 16 buckets, one lock per bucket
![Page 46: Headline in Arial Bold 30pt Making the Linux NFS Server Suck Faster Greg Banks File Serving Technologies, Silicon Graphics, Inc Making The Linux NFS Server](https://reader035.vdocument.in/reader035/viewer/2022062407/56649e175503460f94b02a8c/html5/thumbnails/46.jpg)
Some Hotspots
• duplicate reply cache hash lock– global spinlock
– 1 lookup, 1 insert per non-idempotent call (e.g. WRITE)
• more hash splitting
![Page 47: Headline in Arial Bold 30pt Making the Linux NFS Server Suck Faster Greg Banks File Serving Technologies, Silicon Graphics, Inc Making The Linux NFS Server](https://reader035.vdocument.in/reader035/viewer/2022062407/56649e175503460f94b02a8c/html5/thumbnails/47.jpg)
Some Hotspots
• lock for struct ip_map cache– YA global spinlock
• cached ip_map pointer in struct svc_sock -- for TCP
![Page 48: Headline in Arial Bold 30pt Making the Linux NFS Server Suck Faster Greg Banks File Serving Technologies, Silicon Graphics, Inc Making The Linux NFS Server](https://reader035.vdocument.in/reader035/viewer/2022062407/56649e175503460f94b02a8c/html5/thumbnails/48.jpg)
NUMA Factors: Problem
• Altix; presumably also Opteron, PPC
• CPU scheduler provides poor locality of reference– cold CPU caches
– aggravates hotspots
• ideally, want replies sent from CPUs close to the NIC– e.g. the CPU where the NIC's IRQs go
![Page 49: Headline in Arial Bold 30pt Making the Linux NFS Server Suck Faster Greg Banks File Serving Technologies, Silicon Graphics, Inc Making The Linux NFS Server](https://reader035.vdocument.in/reader035/viewer/2022062407/56649e175503460f94b02a8c/html5/thumbnails/49.jpg)
NUMA Factors: Solution
• make RPC threads node-specific using CPU mask
• only wake threads for packets arriving on local NICs– assumes bound IRQ semantics
– and no irqbalanced or equivalent
![Page 50: Headline in Arial Bold 30pt Making the Linux NFS Server Suck Faster Greg Banks File Serving Technologies, Silicon Graphics, Inc Making The Linux NFS Server](https://reader035.vdocument.in/reader035/viewer/2022062407/56649e175503460f94b02a8c/html5/thumbnails/50.jpg)
NUMA Factors: Solution
• new file /proc/fs/nfsd/pool_threads– sysadmin may get/set number of threads per pool
– default round-robins threads around pools
![Page 51: Headline in Arial Bold 30pt Making the Linux NFS Server Suck Faster Greg Banks File Serving Technologies, Silicon Graphics, Inc Making The Linux NFS Server](https://reader035.vdocument.in/reader035/viewer/2022062407/56649e175503460f94b02a8c/html5/thumbnails/51.jpg)
Mountstorm: Problem
• hundreds of clients try to mount in a few seconds– e.g. job start on compute cluster
• want parallelism, but Linux serialises mounts 3 ways
![Page 52: Headline in Arial Bold 30pt Making the Linux NFS Server Suck Faster Greg Banks File Serving Technologies, Silicon Graphics, Inc Making The Linux NFS Server](https://reader035.vdocument.in/reader035/viewer/2022062407/56649e175503460f94b02a8c/html5/thumbnails/52.jpg)
Mountstorm: Problem
• single threaded portmap
![Page 53: Headline in Arial Bold 30pt Making the Linux NFS Server Suck Faster Greg Banks File Serving Technologies, Silicon Graphics, Inc Making The Linux NFS Server](https://reader035.vdocument.in/reader035/viewer/2022062407/56649e175503460f94b02a8c/html5/thumbnails/53.jpg)
Mountstorm: Problem
• single threaded rpc.mountd
• blocking DNS reverse lookup
• & blocking forward lookup– workaround by adding all
clients to local /etc/hosts
• also responds to “upcall”
from kernel on 1st NFS call
![Page 54: Headline in Arial Bold 30pt Making the Linux NFS Server Suck Faster Greg Banks File Serving Technologies, Silicon Graphics, Inc Making The Linux NFS Server](https://reader035.vdocument.in/reader035/viewer/2022062407/56649e175503460f94b02a8c/html5/thumbnails/54.jpg)
Mountstorm: Problem
• single-threaded lookup of ip_map hashtable
• in kernel, on 1st NFS call
from new address
• spinlock held while traversing
• kernel little-endian 64bit IP
address hashing balance bug– > 99% of ip_map hash entries on one bucket
![Page 55: Headline in Arial Bold 30pt Making the Linux NFS Server Suck Faster Greg Banks File Serving Technologies, Silicon Graphics, Inc Making The Linux NFS Server](https://reader035.vdocument.in/reader035/viewer/2022062407/56649e175503460f94b02a8c/html5/thumbnails/55.jpg)
Mountstorm: Problem
• worst case: mounting takes so long that many clients timeout and the job fails.
![Page 56: Headline in Arial Bold 30pt Making the Linux NFS Server Suck Faster Greg Banks File Serving Technologies, Silicon Graphics, Inc Making The Linux NFS Server](https://reader035.vdocument.in/reader035/viewer/2022062407/56649e175503460f94b02a8c/html5/thumbnails/56.jpg)
Mountstorm: Solution
• simple patch fixes hash problem (thanks, iozone)
• combined with hosts workaround:
can mount 2K clients
![Page 57: Headline in Arial Bold 30pt Making the Linux NFS Server Suck Faster Greg Banks File Serving Technologies, Silicon Graphics, Inc Making The Linux NFS Server](https://reader035.vdocument.in/reader035/viewer/2022062407/56649e175503460f94b02a8c/html5/thumbnails/57.jpg)
Mountstorm: Solution
• multi-threaded rpc.mountd
• surprisingly easy
• modern Linux rpc.mountd keeps state– in files and locks access to them, or
– in kernel
• just fork() some more rpc.mountd processes!
• parallelises hosts lookup
• can mount 2K clients quickly
![Page 58: Headline in Arial Bold 30pt Making the Linux NFS Server Suck Faster Greg Banks File Serving Technologies, Silicon Graphics, Inc Making The Linux NFS Server](https://reader035.vdocument.in/reader035/viewer/2022062407/56649e175503460f94b02a8c/html5/thumbnails/58.jpg)
Duplicate reply cache: Problem
• sidebar: why have a repcache?
• see Olaf Kirch's OLS2006 paper
• non-idempotent (NI) calls
• call succeeds, reply sent, reply lost in network
• client retries, 2nd attempt fails: bad!
![Page 59: Headline in Arial Bold 30pt Making the Linux NFS Server Suck Faster Greg Banks File Serving Technologies, Silicon Graphics, Inc Making The Linux NFS Server](https://reader035.vdocument.in/reader035/viewer/2022062407/56649e175503460f94b02a8c/html5/thumbnails/59.jpg)
Duplicate reply cache: Problem
• repcache keeps copies of replies to NI calls
• every NI call must search before dispatch, insert after dispatch
• e.g. WRITE
• not useful if lifetime of records < client retry time (typ. 1100 ms).
![Page 60: Headline in Arial Bold 30pt Making the Linux NFS Server Suck Faster Greg Banks File Serving Technologies, Silicon Graphics, Inc Making The Linux NFS Server](https://reader035.vdocument.in/reader035/viewer/2022062407/56649e175503460f94b02a8c/html5/thumbnails/60.jpg)
Duplicate reply cache: Problem
• current implementation has fixed size 1024 entries: supports 930 calls/sec
• we want to scale to ~10^5 calls/sec
• so size is 2 orders of magnitude too small
• NFS/TCP rarely suffers from dups
• yet the lock is a global contention point
![Page 61: Headline in Arial Bold 30pt Making the Linux NFS Server Suck Faster Greg Banks File Serving Technologies, Silicon Graphics, Inc Making The Linux NFS Server](https://reader035.vdocument.in/reader035/viewer/2022062407/56649e175503460f94b02a8c/html5/thumbnails/61.jpg)
Duplicate reply cache: Solution
• modernise the repcache!
• automatic expansion of cache records under load
• triggered by largest age of a record falling below threshold
![Page 62: Headline in Arial Bold 30pt Making the Linux NFS Server Suck Faster Greg Banks File Serving Technologies, Silicon Graphics, Inc Making The Linux NFS Server](https://reader035.vdocument.in/reader035/viewer/2022062407/56649e175503460f94b02a8c/html5/thumbnails/62.jpg)
Duplicate reply cache: Solution
• applied hash splitting to reduce contention
• tweaked hash algorithm to reduce contention
![Page 63: Headline in Arial Bold 30pt Making the Linux NFS Server Suck Faster Greg Banks File Serving Technologies, Silicon Graphics, Inc Making The Linux NFS Server](https://reader035.vdocument.in/reader035/viewer/2022062407/56649e175503460f94b02a8c/html5/thumbnails/63.jpg)
Duplicate reply cache: Solution
• implemented hash resizing with lazy rehashing...
• for SGI NAS, not worth the complexity
• manual tuning of the hash size sufficient
![Page 64: Headline in Arial Bold 30pt Making the Linux NFS Server Suck Faster Greg Banks File Serving Technologies, Silicon Graphics, Inc Making The Linux NFS Server](https://reader035.vdocument.in/reader035/viewer/2022062407/56649e175503460f94b02a8c/html5/thumbnails/64.jpg)
CPU scheduler overload: Problem
• Denial of Service with high call load (e.g. rsync)
![Page 65: Headline in Arial Bold 30pt Making the Linux NFS Server Suck Faster Greg Banks File Serving Technologies, Silicon Graphics, Inc Making The Linux NFS Server](https://reader035.vdocument.in/reader035/viewer/2022062407/56649e175503460f94b02a8c/html5/thumbnails/65.jpg)
CPU scheduler overload: Problem
• knfsd wakes a thread for every call
• all 128 threads are runnable but only 4 have a CPU
• load average of ~120 eats the last few% in the scheduler
• only kernel nfsd threads ever run
![Page 66: Headline in Arial Bold 30pt Making the Linux NFS Server Suck Faster Greg Banks File Serving Technologies, Silicon Graphics, Inc Making The Linux NFS Server](https://reader035.vdocument.in/reader035/viewer/2022062407/56649e175503460f94b02a8c/html5/thumbnails/66.jpg)
CPU scheduler overload: Problem
• user-space threads don't schedule for...minutes
• portmap, rpc.mountd do not accept() new connections before client TCP timeout
• new clients cannot mount
![Page 67: Headline in Arial Bold 30pt Making the Linux NFS Server Suck Faster Greg Banks File Serving Technologies, Silicon Graphics, Inc Making The Linux NFS Server](https://reader035.vdocument.in/reader035/viewer/2022062407/56649e175503460f94b02a8c/html5/thumbnails/67.jpg)
CPU scheduler overload: Solution
• limit the # of nfsds woken but not yet on CPU
![Page 68: Headline in Arial Bold 30pt Making the Linux NFS Server Suck Faster Greg Banks File Serving Technologies, Silicon Graphics, Inc Making The Linux NFS Server](https://reader035.vdocument.in/reader035/viewer/2022062407/56649e175503460f94b02a8c/html5/thumbnails/68.jpg)
NFS over UDP: Problem
• bandwidth limited to ~145 MB/s no matter how many CPUs or NICs are used
• unlike TCP, a single socket is used for all UDP traffic
![Page 69: Headline in Arial Bold 30pt Making the Linux NFS Server Suck Faster Greg Banks File Serving Technologies, Silicon Graphics, Inc Making The Linux NFS Server](https://reader035.vdocument.in/reader035/viewer/2022062407/56649e175503460f94b02a8c/html5/thumbnails/69.jpg)
NFS over UDP: Problem
• when replying, knfsd uses the socket as a queue for building packets out of a header and some pages.
• while holding svc_socket->sk_sem
• so the UDP socket is a bottleneck
![Page 70: Headline in Arial Bold 30pt Making the Linux NFS Server Suck Faster Greg Banks File Serving Technologies, Silicon Graphics, Inc Making The Linux NFS Server](https://reader035.vdocument.in/reader035/viewer/2022062407/56649e175503460f94b02a8c/html5/thumbnails/70.jpg)
NFS over UDP: Solution
• multiple UDP sockets for receive
• 1 per NIC
• bound to the NIC (standard linux feature)
• allows multiple sockets to share the same port
• but device binding affects routing,
so can't send on these sockets...
![Page 71: Headline in Arial Bold 30pt Making the Linux NFS Server Suck Faster Greg Banks File Serving Technologies, Silicon Graphics, Inc Making The Linux NFS Server](https://reader035.vdocument.in/reader035/viewer/2022062407/56649e175503460f94b02a8c/html5/thumbnails/71.jpg)
NFS over UDP: Solution
• multiple UDP sockets for send
• 1 per CPU
• socket chosen in NFS reply send path
• new UDP_SENDONLY socket option
• not entered in the UDP port hashtable, cannot receive
![Page 72: Headline in Arial Bold 30pt Making the Linux NFS Server Suck Faster Greg Banks File Serving Technologies, Silicon Graphics, Inc Making The Linux NFS Server](https://reader035.vdocument.in/reader035/viewer/2022062407/56649e175503460f94b02a8c/html5/thumbnails/72.jpg)
Write performance to XFS
• Logic bug in XFS writeback path– On write congestion kupdated incorrectly blocks holding i_sem
– Locks out nfsd
• System can move bits– from network
– or to disk
– but not both at the same time
• Halves NFS write performance
![Page 73: Headline in Arial Bold 30pt Making the Linux NFS Server Suck Faster Greg Banks File Serving Technologies, Silicon Graphics, Inc Making The Linux NFS Server](https://reader035.vdocument.in/reader035/viewer/2022062407/56649e175503460f94b02a8c/html5/thumbnails/73.jpg)
Tunings
• maximum TCP socket buffer sizes
• affects negotiation of TCP window scaling at connect time
• from then on, knfsd manages its own buffer sizes
• tune 'em up high.
![Page 74: Headline in Arial Bold 30pt Making the Linux NFS Server Suck Faster Greg Banks File Serving Technologies, Silicon Graphics, Inc Making The Linux NFS Server](https://reader035.vdocument.in/reader035/viewer/2022062407/56649e175503460f94b02a8c/html5/thumbnails/74.jpg)
Tunings
• tg3 interrupt coalescing parameters
• bump upwards to reduce softirq CPU usage in driver
![Page 75: Headline in Arial Bold 30pt Making the Linux NFS Server Suck Faster Greg Banks File Serving Technologies, Silicon Graphics, Inc Making The Linux NFS Server](https://reader035.vdocument.in/reader035/viewer/2022062407/56649e175503460f94b02a8c/html5/thumbnails/75.jpg)
Tunings
• VM writeback parameters
• bump down dirty_background_ratio, dirty_writeback_centisecs
• try to get dirty pages flushed to disk before the COMMIT call
• alleviate effect of COMMIT latency on write throughput
![Page 76: Headline in Arial Bold 30pt Making the Linux NFS Server Suck Faster Greg Banks File Serving Technologies, Silicon Graphics, Inc Making The Linux NFS Server](https://reader035.vdocument.in/reader035/viewer/2022062407/56649e175503460f94b02a8c/html5/thumbnails/76.jpg)
Tunings
• async export option
• only for the brave
• can improve write performance...or kill it
• unsafe!! data not on stable storage but client thinks it is
![Page 77: Headline in Arial Bold 30pt Making the Linux NFS Server Suck Faster Greg Banks File Serving Technologies, Silicon Graphics, Inc Making The Linux NFS Server](https://reader035.vdocument.in/reader035/viewer/2022062407/56649e175503460f94b02a8c/html5/thumbnails/77.jpg)
Tunings
• no_subtree_check export option
• no security impact if you only export mountpoints
• can save nearly 10% CPU cost per-call
• technically more correct NFS fh semantics
![Page 78: Headline in Arial Bold 30pt Making the Linux NFS Server Suck Faster Greg Banks File Serving Technologies, Silicon Graphics, Inc Making The Linux NFS Server](https://reader035.vdocument.in/reader035/viewer/2022062407/56649e175503460f94b02a8c/html5/thumbnails/78.jpg)
Tunings
• Linux' ARP response behaviour suboptimal
• with shared media, client traffic jumps around randomly between links on ARP timeout
• common config when you have lots of NICs
• affects NUMA locality, reduces performance
• /proc/sys/net/ipv4/conf/$eth/arp_ignore .../arp_announce
![Page 79: Headline in Arial Bold 30pt Making the Linux NFS Server Suck Faster Greg Banks File Serving Technologies, Silicon Graphics, Inc Making The Linux NFS Server](https://reader035.vdocument.in/reader035/viewer/2022062407/56649e175503460f94b02a8c/html5/thumbnails/79.jpg)
Tunings
• ARP cache size
• default size overflows with about 1024 clients
• /proc/sys/net/ipv4/neigh/default/gc_thresh3
![Page 80: Headline in Arial Bold 30pt Making the Linux NFS Server Suck Faster Greg Banks File Serving Technologies, Silicon Graphics, Inc Making The Linux NFS Server](https://reader035.vdocument.in/reader035/viewer/2022062407/56649e175503460f94b02a8c/html5/thumbnails/80.jpg)
Overview
• Introduction
• Principles of Operation
• Performance Factors
• Performance Results
• Future Work
• Questions?
![Page 81: Headline in Arial Bold 30pt Making the Linux NFS Server Suck Faster Greg Banks File Serving Technologies, Silicon Graphics, Inc Making The Linux NFS Server](https://reader035.vdocument.in/reader035/viewer/2022062407/56649e175503460f94b02a8c/html5/thumbnails/81.jpg)
Bandwidth Test
• Throughput for streaming read, TCP, rsize=32K
4 6 80
100200300400500600700800900
1000
# CPUs, NICs, clients
Th
rou
gh
pu
t, M
iB/s Better
Theoretical Maximum
Before
After
![Page 82: Headline in Arial Bold 30pt Making the Linux NFS Server Suck Faster Greg Banks File Serving Technologies, Silicon Graphics, Inc Making The Linux NFS Server](https://reader035.vdocument.in/reader035/viewer/2022062407/56649e175503460f94b02a8c/html5/thumbnails/82.jpg)
Bandwidth Test: CPU Usage
• %sys+%intr CPU usage for streaming read, TCP, rsize=32K
4 6 80
200
400
600
800
1000
# CPUs, NICs, clients
CP
U u
sag
e %
Theoretical Maximum
Before
After
Better
![Page 83: Headline in Arial Bold 30pt Making the Linux NFS Server Suck Faster Greg Banks File Serving Technologies, Silicon Graphics, Inc Making The Linux NFS Server](https://reader035.vdocument.in/reader035/viewer/2022062407/56649e175503460f94b02a8c/html5/thumbnails/83.jpg)
Call Rate Test
• IOPS for in-memory rsync from simulated Linux 2.4 clients, 4 CPUs 4 NICs
0 50 100 150 200 2500
10000
20000
30000
40000
50000
60000
70000
80000
90000
# virtual clients
rsyn
c IO
PS
Before
After
Still going...got bored
Overload
Better
![Page 84: Headline in Arial Bold 30pt Making the Linux NFS Server Suck Faster Greg Banks File Serving Technologies, Silicon Graphics, Inc Making The Linux NFS Server](https://reader035.vdocument.in/reader035/viewer/2022062407/56649e175503460f94b02a8c/html5/thumbnails/84.jpg)
Call Rate Test: CPU Usage
• %sys +%intr CPU usage for in-memory rsync from simulated Linux 2.4 clients
0 50 100 150 200 2500
50
100
150
200
250
300
350
400
450
# virtual clients
%
CPU
usage
Still going...got bored
Overload
BeforeAfter
Better
![Page 85: Headline in Arial Bold 30pt Making the Linux NFS Server Suck Faster Greg Banks File Serving Technologies, Silicon Graphics, Inc Making The Linux NFS Server](https://reader035.vdocument.in/reader035/viewer/2022062407/56649e175503460f94b02a8c/html5/thumbnails/85.jpg)
Performance Results
• More than doubled SPECsfs result
• Made possible the 1st published Altix SPECsfs result
![Page 86: Headline in Arial Bold 30pt Making the Linux NFS Server Suck Faster Greg Banks File Serving Technologies, Silicon Graphics, Inc Making The Linux NFS Server](https://reader035.vdocument.in/reader035/viewer/2022062407/56649e175503460f94b02a8c/html5/thumbnails/86.jpg)
Performance Results
• July 2005: SLES9 SP2 test on customer site "W" with 200 clients: failure
![Page 87: Headline in Arial Bold 30pt Making the Linux NFS Server Suck Faster Greg Banks File Serving Technologies, Silicon Graphics, Inc Making The Linux NFS Server](https://reader035.vdocument.in/reader035/viewer/2022062407/56649e175503460f94b02a8c/html5/thumbnails/87.jpg)
Performance Results
• July 2005: SLES9 SP2 test on customer site "W" with 200 clients: failure
• May 2006: Enhanced NFS test on customer site "P" with 2000 clients: success
![Page 88: Headline in Arial Bold 30pt Making the Linux NFS Server Suck Faster Greg Banks File Serving Technologies, Silicon Graphics, Inc Making The Linux NFS Server](https://reader035.vdocument.in/reader035/viewer/2022062407/56649e175503460f94b02a8c/html5/thumbnails/88.jpg)
Performance Results
• July 2005: SLES9 SP2 test on customer site "W" with 200 clients: failure
• May 2006: Enhanced NFS test on customer site "P" with 2000 clients: success
• Jan 2006: customer “W” again...fingers crossed!
![Page 89: Headline in Arial Bold 30pt Making the Linux NFS Server Suck Faster Greg Banks File Serving Technologies, Silicon Graphics, Inc Making The Linux NFS Server](https://reader035.vdocument.in/reader035/viewer/2022062407/56649e175503460f94b02a8c/html5/thumbnails/89.jpg)
Overview
• Introduction
• Principles of Operation
• Performance Factors
• Performance Results
• Future Work
• Questions?
![Page 90: Headline in Arial Bold 30pt Making the Linux NFS Server Suck Faster Greg Banks File Serving Technologies, Silicon Graphics, Inc Making The Linux NFS Server](https://reader035.vdocument.in/reader035/viewer/2022062407/56649e175503460f94b02a8c/html5/thumbnails/90.jpg)
Read-Ahead Params Cache
• cache of struct raparm so NFS files get server-side readahead behaviour
• replace with an open file cache– avoid fops->release on XFS truncating speculative allocation
– avoid fops->open on some filesystems
![Page 91: Headline in Arial Bold 30pt Making the Linux NFS Server Suck Faster Greg Banks File Serving Technologies, Silicon Graphics, Inc Making The Linux NFS Server](https://reader035.vdocument.in/reader035/viewer/2022062407/56649e175503460f94b02a8c/html5/thumbnails/91.jpg)
Read-Ahead Params Cache
• need to make the cache larger– we use it for writes as well as reads
– current sizing policy depends on #threads
• issue of managing new dentry/vfsmount references– removes all hope of being able to unmount an exported filesystem
![Page 92: Headline in Arial Bold 30pt Making the Linux NFS Server Suck Faster Greg Banks File Serving Technologies, Silicon Graphics, Inc Making The Linux NFS Server](https://reader035.vdocument.in/reader035/viewer/2022062407/56649e175503460f94b02a8c/html5/thumbnails/92.jpg)
One-copy on NFS Write
• NFS writes now require two memcpy– network sk_buff buffers -> nfsd buffer pages
– nfsd buffer pages -> VM page cache
• the 1st of these can be removed
![Page 93: Headline in Arial Bold 30pt Making the Linux NFS Server Suck Faster Greg Banks File Serving Technologies, Silicon Graphics, Inc Making The Linux NFS Server](https://reader035.vdocument.in/reader035/viewer/2022062407/56649e175503460f94b02a8c/html5/thumbnails/93.jpg)
One-copy on NFS Write
• will remove need for most RPC thread buffering– make nfsd memory requirements independent of number of threads
• will require networking support– new APIs to extract data from sockets without copies
• will require rewrite of most of the server XDR code
• not a trivial undertaking
![Page 94: Headline in Arial Bold 30pt Making the Linux NFS Server Suck Faster Greg Banks File Serving Technologies, Silicon Graphics, Inc Making The Linux NFS Server](https://reader035.vdocument.in/reader035/viewer/2022062407/56649e175503460f94b02a8c/html5/thumbnails/94.jpg)
Dynamic Thread Management
• number of nfsd threads is a crucial tuning– Default (4) is almost always too small
– Large (128) is wasteful, and can be harmful
• existing advice for tuning is frequently wrong
• no metrics for correctly choosing a value– existing stats hard to explain & understand, and wrong
![Page 95: Headline in Arial Bold 30pt Making the Linux NFS Server Suck Faster Greg Banks File Serving Technologies, Silicon Graphics, Inc Making The Linux NFS Server](https://reader035.vdocument.in/reader035/viewer/2022062407/56649e175503460f94b02a8c/html5/thumbnails/95.jpg)
Dynamic Thread Management
• want automatic mechanism:
• control loop driven by load metrics
• sets # of threads
• NUMA aware
• manual limits on threads, rates of change
![Page 96: Headline in Arial Bold 30pt Making the Linux NFS Server Suck Faster Greg Banks File Serving Technologies, Silicon Graphics, Inc Making The Linux NFS Server](https://reader035.vdocument.in/reader035/viewer/2022062407/56649e175503460f94b02a8c/html5/thumbnails/96.jpg)
Multi-threaded Portmap
• portmap has read-mostly in-memory database
• not as trivial to MT as rpc.mountd was!
• will help with mountstorm, a little
• code collision with NFS/IPv6 renovation of portmap?
![Page 97: Headline in Arial Bold 30pt Making the Linux NFS Server Suck Faster Greg Banks File Serving Technologies, Silicon Graphics, Inc Making The Linux NFS Server](https://reader035.vdocument.in/reader035/viewer/2022062407/56649e175503460f94b02a8c/html5/thumbnails/97.jpg)
Acknowledgements
• this talk describes work performed at SGI Melbourne, July 2005 – June 2006– thanks for letting me do it
– ...and talk about it.
– thanks for all the cool toys.
![Page 98: Headline in Arial Bold 30pt Making the Linux NFS Server Suck Faster Greg Banks File Serving Technologies, Silicon Graphics, Inc Making The Linux NFS Server](https://reader035.vdocument.in/reader035/viewer/2022062407/56649e175503460f94b02a8c/html5/thumbnails/98.jpg)
Acknowledgements
• kernel & nfs-utils patches described are being submitted
• thanks to code reviewers– Neil Brown, Andrew Morton, Trond Myklebust, Chuck Lever,
Christoph Hellwig, J Bruce Fields and others.
![Page 99: Headline in Arial Bold 30pt Making the Linux NFS Server Suck Faster Greg Banks File Serving Technologies, Silicon Graphics, Inc Making The Linux NFS Server](https://reader035.vdocument.in/reader035/viewer/2022062407/56649e175503460f94b02a8c/html5/thumbnails/99.jpg)
References
• SGI http://www.sgi.com/storage/.
• Olaf Kirch, “Why NFS Sucks”, http://www.linuxsymposium.org/2006/linuxsymposium_procv2.pdf
• PCP http://oss.sgi.com/projects/pcp
• Oprofile http://oprofile.sourceforge.net/
• fsx http://www.freebsd.org/cgi/cvsweb.cgi/src/tools/regression/fsx/
• SPECsfs http://www.spec.org/sfs97r1/
• fsstress http://oss.sgi.com/cgi-bin/cvsweb.cgi/xfs-cmds/xfstests/ltp/
• TBBT http://www.eecs.harvard.edu/sos/papers/P149-zhu.pdf
![Page 100: Headline in Arial Bold 30pt Making the Linux NFS Server Suck Faster Greg Banks File Serving Technologies, Silicon Graphics, Inc Making The Linux NFS Server](https://reader035.vdocument.in/reader035/viewer/2022062407/56649e175503460f94b02a8c/html5/thumbnails/100.jpg)
Advertisement
• SGI Melbourne is hiring!– Are you a Linux kernel engineer?
– Do you know filesystems or networks?
– Want to do QA in an exciting environment?
– Talk to me later
![Page 101: Headline in Arial Bold 30pt Making the Linux NFS Server Suck Faster Greg Banks File Serving Technologies, Silicon Graphics, Inc Making The Linux NFS Server](https://reader035.vdocument.in/reader035/viewer/2022062407/56649e175503460f94b02a8c/html5/thumbnails/101.jpg)
Overview
• Introduction
• Principles of Operation
• Performance Factors
• Performance Results
• Future Work
• Questions?