panasas at the rcf

Robert Petkus – Panasas at the RCF

Panasas at the RCF

HEPiX at SLACFall 2005

Robert Petkus

RHIC/USATLAS Computing FacilityBrookhaven National Laboratory


Centralized File Service

● Single, facility-wide namespace for files.● Uniform, facility-wide “POSIX-like” access to data.● Transparent to end-user.


Traditional SAN● ~220 TB fibre-channel disks in RAID 5 arrays● 13 TB IDE storage● 24 Brocade fibre-channel switches● 37 Sun Solaris 9 servers (E450, V480, V240) using Veritas 4.0 (VxVM, VxFS)● NFS transfer rate: 70MB/sec/server ● I/O throughput to disks: 70 – 90 MB/sec writes, 75 MB/sec reads

Current Storage Implementation


Current Storage Implementation

Panasas● 100 TB on 20 shelves.● 10 Realms.● 2 trunked 1000BaseT network ports (out of 4)

enabled.● Data transfer rate from server ~250 MB/sec max

per shelf.● DirectFLOW client deployed on ~2000 node RHIC

analysis/reconstruction farm.


Current view of the facility


NFS ConcernsNFS● Load balancing● Scaling Issues

– Horizontal (management)– Vertical (performance)

● Security concerns


Veritas IssuesVXFS and VXVM● Pros

– Ability to shrink file systems.– Easy to import/deport volumes among servers.– Dynamic multipathing

● Cons– Quotas broken on file systems > 1 TB– Expensive– Poor customer support


Panasas Features● Fast, scalable, reliable and fault tolerant● Direct and parallel data access● Load balancing (Efficient resource utilization)● Incremental growth● Integrated hardware / software solution● Economic benefits of IDE disk● Security and centralized management● Global namespace● Distributed metadata● Active development● Excellent customer support


Spectrum of Solutions● GPFS: data block striping across disks. Parallel reads/writes.

Fail-over, replication, distributed metadata.● Xrootd: Load balancing, single namespace, proxy servers.

Client access through TXNetFile (ROOT framework)● BlueArc: ASICs dedicated to NFS, network, and filesystem.

Feature-rich, fast and robust NFS solution. ● Ibrix: Meta servers assigned to segments in a disk pool.

Traditional NFS or “Fusion”. Linux LVS model.● Dcache: Management of heterogeneous storage repositories.

Hot spot detection. ● Lustre: Object-based storage solution. Software only. Active

development. Older versions are free.


Various Models

● POSIX-like vs Non POSIX-like● Hardware vs Software● Existing Protocol vs New Protocol● Ethernet vs Fibre Channel● “Unlimited” Scalability vs “Just Fast Enough”● Dedicated vs Non-Dedicated Resources● Proprietary vs “Open”


Panasas Architecture Components of each shelf at the RCF

● 1 Director Blade● 10 Storage Blades● 1 Gigabit internal ethernet ports per blade (11 total)● 4 Gigabit links to external network● 5 or 8 TB raw disk/shelf


Panasas Architecture – Director BladeDirector Blades (The brains)

• File Namespace server(s)• Manages Metadata object map• Coordinates between clients and Storage Blades• Determines “RAID” characteristics of a file.• Determines distribution of file objects over OSD'sSpecifications• 2.4 GHz Intel Xeon• 4 GB RAM• 2 1000BaseT ports• FreeBSD


Panasas Architecture – Storage Blade

Storage Blades (Object Storage Devices or OSD's)• Store and retrieve data objects• Handles I/O to clientSpecifications

● Each Storage Blade (10)• 500GB or 800GB storage (2 SATA 250GB or 400GB

HDD)• 1.2 GHz Intel Celeron• 512 MB RAM• 100/1000BaseT• FreeBSD


Panasas Architecture● ActiveScale Operating System

• Runs on Director Blade• Divides files into data objects, which are arbitrary in size, and stripes

them across storage blades• Dynamically distributes workload across storage blades• Each storage blades is only filled to 90% capacity. The remaining 10%

is reserved for rebuilding parity.● Direct Flow Software

• Installed on the Linux compute node• Direct data path from client to storage blades• Optimizes data layout, caching and prefetching• File is reconstructed at the compute node


Panasas performance snapshots


Issues● No LDAP support● No user or group quotas – volume quotas only

– A new command “pan_du” is in development to list the amount of storage used per user but no enforcement.

● No on-line director blade failover● Can't downgrade ActiveScale● DirectFLOW memory leaks – fixed?● DHCP server stubborn to release addresses – need an excess pool of IP addresses in case a blade crashes.


Issues

● Maxed-out volumes are unstable. Hard volume quotas need to be set at 95% of total usable capacity for protection.● Atime records written to disk on writes not reads (JLAB)● Hard limit on files per directory (350k)● Mass-deletes are not handled gracefully.


Conclusions● Panasas scalability and performance much better

than NFS.● Easy to administrate and reliable.● Responsive, knowledgeable support staff.● Intermittent client-side DirectFLOW crashes are

frustrating on a very large cluster.● Possibility of separating software from hardware?● Emergence of strong competitors.● NFSv4.1?● What role will centralized storage play in a grid

environment ?

panasas at the rcf

Documents