Ceph: A Scalable, High-Performance

Distributed File System

Sage WeilScott BrandtEthan MillerDarrell Long

Carlos MaltzahnUniversity of California, Santa Cruz

Project Goal Reliable, high-performance distributed file system

with excellent scalability Petabytes to exabytes, multi-terabyte files, billions of files Tens or hundreds of thousands of clients simultaneously

accessing same files or directories POSIX interface

Storage systems have long promised scalability, but have failed to deliver Continued reliance on traditional file systems principles

Inode tables Block (or object) list allocation metadata

Passive storage devices

Ceph—Key Design Principles

1. Maximal separation of data and metadata Object-based storage Independent metadata management CRUSH – data distribution function

2. Intelligent disks Reliable Autonomic Distributed Object Store

3. Dynamic metadata management Adaptive and scalable

Outline

1. Maximal separation of data and metadata Object-based storage Independent metadata management CRUSH – data distribution function

2. Intelligent disks Reliable Autonomic Distributed Object Store

3. Dynamic metadata management Adaptive and scalable

Object Interface

File System

Storage component

ApplicationsApplications

Logical Block Interface

File System

Traditional Storage Object-based Storage

Hard Drive Object-based Storage Device (OSD)

Object-based StorageParadigm

Client

Metadata Servers (MDS)

Ceph—Decoupled Data and Metadata

File System

Applications

Storage component

Object-based Storage Device (OSD)

Metadata Manager

Object Storage Devices (OSDs)

Client

CRUSH—Simplifying Metadata Conventionally

Directory contents (filenames) File inodes

Ownership, permissions File size Block list

CRUSH Small “map” completely specifies data distribution Function calculable everywhere used to locate objects Eliminates allocation lists Inodes “collapse” back into small, almost fixed-sized structures

Embed inodes into directories that contain them No more large, cumbersome inode tables

Outline

1. Maximal separation of data and metadata Object-based storage Independent metadata management CRUSH – data distribution function

2. Intelligent disks Reliable Autonomic Distributed Object Store

3. Dynamic metadata management Adaptive and scalable

RADOS

EBOFS

RADOS

EBOFS

RADOS—Reliable AutonomicDistributed Object Store Ceph OSDs are intelligent

Conventional drives only respond to commands OSDs communicate and collaborate with their peers

CRUSH allows us to delegate data replication failure detection failure recovery data migration

OSDs collectively form a single logical object store Reliable Self-managing (autonomic) Distributed

RADOS manages peer and client interaction EBOFS manages local object storage

RADOS –Scalability Failure detection and recovery are distributed

Centralized monitors used only to update map Maps updates are propagated by OSDs themselves

No monitor broadcast necessary

Identical “recovery” procedure used to respond to all map updates OSD failure Cluster expansion

OSDs always collaborate to realize the newly specified data distribution

EBOFS—Low-level object storage Extent and B-tree-based Object File System Non-standard interface and semantics

Asynchronous notification of commits to disk Atomic compound data+metadata updates

Extensive use of copy-on-write Revert to consistent state after failure

User-space implementation We define our own interface—not limited by ill-

suited kernel file system interface Avoid Linux VFS, page cache—designed

under different usage assumptions

RADOS

EBOFS

OSD Performance—EBOFS vs ext3, ReiserFSv3, XFS

EBOFS writes saturate disk for request sizes over 32k

Reads perform significantly better for large write sizes

Outline

1. Maximal separation of data and metadata Object-based storage Independent metadata management CRUSH – data distribution function

2. Intelligent disks Reliable Autonomic Distributed Object Store

3. Dynamic metadata management Adaptive and scalable

Metadata—Traditional Partitioning

Coarse distribution (static subtree partitioning) hierarchical partition preserves locality high management overhead: distribution becomes imbalanced

as file system, workload change Finer distribution (hash-based partitioning)

probabilistically less vulnerable to “hot spots,” workload change destroys locality (ignores underlying hierarchical structure)

Directory Hashing Hash on directory portion

of path only

Coarse partition Fine partition

Static Subtree Partitioning Portions of file hierarchy

are statically assigned to MDS nodes (NFS, AFS, etc.)

File Hashing Metadata distributed

based on hash of full path (or inode #)

Dynamic Subtree Partitioning

Root

Busy directory hashed across many MDS’s

MDS 0

MDS 1

MDS 2

MDS 3

MDS 4

Scalability Arbitrarily partitioned metadata

Adaptability Cope with workload changes over time, and hot spots

Metadata Scalability

Up to 128 MDS nodes, and 250,000 metadata ops/second I/O rates of potentially many terabytes/second Filesystems containing many petabytes (or exabytes?) of data

Conclusions Decoupled metadata improves scalability

Eliminating allocation lists makes metadata simple MDS stays out of I/O path

Intelligent OSDs Manage replication, failure detection, and recovery

CRUSH distribution function makes it possible Global knowledge of complete data distribution Data locations calculated when needed

Dynamic metadata management Preserve locality, improve performance Adapt to varying workloads, hot spots Scale

High-performance and reliability with excellent scalability!

Ongoing and Future Work

Completion of prototype MDS failure recovery Scalable security architecture [Leung, StorageSS ’06]

Quality of service Time travel (snapshots) RADOS improvements

Dynamic replication of objects based on workload Reliability mechanisms: scrubbing, etc.

Thanks!

http://ceph.sourceforge.net/

Support fromLawrence Livermore, Los Alamos, and Sandia

National Laboratories