rakuten leofs - distributed file system

1

S3 Compatible Storage “LeoFS”

Rakuten. Inc,　RIT Yosuke Hara　25/05/2012

2

Table of Contents

1. Motivation

2. Overview

3. Inside of LeoFS

4. WEB Console

3

1. Motivation

4

Which is suitable storage for storing the media files?

? Low ROI Possibility of SPOF Storage Expansion is difficult during increasing data

Why NFS?

5

6

Object Storage Farm for IaaS

7

2. Overview

8

Storage

Gateway

Manager

Overview

9

LeoFS-Manager

LeoFS-Gateway w/Cache Server

LeoFS-Storage

REST over HTTP RPC

Request from Web Application(s)

META Object Store

Storage Engine/Router

META Object Store META Object Store

RPC

Storage Engine/Router Storage Engine/Router

Load Balancer

S3-API

SNMP

WEB Console

System Layout

10

LeoFS-Manager

LeoFS-Gateway w/Cache Server

LeoFS-Storage

REST over HTTP RPC

Request from Web Application(s)

META Object Store

Storage Engine/Router

META Object Store META Object Store

RPC

Storage Engine/Router Storage Engine/Router

Load Balancer

S3-API

SNMP

GUI Console

Gateway HTTP

Request/Response Handling +

w/Object Cache (AWS S3-API)

Manager Cluster

Management

Ring Watcher Node Watcher

Storage Object Storage, Meta data Storage

+ Replicator/Recoverer

System Layout

11

3. Inside of LeoFS

12

Manager Cluster

Storage Cluster (multi master)

Gateway (stateless)

Process Monitor

Erlang RPC

HTTP

Erlang RPC

Erlang RPC

Architecture

13

LeoFS Gateway REST over HTTP (S3-API)

membership (fault-detection)

redundant-manager replicator

queue

Storage Engine

Object Storage Metadata Storage

LeoFS-Storage

read-repairer

redundant-manager membership (fault-detection)

get put delete head

Architecture - Gateway/Storage

RPC

RPC RPC

Cacher

Architecture

14

Erlang Mnesia

Architecture - Manager Cluster

Gateway / Storage Cluster

RING Member / Cluster State

Auth / ACL

Process Monitor

Architecture

15

Architecture

16

High Cost Performance

Monolithic Storage System

Storage Engine For Unstructured Files

Traffic Restrain Mechanism

> File Cache System (Gateway Plugin)

Three “HIGH”

17

Three “HIGH”

High Reliability

NO SPOF

Split Brain Measure

“Erlang OTP” > Nine Nines (99.9999999%)

18

Three “HIGH”

High Scalability

Elastic Storage System

> Able to dynamic attach/detach nodes

> Able to over 100-nodes

> NOT Mesh-connected Mutual Servers

19

Cache Mechanism

20

High I/O efficiency

Low Latency

Gateway Buffer Pool

Gateway

from Client

Slab Alloc Skiplist

request

{$filename, $etag}

response

match: {ok, match} NOT match: {ok, $metadata, $body,}

Cache Mechanism

21

Storage Engine

22

Metadata + Object Storage

Metadata : Keeps an in-memory index of all data. Object Storage : Log structured (append-only) object store.

Object’s Attribute Storage (metadata) Object Storage

LeoFS-Storage

Storage Engine / Replicator / Recoverer

Storage Engine

23

< META DATA >IDFilenameOffsetSizeChecksum

Metadata

Data

Object Container

Retrieve an object from Storage

Header

File

Footer

< META DATA >IDFilenameOffsetSizeChecksum

Storage Engine

Storage Engine

24

Meta Data Server

Append a File

Insert an object into Storage

Data

Add a Metadata

Storage Engine

Storage Engine

25

Reduce unnecessary files

Compaction

Storage Engine

26

4. WEB Console

27

File Manager

Node Stats

Log Search

WEB Console

Web Console

28

Web Console

Log Analyzer / Searcher

Node State Monitor

29

Storage Gateway Manager

Logstatsh

or

Producers and Admins

GUI Console

Web Console System Layout

Web Console

30

Wrap Up

31

Wrap Up

High Cost Performance High Reliability High Scalability

32

33

Thank you for your time