hydrastor: a scalable secondary storage · 2019-02-25 · hydrastor: a scalable secondary storage....

HYDRAstor:a Scalable Secondary Storage

C. Dubnicki, L. Gryz, L. Heldt,M. Kaczmarczyk, W. Kilian,P. Strzelczak, J. Szczepkowski,M. Welnicki

C. Ungureanu

7th USENIX Conference on File and Storage Technologies(FAST '09)

February 26th 2009

HYDRAstor: a Scalable Secondary Storage. 9LivesData, LLC 2

Scalable secondary storage

Characteristics RequirementsHuge amount of data - Scalability (dynamic)

- Low cost per TBSmall backup windows - Very high write performanceDuplication between backup streams

- Global deduplication

Reliable, on-line retrieval - Failure tolerance- High restore performance

Varying value of data - Adjust resilience overhead- Data deletion


Challenges

● High-performance, decentralized

global deduplication

... in a dynamic, distributed system

... with deletion and failures● Combination introduces complexity● Tension between:

● Deduplication and dynamic scalability● Deduplication and on-demand deletion● Failure tolerance and deletion


● Satisfies Scalable secondary storage requirements

● Started as a research project atNEC Laboratories America, in Princeton, NJ

● Successfully commercialized● Today: real-world, commercial system ● Sold by NEC in the US and Japan

● Development of back-end continues at 9LivesData, LLC in Warsaw, Poland● Spinoff from NEC Laboratories


HYDRAstor functionality

● Content addressable storage (CAS)● Vast data repository

● Storing and extracting streams of blocks● Single system image built of independent nodes

● Support for standard access methods● Filesystem, VTL

● Dynamic capacity sharing● Self-recovery from failures● On-demand deletion


Programming Model

● Repository of blocks● Content-addressed● Immutable● Variable-sized

hash=011..0


Programming Model


● Exposed pointers to other blocks

E

hash=011..0

011.

.0


Programming Model



● Trees of blocks E

EE

ERoot1

E

hash=010..1

hash=011..0

011.

.0


Programming Model



● Trees of blocks● DAGs due to deduplication● No cycles possible

E

EE

011.

.0

ERoot1

E

ERoot2hash=010..1

hash=110..0

hash=011..0

011.

.0


Programming Model




● Deletion of whole trees

E

EE

011.

.0

ERoot1

E

ERoot2hash=010..1

hash=110..0

hash=011..0

011.

.0


Programming Model





E

EE

011.

.0

ERoot1

E

ERoot2hash=010..1

hash=110..0

hash=011..0

011.

.0


Programming Model





E 011.

.0

ERoot2

hash=110..0

hash=011..0

011.

.0


Architecture overview

● Standard server-grade hardware running Linux● Scalability on data-center level

Storage Nodes

Access Nodes

NFS / CIFS

Front-end

Back-end(CAS Layer)

InternalNetwork


Data organization:selected requirements

Requirements on scalable storage

Required internaldata services

Failure tolerance ● Identify data resilience reduction● Fast data rebuilding

High performance ● Preserve locality of data streams● Prefetching

Dynamic scalability ● Decentralized data management● Load balancing● Fast data transfer to new location

Deduplication ● Location of potential duplicates● Availability & resiliency verification

On-demand deletion ● Failure-tolerant, distributed deletion


Failure tolerance: erasure coding

Deco

de

Any 3 fragments can be lost

Example: N=8, m=5E

ncod

e

Original block

Ori

gina

l F

ragm

ents

R

edun

dant

F

ragm

ents

● Block erasure-coded into N fragments● Storage overhead tunable


Scalability with DHT: data placement

● Block location: DHT with prefix routing

0 1

01 10 11

empty prefix

00

0

01



● Block location: DHT with prefix routing● Block mapped to hash prefix hash=011..0

0 1

01 10 11

empty prefix

00

Block

0

01



● Block location: DHT with prefix routing● Block mapped to hash prefix● Prefix components

● Hosted on SNs● N components

per prefix

hash=011..0

Block

Node 1Node 6

Node 1Node 5

Node 1Node 4

Node 1Node 3

Node 1Node 2

Node 1Node 1

0 1

1

3

2

0 0

1

2

3

0

1

2

3

2

3

1

0

01 10 11

empty prefix

00

N=4



hash=011..0

Block

Node 1Node 6

Node 1Node 5

Node 1Node 4

Node 1Node 3

Node 1Node 2

Node 1Node 1

0 1

1

3

2

0 0

1

2

3

0

1

2

3

2

3

1

0

01 10 11

empty prefix

00

N=4



per prefix● Store fragments



hash=011..0

Block

Node 1Node 6

Node 1Node 5

Node 1Node 4

Node 1Node 3

Node 1Node 2

Node 1Node 1

0 1

1

3

2

0 0

1

2

3

0

1

2

3

2

3

1

0

01 10 11

empty prefix

00

N=4




● Distributedconsensus



hash=011..0

Block

Node 1Node 6

Node 1Node 5

Node 1Node 4

Node 1Node 3

Node 1Node 2

Node 1Node 1

0 1

1

3

2

0 0

1

2

3

0

1

2

3

2

3

1

0

01 10 11

empty prefix

00

N=4







hash=011..0

Block

Node 1Node 6

Node 1Node 5

Node 1Node 4

Node 1

Node 1

Node 3

Node 2

Node 1Node 1

0 1

1

3

2

0 0

1

2

3

0

1

2

3

2

3

1

0

01 10 11

empty prefix

00

N=4







hash=011..0

Block

Node 1Node 6

Node 1Node 5

Node 1Node 4

Node 1Node 3

Node 1Node 2

Node 1Node 1

0 1

1

3

2

0 0

1

2

3

0

1

2

3

2

3

1

0

01 10 11

empty prefix

00

N=4





● Load balancing


Data organization: synchrun chains

A B EC D F G● Data stream split to blocks



A B EC D F G

Hash 010…

Hash 101…

Hash 110…

Hash 011…

Hash 000…

Hash 011…

Hash 100…

● Data stream split to blocks

● Hashes of blocks computed



A B EC D F G

Hash 010…

Hash 101…

Hash 110…

Hash 011…

Hash 000…

Hash 011…

Hash 100…



● Routing through DHT



A B EC D F G

Hash 010…

Hash 101…

Hash 110…

Hash 011…

Hash 000…

Hash 011…

Hash 100…



● Routing through DHTPrefix 01



A B EC D F G

Hash 010…

Hash 101…

Hash 110…

Hash 011…

Hash 000…

Hash 011…

Hash 100…

Erasure Coding

Compression



● Routing through DHTPrefix 01



A B EC D F G

Hash 010…

Hash 101…

Hash 110…

Hash 011…

Hash 000…

Hash 011…

Hash 100…

Prefix 01

Erasure Coding

Compression




Component0

Component1

Component2

Component3

● Erasure-coded fragments stored by components



A B EC D F G

Hash 010…

Hash 101…

Hash 110…

Hash 011…

Hash 000…

Hash 011…

Hash 100…

Erasure Coding

Compression




A D F

A D F

A D F

A D F

Component0

Component1

Component2

Component3

Prefix 01




A B EC D F G

Hash 010…

Hash 101…

Hash 110…

Hash 011…

Hash 000…

Hash 011…

Hash 100…

Synchrun 1 Synchrun 2 Synchrun 3

Prefix 01

Erasure Coding

Compression

Synchrun




Component0

Component1

Component2

Component3


● Grouped into synchruns

A D F

A D F

A D F

A D F



A B EC D F G

Hash 010…

Hash 101…

Hash 110…

Hash 011…

Hash 000…

Hash 011…

Hash 100…


Prefix 01

Erasure Coding

Compression




Component0

Component1

Component2

Component3

Container



● Containers stored on disks

● Fragment metadata separately from data

Synchrun

A D F

A D F

A D F

A D F



A B EC D F G

Hash 010…

Hash 101…

Hash 110…

Hash 011…

Hash 000…

Hash 011…

Hash 100…


Erasure Coding

Compression




A D F

A D F

A D F

A D F

Component0

Component1

Component2

Component3

Prefix 01



● Containers stored on disks

● Fragment metadata separately from data

● Ordered synchrun chains

● Preserve order & locality

● ManageableContainer Synchrun


Component01:1

Synchrun chains in a dynamic system


Component01:1

System growth: split

Component010:1

Component011:1


Component01:1

Component010:1

Component011:1



Component010:1

Component011:1


Component01:1


Component01:1

Concatenation

Component010:1


Component01:1

Concatenation

Component010:1

Component010:1


Component01:1

Component010:1

Marking blocks to reclaim

Component010:1


Component01:1

Component010:1

Space reclamation & Concatenation

Component010:1

Component010:1


Component01:0

Component01:1

Component01:2

Component01:3

Data Services:Identification of data resiliency level

Missing fragments



Component01:0

Component01:1

Component01:2

Component01:3

Chain scanning


Data services: reconstruction

Component01:0

Component01:1

Component01:2

Component01:3

● Sequential read/write of entire Containers● Erasure decoding and re-encoding



Component01:0

Component01:1

Component01:2

Component01:3



Data services: fast data transfer

Component01:0

Component01:1

Component01:2

Component01:3

Old component

01:3

Location of newnode (DHT)



Component01:0

Component01:1

Component01:2

Component01:3

Old component

01:3

Data transfer



Component01:0

Component01:1

Component01:2

Component01:3

Old component

01:3


Data services for deduplication

● Global: duplicates detected in entire system● DHT routing based on content● Inline deduplication: has to be high-performance

● Prefetching Containers for streams of duplicates● Block hashes stored separately


Data services for deduplication

Component

01:0

Component

01:1

Component

01:2

Component

01:3

hash=011..

Block

Choose completechain

Completeness: “definitely not a duplicate”Deletion interaction: wasn't the block scheduled for deletion?


Data services for deduplicationhash=011..

Block

Component

01:0

Component

01:1

Component

01:2

Component

01:3

Query



Block

Local candidate found

Component

01:0

Component

01:1

Component

01:2

Component

01:3



Block

Candidate verification

Successfuldedup

Component

01:0

Component

01:1

Component

01:2

Component

01:3


On-demand data deletion

● Distributed garbage collection● Per-block reference counter stored per-

fragment● Failure-tolerant

● Block reference counter calculated independently on peer Container chains

● Interference with duplicate elimination:● read-only phase for block tree traversal● space reclamation in background


Writes during node failure

Writing

Reconstruction


Write Scalingnodes added while writing


Questions?

hydrastor: a scalable secondary storage · 2019-02-25 · hydrastor: a scalable secondary storage....

Documents