hydrastor: a scalable secondary storage · 2010-09-08 · hydrastor: a scalable secondary storage....

HYDRAstor:a Scalable Secondary Storage

7th TF-Storage Meeting

September 9th 2010

Łukasz Heldt

Largest Japanese IT company$43 Billion in annual revenue143,000 staffwww.nec.com

Polish R&D company50 engineers and scientistswww.9livesdata.com

Scalable disk based storage for backup with global deduplicationStarted in 2003 in NEC Labs by Cezary Dubnicki2007 Product of the year award by SearchStorage.com2008 Product innovation award by Network Products Guide2009/2010 FAST conference publication in San JoseSold in US and Japan since 2007Will be sold in Poland in 2011 by 9LivesData in coop. with NEC

R&D of criticalbackend component

Owns & sells

HYDRAstor: a Scalable Secondary Storage. 9LivesData, LLC 3

Backup storage

● Tapes are most common, despite:● Sensitive environment requirements● Unreliable restore● Low performance● Manual labor or expensive robots● Problematic replication


Backup storage size

● Usual backup policy● 4-12+ full backups● 7-30+ incremental● Majority of data does

not change● Data compression 2:1

● Secondary storage size:● 5x-20x more than

primary storage● Includes many copies

of the same data● Each data chunk

stored 5-10+ times


Backup storage size

● Usual backup policy● 4-12+ full backups● 7-30+ incremental● Majority of data does

not change● Data compression 2:1

● Secondary storage size:● 5x-20x more than

primary storage● Includes many copies

of the same data● Each data chunk

stored 5-10+ times

High potential for the deduplication technology.


Deduplication

● Save disk space by eliminating duplicates● Sample reduction ratio 10:1 (depends on backup policy)

● Lowers price of gigabyte

B CA

D EA

File A

File B

B CAFile A

Sub-file level deduplication

B C DA E

Stored blocksOnly unique blocks Only unique blocks are storedare stored


Global deduplication

● Prevent silos of deduped data● One system to manage

Global vs. siloed dedup


HYDRAstor product

● Provides● global deduplication using DataRedux™● performance, storage scalability

and data resiliency using Distributed Resilient Data™

9

HYDRAstor deployment

● Interface: CIFS, NFS, Symantec OST

● Marker filtering for: Tivoli, Netbackup, Networker, CommVault


HYDRAstor architecture

● Accelerator Nodes realize performance

● Storage Nodes realize capacity

InternalNetwork

Accelerator Nodes

Storage Nodes

NFS / CIFS / OSTover Ethernet


HYDRAstor architecture

● Accelerator Nodes realize performance

● Storage Nodes realize capacity

InternalNetwork

Accelerator Nodes

Storage Nodes

NFS / CIFS / OSTover Ethernet

Non-disruptivegrid expansion


HYDRAstor scalability

● MiniHYDRA – single server● Storage: 12 TB – 240 TB*● Performance: 1.3 TB / hour

● 2AN 4SN● Storage: 48 TB – 960 TB*● Performance: 3.6 TB / hour

● 20AN 40SN (4 racks)● Storage: 480 TB – 9600 TB*● Performance: 36 TB / hour

* - assuming 20x data reduction through DataRedux™


HYDRAstor scalability

● Slide from Curtis Preston presentationCurtis Preston is a famous storage analyst owning independent consulting company


HYDRAstor other features

● Fully automatic/non-disruptive mgmt● Recovery of lost data resiliency● Periodic data scrubbing● Machine and disk failure recovery

● Configurable redundancy level● erasure coding – better than RAID6

● Optimized replication● Smart resource management


HYDRAstor backenddesign

Details of the design:http://www.usenix.org/events/fast09/tech/full_papers/dubnicki/dubnicki.pdf


Programming Model

● Repository of blocks● Content-addressed● Immutable● Variable-sized

hash=011..0


Programming Model


● Exposed pointers to other blocks

E

hash=011..0

011.

.0


Programming Model



● Trees of blocks E

EE

ERoot1

E

hash=010..1

hash=011..0

011.

.0


Programming Model



● Trees of blocks● DAGs due to deduplication● No cycles possible

E

EE

011.

.0

ERoot1

E

ERoot2hash=010..1

hash=110..0

hash=011..0

011.

.0


Programming Model




● Deletion of whole trees

E

EE

011.

.0

ERoot1

E

ERoot2hash=010..1

hash=110..0

hash=011..0

011.

.0


Programming Model





E

EE

011.

.0

ERoot1

E

ERoot2hash=010..1

hash=110..0

hash=011..0

011.

.0


Programming Model





E 011.

.0

ERoot2

hash=110..0

hash=011..0

011.

.0


Failure tolerance: erasure coding

De

cod e

Any 3 fragments can be lost

Example: N=8, m=5

Enc

ode

Original block

Ori

gina

l F

ragm

e nts

R

edun

d ant

F

ragm

e nts


Failure tolerance: erasure coding

De

cod e

Any 3 fragments can be lost

Example: N=8, m=5

Enc

ode

Original block

Ori

gina

l F

ragm

e nts

R

edun

d ant

F

ragm

e nts

Mirror 3-copy RAID6 Erasure coding

Resiliency 1 2 2 2 3

Overhead 100% 200% 20% 20% 33%

Assuming 12 disks array


Scalability with DHT: data placement

● Block location: DHT with prefix routing

0 1

01 10 11

empty prefix

00

0

01



● Block location: DHT with prefix routing● Block mapped to hash prefix hash=011..0

0 1

01 10 11

empty prefix

00

Block

0

01



● Block location: DHT with prefix routing● Block mapped to hash prefix● Prefix components

● Hosted on SNs● N components

per prefix

hash=011..0

Block

Node 1Node 6

Node 1Node 5

Node 1Node 4

Node 1Node 3

Node 1Node 2

Node 1Node 1

0 1

1

3

2

0 0

1

2

3

0

1

2

3

2

3

1

0

01 10 11

empty prefix

00

N=4



hash=011..0

Block

Node 1Node 6

Node 1Node 5

Node 1Node 4

Node 1Node 3

Node 1Node 2

Node 1Node 1

0 1

1

3

2

0 0

1

2

3

0

1

2

3

2

3

1

0

01 10 11

empty prefix

00

N=4



per prefix● Store fragments



hash=011..0

Block

Node 1Node 6

Node 1Node 5

Node 1Node 4

Node 1Node 3

Node 1Node 2

Node 1Node 1

0 1

1

3

2

0 0

1

2

3

0

1

2

3

2

3

1

0

01 10 11

empty prefix

00

N=4




● Distributedconsensus



hash=011..0

Block

Node 1Node 6

Node 1Node 5

Node 1Node 4

Node 1Node 3

Node 1Node 2

Node 1Node 1

0 1

1

3

2

0 0

1

2

3

0

1

2

3

2

3

1

0

01 10 11

empty prefix

00

N=4







hash=011..0

Block

Node 1Node 6

Node 1Node 5

Node 1Node 4

Node 1

Node 1

Node 3

Node 2

Node 1Node 1

0 1

1

3

2

0 0

1

2

3

0

1

2

3

2

3

1

0

01 10 11

empty prefix

00

N=4







hash=011..0

Block

Node 1Node 6

Node 1Node 5

Node 1Node 4

Node 1Node 3

Node 1Node 2

Node 1Node 1

0 1

1

3

2

0 0

1

2

3

0

1

2

3

2

3

1

0

01 10 11

empty prefix

00

N=4





● Load balancing


Data organization: synchrun chains

A B EC D F G● Data stream split to blocks



A B EC D F G

Hash 010…

Hash 101…

Hash 110…

Hash 011…

Hash 000…

Hash 011…

Hash 100…

● Data stream split to blocks

● Hashes of blocks computed



A B EC D F G

Hash 010…

Hash 101…

Hash 110…

Hash 011…

Hash 000…

Hash 011…

Hash 100…



● Routing through DHT



A B EC D F G

Hash 010…

Hash 101…

Hash 110…

Hash 011…

Hash 000…

Hash 011…

Hash 100…



● Routing through DHTPrefix 01



A B EC D F G

Hash 010…

Hash 101…

Hash 110…

Hash 011…

Hash 000…

Hash 011…

Hash 100…

Erasure Coding

Compression



● Routing through DHTPrefix 01



A B EC D F G

Hash 010…

Hash 101…

Hash 110…

Hash 011…

Hash 000…

Hash 011…

Hash 100…

Prefix 01

Erasure Coding

Compression




Component0

Component1

Component2

Component3

● Erasure-coded fragments stored by components



A B EC D F G

Hash 010…

Hash 101…

Hash 110…

Hash 011…

Hash 000…

Hash 011…

Hash 100…

Erasure Coding

Compression




A D F

A D F

A D F

A D F

Component0

Component1

Component2

Component3

Prefix 01




A B EC D F G

Hash 010…

Hash 101…

Hash 110…

Hash 011…

Hash 000…

Hash 011…

Hash 100…

Synchrun 1 Synchrun 2 Synchrun 3

Prefix 01

Erasure Coding

Compression

Synchrun




Component0

Component1

Component2

Component3


● Grouped into synchruns

A D F

A D F

A D F

A D F



A B EC D F G

Hash 010…

Hash 101…

Hash 110…

Hash 011…

Hash 000…

Hash 011…

Hash 100…


Prefix 01

Erasure Coding

Compression




Component0

Component1

Component2

Component3

Container



● Containers stored on disks

● Fragment metadata separately from data

Synchrun

A D F

A D F

A D F

A D F



A B EC D F G

Hash 010…

Hash 101…

Hash 110…

Hash 011…

Hash 000…

Hash 011…

Hash 100…


Erasure Coding

Compression




A D F

A D F

A D F

A D F

Component0

Component1

Component2

Component3

Prefix 01



● Containers stored on disks

● Fragment metadata separately from data

● Ordered synchrun chains

● Preserve order & locality

● ManageableContainer Synchrun


Component01:0

Component01:1

Component01:2

Component01:3

Data Services:Identification of data resiliency level

Missing fragments



Component01:0

Component01:1

Component01:2

Component01:3

Chain scanning


Data services: reconstruction

Component01:0

Component01:1

Component01:2

Component01:3

● Sequential read/write of entire Containers● Erasure decoding and re-encoding



Component01:0

Component01:1

Component01:2

Component01:3



Data services: fast data transfer

Component01:0

Component01:1

Component01:2

Component01:3

Old component

01:3

Location of newnode (DHT)



Component01:0

Component01:1

Component01:2

Component01:3

Old component

01:3

Data transfer



Component01:0

Component01:1

Component01:2

Component01:3

Old component

01:3


Data services for deduplication

Component

01:0

Component

01:1

Component

01:2

Component

01:3

hash=011..

Block

Choose completechain

Completeness: “definitely not a duplicate”Deletion interaction: wasn't the block scheduled for deletion?


Data services for deduplicationhash=011..

Block

Component

01:0

Component

01:1

Component

01:2

Component

01:3

Query



Block

Local candidate found

Component

01:0

Component

01:1

Component

01:2

Component

01:3



Block

Candidate verification

Successfuldedup

Component

01:0

Component

01:1

Component

01:2

Component

01:3


On-demand data deletion

● Distributed garbage collection● Per-block reference counter stored per-

fragment● Failure-tolerant

● Block reference counter calculated independently on peer Container chains

● Interference with duplicate elimination:● duplicates resurrection after garbage collection● space reclamation in background


Resource management

● Configurable load balancing between:● backup/restore● background tasks (reconstruction, transfer, etc.)● garbage collection

● Shares depend on system state● Assigns priority of tasks automatically

● e.g. reconstruction before transfer or space reclamation

● Maximizes resources utilization


Topics for further discussion

● Features and technical details of HYDRAstor● Sales of HYDRAstor in Poland● Cooperation with 9LivesData on other projects


Questions?

Contact:

[email protected]

mailto:[email protected]

http://www.9livesdata.com/

hydrastor: a scalable secondary storage · 2010-09-08 · hydrastor: a scalable secondary storage....

Documents