distributed storage allocations and a hypergraph conjecture of erd ő s

Distributed Storage Allocationsand a Hypergraph Conjecture of Erdős

Yi-Hsuan Kao1, Alexandros G. Dimakis2,Derek Leong3 4, and Tracey Ho3

1University of Southern California, Los Angeles, California, USA2The University of Texas at Austin, Austin, Texas, USA

3California Institute of Technology, Pasadena, California, USA4Institute for Infocomm Research, Singapore

ISIT 20132013-07-09

Distributed Storage Allocations and a Hypergraph Conjecture of Erdős Slide 2 of 20

Suppose you have a distributed storage systemcomprising 5 storage devices (“nodes”)…

Distributed Storage Allocations: An Example

1 2 3 4 5


2 4

Each node independently fails with probability 1/3, and survives with probability 2/3…

1 2 3 4 5

(1/3)2 (2/3)3 ≈ 0.0329218



2 41 3 5

Each node independently fails with probability 1/3, and survives with probability 2/3…


(1/3)5 ≈ 0.00411523

1 3 52 4


You are given a single data object of (normalized) unit size, and a total storage budget of 7/3…


1 2 3 4 5


You can use any coding schemeto store any amount of coded data in each node,

as long as the total amount of storage usedis at most the given budget 7/3…


1 2 3 4 5



1 2 3 4 5

010010101010010101000101010101000101010111010101001001010001010100

01101010001010101110101010010010100010101001

1010010101000101001110

1010010101000101001110


1 2 3 4 5

010010101010010101000101010101000101010111010101001001010001010100

01101010001010101110101010010010100010101001

1010010101000101001110

1010010101000101001110

?

(1/3)2 (2/3)3 ≈ 0.0329218



For maximum reliability, we need to find

(1) an optimal allocation of the given budget over the nodes, and

(2) an optimal coding scheme

that jointly maximize the probability of successful recovery



Using an appropriate code, successful recovery can occur wheneverthe data collector accesses at least a unit amount of data(= size of the original data object)


1 2 3 4 5

t1 t2

s

A. G. Dimakis et al., “Network coding for distributed storage systems,” Trans. Inf. Theory, Sep 2010.A. Jiang, “Network coding for joint storage and transmission with minimum cost,” in Proc. ISIT, Jul 2006.



1 2 3 4 5


1 2 3 4 5

A 7/15 7/15 7/15 7/15 7/15

B 7/6 7/6 0 0 0

C 2/3 2/3 1/3 1/3 1/3

0.79012

0.88889

0.90535C

RecoveryProbability

n = 5 nodes, access probability p = 2/3, budget T = 7/3



Two access models for the data collector Independently probabilistic access to each node Access to a random fixed-size subset of nodes

Goal: Allocate a given budget T for maximum reliability,i.e., maximize the probability of successful recovery

Combinatorial problem

Distributed Storage Allocations

data (unit size) coded data (size T)

n storage nodes

data collector

D. Leong, A. G. Dimakis, and T. Ho, “Distributed storage allocations,” IEEE Trans. Inf. Theory, Jul. 2012.


Our Conjecture:If budget T is an integer, then the optimal allocation is either

maximal spreading(1/r, ..., 1/r, 0, .., 0)

or

minimal spreading(1, ..., 1, 0, ...,0)

Our conjecture turns out to be equivalent to the Fractional Erdős Conjecture (e.g., see N. Alon, P. Frankl, H. Huang, V. Rödl, A. Rucinski, and B. Sudakov, “Large matchings in uniform hypergraphs and the conjectures of Erdős and Samuels,” Journal of Combinatorial Theory Series A, vol. 119, pp. 1200–1215, 2012.)

Fixed-Size Subset Problem: A Conjecture


The Erdős conjecture is a graph theoretic claim involving hypergraphs and their matching numbers

More intuitive to explain in terms of the dual problem:Minimum Vertex Cover Problem: Given a hypergraph H = (V,E) Assign 0 or 1 to each vertex in V such that each hyperedge in E has a

sum of at least 1 (each hyperedge is “covered” by one or more vertices)

Goal is to minimize the sum over all vertices Vertices = nodes (assigned value is xi) Hyperedges = successful r-subsets of nodes

In the fractional version of the problem, we are allowed to assign a fractional value between 0 and 1 to each vertex

By strong duality, the fractional minimum vertex cover problem and the fractional maximum matching problem have the same optimal solution

Hypergraphs and the Minimum Vertex Cover Problem


Erdős conjectured that given a constraint on the matching number, the maximum number of hyperedges in an r-uniform hypergraph can be one of only two possible values:

Integral Erdős Conjecture (1965)

maximal spreading (1/r, ..., 1/r,0,..,0)

minimal spreading (1,...,1,0,..,0)


Fractional Erdős Conjecture

max spreading min spreading

maximal spreading (1/r, ..., 1/r,0,..,0)

minimal spreading (1,...,1,0,..,0)


If the Integral Erdős Conjecture is true, then we obtain strong bounds on the probability of successful recovery for the fixed-size subset problem

Also, we found new conditions under which the Integral Erdős Conjecture implies the Fractional Erdős Conjecture.

Further Results Involving the Conjectures


We found stronger bounds for the fixed-size subset problem that are independent of the conjectures

These bounds can also be used to improve our earlier bounds for the independent probabilistic access variation of the storage allocation problem

Theorem: For any feasible allocation (x1, ..., xn), i.e., such that

x1 + ... + xn ≤ T and xi ≥ 0 for all i,

the number of successful r-subsets S has the following upper bound:

Proof uses permutation counting arguments similar to Katona’s proof of the Erdős-Ko-Rado theorem

Upper Bounds for the Fixed-Size Subset Problem


Thank You!

distributed storage allocations and a hypergraph conjecture of erd ő s

Documents