Download - Typhoon: An Ultra-Available Archive and Backup System Utilizing Linear-Time Erasure Codes
Typhoon: An Ultra-Available Archive and Backup System
Utilizing Linear-Time Erasure Codes
Typhoon: An Ultra-Available Archive and Backup System
Utilizing Linear-Time Erasure Codes
Part of OceanStore...
Client
Naming/LocationCache
• Erasure Code: a form of data coding that allows lost portions of data to be recovered
• Idea is similar to ECC, except that the algorithm must be told which portions of the data are missing
• Reed Solomon Codes are a common type of Erasure Code, but they are computationally expensive and are usually implemented in hardware
Erasure CodesErasure Codes
Tornado Codes: A Linear-Time Tornado Codes: A Linear-Time Probabilistic Family of Erasure Probabilistic Family of Erasure
CodesCodes
• Tornado Codes are linear time, but use probabilistic
assumptions to “guarantee” that the decoding process will
succeed
• A 1/2 rate Erasure Code will double the size of a file
• Any half of e. file can be used to recreate the original data
• T. Codes also require slightly more than half of the encoded
file, thus trading a network bandwidth for speed
– Inventors of T. Codes report that 5% is typical
• File is divided into nodes of equal size (e.g. 512 bytes)
• Data Nodes are associated with Check Nodes using a series of Bipartite Graphs
• Contents of a Check Node is the XOR of its neighbors
• Bipartite Graphs are created to satisfy mathematical constraints that “guarantee” the recovery process will successfully recover the file
Overview of Encoding Overview of Encoding ProcessProcess
Data Nodes
CheckNodes
Data File
Overview of Encoding Overview of Encoding ProcessProcess
Data Nodes
Data File
Check Nodes
•Once a file is encoded, the data nodes and check nodes are randomly distributed to a set of recipients
MMX: SIMD or Marketing?MMX: SIMD or Marketing?
•There are eight MMX registers•Data in registers can be divided into four different sizes:
MMX: SIMD or Marketing?MMX: SIMD or Marketing?
•There are eight MMX registers•Data in registers can be divided into four different sizes•MMX has 57 instructions for 6 types of operations:
— ADD— SUBTRACT— MULTIPLY— MULTIPLY THEN ADD— COMPARISON— LOGICAL
• AND• NAND• OR• XOR
MMX: SIMD or Marketing?MMX: SIMD or Marketing?
•There are eight MMX registers•Data in registers can be divided into four different sizes•MMX has 57 instructions for 6 types of operations
char array1[512];char array2[512];
for(int i=0; i<512; ++i)array1[i]=array1[i] ^ array2[i];
MMX is 2.3 times faster than this (1.9 w/o pipeline sched.)
MMX: SIMD or Marketing?MMX: SIMD or Marketing?
•There are eight MMX registers•Data in registers can be divided into four different sizes•MMX has 57 instructions for 6 types of operations
char array1[512];char array2[512];
long * array1ptr=(long*)array1;long * array2ptr=(long*)array2;
for(int i=0; i<512/sizeof(long); ++i)array1ptr[i]=array1ptr[i] ^ array2ptr[i];
MMX is 50% faster than this (22% w/o sched.)
MMX: SIMD or Marketing?MMX: SIMD or Marketing?
•There are eight MMX registers•Data in registers can be divided into four different sizes•MMX has 57 instructions for 6 types of operations
char array1[512];char array2[512];
long * array1ptr=(long*)array1;long * array2ptr=(long*)array2;
for(int i=0; i<512; i+=32)xor32fast(array1ptr+i, array2ptr+i);
MMX: SIMD or Marketing?MMX: SIMD or Marketing?inline void xor32bytes(long * array1reg, long* array2reg, long* destreg){
_asm{
mov eax, [array1reg]mov ecx, [array2reg]movq mm0, [eax]movq mm1, [ecx]movq mm2, [eax+8]movq mm3, [ecx+8]movq mm4, [eax+16]movq mm5, [ecx+16]movq mm6, [eax+24]movq mm7, [ecx+24]pxor mm0, mm1 ; 64-bit xorpxor mm2, mm3 ; 64-bit xorpxor mm4, mm5 ; 64-bit xorpxor mm6, mm7 ; 64-bit xormov ecx, [destreg]movq [ecx], mm0 ; store resultmovq [ecx+8], mm2 ; store resultmovq [ecx+16], mm4 ; store resultmovq [ecx+24], mm6 ; store result
}}
MMX: SIMD or Marketing?MMX: SIMD or Marketing?inline void xor32fast(long * array1reg, long* array2reg, long* destreg){
_asm{
mov eax, [array1reg]mov ebx, [array2reg]mov ecx, [destreg]movq mm0, [eax] ; load 1a Umovq mm1, [ebx] ; load 1b Umovq mm2, [eax+8] ; load 2a U Vpxor mm0, mm1 ; xor 1 movq mm3, [ebx+8] ; load 2b Umovq [ecx], mm0 ; store 1 U Vpxor mm2, mm3 ; xor 2 movq mm4, [eax+16] ; load 3a Umovq mm5, [ebx+16] ; load 3b Umovq mm6, [eax+24] ; load 4a U Vpxor mm4, mm5 ; xor 3movq mm7, [ebx+24] ; load 4b Umovq [ecx+8], mm2 ; store 2 U Vpxor mm6, mm7 ; xor 4movq [ecx+16], mm4 ; store 3 Umovq [ecx+24], mm6 ; store 4 U
}}
Overview of Encoding ProcessOverview of Encoding Process
•Server sends storage announcement to a particular set of severs
– Set can be determined/specified using multicast groups, a server list, or some form of DNS address lookup
UDPUDP
Overview of Encoding ProcessOverview of Encoding Process
•Server sends storage announcement to a particular set of severs
– Set can be determined/specified using multicast groups, a server list, or some form of DNS address lookup
MulticastMulticast
Overview of Encoding ProcessOverview of Encoding Process
•Server encodes file•During encoding process, the data nodes and check nodes are [randomly] distributed to other servers
• A set of nodes are received, ideally with random distribution
• Check nodes can be used to recover missing data nodes
• Only check nodes that are missing one neighbor can recreate a data node
• The structure of the graph ensures [w.h.p.] that the encoding process will succeed
– Graph is designed so that there is always at least one check node that is missing only one child
– Data nodes can be used to recover check nodes, but is not important
Overview of Decoding ProcessOverview of Decoding Process
CheckNodes
Data File
Node ReceivedNode Not Received
• A set of nodes are received, ideally with random distribution
• Check nodes can be used to recover missing data nodes
• Only check nodes that are missing one neighbor can recreate a data node
• The structure of the graph ensures [w.h.p.] that the encoding process will succeed
– Graph is designed so that there is always at least one check node that is missing only one child
– Data nodes can be used to recover check nodes, but is not important
Overview of Decoding ProcessOverview of Decoding Process
CheckNodes
Data File
Node ReceivedNode Not Received
• A set of nodes are received, ideally with random distribution
• Check nodes can be used to recover missing data nodes
• Only check nodes that are missing one neighbor can recreate a data node
• The structure of the graph ensures [w.h.p.] that the encoding process will succeed
– Graph is designed so that there is always at least one check node that is missing only one child
– Data nodes can be used to recover check nodes, but is not important
Overview of Decoding ProcessOverview of Decoding Process
CheckNodes
Data File
Node ReceivedNode Not Received
• A set of nodes are received, ideally with random distribution
• Check nodes can be used to recover missing data nodes
• Only check nodes that are missing one neighbor can recreate a data node
• The structure of the graph ensures [w.h.p.] that the encoding process will succeed
– Graph is designed so that there is always at least one check node that is missing only one child
– Data nodes can be used to recover check nodes, but is not important
Overview of Decoding ProcessOverview of Decoding Process
CheckNodes
Data File
Node ReceivedNode Not Received
• A set of nodes are received, ideally with random distribution
• Check nodes can be used to recover missing data nodes
• Only check nodes that are missing one neighbor can recreate a data node
• The structure of the graph ensures [w.h.p.] that the encoding process will succeed
– Graph is designed so that there is always at least one check node that is missing only one child
– Data nodes can be used to recover check nodes, but is not important
Overview of Decoding ProcessOverview of Decoding Process
CheckNodes
Data File
Node ReceivedNode Not Received
• A set of nodes are received, ideally with random distribution
• Check nodes can be used to recover missing data nodes
• Only check nodes that are missing one neighbor can recreate a data node
• The structure of the graph ensures [w.h.p.] that the encoding process will succeed
– Graph is designed so that there is always at least one check node that is missing only one child
– Data nodes can be used to recover check nodes, but is not important
Overview of Decoding ProcessOverview of Decoding Process
CheckNodes
Data File
Node ReceivedNode Not Received
• A set of nodes are received, ideally with random distribution
• Check nodes can be used to recover missing data nodes
• Only check nodes that are missing one neighbor can recreate a data node
• The structure of the graph ensures [w.h.p.] that the encoding process will succeed
– Graph is designed so that there is always at least one check node that is missing only one child
– Data nodes can be used to recover check nodes, but is not important
Overview of Decoding ProcessOverview of Decoding Process
CheckNodes
Data File
Node ReceivedNode Not Received
• A set of nodes are received, ideally with random distribution
• Check nodes can be used to recover missing data nodes
• Only check nodes that are missing one neighbor can recreate a data node
• The structure of the graph ensures [w.h.p.] that the encoding process will succeed
– Graph is designed so that there is always at least one check node that is missing only one child
– Data nodes can be used to recover check nodes, but is not important
Overview of Decoding ProcessOverview of Decoding Process
CheckNodes
Data File
Node ReceivedNode Not Received
• A set of nodes are received, ideally with random distribution
• Check nodes can be used to recover missing data nodes
• Only check nodes that are missing one neighbor can recreate a data node
• The structure of the graph ensures [w.h.p.] that the encoding process will succeed
– Graph is designed so that there is always at least one check node that is missing only one child
– Data nodes can be used to recover check nodes, but is not important
Overview of Decoding ProcessOverview of Decoding Process
CheckNodes
Data File
Node ReceivedNode Not Received
• A set of nodes are received, ideally with random distribution
• Check nodes can be used to recover missing data nodes
• Only check nodes that are missing one neighbor can recreate a data node
• The structure of the graph ensures [w.h.p.] that the encoding process will succeed
– Graph is designed so that there is always at least one check node that is missing only one child
– Data nodes can be used to recover check nodes, but is not important
Overview of Decoding ProcessOverview of Decoding Process
CheckNodes
Data File
Node ReceivedNode Not Received
• A set of nodes are received, ideally with random distribution
• Check nodes can be used to recover missing data nodes
• Only check nodes that are missing one neighbor can recreate a data node
• The structure of the graph ensures [w.h.p.] that the encoding process will succeed
– Graph is designed so that there is always at least one check node that is missing only one child
– Data nodes can be used to recover check nodes, but is not important
Overview of Decoding ProcessOverview of Decoding Process
CheckNodes
Data File
Node ReceivedNode Not Received
• A set of nodes are received, ideally with random distribution
• Check nodes can be used to recover missing data nodes
• Only check nodes that are missing one neighbor can recreate a data node
• The structure of the graph ensures [w.h.p.] that the encoding process will succeed
– Graph is designed so that there is always at least one check node that is missing only one child
– Data nodes can be used to recover check nodes, but is not important
Overview of Decoding ProcessOverview of Decoding Process
CheckNodes
Data File
Node ReceivedNode Not Received
• A set of nodes are received, ideally with random distribution
• Check nodes can be used to recover missing data nodes
• Only check nodes that are missing one neighbor can recreate a data node
• The structure of the graph ensures [w.h.p.] that the encoding process will succeed
– Graph is designed so that there is always at least one check node that is missing only one child
– Data nodes can be used to recover check nodes, but is not important
Overview of Decoding ProcessOverview of Decoding Process
CheckNodes
Data File
Node ReceivedNode Not Received
• A set of nodes are received, ideally with random distribution
• Check nodes can be used to recover missing data nodes
• Only check nodes that are missing one neighbor can recreate a data node
• The structure of the graph ensures [w.h.p.] that the encoding process will succeed
– Graph is designed so that there is always at least one check node that is missing only one child
– Data nodes can be used to recover check nodes, but is not important
Overview of Decoding ProcessOverview of Decoding Process
CheckNodes
Data File
Node ReceivedNode Not Received
Overview of Decoding ProcessOverview of Decoding Process• Server sends file request announcement to a particular set of servers• Retrieves data from multiple servers simultaneously• Recovery process can be performed in parallel with receive (network-based RAID-1)• Depending on data loss pattern, a particular subset of the servers can be selected
• Fastest servers (closest servers, or least utilized servers)• Operational Servers (i.e., some portion of the set is not functioning)
• All servers might be needed in some cases, such as network congestion / packet loss
Client
Naming/LocationCache
ArchitectureArchitecture
ArchitectureArchitecture•What did we implement?
• Client, Cache, Naming and Location Mechanism, Replication mechanism, filestore.
•What did we test?• Communication
•Explicit communication Unicast request•Implicit communication Multicast request
• Network•Distributed servers throughout Berkeley domain.•Simulated network delay by randomizing response time.
• Caching•None for worst case
• Simulation•Strained the Typhoon system by creating requests at the same rate as a 24 hour NFS traces over a 3 hour period.
Tornado - GET avg_proc_time
0
5
10
15
20
25
0 500000 1000000 1500000
File Size (bytes)
Tim
e (
se
c)
client
cache
namingloc
replication
f ilestore
Reed Solom on - GET avg_proc_tim e
0
200
400
600
800
1000
1200
1400
1600
1800
2000
0 1000000 2000000 3000000
File Size (bytes)
Tim
e (s
ec)
Client
Cache
Namingloc
Replication
Filestore
Tornado - Replication
0
5
10
15
20
25
0 500000 1000000 1500000
File Size (bytes)
Tim
e (
se
c)
avg_proc_time
avg_dec_time
avg_comm_time
Reed Solomon - Replication
0
200
400
600
800
1000
1200
1400
1600
1800
2000
0 1000000 2000000 3000000
File Size (bytes)
Tim
e (
se
c)
avg_proc_time
avg_dec_time
avg_comm_time
Reed Solomon - PUT avg_proc_time
0
500
1000
1500
2000
2500
3000
3500
4000
4500
0 1000000 2000000 3000000 4000000
File size (bytes)
Tim
e (s
ec)
Client
Cache
Namingloc
Replication
Filestore
Tornado - PUT avg_proc_time
0
1
2
3
4
5
6
0 1000000 2000000 3000000 4000000
File Size (bytes)
Tim
e (s
ec)
client
cache
namingloc
replication
filestore
Reed Solomon - Replication
0
10
20
30
40
50
60
70
0 200000 400000 600000
File size (bytes)
Tim
e (
se
c)
avg_proc_time
avg_enc_time
avg_comm_time
Tornado - Replication
0
1
2
3
4
5
6
0 1000000 2000000 3000000 4000000
File Size (bytes)
Tim
e (
se
c) avg_proc_time
avg_enc_time
avg_comm_time
Typhoon: An Ultra-Available Archive and Backup System
Utilizing Linear-Time Erasure Codes
Benefits of TyphoonBenefits of Typhoon• Data is ultra-available: up to half of the servers can fail before availability is affected
• Fast file retrieval: data can be retrieved simultaneously from multiple servers– System can choose to use the fastest machines in a set of servers
– Load balancing can be achieved because slow or heavily utilized servers are not used
– Information can be disbursed geographically • Increases the accessibility of data in the event of a major disaster, such as an earthquake
• Can benefit people who travel to remote locations, since data may be closer to them
– Multicast can be used to reduce latency
• Low-overhead algorithms: algorithms for encoding and decoding are linear-time
• Disk overhead of system can be adjusted (typically doubles the size of a file)
ConclusionConclusion
• Tornado Codes are significantly faster than Cauchy-Reed Solomon
• A Typhoon based system can match the the request of a loaded NFS
• Typhoon is a viable solution for increasing the reliability and accessibility of data
ArchitectureArchitecture•What did we implement?
• Client, Cache, Naming and Location Mechanism, Replication Mechanism, filestore.
•What did we test?• Communication
•Explicit communication TCP request, TCP Response.•Implicit communication Multicast request, TCP Response.
• Network•Distributed servers throughout Berkeley domain.•Simulated network delay by randomizing response time.
• Caching•None for worst case
• Simulation•Strained the Typhoon system by creating requests at the same rate as a 24 hour NFS traces over a 3 hour period.