scaling online social networks: extended spar using gossip learning

MotivationAlgorithms

Our ContributionEvaluationConclusion

Scaling Online Social Networks: extended SPARusing Gossip Learning

Presented by: Muhammad Anis uddin NasirCoworker: Maria Stylianou

Supervised by: Sarunas Girdzijauskas

KTH Royal Institute of Technology

December 5, 2012

Muhammad Anis uddin Nasir Scaling Online Social Networks 1/24

1 Motivation

2 AlgorithmsSPARJA-BE-JA

3 Our ContributionChallengesProposed Algorithm

4 EvaluationDatasetsImplementationResults

5 Conclusion

Online Social Networks

Scalability

Hardware Scalability

Application Scalability

Scalability

Hardware Scalability

Application Scalability

Scaling

Vertical Scaling

Full ReplicationData LocalityHigh Cost

Horizontal Scaling

ShardingDisjoint DataPartitioning OSNs

Scaling

Vertical ScalingFull Replication

Data LocalityHigh Cost

Horizontal Scaling

Scaling

Vertical ScalingFull ReplicationData Locality

High Cost

Horizontal Scaling

Scaling

Vertical ScalingFull ReplicationData LocalityHigh Cost

Horizontal Scaling

Scaling

Horizontal Scaling

Scaling

Horizontal ScalingSharding

Disjoint DataPartitioning OSNs

Scaling

Horizontal ScalingShardingDisjoint Data

Partitioning OSNs

Scaling

Horizontal ScalingShardingDisjoint DataPartitioning OSNs

SPARJA-BE-JA

1 Motivation

5 Conclusion

SPARJA-BE-JA

Features

Local Semantics

Load Balancing

Fault Tolerant

Dynamic

Low Replication Overhead

SPARJA-BE-JA

Features

Local Semantics

Load Balancing

Fault Tolerant

Dynamic

SPARJA-BE-JA

Features

Local Semantics

Load Balancing

Fault Tolerant

Dynamic

SPARJA-BE-JA

Features

Local Semantics

Load Balancing

Fault Tolerant

Dynamic

SPARJA-BE-JA

Features

Local Semantics

Load Balancing

Fault Tolerant

Dynamic

SPARJA-BE-JA

Architecture

Partition Manager

Directory Service

Local DirectoryService

ReplicationManager

SPARJA-BE-JA

Architecture

Partition Manager

Directory Service

ReplicationManager

SPARJA-BE-JA

Architecture

Partition Manager

Directory Service

ReplicationManager

SPARJA-BE-JA

Architecture

Partition Manager

Directory Service

ReplicationManager

SPARJA-BE-JA

SPAR Algorithm

Heuristic

Greedy OptimizationLocal Search

Node Add/Remove

Edge Add/Remove

3 Configurations

Server Add/Remove

RedistributionLet it fill

SPARJA-BE-JA

SPAR Algorithm

HeuristicGreedy Optimization

Local Search

Node Add/Remove

Edge Add/Remove

3 Configurations

Server Add/Remove

SPARJA-BE-JA

SPAR Algorithm

HeuristicGreedy OptimizationLocal Search

Node Add/Remove

Edge Add/Remove

3 Configurations

Server Add/Remove

SPARJA-BE-JA

SPAR Algorithm

Node Add/Remove

Edge Add/Remove

3 Configurations

Server Add/Remove

SPARJA-BE-JA

SPAR Algorithm

Node Add/Remove

Edge Add/Remove

3 Configurations

Server Add/Remove

SPARJA-BE-JA

SPAR Algorithm

Node Add/Remove

Edge Add/Remove3 Configurations

Server Add/Remove

SPARJA-BE-JA

SPAR Algorithm

Node Add/Remove

Server Add/Remove

SPARJA-BE-JA

SPAR Algorithm

Node Add/Remove

Server Add/RemoveRedistribution

Let it fill

SPARJA-BE-JA

SPAR Algorithm

Node Add/Remove

Server Add/RemoveRedistributionLet it fill

SPARJA-BE-JA

SPAR Algorithm

Node Add/Remove

SPARJA-BE-JA

SPAR Algorithm

Node Add/Remove

SPARJA-BE-JA

SPAR Algorithm

Node Add/Remove

SPARJA-BE-JA

SPAR Algorithm

Node Add/Remove

SPARJA-BE-JA

Features

Distributed Partitioning

k-way Partitioning

Load Balancing

Low Inter-communicationOverhead

Local Search

SPARJA-BE-JA

Features

k-way Partitioning

Load Balancing

Local Search

SPARJA-BE-JA

Features

k-way Partitioning

Load Balancing

Local Search

SPARJA-BE-JA

Features

k-way Partitioning

Load Balancing

Local Search

SPARJA-BE-JA

Features

k-way Partitioning

Load Balancing

Local Search

SPARJA-BE-JA

Overview

Sampling Policies

LocalRandomHybrid

Swapping Policies

Energy FunctionSimulated Annealing

Algorithm

Hybrid SamplingSimulated Annealing

SPARJA-BE-JA

Overview

Sampling PoliciesLocal

RandomHybrid

Swapping Policies

Algorithm

SPARJA-BE-JA

Overview

Sampling PoliciesLocalRandom

Hybrid

Swapping Policies

Algorithm

SPARJA-BE-JA

Overview

Sampling PoliciesLocalRandomHybrid

Swapping Policies

Algorithm

SPARJA-BE-JA

Overview

Swapping Policies

Algorithm

SPARJA-BE-JA

Overview

Swapping PoliciesEnergy Function

Simulated Annealing

Algorithm

SPARJA-BE-JA

Overview

Swapping PoliciesEnergy FunctionSimulated Annealing

Algorithm

SPARJA-BE-JA

Overview

Algorithm

SPARJA-BE-JA

Overview

AlgorithmHybrid Sampling

Simulated Annealing

SPARJA-BE-JA

Overview

AlgorithmHybrid SamplingSimulated Annealing

ChallengesProposed Algorithm

1 Motivation

5 Conclusion

Challenges

Global View

Partition Manager

Replication Overhead

Load Balancing

Challenges

Global View

Partition Manager

Load Balancing

Challenges

Global View

Partition Manager

Load Balancing

Challenges

Global View

Partition Manager

Load Balancing

Proposed Algorithm

SPAR + JA-BE-JA

Gossip Learning

SimulatedAnnealing

Optimal Replication

Proposed Algorithm

SPAR + JA-BE-JA

Gossip Learning

SimulatedAnnealing

Optimal Replication

Proposed Algorithm

SPAR + JA-BE-JA

Gossip Learning

SimulatedAnnealing

Optimal Replication

Proposed Algorithm

SPAR + JA-BE-JA

Gossip Learning

SimulatedAnnealing

Optimal Replication

DatasetsImplementationResults

1 Motivation

5 Conclusion

Datasets

Synthetic Graphs

Facebook Graphs

0http://snap.stanford.edu/data/Muhammad Anis uddin Nasir Scaling Online Social Networks 16/24

Datasets

Synthetic Graphs

Facebook Graphs

0http://snap.stanford.edu/data/Muhammad Anis uddin Nasir Scaling Online Social Networks 16/24

Datasets

Synthetic GraphsRandomized

ClusteredHighly Clustered

Facebook Graphs

150 nodes, 3386 edges224 nodes, 6384 edges786 nodes, 60050 edges

0https://gephi.org/

Datasets

Synthetic GraphsRandomizedClustered

Highly Clustered

Facebook Graphs

0https://gephi.org/

Datasets

Synthetic GraphsRandomizedClusteredHighly Clustered

Facebook Graphs

0https://gephi.org/

Datasets

Facebook Graphs150 nodes, 3386 edges

224 nodes, 6384 edges786 nodes, 60050 edges

0https://gephi.org/

Datasets

Facebook Graphs150 nodes, 3386 edges224 nodes, 6384 edges

786 nodes, 60050 edges

0https://gephi.org/

Datasets

Facebook Graphs150 nodes, 3386 edges224 nodes, 6384 edges786 nodes, 60050 edges

0https://gephi.org/

Implementation

Proposed Algorithm

MetricReplication Overhead

Evaluation of Replication Overhead

Replication FactorFault tolerance replicas reduce replication overheadProposed Algorithm performs better than SPAR

0replication overhead = number of replicas/number of users

Number of ServersLess Replication overhead in the case of proposed algorithmProposed Algorithm performs better in the case of highclusterization

Number of ServersLess Replication overhead in the case of proposed algorithmProposed Algorithm performs better in case of highclusterization

1 Motivation

5 Conclusion

Conclusion

Distributed social-based partitioning

Local Semantics

Reduced Replication overhead compared to SPAR

Better load balancing using k-way partitioning

Transparent Scaling

Conclusion

Local Semantics

Transparent Scaling

Conclusion

Local Semantics

Transparent Scaling

Conclusion

Local Semantics

Transparent Scaling

Conclusion

Local Semantics

Transparent Scaling

Scaling Online Social Networks: extended SPARusing Gossip Learning

Presented by: Muhammad Anis uddin NasirCoworker: Maria Stylianou

Supervised by: Sarunas Girdzijauskas

KTH Royal Institute of Technology

December 5, 2012