The final presentation of a semester project. Course: Implementation of Distributed Systems (KTH Royal Institute of Technology)

1. Scaling Online Social
Networks (OSNs)
Presented by: Maria Stylianou Coworker: Anis Uddin
Supervisor: Šarūnas Girdzijauskas
KTH - Royal Institute of Technology
Implementation of Distributed Systems
December 6th, 2012

2. Outline
● Motivation
● Current Algorithms
– SPAR
– JA-BE-JA
● Contributions
– Challenges
– Solution
● Evaluation & Conclusions
2

3. Outline
● Motivation
● Current Algorithms
– SPAR
– JA-BE-JA
● Contributions
– Challenges
– Solution
● Evaluation & Conclusions
3

4. “Pandora's box”
Online Social Networks
Source: http://technorati.com/social-media/article/social-networks-theyre-what-every-local/
Motivation-Algorithms-Contribution-Evaluation 4

5. Easy to maintain...
Online Social Networks
Source: http://mastersofmedia.hum.uva.nl/2009/09/14/a-review-of-taken-out-of-context/
Motivation-Algorithms-Contribution-Evaluation 5

6. ...or not!
Online Social Networks
Source: http://mastersofmedia.hum.uva.nl/2009/09/14/a-review-of-taken-out-of-context/
Motivation-Algorithms-Contribution-Evaluation 6

7. Scaling Approaches
Vertical Scaling Horizontal Scaling
● Full Replication ● Adding servers
● Data Locality ● Clean & Disjoint
Partitions
● But:
● But:
– Expensive
– Saturation
– Not applicable in
OSNs
Motivation-Algorithms-Contribution-Evaluation 7

8. Scaling Approaches
Vertical Scaling Horizontal Scaling
● Full Replication ● Adding servers
● Data Locality ● Clean & Disjoint
Partitions
● But:
● But:
– Expensive
– Saturation
– Not applicable in
OSNs
Inefficient
Motivation-Algorithms-Contribution-Evaluation 8

9. Existing 'Solutions' for OSNs
Relational Databases Key-Value Stores
Motivation-Algorithms-Contribution-Evaluation 9

10. Existing 'Solutions' for OSNs
Relational Databases Key-Value Stores
Inefficient
Motivation-Algorithms-Contribution-Evaluation 10

11. Outline
● Motivation
● Current Algorithms
– SPAR
– JA-BE-JA
● Contributions
– Challenges
– Solutions
● Evaluation & Conclusions
11

12. SPAR
Social Partitioning & Replication middle-ware
● Transparent OSN scalability avoids
● Data Locality performance
● Load Balancing bottlenecks
● Fault Tolerance
● Stability
● Replication Overhead Minimization
Motivation-Algorithms-Contribution-Evaluation 12

13. SPAR
Events
● Nodes – Add/Remove
● Edges – Add/Remove
● Servers – Add/Remove
Motivation-Algorithms-Contribution-Evaluation 13

14. SPAR Algorithm
M2
6'
5
1' Create Edge (1,6)
2 5'
3 1
4
5'
M1 6 Master Node
Replica Node
M3
Motivation-Algorithms-Contribution-Evaluation 14

15. SPAR Algorithm
M2
6'
5 Create Edge (1,6)
2 5' 1' C1: Create 6' in M1
Create 1' in M3
3 1
4 6'
5'
M1 6 Master Node
1'
Replica Node
M3
Motivation-Algorithms-Contribution-Evaluation 15

16. SPAR Algorithm
M2
6'
5
Create Edge (1,6)
2 1'
C2: Move 1 to M3
3 1'
4
5'
2' Master Node
M1
6 1 3'
Replica Node
M3 4'
Motivation-Algorithms-Contribution-Evaluation 16

17. SPAR Algorithm
M2
6'
5
Create Edge (1,6)
2 5' 1'
C3: Move 6 to M1
3 1
4 6
M1 Master Node
Replica Node
M3
Motivation-Algorithms-Contribution-Evaluation 17

18. JA-BE-JA
● Distributed Partitioning Algorithm
● K-way Partitioning
● Load Balancing
● Gossip Learning
Motivation-Algorithms-Contribution-Evaluation 18

19. JA-BE-JA - Policies
● Sampling ● Swapping
– Local – Energy Function
● Select neighbors ● Reach minimum
– Random – Simulated Annealing
● Select from random ● Escape from local
walk optima
– Hybrid
● Local & Random
Source: http://socialnetworking.lovetoknow.com/Growth_of_Online_Social_Networking_in_Business
Motivation-Algorithms-Contribution-Evaluation 19

20. Outline
● Motivation
● Current Algorithms
– SPAR
– JA-BE-JA
● Contributions
– Challenges
– Solution
● Evaluation & Conclusions
20

21. Challenges
Global View
Partition Manager requirement
→ Single Point
of Failure
SPAR
SPAR
Replication
Overhead
Motivation-Algorithms-Contribution-Evaluation 21

22. Our Solution
Global View
Partition Manager requirement
→ Single Point
of Failure SPAR Local View
Distributed
&
Partition JA-BE-JA
Manager
Replication
Overhead
Motivation-Algorithms-Contribution-Evaluation 22

23. Our Solution
(wait for it...)
Client Requests
SPAR
Data Store
Servers
Motivation-Algorithms-Contribution-Evaluation 23

24. Our Solution
Client Requests
SPAR
&
JA-BE-JA
JA
BE
JA
Data Store
Servers
Motivation-Algorithms-Contribution-Evaluation 24

25. Outline
● Motivation
● Current Algorithms
– SPAR
– JA-BE-JA
● Contributions
– Challenges
– Solution
● Evaluation & Conclusions
25

26. Implementation
● SPAR
● SPAR-JA
This is SPARJA!
Motivation-Algorithms-Contribution-Evaluation 26

27. Datasets
● Facebook Graphs
by Stanford Network Analysis Project
– #nodes: 150 #edges: ~3000
– #nodes: 224 #edges: ~6000
– #nodes: 786 #edges: ~60000
Source: http://snap.stanford.edu/
Motivation-Algorithms-Contribution-Evaluation 27

28. Datasets
● Synthesized Graphs
– using our own Graph Generator
Graph Visualization Tool
– #nodes: 1000, #degree: 10 https://gephi.org/
Randomized Clustered Highly Clustered
Motivation-Algorithms-Contribution-Evaluation 28

29. Experiments
Replication Overhead on Different Datasets
#k-replicas: 0 (fault tolerance) #Servers: 4
Synthesized Graphs
10000 edges
synth-r: Randomized
synth-c: Clustered
synth-hc:
Highly Clustered
Facebook Graphs
fcbk-1: ~3000 edges
fcbk-2: ~6000 edges
fcbk-3: ~60000 edges
Motivation-Algorithms-Contribution-Evaluation 29

30. Experiments
Replication Overhead vs Replication Factor
K=0
K=2
Motivation-Algorithms-Contribution-Evaluation 30

31. Experiments
Replication Overhead on both algorithms
Fault Tolerance
K=2
synth-hc:
- Highly Clustered
- Synthesized Graph
- 10000 edges
Motivation-Algorithms-Contribution-Evaluation 31

32. Experiments
Replication Overhead on both algorithms
Fault Tolerance
K=2
fcbk-3:
- 3rd facebook graph
- 60,000 edges
Motivation-Algorithms-Contribution-Evaluation 32

33. Conclusions
● SPAR + JA-BE-JA = SPAR-JA
– Highly clustered nodes
– Achieves fault tolerance 'by-default'
– Better than SPAR in case of high clusterization
● Future Work
– More datasets
– Bigger datasets
Motivation-Algorithms-Contribution-Evaluation 33

34. Scaling Online Social
Networks (OSNs)
Presented by: Maria Stylianou Coworker: Anis Uddin
Supervisor: Šarūnas Girdzijauskas
KTH - Royal Institute of Technology
Implementation of Distributed Systems
December 6th, 2012