1. Motivation
Algorithms
Our Contribution
Evaluation
Conclusion
Scaling Online Social Networks: extended SPAR
using Gossip Learning
Presented by: Muhammad Anis uddin Nasir
Coworker: Maria Stylianou
Supervised by: Sarunas Girdzijauskas
KTH Royal Institute of Technology
December 5, 2012
Muhammad Anis uddin Nasir Scaling Online Social Networks 1/24

2. Motivation
Algorithms
Our Contribution
Evaluation
Conclusion
1 Motivation
2 Algorithms
SPAR
JA-BE-JA
3 Our Contribution
Challenges
Proposed Algorithm
4 Evaluation
Datasets
Implementation
Results
5 Conclusion
Muhammad Anis uddin Nasir Scaling Online Social Networks 2/24

3. Motivation
Algorithms
Our Contribution
Evaluation
Conclusion
Online Social Networks
Muhammad Anis uddin Nasir Scaling Online Social Networks 3/24

4. Motivation
Algorithms
Our Contribution
Evaluation
Conclusion
Scalability
Hardware Scalability
Muhammad Anis uddin Nasir Scaling Online Social Networks 4/24

5. Motivation
Algorithms
Our Contribution
Evaluation
Conclusion
Scalability
Hardware Scalability
Application Scalability
Muhammad Anis uddin Nasir Scaling Online Social Networks 4/24

6. Motivation
Algorithms
Our Contribution
Evaluation
Conclusion
Scaling
Vertical Scaling
Muhammad Anis uddin Nasir Scaling Online Social Networks 5/24

7. Motivation
Algorithms
Our Contribution
Evaluation
Conclusion
Scaling
Vertical Scaling
Full Replication
Muhammad Anis uddin Nasir Scaling Online Social Networks 5/24

8. Motivation
Algorithms
Our Contribution
Evaluation
Conclusion
Scaling
Vertical Scaling
Full Replication
Data Locality
Muhammad Anis uddin Nasir Scaling Online Social Networks 5/24

9. Motivation
Algorithms
Our Contribution
Evaluation
Conclusion
Scaling
Vertical Scaling
Full Replication
Data Locality
High Cost
Muhammad Anis uddin Nasir Scaling Online Social Networks 5/24

10. Motivation
Algorithms
Our Contribution
Evaluation
Conclusion
Scaling
Vertical Scaling
Full Replication
Data Locality
High Cost
Horizontal Scaling
Muhammad Anis uddin Nasir Scaling Online Social Networks 5/24

11. Motivation
Algorithms
Our Contribution
Evaluation
Conclusion
Scaling
Vertical Scaling
Full Replication
Data Locality
High Cost
Horizontal Scaling
Sharding
Muhammad Anis uddin Nasir Scaling Online Social Networks 5/24

12. Motivation
Algorithms
Our Contribution
Evaluation
Conclusion
Scaling
Vertical Scaling
Full Replication
Data Locality
High Cost
Horizontal Scaling
Sharding
Disjoint Data
Muhammad Anis uddin Nasir Scaling Online Social Networks 5/24

13. Motivation
Algorithms
Our Contribution
Evaluation
Conclusion
Scaling
Vertical Scaling
Full Replication
Data Locality
High Cost
Horizontal Scaling
Sharding
Disjoint Data
Partitioning OSNs
Muhammad Anis uddin Nasir Scaling Online Social Networks 5/24

14. Motivation
Algorithms
SPAR
Our Contribution
JA-BE-JA
Evaluation
Conclusion
1 Motivation
2 Algorithms
SPAR
JA-BE-JA
3 Our Contribution
Challenges
Proposed Algorithm
4 Evaluation
Datasets
Implementation
Results
5 Conclusion
Muhammad Anis uddin Nasir Scaling Online Social Networks 6/24

15. Motivation
Algorithms
SPAR
Our Contribution
JA-BE-JA
Evaluation
Conclusion
Features
Local Semantics
Muhammad Anis uddin Nasir Scaling Online Social Networks 7/24

16. Motivation
Algorithms
SPAR
Our Contribution
JA-BE-JA
Evaluation
Conclusion
Features
Local Semantics
Load Balancing
Muhammad Anis uddin Nasir Scaling Online Social Networks 7/24

17. Motivation
Algorithms
SPAR
Our Contribution
JA-BE-JA
Evaluation
Conclusion
Features
Local Semantics
Load Balancing
Fault Tolerant
Muhammad Anis uddin Nasir Scaling Online Social Networks 7/24

18. Motivation
Algorithms
SPAR
Our Contribution
JA-BE-JA
Evaluation
Conclusion
Features
Local Semantics
Load Balancing
Fault Tolerant
Dynamic
Muhammad Anis uddin Nasir Scaling Online Social Networks 7/24

19. Motivation
Algorithms
SPAR
Our Contribution
JA-BE-JA
Evaluation
Conclusion
Features
Local Semantics
Load Balancing
Fault Tolerant
Dynamic
Low Replication Overhead
Muhammad Anis uddin Nasir Scaling Online Social Networks 7/24

20. Motivation
Algorithms
SPAR
Our Contribution
JA-BE-JA
Evaluation
Conclusion
Architecture
Partition Manager
Muhammad Anis uddin Nasir Scaling Online Social Networks 8/24

21. Motivation
Algorithms
SPAR
Our Contribution
JA-BE-JA
Evaluation
Conclusion
Architecture
Partition Manager
Directory Service
Muhammad Anis uddin Nasir Scaling Online Social Networks 8/24

22. Motivation
Algorithms
SPAR
Our Contribution
JA-BE-JA
Evaluation
Conclusion
Architecture
Partition Manager
Directory Service
Local Directory
Service
Muhammad Anis uddin Nasir Scaling Online Social Networks 8/24

23. Motivation
Algorithms
SPAR
Our Contribution
JA-BE-JA
Evaluation
Conclusion
Architecture
Partition Manager
Directory Service
Local Directory
Service
Replication
Manager
Muhammad Anis uddin Nasir Scaling Online Social Networks 8/24

24. Motivation
Algorithms
SPAR
Our Contribution
JA-BE-JA
Evaluation
Conclusion
SPAR Algorithm
Heuristic
Muhammad Anis uddin Nasir Scaling Online Social Networks 9/24

25. Motivation
Algorithms
SPAR
Our Contribution
JA-BE-JA
Evaluation
Conclusion
SPAR Algorithm
Heuristic
Greedy Optimization
Muhammad Anis uddin Nasir Scaling Online Social Networks 9/24

26. Motivation
Algorithms
SPAR
Our Contribution
JA-BE-JA
Evaluation
Conclusion
SPAR Algorithm
Heuristic
Greedy Optimization
Local Search
Muhammad Anis uddin Nasir Scaling Online Social Networks 9/24

27. Motivation
Algorithms
SPAR
Our Contribution
JA-BE-JA
Evaluation
Conclusion
SPAR Algorithm
Heuristic
Greedy Optimization
Local Search
Node Add/Remove
Muhammad Anis uddin Nasir Scaling Online Social Networks 9/24

28. Motivation
Algorithms
SPAR
Our Contribution
JA-BE-JA
Evaluation
Conclusion
SPAR Algorithm
Heuristic
Greedy Optimization
Local Search
Node Add/Remove
Edge Add/Remove
Muhammad Anis uddin Nasir Scaling Online Social Networks 9/24

29. Motivation
Algorithms
SPAR
Our Contribution
JA-BE-JA
Evaluation
Conclusion
SPAR Algorithm
Heuristic
Greedy Optimization
Local Search
Node Add/Remove
Edge Add/Remove
3 Configurations
Muhammad Anis uddin Nasir Scaling Online Social Networks 9/24

30. Motivation
Algorithms
SPAR
Our Contribution
JA-BE-JA
Evaluation
Conclusion
SPAR Algorithm
Heuristic
Greedy Optimization
Local Search
Node Add/Remove
Edge Add/Remove
3 Configurations
Server Add/Remove
Muhammad Anis uddin Nasir Scaling Online Social Networks 9/24

31. Motivation
Algorithms
SPAR
Our Contribution
JA-BE-JA
Evaluation
Conclusion
SPAR Algorithm
Heuristic
Greedy Optimization
Local Search
Node Add/Remove
Edge Add/Remove
3 Configurations
Server Add/Remove
Redistribution
Muhammad Anis uddin Nasir Scaling Online Social Networks 9/24

32. Motivation
Algorithms
SPAR
Our Contribution
JA-BE-JA
Evaluation
Conclusion
SPAR Algorithm
Heuristic
Greedy Optimization
Local Search
Node Add/Remove
Edge Add/Remove
3 Configurations
Server Add/Remove
Redistribution
Let it fill
Muhammad Anis uddin Nasir Scaling Online Social Networks 9/24

33. Motivation
Algorithms
SPAR
Our Contribution
JA-BE-JA
Evaluation
Conclusion
SPAR Algorithm
Heuristic
Greedy Optimization
Local Search
Node Add/Remove
Edge Add/Remove
3 Configurations
Server Add/Remove
Redistribution
Let it fill
Muhammad Anis uddin Nasir Scaling Online Social Networks 9/24

34. Motivation
Algorithms
SPAR
Our Contribution
JA-BE-JA
Evaluation
Conclusion
SPAR Algorithm
Heuristic
Greedy Optimization
Local Search
Node Add/Remove
Edge Add/Remove
3 Configurations
Server Add/Remove
Redistribution
Let it fill
Muhammad Anis uddin Nasir Scaling Online Social Networks 9/24

35. Motivation
Algorithms
SPAR
Our Contribution
JA-BE-JA
Evaluation
Conclusion
SPAR Algorithm
Heuristic
Greedy Optimization
Local Search
Node Add/Remove
Edge Add/Remove
3 Configurations
Server Add/Remove
Redistribution
Let it fill
Muhammad Anis uddin Nasir Scaling Online Social Networks 9/24

36. Motivation
Algorithms
SPAR
Our Contribution
JA-BE-JA
Evaluation
Conclusion
SPAR Algorithm
Heuristic
Greedy Optimization
Local Search
Node Add/Remove
Edge Add/Remove
3 Configurations
Server Add/Remove
Redistribution
Let it fill
Muhammad Anis uddin Nasir Scaling Online Social Networks 9/24

37. Motivation
Algorithms
SPAR
Our Contribution
JA-BE-JA
Evaluation
Conclusion
Features
Distributed Partitioning
Muhammad Anis uddin Nasir Scaling Online Social Networks 10/24

38. Motivation
Algorithms
SPAR
Our Contribution
JA-BE-JA
Evaluation
Conclusion
Features
Distributed Partitioning
k-way Partitioning
Muhammad Anis uddin Nasir Scaling Online Social Networks 10/24

39. Motivation
Algorithms
SPAR
Our Contribution
JA-BE-JA
Evaluation
Conclusion
Features
Distributed Partitioning
k-way Partitioning
Load Balancing
Muhammad Anis uddin Nasir Scaling Online Social Networks 10/24

40. Motivation
Algorithms
SPAR
Our Contribution
JA-BE-JA
Evaluation
Conclusion
Features
Distributed Partitioning
k-way Partitioning
Load Balancing
Low Inter-communication
Overhead
Muhammad Anis uddin Nasir Scaling Online Social Networks 10/24

41. Motivation
Algorithms
SPAR
Our Contribution
JA-BE-JA
Evaluation
Conclusion
Features
Distributed Partitioning
k-way Partitioning
Load Balancing
Low Inter-communication
Overhead
Local Search
Muhammad Anis uddin Nasir Scaling Online Social Networks 10/24

42. Motivation
Algorithms
SPAR
Our Contribution
JA-BE-JA
Evaluation
Conclusion
Overview
Sampling Policies
Muhammad Anis uddin Nasir Scaling Online Social Networks 11/24

43. Motivation
Algorithms
SPAR
Our Contribution
JA-BE-JA
Evaluation
Conclusion
Overview
Sampling Policies
Local
Muhammad Anis uddin Nasir Scaling Online Social Networks 11/24

44. Motivation
Algorithms
SPAR
Our Contribution
JA-BE-JA
Evaluation
Conclusion
Overview
Sampling Policies
Local
Random
Muhammad Anis uddin Nasir Scaling Online Social Networks 11/24

45. Motivation
Algorithms
SPAR
Our Contribution
JA-BE-JA
Evaluation
Conclusion
Overview
Sampling Policies
Local
Random
Hybrid
Muhammad Anis uddin Nasir Scaling Online Social Networks 11/24

46. Motivation
Algorithms
SPAR
Our Contribution
JA-BE-JA
Evaluation
Conclusion
Overview
Sampling Policies
Local
Random
Hybrid
Swapping Policies
Muhammad Anis uddin Nasir Scaling Online Social Networks 11/24

47. Motivation
Algorithms
SPAR
Our Contribution
JA-BE-JA
Evaluation
Conclusion
Overview
Sampling Policies
Local
Random
Hybrid
Swapping Policies
Energy Function
Muhammad Anis uddin Nasir Scaling Online Social Networks 11/24

48. Motivation
Algorithms
SPAR
Our Contribution
JA-BE-JA
Evaluation
Conclusion
Overview
Sampling Policies
Local
Random
Hybrid
Swapping Policies
Energy Function
Simulated Annealing
Muhammad Anis uddin Nasir Scaling Online Social Networks 11/24

49. Motivation
Algorithms
SPAR
Our Contribution
JA-BE-JA
Evaluation
Conclusion
Overview
Sampling Policies
Local
Random
Hybrid
Swapping Policies
Energy Function
Simulated Annealing
Algorithm
Muhammad Anis uddin Nasir Scaling Online Social Networks 11/24

50. Motivation
Algorithms
SPAR
Our Contribution
JA-BE-JA
Evaluation
Conclusion
Overview
Sampling Policies
Local
Random
Hybrid
Swapping Policies
Energy Function
Simulated Annealing
Algorithm
Hybrid Sampling
Muhammad Anis uddin Nasir Scaling Online Social Networks 11/24

51. Motivation
Algorithms
SPAR
Our Contribution
JA-BE-JA
Evaluation
Conclusion
Overview
Sampling Policies
Local
Random
Hybrid
Swapping Policies
Energy Function
Simulated Annealing
Algorithm
Hybrid Sampling
Simulated Annealing
Muhammad Anis uddin Nasir Scaling Online Social Networks 11/24

52. Motivation
Algorithms
Challenges
Our Contribution
Proposed Algorithm
Evaluation
Conclusion
1 Motivation
2 Algorithms
SPAR
JA-BE-JA
3 Our Contribution
Challenges
Proposed Algorithm
4 Evaluation
Datasets
Implementation
Results
5 Conclusion
Muhammad Anis uddin Nasir Scaling Online Social Networks 12/24

53. Motivation
Algorithms
Challenges
Our Contribution
Proposed Algorithm
Evaluation
Conclusion
Challenges
Global View
Muhammad Anis uddin Nasir Scaling Online Social Networks 13/24

54. Motivation
Algorithms
Challenges
Our Contribution
Proposed Algorithm
Evaluation
Conclusion
Challenges
Global View
Partition Manager
Muhammad Anis uddin Nasir Scaling Online Social Networks 13/24

55. Motivation
Algorithms
Challenges
Our Contribution
Proposed Algorithm
Evaluation
Conclusion
Challenges
Global View
Partition Manager
Replication Overhead
Muhammad Anis uddin Nasir Scaling Online Social Networks 13/24

56. Motivation
Algorithms
Challenges
Our Contribution
Proposed Algorithm
Evaluation
Conclusion
Challenges
Global View
Partition Manager
Replication Overhead
Load Balancing
Muhammad Anis uddin Nasir Scaling Online Social Networks 13/24

57. Motivation
Algorithms
Challenges
Our Contribution
Proposed Algorithm
Evaluation
Conclusion
Proposed Algorithm
SPAR + JA-BE-JA
Muhammad Anis uddin Nasir Scaling Online Social Networks 14/24

58. Motivation
Algorithms
Challenges
Our Contribution
Proposed Algorithm
Evaluation
Conclusion
Proposed Algorithm
SPAR + JA-BE-JA
Gossip Learning
Muhammad Anis uddin Nasir Scaling Online Social Networks 14/24

59. Motivation
Algorithms
Challenges
Our Contribution
Proposed Algorithm
Evaluation
Conclusion
Proposed Algorithm
SPAR + JA-BE-JA
Gossip Learning
Simulated
Annealing
Muhammad Anis uddin Nasir Scaling Online Social Networks 14/24

60. Motivation
Algorithms
Challenges
Our Contribution
Proposed Algorithm
Evaluation
Conclusion
Proposed Algorithm
SPAR + JA-BE-JA
Gossip Learning
Simulated
Annealing
Optimal Replication
Muhammad Anis uddin Nasir Scaling Online Social Networks 14/24

61. Motivation
Algorithms Datasets
Our Contribution Implementation
Evaluation Results
Conclusion
1 Motivation
2 Algorithms
SPAR
JA-BE-JA
3 Our Contribution
Challenges
Proposed Algorithm
4 Evaluation
Datasets
Implementation
Results
5 Conclusion
Muhammad Anis uddin Nasir Scaling Online Social Networks 15/24

62. Motivation
Algorithms Datasets
Our Contribution Implementation
Evaluation Results
Conclusion
Datasets
Synthetic Graphs
0
http://snap.stanford.edu/data/
Muhammad Anis uddin Nasir Scaling Online Social Networks 16/24

63. Motivation
Algorithms Datasets
Our Contribution Implementation
Evaluation Results
Conclusion
Datasets
Synthetic Graphs
Facebook Graphs
0
http://snap.stanford.edu/data/
Muhammad Anis uddin Nasir Scaling Online Social Networks 16/24

64. Motivation
Algorithms Datasets
Our Contribution Implementation
Evaluation Results
Conclusion
Datasets
Synthetic Graphs
Randomized
Facebook Graphs
0
https://gephi.org/
Muhammad Anis uddin Nasir Scaling Online Social Networks 17/24

65. Motivation
Algorithms Datasets
Our Contribution Implementation
Evaluation Results
Conclusion
Datasets
Synthetic Graphs
Randomized
Clustered
Facebook Graphs
0
https://gephi.org/
Muhammad Anis uddin Nasir Scaling Online Social Networks 17/24

66. Motivation
Algorithms Datasets
Our Contribution Implementation
Evaluation Results
Conclusion
Datasets
Synthetic Graphs
Randomized
Clustered
Highly Clustered
Facebook Graphs
0
https://gephi.org/
Muhammad Anis uddin Nasir Scaling Online Social Networks 17/24

67. Motivation
Algorithms Datasets
Our Contribution Implementation
Evaluation Results
Conclusion
Datasets
Synthetic Graphs
Randomized
Clustered
Highly Clustered
Facebook Graphs
150 nodes, 3386 edges
0
https://gephi.org/
Muhammad Anis uddin Nasir Scaling Online Social Networks 17/24

68. Motivation
Algorithms Datasets
Our Contribution Implementation
Evaluation Results
Conclusion
Datasets
Synthetic Graphs
Randomized
Clustered
Highly Clustered
Facebook Graphs
150 nodes, 3386 edges
224 nodes, 6384 edges
0
https://gephi.org/
Muhammad Anis uddin Nasir Scaling Online Social Networks 17/24

69. Motivation
Algorithms Datasets
Our Contribution Implementation
Evaluation Results
Conclusion
Datasets
Synthetic Graphs
Randomized
Clustered
Highly Clustered
Facebook Graphs
150 nodes, 3386 edges
224 nodes, 6384 edges
786 nodes, 60050 edges
0
https://gephi.org/
Muhammad Anis uddin Nasir Scaling Online Social Networks 17/24

70. Motivation
Algorithms Datasets
Our Contribution Implementation
Evaluation Results
Conclusion
Implementation
SPAR
Proposed Algorithm
Metric
Replication Overhead
Muhammad Anis uddin Nasir Scaling Online Social Networks 18/24

71. Motivation
Algorithms Datasets
Our Contribution Implementation
Evaluation Results
Conclusion
Evaluation of Replication Overhead
Replication Factor
Fault tolerance replicas reduce replication overhead
Proposed Algorithm performs better than SPAR
0
replication overhead = number of replicas/number of users
Muhammad Anis uddin Nasir Scaling Online Social Networks 19/24

72. Motivation
Algorithms Datasets
Our Contribution Implementation
Evaluation Results
Conclusion
Evaluation of Replication Overhead
Number of Servers
Less Replication overhead in the case of proposed algorithm
Proposed Algorithm performs better in the case of high
clusterization
0
replication overhead = number of replicas/number of users
Muhammad Anis uddin Nasir Scaling Online Social Networks 20/24

73. Motivation
Algorithms Datasets
Our Contribution Implementation
Evaluation Results
Conclusion
Evaluation of Replication Overhead
Number of Servers
Less Replication overhead in the case of proposed algorithm
Proposed Algorithm performs better in case of high
clusterization
0
replication overhead = number of replicas/number of users
Muhammad Anis uddin Nasir Scaling Online Social Networks 21/24

74. Motivation
Algorithms
Our Contribution
Evaluation
Conclusion
1 Motivation
2 Algorithms
SPAR
JA-BE-JA
3 Our Contribution
Challenges
Proposed Algorithm
4 Evaluation
Datasets
Implementation
Results
5 Conclusion
Muhammad Anis uddin Nasir Scaling Online Social Networks 22/24

75. Motivation
Algorithms
Our Contribution
Evaluation
Conclusion
Conclusion
Distributed social-based partitioning
Muhammad Anis uddin Nasir Scaling Online Social Networks 23/24

76. Motivation
Algorithms
Our Contribution
Evaluation
Conclusion
Conclusion
Distributed social-based partitioning
Local Semantics
Muhammad Anis uddin Nasir Scaling Online Social Networks 23/24

77. Motivation
Algorithms
Our Contribution
Evaluation
Conclusion
Conclusion
Distributed social-based partitioning
Local Semantics
Reduced Replication overhead compared to SPAR
Muhammad Anis uddin Nasir Scaling Online Social Networks 23/24

78. Motivation
Algorithms
Our Contribution
Evaluation
Conclusion
Conclusion
Distributed social-based partitioning
Local Semantics
Reduced Replication overhead compared to SPAR
Better load balancing using k-way partitioning
Muhammad Anis uddin Nasir Scaling Online Social Networks 23/24

79. Motivation
Algorithms
Our Contribution
Evaluation
Conclusion
Conclusion
Distributed social-based partitioning
Local Semantics
Reduced Replication overhead compared to SPAR
Better load balancing using k-way partitioning
Transparent Scaling
Muhammad Anis uddin Nasir Scaling Online Social Networks 23/24

80. Motivation
Algorithms
Our Contribution
Evaluation
Conclusion
Scaling Online Social Networks: extended SPAR
using Gossip Learning
Presented by: Muhammad Anis uddin Nasir
Coworker: Maria Stylianou
Supervised by: Sarunas Girdzijauskas
KTH Royal Institute of Technology
December 5, 2012
Muhammad Anis uddin Nasir Scaling Online Social Networks 24/24