Clustering is often an essential first step in datamining intended to reduce redundancy, or define data categories. Hierarchical clustering, a widely used clustering technique, can offer a richer representation by suggesting the potential group structures. However, parallelization of such an algorithm is challenging as it exhibits inherent data dependency during the hierarchical tree construction. In this paper, we design a parallel implementation of Single-linkage Hierarchical Clustering by formulating it as a Minimum Spanning Tree problem. We further show that Spark is a natural fit for the parallelization of single-linkage clustering algorithm due to its natural expression of iterative process. Our algorithm can be deployed easily in Amazon’s cloud environment. And a thorough performance evaluation in Amazon’s EC2 verifies that the scalability of our algorithm sustains when the datasets scale up.

1. Hierarchical clustering
using spark
Chen Jin
UberEats

2. Mo#va#on
• Why Clustering
• Why Hierarchical
• Why Spark

3. Hierarchical Clustering
• Agglomera#ve (bo=om up):
– Each point is a cluster ini#ally
– Repeatedly merge the two “nearest” clusters into
one
• Divisive (top down):
– Start with one cluster and recursive

4. Single-Linkage Hierarchical Clustering
(SHC)
Data:
• A simple clustering algorithm
• Define a distance (or dissimilarity)
between cluster
• Ini#alize: every data point is a
cluster
• Iterate
– Compute distance between all
clusters (store for efficiency)
– Merge two closest clusters
• Save both clustering and
sequence of cluster opera#ons
• “dendrogram”

5. Example: Hierarchical Clustering
(Iter 1)
Dendrogram: Data:
Height of the
join indicates
dissimilarity

6. Example: Hierarchical Clustering
(Iter 2)
Dendrogram: Data:
Height of the
join indicates
dissimilarity

7. Example: Hierarchical Clustering
(Iter 3)
Dendrogram: Data:
Height of the
join indicates
dissimilarity

8. Implementa#on
• The total run#me complexity is O(N2logN) and
space complexity is O(N2)
– too expensive for really big datasets
– don’t fit in memory

9. SHAS
Single-linkage Hierarchical clustering Algorithm using Spark
• Paralleliza#on

10. From Clustering to Graph problem
Single-linkage Hierarchical Clustering to
Minimum Spanning Tree
s,
by
to
Based on the theoretical finding [17] that calculating the
SHC dendrogram of a dataset is equivalent to finding the
Minimum Spanning Tree (MST) of a complete weighted
graph, where the vertices are the data points and the edge
weights are the distances between any two points, the SHC
problem with a base dataset D can be formulated as follows:
“Given a complete weighted graph G(D) induced by
the distances between points in D, design a parallel algorithm
to find the MST in the complete weighted graph G(D)”.
To show the process of problem decomposition or complete
graph partition, a toy example is illustrated in Figure 1.
Given an original dataset D, we first divided it into two
disjoint subsets: D1 and D2, thus the complete graph G(D)
is decomposed into to three subgraphs: G(D1), G(D2) and
GB(D1, D2), where GB(D1, D2) is the complete bipartite
graph on datasets D1 and D2. In this way, any possible edge
is assigned to some subgraph, and taking the union of these
subgraphs would return us the original graph. This approach
can be easily extended to s splits, and leads to multiple

11. Problem Decomposi#on
Merge%
D
TD1!
TD2!
T(D1 , D2)%
%
TD%
Split%
Local%%
MST%
D1!
D2!
gB(D1 , D2)%
%
gD2%
gD1%
gD%
%
Fig. 1: Illustration of the divide-and-conquer strategy on
input dataset D. We divide dataset D into two smaller parts,
parallelism (i.e, the
show how we conve
MST finding proble
turns into the graph
Based on the th
SHC dendrogram
Minimum Spanning
graph, where the v
weights are the dis
problem with a bas
“Given a comple
the distances betwe
to find the MST in
To show the proces
graph partition, a
Given an original
disjoint subsets: D1
is decomposed into
GB(D1, D2), where
graph on datasets D

12. Problem Decomposi#on
Merge%
D
TD1!
TD2!
T(D1 , D2)%
%
TD%
Split%
Local%%
MST%
D1!
D2!
gB(D1 , D2)%
%
gD2%
gD1%
gD%
%
Fig. 1: Illustration of the divide-and-conquer strategy on
input dataset D. We divide dataset D into two smaller parts,
parallelism (i.e, the
show how we conve
MST finding proble
turns into the graph
Based on the th
SHC dendrogram
Minimum Spanning
graph, where the v
weights are the dis
problem with a bas
“Given a comple
the distances betwe
to find the MST in
To show the proces
graph partition, a
Given an original
disjoint subsets: D1
is decomposed into
GB(D1, D2), where
graph on datasets D

13. Problem Decomposi#on
Merge%
D
TD1!
TD2!
T(D1 , D2)%
%
TD%
Split%
Local%%
MST%
D1!
D2!
gB(D1 , D2)%
%
gD2%
gD1%
gD%
%
Fig. 1: Illustration of the divide-and-conquer strategy on
input dataset D. We divide dataset D into two smaller parts,
parallelism (i.e, the
show how we conve
MST finding proble
turns into the graph
Based on the th
SHC dendrogram
Minimum Spanning
graph, where the v
weights are the dis
problem with a bas
“Given a comple
the distances betwe
to find the MST in
To show the proces
graph partition, a
Given an original
disjoint subsets: D1
is decomposed into
GB(D1, D2), where
graph on datasets D

14. MST algorithms (1)
• Kruskal
– Implementa#on
• Create a forest F (a set of trees) where each vertex in
the graph is a separate tree.
• Create a set S containing all the edges in the graph
(minHeap)
• While S is nonempty and F is not yet spanning, remove
an smallest edge from S if the removed edge connects
two different trees and then add it to the forest
• O(ElogV) and O(E)

15. MST algorithms (2)
• Prim
– O(V2) and O(V)
• quadra#c #me complexity and linear space complexity.
– Local MST
• For both complete graphs and complete bipar#te
graphs

16. Merge algorithm
• Kruskal Algorithm
– Run on the reducer
• Union-find (disjoint set) data structure
– Union by rank (amor#zed Log(V) per opera#on)
– Find (path compression)
• Merging factor K
– most neighboring subgraphs share half of the data
points
– detect and eliminate incorrect edges at an early stage
and reduce the overall communica#on cost for the
algorithm

17. SHAS
Single-linkage Hierarchical clustering Algorithm using Spark
• Paralleliza#on
• Using Spark

18. SHAS’s Spark driver code
d T to obtain
sub-MSTs and
cess, however,
computational
into multiple
r K such that
h iteration and
MSTs, we use
p track of the
Recall the way
s share half of
taken place only when RDDs are actually needed. It accepts
iterative programs, which create and consume RDDs in a loop.
By using Spark’s Java APIs, Algorithm 1 can be implemented
as a driver program naturally. The main snippet is listed below:
1 JavaRDD<String> subGraphIdRDD = sc
2 .textFile(idFileLoc,numGraphs);
3
4 JavaPairRDD<Integer, Edge> subMSTs =
subGraphIdRDD.flatMapToPair(
5 new LocalMST(filesLoc, numSplits));
6
7 numGraphs = numSplits * numSplits / 2;
8
9 numGraphs = (numGraphs + (K - 1)) / K;
10
11 JavaPairRDD<Integer, Iterable<Edge>>
mstToBeMerged = subMSTs
12 .combineByKey(
13 new CreateCombiner(),
14 new Merger(),
15 new KruskalReducer(numPoints),
16 numGraphs);
17
18 while (numGraphs > 1) {
19 numGraphs = (numGraphs + (K - 1)) / K;
20 mstToBeMerged = mstToBeMerged
21 .mapToPair(new SetPartitionId(K))
22 .reduceByKey(
23 new KruskalReducer(numPoints),
24 numGraphs);
25 }
all the
one M
Cl
from
pay-a
of the
Elasti
host o
applic
also c
“m2.4
8 virt
two 8
types,
intens
PrePar++on
LocalMST
Kruscal-Merge

19. Pre-Par##on Phase
• Pre-Par##on the data points (s splits)
• Input files (tagged with graph type)
– s(s-1)/2 + s
• Complete Bipar#te graphs: s(s-1)/2
• Complete graphs: s
– Given a certain graph type, we apply the
corresponding Prim’s algorithm accordingly
• Load balancing

20. Local Computa#on Phase
• Lazy execu#on
– LocalMST transforma#on starts to be realized only
when reduceBy ac#on takes place
• Loca#on-aware scheduling
– Schedule the reducer as the same node as
mapped results
– Minimize the data shuffle

21. Merge Phase
• K-way merger
– gid = gid/k
• Guarantees that K consecu#ve subgraphs are
processed in the same reduce procedure.
• the number of parallelism decreases by K per
itera#on.

22. Performance
• 2, 000, 000 data points with high dimension
feature
• Achieve 300x speedup on 398 computer cores

23. Data Sets TABLE I: Structural properties of the synthetic-cluster and
synthetic-random testbed
Name Points dimensions size (MByte)
clus100k 100k 5, 10 5, 10
clus500k 500k 5, 10 20, 40
clus2m 2m 5, 10 80, 160
rand100k 100k 5, 10 5, 10
rand500k 500k 5, 10 20, 40
rand2m 2m 5, 10 80, 160
B. Performance
In each experiment, the data is split into a certain num-
ber of partitions evenly without any assumption of the data
Structural proper#es of the synthe#c-cluster and synthe#c-random testbed

24. Performance
Fig. 2: The execution time comparison between Spark andThe execu#on #me comparison between Spark and MapReduce

25. Speedup using ~400 cores

26. Speedup with the merge factor K
cores.

27. Total Remote Bytes Read Per itera#on
CPU load
Hierarc
the structu
number o
quadratic
tolerable
Several eff
algorithm,
tectures an
multi-core
popularize
its implem
Cluster
related to
set of poin
are known
dates back

28. (a) The first iteration

29. (a) The first iteration
(b) One of later iterations
Fig. 6: Snapshot of cluster utilization at the first iteration and one of later iterations.
(b) One of later itera#ons

30. Conclusions
• Reduce to MST problem
• Small data shuffle is the key to achieve linear
speedup

31. Questions
• Data source
– IBM Quest synthe#c data genera#on
• Source code
– h=ps://github.com/xiaocai00/SparkPinkMST
• Paper
– h=p://citeseerx.ist.psu.edu/viewdoc/download?
doi=10.1.1.719.5711&rep=rep1&type=pdf
• Uber is hiring
– cjin@uber.com