Secure Multi-Party Computation Based Privacy Preserving Extreme Learning Machine Algorithm Over Vertically Distributed Data

Introduction
Preliminaries
Privacy-preserving ELM over vertically partitioned data
Experiments
Conclusions
Secure Multi-Party Computation Based Privacy
Preserving Extreme Learning Machine Algorithm
Over Vertically Distributed Data
Ferhat Özgür Çatak
ozgur.catak@tubitak.gov.tr
TÜB˙ITAK - B˙ILGEM - Cyber Security Institute
Kocaeli, Turkey
Nov. 10 2015
The 2015 International Data Mining and Cybersecurity Workshop
With ICONIP2015
Ferhat Özgür Çatak ozgur.catak@tubitak.gov.tr TÜB˙ITAK - B˙ILGEM - Cyber Security Institute Kocaeli, TurkeySecure Multi-Party Computation Based Privacy Preserving Extreme Learnin

Introduction
Preliminaries
Experiments
Conclusions
Table of Contents
1 Introduction
Secure Multiparty Computation
Contributions
2 Preliminaries
ELM
Secure Multi Party Computation
Secure Multi-Party Addition
3 Privacy-preserving ELM over vertically partitioned data
Privacy-preserving ELM
4 Experiments
Experimental setup
Simulation Results
5 Conclusions
Conclusions

Introduction
Preliminaries
Experiments
Conclusions
Contributions
Table of Contents
1 Introduction
Contributions
2 Preliminaries
ELM
4 Experiments
Experimental setup
Simulation Results
5 Conclusions
Conclusions

Introduction
Preliminaries
Experiments
Conclusions
Contributions
Secure Multiparty Computation (SMC)
Deﬁnition
SMC : The goal of creating methods for parties to jointly compute
a function over their inputs while keeping those inputs private.
In an MPC,
a given number of participants, p1, p2, · · · , pN
each has private data, respectively d1, d2, · · · , dN
Participants want to compute the value of a public function on
that private data:
F(d1, d2, ..., dN ) while keeping their own inputs secret.
Challenges :
To protect each participant’s private data, and intermediate results
The computation/communication cost introduced to each
participant shall be aﬀordable
Training data is arbitrarily partitioned

Introduction
Preliminaries
Experiments
Conclusions
Contributions
Contributions
Main contributions of the work
Privacy-preserving Extreme Learning Machine (ELM) training
model
Global ELM classiﬁcation model from the distributed data sets in
multiple parties
Training data set is vertically partitioned among the parties
The ﬁnal distributed model is constructed at an independent party

Introduction
Preliminaries
Experiments
Conclusions
ELM
Table of Contents
1 Introduction
Contributions
2 Preliminaries
ELM
4 Experiments
Experimental setup
Simulation Results
5 Conclusions
Conclusions

Introduction
Preliminaries
Experiments
Conclusions
ELM
Extreme Learning Machine (ELM)
Learning Without Iterative Tuning
All hidden node parameters can be randomly generated without the
knowledge of the training data. That is, for any continuous target
function f and any randomly generated sequence
limL→∞ ||f (x) − fL(x)|| = limL→∞ ||f (x) −
L
i=1 βi G(ai , bi , x)|| = 0
holds with probability one if βi is chosen to minimize
||f (x) − fL(x)||, ∀i
G.-B. Huang, et al., ”Universal approximation using incremental constructive
feedforward networks with random hidden nodes,” IEEE Transactions on Neural
Networks, vol. 17, no. 4, pp. 879-892, 2006.
G.-B. Huang, et al., ”Convex incremental extreme learning machine,”
Neurocomputing, vol. 70, pp. 3056-3062, 2007.
G.-B. Huang, et al., ”Enhanced random search based incremental extreme learning
machine,” Neurocomputing, vol. 71, pp. 3460-3468, 2008.

Introduction
Preliminaries
Experiments
Conclusions
ELM
Figure: Generalized SLFN
Mathematical Model
For N arbitrary distinct samples
(xi , ti ) ∈ Rn × Rm, SLFNs with L
hidden nodes each with output
function, G(ai , bi , x) are
mathematically modeled as
L
i=1
βi G(ai , bi , xj ) = tj , j = 1, ..., N
(1)
(ai , bi ): hidden node parameters.
βi : the weight vector connecting the
ith hidden node and the output node.

Introduction
Preliminaries
Experiments
Conclusions
ELM
Mathematical Model
L
i=1 βi G(ai , bi , xj ) = tj , j = 1, ..., N is equivalent to Hβ = T where,



h(x1)
...
h(xN)


 =



G(a1, b1, x1) · · · G(aL, bL, x1)
...
...
...
G(a1, b1, xN) · · · G(aL, bL, xN)



N×L
(2)
β =



βT
1
...
βT
L



L×m
ve T =



tT
1
...
tT
N



N×m
(3)
H, is called the hidden layer output matrix of the neural network, the ith
column of H is the output of the ith hidden node with respect to inputs ,
x1, ..., xn

Introduction
Preliminaries
Experiments
Conclusions
ELM
Three-Step Learning Model
Given a training set D = {(xi , ti )|xi ∈ Rn
, ti ∈ Rm
, i = 1, · · · , N}, hidden
node output function G(a, b, x), and the number of hidden nodes L,
1 Assign randomly hidden node parameters (ai , bi ), i = 1, · · · , L.
2 Calculate the hidden layer output matrix H.
3 Calculate the output weight β = Ht
T.

Introduction
Preliminaries
Experiments
Conclusions
ELM
Secure Multi-Party Computation
The problem of n players to compute an agreed function f of their
inputs
In vertically partitioned data, each party holds diﬀerent attributes of
same data set.
The partition strategy is shown in Figure 2.
x1,1 · · · x1,t−1 · · · x1,t · · · x1,k
x2,1 · · · x2,t−1 · · · x2,t · · · x2,k
...
...
...
...
...
...
...
xm,1 · · · xm,t−1 · · · xm,t · · · xm,k














P1 · · · Pk
D =
Figure: Vertically partitioned data set D.

Introduction
Preliminaries
Experiments
Conclusions
ELM
Secure Multi-Party Addition I
Assume k ≥ 3 parties, P0, · · · , Pk−1, with party Pi holding value vi
Aim is to compute v =
k−1
i=0
vi
Sum of vi lies in F
P0 randomly chooses a number R ∈ F
P0 adds R to its local value v0,
Sends v0 = R + v0mod|F| to P1
The remaining parties Pi , i = 1, · · · , k − 1
Pi receives V = R +
i−1
j=0
vj mod|F|
Pi sends R + j=1
ivj mod|F| = (vi + V )mod|F|
Finally, P0, subtracts R from the ﬁnal message to compute actual
results.

Introduction
Preliminaries
Experiments
Conclusions
ELM
Secure Multi-Party Addition II
Party0
x0 , R ∈ F
Party1
x1
Party2
x2
Partyk
xk
V = R + x0mod|F|
V = R +
i−1
j=0 xjmod|F|
(xi + V )mod|F|
Figure: Secure Multi-Party Addition.

Introduction
Preliminaries
Experiments
Conclusions
ELM
Secure Multi-Party Addition III
Algorithm 1: Secure multi-party addition
1: procedure SMA(P)
2: P0 : R ← rand(F) P0 randomly chooses a number R
3: V ← R + x0 mod F
4: P0 sends V to node P1
5: for i = 1, · · · , k − 1 do
6: Pi receives V = R +
i−1
j=0
xj mod F
7: Pi computes V = R +
i
j=1
xj mod F = ((xi + V ) mod F)
8: Pi sends V to node Pi+1
9: P0 : V ← (V − R) = (V − R mod F) Actual addition result
10: end procedure

Introduction
Preliminaries
Experiments
Conclusions
Table of Contents
1 Introduction
Contributions
2 Preliminaries
ELM
4 Experiments
Experimental setup
Simulation Results
5 Conclusions
Conclusions

Introduction
Preliminaries
Experiments
Conclusions
Privacy-preserving ELM I
1 Master party creates weight matrix, W ∈ RL×N
2 Master party distributes partition W with same feature size for each
parties.
3 Party P0 creates a random matrix,
R =



rand1,1(F) · · · rand1,L(F)
...
...
...
randN,1(F) · · · randN,L(F)



N×L
4 Party P0 creates perturbated output,
V = R +



x0
1 · w0
1 + b0
1 · · · x0
1 · w0
L + b0
L
...
...
...
x0
N · w0
1 + b0
1 · · · x0
N · w0
L + b0
L



5 for i = 1, · · · , k − 1

Introduction
Preliminaries
Experiments
Conclusions
Privacy-preserving ELM II
Pi computes V = V +



xi
1 · wi
1 + bi
1 · · · xi
1 · wi
L + bi
L
...
...
...
xi
N · wi
1 + bi
1 · · · xi
L · wi
N + bi
N



Pi sends V to Pi+1
6 P0 subtracts random matrix, R, from the received matrix V.
H = (V − R) mod F
7 Hidden layer node weight vector, β, is calculated. β = H†
· T

Introduction
Preliminaries
Experiments
Conclusions
Privacy-preserving ELM III
P arty0
x0
, w, H0
,R ∈ F,
Party1
x1
, H1
, w
Party2
x2
, H2
, w
Partyk
xk−1
,
Hk−1
, w
V0 = R +
H0
mod|F|
V1 = V0+H1
mod|F|
V2 = V1 + H2
mod|F|
Vk = Vk−1 + Hk
mod|F|
Hi
=




G(w1, b1, xi
1) · · · G(wL, bL, xi
1)
.
.
.
...
.
.
.
G(w1, b1, xi
N ) · · · G(wL, bL, xi
N )




N×L
R =




rand1,1(F) · · · rand1,L(F)
.
.
.
...
.
.
.
randN,1(F) · · · randN,L(F)




N×L
H = (V − R) mod F
Figure: Overview of privacy-preserving ELM learning.

Introduction
Preliminaries
Experiments
Conclusions
Experimental setup
Simulation Results
Table of Contents
1 Introduction
Contributions
2 Preliminaries
ELM
4 Experiments
Experimental setup
Simulation Results
5 Conclusions
Conclusions

Introduction
Preliminaries
Experiments
Conclusions
Experimental setup
Simulation Results
Experimental setup I
Our approach is applied to six different data sets to verify its model
effectivity and efficiency.
Table: Description of the testing data sets used in the experiments.
Data set #Train #Classes #Attributes
australian 690 2 14
colon-cancer 62 2 2,000
diabetes 768 2 8
duke breast cancer 44 2 7,129
heart 270 2 13
ionosphere 351 2 34

Introduction
Preliminaries
Experiments
Conclusions
Experimental setup
Simulation Results
Simulation Results I
The number of party size, k: from 3 to number of feature n, of the
data set
For instance, k = 3, and n = 14, then the ﬁrst two party have 5
attributes, and last party has 4 attributes.
The accuracy of secure multi-party computation based ELM is
exactly same for the traditional ELM training algorithm.

Introduction
Preliminaries
Experiments
Conclusions
Experimental setup
Simulation Results
Simulation Results II
Figure shows results of our simulations.
Time scale becomes its steady state position when number of parties
k, moves closer to number of attributes, n.
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1
10
−1
10
0
10
1
10
2
10
3
AttributeS ize(n)
P artyS ize(k)
Timescale
australian colon−cancer diabetes duke heart ionosphere
Figure: Vertically partitioned data set D.

Introduction
Preliminaries
Experiments
Conclusions
Conclusions
Table of Contents
1 Introduction
Contributions
2 Preliminaries
ELM
4 Experiments
Experimental setup
Simulation Results
5 Conclusions
Conclusions

Introduction
Preliminaries
Experiments
Conclusions
Conclusions
Conclusions
Summary
ELM learning algorithm, a new method.
ELM outperforms traditional Single Layer Feed-forward Neural-networks and
Support Vector Machines for Big Data 1
In all ﬁelds that ELM is applied (i.e. medical records, business, government),
privacy is a major concern.
Conclusions
Privacy-preserving learning model is proposed in vertically partitioned data
In multi-party partitioning without sharing the data of each site to
the others.
Master party divides weight vector, and each party calculates the activation
function result with its data and weight vector.
1Cambria, Erik, et al. ”Extreme learning machines [trends & controversies].”
Intelligent Systems, IEEE 28.6 (2013): 30-59.

Introduction
Preliminaries
Experiments
Conclusions
Conclusions
Thank You
Ferhat Özgür Çatak
ozgur.catak@tubitak.gov.tr

Secure Multi-Party Computation Based Privacy Preserving Extreme Learning Machine Algorithm Over Vertically Distributed Data

Recommended

Recommended

More Related Content

What's hot

What's hot (19)

Viewers also liked

Viewers also liked (20)

Similar to Secure Multi-Party Computation Based Privacy Preserving Extreme Learning Machine Algorithm Over Vertically Distributed Data

Similar to Secure Multi-Party Computation Based Privacy Preserving Extreme Learning Machine Algorithm Over Vertically Distributed Data (20)

More from Ferhat Ozgur Catak

More from Ferhat Ozgur Catak (19)

Recently uploaded

Recently uploaded (20)

Secure Multi-Party Computation Based Privacy Preserving Extreme Learning Machine Algorithm Over Vertically Distributed Data