Presentation at the conference Greenmetrics 2016 of the paper "Geographical Load Balancing across Green Datacenters: a Mean Field Analysis" (authors G. Neglia, M. Sereno, G. Bianchi)

1. Geographical Load Balancing
across Green Datacenters:
a Mean Field AnalysisMatteo Sereno
Dipartimento di Informatica
Università di Torino, Italia
Joint work with
G. Neglia (INRIA, SOPHIA ANTIPOLIS)
G. Bianchi (Università di Roma 2, Tor Vergata)

2. Geographical Load Balancing
Geographical Load Balancing (GLB) is a set of
techniques to schedule jobs to geographically
distributed datacenters
GLB has been suggested for data centers hosting
cloud computation
To exploit the electricity price differences across regions.
Greenmetrics, 14 June 2016 –
Antibes Juan les Pins
2

3. Geographical Load Balancing
Several approaches
Qureshi et al. [ACM SIGCOMM 2009]
Reducing the electricity cost in a wholesale market
environment
Lower electricity bill by adapting the load balancing
with dynamic electricity price variation
Liu et al. [ACM SIGMETRICS 2011]
Distributed algorithms for Geographical Load Balancing
Multiple sources for workload.
Incorporated capacity provisioning inside data centers
Investigated how renewable energy can be used to
lower the electricity price.
Lin et al. [IGCC 2012]
Online algorithms
Greenmetrics, 14 June 2016 –
Antibes Juan les Pins
3

4. Our Approach
A simple model for Geographical Load Balancing
Large number of Datacenters (federation of micro-datacenters)
Each datacenter can be powered by renewable or by traditional
energy sources
The renewable energy production (and its price) is modeled as a
stochastic process
The scheduling is decided on the basis of the current state of the
system
Number of jobs for each datacenter
Energy state for each datacenter (renewable / non-renewable)
Federation of N identical datacenters
Aggregated job arrival process with rate lN (Poisson process)
The service time of each job is exponentially distrubuted with
expected value 1/m
Greenmetrics, 14 June 2016 –
Antibes Juan les Pins
4

5. Our Approach (cont’d)
Each datacenter is connected to the power grid but it can be powered by
a renewable energy source (i.e., solar)
Each renewable source can be in two states {S, C}
in S the energy produced is sufficient to power the datacenter
in C the energy produced is negligible and the datacenter
is powered by the grid
Renewable states evolving according a CTMC
When a new job arrives the scheduler assigns it
a) to a datacenter that is available to process it and in state S (renewable),
if any
b) to an available datacenter, if any
c) to a central waiting queue from which the job will be moved to the first
available datacenter
The system works as an M/M/N queue with the characteristic that
available servers (datacenters) in state S get jobs with higher priority
than other servers
Greenmetrics, 14 June 2016 –
Antibes Juan les Pins
5
S C
nC
nS

6. Our Approach (cont’d II)
The system is represented by a Markov chain
𝐽 𝑁 𝑡 , 𝑆 𝑁 𝑡 , 𝐵𝑆
𝑁
𝑡
𝐽 𝑁 𝑡 = number of jobs in the system
𝑆 𝑁 𝑡 = number of servers in state S
𝐵𝑆
𝑁
𝑡 = number of busy servers in state S
The stationary distribution of 𝐽 𝑁 𝑡 and 𝑆 𝑁 𝑡 can be derived
but the characterization of 𝐵𝑆
𝑁
𝑡 is not easy
The system 𝐽 𝑁
𝑡 , 𝑆 𝑁
𝑡 , 𝐵𝑆
𝑁
𝑡 could be studied by using
simulation or by numerical solution
Computational cost increases with N
Even worse if we consider more complex arrival/service
processes and/or renewable energy evolution
Greenmetrics, 14 June 2016 –
Antibes Juan les Pins
6

7. System evolution
Characterization of 𝐵𝑆
𝑁
𝑡 /N that is
needed to quantify how many datacenters
use renewable energy
Greenmetrics, 14 June 2016 –
Antibes Juan les Pins
7
N
S
NN
BSJ ,,
1,1, 
N
S
NN
BSJ
1,,1 
N
S
NN
BSJ
N
S
NN
BSJ ,1, 
N
S
NN
BSJ ,1, 
1,1, 
N
S
NN
BSJ
1,,1 
N
S
NN
BSJ
N
S
NN
BSJ ,,1
S
N
S
N
B
J
n
)
(

S
N
S
N
N
B
S
J
N
n)
(



CN
SN B
S
n)
(

CN
S
B
n
mN
S
B
m)
(
N
S
N
B
J

lN )if(
NN
S
SB 
N
S
NN
BSJ ,,1
lN )if(
NN
S
SB 
Transitions that bring to a change in 𝐵𝑆
𝑁
(for J≤N)
If a new job arrives and there are idle datacenters in state S (𝐵𝑆
𝑁
< 𝑆 𝑁
)
then the job is assigned to one of them and 𝐵𝑆
𝑁
will increase.
Otherwise 𝐵𝑆
𝑁
will stay constant.

8. The fluid Model
We show that the stochastic dynamics of 𝐽 𝑁 𝑡 , 𝑆 𝑁 𝑡 , 𝐵𝑆
𝑁
𝑡
converges (in probability) to a deterministic process (as N
diverges)
Convergence introduced by Kurtz
The limiting process can be described by a system of differential
equations
𝑑𝒙
𝑑𝑡
= 𝒇(𝒙 𝑡 )
𝒇(. ) is called limiting drift function and must satisfy the Lipschitz
property
𝒇(. ) in our case is discontinuous (no Lipschitz property)
To overcome this problem we used the results derived by Gast &
Gaujal (G&G), Performance Evaluation 2012
The dynamics converge to a solution of differential inclusions
(i.e., the function 𝒇(. ) is replaced by a set valued function)
Greenmetrics, 14 June 2016 –
Antibes Juan les Pins
8

9. The fluid Model (cont’d II)
Derivation of the fluid limits for 𝐽 𝑁 𝑡 and 𝑆 𝑁 𝑡
We consider the scaled process 𝐽 𝑁 𝑡 /N and hence we have that
for that process the number of jobs in the system converges to
j* =
l
𝜇
with the same kind of arguments we can derive that the number
of servers in state S (normalized), that is 𝑆 𝑁
𝑡 /N converges to
s* =
n𝑆
n𝑆
+n𝐶
Obviously we would not have needed fluid models to derive the
asymptotic probability that a datacenter is busy or the probability that a
datacenter is powered by renewables
Greenmetrics, 14 June 2016 –
Antibes Juan les Pins
9

10. The fluid Model (cont’d III)
From this it follows that the drift is not continuous
By using the results derived in G&G we show that the
stationary distribution of 𝐵𝑆
𝑁
𝑡 /N converges to b*as N
increases with
𝑏𝑆
∗
= min 𝑠∗
, ρ
n 𝑆 + 𝜇
n 𝑆 + n 𝐶 + 𝜇
Greenmetrics, 14 June 2016 –
Antibes Juan les Pins
10
If a new job arrives and there are idle datacenters in
state S (𝐵𝑆
𝑁
< 𝑆 𝑁
) then the job is assigned to one of them
and 𝐵𝑆
𝑁
will increase.
Otherwise 𝐵𝑆
𝑁
will stay constant.

11. The fluid Model:
accuracy
Greenmetrics, 14 June 2016 –
Antibes Juan les Pins
11
0
0.1
0.2
0.3
0.4
0.5
0 0.2 0.4 0.6 0.8 1
BS/N
Fluid Model
0
0.1
0.2
0.3
0.4
0.5
0 0.2 0.4 0.6 0.8 1
BS/N
Fluid Model
Simulation N=20
0
0.1
0.2
0.3
0.4
0.5
0 0.2 0.4 0.6 0.8 1
BS/N
Fluid Model
Simulation N=20
Simulation N=100
0
0.1
0.2
0.3
0.4
0.5
0 0.2 0.4 0.6 0.8 1
BS/N
Fluid Model
Simulation N=20
Simulation N=100
Simulation N=500
For values of r far from the critical
value for which
𝑠∗
= 𝜌(n 𝑆+𝜇)/(n 𝑆+n 𝐶+𝜇)
corresponding to the non
differentiability of the drift the
approximation is accurate (even for
N=20)
We use a set of parameters
such that 𝑠∗
= 0.5

12. Model Exploitation
The percentage 𝑏𝑆
∗
of datacenters
working and powered by renewables is
limited by 𝑠∗
and by 𝜌, i.e., 𝑏𝑆
∗
= min{𝑠∗
, 𝜌}
Two regimes:
renewables-limited
load-limited
The dispatching algorithm can approach
the bound min 𝑠∗
, 𝜌
Depending on the term
n𝑆+𝜇
n𝑆+n𝐶+𝜇
Greenmetrics, 14 June 2016 –
Antibes Juan les Pins
12

13. Model Exploitation
(Federation vs no-federation)
Percentage of time a datacenter works
and is powered by renewables
Greenmetrics, 14 June 2016 –
Antibes Juan les Pins
13
𝑏𝑆
∗
= min 𝑠∗
, ρ
n 𝑆 + 𝜇
n 𝑆 + n 𝐶 + 𝜇
Federation
ρ ∙ 𝑠∗
no-Federation
but …
ρ<1
n𝑆+𝜇
n𝑆
+n𝐶
+𝜇
>
n𝑆
n𝑆
+n𝐶
= 𝑠∗
>
Every datacenter receives a load
r and can exploit renewables a
fraction s* of the timeThe difference between these
terms (times N) quantifies how
many datacenters work powered
by renewables thanks to the
federation (compared to the “no-
federation” scenario)

14. Model Exploitation
(Federation vs no-federation II)
The average energy cost per datacenter (per time unit)
𝑐𝑓 = ρ − 𝑏𝑆
∗
𝑐 𝑛𝑓 = ρ − ρ ∙ 𝑠∗
Greenmetrics, 14 June 2016 –
Antibes Juan les Pins
14
When r and s* are constant,
b* changes only for the effect
of the ratio (n 𝑆 + n 𝐶) 𝜇
If the ratio (n 𝑆 + n 𝐶) 𝜇 is large the
scheduling is not effective because
a datacenter changes its status S or
C many times before completing the
job
The job takes advantage of
renewable energy on average a
fraction s* of the time,
independently from the status S to C
when the execution starts
We use normalized cost (per time unit)
0 when powered by renewables
1 non-renewables

15. Model Exploitation
(Federation vs no-federation III)
Greenmetrics, 14 June 2016 –
Antibes Juan les Pins
15
The advantage of the federation
converges to 0 as the ratio
(n 𝑆 + n 𝐶) 𝜇 diverges
When r=0.5 the system is always in the
load-limited regime and the advantage of
the federation always decreases as the
ratio (n 𝑆 + n 𝐶) 𝜇 increases.
When r=0.65 the system is initially in the
renewables-limited regime,
the relative gain of the federation is
limited by the average availability of
renewables' energy and it is
independent on the speed of their
dynamics
As the ratio (n 𝑆 + n 𝐶) 𝜇 increases the
system enters in load-limited regime
(the relative improvement in this
regime is independent from r)

16. Model Exploitation
(Correlation)
Energy sources: independent CTMCs … Not very realistic!!!
The CTMC that characterizes a renewable source (S or C) is
modulated by another two states CTMC (G or B) common to all the
different sources
The transition rates n 𝑆 and n 𝐶 (for each renewable source)
depends of the state of modulating CTMC
n 𝑆𝐺, n 𝐶𝐺, and n 𝑆𝐵, n 𝑆𝐵
We consider
𝑠 𝐺
∗
=
n 𝑆𝐺
n 𝑆𝐺 + n 𝐶𝐺
>
n 𝑆𝐵
n 𝑆𝐵 + n 𝑆𝐵
= 𝑠 𝐵
∗
We introduced a correlation coefficient h (0 ≤ h ≤ 1)
0 → the modulating CTMC does not have any effect of the
renewables’ state evolution
1 → maximum effect of the modulating CTMC on the renewables’
state evolution (in particular, all the datacenters are in state S when
the modulating CTMC is in state G, and all the datacenters are in
state C when the modulating CTMC is in state B)
Greenmetrics, 14 June 2016 –
Antibes Juan les Pins
16

17. Greenmetrics, 14 June 2016 –
Antibes Juan les Pins
17
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
0 0.2 0.4 0.6 0.8 1
(cnf-cf)/cnf
=0.3
=0.5
=0.6
=0.8
Model Exploitation
(Correlation)
Interaction between
various parameters:
complex behaviors
We set the parameters
such that the average
percentage of time the
renewable sources can
power datacenters is
constant 𝑠∗
=0.5
(𝑠∗
= 𝑠 𝐺
∗
+𝑠 𝐵
∗
)
As the correlation
parameter h increases
𝑠 𝐺
∗
increases and 𝑠 𝐵
∗
decreases (and vice-versa)
The benefit is non-increasing in h
but depending on the load r there is range of the index h
such that the benefit does not depend on h
The transitions between these ranges depend on the states
renewable-limited / load-limited

18. Conclusions and developments
We use mean field techniques to derive a simple approximated
model for deriving performance measures for GLB strategies
Asymptotic convergence
Comparison with an ad-hoc discrete event simulator
Quantify the effects of different system parameters and different
trade-offs
Developments:
Comparison with real scenarios
Modify the model to make it more realistic
Datacenter heterogeneity
More sophisticated (but yet tractable) models for renewables
Scheduling with “locality” constraints (e.g., use of locality dependent
dispatching costs)
Beyond the GLB
Scheduling with limited knowledge of the datacenter states
Greenmetrics, 14 June 2016 –
Antibes Juan les Pins
18

19. Thank you for the
attention
?
Greenmetrics, 14 June 2016 – Antibes Juan les Pins 19