Distem is a distributed systems emulator. When doing research on Cloud, P2P, High Performance Computing or Grid systems, it can be used to transform an homogenenous cluster (composed of identical nodes) into an experimental platform where nodes have different performance, and are linked together through a complex network topology, making it the ideal tool to benchmark applications targetting such environments.

1. an introduction to
Distem – http://distem.gforge.inria.fr/ 1 / 17

2. Imagine you want to build a race car
Distem – http://distem.gforge.inria.fr/ 2 / 17

3. But you want it to work well
on both dry tracks and wet tracks
Distem – http://distem.gforge.inria.fr/ 3 / 17

4. You could use a simulation :
model the car and the track
and compute how they would interact
Distem – http://distem.gforge.inria.fr/ 4 / 17

5. Or you could use a real world experiment :
build a car, and try it on a real circuit
Distem – http://distem.gforge.inria.fr/ 5 / 17

6. Or you could use a real world experiment :
build a car, and try it on a real circuit
But sometimes, rain is unlikely
Distem – http://distem.gforge.inria.fr/ 5 / 17

7. You could move to another race track . . .
But what if you could change the weather ?
Distem – http://distem.gforge.inria.fr/ 6 / 17

8. You could move to another race track . . .
But what if you could change the weather ?
That’s what Distem does :
Take a platform, and alter it to suit
your experimentation needs
Distem – http://distem.gforge.inria.fr/ 6 / 17

9. An emulator for distributed systems
Take your real application
Run it on a cluster
And use Distem to alter the platform
so it matches the experimental conditions you need
Heterogeneous nodes
Long distance networks
Faults, perf. variations
Grid, Cloud, P2P features
. . .
Distem – http://distem.gforge.inria.fr/ 7 / 17

10. Ideal complement to other tools
SimGrid
Toolkit for the simulation of
distributed applications in
heterogeneous environments
Grid’5000
Large-scale platform
for experiments on
distributed systems
1700 nodes, 7000 cores
Fully reconfigurable
Grid’5000
Distem – http://distem.gforge.inria.fr/ 8 / 17

11. Ideal complement to other tools
SimGrid
Toolkit for the simulation of
distributed applications in
heterogeneous environments
Grid’5000
Large-scale platform
for experiments on
distributed systems
1700 nodes, 7000 cores
Fully reconfigurable
Grid’5000
Distem can run on Grid’5000
Distem – http://distem.gforge.inria.fr/ 8 / 17

12. What can Distem do for you ?
Introduce heterogeneity in an homogeneous cluster
How does your app perform when some nodes are slower ?
Emulate complex network topologies
How does your app perform on a Grid ? on a slower network ?
Inject faults and performance variations during the experiment
How does your app behave when a node crashes ?
When the available CPU time decreases ?
When the available network bandwidth increases ?
Distem – http://distem.gforge.inria.fr/ 9 / 17

13. Introducing heterogeneity
Distem splits real nodes into several virtual nodes
with a different number of cores
with different CPU performance
Each virtual node can be used as a real Linux system
0 1 2 3 4 5 6 7
VN 1 VN 2 VN 3 Virtual node 4
CPU cores
CPUperformance
Distem – http://distem.gforge.inria.fr/ 10 / 17

14. Emulating network topologies
Emulate several local networks linked together
Control available bandwidth and latency on each interface
n3
n1
n2
←5 Mbps, 10ms
10 Mbps, 5ms→
if0
←
1 Mbps, 30ms
1 Mbps, 30ms→
if0
←
100 Mbps, 3ms
100 Mbps, 1ms→
if0
n4
n5
←4 Mbps, 12ms
6 Mbps, 16ms→
if1
←
10 Kbps, 200ms
20 Kbps, 100ms→
if0
←
200 Kbps, 30ms
512 Kbps, 40ms→
if0
Distem – http://distem.gforge.inria.fr/ 11 / 17

15. Distem internals
Distem uses modern Linux features
to steal resources from applications :
LXC, cgroups, cpufreq, iptables, traffic control
Node 1
Node 2
Node 3
Switch
Node 1
Node 2
Node 3
Switch
Distem – http://distem.gforge.inria.fr/ 12 / 17

16. User interfaces
Command-line interface – distem command
Easy to use
Harder to automate
No access to more advanced features
Ruby library
Easy to automate
Access to all features
/ Easy to use if you known Ruby
REST API
Can be used from any language
Requires REST knowledge
Distem – http://distem.gforge.inria.fr/ 13 / 17

17. Overview of a Distem instance
User’s machine
Uses the command line interface,
the Ruby client library
or a REST client
Pnode 2
distemd
Pnode 3
distemd
Coordinator
& Pnode 1
Starts and controls other Pnodes
Keeps the global state of the platform
Is the gateway to Pnodes and Vnodes
distemdREST
REST
Vnodes
Distem – http://distem.gforge.inria.fr/ 14 / 17

18. Example experiment : SCP vs Rsync
Transfer 50 files (total : 5 MB)
Available bandwidth : 10 Mbps
Varying network latency
0 20 40 60 80 100
0
10
20
Emulated latency (ms)
Transfertime(s)
SCP
Rsync
Distem – http://distem.gforge.inria.fr/ 15 / 17

19. Distem scales to thousands of virtual nodes
Executing a command with ClusterShell and TakTuk
0 10,000 20,000 30,000 40,000
0
200
400
600
Virtual nodes
Executiontime(s)
ClusterShell
TakTuk
Distem – http://distem.gforge.inria.fr/ 16 / 17

20. More information ?
http://distem.gforge.inria.fr/
Distem – http://distem.gforge.inria.fr/ 17 / 17