Mbrace plos-slides final

MBrace: Cloud Computing with Monads
Jan Dzik

Nick Palladinos Kostas Rontogiannis
Eirik Tsarpalis Nikolaos Vathis
Nessos Information Technologies, SA

7th Workshop on Programming Languages
and Operating Systems
November 3, 2013

Eirik Tsarpalis (Nessos IT)


PLOS ’13

1 / 29

Introduction

Motivation

Motivation

Distributed Computation is Challenging.
Key to success: choose the right distribution framework.
Each framework tied to particular programming abstraction.
Map-Reduce, Actor model, Dataﬂow model, etc.



PLOS ’13

2 / 29

Introduction

Motivation

Established distributed frameworks

Restrict to specific distribution patterns.
Not expressive enough for certain classes of algorithms.
Difficult to influence task granularity.
Time consuming to deploy, manage and debug.



PLOS ’13

3 / 29

Introduction

What is MBrace?

What is MBrace?

1

A new programming model for the cloud.

2

An elastic, fault tolerant, multitasking cluster infrastructure.



PLOS ’13

4 / 29

Introduction

In This Talk

In This Talk

Concentrate on the programming model.
Distributed Computation.
Distributed Data.

Benchmarks.



PLOS ’13

5 / 29

The MBrace Programming Model

The Cloud Monad


A monad for composing distribution workﬂows.
Essentially a continuation monad that admits distribution.
Based on F# computation expressions.
Inspired by the successful F# asynchronous workﬂows.



PLOS ’13

6 / 29


The Cloud Monad

A Basic cloud workﬂow

let download (url : string) = cloud {
let client = new System.Net.WebClient()
let content = client.DownloadString(url)
return content
} : Cloud<string>



PLOS ’13

7 / 29


The Cloud Monad

Composing cloud workﬂows

let downloadSequential () = cloud {
let! c1 = download "http://m-brace.net/"
let! c2 = download "http://nessos.gr/"
let c = c1 + c2
return c
}



PLOS ’13

8 / 29


Distribution Combinators

Parallel Composition

let downloadParallel () = cloud {
let! c1,c2 =
download "http://m-brace.net/"
<||>
download "http://nessos.gr/"
return c1 + c2
}



PLOS ’13

9 / 29


Distribution Combinators

Distribution Primitives: an overview

Binary parallel operator:
<||> : Cloud<'T> -> Cloud<'U> -> Cloud<'T * 'U>
Variadic parallel combinator:
Cloud.Parallel : Cloud<'T> [] -> Cloud<'T []>
Non-deterministic parallel combinator:
Cloud.Choice : Cloud<'T option> [] -> Cloud<'T option>



PLOS ’13

10 / 29


Additional Constructs

Cloud Monad: additional constructs

Monadic for loops.
Monadic while loops.
Monadic exception handling.



PLOS ’13

11 / 29


Additional Constructs

Example: Inverse squares
let inverseSquares (inputs : int []) = cloud {
let jobs : Cloud<float> [] =
[|
for i in inputs ->
cloud { return 1.0 / float (i * i) }
|]
try
let! results = Cloud.Parallel jobs
return Array.sum results
with :? DivideByZeroException ->
return -1.0
}


PLOS ’13

12 / 29


Evaluation in the Cloud

How is it all executed?

Scheduler/worker cluster organization.
Symbolic execution stack (free monad/trampolines).
Scheduler interprets “monadic skeleton”.
Native “leaf expressions” dispatched to workers.
Symbolic stack winds across multiple machines.



PLOS ’13

13 / 29


Map-Reduce

A Map-Reduce implementation
let rec mapReduce (map : 'T -> Cloud<'R>)
(reduce : 'R -> 'R -> Cloud<'R>)
(identity : 'R)
(input : 'T list) =
cloud {
match input with
| [] -> return identity
| [value] -> return! map value
| _ ->
let left, right = List.split input
let! r1, r2 =
(mapReduce map reduce identity left)
<||>
(mapReduce map reduce identity right)
return! reduce r1 r2
}


PLOS ’13

14 / 29

The Distributed Data Programming Model

Introduction

What about Data Distribution?

MBrace does NOT include a storage service (for now).
Relies on third-party storage services.
Storage Provider plugin architecture.
Out-of-the-box support for FileSystem, SQL and Azure.
Future support for HDFS and Amazon S3.



PLOS ’13

15 / 29


The MBrace Data Programming Model

The MBrace Data Programming Model

Storage services interfaced through data primitives.
Data primitives act as references to distributed resources.
Initialized or updated through the monad.
Come in immutable or mutable ﬂavors.



PLOS ’13

16 / 29


Cloud Ref

Cloud Ref

Simplest distributed data primitive of MBrace.
Generic reference to a stored value.
Conceptually similar to ML ref cells.
Immutable by design.
Cached in worker nodes for performance.



PLOS ’13

17 / 29


Cloud Ref

Cloud Ref: Example

let createRef (inputs : int []) = cloud {
let! ref = CloudRef.New inputs
return ref : CloudRef<int []>
}

let deRef (ref : CloudRef<int []>) = cloud {
let content = ref.Value
return content : int []
}



PLOS ’13

18 / 29


Cloud Ref

Application: Data Sharding
type DistribTree<'T> =
| Leaf of 'T
| Branch of CloudRef<DistribTree<'T>> *
CloudRef<DistribTree<'T>>
let rec map (f : 'T -> 'S) (tree : DistribTree<'T>) =
cloud {
match tree with
| Leaf t -> return! CloudRef.New (Leaf (f t))
| Branch(l,r) ->
let! l', r' = map f l.Value <||> map f r.Value
return! CloudRef.New (Branch(l',r'))
}



PLOS ’13

19 / 29


Cloud File

Cloud File

References ﬁles in the distributed store.
Untyped, immutable, binary blobs.



PLOS ’13

20 / 29


Cloud File

Cloud File : Example

let getSize (file : CloudFile) = cloud {
let! bytes = CloudFile.ReadAllBytes file
return bytes.Length / 1024
}

cloud {
let! files = CloudDir.GetFiles "/path/to/files"
let jobs = Array.map getSize files
let! sizes = Cloud.Parallel jobs
return Array.sum sizes
}



PLOS ’13

21 / 29

The MBrace Framework

Performance

Performance

We tested MBrace against Hadoop.
Both frameworks were run on Windows Azure.
Clusters consisted of 4, 8, 16 and 32 quad-core nodes.
Two algorithms were tested, grep and k-means.
Source code available on github.



PLOS ’13

22 / 29


Performance

Distributed Grep (Windows Azure)

Count occurrences of given pattern from input ﬁles.
Straightforward Map-Reduce algorithm.
Input data was 32, 64, 128 and 256 GB of text.



PLOS ’13

23 / 29


Performance

Distributed Grep (Windows Azure)
400

Time (sec)

300

200

100
MBrace
Hadoop
0

20


40

60
80
worker cores

100


120

PLOS ’13

24 / 29


Performance

k-means Clustering (Windows Azure)

Centroid computation out of a set of vectors.
Iterative algorithm.
Not naturally deﬁnable with Map-Reduce workﬂows.
Hadoop implementation from Apache Mahout library.
Input was 106 , randomly generated, 100-dimensional points.



PLOS ’13

25 / 29


Performance

k-means Clustering (Windows Azure)
MBrace
Hadoop

Time (sec)

1,500

1,000

500

0

20


40

60
80
worker cores


100

120

PLOS ’13

26 / 29

Conclusions & Future Work

Conclusions

A big data platform for the .NET framework.
Language-integrated cloud workﬂows.
User-speciﬁable parallelism patterns and task granularity.
Distributed exception handling.
Pluggable storage services.
Data API integrated with programming model.



PLOS ’13

27 / 29


Future Work

Improved C# support.
A rich library of combinators and parallelism patterns.
A LINQ provider for data parallelism.

Support for the Mono framework and Linux.



PLOS ’13

28 / 29


Thank You!

Questions?

http://m-brace.net



PLOS ’13

29 / 29

Mbrace plos-slides final

Recommandé

Recommandé

Contenu connexe

Tendances

Tendances (20)

En vedette

En vedette (8)

Similaire à Mbrace plos-slides final

Similaire à Mbrace plos-slides final (20)

Dernier

Dernier (20)

Mbrace plos-slides final