Dataflow: the concurrency/parallelism architecture you need

@russel_winder#devoxxuk #dataflowrules Copyright © 2014 Russel Winder
Dataflow:
Russel Winder
@russel_winder
http://www.russel.org.uk
russel@winder.org.uk
The Concurrency/Parallelism
Architecture You Need

What is Dataflow?

What are (in computing†
):
Concurrency:
Structuring solution and code
such that multiple parts may
execute independently and
possibly even at the same
time.
Parallelism:
Execute multiple parts of a
system at the same time on
different processors so as to
get things working faster.
†
In natural language these words have very different meanings.

What is Dataflow?
An architecture comprising channels allowing data to flow from
one operator to another, where each operator has multiple
input channels and multiple output channels, and executes
code only in response to the arrival of data on the inputs.

Historically
Dataflow computers:
– Values flowing between…
–…operators that calculate…
–…new values to pass to…
–…other operators.
Dataflow hardware didn't take
off, but the architecture works
at various scales.
The Manchester Prototype Dataflow Computer
J R Gurd, C C Kirkham, I Watson
CACM 28(1), 1985-01.

Dataflow diagrams have been an
integral part of analysis and design of
information systems since the 1970s
T de Marco, Structured Analysis and Systems Specification,
Yourdon Press, NY, 1978.

Dataflow and Functional
Operators seem like they
might be pure functions,
but…
…they are not necessarily,
operators may have internal
state.
Operators may be referentially
transparent, but they may be
not.
Operators may even have side
effects.

Dataflow is an
event-based
architecture

Dataflow systems are
(possibly)
reactive systems.
Which would make them exceedingly
trendy even if the idea is very old.

Dataflow systems have
no†
shared memory.
†
or at least should have no.

operator
channel

Dataflow systems are
message passing systems.

Each operator must†
be single threaded.
†
or at least should.

Dataflow Frameworks
Scala:
–Future
Akka:
–Dataflow variables, aka
Promise
–Deprecated in favour of Async
Java:
–Pre-8, Future
–8+, CompletableFuture, aka
Promise

Architectural Issue
Each of the aforementioned frameworks assumes that each
operator creates a single value. Communication is by dataflow
variables: each dataflow variable is a thread-safe single
assignment variable.

GPars…
Has dataflow variables
(promises) and tasks and so
can do everything Akka and
Java can offer.
Has DataflowQueue, and so
can create real dataflow
networks.

One does like to code…
…doesn't one.

We need a problem…

A Problem
Calculate mean and standard deviation of a data sample.
¯x =
1
n
∑i=0
n
xi
s =
√ 1
n−1
∑i=0
n
(xi−¯x)2

Amend the Problem
s =
√ 1
n−1 ((∑i=0
n
xi
2
)−n¯x ¯x)
¯x =
1
n
∑i=0
n
xi

Switch to using an IDE for this.Switch to using an IDE for this.
Code Example

Summary
Dataflow is an architecture:
Event-driven, single-threaded
operators communicating by
message passing using
channels.
Dataflow is an easement:
Synchronization is inherent in
the model, and there is no
shared memory, so all
deadlocks are trivial.

Dataflow is a way of harnessing
concurrency and parallelism
in easy to program ways.

GPars is usable from Java
as well as Groovy.

Testing is really Groovy with Spock.

Dataflow is an architecture of
code you need to know.

Dataflow: the concurrency/parallelism architecture you need

Recommandé

Recommandé

Contenu connexe

Similaire à Dataflow: the concurrency/parallelism architecture you need

Similaire à Dataflow: the concurrency/parallelism architecture you need (20)

Plus de Russel Winder

Plus de Russel Winder (20)

Dernier

Dernier (20)

Dataflow: the concurrency/parallelism architecture you need