Distributed Computing Impossibility Proofs

----------------
Impossibility
In the area of Distributed Computing.
----------------© Pawel Szulc, @EncodePanda, paul.szulc@gmail.com 1

NANCY LYNCH. A hundred impossiblility proofs for
distributed computing. In Proceedings of the Eighth
Annual ACM Symposium on Principles of Distributed
Computing, Edmonton, Alberta, Canada, August 1989
http://groups.csail.mit.edu/tds/papers/Lynch/
podc89.pdf
© Pawel Szulc, @EncodePanda, paul.szulc@gmail.com 2

NANCY LYNCH. A hundred impossiblility proofs for
distributed computing. In Proceedings of the Eighth
Annual ACM Symposium on Principles of Distributed
Computing, Edmonton, Alberta, Canada, August 1989
podc89.pdf
"This talk is about impossibility results in the area of
distributed computing. I managed to count over 100 such
impossibility proofs! And my search wasn’t even very
systematic or exhaustive."

ConsensusThe problem of reaching agreement among remote
processes

"The problem of reaching agreement among remote
processes is fundamental problems in distributed
computing."1
1
MICHAEL J. FISCHER, NANCY A. LYNCH, MICHAEL S. PATERSON. Impossibility of distributed consensus with one faulty
process. Journal of the ACM, 32(2):374--382, April 1985 http://groups.csail.mit.edu/tds/papers/Lynch/jacm85.pdf

"A well-known form of the problem is the “transaction
commit problem,” which arises in distributed database
systems"1
1

Replicated State Machine

LESLIE LAMPORT. "Time, Clocks, and the Ordering of
Events in a Distributed System".
https://lamport.azurewebsites.net/pubs/time-clocks.pdf

"The synchronization is specified in terms of a State
Machine, consisting of a set E of possible events, a set S
of possible states, and a function f: E×S -> S. The
relation f(E, S) -> S' means that executing the event E
with the machine in state S causes the machine state to
change to S'."2
2
LESLIE LAMPORT. Time, Clocks, and the Ordering of Events in a Distributed System. https://lamport.azurewebsites.net/
pubs/time-clocks.pdf

Node
State: S
A
B
C

Node
State: S
A
B
C
Client

Node
(S, e) -> S
A
B
C
Client

Node
State: S
A
B
C
Client

Node
State: S
A
B
C
D
Client

Node1 Node2 Node3
Events1 Events2 Events3

Node1 Node2 Node3
Events1 Events2 Events3
Client1 Client2 Client3

Node1 Node2 Node3
Events1
Aggrement1
Events2
Aggrement2
Events3
Aggrement3
Client1 Client2 Client3

"The problem of failure is a difficult one, and it is beyond
the scope of this paper to discuss it in any detail."

LESLIE LAMPORT. "The Part-Time Parliament". ACM
Transactions on Computer Systems 16, 2 (May 1998)
http://lamport.azurewebsites.net/pubs/lamport-
paxos.pdf

LESLIE LAMPORT. "The Part-Time Parliament". ACM
Transactions on Computer Systems 16, 2 (May 1998)
http://lamport.azurewebsites.net/pubs/lamport-
paxos.pdf
"The Paxos algorithm for implementing a fault-tolerant
distributed system has been regarded as difficult to
understand, perhaps because the original presentation
was Greek to many readers"4
4
LESLIE LAMPORT. Paxos Made Simple. ACM SIGACT News (Distributed Computing Column) 32, 4 (Whole Number 121,
December 2001) https://www.microsoft.com/en-us/research/publication/paxos-made-simple/

LESLIE LAMPORT. Paxos Made Simple. ACM SIGACT
News (Distributed Computing Column) 32, 4 (Whole
Number 121, December 2001)
https://www.microsoft.com/en-us/research/
publication/paxos-made-simple/

LESLIE LAMPORT. Paxos Made Simple. ACM SIGACT
News (Distributed Computing Column) 32, 4 (Whole
Number 121, December 2001)
https://www.microsoft.com/en-us/research/
publication/paxos-made-simple/
"At the PODC 2001 conference, I got tired of everyone
saying how difficult it was to understand the Paxos
algorithm (...) the algorithm itself is very simple. So, I
cornered a couple of people at the conference and
explained the algorithm to them orally, with no paper.

"Paxos is an algorithm that is used to achieve consensus
among a distributed set of computers that communicate
via an asynchronous network."4
4

"Assume that agents can communicate with one another
by sending messages."4
"Agents operate at arbitrary speed, may fail by stopping,
and may restart."4
"Messages can take arbitrarily long to be delivered, can
be duplicated, and can be lost, but they are not
corrupted."4
4

MICHAEL J. FISCHER, NANCY A. LYNCH, MICHAEL S.
PATERSON. Impossibility of distributed consensus
with one faulty process. Journal of the ACM, 32(2):
374--382, April 1985
jacm85.pdf

"In this paper, we show the surprising result that no
completely asynchronous consensus protocol can
tolerate even a single unannounced process death. We do
not consider Byzantine failures, and we assume that the
message system is reliable - it delivers all messages
correctly and exactly once. Nevertheless, even with these
assumptions, the stopping of a single process at an
inopportune time can cause any distributed commit
protocol to fail to reach agreement." 1
1

"Paxos is an algorithm that is used to achieve consensus
among a distributed set of computers that communicate
via an asynchronous network."1
1

"Assume that agents can communicate with one another
by sending messages."4
"Agents operate at arbitrary speed, may fail by stopping,
and may restart."4
"Messages can take arbitrarily long to be delivered, can
be duplicated, and can be lost, but they are not
corrupted."4
4

"The consensus problem involves an asynchronous
system of processes, some of which may be unreliable. The
problem is for the reliable processes to agree on a binary
value. In this paper, it is shown that every protocol for
this problem has the possibility of nontermination, even
with only one faulty process." 1
1

"Our impossibility result applies to even a very weak
form of the consensus problem. (...) For the purpose of the
impossibility proof, we require only that some process
eventually make a decision."[^1 ]

Agenda
MICHAEL J. FISCHER, NANCY A. LYNCH, MICHAEL S.
PATERSON. Impossibility of distributed consensus
with one faulty process. Journal of the ACM, 32(2):
374--382, April 1985
1. Detailed explanation of the theorem
2. How that applies to real world?

The Consensus
Our impossibility result applies to even a very weak
form of the consensus problem.
— Assume that every process starts with an initial
value in (0,1). A nonfaulty process decides on a
value in (0, 1) by entering an appropriate decision
state.
— For the purpose of the impossibility proof, we require
only that some process eventually make a decision.

The Consensus
Our impossibility result applies to even a very weak
form of the consensus problem.
— "All nonfaulty processes that make a decision are
required to choose the same value."
— "The trivial solution in which, say, 0 is always chosen
is ruled out by stipulating that both 0 and 1 are
possible decision values."

The Model

Paper assumes totally asynchronous model of
computation
— "processing is completely asynchronous; that is, we
make no assumptions about the relative speeds of
processes"
— "no ability to detect the death of a process, so it is
impossible for one process to tell whether another has
died (stopped entirely) or is just running very slowly."

Paper assumes totally asynchronous model of message
delivery.
— "We make no assumptions (...) about the delay time in
delivering a message"
— "Every message is eventually delivered as long as the
destination process makes infinitely many attempts to
receive, but messages can be delayed, arbitrarily long,
and delivered out of order."

Paper assumes totally asynchronous model of message
delivery.
— "(...) and we assume that the message system is
reliable it delivers all messages correctly and exactly
once"
— "An 'atomic broadcast' capability is assumed", so a
process can send the same message in one step to
all other processes with the knowledge that if any
nonfaulty process receives the message, then all the
nonfaulty processes will.

Paper talks only about crash failures.
— "We do not consider Byzantine failures"
— "Of course, any protocol can be overwhelmed by
faults that are too frequent or too severe, so the best
that one can hope for is a protocol that is tolerant to
a prescribed number of 'expected' faults."

In this paper, we show the
surprising result that no
completely asynchronous
consensus protocol can
tolerate even a single
unannounced process death.

The Formalism

"A major obstacle to progress in the field of distributed
computation is that many of the important algorithms,
especially communications algorithms, seem to be too
complex for rigorous understanding.
Although the designers of these algorithms are often able
to convey an intuitive understanding of how their
algorithms work, it is often difficult to make this intuition
formal and precise."6
6
NANCY LYNCH, MARK TUTTLE Hierarchical Correctness Proofs for Distributed Algorithms https://groups.csail.mit.edu/
tds/papers/Tuttle/Tuttle-thesis.pdf

The Formalism
A concurrent program can be simulated by a sequential
program that produces all interleavings of its threads

The Fromalism
"Processes are modeled as automata"1
1

NANCY LYNCH, MARK TUTTLE "Hierarchical
Correctness Proofs for Distributed Algorithms"
https://groups.csail.mit.edu/tds/papers/Tuttle/Tuttle-
thesis.pdf

NANCY LYNCH, MARK TUTTLE "Hierarchical
Correctness Proofs for Distributed Algorithms"
https://groups.csail.mit.edu/tds/papers/Tuttle/Tuttle-
thesis.pdf
I/O Automaton

I/O Automaton
Automaton
Input
TransitionFunction
Output

I/O Automaton
Automaton
Input Input
TransitionFunction
Output

I/O Automaton
Automaton1
Automaton2
Input
TransitionFunction
Output InputInput
TransitionFunction
Output

"Processes are modeled as automata (with possibly
infinitely many states) that communicate by means of
messages.
In one atomic step a process can attempt to receive a
message, perform local computation on the basis of
whether or not a message was delivered to it (and if so,
which one), and send an arbitrary but finite set of
messages to other processes"1
1

We need to formalise:
1. Process
2. Network
3. Combine those as a single model

Process
xp
TransitionFunction
yp
"A consensus protocol P is an
asynchronous system of N
processes (N > 2). Each process p
has a one-bit input register x, an
output register y with values in (b,
0, 1) ,and an unbounded amount
of internal storage."1
1
MICHAEL J. FISCHER, NANCY A. LYNCH, MICHAEL S. PATERSON.
Impossibility of distributed consensus with one faulty process. Journal
of the ACM, 32(2):374--382, April 1985 http://groups.csail.mit.edu/tds/
papers/Lynch/jacm85.pdf

MessageBuffer
Send Receive
Acts
Outgoing
"Processes communicate by
sending each other messages. A
message is a pair (p, m), where p
is the name of the destination
process and m is a 'message
value'. The message system
maintains a multiset, called the
message buffer, of messages that
have been sent but not yet
delivered. It supports two abstract
operations: Send(p, m) and
1
MICHAEL J. FISCHER, NANCY A. LYNCH, MICHAEL S. PATERSON.
Impossibility of distributed consensus with one faulty process. Journal
of the ACM, 32(2):374--382, April 1985 http://groups.csail.mit.edu/tds/
papers/Lynch/jacm85.pdf

send(p, m): Places (p, m) in the messagebuffer
receive(p): Deletes some message(p, m) from the buffer
and returns m, in which casewe say (p, m) is delivered,
or returns the special null marker 0 and leavesthe buffer
unchanged.

Thus, the message system acts nondeterministically,
subject only to the condition that if receive(p) is
performed infinitely many times, then every message(p,
m) in the messagebuffer is eventually delivered. In
particular, the message system is allowed to return 0 a
finite number of times in response to receive(p), even
though a message(p, m) is present in the buffer.

Processes
P1
xp
TransitionFunction
yp
P2
xp
TransitionFunction
yp
P3
xp
TransitionFunction
yp
MessageBuffer
Send Receive
Acts
Outgoing

Internal State
"The values in the input and output registers, together with
the program counter and internal storage, comprise the
internal state. Initial states prescribe fixed starting values
for all but the input register; in particular, the output
register starts with value b."1
1

Conﬁguration
"A configuration of the system consists of the internal state
of each process, together with the contents of the message
buffer. An initial configuration is one in which each
process starts at an initial state and the message buffer is
empty."1
1

"A step takes one configuration to another. Let C be a
configuration. The step occurs in two phases. First,
receive(p) is performed on the message buffer in C to
obtain a value m. Then, depending on p’s internal state in
C and on m, p enters a new internal state and sends a
finite set of messagesto other processes. Since
processesare deterministic, the step is completely
determined by the pair e = (p, m) which we call an event"1
1

Schedule
A schedule from C is a finite or infinite sequence σ of
events that can be applied, in turn, starting from C. If σ
u is finite, we let σ(C) denote the resulting
configuration, which is said to be reachable from C.

Faulty Process
"A process p is nonfaulty in a run provided that it takes
infinitely many steps, and it is faulty otherwise. "1
1

Theorem"No consensus protocol is totally correct in spite of one
fault."1
1

Lemma 1

LEMMA 1. Suppose that from some configuration C, the
schedules σ1, σ2 lead to configurations C1, C2,
respectively. If the sets of processes taking steps in σ1
and σ2, respectively, are disjoint, then σ2 can be
applied to C1 and σ2 can be applied to C2, and both
lead to the same configuration C3.

PROOF. The result follows at once from the system
definition, since σ1 and σ2 do not interact.

C
C1 C2
C3
σ1
σ2
σ2
σ1

C
C1 C2
C3
σ1
σ2
σ2
σ1
LEMMA 1. Suppose that from
some configuration C, the
schedules σ1, σ2 lead to
configurations C1, C2,
respectively. If the sets of
processes taking steps in σ1 and
σ2, respectively, are disjoint,
then σ2 can be applied to C1 and
σ2 can be applied to C2, and both
lead to the same configuration
C3.

C
some configuration C

C
C1 C2
σ1 σ2
respectively.

C
C1 C2
σ1 σ2

C
C1 C2
C3
σ1
σ2
σ2
σ1
C3.

C
P1
Idle
P2
Idle
P3
Idle
P4
Idle
C1
P1
Running
P2
Idle
P3
Running
P4
Idle
C2
P1
Idle
P2
Running
P3
Idle
P4
Running
C3
P1
Running
P2
Running
P3
Running
P4
Running
σ1
σ2
σ2
σ1

C
C1 C2
C3
σ1
σ2
σ2
σ1
C3.

Theorem"No consensus protocol is totally correct in spite of one
fault."1
1

Proof
"Assume to the contrary that P is a consensus protocol that
is totally correct in spite of one fault. We prove a sequence
of lemmas which eventually lead to a contradiction."1
1

Let C be a configuration and let V be the set of decision
values of configurations reachable from C.
C is bivalent if |V| = 2. C is univalent if |V| = 1
Let us say 0-valent or 1-valent according to the
corresponding decision value.
1. Lemma 2 - P has a bivalent initial configuration.
2. Lemma 3 - From bivalent configuration there is
always at least one reachable bivalent configuration

Lemma 2

Lemma 2: P has a bivalent initial conﬁguration.
"An initial configuration is one in which each process
starts at an initial state and the message buffer is empty."
"Initial states prescribe fixed starting values for all but the
input register; in particular, the output register starts with
value b."

C(0) C(0) C(0) C(1) C(1) C(1)

C(0) C(0) C(0) C(1) C(1) C(1)
"Now consider some deciding run from CO in which
process p takes no steps, and let σ be the associated
schedule. Then σ can be applied to C1 also, and
corresponding configurations in the two runs are identical
except for the internal state of process p."

Lemma 3
Let C be a bivalent configuration of P, and let e = (p, m)
be an event that is applicable to C. Let ℂ be the set of
configurations reachable from C without applying e,
and let 𝔻 = e(ℂ), then 𝔻 contains a bivalent
configuration.

Lemma 3
Consider two neighbors configurations: C0, C1 ∈ ℂ,
such that e(C0) = D0 and e(C1) = D1
Neighbors so C1 = e’(C0) where e’ = (p’, m’).
Two cases:
1. p == p'
2. p === p'

Lemma 3, Case 1. p == p'

Safety and liveness
"Safety: Informally, an algorithm is safe if nothing bad
ever happens. (...)
Liveness: By contrast, an algorithm is live if eventually
something good happens."9
9
NANCY LYNCH AND SETH GILBERT Perspectives on the CAP Theorem https://groups.csail.mit.edu/tds/papers/Gilbert/
Brewer2.pdf

"If a system is wholly synchronous, consensus can be
solved, i.e., the trade-off between safety and liveness can
be avoided. Notably, consensus requires f + 1 rounds, if up
to f servers may crash. Since consensus is impossible in an
asynchronous system if there is even one crash failure, this
motivates the question: how much synchrony is needed to
solve consensus? And do real systems provide that
necessary level of synchrony?9
"
9
NANCY LYNCH AND SETH GILBERT Perspectives on the CAP Theorem https://groups.csail.mit.edu/tds/papers/Gilbert/
Brewer2.pdf

DOLEV, D., DWORK, C., AND STOCKMEYER, L. On the
minimal synchronism needed for distributed
consensus. J. ACM34, 1 (Jan. 1987)
https://dl.acm.org/doi/10.1145/7531.7533

"Most notably, they introduced the idea of eventual
synchrony: a system may experience some periods of
synchrony and some periods of asynchrony, but as long as
it eventually stabilizes and maintains synchrony for a
sufficiently long period of time, we can solve consensus."

CYNTHIA DWORK, LARRY STOCKMEYER AND NANCY
LYNCH Consensus in the Presence of Partial
Synchrony
https://groups.csail.mit.edu/tds/papers/Lynch/
jacm88.pdf

"It’s easy to construct a scenario in which two proposers
each keep issuing a sequence of proposals with increasing
numbers, none of which are ever chosen."4
4

Examples of successful implementations of Paxos
1. Google Chubby (https://research.google/pubs/
pub27897/)
2. Event Store (https://eventstore.com/)

NANCY LYNCH AND SETH GILBERT Perspectives on
the CAP Theorem
https://groups.csail.mit.edu/tds/papers/Gilbert/
Brewer2.pdf

So, wait. Is asynchronous model completely not
thereotically possible in real world?

Before we go...

Thank you!
@EncodePanda

Distributed Computing Impossibility Proofs

Recommandé

Recommandé

Contenu connexe

Similaire à Distributed Computing Impossibility Proofs

Similaire à Distributed Computing Impossibility Proofs (20)

Plus de Pawel Szulc

Plus de Pawel Szulc (20)

Dernier

Dernier (20)

Distributed Computing Impossibility Proofs