Failover and takeover contingency mechanisms for network partition and node failure

Failover and Takeover Contingency Mechanisms
for Network Partition and Node Failure

Macías López, Laura M. Castro, David Cabrero

MADS Research Group – Universidade da Coruña (Spain)

Erlang Workshop
Copenhaguen, 14th September 2012

Erlang Workshop (2012) Fail/Takeover Mechanisms 1 / 25

Why are we (all) here?


Why are we (presenting this work) here?

concurrency!

high-
availability!

distribution!



Unexpected problems
after deployment!

node failures!
system failure!


Outline

1 The system

2 The problems at deployment

3 The solution

4 Final remarks


The system
ADVERTISE

Distributed system for advertisement transmission to on-customer-home
set-top boxes (STBs) over a Digital TV network (iDTV) of a cable operator


The system
ADVERTISE’s requirements

ensure the appropriate coordination of advertising mechanisms:

compilation of events

emission of advertising signals to STBs during a period of time

recording hits (displays) of a speciﬁc piece of advertisement

Major challenge
Management of the size of the communications network:

growing number of operator’s customers (∼ 100.000)


The system
ADVERTISE’s architecture


The system
ADVERTISE’s structure


The system
ADVERTISE as Erlang Distributed Application

To meet its requirements, ADVERTISE was designed

as a distributed application over several nodes


The problems at deployment
The symptoms

ADVERTISE deployment environment

presented some particularities that had not been foreseen:

some nodes showed a tendency to fail more often than others

network partition was common during some time periods (noon,

night)

In this situation. . .
Fault tolerance requirements were not met!


The diagnosis

ADVERTISE was developed and tested over several physical machines


The diagnosis

ADVERTISE was deployed over several virtual machines


The diagnosis

ADVERTISE was deployed over several virtual machines

running on a single physical machine

using a shared hard disk

sharing the network link

sharing with other apps/VMs

Frequent saturation of shared resources was perceived by ADVERTISE
nodes as short network partitions.


The consequences

If nodes lose connectivity, believe that all the others are down and assume

system functions, there are likely to be inconsistencies when connectivity

is restored (duplicated responsibilities, data inconsistencies).

Perceived network partitions led to cascade failovers
Duplicated registration of global names, random killing of conﬂicting

processes, overﬂow and eventual stop of the supervision mechanisms.


The solution

For ADVERTISE, data consistency was more important

than availability:

system could not afford that advertising campaigns, rules,

or media were lost or became inconsistent

instead, it was acceptable that no ads were sent to STBs

(or that they were delayed)

The solution
We re-designed ADVERTISE to be deployed over a minimum of 3 nodes,

and never on an isolated node


The solution
ADVERTISE initialisation


The solution
ADVERTISE boot


The solution
Node integrity check

1 Retrieve the last known population of active nodes Listactives

2 Retrieve the list of all ADVERTISE nodes from the conﬁguration Listall

3 Filter Listall removing ping-unreachable nodes

4 If

(ﬁltered (Listall ) = Listactives ) ∧ (|Listactives | = 1)

ADVERTISE is suspended immediately,

and node is rebooted once connectivity is restored


The solution
Distributed AC check

1 DAC is queried on all nodes, to get PID of ADVERTISE local sup

2 If ∃n ∈ Listall for which ADVERTISE local sup PID could not be
retrieved, node failure is assumed
1 If n ∈ Listactives it means it replies to ping from the global supervisor but
cannot reach others; after a timeout

1 If n ∈ Listactives node failure is conﬁrmed
/
2 If n ∈ Listactives node is up and we reboot it


The solution
Current ADVERTISE deployment

Cluster of 3 virtual nodes, handles an average of 18K STBs per node

with peaks of 23K STBs during prime time

Our tests reached a maximum of 45K STBs per node

System running with no incidents reported in the last 4 months

Most intensive advertising campaign was a 2-month promotion:

displayed over 66 million times, with a peak of 140K times in 1 hour

Average campaign can be displayed a total of 500K, with peaks of up

to 30K in 1 hour during prime time Saturday night


Final remarks
Lessons learned

When designing a distributed Erlang app, one must take into account:

Network security
Network reliability
Network topology
Latency of requests
Heterogeneity of components
Bandwidth
Scalability


Final remarks
Your mileage may vary!

Had ADVERTISE requirements been substantially different

we would probably have favoured

availability over consistency, for instance

And that would be a different story. . .


Questions?

Audience ! thanks

Some images and icons were downloaded from: openclipart.org


Failover and takeover contingency mechanisms for network partition and node failure

Recommandé

Recommandé

Contenu connexe

Similaire à Failover and takeover contingency mechanisms for network partition and node failure

Similaire à Failover and takeover contingency mechanisms for network partition and node failure (20)

Plus de Laura M. Castro

Plus de Laura M. Castro (20)

Dernier

Dernier (20)

Failover and takeover contingency mechanisms for network partition and node failure