In this talk Salvatore Sanfilippo will talk about the internals of his distributed queue Disque. The talk will cover the general architecture of the system, the delivery guarantees provided, its data replication, and finally the best-effort protocols employed in order to avoid useless multiple deliveries of the same messages.

1. Disque
A new distributed message queue
@antirez - Redis Labs

2. Redis roots
• In memory, optional persistence.
• Same protocol.
• BSD license.

3. Asynchronous jobs
execution, micro services
bus, distributed timer

4. API

5. ADDJOB queue job <timeout>

6. Disque Job IDs
DI8497c0098d456946843784d3ea41af5525c741bf05a0SQ
Node ID prefix
(32 bit)
Unique Message ID
(128 bit)
TTL in minutes
(16 bit)

7. GETJOB FROM q1 q2

8. ACKJOB id1 id2 …
Disque is all about explicit acknowledges.

9. Delivery semantics
• At least once by default.
• At most once also available.

10. ADDJOB queue job 0
RETRY 3600

11. ADDJOB queue job 0
TTL 86400

12. ADDJOB queue job 0
DELAY 3600

13. GUARANTEES

14. Synchronous replication
ADDJOB myqueue task1 REPLICATE 3

15. ADDJOB queue job 0
ASYNC

16. Persistent (optionally)
LOADJOB … data …
DELJOB … id …
Append Only File

17. Disque & CAP
• AP.
• Immutable messages (mostly).
• Converge to ACK state.
• CAP “A” availability (single node partition).

18. At least once delivery
• Liveness: eventually the message will be
delivered.
• Safety: messages not yet delivered at least
one time will never be evicted from the
cluster.
• (But if message TTL is reached).

19. At most once delivery
• Safety: messages already dequeued will never be
queued a second time.
• An immediate result of replicating to just one
node, enqueue just one time (retry time set to
zero).

20. Federation:
all nodes are really the same

21. Best effort ordering
Main Design Sacrifice

22. NACK and retries counters
• Alternative for explicit dead letters.
• Counters consistency is best effort.
• (but it does not matters).
• GETJOB exposes the two counters.

23. Disque tries hard to
avoid multiple deliveries.

24. WHY?
• Costly: think at spikes after partitions or at
CP stores to de-dup.
• No de-dup, nor idempotency, in certain
uses, if duplication rate is acceptable.
• Not so hard: worth it.

25. INTERNALS

26. Message states
ACTIVE
QUEUED
ACKED

27. ACTIVE
• Node has a copy.
• Not available for delivery.
• ACTIVE -> QUEUED (On retry timer)
• ACTIVE -> ACKED (On ACK received)

28. QUEUED
• Node has a copy.
• Will deliver via GETJOB.
• QUEUED -> ACTIVE (On delivery)
• QUEUED -> ACKED (On ACK received)

29. ACKED
• Propagate via SETACK!
• Perform Garbage Collection of message.
• ACKED -> EVICTED (on succesful GC)

30. QUEUED
ACTIVE
WILLQUEUE
QUEUED
Sent 500 ms before ACTIVE -> QUEUED
WILLQUEUE MESSAGE

31. QUEUED
ACTIVE
QUEUED MESSAGE on ACTIVE -> QUEUED state change
ACKED
QUEUED
Reset retry timer
QUEUED
Dequeue if ID1 > ID2
QUEUED
SETACK

32. KNOWN SOURCE
ANY OTHER NODE
NEEDJOBS
YOURJOBS
Exponential delay + Broadcast & ad-hoc
NEEDJOBS

33. NEEDJOBS triggers
• Clients blocked with GETJOBS
(and queues are empty)
• Queue drops to zero messages
(and import rate > 0)

34. Message owners
Each node has,
for each message,
a list* of owners
* a possibly inconsistent list

35. Ehm… some C code.
/* Job representation in memory. */
typedef struct job {
char id[JOB_ID_LEN]; /* Job ID. */
unsigned int state:4; /* Job state: one of JOB_STATE_* states. */
unsigned int gc_retry:4;/* GC attempts counter, for exponential delay. */
uint8_t flags; /* Job flags. */
uint16_t repl; /* Replication factor. */
uint32_t etime; /* Job expire time. */
uint64_t ctime; /* Job creation time in ms+counter. */
uint32_t delay; /* Delay before to queue this job for 1st time. */
uint32_t retry; /* Job re-queue time. */
uint16_t num_nacks; /* Number of NACKs this node observed. */
uint16_t num_deliv; /* Number of deliveries this node observed. */
Immutable, converging, inconsistent

36. Ehm… some C code.
robj *queue; /* Job queue name. */
sds body; /* Body, or NULL if job is just an ACK. */
dict *nodes_delivered; /* Nodes that may have a copy. */
dict *nodes_confirmed; /* Nodes that confirmed copy or ack.
mstime_t qtime; /* Next queue time */
mstime_t awakeme; /* Time at which we need to take actions. */
} job;

37. github.com/antirez/disque