Gen AI in Business - Global Trends Report 2024.pdf
Pride and zombie processes, or how not to build a message queue system
1. Pride and zombie processes
Or, how not to develop a message queue system
2. It is a truth universally acknowledged, that
components of a distributed system in
possession of data must be in want of a method
of sharing it.
3. Mama, we have an engagement...
Messaging architectures are generally push; it
is often more efficient.
Data eating zombies like a byte
4. Who should we invite to the dance?
What messages are being sent?
Where are your firewalls and what ports are
open?
What messaging systems do you have?
5. And pray, how should we converse?
Do you have one or many protocols?
If the latter, can you separate the networks?
Can you forward the messages?
6. Issues
Smaller messages are more effective than larger ones
Brokers may not know that consumers are not alive and send
data to them.
Protocols do not talk to each other – without writing code
Network forwarding can be used – at a speed cost
Are relevant firewalls and ports open across the network?
Use messaging gateway pattern rather than direct connections
7. Some references
Zero MQ (http://zeromq.org/)
ZeroMQ Guide - http://zguide.zeromq.org/page:all
RabbitMQ (http://www.rabbitmq.com/)
RabbitMQ in Action - http://www.manning.com/videla/
ActiveMQ (http://activemq.apache.org/)
ActiveMQ in Action - http://www.manning.com/snyder/
Push is good. Might be some cases where push architectures cannot work but then you should re-think using a broker. Or rethink the type of architecture.
Don’t. Ever. Poll. A. Broker.
Remember, zombie processes will appear to be there but might accidentally be sent data from the broker.
What data are you sending? Updates, status control? Great…
File chunks. Can be done, but there are probably more efficient ways of sending the chunks.
What are you allowed to open? Do you have all the firewalls open? Are you sure? Really positive? All the way through?
Do you know what you are sending to? Are you sending to a gateway that handles the implementation or do you have to write a producer/consumer class?
If the latter, how are you going to deal with changing systems?
There is no standard protocol, or rather there are many.
RabbitMQ has plugins to deal with most but ActiveMQ does not support AMQP < 1. ZeroMQ can be plugged into RabbitMQ but not STOMP.
The waltz and the salsa do not mix. They need translation through shovels, bridges, or Camels. Or a very good dance teacher.