SlideShare une entreprise Scribd logo
1  sur  8
Pride and zombie processes

Or, how not to develop a message queue system
It is a truth universally acknowledged, that
components of a distributed system in
possession of data must be in want of a method
of sharing it.
Mama, we have an engagement...




Messaging architectures are generally push; it
is often more efficient.
Data eating zombies like a byte
Who should we invite to the dance?





What messages are being sent?
Where are your firewalls and what ports are
open?
What messaging systems do you have?
And pray, how should we converse?


Do you have one or many protocols?



If the latter, can you separate the networks?



Can you forward the messages?
Issues
Smaller messages are more effective than larger ones
Brokers may not know that consumers are not alive and send
data to them.
Protocols do not talk to each other – without writing code
Network forwarding can be used – at a speed cost
Are relevant firewalls and ports open across the network?
Use messaging gateway pattern rather than direct connections
Some references
Zero MQ (http://zeromq.org/)
ZeroMQ Guide - http://zguide.zeromq.org/page:all
RabbitMQ (http://www.rabbitmq.com/)
RabbitMQ in Action - http://www.manning.com/videla/
ActiveMQ (http://activemq.apache.org/)
ActiveMQ in Action - http://www.manning.com/snyder/
Any questions?
Twitter: iainemsley
Website: austgate.co.uk

Contenu connexe

Similaire à Pride and zombie processes, or how not to build a message queue system

934 Ch1 Networks
934 Ch1  Networks934 Ch1  Networks
934 Ch1 Networkstechbed
 
containing byzantine failures with control zones
containing  byzantine failures with control zonescontaining  byzantine failures with control zones
containing byzantine failures with control zonesvishnuRajan20
 
A Novel High Order Tree for Securing Key Management for Multicast Services
A Novel High Order Tree for Securing Key Management for Multicast ServicesA Novel High Order Tree for Securing Key Management for Multicast Services
A Novel High Order Tree for Securing Key Management for Multicast ServicesIOSR Journals
 
20IT703_PDS_PPT_Unit_I.ppt
20IT703_PDS_PPT_Unit_I.ppt20IT703_PDS_PPT_Unit_I.ppt
20IT703_PDS_PPT_Unit_I.pptsuganthi66742
 
Communcation systems
Communcation systemsCommuncation systems
Communcation systemsMR Z
 
Comparative Analysis and Secure ALM P2P Overlay Multicasting of Various Multi...
Comparative Analysis and Secure ALM P2P Overlay Multicasting of Various Multi...Comparative Analysis and Secure ALM P2P Overlay Multicasting of Various Multi...
Comparative Analysis and Secure ALM P2P Overlay Multicasting of Various Multi...IJERD Editor
 
Distributed Systems
Distributed SystemsDistributed Systems
Distributed Systemsmitali.ray
 
Advance Operating Systems
Advance Operating SystemsAdvance Operating Systems
Advance Operating SystemsRaghu nath
 
Content Sharing over Smartphone-Based Delay-Tolerant Networks
Content Sharing over Smartphone-Based Delay-Tolerant NetworksContent Sharing over Smartphone-Based Delay-Tolerant Networks
Content Sharing over Smartphone-Based Delay-Tolerant NetworksIJERA Editor
 
Message queue architecture
Message queue architectureMessage queue architecture
Message queue architectureMajdee Zoabi
 
Mla Format For Essays Telegraph. Online assignment writing service.
Mla Format For Essays  Telegraph. Online assignment writing service.Mla Format For Essays  Telegraph. Online assignment writing service.
Mla Format For Essays Telegraph. Online assignment writing service.Ashley Smith
 
Grid computing dis
Grid computing disGrid computing dis
Grid computing disgopishna09
 

Similaire à Pride and zombie processes, or how not to build a message queue system (20)

934 Ch1 Networks
934 Ch1  Networks934 Ch1  Networks
934 Ch1 Networks
 
containing byzantine failures with control zones
containing  byzantine failures with control zonescontaining  byzantine failures with control zones
containing byzantine failures with control zones
 
Byzantine
ByzantineByzantine
Byzantine
 
Networking Essay Example
Networking Essay ExampleNetworking Essay Example
Networking Essay Example
 
A Novel High Order Tree for Securing Key Management for Multicast Services
A Novel High Order Tree for Securing Key Management for Multicast ServicesA Novel High Order Tree for Securing Key Management for Multicast Services
A Novel High Order Tree for Securing Key Management for Multicast Services
 
20IT703_PDS_PPT_Unit_I.ppt
20IT703_PDS_PPT_Unit_I.ppt20IT703_PDS_PPT_Unit_I.ppt
20IT703_PDS_PPT_Unit_I.ppt
 
Communcation systems
Communcation systemsCommuncation systems
Communcation systems
 
Networking and internet
Networking and internetNetworking and internet
Networking and internet
 
New media features
New media featuresNew media features
New media features
 
Comparative Analysis and Secure ALM P2P Overlay Multicasting of Various Multi...
Comparative Analysis and Secure ALM P2P Overlay Multicasting of Various Multi...Comparative Analysis and Secure ALM P2P Overlay Multicasting of Various Multi...
Comparative Analysis and Secure ALM P2P Overlay Multicasting of Various Multi...
 
Essay On Network Security
Essay On Network SecurityEssay On Network Security
Essay On Network Security
 
Modules1
Modules1Modules1
Modules1
 
Networks
NetworksNetworks
Networks
 
Distributed Systems
Distributed SystemsDistributed Systems
Distributed Systems
 
HHS_TOC_Glossary EMERSON EDUARDO RODRIGUES
HHS_TOC_Glossary EMERSON EDUARDO RODRIGUESHHS_TOC_Glossary EMERSON EDUARDO RODRIGUES
HHS_TOC_Glossary EMERSON EDUARDO RODRIGUES
 
Advance Operating Systems
Advance Operating SystemsAdvance Operating Systems
Advance Operating Systems
 
Content Sharing over Smartphone-Based Delay-Tolerant Networks
Content Sharing over Smartphone-Based Delay-Tolerant NetworksContent Sharing over Smartphone-Based Delay-Tolerant Networks
Content Sharing over Smartphone-Based Delay-Tolerant Networks
 
Message queue architecture
Message queue architectureMessage queue architecture
Message queue architecture
 
Mla Format For Essays Telegraph. Online assignment writing service.
Mla Format For Essays  Telegraph. Online assignment writing service.Mla Format For Essays  Telegraph. Online assignment writing service.
Mla Format For Essays Telegraph. Online assignment writing service.
 
Grid computing dis
Grid computing disGrid computing dis
Grid computing dis
 

Dernier

How AI, OpenAI, and ChatGPT impact business and software.
How AI, OpenAI, and ChatGPT impact business and software.How AI, OpenAI, and ChatGPT impact business and software.
How AI, OpenAI, and ChatGPT impact business and software.Curtis Poe
 
Scanning the Internet for External Cloud Exposures via SSL Certs
Scanning the Internet for External Cloud Exposures via SSL CertsScanning the Internet for External Cloud Exposures via SSL Certs
Scanning the Internet for External Cloud Exposures via SSL CertsRizwan Syed
 
Human Factors of XR: Using Human Factors to Design XR Systems
Human Factors of XR: Using Human Factors to Design XR SystemsHuman Factors of XR: Using Human Factors to Design XR Systems
Human Factors of XR: Using Human Factors to Design XR SystemsMark Billinghurst
 
The Ultimate Guide to Choosing WordPress Pros and Cons
The Ultimate Guide to Choosing WordPress Pros and ConsThe Ultimate Guide to Choosing WordPress Pros and Cons
The Ultimate Guide to Choosing WordPress Pros and ConsPixlogix Infotech
 
Commit 2024 - Secret Management made easy
Commit 2024 - Secret Management made easyCommit 2024 - Secret Management made easy
Commit 2024 - Secret Management made easyAlfredo García Lavilla
 
Hyperautomation and AI/ML: A Strategy for Digital Transformation Success.pdf
Hyperautomation and AI/ML: A Strategy for Digital Transformation Success.pdfHyperautomation and AI/ML: A Strategy for Digital Transformation Success.pdf
Hyperautomation and AI/ML: A Strategy for Digital Transformation Success.pdfPrecisely
 
Artificial intelligence in cctv survelliance.pptx
Artificial intelligence in cctv survelliance.pptxArtificial intelligence in cctv survelliance.pptx
Artificial intelligence in cctv survelliance.pptxhariprasad279825
 
Connect Wave/ connectwave Pitch Deck Presentation
Connect Wave/ connectwave Pitch Deck PresentationConnect Wave/ connectwave Pitch Deck Presentation
Connect Wave/ connectwave Pitch Deck PresentationSlibray Presentation
 
Designing IA for AI - Information Architecture Conference 2024
Designing IA for AI - Information Architecture Conference 2024Designing IA for AI - Information Architecture Conference 2024
Designing IA for AI - Information Architecture Conference 2024Enterprise Knowledge
 
Are Multi-Cloud and Serverless Good or Bad?
Are Multi-Cloud and Serverless Good or Bad?Are Multi-Cloud and Serverless Good or Bad?
Are Multi-Cloud and Serverless Good or Bad?Mattias Andersson
 
Nell’iperspazio con Rocket: il Framework Web di Rust!
Nell’iperspazio con Rocket: il Framework Web di Rust!Nell’iperspazio con Rocket: il Framework Web di Rust!
Nell’iperspazio con Rocket: il Framework Web di Rust!Commit University
 
DSPy a system for AI to Write Prompts and Do Fine Tuning
DSPy a system for AI to Write Prompts and Do Fine TuningDSPy a system for AI to Write Prompts and Do Fine Tuning
DSPy a system for AI to Write Prompts and Do Fine TuningLars Bell
 
Powerpoint exploring the locations used in television show Time Clash
Powerpoint exploring the locations used in television show Time ClashPowerpoint exploring the locations used in television show Time Clash
Powerpoint exploring the locations used in television show Time Clashcharlottematthew16
 
New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024
New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024
New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024BookNet Canada
 
Vertex AI Gemini Prompt Engineering Tips
Vertex AI Gemini Prompt Engineering TipsVertex AI Gemini Prompt Engineering Tips
Vertex AI Gemini Prompt Engineering TipsMiki Katsuragi
 
"ML in Production",Oleksandr Bagan
"ML in Production",Oleksandr Bagan"ML in Production",Oleksandr Bagan
"ML in Production",Oleksandr BaganFwdays
 
Dev Dives: Streamline document processing with UiPath Studio Web
Dev Dives: Streamline document processing with UiPath Studio WebDev Dives: Streamline document processing with UiPath Studio Web
Dev Dives: Streamline document processing with UiPath Studio WebUiPathCommunity
 
TeamStation AI System Report LATAM IT Salaries 2024
TeamStation AI System Report LATAM IT Salaries 2024TeamStation AI System Report LATAM IT Salaries 2024
TeamStation AI System Report LATAM IT Salaries 2024Lonnie McRorey
 
Anypoint Exchange: It’s Not Just a Repo!
Anypoint Exchange: It’s Not Just a Repo!Anypoint Exchange: It’s Not Just a Repo!
Anypoint Exchange: It’s Not Just a Repo!Manik S Magar
 
Gen AI in Business - Global Trends Report 2024.pdf
Gen AI in Business - Global Trends Report 2024.pdfGen AI in Business - Global Trends Report 2024.pdf
Gen AI in Business - Global Trends Report 2024.pdfAddepto
 

Dernier (20)

How AI, OpenAI, and ChatGPT impact business and software.
How AI, OpenAI, and ChatGPT impact business and software.How AI, OpenAI, and ChatGPT impact business and software.
How AI, OpenAI, and ChatGPT impact business and software.
 
Scanning the Internet for External Cloud Exposures via SSL Certs
Scanning the Internet for External Cloud Exposures via SSL CertsScanning the Internet for External Cloud Exposures via SSL Certs
Scanning the Internet for External Cloud Exposures via SSL Certs
 
Human Factors of XR: Using Human Factors to Design XR Systems
Human Factors of XR: Using Human Factors to Design XR SystemsHuman Factors of XR: Using Human Factors to Design XR Systems
Human Factors of XR: Using Human Factors to Design XR Systems
 
The Ultimate Guide to Choosing WordPress Pros and Cons
The Ultimate Guide to Choosing WordPress Pros and ConsThe Ultimate Guide to Choosing WordPress Pros and Cons
The Ultimate Guide to Choosing WordPress Pros and Cons
 
Commit 2024 - Secret Management made easy
Commit 2024 - Secret Management made easyCommit 2024 - Secret Management made easy
Commit 2024 - Secret Management made easy
 
Hyperautomation and AI/ML: A Strategy for Digital Transformation Success.pdf
Hyperautomation and AI/ML: A Strategy for Digital Transformation Success.pdfHyperautomation and AI/ML: A Strategy for Digital Transformation Success.pdf
Hyperautomation and AI/ML: A Strategy for Digital Transformation Success.pdf
 
Artificial intelligence in cctv survelliance.pptx
Artificial intelligence in cctv survelliance.pptxArtificial intelligence in cctv survelliance.pptx
Artificial intelligence in cctv survelliance.pptx
 
Connect Wave/ connectwave Pitch Deck Presentation
Connect Wave/ connectwave Pitch Deck PresentationConnect Wave/ connectwave Pitch Deck Presentation
Connect Wave/ connectwave Pitch Deck Presentation
 
Designing IA for AI - Information Architecture Conference 2024
Designing IA for AI - Information Architecture Conference 2024Designing IA for AI - Information Architecture Conference 2024
Designing IA for AI - Information Architecture Conference 2024
 
Are Multi-Cloud and Serverless Good or Bad?
Are Multi-Cloud and Serverless Good or Bad?Are Multi-Cloud and Serverless Good or Bad?
Are Multi-Cloud and Serverless Good or Bad?
 
Nell’iperspazio con Rocket: il Framework Web di Rust!
Nell’iperspazio con Rocket: il Framework Web di Rust!Nell’iperspazio con Rocket: il Framework Web di Rust!
Nell’iperspazio con Rocket: il Framework Web di Rust!
 
DSPy a system for AI to Write Prompts and Do Fine Tuning
DSPy a system for AI to Write Prompts and Do Fine TuningDSPy a system for AI to Write Prompts and Do Fine Tuning
DSPy a system for AI to Write Prompts and Do Fine Tuning
 
Powerpoint exploring the locations used in television show Time Clash
Powerpoint exploring the locations used in television show Time ClashPowerpoint exploring the locations used in television show Time Clash
Powerpoint exploring the locations used in television show Time Clash
 
New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024
New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024
New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024
 
Vertex AI Gemini Prompt Engineering Tips
Vertex AI Gemini Prompt Engineering TipsVertex AI Gemini Prompt Engineering Tips
Vertex AI Gemini Prompt Engineering Tips
 
"ML in Production",Oleksandr Bagan
"ML in Production",Oleksandr Bagan"ML in Production",Oleksandr Bagan
"ML in Production",Oleksandr Bagan
 
Dev Dives: Streamline document processing with UiPath Studio Web
Dev Dives: Streamline document processing with UiPath Studio WebDev Dives: Streamline document processing with UiPath Studio Web
Dev Dives: Streamline document processing with UiPath Studio Web
 
TeamStation AI System Report LATAM IT Salaries 2024
TeamStation AI System Report LATAM IT Salaries 2024TeamStation AI System Report LATAM IT Salaries 2024
TeamStation AI System Report LATAM IT Salaries 2024
 
Anypoint Exchange: It’s Not Just a Repo!
Anypoint Exchange: It’s Not Just a Repo!Anypoint Exchange: It’s Not Just a Repo!
Anypoint Exchange: It’s Not Just a Repo!
 
Gen AI in Business - Global Trends Report 2024.pdf
Gen AI in Business - Global Trends Report 2024.pdfGen AI in Business - Global Trends Report 2024.pdf
Gen AI in Business - Global Trends Report 2024.pdf
 

Pride and zombie processes, or how not to build a message queue system

  • 1. Pride and zombie processes Or, how not to develop a message queue system
  • 2. It is a truth universally acknowledged, that components of a distributed system in possession of data must be in want of a method of sharing it.
  • 3. Mama, we have an engagement...   Messaging architectures are generally push; it is often more efficient. Data eating zombies like a byte
  • 4. Who should we invite to the dance?    What messages are being sent? Where are your firewalls and what ports are open? What messaging systems do you have?
  • 5. And pray, how should we converse?  Do you have one or many protocols?  If the latter, can you separate the networks?  Can you forward the messages?
  • 6. Issues Smaller messages are more effective than larger ones Brokers may not know that consumers are not alive and send data to them. Protocols do not talk to each other – without writing code Network forwarding can be used – at a speed cost Are relevant firewalls and ports open across the network? Use messaging gateway pattern rather than direct connections
  • 7. Some references Zero MQ (http://zeromq.org/) ZeroMQ Guide - http://zguide.zeromq.org/page:all RabbitMQ (http://www.rabbitmq.com/) RabbitMQ in Action - http://www.manning.com/videla/ ActiveMQ (http://activemq.apache.org/) ActiveMQ in Action - http://www.manning.com/snyder/

Notes de l'éditeur

  1. Push is good. Might be some cases where push architectures cannot work but then you should re-think using a broker. Or rethink the type of architecture. Don’t. Ever. Poll. A. Broker. Remember, zombie processes will appear to be there but might accidentally be sent data from the broker.
  2. What data are you sending? Updates, status control? Great… File chunks. Can be done, but there are probably more efficient ways of sending the chunks. What are you allowed to open? Do you have all the firewalls open? Are you sure? Really positive? All the way through? Do you know what you are sending to? Are you sending to a gateway that handles the implementation or do you have to write a producer/consumer class? If the latter, how are you going to deal with changing systems?
  3. There is no standard protocol, or rather there are many. RabbitMQ has plugins to deal with most but ActiveMQ does not support AMQP < 1. ZeroMQ can be plugged into RabbitMQ but not STOMP. The waltz and the salsa do not mix. They need translation through shovels, bridges, or Camels. Or a very good dance teacher.