Basic of Message Queue| RabbitMQ | Python | Kafka
Hey Guys , Hope your learning going good. Thought of to be part of your learning journey so will try to help out with the concept of MQ (Message Queue). Hope you like it
What is Message Queue ?
A message queue provides a lightweight buffer which temporarily stores messages, and endpoints that allow software components to connect to the queue in order to send and receive messages. The messages are usually small, and can be things like requests, replies, error messages, or just plain information.
At the simplest level, a message queue is a way for applications and discrete components to send messages between one another in order to reliably communicate. Message queues are typically (but not always) ‘brokers’ that facilitate message passing by providing a protocol or interface which other services can access.
This interface connects producers which create messages and the consumers which then process them. Within the context of a web application, one common case is that the producer is a client application that creates messages based on interactions from the user (i.e user signing up). The consumer in that case is typically daemon processes (i.e rake tasks) that can then process the arriving messages. In other cases, various subsystems will need to communicate with each other via these messages.
Producer : A User Application that sends Messages
Consumer : Consumer is a user application that receives Message
Queue : Queue is buffer that stores messages.
In other technical words , Message Queue python library to publish and subscribe to queue with different adaptor.
Benefits of Message Queue
- Improve Web application page reload time
- Asynchronous Messaging
- Break Larger task into smaller task internally
- Improve Response Timing
- Message queues are asynchronous and reliable etc..
Different Types of MQ
- RabbitMQ is the message queue currently being used by OpenPaaS, so no migration would be necessary. It offers a good and mature community. Some problems regarding clustering have been reported, including message loss and manual reconciliation upon partition. The bad point of this solution is that it doesn’t fit the advanced management features needed by James. We selected it for further investigation, and the quality of its documentation have been a decisive factor
- Kafka is a cutting edge streaming platform. It fulfill the requested features. As it exposes a distributed log, some features of a mail queue are easier to implement. Its community is strong and mature, and it is thought to clustering as a primary concern. Replay is a core concept. However, it’s architecture is complex and involve a ZooKeeper quorum. We selected it as well.
- RocketMQ is a promising, newly born Apache project. However, despite good performances, and an impressive feature set, the community is not very mature, mostly centred around the Alibaba company. The project is still under development. So we considered despite all of its advantages choosing it would not be a wise choice.
- Artemis is the HornetQ, donated to the Apache foundation, and adopted by the ActiveMQ project. A rock-solid message queue, but sadly clustering it is hard. Some old school technologies (including XML!) are involved. Thus we decided to not investigate further.
- NSO is decentralised messaging system. The messaging patterns we want are supported, but only a hack allows to gain *durability*. Clustering is a primary citizen concept, but at the cost of features. AMQP is not supported, replay neither. We decided it was not worth paying the cost of a migration to it.
Some Points to Need to remember
- Priority: You might want to give a higher priority to your organisation email, compared to spam
- Delays: Maybe you don’t want to send too much mails at once. Maybe you need to wait a bit before re-sending an email to a remote mail server in case of errors…
- Management: Mail server administrators expects to inspect the content of the mail queue, remove the elements they want, amongst other…….
Thank you for your Support …!!!!
Python , Message Queue, RabbitMQ