[rabbitmq-discuss] Queue Clustering and Sequencing

Thu Oct 27 18:59:33 BST 2011

Hi Mathias,

thanks for your reply.

> have you tried the obvious approach suggested in your first post, i.e.
> have one queue per consumer and ensure that messages for the same key
> always end up in the same queue?

Yes, that is what we currently are doing (in the single instance
environment).

> > Non-related messages are unnecessarily sequenced. When the key
> > distribution is not exactly even, one queue may have a backup while
> > another is idle.
>
> ...but is that really an issue in practice?

Yes. But a bigger problem is a slow messages. Most messages are
processed in a few hundred milliseconds. But occasionally a message
takes a few seconds to process. That delays the following, unrelaed
messages that happen to be in the same queue. The other queues would
have capacity to process these messages with minimal delay.

> Take a look athttps://github.com/rabbitmq/rabbitmq-consistent-hash-exchange

That helps a lot.

Though one problem remains. When a node fails, the messages on that
node's queues are still stuck there. The consistent-hash-exchange will
re-route future messages to a different queue, avoiding the need to
migrate the consumer. But the message already delivered to the
original queue will not be processed until the node recovers, at which
time they are out of sequence.

Roger