[rabbitmq-discuss] Queue Clustering and Sequencing

[Roger Fischer]

Hello, 

 

I am curious if RabbitMQ addresses the following use case.

 

A single queue. Persistent.

 

A producer creates messages fast (average 10 msg/sec).

 

Consumers process messages slow (average 1 msg/sec). Thus 10 consumers
are required to keep up.

 

Challenge 1: Some messages (< 1%) are related and need to be processed
in order. If two dependent messages are less than 10 messages apart,
they are processed in parallel. That's a problem.

 

Obvious solution: create 10 queues, each with a different key, and each
with only one consumer. But:

* Non-related messages are unnecessarily sequenced. When the key
distribution is not exactly even, one queue may have a backup while
another is idle.

* Complex to configure for differently sized systems.

 

Challenge 1+: earlier message could fail processing and be re-queued
with a delay. Obvious solution doesn't fix that.

 

Desired solution: Queue knows what messages are being processed /
re-queued and holds messages with the same key.

 

But that's not all. Consumers are clustered, with load balancing
(scalability) and failover (availability).

 

Challenge 2a: related messages must still be processed by the same
consumer (and thus loadbalanced to the same node).

 

Challenge 2b: related messages must be processed in order across node
failures.

 

Obvious solution: Solution to 1 and migrate file store and consumer from
failed node to another node. But:

* Need external system to manage migration.

* Very complex to configure for differently sized systems (different
workloads, different node sizes and different cluster sizes).

 

Desired solution: Solution 1 queues, and coordinate across nodes what
keys have messages being processed or re-queued. Will affect throughput
(we need only 250 msg/sec).

 

Thanks...

 

Roger

 

PS: Please copy the response to my email (roger dot fischer at abb dot
Ventyx dot com) as I am not a subscriber of the list.

 



Mindshare 2012 | May 8-10 | Orlando, FL
More details coming soon. 

Dates and location for European Mindshare coming soon.

DISCLAIMER:
**************************************************************************** 
This email message and all attachments transmitted with it are for the sole 
use of the intended recipient(s) and may contain confidential and privileged 
information. Please DO NOT forward this email outside of the recipient's 
Company unless expressly authorized to do so herein. Any unauthorized review, 
use, disclosure or distribution is prohibited. If you are not the intended 
recipient, please contact the sender by reply email and destroy all copies 
of the original message.

Any views expressed in this email message are those of the individual sender 
except where the sender specifically states them to be the views of Ventyx.
****************************************************************************

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20111024/891cd7ad/attachment.htm>