[rabbitmq-discuss] Queue Clustering and Sequencing

[Simon MacMullen]

It's not completely obvious what you're asking. Are you saying that only 
(all?) messages with the same key are dependent on each other or is it 
something else? Are the messages just in dependent pairs or are they 
streams?

Also, please subscribe to the list if you want to ask questions here.

Cheers, Simon

On 24/10/11 21:05, Roger Fischer wrote:
> Hello,
>
> I am curious if RabbitMQ addresses the following use case.
>
> A single queue. Persistent.
>
> A producer creates messages fast (average 10 msg/sec).
>
> Consumers process messages slow (average 1 msg/sec). Thus 10 consumers
> are required to keep up.
>
> Challenge 1: Some messages (< 1%) are related and need to be processed
> in order. If two dependent messages are less than 10 messages apart,
> they are processed in parallel. That’s a problem.
>
> Obvious solution: create 10 queues, each with a different key, and each
> with only one consumer. But:
>
> * Non-related messages are unnecessarily sequenced. When the key
> distribution is not exactly even, one queue may have a backup while
> another is idle.
>
> * Complex to configure for differently sized systems.
>
> Challenge 1+: earlier message could fail processing and be re-queued
> with a delay. Obvious solution doesn’t fix that.
>
> Desired solution: Queue knows what messages are being processed /
> re-queued and holds messages with the same key.
>
> But that’s not all. Consumers are clustered, with load balancing
> (scalability) and failover (availability).
>
> Challenge 2a: related messages must still be processed by the same
> consumer (and thus loadbalanced to the same node).
>
> Challenge 2b: related messages must be processed in order across node
> failures.
>
> Obvious solution: Solution to 1 and migrate file store and consumer from
> failed node to another node. But:
>
> * Need external system to manage migration.
>
> * Very complex to configure for differently sized systems (different
> workloads, different node sizes and different cluster sizes).
>
> Desired solution: Solution 1 queues, and coordinate across nodes what
> keys have messages being processed or re-queued. Will affect throughput
> (we need only 250 msg/sec).
>
> Thanks…
>
> Roger
>
> PS: Please copy the response to my email (roger dot fischer at abb dot
> Ventyx dot com) as I am not a subscriber of the list.
>
>
>
>
> *Mindshare 2012 | May 8-10 | Orlando, FL *
> *More details coming soon. *
>
> *Dates and location for European Mindshare coming soon. *
>
> DISCLAIMER:
> ------------------------------------------------------------------------
> This email message and all attachments transmitted with it are for the
> sole use of the intended recipient(s) and may contain confidential and
> privileged information. Please DO NOT forward this email outside of the
> recipient's Company unless expressly authorized to do so herein. Any
> unauthorized review, use, disclosure or distribution is prohibited. If
> you are not the intended recipient, please contact the sender by reply
> email and destroy all copies of the original message.
>
> Any views expressed in this email message are those of the individual
> sender except where the sender specifically states them to be the views
> of Ventyx.
>
> ------------------------------------------------------------------------
>
>
>
> _______________________________________________
> rabbitmq-discuss mailing list
> rabbitmq-discuss at lists.rabbitmq.com
> https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss


-- 
Simon MacMullen
RabbitMQ, VMware