[rabbitmq-discuss] Confirm consistent hash exchange behavior

[Alvaro Videla]

Hi,

I think this solution can address your problem:

- Create queues 1, 2, .... N
- Bind queues to the CHX
- publish the queue names to a queue called "lock" for example.

Start consumer A, and make it consume _just one message_ from queue "lock",
in ack mode. That message will contain the queue name from where that
consumer needs to get messages.

Then you can start consumer B and do the same, so you get the name of the
next queue where to get the messages.

Because you are using the CH exchange, messages witch certain routing key
will always end up in the same queue. Therefore you can also process
messages in order.

If you need to stop one of those consumers, they need to
basic_reject(requeue=true) the message they first got from the "lock" queue
so another consumer can use that queue. If one of those consumers crashes
then the message with the queue name will get requeued. NOTE: you never ack
any message from the "lock" queue.

Does this help?

Regards,

Alvaro



On Fri, Mar 15, 2013 at 4:46 PM, SteveO <steveolivier22 at gmail.com> wrote:

> Subsequent bindings would have no affect on routing keys that have already
> been seen. When a message with a new routing key arrives, new bindings (or
> even existing ones) could be used for the new routing key.
>
> Once I looked beyond the specific requirements of our application, and
> thinking about distribution, I completely realize the behavior I seek
> breaks
> the nature of the CHX. I understand why it's not doing what I want. So I
> guess what I am looking at is some crazy feature or a completely different
> exchange altogether.
>
> Here are the high level requirements:
> 1. Need to scale
> 2. Need to process messages with the same routing key in order
>
> Here is what is highly desired:
> 1. Elastic scaling of queues
> 2. No back-channel consumer/producer communication
>
> A single queue with multiple consumers gives me req1, but not req2. A
> single
> queue with a single consumer gives me req2, but not req1. By way of a CHX,
> multiple queues and multiple consumers gets me both req1 and req2. However
> with the CHX, if I accomplish scaling up by adding another queue and
> consumer, I can run into situations where I can't guarantee req2. That's
> because I can get messages with the same routing key in more than one
> queue.
> For an alternative method of scale, I can accomplish req1 and req2 by
> adding
> another CHX. However, with our application that would require some
> back-channel communication between producers and consumers which is highly
> undesirable.
>
> A type of "pick and stick" method could work well for us. Our application
> is
> processing messages for events that span a finite duration of time. These
> events could last a few minutes, a few hours or a few days. Messages
> pertaining to a specific event are signified by a GUID in the routing key.
> Once the event is done, messages are consumed and processed, the routing
> key
> for that event is never seen again. I just can't trust the timeliness of my
> consumers, and I have to guarantee message processing order. That's why I
> have to ensure that messages with the same routing key only exist in one
> queue at a time.
>
> I suspect I could have discussed this situation much better if had simply
> stated that rather than questioning the CHX : )
>
> Is this a crazy feature request for a switch to turn this on in the CHX?
> Am I looking at a custom exchange?
> Something else?
>
> Thanks again.
>
> s
>
>
>
> --
> View this message in context:
> http://rabbitmq.1065348.n5.nabble.com/Confirm-consistent-hash-exchange-behavior-tp25458p25471.html
> Sent from the RabbitMQ mailing list archive at Nabble.com.
> _______________________________________________
> rabbitmq-discuss mailing list
> rabbitmq-discuss at lists.rabbitmq.com
> https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20130315/57182ffe/attachment.htm>