[rabbitmq-discuss] Binding to multiple queues in a round-robin style using the PIKA library

[Emile Joubert]

Hi Stephen,

On 03/02/12 15:49, Stephen Young wrote:
> Q1...Qn are the queues I have created
> Each consumer binds to all queues, and has the capability of parsing
> and using data from each queue.
> 
> A request is pushed through a broker and generates a set of queue
> entries, E1...En
> Ej must be processed for Ej+1...En to be processed, the rest are
> independent of the result of Ej.
> Bundling requests together causes the system to be significantly
> slower, as it doesn't capitalize on the heavy parallelism of running
> E1..Ej simultaneously, and once Ej is processed, Ej+1...En
> simultaneously.

This is just a suggestion and might not be possible, but why not treat
Ej completely separately from other entries on the publishing side?
Process Ej perhaps using an RPC call and include the result in all
messages Ej+1..En. Or make the result of Ej available to consumers in
some other way if it is large. You should obtain good parallelism that way.

> When I don't have these consumers, or have a single consumer managing
> the whole system, what seems to be happening is that the consumer will
> stop on one of the queues Qj+m, and read a request, determine that it
> cant complete, send basic_reject, and then read again from the same
> queue. This is not problematic all the time, but occasionally
> generates a loop on that queue by the consumer. What i'd like to do is
> have the consumer act like:
> 
> read queue Qi
> determine blockage, send basic_reject
> read queue Qk
> process, send basic_ack

I'm not familiar with Pika, but can you unsubscribe from Qi before the
basic_reject?


-Emile