[rabbitmq-discuss] Binding to multiple queues in a round-robin style using the PIKA library

[Stephen Young]

Hey Emile,

The approach is currently set up like the following:

Q1...Qn are the queues I have created
Each consumer binds to all queues, and has the capability of parsing
and using data from each queue.

A request is pushed through a broker and generates a set of queue
entries, E1...En
Ej must be processed for Ej+1...En to be processed, the rest are
independent of the result of Ej.
Bundling requests together causes the system to be significantly
slower, as it doesn't capitalize on the heavy parallelism of running
E1..Ej simultaneously, and once Ej is processed, Ej+1...En
simultaneously.

As long as there are dedicated consumers monitoring Ej only, there is
no deadlock as the Ej entries are parsed faster
However, this tends to be a waste of resources, as Ej is not a
particularly heavy process, so those consumers are idle for most of
the time.

When I don't have these consumers, or have a single consumer managing
the whole system, what seems to be happening is that the consumer will
stop on one of the queues Qj+m, and read a request, determine that it
cant complete, send basic_reject, and then read again from the same
queue. This is not problematic all the time, but occasionally
generates a loop on that queue by the consumer. What i'd like to do is
have the consumer act like:

read queue Qi
determine blockage, send basic_reject
read queue Qk
process, send basic_ack

Instead it seems to get stuck in a loop reading Qi.

Let me know if you need any more clarification!


On Feb 3, 7:10 am, Emile Joubert <em... at rabbitmq.com> wrote:
> Hi Stephen,
>
> On 02/02/12 16:36, Stephen Young wrote:
>
>
>
>
>
>
>
>
>
> > I'm working on a system that receives a variety of different message
> > types from different source queues, like the following:
>
> > client binds to Q1, Q2, Q3
> > server posts to Q2
> > client reads request and updates database. If something goes awry, the
> > client sends a basic_reject to requeue the request
>
> > however, this seems to cause some problems in that the client will see
> > a message in Q2, but need to have parsed a message in another queue
> > first. However the clients seem to keep polling the same queue until
> > success. Is there any better way to implement this system? Currently
> > I'm defeating the deadlock by having enough clients + some dedicated
> > clients on the bottleneck queues, but would prefer if each client
> > could run autonomously and read from queues in a pattern / priority of
> > some sort.
>
> I'm reading your question as: How can I organise a workload with
> multiple clients where some messages have dependencies on each other?
>
> If possible publish all the information required to complete a task as a
> single message.
>
> Priorities can be emulated by using different queues, and consuming from
> the queues designated as higher priority more often.
>
> It's hard to give any further advice without more information about how
> messages are intended to match up and how the parsing dependency works.
> Perhaps you could offer more details.
>
> -Emile
> _______________________________________________
> rabbitmq-discuss mailing list
> rabbitmq-disc... at lists.rabbitmq.comhttps://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss