[rabbitmq-discuss] Single vs Multiple Queues

[vishnu]

Hi Alexis
    yeah, I see the argument about how using a single queue means I do the
filtering at the consumer end, but as I see it, it also means I no longer
have to figure out how to prioritise each of the tasks by providing a number
of consumers (though I can imagine that being useful at times).But if I
expect all jobs to as important, but the number of messages to be variable.
Using a single queue seems to give me the advantage that all messages are of
equal priority and so if I get three times the number of messages of one
type as opposed to others, that task is automatically going to get three
times the number of consumers. Do you have any thoughts about this?

This prevents me from having to juggle numbers to optimise different queues
with different pool size. The cost of filtering seems easier if I can make
sure my messages carry that information. I'm trying to figure out if this
advantage I see is deceptive? Is there a way to achieve the same goal with
different queues? For example if I have multiple queues can I make the pool
of consumers subscribe to all queues and pick up the first message that
comes through no matter what? Or does this not make sense.

Another question that might be related is the reason for the "channel"
abstraction. I have a connection over which I create a channel and then
register queues. What are channels for? Why would I have multiple channels
on one connection.

On Wed, Jan 6, 2010 at 12:37 PM, Alexis Richardson <
alexis.richardson at gmail.com> wrote:

> Vishnu
>
> If you hang multiple consumers off a single queue, then that queue's
> state is shared and those consumers share the messages out between
> them "round robin" style.  This is good for the 'master worker'
> pattern for example.  Now, you can also have multiple queues doing a
> master/worker type application.  For example you could have four
> queues, one for 'large' tasks (messages), one for 'medium', one for
> 'small', and one more queue that gets a copy of all messages for
> logging and management.  Each of those queues could be private to one
> consumer, or it could be shared among a 'pool' of consumers (eg worker
> threads).
>
> In the above 'large'. 'medium', 'small' tasks example, you can load
> balance work by size by allocating different numbers of consumers to
> each queue.  So, if you want to filter your flows in the broker, use
> many queues.  Using just one queue forces you to manage any filtering
> in your thread pool.
>
> alexis
>
>
>
> On Wed, Jan 6, 2010 at 3:53 AM, vishnu <pathsny at gmail.com> wrote:
> > Hi
> >    We're looking at building a back end which picks up 3 or 4 different
> > messages. Now we have the option of using a single queue and sending
> > different messages through it, or multiple queues each of which is
> dedicated
> > for a single kind of message.
> > We're wondering what's more common. We're thinking of using Scala with
> lift
> > amqp or the Java api to connect to RabbitMq. On one hand using multiple
> > queues makes it seem like we're going to have to prioritise the number of
> > threads or processes per queue which seems like we're trying to optimise
> > that.
> >
> >
> > _______________________________________________
> > rabbitmq-discuss mailing list
> > rabbitmq-discuss at lists.rabbitmq.com
> > http://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
> >
> >
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20100106/469b1b92/attachment.htm