[rabbitmq-discuss] Single vs Multiple Queues

Wed Jan 6 10:35:06 GMT 2010

Vishnu,

On Wed, Jan 6, 2010 at 10:18 AM, vishnu <pathsny at gmail.com> wrote:
> Hi Alexis
>     yeah, I see the argument about how using a single queue means I do the
> filtering at the consumer end, but as I see it, it also means I no longer
> have to figure out how to prioritise each of the tasks by providing a number
> of consumers (though I can imagine that being useful at times).

Yes and no.

You could easily write your consumers so that they scheduled
themselves across queues on a by-need basis.  For example if a
consumer finds a queue to be empty, then it tries the next queue.  And
so on.  Many strategies are possible.

> But if I
> expect all jobs to as important, but the number of messages to be variable.
> Using a single queue seems to give me the advantage that all messages are of
> equal priority and so if I get three times the number of messages of one
> type as opposed to others, that task is automatically going to get three
> times the number of consumers.

If all messages are of equal priority and you want them to be fairly
shared across all consumers, then using one queue makes sense.

You can always add more queues and more subtle behaviour later!

>  Do you have any thoughts about this?

See above :-)

> This prevents me from having to juggle numbers to optimise different queues
> with different pool size. The cost of filtering seems easier if I can make
> sure my messages carry that information. I'm trying to figure out if this
> advantage I see is deceptive? Is there a way to achieve the same goal with
> different queues? For example if I have multiple queues can I make the pool
> of consumers subscribe to all queues and pick up the first message that
> comes through no matter what? Or does this not make sense.

You can certainly have N queues and connect all M consumers to all N queues.

> Another question that might be related is the reason for the "channel"
> abstraction. I have a connection over which I create a channel and then
> register queues. What are channels for? Why would I have multiple channels
> on one connection.

As I recall, others have asked this question.  IIRC the reason was to
create logically isolated communication links ('channels') without
compelling everyone to open lots of connections.

But perhaps someone can give a better answer.

alexis

> On Wed, Jan 6, 2010 at 12:37 PM, Alexis Richardson
> <alexis.richardson at gmail.com> wrote:
>>
>> Vishnu
>>
>> If you hang multiple consumers off a single queue, then that queue's
>> state is shared and those consumers share the messages out between
>> them "round robin" style.  This is good for the 'master worker'
>> pattern for example.  Now, you can also have multiple queues doing a
>> master/worker type application.  For example you could have four
>> queues, one for 'large' tasks (messages), one for 'medium', one for
>> 'small', and one more queue that gets a copy of all messages for
>> logging and management.  Each of those queues could be private to one
>> consumer, or it could be shared among a 'pool' of consumers (eg worker
>> threads).
>>
>> In the above 'large'. 'medium', 'small' tasks example, you can load
>> balance work by size by allocating different numbers of consumers to
>> each queue.  So, if you want to filter your flows in the broker, use
>> many queues.  Using just one queue forces you to manage any filtering
>> in your thread pool.
>>
>> alexis
>>
>>
>>
>> On Wed, Jan 6, 2010 at 3:53 AM, vishnu <pathsny at gmail.com> wrote:
>> > Hi
>> >    We're looking at building a back end which picks up 3 or 4 different
>> > messages. Now we have the option of using a single queue and sending
>> > different messages through it, or multiple queues each of which is
>> > dedicated
>> > for a single kind of message.
>> > We're wondering what's more common. We're thinking of using Scala with
>> > lift
>> > amqp or the Java api to connect to RabbitMq. On one hand using multiple
>> > queues makes it seem like we're going to have to prioritise the number
>> > of
>> > threads or processes per queue which seems like we're trying to optimise
>> > that.
>> >
>> >
>> > _______________________________________________
>> > rabbitmq-discuss mailing list
>> > rabbitmq-discuss at lists.rabbitmq.com
>> > http://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
>> >
>> >
>
>
> _______________________________________________
> rabbitmq-discuss mailing list
> rabbitmq-discuss at lists.rabbitmq.com
> http://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
>
>