[rabbitmq-discuss] queue limits

[Matthew Sackman]

On Mon, Nov 22, 2010 at 10:11:11AM +1100, Steven Taylor wrote:
> Are there any limits to how long or large a queue can get apart from disk
> space?

Yes. Unless you're using the rabbitmq-toke plugin, there is still a
(very small) per message RAM cost in the form of an index from message
guid to file+offset+size. Ultimately, you can end up with all your RAM
in this index, and thus even though all messages are on disk, you won't
be able to add more messages to a queue.

> Is there a limit to the message size?

Nope, but it must be able to fit in RAM, and it's probably risky to
attempt any message greater than one quarter of the amount of RAM
installed owing to copying of the message etc which is likely to occur.

> Is there a limit to the number of queue?

Yes. Each queue in Rabbit is an Erlang process. By default, we start up
the Erlang VM configured for a million processes, so you should be able
to get close to that, especially off a single connection and channel. If
you need to get to these sorts of numbers then you may well need to
raise the Erlang VM process limit further. This will cost more RAM, but
will allow you to create more AMQP resources.

> Rather than one bus/queue with
> many message types, I'm creating a few queue for each simple units of work +
> having only a few message types per queue.  Does anyone see a problem with
> this approach when using RabbitMQ?

No, but if you need high performance and scalability, you may run into
issues. You may want to look at the queue expiry stuff as well and
obviously judicious use of exclusive and auto-delete flags is warranted.

> I'm at that nervous stage of it looks good, but I'm not quite sure yet.  I
> guess I'm asking for experiences without having fully tested.  I'm hoping to
> avoid potential pitfalls and save some time.

If in doubt, test ;) It _will_ fail at some point - and you need to
discover where that point is and how close it is to your requirements.

Matthew