[rabbitmq-discuss] Should requent queue creation-attachment-deletion be avoided?

[Alexis Richardson]

Davorin

You are definitely thinking along the right lines here.

* Queue creation, binding, etc, carries a small cost in time and space
* For some use cases (eg managing a lot of chat style sessions) it
would be great to have a way to 'hide' bindings temporarily, instead
of deleting them
* A mechanism akin to 'presence' has been mooted but not implemented yet

Space costs:

* The space cost is usually the more critical to considerations of
scale in our practical experience with real apps, to date
* Each binding takes a couple of 100 bytes, last time I asked the
engineering team about this, but this may have improved recently, or
be about to improve with the new persistor - you'll have to ask
Matthew
* Obviously if you have a large machine (8Gb RAM or more) this goes a
long way to helping.
* Memory consumption in RabbitMQ 1.7 is the cost of the entities plus
the bindings plus any messages in memory (messages are stored at most
once per node if they are larger than 60 bytes)
* Per message memory consumption will decrease for persistent
messages, with the new persistor
* For more on this, read
http://www.lshift.net/blog/2009/12/07/rabbitmq-at-the-skills-matter-functional-programming-exchange

Time costs:

* Direct exchange lookup is order log(n) and topic is order n.  Yes,
this could be improved.
* So if you are using topic exchanges at large scale, time is likely
to be a factor too.

Space and time:

* When you run a clustered broker, the routing table ("exchange") is
replicated across nodes.
* Replication is synchronised to the binding lifecycle
* The combination of large scale, heavy binding churn, and clustering,
is therefore not a happy one
* That said almost all problems can be made to go away with
intelligent partitioning or hashing, and reasonably big boxes

Bottom line:

* RabbitMQ can handle pretty large applications on one box provided
you do not do anything daft
* Avoid churn if you can
* The new persistor will help with some cases

alexis







On Sat, Jan 2, 2010 at 12:04 PM, Davorin Rusevljan
<davorin.rusevljan at gmail.com> wrote:
> Hi,
>
> I am thinking how to model my problem with RabbitMQ and AMQP. Let me
> describe it briefly.
>
> There are something like 1000 entities about which central site should
> publish and broadcast information. There are also something like
> 100-1000 clients which can choose only one of those items and receive
> pushed updates to its state. I was thinking about setting an exchange
> to which updates are going to be published, and each client would get
> its private queue attached to it to fetch updates from. The updates
> need not to be persisted or ack-ed.
>
> Now the clients might at some times quite frequently change the entity
> they are monitoring, maybe even once per second if they are edgy. Over
> the period of 6-10 hours that might cumulate to quite large number of
> private queues being created, attached to exchanges and deleted
> subsequently. Is this going to be costly and possible time and
> resource drain?
>
> Maybe queues need not to be deleted but they could be just reattached
> with different keyword pattern? Is that possible and if yes, would it
> also present performance and resource issue?
>
> Or maybe this should be modeled in some other way?
>
> Many thanks!
>
> Davorin Rusevljan
> http://www.cloud208.com/
>
> _______________________________________________
> rabbitmq-discuss mailing list
> rabbitmq-discuss at lists.rabbitmq.com
> http://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
>