[rabbitmq-discuss] performance with thousands of auto_delete queues

[Ask Solem]

On 17 Nov 2011, at 12:29, Muharem Hrnjadovic wrote:

> Hello there!
> 
> We are using rabbitmq rev. 2.3.1 with python-celery for large-scale
> seismic calculations. Celery seems to create an auto_delete queue for
> each task and I am observing a severe performance degradation once
> approx. 15,000 queues have been created (this a tenth of the overall
> calculation).
> 
> Most of these queues have served their purpose, however, and can be
> "garbage collected". Is there a way to request that auto_delete queues
> be actually removed?


The queues you are seeing are the task's progress and results.
If you don't need them you should disable them.  You can do so
by upgrading to the latest version (where results are disabled
by default), or setting @task(ignore_result=True) for individual
tasks, or CELERY_IGNORE_RESULT=True globally.


An auto_delete queue is only deleted when it's empty,
so you have to collect the results.  Another option is to enable
expiration:
   CELERY_AMQP_TASK_RESULT_EXPIRES = 300
This can either be an unsigned integer, or a datetime.timedelta
object describing the number of seconds to pass before the queue
is expired (deleted) when unused. Expiration is also enabled by default
in the latest version, but the expire time is set to 1 day, which may be
too infrequent for your use.