[rabbitmq-discuss] performance with thousands of auto_delete queues

Thu Nov 17 14:42:49 GMT 2011

On 11/17/2011 02:35 PM, Ask Solem wrote:
> 
> On 17 Nov 2011, at 12:29, Muharem Hrnjadovic wrote:
> 
>> Hello there!
>>
>> We are using rabbitmq rev. 2.3.1 with python-celery for large-scale
>> seismic calculations. Celery seems to create an auto_delete queue for
>> each task and I am observing a severe performance degradation once
>> approx. 15,000 queues have been created (this a tenth of the overall
>> calculation).
>>
>> Most of these queues have served their purpose, however, and can be
>> "garbage collected". Is there a way to request that auto_delete queues
>> be actually removed?
> 
> The queues you are seeing are the task's progress and results.
> If you don't need them you should disable them.
We do need the results.

> You can do so by upgrading to the latest version (where results are
> disabled by default), or setting @task(ignore_result=True) for
> individual tasks, or CELERY_IGNORE_RESULT=True globally.

> An auto_delete queue is only deleted when it's empty,
> so you have to collect the results.
How does one collect the results? We do

    result = TaskSet(tasks=subtasks).apply_async()

    # Wait for all subtasks to complete.
    while not result.ready():
        time.sleep(0.25)

    the_results = result.join()

Is there something we need to do beyond that?

[..]

Best regards/Mit freundlichen Grüßen

-- 
Muharem Hrnjadovic <mh at foldr3.com>
Public key id   : B2BBFCFC
Key fingerprint : A5A3 CC67 2B87 D641 103F  5602 219F 6B60 B2BB FCFC

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 900 bytes
Desc: OpenPGP digital signature
URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20111117/5c1fdc6e/attachment.pgp>