[rabbitmq-discuss] rabbitmq-server crashes hard while consuming 31GB of RAM

Wed Dec 21 12:31:09 GMT 2011

On 12/21/2011 12:44 PM, Matthias Radestock wrote:
> Muharem,
> 
> On 21/12/11 11:26, Muharem Hrnjadovic wrote:
>> On 12/21/2011 12:06 PM, Matthias Radestock wrote:
>>> Try doing that *while* it is wedged. And also send us the output of
>>> 'rabbitmqctl report'. Again, while rabbit is wedged.
>> Please see https://bugs.launchpad.net/openquake/+bug/894024/comments/8
>> and https://bugs.launchpad.net/openquake/+bug/894024/comments/9
>> respectively.
>>
>>> My hunch is that you probably have many thousands of celery result
>>> queue processes.
>> Hmm .. I can see that the celery result queues (which are all
>> auto-delete queues) are deleted after the result has been consumed.
>> Why would the respective processes still hang around?
> 
> I was wrong. You have more than 47,000 connections! And the same
> number of channels (unsurprisingly). But you "only" have about 16,000
> queues.
Wow! That means our celery workers are opening a connection per task and
not closing it. We *do* get stuck after approx. 47,000 tasks.
Thank you very much for looking into this and finding the actual
problem!

> Congratulations. This is by far the fattest rabbit I've ever seen
> (except those stuffed full of messages).
Heh :-)

> I have no idea why all these connections, channels and queues are
> there. That's a question for the celery guys.
Right. Now that you have pointed out the root cause I want to check a
couple of things before involving the celery folks.

Best regards/Mit freundlichen Grüßen

-- 
Muharem Hrnjadovic <mh at foldr3.com>
Public key id   : B2BBFCFC
Key fingerprint : A5A3 CC67 2B87 D641 103F  5602 219F 6B60 B2BB FCFC

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 900 bytes
Desc: OpenPGP digital signature
URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20111221/0234d381/attachment.pgp>