[rabbitmq-discuss] Rabbitmq With Celery Problem, Error 541

[Mike Grouchy]

On Fri, Nov 18, 2011 at 1:26 PM, Ask Solem <ask at celeryproject.org> wrote:
> Maybe you should try enabling the pool.  Either way it should retry publishing the result, so maybe this is yet another corner case in amqplib.  It would be great if you could try to create a small test case that is able to reproduce the issue.
>
> --
> Ask Solem
>


Okay,

So I enabled the pool set it to 12 (Seemed like a reasonable number
for the BROKER_POOL_LIMIT). So I ran the task the first time and it
seemed like it ran okay. I just got back from lunch and tried to run
it again to see if maybe enabling the pool fixed the problem, but now
we seem to have some more datapoints here.

I have attached the celery log output (from --INFO ) and the rabbitmq.log

https://gist.github.com/81e6f762410e6e4813aa

So it seems like we are running into a too many processes error, but I
thought the upper bound was around 30K, which I can't imagine I would
be using (even though, there are around 60k tasks).

I can't seem to replicate it consistently, which is making it hard to
narrow down to a specific test case.

Any insight here?

Thanks,
Mike





> On Nov 18, 2011, at 1:15 PM, Mike Grouchy <mgrouchy at gmail.com> wrote:
>
>>
>>
>> On 2011-11-18, at 8:09 AM, Ask Solem <ask at rabbitmq.com> wrote:
>>
>>>
>>> On 18 Nov 2011, at 01:27, Mike Grouchy wrote:
>>>
>>>> Hi There,
>>>>
>>>> I'm having this issue where I have a script that runs on a cronjob
>>>> that runs a rather large amount of tasks against celery(around 60K or
>>>> so tasks). It seems like after I get through a large portion of the
>>>> tasks I start getting error 541's returned from the celery workers
>>>> which seems to be bubbling up from rabbitmq.
>>>>
>>>> Here is the relevant portion of the log https://gist.github.com/c92d42929e514cbb69fa
>>>>
>>>> The weird thing is I am not seeing any error in the rabbitmq logs, but
>>>> as far as I know the error is coming from rabbitmq.
>>>>
>>>> Can anyone point me in the direction on what I can do to prevent this
>>>> from continuing to happen?
>>>>
>>>>
>>>
>>> Seems to me like you have lost the connection to RabbitMQ while trying
>>> to publish task results.
>>>
>>> Are you using BROKER_POOL_LIMIT?
>>
>> No. Currently just the defaults.
>
>