[rabbitmq-discuss] Rabbitmq With Celery Problem, Error 541

[Mike Grouchy]

On Tue, Nov 22, 2011 at 11:14 AM, Ask Solem <ask at rabbitmq.com> wrote:
>
> On 18 Nov 2011, at 18:53, Mike Grouchy wrote:
>>
>>
>> Okay,
>>
>> So I enabled the pool set it to 12 (Seemed like a reasonable number
>> for the BROKER_POOL_LIMIT). So I ran the task the first time and it
>> seemed like it ran okay. I just got back from lunch and tried to run
>> it again to see if maybe enabling the pool fixed the problem, but now
>> we seem to have some more datapoints here.
>>
>> I have attached the celery log output (from --INFO ) and the rabbitmq.log
>>
>> https://gist.github.com/81e6f762410e6e4813aa
>>
>> So it seems like we are running into a too many processes error, but I
>> thought the upper bound was around 30K, which I can't imagine I would
>> be using (even though, there are around 60k tasks).
>>
>> I can't seem to replicate it consistently, which is making it hard to
>> narrow down to a specific test case.
>
>
> I guess you are running out of erlang processes for the result queues.
> The amqp result backend is rather stupid, in that it creates one queue per result.
> But this is necessary since the results should be delivered even though
> there are no consumers currently waiting for it.
>
> I'd suggest that you either switch to one of the other result backends (redis, database, memcached),
>
> or if you use the results in RPC like calls (the one to retrieve the result
> is always the same process as the one that published the task), you could
> create your own backend that creates one queue per publisher instead (of course
> this will eventually lead to problems too if the number of publishers exceed the process limit).  You could also increase the erlang process limit, the answer how to should be in
> the list archives.

Thanks, this is the idea I have been toying around with as
well(switching queues to redis), I think in the long run, it might be
the better solution. I just wasn't sure if there was something that I
had been doing particularly wrong in general.

Thanks a lot for the help and feedback.

Cheers,
Mike