[rabbitmq-discuss] list_queues times out

[Matthew Sackman]

Hi Tyler,

On Thu, Jun 10, 2010 at 02:17:56PM -0400, Tyler Williams wrote:
> I'm having a problem where after running for a while, and with a very large number of messages (10M+, using about 4G RAM), rabbit will get into a state where it is still working, but I cannot list queues. I've been using the rabbitmq-status plugin, and this also stops working, with this message in the logs when I hit the status page:
> 
> =ERROR REPORT==== 8-Jun-2010::01:29:00 ===
> {mochiweb_socket_server,235,
>     {child_error,
>         {timeout,{gen_server2,call,[rabbit_status_web,get_context]}}}}
> 
> Can I do anything else to debug this? Is this a known issue?

That's a coding error either in rabbit_status. It should not be doing any
gen_server2:calls without setting the optional timeout to infinity.

I would advise not using rabbit_status if this issue is causing rabbit
to crash.

The rabbitmqctl correctly sets timeouts and will not crash.

There are times when rabbit can be unresponsive and can take a long time
to respond to even rabbitmqctl - on the order of several minutes. This
is easily considered a bug, and fixing this sort of issue is on our todo
list. However, unresponsiveness does not mean anything's gone wrong -
especially on the 21673 branch. Rabbit should always be able to recover
eventually and eventually move to a state where it can accept more
messages.

That said, if you do have 4GB RAM and over 10M messages, you may need to
start using the rabbitmq-toke plugin which cuts down the per-message RAM
cost to very little (in combination with the 21673 branch), by using
tokyo cabinet for a particular index table which otherwise can take a
couple of hundred bytes per entry. Basically, if you see the high
watermark set in the logs, and then a while later Rabbit reaches a point
where it is inactive (no CPU and no disk use) but the high watermark has
not been cleared, then it's likely you need to start investigating the
toke plugin.

Best wishes,

Matthew