[rabbitmq-discuss] list_queues times out

[Matthew Sackman]

On Thu, Jun 10, 2010 at 03:23:23PM -0400, Tyler Williams wrote:
> I've run rabbitmqctl list_queues and waited for hours with no response. I still see activity in the logs, and I can still connect to rabbit and do stuff, but I don't get any output from list_queues.

That's interesting. That suggests there's just one (or very few) queue
that's blocked or otherwise churning away which is refusing to respond
to your list_queues query. Can you come up with a recipe that reproduces
this? - there's really no what that it should sit there unresponsive for
hours!

> Ok, I'll look into this. This is not exactly the failure mode we're seeing though, because in our case, rabbit is still using cpu and disk. It even clears and resets the watermark. I just can't list the queues at all.

Yup, sounds like it's not a total memory exhaustion issue at all then,
just a temporary issue, but certainly no queue should get stuck for that
long.

> As an addendum, after this happened today, I restarted rabbitmq, and it recovered it's previous queues without error. After using it for about 5 minutes though, it died, this time with an error in the log, which I've posted here: http://pastebin.com/BAaGpJab

That's bizarre. That's a queue trying to read a message off disk and
failing to find it there. Rabbit goes to great lengths to ensure it
remains in a consistent state so this kind of thing should never happen.
Again, if you can come up with a recipe to reproduce this, that would be
great. Just a guess, but have you run out of disk space on whichever
partition Rabbit is storing messages?

Also, which revision are you running? - it's possible you've uncovered a
bug which has since been fixed anyway - the 21673 branch has move quite
a lot over the last couple of weeks so if you're not running the latest,
it's definitely worth updating.

Matthew