[rabbitmq-discuss] RabbitMQ crashed again

[Michael Bridgen]

Hi Ian,

 > Hello all, I'm trying to track down the cause of another RabbitMQ
 > crash that happened to me last night.  The server was clearly no
 > longer working correctly (my consumers stopped receiving messages)
 > but the main producer seemed to be able to continue to publish
 > messages.  On a dev machine I started the producer and then stopped
 > the server entirely, and it stopped producing as expected, but the
 > live producer seemed to think the server was still working, and
 > happily published messages to the broken rabbitmq instance, which is
 > extremely troubling.
 >
 > This does not seem to be caused by the same issue that caused my
 > previous crash (hitting channel limits) - the crash log is different,
 > but still has a reference to a system limit being reached.  RabbitMQ
 > seems very stable for others, so I'm guessing there's some issue in
 > my code base causing this.  Since I disabled the code that was
 > causing all the extra channels to be created, I'm at a bit of a loss
 > trying to understand what is causing these crashes and how I can
 > prevent them - this is starting to become a huge problem.

As you deduced, it looks like you're running into a limit on the number 
of processes.  This is possible with some usages, and there's examples 
of how to increase process limits not long ago on the list which may be 
worth trying out.

For troubleshooting your application, you could keep an eye on the 
things that use processes -- principally connections, channels and 
queues -- by using rabbitmqctl with 'list_connections', 'list_channels' 
(new in 1.7.latest), or 'list_queues'.  ("watch" could be your friend 
here; e.g., "watch -n4 rabbitmqctl list_channels".)

My first suspicion is that channels are somehow remaining open.  If the 
problem remains mysterious, it'll be a help if you can post (ideally 
minimal!) app code.  Happy to look at it.

Please do let us know if this helps, and how you get on.

Regards,
Michael