[rabbitmq-discuss] RabbitMQ crashed again

Wed Mar 31 23:37:40 BST 2010

Thanks for confirming, Michael.  Is it safe to assume that closing a connection will deallocate any associated resources?  I have the example for how to increase the number of processes, so I can increase that and make sure that the connections get reopened on a regular basis, which should hopefully keep whatever is causing this slow resource from crashing the server.

Any thoughts on the behavior I mentioned where the server seemed to continue to accept messages?  I don't see that behavior if I manually stop the server, just in the case of this crash - does that seem at all possible?  I could try to reproduce it locally if that would help.

Thanks,
Ian

On Mar 31, 2010, at 5:11 PM, Michael Bridgen wrote:

> Hi Ian,
> 
> > Hello all, I'm trying to track down the cause of another RabbitMQ
> > crash that happened to me last night.  The server was clearly no
> > longer working correctly (my consumers stopped receiving messages)
> > but the main producer seemed to be able to continue to publish
> > messages.  On a dev machine I started the producer and then stopped
> > the server entirely, and it stopped producing as expected, but the
> > live producer seemed to think the server was still working, and
> > happily published messages to the broken rabbitmq instance, which is
> > extremely troubling.
> >
> > This does not seem to be caused by the same issue that caused my
> > previous crash (hitting channel limits) - the crash log is different,
> > but still has a reference to a system limit being reached.  RabbitMQ
> > seems very stable for others, so I'm guessing there's some issue in
> > my code base causing this.  Since I disabled the code that was
> > causing all the extra channels to be created, I'm at a bit of a loss
> > trying to understand what is causing these crashes and how I can
> > prevent them - this is starting to become a huge problem.
> 
> As you deduced, it looks like you're running into a limit on the number of processes.  This is possible with some usages, and there's examples of how to increase process limits not long ago on the list which may be worth trying out.
> 
> For troubleshooting your application, you could keep an eye on the things that use processes -- principally connections, channels and queues -- by using rabbitmqctl with 'list_connections', 'list_channels' (new in 1.7.latest), or 'list_queues'.  ("watch" could be your friend here; e.g., "watch -n4 rabbitmqctl list_channels".)
> 
> My first suspicion is that channels are somehow remaining open.  If the problem remains mysterious, it'll be a help if you can post (ideally minimal!) app code.  Happy to look at it.
> 
> Please do let us know if this helps, and how you get on.
> 
> Regards,
> Michael