[rabbitmq-discuss] rabbitmq dying

Wed Jun 18 16:23:20 BST 2008

Hi Ben,

Thanks for the quick response. If you have a patch it would be great
because we're going live with rabbitmq in a few weeks. Unfortunately this
problem never showed up during my early tests and is only showing up now
that we're hitting it quite heavily in our dev environment.

Some of our code can produce 50,000 messages in a few minutes and it can
take half an hour to process them. During that time I guess other processes
could also be producing large amounts of messages. Each message can be up
to a few KB in size so it can be quite a bit of data.

I can't give you a simple test case now but I might be able to put one
together. RabbitMQ only dies about every week so it's hard to reproduce.
They way our code works is that a server sends up to 50,000 jobs (messages)
to a job queue. There are 40 consumers that read the jobs and process them
and send the results back to a temporary reply queue. So, no exchanges and
just 1 queue. If several people are using the same instance of RabbitMq,
which might happen, there might be a few queues but no more than 8 or so.

You're right about the clients, when we restart them we do it through a
kill -9 so they don't disconnect gracefully. It may seem strange but the
clients are stateless and lightweight and we've always killed them through
a kill -9.

The RabbitMQ process doesn't actually die. I'm nearly positive about that.
I checked before I did a restart and 'beam' was using lots of RAM and a
little CPU but nothing could connect.

Versions:
RabbitMQ 1.3.0-1
Ubuntu 8.04 amd64 with 4GB RAM
Erlang 1:11.b.5dfsg-11
OpenJDK 1.6.0_06-b02 64-bit

Thanks,

Dave

             "Ben Hood"                                                    
             <0x6e6562 at gmail.c                                             
             om>                                                        To 
             Sent by:                  rabbitmq-discuss at lists.rabbitmq.com 
             rabbitmq-discuss-                                          cc 
             bounces at lists.rab                                             
             bitmq.com                                             Subject 
                                       Re: [rabbitmq-discuss] rabbitmq     
                                       dying                               
             18/06/2008 15:49                                              

Dave,

I've had a quick look at the log file, and it seems that your clients
are dying, which in turn is badly handled in the broker.

This error

Error in process <0.7321.2> on node 'rabbit at vsdlbblue01' with exit
value: {badarg,[{erlang,port_command,[#Port<0.27825>,[<<7 bytes>>,<<36
bytes>>,<<1
byte>>]]},{rabbit_writer,internal_send_command_async,3},{rabbit_writer,handle_message,2},{rabbit_writer,mainloop,1}]}

suggests that the opposing peer is no longer there and causes a follow
on error message:

Error in process <0.30205.1> on node 'rabbit at vsdlbblue01' with exit
value:
{{badmatch,{error,[{exit,{timeout,{gen_server,call,[<0.30206.1>,{notify_down,<0.30204.1>}]}}}]}},[{rabbit_channel,terminate,2},{buffering_proxy,mainloop,4}]}

which is a symptom of the first error not being handled correctly.

There is a bug for this already and will be fixed very soon, please
let us know what the urgency on this is, because we could get a patch
out quicker if necessary.

This is not the complete answer though, which we'll look into, but I
just wanted to give some feedback as soon as possible.

A few questions to help us diagnose this:

- What version of Rabbit are you using?
- Does the Rabbit process actually die or just the TCP listener?

Thanks,

Ben

On Wed, Jun 18, 2008 at 3:23 PM,  <David.Corcoran at edftrading.com> wrote:
>
> Hi,
>
> Recently rabbitmq has been dying on us and I think I've found the
problem.
> What usually happens is that the clients timeout and disconnect (they
have
> a 3 second heartbeat) and reconnecting doesn't work. We get a
> "java.net.ConnectException: Connection refused" exception. The 'beam'
task
> is also currently using 6% CPU and about 2GB of RAM.
>
> The errors look like:
> Mnesia(rabbit at vsdlbblue01): ** WARNING ** Mnesia is overloaded:
{dump_log,
>
> time_threshold}
>
> Mnesia(rabbit at vsdlbblue01): ** WARNING ** Mnesia is overloaded:
{mnesia_tm,
>
> message_queue_len,
>
> [705,850]}
>
> error on TCP connection from 10.80.12.26:47327
> {timeout,{frame_payload,3,1,29421}}
>
> etc.
>
> It looks like we might be leaving messages lying around? If I'm correct
is
> there a way of seeing the queues and which have lots of messages? I've
> attached the last few hours of the log file in case that helps.
>
> Thanks,
>
> Dave
>
> (See attached file: rabbit.zip)
>
> *********************************************************************
> This communication contains confidential information, some or all of
which may be privileged. It is for the intended recipient only and others
must not disclose, distribute, copy, print or rely on this communication.
If an addressing or transmission error has misdirected this communication,
please notify the sender by replying to this e-mail and then delete the
e-mail. E-mail sent to EDF Trading may be monitored by the company. Thank
you.
> EDF Trading Limited
> 80 Victoria Street, 3rd Floor, Cardinal Place, London, SW1E 5JL
> A Company registered in England No. 4255974.
> Switchboard: 020 7061 4000
> EDF Trading Markets Limited is a member of the EDF Trading Limited Group
and is authorised and regulated by the Financial Services Authority.
> VAT number: GB 735 5479 07
> *********************************************************************
> _______________________________________________
> rabbitmq-discuss mailing list
> rabbitmq-discuss at lists.rabbitmq.com
> http://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
>
>

_______________________________________________
rabbitmq-discuss mailing list
rabbitmq-discuss at lists.rabbitmq.com
http://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss

*********************************************************************
This communication contains confidential information, some or all of which may be privileged. It is for the intended recipient only and others must not disclose, distribute, copy, print or rely on this communication. If an addressing or transmission error has misdirected this communication, please notify the sender by replying to this e-mail and then delete the e-mail. E-mail sent to EDF Trading may be monitored by the company. Thank you. 
EDF Trading Limited
80 Victoria Street, 3rd Floor, Cardinal Place, London, SW1E 5JL
A Company registered in England No. 4255974. 
Switchboard: 020 7061 4000
EDF Trading Markets Limited is a member of the EDF Trading Limited Group and is authorised and regulated by the Financial Services Authority.
VAT number: GB 735 5479 07
*********************************************************************