[rabbitmq-discuss] Can't Determine Cause of 541 - INTERNAL_ERROR

[Samuel Duncan]

Hi guys, we are running a few 1.7.1 servers under Linux, and I came 
across a 541 situation with an app we are developing. I was going 
through the testing hosts starting up ones that had died overnight for 
one reason or another (hardware/ network etc), and at a seemingly 
arbitrary point the server stopped allowing queues from a particular 
host to be declared/ used. An application instance uses one connection, 
four exchanges, one channel, and one queue bound to two of the 
exchanges. The queues are declared exclusive. On starting up this 
particular host, I got a 541 (INTERNAL_ERROR) at the point the queue was 
first checked for inbound messages (an outbound message had already been 
successfully sent via one of the four exchanges);

raw_msg = self.chan.basic_get(queue=self.queue, no_ack=True, 
ticket=self.access_ticket)
  File "build/bdist.linux-x86_64/egg/amqplib/client_0_8/channel.py", 
line 2032, in basic_get
  File 
"build/bdist.linux-x86_64/egg/amqplib/client_0_8/abstract_channel.py", 
line 89, in wait
  File "build/bdist.linux-x86_64/egg/amqplib/client_0_8/connection.py", 
line 218, in _wait_method
  File 
"build/bdist.linux-x86_64/egg/amqplib/client_0_8/abstract_channel.py", 
line 105, in wait
  File "build/bdist.linux-x86_64/egg/amqplib/client_0_8/connection.py", 
line 367, in _close
amqplib.client_0_8.exceptions.AMQPConnectionException: (541, 
u'INTERNAL_ERROR', (0, 0), '')


The rabbit server log shows the following. At first we thought it might 
have run out of processes, but I was subsequently able to successfully 
open another application instance on another host. It seems not to like 
this particular queue!

=INFO REPORT==== 23-Feb-2010::09:37:06 ===
accepted TCP connection on 0.0.0.0:5672 from 172.18.1.121:35050

=INFO REPORT==== 23-Feb-2010::09:37:06 ===
starting TCP connection <0.16017.82> from 172.18.1.121:35050

=ERROR REPORT==== 23-Feb-2010::09:37:06 ===
** Generic server <0.16037.82> terminating
** Last message in was {'$gen_cast',
                           {method,
                               {'queue.declare',1,
                                   <<"overwatch.control.avro105b">>,false,
                                   false,true,true,true,[]},
                               none}}
** When Server state == {ch,running,1,<0.16017.82>,<0.16026.82>,undefined,
                            none,
                            {set,0,16,16,8,80,48,
                                 {[],[],[],[],[],[],[],[],[],[],[],[],[],[],
                                  [],[]},
                                 
{{[],[],[],[],[],[],[],[],[],[],[],[],[],[],
                                   [],[]}}},
                            1,
                            {[],[]},
                            {[],[]},
                            <<"overwatch_admin">>,<<"/overwatch">>,<<>>,
                            {dict,0,16,16,8,80,48,
                                  
{[],[],[],[],[],[],[],[],[],[],[],[],[],[],
                                   [],[]},
                                  
{{[],[],[],[],[],[],[],[],[],[],[],[],[],[],
                                    [],[]}}}}
** Reason for termination ==
** {{noproc,{gen_server2,call,
                         
[<0.7426.74>,{claim_queue,<0.16017.82>},infinity]}},
    [{gen_server2,call,3},
     {rabbit_channel,'-handle_method/3-fun-5-',3},
     {rabbit_channel,handle_method,3},
     {rabbit_channel,handle_cast,2},
     {gen_server2,handle_msg,7},
     {proc_lib,init_p_do_apply,3}]}

=ERROR REPORT==== 23-Feb-2010::09:37:06 ===
connection <0.16017.82> (running), channel 1 - error:
{{noproc,{gen_server2,call,[<0.7426.74>,{claim_queue,<0.16017.82>},infinity]}},
 [{gen_server2,call,3},
  {rabbit_channel,'-handle_method/3-fun-5-',3},
  {rabbit_channel,handle_method,3},
  {rabbit_channel,handle_cast,2},
  {gen_server2,handle_msg,7},
  {proc_lib,init_p_do_apply,3}]}

=WARNING REPORT==== 23-Feb-2010::09:37:06 ===
Non-AMQP exit reason '{{noproc,
                           {gen_server2,call,
                               [<0.7426.74>,
                                {claim_queue,<0.16017.82>},
                                infinity]}},
                       [{gen_server2,call,3},
                        {rabbit_channel,'-handle_method/3-fun-5-',3},
                        {rabbit_channel,handle_method,3},
                        {rabbit_channel,handle_cast,2},
                        {gen_server2,handle_msg,7},
                        {proc_lib,init_p_do_apply,3}]}'

I tried deleting the queue which, as expected, failed because it 
shouldn't exist until the host has started up the application instance - 
so doesn't seem to be a problem with exclusivity. I have checked with 
rabbitmqctl, and the queue is definitely not hanging around. Anyone have 
any ideas about what might be causing this, leads to follow etc?

Sam