[rabbitmq-discuss] questions about log messages and problems with connections

[Simon MacMullen]

On 09/09/11 09:53, Andrea Rosa wrote:
> in the rabbitmq file I found this error:
> =ERROR REPORT==== 8-Sep-2011::14:13:39 ===
> exception on TCP connection<0.27709.5838>  from ipaddr:port
> {inet_error,etimedout}
>
> When this error occurs?
> Unfortunately this problem affects 70% of our nodes!

We've only seen this error before when there's a TCP load balancer or 
firewall in between Rabbit and its clients that is somehow futzing with 
TCP connections - could this be happening to you?

> Analysing the rabbitmq log file I found other strange messages:
> =WARNING REPORT==== 8-Sep-2011::14:49:08 ===
> exception on TCP connection<0.11485.6171>  from ipaddr:port
> connection_closed_abruptly
>
> This message is repeated a lot of times in the log file, but It seems
> that didn't affect the behaviour of our application.
> It's normal to have so much WARNING messages?

That just means that a connection was closed at the TCP level without 
going through the AMQP close handshaking. If you're seeing this all the 
time and it's not associated with clients getting disconnected when you 
don't expect it, it could be a badly-written client? Which client are 
you using?

> And finally these messages:
> =INFO REPORT==== 8-Sep-2011::13:13:13 ===
>      alarm_handler: {clear,file_descriptor_limit}
>
> =INFO REPORT==== 8-Sep-2011::13:13:13 ===
>      alarm_handler: {set,{file_descriptor_limit,[]}}
>
> These are related with socket problems?

Ah, that's interesting. On Unix every TCP connection requires a file 
descriptor and often operating systems are configured with a fairly low 
limit (1024 is fairly typical). And Rabbit needs to keep some file 
descriptors back for... files. If you want Rabbit to support large 
numbers of connections, look into raising the limit with limits.conf.

However, even if you're running into the limit then this shouldn't kill 
existing connections, just make Rabbit unable to accept new ones, so I 
think you still have multiple issues to fix I'm afraid.

Cheers, Simon

-- 
Simon MacMullen
RabbitMQ, VMware