[rabbitmq-discuss] Erlang crashes reports

Thu Sep 16 14:19:06 BST 2010

Le 16 sept. 10 à 14:12, Emile Joubert a écrit :

Hi,
thanks for the support. The same problem happens both with the version  
2.0.0 and 2.1.0 !!
I have just performed the upgrade on our development environment, but  
it does not seem to solve this issue.

>
> Hi Romary,
>
> Thanks for this error report. Can you confirm whether the logfile
> contained anything relevant before the broker became unavailable?

Nope, the broker log does not contain anything relevant. Here is an  
extract of the log file when the broker crashes :

=INFO REPORT==== 16-Sep-2010::14:47:47 ===
Limiting to approx 65435 file handles (58889 sockets)

=INFO REPORT==== 16-Sep-2010::14:47:47 ===
Memory limit set to 1609MB.

=INFO REPORT==== 16-Sep-2010::14:47:47 ===
msg_store_transient: using rabbit_msg_store_ets_index to provide index

=INFO REPORT==== 16-Sep-2010::14:47:47 ===
msg_store_persistent: using rabbit_msg_store_ets_index to provide index

=WARNING REPORT==== 16-Sep-2010::14:47:47 ===
msg_store_persistent: rebuilding indices from scratch

=INFO REPORT==== 16-Sep-2010::14:47:47 ===
started TCP Listener on 0.0.0.0:5672

=INFO REPORT==== 16-Sep-2010::14:47:47 ===
started SSL Listener on 0.0.0.0:5671

> Do you
> override any default configuration variables in the broker config  
> file?
>

Ho yes we did :

here is the /etc/rabbitmq/rabbitmq.config

[
   {rabbit, [
      {vm_memory_high_watermark, 0.8},
      {ssl_listeners, [{"0.0.0.0",5671}]},
      {ssl_options, [{cacertfile,"/var/lib/rabbitmq/ssl/certificate- 
authority/schneider-ca.pem"},
                     {certfile,"/var/lib/rabbitmq/ssl/broker-certs/ 
broker-cert.pem"},
                     {keyfile,"/var/lib/rabbitmq/ssl/broker-certs/ 
broker-key.pem"},
                     {verify,verify_none},
                     {fail_if_no_peer_cert,false}]}
    ]}
].

and ther is the /etc/rabbitmq/rabbitmq.conf

# increase the max number of Erlang lightweight process
SERVER_ERL_ARGS="$SERVER_ERL_ARGS +P 1000000"
RABBITMQ_MULTI_ERL_ARGS="$RABBITMQ_MULTI_ERL_ARGS +P 1000000"

finaly the /etc/default/rabbitmq

# 65,635 maximum connections per rabbitmq process
ulimit -n 65535

> If you find memory usage growing with SSL then try using Erlang R14B.
> (Also see OTP-8810) What version of Erlang are you currently using?

Yes we are on our way to upgrade to Erlang R14B too !! I will let you  
know if it solves any of our troubles !

> Regards
>
> Emile
>

Thanks for all,

Romary.

>
>
>
>
> On 16/09/10 10:32, Romary Kremer wrote:
>> We are performing evaluation on rabbitmq message broker, and we
>> currently encounter difficulties with release 2.0.0:
>>
>> - Our application implies 10 000 peers producing messages  
>> periodically
>> to a unique queue. This queue is listen asynchronously by another  
>> peer.
>> - All peer are written in Java.
>> - The production rate of a single peer is 4 messages / hour.
>> - We can simulate a time-consuming task in the consumer callback,
>> simulating more or less fast consumer.
>> - we are using SSL certificate on the broker side to allow the peer  
>> to
>> authenticate the broker.
>> - we have noticed that the use of SSL as dramatic incidence on the
>> memory occupied by Rabbitmq process
>>
>> Since we upgraded to version *2.0.0*, we are no longer able to make a
>> test scenario running. The symptoms are listed bellow :
>>
>> on the broker console first, we get the message :
>>
>> Erlang has closed
>> Error: unable to connect to node rabbit at murphys: nodedown
>> diagnostics:
>> - nodes and their ports on murphys: [{rabbitmqctl22609,42767}]
>> - current node: rabbitmqctl22609 at murphys
>> - current node home dir: /var/lib/rabbitmq
>> - current node cookie hash: qu0gh1hg7j7LKyzK0GLk+A==
>>
>> we have kept the erl_crash.dump in case, but since i's about 200  
>> MB  I
>> cannot do nothing to send it to you.
>> Maybe some one can give us some hints or some indicators to look  
>> out in
>> the dump to help diagnostics, but we are not Erlang fluent !
>>
>> What we know for sure is that the crash happens while the 10 000
>> connections are established, at the beginning of the test.
>> We have monitored the  number of connections established and the  
>> crashes
>> happens always around 4500 - 5000 connections, but never the same  
>> exact
>> number.
>> We also tried with and without SSL but this does not help at all  
>> (same
>> symptoms).
>>
>> On the client side, our application registers a ShutdownListener to
>> implement a connection retry logic upon shutdown.
>> The retries always failed with the error : connection refused.
>>
>> here are some figures we gathered during the test start up about the
>> maximum number of connection established before it crashes
>>
>> - with SSL : 5404, 4493, 4399
>>
>> - without SSL : 4673
>> we dont think that the problem is about file descriptors since we
>> haven't changed anything in the configuration when we upgraded to  
>> 2.0.0.
>> The same test used to run successfully on previous version of the  
>> broker
>> (1.7.2, and 1.8.1).
>> Moreover, the rabbit_status plugin tells us we have enough file
>> descriptors as well as erlang processes
>> - file descriptors (used / available)= 34 / 65535
>> - elrang processes (used / available)= 160 / 1 000 000
>> - memory (used / available)= 40 MB / 1609 MB
>>
>> We haven't try the *2.1.0* yet because we would like to have your
>> feedback about this issue before.
>>
>> We would appreciate your feedbacks on that point before we migrate to
>> release 2.1.0.
>>
>> Best regards,
>>
>> Romary.

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20100916/e8e708fa/attachment.htm>