Hi guys,<div><br></div><div>I seem to have a problem where RabbitMQ will stop accepting channel requests after either a certain amount of messages are in the queue, or once I have sent 200-300 connection requests.</div><div>
<br></div><div>I have done this using the Tracer and without the Tracer, and experience the exact same problem each time.�I'm also not using any custom rabbitmq config.</div><div><br></div><div>At first I thought it was a memory problem, but if I restart RabbitMQ, sometimes it will start accepting connections again and allow me to put more items on the queue, but then other times it won't.</div>
<div><br></div><div>Whenever I delete the mnesia folder, it will start working again, but then after a few thousands messages (20k+) it will randomly stop again.</div><div><br></div><div>Currently, I am at 324k messages in the queue.</div>
<div><br></div><div>Does anyone have any ideas what could be causing this?</div><div><br></div><div>Cheers</div><div><br></div><div>Cal</div><div><br></div><div><br></div><div><br></div><div>System info</div><div>-----------------</div>
<div><div>�<a href="mailto:root@mail01.dmemedialtd.com">root@mail01.dmemedialtd.com</a> [/var/log/rabbitmq] > uname -a</div><div>Linux <a href="http://mail01.dmemedialtd.com">mail01.dmemedialtd.com</a> 2.6.26-2-amd64 #1 SMP Sun Jun 20 20:16:30 UTC 2010 x86_64 GNU/Linux</div>
</div><div><br></div><div><div>�<a href="mailto:root@mail01.dmemedialtd.com">root@mail01.dmemedialtd.com</a> [/var/log/rabbitmq] > cat /etc/issue</div><div>Debian GNU/Linux 5.0 \n \l</div><div><br></div></div><div><div>
top - 14:42:53 up 49 days, 15:35, �2 users, �load average: 0.00, 0.00, 0.00</div><div>Tasks: �60 total, � 1 running, �59 sleeping, � 0 stopped, � 0 zombie</div><div>Cpu0 �: �0.0%us, �0.0%sy, �0.0%ni, 99.9%id, �0.0%wa, �0.0%hi, �0.0%si, �0.0%st</div>
<div>Cpu1 �: �0.0%us, �0.0%sy, �0.0%ni, 99.9%id, �0.0%wa, �0.0%hi, �0.0%si, �0.0%st</div><div>Mem: � 1027932k total, � 766764k used, � 261168k free, � �25584k buffers</div><div>Swap: � 971892k total, � 212020k used, � 759872k free, � 288416k cached</div>
</div><div><br></div><div><div>��PID USER � � �PR �NI �VIRT �RES �SHR S %CPU %MEM � �TIME+ �COMMAND</div><div>�4682 rabbitmq �20 � 0 �610m 349m 2020 S � �0 34.8 � 0:28.47 beam.smp</div><div>�4735 rabbitmq �20 � 0 �3772 �372 �368 S � �0 �0.0 � 0:00.00 cpu_sup</div>
<div>�4764 rabbitmq �20 � 0 10492 �392 �388 S � �0 �0.0 � 0:00.00 inet_gethost</div><div>�4766 rabbitmq �20 � 0 16784 �560 �556 S � �0 �0.1 � 0:00.00 inet_gethost</div><div>14382 rabbitmq �20 � 0 11004 �940 �332 S � �0 �0.1 � 0:01.50 epmd</div>
</div><div><br></div><div><div>rabbitmq 14382 �0.0 �0.0 �11004 � 940 ? � � � �S � �Aug12 � 0:01 /usr/lib/erlang/erts-5.6.3/bin/epmd -daemon</div><div>rabbitmq �4682 �0.9 34.7 624684 357480 ? � � � Ssl �13:54 � 0:28 /usr/lib/erlang/erts-5.6.3/bin/beam.smp -W w -K true -A30 -P 1048576 -- -root /usr/lib/erlang -progname erl -- -home /var/lib/rabbitmq -pa /usr/lib/rabbitmq/lib/rabbitmq_server-1.8.1/sbin/../ebin -noshell -noinput -s rabbit -sname rabbit@mail01 -boot start_sasl -kernel inet_default_listen_options [{nodelay,true}] -kernel inet_default_connect_options [{nodelay,true}] -sasl errlog_type error -kernel error_logger {file,"/var/log/rabbitmq/rabbit@mail01.log"} -sasl sasl_error_logger {file,"/var/log/rabbitmq/rabbit@mail01-sasl.log"} -os_mon start_cpu_sup true -os_mon start_disksup false -os_mon start_memsup false -mnesia dir "/var/lib/rabbitmq/mnesia/rabbit@mail01" -noshell -noinput</div>
<div>rabbitmq �4735 �0.0 �0.0 � 3772 � 372 ? � � � �Ss � 13:54 � 0:00 �\_ /usr/lib/erlang/lib/os_mon-2.1.6/priv/bin/cpu_sup</div><div>rabbitmq �4764 �0.0 �0.0 �10492 � 392 ? � � � �Ss � 13:54 � 0:00 �\_ inet_gethost 4</div>
<div>rabbitmq �4766 �0.0 �0.0 �16784 � 560 ? � � � �S � �13:54 � 0:00 � � �\_ inet_gethost 4</div></div><div><br></div><div><br></div><div><div>�<a href="mailto:root@mail01.dmemedialtd.com">root@mail01.dmemedialtd.com</a> [~/rabbitmq-java-client-1.4.0/build/dist] > rabbitmqctl list_queues</div>
<div>Listing queues ...</div><div>po_box 324722</div><div>...done.</div></div><div><br></div><div><br></div><div><div>�<a href="mailto:root@mail01.dmemedialtd.com">root@mail01.dmemedialtd.com</a> [/var/lib/rabbitmq/mnesia/rabbit@mail01] > ls -la</div>
<div>total 162608</div><div>drwxr-xr-x 2 rabbitmq rabbitmq � � 4096 2010-08-18 14:00 .</div><div>drwxr-xr-x 3 rabbitmq rabbitmq � � 4096 2010-08-12 21:54 ..</div><div>-rw-r--r-- 1 rabbitmq rabbitmq � � �155 2010-08-18 14:00 DECISION_TAB.LOG</div>
<div>-rw-r--r-- 1 rabbitmq rabbitmq � � � 91 2010-08-18 14:00 LATEST.LOG</div><div>-rw-r--r-- 1 rabbitmq rabbitmq � � � �8 2010-08-12 21:54 rabbit_config.DCD</div><div>-rw-r--r-- 1 rabbitmq rabbitmq � � �822 2010-08-18 13:31 rabbit_durable_exchange.DCD</div>
<div>-rw-r--r-- 1 rabbitmq rabbitmq � � �198 2010-08-18 13:54 rabbit_durable_queue.DCD</div><div>-rw-r--r-- 1 rabbitmq rabbitmq � � �271 2010-08-18 13:57 rabbit_durable_queue.DCL</div><div>-rw-r--r-- 1 rabbitmq rabbitmq � � �348 2010-08-18 13:31 rabbit_durable_route.DCD</div>
<div>-rw-r--r-- 1 rabbitmq rabbitmq 83128355 2010-08-18 13:55 rabbit_persister.LOG</div><div>-rw-r--r-- 1 rabbitmq rabbitmq 83128355 2010-08-18 13:55 rabbit_persister.LOG.previous</div><div>-rw-r--r-- 1 rabbitmq rabbitmq � � � �3 2010-08-18 13:54 rabbit_serial</div>
<div>-rw-r--r-- 1 rabbitmq rabbitmq � � �127 2010-08-18 13:31 rabbit_user.DCD</div><div>-rw-r--r-- 1 rabbitmq rabbitmq � � �185 2010-08-18 13:31 rabbit_user_permission.DCD</div><div>-rw-r--r-- 1 rabbitmq rabbitmq � � �581 2010-08-18 14:00 rabbit_user_permission.DCL</div>
<div>-rw-r--r-- 1 rabbitmq rabbitmq � � �126 2010-08-18 13:31 rabbit_vhost.DCD</div><div>-rw-r--r-- 1 rabbitmq rabbitmq � �12389 2010-08-12 21:54 schema.DAT</div></div><div><br></div><div><br></div><div><br></div><div>Client Side:</div>
<div>-------------------</div><div><div>>>> conn = amqp.Connection(host="<a href="http://10.1.3.2:5673">10.1.3.2:5673</a>", userid="guest", password="guest", virtual_host="/", insist=True)</div>
<div>>>> chan = conn.channel()</div></div><div>�---- Blocks forever and a day ----</div><div>�---- Press CTRL + C ----</div><div><div>^CTraceback (most recent call last):</div><div>��File "<stdin>", line 1, in <module></div>
<div>��File "build/bdist.linux-x86_64/egg/amqplib/client_0_8/connection.py", line 230, in channel</div><div>��File "build/bdist.linux-x86_64/egg/amqplib/client_0_8/channel.py", line 82, in __init__</div>
<div>��File "build/bdist.linux-x86_64/egg/amqplib/client_0_8/channel.py", line 471, in _x_open</div><div>��File "build/bdist.linux-x86_64/egg/amqplib/client_0_8/abstract_channel.py", line 89, in wait</div>
<div>��File "build/bdist.linux-x86_64/egg/amqplib/client_0_8/connection.py", line 198, in _wait_method</div><div>��File "build/bdist.linux-x86_64/egg/amqplib/client_0_8/method_framing.py", line 212, in read_method</div>
<div>��File "build/bdist.linux-x86_64/egg/amqplib/client_0_8/method_framing.py", line 127, in _next_method</div><div>��File "build/bdist.linux-x86_64/egg/amqplib/client_0_8/transport.py", line 109, in read_frame</div>
<div>��File "build/bdist.linux-x86_64/egg/amqplib/client_0_8/transport.py", line 200, in _read</div><div>KeyboardInterrupt</div></div><div><br></div><div>Tracer proxy result:</div><div>-------------------</div><div>
<div>�<a href="mailto:root@mail01.dmemedialtd.com">root@mail01.dmemedialtd.com</a> [~/rabbitmq-java-client-1.4.0/build/dist] > ./runjava.sh com.rabbitmq.tools.Tracer</div><div>Usage: Tracer [<listenport> [<connecthost> [<connectport>]]]</div>
<div>Invoked as: Tracer 5673 localhost 5672</div><div>com.rabbitmq.tools.Tracer.WITHHOLD_INBOUND_HEARTBEATS = false</div><div>com.rabbitmq.tools.Tracer.WITHHOLD_OUTBOUND_HEARTBEATS = false</div><div>com.rabbitmq.tools.Tracer.NO_ASSEMBLE_FRAMES = false</div>
<div>com.rabbitmq.tools.Tracer.NO_DECODE_FRAMES = false</div><div><br></div><div>---- this is where the connection is started ----</div><div>1282138358084: conn#0 ch#0 <- {#method<connection.start>(version-major=8,version-minor=0,server properties={platform=Erlang/OTP, product=RabbitMQ, version=1.8.1, copyright=Copyright (C) 2007-2010 LShift Ltd., Cohesive Financial Technologies LLC., and Rabbit Technologies Ltd., information=Licensed under the MPL. �See <a href="http://www.rabbitmq.com/">http://www.rabbitmq.com/</a>},mechanisms=PLAIN AMQPLAIN,locales=en_US),null,""}</div>
<div>1282138358084: conn#0 ch#0 <- {#method<connection.tune>(channel-max=0,frame-max=131072,heartbeat=0),null,""}</div><div>1282138358086: conn#0 ch#0 -> {#method<connection.start-ok>(client-properties={library=Python amqplib, library_version=0.6.1},mechanism=AMQPLAIN,response=LOGINSguesPASSWORDSguest,locale=en_US),null,""}</div>
<div>1282138358124: conn#0 ch#0 -> {#method<connection.tune-ok>(channel-max=65535,frame-max=131072,heartbeat=0),null,""}</div><div>1282138358125: conn#0 ch#0 -> {#method<connection.open>(virtual-host=/,capabilities=,insist=true),null,""}</div>
<div>1282138358164: conn#0 ch#0 <- {#method<connection.open-ok>(known-hosts=mail01:5672),null,""}</div><div><br></div><div>---- this is where the channel is opened ----</div><div>1282138359932: conn#0 ch#1 -> {#method<channel.open>(out-of-band=),null,""}</div>
</div><div>---- nothing comes back afterwards ----</div><div><br></div><div><br></div><div><div>�<a href="mailto:root@mail01.dmemedialtd.com">root@mail01.dmemedialtd.com</a> [~] > rabbitmqctl list_connections</div><div>
Listing connections ...</div><div>guest � 127.0.0.1 � � � 43467 � running</div><div>...done.</div></div><div><br></div><div><div>�<a href="mailto:root@mail01.dmemedialtd.com">root@mail01.dmemedialtd.com</a> [~] > rabbitmqctl list_channels</div>
<div>Listing channels ...</div><div><rabbit@mail01.3566.0> �guest � false � 0 � � � 0</div><div>...done.</div></div><div><br></div><div>Logs</div><div>----------------</div><div><div>�<a href="mailto:root@mail01.dmemedialtd.com">root@mail01.dmemedialtd.com</a> [/var/log/rabbitmq] > tail -n 20 rabbit@mail01.log</div>
<div><br></div><div>=WARNING REPORT==== 18-Aug-2010::14:28:49 ===</div><div>exception on TCP connection <0.873.0> from <a href="http://10.1.3.1:36844">10.1.3.1:36844</a></div><div>connection_closed_abruptly</div><div>
<br></div><div>=INFO REPORT==== 18-Aug-2010::14:28:49 ===</div><div>closing TCP connection <0.873.0> from <a href="http://10.1.3.1:36844">10.1.3.1:36844</a></div><div><br></div><div>=WARNING REPORT==== 18-Aug-2010::14:31:54 ===</div>
<div>exception on TCP connection <0.3212.0> from <a href="http://127.0.0.1:57206">127.0.0.1:57206</a></div><div>connection_closed_abruptly</div><div><br></div><div>=INFO REPORT==== 18-Aug-2010::14:31:54 ===</div><div>
closing TCP connection <0.3212.0> from <a href="http://127.0.0.1:57206">127.0.0.1:57206</a></div><div><br></div><div>=INFO REPORT==== 18-Aug-2010::14:32:37 ===</div><div>accepted TCP connection on <a href="http://0.0.0.0:5672">0.0.0.0:5672</a> from <a href="http://127.0.0.1:43467">127.0.0.1:43467</a></div>
<div><br></div><div>=INFO REPORT==== 18-Aug-2010::14:32:37 ===</div><div>starting TCP connection <0.3559.0> from <a href="http://127.0.0.1:43467">127.0.0.1:43467</a></div></div><div><br></div><div><br></div><br clear="all">
<br>-- <br><p style="color:rgb(0, 51, 102);font-weight:bold"><span style="border-collapse:separate;font-family:arial;line-height:normal;font-size:small">Cal Leeming</span></p><p style="color:rgb(0, 51, 102);font-weight:bold">
Operational Security & Support Team<br></p><p style="border-bottom-width:1px;border-bottom-style:solid;border-bottom-color:rgb(204, 204, 204);padding-bottom:25px"><b>Out of Hours:�</b>+44 (07534) 971120 |�<b>Support Tickets:�</b><a href="mailto:support@simplicitymedialtd.co.uk" style="color:rgb(49, 132, 173)" target="_blank">support@simplicitymedialtd.co.uk</a>�<br>
<b>Fax:�</b>+44 (02476) 578987�|�<b>Email:�</b><a href="mailto:cal.leeming@simplicitymedialtd.co.uk" style="color:rgb(49, 132, 173)" target="_blank">cal.leeming@simplicitymedialtd.co.uk</a>�</p><div><span style="line-height:13px;font-size:10px;color:rgb(185, 184, 184)">Simplicity Media Ltd. All rights reserved.<br>
</span></div><div><span style="line-height:13px;font-size:10px;color:rgb(185, 184, 184)">Registered company number�7143564<span></span><span></span></span></div><br>