[rabbitmq-discuss] Problems with RabbitMQ not responding on channel request
Cal Leeming [Simplicity Media Ltd]
cal.leeming at simplicitymedialtd.co.uk
Wed Aug 18 17:01:01 BST 2010
Also,
I have uploaded the tcpdump pcap of the connection.
http://www.wikifortio.com/540101/trace.pcap
In this pcap, I had attempted to connect several times, and then CTRL + C
the python app after a few seconds.
Cheers
Cal
On Wed, Aug 18, 2010 at 2:48 PM, Cal Leeming [Simplicity Media Ltd] <
cal.leeming at simplicitymedialtd.co.uk> wrote:
> Hi guys,
>
> I seem to have a problem where RabbitMQ will stop accepting channel
> requests after either a certain amount of messages are in the queue, or once
> I have sent 200-300 connection requests.
>
> I have done this using the Tracer and without the Tracer, and experience
> the exact same problem each time. I'm also not using any custom rabbitmq
> config.
>
> At first I thought it was a memory problem, but if I restart RabbitMQ,
> sometimes it will start accepting connections again and allow me to put more
> items on the queue, but then other times it won't.
>
> Whenever I delete the mnesia folder, it will start working again, but then
> after a few thousands messages (20k+) it will randomly stop again.
>
> Currently, I am at 324k messages in the queue.
>
> Does anyone have any ideas what could be causing this?
>
> Cheers
>
> Cal
>
>
>
> System info
> -----------------
> root at mail01.dmemedialtd.com [/var/log/rabbitmq] > uname -a
> Linux mail01.dmemedialtd.com 2.6.26-2-amd64 #1 SMP Sun Jun 20 20:16:30 UTC
> 2010 x86_64 GNU/Linux
>
> root at mail01.dmemedialtd.com [/var/log/rabbitmq] > cat /etc/issue
> Debian GNU/Linux 5.0 \n \l
>
> top - 14:42:53 up 49 days, 15:35, 2 users, load average: 0.00, 0.00, 0.00
> Tasks: 60 total, 1 running, 59 sleeping, 0 stopped, 0 zombie
> Cpu0 : 0.0%us, 0.0%sy, 0.0%ni, 99.9%id, 0.0%wa, 0.0%hi, 0.0%si,
> 0.0%st
> Cpu1 : 0.0%us, 0.0%sy, 0.0%ni, 99.9%id, 0.0%wa, 0.0%hi, 0.0%si,
> 0.0%st
> Mem: 1027932k total, 766764k used, 261168k free, 25584k buffers
> Swap: 971892k total, 212020k used, 759872k free, 288416k cached
>
> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
> 4682 rabbitmq 20 0 610m 349m 2020 S 0 34.8 0:28.47 beam.smp
> 4735 rabbitmq 20 0 3772 372 368 S 0 0.0 0:00.00 cpu_sup
> 4764 rabbitmq 20 0 10492 392 388 S 0 0.0 0:00.00 inet_gethost
> 4766 rabbitmq 20 0 16784 560 556 S 0 0.1 0:00.00 inet_gethost
> 14382 rabbitmq 20 0 11004 940 332 S 0 0.1 0:01.50 epmd
>
> rabbitmq 14382 0.0 0.0 11004 940 ? S Aug12 0:01
> /usr/lib/erlang/erts-5.6.3/bin/epmd -daemon
> rabbitmq 4682 0.9 34.7 624684 357480 ? Ssl 13:54 0:28
> /usr/lib/erlang/erts-5.6.3/bin/beam.smp -W w -K true -A30 -P 1048576 --
> -root /usr/lib/erlang -progname erl -- -home /var/lib/rabbitmq -pa
> /usr/lib/rabbitmq/lib/rabbitmq_server-1.8.1/sbin/../ebin -noshell -noinput
> -s rabbit -sname rabbit at mail01 -boot start_sasl -kernel
> inet_default_listen_options [{nodelay,true}] -kernel
> inet_default_connect_options [{nodelay,true}] -sasl errlog_type error
> -kernel error_logger {file,"/var/log/rabbitmq/rabbit at mail01.log"} -sasl
> sasl_error_logger {file,"/var/log/rabbitmq/rabbit at mail01-sasl.log"}
> -os_mon start_cpu_sup true -os_mon start_disksup false -os_mon start_memsup
> false -mnesia dir "/var/lib/rabbitmq/mnesia/rabbit at mail01" -noshell
> -noinput
> rabbitmq 4735 0.0 0.0 3772 372 ? Ss 13:54 0:00 \_
> /usr/lib/erlang/lib/os_mon-2.1.6/priv/bin/cpu_sup
> rabbitmq 4764 0.0 0.0 10492 392 ? Ss 13:54 0:00 \_
> inet_gethost 4
> rabbitmq 4766 0.0 0.0 16784 560 ? S 13:54 0:00 \_
> inet_gethost 4
>
>
> root at mail01.dmemedialtd.com [~/rabbitmq-java-client-1.4.0/build/dist] >
> rabbitmqctl list_queues
> Listing queues ...
> po_box 324722
> ...done.
>
>
> root at mail01.dmemedialtd.com [/var/lib/rabbitmq/mnesia/rabbit at mail01] > ls
> -la
> total 162608
> drwxr-xr-x 2 rabbitmq rabbitmq 4096 2010-08-18 14:00 .
> drwxr-xr-x 3 rabbitmq rabbitmq 4096 2010-08-12 21:54 ..
> -rw-r--r-- 1 rabbitmq rabbitmq 155 2010-08-18 14:00 DECISION_TAB.LOG
> -rw-r--r-- 1 rabbitmq rabbitmq 91 2010-08-18 14:00 LATEST.LOG
> -rw-r--r-- 1 rabbitmq rabbitmq 8 2010-08-12 21:54 rabbit_config.DCD
> -rw-r--r-- 1 rabbitmq rabbitmq 822 2010-08-18 13:31
> rabbit_durable_exchange.DCD
> -rw-r--r-- 1 rabbitmq rabbitmq 198 2010-08-18 13:54
> rabbit_durable_queue.DCD
> -rw-r--r-- 1 rabbitmq rabbitmq 271 2010-08-18 13:57
> rabbit_durable_queue.DCL
> -rw-r--r-- 1 rabbitmq rabbitmq 348 2010-08-18 13:31
> rabbit_durable_route.DCD
> -rw-r--r-- 1 rabbitmq rabbitmq 83128355 2010-08-18 13:55
> rabbit_persister.LOG
> -rw-r--r-- 1 rabbitmq rabbitmq 83128355 2010-08-18 13:55
> rabbit_persister.LOG.previous
> -rw-r--r-- 1 rabbitmq rabbitmq 3 2010-08-18 13:54 rabbit_serial
> -rw-r--r-- 1 rabbitmq rabbitmq 127 2010-08-18 13:31 rabbit_user.DCD
> -rw-r--r-- 1 rabbitmq rabbitmq 185 2010-08-18 13:31
> rabbit_user_permission.DCD
> -rw-r--r-- 1 rabbitmq rabbitmq 581 2010-08-18 14:00
> rabbit_user_permission.DCL
> -rw-r--r-- 1 rabbitmq rabbitmq 126 2010-08-18 13:31 rabbit_vhost.DCD
> -rw-r--r-- 1 rabbitmq rabbitmq 12389 2010-08-12 21:54 schema.DAT
>
>
>
> Client Side:
> -------------------
> >>> conn = amqp.Connection(host="10.1.3.2:5673", userid="guest",
> password="guest", virtual_host="/", insist=True)
> >>> chan = conn.channel()
> ---- Blocks forever and a day ----
> ---- Press CTRL + C ----
> ^CTraceback (most recent call last):
> File "<stdin>", line 1, in <module>
> File "build/bdist.linux-x86_64/egg/amqplib/client_0_8/connection.py",
> line 230, in channel
> File "build/bdist.linux-x86_64/egg/amqplib/client_0_8/channel.py", line
> 82, in __init__
> File "build/bdist.linux-x86_64/egg/amqplib/client_0_8/channel.py", line
> 471, in _x_open
> File
> "build/bdist.linux-x86_64/egg/amqplib/client_0_8/abstract_channel.py", line
> 89, in wait
> File "build/bdist.linux-x86_64/egg/amqplib/client_0_8/connection.py",
> line 198, in _wait_method
> File "build/bdist.linux-x86_64/egg/amqplib/client_0_8/method_framing.py",
> line 212, in read_method
> File "build/bdist.linux-x86_64/egg/amqplib/client_0_8/method_framing.py",
> line 127, in _next_method
> File "build/bdist.linux-x86_64/egg/amqplib/client_0_8/transport.py", line
> 109, in read_frame
> File "build/bdist.linux-x86_64/egg/amqplib/client_0_8/transport.py", line
> 200, in _read
> KeyboardInterrupt
>
> Tracer proxy result:
> -------------------
> root at mail01.dmemedialtd.com [~/rabbitmq-java-client-1.4.0/build/dist] >
> ./runjava.sh com.rabbitmq.tools.Tracer
> Usage: Tracer [<listenport> [<connecthost> [<connectport>]]]
> Invoked as: Tracer 5673 localhost 5672
> com.rabbitmq.tools.Tracer.WITHHOLD_INBOUND_HEARTBEATS = false
> com.rabbitmq.tools.Tracer.WITHHOLD_OUTBOUND_HEARTBEATS = false
> com.rabbitmq.tools.Tracer.NO_ASSEMBLE_FRAMES = false
> com.rabbitmq.tools.Tracer.NO_DECODE_FRAMES = false
>
> ---- this is where the connection is started ----
> 1282138358084: conn#0 ch#0 <-
> {#method<connection.start>(version-major=8,version-minor=0,server
> properties={platform=Erlang/OTP, product=RabbitMQ, version=1.8.1,
> copyright=Copyright (C) 2007-2010 LShift Ltd., Cohesive Financial
> Technologies LLC., and Rabbit Technologies Ltd., information=Licensed under
> the MPL. See http://www.rabbitmq.com/},mechanisms=PLAIN
> AMQPLAIN,locales=en_US),null,""}
> 1282138358084: conn#0 ch#0 <-
> {#method<connection.tune>(channel-max=0,frame-max=131072,heartbeat=0),null,""}
> 1282138358086: conn#0 ch#0 ->
> {#method<connection.start-ok>(client-properties={library=Python amqplib,
> library_version=0.6.1},mechanism=AMQPLAIN,response=LOGINSguesPASSWORDSguest,locale=en_US),null,""}
> 1282138358124: conn#0 ch#0 ->
> {#method<connection.tune-ok>(channel-max=65535,frame-max=131072,heartbeat=0),null,""}
> 1282138358125: conn#0 ch#0 ->
> {#method<connection.open>(virtual-host=/,capabilities=,insist=true),null,""}
> 1282138358164: conn#0 ch#0 <-
> {#method<connection.open-ok>(known-hosts=mail01:5672),null,""}
>
> ---- this is where the channel is opened ----
> 1282138359932: conn#0 ch#1 -> {#method<channel.open>(out-of-band=),null,""}
> ---- nothing comes back afterwards ----
>
>
> root at mail01.dmemedialtd.com [~] > rabbitmqctl list_connections
> Listing connections ...
> guest 127.0.0.1 43467 running
> ...done.
>
> root at mail01.dmemedialtd.com [~] > rabbitmqctl list_channels
> Listing channels ...
> <rabbit at mail01.3566.0> guest false 0 0
> ...done.
>
> Logs
> ----------------
> root at mail01.dmemedialtd.com [/var/log/rabbitmq] > tail -n 20
> rabbit at mail01.log
>
> =WARNING REPORT==== 18-Aug-2010::14:28:49 ===
> exception on TCP connection <0.873.0> from 10.1.3.1:36844
> connection_closed_abruptly
>
> =INFO REPORT==== 18-Aug-2010::14:28:49 ===
> closing TCP connection <0.873.0> from 10.1.3.1:36844
>
> =WARNING REPORT==== 18-Aug-2010::14:31:54 ===
> exception on TCP connection <0.3212.0> from 127.0.0.1:57206
> connection_closed_abruptly
>
> =INFO REPORT==== 18-Aug-2010::14:31:54 ===
> closing TCP connection <0.3212.0> from 127.0.0.1:57206
>
> =INFO REPORT==== 18-Aug-2010::14:32:37 ===
> accepted TCP connection on 0.0.0.0:5672 from 127.0.0.1:43467
>
> =INFO REPORT==== 18-Aug-2010::14:32:37 ===
> starting TCP connection <0.3559.0> from 127.0.0.1:43467
>
>
>
>
> --
>
> Cal Leeming
>
> Operational Security & Support Team
>
> *Out of Hours: *+44 (07534) 971120 | *Support Tickets: *
> support at simplicitymedialtd.co.uk
> *Fax: *+44 (02476) 578987 | *Email: *cal.leeming at simplicitymedialtd.co.uk
> Simplicity Media Ltd. All rights reserved.
> Registered company number 7143564
>
>
--
Cal Leeming
Operational Security & Support Team
*Out of Hours: *+44 (07534) 971120 | *Support Tickets: *
support at simplicitymedialtd.co.uk
*Fax: *+44 (02476) 578987 | *Email: *cal.leeming at simplicitymedialtd.co.uk
Simplicity Media Ltd. All rights reserved.
Registered company number 7143564
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20100818/24db1069/attachment-0001.htm>
More information about the rabbitmq-discuss
mailing list