[rabbitmq-discuss] Problems with RabbitMQ not responding on channel request

• Previous message: [rabbitmq-discuss] Problems with RabbitMQ not responding on channel request
• Next message: [rabbitmq-discuss] Problems with RabbitMQ not responding on channel request
• Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]

Sorry guys, previous pcap only contained client data.

Here is another one with the server responses too!

http://www.wikifortio.com/822943/trace%20(1).pcap

Cal

On Wed, Aug 18, 2010 at 5:01 PM, Cal Leeming [Simplicity Media Ltd] <
cal.leeming at simplicitymedialtd.co.uk> wrote:

> Also,
>
> I have uploaded the tcpdump pcap of the connection.
>
> http://www.wikifortio.com/540101/trace.pcap
>
> In this pcap, I had attempted to connect several times, and then CTRL + C
> the python app after a few seconds.
>
> Cheers
>
> Cal
>
> On Wed, Aug 18, 2010 at 2:48 PM, Cal Leeming [Simplicity Media Ltd] <
> cal.leeming at simplicitymedialtd.co.uk> wrote:
>
>> Hi guys,
>>
>> I seem to have a problem where RabbitMQ will stop accepting channel
>> requests after either a certain amount of messages are in the queue, or once
>> I have sent 200-300 connection requests.
>>
>> I have done this using the Tracer and without the Tracer, and experience
>> the exact same problem each time. I'm also not using any custom rabbitmq
>> config.
>>
>> At first I thought it was a memory problem, but if I restart RabbitMQ,
>> sometimes it will start accepting connections again and allow me to put more
>> items on the queue, but then other times it won't.
>>
>> Whenever I delete the mnesia folder, it will start working again, but then
>> after a few thousands messages (20k+) it will randomly stop again.
>>
>> Currently, I am at 324k messages in the queue.
>>
>> Does anyone have any ideas what could be causing this?
>>
>> Cheers
>>
>> Cal
>>
>>
>>
>> System info
>> -----------------
>>  root at mail01.dmemedialtd.com [/var/log/rabbitmq] > uname -a
>> Linux mail01.dmemedialtd.com 2.6.26-2-amd64 #1 SMP Sun Jun 20 20:16:30
>> UTC 2010 x86_64 GNU/Linux
>>
>>  root at mail01.dmemedialtd.com [/var/log/rabbitmq] > cat /etc/issue
>> Debian GNU/Linux 5.0 \n \l
>>
>>  top - 14:42:53 up 49 days, 15:35,  2 users,  load average: 0.00, 0.00,
>> 0.00
>> Tasks:  60 total,   1 running,  59 sleeping,   0 stopped,   0 zombie
>> Cpu0  :  0.0%us,  0.0%sy,  0.0%ni, 99.9%id,  0.0%wa,  0.0%hi,  0.0%si,
>>  0.0%st
>> Cpu1  :  0.0%us,  0.0%sy,  0.0%ni, 99.9%id,  0.0%wa,  0.0%hi,  0.0%si,
>>  0.0%st
>> Mem:   1027932k total,   766764k used,   261168k free,    25584k buffers
>> Swap:   971892k total,   212020k used,   759872k free,   288416k cached
>>
>>   PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
>>  4682 rabbitmq  20   0  610m 349m 2020 S    0 34.8   0:28.47 beam.smp
>>  4735 rabbitmq  20   0  3772  372  368 S    0  0.0   0:00.00 cpu_sup
>>  4764 rabbitmq  20   0 10492  392  388 S    0  0.0   0:00.00 inet_gethost
>>  4766 rabbitmq  20   0 16784  560  556 S    0  0.1   0:00.00 inet_gethost
>> 14382 rabbitmq  20   0 11004  940  332 S    0  0.1   0:01.50 epmd
>>
>> rabbitmq 14382  0.0  0.0  11004   940 ?        S    Aug12   0:01
>> /usr/lib/erlang/erts-5.6.3/bin/epmd -daemon
>> rabbitmq  4682  0.9 34.7 624684 357480 ?       Ssl  13:54   0:28
>> /usr/lib/erlang/erts-5.6.3/bin/beam.smp -W w -K true -A30 -P 1048576 --
>> -root /usr/lib/erlang -progname erl -- -home /var/lib/rabbitmq -pa
>> /usr/lib/rabbitmq/lib/rabbitmq_server-1.8.1/sbin/../ebin -noshell -noinput
>> -s rabbit -sname rabbit at mail01 -boot start_sasl -kernel
>> inet_default_listen_options [{nodelay,true}] -kernel
>> inet_default_connect_options [{nodelay,true}] -sasl errlog_type error
>> -kernel error_logger {file,"/var/log/rabbitmq/rabbit at mail01.log"} -sasl
>> sasl_error_logger {file,"/var/log/rabbitmq/rabbit at mail01-sasl.log"}
>> -os_mon start_cpu_sup true -os_mon start_disksup false -os_mon start_memsup
>> false -mnesia dir "/var/lib/rabbitmq/mnesia/rabbit at mail01" -noshell
>> -noinput
>> rabbitmq  4735  0.0  0.0   3772   372 ?        Ss   13:54   0:00  \_
>> /usr/lib/erlang/lib/os_mon-2.1.6/priv/bin/cpu_sup
>> rabbitmq  4764  0.0  0.0  10492   392 ?        Ss   13:54   0:00  \_
>> inet_gethost 4
>> rabbitmq  4766  0.0  0.0  16784   560 ?        S    13:54   0:00      \_
>> inet_gethost 4
>>
>>
>>  root at mail01.dmemedialtd.com [~/rabbitmq-java-client-1.4.0/build/dist] >
>> rabbitmqctl list_queues
>> Listing queues ...
>> po_box 324722
>> ...done.
>>
>>
>>  root at mail01.dmemedialtd.com [/var/lib/rabbitmq/mnesia/rabbit at mail01] >
>> ls -la
>> total 162608
>> drwxr-xr-x 2 rabbitmq rabbitmq     4096 2010-08-18 14:00 .
>> drwxr-xr-x 3 rabbitmq rabbitmq     4096 2010-08-12 21:54 ..
>> -rw-r--r-- 1 rabbitmq rabbitmq      155 2010-08-18 14:00 DECISION_TAB.LOG
>> -rw-r--r-- 1 rabbitmq rabbitmq       91 2010-08-18 14:00 LATEST.LOG
>> -rw-r--r-- 1 rabbitmq rabbitmq        8 2010-08-12 21:54 rabbit_config.DCD
>> -rw-r--r-- 1 rabbitmq rabbitmq      822 2010-08-18 13:31
>> rabbit_durable_exchange.DCD
>> -rw-r--r-- 1 rabbitmq rabbitmq      198 2010-08-18 13:54
>> rabbit_durable_queue.DCD
>> -rw-r--r-- 1 rabbitmq rabbitmq      271 2010-08-18 13:57
>> rabbit_durable_queue.DCL
>> -rw-r--r-- 1 rabbitmq rabbitmq      348 2010-08-18 13:31
>> rabbit_durable_route.DCD
>> -rw-r--r-- 1 rabbitmq rabbitmq 83128355 2010-08-18 13:55
>> rabbit_persister.LOG
>> -rw-r--r-- 1 rabbitmq rabbitmq 83128355 2010-08-18 13:55
>> rabbit_persister.LOG.previous
>> -rw-r--r-- 1 rabbitmq rabbitmq        3 2010-08-18 13:54 rabbit_serial
>> -rw-r--r-- 1 rabbitmq rabbitmq      127 2010-08-18 13:31 rabbit_user.DCD
>> -rw-r--r-- 1 rabbitmq rabbitmq      185 2010-08-18 13:31
>> rabbit_user_permission.DCD
>> -rw-r--r-- 1 rabbitmq rabbitmq      581 2010-08-18 14:00
>> rabbit_user_permission.DCL
>> -rw-r--r-- 1 rabbitmq rabbitmq      126 2010-08-18 13:31 rabbit_vhost.DCD
>> -rw-r--r-- 1 rabbitmq rabbitmq    12389 2010-08-12 21:54 schema.DAT
>>
>>
>>
>> Client Side:
>> -------------------
>> >>> conn = amqp.Connection(host="10.1.3.2:5673", userid="guest",
>> password="guest", virtual_host="/", insist=True)
>> >>> chan = conn.channel()
>>  ---- Blocks forever and a day ----
>>  ---- Press CTRL + C ----
>> ^CTraceback (most recent call last):
>>   File "<stdin>", line 1, in <module>
>>   File "build/bdist.linux-x86_64/egg/amqplib/client_0_8/connection.py",
>> line 230, in channel
>>   File "build/bdist.linux-x86_64/egg/amqplib/client_0_8/channel.py", line
>> 82, in __init__
>>   File "build/bdist.linux-x86_64/egg/amqplib/client_0_8/channel.py", line
>> 471, in _x_open
>>   File
>> "build/bdist.linux-x86_64/egg/amqplib/client_0_8/abstract_channel.py", line
>> 89, in wait
>>   File "build/bdist.linux-x86_64/egg/amqplib/client_0_8/connection.py",
>> line 198, in _wait_method
>>   File
>> "build/bdist.linux-x86_64/egg/amqplib/client_0_8/method_framing.py", line
>> 212, in read_method
>>   File
>> "build/bdist.linux-x86_64/egg/amqplib/client_0_8/method_framing.py", line
>> 127, in _next_method
>>   File "build/bdist.linux-x86_64/egg/amqplib/client_0_8/transport.py",
>> line 109, in read_frame
>>   File "build/bdist.linux-x86_64/egg/amqplib/client_0_8/transport.py",
>> line 200, in _read
>> KeyboardInterrupt
>>
>> Tracer proxy result:
>> -------------------
>>   root at mail01.dmemedialtd.com [~/rabbitmq-java-client-1.4.0/build/dist] >
>> ./runjava.sh com.rabbitmq.tools.Tracer
>> Usage: Tracer [<listenport> [<connecthost> [<connectport>]]]
>> Invoked as: Tracer 5673 localhost 5672
>> com.rabbitmq.tools.Tracer.WITHHOLD_INBOUND_HEARTBEATS = false
>> com.rabbitmq.tools.Tracer.WITHHOLD_OUTBOUND_HEARTBEATS = false
>> com.rabbitmq.tools.Tracer.NO_ASSEMBLE_FRAMES = false
>> com.rabbitmq.tools.Tracer.NO_DECODE_FRAMES = false
>>
>> ---- this is where the connection is started ----
>> 1282138358084: conn#0 ch#0 <-
>> {#method<connection.start>(version-major=8,version-minor=0,server
>> properties={platform=Erlang/OTP, product=RabbitMQ, version=1.8.1,
>> copyright=Copyright (C) 2007-2010 LShift Ltd., Cohesive Financial
>> Technologies LLC., and Rabbit Technologies Ltd., information=Licensed under
>> the MPL.  See http://www.rabbitmq.com/},mechanisms=PLAIN
>> AMQPLAIN,locales=en_US),null,""}
>> 1282138358084: conn#0 ch#0 <-
>> {#method<connection.tune>(channel-max=0,frame-max=131072,heartbeat=0),null,""}
>> 1282138358086: conn#0 ch#0 ->
>> {#method<connection.start-ok>(client-properties={library=Python amqplib,
>> library_version=0.6.1},mechanism=AMQPLAIN,response=LOGINSguesPASSWORDSguest,locale=en_US),null,""}
>> 1282138358124: conn#0 ch#0 ->
>> {#method<connection.tune-ok>(channel-max=65535,frame-max=131072,heartbeat=0),null,""}
>> 1282138358125: conn#0 ch#0 ->
>> {#method<connection.open>(virtual-host=/,capabilities=,insist=true),null,""}
>> 1282138358164: conn#0 ch#0 <-
>> {#method<connection.open-ok>(known-hosts=mail01:5672),null,""}
>>
>> ---- this is where the channel is opened ----
>> 1282138359932: conn#0 ch#1 ->
>> {#method<channel.open>(out-of-band=),null,""}
>> ---- nothing comes back afterwards ----
>>
>>
>>  root at mail01.dmemedialtd.com [~] > rabbitmqctl list_connections
>>  Listing connections ...
>> guest   127.0.0.1       43467   running
>> ...done.
>>
>>  root at mail01.dmemedialtd.com [~] > rabbitmqctl list_channels
>> Listing channels ...
>> <rabbit at mail01.3566.0>  guest   false   0       0
>> ...done.
>>
>> Logs
>> ----------------
>>  root at mail01.dmemedialtd.com [/var/log/rabbitmq] > tail -n 20
>> rabbit at mail01.log
>>
>> =WARNING REPORT==== 18-Aug-2010::14:28:49 ===
>> exception on TCP connection <0.873.0> from 10.1.3.1:36844
>> connection_closed_abruptly
>>
>> =INFO REPORT==== 18-Aug-2010::14:28:49 ===
>> closing TCP connection <0.873.0> from 10.1.3.1:36844
>>
>> =WARNING REPORT==== 18-Aug-2010::14:31:54 ===
>> exception on TCP connection <0.3212.0> from 127.0.0.1:57206
>> connection_closed_abruptly
>>
>> =INFO REPORT==== 18-Aug-2010::14:31:54 ===
>>  closing TCP connection <0.3212.0> from 127.0.0.1:57206
>>
>> =INFO REPORT==== 18-Aug-2010::14:32:37 ===
>> accepted TCP connection on 0.0.0.0:5672 from 127.0.0.1:43467
>>
>> =INFO REPORT==== 18-Aug-2010::14:32:37 ===
>> starting TCP connection <0.3559.0> from 127.0.0.1:43467
>>
>>
>>
>>
>> --
>>
>> Cal Leeming
>>
>> Operational Security & Support Team
>>
>> *Out of Hours: *+44 (07534) 971120 | *Support Tickets: *
>> support at simplicitymedialtd.co.uk
>> *Fax: *+44 (02476) 578987 | *Email: *cal.leeming at simplicitymedialtd.co.uk
>>
>> Simplicity Media Ltd. All rights reserved.
>> Registered company number 7143564
>>
>>
>
>
> --
>
> Cal Leeming
>
> Operational Security & Support Team
>
> *Out of Hours: *+44 (07534) 971120 | *Support Tickets: *
> support at simplicitymedialtd.co.uk
> *Fax: *+44 (02476) 578987 | *Email: *cal.leeming at simplicitymedialtd.co.uk
> Simplicity Media Ltd. All rights reserved.
> Registered company number 7143564
>
>


-- 

Cal Leeming

Operational Security & Support Team

*Out of Hours: *+44 (07534) 971120 | *Support Tickets: *
support at simplicitymedialtd.co.uk
*Fax: *+44 (02476) 578987 | *Email: *cal.leeming at simplicitymedialtd.co.uk
Simplicity Media Ltd. All rights reserved.
Registered company number 7143564
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20100818/e251cab9/attachment-0001.htm>

• Previous message: [rabbitmq-discuss] Problems with RabbitMQ not responding on channel request
• Next message: [rabbitmq-discuss] Problems with RabbitMQ not responding on channel request
• Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]