[rabbitmq-discuss] Perl Net::RabbitMQ and failure conditions

[Alexis Richardson]

Ronald

I spoke with the author, Theo Schlossnagle, who permitted to post his
reply here:

-
You must catch SIGPIPE if you plan to handle busted writes on
sockets. This is straight forward from perl.

The publishes don't erroneously indicated success as the writes() and
synchronous. The remote is down, but the TCP socket hasn't realized it
yet... So you write(), which is fine (as the socket hasn't flushed).
So, if it flushes into the kernel and the kernel knows the socket is
busted -> SIGPIPE, if not, then you'll get a successful publish.  That
behaviour is expected, no?

The hang in DESTROY sounds like a bug -- I have not seen that one on
my side.  Would like some more information -- user-space stack trace
of the hung perl process w/ symbols would be awesome.

You can also open a issue on github for the specific problems
remaining unresolved:

https://github.com/omniti-labs/Net--RabbitMQ
-

Hope this helps!

alexis



On Tue, May 31, 2011 at 6:34 PM, Ronald J Kimball
<rkimball at pangeamedia.com> wrote:
> Hi, thanks for the response.  Unfortunately, I haven't really found an
> answer.
>
> We upgraded RabbitMQ to 2.4.1, after which Net::RabbitMQ no longer hangs
> when the remote server is up but the remote app is down.
>
> However, Net::RabbitMQ still returns a false success when the remote server
> or app goes down after the publisher connects to the local node.  (The queue
> is located on the remote app.)
>
> For now, we've switched to having a single node.  This decreases our
> throughput slightly, but at least we don't risk dropping messages.
>
> I think we must be doing something wrong, but I don't know what...
>
> Ronald
>
>
> On Tue, May 31, 2011 at 7:41 AM, Alexis Richardson <alexis at rabbitmq.com>
> wrote:
>>
>> Ronald
>>
>> Were you able to find an answer to this?
>>
>> alexis
>>
>>
>>
>> On Thu, May 19, 2011 at 9:44 PM, Ronald J Kimball
>> <rkimball at pangeamedia.com> wrote:
>> > I'm trying to understand how RabbitMQ behaves under failure conditions,
>> > specifically when using the Net::RabbitMQ Perl module.
>> >
>> > We have RabbitMQ set up with three nodes, one on the backend, which
>> > holds
>> > the actual queues, and two on the frontend, which receive publish
>> > requests.
>> >
>> > RabbitMQ 2.2.0
>> > Net::RabbitMQ 0.1.8
>> >
>> >
>> > Below is the chart of behaviors that I have observed.
>> >
>> > "+" means the behavior is as I expected: either the message is
>> > successfully
>> > queued, or an error is thrown.
>> > "-" means the behavior is not as I expected, but can be managed:
>> > specifically, the process receives a SIGPIPE, which I can trap and
>> > recover
>> > from.
>> > "!" means the behavior is not as I expected, and cannot be managed: the
>> > message is not queued but no error is thrown and/or the process hangs.
>> >
>> > "App down" means I ran `rabbitmqctl stop_app`.  "Daemon down" means I
>> > ran
>> > `rabbitmqctl stop`.
>> >
>> >   As publishing process starts up:
>> >
>> >     Remote daemon up, app up
>> >       + Everything okay
>> >
>> >     Remote daemon up, app down
>> >       ! Net::RabbitMQ hangs when declaring queue
>> >
>> >     Remote daemon down
>> >       + Net::RabbitMQ throws error when declaring queue
>> >         "Declaring queue: server channel error 404, message: NOT_FOUND -
>> > no
>> > queue 'test' in vhost '/'"
>> >
>> >     Local daemon up, app down
>> >       + Net::RabbitMQ throws error when connecting
>> >         "Opening socket: Connection refused"
>> >
>> >     Local daemon down
>> >       + Net::RabbitMQ throws error when connecting
>> >         "Opening socket: Connection refused"
>> >
>> >
>> >   After publishing process starts up:
>> >
>> >     Local app goes down
>> >       - Process receives SIGPIPE
>> >
>> >     Local daemon goes down
>> >       - Process receives SIGPIPE
>> >
>> >     Remote app goes down
>> >       ! Net::RabbitMQ falsely indicates success when publishing, then
>> > hangs
>> > (in DESTROY?)
>> >
>> >     Remote daemon goes down
>> >       ! Net::RabbitMQ falsely indicates success when publishing
>> >
>> >
>> >     Local app goes down, comes back up
>> >       - Process receives SIGPIPE
>> >
>> >     Local daemon goes down, comes back up
>> >       - Process receives SIGPIPE
>> >
>> >     Remote app goes down, comes back up
>> >       + Everything okay
>> >
>> >     Remote daemon goes down, comes back up
>> >       + Everything okay
>> >
>> >
>> > Have other people had these problems with Net::RabbitMQ?  Can we resolve
>> > these issues by changing something in our RabbitMQ configuration?
>> >
>> >
>> > thanks,
>> > Ronald
>> >
>> >
>> > P.S. Here's my script.
>> >
>> > #!/usr/local/bin/perl
>> >
>> > use strict;
>> > use warnings;
>> >
>> > use Net::RabbitMQ;
>> >
>> > $| = 1;
>> >
>> > $SIG{'PIPE'} = sub { die "SIGPIPE\n" };
>> >
>> > my $mq = Net::RabbitMQ->new();
>> >
>> > alarm(10);
>> >
>> > print "Connecting\n";
>> > $mq->connect('localhost', { user => 'engagement', password => '********'
>> > })
>> >   or die "Can't connect to RabbitMQ\n";
>> >
>> > print "Opening channel\n";
>> > $mq->channel_open(1);
>> >
>> > print "Declaring exchange\n";
>> > $mq->exchange_declare(1, 'ee_exchange', { durable => 1 });
>> >
>> > print "Declaring queue\n";
>> > $mq->queue_declare(1, 'test', { durable => 1 });
>> >
>> > print "Binding queue\n";
>> > $mq->queue_bind(1, 'test', 'ee_exchange', 'ee_test');
>> >
>> > alarm(0);
>> >
>> > print "> ";
>> > <>;
>> >
>> > alarm(10);
>> >
>> > print "Publishing message\n";
>> > my $rc =
>> >   $mq->publish(1, 'ee_test', 'hello world!',
>> >                { exchange => 'ee_exchange', mandatory => 1, immediate =>
>> > 0
>> > },
>> >                { delivery_mode => 2 });
>> > print "Result: $rc\n";
>> >
>> > __END__
>> >
>> >
>> > _______________________________________________
>> > rabbitmq-discuss mailing list
>> > rabbitmq-discuss at lists.rabbitmq.com
>> > https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
>> >
>> >
>
>