[rabbitmq-discuss] Perl Net::RabbitMQ and failure conditions
Ronald J Kimball
rkimball at pangeamedia.com
Fri Jun 3 17:39:47 BST 2011
Hi Alexis, thanks for getting in touch with Theo. However, I'm not sure
this addresses my issue.
I think he's using "remote" here to refer to the Rabbit process on the other
end of the socket from my publisher. If that Rabbit process goes down, the
publisher does receive a SIGPIPE. However, that is actually my "local"
process. My "remote" process is the Rabbit node on another machine that
holds the queue. My publisher connects to the local Rabbit, and the local
Rabbit connects to the remote Rabbit. If the remote Rabbit goes down, there
is no SIGPIPE in my publisher.
The hang in DESTROY was resolved by upgrading Rabbit itself.
I'll contact him myself and let you know what I find out.
Ronald
On Fri, Jun 3, 2011 at 11:47 AM, Alexis Richardson <alexis at rabbitmq.com>wrote:
> Ronald
>
> I spoke with the author, Theo Schlossnagle, who permitted to post his
> reply here:
>
> -
> You must catch SIGPIPE if you plan to handle busted writes on
> sockets. This is straight forward from perl.
>
> The publishes don't erroneously indicated success as the writes() and
> synchronous. The remote is down, but the TCP socket hasn't realized it
> yet... So you write(), which is fine (as the socket hasn't flushed).
> So, if it flushes into the kernel and the kernel knows the socket is
> busted -> SIGPIPE, if not, then you'll get a successful publish. That
> behaviour is expected, no?
>
> The hang in DESTROY sounds like a bug -- I have not seen that one on
> my side. Would like some more information -- user-space stack trace
> of the hung perl process w/ symbols would be awesome.
>
> You can also open a issue on github for the specific problems
> remaining unresolved:
>
> https://github.com/omniti-labs/Net--RabbitMQ
> -
>
> Hope this helps!
>
> alexis
>
>
>
> On Tue, May 31, 2011 at 6:34 PM, Ronald J Kimball
> <rkimball at pangeamedia.com> wrote:
> > Hi, thanks for the response. Unfortunately, I haven't really found an
> > answer.
> >
> > We upgraded RabbitMQ to 2.4.1, after which Net::RabbitMQ no longer hangs
> > when the remote server is up but the remote app is down.
> >
> > However, Net::RabbitMQ still returns a false success when the remote
> server
> > or app goes down after the publisher connects to the local node. (The
> queue
> > is located on the remote app.)
> >
> > For now, we've switched to having a single node. This decreases our
> > throughput slightly, but at least we don't risk dropping messages.
> >
> > I think we must be doing something wrong, but I don't know what...
> >
> > Ronald
> >
> >
> > On Tue, May 31, 2011 at 7:41 AM, Alexis Richardson <alexis at rabbitmq.com>
> > wrote:
> >>
> >> Ronald
> >>
> >> Were you able to find an answer to this?
> >>
> >> alexis
> >>
> >>
> >>
> >> On Thu, May 19, 2011 at 9:44 PM, Ronald J Kimball
> >> <rkimball at pangeamedia.com> wrote:
> >> > I'm trying to understand how RabbitMQ behaves under failure
> conditions,
> >> > specifically when using the Net::RabbitMQ Perl module.
> >> >
> >> > We have RabbitMQ set up with three nodes, one on the backend, which
> >> > holds
> >> > the actual queues, and two on the frontend, which receive publish
> >> > requests.
> >> >
> >> > RabbitMQ 2.2.0
> >> > Net::RabbitMQ 0.1.8
> >> >
> >> >
> >> > Below is the chart of behaviors that I have observed.
> >> >
> >> > "+" means the behavior is as I expected: either the message is
> >> > successfully
> >> > queued, or an error is thrown.
> >> > "-" means the behavior is not as I expected, but can be managed:
> >> > specifically, the process receives a SIGPIPE, which I can trap and
> >> > recover
> >> > from.
> >> > "!" means the behavior is not as I expected, and cannot be managed:
> the
> >> > message is not queued but no error is thrown and/or the process hangs.
> >> >
> >> > "App down" means I ran `rabbitmqctl stop_app`. "Daemon down" means I
> >> > ran
> >> > `rabbitmqctl stop`.
> >> >
> >> > As publishing process starts up:
> >> >
> >> > Remote daemon up, app up
> >> > + Everything okay
> >> >
> >> > Remote daemon up, app down
> >> > ! Net::RabbitMQ hangs when declaring queue
> >> >
> >> > Remote daemon down
> >> > + Net::RabbitMQ throws error when declaring queue
> >> > "Declaring queue: server channel error 404, message: NOT_FOUND
> -
> >> > no
> >> > queue 'test' in vhost '/'"
> >> >
> >> > Local daemon up, app down
> >> > + Net::RabbitMQ throws error when connecting
> >> > "Opening socket: Connection refused"
> >> >
> >> > Local daemon down
> >> > + Net::RabbitMQ throws error when connecting
> >> > "Opening socket: Connection refused"
> >> >
> >> >
> >> > After publishing process starts up:
> >> >
> >> > Local app goes down
> >> > - Process receives SIGPIPE
> >> >
> >> > Local daemon goes down
> >> > - Process receives SIGPIPE
> >> >
> >> > Remote app goes down
> >> > ! Net::RabbitMQ falsely indicates success when publishing, then
> >> > hangs
> >> > (in DESTROY?)
> >> >
> >> > Remote daemon goes down
> >> > ! Net::RabbitMQ falsely indicates success when publishing
> >> >
> >> >
> >> > Local app goes down, comes back up
> >> > - Process receives SIGPIPE
> >> >
> >> > Local daemon goes down, comes back up
> >> > - Process receives SIGPIPE
> >> >
> >> > Remote app goes down, comes back up
> >> > + Everything okay
> >> >
> >> > Remote daemon goes down, comes back up
> >> > + Everything okay
> >> >
> >> >
> >> > Have other people had these problems with Net::RabbitMQ? Can we
> resolve
> >> > these issues by changing something in our RabbitMQ configuration?
> >> >
> >> >
> >> > thanks,
> >> > Ronald
> >> >
> >> >
> >> > P.S. Here's my script.
> >> >
> >> > #!/usr/local/bin/perl
> >> >
> >> > use strict;
> >> > use warnings;
> >> >
> >> > use Net::RabbitMQ;
> >> >
> >> > $| = 1;
> >> >
> >> > $SIG{'PIPE'} = sub { die "SIGPIPE\n" };
> >> >
> >> > my $mq = Net::RabbitMQ->new();
> >> >
> >> > alarm(10);
> >> >
> >> > print "Connecting\n";
> >> > $mq->connect('localhost', { user => 'engagement', password =>
> '********'
> >> > })
> >> > or die "Can't connect to RabbitMQ\n";
> >> >
> >> > print "Opening channel\n";
> >> > $mq->channel_open(1);
> >> >
> >> > print "Declaring exchange\n";
> >> > $mq->exchange_declare(1, 'ee_exchange', { durable => 1 });
> >> >
> >> > print "Declaring queue\n";
> >> > $mq->queue_declare(1, 'test', { durable => 1 });
> >> >
> >> > print "Binding queue\n";
> >> > $mq->queue_bind(1, 'test', 'ee_exchange', 'ee_test');
> >> >
> >> > alarm(0);
> >> >
> >> > print "> ";
> >> > <>;
> >> >
> >> > alarm(10);
> >> >
> >> > print "Publishing message\n";
> >> > my $rc =
> >> > $mq->publish(1, 'ee_test', 'hello world!',
> >> > { exchange => 'ee_exchange', mandatory => 1, immediate
> =>
> >> > 0
> >> > },
> >> > { delivery_mode => 2 });
> >> > print "Result: $rc\n";
> >> >
> >> > __END__
> >> >
> >> >
> >> > _______________________________________________
> >> > rabbitmq-discuss mailing list
> >> > rabbitmq-discuss at lists.rabbitmq.com
> >> > https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
> >> >
> >> >
> >
> >
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20110603/afc55274/attachment.htm>
More information about the rabbitmq-discuss
mailing list