[rabbitmq-discuss] RabbitMQ Consumer timeout (Alan Antonuk)
Lothe_Shashank
Lothe_Shashank at mahindrasatyam.com
Fri Jul 5 14:52:36 BST 2013
Dear Alan,
I download rabbitmq-c master package from and tried newly added amqp_simple_wait_frame_noblock API. I modified amqp_listen.c example and instead of amqp_simple_wait_frame I used amqp_simple_wait_frame_noblock with different timeout values. However this API is failing immediately with return value as -13. What this return value means? Is there any other way of using timeout so that if I don't receive a frame within say 10ms I can reject that message.
Another problem I am facing with amqp_basic_reject API. In amqp_listen.c example I called this API (with requeue=1) and I got following console output. After that it is not able to receive any message sent by amqp_send_string example.
00000000: 48 65 6C 6C 6F 20 57 6F : 72 6C 64 Hello World
0000000B:
Result 0
Frame type 1, channel 1
Method AMQP_CHANNEL_CLOSE_METHOD
One more problem I would like to report. I have created a client 7 different queues with 7 different channels and bound with 7 different binding keys on the same connection with direct exchange. The listening part of these 7 queues are with 7 different threads listening for server input. When I am sending message from server with corresponding routing keys the messages are delivered to different queue than the expected queue. (I believe it is called Shunting messages from one queue to another). What could be the problem? Do I have to do something from rabbitmq server to targeted delivery? I tried by making queues exclusive but got the same result.
Please help.
Thanks and regards,
Shashank
---------- Forwarded message ----------
From: <rabbitmq-discuss-request at lists.rabbitmq.com<mailto:rabbitmq-discuss-request at lists.rabbitmq.com>>
Date: Wed, Jun 26, 2013 at 4:30 PM
Subject: rabbitmq-discuss Digest, Vol 73, Issue 27
To: rabbitmq-discuss at lists.rabbitmq.com<mailto:rabbitmq-discuss at lists.rabbitmq.com>
Send rabbitmq-discuss mailing list submissions to
rabbitmq-discuss at lists.rabbitmq.com<mailto:rabbitmq-discuss at lists.rabbitmq.com>
To subscribe or unsubscribe via the World Wide Web, visit
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
or, via email, send a message with subject or body 'help' to
rabbitmq-discuss-request at lists.rabbitmq.com<mailto:rabbitmq-discuss-request at lists.rabbitmq.com>
You can reach the person managing the list at
rabbitmq-discuss-owner at lists.rabbitmq.com<mailto:rabbitmq-discuss-owner at lists.rabbitmq.com>
When replying, please edit your Subject line so it is more specific
than "Re: Contents of rabbitmq-discuss digest..."
Today's Topics:
1. Determining Required Erlang Version for Upgrade (Richard Raseley)
2. Re: RabbitMQ Consumer timeout (Alan Antonuk)
3. Re: RabbitMQ's Mirroring results in weird behavior when slave
goes down (thomas)
4. Re: Determining Required Erlang Version for Upgrade
(Michael Klishin)
5. Re: rabbitmq Error:{cannot_discover_cluster, "the nodes
provided are either offline or not running"} (Emile Joubert)
6. Re: Determining Required Erlang Version for Upgrade
(Emile Joubert)
7. Re: RabbitMQ won't start after the disk went full (Emile Joubert)
8. Re: RabbitMQ's Mirroring results in weird behavior when slave
goes down (Tim Watson)
----------------------------------------------------------------------
Message: 1
Date: Tue, 25 Jun 2013 12:39:15 -0700
From: Richard Raseley <richard at raseley.com<mailto:richard at raseley.com>>
Subject: [rabbitmq-discuss] Determining Required Erlang Version for
Upgrade
To: RabbitMQ-Discuss Mailing List
<rabbitmq-discuss at lists.rabbitmq.com<mailto:rabbitmq-discuss at lists.rabbitmq.com>>
Message-ID:
<CAO=6ox_tTVUHNnd4VwPr2zO+iGnjb1Kr5fYT0OEadj-urYRiAA at mail.gmail.com<mailto:6ox_tTVUHNnd4VwPr2zO%2BiGnjb1Kr5fYT0OEadj-urYRiAA at mail.gmail.com>>
Content-Type: text/plain; charset="iso-8859-1"
I am in the process of drafting a plan to upgrade several 3.0.x instances
to 3.1.2. I cannot seem to find in the release notes the specific version
of Erlang that is required for the new versions.
Would someone please direct me to a version requirement matrix (assuming
one exists)?
Regards,
Richard
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20130625/196f9e67/attachment.html>
------------------------------
Message: 2
Date: Tue, 25 Jun 2013 17:36:48 -0700
From: Alan Antonuk <alan.antonuk at gmail.com<mailto:alan.antonuk at gmail.com>>
Subject: Re: [rabbitmq-discuss] RabbitMQ Consumer timeout
To: Discussions about RabbitMQ <rabbitmq-discuss at lists.rabbitmq.com<mailto:rabbitmq-discuss at lists.rabbitmq.com>>
Message-ID:
<CAAt2poJAP5z0-gsdNFaNaH0qzR_FLcxwK_tQEA_tmb=d2CoZiA at mail.gmail.com<mailto:d2CoZiA at mail.gmail.com>>
Content-Type: text/plain; charset="iso-8859-1"
Antony;
rabbitmq-c has very recently added an API that allows for specifying a
timeout when waiting for frames from the broker (see
https://github.com/alanxz/rabbitmq-c/blob/master/librabbitmq/amqp.h#L495 ).
With this API is possible to implement something with a similar effect.
I will warn you that this is only in the development branch and thus hasn't
received the same amount of testing that tagged versions have, but I would
encourage you to try it out and report back any issues that come up.
-Alan
On Tue, Jun 25, 2013 at 4:56 AM, antony <antony.pulicken at gmail.com<mailto:antony.pulicken at gmail.com>> wrote:
> Hi,
>
> This is my scenario:
>
> I have multiple instances of listeners waiting on different temp queues. In
> some of the cases, there won't be any messages published to these temp
> queues and I want these listeners to timeout so that it can free up the
> connections.
>
> I guess in Java, we can achieve this by using the overloaded method in the
> QueueingConsumer class that takes timeout as a parameter as shown below:
>
>
> /QueueingConsumer.Delivery delivery = null;
> long timeout = 2 * 60 * 1000; // 2 minutes in milliseconds
> delivery = queuingConsumer.nextDelivery(*timeout*);
> if (delivery == null) {
> // shut down your consumer here - no events arrived
> // before the timeout was reached
> }
> else {
> // process the delivered message here
> }/
>
>
> 1) Does RabbitMQ C client also has a similar feature ?
>
> 2) Are there better approaches to handle this timeouts ?
>
> Please let me know.
>
> Thanks and Regards,
> Antony.
>
>
>
>
> --
> View this message in context:
> http://rabbitmq.1065348.n5.nabble.com/RabbitMQ-Consumer-timeout-tp27590.html
> Sent from the RabbitMQ mailing list archive at Nabble.com.
> _______________________________________________
> rabbitmq-discuss mailing list
> rabbitmq-discuss at lists.rabbitmq.com<mailto:rabbitmq-discuss at lists.rabbitmq.com>
> https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20130625/3468600c/attachment-0001.htm>
------------------------------
Message: 3
Date: Tue, 25 Jun 2013 22:24:26 -0700 (PDT)
From: thomas <taihock at gmail.com<mailto:taihock at gmail.com>>
Subject: Re: [rabbitmq-discuss] RabbitMQ's Mirroring results in weird
behavior when slave goes down
To: rabbitmq-discuss at lists.rabbitmq.com<mailto:rabbitmq-discuss at lists.rabbitmq.com>
Message-ID: <1372224266044-27611.post at n5.nabble.com<mailto:1372224266044-27611.post at n5.nabble.com>>
Content-Type: text/plain; charset=us-ascii
Hi Tim,
Thanks for the reply. I am using rabbitmq 3.1.1 with autoheal for
cluster_partition_handling.
My test is just to find out the behavior of rabbitmq master should its slave
goes down. For my case, initially I will have rabbit at A, rabbit at B and
rabbit at C running. Then I will send out messages from client to rabbit at A
which is the master throughout the test. Soon after the client started to
send out messages, I will bring down a slave which in my case is rabbit at B by
closing its network connection.
I have changed the net_ticktime to 60 and carry out the same test using the
same code but this time round i increases the number of messages to 500,000
so as to better observe the behavior of sending messages for more than a
minute.
My observation is still similar to that of my previous test where
net_ticktime is 2.
Approximately 10 seconds after I shut down the network connection of
rabbit at B, the sending of messages to rabbit at A comes to a pause for close to
a minute and then continues as per normal.
Trying the same test but with no mirroring does not have any disruption.
By the way, I have try out the same test for rabbitmq mirroring except that
there will be a delay of 10ms between sending every message and there was no
disruption.
>From my observation it can be seen that RabbitMQ server will exhibit weird
behavior under stress condition when a slave goes down. I presume this weird
behavior will only surface under stress condition.
Is there a rough estimate for the maximum load that a RabbitMQ server can
take per second for mirroring? Thanks.
--
View this message in context: http://rabbitmq.1065348.n5.nabble.com/RabbitMQ-s-Mirroring-results-in-weird-behavior-when-slave-goes-down-tp27572p27611.html
Sent from the RabbitMQ mailing list archive at Nabble.com.
------------------------------
Message: 4
Date: Wed, 26 Jun 2013 12:19:53 +0400
From: Michael Klishin <michael.s.klishin at gmail.com<mailto:michael.s.klishin at gmail.com>>
Subject: Re: [rabbitmq-discuss] Determining Required Erlang Version
for Upgrade
To: Discussions about RabbitMQ <rabbitmq-discuss at lists.rabbitmq.com<mailto:rabbitmq-discuss at lists.rabbitmq.com>>
Message-ID:
<CAE3HoVQG8o6uiMGCU2RmasAv3qVObSF9EdN-+YKqe6-Scjd+-w at mail.gmail.com<mailto:CAE3HoVQG8o6uiMGCU2RmasAv3qVObSF9EdN-%2BYKqe6-Scjd%2B-w at mail.gmail.com>>
Content-Type: text/plain; charset="utf-8"
2013/6/25 Richard Raseley <richard at raseley.com<mailto:richard at raseley.com>>
> I am in the process of drafting a plan to upgrade several 3.0.x instances
> to 3.1.2. I cannot seem to find in the release notes the specific version
> of Erlang that is required for the new versions.
Richard,
I don't think the version requirement changed between 3.0.x and 3.1.x.
--
MK
http://github.com/michaelklishin
http://twitter.com/michaelklishin
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20130626/45de3926/attachment-0001.htm>
------------------------------
Message: 5
Date: Wed, 26 Jun 2013 10:22:40 +0100
From: Emile Joubert <emile at rabbitmq.com<mailto:emile at rabbitmq.com>>
Subject: Re: [rabbitmq-discuss] rabbitmq
Error:{cannot_discover_cluster, "the nodes provided are either offline
or not running"}
To: Luis Rodriguez <lrodriguez1995 at yahoo.com<mailto:lrodriguez1995 at yahoo.com>>, Discussions about
RabbitMQ <rabbitmq-discuss at lists.rabbitmq.com<mailto:rabbitmq-discuss at lists.rabbitmq.com>>
Message-ID: <51CAB2E0.9010805 at rabbitmq.com<mailto:51CAB2E0.9010805 at rabbitmq.com>>
Content-Type: text/plain; charset=ISO-8859-1
Hi
On 25/06/13 18:38, Luis Rodriguez wrote:
> I am running a windows host machine along with three vm's with centos
> 5.9 , rabbitmq-server-3.0.3, and erlang installed on all three nodes.
Make sure all nodes are using the same version of Erlang.
> (I am able to ping between nodes fine)
A number of TCP ports need to be open between cluster nodes, which ping
doesn't guarantee. See
http://www.rabbitmq.com/clustering.html#firewall
> the cookie hash however, doesn't change regardless of how many times
> I stop (rabbitmqctl stop) and start.. I don't really know what I'm
> missing.
If you change the .erlang.cookie and the cookie hash doesn't change then
you are probably copying to the wrong directory. You can discover a
running node's own idea of where its home directory is by executing
rabbitmqctl eval 'init:get_argument(home).'
Also take note that the cookie and the cookie hash will not be the same.
The hash is displayed in diagnostic output instead of the cookie as a
security precaution. If you have all the cookies synchronised correctly
then the hashes will also be the same.
> when I do rabbitctl join_cluster I get: rabbitmq
> Error:{cannot_discover_cluster,"the nodes provided are either offline
> or not running"}
As per the troubleshooting guide
http://www.rabbitmq.com/troubleshooting.html
if that message is accompanied by a broker logfile entry "Connection
attempt from disallowed node" then the erlang cookies are not set up
correctly. Otherwise the problem is network related.
-Emile
------------------------------
Message: 6
Date: Wed, 26 Jun 2013 10:25:52 +0100
From: Emile Joubert <emile at rabbitmq.com<mailto:emile at rabbitmq.com>>
Subject: Re: [rabbitmq-discuss] Determining Required Erlang Version
for Upgrade
To: Discussions about RabbitMQ <rabbitmq-discuss at lists.rabbitmq.com<mailto:rabbitmq-discuss at lists.rabbitmq.com>>
Message-ID: <51CAB3A0.8070707 at rabbitmq.com<mailto:51CAB3A0.8070707 at rabbitmq.com>>
Content-Type: text/plain; charset=ISO-8859-1
Hi Richard,
On 25/06/13 20:39, Richard Raseley wrote:
> Would someone please direct me to a version requirement matrix (assuming
> one exists)?
See http://www.rabbitmq.com/which-erlang.html
-Emile
------------------------------
Message: 7
Date: Wed, 26 Jun 2013 10:35:45 +0100
From: Emile Joubert <emile at rabbitmq.com<mailto:emile at rabbitmq.com>>
Subject: Re: [rabbitmq-discuss] RabbitMQ won't start after the disk
went full
To: Discussions about RabbitMQ <rabbitmq-discuss at lists.rabbitmq.com<mailto:rabbitmq-discuss at lists.rabbitmq.com>>
Cc: photon <faycal.noushi at gmail.com<mailto:faycal.noushi at gmail.com>>
Message-ID: <51CAB5F1.20202 at rabbitmq.com<mailto:51CAB5F1.20202 at rabbitmq.com>>
Content-Type: text/plain; charset=ISO-8859-1
Hi,
On 25/06/13 14:38, photon wrote:
> I have an issue concerning RabbitMQ. Once my disk got full, RabbitMQ became
> unusable. This didn't change even after i made the disk usage drop
> dramatically.
You either need to free up more space or lower the disk_free_limit. See
the configuration guide:
http://www.rabbitmq.com/configure.html
> Error description:
> {error,corrupt_cluster_status_files,[]}
This is a separate problem and can be solved either by removing the
entire Mnesia directory or by removing the cluster_nodes.config file in
the Mnesia directory and reclustering if necessary.
> =INFO REPORT==== 25-Jun-2013::09:10:49 ===
> Disk free space insufficient. Free bytes:645095424 Limit:1000000000
This shows that there is insufficient free disk space. The broker will
not accept any messages until there is enough free disk space. See
http://www.rabbitmq.com/memory.html#diskfreesup
-Emile
------------------------------
Message: 8
Date: Wed, 26 Jun 2013 11:38:36 +0100
From: Tim Watson <tim at rabbitmq.com<mailto:tim at rabbitmq.com>>
Subject: Re: [rabbitmq-discuss] RabbitMQ's Mirroring results in weird
behavior when slave goes down
To: Discussions about RabbitMQ <rabbitmq-discuss at lists.rabbitmq.com<mailto:rabbitmq-discuss at lists.rabbitmq.com>>
Message-ID: <2F050250-E7F1-4543-B3C3-239885F7C9CB at rabbitmq.com<mailto:2F050250-E7F1-4543-B3C3-239885F7C9CB at rabbitmq.com>>
Content-Type: text/plain; charset=us-ascii
Hi Thomas,
Here are my thoughts (inline below). I'm not the greatest clustering/HA expert, but from what I can gather, I think what you're seeing might be a result of the combination of a network partition and the `autoheal' mode.
On 26 Jun 2013, at 06:24, thomas wrote:
> Hi Tim,
>
> Thanks for the reply. I am using rabbitmq 3.1.1 with autoheal for
> cluster_partition_handling.
>
> My test is just to find out the behavior of rabbitmq master should its slave
> goes down.
This is clearly documented on the website - RabbitMQ clusters /do not/ handle network partitions well. Things will probably go wrong. If the only issue you're running into is a busy producer getting blocked then that's great - the worst cases could have involved lost messages and all sorts.
Now, the intention with HA is *not* to go belly up in the face of disappearing slaves - quite the opposite. Quite why the master is suddenly blocking the producer I'm not sure - I'll discuss this with the rest of the team but it doesn't sound quite right to me.
> Approximately 10 seconds after I shut down the network connection of
> rabbit at B, the sending of messages to rabbit at A comes to a pause for close to
> a minute and then continues as per normal.
>
Just to be clear, in a RabbitMQ cluster there is no concept of a master node - only mirror queues introduce the master/slave concept, and then only for queues that are mirrored. Each mirrored queue's master can end up on different nodes, depending on which you were connect to when it was declared.
> Trying the same test but with no mirroring does not have any disruption.
>
That is not surprising, since the queue you're connected to isn't trying to replicate to any other nodes.
> By the way, I have try out the same test for rabbitmq mirroring except that
> there will be a delay of 10ms between sending every message and there was no
> disruption.
>
Can you share your test code? I'm surprised there is such a big difference, but still... If you've got a decent automated test that consistently replicates this failure mode and you're able to share it, we might be able to use it to make improvements to partition handling in the future. If not code, a test script (and description of how you're disabling the network link) with precise steps would do.
> From my observation it can be seen that RabbitMQ server will exhibit weird
> behavior under stress condition when a slave goes down. I presume this weird
> behavior will only surface under stress condition.
>
As I've stated before, clustering is not designed to handle network partitions. In previous versions of rabbit, when a partition occurred it was up to a system administrator to handle it. This usually involved stopping some (or all) of the nodes and restarting them. The 'autoheal' feature simply attempts to replicate what a sysadmin might do, and therefore (whilst cluster nodes are restarting) there can be delays. The intention is not to block the master (or producers), but depending on how the cluster partition is handled, that might be one of the potential side effects.
We will certainly take a look to see if this is avoidable - I can't comment on that right now - and being able to repeat the tests could help with that too.
> Is there a rough estimate for the maximum load that a RabbitMQ server can
> take per second for mirroring? Thanks.
>
I'm not sure whether mirroring is really the problem here. Mirrored queues can handle plenty of load, although there /is/ some performance slow down due to replication (and esp. so if you're using publisher confirms) but there is a not a specific limit - this is hardware/resource and use-case dependent. I suspect the problem /you/ have could be a combination of network partition and cluster partition resolution (i.e., autoheal) mode.
Based on http://www.rabbitmq.com/partitions.html, what I'd suggest is that you change 'autoheal' to 'ignore' and try your test again. That will either confirm or eliminate my suspicion quickly enough.
Cheers,
Tim
------------------------------
_______________________________________________
rabbitmq-discuss mailing list
rabbitmq-discuss at lists.rabbitmq.com<mailto:rabbitmq-discuss at lists.rabbitmq.com>
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
End of rabbitmq-discuss Digest, Vol 73, Issue 27
************************************************
________________________________
DISCLAIMER:
This email (including any attachments) is intended for the sole use of the intended recipient/s and may contain material that is CONFIDENTIAL AND PRIVATE COMPANY INFORMATION. Any review or reliance by others or copying or distribution or forwarding of any or all of the contents in this message is STRICTLY PROHIBITED. If you are not the intended recipient, please contact the sender by email and delete all copies; your cooperation in this regard is appreciated.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20130705/7063923c/attachment.htm>
More information about the rabbitmq-discuss
mailing list