From james at archfashion.com.au Tue Apr 1 03:48:08 2014 From: james at archfashion.com.au (James Andrew-Smith) Date: Tue, 1 Apr 2014 13:48:08 +1100 Subject: [rabbitmq-discuss] Do WebStomp/Sock.JS require a constant connection? Message-ID: Hi, I've set up a Sock.js/Web Stomp server, everything works, but I don't understand how to deal with the break in connection when a user navigates our site. If a user receives a message when the page is reloading (or if the browser is reconnected), when they reconnect the message isn't coming through. I feel like I am missing something obvious. James -------------- next part -------------- An HTML attachment was scrubbed... URL: From carl.hoerberg at gmail.com Tue Apr 1 05:10:14 2014 From: carl.hoerberg at gmail.com (carlhoerberg) Date: Mon, 31 Mar 2014 21:10:14 -0700 (PDT) Subject: [rabbitmq-discuss] cluster meltdown Message-ID: <1396325414968-34453.post@n5.nabble.com> We have a two node cluster, an AWS node froze for us, we could neither start or stop it. That made the first node unresponsive to mgmt db actions, all API request timed out. We restart the first node but a lot of queues are then inaccessible: =ERROR REPORT==== 1-Apr-2014::02:58:03 === connection <0.5559.279>, channel 1 - soft error: {amqp_error,not_found, "home node 'rabbit at node2' of durable queue 'celery' in vhost 'vhost1' is down or inaccessible", 'queue.declare'} We issue rabbitmqctl forget_cluster_node rabbit at node2 as we still can't access node2. Node1 continue to report a lot of "home node of queue is down". Node2 has now restarted, but can't join the cluster. Is there a way to rejoin the cluster without resetting? We reset node2 and tries to join_cluster again but with the following result: Clustering node 'rabbit at node2' with 'rabbit at node1' ... ...done (already_member). node2# rabbitmqctl cluster_status Cluster status of node 'rabbit at node2' ... [{nodes,[{disc,['rabbit at node2']}]}, {running_nodes,['rabbit at node2']}, {partitions,[]}] ...done. But start_app doesn't join node1. node1# rabbitmqctl cluster_status Cluster status of node 'rabbit at node1' ... [{nodes,[{disc,['rabbit at node1','rabbit at node2']}]}, {running_nodes,['rabbit at node1']}, {partitions,[]}] ...done. node2# rabbitmqctl update_cluster_nodes rabbit at node1 Now node2 understands that it's clustered with node1 and with start_app it starts and joins node1. RabbitMQ 3.2.3, Erlang R16B03-1 -- View this message in context: http://rabbitmq.1065348.n5.nabble.com/cluster-meltdown-tp34453.html Sent from the RabbitMQ mailing list archive at Nabble.com. From mklishin at gopivotal.com Tue Apr 1 05:32:44 2014 From: mklishin at gopivotal.com (Michael Klishin) Date: Tue, 1 Apr 2014 08:32:44 +0400 Subject: [rabbitmq-discuss] Do WebStomp/Sock.JS require a constant connection? In-Reply-To: References: Message-ID: On 1 April 2014 at 06:49:23, James Andrew-Smith (james at archfashion.com.au) wrote: > > I've set up a Sock.js/Web Stomp server, everything works, but > I don't understand how to deal with the break in connection when > a user navigates our site. Build a navigation that does not require page reloading (?single page apps?).? -- MK Software Engineer, Pivotal/RabbitMQ From 1517427523 at qq.com Tue Apr 1 06:05:51 2014 From: 1517427523 at qq.com (=?ISO-8859-1?B?dGhvbWFz?=) Date: Tue, 1 Apr 2014 13:05:51 +0800 Subject: [rabbitmq-discuss] How long interval fetching message from MQ SERVER? Message-ID: I have a question in RPC callBack queue,if i use while(true) to keep receiving message,How long interval fetching message from MQ SERVER? -------------- next part -------------- An HTML attachment was scrubbed... URL: From mklishin at gopivotal.com Tue Apr 1 06:08:05 2014 From: mklishin at gopivotal.com (Michael Klishin) Date: Tue, 1 Apr 2014 09:08:05 +0400 Subject: [rabbitmq-discuss] How long interval fetching message from MQ SERVER? In-Reply-To: References: Message-ID: On 1 April 2014 at 09:07:00, thomas (1517427523 at qq.com) wrote: > > I have a question in RPC callBack queue,if i use while(true) > to keep receiving message,How long interval fetching message > from MQ SERVER? There is no timeout in the protocol (and if you consume with basic.consume, it does not really make sense). So it depends on your client library.? -- MK Software Engineer, Pivotal/RabbitMQ From 1517427523 at qq.com Tue Apr 1 06:16:48 2014 From: 1517427523 at qq.com (=?gb18030?B?dGhvbWFz?=) Date: Tue, 1 Apr 2014 13:16:48 +0800 Subject: [rabbitmq-discuss] =?gb18030?b?u9i4tKO6ICBIb3cgbG9uZyBpbnRlcnZh?= =?gb18030?q?l_fetching_message_from_MQSERVER=3F?= In-Reply-To: References: Message-ID: Thanks for answering my question! pls check my code below: String message = client.getNextMessage(); if (message != null && !message.equals("")) { client.sendMessage(imqRec.handler(message)); } If I use while(true) loop continuously monitor RPC queue, it would be very resource intensive, whether you grab a message per sec or anything else? ------------------ ???? ------------------ ???: "Michael Klishin";; ????: 2014?4?1?(???) ??1:08 ???: "Discussions about RabbitMQ"; ??: Re: [rabbitmq-discuss] How long interval fetching message from MQSERVER? On 1 April 2014 at 09:07:00, thomas (1517427523 at qq.com) wrote: > > I have a question in RPC callBack queue,if i use while(true) > to keep receiving message,How long interval fetching message > from MQ SERVER? There is no timeout in the protocol (and if you consume with basic.consume, it does not really make sense). So it depends on your client library. -- MK Software Engineer, Pivotal/RabbitMQ _______________________________________________ rabbitmq-discuss mailing list rabbitmq-discuss at lists.rabbitmq.com https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: From 1517427523 at qq.com Tue Apr 1 06:23:19 2014 From: 1517427523 at qq.com (=?gb18030?B?dGhvbWFz?=) Date: Tue, 1 Apr 2014 13:23:19 +0800 Subject: [rabbitmq-discuss] =?gb18030?q?whether_you_grab_a_message_per_sec?= =?gb18030?q?_or_anything_else=A3=BF?= Message-ID: pls check my code below: String message = client.getNextMessage(); if (message != null && !message.equals("")) { client.sendMessage(imqRec.handler(message)); } If I use while(true) loop continuously monitor RPC queue, it would be very resource intensive, whether you grab a message per sec or anything else? -------------- next part -------------- An HTML attachment was scrubbed... URL: From 1517427523 at qq.com Tue Apr 1 06:33:25 2014 From: 1517427523 at qq.com (=?ISO-8859-1?B?dGhvbWFz?=) Date: Tue, 1 Apr 2014 13:33:25 +0800 Subject: [rabbitmq-discuss] Thread more question Message-ID: my system have a problem we use thread to send message to mq,and the thread will sleep when it have no message to send,but i found that linux thread is become more and more,what happend? -------------- next part -------------- An HTML attachment was scrubbed... URL: From james at archfashion.com.au Tue Apr 1 06:43:49 2014 From: james at archfashion.com.au (James Andrew-Smith) Date: Tue, 1 Apr 2014 16:43:49 +1100 Subject: [rabbitmq-discuss] Do WebStomp/Sock.JS require a constant connection? In-Reply-To: References: Message-ID: If a single page app isn't feasible - are there any alternatives? Can I make an explicit AJAX call when the page loads or something? Or should I keep a history of recent messages in a cache somewhere? On Tue, Apr 1, 2014 at 3:32 PM, Michael Klishin wrote: > On 1 April 2014 at 06:49:23, James Andrew-Smith (james at archfashion.com.au) > wrote: > > > I've set up a Sock.js/Web Stomp server, everything works, but > > I don't understand how to deal with the break in connection when > > a user navigates our site. > > Build a navigation that does not require page reloading > ("single page apps"). > -- > MK > > Software Engineer, Pivotal/RabbitMQ > _______________________________________________ > rabbitmq-discuss mailing list > rabbitmq-discuss at lists.rabbitmq.com > https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mklishin at gopivotal.com Tue Apr 1 07:47:19 2014 From: mklishin at gopivotal.com (Michael Klishin) Date: Tue, 1 Apr 2014 10:47:19 +0400 Subject: [rabbitmq-discuss] Thread more question In-Reply-To: References: Message-ID: On 1 April 2014 at 09:34:34, thomas (1517427523 at qq.com) wrote: > > we use thread to send message to mq,and the thread will sleep > when it have no message to send,but i found that linux thread is > become more and more,what happend? It?s not really possible to suggest something without seeing your code. if you open a new connection per temporarily created thread, then make sure you close the connection. In general, c onnections to RabbitMQ are supposed to be long lived. -- MK Software Engineer, Pivotal/RabbitMQ From simon at rabbitmq.com Tue Apr 1 10:02:34 2014 From: simon at rabbitmq.com (Simon MacMullen) Date: Tue, 01 Apr 2014 10:02:34 +0100 Subject: [rabbitmq-discuss] RabbitMQ 3.2.4 keeps crashing In-Reply-To: References: Message-ID: <533A80AA.8030805@rabbitmq.com> On 31/03/2014 7:42PM, Sergey Dobrovolskiy wrote: > that's the result from sasl log: Is there anything in the main log? That looks like Mnesia shutting down in response to an earlier error. Cheers, Simon -- Simon MacMullen RabbitMQ, Pivotal From stammailbox at gmail.com Tue Apr 1 12:02:10 2014 From: stammailbox at gmail.com (rails) Date: Tue, 1 Apr 2014 14:02:10 +0300 Subject: [rabbitmq-discuss] Is there a way to lock the ability to create rabbitmq amqp entities from clients Message-ID: I am trying to re-org the usage of rabbit in my organization. I have one main system that many clients subscribe to its changes. My goal is to prevent abuse to the Rabbitmq broker that will be caused due to bad usage. 1. Is there a way to create all the queues in advance (i.e. using load-definitions ) and than not to allow any more queues creation. All of my queues are pretty much expexted. 2. Is there a way to control number of other amqp entities? An un-experienced devleoper could open too much connections or too many channels. Can I prevent that? Thanks. -------------- next part -------------- An HTML attachment was scrubbed... URL: From mklishin at gopivotal.com Tue Apr 1 12:05:59 2014 From: mklishin at gopivotal.com (Michael Klishin) Date: Tue, 1 Apr 2014 15:05:59 +0400 Subject: [rabbitmq-discuss] Is there a way to lock the ability to create rabbitmq amqp entities from clients In-Reply-To: References: Message-ID: On 1 April 2014 at 15:04:25, rails (stammailbox at gmail.com) wrote: > > 1. Is there a way to create all the queues in advance (i.e. using > load-definitions ) and than not to allow any more queues creation. > All of my queues are pretty much expexted. See?http://www.rabbitmq.com/access-control.html. > 2. Is there a way to control number of other amqp entities? > An un-experienced devleoper could open too much connections > or too many channels. No but you can monitor connections and channels via HTTP API. This is what hosted RabbitMQ services (e.g. CloudAMQP) ?do. -- MK Software Engineer, Pivotal/RabbitMQ From dilip.aim11 at gmail.com Tue Apr 1 12:49:09 2014 From: dilip.aim11 at gmail.com (dilip.aim11) Date: Tue, 1 Apr 2014 04:49:09 -0700 (PDT) Subject: [rabbitmq-discuss] Alternate exchange in Rabbit MQ Message-ID: <1396352949431-34465.post@n5.nabble.com> Hi All , I am trying to use Alternate exchange Property of Rabbit MQ - This is My Step - 1 I created exchange Called - AEX 2 Create Q - TestQ 3 Bind with RK - RK Now - 4 - Created Another Exchange - EmptyEX - in Alternate exchange Text Box given - AEX -------------- Both Exchange are Direct - Now when i am simply publishing the message to EmptyEX from management window - i AM GETTING MESSAGE - Message published, but not routed. but my message is not coming to Q - TestQ i am following http://www.rabbitmq.com/ae.html Please help me on this where i am wrong . Thanks . -- View this message in context: http://rabbitmq.1065348.n5.nabble.com/Alternate-exchange-in-Rabbit-MQ-tp34465.html Sent from the RabbitMQ mailing list archive at Nabble.com. From mklishin at gopivotal.com Tue Apr 1 13:03:40 2014 From: mklishin at gopivotal.com (Michael Klishin) Date: Tue, 1 Apr 2014 16:03:40 +0400 Subject: [rabbitmq-discuss] Alternate exchange in Rabbit MQ In-Reply-To: <1396352949431-34465.post@n5.nabble.com> References: <1396352949431-34465.post@n5.nabble.com> Message-ID: ?On 1 April 2014 at 15:50:11, dilip.aim11 (dilip.aim11 at gmail.com) wrote: > > 1 I created exchange Called - AEX > 2 Create Q - TestQ > 3 Bind with RK - RK > > Now - > 4 - Created Another Exchange - EmptyEX - > in Alternate exchange Text Box given - AEX > > -------------- > Both Exchange are Direct - > Now when i am simply publishing the message to EmptyEX from management > window - > i AM GETTING MESSAGE - Message published, but not routed. > but my message is not coming to Q - TestQ > i am following > http://www.rabbitmq.com/ae.html > > Please help me on this where i am wrong . Can you post some relevant parts of your code? The problem can be with AEX bindings to TestQ but can also be elsewhere. -- MK Software Engineer, Pivotal/RabbitMQ From dilip.aim11 at gmail.com Tue Apr 1 13:16:05 2014 From: dilip.aim11 at gmail.com (dilip.aim11) Date: Tue, 1 Apr 2014 05:16:05 -0700 (PDT) Subject: [rabbitmq-discuss] Alternate exchange in Rabbit MQ In-Reply-To: References: <1396352949431-34465.post@n5.nabble.com> Message-ID: <1396354565964-34467.post@n5.nabble.com> Hi i am using WEB UI so there is no code involved *The problem can be with AEX bindings to TestQ but can also be elsewhere* i already verified if i m publishing message from AEX . message is coming to TestQ -- View this message in context: http://rabbitmq.1065348.n5.nabble.com/Alternate-exchange-in-Rabbit-MQ-tp34465p34467.html Sent from the RabbitMQ mailing list archive at Nabble.com. From mklishin at gopivotal.com Tue Apr 1 13:22:10 2014 From: mklishin at gopivotal.com (Michael Klishin) Date: Tue, 1 Apr 2014 16:22:10 +0400 Subject: [rabbitmq-discuss] Alternate exchange in Rabbit MQ In-Reply-To: <1396354565964-34467.post@n5.nabble.com> References: <1396352949431-34465.post@n5.nabble.com> <1396354565964-34467.post@n5.nabble.com> Message-ID: On 1 April 2014 at 16:17:09, dilip.aim11 (dilip.aim11 at gmail.com) wrote: > > i already verified if i m publishing message from AEX . message > is coming > to TestQ Try with two exchanges of type fanout first. It should make no difference if you publish to AEX (bound to a queue) or to another exchange that sets up AEX as an alternate exchange. So perhaps you have a routing key mismatch with direct exchanges.? -- MK Software Engineer, Pivotal/RabbitMQ From dilip.aim11 at gmail.com Tue Apr 1 13:38:55 2014 From: dilip.aim11 at gmail.com (dilip.aim11) Date: Tue, 1 Apr 2014 05:38:55 -0700 (PDT) Subject: [rabbitmq-discuss] Alternate exchange in Rabbit MQ In-Reply-To: References: <1396352949431-34465.post@n5.nabble.com> <1396354565964-34467.post@n5.nabble.com> Message-ID: <1396355935281-34469.post@n5.nabble.com> Hi , if for EmptyEX i m publishing message with RK - so message is going to TestQ. is it AE ? my requirement something . that my message should go all the Queue Of AEX . i can not make AEX as Fanout . There is anything else from which i can achieve this ?? Thnkx for ur suggestion .. -- View this message in context: http://rabbitmq.1065348.n5.nabble.com/Alternate-exchange-in-Rabbit-MQ-tp34465p34469.html Sent from the RabbitMQ mailing list archive at Nabble.com. From mklishin at gopivotal.com Tue Apr 1 13:49:36 2014 From: mklishin at gopivotal.com (Michael Klishin) Date: Tue, 1 Apr 2014 16:49:36 +0400 Subject: [rabbitmq-discuss] Alternate exchange in Rabbit MQ In-Reply-To: <1396355935281-34469.post@n5.nabble.com> References: <1396352949431-34465.post@n5.nabble.com> <1396354565964-34467.post@n5.nabble.com> <1396355935281-34469.post@n5.nabble.com> Message-ID: On 1 April 2014 at 16:39:57, dilip.aim11 (dilip.aim11 at gmail.com) wrote: > > if for EmptyEX i m publishing message with RK - > so message is going to TestQ. > is it AE ? A message published to AE with routing key RK should be routed to TestQ. Then, when you publish it to another exchange *and* it cannot be routed anywhere, it will be [re]published to AE. > my requirement something . that my message should go all the Queue > Of AEX . > i can not make AEX as Fanout . Do you mean ?shoult not go to all the queues?? > There is anything else from which i can achieve this ?? My guess is that if your exchange E performs the routing, then messages never reach AE. In any case, my suggestion to use fanouts was to see if the problem is with routing or your exchange attributes (alternate exchange setup). Fanouts completely avoid all the issues with routing keys.? -- MK Software Engineer, Pivotal/RabbitMQ From johnwdavis at gmail.com Tue Apr 1 16:57:36 2014 From: johnwdavis at gmail.com (John Davis) Date: Tue, 1 Apr 2014 08:57:36 -0700 (PDT) Subject: [rabbitmq-discuss] Installation Guide for rabbitmq Java client missing Message-ID: <48680c81-2ba3-49b0-85ae-abfad9e729d2@googlegroups.com> If developer is trying to prepare a Windows 7 Pro laptop to run a Java client app (which connects to a rabbitmq broker on another Windows server), I assume the following: - JVM must be installed - RabbitMQ Java client must be installed. - The distribution package for the app should be on the client server. Obviously, the client's Windows machine would need to be an Erlang node. Would Erland need to be downloaded and installed prior to installing rabbitmq Java client? Although it is likely fairly simple, I think that an Installation guide for Java Client would be a welcome addition to the documentation (similar to the installation guide provided for rabbitmq Server). -------------- next part -------------- An HTML attachment was scrubbed... URL: From jkuch at gopivotal.com Tue Apr 1 17:09:30 2014 From: jkuch at gopivotal.com (Jerry Kuch) Date: Tue, 1 Apr 2014 09:09:30 -0700 Subject: [rabbitmq-discuss] Installation Guide for rabbitmq Java client missing In-Reply-To: <48680c81-2ba3-49b0-85ae-abfad9e729d2@googlegroups.com> References: <48680c81-2ba3-49b0-85ae-abfad9e729d2@googlegroups.com> Message-ID: On Tue, Apr 1, 2014 at 8:57 AM, John Davis wrote: > If developer is trying to prepare a Windows 7 Pro laptop to run a Java > client app (which connects to a rabbitmq broker on another Windows server), > I assume the following: > > - JVM must be installed > - RabbitMQ Java client must be installed. > - The distribution package for the app should be on the client server. > > I don't quite know what "the client server" means in the third bullet point above. > Obviously, the client's Windows machine would need to be an Erlang node. > False: The RabbitMQ Java client is just a pure Java library that speaks the AMQP wire protocol. Nothing Erlang-ish needs to be on the machine with it. Defining a language agnostic wire protocol and command set is one of AMQP's strengths, and a key reason why it makes polyglot systems, such as in your case a Java client application talking to a message broker, RabbitMQ, that happens to be written in Erlang, viable. > Would Erland need to be downloaded and installed prior to installing > rabbitmq Java client? > No, as per my prior paragraph, so there's less work for you to do. > Although it is likely fairly simple, I think that an Installation guide > for Java Client would be a welcome addition to the documentation (similar > to the installation guide provided for rabbitmq Server). > It's just a plain old JAR file. You can drop it on your classpath however you like, and pull it into your project via Maven, Ivy, SBT or the like from a suitable repository, etc. Best regards, Jerry -------------- next part -------------- An HTML attachment was scrubbed... URL: From ambatinr at hotmail.com Tue Apr 1 20:51:11 2014 From: ambatinr at hotmail.com (anate) Date: Tue, 1 Apr 2014 12:51:11 -0700 (PDT) Subject: [rabbitmq-discuss] How do I delete a persistent message after successul consuming Message-ID: <1396381871418-34473.post@n5.nabble.com> I have a durable and persistent message and trying to consume it and insert into the database. If everything goes well(no exception) then I would like to delete a message from queue. I don't see any examples online using C#. Could you please let me know? try{ using (IConnection connection = connectionFactory.CreateConnection()) { using (var channel = connection.CreateModel()) { channel.QueueDeclare("hello-world-queue", false, true, false, null); var consumer = new QueueingBasicConsumer(channel); channel.BasicConsume("hello-world-queue", true, consumer); while (true) { var ea = (BasicDeliverEventArgs)consumer.Queue.Dequeue(); //Insert data into database // delete message from queue } } } } catch(ex as Exception) { // show error message } -- View this message in context: http://rabbitmq.1065348.n5.nabble.com/How-do-I-delete-a-persistent-message-after-successul-consuming-tp34473.html Sent from the RabbitMQ mailing list archive at Nabble.com. From videlalvaro at gmail.com Tue Apr 1 20:56:16 2014 From: videlalvaro at gmail.com (Alvaro Videla) Date: Tue, 1 Apr 2014 21:56:16 +0200 Subject: [rabbitmq-discuss] How do I delete a persistent message after successul consuming In-Reply-To: <1396381871418-34473.post@n5.nabble.com> References: <1396381871418-34473.post@n5.nabble.com> Message-ID: Hi, See "Message acknowledgment" here: http://www.rabbitmq.com/tutorials/tutorial-two-dotnet.html Regards, Alvaro On Tue, Apr 1, 2014 at 9:51 PM, anate wrote: > I have a durable and persistent message and trying to consume it and insert > into the database. If everything goes well(no exception) then I would like > to delete a message from queue. I don't see any examples online using C#. > Could you please let me know? > > try{ > using (IConnection connection = > connectionFactory.CreateConnection()) > { > using (var channel = connection.CreateModel()) > { > channel.QueueDeclare("hello-world-queue", false, true, > false, null); > > var consumer = new QueueingBasicConsumer(channel); > channel.BasicConsume("hello-world-queue", true, > consumer); > > while (true) > { > var ea = > (BasicDeliverEventArgs)consumer.Queue.Dequeue(); > //Insert data into database > // delete message from queue > } > } > } > } > catch(ex as Exception) > { > // show error message > } > > > > > > -- > View this message in context: http://rabbitmq.1065348.n5.nabble.com/How-do-I-delete-a-persistent-message-after-successul-consuming-tp34473.html > Sent from the RabbitMQ mailing list archive at Nabble.com. > _______________________________________________ > rabbitmq-discuss mailing list > rabbitmq-discuss at lists.rabbitmq.com > https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss From mklishin at gopivotal.com Tue Apr 1 20:55:55 2014 From: mklishin at gopivotal.com (Michael Klishin) Date: Tue, 1 Apr 2014 23:55:55 +0400 Subject: [rabbitmq-discuss] How do I delete a persistent message after successul consuming In-Reply-To: <1396381871418-34473.post@n5.nabble.com> References: <1396381871418-34473.post@n5.nabble.com> Message-ID: ?On 1 April 2014 at 23:52:13, anate (ambatinr at hotmail.com) wrote: > > I have a durable and persistent message and trying to consume > it and insert > into the database. If everything goes well(no exception) then > I would like > to delete a message from queue. I don't see any examples online > using C#. > Could you please let me know? Consumed and acknowledged messages can be considered deleted. You consume with automatic acknowledgements on, so as soon as you get a delivery, you can consider the message deleted. -- MK Software Engineer, Pivotal/RabbitMQ From ajkedar at gmail.com Wed Apr 2 05:40:24 2014 From: ajkedar at gmail.com (Amol Kedar) Date: Tue, 1 Apr 2014 21:40:24 -0700 (PDT) Subject: [rabbitmq-discuss] Patterns with RabbitMQ Message-ID: <903570fa-62d2-42c5-b491-068d60696c87@googlegroups.com> We are trying to implement message queueing for our multi-server and multi datacenter application (distributed ) I have a general question is it a good idea to have 1 big queue created with different routing keys based on where the server is location and can the consumers in those data centers intelligently only pick the messages from the queue that belong to them? if its confusing i can be bit more descriptive... -------------- next part -------------- An HTML attachment was scrubbed... URL: From ajkedar at gmail.com Wed Apr 2 05:41:28 2014 From: ajkedar at gmail.com (Amol Kedar) Date: Tue, 1 Apr 2014 21:41:28 -0700 (PDT) Subject: [rabbitmq-discuss] Network partitioning In-Reply-To: <96bb27bb-58bc-4db6-99f2-abaaacd02f63@googlegroups.com> References: <96bb27bb-58bc-4db6-99f2-abaaacd02f63@googlegroups.com> Message-ID: <33893f49-0b2f-42fe-8b90-0e1ff14ac51b@googlegroups.com> were you using the federation plugin? On Thursday, March 20, 2014 5:41:22 AM UTC-4, Geoffrey Samper wrote: > > Hi, > > > We have a cluster with 3 nodes (Rabbit 1, Rabbit 3, rabbit 4). Sometimes > we have problems of network partitioning. Over the day everything runs fine > but at night 3AM and 4AM arround we expierence network partitioning. Most > of the time between 3 and 1. What strange is the log of rabbit 1 is > contains "Generic server <0.263.0> terminating in the log". > > Can anyone help me with this issue. The logs are below > > > RABBIT 04 > > =========== > > INFO REPORT==== 20-Mar-2014::03:41:27 === > Mirrored-queue (queue 'ha.PageDeletedConsumer' in vhost 'A105'): Master > saw deaths of mirrors > > =INFO REPORT==== 20-Mar-2014::03:41:27 === > Mirrored-queue (queue 'ha.services_p14' in vhost 'A105'): Master > saw deaths of mirrors > > =INFO REPORT==== 20-Mar-2014::03:41:27 === > Mirrored-queue (queue 'ha.MNT_grbhstmntp01' in vhost 'A105'): Master > saw deaths of mirrors > > =INFO REPORT==== 20-Mar-2014::03:41:27 === > Mirrored-queue (queue 'ha.services_p05' in vhost 'A105'): Slave > saw deaths of mirrors > > =INFO REPORT==== 20-Mar-2014::03:41:27 === > Mirrored-queue (queue 'ha.services_p09' in vhost 'A105'): Master > saw deaths of mirrors > > Rabbit 03 > ===================== > =INFO REPORT==== 20-Mar-2014::03:41:27 === > rabbit on node rabbit at P01 down > > =ERROR REPORT==== 20-Mar-2014::03:41:27 === > Mnesia(rabbit at P03): ** ERROR ** mnesia_event got {inconsistent_database, > running_partitioned_network, rabbit at P01} > > =INFO REPORT==== 20-Mar-2014::03:41:27 === > Mirrored-queue (queue 'ha.services_p21' in vhost 'A105'): Slave > saw deaths of mirrors > > =INFO REPORT==== 20-Mar-2014::03:41:27 === > Mirrored-queue (queue 'ha.services_p22' in vhost 'A105'): Slave > saw deaths of mirrors > > =INFO REPORT==== 20-Mar-2014::03:41:27 === > > Mirrored-queue (queue 'ha.PageDeletedConsumer' in vhost 'A105.001'): Slave > saw deaths of mirrors > > =INFO REPORT==== 20-Mar-2014::03:41:27 === > Mirrored-queue (queue 'ha.services_p04' in vhost 'A105.001'): Slave > saw deaths of mirrors > > =INFO REPORT==== 20-Mar-2014::03:41:27 === > Mirrored-queue (queue 'ha.MNT_grbhstmntp02' in vhost 'A105'): Slave > saw deaths of mirrors > > =INFO REPORT==== 20-Mar-2014::03:41:27 === > Mirrored-queue (queue 'ha.services_p03' in vhost 'A105.001'): Slave > saw deaths of mirrors > > =INFO REPORT==== 20-Mar-2014::03:41:27 === > Mirrored-queue (queue 'ha.MNT_grbhstmntp01' in vhost 'A105.001'): Slave > saw deaths of mirrors > > =INFO REPORT==== 20-Mar-2014::03:41:27 === > Mirrored-queue (queue 'ha.services_p20' in vhost 'A105.001'): Slave > saw deaths of mirrors > > =INFO REPORT==== 20-Mar-2014::03:41:27 === > Mirrored-queue (queue 'ha.PageDeleteConsumer' in vhost 'A105.001'): Slave > saw deaths of mirrors > > Rabbit 01 > =================================== > > =INFO REPORT==== 20-Mar-2014::03:41:14 === > rabbit on node rabbit at P03 down > > =ERROR REPORT==== 20-Mar-2014::03:41:15 === > Mnesia(rabbit at P01): ** ERROR ** mnesia_event got {inconsistent_database, > running_partitioned_network, rabbit at P03} > > > =ERROR REPORT==== 20-Mar-2014::03:41:15 === > > ** Generic server <0.263.0> terminating > > ** Last message in was {'DOWN',#Ref<0.0.0.1692>,process,<4625.265.0>, > > noconnection} > > ** When Server state == {state, > > {31,<0.263.0>}, > > {{28,<4625.265.0>},#Ref<0.0.0.1692>}, > > {{25,<5158.263.0>},#Ref<0.0.0.1693>}, > > {resource,<<"A105">>,queue, > > <<"ha.services_p14">>}, > > rabbit_mirror_queue_slave, > > {32, > > [{{25,<5158.263.0>}, > > {view_member, > > {25,<5158.263.0>}, > > [], > > {31,<0.263.0>}, > > {28,<4625.265.0>}}}, > > {{28,<4625.265.0>}, > > {view_member, > > {28,<4625.265.0>}, > > [], > > {25,<5158.263.0>}, > > {31,<0.263.0>}}}, > > {{31,<0.263.0>}, > > {view_member, > > {31,<0.263.0>}, > > [], > > {28,<4625.265.0>}, > > {25,<5158.263.0>}}}]}, > > 0, > > [{{25,<5158.263.0>},{member,{[],[]},6,6}}, > > {{28,<4625.265.0>},{member,{[],[]},1,1}}, > > {{31,<0.263.0>},{member,{[],[]},0,0}}], > > [<0.262.0>], > > {[],[]}, > > [],0,undefined, > > #Fun} > > ** Reason for termination == > > ** {function_clause,[{orddict,fetch, > > [{31,<0.263.0>},[]], > > [{file,"orddict.erl"},{line,72}]}, > > {gm,check_neighbours,1,[]}, > > {gm,handle_info,2,[]}, > > {gen_server2,handle_msg,2,[]}, > > > {proc_lib,wake_up,3,[{file,"proc_lib.erl"},{line,249}]}]} > =INFO REPORT==== 20-Mar-2014::03:41:15 === > > Mirrored-queue (queue 'ha.MNT_Pages_grbhstmntp01' in vhost 'A105'): Slave > saw deaths of mirrors > > =INFO REPORT==== 20-Mar-2014::03:41:15 === > Mirrored-queue (queue 'ha.services_p17' in vhost 'A105'): Slave > saw deaths of mirrors > > =INFO REPORT==== 20-Mar-2014::03:41:15 === > Mirrored-queue (queue 'ha.services_p06' in vhost 'A105'): Slave > saw deaths of mirrors > > =INFO REPORT==== 20-Mar-2014::03:41:15 === > Mirrored-queue (queue 'ha.services_p06' in vhost 'A105'): Promoting slave > to master > > =INFO REPORT==== 20-Mar-2014::03:41:16 === > Mirrored-queue (queue 'ha.ElasticSearchConsumer' in vhost 'A105.001'): > Slave saw deaths of mirrors > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mcintoshj at gmail.com Wed Apr 2 05:52:20 2014 From: mcintoshj at gmail.com (McIntosh Jason) Date: Tue, 1 Apr 2014 23:52:20 -0500 Subject: [rabbitmq-discuss] Patterns with RabbitMQ In-Reply-To: <903570fa-62d2-42c5-b491-068d60696c87@googlegroups.com> References: <903570fa-62d2-42c5-b491-068d60696c87@googlegroups.com> Message-ID: Amol: I'd highly suggest reading the basic tutorials. They're very good: https://www.rabbitmq.com/getstarted.html If I understand what you're asking queues won't work the way you've described. Really though multi homed distributed data center design is moderately to very complex. So there's no easy "this is how you do it" as it depends on goals and topology. There isn't a simple answer other than "the possibilities are endless" Jason Sent from my iPhone > On Apr 1, 2014, at 11:40 PM, Amol Kedar wrote: > > We are trying to implement message queueing for our multi-server and multi datacenter application (distributed ) > I have a general question > is it a good idea to have 1 big queue created with different routing keys based on where the server is location and can the consumers in those data centers intelligently only pick the messages from the queue that belong to them? > > if its confusing i can be bit more descriptive... > > _______________________________________________ > rabbitmq-discuss mailing list > rabbitmq-discuss at lists.rabbitmq.com > https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss From towen27 at gmail.com Tue Apr 1 01:21:08 2014 From: towen27 at gmail.com (towen) Date: Mon, 31 Mar 2014 17:21:08 -0700 (PDT) Subject: [rabbitmq-discuss] Encryption between RabbitMQ cluster nodes Message-ID: <37ad40ba-8654-42cb-a627-4aee542afbbc@googlegroups.com> I was wondering if it was possible to encrypt the messages between rabbitmq cluster nodes. From what I understand, by default they listen to 4369, but communicate via clear text. I was wondering if it was possible to encrypt this via SSL, TLS, etc. Thanks! -------------- next part -------------- An HTML attachment was scrubbed... URL: From virus.vinay at gmail.com Tue Apr 1 13:31:00 2014 From: virus.vinay at gmail.com (Vinay Nayak) Date: Tue, 1 Apr 2014 05:31:00 -0700 (PDT) Subject: [rabbitmq-discuss] RabbitMQ .Net Client connecting without a passphrase Message-ID: Hi, We have managed to get an authenticate-authorise with server-client certificates set-up on our RabbitMQ server. However the fact that in the client code we pass a certificate with a key (i.e. p12 file) and a passphrase to connect to the server makes us a little uncomfortable. The fact that we are passing a Passphrase implies that RabbitMQ uses the passphrase to decrypt the p12 file, retrieve the key, use the key to get the CA details from the certificate and then check if the CA is trusted or not; instead of RabbitMQ contacting the CA server to verify the certificate presented by the client. The above can be absolute bollocks, if it is can someone please explain what goes under the hood. Also is there any way to connect from the client using certificates without giving out too much sensitive information. Thanks in advance!! ~Vinay -------------- next part -------------- An HTML attachment was scrubbed... URL: From matthias at rabbitmq.com Wed Apr 2 09:05:06 2014 From: matthias at rabbitmq.com (Matthias Radestock) Date: Wed, 02 Apr 2014 09:05:06 +0100 Subject: [rabbitmq-discuss] RabbitMQ .Net Client connecting without a passphrase In-Reply-To: References: Message-ID: <533BC4B2.1020004@rabbitmq.com> On 01/04/14 13:31, Vinay Nayak wrote: > We have managed to get an authenticate-authorise with server-client > certificates set-up on our RabbitMQ server. > However the fact that in the client code we pass a certificate with a > key (i.e. p12 file) and a passphrase to connect to the server makes us a > little uncomfortable. > > The fact that we are passing a Passphrase implies that RabbitMQ uses the > passphrase to decrypt the p12 file, retrieve the key, use the key to get > the CA details from the certificate and then check if the CA is trusted > or not; instead of RabbitMQ contacting the CA server to verify the > certificate presented by the client. > The above can be absolute bollocks, if it is can someone please explain > what goes under the hood. I am afraid the above is indeed absolute bollocks :) The client needs the key for its own certificate, just like the server needs the key for its own certificate. That's how PKI works - each party needs to know their own key (only). The key is not passed to the other party. Also, there is no such thing as a "CA server"; cert validation is a local operation. Regards, Matthias. From matthias at rabbitmq.com Wed Apr 2 09:17:48 2014 From: matthias at rabbitmq.com (Matthias Radestock) Date: Wed, 02 Apr 2014 09:17:48 +0100 Subject: [rabbitmq-discuss] Encryption between RabbitMQ cluster nodes In-Reply-To: <37ad40ba-8654-42cb-a627-4aee542afbbc@googlegroups.com> References: <37ad40ba-8654-42cb-a627-4aee542afbbc@googlegroups.com> Message-ID: <533BC7AC.40005@rabbitmq.com> On 01/04/14 01:21, towen wrote: > I was wondering if it was possible to encrypt the messages between > rabbitmq cluster nodes. From what I understand, by default they listen > to 4369, but communicate via clear text. I was wondering if it was > possible to encrypt this via SSL, TLS, etc. Thanks! http://rabbitmq.1065348.n5.nabble.com/RabbitMQ-SSL-and-Erlang-SSL-td34403.html From videlalvaro at gmail.com Wed Apr 2 10:10:57 2014 From: videlalvaro at gmail.com (Alvaro Videla) Date: Wed, 2 Apr 2014 11:10:57 +0200 Subject: [rabbitmq-discuss] RabbitMQ SSL and Erlang SSL In-Reply-To: <1396293255637-34447.post@n5.nabble.com> References: <1396050220968-34403.post@n5.nabble.com> <1396293255637-34447.post@n5.nabble.com> Message-ID: Hi, For embedded Erlang we've only done it for Mac users. I know it's not the best situation, but this file could point you on how to get an embedded Erlang for your OS distribution: http://hg.rabbitmq.com/rabbitmq-server/file/0e251dd3817a/packaging/standalone/Makefile Regards, Alvaro On Mon, Mar 31, 2014 at 9:14 PM, ranjitiyer wrote: > Thanks. Would be good if RabbitMQ could address these issues (in addition to > embedded Erlang). That would hugely simplify the kind of effort I am going > through for including RabbitMQ in a COTS product. > > Ranjit > > > > -- > View this message in context: http://rabbitmq.1065348.n5.nabble.com/RabbitMQ-SSL-and-Erlang-SSL-tp34403p34447.html > Sent from the RabbitMQ mailing list archive at Nabble.com. > _______________________________________________ > rabbitmq-discuss mailing list > rabbitmq-discuss at lists.rabbitmq.com > https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss From matthias at rabbitmq.com Wed Apr 2 10:12:45 2014 From: matthias at rabbitmq.com (Matthias Radestock) Date: Wed, 02 Apr 2014 10:12:45 +0100 Subject: [rabbitmq-discuss] Service stops automatically, when web management console is activated In-Reply-To: <00f801cf4906$dd87ce10$98976a30$@dataverse.gr> References: <00c401cf48fc$a33212c0$e9963840$@dataverse.gr> <00cf01cf48ff$a59c2d90$f0d488b0$@dataverse.gr> <00f801cf4906$dd87ce10$98976a30$@dataverse.gr> Message-ID: <533BD48D.3020308@rabbitmq.com> On 26/03/14 15:20, Kostas wrote: > I've put the provided example config file (rabbitmq.config.example) in > %APPDATA%\RabbitMQ\rabbitmq.config. > > I've modified this part: > > {rabbitmq_management, > > [{listener, [{port, 777}]}] > > }, > > I?ve re-installed the service. > > But now, when I start the service it crashes (Windows report an error) > and there are no log messages in the 2 log files > (%APPDATA%\RabbitMQ\log\) nor in Windows Event Viewer. > > Am I missing something? You may want to pick a port >= 1024. Matthias From bison_art at o2.pl Wed Apr 2 11:02:18 2014 From: bison_art at o2.pl (rabbit15) Date: Wed, 2 Apr 2014 03:02:18 -0700 (PDT) Subject: [rabbitmq-discuss] How join 2 queues in 1 ? Message-ID: <1396432938285-34485.post@n5.nabble.com> Hi I want to join (concat) 2 queues in 1 queue. My scenario is : I have added messages in two separate queues, after some time I want to make concatenation of my messages from second queue on the end of my first queue. Best Regrads, Artur -- View this message in context: http://rabbitmq.1065348.n5.nabble.com/How-join-2-queues-in-1-tp34485.html Sent from the RabbitMQ mailing list archive at Nabble.com. From videlalvaro at gmail.com Wed Apr 2 11:07:26 2014 From: videlalvaro at gmail.com (Alvaro Videla) Date: Wed, 2 Apr 2014 12:07:26 +0200 Subject: [rabbitmq-discuss] How join 2 queues in 1 ? In-Reply-To: <1396432938285-34485.post@n5.nabble.com> References: <1396432938285-34485.post@n5.nabble.com> Message-ID: Hi, There's no automatic way of doing this*. You can either: - consume messages from one queue and republish on the second one, or - setup a dead letter exchange with the second queue bound to it, and then consume and reject the messages from the first queue so they get dead lettered: http://www.rabbitmq.com/dlx.html *There's also the Shovel plugin that could do this "automatically": http://www.rabbitmq.com/shovel.html Regards, Alvaro On Wed, Apr 2, 2014 at 12:02 PM, rabbit15 wrote: > Hi I want to join (concat) 2 queues in 1 queue. > My scenario is : I have added messages in two separate queues, after some > time I want to make concatenation of my messages from second queue on the > end of my first queue. > Best Regrads, > Artur > > > > -- > View this message in context: http://rabbitmq.1065348.n5.nabble.com/How-join-2-queues-in-1-tp34485.html > Sent from the RabbitMQ mailing list archive at Nabble.com. > _______________________________________________ > rabbitmq-discuss mailing list > rabbitmq-discuss at lists.rabbitmq.com > https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss From matthias at rabbitmq.com Wed Apr 2 11:17:02 2014 From: matthias at rabbitmq.com (Matthias Radestock) Date: Wed, 02 Apr 2014 11:17:02 +0100 Subject: [rabbitmq-discuss] How join 2 queues in 1 ? In-Reply-To: References: <1396432938285-34485.post@n5.nabble.com> Message-ID: <533BE39E.1050707@rabbitmq.com> On 02/04/14 11:07, Alvaro Videla wrote: > *There's also the Shovel plugin that could do this "automatically": > http://www.rabbitmq.com/shovel.html The upcoming RabbitMQ 3.3.0 release introduces dynamic shovels, including shovels that delete themselves after emptying the source queue. http://next.rabbitmq.com/shovel-dynamic.html Matthias. From bison_art at o2.pl Wed Apr 2 13:00:26 2014 From: bison_art at o2.pl (rabbit15) Date: Wed, 2 Apr 2014 05:00:26 -0700 (PDT) Subject: [rabbitmq-discuss] How join 2 queues in 1 ? In-Reply-To: <533BE39E.1050707@rabbitmq.com> References: <1396432938285-34485.post@n5.nabble.com> <533BE39E.1050707@rabbitmq.com> Message-ID: <1396440026890-34488.post@n5.nabble.com> Thanks a lot! Artur -- View this message in context: http://rabbitmq.1065348.n5.nabble.com/How-join-2-queues-in-1-tp34485p34488.html Sent from the RabbitMQ mailing list archive at Nabble.com. From steve at rabbitmq.com Wed Apr 2 14:34:55 2014 From: steve at rabbitmq.com (Steve Powell) Date: Wed, 2 Apr 2014 14:34:55 +0100 Subject: [rabbitmq-discuss] How join 2 queues in 1 ? In-Reply-To: <533BE39E.1050707@rabbitmq.com> References: <1396432938285-34485.post@n5.nabble.com> <533BE39E.1050707@rabbitmq.com> Message-ID: <34F98963-FFF5-42C0-98FF-166509D9721E@rabbitmq.com> Hmm; that?s interesting. I might be able to get JMS queue selectors working with a feature like this. Steve Powell [Cell: +44-7815-838-558] Good design [?] is as little design as possible. Dieter Rams On 2 Apr 2014, at 11:17, Matthias Radestock wrote: > On 02/04/14 11:07, Alvaro Videla wrote: >> *There's also the Shovel plugin that could do this "automatically": >> http://www.rabbitmq.com/shovel.html > > The upcoming RabbitMQ 3.3.0 release introduces dynamic shovels, including shovels that delete themselves after emptying the source queue. > > http://next.rabbitmq.com/shovel-dynamic.html > > Matthias. > _______________________________________________ > rabbitmq-discuss mailing list > rabbitmq-discuss at lists.rabbitmq.com > https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: From simon at rabbitmq.com Wed Apr 2 15:45:18 2014 From: simon at rabbitmq.com (Simon MacMullen) Date: Wed, 02 Apr 2014 15:45:18 +0100 Subject: [rabbitmq-discuss] RabbitMQ 3.3.0 released Message-ID: <533C227E.1090003@rabbitmq.com> The RabbitMQ team is pleased to announce the release of RabbitMQ 3.3.0. This release improves performance in a variety of conditions, adds monitoring information to identify performance bottlenecks, adds dynamically manageable shovels, and allows Java-based clients to reconnect automatically after network failure. This release also corrects a number of defects in the broker and plugins, as well as introducing a host of smaller features. See the release notes at: http://www.rabbitmq.com/release-notes/README-3.3.0.txt for more information. The new release can be downloaded from: http://www.rabbitmq.com/download.html As always, we welcome any questions, bug reports, and other feedback on this release, as well as general suggestions for features and enhancements in future releases. Mail us via the RabbitMQ discussion list. Regards, The RabbitMQ Team (http://www.rabbitmq.com) From simon at rabbitmq.com Wed Apr 2 15:48:51 2014 From: simon at rabbitmq.com (Simon MacMullen) Date: Wed, 02 Apr 2014 15:48:51 +0100 Subject: [rabbitmq-discuss] RabbitMQ 3.3.0: "guest" user changes Message-ID: <533C2353.5040804@rabbitmq.com> This release changes the behaviour of the default "guest" user. Previously the guest user could be accessed from any network location. In RabbitMQ 3.3.0 and later, the guest user can only be accessed via localhost by default. Any other users you create will not (by default) be restricted in this way. See http://www.rabbitmq.com/access-control.html for more information. Cheers, Simon -- Simon MacMullen RabbitMQ, Pivotal From Petr.Gotthard at Honeywell.com Wed Apr 2 17:11:10 2014 From: Petr.Gotthard at Honeywell.com (Gotthard, Petr) Date: Wed, 2 Apr 2014 16:11:10 +0000 Subject: [rabbitmq-discuss] badmatch when unbinding queue in web management In-Reply-To: <53394BC6.7060909@rabbitmq.com> References: <53354DFD.9040603@rabbitmq.com> <53394BC6.7060909@rabbitmq.com> Message-ID: Hi Matthias, It seems that the 3.3.0 does fix this issue for me. Thanks, Petr -----Original Message----- From: Matthias Radestock [mailto:matthias at rabbitmq.com] Sent: 31. b?ezna 2014 13:05 To: Discussions about RabbitMQ Cc: Gotthard, Petr Subject: Re: [rabbitmq-discuss] badmatch when unbinding queue in web management Petr, On 28/03/14 11:36, Gotthard, Petr wrote: > I'm not sure this is the case; we don't have multiple identical > bindings with different arguments. > > > In the meantime we discovered the scenario that causes the web > unbinds to fail: -- client creates a binding with some parameters -- > clients unbinds with different parameters (which will fail) and > disconnects immediately We've had no joy reproducing this. If this is easily reproducible for you, would you mind grabbing a nightly build and repeating your experiment? Matthias. From Petr.Gotthard at Honeywell.com Wed Apr 2 17:14:33 2014 From: Petr.Gotthard at Honeywell.com (Gotthard, Petr) Date: Wed, 2 Apr 2014 16:14:33 +0000 Subject: [rabbitmq-discuss] Removal of the 'impersonator' tag from 3.3.0 Message-ID: Hello, May I ask why the 'impersonator' tag has been removed from 3.3.0? I thought it was a useful feature I will miss it. Regards, Petr -------------- next part -------------- An HTML attachment was scrubbed... URL: From videlalvaro at gmail.com Wed Apr 2 17:16:23 2014 From: videlalvaro at gmail.com (Alvaro Videla) Date: Wed, 2 Apr 2014 18:16:23 +0200 Subject: [rabbitmq-discuss] Issues with flow control In-Reply-To: References: <5322D5EE.8000300@rabbitmq.com> Message-ID: Hi Mat?as, Today we released a new version of RabbitMQ, perhaps bug 25868 is related to what you were getting. See here: http://www.rabbitmq.com/release-notes/README-3.3.0.txt Regards, Alvaro On Fri, Mar 14, 2014 at 4:01 PM, Mat?as Waisgold wrote: > Not permanently but for long terms > > > 2014-03-14 12:00 GMT-03:00 Michael Klishin : > >> >> On 14 Mar 2014, at 18:51, Mat?as Waisgold wrote: >> >> > rabbitmqctl eval 'rabbit_alarm:get_alarms().' >> > >> > [] >> > >> > ...done. >> > >> >> This means that there are no alarms across the cluster. Are your clients >> are permanently blocked >> or it just happens from time to time? >> >> MK >> >> Software Engineer, Pivotal/RabbitMQ >> >> >> _______________________________________________ >> rabbitmq-discuss mailing list >> rabbitmq-discuss at lists.rabbitmq.com >> https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss > > > > _______________________________________________ > rabbitmq-discuss mailing list > rabbitmq-discuss at lists.rabbitmq.com > https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss > From simon at rabbitmq.com Wed Apr 2 17:21:33 2014 From: simon at rabbitmq.com (Simon MacMullen) Date: Wed, 02 Apr 2014 17:21:33 +0100 Subject: [rabbitmq-discuss] Removal of the 'impersonator' tag from 3.3.0 In-Reply-To: References: Message-ID: <533C390D.2060907@rabbitmq.com> On 02/04/14 17:14, Gotthard, Petr wrote: > May I ask why the 'impersonator' tag has been removed from 3.3.0? I > thought it was a useful feature I will miss it. It was really an internal implementation detail of the federation plugin that "escaped". We removed it because we had no idea people would find it useful. Can I ask what you were doing with it? It would not be impossible to bring it back, but I am genuinely curious as to why anyone would need it. Cheers, Simon -- Simon MacMullen RabbitMQ, Pivotal From Petr.Gotthard at Honeywell.com Wed Apr 2 18:11:04 2014 From: Petr.Gotthard at Honeywell.com (Gotthard, Petr) Date: Wed, 2 Apr 2014 17:11:04 +0000 Subject: [rabbitmq-discuss] Removal of the 'impersonator' tag from 3.3.0 In-Reply-To: <533C390D.2060907@rabbitmq.com> References: <533C390D.2060907@rabbitmq.com> Message-ID: It would be great if you could bring it back. We need a two-way federation but have both federation clients in one broker only. Therefore we have a (custom) federation plug-in (running in the upstream broker) that re-publishes data to a downstream exchange. The downstream broker validates the User-IDs, but it doesn't know this was a federation, so to make this work we have either to: - modify the "user_id" in the upstream before re-publishing down-- but then we lose the information about the actual publisher and all upstream users will be represented by the broker username, or - assign the "impersonator" tag to the username used by upstream when connecting to downstream, so it can re-publish data down on behalf of others. In other words, the "escaped" implementation detail was used to implement user_id validation in upstream-controlled federations. Does it make sense? Petr -----Original Message----- From: Simon MacMullen [mailto:simon at rabbitmq.com] Sent: 2. dubna 2014 18:22 To: Discussions about RabbitMQ; Gotthard, Petr Subject: Re: [rabbitmq-discuss] Removal of the 'impersonator' tag from 3.3.0 On 02/04/14 17:14, Gotthard, Petr wrote: > May I ask why the 'impersonator' tag has been removed from 3.3.0? I > thought it was a useful feature I will miss it. It was really an internal implementation detail of the federation plugin that "escaped". We removed it because we had no idea people would find it useful. Can I ask what you were doing with it? It would not be impossible to bring it back, but I am genuinely curious as to why anyone would need it. Cheers, Simon -- Simon MacMullen RabbitMQ, Pivotal -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/x-pkcs7-signature Size: 5048 bytes Desc: not available URL: From michael.s.klishin at gmail.com Wed Apr 2 20:06:04 2014 From: michael.s.klishin at gmail.com (Michael Klishin) Date: Wed, 2 Apr 2014 23:06:04 +0400 Subject: [rabbitmq-discuss] ANN Langohr 2.8 is released Message-ID: Langohr [1] is a small, feature complete Clojure RabbitMQ client. Release notes: http://blog.clojurewerkz.org/blog/2014/04/02/langohr-2-dot-8-1-is-released/ 1. http://clojurerabbitmq.info -- MK http://github.com/michaelklishin http://twitter.com/michaelklishin -------------- next part -------------- An HTML attachment was scrubbed... URL: From mwaisgold at gmail.com Wed Apr 2 20:13:32 2014 From: mwaisgold at gmail.com (=?ISO-8859-1?Q?Mat=EDas_Waisgold?=) Date: Wed, 2 Apr 2014 16:13:32 -0300 Subject: [rabbitmq-discuss] Issues with flow control In-Reply-To: References: <5322D5EE.8000300@rabbitmq.com> Message-ID: Cool thank's. I'll try it. 2014-04-02 13:16 GMT-03:00 Alvaro Videla : > Hi Mat?as, > > Today we released a new version of RabbitMQ, perhaps bug 25868 is > related to what you were getting. See here: > http://www.rabbitmq.com/release-notes/README-3.3.0.txt > > Regards, > > Alvaro > > On Fri, Mar 14, 2014 at 4:01 PM, Mat?as Waisgold > wrote: > > Not permanently but for long terms > > > > > > 2014-03-14 12:00 GMT-03:00 Michael Klishin : > > > >> > >> On 14 Mar 2014, at 18:51, Mat?as Waisgold wrote: > >> > >> > rabbitmqctl eval 'rabbit_alarm:get_alarms().' > >> > > >> > [] > >> > > >> > ...done. > >> > > >> > >> This means that there are no alarms across the cluster. Are your clients > >> are permanently blocked > >> or it just happens from time to time? > >> > >> MK > >> > >> Software Engineer, Pivotal/RabbitMQ > >> > >> > >> _______________________________________________ > >> rabbitmq-discuss mailing list > >> rabbitmq-discuss at lists.rabbitmq.com > >> https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss > > > > > > > > _______________________________________________ > > rabbitmq-discuss mailing list > > rabbitmq-discuss at lists.rabbitmq.com > > https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss > > > _______________________________________________ > rabbitmq-discuss mailing list > rabbitmq-discuss at lists.rabbitmq.com > https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss > -------------- next part -------------- An HTML attachment was scrubbed... URL: From kopias.peter at gmail.com Wed Apr 2 20:19:48 2014 From: kopias.peter at gmail.com (Peter Kopias) Date: Wed, 2 Apr 2014 21:19:48 +0200 Subject: [rabbitmq-discuss] Upgrade fail Message-ID: Hi. I've tried to upgrade my cluster to 3.3.0. Disclosure: we had seen some strange artifacts of (maybe mnesia) failures, as there were a few queues that showed up in the lists, but I could never remove them or even query their properties. I hoped that the upgrade would solve this. Stop node2, node3, node1. (as node1 should run be the last and first) node1: apt-get upgrade node1: service rabbitmq-server start /var/log/rabbitmq/startup_log RabbitMQ 3.3.0. Copyright (C) 2007-2013 GoPivotal, Inc. ## ## Licensed under the MPL. See http://www.rabbitmq.com/ ## ## ########## Logs: /var/log/rabbitmq/rabbit at node1.log ###### ## /var/log/rabbitmq/rabbit at node1-sasl.log ########## Starting broker...dets: file "/var/lib/rabbitmq/mnesia/rabbit at node1/schema.DAT" not properly closed, repairing ... BOOT FAILED =========== Timeout contacting cluster nodes: [rabbit at node3,rabbit at node2]. DIAGNOSTICS =========== attempted to contact: [rabbit at node3,rabbit at node2] rabbit at node3: * found rabbit (port 25672) * TCP connection succeeded * suggestion: hostname mismatch? * suggestion: is the cookie set correctly? rabbit at node2: * found rabbit (port 25672) * TCP connection succeeded * suggestion: hostname mismatch? * suggestion: is the cookie set correctly? current node details: - node name: rabbit at node1 - home dir: /var/lib/rabbitmq - cookie hash: XSquoZCiCPIfZlTilwy1Dg== BOOT FAILED =========== Error description: {could_not_start,rabbit, {bad_return, {{rabbit,start,[normal,[]]}, {'EXIT', {rabbit,failure_during_boot, {error, {timeout_waiting_for_tables, [rabbit_listener,rabbit_semi_durable_route, rabbit_route,rabbit_reverse_route, rabbit_topic_trie_node,rabbit_topic_trie_edge, rabbit_topic_trie_binding,rabbit_exchange, rabbit_exchange_serial,rabbit_queue,gm_group, mirrored_sup_childspec]}}}}}}} Log files (may contain more information): /var/log/rabbitmq/rabbit at node1.log /var/log/rabbitmq/rabbit at node1-sasl.log {"init terminating in do_boot",{rabbit,failure_during_boot,{could_not_start,rabbit,{bad_return,{{rabbit,start,[normal,[]]},{'EXIT',{rabbit,failure_during_boot,{error,{timeout_waiting_for_tables,[ra bbit_listener,rabbit_semi_durable_route,rabbit_route,rabbit_reverse_route,rabbit_topic_trie_node,rabbit_topic_trie_edge,rabbit_topic_trie_binding,rabbit_exchange,rabbit_exchange_serial,rabbit_queue ,gm_group,mirrored_sup_childspec]}}}}}}}}} WHY is it waiting for node2, node3, if node1 was the last to stop, and it should come online in itself without them? The cookie is the same, the hostnames are correct. Although: /var/log/rabbitmq/startup_err Crash dump was written to: erl_crash.dump init terminating in do_boot () /var/log/rabbitmq/rabbit at node1-sasl.log =CRASH REPORT==== 2-Apr-2014::19:51:57 === crasher: initial call: application_master:init/4 pid: <0.118.0> registered_name: [] exception exit: {bad_return, {{rabbit,start,[normal,[]]}, {'EXIT', {rabbit,failure_during_boot, {error, {timeout_waiting_for_tables, [rabbit_listener,rabbit_semi_durable_route, rabbit_route,rabbit_reverse_route, rabbit_topic_trie_node,rabbit_topic_trie_edge, rabbit_topic_trie_binding,rabbit_exchange, rabbit_exchange_serial,rabbit_queue,gm_group, mirrored_sup_childspec]}}}}}} in function application_master:init/4 (application_master.erl, line 133) ancestors: [<0.117.0>] messages: [{'EXIT',<0.119.0>,normal}] links: [<0.117.0>,<0.7.0>] dictionary: [] trap_exit: true status: running heap_size: 4185 stack_size: 27 reductions: 322 neighbours: rabbitmq.config: [{rabbit, [{vm_memory_high_watermark, 0.45},{vm_memory_high_watermark, 0.5},{disk_free_limit, 10000000000},{hipe_compile,false}]}]. erl_crash.dump =erl_crash_dump:0.2 Wed Apr 2 19:51:58 2014 Slogan: init terminating in do_boot () System version: Erlang R16B01 (erts-5.10.2) [source] [64-bit] [smp:4:4] [async-threads:30] [hipe] [kernel-poll:true] Compiled: Wed Jul 10 10:49:45 2013 Taints: Atoms: 22590 =memory total: 41536280 processes: 14801392 processes_used: 14801320 system: 26734888 atom: 703377 atom_used: 673775 binary: 1330608 code: 19294518 ets: 859648 =hash_table:atom_tab size: 19289 used: 13349 objs: 22590 depth: 8 =index_table:atom_tab size: 23552 limit: 1048576 entries: 22590 =hash_table:module_code size: 397 used: 288 objs: 539 depth: 5 =index_table:module_code size: 1024 limit: 65536 entries: 539 =hash_table:export_list size: 6421 used: 4793 objs: 8988 depth: 6 =index_table:export_list size: 9216 limit: 524288 entries: 8988 =hash_table:export_list size: 6421 used: 4788 objs: 8956 depth: 6 =hash_table:process_reg size: 47 used: 24 objs: 31 depth: 2 =hash_table:fun_table size: 3203 used: 2189 objs: 4009 depth: 6 =hash_table:node_table size: 5 used: 3 objs: 4 depth: 2 =hash_table:dist_table size: 5 used: 3 objs: 4 depth: 2 =allocated_areas sys_misc: 126685 static: 524928 atom_space: 360536 330934 atom_table: 342841 module_table: 187920 export_table: 375636 export_list: 1581888 register_table: 468 fun_table: 25714 module_refs: 25824 loaded_code: 17097536 dist_table: 99875 node_table: 323 bits_bufs_size: 0 bif_timer: 80200 link_lh: 208 process_table: 12582912 port_table: 786432 ets_misc: 49192 =allocator:sys_alloc option e: true option m: libc option tt: 131072 option tp: 0 =allocator:temp_alloc[0] versions: 2.1 3.0 option e: true option t: option acul: %d option ramv: 5 option sbct: 5865745 option asbcst: 524288 option rsbcst: 4145152 option rsbcmt: 90 option rmbcmt: 80 option mmbcs: 100 option mmsbc: 131072 option mmmbc: 256 option lmbcs: 18446744073709551615 option smbcs: 10485760 option mbcgs: 1048576 option mbsd: 3 option as: gf mbcs blocks: 0 4 4 mbcs blocks size: 0 9848 9848 mbcs carriers: 1 1 1 mbcs mseg carriers: 1 mbcs sys_alloc carriers: 0 mbcs carriers size: 1048576 1048576 1048576 mbcs mseg carriers size: 1048576 mbcs sys_alloc carriers size: 0 sbcs blocks: 0 0 0 sbcs blocks size: 0 0 0 sbcs carriers: 0 0 0 sbcs mseg carriers: 0 sbcs sys_alloc carriers: 0 sbcs carriers size: 0 0 0 sbcs mseg carriers size: 0 sbcs sys_alloc carriers size: 0 temp_alloc calls: 8506 temp_free calls: 8506 temp_realloc calls: 1 mseg_alloc calls: 1 mseg_dealloc calls: 0 mseg_realloc calls: 0 sys_alloc calls: 0 sys_free calls: 0 sys_realloc calls: 0 =allocator:temp_alloc[1] versions: 0.9 3.0 option e: true option t: option acul: %d option ramv: 5 option sbct: 5865745 option asbcst: 524288 option rsbcst: 4145152 option rsbcmt: 90 option rmbcmt: 80 option mmbcs: 100 option mmsbc: 131072 option mmmbc: 256 ...... .... =allocator:alloc_util option mmc: 1024 option ycs: 1048576 =allocator:instr option m: false option s: false option t: false =proc:<0.0.0> State: Running Name: init Spawned as: otp_ring0:start/2 Spawned by: [] Started: Wed Apr 2 19:51:25 2014 Message queue length: 0 Number of heap fragments: 0 Heap fragment data: 0 Link list: [<0.3.0>, <0.7.0>, <0.6.0>] Reductions: 57344 Stack+heap: 6772 OldHeap: 6772 Heap unused: 3114 OldHeap unused: 6325 Memory: 109264 Program counter: 0x00007f94178c6f80 (init:boot_loop/2 + 64) CP: 0x0000000000000000 (invalid) =proc:<0.3.0> State: Waiting Name: erl_prim_loader Spawned as: erlang:apply/2 Spawned by: <0.2.0> Started: Wed Apr 2 19:51:25 2014 Message queue length: 0 Number of heap fragments: 0 Heap fragment data: 0 Link list: [<0.0.0>, #Port<0.0>] Reductions: 3093307 Stack+heap: 1598 OldHeap: 4185 Heap unused: 945 OldHeap unused: 2668 Memory: 47136 Program counter: 0x00007f94179369b8 (erl_prim_loader:loop/3 + 176) CP: 0x0000000000000000 (invalid) arity = 0 =proc:<0.6.0> State: Waiting Name: error_logger Spawned as: proc_lib:init_p/5 Spawned by: <0.2.0> Started: Wed Apr 2 19:51:25 2014 Message queue length: 0 Number of heap fragments: 0 Heap fragment data: 0 Link list: [<0.30.0>, <0.0.0>, {from,<0.46.0>,#Ref<0.0.0.64>}, {from,<0.47.0>,#Ref<0.0.0.69>}] Reductions: 38274 Stack+heap: 4185 OldHeap: 6772 Heap unused: 2304 OldHeap unused: 5548 Memory: 88776 Program counter: 0x00007f9417962c50 (gen_event:fetch_msg/5 + 80) CP: 0x0000000000000000 (invalid) arity = 0 =proc:<0.7.0> State: Waiting Name: application_controller Spawned as: erlang:apply/2 Spawned by: <0.2.0> Started: Wed Apr 2 19:51:25 2014 Message queue length: 0 Number of heap fragments: 0 Heap fragment data: 0 Link list: [<0.0.0>, <0.35.0>, <0.9.0>] Reductions: 208958 Stack+heap: 6772 OldHeap: 28690 Heap unused: 3968 OldHeap unused: 19495 Memory: 284712 Program counter: 0x00007f941638a1c8 (gen_server:loop/6 + 264) CP: 0x0000000000000000 (invalid) arity = 0 =proc:<0.9.0> State: Waiting Spawned as: proc_lib:init_p/5 Spawned by: <0.8.0> Started: Wed Apr 2 19:51:25 2014 Message queue length: 0 Number of heap fragments: 0 Heap fragment data: 0 Link list: [<0.10.0>, <0.7.0>] Reductions: 44 Stack+heap: 376 OldHeap: 376 Heap unused: 335 OldHeap unused: 171 Memory: 6992 Program counter: 0x00007f941639b430 (application_master:main_loop/2 + 64) CP: 0x0000000000000000 (invalid) arity = 0 =proc:<0.10.0> State: Waiting Spawned as: application_master:start_it/4 Spawned by: <0.9.0> Started: Wed Apr 2 19:51:25 2014 Message queue length: 0 Number of heap fragments: 0 Heap fragment data: 0 Link list: [<0.11.0>, <0.9.0>] Reductions: 69 Stack+heap: 233 OldHeap: 0 Heap unused: 86 OldHeap unused: 0 Memory: 2736 Program counter: 0x00007f941639d720 (application_master:loop_it/4 + 72) CP: 0x0000000000000000 (invalid) arity = 0 =proc:<0.11.0> State: Waiting Name: kernel_sup Spawned as: proc_lib:init_p/5 Spawned by: <0.10.0> Started: Wed Apr 2 19:51:25 2014 Message queue length: 0 Number of heap fragments: 0 Heap fragment data: 0 Link list: [<0.10.0>, <0.13.0>, <0.12.0>, <0.18.0>, <0.25.0>, <0.24.0>, <0.16.0>, <0.27.0>, <0.32.0>, <0.31.0>, <0.29.0>, <0.26.0>] Reductions: 52300 Stack+heap: 6772 OldHeap: 28690 Heap unused: 3529 OldHeap unused: 24822 Memory: 285072 Program counter: 0x00007f941638a1c8 (gen_server:loop/6 + 264) CP: 0x0000000000000000 (invalid) arity = 0 =proc:<0.12.0> State: Waiting Name: rex Spawned as: proc_lib:init_p/5 Spawned by: <0.11.0> Started: Wed Apr 2 19:51:25 2014 Message queue length: 0 Number of heap fragments: 0 Heap fragment data: 0 Link list: [<0.11.0>, {from,<22585.2.0>,#Ref<22585.0.0.50>}, {to,<0.155.0>,#Ref<0.0.0.767>}] Reductions: 858 Stack+heap: 2586 OldHeap: 376 Heap unused: 1188 OldHeap unused: 350 Memory: 24824 Program counter: 0x00007f941638a1c8 (gen_server:loop/6 + 264) CP: 0x0000000000000000 (invalid) arity = 0 =proc:<0.13.0> State: Waiting Name: global_name_server Spawned as: proc_lib:init_p/5 Spawned by: <0.11.0> Started: Wed Apr 2 19:51:25 2014 Message queue length: 0 Number of heap fragments: 0 Heap fragment data: 0 Link list: [<0.11.0>, <0.15.0>, <0.14.0>] Reductions: 757 Stack+heap: 610 OldHeap: 376 Heap unused: 353 OldHeap unused: 280 Memory: 8904 Program counter: 0x00007f941638a1c8 (gen_server:loop/6 + 264) CP: 0x0000000000000000 (invalid) arity = 0 =proc:<0.14.0> State: Waiting Spawned as: erlang:apply/2 Spawned by: <0.13.0> Started: Wed Apr 2 19:51:25 2014 Message queue length: 0 Number of heap fragments: 0 Heap fragment data: 0 Link list: [<0.13.0>] Reductions: 1023 Stack+heap: 1598 OldHeap: 376 Heap unused: 1088 OldHeap unused: 308 I can't start node2 and node3, as they were stopped before node1, so they won't start till it gets up or the locks get removed. So currently I've got a three legged cluster down on its six knees. --- I'm trying to start node2 and node3 to have at least one running but they always fail, even if I copied back the previous disk state, and upgraded again... Not a single node running currently, looks like I have to: - reinstall each rabbitmq-server with clean /var/lib/rabbitmq dirs - rebuild cluster - rebuild the virtual hosts - recreate all users and permissions - all queues and exchanges and policies Thats not the way upgrade should happen. :( Peter -------------- next part -------------- An HTML attachment was scrubbed... URL: From bernie at step.polymtl.ca Wed Apr 2 21:02:19 2014 From: bernie at step.polymtl.ca (Bertrand Guay-Paquet) Date: Wed, 02 Apr 2014 16:02:19 -0400 Subject: [rabbitmq-discuss] How to shutdown cleanly a Java application using Consumers? Message-ID: <533C6CCB.6070505@step.polymtl.ca> Hello, I'm using the RabbitMQ Java client API and need some guidance on the proper application shutdown procedure. Here's how I start the application: 1-Create a Connection 2-Create different types of consumers, each with its own channel, and call channel.basicConsume("queue", false, consumer) 3-Let it all run This works great, but I can't figure out how to cleanly shutdown the application. If I simply close the Connection, each created channel immediately (or soon enough) becomes invalid and any Consumer currently doing some work fails when trying to ack their current message or perform any other action on the channel. I'd like to let the consumers finish whatever message they're processing and then close everything down. I guess I need to keep track of the created Consumers and somehow signal them to stop accepting new messages and after they're all done with their current job, close the connection? Is that possible or is there another way? I haven't found any management methods for the consumer classes to control or query their status. The information I found so far is related to manually created threads that poll the queues to process messages. In that case, it's really easy because I can just set a flag on each runnable to exit after processing their current message and join on all the threads before closing the underlying connection. So this leads me to ask, as a side note: are Consumers the way to go to use RabbitMQ in real-world scenarios or should I poll on the queues? It seems to me that Consumers would be the better choice (polling is bad), but if they're less powerful, perhaps they're not a silver bullet in my case. Thank you, Bertrand From mcintoshj at gmail.com Wed Apr 2 21:10:55 2014 From: mcintoshj at gmail.com (Jason McIntosh) Date: Wed, 2 Apr 2014 15:10:55 -0500 Subject: [rabbitmq-discuss] How to shutdown cleanly a Java application using Consumers? In-Reply-To: <533C6CCB.6070505@step.polymtl.ca> References: <533C6CCB.6070505@step.polymtl.ca> Message-ID: We use consumers here with the spring libraries. If you use them, there's a "Shutdown" command you can call as needed. We've had pretty good luck with it so far. The spring consumer stuff manages how many consumers are running, connection threading, etc. - and it has very good shutdown hooks. Jason On Wed, Apr 2, 2014 at 3:02 PM, Bertrand Guay-Paquet wrote: > Hello, > > I'm using the RabbitMQ Java client API and need some guidance on the > proper application shutdown procedure. > > Here's how I start the application: > 1-Create a Connection > 2-Create different types of consumers, each with its own channel, and call > channel.basicConsume("queue", false, consumer) > 3-Let it all run > > This works great, but I can't figure out how to cleanly shutdown the > application. If I simply close the Connection, each created channel > immediately (or soon enough) becomes invalid and any Consumer currently > doing some work fails when trying to ack their current message or perform > any other action on the channel. I'd like to let the consumers finish > whatever message they're processing and then close everything down. I guess > I need to keep track of the created Consumers and somehow signal them to > stop accepting new messages and after they're all done with their current > job, close the connection? Is that possible or is there another way? I > haven't found any management methods for the consumer classes to control or > query their status. > > The information I found so far is related to manually created threads that > poll the queues to process messages. In that case, it's really easy because > I can just set a flag on each runnable to exit after processing their > current message and join on all the threads before closing the underlying > connection. So this leads me to ask, as a side note: are Consumers the way > to go to use RabbitMQ in real-world scenarios or should I poll on the > queues? It seems to me that Consumers would be the better choice (polling > is bad), but if they're less powerful, perhaps they're not a silver bullet > in my case. > > Thank you, > Bertrand > _______________________________________________ > rabbitmq-discuss mailing list > rabbitmq-discuss at lists.rabbitmq.com > https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss > -- Jason McIntosh https://github.com/jasonmcintosh/ 573-424-7612 -------------- next part -------------- An HTML attachment was scrubbed... URL: From bernie at step.polymtl.ca Wed Apr 2 21:17:00 2014 From: bernie at step.polymtl.ca (Bertrand Guay-Paquet) Date: Wed, 02 Apr 2014 16:17:00 -0400 Subject: [rabbitmq-discuss] How to shutdown cleanly a Java application using Consumers? In-Reply-To: References: <533C6CCB.6070505@step.polymtl.ca> Message-ID: <533C703C.6030509@step.polymtl.ca> An HTML attachment was scrubbed... URL: From mcintoshj at gmail.com Wed Apr 2 21:35:09 2014 From: mcintoshj at gmail.com (Jason McIntosh) Date: Wed, 2 Apr 2014 15:35:09 -0500 Subject: [rabbitmq-discuss] How to shutdown cleanly a Java application using Consumers? In-Reply-To: <533C703C.6030509@step.polymtl.ca> References: <533C6CCB.6070505@step.polymtl.ca> <533C703C.6030509@step.polymtl.ca> Message-ID: Real quick sample code from the Spring AMQP site: ConnectionFactory cf = new CachingConnectionFactory(); // set up the queue, exchange, binding on the broker RabbitAdmin admin = new RabbitAdmin(cf); Queue queue = new Queue("myQueue"); admin.declareQueue(queue); TopicExchange exchange = new TopicExchange("myExchange"); admin.declareExchange(exchange); admin.declareBinding( BindingBuilder.bind(queue).to(exchange).with("foo.*")); // set up the listener and container SimpleMessageListenerContainer container = new SimpleMessageListenerContainer(cf); Object listener = new Object() { public void handleMessage(String foo) { System.out.println(foo); } }; MessageListenerAdapter adapter = new MessageListenerAdapter(listener); container.setMessageListener(adapter); container.setQueueNames("myQueue"); container.start(); // send something RabbitTemplate template = new RabbitTemplate(cf); template.convertAndSend("myExchange", "foo.bar", "Hello, world!"); Thread.sleep(1000); container.stop(); ON a war, you add shutdown hooks to the web.xml file. In a non-web environment, there are shutdown hooks through the Runtime: http://docs.oracle.com/javase/7/docs/api/java/lang/Runtime.html In the end, they all call container.shutdown: http://docs.spring.io/spring-amqp/docs/1.2.1.RELEASE/api/org/springframework/amqp/rabbit/listener/SimpleMessageListenerContainer.html Code for that I found here: http://grepcode.com/file/repo1.maven.org/maven2/org.springframework.amqp/spring-rabbit/1.0.0.RELEASE/org/springframework/amqp/rabbit/listener/SimpleMessageListenerContainer.java The spring libraries are fantastic for RabbitMQ work (though sometimes there are some headaches on "Which factory is this? Which MessageProperties do I use?" kinda stuff, but overall they're not bad and they're getting better. I tend to be very careful on how much abstraction I add to my systems (hence why Groovy drives me nuts), but overall spring is has been very clean implementation wise. Jason On Wed, Apr 2, 2014 at 3:17 PM, Bertrand Guay-Paquet wrote: > Hi Jason, > > Thanks for the hint, but I'm unfortunately not using Spring for this. Do > you know how Spring achieves this "shutdown" command? I don't know Spring > at all but I'll have a look at their doc. > > Regards, > Bertrand > > > On 02/04/2014 4:10 PM, Jason McIntosh wrote: > > We use consumers here with the spring libraries. If you use them, > there's a "Shutdown" command you can call as needed. We've had pretty good > luck with it so far. The spring consumer stuff manages how many consumers > are running, connection threading, etc. - and it has very good shutdown > hooks. > Jason > > > On Wed, Apr 2, 2014 at 3:02 PM, Bertrand Guay-Paquet < > bernie at step.polymtl.ca> wrote: > >> Hello, >> >> I'm using the RabbitMQ Java client API and need some guidance on the >> proper application shutdown procedure. >> >> Here's how I start the application: >> 1-Create a Connection >> 2-Create different types of consumers, each with its own channel, and >> call channel.basicConsume("queue", false, consumer) >> 3-Let it all run >> >> This works great, but I can't figure out how to cleanly shutdown the >> application. If I simply close the Connection, each created channel >> immediately (or soon enough) becomes invalid and any Consumer currently >> doing some work fails when trying to ack their current message or perform >> any other action on the channel. I'd like to let the consumers finish >> whatever message they're processing and then close everything down. I guess >> I need to keep track of the created Consumers and somehow signal them to >> stop accepting new messages and after they're all done with their current >> job, close the connection? Is that possible or is there another way? I >> haven't found any management methods for the consumer classes to control or >> query their status. >> >> The information I found so far is related to manually created threads >> that poll the queues to process messages. In that case, it's really easy >> because I can just set a flag on each runnable to exit after processing >> their current message and join on all the threads before closing the >> underlying connection. So this leads me to ask, as a side note: are >> Consumers the way to go to use RabbitMQ in real-world scenarios or should I >> poll on the queues? It seems to me that Consumers would be the better >> choice (polling is bad), but if they're less powerful, perhaps they're not >> a silver bullet in my case. >> >> Thank you, >> Bertrand >> _______________________________________________ >> rabbitmq-discuss mailing list >> rabbitmq-discuss at lists.rabbitmq.com >> https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss >> > > > > -- > Jason McIntosh > https://github.com/jasonmcintosh/ > 573-424-7612 > > > _______________________________________________ > rabbitmq-discuss mailing listrabbitmq-discuss at lists.rabbitmq.comhttps://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss > > > > _______________________________________________ > rabbitmq-discuss mailing list > rabbitmq-discuss at lists.rabbitmq.com > https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss > > -- Jason McIntosh https://github.com/jasonmcintosh/ 573-424-7612 -------------- next part -------------- An HTML attachment was scrubbed... URL: From ambatinr at hotmail.com Wed Apr 2 21:39:26 2014 From: ambatinr at hotmail.com (anate) Date: Wed, 2 Apr 2014 13:39:26 -0700 (PDT) Subject: [rabbitmq-discuss] SharedQueue closed error Message-ID: <1396471166545-34504.post@n5.nabble.com> I declared a queue with persistent message using PHP and trying to consume the message in C#. I received 'SharedQueue closed error' at this line. channel.BasicAck(ea.DeliveryTag, false); I am assuming the persistent message shouldn't delete from queue but after Dequeue() message getting deleted. Could you please let me know if there are any suggestions. PHP(Publish) ----- $channel->queue_declare('hello-queue', false, true, false, false); $message = "Hello World...."; try{ $msg = new AMQPMessage($message, array('delivery_mode' => 2) # make message persistent ); $channel->basic_publish($msg, '', 'hello-queue'); } catch (AMQPChannelException $exception) { echo "The channel is not open (publishing message on queue).\n"; } Consume(C#) --------------- using (IConnection connection = connectionFactory.CreateConnection()) { using (var channel = connection.CreateModel()) { channel.QueueDeclare("hello-queue", true, false, false, null); var consumer = new QueueingBasicConsumer(channel); channel.BasicConsume("hello-queue", true, consumer); while (true) { try { var ea = (BasicDeliverEventArgs)consumer.Queue.Dequeue(); InsertData(message); channel.BasicAck(ea.DeliveryTag, false); } catch (RabbitMQ.Client.Exceptions.OperationInterruptedException ex) { break; } } } } -- View this message in context: http://rabbitmq.1065348.n5.nabble.com/SharedQueue-closed-error-tp34504.html Sent from the RabbitMQ mailing list archive at Nabble.com. From ajkedar at gmail.com Wed Apr 2 21:43:53 2014 From: ajkedar at gmail.com (Amol Kedar) Date: Wed, 2 Apr 2014 13:43:53 -0700 (PDT) Subject: [rabbitmq-discuss] Consumer management In-Reply-To: References: Message-ID: <89f68acc-5224-4af7-b4df-bc18944f0c58@googlegroups.com> Hi Bill, I'd like to know how the supervisord has worked out for you or have you moved on to something else? we are trying to implement something on our own but saw this thread and might give it a try for spawning our consumers On Monday, January 31, 2011 10:01:13 AM UTC-5, Bill Moseley wrote: > > How do you manage your consumer/worker processes? Does anyone look at the > queue lengths and automatically adjust the number of consumers? > > Is anyone using http://supervisord.org/ for managing consumer processes? > > > > I have a pool of servers to run a small collection of consumers that do > different tasks. As load dictates (by looking at messages waiting in > queues) I'd like to be able to dynamically re-balance the mixture of > consumers. For example, if a lot of requests come in for report generation > and I have a lot of idle consumers that do image manipulation I'd like to > reduce the number of image processing consumers and add more report > generation consumers. > > (Many of my consumers are pretty simple in that they just run some command > line tool.) > > What's your approach? > > > And do you tend to write consumers that are small stand-alone programs > (something easy for Supervisor to manage) or write larger (perhaps more > complex) stand-alone apps that listen to a queue and fork (or threads) > workers do handle incoming messages? > > > > -- > Bill Moseley > mos... at hank.org > -------------- next part -------------- An HTML attachment was scrubbed... URL: From videlalvaro at gmail.com Wed Apr 2 21:45:19 2014 From: videlalvaro at gmail.com (Alvaro Videla) Date: Wed, 2 Apr 2014 22:45:19 +0200 Subject: [rabbitmq-discuss] SharedQueue closed error In-Reply-To: <1396471166545-34504.post@n5.nabble.com> References: <1396471166545-34504.post@n5.nabble.com> Message-ID: If you ack the message on the consumer side, then the message will be removed from the queue. On Wed, Apr 2, 2014 at 10:39 PM, anate wrote: > I declared a queue with persistent message using PHP and trying to consume > the message in C#. I received 'SharedQueue closed error' at this line. > channel.BasicAck(ea.DeliveryTag, false); > I am assuming the persistent message shouldn't delete from queue but after > Dequeue() message getting deleted. Could you please let me know if there are > any suggestions. > > PHP(Publish) > ----- > $channel->queue_declare('hello-queue', false, true, false, false); > > $message = "Hello World...."; > > try{ > > $msg = new AMQPMessage($message, > array('delivery_mode' => 2) # make message > persistent > ); > $channel->basic_publish($msg, '', 'hello-queue'); > } > catch (AMQPChannelException $exception) > { > echo "The channel is not open (publishing message on queue).\n"; > } > > Consume(C#) > --------------- > > using (IConnection connection = > connectionFactory.CreateConnection()) > { > using (var channel = connection.CreateModel()) > { > channel.QueueDeclare("hello-queue", true, false, > false, null); > > var consumer = new QueueingBasicConsumer(channel); > channel.BasicConsume("hello-queue", true, consumer); > > while (true) > { > try > { > var ea = > (BasicDeliverEventArgs)consumer.Queue.Dequeue(); > > InsertData(message); > > channel.BasicAck(ea.DeliveryTag, false); > } > catch > (RabbitMQ.Client.Exceptions.OperationInterruptedException ex) > { > break; > } > } > } > } > > > > -- > View this message in context: http://rabbitmq.1065348.n5.nabble.com/SharedQueue-closed-error-tp34504.html > Sent from the RabbitMQ mailing list archive at Nabble.com. > _______________________________________________ > rabbitmq-discuss mailing list > rabbitmq-discuss at lists.rabbitmq.com > https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss From ambatinr at hotmail.com Wed Apr 2 21:50:53 2014 From: ambatinr at hotmail.com (anate) Date: Wed, 2 Apr 2014 13:50:53 -0700 (PDT) Subject: [rabbitmq-discuss] SharedQueue closed error In-Reply-To: References: <1396471166545-34504.post@n5.nabble.com> Message-ID: <1396471853608-34507.post@n5.nabble.com> Thank you. I am not sure why am I getting SharedQueue closed error message. Could you please let me know? -- View this message in context: http://rabbitmq.1065348.n5.nabble.com/SharedQueue-closed-error-tp34504p34507.html Sent from the RabbitMQ mailing list archive at Nabble.com. From grussell at gopivotal.com Wed Apr 2 21:50:56 2014 From: grussell at gopivotal.com (Gary Russell) Date: Wed, 2 Apr 2014 16:50:56 -0400 Subject: [rabbitmq-discuss] How to shutdown cleanly a Java application using Consumers? In-Reply-To: References: <533C6CCB.6070505@step.polymtl.ca> <533C703C.6030509@step.polymtl.ca> Message-ID: Internally, in essence, the listener container tells each consumer to issue a basicCancel for its consumer tag(s) (one per queue) and waits for all the cancelOKs and close its channel. When all the consumer instances have stopped, the container stops. The architecture is such that we allow any pre-fetched messages that have already been passed to the container to process before the cancel is issued, while preventing any new deliveries. Hence, any prefetched messages in the container complete processing, while those still in the rabbit client, and not yet handed over to the container will be un-acked and redelivered later. On Wed, Apr 2, 2014 at 4:35 PM, Jason McIntosh wrote: > Real quick sample code from the Spring AMQP site: > > > ConnectionFactory cf = new CachingConnectionFactory(); > > // set up the queue, exchange, binding on the broker > RabbitAdmin admin = new RabbitAdmin(cf); > Queue queue = new Queue("myQueue"); > admin.declareQueue(queue); > TopicExchange exchange = new TopicExchange("myExchange"); > admin.declareExchange(exchange); > admin.declareBinding( > BindingBuilder.bind(queue).to(exchange).with("foo.*")); > > // set up the listener and container > SimpleMessageListenerContainer container = > new SimpleMessageListenerContainer(cf); > Object listener = new Object() { > public void handleMessage(String foo) { > System.out.println(foo); > } > }; > MessageListenerAdapter adapter = new MessageListenerAdapter(listener); > container.setMessageListener(adapter); > container.setQueueNames("myQueue"); > container.start(); > > // send something > RabbitTemplate template = new RabbitTemplate(cf); > template.convertAndSend("myExchange", "foo.bar", "Hello, world!"); > Thread.sleep(1000); > container.stop(); > > > ON a war, you add shutdown hooks to the web.xml file. In a non-web environment, there are shutdown hooks through the Runtime: > http://docs.oracle.com/javase/7/docs/api/java/lang/Runtime.html > > In the end, they all call container.shutdown: > http://docs.spring.io/spring-amqp/docs/1.2.1.RELEASE/api/org/springframework/amqp/rabbit/listener/SimpleMessageListenerContainer.html > > Code for that I found here: > http://grepcode.com/file/repo1.maven.org/maven2/org.springframework.amqp/spring-rabbit/1.0.0.RELEASE/org/springframework/amqp/rabbit/listener/SimpleMessageListenerContainer.java > > The spring libraries are fantastic for RabbitMQ work (though sometimes there are some headaches on "Which factory is this? Which MessageProperties do I use?" kinda stuff, but overall they're not bad and they're getting better. I tend to be very careful on how much abstraction I add to my systems (hence why Groovy drives me nuts), but overall spring is has been very clean implementation wise. > > Jason > > > > > > On Wed, Apr 2, 2014 at 3:17 PM, Bertrand Guay-Paquet < > bernie at step.polymtl.ca> wrote: > >> Hi Jason, >> >> Thanks for the hint, but I'm unfortunately not using Spring for this. Do >> you know how Spring achieves this "shutdown" command? I don't know Spring >> at all but I'll have a look at their doc. >> >> Regards, >> Bertrand >> >> >> On 02/04/2014 4:10 PM, Jason McIntosh wrote: >> >> We use consumers here with the spring libraries. If you use them, >> there's a "Shutdown" command you can call as needed. We've had pretty good >> luck with it so far. The spring consumer stuff manages how many consumers >> are running, connection threading, etc. - and it has very good shutdown >> hooks. >> Jason >> >> >> On Wed, Apr 2, 2014 at 3:02 PM, Bertrand Guay-Paquet < >> bernie at step.polymtl.ca> wrote: >> >>> Hello, >>> >>> I'm using the RabbitMQ Java client API and need some guidance on the >>> proper application shutdown procedure. >>> >>> Here's how I start the application: >>> 1-Create a Connection >>> 2-Create different types of consumers, each with its own channel, and >>> call channel.basicConsume("queue", false, consumer) >>> 3-Let it all run >>> >>> This works great, but I can't figure out how to cleanly shutdown the >>> application. If I simply close the Connection, each created channel >>> immediately (or soon enough) becomes invalid and any Consumer currently >>> doing some work fails when trying to ack their current message or perform >>> any other action on the channel. I'd like to let the consumers finish >>> whatever message they're processing and then close everything down. I guess >>> I need to keep track of the created Consumers and somehow signal them to >>> stop accepting new messages and after they're all done with their current >>> job, close the connection? Is that possible or is there another way? I >>> haven't found any management methods for the consumer classes to control or >>> query their status. >>> >>> The information I found so far is related to manually created threads >>> that poll the queues to process messages. In that case, it's really easy >>> because I can just set a flag on each runnable to exit after processing >>> their current message and join on all the threads before closing the >>> underlying connection. So this leads me to ask, as a side note: are >>> Consumers the way to go to use RabbitMQ in real-world scenarios or should I >>> poll on the queues? It seems to me that Consumers would be the better >>> choice (polling is bad), but if they're less powerful, perhaps they're not >>> a silver bullet in my case. >>> >>> Thank you, >>> Bertrand >>> _______________________________________________ >>> rabbitmq-discuss mailing list >>> rabbitmq-discuss at lists.rabbitmq.com >>> https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss >>> >> >> >> >> -- >> Jason McIntosh >> https://github.com/jasonmcintosh/ >> 573-424-7612 >> >> >> _______________________________________________ >> rabbitmq-discuss mailing listrabbitmq-discuss at lists.rabbitmq.comhttps://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss >> >> >> >> _______________________________________________ >> rabbitmq-discuss mailing list >> rabbitmq-discuss at lists.rabbitmq.com >> https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss >> >> > > > -- > Jason McIntosh > https://github.com/jasonmcintosh/ > 573-424-7612 > > _______________________________________________ > rabbitmq-discuss mailing list > rabbitmq-discuss at lists.rabbitmq.com > https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mklishin at gopivotal.com Wed Apr 2 21:50:42 2014 From: mklishin at gopivotal.com (Michael Klishin) Date: Thu, 3 Apr 2014 00:50:42 +0400 Subject: [rabbitmq-discuss] SharedQueue closed error In-Reply-To: <1396471166545-34504.post@n5.nabble.com> References: <1396471166545-34504.post@n5.nabble.com> Message-ID: On 3 April 2014 at 00:40:40, anate (ambatinr at hotmail.com) wrote: > > I declared a queue with persistent message using PHP and trying > to consume > the message in C#. I received 'SharedQueue closed error' at this > line. What is in RabbitMQ log (when you see the exception)? > channel.BasicAck(ea.DeliveryTag, false); > I am assuming the persistent message shouldn't delete from queue > but after > Dequeue() message getting deleted. Your understanding is not correct. Persistent messages survive RabbitMQ restart. They are deleted from the queue when they are acknowledged. Since you use?channel.BasicConsume("hello-queue", true, consumer), messages are considered to be acknowledged as soon as RabbitMQ sends them out to consumers. There is no need to ack them (and acking them will result in a protocol exception). QueueingConsumer will accumulate deliveries locally in a shared queue. My guess is that QueueingConsumer shuts the shared internal queue down when a protocol exception occurs. Judging from your code you don?t need to use a QueueingConsumer. Simply subclass DefaultConsumer and override HandleDelivery. HTH.? -- MK Software Engineer, Pivotal/RabbitMQ From bernie at step.polymtl.ca Wed Apr 2 22:52:00 2014 From: bernie at step.polymtl.ca (Bertrand Guay-Paquet) Date: Wed, 02 Apr 2014 17:52:00 -0400 Subject: [rabbitmq-discuss] How to shutdown cleanly a Java application using Consumers? In-Reply-To: References: <533C6CCB.6070505@step.polymtl.ca> <533C703C.6030509@step.polymtl.ca> Message-ID: <533C8680.5070908@step.polymtl.ca> Thanks Jason and Gary, Your replies really helped and I almost got it now. At first, I didn't know that handleCancelOk only runs when a consumer is not handling anything else. That is, only a single handle*() method can run at the same time on any given consumer instance. Thinking about it now, I realize this was explained in the doc with the phrase "Each Channel has its own dispatch thread." So I created my subclass of DefaultConsumer (BasicConsumer) which implements isCancelled() and handleCancelOk() which sets a "cancelled flag" to true. Is there already a built-in way to know if a consumer was cancelled? I couldn't find it myself... Here are my start and stop sequences which almost completely work. I haven't found how to forcefully stop a badly behaving Consumer after a given timeout at the end of the stop sequence. How can this be achieved? Start sequence: consumers = new ArrayList(); consumers.add(...) Stop sequence: // Cancel all consumers for (BasicConsumer consumer : consumers) { try { consumer.getChannel().basicCancel(consumer.getConsumerTag()); } catch (Exception e) { // report } } // Wait for all consumers to be cancelled Timeout timeout = ...; while (!consumers.isEmpty() && !timeout.isElapsed()) { // Remove cancelled consumers for (Iterator iterator = consumers.iterator(); iterator.hasNext();) { if (iterator.next().isCancelled()) iterator.remove(); } } // Force close remaining (timed-out) consumers for (BasicConsumer consumer : consumers) { consumer.getChannel().abort(); // <---- doesn'k kill the consumer // <---------------- How do I kill a timed-out consumer? } connection.close(); Regards, Bertrand From grussell at gopivotal.com Wed Apr 2 23:23:33 2014 From: grussell at gopivotal.com (Gary Russell) Date: Wed, 2 Apr 2014 18:23:33 -0400 Subject: [rabbitmq-discuss] How to shutdown cleanly a Java application using Consumers? In-Reply-To: <533C8680.5070908@step.polymtl.ca> References: <533C6CCB.6070505@step.polymtl.ca> <533C703C.6030509@step.polymtl.ca> <533C8680.5070908@step.polymtl.ca> Message-ID: In Spring-AMQP, we don't asynchronously cancel the consumer, we tell the consumer it needs to cancel itself when it's finished its current work. IIRC, if the consumer is running when you cancel it, its basicAck will fail because you've already cancelled it and the broker has re-queued that message. Effectively we tell the consumer to emit the cancel immediately after the ack. >stop a badly behaving Consumer It depends on how badly behaved it is; if it never executes any interruptible code (say it's stuck in a while() loop), you are out of luck. If it does stuff that's interruptible (acquire locks, etc, etc) then you can interrupt it. I don't know enough about the rabbit client internals as to whether there is a mechanism to force it to interrupt the dispatcher thread (I didn't see anything after a quick look) so you might have to roll your own (capture a reference to the thread in handleDelivery and interrupt it after your timeout). When all else fails, System.exit() is your friend. Gary On Wed, Apr 2, 2014 at 5:52 PM, Bertrand Guay-Paquet wrote: > Thanks Jason and Gary, > > Your replies really helped and I almost got it now. At first, I didn't > know that handleCancelOk only runs when a consumer is not handling anything > else. That is, only a single handle*() method can run at the same time on > any given consumer instance. Thinking about it now, I realize this was > explained in the doc with the phrase "Each Channel has its own dispatch > thread." > > So I created my subclass of DefaultConsumer (BasicConsumer) which > implements isCancelled() and handleCancelOk() which sets a "cancelled flag" > to true. Is there already a built-in way to know if a consumer was > cancelled? I couldn't find it myself... > > Here are my start and stop sequences which almost completely work. I > haven't found how to forcefully stop a badly behaving Consumer after a > given timeout at the end of the stop sequence. How can this be achieved? > > Start sequence: > consumers = new ArrayList(); > consumers.add(...) > > Stop sequence: > // Cancel all consumers > for (BasicConsumer consumer : consumers) { > try { > consumer.getChannel().basicCancel(consumer.getConsumerTag()); > } catch (Exception e) { > // report > } > } > > // Wait for all consumers to be cancelled > Timeout timeout = ...; > while (!consumers.isEmpty() && !timeout.isElapsed()) { > // Remove cancelled consumers > for (Iterator iterator = consumers.iterator(); > iterator.hasNext();) { > if (iterator.next().isCancelled()) > iterator.remove(); > } > } > > // Force close remaining (timed-out) consumers > for (BasicConsumer consumer : consumers) { > consumer.getChannel().abort(); // <---- doesn'k kill the consumer > // <---------------- How do I kill a timed-out consumer? > } > connection.close(); > > Regards, > Bertrand > _______________________________________________ > rabbitmq-discuss mailing list > rabbitmq-discuss at lists.rabbitmq.com > https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss > -------------- next part -------------- An HTML attachment was scrubbed... URL: From grussell at gopivotal.com Wed Apr 2 23:34:42 2014 From: grussell at gopivotal.com (Gary Russell) Date: Wed, 2 Apr 2014 18:34:42 -0400 Subject: [rabbitmq-discuss] How to shutdown cleanly a Java application using Consumers? In-Reply-To: References: <533C6CCB.6070505@step.polymtl.ca> <533C703C.6030509@step.polymtl.ca> <533C8680.5070908@step.polymtl.ca> Message-ID: I just took another look at the client - with the default (internal private) ExecutorService, it does an orderly shutdown() which won't interrupt the threads. If you pass in your own ExecutorService to newConnection() you can call shutDownNow() which *will* interrupt the dispatcher threads (if they are doing something that's interruptible). See ExecutorService.shutDownNow() for more information. Gary On Wed, Apr 2, 2014 at 6:23 PM, Gary Russell wrote: > In Spring-AMQP, we don't asynchronously cancel the consumer, we tell the > consumer it needs to cancel itself when it's finished its current work. > IIRC, if the consumer is running when you cancel it, its basicAck will fail > because you've already cancelled it and the broker has re-queued that > message. Effectively we tell the consumer to emit the cancel immediately > after the ack. > > >stop a badly behaving Consumer > > It depends on how badly behaved it is; if it never executes any > interruptible code (say it's stuck in a while() loop), you are out of luck. > If it does stuff that's interruptible (acquire locks, etc, etc) then you > can interrupt it. > > I don't know enough about the rabbit client internals as to whether there > is a mechanism to force it to interrupt the dispatcher thread (I didn't see > anything after a quick look) so you might have to roll your own (capture a > reference to the thread in handleDelivery and interrupt it after your > timeout). > > When all else fails, System.exit() is your friend. > > Gary > > > > On Wed, Apr 2, 2014 at 5:52 PM, Bertrand Guay-Paquet < > bernie at step.polymtl.ca> wrote: > >> Thanks Jason and Gary, >> >> Your replies really helped and I almost got it now. At first, I didn't >> know that handleCancelOk only runs when a consumer is not handling anything >> else. That is, only a single handle*() method can run at the same time on >> any given consumer instance. Thinking about it now, I realize this was >> explained in the doc with the phrase "Each Channel has its own dispatch >> thread." >> >> So I created my subclass of DefaultConsumer (BasicConsumer) which >> implements isCancelled() and handleCancelOk() which sets a "cancelled flag" >> to true. Is there already a built-in way to know if a consumer was >> cancelled? I couldn't find it myself... >> >> Here are my start and stop sequences which almost completely work. I >> haven't found how to forcefully stop a badly behaving Consumer after a >> given timeout at the end of the stop sequence. How can this be achieved? >> >> Start sequence: >> consumers = new ArrayList(); >> consumers.add(...) >> >> Stop sequence: >> // Cancel all consumers >> for (BasicConsumer consumer : consumers) { >> try { >> consumer.getChannel().basicCancel(consumer.getConsumerTag()); >> } catch (Exception e) { >> // report >> } >> } >> >> // Wait for all consumers to be cancelled >> Timeout timeout = ...; >> while (!consumers.isEmpty() && !timeout.isElapsed()) { >> // Remove cancelled consumers >> for (Iterator iterator = consumers.iterator(); >> iterator.hasNext();) { >> if (iterator.next().isCancelled()) >> iterator.remove(); >> } >> } >> >> // Force close remaining (timed-out) consumers >> for (BasicConsumer consumer : consumers) { >> consumer.getChannel().abort(); // <---- doesn'k kill the consumer >> // <---------------- How do I kill a timed-out consumer? >> } >> connection.close(); >> >> Regards, >> Bertrand >> _______________________________________________ >> rabbitmq-discuss mailing list >> rabbitmq-discuss at lists.rabbitmq.com >> https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From riccardi at systran.fr Thu Apr 3 09:26:53 2014 From: riccardi at systran.fr (Thomas Riccardi) Date: Thu, 03 Apr 2014 10:26:53 +0200 Subject: [rabbitmq-discuss] Wrong link to changelog for RabbitMQ Server 3.3.0 Message-ID: <1396513613.4571.42.camel@CLM3X4J.systran.local> Hi, The link to the 3.3.0 changelog here http://www.rabbitmq.com/changelog.html points to 3.2.0, not 3.3.0. The 3.3.0 page exists though: http://www.rabbitmq.com/release-notes/README-3.3.0.txt Thanks, Thomas From matthias at rabbitmq.com Thu Apr 3 09:40:18 2014 From: matthias at rabbitmq.com (Matthias Radestock) Date: Thu, 03 Apr 2014 09:40:18 +0100 Subject: [rabbitmq-discuss] Wrong link to changelog for RabbitMQ Server 3.3.0 In-Reply-To: <1396513613.4571.42.camel@CLM3X4J.systran.local> References: <1396513613.4571.42.camel@CLM3X4J.systran.local> Message-ID: <533D1E72.7090100@rabbitmq.com> On 03/04/14 09:26, Thomas Riccardi wrote: > The link to the 3.3.0 changelog here > http://www.rabbitmq.com/changelog.html points to 3.2.0, not 3.3.0. Well spotted. Fixed. Matthias. From michael.s.klishin at gmail.com Thu Apr 3 10:38:15 2014 From: michael.s.klishin at gmail.com (Michael Klishin) Date: Thu, 3 Apr 2014 13:38:15 +0400 Subject: [rabbitmq-discuss] ANN Langohr 2.8.2 is released Message-ID: Langohr 2.8.2 has no change log entry: there are no user facing changes from 2.8.1, only some corrections to make it compile against RabbitMQ Java client 3.3.0 from scratch (after `lein clean`). -- MK http://github.com/michaelklishin http://twitter.com/michaelklishin -------------- next part -------------- An HTML attachment was scrubbed... URL: From klk at mmw.ag Thu Apr 3 13:17:05 2014 From: klk at mmw.ag (Hans-Joachim Kliemeck) Date: Thu, 03 Apr 2014 14:17:05 +0200 Subject: [rabbitmq-discuss] Extending the java rabbitmq client Message-ID: <533D5141.6080400@mmw.ag> Hey guys, i'm new to the rabbitmq client list and want to extend the client regarding a simple failover. Since i did not found any information where to submit a patch, i hope that i'm submitting my change to the correct mailing list. I've discovered that there is already a possibility to auto-reconnect the client ("automaticRecovery") but i want to make advantage of the multiple addresses capability on newConnection. Attached a patch for the change. greets Hans-Joachim Kliemeck -- marken mehrwert - brand added value AG Tel.: +49 (0) 621-761395-748 Tel.: +49 (0) 175-5276327 Julius-Hatry-Strasse 1, 68163 Mannheim, Germany Vorstand: Frank Schwarz Aufsichtsrat: Prof. Dr. Stefan Jugel (Vorsitzender), Prof. Dr. Michael Woywode, Prof. Dr. h.c. Dietmar von Hoyningen Huene USt-IdNr. DE258032509, Registergericht Mannheim HRB 704123 -------------- next part -------------- A non-text attachment was scrubbed... Name: ConnectionFactory.java.patch Type: text/x-patch Size: 3734 bytes Desc: not available URL: From mklishin at gopivotal.com Thu Apr 3 13:20:58 2014 From: mklishin at gopivotal.com (Michael Klishin) Date: Thu, 3 Apr 2014 16:20:58 +0400 Subject: [rabbitmq-discuss] Extending the java rabbitmq client In-Reply-To: <533D5141.6080400@mmw.ag> References: <533D5141.6080400@mmw.ag> Message-ID: ?On 3 April 2014 at 16:17:49, Hans-Joachim Kliemeck (klk at mmw.ag) wrote: > > I've discovered that there is already a possibility to auto-reconnect > the client ("automaticRecovery") but i want to make advantage > of the > multiple addresses capability on newConnection. Attached > a patch for the > change. If you use ConnectionFactory#newConnection with an array of addresses, they will be shuffled before every connection attempt: http://hg.rabbitmq.com/rabbitmq-java-client/file/53441f4809ac/src/com/rabbitmq/client/impl/recovery/RecoveryAwareAMQConnectionFactory.java#l32 -- MK Software Engineer, Pivotal/RabbitMQ From klk at mmw.ag Thu Apr 3 13:30:57 2014 From: klk at mmw.ag (Hans-Joachim Kliemeck) Date: Thu, 03 Apr 2014 14:30:57 +0200 Subject: [rabbitmq-discuss] Extending the java rabbitmq client In-Reply-To: References: <533D5141.6080400@mmw.ag> Message-ID: <533D5481.2050102@mmw.ag> Hey, sure, but the current implementation lacks of support regarding the URI configuration method (which i thing is very sexy). It is possible that the patch will be included or do you have any objections against it? Regarding the shuffling, maybe it is better to call them additionalHosts or something like that. greets Hans-Joachim Am 03.04.2014 14:20, schrieb Michael Klishin: > On 3 April 2014 at 16:17:49, Hans-Joachim Kliemeck (klk at mmw.ag) wrote: >>> I've discovered that there is already a possibility to auto-reconnect >> the client ("automaticRecovery") but i want to make advantage >> of the >> multiple addresses capability on newConnection. Attached >> a patch for the >> change. > > If you use ConnectionFactory#newConnection with an array of addresses, > they will be shuffled before every connection attempt: > > http://hg.rabbitmq.com/rabbitmq-java-client/file/53441f4809ac/src/com/rabbitmq/client/impl/recovery/RecoveryAwareAMQConnectionFactory.java#l32 > -- > MK > > Software Engineer, Pivotal/RabbitMQ > _______________________________________________ > rabbitmq-discuss mailing list > rabbitmq-discuss at lists.rabbitmq.com > https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss > -- marken mehrwert - brand added value AG Tel.: +49 (0) 621-761395-748 Tel.: +49 (0) 175-5276327 Julius-Hatry-Strasse 1, 68163 Mannheim, Germany Vorstand: Frank Schwarz Aufsichtsrat: Prof. Dr. Stefan Jugel (Vorsitzender), Prof. Dr. Michael Woywode, Prof. Dr. h.c. Dietmar von Hoyningen Huene USt-IdNr. DE258032509, Registergericht Mannheim HRB 704123 From simon at rabbitmq.com Thu Apr 3 14:11:23 2014 From: simon at rabbitmq.com (Simon MacMullen) Date: Thu, 03 Apr 2014 14:11:23 +0100 Subject: [rabbitmq-discuss] Upgrade fail In-Reply-To: References: Message-ID: <533D5DFB.5080405@rabbitmq.com> On 02/04/14 20:19, Peter Kopias wrote: > Hi. > > I've tried to upgrade my cluster to 3.3.0. > BOOT FAILED > =========== > > Timeout contacting cluster nodes: [rabbit at node3,rabbit at node2]. > > DIAGNOSTICS > =========== > > attempted to contact: [rabbit at node3,rabbit at node2] > > rabbit at node3: > * found rabbit (port 25672) > * TCP connection succeeded > * suggestion: hostname mismatch? > * suggestion: is the cookie set correctly? > rabbit at node2: > * found rabbit (port 25672) > * TCP connection succeeded > * suggestion: hostname mismatch? > * suggestion: is the cookie set correctly? This is quite weird. The diagnostics indicate that there actually is some process up, registered with epmd and listening on the Erlang distribution port for both node2 and node3. > WHY is it waiting for node2, node3, if node1 was the last to stop, and > it should come online in itself without them? Very good question. I was able to reproduce this precise error message by doing the equivalent of: (stop node1) rabbitmqctl -n node2 stop_app rabbitmqctl -n node3 stop_app (start node1) i./e. leaving the Erlang VM running but stopping RabbitMQ (and Mnesia) on node2 and node3. Is there any possibility at all you could have done something like this? Is there a beam.smp process running on those nodes? I am especially puzzled since the nodes have registered 25672 as a distribution port - prior to 3.3.0 the distribution port would be chosen at random, or you could configure it. So the presence of 25672 strongly indicates that those nodes have been upgraded to 3.3.0 and are running an Erlang VM - with RabbitMQ stopped? > The cookie is the same, the hostnames are correct. If I am correct about the diagnosis above then the diagnostics don't currently handle this situation well; they don't check for the case where Erlang is running but RabbitMQ isn't. This will get fixed in the next release. > I'm trying to start node2 and node3 to have at least one running but > they always fail, even if I copied back the previous disk state, and > upgraded again... How do they fail? Because it sounds like something is running. > Not a single node running currently, looks like I have to: > - reinstall each rabbitmq-server with clean /var/lib/rabbitmq dirs > - rebuild cluster > - rebuild the virtual hosts > - recreate all users and permissions > - all queues and exchanges and policies > > Thats not the way upgrade should happen. :( No, it really isn't. :-( Cheers, Simon -- Simon MacMullen RabbitMQ, Pivotal From mklishin at gopivotal.com Thu Apr 3 14:13:31 2014 From: mklishin at gopivotal.com (Michael Klishin) Date: Thu, 3 Apr 2014 17:13:31 +0400 Subject: [rabbitmq-discuss] Extending the java rabbitmq client In-Reply-To: <533D5481.2050102@mmw.ag> References: <533D5141.6080400@mmw.ag> <533D5481.2050102@mmw.ag> Message-ID: On 3 April 2014 at 16:31:31, Hans-Joachim Kliemeck (klk at mmw.ag) wrote: > > sure, but the current implementation lacks of support regarding > the URI > configuration method (which i thing is very sexy). It is possible > that > the patch will be included or do you have any objections against > it? I?m not against this feature but there are two issues with your patch as it is: 1. It entirely lacks tests 2. It makes the API confusing: there are 2 ways of providing multiple addresses ? ?and it?s not clear how newConnection?s addresses is different from?setFailoverHosts. I?d rather provide a way to set extra addresses which are then merged with the list given to newConnection or via setUri: factory.setClusterNodes(list); or similar. Picking a good name is a very important part.? -- MK Software Engineer, Pivotal/RabbitMQ From simon at rabbitmq.com Thu Apr 3 14:20:10 2014 From: simon at rabbitmq.com (Simon MacMullen) Date: Thu, 03 Apr 2014 14:20:10 +0100 Subject: [rabbitmq-discuss] Removal of the 'impersonator' tag from 3.3.0 In-Reply-To: References: <533C390D.2060907@rabbitmq.com> Message-ID: <533D600A.6040701@rabbitmq.com> On 02/04/14 18:11, Gotthard, Petr wrote: > In other words, the "escaped" implementation detail was used to implement > user_id validation in upstream-controlled federations. Does it make sense? Thanks, that does. And I can't think of a good way to do what you want without it. Hmm. Cheers, Simon -- Simon MacMullen RabbitMQ, Pivotal From videlalvaro at gmail.com Thu Apr 3 15:04:28 2014 From: videlalvaro at gmail.com (Alvaro Videla) Date: Thu, 3 Apr 2014 16:04:28 +0200 Subject: [rabbitmq-discuss] ANN new release for Recent History Exchange Message-ID: Hi, There's a new release of the Recent History Exchange. There's only a small change to accommodate to an API change in the broker, so there's no user facing changes. See more here: https://github.com/videlalvaro/rabbitmq-recent-history-exchange Regards, Alvaro From stammailbox at gmail.com Thu Apr 3 15:06:22 2014 From: stammailbox at gmail.com (rails) Date: Thu, 3 Apr 2014 17:06:22 +0300 Subject: [rabbitmq-discuss] Java Client Using DefaultConsumer vs QueueingConsumer Message-ID: 1. My DemoConsumer inherits from DefaultConsumer I have noticed that working this way handleDelivery() is invoked from ThreadPool. (printing Thread.currentThread().getName() I see pool-1-thread-1/2/3/4 eachtime. I have also tested it several times and saw that the order is saved. Just to make sure - since different threads call handle delivery - will it mess my order? 2. QueueingConsumer All of the java tutorial use QueueingConsumer to consume messages. In the API Docs it is mentioned as a deprecated class. Should I change my code to inherit from DefaultConsumer use it? Is the tutorial outdated? Thanks. -------------- next part -------------- An HTML attachment was scrubbed... URL: From ambatinr at hotmail.com Thu Apr 3 15:09:17 2014 From: ambatinr at hotmail.com (anate) Date: Thu, 3 Apr 2014 07:09:17 -0700 (PDT) Subject: [rabbitmq-discuss] SharedQueue closed error In-Reply-To: References: <1396471166545-34504.post@n5.nabble.com> Message-ID: <1396534157179-34528.post@n5.nabble.com> Thank you for the reply. I am trying to insert the data into database and would like to keep the message in RabbitMQ if the insert fails. I am not sure if I am in right path. Could you please let me know more info about "subclass DefaultConsumer and override HandleDelivery"? Please let me know if there are any code examples. > QueueingConsumer will accumulate deliveries locally in a shared queue. > My guess is that QueueingConsumer shuts the shared internal queue down > when a protocol exception occurs. > Judging from your code you don?t need to use a QueueingConsumer. Simply > subclass DefaultConsumer and override HandleDelivery. -- View this message in context: http://rabbitmq.1065348.n5.nabble.com/SharedQueue-closed-error-tp34504p34528.html Sent from the RabbitMQ mailing list archive at Nabble.com. From ambatinr at hotmail.com Thu Apr 3 15:20:17 2014 From: ambatinr at hotmail.com (anate) Date: Thu, 3 Apr 2014 07:20:17 -0700 (PDT) Subject: [rabbitmq-discuss] SharedQueue closed error In-Reply-To: References: <1396471166545-34504.post@n5.nabble.com> Message-ID: <1396534817495-34529.post@n5.nabble.com> Here is the error message in the logs. {amqp_error,precondition_failed, "parameters for queue 'hello-queue' in vhost '/' not equivalent", 'queue.declare'} channel 1 - soft error: {amqp_error,precondition_failed,"unknown delivery tag 1",'basic.ack'}: connection_closed_abruptly -- View this message in context: http://rabbitmq.1065348.n5.nabble.com/SharedQueue-closed-error-tp34504p34529.html Sent from the RabbitMQ mailing list archive at Nabble.com. From mklishin at gopivotal.com Thu Apr 3 16:00:13 2014 From: mklishin at gopivotal.com (Michael Klishin) Date: Thu, 3 Apr 2014 19:00:13 +0400 Subject: [rabbitmq-discuss] SharedQueue closed error In-Reply-To: <1396534157179-34528.post@n5.nabble.com> References: <1396471166545-34504.post@n5.nabble.com> <1396534157179-34528.post@n5.nabble.com> Message-ID: On 3 April 2014 at 18:10:27, anate (ambatinr at hotmail.com) wrote: > > Thank you for the reply. I am trying to insert the data into database > and > would like to keep the message in RabbitMQ if the insert fails. > I am not > sure if I am in right path You need to use manual acknowledgement mode: http://www.rabbitmq.com/tutorials/tutorial-two-dotnet.html? -- MK Software Engineer, Pivotal/RabbitMQ From mklishin at gopivotal.com Thu Apr 3 16:01:47 2014 From: mklishin at gopivotal.com (Michael Klishin) Date: Thu, 3 Apr 2014 19:01:47 +0400 Subject: [rabbitmq-discuss] SharedQueue closed error In-Reply-To: <1396534817495-34529.post@n5.nabble.com> References: <1396471166545-34504.post@n5.nabble.com> <1396534817495-34529.post@n5.nabble.com> Message-ID: On 3 April 2014 at 18:21:29, anate (ambatinr at hotmail.com) wrote: > > {amqp_error,precondition_failed, > "parameters for queue 'hello-queue' in vhost '/' not > equivalent", > 'queue.declare'} > > channel 1 - soft error: > {amqp_error,precondition_failed,"unknown delivery tag > 1",'basic.ack'}: > connection_closed_abruptly This is exactly the error I mentioned earlier in this thread. Use manual acknowledgements and set a basic.qos limit (say, 25-50 messages) on the channel you use. basic.qos and requeueing are covered in Tutorial 2: http://www.rabbitmq.com/tutorials/tutorial-two-dotnet.html? -- MK Software Engineer, Pivotal/RabbitMQ From mklishin at gopivotal.com Thu Apr 3 16:04:08 2014 From: mklishin at gopivotal.com (Michael Klishin) Date: Thu, 3 Apr 2014 19:04:08 +0400 Subject: [rabbitmq-discuss] Java Client Using DefaultConsumer vs QueueingConsumer In-Reply-To: References: Message-ID: On 3 April 2014 at 18:09:06, rails (stammailbox at gmail.com) wrote: > > 1. My DemoConsumer inherits from DefaultConsumer > I have noticed that working this way handleDelivery() is invoked > from ThreadPool. > (printing Thread.currentThread().getName() I see pool-1-thread-1/2/3/4 > eachtime. > I have also tested it several times and saw that the order is saved. > Just to make sure - since different threads call handle delivery > - will it mess my order? By default the pool has 5 threads, so yes. You can provide your own executor on ConnectionFactory, e.g. to use a single threaded executor. > 2. QueueingConsumer All of the java tutorial use QueueingConsumer > to consume messages. > In the API Docs it is mentioned as a deprecated class. > Should I change my code to inherit from DefaultConsumer use it? > Is the tutorial outdated? I believe it was undeprecated after some of its limitations were documented . On what pages do you see this mentioned? -- MK Software Engineer, Pivotal/RabbitMQ From bernie at step.polymtl.ca Thu Apr 3 17:28:30 2014 From: bernie at step.polymtl.ca (Bertrand Guay-Paquet) Date: Thu, 03 Apr 2014 12:28:30 -0400 Subject: [rabbitmq-discuss] How to shutdown cleanly a Java application using Consumers? In-Reply-To: References: <533C6CCB.6070505@step.polymtl.ca> <533C703C.6030509@step.polymtl.ca> <533C8680.5070908@step.polymtl.ca> Message-ID: <533D8C2E.9090401@step.polymtl.ca> An HTML attachment was scrubbed... URL: From simon at rabbitmq.com Thu Apr 3 17:47:15 2014 From: simon at rabbitmq.com (Simon MacMullen) Date: Thu, 03 Apr 2014 17:47:15 +0100 Subject: [rabbitmq-discuss] How to shutdown cleanly a Java application using Consumers? In-Reply-To: <533D8C2E.9090401@step.polymtl.ca> References: <533C6CCB.6070505@step.polymtl.ca> <533C703C.6030509@step.polymtl.ca> <533C8680.5070908@step.polymtl.ca> <533D8C2E.9090401@step.polymtl.ca> Message-ID: <533D9093.2020304@rabbitmq.com> On 03/04/14 17:28, Bertrand Guay-Paquet wrote: > I tested this on RabbitMQ 3.2.4 and calling basicCancel for a consumer > tag does not interfere with the ability to basicAck a message currently > being processed That is correct. > (I used Thread.sleep() in the consumer to ensure the > operation order). The handleCancelOk() method is only called after the > handleDelivery() method returns. That is also correct. > From what I could piece together, calling basicCancel queues the > channel rpc command for execution after whatever is already queued. The > consumer even consumes its buffered messages (basicQos > 1) before > processing the cancel command. So there has been a certain lack of clarity in this thread. Two key points: * It is absolutely fine and normal to acknowledge a message after its consumer has been cancelled. The message remains acknowledgeable until the channel it was delivered on closes. * Once you have received basic.cancel-ok, you will not receive any more deliveries for that consumer. So the way to cleanly[0] shut down a consumer in the Java client is: 1) Invoke basicCancel() 2) Process and acknowledge messages as usual 3) When handleCancelOk() gets called and you have acknowledged all messages you received before it, you can then consider yourself done, close channels and connections as appropriate. [0] I am assuming "cleanly" here means without causing messages to be redelivered. Otherwise you can just pull the network cable :-) Cheers, Simon -- Simon MacMullen RabbitMQ, Pivotal From michael.laing at nytimes.com Thu Apr 3 17:54:20 2014 From: michael.laing at nytimes.com (Laing, Michael) Date: Thu, 3 Apr 2014 12:54:20 -0400 Subject: [rabbitmq-discuss] How to shutdown cleanly a Java application using Consumers? In-Reply-To: <533D9093.2020304@rabbitmq.com> References: <533C6CCB.6070505@step.polymtl.ca> <533C703C.6030509@step.polymtl.ca> <533C8680.5070908@step.polymtl.ca> <533D8C2E.9090401@step.polymtl.ca> <533D9093.2020304@rabbitmq.com> Message-ID: I found this thread interesting but... we just pull the cable and let a peer take over :) ml On Thu, Apr 3, 2014 at 12:47 PM, Simon MacMullen wrote: > On 03/04/14 17:28, Bertrand Guay-Paquet wrote: > >> I tested this on RabbitMQ 3.2.4 and calling basicCancel for a consumer >> tag does not interfere with the ability to basicAck a message currently >> being processed >> > > That is correct. > > > (I used Thread.sleep() in the consumer to ensure the >> operation order). The handleCancelOk() method is only called after the >> handleDelivery() method returns. >> > > That is also correct. > > > From what I could piece together, calling basicCancel queues the >> channel rpc command for execution after whatever is already queued. The >> consumer even consumes its buffered messages (basicQos > 1) before >> processing the cancel command. >> > > So there has been a certain lack of clarity in this thread. Two key points: > > * It is absolutely fine and normal to acknowledge a message after its > consumer has been cancelled. The message remains acknowledgeable until the > channel it was delivered on closes. > > * Once you have received basic.cancel-ok, you will not receive any more > deliveries for that consumer. > > So the way to cleanly[0] shut down a consumer in the Java client is: > > 1) Invoke basicCancel() > 2) Process and acknowledge messages as usual > 3) When handleCancelOk() gets called and you have acknowledged all > messages you received before it, you can then consider yourself done, close > channels and connections as appropriate. > > [0] I am assuming "cleanly" here means without causing messages to be > redelivered. Otherwise you can just pull the network cable :-) > > Cheers, Simon > > -- > Simon MacMullen > RabbitMQ, Pivotal > > _______________________________________________ > rabbitmq-discuss mailing list > rabbitmq-discuss at lists.rabbitmq.com > https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss > -------------- next part -------------- An HTML attachment was scrubbed... URL: From michael.laing at nytimes.com Thu Apr 3 18:30:12 2014 From: michael.laing at nytimes.com (Laing, Michael) Date: Thu, 3 Apr 2014 13:30:12 -0400 Subject: [rabbitmq-discuss] shovel auth change in 3.3.0? Message-ID: It appears that either the the source brokers or the destination brokers must now have a password provided even when using direct connection to both vhosts, i.e. the shovel is connecting vhosts on the same cluster. Formerly no password was required for either. This is not a big deal, as we do this for dev only and I lazily did not put the passwords in, but it is a change. ml -------------- next part -------------- An HTML attachment was scrubbed... URL: From ajkedar at gmail.com Thu Apr 3 20:39:45 2014 From: ajkedar at gmail.com (Amol Kedar) Date: Thu, 3 Apr 2014 12:39:45 -0700 (PDT) Subject: [rabbitmq-discuss] unable to login to management console after 3.3 upgrade of rabbitmq Message-ID: I am running a single instance node on EC2 and i upgraded to rabbit 3.3 today, once i finished the upgrade i was no longer able to login to the web base management console? what can i check? -------------- next part -------------- An HTML attachment was scrubbed... URL: From ajkedar at gmail.com Thu Apr 3 21:04:23 2014 From: ajkedar at gmail.com (Amol Kedar) Date: Thu, 3 Apr 2014 13:04:23 -0700 (PDT) Subject: [rabbitmq-discuss] unable to login to management console after 3.3 upgrade of rabbitmq In-Reply-To: References: Message-ID: <3134bfe1-d08a-411d-b16e-64c24b983be4@googlegroups.com> well i am only able to login with monitoring user but not the regular application user? what could be the cause? did the upgrade from 3.1 to 3.3 cause this? or is it ec2 amazon configuration? On Thursday, April 3, 2014 3:39:45 PM UTC-4, Amol Kedar wrote: > > I am running a single instance node on EC2 and i upgraded to rabbit 3.3 > today, once i finished the upgrade i was no longer able to login to the web > base management console? > what can i check? > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ajkedar at gmail.com Thu Apr 3 21:43:47 2014 From: ajkedar at gmail.com (Amol Kedar) Date: Thu, 3 Apr 2014 13:43:47 -0700 (PDT) Subject: [rabbitmq-discuss] unable to login to management console after 3.3 upgrade of rabbitmq In-Reply-To: References: Message-ID: <3a16c303-dd0e-49e5-af41-525d19ee30d3@googlegroups.com> i think i found my answer that in 3.3 the guest user can only login via localhost http://www.rabbitmq.com/access-control.html On Thursday, April 3, 2014 3:39:45 PM UTC-4, Amol Kedar wrote: > > I am running a single instance node on EC2 and i upgraded to rabbit 3.3 > today, once i finished the upgrade i was no longer able to login to the web > base management console? > what can i check? > -------------- next part -------------- An HTML attachment was scrubbed... URL: From fred at dushin.net Fri Apr 4 01:19:25 2014 From: fred at dushin.net (Dushin Fred) Date: Thu, 3 Apr 2014 20:19:25 -0400 Subject: [rabbitmq-discuss] rabbitmq-c and SimpleAmqpClient SSL authentication In-Reply-To: References: Message-ID: <632F1629-0DFA-446E-A08E-77ED80DE7CD8@dushin.net> On Mar 30, 2014, at 8:46 PM, Alan Antonuk wrote: > This is probably what you want: https://github.com/alanxz/SimpleAmqpClient/pull/85. In the future feel free to open a pull-request against the library on github. The error you were seeing with boost::make_shared most likely had to do its limit of 10 arguments. Looks good to me. Let me run it through some testing. -Fred -------------- next part -------------- An HTML attachment was scrubbed... URL: From fred at dushin.net Fri Apr 4 01:19:28 2014 From: fred at dushin.net (Dushin Fred) Date: Thu, 3 Apr 2014 20:19:28 -0400 Subject: [rabbitmq-discuss] rabbitmq-c and SimpleAmqpClient SSL authentication In-Reply-To: References: Message-ID: <4B694A5D-3D69-4699-B940-C79ED5E5D57A@dushin.net> And I suppose we would want to plumb this through as a parameter to the SimpleAmqpClient API, as well, no? I can try that. -Fred On Mar 31, 2014, at 1:31 AM, Alan Antonuk wrote: > I haven't tried it out yet, but here's a PR adding the EXTERNAL SASL mechanism to rabbitmq-c. > > https://github.com/alanxz/rabbitmq-c/pull/179 > > Feel free to try it out and let me know how it works. > > -Alan > > > On Sun, Mar 30, 2014 at 5:46 PM, Alan Antonuk wrote: > Fred; > > > On Sun, Mar 30, 2014 at 1:29 PM, Dushin Fred wrote: > Hi Folks, > > I have run into two issues with the (admittedly unsupported) rabbitmq-c and SimpleAmqpClient libraries. (I realize I am hijacking the RabbtiMQ mailing list for this topic, but I am guessing the Pivotal folks don't mind, because this should result in an improvement of the overall RabbitMQ ecosystem.) > > Currently this mailing list is the best place to have discussions concerning these two libraries. > > The first issue is with the SimpleAmqpClient library. It appears that there is no knob in the Channel::CreateSecureChannel operation to disable hostname verification of the RabbitMQ server. There is a knob in the rabbitmq-c API (amqp_ssl_socket_set_verify), but you need a socket to call it, and the only reasonable place to do that is in the Channel constructor. I am including a patch off the github 2.3 tag for doing just that, though for some reason I did not track down, I could not get boost::make_shared take my new constructor signature, so I just used the shared_ptr constructor in the raw. That may not fit the current model, but it seems harmless for testing. > > This is probably what you want: https://github.com/alanxz/SimpleAmqpClient/pull/85. In the future feel free to open a pull-request against the library on github. The error you were seeing with boost::make_shared most likely had to do its limit of 10 arguments. > > > If I look at the rabbitmq-c code, I see that the C client library and API only seems to support the PLAIN SASL method. > > That is correct. I'm open to adding support for additional SASL mechanisms to rabbitmq-c. If the implementation of the SASL mechanism is anything more than trivial (e.g., PLAIN really is dead-simple), rabbitmq-c should probably link in a thoroughly tested external library (like cyrus-SASL). Given use of these alternate SASL mechanisms appears to be very low, any external library dependancies should remain optional. I can provide some other hints to get started adding this to rabbitmq-c if you so desire. > > Any pointers to docs on the EXTERNAL mechanism (at least WRT SSL client auth) or existing code would be greatly appreciated. > > The RFC for SASL includes a pretty good description of how the SASL EXTERNAL method should operate: http://tools.ietf.org/html/rfc4422#appendix-A (it actually looks pretty simple to implement). > > > -Alan > > > _______________________________________________ > rabbitmq-discuss mailing list > rabbitmq-discuss at lists.rabbitmq.com > https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: From fred at dushin.net Fri Apr 4 01:19:31 2014 From: fred at dushin.net (Dushin Fred) Date: Thu, 3 Apr 2014 20:19:31 -0400 Subject: [rabbitmq-discuss] rabbitmq-c memory leak? Message-ID: I have written a simple test program using the SimpleAqmpClient library (which in turn uses the rabbitmq-c library), which seems to illustrate a memory leak, of sorts. The program is a simple main, which pre-allocates a string message, and then calls BasicPublish on a channel in a loop, all single threaded. Effectively: auto envelope = BasicMessage::Create(random_data(message_size)); for (unsigned i = 0; i < num_messages; ++i) { channel->BasicPublish(exchange_name, routing_key, envelope); } The problem is that the memory image of this process seems to grow without bounds. The memory all seems to be allocated in amqp_pool_alloc (as expected), when calling wait_frame_inner -> amqp_handle_input. The memory is not orphaned -- i.e., it seems to get cleaned up at shutdown, so it doesn't show up as a leak in a leak detector tool, such a Instruments/DTrace, or valgrind, or purify, etc. But I don't see where the memory is every de-allocated, which will spell doom for an application that is designed to never terminate. I am attaching a screenshot from an Instruments run, to give some context. The code is here: https://github.com/fadushin/sandbox/blob/master/cpp/rabbitmq/sender.cpp This was run against RabbitMQ 3.2.4, using rabbitmq-c 0.5.0 and SimpleAmqpClient 2.3, compiled using Clang 5.0 on OS X.9 (Mavericks). Has anyone seen this behavior? Am I missing something in my use of BasicPublish, that is not telling the rabbitmq-c library to free memory in the "amqp pool"? It doesn't seem to make a difference whether I am actively pulling messages off the queue from a separate process, or even whether there is a queue bound to the exchange to which I am publishing messages. (Not that I would expect it to) -Fred encl. -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: PastedGraphic-3.png Type: image/png Size: 422159 bytes Desc: not available URL: From arun.achutha at gmail.com Fri Apr 4 08:17:37 2014 From: arun.achutha at gmail.com (arun kumar) Date: Fri, 4 Apr 2014 12:47:37 +0530 Subject: [rabbitmq-discuss] shovel auth change in 3.3.0? In-Reply-To: References: Message-ID: Hi Laing, Michael i'm john working on python modules i have little doubt about the parameters in callback method what we have in the tutorials i'm very new any suggesstions will help me alot Regards, John On Thu, Apr 3, 2014 at 11:00 PM, Laing, Michael wrote: > It appears that either the the source brokers or the destination brokers > must now have a password provided even when using direct connection to both > vhosts, i.e. the shovel is connecting vhosts on the same cluster. > > Formerly no password was required for either. > > This is not a big deal, as we do this for dev only and I lazily did not > put the passwords in, but it is a change. > > ml > > _______________________________________________ > rabbitmq-discuss mailing list > rabbitmq-discuss at lists.rabbitmq.com > https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jean-philippe.noel at optiflows.com Fri Apr 4 10:37:08 2014 From: jean-philippe.noel at optiflows.com (=?ISO-8859-1?Q?Jean-Philippe_No=EBl?=) Date: Fri, 4 Apr 2014 11:37:08 +0200 Subject: [rabbitmq-discuss] Acknowledgement in mirrored queues Message-ID: <533E7D44.1010501@optiflows.com> Hi, I am trying to understand what happens when a consumer acks a message from a mirrored queue. Does this ack remain in the local queue or is it replicated on the other queues ? Thank you for your help Jean-Philippe From simon at rabbitmq.com Fri Apr 4 11:17:22 2014 From: simon at rabbitmq.com (Simon MacMullen) Date: Fri, 04 Apr 2014 11:17:22 +0100 Subject: [rabbitmq-discuss] Acknowledgement in mirrored queues In-Reply-To: <533E7D44.1010501@optiflows.com> References: <533E7D44.1010501@optiflows.com> Message-ID: <533E86B2.6090708@rabbitmq.com> The ack is replicated to all mirrors. Cheers, Simon On 04/04/14 10:37, Jean-Philippe No?l wrote: > Hi, > > I am trying to understand what happens when a consumer acks a message > from a mirrored queue. > > Does this ack remain in the local queue or is it replicated on the other > queues ? > > Thank you for your help > > Jean-Philippe > > _______________________________________________ > rabbitmq-discuss mailing list > rabbitmq-discuss at lists.rabbitmq.com > https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss -- Simon MacMullen RabbitMQ, Pivotal From simon at rabbitmq.com Fri Apr 4 11:31:22 2014 From: simon at rabbitmq.com (Simon MacMullen) Date: Fri, 04 Apr 2014 11:31:22 +0100 Subject: [rabbitmq-discuss] shovel auth change in 3.3.0? In-Reply-To: References: Message-ID: <533E89FA.3000000@rabbitmq.com> On 03/04/14 18:30, Laing, Michael wrote: > It appears that either the the source brokers or the destination brokers > must now have a password provided even when using direct connection to > both vhosts, i.e. the shovel is connecting vhosts on the same cluster. > > Formerly no password was required for either. That really should not be happening - and in fact I just tested it and I could still set up a shovel specifying username but no password over a direct connection. Furthermore, in 3.3.0 if you specify no username for a direct connection it should always work rather than requiring the guest user to exist. So I'm curious: how exactly are you doing this? What error do you see? Cheers, Simon -- Simon MacMullen RabbitMQ, Pivotal From michael.laing at nytimes.com Fri Apr 4 12:52:34 2014 From: michael.laing at nytimes.com (Laing, Michael) Date: Fri, 4 Apr 2014 07:52:34 -0400 Subject: [rabbitmq-discuss] shovel auth change in 3.3.0? In-Reply-To: <533E89FA.3000000@rabbitmq.com> References: <533E89FA.3000000@rabbitmq.com> Message-ID: Thanks I'll try to make a test case - I've already added passwords to our dev environment so all works as it did before now. It was an auth error. As I was updating them, I noticed that, although the passwords had been left out, the colon was still there, so it may have been a parsing difference. Of course the colon shouldn't have been there. ml On Fri, Apr 4, 2014 at 6:31 AM, Simon MacMullen wrote: > On 03/04/14 18:30, Laing, Michael wrote: > >> It appears that either the the source brokers or the destination brokers >> must now have a password provided even when using direct connection to >> both vhosts, i.e. the shovel is connecting vhosts on the same cluster. >> >> Formerly no password was required for either. >> > > That really should not be happening - and in fact I just tested it and I > could still set up a shovel specifying username but no password over a > direct connection. > > Furthermore, in 3.3.0 if you specify no username for a direct connection > it should always work rather than requiring the guest user to exist. > > So I'm curious: how exactly are you doing this? What error do you see? > > Cheers, Simon > > -- > Simon MacMullen > RabbitMQ, Pivotal > -------------- next part -------------- An HTML attachment was scrubbed... URL: From albert.lacki at gmail.com Fri Apr 4 08:34:04 2014 From: albert.lacki at gmail.com (=?UTF-8?Q?Albert_=C5=81=C4=85cki?=) Date: Fri, 4 Apr 2014 00:34:04 -0700 (PDT) Subject: [rabbitmq-discuss] Wrong link to 3.3.0 release notes Message-ID: <3dd7c17e-347a-48be-b8f0-99b672564968@googlegroups.com> Please update 3.3.0 release notes link on changelog web page (http://www.rabbitmq.com/changelog.html) "(changes)" in 3.3.0 links to http://www.rabbitmq.com/release-notes/README-3.2.0.txt but should links to http://www.rabbitmq.com/release-notes/README-3.3.0.txt Albert -------------- next part -------------- An HTML attachment was scrubbed... URL: From bieniekmat at gmail.com Thu Apr 3 09:07:30 2014 From: bieniekmat at gmail.com (Mateusz Bieniek) Date: Thu, 3 Apr 2014 01:07:30 -0700 (PDT) Subject: [rabbitmq-discuss] Do not require an acknowledgement for a single message Message-ID: Hi, We use a simple model: Server publishes a message. Client side processes the message and then acknowledges it. If the client is shutdown while processing the message, it will receive it again since the message has not been acknowledged. Now we have a special case. We want to publish a reboot command. In this case, the client reboots itself as part of the processing, and so never acknowledges the reboot message. So when it launches, it consumes the message again and the cycle beings. Updating the client side is highly undesirable, so please assume it's not a possibility. Is there a way we can publish a single message and omit acknowledgement for it? Many thanks, Mateusz -------------- next part -------------- An HTML attachment was scrubbed... URL: From mauro.gandelli at gmail.com Thu Apr 3 16:47:57 2014 From: mauro.gandelli at gmail.com (Mauro Gandelli) Date: Thu, 3 Apr 2014 08:47:57 -0700 (PDT) Subject: [rabbitmq-discuss] When disk usage is below threshold, how do I prevent my consumers from blocking when closing a channel? Message-ID: On my application, after calling IModel.BasicPublish(); , if I call IModel.Close() afterwards the app blocks. In my development machine I noticed that the call unlocks and proceeds when I free disk space above threshold (that's how I figured out that it was the Flow Control mechanism that blocked my connection). In the production environment I need my clients not to block when this happens. According to this article on Flow Control: There are two flow control mechanisms in RabbitMQ. Both work by exerting > TCP backpressure on connections that are publishing too fast. They are: > > - A per-connection mechanism that prevents messages being published > faster than they can be routed to queues. > > > - A global mechanism that prevents any messages from being published > when the memory usage exceeds > a configured threshold or free disk space drops below a configured > threshold. > > Both mechanisms will temporarily *block* connections - the server will > pause reading from the sockets of connected clients which send > content-bearing methods (such as basic.publish) which have been blocked. > Connection heartbeat monitoring will be disabled too. on the bottom of that article, there's a link to this other article, which shows a way to handle blocked connections in both Java and .NET (I'm using .NET API) through events. What is the best way to handle this situation in the client? Do I have to implement those event handlers and prevent any future calls to IModel.BasicPublish() when the connection is blocked? RabbitMQ Version: 3.1.0 Client .NET Version: 3.0.4.0 -------------- next part -------------- An HTML attachment was scrubbed... URL: From mklishin at gopivotal.com Fri Apr 4 13:19:15 2014 From: mklishin at gopivotal.com (Michael Klishin) Date: Fri, 4 Apr 2014 16:19:15 +0400 Subject: [rabbitmq-discuss] Do not require an acknowledgement for a single message In-Reply-To: References: Message-ID: On 4 April 2014 at 16:18:02, Mateusz Bieniek (bieniekmat at gmail.com) wrote: > > Now we have a special case. We want to publish a reboot command. > In this case, the client reboots itself as part of the processing, > and so never acknowledges the reboot message. So when it launches, > it consumes the message again and the cycle beings. > > Updating the client side is highly undesirable, so please assume > it's not a possibility. Is there a way we can publish a single message > and omit acknowledgement for it? No. You need to register another consumer which will use automatic acknowledgements and perform restarts.? -- MK Software Engineer, Pivotal/RabbitMQ From mklishin at gopivotal.com Fri Apr 4 13:19:51 2014 From: mklishin at gopivotal.com (Michael Klishin) Date: Fri, 4 Apr 2014 16:19:51 +0400 Subject: [rabbitmq-discuss] Do not require an acknowledgement for a single message In-Reply-To: References: Message-ID: ?On 4 April 2014 at 16:18:02, Mateusz Bieniek (bieniekmat at gmail.com) wrote: > > Now we have a special case. We want to publish a reboot command. > In this case, the client reboots itself as part of the processing, > and so never acknowledges the reboot message. So when it launches, > it consumes the message again and the cycle beings. > > Updating the client side is highly undesirable, so please assume > it's not a possibility. Is there a way we can publish a single message > and omit acknowledgement for it? What you can do is check ?redelivered? property and if a message is of type ?reboot? but is a redelivery, ignore it. -- MK Software Engineer, Pivotal/RabbitMQ From simon at rabbitmq.com Fri Apr 4 13:21:01 2014 From: simon at rabbitmq.com (Simon MacMullen) Date: Fri, 04 Apr 2014 13:21:01 +0100 Subject: [rabbitmq-discuss] Wrong link to 3.3.0 release notes In-Reply-To: <3dd7c17e-347a-48be-b8f0-99b672564968@googlegroups.com> References: <3dd7c17e-347a-48be-b8f0-99b672564968@googlegroups.com> Message-ID: <533EA3AD.1090207@rabbitmq.com> On 04/04/14 08:34, Albert ??cki wrote: > Please update 3.3.0 release notes link on changelog web page Ah, yes. Odd, Matthias claimed to have fixed that already. Anyway, it's right now. Cheers, Simon -- Simon MacMullen RabbitMQ, Pivotal From mklishin at gopivotal.com Fri Apr 4 13:29:53 2014 From: mklishin at gopivotal.com (Michael Klishin) Date: Fri, 4 Apr 2014 16:29:53 +0400 Subject: [rabbitmq-discuss] When disk usage is below threshold, how do I prevent my consumers from blocking when closing a channel? In-Reply-To: References: Message-ID: ?On 4 April 2014 at 16:20:18, Mauro Gandelli (mauro.gandelli at gmail.com) wrote: > > What is the best way to handle this situation in the client? Do > I have to implement those event handlers and prevent any future > calls to IModel.BasicPublish() when the connection is blocked? Only connections that publish are blocked, so use a separate connection for consumers. -- MK Software Engineer, Pivotal/RabbitMQ From Vinod.Vasudevan at newedge.com Fri Apr 4 15:02:32 2014 From: Vinod.Vasudevan at newedge.com (Vinod Vasudevan) Date: Fri, 4 Apr 2014 22:02:32 +0800 Subject: [rabbitmq-discuss] AUTO: Vinod Vasudevan is out of the office Message-ID: I am out of the office until 06/04/2014. I am out of office . Please escalate to Wendy Zhao/ Anant Deshmukh in case of urgent issues. Send a mail to glb-middlewareservices for regular queries. Note: This is an automated response to your message "rabbitmq-discuss Digest, Vol 83, Issue 4" sent on 4/4/2014 7:00:01 PM. This is the only notification you will receive while this person is away. This electronic communication and its contents are intended for the recipient only and may contain confidential, non-public and/or privileged information. If you are not the intended recipient and/or have received this electronic communication in error, do not read, duplicate or distribute. Please immediately advise the sender and delete this electronic communication and its contents from your system (if permitted by law). No waiver of confidentiality or privilege is made by transmission in error. Newedge makes no representation or warranty that the information contained herein is accurate, complete, fair or correct or that any transaction is appropriate for any person and it should not be relied on as such. All information is subject to change without notice. The information herein is subject to change without notice and is not to be construed as an offer to buy or sell any financial instruments or securities. If this content is generated by or on behalf of Newedge USA, LLC and includes an analysis of the price or market for any derivatives, it should be construed as a solicitation for the purpose of the relevant CFTC Rules. This communication is for information purposes only. Any market or other views expressed herein are those of the sender only as of the date indicated and not necessarily those of Newedge. E-mails may not be secure or error free and information could be lost, destroyed, incomplete, delayed, altered, intercepted, corrupted or fail to be delivered. Newedge makes no representation that this e-mail or any attachments are free of computer virus or other defects or inherent risks and accepts no responsibility for any loss or damage or liability of any kind arising therefrom. Newedge reserves the right to retain all messages. "Newedge" refers to Newedge Group SA and all of its worldwide branches and subsidiaries. Newedge Group SA and its branches are lead regulated by the Autorit??e Contr??le de Prudentiel. Newedge Group SA is also regulated by the Autorit??es March??Fi des Financiers in France. Newedge UK Financial Limited is authorized and regulated by the Financial Conduct Authority (FCA). Newedge Group (Dubai Branch) is regulated by the Dubai Financial Services Authority. Newedge Group (Zurich and Dubai branches) and Newedge UK Financial Limited do not deal with, or for, Retail Clients (as defined under MiFID, FCA rules and Dubai Financial Services Authority). Only Newedge USA, LLC is a member of FINRA and SIPC (SIPC only pertains to securities-related transactions and positions). Only Newedge Canada Inc. is a member of the Canadian Investor Protection Fund. Not all products or services are available from all Newedge organizations or personnel. If necessary, please consult our local office for details. For further information about Newedge and our services, please see our website at www.newedge.com. From simon at rabbitmq.com Fri Apr 4 15:07:24 2014 From: simon at rabbitmq.com (Simon MacMullen) Date: Fri, 04 Apr 2014 15:07:24 +0100 Subject: [rabbitmq-discuss] AUTO: Vinod Vasudevan is out of the office In-Reply-To: References: Message-ID: <533EBC9C.4050206@rabbitmq.com> Apologies to all. Normally these things are supposed to get automatically filtered out. The user responsible has been unsubscribed. Cheers, Simon On 04/04/14 15:02, Vinod Vasudevan wrote: > > I am out of the office until 06/04/2014. > > I am out of office . Please escalate to Wendy Zhao/ Anant Deshmukh in case > of urgent issues. > Send a mail to glb-middlewareservices for regular queries. > > > Note: This is an automated response to your message "rabbitmq-discuss > Digest, Vol 83, Issue 4" sent on 4/4/2014 7:00:01 PM. > > This is the only notification you will receive while this person is away. > > > This electronic communication and its contents are intended for the recipient > only and may contain confidential, non-public and/or privileged information. > If you are not the intended recipient and/or have received this electronic > communication in error, do not read, duplicate or distribute. Please > immediately advise the sender and delete this electronic communication and its > contents from your system (if permitted by law). No waiver of confidentiality > or privilege is made by transmission in error. > Newedge makes no representation or warranty that the information contained > herein is accurate, complete, fair or correct or that any transaction is > appropriate for any person and it should not be relied on as such. All > information is subject to change without notice. The information herein is > subject to change without notice and is not to be construed as an offer to buy > or sell any financial instruments or securities. If this content is generated > by or on behalf of Newedge USA, LLC and includes an analysis of the price or > market for any derivatives, it should be construed as a solicitation for the > purpose of the relevant CFTC Rules. This communication is for information > purposes only. Any market or other views expressed herein are those of the > sender only as of the date indicated and not necessarily those of Newedge. > E-mails may not be secure or error free and information could be lost, > destroyed, incomplete, delayed, altered, intercepted, corrupted or fail to be > delivered. Newedge makes no representation that this e-mail or any attachments > are free of computer virus or other defects or inherent risks and accepts no > responsibility for any loss or damage or liability of any kind arising > therefrom. Newedge reserves the right to retain all messages. > "Newedge" refers to Newedge Group SA and all of its worldwide branches and > subsidiaries. Newedge Group SA and its branches are lead regulated by the > Autorit??e Contr??le de Prudentiel. Newedge Group SA is also regulated by the > Autorit??es March??Fi des Financiers in France. Newedge UK Financial Limited is > authorized and regulated by the Financial Conduct Authority (FCA). Newedge > Group (Dubai Branch) is regulated by the Dubai Financial Services Authority. > Newedge Group (Zurich and Dubai branches) and Newedge UK Financial Limited do > not deal with, or for, Retail Clients (as defined under MiFID, FCA rules and > Dubai Financial Services Authority). Only Newedge USA, LLC is a member of > FINRA and SIPC (SIPC only pertains to securities-related transactions and > positions). Only Newedge Canada Inc. is a member of the Canadian Investor > Protection Fund. Not all products or services are available from all Newedge > organizations or personnel. If necessary, please consult our local office for > details. > > For further information about Newedge and our services, please see our website > at www.newedge.com. > > > > _______________________________________________ > rabbitmq-discuss mailing list > rabbitmq-discuss at lists.rabbitmq.com > https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss > -- Simon MacMullen RabbitMQ, Pivotal From michael.laing at nytimes.com Fri Apr 4 16:37:15 2014 From: michael.laing at nytimes.com (Laing, Michael) Date: Fri, 4 Apr 2014 11:37:15 -0400 Subject: [rabbitmq-discuss] shovel auth change in 3.3.0? In-Reply-To: References: <533E89FA.3000000@rabbitmq.com> Message-ID: Here's an example - one shovel fails and the other works. The difference is the colon which, strictly speaking, perhaps shouldn't have been there - but used to work: [ > {rabbitmq_shovel, [ > {shovels, [ > {'fails', [ > {sources, [ > {broker, "amqp://fabrik.user.test:@/ > test.fabrik.dev.nytimes.com"} > ]}, > {destinations, [ > {brokers, [ > "amqp://fabrik.user.test:@/ > core.fabrik.dev.nytimes.com" > ]} > ]}, > {queue, <<"ha.fabrik">>} > ]}, > {'works', [ > {sources, [ > {broker, "amqp://fabrik.user.test@/ > test.fabrik.dev.nytimes.com"} > ]}, > {destinations, [ > {brokers, [ > "amqp://fabrik.user.test@/ > core.fabrik.dev.nytimes.com" > ]} > ]}, > {queue, <<"ha.fabrik">>} > ]} > ]} % /shovels > ]} % /rabbitmq_shovel > ]. Here's the output from the management console for the failed shovel: {{badmatch,{error,{auth_failure,"Refused"}}}, > [{rabbit_shovel_worker,make_conn_and_chan,1, > > [{file,"rabbitmq-shovel/src/rabbit_shovel_worker.erl"}, > {line,226}]}, > {rabbit_shovel_worker,handle_cast,2, > > [{file,"rabbitmq-shovel/src/rabbit_shovel_worker.erl"}, > {line,60}]}, > {gen_server2,handle_msg,2,[{file,"src/gen_server2.erl"},{line,1022}]}, > {proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,239}]}]} On Fri, Apr 4, 2014 at 7:52 AM, Laing, Michael wrote: > Thanks I'll try to make a test case - I've already added passwords to our > dev environment so all works as it did before now. It was an auth error. > > As I was updating them, I noticed that, although the passwords had been > left out, the colon was still there, so it may have been a parsing > difference. Of course the colon shouldn't have been there. > > ml > > > On Fri, Apr 4, 2014 at 6:31 AM, Simon MacMullen wrote: > >> On 03/04/14 18:30, Laing, Michael wrote: >> >>> It appears that either the the source brokers or the destination brokers >>> must now have a password provided even when using direct connection to >>> both vhosts, i.e. the shovel is connecting vhosts on the same cluster. >>> >>> Formerly no password was required for either. >>> >> >> That really should not be happening - and in fact I just tested it and I >> could still set up a shovel specifying username but no password over a >> direct connection. >> >> Furthermore, in 3.3.0 if you specify no username for a direct connection >> it should always work rather than requiring the guest user to exist. >> >> So I'm curious: how exactly are you doing this? What error do you see? >> >> Cheers, Simon >> >> -- >> Simon MacMullen >> RabbitMQ, Pivotal >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve at rabbitmq.com Fri Apr 4 17:20:29 2014 From: steve at rabbitmq.com (Steve Powell) Date: Fri, 4 Apr 2014 17:20:29 +0100 Subject: [rabbitmq-discuss] Java Client Using DefaultConsumer vs QueueingConsumer In-Reply-To: References: Message-ID: <52515210-1BB8-4739-802E-200DD77422E8@rabbitmq.com> No it won?t. The normal ordering guarantees are observed. The thread dispatching algorithm ensures that consumers on the same channel are called serially. Each call might run on a different thread (or the same thread ? the calls might be batched if many messages arrive at once), but, on a particular channel, there is at most one thread working on a consumer callback (like handleDelivery()) at any time. I thought this was clearly spelt out in the Java Client Doc, but I can only find this: Callbacks to Consumers are dispatched on a thread separate from the thread managed by the Connection. This means that Consumers can safely call blocking methods on the Connection or Channel, such asqueueDeclare, txCommit, basicCancel or basicPublish. Each Channel has its own dispatch thread. For the most common use case of one Consumer per Channel, this means Consumers do not hold up other Consumers. If you have multiple Consumers per Channel be aware that a long-running Consumer may hold up dispatch of callbacks to other Consumers on that Channel. which occurs (buried) in the Channel Thread Safety section. Maybe this should be improved (and a note put in the Consumer Thread Pool section). Steve Powell [Cell: +44-7815-838-558] Good design [?] is as little design as possible. Dieter Rams On 3 Apr 2014, at 16:04, Michael Klishin wrote: >> I have also tested it several times and saw that the order is saved. >> Just to make sure - since different threads call handle delivery >> - will it mess my order? > > By default the pool has 5 threads, so yes. You can provide your own > executor on ConnectionFactory, e.g. to use a single threaded executor. -------------- next part -------------- An HTML attachment was scrubbed... URL: From ryankbrown at gmail.com Fri Apr 4 20:44:46 2014 From: ryankbrown at gmail.com (Ryan Brown) Date: Fri, 4 Apr 2014 13:44:46 -0600 Subject: [rabbitmq-discuss] Problem with publish confirms Message-ID: I have an application that acts as, essentially, a rest wrapper around rabbitmq. (There is obviously some special logic that makes it necessary to build our own). Some background: We have been using this application (current code available at https://github.com/PearsonEducation/subpub) as an internal pub/sub for several years now. It has now grown dramatically in importance to our vision and we are beginning to get inundated with requests for features and enhancements. One of those enhancements is to allow publishers to specify with a header whether they want the application to require confirmed publishing to rabbit. We had begun this process by checking for the header value and passing down the chain. However, we are not currently setting-up a channel that is set for publish confirms. Anyway, TMI... The state I'm in now is that, on startup, we create connections to each of our brokers. In this process we create a channel and save it in state. I have now added logic to create a second channel on that connection that sets: #'confirm.select_ok'{} = amqp_channel:call(ConfirmedChannel, #'confirm.select'{}), And saves it to state. Then on publish, I select which one to use based on the values passed-down the stack from message intake. Set self() as the confirm handler and respond with a wait so that up the stack I can initiate a receive loop in the message intake which is not a gen_server. I have debug messages throughout and it appears that everything is happening as expected. I trace the message through the publish path. To being published and passing the wait back up to the receiver. I also see where the basic.ack is received and the publish_confirm acknowledgement passed all the way up to the receive loop. However, then things go awry. I get the attached dump. My take-away, although likely I am wrong, is that somewhere, apparently in amqp_balanced_publisher, a gen_server call is timing-out while waiting for a response. But, I seem to be able to trace the responses through the whole stack so I am thoroughly confused. I realize this is vague and additional details/code are likely still needed. Please let me know. I've spent way too much time on this already and it's getting quite frustrating. And I know I am likely missing something obvious but am finding supporting documentation for erlang lacking. But I'm probably just looking in the right places. Best regards, Ryan -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: pub_confirm_dump Type: application/octet-stream Size: 41657 bytes Desc: not available URL: From srikanthtns at gmail.com Fri Apr 4 23:27:47 2014 From: srikanthtns at gmail.com (srikanth tns) Date: Fri, 4 Apr 2014 15:27:47 -0700 Subject: [rabbitmq-discuss] Rabbitmq queues dropped and recreated after a while Message-ID: Hi we are noticing this weird behaviour with rabbitmq from last 3 days . we are using rabbitmq as middleware for mcollective. The number of nodes we have connected to rabbitmq were 7000 last week, this week the number of nodes came up to 8000 , so we started seeing the connections dropping out from 8000 to 50 after a period of time and after 2 hours, the connections are back to 8000. Can we know the reason why the connections shutdown all of sudden and how do they reconnect back ? Also we noticed that till we restart one of the nodes mcollective services , the connections on rabbitmq dont come up. Thanks for your help, Srikanth -------------- next part -------------- An HTML attachment was scrubbed... URL: From marco at metm.org Sat Apr 5 02:38:29 2014 From: marco at metm.org (Marco Scoffier) Date: Fri, 04 Apr 2014 21:38:29 -0400 Subject: [rabbitmq-discuss] rabbitmq-c corrupt content frame from ARM client Message-ID: <533F5E95.7080906@metm.org> Hello, I apologize if this has been discussed before. I checked the archives but did not find any mention of the issue. I've compiled rabbitmq-c [1] for the arm architecture [2] from and am trying to send messages to a broker running on another machine [3]. The rabbitmq-c examples run perfectly on the 64bit linux, same code compiled for arm device and running there sends a corrupt packet. [1] https://github.com/alanxz/rabbitmq-c [2] armv6 linaro toolchain [3] linux 64bit rabbitmq-server ppa for ubuntu. running on 12.04 I'm a little stuck about how to debug further and am looking for some pointers where to look in the code or what extra information I can provide. Wireshark shows three packets for the amqp_sendstring example ARM device/Source = 192.168.1.117 64bit broker/Destination = 192.168.1.133 Source Destination protocol length info --------------+---------------+--------+------+------------ 192.168.1.117 192.168.1.133 AMQP 74 Protocol-Header 9-1 192.168.1.133 192.168.1.117 AMQP 522 Connection.Start 192.168.1.117 192.168.1.133 AMQP 190 Content-Header wireshark's amqp frame dissection shows an error on the 3rd Content-Header packet: Expert Info (Error/Protocol): Unknown header class 1290 Message: Unknown header class 1290 Severity level: Error Group: Protocol the rabbitmq-server on the linux machine provides this log message: =ERROR REPORT==== 3-Apr-2014::15:05:35 === closing AMQP connection <0.380.0> (192.168.1.117:52890 -> 192.168.1.133:5672): {handshake_error,starting,0, {amqp_error,frame_error, "type 0, first 16 octets = <<5,10,0,11,0,0,0,80,98,112,114,111,100,117,99,116>>...", none}} repeat that the same code compiled for 64bit linux same command works as expected. Thanks for any pointers, Marco From alan.antonuk at gmail.com Sat Apr 5 16:28:08 2014 From: alan.antonuk at gmail.com (Alan Antonuk) Date: Sat, 5 Apr 2014 08:28:08 -0700 Subject: [rabbitmq-discuss] rabbitmq-c and SimpleAmqpClient SSL authentication In-Reply-To: <4B694A5D-3D69-4699-B940-C79ED5E5D57A@dushin.net> References: <4B694A5D-3D69-4699-B940-C79ED5E5D57A@dushin.net> Message-ID: Yes, that should be done at some point. Feel free to open a PR on SimpleAmqpClient when you've got something. -Alan On Thu, Apr 3, 2014 at 5:19 PM, Dushin Fred wrote: > And I suppose we would want to plumb this through as a parameter to the > SimpleAmqpClient API, as well, no? I can try that. > > -Fred > > On Mar 31, 2014, at 1:31 AM, Alan Antonuk wrote: > > I haven't tried it out yet, but here's a PR adding the EXTERNAL SASL > mechanism to rabbitmq-c. > > https://github.com/alanxz/rabbitmq-c/pull/179 > > Feel free to try it out and let me know how it works. > > -Alan > > > On Sun, Mar 30, 2014 at 5:46 PM, Alan Antonuk wrote: > >> Fred; >> >> >> On Sun, Mar 30, 2014 at 1:29 PM, Dushin Fred wrote: >> >>> Hi Folks, >>> >>> I have run into two issues with the (admittedly unsupported) rabbitmq-c >>> and SimpleAmqpClient libraries. (I realize I am hijacking the RabbtiMQ >>> mailing list for this topic, but I am guessing the Pivotal folks don't >>> mind, because this should result in an improvement of the overall RabbitMQ >>> ecosystem.) >>> >> >> Currently this mailing list is the best place to have discussions >> concerning these two libraries. >> >>> >>> The first issue is with the SimpleAmqpClient library. It appears that >>> there is no knob in the Channel::CreateSecureChannel operation to disable >>> hostname verification of the RabbitMQ server. There is a knob in the >>> rabbitmq-c API (amqp_ssl_socket_set_verify), but you need a socket to call >>> it, and the only reasonable place to do that is in the Channel constructor. >>> I am including a patch off the github 2.3 tag for doing just that, though >>> for some reason I did not track down, I could not get boost::make_shared >>> take my new constructor signature, so I just used the shared_ptr >>> constructor in the raw. That may not fit the current model, but it seems >>> harmless for testing. >>> >> >> This is probably what you want: >> https://github.com/alanxz/SimpleAmqpClient/pull/85. In the future feel >> free to open a pull-request against the library on github. The error you >> were seeing with boost::make_shared most likely had to do its limit of 10 >> arguments. >> >>> >>> >>> If I look at the rabbitmq-c code, I see that the C client library and >>> API only seems to support the PLAIN SASL method. >> >> >> That is correct. I'm open to adding support for additional SASL >> mechanisms to rabbitmq-c. If the implementation of the SASL mechanism is >> anything more than trivial (e.g., PLAIN really is dead-simple), rabbitmq-c >> should probably link in a thoroughly tested external library (like >> cyrus-SASL). Given use of these alternate SASL mechanisms appears to be >> very low, any external library dependancies should remain optional. I can >> provide some other hints to get started adding this to rabbitmq-c if you so >> desire. >> >> >>> Any pointers to docs on the EXTERNAL mechanism (at least WRT SSL client >>> auth) or existing code would be greatly appreciated. >>> >> >> The RFC for SASL includes a pretty good description of how the SASL >> EXTERNAL method should operate: >> http://tools.ietf.org/html/rfc4422#appendix-A (it actually looks pretty >> simple to implement). >> >> >> -Alan >> >> > _______________________________________________ > rabbitmq-discuss mailing list > rabbitmq-discuss at lists.rabbitmq.com > https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss > > > > _______________________________________________ > rabbitmq-discuss mailing list > rabbitmq-discuss at lists.rabbitmq.com > https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jonathan.bouzekri at gmail.com Sat Apr 5 16:40:00 2014 From: jonathan.bouzekri at gmail.com (Jonathan Bouzekri) Date: Sat, 5 Apr 2014 08:40:00 -0700 (PDT) Subject: [rabbitmq-discuss] Apache/PHP_mod and keeping connection alive Message-ID: Hi, I bought the Rabbit In Action book. The authors said that you should not open a connection to Rabbit every single time you want to send a message. It seems logic when you know the channel mecanism however in a HTTP context with a server web, I don't see how you can do it. Especially in an Apache/PHP mod context. You can't share a connection between requests in PHP because each request has its own process and scope and don't share anything (or I don't know of any method to do it). So suppose you must send a message through Rabbit each time a user make a GET request to your server, you will open a connection, create an exchange (maybe a queue too), send a message, close the connection. It will be time consumming and maybe overload rabbit. Is there another solution ? I am thinking about queueing my message in a simple SQL table (because I already open the connection to the server at each request) and process each line with a batch which will send the message to Rabbit but It seems like building another queue mecanism from scratch .... Thanks in advance for your suggestion. Best regards -------------- next part -------------- An HTML attachment was scrubbed... URL: From alan.antonuk at gmail.com Sat Apr 5 16:44:21 2014 From: alan.antonuk at gmail.com (Alan Antonuk) Date: Sat, 5 Apr 2014 08:44:21 -0700 Subject: [rabbitmq-discuss] rabbitmq-c memory leak? In-Reply-To: References: Message-ID: SimpleAmqpClient should call MaybeReleaseBuffersOnChannel, which if it has no frames enqueued, should call amqp_maybe_release_buffers_on_channel(), which if rabbitmq-c doesn't have any frames lying around will recycle the pool associated with that channel. If this really is a bug (possible...) it would be good to know which channel(s) SimpleAmqpClient is publishing on, and which channel(s) are growing without bound (e.g., which entry in amqp_connection_state_t.pool_table). -Alan On Thu, Apr 3, 2014 at 5:19 PM, Dushin Fred wrote: > > I have written a simple test program using the SimpleAqmpClient library > (which in turn uses the rabbitmq-c library), which seems to illustrate a > memory leak, of sorts. > > The program is a simple main, which pre-allocates a string message, and > then calls BasicPublish on a channel in a loop, all single threaded. > Effectively: > > auto envelope = BasicMessage::Create(random_data(message_size)); > for (unsigned i = 0; i < num_messages; ++i) > { > channel->BasicPublish(exchange_name, routing_key, envelope); > } > > The problem is that the memory image of this process seems to grow without > bounds. The memory all seems to be allocated in amqp_pool_alloc (as > expected), when calling wait_frame_inner -> amqp_handle_input. The memory > is not orphaned -- i.e., it seems to get cleaned up at shutdown, so it > doesn't show up as a leak in a leak detector tool, such a > Instruments/DTrace, or valgrind, or purify, etc. But I don't see where the > memory is every de-allocated, which will spell doom for an application that > is designed to never terminate. > > I am attaching a screenshot from an Instruments run, to give some context. > The code is here: > > https://github.com/fadushin/sandbox/blob/master/cpp/rabbitmq/sender.cpp > > This was run against RabbitMQ 3.2.4, using rabbitmq-c 0.5.0 and > SimpleAmqpClient 2.3, compiled using Clang 5.0 on OS X.9 (Mavericks). > > Has anyone seen this behavior? Am I missing something in my use of > BasicPublish, that is not telling the rabbitmq-c library to free memory in > the "amqp pool"? It doesn't seem to make a difference whether I am > actively pulling messages off the queue from a separate process, or even > whether there is a queue bound to the exchange to which I am publishing > messages. (Not that I would expect it to) > > -Fred > > encl. > > > > _______________________________________________ > rabbitmq-discuss mailing list > rabbitmq-discuss at lists.rabbitmq.com > https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss > > -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: PastedGraphic-3.png Type: image/png Size: 422159 bytes Desc: not available URL: From videlalvaro at gmail.com Sat Apr 5 16:51:41 2014 From: videlalvaro at gmail.com (Alvaro Videla) Date: Sat, 5 Apr 2014 17:51:41 +0200 Subject: [rabbitmq-discuss] Apache/PHP_mod and keeping connection alive In-Reply-To: References: Message-ID: Hi, Note that you don't need to declare the queue/exchange every time. You can have that setup in advance and use your connection to just send the message. About RabbitMQ being overloaded by this, unless you are on the several thousands, I doubt it would be a problem. As always, it's better to benchmark this yourself. Regards, Alvaro On Sat, Apr 5, 2014 at 5:40 PM, Jonathan Bouzekri wrote: > Hi, > > I bought the Rabbit In Action book. The authors said that you should not > open a connection to Rabbit every single time you want to send a message. It > seems logic when you know the channel mecanism however in a HTTP context > with a server web, I don't see how you can do it. Especially in an > Apache/PHP mod context. You can't share a connection between requests in PHP > because each request has its own process and scope and don't share anything > (or I don't know of any method to do it). > > So suppose you must send a message through Rabbit each time a user make a > GET request to your server, you will open a connection, create an exchange > (maybe a queue too), send a message, close the connection. It will be time > consumming and maybe overload rabbit. Is there another solution ? I am > thinking about queueing my message in a simple SQL table (because I already > open the connection to the server at each request) and process each line > with a batch which will send the message to Rabbit but It seems like > building another queue mecanism from scratch .... > > Thanks in advance for your suggestion. > > Best regards > > _______________________________________________ > rabbitmq-discuss mailing list > rabbitmq-discuss at lists.rabbitmq.com > https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss > From alan.antonuk at gmail.com Sun Apr 6 05:48:41 2014 From: alan.antonuk at gmail.com (Alan Antonuk) Date: Sat, 5 Apr 2014 21:48:41 -0700 Subject: [rabbitmq-discuss] rabbitmq-c corrupt content frame from ARM client In-Reply-To: <533F5E95.7080906@metm.org> References: <533F5E95.7080906@metm.org> Message-ID: Marco; >From the wireshark info, it sounds like the rabbitmq-c client is sending completely the wrong data. It should send a Connection.Start-Ok method and not a content-header frame (e.g., the frame type should be 1 first few bytes of that frame should be: 0x01 {Method Frame}, 0x00, 0x00 {Channel 0}, 0x00, 0x00 0xXX, 0xXX {Big Endian length of frame in bytes} ). I'm also not sure what to to think about the sent header length: that should be 8 bytes or 64 bits, I suspect thats a wireshark oddity though. The octets reported by the RabbitMQ server error message don't really make sense to me: I'm having trouble correlating what is reported vs what I think should be, even if the system is doing an incorrect byteswap. I'm not 100% sure we're clear of any unaligned access issues, but I would guess we'd be hitting bus-errors in that case, though I'm not quite sure Would you be willing to send me a filtered down wireshark log of the failed handshake? -Alan On Fri, Apr 4, 2014 at 6:38 PM, Marco Scoffier wrote: > Hello, > > I apologize if this has been discussed before. I checked the archives but > did not find any mention of the issue. > > I've compiled rabbitmq-c [1] for the arm architecture [2] from and am > trying to send messages to a broker running on another machine [3]. The > rabbitmq-c examples run perfectly on the 64bit linux, same code compiled > for arm device and running there sends a corrupt packet. > > [1] https://github.com/alanxz/rabbitmq-c > [2] armv6 linaro toolchain > [3] linux 64bit rabbitmq-server ppa for ubuntu. running on 12.04 > > I'm a little stuck about how to debug further and am looking for some > pointers where to look in the code or what extra information I can provide. > > Wireshark shows three packets for the amqp_sendstring example > > ARM device/Source = 192.168.1.117 > 64bit broker/Destination = 192.168.1.133 > > Source Destination protocol length info > --------------+---------------+--------+------+------------ > 192.168.1.117 192.168.1.133 AMQP 74 Protocol-Header 9-1 > 192.168.1.133 192.168.1.117 AMQP 522 Connection.Start > 192.168.1.117 192.168.1.133 AMQP 190 Content-Header > > wireshark's amqp frame dissection shows an error on the 3rd Content-Header > packet: > > Expert Info (Error/Protocol): Unknown header class 1290 > Message: Unknown header class 1290 > Severity level: Error > Group: Protocol > > the rabbitmq-server on the linux machine provides this log message: > > =ERROR REPORT==== 3-Apr-2014::15:05:35 === > closing AMQP connection <0.380.0> (192.168.1.117:52890 -> > 192.168.1.133:5672): > {handshake_error,starting,0, > {amqp_error,frame_error, > "type 0, first 16 octets = > <<5,10,0,11,0,0,0,80,98,112,114,111,100,117,99,116>>...", > none}} > > repeat that the same code compiled for 64bit linux same command works as > expected. > > Thanks for any pointers, > > Marco > _______________________________________________ > rabbitmq-discuss mailing list > rabbitmq-discuss at lists.rabbitmq.com > https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss > -------------- next part -------------- An HTML attachment was scrubbed... URL: From matthias at rabbitmq.com Sun Apr 6 07:18:10 2014 From: matthias at rabbitmq.com (Matthias Radestock) Date: Sun, 06 Apr 2014 07:18:10 +0100 Subject: [rabbitmq-discuss] =?utf-8?b?5Zue5aSN77yaICBIb3cgbG9uZyBpbnRlcnZh?= =?utf-8?q?l_fetching_message_from_MQSERVER=3F?= In-Reply-To: References: Message-ID: <5340F1A2.1010104@rabbitmq.com> On 01/04/14 06:16, thomas wrote: > pls check my code below: > > String message = client.getNextMessage(); > if (message != null && !message.equals("")) { > client.sendMessage(imqRec.handler(message)); > } > > > If I use while(true) loop continuously monitor RPC queue, it would be > very resource intensive, whether you grab a message per sec or anything > else? You haven't told us what client library you are using. If client.getNextMessage() is a non-blocking operation that, say, returns null when there are no messages in the queue then yes, the above will be very resource intensive if invoked in a tight loop. Less so, obviously, if you insert a 1 sec delay. Such polling is a really bad idea. Please read the RabbitMQ tutorials and familiarise yourself with the basic.consume AMQP operation, which does not require polling. Matthias. From stammailbox at gmail.com Sun Apr 6 07:32:29 2014 From: stammailbox at gmail.com (rails) Date: Sun, 6 Apr 2014 09:32:29 +0300 Subject: [rabbitmq-discuss] Java Client Using DefaultConsumer vs QueueingConsumer In-Reply-To: References: Message-ID: Thanks. On this one. Search for deprecated in the page On Thu, Apr 3, 2014 at 6:04 PM, Michael Klishin wrote: > On 3 April 2014 at 18:09:06, rails (stammailbox at gmail.com) wrote: > > > 1. My DemoConsumer inherits from DefaultConsumer > > I have noticed that working this way handleDelivery() is invoked > > from ThreadPool. > > (printing Thread.currentThread().getName() I see pool-1-thread-1/2/3/4 > > eachtime. > > I have also tested it several times and saw that the order is saved. > > Just to make sure - since different threads call handle delivery > > - will it mess my order? > > By default the pool has 5 threads, so yes. You can provide your own > executor on ConnectionFactory, e.g. to use a single threaded executor. > > > 2. QueueingConsumer All of the java tutorial use QueueingConsumer > > to consume messages. > > In the API Docs it is mentioned as a deprecated class. > > Should I change my code to inherit from DefaultConsumer use it? > > Is the tutorial outdated? > > I believe it was undeprecated after some of its limitations > were documented . On what pages do you see this mentioned? > -- > MK > > Software Engineer, Pivotal/RabbitMQ > _______________________________________________ > rabbitmq-discuss mailing list > rabbitmq-discuss at lists.rabbitmq.com > https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss > -------------- next part -------------- An HTML attachment was scrubbed... URL: From marco at metm.org Sun Apr 6 08:16:09 2014 From: marco at metm.org (Marco Scoffier) Date: Sun, 06 Apr 2014 03:16:09 -0400 Subject: [rabbitmq-discuss] rabbitmq-c corrupt content frame from ARM client In-Reply-To: References: <533F5E95.7080906@metm.org> Message-ID: <5340FF39.3050607@metm.org> Sure, I dumped the 3 packets to a separate file. This is the binary data which you can open in wireshark: http://metm.org/images/amqp_handshake_wireshark.log Thanks for looking into this and happy to provide any additional tests, Marco On 04/06/2014 12:48 AM, Alan Antonuk wrote: > Marco; > > From the wireshark info, it sounds like the rabbitmq-c client is > sending completely the wrong data. It should send a Connection.Start-Ok > method and not a content-header frame (e.g., the frame type should be 1 > first few bytes of that frame should be: 0x01 {Method Frame}, 0x00, 0x00 > {Channel 0}, 0x00, 0x00 0xXX, 0xXX {Big Endian length of frame in bytes} > ). I'm also not sure what to to think about the sent header length: that > should be 8 bytes or 64 bits, I suspect thats a wireshark oddity though. > > The octets reported by the RabbitMQ server error message don't really > make sense to me: I'm having trouble correlating what is reported vs > what I think should be, even if the system is doing an incorrect > byteswap. I'm not 100% sure we're clear of any unaligned access issues, > but I would guess we'd be hitting bus-errors in that case, though I'm > not quite sure > > Would you be willing to send me a filtered down wireshark log of the > failed handshake? > > -Alan > > > On Fri, Apr 4, 2014 at 6:38 PM, Marco Scoffier > wrote: > > Hello, > > I apologize if this has been discussed before. I checked the > archives but did not find any mention of the issue. > > I've compiled rabbitmq-c [1] for the arm architecture [2] from and > am trying to send messages to a broker running on another machine > [3]. The rabbitmq-c examples run perfectly on the 64bit linux, same > code compiled for arm device and running there sends a corrupt packet. > > [1] https://github.com/alanxz/__rabbitmq-c > > [2] armv6 linaro toolchain > [3] linux 64bit rabbitmq-server ppa for ubuntu. running on 12.04 > > I'm a little stuck about how to debug further and am looking for > some pointers where to look in the code or what extra information I > can provide. > > Wireshark shows three packets for the amqp_sendstring example > > ARM device/Source = 192.168.1.117 > 64bit broker/Destination = 192.168.1.133 > > Source Destination protocol length info > --------------+---------------__+--------+------+------------ > 192.168.1.117 192.168.1.133 AMQP 74 Protocol-Header 9-1 > 192.168.1.133 192.168.1.117 AMQP 522 Connection.Start > 192.168.1.117 192.168.1.133 AMQP 190 Content-Header > > wireshark's amqp frame dissection shows an error on the 3rd > Content-Header packet: > > Expert Info (Error/Protocol): Unknown header class 1290 > Message: Unknown header class 1290 > Severity level: Error > Group: Protocol > > the rabbitmq-server on the linux machine provides this log message: > > =ERROR REPORT==== 3-Apr-2014::15:05:35 === > closing AMQP connection <0.380.0> (192.168.1.117:52890 > -> 192.168.1.133:5672 > ): > {handshake_error,starting,0, > {amqp_error,frame_error, > "type 0, first 16 octets = > <<5,10,0,11,0,0,0,80,98,112,__114,111,100,117,99,116>>...", > none}} > > repeat that the same code compiled for 64bit linux same command > works as expected. > > Thanks for any pointers, > > Marco > _________________________________________________ > rabbitmq-discuss mailing list > rabbitmq-discuss at lists.__rabbitmq.com > > https://lists.rabbitmq.com/__cgi-bin/mailman/listinfo/__rabbitmq-discuss > > > > > > _______________________________________________ > rabbitmq-discuss mailing list > rabbitmq-discuss at lists.rabbitmq.com > https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss > From marco at metm.org Sun Apr 6 10:25:55 2014 From: marco at metm.org (Marco Scoffier) Date: Sun, 06 Apr 2014 05:25:55 -0400 Subject: [rabbitmq-discuss] rabbitmq-c corrupt content frame from ARM client In-Reply-To: <5340FF39.3050607@metm.org> References: <533F5E95.7080906@metm.org> <5340FF39.3050607@metm.org> Message-ID: <53411DA3.9080208@metm.org> Hi Alan, I just re-uploaded the file to include all the packets around the handshake (not just the 3 I get when filtering for AMQP) as you mentioned wireshark itself might also have parsing issues. Marco On 04/06/2014 03:16 AM, Marco Scoffier wrote: > Sure, > > I dumped the 3 packets to a separate file. This is the binary data which > you can open in wireshark: > > http://metm.org/images/amqp_handshake_wireshark.log > > Thanks for looking into this and happy to provide any additional tests, > > Marco > > On 04/06/2014 12:48 AM, Alan Antonuk wrote: >> Marco; >> >> From the wireshark info, it sounds like the rabbitmq-c client is >> sending completely the wrong data. It should send a Connection.Start-Ok >> method and not a content-header frame (e.g., the frame type should be 1 >> first few bytes of that frame should be: 0x01 {Method Frame}, 0x00, 0x00 >> {Channel 0}, 0x00, 0x00 0xXX, 0xXX {Big Endian length of frame in bytes} >> ). I'm also not sure what to to think about the sent header length: that >> should be 8 bytes or 64 bits, I suspect thats a wireshark oddity though. >> >> The octets reported by the RabbitMQ server error message don't really >> make sense to me: I'm having trouble correlating what is reported vs >> what I think should be, even if the system is doing an incorrect >> byteswap. I'm not 100% sure we're clear of any unaligned access issues, >> but I would guess we'd be hitting bus-errors in that case, though I'm >> not quite sure >> >> Would you be willing to send me a filtered down wireshark log of the >> failed handshake? >> >> -Alan >> >> >> On Fri, Apr 4, 2014 at 6:38 PM, Marco Scoffier > > wrote: >> >> Hello, >> >> I apologize if this has been discussed before. I checked the >> archives but did not find any mention of the issue. >> >> I've compiled rabbitmq-c [1] for the arm architecture [2] from and >> am trying to send messages to a broker running on another machine >> [3]. The rabbitmq-c examples run perfectly on the 64bit linux, same >> code compiled for arm device and running there sends a corrupt >> packet. >> >> [1] https://github.com/alanxz/__rabbitmq-c >> >> [2] armv6 linaro toolchain >> [3] linux 64bit rabbitmq-server ppa for ubuntu. running on 12.04 >> >> I'm a little stuck about how to debug further and am looking for >> some pointers where to look in the code or what extra information I >> can provide. >> >> Wireshark shows three packets for the amqp_sendstring example >> >> ARM device/Source = 192.168.1.117 >> 64bit broker/Destination = 192.168.1.133 >> >> Source Destination protocol length info >> --------------+---------------__+--------+------+------------ >> 192.168.1.117 192.168.1.133 AMQP 74 Protocol-Header 9-1 >> 192.168.1.133 192.168.1.117 AMQP 522 Connection.Start >> 192.168.1.117 192.168.1.133 AMQP 190 Content-Header >> >> wireshark's amqp frame dissection shows an error on the 3rd >> Content-Header packet: >> >> Expert Info (Error/Protocol): Unknown header class 1290 >> Message: Unknown header class 1290 >> Severity level: Error >> Group: Protocol >> >> the rabbitmq-server on the linux machine provides this log message: >> >> =ERROR REPORT==== 3-Apr-2014::15:05:35 === >> closing AMQP connection <0.380.0> (192.168.1.117:52890 >> -> 192.168.1.133:5672 >> ): >> {handshake_error,starting,0, >> {amqp_error,frame_error, >> "type 0, first 16 octets = >> <<5,10,0,11,0,0,0,80,98,112,__114,111,100,117,99,116>>...", >> none}} >> >> repeat that the same code compiled for 64bit linux same command >> works as expected. >> >> Thanks for any pointers, >> >> Marco >> _________________________________________________ >> rabbitmq-discuss mailing list >> rabbitmq-discuss at lists.__rabbitmq.com >> >> >> https://lists.rabbitmq.com/__cgi-bin/mailman/listinfo/__rabbitmq-discuss >> >> >> >> >> >> >> _______________________________________________ >> rabbitmq-discuss mailing list >> rabbitmq-discuss at lists.rabbitmq.com >> https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss >> > > _______________________________________________ > rabbitmq-discuss mailing list > rabbitmq-discuss at lists.rabbitmq.com > https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss From dmitry.andrianov at alertme.com Sun Apr 6 12:18:12 2014 From: dmitry.andrianov at alertme.com (Dmitry Andrianov) Date: Sun, 06 Apr 2014 12:18:12 +0100 Subject: [rabbitmq-discuss] 3.3.0 Message-ID: <534137F4.3090206@alertme.com> Hello. It looks like one of our RabbitMQ plugins stopped working after we upgraded to 3.0.0. Of course it can be that we are doing something wrong, I do not know any details yet, just wanted to check if there is a known change I should be aware of. The plugin that stopped working was hooking into the events to do its stuff when new exchange is created. -export([init/1, handle_call/2, handle_event/2, handle_info/2, terminate/2, code_change/3]). ... handle_event(Event = #event{type = exchange_created, props = [{name, ExchangeName = #resource{}} | _]}, State) -> rabbit_log:info("handle_event (Event=~p, State=~p)~n", [Event, State]), ... the INFO which used to appear in the log for 3.1.1 to 3.2.3 does not appear anymore. I downgraded for now until I have time to debug it properly. Thanks This email is for the use of the intended recipient(s) only. If you have received this email in error, please notify the sender immediately and then delete it. If you are not the intended recipient, you must not use, disclose or distribute this email without the author's prior permission. AlertMe.com Ltd. is not responsible for any personal views expressed in this message or any attachments that are those of the individual sender. AlertMe.com Ltd, 30 Station Road, Cambridge, CB1 2RE, UK. Registered in England, Company number 578 2908, VAT registration number GB 895 9914 42. -------------- next part -------------- An HTML attachment was scrubbed... URL: From mklishin at gopivotal.com Sun Apr 6 14:33:05 2014 From: mklishin at gopivotal.com (Michael Klishin) Date: Sun, 6 Apr 2014 17:33:05 +0400 Subject: [rabbitmq-discuss] Rabbitmq queues dropped and recreated after a while In-Reply-To: References: Message-ID: On 5 April 2014 at 02:30:04, srikanth tns (srikanthtns at gmail.com) wrote: > > The number of nodes we have connected to rabbitmq were 7000 last > week, this week the number of nodes came up to 8000 , so we started > seeing the connections dropping out from 8000 to 50 after a period > of time and after 2 hours, the connections are back to 8000. Can > we know the reason why the connections shutdown all of sudden > and how do they reconnect back ? Also we noticed that till we restart > one of the nodes mcollective services , the connections on rabbitmq > dont come up. What?s in RabbitMQ log files?? -- MK Software Engineer, Pivotal/RabbitMQ From vish.ramachandran at gmail.com Sun Apr 6 21:48:55 2014 From: vish.ramachandran at gmail.com (Viswanathan Ramachandran) Date: Sun, 6 Apr 2014 13:48:55 -0700 Subject: [rabbitmq-discuss] Dynamic Shovel and HA with 3.3 Message-ID: Hi, I have a bunch of questions on HA + shoveling. In our setup, we plan to shovel from one cluster with HA queues to another cluster with HA queues as well. 1. Is this possible with dynamic shovels? I read that shovel queues would be created with no args, so ha-policy may not be possible. 2. If the source queues (in the local exchange) mentioned in the dynamic shovels are pre-created with x-ha-policy = all argument is it a valid configuration? 3. Or is it necessary to set a HA policy for the pre-created queue from which messages would be shoveled? 4. If master fails in an HA queue consumed by shovel in slave, will shovel take care of re-consumption from new queue? With the 3.3 enhancement of clients not needing to reconsume, I guess this should work seamlessly with shovels as well? 5. Dynamic shovels don't allow detailed configuration. Are the queues declared configured for no loss in messages? Namely durable, acknowledge on explicit confirm etc? Thanks Vish -------------- next part -------------- An HTML attachment was scrubbed... URL: From aimirim.software at gmail.com Sun Apr 6 22:39:42 2014 From: aimirim.software at gmail.com (=?UTF-8?Q?S=C3=A9rgio_Ferreira?=) Date: Sun, 6 Apr 2014 14:39:42 -0700 (PDT) Subject: [rabbitmq-discuss] rabbitmq-server start "init terminating in do_boot" Message-ID: <8c8cb061-99e1-4517-81e7-6434477d731d@googlegroups.com> Hi, I'm trying to install and run the service on my laptop with Windows 8.1. I'm getting this error over and over again. I tried with the version 3.3.0 and earlier version to. after searching for hours for a solution in blog posts and this forum I was unable to find a solution. I'm currently using this service in a VPS on a ubuntu machine working fine. can someone help me out? Error message: {"init terminating in do_boot",{{badmatch,undefined},[{rabbit,maybe_hipe_compile ,0,[]},{init,eval_script,8,[{file,"init.erl"},{line,842}]},{init,do_boot,3,[{fil e,"init.erl"},{line,748}]}]}} init terminating in do_boot () -------------- next part -------------- An HTML attachment was scrubbed... URL: From srikanthtns at gmail.com Mon Apr 7 00:01:39 2014 From: srikanthtns at gmail.com (srikanth tns) Date: Sun, 6 Apr 2014 16:01:39 -0700 Subject: [rabbitmq-discuss] Rabbitmq queues dropped and recreated after a while In-Reply-To: References: Message-ID: This is what we see in the logs.. ending STOMP connection <0.13770.0> from 10.233.243.155:55298 =INFO REPORT==== 6-Apr-2014::22:45:02 === ending STOMP connection <0.6415.0> from 10.233.243.156:33153 =INFO REPORT==== 6-Apr-2014::22:45:02 === ending STOMP connection <0.10014.0> from 10.233.243.157:36118 =INFO REPORT==== 6-Apr-2014::22:45:02 === ending STOMP connection <0.2265.0> from 10.233.243.158:45859 =INFO REPORT==== 6-Apr-2014::22:45:03 === ending STOMP connection <0.11893.0> from 10.233.243.159:44710 =INFO REPORT==== 6-Apr-2014::22:45:03 === ending STOMP connection <0.6531.0> from 10.233.243.169:60762 On Sun, Apr 6, 2014 at 6:33 AM, Michael Klishin wrote: > On 5 April 2014 at 02:30:04, srikanth tns (srikanthtns at gmail.com) wrote: > > > The number of nodes we have connected to rabbitmq were 7000 last > > week, this week the number of nodes came up to 8000 , so we started > > seeing the connections dropping out from 8000 to 50 after a period > > of time and after 2 hours, the connections are back to 8000. Can > > we know the reason why the connections shutdown all of sudden > > and how do they reconnect back ? Also we noticed that till we restart > > one of the nodes mcollective services , the connections on rabbitmq > > dont come up. > > What's in RabbitMQ log files? > -- > MK > > Software Engineer, Pivotal/RabbitMQ > _______________________________________________ > rabbitmq-discuss mailing list > rabbitmq-discuss at lists.rabbitmq.com > https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ravindrapandey1982 at gmail.com Mon Apr 7 07:07:24 2014 From: ravindrapandey1982 at gmail.com (Ravi Pandey) Date: Mon, 7 Apr 2014 11:37:24 +0530 Subject: [rabbitmq-discuss] How to make RabbitMQ worker listening continuously Message-ID: Hi I am implementing rabbitMQ with more than 3 workers processing. To test it i need to execute worker file each time but i don't want that. I want my worker script to listen all request continuously without manually executing worker file. Many people suggested CRON but I don't want because if previous run hasn't finished then overlap can cause serious issues. Is there any way to run my worker script continuously in background? -------------- next part -------------- An HTML attachment was scrubbed... URL: From michael.s.klishin at gmail.com Mon Apr 7 07:35:49 2014 From: michael.s.klishin at gmail.com (Michael Klishin) Date: Mon, 7 Apr 2014 10:35:49 +0400 Subject: [rabbitmq-discuss] Rabbitmq queues dropped and recreated after a while In-Reply-To: References: Message-ID: 2014-04-07 3:01 GMT+04:00 srikanth tns : > =INFO REPORT==== 6-Apr-2014::22:45:02 === > ending STOMP connection <0.6415.0> from 10.233.243.156:33153 > It means that a client has disconnected or, possibly, an unrecoverable error was received (but then there should be other messages in the log). Can there be clients that disconnect at will or intermediaries (e.g. firewalls) that close TCP connections, say due to inactivity? Note: this particular error message was removed from RabbitMQ STOMP plugin around Jan 2012. Consider upgrading. -- MK http://github.com/michaelklishin http://twitter.com/michaelklishin -------------- next part -------------- An HTML attachment was scrubbed... URL: From michael.s.klishin at gmail.com Mon Apr 7 07:36:43 2014 From: michael.s.klishin at gmail.com (Michael Klishin) Date: Mon, 7 Apr 2014 10:36:43 +0400 Subject: [rabbitmq-discuss] How to make RabbitMQ worker listening continuously In-Reply-To: References: Message-ID: 2014-04-07 10:07 GMT+04:00 Ravi Pandey : > I want my worker script to listen all request continuously without > manually executing worker file. Many people suggested CRON but I don't want > because if previous run hasn't finished then overlap can cause serious > issues. > > Is there any way to run my worker script continuously in background? > It depends on the language and client used. What do you use? Do you consume using basic.consume or basic.get? -- MK http://github.com/michaelklishin http://twitter.com/michaelklishin -------------- next part -------------- An HTML attachment was scrubbed... URL: From michael.s.klishin at gmail.com Mon Apr 7 07:38:36 2014 From: michael.s.klishin at gmail.com (Michael Klishin) Date: Mon, 7 Apr 2014 10:38:36 +0400 Subject: [rabbitmq-discuss] rabbitmq-server start "init terminating in do_boot" In-Reply-To: <8c8cb061-99e1-4517-81e7-6434477d731d@googlegroups.com> References: <8c8cb061-99e1-4517-81e7-6434477d731d@googlegroups.com> Message-ID: 2014-04-07 1:39 GMT+04:00 S?rgio Ferreira : > {{badmatch,undefined},[{rabbit,maybe_hipe_compile > ,0,[] > What Erlang version do you run? Are you sure it's the same on both Windows and Linux machines? -- MK http://github.com/michaelklishin http://twitter.com/michaelklishin -------------- next part -------------- An HTML attachment was scrubbed... URL: From mklishin at gopivotal.com Mon Apr 7 07:54:21 2014 From: mklishin at gopivotal.com (Michael Klishin) Date: Mon, 7 Apr 2014 10:54:21 +0400 Subject: [rabbitmq-discuss] Dynamic Shovel and HA with 3.3 In-Reply-To: References: Message-ID: On 7 April 2014 at 00:51:13, Viswanathan Ramachandran (vish.ramachandran at gmail.com) wrote: > > 1. Is this possible with dynamic shovels? I read that shovel > queues would be created with no args, so ha-policy may not be possible. > > 2. If the source queues (in the local exchange) mentioned in the > dynamic shovels are pre-created with x-ha-policy = all argument > is it a valid configuration? From : ?If the source queue does not exist on the source broker, it will be declared as a durable queue with no arguments.? So you can declare a queue first or use policies to make it mirrored, both ways should work and won?t interfere with shovels.? -- MK Software Engineer, Pivotal/RabbitMQ From aimirim.software at gmail.com Mon Apr 7 07:58:50 2014 From: aimirim.software at gmail.com (=?ISO-8859-1?Q?S=E9rgio_Ferreira?=) Date: Mon, 7 Apr 2014 07:58:50 +0100 Subject: [rabbitmq-discuss] rabbitmq-server start "init terminating in do_boot" In-Reply-To: References: <8c8cb061-99e1-4517-81e7-6434477d731d@googlegroups.com> Message-ID: Hi Michael, I believe that I've tried that to. I'll try matching the exact same versions. Does rabbitmq need a specific version of earning? Thanks for the reply, S?rgio Em 07/04/2014 07:41, "Michael Klishin" escreveu: > 2014-04-07 1:39 GMT+04:00 S?rgio Ferreira : > >> {{badmatch,undefined},[{rabbit,maybe_hipe_compile >> ,0,[] >> > > What Erlang version do you run? Are you sure it's the same on both Windows > and Linux > machines? > -- > MK > > http://github.com/michaelklishin > http://twitter.com/michaelklishin > > _______________________________________________ > rabbitmq-discuss mailing list > rabbitmq-discuss at lists.rabbitmq.com > https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mklishin at gopivotal.com Mon Apr 7 07:59:53 2014 From: mklishin at gopivotal.com (Michael Klishin) Date: Mon, 7 Apr 2014 10:59:53 +0400 Subject: [rabbitmq-discuss] Dynamic Shovel and HA with 3.3 In-Reply-To: References: Message-ID: On 7 April 2014 at 00:51:13, Viswanathan Ramachandran (vish.ramachandran at gmail.com) wrote: > > 4. If master fails in an HA queue consumed by shovel in slave, > will shovel take care of re-consumption from new queue? With > the 3.3 enhancement of clients not needing to reconsume, I guess > this should work seamlessly with shovels as well? Shovel uses a regular (Erlang) client, so yes, it should.? > 5. Dynamic shovels don't allow detailed configuration. Are > the queues declared configured for no loss in messages? Namely > durable, acknowledge on explicit confirm etc? When Shovel declares a queue because it does not exist, it is declared as durable. Explicit acknowledgements are used by default but this can be user-configured. -- MK Software Engineer, Pivotal/RabbitMQ From mklishin at gopivotal.com Mon Apr 7 08:00:14 2014 From: mklishin at gopivotal.com (Michael Klishin) Date: Mon, 7 Apr 2014 11:00:14 +0400 Subject: [rabbitmq-discuss] Dynamic Shovel and HA with 3.3 In-Reply-To: References: Message-ID: On 7 April 2014 at 10:54:50, Michael Klishin (michael at rabbitmq.com) wrote: > > From : > > ?If the source queue does not exist on the source broker, it will > be declared as a durable queue with no arguments.? > > So you can declare a queue first or use policies to make it mirrored, > both ways should > work and won?t interfere with shovels. Forgot the link: http://www.rabbitmq.com/shovel-dynamic.html? -- MK Software Engineer, Pivotal/RabbitMQ From mklishin at gopivotal.com Mon Apr 7 08:02:03 2014 From: mklishin at gopivotal.com (Michael Klishin) Date: Mon, 7 Apr 2014 11:02:03 +0400 Subject: [rabbitmq-discuss] rabbitmq-server start "init terminating in do_boot" In-Reply-To: References: <8c8cb061-99e1-4517-81e7-6434477d731d@googlegroups.com> Message-ID: On 7 April 2014 at 11:01:14, S?rgio Ferreira (aimirim.software at gmail.com) wrote: > > > I believe that I've tried that to. I'll try matching the exact > same versions. > > > Does rabbitmq need a specific version http://www.rabbitmq.com/which-erlang.html? -- MK Software Engineer, Pivotal/RabbitMQ From ravindrapandey1982 at gmail.com Mon Apr 7 08:21:17 2014 From: ravindrapandey1982 at gmail.com (Ravi Pandey) Date: Mon, 7 Apr 2014 12:51:17 +0530 Subject: [rabbitmq-discuss] How to make RabbitMQ worker listening continuously In-Reply-To: References: Message-ID: I am using PHP with basic.consume On Mon, Apr 7, 2014 at 12:06 PM, Michael Klishin < michael.s.klishin at gmail.com> wrote: > > 2014-04-07 10:07 GMT+04:00 Ravi Pandey : > > I want my worker script to listen all request continuously without >> manually executing worker file. Many people suggested CRON but I don't want >> because if previous run hasn't finished then overlap can cause serious >> issues. >> >> Is there any way to run my worker script continuously in background? >> > > It depends on the language and client used. What do you use? Do you > consume using basic.consume or basic.get? > -- > MK > > http://github.com/michaelklishin > http://twitter.com/michaelklishin > > _______________________________________________ > rabbitmq-discuss mailing list > rabbitmq-discuss at lists.rabbitmq.com > https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From videlalvaro at gmail.com Mon Apr 7 08:30:46 2014 From: videlalvaro at gmail.com (Alvaro Videla) Date: Mon, 7 Apr 2014 09:30:46 +0200 Subject: [rabbitmq-discuss] How to make RabbitMQ worker listening continuously In-Reply-To: References: Message-ID: Hi, Have you seen the examples here: https://github.com/videlalvaro/php-amqplib/tree/master/demo, specially this one https://github.com/videlalvaro/php-amqplib/blob/master/demo/amqp_consumer.php ? This is also explained in our tutorials: http://www.rabbitmq.com/tutorials/tutorial-one-php.html Regards, Alvaro On Mon, Apr 7, 2014 at 9:21 AM, Ravi Pandey wrote: > I am using PHP with basic.consume > > > On Mon, Apr 7, 2014 at 12:06 PM, Michael Klishin > wrote: >> >> >> 2014-04-07 10:07 GMT+04:00 Ravi Pandey : >> >>> I want my worker script to listen all request continuously without >>> manually executing worker file. Many people suggested CRON but I don't want >>> because if previous run hasn't finished then overlap can cause serious >>> issues. >>> >>> Is there any way to run my worker script continuously in background? >> >> >> It depends on the language and client used. What do you use? Do you >> consume using basic.consume or basic.get? >> -- >> MK >> >> http://github.com/michaelklishin >> http://twitter.com/michaelklishin >> >> _______________________________________________ >> rabbitmq-discuss mailing list >> rabbitmq-discuss at lists.rabbitmq.com >> https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss >> > > > _______________________________________________ > rabbitmq-discuss mailing list > rabbitmq-discuss at lists.rabbitmq.com > https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss > From ravindrapandey1982 at gmail.com Mon Apr 7 08:41:01 2014 From: ravindrapandey1982 at gmail.com (Ravi Pandey) Date: Mon, 7 Apr 2014 13:11:01 +0530 Subject: [rabbitmq-discuss] How to make RabbitMQ worker listening continuously In-Reply-To: References: Message-ID: Yes I am using this library but first I need to run amqp_consumer.php manually to process all messages and I don't want to execute amqp_consumer.php file manually everytime. On Mon, Apr 7, 2014 at 1:00 PM, Alvaro Videla wrote: > Hi, > > Have you seen the examples here: > https://github.com/videlalvaro/php-amqplib/tree/master/demo, specially > this one > https://github.com/videlalvaro/php-amqplib/blob/master/demo/amqp_consumer.php > ? > > This is also explained in our tutorials: > http://www.rabbitmq.com/tutorials/tutorial-one-php.html > > Regards, > > Alvaro > > On Mon, Apr 7, 2014 at 9:21 AM, Ravi Pandey > wrote: > > I am using PHP with basic.consume > > > > > > On Mon, Apr 7, 2014 at 12:06 PM, Michael Klishin > > wrote: > >> > >> > >> 2014-04-07 10:07 GMT+04:00 Ravi Pandey : > >> > >>> I want my worker script to listen all request continuously without > >>> manually executing worker file. Many people suggested CRON but I don't > want > >>> because if previous run hasn't finished then overlap can cause serious > >>> issues. > >>> > >>> Is there any way to run my worker script continuously in background? > >> > >> > >> It depends on the language and client used. What do you use? Do you > >> consume using basic.consume or basic.get? > >> -- > >> MK > >> > >> http://github.com/michaelklishin > >> http://twitter.com/michaelklishin > >> > >> _______________________________________________ > >> rabbitmq-discuss mailing list > >> rabbitmq-discuss at lists.rabbitmq.com > >> https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss > >> > > > > > > _______________________________________________ > > rabbitmq-discuss mailing list > > rabbitmq-discuss at lists.rabbitmq.com > > https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss > > > _______________________________________________ > rabbitmq-discuss mailing list > rabbitmq-discuss at lists.rabbitmq.com > https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss > -------------- next part -------------- An HTML attachment was scrubbed... URL: From videlalvaro at gmail.com Mon Apr 7 09:14:14 2014 From: videlalvaro at gmail.com (Alvaro Videla) Date: Mon, 7 Apr 2014 10:14:14 +0200 Subject: [rabbitmq-discuss] How to make RabbitMQ worker listening continuously In-Reply-To: References: Message-ID: If I understand you correctly, then you are probably looking for this tool: http://supervisord.org/ On Mon, Apr 7, 2014 at 9:41 AM, Ravi Pandey wrote: > Yes I am using this library but first I need to run amqp_consumer.php > manually to process all messages and I don't want to execute > amqp_consumer.php file manually everytime. > > > On Mon, Apr 7, 2014 at 1:00 PM, Alvaro Videla wrote: >> >> Hi, >> >> Have you seen the examples here: >> https://github.com/videlalvaro/php-amqplib/tree/master/demo, specially >> this one >> https://github.com/videlalvaro/php-amqplib/blob/master/demo/amqp_consumer.php >> ? >> >> This is also explained in our tutorials: >> http://www.rabbitmq.com/tutorials/tutorial-one-php.html >> >> Regards, >> >> Alvaro >> >> On Mon, Apr 7, 2014 at 9:21 AM, Ravi Pandey >> wrote: >> > I am using PHP with basic.consume >> > >> > >> > On Mon, Apr 7, 2014 at 12:06 PM, Michael Klishin >> > wrote: >> >> >> >> >> >> 2014-04-07 10:07 GMT+04:00 Ravi Pandey : >> >> >> >>> I want my worker script to listen all request continuously without >> >>> manually executing worker file. Many people suggested CRON but I don't >> >>> want >> >>> because if previous run hasn't finished then overlap can cause serious >> >>> issues. >> >>> >> >>> Is there any way to run my worker script continuously in background? >> >> >> >> >> >> It depends on the language and client used. What do you use? Do you >> >> consume using basic.consume or basic.get? >> >> -- >> >> MK >> >> >> >> http://github.com/michaelklishin >> >> http://twitter.com/michaelklishin >> >> >> >> _______________________________________________ >> >> rabbitmq-discuss mailing list >> >> rabbitmq-discuss at lists.rabbitmq.com >> >> https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss >> >> >> > >> > >> > _______________________________________________ >> > rabbitmq-discuss mailing list >> > rabbitmq-discuss at lists.rabbitmq.com >> > https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss >> > >> _______________________________________________ >> rabbitmq-discuss mailing list >> rabbitmq-discuss at lists.rabbitmq.com >> https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss > > > > _______________________________________________ > rabbitmq-discuss mailing list > rabbitmq-discuss at lists.rabbitmq.com > https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss > From ravindrapandey1982 at gmail.com Mon Apr 7 10:17:01 2014 From: ravindrapandey1982 at gmail.com (Ravi Pandey) Date: Mon, 7 Apr 2014 14:47:01 +0530 Subject: [rabbitmq-discuss] How to make RabbitMQ worker listening continuously In-Reply-To: References: Message-ID: Yes - I assumed RabbitMQ is providing this functionality as RabbitMQ server is running continuously in background. Doesn't it? On Mon, Apr 7, 2014 at 1:44 PM, Alvaro Videla wrote: > If I understand you correctly, then you are probably looking for this > tool: http://supervisord.org/ > > On Mon, Apr 7, 2014 at 9:41 AM, Ravi Pandey > wrote: > > Yes I am using this library but first I need to run amqp_consumer.php > > manually to process all messages and I don't want to execute > > amqp_consumer.php file manually everytime. > > > > > > On Mon, Apr 7, 2014 at 1:00 PM, Alvaro Videla > wrote: > >> > >> Hi, > >> > >> Have you seen the examples here: > >> https://github.com/videlalvaro/php-amqplib/tree/master/demo, specially > >> this one > >> > https://github.com/videlalvaro/php-amqplib/blob/master/demo/amqp_consumer.php > >> ? > >> > >> This is also explained in our tutorials: > >> http://www.rabbitmq.com/tutorials/tutorial-one-php.html > >> > >> Regards, > >> > >> Alvaro > >> > >> On Mon, Apr 7, 2014 at 9:21 AM, Ravi Pandey > >> wrote: > >> > I am using PHP with basic.consume > >> > > >> > > >> > On Mon, Apr 7, 2014 at 12:06 PM, Michael Klishin > >> > wrote: > >> >> > >> >> > >> >> 2014-04-07 10:07 GMT+04:00 Ravi Pandey >: > >> >> > >> >>> I want my worker script to listen all request continuously without > >> >>> manually executing worker file. Many people suggested CRON but I > don't > >> >>> want > >> >>> because if previous run hasn't finished then overlap can cause > serious > >> >>> issues. > >> >>> > >> >>> Is there any way to run my worker script continuously in background? > >> >> > >> >> > >> >> It depends on the language and client used. What do you use? Do you > >> >> consume using basic.consume or basic.get? > >> >> -- > >> >> MK > >> >> > >> >> http://github.com/michaelklishin > >> >> http://twitter.com/michaelklishin > >> >> > >> >> _______________________________________________ > >> >> rabbitmq-discuss mailing list > >> >> rabbitmq-discuss at lists.rabbitmq.com > >> >> https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss > >> >> > >> > > >> > > >> > _______________________________________________ > >> > rabbitmq-discuss mailing list > >> > rabbitmq-discuss at lists.rabbitmq.com > >> > https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss > >> > > >> _______________________________________________ > >> rabbitmq-discuss mailing list > >> rabbitmq-discuss at lists.rabbitmq.com > >> https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss > > > > > > > > _______________________________________________ > > rabbitmq-discuss mailing list > > rabbitmq-discuss at lists.rabbitmq.com > > https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss > > > _______________________________________________ > rabbitmq-discuss mailing list > rabbitmq-discuss at lists.rabbitmq.com > https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss > -------------- next part -------------- An HTML attachment was scrubbed... URL: From videlalvaro at gmail.com Mon Apr 7 10:26:36 2014 From: videlalvaro at gmail.com (Alvaro Videla) Date: Mon, 7 Apr 2014 11:26:36 +0200 Subject: [rabbitmq-discuss] How to make RabbitMQ worker listening continuously In-Reply-To: References: Message-ID: RabbitMQ is running in the background, yes, but it has nothing to do on how you run your own workers. This is not related to PHP, Java or any other language on which you implement your workers. You have to decide how you want to run them. RabbitMQ will *just* take care of passing the messages around, (plus durability guarantees and so on). On Mon, Apr 7, 2014 at 11:17 AM, Ravi Pandey wrote: > Yes - I assumed RabbitMQ is providing this functionality as RabbitMQ server > is running continuously in background. Doesn't it? > > > On Mon, Apr 7, 2014 at 1:44 PM, Alvaro Videla wrote: >> >> If I understand you correctly, then you are probably looking for this >> tool: http://supervisord.org/ >> >> On Mon, Apr 7, 2014 at 9:41 AM, Ravi Pandey >> wrote: >> > Yes I am using this library but first I need to run amqp_consumer.php >> > manually to process all messages and I don't want to execute >> > amqp_consumer.php file manually everytime. >> > >> > >> > On Mon, Apr 7, 2014 at 1:00 PM, Alvaro Videla >> > wrote: >> >> >> >> Hi, >> >> >> >> Have you seen the examples here: >> >> https://github.com/videlalvaro/php-amqplib/tree/master/demo, specially >> >> this one >> >> >> >> https://github.com/videlalvaro/php-amqplib/blob/master/demo/amqp_consumer.php >> >> ? >> >> >> >> This is also explained in our tutorials: >> >> http://www.rabbitmq.com/tutorials/tutorial-one-php.html >> >> >> >> Regards, >> >> >> >> Alvaro >> >> >> >> On Mon, Apr 7, 2014 at 9:21 AM, Ravi Pandey >> >> wrote: >> >> > I am using PHP with basic.consume >> >> > >> >> > >> >> > On Mon, Apr 7, 2014 at 12:06 PM, Michael Klishin >> >> > wrote: >> >> >> >> >> >> >> >> >> 2014-04-07 10:07 GMT+04:00 Ravi Pandey >> >> >> : >> >> >> >> >> >>> I want my worker script to listen all request continuously without >> >> >>> manually executing worker file. Many people suggested CRON but I >> >> >>> don't >> >> >>> want >> >> >>> because if previous run hasn't finished then overlap can cause >> >> >>> serious >> >> >>> issues. >> >> >>> >> >> >>> Is there any way to run my worker script continuously in >> >> >>> background? >> >> >> >> >> >> >> >> >> It depends on the language and client used. What do you use? Do you >> >> >> consume using basic.consume or basic.get? >> >> >> -- >> >> >> MK >> >> >> >> >> >> http://github.com/michaelklishin >> >> >> http://twitter.com/michaelklishin >> >> >> >> >> >> _______________________________________________ >> >> >> rabbitmq-discuss mailing list >> >> >> rabbitmq-discuss at lists.rabbitmq.com >> >> >> https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss >> >> >> >> >> > >> >> > >> >> > _______________________________________________ >> >> > rabbitmq-discuss mailing list >> >> > rabbitmq-discuss at lists.rabbitmq.com >> >> > https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss >> >> > >> >> _______________________________________________ >> >> rabbitmq-discuss mailing list >> >> rabbitmq-discuss at lists.rabbitmq.com >> >> https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss >> > >> > >> > >> > _______________________________________________ >> > rabbitmq-discuss mailing list >> > rabbitmq-discuss at lists.rabbitmq.com >> > https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss >> > >> _______________________________________________ >> rabbitmq-discuss mailing list >> rabbitmq-discuss at lists.rabbitmq.com >> https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss > > > > _______________________________________________ > rabbitmq-discuss mailing list > rabbitmq-discuss at lists.rabbitmq.com > https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss > From simon at rabbitmq.com Mon Apr 7 10:46:04 2014 From: simon at rabbitmq.com (Simon MacMullen) Date: Mon, 07 Apr 2014 10:46:04 +0100 Subject: [rabbitmq-discuss] 3.3.0 In-Reply-To: <534137F4.3090206@alertme.com> References: <534137F4.3090206@alertme.com> Message-ID: <534273DC.7020707@rabbitmq.com> The #event{} record has changed in a source-compatible(ish) but binary-incompatible way. So you will definitely need to recompile if you have not already done so. You might also want to make a small source change to get better behaviour in the event of events being reissued. The reason it's changed is so that you can filter out reissued events which were requested (by calling rabbit:force_event_refresh/1) by something that was not your plugin, by checking the 'reference' field. An example would be when the mgmt DB fails over: it calls rabbit:force_event_refresh/1 and thus a load of events are emitted for things which already exist. If you never call rabbit:force_event_refresh/1 you can just match on reference = none to filter these out. Cheers, Simon On 06/04/2014 12:18, Dmitry Andrianov wrote: > Hello. > It looks like one of our RabbitMQ plugins stopped working after we > upgraded to 3.0.0. > Of course it can be that we are doing something wrong, I do not know any > details yet, just wanted to check if there is a known change I should be > aware of. > > The plugin that stopped working was hooking into the events to do its > stuff when new exchange is created. > > > -export([init/1, handle_call/2, handle_event/2, handle_info/2, > terminate/2, code_change/3]). > ... > > handle_event(Event = #event{type = exchange_created, props = [{name, > ExchangeName = #resource{}} | _]}, > State) -> > > rabbit_log:info("handle_event (Event=~p, State=~p)~n", [Event, State]), > ... > > > the INFO which used to appear in the log for 3.1.1 to 3.2.3 does not > appear anymore. > I downgraded for now until I have time to debug it properly. > > Thanks > This email is for the use of the intended recipient(s) only. If you have > received this email in error, please notify the sender immediately and > then delete it. If you are not the intended recipient, you must not use, > disclose or distribute this email without the author's prior permission. > AlertMe.com Ltd. is not responsible for any personal views expressed in > this message or any attachments that are those of the individual sender. > > AlertMe.com Ltd, 30 Station Road, Cambridge, CB1 2RE, UK. > Registered in England, Company number 578 2908, VAT registration number > GB 895 9914 42. > > > _______________________________________________ > rabbitmq-discuss mailing list > rabbitmq-discuss at lists.rabbitmq.com > https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss > From simon at rabbitmq.com Mon Apr 7 10:51:29 2014 From: simon at rabbitmq.com (Simon MacMullen) Date: Mon, 07 Apr 2014 10:51:29 +0100 Subject: [rabbitmq-discuss] Dynamic Shovel and HA with 3.3 In-Reply-To: References: Message-ID: <53427521.2060307@rabbitmq.com> On 06/04/2014 21:48, Viswanathan Ramachandran wrote: > 2. If the source queues (in the local exchange) mentioned in the dynamic > shovels are pre-created with x-ha-policy = all argument is it a valid > configuration? > > 3. Or is it necessary to set a HA policy for the pre-created queue from > which messages would be shoveled? Michael already answered your questions, but it's worth pointing out that HA was only configured with "x-ha-policy" in RabbitMQ 2.x. Unless you are talking about shovelling from 2.x to 3.3, you should forget "x-ha-policy" and configure mirroring as described here: http://www.rabbitmq.com/ha.html#genesis Cheers, Simon From dclark at nexatech.com Fri Apr 4 18:03:54 2014 From: dclark at nexatech.com (Dave Clark) Date: Fri, 4 Apr 2014 17:03:54 +0000 Subject: [rabbitmq-discuss] async C client Message-ID: <260f563f36654f50bcdac926e46a5b8d@BLUPR07MB068.namprd07.prod.outlook.com> I am trying to implement a C client to RabbitMQ and would like to receive messages asynchronously but I do not see any callback mechanism in the C interface. Is there one or do I have to implement my own? -------------- next part -------------- An HTML attachment was scrubbed... URL: From jbabrams at gmail.com Sat Apr 5 17:01:14 2014 From: jbabrams at gmail.com (Joe Abrams) Date: Sat, 5 Apr 2014 09:01:14 -0700 (PDT) Subject: [rabbitmq-discuss] Trouble with SSL on my newly installed RabbitMQ instance Message-ID: Hello, I'm trying to get SSL working on my Rabbit server, following the instructions at https://www.rabbitmq.com/ssl.html verbatim, but am getting this error when making connections: =INFO REPORT==== 5-Apr-2014::15:28:48 === accepting AMQP connection <0.678.0> (.....) =ERROR REPORT==== 5-Apr-2014::15:28:53 === error on AMQP connection <0.678.0>: {ssl_upgrade_error,{options,{cacertfile,[47,11... I'm following the SSL troubleshooting guide and get the above error when trying to 'Attempt SSL connection to Broker' from the same machine. It looks like my config *is* being read as I see this in my logs: =INFO REPORT==== 5-Apr-2014::14:38:13 === started SSL Listener on [::]:5671 +++ My system details +++ Ubuntu 13.10 (I'm on ec2) SSL version: [{ssl_app,"5.3"}, {supported,['tlsv1.2','tlsv1.1',tlsv1,sslv3]}, {available,['tlsv1.2','tlsv1.1',tlsv1,sslv3]}] RabbitMQ 3.3.0, Erlang R16B01 I've banged my head on this for a day now and am not making progress. Any help would be greatly appreciate. Thanks -Joe -------------- next part -------------- An HTML attachment was scrubbed... URL: From helloworld.hei at gmail.com Sun Apr 6 07:07:10 2014 From: helloworld.hei at gmail.com (helloworld.hei at gmail.com) Date: Sat, 5 Apr 2014 23:07:10 -0700 (PDT) Subject: [rabbitmq-discuss] MQTT plugin CONNACK return code 0x05 Message-ID: <7e743fad-b24d-4641-90e2-075cc708c736@googlegroups.com> Version: RabbitMQ 3.3.0 release my mqtt client CONNECT success and SUBSCRIBE success, but then return CONNACK message: 0x05 Connection Refused: not authorized and get the warning log: =WARNING REPORT==== MQTT login failed for "guest" access_refused (access must be from localhost) I had created issue in github . MQTT plugin documentation has allow_anonymous option, but i fail to use anonymous . -------------- next part -------------- An HTML attachment was scrubbed... URL: From helloworld.hei at gmail.com Sun Apr 6 10:33:58 2014 From: helloworld.hei at gmail.com (helloworld.hei at gmail.com) Date: Sun, 6 Apr 2014 02:33:58 -0700 (PDT) Subject: [rabbitmq-discuss] MQTT plugin CONNACK return code 0x05 In-Reply-To: <7e743fad-b24d-4641-90e2-075cc708c736@googlegroups.com> References: <7e743fad-b24d-4641-90e2-075cc708c736@googlegroups.com> Message-ID: The problem solved. details on github issues/ ? 2014?4?6????UTC+8??2?07?10??hellowo... at gmail.com??? > > Version: RabbitMQ 3.3.0 release > > my mqtt client CONNECT success and SUBSCRIBE success, but then return > CONNACK message: > > 0x05 Connection Refused: not authorized > > and get the warning log: > > =WARNING REPORT==== > MQTT login failed for "guest" access_refused (access must be from > localhost) > > I had created issue in github > . > > MQTT plugin documentation has allow_anonymous > option, but i fail to use anonymous . > -------------- next part -------------- An HTML attachment was scrubbed... URL: From kamil at kieliszczyk.net Mon Apr 7 09:08:14 2014 From: kamil at kieliszczyk.net (Kamil Kieliszczyk) Date: Mon, 07 Apr 2014 10:08:14 +0200 Subject: [rabbitmq-discuss] How to make RabbitMQ worker listening continuously In-Reply-To: References: Message-ID: <53425CEE.8010701@kieliszczyk.net> Have you read this? http://www.rabbitmq.com/tutorials/tutorial-one-php.html On 04/07/2014 09:41 AM, Ravi Pandey wrote: > Yes I am using this library but first I need to run amqp_consumer.php > manually to process all messages and I don't want to execute > amqp_consumer.php file manually everytime. > > > On Mon, Apr 7, 2014 at 1:00 PM, Alvaro Videla > wrote: > > Hi, > > Have you seen the examples here: > https://github.com/videlalvaro/php-amqplib/tree/master/demo, specially > this one > https://github.com/videlalvaro/php-amqplib/blob/master/demo/amqp_consumer.php > ? > > This is also explained in our tutorials: > http://www.rabbitmq.com/tutorials/tutorial-one-php.html > > Regards, > > Alvaro > > On Mon, Apr 7, 2014 at 9:21 AM, Ravi Pandey > > wrote: > > I am using PHP with basic.consume > > > > > > On Mon, Apr 7, 2014 at 12:06 PM, Michael Klishin > > > wrote: > >> > >> > >> 2014-04-07 10:07 GMT+04:00 Ravi Pandey > >: > >> > >>> I want my worker script to listen all request continuously without > >>> manually executing worker file. Many people suggested CRON but > I don't want > >>> because if previous run hasn't finished then overlap can cause > serious > >>> issues. > >>> > >>> Is there any way to run my worker script continuously in > background? > >> > >> > >> It depends on the language and client used. What do you use? Do you > >> consume using basic.consume or basic.get? > >> -- > >> MK > >> > >> http://github.com/michaelklishin > >> http://twitter.com/michaelklishin > >> > >> _______________________________________________ > >> rabbitmq-discuss mailing list > >> rabbitmq-discuss at lists.rabbitmq.com > > >> > https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss > >> > > > > > > _______________________________________________ > > rabbitmq-discuss mailing list > > rabbitmq-discuss at lists.rabbitmq.com > > > https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss > > > _______________________________________________ > rabbitmq-discuss mailing list > rabbitmq-discuss at lists.rabbitmq.com > > https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss > > > > > _______________________________________________ > rabbitmq-discuss mailing list > rabbitmq-discuss at lists.rabbitmq.com > https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: From borisvukaso at gmail.com Mon Apr 7 10:31:32 2014 From: borisvukaso at gmail.com (Boris Vukasovic) Date: Mon, 7 Apr 2014 11:31:32 +0200 Subject: [rabbitmq-discuss] Upgrade to 3.3.0 - now binding only licalhost Message-ID: <000f01cf5244$2b6de5b0$8249b110$@gmail.com> Hi - I use SSL for management plugin and ports 5672 and 5673 for access (5673 is SSL). After upgrade management plugin displays 'User can only login via localhost' and port 5673 is not accessible. The server status line ('service rabbitmq-server status ') reads: {listeners,[{clustering,25672,"::"},{amqp,5672,"::"}]}, Is it possible that /etc/rabbitmq/rabbitmq.config is not read any more? This is its content: [ {rabbit, [ {tcp_listeners, [5672] }, {ssl_listeners, [5673] }, {ssl_options, [ {cacertfile, "/etc/rabbitmq/ssl/ca/cacert.cer" }, {certfile, "/etc/rabbitmq/ssl/server/XXXXXXXXXX.cert.pem" }, {keyfile, "/etc/rabbitmq/ssl/server/ XXXXXXXXXX.key.pem" }, {verify, verify_none}, {fail_if_no_peer_cert, false }]} ]}, {rabbitmq_management, [{listener, [{port, 15672}, {ssl, true}, {ssl_opts, [{cacertfile, "/etc/rabbitmq/ssl/ca/cacert.pem"}, {certfile, "/etc/rabbitmq/ssl/server/ XXXXXXXXXX.cert.pem"}, {keyfile, "/etc/rabbitmq/ssl/server/ XXXXXXXXXX.key.pem"}]} ]} ]} ]. Regards, Boris -------------- next part -------------- An HTML attachment was scrubbed... URL: From simon at rabbitmq.com Mon Apr 7 11:05:13 2014 From: simon at rabbitmq.com (Simon MacMullen) Date: Mon, 07 Apr 2014 11:05:13 +0100 Subject: [rabbitmq-discuss] Trouble with SSL on my newly installed RabbitMQ instance In-Reply-To: References: Message-ID: <53427859.8010804@rabbitmq.com> On 05/04/2014 17:01, Joe Abrams wrote: > =ERROR REPORT==== 5-Apr-2014::15:28:53 === > error on AMQP connection <0.678.0>: > {ssl_upgrade_error,{options,{cacertfile,[47,11... Hmm, could you post your configuration file? > I'm following the SSL troubleshooting guide and get the above error when > trying to 'Attempt SSL connection to Broker' from the same machine. > > It looks like my config *is* being read as I see this in my logs: > > =INFO REPORT==== 5-Apr-2014::14:38:13 === > started SSL Listener on [::]:5671 That only proves that the ssl_listeners item was read I'm afraid. Cheers, Simon From simon at rabbitmq.com Mon Apr 7 11:06:55 2014 From: simon at rabbitmq.com (Simon MacMullen) Date: Mon, 07 Apr 2014 11:06:55 +0100 Subject: [rabbitmq-discuss] Upgrade to 3.3.0 - now binding only licalhost In-Reply-To: <000f01cf5244$2b6de5b0$8249b110$@gmail.com> References: <000f01cf5244$2b6de5b0$8249b110$@gmail.com> Message-ID: <534278BF.8000109@rabbitmq.com> On 07/04/2014 10:31, Boris Vukasovic wrote: > After upgrade management plugin displays ?User can only login via > localhost? and port 5673 is not accessible. 3.3.0 banned the "guest" user from accessing via anything other than localhost, by default. Please see http://www.rabbitmq.com/access-control.html Cheers, Simon From mklishin at gopivotal.com Mon Apr 7 11:06:23 2014 From: mklishin at gopivotal.com (Michael Klishin) Date: Mon, 7 Apr 2014 14:06:23 +0400 Subject: [rabbitmq-discuss] Trouble with SSL on my newly installed RabbitMQ instance In-Reply-To: References: Message-ID: On 7 April 2014 at 13:58:47, Joe Abrams (jbabrams at gmail.com) wrote: > > I've banged my head on this for a day now and am not making progress. > Any help would be greatly appreciate. Have you tried connecting with openssl s_client and using openssl s_server to verify your certificates and server key? There can be many reasons to?ssl_upgrade_error and openssl s_client/s_server will output more detailed messages when there?s an error.? -- MK Software Engineer, Pivotal/RabbitMQ From mklishin at gopivotal.com Mon Apr 7 11:07:57 2014 From: mklishin at gopivotal.com (Michael Klishin) Date: Mon, 7 Apr 2014 14:07:57 +0400 Subject: [rabbitmq-discuss] Upgrade to 3.3.0 - now binding only licalhost In-Reply-To: <000f01cf5244$2b6de5b0$8249b110$@gmail.com> References: <000f01cf5244$2b6de5b0$8249b110$@gmail.com> Message-ID: On 7 April 2014 at 14:05:41, Boris Vukasovic (borisvukaso at gmail.com) wrote: > > After upgrade management plugin displays ?User can only login > via localhost? and port 5673 is not accessible. Are you connecting from another host as ?guest?? See?http://www.rabbitmq.com/blog/2014/04/02/breaking-things-with-rabbitmq-3-3/? -- MK Software Engineer, Pivotal/RabbitMQ From simon at rabbitmq.com Mon Apr 7 11:20:19 2014 From: simon at rabbitmq.com (Simon MacMullen) Date: Mon, 07 Apr 2014 11:20:19 +0100 Subject: [rabbitmq-discuss] Problem with publish confirms In-Reply-To: References: Message-ID: <53427BE3.8000209@rabbitmq.com> It's going to be hard to debug this without seeing your code, especially since it sounds like it is quite likely to be a general Erlang problem. Having said that, here are some general pointers: On 04/04/2014 20:44, Ryan Brown wrote: > Then on publish, I select which one to use based > on the values passed-down the stack from message intake. Set self() as > the confirm handler and respond with a wait so that up the stack I can > initiate a receive loop in the message intake which is not a gen_server. Have you seen amqp_channel:wait_for_confirms/1? That might be easier than handling the confirms yourself if all you are going to do is wait for them anyway. > I have debug messages throughout and it appears that everything is > happening as expected. I trace the message through the publish path. To > being published and passing the wait back up to the receiver. I also see > where the basic.ack is received and the publish_confirm acknowledgement > passed all the way up to the receive loop. However, then things go awry. > I get the attached dump. It's very hard to see what is going wrong here in detail, but the surface issue is that your gen_server took more than 5 seconds to respond to a request - note that gen_server:call/2 will time out after 5 seconds. For this reason we never use gen_server:call/2 (we have a CI task that explodes if we accidentally commit code using it!); we always use call/3 with a timeout of 'infinity'. IMO this is a misfeature in Erlang. > My take-away, although likely I am wrong, is that somewhere, apparently > in amqp_balanced_publisher, a gen_server call is timing-out while > waiting for a response. But, I seem to be able to trace the responses > through the whole stack so I am thoroughly confused. That's correct. Of course, it should be quite rare / require heavy load for the 5s timeout to fail because the operation simply took too long. It's quite possible that the gen_server you are calling into is deadlocked or stuck in some other way. I would debug this as follows: 1) Change the call/2 to call/3 with infinity. 2) If the program now hangs rather than times out, launch 'observer' (http://www.erlang.org/doc/apps/observer/observer_ug.html), go and find the server you are calling into, and see what it's current stacktrace is. You may wish to give this process a registered name temporarily so you can find it more easily. Cheers, Simon From dev440 at gmail.com Mon Apr 7 11:36:53 2014 From: dev440 at gmail.com (chetan dev) Date: Mon, 7 Apr 2014 16:06:53 +0530 Subject: [rabbitmq-discuss] Raise an event on message receiving Message-ID: Hi, Is there any event triggered when queue receives a message, without consuming the message -- Cheten Dev -------------- next part -------------- An HTML attachment was scrubbed... URL: From simon at rabbitmq.com Mon Apr 7 11:47:16 2014 From: simon at rabbitmq.com (Simon MacMullen) Date: Mon, 07 Apr 2014 11:47:16 +0100 Subject: [rabbitmq-discuss] shovel auth change in 3.3.0? In-Reply-To: References: <533E89FA.3000000@rabbitmq.com> Message-ID: <53428234.2000007@rabbitmq.com> On 04/04/2014 16:37, Laing, Michael wrote: > Here's an example - one shovel fails and the other works. The difference > is the colon which, strictly speaking, perhaps shouldn't have been there > - but used to work: Ah right, reproduced. The difference is in how URI parsing works - in 3.2.x, "amqp://user:@/vhost" would be treated as a direct connection with no password (i.e. assumed authenticated), while in 3.3.x it would be treated as a direct connection with a password of "". So it's probably a bug. I just can't decide if it's a bug in 3.2.x or 3.3.0... Cheers, Simon From mklishin at gopivotal.com Mon Apr 7 11:49:31 2014 From: mklishin at gopivotal.com (Michael Klishin) Date: Mon, 7 Apr 2014 14:49:31 +0400 Subject: [rabbitmq-discuss] Raise an event on message receiving In-Reply-To: References: Message-ID: On 7 April 2014 at 14:39:13, chetan dev (dev440 at gmail.com) wrote: > > Is there any event triggered when queue receives a message, > without consuming the message http://www.rabbitmq.com/firehose.html is close to what you want.? -- MK Software Engineer, Pivotal/RabbitMQ From dev440 at gmail.com Mon Apr 7 12:14:47 2014 From: dev440 at gmail.com (chetan dev) Date: Mon, 7 Apr 2014 16:44:47 +0530 Subject: [rabbitmq-discuss] Raise an event on message receiving In-Reply-To: References: Message-ID: Hi, Thanks for the reply , but i dont think i need tracer . I just want an event raise whenever there is a new message without consuming the message Thanks On Mon, Apr 7, 2014 at 4:19 PM, Michael Klishin wrote: > On 7 April 2014 at 14:39:13, chetan dev (dev440 at gmail.com) wrote: > > > Is there any event triggered when queue receives a message, > > without consuming the message > > http://www.rabbitmq.com/firehose.html is close to what you want. > -- > MK > > Software Engineer, Pivotal/RabbitMQ > _______________________________________________ > rabbitmq-discuss mailing list > rabbitmq-discuss at lists.rabbitmq.com > https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss > -- Cheten Dev -------------- next part -------------- An HTML attachment was scrubbed... URL: From michael.laing at nytimes.com Mon Apr 7 12:16:23 2014 From: michael.laing at nytimes.com (Laing, Michael) Date: Mon, 7 Apr 2014 07:16:23 -0400 Subject: [rabbitmq-discuss] shovel auth change in 3.3.0? In-Reply-To: <53428234.2000007@rabbitmq.com> References: <533E89FA.3000000@rabbitmq.com> <53428234.2000007@rabbitmq.com> Message-ID: Hmmm - yes that's a problem. The amqp URI BNF in the rabbitmq docs includes this: amqp_userinfo = username [ ":" password ] > username = *( unreserved / pct-encoded / sub-delims ) > password = *( unreserved / pct-encoded / sub-delims ) which indicates that an empty password is acceptable in the URI, unfortunately. Of course that doesn't mean the shovel has to accept it. Possibly '1*' should be used instead of '*' above...? It's a pretty small buglet in the scheme of things... ml On Mon, Apr 7, 2014 at 6:47 AM, Simon MacMullen wrote: > On 04/04/2014 16:37, Laing, Michael wrote: > >> Here's an example - one shovel fails and the other works. The difference >> is the colon which, strictly speaking, perhaps shouldn't have been there >> - but used to work: >> > > Ah right, reproduced. The difference is in how URI parsing works - in > 3.2.x, "amqp://user:@/vhost" would be treated as a direct connection with > no password (i.e. assumed authenticated), while in 3.3.x it would be > treated as a direct connection with a password of "". > > So it's probably a bug. I just can't decide if it's a bug in 3.2.x or > 3.3.0... > > Cheers, Simon > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mklishin at gopivotal.com Mon Apr 7 12:20:26 2014 From: mklishin at gopivotal.com (Michael Klishin) Date: Mon, 7 Apr 2014 15:20:26 +0400 Subject: [rabbitmq-discuss] Raise an event on message receiving In-Reply-To: References: Message-ID: On 7 April 2014 at 15:17:08, chetan dev (dev440 at gmail.com) wrote: > > I just want an event raise whenever there is a new message without > consuming the message How do you expect to consume the events?? -- MK Software Engineer, Pivotal/RabbitMQ From dev440 at gmail.com Mon Apr 7 12:25:07 2014 From: dev440 at gmail.com (chetan dev) Date: Mon, 7 Apr 2014 16:55:07 +0530 Subject: [rabbitmq-discuss] Raise an event on message receiving In-Reply-To: References: Message-ID: hi, Whenever i will receive an event i will call a service from there i will consume the message Thanks On Mon, Apr 7, 2014 at 4:50 PM, Michael Klishin wrote: > On 7 April 2014 at 15:17:08, chetan dev (dev440 at gmail.com) wrote: > > > I just want an event raise whenever there is a new message without > > consuming the message > > How do you expect to consume the events? > -- > MK > > Software Engineer, Pivotal/RabbitMQ > _______________________________________________ > rabbitmq-discuss mailing list > rabbitmq-discuss at lists.rabbitmq.com > https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss > -- Cheten Dev B.Tech Final Year Dept. of Electrical Engg. IIT Delhi, New Delhi ph 8527333215 -------------- next part -------------- An HTML attachment was scrubbed... URL: From mklishin at gopivotal.com Mon Apr 7 12:29:05 2014 From: mklishin at gopivotal.com (Michael Klishin) Date: Mon, 7 Apr 2014 15:29:05 +0400 Subject: [rabbitmq-discuss] Raise an event on message receiving In-Reply-To: References: Message-ID: On 7 April 2014 at 15:27:31, chetan dev (dev440 at gmail.com) wrote: > > Whenever i will receive an event i will call a service from there > i will consume the message That?s not what I?m asking. How do you expect to be notified of such events? If you want to consume the message when it?s ready, you already have ?* basic.consume ?* queue.declare-ok which carries a number of messages ready ?* a way to retrieve queue info via HTTP API? -- MK Software Engineer, Pivotal/RabbitMQ From videlalvaro at gmail.com Mon Apr 7 13:45:26 2014 From: videlalvaro at gmail.com (Alvaro Videla) Date: Mon, 7 Apr 2014 14:45:26 +0200 Subject: [rabbitmq-discuss] Raise an event on message receiving In-Reply-To: References: Message-ID: Hi, On Mon, Apr 7, 2014 at 1:25 PM, chetan dev wrote: > Whenever i will receive an event i will call a service from there i will > consume the message May I ask what are you trying to do? Why not let the service consume the messages them? It seems you want to implement functionality that's already provided by RabbitMQ. Regards, Alvaro From steve at rabbitmq.com Mon Apr 7 15:04:08 2014 From: steve at rabbitmq.com (Steve Powell) Date: Mon, 7 Apr 2014 15:04:08 +0100 Subject: [rabbitmq-discuss] Java Client Using DefaultConsumer vs QueueingConsumer In-Reply-To: References: Message-ID: <52154277-4F84-44ED-9D21-A4C034EB7B41@rabbitmq.com> Thanks for the reminder; I thought I?d written it down somewhere :-). Steve Powell [Cell: +44-7815-838-558] Good design [?] is as little design as possible. Dieter Rams On 6 Apr 2014, at 07:32, rails wrote: > On this one. > Search for deprecated in the page -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve at rabbitmq.com Mon Apr 7 15:18:59 2014 From: steve at rabbitmq.com (Steve Powell) Date: Mon, 7 Apr 2014 15:18:59 +0100 Subject: [rabbitmq-discuss] basic.qos explanation Message-ID: The paragraph on http://www.rabbitmq.com/consumer-prefetch.html > Example - multiple consumers sharing the limit > The AMQP specification does not explain what happens if you invoke basic.qos multiple times with different global values. RabbitMQ interprets this as meaning that the two prefetch limits should be enforced independently of each other; consumers will only receive new messages when both limits on unacknowledged messages are met. > Has a curious last phrase (underscored). I think the phrase should be something like: consumers will only receive new messages when neither limit on unacknowledged messages are reached. (modulo the meaning of limits ? are met.) (Perhaps it should be breached instead of reached.) Steve Powell [Cell: +44-7815-838-558] Good design [?] is as little design as possible. Dieter Rams -------------- next part -------------- An HTML attachment was scrubbed... URL: From vish.ramachandran at gmail.com Mon Apr 7 16:31:45 2014 From: vish.ramachandran at gmail.com (Viswanathan Ramachandran) Date: Mon, 7 Apr 2014 08:31:45 -0700 Subject: [rabbitmq-discuss] Dynamic Shovel and HA with 3.3 In-Reply-To: <53427521.2060307@rabbitmq.com> References: <53427521.2060307@rabbitmq.com> Message-ID: Thanks all for your prompt replies. Cheers Vish On Mon, Apr 7, 2014 at 2:51 AM, Simon MacMullen wrote: > On 06/04/2014 21:48, Viswanathan Ramachandran wrote: > >> 2. If the source queues (in the local exchange) mentioned in the dynamic >> shovels are pre-created with x-ha-policy = all argument is it a valid >> configuration? >> >> 3. Or is it necessary to set a HA policy for the pre-created queue from >> which messages would be shoveled? >> > > Michael already answered your questions, but it's worth pointing out that > HA was only configured with "x-ha-policy" in RabbitMQ 2.x. Unless you are > talking about shovelling from 2.x to 3.3, you should forget "x-ha-policy" > and configure mirroring as described here: http://www.rabbitmq.com/ha. > html#genesis > > Cheers, Simon > -------------- next part -------------- An HTML attachment was scrubbed... URL: From michael.sander at gmail.com Mon Apr 7 16:39:32 2014 From: michael.sander at gmail.com (Michael Sander) Date: Mon, 7 Apr 2014 08:39:32 -0700 (PDT) Subject: [rabbitmq-discuss] Rabbit MQ Crash Message-ID: <4da4be68-9e3b-4bba-aaa9-e262df330d6f@googlegroups.com> Rabbit MQ seemed to have crashed in the middle of the night, and it's the second time in the past day. No idea why. =CRASH REPORT==== 7-Apr-2014::10:08:34 === crasher: initial call: rabbit_disk_monitor:init/1 pid: <0.3842.0> registered_name: rabbit_disk_monitor exception exit: {{badmatch,[]}, [{rabbit_disk_monitor,parse_free_unix,1,[]}, {rabbit_disk_monitor,internal_update,1,[]}, {rabbit_disk_monitor,handle_info,2,[]}, {gen_server,handle_msg,5, [{file,"gen_server.erl"},{line,607}]}, {proc_lib,init_p_do_apply,3, [{file,"proc_lib.erl"},{line,227}]}]} in function gen_server:terminate/6 (gen_server.erl, line 747) ancestors: [rabbit_disk_monitor_sup,rabbit_sup,<0.157.0>] messages: [{'$gen_call',{<0.3843.0>,#Ref<0.0.1.41682>}, get_disk_free_limit}] links: [<0.180.0>] dictionary: [] trap_exit: false status: running heap_size: 1597 stack_size: 24 reductions: 827021 neighbours: =SUPERVISOR REPORT==== 7-Apr-2014::10:08:35 === Supervisor: {local,rabbit_disk_monitor_sup} Context: child_terminated Reason: {{badmatch,[]}, [{rabbit_disk_monitor,parse_free_unix,1,[]}, {rabbit_disk_monitor,internal_update,1,[]}, {rabbit_disk_monitor,handle_info,2,[]}, {gen_server,handle_msg,5, [{file,"gen_server.erl"},{line,607}]}, {proc_lib,init_p_do_apply,3, [{file,"proc_lib.erl"},{line,227}]}]} Offender: [{pid,<0.3842.0>}, {name,rabbit_disk_monitor}, {mfargs,{rabbit_disk_monitor,start_link,[50000000]}}, {restart_type,transient}, {shutdown,4294967295}, {child_type,worker}] =CRASH REPORT==== 7-Apr-2014::10:08:35 === crasher: initial call: rabbit_mgmt_external_stats:init/1 pid: <0.3843.0> registered_name: rabbit_mgmt_external_stats exception exit: {{{badmatch,[]}, [{rabbit_disk_monitor,parse_free_unix,1,[]}, {rabbit_disk_monitor,internal_update,1,[]}, {rabbit_disk_monitor,handle_info,2,[]}, {gen_server,handle_msg,5, [{file,"gen_server.erl"},{line,607}]}, {proc_lib,init_p_do_apply,3, [{file,"proc_lib.erl"},{line,227}]}]}, {gen_server,call, [rabbit_disk_monitor,get_disk_free_limit, infinity]}} in function gen_server:terminate/6 (gen_server.erl, line 747) ancestors: [rabbit_mgmt_agent_sup,<0.236.0>] messages: [] links: [<0.237.0>] dictionary: [] trap_exit: false status: running heap_size: 6765 stack_size: 24 reductions: 2011976 neighbours: =SUPERVISOR REPORT==== 7-Apr-2014::10:08:35 === Supervisor: {local,rabbit_mgmt_agent_sup} Context: child_terminated Reason: {{{badmatch,[]}, [{rabbit_disk_monitor,parse_free_unix,1,[]}, {rabbit_disk_monitor,internal_update,1,[]}, {rabbit_disk_monitor,handle_info,2,[]}, {gen_server,handle_msg,5, [{file,"gen_server.erl"},{line,607}]}, {proc_lib,init_p_do_apply,3, [{file,"proc_lib.erl"},{line,227}]}]}, {gen_server,call, [rabbit_disk_monitor,get_disk_free_limit, infinity]}} Offender: [{pid,<0.3843.0>}, {name,rabbit_mgmt_external_stats}, {mfargs,{rabbit_mgmt_external_stats,start_link,[]}}, {restart_type,permanent}, {shutdown,5000}, {child_type,worker}] Some other basic linux info: $ cat /etc/*-release PRETTY_NAME="Debian GNU/Linux 7 (wheezy)" NAME="Debian GNU/Linux" VERSION_ID="7" VERSION="7 (wheezy)" ID=debian ANSI_COLOR="1;31" -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve at rabbitmq.com Mon Apr 7 17:23:11 2014 From: steve at rabbitmq.com (Steve Powell) Date: Mon, 7 Apr 2014 17:23:11 +0100 Subject: [rabbitmq-discuss] basic.qos explanation In-Reply-To: References: Message-ID: <1AE3F789-0642-4768-9460-8ECC52C1356D@rabbitmq.com> Of course I meant ?is reached?. Tsk. Steve Powell [Cell: +44-7815-838-558] Good design [?] is as little design as possible. Dieter Rams On 7 Apr 2014, at 15:18, Steve Powell wrote: > when neither limit on unacknowledged messages are reached -------------- next part -------------- An HTML attachment was scrubbed... URL: From ryankbrown at gmail.com Mon Apr 7 18:07:17 2014 From: ryankbrown at gmail.com (Ryan Brown) Date: Mon, 7 Apr 2014 11:07:17 -0600 Subject: [rabbitmq-discuss] Problem with publish confirms In-Reply-To: <53427BE3.8000209@rabbitmq.com> References: <53427BE3.8000209@rabbitmq.com> Message-ID: You rock Simon! So much great information. I found my problem and it was indeed a gen_server timeout due to a missed return in my confirmation chain going back up the chain to the rest interface where I respond to the client that posted the message for publishing. However, that said, I am curious about the option of just using amqp_channel:wait_for_confirms/1. I did not use this option for a couple of reasons. First was the fact that using the registered handler is mentioned quite often as the preferred option. My understanding was that using the amqp_channel:wait_for_confirms would cause the channel to wait for a response before another message could be published on that channel. If this is true, it seems like it could slow-down publishing more than registering a handler and letting publishing continue. Is this correct? Or am I misunderstanding the whole workflow? Speaking of misunderstandings; the second reason I did not use wait_for_confirms was that I didn't completely understand the definition: "Wait until all messages published since the last call have been either ack'd or nack'd by the broker; or until timeout elapses." What exactly does it mean by the "last call"? This sounded to me as if it may get in a situation that if calls are coming-in rapidly enough that they are publishing new messages prior to the previous message getting ack'd, it would not respond until all such queued messages were ack'd/nack'd. I realized this was unlikely, but could not come-up with documentation clear enough to convince me otherwise and this would not work for me as I need to respond to each publish individually. Thanks again for the, characteristically, excellent and comprehensive reponse. Best, Ryan On Mon, Apr 7, 2014 at 4:20 AM, Simon MacMullen wrote: > It's going to be hard to debug this without seeing your code, especially > since it sounds like it is quite likely to be a general Erlang problem. > > Having said that, here are some general pointers: > > On 04/04/2014 20:44, Ryan Brown wrote: > > >> Then on publish, I select which one to use based >> on the values passed-down the stack from message intake. Set self() as >> the confirm handler and respond with a wait so that up the stack I can >> initiate a receive loop in the message intake which is not a gen_server. >> > > Have you seen amqp_channel:wait_for_confirms/1? That might be easier than > handling the confirms yourself if all you are going to do is wait for them > anyway. > > I have debug messages throughout and it appears that everything is >> happening as expected. I trace the message through the publish path. To >> being published and passing the wait back up to the receiver. I also see >> where the basic.ack is received and the publish_confirm acknowledgement >> passed all the way up to the receive loop. However, then things go awry. >> I get the attached dump. >> > > It's very hard to see what is going wrong here in detail, but the surface > issue is that your gen_server took more than 5 seconds to respond to a > request - note that gen_server:call/2 will time out after 5 seconds. > > For this reason we never use gen_server:call/2 (we have a CI task that > explodes if we accidentally commit code using it!); we always use call/3 > with a timeout of 'infinity'. IMO this is a misfeature in Erlang. > > My take-away, although likely I am wrong, is that somewhere, apparently >> in amqp_balanced_publisher, a gen_server call is timing-out while >> waiting for a response. But, I seem to be able to trace the responses >> through the whole stack so I am thoroughly confused. >> > > That's correct. Of course, it should be quite rare / require heavy load > for the 5s timeout to fail because the operation simply took too long. It's > quite possible that the gen_server you are calling into is deadlocked or > stuck in some other way. > > I would debug this as follows: > > 1) Change the call/2 to call/3 with infinity. > 2) If the program now hangs rather than times out, launch 'observer' ( > http://www.erlang.org/doc/apps/observer/observer_ug.html), go and find > the server you are calling into, and see what it's current stacktrace is. > You may wish to give this process a registered name temporarily so you can > find it more easily. > > Cheers, Simon > > -- -rb -------------- next part -------------- An HTML attachment was scrubbed... URL: From alan.antonuk at gmail.com Mon Apr 7 18:22:02 2014 From: alan.antonuk at gmail.com (Alan Antonuk) Date: Mon, 7 Apr 2014 10:22:02 -0700 Subject: [rabbitmq-discuss] async C client In-Reply-To: <260f563f36654f50bcdac926e46a5b8d@BLUPR07MB068.namprd07.prod.outlook.com> References: <260f563f36654f50bcdac926e46a5b8d@BLUPR07MB068.namprd07.prod.outlook.com> Message-ID: Dave; Assuming you're talking about the rabbitmq-c client library, currently the API does not have a callback API for dealing with asynchronous events. (This is something that sorely needs to be added to rabbitmq-c, but requires a bit of rearchitecting with the way rabbitmq-c works) Depending on what exactly you're trying to do, I would recommend putting the amqp_connection_state_t object in its own thread, and then firing off events to other threads when you receive messages. -Alan On Fri, Apr 4, 2014 at 10:03 AM, Dave Clark wrote: > I am trying to implement a C client to RabbitMQ and would like to > receive messages asynchronously but I do not see any callback mechanism in > the C interface. Is there one or do I have to implement my own? > > _______________________________________________ > rabbitmq-discuss mailing list > rabbitmq-discuss at lists.rabbitmq.com > https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From simon at rabbitmq.com Mon Apr 7 18:25:03 2014 From: simon at rabbitmq.com (Simon MacMullen) Date: Mon, 07 Apr 2014 18:25:03 +0100 Subject: [rabbitmq-discuss] Problem with publish confirms In-Reply-To: References: <53427BE3.8000209@rabbitmq.com> Message-ID: <5342DF6F.2060602@rabbitmq.com> On 07/04/2014 18:07, Ryan Brown wrote: > However, that said, I am curious about the option of just using > amqp_channel:wait_for___confirms/1. I did not use this option for a > couple of reasons. > > First was the fact that using the registered handler is mentioned quite > often as the preferred option. My understanding was that using the > amqp_channel:wait_for___confirms would cause the channel to wait for a > response before another message could be published on that channel. If > this is true, it seems like it could slow-down publishing more than > registering a handler and letting publishing continue. Is this correct? > Or am I misunderstanding the whole workflow? It's possible I misunderstood you. I thought you were waiting for the confirm - i.e. you were effectively reimplementing wait_for_confirms() anyway. Yes, it is slower to call wait_for_confirms() (especially if you call it frequently) as you will stop publishing. > Speaking of misunderstandings; the second reason I did not use > wait_for_confirms was that I didn't completely understand the definition: > > "Wait until all messages published since the last call have been either > ack'd or nack'd by the broker; or until timeout elapses." > > What exactly does it mean by the "last call"? The last call to wait_for_confirms(). > This sounded to me as if > it may get in a situation that if calls are coming-in rapidly enough > that they are publishing new messages prior to the previous message > getting ack'd, it would not respond until all such queued messages were > ack'd/nack'd. Yes, that's correct. Cheers, Simon From simon at rabbitmq.com Mon Apr 7 18:28:58 2014 From: simon at rabbitmq.com (Simon MacMullen) Date: Mon, 07 Apr 2014 18:28:58 +0100 Subject: [rabbitmq-discuss] basic.qos explanation In-Reply-To: References: Message-ID: <5342E05A.3030607@rabbitmq.com> On 07/04/2014 15:18, Steve Powell wrote: > Has a curious last phrase (underscored). Hmm, that is not completely clear. Updated to "...when neither limit on unacknowledged messages has been reached" . Cheers, Simon From alan.antonuk at gmail.com Mon Apr 7 18:40:20 2014 From: alan.antonuk at gmail.com (Alan Antonuk) Date: Mon, 7 Apr 2014 10:40:20 -0700 Subject: [rabbitmq-discuss] rabbitmq-c corrupt content frame from ARM client In-Reply-To: <53411DA3.9080208@metm.org> References: <533F5E95.7080906@metm.org> <5340FF39.3050607@metm.org> <53411DA3.9080208@metm.org> Message-ID: Marco - Thanks I got the file and was able to look at it in Wireshark. Upon first glance I see two issues: the frame type is wrong (the first byte should be 0x01 instead of 0x02), and there appears to be an added 0x05 octet right at the beginning of the frame payload (it should be 0x00, 0x0a, 0x00, 0x0b). There may also be some other issues as I didn't look more deeply than that. Question for you: is your armv6 system running in little or big-endian mode? It appears that we've correctly detected the endianness of the system, but I'm not 100% sure on that. Also when compiling the library did you notice any compiler warnings? (Enable -Wall -Wextra if necessary). *To one of the RabbitMQ broker developers*: for the following message in the RabbitMQ broker logs: =ERROR REPORT==== 3-Apr-2014::15:05:35 === closing AMQP connection <0.380.0> (192.168.1.117:52890 -> 192.168.1.133:5672 ): {handshake_error,starting,0, {amqp_error,frame_error, "type 0, first 16 octets = <<5,10,0,11,0,0,0,80,98,112,__114,111,100,117,99,116>>...", none}} Are those first 16 octets including or not including the frame header? -Alan On Sun, Apr 6, 2014 at 2:25 AM, Marco Scoffier wrote: > Hi Alan, > > I just re-uploaded the file to include all the packets around the > handshake (not just the 3 I get when filtering for AMQP) as you mentioned > wireshark itself might also have parsing issues. > > Marco > > > On 04/06/2014 03:16 AM, Marco Scoffier wrote: > >> Sure, >> >> I dumped the 3 packets to a separate file. This is the binary data which >> you can open in wireshark: >> >> http://metm.org/images/amqp_handshake_wireshark.log >> >> Thanks for looking into this and happy to provide any additional tests, >> >> Marco >> >> On 04/06/2014 12:48 AM, Alan Antonuk wrote: >> >>> Marco; >>> >>> From the wireshark info, it sounds like the rabbitmq-c client is >>> sending completely the wrong data. It should send a Connection.Start-Ok >>> method and not a content-header frame (e.g., the frame type should be 1 >>> first few bytes of that frame should be: 0x01 {Method Frame}, 0x00, 0x00 >>> {Channel 0}, 0x00, 0x00 0xXX, 0xXX {Big Endian length of frame in bytes} >>> ). I'm also not sure what to to think about the sent header length: that >>> should be 8 bytes or 64 bits, I suspect thats a wireshark oddity though. >>> >>> The octets reported by the RabbitMQ server error message don't really >>> make sense to me: I'm having trouble correlating what is reported vs >>> what I think should be, even if the system is doing an incorrect >>> byteswap. I'm not 100% sure we're clear of any unaligned access issues, >>> but I would guess we'd be hitting bus-errors in that case, though I'm >>> not quite sure >>> >>> Would you be willing to send me a filtered down wireshark log of the >>> failed handshake? >>> >>> -Alan >>> >>> >>> On Fri, Apr 4, 2014 at 6:38 PM, Marco Scoffier >> > wrote: >>> >>> Hello, >>> >>> I apologize if this has been discussed before. I checked the >>> archives but did not find any mention of the issue. >>> >>> I've compiled rabbitmq-c [1] for the arm architecture [2] from and >>> am trying to send messages to a broker running on another machine >>> [3]. The rabbitmq-c examples run perfectly on the 64bit linux, same >>> code compiled for arm device and running there sends a corrupt >>> packet. >>> >>> [1] https://github.com/alanxz/__rabbitmq-c >>> >>> [2] armv6 linaro toolchain >>> [3] linux 64bit rabbitmq-server ppa for ubuntu. running on 12.04 >>> >>> I'm a little stuck about how to debug further and am looking for >>> some pointers where to look in the code or what extra information I >>> can provide. >>> >>> Wireshark shows three packets for the amqp_sendstring example >>> >>> ARM device/Source = 192.168.1.117 >>> 64bit broker/Destination = 192.168.1.133 >>> >>> Source Destination protocol length info >>> --------------+---------------__+--------+------+------------ >>> 192.168.1.117 192.168.1.133 AMQP 74 Protocol-Header 9-1 >>> 192.168.1.133 192.168.1.117 AMQP 522 Connection.Start >>> 192.168.1.117 192.168.1.133 AMQP 190 Content-Header >>> >>> wireshark's amqp frame dissection shows an error on the 3rd >>> Content-Header packet: >>> >>> Expert Info (Error/Protocol): Unknown header class 1290 >>> Message: Unknown header class 1290 >>> Severity level: Error >>> Group: Protocol >>> >>> the rabbitmq-server on the linux machine provides this log message: >>> >>> =ERROR REPORT==== 3-Apr-2014::15:05:35 === >>> closing AMQP connection <0.380.0> (192.168.1.117:52890 >>> -> 192.168.1.133:5672 >>> ): >>> {handshake_error,starting,0, >>> {amqp_error,frame_error, >>> "type 0, first 16 octets = >>> <<5,10,0,11,0,0,0,80,98,112,__114,111,100,117,99,116>>...", >>> none}} >>> >>> repeat that the same code compiled for 64bit linux same command >>> works as expected. >>> >>> Thanks for any pointers, >>> >>> Marco >>> _________________________________________________ >>> rabbitmq-discuss mailing list >>> rabbitmq-discuss at lists.__rabbitmq.com >>> >>> >>> https://lists.rabbitmq.com/__cgi-bin/mailman/listinfo/__rabbitmq-discuss >>> >>> >>> >>> >>> >>> >>> _______________________________________________ >>> rabbitmq-discuss mailing list >>> rabbitmq-discuss at lists.rabbitmq.com >>> https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss >>> >>> >> _______________________________________________ >> rabbitmq-discuss mailing list >> rabbitmq-discuss at lists.rabbitmq.com >> https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss >> > > _______________________________________________ > rabbitmq-discuss mailing list > rabbitmq-discuss at lists.rabbitmq.com > https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss > -------------- next part -------------- An HTML attachment was scrubbed... URL: From simon at rabbitmq.com Mon Apr 7 18:49:52 2014 From: simon at rabbitmq.com (Simon MacMullen) Date: Mon, 07 Apr 2014 18:49:52 +0100 Subject: [rabbitmq-discuss] rabbitmq-c corrupt content frame from ARM client In-Reply-To: References: <533F5E95.7080906@metm.org> <5340FF39.3050607@metm.org> <53411DA3.9080208@metm.org> Message-ID: <5342E540.2010108@rabbitmq.com> On 07/04/2014 18:40, Alan Antonuk wrote: > > *To one of the RabbitMQ broker developers*: for the following message in > the RabbitMQ broker logs: > > =ERROR REPORT==== 3-Apr-2014::15:05:35 === > closing AMQP connection <0.380.0> (192.168.1.117:52890 > > > -> > 192.168.1.133:5672 > >): > > {handshake_error,starting,0, > {amqp_error,frame_error, > "type 0, first 16 octets = > <<5,10,0,11,0,0,0,80,98,112,____114,111,100,117,99,116>>...", > > none}} > > Are those first 16 octets including or not including the frame header? No, they don't. 5 is the first byte of the frame body. The frame was decoded as type 0, channel 0. The type 0 is of course rather dubious. Unfortunately that error message doesn't include the frame payload size, so some information for the frame header is missing. Cheers, Simon From simon at rabbitmq.com Mon Apr 7 19:03:26 2014 From: simon at rabbitmq.com (Simon MacMullen) Date: Mon, 07 Apr 2014 19:03:26 +0100 Subject: [rabbitmq-discuss] Rabbit MQ Crash In-Reply-To: <4da4be68-9e3b-4bba-aaa9-e262df330d6f@googlegroups.com> References: <4da4be68-9e3b-4bba-aaa9-e262df330d6f@googlegroups.com> Message-ID: <5342E86E.4060508@rabbitmq.com> On 07/04/2014 16:39, Michael Sander wrote: > Rabbit MQ seemed to have crashed in the middle of the night, and it's > the second time in the past day. No idea why. Are those timestamps (7-Apr-2014::10:08:34) consistent with when the server crashed? Is that all that is in the sasl log or is there anything else? Cheers, Simon From ryankbrown at gmail.com Mon Apr 7 19:49:59 2014 From: ryankbrown at gmail.com (Ryan Brown) Date: Mon, 7 Apr 2014 12:49:59 -0600 Subject: [rabbitmq-discuss] Problem with publish confirms In-Reply-To: <5342DF6F.2060602@rabbitmq.com> References: <53427BE3.8000209@rabbitmq.com> <5342DF6F.2060602@rabbitmq.com> Message-ID: Thank you Simon. Now I think I have my head wrapped around this whole thing. Best, Ryan On Mon, Apr 7, 2014 at 11:25 AM, Simon MacMullen wrote: > On 07/04/2014 18:07, Ryan Brown wrote: > >> However, that said, I am curious about the option of just using >> amqp_channel:wait_for___confirms/1. I did not use this option for a >> >> couple of reasons. >> >> First was the fact that using the registered handler is mentioned quite >> often as the preferred option. My understanding was that using the >> amqp_channel:wait_for___confirms would cause the channel to wait for a >> >> response before another message could be published on that channel. If >> this is true, it seems like it could slow-down publishing more than >> registering a handler and letting publishing continue. Is this correct? >> Or am I misunderstanding the whole workflow? >> > > It's possible I misunderstood you. I thought you were waiting for the > confirm - i.e. you were effectively reimplementing wait_for_confirms() > anyway. Yes, it is slower to call wait_for_confirms() (especially if you > call it frequently) as you will stop publishing. > > > Speaking of misunderstandings; the second reason I did not use >> wait_for_confirms was that I didn't completely understand the definition: >> >> "Wait until all messages published since the last call have been either >> ack'd or nack'd by the broker; or until timeout elapses." >> >> What exactly does it mean by the "last call"? >> > > The last call to wait_for_confirms(). > > > This sounded to me as if >> it may get in a situation that if calls are coming-in rapidly enough >> that they are publishing new messages prior to the previous message >> getting ack'd, it would not respond until all such queued messages were >> ack'd/nack'd. >> > > Yes, that's correct. > > Cheers, Simon > -- -rb -------------- next part -------------- An HTML attachment was scrubbed... URL: From michael.sander at gmail.com Mon Apr 7 21:22:04 2014 From: michael.sander at gmail.com (Michael Sander) Date: Mon, 7 Apr 2014 16:22:04 -0400 Subject: [rabbitmq-discuss] Rabbit MQ Crash In-Reply-To: <5342E86E.4060508@rabbitmq.com> References: <4da4be68-9e3b-4bba-aaa9-e262df330d6f@googlegroups.com> <5342E86E.4060508@rabbitmq.com> Message-ID: Hi Simon, Those timestamps seem accurate. They are in UTC, so it was really around 6AM ET (where I am). Those were in the sasl logs, I couldn't find anything interesting in the normal rabbitmq log at the same time. Where else should I be looking for relevant logging info? Also, is this crash really due to the disk monitor? It seems like if there is a problem with the disk monitor, it shouldn't take down all of rabbitmq. Michael Sander On Mon, Apr 7, 2014 at 2:03 PM, Simon MacMullen wrote: > On 07/04/2014 16:39, Michael Sander wrote: > >> Rabbit MQ seemed to have crashed in the middle of the night, and it's >> the second time in the past day. No idea why. >> > > Are those timestamps (7-Apr-2014::10:08:34) consistent with when the > server crashed? Is that all that is in the sasl log or is there anything > else? > > Cheers, Simon > -------------- next part -------------- An HTML attachment was scrubbed... URL: From matthias at rabbitmq.com Mon Apr 7 21:58:37 2014 From: matthias at rabbitmq.com (Matthias Radestock) Date: Mon, 07 Apr 2014 21:58:37 +0100 Subject: [rabbitmq-discuss] Rabbit MQ Crash In-Reply-To: References: <4da4be68-9e3b-4bba-aaa9-e262df330d6f@googlegroups.com> <5342E86E.4060508@rabbitmq.com> Message-ID: <5343117D.7080405@rabbitmq.com> On 07/04/14 21:22, Michael Sander wrote: > Those timestamps seem accurate. They are in UTC, so it was really > around 6AM ET (where I am). Those were in the sasl logs, I couldn't > find anything interesting in the normal rabbitmq log at the same time. Did you post the *entire* contents of the SASL log? > Where else should I be looking for relevant logging info? The sasl and normal log is all we need. > Also, is this crash really due to the disk monitor? It seems like if > there is a problem with the disk monitor, it shouldn't take down all of > rabbitmq. Quite, but the odd thing here is that this appears to be a transient error. Rabbit will disable disk monitoring when it fails on startup, whereas the situation here is that disk monitoring works initially and then fails at some point. When that happens rabbit will attempt to restart the disk monitor, and if that fails a few times *then* it will die. We have some suspicion that the underlying cause might be a system update, in particular one that installs/updates Erlang. So it is worth checking that what cron jobs might be running on your system at 6am. Matthias. From matthias at rabbitmq.com Mon Apr 7 22:23:14 2014 From: matthias at rabbitmq.com (Matthias Radestock) Date: Mon, 07 Apr 2014 22:23:14 +0100 Subject: [rabbitmq-discuss] Rabbit MQ Crash In-Reply-To: References: <4da4be68-9e3b-4bba-aaa9-e262df330d6f@googlegroups.com> <5342E86E.4060508@rabbitmq.com> <5343117D.7080405@rabbitmq.com> Message-ID: <53431742.5000008@rabbitmq.com> Michael, On 07/04/14 22:06, Michael Sander wrote: > I am using a relatively memory-limited machine (n1-highcpu-2 on google > compute engine, details here > ), and > there's a possibility that I ran out of memory at some point, but I did > not see any direct evidence of that. Ah! Your original report did not mention that this is on GCE. Looking at our bug tracker, I see now that you reported the same issue previously, back in October: http://rabbitmq.1065348.n5.nabble.com/rabbitmq-nodedown-nodedown-Generic-server-rabbit-disk-monitor-terminating-td30939.html. > The logs I sent were just snippets. The complete logs should hopefully shed some further light on the root cause. Matthias. From vish.ramachandran at gmail.com Tue Apr 8 01:32:27 2014 From: vish.ramachandran at gmail.com (vish.ramachandran) Date: Mon, 7 Apr 2014 17:32:27 -0700 (PDT) Subject: [rabbitmq-discuss] Consumer Cancellation in 3.3 Message-ID: <1396917147135-34629.post@n5.nabble.com> Hi, We are planning to use the sstone client at https://github.com/sstone/amqp-client for our scala based RabbitMQ client. The library depends on RMQ java client version 3.2.1 "com.rabbitmq" % "amqp-client" % "3.2.1" The sstone amqp client does not handle consumer cancelations in that it does not reconsume from queue when this notification is received. Bug has been filed and is being worked on. However, if we upgrade our servers to RabbitMQ 3.3 can we use the same version of the library safely. I noticed that RabbitMQ 3.3 does not require clients to reconsume when master fails in a HA situation. The enhancement I am referring to is: 26070 automatically reconsume when mirrored queues fail over (and introduce x-cancel-on-ha-failover argument for the old behaviour) Thanks Vish -- View this message in context: http://rabbitmq.1065348.n5.nabble.com/Consumer-Cancellation-in-3-3-tp34629.html Sent from the RabbitMQ mailing list archive at Nabble.com. From marco at metm.org Tue Apr 8 01:34:36 2014 From: marco at metm.org (Marco Scoffier) Date: Mon, 07 Apr 2014 20:34:36 -0400 Subject: [rabbitmq-discuss] rabbitmq-c corrupt content frame from ARM client In-Reply-To: References: <533F5E95.7080906@metm.org> <5340FF39.3050607@metm.org> <53411DA3.9080208@metm.org> Message-ID: <5343441C.6000806@metm.org> Hey Alan, I'm sorry but I will have to get back to you in a day or two. I would love to help get to the bottom of this. Marco On 4/7/14, 1:40 PM, Alan Antonuk wrote: > Marco - Thanks I got the file and was able to look at it in Wireshark. > Upon first glance I see two issues: the frame type is wrong (the first > byte should be 0x01 instead of 0x02), and there appears to be an added > 0x05 octet right at the beginning of the frame payload (it should be > 0x00, 0x0a, 0x00, 0x0b). There may also be some other issues as I didn't > look more deeply than that. Question for you: is your armv6 system > running in little or big-endian mode? It appears that we've correctly > detected the endianness of the system, but I'm not 100% sure on that. > > Also when compiling the library did you notice any compiler warnings? > (Enable -Wall -Wextra if necessary). > > *To one of the RabbitMQ broker developers*: for the following message in > the RabbitMQ broker logs: > > =ERROR REPORT==== 3-Apr-2014::15:05:35 === > closing AMQP connection <0.380.0> (192.168.1.117:52890 > > > -> > 192.168.1.133:5672 > >): > > {handshake_error,starting,0, > {amqp_error,frame_error, > "type 0, first 16 octets = > <<5,10,0,11,0,0,0,80,98,112,____114,111,100,117,99,116>>...", > > none}} > > Are those first 16 octets including or not including the frame header? > > -Alan > > > > On Sun, Apr 6, 2014 at 2:25 AM, Marco Scoffier > wrote: > > Hi Alan, > > I just re-uploaded the file to include all the packets around the > handshake (not just the 3 I get when filtering for AMQP) as you > mentioned wireshark itself might also have parsing issues. > > Marco > > > On 04/06/2014 03:16 AM, Marco Scoffier wrote: > > Sure, > > I dumped the 3 packets to a separate file. This is the binary > data which > you can open in wireshark: > > http://metm.org/images/amqp___handshake_wireshark.log > > > Thanks for looking into this and happy to provide any additional > tests, > > Marco > > On 04/06/2014 12:48 AM, Alan Antonuk wrote: > > Marco; > > From the wireshark info, it sounds like the rabbitmq-c > client is > sending completely the wrong data. It should send a > Connection.Start-Ok > method and not a content-header frame (e.g., the frame type > should be 1 > first few bytes of that frame should be: 0x01 {Method > Frame}, 0x00, 0x00 > {Channel 0}, 0x00, 0x00 0xXX, 0xXX {Big Endian length of > frame in bytes} > ). I'm also not sure what to to think about the sent header > length: that > should be 8 bytes or 64 bits, I suspect thats a wireshark > oddity though. > > The octets reported by the RabbitMQ server error message > don't really > make sense to me: I'm having trouble correlating what is > reported vs > what I think should be, even if the system is doing an incorrect > byteswap. I'm not 100% sure we're clear of any unaligned > access issues, > but I would guess we'd be hitting bus-errors in that case, > though I'm > not quite sure > > Would you be willing to send me a filtered down wireshark > log of the > failed handshake? > > -Alan > > > On Fri, Apr 4, 2014 at 6:38 PM, Marco Scoffier > > >> wrote: > > Hello, > > I apologize if this has been discussed before. I > checked the > archives but did not find any mention of the issue. > > I've compiled rabbitmq-c [1] for the arm architecture > [2] from and > am trying to send messages to a broker running on > another machine > [3]. The rabbitmq-c examples run perfectly on the 64bit > linux, same > code compiled for arm device and running there sends a > corrupt > packet. > > [1] https://github.com/alanxz/____rabbitmq-c > > > > [2] armv6 linaro toolchain > [3] linux 64bit rabbitmq-server ppa for ubuntu. running > on 12.04 > > I'm a little stuck about how to debug further and am > looking for > some pointers where to look in the code or what extra > information I > can provide. > > Wireshark shows three packets for the amqp_sendstring > example > > ARM device/Source = 192.168.1.117 > 64bit broker/Destination = 192.168.1.133 > > Source Destination protocol length info > > --------------+---------------____+--------+------+-----------__- > 192.168.1.117 192.168.1.133 AMQP 74 > Protocol-Header 9-1 > 192.168.1.133 192.168.1.117 AMQP 522 > Connection.Start > 192.168.1.117 192.168.1.133 AMQP 190 > Content-Header > > wireshark's amqp frame dissection shows an error on the 3rd > Content-Header packet: > > Expert Info (Error/Protocol): Unknown header class 1290 > Message: Unknown header class 1290 > Severity level: Error > Group: Protocol > > the rabbitmq-server on the linux machine provides this > log message: > > =ERROR REPORT==== 3-Apr-2014::15:05:35 === > closing AMQP connection <0.380.0> (192.168.1.117:52890 > > -> 192.168.1.133:5672 > > ): > {handshake_error,starting,0, > {amqp_error,frame_error, > "type 0, first 16 octets = > > <<5,10,0,11,0,0,0,80,98,112,____114,111,100,117,99,116>>...", > none}} > > repeat that the same code compiled for 64bit linux same > command > works as expected. > > Thanks for any pointers, > > Marco > ___________________________________________________ > rabbitmq-discuss mailing list > rabbitmq-discuss at lists.__rabbi__tmq.com > > > > > https://lists.rabbitmq.com/____cgi-bin/mailman/listinfo/____rabbitmq-discuss > > > > > > > > > _________________________________________________ > rabbitmq-discuss mailing list > rabbitmq-discuss at lists.__rabbitmq.com > > https://lists.rabbitmq.com/__cgi-bin/mailman/listinfo/__rabbitmq-discuss > > > > _________________________________________________ > rabbitmq-discuss mailing list > rabbitmq-discuss at lists.__rabbitmq.com > > https://lists.rabbitmq.com/__cgi-bin/mailman/listinfo/__rabbitmq-discuss > > > > _________________________________________________ > rabbitmq-discuss mailing list > rabbitmq-discuss at lists.__rabbitmq.com > > https://lists.rabbitmq.com/__cgi-bin/mailman/listinfo/__rabbitmq-discuss > > > > > > _______________________________________________ > rabbitmq-discuss mailing list > rabbitmq-discuss at lists.rabbitmq.com > https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss > From srikanthtns at gmail.com Tue Apr 8 04:31:30 2014 From: srikanthtns at gmail.com (srikanth tns) Date: Mon, 7 Apr 2014 20:31:30 -0700 Subject: [rabbitmq-discuss] Rabbitmq queues dropped and recreated after a while In-Reply-To: References: Message-ID: we have not seen any error messages on the logs , Do you think the firewall is dropping the connections if they are idle ? If the connections are dropping off , what trigger that connections to be re-established back ? and after how long does it start reconnecting ? On Sun, Apr 6, 2014 at 11:35 PM, Michael Klishin < michael.s.klishin at gmail.com> wrote: > 2014-04-07 3:01 GMT+04:00 srikanth tns : > > =INFO REPORT==== 6-Apr-2014::22:45:02 === >> ending STOMP connection <0.6415.0> from 10.233.243.156:33153 >> > > It means that a client has disconnected or, possibly, an unrecoverable > error was received (but then there should be other messages in the log). > > Can there be clients that disconnect at will or intermediaries (e.g. > firewalls) that close > TCP connections, say due to inactivity? > > Note: this particular error message was removed from RabbitMQ STOMP plugin > around Jan 2012. > Consider upgrading. > -- > MK > > http://github.com/michaelklishin > http://twitter.com/michaelklishin > > _______________________________________________ > rabbitmq-discuss mailing list > rabbitmq-discuss at lists.rabbitmq.com > https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From matthias at rabbitmq.com Tue Apr 8 06:18:02 2014 From: matthias at rabbitmq.com (Matthias Radestock) Date: Tue, 08 Apr 2014 06:18:02 +0100 Subject: [rabbitmq-discuss] Rabbit MQ Crash In-Reply-To: References: <4da4be68-9e3b-4bba-aaa9-e262df330d6f@googlegroups.com> <5342E86E.4060508@rabbitmq.com> <5343117D.7080405@rabbitmq.com> <53431742.5000008@rabbitmq.com> Message-ID: <5343868A.3060207@rabbitmq.com> Michael, On 08/04/14 02:50, Michael Sander wrote: > Full logs are attached. You'll notice that it crashes pretty often now. The disk_monitor is crashing frequently, yes, but in none of the instances in the logs that actually took down rabbit (notice that there are no rabbit starts recorded in the rabbit.log); the disk_monitor restarts just fine and the bunny lives on. Do you have the logs covering the time period around the crash? > Here are the output of the commands > > $ sudo rabbitmqctl eval 'rabbit_misc:os_cmd("/bin/df -kP > /var/lib/rabbitmq/mnesia/")' > Error: syntax error before: Ah, sorry, missed a full stop. Should be sudo rabbitmqctl eval 'rabbit_misc:os_cmd("/bin/df -kP /var/lib/rabbitmq/mnesia/").' Matthias. From mklishin at gopivotal.com Tue Apr 8 06:34:50 2014 From: mklishin at gopivotal.com (Michael Klishin) Date: Tue, 8 Apr 2014 09:34:50 +0400 Subject: [rabbitmq-discuss] Consumer Cancellation in 3.3 In-Reply-To: <1396917147135-34629.post@n5.nabble.com> References: <1396917147135-34629.post@n5.nabble.com> Message-ID: On 8 April 2014 at 04:33:35, vish.ramachandran (vish.ramachandran at gmail.com) wrote: > > However, if we upgrade our servers to RabbitMQ 3.3 can we use > the same > version of the library safely. I noticed that RabbitMQ 3.3 does > not require > clients to reconsume when master fails in a HA situation. The > enhancement I > am referring to is: > > 26070 automatically reconsume when mirrored queues fail over > (and > introduce x-cancel-on-ha-failover argument for the old behaviour) Your understanding is correct.? -- MK Software Engineer, Pivotal/RabbitMQ From mklishin at gopivotal.com Tue Apr 8 06:35:54 2014 From: mklishin at gopivotal.com (Michael Klishin) Date: Tue, 8 Apr 2014 09:35:54 +0400 Subject: [rabbitmq-discuss] Rabbitmq queues dropped and recreated after a while In-Reply-To: References: Message-ID: On 8 April 2014 at 07:33:50, srikanth tns (srikanthtns at gmail.com) wrote: > > If the connections are dropping off , what trigger that connections > to be re-established back ? and after how long does it start reconnecting > ? Clients may choose to reconnect. Your apps may choose to reconnect.? -- MK Software Engineer, Pivotal/RabbitMQ From stammailbox at gmail.com Tue Apr 8 08:27:13 2014 From: stammailbox at gmail.com (rails) Date: Tue, 8 Apr 2014 10:27:13 +0300 Subject: [rabbitmq-discuss] Rabbitrmq thresholds defintions Message-ID: Trying to create a complicated scenario I successfully made my dev VM crash :-) (Did that by creating two DLX that routed messages between them until ram exploded) Now, having this experience made me think I need to set thresholds in production so my vm will always be responsive. Are there ways to control parameteres such as ram Precentages/CPU/Disk Space so my whole VM wouldn't be unresponsive? What are the best prcatices? Thanks. -------------- next part -------------- An HTML attachment was scrubbed... URL: From mklishin at gopivotal.com Tue Apr 8 08:35:51 2014 From: mklishin at gopivotal.com (Michael Klishin) Date: Tue, 8 Apr 2014 11:35:51 +0400 Subject: [rabbitmq-discuss] Rabbitrmq thresholds defintions In-Reply-To: References: Message-ID: On 8 April 2014 at 11:29:35, rails (stammailbox at gmail.com) wrote: > > Are there ways to control parameteres such as ram Precentages/CPU/Disk > Space so my whole VM wouldn't be unresponsive? > What are the best prcatices? Run RabbitMQ in a VM (or a set of thee) and use VM resource controls. Also, cycles are mentioned on?http://rabbitmq.com/dlx.html: perhaps message expiry can be used in your case.? -- MK Software Engineer, Pivotal/RabbitMQ From mes65 at cornell.edu Mon Apr 7 22:06:31 2014 From: mes65 at cornell.edu (Michael Sander) Date: Mon, 7 Apr 2014 17:06:31 -0400 Subject: [rabbitmq-discuss] Rabbit MQ Crash In-Reply-To: <5343117D.7080405@rabbitmq.com> References: <4da4be68-9e3b-4bba-aaa9-e262df330d6f@googlegroups.com> <5342E86E.4060508@rabbitmq.com> <5343117D.7080405@rabbitmq.com> Message-ID: Hi Matthias, I don't automatically update anything. The only cron jobs I have are those that start the rabbitmq service. In theory, they shouldn't do anything because rabbitmq should already be running. (I only have them in the event there is a rabbitmq crash). The logs I sent were just snippets. I will send complete logs, but I am not in front of a terminal and it will take me another 3 or 4 hours. I am using a relatively memory-limited machine (n1-highcpu-2 on google compute engine, details here), and there's a possibility that I ran out of memory at some point, but I did not see any direct evidence of that. I greatly appreciate you looking into this. Best, Michael Sander On Mon, Apr 7, 2014 at 4:58 PM, Matthias Radestock wrote: > On 07/04/14 21:22, Michael Sander wrote: > >> Those timestamps seem accurate. They are in UTC, so it was really >> around 6AM ET (where I am). Those were in the sasl logs, I couldn't >> find anything interesting in the normal rabbitmq log at the same time. >> > > Did you post the *entire* contents of the SASL log? > > > Where else should I be looking for relevant logging info? >> > > The sasl and normal log is all we need. > > > Also, is this crash really due to the disk monitor? It seems like if >> there is a problem with the disk monitor, it shouldn't take down all of >> rabbitmq. >> > > Quite, but the odd thing here is that this appears to be a transient > error. Rabbit will disable disk monitoring when it fails on startup, > whereas the situation here is that disk monitoring works initially and then > fails at some point. When that happens rabbit will attempt to restart the > disk monitor, and if that fails a few times *then* it will die. > > We have some suspicion that the underlying cause might be a system update, > in particular one that installs/updates Erlang. So it is worth checking > that what cron jobs might be running on your system at 6am. > > Matthias. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mes65 at cornell.edu Mon Apr 7 22:29:56 2014 From: mes65 at cornell.edu (Michael Sander) Date: Mon, 7 Apr 2014 17:29:56 -0400 Subject: [rabbitmq-discuss] Rabbit MQ Crash In-Reply-To: <53431742.5000008@rabbitmq.com> References: <4da4be68-9e3b-4bba-aaa9-e262df330d6f@googlegroups.com> <5342E86E.4060508@rabbitmq.com> <5343117D.7080405@rabbitmq.com> <53431742.5000008@rabbitmq.com> Message-ID: Wow, I had completely forgotten about that, I apologies for the duplicate. It also looks like I dropped the ball on answering your questions. Again my apologies, I was very busy at that time. In a few hours I will send you: - The complete sasl and normal logs - The output of the command rabbitmqctl eval 'rabbit_misc:os_cmd("/bin/df -kP /var/lib/rabbitmq/mnesia/") Best, Michael Sander On Mon, Apr 7, 2014 at 5:23 PM, Matthias Radestock wrote: > Michael, > > > On 07/04/14 22:06, Michael Sander wrote: > >> I am using a relatively memory-limited machine (n1-highcpu-2 on google >> compute engine, details here >> ), and >> >> there's a possibility that I ran out of memory at some point, but I did >> not see any direct evidence of that. >> > > Ah! Your original report did not mention that this is on GCE. Looking at > our bug tracker, I see now that you reported the same issue previously, > back in October: http://rabbitmq.1065348.n5.nabble.com/rabbitmq-nodedown- > nodedown-Generic-server-rabbit-disk-monitor-terminating-td30939.html. > > > The logs I sent were just snippets. >> > > The complete logs should hopefully shed some further light on the root > cause. > > Matthias. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mes65 at cornell.edu Tue Apr 8 02:50:54 2014 From: mes65 at cornell.edu (Michael Sander) Date: Mon, 7 Apr 2014 21:50:54 -0400 Subject: [rabbitmq-discuss] Rabbit MQ Crash In-Reply-To: References: <4da4be68-9e3b-4bba-aaa9-e262df330d6f@googlegroups.com> <5342E86E.4060508@rabbitmq.com> <5343117D.7080405@rabbitmq.com> <53431742.5000008@rabbitmq.com> Message-ID: Hi Matthias, Full logs are attached. You'll notice that it crashes pretty often now. Here are the output of the commands $ sudo rabbitmqctl eval 'rabbit_misc:os_cmd("/bin/df -kP /var/lib/rabbitmq/mnesia/")' Error: syntax error before: $ /bin/df -kP /var/lib/rabbitmq/mnesia/ Filesystem 1024-blocks Used Available Capacity Mounted on /dev/disk/by-uuid/36fd30d4-ea87-419f-a6a4-a1a3cf290ff1 10320184 6358284 3437664 65% / I just noticed the following in the non sasl log, I should have included it in the original: =ERROR REPORT==== 7-Apr-2014::10:08:34 === ** Generic server rabbit_disk_monitor terminating ** Last message in was update ** When Server state == {state,"/var/lib/rabbitmq/mnesia/rabbit at ocr-proc-2", 50000000,3702104064,100,10000, #Ref<0.0.1.41329>,false} ** Reason for termination == ** {{badmatch,[]}, [{rabbit_disk_monitor,parse_free_unix,1,[]}, {rabbit_disk_monitor,internal_update,1,[]}, {rabbit_disk_monitor,handle_info,2,[]}, {gen_server,handle_msg,5,[{file,"gen_server.erl"},{line,607}]}, {proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,227}]}]} =ERROR REPORT==== 7-Apr-2014::10:08:35 === ** Generic server rabbit_mgmt_external_stats terminating ** Last message in was emit_update ** When Server state == {state,32768} ** Reason for termination == ** {{{badmatch,[]}, [{rabbit_disk_monitor,parse_free_unix,1,[]}, {rabbit_disk_monitor,internal_update,1,[]}, {rabbit_disk_monitor,handle_info,2,[]}, {gen_server,handle_msg,5,[{file,"gen_server.erl"},{line,607}]}, {proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,227}]}]}, {gen_server,call,[rabbit_disk_monitor,get_disk_free_limit,infinity]}} Michael Sander mes65 at cornell.edu 607-227-9859 On Mon, Apr 7, 2014 at 5:29 PM, Michael Sander wrote: > Wow, I had completely forgotten about that, I apologies for the duplicate. > It also looks like I dropped the ball on answering your questions. Again > my apologies, I was very busy at that time. > > In a few hours I will send you: > - The complete sasl and normal logs > - The output of the command rabbitmqctl eval 'rabbit_misc:os_cmd("/bin/df > -kP /var/lib/rabbitmq/mnesia/") > > Best, > > Michael Sander > > > On Mon, Apr 7, 2014 at 5:23 PM, Matthias Radestock wrote: > >> Michael, >> >> >> On 07/04/14 22:06, Michael Sander wrote: >> >>> I am using a relatively memory-limited machine (n1-highcpu-2 on google >>> compute engine, details here >>> ), and >>> >>> there's a possibility that I ran out of memory at some point, but I did >>> not see any direct evidence of that. >>> >> >> Ah! Your original report did not mention that this is on GCE. Looking at >> our bug tracker, I see now that you reported the same issue previously, >> back in October: http://rabbitmq.1065348.n5.nabble.com/rabbitmq-nodedown- >> nodedown-Generic-server-rabbit-disk-monitor-terminating-td30939.html. >> >> >> The logs I sent were just snippets. >>> >> >> The complete logs should hopefully shed some further light on the root >> cause. >> >> Matthias. >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: rabbit-sasl.log Type: application/octet-stream Size: 37076 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: rabbit.log Type: application/octet-stream Size: 1872137 bytes Desc: not available URL: From mes65 at cornell.edu Tue Apr 8 08:33:17 2014 From: mes65 at cornell.edu (Michael Sander) Date: Tue, 8 Apr 2014 03:33:17 -0400 Subject: [rabbitmq-discuss] Rabbit MQ Crash In-Reply-To: <5343868A.3060207@rabbitmq.com> References: <4da4be68-9e3b-4bba-aaa9-e262df330d6f@googlegroups.com> <5342E86E.4060508@rabbitmq.com> <5343117D.7080405@rabbitmq.com> <53431742.5000008@rabbitmq.com> <5343868A.3060207@rabbitmq.com> Message-ID: Hi Matthais, What I sent you was everything I had. However, I did check ps -aux after the crash and rabbitmq-server was definitely not in there. I will turn off the cron jobs that automatically restart rabbitmq, and I'll let you know if I see it again. Here is the output of the command. $ sudo rabbitmqctl eval 'rabbit_misc:os_cmd("/bin/df -kP /var/lib/rabbitmq/mnes ia/").' "Filesystem 1024-blocks Used Available Capacity Mounted on\n/dev/disk/by-uuid/36fd30d4-ea87-419f-a6a4-a1a3cf290ff1 10320184 6348300 3447648 65% /\n" ...done. Also, I'm not sure whether it will help you, but attached is a screenshot of the rabbitmq console. If you see the start of the top chart at 19:00, there is a sharp increase in the queued messages. That's when I restarted rabbitmq after the crash. Everything before that was flat. Another point to note is that it currently says that the disk space is unavailable. I definitely remember seeing a value there at some point before, I don't know what causes that to occur. I've turned off my rabbimq auto-start cron jobs, I'll let you know if I see the crash again. Thanks again. Best, Michael Sander On Tue, Apr 8, 2014 at 1:18 AM, Matthias Radestock wrote: > Michael, > > > On 08/04/14 02:50, Michael Sander wrote: > >> Full logs are attached. You'll notice that it crashes pretty often now. >> > > The disk_monitor is crashing frequently, yes, but in none of the instances > in the logs that actually took down rabbit (notice that there are no rabbit > starts recorded in the rabbit.log); the disk_monitor restarts just fine and > the bunny lives on. > > Do you have the logs covering the time period around the crash? > > > Here are the output of the commands >> >> $ sudo rabbitmqctl eval 'rabbit_misc:os_cmd("/bin/df -kP >> /var/lib/rabbitmq/mnesia/")' >> Error: syntax error before: >> > > Ah, sorry, missed a full stop. Should be > > sudo rabbitmqctl eval 'rabbit_misc:os_cmd("/bin/df -kP > /var/lib/rabbitmq/mnesia/").' > > > Matthias. > -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: screenshot.png Type: image/png Size: 402745 bytes Desc: not available URL: From mes65 at cornell.edu Tue Apr 8 08:42:28 2014 From: mes65 at cornell.edu (Michael Sander) Date: Tue, 8 Apr 2014 03:42:28 -0400 Subject: [rabbitmq-discuss] Rabbit MQ Crash In-Reply-To: References: <4da4be68-9e3b-4bba-aaa9-e262df330d6f@googlegroups.com> <5342E86E.4060508@rabbitmq.com> <5343117D.7080405@rabbitmq.com> <53431742.5000008@rabbitmq.com> <5343868A.3060207@rabbitmq.com> Message-ID: Hi Matthias, I just checked rabbit the second after sending that, and it appears to have crashed. Here is some output, that you may find useful. Notice that erlang appears to be alive even though rabbitmq is not. ~$ ps aux|grep rabbit rabbitmq 1761 0.0 0.0 10836 160 ? S Apr07 0:01 /usr/lib/erlang/erts-5.9.1/bin/epmd -daemon 1001 26076 0.0 0.0 6308 600 pts/1 S+ 07:34 0:00 grep rabbit ~$ ps aux|grep erlan rabbitmq 1761 0.0 0.0 10836 160 ? S Apr07 0:01 /usr/lib/erlang/erts-5.9.1/bin/epmd -daemon 1001 26078 0.0 0.0 6308 600 pts/1 S+ 07:34 0:00 grep erlan ~$ df -h Filesystem Size Used Avail Use% Mounted on rootfs 9.9G 6.1G 3.3G 65% / udev 10M 0 10M 0% /dev tmpfs 181M 128K 181M 1% /run /dev/disk/by-uuid/36fd30d4-ea87-419f-a6a4-a1a3cf290ff1 9.9G 6.1G 3.3G 65% / tmpfs 5.0M 0 5.0M 0% /run/lock tmpfs 362M 0 362M 0% /run/shm ~$ sudo df -h Filesystem Size Used Avail Use% Mounted on rootfs 9.9G 6.1G 3.3G 65% / udev 10M 0 10M 0% /dev tmpfs 181M 128K 181M 1% /run /dev/disk/by-uuid/36fd30d4-ea87-419f-a6a4-a1a3cf290ff1 9.9G 6.1G 3.3G 65% / tmpfs 5.0M 0 5.0M 0% /run/lock tmpfs 362M 0 362M 0% /run/shm ~$ tail -n 100 /var/log/rabbitmq/rabbit at ocr-proc-2-sasl.log ~$ tail -n 100 /var/log/rabbitmq/rabbit at ocr-proc-2-sasl.log.1 crasher: initial call: rabbit_disk_monitor:init/1 pid: <0.19499.0> registered_name: [] exception exit: unsupported_platform in function gen_server:init_it/6 (gen_server.erl, line 320) ancestors: [rabbit_disk_monitor_sup,rabbit_sup,<0.157.0>] messages: [] links: [<0.180.0>] dictionary: [] trap_exit: false status: running heap_size: 6765 stack_size: 24 reductions: 13592 neighbours: =SUPERVISOR REPORT==== 8-Apr-2014::00:38:08 === Supervisor: {local,rabbit_disk_monitor_sup} Context: start_error Reason: unsupported_platform Offender: [{pid,{restarting,<0.5000.0>}}, {name,rabbit_disk_monitor}, {mfargs,{rabbit_disk_monitor,start_link,[50000000]}}, {restart_type,transient}, {shutdown,4294967295}, {child_type,worker}] =CRASH REPORT==== 8-Apr-2014::00:38:08 === crasher: initial call: rabbit_disk_monitor:init/1 pid: <0.19502.0> registered_name: [] exception exit: unsupported_platform in function gen_server:init_it/6 (gen_server.erl, line 320) ancestors: [rabbit_disk_monitor_sup,rabbit_sup,<0.157.0>] messages: [] links: [<0.180.0>] dictionary: [] trap_exit: false status: running heap_size: 6765 stack_size: 24 reductions: 13592 neighbours: =SUPERVISOR REPORT==== 8-Apr-2014::00:38:08 === Supervisor: {local,rabbit_disk_monitor_sup} Context: start_error Reason: unsupported_platform Offender: [{pid,{restarting,<0.5000.0>}}, {name,rabbit_disk_monitor}, {mfargs,{rabbit_disk_monitor,start_link,[50000000]}}, {restart_type,transient}, {shutdown,4294967295}, {child_type,worker}] =CRASH REPORT==== 8-Apr-2014::00:38:08 === crasher: initial call: rabbit_disk_monitor:init/1 pid: <0.19505.0> registered_name: [] exception exit: unsupported_platform in function gen_server:init_it/6 (gen_server.erl, line 320) ancestors: [rabbit_disk_monitor_sup,rabbit_sup,<0.157.0>] messages: [] links: [<0.180.0>] dictionary: [] trap_exit: false status: running heap_size: 6765 stack_size: 24 reductions: 13592 neighbours: =SUPERVISOR REPORT==== 8-Apr-2014::00:38:08 === Supervisor: {local,rabbit_disk_monitor_sup} Context: start_error Reason: unsupported_platform Offender: [{pid,{restarting,<0.5000.0>}}, {name,rabbit_disk_monitor}, {mfargs,{rabbit_disk_monitor,start_link,[50000000]}}, {restart_type,transient}, {shutdown,4294967295}, {child_type,worker}] =SUPERVISOR REPORT==== 8-Apr-2014::00:38:08 === Supervisor: {local,rabbit_disk_monitor_sup} Context: shutdown Reason: reached_max_restart_intensity Offender: [{pid,{restarting,<0.5000.0>}}, {name,rabbit_disk_monitor}, {mfargs,{rabbit_disk_monitor,start_link,[50000000]}}, {restart_type,transient}, {shutdown,4294967295}, {child_type,worker}] ~$ tail -n 100 /var/log/rabbitmq/rabbit at ocr-proc-2.log =WARNING REPORT==== 8-Apr-2014::07:29:47 === closing AMQP connection <0.20361.1> (127.0.0.1:48568 -> 127.0.0.1:5672): connection_closed_abruptly =WARNING REPORT==== 8-Apr-2014::07:29:47 === closing AMQP connection <0.20392.1> (127.0.0.1:48586 -> 127.0.0.1:5672): connection_closed_abruptly =WARNING REPORT==== 8-Apr-2014::07:29:49 === closing AMQP connection <0.20401.1> (127.0.0.1:48589 -> 127.0.0.1:5672): connection_closed_abruptly =WARNING REPORT==== 8-Apr-2014::07:29:50 === closing AMQP connection <0.22633.1> (127.0.0.1:50329 -> 127.0.0.1:5672): connection_closed_abruptly =WARNING REPORT==== 8-Apr-2014::07:29:51 === closing AMQP connection <0.16156.1> (127.0.0.1:44692 -> 127.0.0.1:5672): connection_closed_abruptly =INFO REPORT==== 8-Apr-2014::07:30:11 === accepting AMQP connection <0.22761.1> (127.0.0.1:50370 -> 127.0.0.1:5672) =WARNING REPORT==== 8-Apr-2014::07:30:11 === closing AMQP connection <0.22608.1> (127.0.0.1:50316 -> 127.0.0.1:5672): connection_closed_abruptly =INFO REPORT==== 8-Apr-2014::07:30:11 === accepting AMQP connection <0.22774.1> (127.0.0.1:50371 -> 127.0.0.1:5672) =INFO REPORT==== 8-Apr-2014::07:30:11 === accepting AMQP connection <0.22777.1> (127.0.0.1:50372 -> 127.0.0.1:5672) =INFO REPORT==== 8-Apr-2014::07:30:26 === accepting AMQP connection <0.22796.1> (127.0.0.1:50383 -> 127.0.0.1:5672) =INFO REPORT==== 8-Apr-2014::07:30:26 === accepting AMQP connection <0.22805.1> (127.0.0.1:50384 -> 127.0.0.1:5672) =INFO REPORT==== 8-Apr-2014::07:30:26 === accepting AMQP connection <0.22810.1> (127.0.0.1:50385 -> 127.0.0.1:5672) =INFO REPORT==== 8-Apr-2014::07:30:27 === accepting AMQP connection <0.22825.1> (127.0.0.1:50386 -> 127.0.0.1:5672) =INFO REPORT==== 8-Apr-2014::07:30:29 === accepting AMQP connection <0.22834.1> (127.0.0.1:50387 -> 127.0.0.1:5672) =INFO REPORT==== 8-Apr-2014::07:30:30 === accepting AMQP connection <0.22843.1> (127.0.0.1:50388 -> 127.0.0.1:5672) =INFO REPORT==== 8-Apr-2014::07:30:31 === accepting AMQP connection <0.22852.1> (127.0.0.1:50389 -> 127.0.0.1:5672) =INFO REPORT==== 8-Apr-2014::07:30:34 === accepting AMQP connection <0.22863.1> (127.0.0.1:50394 -> 127.0.0.1:5672) =INFO REPORT==== 8-Apr-2014::07:30:34 === accepting AMQP connection <0.22866.1> (127.0.0.1:50395 -> 127.0.0.1:5672) =WARNING REPORT==== 8-Apr-2014::07:30:36 === closing AMQP connection <0.22852.1> (127.0.0.1:50389 -> 127.0.0.1:5672): connection_closed_abruptly =INFO REPORT==== 8-Apr-2014::07:30:36 === accepting AMQP connection <0.22883.1> (127.0.0.1:50399 -> 127.0.0.1:5672) =WARNING REPORT==== 8-Apr-2014::07:30:37 === closing AMQP connection <0.22761.1> (127.0.0.1:50370 -> 127.0.0.1:5672): connection_closed_abruptly =WARNING REPORT==== 8-Apr-2014::07:30:38 === closing AMQP connection <0.22796.1> (127.0.0.1:50383 -> 127.0.0.1:5672): connection_closed_abruptly =INFO REPORT==== 8-Apr-2014::07:30:39 === accepting AMQP connection <0.22893.1> (127.0.0.1:50403 -> 127.0.0.1:5672) =WARNING REPORT==== 8-Apr-2014::07:30:39 === closing AMQP connection <0.22810.1> (127.0.0.1:50385 -> 127.0.0.1:5672): connection_closed_abruptly =INFO REPORT==== 8-Apr-2014::07:30:41 === accepting AMQP connection <0.22902.1> (127.0.0.1:50409 -> 127.0.0.1:5672) =INFO REPORT==== 8-Apr-2014::07:30:43 === accepting AMQP connection <0.22913.1> (127.0.0.1:50411 -> 127.0.0.1:5672) =INFO REPORT==== 8-Apr-2014::07:30:49 === accepting AMQP connection <0.22925.1> (127.0.0.1:50420 -> 127.0.0.1:5672) =INFO REPORT==== 8-Apr-2014::07:30:49 === accepting AMQP connection <0.22928.1> (127.0.0.1:50421 -> 127.0.0.1:5672) =WARNING REPORT==== 8-Apr-2014::07:30:50 === closing AMQP connection <0.22660.1> (127.0.0.1:50332 -> 127.0.0.1:5672): connection_closed_abruptly =INFO REPORT==== 8-Apr-2014::07:30:51 === accepting AMQP connection <0.22945.1> (127.0.0.1:50423 -> 127.0.0.1:5672) ~$ tail -n 100 /var/log/rabbitmq/shutdown_err /usr/lib/rabbitmq/bin/rabbitmqctl: 1: /etc/rabbitmq/rabbitmq-env.conf: ocr-proc-2=rabbit at localhost: not found ~$ tail -n 100 /var/log/rabbitmq/shutdown_log Stopping and halting node 'rabbit at ocr-proc-2' ... ...done. ~$ tail -n 100 /var/log/rabbitmq/startup_err /usr/lib/rabbitmq/bin/rabbitmq-server: 1: /etc/rabbitmq/rabbitmq-env.conf: ocr-proc-2=rabbit at localhost: not found Killed ~$ tail -n 100 /var/log/rabbitmq/startup_log RabbitMQ 3.2.4. Copyright (C) 2007-2013 GoPivotal, Inc. ## ## Licensed under the MPL. See http://www.rabbitmq.com/ ## ## ########## Logs: /var/log/rabbitmq/rabbit at ocr-proc-2.log ###### ## /var/log/rabbitmq/rabbit at ocr-proc-2-sasl.log ########## Starting broker... completed with 6 plugins. ? Michael Sander mes65 at cornell.edu 607-227-9859 On Tue, Apr 8, 2014 at 3:33 AM, Michael Sander wrote: > Hi Matthais, > > What I sent you was everything I had. However, I did check ps -aux after > the crash and rabbitmq-server was definitely not in there. I will turn off > the cron jobs that automatically restart rabbitmq, and I'll let you know if > I see it again. > > Here is the output of the command. > > $ sudo rabbitmqctl eval 'rabbit_misc:os_cmd("/bin/df -kP > /var/lib/rabbitmq/mnes > ia/").' > "Filesystem 1024-blocks > Used Available Capacity Mounted > on\n/dev/disk/by-uuid/36fd30d4-ea87-419f-a6a4-a1a3cf290ff1 10320184 > 6348300 3447648 65% /\n" > ...done. > > > Also, I'm not sure whether it will help you, but attached is a screenshot > of the rabbitmq console. If you see the start of the top chart at 19:00, > there is a sharp increase in the queued messages. That's when I restarted > rabbitmq after the crash. Everything before that was flat. Another point > to note is that it currently says that the disk space is unavailable. I > definitely remember seeing a value there at some point before, I don't know > what causes that to occur. > > I've turned off my rabbimq auto-start cron jobs, I'll let you know if I > see the crash again. > > Thanks again. > > Best, > > Michael Sander > > > > On Tue, Apr 8, 2014 at 1:18 AM, Matthias Radestock wrote: > >> Michael, >> >> >> On 08/04/14 02:50, Michael Sander wrote: >> >>> Full logs are attached. You'll notice that it crashes pretty often now. >>> >> >> The disk_monitor is crashing frequently, yes, but in none of the >> instances in the logs that actually took down rabbit (notice that there are >> no rabbit starts recorded in the rabbit.log); the disk_monitor restarts >> just fine and the bunny lives on. >> >> Do you have the logs covering the time period around the crash? >> >> >> Here are the output of the commands >>> >>> $ sudo rabbitmqctl eval 'rabbit_misc:os_cmd("/bin/df -kP >>> /var/lib/rabbitmq/mnesia/")' >>> Error: syntax error before: >>> >> >> Ah, sorry, missed a full stop. Should be >> >> sudo rabbitmqctl eval 'rabbit_misc:os_cmd("/bin/df -kP >> /var/lib/rabbitmq/mnesia/").' >> >> >> Matthias. >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mes65 at cornell.edu Tue Apr 8 08:49:31 2014 From: mes65 at cornell.edu (Michael Sander) Date: Tue, 8 Apr 2014 03:49:31 -0400 Subject: [rabbitmq-discuss] Rabbit MQ Crash In-Reply-To: References: <4da4be68-9e3b-4bba-aaa9-e262df330d6f@googlegroups.com> <5342E86E.4060508@rabbitmq.com> <5343117D.7080405@rabbitmq.com> <53431742.5000008@rabbitmq.com> <5343868A.3060207@rabbitmq.com> Message-ID: Unfortunately, I had to restart rabbitmq because it's on a production machine. Attached is screenshot after the restart. Notice that this time, free disk space limit is available. Please let me know what additional information you would like. If it goes down again, I will collect it. Michael Sander On Tue, Apr 8, 2014 at 3:42 AM, Michael Sander wrote: > Hi Matthias, > > I just checked rabbit the second after sending that, and it appears to > have crashed. Here is some output, that you may find useful. Notice that > erlang appears to be alive even though rabbitmq is not. > > ~$ ps aux|grep rabbit > rabbitmq 1761 0.0 0.0 10836 160 ? S Apr07 0:01 > /usr/lib/erlang/erts-5.9.1/bin/epmd -daemon > 1001 26076 0.0 0.0 6308 600 pts/1 S+ 07:34 0:00 grep > rabbit > ~$ ps aux|grep erlan > rabbitmq 1761 0.0 0.0 10836 160 ? S Apr07 0:01 > /usr/lib/erlang/erts-5.9.1/bin/epmd -daemon > 1001 26078 0.0 0.0 6308 600 pts/1 S+ 07:34 0:00 grep erlan > ~$ df -h > Filesystem Size Used Avail > Use% Mounted on > rootfs 9.9G 6.1G 3.3G > 65% / > udev 10M 0 10M > 0% /dev > tmpfs 181M 128K 181M > 1% /run > /dev/disk/by-uuid/36fd30d4-ea87-419f-a6a4-a1a3cf290ff1 9.9G 6.1G 3.3G > 65% / > tmpfs 5.0M 0 5.0M > 0% /run/lock > tmpfs 362M 0 362M > 0% /run/shm > ~$ sudo df -h > Filesystem Size Used Avail > Use% Mounted on > rootfs 9.9G 6.1G 3.3G > 65% / > udev 10M 0 10M > 0% /dev > tmpfs 181M 128K 181M > 1% /run > /dev/disk/by-uuid/36fd30d4-ea87-419f-a6a4-a1a3cf290ff1 9.9G 6.1G 3.3G > 65% / > tmpfs 5.0M 0 5.0M > 0% /run/lock > tmpfs 362M 0 362M > 0% /run/shm > ~$ tail -n 100 /var/log/rabbitmq/rabbit at ocr-proc-2-sasl.log > ~$ tail -n 100 /var/log/rabbitmq/rabbit at ocr-proc-2-sasl.log.1 > crasher: > initial call: rabbit_disk_monitor:init/1 > pid: <0.19499.0> > registered_name: [] > exception exit: unsupported_platform > in function gen_server:init_it/6 (gen_server.erl, line 320) > ancestors: [rabbit_disk_monitor_sup,rabbit_sup,<0.157.0>] > messages: [] > links: [<0.180.0>] > dictionary: [] > trap_exit: false > status: running > heap_size: 6765 > stack_size: 24 > reductions: 13592 > neighbours: > > =SUPERVISOR REPORT==== 8-Apr-2014::00:38:08 === > Supervisor: {local,rabbit_disk_monitor_sup} > Context: start_error > Reason: unsupported_platform > Offender: [{pid,{restarting,<0.5000.0>}}, > {name,rabbit_disk_monitor}, > {mfargs,{rabbit_disk_monitor,start_link,[50000000]}}, > {restart_type,transient}, > {shutdown,4294967295}, > {child_type,worker}] > > > =CRASH REPORT==== 8-Apr-2014::00:38:08 === > crasher: > initial call: rabbit_disk_monitor:init/1 > pid: <0.19502.0> > registered_name: [] > exception exit: unsupported_platform > in function gen_server:init_it/6 (gen_server.erl, line 320) > ancestors: [rabbit_disk_monitor_sup,rabbit_sup,<0.157.0>] > messages: [] > links: [<0.180.0>] > dictionary: [] > trap_exit: false > status: running > heap_size: 6765 > stack_size: 24 > reductions: 13592 > neighbours: > > =SUPERVISOR REPORT==== 8-Apr-2014::00:38:08 === > Supervisor: {local,rabbit_disk_monitor_sup} > Context: start_error > Reason: unsupported_platform > Offender: [{pid,{restarting,<0.5000.0>}}, > {name,rabbit_disk_monitor}, > {mfargs,{rabbit_disk_monitor,start_link,[50000000]}}, > {restart_type,transient}, > {shutdown,4294967295}, > {child_type,worker}] > > > =CRASH REPORT==== 8-Apr-2014::00:38:08 === > crasher: > initial call: rabbit_disk_monitor:init/1 > pid: <0.19505.0> > registered_name: [] > exception exit: unsupported_platform > in function gen_server:init_it/6 (gen_server.erl, line 320) > ancestors: [rabbit_disk_monitor_sup,rabbit_sup,<0.157.0>] > messages: [] > links: [<0.180.0>] > dictionary: [] > trap_exit: false > status: running > heap_size: 6765 > stack_size: 24 > reductions: 13592 > neighbours: > > =SUPERVISOR REPORT==== 8-Apr-2014::00:38:08 === > Supervisor: {local,rabbit_disk_monitor_sup} > Context: start_error > Reason: unsupported_platform > Offender: [{pid,{restarting,<0.5000.0>}}, > {name,rabbit_disk_monitor}, > {mfargs,{rabbit_disk_monitor,start_link,[50000000]}}, > {restart_type,transient}, > {shutdown,4294967295}, > {child_type,worker}] > > > =SUPERVISOR REPORT==== 8-Apr-2014::00:38:08 === > Supervisor: {local,rabbit_disk_monitor_sup} > Context: shutdown > Reason: reached_max_restart_intensity > Offender: [{pid,{restarting,<0.5000.0>}}, > {name,rabbit_disk_monitor}, > {mfargs,{rabbit_disk_monitor,start_link,[50000000]}}, > {restart_type,transient}, > {shutdown,4294967295}, > {child_type,worker}] > ~$ tail -n 100 /var/log/rabbitmq/rabbit at ocr-proc-2.log > =WARNING REPORT==== 8-Apr-2014::07:29:47 === > closing AMQP connection <0.20361.1> (127.0.0.1:48568 -> 127.0.0.1:5672): > connection_closed_abruptly > > =WARNING REPORT==== 8-Apr-2014::07:29:47 === > closing AMQP connection <0.20392.1> (127.0.0.1:48586 -> 127.0.0.1:5672): > connection_closed_abruptly > > =WARNING REPORT==== 8-Apr-2014::07:29:49 === > closing AMQP connection <0.20401.1> (127.0.0.1:48589 -> 127.0.0.1:5672): > connection_closed_abruptly > > =WARNING REPORT==== 8-Apr-2014::07:29:50 === > closing AMQP connection <0.22633.1> (127.0.0.1:50329 -> 127.0.0.1:5672): > connection_closed_abruptly > > =WARNING REPORT==== 8-Apr-2014::07:29:51 === > closing AMQP connection <0.16156.1> (127.0.0.1:44692 -> 127.0.0.1:5672): > connection_closed_abruptly > > =INFO REPORT==== 8-Apr-2014::07:30:11 === > accepting AMQP connection <0.22761.1> (127.0.0.1:50370 -> 127.0.0.1:5672) > > =WARNING REPORT==== 8-Apr-2014::07:30:11 === > closing AMQP connection <0.22608.1> (127.0.0.1:50316 -> 127.0.0.1:5672): > connection_closed_abruptly > > =INFO REPORT==== 8-Apr-2014::07:30:11 === > accepting AMQP connection <0.22774.1> (127.0.0.1:50371 -> 127.0.0.1:5672) > > =INFO REPORT==== 8-Apr-2014::07:30:11 === > accepting AMQP connection <0.22777.1> (127.0.0.1:50372 -> 127.0.0.1:5672) > > =INFO REPORT==== 8-Apr-2014::07:30:26 === > accepting AMQP connection <0.22796.1> (127.0.0.1:50383 -> 127.0.0.1:5672) > > =INFO REPORT==== 8-Apr-2014::07:30:26 === > accepting AMQP connection <0.22805.1> (127.0.0.1:50384 -> 127.0.0.1:5672) > > =INFO REPORT==== 8-Apr-2014::07:30:26 === > accepting AMQP connection <0.22810.1> (127.0.0.1:50385 -> 127.0.0.1:5672) > > =INFO REPORT==== 8-Apr-2014::07:30:27 === > accepting AMQP connection <0.22825.1> (127.0.0.1:50386 -> 127.0.0.1:5672) > > =INFO REPORT==== 8-Apr-2014::07:30:29 === > accepting AMQP connection <0.22834.1> (127.0.0.1:50387 -> 127.0.0.1:5672) > > =INFO REPORT==== 8-Apr-2014::07:30:30 === > accepting AMQP connection <0.22843.1> (127.0.0.1:50388 -> 127.0.0.1:5672) > > =INFO REPORT==== 8-Apr-2014::07:30:31 === > accepting AMQP connection <0.22852.1> (127.0.0.1:50389 -> 127.0.0.1:5672) > > =INFO REPORT==== 8-Apr-2014::07:30:34 === > accepting AMQP connection <0.22863.1> (127.0.0.1:50394 -> 127.0.0.1:5672) > > =INFO REPORT==== 8-Apr-2014::07:30:34 === > accepting AMQP connection <0.22866.1> (127.0.0.1:50395 -> 127.0.0.1:5672) > > =WARNING REPORT==== 8-Apr-2014::07:30:36 === > closing AMQP connection <0.22852.1> (127.0.0.1:50389 -> 127.0.0.1:5672): > connection_closed_abruptly > > =INFO REPORT==== 8-Apr-2014::07:30:36 === > accepting AMQP connection <0.22883.1> (127.0.0.1:50399 -> 127.0.0.1:5672) > > =WARNING REPORT==== 8-Apr-2014::07:30:37 === > closing AMQP connection <0.22761.1> (127.0.0.1:50370 -> 127.0.0.1:5672): > connection_closed_abruptly > > =WARNING REPORT==== 8-Apr-2014::07:30:38 === > closing AMQP connection <0.22796.1> (127.0.0.1:50383 -> 127.0.0.1:5672): > connection_closed_abruptly > > =INFO REPORT==== 8-Apr-2014::07:30:39 === > accepting AMQP connection <0.22893.1> (127.0.0.1:50403 -> 127.0.0.1:5672) > > =WARNING REPORT==== 8-Apr-2014::07:30:39 === > closing AMQP connection <0.22810.1> (127.0.0.1:50385 -> 127.0.0.1:5672): > connection_closed_abruptly > > =INFO REPORT==== 8-Apr-2014::07:30:41 === > accepting AMQP connection <0.22902.1> (127.0.0.1:50409 -> 127.0.0.1:5672) > > =INFO REPORT==== 8-Apr-2014::07:30:43 === > accepting AMQP connection <0.22913.1> (127.0.0.1:50411 -> 127.0.0.1:5672) > > =INFO REPORT==== 8-Apr-2014::07:30:49 === > accepting AMQP connection <0.22925.1> (127.0.0.1:50420 -> 127.0.0.1:5672) > > =INFO REPORT==== 8-Apr-2014::07:30:49 === > accepting AMQP connection <0.22928.1> (127.0.0.1:50421 -> 127.0.0.1:5672) > > =WARNING REPORT==== 8-Apr-2014::07:30:50 === > closing AMQP connection <0.22660.1> (127.0.0.1:50332 -> 127.0.0.1:5672): > connection_closed_abruptly > > =INFO REPORT==== 8-Apr-2014::07:30:51 === > accepting AMQP connection <0.22945.1> (127.0.0.1:50423 -> 127.0.0.1:5672) > ~$ tail -n 100 /var/log/rabbitmq/shutdown_err > /usr/lib/rabbitmq/bin/rabbitmqctl: 1: /etc/rabbitmq/rabbitmq-env.conf: > ocr-proc-2=rabbit at localhost: not found > ~$ tail -n 100 /var/log/rabbitmq/shutdown_log > Stopping and halting node 'rabbit at ocr-proc-2' ... > ...done. > ~$ tail -n 100 /var/log/rabbitmq/startup_err > /usr/lib/rabbitmq/bin/rabbitmq-server: 1: /etc/rabbitmq/rabbitmq-env.conf: > ocr-proc-2=rabbit at localhost: not found > Killed > ~$ tail -n 100 /var/log/rabbitmq/startup_log > > RabbitMQ 3.2.4. Copyright (C) 2007-2013 GoPivotal, Inc. > ## ## Licensed under the MPL. See http://www.rabbitmq.com/ > ## ## > ########## Logs: /var/log/rabbitmq/rabbit at ocr-proc-2.log > ###### ## /var/log/rabbitmq/rabbit at ocr-proc-2-sasl.log > ########## > Starting broker... completed with 6 plugins. > > > > > Michael Sander > mes65 at cornell.edu > 607-227-9859 > > > On Tue, Apr 8, 2014 at 3:33 AM, Michael Sander wrote: > >> Hi Matthais, >> >> What I sent you was everything I had. However, I did check ps -aux after >> the crash and rabbitmq-server was definitely not in there. I will turn off >> the cron jobs that automatically restart rabbitmq, and I'll let you know if >> I see it again. >> >> Here is the output of the command. >> >> $ sudo rabbitmqctl eval 'rabbit_misc:os_cmd("/bin/df -kP >> /var/lib/rabbitmq/mnes >> ia/").' >> "Filesystem 1024-blocks >> Used Available Capacity Mounted >> on\n/dev/disk/by-uuid/36fd30d4-ea87-419f-a6a4-a1a3cf290ff1 10320184 >> 6348300 3447648 65% /\n" >> ...done. >> >> >> Also, I'm not sure whether it will help you, but attached is a screenshot >> of the rabbitmq console. If you see the start of the top chart at 19:00, >> there is a sharp increase in the queued messages. That's when I restarted >> rabbitmq after the crash. Everything before that was flat. Another point >> to note is that it currently says that the disk space is unavailable. I >> definitely remember seeing a value there at some point before, I don't know >> what causes that to occur. >> >> I've turned off my rabbimq auto-start cron jobs, I'll let you know if I >> see the crash again. >> >> Thanks again. >> >> Best, >> >> Michael Sander >> >> >> >> On Tue, Apr 8, 2014 at 1:18 AM, Matthias Radestock > > wrote: >> >>> Michael, >>> >>> >>> On 08/04/14 02:50, Michael Sander wrote: >>> >>>> Full logs are attached. You'll notice that it crashes pretty often now. >>>> >>> >>> The disk_monitor is crashing frequently, yes, but in none of the >>> instances in the logs that actually took down rabbit (notice that there are >>> no rabbit starts recorded in the rabbit.log); the disk_monitor restarts >>> just fine and the bunny lives on. >>> >>> Do you have the logs covering the time period around the crash? >>> >>> >>> Here are the output of the commands >>>> >>>> $ sudo rabbitmqctl eval 'rabbit_misc:os_cmd("/bin/df -kP >>>> /var/lib/rabbitmq/mnesia/")' >>>> Error: syntax error before: >>>> >>> >>> Ah, sorry, missed a full stop. Should be >>> >>> sudo rabbitmqctl eval 'rabbit_misc:os_cmd("/bin/df -kP >>> /var/lib/rabbitmq/mnesia/").' >>> >>> >>> Matthias. >>> >> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: screenshot2.png Type: image/png Size: 219630 bytes Desc: not available URL: From matwey.kornilov at gmail.com Tue Apr 8 12:30:45 2014 From: matwey.kornilov at gmail.com (Matwey V. Kornilov) Date: Tue, 08 Apr 2014 15:30:45 +0400 Subject: [rabbitmq-discuss] Idempotent version of queue.unbind Message-ID: Hi, queue.unbind make an assumption that the binding must exists before it unbinded. is there a way to have a version of queue.unbind not to making such an assumption? I mean silently success when binding not exists. From mklishin at gopivotal.com Tue Apr 8 12:33:39 2014 From: mklishin at gopivotal.com (Michael Klishin) Date: Tue, 8 Apr 2014 15:33:39 +0400 Subject: [rabbitmq-discuss] Idempotent version of queue.unbind In-Reply-To: References: Message-ID: On 8 April 2014 at 15:32:26, Matwey V. Kornilov (matwey.kornilov at gmail.com) wrote: > > queue.unbind make an assumption that the binding must exists > before it > unbinded. > > is there a way to have a version of queue.unbind not to making such > an > assumption? I mean silently success when binding not exists. queue.unbind is idempotent as of RabbitMQ 3.2.? -- MK Software Engineer, Pivotal/RabbitMQ From simon at rabbitmq.com Tue Apr 8 12:36:33 2014 From: simon at rabbitmq.com (Simon MacMullen) Date: Tue, 08 Apr 2014 12:36:33 +0100 Subject: [rabbitmq-discuss] Idempotent version of queue.unbind In-Reply-To: References: Message-ID: <5343DF41.8040207@rabbitmq.com> On 08/04/14 12:30, Matwey V. Kornilov wrote: > Hi, Hi. > queue.unbind make an assumption that the binding must exists before it > unbinded. You don't mention which version of RabbitMQ server you're running. > is there a way to have a version of queue.unbind not to making such an > assumption? I mean silently success when binding not exists. ...which matters because queue.unbind should be idempotent as of 3.2.0; if you're seeing a lack of idempotency in a recent version then that's a bug. Cheers, Simon -- Simon MacMullen RabbitMQ, Pivotal From matwey.kornilov at gmail.com Tue Apr 8 12:43:03 2014 From: matwey.kornilov at gmail.com (Matwey V. Kornilov) Date: Tue, 8 Apr 2014 15:43:03 +0400 Subject: [rabbitmq-discuss] Idempotent version of queue.unbind In-Reply-To: <5343DF41.8040207@rabbitmq.com> References: <5343DF41.8040207@rabbitmq.com> Message-ID: 2014-04-08 15:36 GMT+04:00 Simon MacMullen : > You don't mention which version of RabbitMQ server you're running. I am running 3.1.3 and I thought the behaviour is defined by AMQP protocol specification. And this was not clear from docs, so I just tested. > ...which matters because queue.unbind should be idempotent as of 3.2.0; if > you're seeing a lack of idempotency in a recent version then that's a bug. Ok, makes much sense. -- With best regards, Matwey V. Kornilov http://blog.matwey.name xmpp:0x2207 at jabber.ru From simon at rabbitmq.com Tue Apr 8 12:46:15 2014 From: simon at rabbitmq.com (Simon MacMullen) Date: Tue, 08 Apr 2014 12:46:15 +0100 Subject: [rabbitmq-discuss] Idempotent version of queue.unbind In-Reply-To: References: <5343DF41.8040207@rabbitmq.com> Message-ID: <5343E187.7060402@rabbitmq.com> On 08/04/14 12:43, Matwey V. Kornilov wrote: > I am running 3.1.3 and I thought the behaviour is defined by AMQP > protocol specification. It is - but we decided to be different in 3.2.0 anyway since the idempotent behaviour makes more sense. > And this was not clear from docs, so I just tested. It is documented here: http://www.rabbitmq.com/specification.html#method-status-queue.unbind although that is admittedly not the most findable thing on the website. Cheers, Simon -- Simon MacMullen RabbitMQ, Pivotal From pat at munkiisoft.com Tue Apr 8 14:06:57 2014 From: pat at munkiisoft.com (Patrick Long) Date: Tue, 8 Apr 2014 14:06:57 +0100 Subject: [rabbitmq-discuss] Diagnosing errors from SASL log Message-ID: One of our clusters had a problem over the weekend. Ops were upgrading the Load Balancer and restatrted the RabbitMQ service on both nodes at 15:00. Everything seemed to come back up Ok but the following errors started showing later in the day at 17:15 I have copied some of the SASL log below. Any ideas why this would have started happening? =CRASH REPORT==== 5-Apr-2014::19:14:34 === crasher: initial call: gen:init_it/6 pid: <0.9454.1> registered_name: [] exception exit: {function_clause, [{rabbit_channel,handle_info, [{{#Ref<0.0.1.206248>,rabbit at NWPAPPRMQA01}, [{ok,<8069.506.0>,{ok,0,0}}]}, {ch,running,rabbit_framing_amqp_0_9_1,1,<0.9451.1>, <0.9451.1>,<0.9445.1>, <<"">>, {lstate,<0.9453.1>,false,false}, none,1, {[],[]}, {user,<<"guest">>, [administrator], rabbit_auth_backend_internal, {internal_user,<<"guest">>, <<54,19,230,202,176,82,197,60,61,40,94,249,70,83,81, 243,160,53,79,216>>, [administrator]}}, <<"/">>,<<"*aliveness-test*">>,..... =SUPERVISOR REPORT==== 5-Apr-2014::19:14:34 === Supervisor: {<0.9446.1>,amqp_channel_sup_sup} Context: child_terminated Reason: {function_clause, [{rabbit_channel,handle_info, [{{#Ref<0.0.1.206248>,rabbit at NWPAPPRMQA01}, [{ok,<8069.506.0>,{ok,0,0}}]}, {ch,running,rabbit_framing_amqp_0_9_1,1,<0.9451.1>, <0.9451.1>,<0.9445.1>, <<"">>, {lstate,<0.9453.1>,false,false}, none,1, {[],[]}, {user,<<"guest">>, [administrator], rabbit_auth_backend_internal, {internal_user,<<"guest">>, <<54,19,230,202,176,82,197,60,61,40,94, 249,70,83,81,243,160,53,79,216>>, [administrator]}}, <<"/">>,<<"aliveness-test">>, =SUPERVISOR REPORT==== 5-Apr-2014::19:14:34 === Supervisor: {<0.9452.1>,rabbit_channel_sup} Context: shutdown Reason: reached_max_restart_intensity Offender: [{pid,<0.9454.1>}, {name,channel}, {mfargs, {rabbit_channel,start_link, [1,<0.9451.1>,<0.9451.1>,<0.9445.1>, <<"">>, rabbit_framing_amqp_0_9_1, {user,<<"guest">>, [administrator], rabbit_auth_backend_internal, {internal_user,<<"guest">>, <<54,19,230,202,176,82,197,60,61,40,94,249, 70,83,81,243,160,53,79,216>>, [administrator]}}, <<"/">>, [{<<"publisher_confirms">>,bool,true}, {<<"exchange_exchange_bindings">>,bool,true}, {<<"basic.nack">>,bool,true}, {<<"consumer_cancel_notify">>,bool,true}, {<<"connection.blocked">>,bool,true}, {<<"authentication_failure_close">>,bool,true}], <0.9448.1>,<0.9453.1>]}}, {restart_type,intrinsic}, {shutdown,4294967295}, {child_type,worker}] =SUPERVISOR REPORT==== 5-Apr-2014::19:14:34 === Supervisor: {<0.362.0>,mirrored_supervisor} Context: child_terminated Reason: killed Offender: [{pid,<0.9479.1>}, {name,rabbit_mgmt_db}, {mfargs,{rabbit_mgmt_db,start_link,[]}}, {restart_type,permanent}, {shutdown,4294967295}, {child_type,worker}] =SUPERVISOR REPORT==== 5-Apr-2014::19:14:35 === Supervisor: {<0.362.0>,mirrored_supervisor} Context: start_error Reason: {already_started,<8069.464.0>} Offender: [{pid,<0.9479.1>}, {name,rabbit_mgmt_db}, {mfargs,{rabbit_mgmt_db,start_link,[]}}, {restart_type,permanent}, {shutdown,4294967295}, {child_type,worker}] Thanks -- Patrick Long - Munkiisoft Ltd -------------- next part -------------- An HTML attachment was scrubbed... URL: From matthias at rabbitmq.com Tue Apr 8 14:14:00 2014 From: matthias at rabbitmq.com (Matthias Radestock) Date: Tue, 08 Apr 2014 14:14:00 +0100 Subject: [rabbitmq-discuss] Diagnosing errors from SASL log In-Reply-To: References: Message-ID: <5343F618.3060604@rabbitmq.com> On 08/04/14 14:06, Patrick Long wrote: > One of our clusters had a problem over the weekend. Ops were upgrading > the Load Balancer and restatrted the RabbitMQ service on both nodes at > 15:00. Everything seemed to come back up Ok but the following errors > started showing later in the day at 17:15 > > I have copied some of the SASL log below. Any ideas why this would have > started happening? What version of RabbitMQ are you running, and is it consistent across the cluster? Matthias. From pat at munkiisoft.com Tue Apr 8 14:17:27 2014 From: pat at munkiisoft.com (Patrick Long) Date: Tue, 8 Apr 2014 14:17:27 +0100 Subject: [rabbitmq-discuss] Diagnosing errors from SASL log In-Reply-To: <5343F618.3060604@rabbitmq.com> References: <5343F618.3060604@rabbitmq.com> Message-ID: Sorry should have said that at the top. RabbitMQ 3.2.4 Erlang R16B02 Windows Server 2008 R2 Consistent across cluster On 8 April 2014 14:14, Matthias Radestock wrote: > On 08/04/14 14:06, Patrick Long wrote: > >> One of our clusters had a problem over the weekend. Ops were upgrading >> the Load Balancer and restatrted the RabbitMQ service on both nodes at >> 15:00. Everything seemed to come back up Ok but the following errors >> started showing later in the day at 17:15 >> >> I have copied some of the SASL log below. Any ideas why this would have >> started happening? >> > > What version of RabbitMQ are you running, and is it consistent across the > cluster? > > Matthias. > -- Patrick Long - Munkiisoft Ltd -------------- next part -------------- An HTML attachment was scrubbed... URL: From aleroux at splio.fr Tue Apr 8 10:17:09 2014 From: aleroux at splio.fr (Amaury LEROUX DE LENS) Date: Tue, 8 Apr 2014 11:17:09 +0200 Subject: [rabbitmq-discuss] Unable to start RabbitMQ after upgrading to 3.3.0 Message-ID: Hi, I'm using RabbitMQ on MacOSX, installed via HomeBrew and since i upgraded to 3.3.0 i'm not able to start RabbitMQ again. I just got this error report in log files. Any help would be appreciated. Thanks, Amaury =INFO REPORT==== 8-Apr-2014::11:09:05 === > Starting RabbitMQ 3.3.0 on Erlang R16B02 > Copyright (C) 2007-2013 GoPivotal, Inc. > Licensed under the MPL. See http://www.rabbitmq.com/ > =INFO REPORT==== 8-Apr-2014::11:09:05 === > node : rabbit at localhost > home dir : /var/root > config file(s) : (none) > cookie hash : 5IMb6fXQduv+drM042y32A== > log : /usr/local/var/log/rabbitmq/rabbit at localhost.log > sasl log : /usr/local/var/log/rabbitmq/rabbit at localhost-sasl.log > database dir : /usr/local/var/lib/rabbitmq/mnesia/rabbit at localhost > =INFO REPORT==== 8-Apr-2014::11:09:06 === > Limiting to approx 156 file handles (138 sockets) > =INFO REPORT==== 8-Apr-2014::11:09:06 === > Memory limit set to 2808MB of 7021MB total. > =INFO REPORT==== 8-Apr-2014::11:09:06 === > Disk free limit set to 50MB > =INFO REPORT==== 8-Apr-2014::11:09:06 === > msg_store_transient: using rabbit_msg_store_ets_index to provide index > =INFO REPORT==== 8-Apr-2014::11:09:06 === > msg_store_persistent: using rabbit_msg_store_ets_index to provide index > =INFO REPORT==== 8-Apr-2014::11:09:06 === > Error description: > {badmatch,{error,not_found}} > Log files (may contain more information): > /usr/local/var/log/rabbitmq/rabbit at localhost.log > /usr/local/var/log/rabbitmq/rabbit at localhost-sasl.log > Stack trace: > [{rabbit_binding,recover_semi_durable_route,3, > [{file,"src/rabbit_binding.erl"},{line,127}]}, > {rabbit_binding,'-recover/2-lc$^0/1-0-',3, > [{file,"src/rabbit_binding.erl"},{line,117}]}, > > {rabbit_binding,recover,2,[{file,"src/rabbit_binding.erl"},{line,117}]}, > {rabbit,recover,0,[{file,"src/rabbit.erl"},{line,620}]}, > {rabbit,'-run_boot_step/1-lc$^1/1-1-',1, > [{file,"src/rabbit.erl"},{line,505}]}, > {rabbit,run_boot_step,1,[{file,"src/rabbit.erl"},{line,504}]}, > {rabbit,'-start/2-lc$^0/1-0-',1,[{file,"src/rabbit.erl"},{line,471}]}, > {rabbit,start,2,[{file,"src/rabbit.erl"},{line,471}]}] > > =INFO REPORT==== 8-Apr-2014::11:09:07 === > Error description: > {could_not_start,rabbit, > {bad_return, > {{rabbit,start,[normal,[]]}, > {'EXIT', > > {rabbit,failure_during_boot,{badmatch,{error,not_found}}}}}}} > Log files (may contain more information): > /usr/local/var/log/rabbitmq/rabbit at localhost.log > /usr/local/var/log/rabbitmq/rabbit at localhost-sasl.log Amaury Leroux de Lens -------------- next part -------------- An HTML attachment was scrubbed... URL: From trash.collector at seyb.com Tue Apr 8 10:36:53 2014 From: trash.collector at seyb.com (Dave S) Date: Tue, 8 Apr 2014 02:36:53 -0700 (PDT) Subject: [rabbitmq-discuss] Testing Lyra recovery with the java client best practices ? Message-ID: <7e0a49e6-4a4c-45c5-b0e2-48a1e6d93102@googlegroups.com> Hello All I was wondering if any one has had much success using Lyra https://github.com/jhalterman/lyra >From my testing so far it appears to do exactly what it says it will do However My problem is that when I try to use the recovered consumer the same shutdown exception is thrown that cause the recovery in the first place. My simple test case is to kill the connection using the web management console after sending a number of messages. There must be something here I am not understand correctly. What I would expect is that after the connection and consumer is recovered any un acked messages would be republished but no the forced disconnect that caused the recovery in the first place. Is there a way to over come this or is forcefully killing the connection not a valid recovery test case ? Here is test code import com.rabbitmq.client.Channel; import com.rabbitmq.client.ConnectionFactory; import com.rabbitmq.client.QueueingConsumer; import net.jodah.lyra.ConnectionOptions; import net.jodah.lyra.Connections; import net.jodah.lyra.config.Config; import net.jodah.lyra.config.ConfigurableConnection; import net.jodah.lyra.config.RecoveryPolicies; public class RecoveryTest { private static RecoveryTest test; private String EXCHANGE_NAME = "ORDER_UPDATES"; private String EXCHANGE_HOST = "localhost"; private String EXCHANGE_USER = "recover"; private String EXCHANGE_PASS = "recover"; private String QUEUE = "TEST_UPDATES"; private String TOPIC = "*.#"; private final Config config = new Config().withRecoveryPolicy(RecoveryPolicies.recoverAlways());; private ConnectionOptions options; private Channel channel; private ConnectionFactory factory; private ConfigurableConnection RecoveryConnection; private QueueingConsumer consumer; public static void main(String[] args) throws Exception { test = new RecoveryTest(); test.run(); } public RecoveryTest() throws Exception { options = new ConnectionOptions().withAddresses(EXCHANGE_HOST); factory = options.getConnectionFactory(); factory.setUsername(EXCHANGE_USER); factory.setPassword(EXCHANGE_PASS); RecoveryConnection = Connections.create(options, config); channel = RecoveryConnection.createChannel(); channel.queueDeclare(QUEUE, true, false, false, null); channel.queueBind(QUEUE, EXCHANGE_NAME, TOPIC); consumer = new QueueingConsumer(channel); boolean autoAck = true; channel.basicConsume(QUEUE, autoAck, consumer); } public void run() { QueueingConsumer.Delivery delivery = null; System.out.println("Starting Test Run"); while(true){ try { delivery = consumer.nextDelivery(); } catch (Exception ex) { System.out.println("Somthing went wrong ..." + ex.getLocalizedMessage()); } byte [] msg = delivery.getBody(); System.out.println(String.valueOf(new String(msg))); try { System.out.println("Lets take a rest"); Thread.sleep(8000); } catch (InterruptedException ex) { System.out.println("We have been intrupted"); } } } } [main] INFO net.jodah.lyra.internal.ConnectionHandler - Creating connection cxn-1 to [localhost] [main] INFO net.jodah.lyra.internal.ConnectionHandler - Created connection cxn-1 to amqp://127.0.0.1:5672/ [main] INFO net.jodah.lyra.internal.ConnectionHandler - Created channel-1 on cxn-1 [main] INFO net.jodah.lyra.internal.ChannelHandler - Created consumer-amq.ctag-vUb0yGdYSY3Cq6eJEARsvA of TEST_UPDATES via channel-1 on cxn-1 Starting Test Run hello Lets take a rest hello Lets take a rest hello again Lets take a rest This is a new message Lets take a rest [AMQP Connection 127.0.0.1:5672] ERROR net.jodah.lyra.internal.ChannelHandler - Channel channel-1 on cxn-1 was closed unexpectedly Somthing went wrong ...connection error; reason: {#method(reply-code=320, reply-text=CONNECTION_FORCED - Closed via management plugin, class-id=0, method-id=0), null, ""} This is a new message Lets take a rest [AMQP Connection 127.0.0.1:5672] ERROR net.jodah.lyra.internal.ConnectionHandler - Connection cxn-1 was closed unexpectedly [lyra-recovery-1] INFO net.jodah.lyra.internal.ConnectionHandler - Recovering connection cxn-1 to [localhost] [lyra-recovery-1] INFO net.jodah.lyra.internal.ConnectionHandler - Recovered connection cxn-1 to amqp://127.0.0.1:5672/ [lyra-recovery-1] INFO net.jodah.lyra.internal.ConnectionHandler - Recovering queue binding from ORDER_UPDATES to TEST_UPDATES with *.# via cxn-1 [lyra-recovery-1] INFO net.jodah.lyra.internal.ChannelHandler - Recovering channel-1 on cxn-1 [lyra-recovery-1] INFO net.jodah.lyra.internal.ChannelHandler - Recovered channel-1 on cxn-1 [lyra-recovery-1] INFO net.jodah.lyra.internal.ChannelHandler - Recovering consumer-amq.ctag-vUb0yGdYSY3Cq6eJEARsvA of TEST_UPDATES via channel-1 on cxn-1 Somthing went wrong ...connection error; reason: {#method(reply-code=320, reply-text=CONNECTION_FORCED - Closed via management plugin, class-id=0, method-id=0), null, ""} This is a new message Lets take a rest Somthing went wrong ...connection error; reason: {#method(reply-code=320, reply-text=CONNECTION_FORCED - Closed via management plugin, class-id=0, method-id=0), null, ""} This is a new message loop for ever ...... -------------- next part -------------- An HTML attachment was scrubbed... URL: From stammailbox at gmail.com Tue Apr 8 16:36:17 2014 From: stammailbox at gmail.com (rails) Date: Tue, 8 Apr 2014 18:36:17 +0300 Subject: [rabbitmq-discuss] surviving a consumer disconnection using DefaultConsumer Message-ID: using java client QueueingConsumer I wrote the following code to assist consumer recovery while (true) { try{ .. create connection .. create channel & queue while (true) { final QueueingConsumer.Delivery delivery = consumer.nextDelivery(); .... } catch { handleException() } } And every time I restarted rabbit it survived cause it created the connection from start. Now, I understand I should be working with DefaultConsumer. So I have this method public void handleDelivery(...) throws IOException { } and my problems are : 1. I cant catch the exception inside and recreate the connection 2. This only recovers when the ack fails. What if it fails in another stage of the process? How should I recover these fails? 3. Thanks. -------------- next part -------------- An HTML attachment was scrubbed... URL: From simon at rabbitmq.com Tue Apr 8 17:32:36 2014 From: simon at rabbitmq.com (Simon MacMullen) Date: Tue, 08 Apr 2014 17:32:36 +0100 Subject: [rabbitmq-discuss] Heartbleed SSL vulnerability does not affect RabbitMQ Message-ID: <534424A4.50600@rabbitmq.com> Just in case people are worrying about this: Erlang uses OpenSSL for crypto functions but does *not* use OpenSSL's implementation of the SSL / TLS protocol. So even if you have a bad version of OpenSSL linked to Erlang / RabbitMQ you are still safe. See http://erlang.org/pipermail/erlang-questions/2014-April/078538.html for a bit more detail. Cheers, Simon -- Simon MacMullen RabbitMQ, Pivotal From simon at rabbitmq.com Tue Apr 8 17:53:10 2014 From: simon at rabbitmq.com (Simon MacMullen) Date: Tue, 08 Apr 2014 17:53:10 +0100 Subject: [rabbitmq-discuss] Rabbitrmq thresholds defintions In-Reply-To: References: Message-ID: <53442976.5030105@rabbitmq.com> On 08/04/14 08:27, rails wrote: > Trying to create a complicated scenario I successfully made my dev VM > crash :-) > (Did that by creating two DLX that routed messages between them until > ram exploded) I was going to say "but you shouldn't be able to do that; DLX cycles are detected when the entire cycle is due to TTL expiry". But I'm guessing that you had DLX due to max length instead. Hmm, we should not allow cycles involving that either. Really a cycle should have to involve outside intervention (i.e. nack) in order to be permitted. Filing a bug... Cheers, Simon -- Simon MacMullen RabbitMQ, Pivotal From mklishin at gopivotal.com Tue Apr 8 19:18:56 2014 From: mklishin at gopivotal.com (Michael Klishin) Date: Tue, 8 Apr 2014 22:18:56 +0400 Subject: [rabbitmq-discuss] surviving a consumer disconnection using DefaultConsumer In-Reply-To: References: Message-ID: On 8 April 2014 at 19:38:38, rails (stammailbox at gmail.com) wrote: > > I cant catch the exception inside and recreate the connection > This only recovers when the ack fails. What if it fails in another > stage of the process? How should I recover these fails? What are you trying to recover from? If you want to recover from network failures, you need to use shutdown listeners, not exceptions.? -- MK Software Engineer, Pivotal/RabbitMQ From josephrouphael at gmail.com Tue Apr 8 19:44:11 2014 From: josephrouphael at gmail.com (joseph rouphael) Date: Tue, 8 Apr 2014 21:44:11 +0300 Subject: [rabbitmq-discuss] Performance bottleneck on inter-node connection Message-ID: I am facing a performance bottleneck on the inter-node connection between two nodes in a cluster. Cluster configuration is as follow: Nodes: 2 (node1 and node2) OS: RHEL6 RabbitMQ version: 3.3.0 Erlang version: R16B03 rabbitmq.config: [{rabbit, [ {cluster_nodes, {['rabbit at node1', 'rabbit at node2'], disc}}, {hipe_compile, true}, {delegate_count, 32}, {loopback_users, []} ] }]. Queues are located on node1. When producing/consuming messages of 1KB on the same node (node 1), I am able to achieve up to 100K TPS, even more. If I produce messages on node 2, and consume messages from node 1; I am not able to hit more then 15K TPS. Knowing that the CPU load on the two nodes is very relaxed (90% idle), and no bottleneck on the network bandwidth between the nodes. The same, if I produce on node 1, and consume on node 2; I am not able to hit more then 15K TPS I am suspecting a limitation on the inter-node connection between the nodes. Is there a way to create multiple links between the nodes to scale up? If not possible to create multiple links, Is there a way to optimize the link performance? Is there something else I can do to increase the performance between nodes? Thanks for your help. -------------- next part -------------- An HTML attachment was scrubbed... URL: From kopias.peter at gmail.com Wed Apr 9 00:48:51 2014 From: kopias.peter at gmail.com (Peter Kopias) Date: Wed, 9 Apr 2014 01:48:51 +0200 Subject: [rabbitmq-discuss] Upgrade fail In-Reply-To: <533D5DFB.5080405@rabbitmq.com> References: <533D5DFB.5080405@rabbitmq.com> Message-ID: > > I've tried to upgrade my cluster to 3.3.0. >> > > > > As I've wrote before, I did a clean install, to drop the inherited possibly damaged mnesia files, so the previous installation can not be tested. But of course I've got all the logs. >> This is quite weird. The diagnostics indicate that there actually is some > process up, registered with epmd and listening on the Erlang distribution > port for both node2 and node3. I've seen this quite a few times. Just starting and stopping rabbitmq-server leaves one running epmd process in memory! I've noticed it before, but it looks like it's intended. For example (after the fresh rabbit install): petros at node3:~$ ps aux|grep rabbitmq rabbitmq 1223 0.0 0.0 7496 320 ? S 00:41 0:00 /usr/lib/erlang/erts-5.10.2/bin/epmd -daemon rabbitmq 2228 0.0 0.0 4440 624 ? S 00:42 0:00 /bin/sh /usr/sbin/rabbitmq-server rabbitmq 2268 2.1 0.6 1002616 51564 ? Sl 00:42 0:13 /usr/lib/erlang/erts-5.10.2/bin/beam.smp -W w -K true -A30 -P 1048576 -- -root /usr/lib/erlang -progname erl -- -home /var/lib/rabbitmq -- -pa /usr/lib/rabbitmq/lib/rabbitmq_server-3.3.0/sbin/../ebin -noshell -noinput -s rabbit boot -sname rabbit at node3 -boot start_sasl -config /etc/rabbitmq/rabbitmq -kernel inet_default_connect_options [{nodelay,true}] -sasl errlog_type error -sasl sasl_error_logger false -rabbit error_logger {file,"/var/log/rabbitmq/rabbit at node3.log"} -rabbit sasl_error_logger {file,"/var/log/rabbitmq/rabbit at node3-sasl.log"} -rabbit enabled_plugins_file "/etc/rabbitmq/enabled_plugins" -rabbit plugins_dir "/usr/lib/rabbitmq/lib/rabbitmq_server-3.3.0/sbin/../plugins" -rabbit plugins_expand_dir "/var/lib/rabbitmq/mnesia/rabbit at node3-plugins-expand" -os_mon start_cpu_sup false -os_mon start_disksup false -os_mon start_memsup false -mnesia dir "/var/lib/rabbitmq/mnesia/rabbit at node3" -kernel inet_dist_listen_min 25672 -kernel inet_dist_listen_max 25672 rabbitmq 2368 0.0 0.0 7460 428 ? Ss 00:42 0:00 inet_gethost 4 rabbitmq 2369 0.0 0.0 13788 832 ? S 00:42 0:00 inet_gethost 4 petros 5381 0.0 0.0 11196 948 pts/0 S+ 00:52 0:00 grep --color=auto rabbitmq petros at node3:~$ sudo service rabbitmq-server stop * Stopping message broker rabbitmq-server [ OK ] petros at node3:~$ ps aux|grep rabbitmq rabbitmq 1223 0.0 0.0 7496 320 ? S 00:41 0:00 /usr/lib/erlang/erts-5.10.2/bin/epmd -daemon petros 5505 0.0 0.0 11196 944 pts/0 S+ 00:53 0:00 grep --color=auto rabbitmq petros at node3:~$ So after stopping, one erlang process still runs, who knows why. :) petros at node3:~$ sudo lsof |grep epmd epmd 1223 rabbitmq cwd DIR 8,1 4096 2 / epmd 1223 rabbitmq rtd DIR 8,1 4096 2 / epmd 1223 rabbitmq txt REG 8,1 51856 936647 /usr/lib/erlang/erts-5.10.2/bin/epmd epmd 1223 rabbitmq mem REG 8,1 1853400 394057 /lib/x86_64-linux-gnu/libc-2.17.so epmd 1223 rabbitmq mem REG 8,1 1063328 394030 /lib/x86_64-linux-gnu/libm-2.17.so epmd 1223 rabbitmq mem REG 8,1 149312 394033 /lib/x86_64-linux-gnu/ld-2.17.so epmd 1223 rabbitmq 0r CHR 1,3 0t0 1029 /dev/null epmd 1223 rabbitmq 1w CHR 1,3 0t0 1029 /dev/null epmd 1223 rabbitmq 2w CHR 1,3 0t0 1029 /dev/null epmd 1223 rabbitmq 3u IPv4 10155 0t0 TCP *:epmd (LISTEN) and it's listening too... > > WHY is it waiting for node2, node3, if node1 was the last to stop, and >> it should come online in itself without them? >> > > Very good question. > > I was able to reproduce this precise error message by doing the equivalent > of: > > (stop node1) > rabbitmqctl -n node2 stop_app > rabbitmqctl -n node3 stop_app > (start node1) > > i./e. leaving the Erlang VM running but stopping RabbitMQ (and Mnesia) on > node2 and node3. > > Is there any possibility at all you could have done something like this? > Is there a beam.smp process running on those nodes? > I don't run rabbitmqctl stop_app commands by hand, I've got these things (creating a cluster, changing nodetype) scripted, so no, that's not the case. Even if I run something that needs stop_app, there was a start_app in a second. I am especially puzzled since the nodes have registered 25672 as a > distribution port - prior to 3.3.0 the distribution port would be chosen at > random, or you could configure it. So the presence of 25672 strongly > indicates that those nodes have been upgraded to 3.3.0 and are running an > Erlang VM - with RabbitMQ stopped? I think that the standard apt-get process is doing something like this: service rabbitmq-server stop upgrade service rabbitmq-server start but as we've already identified rabbitmq-server stop does leave an epmd running, as it did with 3.1.3, 3.2.2, 3.2.3, 3.2.4, 3.3.0 so after upgradeing the epmd isn't fresh, it may have previous settings, apps running. > > The cookie is the same, the hostnames are correct. >> > > If I am correct about the diagnosis above then the diagnostics don't > currently handle this situation well; they don't check for the case where > Erlang is running but RabbitMQ isn't. This will get fixed in the next > release. Intensive logging would help too. > > I'm trying to start node2 and node3 to have at least one running but >> they always fail, even if I copied back the previous disk state, and >> upgraded again... >> > > How do they fail? Because it sounds like something is running. Node2 and node3 always exited with the message that they are not the one stopped the last, and if I removed the locks they tried to connect to each other - while repairing the mnesia db, and as the network connect failed (as they were not accepting connections probably because the db repair was on), they quit, leaving the repair in half. So the nodes were not able to communicate because of the db repair (this is a theory only), and as the timeout arrived they quit before the repair would finish. (It would be nice to have a message like "starting repair" and "repair finished", as the log message currently is not clear about the state of the repair, I'm not sure that if the repair ends I'll get a logitem.) ==== cut here ==== At this very moment I have the following situation (after the clean 3.3.0 install on all nodes, and cluster rebuilt): node1,node2,node3 is after an apt-get upgrade, and needs restarting node1 restart done, fine. node3 restart done, fine. node2: sudo reboot, and it's just waiting forever (without the usual conneciton closed by foreign host) root at node2:~# rabbitmqctl status Status of node rabbit at node2 ... Error: unable to connect to node rabbit at node2: nodedown DIAGNOSTICS =========== attempted to contact: [rabbit at node2] rabbit at node2: * rabbit seems not to be running at all * other nodes on node2: [rabbitmqctl13579] current node details: - node name: rabbitmqctl14147 at node2 - home dir: /var/lib/rabbitmq - cookie hash: XSquoZCiCPIfZlTilwy1Dg== .... root at node2:~# ps aux|grep rabbit rabbitmq 1221 0.0 0.0 9416 2372 ? S Apr02 0:10 /usr/lib/erlang/erts-5.10.2/bin/epmd -daemon rabbitmq 2189 0.0 0.0 4440 624 ? S Apr02 0:00 /bin/sh /usr/sbin/rabbitmq-server rabbitmq 2236 2.3 0.6 995960 53324 ? Sl Apr02 211:08 /usr/lib/erlang/erts-5.10.2/bin/beam.smp -W w -K true -A30 -P 1048576 -- -root /usr/lib/erlang -progname erl -- -home /var/lib/rabbitmq -- -pa /usr/lib/rabbitmq/lib/rabbitmq_server-3.3.0/sbin/../ebin -noshell -noinput -s rabbit boot -sname rabbit at node2 -boot start_sasl -config /etc/rabbitmq/rabbitmq -kernel inet_default_connect_options [{nodelay,true}] -sasl errlog_type error -sasl sasl_error_logger false -rabbit error_logger {file,"/var/log/rabbitmq/rabbit at node2.log"} -rabbit sasl_error_logger {file,"/var/log/rabbitmq/rabbit at node2-sasl.log"} -rabbit enabled_plugins_file "/etc/rabbitmq/enabled_plugins" -rabbit plugins_dir "/usr/lib/rabbitmq/lib/rabbitmq_server-3.3.0/sbin/../plugins" -rabbit plugins_expand_dir "/var/lib/rabbitmq/mnesia/rabbit at node2-plugins-expand" -os_mon start_cpu_sup false -os_mon start_disksup false -os_mon start_memsup false -mnesia dir "/var/lib/rabbitmq/mnesia/rabbit at node2" -kernel inet_dist_listen_min 25672 -kernel inet_dist_listen_max 25672 root 13513 0.0 0.0 4440 776 ? S 00:42 0:00 /bin/sh /etc/rc6.d/K20rabbitmq-server stop root 13568 0.0 0.0 4440 680 ? S 00:42 0:00 /bin/sh /usr/sbin/rabbitmqctl stop /var/run/rabbitmq/pid root 13577 0.0 0.0 53204 1652 ? S 00:42 0:00 su rabbitmq -s /bin/sh -c /usr/lib/rabbitmq/bin/rabbitmqctl "stop" "/var/run/rabbitmq/pid" rabbitmq 13578 0.0 0.0 4440 624 ? Ss 00:42 0:00 sh -c /usr/lib/rabbitmq/bin/rabbitmqctl "stop" "/var/run/rabbitmq/pid" rabbitmq 13579 0.1 0.1 437204 14676 ? Sl 00:42 0:01 /usr/lib/erlang/erts-5.10.2/bin/beam.smp -- -root /usr/lib/erlang -progname erl -- -home /var/lib/rabbitmq -- -pa /usr/lib/rabbitmq/lib/rabbitmq_server-3.3.0/sbin/../ebin -noshell -noinput -hidden -sname rabbitmqctl13579 -boot start_clean -s rabbit_control_main -nodename rabbit at node2 -extra stop /var/run/rabbitmq/pid rabbitmq 13611 0.0 0.0 7460 432 ? Ss 00:42 0:00 inet_gethost 4 rabbitmq 13612 0.0 0.0 9560 636 ? S 00:42 0:00 inet_gethost 4 root 14958 0.0 0.0 11196 948 pts/0 S+ 01:00 0:00 grep --color=auto rabbit NOTE that, we have the "K20rabbitmq-server stop" running, we have the rabbitmq processes too, but currently neither rabbitmqctl nor management plugin is accessible, and the reboot process hung. This was the very reason about month ago, we first killed the rabbitmq processes. After that we thought "maybe we corrupted mnesia with the kill, and we don't know why it hang for the first time". Unfortunately as the computer is in a "going down to reboot" state, my only shell is the logged in one (one can't get a new connection while the system is going down).... but by starting sshd manually I can have this node powered up for a while. Log files: the sasl log on node2 is empty (yeeah) shutdown_err and startup_err is empty (whoa) startup_log is normal shutdown_log "Stopping and halting node rabbit at node2 ... " and it has the timestamp of the reboot command node2.log: "=INFO REPORT==== 9-Apr-2014::00:42:31 === Stopping RabbitMQ =INFO REPORT==== 9-Apr-2014::00:42:31 === stopped TCP Listener on [::]:5672 ....a few closed connections.... =INFO REPORT==== 9-Apr-2014::00:42:32 === Halting Erlang VM " so it looks like, that rabbitmq really exited 25 minutes ago, but the stop script won't exit, as the processes still running. Could the problem be the inet_gethost process? Unfortunately I've seen dns problems in our networks, so that could be a problem, but that should not hang the shutdown. The running rabbit processes and their open files are here: http://pastebin.com/mLKf5mEu root at node2:~# pstree init???dbus-daemon ... ??epmd ??rabbitmq-server???beam.smp???38*[{beam.smp}] ??rc???K20rabbitmq-ser???rabbitmqctl???su???sh???beam.smp???inet_gethost???inet_gethost ? ??18*[{beam.smp}] ... So, I'm looking to find the hangup process... straceing and killing them. The first try is the gethosts: root at node2:~# strace -p 13612 Process 13612 attached read(0, .... ^CProcess 13612 detached root at node2:~# strace -p 13611 Process 13611 attached select(4, [0 3], NULL, NULL, NULL .... ^CProcess 13611 detached root at node2:~# kill 13612 root at node2:~# kill 13611 weee, after this second kill we've got the inet_gethost respawned, so the beam.smp (pid 13579) still works (but it should have already quit :) ) We know that some from these are waiting: ??rc(13415)???K20rabbitmq-ser(13513)???rabbitmqctl(13568)???su(13577)???sh(13578)???beam.smp(13579)???inet_gethost(16346) ? ??{beam.smp}(13591) ? ??{beam.smp}(13592) ? ??{beam.smp}(13593) ... ? ??{beam.smp}(13608) Any ideas? How can I debug this? Specific questions? I will leave this system in this state for now, to allow us to diagnose the situation. Unfortunately I don't know erlang, but I can try anything if I can help solve these hangups. As I've said this is the same behaviour as 3.1.3, 3.2.2, 3.2.3 and so on worked, they sometimes just hung, then we killed them and had that bad feeling in our hearts, that we may broke the db. I won't kill this time. :) If you can "remote control" me, I may even try to erlang debug the processes, to guess what they're waiting for. Thank you for your help, I hope these hangups could get eliminated. Have a nice day, Peter -------------- next part -------------- An HTML attachment was scrubbed... URL: From kopias.peter at gmail.com Wed Apr 9 00:51:52 2014 From: kopias.peter at gmail.com (Peter Kopias) Date: Wed, 9 Apr 2014 01:51:52 +0200 Subject: [rabbitmq-discuss] Encryption between RabbitMQ cluster nodes In-Reply-To: <37ad40ba-8654-42cb-a627-4aee542afbbc@googlegroups.com> References: <37ad40ba-8654-42cb-a627-4aee542afbbc@googlegroups.com> Message-ID: You may try to use ssh tunnels. Not much simpler to set up though. :) On Tue, Apr 1, 2014 at 2:21 AM, towen wrote: > I was wondering if it was possible to encrypt the messages between > rabbitmq cluster nodes. From what I understand, by default they listen to > 4369, but communicate via clear text. I was wondering if it was possible to > encrypt this via SSL, TLS, etc. Thanks! > > _______________________________________________ > rabbitmq-discuss mailing list > rabbitmq-discuss at lists.rabbitmq.com > https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From srinath.c at gmail.com Wed Apr 9 02:33:58 2014 From: srinath.c at gmail.com (Srinath C) Date: Wed, 9 Apr 2014 07:03:58 +0530 Subject: [rabbitmq-discuss] Troubleshooting handleCancel() Message-ID: Hi, I'm looking for help in troubleshooting a scenario where my consumers stopped processing messages from the queue they were listening on. *Setup*: A 2 node RabbitMQ cluster running 3.2.4. Both are disc nodes. A topic E (durable=true, auto-delete=false) A queue Q (auto-delete=true, exclusive=false, durable=false) Q binds to E using binding key as # C the creator & consumer of Q is running on a JVM using Lyra to handle recovery of connections. *Scenario*: One of the rabbitmq nodes were brought down for testing. It was observed that no more messages were being consumed from the exchange. (This is not always reproducible using the same steps) *Observations*: Q existed on the remaining rabbitmq node Q was not bound to E (hence no message were consumed) *Question*: Since the consumer C is using Lyra, it would have re-created the queue and bindings to exchanges in case of connection failures (it has been configured to do so). This has been tested several times and it works. But in this particular case, there were no logs from Lyra indicating a connection failure and recovery. Indicating that the consumer was connected to the node that was not brought down. The only details in the consumer logs were a "handleCancel" called on the consumer. What are the scenarios where this is called? I can simulate this scenario by explicitly logging into the console and deleting the queue or the exchange.... Thanks in advance, Srinath. -------------- next part -------------- An HTML attachment was scrubbed... URL: From pmaisenovich at blizzard.com Wed Apr 9 06:15:02 2014 From: pmaisenovich at blizzard.com (Pavel) Date: Tue, 8 Apr 2014 22:15:02 -0700 (PDT) Subject: [rabbitmq-discuss] Possible memory leak in the management plugin In-Reply-To: <520CB29A.1060304@rabbitmq.com> References: <8a1af6ed-c599-42a3-85c3-c5dc22a99bd8@googlegroups.com> <520CB29A.1060304@rabbitmq.com> Message-ID: <1397020502914-34663.post@n5.nabble.com> Hello, We are running RabbitMQ 3.1.5 on Erlang R15B. Last week we had a couple incidents, where RabbitMQ would go from stable ~1Gb RAM to ~10Gb in ~10 mins and keep growing up to the high watermark. It was quickly identified that mgmt_db is a culprit, so we disabled management plugin and attempted to reproduce the issue in the lab. Although the lab test wasn't exactly representing the production setup, we managed to achieve similar behavior with rapid unbounded growth of mgmt_db using the following scenario: - declare 1000 exchanges with 1 queue (maxlength=5) bound to each one - run a single threaded script that will - create 1 connection with 1000 channels - publish 1000 messages (1 per channel) to each of exchanges sequentially Here is the output for the rabbit_mgmt_db process after running test above for about a minute: https://gist.github.com/maisenovich/10226474 I'm very new to Erlang, so couple questions about this output: Q1: There are three ets tables (as far as I understand) that have million+ items and keep growing rapidly (5754973, 5759070 and 5763167). What are those? Q2: Total memory used by ets storage (~120Mb) is far less than total memory used by rabbit_mgmt_db process (~3.5Gb). Is there any way to inspect what consumes the most of the memory? It was noticed also, that in the configuration described above (1000 exchanges x 1 queue each) stats events for publishing channels become very big. It also seems that increasing collect_statistics_interval from default 5 seconds to 60 seconds helps - memory still goes up after first iteration, but then stays within ~2Gb (test was running for at least 1 hour with no crash). Here is the output of fprof analysis for ~5 seconds of the rabbit_mgmt_db process during the test at a slower publish rate: https://gist.github.com/maisenovich/10226881 Q3: Is there any better way to profile Erlang/Rabbit? Attempts to use fprof under load that is causing the crash didn't work so far. We are suspecting that the issue is caused by massive channel_stats events which at the high enough rate causing Erlang GC to start falling behind. The function invocation below (from linked profile) is particularly raising questions: {[{{gb_trees,update,3}, 2892, 138.949, 43.548}, {{gb_trees,update_1,3}, 19243, 0.000, 94.664}], { {gb_trees,update_1,3}, 22135, 138.949, 138.212}, % [{garbage_collect, 60, 0.737, 0.737}, {{gb_trees,update_1,3}, 19243, 0.000, 94.664}]}. Q4: Any better theories on what might be causing the unbounded memory growth? Any GC tuning you would recommend to attempt? Thank you! -- View this message in context: http://rabbitmq.1065348.n5.nabble.com/Possible-memory-leak-in-the-management-plugin-tp27414p34663.html Sent from the RabbitMQ mailing list archive at Nabble.com. From mklishin at gopivotal.com Wed Apr 9 08:56:39 2014 From: mklishin at gopivotal.com (Michael Klishin) Date: Wed, 9 Apr 2014 11:56:39 +0400 Subject: [rabbitmq-discuss] Troubleshooting handleCancel() In-Reply-To: References: Message-ID: ?On 9 April 2014 at 05:36:42, Srinath C (srinath.c at gmail.com) wrote: > The only details in the consumer logs were a "handleCancel" called > on the consumer. What are the scenarios where this is called? > I can simulate this scenario by explicitly logging into the console > and deleting the queue or the exchange.... 1. When the queue a consumer is on is deleted 2. (prior to 3.3.0) when queue master node fails Deleting an exchange cannot possibly result in basic.cancel being sent to a consumer. -- MK Software Engineer, Pivotal/RabbitMQ From mklishin at gopivotal.com Wed Apr 9 08:57:56 2014 From: mklishin at gopivotal.com (Michael Klishin) Date: Wed, 9 Apr 2014 11:57:56 +0400 Subject: [rabbitmq-discuss] Troubleshooting handleCancel() In-Reply-To: References: Message-ID: On 9 April 2014 at 05:36:42, Srinath C (srinath.c at gmail.com) wrote: > > Scenario: > One of the rabbitmq nodes were brought down for testing. > It was observed that no more messages were being consumed from > the exchange. What?s in RabbitMQ log? Messages are not consumed from exchanges. Unlike queues, exchanges and bindings have no masters in the cluster, they are always accessible from every node. You are not specifying what node your publisher and consumer were connected to at the time a node was shut down.? -- MK Software Engineer, Pivotal/RabbitMQ From stammailbox at gmail.com Wed Apr 9 08:58:29 2014 From: stammailbox at gmail.com (rails) Date: Wed, 9 Apr 2014 10:58:29 +0300 Subject: [rabbitmq-discuss] surviving a consumer disconnection using DefaultConsumer In-Reply-To: References: Message-ID: Hi, Did you mean DefaultConsumer.handleShutdownSignal? I just found that in the new 3.3.0 version the following methods were added connectionFactory.setAutomaticRecoveryEnabled(true); connectionFactory.setTopologyRecoveryEnabled(true); Just what I needed. Thanks. On Tue, Apr 8, 2014 at 9:18 PM, Michael Klishin wrote: > On 8 April 2014 at 19:38:38, rails (stammailbox at gmail.com) wrote: > > > I cant catch the exception inside and recreate the connection > > This only recovers when the ack fails. What if it fails in another > > stage of the process? How should I recover these fails? > > What are you trying to recover from? If you want to recover from network > failures, > you need to use shutdown listeners, not exceptions. > -- > MK > > Software Engineer, Pivotal/RabbitMQ > _______________________________________________ > rabbitmq-discuss mailing list > rabbitmq-discuss at lists.rabbitmq.com > https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mklishin at gopivotal.com Wed Apr 9 09:01:41 2014 From: mklishin at gopivotal.com (Michael Klishin) Date: Wed, 9 Apr 2014 12:01:41 +0400 Subject: [rabbitmq-discuss] Troubleshooting handleCancel() In-Reply-To: References: Message-ID: On 9 April 2014 at 05:36:42, Srinath C (srinath.c at gmail.com) wrote: > > Scenario: > One of the rabbitmq nodes were brought down for testing. > It was observed that no more messages were being consumed Note that RabbitMQ 3.3.0 will automatically re-register consumers on a queue that had master failure. So I suggest giving your scenario a try with 3.3.? -- MK Software Engineer, Pivotal/RabbitMQ From mklishin at gopivotal.com Wed Apr 9 09:18:18 2014 From: mklishin at gopivotal.com (Michael Klishin) Date: Wed, 9 Apr 2014 12:18:18 +0400 Subject: [rabbitmq-discuss] surviving a consumer disconnection using DefaultConsumer In-Reply-To: References: Message-ID: On 9 April 2014 at 12:01:41, rails (stammailbox at gmail.com) wrote: > > Did you mean DefaultConsumer.handleShutdownSignal? See Shutdown Protocol in [1].? > I just found that in the new 3.3.0 version the following methods > were added > connectionFactory.setAutomaticRecoveryEnabled(true); > connectionFactory.setTopologyRecoveryEnabled(true); They are not just for consumer recovery, although it?s recommended that you use this feature or Lyra instead of rolling you own. See?Automatic Recovery From Network Failures in [1]. 1.?http://rabbitmq.com/api-guide.html -- MK Software Engineer, Pivotal/RabbitMQ From carl.hoerberg at gmail.com Wed Apr 9 09:56:28 2014 From: carl.hoerberg at gmail.com (carlhoerberg) Date: Wed, 9 Apr 2014 01:56:28 -0700 (PDT) Subject: [rabbitmq-discuss] mirror loggning 3.3.0 Message-ID: <1397033788841-34669.post@n5.nabble.com> Upgraded to a cluster to rabbitmq 3.3.0, and then lines like this popped up in the log: =INFO REPORT==== 8-Apr-2014::14:27:31 === ERROR: "Mirrored ~s: Synchronising ~s: ~p messages to synchronise~n" - ["queue 'myqueue' in vhost 'vhost1'", 0] -- View this message in context: http://rabbitmq.1065348.n5.nabble.com/mirror-loggning-3-3-0-tp34669.html Sent from the RabbitMQ mailing list archive at Nabble.com. From arajure at gmail.com Wed Apr 9 09:54:27 2014 From: arajure at gmail.com (Abhay Rajure) Date: Wed, 9 Apr 2014 08:54:27 +0000 (UTC) Subject: [rabbitmq-discuss] rabbitmq-c memory leak? References: Message-ID: Dushin Fred writes: > > > I have written a simple test program using the SimpleAqmpClient library (which in turn uses the rabbitmq-c > library), which seems to illustrate a memory leak, of sorts. > > The program is a simple main, which pre-allocates a string message, and then calls BasicPublish on a > channel in a loop, all single threaded. Effectively: > > auto envelope = BasicMessage::Create(random_data(message_size)); > for (unsigned i = 0; i < num_messages; ++i) > { > channel->BasicPublish(exchange_name, routing_key, envelope); > } > > The problem is that the memory image of this process seems to grow without bounds. The memory all seems to be > allocated in amqp_pool_alloc (as expected), when calling wait_frame_inner -> amqp_handle_input. The > memory is not orphaned -- i.e., it seems to get cleaned up at shutdown, so it doesn't show up as a leak in a leak > detector tool, such a Instruments/DTrace, or valgrind, or purify, etc. But I don't see where the memory is > every de-allocated, which will spell doom for an application that is designed to never terminate. > > I am attaching a screenshot from an Instruments run, to give some context. The code is here: > > https://github.com/fadushin/sandbox/blob/master/cpp/rabbitmq/sender.cpp > > This was run against RabbitMQ 3.2.4, using rabbitmq-c 0.5.0 and SimpleAmqpClient 2.3, compiled using > Clang 5.0 on OS X.9 (Mavericks). > > Has anyone seen this behavior? Am I missing something in my use of BasicPublish, that is not telling the > rabbitmq-c library to free memory in the "amqp pool"? It doesn't seem to make a difference whether I am > actively pulling messages off the queue from a separate process, or even whether there is a queue bound to > the exchange to which I am publishing messages. (Not that I would expect it to) > > -Fred > > encl. > > > > > I have written a simple test program using the SimpleAqmpClient library (which in turn uses the rabbitmq-c > library), which seems to illustrate a memory leak, of sorts. > > The program is a simple main, which pre-allocates a string message, and then calls BasicPublish on a > channel in a loop, all single threaded. Effectively: > > auto envelope = BasicMessage::Create(random_data(message_size)); > for (unsigned i = 0; i < num_messages; ++i) > { > channel->BasicPublish(exchange_name, routing_key, envelope); > } > > The problem is that the memory image of this process seems to grow without bounds. The memory all seems to be > allocated in amqp_pool_alloc (as expected), when calling wait_frame_inner -> amqp_handle_input. The > memory is not orphaned -- i.e., it seems to get cleaned up at shutdown, so it doesn't show up as a leak in a leak > detector tool, such a Instruments/DTrace, or valgrind, or purify, etc. But I don't see where the memory is > every de-allocated, which will spell doom for an application that is designed to never terminate. > > I am attaching a screenshot from an Instruments run, to give some context. The code is here: > > https://github.com/fadushin/sandbox/blob/master/cpp/rabbitmq/sender.cpp > > This was run against RabbitMQ 3.2.4, using rabbitmq-c 0.5.0 and SimpleAmqpClient 2.3, compiled using > Clang 5.0 on OS X.9 (Mavericks). > > Has anyone seen this behavior? Am I missing something in my use of BasicPublish, that is not telling the > rabbitmq-c library to free memory in the "amqp pool"? It doesn't seem to make a difference whether I am > actively pulling messages off the queue from a separate process, or even whether there is a queue bound to > the exchange to which I am publishing messages. (Not that I would expect it to) > > -Fred > > encl. > > Seems like the issue is in SimpleAmqpClient's ChannelImpl::MaybeReleaseBufferOnChannel() method, it fails to call amqp_maybe_release_buffers_on_channel() because by the time it is called, it can't find channel in m_open_channels(bug?). Try this -- In File SimpleAmqpClient/src/ChannelImpl.cpp, call amqp_maybe_release_buffers_on_channel() in ReturnChannel(). void ChannelImpl::ReturnChannel(amqp_chanel_t channel) { +amqp_maybe_release_buffers_on_channel(m_connection, channel); m_free_channels.push(channel); } -Abhay From carl.hoerberg at gmail.com Wed Apr 9 10:02:02 2014 From: carl.hoerberg at gmail.com (carlhoerberg) Date: Wed, 9 Apr 2014 02:02:02 -0700 (PDT) Subject: [rabbitmq-discuss] HiPE slower Message-ID: <1397034121943-34671.post@n5.nabble.com> Did some performance testing on a c3.4xlarge machine, with and without HiPE: Without HiPE: $ rabbitmq-java-client-bin-3.3.0/runjava.sh com.rabbitmq.examples.PerfTest --uri $URI --time 20 --autoack -x 8 -y 16 sending rate avg: 17382 msg/s recving rate avg: 240834 msg/s With HiPE: $ rabbitmq-java-client-bin-3.3.0/runjava.sh com.rabbitmq.examples.PerfTest --uri $URI --time 20 --autoack -x 8 -y 16 sending rate avg: 5328 msg/s recving rate avg: 55698 msg/s Any idea why HiPE is slower on a 16 core machine? On smaller instances, like m3.medium HiPE gives about 80% _better_ performance than without. -- View this message in context: http://rabbitmq.1065348.n5.nabble.com/HiPE-slower-tp34671.html Sent from the RabbitMQ mailing list archive at Nabble.com. From simon at rabbitmq.com Wed Apr 9 11:33:21 2014 From: simon at rabbitmq.com (Simon MacMullen) Date: Wed, 09 Apr 2014 11:33:21 +0100 Subject: [rabbitmq-discuss] Possible memory leak in the management plugin In-Reply-To: <1397020502914-34663.post@n5.nabble.com> References: <8a1af6ed-c599-42a3-85c3-c5dc22a99bd8@googlegroups.com> <520CB29A.1060304@rabbitmq.com> <1397020502914-34663.post@n5.nabble.com> Message-ID: <534521F1.9030707@rabbitmq.com> On 09/04/14 06:15, Pavel wrote: > Hello, Hello. And first of all, thanks for a very detailed email! > It > was quickly identified that mgmt_db is a culprit, so we disabled management > plugin and attempted to reproduce the issue in the lab. Based on what you've said below, it looks like the fine grained stats are causing the problem, so if you want to re-enable mgmt you might want to add {rabbitmq_management_agent, [{force_fine_statistics, false}]} in your configuration - this will stop mgmt showing message rates but I think it would remove the performance issue you're seeing. > Q1: There are three ets tables (as far as I understand) that have million+ > items and keep growing rapidly (5754973, 5759070 and 5763167). What are > those? Those numbers are just generated IDs, so they will be different for each run of the database. But in your case they are: 5754973 - aggregated_stats 5759070 - aggregated_stats_index 5763167 - old_stats In (very) short, aggregated_stats contains all the stats for things which have history, aggregated_stats_index provides some alternate indexes to provide fast lookups into aggregated_stats, and old_stats retains the (N-1) version of the raw stats emitted by channels (so that we can get the delta to the previous one when aggregating). All of these I would expect to see having a size that is proportional to the number of distinct (channel -> exchange), (channel -> exchange -> queue) and (queue -> channel) publish and deliver events. So with large fanout between channels and exchanges I would expect to see lots of records in these tables. However, I would not expect to see the tables grow indefinitely. They should reach a stable size, and then drop back to (comparatively near) zero once all the channels are closed (they will still contain per-queue, per-exchange, per-vhost records). So are they growing without bound? What happens when you close all the channels? > Q2: Total memory used by ets storage (~120Mb) is far less than total memory > used by rabbit_mgmt_db process (~3.5Gb). Is there any way to inspect what > consumes the most of the memory? That's the weird part. The vast majority of data for the mgmt DB should be in those ETS tables; the process memory should only include the database's state record (which you printed with sys:get_status/1, it's tiny), its stack (also tiny) and any garbage that hasn't yet been collected. Hmm. That last one is an interesting question. What does rabbitmqctl eval 'erlang:garbage_collect(global:whereis_name(rabbit_mgmt_db)).' do to your memory use? The database will be generating heaps (sorry) of garbage in your test; Erlang's GC should be kicking in frequently but I wonder if somehow it is not. > It was noticed also, that in the configuration described above (1000 > exchanges x 1 queue each) stats events for publishing channels become very > big. Yes, again they contain data for each exchange / queue the channel publishes to. > It also seems that increasing collect_statistics_interval from default > 5 seconds to 60 seconds helps - memory still goes up after first iteration, > but then stays within ~2Gb (test was running for at least 1 hour with no > crash). That makes sense. > Here is the output of fprof analysis for ~5 seconds of the rabbit_mgmt_db > process during the test at a slower publish rate: > https://gist.github.com/maisenovich/10226881 > > Q3: Is there any better way to profile Erlang/Rabbit? Attempts to use fprof > under load that is causing the crash didn't work so far. Yeah, fprof has a huge performance impact since it is of the "trace every function call" school. I am not aware of a sampling profiler for Erlang I'm afraid. > We are suspecting that the issue is caused by massive channel_stats events > which at the high enough rate causing Erlang GC to start falling behind. The > function invocation below (from linked profile) is particularly raising > questions: That seems sort of plausible. But Erlang does not GC like Java - each process has its own heap and the GC takes place "within" that process, it's not a separate thread or anything, so it shouldn?t be able to fall behind just because a process is busy. I am still very suspicious of GC in your case though, forcing a manual GC as above would be very helpful. > Q4: Any better theories on what might be causing the unbounded memory > growth? Any GC tuning you would recommend to attempt? Not yet. I'd like to verify that GC is the problem first, then we can look at how to do it better. Cheers, Simon -- Simon MacMullen RabbitMQ, Pivotal From michael.s.klishin at gmail.com Wed Apr 9 11:45:23 2014 From: michael.s.klishin at gmail.com (Michael Klishin) Date: Wed, 9 Apr 2014 14:45:23 +0400 Subject: [rabbitmq-discuss] ANN Langohr 2.9.0 is released Message-ID: Langohr [1] is a small, feature complete Clojure client for RabbitMQ. Release notes: http://blog.clojurewerkz.org/blog/2014/04/09/langohr-2-dot-9-0-is-released/ 1. http://clojurerabbitmq.info -- MK http://github.com/michaelklishin http://twitter.com/michaelklishin -------------- next part -------------- An HTML attachment was scrubbed... URL: From mklishin at gopivotal.com Wed Apr 9 11:46:45 2014 From: mklishin at gopivotal.com (Michael Klishin) Date: Wed, 9 Apr 2014 14:46:45 +0400 Subject: [rabbitmq-discuss] ANN Bunny 1.1.9 is released Message-ID: Bunny [1] is a Ruby RabbitMQ client that focuses on ease of use. Release notes: http://blog.rubyrabbitmq.info/blog/2014/04/09/bunny-1-dot-1-9-is-released/ Note that there will be no more 1.1.x releases. Please move to 1.2.x (will be announced shortly).? 1. http://rubybunny.info -- MK Software Engineer, Pivotal/RabbitMQ From mklishin at gopivotal.com Wed Apr 9 11:48:22 2014 From: mklishin at gopivotal.com (Michael Klishin) Date: Wed, 9 Apr 2014 14:48:22 +0400 Subject: [rabbitmq-discuss] ANN Bunny 1.2.0 is released Message-ID: Bunny [1] is a Ruby RabbitMQ client that focuses on ease of use.? Release notes:? http://blog.rubyrabbitmq.info/blog/2014/04/09/bunny-1-dot-2-0-is-released/ Note that there will be no more 1.1.x releases. Please move to? 1.2.0, it is a drop-in replacement. 1. http://rubybunny.info? -- MK Software Engineer, Pivotal/RabbitMQ From simon at rabbitmq.com Wed Apr 9 11:49:42 2014 From: simon at rabbitmq.com (Simon MacMullen) Date: Wed, 09 Apr 2014 11:49:42 +0100 Subject: [rabbitmq-discuss] mirror loggning 3.3.0 In-Reply-To: <1397033788841-34669.post@n5.nabble.com> References: <1397033788841-34669.post@n5.nabble.com> Message-ID: <534525C6.8080600@rabbitmq.com> On 09/04/14 09:56, carlhoerberg wrote: > Upgraded to a cluster to rabbitmq 3.3.0, and then lines like this popped up > in the log: > > =INFO REPORT==== 8-Apr-2014::14:27:31 === > ERROR: "Mirrored ~s: Synchronising ~s: ~p messages to synchronise~n" - > ["queue 'myqueue' in vhost 'vhost1'", 0] Ah thanks, a cosmetic issue but one that should be easy to fix. Cheers, Simon -- Simon MacMullen RabbitMQ, Pivotal From stammailbox at gmail.com Wed Apr 9 11:56:09 2014 From: stammailbox at gmail.com (rails) Date: Wed, 9 Apr 2014 13:56:09 +0300 Subject: [rabbitmq-discuss] surviving a consumer disconnection using DefaultConsumer In-Reply-To: References: Message-ID: Thanks Michael. Please explain: 1. "not just for consumer recovery," Did you mean by that topology recovery or publisher recovery? What else is supported? 2. What are the benefits of lyra now when the auto recovery feature exists? I have been having a hard time understand Lyra project (code in the lyra readme at github doesnt actually compile nor self explained) Thanks. On Wed, Apr 9, 2014 at 11:18 AM, Michael Klishin wrote: > On 9 April 2014 at 12:01:41, rails (stammailbox at gmail.com) wrote: > > > Did you mean DefaultConsumer.handleShutdownSignal? > > See Shutdown Protocol in [1]. > > > I just found that in the new 3.3.0 version the following methods > > were added > > connectionFactory.setAutomaticRecoveryEnabled(true); > > connectionFactory.setTopologyRecoveryEnabled(true); > > They are not just for consumer recovery, although it's recommended that > you use > this feature or Lyra instead of rolling you own. See Automatic Recovery > From Network Failures > in [1]. > > 1. http://rabbitmq.com/api-guide.html > -- > MK > > Software Engineer, Pivotal/RabbitMQ > _______________________________________________ > rabbitmq-discuss mailing list > rabbitmq-discuss at lists.rabbitmq.com > https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mklishin at gopivotal.com Wed Apr 9 12:04:05 2014 From: mklishin at gopivotal.com (Michael Klishin) Date: Wed, 9 Apr 2014 15:04:05 +0400 Subject: [rabbitmq-discuss] surviving a consumer disconnection using DefaultConsumer In-Reply-To: References: Message-ID: ?On 9 April 2014 at 14:58:24, rails (stammailbox at gmail.com) wrote: > > Please explain: > 1. "not just for consumer recovery," > Did you mean by that topology recovery or publisher recovery? > What else is supported? Automatic recovery recovers connections and channels, what topology recovery does is explained in the docs: http://rabbitmq.com/api-guide.html > 2. What are the benefits of lyra now when the auto recovery feature > exists? > I have been having a hard time understand Lyra project (code in > the lyra readme at github doesnt actually compile nor self explained) Lyra is more flexible and (AFAIK) supports retries for publishers, which the official Java client currently does not. -- MK Software Engineer, Pivotal/RabbitMQ From stammailbox at gmail.com Wed Apr 9 12:07:38 2014 From: stammailbox at gmail.com (rails) Date: Wed, 9 Apr 2014 14:07:38 +0300 Subject: [rabbitmq-discuss] Java client auto recovery - a few questions and notes Message-ID: Hi, Now that the new auto recovery feature is out I want to know if it supports any scenario on the publisher side. what is the best way to call basicPublish if I want it to auto-retry ? (I dont want to loose data that failed publishing) Thanks. * Notes regarding the documentation in api-guide.html 1. in Connection Recovery section factory.setAutomaticRecovery(true) should be factory.setAutomaticRecoveryEnabled(true) 2. in Topology Recovery section a, I dont think the code snippet is correct - it doesnt refer to the topology. rather it calls factory.setAutomaticRecovery(true) b. I think the topology should contain info on what todo when everything was declared durable=true and autodelete=false. -------------- next part -------------- An HTML attachment was scrubbed... URL: From stammailbox at gmail.com Wed Apr 9 12:09:40 2014 From: stammailbox at gmail.com (rails) Date: Wed, 9 Apr 2014 14:09:40 +0300 Subject: [rabbitmq-discuss] surviving a consumer disconnection using DefaultConsumer In-Reply-To: References: Message-ID: Thanks. Just asked it in another thread (didn't want to mix thread topics). should I combine auto-recovery true and lyra client? To your information. Is the Lyra compliant with the new rabbit client version? On Wed, Apr 9, 2014 at 2:04 PM, Michael Klishin wrote: > On 9 April 2014 at 14:58:24, rails (stammailbox at gmail.com) wrote: > > > Please explain: > > 1. "not just for consumer recovery," > > Did you mean by that topology recovery or publisher recovery? > > What else is supported? > > Automatic recovery recovers connections and channels, what topology > recovery > does is explained in the docs: > http://rabbitmq.com/api-guide.html > > > 2. What are the benefits of lyra now when the auto recovery feature > > exists? > > I have been having a hard time understand Lyra project (code in > > the lyra readme at github doesnt actually compile nor self explained) > > Lyra is more flexible and (AFAIK) supports retries for publishers, which > the official Java client currently does not. > -- > MK > > Software Engineer, Pivotal/RabbitMQ > _______________________________________________ > rabbitmq-discuss mailing list > rabbitmq-discuss at lists.rabbitmq.com > https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mklishin at gopivotal.com Wed Apr 9 12:12:40 2014 From: mklishin at gopivotal.com (Michael Klishin) Date: Wed, 9 Apr 2014 15:12:40 +0400 Subject: [rabbitmq-discuss] Java client auto recovery - a few questions and notes In-Reply-To: References: Message-ID: On 9 April 2014 at 15:09:52, rails (stammailbox at gmail.com) wrote: > > Now that the new auto recovery feature is out I want to know if > it supports any scenario on the publisher side. > what is the best way to call basicPublish if I want it to auto-retry > ? Accumulate deliveries and retry manually. Use publisher confirms. Or use Lyra. > b. I think the topology should contain info on what todo when everything > was declared durable=true and autodelete=false. This case is no exception to how topology recovery works in general. I?ll take a look at the typos in the docs.? -- MK Software Engineer, Pivotal/RabbitMQ From mklishin at gopivotal.com Wed Apr 9 12:14:11 2014 From: mklishin at gopivotal.com (Michael Klishin) Date: Wed, 9 Apr 2014 15:14:11 +0400 Subject: [rabbitmq-discuss] surviving a consumer disconnection using DefaultConsumer In-Reply-To: References: Message-ID: ?On 9 April 2014 at 15:12:16, rails (stammailbox at gmail.com) wrote: > > should I combine auto-recovery true and lyra client? They will step on each other?s toes. > To your information. Is the Lyra compliant with the new rabbit > client version? Automatic recovery in the Java client is disabled by default, so Lyra and Spring AMQP should work just fine with it. That?s largely why it is disabled by default. -- MK Software Engineer, Pivotal/RabbitMQ From mklishin at gopivotal.com Wed Apr 9 12:21:13 2014 From: mklishin at gopivotal.com (Michael Klishin) Date: Wed, 9 Apr 2014 15:21:13 +0400 Subject: [rabbitmq-discuss] Java client auto recovery - a few questions and notes In-Reply-To: References: Message-ID: On 9 April 2014 at 15:14:28, Michael Klishin (mklishin at gopivotal.com) wrote: > > I?ll take a look at the typos in the docs. Corrected.? -- MK Software Engineer, Pivotal/RabbitMQ From simon at rabbitmq.com Wed Apr 9 12:43:05 2014 From: simon at rabbitmq.com (Simon MacMullen) Date: Wed, 09 Apr 2014 12:43:05 +0100 Subject: [rabbitmq-discuss] Upgrade fail In-Reply-To: References: <533D5DFB.5080405@rabbitmq.com> Message-ID: <53453249.90808@rabbitmq.com> On 09/04/14 00:48, Peter Kopias wrote: > Just starting and stopping rabbitmq-server leaves one running epmd > process in memory! I've noticed it before, but it looks like it's intended. Yes. The epmd process staying around is not related to anything else you're seeing, that's how epmd is supposed to work. epmd is a very simple daemon which just maps names to port numbers for Erlang (beam) processes. It doesn't contain any other state. > Node2 and node3 always exited with the message that they are not the > one stopped the last, and if I removed the locks they tried to connect > to each other - while repairing the mnesia db, and as the network > connect failed (as they were not accepting connections probably because > the db repair was on), they quit, leaving the repair in half. So the > nodes were not able to communicate because of the db repair (this is a > theory only), and as the timeout arrived they quit before the repair > would finish. (It would be nice to have a message like "starting repair" > and "repair finished", as the log message currently is not clear about > the state of the repair, I'm not sure that if the repair ends I'll get a > logitem.) If you are talking about the upgrades, then "mnesia upgrades: N to apply" marks the start, and "mnesia upgrades: All upgrades applied successfully" marks the end. > node2: sudo reboot, and it's just waiting forever (without the usual > conneciton closed by foreign host) > > root at node2:~# rabbitmqctl status > Status of node rabbit at node2 ... > Error: unable to connect to node rabbit at node2: nodedown > > DIAGNOSTICS > =========== > > attempted to contact: [rabbit at node2] > > rabbit at node2: > * rabbit seems not to be running at all > * other nodes on node2: [rabbitmqctl13579] > NOTE that, we have the "K20rabbitmq-server stop" running, we have the > rabbitmq processes too, but currently neither rabbitmqctl nor management > plugin is accessible, and the reboot process hung. So this is very weird. The beam process is still around, but everything in the shutdown appears to have succeeded. By the time "Halting Erlang VM" appears in the logs RabbitMQ is completely stopped, that log message is literally the last thing we do before telling the VM to stop. > Could the problem be the inet_gethost process? Unfortunately I've seen > dns problems in our networks, so that could be a problem, but that > should not hang the shutdown. No, it really shouldn't. The beam process has already unregistered from epmd (hence "rabbit seems not to be running at all" in the status command above). > The running rabbit processes and their open files are here: > > http://pastebin.com/mLKf5mEu And it seems to have few files open, and to have closed all network sockets. > Any ideas? How can I debug this? > Specific questions? I am not full of ideas here, this looks like more of an issue with Erlang than RabbitMQ. But some things to look into: The beam process (2236 in your case) is the interesting one; it's the thing that should have shut down but hasn't; everything else is pretty much as expected. Unfortunately although it's an Erlang process it has stopped listening for Erlang distribution messages so debugging it is likely to be hard. Having said that, strace might give some clue as to what it's doing, and the small bright side is that so much within that process has already shut down that *anything* it is still doing is a good candidate for what's making it stuck. So what does strace say for it? > Thank you for your help, I hope these hangups could get eliminated. I would hope so. We don't have much to go on though. Cheers, Simon -- Simon MacMullen RabbitMQ, Pivotal From stammailbox at gmail.com Wed Apr 9 13:28:07 2014 From: stammailbox at gmail.com (rails) Date: Wed, 9 Apr 2014 15:28:07 +0300 Subject: [rabbitmq-discuss] surviving a consumer disconnection using DefaultConsumer In-Reply-To: References: Message-ID: Thanks a lot for your help. I will try not to keep it long - just to summarize what I have been through and get your recommendation - I want to create wrappers for the rabbit client in my project that will standard the calls to rabbit - (so it will hide the settings from the other programmers which don't really care about them ) - If I want both to support auto-recovery - I should use the rabbitMQ feature for the consumer and for the producer I should wrap lyra? Thanks. On Wed, Apr 9, 2014 at 2:14 PM, Michael Klishin wrote: > On 9 April 2014 at 15:12:16, rails (stammailbox at gmail.com) wrote: > > > should I combine auto-recovery true and lyra client? > > They will step on each other's toes. > > > To your information. Is the Lyra compliant with the new rabbit > > client version? > > Automatic recovery in the Java client is disabled by default, so Lyra > and Spring AMQP should work just fine with it. That's largely why it is > disabled by default. > -- > MK > > Software Engineer, Pivotal/RabbitMQ > _______________________________________________ > rabbitmq-discuss mailing list > rabbitmq-discuss at lists.rabbitmq.com > https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss > -------------- next part -------------- An HTML attachment was scrubbed... URL: From stammailbox at gmail.com Wed Apr 9 13:46:54 2014 From: stammailbox at gmail.com (rails) Date: Wed, 9 Apr 2014 15:46:54 +0300 Subject: [rabbitmq-discuss] Java client auto recovery - a few questions and notes In-Reply-To: References: Message-ID: Regarding using Lyra for publisher side - There is no publisher documentation in the site. On Wed, Apr 9, 2014 at 2:12 PM, Michael Klishin wrote: > On 9 April 2014 at 15:09:52, rails (stammailbox at gmail.com) wrote: > > > Now that the new auto recovery feature is out I want to know if > > it supports any scenario on the publisher side. > > what is the best way to call basicPublish if I want it to auto-retry > > ? > > Accumulate deliveries and retry manually. Use publisher confirms. > > Or use Lyra. > > > b. I think the topology should contain info on what todo when everything > > was declared durable=true and autodelete=false. > > This case is no exception to how topology recovery works in general. > > I'll take a look at the typos in the docs. > -- > MK > > Software Engineer, Pivotal/RabbitMQ > _______________________________________________ > rabbitmq-discuss mailing list > rabbitmq-discuss at lists.rabbitmq.com > https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss > -------------- next part -------------- An HTML attachment was scrubbed... URL: From simon at rabbitmq.com Wed Apr 9 14:57:17 2014 From: simon at rabbitmq.com (Simon MacMullen) Date: Wed, 09 Apr 2014 14:57:17 +0100 Subject: [rabbitmq-discuss] Removal of the 'impersonator' tag from 3.3.0 In-Reply-To: <533D600A.6040701@rabbitmq.com> References: <533C390D.2060907@rabbitmq.com> <533D600A.6040701@rabbitmq.com> Message-ID: <534551BD.1010804@rabbitmq.com> On 03/04/14 14:20, Simon MacMullen wrote: > On 02/04/14 18:11, Gotthard, Petr wrote: >> In other words, the "escaped" implementation detail was used to implement >> user_id validation in upstream-controlled federations. Does it make >> sense? > > Thanks, that does. And I can't think of a good way to do what you want > without it. Hmm. OK, the 'impersonator' tag is back in nightly builds, and will be in 3.3.1. Cheers, Simon -- Simon MacMullen RabbitMQ, Pivotal From mes65 at cornell.edu Wed Apr 9 09:15:29 2014 From: mes65 at cornell.edu (Michael Sander) Date: Wed, 9 Apr 2014 04:15:29 -0400 Subject: [rabbitmq-discuss] Rabbit MQ Crash In-Reply-To: References: <4da4be68-9e3b-4bba-aaa9-e262df330d6f@googlegroups.com> <5342E86E.4060508@rabbitmq.com> <5343117D.7080405@rabbitmq.com> <53431742.5000008@rabbitmq.com> <5343868A.3060207@rabbitmq.com> Message-ID: Anything I can do to move the ball forward here? This keeps on happening. Michael Sander mes65 at cornell.edu 607-227-9859 On Tue, Apr 8, 2014 at 3:49 AM, Michael Sander wrote: > Unfortunately, I had to restart rabbitmq because it's on a production > machine. Attached is screenshot after the restart. Notice that this time, > free disk space limit is available. > > Please let me know what additional information you would like. If it goes > down again, I will collect it. > > Michael Sander > > > > On Tue, Apr 8, 2014 at 3:42 AM, Michael Sander wrote: > >> Hi Matthias, >> >> I just checked rabbit the second after sending that, and it appears to >> have crashed. Here is some output, that you may find useful. Notice that >> erlang appears to be alive even though rabbitmq is not. >> >> ~$ ps aux|grep rabbit >> rabbitmq 1761 0.0 0.0 10836 160 ? S Apr07 0:01 >> /usr/lib/erlang/erts-5.9.1/bin/epmd -daemon >> 1001 26076 0.0 0.0 6308 600 pts/1 S+ 07:34 0:00 grep >> rabbit >> ~$ ps aux|grep erlan >> rabbitmq 1761 0.0 0.0 10836 160 ? S Apr07 0:01 >> /usr/lib/erlang/erts-5.9.1/bin/epmd -daemon >> 1001 26078 0.0 0.0 6308 600 pts/1 S+ 07:34 0:00 grep >> erlan >> ~$ df -h >> Filesystem Size Used Avail >> Use% Mounted on >> rootfs 9.9G 6.1G 3.3G >> 65% / >> udev 10M 0 10M >> 0% /dev >> tmpfs 181M 128K 181M >> 1% /run >> /dev/disk/by-uuid/36fd30d4-ea87-419f-a6a4-a1a3cf290ff1 9.9G 6.1G 3.3G >> 65% / >> tmpfs 5.0M 0 5.0M >> 0% /run/lock >> tmpfs 362M 0 362M >> 0% /run/shm >> ~$ sudo df -h >> Filesystem Size Used Avail >> Use% Mounted on >> rootfs 9.9G 6.1G 3.3G >> 65% / >> udev 10M 0 10M >> 0% /dev >> tmpfs 181M 128K 181M >> 1% /run >> /dev/disk/by-uuid/36fd30d4-ea87-419f-a6a4-a1a3cf290ff1 9.9G 6.1G 3.3G >> 65% / >> tmpfs 5.0M 0 5.0M >> 0% /run/lock >> tmpfs 362M 0 362M >> 0% /run/shm >> ~$ tail -n 100 /var/log/rabbitmq/rabbit at ocr-proc-2-sasl.log >> ~$ tail -n 100 /var/log/rabbitmq/rabbit at ocr-proc-2-sasl.log.1 >> crasher: >> initial call: rabbit_disk_monitor:init/1 >> pid: <0.19499.0> >> registered_name: [] >> exception exit: unsupported_platform >> in function gen_server:init_it/6 (gen_server.erl, line 320) >> ancestors: [rabbit_disk_monitor_sup,rabbit_sup,<0.157.0>] >> messages: [] >> links: [<0.180.0>] >> dictionary: [] >> trap_exit: false >> status: running >> heap_size: 6765 >> stack_size: 24 >> reductions: 13592 >> neighbours: >> >> =SUPERVISOR REPORT==== 8-Apr-2014::00:38:08 === >> Supervisor: {local,rabbit_disk_monitor_sup} >> Context: start_error >> Reason: unsupported_platform >> Offender: [{pid,{restarting,<0.5000.0>}}, >> {name,rabbit_disk_monitor}, >> {mfargs,{rabbit_disk_monitor,start_link,[50000000]}}, >> {restart_type,transient}, >> {shutdown,4294967295}, >> {child_type,worker}] >> >> >> =CRASH REPORT==== 8-Apr-2014::00:38:08 === >> crasher: >> initial call: rabbit_disk_monitor:init/1 >> pid: <0.19502.0> >> registered_name: [] >> exception exit: unsupported_platform >> in function gen_server:init_it/6 (gen_server.erl, line 320) >> ancestors: [rabbit_disk_monitor_sup,rabbit_sup,<0.157.0>] >> messages: [] >> links: [<0.180.0>] >> dictionary: [] >> trap_exit: false >> status: running >> heap_size: 6765 >> stack_size: 24 >> reductions: 13592 >> neighbours: >> >> =SUPERVISOR REPORT==== 8-Apr-2014::00:38:08 === >> Supervisor: {local,rabbit_disk_monitor_sup} >> Context: start_error >> Reason: unsupported_platform >> Offender: [{pid,{restarting,<0.5000.0>}}, >> {name,rabbit_disk_monitor}, >> {mfargs,{rabbit_disk_monitor,start_link,[50000000]}}, >> {restart_type,transient}, >> {shutdown,4294967295}, >> {child_type,worker}] >> >> >> =CRASH REPORT==== 8-Apr-2014::00:38:08 === >> crasher: >> initial call: rabbit_disk_monitor:init/1 >> pid: <0.19505.0> >> registered_name: [] >> exception exit: unsupported_platform >> in function gen_server:init_it/6 (gen_server.erl, line 320) >> ancestors: [rabbit_disk_monitor_sup,rabbit_sup,<0.157.0>] >> messages: [] >> links: [<0.180.0>] >> dictionary: [] >> trap_exit: false >> status: running >> heap_size: 6765 >> stack_size: 24 >> reductions: 13592 >> neighbours: >> >> =SUPERVISOR REPORT==== 8-Apr-2014::00:38:08 === >> Supervisor: {local,rabbit_disk_monitor_sup} >> Context: start_error >> Reason: unsupported_platform >> Offender: [{pid,{restarting,<0.5000.0>}}, >> {name,rabbit_disk_monitor}, >> {mfargs,{rabbit_disk_monitor,start_link,[50000000]}}, >> {restart_type,transient}, >> {shutdown,4294967295}, >> {child_type,worker}] >> >> >> =SUPERVISOR REPORT==== 8-Apr-2014::00:38:08 === >> Supervisor: {local,rabbit_disk_monitor_sup} >> Context: shutdown >> Reason: reached_max_restart_intensity >> Offender: [{pid,{restarting,<0.5000.0>}}, >> {name,rabbit_disk_monitor}, >> {mfargs,{rabbit_disk_monitor,start_link,[50000000]}}, >> {restart_type,transient}, >> {shutdown,4294967295}, >> {child_type,worker}] >> ~$ tail -n 100 /var/log/rabbitmq/rabbit at ocr-proc-2.log >> =WARNING REPORT==== 8-Apr-2014::07:29:47 === >> closing AMQP connection <0.20361.1> (127.0.0.1:48568 -> 127.0.0.1:5672): >> connection_closed_abruptly >> >> =WARNING REPORT==== 8-Apr-2014::07:29:47 === >> closing AMQP connection <0.20392.1> (127.0.0.1:48586 -> 127.0.0.1:5672): >> connection_closed_abruptly >> >> =WARNING REPORT==== 8-Apr-2014::07:29:49 === >> closing AMQP connection <0.20401.1> (127.0.0.1:48589 -> 127.0.0.1:5672): >> connection_closed_abruptly >> >> =WARNING REPORT==== 8-Apr-2014::07:29:50 === >> closing AMQP connection <0.22633.1> (127.0.0.1:50329 -> 127.0.0.1:5672): >> connection_closed_abruptly >> >> =WARNING REPORT==== 8-Apr-2014::07:29:51 === >> closing AMQP connection <0.16156.1> (127.0.0.1:44692 -> 127.0.0.1:5672): >> connection_closed_abruptly >> >> =INFO REPORT==== 8-Apr-2014::07:30:11 === >> accepting AMQP connection <0.22761.1> (127.0.0.1:50370 -> 127.0.0.1:5672) >> >> =WARNING REPORT==== 8-Apr-2014::07:30:11 === >> closing AMQP connection <0.22608.1> (127.0.0.1:50316 -> 127.0.0.1:5672): >> connection_closed_abruptly >> >> =INFO REPORT==== 8-Apr-2014::07:30:11 === >> accepting AMQP connection <0.22774.1> (127.0.0.1:50371 -> 127.0.0.1:5672) >> >> =INFO REPORT==== 8-Apr-2014::07:30:11 === >> accepting AMQP connection <0.22777.1> (127.0.0.1:50372 -> 127.0.0.1:5672) >> >> =INFO REPORT==== 8-Apr-2014::07:30:26 === >> accepting AMQP connection <0.22796.1> (127.0.0.1:50383 -> 127.0.0.1:5672) >> >> =INFO REPORT==== 8-Apr-2014::07:30:26 === >> accepting AMQP connection <0.22805.1> (127.0.0.1:50384 -> 127.0.0.1:5672) >> >> =INFO REPORT==== 8-Apr-2014::07:30:26 === >> accepting AMQP connection <0.22810.1> (127.0.0.1:50385 -> 127.0.0.1:5672) >> >> =INFO REPORT==== 8-Apr-2014::07:30:27 === >> accepting AMQP connection <0.22825.1> (127.0.0.1:50386 -> 127.0.0.1:5672) >> >> =INFO REPORT==== 8-Apr-2014::07:30:29 === >> accepting AMQP connection <0.22834.1> (127.0.0.1:50387 -> 127.0.0.1:5672) >> >> =INFO REPORT==== 8-Apr-2014::07:30:30 === >> accepting AMQP connection <0.22843.1> (127.0.0.1:50388 -> 127.0.0.1:5672) >> >> =INFO REPORT==== 8-Apr-2014::07:30:31 === >> accepting AMQP connection <0.22852.1> (127.0.0.1:50389 -> 127.0.0.1:5672) >> >> =INFO REPORT==== 8-Apr-2014::07:30:34 === >> accepting AMQP connection <0.22863.1> (127.0.0.1:50394 -> 127.0.0.1:5672) >> >> =INFO REPORT==== 8-Apr-2014::07:30:34 === >> accepting AMQP connection <0.22866.1> (127.0.0.1:50395 -> 127.0.0.1:5672) >> >> =WARNING REPORT==== 8-Apr-2014::07:30:36 === >> closing AMQP connection <0.22852.1> (127.0.0.1:50389 -> 127.0.0.1:5672): >> connection_closed_abruptly >> >> =INFO REPORT==== 8-Apr-2014::07:30:36 === >> accepting AMQP connection <0.22883.1> (127.0.0.1:50399 -> 127.0.0.1:5672) >> >> =WARNING REPORT==== 8-Apr-2014::07:30:37 === >> closing AMQP connection <0.22761.1> (127.0.0.1:50370 -> 127.0.0.1:5672): >> connection_closed_abruptly >> >> =WARNING REPORT==== 8-Apr-2014::07:30:38 === >> closing AMQP connection <0.22796.1> (127.0.0.1:50383 -> 127.0.0.1:5672): >> connection_closed_abruptly >> >> =INFO REPORT==== 8-Apr-2014::07:30:39 === >> accepting AMQP connection <0.22893.1> (127.0.0.1:50403 -> 127.0.0.1:5672) >> >> =WARNING REPORT==== 8-Apr-2014::07:30:39 === >> closing AMQP connection <0.22810.1> (127.0.0.1:50385 -> 127.0.0.1:5672): >> connection_closed_abruptly >> >> =INFO REPORT==== 8-Apr-2014::07:30:41 === >> accepting AMQP connection <0.22902.1> (127.0.0.1:50409 -> 127.0.0.1:5672) >> >> =INFO REPORT==== 8-Apr-2014::07:30:43 === >> accepting AMQP connection <0.22913.1> (127.0.0.1:50411 -> 127.0.0.1:5672) >> >> =INFO REPORT==== 8-Apr-2014::07:30:49 === >> accepting AMQP connection <0.22925.1> (127.0.0.1:50420 -> 127.0.0.1:5672) >> >> =INFO REPORT==== 8-Apr-2014::07:30:49 === >> accepting AMQP connection <0.22928.1> (127.0.0.1:50421 -> 127.0.0.1:5672) >> >> =WARNING REPORT==== 8-Apr-2014::07:30:50 === >> closing AMQP connection <0.22660.1> (127.0.0.1:50332 -> 127.0.0.1:5672): >> connection_closed_abruptly >> >> =INFO REPORT==== 8-Apr-2014::07:30:51 === >> accepting AMQP connection <0.22945.1> (127.0.0.1:50423 -> 127.0.0.1:5672) >> ~$ tail -n 100 /var/log/rabbitmq/shutdown_err >> /usr/lib/rabbitmq/bin/rabbitmqctl: 1: /etc/rabbitmq/rabbitmq-env.conf: >> ocr-proc-2=rabbit at localhost: not found >> ~$ tail -n 100 /var/log/rabbitmq/shutdown_log >> Stopping and halting node 'rabbit at ocr-proc-2' ... >> ...done. >> ~$ tail -n 100 /var/log/rabbitmq/startup_err >> /usr/lib/rabbitmq/bin/rabbitmq-server: 1: >> /etc/rabbitmq/rabbitmq-env.conf: ocr-proc-2=rabbit at localhost: not found >> Killed >> ~$ tail -n 100 /var/log/rabbitmq/startup_log >> >> RabbitMQ 3.2.4. Copyright (C) 2007-2013 GoPivotal, Inc. >> ## ## Licensed under the MPL. See http://www.rabbitmq.com/ >> ## ## >> ########## Logs: /var/log/rabbitmq/rabbit at ocr-proc-2.log >> ###### ## /var/log/rabbitmq/rabbit at ocr-proc-2-sasl.log >> ########## >> Starting broker... completed with 6 plugins. >> >> >> >> >> Michael Sander >> mes65 at cornell.edu >> 607-227-9859 >> >> >> On Tue, Apr 8, 2014 at 3:33 AM, Michael Sander wrote: >> >>> Hi Matthais, >>> >>> What I sent you was everything I had. However, I did check ps -aux after >>> the crash and rabbitmq-server was definitely not in there. I will turn off >>> the cron jobs that automatically restart rabbitmq, and I'll let you know if >>> I see it again. >>> >>> Here is the output of the command. >>> >>> $ sudo rabbitmqctl eval 'rabbit_misc:os_cmd("/bin/df -kP >>> /var/lib/rabbitmq/mnes >>> ia/").' >>> "Filesystem 1024-blocks >>> Used Available Capacity Mounted >>> on\n/dev/disk/by-uuid/36fd30d4-ea87-419f-a6a4-a1a3cf290ff1 10320184 >>> 6348300 3447648 65% /\n" >>> ...done. >>> >>> >>> Also, I'm not sure whether it will help you, but attached is a >>> screenshot of the rabbitmq console. If you see the start of the top chart >>> at 19:00, there is a sharp increase in the queued messages. That's when I >>> restarted rabbitmq after the crash. Everything before that was flat. >>> Another point to note is that it currently says that the disk space is >>> unavailable. I definitely remember seeing a value there at some point >>> before, I don't know what causes that to occur. >>> >>> I've turned off my rabbimq auto-start cron jobs, I'll let you know if I >>> see the crash again. >>> >>> Thanks again. >>> >>> Best, >>> >>> Michael Sander >>> >>> >>> >>> On Tue, Apr 8, 2014 at 1:18 AM, Matthias Radestock < >>> matthias at rabbitmq.com> wrote: >>> >>>> Michael, >>>> >>>> >>>> On 08/04/14 02:50, Michael Sander wrote: >>>> >>>>> Full logs are attached. You'll notice that it crashes pretty often >>>>> now. >>>>> >>>> >>>> The disk_monitor is crashing frequently, yes, but in none of the >>>> instances in the logs that actually took down rabbit (notice that there are >>>> no rabbit starts recorded in the rabbit.log); the disk_monitor restarts >>>> just fine and the bunny lives on. >>>> >>>> Do you have the logs covering the time period around the crash? >>>> >>>> >>>> Here are the output of the commands >>>>> >>>>> $ sudo rabbitmqctl eval 'rabbit_misc:os_cmd("/bin/df -kP >>>>> /var/lib/rabbitmq/mnesia/")' >>>>> Error: syntax error before: >>>>> >>>> >>>> Ah, sorry, missed a full stop. Should be >>>> >>>> sudo rabbitmqctl eval 'rabbit_misc:os_cmd("/bin/df -kP >>>> /var/lib/rabbitmq/mnesia/").' >>>> >>>> >>>> Matthias. >>>> >>> >>> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From tbuehlmann.android at googlemail.com Wed Apr 9 10:32:50 2014 From: tbuehlmann.android at googlemail.com (=?UTF-8?Q?Tobias_B=C3=BChlmann?=) Date: Wed, 9 Apr 2014 02:32:50 -0700 (PDT) Subject: [rabbitmq-discuss] Design Decision Message-ID: <2f4c1429-5142-4a8f-970e-d0cfb2c63eac@googlegroups.com> Hey, I'm pretty new to RabbitMQ and AMQP in general and I have the feeling my design decisions regarding this problem might be wrong. So, the problem: I have a relational database with a notifications table where each notification has many associated apps. Each app has itself many devices. Now, a producer will check the table for new notifications with a specific state. If there are any, grab them and produce a message for each device in each app associated with the new notification. After that, some consumer workers will work the queue(s) off and deliver the messages to the devices. Now, since the message payload will differ depending on the app type, my idea was to dynamically generate queues for each app for a notification. Having a notification with ID=42 and associated apps with IDS=110, 112, I would generate the queues "42_110" and "42_112". Then, consumer workers would be notified about the new queues and work them off. As the workers need the notification and app data in addition to the message payload itself, they could easily get it by splitting the queue name at "_" and querying the database. Despite I think this could work, it just doesn't feel right. Is there any opinionated way to go? How would you design it? Appreciating any help. :) -------------- next part -------------- An HTML attachment was scrubbed... URL: From maximov at rutoll.ru Wed Apr 9 10:46:21 2014 From: maximov at rutoll.ru (Nikolay Maximov) Date: Wed, 9 Apr 2014 02:46:21 -0700 (PDT) Subject: [rabbitmq-discuss] =?utf-8?q?=C2=A0Installing_broker_on_each_node?= =?utf-8?q?_instead_of_client-only_approach?= Message-ID: <2efb248a-473f-4d71-8caa-a6df339278a8@googlegroups.com> Hello, I have a network of nodes which has some events to be saved in local store and queued for further processing in central back office. Please advice what is better to manage and costs less: to have local store and application which store message first and queue it then versus local rabbitmq broker to do the same thing with persistent queue. I am looking for best practice in this approach. Does someone has an exprience in such design where broker installed on each node? -- Nikolay -------------- next part -------------- An HTML attachment was scrubbed... URL: From manjay.bit2k7 at gmail.com Wed Apr 9 12:39:04 2014 From: manjay.bit2k7 at gmail.com (Rafael) Date: Wed, 9 Apr 2014 04:39:04 -0700 (PDT) Subject: [rabbitmq-discuss] RabbitMQ callback using some listener Message-ID: <814f016c-1c52-4b43-9a65-4754de393e47@googlegroups.com> Apology if it's very basic question. I am new to RabbitMQ. I have one requirement where My receiver should be notified if There is something written in Queue by sender. I don't want to implement a solution where I will check every time in queue and if its there I will deque. I dont want like this where I will check every time in a queue rather please provide some solution where I will get notification while something written in queue. while (true) {var ea =(BasicDeliverEventArgs)consumer.Queue.Dequeue();var body = ea.Body;var message = Encoding.UTF8.GetString(body);Console.WriteLine(" [x] Received {0}", message); int dots = message.Split('.').Length - 1;Thread.Sleep(dots * 1000); Console.WriteLine(" [x] Done"); channel.BasicAck(ea.DeliveryTag, false); } Thanks in Advance -------------- next part -------------- An HTML attachment was scrubbed... URL: From videlalvaro at gmail.com Wed Apr 9 15:13:19 2014 From: videlalvaro at gmail.com (Alvaro Videla) Date: Wed, 9 Apr 2014 16:13:19 +0200 Subject: [rabbitmq-discuss] Design Decision In-Reply-To: <2f4c1429-5142-4a8f-970e-d0cfb2c63eac@googlegroups.com> References: <2f4c1429-5142-4a8f-970e-d0cfb2c63eac@googlegroups.com> Message-ID: Hi, Have you looked into the topic exchange? Cheers, Alvaro On Wed, Apr 9, 2014 at 11:32 AM, Tobias B?hlmann wrote: > Hey, > > I'm pretty new to RabbitMQ and AMQP in general and I have the feeling my > design decisions regarding this problem might be wrong. So, the problem: I > have a relational database with a notifications table where each > notification has many associated apps. Each app has itself many devices. > Now, a producer will check the table for new notifications with a specific > state. If there are any, grab them and produce a message for each device in > each app associated with the new notification. After that, some consumer > workers will work the queue(s) off and deliver the messages to the devices. > > Now, since the message payload will differ depending on the app type, my > idea was to dynamically generate queues for each app for a notification. > Having a notification with ID=42 and associated apps with IDS=110, 112, I > would generate the queues "42_110" and "42_112". Then, consumer workers > would be notified about the new queues and work them off. As the workers > need the notification and app data in addition to the message payload > itself, they could easily get it by splitting the queue name at "_" and > querying the database. > > Despite I think this could work, it just doesn't feel right. Is there any > opinionated way to go? How would you design it? Appreciating any help. :) > > _______________________________________________ > rabbitmq-discuss mailing list > rabbitmq-discuss at lists.rabbitmq.com > https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss > From matthias at rabbitmq.com Wed Apr 9 15:20:18 2014 From: matthias at rabbitmq.com (Matthias Radestock) Date: Wed, 09 Apr 2014 15:20:18 +0100 Subject: [rabbitmq-discuss] Possible memory leak in the management plugin In-Reply-To: <1397020502914-34663.post@n5.nabble.com> References: <8a1af6ed-c599-42a3-85c3-c5dc22a99bd8@googlegroups.com> <520CB29A.1060304@rabbitmq.com> <1397020502914-34663.post@n5.nabble.com> Message-ID: <53455722.6050906@rabbitmq.com> On 09/04/14 06:15, Pavel wrote: > We are running RabbitMQ 3.1.5 on Erlang R15B. > > Last week we had a couple incidents, where RabbitMQ would go from stable > ~1Gb RAM to ~10Gb in ~10 mins and keep growing up to the high watermark. It > was quickly identified that mgmt_db is a culprit, so we disabled management > plugin and attempted to reproduce the issue in the lab. Although the lab > test wasn't exactly representing the production setup, we managed to achieve > similar behavior with rapid unbounded growth of mgmt_db using the following > scenario: > > - declare 1000 exchanges with 1 queue (maxlength=5) bound to each one > - run a single threaded script that will > - create 1 connection with 1000 channels > - publish 1000 messages (1 per channel) to each of exchanges sequentially I am unable to reproduce this with the attached RabbitMQ Erlang Client code. This stabilises at around 150MB total memory use reported in the management UI, with just 11MB of that taken up by the management DB. Matthias. -------------- next part -------------- A non-text attachment was scrubbed... Name: loadsa_channels.erl Type: text/x-erlang Size: 1042 bytes Desc: not available URL: From srinath.c at gmail.com Wed Apr 9 16:04:57 2014 From: srinath.c at gmail.com (Srinath C) Date: Wed, 9 Apr 2014 20:34:57 +0530 Subject: [rabbitmq-discuss] Troubleshooting handleCancel() In-Reply-To: References: Message-ID: Hi MK, Thanks for your inputs. Below are my replies to your queries: a, I can say for sure that there is part of this deployment which attempts to explicit delete a queue or an exchange. b, Queue master node failure could be a possibility. But I'm not sure if the failed node was the master. How do you find it out by the way? c, I have attached the rabbitmq logs for the surviving node. I have cropped the logs to a few minutes before and after the handleCancel was called. d, I can say for sure that publisher was connected to the surviving node. The consumer - I believe was also connected to the surviving node because of the fact that lyra would have otherwise recovered the queue and bindings. e, Sure, will try out 3.3 to see if this issue occurs again. Thanks again, Srinath. On Wed, Apr 9, 2014 at 1:31 PM, Michael Klishin wrote: > On 9 April 2014 at 05:36:42, Srinath C (srinath.c at gmail.com) wrote: > > > Scenario: > > One of the rabbitmq nodes were brought down for testing. > > It was observed that no more messages were being consumed > > Note that RabbitMQ 3.3.0 will automatically re-register consumers > on a queue that had master failure. So I suggest giving your scenario > a try with 3.3. > -- > MK > > Software Engineer, Pivotal/RabbitMQ > _______________________________________________ > rabbitmq-discuss mailing list > rabbitmq-discuss at lists.rabbitmq.com > https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss > -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: rabbitmq_survivor.log Type: application/octet-stream Size: 36663 bytes Desc: not available URL: From matthias at rabbitmq.com Wed Apr 9 16:33:36 2014 From: matthias at rabbitmq.com (Matthias Radestock) Date: Wed, 09 Apr 2014 16:33:36 +0100 Subject: [rabbitmq-discuss] Rabbit MQ Crash In-Reply-To: References: <4da4be68-9e3b-4bba-aaa9-e262df330d6f@googlegroups.com> <5342E86E.4060508@rabbitmq.com> <5343117D.7080405@rabbitmq.com> <53431742.5000008@rabbitmq.com> <5343868A.3060207@rabbitmq.com> Message-ID: <53456850.2070205@rabbitmq.com> On 09/04/14 09:15, Michael Sander wrote: > Anything I can do to move the ball forward here? This keeps on > happening. Evidently on GCE every now and then the output of 'df' is mangled, or there is some other problem when rabbit is executing the 'df' command. We'll work around that in the next bugfix release. Prior to that, the fix will be in the nightly builds in a few days. Matthias. From simon at rabbitmq.com Wed Apr 9 16:47:16 2014 From: simon at rabbitmq.com (Simon MacMullen) Date: Wed, 09 Apr 2014 16:47:16 +0100 Subject: [rabbitmq-discuss] Rabbit MQ Crash In-Reply-To: <53456850.2070205@rabbitmq.com> References: <4da4be68-9e3b-4bba-aaa9-e262df330d6f@googlegroups.com> <5342E86E.4060508@rabbitmq.com> <5343117D.7080405@rabbitmq.com> <53431742.5000008@rabbitmq.com> <5343868A.3060207@rabbitmq.com> <53456850.2070205@rabbitmq.com> Message-ID: <53456B84.6010906@rabbitmq.com> On 09/04/14 16:33, Matthias Radestock wrote: > Prior to that, the fix will be in the nightly builds in a few days. Actually it will be in tonight's build, I just merged it. It would be helpful if you can report back and let us know if this fixes things for you. Cheers, Simon -- Simon MacMullen RabbitMQ, Pivotal From pmaisenovich at blizzard.com Wed Apr 9 16:51:25 2014 From: pmaisenovich at blizzard.com (Pavel) Date: Wed, 9 Apr 2014 08:51:25 -0700 (PDT) Subject: [rabbitmq-discuss] Possible memory leak in the management plugin In-Reply-To: <534521F1.9030707@rabbitmq.com> References: <8a1af6ed-c599-42a3-85c3-c5dc22a99bd8@googlegroups.com> <520CB29A.1060304@rabbitmq.com> <1397020502914-34663.post@n5.nabble.com> <534521F1.9030707@rabbitmq.com> Message-ID: <1397058685615-34697.post@n5.nabble.com> Hi Simon, Thank you for a quick and detailed answer! Getting to the root of this issue is very important to us. > if you want to re-enable mgmt you might want to add > {rabbitmq_management_agent, [{force_fine_statistics, false}]} > in your configuration We definitely want to have mgmt plugin running and we did re-enable it after implementing some work on reducing number of channels/exchanges a bit and increasing collect_statistics_interval as it proved to be helpful in the lab tests. We will consider turning off force_fine_statistics if the issue happens again instead turning off entire plugin. > So are they growing without bound? What happens when you close all the > channels? It's hard to say how far they will grow, but these three (aggregated_stats, aggregated_stats_index, old_stats) definitely only go up during the test. See below for more info. Killing publishing channels does bring these three tables and mgmt_db memory back to normal size (~6Mb) after some short time. So the issue definitely requires continuous activity on channels producing massive stats events to happen. >What does >rabbitmqctl eval 'erlang:garbage_collect(global:whereis_name(rabbit_mgmt_db)).' >do to your memory use? I've repeated the test described earlier while running /usr/sbin/rabbitmqctl eval '{process_info(global:whereis_name(rabbit_mgmt_db),memory),[{T, ets:info(T,size), ets:info(T,memory)} || T <-lists:sort(ets:all()), rabbit_mgmt_db <- [ets:info(T, name)]]}.' every second. Process memory (and sizes of those aggregated/old stats data structures) went quickly up to 2069410400 and then stopped. After about 30 seconds I checked rabbitmqctl status and it reported that mgmt_db was using 3795297904! [root at lab-rmq02 pmaisenovich]# /usr/sbin/rabbitmqctl status | grep mgmt_db {mgmt_db,3795297904}, [root at lab-rmq02 pmaisenovich]# /usr/sbin/rabbitmqctl eval '{process_info(global:whereis_name(rabbit_mgmt_db),memory),[{T, ets:info(T,size), ets:info(T,memory)} || T <-lists:sort(ets:all()), rabbit_mgmt_db <- [ets:info(T, name)]]}.' {{memory,2069410496}, [{5734484,1046,205243}, {5738585,5,906}, {5742682,2006,136495}, {5746779,1,175}, {5750876,1,175}, {5754973,1,1059}, {5759070,895752,48244117}, {5763167,1777748,44462528}, {5767264,1777799,124621170}]} ...done. Clearly we have some memory missing: ETS tables report ~225Mb, process_info(rabbit_mgmt_db, memory) reports ~2Gb, rabbitmqctl status reports ~3.8Gb for mgmt_db. Q5: What else is included in mgmt_db size when reported by rabbitmqctl status? Running garbage collection (erlang:garbage_collect(global:whereis_name(rabbit_mgmt_db))) did instantly reduce the mgmt_db size: [root at lab-rmq02 pmaisenovich]# /usr/sbin/rabbitmqctl status | grep mgmt_db {mgmt_db,3853521792}, [root at lab-rmq02 pmaisenovich]# /usr/sbin/rabbitmqctl eval 'erlang:garbage_collect(global:whereis_name(rabbit_mgmt_db)).' true ...done. [root at lab-rmq02 pmaisenovich]# /usr/sbin/rabbitmqctl status | grep mgmt_db {mgmt_db,1804503848}, And immediately cleaned up rabbit_mgmt_db memory (even too much so): {{memory,2069410400}, [{5734484,1046,205243}, {5738585,5,906}, {5742682,2006,136495}, {5746779,1,175}, {5750876,1,175}, {5754973,1,1059}, {5759070,916685,51561447}, {5763167,1819614,45509178}, {5767264,1819847,127559980}]} ...done. {{memory,5960}, [{5734484,1046,205243}, {5738585,5,906}, {5742682,2006,136495}, {5746779,1,175}, {5750876,1,175}, {5754973,1,1059}, {5759070,916685,51561447}, {5763167,1819614,45509178}, {5767264,1819847,127559980}]} ...done. Q6: In the last snapshot above process_info(rabbit_mgmt_db, memory) is much smaller than ETS numbers right below it. Are those not included in the process memory calculation? Q7: Note, that ETS table sizes didn't go down at all. Isn't GC supposed to clean those up? Furthermore, in 4 seconds after manual GC run, the process_info(rabbit_mgmt_db, memory) went up from 5960 to 783979640 and kept growing up to a certain (different to previous) limit. Here is a full log of process memory snapshots during this test (including drop after GC run): https://gist.github.com/maisenovich/10283607 Note, that I ran GC a couple more times during the test, with the same effect - memory would go down briefly and quickly ramp back up. Finally, "rabbitmqctl status" output for Rabbit used in the test above (while idle, after publishing channels were terminated) just in case: https://gist.github.com/maisenovich/10284388 Thanks! -- View this message in context: http://rabbitmq.1065348.n5.nabble.com/Possible-memory-leak-in-the-management-plugin-tp27414p34697.html Sent from the RabbitMQ mailing list archive at Nabble.com. From matthias at rabbitmq.com Wed Apr 9 17:01:31 2014 From: matthias at rabbitmq.com (Matthias Radestock) Date: Wed, 09 Apr 2014 17:01:31 +0100 Subject: [rabbitmq-discuss] Diagnosing errors from SASL log In-Reply-To: References: <5343F618.3060604@rabbitmq.com> Message-ID: <53456EDB.5030405@rabbitmq.com> On 08/04/14 14:17, Patrick Long wrote: > RabbitMQ 3.2.4 > Erlang R16B02 > Windows Server 2008 R2 > > Consistent across cluster Have you got the complete sasl logs from the time the servers were restarted to when you were seeing the errors? Matthias. From pmaisenovich at blizzard.com Wed Apr 9 17:06:27 2014 From: pmaisenovich at blizzard.com (Pavel) Date: Wed, 9 Apr 2014 09:06:27 -0700 (PDT) Subject: [rabbitmq-discuss] Possible memory leak in the management plugin In-Reply-To: <53455722.6050906@rabbitmq.com> References: <8a1af6ed-c599-42a3-85c3-c5dc22a99bd8@googlegroups.com> <520CB29A.1060304@rabbitmq.com> <1397020502914-34663.post@n5.nabble.com> <53455722.6050906@rabbitmq.com> Message-ID: <1397059587422-34699.post@n5.nabble.com> Hi Matthias, Thanks for your reply! I'm not 100% sure, but it seems your publishing pattern is slightly different. This is what we do in our test: private void publish(int exchangeCount) throws IOException { for (int i = 0; i < exchangeCount; ++i) { // for each of 1000 exchanges for (Channel c : channels) { // for each of 1000 channels // Channel.basicPublish(exchange, routingKey, props, byte[] body) c.basicPublish("foo-" + i, "foo." + i, null, "foo".getBytes()); } } } I will clean up and publish entire source later today. Thanks! -- View this message in context: http://rabbitmq.1065348.n5.nabble.com/Possible-memory-leak-in-the-management-plugin-tp27414p34699.html Sent from the RabbitMQ mailing list archive at Nabble.com. From matthias at rabbitmq.com Wed Apr 9 17:09:25 2014 From: matthias at rabbitmq.com (Matthias Radestock) Date: Wed, 09 Apr 2014 17:09:25 +0100 Subject: [rabbitmq-discuss] Possible memory leak in the management plugin In-Reply-To: <1397059587422-34699.post@n5.nabble.com> References: <8a1af6ed-c599-42a3-85c3-c5dc22a99bd8@googlegroups.com> <520CB29A.1060304@rabbitmq.com> <1397020502914-34663.post@n5.nabble.com> <53455722.6050906@rabbitmq.com> <1397059587422-34699.post@n5.nabble.com> Message-ID: <534570B5.5020705@rabbitmq.com> On 09/04/14 17:06, Pavel wrote: > I'm not 100% sure, but it seems your publishing pattern is slightly > different. This is what we do in our test: Ah. You are publishing to each exchange in each channel. Got it. Will give that a go. Matthias. From matthias at rabbitmq.com Wed Apr 9 17:30:50 2014 From: matthias at rabbitmq.com (Matthias Radestock) Date: Wed, 09 Apr 2014 17:30:50 +0100 Subject: [rabbitmq-discuss] Diagnosing errors from SASL log In-Reply-To: References: <5343F618.3060604@rabbitmq.com> <53456EDB.5030405@rabbitmq.com> Message-ID: <534575BA.1040306@rabbitmq.com> (+list) On 09/04/14 17:17, Patrick Long wrote: > I have attached the SASL logs from the time of the initial restart > (15:13) until the end of that day cheers. please also send the rabbit.logs. Matthias. From simon at rabbitmq.com Wed Apr 9 17:33:56 2014 From: simon at rabbitmq.com (Simon MacMullen) Date: Wed, 09 Apr 2014 17:33:56 +0100 Subject: [rabbitmq-discuss] Possible memory leak in the management plugin In-Reply-To: <1397058685615-34697.post@n5.nabble.com> References: <8a1af6ed-c599-42a3-85c3-c5dc22a99bd8@googlegroups.com> <520CB29A.1060304@rabbitmq.com> <1397020502914-34663.post@n5.nabble.com> <534521F1.9030707@rabbitmq.com> <1397058685615-34697.post@n5.nabble.com> Message-ID: <53457674.1040704@rabbitmq.com> On 09/04/14 16:51, Pavel wrote: > Clearly we have some memory missing: ETS tables report ~225Mb, > process_info(rabbit_mgmt_db, memory) reports ~2Gb, rabbitmqctl status > reports ~3.8Gb for mgmt_db. > > Q5: What else is included in mgmt_db size when reported by rabbitmqctl > status? Nothing. But ets:info(Table, memory) reports a number of words, not bytes. So it needs to be multiplied by 8 (assuming a 64 bit system). rabbitmqctl status does that for you, hence ~3.8Gb. > Running garbage collection > (erlang:garbage_collect(global:whereis_name(rabbit_mgmt_db))) did instantly > reduce the mgmt_db size: > > [root at lab-rmq02 pmaisenovich]# /usr/sbin/rabbitmqctl status | grep mgmt_db > {mgmt_db,3853521792}, > [root at lab-rmq02 pmaisenovich]# /usr/sbin/rabbitmqctl eval > 'erlang:garbage_collect(global:whereis_name(rabbit_mgmt_db)).' > true > ...done. > [root at lab-rmq02 pmaisenovich]# /usr/sbin/rabbitmqctl status | grep mgmt_db > {mgmt_db,1804503848}, > > And immediately cleaned up rabbit_mgmt_db memory (even too much so): > > {{memory,2069410400}, > [{5734484,1046,205243}, > {5738585,5,906}, > {5742682,2006,136495}, > {5746779,1,175}, > {5750876,1,175}, > {5754973,1,1059}, > {5759070,916685,51561447}, > {5763167,1819614,45509178}, > {5767264,1819847,127559980}]} > ...done. > {{memory,5960}, > [{5734484,1046,205243}, > {5738585,5,906}, > {5742682,2006,136495}, > {5746779,1,175}, > {5750876,1,175}, > {5754973,1,1059}, > {5759070,916685,51561447}, > {5763167,1819614,45509178}, > {5767264,1819847,127559980}]} > ...done. > > Q6: In the last snapshot above process_info(rabbit_mgmt_db, memory) is much > smaller than ETS numbers right below it. Are those not included in the > process memory calculation? No, they're not. The "rabbitmqctl status" output adds the process memory and ETS memory together. > Q7: Note, that ETS table sizes didn't go down at all. Isn't GC supposed to > clean those up? No, the GC is only for the process memory. ETS is more like a database, rows get deleted when you tell it to. > Furthermore, in 4 seconds after manual GC run, the > process_info(rabbit_mgmt_db, memory) went up from 5960 to 783979640 and kept > growing up to a certain (different to previous) limit. Ouch. So I think there are two issues here: 1) The combination of large numbers of exchanges / queues being published to by large numbers of channels is something that the management database inherently struggles with; it maintains data for all those combinations, so if you have huge numbers of them, you'll have huge amounts of data. Currently there's no switch to say "I just want message rates per queue, exchange or channel, not per combination of those things". If there were, it would allow the mgmt database to cut its memory usage a lot (although it would still be dealing with a lot of incoming data). So your only option is to switch off message rates altogether, with {force_fine_statistics, false}. 2) The management database is not getting GCed enough by Erlang when it is very busy. It's probable that we could do something about that, although the fact that the amount of garbage goes back up so quickly is alarming. So in the short term, {force_fine_statistics, false} is the best option you have. In the longer term we may be able to have the mgmt plugin still display some message rates at a lower cost. Cheers, Simon -- Simon MacMullen RabbitMQ, Pivotal From aaron.kenny at gmail.com Wed Apr 9 16:09:48 2014 From: aaron.kenny at gmail.com (aaron kenny) Date: Wed, 9 Apr 2014 08:09:48 -0700 (PDT) Subject: [rabbitmq-discuss] set content_type using PHP Message-ID: I'm struggling to find out how to set the content_type='text/json'. ive seen it set before via th basic_publish() function but not for PHP. Any ideas. -------------- next part -------------- An HTML attachment was scrubbed... URL: From simon at rabbitmq.com Wed Apr 9 18:07:04 2014 From: simon at rabbitmq.com (Simon MacMullen) Date: Wed, 09 Apr 2014 18:07:04 +0100 Subject: [rabbitmq-discuss] Rabbit MQ Crash In-Reply-To: References: <4da4be68-9e3b-4bba-aaa9-e262df330d6f@googlegroups.com> <5342E86E.4060508@rabbitmq.com> <5343117D.7080405@rabbitmq.com> <53431742.5000008@rabbitmq.com> <5343868A.3060207@rabbitmq.com> <53456850.2070205@rabbitmq.com> <53456B84.6010906@rabbitmq.com> Message-ID: <53457E38.90600@rabbitmq.com> On 09/04/14 18:04, Michael Sander wrote: > I normally just pull it from your apt packages. I've been pretty busy > the past few days, so it may take me a day or two to > pull/compile/install the source. The nightly builds include .deb packages, you don't need to compile. You can download from http://www.rabbitmq.com/nightlies/rabbitmq-server/ (err, in about 12 hours). Cheers, Simon -- Simon MacMullen RabbitMQ, Pivotal From simon at rabbitmq.com Wed Apr 9 18:14:21 2014 From: simon at rabbitmq.com (Simon MacMullen) Date: Wed, 09 Apr 2014 18:14:21 +0100 Subject: [rabbitmq-discuss] The RabbitMQ team is recruiting Message-ID: <53457FED.1000505@rabbitmq.com> http://careers.stackoverflow.com/jobs/53520/senior-software-engineer-rabbitmq-pivotal Cheers, Simon -- Simon MacMullen RabbitMQ, Pivotal From ambatinr at hotmail.com Wed Apr 9 20:00:13 2014 From: ambatinr at hotmail.com (anate) Date: Wed, 9 Apr 2014 12:00:13 -0700 (PDT) Subject: [rabbitmq-discuss] How to increase the size limit for message? Message-ID: <1397070013452-34706.post@n5.nabble.com> My messages are getting 25 MB sometimes. I am getting the following error message. Fatal error: Allowed memory size of 134217728 bytes exhausted (tried to allocate 43065995 bytes) How to increase the size limit for Queue message? Please let me know. -- View this message in context: http://rabbitmq.1065348.n5.nabble.com/How-to-increase-the-size-limit-for-message-tp34706.html Sent from the RabbitMQ mailing list archive at Nabble.com. From mklishin at gopivotal.com Wed Apr 9 20:18:37 2014 From: mklishin at gopivotal.com (Michael Klishin) Date: Wed, 9 Apr 2014 23:18:37 +0400 Subject: [rabbitmq-discuss] set content_type using PHP In-Reply-To: References: Message-ID: On 9 April 2014 at 20:58:34, aaron kenny (aaron.kenny at gmail.com) wrote: > > I'm struggling to find out how to set the content_type='text/json'. > ive seen it set before via th basic_publish() function but not > for PHP. Any ideas. https://github.com/videlalvaro/php-amqplib/blob/master/demo/amqp_publisher.php#L41? -- MK Software Engineer, Pivotal/RabbitMQ From videlalvaro at gmail.com Wed Apr 9 20:25:32 2014 From: videlalvaro at gmail.com (Alvaro Videla) Date: Wed, 9 Apr 2014 21:25:32 +0200 Subject: [rabbitmq-discuss] How to increase the size limit for message? In-Reply-To: <1397070013452-34706.post@n5.nabble.com> References: <1397070013452-34706.post@n5.nabble.com> Message-ID: Hi, Are you using PHP, because this looks like a PHP error. If that's the case, then you need to increase the memory allowed for your PHP script. See: http://www.php.net/manual/en/ini.core.php#ini.memory-limit Regards, Alvaro On Wed, Apr 9, 2014 at 9:00 PM, anate wrote: > My messages are getting 25 MB sometimes. I am getting the following error > message. > > Fatal error: Allowed memory size of 134217728 bytes exhausted (tried to > allocate 43065995 bytes) > > How to increase the size limit for Queue message? Please let me know. > > > > -- > View this message in context: http://rabbitmq.1065348.n5.nabble.com/How-to-increase-the-size-limit-for-message-tp34706.html > Sent from the RabbitMQ mailing list archive at Nabble.com. > _______________________________________________ > rabbitmq-discuss mailing list > rabbitmq-discuss at lists.rabbitmq.com > https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss From mklishin at gopivotal.com Wed Apr 9 20:33:27 2014 From: mklishin at gopivotal.com (Michael Klishin) Date: Wed, 9 Apr 2014 23:33:27 +0400 Subject: [rabbitmq-discuss] How to increase the size limit for message? In-Reply-To: <1397070013452-34706.post@n5.nabble.com> References: <1397070013452-34706.post@n5.nabble.com> Message-ID: On 9 April 2014 at 23:01:25, anate (ambatinr at hotmail.com) wrote: > > Fatal error: Allowed memory size of 134217728 bytes exhausted > (tried to > allocate 43065995 bytes) > > How to increase the size limit for Queue message? Please let me > know. There is no message size or queue size limit.? What?s in RabbitMQ log? where does the error come from? What?s your VM high memory watermark setting? What versions of RabbitMQ and Erlang do you run on what OS?? See also http://www.rabbitmq.com/memory.html and http://www.rabbitmq.com/blog/2014/01/23/preventing-unbounded-buffers-with-rabbitmq/ -- MK Software Engineer, Pivotal/RabbitMQ From matthias at rabbitmq.com Wed Apr 9 20:34:50 2014 From: matthias at rabbitmq.com (Matthias Radestock) Date: Wed, 09 Apr 2014 20:34:50 +0100 Subject: [rabbitmq-discuss] RabbitMQ callback using some listener In-Reply-To: <814f016c-1c52-4b43-9a65-4754de393e47@googlegroups.com> References: <814f016c-1c52-4b43-9a65-4754de393e47@googlegroups.com> Message-ID: <5345A0DA.7070109@rabbitmq.com> On 09/04/14 12:39, Rafael wrote: > Apology if it's very basic question. I am new to RabbitMQ. > > I have one requirement where My receiver should be notified if There is > something written in Queue by sender. I don't want to implement a > solution where I will check every time in queue and if its there I will > deque. > > I dont want like this where I will check every time in a queue rather > please provide some solution where I will get notification while > something written in queue. See https://github.com/rabbitmq/rabbitmq-dotnet-client/blob/master/projects/examples/client/LowlevelLogTail/src/examples/LowlevelLogTail.cs for an example. Matthias. From ambatinr at hotmail.com Wed Apr 9 20:39:32 2014 From: ambatinr at hotmail.com (anate) Date: Wed, 9 Apr 2014 12:39:32 -0700 (PDT) Subject: [rabbitmq-discuss] consumer.Queue.Dequeue() timeout if there are no messages Message-ID: <1397072372771-34711.post@n5.nabble.com> I have C# Console application. I am getting timeout consumer.Queue.Dequeue() at this line when there are no messages in the queue. I would like to come out from the program if there are no messages. How to handle this scenario? Please let me know. using (IConnection connection = connectionFactory.CreateConnection()) { using (var channel = connection.CreateModel()) { channel.QueueDeclare("hello-world-queue", true, false, false, null); channel.BasicQos(0, 1, false); var consumer = new QueueingBasicConsumer(channel); channel.BasicConsume("hello-world-queue", false, consumer); while (true) { try { var ea = (BasicDeliverEventArgs)consumer.Queue.Dequeue(); -- View this message in context: http://rabbitmq.1065348.n5.nabble.com/consumer-Queue-Dequeue-timeout-if-there-are-no-messages-tp34711.html Sent from the RabbitMQ mailing list archive at Nabble.com. From matthias at rabbitmq.com Wed Apr 9 20:41:04 2014 From: matthias at rabbitmq.com (Matthias Radestock) Date: Wed, 09 Apr 2014 20:41:04 +0100 Subject: [rabbitmq-discuss] HiPE slower In-Reply-To: <1397034121943-34671.post@n5.nabble.com> References: <1397034121943-34671.post@n5.nabble.com> Message-ID: <5345A250.4040300@rabbitmq.com> On 09/04/14 10:02, carlhoerberg wrote: > Any idea why HiPE is slower on a 16 core machine? Nope. Could be an Erlang scheduler quirk. Make sure that you are running the latest version and start experimenting with the various scheduling related options to the Erlang VM (there are loads, but +swt <...> is one that I've seen make a huge difference in the past). Matthias. From ambatinr at hotmail.com Wed Apr 9 20:45:41 2014 From: ambatinr at hotmail.com (anate) Date: Wed, 9 Apr 2014 12:45:41 -0700 (PDT) Subject: [rabbitmq-discuss] How to increase the size limit for message? In-Reply-To: References: <1397070013452-34706.post@n5.nabble.com> Message-ID: <1397072741615-34713.post@n5.nabble.com> Yes. I am using PHP. I looked at PHP.ini file for memory_limit. It was set to 128M. Why do I have this issue? Please let me know. -- View this message in context: http://rabbitmq.1065348.n5.nabble.com/How-to-increase-the-size-limit-for-message-tp34706p34713.html Sent from the RabbitMQ mailing list archive at Nabble.com. From mklishin at gopivotal.com Wed Apr 9 20:54:32 2014 From: mklishin at gopivotal.com (Michael Klishin) Date: Wed, 9 Apr 2014 23:54:32 +0400 Subject: [rabbitmq-discuss] How to increase the size limit for message? In-Reply-To: <1397072741615-34713.post@n5.nabble.com> References: <1397070013452-34706.post@n5.nabble.com> <1397072741615-34713.post@n5.nabble.com> Message-ID: On 9 April 2014 at 23:46:55, anate (ambatinr at hotmail.com) wrote: > > Yes. I am using PHP. I looked at PHP.ini file for memory_limit. > It was set to > 128M. Why do I have this issue? Please let me know. Where does the error come from, PHP or RabbitMQ/Erlang VM? You certainly can publish messages 25 MB in size fast enough for the client to quickly exhaust 128 MB.? -- MK Software Engineer, Pivotal/RabbitMQ From videlalvaro at gmail.com Wed Apr 9 20:55:25 2014 From: videlalvaro at gmail.com (Alvaro Videla) Date: Wed, 9 Apr 2014 21:55:25 +0200 Subject: [rabbitmq-discuss] How to increase the size limit for message? In-Reply-To: <1397072741615-34713.post@n5.nabble.com> References: <1397070013452-34706.post@n5.nabble.com> <1397072741615-34713.post@n5.nabble.com> Message-ID: This issue is not related to RabbitMQ. It might well be that you are going well over the 128MB limit you set. Also make sure PHP is loading the right php.ini file On Wed, Apr 9, 2014 at 9:45 PM, anate wrote: > Yes. I am using PHP. I looked at PHP.ini file for memory_limit. It was set to > 128M. Why do I have this issue? Please let me know. > > > > -- > View this message in context: http://rabbitmq.1065348.n5.nabble.com/How-to-increase-the-size-limit-for-message-tp34706p34713.html > Sent from the RabbitMQ mailing list archive at Nabble.com. > _______________________________________________ > rabbitmq-discuss mailing list > rabbitmq-discuss at lists.rabbitmq.com > https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss From james.gardner at noaa.gov Wed Apr 9 21:00:29 2014 From: james.gardner at noaa.gov (James Gardner) Date: Wed, 09 Apr 2014 15:00:29 -0500 Subject: [rabbitmq-discuss] How to increase the size limit for message? In-Reply-To: References: <1397070013452-34706.post@n5.nabble.com> <1397072741615-34713.post@n5.nabble.com> Message-ID: <5345A6DD.9030808@noaa.gov> Which PHP library are you using for AMQP interaction - this one: http://pecl.php.net/package/amqp or this one: https://github.com/videlalvaro/php-amqplib ? On 04/09/2014 02:54 PM, Michael Klishin wrote: > On 9 April 2014 at 23:46:55, anate (ambatinr at hotmail.com) wrote: >>> Yes. I am using PHP. I looked at PHP.ini file for memory_limit. >> It was set to >> 128M. Why do I have this issue? Please let me know. > Where does the error come from, PHP or RabbitMQ/Erlang VM? > > You certainly can publish messages 25 MB in size fast enough for the client > to quickly exhaust 128 MB. > -- > MK > > Software Engineer, Pivotal/RabbitMQ > _______________________________________________ > rabbitmq-discuss mailing list > rabbitmq-discuss at lists.rabbitmq.com > https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss From mklishin at gopivotal.com Wed Apr 9 21:00:29 2014 From: mklishin at gopivotal.com (Michael Klishin) Date: Thu, 10 Apr 2014 00:00:29 +0400 Subject: [rabbitmq-discuss] consumer.Queue.Dequeue() timeout if there are no messages In-Reply-To: <1397072372771-34711.post@n5.nabble.com> References: <1397072372771-34711.post@n5.nabble.com> Message-ID: ?On 9 April 2014 at 23:40:47, anate (ambatinr at hotmail.com) wrote: > > I would like to come > out from the program if there are no messages. How to handle this > scenario? You get a timeout when there are no messages to deliver immediately. Presumably that?s what you want and you should terminate your program in such case, but RabbitMQ consumers are typically long running and are not started periodically. N.B. there is a SharedQueue#Dequeue version which accepts the timeout you want. -- MK Software Engineer, Pivotal/RabbitMQ From ambatinr at hotmail.com Wed Apr 9 21:05:48 2014 From: ambatinr at hotmail.com (anate) Date: Wed, 9 Apr 2014 13:05:48 -0700 (PDT) Subject: [rabbitmq-discuss] How to increase the size limit for message? In-Reply-To: <5345A6DD.9030808@noaa.gov> References: <1397070013452-34706.post@n5.nabble.com> <1397072741615-34713.post@n5.nabble.com> <5345A6DD.9030808@noaa.gov> Message-ID: <1397073948658-34718.post@n5.nabble.com> I am using https://github.com/videlalvaro/php-amqplib. -- View this message in context: http://rabbitmq.1065348.n5.nabble.com/How-to-increase-the-size-limit-for-message-tp34706p34718.html Sent from the RabbitMQ mailing list archive at Nabble.com. From mklishin at gopivotal.com Wed Apr 9 21:24:45 2014 From: mklishin at gopivotal.com (Michael Klishin) Date: Thu, 10 Apr 2014 00:24:45 +0400 Subject: [rabbitmq-discuss] How to increase the size limit for message? In-Reply-To: <1397073948658-34718.post@n5.nabble.com> References: <1397070013452-34706.post@n5.nabble.com> <1397072741615-34713.post@n5.nabble.com> <5345A6DD.9030808@noaa.gov> <1397073948658-34718.post@n5.nabble.com> Message-ID: On 10 April 2014 at 00:06:59, anate (ambatinr at hotmail.com) wrote: > > I am using https://github.com/videlalvaro/php-amqplib. > Please confirm that the error message is coming from PHP.? -- MK Software Engineer, Pivotal/RabbitMQ From ambatinr at hotmail.com Wed Apr 9 21:30:22 2014 From: ambatinr at hotmail.com (anate) Date: Wed, 9 Apr 2014 13:30:22 -0700 (PDT) Subject: [rabbitmq-discuss] consumer.Queue.Dequeue() timeout if there are no messages In-Reply-To: References: <1397072372771-34711.post@n5.nabble.com> Message-ID: <1397075422633-34720.post@n5.nabble.com> I have consumer.exe C# program and I would like to end the program when there are no messages. I tried the following code but no luck. Because I have to run this .exe program with specified user account and periodic intervals. Please let me know if I am in right path. try { var ea = (BasicDeliverEventArgs)consumer.Queue.Dequeue(); } catch(TimeoutException ex) { break; } -- View this message in context: http://rabbitmq.1065348.n5.nabble.com/consumer-Queue-Dequeue-timeout-if-there-are-no-messages-tp34711p34720.html Sent from the RabbitMQ mailing list archive at Nabble.com. From ambatinr at hotmail.com Wed Apr 9 21:33:00 2014 From: ambatinr at hotmail.com (anate) Date: Wed, 9 Apr 2014 13:33:00 -0700 (PDT) Subject: [rabbitmq-discuss] How to increase the size limit for message? In-Reply-To: References: <1397070013452-34706.post@n5.nabble.com> <1397072741615-34713.post@n5.nabble.com> <5345A6DD.9030808@noaa.gov> <1397073948658-34718.post@n5.nabble.com> Message-ID: <1397075580194-34721.post@n5.nabble.com> I think it's coming from PHP. I am trying to locate RabbitMQ log file. I am using RabbitMQ 3.2.3 and Erlang 5.10.4. I have Windows server 2008. -- View this message in context: http://rabbitmq.1065348.n5.nabble.com/How-to-increase-the-size-limit-for-message-tp34706p34721.html Sent from the RabbitMQ mailing list archive at Nabble.com. From videlalvaro at gmail.com Wed Apr 9 21:38:39 2014 From: videlalvaro at gmail.com (Alvaro Videla) Date: Wed, 9 Apr 2014 22:38:39 +0200 Subject: [rabbitmq-discuss] How to increase the size limit for message? In-Reply-To: <1397075580194-34721.post@n5.nabble.com> References: <1397070013452-34706.post@n5.nabble.com> <1397072741615-34713.post@n5.nabble.com> <5345A6DD.9030808@noaa.gov> <1397073948658-34718.post@n5.nabble.com> <1397075580194-34721.post@n5.nabble.com> Message-ID: On Wed, Apr 9, 2014 at 10:33 PM, anate wrote: > I think it's coming from PHP. I am trying to locate RabbitMQ log file. The error you are seeing *is* coming from PHP: https://www.google.com/search?q=Fatal+error%3A+Allowed+memory+size+of+134217728+bytes+exhausted&oq=Fatal+error%3A+Allowed+memory+size+of+134217728+bytes+exhausted&aqs=chrome..69i57.214j0j7&sourceid=chrome&es_sm=91&ie=UTF-8 From ambatinr at hotmail.com Wed Apr 9 21:44:28 2014 From: ambatinr at hotmail.com (anate) Date: Wed, 9 Apr 2014 13:44:28 -0700 (PDT) Subject: [rabbitmq-discuss] How to increase the size limit for message? In-Reply-To: References: <1397070013452-34706.post@n5.nabble.com> <1397072741615-34713.post@n5.nabble.com> <5345A6DD.9030808@noaa.gov> <1397073948658-34718.post@n5.nabble.com> Message-ID: <1397076268104-34723.post@n5.nabble.com> I increased to memory_size to 512MB in PHP.ini file. It works now. Thank you for the support. -- View this message in context: http://rabbitmq.1065348.n5.nabble.com/How-to-increase-the-size-limit-for-message-tp34706p34723.html Sent from the RabbitMQ mailing list archive at Nabble.com. From fred at dushin.net Wed Apr 9 23:17:57 2014 From: fred at dushin.net (Dushin Fred) Date: Wed, 9 Apr 2014 18:17:57 -0400 Subject: [rabbitmq-discuss] rabbitmq-c memory leak? In-Reply-To: References: Message-ID: <673BC989-D49B-4F23-B57B-04D7502FDDEB@dushin.net> That works for me, too. Thanks, Abhay. Alan, would you like a pull request? Is there a timeline for folding this an the other recent fixes into master? Anything I can do to help expedite? -Fred On Apr 9, 2014, at 4:54 AM, Abhay Rajure wrote: > Try this -- > > In File SimpleAmqpClient/src/ChannelImpl.cpp, > call amqp_maybe_release_buffers_on_channel() in ReturnChannel(). > > void ChannelImpl::ReturnChannel(amqp_chanel_t channel) > { > +amqp_maybe_release_buffers_on_channel(m_connection, channel); > m_free_channels.push(channel); > } -------------- next part -------------- An HTML attachment was scrubbed... URL: From pmaisenovich at blizzard.com Thu Apr 10 06:03:57 2014 From: pmaisenovich at blizzard.com (Pavel) Date: Wed, 9 Apr 2014 22:03:57 -0700 (PDT) Subject: [rabbitmq-discuss] Possible memory leak in the management plugin In-Reply-To: <53457674.1040704@rabbitmq.com> References: <8a1af6ed-c599-42a3-85c3-c5dc22a99bd8@googlegroups.com> <520CB29A.1060304@rabbitmq.com> <1397020502914-34663.post@n5.nabble.com> <534521F1.9030707@rabbitmq.com> <1397058685615-34697.post@n5.nabble.com> <53457674.1040704@rabbitmq.com> Message-ID: <1397106237672-34725.post@n5.nabble.com> Thanks for answers, Simon! Now everything makes much more sense! Here is the program I used (requires com.rabbitmq:amqp-client:3.3.0 in classpath): https://gist.github.com/maisenovich/10339925 Running it with "1000 0 10000" params (1K channels, no delay on publish, 10K iterations) blows my Rabbit's RAM to 4G (high watermark) quick, then publishing channels become blocked and Rabbit struggles to free up some memory. As you suggested, with {force_fine_statistics,false} Rabbit was able to survive the same test, stabilizing around 2Gb mark with most RAM taken by connection_stats and queue_procs. So it is certainly a working option, although at the cost of reduced visibility. It's worth noting, that in my test publish rate was around 80K/s, which generates huge flood of channel_stats events and like you said, kind of expected to crash management plugin. However the incident we had in production happened under much lower publish rate (we believe), so it still leaves uncertainty about the exact cause. Then I added a modification to the test - every time message is published to a random exchange (instead of publishing 1000 messages to each exchange sequentially). This produced an interesting result: rabbit_mgmt_db process was GCed properly and remained at stable memory use. However aggregated_stats table kept growing in size until all RAM (up to a high watermark) was taken. For example, {process_info(rabbit_mgmt_db, memory),ETS sizes} went from {{memory,542484424}, [{5730386,1046,205243}, {5734488,2005,384904}, {5738585,2006,126545}, {5742682,1,175}, {5746779,1,175}, {5750876,1,1059}, {5754973,1009026,79415774}, {5759070,2001250,49359758}, {5763167,1003620,57435178}]} at one point to this some time later: {{memory,434953160}, [{5730386,1046,205243}, {5734488,2005,384904}, {5738585,2006,132875}, {5742682,1,175}, {5746779,1,175}, {5750876,1,1059}, {5754973,1009026,147761213}, {5759070,2001250,49359758}, {5763167,1003834,57452431}]} Note, that number of items for (5754973) didn't change, but the size almost doubled. I was trying to understand "Event-GCing" portion of rabbit_mgmt_stats.erl , but that's beyond me at the moment. Could you please describe in a few words, how and when aggregated_stats supposed to be cleaned up? Also, from your earlier reply it sounds like closing and reopening channels (sometimes) would help to keep mgmt_db from expanding. Is this something generally recommended to do (short-live vs long-live channels)? Thanks! -- View this message in context: http://rabbitmq.1065348.n5.nabble.com/Possible-memory-leak-in-the-management-plugin-tp27414p34725.html Sent from the RabbitMQ mailing list archive at Nabble.com. From alan.antonuk at gmail.com Thu Apr 10 06:11:19 2014 From: alan.antonuk at gmail.com (Alan Antonuk) Date: Wed, 9 Apr 2014 22:11:19 -0700 Subject: [rabbitmq-discuss] rabbitmq-c memory leak? In-Reply-To: <673BC989-D49B-4F23-B57B-04D7502FDDEB@dushin.net> References: <673BC989-D49B-4F23-B57B-04D7502FDDEB@dushin.net> Message-ID: What version of SimpleAmqpClient are you using? The lines surrounding your change don't seem to match what is currentlyin the master branch of SimpleAmqpClient. I believe behavior you're seeing was addressed a while ago (see Issue #56). Please download the master branch and build from that, you will also need rabbitmq-c v0.4.1 or newer for this to work. FYI: the patch you are suggesting using will likely cause undefined behavior. Because of the way rabbitmq-c's memory management works, the SimpleAmqpClient cannot call amqp_maybe_release_buffers() until it no longer holds any references to frames returned from amqp_simple_wait_frame(). Once amqp_maybe_release_buffers() (or amqp_maybe_release_buffers_on_channel()) any outstanding frames are considered freed and their use will lead to undefined behavior. SimpleAmqpClient maintains a list of frames that it has seen from the broker, in ChannelImpl::MaybeReleaseBuffersOnChannel it checks to see if there are any frames that it still has a reference to before calling amqp_maybe_release_buffers() -Alan On Wed, Apr 9, 2014 at 3:17 PM, Dushin Fred wrote: > That works for me, too. Thanks, Abhay. > > Alan, would you like a pull request? Is there a timeline for folding this > an the other recent fixes into master? Anything I can do to help expedite? > > -Fred > > On Apr 9, 2014, at 4:54 AM, Abhay Rajure wrote: > > Try this -- > > In File SimpleAmqpClient/src/ChannelImpl.cpp, > call amqp_maybe_release_buffers_on_channel() in ReturnChannel(). > > void ChannelImpl::ReturnChannel(amqp_chanel_t channel) > { > +amqp_maybe_release_buffers_on_channel(m_connection, channel); > m_free_channels.push(channel); > } > > > > _______________________________________________ > rabbitmq-discuss mailing list > rabbitmq-discuss at lists.rabbitmq.com > https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From matthias at rabbitmq.com Thu Apr 10 07:10:48 2014 From: matthias at rabbitmq.com (Matthias Radestock) Date: Thu, 10 Apr 2014 07:10:48 +0100 Subject: [rabbitmq-discuss] RabbitMQ callback using some listener In-Reply-To: <9550239.166765.1397108539753.JavaMail.nabble@sam.nabble.com> References: <9550239.166765.1397108539753.JavaMail.nabble@sam.nabble.com> Message-ID: <534635E8.7050902@rabbitmq.com> please keep the list on cc. On 10/04/14 06:42, manjay.bit2k7 at gmail.com wrote: > Thanks for your response. But I don't want While(true ) thing. It > checks each time in queue. I want something like notifier. Which will > notify Receiver while message is received. That is exactly what the example I linked to - https://github.com/rabbitmq/rabbitmq-dotnet-client/blob/master/projects/examples/client/LowlevelLogTail/src/examples/LowlevelLogTail.cs - shows: HandleBasicDelivery is invoked when a message arrives. No "while (true)" looping involved. The "while (true)" you are seeing there is for decoding individual messages, which, in this example, just so happen to be streams of objects. Matthias. From cfautsch at goodgamestudios.com Thu Apr 10 07:15:41 2014 From: cfautsch at goodgamestudios.com (Claire Fautsch) Date: Thu, 10 Apr 2014 08:15:41 +0200 Subject: [rabbitmq-discuss] Blocking Queue - Close connection by client In-Reply-To: References: Message-ID: Hi Michael and all, Unfortunatelly calling the abort function instead of close, will show exactly the same behaviour, as there is still a flush at the end (I also tested this, and indeed not a solution). The only difference is that Exceptions/Stacktraces are ignroed. The only option I see so far to avoid hanging here, is to set a timeout to the close method. Regards Claire 2014-03-03 9:18 GMT+01:00 Claire Fautsch : > Hi, > > we have following issue with blocking queues: > > Once the memory watermark is reached, the connection from the client gets > blocked by the RabbitMQ server and it is no longer possible to publish > messages. > > If this happens, we are for some time still able to publish messages, > however they remain in the socket's write buffer and are not actually > published until the connection is un-blocked again. Once this buffer is > full, publishing blocks completely. > > After some research we found out, that with a newer version of the > official Java client, it is possible to use blocked connection > notifications to react on this situation. > > Our prefered reaction would be to close the connection, and re-connect to > another broker. Here however comes the problem: It is not possible to close > the connection to the RabbitMQ, as the method to do so, involves a flush on > the socket before the socket is closed and this is blocked as well. > > Is there any possibility for a force-closed on the client side? Or is the > only possibility to open a new connection without closing the old one, and > "let it die"? > > Re-writing the SocketFrameHandler would of course also be a possibility, > but one we would (if possible) like to avoid. (see also discussion here : > https://github.com/rabbitmq/rabbitmq-java-client/issues/11) > > > Additionally, when we see blocking connections occurring, we frequently > see following stack trace in our logs, where we are not absolutely sure > where it comes from, or rather why it occurs > > com.rabbitmq.client.ShutdownSignalException: connection error; reason: > java.io.EOFException > > at com.rabbitmq.utility.ValueOrException.getValue(ValueOrException.java:67) > > at > com.rabbitmq.utility.BlockingValueOrException.uninterruptibleGetValue(BlockingValueOrException.java:37) > > at > com.rabbitmq.client.impl.AMQChannel$BlockingRpcContinuation.getReply(AMQChannel.java:349) > > at com.rabbitmq.client.impl.ChannelN.close(ChannelN.java:569) > > at com.rabbitmq.client.impl.ChannelN.close(ChannelN.java:501) > > ... > > Caused by: java.io.EOFException > > at com.rabbitmq.client.impl.ChannelN.close(ChannelN.java:494)at > java.io.DataInputStream.readUnsignedByte(Unknown Source) > > at com.rabbitmq.client.impl.Frame.readFrom(Frame.java:95) > > at > com.rabbitmq.client.impl.SocketFrameHandler.readFrame(SocketFrameHandler.java:131) > > at > com.rabbitmq.client.impl.AMQConnection$MainLoop.run(AMQConnection.java:515) > > > For information, we are currently using Java client, > com.rabbitmq:amqp-client version 3.1.4 > > and com.zanox.lib.rabbiteasy version:rabbiteasy-core version 1.2.0, for > the connection handling (SingleConnectionFactory) and publishing. > > Thanks in advance for any hint or ideas. > > Regards > > Claire > > > > > -- *Claire Fautsch* Server Developer cfautsch at goodgamestudios.com Goodgame Studios Theodorstr. 42-90, House 9 22761 Hamburg, Germany Phone: +49 (0)40 219 880 -0 *www.goodgamestudios.com * Goodgame Studios is a branch of Altigi GmbH Altigi GmbH, District court Hamburg, HRB 99869 Board of directors: Dr. Kai Wawrzinek, Dr. Christian Wawrzinek, Fabian Ritter -------------- next part -------------- An HTML attachment was scrubbed... URL: From carl.hoerberg at gmail.com Thu Apr 10 08:05:00 2014 From: carl.hoerberg at gmail.com (carlhoerberg) Date: Thu, 10 Apr 2014 00:05:00 -0700 (PDT) Subject: [rabbitmq-discuss] HiPE slower In-Reply-To: <5345A250.4040300@rabbitmq.com> References: <1397034121943-34671.post@n5.nabble.com> <5345A250.4040300@rabbitmq.com> Message-ID: <1397113500168-34729.post@n5.nabble.com> Yes, using the latest versions of both erlang and rabbitmq. Ok, thank you for the pointer to scheduler configurations! -- View this message in context: http://rabbitmq.1065348.n5.nabble.com/HiPE-slower-tp34671p34729.html Sent from the RabbitMQ mailing list archive at Nabble.com. From joey.jiaojg at gmail.com Thu Apr 10 09:06:41 2014 From: joey.jiaojg at gmail.com (Joey Jiao) Date: Thu, 10 Apr 2014 16:06:41 +0800 Subject: [rabbitmq-discuss] Fwd: flow control shortcut In-Reply-To: References: Message-ID: Hi, Can anyone kindly answer this question? Or I have to rely on myself to investigate a patchset? ---------- Forwarded message ---------- From: Joey Jiao Date: 2014-03-26 16:01 GMT+08:00 Subject: flow control shortcut To: Discussions about RabbitMQ Hi, Is there a flow control shortcut to block all connections or all pub connections or all consume connections? And is it safe to reboot machine or restart rabbitmq service at any time? -- -Joey Jiao -- -Joey Jiao -------------- next part -------------- An HTML attachment was scrubbed... URL: From mklishin at gopivotal.com Thu Apr 10 09:11:26 2014 From: mklishin at gopivotal.com (Michael Klishin) Date: Thu, 10 Apr 2014 12:11:26 +0400 Subject: [rabbitmq-discuss] Fwd: flow control shortcut In-Reply-To: References: Message-ID: On 10 April 2014 at 12:09:12, Joey Jiao (joey.jiaojg at gmail.com) wrote: > > Can anyone kindly answer this question? There were 2 replies to your original question: ? http://markmail.org/thread/rvqhhopa5eynsnwu -- MK Software Engineer, Pivotal/RabbitMQ From matthias at rabbitmq.com Thu Apr 10 09:57:25 2014 From: matthias at rabbitmq.com (Matthias Radestock) Date: Thu, 10 Apr 2014 09:57:25 +0100 Subject: [rabbitmq-discuss] Performance bottleneck on inter-node connection In-Reply-To: References: Message-ID: <53465CF5.20607@rabbitmq.com> On 08/04/14 19:44, joseph rouphael wrote: > I am facing a performance bottleneck on the inter-node connection > between two nodes in a cluster. > [...] > {hipe_compile, true}, Have you tried this w/o hipe? > {delegate_count, 32}, Why did you set this? > Queues are located on node1. > > When producing/consuming messages of 1KB on the same node (node 1), I am > able to achieve up to 100K TPS, even more. > > If I produce messages on node 2, and consume messages from node 1; I am > not able to hit more then 15K TPS. Knowing that the CPU load on the two > nodes is very relaxed (90% idle), and no bottleneck on the network > bandwidth between the nodes. > > The same, if I produce on node 1, and consume on node 2; I am not able > to hit more then 15K TPS > > I am suspecting a limitation on the inter-node connection between the nodes. > > > Is there a way to create multiple links between the nodes to scale up? > > If not possible to create multiple links, Is there a way to optimize the > link performance? > > Is there something else I can do to increase the performance between nodes? I suspect your test is latency sensitive rather than bandwidth sensitive, presumably because you are doing something synchronous, e.g. publish a messages and wait for it to arrive at the consumer, then send the next message, etc. So please explain what your test actually does. How are you measuring TPS? Matthias. From mklishin at gopivotal.com Thu Apr 10 10:10:41 2014 From: mklishin at gopivotal.com (Michael Klishin) Date: Thu, 10 Apr 2014 13:10:41 +0400 Subject: [rabbitmq-discuss] Do WebStomp/Sock.JS require a constant connection? In-Reply-To: References: Message-ID: On 1 April 2014 at 09:44:54, James Andrew-Smith (james at archfashion.com.au) wrote: > > Can I make an explicit AJAX call when the page loads or something? > Or should I keep a history of recent messages in a cache somewhere? You can store some state on the server and load it using ?regular? HTTP requests, then use it when opening a WebSTOMP connection.? -- MK Software Engineer, Pivotal/RabbitMQ From mklishin at gopivotal.com Thu Apr 10 10:13:29 2014 From: mklishin at gopivotal.com (Michael Klishin) Date: Thu, 10 Apr 2014 13:13:29 +0400 Subject: [rabbitmq-discuss] Error during WebSocket handshake: Sent non-empty 'Sec-WebSocket-Protocol' header but no response was received In-Reply-To: <1395406371281-34248.post@n5.nabble.com> References: <1395406371281-34248.post@n5.nabble.com> Message-ID: On 21 March 2014 at 16:53:47, joshua__lim (joshua__lim at hotmail.com) wrote: > > WebSocket connection to 'ws://localhost:15674/stomp/websocket' > failed: Error > during WebSocket handshake: Sent non-empty 'Sec-WebSocket-Protocol' > header > but no response was received :8080/stomp.js:374 > > Line 374 is "ws = new klass(url, protocols);?. RabbitMQ Web STOMP uses /stomp. Have you seen/tried code examples on?http://www.rabbitmq.com/web-stomp.html?? -- MK Software Engineer, Pivotal/RabbitMQ From matthias at rabbitmq.com Thu Apr 10 10:21:46 2014 From: matthias at rabbitmq.com (Matthias Radestock) Date: Thu, 10 Apr 2014 10:21:46 +0100 Subject: [rabbitmq-discuss] Unable to start RabbitMQ after upgrading to 3.3.0 In-Reply-To: References: Message-ID: <534662AA.5000307@rabbitmq.com> Amaury, On 08/04/14 10:17, Amaury LEROUX DE LENS wrote: > I'm using RabbitMQ on MacOSX, installed via HomeBrew and since i > upgraded to 3.3.0 i'm not able to start RabbitMQ again. > > I just got this error report in log files. > [...] > =INFO REPORT==== 8-Apr-2014::11:09:06 === > Error description: > {badmatch,{error,not_found}} > Log files (may contain more information): > /usr/local/var/log/rabbitmq/rabbit at localhost.log > /usr/local/var/log/rabbitmq/rabbit at localhost-sasl.log > Stack trace: > [{rabbit_binding,recover_semi_durable_route,3, > [{file,"src/rabbit_binding.erl"},{line,127}]}, > {rabbit_binding,'-recover/2-lc$^0/1-0-',3, > [{file,"src/rabbit_binding.erl"},{line,117}]}, > > {rabbit_binding,recover,2,[{file,"src/rabbit_binding.erl"},{line,117}]}, > {rabbit,recover,0,[{file,"src/rabbit.erl"},{line,620}]}, > {rabbit,'-run_boot_step/1-lc$^1/1-1-',1, > [{file,"src/rabbit.erl"},{line,505}]}, > {rabbit,run_boot_step,1,[{file,"src/rabbit.erl"},{line,504}]}, > > {rabbit,'-start/2-lc$^0/1-0-',1,[{file,"src/rabbit.erl"},{line,471}]}, > {rabbit,start,2,[{file,"src/rabbit.erl"},{line,471}]}] This indicates a serious inconsistency in RabbitMQ's database. Would you mind posting a zip/tgz of the database dir (/usr/local/var/lib/rabbitmq/mnesia/rabbit at localhost) somewhere? After that, if you want to get going and don't mind losing existing data, simply remove that dir. Regards, Matthias. From mes65 at cornell.edu Wed Apr 9 18:04:16 2014 From: mes65 at cornell.edu (Michael Sander) Date: Wed, 9 Apr 2014 13:04:16 -0400 Subject: [rabbitmq-discuss] Rabbit MQ Crash In-Reply-To: <53456B84.6010906@rabbitmq.com> References: <4da4be68-9e3b-4bba-aaa9-e262df330d6f@googlegroups.com> <5342E86E.4060508@rabbitmq.com> <5343117D.7080405@rabbitmq.com> <53431742.5000008@rabbitmq.com> <5343868A.3060207@rabbitmq.com> <53456850.2070205@rabbitmq.com> <53456B84.6010906@rabbitmq.com> Message-ID: Awesome. I normally just pull it from your apt packages. I've been pretty busy the past few days, so it may take me a day or two to pull/compile/install the source. Thank you again. Michael Sander On Wed, Apr 9, 2014 at 11:47 AM, Simon MacMullen wrote: > On 09/04/14 16:33, Matthias Radestock wrote: > >> Prior to that, the fix will be in the nightly builds in a few days. >> > > Actually it will be in tonight's build, I just merged it. > > It would be helpful if you can report back and let us know if this fixes > things for you. > > Cheers, Simon > > -- > Simon MacMullen > RabbitMQ, Pivotal > -------------- next part -------------- An HTML attachment was scrubbed... URL: From tbuehlmann.android at googlemail.com Thu Apr 10 06:06:12 2014 From: tbuehlmann.android at googlemail.com (=?UTF-8?Q?Tobias_B=C3=BChlmann?=) Date: Wed, 9 Apr 2014 22:06:12 -0700 (PDT) Subject: [rabbitmq-discuss] Design Decision In-Reply-To: References: <2f4c1429-5142-4a8f-970e-d0cfb2c63eac@googlegroups.com> Message-ID: <23aa8661-6642-41e4-92c3-6776168b715e@googlegroups.com> Actually, I haven't, but did now. My impression is that I need to have "predefined" queues in order to make this work with a topic exchange. I mean, what would my topic words be like? ids? From qihangzp at gmail.com Thu Apr 10 10:02:30 2014 From: qihangzp at gmail.com (qihang zhang) Date: Thu, 10 Apr 2014 02:02:30 -0700 (PDT) Subject: [rabbitmq-discuss] Does rabbitmq support publishing message with a wildcard? Message-ID: <05f691bb-8f45-4356-ba1f-ef2e68c50244@googlegroups.com> Hi all, Does rabbitmq support publishing message with wildcard? Something I wish like this: declare(exchange, 'topic') bind(q1, exchange, 'user1.client1') bind(q2, exchange, 'user1.client2') bind(q3, exchange, 'user1.client3') exchange.publish(msg, routing_key='user1.*') # all of q1, q2, q3 would receive this msg exchange.publish(msg, routing_key='user1.client1') # only q1 would receive this msg I read the doc and searched about this, found it seems that only binding with wildcart is supported, Now I have to write codes like: declare(exchange_fanout, 'fanout') bind(q1, exchange_fanout) bind(q2, exchange_fanout) bind(q3, exchange_fanout) declare(exchange_direct, 'direct') bind(q1, exchange_direct, 'user1.client1') bind(q2, exchange_direct, 'user1.client2') bind(q3, exchange_direct, 'user1.client3') exchange_fanout.publish(msg) # all of q1, q2, q3 would receive this msg exchange_direct.publish(msg, routing_key='user1.client1') # only q1 would receive this msg Is there other clearer way to do these? I think this is a pretty common thing when we want publish msgs peer-to-peer and peer-to-group. Many thanks! :) -------------- next part -------------- An HTML attachment was scrubbed... URL: From simon at rabbitmq.com Thu Apr 10 11:06:49 2014 From: simon at rabbitmq.com (Simon MacMullen) Date: Thu, 10 Apr 2014 11:06:49 +0100 Subject: [rabbitmq-discuss] Possible memory leak in the management plugin In-Reply-To: <1397106237672-34725.post@n5.nabble.com> References: <8a1af6ed-c599-42a3-85c3-c5dc22a99bd8@googlegroups.com> <520CB29A.1060304@rabbitmq.com> <1397020502914-34663.post@n5.nabble.com> <534521F1.9030707@rabbitmq.com> <1397058685615-34697.post@n5.nabble.com> <53457674.1040704@rabbitmq.com> <1397106237672-34725.post@n5.nabble.com> Message-ID: <53466D39.4050906@rabbitmq.com> On 10/04/14 06:03, Pavel wrote: > Thanks for answers, Simon! Now everything makes much more sense! > > Here is the program I used (requires com.rabbitmq:amqp-client:3.3.0 in > classpath): > https://gist.github.com/maisenovich/10339925 Thank you! > As you suggested, with {force_fine_statistics,false} Rabbit was able > to survive the same test, stabilizing around 2Gb mark with most RAM taken by > connection_stats and queue_procs. So it is certainly a working option, > although at the cost of reduced visibility. That's good to hear. > Then I added a modification to the test - every time message is published to > a random exchange (instead of publishing 1000 messages to each exchange > sequentially). This produced an interesting result: rabbit_mgmt_db process > was GCed properly and remained at stable memory use. However > aggregated_stats table kept growing in size until all RAM (up to a high > watermark) was taken. > > For example, {process_info(rabbit_mgmt_db, memory),ETS sizes} went from > > {{memory,542484424}, > [{5730386,1046,205243}, > {5734488,2005,384904}, > {5738585,2006,126545}, > {5742682,1,175}, > {5746779,1,175}, > {5750876,1,1059}, > {5754973,1009026,79415774}, > {5759070,2001250,49359758}, > {5763167,1003620,57435178}]} > > at one point to this some time later: > > {{memory,434953160}, > [{5730386,1046,205243}, > {5734488,2005,384904}, > {5738585,2006,132875}, > {5742682,1,175}, > {5746779,1,175}, > {5750876,1,1059}, > {5754973,1009026,147761213}, > {5759070,2001250,49359758}, > {5763167,1003834,57452431}]} > > Note, that number of items for (5754973) didn't change, but the size almost > doubled. This is not too surprising. That table contains one row for every combination of things that can show message rates, and each row contains some history for that thing. > I was trying to understand "Event-GCing" portion of > rabbit_mgmt_stats.erl > > , but that's beyond me at the moment. Could you please describe in a few > words, how and when aggregated_stats supposed to be cleaned up? The GCing is about deleting old history from each row. This is a relatively expensive operation, so the DB loops round GCing 1% of rows (or 100 rows, whichever is larger) every 5s. That means that we can keep a bit more history around than we're configured to, just because we haven't got round to GCing it yet. (Probably this is obvious but: this "GC" has nothing to do with the Erlang VM GC we were previously discussing.) > Also, from your earlier reply it sounds like closing and reopening channels > (sometimes) would help to keep mgmt_db from expanding. Is this something > generally recommended to do (short-live vs long-live channels)? That will short-cut the history GCing above, since it will just drop all the rows relating to that channel. You might possibly decide to do that to work round this performance issue, but it's certainly not "best practice" - you should be able to have long-lived channels! Cheers, Simon -- Simon MacMullen RabbitMQ, Pivotal From mklishin at gopivotal.com Thu Apr 10 11:07:12 2014 From: mklishin at gopivotal.com (Michael Klishin) Date: Thu, 10 Apr 2014 14:07:12 +0400 Subject: [rabbitmq-discuss] Design Decision In-Reply-To: <2f4c1429-5142-4a8f-970e-d0cfb2c63eac@googlegroups.com> References: <2f4c1429-5142-4a8f-970e-d0cfb2c63eac@googlegroups.com> Message-ID: On 9 April 2014 at 18:03:44, Tobias B?hlmann (tbuehlmann.android at googlemail.com) wrote: > > So, the problem: I have a relational database with a notifications > table where each notification has many associated apps. Each > app has itself many devices. Now, a producer will check the table > for new notifications with a specific state. If there are any, > grab them and produce a message for each device in each app associated > with the new notification. After that, some consumer workers > will work the queue(s) off and deliver the messages to the devices. Tobias, Can you elaborate how apps and devices are named? Does the publisher know much about them? This is a very important detail. If I understand your scenario correctly, all devices across all apps should get notifications of a specific kind. If so, you can make your consumers declare server-named queues (their names won?t be known to the publisher) and bind them to a topic exchange with a pattern like this: notifications.[type] As long as routing can happen on the type, the publisher won?t have to know anything about consumers. This is largely the point of the exchange/queue separation in the protocol.? -- MK Software Engineer, Pivotal/RabbitMQ From simon at rabbitmq.com Thu Apr 10 11:08:15 2014 From: simon at rabbitmq.com (Simon MacMullen) Date: Thu, 10 Apr 2014 11:08:15 +0100 Subject: [rabbitmq-discuss] Does rabbitmq support publishing message with a wildcard? In-Reply-To: <05f691bb-8f45-4356-ba1f-ef2e68c50244@googlegroups.com> References: <05f691bb-8f45-4356-ba1f-ef2e68c50244@googlegroups.com> Message-ID: <53466D8F.30106@rabbitmq.com> On 10/04/14 10:02, qihang zhang wrote: > Does rabbitmq support publishing message with wildcard? Not out of the box. Alvaro wrote a plugin to do this: https://github.com/videlalvaro/rabbitmq-rtopic-exchange Cheers, Simon -- Simon MacMullen RabbitMQ, Pivotal From mklishin at gopivotal.com Thu Apr 10 11:08:41 2014 From: mklishin at gopivotal.com (Michael Klishin) Date: Thu, 10 Apr 2014 14:08:41 +0400 Subject: [rabbitmq-discuss] Does rabbitmq support publishing message with a wildcard? In-Reply-To: <05f691bb-8f45-4356-ba1f-ef2e68c50244@googlegroups.com> References: <05f691bb-8f45-4356-ba1f-ef2e68c50244@googlegroups.com> Message-ID: On 10 April 2014 at 14:02:04, qihang zhang (qihangzp at gmail.com) wrote: > > Does rabbitmq support publishing message with wildcard? > Something I wish like this: > declare(exchange, 'topic') > > bind(q1, exchange, 'user1.client1') > > bind(q2, exchange, 'user1.client2') > > bind(q3, exchange, 'user1.client3') > > > > exchange.publish(msg, routing_key='user1.*') # all of q1, > q2, q3 would receive this msg > > exchange.publish(msg, routing_key='user1.client1') # > only q1 would receive this msg > > > I read the doc and searched about this, found it seems that only > binding with wildcart is supported, > Now I have to write codes like: > > declare(exchange_fanout, 'fanout') > > > > bind(q1, exchange_fanout) > > > > bind(q2, exchange_fanout) > > bind(q3, exchange_fanout) > > > > declare(exchange_direct, 'direct') > > bind(q1, exchange_direct, 'user1.client1') > > > > bind(q2, exchange_direct, 'user1.client2') > > > > bind(q3, exchange_direct, 'user1.client3') > > > > > > exchange_fanout.publish(msg) # all of q1, q2, q3 would receive > this msg > > exchange_direct.publish(msg, routing_key='user1.client1') > # only q1 would receive this msg > > > Is there other clearer way to do these? I think this is a pretty > common thing when we want publish msgs peer-to-peer and peer-to-group. > Many thanks! :) This is what topic exchanges are for, see tutorial 5 (and maybe 3, to contrast): http://www.rabbitmq.com/getstarted.html? -- MK Software Engineer, Pivotal/RabbitMQ From ravindrapandey1982 at gmail.com Thu Apr 10 12:05:45 2014 From: ravindrapandey1982 at gmail.com (Ravi Pandey) Date: Thu, 10 Apr 2014 16:35:45 +0530 Subject: [rabbitmq-discuss] (no subject) Message-ID: Hi I have a question - How to keep continuous connection between consumer and producer? I mean when producer sends messages - consumer process it and send back with callback queue if I used rpc procedure and connection will closed. Now I want connection between consumer and producer should be open. How to achieve this? Thanks Ravi -------------- next part -------------- An HTML attachment was scrubbed... URL: From ravindrapandey1982 at gmail.com Thu Apr 10 12:06:17 2014 From: ravindrapandey1982 at gmail.com (Ravi Pandey) Date: Thu, 10 Apr 2014 16:36:17 +0530 Subject: [rabbitmq-discuss] Continuous connection between consumer and producer Message-ID: Hi I have a question - How to keep continuous connection between consumer and producer? I mean when producer sends messages - consumer process it and send back with callback queue if I used rpc procedure and connection will closed. Now I want connection between consumer and producer should be open. How to achieve this? Thanks -------------- next part -------------- An HTML attachment was scrubbed... URL: From Mahendrakumary at neelinfo.com Thu Apr 10 11:56:18 2014 From: Mahendrakumary at neelinfo.com (Mahendrakumar Yadavalli) Date: Thu, 10 Apr 2014 03:56:18 -0700 Subject: [rabbitmq-discuss] RabbitMQ Message consuming issue (when consumer keepting idle) Message-ID: <0AE0D43E7CAC0342AE4BCDD4A4F5F452011AE700@NII-WINSRV.niiwin.local> Hi Teaches, This is Mahendra working as software engineer . In our project we are using RabbitMq to push the messages from server to client applications. Here client is android based application. Whenever there is a change in Database, we need to pushing the new data to the clients without client server call. We got the sample code form the RabbitMQ site and implemented an application where the producer implemented in web application and consumer was implemented in android application. Both the producer and consumer are working fine. We got an issue like "when the android device kept idle for a long time, and start working on it after some time, then messages are not consuming from consumer side. once we delete the queue then it will work fine". The problem is when we kept the client device idle for long time this issue occurs. I have gone through the google ,I didn't find the solution. Please help me. Thanks in Adv. Can you please find the attached mail which contains Android client consumer application sample code(from android client application). And Here is the sample code for java producer code(from java application): Whenever we need to push the data from producer to consumer, we just invoke this below "pushData"method. Sample code: public void pushData(Set queueNames, String message) { try { conn = factory.newConnection(); channel = conn.createChannel(); } catch (Exception e) { log.error("Error while creating connetion/channel for RabbitMQ Server:"+e); } try { for (String queueName : queueNames) { try{ // channel.exchangeDeclare(EXCHANGE_NAME, "direct", true); channel.queueDeclare(queueName, false, false, false, null); channel.basicPublish("", queueName, null, message.getBytes()); } catch (Exception e) { log.error("Error while RabbitMQ Server publishing the message"+ message+ "\n and for the queue "+queueName+"\n" +e); } } message = ""; channel.close(); conn.close(); } catch (IOException e) { log.error("Error while RabbitMQ Server publishing the message"+ e); } } Let me know any information needed. Thanks&Regards Mahendra.y Software Engineer Neelinfo Software Solutions,Hyderbad,India. -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- An embedded message was scrubbed... From: Manasa Potula Subject: RabbitMQ Client side consumer code. Date: Thu, 10 Apr 2014 03:04:54 -0700 Size: 33520 URL: From tim at rabbitmq.com Thu Apr 10 12:37:57 2014 From: tim at rabbitmq.com (Tim Watson) Date: Thu, 10 Apr 2014 12:37:57 +0100 Subject: [rabbitmq-discuss] Continuous connection between consumer and producer In-Reply-To: References: Message-ID: <820C6E57-7289-4D29-951E-2A9939EB80CF@rabbitmq.com> Producers and consumers are typically decoupled in messaging systems, but the presence of an intermediary (i.e., the broker). What client library/language are you using? The connections between the producer/broker and consumer/broker will be as long-lived as the code that sets them up and closes them. Cheers, Tim On 10 Apr 2014, at 12:06, Ravi Pandey wrote: > Hi > > I have a question - How to keep continuous connection between consumer and producer? > > I mean when producer sends messages - consumer process it and send back with callback queue if I used rpc procedure and connection will closed. > > Now I want connection between consumer and producer should be open. How to achieve this? > > Thanks > _______________________________________________ > rabbitmq-discuss mailing list > rabbitmq-discuss at lists.rabbitmq.com > https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: From fred at dushin.net Thu Apr 10 12:45:55 2014 From: fred at dushin.net (Dushin Fred) Date: Thu, 10 Apr 2014 07:45:55 -0400 Subject: [rabbitmq-discuss] rabbitmq-c memory leak? In-Reply-To: References: <673BC989-D49B-4F23-B57B-04D7502FDDEB@dushin.net> Message-ID: Sorry, I should have been more clear. The original issue I hit was from a build off the 2.3 tag (418f31f). So the patch I applied was slightly different: > git diff diff --git a/src/ChannelImpl.cpp b/src/ChannelImpl.cpp index 9749556..864502b 100644 --- a/src/ChannelImpl.cpp +++ b/src/ChannelImpl.cpp @@ -152,6 +152,7 @@ amqp_channel_t ChannelImpl::GetChannel() void ChannelImpl::ReturnChannel(amqp_channel_t channel) { + amqp_maybe_release_buffers_on_channel(m_connection, channel); m_channels.at(channel) = CS_Open; m_last_used_channel = channel; } I fully respect that this may not be the "right" fix, but in my limited scenario (the sender.cpp program I posted off github), it does the trick. I have done zero analysis of the right fix, but I'd be happy to crack open the code and look into it, if the root cause of the bug is not immediately evident to you. Otherwise, if you have any thoughts, feel free to let me know. And of course I would prefer (as I suspect many would) that the fix go into master, as this bug seems to effectively render the C++ client library unsuitable for production, and I would prefer not to maintain a fix that is not endorsed by the author/maintainer. -Fred On Apr 10, 2014, at 1:11 AM, Alan Antonuk wrote: > What version of SimpleAmqpClient are you using? The lines surrounding your change don't seem to match what is currently in the master branch of SimpleAmqpClient. I believe behavior you're seeing was addressed a while ago (see Issue #56). Please download the master branch and build from that, you will also need rabbitmq-c v0.4.1 or newer for this to work. > > FYI: the patch you are suggesting using will likely cause undefined behavior. Because of the way rabbitmq-c's memory management works, the SimpleAmqpClient cannot call amqp_maybe_release_buffers() until it no longer holds any references to frames returned from amqp_simple_wait_frame(). Once amqp_maybe_release_buffers() (or amqp_maybe_release_buffers_on_channel()) any outstanding frames are considered freed and their use will lead to undefined behavior. SimpleAmqpClient maintains a list of frames that it has seen from the broker, in ChannelImpl::MaybeReleaseBuffersOnChannel it checks to see if there are any frames that it still has a reference to before calling amqp_maybe_release_buffers() > > -Alan > > > On Wed, Apr 9, 2014 at 3:17 PM, Dushin Fred wrote: > That works for me, too. Thanks, Abhay. > > Alan, would you like a pull request? Is there a timeline for folding this an the other recent fixes into master? Anything I can do to help expedite? > > -Fred > > On Apr 9, 2014, at 4:54 AM, Abhay Rajure wrote: > >> Try this -- >> >> In File SimpleAmqpClient/src/ChannelImpl.cpp, >> call amqp_maybe_release_buffers_on_channel() in ReturnChannel(). >> >> void ChannelImpl::ReturnChannel(amqp_chanel_t channel) >> { >> +amqp_maybe_release_buffers_on_channel(m_connection, channel); >> m_free_channels.push(channel); >> } > > > _______________________________________________ > rabbitmq-discuss mailing list > rabbitmq-discuss at lists.rabbitmq.com > https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss > > > _______________________________________________ > rabbitmq-discuss mailing list > rabbitmq-discuss at lists.rabbitmq.com > https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: From mklishin at gopivotal.com Thu Apr 10 12:47:59 2014 From: mklishin at gopivotal.com (Michael Klishin) Date: Thu, 10 Apr 2014 15:47:59 +0400 Subject: [rabbitmq-discuss] Continuous connection between consumer and producer In-Reply-To: References: Message-ID: ?On 10 April 2014 at 15:10:03, Ravi Pandey (ravindrapandey1982 at gmail.com) wrote: > > I mean when producer sends messages - consumer process it and > send back with callback queue if I used rpc procedure and connection > will closed. The request/reply pattern does not assume either publisher or producer has to close the connection. > Now I want connection between consumer and producer should be > open. How to achieve this? Have a consumer for replies in the publisher app. Note that the notion of ?connection between producer and consumer? is not really true: both are connected to RabbitMQ and communicate with messages asynchronously. -- MK Software Engineer, Pivotal/RabbitMQ From mklishin at gopivotal.com Thu Apr 10 12:51:50 2014 From: mklishin at gopivotal.com (Michael Klishin) Date: Thu, 10 Apr 2014 15:51:50 +0400 Subject: [rabbitmq-discuss] RabbitMQ Message consuming issue (when consumer keepting idle) In-Reply-To: <0AE0D43E7CAC0342AE4BCDD4A4F5F452011AE700@NII-WINSRV.niiwin.local> References: <0AE0D43E7CAC0342AE4BCDD4A4F5F452011AE700@NII-WINSRV.niiwin.local> Message-ID: On 10 April 2014 at 14:56:28, Mahendrakumar Yadavalli (mahendrakumary at neelinfo.com) wrote: > > Both the producer and consumer are working fine. We got an issue > like ?when the android device kept idle for a long time, and start > working on it after some time, then messages are not consuming > from consumer side. once we delete the queue then it will work > fine?. The problem is when we kept the client device idle for long > time this issue occurs. There have to be some error messages somewhere. One possible reason is that when a device goes idle for a long time, Android hibernates the app entirely and it does not send out heartbeats, so RabbitMQ considers the connection dead. There should be warnings in RabbitMQ log about this. -- MK Software Engineer, Pivotal/RabbitMQ From ravindrapandey1982 at gmail.com Thu Apr 10 12:57:37 2014 From: ravindrapandey1982 at gmail.com (Ravi Pandey) Date: Thu, 10 Apr 2014 17:27:37 +0530 Subject: [rabbitmq-discuss] Continuous connection between consumer and producer In-Reply-To: <820C6E57-7289-4D29-951E-2A9939EB80CF@rabbitmq.com> References: <820C6E57-7289-4D29-951E-2A9939EB80CF@rabbitmq.com> Message-ID: I am using php-amqplib On Thu, Apr 10, 2014 at 5:07 PM, Tim Watson wrote: > Producers and consumers are typically decoupled in messaging systems, but > the presence of an intermediary (i.e., the broker). What client > library/language are you using? The connections between the producer/broker > and consumer/broker will be as long-lived as the code that sets them up and > closes them. > > Cheers, > Tim > > On 10 Apr 2014, at 12:06, Ravi Pandey wrote: > > Hi > > I have a question - How to keep continuous connection between consumer and > producer? > > I mean when producer sends messages - consumer process it and send back > with callback queue if I used rpc procedure and connection will closed. > > Now I want connection between consumer and producer should be open. How to > achieve this? > > Thanks > _______________________________________________ > rabbitmq-discuss mailing list > rabbitmq-discuss at lists.rabbitmq.com > https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss > > > > _______________________________________________ > rabbitmq-discuss mailing list > rabbitmq-discuss at lists.rabbitmq.com > https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From tim at rabbitmq.com Thu Apr 10 13:21:01 2014 From: tim at rabbitmq.com (Tim Watson) Date: Thu, 10 Apr 2014 13:21:01 +0100 Subject: [rabbitmq-discuss] Continuous connection between consumer and producer In-Reply-To: References: <820C6E57-7289-4D29-951E-2A9939EB80CF@rabbitmq.com> Message-ID: <5CC15C7A-BD85-4E2D-811A-BD9B29D9C1EC@rabbitmq.com> On 10 Apr 2014, at 12:57, Ravi Pandey wrote: > I am using php-amqplib > That's perhaps why we see this "disconnecting" business then, since PHP doesn't support long-lived connections. Not sure what you can do about that, given the way PHP works. Perhaps someone with more PHP expertise can comment? As Michael points out though, it's really not an issue when working with request/reply queues, since the connection does not have to be long-lived in order for this to work. Cheers, Tim From ravindrapandey1982 at gmail.com Thu Apr 10 13:29:21 2014 From: ravindrapandey1982 at gmail.com (Ravi Pandey) Date: Thu, 10 Apr 2014 17:59:21 +0530 Subject: [rabbitmq-discuss] Continuous connection between consumer and producer In-Reply-To: <5CC15C7A-BD85-4E2D-811A-BD9B29D9C1EC@rabbitmq.com> References: <820C6E57-7289-4D29-951E-2A9939EB80CF@rabbitmq.com> <5CC15C7A-BD85-4E2D-811A-BD9B29D9C1EC@rabbitmq.com> Message-ID: I want connection between consumer and producer to be open. My requirement is to send messages to producer, so my assumption is that if connection between them is open forever then I can send messages through that open connection :) On Thu, Apr 10, 2014 at 5:51 PM, Tim Watson wrote: > On 10 Apr 2014, at 12:57, Ravi Pandey wrote: > > > I am using php-amqplib > > > > That's perhaps why we see this "disconnecting" business then, since PHP > doesn't support long-lived connections. Not sure what you can do about > that, given the way PHP works. Perhaps someone with more PHP expertise can > comment? > > As Michael points out though, it's really not an issue when working with > request/reply queues, since the connection does not have to be long-lived > in order for this to work. > > Cheers, > Tim > _______________________________________________ > rabbitmq-discuss mailing list > rabbitmq-discuss at lists.rabbitmq.com > https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss > -------------- next part -------------- An HTML attachment was scrubbed... URL: From videlalvaro at gmail.com Thu Apr 10 13:42:07 2014 From: videlalvaro at gmail.com (Alvaro Videla) Date: Thu, 10 Apr 2014 14:42:07 +0200 Subject: [rabbitmq-discuss] (no subject) In-Reply-To: References: Message-ID: Hi, RabbitMQ is a broker based architecture, which means your consumers and producers will have to be connected to the broker. Regards, Alvaro On Thu, Apr 10, 2014 at 1:05 PM, Ravi Pandey wrote: > Hi > > I have a question - How to keep continuous connection between consumer and > producer? > > I mean when producer sends messages - consumer process it and send back with > callback queue if I used rpc procedure and connection will closed. > > Now I want connection between consumer and producer should be open. How to > achieve this? > > Thanks > Ravi > > _______________________________________________ > rabbitmq-discuss mailing list > rabbitmq-discuss at lists.rabbitmq.com > https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss > From pat at munkiisoft.com Thu Apr 10 13:44:06 2014 From: pat at munkiisoft.com (Patrick Long) Date: Thu, 10 Apr 2014 13:44:06 +0100 Subject: [rabbitmq-discuss] Diagnosing errors from SASL log In-Reply-To: <534575BA.1040306@rabbitmq.com> References: <5343F618.3060604@rabbitmq.com> <53456EDB.5030405@rabbitmq.com> <534575BA.1040306@rabbitmq.com> Message-ID: I have attached the rabbit logs from that time On 9 April 2014 17:30, Matthias Radestock wrote: > (+list) > > On 09/04/14 17:17, Patrick Long wrote: > >> I have attached the SASL logs from the time of the initial restart >> (15:13) until the end of that day >> > > cheers. please also send the rabbit.logs. > > Matthias. > -- Patrick Long - Munkiisoft Ltd -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: A01-rabbit.log Type: application/octet-stream Size: 17781 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: A02-rabbit.log Type: application/octet-stream Size: 33133 bytes Desc: not available URL: From mklishin at gopivotal.com Thu Apr 10 13:49:01 2014 From: mklishin at gopivotal.com (Michael Klishin) Date: Thu, 10 Apr 2014 16:49:01 +0400 Subject: [rabbitmq-discuss] Continuous connection between consumer and producer In-Reply-To: References: <820C6E57-7289-4D29-951E-2A9939EB80CF@rabbitmq.com> <5CC15C7A-BD85-4E2D-811A-BD9B29D9C1EC@rabbitmq.com> Message-ID: On 10 April 2014 at 16:31:38, Ravi Pandey (ravindrapandey1982 at gmail.com) wrote: > > I want connection between consumer and producer to be open. > My requirement is to send messages to producer, so my assumption > is that if connection between them is open forever then I can send > messages through that open connection :) Perhaps we should reiterate. ?* There is *no* direct connection from producer to consumer. Both connect to RabbitMQ. ?* PHP?s request model makes it at least challenging to keep long-lived connections ?* Consumers which do not run as part of the Web server can and should use long lived connections? -- MK Software Engineer, Pivotal/RabbitMQ From aaron.kenny at gmail.com Thu Apr 10 13:30:56 2014 From: aaron.kenny at gmail.com (aaron kenny) Date: Thu, 10 Apr 2014 05:30:56 -0700 (PDT) Subject: [rabbitmq-discuss] connot connect Message-ID: <64c98912-00fd-41b3-aa47-26d033b9a2ad@googlegroups.com> Hello All, below script wont create a connection for me and I'm a little unsure how to determine what might be causing the issue. Im connecting to a server which im told defo works. As a test I changed the connection host to local/guest/guest and it work fine. Any ideas would be much appriciated as im totally out of ideas. channel(); $channel->exchange_declare('gbs_prod_x2', 'direct', false, false, false); $channel->queue_declare("", false, true, true, false); if(!empty($argv[1])) { $data = implode(' ', array_slice($argv, 2)); } else if(isset($_REQUEST['data'])) { $data = urldecode($_REQUEST['data']); } else { $data = "Hello from Aaron in London"; } //SET THE SEVERITY if(empty($severity)) { if(isset($_REQUEST['severity'])) { $severity = $_REQUEST['severity']; } else if(!empty($argv[1])) { $severity = $argv[1]; } else { $severity = "info"; } } echo $severity; $msg = new AMQPMessage($data); $channel->basic_publish($msg, 'direct_logs', $severity); echo " [x] Sent ",$severity,':',$data," \n"; $channel->close(); $connection->close(); ?> -------------- next part -------------- An HTML attachment was scrubbed... URL: From mklishin at gopivotal.com Thu Apr 10 14:18:50 2014 From: mklishin at gopivotal.com (Michael Klishin) Date: Thu, 10 Apr 2014 17:18:50 +0400 Subject: [rabbitmq-discuss] connot connect In-Reply-To: <64c98912-00fd-41b3-aa47-26d033b9a2ad@googlegroups.com> References: <64c98912-00fd-41b3-aa47-26d033b9a2ad@googlegroups.com> Message-ID: On 10 April 2014 at 17:16:14, aaron kenny (aaron.kenny at gmail.com) wrote: > > below script wont create a connection for me and I'm a little > unsure how to determine what might be causing the issue. Im connecting > to a server which im told defo works. As a test I changed the connection > host to local/guest/guest and it work fine. As of RabbitMQ 3.3, guest can only connect from localhost. This can be changed but you really should create a separate user if you intend to connect from remote machines: http://www.rabbitmq.com/blog/2014/04/02/breaking-things-with-rabbitmq-3-3/ http://www.rabbitmq.com/access-control.html? -- MK Software Engineer, Pivotal/RabbitMQ From simon at rabbitmq.com Thu Apr 10 14:20:36 2014 From: simon at rabbitmq.com (Simon MacMullen) Date: Thu, 10 Apr 2014 14:20:36 +0100 Subject: [rabbitmq-discuss] Announcing the community plugin repository Message-ID: <53469AA4.9010803@rabbitmq.com> Hi all. There are quite a few interesting plugins for RabbitMQ which don't ship with the broker. The trouble is that it's been hard to find these plugins, and it's often been hard to install them too. So today I'd like to present this: http://www.rabbitmq.com/community-plugins.html It's a directory of some interesting plugins that are available, with a consistent way to find binary (and source) downloads. The idea is that we'll build all the plugins when a new release make it necessary, and when a plugin author asks us to :) We're only going to add plugins with the author's permission. So currently the only plugins there are ones that were written by RabbitMQ team members. So if you've written a plugin and would like us to add it to the repository, please get in touch! Cheers, Simon -- Simon MacMullen RabbitMQ, Pivotal From alan.antonuk at gmail.com Thu Apr 10 14:21:30 2014 From: alan.antonuk at gmail.com (Alan Antonuk) Date: Thu, 10 Apr 2014 13:21:30 +0000 Subject: [rabbitmq-discuss] rabbitmq-c memory leak? References: <673BC989-D49B-4F23-B57B-04D7502FDDEB@dushin.net> Message-ID: Do you get the same problems building it off of the master branch? On Thu, Apr 10, 2014 at 4:46 AM, Dushin Fred wrote: > Sorry, I should have been more clear. > > The original issue I hit was from a build off the 2.3 tag (418f31f). So > the patch I applied was slightly different: > > > git diff > diff --git a/src/ChannelImpl.cpp b/src/ChannelImpl.cpp > index 9749556..864502b 100644 > --- a/src/ChannelImpl.cpp > +++ b/src/ChannelImpl.cpp > @@ -152,6 +152,7 @@ amqp_channel_t ChannelImpl::GetChannel() > > void ChannelImpl::ReturnChannel(amqp_channel_t channel) > { > + amqp_maybe_release_buffers_on_channel(m_connection, channel); > m_channels.at(channel) = CS_Open; > m_last_used_channel = channel; > } > > I fully respect that this may not be the "right" fix, but in my limited > scenario (the sender.cpp program I posted off github), it does the trick. > > I have done zero analysis of the right fix, but I'd be happy to crack open > the code and look into it, if the root cause of the bug is not immediately > evident to you. Otherwise, if you have any thoughts, feel free to let me > know. > > And of course I would prefer (as I suspect many would) that the fix go > into master, as this bug seems to effectively render the C++ client library > unsuitable for production, and I would prefer not to maintain a fix that is > not endorsed by the author/maintainer. > > -Fred > > On Apr 10, 2014, at 1:11 AM, Alan Antonuk wrote: > > What version of SimpleAmqpClient are you using? The lines surrounding your > change don't seem to match what is currentlyin the master branch of SimpleAmqpClient. I believe behavior you're seeing > was addressed a while ago (see Issue #56). > Please download the master branch and build from that, you will also need > rabbitmq-c v0.4.1 or newer for this to work. > > FYI: the patch you are suggesting using will likely cause undefined > behavior. Because of the way rabbitmq-c's memory management works, the > SimpleAmqpClient cannot call amqp_maybe_release_buffers() until it no > longer holds any references to frames returned from > amqp_simple_wait_frame(). Once amqp_maybe_release_buffers() (or > amqp_maybe_release_buffers_on_channel()) any outstanding frames are > considered freed and their use will lead to undefined behavior. > SimpleAmqpClient maintains a list of frames that it has seen from the > broker, in ChannelImpl::MaybeReleaseBuffersOnChannel it > checks to see if there are any frames that it still has a reference to > before calling amqp_maybe_release_buffers() > > -Alan > > > On Wed, Apr 9, 2014 at 3:17 PM, Dushin Fred wrote: > >> That works for me, too. Thanks, Abhay. >> >> Alan, would you like a pull request? Is there a timeline for folding >> this an the other recent fixes into master? Anything I can do to help >> expedite? >> >> -Fred >> >> On Apr 9, 2014, at 4:54 AM, Abhay Rajure wrote: >> >> Try this -- >> >> In File SimpleAmqpClient/src/ChannelImpl.cpp, >> call amqp_maybe_release_buffers_on_channel() in ReturnChannel(). >> >> void ChannelImpl::ReturnChannel(amqp_chanel_t channel) >> { >> +amqp_maybe_release_buffers_on_channel(m_connection, channel); >> m_free_channels.push(channel); >> } >> >> >> >> _______________________________________________ >> rabbitmq-discuss mailing list >> rabbitmq-discuss at lists.rabbitmq.com >> https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss >> >> > _______________________________________________ > rabbitmq-discuss mailing list > rabbitmq-discuss at lists.rabbitmq.com > https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss > > > _______________________________________________ > rabbitmq-discuss mailing list > rabbitmq-discuss at lists.rabbitmq.com > https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss > -------------- next part -------------- An HTML attachment was scrubbed... URL: From videlalvaro at gmail.com Thu Apr 10 14:23:01 2014 From: videlalvaro at gmail.com (Alvaro Videla) Date: Thu, 10 Apr 2014 15:23:01 +0200 Subject: [rabbitmq-discuss] Announcing the community plugin repository In-Reply-To: <53469AA4.9010803@rabbitmq.com> References: <53469AA4.9010803@rabbitmq.com> Message-ID: This is great, thanks! On Thu, Apr 10, 2014 at 3:20 PM, Simon MacMullen wrote: > Hi all. > > There are quite a few interesting plugins for RabbitMQ which don't ship with > the broker. The trouble is that it's been hard to find these plugins, and > it's often been hard to install them too. So today I'd like to present this: > > http://www.rabbitmq.com/community-plugins.html > > It's a directory of some interesting plugins that are available, with a > consistent way to find binary (and source) downloads. The idea is that we'll > build all the plugins when a new release make it necessary, and when a > plugin author asks us to :) > > We're only going to add plugins with the author's permission. So currently > the only plugins there are ones that were written by RabbitMQ team members. > > So if you've written a plugin and would like us to add it to the repository, > please get in touch! > > Cheers, Simon > > -- > Simon MacMullen > RabbitMQ, Pivotal > _______________________________________________ > rabbitmq-discuss mailing list > rabbitmq-discuss at lists.rabbitmq.com > https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss From matthias at rabbitmq.com Thu Apr 10 14:30:03 2014 From: matthias at rabbitmq.com (Matthias Radestock) Date: Thu, 10 Apr 2014 14:30:03 +0100 Subject: [rabbitmq-discuss] Diagnosing errors from SASL log In-Reply-To: References: <5343F618.3060604@rabbitmq.com> <53456EDB.5030405@rabbitmq.com> <534575BA.1040306@rabbitmq.com> Message-ID: <53469CDB.50001@rabbitmq.com> On 10/04/14 13:44, Patrick Long wrote: > I have attached the rabbit logs from that time Excellent. That confirms what I suspected, namely that, as far as rabbit is concerned, a network partition occurred. That can make the bunny quite unhappy. Matthias. From ravindrapandey1982 at gmail.com Thu Apr 10 14:46:44 2014 From: ravindrapandey1982 at gmail.com (Ravi Pandey) Date: Thu, 10 Apr 2014 19:16:44 +0530 Subject: [rabbitmq-discuss] How many max queues RabbitMQ can handle at a time? Message-ID: Hi, I want to know how many maximum queues RabbitMQ can handle on a single server? Does it depend on RAM? Does it depends on erlang processes? Thanks Ravi -------------- next part -------------- An HTML attachment was scrubbed... URL: From mklishin at gopivotal.com Thu Apr 10 14:50:54 2014 From: mklishin at gopivotal.com (Michael Klishin) Date: Thu, 10 Apr 2014 17:50:54 +0400 Subject: [rabbitmq-discuss] How many max queues RabbitMQ can handle at a time? In-Reply-To: References: Message-ID: On 10 April 2014 at 17:48:56, Ravi Pandey (ravindrapandey1982 at gmail.com) wrote: > > > I want to know how many maximum queues RabbitMQ can handle on a > single server? > Does it depend on RAM? Does it depends on erlang processes? Both but primarily on RAM. Tens of thousands of queues should be very realistic on reasonably recent hardware. Benchmark your own environment to find out more.? -- MK Software Engineer, Pivotal/RabbitMQ From simon at rabbitmq.com Thu Apr 10 15:03:29 2014 From: simon at rabbitmq.com (Simon MacMullen) Date: Thu, 10 Apr 2014 15:03:29 +0100 Subject: [rabbitmq-discuss] Upgrade fail In-Reply-To: <53453249.90808@rabbitmq.com> References: <533D5DFB.5080405@rabbitmq.com> <53453249.90808@rabbitmq.com> Message-ID: <5346A4B1.9020408@rabbitmq.com> On 09/04/14 12:43, Simon MacMullen wrote: > I am not full of ideas here, this looks like more of an issue with > Erlang than RabbitMQ. Interestingly, the release notes for Erlang 17.0 contain: OTP-11558 Fixed a bug where starting Erlang without having an open stdin on fd 0 would sometimes deadlock the emulator when terminating. which sounds like it could describe what you're experiencing. Cheers, Simon -- Simon MacMullen RabbitMQ, Pivotal From ravindrapandey1982 at gmail.com Thu Apr 10 15:14:04 2014 From: ravindrapandey1982 at gmail.com (Ravi Pandey) Date: Thu, 10 Apr 2014 19:44:04 +0530 Subject: [rabbitmq-discuss] How many max queues RabbitMQ can handle at a time? In-Reply-To: References: Message-ID: basically I want technical answer For Ex. 1Gb RAM can hangle this much request ..like this - so that I can decide should I increase RAM or not. Can RabbitMQ Server handle 10 million queues? how much memory will my server need? On Thu, Apr 10, 2014 at 7:20 PM, Michael Klishin wrote: > On 10 April 2014 at 17:48:56, Ravi Pandey (ravindrapandey1982 at gmail.com) > wrote: > > > > > I want to know how many maximum queues RabbitMQ can handle on a > > single server? > > Does it depend on RAM? Does it depends on erlang processes? > > Both but primarily on RAM. Tens of thousands of queues should be very > realistic on reasonably recent hardware. > > Benchmark your own environment to find out more. > -- > MK > > Software Engineer, Pivotal/RabbitMQ > _______________________________________________ > rabbitmq-discuss mailing list > rabbitmq-discuss at lists.rabbitmq.com > https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mklishin at gopivotal.com Thu Apr 10 15:37:37 2014 From: mklishin at gopivotal.com (Michael Klishin) Date: Thu, 10 Apr 2014 18:37:37 +0400 Subject: [rabbitmq-discuss] How many max queues RabbitMQ can handle at a time? In-Reply-To: References: Message-ID: On 10 April 2014 at 18:16:21, Ravi Pandey (ravindrapandey1982 at gmail.com) wrote: > > basically I want technical answer For Ex. 1Gb RAM can hangle > this much request What matters a lot more is how many messages there are in the queues during peaks and their sizes. I just created 30,000 empty queues on a 64 bit machine and see Erlang VM consuming 589 MB of RAM on peak (slightly lower after some time). The most technical answer I can think of is: benchmark on your own OS/architecture/hardware/workload.? -- MK Software Engineer, Pivotal/RabbitMQ From mklishin at gopivotal.com Thu Apr 10 15:41:10 2014 From: mklishin at gopivotal.com (Michael Klishin) Date: Thu, 10 Apr 2014 18:41:10 +0400 Subject: [rabbitmq-discuss] How many max queues RabbitMQ can handle at a time? In-Reply-To: References: Message-ID: On 10 April 2014 at 18:38:08, Michael Klishin (michael at rabbitmq.com) wrote: > > I just created 30,000 empty queues on a 64 bit machine and see > Erlang VM consuming > 589 MB of RAM on peak (slightly lower after some time). Then I added 20,000 more: 1 GB. All queues were exclusive. After shutting down the process used to create them, VM RAM consumption quickly went up to ~ 2 GB then decreased to ~50 MB I had from the start. So regardless of the absolute value you get at one point, the number can still vary over time and you need to account for that.? -- MK Software Engineer, Pivotal/RabbitMQ From videlalvaro at gmail.com Thu Apr 10 15:48:09 2014 From: videlalvaro at gmail.com (Alvaro Videla) Date: Thu, 10 Apr 2014 16:48:09 +0200 Subject: [rabbitmq-discuss] Introducing the RabbitMQ Sharding Plugin Message-ID: Hi, We have just released the RabbitMQ Sharding Plugin: The plugin allows you to partition messages across sharded queues. Adding new nodes to the cluster, automatically scales the shard size. Learn more here: https://github.com/rabbitmq/rabbitmq-sharding Any kind of feedback or questions about the plugin is welcomed. Regards, Alvaro From gavinmroy at gmail.com Thu Apr 10 16:17:40 2014 From: gavinmroy at gmail.com (Gavin M. Roy) Date: Thu, 10 Apr 2014 11:17:40 -0400 Subject: [rabbitmq-discuss] Introducing the RabbitMQ Sharding Plugin In-Reply-To: References: Message-ID: Looks great, looks like I have to update another chapter ;-) On April 10, 2014 at 10:49:51 AM, Alvaro Videla (videlalvaro at gmail.com) wrote: Hi, We have just released the RabbitMQ Sharding Plugin: The plugin allows you to partition messages across sharded queues. Adding new nodes to the cluster, automatically scales the shard size. Learn more here: https://github.com/rabbitmq/rabbitmq-sharding Any kind of feedback or questions about the plugin is welcomed. Regards, Alvaro _______________________________________________ rabbitmq-discuss mailing list rabbitmq-discuss at lists.rabbitmq.com https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: From fred at dushin.net Thu Apr 10 16:52:25 2014 From: fred at dushin.net (fred at dushin.net) Date: Thu, 10 Apr 2014 11:52:25 -0400 Subject: [rabbitmq-discuss] rabbitmq-c memory leak? In-Reply-To: References: <673BC989-D49B-4F23-B57B-04D7502FDDEB@dushin.net> Message-ID: <65AD45E4-5547-4804-A4DB-461449F9968F@dushin.net> I am happy to report that by upgrading to RabbitMQ-c 0.5.0 and SimpleAmqpClient master (2e1b4e3), the leak has disappeared. I apologize for the false alarm, though it does seem like the combination of SimpleAmqpClient 2.3 and RabbitMQ-c 0.4.0 brought about the issue. Looking at the diffs after 2.3, I don't see any obvious culprits, but I am not intimately familiar with the code. In any event, problem solved -- thanks, Alan! -Fred > On Apr 10, 2014, at 9:21 AM, Alan Antonuk wrote: > > Do you get the same problems building it off of the master branch? >> On Thu, Apr 10, 2014 at 4:46 AM, Dushin Fred wrote: >> Sorry, I should have been more clear. >> >> The original issue I hit was from a build off the 2.3 tag (418f31f). So the patch I applied was slightly different: >> >> > git diff >> diff --git a/src/ChannelImpl.cpp b/src/ChannelImpl.cpp >> index 9749556..864502b 100644 >> --- a/src/ChannelImpl.cpp >> +++ b/src/ChannelImpl.cpp >> @@ -152,6 +152,7 @@ amqp_channel_t ChannelImpl::GetChannel() >> >> void ChannelImpl::ReturnChannel(amqp_channel_t channel) >> { >> + amqp_maybe_release_buffers_on_channel(m_connection, channel); >> m_channels.at(channel) = CS_Open; >> m_last_used_channel = channel; >> } >> >> I fully respect that this may not be the "right" fix, but in my limited scenario (the sender.cpp program I posted off github), it does the trick. >> >> I have done zero analysis of the right fix, but I'd be happy to crack open the code and look into it, if the root cause of the bug is not immediately evident to you. Otherwise, if you have any thoughts, feel free to let me know. >> >> And of course I would prefer (as I suspect many would) that the fix go into master, as this bug seems to effectively render the C++ client library unsuitable for production, and I would prefer not to maintain a fix that is not endorsed by the author/maintainer. >> >> -Fred >> >>> On Apr 10, 2014, at 1:11 AM, Alan Antonuk wrote: >>> >>> What version of SimpleAmqpClient are you using? The lines surrounding your change don't seem to match what is currently in the master branch of SimpleAmqpClient. I believe behavior you're seeing was addressed a while ago (see Issue #56). Please download the master branch and build from that, you will also need rabbitmq-c v0.4.1 or newer for this to work. >>> >>> FYI: the patch you are suggesting using will likely cause undefined behavior. Because of the way rabbitmq-c's memory management works, the SimpleAmqpClient cannot call amqp_maybe_release_buffers() until it no longer holds any references to frames returned from amqp_simple_wait_frame(). Once amqp_maybe_release_buffers() (or amqp_maybe_release_buffers_on_channel()) any outstanding frames are considered freed and their use will lead to undefined behavior. SimpleAmqpClient maintains a list of frames that it has seen from the broker, in ChannelImpl::MaybeReleaseBuffersOnChannel it checks to see if there are any frames that it still has a reference to before calling amqp_maybe_release_buffers() >>> >>> -Alan >>> >>> >>>> On Wed, Apr 9, 2014 at 3:17 PM, Dushin Fred wrote: >>>> That works for me, too. Thanks, Abhay. >>>> >>>> Alan, would you like a pull request? Is there a timeline for folding this an the other recent fixes into master? Anything I can do to help expedite? >>>> >>>> -Fred >>>> >>>>> On Apr 9, 2014, at 4:54 AM, Abhay Rajure wrote: >>>>> >>>>> Try this -- >>>>> >>>>> In File SimpleAmqpClient/src/ChannelImpl.cpp, >>>>> call amqp_maybe_release_buffers_on_channel() in ReturnChannel(). >>>>> >>>>> void ChannelImpl::ReturnChannel(amqp_chanel_t channel) >>>>> { >>>>> +amqp_maybe_release_buffers_on_channel(m_connection, channel); >>>>> m_free_channels.push(channel); >>>>> } >>>> >>>> >>>> _______________________________________________ >>>> rabbitmq-discuss mailing list >>>> rabbitmq-discuss at lists.rabbitmq.com >>>> https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss >>> >>> _______________________________________________ >>> rabbitmq-discuss mailing list >>> rabbitmq-discuss at lists.rabbitmq.com >>> https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss >> >> _______________________________________________ >> rabbitmq-discuss mailing list >> rabbitmq-discuss at lists.rabbitmq.com >> https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss > _______________________________________________ > rabbitmq-discuss mailing list > rabbitmq-discuss at lists.rabbitmq.com > https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: From matthias at rabbitmq.com Thu Apr 10 17:01:48 2014 From: matthias at rabbitmq.com (Matthias Radestock) Date: Thu, 10 Apr 2014 17:01:48 +0100 Subject: [rabbitmq-discuss] Performance bottleneck on inter-node connection In-Reply-To: References: <53465CF5.20607@rabbitmq.com> Message-ID: <5346C06C.8090609@rabbitmq.com> (re-adding the list) On 10/04/14 16:42, joseph rouphael wrote: > On Thu, Apr 10, 2014 at 11:57 AM, Matthias Radestock > > wrote: > Have you tried this w/o hipe? > In fact, HIPE is not enabled in Erlang, so this config has no effect. > (You can consider Hipe off) ok, I'd still prefer if you took out that setting. > {delegate_count, 32}, > Why did you set this? > I have used this in a try to increase the number of delegate processes > used for intra-cluster configuration. Hoping that will increase > inter-node communication performance (But this had no effect) Ditto. > In fact, I am not using any synchronous mechanism. I am publishing > messages with no confirm.select. > I am using a c client (based on rabbitmq-c) to publish messages with the > following config: > - Non persistent > - No confirm.select > - Default Direct exchange > - Routing key equal to queue_name > - Message size 1KB > I am running 10 client processes which are publishing in parallel to 10 > different queues. > Rate is regulated on the client, I can force the publsih rate in the C > client. > > I am running as well 10 other client processes to consume full-speed (no > rate regulation) from the 10 queues. Do the consumers use acknowledgements? > I am using the same setup to publish/consume messages on one node (node > 1 where queues are created), with no issue. I can hit up to 100K TPS (5K > TPS on each client process). > The problem appears only when I try to publish on node1 and consume from > node2 (or publish on node2 and consume from node1). The problem appears > only when there is an inter-node activity. So the 100kHz is the aggregate publishing and consuming rate, i.e. 10 x 5kHz publishing and 10 x 5kHz consuming? What results do you get in the single-node and multi-node scenarios when running the tests with just a single queue and single producer and single consumer? Regards, Matthias. From joshua__lim at hotmail.com Thu Apr 10 17:02:10 2014 From: joshua__lim at hotmail.com (joshua__lim) Date: Thu, 10 Apr 2014 09:02:10 -0700 (PDT) Subject: [rabbitmq-discuss] Error during WebSocket handshake: Sent non-empty 'Sec-WebSocket-Protocol' header but no response was received In-Reply-To: References: <1395406371281-34248.post@n5.nabble.com> Message-ID: Hi Michael, I've tried ws://localhost:15674/stomp and got a 200 error instead. ws://localhost:15674/stomp/websocket appears to be correct. Please see this thread - http://rabbitmq.1065348.n5.nabble.com/RabbitMQ-s-WebSocket-broken-td30408.html Rgds,Joshua Date: Thu, 10 Apr 2014 02:15:07 -0700 From: ml-node+s1065348n34734h68 at n5.nabble.com To: joshua__lim at hotmail.com Subject: Re: Error during WebSocket handshake: Sent non-empty 'Sec-WebSocket-Protocol' header but no response was received On 21 March 2014 at 16:53:47, joshua__lim ([hidden email]) wrote: > > WebSocket connection to 'ws://localhost:15674/stomp/websocket' > failed: Error > during WebSocket handshake: Sent non-empty 'Sec-WebSocket-Protocol' > header > but no response was received :8080/stomp.js:374 > > Line 374 is "ws = new klass(url, protocols);?. RabbitMQ Web STOMP uses /stomp. Have you seen/tried code examples on http://www.rabbitmq.com/web-stomp.html? -- MK Software Engineer, Pivotal/RabbitMQ _______________________________________________ rabbitmq-discuss mailing list [hidden email] https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss If you reply to this email, your message will be added to the discussion below: http://rabbitmq.1065348.n5.nabble.com/Error-during-WebSocket-handshake-Sent-non-empty-Sec-WebSocket-Protocol-header-but-no-response-was-red-tp34248p34734.html To unsubscribe from Error during WebSocket handshake: Sent non-empty 'Sec-WebSocket-Protocol' header but no response was received, click here. NAML -- View this message in context: http://rabbitmq.1065348.n5.nabble.com/Error-during-WebSocket-handshake-Sent-non-empty-Sec-WebSocket-Protocol-header-but-no-response-was-red-tp34248p34771.html Sent from the RabbitMQ mailing list archive at Nabble.com. -------------- next part -------------- An HTML attachment was scrubbed... URL: From dvl at biglist.com Thu Apr 10 17:22:47 2014 From: dvl at biglist.com (Dan Langille) Date: Thu, 10 Apr 2014 12:22:47 -0400 Subject: [rabbitmq-discuss] trouble starting 3.3.0 on FreeBSD 9.2 Message-ID: <1E2AC496-6BE3-4E02-B23C-8B33BEE6E4B3@biglist.com> I am having trouble starting RabbitMQ 3.3.0 with erlang-16.b.03.1,3 on FreeBSD 9.2 I have been following https://www.rabbitmq.com/configure.html /usr/local/etc/rabbitmq/rabbitmq-env.conf contains this (sally is the host I'm working on.): NODENAME=bunny at sally I issue this command: $ sudo service rabbitmq start Warning: PID file not written; -detached was passed. And see these log entries: *** /var/log/rabbitmq//bunny at sally.log *** =INFO REPORT==== 10-Apr-2014::16:13:25 === Starting RabbitMQ 3.3.0 on Erlang R16B03-1 Copyright (C) 2007-2013 GoPivotal, Inc. Licensed under the MPL. See http://www.rabbitmq.com/ =INFO REPORT==== 10-Apr-2014::16:13:25 === node : bunny at sally home dir : /var/db/rabbitmq config file(s) : /usr/local/etc/rabbitmq/rabbitmq.config cookie hash : bVe6vw3baQne66uSp3IOiQ== log : /var/log/rabbitmq/bunny at sally.log sasl log : /var/log/rabbitmq/bunny at sally-sasl.log database dir : /var/db/rabbitmq/mnesia/bunny at sally =INFO REPORT==== 10-Apr-2014::16:13:25 === Limiting to approx 10995 file handles (9893 sockets) =INFO REPORT==== 10-Apr-2014::16:13:25 === Memory limit set to 6328MB of 15822MB total. =INFO REPORT==== 10-Apr-2014::16:13:25 === Disk free limit set to 50MB =INFO REPORT==== 10-Apr-2014::16:13:25 === msg_store_transient: using rabbit_msg_store_ets_index to provide index =INFO REPORT==== 10-Apr-2014::16:13:25 === Error description: {badmatch, {error, {{{badmatch, {error, {"/var/db/rabbitmq/mnesia/bunny at sally/msg_stor...", eacces}}}, [{rabbit_msg_store,init,1, [{file,"src/rabbit_msg_store.erl"},{line,666}]}, {gen_server2,init_it,6, [{file,"src/gen_server2.erl"},{line,554}]}, {proc_lib,init_p_do_apply,3, [{file,"proc_lib.erl"},{line,239}]}]}, {child,undefined,msg_store_transient, {rabbit_msg_store,start_link, [msg_store_transient, "/var/db/rabbitmq/mnesia/bunny at sally",undefined, {#Fun,ok}]}, transient,4294967295,worker, [rabbit_msg_store]}}}} Log files (may contain more information): /var/log/rabbitmq/bunny at sally.log /var/log/rabbitmq/bunny at sally-sasl.log Stack trace: [{rabbit_variable_queue,start_msg_store,2, [{file,"src/rabbit_variable_queue.erl"}, {line,407}]}, {rabbit_variable_queue,start,1, [{file,"src/rabbit_variable_queue.erl"}, {line,392}]}, {rabbit_amqqueue,recover,0,[{file,"src/rabbit_amqqueue.erl"},{line,208}]}, {rabbit,recover,0,[{file,"src/rabbit.erl"},{line,619}]}, {rabbit,'-run_boot_step/1-lc$^1/1-1-',1, [{file,"src/rabbit.erl"},{line,505}]}, {rabbit,run_boot_step,1,[{file,"src/rabbit.erl"},{line,504}]}, {rabbit,'-start/2-lc$^0/1-0-',1,[{file,"src/rabbit.erl"},{line,471}]}, {rabbit,start,2,[{file,"src/rabbit.erl"},{line,471}]}] *** /var/log/rabbitmq//bunny at sally-sasl.log *** =CRASH REPORT==== 10-Apr-2014::16:13:25 === crasher: initial call: gen:init_it/7 pid: <0.185.0> registered_name: [] exception exit: {{badmatch, {error, {"/var/db/rabbitmq/mnesia/bunny at sally/msg_s...", eacces}}}, [{rabbit_msg_store,init,1, [{file,"src/rabbit_msg_store.erl"},{line,666}]}, {gen_server2,init_it,6, [{file,"src/gen_server2.erl"},{line,554}]}, {proc_lib,init_p_do_apply,3, [{file,"proc_lib.erl"},{line,239}]}]} in function gen_server2:init_it/6 (src/gen_server2.erl, line 588) ancestors: [rabbit_sup,<0.113.0>] messages: [] links: [<0.114.0>] dictionary: [] trap_exit: true status: running heap_size: 1598 stack_size: 27 reductions: 1790 neighbours: *** /var/log/rabbitmq//bunny at sally.log *** =INFO REPORT==== 10-Apr-2014::16:13:26 === Error description: {could_not_start,rabbit, {bad_return, {{rabbit,start,[normal,[]]}, {'EXIT', {rabbit,failure_during_boot, {badmatch, {error, {{{badmatch,{error,{"/v...",eacces}}}, [{rabbit_msg_store,init,1, [{file,"..."},{line,666}]}, {gen_server2,init_it,6, [{file,"..."},{line,'...'}]}, {proc_lib,init_p_do_apply,3, [{file,'...'},{'...'}]}]}, {child,undefined,msg_store_transient, {rabbit_msg_store,start_link, [msg_store_transient,"/var...",undefined, {#Fun, ok}]}, transient,4294967295,worker, [rabbit_msg_store]}}}}}}}}} Log files (may contain more information): /var/log/rabbitmq/bunny at sally.log /var/log/rabbitmq/bunny at sally-sasl.log *** /var/log/rabbitmq//bunny at sally-sasl.log *** =CRASH REPORT==== 10-Apr-2014::16:13:26 === crasher: initial call: application_master:init/4 pid: <0.112.0> registered_name: [] exception exit: {bad_return, {{rabbit,start,[normal,[]]}, {'EXIT', {rabbit,failure_during_boot, {badmatch, {error, {{{badmatch,{error,{'...'}}}, [{rabbit_msg_store,init,1,'...'}, {gen_server2,init_it,'...'}, {proc_lib,'...'}]}, {child,undefined,msg_store_transient, {rabbit_msg_store,start_link,['...']}, transient,4294967295,worker, [rabbit_msg_store]}}}}}}}} in function application_master:init/4 (application_master.erl, line 133) ancestors: [<0.111.0>] messages: [{'EXIT',<0.113.0>,normal}] links: [<0.111.0>,<0.7.0>] dictionary: [] trap_exit: true status: running heap_size: 2586 stack_size: 27 reductions: 329 neighbours: ? Dan Langille dvl at biglist.com -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 496 bytes Desc: Message signed with OpenPGP using GPGMail URL: From simon at rabbitmq.com Thu Apr 10 17:26:59 2014 From: simon at rabbitmq.com (Simon MacMullen) Date: Thu, 10 Apr 2014 17:26:59 +0100 Subject: [rabbitmq-discuss] trouble starting 3.3.0 on FreeBSD 9.2 In-Reply-To: <1E2AC496-6BE3-4E02-B23C-8B33BEE6E4B3@biglist.com> References: <1E2AC496-6BE3-4E02-B23C-8B33BEE6E4B3@biglist.com> Message-ID: <5346C653.4000906@rabbitmq.com> On 10/04/14 17:22, Dan Langille wrote: > {{{badmatch, > {error, > {"/var/db/rabbitmq/mnesia/bunny at sally/msg_stor...", > eacces}}} eacces -> the server did not have permission to read (or write?) the Mnesia directory. Are you using a FreeBSD port? (Is there one?) If so the port maintainer should look into it. If you have manually installed then... you need to make sure that path is owned by whichever user runs RabbitMQ. Cheers, Simon -- Simon MacMullen RabbitMQ, Pivotal From dvl at biglist.com Thu Apr 10 17:35:07 2014 From: dvl at biglist.com (Dan Langille) Date: Thu, 10 Apr 2014 12:35:07 -0400 Subject: [rabbitmq-discuss] trouble starting 3.3.0 on FreeBSD 9.2 In-Reply-To: <5346C653.4000906@rabbitmq.com> References: <1E2AC496-6BE3-4E02-B23C-8B33BEE6E4B3@biglist.com> <5346C653.4000906@rabbitmq.com> Message-ID: <0E877B4B-DE55-4AAF-93D0-22F727CAB1E9@biglist.com> On Apr 10, 2014, at 12:26 PM, Simon MacMullen wrote: > On 10/04/14 17:22, Dan Langille wrote: >> {{{badmatch, >> {error, >> {"/var/db/rabbitmq/mnesia/bunny at sally/msg_stor...", >> eacces}}} > > eacces -> the server did not have permission to read (or write?) the Mnesia directory. $ ls -ld /var/db/rabbitmq \ > /var/db/rabbitmq/mnesia \ > /var/db/rabbitmq/mnesia/bunny\@sally \ > /var/db/rabbitmq/mnesia/bunny\@sally/* drwxr-xr-x 3 rabbitmq rabbitmq 4 Apr 9 20:40 /var/db/rabbitmq drwxr-xr-x 6 rabbitmq rabbitmq 8 Apr 10 16:24 /var/db/rabbitmq/mnesia drwxr-xr-x 4 rabbitmq rabbitmq 19 Apr 10 16:24 /var/db/rabbitmq/mnesia/bunny at sally -rw-r--r-- 1 rabbitmq rabbitmq 149 Apr 10 16:24 /var/db/rabbitmq/mnesia/bunny at sally/DECISION_TAB.LOG -rw-r--r-- 1 rabbitmq rabbitmq 144 Apr 10 16:24 /var/db/rabbitmq/mnesia/bunny at sally/LATEST.LOG -rw-r--r-- 1 rabbitmq rabbitmq 31 Apr 10 16:24 /var/db/rabbitmq/mnesia/bunny at sally/cluster_nodes.config drwxr-xr-x 2 rabbitmq rabbitmq 3 Apr 10 15:32 /var/db/rabbitmq/mnesia/bunny at sally/msg_store_persistent drwxr-xr-x 2 root rabbitmq 3 Apr 10 15:32 /var/db/rabbitmq/mnesia/bunny at sally/msg_store_transient -rw-r--r-- 1 rabbitmq rabbitmq 15 Apr 10 16:24 /var/db/rabbitmq/mnesia/bunny at sally/nodes_running_at_shutdown -rw-r--r-- 1 rabbitmq rabbitmq 1110 Apr 10 15:30 /var/db/rabbitmq/mnesia/bunny at sally/rabbit_durable_exchange.DCD -rw-r--r-- 1 rabbitmq rabbitmq 8 Apr 10 14:59 /var/db/rabbitmq/mnesia/bunny at sally/rabbit_durable_queue.DCD -rw-r--r-- 1 rabbitmq rabbitmq 8 Apr 10 14:59 /var/db/rabbitmq/mnesia/bunny at sally/rabbit_durable_route.DCD -rw-r--r-- 1 rabbitmq rabbitmq 168 Apr 10 15:30 /var/db/rabbitmq/mnesia/bunny at sally/rabbit_runtime_parameters.DCD -rw-r--r-- 1 rabbitmq rabbitmq 4 Apr 10 16:24 /var/db/rabbitmq/mnesia/bunny at sally/rabbit_serial -rw-r--r-- 1 rabbitmq rabbitmq 231 Apr 10 15:30 /var/db/rabbitmq/mnesia/bunny at sally/rabbit_user.DCD -rw-r--r-- 1 rabbitmq rabbitmq 182 Apr 10 15:30 /var/db/rabbitmq/mnesia/bunny at sally/rabbit_user_permission.DCD -rw-r--r-- 1 rabbitmq rabbitmq 123 Apr 10 15:30 /var/db/rabbitmq/mnesia/bunny at sally/rabbit_vhost.DCD -rw-r--r-- 1 rabbitmq rabbitmq 5464 Apr 10 16:24 /var/db/rabbitmq/mnesia/bunny at sally/recovery.dets -rw-r--r-- 1 rabbitmq rabbitmq 19105 Apr 10 14:59 /var/db/rabbitmq/mnesia/bunny at sally/schema.DAT -rw-r--r-- 1 rabbitmq rabbitmq 272 Apr 10 14:59 /var/db/rabbitmq/mnesia/bunny at sally/schema_version Based on that, I foolhardily tried this: chown rabbitmq /var/db/rabbitmq/mnesia/bunny at sally/msg_store_transient $ sudo service rabbitmq start Warning: PID file not written; -detached was passed. [dvl at sally /var/db/rabbitmq/mnesia]$ ps auwx | grep rabb rabbitmq 23212 3.8 0.3 91268 45032 ?? SJ 4:32PM 0:00.78 /usr/local/lib/erlang/erts-5.10.4/bin/beam.smp -W w -A30 -P 1048576 -- -root /usr/local/lib/erlang -progname erl -- -home /var/db/rabbitm rabbitmq 23242 1.4 0.0 14264 1800 ?? SsJ 4:32PM 0:00.00 inet_gethost 4 rabbitmq 23243 1.4 0.0 14264 1888 ?? SJ 4:32PM 0:00.00 inet_gethost 4 dvl 23246 0.0 0.0 16312 1884 3 R+J 4:32PM 0:00.00 grep rabb dvl 23181 0.0 0.0 9944 1568 0 S+J 4:32PM 0:00.00 xtail /var/log/rabbitmq/ That was the issue. I thank you. > Are you using a FreeBSD port? (Is there one?) Yes: http://www.freshports.org/net/rabbitmq/ (disclosure, that's my website) > If so the port maintainer should look into it. If you have manually installed then... you need to make sure that path is owned by whichever user runs RabbitMQ. I have a feeling that errant file may have been created by my own misdoings. But I'm willing to test that by trying this again on a new host. I'll report back soon. [I apologize for the rude things I said while trying to get this working...] ? Dan Langille dvl at biglist.com -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 496 bytes Desc: Message signed with OpenPGP using GPGMail URL: From arajure at gmail.com Thu Apr 10 17:56:49 2014 From: arajure at gmail.com (Abhay Rajure) Date: Thu, 10 Apr 2014 16:56:49 +0000 (UTC) Subject: [rabbitmq-discuss] rabbitmq-c memory leak? References: <673BC989-D49B-4F23-B57B-04D7502FDDEB@dushin.net> <65AD45E4-5547-4804-A4DB-461449F9968F@dushin.net> Message-ID: fred at ... writes: > > > I am happy to report that by upgrading to RabbitMQ-c 0.5.0 and SimpleAmqpClient master (2e1b4e3), the leak has disappeared. > > I apologize for the false alarm, though it does seem like the combination of SimpleAmqpClient 2.3 and RabbitMQ-c 0.4.0 brought about the issue. > > Looking at the diffs after 2.3, I don't see any obvious culprits, but I am not intimately familiar with the code. > > In any event, problem solved -- thanks, Alan!-Fred > Thanks Fred for the update! I can confirm as well that with the latest master and the leak seems to have disappeared. Thanks Alan & Fred! -Abhay From dvl at biglist.com Thu Apr 10 18:05:06 2014 From: dvl at biglist.com (Dan Langille) Date: Thu, 10 Apr 2014 13:05:06 -0400 Subject: [rabbitmq-discuss] trouble starting 3.3.0 on FreeBSD 9.2 In-Reply-To: <0E877B4B-DE55-4AAF-93D0-22F727CAB1E9@biglist.com> References: <1E2AC496-6BE3-4E02-B23C-8B33BEE6E4B3@biglist.com> <5346C653.4000906@rabbitmq.com> <0E877B4B-DE55-4AAF-93D0-22F727CAB1E9@biglist.com> Message-ID: <4CCD8813-AD7C-4922-BE00-B6D1137D9B8F@biglist.com> On Apr 10, 2014, at 12:35 PM, Dan Langille wrote: > On Apr 10, 2014, at 12:26 PM, Simon MacMullen wrote: > >> On 10/04/14 17:22, Dan Langille wrote: >>> {{{badmatch, >>> {error, >>> {"/var/db/rabbitmq/mnesia/bunny at sally/msg_stor...", >>> eacces}}} >> >> eacces -> the server did not have permission to read (or write?) the Mnesia directory. [snip] > drwxr-xr-x 2 root rabbitmq 3 Apr 10 15:32 /var/db/rabbitmq/mnesia/bunny at sally/msg_store_transient [snip] > I have a feeling that errant file may have been created by my own misdoings. But I'm willing to test that by trying this again on a new host. I'll report back soon. I have tried this again on a clean host, installing the same packages. I did not encounter the same problem. I am not sure how the root chown'd directory happened, but I think we will all agree it was my fault. For what it's worth, when I was testing with this: NODENAME=bunny at 192.168.11.54 The server would not start: # service rabbitmq onestart Warning: PID file not written; -detached was passed. Error: unable to connect to node 'bunny at 192.168.11.54': nodedown DIAGNOSTICS =========== attempted to contact: ['bunny at 192.168.11.54'] bunny at 192.168.11.54: * bunny seems not to be running at all * other nodes on 192.168.11.54: [rabbitmqctl26754] current node details: - node name: rabbitmqctl26754 at testing - home dir: /var/db/rabbitmq - cookie hash: oGo1EYRWNeUldVp0J/8vcw== However, when I added an entry for testing in /etc/hosts and switched to: NODENAME=bunny at testing ... the server would start. -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 496 bytes Desc: Message signed with OpenPGP using GPGMail URL: From arajure at gmail.com Thu Apr 10 19:51:04 2014 From: arajure at gmail.com (Abhay Rajure) Date: Thu, 10 Apr 2014 18:51:04 +0000 (UTC) Subject: [rabbitmq-discuss] rabbitmq-c memory leak? References: <673BC989-D49B-4F23-B57B-04D7502FDDEB@dushin.net> Message-ID: Alan Antonuk writes: > On Apr 10, 2014, at 1:11 AM, Alan Antonuk wrote: > > > What version of SimpleAmqpClient are you using? The lines surrounding your change don't seem to match what is currently in the master branch of SimpleAmqpClient. I believe behavior you're seeing was addressed a while ago (see Issue #56). Please download the master branch and build from that, you will also need rabbitmq-c v0.4.1 or newer for this to work. > > > > -Alan > Hi Alan, Can you recommend a specific tag we can use from the master that addresses this memory leak issue you've mentioned (Issue #56)? Or is it safe to use the latest from master (unreleased) version? Thanks, Abhay From pmaisenovich at blizzard.com Thu Apr 10 20:19:52 2014 From: pmaisenovich at blizzard.com (Pavel) Date: Thu, 10 Apr 2014 12:19:52 -0700 (PDT) Subject: [rabbitmq-discuss] Possible memory leak in the management plugin In-Reply-To: <53466D39.4050906@rabbitmq.com> References: <8a1af6ed-c599-42a3-85c3-c5dc22a99bd8@googlegroups.com> <520CB29A.1060304@rabbitmq.com> <1397020502914-34663.post@n5.nabble.com> <534521F1.9030707@rabbitmq.com> <1397058685615-34697.post@n5.nabble.com> <53457674.1040704@rabbitmq.com> <1397106237672-34725.post@n5.nabble.com> <53466D39.4050906@rabbitmq.com> Message-ID: <1397157592459-34779.post@n5.nabble.com> > That table [aggregated_stats] contains one row for every > combination of things that can show message rates, and each row contains > some history for that thing. > > The GCing is about deleting old history from each row. This is a > relatively expensive operation, so the DB loops round GCing 1% of rows > (or 100 rows, whichever is larger) every 5s. That means that we can keep > a bit more history around than we're configured to, just because we > haven't got round to GCing it yet. It sounds like 1) If at least 1 message is published to each channel/exchange/queue within retention period (10s by default for detailed stats) - entire history data will be kept in memory (aggregated_stats) and solely rely on clean up process ("GC-ing") to remove old records. 2) Because the GC-ing process has constant rate (100 or 1% rows each 5 seconds), there is always a possibility that aggregated_stats table growth will outpace the clean up efforts. In other words, if (events rate x number of data points) is high enough, mgmt_db will grow continuously until Rabbit hits high watermark and starts to throttle publishers/consumers. This is very concerning as while Rabbit might be stable at the moment it could begin to explode some time in a future depending on concurrency. As far as I can tell, currently rows for GC are selected randomly. Is there a way cycle them so that entire aggregated_stats table is guaranteed to get a GC pass every 500 seconds (if size is 10000+)? This should increase the concurrency required to outpace GC. Also, would you consider making GC_MIN_ROWS and GC_MIN_RATIO configurable externally? It will give some room for tuning, but essentially the best solution would be to make the GC-ing process dynamic and memory aware. Speaking of which, what you think about monitoring the aggregated_stats table size and dynamically changing the row count for GC-ing process run up to a certain threshold? Since it can't grow forever (supposedly, due to performance implications you mentioned), in addition it would be great to have some sort of high watermark based throttling for fine grained stats, similar to how transient queues begin to page out to disk at certain level. So in the event of GC-ing process falling behind aggregated_stats will not consume all RAM and force Rabbit to throttle publishers, but instead downgrade the monitoring level (and alert!) as a prevention measure. Thanks! -- View this message in context: http://rabbitmq.1065348.n5.nabble.com/Possible-memory-leak-in-the-management-plugin-tp27414p34779.html Sent from the RabbitMQ mailing list archive at Nabble.com. From Anita.Yip at RealPage.com Thu Apr 10 20:22:28 2014 From: Anita.Yip at RealPage.com (Anita Yip) Date: Thu, 10 Apr 2014 19:22:28 +0000 Subject: [rabbitmq-discuss] frame_too_large error while connecting to Message-ID: I keep getting this error some times when I start my node.js app which connects to rabbitmq servers during app start up. Even after getting this error, after couple of retries, it is able to connect properly. Can you please let me know what could be causing this issue and how I can overcome it. Unhandled connection error: FRAME_ERROR - type 65, all octets = <<>>: {frame_too_large,1342177289,131064} { [Error: FRAME_ERROR - type 65, all octets = <<>>: {frame_too_large,1342177289,131064}] code: 501 } Anita Yip RealPage, Inc. 360 Third Street Suite 450 San Francisco, CA 94107 Office: 415-395-2109 Cell: 415-200-7312 Anita.yip at realpage.com www.realpage.com Connect with RealPage: [Description: Description: Description: cid:image003.jpg at 01CE5184.4C0A9D20][Description: Description: Description: cid:image005.gif at 01CE5184.4C0A9D20] [Description: Description: Description: cid:image006.gif at 01CE5184.4C0A9D20] [Description: Description: Description: cid:image007.png at 01CE5184.4C0A9D20] [Description: Description: Description: cid:image008.jpg at 01CE5184.4C0A9D20] [Description: Description: Description: cid:image009.jpg at 01CE5184.4C0A9D20] ________________________________ This message is intended only for the use of the individual(s) or entity to which it is addressed and may contain information that is privileged, confidential, and/or proprietary to RealPage and its affiliated companies. If the reader of this message is not the intended recipient, you are hereby notified that any dissemination, distribution, forwarding or copying of this communication is prohibited without the express permission of the sender. If you have received this communication in error, please notify the sender immediately and delete the original message. -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: 4AC7ACDF-CDB3-4930-A66C-481C9E292AE1[68].png Type: image/png Size: 1722 bytes Desc: 4AC7ACDF-CDB3-4930-A66C-481C9E292AE1[68].png URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: 00859B61-FDB7-4823-A9ED-23D3130E18AC[68].png Type: image/png Size: 1360 bytes Desc: 00859B61-FDB7-4823-A9ED-23D3130E18AC[68].png URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: 91E364DD-5CE6-493B-8292-4DD276F2B5D4[68].png Type: image/png Size: 1254 bytes Desc: 91E364DD-5CE6-493B-8292-4DD276F2B5D4[68].png URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: 72716D1A-8A3D-46B6-A89B-999285564183[68].png Type: image/png Size: 4781 bytes Desc: 72716D1A-8A3D-46B6-A89B-999285564183[68].png URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: 2AC40763-6D00-4A59-9A0C-A5525B44EC57[68].png Type: image/png Size: 1875 bytes Desc: 2AC40763-6D00-4A59-9A0C-A5525B44EC57[68].png URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: 4CF393B6-2ADA-418D-AC2B-5D67FA784399[68].png Type: image/png Size: 1885 bytes Desc: 4CF393B6-2ADA-418D-AC2B-5D67FA784399[68].png URL: From mklishin at gopivotal.com Thu Apr 10 20:50:19 2014 From: mklishin at gopivotal.com (Michael Klishin) Date: Thu, 10 Apr 2014 23:50:19 +0400 Subject: [rabbitmq-discuss] frame_too_large error while connecting to In-Reply-To: References: Message-ID: ?On 10 April 2014 at 23:23:46, Anita Yip (anita.yip at realpage.com) wrote: > > Unhandled connection error: FRAME_ERROR - type 65, all octets > = <<>>: {frame_too_large,1342177289,131064(tel://1342177289,131064)} > { [Error: FRAME_ERROR - type 65, all octets = <<>>: {frame_too_large,1342177289,131064(tel://1342177289,131064)}] > code: 501 } This almost certainly indicates a client library bug. What do you use? Can you post a small script/program that reproduces the problem most of the time? -- MK Software Engineer, Pivotal/RabbitMQ From mcintoshj at gmail.com Thu Apr 10 20:55:32 2014 From: mcintoshj at gmail.com (Jason McIntosh) Date: Thu, 10 Apr 2014 14:55:32 -0500 Subject: [rabbitmq-discuss] Rabbit startup command is hanging Message-ID: Not sure what's going on here. Just ugpraded my cluster from 3.2.3 to 3.2.4 (including a restart of the machine). On startup, two of my initial nodes started fine, but when the third node in the cluster started, the "/etc/init.d/rabbitmq-server start" just sits at "Starting rabbitmq-server: " without ever finishing. Doing a rabbitmqctl status shows: Status of node cluster at rabbitmqm11p ... [{pid,62505}, {running_applications,[{os_mon,"CPO CXC 138 46","2.2.14"}, {inets,"INETS CXC 138 49","5.9.8"}, {mnesia,"MNESIA CXC 138 12","4.11"}, {amqp_client,"RabbitMQ AMQP Client","3.2.4"}, {xmerl,"XML parser","1.3.6"}, {eldap,"Ldap api","1.0.2"}, {sasl,"SASL CXC 138 11","2.3.4"}, {stdlib,"ERTS CXC 138 10","1.19.4"}, {kernel,"ERTS CXC 138 10","2.16.4"}]}, {os,{unix,linux}}, {erlang_version,"Erlang R16B03-1 (erts-5.10.4) [source] [64-bit] [smp:24:24] [async-threads:30] [hipe] [kernel-poll:true]\n"}, {memory,[{total,48504352}, {connection_procs,2808}, {queue_procs,0}, {plugins,0}, {other_proc,16290632}, {mnesia,1783536}, {mgmt_db,0}, {msg_index,0}, {other_ets,1120896}, {binary,725448}, {code,19691642}, {atom,703377}, {other_system,8186013}]}, {file_descriptors,[{total_limit,12188}, {total_used,0}, {sockets_limit,10967}, {sockets_used,0}]}, {processes,[{limit,1048576},{used,117}]}, {run_queue,0}, {uptime,83}] ...done. In the web management interface, I see this: Node statistics not available Memory details Connections 2.7kB Queues 0B Plugins 0B Other process memory 16MB Mnesia 1.7MB Message store index 0B Management database 0B Other ETS tables 1.1MB Binaries 708kB Code 19MB Atoms 687kB Other system 7.8MB SO rabbit appears to have sort of started, but certain things are not started (e.g. plugins). Plugins list is: [e] amqp_client 3.2.4 [ ] cowboy 0.5.0-rmq3.2.4-git4b93c2d [ ] eldap 3.2.4-gite309de4 [e] mochiweb 2.7.0-rmq3.2.4-git680dba8 [ ] rabbitmq_amqp1_0 3.2.4 [E] rabbitmq_auth_backend_ldap 3.2.4 [ ] rabbitmq_auth_mechanism_ssl 3.2.4 [E] rabbitmq_consistent_hash_exchange 3.2.4 [E] rabbitmq_federation 3.2.4 [E] rabbitmq_federation_management 3.2.4 [ ] rabbitmq_jsonrpc 3.2.4 [ ] rabbitmq_jsonrpc_channel 3.2.4 [ ] rabbitmq_jsonrpc_channel_examples 3.2.4 [E] rabbitmq_management 3.2.4 [E] rabbitmq_management_agent 3.2.4 [E] rabbitmq_management_visualiser 3.2.4 [ ] rabbitmq_mqtt 3.2.4 [E] rabbitmq_shovel 3.2.4 [E] rabbitmq_shovel_management 3.2.4 [ ] rabbitmq_stomp 3.2.4 [ ] rabbitmq_tracing 3.2.4 [e] rabbitmq_web_dispatch 3.2.4 [ ] rabbitmq_web_stomp 3.2.4 [ ] rabbitmq_web_stomp_examples 3.2.4 [ ] rfc4627_jsonrpc 3.2.4-git5e67120 [ ] sockjs 0.3.4-rmq3.2.4-git3132eb9 [e] webmachine 1.10.3-rmq3.2.4-gite9359c7 Any suggestions on next steps on debugging this? Or what I can do to get this back up and in a "healthy" state? Thanks! Jason -- Jason McIntosh https://github.com/jasonmcintosh/ 573-424-7612 -------------- next part -------------- An HTML attachment was scrubbed... URL: From matthias at rabbitmq.com Thu Apr 10 20:57:32 2014 From: matthias at rabbitmq.com (Matthias Radestock) Date: Thu, 10 Apr 2014 20:57:32 +0100 Subject: [rabbitmq-discuss] Possible memory leak in the management plugin In-Reply-To: <1397157592459-34779.post@n5.nabble.com> References: <8a1af6ed-c599-42a3-85c3-c5dc22a99bd8@googlegroups.com> <520CB29A.1060304@rabbitmq.com> <1397020502914-34663.post@n5.nabble.com> <534521F1.9030707@rabbitmq.com> <1397058685615-34697.post@n5.nabble.com> <53457674.1040704@rabbitmq.com> <1397106237672-34725.post@n5.nabble.com> <53466D39.4050906@rabbitmq.com> <1397157592459-34779.post@n5.nabble.com> Message-ID: <5346F7AC.2090703@rabbitmq.com> On 10/04/14 20:19, Pavel wrote: > 2) Because the GC-ing process has constant rate (100 or 1% rows each > 5 seconds), there is always a possibility that aggregated_stats table > growth will outpace the clean up efforts. No, there isn't. > As far as I can tell, currently rows for GC are selected randomly. No, they are not. > Is there a way cycle them so that entire aggregated_stats table is > guaranteed to get a GC pass every 500 seconds (if size is 10000+)? That is exactly what happens. Matthias. From mcintoshj at gmail.com Thu Apr 10 21:07:34 2014 From: mcintoshj at gmail.com (Jason McIntosh) Date: Thu, 10 Apr 2014 15:07:34 -0500 Subject: [rabbitmq-discuss] Introducing the RabbitMQ Sharding Plugin In-Reply-To: References: Message-ID: Off hand, this new exchange looks pretty interesting. We sorta do sharding via x-consistent-hashses right now. It does require more manual work/setup to do it, but it works quite well for our setup. This though looks like it'd make things much simpler - particularly on the consumer side! A few things I wanted to check on though: Shoveling config - we use shovel to publish from a bunch of "cloud" rabbit's to a central server. We "shard" our queues to get around WAN latency. SO I'd be curious how this will operate with shovels. I'm assuming just like any other consumer - e.g. "amqp://@localhost", consume from queue "images"? The one thing I didn't see in the post was how many consumers you'd need. It sounds like you'd need at least N consumers where N is the number of nodes in the shard? With the shoveling config, we'd need to dynamically grow consumers to adapt to the cluster size it sounds like. Just a few thoughts right now - the consumer side is where I've got some concerns on how this would work, Jason On Thu, Apr 10, 2014 at 10:17 AM, Gavin M. Roy wrote: > Looks great, looks like I have to update another chapter ;-) > > On April 10, 2014 at 10:49:51 AM, Alvaro Videla (videlalvaro at gmail.com) > wrote: > > Hi, > > We have just released the RabbitMQ Sharding Plugin: > > The plugin allows you to partition messages across sharded queues. > Adding new nodes to the cluster, automatically scales the shard size. > > Learn more here: https://github.com/rabbitmq/rabbitmq-sharding > > Any kind of feedback or questions about the plugin is welcomed. > > Regards, > > Alvaro > _______________________________________________ > rabbitmq-discuss mailing list > rabbitmq-discuss at lists.rabbitmq.com > https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss > > > _______________________________________________ > rabbitmq-discuss mailing list > rabbitmq-discuss at lists.rabbitmq.com > https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss > > -- Jason McIntosh https://github.com/jasonmcintosh/ 573-424-7612 -------------- next part -------------- An HTML attachment was scrubbed... URL: From videlalvaro at gmail.com Thu Apr 10 21:10:40 2014 From: videlalvaro at gmail.com (Alvaro Videla) Date: Thu, 10 Apr 2014 22:10:40 +0200 Subject: [rabbitmq-discuss] Introducing the RabbitMQ Sharding Plugin In-Reply-To: References: Message-ID: Hi, On Thu, Apr 10, 2014 at 10:07 PM, Jason McIntosh wrote: > The one thing I didn't see in the post was how many consumers you'd need. > It sounds like you'd need at least N consumers where N is the number of > nodes in the shard? With the shoveling config, we'd need to dynamically > grow consumers to adapt to the cluster size it sounds like. > Yes, at the moment you would need 1 consumer per shard. So if the shard-per-node value is 4, then you'd need 4 consumers per node in the cluster. Regards, Alvaro From alan.antonuk at gmail.com Thu Apr 10 22:09:36 2014 From: alan.antonuk at gmail.com (Alan Antonuk) Date: Thu, 10 Apr 2014 14:09:36 -0700 Subject: [rabbitmq-discuss] rabbitmq-c memory leak? In-Reply-To: References: <673BC989-D49B-4F23-B57B-04D7502FDDEB@dushin.net> Message-ID: The master branch in general pretty safe to use. I do make an effort to have whatever is in the published master branch be stable. Feature branches are up for grabs and likely unstable until merged. -Alan On Thu, Apr 10, 2014 at 11:51 AM, Abhay Rajure wrote: > Alan Antonuk writes: > > > On Apr 10, 2014, at 1:11 AM, Alan Antonuk > wrote: > > > > > > What version of SimpleAmqpClient are you using? The lines surrounding > your > change don't seem to match what is currently in the master branch of > SimpleAmqpClient. I believe behavior you're seeing was addressed a while > ago > (see Issue #56). Please download the master branch and build from that, you > will also need rabbitmq-c v0.4.1 or newer for this to work. > > > > > > > > -Alan > > > > Hi Alan, > Can you recommend a specific tag we can use from the master that addresses > this memory leak issue you've mentioned (Issue #56)? > > Or is it safe to use the latest from master (unreleased) version? > > Thanks, > Abhay > > > _______________________________________________ > rabbitmq-discuss mailing list > rabbitmq-discuss at lists.rabbitmq.com > https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mcintoshj at gmail.com Thu Apr 10 22:22:40 2014 From: mcintoshj at gmail.com (Jason McIntosh) Date: Thu, 10 Apr 2014 16:22:40 -0500 Subject: [rabbitmq-discuss] Rabbit startup command is hanging In-Reply-To: References: Message-ID: SO now the fun part. I decided to try and rebuild the middle node (I have boxes 10, 11 and 12). However, I can't get the middle node to reconnect to the cluster. Removing it's mnesia directory allowed it to start, but it can't rejoin the cluster. SO I tried removing the node from the cluster, e.g.: rabbitmqctl -n cluster at rabbitmqm10 forget_cluster_node cluster at rabbitmqm11 But the above never responds - it's just sitting there hanging. rabbitmqctl -n cluster at rabbitmqm11 status FROM the other nodes all works fine. I'm about at a loss as to how the heck to repair things. I can't remove the node from the cluster, I can't start it with the mnesia directory in it's current state, and removing the mnesia directory and trying to add it back in is failing - it fails with "....done (already_member).". Trying to do rabbitmqctl update_cluster_nodes cluster at rabbitmqm10 is sitting there doing nothing and not responding either. I'm starting to really worry I'm going to have to completely rebuild my cluster... Jason On Thu, Apr 10, 2014 at 2:55 PM, Jason McIntosh wrote: > Not sure what's going on here. Just ugpraded my cluster from 3.2.3 to > 3.2.4 (including a restart of the machine). On startup, two of my initial > nodes started fine, but when the third node in the cluster started, the > "/etc/init.d/rabbitmq-server start" just sits at "Starting rabbitmq-server: > " without ever finishing. Doing a rabbitmqctl status shows: > Status of node cluster at rabbitmqm11p ... > [{pid,62505}, > {running_applications,[{os_mon,"CPO CXC 138 46","2.2.14"}, > {inets,"INETS CXC 138 49","5.9.8"}, > {mnesia,"MNESIA CXC 138 12","4.11"}, > {amqp_client,"RabbitMQ AMQP Client","3.2.4"}, > {xmerl,"XML parser","1.3.6"}, > {eldap,"Ldap api","1.0.2"}, > {sasl,"SASL CXC 138 11","2.3.4"}, > {stdlib,"ERTS CXC 138 10","1.19.4"}, > {kernel,"ERTS CXC 138 10","2.16.4"}]}, > {os,{unix,linux}}, > {erlang_version,"Erlang R16B03-1 (erts-5.10.4) [source] [64-bit] > [smp:24:24] [async-threads:30] [hipe] [kernel-poll:true]\n"}, > {memory,[{total,48504352}, > {connection_procs,2808}, > {queue_procs,0}, > {plugins,0}, > {other_proc,16290632}, > {mnesia,1783536}, > {mgmt_db,0}, > {msg_index,0}, > {other_ets,1120896}, > {binary,725448}, > {code,19691642}, > {atom,703377}, > {other_system,8186013}]}, > {file_descriptors,[{total_limit,12188}, > {total_used,0}, > {sockets_limit,10967}, > {sockets_used,0}]}, > {processes,[{limit,1048576},{used,117}]}, > {run_queue,0}, > {uptime,83}] > ...done. > > > In the web management interface, I see this: > Node statistics not available > Memory details > > Connections 2.7kB Queues 0B Plugins 0B Other process memory 16MB > Mnesia 1.7MB Message store index 0B Management database 0B Other ETS > tables 1.1MB Binaries 708kB Code 19MB Atoms 687kB Other system 7.8MB > > > SO rabbit appears to have sort of started, but certain things are not > started (e.g. plugins). Plugins list is: > [e] amqp_client 3.2.4 > [ ] cowboy 0.5.0-rmq3.2.4-git4b93c2d > [ ] eldap 3.2.4-gite309de4 > [e] mochiweb 2.7.0-rmq3.2.4-git680dba8 > [ ] rabbitmq_amqp1_0 3.2.4 > [E] rabbitmq_auth_backend_ldap 3.2.4 > [ ] rabbitmq_auth_mechanism_ssl 3.2.4 > [E] rabbitmq_consistent_hash_exchange 3.2.4 > [E] rabbitmq_federation 3.2.4 > [E] rabbitmq_federation_management 3.2.4 > [ ] rabbitmq_jsonrpc 3.2.4 > [ ] rabbitmq_jsonrpc_channel 3.2.4 > [ ] rabbitmq_jsonrpc_channel_examples 3.2.4 > [E] rabbitmq_management 3.2.4 > [E] rabbitmq_management_agent 3.2.4 > [E] rabbitmq_management_visualiser 3.2.4 > [ ] rabbitmq_mqtt 3.2.4 > [E] rabbitmq_shovel 3.2.4 > [E] rabbitmq_shovel_management 3.2.4 > [ ] rabbitmq_stomp 3.2.4 > [ ] rabbitmq_tracing 3.2.4 > [e] rabbitmq_web_dispatch 3.2.4 > [ ] rabbitmq_web_stomp 3.2.4 > [ ] rabbitmq_web_stomp_examples 3.2.4 > [ ] rfc4627_jsonrpc 3.2.4-git5e67120 > [ ] sockjs 0.3.4-rmq3.2.4-git3132eb9 > [e] webmachine 1.10.3-rmq3.2.4-gite9359c7 > > > Any suggestions on next steps on debugging this? Or what I can do to get > this back up and in a "healthy" state? > > Thanks! > Jason > > > > > -- > Jason McIntosh > https://github.com/jasonmcintosh/ > 573-424-7612 > -- Jason McIntosh https://github.com/jasonmcintosh/ 573-424-7612 -------------- next part -------------- An HTML attachment was scrubbed... URL: From gavinmroy at gmail.com Thu Apr 10 22:22:37 2014 From: gavinmroy at gmail.com (Gavin M. Roy) Date: Thu, 10 Apr 2014 17:22:37 -0400 Subject: [rabbitmq-discuss] ANN rabbitpy 0.16.0 is released Message-ID: I have pushed rabbitpy 0.16.0 to pypi and it is available for use. It is considered the latest and greatest stable release and all users are urged to upgrade. This release is primarily a bug fix release: - Fix an issue with no_ack=True consumer cancellation causing an exception on Channel.Close - Fix bugs with exchange and queue unbinding sending the wrong frame type - Fix for gevent to add wait on the SOCKET_OPENED event when connecting? - Deal with str message body values in Python 3 by casting to bytes and encoding as UTF-8. Thank you to contributors to this version:?Brian McFadden, SuNbiT, and kristaps For more information, including installation instructions, please visit https://pypi.python.org/pypi/rabbitpy or the documentation at https://rabbitpy.readthedocs.org? If you have any issues, please be sure to use the issue tracker at https://github.com/gmr/rabbitpy/issues Regards, Gavin -------------- next part -------------- An HTML attachment was scrubbed... URL: From ryankbrown at gmail.com Thu Apr 10 22:35:52 2014 From: ryankbrown at gmail.com (Ryan Brown) Date: Thu, 10 Apr 2014 15:35:52 -0600 Subject: [rabbitmq-discuss] Forcing publish/confirm to return a nack Message-ID: I am just about finished working through my new publish/confirm workflow and would like to test non-standard workflows in my integration environment. Would there be a way that I could force rabbit to return, say, a basic.nack? -- -rb -------------- next part -------------- An HTML attachment was scrubbed... URL: From pmaisenovich at blizzard.com Fri Apr 11 00:17:34 2014 From: pmaisenovich at blizzard.com (Pavel) Date: Thu, 10 Apr 2014 16:17:34 -0700 (PDT) Subject: [rabbitmq-discuss] Possible memory leak in the management plugin In-Reply-To: <5346F7AC.2090703@rabbitmq.com> References: <520CB29A.1060304@rabbitmq.com> <1397020502914-34663.post@n5.nabble.com> <534521F1.9030707@rabbitmq.com> <1397058685615-34697.post@n5.nabble.com> <53457674.1040704@rabbitmq.com> <1397106237672-34725.post@n5.nabble.com> <53466D39.4050906@rabbitmq.com> <1397157592459-34779.post@n5.nabble.com> <5346F7AC.2090703@rabbitmq.com> Message-ID: <1397171854071-34790.post@n5.nabble.com> > 2) Because the GC-ing process has constant rate (100 or 1% rows each > 5 seconds), there is always a possibility that aggregated_stats table > growth will outpace the clean up efforts. > No, there isn't. Well, something is happening and I'm just trying to get down to the cause. That theory seemed plausible given my test results. I've been running a throttled test (RabbitSmasher 1000 1 1000000 1000 false true) which was publishing at 500/s rate rotating through 1000 channels and 1000 exchanges on every message for ~10 hours. aggregated_stats table was growing steadily for almost 4 hours straight until it reached 2.5Gb when Rabbit RAM hit high watermark of 4Gb and publisher channels start getting throttled. From there Rabbit started flipping between "publisher runs, memory grows" and "publisher blocked, memory GCed" states for next 6 hours. One thing I noticed is that during "publisher blocked" phase the memory used by aggregated_stats would go down, but it's size wouldn't. The number of items in that table will always keep growing for the duration of entire test (albeit much slower since hitting watermark threshold) up to 1006782 records when I stopped the script. If I understood it right, that table contains at least one record per each channel x exchange and channel x queue permutation, which is 2M in my case. Perhaps if I had enough memory before high watermark to keep (short) history for each of 2M permutations of stats data points it will eventually stabilize at that point? I'll run more tests to check that. I have ETS memory snapshot logs for this 10 hours test if you are interested to see it yourself. -- View this message in context: http://rabbitmq.1065348.n5.nabble.com/Possible-memory-leak-in-the-management-plugin-tp27414p34790.html Sent from the RabbitMQ mailing list archive at Nabble.com. From pmaisenovich at blizzard.com Fri Apr 11 04:12:41 2014 From: pmaisenovich at blizzard.com (Pavel) Date: Thu, 10 Apr 2014 20:12:41 -0700 (PDT) Subject: [rabbitmq-discuss] Possible memory leak in the management plugin In-Reply-To: <1397171854071-34790.post@n5.nabble.com> References: <520CB29A.1060304@rabbitmq.com> <1397020502914-34663.post@n5.nabble.com> <534521F1.9030707@rabbitmq.com> <1397058685615-34697.post@n5.nabble.com> <53457674.1040704@rabbitmq.com> <1397106237672-34725.post@n5.nabble.com> <53466D39.4050906@rabbitmq.com> <1397157592459-34779.post@n5.nabble.com> <5346F7AC.2090703@rabbitmq.com> <1397171854071-34790.post@n5.nabble.com> Message-ID: <1397185961431-34791.post@n5.nabble.com> I'm attaching three tests data. In Test 2 and 3 there was 1000 channels x 999 exchanges with only difference in publish rate. Both tests led to Rabbit running OOM (high watermark) and throttling publishers. One new thing I noted after plotting the mgmt_stats_db used memory is that it spikes rapidly shortly before throttling triggered. This is not related to continuous aggregated_stats growth (or it doesn't seem so), but also very concerning. In Test 4 (which is still running) I've reduced number of channels and exchanges by half (500 and 499) to prove the theory of aggregated_stats growth. As you can see from the plot, the number of items in aggregated_stats table have not changed for about an hour, yet the size of the table continues to grow. It's worth noting, that while Rabbit didn't reach high watermark (current usage is [{total,2884223408}, {mgmt_db,1764414344}]) in this test (yet), the management plugin UI is completely unresponsive. The test continues to publish at a steady 500/s rate and I'll leave for the night to see how far it will push the RAM. Test_2_-_1000ch_x_999ex_sequential_publish_with_2ms_delay_(300_per_sec)_-_2_hours.xlsx Test_3_-_1000ch_x_999ex_sequential_publish_with_3ms_delay_(250_per_sec)_-_1_hour.xlsx Test_4_-_500ch_x_499ex_sequential_publish_with_1ms_delay_(500_per_sec).xlsx -- View this message in context: http://rabbitmq.1065348.n5.nabble.com/Possible-memory-leak-in-the-management-plugin-tp27414p34791.html Sent from the RabbitMQ mailing list archive at Nabble.com. From arunrao.seattle at gmail.com Fri Apr 11 06:20:41 2014 From: arunrao.seattle at gmail.com (Arun Rao) Date: Thu, 10 Apr 2014 22:20:41 -0700 Subject: [rabbitmq-discuss] Heartbeat vs timeout in .NET code Message-ID: In .NET client (3.1.4.1), is there a timeout value separately for each connection in addition to heartbeat? I see the timeout value shown in Management URL under connections tab (under Network section of the table) is heartbeat negotiated value by the client? Thanks! -------------- next part -------------- An HTML attachment was scrubbed... URL: From mklishin at gopivotal.com Fri Apr 11 07:41:57 2014 From: mklishin at gopivotal.com (Michael Klishin) Date: Fri, 11 Apr 2014 10:41:57 +0400 Subject: [rabbitmq-discuss] Heartbeat vs timeout in .NET code In-Reply-To: References: Message-ID: On 11 April 2014 at 09:22:55, Arun Rao (arunrao.seattle at gmail.com) wrote: > > In .NET client (3.1.4.1), is there a timeout value separately > for each connection in addition to heartbeat? I see the timeout > value shown in Management URL under connections tab (under Network > section of the table) is heartbeat negotiated value by the client? It is the effective heartbeat interval, yes.? -- MK Software Engineer, Pivotal/RabbitMQ From arunrao.seattle at gmail.com Fri Apr 11 08:49:14 2014 From: arunrao.seattle at gmail.com (Arun Rao) Date: Fri, 11 Apr 2014 00:49:14 -0700 Subject: [rabbitmq-discuss] Heartbeat vs timeout in .NET code In-Reply-To: References: Message-ID: Thanks Michael! So, is that timeout value just for UI or is that also when RabbitMQ connection times out due to inactivity (in which case timeout precedes heartbeat)? On Thu, Apr 10, 2014 at 11:41 PM, Michael Klishin wrote: > On 11 April 2014 at 09:22:55, Arun Rao (arunrao.seattle at gmail.com) wrote: > > > In .NET client (3.1.4.1), is there a timeout value separately > > for each connection in addition to heartbeat? I see the timeout > > value shown in Management URL under connections tab (under Network > > section of the table) is heartbeat negotiated value by the client? > > It is the effective heartbeat interval, yes. > -- > MK > > Software Engineer, Pivotal/RabbitMQ > _______________________________________________ > rabbitmq-discuss mailing list > rabbitmq-discuss at lists.rabbitmq.com > https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss > -------------- next part -------------- An HTML attachment was scrubbed... URL: From stuart.king at me.com Fri Apr 11 03:02:58 2014 From: stuart.king at me.com (Stuart King) Date: Thu, 10 Apr 2014 19:02:58 -0700 Subject: [rabbitmq-discuss] MQTT SSL handshake failures cause server lockup Message-ID: Hi, I?m running RabbitMQ with the MQTT adaptor enabled along with SSL. I?ve discovered that if a small number of connections fail the SSL handshake (e.g. the client rejecting the server?s certificate because it doesn?t recognize the certificate authority) it prevents other SSL connections being established to the server for a short period of time. If a ?bad client? (i.e. one that fails the SSL handshake) keeps trying to reconnect then it effectively locks up the server. A simple way I found to demonstrate this is by creating my own certificate authority, certificates, keys etc for RabbitMQ as per the instructions at https://www.rabbitmq.com/ssl.html and adding the relevant ssl options to the RabbitMQ config file but put the ssl_listeners element into the rabbitmq_mqtt tuple. Then, implement a basic MQTT client in Java using the Paho client libraries. When the client is ran, an exception is thrown as expected since the certificate isn?t trusted. If this client is put into a basic load test, where a new client is created and attempts to connect to the server every 2 seconds, it will prevent other clients connecting. This can be observed by simply using the "openssl s_client? command, which will just hang during the time the Java clients are trying to connect. A sample from the RabbitMQ log file when this occurs is: =ERROR REPORT==== 10-Apr-2014::21:49:33 === SSL: certify: ssl_connection.erl:1724:Fatal error: certificate unknown =ERROR REPORT==== 10-Apr-2014::21:49:38 === ** Generic server <0.689.0> terminating ** Last message in was {inet_async,#Port<0.15200>,48667,{ok,#Port<0.16417>}} ** When Server state == {state, {rabbit_mqtt_sup,start_ssl_client, [[{cacertfile,"/home/sking/ssl/cacert.pem"}, {certfile,"/home/sking/ssl/cert.pem"}, {keyfile,"/home/sking/ssl/key.pem"}, {verify,verify_none}, {fail_if_no_peer_cert,false}]]}, #Port<0.15200>,48667} ** Reason for termination == ** {timeout,{gen_server2,call, [<0.718.0>, {go,#Port<0.16417>, #Fun}]}} =ERROR REPORT==== 10-Apr-2014::21:49:38 === ** Generic server <0.718.0> terminating ** Last message in was {go,#Port<0.16417>,#Fun} ** When Server state == undefined ** Reason for termination == ** {{badmatch,{error,{ssl_upgrade_error,"certificate unknown"}}}, [{rabbit_mqtt_reader,handle_call,3,[]}, {gen_server2,handle_msg,2,[]}, {proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,227}]}]} That particular example is on a server running RabbitMQ 3.1.0 / Erlang R15B03, however I have also reproduced it on the more recent RabbitMQ 3.3.0 / Erlang R16B03-1. I also ran a similar test using AMQP and could not reproduce, so this issue only seems to be with using MQTT over SSL. Regards, Stuart -------------- next part -------------- An HTML attachment was scrubbed... URL: From mklishin at gopivotal.com Fri Apr 11 09:34:44 2014 From: mklishin at gopivotal.com (Michael Klishin) Date: Fri, 11 Apr 2014 12:34:44 +0400 Subject: [rabbitmq-discuss] Heartbeat vs timeout in .NET code In-Reply-To: References: Message-ID: On 11 April 2014 at 11:51:27, Arun Rao (arunrao.seattle at gmail.com) wrote: > > So, is that timeout value just for UI or is that also when RabbitMQ > connection times out due to inactivity (in which case timeout > precedes heartbeat)? It is a interval at which client and server exchange heartbeat frames. 2 skipped heartbeats mean the peer should be considered unreachable.? -- MK Software Engineer, Pivotal/RabbitMQ From mklishin at gopivotal.com Fri Apr 11 09:44:25 2014 From: mklishin at gopivotal.com (Michael Klishin) Date: Fri, 11 Apr 2014 12:44:25 +0400 Subject: [rabbitmq-discuss] MQTT SSL handshake failures cause server lockup In-Reply-To: References: Message-ID: On 11 April 2014 at 12:34:56, Stuart King (stuart.king at me.com) wrote: > > A simple way I found to demonstrate this is by creating my own > certificate authority, certificates, keys etc for RabbitMQ > as per the instructions at https://www.rabbitmq.com/ssl.html > and adding the relevant ssl options to the RabbitMQ config file > but put the ssl_listeners element into the rabbitmq_mqtt tuple. > Then, implement a basic MQTT client in Java using the Paho client > libraries. When the client is ran, an exception is thrown as expected > since the certificate isn?t trusted. If this client is put into > a basic load test, where a new client is created and attempts to > connect to the server every 2 seconds, it will prevent other clients > connecting. This can be observed by simply using the "openssl > s_client? command, which will just hang during the time the Java > clients are trying to connect. Thanks for reporting, a bug is filed.? -- MK Software Engineer, Pivotal/RabbitMQ From arunrao.seattle at gmail.com Fri Apr 11 09:47:43 2014 From: arunrao.seattle at gmail.com (Arun Rao) Date: Fri, 11 Apr 2014 01:47:43 -0700 Subject: [rabbitmq-discuss] Heartbeat vs timeout in .NET code In-Reply-To: References: Message-ID: Actually, I know that already - Client sends a 62 byte packet (of Type: Heartbeat) to server and server responds with a zero (payload) length packet as the response. My specific question was - When does .NET RabbitMQ client that negotiates only Heartbeat argument with the RabbitMQ server timeout? Is that again 2 skipped heartbeats as you mentioned above? If so, Timeout value in Management UI for .NET clients should be twice the value? Thanks! On Fri, Apr 11, 2014 at 1:34 AM, Michael Klishin wrote: > On 11 April 2014 at 11:51:27, Arun Rao (arunrao.seattle at gmail.com) wrote: > > > So, is that timeout value just for UI or is that also when RabbitMQ > > connection times out due to inactivity (in which case timeout > > precedes heartbeat)? > > It is a interval at which client and server exchange heartbeat frames. > 2 skipped heartbeats mean the peer should be considered unreachable. > -- > MK > > Software Engineer, Pivotal/RabbitMQ > _______________________________________________ > rabbitmq-discuss mailing list > rabbitmq-discuss at lists.rabbitmq.com > https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mklishin at gopivotal.com Fri Apr 11 09:51:44 2014 From: mklishin at gopivotal.com (Michael Klishin) Date: Fri, 11 Apr 2014 12:51:44 +0400 Subject: [rabbitmq-discuss] Heartbeat vs timeout in .NET code In-Reply-To: References: Message-ID: ?On 11 April 2014 at 12:50:00, Arun Rao (arunrao.seattle at gmail.com) wrote: > > My specific question was - When does .NET RabbitMQ client that > negotiates only Heartbeat argument with the RabbitMQ server > timeout? Is that again 2 skipped heartbeats as you mentioned > above?? After 2 missed heartbeats from server. > If so, Timeout value in Management UI for .NET clients > should be twice the value? Sounds like it and I suspect the name ?timeout? wasn?t as clear as we expected to be. -- MK Software Engineer, Pivotal/RabbitMQ From dmalinovsky at thumbtack.net Fri Apr 11 11:28:02 2014 From: dmalinovsky at thumbtack.net (Dmitry Malinovsky) Date: Fri, 11 Apr 2014 03:28:02 -0700 (PDT) Subject: [rabbitmq-discuss] CRL support & verify_fun Message-ID: <880cfca2-4982-4c9c-9090-f7092f4020bb@googlegroups.com> Hi, I need help with a custom verify_fun to have an ability to read CRL pem file and validate client certificates (since RabbitMQ/Erlang does not support it). I'm not Erlang programmer (I'm more about Python), and I have some troubles with understanding things. Documentation says that I can specify a custom verify_fun this way: {verify_fun, {Module, Function}} Where should I put my Module (e.g. mymod.beam) contains Function (e.g. check_crl/1) to make sure that RabbitMQ will find and use this Module? Should I specify Function's arity in the config file? Thanks! -------------- next part -------------- An HTML attachment was scrubbed... URL: From simon at rabbitmq.com Fri Apr 11 12:49:08 2014 From: simon at rabbitmq.com (Simon MacMullen) Date: Fri, 11 Apr 2014 12:49:08 +0100 Subject: [rabbitmq-discuss] Forcing publish/confirm to return a nack In-Reply-To: References: Message-ID: <5347D6B4.6050302@rabbitmq.com> On 10/04/2014 22:35, Ryan Brown wrote: > I am just about finished working through my new publish/confirm workflow > and would like to test non-standard workflows in my integration > environment. Would there be a way that I could force rabbit to return, > say, a basic.nack? The confirm mechanism will only return a nack on a queue crash - not on queue deletion or queue's node going down. So this is comparatively hard to test, since you're not exactly meant to be able to provoke a queue to crash. About the only way you can do it is with something like: rabbitmqctl eval '{ok, Q} = rabbit_amqqueue:lookup(rabbit_misc:r(<<"/">>, queue, <<"test">>)), exit(element(7, Q), die).' Cheers, Simon From mklishin at gopivotal.com Fri Apr 11 13:30:27 2014 From: mklishin at gopivotal.com (Michael Klishin) Date: Fri, 11 Apr 2014 16:30:27 +0400 Subject: [rabbitmq-discuss] ANN Bunny 1.2.1 is released Message-ID: Bunny [1] is a Ruby RabbitMQ client that focuses on ease of use. Release notes: http://blog.rubyrabbitmq.info/blog/2014/04/11/bunny-1-dot-2-1-is-released/ 1. http://rubybunny.info? -- MK Software Engineer, Pivotal/RabbitMQ From simon at rabbitmq.com Fri Apr 11 14:51:21 2014 From: simon at rabbitmq.com (Simon MacMullen) Date: Fri, 11 Apr 2014 14:51:21 +0100 Subject: [rabbitmq-discuss] Possible memory leak in the management plugin In-Reply-To: <1397171854071-34790.post@n5.nabble.com> References: <520CB29A.1060304@rabbitmq.com> <1397020502914-34663.post@n5.nabble.com> <534521F1.9030707@rabbitmq.com> <1397058685615-34697.post@n5.nabble.com> <53457674.1040704@rabbitmq.com> <1397106237672-34725.post@n5.nabble.com> <53466D39.4050906@rabbitmq.com> <1397157592459-34779.post@n5.nabble.com> <5346F7AC.2090703@rabbitmq.com> <1397171854071-34790.post@n5.nabble.com> Message-ID: <5347F359.7010504@rabbitmq.com> On 11/04/2014 00:17, Pavel wrote: > aggregated_stats table was > growing steadily for almost 4 hours straight until it reached 2.5Gb One thing occurs to me: the ETS GC part of the management DB decides on how many rows to GC per iteration in advance, at the start of each run through the table. So if you started with 10,000 rows in the table, it would decide to read 100 rows at a time. If the table then increased to 1,000,000 rows it would still only manage 100 at a time until it had traversed the table completely - taking most of a day to do so. And your test rather resembles this case. So the GC code probably needs to be smarter about how it does this. Cheers, Simon From mcintoshj at gmail.com Fri Apr 11 15:20:29 2014 From: mcintoshj at gmail.com (Jason McIntosh) Date: Fri, 11 Apr 2014 09:20:29 -0500 Subject: [rabbitmq-discuss] Rabbit startup command is hanging In-Reply-To: References: Message-ID: SO a final email on this. I ended up having to kill all the processes on all nodes in the cluster, then starting them back up in order to recover. At that point, the node that wouldn't rejoin the cluster came online and started syncing messages and responding fine. I'm guessing I had a deadlock someplace though I'm not totally sure where it would be. I'll keep an eye on this and see what else I can discover. *SIGH* I really need to learn to debug and work with erlang better, Thanks all, Jason On Thu, Apr 10, 2014 at 4:22 PM, Jason McIntosh wrote: > SO now the fun part. I decided to try and rebuild the middle node (I have > boxes 10, 11 and 12). However, I can't get the middle node to reconnect to > the cluster. Removing it's mnesia directory allowed it to start, but it > can't rejoin the cluster. SO I tried removing the node from the cluster, > e.g.: > > rabbitmqctl -n cluster at rabbitmqm10 forget_cluster_node cluster at rabbitmqm11 > > But the above never responds - it's just sitting there hanging. > > rabbitmqctl -n cluster at rabbitmqm11 status FROM the other nodes all works > fine. I'm about at a loss as to how the heck to repair things. I can't > remove the node from the cluster, I can't start it with the mnesia > directory in it's current state, and removing the mnesia directory and > trying to add it back in is failing - it fails with "....done > (already_member).". Trying to do rabbitmqctl update_cluster_nodes > cluster at rabbitmqm10 is sitting there doing nothing and not responding > either. > > > I'm starting to really worry I'm going to have to completely rebuild my > cluster... > Jason > > > > On Thu, Apr 10, 2014 at 2:55 PM, Jason McIntosh wrote: > >> Not sure what's going on here. Just ugpraded my cluster from 3.2.3 to >> 3.2.4 (including a restart of the machine). On startup, two of my initial >> nodes started fine, but when the third node in the cluster started, the >> "/etc/init.d/rabbitmq-server start" just sits at "Starting rabbitmq-server: >> " without ever finishing. Doing a rabbitmqctl status shows: >> Status of node cluster at rabbitmqm11p ... >> [{pid,62505}, >> {running_applications,[{os_mon,"CPO CXC 138 46","2.2.14"}, >> {inets,"INETS CXC 138 49","5.9.8"}, >> {mnesia,"MNESIA CXC 138 12","4.11"}, >> {amqp_client,"RabbitMQ AMQP Client","3.2.4"}, >> {xmerl,"XML parser","1.3.6"}, >> {eldap,"Ldap api","1.0.2"}, >> {sasl,"SASL CXC 138 11","2.3.4"}, >> {stdlib,"ERTS CXC 138 10","1.19.4"}, >> {kernel,"ERTS CXC 138 10","2.16.4"}]}, >> {os,{unix,linux}}, >> {erlang_version,"Erlang R16B03-1 (erts-5.10.4) [source] [64-bit] >> [smp:24:24] [async-threads:30] [hipe] [kernel-poll:true]\n"}, >> {memory,[{total,48504352}, >> {connection_procs,2808}, >> {queue_procs,0}, >> {plugins,0}, >> {other_proc,16290632}, >> {mnesia,1783536}, >> {mgmt_db,0}, >> {msg_index,0}, >> {other_ets,1120896}, >> {binary,725448}, >> {code,19691642}, >> {atom,703377}, >> {other_system,8186013}]}, >> {file_descriptors,[{total_limit,12188}, >> {total_used,0}, >> {sockets_limit,10967}, >> {sockets_used,0}]}, >> {processes,[{limit,1048576},{used,117}]}, >> {run_queue,0}, >> {uptime,83}] >> ...done. >> >> >> In the web management interface, I see this: >> Node statistics not available >> Memory details >> >> Connections 2.7kB Queues 0B Plugins 0B Other process memory 16MB >> Mnesia 1.7MB Message store index 0B Management database 0B Other ETS >> tables 1.1MB Binaries 708kB Code 19MB Atoms 687kB Other system 7.8MB >> >> >> SO rabbit appears to have sort of started, but certain things are not >> started (e.g. plugins). Plugins list is: >> [e] amqp_client 3.2.4 >> [ ] cowboy 0.5.0-rmq3.2.4-git4b93c2d >> [ ] eldap 3.2.4-gite309de4 >> [e] mochiweb 2.7.0-rmq3.2.4-git680dba8 >> [ ] rabbitmq_amqp1_0 3.2.4 >> [E] rabbitmq_auth_backend_ldap 3.2.4 >> [ ] rabbitmq_auth_mechanism_ssl 3.2.4 >> [E] rabbitmq_consistent_hash_exchange 3.2.4 >> [E] rabbitmq_federation 3.2.4 >> [E] rabbitmq_federation_management 3.2.4 >> [ ] rabbitmq_jsonrpc 3.2.4 >> [ ] rabbitmq_jsonrpc_channel 3.2.4 >> [ ] rabbitmq_jsonrpc_channel_examples 3.2.4 >> [E] rabbitmq_management 3.2.4 >> [E] rabbitmq_management_agent 3.2.4 >> [E] rabbitmq_management_visualiser 3.2.4 >> [ ] rabbitmq_mqtt 3.2.4 >> [E] rabbitmq_shovel 3.2.4 >> [E] rabbitmq_shovel_management 3.2.4 >> [ ] rabbitmq_stomp 3.2.4 >> [ ] rabbitmq_tracing 3.2.4 >> [e] rabbitmq_web_dispatch 3.2.4 >> [ ] rabbitmq_web_stomp 3.2.4 >> [ ] rabbitmq_web_stomp_examples 3.2.4 >> [ ] rfc4627_jsonrpc 3.2.4-git5e67120 >> [ ] sockjs 0.3.4-rmq3.2.4-git3132eb9 >> [e] webmachine 1.10.3-rmq3.2.4-gite9359c7 >> >> >> Any suggestions on next steps on debugging this? Or what I can do to get >> this back up and in a "healthy" state? >> >> Thanks! >> Jason >> >> >> >> >> -- >> Jason McIntosh >> https://github.com/jasonmcintosh/ >> 573-424-7612 >> > > > > -- > Jason McIntosh > https://github.com/jasonmcintosh/ > 573-424-7612 > -- Jason McIntosh https://github.com/jasonmcintosh/ 573-424-7612 -------------- next part -------------- An HTML attachment was scrubbed... URL: From Anita.Yip at RealPage.com Fri Apr 11 16:30:27 2014 From: Anita.Yip at RealPage.com (Anita Yip) Date: Fri, 11 Apr 2014 15:30:27 +0000 Subject: [rabbitmq-discuss] frame_too_large error while connecting to In-Reply-To: References: Message-ID: I am trying to connect to a cluster of 2 RabbitMQ servers, and using 'node-amqp' module. Recently, the servers are open to public access for a demo purpose. Before that I never saw this issue. And even now, if I connect to the servers using their hostnames via vpn, I do not see this issue. But if I use the ip addresses, whether or not I am on vpn, the issue occurs. Anita Yip RealPage, Inc.360 Third Street Suite 450 San Francisco, CA 94107 Office: 415-395-2109 Cell: 415-200-7312 Anita.yip at realpage.com www.realpage.com Connect with RealPage: On 4/10/14 12:50 PM, "Michael Klishin" wrote: > On 10 April 2014 at 23:23:46, Anita Yip (anita.yip at realpage.com) wrote: >> > Unhandled connection error: FRAME_ERROR - type 65, all octets >> = <<>>: {frame_too_large,1342177289,131064(tel://1342177289,131064)} >> { [Error: FRAME_ERROR - type 65, all octets = <<>>: >>{frame_too_large,1342177289,131064(tel://1342177289,131064)}] >> code: 501 } > >This almost certainly indicates a client library bug. What do you use? >Can you post a small script/program that reproduces the problem most of >the time? >-- >MK > >Software Engineer, Pivotal/RabbitMQ >_______________________________________________ >rabbitmq-discuss mailing list >rabbitmq-discuss at lists.rabbitmq.com >https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss ________________________________ This message is intended only for the use of the individual(s) or entity to which it is addressed and may contain information that is privileged, confidential, and/or proprietary to RealPage and its affiliated companies. If the reader of this message is not the intended recipient, you are hereby notified that any dissemination, distribution, forwarding or copying of this communication is prohibited without the express permission of the sender. If you have received this communication in error, please notify the sender immediately and delete the original message. From josephrouphael at gmail.com Fri Apr 11 17:37:21 2014 From: josephrouphael at gmail.com (joseph rouphael) Date: Fri, 11 Apr 2014 19:37:21 +0300 Subject: [rabbitmq-discuss] Fwd: Performance bottleneck on inter-node connection In-Reply-To: References: <53465CF5.20607@rabbitmq.com> <5346C06C.8090609@rabbitmq.com> Message-ID: ---------- Forwarded message ---------- From: joseph rouphael Date: Fri, Apr 11, 2014 at 4:11 PM Subject: Re: [rabbitmq-discuss] Performance bottleneck on inter-node connection To: Matthias Radestock On Thu, Apr 10, 2014 at 7:01 PM, Matthias Radestock wrote: > (re-adding the list) > > > On 10/04/14 16:42, joseph rouphael wrote: > >> On Thu, Apr 10, 2014 at 11:57 AM, Matthias Radestock >> > wrote: >> Have you tried this w/o hipe? >> In fact, HIPE is not enabled in Erlang, so this config has no effect. >> (You can consider Hipe off) >> > > ok, I'd still prefer if you took out that setting. OK, setting removed. (Test re-done with same result) > > > {delegate_count, 32}, >> Why did you set this? >> I have used this in a try to increase the number of delegate processes >> used for intra-cluster configuration. Hoping that will increase >> inter-node communication performance (But this had no effect) >> > > Ditto. > > > In fact, I am not using any synchronous mechanism. I am publishing >> messages with no confirm.select. >> I am using a c client (based on rabbitmq-c) to publish messages with the >> following config: >> - Non persistent >> - No confirm.select >> - Default Direct exchange >> - Routing key equal to queue_name >> - Message size 1KB >> I am running 10 client processes which are publishing in parallel to 10 >> different queues. >> Rate is regulated on the client, I can force the publsih rate in the C >> client. >> >> I am running as well 10 other client processes to consume full-speed (no >> rate regulation) from the 10 queues. >> > > Do the consumers use acknowledgements? Yes. Consumers are acknowledging the messages as soon as they are received (No processing in between) I have re-done the test with no-ack and I have encountered the same bottleneck.The rate achieved with no-ack was fluctuating between 18KHZ and 27KHZ (The expected rate is 100KHZ) > > I am using the same setup to publish/consume messages on one node (node >> 1 where queues are created), with no issue. I can hit up to 100K TPS (5K >> TPS on each client process). >> The problem appears only when I try to publish on node1 and consume from >> node2 (or publish on node2 and consume from node1). The problem appears >> only when there is an inter-node activity. >> > > So the 100kHz is the aggregate publishing and consuming rate, i.e. 10 x > 5kHz publishing and 10 x 5kHz consuming? > Sorry my mistake, there are 20 processes with 20 queues (Not 10). So 100KHZ (20 x 5) producing and 100KHZ (20 x 5) consuming > > What results do you get in the single-node and multi-node scenarios when > running the tests with just a single queue and single producer and single > consumer? > > Single producer/consumer/queue on single node max rate achieved: 11KHZ Single producer/consumer/queue on dual nodes max rate achieved: ~ 11KHZ The impact of inter-node is not seen here because the traffic is limited by the single client connection/process. In single consumer/producer connection, the throughput is limited by the client connection itself not by the inter-node connection. Here are below a summary of the tests performed: Test1: Single node, 20 queues, ACK on ------------------------------------ Nodes: Single node Producer Processes: 20 Producer process regulate rate: 5KHZ Acknowledgment: ON Expected rate: 100 KHZ Achieved rate: Steady at 100 KHZ CPU idle: 33% Note: No bottleneck detected Test 2: Single node, 20 queues, ACK off --------------------------------------- Nodes: Single node Producer Processes: 20 Producer process regulate rate: 5KHZ Acknowledgment: OFF Expected rate: 100 KHZ Achieved rate: Steady at 100 KHZ CPU idle: 57% Note: No bottleneck detected Test 3: Single node, 1 queue, ACK on ------------------------------------ Nodes: Single node Producer Processes: 1 Producer process regulate rate: 20KHZ Acknowledgment: ON Expected rate: 20 KHZ Achieved rate: Steady 11 KHZ CPU idle: 94% Note: Bottleneck on single client connection Test 4: Single node, 1 queue, ACK off ------------------------------------ Nodes: Single node Producer Processes: 1 Producer process regulate rate: 20KHZ Acknowledgment: OFF Expected rate: 20 KHZ Achieved rate: Steady 11 KHZ CPU idle: 96% Note: Bottleneck on single client connection Test 5: Dual node, Queues/Producers on same node, 20 queue, ACK on ------------------------------------------------------------------ Nodes: Double node Queues created on: Node1 Producers on: node 1 Consumers on: node 2 Producer Processes: 20 Producer regulate rate per process: 5KHZ Acknowledgment: ON Expected rate: 100KHZ Achieved rate: Fluctuating between 12KHZ and 21KHZ CPU idle: 88% Note: Bottleneck on inter-node connection Test 6: Dual node, Queues/Producers on same node, 20 queue, ACK off ------------------------------------------------------------------- Nodes: Double node Queues created on: Node1 Producers on: node 1 Consumers on: node 2 Producer Processes: 20 Producer regulate rate per process: 5KHZ Acknowledgment: OFF Expected rate: 100KHZ Achieved rate: Fluctuating between 18KHZ and 27KHZ CPU idle: 91% Note: Bottleneck on inter-node connection Test 7: Dual node, Queues/Consumers on same node, 20 queue, ACK on ------------------------------------------------------------------ Nodes: Double node Queues created on: Node1 Producers on: node 2 Consumers on: node 1 Producer Processes: 20 Producer regulate rate per process: 5KHZ Acknowledgment: ON Expected rate: 100KHZ Achieved rate: Fluctuating between 10KHZ and 20KHZ CPU idle: 91% Note: Bottleneck on inter-node connection Test 8: Dual node, Queues/Consumers on same node, 20 queue, ACK off ------------------------------------------------------------------- Nodes: Double node Queues created on: Node1 Producers on: node 2 Consumers on: node 1 Producer Processes: 20 Producer regulate rate per process: 5KHZ Acknowledgment: OFF Expected rate: 100KHZ Achieved rate: Fluctuating between 10KHZ and 20KHZ CPU idle: 95% Note: Bottleneck on inter-node connection Test 9: Dual node, 1 queue, ACK off ------------------------------------ Nodes: Double node Queues created on: Node1 Producers on: node 2 Consumers on: node 1 Producer Processes: 1 Producer regulate rate per process: 20KHZ Acknowledgment: OFF Expected rate: 20KHZ Achieved rate: Steady 10.5KHZ CPU idle: 96% Note: Bottleneck on single client connection Test 10: Dual node, 1 queue, ACK on ------------------------------------ Nodes: Double node Queues created on: Node1 Producers on: node 2 Consumers on: node 1 Producer Processes: 1 Producer regulate rate per process: 20KHZ Acknowledgment: ON Expected rate: 20KHZ Achieved rate: Steady 10.5KHZ CPU idle: 95% Note: Bottleneck on single client connection > Regards, > > Matthias. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From loganathan.ms at gmail.com Fri Apr 11 20:05:56 2014 From: loganathan.ms at gmail.com (Loganathan Sellappa) Date: Fri, 11 Apr 2014 12:05:56 -0700 (PDT) Subject: [rabbitmq-discuss] How to access RabbitMq publicly Message-ID: HI All, I have installed & setup the Rabbitmq on Centos remote server. Later I created an file "rabbitmq.config" and added the line *[{rabbit, [{loopback_users, []}]}] and then restarted the rabbitmq server. Again *tried to login the rabbitmq management web interface from my local machine using the guest credentials, but getting *login failed* error message. Can anybody let me know the proper way to empty the loopback user settings for Rabbitmq. Regards, Loganathanne -------------- next part -------------- An HTML attachment was scrubbed... URL: From mklishin at gopivotal.com Fri Apr 11 21:56:53 2014 From: mklishin at gopivotal.com (Michael Klishin) Date: Sat, 12 Apr 2014 00:56:53 +0400 Subject: [rabbitmq-discuss] frame_too_large error while connecting to In-Reply-To: References: Message-ID: On 11 April 2014 at 19:31:51, Anita Yip (anita.yip at realpage.com) wrote: > > I am trying to connect to a cluster of 2 RabbitMQ servers, and > using > 'node-amqp' module. Recently, the servers are open to public > access for a > demo purpose. Before that I never saw this issue. And even now, > if I > connect to the servers using their hostnames via vpn, I do not > see this > issue. But if I use the ip addresses, whether or not I am on vpn, > the > issue occurs. This suggests an issue in the client . node-amqp is not particularly actively maintained, I recommend amqp.node [1]. 1.?https://github.com/squaremo/amqp.node/ -- MK Software Engineer, Pivotal/RabbitMQ From mklishin at gopivotal.com Fri Apr 11 21:59:56 2014 From: mklishin at gopivotal.com (Michael Klishin) Date: Sat, 12 Apr 2014 00:59:56 +0400 Subject: [rabbitmq-discuss] How to access RabbitMq publicly In-Reply-To: References: Message-ID: On 11 April 2014 at 23:08:09, Loganathan Sellappa (loganathan.ms at gmail.com) wrote: > > I have installed & setup the Rabbitmq on Centos remote server. > Later I created an file "rabbitmq.config" and added the line > [{rabbit, [{loopback_users, []}]}] and then restarted the > rabbitmq server. Again tried to login the rabbitmq management > web interface from my local machine using the guest credentials, > but getting login failed error message. Can anybody let me know > the proper way to empty the loopback user settings for Rabbitmq. How did you install RabbitMQ and where the config file is located?? -- MK Software Engineer, Pivotal/RabbitMQ From mparrish at catbird.com Fri Apr 11 22:59:48 2014 From: mparrish at catbird.com (Mike Parrish) Date: Fri, 11 Apr 2014 14:59:48 -0700 Subject: [rabbitmq-discuss] Consulting services Message-ID: Hi, We're developing a plugin to do authorization/authentication on a Windows box which replaces username/password searches in the mnesia database with logins to the Windows API. Erlang/NIF project. Is there a group within RabbitMQ/VMWare that does integration services for this type of project? Regards, Mike Parrish Catbird 831-295-4237 -------------- next part -------------- An HTML attachment was scrubbed... URL: From videlalvaro at gmail.com Fri Apr 11 23:12:22 2014 From: videlalvaro at gmail.com (Alvaro Videla) Date: Sat, 12 Apr 2014 00:12:22 +0200 Subject: [rabbitmq-discuss] Consulting services In-Reply-To: References: Message-ID: Hi Mike, Please contact support at rabbitmq.com for these kind of inquiries. Regards, Alvaro On Fri, Apr 11, 2014 at 11:59 PM, Mike Parrish wrote: > Hi, > > We're developing a plugin to do authorization/authentication on a Windows > box which replaces username/password searches in the mnesia database with > logins to the Windows API. Erlang/NIF project. > > Is there a group within RabbitMQ/VMWare that does integration services for > this type of project? > > > Regards, > Mike Parrish > Catbird > 831-295-4237 > > _______________________________________________ > rabbitmq-discuss mailing list > rabbitmq-discuss at lists.rabbitmq.com > https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss > From mparrish at catbird.com Fri Apr 11 23:17:02 2014 From: mparrish at catbird.com (Mike Parrish) Date: Fri, 11 Apr 2014 15:17:02 -0700 Subject: [rabbitmq-discuss] Consulting services In-Reply-To: References: Message-ID: Sent. Thanks for the pointer. On Fri, Apr 11, 2014 at 3:12 PM, Alvaro Videla wrote: > Hi Mike, > > Please contact support at rabbitmq.com for these kind of inquiries. > > Regards, > > Alvaro > > > > On Fri, Apr 11, 2014 at 11:59 PM, Mike Parrish > wrote: > > Hi, > > > > We're developing a plugin to do authorization/authentication on a Windows > > box which replaces username/password searches in the mnesia database with > > logins to the Windows API. Erlang/NIF project. > > > > Is there a group within RabbitMQ/VMWare that does integration services > for > > this type of project? > > > > > > Regards, > > Mike Parrish > > Catbird > > 831-295-4237 > > > > _______________________________________________ > > rabbitmq-discuss mailing list > > rabbitmq-discuss at lists.rabbitmq.com > > https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss > > > _______________________________________________ > rabbitmq-discuss mailing list > rabbitmq-discuss at lists.rabbitmq.com > https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss > -------------- next part -------------- An HTML attachment was scrubbed... URL: From james at bellenger.org Fri Apr 11 23:38:31 2014 From: james at bellenger.org (James Bellenger) Date: Fri, 11 Apr 2014 22:38:31 +0000 Subject: [rabbitmq-discuss] java client reconnect error: attempt to reuse consumer tag Message-ID: <0000014552f1370c-4c840fae-ccf3-4db4-9726-b84650b22206-000000@email.amazonses.com> Hello. I've been testing the client autorecovery support in 3.3.0 and have had some issues getting it to work. On my machine, recovery appears to work at fewer than about 5 consumers. Anything over that, and the client gets a stream of TopologyRecoveryExceptions and is never able to reconnect. I've looked at the client source and haven't seen any obvious culprit. Has anyone else seen this? TEST SETUP: client jar: amqp-client-3.3.0.jar server: rabbitmq-server 3.1.3-1 (from ubuntu 13.10) Triggering autorecovery via 'service rabbitmq-server restart' CLIENT CODE (scala): package rabbittest import com.rabbitmq.client._ object RecoveryTest { // the consumer count matters. At ~5 consumers the client gets put into a bad state and never reconnects // Fewer than 5, the client is usually able to reconnect val consumers = 10 def main(args: Array[String]) { val factory = new ConnectionFactory() factory.setUri("amqp://guest at localhost:5672/test") factory.setAutomaticRecoveryEnabled(true) val conn = factory.newConnection() for (x <- 1 to consumers) { val chan = conn.createChannel() val result = chan.queueDeclare() val tag = chan.basicConsume(result.getQueue, new DefaultConsumer(chan)) println(s"channel=${chan.getChannelNumber} queue=${result.getQueue} consumer=$tag") } } } GOOD LOGS: channel=1 queue=amq.gen-KvCBOoSjkyhpqle8GMGB0Q consumer=amq.ctag-4WBZRhTLhp_mnQDCXs1NGg channel=2 queue=amq.gen-bNUrlmqi2nHZiunZ9qy-UA consumer=amq.ctag-2_GR91qniV39a-qTtK9Beg channel=3 queue=amq.gen-li-v7cR25J7kmQ-YfcqbCQ consumer=amq.ctag-llfj8Zf1yFCnIGxpU0EI8A BAD LOGS (topology exceptions repeat on every reconnect attempt) channel=1 queue=amq.gen-TIierQJcDNd93QzX8gYuww consumer=amq.ctag-grxPT0_DsPA7WccDmbNBrg channel=2 queue=amq.gen-JaXYQy218NUlWGQCubjjhg consumer=amq.ctag-Toh8E8Ogvmb3UNkKcJszxg channel=3 queue=amq.gen-VzRS5YVgjP7Lz3UfhxYBgA consumer=amq.ctag-LOiutBNDPk8Dl4gUWxJgdg channel=4 queue=amq.gen-JNJAT8cGvVPRePX43hSPtg consumer=amq.ctag-0nyXfLeLl9lsApz3XsT0Iw channel=5 queue=amq.gen-Biake-GSvzyuCFe1b-4FVA consumer=amq.ctag-xtZbH08UoJnJKhi52vn_1g channel=6 queue=amq.gen-apsJKNLSgN3Dv0VBqPLqSQ consumer=amq.ctag-OwrpVcvqzqB3aQ42PdQY2g channel=7 queue=amq.gen-KxXs5Hkd1W2I1WPxIydpMQ consumer=amq.ctag-KPZzHfmW2f7Cni5u4gQCvA channel=8 queue=amq.gen-ZeLhlueGGsAXW_GhuOVS5w consumer=amq.ctag-54Raw5XNJp2W1-uQUcP2aw channel=9 queue=amq.gen-UZW1yR4QG2Sz90RM9CulQA consumer=amq.ctag-kDDpMuN3vaP-MLeH4Fk7oA channel=10 queue=amq.gen-Ejgc9jZrg6q1UH3qN2VXuA consumer=amq.ctag-3OxICESMijMUMgZOURXNmw Caught an exception when recovering topology Caught an exception while recovering consumer amq.ctag-LOiutBNDPk8Dl4gUWxJgdg com.rabbitmq.client.TopologyRecoveryException: Caught an exception while recovering consumer amq.ctag-LOiutBNDPk8Dl4gUWxJgdg at com.rabbitmq.client.impl.recovery.AutorecoveringConnection.recoverConsumers(AutorecoveringConnection.java:488) at com.rabbitmq.client.impl.recovery.AutorecoveringConnection.beginAutomaticRecovery(AutorecoveringConnection.java:365) at com.rabbitmq.client.impl.recovery.AutorecoveringConnection.access$000(AutorecoveringConnection.java:47) at com.rabbitmq.client.impl.recovery.AutorecoveringConnection$1.shutdownCompleted(AutorecoveringConnection.java:344) at com.rabbitmq.client.impl.ShutdownNotifierComponent.notifyListeners(ShutdownNotifierComponent.java:75) at com.rabbitmq.client.impl.AMQConnection$MainLoop.run(AMQConnection.java:572) at java.lang.Thread.run(Thread.java:744) Caused by: java.io.IOException at com.rabbitmq.client.impl.AMQChannel.wrap(AMQChannel.java:106) at com.rabbitmq.client.impl.AMQChannel.wrap(AMQChannel.java:102) at com.rabbitmq.client.impl.ChannelN.basicConsume(ChannelN.java:995) at com.rabbitmq.client.impl.recovery.AutorecoveringChannel.basicConsume(AutorecoveringChannel.java:312) at com.rabbitmq.client.impl.recovery.RecordedConsumer.recover(RecordedConsumer.java:45) at com.rabbitmq.client.impl.recovery.AutorecoveringConnection.recoverConsumers(AutorecoveringConnection.java:481) ... 6 more Caused by: com.rabbitmq.client.ShutdownSignalException: connection error; reason: #method(reply-code=530, reply-text=NOT_ALLOWED - attempt to reuse consumer tag 'amq.ctag-LOiutBNDPk8Dl4gUWxJgdg', class-id=60, method-id=20) at com.rabbitmq.utility.ValueOrException.getValue(ValueOrException.java:67) at com.rabbitmq.utility.BlockingValueOrException.uninterruptibleGetValue(BlockingValueOrException.java:33) at com.rabbitmq.client.impl.AMQChannel$BlockingRpcContinuation.getReply(AMQChannel.java:343) at com.rabbitmq.client.impl.ChannelN.basicConsume(ChannelN.java:993) ... 9 more Caught an exception when recovering topology Caught an exception while recovering consumer amq.ctag-54Raw5XNJp2W1-uQUcP2aw com.rabbitmq.client.TopologyRecoveryException: Caught an exception while recovering consumer amq.ctag-54Raw5XNJp2W1-uQUcP2aw at com.rabbitmq.client.impl.recovery.AutorecoveringConnection.recoverConsumers(AutorecoveringConnection.java:488) at com.rabbitmq.client.impl.recovery.AutorecoveringConnection.beginAutomaticRecovery(AutorecoveringConnection.java:365) at com.rabbitmq.client.impl.recovery.AutorecoveringConnection.access$000(AutorecoveringConnection.java:47) at com.rabbitmq.client.impl.recovery.AutorecoveringConnection$1.shutdownCompleted(AutorecoveringConnection.java:344) at com.rabbitmq.client.impl.ShutdownNotifierComponent.notifyListeners(ShutdownNotifierComponent.java:75) at com.rabbitmq.client.impl.AMQConnection$MainLoop.run(AMQConnection.java:572) at java.lang.Thread.run(Thread.java:744) Caused by: com.rabbitmq.client.AlreadyClosedException: connectionconnection error; reason: #method(reply-code=530, reply-text=NOT_ALLOWED - attempt to reuse consumer tag 'amq.ctag-LOiutBNDPk8Dl4gUWxJgdg', class-id=60, method-id=20) at com.rabbitmq.client.impl.AMQChannel.ensureIsOpen(AMQChannel.java:190) at com.rabbitmq.client.impl.AMQChannel.rpc(AMQChannel.java:223) at com.rabbitmq.client.impl.ChannelN.basicConsume(ChannelN.java:981) at com.rabbitmq.client.impl.recovery.AutorecoveringChannel.basicConsume(AutorecoveringChannel.java:312) at com.rabbitmq.client.impl.recovery.RecordedConsumer.recover(RecordedConsumer.java:45) at com.rabbitmq.client.impl.recovery.AutorecoveringConnection.recoverConsumers(AutorecoveringConnection.java:481) ... 6 more From mklishin at gopivotal.com Sat Apr 12 09:10:17 2014 From: mklishin at gopivotal.com (Michael Klishin) Date: Sat, 12 Apr 2014 12:10:17 +0400 Subject: [rabbitmq-discuss] ANN DO NOT use amqp gem unless you already use EventMachine Message-ID: As the maintainer of 3 most widely used Ruby RabbitMQ clients, I have a little announcement to make. I?ve been meaning to suggest this in amqp gem README for a long time and finally it?s time to do so: if you don?t use EventMachine, avoid amqp gem. There is no reason to use it over Bunny or March Hare. The oddities of EventMachine, mostly around error handling, make amqp gem a pain for beginners. The library can only hide/work around some of it. Bunny and March Hare are much better clients in what they focus on: ease of use (the former) or best possible throughput/latency (the latter). This is now stated loud and clear in the README: https://github.com/ruby-amqp/amqp#a-word-of-warning-use-this-only-if-you-already-use-eventmachine but I wanted to mention it here as well. Happy hacking. -- MK Software Engineer, Pivotal/RabbitMQ From mklishin at gopivotal.com Sat Apr 12 11:36:32 2014 From: mklishin at gopivotal.com (Michael Klishin) Date: Sat, 12 Apr 2014 14:36:32 +0400 Subject: [rabbitmq-discuss] java client reconnect error: attempt to reuse consumer tag In-Reply-To: <0000014552f1370c-4c840fae-ccf3-4db4-9726-b84650b22206-000000@email.amazonses.com> References: <0000014552f1370c-4c840fae-ccf3-4db4-9726-b84650b22206-000000@email.amazonses.com> Message-ID: ?On 12 April 2014 at 02:39:53, James Bellenger (james at bellenger.org) wrote: > > On my machine, recovery appears to work at fewer than about 5 > consumers. Anything over that, and the client gets a stream of > TopologyRecoveryExceptions and is never able to reconnect. This is annoying but reconnection happens before topology recovery, so it is able to reconnect. > I've looked at the client source and haven't seen any obvious > culprit. I cannot immediately say what may be going here but there is one default = 5 in the client. ConsumerWorkService uses a 5 thread executor by default (instantiated with?Executors.newFixedThreadPool). Try setting a custom executor that has, say, 64 threads and configure ConnectionFactory with it, and let us know how it goes. -- MK Software Engineer, Pivotal/RabbitMQ From greg.poirier at opower.com Sat Apr 12 15:08:19 2014 From: greg.poirier at opower.com (Greg Poirier) Date: Sat, 12 Apr 2014 07:08:19 -0700 Subject: [rabbitmq-discuss] RabbitMQ Deployment Architectures (for lack of a better subject) Message-ID: I am trying to determine if it is possible/feasible to provide a single "router" of RabbitMQ systems to backing "storage" RabbitMQ systems. Some explanation would probably be helpful... Not all of our services have the same SLA. Some can tolerate certain amounts of message loss... others cannot tolerate message loss well at all. In order to address this, I'm looking at deploying several different configurations of RabbitMQ clusters. That being said, I'd like for services not to have to know which RabbitMQ cluster they have to connect to. I would like to effectively have a set of "router" hosts that are all configured identically that federate exchanges to the various "storage" clusters that take care of our SLA requirements (mirror count, number of hosts, amount of disk/memory, etc). Is this kind of configuration possible? If both consumers and producers connect to different, non-clustered "router" systems, how do I configure federation to achieve this? -------------- next part -------------- An HTML attachment was scrubbed... URL: From mklishin at gopivotal.com Sat Apr 12 16:06:30 2014 From: mklishin at gopivotal.com (Michael Klishin) Date: Sat, 12 Apr 2014 19:06:30 +0400 Subject: [rabbitmq-discuss] RabbitMQ Deployment Architectures (for lack of a better subject) In-Reply-To: References: Message-ID: On 12 April 2014 at 18:09:43, Greg Poirier (greg.poirier at opower.com) wrote: > > Not all of our services have the same SLA. Some can tolerate certain > amounts of message loss... others cannot tolerate message loss > well at all. In order to address this, I'm looking at deploying > several different configurations of RabbitMQ clusters. > > That being said, I'd like for services not to have to know which > RabbitMQ cluster they have to connect to. I would like to effectively > have a set of "router" hosts that are all configured identically > that federate exchanges to the various "storage" clusters that > take care of our SLA requirements (mirror count, number of hosts, > amount of disk/memory, etc). > Is this kind of configuration possible? One straightforward way of ?routing? would be to use multiple exchanges that are federated to different ?storage? clusters. You can name them sla1, sla2 and so on, for example. > If both consumers and producers connect to different, non-clustered > "router" systems, how do I configure federation to achieve this? Publishers publish to different exchanges in the ?router? cluster, federation propagates messages to N ?storage? clusters where consumers declare the queues they need and bind them to federated [destination] exchanges. Then consume messages as if they were published to their cluster. If you have any specific examples you need help them, please clarify.? -- MK Software Engineer, Pivotal/RabbitMQ From srinath.c at gmail.com Sat Apr 12 16:25:16 2014 From: srinath.c at gmail.com (Srinath C) Date: Sat, 12 Apr 2014 20:55:16 +0530 Subject: [rabbitmq-discuss] Troubleshooting handleCancel() In-Reply-To: References: Message-ID: Hi, Just checking if the logs provide any insights into the probable causes? Regards, Srinath. On Wed, Apr 9, 2014 at 8:34 PM, Srinath C wrote: > Hi MK, > Thanks for your inputs. Below are my replies to your queries: > a, I can say for sure that there is part of this deployment which attempts > to explicit delete a queue or an exchange. > b, Queue master node failure could be a possibility. But I'm not sure if > the failed node was the master. How do you find it out by the way? > c, I have attached the rabbitmq logs for the surviving node. I have > cropped the logs to a few minutes before and after the handleCancel was > called. > d, I can say for sure that publisher was connected to the surviving node. > The consumer - I believe was also connected to the surviving node because > of the fact that lyra would have otherwise recovered the queue and bindings. > e, Sure, will try out 3.3 to see if this issue occurs again. > > Thanks again, > Srinath. > > > On Wed, Apr 9, 2014 at 1:31 PM, Michael Klishin wrote: > >> On 9 April 2014 at 05:36:42, Srinath C (srinath.c at gmail.com) wrote: >> > > Scenario: >> > One of the rabbitmq nodes were brought down for testing. >> > It was observed that no more messages were being consumed >> >> Note that RabbitMQ 3.3.0 will automatically re-register consumers >> on a queue that had master failure. So I suggest giving your scenario >> a try with 3.3. >> -- >> MK >> >> Software Engineer, Pivotal/RabbitMQ >> _______________________________________________ >> rabbitmq-discuss mailing list >> rabbitmq-discuss at lists.rabbitmq.com >> https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From greg.poirier at opower.com Sat Apr 12 18:55:27 2014 From: greg.poirier at opower.com (Greg Poirier) Date: Sat, 12 Apr 2014 10:55:27 -0700 Subject: [rabbitmq-discuss] RabbitMQ Deployment Architectures (for lack of a better subject) In-Reply-To: References: Message-ID: On Sat, Apr 12, 2014 at 8:06 AM, Michael Klishin wrote: > One straightforward way of "routing" would be to use multiple exchanges > that are federated to different "storage" clusters. > > You can name them sla1, sla2 and so on, for example. Publishing definitely seems to be the easy part. This is more or less what I had planned to do. The trick of course comes with the consumers. > > If both consumers and producers connect to different, non-clustered > > "router" systems, how do I configure federation to achieve this? > > Publishers publish to different exchanges in the "router" cluster, > federation > propagates messages to N "storage" clusters where consumers declare the > queues > they need and bind them to federated [destination] exchanges. Then consume > messages as if they were published to their cluster. > So this would require, for example, the consumers to connect directly to the storage clusters in order to instantiate their queues and then federate those queues to exchanges on the router systems? -------------- next part -------------- An HTML attachment was scrubbed... URL: From mklishin at gopivotal.com Sat Apr 12 18:59:52 2014 From: mklishin at gopivotal.com (Michael Klishin) Date: Sat, 12 Apr 2014 21:59:52 +0400 Subject: [rabbitmq-discuss] RabbitMQ Deployment Architectures (for lack of a better subject) In-Reply-To: References: Message-ID: On 12 April 2014 at 21:56:40, Greg Poirier (greg.poirier at opower.com) wrote: > > So this would require, for example, the consumers to connect > directly to the storage clusters Yes. I thought the point was for consumers to use these ?storage? clusters. ?in order to instantiate their > queues and then federate those queues to exchanges on the router > systems? Consumers won?t federate anything. They will set up the routing as if the publishers were publishing to their local ?storage? cluster. This will require them knowing only what exchange gets what messages and nothing else. This is pretty straightforward. The hard part would be replying to publishers that publish to a different cluster. This is something that?s currently not addressed by the Federation plugin at all. But in a fair share of cases communication between publishers and consumers is uni-directional.? -- MK Software Engineer, Pivotal/RabbitMQ From greg.poirier at opower.com Sat Apr 12 19:29:23 2014 From: greg.poirier at opower.com (Greg Poirier) Date: Sat, 12 Apr 2014 11:29:23 -0700 Subject: [rabbitmq-discuss] RabbitMQ Deployment Architectures (for lack of a better subject) In-Reply-To: References: Message-ID: On Sat, Apr 12, 2014 at 10:59 AM, Michael Klishin wrote: > > Yes. I thought the point was for consumers to use these "storage" clusters. It is, really. I think I have another idea though. What I _really_ need is simply something that sits between AMQP clients and RabbitMQ that routes to individual clusters based on vhost. -------------- next part -------------- An HTML attachment was scrubbed... URL: From mklishin at gopivotal.com Sat Apr 12 19:34:41 2014 From: mklishin at gopivotal.com (Michael Klishin) Date: Sat, 12 Apr 2014 22:34:41 +0400 Subject: [rabbitmq-discuss] RabbitMQ Deployment Architectures (for lack of a better subject) In-Reply-To: References: Message-ID: On 12 April 2014 at 22:30:36, Greg Poirier (greg.poirier at opower.com) wrote: > > It is, really. I think I have another idea though. What I _really_ > need is simply something that sits between AMQP clients and RabbitMQ > that routes to individual clusters based on vhost. A small service that clients ask where to connect before doing so sounds reasonable.? -- MK Software Engineer, Pivotal/RabbitMQ From greg.poirier at opower.com Sat Apr 12 20:11:32 2014 From: greg.poirier at opower.com (Greg Poirier) Date: Sat, 12 Apr 2014 12:11:32 -0700 Subject: [rabbitmq-discuss] RabbitMQ Deployment Architectures (for lack of a better subject) In-Reply-To: References: Message-ID: On Sat, Apr 12, 2014 at 11:34 AM, Michael Klishin wrote: > A small service that clients ask where to connect before doing so > sounds reasonable. > Yup. I've got some ideas on how to do this with configuration management once we have made it so that all services are deployed by Chef. It's this interim state that we're in that is making things atrociously difficult. -------------- next part -------------- An HTML attachment was scrubbed... URL: From james at bellenger.org Sat Apr 12 21:32:57 2014 From: james at bellenger.org (James Bellenger) Date: Sat, 12 Apr 2014 20:32:57 +0000 Subject: [rabbitmq-discuss] java client reconnect error: attempt to reuse consumer tag In-Reply-To: References: <0000014552f1370c-4c840fae-ccf3-4db4-9726-b84650b22206-000000@email.amazonses.com> Message-ID: <0000014557a49e52-18d53c41-5f6a-4bd1-b035-df84ed467e2f-000000@email.amazonses.com> Thanks for the pointer, Michael. In looking at the client code again, AutorecoveringConnection.recoverConsumers modifies the consumer map while iterating over it. This can change the map ordering and lead to recovering the same consumer twice. It looks like a similar thing happens in recoverQueues. I've attached a small patch that fixes the issue. Thanks! James On 2014-04-12 03:36, Michael Klishin wrote: > ?On 12 April 2014 at 02:39:53, James Bellenger (james at bellenger.org) > wrote: >> > On my machine, recovery appears to work at fewer than about 5 >> consumers. Anything over that, and the client gets a stream of >> TopologyRecoveryExceptions and is never able to reconnect. > > This is annoying but reconnection happens before topology recovery, > so > it is able to reconnect. > >> I've looked at the client source and haven't seen any obvious >> culprit. > > I cannot immediately say what may be going here but there is one > default = 5 in the client. > > ConsumerWorkService uses a 5 thread executor by default (instantiated > with?Executors.newFixedThreadPool). Try setting a custom executor > that > has, say, 64 threads and configure ConnectionFactory with it, and let > us know > how it goes. > -- > MK > > Software Engineer, Pivotal/RabbitMQ -------------- next part -------------- A non-text attachment was scrubbed... Name: recovery.patch Type: text/x-java Size: 1411 bytes Desc: not available URL: From mklishin at gopivotal.com Sat Apr 12 21:44:58 2014 From: mklishin at gopivotal.com (Michael Klishin) Date: Sun, 13 Apr 2014 00:44:58 +0400 Subject: [rabbitmq-discuss] java client reconnect error: attempt to reuse consumer tag In-Reply-To: <0000014557a49e52-18d53c41-5f6a-4bd1-b035-df84ed467e2f-000000@email.amazonses.com> References: <0000014552f1370c-4c840fae-ccf3-4db4-9726-b84650b22206-000000@email.amazonses.com> <0000014557a49e52-18d53c41-5f6a-4bd1-b035-df84ed467e2f-000000@email.amazonses.com> Message-ID: ?On 13 April 2014 at 00:32:58, James Bellenger (james at bellenger.org) wrote: > > Thanks for the pointer, Michael. > In looking at the client code again, > AutorecoveringConnection.recoverConsumers modifies the > consumer map > while iterating over it. This can change the map ordering and > lead to > recovering the same consumer twice. It looks like a similar thing > happens in recoverQueues. I've attached a small patch that fixes > the > issue. Doh. It?s a real blunder because I already fixed this exact issue twice in 2 Ruby clients ;) Thank you, a fix should be in nightlies early next week and in 3.3.1 in a few weeks. -- MK Software Engineer, Pivotal/RabbitMQ From mklishin at gopivotal.com Sat Apr 12 23:58:12 2014 From: mklishin at gopivotal.com (Michael Klishin) Date: Sun, 13 Apr 2014 02:58:12 +0400 Subject: [rabbitmq-discuss] java client reconnect error: attempt to reuse consumer tag In-Reply-To: References: <0000014552f1370c-4c840fae-ccf3-4db4-9726-b84650b22206-000000@email.amazonses.com> <0000014557a49e52-18d53c41-5f6a-4bd1-b035-df84ed467e2f-000000@email.amazonses.com> Message-ID: On 13 April 2014 at 00:45:26, Michael Klishin (mklishin at gopivotal.com) wrote: > > Thank you, a fix should be in nightlies early next week and in > 3.3.1 in a few weeks. https://github.com/rabbitmq/rabbitmq-java-client/commit/ef393cbdc16f24fa92c24cf0bc9b870957bfa2a1 -- MK Software Engineer, Pivotal/RabbitMQ From Anita.Yip at RealPage.com Sun Apr 13 03:28:36 2014 From: Anita.Yip at RealPage.com (Anita Yip) Date: Sun, 13 Apr 2014 02:28:36 +0000 Subject: [rabbitmq-discuss] frame_too_large error while connecting to In-Reply-To: References: Message-ID: Hi Michael, Thanks for the suggestion. Is amqp.node the node.js library that rabbitmq is officially recommending? Though this module seems to be more active, lack of clear documentation and other github stats are less compelling to make the move. Is this module good enough to use in a production environment with heavy message loads? On 4/11/14 1:56 PM, "Michael Klishin" wrote: >On 11 April 2014 at 19:31:51, Anita Yip (anita.yip at realpage.com) wrote: >> > I am trying to connect to a cluster of 2 RabbitMQ servers, and >> using >> 'node-amqp' module. Recently, the servers are open to public >> access for a >> demo purpose. Before that I never saw this issue. And even now, >> if I >> connect to the servers using their hostnames via vpn, I do not >> see this >> issue. But if I use the ip addresses, whether or not I am on vpn, >> the >> issue occurs. > >This suggests an issue in the client . >node-amqp is not particularly actively maintained, I recommend amqp.node >[1]. > >1. https://github.com/squaremo/amqp.node/ >-- >MK > >Software Engineer, Pivotal/RabbitMQ >_______________________________________________ >rabbitmq-discuss mailing list >rabbitmq-discuss at lists.rabbitmq.com >https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss ________________________________ This message is intended only for the use of the individual(s) or entity to which it is addressed and may contain information that is privileged, confidential, and/or proprietary to RealPage and its affiliated companies. If the reader of this message is not the intended recipient, you are hereby notified that any dissemination, distribution, forwarding or copying of this communication is prohibited without the express permission of the sender. If you have received this communication in error, please notify the sender immediately and delete the original message. From mklishin at gopivotal.com Sun Apr 13 06:42:32 2014 From: mklishin at gopivotal.com (Michael Klishin) Date: Sun, 13 Apr 2014 09:42:32 +0400 Subject: [rabbitmq-discuss] frame_too_large error while connecting to In-Reply-To: References: Message-ID: On 13 April 2014 at 06:29:48, Anita Yip (anita.yip at realpage.com) wrote: > > Is amqp.node the node.js library > that rabbitmq is officially recommending? It is the client that will be used when we add a Node.js port of our tutorials. It is also maintained by a former RabbitMQ team member. > Though this module > seems to be > more active, lack of clear documentation and other github stats > are less > compelling to make the move.? GitHub stats won?t tell you that node-amqp maintainer is no longer interested in maintaining it. There?s tutorial ports: https://github.com/squaremo/amqp.node/tree/master/examples/tutorials and API reference: http://squaremo.github.io/amqp.node/doc/channel_api.html > Is this module good enough to use > in a > production environment with heavy message loads? It is.? -- MK Software Engineer, Pivotal/RabbitMQ From Anita.Yip at RealPage.com Mon Apr 14 01:48:47 2014 From: Anita.Yip at RealPage.com (Anita Yip) Date: Mon, 14 Apr 2014 00:48:47 +0000 Subject: [rabbitmq-discuss] frame_too_large error while connecting to In-Reply-To: References: Message-ID: Ok, thank you! On 4/12/14 10:42 PM, "Michael Klishin" wrote: >On 13 April 2014 at 06:29:48, Anita Yip (anita.yip at realpage.com) wrote: >> > Is amqp.node the node.js library >> that rabbitmq is officially recommending? > >It is the client that will be used when we add a Node.js port of >our tutorials. It is also maintained by a former RabbitMQ >team member. > >> Though this module >> seems to be >> more active, lack of clear documentation and other github stats >> are less >> compelling to make the move. > >GitHub stats won?t tell you that node-amqp maintainer is no longer >interested in maintaining it. > >There?s tutorial ports: > >https://github.com/squaremo/amqp.node/tree/master/examples/tutorials > >and API reference: > >http://squaremo.github.io/amqp.node/doc/channel_api.html > >> Is this module good enough to use >> in a >> production environment with heavy message loads? > >It is. >-- >MK > >Software Engineer, Pivotal/RabbitMQ >_______________________________________________ >rabbitmq-discuss mailing list >rabbitmq-discuss at lists.rabbitmq.com >https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss ________________________________ This message is intended only for the use of the individual(s) or entity to which it is addressed and may contain information that is privileged, confidential, and/or proprietary to RealPage and its affiliated companies. If the reader of this message is not the intended recipient, you are hereby notified that any dissemination, distribution, forwarding or copying of this communication is prohibited without the express permission of the sender. If you have received this communication in error, please notify the sender immediately and delete the original message. From alan.antonuk at gmail.com Mon Apr 14 06:44:57 2014 From: alan.antonuk at gmail.com (Alan Antonuk) Date: Mon, 14 Apr 2014 05:44:57 +0000 Subject: [rabbitmq-discuss] ANN: SimpleAmqpClient v2.4.0 Released Message-ID: SimpleAmqpClient is a developer-friendly C++ wrapper around rabbitmq-c, focused around a light-weight RPC use-case New features for SimpleAmqpClient v2.4: - New API: BasicReject - New API: DeclareQueueWithCounts - Support for consumer cancellation notification - Enhanced BasicConsumeMessage to wait on multiple consumer tags. - Enhancements in frame handling code improves worst-case memory usage under a variety of conditions. - and more... The complete ChangeLog can be found: *https://github.com/alanxz/SimpleAmqpClient/blob/v2.4.0/ChangeLog.md * -------------- next part -------------- An HTML attachment was scrubbed... URL: From james at bellenger.org Fri Apr 11 23:29:13 2014 From: james at bellenger.org (James Bellenger) Date: Fri, 11 Apr 2014 15:29:13 -0700 (PDT) Subject: [rabbitmq-discuss] java client reconnect error: attempt to reuse consumer tag Message-ID: <784624f3-9bcb-4336-8042-8329f7ed2814@googlegroups.com> Hello. I've been testing the client autorecovery support in 3.3.0 and have had some issues getting it to work. On my machine, recovery appears to work at fewer than about 5 consumers. Anything over that, and the client gets a stream of TopologyRecoveryExceptions and is never able to reconnect. I've looked at the client source and haven't seen any obvious culprit. Has anyone else seen this? *Test Setup:* client jar: amqp-client-3.3.0.jar server: rabbitmq-server 3.1.3-1 (from ubuntu 13.10) Triggering autorecovery via 'service rabbitmq-server restart' *Client code (scala):* package rabbittest import com.rabbitmq.client._ object RecoveryTest { // the consumer count matters. At ~5 consumers the client gets put into a bad state and never reconnects // Fewer than 5, the client is usually able to reconnect val consumers = 10 def main(args: Array[String]) { val factory = new ConnectionFactory() factory.setUri("amqp://guest at localhost:5672/test") factory.setAutomaticRecoveryEnabled(true) val conn = factory.newConnection() for (x <- 1 to consumers) { val chan = conn.createChannel() val result = chan.queueDeclare() val tag = chan.basicConsume(result.getQueue, new DefaultConsumer(chan)) println(s"channel=${chan.getChannelNumber} queue=${result.getQueue} consumer=$tag") } } } *Good Logs* channel=1 queue=amq.gen-KvCBOoSjkyhpqle8GMGB0Q consumer=amq.ctag-4WBZRhTLhp_mnQDCXs1NGg channel=2 queue=amq.gen-bNUrlmqi2nHZiunZ9qy-UA consumer=amq.ctag-2_GR91qniV39a-qTtK9Beg channel=3 queue=amq.gen-li-v7cR25J7kmQ-YfcqbCQ consumer=amq.ctag-llfj8Zf1yFCnIGxpU0EI8A *Bad Logs (topology exceptions repeat on every reconnect attempt)* channel=1 queue=amq.gen-TIierQJcDNd93QzX8gYuww consumer=amq.ctag-grxPT0_DsPA7WccDmbNBrg channel=2 queue=amq.gen-JaXYQy218NUlWGQCubjjhg consumer=amq.ctag-Toh8E8Ogvmb3UNkKcJszxg channel=3 queue=amq.gen-VzRS5YVgjP7Lz3UfhxYBgA consumer=amq.ctag-LOiutBNDPk8Dl4gUWxJgdg channel=4 queue=amq.gen-JNJAT8cGvVPRePX43hSPtg consumer=amq.ctag-0nyXfLeLl9lsApz3XsT0Iw channel=5 queue=amq.gen-Biake-GSvzyuCFe1b-4FVA consumer=amq.ctag-xtZbH08UoJnJKhi52vn_1g channel=6 queue=amq.gen-apsJKNLSgN3Dv0VBqPLqSQ consumer=amq.ctag-OwrpVcvqzqB3aQ42PdQY2g channel=7 queue=amq.gen-KxXs5Hkd1W2I1WPxIydpMQ consumer=amq.ctag-KPZzHfmW2f7Cni5u4gQCvA channel=8 queue=amq.gen-ZeLhlueGGsAXW_GhuOVS5w consumer=amq.ctag-54Raw5XNJp2W1-uQUcP2aw channel=9 queue=amq.gen-UZW1yR4QG2Sz90RM9CulQA consumer=amq.ctag-kDDpMuN3vaP-MLeH4Fk7oA channel=10 queue=amq.gen-Ejgc9jZrg6q1UH3qN2VXuA consumer=amq.ctag-3OxICESMijMUMgZOURXNmw Caught an exception when recovering topology Caught an exception while recovering consumer amq.ctag-LOiutBNDPk8Dl4gUWxJgdg com.rabbitmq.client.TopologyRecoveryException: Caught an exception while recovering consumer amq.ctag-LOiutBNDPk8Dl4gUWxJgdg at com.rabbitmq.client.impl.recovery.AutorecoveringConnection.recoverConsumers(AutorecoveringConnection.java:488) at com.rabbitmq.client.impl.recovery.AutorecoveringConnection.beginAutomaticRecovery(AutorecoveringConnection.java:365) at com.rabbitmq.client.impl.recovery.AutorecoveringConnection.access$000(AutorecoveringConnection.java:47) at com.rabbitmq.client.impl.recovery.AutorecoveringConnection$1.shutdownCompleted(AutorecoveringConnection.java:344) at com.rabbitmq.client.impl.ShutdownNotifierComponent.notifyListeners(ShutdownNotifierComponent.java:75) at com.rabbitmq.client.impl.AMQConnection$MainLoop.run(AMQConnection.java:572) at java.lang.Thread.run(Thread.java:744) Caused by: java.io.IOException at com.rabbitmq.client.impl.AMQChannel.wrap(AMQChannel.java:106) at com.rabbitmq.client.impl.AMQChannel.wrap(AMQChannel.java:102) at com.rabbitmq.client.impl.ChannelN.basicConsume(ChannelN.java:995) at com.rabbitmq.client.impl.recovery.AutorecoveringChannel.basicConsume(AutorecoveringChannel.java:312) at com.rabbitmq.client.impl.recovery.RecordedConsumer.recover(RecordedConsumer.java:45) at com.rabbitmq.client.impl.recovery.AutorecoveringConnection.recoverConsumers(AutorecoveringConnection.java:481) ... 6 more Caused by: com.rabbitmq.client.ShutdownSignalException: connection error; reason: #method(reply-code=530, reply-text=NOT_ALLOWED - attempt to reuse consumer tag 'amq.ctag-LOiutBNDPk8Dl4gUWxJgdg', class-id=60, method-id=20) at com.rabbitmq.utility.ValueOrException.getValue(ValueOrException.java:67) at com.rabbitmq.utility.BlockingValueOrException.uninterruptibleGetValue(BlockingValueOrException.java:33) at com.rabbitmq.client.impl.AMQChannel$BlockingRpcContinuation.getReply(AMQChannel.java:343) at com.rabbitmq.client.impl.ChannelN.basicConsume(ChannelN.java:993) ... 9 more Caught an exception when recovering topology Caught an exception while recovering consumer amq.ctag-54Raw5XNJp2W1-uQUcP2aw com.rabbitmq.client.TopologyRecoveryException: Caught an exception while recovering consumer amq.ctag-54Raw5XNJp2W1-uQUcP2aw at com.rabbitmq.client.impl.recovery.AutorecoveringConnection.recoverConsumers(AutorecoveringConnection.java:488) at com.rabbitmq.client.impl.recovery.AutorecoveringConnection.beginAutomaticRecovery(AutorecoveringConnection.java:365) at com.rabbitmq.client.impl.recovery.AutorecoveringConnection.access$000(AutorecoveringConnection.java:47) at com.rabbitmq.client.impl.recovery.AutorecoveringConnection$1.shutdownCompleted(AutorecoveringConnection.java:344) at com.rabbitmq.client.impl.ShutdownNotifierComponent.notifyListeners(ShutdownNotifierComponent.java:75) at com.rabbitmq.client.impl.AMQConnection$MainLoop.run(AMQConnection.java:572) at java.lang.Thread.run(Thread.java:744) Caused by: com.rabbitmq.client.AlreadyClosedException: connectionconnection error; reason: #method(reply-code=530, reply-text=NOT_ALLOWED - attempt to reuse consumer tag 'amq.ctag-LOiutBNDPk8Dl4gUWxJgdg', class-id=60, method-id=20) at com.rabbitmq.client.impl.AMQChannel.ensureIsOpen(AMQChannel.java:190) at com.rabbitmq.client.impl.AMQChannel.rpc(AMQChannel.java:223) at com.rabbitmq.client.impl.ChannelN.basicConsume(ChannelN.java:981) at com.rabbitmq.client.impl.recovery.AutorecoveringChannel.basicConsume(AutorecoveringChannel.java:312) at com.rabbitmq.client.impl.recovery.RecordedConsumer.recover(RecordedConsumer.java:45) at com.rabbitmq.client.impl.recovery.AutorecoveringConnection.recoverConsumers(AutorecoveringConnection.java:481) ... 6 more -------------- next part -------------- An HTML attachment was scrubbed... URL: From maxime.montinet at gmail.com Sat Apr 12 04:13:47 2014 From: maxime.montinet at gmail.com (Maxime Montinet) Date: Fri, 11 Apr 2014 20:13:47 -0700 (PDT) Subject: [rabbitmq-discuss] Can't send payloads above 80kB to a RabbitMQ server ? Message-ID: Hi, I'm trying to send base64 encoded chunks of data through a RabbitMQ server, from a Python client. It goes perfectly fine for payloads in the 1 to ~80kB range, but anything above that and I get a connection reset, with an error saying Resource temporarily unavailable. There are absolutely no errors in the server log when this happens. I have tried this with Python clients, using both pika and rabbitpy, and also using different versions of rabbitmq-server (2.8.4 on a Debian host, and 3.2.4 on an OSX 10.9 host), the error and its circumstances are the same. There was no custom configuration whatsoever done on the test servers. Here's a link to the script I used for my tests, host/usernames blanked out : http://pastebin.com/xWEFeM2K . As you can see, fairly simple test setup, and we can see the error appearing at approx. 80kB. All the data that's send to the broker is sane (Just a bunch of base64-encoded random junk). I have also tried to simply reconnect to the broker when this happens, and trying again, but as expected my publication is always rejected. As I understood it, there wasn't really any limit on the size of the messages a rabbitmq broker was capable of handling, am I in the wrong here ? Does anyone know what I can do to sort this out ? I can provide information as needed, just ask :) Regards -------------- next part -------------- An HTML attachment was scrubbed... URL: From mklishin at gopivotal.com Mon Apr 14 09:05:29 2014 From: mklishin at gopivotal.com (Michael Klishin) Date: Mon, 14 Apr 2014 12:05:29 +0400 Subject: [rabbitmq-discuss] java client reconnect error: attempt to reuse consumer tag In-Reply-To: <784624f3-9bcb-4336-8042-8329f7ed2814@googlegroups.com> References: <784624f3-9bcb-4336-8042-8329f7ed2814@googlegroups.com> Message-ID: ?On 14 April 2014 at 12:01:52, James Bellenger (james at bellenger.org) wrote: > > I've been testing the client autorecovery support in 3.3.0 > and have had some issues getting it to work. > On my machine, recovery appears to work at fewer than about 5 consumers. > Anything over that, and the client gets a stream of TopologyRecoveryExceptions > and is never able to reconnect. > > I've looked at the client source and haven't seen any obvious > culprit. Has anyone else seen this? This message was posted via Google Groups and took some time to get synced. See ?http://markmail.org/thread/6vxalpggsh45k4bn for a resolution. -- MK Software Engineer, Pivotal/RabbitMQ From mklishin at gopivotal.com Mon Apr 14 09:07:06 2014 From: mklishin at gopivotal.com (Michael Klishin) Date: Mon, 14 Apr 2014 12:07:06 +0400 Subject: [rabbitmq-discuss] Can't send payloads above 80kB to a RabbitMQ server ? In-Reply-To: References: Message-ID: On 14 April 2014 at 12:03:43, Maxime Montinet (maxime.montinet at gmail.com) wrote: > > As I understood it, there wasn't really any limit on the size > of the messages a rabbitmq broker was capable of handling, am > I in the wrong here ? Does anyone know what I can do to sort this out > ? Max message size is 2 GB as of 3.2, I believe. There was no limit earlier. > I can provide information as needed, just ask :) What?s in RabbitMQ logs? Are there any firewalls between client and server?? -- MK Software Engineer, Pivotal/RabbitMQ From mklishin at gopivotal.com Mon Apr 14 09:17:26 2014 From: mklishin at gopivotal.com (Michael Klishin) Date: Mon, 14 Apr 2014 12:17:26 +0400 Subject: [rabbitmq-discuss] Can't send payloads above 80kB to a RabbitMQ server ? In-Reply-To: References: Message-ID: On 14 April 2014 at 12:03:43, Maxime Montinet (maxime.montinet at gmail.com) wrote: > > It goes perfectly fine for payloads in the 1 to ~80kB range, but > anything above that and I get a connection reset, with an error > saying Resource temporarily unavailable. > > There are absolutely no errors in the server log when this happens. > Missed this part. This may mean an intermediary (e.g. a firewall) is closing TCP connection but then there would be abrupt connection closure messages in RabbitMQ log. I will try to reproduce it.? -- MK Software Engineer, Pivotal/RabbitMQ From mklishin at gopivotal.com Mon Apr 14 09:26:52 2014 From: mklishin at gopivotal.com (Michael Klishin) Date: Mon, 14 Apr 2014 12:26:52 +0400 Subject: [rabbitmq-discuss] Can't send payloads above 80kB to a RabbitMQ server ? In-Reply-To: References: Message-ID: On 14 April 2014 at 12:19:12, Michael Klishin (mklishin at gopivotal.com) wrote: > > I will try to reproduce it. I get $ python publisher.py Opening connection Opening channel Declaring queue Payload size is 13656 bytes Sending payload Payload size is 27312 bytes Sending payload Payload size is 40968 bytes Sending payload Payload size is 54624 bytes Sending payload Payload size is 68280 bytes Sending payload Payload size is 81936 bytes Sending payload Payload size is 95592 bytes Sending payload Payload size is 109248 bytes Sending payload Payload size is 122904 bytes Sending payload Payload size is 136560 bytes Sending payload Payload size is 150216 bytes Sending payload Payload size is 163872 bytes Sending payload Payload size is 177528 bytes Sending payload Payload size is 191184 bytes Sending payload Payload size is 204840 bytes Sending payload Payload size is 218496 bytes Sending payload Payload size is 232152 bytes Sending payload Payload size is 245808 bytes Sending payload Payload size is 259464 bytes Sending payload with RabbitMQ 3.3.0. -- MK Software Engineer, Pivotal/RabbitMQ From matthias at rabbitmq.com Mon Apr 14 10:00:51 2014 From: matthias at rabbitmq.com (Matthias Radestock) Date: Mon, 14 Apr 2014 10:00:51 +0100 Subject: [rabbitmq-discuss] Fwd: Performance bottleneck on inter-node connection In-Reply-To: References: <53465CF5.20607@rabbitmq.com> <5346C06C.8090609@rabbitmq.com> Message-ID: <534BA3C3.3080607@rabbitmq.com> On 11/04/14 17:37, joseph rouphael wrote: > Test 8: Dual node, Queues/Consumers on same node, 20 queue, ACK off > ------------------------------------------------------------------- > Nodes: Double node > Queues created on: Node1 > Producers on: node 2 > Consumers on: node 1 > Producer Processes: 20 > Producer regulate rate per process: 5KHZ > Acknowledgment: OFF > Expected rate: 100KHZ > Achieved rate: Fluctuating between 10KHZ and 20KHZ > CPU idle: 95% > Note: Bottleneck on inter-node connection What happens when you run the above test with both nodes on the same machine? Also, you really shouldn't need 20 producers/consumers to saturate a broker, unless the machine has >20 cores. So I'd be interested to hear what the smallest producer/consumer count is at which you see a significant performance difference between the single node and dual node case. Finally, it would be helpful if you ran your tests using RabbitMQ's standard performance testing tool, PerfTest - http://www.rabbitmq.com/java-tools.html#perftest. In order to replicate the above scenario you'd run 20 PerfTest instances with options -u q -x 0 -a and 20 instances with -u q -y 0 with n in 1..20. Regards, Matthias. From matthias at rabbitmq.com Mon Apr 14 10:20:56 2014 From: matthias at rabbitmq.com (Matthias Radestock) Date: Mon, 14 Apr 2014 10:20:56 +0100 Subject: [rabbitmq-discuss] =?iso-8859-1?q?=A0Installing_broker_on_each_no?= =?iso-8859-1?q?de_instead_of_client-only_approach?= In-Reply-To: <2efb248a-473f-4d71-8caa-a6df339278a8@googlegroups.com> References: <2efb248a-473f-4d71-8caa-a6df339278a8@googlegroups.com> Message-ID: <534BA878.5070301@rabbitmq.com> On 09/04/14 10:46, Nikolay Maximov wrote: > I have a network of nodes which has some events to be saved in local > store and queued for further processing in central back office. Please > advice what is better to manage and costs less: to have local store and > application which store message first and queue it then versus local > rabbitmq broker to do the same thing with persistent queue. I am looking > for best practice in this approach. Does someone has an exprience in > such design where broker installed on each node? A star-shaped RabbitMQ exchange federation (http://www.rabbitmq.com/federated-exchanges.html) - with local nodes serving as resilient store-and-forward buffers to a central broker - would work here and is quite a common configuration for exactly the scenario you are describing. Matthias. From matthias at rabbitmq.com Mon Apr 14 10:58:29 2014 From: matthias at rabbitmq.com (Matthias Radestock) Date: Mon, 14 Apr 2014 10:58:29 +0100 Subject: [rabbitmq-discuss] Fwd: Performance bottleneck on inter-node connection In-Reply-To: <534BA3C3.3080607@rabbitmq.com> References: <53465CF5.20607@rabbitmq.com> <5346C06C.8090609@rabbitmq.com> <534BA3C3.3080607@rabbitmq.com> Message-ID: <534BB145.8000807@rabbitmq.com> On 14/04/14 10:00, Matthias Radestock wrote: > run 20 PerfTest instances with options > -u q -x 0 -a > and 20 instances with > -u q -y 0 > with n in 1..20. Slight correction. Make that -u q -k q -x 0 -a and -u q -k q -y 0 Matthias. From Chris.Nicel at 15below.com Mon Apr 14 11:53:30 2014 From: Chris.Nicel at 15below.com (Chris Nicel) Date: Mon, 14 Apr 2014 05:53:30 -0500 Subject: [rabbitmq-discuss] Issue deleting Queue Message-ID: Hi All, A little back ground. We're using highly available queues in a two node cluster running on Ubuntu 12.04LTS and RAbbitMQ server 3.2.2. The ha policy is set to automatically mirror each queue to the other node for failover. Our application code handles the failover and is aware of both rabbit servers. I am having trouble deleting a queue in a vhost. The queue name contains some special chars which may be causing the management plugin some troubles when displaying the queue information. When I click on the queue name in the management plugin I get the following error message: Not found The object you clicked on was not found; it may have been deleted on the server. Queue name is: Applications_Pasngr_Events_Contract_Messages+DataSourceChange:Applications_Pasngr_Events_Contract I suspect it is the + that is the problem. If someone can help me delete it I promise not to make another queue with a + in.... Which was probably a silly idea in the first place... I have tried rabbitmqctl but that doesn't have any delete queues options. I have also tried rabbitmqadmin python script (from http://brokerurl:15672/cli) with the command line (both with and without the quotes): python rabbitmqadmin -V applications.pasngr delete queue name="Applications_Pasngr_Events_Contract_Messages+DataSourceChange: Applications_Pasngr_Events_Contract" The command line returns saying that the queue has been deleted but looking at the management plugin it appears to be only deleted from one node. (management plugin shows a red +1 indicating it's not mirrored). The HA policy appears to immediately recreate the deleted queue from the other node. I have tried deleting the queue from both nodes in the same way but I can't seem to do it synchronously enough to make it actually disappear. Any help would be greatly appreciated. Cheers Chris Nicel Senior Systems Engineer 15below ltd : 15below Australia pty. ltd. t: +44 1273 764230 e: chris.nicel at 15below.com w: 15below.com [cid:image001.png at 01CF57D8.27B5EBE0] @15below_travel 15below Limited: Company registered in England and Wales No 3945289 Registered Office: Lyndean House, 43-46 Queens Road, Brighton BN1 3XB, United Kingdom 15below Australia Pty Limited: ABN 25 132 716 379 Level 21, Tower 2 Darling Park, 201 Sussex Street, Sydney, NSW 2000, Australia Please think about the environment before printing this email. ************************************************************************ This email and any attachments may be confidential and/or legally privileged and are solely for the use of the intended recipient. If you have received this email in error please contact the sender. Any views or opinions expressed within this e-mail are solely those of the sender, and do not necessarily represent those of 15below unless otherwise specifically stated. Although 15below has taken every reasonable precaution to ensure that any attachment to this e-mail has been checked for viruses, it is strongly recommended that you carry out your own virus check before opening any attachment, as we cannot accept liability for any damage sustained as a result of software virus infection. -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image001.png Type: image/png Size: 808 bytes Desc: image001.png URL: From matthias at rabbitmq.com Mon Apr 14 12:19:15 2014 From: matthias at rabbitmq.com (Matthias Radestock) Date: Mon, 14 Apr 2014 12:19:15 +0100 Subject: [rabbitmq-discuss] CRL support & verify_fun In-Reply-To: <880cfca2-4982-4c9c-9090-f7092f4020bb@googlegroups.com> References: <880cfca2-4982-4c9c-9090-f7092f4020bb@googlegroups.com> Message-ID: <534BC433.8070309@rabbitmq.com> On 11/04/14 11:28, Dmitry Malinovsky wrote: > I need help with a custom verify_fun > [...] > Where should I put my Module (e.g. mymod.beam) contains Function (e.g. > check_crl/1) to make sure that RabbitMQ will find and use this Module? Write a custom plug-in. https://www.rabbitmq.com/plugin-development.html > Should I specify Function's arity in the config file? No. Matthias. From simon at rabbitmq.com Mon Apr 14 12:22:12 2014 From: simon at rabbitmq.com (Simon MacMullen) Date: Mon, 14 Apr 2014 12:22:12 +0100 Subject: [rabbitmq-discuss] Issue deleting Queue In-Reply-To: References: Message-ID: <534BC4E4.30207@rabbitmq.com> Hi Chris. I think the "+" is likely to be a red herring; you certainly should be able to use "+" in a queue name and expect things to work. Are there any errors in the logs referencing the name of this queue? (I should expect there probably are.) Cheers, Simon On 14/04/2014 11:53, Chris Nicel wrote: > Hi All, > > A little back ground. We?re using highly available queues in a two node > cluster running on Ubuntu 12.04LTS and RAbbitMQ server 3.2.2. The ha > policy is set to automatically mirror each queue to the other node for > failover. Our application code handles the failover and is aware of both > rabbit servers. > > I am having trouble deleting a queue in a vhost. The queue name contains > some special chars which may be causing the management plugin some > troubles when displaying the queue information. When I click on the > queue name in the management plugin I get the following error message: > > Not found > > The object you clicked on was not found; it may have been deleted on the > server. > > Queue name is: > *Applications_Pasngr_Events_Contract_Messages+DataSourceChange:Applications_Pasngr_Events_Contract* > > > I suspect it is the + that is the problem. If someone can help me delete > it I promise not to make another queue with a + in?. Which was probably > a silly idea in the first place? > > I have tried rabbitmqctl but that doesn?t have any delete queues > options. I have also tried rabbitmqadmin python script (from > http://brokerurl:15672/cli) with the command line (both with and without > the quotes): > > python rabbitmqadmin -V applications.pasngr delete queue > name="Applications_Pasngr_Events_Contract_Messages+DataSourceChange: > Applications_Pasngr_Events_Contract" > > The command line returns saying that the queue has been deleted but > looking at the management plugin it appears to be only deleted from one > node. (management plugin shows a red +1 indicating it?s not mirrored). > The HA policy appears to immediately recreate the deleted queue from the > other node. I have tried deleting the queue from both nodes in the same > way but I can?t seem to do it synchronously enough to make it actually > disappear. > > Any help would be greatly appreciated. > > Cheers > > *Chris Nicel* > > Senior Systems Engineer > > 15below ltd : 15below Australia pty. ltd. > > *t:***+44 1273 764230 > > *e*: chris.nicel at 15below.com > *w:*15below.com > > *@15below_travel > > > 15below Limited: Company registered in England and Wales No 3945289 > Registered Office: Lyndean House, 43-46 Queens Road, Brighton BN1 3XB, > United Kingdom > > 15below Australia Pty Limited: ABN 25 132 716 379 > Level 21, Tower 2 Darling Park, 201 Sussex Street, Sydney, NSW 2000, > Australia > > Please think about the environment before printing this email. > > ************************************************************************ > This email and any attachments may be confidential and/or legally > privileged and are solely for the use of the intended recipient. If you > have received this email in error please contact the sender. Any views > or opinions expressed within this e-mail are solely those of the sender, > and do not necessarily represent those of 15below unless otherwise > specifically stated. Although 15below has taken every reasonable > precaution to ensure that any attachment to this e-mail has been checked > for viruses, it is strongly recommended that you carry out your own > virus check before opening any attachment, as we cannot accept liability > for any damage sustained as a result of software virus infection. > > > _______________________________________________ > rabbitmq-discuss mailing list > rabbitmq-discuss at lists.rabbitmq.com > https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss > From videlalvaro at gmail.com Mon Apr 14 12:52:38 2014 From: videlalvaro at gmail.com (Alvaro Videla) Date: Mon, 14 Apr 2014 13:52:38 +0200 Subject: [rabbitmq-discuss] Issue deleting Queue In-Reply-To: References: Message-ID: Hi, Have you tried encoding the + sign as a "%2B"? Regards, Alvaro On Mon, Apr 14, 2014 at 12:53 PM, Chris Nicel wrote: > Hi All, > > > > A little back ground. We?re using highly available queues in a two node > cluster running on Ubuntu 12.04LTS and RAbbitMQ server 3.2.2. The ha policy > is set to automatically mirror each queue to the other node for failover. > Our application code handles the failover and is aware of both rabbit > servers. > > > > I am having trouble deleting a queue in a vhost. The queue name contains > some special chars which may be causing the management plugin some troubles > when displaying the queue information. When I click on the queue name in > the management plugin I get the following error message: > > > > Not found > > The object you clicked on was not found; it may have been deleted on the > server. > > > > > > Queue name is: > *Applications_Pasngr_Events_Contract_Messages+DataSourceChange:Applications_Pasngr_Events_Contract* > > > > I suspect it is the + that is the problem. If someone can help me delete > it I promise not to make another queue with a + in?. Which was probably a > silly idea in the first place? > > > > I have tried rabbitmqctl but that doesn?t have any delete queues options. > I have also tried rabbitmqadmin python script (from > http://brokerurl:15672/cli) with the command line (both with and without > the quotes): > > > > python rabbitmqadmin -V applications.pasngr delete queue > name="Applications_Pasngr_Events_Contract_Messages+DataSourceChange: > Applications_Pasngr_Events_Contract" > > > > The command line returns saying that the queue has been deleted but > looking at the management plugin it appears to be only deleted from one > node. (management plugin shows a red +1 indicating it?s not mirrored). The > HA policy appears to immediately recreate the deleted queue from the other > node. I have tried deleting the queue from both nodes in the same way but I > can?t seem to do it synchronously enough to make it actually disappear. > > > > Any help would be greatly appreciated. > > > > Cheers > > > > *Chris Nicel* > > Senior Systems Engineer > > 15below ltd : 15below Australia pty. ltd. > > > > *t:* +44 1273 764230 > > *e*: chris.nicel at 15below.com > *w:* 15below.com > > [image: *] @15below_travel > > > > 15below Limited: Company registered in England and Wales No 3945289 > Registered Office: Lyndean House, 43-46 Queens Road, Brighton BN1 3XB, > United Kingdom > > 15below Australia Pty Limited: ABN 25 132 716 379 > Level 21, Tower 2 Darling Park, 201 Sussex Street, Sydney, NSW 2000, > Australia > > Please think about the environment before printing this email. > > ************************************************************************ > This email and any attachments may be confidential and/or legally > privileged and are solely for the use of the intended recipient. If you > have received this email in error please contact the sender. Any views or > opinions expressed within this e-mail are solely those of the sender, and > do not necessarily represent those of 15below unless otherwise specifically > stated. Although 15below has taken every reasonable precaution to ensure > that any attachment to this e-mail has been checked for viruses, it is > strongly recommended that you carry out your own virus check before opening > any attachment, as we cannot accept liability for any damage sustained as a > result of software virus infection. > > _______________________________________________ > rabbitmq-discuss mailing list > rabbitmq-discuss at lists.rabbitmq.com > https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss > > -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image001.png Type: image/png Size: 808 bytes Desc: not available URL: From mklishin at gopivotal.com Mon Apr 14 13:03:56 2014 From: mklishin at gopivotal.com (Michael Klishin) Date: Mon, 14 Apr 2014 16:03:56 +0400 Subject: [rabbitmq-discuss] Blocking Queue - Close connection by client In-Reply-To: References: Message-ID: On 10 April 2014 at 10:16:49, Claire Fautsch (cfautsch at goodgamestudios.com) wrote: > > Unfortunatelly calling the abort function instead of close, > will show exactly the same behaviour, as there is still a flush > at the end (I also tested this, and indeed not a solution). The > only difference is that Exceptions/Stacktraces are ignroed. > The only option I see so far to avoid hanging here, is to set a timeout > to the close method. Your observations are correct. N.B. if you have a connection blocked, opening a new one will only help if the new connection doesn?t publish anything. Which suggests you can separate connections that consume and publish from the start.? -- MK Software Engineer, Pivotal/RabbitMQ From mklishin at gopivotal.com Mon Apr 14 13:06:32 2014 From: mklishin at gopivotal.com (Michael Klishin) Date: Mon, 14 Apr 2014 16:06:32 +0400 Subject: [rabbitmq-discuss] Troubleshooting handleCancel() In-Reply-To: References: Message-ID: On 12 April 2014 at 19:27:51, Srinath C (srinath.c at gmail.com) wrote: > > Just checking if the logs provide any insights into the probable > causes? Unfortunately, there?s nothing in the log that seems relevant to what you?re describing.? -- MK Software Engineer, Pivotal/RabbitMQ From Chris.Nicel at 15below.com Mon Apr 14 13:08:10 2014 From: Chris.Nicel at 15below.com (Chris Nicel) Date: Mon, 14 Apr 2014 07:08:10 -0500 Subject: [rabbitmq-discuss] Issue deleting Queue In-Reply-To: References: Message-ID: Hi Alvaro, The management plugin encodes the + as a %2B for me when I click on the queue name but this returns the ?Not Found? error below. The rabbitmqadmin python script also appears to encode it as %2B as well. I just tried with %2B in the queue name and it was encoded to %252B. Also, it appears that the queue is deleted from one node and then recreated by the ha policy on the node it was deleted from. Cheers Chris From: rabbitmq-discuss [mailto:rabbitmq-discuss-bounces at lists.rabbitmq.com] On Behalf Of Alvaro Videla Sent: 14 April 2014 12:53 To: Discussions about RabbitMQ Subject: Re: [rabbitmq-discuss] Issue deleting Queue Hi, Have you tried encoding the + sign as a "%2B"? Regards, Alvaro On Mon, Apr 14, 2014 at 12:53 PM, Chris Nicel > wrote: Hi All, A little back ground. We?re using highly available queues in a two node cluster running on Ubuntu 12.04LTS and RAbbitMQ server 3.2.2. The ha policy is set to automatically mirror each queue to the other node for failover. Our application code handles the failover and is aware of both rabbit servers. I am having trouble deleting a queue in a vhost. The queue name contains some special chars which may be causing the management plugin some troubles when displaying the queue information. When I click on the queue name in the management plugin I get the following error message: Not found The object you clicked on was not found; it may have been deleted on the server. Queue name is: Applications_Pasngr_Events_Contract_Messages+DataSourceChange:Applications_Pasngr_Events_Contract I suspect it is the + that is the problem. If someone can help me delete it I promise not to make another queue with a + in?. Which was probably a silly idea in the first place? I have tried rabbitmqctl but that doesn?t have any delete queues options. I have also tried rabbitmqadmin python script (from http://brokerurl:15672/cli) with the command line (both with and without the quotes): python rabbitmqadmin -V applications.pasngr delete queue name="Applications_Pasngr_Events_Contract_Messages+DataSourceChange: Applications_Pasngr_Events_Contract" The command line returns saying that the queue has been deleted but looking at the management plugin it appears to be only deleted from one node. (management plugin shows a red +1 indicating it?s not mirrored). The HA policy appears to immediately recreate the deleted queue from the other node. I have tried deleting the queue from both nodes in the same way but I can?t seem to do it synchronously enough to make it actually disappear. Any help would be greatly appreciated. Cheers Chris Nicel Senior Systems Engineer 15below ltd : 15below Australia pty. ltd. t: +44 1273 764230 e: chris.nicel at 15below.com w: 15below.com [cid:image002.png at 01CF57E2.95E80530] @15below_travel 15below Limited: Company registered in England and Wales No 3945289 Registered Office: Lyndean House, 43-46 Queens Road, Brighton BN1 3XB, United Kingdom 15below Australia Pty Limited: ABN 25 132 716 379 Level 21, Tower 2 Darling Park, 201 Sussex Street, Sydney, NSW 2000, Australia Please think about the environment before printing this email. ************************************************************************ This email and any attachments may be confidential and/or legally privileged and are solely for the use of the intended recipient. If you have received this email in error please contact the sender. Any views or opinions expressed within this e-mail are solely those of the sender, and do not necessarily represent those of 15below unless otherwise specifically stated. Although 15below has taken every reasonable precaution to ensure that any attachment to this e-mail has been checked for viruses, it is strongly recommended that you carry out your own virus check before opening any attachment, as we cannot accept liability for any damage sustained as a result of software virus infection. _______________________________________________ rabbitmq-discuss mailing list rabbitmq-discuss at lists.rabbitmq.com https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss 15below Limited: Company registered in England and Wales No 3945289 Registered Office: Lyndean House, 43-46 Queens Road, Brighton BN1 3XB, United Kingdom 15below Australia Pty Limited: ABN 25 132 716 379 Level 21, Tower 2 Darling Park, 201 Sussex Street, Sydney, NSW 2000, Australia Please think about the environment before printing this email. ************************************************************************ This email and any attachments may be confidential and/or legally privileged and are solely for the use of the intended recipient. If you have received this email in error please contact the sender. Any views or opinions expressed within this e-mail are solely those of the sender, and do not necessarily represent those of 15below unless otherwise specifically stated. Although 15below has taken every reasonable precaution to ensure that any attachment to this e-mail has been checked for viruses, it is strongly recommended that you carry out your own virus check before opening any attachment, as we cannot accept liability for any damage sustained as a result of software virus infection. -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image002.png Type: image/png Size: 504 bytes Desc: image002.png URL: From videlalvaro at gmail.com Mon Apr 14 13:10:07 2014 From: videlalvaro at gmail.com (Alvaro Videla) Date: Mon, 14 Apr 2014 14:10:07 +0200 Subject: [rabbitmq-discuss] Issue deleting Queue In-Reply-To: References: Message-ID: I've just tried creating a queue with the name you provided using an AMQP client. Then I went to the management UI and I was able to successfully delete the queue. So as Simon says, the error should be somewhere else. On Mon, Apr 14, 2014 at 1:52 PM, Alvaro Videla wrote: > Hi, > > Have you tried encoding the + sign as a "%2B"? > > Regards, > > Alvaro > > > On Mon, Apr 14, 2014 at 12:53 PM, Chris Nicel wrote: > >> Hi All, >> >> >> >> A little back ground. We?re using highly available queues in a two node >> cluster running on Ubuntu 12.04LTS and RAbbitMQ server 3.2.2. The ha policy >> is set to automatically mirror each queue to the other node for failover. >> Our application code handles the failover and is aware of both rabbit >> servers. >> >> >> >> I am having trouble deleting a queue in a vhost. The queue name contains >> some special chars which may be causing the management plugin some troubles >> when displaying the queue information. When I click on the queue name in >> the management plugin I get the following error message: >> >> >> >> Not found >> >> The object you clicked on was not found; it may have been deleted on the >> server. >> >> >> >> >> >> Queue name is: >> *Applications_Pasngr_Events_Contract_Messages+DataSourceChange:Applications_Pasngr_Events_Contract* >> >> >> >> I suspect it is the + that is the problem. If someone can help me delete >> it I promise not to make another queue with a + in?. Which was probably a >> silly idea in the first place? >> >> >> >> I have tried rabbitmqctl but that doesn?t have any delete queues options. >> I have also tried rabbitmqadmin python script (from >> http://brokerurl:15672/cli) with the command line (both with and without >> the quotes): >> >> >> >> python rabbitmqadmin -V applications.pasngr delete queue >> name="Applications_Pasngr_Events_Contract_Messages+DataSourceChange: >> Applications_Pasngr_Events_Contract" >> >> >> >> The command line returns saying that the queue has been deleted but >> looking at the management plugin it appears to be only deleted from one >> node. (management plugin shows a red +1 indicating it?s not mirrored). The >> HA policy appears to immediately recreate the deleted queue from the other >> node. I have tried deleting the queue from both nodes in the same way but I >> can?t seem to do it synchronously enough to make it actually disappear. >> >> >> >> Any help would be greatly appreciated. >> >> >> >> Cheers >> >> >> >> *Chris Nicel* >> >> Senior Systems Engineer >> >> 15below ltd : 15below Australia pty. ltd. >> >> >> >> *t:* +44 1273 764230 >> >> *e*: chris.nicel at 15below.com >> *w:* 15below.com >> >> [image: *] @15below_travel >> >> >> >> 15below Limited: Company registered in England and Wales No 3945289 >> Registered Office: Lyndean House, 43-46 Queens Road, Brighton BN1 3XB, >> United Kingdom >> >> 15below Australia Pty Limited: ABN 25 132 716 379 >> Level 21, Tower 2 Darling Park, 201 Sussex Street, Sydney, NSW 2000, >> Australia >> >> Please think about the environment before printing this email. >> >> ************************************************************************ >> This email and any attachments may be confidential and/or legally >> privileged and are solely for the use of the intended recipient. If you >> have received this email in error please contact the sender. Any views or >> opinions expressed within this e-mail are solely those of the sender, and >> do not necessarily represent those of 15below unless otherwise specifically >> stated. Although 15below has taken every reasonable precaution to ensure >> that any attachment to this e-mail has been checked for viruses, it is >> strongly recommended that you carry out your own virus check before opening >> any attachment, as we cannot accept liability for any damage sustained as a >> result of software virus infection. >> >> _______________________________________________ >> rabbitmq-discuss mailing list >> rabbitmq-discuss at lists.rabbitmq.com >> https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss >> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image001.png Type: image/png Size: 808 bytes Desc: not available URL: From Chris.Nicel at 15below.com Mon Apr 14 13:11:14 2014 From: Chris.Nicel at 15below.com (Chris Nicel) Date: Mon, 14 Apr 2014 07:11:14 -0500 Subject: [rabbitmq-discuss] Issue deleting Queue In-Reply-To: <534BC4E4.30207@rabbitmq.com> References: <534BC4E4.30207@rabbitmq.com> Message-ID: Hi Simon, Further to my previous email, if you create a queue in the management plugin with a + symbol in, the + seems to get stripped out of the queue name. This is feeling like a bug in the management plugin. Although still curious that I can't delete it from rabbitmqadmin either. And even more curious that rabbitmqadmin claims to have deleted the node, the management plugin shows that the queue is not mirrored briefly while it is recreated on the node it was deleted from. Cheers Chris -----Original Message----- From: Simon MacMullen [mailto:simon at rabbitmq.com] Sent: 14 April 2014 12:22 To: Discussions about RabbitMQ Cc: Chris Nicel Subject: Re: [rabbitmq-discuss] Issue deleting Queue Hi Chris. I think the "+" is likely to be a red herring; you certainly should be able to use "+" in a queue name and expect things to work. Are there any errors in the logs referencing the name of this queue? (I should expect there probably are.) Cheers, Simon On 14/04/2014 11:53, Chris Nicel wrote: > Hi All, > > A little back ground. We're using highly available queues in a two > node cluster running on Ubuntu 12.04LTS and RAbbitMQ server 3.2.2. The > ha policy is set to automatically mirror each queue to the other node > for failover. Our application code handles the failover and is aware > of both rabbit servers. > > I am having trouble deleting a queue in a vhost. The queue name > contains some special chars which may be causing the management plugin > some troubles when displaying the queue information. When I click on > the queue name in the management plugin I get the following error message: > > Not found > > The object you clicked on was not found; it may have been deleted on > the server. > > Queue name is: > *Applications_Pasngr_Events_Contract_Messages+DataSourceChange:Applica > tions_Pasngr_Events_Contract* > ations.pasngr/Zorrillo_Applications_Pasngr_Events_Contract_Messages%2B > DataSourceChange%3AZorrillo_Applications_Pasngr_Events_Contract> > > I suspect it is the + that is the problem. If someone can help me > delete it I promise not to make another queue with a + in.... Which was > probably a silly idea in the first place... > > I have tried rabbitmqctl but that doesn't have any delete queues > options. I have also tried rabbitmqadmin python script (from > http://brokerurl:15672/cli) with the command line (both with and > without the quotes): > > python rabbitmqadmin -V applications.pasngr delete queue > name="Applications_Pasngr_Events_Contract_Messages+DataSourceChange: > Applications_Pasngr_Events_Contract" > > The command line returns saying that the queue has been deleted but > looking at the management plugin it appears to be only deleted from > one node. (management plugin shows a red +1 indicating it's not mirrored). > The HA policy appears to immediately recreate the deleted queue from > the other node. I have tried deleting the queue from both nodes in the > same way but I can't seem to do it synchronously enough to make it > actually disappear. > > Any help would be greatly appreciated. > > Cheers > > *Chris Nicel* > > Senior Systems Engineer > > 15below ltd : 15below Australia pty. ltd. > > *t:***+44 1273 764230 > > *e*: chris.nicel at 15below.com > *w:*15below.com > > *@15below_travel > > > 15below Limited: Company registered in England and Wales No 3945289 > Registered Office: Lyndean House, 43-46 Queens Road, Brighton BN1 3XB, > United Kingdom > > 15below Australia Pty Limited: ABN 25 132 716 379 Level 21, Tower 2 > Darling Park, 201 Sussex Street, Sydney, NSW 2000, Australia > > Please think about the environment before printing this email. > > ********************************************************************** > ** This email and any attachments may be confidential and/or legally > privileged and are solely for the use of the intended recipient. If > you have received this email in error please contact the sender. Any > views or opinions expressed within this e-mail are solely those of the > sender, and do not necessarily represent those of 15below unless > otherwise specifically stated. Although 15below has taken every > reasonable precaution to ensure that any attachment to this e-mail has > been checked for viruses, it is strongly recommended that you carry > out your own virus check before opening any attachment, as we cannot > accept liability for any damage sustained as a result of software virus infection. > > > _______________________________________________ > rabbitmq-discuss mailing list > rabbitmq-discuss at lists.rabbitmq.com > https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss > 15below Limited: Company registered in England and Wales No 3945289 Registered Office: Lyndean House, 43-46 Queens Road, Brighton BN1 3XB, United Kingdom 15below Australia Pty Limited: ABN 25 132 716 379 Level 21, Tower 2 Darling Park, 201 Sussex Street, Sydney, NSW 2000, Australia Please think about the environment before printing this email. ************************************************************************ This email and any attachments may be confidential and/or legally privileged and are solely for the use of the intended recipient. If you have received this email in error please contact the sender. Any views or opinions expressed within this e-mail are solely those of the sender, and do not necessarily represent those of 15below unless otherwise specifically stated. Although 15below has taken every reasonable precaution to ensure that any attachment to this e-mail has been checked for viruses, it is strongly recommended that you carry out your own virus check before opening any attachment, as we cannot accept liability for any damage sustained as a result of software virus infection. From mklishin at gopivotal.com Mon Apr 14 13:11:31 2014 From: mklishin at gopivotal.com (Michael Klishin) Date: Mon, 14 Apr 2014 16:11:31 +0400 Subject: [rabbitmq-discuss] Error during WebSocket handshake: Sent non-empty 'Sec-WebSocket-Protocol' header but no response was received In-Reply-To: References: <1395406371281-34248.post@n5.nabble.com> Message-ID: On 10 April 2014 at 20:03:53, joshua__lim (joshua__lim at hotmail.com) wrote: > > I've tried ws://localhost:15674/stomp and got a 200 error > instead. > > ws://localhost:15674/stomp/websocket appears to be correct. OK, so it?s likely a hole in our implementation: https://tools.ietf.org/html/rfc6455#section-11.3.4? -- MK Software Engineer, Pivotal/RabbitMQ From Chris.Nicel at 15below.com Mon Apr 14 13:15:40 2014 From: Chris.Nicel at 15below.com (Chris Nicel) Date: Mon, 14 Apr 2014 07:15:40 -0500 Subject: [rabbitmq-discuss] Issue deleting Queue In-Reply-To: References: Message-ID: Hi Alvaro, What version of rabbitmq are you using? This is occurring on a 3.2.2 installation. Cheers Chris From: rabbitmq-discuss [mailto:rabbitmq-discuss-bounces at lists.rabbitmq.com] On Behalf Of Alvaro Videla Sent: 14 April 2014 13:10 To: Discussions about RabbitMQ Subject: Re: [rabbitmq-discuss] Issue deleting Queue I've just tried creating a queue with the name you provided using an AMQP client. Then I went to the management UI and I was able to successfully delete the queue. So as Simon says, the error should be somewhere else. On Mon, Apr 14, 2014 at 1:52 PM, Alvaro Videla > wrote: Hi, Have you tried encoding the + sign as a "%2B"? Regards, Alvaro On Mon, Apr 14, 2014 at 12:53 PM, Chris Nicel > wrote: Hi All, A little back ground. We?re using highly available queues in a two node cluster running on Ubuntu 12.04LTS and RAbbitMQ server 3.2.2. The ha policy is set to automatically mirror each queue to the other node for failover. Our application code handles the failover and is aware of both rabbit servers. I am having trouble deleting a queue in a vhost. The queue name contains some special chars which may be causing the management plugin some troubles when displaying the queue information. When I click on the queue name in the management plugin I get the following error message: Not found The object you clicked on was not found; it may have been deleted on the server. Queue name is: Applications_Pasngr_Events_Contract_Messages+DataSourceChange:Applications_Pasngr_Events_Contract I suspect it is the + that is the problem. If someone can help me delete it I promise not to make another queue with a + in?. Which was probably a silly idea in the first place? I have tried rabbitmqctl but that doesn?t have any delete queues options. I have also tried rabbitmqadmin python script (from http://brokerurl:15672/cli) with the command line (both with and without the quotes): python rabbitmqadmin -V applications.pasngr delete queue name="Applications_Pasngr_Events_Contract_Messages+DataSourceChange: Applications_Pasngr_Events_Contract" The command line returns saying that the queue has been deleted but looking at the management plugin it appears to be only deleted from one node. (management plugin shows a red +1 indicating it?s not mirrored). The HA policy appears to immediately recreate the deleted queue from the other node. I have tried deleting the queue from both nodes in the same way but I can?t seem to do it synchronously enough to make it actually disappear. Any help would be greatly appreciated. Cheers Chris Nicel Senior Systems Engineer 15below ltd : 15below Australia pty. ltd. t: +44 1273 764230 e: chris.nicel at 15below.com w: 15below.com [cid:image002.png at 01CF57E3.A2265E90] @15below_travel 15below Limited: Company registered in England and Wales No 3945289 Registered Office: Lyndean House, 43-46 Queens Road, Brighton BN1 3XB, United Kingdom 15below Australia Pty Limited: ABN 25 132 716 379 Level 21, Tower 2 Darling Park, 201 Sussex Street, Sydney, NSW 2000, Australia Please think about the environment before printing this email. ************************************************************************ This email and any attachments may be confidential and/or legally privileged and are solely for the use of the intended recipient. If you have received this email in error please contact the sender. Any views or opinions expressed within this e-mail are solely those of the sender, and do not necessarily represent those of 15below unless otherwise specifically stated. Although 15below has taken every reasonable precaution to ensure that any attachment to this e-mail has been checked for viruses, it is strongly recommended that you carry out your own virus check before opening any attachment, as we cannot accept liability for any damage sustained as a result of software virus infection. _______________________________________________ rabbitmq-discuss mailing list rabbitmq-discuss at lists.rabbitmq.com https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss 15below Limited: Company registered in England and Wales No 3945289 Registered Office: Lyndean House, 43-46 Queens Road, Brighton BN1 3XB, United Kingdom 15below Australia Pty Limited: ABN 25 132 716 379 Level 21, Tower 2 Darling Park, 201 Sussex Street, Sydney, NSW 2000, Australia Please think about the environment before printing this email. ************************************************************************ This email and any attachments may be confidential and/or legally privileged and are solely for the use of the intended recipient. If you have received this email in error please contact the sender. Any views or opinions expressed within this e-mail are solely those of the sender, and do not necessarily represent those of 15below unless otherwise specifically stated. Although 15below has taken every reasonable precaution to ensure that any attachment to this e-mail has been checked for viruses, it is strongly recommended that you carry out your own virus check before opening any attachment, as we cannot accept liability for any damage sustained as a result of software virus infection. -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image002.png Type: image/png Size: 504 bytes Desc: image002.png URL: From videlalvaro at gmail.com Mon Apr 14 13:19:50 2014 From: videlalvaro at gmail.com (Alvaro Videla) Date: Mon, 14 Apr 2014 14:19:50 +0200 Subject: [rabbitmq-discuss] Issue deleting Queue In-Reply-To: References: Message-ID: I've tried 3.3.0, from the standalone generic unix tarball. Also I don't think this is related with the rabbitmq version, since we haven't fixed a bug on this regard. On Mon, Apr 14, 2014 at 2:15 PM, Chris Nicel wrote: > Hi Alvaro, > > > > What version of rabbitmq are you using? This is occurring on a 3.2.2 > installation. > > > > Cheers > > > > Chris > > > > *From:* rabbitmq-discuss [mailto: > rabbitmq-discuss-bounces at lists.rabbitmq.com] *On Behalf Of *Alvaro Videla > *Sent:* 14 April 2014 13:10 > *To:* Discussions about RabbitMQ > > *Subject:* Re: [rabbitmq-discuss] Issue deleting Queue > > > > I've just tried creating a queue with the name you provided using an AMQP > client. > > > > Then I went to the management UI and I was able to successfully delete the > queue. > > > > So as Simon says, the error should be somewhere else. > > > > On Mon, Apr 14, 2014 at 1:52 PM, Alvaro Videla > wrote: > > Hi, > > > > Have you tried encoding the + sign as a "%2B"? > > > > Regards, > > > Alvaro > > > > On Mon, Apr 14, 2014 at 12:53 PM, Chris Nicel > wrote: > > Hi All, > > > > A little back ground. We?re using highly available queues in a two node > cluster running on Ubuntu 12.04LTS and RAbbitMQ server 3.2.2. The ha policy > is set to automatically mirror each queue to the other node for failover. > Our application code handles the failover and is aware of both rabbit > servers. > > > > I am having trouble deleting a queue in a vhost. The queue name contains > some special chars which may be causing the management plugin some troubles > when displaying the queue information. When I click on the queue name in > the management plugin I get the following error message: > > > > Not found > > The object you clicked on was not found; it may have been deleted on the > server. > > > > > > Queue name is: > *Applications_Pasngr_Events_Contract_Messages+DataSourceChange:Applications_Pasngr_Events_Contract* > > > > I suspect it is the + that is the problem. If someone can help me delete > it I promise not to make another queue with a + in?. Which was probably a > silly idea in the first place? > > > > I have tried rabbitmqctl but that doesn?t have any delete queues options. > I have also tried rabbitmqadmin python script (from > http://brokerurl:15672/cli) with the command line (both with and without > the quotes): > > > > python rabbitmqadmin -V applications.pasngr delete queue > name="Applications_Pasngr_Events_Contract_Messages+DataSourceChange: > Applications_Pasngr_Events_Contract" > > > > The command line returns saying that the queue has been deleted but > looking at the management plugin it appears to be only deleted from one > node. (management plugin shows a red +1 indicating it?s not mirrored). The > HA policy appears to immediately recreate the deleted queue from the other > node. I have tried deleting the queue from both nodes in the same way but I > can?t seem to do it synchronously enough to make it actually disappear. > > > > Any help would be greatly appreciated. > > > > Cheers > > > > *Chris Nicel* > > Senior Systems Engineer > > 15below ltd : 15below Australia pty. ltd. > > > > *t: *+44 1273 764230 > > *e*: chris.nicel at 15below.com > *w:* 15below.com > > [image: *] @15below_travel > > > > > 15below Limited: Company registered in England and Wales No 3945289 > Registered Office: Lyndean House, 43-46 Queens Road, Brighton BN1 3XB, > United Kingdom > > 15below Australia Pty Limited: ABN 25 132 716 379 > Level 21, Tower 2 Darling Park, 201 Sussex Street, Sydney, NSW 2000, > Australia > > Please think about the environment before printing this email. > > ************************************************************************ > This email and any attachments may be confidential and/or legally > privileged and are solely for the use of the intended recipient. If you > have received this email in error please contact the sender. Any views or > opinions expressed within this e-mail are solely those of the sender, and > do not necessarily represent those of 15below unless otherwise specifically > stated. Although 15below has taken every reasonable precaution to ensure > that any attachment to this e-mail has been checked for viruses, it is > strongly recommended that you carry out your own virus check before opening > any attachment, as we cannot accept liability for any damage sustained as a > result of software virus infection. > > > > _______________________________________________ > rabbitmq-discuss mailing list > rabbitmq-discuss at lists.rabbitmq.com > https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss > > > > > > 15below Limited: Company registered in England and Wales No 3945289 > Registered Office: Lyndean House, 43-46 Queens Road, Brighton BN1 3XB, > United Kingdom > > 15below Australia Pty Limited: ABN 25 132 716 379 > Level 21, Tower 2 Darling Park, 201 Sussex Street, Sydney, NSW 2000, > Australia > > Please think about the environment before printing this email. > > ************************************************************************ > This email and any attachments may be confidential and/or legally > privileged and are solely for the use of the intended recipient. If you > have received this email in error please contact the sender. Any views or > opinions expressed within this e-mail are solely those of the sender, and > do not necessarily represent those of 15below unless otherwise specifically > stated. Although 15below has taken every reasonable precaution to ensure > that any attachment to this e-mail has been checked for viruses, it is > strongly recommended that you carry out your own virus check before opening > any attachment, as we cannot accept liability for any damage sustained as a > result of software virus infection. > > _______________________________________________ > rabbitmq-discuss mailing list > rabbitmq-discuss at lists.rabbitmq.com > https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss > > -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image002.png Type: image/png Size: 504 bytes Desc: not available URL: From matthias at rabbitmq.com Mon Apr 14 13:22:21 2014 From: matthias at rabbitmq.com (Matthias Radestock) Date: Mon, 14 Apr 2014 13:22:21 +0100 Subject: [rabbitmq-discuss] Issue deleting Queue In-Reply-To: References: <534BC4E4.30207@rabbitmq.com> Message-ID: <534BD2FD.6000405@rabbitmq.com> On 14/04/14 13:11, Chris Nicel wrote: > Further to my previous email, if you create a queue in the management > plugin with a + symbol in, the + seems to get stripped out of the > queue name. No, it's not. Have just tried that on http://dev.rabbitmq.com/mgmt/, which is running 3.2.3. As Simon said, the problem has likely nothing to do with the queue name, so please check the server logs for any errors. Matthias. From mklishin at gopivotal.com Mon Apr 14 13:44:46 2014 From: mklishin at gopivotal.com (Michael Klishin) Date: Mon, 14 Apr 2014 16:44:46 +0400 Subject: [rabbitmq-discuss] Error during WebSocket handshake: Sent non-empty 'Sec-WebSocket-Protocol' header but no response was received In-Reply-To: References: <1395406371281-34248.post@n5.nabble.com> Message-ID: On 14 April 2014 at 16:14:22, Michael Klishin (mklishin at gopivotal.com) wrote: > > OK, so it?s likely a hole in our implementation: > > https://tools.ietf.org/html/rfc6455#section-11.3.4 This should be fixed in 3.3.0. Can you try that version and let us know how it goes?? -- MK Software Engineer, Pivotal/RabbitMQ From Chris.Nicel at 15below.com Mon Apr 14 14:25:49 2014 From: Chris.Nicel at 15below.com (Chris Nicel) Date: Mon, 14 Apr 2014 08:25:49 -0500 Subject: [rabbitmq-discuss] Issue deleting Queue Message-ID: Hi Matthias, I will check the logs again for errors. Ok, I think I have worked out what is occurring here.... someone has some code running somewhere that is recreating this queue each time I delete it. Many apologies for not spotting this sooner. We are still seeing the below when we try to view a queue with a + in the management plugin. Not found The object you clicked on was not found; it may have been deleted on the server. I guess this is related to us being on 3.2.2 as it doesn't appear to be an issue on 3.2.3. I have just tried on http://dev.rabbitmq.com/mgmt/ which allowed me to create the queue and then view it in the management plugin. I also noticed the version of erlang you're using in http://dev.rabbitmq.com/mgmt/ is only 14A. We are using 16B03. But I guess this is probably not related? Cheers Chris -----Original Message----- From: Matthias Radestock [mailto:matthias at rabbitmq.com] Sent: 14 April 2014 13:22 To: Discussions about RabbitMQ Cc: Chris Nicel Subject: Re: [rabbitmq-discuss] Issue deleting Queue On 14/04/14 13:11, Chris Nicel wrote: > Further to my previous email, if you create a queue in the management > plugin with a + symbol in, the + seems to get stripped out of the > queue name. No, it's not. Have just tried that on http://dev.rabbitmq.com/mgmt/, which is running 3.2.3. As Simon said, the problem has likely nothing to do with the queue name, so please check the server logs for any errors. Matthias. 15below Limited: Company registered in England and Wales No 3945289 Registered Office: Lyndean House, 43-46 Queens Road, Brighton BN1 3XB, United Kingdom 15below Australia Pty Limited: ABN 25 132 716 379 Level 21, Tower 2 Darling Park, 201 Sussex Street, Sydney, NSW 2000, Australia Please think about the environment before printing this email. ************************************************************************ This email and any attachments may be confidential and/or legally privileged and are solely for the use of the intended recipient. If you have received this email in error please contact the sender. Any views or opinions expressed within this e-mail are solely those of the sender, and do not necessarily represent those of 15below unless otherwise specifically stated. Although 15below has taken every reasonable precaution to ensure that any attachment to this e-mail has been checked for viruses, it is strongly recommended that you carry out your own virus check before opening any attachment, as we cannot accept liability for any damage sustained as a result of software virus infection. From js.future.software at gmail.com Mon Apr 14 14:04:30 2014 From: js.future.software at gmail.com (js.future.software at gmail.com) Date: Mon, 14 Apr 2014 06:04:30 -0700 (PDT) Subject: [rabbitmq-discuss] Recover cluster from crash Message-ID: <665d6495-ca13-465d-b741-142f5835a6f0@googlegroups.com> Hello, we are testing RabbitMQ for future replace MSMQ. Someday ago we had a crash with one node. Cluster is build with 3 nodes HA on Windows machines - RabbitMQ instances are services. This node has been reinstalled. After this cluster failed to start. Now on other two machines we get information that can not connect to other nodes. We tried set enviroment variable RABBITMQ_NODE_ONLY, run service and executed command to remove node "rabbitmqctl forget_cluster_node --offline xxx at nodeName". On running this command we get information that node is not found "Error: {not_a_cluster_node,"The node selected is not in the cluster."}" what to do with this? We don't want to reinstall all. If we start it normally we get exception in log that: *"* *Error description: {error,{inconsistent_cluster,"Node rabbit at X thinks it's clustered with node rabbit at Y, but rabbit at Y..."}}* *"* Please help us. -------------- next part -------------- An HTML attachment was scrubbed... URL: From patrickjnewell at gmail.com Mon Apr 14 14:12:32 2014 From: patrickjnewell at gmail.com (patrickjnewell at gmail.com) Date: Mon, 14 Apr 2014 06:12:32 -0700 (PDT) Subject: [rabbitmq-discuss] Queue Deleted and Permissions Corrupted While Testing Flow Control Message-ID: Hello, While testing disk-based flow control, the queue my test client was consuming from suddenly got deleted and the user permissions appeared to get corrupted. I was running the broker on a CentOS 6.5 VM and had recently upgraded the broker from 3.2.1 to 3.3.0. The broker had 32 GB of available disk space with a 48 MB low watermark. The performed the test as follows: 1) Start a producer that sends 4 million persistent, 10 kb messages to a durable queue 2) Wait for the producer to block due to disk-based flow control (right around 3.1 million messages) 3) Start a consumer to drain the queue (prefetch count of 8, acknowledges one message at a time), allowing the producer to continue to send messages About 1 minute into step 3, the producer and consumer stopped suddenly and were disconnected about 1 minute later (the consumer received a Cancel). The queue disappeared from the management dashboard around the same time. About 30 seconds later, I was unable to use the dashboard at all -- my credentials appeared to be invalid and I was seeing 500 errors in the browser. The 500 error was as follows: {error,badarg, [{gen_event,send,2}, {rabbit_access_control,check_user_login,2}, {rabbit_mgmt_util,is_authorized,6}, {webmachine_resource,resource_call,3}, {webmachine_resource,do,3} {webmachine_decision_core,resource_call,1}, {webmachine_decision_core,decision,1}, {webmachine_decision_core,handle_request,2}]} I ran rabbitmqctl and saw that the users and virtual hosts I had created still existed. However, all exchanges and queues were no longer listed. Additionally, rabbitmqctl appeared to crash whenever I ran "list_user_permissions" or "list_permissions": [root at pnewellrabbitmqvm ~]# rabbitmqctl list_user_permissions admin Listing permissions for user "admin" ... Error: {noproc,{gen_server2,call, [worker_pool,{next_free,<4967.1308.0>},infinity]}} [root at pnewellrabbitmqvm ~]# rabbitmqctl list_permissions Listing permissions in vhost "/" ... Error: {noproc,{gen_server2,call, [worker_pool,{next_free,<4967.1317.0>},infinity]}} The sasl.log file listed several crash reports. It looks like the persistent message store process(?) crashed when it was reading a message from disk, which caused subsequent crashes. I included the log output below. After resetting the broker (rabbitmqctl stop_app, reset, start_app) and recreating the objects, I ran the same test again and reproduced the same issue. This time, however, the producer had managed to finish sending its messages and the consumer had received about 1.3 million messages before the queue and permissions crashed. I re-ran the same test 3 or 4 more times, but did not reproduce the issue. Any idea what might have caused this? My best guess is that something might have gotten corrupted when I originally performed the upgrade -- I stopped the service (/sbin/service rabbitmq-server stop), upgraded the broker and restarted it. The broker was empty with no messages persisted at the time of the upgrade. Any help would be much appreciated. Thanks, -Pat Here's the cleaned up sasl-log output (if I took out anything important, please let me know): =CRASH REPORT==== 9-Apr-2014::18:42:05 === crasher: initial call: gen:init_it/6 pid: <0.214.0> registered_name: [] exception exit: {{badmatch, {error, {misread, [{old_state, {client_msstate,msg_store_persistent, <<184,251,38,246,192,82,244,149,210,80,115,109,162, 244,176,61>>, {dict,1,16,16,8,80,48, {[],[],[],[],'...'}, {{'...'}}}, {state,233544,"/data..."}, rabbit_msg_store_ets_index,"/data/rabbit...", <0.210.0>,237641,229447,241738,245835}}, {file_num,681}, {offset,16669770}, {msg_id, <<160,82,77,79,243,175,93,42,1,219,170,168,159,121, 222,119>>}, {read, {ok, <<0,0,0,0,0,0,41,160,160,82,77,79,243,175,93,42,1, 219,170,168,159,121,222,119,131,104,6,100,0,13, 98,97,115,105,99,95,109,101,115,115,97,103,101, 104,4,100,0,8,114,101,115,111,117,114,99,101, 109,0,0,0,2,100,49,100,0,8,101,120,99,104,97, 110,103,101,109,0,0,0,19,66,105,108,108,105,110, 103,78,111,116,105,102,105,99,97,116,105,111, 110,108,0,0,0,1,109,0,0,0,10,97,112,112,49,46, 100,101,98,117,103,106,104,6,100,0,7,99,111,110, 116,101,110,116,97,60,100,0,4,110,111,110,101, 109,0,0,0,29,152,0,24,97,112,112,108,105,99,97, 116,105,111,110,47,111,99,116,101,116,45,115, 116,114,101,97,109,2,0,100,0,25,114,97,98,98, 105,116,95,102,114,97,109,105,110,103,95,97,109, 113,112,95,48,95,57,95,49,108,0,0,0,1,109,0,0, 40,183,123,34,117,117,105,100,34,58,34,52,48,53, 97,98,51,48,97,45,53,57,55,98,45,52,102,97,57, 45,57,101,100,100,45,50,102,100,97,51,56,57,50, 57,100,51,50,34,44,34,115,101,113,117,101,110, 99,101,34,58,49,56,55,53,54,56,44,34,97,112,112, 108,105,99,97,116,105,111,110,34,58,34,97,112, 112,48,34,44,34,101,118,101,110,116,34,58,34,69, 82,82,79,82,34,44,34,100,97,116,97,34,58,123,34, 115,116,114,105,110,103,34,58,34,97,97,97,97,97, 97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97, 97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97, 97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97, 97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97, 97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97, 97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97, 97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97, 97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97, 97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97, 97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97, 97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97, 97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97, 97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97, 97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97, 97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97, 97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97, 97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97, 97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97, 97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97, 97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97, 97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97, 97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97, 97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97, 97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97, 97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97, 97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97, 97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97, 97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97, 97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97, 97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97, 97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97, 97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97, 97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97, 97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97, 97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97, 97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97, 97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97, 97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97, 97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97, 97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97, 97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97, 97,97,97,97,46,46,46>>}}, {proc_dict, [{random_seed,{1397,21447,511}}, {{"...",fhc_file},{file,1,true}}, {'$ancestors', [rabbit_amqqueue_sup,rabbit_sup,<0.145.0>]}, {process_name,{rabbit_amqqueue_process,{'...'}}}, {fhc_age_tree,{2,{'...'}}}, {{ch,<0.901.0>}, {cr,<0.901.0>,#Ref<0.0.0.27418>,'...'}}, {credit_blocked,[]}, {{#Ref<0.0.0.1152>,fhc_handle},{handle,'...'}}, {{"...",'...'},{'...'}}, {{'...'},{'...'}}, {{'...'},50}, {'$initial_call',{'...'}}, {guid,{'...'}}, {{'...'},1350}]}]}}}, [{rabbit_msg_store,read_from_disk,2}, {rabbit_msg_store,client_read3,3}, {rabbit_msg_store,safe_ets_update_counter,5}, {rabbit_variable_queue,with_msg_store_state,3}, {rabbit_variable_queue,read_msg,3}, {rabbit_variable_queue,fetch,2}, {rabbit_mirror_queue_master,fetch,2}, {rabbit_amqqueue_process,fetch,2}]} in function gen_server2:terminate/3 ancestors: [rabbit_amqqueue_sup,rabbit_sup,<0.145.0>] messages: [{'$gen_cast', {deliver, {delivery,false,false,<0.552.0>, {basic_message, {resource,<<"d1">>,exchange,<<"BillingNotification">>}, [<<"app1.debug">>], {content,60, {'P_basic',<<"application/octet-stream">>,undefined, undefined,2,0,undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined,undefined}, <<152,0,24,97,112,112,108,105,99,97,116,105,111,110,47, 111,99,116,101,116,45,115,116,114,101,97,109,2,0>>, rabbit_framing_amqp_0_9_1, [<<"{\"uuid\":\"dae09ba7-bf47-455c-b512-49d434fb7c3f\",\"sequence\":2363388,\"application\":\"app2\",\"event\":\"DEBUG\",\"data\":{\"string\":\"aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa...">>]}, <<30,161,133,41,140,102,238,100,34,46,141,56,48,162,217, 14>>, true}, undefined}, false,flow}}, // Lots of queued up messages, acks, etc... '...'] links: [<0.211.0>] dictionary: [{random_seed,{1397,21447,511}}, {process_name, {rabbit_amqqueue_process, {resource,<<"d1">>,queue,<<"Appconnect">>}}}, {fhc_age_tree,{0,nil}}, {{ch,<0.901.0>}, {cr,<0.901.0>,#Ref<0.0.0.27418>, {[{187567,<<"amq.ctag-UzwVuTVR7OxfhipZ4Nls1Q">>}, {187566,<<"amq.ctag-UzwVuTVR7OxfhipZ4Nls1Q">>}], [{187561,<<"amq.ctag-UzwVuTVR7OxfhipZ4Nls1Q">>}, {187562,<<"amq.ctag-UzwVuTVR7OxfhipZ4Nls1Q">>}, {187563,<<"amq.ctag-UzwVuTVR7OxfhipZ4Nls1Q">>}, {187564,<<"amq.ctag-UzwVuTVR7OxfhipZ4Nls1Q">>}, {187565,<<"amq.ctag-UzwVuTVR7OxfhipZ4Nls1Q">>}]}, 1, {queue,[],[],0}, {qstate,<0.900.0>,dormant, {1, {<<"amq.ctag-UzwVuTVR7OxfhipZ4Nls1Q">>, {credit,1,auto}, nil,nil}}}, 168}}, {credit_blocked,[]}, {{xtype_to_module,direct},rabbit_exchange_type_direct}, {{xtype_to_module,topic},rabbit_exchange_type_topic}, {{credit_to,<0.552.0>},50}, {guid,{{230980502,69406936,3284697005,3000754076},0}}, {{credit_from,<0.205.0>},1350}] trap_exit: true status: running heap_size: 1346269 stack_size: 24 reductions: 1748356441 neighbours: =SUPERVISOR REPORT==== 9-Apr-2014::18:42:05 === Supervisor: {local,rabbit_amqqueue_sup} Context: child_terminated Reason: {{badmatch, {error, {misread, [{old_state, {client_msstate,msg_store_persistent, <<184,251,38,246,192,82,244,149,210,80,115,109,162,244, 176,61>>, {dict,1,16,16,8,80,48, {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]}, {{[],[],[],[],[],[],[],[],[], [['...']], [],[],[],[],[],[]}}}, {state,233544,"/data/rabbitmq/mnesia/rabbit at pn..."}, rabbit_msg_store_ets_index, "/data/rabbitmq/mnesia/rabbit at pnewellra...",<0.210.0>, 237641,229447,241738,245835}}, {file_num,681}, {offset,16669770}, {msg_id, <<160,82,77,79,243,175,93,42,1,219,170,168,159,121,222, 119>>}, {read, {ok, <<0,0,0,0,0,0,41,160,160,82,77,79,243,175,93,42,1, 219,170,168,159,121,222,119,131,104,6,100,0,13,98, 97,115,105,99,95,109,101,115,115,97,103,101,104,4, 100,0,8,114,101,115,111,117,114,99,101,109,0,0,0,2, 100,49,100,0,8,101,120,99,104,97,110,103,101,109,0, 0,0,19,66,105,108,108,105,110,103,78,111,116,105, 102,105,99,97,116,105,111,110,108,0,0,0,1,109,0,0, 0,10,97,112,112,49,46,100,101,98,117,103,106,104,6, 100,0,7,99,111,110,116,101,110,116,97,60,100,0,4, 110,111,110,101,109,0,0,0,29,152,0,24,97,112,112, 108,105,99,97,116,105,111,110,47,111,99,116,101, 116,45,115,116,114,101,97,109,2,0,100,0,25,114,97, 98,98,105,116,95,102,114,97,109,105,110,103,95,97, 109,113,112,95,48,95,57,95,49,108,0,0,0,1,109,0,0, 40,183,123,34,117,117,105,100,34,58,34,52,48,53,97, 98,51,48,97,45,53,57,55,98,45,52,102,97,57,45,57, 101,100,100,45,50,102,100,97,51,56,57,50,57,100,51, 50,34,44,34,115,101,113,117,101,110,99,101,34,58, 49,56,55,53,54,56,44,34,97,112,112,108,105,99,97, 116,105,111,110,34,58,34,97,112,112,48,34,44,34, 101,118,101,110,116,34,58,34,69,82,82,79,82,34,44, 34,100,97,116,97,34,58,123,34,115,116,114,105,110, 103,34,58,34,97,97,97,97,97,97,97,97,97,97,97,97, 97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97, 97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97, 97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97, 97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97, 97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97, 97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97, 97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97, 97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97, 97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97, 97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97, 97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97, 97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97, 97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97, 97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97, 97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97, 97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97, 97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97, 97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97, 97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97, 97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97, 97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97, 97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97, 97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97, 97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97, 97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97, 97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97, 97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97, 97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97, 97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97, 97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97, 97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97, 97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97, 97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97, 97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97, 97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97, 97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97, 97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97, 97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97, 97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97, 97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97, 97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97, 97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97, 97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97, 97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97, 97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97, 97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97, 97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97, 97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97, 97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97, 97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97, 97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97, 97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97, 97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97, 97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97, 97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97, 97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97, 97,46,46,46>>}}, {proc_dict, [{random_seed,{1397,21447,511}}, {{"/data/rabbitmq/mnesia/ra...",fhc_file}, {file,1,true}}, {'$ancestors', [rabbit_amqqueue_sup,rabbit_sup,<0.145.0>]}, {process_name, {rabbit_amqqueue_process, {resource,<<"d1">>,queue,<<"Appconnect">>}}}, {fhc_age_tree, {2, {{1397,83251,807183}, #Ref<0.0.0.1152>,nil, {{1397,83251,808277},#Ref<0.0.0.31857>,nil,nil}}}}, {{ch,<0.901.0>}, {cr,<0.901.0>,#Ref<0.0.0.27418>, {[{187567,<<"amq.ctag-UzwVuTVR7OxfhipZ4Nls1Q">>}, {187566,<<"amq.ctag-UzwVuTVR7OxfhipZ4Nls1Q">>}], [{187561,<<"amq.ctag-UzwVuTVR7OxfhipZ4Nls1Q">>}, {187562,<<"amq.ctag-UzwVuTVR7OxfhipZ4Nls1Q">>}, {187563,<<"amq.ctag-UzwVuTVR7OxfhipZ4Nls1Q">>}, {187564,'...'}, {'...'}]}, 1, {queue,[],[],0}, {qstate,<0.900.0>,dormant,{1,{'...'}}}, 168}}, {credit_blocked,[]}, {{#Ref<0.0.0.1152>,fhc_handle}, {handle, {file_descriptor,prim_file,{#Port<0.13248>,15}}, 0,false,291048,infinity, [[<<192,0,0,0,0,2,220,168>>], [<<128,0,0,0,0,2,220,175>>], [<<192,0,0,0,0,2,220,167>>], [<<128,0,0,0,0,2,220,174>>], [<<192,0,0,0,0,2,220,166>>], [<<128,0,0,0,0,2,220,173>>], [<<192,0,0,0,0,2,220,165>>], ['...'], ['...'], ['...'], ['...'], ['...'], ['...'], ['...'], '...'], true,"/data/rab...", [write,binary,raw,read], [{write_buffer,infinity}], true,true, {1397,83251,807183}}}, {{"/data/rabbitmq/mn...",fhc_file},{file,1,false}}, {{#Ref<0.0.0.31857>,fhc_handle}, {handle, {file_descriptor,prim_file,{#Port<0.18591>,23}}, 16680435,false,0,1048576,[],false,"/data/r...", [raw,binary,read], [{write_buffer,'...'}], false,true, {1397,83251,808277}}}, {{credit_to,<0.552.0>},50}, {'$initial_call',{gen,init_it,6}}, {guid,{{230980502,69406936,3284697005,3000754076},0}}, {{credit_from,<0.205.0>},1350}]}]}}}, [{rabbit_msg_store,read_from_disk,2}, {rabbit_msg_store,client_read3,3}, {rabbit_msg_store,safe_ets_update_counter,5}, {rabbit_variable_queue,with_msg_store_state,3}, {rabbit_variable_queue,read_msg,3}, {rabbit_variable_queue,fetch,2}, {rabbit_mirror_queue_master,fetch,2}, {rabbit_amqqueue_process,fetch,2}]} Offender: [{pid,<0.214.0>}, {name,rabbit_amqqueue}, {mfargs,{rabbit_amqqueue_process,start_link,undefined}}, {restart_type,temporary}, {shutdown,4294967295}, {child_type,worker}] =CRASH REPORT==== 9-Apr-2014::18:44:04 === crasher: initial call: gen:init_it/7 pid: <0.205.0> registered_name: msg_store_persistent exception exit: {{case_clause, {false, {msg_location, <<94,44,8,71,59,30,61,222,43,254,204,232,194, 42,158,238>>, 0,681,16584450,10665}}}, [{rabbit_msg_store,remove_message,3}, {rabbit_msg_store,'-handle_cast/2-fun-0-',3}, {lists,foldl,3}, {rabbit_msg_store,handle_cast,2}, {gen_server2,handle_msg,2}, {proc_lib,wake_up,3}]} in function gen_server2:terminate/3 ancestors: [rabbit_sup,<0.145.0>] messages: [{'$gen_cast',{delete_file,1634,16787111}}, {'$gen_cast',{delete_file,1654,16787101}}, {'$gen_cast', {client_dying, <<184,251,38,246,192,82,244,149,210,80,115,109,162,244, 176,61>>}}, {'$gen_cast', {client_delete, <<184,251,38,246,192,82,244,149,210,80,115,109,162,244, 176,61>>}}, {'DOWN',#Ref<0.0.0.1105>,process,<0.214.0>, {{badmatch, {error, {misread, [{old_state, {client_msstate,msg_store_persistent, <<184,251,38,246,192,82,244,149,210,80,115,109,162, 244,176,61>>, {dict,1,'...'}, {state,'...'}, rabbit_msg_store_ets_index,"...",<0.210.0>,237641, 229447,241738,245835}}, {file_num,681}, {offset,16669770}, {msg_id, <<160,82,77,79,243,175,93,42,1,219,170,168,159,121, 222,119>>}, {read, {ok, <<0,0,0,0,0,0,41,160,160,82,77,79,243,175,93,42,1, 219,170,168,159,121,222,119,131,104,6,100,0,13, 98,97,115,105,99,95,109,101,115,115,97,103,101, 104,4,100,0,8,114,101,115,111,117,114,99,101,109, 0,0,0,2,100,49,100,0,8,101,120,99,104,97,110,103, 101,109,0,0,0,19,66,105,108,108,105,110,103,78, 111,116,105,102,105,99,97,116,105,111,110,108,0, 0,0,1,109,0,0,0,10,97,112,112,49,46,100,101,98, 117,103,106,104,6,100,0,7,99,111,110,116,101,110, 116,97,60,100,0,4,110,111,110,101,109,0,0,0,29, 152,0,24,97,112,112,108,105,99,97,116,105,111, 110,47,111,99,116,101,116,45,115,116,114,101,97, 109,2,0,100,0,25,114,97,98,98,105,116,95,102,114, 97,109,105,110,103,95,97,109,113,112,95,48,95,57, 95,49,108,0,0,0,1,109,0,0,40,183,123,34,117,117, 105,100,34,58,34,52,48,53,97,98,51,48,97,45,53, 57,55,98,45,52,102,97,57,45,57,101,100,100,45,50, 102,100,97,51,56,57,50,57,100,51,50,34,44,34,115, 101,113,117,101,110,99,101,34,58,49,56,55,53,54, 56,44,34,97,112,112,108,105,99,97,116,105,111, 110,34,58,34,97,112,112,48,34,44,34,101,118,101, 110,116,34,58,34,69,82,82,79,82,34,44,34,100,97, 116,97,34,58,123,34,115,116,114,105,110,103,34, 58,34,97,97,97,97,97,97,97,97,97,97,97,97,97,97, 97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97, 97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97, 97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97, 97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97, 97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97, 97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97, 97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97, 97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97, 97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97, 97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97, 97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97, 97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97, 97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97, 97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97, 97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97, 97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97, 97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97, 97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97, 97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97, 97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97, 97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97, 97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97, 97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97, 97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97, 97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97, 97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97, 97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97, 97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97, 97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97, 97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97, 97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97, 97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97, 97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97, 97,97,97,97,97,97,97,97,97,97,97,97,97,97,97,97, 97,97,97,97,97,97,97,46,46,46>>}}, {proc_dict, [{'...'}, {'...'}, {'...'}, {'...'}, {'...'}, {'...'}, {'...'}, '...']}]}}}, [{rabbit_msg_store,read_from_disk,2}, {rabbit_msg_store,client_read3,3}, {rabbit_msg_store,safe_ets_update_counter,5}, {rabbit_variable_queue,with_msg_store_state,3}, {rabbit_variable_queue,read_msg,3}, {rabbit_variable_queue,fetch,2}, {rabbit_mirror_queue_master,fetch,2}, {rabbit_amqqueue_process,fetch,2}]}}, {'$gen_cast',{delete_file,1653,16787145}}, {'$gen_cast',{delete_file,1652,16787106}}, {'$gen_cast',{delete_file,1651,16787067}}, {'$gen_cast',{delete_file,1650,16787153}}, {'$gen_cast',{delete_file,1649,16787131}}, {'$gen_cast',{delete_file,1648,16787160}}, {'$gen_cast',{delete_file,1647,16787155}}, {'$gen_cast',{delete_file,1646,16787099}}, {'$gen_cast',{delete_file,1645,16787086}}, {'$gen_cast',{delete_file,1665,16787076}}, {'$gen_cast',{delete_file,1664,16787088}}, {'$gen_cast',{delete_file,1663,16787115}}, {'$gen_cast',{delete_file,1662,16787099}}, {'$gen_cast',{delete_file,1661,16787074}}, {'$gen_cast',{delete_file,1660,16787049}}, {'$gen_cast',{delete_file,1659,16787133}}, {'$gen_cast',{delete_file,1658,16787128}}, {'$gen_cast',{delete_file,1657,16787110}}, {'$gen_cast',{delete_file,1656,16787049}}, {'$gen_cast',{delete_file,1655,16787126}}, {'$gen_cast',{delete_file,1675,16787031}}, {'$gen_cast',{delete_file,1674,16787101}}, {'$gen_cast',{delete_file,1673,16787040}}, {'$gen_cast',{delete_file,1672,16787094}}, {'$gen_cast',{delete_file,1671,16787100}}, {'$gen_cast',{delete_file,1670,16787017}}, {'$gen_cast',{delete_file,1669,16787096}}, {'$gen_cast',{delete_file,1668,16787175}}, {'$gen_cast',{delete_file,1667,16787143}}, {'$gen_cast',{delete_file,1666,16787019}}, {'$gen_cast',{delete_file,1686,16787085}}, {'$gen_cast',{delete_file,1685,16787079}}, {'$gen_cast',{delete_file,1684,16787117}}, {'$gen_cast',{delete_file,1683,16787198}}, {'$gen_cast',{delete_file,1682,16787109}}, {'$gen_cast',{delete_file,1681,16787141}}, {'$gen_cast',{delete_file,1680,16787142}}, {'$gen_cast',{delete_file,1679,16787100}}, {'$gen_cast',{delete_file,1678,16787137}}, {'$gen_cast',{delete_file,1677,16787122}}, {'$gen_cast',{delete_file,1676,16787092}}, {'$gen_cast',{delete_file,1696,16787007}}, {'$gen_cast',{delete_file,1695,16787153}}, {'$gen_cast',{delete_file,1694,16787025}}, {'$gen_cast',{delete_file,1693,16787045}}, {'$gen_cast',{delete_file,1692,16787134}}, {'$gen_cast',{delete_file,1691,16787109}}, {'$gen_cast',{delete_file,1690,16787153}}, {'$gen_cast',{delete_file,1689,16787121}}, {'$gen_cast',{delete_file,1688,16787109}}, {'$gen_cast',{delete_file,1687,16787047}}, {'$gen_cast',{delete_file,1706,16787131}}, {'$gen_cast',{delete_file,1705,16787087}}, {'$gen_cast',{delete_file,1704,16787098}}, {'$gen_cast',{delete_file,1703,16787121}}, {'$gen_cast',{delete_file,1702,16787086}}, {'$gen_cast',{delete_file,1701,16787059}}, {'$gen_cast',{delete_file,1700,16787084}}, {'$gen_cast',{delete_file,1699,'...'}}, {'$gen_cast',{delete_file,'...'}}, {'$gen_cast',{'...'}}, {'$gen_cast',{'...'}}, {'$gen_cast',{'...'}}, {'$gen_cast',{'...'}}, {'$gen_cast',{'...'}}, {'$gen_cast',{'...'}}, {'$gen_cast',{'...'}}, {'$gen_cast','...'}, {'...'}, {'...'}, {'...'}, {'...'}, {'...'}, {'...'}, {'...'}, '...'] links: [<0.146.0>] dictionary: [{{credit_to,<0.214.0>},350},{fhc_age_tree,{0,nil}}] trap_exit: true status: running heap_size: 832040 stack_size: 24 reductions: 790489983 neighbours: =SUPERVISOR REPORT==== 9-Apr-2014::18:44:04 === Supervisor: {local,rabbit_sup} Context: child_terminated Reason: {{case_clause, {false, {msg_location, <<94,44,8,71,59,30,61,222,43,254,204,232,194,42, 158,238>>, 0,681,16584450,10665}}}, [{rabbit_msg_store,remove_message,3}, {rabbit_msg_store,'-handle_cast/2-fun-0-',3}, {lists,foldl,3}, {rabbit_msg_store,handle_cast,2}, {gen_server2,handle_msg,2}, {proc_lib,wake_up,3}]} Offender: [{pid,<0.205.0>}, {name,msg_store_persistent}, {mfargs, {rabbit_msg_store,start_link, [msg_store_persistent, "/data/rabbitmq/mnesia/rabbit at pnewellrabbitmqvm", [<<197,113,14,78,169,98,103,74,158,178,5,165,183, 174,221,28>>, <<186,194,196,141,2,108,179,221,99,117,59,65,250, 235,145,61>>, <<184,251,38,246,192,82,244,149,210,80,115,109,162, 244,176,61>>, <<123,55,234,4,244,110,71,11,21,208,154,4,170,7,65, 98>>, <<245,180,254,87,28,109,130,169,124,40,55,231,11, 150,52,177>>, <<162,87,146,34,151,178,125,226,157,199,2,24,48, 57,175,102>>], {#Fun, {start, [{resource,<<"d1-cbs">>,queue, <<"Appconnect">>}, {resource,<<"/">>,queue,<<"Appconnect">>}, {resource,<<"d1">>,queue,<<"Appconnect">>}, {resource,<<"d1-cbs">>,queue,<<"ApiUsage">>}, {resource,<<"/">>,queue,<<"ApiUsage">>}, {resource,<<"d1">>,queue, <<"ApiUsage">>}]}}]}}, {restart_type,transient}, {shutdown,4294967295}, {child_type,worker}] =SUPERVISOR REPORT==== 9-Apr-2014::18:44:04 === Supervisor: {local,rabbit_sup} Context: shutdown Reason: reached_max_restart_intensity Offender: [{pid,<0.205.0>}, {name,msg_store_persistent}, {mfargs, {rabbit_msg_store,start_link, [msg_store_persistent, "/data/rabbitmq/mnesia/rabbit at pnewellrabbitmqvm", [<<197,113,14,78,169,98,103,74,158,178,5,165,183, 174,221,28>>, <<186,194,196,141,2,108,179,221,99,117,59,65,250, 235,145,61>>, <<184,251,38,246,192,82,244,149,210,80,115,109,162, 244,176,61>>, <<123,55,234,4,244,110,71,11,21,208,154,4,170,7,65, 98>>, <<245,180,254,87,28,109,130,169,124,40,55,231,11, 150,52,177>>, <<162,87,146,34,151,178,125,226,157,199,2,24,48, 57,175,102>>], {#Fun, {start, [{resource,<<"d1-cbs">>,queue, <<"Appconnect">>}, {resource,<<"/">>,queue,<<"Appconnect">>}, {resource,<<"d1">>,queue,<<"Appconnect">>}, {resource,<<"d1-cbs">>,queue,<<"ApiUsage">>}, {resource,<<"/">>,queue,<<"ApiUsage">>}, {resource,<<"d1">>,queue, <<"ApiUsage">>}]}}]}}, {restart_type,transient}, {shutdown,4294967295}, {child_type,worker}] =CRASH REPORT==== 9-Apr-2014::18:44:04 === crasher: initial call: gen:init_it/6 pid: <0.217.0> registered_name: [] exception exit: {noproc, {gen_server2,call, [msg_store_persistent, {client_terminate, <<162,87,146,34,151,178,125,226,157,199,2,24, 48,57,175,102>>}, infinity]}} in function gen_server2:terminate/3 ancestors: [rabbit_amqqueue_sup,rabbit_sup,<0.145.0>] messages: [] links: [<0.223.0>] dictionary: [{random_seed,{1397,21447,29346}}, {process_name, {rabbit_amqqueue_process, {resource,<<"d1">>,queue,<<"ApiUsage">>}}}, {guid,{{2720735611,2534855728,3608505330,4142421861},0}}] trap_exit: true status: running heap_size: 1597 stack_size: 24 reductions: 49987 neighbours: neighbour: [{pid,<0.226.0>}, {registered_name,[]}, {initial_call, {gen,init_it, ['Argument__1','Argument__2','Argument__3', 'Argument__4','Argument__5','Argument__6']}}, {current_function,{erlang,hibernate,3}}, {ancestors, [<0.223.0>,<0.217.0>,rabbit_amqqueue_sup,rabbit_sup, <0.145.0>]}, {messages,[]}, {links,[<0.223.0>]}, {dictionary, [{random_seed,{1397,21447,25352}}, {process_name, {gm,{resource,<<"d1">>,queue,<<"ApiUsage">>}}}]}, {trap_exit,false}, {status,waiting}, {heap_size,233}, {stack_size,0}, {reductions,244}] neighbour: [{pid,<0.223.0>}, {registered_name,[]}, {initial_call, {gen,init_it, ['Argument__1','Argument__2','Argument__3', 'Argument__4','Argument__5','Argument__6']}}, {current_function,{erlang,hibernate,3}}, {ancestors, [<0.217.0>,rabbit_amqqueue_sup,rabbit_sup,<0.145.0>]}, {messages,[]}, {links,[<0.217.0>,<0.226.0>]}, {dictionary, [{process_name, {rabbit_mirror_queue_coordinator, {resource,<<"d1">>,queue,<<"ApiUsage">>}}}]}, {trap_exit,false}, {status,waiting}, {heap_size,377}, {stack_size,0}, {reductions,120}] =CRASH REPORT==== 9-Apr-2014::18:44:04 === crasher: initial call: gen:init_it/6 pid: <0.216.0> registered_name: [] exception exit: {noproc, {gen_server2,call, [msg_store_persistent, {client_terminate, <<245,180,254,87,28,109,130,169,124,40,55, 231,11,150,52,177>>}, infinity]}} in function gen_server2:terminate/3 ancestors: [rabbit_amqqueue_sup,rabbit_sup,<0.145.0>] messages: [] links: [<0.221.0>] dictionary: [{random_seed,{1397,21447,28485}}, {process_name, {rabbit_amqqueue_process, {resource,<<"/">>,queue,<<"ApiUsage">>}}}, {guid,{{2480445846,550087633,2809237486,3458201455},0}}] trap_exit: true status: running heap_size: 1597 stack_size: 24 reductions: 6301 neighbours: neighbour: [{pid,<0.224.0>}, {registered_name,[]}, {initial_call, {gen,init_it, ['Argument__1','Argument__2','Argument__3', 'Argument__4','Argument__5','Argument__6']}}, {current_function,{erlang,hibernate,3}}, {ancestors, [<0.221.0>,<0.216.0>,rabbit_amqqueue_sup,rabbit_sup, <0.145.0>]}, {messages,[]}, {links,[<0.221.0>]}, {dictionary, [{random_seed,{1397,21447,25330}}, {process_name, {gm,{resource,<<"/">>,queue,<<"ApiUsage">>}}}]}, {trap_exit,false}, {status,waiting}, {heap_size,233}, {stack_size,0}, {reductions,244}] neighbour: [{pid,<0.221.0>}, {registered_name,[]}, {initial_call, {gen,init_it, ['Argument__1','Argument__2','Argument__3', 'Argument__4','Argument__5','Argument__6']}}, {current_function,{erlang,hibernate,3}}, {ancestors, [<0.216.0>,rabbit_amqqueue_sup,rabbit_sup,<0.145.0>]}, {messages,[]}, {links,[<0.216.0>,<0.224.0>]}, {dictionary, [{process_name, {rabbit_mirror_queue_coordinator, {resource,<<"/">>,queue,<<"ApiUsage">>}}}]}, {trap_exit,false}, {status,waiting}, {heap_size,377}, {stack_size,0}, {reductions,120}] =CRASH REPORT==== 9-Apr-2014::18:44:04 === crasher: initial call: gen:init_it/6 pid: <0.215.0> registered_name: [] exception exit: {noproc, {gen_server2,call, [msg_store_persistent, {client_terminate, <<123,55,234,4,244,110,71,11,21,208,154,4, 170,7,65,98>>}, infinity]}} in function gen_server2:terminate/3 ancestors: [rabbit_amqqueue_sup,rabbit_sup,<0.145.0>] messages: [] links: [<0.222.0>] dictionary: [{random_seed,{1397,21447,29365}}, {process_name, {rabbit_amqqueue_process, {resource,<<"d1-cbs">>,queue,<<"ApiUsage">>}}}, {guid,{{1612108673,2871644604,1399026086,1533985467},0}}] trap_exit: true status: running heap_size: 1597 stack_size: 24 reductions: 49896 neighbours: neighbour: [{pid,<0.225.0>}, {registered_name,[]}, {initial_call, {gen,init_it, ['Argument__1','Argument__2','Argument__3', 'Argument__4','Argument__5','Argument__6']}}, {current_function,{erlang,hibernate,3}}, {ancestors, [<0.222.0>,<0.215.0>,rabbit_amqqueue_sup,rabbit_sup, <0.145.0>]}, {messages,[]}, {links,[<0.222.0>]}, {dictionary, [{random_seed,{1397,21447,25344}}, {process_name, {gm,{resource,<<"d1-cbs">>,queue, <<"ApiUsage">>}}}]}, {trap_exit,false}, {status,waiting}, {heap_size,233}, {stack_size,0}, {reductions,244}] neighbour: [{pid,<0.222.0>}, {registered_name,[]}, {initial_call, {gen,init_it, ['Argument__1','Argument__2','Argument__3', 'Argument__4','Argument__5','Argument__6']}}, {current_function,{erlang,hibernate,3}}, {ancestors, [<0.215.0>,rabbit_amqqueue_sup,rabbit_sup,<0.145.0>]}, {messages,[]}, {links,[<0.215.0>,<0.225.0>]}, {dictionary, [{process_name, {rabbit_mirror_queue_coordinator, {resource,<<"d1-cbs">>,queue, <<"ApiUsage">>}}}]}, {trap_exit,false}, {status,waiting}, {heap_size,377}, {stack_size,0}, {reductions,120}] =CRASH REPORT==== 9-Apr-2014::18:44:04 === crasher: initial call: gen:init_it/6 pid: <0.213.0> registered_name: [] exception exit: {noproc, {gen_server2,call, [msg_store_persistent, {client_terminate, <<186,194,196,141,2,108,179,221,99,117,59,65, 250,235,145,61>>}, infinity]}} in function gen_server2:terminate/3 ancestors: [rabbit_amqqueue_sup,rabbit_sup,<0.145.0>] messages: [] links: [<0.227.0>] dictionary: [{random_seed,{1397,21447,30135}}, {process_name, {rabbit_amqqueue_process, {resource,<<"/">>,queue,<<"Appconnect">>}}}, {guid,{{2712885445,3207113857,3507961840,57524982},0}}] trap_exit: true status: running heap_size: 2584 stack_size: 24 reductions: 51725 neighbours: neighbour: [{pid,<0.228.0>}, {registered_name,[]}, {initial_call, {gen,init_it, ['Argument__1','Argument__2','Argument__3', 'Argument__4','Argument__5','Argument__6']}}, {current_function,{erlang,hibernate,3}}, {ancestors, [<0.227.0>,<0.213.0>,rabbit_amqqueue_sup,rabbit_sup, <0.145.0>]}, {messages,[]}, {links,[<0.227.0>]}, {dictionary, [{random_seed,{1397,21447,25502}}, {process_name, {gm,{resource,<<"/">>,queue,<<"Appconnect">>}}}]}, {trap_exit,false}, {status,waiting}, {heap_size,233}, {stack_size,0}, {reductions,245}] neighbour: [{pid,<0.227.0>}, {registered_name,[]}, {initial_call, {gen,init_it, ['Argument__1','Argument__2','Argument__3', 'Argument__4','Argument__5','Argument__6']}}, {current_function,{erlang,hibernate,3}}, {ancestors, [<0.213.0>,rabbit_amqqueue_sup,rabbit_sup,<0.145.0>]}, {messages,[]}, {links,[<0.213.0>,<0.228.0>]}, {dictionary, [{process_name, {rabbit_mirror_queue_coordinator, {resource,<<"/">>,queue,<<"Appconnect">>}}}]}, {trap_exit,false}, {status,waiting}, {heap_size,377}, {stack_size,0}, {reductions,119}] =CRASH REPORT==== 9-Apr-2014::18:44:04 === crasher: initial call: gen:init_it/6 pid: <0.212.0> registered_name: [] exception exit: {noproc, {gen_server2,call, [msg_store_persistent, {client_terminate, <<197,113,14,78,169,98,103,74,158,178,5,165, 183,174,221,28>>}, infinity]}} in function gen_server2:terminate/3 ancestors: [rabbit_amqqueue_sup,rabbit_sup,<0.145.0>] messages: [] links: [<0.219.0>] dictionary: [{random_seed,{1397,21447,24766}}, {process_name, {rabbit_amqqueue_process, {resource,<<"d1-cbs">>,queue,<<"Appconnect">>}}}, {guid,{{3043230381,2235775552,2320571086,906396196},0}}] trap_exit: true status: running heap_size: 2584 stack_size: 24 reductions: 6321 neighbours: neighbour: [{pid,<0.220.0>}, {registered_name,[]}, {initial_call, {gen,init_it, ['Argument__1','Argument__2','Argument__3', 'Argument__4','Argument__5','Argument__6']}}, {current_function,{erlang,hibernate,3}}, {ancestors, [<0.219.0>,<0.212.0>,rabbit_amqqueue_sup,rabbit_sup, <0.145.0>]}, {messages,[]}, {links,[<0.219.0>]}, {dictionary, [{random_seed,{1397,21447,23446}}, {process_name, {gm,{resource,<<"d1-cbs">>,queue, <<"Appconnect">>}}}]}, {trap_exit,false}, {status,waiting}, {heap_size,233}, {stack_size,0}, {reductions,245}] neighbour: [{pid,<0.219.0>}, {registered_name,[]}, {initial_call, {gen,init_it, ['Argument__1','Argument__2','Argument__3', 'Argument__4','Argument__5','Argument__6']}}, {current_function,{erlang,hibernate,3}}, {ancestors, [<0.212.0>,rabbit_amqqueue_sup,rabbit_sup,<0.145.0>]}, {messages,[]}, {links,[<0.212.0>,<0.220.0>]}, {dictionary, [{process_name, {rabbit_mirror_queue_coordinator, {resource,<<"d1-cbs">>,queue, <<"Appconnect">>}}}]}, {trap_exit,false}, {status,waiting}, {heap_size,377}, {stack_size,0}, {reductions,119}] =SUPERVISOR REPORT==== 9-Apr-2014::18:44:04 === Supervisor: {local,rabbit_amqqueue_sup} Context: shutdown_error Reason: {noproc, {gen_server2,call, [msg_store_persistent, {client_terminate, <<186,194,196,141,2,108,179,221,99,117,59,65, 250,235,145,61>>}, infinity]}} Offender: [{nb_children,1}, {name,rabbit_amqqueue}, {mfargs,{rabbit_amqqueue_process,start_link,[]}}, {restart_type,temporary}, {shutdown,4294967295}, {child_type,worker}] =SUPERVISOR REPORT==== 9-Apr-2014::18:44:04 === Supervisor: {local,rabbit_amqqueue_sup} Context: shutdown_error Reason: {noproc, {gen_server2,call, [msg_store_persistent, {client_terminate, <<162,87,146,34,151,178,125,226,157,199,2,24,48, 57,175,102>>}, infinity]}} Offender: [{nb_children,1}, {name,rabbit_amqqueue}, {mfargs,{rabbit_amqqueue_process,start_link,[]}}, {restart_type,temporary}, {shutdown,4294967295}, {child_type,worker}] =SUPERVISOR REPORT==== 9-Apr-2014::18:44:04 === Supervisor: {local,rabbit_amqqueue_sup} Context: shutdown_error Reason: {noproc, {gen_server2,call, [msg_store_persistent, {client_terminate, <<197,113,14,78,169,98,103,74,158,178,5,165,183, 174,221,28>>}, infinity]}} Offender: [{nb_children,1}, {name,rabbit_amqqueue}, {mfargs,{rabbit_amqqueue_process,start_link,[]}}, {restart_type,temporary}, {shutdown,4294967295}, {child_type,worker}] =SUPERVISOR REPORT==== 9-Apr-2014::18:44:04 === Supervisor: {local,rabbit_amqqueue_sup} Context: shutdown_error Reason: {noproc, {gen_server2,call, [msg_store_persistent, {client_terminate, <<245,180,254,87,28,109,130,169,124,40,55,231, 11,150,52,177>>}, infinity]}} Offender: [{nb_children,1}, {name,rabbit_amqqueue}, {mfargs,{rabbit_amqqueue_process,start_link,[]}}, {restart_type,temporary}, {shutdown,4294967295}, {child_type,worker}] =SUPERVISOR REPORT==== 9-Apr-2014::18:44:04 === Supervisor: {local,rabbit_amqqueue_sup} Context: shutdown_error Reason: {noproc, {gen_server2,call, [msg_store_persistent, {client_terminate, <<123,55,234,4,244,110,71,11,21,208,154,4,170,7, 65,98>>}, infinity]}} Offender: [{nb_children,1}, {name,rabbit_amqqueue}, {mfargs,{rabbit_amqqueue_process,start_link,[]}}, {restart_type,temporary}, {shutdown,4294967295}, {child_type,worker}] (This same crash report is repeated several times afterwards) =CRASH REPORT==== 9-Apr-2014::18:44:08 === crasher: initial call: rabbit_mgmt_external_stats:init/1 pid: <0.248.0> registered_name: rabbit_mgmt_external_stats exception exit: {noproc, {gen_server,call, [rabbit_node_monitor,partitions,infinity]}} in function gen_server:terminate/6 ancestors: [rabbit_mgmt_agent_sup,<0.246.0>] messages: [] links: [<0.247.0>] dictionary: [] trap_exit: false status: running heap_size: 1597 stack_size: 24 reductions: 375236 neighbours: -------------- next part -------------- An HTML attachment was scrubbed... URL: From simon at rabbitmq.com Mon Apr 14 14:48:02 2014 From: simon at rabbitmq.com (Simon MacMullen) Date: Mon, 14 Apr 2014 14:48:02 +0100 Subject: [rabbitmq-discuss] Issue deleting Queue In-Reply-To: References: Message-ID: <534BE712.4060907@rabbitmq.com> On 14/04/14 14:25, Chris Nicel wrote: > We are still seeing the below when we try to view a queue with a + in > the management plugin. > > Not found > > The object you clicked on was not found; it may have been deleted on > the server. > > I guess this is related to us being on 3.2.2 as it doesn't appear to > be an issue on 3.2.3. I have just tried > onhttp://dev.rabbitmq.com/mgmt/ which allowed me to create the queue > and then view it in the management plugin. I also noticed the version > of erlang you're using inhttp://dev.rabbitmq.com/mgmt/ is only 14A. > We are using 16B03. But I guess this is probably not related? Nothing in this region should have changed between 3.2.2 and 3.3.0, nor should it be related to the Erlang version used. If you're seeing this bug on your machine but not others then I wonder if you are going via some broken proxy? We have definitely seen proxies break URL escaping before now. Cheers, Simon -- Simon MacMullen RabbitMQ, Pivotal From Chris.Nicel at 15below.com Mon Apr 14 14:49:28 2014 From: Chris.Nicel at 15below.com (Chris Nicel) Date: Mon, 14 Apr 2014 08:49:28 -0500 Subject: [rabbitmq-discuss] Issue deleting Queue In-Reply-To: <534BE712.4060907@rabbitmq.com> References: <534BE712.4060907@rabbitmq.com> Message-ID: We are using nginx as a proxy. I don't believe it supports the nocanon option that is mentioned in your sample apache proxy config. I will investigate this further. Cheers Chris -----Original Message----- From: Simon MacMullen [mailto:simon at rabbitmq.com] Sent: 14 April 2014 14:48 To: Discussions about RabbitMQ; Chris Nicel Subject: Re: [rabbitmq-discuss] Issue deleting Queue On 14/04/14 14:25, Chris Nicel wrote: > We are still seeing the below when we try to view a queue with a + in > the management plugin. > > Not found > > The object you clicked on was not found; it may have been deleted on > the server. > > I guess this is related to us being on 3.2.2 as it doesn't appear to > be an issue on 3.2.3. I have just tried > onhttp://dev.rabbitmq.com/mgmt/ which allowed me to create the queue > and then view it in the management plugin. I also noticed the version > of erlang you're using inhttp://dev.rabbitmq.com/mgmt/ is only 14A. > We are using 16B03. But I guess this is probably not related? Nothing in this region should have changed between 3.2.2 and 3.3.0, nor should it be related to the Erlang version used. If you're seeing this bug on your machine but not others then I wonder if you are going via some broken proxy? We have definitely seen proxies break URL escaping before now. Cheers, Simon -- Simon MacMullen RabbitMQ, Pivotal 15below Limited: Company registered in England and Wales No 3945289 Registered Office: Lyndean House, 43-46 Queens Road, Brighton BN1 3XB, United Kingdom 15below Australia Pty Limited: ABN 25 132 716 379 Level 21, Tower 2 Darling Park, 201 Sussex Street, Sydney, NSW 2000, Australia Please think about the environment before printing this email. ************************************************************************ This email and any attachments may be confidential and/or legally privileged and are solely for the use of the intended recipient. If you have received this email in error please contact the sender. Any views or opinions expressed within this e-mail are solely those of the sender, and do not necessarily represent those of 15below unless otherwise specifically stated. Although 15below has taken every reasonable precaution to ensure that any attachment to this e-mail has been checked for viruses, it is strongly recommended that you carry out your own virus check before opening any attachment, as we cannot accept liability for any damage sustained as a result of software virus infection. From matthias at rabbitmq.com Mon Apr 14 15:30:35 2014 From: matthias at rabbitmq.com (Matthias Radestock) Date: Mon, 14 Apr 2014 15:30:35 +0100 Subject: [rabbitmq-discuss] =?koi8-r?b?mkluc3RhbGxpbmcgYnJva2VyIG9uIGVh?= =?koi8-r?b?Y2ggbm9kZSBpbnN0ZWFkIG9mIGNsaWVudC1vbmx5IGFwcHJvYWNo?= In-Reply-To: <8F9C4C4C-5EA3-403E-8CC9-373E98EC637E@rutoll.ru> References: <2efb248a-473f-4d71-8caa-a6df339278a8@googlegroups.com> <534BA878.5070301@rabbitmq.com> <8F9C4C4C-5EA3-403E-8CC9-373E98EC637E@rutoll.ru> Message-ID: <534BF10B.4050409@rabbitmq.com> On 14/04/14 15:04, ??????? ???????? wrote: > Thank you, Matthias. Is there any limits for queue size to be reached > while working in autonomy/degrade mode? I need this to decide how > long my setup could work without connectivity. The limiting factors are memory and disk space. When memory gets tight and messages are fully paged out to disk, each message requires about 122 bytes of memory (on a 64-bit machine), and about 255 bytes + payload size of disk space. Matthias. From maximov at rutoll.ru Mon Apr 14 15:04:51 2014 From: maximov at rutoll.ru (=?koi8-r?B?7snLz8zByiDtwcvTyc3P1w==?=) Date: Mon, 14 Apr 2014 18:04:51 +0400 Subject: [rabbitmq-discuss] =?koi8-r?b?mkluc3RhbGxpbmcgYnJva2VyIG9uIGVh?= =?koi8-r?b?Y2ggbm9kZSBpbnN0ZWFkIG9mIGNsaWVudC1vbmx5IGFwcHJvYWNo?= In-Reply-To: <534BA878.5070301@rabbitmq.com> References: <2efb248a-473f-4d71-8caa-a6df339278a8@googlegroups.com> <534BA878.5070301@rabbitmq.com> Message-ID: <8F9C4C4C-5EA3-403E-8CC9-373E98EC637E@rutoll.ru> Thank you, Matthias. Is there any limits for queue size to be reached while working in autonomy/degrade mode? I need this to decide how long my setup could work without connectivity. -- Nikolay 14 ???. 2014 ?., ? 13:20, Matthias Radestock ???????(?): > On 09/04/14 10:46, Nikolay Maximov wrote: >> I have a network of nodes which has some events to be saved in local >> store and queued for further processing in central back office. Please >> advice what is better to manage and costs less: to have local store and >> application which store message first and queue it then versus local >> rabbitmq broker to do the same thing with persistent queue. I am looking >> for best practice in this approach. Does someone has an exprience in >> such design where broker installed on each node? > > A star-shaped RabbitMQ exchange federation (http://www.rabbitmq.com/federated-exchanges.html) - with local nodes serving as resilient store-and-forward buffers to a central broker - would work here and is quite a common configuration for exactly the scenario you are describing. > > Matthias. > From mklishin at gopivotal.com Mon Apr 14 16:30:33 2014 From: mklishin at gopivotal.com (Michael Klishin) Date: Mon, 14 Apr 2014 19:30:33 +0400 Subject: [rabbitmq-discuss] Recover cluster from crash In-Reply-To: <665d6495-ca13-465d-b741-142f5835a6f0@googlegroups.com> References: <665d6495-ca13-465d-b741-142f5835a6f0@googlegroups.com> Message-ID: On 14 April 2014 at 17:34:33, js.future.software at gmail.com (js.future.software at gmail.com) wrote: > > Error description: > {error,{inconsistent_cluster,"Node rabbit at X thinks it's > clustered with node rabbit at Y, but rabbit at Y??}} Have you tried resetting the node that?s failed?? -- MK Software Engineer, Pivotal/RabbitMQ From Chris.Nicel at 15below.com Mon Apr 14 16:33:38 2014 From: Chris.Nicel at 15below.com (Chris Nicel) Date: Mon, 14 Apr 2014 10:33:38 -0500 Subject: [rabbitmq-discuss] Issue deleting Queue In-Reply-To: References: <534BE712.4060907@rabbitmq.com> Message-ID: Thanks for all the help on this one. Nginx is the cause of our issues. Although it works perfectly until someone introduced a + into their queue name. Cheers Chris -----Original Message----- From: rabbitmq-discuss [mailto:rabbitmq-discuss-bounces at lists.rabbitmq.com] On Behalf Of Chris Nicel Sent: 14 April 2014 14:49 To: Simon MacMullen; Discussions about RabbitMQ Subject: Re: [rabbitmq-discuss] Issue deleting Queue We are using nginx as a proxy. I don't believe it supports the nocanon option that is mentioned in your sample apache proxy config. I will investigate this further. Cheers Chris -----Original Message----- From: Simon MacMullen [mailto:simon at rabbitmq.com] Sent: 14 April 2014 14:48 To: Discussions about RabbitMQ; Chris Nicel Subject: Re: [rabbitmq-discuss] Issue deleting Queue On 14/04/14 14:25, Chris Nicel wrote: > We are still seeing the below when we try to view a queue with a + in > the management plugin. > > Not found > > The object you clicked on was not found; it may have been deleted on > the server. > > I guess this is related to us being on 3.2.2 as it doesn't appear to > be an issue on 3.2.3. I have just tried > onhttp://dev.rabbitmq.com/mgmt/ which allowed me to create the queue > and then view it in the management plugin. I also noticed the version > of erlang you're using inhttp://dev.rabbitmq.com/mgmt/ is only 14A. > We are using 16B03. But I guess this is probably not related? Nothing in this region should have changed between 3.2.2 and 3.3.0, nor should it be related to the Erlang version used. If you're seeing this bug on your machine but not others then I wonder if you are going via some broken proxy? We have definitely seen proxies break URL escaping before now. Cheers, Simon -- Simon MacMullen RabbitMQ, Pivotal 15below Limited: Company registered in England and Wales No 3945289 Registered Office: Lyndean House, 43-46 Queens Road, Brighton BN1 3XB, United Kingdom 15below Australia Pty Limited: ABN 25 132 716 379 Level 21, Tower 2 Darling Park, 201 Sussex Street, Sydney, NSW 2000, Australia Please think about the environment before printing this email. ************************************************************************ This email and any attachments may be confidential and/or legally privileged and are solely for the use of the intended recipient. If you have received this email in error please contact the sender. Any views or opinions expressed within this e-mail are solely those of the sender, and do not necessarily represent those of 15below unless otherwise specifically stated. Although 15below has taken every reasonable precaution to ensure that any attachment to this e-mail has been checked for viruses, it is strongly recommended that you carry out your own virus check before opening any attachment, as we cannot accept liability for any damage sustained as a result of software virus infection. _______________________________________________ rabbitmq-discuss mailing list rabbitmq-discuss at lists.rabbitmq.com https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss 15below Limited: Company registered in England and Wales No 3945289 Registered Office: Lyndean House, 43-46 Queens Road, Brighton BN1 3XB, United Kingdom 15below Australia Pty Limited: ABN 25 132 716 379 Level 21, Tower 2 Darling Park, 201 Sussex Street, Sydney, NSW 2000, Australia Please think about the environment before printing this email. ************************************************************************ This email and any attachments may be confidential and/or legally privileged and are solely for the use of the intended recipient. If you have received this email in error please contact the sender. Any views or opinions expressed within this e-mail are solely those of the sender, and do not necessarily represent those of 15below unless otherwise specifically stated. Although 15below has taken every reasonable precaution to ensure that any attachment to this e-mail has been checked for viruses, it is strongly recommended that you carry out your own virus check before opening any attachment, as we cannot accept liability for any damage sustained as a result of software virus infection. From arunrao.seattle at gmail.com Mon Apr 14 16:39:29 2014 From: arunrao.seattle at gmail.com (Arun Rao) Date: Mon, 14 Apr 2014 08:39:29 -0700 Subject: [rabbitmq-discuss] Heartbeat vs timeout in .NET code In-Reply-To: References: Message-ID: Hi MK - Are there any use cases when RabbitMQ .NET/Java client code closes connection due to missed heartbeats? Thanks for your help! On Fri, Apr 11, 2014 at 1:51 AM, Michael Klishin wrote: > On 11 April 2014 at 12:50:00, Arun Rao (arunrao.seattle at gmail.com) wrote: > > > My specific question was - When does .NET RabbitMQ client that > > negotiates only Heartbeat argument with the RabbitMQ server > > timeout? Is that again 2 skipped heartbeats as you mentioned > > above? > > After 2 missed heartbeats from server. > > > If so, Timeout value in Management UI for .NET clients > > should be twice the value? > > Sounds like it and I suspect the name ?timeout? wasn?t as clear > as we expected to be. > -- > MK > > Software Engineer, Pivotal/RabbitMQ > _______________________________________________ > rabbitmq-discuss mailing list > rabbitmq-discuss at lists.rabbitmq.com > https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mklishin at gopivotal.com Mon Apr 14 16:46:59 2014 From: mklishin at gopivotal.com (Michael Klishin) Date: Mon, 14 Apr 2014 19:46:59 +0400 Subject: [rabbitmq-discuss] Heartbeat vs timeout in .NET code In-Reply-To: References: Message-ID: On 14 April 2014 at 19:41:44, Arun Rao (arunrao.seattle at gmail.com) wrote: > > Are there any use cases when RabbitMQ .NET/Java client code > closes connection due to missed heartbeats? Every correct client and RabbitMQ must close TCP connection when 2 heartbeats are skipped by the peer.? -- MK Software Engineer, Pivotal/RabbitMQ From stuart.king at me.com Mon Apr 14 18:00:19 2014 From: stuart.king at me.com (Stuart King) Date: Mon, 14 Apr 2014 10:00:19 -0700 Subject: [rabbitmq-discuss] MQTT SSL handshake failures causes server lockup Message-ID: Hi, I?m running RabbitMQ with the MQTT adaptor enabled along with SSL. I?ve discovered that if a small number of connections fail the SSL handshake (e.g. the client rejecting the server?s certificate because it doesn?t recognize the certificate authority) it prevents other SSL connections being established to the server for a short period of time. If a ?bad client? (i.e. one that fails the SSL handshake) keeps trying to reconnect then it effectively locks up the server. A simple way I found to demonstrate this is by creating my own certificate authority, certificates, keys etc for RabbitMQ as per the instructions at https://www.rabbitmq.com/ssl.html and adding the relevant ssl options to the RabbitMQ config file but put the ssl_listeners element into the rabbitmq_mqtt tuple. Then, implement a basic MQTT client in Java using the Paho client libraries. When the client is ran, an exception is thrown as expected since the certificate isn?t trusted. If this client is put into a basic load test, where a new client is created and attempts to connect to the server every 2 seconds, it will prevent other clients connecting. This can be observed by simply using the "openssl s_client? command, which will just hang during the time the Java clients are trying to connect. A sample from the RabbitMQ log file when this occurs is: =ERROR REPORT==== 10-Apr-2014::21:49:33 === SSL: certify: ssl_connection.erl:1724:Fatal error: certificate unknown =ERROR REPORT==== 10-Apr-2014::21:49:38 === ** Generic server <0.689.0> terminating ** Last message in was {inet_async,#Port<0.15200>,48667,{ok,#Port<0.16417>}} ** When Server state == {state, {rabbit_mqtt_sup,start_ssl_client, [[{cacertfile,"/home/sking/ssl/cacert.pem"}, {certfile,"/home/sking/ssl/cert.pem"}, {keyfile,"/home/sking/ssl/key.pem"}, {verify,verify_none}, {fail_if_no_peer_cert,false}]]}, #Port<0.15200>,48667} ** Reason for termination == ** {timeout,{gen_server2,call, [<0.718.0>, {go,#Port<0.16417>, #Fun}]}} =ERROR REPORT==== 10-Apr-2014::21:49:38 === ** Generic server <0.718.0> terminating ** Last message in was {go,#Port<0.16417>,#Fun} ** When Server state == undefined ** Reason for termination == ** {{badmatch,{error,{ssl_upgrade_error,"certificate unknown"}}}, [{rabbit_mqtt_reader,handle_call,3,[]}, {gen_server2,handle_msg,2,[]}, {proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,227}]}]} That particular example is on a server running RabbitMQ 3.1.0 / Erlang R15B03, however I have also reproduced it on the more recent RabbitMQ 3.3.0 / Erlang R16B03-1. I also ran a similar test using AMQP and could not reproduce, so this issue only seems to be with using MQTT over SSL. Regards, Stuart -------------- next part -------------- An HTML attachment was scrubbed... URL: From mklishin at gopivotal.com Mon Apr 14 18:04:03 2014 From: mklishin at gopivotal.com (Michael Klishin) Date: Mon, 14 Apr 2014 21:04:03 +0400 Subject: [rabbitmq-discuss] MQTT SSL handshake failures causes server lockup In-Reply-To: References: Message-ID: On 14 April 2014 at 21:01:43, Stuart King (stuart.king at me.com) wrote: > > That particular example is on a server running RabbitMQ 3.1.0 > / Erlang R15B03, however I have also reproduced it on the more > recent RabbitMQ 3.3.0 / Erlang R16B03-1. > > I also ran a similar test using AMQP and could not reproduce, so > this issue only seems to be with using MQTT over SSL. ? -- MK Software Engineer, Pivotal/RabbitMQ From mklishin at gopivotal.com Mon Apr 14 18:04:35 2014 From: mklishin at gopivotal.com (Michael Klishin) Date: Mon, 14 Apr 2014 21:04:35 +0400 Subject: [rabbitmq-discuss] MQTT SSL handshake failures cause server lockup In-Reply-To: References: Message-ID: On 11 April 2014 at 12:44:58, Michael Klishin (michael at rabbitmq.com) wrote: > > Thanks for reporting, a bug is filed. We?ve fixed this a few days ago: http://hg.rabbitmq.com/rabbitmq-mqtt/rev/4fb81a2c5607 it?s pending QA and should make it into 3.3.1.? -- MK Software Engineer, Pivotal/RabbitMQ From stuart.king at me.com Mon Apr 14 18:11:06 2014 From: stuart.king at me.com (Stuart King) Date: Mon, 14 Apr 2014 10:11:06 -0700 Subject: [rabbitmq-discuss] MQTT SSL handshake failures cause server lockup In-Reply-To: References: Message-ID: <9F20C44A-AD40-42F6-918B-F73BE9651D8C@me.com> Fantastic, thanks. Will look out for the next release. Regards, Stuart On Apr 14, 2014, at 10:04 AM, Michael Klishin wrote: > On 11 April 2014 at 12:44:58, Michael Klishin (michael at rabbitmq.com) wrote: >>> Thanks for reporting, a bug is filed. > > We?ve fixed this a few days ago: > > http://hg.rabbitmq.com/rabbitmq-mqtt/rev/4fb81a2c5607 > > it?s pending QA and should make it into 3.3.1. > -- > MK > > Software Engineer, Pivotal/RabbitMQ > _______________________________________________ > rabbitmq-discuss mailing list > rabbitmq-discuss at lists.rabbitmq.com > https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss From arunrao.seattle at gmail.com Mon Apr 14 20:26:32 2014 From: arunrao.seattle at gmail.com (Arun Rao) Date: Mon, 14 Apr 2014 12:26:32 -0700 Subject: [rabbitmq-discuss] Heartbeat vs timeout in .NET code In-Reply-To: References: Message-ID: Sorry guys for extending this topic further and further. What if I disabled heartbeat? *On the producer side *- When a producer tries to send a message, .NET client code will catch Connection Shutdown exception and try to reinitiate the connection. Depending on the traffic on the connection, this could be caught within a second if the connection was active sending data or within several minutes if the connection was idle. *On the consumer side* - How frequently does the consumer check the queue for messages? I thought that even without heartbeats Consumers will catch the following exception: System.IO.IOException: Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host. ---> System.Net.Sockets.SocketException: An existing connection was forcibly closed by the remote host at System.Net.Sockets.NetworkStream.Read(Byte[] buffer, Int32 offset, Int32 size) --- End of inner exception stack trace --- at RabbitMQ.Client.Impl.Frame.ReadFrom(NetworkBinaryReader reader) at RabbitMQ.Client.Impl.SocketFrameHandler_0_9.ReadFrame() at RabbitMQ.Client.Impl.ConnectionBase.MainLoopIteration() at RabbitMQ.Client.Impl.ConnectionBase.MainLoop() Thanks! On Mon, Apr 14, 2014 at 8:46 AM, Michael Klishin wrote: > On 14 April 2014 at 19:41:44, Arun Rao (arunrao.seattle at gmail.com) wrote: > > > Are there any use cases when RabbitMQ .NET/Java client code > > closes connection due to missed heartbeats? > > Every correct client and RabbitMQ must close TCP connection when 2 > heartbeats > are skipped by the peer. > -- > MK > > Software Engineer, Pivotal/RabbitMQ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From kopias.peter at gmail.com Tue Apr 15 00:23:32 2014 From: kopias.peter at gmail.com (Peter Kopias) Date: Tue, 15 Apr 2014 01:23:32 +0200 Subject: [rabbitmq-discuss] Upgrade fail In-Reply-To: <53453249.90808@rabbitmq.com> References: <533D5DFB.5080405@rabbitmq.com> <53453249.90808@rabbitmq.com> Message-ID: Hi! I've just noticed I had this email unsent, waiting in draft for a few days. So this is just for closing down the thread with "feedback". It seems like, that erlang hangs sometimes, waiting for an empty list of files, with no timeouts... it's probably fixed in version 17. Thank you for all your help, Peter > Yes. The epmd process staying around is not related to anything else > you're seeing, that's how epmd is supposed to work. > > epmd is a very simple daemon which just maps names to port numbers for > Erlang (beam) processes. It doesn't contain any other state. Ok. BTW IMO the "unix way" would be that if I start and stop something, the machine should be in the state it was before. No processes left. :) But I can live with this. :) We all like our systems well behaved and deterministic. :) Node2 and node3 always exited with the message that they are not the >> one stopped the last, and if I removed the locks they tried to connect >> to each other - while repairing the mnesia db, and as the network >> connect failed (as they were not accepting connections probably because >> the db repair was on), they quit, leaving the repair in half. So the >> nodes were not able to communicate because of the db repair (this is a >> theory only), and as the timeout arrived they quit before the repair >> would finish. (It would be nice to have a message like "starting repair" >> and "repair finished", as the log message currently is not clear about >> the state of the repair, I'm not sure that if the repair ends I'll get a >> logitem.) >> > > If you are talking about the upgrades, then > No. No upgrades. It was telling me, that some db files are damaged, and needs to repair them. I'll copy the messages here, but they don't contain too much information about what's the problem. ... > NOTE that, we have the "K20rabbitmq-server stop" running, we have the >> rabbitmq processes too, but currently neither rabbitmqctl nor management >> plugin is accessible, and the reboot process hung. >> > > So this is very weird. The beam process is still around, but everything in > the shutdown appears to have succeeded. > > By the time "Halting Erlang VM" appears in the logs RabbitMQ is completely > stopped, that log message is literally the last thing we do before telling > the VM to stop. > > >> No, it really shouldn't. > > The beam process has already unregistered from epmd (hence "rabbit seems > not to be running at all" in the status command above). > > > The running rabbit processes and their open files are here: >> >> http://pastebin.com/mLKf5mEu >> > > And it seems to have few files open, and to have closed all network > sockets. > > > Any ideas? How can I debug this? >> Specific questions? >> > > I am not full of ideas here, this looks like more of an issue with Erlang > than RabbitMQ. But some things to look into: > > The beam process (2236 in your case) is the interesting one; it's the > thing that should have shut down but hasn't; everything else is pretty much > as expected. Unfortunately although it's an Erlang process it has stopped > listening for Erlang distribution messages so debugging it is likely to be > hard. Having said that, strace might give some clue as to what it's doing, > and the small bright side is that so much within that process has already > shut down that *anything* it is still doing is a good candidate for what's > making it stuck. > > So what does strace say for it? > > CUT STRAIGHT: This is probably the erlang bug that's been fixed in 17, as you've written in the next email. # strace -p 2236 Process 2236 attached select(0, NULL, NULL, NULL, NULL that means: - zero filedescriptors to look for - empty read filedescriptorlist - empty write filedescriptorlist - empty exception filedescriptor list - empty timeout.... from select(2): "If timeout is NULL (no timeout), select() can block indefinitely." This is the cpu-friendly version of while(1); :) I'll try to unbalance this via sending the process signals sigint, sighup, sigterm, sigkill... Let's try: SIGINT (2) ..... result: Connection to ... closed by remote host. (Note: we we're in a shutdown, when the hangup happened :D) So, the problem is a "wait for no files with no timeout", which MAY be the same bug, erlang 17 has fixed, but we don't know for sure. :) If someone founds a hanging rabbit process after stopping, strace it and check whether its a 'select (0,null,null,null', :) Thanks Simon for your help again! Bye, Peter Thank you for your help, I hope these hangups could get eliminated. >> > > I would hope so. We don't have much to go on though. > > > Cheers, Simon > > -- > Simon MacMullen > RabbitMQ, Pivotal > -------------- next part -------------- An HTML attachment was scrubbed... URL: From brandon.munroe at gmail.com Tue Apr 15 00:27:16 2014 From: brandon.munroe at gmail.com (Brandon Munroe) Date: Mon, 14 Apr 2014 17:27:16 -0600 Subject: [rabbitmq-discuss] RabbitMQ Causing Segmentation Fault on FreeBSD Message-ID: We're trying to run a RabbitMQ instance on FreeBSD 10.0 with 10k concurrent connections/queues, and we are experiencing a segmentation fault after letting the system idle for a little while (half an hour or so). During this time, we just connect the 10k clients and pass no messages. Does anyone have any ideas on where we could start looking to diagnose the problem, or has anyone encountered similar problems? Thanks in advance. -Brandon -------------- next part -------------- An HTML attachment was scrubbed... URL: From gavinmroy at gmail.com Tue Apr 15 00:30:48 2014 From: gavinmroy at gmail.com (Gavin M. Roy) Date: Mon, 14 Apr 2014 19:30:48 -0400 Subject: [rabbitmq-discuss] ANN: PostgreSQL Listen Exchange v0.1.0 Message-ID: I am pleased to announce the initial release of the PostgreSQL Listen Exchange.? The exchange allows for non-intrusive publishing from PostgreSQL versions 9.0 or higher by using Postgres?s built-in LISTEN/NOTIFY functionality and should be considered experimental. PostgreSQL connection information can be configured via policy or arguments when declaring the exchange. ? Instructions and a pre-built binary download for RabbitMQ v3.3.0 are available at: https://github.com/aweber/pgsql-listen-exchange#readme Cheers, Gavin -------------- next part -------------- An HTML attachment was scrubbed... URL: From mklishin at gopivotal.com Tue Apr 15 05:39:00 2014 From: mklishin at gopivotal.com (Michael Klishin) Date: Tue, 15 Apr 2014 08:39:00 +0400 Subject: [rabbitmq-discuss] Heartbeat vs timeout in .NET code In-Reply-To: References: Message-ID: On 14 April 2014 at 23:28:43, Arun Rao (arunrao.seattle at gmail.com) wrote: > > Sorry guys for extending this topic further and further. What > if I disabled heartbeat? Then you get ?regular? TCP connection loss detection, which is quite slow (TCP tries to redeliver first). > On the producer side - When a producer tries to send a message, > .NET client code will catch Connection Shutdown exception and > try to reinitiate the connection. Depending on the traffic on > the connection, this could be caught within a second if the connection > was active sending data or within several minutes if the connection > was idle. If the peer is unreachable and did not close TCP connection, then no, it can take well more than a second, depending on TCP stack configuration. > On the consumer side - How frequently does the consumer check > the queue for messages?? Consumers that use basic.consume *never* check for messages. Messages are sent to them by RabbitMQ. > I thought that even without heartbeats > Consumers will catch the following exception: > > System.IO.IOException: Unable to read data from the transport > connection: An existing connection was forcibly closed by the > remote host. ---> System.Net.Sockets.SocketException: An > existing connection was forcibly closed by the remote host This sounds like an exception after a connection was closed via RabbitMQ management plugin. This involves closing TCP connection properly, so the socket is immediately ?marked" as closed by the OS. In this scenario, it?s all good. When either peer becomes unreachable, however, it can take a while (up to multiple minutes is not unheard of) to detect. This is why heartbeats were introduced in the protocol.? -- MK Software Engineer, Pivotal/RabbitMQ From thebrodizzle at gmail.com Tue Apr 15 02:16:17 2014 From: thebrodizzle at gmail.com (Danny Brody) Date: Mon, 14 Apr 2014 18:16:17 -0700 (PDT) Subject: [rabbitmq-discuss] RabbitMQ SSL: certify: tls_handshake.erl:718:Fatal error: handshake failure Message-ID: <6a8a9329-1d31-4f5d-9133-ca40b26f9d89@googlegroups.com> Hi guys, I have looked all over and tried numerous languages to try to get ssl producers working on rabbitmq. Here are the errors i am getting that are in the rabbitmq log file. =ERROR REPORT==== 15-Apr-2014::01:01:56 === SSL: certify: tls_handshake.erl:718:Fatal error: handshake failure =ERROR REPORT==== 15-Apr-2014::01:02:01 === error on AMQP connection <0.31263.4>: {ssl_upgrade_error,{tls_alert,[104,97,110,100,115,104,97,107,101,32,102,97,105,108,117,114,101]}}... On the client side with nodejs i get a pretty descriptive error output: node app.js [Error: 140489250436928:error:14094410:SSL routines:SSL3_READ_BYTES:sslv3 alert handshake failure:../deps/openssl/openssl/ssl/s3_pkt.c:1256:SSL alert number 40 ] with php i get: Message: Error Connecting to server(0): Anyways Im trying to produce messages to a rabbitmq server and i generated the certificates properly and the publishers are working with java w/ spring framework, but i wanted to benchmark the performance with nodejs and php, so a simple producer script is all i need. BTW: everything worked fine with regular producing code but the issues happened once ssl was added to the system. I do i have a self signed certificate with a key and cert file. openssl s_client -msg -connect xxx.xxx.xxx.xxx:5673 -cert /home/ubuntu/dannyTest/ssl/server/cert.pem -key /home/ubuntu/dannyTest/ssl/server/key.pem -CAfile /home/ubuntu/dannyTest/ssl/ca/cacert.pem gives the following output with two errors being: 140478781478560:error:14094410:SSL routines:SSL3_READ_BYTES:sslv3 alert handshake failure:s3_pkt.c:1256:SSL alert number 40 140478781478560:error:140790E5:SSL routines:SSL23_WRITE:ssl handshake failure:s23_lib.c:177: CONNECTED(00000003) >>> TLS 1.1 [length 00dc] 01 00 00 d8 03 02 53 4c 87 7a 0b 1c b4 67 8e 4b 55 64 fe 11 ee 57 99 55 78 ec f7 d4 b0 6d 7d 3d d0 ed a9 b2 1f cf 00 00 66 c0 14 c0 0a c0 22 c0 21 00 39 00 38 00 88 00 87 c0 0f c0 05 00 35 00 84 c0 12 c0 08 c0 1c c0 1b 00 16 00 13 c0 0d c0 03 00 0a c0 13 c0 09 c0 1f c0 1e 00 33 00 32 00 9a 00 99 00 45 00 44 c0 0e c0 04 00 2f 00 96 00 41 c0 11 c0 07 c0 0c c0 02 00 05 00 04 00 15 00 12 00 09 00 14 00 11 00 08 00 06 00 03 00 ff 01 00 00 49 00 0b 00 04 03 00 01 02 00 0a 00 34 00 32 00 0e 00 0d 00 19 00 0b 00 0c 00 18 00 09 00 0a 00 16 00 17 00 08 00 06 00 07 00 14 00 15 00 04 00 05 00 12 00 13 00 01 00 02 00 03 00 0f 00 10 00 11 00 23 00 00 00 0f 00 01 01 <<< TLS 1.1 [length 0051] 02 00 00 4d 03 02 53 4c 87 7a b1 28 bc d3 24 ce 51 56 4a 38 81 68 b7 24 4d cb 9f bf a6 1a 76 00 99 c0 a1 6d 50 32 20 e5 6b 0f 75 f0 66 49 b0 8b 25 88 70 71 19 70 ed d7 a2 06 42 d5 3e fd 66 f8 f9 5c f7 c9 31 f2 34 00 35 00 00 05 ff 01 00 01 00 <<< TLS 1.1 [length 05b5] 0b 00 05 b1 00 05 ae 00 02 e8 30 82 02 e4 30 82 01 cc a0 03 02 01 02 02 01 01 30 0d 06 09 2a 86 48 86 f7 0d 01 01 05 05 00 30 0e 31 0c 30 0a 06 03 55 04 03 13 03 41 43 52 30 1e 17 0d 31 34 30 33 32 38 31 33 35 38 33 33 5a 17 0d 31 35 30 33 32 38 31 33 35 38 33 33 5a 30 2c 31 19 30 17 06 03 55 04 03 13 10 69 70 2d 31 30 2d 31 38 35 2d 34 36 2d 31 32 30 31 0f 30 0d 06 03 55 04 0a 13 06 73 65 72 76 65 72 30 82 01 22 30 0d 06 09 2a 86 48 86 f7 0d 01 01 01 05 00 03 82 01 0f 00 30 82 01 0a 02 82 01 01 00 b3 e4 6e 5a 91 62 b9 bd 73 80 3b f5 4b 81 0a eb 35 cf 38 ce 1b 64 bb 51 2d 0c bb 2c 27 fc 9f c0 48 5a 09 3d ed a7 7c 4b bc 41 12 26 89 62 d5 8e 29 43 23 b0 2a 9e d1 c3 06 23 e1 4f 40 0c e5 4e 6b 38 af 25 22 ae c6 52 9f a5 5a 24 cb 77 ab cd 76 0d a1 29 6b 0c bb 14 95 90 2a d9 60 a3 35 de 6e 2a 6e 46 f2 c7 55 1a e8 34 e4 cb af 33 2d a8 de 1a 67 75 6a 3e 33 45 ba 6d c4 4d df 17 79 0a a9 bb 5e fc 46 20 a5 35 e1 7a 6d b2 95 ab 1c 1d 01 98 54 94 91 65 ed 68 2d f0 95 58 e5 e3 3d 7d 63 20 b5 77 aa 5e 3f fd 04 a1 f4 86 fe d7 33 66 29 98 d4 fb 0d f8 1c fc 01 e3 98 be 7c 47 7e eb b5 7e fc 35 54 ac f9 47 11 07 4b f5 90 46 27 c9 56 0a ce 57 45 5a 4c c8 d2 dd d1 6d 30 78 99 62 68 23 97 a9 dc 20 24 1c 8d 09 b9 fb dc 53 ee c1 11 80 4d 93 a2 2d 33 27 ab 52 1d 40 c2 1f dc 29 02 03 01 00 01 a3 2f 30 2d 30 09 06 03 55 1d 13 04 02 30 00 30 0b 06 03 55 1d 0f 04 04 03 02 05 20 30 13 06 03 55 1d 25 04 0c 30 0a 06 08 2b 06 01 05 05 07 03 01 30 0d 06 09 2a 86 48 86 f7 0d 01 01 05 05 00 03 82 01 01 00 26 f2 67 73 14 ce 16 7a aa 00 28 a6 0d 45 48 be f0 b9 d2 d7 f7 20 c4 7d 48 7e 90 af 43 0f b5 84 76 1c e4 eb a2 15 3f aa 53 63 a8 69 c7 47 df a7 00 06 0f 06 22 d0 2d 36 a5 3c 76 ce 79 03 0f 75 cc e8 df f5 fc 88 c2 b3 b3 f7 02 c0 4d 20 2d 16 91 a2 33 e6 f1 4a ee b0 3c d2 5f 1d 60 b4 83 b6 ec 3f cc 75 25 6b 5a 86 24 ad d1 8d 9f d7 9e 45 b5 49 f7 3c 89 41 63 7f a8 5b a1 90 9d e9 59 ea bb 3a af 0d 40 ab 8e f9 4f 70 be 79 87 ee b5 91 76 62 89 c2 8f 37 c6 66 00 52 ac 4b bf 65 51 7d 05 91 a5 8a ef d6 3f e0 52 4a 9e 07 cd 1a 62 b8 58 6e d9 ca 5a a2 de b9 ff 49 e2 08 f9 22 ed 54 30 f7 f3 5f bd 41 2e f9 81 28 67 aa b9 18 25 cb 7e 86 9a 77 56 f0 29 45 a2 2e 77 b5 ff 01 5c 4a 40 62 40 02 c7 72 62 90 9a e4 be c6 13 4d 8f ba 44 99 85 e6 dc dc 7b 92 ae e8 21 37 d1 90 dc 88 00 02 c0 30 82 02 bc 30 82 01 a4 a0 03 02 01 02 02 09 00 a0 b9 e9 91 cb d4 f2 3c 30 0d 06 09 2a 86 48 86 f7 0d 01 01 05 05 00 30 0e 31 0c 30 0a 06 03 55 04 03 13 03 41 43 52 30 1e 17 0d 31 34 30 33 32 38 31 33 35 35 30 31 5a 17 0d 31 35 30 33 32 38 31 33 35 35 30 31 5a 30 0e 31 0c 30 0a 06 03 55 04 03 13 03 41 43 52 30 82 01 22 30 0d 06 09 2a 86 48 86 f7 0d 01 01 01 05 00 03 82 01 0f 00 30 82 01 0a 02 82 01 01 00 aa 39 16 64 fb 96 41 8e 95 8e e6 9f c5 01 37 b2 63 1f b2 c4 17 aa 33 d8 b1 5f 5e 2e 93 ce 3a 7a a0 db 58 29 df ad 21 46 20 5c 55 c6 91 5b 5f 6f 43 3c 21 5a 17 31 02 d5 a6 97 dd 27 59 7b 4e 1d d7 db 92 88 fb 30 93 eb 44 70 be 7d 57 69 4a 98 58 fc ad a1 b4 67 b1 07 8d 81 a8 13 d5 00 cc c7 4a 46 33 4b e1 30 00 90 d3 75 da 55 1a 82 15 9c 28 73 90 d7 de 41 e6 bc 05 e7 1b 3d c3 8c cb 67 2d d5 4b 2b f8 f2 ef 64 d8 e0 35 e8 8c 38 5e 05 67 74 ea 94 5c 2b 74 83 76 df 31 60 d7 eb b1 25 2c 29 9f 6a da c0 08 0e df 4f 2b 1b fd 5c 93 08 ee 5b b6 e4 66 70 f8 3f 8d 76 f6 2b ce 8a 89 42 7b 21 21 36 a8 c0 22 f2 4e b2 6f a9 d5 33 5f 38 3a f0 9a 57 37 66 a2 6b 73 76 41 75 4f b8 ad 14 ae a3 97 77 81 1d 1f 65 7e 54 0b ae 99 b2 a0 31 e7 9c 84 c5 a3 de 18 10 05 5c 83 c1 14 f2 5d 75 02 03 01 00 01 a3 1d 30 1b 30 0c 06 03 55 1d 13 04 05 30 03 01 01 ff 30 0b 06 03 55 1d 0f 04 04 03 02 01 06 30 0d 06 09 2a 86 48 86 f7 0d 01 01 05 05 00 03 82 01 01 00 35 1c e0 6f fc 31 46 d2 97 f0 95 18 2f 4b 93 0c cf 5e 09 d5 d2 3f cf 5b e0 2b 8a 76 21 59 3a c7 ba 2d 89 64 c0 dd bb ce 3c e8 20 28 5b 50 3c c4 3a eb 95 d4 5f d4 c2 d5 cd 80 fc cd 14 79 1f c8 f9 7a c6 79 f9 ad 1f a1 47 f7 4c a7 d9 d0 07 79 dd c3 e8 32 df 03 c7 39 72 7f f6 37 48 72 1d f9 3e 33 6d d5 01 04 35 3e c5 30 d4 c7 ee c5 a8 49 b0 53 87 23 a5 ec 0d ae 79 1e 87 c3 b6 e2 a4 15 ac 4e 1e f8 f4 b8 a5 4f 6a b2 a8 78 6c 04 ac c2 30 ff 02 47 e5 c7 be df 7e aa eb 60 14 19 96 38 fc 03 45 a7 38 36 2d d3 9e 67 c9 f5 eb 4f 34 44 fb 7d 2f 57 7f 6d 2d 4f 13 f7 8c ac a1 1b 14 0d 47 8b fc 4f 3b 0f f9 4c 69 41 4b 57 65 a2 fe 31 1d 4c 1b 2d 9c a7 f2 c6 50 1d b3 34 f4 89 33 1d 1a ad 58 14 eb e5 16 e1 74 9f d4 8d de 5b 27 6b 5f 64 dc b0 a1 70 33 4b f3 b7 df ae 0e 8a 1a 4a depth=1 CN = ACR verify return:1 depth=0 CN = XXX.XXX.XXX.XXX, O = server verify return:1 <<< TLS 1.1 [length 001a] 0d 00 00 16 01 01 00 12 00 10 30 0e 31 0c 30 0a 06 03 55 04 03 13 03 41 43 52 <<< TLS 1.1 [length 0004] 0e 00 00 00 >>> TLS 1.1 [length 05b5] 0b 00 05 b1 00 05 ae 00 02 e8 30 82 02 e4 30 82 01 cc a0 03 02 01 02 02 01 01 30 0d 06 09 2a 86 48 86 f7 0d 01 01 05 05 00 30 0e 31 0c 30 0a 06 03 55 04 03 13 03 41 43 52 30 1e 17 0d 31 34 30 33 32 38 31 33 35 38 33 33 5a 17 0d 31 35 30 33 32 38 31 33 35 38 33 33 5a 30 2c 31 19 30 17 06 03 55 04 03 13 10 69 70 2d 31 30 2d 31 38 35 2d 34 36 2d 31 32 30 31 0f 30 0d 06 03 55 04 0a 13 06 73 65 72 76 65 72 30 82 01 22 30 0d 06 09 2a 86 48 86 f7 0d 01 01 01 05 00 03 82 01 0f 00 30 82 01 0a 02 82 01 01 00 b3 e4 6e 5a 91 62 b9 bd 73 80 3b f5 4b 81 0a eb 35 cf 38 ce 1b 64 bb 51 2d 0c bb 2c 27 fc 9f c0 48 5a 09 3d ed a7 7c 4b bc 41 12 26 89 62 d5 8e 29 43 23 b0 2a 9e d1 c3 06 23 e1 4f 40 0c e5 4e 6b 38 af 25 22 ae c6 52 9f a5 5a 24 cb 77 ab cd 76 0d a1 29 6b 0c bb 14 95 90 2a d9 60 a3 35 de 6e 2a 6e 46 f2 c7 55 1a e8 34 e4 cb af 33 2d a8 de 1a 67 75 6a 3e 33 45 ba 6d c4 4d df 17 79 0a a9 bb 5e fc 46 20 a5 35 e1 7a 6d b2 95 ab 1c 1d 01 98 54 94 91 65 ed 68 2d f0 95 58 e5 e3 3d 7d 63 20 b5 77 aa 5e 3f fd 04 a1 f4 86 fe d7 33 66 29 98 d4 fb 0d f8 1c fc 01 e3 98 be 7c 47 7e eb b5 7e fc 35 54 ac f9 47 11 07 4b f5 90 46 27 c9 56 0a ce 57 45 5a 4c c8 d2 dd d1 6d 30 78 99 62 68 23 97 a9 dc 20 24 1c 8d 09 b9 fb dc 53 ee c1 11 80 4d 93 a2 2d 33 27 ab 52 1d 40 c2 1f dc 29 02 03 01 00 01 a3 2f 30 2d 30 09 06 03 55 1d 13 04 02 30 00 30 0b 06 03 55 1d 0f 04 04 03 02 05 20 30 13 06 03 55 1d 25 04 0c 30 0a 06 08 2b 06 01 05 05 07 03 01 30 0d 06 09 2a 86 48 86 f7 0d 01 01 05 05 00 03 82 01 01 00 26 f2 67 73 14 ce 16 7a aa 00 28 a6 0d 45 48 be f0 b9 d2 d7 f7 20 c4 7d 48 7e 90 af 43 0f b5 84 76 1c e4 eb a2 15 3f aa 53 63 a8 69 c7 47 df a7 00 06 0f 06 22 d0 2d 36 a5 3c 76 ce 79 03 0f 75 cc e8 df f5 fc 88 c2 b3 b3 f7 02 c0 4d 20 2d 16 91 a2 33 e6 f1 4a ee b0 3c d2 5f 1d 60 b4 83 b6 ec 3f cc 75 25 6b 5a 86 24 ad d1 8d 9f d7 9e 45 b5 49 f7 3c 89 41 63 7f a8 5b a1 90 9d e9 59 ea bb 3a af 0d 40 ab 8e f9 4f 70 be 79 87 ee b5 91 76 62 89 c2 8f 37 c6 66 00 52 ac 4b bf 65 51 7d 05 91 a5 8a ef d6 3f e0 52 4a 9e 07 cd 1a 62 b8 58 6e d9 ca 5a a2 de b9 ff 49 e2 08 f9 22 ed 54 30 f7 f3 5f bd 41 2e f9 81 28 67 aa b9 18 25 cb 7e 86 9a 77 56 f0 29 45 a2 2e 77 b5 ff 01 5c 4a 40 62 40 02 c7 72 62 90 9a e4 be c6 13 4d 8f ba 44 99 85 e6 dc dc 7b 92 ae e8 21 37 d1 90 dc 88 00 02 c0 30 82 02 bc 30 82 01 a4 a0 03 02 01 02 02 09 00 a0 b9 e9 91 cb d4 f2 3c 30 0d 06 09 2a 86 48 86 f7 0d 01 01 05 05 00 30 0e 31 0c 30 0a 06 03 55 04 03 13 03 41 43 52 30 1e 17 0d 31 34 30 33 32 38 31 33 35 35 30 31 5a 17 0d 31 35 30 33 32 38 31 33 35 35 30 31 5a 30 0e 31 0c 30 0a 06 03 55 04 03 13 03 41 43 52 30 82 01 22 30 0d 06 09 2a 86 48 86 f7 0d 01 01 01 05 00 03 82 01 0f 00 30 82 01 0a 02 82 01 01 00 aa 39 16 64 fb 96 41 8e 95 8e e6 9f c5 01 37 b2 63 1f b2 c4 17 aa 33 d8 b1 5f 5e 2e 93 ce 3a 7a a0 db 58 29 df ad 21 46 20 5c 55 c6 91 5b 5f 6f 43 3c 21 5a 17 31 02 d5 a6 97 dd 27 59 7b 4e 1d d7 db 92 88 fb 30 93 eb 44 70 be 7d 57 69 4a 98 58 fc ad a1 b4 67 b1 07 8d 81 a8 13 d5 00 cc c7 4a 46 33 4b e1 30 00 90 d3 75 da 55 1a 82 15 9c 28 73 90 d7 de 41 e6 bc 05 e7 1b 3d c3 8c cb 67 2d d5 4b 2b f8 f2 ef 64 d8 e0 35 e8 8c 38 5e 05 67 74 ea 94 5c 2b 74 83 76 df 31 60 d7 eb b1 25 2c 29 9f 6a da c0 08 0e df 4f 2b 1b fd 5c 93 08 ee 5b b6 e4 66 70 f8 3f 8d 76 f6 2b ce 8a 89 42 7b 21 21 36 a8 c0 22 f2 4e b2 6f a9 d5 33 5f 38 3a f0 9a 57 37 66 a2 6b 73 76 41 75 4f b8 ad 14 ae a3 97 77 81 1d 1f 65 7e 54 0b ae 99 b2 a0 31 e7 9c 84 c5 a3 de 18 10 05 5c 83 c1 14 f2 5d 75 02 03 01 00 01 a3 1d 30 1b 30 0c 06 03 55 1d 13 04 05 30 03 01 01 ff 30 0b 06 03 55 1d 0f 04 04 03 02 01 06 30 0d 06 09 2a 86 48 86 f7 0d 01 01 05 05 00 03 82 01 01 00 35 1c e0 6f fc 31 46 d2 97 f0 95 18 2f 4b 93 0c cf 5e 09 d5 d2 3f cf 5b e0 2b 8a 76 21 59 3a c7 ba 2d 89 64 c0 dd bb ce 3c e8 20 28 5b 50 3c c4 3a eb 95 d4 5f d4 c2 d5 cd 80 fc cd 14 79 1f c8 f9 7a c6 79 f9 ad 1f a1 47 f7 4c a7 d9 d0 07 79 dd c3 e8 32 df 03 c7 39 72 7f f6 37 48 72 1d f9 3e 33 6d d5 01 04 35 3e c5 30 d4 c7 ee c5 a8 49 b0 53 87 23 a5 ec 0d ae 79 1e 87 c3 b6 e2 a4 15 ac 4e 1e f8 f4 b8 a5 4f 6a b2 a8 78 6c 04 ac c2 30 ff 02 47 e5 c7 be df 7e aa eb 60 14 19 96 38 fc 03 45 a7 38 36 2d d3 9e 67 c9 f5 eb 4f 34 44 fb 7d 2f 57 7f 6d 2d 4f 13 f7 8c ac a1 1b 14 0d 47 8b fc 4f 3b 0f f9 4c 69 41 4b 57 65 a2 fe 31 1d 4c 1b 2d 9c a7 f2 c6 50 1d b3 34 f4 89 33 1d 1a ad 58 14 eb e5 16 e1 74 9f d4 8d de 5b 27 6b 5f 64 dc b0 a1 70 33 4b f3 b7 df ae 0e 8a 1a 4a >>> TLS 1.1 [length 0106] 10 00 01 02 01 00 ab 94 c0 47 a6 53 2e 31 76 11 72 aa b6 97 f4 fe 15 e2 57 bc cd 4f 64 33 0e 20 57 73 80 58 bb 7d 9c 93 79 29 40 96 81 f4 3f 5e 72 ec 48 3e c8 c0 43 7b a5 95 ed b7 78 05 1b e5 61 c6 79 d6 3d d2 f1 8d 56 ff 69 10 7d 0e c3 aa 8d f4 e2 02 54 7e 09 47 0e 56 9e db 29 74 30 36 14 11 02 e5 6a b2 12 69 c4 76 70 55 a1 d5 83 4f 2e 96 4f a6 4e de bb fe 9c 7b d4 19 13 d0 4e 11 06 53 3f 6d 2a 47 7d b0 aa 2e 76 87 f5 12 a1 af af a9 bb c6 0f 5f 5c a5 48 8c 56 3a de 77 b2 97 f2 5d f8 ec 8b fa 8d 9b b2 94 89 0d 74 b3 07 58 5e 21 6e dd c6 fd 12 04 3a 11 28 c4 f5 92 30 ec 02 c6 b0 e0 84 8a b3 b2 d8 97 5d 66 5e 42 f0 01 c2 47 c6 e3 57 45 ff 93 a0 b6 b2 1c 1d e2 16 fd 2d 85 cc 90 c2 a1 b8 4a 0a 27 3a 36 4f ec 36 d7 28 ef 39 c4 e7 a5 db e6 72 0f 49 50 a5 05 19 ba a3 fb 13 60 a8 4f >>> TLS 1.1 [length 0106] 0f 00 01 02 01 00 85 43 26 0d a5 b1 92 b5 d6 1b 6d ef d7 e1 e1 91 a9 7d 42 09 cc 52 cd 1d 54 3b 9a 86 ea 36 b4 b0 46 30 15 ec d4 4c c4 fd b4 d7 93 6e dd be a5 4f 05 65 70 04 7c 83 5a 25 07 bf 58 35 41 58 18 e7 3e 90 6c 7a d2 4b 8c e6 80 6f f5 34 d1 82 72 f6 3c ae b0 ee df 2c 63 8b 73 5e 66 6c 0d dd 8f cd 85 3f 6a 2d ef 74 73 cf 34 4f 91 11 7e ee 68 a1 de 4c 73 c1 35 82 a7 8e 79 1c 56 ba ea 6c 2c 3f fb e9 74 28 96 71 7d d0 16 5d 1d 8f 65 32 aa 2a 70 03 e9 7c ae 6d d4 bc e3 86 d6 5c 2b 6e 46 6f 2a 53 2a fe 24 a2 f3 88 12 68 89 fe 89 fe 3f ed ce dd f1 89 e5 dc fb 09 29 d3 4c c2 38 de aa 71 29 3c ee b4 b0 3d 31 0c 2a 1c d2 fb a1 b1 d2 50 cd 94 d4 64 af f4 9e e1 df 87 fb 06 96 ca a6 f0 18 b7 85 a5 70 51 26 9a 18 5a b0 4e f9 e0 d3 de d2 32 10 03 ab b9 9d 4a ce 6f 8a 3c a7 d4 85 fe >>> TLS 1.1 [length 0001] 01 >>> TLS 1.1 [length 0010] 14 00 00 0c 19 93 67 da 00 96 95 41 a1 00 8b 11 <<< TLS 1.1 [length 0002] 02 28 140478781478560:error:14094410:SSL routines:SSL3_READ_BYTES:sslv3 alert handshake failure:s3_pkt.c:1256:SSL alert number 40 140478781478560:error:140790E5:SSL routines:SSL23_WRITE:ssl handshake failure:s23_lib.c:177: --- Certificate chain 0 s:/CN=XXX.XXX.XXX.XXX/O=server i:/CN=ACR 1 s:/CN=ACR i:/CN=ACR --- Server certificate -----BEGIN CERTIFICATE----- ---------------------REMOVING CERTIFICATE TEXT FOR SECURITY REASONS JUST IN CASE ----------------- -----END CERTIFICATE----- subject=/CN=XXX.XXX.XXX.XXX/O=server issuer=/CN=ACR --- Acceptable client certificate CA names /CN=ACR --- SSL handshake has read 1599 bytes and written 2075 bytes --- New, TLSv1/SSLv3, Cipher is AES256-SHA Server public key is 2048 bit Secure Renegotiation IS supported Compression: NONE Expansion: NONE SSL-Session: Protocol : TLSv1.1 Cipher : AES256-SHA Session-ID: E56B0F75F06649B08B258870711970EDD7A20642D53EFD66F8F95CF7C931F234 Session-ID-ctx: Master-Key: 662A3A97A8F3F9BC08F7D62AD8E05BE6747963FD36B5653D7D37035C8658EC524100C9CEFFB2D3C29A825DAB18DB3C22 Key-Arg : None PSK identity: None PSK identity hint: None SRP username: None Start Time: 1397524346 Timeout : 300 (sec) Verify return code: 0 (ok) --- I have scoured google and tried all i could. All help is appreciated! -------------- next part -------------- An HTML attachment was scrubbed... URL: From juno at algorithmictradinggroup.com Tue Apr 15 03:39:53 2014 From: juno at algorithmictradinggroup.com (AJ) Date: Mon, 14 Apr 2014 19:39:53 -0700 (PDT) Subject: [rabbitmq-discuss] Ready, Unacked and Total messages of Queues Message-ID: <178d6430-4bd2-463e-8b4f-29bd3d532283@googlegroups.com> Hi all, I am a newbie to RabbitMQ. Would anyone please kindly explain more on the 'Ready', 'Unacked' and 'Total' messages of a queue? I have a number of .NET applications communicating to each other with RabbitMQ. Unfortunately, randomly happens, some of the queues are having high # in Total and Unacked, and some of the queues are having high# in Ready and Total. I do not know under what situations will the queues having these 2 different situations? So that I can trace to the application code and see if there are anything wrong with the code. This may be a very trivial problems to you all, yet I cannot find any relevant documentations on it, please kindly let me know if there are any. Thanks very much! -------------- next part -------------- An HTML attachment was scrubbed... URL: From mklishin at gopivotal.com Tue Apr 15 07:01:00 2014 From: mklishin at gopivotal.com (Michael Klishin) Date: Tue, 15 Apr 2014 10:01:00 +0400 Subject: [rabbitmq-discuss] Ready, Unacked and Total messages of Queues In-Reply-To: <178d6430-4bd2-463e-8b4f-29bd3d532283@googlegroups.com> References: <178d6430-4bd2-463e-8b4f-29bd3d532283@googlegroups.com> Message-ID: On 15 April 2014 at 09:52:51, AJ (juno at algorithmictradinggroup.com) wrote: > > Would anyone please kindly explain more on the 'Ready', 'Unacked' > and 'Total' messages of a queue? > I have a number of .NET applications communicating to each other > with RabbitMQ. > Unfortunately, randomly happens, some of the queues are having > high # in Total and Unacked, and some of the queues are having high# > in Ready and Total. > I do not know under what situations will the queues having these > 2 different situations? So that I can trace to the application > code and see if there are anything wrong with the code. Messages delivered by RabbitMQ to consumers can be acknowledged automatically (as soon as they are sent out) or manually using IModel#BasicAck. You decide which mode to use. Unacknowledged messages are delivered but not yet acknowledged. If the consumer(s) processing them fails, the messages will be requeued. ?Ready? messages are not delivered and simply sitting in a queue. Total = ready + unacknowledged.? See?http://www.rabbitmq.com/tutorials/tutorial-two-dotnet.html. -- MK Software Engineer, Pivotal/RabbitMQ From mklishin at gopivotal.com Tue Apr 15 07:06:19 2014 From: mklishin at gopivotal.com (Michael Klishin) Date: Tue, 15 Apr 2014 10:06:19 +0400 Subject: [rabbitmq-discuss] RabbitMQ SSL: certify: tls_handshake.erl:718:Fatal error: handshake failure In-Reply-To: <6a8a9329-1d31-4f5d-9133-ca40b26f9d89@googlegroups.com> References: <6a8a9329-1d31-4f5d-9133-ca40b26f9d89@googlegroups.com> Message-ID: On 15 April 2014 at 09:51:43, Danny Brody (thebrodizzle at gmail.com) wrote: > > =ERROR REPORT==== 15-Apr-2014(http://airmail.calendar/2014-04-15%2012:00:00%20GMT+4)::01:01:56 > === > SSL: certify: tls_handshake.erl:718:Fatal error: handshake > failure > > =ERROR REPORT==== 15-Apr-2014(http://airmail.calendar/2014-04-15%2012:00:00%20GMT+4)::01:02:01 > === > error on AMQP connection <0.31263.4>: {ssl_upgrade_error,{tls_alert,[104,97,110,100,115,104,97,107,101,32,102,97,105,108,117,114,101]}}... > > > On the client side with nodejs i get a pretty descriptive error > output: > node app.js > [Error: 140489250436928:error:14094410:SSL routines:SSL3_READ_BYTES:sslv3 > alert handshake failure:../deps/openssl/openssl/ssl/s3_pkt.c:1256:SSL > alert number 40 > ] > > with php i get: > Message: Error Connecting to server(0): I suspect there are no ciphers that client and server can agree on. Server seems to select?AES256-SHA, can you try connecting to openssl s_server to see what the client supports? Also, what OpenSSL, PHP and Erlang versions do you run?? -- MK Software Engineer, Pivotal/RabbitMQ From matthias at rabbitmq.com Tue Apr 15 08:23:32 2014 From: matthias at rabbitmq.com (Matthias Radestock) Date: Tue, 15 Apr 2014 08:23:32 +0100 Subject: [rabbitmq-discuss] Ready, Unacked and Total messages of Queues In-Reply-To: References: <178d6430-4bd2-463e-8b4f-29bd3d532283@googlegroups.com> Message-ID: <534CDE74.60201@rabbitmq.com> On 15/04/14 07:01, Michael Klishin wrote: > On 15 April 2014 at 09:52:51, AJ (juno at algorithmictradinggroup.com) wrote: >>> Would anyone please kindly explain more on the 'Ready', 'Unacked' >> and 'Total' messages of a queue? > > Messages delivered by RabbitMQ to consumers can be acknowledged automatically > (as soon as they are sent out) or manually using IModel#BasicAck. > > ?Ready? messages are not delivered and simply sitting in a queue. > Total = ready + unacknowledged. Btw, the meaning of these is explained in the management UI's built-in help - click on the '?' above the chart. Matthias. From juno at algorithmictradinggroup.com Tue Apr 15 08:35:47 2014 From: juno at algorithmictradinggroup.com (Juno Chan) Date: Tue, 15 Apr 2014 15:35:47 +0800 Subject: [rabbitmq-discuss] Ready, Unacked and Total messages of Queues In-Reply-To: <534CDE74.60201@rabbitmq.com> References: <178d6430-4bd2-463e-8b4f-29bd3d532283@googlegroups.com> <534CDE74.60201@rabbitmq.com> Message-ID: Hi Matthias, thanks very much. So, for the queues having high values in ready& total # of messages(even go up to 300K messages), rabbit MQ performance won't get affected? only those queues having high values in unacked and total would harm the rabbitMQ performance, is that correct? BTW, for 2 applications subscribing to the same subscribe channel, 2 separate queues are created, if for some reasons, one of them has problems and cause the queue stuck (e.g. high # unack messages) will another queue being affected? Thanks very much On Tue, Apr 15, 2014 at 3:23 PM, Matthias Radestock wrote: > On 15/04/14 07:01, Michael Klishin wrote: > >> On 15 April 2014 at 09:52:51, AJ (juno at algorithmictradinggroup.com) >> wrote: >> >>> Would anyone please kindly explain more on the 'Ready', 'Unacked' >>>> >>> and 'Total' messages of a queue? >>> >> >> Messages delivered by RabbitMQ to consumers can be acknowledged >> automatically >> (as soon as they are sent out) or manually using IModel#BasicAck. >> >> ?Ready? messages are not delivered and simply sitting in a queue. >> Total = ready + unacknowledged. >> > > Btw, the meaning of these is explained in the management UI's built-in > help - click on the '?' above the chart. > > Matthias. > -- Juno Chan Algorithmic Trading Group (ATG) Ltd www.algorithmictradinggroup.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From mklishin at gopivotal.com Tue Apr 15 08:42:30 2014 From: mklishin at gopivotal.com (Michael Klishin) Date: Tue, 15 Apr 2014 11:42:30 +0400 Subject: [rabbitmq-discuss] Ready, Unacked and Total messages of Queues In-Reply-To: References: <178d6430-4bd2-463e-8b4f-29bd3d532283@googlegroups.com> <534CDE74.60201@rabbitmq.com> Message-ID: On 15 April 2014 at 11:37:39, Juno Chan (juno at algorithmictradinggroup.com) wrote: > > thanks very much. So, for the queues having high values in ready& > total # of messages(even go up to 300K messages), rabbit MQ performance > won't get affected? > only those queues having high values in unacked and total would > harm the rabbitMQ performance, is that correct? Unacknowledged messages are still kept around by RabbitMQ until they are acknowledged or rejected, so it?s the total number of messages that really ?matters. > BTW, for 2 applications subscribing to the same subscribe channel, > 2 separate queues are created, if for some reasons, one of them > has problems and cause the queue stuck (e.g. high # unack messages) > will another queue being affected? A single queue can only use a single CPU core. The number of messages in the longest matter a lot less compared to how much disk I/O RabbitMQ has to do, either due to memory pressure or a high rate of messages published as persistent and routed to durable queues. Avoid overloaded queues as much as possible. See also http://www.rabbitmq.com/blog/2014/01/23/preventing-unbounded-buffers-with-rabbitmq/ http://www.rabbitmq.com/blog/2014/04/10/consumer-bias-in-rabbitmq-3-3/ http://www.rabbitmq.com/blog/2014/04/14/finding-bottlenecks-with-rabbitmq-3-3/ -- MK Software Engineer, Pivotal/RabbitMQ From juno at algorithmictradinggroup.com Tue Apr 15 08:55:45 2014 From: juno at algorithmictradinggroup.com (Juno Chan) Date: Tue, 15 Apr 2014 15:55:45 +0800 Subject: [rabbitmq-discuss] Ready, Unacked and Total messages of Queues In-Reply-To: References: <178d6430-4bd2-463e-8b4f-29bd3d532283@googlegroups.com> <534CDE74.60201@rabbitmq.com> Message-ID: thanks very much MK So it's good to hear that #ready msgs high is not impacting RabbitMQ performance. because we encountered an issues like the total messages in rabbit was up to 300K, while some applications have queues of extremely high #ready, but no #unack, while some applications have queues of high #unack, but no #ready. thats helps me to troubleshoot which application I should look into. Thanks. Cheers On Tue, Apr 15, 2014 at 3:42 PM, Michael Klishin wrote: > On 15 April 2014 at 11:37:39, Juno Chan (juno at algorithmictradinggroup.com) > wrote: > > > thanks very much. So, for the queues having high values in ready& > > total # of messages(even go up to 300K messages), rabbit MQ performance > > won't get affected? > > only those queues having high values in unacked and total would > > harm the rabbitMQ performance, is that correct? > > Unacknowledged messages are still kept around by RabbitMQ until they are > acknowledged > or rejected, so it?s the total number of messages that really matters. > > > BTW, for 2 applications subscribing to the same subscribe channel, > > 2 separate queues are created, if for some reasons, one of them > > has problems and cause the queue stuck (e.g. high # unack messages) > > will another queue being affected? > > A single queue can only use a single CPU core. The number of messages in > the longest > matter a lot less compared to how much disk I/O RabbitMQ has to do, either > due to > memory pressure or a high rate of messages published as persistent and > routed to > durable queues. > > Avoid overloaded queues as much as possible. See also > > > http://www.rabbitmq.com/blog/2014/01/23/preventing-unbounded-buffers-with-rabbitmq/ > http://www.rabbitmq.com/blog/2014/04/10/consumer-bias-in-rabbitmq-3-3/ > > http://www.rabbitmq.com/blog/2014/04/14/finding-bottlenecks-with-rabbitmq-3-3/ > -- > MK > > Software Engineer, Pivotal/RabbitMQ > -- Juno Chan Algorithmic Trading Group (ATG) Ltd www.algorithmictradinggroup.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From simon at rabbitmq.com Tue Apr 15 08:58:09 2014 From: simon at rabbitmq.com (Simon MacMullen) Date: Tue, 15 Apr 2014 08:58:09 +0100 Subject: [rabbitmq-discuss] RabbitMQ Causing Segmentation Fault on FreeBSD In-Reply-To: References: Message-ID: <534CE691.7010203@rabbitmq.com> I assume you mean the Erlang VM is segfaulting (or at least terminating abruptly)? Is there anything in the logs? Does anything get printed to standard output / error just before the termination? Which versions of RabbitMQ / Erlang are you running? Have you enabled HiPE? Cheers, Simon On 15/04/2014 00:27, Brandon Munroe wrote: > We're trying to run a RabbitMQ instance on FreeBSD 10.0 with 10k > concurrent connections/queues, and we are experiencing a segmentation > fault after letting the system idle for a little while (half an hour or > so). During this time, we just connect the 10k clients and pass no > messages. Does anyone have any ideas on where we could start looking to > diagnose the problem, or has anyone encountered similar problems? > > Thanks in advance. > > -Brandon > > > _______________________________________________ > rabbitmq-discuss mailing list > rabbitmq-discuss at lists.rabbitmq.com > https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss > From mklishin at gopivotal.com Tue Apr 15 08:58:35 2014 From: mklishin at gopivotal.com (Michael Klishin) Date: Tue, 15 Apr 2014 11:58:35 +0400 Subject: [rabbitmq-discuss] Ready, Unacked and Total messages of Queues In-Reply-To: References: <178d6430-4bd2-463e-8b4f-29bd3d532283@googlegroups.com> <534CDE74.60201@rabbitmq.com> Message-ID: On 15 April 2014 at 11:56:25, Juno Chan (juno at algorithmictradinggroup.com) wrote: > > So it's good to hear that #ready msgs high is not impacting RabbitMQ > performance. Yes, it will, as soon as there are so many messages that RabbitMQ has to perform a lot of disk I/O. > because we encountered an issues like the total messages in rabbit > was up to 300K, while some applications have queues of extremely > high #ready, but no #unack, while some applications have queues > of high #unack, but no #ready. My point was that you should avoid overloaded queues and there are multiple features that help you with that even if you can?t have more consumer capacity that producer one.? -- MK Software Engineer, Pivotal/RabbitMQ From juno at algorithmictradinggroup.com Tue Apr 15 09:13:48 2014 From: juno at algorithmictradinggroup.com (Juno Chan) Date: Tue, 15 Apr 2014 16:13:48 +0800 Subject: [rabbitmq-discuss] Ready, Unacked and Total messages of Queues In-Reply-To: References: <178d6430-4bd2-463e-8b4f-29bd3d532283@googlegroups.com> <534CDE74.60201@rabbitmq.com> Message-ID: Hi MK, I have set the per queue message TTL value to a small values for some non-critical queues I have read the below link http://www.rabbitmq.com/blog/2012/05/11/some-queuing-theory-throughput-latency-and-bandwidth/ but I don't quite understand how prefetch helps, could you please explain more on that? thanks. If this really helps, how can it be set with .NET code? the tutorials only shows python codes. thanks very much for your help. Cheers On Tue, Apr 15, 2014 at 3:58 PM, Michael Klishin wrote: > > > On 15 April 2014 at 11:56:25, Juno Chan (juno at algorithmictradinggroup.com) > wrote: > > > So it's good to hear that #ready msgs high is not impacting RabbitMQ > > performance. > > Yes, it will, as soon as there are so many messages that RabbitMQ has to > perform > a lot of disk I/O. > > > because we encountered an issues like the total messages in rabbit > > was up to 300K, while some applications have queues of extremely > > high #ready, but no #unack, while some applications have queues > > of high #unack, but no #ready. > > My point was that you should avoid overloaded queues and there are multiple > features that help you with that even if you can?t have more consumer > capacity > that producer one. > -- > MK > > Software Engineer, Pivotal/RabbitMQ > -- Juno Chan Algorithmic Trading Group (ATG) Ltd www.algorithmictradinggroup.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From mklishin at gopivotal.com Tue Apr 15 09:19:47 2014 From: mklishin at gopivotal.com (Michael Klishin) Date: Tue, 15 Apr 2014 12:19:47 +0400 Subject: [rabbitmq-discuss] Ready, Unacked and Total messages of Queues In-Reply-To: References: <178d6430-4bd2-463e-8b4f-29bd3d532283@googlegroups.com> <534CDE74.60201@rabbitmq.com> Message-ID: On 15 April 2014 at 12:14:28, Juno Chan (juno at algorithmictradinggroup.com) wrote: > > but I don't quite understand how prefetch helps, could you please > explain more on that? thanks. basic.qos controls how many messages can be in progress (delivered to consumers) before at least one of them is acknowledged or rejected. This setting is per-channel, there is no limit by default.? > If this really helps, how can it be set with .NET code? the tutorials > only shows python codes. http://www.rabbitmq.com/tutorials/tutorial-two-dotnet.html -- MK Software Engineer, Pivotal/RabbitMQ From mklishin at gopivotal.com Tue Apr 15 10:00:57 2014 From: mklishin at gopivotal.com (Michael Klishin) Date: Tue, 15 Apr 2014 13:00:57 +0400 Subject: [rabbitmq-discuss] Ready, Unacked and Total messages of Queues In-Reply-To: References: <178d6430-4bd2-463e-8b4f-29bd3d532283@googlegroups.com> <534CDE74.60201@rabbitmq.com> Message-ID: Please keep rabbitmq-discuss in ?CC. On 15 April 2014 at 12:53:15, Juno Chan (juno at algorithmictradinggroup.com) wrote: > > For testing purposes, I tried to set the per queue message TTL > value to 100 ms, and the queue really grows up more slowly(both > ready and unack), and it drops messages too Expired messages are deleted, so this is expected. > > However, after I added the Channel.BasicQos(0, 1, false), with > x-message-ttl to a higher value(e.g. 1min), when I set a breakpoint, > the queue was having a relatively low unack, but the ready # messages > are going up quite high, may I know is that the normal behaviour > as well? It just means that your consumer(s) cannot keep up with publisher(s). Add more consumers or throttle publishers. -- MK Software Engineer, Pivotal/RabbitMQ From matthias at rabbitmq.com Tue Apr 15 10:23:37 2014 From: matthias at rabbitmq.com (Matthias Radestock) Date: Tue, 15 Apr 2014 10:23:37 +0100 Subject: [rabbitmq-discuss] Ready, Unacked and Total messages of Queues In-Reply-To: References: <178d6430-4bd2-463e-8b4f-29bd3d532283@googlegroups.com> <534CDE74.60201@rabbitmq.com> Message-ID: <534CFA99.6090502@rabbitmq.com> On 15/04/14 10:00, Michael Klishin wrote: > On 15 April 2014 at 12:53:15, Juno Chan (juno at algorithmictradinggroup.com) wrote: >> However, after I added the Channel.BasicQos(0, 1, false), with >> x-message-ttl to a higher value(e.g. 1min), when I set a breakpoint, >> the queue was having a relatively low unack, but the ready # messages >> are going up quite high, may I know is that the normal behaviour >> as well? > > It just means that your consumer(s) cannot keep up with publisher(s). > Add more consumers or throttle publishers. ...or increase your prefetch count. Matthias. From juno at algorithmictradinggroup.com Tue Apr 15 10:51:00 2014 From: juno at algorithmictradinggroup.com (Juno Chan) Date: Tue, 15 Apr 2014 17:51:00 +0800 Subject: [rabbitmq-discuss] Ready, Unacked and Total messages of Queues In-Reply-To: <534CFA99.6090502@rabbitmq.com> References: <178d6430-4bd2-463e-8b4f-29bd3d532283@googlegroups.com> <534CDE74.60201@rabbitmq.com> <534CFA99.6090502@rabbitmq.com> Message-ID: Thanks much MK! BTW, are we able to query the total number of messages published to rabbit since up? or if we can get the overall publish rate * uptime is also good. Can either be on management UI or by api query On Tue, Apr 15, 2014 at 5:23 PM, Matthias Radestock wrote: > On 15/04/14 10:00, Michael Klishin wrote: > >> On 15 April 2014 at 12:53:15, Juno Chan (juno at algorithmictradinggroup.com) >> wrote: >> >>> However, after I added the Channel.BasicQos(0, 1, false), with >>> x-message-ttl to a higher value(e.g. 1min), when I set a breakpoint, >>> the queue was having a relatively low unack, but the ready # messages >>> are going up quite high, may I know is that the normal behaviour >>> as well? >>> >> >> It just means that your consumer(s) cannot keep up with publisher(s). >> Add more consumers or throttle publishers. >> > > ...or increase your prefetch count. > > Matthias. > -- Juno Chan Algorithmic Trading Group (ATG) Ltd www.algorithmictradinggroup.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From mklishin at gopivotal.com Tue Apr 15 10:55:04 2014 From: mklishin at gopivotal.com (Michael Klishin) Date: Tue, 15 Apr 2014 13:55:04 +0400 Subject: [rabbitmq-discuss] Ready, Unacked and Total messages of Queues In-Reply-To: References: <178d6430-4bd2-463e-8b4f-29bd3d532283@googlegroups.com> <534CDE74.60201@rabbitmq.com> <534CFA99.6090502@rabbitmq.com> Message-ID: On 15 April 2014 at 13:52:51, Juno Chan (juno at algorithmictradinggroup.com) wrote: > > BTW, are we able to query the total number of messages published > to rabbit since up? or if we can get the overall publish rate * uptime > is also good. > Can either be on management UI or by api query You can access message rates over HTTP API but not the total number of messages for the entire cluster lifetime: http://www.rabbitmq.com/management.html http://hg.rabbitmq.com/rabbitmq-management/raw-file/rabbitmq_v3_3_0/priv/www/api/index.html -- MK Software Engineer, Pivotal/RabbitMQ From juno at algorithmictradinggroup.com Tue Apr 15 11:04:54 2014 From: juno at algorithmictradinggroup.com (Juno Chan) Date: Tue, 15 Apr 2014 18:04:54 +0800 Subject: [rabbitmq-discuss] Ready, Unacked and Total messages of Queues In-Reply-To: References: <178d6430-4bd2-463e-8b4f-29bd3d532283@googlegroups.com> <534CDE74.60201@rabbitmq.com> <534CFA99.6090502@rabbitmq.com> Message-ID: It's okay, is the uptime available as well? so that I can calculate the stats manually. On Tue, Apr 15, 2014 at 5:55 PM, Michael Klishin wrote: > On 15 April 2014 at 13:52:51, Juno Chan (juno at algorithmictradinggroup.com) > wrote: > > > BTW, are we able to query the total number of messages published > > to rabbit since up? or if we can get the overall publish rate * uptime > > is also good. > > Can either be on management UI or by api query > > You can access message rates over HTTP API but not the total number of > messages > for the entire cluster lifetime: > > http://www.rabbitmq.com/management.html > > http://hg.rabbitmq.com/rabbitmq-management/raw-file/rabbitmq_v3_3_0/priv/www/api/index.html > -- > MK > > Software Engineer, Pivotal/RabbitMQ > -- Juno Chan Algorithmic Trading Group (ATG) Ltd www.algorithmictradinggroup.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From bison_art at o2.pl Tue Apr 15 13:52:52 2014 From: bison_art at o2.pl (rabbit15) Date: Tue, 15 Apr 2014 05:52:52 -0700 (PDT) Subject: [rabbitmq-discuss] RabbitMQ Cluster node Message-ID: <1397566372860-34893.post@n5.nabble.com> Hi, I have problem with one node in my cluster: rabbitmq/sbin/rabbitmqctl start_app Starting node 'rabbit at centos65-node21' ... Error: {badarg,[{erlang,register,[rabbit_boot,<5310.561.0>],[]}, {rabbit,start_it,1,[]}, {rpc,'-handle_call_call/6-fun-0-',5, [{file,"rpc.erl"},{line,203}]}]} Could you explain me what is a problem ? Best Regards, Artur -- View this message in context: http://rabbitmq.1065348.n5.nabble.com/RabbitMQ-Cluster-node-tp34893.html Sent from the RabbitMQ mailing list archive at Nabble.com. From josephrouphael at gmail.com Tue Apr 15 14:20:20 2014 From: josephrouphael at gmail.com (joseph rouphael) Date: Tue, 15 Apr 2014 16:20:20 +0300 Subject: [rabbitmq-discuss] Fwd: Performance bottleneck on inter-node connection In-Reply-To: <534BA3C3.3080607@rabbitmq.com> References: <53465CF5.20607@rabbitmq.com> <5346C06C.8090609@rabbitmq.com> <534BA3C3.3080607@rabbitmq.com> Message-ID: On Mon, Apr 14, 2014 at 12:00 PM, Matthias Radestock wrote: > On 11/04/14 17:37, joseph rouphael wrote: > >> Test 8: Dual node, Queues/Consumers on same node, 20 queue, ACK off >> ------------------------------------------------------------------- >> Nodes: Double node >> Queues created on: Node1 >> Producers on: node 2 >> Consumers on: node 1 >> Producer Processes: 20 >> Producer regulate rate per process: 5KHZ >> Acknowledgment: OFF >> Expected rate: 100KHZ >> Achieved rate: Fluctuating between 10KHZ and 20KHZ >> CPU idle: 95% >> Note: Bottleneck on inter-node connection >> > > What happens when you run the above test with both nodes on the same > machine? > The result is almost the same: Fluctuating between 10 and 23KHZ (Expected 100KHZ). Here is below the summary: Test 11: Dual nodes same machine, Queues/Consumers on same node, 20 queue, ACK on/off ------------------------------------------------------------ ------------------------------------------------------------------------------- Nodes: Double nodes same machine Queues created on: Node1 Producers on: node 2 Consumers on: node 1 Producer Processes: 20 Producer regulate rate per process: 5KHZ Acknowledgment: OFF or ON same result Expected rate: 100KHZ Achieved rate: Fluctuating between 10KHZ and 23KHZ CPU idle: 82% Note: Bottleneck on inter-node connection > Also, you really shouldn't need 20 producers/consumers to saturate a > broker, unless the machine has >20 cores. So I'd be interested to hear what > the smallest producer/consumer count is at which you see a significant > performance difference between the single node and dual node case. > The machine I am working on has 32 cores. So I am relaxed with the numbers of processes. 1 producer/consumer achieve a maximum of 11 KHZ (No inter-node impact. On single node it achieves the same) 2 producers/consumers achieve a maximum of ~ 20 KHZ (Minor inter-node impact - expected 22KHZ) 3 producers/consumers achieve a maximum of ~ 20 KHZ (Major inter-node impact - expected 33KHZ) > > Finally, it would be helpful if you ran your tests using RabbitMQ's > standard performance testing tool, PerfTest - > http://www.rabbitmq.com/java-tools.html#perftest. In order to replicate > the above scenario you'd run 20 PerfTest instances with options > -u q -x 0 -a > and 20 instances with > -u q -y 0 > with n in 1..20. > > perftest ran as advised and the test produced the same result. There is a real inter-node bottleneck. Results of perftest: Test 1: Double nodes same machine ---------- Node1 and Node2 on same machine Node1 listening on port 5672 Node2 listening on port 5673 20 consumers running full speed 20 producers running full speed Message size 1KB Max rate achieved: 16KHZ CPU Idle 92% Note: Inter-node bottleneck The script used to run the test: #!/bin/sh for i in 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 do ./runjava.sh com.rabbitmq.examples.PerfTest -k queue$i -x 0 -a -h amqp:// 193.100.200.130:5673 -s 1000 & ./runjava.sh com.rabbitmq.examples.PerfTest -k queue$i -y 0 -h amqp:// 193.100.200.130:5672 -s 1000 & done Test 2: Single node --------- Single node: Node1 Node1 listening on port 5672 20 consumers running full speed 20 producers running full speed Message size 1KB Max rate achieved: 200KHZ CPU Idle 12% The script used to run the test: #!/bin/sh for i in 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 do ./runjava.sh com.rabbitmq.examples.PerfTest -k queue$i -x 0 -a -h amqp:// 193.100.200.130:5672 -s 1000 & ./runjava.sh com.rabbitmq.examples.PerfTest -k queue$i -y 0 -h amqp:// 193.100.200.130:5672 -s 1000 & done > > Regards, > > Matthias. > _______________________________________________ > rabbitmq-discuss mailing list > rabbitmq-discuss at lists.rabbitmq.com > https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss > -------------- next part -------------- An HTML attachment was scrubbed... URL: From dmalinovsky at thumbtack.net Tue Apr 15 13:02:06 2014 From: dmalinovsky at thumbtack.net (Dmitry Malinovsky) Date: Tue, 15 Apr 2014 19:02:06 +0700 Subject: [rabbitmq-discuss] CRL support & verify_fun In-Reply-To: <534BC433.8070309@rabbitmq.com> References: <880cfca2-4982-4c9c-9090-f7092f4020bb@googlegroups.com> <534BC433.8070309@rabbitmq.com> Message-ID: On Mon, Apr 14, 2014 at 6:19 PM, Matthias Radestock wrote: > Write a custom plug-in. https://www.rabbitmq.com/plugin-development.html > Should I really implement all this application stuff just to have an ability to get access to a single function? Also, this "plugin" will not have any behavior, it is not even an application, so I'm really stuck. Thanks, -- Dmitry Malinovsky Support Engineer Thumbtack Technology Email: dmalinovsky at thumbtack.net Skype: d_malinoff -------------- next part -------------- An HTML attachment was scrubbed... URL: From tim at rabbitmq.com Tue Apr 15 14:37:15 2014 From: tim at rabbitmq.com (Tim Watson) Date: Tue, 15 Apr 2014 14:37:15 +0100 Subject: [rabbitmq-discuss] CRL support & verify_fun In-Reply-To: References: <880cfca2-4982-4c9c-9090-f7092f4020bb@googlegroups.com> <534BC433.8070309@rabbitmq.com> Message-ID: On 15 Apr 2014, at 13:02, Dmitry Malinovsky wrote: > On Mon, Apr 14, 2014 at 6:19 PM, Matthias Radestock wrote: > Write a custom plug-in. https://www.rabbitmq.com/plugin-development.html > > Should I really implement all this application stuff just to have an ability to get access to a single function? Also, this "plugin" will not have any behavior, it is not even an application, so I'm really stuck. > Well, Rabbit (and in this case actually, Erlang) needs to know about the "application" so it knows to load the various modules (i.e., code) at runtime. Also note that in Erlang, applications come in both active and passive guises, with applications that have no processes treated in much the same way as those that do. The latter kind are often called "library applications", and still need to specify an application config file so the Erlang runtime knows what to do with them. Cheers, Tim -------------- next part -------------- An HTML attachment was scrubbed... URL: From mcintoshj at gmail.com Tue Apr 15 16:06:02 2014 From: mcintoshj at gmail.com (Jason McIntosh) Date: Tue, 15 Apr 2014 10:06:02 -0500 Subject: [rabbitmq-discuss] Installing broker on each node instead of client-only approach Message-ID: Just a note - we use local shovels on the distributed nodes instead of federation on the central node so that the "central" rabbitmq server doesn't require knowledge of the remote nodes, e.g. cloud computers which can come up and down as you wish, and the central piece doesn't have to know about them, it just sits there waiting for data. Jason On Mon, Apr 14, 2014 at 9:30 AM, Matthias Radestock wrote: > On 14/04/14 15:04, ??????? ???????? wrote: > >> Thank you, Matthias. Is there any limits for queue size to be reached >> while working in autonomy/degrade mode? I need this to decide how >> long my setup could work without connectivity. >> > > The limiting factors are memory and disk space. When memory gets tight and > messages are fully paged out to disk, each message requires about 122 bytes > of memory (on a 64-bit machine), and about 255 bytes + payload size of disk > space. > > > Matthias. > _______________________________________________ > rabbitmq-discuss mailing list > rabbitmq-discuss at lists.rabbitmq.com > https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss > -- Jason McIntosh https://github.com/jasonmcintosh/ 573-424-7612 -------------- next part -------------- An HTML attachment was scrubbed... URL: From mcintoshj at gmail.com Tue Apr 15 16:11:36 2014 From: mcintoshj at gmail.com (Jason McIntosh) Date: Tue, 15 Apr 2014 10:11:36 -0500 Subject: [rabbitmq-discuss] RabbitMQ Deployment Architectures (for lack of a better subject) In-Reply-To: References: Message-ID: We do something like this using dns and search lists. e.g. we shovel to an "rabbitbroker" which using search list on low SLA boxes "rabbitrbroker.batchprocessing.internaldomain" yet on higher priority systems, that has a different search list that resolves to "rabbitbroker.interactive.internaldomain" This way our queues, exchanges, configurations are all identical, we just change where the shovels point to. Jason On Sat, Apr 12, 2014 at 2:11 PM, Greg Poirier wrote: > On Sat, Apr 12, 2014 at 11:34 AM, Michael Klishin wrote: > >> A small service that clients ask where to connect before doing so >> sounds reasonable. >> > > Yup. I've got some ideas on how to do this with configuration management > once we have made it so that all services are deployed by Chef. It's this > interim state that we're in that is making things atrociously difficult. > > _______________________________________________ > rabbitmq-discuss mailing list > rabbitmq-discuss at lists.rabbitmq.com > https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss > > -- Jason McIntosh https://github.com/jasonmcintosh/ 573-424-7612 -------------- next part -------------- An HTML attachment was scrubbed... URL: From brandon.munroe at gmail.com Tue Apr 15 17:25:58 2014 From: brandon.munroe at gmail.com (Brandon Munroe) Date: Tue, 15 Apr 2014 10:25:58 -0600 Subject: [rabbitmq-discuss] RabbitMQ Causing Segmentation Fault on FreeBSD In-Reply-To: <534CE691.7010203@rabbitmq.com> References: <534CE691.7010203@rabbitmq.com> Message-ID: Yep, the VM is segfaulting (it exits with the message "Segmentation fault (core dumped)"). The message in the logs reads: ** Generic server rabbit_mgmt_external_stats terminating ** Last message in was emit_update ** When Server state == {state,707130} ** Reason for termination == ** {function_clause,[{lists,nth,[1,[]],[{file,"lists.erl"},{line,168}]}, {rabbit_mgmt_external_stats,'-get_used_fd/1-fun-1-',2,[]}, {lists,'-filter/2-lc$^0/1-0-',2, [{file,"lists.erl"},{line,1271}]}, {lists,'-filter/2-lc$^0/1-0-',2, [{file,"lists.erl"},{line,1271}]}, {rabbit_mgmt_external_stats,get_used_fd,1,[]}, {rabbit_mgmt_external_stats,'-infos/2-lc$^0/1-0-',2,[]}, {rabbit_mgmt_external_stats,'-infos/2-lc$^0/1-0-',2,[]}, {rabbit_mgmt_external_stats,emit_update,1,[]}]} The only message on stdout/stderr is the one above. We're running RabbitMQ 3.2.4 on Erlang R16B03-1, with HiPE compiled in (though we didn't specify hipe_compile in the RabbitMQ configuration). -Brandon On Tue, Apr 15, 2014 at 1:58 AM, Simon MacMullen wrote: > I assume you mean the Erlang VM is segfaulting (or at least terminating > abruptly)? > > Is there anything in the logs? Does anything get printed to standard > output / error just before the termination? Which versions of RabbitMQ / > Erlang are you running? Have you enabled HiPE? > > Cheers, Simon > > > On 15/04/2014 00:27, Brandon Munroe wrote: > >> We're trying to run a RabbitMQ instance on FreeBSD 10.0 with 10k >> concurrent connections/queues, and we are experiencing a segmentation >> fault after letting the system idle for a little while (half an hour or >> so). During this time, we just connect the 10k clients and pass no >> messages. Does anyone have any ideas on where we could start looking to >> diagnose the problem, or has anyone encountered similar problems? >> >> Thanks in advance. >> >> -Brandon >> >> >> _______________________________________________ >> rabbitmq-discuss mailing list >> rabbitmq-discuss at lists.rabbitmq.com >> https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss >> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mpietrek at skytap.com Tue Apr 15 23:09:24 2014 From: mpietrek at skytap.com (Matt Pietrek) Date: Tue, 15 Apr 2014 15:09:24 -0700 Subject: [rabbitmq-discuss] Queue disappears during partition/autoheal Message-ID: This is rabbitmq 3.2.4, running in a 2 node cluster with all queues in ha. There was a queue called cmcmd declared like this: arguments:x-ha-policy:alldurable:true At some point we saw a network partition (see below). It appears that Autoheal eventually worked, but afterwards the cmcmd queue wasn't on the broker. Here's the Autoheal sequence (note the big time gap waiting for the sea5m1mq1 shutdown): ----------- 21:57 mpietrek at foo:/$ grep heal 2014-04-14-mq.log 2014-04-14 18:02:35 sea5m1mq2 [info] rabbit at sea5m1mq2.log: Auto*heal*request sent to rabbit at sea5m1mq1 2014-04-14 18:02:35 sea5m1mq2 [info] rabbit at sea5m1mq2.log: Auto*heal*: I am the winner, waiting for [rabbit at sea5m1mq1] to stop 2014-04-14 18:02:35 sea5m1mq2 [info] rabbit at sea5m1mq2.log: Auto*heal*: final node has stopped, starting... 2014-04-14 18:57:38 sea5m1mq1 [info] rabbit at sea5m1mq1.log: Auto*heal*request received from rabbit at sea5m1mq2 2014-04-14 18:57:38 sea5m1mq1 [info] rabbit at sea5m1mq1.log: Auto*heal*decision 2014-04-14 18:57:38 sea5m1mq1 [info] rabbit at sea5m1mq1.log: Auto*heal*: we were selected to restart; winner is rabbit at sea5m1mq2 ------------ And the rabbit at sea5m1mq2 log spew: =ERROR REPORT==== 14-Apr-2014::18:02:30 === ** Node rabbit at sea5m1mq1 not responding ** ** Removing (timedout) connection ** =INFO REPORT==== 14-Apr-2014::18:02:30 === rabbit on node rabbit at sea5m1mq1 down =ERROR REPORT==== 14-Apr-2014::18:02:30 === Mnesia(rabbit at sea5m1mq2): ** ERROR ** mnesia_event got {inconsistent_database, running_partitioned_network, rabbit at sea5m1mq1} =INFO REPORT==== 14-Apr-2014::18:02:30 === Statistics database started. =INFO REPORT==== 14-Apr-2014::18:02:30 === Autoheal request sent to rabbit at sea5m1mq1 =ERROR REPORT==== 14-Apr-2014::18:02:30 === ** Generic server <0.204.0> terminating ** Last message in was {mnesia_locker,rabbit at sea5m1mq1,granted} ** When Server state == {state,2,{from,<0.302.0>,#Ref<0.0.1372.163190>}} ** Reason for termination == ** {unexpected_info,{mnesia_locker,rabbit at sea5m1mq1,granted}} =ERROR REPORT==== 14-Apr-2014::18:02:30 === ** Generic server <0.302.0> terminating ** Last message in was {'DOWN',#Ref<0.0.0.2733>,process,<2782.309.0>, noconnection} ** When Server state == {state, {0,<0.302.0>}, {{0,<2782.309.0>},#Ref<0.0.0.2733>}, {{0,<2782.309.0>},#Ref<0.0.0.2734>}, {resource,<<"/">>,queue,<<"cmcmd">>}, rabbit_mirror_queue_coordinator, {1, [{{0,<0.302.0>}, {view_member, {0,<0.302.0>}, [], {0,<2782.309.0>}, {0,<2782.309.0>}}}, {{0,<2782.309.0>}, {view_member, {0,<2782.309.0>}, [], {0,<0.302.0>}, {0,<0.302.0>}}}]}, 0, [{{0,<0.302.0>},{member,{[],[]},0,0}}, {{0,<2782.309.0>},{member,{[],[]},0,0}}], [<0.301.0>], {[],[]}, [],0,undefined, #Fun} ** Reason for termination == ** {noproc,{gen_server2,call, [<0.204.0>, {submit,#Fun,<0.302.0>}, infinity]}} =ERROR REPORT==== 14-Apr-2014::18:02:30 === ** Generic server <0.203.0> terminating ** Last message in was {mnesia_locker,rabbit at sea5m1mq1,granted} ** When Server state == {state,1,undefined} ** Reason for termination == -------------- next part -------------- An HTML attachment was scrubbed... URL: From juno at algorithmictradinggroup.com Wed Apr 16 04:39:52 2014 From: juno at algorithmictradinggroup.com (Juno Chan) Date: Wed, 16 Apr 2014 11:39:52 +0800 Subject: [rabbitmq-discuss] Ready, Unacked and Total messages of Queues In-Reply-To: <534CFA99.6090502@rabbitmq.com> References: <178d6430-4bd2-463e-8b4f-29bd3d532283@googlegroups.com> <534CDE74.60201@rabbitmq.com> <534CFA99.6090502@rabbitmq.com> Message-ID: Hi Matthias and MK, In addition, when I set the prefetch count to 1, there should be only 1 message delivered to the queue, why in my testing code, there are still unack message#>1 ? if the prefetch count is 1, unack message should always be in (0,1), and ready message #will be increasing. will the increasing of ready message count harming the performance on publisher (i.e. cannot publish anything, or causing publishing application/machine got stuck)? Though ready message # theoretically does not harm rabbitMQ if not much diskI/O Thanks very much On Tue, Apr 15, 2014 at 5:23 PM, Matthias Radestock wrote: > On 15/04/14 10:00, Michael Klishin wrote: > >> On 15 April 2014 at 12:53:15, Juno Chan (juno at algorithmictradinggroup.com) >> wrote: >> >>> However, after I added the Channel.BasicQos(0, 1, false), with >>> x-message-ttl to a higher value(e.g. 1min), when I set a breakpoint, >>> the queue was having a relatively low unack, but the ready # messages >>> are going up quite high, may I know is that the normal behaviour >>> as well? >>> >> >> It just means that your consumer(s) cannot keep up with publisher(s). >> Add more consumers or throttle publishers. >> > > ...or increase your prefetch count. > > Matthias. > -- Juno Chan Algorithmic Trading Group (ATG) Ltd www.algorithmictradinggroup.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From sashimw at gmail.com Wed Apr 16 05:19:54 2014 From: sashimw at gmail.com (Sashika Wijesinghe) Date: Wed, 16 Apr 2014 09:49:54 +0530 Subject: [rabbitmq-discuss] Monitoring message details that moves through RabbitMQ Message-ID: Hi, Is there any plugin to monitor the messages move through the RabbitMQ to the server? I saw there was a Splunk plugin. But as it mentioned it will only capable of monitoring RabbitMQ health and performance. But I want to monitor message details that moves through the queue. Please help on this Thanks in advance. Regards, Sashi -------------- next part -------------- An HTML attachment was scrubbed... URL: From simon at rabbitmq.com Wed Apr 16 07:50:57 2014 From: simon at rabbitmq.com (Simon MacMullen) Date: Wed, 16 Apr 2014 07:50:57 +0100 Subject: [rabbitmq-discuss] Monitoring message details that moves through RabbitMQ In-Reply-To: References: Message-ID: <534E2851.8030609@rabbitmq.com> On 16/04/2014 05:19, Sashika Wijesinghe wrote: > Is there any plugin to monitor the messages move through the RabbitMQ to > the server? I saw there was a Splunk plugin. But as it mentioned it will > only capable of monitoring RabbitMQ health and performance. But I want > to monitor message details that moves through the queue. See http://www.rabbitmq.com/firehose.html Cheers, Simon From mklishin at gopivotal.com Wed Apr 16 09:43:21 2014 From: mklishin at gopivotal.com (Michael Klishin) Date: Wed, 16 Apr 2014 12:43:21 +0400 Subject: [rabbitmq-discuss] Ready, Unacked and Total messages of Queues In-Reply-To: References: <178d6430-4bd2-463e-8b4f-29bd3d532283@googlegroups.com> <534CDE74.60201@rabbitmq.com> <534CFA99.6090502@rabbitmq.com> Message-ID: On 16 April 2014 at 07:41:47, Juno Chan (juno at algorithmictradinggroup.com) wrote: > > In addition, when I set the prefetch count to 1, there should > be only 1 message delivered to the queue, why in my testing code, > there are still unack message#>1 ? prefetch works per channel. If you use multiple channels or connections, there can (and will) be multiple messages unacknowledged at a time. > if the prefetch count is 1, unack message should always be in (0,1), > and ready message #will be increasing. will the increasing of > ready message count harming the performance on publisher (i.e. > cannot publish anything, or causing publishing application/machine > got stuck)? Though ready message # theoretically does not harm > rabbitMQ if not much diskI/O This question was largely answered earlier. If a memory or available disk space alarm is triggered, publishing connections will be blocked: http://rabbitmq.com/memory.html -- MK Software Engineer, Pivotal/RabbitMQ From mklishin at gopivotal.com Wed Apr 16 10:39:11 2014 From: mklishin at gopivotal.com (Michael Klishin) Date: Wed, 16 Apr 2014 13:39:11 +0400 Subject: [rabbitmq-discuss] RabbitMQ Message consuming issue (when consumer keepting idle) In-Reply-To: <0AE0D43E7CAC0342AE4BCDD4A4F5F452011AE87B@NII-WINSRV.niiwin.local> References: <0AE0D43E7CAC0342AE4BCDD4A4F5F452011AE700@NII-WINSRV.niiwin.local> <0AE0D43E7CAC0342AE4BCDD4A4F5F452011AE87B@NII-WINSRV.niiwin.local> Message-ID: On 16 April 2014 at 13:35:35, Mahendrakumar Yadavalli (mahendrakumary at neelinfo.com) wrote: > > We didn?t get any error/warning messages in that specified > scenario. > > But we are facing another problem like "some of the messages are > disappeared from the queue even though they are not get consumed > by the client", Where we are getting some exceptions in our consumer > logs. Can you please find the attached mail. Suggest me the suitable > fix. The exception you get is a clean shutdown (when the app itself calls Channel#close). It certainly looks like an app problem to me. Can your messages be published with expiration set? Can there be expiration set on queues? And so on. ? http://www.rabbitmq.com/ttl.html -- MK Software Engineer, Pivotal/RabbitMQ From simon at rabbitmq.com Wed Apr 16 11:12:06 2014 From: simon at rabbitmq.com (Simon MacMullen) Date: Wed, 16 Apr 2014 11:12:06 +0100 Subject: [rabbitmq-discuss] RabbitMQ Cluster node In-Reply-To: <1397566372860-34893.post@n5.nabble.com> References: <1397566372860-34893.post@n5.nabble.com> Message-ID: <534E5776.90602@rabbitmq.com> The node thinks start_app is already happening (i.e, it has not finished starting but it is starting). Probably the best way to make progress is to stop the node completely then start it again. Cheers, Simon On 15/04/14 13:52, rabbit15 wrote: > Hi, I have problem with one node in my cluster: > > rabbitmq/sbin/rabbitmqctl start_app > Starting node 'rabbit at centos65-node21' ... > Error: {badarg,[{erlang,register,[rabbit_boot,<5310.561.0>],[]}, > {rabbit,start_it,1,[]}, > {rpc,'-handle_call_call/6-fun-0-',5, > [{file,"rpc.erl"},{line,203}]}]} > > Could you explain me what is a problem ? > > Best Regards, > Artur > > > > > -- > View this message in context: http://rabbitmq.1065348.n5.nabble.com/RabbitMQ-Cluster-node-tp34893.html > Sent from the RabbitMQ mailing list archive at Nabble.com. > _______________________________________________ > rabbitmq-discuss mailing list > rabbitmq-discuss at lists.rabbitmq.com > https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss > -- Simon MacMullen RabbitMQ, Pivotal From simon at rabbitmq.com Wed Apr 16 13:08:46 2014 From: simon at rabbitmq.com (Simon MacMullen) Date: Wed, 16 Apr 2014 13:08:46 +0100 Subject: [rabbitmq-discuss] Queue disappears during partition/autoheal In-Reply-To: References: Message-ID: <534E72CE.5010001@rabbitmq.com> On 15/04/14 23:09, Matt Pietrek wrote: > This is rabbitmq 3.2.4, running in a 2 node cluster with all queues in ha. > At some point we saw a network partition (see below). It appears that > Autoheal eventually worked, but afterwards the cmcmd queue wasn't on the > broker. > =ERROR REPORT==== 14-Apr-2014::18:02:30 === > ** Generic server <0.204.0> terminating > ** Last message in was {mnesia_locker,rabbit at sea5m1mq1,granted} > ** When Server state == {state,2,{from,<0.302.0>,#Ref<0.0.1372.163190>}} > ** Reason for termination == > ** {unexpected_info,{mnesia_locker,rabbit at sea5m1mq1,granted}} So this is something we've seen before in the case of short-lived partitions; something in Mnesia is sending a stray {mnesia_locker, ..., ...} message to a process that isn't expecting it after the partition, killing the process in question. The release notes for Erlang 17.0 contain: OTP-11497 To prevent a race condition if there is a short communication problem when node-down and node-up events are received. They are now stored and later checked if the node came up just before mnesia flagged the node as down. (Thanks to Jonas Falkevik ) which sounds like the same thing. So it is quite possible that this is fixed in Erlang 17.0. Cheers, Simon -- Simon MacMullen RabbitMQ, Pivotal From Mahendrakumary at neelinfo.com Wed Apr 16 10:35:28 2014 From: Mahendrakumary at neelinfo.com (Mahendrakumar Yadavalli) Date: Wed, 16 Apr 2014 02:35:28 -0700 Subject: [rabbitmq-discuss] RabbitMQ Message consuming issue (when consumer keepting idle) References: <0AE0D43E7CAC0342AE4BCDD4A4F5F452011AE700@NII-WINSRV.niiwin.local> Message-ID: <0AE0D43E7CAC0342AE4BCDD4A4F5F452011AE87B@NII-WINSRV.niiwin.local> Hi Michael Klishin, We didn?t get any error/warning messages in that specified scenario. But we are facing another problem like "some of the messages are disappeared from the queue even though they are not get consumed by the client", Where we are getting some exceptions in our consumer logs. Can you please find the attached mail. Suggest me the suitable fix. Thanks&Regards Mahendra.y -----Original Message----- From: Michael Klishin [mailto:mklishin at gopivotal.com] Sent: Thursday, April 10, 2014 5:22 PM To: Mahendrakumar Yadavalli Cc: Discussions about RabbitMQ Subject: Re: RabbitMQ Message consuming issue (when consumer keepting idle) On 10 April 2014 at 14:56:28, Mahendrakumar Yadavalli (mahendrakumary at neelinfo.com) wrote: > > Both the producer and consumer are working fine. We got an issue > like ?when the android device kept idle for a long time, and start > working on it after some time, then messages are not consuming from > consumer side. once we delete the queue then it will work fine?. The > problem is when we kept the client device idle for long time this > issue occurs. There have to be some error messages somewhere. One possible reason is that when a device goes idle for a long time, Android hibernates the app entirely and it does not send out heartbeats, so RabbitMQ considers the connection dead. There should be warnings in RabbitMQ log about this. -- MK Software Engineer, Pivotal/RabbitMQ -------------- next part -------------- An embedded message was scrubbed... From: Manasa Potula Subject: Exception in RabbitMQ Date: Wed, 16 Apr 2014 02:19:29 -0700 Size: 67524 URL: From dubois.sebastien66 at gmail.com Wed Apr 16 15:09:21 2014 From: dubois.sebastien66 at gmail.com (Sebastien Dubois) Date: Wed, 16 Apr 2014 07:09:21 -0700 (PDT) Subject: [rabbitmq-discuss] RabbitMQ user accounts without management access? Message-ID: Hi all, We have an application that exposes a notification channel through a RabbitMQ broker. We would like to give access to our application users so that their own clients can have queues that they can bind to our notification RabbitMQ exchange (using the account host/user/password/vhost info), but without giving them access to the web-based rabbit management (i.e. the one that can be reached by default on port 15672), for security purposes. Is that possible? From what we can understand, you can have read-only user accounts, but that does not prevent access to the web-based management application. Thanks in advance /Sebastien -------------- next part -------------- An HTML attachment was scrubbed... URL: From simon at rabbitmq.com Wed Apr 16 15:17:45 2014 From: simon at rabbitmq.com (Simon MacMullen) Date: Wed, 16 Apr 2014 15:17:45 +0100 Subject: [rabbitmq-discuss] RabbitMQ user accounts without management access? In-Reply-To: References: Message-ID: <534E9109.9090602@rabbitmq.com> That is possible, see http://www.rabbitmq.com/management.html#permissions. In short, do not give the users the 'management' tag and you'll be fine. Cheers, Simon On 16/04/14 15:09, Sebastien Dubois wrote: > > Hi all, > > We have an application that exposes a notification channel through a > RabbitMQ broker. We would like to give access to our application users > so that their own clients can have queues that they can bind to our > notification RabbitMQ exchange (using the account > host/user/password/vhost info), but without giving them access to the > web-based rabbit management (i.e. the one that can be reached by default > on port 15672), for security purposes. > > Is that possible? From what we can understand, you can have read-only > user accounts, but that does not prevent access to the web-based > management application. > > Thanks in advance > /Sebastien > > > _______________________________________________ > rabbitmq-discuss mailing list > rabbitmq-discuss at lists.rabbitmq.com > https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss > -- Simon MacMullen RabbitMQ, Pivotal From gavinmroy at gmail.com Wed Apr 16 18:46:47 2014 From: gavinmroy at gmail.com (Gavin M. Roy) Date: Wed, 16 Apr 2014 13:46:47 -0400 Subject: [rabbitmq-discuss] Testing Plugin / Internals Question Message-ID: I?m working on a plugin that will let me trigger some internal behaviors in RabbitMQ from pika or any other RabbitMQ client for acceptance testing purposes. ?In doing so, I?ve been trying to trace back how to get the connection pid() from the channel that is passed as the sender in the #delivery record passed into rabbit_exchange_type:route/2. When I?m issuing rabbit_channel:info(pid()) for the sender, it seems to hang and I was hoping that was where I could get the connection pid. Does anyone have any suggestions as to where I can get that info? Ultimately for this particular piece of functionality, I?m looking to trigger rabbit_reader:conserve_resources/3 to force a Connection.Blocked frame to be sent. Any alternative ways to trigger that from either the exchange or an interceptor would be helpful as well. Thanks in advance, Gavin -------------- next part -------------- An HTML attachment was scrubbed... URL: From mpietrek at skytap.com Wed Apr 16 19:04:41 2014 From: mpietrek at skytap.com (Matt Pietrek) Date: Wed, 16 Apr 2014 11:04:41 -0700 Subject: [rabbitmq-discuss] Queue disappears during partition/autoheal In-Reply-To: <534E72CE.5010001@rabbitmq.com> References: <534E72CE.5010001@rabbitmq.com> Message-ID: Thanks Simon. One more question: > So this is something we've seen before in the case of short-lived partitions; something in Mnesia is sending a stray {mnesia_locker, ..., ...} message to a process that isn't expecting it after the partition, killing the process in question. Do you have a sense whether this behavior is specific to having Autoheal enabled? In other words, if we didn't have Autoheal in effect, could it still happen? On Wed, Apr 16, 2014 at 5:08 AM, Simon MacMullen wrote: > On 15/04/14 23:09, Matt Pietrek wrote: > >> This is rabbitmq 3.2.4, running in a 2 node cluster with all queues in ha. >> > > At some point we saw a network partition (see below). It appears that >> Autoheal eventually worked, but afterwards the cmcmd queue wasn't on the >> broker. >> > > =ERROR REPORT==== 14-Apr-2014::18:02:30 === >> ** Generic server <0.204.0> terminating >> ** Last message in was {mnesia_locker,rabbit at sea5m1mq1,granted} >> ** When Server state == {state,2,{from,<0.302.0>,#Ref<0.0.1372.163190>}} >> ** Reason for termination == >> ** {unexpected_info,{mnesia_locker,rabbit at sea5m1mq1,granted}} >> > > So this is something we've seen before in the case of short-lived > partitions; something in Mnesia is sending a stray {mnesia_locker, ..., > ...} message to a process that isn't expecting it after the partition, > killing the process in question. > > The release notes for Erlang 17.0 contain: > > OTP-11497 To prevent a race condition if there is a short communication > problem when node-down and node-up events are received. They > are now stored and later checked if the node came up just > before mnesia flagged the node as down. (Thanks to Jonas > Falkevik ) > > which sounds like the same thing. > > So it is quite possible that this is fixed in Erlang 17.0. > > Cheers, Simon > > -- > Simon MacMullen > RabbitMQ, Pivotal > -------------- next part -------------- An HTML attachment was scrubbed... URL: From gavinmroy at gmail.com Wed Apr 16 21:42:56 2014 From: gavinmroy at gmail.com (Gavin M. Roy) Date: Wed, 16 Apr 2014 16:42:56 -0400 Subject: [rabbitmq-discuss] ANN: rabbitpy 0.17.0 Message-ID: rabbitpy 0.17.0 is now available for download from pypi. It is considered the latest and greatest stable release and all users are urged to upgrade. This release is primarily a bug fix release: - Refactor cross-thread communication for RabbitMQ to client RPC requests - Fix cross-thread exceptions being raised - Fix shutdown behavior when an exception is thrown in the IO thread - Balance the read/write ratio on the socket fixing acknowledgement write buffering issues - Ensure compatibility with Python 3.4 - Documentation update to break out version history to a stand-alone document For more information, including installation instructions, please visit?https://pypi.python.org/pypi/rabbitpy?or the documentation at?https://rabbitpy.readthedocs.org? If you have any issues, please be sure to use the issue tracker at?https://github.com/gmr/rabbitpy/issues Regards, Gavin -------------- next part -------------- An HTML attachment was scrubbed... URL: From vish.ramachandran at gmail.com Wed Apr 16 22:38:55 2014 From: vish.ramachandran at gmail.com (vish.ramachandran) Date: Wed, 16 Apr 2014 14:38:55 -0700 (PDT) Subject: [rabbitmq-discuss] Supporting both SSL+Password as well as SSL+Certificate based authentication Message-ID: <1397684335559-34914.post@n5.nabble.com> We have a use case where we want to support both 1. "External" passwordless certificate based authentication based on rabbitmq-auth-mechanism-ssl plugin 2. "Plain" password based authentication with SSL turned on without client needing to present a certificate. This is precisely what is described at https://www.rabbitmq.com/ssl.html#enabling-ssl example. Below is the configuration in question. [ {rabbit, [ {ssl_listeners, [5671]}, {ssl_options, [{cacertfile,"/opt/ssl4/cacert.pem"}, {certfile,"/opt/ssl4/cert.pem"}, {keyfile,"/opt/ssl4/key.pem"}, {verify,verify_peer}, {fail_if_no_peer_cert,true}]}, {auth_mechanisms, ['PLAIN', 'AMQPLAIN', 'EXTERNAL']}, {ssl_cert_login_from, common_name} ]} ]. With above configuration, use case #1 is possible, however use case #2 fails when client does not present a certificate since fail_if_no_peer_certificate is set to true. If I change fail_if_no_peer_certificate to false, use case #2 works, use case #1 fails with an "unsafe configuration error". Can we configure RMQ to support both by 1. verifying the certificate if one is presented along with request to do external authentication 2. expect username/password if certificate is not presented and plain authentication is chosen 3. Fail if external authentication is chosen and no certificate is presented It does not seem right to mandate that password based clients also present valid certificate. If they could, then there is no need for password based authentication. -- View this message in context: http://rabbitmq.1065348.n5.nabble.com/Supporting-both-SSL-Password-as-well-as-SSL-Certificate-based-authentication-tp34914.html Sent from the RabbitMQ mailing list archive at Nabble.com. From tijunaitis at gmail.com Wed Apr 16 23:05:38 2014 From: tijunaitis at gmail.com (=?UTF-8?Q?Raimondas_Tij=C5=ABnaitis?=) Date: Wed, 16 Apr 2014 15:05:38 -0700 (PDT) Subject: [rabbitmq-discuss] performance degradation on unrelated queues Message-ID: Hi, I'm experiencing issue when slow publishing to one queue (because of slow consumers) affects other unrelated queue message publishing (publishing into bus also slows down). Rabbit itself is far from it's limits (reaches only 300 msgs/s). When slow publishing stops everything goes back to normal. Have anybody else experienced such issues? Is it possible that such behavior is by design or some configuration specifics? Thanks. -------------- next part -------------- An HTML attachment was scrubbed... URL: From dejan.glozic at gmail.com Wed Apr 16 23:29:41 2014 From: dejan.glozic at gmail.com (Dejan Glozic) Date: Wed, 16 Apr 2014 15:29:41 -0700 (PDT) Subject: [rabbitmq-discuss] Cannot get MQTT->AMQP to work Message-ID: <1cbd1e1b-5957-4340-9cce-b6f062306b81@googlegroups.com> I am banging my head on my desk the whole afternoon trying to figure this out. I am trying to publish a message into an MQTT topic using Eclipse MQTT Paho client, and receive it using RabbitMQ AMQP Java client. I am using RabbitMQ for Windows, version 3.2.4. >From what I was reading, MQTT plug-in will publish into the default 'amq.topic' exchange, and I can bind a queue to that exchange and start listening. When I publish a message into the exchange using AMQP client, the AMQP client on the receiving end will get it. When I publish a message using MQTT client, another MQTT client on the receiving end will get it. But for the love of God I cannot get the cross-protocol to work. Here is the MQTT client that publishes: MqttClient client; *try* { client = *new* MqttClient("tcp://"+host+":1883", "pahomqttpublish1"); client.connect(); MqttMessage message = *new* MqttMessage(messageText.getBytes()); client.publish(queueName, message); System.*out*.println(" [x] Sent to MQTT topic '" +queueName+"': "+ message + "'"); client.disconnect(); } *catch* (MqttException e) { // *TODO* Auto-generated catch block e.printStackTrace(); } And here is the AMQP client that is supposed to receive it: public static final String QUEUE_NAME="workerQueue"; *public* *static* *void* main(String[] args) *throws* IOException, InterruptedException { ConnectionFactory factory = *new* ConnectionFactory(); factory.setHost("localhost"); Connection connection = factory.newConnection(); Channel channel = connection.createChannel(); channel.queueDeclare(QUEUE_NAME, *true*, *false*, *false*, *null*); channel.queueBind(QUEUE_NAME, "amq.topic", "builds"); System.*out*.println(" [*] Waiting for messages. To exit press CTRL+C"); QueueingConsumer consumer = *new* QueueingConsumer(channel); channel.basicConsume(QUEUE_NAME, *true*, consumer); * while* (*true*) { QueueingConsumer.Delivery delivery = consumer.nextDelivery(); String message = *new* String(delivery.getBody()); String routingKey = delivery.getEnvelope().getRoutingKey(); System.*out*.println(" [x] Received '" + routingKey + "':'" + message + "'"); } } The client above will receive the message when published into "amq.topic" exchange by another AMQP client. Anything stupid and obvious I am missing? Dejan -------------- next part -------------- An HTML attachment was scrubbed... URL: From mklishin at gopivotal.com Thu Apr 17 00:09:20 2014 From: mklishin at gopivotal.com (Michael Klishin) Date: Thu, 17 Apr 2014 03:09:20 +0400 Subject: [rabbitmq-discuss] performance degradation on unrelated queues In-Reply-To: References: Message-ID: On 17 April 2014 at 03:00:53, Raimondas Tij?naitis (tijunaitis at gmail.com) wrote: > > 'm experiencing issue when slow publishing to one queue (because > of slow consumers) affects other unrelated queue message publishing > (publishing into bus also slows down). > > Rabbit itself is far from it's limits (reaches only 300 msgs/s). > When slow publishing stops everything goes back to normal. Can you post some example code? You say ?slow publishing because of slow consumers?, that?s a little confusing.? -- MK Software Engineer, Pivotal/RabbitMQ From mklishin at gopivotal.com Thu Apr 17 00:14:00 2014 From: mklishin at gopivotal.com (Michael Klishin) Date: Thu, 17 Apr 2014 03:14:00 +0400 Subject: [rabbitmq-discuss] Cannot get MQTT->AMQP to work In-Reply-To: <1cbd1e1b-5957-4340-9cce-b6f062306b81@googlegroups.com> References: <1cbd1e1b-5957-4340-9cce-b6f062306b81@googlegroups.com> Message-ID: On 17 April 2014 at 03:03:00, Dejan Glozic (dejan.glozic at gmail.com) wrote: > > client.publish(queueName, message); What?s the value of queueName here?? -- MK Software Engineer, Pivotal/RabbitMQ From dejan.glozic at gmail.com Thu Apr 17 03:05:01 2014 From: dejan.glozic at gmail.com (Dejan Glozic) Date: Wed, 16 Apr 2014 19:05:01 -0700 (PDT) Subject: [rabbitmq-discuss] Cannot get MQTT->AMQP to work In-Reply-To: <1cbd1e1b-5957-4340-9cce-b6f062306b81@googlegroups.com> References: <1cbd1e1b-5957-4340-9cce-b6f062306b81@googlegroups.com> Message-ID: Just realized the example is not clear - the value of 'queueName' on the MQTT side is 'builds' - the same as the value for the 'routingKey' on the AMQP side used for channel.queueBind. Dejan On Wednesday, April 16, 2014 6:29:41 PM UTC-4, Dejan Glozic wrote: > > I am banging my head on my desk the whole afternoon trying to figure this > out. I am trying to publish a message into an MQTT topic using Eclipse MQTT > Paho client, and receive it using RabbitMQ AMQP Java client. > > I am using RabbitMQ for Windows, version 3.2.4. > > From what I was reading, MQTT plug-in will publish into the default > 'amq.topic' exchange, and I can bind a queue to that exchange and start > listening. > > When I publish a message into the exchange using AMQP client, the AMQP > client on the receiving end will get it. > > When I publish a message using MQTT client, another MQTT client on the > receiving end will get it. > > But for the love of God I cannot get the cross-protocol to work. > > Here is the MQTT client that publishes: > > > MqttClient client; > > *try* { > > client = *new* MqttClient("tcp://"+host+":1883", "pahomqttpublish1"); > > client.connect(); > > MqttMessage message = *new* MqttMessage(messageText.getBytes()); > > client.publish(queueName, message); > > System.*out*.println(" [x] Sent to MQTT topic '" +queueName+"': "+ > message + "'"); > > client.disconnect(); > > } *catch* (MqttException e) { > > // *TODO* Auto-generated catch block > > e.printStackTrace(); > > } > > > And here is the AMQP client that is supposed to receive it: > > public static final String QUEUE_NAME="workerQueue"; > > *public* *static* *void* main(String[] args) *throws* IOException, > InterruptedException { > > ConnectionFactory factory = *new* ConnectionFactory(); > > factory.setHost("localhost"); > > Connection connection = factory.newConnection(); > > Channel channel = connection.createChannel(); > > channel.queueDeclare(QUEUE_NAME, *true*, *false*, *false*, *null*); > > channel.queueBind(QUEUE_NAME, "amq.topic", "builds"); > > System.*out*.println(" [*] Waiting for messages. To exit press CTRL+C" > ); > > > QueueingConsumer consumer = *new* QueueingConsumer(channel); > > channel.basicConsume(QUEUE_NAME, *true*, consumer); > > * while* (*true*) { > > QueueingConsumer.Delivery delivery = consumer.nextDelivery(); > > String message = *new* String(delivery.getBody()); > > String routingKey = delivery.getEnvelope().getRoutingKey(); > > System.*out*.println(" [x] Received '" + routingKey + "':'" + > message + "'"); > > } > > } > > > The client above will receive the message when published into "amq.topic" > exchange by another AMQP client. > > Anything stupid and obvious I am missing? > > Dejan > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From spicoflorin at gmail.com Thu Apr 17 09:15:13 2014 From: spicoflorin at gmail.com (spicoflorin at gmail.com) Date: Thu, 17 Apr 2014 01:15:13 -0700 (PDT) Subject: [rabbitmq-discuss] Understanding the topic exchange behavior Message-ID: Hello! In the tutorial about the topic exchange http://www.rabbitmq.com/tutorials/tutorial-five-java.html it is mentioned "When a queue is bound with "#" (hash) binding key - it will receive all the messages, regardless of the routing key - like in fanout exchange. When special characters "*" (star) and "#" (hash) aren't used in bindings, the topic exchange will behave just like a direct one." The spotted issue is related with the last statement "When special characters "*" (star) and "#" (hash) aren't used in bindings, the topic exchange will behave just like a direct one." Here is the scenario that contradicts this statement. Suppose that you have: - a producer that is sending messages using routing key "kern.critical" -two consumers that are consuming using the same routing key "kern.critical". In my understanding, the behavior for this case should be, that the consumers will consume in a round robin fashion messages (direct exchange). In reality, both consumers will receive the same sent messages, and that is not the case of direct exchange contract. 1. Is this an issue of the documentation or my understanding is wrong? 2. When using the topic exchange, all the consumers that are listening to a specific bounded queue will receive the same messages. In the given example from the tutorial, if Q1 has two consumers, then both of them will receive same sent messages. Is it possible when have multiple consumers to consume in a round robin fashion? I look forward for your answers. Thank you very much. Regards, Florin -------------- next part -------------- An HTML attachment was scrubbed... URL: From mklishin at gopivotal.com Thu Apr 17 09:25:15 2014 From: mklishin at gopivotal.com (Michael Klishin) Date: Thu, 17 Apr 2014 12:25:15 +0400 Subject: [rabbitmq-discuss] Understanding the topic exchange behavior In-Reply-To: References: Message-ID: On 17 April 2014 at 12:17:27, spicoflorin at gmail.com (spicoflorin at gmail.com) wrote: > > > Suppose that you have: > > > - a producer that is sending messages using routing key "kern.critical" > > > -two consumers that are consuming using the same routing key > "kern.critical". > > > In my understanding, the behavior for this case should be, that > the consumers will consume in a round robin fashion messages > (direct exchange). In reality, both consumers will receive > the same sent messages, and that is not the case of direct exchange > contract. > > > 1. Is this an issue of the documentation or my understanding is > wrong? You are confusing *routing* and consumption. Routing keys used by queue.bind are used to distribute messages between queues. Then consumers consume them in the round robin fashion (given that both use channels with qos = 1). It?s true that direct exchange can route a message to multiple queues. > 2. When using the topic exchange, all the consumers that are listening > to a specific bounded queue will receive the same messages. In > the given example from the tutorial, if Q1 has two consumers, > then both of them will receive same sent messages. If a queue has two consumers, they *will not* receive the same message. Again, you are confusing routing and consumption See [1]. 1.?http://www.rabbitmq.com/tutorials/amqp-concepts.html? -- MK Software Engineer, Pivotal/RabbitMQ From mklishin at gopivotal.com Thu Apr 17 09:27:40 2014 From: mklishin at gopivotal.com (Michael Klishin) Date: Thu, 17 Apr 2014 12:27:40 +0400 Subject: [rabbitmq-discuss] Cannot get MQTT->AMQP to work In-Reply-To: References: <1cbd1e1b-5957-4340-9cce-b6f062306b81@googlegroups.com> Message-ID: On 17 April 2014 at 06:07:19, Dejan Glozic (dejan.glozic at gmail.com) wrote: > > Just realized the example is not clear - the value of 'queueName' > on the MQTT side is 'builds' - the same as the value for the 'routingKey' > on the AMQP side used for channel.queueBind. I just tried reproducing your problem with a couple of scripts and my consumer receives the messages published over MQTT: https://gist.github.com/michaelklishin/fb4ae5a53a537cc73e4f Use RabbitMQ management UI to inspect exchange message rates, bindings and consumers. -- MK Software Engineer, Pivotal/RabbitMQ From simon at rabbitmq.com Thu Apr 17 12:55:12 2014 From: simon at rabbitmq.com (Simon MacMullen) Date: Thu, 17 Apr 2014 12:55:12 +0100 Subject: [rabbitmq-discuss] Supporting both SSL+Password as well as SSL+Certificate based authentication In-Reply-To: <1397684335559-34914.post@n5.nabble.com> References: <1397684335559-34914.post@n5.nabble.com> Message-ID: <534FC120.5050504@rabbitmq.com> On 16/04/2014 22:38, vish.ramachandran wrote: > It does not seem right to mandate that password based clients also present > valid certificate. If they could, then there is no need for password based > authentication. I'm afraid that is the requirement at the moment. A future release may improve this situation. Cheers, Simon From simon at rabbitmq.com Thu Apr 17 13:01:56 2014 From: simon at rabbitmq.com (Simon MacMullen) Date: Thu, 17 Apr 2014 13:01:56 +0100 Subject: [rabbitmq-discuss] Testing Plugin / Internals Question In-Reply-To: References: Message-ID: <534FC2B4.3040300@rabbitmq.com> On 16/04/2014 18:46, Gavin M. Roy wrote: > I?m working on a plugin that will let me trigger some internal behaviors > in RabbitMQ from pika or any other RabbitMQ client for acceptance > testing purposes. In doing so, I?ve been trying to trace back how to > get the connection pid() from the channel that is passed as the sender > in the #delivery record passed into rabbit_exchange_type:route/2. > > When I?m issuing rabbit_channel:info(pid()) for the sender, it seems to > hang and I was hoping that was where I could get the connection pid. It's not completely clear to me what you are doing, but if this is an exchange type then bear in mind those callbacks are called by the channel - and having the channel call into itself will deadlock. > Does anyone have any suggestions as to where I can get that info? > Ultimately for this particular piece of functionality, I?m looking to > trigger rabbit_reader:conserve_resources/3 to force a Connection.Blocked > frame to be sent. Any alternative ways to trigger that from either the > exchange or an interceptor would be helpful as well. If you really want to do that, you could spin up a separate process that calls into the channel to get its connection pid and then call conserve_resources(). Personally I'd just trigger the condition broker-wide with "rabbitmqctl set_vm_memory_high_watermark 0.000001" or similar. Cheers, Simon From bison_art at o2.pl Thu Apr 17 14:32:10 2014 From: bison_art at o2.pl (rabbit15) Date: Thu, 17 Apr 2014 06:32:10 -0700 (PDT) Subject: [rabbitmq-discuss] RabbitMQ Cluster node In-Reply-To: <534E5776.90602@rabbitmq.com> References: <1397566372860-34893.post@n5.nabble.com> <534E5776.90602@rabbitmq.com> Message-ID: <1397741530680-34925.post@n5.nabble.com> Thank you ! Cheers, Artur -- View this message in context: http://rabbitmq.1065348.n5.nabble.com/RabbitMQ-Cluster-node-tp34893p34925.html Sent from the RabbitMQ mailing list archive at Nabble.com. From matthias at rabbitmq.com Thu Apr 17 14:56:32 2014 From: matthias at rabbitmq.com (Matthias Radestock) Date: Thu, 17 Apr 2014 14:56:32 +0100 Subject: [rabbitmq-discuss] Fwd: Performance bottleneck on inter-node connection In-Reply-To: References: <53465CF5.20607@rabbitmq.com> <5346C06C.8090609@rabbitmq.com> <534BA3C3.3080607@rabbitmq.com> Message-ID: <534FDD90.30805@rabbitmq.com> On 15/04/14 14:20, joseph rouphael wrote: > On Mon, Apr 14, 2014 at 12:00 PM, Matthias Radestock > > wrote: > What happens when you run the above test with both nodes on the same > machine? > > The result is almost the same: Fluctuating between 10 and 23KHZ Thanks for running all these tests. I've reproduced your observations. What we are seeing here is, at least partially, an issue with the Erlang scheduler. I noticed that when running the test with PerfTest, single producer, single consumer, two nodes, the consuming node was using 100% CPU, i.e. the equivalent of a single core. That is usually an indication of an Erlang scheduler issue since there is an appreciable degree of parallelism that should be exploitable. For this particular test I managed to double performance by supplying the "+swt very_low" or "+sfwi 1" flags to Erlang via RABBITMQ_SERVER_ERL_ARGS. So these flags, plus the other +s... flags of the Erlang VM, are worth playing with. Matthias. From Dayton_Gomez at symantec.com Thu Apr 17 14:58:12 2014 From: Dayton_Gomez at symantec.com (Dayton Gomez) Date: Thu, 17 Apr 2014 06:58:12 -0700 Subject: [rabbitmq-discuss] Defining Default Policy? Message-ID: Hello, I'm looking for a way to define a high-availability policy automatically. After much googling, I've not found an easy way to do so (like definitions in a config file). I do have some provisioning software, but was hoping for a native way to set things up. Any pointers are much appreciated, Dayton -------------- next part -------------- An HTML attachment was scrubbed... URL: From Anita.Yip at RealPage.com Thu Apr 17 17:48:04 2014 From: Anita.Yip at RealPage.com (Anita Yip) Date: Thu, 17 Apr 2014 16:48:04 +0000 Subject: [rabbitmq-discuss] ready-only access for users Message-ID: Hi, Is it possible to have read-only access permissions for a RabbitMQ user? If yes, can you please show me how to configure that. Basically, I want a ready-only user access, so that I will be able to monitor the stats on production rabbitmq management UI. The read-only user should not be able to create/update/delete anything related to rabbitmq. Thanks, Anita Yip ________________________________ This message is intended only for the use of the individual(s) or entity to which it is addressed and may contain information that is privileged, confidential, and/or proprietary to RealPage and its affiliated companies. If the reader of this message is not the intended recipient, you are hereby notified that any dissemination, distribution, forwarding or copying of this communication is prohibited without the express permission of the sender. If you have received this communication in error, please notify the sender immediately and delete the original message. -------------- next part -------------- An HTML attachment was scrubbed... URL: From mcintoshj at gmail.com Thu Apr 17 17:54:24 2014 From: mcintoshj at gmail.com (Jason McIntosh) Date: Thu, 17 Apr 2014 11:54:24 -0500 Subject: [rabbitmq-discuss] Fwd: Performance bottleneck on inter-node connection In-Reply-To: <534FDD90.30805@rabbitmq.com> References: <53465CF5.20607@rabbitmq.com> <5346C06C.8090609@rabbitmq.com> <534BA3C3.3080607@rabbitmq.com> <534FDD90.30805@rabbitmq.com> Message-ID: Watch also NUMA stuff on these bigger multi-core systems. I've seen some very interesting effects due to NUMA and how it affects erlang. Look at these posts for some more details: http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/2013-January/024790.html Typically a system with as many cores as you've got requires some more "tweaking" to get good performance out of it. One of the best articles I'd found was this: http://www.agoragames.com/blog/2011/06/24/of-penguins-rabbits-and-buses/ I don't THINK this would have an impact on your issues with inter-node communications but I have to toss it out there. Also, another fun thing we saw was long DNS lookups causing some odd issues as well. Jason On Thu, Apr 17, 2014 at 8:56 AM, Matthias Radestock wrote: > On 15/04/14 14:20, joseph rouphael wrote: > >> On Mon, Apr 14, 2014 at 12:00 PM, Matthias Radestock >> > wrote: >> What happens when you run the above test with both nodes on the same >> machine? >> >> The result is almost the same: Fluctuating between 10 and 23KHZ >> > > Thanks for running all these tests. I've reproduced your observations. > > What we are seeing here is, at least partially, an issue with the Erlang > scheduler. > > I noticed that when running the test with PerfTest, single producer, > single consumer, two nodes, the consuming node was using 100% CPU, i.e. the > equivalent of a single core. That is usually an indication of an Erlang > scheduler issue since there is an appreciable degree of parallelism that > should be exploitable. > > For this particular test I managed to double performance by supplying the > "+swt very_low" or "+sfwi 1" flags to Erlang via RABBITMQ_SERVER_ERL_ARGS. > > So these flags, plus the other +s... flags of the Erlang VM, are worth > playing with. > > > Matthias. > _______________________________________________ > rabbitmq-discuss mailing list > rabbitmq-discuss at lists.rabbitmq.com > https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss > -- Jason McIntosh https://github.com/jasonmcintosh/ 573-424-7612 -------------- next part -------------- An HTML attachment was scrubbed... URL: From tim at rabbitmq.com Thu Apr 17 18:21:17 2014 From: tim at rabbitmq.com (Tim Watson) Date: Thu, 17 Apr 2014 18:21:17 +0100 Subject: [rabbitmq-discuss] ready-only access for users In-Reply-To: References: Message-ID: <15545A0D-0A0D-4E95-89D9-3C6A5AAC046B@rabbitmq.com> All the details for setting up permissions are outlined in http://www.rabbitmq.com/access-control.html. The additional access controls offered for management users are outlined in http://www.rabbitmq.com/management.html#permissions. I'm not sure you can tweak things to do exactly what you're asking for, but in terms of CRUD operations, 'management' users can only delete/close their own channels and connections (i.e., those that user established themselves) in addition to whatever AMQP permissions you grant them. Cheers, Tim On 17 Apr 2014, at 17:48, Anita Yip wrote: > Hi, > Is it possible to have read-only access permissions for a RabbitMQ user? If yes, can you please show me how to configure that. Basically, I want a ready-only user access, so that I will be able to monitor the stats on production rabbitmq management UI. The read-only user should not be able to create/update/delete anything related to rabbitmq. > Thanks, > Anita Yip > > > > > This message is intended only for the use of the individual(s) or entity to which it is addressed and may contain information that is privileged, confidential, and/or proprietary to RealPage and its affiliated companies. If the reader of this message is not the intended recipient, you are hereby notified that any dissemination, distribution, forwarding or copying of this communication is prohibited without the express permission of the sender. If you have received this communication in error, please notify the sender immediately and delete the original message. > _______________________________________________ > rabbitmq-discuss mailing list > rabbitmq-discuss at lists.rabbitmq.com > https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: From matthias at rabbitmq.com Thu Apr 17 18:28:33 2014 From: matthias at rabbitmq.com (Matthias Radestock) Date: Thu, 17 Apr 2014 18:28:33 +0100 Subject: [rabbitmq-discuss] ready-only access for users In-Reply-To: <15545A0D-0A0D-4E95-89D9-3C6A5AAC046B@rabbitmq.com> References: <15545A0D-0A0D-4E95-89D9-3C6A5AAC046B@rabbitmq.com> Message-ID: <53500F41.9030707@rabbitmq.com> On 17/04/14 18:21, Tim Watson wrote: > All the details for setting up permissions are outlined in > http://www.rabbitmq.com/access-control.html. The additional access > controls offered for management users are outlined in > http://www.rabbitmq.com/management.html#permissions. I'm not sure you > can tweak things to do exactly what you're asking for, but in terms of > CRUD operations, 'management' users can only delete/close their own > channels and connections (i.e., those that user established themselves) > in addition to whatever AMQP permissions you grant them. A user with with the 'monitoring' tag and no permissions in any vhost should provide exactly what OP asked for. Matthias. From dejan at ca.ibm.com Thu Apr 17 19:47:51 2014 From: dejan at ca.ibm.com (Dejan Glozic) Date: Thu, 17 Apr 2014 14:47:51 -0400 Subject: [rabbitmq-discuss] Cannot get MQTT->AMQP to work In-Reply-To: References: <1cbd1e1b-5957-4340-9cce-b6f062306b81@googlegroups.com> Message-ID: An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/jpeg Size: 518 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/jpeg Size: 638 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/jpeg Size: 542 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/gif Size: 1851 bytes Desc: not available URL: From jaishreegn at gmail.com Thu Apr 17 22:34:14 2014 From: jaishreegn at gmail.com (jayashree gn) Date: Thu, 17 Apr 2014 15:34:14 -0600 Subject: [rabbitmq-discuss] cluster_status changes with rabbitmq - 3.3.0 Message-ID: Hi All, On new version of rabbitmq 3.3.0, the rabbitmqctl cluster_status commad looks like this Cluster status of node 'rabbit at ec2-54-196-175-155' ... [{nodes,[{disc,['rabbit at rabbitmq_node01','rabbitmq_node02', 'rabbitmq_node03']}]}, {running_nodes,['rabbitmq_node01','rabbitmq_node02', 'rabbitmq_node03']}, {cluster_name,<<"rabbitmq_node01">>}, {partitions,[]}] ...done. What is the new addition {cluster_name,<<"rabbitmq_node01">>} to the status?? All i can infer is its the master node that the slave nodes connect to. But if the master node is terminated, then it still renames in the {cluster_name,<<"rabbitmq_node01">>}, Could not find any information on this new chnange in documentation for What?s new in RabbitMQ 3.0 Anybody familiar with what this new change meant to do? Thank You!! -------------- next part -------------- An HTML attachment was scrubbed... URL: From jaishreegn at gmail.com Thu Apr 17 22:36:59 2014 From: jaishreegn at gmail.com (jayashree gn) Date: Thu, 17 Apr 2014 15:36:59 -0600 Subject: [rabbitmq-discuss] cluster_status changes with rabbitmq - 3.3.0 In-Reply-To: References: Message-ID: > Hi All, > On new version of rabbitmq 3.3.0, the rabbitmqctl cluster_status commad > looks like this > > Cluster status of node 'rabbit at ec2-54-196-175-155' ... > [{nodes,[{disc,['rabbit at rabbitmq_node01','rabbitmq_node02', > 'rabbitmq_node03']}]}, > {running_nodes,['rabbitmq_node01','rabbitmq_node02', > 'rabbitmq_node03']}, > {cluster_name,<<"rabbitmq_node01">>}, > {partitions,[]}] > ...done. > > What is the new addition {cluster_name,<<"rabbitmq_node01">>} to the > status?? All i can infer is its the master node that the slave nodes > connect to. > But if the master node is terminated, then it still renames in the > {cluster_name,<<"rabbitmq_node01">>}, > > Could not find any information on this new chnange in documentation for What?s > new in RabbitMQ 3.0 > > Anybody familiar with what this new change meant to do? > > Thank You!! > -------------- next part -------------- An HTML attachment was scrubbed... URL: From matthias at rabbitmq.com Thu Apr 17 22:54:42 2014 From: matthias at rabbitmq.com (Matthias Radestock) Date: Thu, 17 Apr 2014 22:54:42 +0100 Subject: [rabbitmq-discuss] cluster_status changes with rabbitmq - 3.3.0 In-Reply-To: References: Message-ID: <53504DA2.2000601@rabbitmq.com> On 17/04/14 22:34, jayashree gn wrote: > What is the new addition {cluster_name,<<"rabbitmq_node01">>} to the > status?? http://www.rabbitmq.com/man/rabbitmqctl.1.man.html#set_cluster_name From Jesse.Smithnosky at emc.com Fri Apr 18 01:15:41 2014 From: Jesse.Smithnosky at emc.com (Smithnosky, Jesse) Date: Thu, 17 Apr 2014 20:15:41 -0400 Subject: [rabbitmq-discuss] Dynamic Shovels Message-ID: Hi Everyone, I'm trying to get the new (in version 3.3) Dynamic Shovels feature up and running, but I'm seeing some weird behavior using curl and not having much luck setting it up in java either. Here is the curl command I am trying: curl -i -u guest:guest -H "content-type:application/json" -XPUT -d'{"value":{"src-uri":"amqp://","src-queue":"my-queue","dest-uri":"amqp://remote-server","dest-queue":"another-queue"}}' http://localhost:15672/api/parameters/shovel/%2f/edShovel HTTP/1.1 204 No Content Server: MochiWeb/1.1 WebMachine/1.10.0 (never breaks eye contact) Date: Thu, 17 Apr 2014 23:16:58 GMT Content-Type: application/json Content-Length: 0 It appears that the shovel is getting created correctly, but a 204 is coming back. This is really just a proof of concept, but it has me a little worried about whether or not I can get it working in Java. I'm trying to do something like this: String jsonString = jackson.writeValueAsString(shovel); HttpClient client = HttpClientBuilder.create().build(); HttpPost post = new HttpPost(RABBIT_URL_BASE+vhost+"/"+shovelName); post.setEntity(new StringEntity(jsonString)); post.addHeader("content-type", "application/json"); String userPassword = "guest:guest"; String encoding = DatatypeConverter.printBase64Binary(userPassword.getBytes("UTF-8")); post.setHeader( "Authorization", "Basic " + encoding ); HttpResponse response = client.execute(post); No matter what I seem to do, however, I'm getting back a 405 Method not allowed response. Has anyone gotten these working yet? Am I doing something obvious incorrectly? Any help is appreciated. Thanks, Jesse -------------- next part -------------- An HTML attachment was scrubbed... URL: From joey.jiaojg at gmail.com Fri Apr 18 03:13:39 2014 From: joey.jiaojg at gmail.com (Joey Jiao) Date: Fri, 18 Apr 2014 10:13:39 +0800 Subject: [rabbitmq-discuss] Exchange.Unknown Message-ID: Hi, I received unknown amqp package (40, 41) and (40, 51) (40, 41) is exchange.unbind_ok in celery exception defination. Any idea on (40,51)? Should I upgrade amqp protocol or rabbitmq only supports 0.9.1 right now? -- -Joey Jiao -------------- next part -------------- An HTML attachment was scrubbed... URL: From srikanthtns at gmail.com Fri Apr 18 07:27:52 2014 From: srikanthtns at gmail.com (srikanth tns) Date: Thu, 17 Apr 2014 23:27:52 -0700 Subject: [rabbitmq-discuss] RabbitMQ Clustering and HA Approach Message-ID: Hi We have installed rabbitmq 3.2.4 for using it with mcollective architecture . We have setup the rabbitmq cluster and set up the HA mirrored queues. Currently in PROD, we have 15000 nodes in the network trying to use rabbitmq for rpc communication and they might increase . The questions I have is 1) we are planning to have 3 node cluster . It appears like all the queues are connected to the master. Will all these queue connections create load on master rabbitmq ? for ex , 15000 tcp connections to same port/ip. or will the other nodes in the cluster be able to handle the load and the messages from mcollective ? can you brief me how the messages are routed to the queues within the nodes ? 2) we have HA'ed the queues with ha_mode=all , for these 3 nodes , will this cause any performance issue ? if the master is down and come over a new slave node to the cluster , will it handle the load as before ? 3) what is the scalability solution we need to look into incase we have more connections coming into the application ? what we are looking for is a better approach to load balance rabbitmq, its queues and manage HA correctly. Thanks Srikanth -------------- next part -------------- An HTML attachment was scrubbed... URL: From petar.georgie at gmail.com Fri Apr 18 08:41:40 2014 From: petar.georgie at gmail.com (Petar Georgiev) Date: Fri, 18 Apr 2014 00:41:40 -0700 (PDT) Subject: [rabbitmq-discuss] reciving duplicate messages Message-ID: Hello all, I'm using mirrored rabbit mq servers. Sometimes on high load i recive duplicate massage - with same delivery tag. I would like to know what is the reason for recieving duplicate messages? When acknowledge a message twice i got shut down signal and the consumer is disconected. Is there some good practice for this situation? Best Regards, Petar -------------- next part -------------- An HTML attachment was scrubbed... URL: From mklishin at gopivotal.com Fri Apr 18 09:20:18 2014 From: mklishin at gopivotal.com (Michael Klishin) Date: Fri, 18 Apr 2014 12:20:18 +0400 Subject: [rabbitmq-discuss] Exchange.Unknown In-Reply-To: References: Message-ID: On 18 April 2014 at 06:16:11, Joey Jiao (joey.jiaojg at gmail.com) wrote: > > (40, 41) is exchange.unbind_ok in celery exception defination. > Any idea on (40,51)? {40, 51} is exchange.unbind-ok in 0-9-1 . {40, 41} does not exist (exchange.unbind is {40, 40}). Since exchange-to-exchange bindings is a RabbitMQ extensions to AMQO 0-9-1, you must use a 0-9-1 client in order to use it. -- MK Software Engineer, Pivotal/RabbitMQ From mklishin at gopivotal.com Fri Apr 18 09:24:59 2014 From: mklishin at gopivotal.com (Michael Klishin) Date: Fri, 18 Apr 2014 12:24:59 +0400 Subject: [rabbitmq-discuss] Dynamic Shovels In-Reply-To: References: Message-ID: On 18 April 2014 at 04:17:02, Smithnosky, Jesse (jesse.smithnosky at emc.com) wrote: > > > curl -i -u guest:guest -H "content-type:application/json" > -XPUT -d'{"value":{"src-uri":"amqp://","src-queue":"my-queue","dest-uri":"amqp://remote-server","dest-queue":"another-queue"}}' > http://localhost:15672/api/parameters/shovel/%2f/edShovel -XPUT means you use HTTP method PUT. ? > > > HTTP/1.1 204 No Content > > > Server: MochiWeb/1.1 WebMachine/1.10.0 (never breaks eye > contact) > > > Date: Thu, 17 Apr 2014 23:16:58(http://airmail.calendar/2014-04-17%2023:16:58%20GMT+4) > GMT > > > Content-Type: application/json > > > Content-Length: 0 > > > > > > It appears that the shovel is getting created correctly, but > a 204 is coming back. 204 is a successful response meaning ?No Content?. ?This is really just a proof of concept, but > it has me a little worried about whether or not I can get it working > in Java. I?m trying to do something like this: > > > > > > String jsonString = jackson.writeValueAsString(shovel); > > > HttpClient client = HttpClientBuilder.create().build(); > > > HttpPost post = new HttpPost(RABBIT_URL_BASE+vhost+"/"+shovelName); > > > post.setEntity(new StringEntity(jsonString)); > > > post.addHeader("content-type", "application/json"); > > > String userPassword = "guest:guest"; > > > String encoding = DatatypeConverter.printBase64Binary(userPassword.getBytes("UTF-8")); > > > post.setHeader( "Authorization", "Basic " + encoding ); > > > HttpResponse response = client.execute(post); > > > > > > No matter what I seem to do, however, I?m getting back a 405 Method > not allowed response. Has anyone gotten these working yet? Am > I doing something obvious incorrectly? Any help is appreciated. > You use POST in your Java code but PUT with curl. HTTP API expects a PUT and rejects POSTs.? -- MK Software Engineer, Pivotal/RabbitMQ From mklishin at gopivotal.com Fri Apr 18 09:31:06 2014 From: mklishin at gopivotal.com (Michael Klishin) Date: Fri, 18 Apr 2014 12:31:06 +0400 Subject: [rabbitmq-discuss] RabbitMQ Clustering and HA Approach In-Reply-To: References: Message-ID: On 18 April 2014 at 10:30:08, srikanth tns (srikanthtns at gmail.com) wrote: > > 1) we are planning to have 3 node cluster . It appears like all > the queues are connected to the master. Will all these queue connections > create load on master rabbitmq ? for ex , 15000 tcp connections > to same port/ip. > or will the other nodes in the cluster be able to handle the load > and the messages from mcollective ? can you brief me how the messages > are routed to the queues within the nodes ? 15K connections to a single node is not very many but your client can connect to any node, so it can be, say, 5K per node. Make sure your OS settings allow RabbitMQ to use enough file descriptors. > 2) we have HA'ed the queues with ha_mode=all , for these 3 nodes > , will this cause any performance issue ? if the master is down > and come over a new slave node to the cluster , will it handle the > load as before ? Mirroring to 3 nodes requires RabbitMQ to do more work than mirroring to 2 or 1 node. However, what matters more is whether your consumers connect to queue masters. Every queue in a cluster has a master node and other nodes forward requests (that involve the queue) to it. Publishing and consuming from master improves data locality but also requires your clients to be a bit smarter when reconnecting. Queue master can be determined using HTTP API. > 3) what is the scalability solution we need to look into incase > we have more connections coming into the application ? You can simply add nodes and distribute connections between them roughly evenly. It?s not uncommon to put a load balancer in front of a cluster to help with this. It really depends on if your message rates and latency requirements require publishing and consuming from master. Simply distributing connections between node is pretty trivial. HTH.? -- MK Software Engineer, Pivotal/RabbitMQ From mklishin at gopivotal.com Fri Apr 18 09:33:01 2014 From: mklishin at gopivotal.com (Michael Klishin) Date: Fri, 18 Apr 2014 12:33:01 +0400 Subject: [rabbitmq-discuss] reciving duplicate messages In-Reply-To: References: Message-ID: On 18 April 2014 at 11:43:51, Petar Georgiev (petar.georgie at gmail.com) wrote: > > I'm using mirrored rabbit mq servers. Sometimes on high load > i recive duplicate massage - with same delivery tag. I would like > to know what is the reason for recieving duplicate messages? > When acknowledge a message twice i got shut down signal and the > consumer is disconected. Is there some good practice for this > situation? Applications need to be prepared to handle redeliveries because they are inevitable in certain scenarios that involve consumer and node failures. Redelivered messages have ?redelivered? property set to true on them.? -- MK Software Engineer, Pivotal/RabbitMQ From matthias at rabbitmq.com Fri Apr 18 09:44:31 2014 From: matthias at rabbitmq.com (Matthias Radestock) Date: Fri, 18 Apr 2014 09:44:31 +0100 Subject: [rabbitmq-discuss] reciving duplicate messages In-Reply-To: References: Message-ID: <5350E5EF.1010709@rabbitmq.com> On 18/04/14 09:33, Michael Klishin wrote: > On 18 April 2014 at 11:43:51, Petar Georgiev > (petar.georgie at gmail.com) wrote: >>> I'm using mirrored rabbit mq servers. Sometimes on high load >> i recive duplicate massage - with same delivery tag. I would like >> to know what is the reason for recieving duplicate messages? When >> acknowledge a message twice i got shut down signal and the consumer >> is disconected. Is there some good practice for this situation? > > Applications need to be prepared to handle redeliveries because they > are inevitable in certain scenarios that involve consumer and node > failures. Redelivered messages have ?redelivered? property set to > true on them. However, you should certainly not receive multiple messages with the same delivery tag *on the same channel*. The odds of the RabbitMQ server having a bug in this regard are close to zero; most likely there is a bug in your application code that conflates deliveries from multiple channels, possibly after a reconnection. Matthias. From bra at fsn.hu Fri Apr 18 10:11:09 2014 From: bra at fsn.hu (Nagy, Attila) Date: Fri, 18 Apr 2014 11:11:09 +0200 Subject: [rabbitmq-discuss] Federated queues with durable topic subscriptions over STOMP and ACLs Message-ID: <5350EC2D.1030602@fsn.hu> Hi, I use (shared) durable subscriptions to topics via STOMP, which I would like to federate between rabbitmq nodes with the semantics offered by queue federation and durable topics. And I would like to use SSL certificate based ACLs for those topics, so a given user should only access /topic/routing.key (eg: can only read or write from/to /topic/accounting.* or /topic/accounting.company1.# etc). I can use SSL. I can use federation (by federating stomp-dsub queues). Everything works. Except I can't use durable topics with ACLs. What I could find so far is this post: http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/2011-March/011762.html which states that I should use an exchange, but with that I can't use durable subscriptions. Is this possible somehow with rabbitmq? It seems the only missing piece here is to extend permissions to routing keys. Or is there another way to do this, without topics? Thanks, From mklishin at gopivotal.com Fri Apr 18 10:22:40 2014 From: mklishin at gopivotal.com (Michael Klishin) Date: Fri, 18 Apr 2014 13:22:40 +0400 Subject: [rabbitmq-discuss] Federated queues with durable topic subscriptions over STOMP and ACLs In-Reply-To: <5350EC2D.1030602@fsn.hu> References: <5350EC2D.1030602@fsn.hu> Message-ID: On 18 April 2014 at 13:12:36, Nagy, Attila (bra at fsn.hu) wrote: > > I can use SSL. I can use federation (by federating stomp-dsub > queues). > Everything works. > Except I can't use durable topics with ACLs. > > What I could find so far is this post: > http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/2011-March/011762.html > which states that I should use an exchange, but with that I can't > use > durable subscriptions. > > Is this possible somehow with rabbitmq? > It seems the only missing piece here is to extend permissions > to routing > keys. > Or is there another way to do this, without topics? I?m afraid what Rob said 3 years ago is still true today. You can try to work around it using exchange-to-exchange bindings but that is ?* Not something that exists in STOMP, can be done in STOMP or fits its semantics well ?* More complicated to debug ?* Less efficient So, for every topic you?d declare a fanout exchange, bind it to amq.topic and try durable STOMP subscriptions while publishing to the exchanges that represent topics. This e2e stuff is largely static and can be done once by a small service that is started before the rest of your system. It *should* work although I haven?t tried it on my end.? For more info see Durable Topic Subscriptions in [1] and [2]. 1.?http://www.rabbitmq.com/stomp.html 2.?http://www.rabbitmq.com/e2e.html -- MK Software Engineer, Pivotal/RabbitMQ From bra at fsn.hu Fri Apr 18 10:37:36 2014 From: bra at fsn.hu (Nagy, Attila) Date: Fri, 18 Apr 2014 11:37:36 +0200 Subject: [rabbitmq-discuss] Federated queues with durable topic subscriptions over STOMP and ACLs In-Reply-To: References: <5350EC2D.1030602@fsn.hu> Message-ID: <5350F260.4070504@fsn.hu> On 04/18/14 11:22, Michael Klishin wrote: > On 18 April 2014 at 13:12:36, Nagy, Attila (bra at fsn.hu) wrote: >>> I can use SSL. I can use federation (by federating stomp-dsub >> queues). >> Everything works. >> Except I can't use durable topics with ACLs. >> >> What I could find so far is this post: >> http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/2011-March/011762.html >> which states that I should use an exchange, but with that I can't >> use >> durable subscriptions. >> >> Is this possible somehow with rabbitmq? >> It seems the only missing piece here is to extend permissions >> to routing >> keys. >> Or is there another way to do this, without topics? > I?m afraid what Rob said 3 years ago is still true today. You can try to work > around it using exchange-to-exchange bindings but that is Will try to understand this, but until that a new question has arisen in my mind: from the STOMP page it seems amq.topic has a special meaning in STOMP, the name is what triggers this function: "For SUBSCRIBE frames, a shared queue is created for each distinct subscription ID x destination pair, and bound to the amq.topic exchange with routing key . A subscription is created against the queue." Wouldn't be possible to extend this to take the exchange's type into account as well? I mean, would it work to create a new topic exchange and when I durably subscribe to /exchange/myownexchange.routing.key from STOMP, it would be treated the same as I would subscribe to a topic durably if that exchange is of topic type? From mklishin at gopivotal.com Fri Apr 18 10:42:09 2014 From: mklishin at gopivotal.com (Michael Klishin) Date: Fri, 18 Apr 2014 13:42:09 +0400 Subject: [rabbitmq-discuss] Federated queues with durable topic subscriptions over STOMP and ACLs In-Reply-To: <5350F260.4070504@fsn.hu> References: <5350EC2D.1030602@fsn.hu> <5350F260.4070504@fsn.hu> Message-ID: On 18 April 2014 at 13:38:52, Nagy, Attila (bra at fsn.hu) wrote: > > Will try to understand this, but until that a new question has > arisen in > my mind: > from the STOMP page it seems amq.topic has a special meaning in > STOMP, > the name is what triggers this function: > "For SUBSCRIBE frames, a shared queue is created for each distinct > subscription ID x destination pair, and bound to the amq.topic > exchange > with routing key . A subscription is created against the > queue." > > Wouldn't be possible to extend this to take the exchange's type > into > account as well? > > I mean, would it work to create a new topic exchange and when I durably > subscribe to /exchange/myownexchange.routing.key from STOMP, > it would be > treated the same as I would subscribe to a topic durably if that > exchange is of topic type? I recommend that you read http://www.rabbitmq.com/tutorials/amqp-concepts.html You do not subscribe to exchanges in AMQP 0-9-1 and the STOMP plugin is built on top of AMQP 0-9-1 entities. What you need is a layer (exchanges) to enforce ACL. From there messages should be routed exactly the same as STOMP plugin with durable subscriptions would route them, that?s why I recommended fanout exchanges and e2e: this is a way to make messages flowing into N exchanges to be routed to amq.topic. From there it should work just as described in the durable subscriptions section. -- MK Software Engineer, Pivotal/RabbitMQ From mklishin at gopivotal.com Fri Apr 18 11:03:45 2014 From: mklishin at gopivotal.com (Michael Klishin) Date: Fri, 18 Apr 2014 14:03:45 +0400 Subject: [rabbitmq-discuss] Federated queues with durable topic subscriptions over STOMP and ACLs In-Reply-To: <5350F712.6080203@fsn.hu> References: <5350EC2D.1030602@fsn.hu> <5350F260.4070504@fsn.hu> <5350F712.6080203@fsn.hu> Message-ID: On 18 April 2014 at 13:57:41, Nagy, Attila (bra at fsn.hu) wrote: Please keep rabbitmq-discuss on CC.? > > What I'm trying to ask (sorry if I can't describe it properly) > is: > wouldn't it work if the STOMP adapter would check the exchange's > type > when evaluating the subscription and if it's a topic and persistent > is > true, it would do the same: > bind a named queue to the exchange (here, a custom exchange, not > an > amq.*) with the routing key specified. Perhaps that could work but it may be too specific a use case to make STOMP plugin work like that. Plus it will take a while for a new version to come out (a few months). > > What you need is a layer (exchanges) to enforce ACL. From there > messages > > should be routed exactly the same as STOMP plugin with durable > subscriptions > > would route them, that?s why I recommended fanout exchanges > and e2e: this is > > a way to make messages flowing into N exchanges to be routed to > amq.topic > Which may solve my problem on the publisher side if I got it right, > but > what happens on the consumer side? > I think I still need to give configure rights to them onto amq.topic > to > make the binding. As far as I understand your problem only exists on the publisher side? Nothing changes for the consumer. If you need to have ACL for consumers, too, given you know queue names upfront you can use the features described on http://www.rabbitmq.com/access-control.html -- MK Software Engineer, Pivotal/RabbitMQ From bra at fsn.hu Fri Apr 18 11:34:56 2014 From: bra at fsn.hu (Nagy, Attila) Date: Fri, 18 Apr 2014 12:34:56 +0200 Subject: [rabbitmq-discuss] Federated queues with durable topic subscriptions over STOMP and ACLs In-Reply-To: References: <5350EC2D.1030602@fsn.hu> <5350F260.4070504@fsn.hu> <5350F712.6080203@fsn.hu> Message-ID: <5350FFD0.1090306@fsn.hu> On 04/18/14 12:03, Michael Klishin wrote: > On 18 April 2014 at 13:57:41, Nagy, Attila (bra at fsn.hu) wrote: > > Please keep rabbitmq-discuss on CC. Sorry, it seems my shift key is defunct. > >>> What I'm trying to ask (sorry if I can't describe it properly) >> is: >> wouldn't it work if the STOMP adapter would check the exchange's >> type >> when evaluating the subscription and if it's a topic and persistent >> is >> true, it would do the same: >> bind a named queue to the exchange (here, a custom exchange, not >> an >> amq.*) with the routing key specified. > Perhaps that could work but it may be too specific a use case to > make STOMP plugin work like that. Plus it will take a while for > a new version to come out (a few months). I'm surprised this counts as too specific. This can be done programatically in AMQP by "manually" declaring and binding exchanged and queues, but can't be done via STOMP. I agree that it can be complex to implement. >> amq.topic >> Which may solve my problem on the publisher side if I got it right, >> but >> what happens on the consumer side? >> I think I still need to give configure rights to them onto amq.topic >> to >> make the binding. > As far as I understand your problem only exists on the publisher side? > Nothing changes for the consumer. If you need to have ACL for consumers, too, > given you know queue names upfront you can use the features described on > > http://www.rabbitmq.com/access-control.html > No, the consumer side is also problematic. First of all, the queue name is generated, but that could be worked around by first checking what rabbitmq gives and then set the ACLs accordingly. The problem here is the binding is always done to amq.topic, so I have to give more rights to the consumer that would otherwise needed (bind to the named exchange, or to amq.topic's given routing key, if ACLs could go into that property). I guess it's easier for me now if I make this declarations and binds in the brokers, also, this makes it possible to properly ACL the consumers. Thank you very much for your quick responses! From joey.jiaojg at gmail.com Fri Apr 18 12:36:21 2014 From: joey.jiaojg at gmail.com (Joey Jiao) Date: Fri, 18 Apr 2014 19:36:21 +0800 Subject: [rabbitmq-discuss] Exchange.Unknown In-Reply-To: References: Message-ID: Thanks, it's (40,40), typo. I'm using celery as the client. It should also support these two msg_id. but the bin/celery might has the issue which doesn't include exchange.unbind and unbind_ok. Might be better check with them. 2014-04-18 16:20 GMT+08:00 Michael Klishin : > On 18 April 2014 at 06:16:11, Joey Jiao (joey.jiaojg at gmail.com) wrote: > > > (40, 41) is exchange.unbind_ok in celery exception defination. > > Any idea on (40,51)? > > {40, 51} is exchange.unbind-ok in 0-9-1 . {40, 41} does not exist > (exchange.unbind is {40, 40}). > > Since exchange-to-exchange bindings is a RabbitMQ extensions to AMQO 0-9-1, > you must use a 0-9-1 client in order to use it. > -- > MK > > Software Engineer, Pivotal/RabbitMQ > -- -Joey Jiao -------------- next part -------------- An HTML attachment was scrubbed... URL: From mklishin at gopivotal.com Fri Apr 18 12:41:07 2014 From: mklishin at gopivotal.com (Michael Klishin) Date: Fri, 18 Apr 2014 15:41:07 +0400 Subject: [rabbitmq-discuss] Exchange.Unknown with Celery In-Reply-To: References: Message-ID: On 18 April 2014 at 15:36:42, Joey Jiao (joey.jiaojg at gmail.com) wrote: > > I'm using celery as the client. It should also support these > two msg_id. > but the bin/celery might has the issue which doesn't include > exchange.unbind and unbind_ok. Might be better check with them. Perhaps Ask (Celery author) can shed some light on what client Celery uses and what protocol version it targets. Meanwhile, can you post something that explains how you run Celery and what the error message looks like?? -- MK Software Engineer, Pivotal/RabbitMQ From sp345s at att.com Fri Apr 18 13:02:57 2014 From: sp345s at att.com (PATAR, SAGAR) Date: Fri, 18 Apr 2014 12:02:57 +0000 Subject: [rabbitmq-discuss] Federation Queue / Exchange Message-ID: <59BF56992750434F89F8EBE141E64B1D01875474@GAALPA1MSGUSR9C.ITServices.sbc.com> Greetings .. We have 2 separate clustered environments with 2 nodes each and have consumers on both these clusters listening on specific header property values ... We want to forward messages NOT consumed on one cluster/node to other cluster if the messages are not consumed for more than a specific time say 30 sec or so .. Looks like federation queues are close to our requirement.... But not sure if we can forward only messages which are NOT consumed to other cluster .. Pl. suggest -------------- next part -------------- An HTML attachment was scrubbed... URL: From sp345s at att.com Fri Apr 18 13:07:06 2014 From: sp345s at att.com (PATAR, SAGAR) Date: Fri, 18 Apr 2014 12:07:06 +0000 Subject: [rabbitmq-discuss] JMS API Message-ID: <59BF56992750434F89F8EBE141E64B1D0187548C@GAALPA1MSGUSR9C.ITServices.sbc.com> Greetings .. Can we get a trail version of the spring / rabbitmq / JMS libraries and some examples , for our JMS clients to connect to Rabbit MQ Broker /Exchanges .. -------------- next part -------------- An HTML attachment was scrubbed... URL: From mklishin at gopivotal.com Fri Apr 18 13:43:03 2014 From: mklishin at gopivotal.com (Michael Klishin) Date: Fri, 18 Apr 2014 16:43:03 +0400 Subject: [rabbitmq-discuss] Federation Queue / Exchange In-Reply-To: <59BF56992750434F89F8EBE141E64B1D01875474@GAALPA1MSGUSR9C.ITServices.sbc.com> References: <59BF56992750434F89F8EBE141E64B1D01875474@GAALPA1MSGUSR9C.ITServices.sbc.com> Message-ID: On 18 April 2014 at 16:04:44, PATAR, SAGAR (sp345s at att.com) wrote: > > Looks like federation queues are close to our requirement?. > But not sure if we can forward only messages which are NOT consumed > to other cluster Federated queues are indeed close and favour local consumers but the exact message distribution logic may or may not be what you expect. Give it a try.? -- MK Software Engineer, Pivotal/RabbitMQ From japperly at gopivotal.com Fri Apr 18 14:53:50 2014 From: japperly at gopivotal.com (Jim Apperly) Date: Fri, 18 Apr 2014 14:53:50 +0100 Subject: [rabbitmq-discuss] JMS API In-Reply-To: <59BF56992750434F89F8EBE141E64B1D0187548C@GAALPA1MSGUSR9C.ITServices.sbc.com> References: <59BF56992750434F89F8EBE141E64B1D0187548C@GAALPA1MSGUSR9C.ITServices.sbc.com> Message-ID: You can download a 60 day evaluation of the commercial packages from here: https://network.gopivotal.com/products/pivotal-rabbitmq Thanks Jim On Friday, 18 April 2014, PATAR, SAGAR wrote: > Greetings .. > > > > Can we get a trail version of the spring / rabbitmq / JMS libraries and > some examples , for our JMS clients to connect to Rabbit MQ Broker > /Exchanges .. > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From matthias at rabbitmq.com Fri Apr 18 14:55:04 2014 From: matthias at rabbitmq.com (Matthias Radestock) Date: Fri, 18 Apr 2014 14:55:04 +0100 Subject: [rabbitmq-discuss] Queue Deleted and Permissions Corrupted While Testing Flow Control In-Reply-To: References: Message-ID: <53512EB8.704@rabbitmq.com> On 14/04/14 14:12, patrickjnewell at gmail.com wrote: > While testing disk-based flow control, the queue my test client was > consuming from suddenly got deleted and the user permissions appeared to > get corrupted. Did the machine actually run out of disk space during your test? That certainly would be bad news, and data corruption is almost inevitable in that situation; restarting the broker (after making some space, obviously) should hopefully recover what is recoverable. > Any idea what might have caused this? My best guess is that something > might have gotten corrupted when I originally performed the upgrade Unlikely. Matthias. From matthias at rabbitmq.com Fri Apr 18 15:16:50 2014 From: matthias at rabbitmq.com (Matthias Radestock) Date: Fri, 18 Apr 2014 15:16:50 +0100 Subject: [rabbitmq-discuss] RabbitMQ Causing Segmentation Fault on FreeBSD In-Reply-To: References: <534CE691.7010203@rabbitmq.com> Message-ID: <535133D2.5020105@rabbitmq.com> Brandon, On 15/04/14 17:25, Brandon Munroe wrote: > The message in the logs reads: > > ** Generic server rabbit_mgmt_external_stats terminating > ** Last message in was emit_update > ** When Server state == {state,707130} > ** Reason for termination == > ** {function_clause,[{lists,nth,[1,[]],[{file,"lists.erl"},{line,168}]}, > > {rabbit_mgmt_external_stats,'-get_used_fd/1-fun-1-',2,[]}, > {lists,'-filter/2-lc$^0/1-0-',2, > [{file,"lists.erl"},{line,1271}]}, > {lists,'-filter/2-lc$^0/1-0-',2, > [{file,"lists.erl"},{line,1271}]}, > {rabbit_mgmt_external_stats,get_used_fd,1,[]}, > > {rabbit_mgmt_external_stats,'-infos/2-lc$^0/1-0-',2,[]}, > > {rabbit_mgmt_external_stats,'-infos/2-lc$^0/1-0-',2,[]}, > {rabbit_mgmt_external_stats,emit_update,1,[]}]} On BSD RabbitMQ invokes 'fstat -p ' to determine the number of used file descriptors. In the above it is tripping over the parsing of the output. If you can, please post the output of that command just before rabbit dies. Meanwhile I've filed a bug to make our parsing code more robust. Matthias. From joey.jiaojg at gmail.com Fri Apr 18 15:31:38 2014 From: joey.jiaojg at gmail.com (Joey Jiao) Date: Fri, 18 Apr 2014 22:31:38 +0800 Subject: [rabbitmq-discuss] Exchange.Unknown with Celery In-Reply-To: References: Message-ID: I use tcpdump to get the amqp protocol packet, which says 'Exchange.Unknown' in INFO field. In AMQP section, I can see the error message is 'Unknown exchange message 40' or 'Unknown exchange message 51'. I have searched the celery 3.0.18 which I'm using, only bin/celery.py has similar error message 'Unknown {0.name} message {1}', which can be trigger by using command like 'celery amqp exchange.unbind' or 'celery amqp exchange.unbind_ok'. I haven't found which process to call it, but I'm still checking with other colleagues using celery. I'm not at that computer, so cannot share the packet I captured. But the error I captured is from one broker to another and we are using federation exchanges and federation queues. And yes, I have posted the same question to celery-users group. 2014-04-18 19:41 GMT+08:00 Michael Klishin : > On 18 April 2014 at 15:36:42, Joey Jiao (joey.jiaojg at gmail.com) wrote: > > > I'm using celery as the client. It should also support these > > two msg_id. > > but the bin/celery might has the issue which doesn't include > > exchange.unbind and unbind_ok. Might be better check with them. > > Perhaps Ask (Celery author) can shed some light on what client Celery uses > and what protocol version it targets. > > Meanwhile, can you post something that explains how you run Celery and > what the error message looks like? > -- > MK > > Software Engineer, Pivotal/RabbitMQ > -- -Joey Jiao -------------- next part -------------- An HTML attachment was scrubbed... URL: From mklishin at gopivotal.com Fri Apr 18 15:34:13 2014 From: mklishin at gopivotal.com (Michael Klishin) Date: Fri, 18 Apr 2014 18:34:13 +0400 Subject: [rabbitmq-discuss] Exchange.Unknown with Celery In-Reply-To: References: Message-ID: On 18 April 2014 at 18:31:58, Joey Jiao (joey.jiaojg at gmail.com) wrote: > > I use tcpdump to get the amqp protocol packet, which says 'Exchange.Unknown' > in INFO field. This can also mean that the AMQP dissector is unaware of RabbitMQ extensions or even version 0-9-1. If you get any errors in Celery?s end, post them.? -- MK Software Engineer, Pivotal/RabbitMQ From brandon.munroe at gmail.com Fri Apr 18 18:15:41 2014 From: brandon.munroe at gmail.com (Brandon Munroe) Date: Fri, 18 Apr 2014 11:15:41 -0600 Subject: [rabbitmq-discuss] RabbitMQ Causing Segmentation Fault on FreeBSD In-Reply-To: <535133D2.5020105@rabbitmq.com> References: <534CE691.7010203@rabbitmq.com> <535133D2.5020105@rabbitmq.com> Message-ID: Here's the output of the command around (within 10s) when it segfaults. I've truncated a bunch of similar lines, since there are more than 25k lines. USER CMD PID FD MOUNT INUM MODE SZ|DV R/W brandon beam.smp 65462 text / 17977559 -r-xr-xr-x 2397040 r brandon beam.smp 65462 ctty /dev 89 crw--w---- pts/0 rw brandon beam.smp 65462 wd / 15810621 drwxr-xr-x 1024 r brandon beam.smp 65462 root / 2 drwxr-xr-x 1024 r brandon beam.smp 65462 0 /dev 89 crw--w---- pts/0 rw brandon beam.smp 65462 1 /dev 89 crw--w---- pts/0 rw brandon beam.smp 65462 2 /dev 89 crw--w---- pts/0 rw brandon beam.smp 65462 3 brandon beam.smp 65462 4* pipe fffff8000a978000 <-> fffff8000a978160 0 rw brandon beam.smp 65462 5* pipe fffff8000a978160 <-> fffff8000a978000 0 rw brandon beam.smp 65462 6* pipe fffff8008b5b22e8 <-> fffff8008b5b2448 0 rw brandon beam.smp 65462 7* pipe fffff8008b5b2448 <-> fffff8008b5b22e8 0 rw brandon beam.smp 65462 8* internet stream tcp fffff8027d821000 brandon beam.smp 65462 9* internet stream tcp fffff80566596800 brandon beam.smp 65462 10 - 15810857 -rw-r--r-- 3093764 w brandon beam.smp 65462 11 - 15810858 -rw-r--r-- 0 w brandon beam.smp 65462 12 - 15891126 -rw-r--r-- 96 rw brandon beam.smp 65462 13 - 15891137 -rw-r--r-- 0 rw brandon beam.smp 65462 14 - 15891139 -rw-r--r-- 0 rw brandon beam.smp 65462 15* internet6 stream tcp fffff80458487800 brandon beam.smp 65462 16* pipe fffff803c3f4b000 <-> fffff803c3f4b160 0 rw brandon beam.smp 65462 17* internet stream tcp fffff80458449400 brandon beam.smp 65462 18* internet stream tcp fffff802be9b4c00 brandon beam.smp 65462 19* pipe fffff8000a9bca18 <-> fffff8000a9bc8b8 0 rw brandon beam.smp 65462 20* internet stream tcp fffff802bee95400 brandon beam.smp 65462 21* internet stream tcp fffff802be4ad400 brandon beam.smp 65462 22* pipe fffff801ea5f52e8 <-> fffff801ea5f5448 0 rw brandon beam.smp 65462 23* internet stream tcp fffff804edad0800 brandon beam.smp 65462 24* internet stream tcp fffff80249ff8400 brandon beam.smp 65462 25* internet stream tcp fffff80566026400 ... brandon beam.smp 65462 25018* internet stream tcp fffff8045848d000 brandon beam.smp 65462 25019* internet stream tcp fffff805662fb400 brandon beam.smp 65462 25020* internet stream tcp fffff804a4633400 brandon beam.smp 65462 25021* internet stream tcp fffff802495a8000 brandon beam.smp 65462 25024* pipe fffff8049522a160 <-> fffff8049522a000 0 rw Let me know if you need anything else, or if you spot anything that I can do to mitigate the issue. -Brandon On Fri, Apr 18, 2014 at 8:16 AM, Matthias Radestock wrote: > Brandon, > > > On 15/04/14 17:25, Brandon Munroe wrote: > >> The message in the logs reads: >> >> ** Generic server rabbit_mgmt_external_stats terminating >> ** Last message in was emit_update >> ** When Server state == {state,707130} >> ** Reason for termination == >> ** {function_clause,[{lists,nth,[1,[]],[{file,"lists.erl"},{line,168}]}, >> >> {rabbit_mgmt_external_stats,'-get_used_fd/1-fun-1-',2,[]}, >> {lists,'-filter/2-lc$^0/1-0-',2, >> [{file,"lists.erl"},{line,1271}]}, >> {lists,'-filter/2-lc$^0/1-0-',2, >> [{file,"lists.erl"},{line,1271}]}, >> {rabbit_mgmt_external_stats,get_used_fd,1,[]}, >> >> {rabbit_mgmt_external_stats,'-infos/2-lc$^0/1-0-',2,[]}, >> >> {rabbit_mgmt_external_stats,'-infos/2-lc$^0/1-0-',2,[]}, >> {rabbit_mgmt_external_stats,emit_update,1,[]}]} >> > > On BSD RabbitMQ invokes 'fstat -p ' to determine the number of used > file descriptors. In the above it is tripping over the parsing of the > output. If you can, please post the output of that command just before > rabbit dies. Meanwhile I've filed a bug to make our parsing code more > robust. > > > Matthias. > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From matthias at rabbitmq.com Fri Apr 18 18:29:52 2014 From: matthias at rabbitmq.com (Matthias Radestock) Date: Fri, 18 Apr 2014 18:29:52 +0100 Subject: [rabbitmq-discuss] RabbitMQ Causing Segmentation Fault on FreeBSD In-Reply-To: References: <534CE691.7010203@rabbitmq.com> <535133D2.5020105@rabbitmq.com> Message-ID: <53516110.9040007@rabbitmq.com> On 18/04/14 18:15, Brandon Munroe wrote: > Here's the output of the command around (within 10s) when it segfaults. > I've truncated a bunch of similar lines, since there are more than 25k > lines. Would you mind sending me the complete output directly? Matthias. From matthias at rabbitmq.com Fri Apr 18 19:30:17 2014 From: matthias at rabbitmq.com (Matthias Radestock) Date: Fri, 18 Apr 2014 19:30:17 +0100 Subject: [rabbitmq-discuss] RabbitMQ Causing Segmentation Fault on FreeBSD In-Reply-To: References: <534CE691.7010203@rabbitmq.com> <535133D2.5020105@rabbitmq.com> <53516110.9040007@rabbitmq.com> Message-ID: <53516F39.2050709@rabbitmq.com> (adding list back) On 18/04/14 18:32, Brandon Munroe wrote: > Yep, here you go. Nothing wrong with that. I'm afraid that until we've patched the code to be more defensive, the only mitigation I can think of is to disable the management plug-in. Or, possibly, reducing the fd count. Matthias. From brandon.munroe at gmail.com Fri Apr 18 19:48:25 2014 From: brandon.munroe at gmail.com (Brandon Munroe) Date: Fri, 18 Apr 2014 12:48:25 -0600 Subject: [rabbitmq-discuss] RabbitMQ Causing Segmentation Fault on FreeBSD In-Reply-To: <53516F39.2050709@rabbitmq.com> References: <534CE691.7010203@rabbitmq.com> <535133D2.5020105@rabbitmq.com> <53516110.9040007@rabbitmq.com> <53516F39.2050709@rabbitmq.com> Message-ID: Okay, thanks. Is there something specific I should watch for in the change logs to know when it should be okay to re-enable the plug-in? -Brandon On Fri, Apr 18, 2014 at 12:30 PM, Matthias Radestock wrote: > (adding list back) > > On 18/04/14 18:32, Brandon Munroe wrote: > >> Yep, here you go. >> > > Nothing wrong with that. > > I'm afraid that until we've patched the code to be more defensive, the > only mitigation I can think of is to disable the management plug-in. Or, > possibly, reducing the fd count. > > > Matthias. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From matthias at rabbitmq.com Fri Apr 18 22:25:04 2014 From: matthias at rabbitmq.com (Matthias Radestock) Date: Fri, 18 Apr 2014 22:25:04 +0100 Subject: [rabbitmq-discuss] RabbitMQ Causing Segmentation Fault on FreeBSD In-Reply-To: References: <534CE691.7010203@rabbitmq.com> <535133D2.5020105@rabbitmq.com> <53516110.9040007@rabbitmq.com> <53516F39.2050709@rabbitmq.com> Message-ID: <53519830.20002@rabbitmq.com> On 18/04/14 19:48, Brandon Munroe wrote: > Is there something specific I should watch for in the change logs to > know when it should be okay to re-enable the plug-in? We are tracking this issue under bug# 26134. Matthias. From brandon.munroe at gmail.com Sat Apr 19 00:17:25 2014 From: brandon.munroe at gmail.com (Brandon Munroe) Date: Fri, 18 Apr 2014 17:17:25 -0600 Subject: [rabbitmq-discuss] RabbitMQ Causing Segmentation Fault on FreeBSD In-Reply-To: <53519830.20002@rabbitmq.com> References: <534CE691.7010203@rabbitmq.com> <535133D2.5020105@rabbitmq.com> <53516110.9040007@rabbitmq.com> <53516F39.2050709@rabbitmq.com> <53519830.20002@rabbitmq.com> Message-ID: Great, thanks! -Brandon On Apr 18, 2014 3:25 PM, "Matthias Radestock" wrote: > On 18/04/14 19:48, Brandon Munroe wrote: > >> Is there something specific I should watch for in the change logs to >> know when it should be okay to re-enable the plug-in? >> > > We are tracking this issue under bug# 26134. > > Matthias. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From srividhya.umashanker at hp.com Sat Apr 19 15:42:57 2014 From: srividhya.umashanker at hp.com (Umashanker, Srividhya) Date: Sat, 19 Apr 2014 14:42:57 +0000 Subject: [rabbitmq-discuss] 3 Node Federation : Messages to be handled by other nodes and ignored by local node Message-ID: <5D5078F9198B04449B9D74EA9E7F3D580D11C4C5@G1W3640.americas.hpqcorp.net> Problem to solve : a. I want federation of 3 nodes , messages published on one node to be available on all other nodes as well b. The local should not handle it, but all other nodes in the federation should handle it. What did I configure a. Node 1: Upstream - node2 , node3 Node 2: Upstream - node1, node3 Node 3: upstream - node1, node2 b. All three nodes have an exchange called "federated-x" c. Federated-q for federated-x with NO routing key The following is what I tried. 1. Federate only exchange - When I publish on node1's federated-x, I listened on node1, node2, node3's federated-q. I received the messages on all 3 queues. 2. Federate only queue - When I publish on node1's federated-x, I listened on node1, node2, node3's federated-q. a. Only node1 local receives it. b. When node1 is not listening, the 1st consumer alone recives not all 3. Federate both exchanges and queues - Did not work as expected. Is the above the expected behavior? Can anyone else suggest me other ways to handle it? I can achieve mostly with exchange federation. But I want the local node to ignore the message. Any other way, than programmatically ignoring it when message comes without x-federation headers? Please help. -Vidhya -------------- next part -------------- An HTML attachment was scrubbed... URL: From michael.s.klishin at gmail.com Sat Apr 19 15:57:45 2014 From: michael.s.klishin at gmail.com (Michael Klishin) Date: Sat, 19 Apr 2014 18:57:45 +0400 Subject: [rabbitmq-discuss] 3 Node Federation : Messages to be handled by other nodes and ignored by local node In-Reply-To: <5D5078F9198B04449B9D74EA9E7F3D580D11C4C5@G1W3640.americas.hpqcorp.net> References: <5D5078F9198B04449B9D74EA9E7F3D580D11C4C5@G1W3640.americas.hpqcorp.net> Message-ID: 2014-04-19 18:42 GMT+04:00 Umashanker, Srividhya < srividhya.umashanker at hp.com>: > *The following is what I tried.* > > 1. Federate only exchange - When I publish on node1?s > federated-x, I listened on node1, node2, node3?s federated-q. > > I received the messages on all 3 queues. > > 2. Federate only queue - When I publish on node1?s > federated-x, I listened on node1, node2, node3?s federated-q. > > a. Only node1 local receives it. > > b. When node1 is not listening, the 1st consumer alone recives not > all > > 3. Federate both exchanges and queues ? Did not work as expected. > > > > Is the above the expected behavior? Can anyone else suggest me other ways > to handle it? > > > > I can achieve mostly with exchange federation. But I want the local node > to ignore the message. Any other way, than programmatically ignoring it > when message comes without x-federation headers? > As described in http://www.rabbitmq.com/federated-queues.html, federated queues distributed messages between federated clusters and prefer local consumers. This sounds like the opposite of what you want. Federated exchanges replicate message streams to other clusters. How the messages are routed or consumed is completely up to you. So simply set up a topology in clusters 2 and 3 and have consumers there. Cluster 1 won't have any queues bound to the federated exchange(s) and won't have any consumers. -- MK http://github.com/michaelklishin http://twitter.com/michaelklishin -------------- next part -------------- An HTML attachment was scrubbed... URL: From bjoernhaeuser at gmail.com Sat Apr 19 16:54:19 2014 From: bjoernhaeuser at gmail.com (=?iso-8859-1?Q?Bj=F6rn_H=E4user?=) Date: Sat, 19 Apr 2014 17:54:19 +0200 Subject: [rabbitmq-discuss] 3.2.4: Segfaults with HiPe enabled Message-ID: Hello, to cope with some performance degradations we enabled HiPe two weeks ago. We now have problems that every 6 weeks every node crashes with this error message: Slogan: std_alloc: Cannot allocate 1117203264527168 bytes of memory (of type "arg_reg"). You can find the complete erl_crash.dump here: https://files.rebuy-cdn.de/erl_crash.dump - the certificate is not valid. Can we anything do to have HiPe enabled without these crashes? ;) Thanks in advance Bj?rn From bjoernhaeuser at gmail.com Sat Apr 19 16:48:25 2014 From: bjoernhaeuser at gmail.com (=?iso-8859-1?Q?Bj=F6rn_H=E4user?=) Date: Sat, 19 Apr 2014 17:48:25 +0200 Subject: [rabbitmq-discuss] 3.2.4: Segfaults with HiPe enabled Message-ID: <8B3D0985-174A-46BF-B665-C379145664F6@gmail.com> Hello, to cope with some performance degradations we enabled HiPe two weeks ago. We now have problems that every 6 weeks every node crashes with this error message: Slogan: std_alloc: Cannot allocate 1117203264527168 bytes of memory (of type "arg_reg"). You can find the complete erl_crash.dump here: https://files.rebuy-cdn.de/erl_crash.dump - the certificate is not valid. Can we anything do to have HiPe enabled without these crashes? ;) Thanks in advance Bj?rn From joey.jiaojg at gmail.com Mon Apr 21 07:16:02 2014 From: joey.jiaojg at gmail.com (Joey Jiao) Date: Mon, 21 Apr 2014 14:16:02 +0800 Subject: [rabbitmq-discuss] Exchange.Unknown with Celery In-Reply-To: References: Message-ID: My mistake, I use wireshark to parse but it uses default AMQP protocol not the rabbitmq extended one. 2014-04-18 22:34 GMT+08:00 Michael Klishin : > On 18 April 2014 at 18:31:58, Joey Jiao (joey.jiaojg at gmail.com) wrote: > > > I use tcpdump to get the amqp protocol packet, which says > 'Exchange.Unknown' > > in INFO field. > > This can also mean that the AMQP dissector is unaware of RabbitMQ > extensions or > even version 0-9-1. > > If you get any errors in Celery?s end, post them. > -- > MK > > Software Engineer, Pivotal/RabbitMQ > -- -Joey Jiao -------------- next part -------------- An HTML attachment was scrubbed... URL: From matthias at rabbitmq.com Mon Apr 21 08:41:09 2014 From: matthias at rabbitmq.com (Matthias Radestock) Date: Mon, 21 Apr 2014 08:41:09 +0100 Subject: [rabbitmq-discuss] 3.2.4: Segfaults with HiPe enabled In-Reply-To: References: Message-ID: <5354CB95.2040201@rabbitmq.com> On 19/04/14 16:54, Bj?rn H?user wrote: > Can we anything do to have HiPe enabled without these crashes? ;) Try to 1) upgrade Erlang - bugs in HiPE get fixed in pretty much every new release 2) upgrade RabbitMQ - one possible cause for the memory allocation failure is that a large RabbitMQ process crashed and Erlang then died trying to display the process state. We addressed that issue in 3.3.0 Matthias. From matthias at rabbitmq.com Mon Apr 21 08:45:06 2014 From: matthias at rabbitmq.com (Matthias Radestock) Date: Mon, 21 Apr 2014 08:45:06 +0100 Subject: [rabbitmq-discuss] Queue disappears during partition/autoheal In-Reply-To: References: <534E72CE.5010001@rabbitmq.com> Message-ID: <5354CC82.6010100@rabbitmq.com> On 16/04/14 19:04, Matt Pietrek wrote: > > So this is something we've seen before in the case of short-lived > partitions; something in Mnesia is sending a stray {mnesia_locker, ..., > ...} message to a process that isn't expecting it after the partition, > killing the process in question. > > Do you have a sense whether this behavior is specific to having Autoheal > enabled? In other words, if we didn't have Autoheal in effect, could it > still happen? It could still happen. And even, it turns out, in R17 :( It conceivably is less likely to occur w/o autoheal though. Matthias. From bjoernhaeuser at gmail.com Mon Apr 21 09:32:42 2014 From: bjoernhaeuser at gmail.com (=?UTF-8?B?QmrDtnJuIEjDpHVzZXI=?=) Date: Mon, 21 Apr 2014 10:32:42 +0200 Subject: [rabbitmq-discuss] Queue disappears during partition/autoheal In-Reply-To: <5354CC82.6010100@rabbitmq.com> References: <534E72CE.5010001@rabbitmq.com> <5354CC82.6010100@rabbitmq.com> Message-ID: Hi Matthias, ok, we will upgrade to newest rabbit mq. We are also running R16B03. Should wo go for R17? Thanks Bj?rn 2014-04-21 9:45 GMT+02:00 Matthias Radestock : > On 16/04/14 19:04, Matt Pietrek wrote: >> >> > So this is something we've seen before in the case of short-lived >> partitions; something in Mnesia is sending a stray {mnesia_locker, ..., >> ...} message to a process that isn't expecting it after the partition, >> killing the process in question. >> >> Do you have a sense whether this behavior is specific to having Autoheal >> enabled? In other words, if we didn't have Autoheal in effect, could it >> still happen? > > > It could still happen. And even, it turns out, in R17 :( > > It conceivably is less likely to occur w/o autoheal though. > > > Matthias. > > > _______________________________________________ > rabbitmq-discuss mailing list > rabbitmq-discuss at lists.rabbitmq.com > https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss From mcintoshj at gmail.com Mon Apr 21 17:43:32 2014 From: mcintoshj at gmail.com (Jason McIntosh) Date: Mon, 21 Apr 2014 11:43:32 -0500 Subject: [rabbitmq-discuss] Flow control behavior Message-ID: I could have sworn at one point, flow control would kick in when consumers couldn't keep up with publish rates. I recall running tests about a year ago where consumer slow downs would slow down publishes. But I could also be out of my mind with more than a few loose screws (actually I know I've got a few loose). But anyways, while testing publishing, it appears flow control only kicks in when the queues can't keep up due to disk/io or ram utilization. The rabbit docs seem to verify this. Did something change on this or am I imaging how things used to be? This is my test setup: 1 node, publishes via shovel to two other servers. Publish at 800 msgs/sec to node 1, shovel at 800/sec, consume at 15 messages/sec from other two servers. Publish at 800 msgs/sec to node 1, no shovel, consume at 15 messages/sec from node 1 Observations: 1) It appears that as long as rabbit has ram and disk space, i can continue to publish at 800/sec no problem. Even if the remote side slows down to 400/sec over the shovel, rabbit on node 1 continues to publish and backlog locally as long as node 1 has ram/disk capacity. 2) Remote sides going up and down have no impact on publishing rates, other than possibly slowing things down due to having to read disks to get the messages backlogged to disk. Was there ever a time where flow control was based upon consumer rate not queue rates? Jason -- Jason McIntosh https://github.com/jasonmcintosh/ 573-424-7612 -------------- next part -------------- An HTML attachment was scrubbed... URL: From mklishin at gopivotal.com Mon Apr 21 18:45:46 2014 From: mklishin at gopivotal.com (Michael Klishin) Date: Mon, 21 Apr 2014 21:45:46 +0400 Subject: [rabbitmq-discuss] Flow control behavior In-Reply-To: References: Message-ID: On 21 April 2014 at 20:45:45, Jason McIntosh (mcintoshj at gmail.com) wrote: > > Was there ever a time where flow control was based upon consumer > rate not queue rates? There is flow control based on various Erlang processes (including queues, protocol reader and consumers) not keeping up with the messages that are being sent to them. This flow control is temporary, while resource-based can lead to publishers being blocked for a longer time (until resource alarms clear). Does this answer your question?? -- MK Software Engineer, Pivotal/RabbitMQ From mcintoshj at gmail.com Mon Apr 21 19:07:55 2014 From: mcintoshj at gmail.com (Jason McIntosh) Date: Mon, 21 Apr 2014 13:07:55 -0500 Subject: [rabbitmq-discuss] Flow control behavior In-Reply-To: References: Message-ID: Hrmm, I guess it sorta does. I thought though, at least, I recall testing about a year ago, where if a consumer was able to consume at only 15 messages a second, then publishing would be restricted (via flow control) to 15 messages a second. While testing today, this doesn't appear to be the case - that publishing is only restricted by the capacities of the rabbit server (or cluster) locally. Jason On Mon, Apr 21, 2014 at 12:45 PM, Michael Klishin wrote: > On 21 April 2014 at 20:45:45, Jason McIntosh (mcintoshj at gmail.com) wrote: > > > Was there ever a time where flow control was based upon consumer > > rate not queue rates? > > There is flow control based on various Erlang processes (including queues, > protocol reader > and consumers) not keeping up with the messages that are being sent to > them. This > flow control is temporary, while resource-based can lead to publishers > being blocked > for a longer time (until resource alarms clear). > > Does this answer your question? > -- > MK > > Software Engineer, Pivotal/RabbitMQ > -- Jason McIntosh https://github.com/jasonmcintosh/ 573-424-7612 -------------- next part -------------- An HTML attachment was scrubbed... URL: From matthias at rabbitmq.com Mon Apr 21 19:16:11 2014 From: matthias at rabbitmq.com (Matthias Radestock) Date: Mon, 21 Apr 2014 19:16:11 +0100 Subject: [rabbitmq-discuss] Flow control behavior In-Reply-To: References: Message-ID: <5355606B.1030708@rabbitmq.com> On 21/04/14 19:07, Jason McIntosh wrote: > Hrmm, I guess it sorta does. I thought though, at least, I recall > testing about a year ago, where if a consumer was able to consume at > only 15 messages a second, then publishing would be restricted (via flow > control) to 15 messages a second. While testing today, this doesn't > appear to be the case - that publishing is only restricted by the > capacities of the rabbit server (or cluster) locally. It's always been like that. What has changed a few times though is how busy queues prioritise work relating to producers vs consumers. See http://www.rabbitmq.com/blog/2014/04/10/consumer-bias-in-rabbitmq-3-3/ Matthias. From Rouzbeh.Farahmand at Desire2Learn.com Mon Apr 21 22:09:54 2014 From: Rouzbeh.Farahmand at Desire2Learn.com (Rouzbeh Farahmand) Date: Mon, 21 Apr 2014 21:09:54 +0000 Subject: [rabbitmq-discuss] The meaning of latency in the PerfTest module Message-ID: <6B4CB2165485294B8C9571963FDDBD4822CE83AA@KTVCAEX-MB-01.desire2learn.d2l> Hello all, I have a question regarding the way latency is calculated in the PefTest module. When we say latency (time taken for a sent message to be received), do we mean the time it took for the publisher to put a message to a queue and got an ACK back. ( i.e. latency = wall-time of sending the message by publisher - wall-time of receiving the ACK by the publisher) Or Do we mean the time that it takes to publish a message to a queue and consume the same message from the same queue? . ( i.e. latency = wall-time of sending the message by publisher - wall-time of receiving by the consumer) I had a feeling that it must be the former ( because usually the consumer is another party), but my confusion is due to the following scenarios: When I run the PerfTest like this: One publisher - One Consumer (with publisher confirms), I get the latency calculated for me. So far so good! D:\abbitmq-java-client-bin-3.3.0> runjava.bat com.rabbitmq.examples.PerfTest -x 1 -y 1 -c 1000 starting consumer #0 starting producer #0 time: 1.000s, sent: 16562 msg/s, confirmed: 16138 msg/s, nacked: 0 msg/s, received: 16137 msg/s, min/avg/max latency: 5246/26151/85175 microseconds However, when I run this test: one publisher - one consumer (with publisher confirms) with auto-ack set (i.e. no Ack). I still get the latency calculated for me. So the PerfTest could have not possibly calculated the latency based on the time it received the Ack. So does it calculate it based on the time it consumes the message and receives it at its own end? D:\rabbitmq-java-client-bin-3.3.0> runjava.bat com.rabbitmq.examples.PerfTest -a -x 1 -y 1 -c 1000 starting consumer #0 starting producer #0 time: 1.000s, sent: 18510 msg/s, confirmed: 18074 msg/s, nacked: 0 msg/s, received: 18073 msg/s, min/avg/max latency: 650/20666/44193 microseconds Whereas here, I have one publisher and no consumer, and I don't get the latency because in this scenario, I have no consumer, so no queue is created, and we don't deliver a message to a queue, and hence we don't have the time of consumption or ack receive ?! D:\ \rabbitmq-java-client-bin-3.3.0> runjava.bat com.rabbitmq.examples.PerfTest -x 1 -y 0 -c 1000 starting producer #0 time: 1.000s, sent: 47299 msg/s, confirmed: 46835 msg/s, nacked: 0 msg/s Your help and clarification is much appreciated. Cheers, Rouzbeh [D2L logo] Rouzbeh Farahmand Software Developer Desire2Learn Incorporated rouzbeh.farahmand at Desire2Learn.com www.Desire2Learn.com -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image001.png Type: image/png Size: 1545 bytes Desc: image001.png URL: From wbosland.prive at gmail.com Mon Apr 21 15:24:07 2014 From: wbosland.prive at gmail.com (Wouter Bosland) Date: Mon, 21 Apr 2014 07:24:07 -0700 (PDT) Subject: [rabbitmq-discuss] Windows RabbitMQ Cluster Message-ID: <8a1539c0-f76f-44b2-af75-63d1956ddcdd@googlegroups.com> Hello everyone, I have 2 virtual machines running Windows 7 with RabbitMQ 3.3.0 (firewall off) and followed this tutorial: https://www.rabbitmq.com/clustering.html This is what I did: 1. start rabbitmq-server on both machines 2. stop rabbitmq-server on machine 2 (slave) using: rabbitmqctl stop 3. copy the .erlang.cookie from machine 1 (master) to machine 2 (slave) 4. start rabbitmq-server again on machine 2 (slave) with the new cookie 5. run the command: "rabbitmqctl stop_app" on machine 2 (slave) 6. run the command: ''rabbitmqctl join_cluster rabbit at master'' on machine 2 (slave) Clustering node 'rabbit at WIN-PK8SONHOIBH' with rabbit at master ... Error: unable to connect to nodes [rabbit at master]: nodedown DIAGNOSTICS =========== attempted to contact: [rabbit at master] rabbit at master: * found rabbit (port 25672) * TCP connection succeeded * suggestion: hostname mismatch? * suggestion: is the cookie set correctly? current node details: - node name: 'rabbitmqctl2665551 at WIN-PK8SONHOIBH' - home dir: C:\Users\RabbitMQ2 - cookie hash: *zF3bJzufFyuGd7zyNAb0UQ==* *If I try the other way around clustering master to slave I get:* Clustering node 'rabbit at WIN-2BQPAJ1S03R' with rabbit at slave ... Error: unable to connect to nodes [rabbit at slave]: nodedown DIAGNOSTICS =========== attempted to contact: [rabbit at slave] rabbit at slave: * found rabbit (port 25672) * TCP connection succeeded * suggestion: hostname mismatch? * suggestion: is the cookie set correctly? current node details: - node name: 'rabbitmqctl2366873 at WIN-2BQPAJ1S03R' - home dir: C:\Users\RabbitMQ1 - cookie hash: *zF3bJzufFyuGd7zyNAb0UQ==* *The cookie hash is the same so what could be the problem?* -------------- next part -------------- An HTML attachment was scrubbed... URL: From xeonmailinglist at gmail.com Mon Apr 21 19:07:42 2014 From: xeonmailinglist at gmail.com (xeon Mailinglist) Date: Mon, 21 Apr 2014 11:07:42 -0700 (PDT) Subject: [rabbitmq-discuss] UNEXPECTED_FRAME - expected content header for class 60, got non content header frame instead Message-ID: I am trying to run python multiprocessing to execute a remote function that I call with the help of celery. My program runs ok if I just run one process. But when I run more than one process, I get the error below. Is it because of incompatible versions in python plugins and rabbitmq? Process Process-1: Traceback (most recent call last): File "/usr/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap self.run() File "/usr/lib/python2.7/multiprocessing/process.py", line 114, in run self._target(*self._args, **self._kwargs) File "testHello.py", line 16, in test_hello_aux print output.get() File "/usr/local/lib/python2.7/dist-packages/celery/result.py", line 169, in get no_ack=no_ack, File "/usr/local/lib/python2.7/dist-packages/celery/backends/amqp.py", line 155, in wait_for on_interval=on_interval) File "/usr/local/lib/python2.7/dist-packages/celery/backends/amqp.py", line 229, in consume no_ack=no_ack, accept=self.accept) as consumer: File "/usr/local/lib/python2.7/dist-packages/kombu/messaging.py", line 359, in __init__ self.revive(self.channel) File "/usr/local/lib/python2.7/dist-packages/kombu/messaging.py", line 371, in revive self.declare() File "/usr/local/lib/python2.7/dist-packages/kombu/messaging.py", line 381, in declare queue.declare() File "/usr/local/lib/python2.7/dist-packages/kombu/entity.py", line 505, in declare self.queue_declare(nowait, passive=False) File "/usr/local/lib/python2.7/dist-packages/kombu/entity.py", line 531, in queue_declare nowait=nowait) File "/usr/local/lib/python2.7/dist-packages/amqp/channel.py", line 1254, in queue_declare self._send_method((50, 10), args) File "/usr/local/lib/python2.7/dist-packages/amqp/abstract_channel.py", line 56, in _send_method self.channel_id, method_sig, args, content, File "/usr/local/lib/python2.7/dist-packages/amqp/method_framing.py", line 221, in write_method write_frame(1, channel, payload) File "/usr/local/lib/python2.7/dist-packages/amqp/transport.py", line 177, in write_frame frame_type, channel, size, payload, 0xce, File "/usr/lib/python2.7/socket.py", line 224, in meth return getattr(self._sock,name)(*args) error: [Errno 32] Broken pipe Process Process-2: Traceback (most recent call last): File "/usr/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap self.run() File "/usr/lib/python2.7/multiprocessing/process.py", line 114, in run self._target(*self._args, **self._kwargs) File "testHello.py", line 16, in test_hello_aux print output.get() File "/usr/local/lib/python2.7/dist-packages/celery/result.py", line 169, in get no_ack=no_ack, File "/usr/local/lib/python2.7/dist-packages/celery/backends/amqp.py", line 155, in wait_for on_interval=on_interval) File "/usr/local/lib/python2.7/dist-packages/celery/backends/amqp.py", line 229, in consume no_ack=no_ack, accept=self.accept) as consumer: File "/usr/local/lib/python2.7/dist-packages/kombu/messaging.py", line 359, in __init__ Process Process-3: self.revive(self.channel) Traceback (most recent call last): File "/usr/local/lib/python2.7/dist-packages/kombu/messaging.py", line 371, in revive self.declare() File "/usr/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap File "/usr/local/lib/python2.7/dist-packages/kombu/messaging.py", line 381, in declare queue.declare() File "/usr/local/lib/python2.7/dist-packages/kombu/entity.py", line 504, in declare self.run() self.exchange.declare(nowait) File "/usr/lib/python2.7/multiprocessing/process.py", line 114, in run File "/usr/local/lib/python2.7/dist-packages/kombu/entity.py", line 166, in declare self._target(*self._args, **self._kwargs) nowait=nowait, passive=passive, File "testHello.py", line 16, in test_hello_aux File "/usr/local/lib/python2.7/dist-packages/amqp/channel.py", line 613, in exchange_declare print output.get() File "/usr/local/lib/python2.7/dist-packages/celery/result.py", line 169, in get no_ack=no_ack, File "/usr/local/lib/python2.7/dist-packages/celery/backends/amqp.py", line 155, in wait_for on_interval=on_interval) File "/usr/local/lib/python2.7/dist-packages/celery/backends/amqp.py", line 229, in consume no_ack=no_ack, accept=self.accept) as consumer: File "/usr/local/lib/python2.7/dist-packages/kombu/messaging.py", line 359, in __init__ self._send_method((40, 10), args) File "/usr/local/lib/python2.7/dist-packages/amqp/abstract_channel.py", line 56, in _send_method self.channel_id, method_sig, args, content, File "/usr/local/lib/python2.7/dist-packages/amqp/method_framing.py", line 221, in write_method self.revive(self.channel) File "/usr/local/lib/python2.7/dist-packages/kombu/messaging.py", line 371, in revive self.declare() File "/usr/local/lib/python2.7/dist-packages/kombu/messaging.py", line 381, in declare write_frame(1, channel, payload) queue.declare() File "/usr/local/lib/python2.7/dist-packages/amqp/transport.py", line 177, in write_frame File "/usr/local/lib/python2.7/dist-packages/kombu/entity.py", line 504, in declare frame_type, channel, size, payload, 0xce, File "/usr/lib/python2.7/socket.py", line 224, in meth self.exchange.declare(nowait) File "/usr/local/lib/python2.7/dist-packages/kombu/entity.py", line 166, in declare nowait=nowait, passive=passive, File "/usr/local/lib/python2.7/dist-packages/amqp/channel.py", line 620, in exchange_declare return getattr(self._sock,name)(*args) error: [Errno 32] Broken pipe (40, 11), # Channel.exchange_declare_ok File "/usr/local/lib/python2.7/dist-packages/amqp/abstract_channel.py", line 67, in wait self.channel_id, allowed_methods) File "/usr/local/lib/python2.7/dist-packages/amqp/connection.py", line 237, in _wait_method self.method_reader.read_method() File "/usr/local/lib/python2.7/dist-packages/amqp/method_framing.py", line 189, in read_method raise m error: [Errno 104] Connection reset by peer Process Process-4: Traceback (most recent call last): File "/usr/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap self.run() File "/usr/lib/python2.7/multiprocessing/process.py", line 114, in run self._target(*self._args, **self._kwargs) File "testHello.py", line 16, in test_hello_aux print output.get() File "/usr/local/lib/python2.7/dist-packages/celery/result.py", line 169, in get no_ack=no_ack, File "/usr/local/lib/python2.7/dist-packages/celery/backends/amqp.py", line 155, in wait_for on_interval=on_interval) File "/usr/local/lib/python2.7/dist-packages/celery/backends/amqp.py", line 229, in consume no_ack=no_ack, accept=self.accept) as consumer: File "/usr/local/lib/python2.7/dist-packages/kombu/messaging.py", line 359, in __init__ self.revive(self.channel) File "/usr/local/lib/python2.7/dist-packages/kombu/messaging.py", line 371, in revive self.declare() File "/usr/local/lib/python2.7/dist-packages/kombu/messaging.py", line 381, in declare queue.declare() File "/usr/local/lib/python2.7/dist-packages/kombu/entity.py", line 505, in declare self.queue_declare(nowait, passive=False) File "/usr/local/lib/python2.7/dist-packages/kombu/entity.py", line 531, in queue_declare nowait=nowait) File "/usr/local/lib/python2.7/dist-packages/amqp/channel.py", line 1258, in queue_declare (50, 11), # Channel.queue_declare_ok File "/usr/local/lib/python2.7/dist-packages/amqp/abstract_channel.py", line 67, in wait self.channel_id, allowed_methods) File "/usr/local/lib/python2.7/dist-packages/amqp/connection.py", line 270, in _wait_method self.wait() File "/usr/local/lib/python2.7/dist-packages/amqp/abstract_channel.py", line 69, in wait return self.dispatch_method(method_sig, args, content) File "/usr/local/lib/python2.7/dist-packages/amqp/abstract_channel.py", line 87, in dispatch_method return amqp_method(self, args) File "/usr/local/lib/python2.7/dist-packages/amqp/connection.py", line 526, in _close (class_id, method_id), ConnectionError) UnexpectedFrame: Basic.publish: (505) UNEXPECTED_FRAME - expected content header for class 60, got non content header frame instead -------------- next part -------------- An HTML attachment was scrubbed... URL: From mklishin at gopivotal.com Mon Apr 21 22:41:01 2014 From: mklishin at gopivotal.com (Michael Klishin) Date: Tue, 22 Apr 2014 01:41:01 +0400 Subject: [rabbitmq-discuss] Windows RabbitMQ Cluster In-Reply-To: <8a1539c0-f76f-44b2-af75-63d1956ddcdd@googlegroups.com> References: <8a1539c0-f76f-44b2-af75-63d1956ddcdd@googlegroups.com> Message-ID: On 22 April 2014 at 01:38:55, Wouter Bosland (wbosland.prive at gmail.com) wrote: > > The cookie hash is the same so what could be the problem? Are connections on port 4369 accessible on/to both nodes? It is used by the Erlang runtime for inter-host communication.? -- MK Software Engineer, Pivotal/RabbitMQ From mklishin at gopivotal.com Mon Apr 21 22:54:54 2014 From: mklishin at gopivotal.com (Michael Klishin) Date: Tue, 22 Apr 2014 01:54:54 +0400 Subject: [rabbitmq-discuss] UNEXPECTED_FRAME - expected content header for class 60, got non content header frame instead In-Reply-To: References: Message-ID: On 22 April 2014 at 01:41:11, xeon Mailinglist (xeonmailinglist at gmail.com) wrote: > > Is it because of incompatible versions in python plugins and > rabbitmq? The issue is with concurrent publishing on the same channel. Channels should not be shared between threads/etc. For every publish you do, your client sends at least 2 frames on the wire: [basic.publish method][content header][payload]* RabbitMQ expects frames to arrive in this order. With concurrent publishing on the same channel, frames can get interleaved like so: [basic.publish method 1][content header 1][basic.publish method 2][content header 2][payload 1][payload 2] or get interleaved with other protocol methods, e.g. basic.consume in your case. I?m not familiar with Celery enough to tell how to resolve this. Using a connection or at least channel per multiprocessing ?thread? should help.? -- MK Software Engineer, Pivotal/RabbitMQ From dougbenjaminatwork at gmail.com Tue Apr 22 03:58:34 2014 From: dougbenjaminatwork at gmail.com (Doug Benjamin) Date: Mon, 21 Apr 2014 21:58:34 -0500 Subject: [rabbitmq-discuss] x509 proxy file for user authentication Message-ID: Hello, I was wondering about the status of using a x509 proxy file for user authentication using a python client code and rabbitmq server. If possible I would like to avoid having to pass a password for authentication, as I would have to place the password in a text file on the client machine. (Does code exist to use an encrypted shared secret?) When connecting to a globus gridftp server, I can use my x509 proxy file. The globus gridftp server contains a text file that maps DN to user name. I would like to do the same with my rabbitmq server. I would like to use the ssl certificate log into the rabbit management and rabbit server ports. When I try to connect to the rabbit management port I connect vs https but get a login screen requesting username/password. I am sure that I have something misconfigured. Here is what I have for my rabbit config file. [ {rabbit, [ {auth_mechanisms,['EXTERNAL']}, {tcp_listeners, []}, {ssl_listeners, [5671]}, {ssl_options, [{cacertfile,"/var/lib/rabbitmq/certs/all_cacerts.pem"}, {certfile,"/var/lib/rabbitmq/certs/hostcert.pem"}, {keyfile,"/var/lib/rabbitmq/certs/hostkey.pem"}, {verify,verify_peer}, {fail_if_no_peer_cert,true}]} ]}, {rabbitmq_stomp, [{tcp_listeners,[]}, {ssl_cert_login, true}, {ssl_listeners, [61614]}]}, {rabbitmq_management, [{ listener, [ {port,15671}, {ssl,true}, {ssl_opts, [{cacertfile,"/var/lib/rabbitmq/certs/all_cacerts.pem"}, {certfile,"/var/lib/rabbitmq/certs/hostcert.pem"}, {keyfile,"/var/lib/rabbitmq/certs/hostkey.pem"}]} ]} ]} ]. Thanks, Doug Benjamin -------------- next part -------------- An HTML attachment was scrubbed... URL: From arunrao.seattle at gmail.com Tue Apr 22 06:55:34 2014 From: arunrao.seattle at gmail.com (Arun Rao) Date: Mon, 21 Apr 2014 22:55:34 -0700 Subject: [rabbitmq-discuss] Exchange Master failure? Message-ID: Hello - I have a 4 node cluster and exchanges are HA'ed through a policy. When one node (randomly chosen as Exchange master?) fails, I see delivery rate dropping to zero for several seconds across all queues in that cluster. (RabbitMQ Server version: 3.1.3) Can someone advise the logic behind Exchanges in a cluster? Thanks! -------------- next part -------------- An HTML attachment was scrubbed... URL: From ashjain2 at gmail.com Tue Apr 22 06:56:39 2014 From: ashjain2 at gmail.com (Ashish Jain) Date: Tue, 22 Apr 2014 11:26:39 +0530 Subject: [rabbitmq-discuss] Incredible problem with rabbitmq Message-ID: Guys, I am having a very different problem with rabbitmq. Here is the use case 1) I have 2 rabbitmq installed. One is rmq1 and other is rmq2. 2) I have 50 consumers for both the rabbitmq instances. 3) I am seeing that all the rabbitmq consumers to rmq1 are hanged after sometime however the consumers to rmq2 are up and running fine and are never in a hang state. 4) When I restart rmq1 all the consumers reconnect and are fine for sometime however again they are hanged. 5) So as to debug this issue I used "telnet rmq1 5672" and "telnet rmq2 5672". In case of "telnet rmq1 5672" I see that I am not able to close the connection at all. However "telnet rmq2 5672" once I hit a key I am able to close the connection and the message is "Escape character is '^]'. ^C ^CConnection closed by foreign host." Do you guys know in #5 why is there a difference in behavior? --Ashish -------------- next part -------------- An HTML attachment was scrubbed... URL: From arunrao.seattle at gmail.com Tue Apr 22 07:04:30 2014 From: arunrao.seattle at gmail.com (Arun Rao) Date: Mon, 21 Apr 2014 23:04:30 -0700 Subject: [rabbitmq-discuss] Incredible problem with rabbitmq In-Reply-To: References: Message-ID: Your logs on rmq1 say anything? I have seen this behavior due to Disk IO being terribly slow on one of my machines. On Mon, Apr 21, 2014 at 10:56 PM, Ashish Jain wrote: > Guys, > > I am having a very different problem with rabbitmq. Here is the use case > > 1) I have 2 rabbitmq installed. One is rmq1 and other is rmq2. > 2) I have 50 consumers for both the rabbitmq instances. > 3) I am seeing that all the rabbitmq consumers to rmq1 are hanged after > sometime however the consumers to rmq2 are up and running fine and are > never in a hang state. > 4) When I restart rmq1 all the consumers reconnect and are fine for > sometime however again they are hanged. > 5) So as to debug this issue I used "telnet rmq1 5672" and "telnet rmq2 > 5672". In case of "telnet rmq1 5672" I see that I am not able to close the > connection at all. > However "telnet rmq2 5672" once I hit a key I am able to close the > connection and the message is "Escape character is '^]'. ^C ^CConnection > closed by foreign host." > > Do you guys know in #5 why is there a difference in behavior? > > --Ashish > > > _______________________________________________ > rabbitmq-discuss mailing list > rabbitmq-discuss at lists.rabbitmq.com > https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From matthias at rabbitmq.com Tue Apr 22 07:26:13 2014 From: matthias at rabbitmq.com (Matthias Radestock) Date: Tue, 22 Apr 2014 07:26:13 +0100 Subject: [rabbitmq-discuss] Windows RabbitMQ Cluster In-Reply-To: <8a1539c0-f76f-44b2-af75-63d1956ddcdd@googlegroups.com> References: <8a1539c0-f76f-44b2-af75-63d1956ddcdd@googlegroups.com> Message-ID: <53560B85.1030404@rabbitmq.com> On 21/04/14 15:24, Wouter Bosland wrote: > 6. run the command: ''rabbitmqctl join_cluster rabbit at master'' on > machine 2 (slave) > > Clustering node 'rabbit at WIN-PK8SONHOIBH' with rabbit at master ... > Error: unable to connect to nodes [rabbit at master]: nodedown > [...] > *If I try the other way around clustering master to slave I get:* > > Clustering node 'rabbit at WIN-2BQPAJ1S03R' with rabbit at slave ... > Error: unable to connect to nodes [rabbit at slave]: nodedown > [...] > * suggestion: hostname mismatch? > * suggestion: is the cookie set correctly? > [...] > *The cookie hash is the same so what could be the problem?* See the 2nd suggestion :) It appears that your nodes know themselves as rabbit at WIN-<...>, but you attempt to address them as rabbit@{master,slave}. Now, evidently master/slave do resolve, since rabbitmqctl can find the machine, but for communication to succeed the name of the node must match how it is being addressed. So either address nodes by their WIN-<...> name or alter the node name on startup by setting RABBITMQ_NODENAME. Matthias. From ashjain2 at gmail.com Tue Apr 22 07:30:14 2014 From: ashjain2 at gmail.com (Ashish Jain) Date: Tue, 22 Apr 2014 12:00:14 +0530 Subject: [rabbitmq-discuss] Incredible problem with rabbitmq In-Reply-To: References: Message-ID: Logs do not say anything :(. For me there is not much of disk IO. It is just disconnecting and re-connecting to rmq. --Ashish On Tue, Apr 22, 2014 at 11:34 AM, Arun Rao wrote: > Your logs on rmq1 say anything? I have seen this behavior due to Disk IO > being terribly slow on one of my machines. > > > On Mon, Apr 21, 2014 at 10:56 PM, Ashish Jain wrote: > >> Guys, >> >> I am having a very different problem with rabbitmq. Here is the use case >> >> 1) I have 2 rabbitmq installed. One is rmq1 and other is rmq2. >> 2) I have 50 consumers for both the rabbitmq instances. >> 3) I am seeing that all the rabbitmq consumers to rmq1 are hanged after >> sometime however the consumers to rmq2 are up and running fine and are >> never in a hang state. >> 4) When I restart rmq1 all the consumers reconnect and are fine for >> sometime however again they are hanged. >> 5) So as to debug this issue I used "telnet rmq1 5672" and "telnet rmq2 >> 5672". In case of "telnet rmq1 5672" I see that I am not able to close the >> connection at all. >> However "telnet rmq2 5672" once I hit a key I am able to close the >> connection and the message is "Escape character is '^]'. ^C ^CConnection >> closed by foreign host." >> >> Do you guys know in #5 why is there a difference in behavior? >> >> --Ashish >> >> >> _______________________________________________ >> rabbitmq-discuss mailing list >> rabbitmq-discuss at lists.rabbitmq.com >> https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss >> >> > > _______________________________________________ > rabbitmq-discuss mailing list > rabbitmq-discuss at lists.rabbitmq.com > https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From matthias at rabbitmq.com Tue Apr 22 07:36:43 2014 From: matthias at rabbitmq.com (Matthias Radestock) Date: Tue, 22 Apr 2014 07:36:43 +0100 Subject: [rabbitmq-discuss] Incredible problem with rabbitmq In-Reply-To: References: Message-ID: <53560DFB.9060408@rabbitmq.com> On 22/04/14 07:30, Ashish Jain wrote: > Logs do not say anything :(. I find that hard to believe. Please post them here. Also, please post the output of 'rabbitmqctl status' for both nodes. Matthias. From wbosland.prive at gmail.com Tue Apr 22 07:56:22 2014 From: wbosland.prive at gmail.com (Wouter Bosland) Date: Tue, 22 Apr 2014 08:56:22 +0200 Subject: [rabbitmq-discuss] Windows RabbitMQ Cluster In-Reply-To: <53560B85.1030404@rabbitmq.com> References: <8a1539c0-f76f-44b2-af75-63d1956ddcdd@googlegroups.com> <53560B85.1030404@rabbitmq.com> Message-ID: Your 2nd suggestion solved it. Thank you very much :) I changed the host file and it now uses the name of my nodes. How do I change the node name via console? 2014-04-22 8:26 GMT+02:00 Matthias Radestock : > On 21/04/14 15:24, Wouter Bosland wrote: > >> 6. run the command: ''rabbitmqctl join_cluster rabbit at master'' on >> machine 2 (slave) >> >> Clustering node 'rabbit at WIN-PK8SONHOIBH' with rabbit at master ... >> Error: unable to connect to nodes [rabbit at master]: nodedown >> [...] >> *If I try the other way around clustering master to slave I get:* >> >> Clustering node 'rabbit at WIN-2BQPAJ1S03R' with rabbit at slave ... >> Error: unable to connect to nodes [rabbit at slave]: nodedown >> [...] >> * suggestion: hostname mismatch? >> * suggestion: is the cookie set correctly? >> [...] >> *The cookie hash is the same so what could be the problem?* >> > > See the 2nd suggestion :) > > It appears that your nodes know themselves as rabbit at WIN-<...>, but you > attempt to address them as rabbit@{master,slave}. > > Now, evidently master/slave do resolve, since rabbitmqctl can find the > machine, but for communication to succeed the name of the node must match > how it is being addressed. > > So either address nodes by their WIN-<...> name or alter the node name on > startup by setting RABBITMQ_NODENAME. > > Matthias. > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From SENTHIL.KUMARASAMY at GMAIL.COM Tue Apr 22 01:23:29 2014 From: SENTHIL.KUMARASAMY at GMAIL.COM (Sen) Date: Mon, 21 Apr 2014 17:23:29 -0700 (PDT) Subject: [rabbitmq-discuss] RabbimtMQ Managemnet API Message-ID: <1398126209271-34984.post@n5.nabble.com> I am trying to log the http request to a file, Here is the config, {rabbitmq_management, [{http_log_dir, "c:\\log"}]}, I would like to change the log format to include the time it took to execute the request. My question is, How and where do i make that change so that it include the time taken. Thanks Sen -- View this message in context: http://rabbitmq.1065348.n5.nabble.com/RabbimtMQ-Managemnet-API-tp34984.html Sent from the RabbitMQ mailing list archive at Nabble.com. From sasaki_hr at ariel-networks.com Tue Apr 22 07:43:54 2014 From: sasaki_hr at ariel-networks.com (=?ISO-2022-JP?B?GyRCOjQhOUxaGyhCIBskQjlAO0sbKEI=?=) Date: Tue, 22 Apr 2014 15:43:54 +0900 Subject: [rabbitmq-discuss] ***UNCHECKED*** cannot start RabbitMQ service on Windows Server 2012 R2 Message-ID: <53560FAA.5020003@ariel-networks.com> Hi, Rabbit MQ Term. This is Hiroshi Sasaki. I mail you because I'm warry about Rabbit MQ service. Rabbit MQ doesn't start on Windows Server 2012 R2. # When click "start" Rabbit MQ service on "Management of Computer" # Rabbit MQ looks like start properly, but reload the display by F5, # The state of Rabbit MQ service is not started. I could start Rabbit MQ properly on Windows Server 2008 R2 by the same way. I would like you to give me advice (how to try fix). I'll send you the log file when it's created at the time of try to start Rabbit MQ service. the password of the added file on this mail is "sasaki". best regards. -------------- next part -------------- A non-text attachment was scrubbed... Name: rabbitMQ.zip Type: application/x-zip-compressed Size: 74721 bytes Desc: not available URL: From ashjain2 at gmail.com Tue Apr 22 09:08:07 2014 From: ashjain2 at gmail.com (Ashish Jain) Date: Tue, 22 Apr 2014 13:38:07 +0530 Subject: [rabbitmq-discuss] Incredible problem with rabbitmq In-Reply-To: <53560DFB.9060408@rabbitmq.com> References: <53560DFB.9060408@rabbitmq.com> Message-ID: Due to confidentiality clause I cannot take the logs out of the environment. However in rmq1 logs there are few errors like this exception on TCP connection XXXXXXXXXXXXXXXXXXXXXXXXXXX {handshake_timeout,frame_header} exception on TCP connection {timeout,running} One more point to add here rmq1 and rmq2 are in 2 different networks. Here the client is in same network as rmq2 however client and rmq1 are in different network. Does this help or further information is required to debug this issue? On Tue, Apr 22, 2014 at 12:06 PM, Matthias Radestock wrote: > On 22/04/14 07:30, Ashish Jain wrote: > >> Logs do not say anything :(. >> > > I find that hard to believe. Please post them here. > > Also, please post the output of 'rabbitmqctl status' for both nodes. > > Matthias. > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From matthias at rabbitmq.com Tue Apr 22 09:12:38 2014 From: matthias at rabbitmq.com (Matthias Radestock) Date: Tue, 22 Apr 2014 09:12:38 +0100 Subject: [rabbitmq-discuss] Incredible problem with rabbitmq In-Reply-To: References: <53560DFB.9060408@rabbitmq.com> Message-ID: <53562476.3080104@rabbitmq.com> On 22/04/14 09:08, Ashish Jain wrote: > Due to confidentiality clause I cannot take the logs out of the > environment. However in rmq1 logs there are few errors like this > exception on TCP connection XXXXXXXXXXXXXXXXXXXXXXXXXXX > {handshake_timeout,frame_header} > > exception on TCP connection {timeout,running} > > One more point to add here rmq1 and rmq2 are in 2 different networks. > Here the client is in same network as rmq2 however client and rmq1 are > in different network. > > Does this help or further information is required to debug this issue? Could be a network routing issue - packets from the client get through to the server but not vice versa. That's also consistent with your 'telnet' observation. I suggest you repeat your telnet experiment while logged into the same machine (i.e. while on rmq1 telnet to rabbitmq on that machine); if that works but remote connections don't then it's definitely a network issue. Matthias. From mklishin at gopivotal.com Tue Apr 22 10:10:50 2014 From: mklishin at gopivotal.com (Michael Klishin) Date: Tue, 22 Apr 2014 13:10:50 +0400 Subject: [rabbitmq-discuss] Exchange Master failure? In-Reply-To: References: Message-ID: ?On 22 April 2014 at 09:57:47, Arun Rao (arunrao.seattle at gmail.com) wrote: > > I have a 4 node cluster and exchanges are HA'ed through a policy. > When one node (randomly chosen as Exchange master?) fails, I > see delivery rate dropping to zero for several seconds across > all queues in that cluster. (RabbitMQ Server version: 3.1.3) > > Can someone advise the logic behind Exchanges in a cluster? Exchanges cannot be HA?ed, or rather, they always are because exchanges are just routing tables. Yes, queues in RabbitMQ have a master and if it fails, it take some time to detect this and elect a new master. During this time there will be no deliveries for that queue because master node is in charge of that. -- MK Software Engineer, Pivotal/RabbitMQ From mklishin at gopivotal.com Tue Apr 22 10:15:10 2014 From: mklishin at gopivotal.com (Michael Klishin) Date: Tue, 22 Apr 2014 13:15:10 +0400 Subject: [rabbitmq-discuss] ***UNCHECKED*** cannot start RabbitMQ service on Windows Server 2012 R2 In-Reply-To: <53560FAA.5020003@ariel-networks.com> References: <53560FAA.5020003@ariel-networks.com> Message-ID: ?On 22 April 2014 at 12:01:41, ??? ?? (sasaki_hr at ariel-networks.com) wrote: > > I'll send you the log file when it's created at the time of try > to start Rabbit MQ service. Erlang tries to run?d:/Program Files (x86)/erl5.8.5/lib/os_mon-2.2.7/priv/bin/win32sysinfo.exe but that file isn?t there (or the VM process has no permission to access it?). -- MK Software Engineer, Pivotal/RabbitMQ From alix.baggio at gmail.com Tue Apr 22 11:02:00 2014 From: alix.baggio at gmail.com (Alix B) Date: Tue, 22 Apr 2014 03:02:00 -0700 (PDT) Subject: [rabbitmq-discuss] set content_type using PHP In-Reply-To: References: Message-ID: You can set the content_type for a AMQPMessage public function foo(AMQPMessage &$response) { $response->set('content_type', 'application/json'); /* ... */ } Le mercredi 9 avril 2014 17:09:48 UTC+2, aaron kenny a ?crit : > > I'm struggling to find out how to set the content_type='text/json'. ive > seen it set before via th basic_publish() function but not for PHP. Any > ideas. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From josephrouphael at gmail.com Tue Apr 22 14:06:37 2014 From: josephrouphael at gmail.com (joseph rouphael) Date: Tue, 22 Apr 2014 16:06:37 +0300 Subject: [rabbitmq-discuss] Fwd: Performance bottleneck on inter-node connection In-Reply-To: <534FDD90.30805@rabbitmq.com> References: <53465CF5.20607@rabbitmq.com> <5346C06C.8090609@rabbitmq.com> <534BA3C3.3080607@rabbitmq.com> <534FDD90.30805@rabbitmq.com> Message-ID: Thanks for your note regarding the scheduler. Indeed the performance doubled once I set +sfwi flag to 1. I could achieve 40KHZ inter-node instead of 20KHZ (Expected 100KHZ) I will try to play with other scheduler flags and see what will happen Thanks again. I would appreciate updating me if you got the chance to have more info related to this inter-node bottleneck. On Thu, Apr 17, 2014 at 4:56 PM, Matthias Radestock wrote: > On 15/04/14 14:20, joseph rouphael wrote: > >> On Mon, Apr 14, 2014 at 12:00 PM, Matthias Radestock >> > wrote: >> What happens when you run the above test with both nodes on the same >> machine? >> >> The result is almost the same: Fluctuating between 10 and 23KHZ >> > > Thanks for running all these tests. I've reproduced your observations. > > What we are seeing here is, at least partially, an issue with the Erlang > scheduler. > > I noticed that when running the test with PerfTest, single producer, > single consumer, two nodes, the consuming node was using 100% CPU, i.e. the > equivalent of a single core. That is usually an indication of an Erlang > scheduler issue since there is an appreciable degree of parallelism that > should be exploitable. > > For this particular test I managed to double performance by supplying the > "+swt very_low" or "+sfwi 1" flags to Erlang via RABBITMQ_SERVER_ERL_ARGS. > > So these flags, plus the other +s... flags of the Erlang VM, are worth > playing with. > > Matthias. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From josephrouphael at gmail.com Tue Apr 22 14:17:52 2014 From: josephrouphael at gmail.com (joseph rouphael) Date: Tue, 22 Apr 2014 16:17:52 +0300 Subject: [rabbitmq-discuss] Fwd: Performance bottleneck on inter-node connection In-Reply-To: References: <53465CF5.20607@rabbitmq.com> <5346C06C.8090609@rabbitmq.com> <534BA3C3.3080607@rabbitmq.com> <534FDD90.30805@rabbitmq.com> Message-ID: Thanks Jason for sharing these articles. I will get a look on them to see if they can help. On Thu, Apr 17, 2014 at 7:54 PM, Jason McIntosh wrote: > Watch also NUMA stuff on these bigger multi-core systems. I've seen some > very interesting effects due to NUMA and how it affects erlang. Look at > these posts for some more details: > > http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/2013-January/024790.html > > Typically a system with as many cores as you've got requires some more > "tweaking" to get good performance out of it. One of the best articles I'd > found was this: > http://www.agoragames.com/blog/2011/06/24/of-penguins-rabbits-and-buses/ > > > I don't THINK this would have an impact on your issues with inter-node > communications but I have to toss it out there. Also, another fun thing we > saw was long DNS lookups causing some odd issues as well. > Jason > > > On Thu, Apr 17, 2014 at 8:56 AM, Matthias Radestock > wrote: > >> On 15/04/14 14:20, joseph rouphael wrote: >> >>> On Mon, Apr 14, 2014 at 12:00 PM, Matthias Radestock >>> > wrote: >>> What happens when you run the above test with both nodes on the same >>> machine? >>> >>> The result is almost the same: Fluctuating between 10 and 23KHZ >>> >> >> Thanks for running all these tests. I've reproduced your observations. >> >> What we are seeing here is, at least partially, an issue with the Erlang >> scheduler. >> >> I noticed that when running the test with PerfTest, single producer, >> single consumer, two nodes, the consuming node was using 100% CPU, i.e. the >> equivalent of a single core. That is usually an indication of an Erlang >> scheduler issue since there is an appreciable degree of parallelism that >> should be exploitable. >> >> For this particular test I managed to double performance by supplying the >> "+swt very_low" or "+sfwi 1" flags to Erlang via RABBITMQ_SERVER_ERL_ARGS. >> >> So these flags, plus the other +s... flags of the Erlang VM, are worth >> playing with. >> >> >> Matthias. >> >> _______________________________________________ >> rabbitmq-discuss mailing list >> rabbitmq-discuss at lists.rabbitmq.com >> https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss >> > > > > -- > Jason McIntosh > https://github.com/jasonmcintosh/ > 573-424-7612 > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jmarlin at saucontech.com Tue Apr 22 15:34:15 2014 From: jmarlin at saucontech.com (Joseph Marlin) Date: Tue, 22 Apr 2014 10:34:15 -0400 (EDT) Subject: [rabbitmq-discuss] Zero message rates in management interface In-Reply-To: <145365085.115208545.1398176352596.JavaMail.root@saucontech.com> Message-ID: <450825394.115260544.1398177255446.JavaMail.root@saucontech.com> All of our RabbitMQ 3.3.0 queues are incorrectly reporting message rates or number of queued messages for the previous ~15 seconds in the web management interface. This can be seen by scrutinizing the following two images. In the first image, http://i.imgur.com/LiFFyTw.png , at 10:21:30, originally a zero message rate is reported, and the the queue size reported is incorrectly still at the same value from fifteen seconds ago. In the second image, http://i.imgur.com/9hxhKd6.png , you can see that the management interface now displays the correct data for the previous image's time, 10:21:30, and furthermore, the problem is continuing to occur at 10:22:00. It continues on this way, never showing the data for the previous few seconds, but eventually, after enough time has passed, it will display it. Problem is the same across browsers. Any ideas? Thanks much! Status of node rabbit at queue ... [{pid,995}, {running_applications, [{rabbitmq_management_visualiser,"RabbitMQ Visualiser","3.3.0"}, {rabbitmq_management,"RabbitMQ Management Console","3.3.0"}, {rabbitmq_management_agent,"RabbitMQ Management Agent","3.3.0"}, {rabbit,"RabbitMQ","3.3.0"}, {os_mon,"CPO CXC 138 46","2.2.7"}, {rabbitmq_web_dispatch,"RabbitMQ Web Dispatcher","3.3.0"}, {webmachine,"webmachine","1.10.3-rmq3.3.0-gite9359c7"}, {mochiweb,"MochiMedia Web Server","2.7.0-rmq3.3.0-git680dba8"}, {xmerl,"XML parser","1.2.10"}, {cowboy,"Small, fast, modular HTTP server.","0.5.0-rmq3.3.0-git4b93c2d"}, {sockjs,"SockJS","0.3.4-rmq3.3.0-git3132eb9"}, {inets,"INETS CXC 138 49","5.7.1"}, {mnesia,"MNESIA CXC 138 12","4.5"}, {amqp_client,"RabbitMQ AMQP Client","3.3.0"}, {sasl,"SASL CXC 138 11","2.1.10"}, {stdlib,"ERTS CXC 138 10","1.17.5"}, {kernel,"ERTS CXC 138 10","2.14.5"}]}, {os,{unix,linux}}, {erlang_version, "Erlang R14B04 (erts-5.8.5) [source] [64-bit] [smp:2:2] [rq:2] [async-threads:30] [kernel-poll:true]\n"}, {memory, [{total,66382512}, {connection_procs,12156344}, {queue_procs,5212592}, {plugins,1461448}, {other_proc,10577128}, {mnesia,227464}, {mgmt_db,7024816}, {msg_index,547848}, {other_ets,3443176}, {binary,4247976}, {code,17811775}, {atom,1620513}, {other_system,2051432}]}, {alarms,[]}, {listeners,[{clustering,25672,"::"},{amqp,5672,"::"}]}, {vm_memory_high_watermark,0.4}, {vm_memory_limit,840315699}, {disk_free_limit,50000000}, {disk_free,27659542528}, {file_descriptors, [{total_limit,924}, {total_used,28}, {sockets_limit,829}, {sockets_used,16}]}, {processes,[{limit,1048576},{used,1641}]}, {run_queue,0}, {uptime,43659}] ...done. From bjoernhaeuser at gmail.com Tue Apr 22 17:45:18 2014 From: bjoernhaeuser at gmail.com (=?iso-8859-1?Q?Bj=F6rn_H=E4user?=) Date: Tue, 22 Apr 2014 18:45:18 +0200 Subject: [rabbitmq-discuss] 3.2.4: Segfaults with HiPe enabled In-Reply-To: <5354CB95.2040201@rabbitmq.com> References: <5354CB95.2040201@rabbitmq.com> Message-ID: <47148C41-E224-4733-8FF1-442F433CECEA@gmail.com> Hello, sorry for answering in the wrong thread. We will upgrade to newest RabbitMQ. We are also running R16B03. Should we also go for R17? Thanks Bj?rn On 21.04.2014, at 09:41, Matthias Radestock wrote: > On 19/04/14 16:54, Bj?rn H?user wrote: >> Can we anything do to have HiPe enabled without these crashes? ;) > > Try to > > 1) upgrade Erlang - bugs in HiPE get fixed in pretty much every new release > > 2) upgrade RabbitMQ - one possible cause for the memory allocation failure is that a large RabbitMQ process crashed and Erlang then died trying to display the process state. We addressed that issue in 3.3.0 > > Matthias. From matthias at rabbitmq.com Tue Apr 22 17:55:19 2014 From: matthias at rabbitmq.com (Matthias Radestock) Date: Tue, 22 Apr 2014 17:55:19 +0100 Subject: [rabbitmq-discuss] 3.2.4: Segfaults with HiPe enabled In-Reply-To: <47148C41-E224-4733-8FF1-442F433CECEA@gmail.com> References: <5354CB95.2040201@rabbitmq.com> <47148C41-E224-4733-8FF1-442F433CECEA@gmail.com> Message-ID: <53569EF7.8050108@rabbitmq.com> On 22/04/14 17:45, Bj?rn H?user wrote: > We will upgrade to newest RabbitMQ. We are also running R16B03. > Should we also go for R17? Yes. (I knew you were running R16B03 from the crash dump) Matthias. From bjoernhaeuser at gmail.com Tue Apr 22 18:35:51 2014 From: bjoernhaeuser at gmail.com (=?windows-1252?Q?Bj=F6rn_H=E4user?=) Date: Tue, 22 Apr 2014 19:35:51 +0200 Subject: [rabbitmq-discuss] Scaling with RabbitMQ Message-ID: <4A7DAC2A-2561-4D00-87B5-285C988741DC@gmail.com> Hello, we are currently reviewing our current architecture, which is mostly based on RabbitMQ and Java. To keep up with the expected growth of our whole system, I want to discuss if our current approach of using RabbitMQ is in a good shape. We have several services, each running with at least two instances. Each service registers several queues (between 5 and 25). Each queue has two bindings. Every instance has one connection to rabbitmq, for every queue we are opening a new channel. One additional channel is used for publishing messages. The services communicate with each other through one topic exchange. We are not using any wild card bindings currently and do not plan to use many of them in the near future. We are not aiming for maximal throughput, our goal is maximum durability. Therefore we are using delivery_mode = 2, durable and ha queues. We are currently running a cluster which consists of two nodes (each has 4GB RAM and 4 CPUs, using EXT4). These are the numbers of our production system: 34 Connections 431 Channels 227 Queues 393 Consumers I tried to find these information on the rabbitmq website, but was not able to find them. Here are the questions I currently don?t have an answer to: 1. Topic exchange is the slowest exchange type. Direct exchange would be faster. We would lose the ability to have wildcard bindings. Can someone put a number of how much would we gain when we switch to a direct exchange? 2. Publishing rate depends on how many queues have a binding for the published message. We use java class names as routing keys. (com.foo.bar.xxx.zzz). The messages start to differ from the xxx part. Does the structure of the routing key influences publishing speed? What would be the impact of drastically increasing the number of queues (and bindings)? I am thinking of having thousand queues. 3. Is it wise to run everything over the same exchange? As far as I read on the mailing list, different exchanges on the same cluster do not influence each other - as long as there are enough physical resources. 4. How do we know, that we hit some limit of rabbitmq by using it the wrong way? Is there any way to see and monitor internal rabbitmq metrics? What?s your favorite tool to monitor RabbitMQ? 5. What do you think about having a Channel per Queue? That decision made us able to set the prefetch size and thread count per queue on the client side. Finally I want to thank you for reading my mail. If this is complete nonsense, please tell me. I am new to the land of rabbits (sorry for this bad joke). And additionally I want to thank every developer who puts his time into rabbitmq. It is an awesome piece of software. Thanks Bj?rn From 0x6e6562 at gmail.com Tue Apr 22 18:46:43 2014 From: 0x6e6562 at gmail.com (Ben Hood) Date: Tue, 22 Apr 2014 18:46:43 +0100 Subject: [rabbitmq-discuss] Prefetch fields in management UI Message-ID: Hi all, Does anybody know what the difference is on the per-queue view in the management UI between the prefetch value in the details pane versus the value reported in the list of consumers? I've appended a couple of screenshots to better highlight the point (with and without the qos sharing flag). Cheers, Ben PS This is a 3.3.0/16B03/OSX 10.9.2 combo. -------------- next part -------------- A non-text attachment was scrubbed... Name: Screenshot 2014-04-22 18.41.24.png Type: image/png Size: 31610 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Screenshot 2014-04-22 18.41.46.png Type: image/png Size: 31506 bytes Desc: not available URL: From 0x6e6562 at gmail.com Tue Apr 22 19:19:46 2014 From: 0x6e6562 at gmail.com (Ben Hood) Date: Tue, 22 Apr 2014 19:19:46 +0100 Subject: [rabbitmq-discuss] Prefetch fields in management UI In-Reply-To: References: Message-ID: On Tue, Apr 22, 2014 at 6:46 PM, Ben Hood <0x6e6562 at gmail.com> wrote: > Does anybody know what the difference is on the per-queue view in the > management UI between the prefetch value in the details pane versus > the value reported in the list of consumers? I've appended a couple of > screenshots to better highlight the point (with and without the qos > sharing flag). I have a feeling that this might be a client issue. The screenshots I sent were made using an the Go amqp driver - the official Java client doesn't seem to have this issue. Sorry about the noise. From mklishin at gopivotal.com Tue Apr 22 19:34:50 2014 From: mklishin at gopivotal.com (Michael Klishin) Date: Tue, 22 Apr 2014 22:34:50 +0400 Subject: [rabbitmq-discuss] Scaling with RabbitMQ In-Reply-To: <4A7DAC2A-2561-4D00-87B5-285C988741DC@gmail.com> References: <4A7DAC2A-2561-4D00-87B5-285C988741DC@gmail.com> Message-ID: On 22 April 2014 at 21:37:54, Bj?rn H?user (bjoernhaeuser at gmail.com) wrote: > > 1. Topic exchange is the slowest exchange type. Direct exchange > would be faster. We would lose the ability to have wildcard bindings. > Can someone put a number of how much would we gain when we switch > to a direct exchange? I?d recommend doing your own benchmarks instead of believing numbers posted on the Internet. > 2. Publishing rate depends on how many queues have a binding for > the published message. We use java class names as routing keys. > (com.foo.bar.xxx.zzz). The messages start to differ from the > xxx part. Does the structure of the routing key influences publishing > speed? What would be the impact of drastically increasing the > number of queues (and bindings)? I am thinking of having thousand > queues. Topic routing uses a trie internally which doesn?t get much slower as the number of bindings increases or topic length grows. However, since every queue gets a copy of the message, there will be some impact, in particular with larger messages. > > 3. Is it wise to run everything over the same exchange? As far as > I read on the mailing list, different exchanges on the same cluster > do not influence each other - as long as there are enough physical > resources. Exchanges are just named routing tables. Using a single exchange vs multiple exchanges of the same type should make no difference. > > 4. How do we know, that we hit some limit of rabbitmq by using it > the wrong way? Is there any way to see and monitor internal rabbitmq > metrics? What?s your favorite tool to monitor RabbitMQ? RabbitMQ management UI and HTTP API that ships with it. > 5. What do you think about having a Channel per Queue? That decision > made us able to set the prefetch size and thread count per queue > on the client side. Every channel consumes a bit of RAM but it should work fine.? -- MK Software Engineer, Pivotal/RabbitMQ From Anita.Yip at RealPage.com Tue Apr 22 22:23:09 2014 From: Anita.Yip at RealPage.com (Anita Yip) Date: Tue, 22 Apr 2014 21:23:09 +0000 Subject: [rabbitmq-discuss] RabbitMQ HA set up not working as expected Message-ID: I have a question regarding the High Availability set up for RabbitMq servers. I have 2 RabbitMq servers set up to be in HA mode, and the queues in one server is automatically replicated on the other server. But when I bring one of those servers down, if I stop the server on which I primarily created the queue, then I can no longer access that queue even on server-2. And once I restart the first server, the queue is safely retained. Is this the intended behavior? How can I configure the servers/queues so that even if one of the RabbitMQ servers is down, the other should be able to handle the requests seemlessly. Thanks, Anita ________________________________ This message is intended only for the use of the individual(s) or entity to which it is addressed and may contain information that is privileged, confidential, and/or proprietary to RealPage and its affiliated companies. If the reader of this message is not the intended recipient, you are hereby notified that any dissemination, distribution, forwarding or copying of this communication is prohibited without the express permission of the sender. If you have received this communication in error, please notify the sender immediately and delete the original message. -------------- next part -------------- An HTML attachment was scrubbed... URL: From Anita.Yip at RealPage.com Wed Apr 23 00:03:50 2014 From: Anita.Yip at RealPage.com (Anita Yip) Date: Tue, 22 Apr 2014 23:03:50 +0000 Subject: [rabbitmq-discuss] RabbitMQ HA set up not working as expected Message-ID: Ignore the below email, as it turned out the set up is not correct. On a similar note, can you please let me know if high availability set up is possible at a cluster level? If I have Rs1 and Rs2 rabbit servers in cluster1 and Rs3 and Rs4 servers in cluster 2, how can I configure Rs1 and Rs2 to load balance the requests and have cluster1 set up in high availability with cluster2? From: Anita Yip > Reply-To: Discussions about RabbitMQ > Date: Tuesday, April 22, 2014 2:23 PM To: Discussions about RabbitMQ > Subject: [rabbitmq-discuss] RabbitMQ HA set up not working as expected I have a question regarding the High Availability set up for RabbitMq servers. I have 2 RabbitMq servers set up to be in HA mode, and the queues in one server is automatically replicated on the other server. But when I bring one of those servers down, if I stop the server on which I primarily created the queue, then I can no longer access that queue even on server-2. And once I restart the first server, the queue is safely retained. Is this the intended behavior? How can I configure the servers/queues so that even if one of the RabbitMQ servers is down, the other should be able to handle the requests seemlessly. Thanks, Anita ________________________________ This message is intended only for the use of the individual(s) or entity to which it is addressed and may contain information that is privileged, confidential, and/or proprietary to RealPage and its affiliated companies. If the reader of this message is not the intended recipient, you are hereby notified that any dissemination, distribution, forwarding or copying of this communication is prohibited without the express permission of the sender. If you have received this communication in error, please notify the sender immediately and delete the original message. ________________________________ This message is intended only for the use of the individual(s) or entity to which it is addressed and may contain information that is privileged, confidential, and/or proprietary to RealPage and its affiliated companies. If the reader of this message is not the intended recipient, you are hereby notified that any dissemination, distribution, forwarding or copying of this communication is prohibited without the express permission of the sender. If you have received this communication in error, please notify the sender immediately and delete the original message. -------------- next part -------------- An HTML attachment was scrubbed... URL: From jkuch at gopivotal.com Wed Apr 23 00:08:28 2014 From: jkuch at gopivotal.com (Jerry Kuch) Date: Tue, 22 Apr 2014 16:08:28 -0700 Subject: [rabbitmq-discuss] RabbitMQ HA set up not working as expected In-Reply-To: References: Message-ID: On Tue, Apr 22, 2014 at 4:03 PM, Anita Yip wrote: > Ignore the below email, as it turned out the set up is not correct. On > a similar note, can you please let me know if high availability set up is > possible at a cluster level? > In old Rabbits HA/mirroring is a property of the queue, granted to it when the queue is declared. In newer Rabbits, HA is defined by a policy, which applies to a pattern specifying the names of the objects the policy is supposed to modify. For details see here: http://www.rabbitmq.com/ha.html Are you sure you want to mirror absolutely all of the queues? -------------- next part -------------- An HTML attachment was scrubbed... URL: From 0x6e6562 at gmail.com Wed Apr 23 00:21:56 2014 From: 0x6e6562 at gmail.com (Ben Hood) Date: Wed, 23 Apr 2014 00:21:56 +0100 Subject: [rabbitmq-discuss] Prefetch fields in management UI In-Reply-To: References: Message-ID: On Tue, Apr 22, 2014 at 7:19 PM, Ben Hood <0x6e6562 at gmail.com> wrote: > I have a feeling that this might be a client issue. The screenshots I > sent were made using an the Go amqp driver - the official Java client > doesn't seem to have this issue. Sorry about the noise. Turns out this issue is reproducible with the official Java client, but it might just be due to lack of RTFM. So qos does not appear to take effect if the command is submitted after the call to subscribe to a queue, with in the context of a single channel. So to fix this issue, all I needed to do is to make sure that setting the prefetch happens before subscribing to a queue. I never realized that egress qos is order dependent - I had just assumed that that it is a changeable value within the context of the limiter. What puzzled me is that the same app running against 3.2.4 does not (anecdotally) seem to care as much about the command order as the current release. I probably need to re-read the documentation. From lepolac at gmail.com Tue Apr 22 20:02:16 2014 From: lepolac at gmail.com (lepolac) Date: Tue, 22 Apr 2014 12:02:16 -0700 (PDT) Subject: [rabbitmq-discuss] Clustering and scaling Message-ID: <1398193336627-35008.post@n5.nabble.com> Hello, I have a question regarding how clustering helps to scale. In a context where you mainly have clustered queues with sync policy, everything must hit the master first to get then propagated to the rest of the cluster. One can then see the master as the bottleneck. If I have a total of X connections, with a total of Y msg/s, am I really better of with let's say 5 nodes (so X/5 connections/node et Y/5 msg/s/node assuming we're in an ideal world....) compared to one node ? I'm not talking about the resiliency side of things which is pretty obvious. To sum up, can clustering be considered as more than adding resiliency and help to scale up, or is exchange federation a better option for this ? Regards -- View this message in context: http://rabbitmq.1065348.n5.nabble.com/Clustering-and-scaling-tp35008.html Sent from the RabbitMQ mailing list archive at Nabble.com. From lepolac at gmail.com Tue Apr 22 20:03:35 2014 From: lepolac at gmail.com (lepolac) Date: Tue, 22 Apr 2014 12:03:35 -0700 (PDT) Subject: [rabbitmq-discuss] Partition handling configuration Message-ID: <1398193415066-35009.post@n5.nabble.com> Hi, I would like to know how is the partition handling configuration propagated within a cluster. If I have say 2 nodes with autoheal and 3 nodes without anything specified, will the autoheal propagate ? What happens if you have autoheal on some and pause_minority on the rest, does the majority of nodes with the same config win ? Regards -- View this message in context: http://rabbitmq.1065348.n5.nabble.com/Partition-handling-configuration-tp35009.html Sent from the RabbitMQ mailing list archive at Nabble.com. From lepolac at gmail.com Tue Apr 22 20:08:20 2014 From: lepolac at gmail.com (lepolac) Date: Tue, 22 Apr 2014 12:08:20 -0700 (PDT) Subject: [rabbitmq-discuss] Federated exchange and sync delivery Message-ID: <1398193700805-35010.post@n5.nabble.com> Hi, In the documentation it appears that the federation is seen as asynchronous. What are exactly the delivery guarantees implemented in the federation ? When are messages acknowledged by the downstream ? Is there an equivalent of publisher confirms used internally for persistent messaging ? On the consuming side, when does a message disappear from the queue ? How far a downstream exchange lags behind its upstream peer ? The overall question I'm trying to answer is, is federation viable for BCP purposes or is it too loosely coupled to take this risk. Thanks ! -- View this message in context: http://rabbitmq.1065348.n5.nabble.com/Federated-exchange-and-sync-delivery-tp35010.html Sent from the RabbitMQ mailing list archive at Nabble.com. From tjanczuk33 at gmail.com Wed Apr 23 01:43:08 2014 From: tjanczuk33 at gmail.com (Tomasz Janczuk) Date: Tue, 22 Apr 2014 17:43:08 -0700 (PDT) Subject: [rabbitmq-discuss] Preventing DoS in a multi-tenant Rabbit deployment Message-ID: <4f91d062-8c04-44e6-90e0-f852e11e79db@googlegroups.com> What is the best mechanism to prevent authenticated DoS attacks in a multi-tenant Rabbit deployment? Is there one? By authenticated DoS attack I mean an attack in which an authenticated tenant causes a level of resource consumption in a Rabbit deployment that prevents other tenants from using the service or severely degrades the performance. Vhosts appear to support a level of entity isolation that is desired between tenants in a multi-tenant deployment, but do they support setting limits to prevent DoS? For example, limiting the message throughput, number of channels and connections, message sizes, number of queues and exchanges? Is process-isolation an adequate solution? Does assigning individual Rabbit nodes to tenants allow limits to be set to prevent DoS? Or is using dedicated VMs the only way to achieve the level of isolation that will prevent authenticated DoS? Thanks, Tomasz Janczuk -------------- next part -------------- An HTML attachment was scrubbed... URL: From simon at rabbitmq.com Wed Apr 23 08:38:16 2014 From: simon at rabbitmq.com (Simon MacMullen) Date: Wed, 23 Apr 2014 08:38:16 +0100 Subject: [rabbitmq-discuss] The meaning of latency in the PerfTest module In-Reply-To: <6B4CB2165485294B8C9571963FDDBD4822CE83AA@KTVCAEX-MB-01.desire2learn.d2l> References: <6B4CB2165485294B8C9571963FDDBD4822CE83AA@KTVCAEX-MB-01.desire2learn.d2l> Message-ID: <53576DE8.7000000@rabbitmq.com> On 21/04/2014 22:09, Rouzbeh Farahmand wrote: > Do we mean the time that it takes to publish a message to a queue and > consume the same message from the same queue? . ( i.e. latency = > wall-time of sending the message by publisher ? wall-time of receiving > by the consumer) This one. Note that you get latency numbers out of PerfTest when you have a producer and a consumer, but you don't have to use publisher confirms. Cheers, Simon From simon at rabbitmq.com Wed Apr 23 08:39:59 2014 From: simon at rabbitmq.com (Simon MacMullen) Date: Wed, 23 Apr 2014 08:39:59 +0100 Subject: [rabbitmq-discuss] RabbimtMQ Managemnet API In-Reply-To: <1398126209271-34984.post@n5.nabble.com> References: <1398126209271-34984.post@n5.nabble.com> Message-ID: <53576E4F.4010403@rabbitmq.com> I'm afraid the log format in management is just what's offered by Webmachine. You can't alter the format, or even the rotation frequency. This might hcange one day, but it's not a high priority. Cheers, Simon On 22/04/2014 01:23, Sen wrote: > I am trying to log the http request to a file, Here is the config, > {rabbitmq_management, [{http_log_dir, "c:\\log"}]}, > I would like to change the log format to include the time it took to execute > the request. > My question is, How and where do i make that change so that it include the > time taken. > > Thanks > Sen > > > > -- > View this message in context: http://rabbitmq.1065348.n5.nabble.com/RabbimtMQ-Managemnet-API-tp34984.html > Sent from the RabbitMQ mailing list archive at Nabble.com. > _______________________________________________ > rabbitmq-discuss mailing list > rabbitmq-discuss at lists.rabbitmq.com > https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss > From simon at rabbitmq.com Wed Apr 23 08:42:18 2014 From: simon at rabbitmq.com (Simon MacMullen) Date: Wed, 23 Apr 2014 08:42:18 +0100 Subject: [rabbitmq-discuss] x509 proxy file for user authentication In-Reply-To: References: Message-ID: <53576EDA.8060804@rabbitmq.com> On 22/04/2014 03:58, Doug Benjamin wrote: > I would like to use the ssl certificate log into the rabbit management > and rabbit server ports You can use the EXTERNAL mechanism to connect AMQP without specifying a password. There's no equivalent support in management though I'm afraid. Cheers, Simon From simon at rabbitmq.com Wed Apr 23 09:02:51 2014 From: simon at rabbitmq.com (Simon MacMullen) Date: Wed, 23 Apr 2014 09:02:51 +0100 Subject: [rabbitmq-discuss] Defining Default Policy? In-Reply-To: References: Message-ID: <535773AB.3060902@rabbitmq.com> On 17/04/2014 14:58, Dayton Gomez wrote: > I'm looking for a way to define a high-availability policy > automatically. After much googling, I've not found an easy way to do so > (like definitions in a config file). I do have some provisioning > software, but was hoping for a native way to set things up. http://www.rabbitmq.com/management.html#load-definitions might be what you're looking for? Cheers, Simon From simon at rabbitmq.com Wed Apr 23 09:04:25 2014 From: simon at rabbitmq.com (Simon MacMullen) Date: Wed, 23 Apr 2014 09:04:25 +0100 Subject: [rabbitmq-discuss] Partition handling configuration In-Reply-To: <1398193415066-35009.post@n5.nabble.com> References: <1398193415066-35009.post@n5.nabble.com> Message-ID: <53577409.4020305@rabbitmq.com> On 22/04/2014 20:03, lepolac wrote: > I would like to know how is the partition handling configuration propagated > within a cluster. > If I have say 2 nodes with autoheal and 3 nodes without anything specified, > will the autoheal propagate ? No, you need to set it the same on all nodes. > What happens if you have autoheal on some and pause_minority on the rest, > does the majority of nodes with the same config win ? Things will work strangely. Cheers, Simon From simon at rabbitmq.com Wed Apr 23 09:08:18 2014 From: simon at rabbitmq.com (Simon MacMullen) Date: Wed, 23 Apr 2014 09:08:18 +0100 Subject: [rabbitmq-discuss] Prefetch fields in management UI In-Reply-To: References: Message-ID: <535774F2.4070200@rabbitmq.com> On 23/04/2014 00:21, Ben Hood wrote: > So qos does not appear to > take effect if the command is submitted after the call to subscribe to > a queue, with in the context of a single channel. So to fix this > issue, all I needed to do is to make sure that setting the prefetch > happens before subscribing to a queue. I never realized that egress > qos is order dependent - I had just assumed that that it is a > changeable value within the context of the limiter. Note that the semantics of qos have changed in 3.3.0: http://www.rabbitmq.com/consumer-prefetch.html Per-consumer prefetch will be fixed at the moment the consumer is created. Per-channel prefetch can be changed after the fact, as before. But it was never a great idea to do that since if you set qos after consuming then the broker can deliver as many messages as it wants in between the two commands, which is unlikely to be what you want. Cheers, Simon From simon at rabbitmq.com Wed Apr 23 09:49:47 2014 From: simon at rabbitmq.com (Simon MacMullen) Date: Wed, 23 Apr 2014 09:49:47 +0100 Subject: [rabbitmq-discuss] Zero message rates in management interface In-Reply-To: <450825394.115260544.1398177255446.JavaMail.root@saucontech.com> References: <450825394.115260544.1398177255446.JavaMail.root@saucontech.com> Message-ID: <53577EAB.2010005@rabbitmq.com> Do you have a cluster? Is there clock skew between the nodes? Cheers, Simon On 22/04/2014 15:34, Joseph Marlin wrote: > All of our RabbitMQ 3.3.0 queues are incorrectly reporting message rates or number of queued messages for the previous ~15 seconds in the web management interface. > > This can be seen by scrutinizing the following two images. In the first image, http://i.imgur.com/LiFFyTw.png , at 10:21:30, originally a zero message rate is reported, and the the queue size reported is incorrectly still at the same value from fifteen seconds ago. > > In the second image, http://i.imgur.com/9hxhKd6.png , you can see that the management interface now displays the correct data for the previous image's time, 10:21:30, and furthermore, the problem is continuing to occur at 10:22:00. It continues on this way, never showing the data for the previous few seconds, but eventually, after enough time has passed, it will display it. > > Problem is the same across browsers. > > Any ideas? Thanks much! > > > > > > Status of node rabbit at queue ... > [{pid,995}, > {running_applications, > [{rabbitmq_management_visualiser,"RabbitMQ Visualiser","3.3.0"}, > {rabbitmq_management,"RabbitMQ Management Console","3.3.0"}, > {rabbitmq_management_agent,"RabbitMQ Management Agent","3.3.0"}, > {rabbit,"RabbitMQ","3.3.0"}, > {os_mon,"CPO CXC 138 46","2.2.7"}, > {rabbitmq_web_dispatch,"RabbitMQ Web Dispatcher","3.3.0"}, > {webmachine,"webmachine","1.10.3-rmq3.3.0-gite9359c7"}, > {mochiweb,"MochiMedia Web Server","2.7.0-rmq3.3.0-git680dba8"}, > {xmerl,"XML parser","1.2.10"}, > {cowboy,"Small, fast, modular HTTP server.","0.5.0-rmq3.3.0-git4b93c2d"}, > {sockjs,"SockJS","0.3.4-rmq3.3.0-git3132eb9"}, > {inets,"INETS CXC 138 49","5.7.1"}, > {mnesia,"MNESIA CXC 138 12","4.5"}, > {amqp_client,"RabbitMQ AMQP Client","3.3.0"}, > {sasl,"SASL CXC 138 11","2.1.10"}, > {stdlib,"ERTS CXC 138 10","1.17.5"}, > {kernel,"ERTS CXC 138 10","2.14.5"}]}, > {os,{unix,linux}}, > {erlang_version, > "Erlang R14B04 (erts-5.8.5) [source] [64-bit] [smp:2:2] [rq:2] [async-threads:30] [kernel-poll:true]\n"}, > {memory, > [{total,66382512}, > {connection_procs,12156344}, > {queue_procs,5212592}, > {plugins,1461448}, > {other_proc,10577128}, > {mnesia,227464}, > {mgmt_db,7024816}, > {msg_index,547848}, > {other_ets,3443176}, > {binary,4247976}, > {code,17811775}, > {atom,1620513}, > {other_system,2051432}]}, > {alarms,[]}, > {listeners,[{clustering,25672,"::"},{amqp,5672,"::"}]}, > {vm_memory_high_watermark,0.4}, > {vm_memory_limit,840315699}, > {disk_free_limit,50000000}, > {disk_free,27659542528}, > {file_descriptors, > [{total_limit,924}, > {total_used,28}, > {sockets_limit,829}, > {sockets_used,16}]}, > {processes,[{limit,1048576},{used,1641}]}, > {run_queue,0}, > {uptime,43659}] > ...done. > > > > > _______________________________________________ > rabbitmq-discuss mailing list > rabbitmq-discuss at lists.rabbitmq.com > https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss > From simon at rabbitmq.com Wed Apr 23 09:53:29 2014 From: simon at rabbitmq.com (Simon MacMullen) Date: Wed, 23 Apr 2014 09:53:29 +0100 Subject: [rabbitmq-discuss] Clustering and scaling In-Reply-To: <1398193336627-35008.post@n5.nabble.com> References: <1398193336627-35008.post@n5.nabble.com> Message-ID: <53577F89.9050203@rabbitmq.com> On 22/04/2014 20:02, lepolac wrote: > I'm not talking about the resiliency side of things which is pretty obvious. > To sum up, can clustering be considered as more than adding resiliency and > help to scale up, or is exchange federation a better option for this ? Well, you don't need to have all queues mirrored on all nodes; if you have unmirrored queues or those with ha-mode "nodes" or "exactly" then you can get more queue-capacity across a cluster. Note also that connection and channel processes will take some CPU; distributing that around a cluster will also be beneficial. But federation might be better for your case; the difference is more to do with reliable vs unreliable networks than anything else. See http://www.rabbitmq.com/distributed.html if you haven't already seen it. Cheers, Simon From mklishin at gopivotal.com Wed Apr 23 10:56:16 2014 From: mklishin at gopivotal.com (Michael Klishin) Date: Wed, 23 Apr 2014 13:56:16 +0400 Subject: [rabbitmq-discuss] Federated exchange and sync delivery In-Reply-To: <1398193700805-35010.post@n5.nabble.com> References: <1398193700805-35010.post@n5.nabble.com> Message-ID: On 23 April 2014 at 11:20:35, lepolac (lepolac at gmail.com) wrote: > > What are exactly the delivery guarantees implemented in the > federation ? > When are messages acknowledged by the downstream ? > Is there an equivalent of publisher confirms used internally > for persistent > messaging ? Publisher confirms and persistent storage are orthogonal things. Federation uses regular AMQP client connections between nodes, with confirms and handling of network failures. > The overall question I'm trying to answer is, is federation viable > for BCP > purposes or is it too loosely coupled to take this risk. What is BCP? Federation is designed to be used over WAN links (of course, you can use it over LAN if you want, too). Design goals are listed on?http://www.rabbitmq.com/federation.html.? -- MK Software Engineer, Pivotal/RabbitMQ From mklishin at gopivotal.com Wed Apr 23 10:57:37 2014 From: mklishin at gopivotal.com (Michael Klishin) Date: Wed, 23 Apr 2014 13:57:37 +0400 Subject: [rabbitmq-discuss] Preventing DoS in a multi-tenant Rabbit deployment In-Reply-To: <4f91d062-8c04-44e6-90e0-f852e11e79db@googlegroups.com> References: <4f91d062-8c04-44e6-90e0-f852e11e79db@googlegroups.com> Message-ID: On 23 April 2014 at 11:22:29, Tomasz Janczuk (tjanczuk33 at gmail.com) wrote: > > What is the best mechanism to prevent authenticated DoS attacks > in a multi-tenant Rabbit deployment? Is there one? > > By authenticated DoS attack I mean an attack in which an authenticated > tenant causes a level of resource consumption in a Rabbit deployment > that prevents other tenants from using the service or severely > degrades the performance. Use HTTP API to monitor connections, close all but 1 (or however many you want) from a username that has too many of them. I?m pretty sure that?s how hosted RabbitMQ services (e.g. CloudAMQP) do it.? -- MK Software Engineer, Pivotal/RabbitMQ From 0x6e6562 at gmail.com Wed Apr 23 13:55:49 2014 From: 0x6e6562 at gmail.com (Ben Hood) Date: Wed, 23 Apr 2014 13:55:49 +0100 Subject: [rabbitmq-discuss] Prefetch fields in management UI In-Reply-To: <535774F2.4070200@rabbitmq.com> References: <535774F2.4070200@rabbitmq.com> Message-ID: On Wed, Apr 23, 2014 at 9:08 AM, Simon MacMullen wrote: > Note that the semantics of qos have changed in 3.3.0: > > http://www.rabbitmq.com/consumer-prefetch.html > > Per-consumer prefetch will be fixed at the moment the consumer is created. > Per-channel prefetch can be changed after the fact, as before. But it was > never a great idea to do that since if you set qos after consuming then the > broker can deliver as many messages as it wants in between the two commands, > which is unlikely to be what you want. Yep, that all makes sense and the change in the scope of qos in 3.3 is really good idea. So the take home is that you could have an app with a bug in it (issuing qos after the consume), then a pre-3.3 broker would mask this issue for you. So app only then becomes really broken after upgrading to 3.3. But the fix is simple. Cheers, Ben From Rouzbeh.Farahmand at Desire2Learn.com Wed Apr 23 14:40:17 2014 From: Rouzbeh.Farahmand at Desire2Learn.com (Rouzbeh Farahmand) Date: Wed, 23 Apr 2014 13:40:17 +0000 Subject: [rabbitmq-discuss] The meaning of latency in the PerfTest module In-Reply-To: <53576DE8.7000000@rabbitmq.com> References: <6B4CB2165485294B8C9571963FDDBD4822CE83AA@KTVCAEX-MB-01.desire2learn.d2l> <53576DE8.7000000@rabbitmq.com> Message-ID: <6B4CB2165485294B8C9571963FDDBD4822CE851A@KTVCAEX-MB-01.desire2learn.d2l> Thanks, very much. We will update our definition of latency in our benchmarking accordingly. Cheers, Rouzbeh -----Original Message----- From: Simon MacMullen [mailto:simon at rabbitmq.com] Sent: April-23-14 3:38 AM To: Discussions about RabbitMQ; Rouzbeh Farahmand Subject: Re: [rabbitmq-discuss] The meaning of latency in the PerfTest module On 21/04/2014 22:09, Rouzbeh Farahmand wrote: > Do we mean the time that it takes to publish a message to a queue and > consume the same message from the same queue? . ( i.e. latency = > wall-time of sending the message by publisher - wall-time of receiving > by the consumer) This one. Note that you get latency numbers out of PerfTest when you have a producer and a consumer, but you don't have to use publisher confirms. Cheers, Simon From Anita.Yip at RealPage.com Wed Apr 23 17:43:30 2014 From: Anita.Yip at RealPage.com (Anita Yip) Date: Wed, 23 Apr 2014 16:43:30 +0000 Subject: [rabbitmq-discuss] RabbitMQ HA set up not working as expected In-Reply-To: References: Message-ID: To achieve High Availability, I do need to mirror all the queues, right? Or is there any other way. My basic requirement is that when one Rabbit server goes down, the apps connecting to it should not get affected and the messages should not be lost. Anita Yip RealPage, Inc. 360 Third Street Suite 450 San Francisco, CA 94107 Office: 415-395-2109 Anita.yip at realpage.com www.realpage.com Connect with RealPage: [Description: Description: Description: cid:image003.jpg at 01CE5184.4C0A9D20][Description: Description: Description: cid:image005.gif at 01CE5184.4C0A9D20] [Description: Description: Description: cid:image006.gif at 01CE5184.4C0A9D20] [Description: Description: Description: cid:image007.png at 01CE5184.4C0A9D20] [Description: Description: Description: cid:image008.jpg at 01CE5184.4C0A9D20] [Description: Description: Description: cid:image009.jpg at 01CE5184.4C0A9D20] From: Jerry Kuch > Reply-To: Discussions about RabbitMQ > Date: Tuesday, April 22, 2014 4:08 PM To: Discussions about RabbitMQ > Subject: Re: [rabbitmq-discuss] RabbitMQ HA set up not working as expected On Tue, Apr 22, 2014 at 4:03 PM, Anita Yip > wrote: Ignore the below email, as it turned out the set up is not correct. On a similar note, can you please let me know if high availability set up is possible at a cluster level? In old Rabbits HA/mirroring is a property of the queue, granted to it when the queue is declared. In newer Rabbits, HA is defined by a policy, which applies to a pattern specifying the names of the objects the policy is supposed to modify. For details see here: http://www.rabbitmq.com/ha.html Are you sure you want to mirror absolutely all of the queues? ________________________________ This message is intended only for the use of the individual(s) or entity to which it is addressed and may contain information that is privileged, confidential, and/or proprietary to RealPage and its affiliated companies. If the reader of this message is not the intended recipient, you are hereby notified that any dissemination, distribution, forwarding or copying of this communication is prohibited without the express permission of the sender. If you have received this communication in error, please notify the sender immediately and delete the original message. -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: 4AC7ACDF-CDB3-4930-A66C-481C9E292AE1[115].png Type: image/png Size: 1722 bytes Desc: 4AC7ACDF-CDB3-4930-A66C-481C9E292AE1[115].png URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: 00859B61-FDB7-4823-A9ED-23D3130E18AC[115].png Type: image/png Size: 1360 bytes Desc: 00859B61-FDB7-4823-A9ED-23D3130E18AC[115].png URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: 91E364DD-5CE6-493B-8292-4DD276F2B5D4[115].png Type: image/png Size: 1254 bytes Desc: 91E364DD-5CE6-493B-8292-4DD276F2B5D4[115].png URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: 72716D1A-8A3D-46B6-A89B-999285564183[115].png Type: image/png Size: 4781 bytes Desc: 72716D1A-8A3D-46B6-A89B-999285564183[115].png URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: 2AC40763-6D00-4A59-9A0C-A5525B44EC57[115].png Type: image/png Size: 1875 bytes Desc: 2AC40763-6D00-4A59-9A0C-A5525B44EC57[115].png URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: 4CF393B6-2ADA-418D-AC2B-5D67FA784399[115].png Type: image/png Size: 1885 bytes Desc: 4CF393B6-2ADA-418D-AC2B-5D67FA784399[115].png URL: From jkuch at gopivotal.com Wed Apr 23 17:50:26 2014 From: jkuch at gopivotal.com (Jerry Kuch) Date: Wed, 23 Apr 2014 09:50:26 -0700 Subject: [rabbitmq-discuss] RabbitMQ HA set up not working as expected In-Reply-To: References: Message-ID: Please consult the documentation on HA via mirrored queues at: http://www.rabbitmq.com/ha.html One wants to be careful about how one uses the term "high availability." Normally a queue will reside on only one node of a cluster, and its availability will be subject to the health of that node. HA via mirrored queues allows you to specify that a queue or queues will be replicated on more than one node of a cluster, thereby guarding against failure of what would otherwise be a single node holding the queue. So do you want to mirror all queues? Maybe, but it depends on your actual requirements. HA via mirrored queues, as the above link explains, isn't a switch one throws once on the cluster and walks away from, ticking off a box on one's checklist marked "High Availability: Done." If you want to guard against message loss in the presence of failures, there are other things your applications will have to think about, particularly around the transfer of responsibility for messages as they pass in and out of the broker, whether by using a proper acknowledgement discipline with consumers, using publisher confirms on the producer side, or using transactions, and handling the various failure notifications that can happen when a connection or cluster node go down, potentially retransmitting on demand, being prepared to see a duplicated message under certain failure syndromes, etc. These issues are all discussed in the above link. Best regards, Jerry On Wed, Apr 23, 2014 at 9:43 AM, Anita Yip wrote: > To achieve High Availability, I do need to mirror all the queues, > right? Or is there any other way. My basic requirement is that when one > Rabbit server goes down, the apps connecting to it should not get affected > and the messages should not be lost. > > *Anita Yip* > RealPage, Inc. > 360 Third Street Suite 450 > San Francisco, CA 94107 > > Office: 415-395-2109 > Anita.yip at realpage.com > www.realpage.com > > *Connect with RealPage:* > > [image: Description: Description: Description: > cid:image003.jpg at 01CE5184.4C0A9D20] [image: > Description: Description: Description: cid:image005.gif at 01CE5184.4C0A9D20] > [image: Description: Description: Description: > cid:image006.gif at 01CE5184.4C0A9D20] [image: > Description: Description: Description: cid:image007.png at 01CE5184.4C0A9D20] > [image: Description: Description: Description: > cid:image008.jpg at 01CE5184.4C0A9D20] > [image: Description: Description: Description: > cid:image009.jpg at 01CE5184.4C0A9D20] > > > From: Jerry Kuch > > Reply-To: Discussions about RabbitMQ > Date: Tuesday, April 22, 2014 4:08 PM > > To: Discussions about RabbitMQ > Subject: Re: [rabbitmq-discuss] RabbitMQ HA set up not working as expected > > On Tue, Apr 22, 2014 at 4:03 PM, Anita Yip wrote: > >> Ignore the below email, as it turned out the set up is not correct. On >> a similar note, can you please let me know if high availability set up is >> possible at a cluster level? >> > > In old Rabbits HA/mirroring is a property of the queue, granted to it > when the queue is declared. > > In newer Rabbits, HA is defined by a policy, which applies to a pattern > specifying the names of the objects the policy is supposed to modify. For > details see here: http://www.rabbitmq.com/ha.html > > Are you sure you want to mirror absolutely all of the queues? > > > ------------------------------ > > This message is intended only for the use of the individual(s) or entity > to which it is addressed and may contain information that is privileged, > confidential, and/or proprietary to RealPage and its affiliated companies. > If the reader of this message is not the intended recipient, you are hereby > notified that any dissemination, distribution, forwarding or copying of > this communication is prohibited without the express permission of the > sender. If you have received this communication in error, please notify the > sender immediately and delete the original message. > > _______________________________________________ > rabbitmq-discuss mailing list > rabbitmq-discuss at lists.rabbitmq.com > https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss > > -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: 4AC7ACDF-CDB3-4930-A66C-481C9E292AE1[115].png Type: image/png Size: 1722 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: 4CF393B6-2ADA-418D-AC2B-5D67FA784399[115].png Type: image/png Size: 1885 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: 2AC40763-6D00-4A59-9A0C-A5525B44EC57[115].png Type: image/png Size: 1875 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: 91E364DD-5CE6-493B-8292-4DD276F2B5D4[115].png Type: image/png Size: 1254 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: 72716D1A-8A3D-46B6-A89B-999285564183[115].png Type: image/png Size: 4781 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: 00859B61-FDB7-4823-A9ED-23D3130E18AC[115].png Type: image/png Size: 1360 bytes Desc: not available URL: From sp345s at att.com Wed Apr 23 20:18:38 2014 From: sp345s at att.com (PATAR, SAGAR) Date: Wed, 23 Apr 2014 19:18:38 +0000 Subject: [rabbitmq-discuss] NullPointerException Message-ID: <59BF56992750434F89F8EBE141E64B1D018769B2@GAALPA1MSGUSR9C.ITServices.sbc.com> We have a multithread java program running in JBOSS .. where each thread sends a message to the RABBIT MQ Exchange .. We are using the Spring config file to make the connections / channels etc ... Currently we have requestedChannelMax=1 .. Occasionally we see the below exception ... The exception is mis leading and not sure if its because eof the channel availability ... 2014-04-23 14:28:07,279 ERROR [DeviceOIDUpdateMBusTask] - com.att.dlife.dlcore.mbus.client.common.DLCoreMessageBusException: org.springframework.amqp.UncategorizedAmqpException: java.lang.NullPointerException at org.springframework.amqp.rabbit.support.RabbitExceptionTranslator.convertRabbitAccessException(RabbitExceptionTranslator.java:60) at org.springframework.amqp.rabbit.connection.RabbitAccessor.convertRabbitAccessException(RabbitAccessor.java:108) at org.springframework.amqp.rabbit.core.RabbitTemplate.execute(RabbitTemplate.java:632) at org.springframework.amqp.rabbit.core.RabbitTemplate.send(RabbitTemplate.java:349) at org.springframework.amqp.rabbit.core.RabbitTemplate.send(RabbitTemplate.java:343) at com.att.dlife.dlcore.mbus.client.publisher.AmqpMessageSenderImpl.send(AmqpMessageSenderImpl.java:71) at com.xanboo.core.mbus.DeviceOIDUpdateMBusTask.run(DeviceOIDUpdateMBusTask.java:156) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) Caused by: java.lang.NullPointerException at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.springframework.amqp.rabbit.connection.CachingConnectionFactory$CachedChannelInvocationHandler.invoke(CachingConnectionFactory.java:348) at $Proxy207.basicPublish(Unknown Source) at org.springframework.amqp.rabbit.core.RabbitTemplate.doSend(RabbitTemplate.java:675) at org.springframework.amqp.rabbit.core.RabbitTemplate$1.doInRabbit(RabbitTemplate.java:351) at org.springframework.amqp.rabbit.core.RabbitTemplate.execute(RabbitTemplate.java:627) ... 7 more at com.att.dlife.dlcore.mbus.client.publisher.AmqpMessageSenderImpl.send(AmqpMessageSenderImpl.java:94) at com.xanboo.core.mbus.DeviceOIDUpdateMBusTask.run(DeviceOIDUpdateMBusTask.java:156) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) -------------- next part -------------- An HTML attachment was scrubbed... URL: From grussell at gopivotal.com Wed Apr 23 20:35:25 2014 From: grussell at gopivotal.com (Gary Russell) Date: Wed, 23 Apr 2014 15:35:25 -0400 Subject: [rabbitmq-discuss] NullPointerException In-Reply-To: <59BF56992750434F89F8EBE141E64B1D018769B2@GAALPA1MSGUSR9C.ITServices.sbc.com> References: <59BF56992750434F89F8EBE141E64B1D018769B2@GAALPA1MSGUSR9C.ITServices.sbc.com> Message-ID: Please open a JIRA issue here: https://jira.spring.io/browse/AMQP and we'll take a look. Please be sure to indicate which version of Spring AMQP you are using. Thanks Gary On Wed, Apr 23, 2014 at 3:18 PM, PATAR, SAGAR wrote: > We have a multithread java program running in JBOSS .. where each thread > sends a message to the RABBIT MQ Exchange .. > > We are using the Spring config file to make the connections / channels > etc ... > > Currently we have requestedChannelMax=1 .. Occasionally we see the below > exception ... > > The exception is mis leading and not sure if its because eof the channel > availability ... > > > > 2014-04-23 14:28:07,279 ERROR [DeviceOIDUpdateMBusTask] - > com.att.dlife.dlcore.mbus.client.common.DLCoreMessageBusException: > org.springframework.amqp.UncategorizedAmqpException: > java.lang.NullPointerException > > at > org.springframework.amqp.rabbit.support.RabbitExceptionTranslator.convertRabbitAccessException(RabbitExceptionTranslator.java:60) > > at > org.springframework.amqp.rabbit.connection.RabbitAccessor.convertRabbitAccessException(RabbitAccessor.java:108) > > at > org.springframework.amqp.rabbit.core.RabbitTemplate.execute(RabbitTemplate.java:632) > > at > org.springframework.amqp.rabbit.core.RabbitTemplate.send(RabbitTemplate.java:349) > > at > org.springframework.amqp.rabbit.core.RabbitTemplate.send(RabbitTemplate.java:343) > > at > com.att.dlife.dlcore.mbus.client.publisher.AmqpMessageSenderImpl.send(AmqpMessageSenderImpl.java:71) > > at > com.xanboo.core.mbus.DeviceOIDUpdateMBusTask.run(DeviceOIDUpdateMBusTask.java:156) > > at > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > > at java.lang.Thread.run(Thread.java:662) > > Caused by: java.lang.NullPointerException > > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) > > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) > > at java.lang.reflect.Method.invoke(Method.java:597) > > at > org.springframework.amqp.rabbit.connection.CachingConnectionFactory$CachedChannelInvocationHandler.invoke(CachingConnectionFactory.java:348) > > at $Proxy207.basicPublish(Unknown Source) > > at > org.springframework.amqp.rabbit.core.RabbitTemplate.doSend(RabbitTemplate.java:675) > > at > org.springframework.amqp.rabbit.core.RabbitTemplate$1.doInRabbit(RabbitTemplate.java:351) > > at > org.springframework.amqp.rabbit.core.RabbitTemplate.execute(RabbitTemplate.java:627) > > ... 7 more > > > > at > com.att.dlife.dlcore.mbus.client.publisher.AmqpMessageSenderImpl.send(AmqpMessageSenderImpl.java:94) > > at > com.xanboo.core.mbus.DeviceOIDUpdateMBusTask.run(DeviceOIDUpdateMBusTask.java:156) > > at > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > > at java.lang.Thread.run(Thread.java:662) > > _______________________________________________ > rabbitmq-discuss mailing list > rabbitmq-discuss at lists.rabbitmq.com > https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mklishin at gopivotal.com Wed Apr 23 20:38:15 2014 From: mklishin at gopivotal.com (Michael Klishin) Date: Wed, 23 Apr 2014 23:38:15 +0400 Subject: [rabbitmq-discuss] NullPointerException In-Reply-To: References: <59BF56992750434F89F8EBE141E64B1D018769B2@GAALPA1MSGUSR9C.ITServices.sbc.com> Message-ID: On 23 April 2014 at 23:36:36, Gary Russell (grussell at gopivotal.com) wrote: > > Please open a JIRA issue here: https://jira.spring.io/browse/AMQP > and we'll take a look. > > Please be sure to indicate which version of Spring AMQP you are > using. And also post a code snippet that briefly describes what you?re trying to do. NPEs in multithreaded code can stem from lazy initialization in application code, for example.? -- MK Software Engineer, Pivotal/RabbitMQ From lepolac at gmail.com Wed Apr 23 21:57:50 2014 From: lepolac at gmail.com (lepolac) Date: Wed, 23 Apr 2014 13:57:50 -0700 (PDT) Subject: [rabbitmq-discuss] Federated exchange and sync delivery In-Reply-To: References: <1398193700805-35010.post@n5.nabble.com> Message-ID: <1398286670878-35036.post@n5.nabble.com> Hi, Thanks for the reply, sorry, by BCP I meant disaster recovery, as in having a cluster in another datacenter with a state/msg content that is as close to the upstream cluster as possible. If a publisher pushes a message on the upstream, does it get the confirm after the downstream has got/confirmed the message ? -- View this message in context: http://rabbitmq.1065348.n5.nabble.com/Federated-exchange-and-sync-delivery-tp35010p35036.html Sent from the RabbitMQ mailing list archive at Nabble.com. From mklishin at gopivotal.com Wed Apr 23 22:03:40 2014 From: mklishin at gopivotal.com (Michael Klishin) Date: Thu, 24 Apr 2014 01:03:40 +0400 Subject: [rabbitmq-discuss] Federated exchange and sync delivery In-Reply-To: <1398286670878-35036.post@n5.nabble.com> References: <1398193700805-35010.post@n5.nabble.com> <1398286670878-35036.post@n5.nabble.com> Message-ID: On 24 April 2014 at 00:59:03, lepolac (lepolac at gmail.com) wrote: > > If a publisher pushes a message on the upstream, does it get the > confirm > after the downstream has got/confirmed the message ? Exchange federation uses publisher confirms.? -- MK Software Engineer, Pivotal/RabbitMQ From lepolac at gmail.com Wed Apr 23 22:14:24 2014 From: lepolac at gmail.com (lepolac) Date: Wed, 23 Apr 2014 14:14:24 -0700 (PDT) Subject: [rabbitmq-discuss] Partition handling configuration In-Reply-To: <53577409.4020305@rabbitmq.com> References: <1398193415066-35009.post@n5.nabble.com> <53577409.4020305@rabbitmq.com> Message-ID: <1398287664636-35038.post@n5.nabble.com> Thanks :) -- View this message in context: http://rabbitmq.1065348.n5.nabble.com/Partition-handling-configuration-tp35009p35038.html Sent from the RabbitMQ mailing list archive at Nabble.com. From lepolac at gmail.com Wed Apr 23 23:35:31 2014 From: lepolac at gmail.com (lepolac) Date: Wed, 23 Apr 2014 15:35:31 -0700 (PDT) Subject: [rabbitmq-discuss] Federation link and partition handling Message-ID: <1398292531569-35039.post@n5.nabble.com> Hi, When there's a partition on a cluster that's configured in ignore or autoheal, where both sides will stay active, a second federation link seems to be created. Are both links active ? Will the downstream broker get messages from both partitions ? How will they be merged back ? Thanks. -- View this message in context: http://rabbitmq.1065348.n5.nabble.com/Federation-link-and-partition-handling-tp35039.html Sent from the RabbitMQ mailing list archive at Nabble.com. From lepolac at gmail.com Wed Apr 23 23:38:11 2014 From: lepolac at gmail.com (lepolac) Date: Wed, 23 Apr 2014 15:38:11 -0700 (PDT) Subject: [rabbitmq-discuss] Federated exchange and sync delivery In-Reply-To: References: <1398193700805-35010.post@n5.nabble.com> <1398286670878-35036.post@n5.nabble.com> Message-ID: <1398292691562-35040.post@n5.nabble.com> So is the following flow accurate ? 1 2 publisher --------> upstream ------> downstream <--confirm--- <--confirm-- 4 3 Or would the upstream send the confirm back before the message reaches the downstream ? -- View this message in context: http://rabbitmq.1065348.n5.nabble.com/Federated-exchange-and-sync-delivery-tp35010p35040.html Sent from the RabbitMQ mailing list archive at Nabble.com. From Anita.Yip at RealPage.com Thu Apr 24 01:22:02 2014 From: Anita.Yip at RealPage.com (Anita Yip) Date: Thu, 24 Apr 2014 00:22:02 +0000 Subject: [rabbitmq-discuss] RabbitMQ HA set up not working as expected In-Reply-To: References: Message-ID: Whatever you mention in the last para, we are already doing it. My question is mainly about the HA set up of the clusters. Again, HA set up within a cluster is what we already have and I could find that documentation on the net. But is it possible to set up HA across clusters? From: Jerry Kuch > Reply-To: Discussions about RabbitMQ > Date: Wednesday, April 23, 2014 9:50 AM To: Discussions about RabbitMQ > Subject: Re: [rabbitmq-discuss] RabbitMQ HA set up not working as expected Please consult the documentation on HA via mirrored queues at: http://www.rabbitmq.com/ha.html One wants to be careful about how one uses the term "high availability." Normally a queue will reside on only one node of a cluster, and its availability will be subject to the health of that node. HA via mirrored queues allows you to specify that a queue or queues will be replicated on more than one node of a cluster, thereby guarding against failure of what would otherwise be a single node holding the queue. So do you want to mirror all queues? Maybe, but it depends on your actual requirements. HA via mirrored queues, as the above link explains, isn't a switch one throws once on the cluster and walks away from, ticking off a box on one's checklist marked "High Availability: Done." If you want to guard against message loss in the presence of failures, there are other things your applications will have to think about, particularly around the transfer of responsibility for messages as they pass in and out of the broker, whether by using a proper acknowledgement discipline with consumers, using publisher confirms on the producer side, or using transactions, and handling the various failure notifications that can happen when a connection or cluster node go down, potentially retransmitting on demand, being prepared to see a duplicated message under certain failure syndromes, etc. These issues are all discussed in the above link. Best regards, Jerry On Wed, Apr 23, 2014 at 9:43 AM, Anita Yip > wrote: To achieve High Availability, I do need to mirror all the queues, right? Or is there any other way. My basic requirement is that when one Rabbit server goes down, the apps connecting to it should not get affected and the messages should not be lost. Anita Yip RealPage, Inc. 360 Third Street Suite 450 San Francisco, CA 94107 Office: 415-395-2109 Anita.yip at realpage.com www.realpage.com Connect with RealPage: [Description: Description: Description: cid:image003.jpg at 01CE5184.4C0A9D20][Description: Description: Description: cid:image005.gif at 01CE5184.4C0A9D20] [Description: Description: Description: cid:image006.gif at 01CE5184.4C0A9D20] [Description: Description: Description: cid:image007.png at 01CE5184.4C0A9D20] [Description: Description: Description: cid:image008.jpg at 01CE5184.4C0A9D20] [Description: Description: Description: cid:image009.jpg at 01CE5184.4C0A9D20] From: Jerry Kuch > Reply-To: Discussions about RabbitMQ > Date: Tuesday, April 22, 2014 4:08 PM To: Discussions about RabbitMQ > Subject: Re: [rabbitmq-discuss] RabbitMQ HA set up not working as expected On Tue, Apr 22, 2014 at 4:03 PM, Anita Yip > wrote: Ignore the below email, as it turned out the set up is not correct. On a similar note, can you please let me know if high availability set up is possible at a cluster level? In old Rabbits HA/mirroring is a property of the queue, granted to it when the queue is declared. In newer Rabbits, HA is defined by a policy, which applies to a pattern specifying the names of the objects the policy is supposed to modify. For details see here: http://www.rabbitmq.com/ha.html Are you sure you want to mirror absolutely all of the queues? ________________________________ This message is intended only for the use of the individual(s) or entity to which it is addressed and may contain information that is privileged, confidential, and/or proprietary to RealPage and its affiliated companies. If the reader of this message is not the intended recipient, you are hereby notified that any dissemination, distribution, forwarding or copying of this communication is prohibited without the express permission of the sender. If you have received this communication in error, please notify the sender immediately and delete the original message. _______________________________________________ rabbitmq-discuss mailing list rabbitmq-discuss at lists.rabbitmq.com https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss ________________________________ This message is intended only for the use of the individual(s) or entity to which it is addressed and may contain information that is privileged, confidential, and/or proprietary to RealPage and its affiliated companies. If the reader of this message is not the intended recipient, you are hereby notified that any dissemination, distribution, forwarding or copying of this communication is prohibited without the express permission of the sender. If you have received this communication in error, please notify the sender immediately and delete the original message. -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: 4AC7ACDF-CDB3-4930-A66C-481C9E292AE1[115].png Type: image/png Size: 1722 bytes Desc: 4AC7ACDF-CDB3-4930-A66C-481C9E292AE1[115].png URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: 4CF393B6-2ADA-418D-AC2B-5D67FA784399[115].png Type: image/png Size: 1885 bytes Desc: 4CF393B6-2ADA-418D-AC2B-5D67FA784399[115].png URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: 2AC40763-6D00-4A59-9A0C-A5525B44EC57[115].png Type: image/png Size: 1875 bytes Desc: 2AC40763-6D00-4A59-9A0C-A5525B44EC57[115].png URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: 91E364DD-5CE6-493B-8292-4DD276F2B5D4[115].png Type: image/png Size: 1254 bytes Desc: 91E364DD-5CE6-493B-8292-4DD276F2B5D4[115].png URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: 72716D1A-8A3D-46B6-A89B-999285564183[115].png Type: image/png Size: 4781 bytes Desc: 72716D1A-8A3D-46B6-A89B-999285564183[115].png URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: 00859B61-FDB7-4823-A9ED-23D3130E18AC[115].png Type: image/png Size: 1360 bytes Desc: 00859B61-FDB7-4823-A9ED-23D3130E18AC[115].png URL: From jkuch at gopivotal.com Thu Apr 24 03:08:49 2014 From: jkuch at gopivotal.com (Jerry Kuch) Date: Wed, 23 Apr 2014 19:08:49 -0700 Subject: [rabbitmq-discuss] RabbitMQ HA set up not working as expected In-Reply-To: References: Message-ID: No. Mirrored queues rely on Erlang distribution for their operation and thus only work within a cluster. On Wed, Apr 23, 2014 at 5:22 PM, Anita Yip wrote: > Whatever you mention in the last para, we are already doing it. My > question is mainly about the HA set up of the clusters. Again, HA set up > within a cluster is what we already have and I could find that > documentation on the net. But is it possible to set up HA across clusters? > > > From: Jerry Kuch > Reply-To: Discussions about RabbitMQ > Date: Wednesday, April 23, 2014 9:50 AM > > To: Discussions about RabbitMQ > Subject: Re: [rabbitmq-discuss] RabbitMQ HA set up not working as expected > > Please consult the documentation on HA via mirrored queues at: > > http://www.rabbitmq.com/ha.html > > One wants to be careful about how one uses the term "high availability." > Normally a queue will reside on only one node of a cluster, and its > availability will be subject to the health of that node. HA via mirrored > queues allows you to specify that a queue or queues will be replicated on > more than one node of a cluster, thereby guarding against failure of what > would otherwise be a single node holding the queue. > > So do you want to mirror all queues? Maybe, but it depends on your > actual requirements. > HA via mirrored queues, as the above link explains, isn't a switch one > throws once on the cluster and walks away from, ticking off a box on one's > checklist marked "High Availability: Done." > > If you want to guard against message loss in the presence of failures, > there are other things your applications will have to think about, > particularly around the transfer of responsibility for messages as they > pass in and out of the broker, whether by using a proper acknowledgement > discipline with consumers, using publisher confirms on the producer side, > or using transactions, and handling the various failure notifications that > can happen when a connection or cluster node go down, potentially > retransmitting on demand, being prepared to see a duplicated message under > certain failure syndromes, etc. These issues are all discussed in the > above link. > > Best regards, > Jerry > > > > > On Wed, Apr 23, 2014 at 9:43 AM, Anita Yip wrote: > >> To achieve High Availability, I do need to mirror all the queues, >> right? Or is there any other way. My basic requirement is that when one >> Rabbit server goes down, the apps connecting to it should not get affected >> and the messages should not be lost. >> >> *Anita Yip* >> RealPage, Inc. >> 360 Third Street Suite 450 >> San Francisco, CA 94107 >> >> Office: 415-395-2109 >> Anita.yip at realpage.com >> www.realpage.com >> >> *Connect with RealPage:* >> >> [image: Description: Description: Description: >> cid:image003.jpg at 01CE5184.4C0A9D20] [image: >> Description: Description: Description: cid:image005.gif at 01CE5184.4C0A9D20] >> [image: Description: Description: Description: >> cid:image006.gif at 01CE5184.4C0A9D20] [image: >> Description: Description: Description: cid:image007.png at 01CE5184.4C0A9D20] >> [image: Description: Description: Description: >> cid:image008.jpg at 01CE5184.4C0A9D20] >> [image: Description: Description: Description: >> cid:image009.jpg at 01CE5184.4C0A9D20] >> >> >> From: Jerry Kuch >> >> Reply-To: Discussions about RabbitMQ > > >> Date: Tuesday, April 22, 2014 4:08 PM >> >> To: Discussions about RabbitMQ >> Subject: Re: [rabbitmq-discuss] RabbitMQ HA set up not working as >> expected >> >> On Tue, Apr 22, 2014 at 4:03 PM, Anita Yip wrote: >> >>> Ignore the below email, as it turned out the set up is not correct. >>> On a similar note, can you please let me know if high availability set up >>> is possible at a cluster level? >>> >> >> In old Rabbits HA/mirroring is a property of the queue, granted to it >> when the queue is declared. >> >> In newer Rabbits, HA is defined by a policy, which applies to a pattern >> specifying the names of the objects the policy is supposed to modify. For >> details see here: http://www.rabbitmq.com/ha.html >> >> Are you sure you want to mirror absolutely all of the queues? >> >> >> ------------------------------ >> >> This message is intended only for the use of the individual(s) or entity >> to which it is addressed and may contain information that is privileged, >> confidential, and/or proprietary to RealPage and its affiliated companies. >> If the reader of this message is not the intended recipient, you are hereby >> notified that any dissemination, distribution, forwarding or copying of >> this communication is prohibited without the express permission of the >> sender. If you have received this communication in error, please notify the >> sender immediately and delete the original message. >> >> _______________________________________________ >> rabbitmq-discuss mailing list >> rabbitmq-discuss at lists.rabbitmq.com >> https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss >> >> > > ------------------------------ > > This message is intended only for the use of the individual(s) or entity > to which it is addressed and may contain information that is privileged, > confidential, and/or proprietary to RealPage and its affiliated companies. > If the reader of this message is not the intended recipient, you are hereby > notified that any dissemination, distribution, forwarding or copying of > this communication is prohibited without the express permission of the > sender. If you have received this communication in error, please notify the > sender immediately and delete the original message. > > _______________________________________________ > rabbitmq-discuss mailing list > rabbitmq-discuss at lists.rabbitmq.com > https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss > > -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: 4CF393B6-2ADA-418D-AC2B-5D67FA784399[115].png Type: image/png Size: 1885 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: 00859B61-FDB7-4823-A9ED-23D3130E18AC[115].png Type: image/png Size: 1360 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: 72716D1A-8A3D-46B6-A89B-999285564183[115].png Type: image/png Size: 4781 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: 91E364DD-5CE6-493B-8292-4DD276F2B5D4[115].png Type: image/png Size: 1254 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: 4AC7ACDF-CDB3-4930-A66C-481C9E292AE1[115].png Type: image/png Size: 1722 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: 2AC40763-6D00-4A59-9A0C-A5525B44EC57[115].png Type: image/png Size: 1875 bytes Desc: not available URL: From srikanthtns at gmail.com Thu Apr 24 05:10:00 2014 From: srikanthtns at gmail.com (srikanth tns) Date: Wed, 23 Apr 2014 21:10:00 -0700 Subject: [rabbitmq-discuss] RabbitMQ HA Setup Message-ID: We are using 3 rabbitmq nodes setup in the cluster . Our client application is mcollective. We have made sure the queues are load balanced based on the number of connections for each node in the cluster. we have setup HA all mode for queue mirroring in the cluster. In the failover scenario, if one of the nodes in the cluster goes down , the queues dedicated for the failed node are getting failed over to the next master in the cluster. Now when the failed node is back into service , the queues are not assigned back to its owner. They are still connected to master. Is there any way we can have the rabbitmq to send back the queues to its owner? But when I restart the client application , the queues are back to the owner , but in failover mode the queues stick to the new master and never come back to old master. Thanks Srikanth -------------- next part -------------- An HTML attachment was scrubbed... URL: From mklishin at gopivotal.com Thu Apr 24 06:40:00 2014 From: mklishin at gopivotal.com (Michael Klishin) Date: Thu, 24 Apr 2014 09:40:00 +0400 Subject: [rabbitmq-discuss] RabbitMQ HA set up not working as expected In-Reply-To: References: Message-ID: On 24 April 2014 at 04:23:21, Anita Yip (anita.yip at realpage.com) wrote: > > But is it possible to set up HA across clusters? You can create a standby cluster with federated exchanges. See http://rabbitmq.com/federation.html? -- MK Software Engineer, Pivotal/RabbitMQ From mklishin at gopivotal.com Thu Apr 24 06:42:36 2014 From: mklishin at gopivotal.com (Michael Klishin) Date: Thu, 24 Apr 2014 09:42:36 +0400 Subject: [rabbitmq-discuss] RabbitMQ HA Setup In-Reply-To: References: Message-ID: On 24 April 2014 at 08:12:17, srikanth tns (srikanthtns at gmail.com) wrote: > > Is there any way we can have the rabbitmq to send back the queues > to its owner? > But when I restart the client application , the queues are back > to the owner , but in failover mode the queues stick to the new master > and never come back to old master. No. This would contradict the whole idea of HA. What if the node that used to host master of a queue is decommissioned and is never coming back? How should things work in that case?? -- MK Software Engineer, Pivotal/RabbitMQ From mklishin at gopivotal.com Thu Apr 24 08:11:50 2014 From: mklishin at gopivotal.com (Michael Klishin) Date: Thu, 24 Apr 2014 11:11:50 +0400 Subject: [rabbitmq-discuss] Federated exchange and sync delivery In-Reply-To: <1398292691562-35040.post@n5.nabble.com> References: <1398193700805-35010.post@n5.nabble.com> <1398286670878-35036.post@n5.nabble.com> <1398292691562-35040.post@n5.nabble.com> Message-ID: On 24 April 2014 at 02:39:21, lepolac (lepolac at gmail.com) wrote: > > Or would the upstream send the confirm back before the message > reaches the > downstream ? Yes. Publisher confirms between a publisher (client) and RabbitMQ node are unaware of federation links.? -- MK Software Engineer, Pivotal/RabbitMQ From bra at fsn.hu Thu Apr 24 09:22:13 2014 From: bra at fsn.hu (Nagy, Attila) Date: Thu, 24 Apr 2014 10:22:13 +0200 Subject: [rabbitmq-discuss] Exclusive consumers via STOMP Message-ID: <5358C9B5.9020305@fsn.hu> Hi, I need exclusive consumer functionality via STOMP in a federated (queue) broker environment (I know it can't be perfect in the case of a network split, but I could handle that in a lower level). I've just found this message: http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/2012-March/019183.html BTW, ActiveMQ's same functionality is described here: http://activemq.apache.org/exclusive-consumer.html and here for STOMP: https://activemq.apache.org/stomp.html (activemq.exclusive subscribe header) I've tried it with a durable queue. I could successfully subscribe to the queue from two machines and instances of the same consumer via STOMP and got messages (round robin) to each of them, regardless I've specified exclusive: True in the subscribe frame. I'm using 3.3.0. Wasn't this implemented? And if this would be implemented: would it be possible to implement it in a way that it works with queue federation? Like other brokers can't consume from the queue if the local broker has a consumer active. If that disconnects, and one of the remote brokers have a consumer, it would be able to consume. Thanks, From mirco.babin at gmail.com Thu Apr 24 09:38:26 2014 From: mirco.babin at gmail.com (Mirco Babin) Date: Thu, 24 Apr 2014 01:38:26 -0700 (PDT) Subject: [rabbitmq-discuss] rabbitmq-dotnet-client-3.3.0: 2 unit tests fail Message-ID: Dear reader, I want to report 2 unit test errors in the RabbitMQ dot Net client. I took the following steps: 1) Downloaded rabbitmq-dotnet-client-3.3.0.zip from the RabbitMQ website. 2) Extracted the zip file and renamed the Local.props.example to Local.props. 3) Modified Local.props to target the dot Net 4.5 framework v4.5 4) Opened the RabbitMQDotNetClient.sln in Visual Studio 2012 Professional 5) Confirmed the migration of the solution 6) Batch builded everything in the solution. 7) Downloaded and installed NUnit 2.6.3 8) Started NUnit gui (from the start menu), selected open project and selected rabbitmq-dotnet-client-3.3.0\projects\client\Unit\build\bin\unit-tests.dll 9) Finally hit the "run" button in NUnit gui Nunit reports 2 errors: 1) RabbitMQ.Client.Unit.TestConnectionBlocked.TestConnectionBlockedNotification: Unblock notification not received. 2) This error is probably due to locale issues. I have the Dutch Locale, and a decimal point is a "," in The Netherlands. (e.g. Dutch: 123.456,78 means integer part 123456 and decimal part 78) RabbitMQ.Client.Unit.TestStreamWireFormatting.TestSingleDecoding2: String lengths are both 5. Strings differ at index 1. Expected: "1.234" But was: "1,234" ------------^ Kind regards, Mirco Babin -------------- next part -------------- An HTML attachment was scrubbed... URL: From videlalvaro at gmail.com Thu Apr 24 10:43:52 2014 From: videlalvaro at gmail.com (Alvaro Videla) Date: Thu, 24 Apr 2014 11:43:52 +0200 Subject: [rabbitmq-discuss] Scaling with RabbitMQ In-Reply-To: References: <4A7DAC2A-2561-4D00-87B5-285C988741DC@gmail.com> Message-ID: Hi, On Tue, Apr 22, 2014 at 8:34 PM, Michael Klishin wrote: > Topic routing uses a trie internally which doesn?t get much slower > as the number of bindings increases or topic length grows. However, > since every queue gets a copy of the message, there will be some impact, > in particular with larger messages. Actually RabbitMQ keeps just one copy of each message, and then it keeps and index to know which message belongs to which queue. So if a message gets routed to 10 queues for example, it won't be stored 10 times. Regards, Alvaro From mklishin at gopivotal.com Thu Apr 24 11:16:55 2014 From: mklishin at gopivotal.com (Michael Klishin) Date: Thu, 24 Apr 2014 14:16:55 +0400 Subject: [rabbitmq-discuss] rabbitmq-dotnet-client-3.3.0: 2 unit tests fail In-Reply-To: References: Message-ID: ?On 24 April 2014 at 13:31:35, Mirco Babin (mirco.babin at gmail.com) wrote: > > Nunit reports 2 errors: > 1) > RabbitMQ.Client.Unit.TestConnectionBlocked.TestConnectionBlockedNotification: > Unblock notification not received. You need to run tests from the umbrella repo, with rabbitmq-server running from ../rabbitmq-server then the test above will trigger a resource alarm successfully > 2) This error is probably due to locale issues. I have the Dutch > Locale, and a decimal point is a "," in The Netherlands. (e.g. > Dutch: 123.456,78 means integer part 123456 and decimal part > 78) > RabbitMQ.Client.Unit.TestStreamWireFormatting.TestSingleDecoding2: > String lengths are both 5. Strings differ at index 1. > Expected: "1.234" > But was: "1,234" > ------------^ This is a locale-specific value. It would be nice to make it less environment-dependent but you can see that the value is basically what?s expected. -- MK Software Engineer, Pivotal/RabbitMQ From mklishin at gopivotal.com Thu Apr 24 11:20:38 2014 From: mklishin at gopivotal.com (Michael Klishin) Date: Thu, 24 Apr 2014 14:20:38 +0400 Subject: [rabbitmq-discuss] Exclusive consumers via STOMP In-Reply-To: <5358C9B5.9020305@fsn.hu> References: <5358C9B5.9020305@fsn.hu> Message-ID: On 24 April 2014 at 12:23:38, Nagy, Attila (bra at fsn.hu) wrote: > > I've tried it with a durable queue. I could successfully subscribe > to > the queue from two machines and instances of the same consumer > via STOMP > and got messages (round robin) to each of them, regardless I've > specified exclusive: True in the subscribe frame. > I'm using 3.3.0. > Wasn't this implemented? Possibly. In RabbitMQ consumer exclusivity means no more than 1 consumer can be added on a given queue. This includes federation links for federated queues. I need to run some tests to see how queue federation works in such case.? -- MK Software Engineer, Pivotal/RabbitMQ From loganathan.ms at gmail.com Thu Apr 24 12:32:13 2014 From: loganathan.ms at gmail.com (Loganathan Sellapa) Date: Thu, 24 Apr 2014 17:02:13 +0530 Subject: [rabbitmq-discuss] How do we connect with AMQP without username password Message-ID: Hi All, I am Loganathan, ROR developer. Currently I am using ruby-amqp gem for RabbitMq communication. I have enabled the SSL and I am able to establish the connection only with username & password with the certificates. But I would like to connect to the RabbitMq without user credentials. Is their any way to do it? One of Java application is connecting to the RabbitMq only with Keystore & trust keystore. *AMQP RUBY :* *settings = {:host => "10.146.14.115", :port => 5671, :vhost => "vchs", :user => "vchsadmin", :pass => "changeme",:ssl => {:cert_chain_file => "/home/logan/bks/ipam_service/client1.vchslabs.vmware.com/cert.pem ", :private_key_file => "/home/logan/bks/ipam_service/client1.vchslabs.vmware.com/key.pem "}}* *AMQP.start(settings) do |connection|* *puts "Connected, authenticated. TLS seems to work." connection.disconnect { puts "Now closing the connection..."; EventMachine.stop }* *end* *RABBITMQ CONFIGURATIONS:* [ {rabbit, [ {ssl_listeners, [5671]}, {ssl_options, [{cacertfile,"/opt/ssl/cacert.pem"}, {certfile,"/opt/ssl/cert.pem"}, {keyfile,"/opt/ssl/key.pem"}, {verify,verify_peer}, {fail_if_no_peer_cert,true}]}, {auth_mechanisms, ['PLAIN', 'AMQPLAIN', 'EXTERNAL']}, {ssl_cert_login_from, common_name} ]} ]. regards, Loganathan Mob: +91 7760780741 | +91 9944414388 Skype: loganathan.sellappa ViewMe -------------- next part -------------- An HTML attachment was scrubbed... URL: From mklishin at gopivotal.com Thu Apr 24 12:38:41 2014 From: mklishin at gopivotal.com (Michael Klishin) Date: Thu, 24 Apr 2014 15:38:41 +0400 Subject: [rabbitmq-discuss] How do we connect with AMQP without username password In-Reply-To: References: Message-ID: On 24 April 2014 at 15:35:05, Loganathan Sellapa (loganathan.ms at gmail.com) wrote: > > I have enabled the SSL and I am able to establish the connection > only with username & password with the certificates. But I would > like to connect to the RabbitMq without user credentials. Is > their any way to do it? amqp gem and Bunny support authentication mechanisms (e.g. EXTERNAL), :auth_mechanism is the option name. However, I don?t know if anyone has tried using them with the SSL auth mechanism plugin. Try it.? -- MK Software Engineer, Pivotal/RabbitMQ From bra at fsn.hu Thu Apr 24 12:53:52 2014 From: bra at fsn.hu (Nagy, Attila) Date: Thu, 24 Apr 2014 13:53:52 +0200 Subject: [rabbitmq-discuss] Exclusive consumers via STOMP In-Reply-To: References: <5358C9B5.9020305@fsn.hu> Message-ID: <5358FB50.4010006@fsn.hu> On 04/24/14 12:20, Michael Klishin wrote: > On 24 April 2014 at 12:23:38, Nagy, Attila (bra at fsn.hu) wrote: >>> I've tried it with a durable queue. I could successfully subscribe >> to >> the queue from two machines and instances of the same consumer >> via STOMP >> and got messages (round robin) to each of them, regardless I've >> specified exclusive: True in the subscribe frame. >> I'm using 3.3.0. >> Wasn't this implemented? > Possibly. In RabbitMQ consumer exclusivity means no more than 1 consumer can > be added on a given queue. This includes federation links for federated queues. Exactly what I need. > > I need to run some tests to see how queue federation works in such case. > How does queue federation consumes from the queue? I mean it's not just a normal consumer, because if nobody is on the other side, it won't consume. Or it just starts consuming if there are somebody? But there are even more conditions need to be true to get messages to it (described on the federation page), so this is not so simple. Also, if consumer A on broker A consumes from the queue, but broker B can't consume from broker A's queue, I assume consumer B on broker B will be able to consume from the queue on broker B, so this essentially doesn't work in a federated setup. Or is there something which disallows this? Thanks, From amalgjose at gmail.com Thu Apr 24 12:10:45 2014 From: amalgjose at gmail.com (AMAL G JOSE) Date: Thu, 24 Apr 2014 04:10:45 -0700 (PDT) Subject: [rabbitmq-discuss] I am getting an Error while trying to send messages to a queue. "ACCESS_REFUSED - Login was refused using authentication mechanism PLAIN. For details see the broker logfile." Message-ID: <152f9b10-e3d4-4cb8-bef2-68dce4a56833@googlegroups.com> I have configured rabbitMQ in my laptop and wrote a sender and receiver program. It works fine, if I set the *factory.setHost(prop.getProperty("hostname")); *property with* localhost.* But when I am giving the ipadress, I am getting an error * "ACCESS_REFUSED - Login was refused using authentication mechanism PLAIN. For details see the broker logfile."* I think this is due to some user permissions. Can anyone help me in solving this issue. -------------- next part -------------- An HTML attachment was scrubbed... URL: From mklishin at gopivotal.com Thu Apr 24 13:11:20 2014 From: mklishin at gopivotal.com (Michael Klishin) Date: Thu, 24 Apr 2014 16:11:20 +0400 Subject: [rabbitmq-discuss] I am getting an Error while trying to send messages to a queue. "ACCESS_REFUSED - Login was refused using authentication mechanism PLAIN. For details see the broker logfile." In-Reply-To: <152f9b10-e3d4-4cb8-bef2-68dce4a56833@googlegroups.com> References: <152f9b10-e3d4-4cb8-bef2-68dce4a56833@googlegroups.com> Message-ID: On 24 April 2014 at 15:57:07, AMAL G JOSE (amalgjose at gmail.com) wrote: > > But when I am giving the ipadress, I am getting an error "ACCESS_REFUSED > - Login was refused using authentication mechanism PLAIN. For > details see the broker logfile.? ?You likely connect as ?guest?? Then see?http://www.rabbitmq.com/blog/2014/04/02/breaking-things-with-rabbitmq-3-3/ -- MK Software Engineer, Pivotal/RabbitMQ From loganathan.ms at gmail.com Thu Apr 24 13:45:12 2014 From: loganathan.ms at gmail.com (Loganathan Sellapa) Date: Thu, 24 Apr 2014 18:15:12 +0530 Subject: [rabbitmq-discuss] How do we connect with AMQP without username password In-Reply-To: References: Message-ID: HI Michael, Thanks Its working with EXTERNAL auth mechanism. regards, Loganathan Mob: +91 7760780741 | +91 9944414388 Skype: loganathan.sellappa ViewMe On Thu, Apr 24, 2014 at 5:08 PM, Michael Klishin wrote: > On 24 April 2014 at 15:35:05, Loganathan Sellapa (loganathan.ms at gmail.com) > wrote: > > > I have enabled the SSL and I am able to establish the connection > > only with username & password with the certificates. But I would > > like to connect to the RabbitMq without user credentials. Is > > their any way to do it? > > amqp gem and Bunny support authentication mechanisms (e.g. EXTERNAL), > :auth_mechanism is > the option name. > > However, I don?t know if anyone has tried using them with the SSL auth > mechanism plugin. > Try it. > -- > MK > > Software Engineer, Pivotal/RabbitMQ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From vish.ramachandran at gmail.com Thu Apr 24 18:01:55 2014 From: vish.ramachandran at gmail.com (vish.ramachandran) Date: Thu, 24 Apr 2014 10:01:55 -0700 (PDT) Subject: [rabbitmq-discuss] High Availability and Load Balancers Message-ID: <1398358915354-35058.post@n5.nabble.com> Hello RabbitMq Team, We are caught in a decision point on whether to choose a load balancer in front of cluster members or to choose a setup where the list of cluster members is baked into client configuration. Data points: 1. We are using clustered rmqs mainly for high availability. Our queues are set up for HA in this setup. 2. Scalability is not a concern yet. We don't expect to add new members to the cluster dynamically. Dynamic DNS is a possibility for recovering any failed nodes. 3. We are using libraries like Spring AMQP and SStone that provide for automatic reconnect/failover. This takes care of consumption. We also plan to design clients to retry publishing upon failure. Questions: 1. We would like to detect failed connections quickly on the client side. We wonder whether TCP load balancers do a good job of detecting failed connections or sit on a bad connection till a real problem is seen. 2. If clients deal with the cluster members directly, is it any better? Can the RMQ client library (like sstone or spring amqp) do a better job at detecting failures quicker than load balancers? 3. Can the actual consumers and publishers (clients of the RMQ libraries) take any special action to detect and recover from failures quickly? Thanks Vish Thanks Vish -- View this message in context: http://rabbitmq.1065348.n5.nabble.com/High-Availability-and-Load-Balancers-tp35058.html Sent from the RabbitMQ mailing list archive at Nabble.com. From srinatsr at cisco.com Thu Apr 24 19:44:20 2014 From: srinatsr at cisco.com (Srinath Sridharan -X (srinatsr - ZENSAR TECHNOLOGIES INC at Cisco)) Date: Thu, 24 Apr 2014 18:44:20 +0000 Subject: [rabbitmq-discuss] How to monitor RabbitMQ queues Message-ID: <1DB4F5481AB296448E8A5B5BC9D336541CB02033@xmb-aln-x10.cisco.com> Hello, I need to monitor the rabbitmq queues, to know what messages are coming in and who is consuming the messages. Basically I need to monitor the queues and it's status. We have implemented RabbitMQ in Java. Regards ...* Srinath -------------- next part -------------- An HTML attachment was scrubbed... URL: From arunrao.seattle at gmail.com Thu Apr 24 19:48:11 2014 From: arunrao.seattle at gmail.com (Arun Rao) Date: Thu, 24 Apr 2014 11:48:11 -0700 Subject: [rabbitmq-discuss] How to monitor RabbitMQ queues In-Reply-To: <1DB4F5481AB296448E8A5B5BC9D336541CB02033@xmb-aln-x10.cisco.com> References: <1DB4F5481AB296448E8A5B5BC9D336541CB02033@xmb-aln-x10.cisco.com> Message-ID: You can do it automatically using RabbitMQ API : http://hg.rabbitmq.com/rabbitmq-management/raw-file/3646dee55e02/priv/www-api/help.html On Thu, Apr 24, 2014 at 11:44 AM, Srinath Sridharan -X (srinatsr - ZENSAR TECHNOLOGIES INC at Cisco) wrote: > Hello, > > > > I need to monitor the rabbitmq queues, to know what messages are coming in > and who is consuming the messages. > > > > Basically I need to monitor the queues and it?s status. > > > > We have implemented RabbitMQ in Java. > > > > > > > > Regards ?*?* > > Srinath > > > > _______________________________________________ > rabbitmq-discuss mailing list > rabbitmq-discuss at lists.rabbitmq.com > https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mcintoshj at gmail.com Thu Apr 24 19:50:51 2014 From: mcintoshj at gmail.com (Jason McIntosh) Date: Thu, 24 Apr 2014 13:50:51 -0500 Subject: [rabbitmq-discuss] How to monitor RabbitMQ queues In-Reply-To: <1DB4F5481AB296448E8A5B5BC9D336541CB02033@xmb-aln-x10.cisco.com> References: <1DB4F5481AB296448E8A5B5BC9D336541CB02033@xmb-aln-x10.cisco.com> Message-ID: It sounds like you want to monitor not just the queues but your connections and what the connections are bound to? If you look at the API it might give you some of this: http://server:15672/api/queues/ http://server:15672/api/connections/ I've got monitoring with Zabbix right now: https://github.com/jasonmcintosh/rabbitmq-zabbix And though I don't have message rights, I do track how many consumers are on each queue and other information. Jason On Thu, Apr 24, 2014 at 1:44 PM, Srinath Sridharan -X (srinatsr - ZENSAR TECHNOLOGIES INC at Cisco) wrote: > Hello, > > > > I need to monitor the rabbitmq queues, to know what messages are coming in > and who is consuming the messages. > > > > Basically I need to monitor the queues and it?s status. > > > > We have implemented RabbitMQ in Java. > > > > > > > > Regards ?*?* > > Srinath > > > > _______________________________________________ > rabbitmq-discuss mailing list > rabbitmq-discuss at lists.rabbitmq.com > https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss > > -- Jason McIntosh https://github.com/jasonmcintosh/ 573-424-7612 -------------- next part -------------- An HTML attachment was scrubbed... URL: From arunrao.seattle at gmail.com Thu Apr 24 19:52:46 2014 From: arunrao.seattle at gmail.com (Arun Rao) Date: Thu, 24 Apr 2014 11:52:46 -0700 Subject: [rabbitmq-discuss] High Availability and Load Balancers In-Reply-To: <1398358915354-35058.post@n5.nabble.com> References: <1398358915354-35058.post@n5.nabble.com> Message-ID: I am using Load Balancer (F5). I am using Producer VIP separately from Consumer VIP. This gives me a lot of flexibility when doing maintenance. For Producer VIP, I am using a API health http://:15672/api/aliveness-test/ and looking for "status":"ok" result. For consumer VIP, I am just using TCP Half-open. HTH, -arun. On Thu, Apr 24, 2014 at 10:01 AM, vish.ramachandran < vish.ramachandran at gmail.com> wrote: > Hello RabbitMq Team, > > We are caught in a decision point on whether to choose a load balancer in > front of cluster members or to choose a setup where the list of cluster > members is baked into client configuration. > > Data points: > > 1. We are using clustered rmqs mainly for high availability. Our queues are > set up for HA in this setup. > > 2. Scalability is not a concern yet. We don't expect to add new members to > the cluster dynamically. Dynamic DNS is a possibility for recovering any > failed nodes. > > 3. We are using libraries like Spring AMQP and SStone that provide for > automatic reconnect/failover. This takes care of consumption. We also plan > to design clients to retry publishing upon failure. > > > Questions: > > 1. We would like to detect failed connections quickly on the client side. > We > wonder whether TCP load balancers do a good job of detecting failed > connections or sit on a bad connection till a real problem is seen. > > 2. If clients deal with the cluster members directly, is it any better? Can > the RMQ client library (like sstone or spring amqp) do a better job at > detecting failures quicker than load balancers? > > 3. Can the actual consumers and publishers (clients of the RMQ libraries) > take any special action to detect and recover from failures quickly? > > > Thanks > Vish > > > Thanks > Vish > > > > > -- > View this message in context: > http://rabbitmq.1065348.n5.nabble.com/High-Availability-and-Load-Balancers-tp35058.html > Sent from the RabbitMQ mailing list archive at Nabble.com. > _______________________________________________ > rabbitmq-discuss mailing list > rabbitmq-discuss at lists.rabbitmq.com > https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss > -------------- next part -------------- An HTML attachment was scrubbed... URL: From srinatsr at cisco.com Thu Apr 24 20:16:14 2014 From: srinatsr at cisco.com (Srinath Sridharan -X (srinatsr - ZENSAR TECHNOLOGIES INC at Cisco)) Date: Thu, 24 Apr 2014 19:16:14 +0000 Subject: [rabbitmq-discuss] How to monitor RabbitMQ queues In-Reply-To: References: <1DB4F5481AB296448E8A5B5BC9D336541CB02033@xmb-aln-x10.cisco.com> Message-ID: <1DB4F5481AB296448E8A5B5BC9D336541CB020A6@xmb-aln-x10.cisco.com> 1. Utilize RabbitMQ/Plugin APIs to obtain queuing data/activities 2. Gather the status for RabbitMQ nodes and queues 3. Create a monitor queue for tracking all activities I need to achieve the above things? Regards ?? Srinath From: rabbitmq-discuss [mailto:rabbitmq-discuss-bounces at lists.rabbitmq.com] On Behalf Of Jason McIntosh Sent: Thursday, April 24, 2014 11:51 AM To: Discussions about RabbitMQ Subject: Re: [rabbitmq-discuss] How to monitor RabbitMQ queues It sounds like you want to monitor not just the queues but your connections and what the connections are bound to? If you look at the API it might give you some of this: http://server:15672/api/queues/ http://server:15672/api/connections/ I've got monitoring with Zabbix right now: https://github.com/jasonmcintosh/rabbitmq-zabbix And though I don't have message rights, I do track how many consumers are on each queue and other information. Jason On Thu, Apr 24, 2014 at 1:44 PM, Srinath Sridharan -X (srinatsr - ZENSAR TECHNOLOGIES INC at Cisco) > wrote: Hello, I need to monitor the rabbitmq queues, to know what messages are coming in and who is consuming the messages. Basically I need to monitor the queues and it?s status. We have implemented RabbitMQ in Java. Regards ?? Srinath _______________________________________________ rabbitmq-discuss mailing list rabbitmq-discuss at lists.rabbitmq.com https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss -- Jason McIntosh https://github.com/jasonmcintosh/ 573-424-7612 -------------- next part -------------- An HTML attachment was scrubbed... URL: From ron.cordell at gmail.com Thu Apr 24 20:48:53 2014 From: ron.cordell at gmail.com (Ron Cordell) Date: Thu, 24 Apr 2014 12:48:53 -0700 Subject: [rabbitmq-discuss] High Availability and Load Balancers In-Reply-To: References: <1398358915354-35058.post@n5.nabble.com> Message-ID: We have a similar setup with our F5 LB - exposing a VIP to the application and checking the health of the nodes from the F5 by querying the api. Cheers, -ronc On Thu, Apr 24, 2014 at 11:52 AM, Arun Rao wrote: > I am using Load Balancer (F5). I am using Producer VIP separately from > Consumer VIP. This gives me a lot of flexibility when doing maintenance. > For Producer VIP, I am using a API health http://:15672/api/aliveness-test/ > and looking for "status":"ok" result. > > For consumer VIP, I am just using TCP Half-open. > > HTH, > -arun. > > > On Thu, Apr 24, 2014 at 10:01 AM, vish.ramachandran < > vish.ramachandran at gmail.com> wrote: > >> Hello RabbitMq Team, >> >> We are caught in a decision point on whether to choose a load balancer in >> front of cluster members or to choose a setup where the list of cluster >> members is baked into client configuration. >> >> Data points: >> >> 1. We are using clustered rmqs mainly for high availability. Our queues >> are >> set up for HA in this setup. >> >> 2. Scalability is not a concern yet. We don't expect to add new members to >> the cluster dynamically. Dynamic DNS is a possibility for recovering any >> failed nodes. >> >> 3. We are using libraries like Spring AMQP and SStone that provide for >> automatic reconnect/failover. This takes care of consumption. We also plan >> to design clients to retry publishing upon failure. >> >> >> Questions: >> >> 1. We would like to detect failed connections quickly on the client side. >> We >> wonder whether TCP load balancers do a good job of detecting failed >> connections or sit on a bad connection till a real problem is seen. >> >> 2. If clients deal with the cluster members directly, is it any better? >> Can >> the RMQ client library (like sstone or spring amqp) do a better job at >> detecting failures quicker than load balancers? >> >> 3. Can the actual consumers and publishers (clients of the RMQ libraries) >> take any special action to detect and recover from failures quickly? >> >> >> Thanks >> Vish >> >> >> Thanks >> Vish >> >> >> >> >> -- >> View this message in context: >> http://rabbitmq.1065348.n5.nabble.com/High-Availability-and-Load-Balancers-tp35058.html >> Sent from the RabbitMQ mailing list archive at Nabble.com. >> _______________________________________________ >> rabbitmq-discuss mailing list >> rabbitmq-discuss at lists.rabbitmq.com >> https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss >> > > > _______________________________________________ > rabbitmq-discuss mailing list > rabbitmq-discuss at lists.rabbitmq.com > https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From michael at rabbitmq.com Thu Apr 24 21:31:01 2014 From: michael at rabbitmq.com (Michael Klishin) Date: Fri, 25 Apr 2014 00:31:01 +0400 Subject: [rabbitmq-discuss] How to monitor RabbitMQ queues In-Reply-To: <1DB4F5481AB296448E8A5B5BC9D336541CB020A6@xmb-aln-x10.cisco.com> References: <1DB4F5481AB296448E8A5B5BC9D336541CB02033@xmb-aln-x10.cisco.com> <1DB4F5481AB296448E8A5B5BC9D336541CB020A6@xmb-aln-x10.cisco.com> Message-ID: As it was mentioned earlier, everything you need is provided by RabbitMQ HTTP API. MK > On 24/04/2014, at 23:16, "Srinath Sridharan -X (srinatsr - ZENSAR TECHNOLOGIES INC at Cisco)" wrote: > > Utilize RabbitMQ/Plugin APIs to obtain queuing data/activities > > 2. Gather the status for RabbitMQ nodes and queues > > 3. Create a monitor queue for tracking all activities > > > > I need to achieve the above things? -------------- next part -------------- An HTML attachment was scrubbed... URL: From goyalk at vmware.com Fri Apr 25 02:16:45 2014 From: goyalk at vmware.com (Kapil Goyal) Date: Thu, 24 Apr 2014 18:16:45 -0700 (PDT) Subject: [rabbitmq-discuss] NC1/CS6 marking of RabbitMQ packets Message-ID: Hi, We have an ask to mark all packets exchanged between RabbitMQ broker and client applications to be marked with NC1/CS6. Any suggestions on how this can be worked out? Thanks Kapil -------------- next part -------------- An HTML attachment was scrubbed... URL: From mklishin at gopivotal.com Fri Apr 25 09:47:42 2014 From: mklishin at gopivotal.com (Michael Klishin) Date: Fri, 25 Apr 2014 12:47:42 +0400 Subject: [rabbitmq-discuss] NC1/CS6 marking of RabbitMQ packets In-Reply-To: References: Message-ID: On 25 April 2014 at 05:18:10, Kapil Goyal (goyalk at vmware.com) wrote: > > We have an ask to mark all packets exchanged between RabbitMQ > broker and client applications to be marked with NC1/CS6. Any > suggestions on how this can be worked out? RabbitMQ uses TCP connections and is not aware of?MPLS or similar mechanisms. Presumably you can use destination host and port (5672 and 5671 by default for RabbitMQ).? -- MK Software Engineer, Pivotal/RabbitMQ From qiujixiong at 163.com Fri Apr 25 10:09:49 2014 From: qiujixiong at 163.com (qiujixiong) Date: Fri, 25 Apr 2014 17:09:49 +0800 (CST) Subject: [rabbitmq-discuss] Why rabbitmq stop pushing message to consumer Message-ID: <6ed96fd1.d2d0.1459825dc8e.Coremail.qiujixiong@163.com> Hi, I have a problem with rabbitmq recently . The message in queue accumulated and the queue become more and more bigger(about hundred thousands of messages in queue). Then I found rabbitmq did not push message to consumers and all consumers were blocked on tcp receive function. While the publishers were continue publishing message at that time. (When I found the channel was blocked, I would try to re-create a connection to publish message. Am I wrong to publish message in such a way?) Is stopping pushing message to consumers the reason of message accumulated or the result of message accumulated? Another thing, the message is durable, and disk io at that time was almost full. May this be the reason of message accumulated? Then, still, why consumers could not receive message from rabbitmq? Can anyone help? Thanks very much in advanced! Here is the status of rabbitmq I dump when problem occur: Reporting server status on {{2014,4,23},{14,50,1}} ... Status of node rabbit4 at rabbitmq153 ... [{pid,8934}, {running_applications, [{rabbitmq_federation_management,"RabbitMQ Federation Management", "3.1.5"}, {rabbitmq_management,"RabbitMQ Management Console","3.1.5"}, {rabbitmq_federation,"RabbitMQ Federation","3.1.5"}, {rabbitmq_management_agent,"RabbitMQ Management Agent","3.1.5"}, {rabbit,"RabbitMQ","3.1.5"}, {os_mon,"CPO CXC 138 46","2.2.7"}, {rabbitmq_web_dispatch,"RabbitMQ Web Dispatcher","3.1.5"}, {webmachine,"webmachine","1.10.3-rmq3.1.5-gite9359c7"}, {mochiweb,"MochiMedia Web Server","2.7.0-rmq3.1.5-git680dba8"}, {xmerl,"XML parser","1.2.10"}, {inets,"INETS CXC 138 49","5.7.1"}, {mnesia,"MNESIA CXC 138 12","4.5"}, {amqp_client,"RabbitMQ AMQP Client","3.1.5"}, {sasl,"SASL CXC 138 11","2.1.10"}, {stdlib,"ERTS CXC 138 10","1.17.5"}, {kernel,"ERTS CXC 138 10","2.14.5"}]}, {os,{unix,linux}}, {erlang_version, "Erlang R14B04 (erts-5.8.5) [source] [64-bit] [smp:16:16] [rq:16] [async-threads:30] [kernel-poll:true]\n"}, {memory, [{total,2384958696}, {connection_procs,243278664}, {queue_procs,1295680360}, {plugins,2061384}, {other_proc,189406552}, {mnesia,138736}, {mgmt_db,54466288}, {msg_index,8958408}, {other_ets,37974792}, {binary,367954632}, {code,17812261}, {atom,2285609}, {other_system,164941010}]}, {vm_memory_high_watermark,0.9}, {vm_memory_limit,15137044070}, {disk_free_limit,1000000000}, {disk_free,219615227904}, {file_descriptors, [{total_limit,262040}, {total_used,4426}, {sockets_limit,235834}, {sockets_used,4422}]}, {processes,[{limit,1048576},{used,39049}]}, {run_queue,12}, {uptime,3590787}] Cluster status of node rabbit4 at rabbitmq153 ... [{nodes,[{disc,[rabbit3 at rabbitmq152,rabbit4 at rabbitmq153]}]}, {running_nodes,[rabbit3 at rabbitmq152,rabbit4 at rabbitmq153]}, {partitions,[]}] Application environment of node rabbit4 at rabbitmq153 ... [{auth_backends,[rabbit_auth_backend_internal]}, {auth_mechanisms,['PLAIN','AMQPLAIN']}, {backing_queue_module,rabbit_variable_queue}, {cluster_nodes,{[],disc}}, {cluster_partition_handling,ignore}, {collect_statistics,fine}, {collect_statistics_interval,5000}, {default_permissions,[<<".*">>,<<".*">>,<<".*">>]}, {default_user,<<"guest">>}, {default_user_tags,[administrator]}, {default_vhost,<<"/">>}, {delegate_count,16}, {disk_free_limit,1000000000}, {enabled_plugins_file,"/etc/rabbitmq/enabled_plugins"}, {error_logger,{file,"/data/yy/log/rabbitmq/rabbit4 at rabbitmq153.log"}}, {frame_max,131072}, {heartbeat,600}, {hipe_compile,false}, {hipe_modules,[rabbit_reader,rabbit_channel,gen_server2,rabbit_exchange, rabbit_command_assembler,rabbit_framing_amqp_0_9_1, rabbit_basic,rabbit_event,lists,queue,priority_queue, rabbit_router,rabbit_trace,rabbit_misc,rabbit_binary_parser, rabbit_exchange_type_direct,rabbit_guid,rabbit_net, rabbit_amqqueue_process,rabbit_variable_queue, rabbit_binary_generator,rabbit_writer,delegate,gb_sets,lqueue, sets,orddict,rabbit_amqqueue,rabbit_limiter,gb_trees, rabbit_queue_index,rabbit_exchange_decorator,gen,dict,ordsets, file_handle_cache,rabbit_msg_store,array, rabbit_msg_store_ets_index,rabbit_msg_file, rabbit_exchange_type_fanout,rabbit_exchange_type_topic,mnesia, mnesia_lib,rpc,mnesia_tm,qlc,sofs,proplists,credit_flow,pmon, ssl_connection,tls_connection,ssl_record,tls_record,gen_fsm, ssl]}, {included_applications,[]}, {log_levels,[{connection,info}]}, {msg_store_file_size_limit,16777216}, {msg_store_index_module,rabbit_msg_store_ets_index}, {plugins_dir,"/usr/lib/rabbitmq/lib/rabbitmq_server-3.1.5/sbin/../plugins"}, {plugins_expand_dir,"/data/rabbitmq/rabbit4 at rabbitmq153-plugins-expand"}, {queue_index_max_journal_entries,65536}, {reverse_dns_lookups,false}, {sasl_error_logger,{file,"/data/yy/log/rabbitmq/rabbit4 at rabbitmq153-sasl.log"}}, {server_properties,[]}, {ssl_apps,[asn1,crypto,public_key,ssl]}, {ssl_cert_login_from,distinguished_name}, {ssl_listeners,[]}, {ssl_options,[]}, {tcp_listen_options,[binary, {packet,raw}, {reuseaddr,true}, {backlog,128}, {nodelay,true}, {linger,{true,0}}, {exit_on_close,false}]}, {tcp_listeners,[5672]}, {trace_vhosts,[]}, {vm_memory_high_watermark,0.9}, {vm_memory_high_watermark_paging_ratio,0.9}] Status of node rabbit3 at rabbitmq152 ... [{pid,21118}, {running_applications, [{rabbitmq_federation_management,"RabbitMQ Federation Management", "3.1.5"}, {rabbitmq_management,"RabbitMQ Management Console","3.1.5"}, {rabbitmq_federation,"RabbitMQ Federation","3.1.5"}, {rabbitmq_management_agent,"RabbitMQ Management Agent","3.1.5"}, {rabbit,"RabbitMQ","3.1.5"}, {os_mon,"CPO CXC 138 46","2.2.7"}, {rabbitmq_web_dispatch,"RabbitMQ Web Dispatcher","3.1.5"}, {webmachine,"webmachine","1.10.3-rmq3.1.5-gite9359c7"}, {mochiweb,"MochiMedia Web Server","2.7.0-rmq3.1.5-git680dba8"}, {xmerl,"XML parser","1.2.10"}, {inets,"INETS CXC 138 49","5.7.1"}, {mnesia,"MNESIA CXC 138 12","4.5"}, {amqp_client,"RabbitMQ AMQP Client","3.1.5"}, {sasl,"SASL CXC 138 11","2.1.10"}, {stdlib,"ERTS CXC 138 10","1.17.5"}, {kernel,"ERTS CXC 138 10","2.14.5"}]}, {os,{unix,linux}}, {erlang_version, "Erlang R14B04 (erts-5.8.5) [source] [64-bit] [smp:16:16] [rq:16] [async-threads:30] [kernel-poll:true]\n"}, {memory, [{total,1948612952}, {connection_procs,273728912}, {queue_procs,1262564272}, {plugins,577536}, {other_proc,15237064}, {mnesia,138720}, {mgmt_db,9904}, {msg_index,3223064}, {other_ets,27246728}, {binary,198736656}, {code,17812261}, {atom,1624553}, {other_system,147713282}]}, {vm_memory_high_watermark,0.9}, {vm_memory_limit,15136985088}, {disk_free_limit,1000000000}, {disk_free,221611429888}, {file_descriptors, [{total_limit,262040}, {total_used,1010}, {sockets_limit,235834}, {sockets_used,1006}]}, {processes,[{limit,1048576},{used,9175}]}, {run_queue,0}, {uptime,47734}] Cluster status of node rabbit3 at rabbitmq152 ... [{nodes,[{disc,[rabbit3 at rabbitmq152,rabbit4 at rabbitmq153]}]}, {running_nodes,[rabbit4 at rabbitmq153,rabbit3 at rabbitmq152]}, {partitions,[]}] Application environment of node rabbit3 at rabbitmq152 ... [{auth_backends,[rabbit_auth_backend_internal]}, {auth_mechanisms,['PLAIN','AMQPLAIN']}, {backing_queue_module,rabbit_variable_queue}, {cluster_nodes,{[],disc}}, {cluster_partition_handling,ignore}, {collect_statistics,fine}, {collect_statistics_interval,5000}, {default_permissions,[<<".*">>,<<".*">>,<<".*">>]}, {default_user,<<"guest">>}, {default_user_tags,[administrator]}, {default_vhost,<<"/">>}, {delegate_count,16}, {disk_free_limit,1000000000}, {enabled_plugins_file,"/etc/rabbitmq/enabled_plugins"}, {error_logger,{file,"/data/yy/log/rabbitmq/rabbit3 at rabbitmq152.log"}}, {frame_max,131072}, {heartbeat,600}, {hipe_compile,false}, {hipe_modules,[rabbit_reader,rabbit_channel,gen_server2,rabbit_exchange, rabbit_command_assembler,rabbit_framing_amqp_0_9_1, rabbit_basic,rabbit_event,lists,queue,priority_queue, rabbit_router,rabbit_trace,rabbit_misc,rabbit_binary_parser, rabbit_exchange_type_direct,rabbit_guid,rabbit_net, rabbit_amqqueue_process,rabbit_variable_queue, rabbit_binary_generator,rabbit_writer,delegate,gb_sets,lqueue, sets,orddict,rabbit_amqqueue,rabbit_limiter,gb_trees, rabbit_queue_index,rabbit_exchange_decorator,gen,dict,ordsets, file_handle_cache,rabbit_msg_store,array, rabbit_msg_store_ets_index,rabbit_msg_file, rabbit_exchange_type_fanout,rabbit_exchange_type_topic,mnesia, mnesia_lib,rpc,mnesia_tm,qlc,sofs,proplists,credit_flow,pmon, ssl_connection,tls_connection,ssl_record,tls_record,gen_fsm, ssl]}, {included_applications,[]}, {log_levels,[{connection,info}]}, {msg_store_file_size_limit,16777216}, {msg_store_index_module,rabbit_msg_store_ets_index}, {plugins_dir,"/usr/lib/rabbitmq/lib/rabbitmq_server-3.1.5/sbin/../plugins"}, {plugins_expand_dir,"/data/rabbitmq/rabbit3 at rabbitmq152-plugins-expand"}, {queue_index_max_journal_entries,65536}, {reverse_dns_lookups,false}, {sasl_error_logger,{file,"/data/yy/log/rabbitmq/rabbit3 at rabbitmq152-sasl.log"}}, {server_properties,[]}, {ssl_apps,[asn1,crypto,public_key,ssl]}, {ssl_cert_login_from,distinguished_name}, {ssl_listeners,[]}, {ssl_options,[]}, {tcp_listen_options,[binary, {packet,raw}, {reuseaddr,true}, {backlog,128}, {nodelay,true}, {linger,{true,0}}, {exit_on_close,false}]}, {tcp_listeners,[5672]}, {trace_vhosts,[]}, {vm_memory_high_watermark,0.9}, {vm_memory_high_watermark_paging_ratio,0.9}] Connections: (the report stopped here and no further info show) -------------- next part -------------- An HTML attachment was scrubbed... URL: From mklishin at gopivotal.com Fri Apr 25 12:58:36 2014 From: mklishin at gopivotal.com (Michael Klishin) Date: Fri, 25 Apr 2014 15:58:36 +0400 Subject: [rabbitmq-discuss] Why rabbitmq stop pushing message to consumer In-Reply-To: <6ed96fd1.d2d0.1459825dc8e.Coremail.qiujixiong@163.com> References: <6ed96fd1.d2d0.1459825dc8e.Coremail.qiujixiong@163.com> Message-ID: On 25 April 2014 at 13:11:14, qiujixiong (qiujixiong at 163.com) wrote: > > Is stopping pushing message to consumers the reason of message > accumulated or the result of message accumulated? Another thing, > the message is durable, and disk io at that time was almost full. > May this be the reason of message accumulated? Then, still, why > consumers could not receive message from rabbitmq? Resource-based flow control should only block publishers. Re-creating connections won?t help: every publishing connection will be blocked. Consumers should not be blocked but nearly running out of disk space sounds alarming and can prevent RabbitMQ from moving messages to disk or very heavy OS paging . What?s in RabbitMQ log? There should be disk and RAM alarms logged. rabbitmqctl status indicates disk limit alarm should be on. -- MK Software Engineer, Pivotal/RabbitMQ From mcintoshj at gmail.com Fri Apr 25 14:30:48 2014 From: mcintoshj at gmail.com (Jason McIntosh) Date: Fri, 25 Apr 2014 08:30:48 -0500 Subject: [rabbitmq-discuss] High Availability and Load Balancers In-Reply-To: References: <1398358915354-35058.post@n5.nabble.com> Message-ID: Ditto on Ron. Some things to be aware of: Most load balancers have monitors that check the status. This half-opens a connection to the rabbit ports, so rabbit will log all of these as info and warning level messages. It tends to flood the logs. Also, watch your heartbeats. We hit an issue initially with our F5's where we had a short TCP/IP timeout and rabbit's heartbeat was longer, so we were getting disconnects every 5 minutes. But, this config has worked well for us for at least two years now :) We back clusters with this, and individual non clustered nodes as well. Jason On Thu, Apr 24, 2014 at 2:48 PM, Ron Cordell wrote: > We have a similar setup with our F5 LB - exposing a VIP to the application > and checking the health of the nodes from the F5 by querying the api. > > Cheers, > > -ronc > > > On Thu, Apr 24, 2014 at 11:52 AM, Arun Rao wrote: > >> I am using Load Balancer (F5). I am using Producer VIP separately from >> Consumer VIP. This gives me a lot of flexibility when doing maintenance. >> For Producer VIP, I am using a API health http://:15672/api/aliveness-test/ >> and looking for "status":"ok" result. >> >> For consumer VIP, I am just using TCP Half-open. >> >> HTH, >> -arun. >> >> >> On Thu, Apr 24, 2014 at 10:01 AM, vish.ramachandran < >> vish.ramachandran at gmail.com> wrote: >> >>> Hello RabbitMq Team, >>> >>> We are caught in a decision point on whether to choose a load balancer in >>> front of cluster members or to choose a setup where the list of cluster >>> members is baked into client configuration. >>> >>> Data points: >>> >>> 1. We are using clustered rmqs mainly for high availability. Our queues >>> are >>> set up for HA in this setup. >>> >>> 2. Scalability is not a concern yet. We don't expect to add new members >>> to >>> the cluster dynamically. Dynamic DNS is a possibility for recovering any >>> failed nodes. >>> >>> 3. We are using libraries like Spring AMQP and SStone that provide for >>> automatic reconnect/failover. This takes care of consumption. We also >>> plan >>> to design clients to retry publishing upon failure. >>> >>> >>> Questions: >>> >>> 1. We would like to detect failed connections quickly on the client >>> side. We >>> wonder whether TCP load balancers do a good job of detecting failed >>> connections or sit on a bad connection till a real problem is seen. >>> >>> 2. If clients deal with the cluster members directly, is it any better? >>> Can >>> the RMQ client library (like sstone or spring amqp) do a better job at >>> detecting failures quicker than load balancers? >>> >>> 3. Can the actual consumers and publishers (clients of the RMQ libraries) >>> take any special action to detect and recover from failures quickly? >>> >>> >>> Thanks >>> Vish >>> >>> >>> Thanks >>> Vish >>> >>> >>> >>> >>> -- >>> View this message in context: >>> http://rabbitmq.1065348.n5.nabble.com/High-Availability-and-Load-Balancers-tp35058.html >>> Sent from the RabbitMQ mailing list archive at Nabble.com. >>> _______________________________________________ >>> rabbitmq-discuss mailing list >>> rabbitmq-discuss at lists.rabbitmq.com >>> https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss >>> >> >> >> _______________________________________________ >> rabbitmq-discuss mailing list >> rabbitmq-discuss at lists.rabbitmq.com >> https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss >> >> > > _______________________________________________ > rabbitmq-discuss mailing list > rabbitmq-discuss at lists.rabbitmq.com > https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss > > -- Jason McIntosh https://github.com/jasonmcintosh/ 573-424-7612 -------------- next part -------------- An HTML attachment was scrubbed... URL: From c1ear at sympatico.ca Fri Apr 25 14:35:57 2014 From: c1ear at sympatico.ca (prex) Date: Fri, 25 Apr 2014 06:35:57 -0700 (PDT) Subject: [rabbitmq-discuss] RabbitMQ on Windows XP - Failed to create cookie file In-Reply-To: <2C7E92C7E7E92A4E850BC32CB8347DB402CF5206@chsa1068.share.beluni.net> References: <2C7E92C7E7E92A4E850BC32CB8347DB402CF5206@chsa1068.share.beluni.net> Message-ID: <1398432957801-35071.post@n5.nabble.com> I realize that this thread is old but I just wanted to add some more insight here. I'm also seeing this problem with latest version of RabbitMQ (3.1.0) and it seems there is an issue when the Windows AD home path is set on the account RabbitMQ is being configured under. Some organizations set Windows users home dir to a shared location (usually z:\ ). RabbitMQ however seems to have issues with this regardless what permissions are set for this directory. Directories local to the machine RabbitMQ is configured on are OK. -- View this message in context: http://rabbitmq.1065348.n5.nabble.com/RabbitMQ-on-Windows-XP-Failed-to-create-cookie-file-tp10460p35071.html Sent from the RabbitMQ mailing list archive at Nabble.com. From simon at rabbitmq.com Fri Apr 25 14:46:28 2014 From: simon at rabbitmq.com (Simon MacMullen) Date: Fri, 25 Apr 2014 14:46:28 +0100 Subject: [rabbitmq-discuss] Exclusive consumers via STOMP In-Reply-To: <5358FB50.4010006@fsn.hu> References: <5358C9B5.9020305@fsn.hu> <5358FB50.4010006@fsn.hu> Message-ID: <535A6734.1080502@rabbitmq.com> On 24/04/2014 12:53, Nagy, Attila wrote: >> Possibly. In RabbitMQ consumer exclusivity means no more than 1 >> consumer can >> be added on a given queue. This includes federation links for >> federated queues. > Exactly what I need. It's worth noting that no more than one consumer can be on one individual queue, not one set of queues that are federated together. >> I need to run some tests to see how queue federation works in such case. >> > How does queue federation consumes from the queue? I mean it's not just > a normal consumer, because if nobody is on the other side, it won't > consume. Or it just starts consuming if there are somebody? > But there are even more conditions need to be true to get messages to it > (described on the federation page), so this is not so simple. It is a low priority (http://www.rabbitmq.com/consumer-priority.html) consumer that starts and stops according to whether the downstream "needs" messages. > Also, if consumer A on broker A consumes from the queue, but broker B > can't consume from broker A's queue, I assume consumer B on broker B > will be able to consume from the queue on broker B, so this essentially > doesn't work in a federated setup. That's correct. > Or is there something which disallows this? No. And really there couldn't be; in order to do that you'd need some distributed shared state, which would have to be Consistent and Available, thius not Partition Tolerant, thus not really compatible with how federation works. Cheers, Simon From michael.laing at nytimes.com Fri Apr 25 16:10:55 2014 From: michael.laing at nytimes.com (Laing, Michael) Date: Fri, 25 Apr 2014 11:10:55 -0400 Subject: [rabbitmq-discuss] Set x-priority on a static shovel Message-ID: Is this possible? If so what is the syntax? I would like to use this to adjust the priorities of the shovels that distribute load from a single input queue to clusters in other data centers. Thanks, Michael -------------- next part -------------- An HTML attachment was scrubbed... URL: From simon at rabbitmq.com Fri Apr 25 16:24:09 2014 From: simon at rabbitmq.com (Simon MacMullen) Date: Fri, 25 Apr 2014 16:24:09 +0100 Subject: [rabbitmq-discuss] Set x-priority on a static shovel In-Reply-To: References: Message-ID: <535A7E19.9080907@rabbitmq.com> On 25/04/2014 16:10, Laing, Michael wrote: > Is this possible? If so what is the syntax? It's not I'm afraid, the basic.consume method's arguments can't be configured. I've filed a bug for that. Cheers, Simon From michael.laing at nytimes.com Fri Apr 25 16:42:28 2014 From: michael.laing at nytimes.com (Laing, Michael) Date: Fri, 25 Apr 2014 11:42:28 -0400 Subject: [rabbitmq-discuss] Set x-priority on a static shovel In-Reply-To: <535A7E19.9080907@rabbitmq.com> References: <535A7E19.9080907@rabbitmq.com> Message-ID: Thanks. I general it's useful to be able to configure pretty much everything in the shovel :) On Fri, Apr 25, 2014 at 11:24 AM, Simon MacMullen wrote: > On 25/04/2014 16:10, Laing, Michael wrote: > >> Is this possible? If so what is the syntax? >> > > It's not I'm afraid, the basic.consume method's arguments can't be > configured. I've filed a bug for that. > > Cheers, Simon > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jeff at ocjtech.us Fri Apr 25 16:44:10 2014 From: jeff at ocjtech.us (Jeffrey Ollie) Date: Fri, 25 Apr 2014 10:44:10 -0500 Subject: [rabbitmq-discuss] NC1/CS6 marking of RabbitMQ packets In-Reply-To: References: Message-ID: On Thu, Apr 24, 2014 at 8:16 PM, Kapil Goyal wrote: > > We have an ask to mark all packets exchanged between RabbitMQ broker and > client applications to be marked with NC1/CS6. Any suggestions on how this > can be worked out? If your hosts are using Linux you can set the DSCP markings using iptables without modifying the source: http://www.linuxtopia.org/Linux_Firewall_iptables/x4172.html Otherwise it would involve modifying the code of the server and all of the client libraries to set the IP_TOS option on all of the sockets that are created. -- Jeff Ollie From goyalk at vmware.com Fri Apr 25 18:28:56 2014 From: goyalk at vmware.com (Kapil Goyal) Date: Fri, 25 Apr 2014 10:28:56 -0700 (PDT) Subject: [rabbitmq-discuss] NC1/CS6 marking of RabbitMQ packets In-Reply-To: References: Message-ID: <1790225943.4428834.1398446936527.JavaMail.root@vmware.com> Thanks Jeff. Is there any RabbitMQ configuration that accepts socket-options? The description of 'tcp_listen_options' says it accepts options for sockets. Can this be utilized? Thanks Kapil ----- Original Message ----- From: "Jeffrey Ollie" To: "Discussions about RabbitMQ" Sent: Friday, April 25, 2014 8:44:10 AM Subject: Re: [rabbitmq-discuss] NC1/CS6 marking of RabbitMQ packets On Thu, Apr 24, 2014 at 8:16 PM, Kapil Goyal wrote: > > We have an ask to mark all packets exchanged between RabbitMQ broker and > client applications to be marked with NC1/CS6. Any suggestions on how this > can be worked out? If your hosts are using Linux you can set the DSCP markings using iptables without modifying the source: https://urldefense.proofpoint.com/v1/url?u=http://www.linuxtopia.org/Linux_Firewall_iptables/x4172.html&k=oIvRg1%2BdGAgOoM1BIlLLqw%3D%3D%0A&r=JF6BIRiITniYYZhbhzlgNvj%2FeB0%2BYbRBSr0f8NG91qs%3D%0A&m=kJYPJn%2BN%2Bv9KWf%2FRg9Pj5TxTAchQLarnZnAnCjDHjAY%3D%0A&s=0c259dd5fd00de315800a05f44dbbb94cc94a73c75f5e0e3e93a512fc8c19f3e Otherwise it would involve modifying the code of the server and all of the client libraries to set the IP_TOS option on all of the sockets that are created. -- Jeff Ollie _______________________________________________ rabbitmq-discuss mailing list rabbitmq-discuss at lists.rabbitmq.com https://urldefense.proofpoint.com/v1/url?u=https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss&k=oIvRg1%2BdGAgOoM1BIlLLqw%3D%3D%0A&r=JF6BIRiITniYYZhbhzlgNvj%2FeB0%2BYbRBSr0f8NG91qs%3D%0A&m=kJYPJn%2BN%2Bv9KWf%2FRg9Pj5TxTAchQLarnZnAnCjDHjAY%3D%0A&s=3dea37c85208707fb6062b1f3e148bd791b0ed614e12a4cf5eeb927079935526 From arunrao.seattle at gmail.com Fri Apr 25 18:29:31 2014 From: arunrao.seattle at gmail.com (Arun Rao) Date: Fri, 25 Apr 2014 10:29:31 -0700 Subject: [rabbitmq-discuss] High Availability and Load Balancers In-Reply-To: References: <1398358915354-35058.post@n5.nabble.com> Message-ID: *>> This half-opens a connection to the rabbit ports, so rabbit will log all of these as info and warning level messages. It tends to flood the logs.* I set my /etc/rabbitmq/rabbitmq.config with the following line, i dont get a log line for each half tcp-open: log_levels, [{connection,error}] On Fri, Apr 25, 2014 at 6:30 AM, Jason McIntosh wrote: > Ditto on Ron. Some things to be aware of: > > Most load balancers have monitors that check the status. This half-opens > a connection to the rabbit ports, so rabbit will log all of these as info > and warning level messages. It tends to flood the logs. Also, watch your > heartbeats. We hit an issue initially with our F5's where we had a short > TCP/IP timeout and rabbit's heartbeat was longer, so we were getting > disconnects every 5 minutes. But, this config has worked well for us for > at least two years now :) We back clusters with this, and individual non > clustered nodes as well. > > Jason > > > On Thu, Apr 24, 2014 at 2:48 PM, Ron Cordell wrote: > >> We have a similar setup with our F5 LB - exposing a VIP to the >> application and checking the health of the nodes from the F5 by querying >> the api. >> >> Cheers, >> >> -ronc >> >> >> On Thu, Apr 24, 2014 at 11:52 AM, Arun Rao wrote: >> >>> I am using Load Balancer (F5). I am using Producer VIP separately from >>> Consumer VIP. This gives me a lot of flexibility when doing maintenance. >>> For Producer VIP, I am using a API health http://:15672/api/aliveness-test/ >>> and looking for "status":"ok" result. >>> >>> For consumer VIP, I am just using TCP Half-open. >>> >>> HTH, >>> -arun. >>> >>> >>> On Thu, Apr 24, 2014 at 10:01 AM, vish.ramachandran < >>> vish.ramachandran at gmail.com> wrote: >>> >>>> Hello RabbitMq Team, >>>> >>>> We are caught in a decision point on whether to choose a load balancer >>>> in >>>> front of cluster members or to choose a setup where the list of cluster >>>> members is baked into client configuration. >>>> >>>> Data points: >>>> >>>> 1. We are using clustered rmqs mainly for high availability. Our queues >>>> are >>>> set up for HA in this setup. >>>> >>>> 2. Scalability is not a concern yet. We don't expect to add new members >>>> to >>>> the cluster dynamically. Dynamic DNS is a possibility for recovering any >>>> failed nodes. >>>> >>>> 3. We are using libraries like Spring AMQP and SStone that provide for >>>> automatic reconnect/failover. This takes care of consumption. We also >>>> plan >>>> to design clients to retry publishing upon failure. >>>> >>>> >>>> Questions: >>>> >>>> 1. We would like to detect failed connections quickly on the client >>>> side. We >>>> wonder whether TCP load balancers do a good job of detecting failed >>>> connections or sit on a bad connection till a real problem is seen. >>>> >>>> 2. If clients deal with the cluster members directly, is it any better? >>>> Can >>>> the RMQ client library (like sstone or spring amqp) do a better job at >>>> detecting failures quicker than load balancers? >>>> >>>> 3. Can the actual consumers and publishers (clients of the RMQ >>>> libraries) >>>> take any special action to detect and recover from failures quickly? >>>> >>>> >>>> Thanks >>>> Vish >>>> >>>> >>>> Thanks >>>> Vish >>>> >>>> >>>> >>>> >>>> -- >>>> View this message in context: >>>> http://rabbitmq.1065348.n5.nabble.com/High-Availability-and-Load-Balancers-tp35058.html >>>> Sent from the RabbitMQ mailing list archive at Nabble.com. >>>> _______________________________________________ >>>> rabbitmq-discuss mailing list >>>> rabbitmq-discuss at lists.rabbitmq.com >>>> https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss >>>> >>> >>> >>> _______________________________________________ >>> rabbitmq-discuss mailing list >>> rabbitmq-discuss at lists.rabbitmq.com >>> https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss >>> >>> >> >> _______________________________________________ >> rabbitmq-discuss mailing list >> rabbitmq-discuss at lists.rabbitmq.com >> https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss >> >> > > > -- > Jason McIntosh > https://github.com/jasonmcintosh/ > 573-424-7612 > > _______________________________________________ > rabbitmq-discuss mailing list > rabbitmq-discuss at lists.rabbitmq.com > https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From marek.stoj at gmail.com Fri Apr 25 14:08:03 2014 From: marek.stoj at gmail.com (=?UTF-8?Q?Marek_St=C3=B3j?=) Date: Fri, 25 Apr 2014 06:08:03 -0700 (PDT) Subject: [rabbitmq-discuss] Recover cluster from crash In-Reply-To: <665d6495-ca13-465d-b741-142f5835a6f0@googlegroups.com> References: <665d6495-ca13-465d-b741-142f5835a6f0@googlegroups.com> Message-ID: <4bed6a8a-f896-4e2b-b981-907045e61d7e@googlegroups.com> Turns out the problem is that the environment variable RABBITMQ_NODE_ONLY is not taken into account on Windows version of RabbitMQ server (sic!). One can workaround this by manually removing ''-s rabbit boot ^" line from rabbitmq-server.bat. On Monday, April 14, 2014 3:04:30 PM UTC+2, js.future... at gmail.com wrote: > > Hello, > > we are testing RabbitMQ for future replace MSMQ. Someday ago we had a > crash with one node. Cluster is build with 3 nodes HA on Windows machines - > RabbitMQ instances are services. This node has been reinstalled. After this > cluster failed to start. Now on other two machines we get information that > can not connect to other nodes. We tried set enviroment variable RABBITMQ_NODE_ONLY, > run service and executed command to remove node "rabbitmqctl > forget_cluster_node --offline xxx at nodeName". On running this command we get > information that node is not found "Error: {not_a_cluster_node,"The node > selected is not in the cluster."}" what to do with this? We don't want to > reinstall all. If we start it normally we get exception in log that: > > *"* > > *Error description: {error,{inconsistent_cluster,"Node rabbit at X thinks > it's clustered with node rabbit at Y, but rabbit at Y..."}}* > *"* > > Please help us. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jeff at ocjtech.us Fri Apr 25 18:42:56 2014 From: jeff at ocjtech.us (Jeffrey Ollie) Date: Fri, 25 Apr 2014 12:42:56 -0500 Subject: [rabbitmq-discuss] NC1/CS6 marking of RabbitMQ packets In-Reply-To: <1790225943.4428834.1398446936527.JavaMail.root@vmware.com> References: <1790225943.4428834.1398446936527.JavaMail.root@vmware.com> Message-ID: On Fri, Apr 25, 2014 at 12:28 PM, Kapil Goyal wrote: > > Is there any RabbitMQ configuration that accepts socket-options? The description of 'tcp_listen_options' says it accepts options for sockets. Can this be utilized? You'd have to ask someone with more Erlang experience that I have, but I doubt it. -- Jeff Ollie From bra at fsn.hu Fri Apr 25 21:15:48 2014 From: bra at fsn.hu (Attila Nagy) Date: Fri, 25 Apr 2014 22:15:48 +0200 Subject: [rabbitmq-discuss] Exclusive consumers via STOMP In-Reply-To: <535A6734.1080502@rabbitmq.com> References: <5358C9B5.9020305@fsn.hu> <5358FB50.4010006@fsn.hu> <535A6734.1080502@rabbitmq.com> Message-ID: <535AC274.2040403@fsn.hu> An HTML attachment was scrubbed... URL: From dubois.sebastien66 at gmail.com Fri Apr 25 22:34:22 2014 From: dubois.sebastien66 at gmail.com (Sebastien Dubois) Date: Fri, 25 Apr 2014 14:34:22 -0700 (PDT) Subject: [rabbitmq-discuss] New RabbitMQ 3.3.0 Web_stomp SSL problems Message-ID: Hi all, Up to recently we've been using RabbitMQ 3.2.3/Erlang R15B01 with the web_Stomp plugin over SSL. Since this was not supported, we had to rebuilt the web_stomp plugin using a patched branch as described at https://gist.github.com/berico-rclayton/5475365 (also see https://github.com/rabbitmq/rabbitmq-web-stomp/pull/3). This has been working fine for months. We recently realized that the fix for SSL support in web_stomp was systemized in RabbitMQ 3.3.0/Erlang R16B03. So we installed it and updated our configuration. However, using the exact same setup and self-signed keys/certificates, we cannot get the new version to work properly while using SSL over stomp. Connecting to https://:15678/stomp just fails, although from the log web_stomp seems to be listening correctly on port 15678. We tried to regenerate a new set of keys, but it did not do anything. Our old rabbitMQ config (patched 3.2.3 web_stomp) was: {rabbitmq_web_stomp, [ {ssl_enabled, true}, {https_port, 15678}, {ssl_key_file, "/usr/local/ssl/private/server.key"}, {ssl_key_password, "password"}, {ssl_ca_certificate_file, "/usr/local/ssl/crt/public.crt"}, {ssl_certificate_file, "/usr/local/ssl/crt/public.crt"} ] }, And the corresponding new config on RabbitMQ 3.3.0 is {rabbitmq_web_stomp, [{ssl_config, [{port, 15678}, {backlog, 1024}, {certfile, "/usr/local/ssl/crt/public.crt"}, {keyfile, "/usr/local/ssl/private/server.key"}, {cacertfile, "/usr/local/ssl/crt/public.crt"}, {password, "password"} ]} ]}, Is our config ok? does anybody experienced similar problems or have any idea what we could be doing wrong? Thanks in advance, /Sebas -------------- next part -------------- An HTML attachment was scrubbed... URL: From mklishin at gopivotal.com Fri Apr 25 22:43:29 2014 From: mklishin at gopivotal.com (Michael Klishin) Date: Sat, 26 Apr 2014 01:43:29 +0400 Subject: [rabbitmq-discuss] New RabbitMQ 3.3.0 Web_stomp SSL problems In-Reply-To: References: Message-ID: ?On 26 April 2014 at 01:36:37, Sebastien Dubois (dubois.sebastien66 at gmail.com) wrote: > Is our config ok?? Seems fine. > does anybody experienced similar problems > or have any idea what we could be doing wrong? And there is nothing in both RabbitMQ logs? Have you tried with a non-password protected key, just out of curiosity? -- MK Software Engineer, Pivotal/RabbitMQ From mklishin at gopivotal.com Fri Apr 25 22:45:04 2014 From: mklishin at gopivotal.com (Michael Klishin) Date: Sat, 26 Apr 2014 01:45:04 +0400 Subject: [rabbitmq-discuss] New RabbitMQ 3.3.0 Web_stomp SSL problems In-Reply-To: References: Message-ID: On 26 April 2014 at 01:36:37, Sebastien Dubois (dubois.sebastien66 at gmail.com) wrote: > > Connecting to https://:15678/stomp just > fails, although from the log web_stomp seems to be listening > correctly on port 15678(tel://port%2015678). We tried to regenerate > a new set of keys, but it did not do anything. Have you tried connecting with openssl s_client? -- MK Software Engineer, Pivotal/RabbitMQ From mklishin at gopivotal.com Fri Apr 25 23:04:53 2014 From: mklishin at gopivotal.com (Michael Klishin) Date: Sat, 26 Apr 2014 02:04:53 +0400 Subject: [rabbitmq-discuss] Recover cluster from crash In-Reply-To: <4bed6a8a-f896-4e2b-b981-907045e61d7e@googlegroups.com> References: <665d6495-ca13-465d-b741-142f5835a6f0@googlegroups.com> <4bed6a8a-f896-4e2b-b981-907045e61d7e@googlegroups.com> Message-ID: On 25 April 2014 at 21:44:05, Marek St?j (marek.stoj at gmail.com) wrote: > > Turns out the problem is that the environment variable RABBITMQ_NODE_ONLY > is not taken into account on Windows version of RabbitMQ server > (sic!). One can workaround this by manually removing ''-s rabbit > boot ^" line from rabbitmq-server.bat. FTR, Marek has submitted a pull request for this: https://github.com/rabbitmq/rabbitmq-server/pull/34 that we will take a look at soon. -- MK Software Engineer, Pivotal/RabbitMQ From goyalk at vmware.com Sat Apr 26 00:00:08 2014 From: goyalk at vmware.com (Kapil Goyal) Date: Fri, 25 Apr 2014 16:00:08 -0700 (PDT) Subject: [rabbitmq-discuss] NC1/CS6 marking of RabbitMQ packets In-Reply-To: References: <1790225943.4428834.1398446936527.JavaMail.root@vmware.com> Message-ID: <6167c5fc.00000e74.000000a5@KAPILPC.vmware.com> Setting 'tos' option to 192 in the 'tcp_listen_options' section worked out for me. Now packets sent by RMQ are marked with NC1. I also overrode the 'configureSocket' method of the ConnectionFactory class in RMQ Java client to add 'socket.setTrafficClass(int_value)', which is supposed to do the same thing as 'tos' does for RMQ. But, the packets going from my Java client to RMQ are not marked with NC1. Any suggestions? Thanks Kapil -----Original Message----- From: rabbitmq-discuss [mailto:rabbitmq-discuss-bounces at lists.rabbitmq.com] On Behalf Of Jeffrey Ollie Sent: Friday, April 25, 2014 10:43 AM To: Discussions about RabbitMQ Subject: Re: [rabbitmq-discuss] NC1/CS6 marking of RabbitMQ packets On Fri, Apr 25, 2014 at 12:28 PM, Kapil Goyal wrote: > > Is there any RabbitMQ configuration that accepts socket-options? The description of 'tcp_listen_options' says it accepts options for sockets. Can this be utilized? You'd have to ask someone with more Erlang experience that I have, but I doubt it. -- Jeff Ollie _______________________________________________ rabbitmq-discuss mailing list rabbitmq-discuss at lists.rabbitmq.com https://urldefense.proofpoint.com/v1/url?u=https://lists.rabbitmq.com/cgi- bin/mailman/listinfo/rabbitmq-discuss&k=oIvRg1%2BdGAgOoM1BIlLLqw%3D%3D%0A& r=JF6BIRiITniYYZhbhzlgNvj%2FeB0%2BYbRBSr0f8NG91qs%3D%0A&m=lZzjfRewdYFO96Ph iLJ5dqZ5GP4Quu%2FdHXThwlqGYHg%3D%0A&s=fb978ad865298d3081a9b88340e6a7047e22 46ba57996f7e73e1b623da0bcc7f From scott.kendall.mcfadden at gmail.com Fri Apr 25 23:55:51 2014 From: scott.kendall.mcfadden at gmail.com (Scott McFadden) Date: Fri, 25 Apr 2014 15:55:51 -0700 (PDT) Subject: [rabbitmq-discuss] RabbitMQ.Client.Exceptions.AuthenticationFailureException / ACCESS_REFUSED Message-ID: <59c66ed4-2b3a-4c6b-8464-39c7918e1a38@googlegroups.com> Was running RabbitMQ 3.3.0 for Windows but our server crashed. Recently moved rabbitmq to a hyperv windows virtual server and trying to get set back up. Trying to connect from client to new RabbitMqServer on hyper-v 2008 virtual server using guest/guest default credentials and get this error: None of the specified endpoints were reachable\nEndpoints attempted:\n------------------------------------------------\nendpoint=amqp-0-9://devtest01.acme.com:5672, attempts=1\nRabbitMQ.Client.Exceptions.AuthenticationFailureException: ACCESS_REFUSED - Login was refused using authentication mechanism PLAIN. For details see the broker logfile.\r\n at RabbitMQ.Client.Impl.ConnectionBase.StartAndTune()\r\n at RabbitMQ.Client.Framing.Impl.v0_9_1.Connection.Open(Boolean insist)\r\n at RabbitMQ.Client.Impl.ConnectionBase..ctor(ConnectionFactory factory, Boolean insist, IFrameHandler frameHandler)\r\n at RabbitMQ.Client.Framing.Impl.v0_9_1.Connection..ctor(ConnectionFactory factory, Boolean insist, IFrameHandler frameHandler)\r\n at RabbitMQ.Client.Framing.Impl.v0_9_1.ProtocolBase.CreateConnection(ConnectionFactory factory, Boolean insist, IFrameHandler frameHandler)\r\n at RabbitMQ.Client.ConnectionFactory.FollowRedirectChain(Int32 maxRedirects, IDictionary`2 connectionAttempts, IDictionary`2 co nnectionErrors, AmqpTcpEndpoint[]& mostRecentKnownHosts, AmqpTcpEndpoint endpoint)\n================================================\nStack trace:\n at RabbitMQ.Client.ConnectionFactory.CreateConnection(Int32 maxRedirects)\r\n at RabbitMQ.Client.ConnectionFactory.CreateConnection()\r\n at Client is using RabbitMQ.Client for .NET 3.2.4.0. Here is the .NET 4.51 code: var cf = new ConnectionFactory() { HostName = this.HostName, UserName = this.UserName, Password = this.Password }; m_connection = cf.CreateConnection(); m_model = m_connection.CreateModel(); How do you troubleshoot RabbitMQ connectivity issues for windows server / client? thanks scott -------------- next part -------------- An HTML attachment was scrubbed... URL: From scott.kendall.mcfadden at gmail.com Sat Apr 26 02:47:35 2014 From: scott.kendall.mcfadden at gmail.com (Scott McFadden) Date: Fri, 25 Apr 2014 18:47:35 -0700 (PDT) Subject: [rabbitmq-discuss] I am getting an Error while trying to send messages to a queue. "ACCESS_REFUSED - Login was refused using authentication mechanism PLAIN. For details see the broker logfile." In-Reply-To: <152f9b10-e3d4-4cb8-bef2-68dce4a56833@googlegroups.com> References: <152f9b10-e3d4-4cb8-bef2-68dce4a56833@googlegroups.com> Message-ID: Any progress? I may be experiencing something similar. On Thursday, April 24, 2014 6:10:45 AM UTC-5, AMAL G JOSE wrote: > > I have configured rabbitMQ in my laptop and wrote a sender and receiver > program. > It works fine, if I set the *factory.setHost(prop.getProperty("hostname")); > *property with* localhost.* > But when I am giving the ipadress, I am getting an error * "ACCESS_REFUSED > - Login was refused using authentication mechanism PLAIN. For details see > the broker logfile."* > I think this is due to some user permissions. Can anyone help me in > solving this issue. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From scott.kendall.mcfadden at gmail.com Sat Apr 26 03:21:48 2014 From: scott.kendall.mcfadden at gmail.com (Scott McFadden) Date: Fri, 25 Apr 2014 19:21:48 -0700 (PDT) Subject: [rabbitmq-discuss] RabbitMQ.Client.Exceptions.AuthenticationFailureException / ACCESS_REFUSED In-Reply-To: <59c66ed4-2b3a-4c6b-8464-39c7918e1a38@googlegroups.com> References: <59c66ed4-2b3a-4c6b-8464-39c7918e1a38@googlegroups.com> Message-ID: <08cb1b4f-1537-40a2-8309-90c4e3326379@googlegroups.com> Here are the permissions for default host. C:\Program Files (x86)\RabbitMQ Server\rabbitmq_server-3.3.0\sbin>rabbitmqctl.ba t list_permissions -p / Listing permissions in vhost "/" ... guest .* .* .* ...done. On Friday, April 25, 2014 5:55:51 PM UTC-5, Scott McFadden wrote: > > Was running RabbitMQ 3.3.0 for Windows but our server crashed. Recently > moved rabbitmq to a hyperv windows virtual server and trying to get set > back up. Trying to connect from client to new RabbitMqServer on hyper-v > 2008 virtual server using guest/guest default credentials and get this > error: > > > None of the specified endpoints were reachable\nEndpoints > attempted:\n------------------------------------------------\nendpoint=amqp-0-9:// > devtest01.acme.com:5672, > attempts=1\nRabbitMQ.Client.Exceptions.AuthenticationFailureException: > ACCESS_REFUSED - Login was refused using authentication mechanism PLAIN. > For details see the broker logfile.\r\n at > RabbitMQ.Client.Impl.ConnectionBase.StartAndTune()\r\n at > RabbitMQ.Client.Framing.Impl.v0_9_1.Connection.Open(Boolean insist)\r\n > at RabbitMQ.Client.Impl.ConnectionBase..ctor(ConnectionFactory factory, > Boolean insist, IFrameHandler frameHandler)\r\n at > RabbitMQ.Client.Framing.Impl.v0_9_1.Connection..ctor(ConnectionFactory > factory, Boolean insist, IFrameHandler frameHandler)\r\n at > RabbitMQ.Client.Framing.Impl.v0_9_1.ProtocolBase.CreateConnection(ConnectionFactory > factory, Boolean insist, IFrameHandler frameHandler)\r\n at > RabbitMQ.Client.ConnectionFactory.FollowRedirectChain(Int32 maxRedirects, > IDictionary`2 connectionAttempts, IDictionary`2 co > nnectionErrors, AmqpTcpEndpoint[]& mostRecentKnownHosts, AmqpTcpEndpoint > endpoint)\n================================================\nStack trace:\n > at RabbitMQ.Client.ConnectionFactory.CreateConnection(Int32 > maxRedirects)\r\n at > RabbitMQ.Client.ConnectionFactory.CreateConnection()\r\n at > > Client is using RabbitMQ.Client for .NET 3.2.4.0. > > Here is the .NET 4.51 code: > > var cf = new ConnectionFactory() { HostName = this.HostName, UserName = > this.UserName, Password = this.Password }; > m_connection = cf.CreateConnection(); > m_model = m_connection.CreateModel(); > > How do you troubleshoot RabbitMQ connectivity issues for windows server / > client? > > > thanks > > > scott > -------------- next part -------------- An HTML attachment was scrubbed... URL: From scott.kendall.mcfadden at gmail.com Sat Apr 26 05:29:51 2014 From: scott.kendall.mcfadden at gmail.com (Scott McFadden) Date: Fri, 25 Apr 2014 21:29:51 -0700 (PDT) Subject: [rabbitmq-discuss] RabbitMQ.Client.Exceptions.AuthenticationFailureException / ACCESS_REFUSED In-Reply-To: <59c66ed4-2b3a-4c6b-8464-39c7918e1a38@googlegroups.com> References: <59c66ed4-2b3a-4c6b-8464-39c7918e1a38@googlegroups.com> Message-ID: Not so quick update. Tried looking at broker log file on server but couldn't find it in default locations specified by documentation. Ended up setting a custom WINDOWS environment variable RABBITMQ_LOG_BASE to a custom folder I created. Had to uninstall rabbitmq service / reinstall rabbit mq service to get it to finally use my custom log folder. Once that happened I was able to glean the following useful info from the log file: =ERROR REPORT==== 25-Apr-2014::23:19:49 === closing AMQP connection <0.278.0> (172.17.9.14:61333 -> 172.17.9.24:5672): {handshake_error,starting,0, {amqp_error,access_refused, "PLAIN login refused: user 'guest' can only connect via localhost", 'connection.start_ok'}} So apparently RabbitMQ no longer allows guest login outside of localhost. Our previous version of RabbitMQ did allow this. Apparently this was a breaking change some where along the way. -------------- next part -------------- An HTML attachment was scrubbed... URL: From scott.kendall.mcfadden at gmail.com Sat Apr 26 06:27:36 2014 From: scott.kendall.mcfadden at gmail.com (Scott McFadden) Date: Fri, 25 Apr 2014 22:27:36 -0700 (PDT) Subject: [rabbitmq-discuss] I am getting an Error while trying to send messages to a queue. "ACCESS_REFUSED - Login was refused using authentication mechanism PLAIN. For details see the broker logfile." In-Reply-To: <152f9b10-e3d4-4cb8-bef2-68dce4a56833@googlegroups.com> References: <152f9b10-e3d4-4cb8-bef2-68dce4a56833@googlegroups.com> Message-ID: <5214f6eb-a4f9-4b9f-a214-e63fe02b3751@googlegroups.com> Go look at the server log file. If you are using guest/guest, latest RabbitMQ no longer allows guest to connect to anything other than localhost. On Thursday, April 24, 2014 6:10:45 AM UTC-5, AMAL G JOSE wrote: > > I have configured rabbitMQ in my laptop and wrote a sender and receiver > program. > It works fine, if I set the *factory.setHost(prop.getProperty("hostname")); > *property with* localhost.* > But when I am giving the ipadress, I am getting an error * "ACCESS_REFUSED > - Login was refused using authentication mechanism PLAIN. For details see > the broker logfile."* > I think this is due to some user permissions. Can anyone help me in > solving this issue. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From brett.r.cameron at gmail.com Sat Apr 26 09:44:30 2014 From: brett.r.cameron at gmail.com (Brett Cameron) Date: Sat, 26 Apr 2014 20:44:30 +1200 Subject: [rabbitmq-discuss] RabbitMQ.Client.Exceptions.AuthenticationFailureException / ACCESS_REFUSED In-Reply-To: References: <59c66ed4-2b3a-4c6b-8464-39c7918e1a38@googlegroups.com> Message-ID: Scott, See http://www.rabbitmq.com/access-control.html Brett On Sat, Apr 26, 2014 at 4:29 PM, Scott McFadden < scott.kendall.mcfadden at gmail.com> wrote: > Not so quick update. Tried looking at broker log file on server but > couldn't find it in default locations specified by documentation. Ended up > setting a custom WINDOWS environment variable RABBITMQ_LOG_BASE to a custom > folder I created. Had to uninstall rabbitmq service / reinstall rabbit mq > service to get it to finally use my custom log folder. Once that happened > I was able to glean the following useful info from the log file: > > =ERROR REPORT==== 25-Apr-2014::23:19:49 === > closing AMQP connection <0.278.0> (172.17.9.14:61333 -> 172.17.9.24:5672): > {handshake_error,starting,0, > {amqp_error,access_refused, > "PLAIN login refused: user 'guest' can only > connect via localhost", > 'connection.start_ok'}} > > So apparently RabbitMQ no longer allows guest login outside of localhost. > Our previous version of RabbitMQ did allow this. Apparently this was a > breaking change some where along the way. > > > > _______________________________________________ > rabbitmq-discuss mailing list > rabbitmq-discuss at lists.rabbitmq.com > https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From senthilbasuvaraj at gmail.com Sat Apr 26 10:41:21 2014 From: senthilbasuvaraj at gmail.com (Senthil Basuva Raj) Date: Sat, 26 Apr 2014 15:11:21 +0530 Subject: [rabbitmq-discuss] ERROR: epmd error for host INLN50800100A: address (cannot connect to host/port) Message-ID: Hi Forum, I installed erlang and rabbitmq. First time, when I installed, I was able to run rabbitmq and use it successfully. But i restarted again today, but I am getting the following error. ERROR: epmd error for host INLN50800100A: address (cannot connect to host/port) Can you suggest, what I am missing. I could see the following error in rabbitmq logs =INFO REPORT==== 25-Apr-2014::07:12:11 === Error description: {could_not_start,rabbit, {bad_return, {{rabbit,start,[normal,[]]}, {'EXIT',{rabbit,failure_during_boot,{badmatch,noport}}}}}} Log files (may contain more information): C:/Users/I048548/AppData/Roaming/RabbitMQ/log/rabbit at INLN50800100A.log C:/Users/I048548/AppData/Roaming/RabbitMQ/log/rabbit at INLN50800100A-sasl.log Thanks, Senti. -------------- next part -------------- An HTML attachment was scrubbed... URL: From karthikeyan.manoharan at barclays.com Sat Apr 26 11:09:59 2014 From: karthikeyan.manoharan at barclays.com (karthikeyan.manoharan at barclays.com) Date: Sat, 26 Apr 2014 11:09:59 +0100 Subject: [rabbitmq-discuss] RabbitMQ.Client.Exceptions.AuthenticationFailureException / ACCESS_REFUSED In-Reply-To: Message-ID: Nearly more than Lakhs of mails . Still not stable. I'm fan of rabbit MQ. When will this come stable with proper intructions? Why is it not preffered over IBM MQ? Thanks Karthik From: Scott McFadden [mailto:scott.kendall.mcfadden at gmail.com] Sent: Saturday, April 26, 2014 09:59 AM To: rabbitmq-discuss at googlegroups.com Subject: Re: [rabbitmq-discuss] RabbitMQ.Client.Exceptions.AuthenticationFailureException / ACCESS_REFUSED Not so quick update. Tried looking at broker log file on server but couldn't find it in default locations specified by documentation. Ended up setting a custom WINDOWS environment variable RABBITMQ_LOG_BASE to a custom folder I created. Had to uninstall rabbitmq service / reinstall rabbit mq service to get it to finally use my custom log folder. Once that happened I was able to glean the following useful info from the log file: =ERROR REPORT==== 25-Apr-2014::23:19:49 === closing AMQP connection <0.278.0> (172.17.9.14:61333 -> 172.17.9.24:5672): {handshake_error,starting,0, {amqp_error,access_refused, "PLAIN login refused: user 'guest' can only connect via localhost", 'connection.start_ok'}} So apparently RabbitMQ no longer allows guest login outside of localhost. Our previous version of RabbitMQ did allow this. Apparently this was a breaking change some where along the way. This e-mail and any attachments are confidential and intended solely for the addressee and may also be privileged or exempt from disclosure under applicable law. If you are not the addressee, or have received this e-mail in error, please notify the sender immediately, delete it from your system and do not copy, disclose or otherwise act upon any part of this e-mail or its attachments. Internet communications are not guaranteed to be secure or virus-free. The Barclays Group does not accept responsibility for any loss arising from unauthorised access to, or interference with, any Internet communications by any third party, or from the transmission of any viruses. Replies to this e-mail may be monitored by the Barclays Group for operational or business reasons. Any opinion or other information in this e-mail or its attachments that does not relate to the business of the Barclays Group is personal to the sender and is not given or endorsed by the Barclays Group. Barclays Bank PLC. Registered in England and Wales (registered no. 1026167). Registered Office: 1 Churchill Place, London, E14 5HP, United Kingdom. Barclays Bank PLC is authorised by the Prudential Regulation Authority and regulated by the Financial Conduct Authority and the Prudential Regulation Authority (Financial Services Register No. 122702). -------------- next part -------------- An HTML attachment was scrubbed... URL: From mklishin at gopivotal.com Sat Apr 26 11:57:32 2014 From: mklishin at gopivotal.com (Michael Klishin) Date: Sat, 26 Apr 2014 14:57:32 +0400 Subject: [rabbitmq-discuss] RabbitMQ.Client.Exceptions.AuthenticationFailureException / ACCESS_REFUSED In-Reply-To: References: Message-ID: ?On 26 April 2014 at 14:13:36, karthikeyan.manoharan at barclays.com (karthikeyan.manoharan at barclays.com) wrote: > > Still not stable. Can you be more specific? This thread is about limiting the guest user to localhost, which is a sensible security measure. -- MK Software Engineer, Pivotal/RabbitMQ From mklishin at gopivotal.com Sat Apr 26 11:58:47 2014 From: mklishin at gopivotal.com (Michael Klishin) Date: Sat, 26 Apr 2014 14:58:47 +0400 Subject: [rabbitmq-discuss] ERROR: epmd error for host INLN50800100A: address (cannot connect to host/port) In-Reply-To: References: Message-ID: ?On 26 April 2014 at 13:44:03, Senthil Basuva Raj (senthilbasuvaraj at gmail.com) wrote: > > ERROR: epmd error for host INLN50800100A: address (cannot > connect to host/port) Make sure that host?INLN50800100A resolves from the machine itself and port 4369 is not blocked by a firewall. -- MK Software Engineer, Pivotal/RabbitMQ From mklishin at gopivotal.com Sat Apr 26 12:00:19 2014 From: mklishin at gopivotal.com (Michael Klishin) Date: Sat, 26 Apr 2014 15:00:19 +0400 Subject: [rabbitmq-discuss] I am getting an Error while trying to send messages to a queue. "ACCESS_REFUSED - Login was refused using authentication mechanism PLAIN. For details see the broker logfile." In-Reply-To: <5214f6eb-a4f9-4b9f-a214-e63fe02b3751@googlegroups.com> References: <152f9b10-e3d4-4cb8-bef2-68dce4a56833@googlegroups.com> <5214f6eb-a4f9-4b9f-a214-e63fe02b3751@googlegroups.com> Message-ID: On 26 April 2014 at 12:36:16, Scott McFadden (scott.kendall.mcfadden at gmail.com) wrote: > > Go look at the server log file. If you are using guest/guest, > latest RabbitMQ no longer allows guest to connect to anything > other than localhost. Correct, which is mentioned early in the breaking changes blog post: http://www.rabbitmq.com/blog/2014/04/02/breaking-things-with-rabbitmq-3-3/ The old behaviour also can be re-enabled: http://www.rabbitmq.com/access-control.html? -- MK Software Engineer, Pivotal/RabbitMQ From senthilbasuvaraj at gmail.com Sat Apr 26 12:17:21 2014 From: senthilbasuvaraj at gmail.com (Senthil Basuva Raj) Date: Sat, 26 Apr 2014 16:47:21 +0530 Subject: [rabbitmq-discuss] |Spam| Re: ERROR: epmd error for host INLN50800100A: address (cannot connect to host/port) In-Reply-To: References: Message-ID: Hi Micheal, I tried the same also, I have added the port open in my firewall settings.please find the screen shot. Please suggest, if I am missing anything more than this. [image: Inline image 1] Thanks, Senti On Sat, Apr 26, 2014 at 4:28 PM, Michael Klishin wrote: > On 26 April 2014 at 13:44:03, Senthil Basuva Raj ( > senthilbasuvaraj at gmail.com) wrote: > > > ERROR: epmd error for host INLN50800100A: address (cannot > > connect to host/port) > > Make sure that host INLN50800100A resolves from the machine itself and port > 4369 is not blocked by a firewall. > -- > MK > > Software Engineer, Pivotal/RabbitMQ > -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image.png Type: image/png Size: 84166 bytes Desc: not available URL: From stammailbox at gmail.com Sat Apr 26 15:47:13 2014 From: stammailbox at gmail.com (rails) Date: Sat, 26 Apr 2014 17:47:13 +0300 Subject: [rabbitmq-discuss] =?utf-8?q?RabbitMQ_-_an_example_for_a_?= =?utf-8?b?4oCcTmFja+KAnSBuZWVk?= Message-ID: When do I need to respond with a nack on my RabbitMQ listener? So far I have never witness an error in the message over the network. I guess the network layer should take care of it, no? If I get an error message from the publisher a nack will just leave it in the queue, wont it? Thanks. -------------- next part -------------- An HTML attachment was scrubbed... URL: From mklishin at gopivotal.com Sat Apr 26 15:56:04 2014 From: mklishin at gopivotal.com (Michael Klishin) Date: Sat, 26 Apr 2014 18:56:04 +0400 Subject: [rabbitmq-discuss] RabbitMQ - an example for a =?utf-8?Q?=E2=80=9CNack=E2=80=9D_?=need In-Reply-To: References: Message-ID: On April 26, 2014 at 6:49:27 PM, rails (stammailbox at gmail.com) wrote: > > > When do I need to respond with a nack on my RabbitMQ listener? > So far I have never witness an error in the message over the network. See?http://markmail.org/thread/ayb6j3bwmh3zsww6 > I guess the network layer should take care of it, no? Network layer cannot possibly take care of an issue with RabbitMQ node. > If I get an error message from the publisher a nack will just leave > it in the queue, wont it? Are you talking about publisher confirms or consumer ones? With publisher confirms something can go wrong before a message is routed to a queue, or it can route nowhere.? -- MK From mklishin at gopivotal.com Sat Apr 26 15:58:42 2014 From: mklishin at gopivotal.com (Michael Klishin) Date: Sat, 26 Apr 2014 18:58:42 +0400 Subject: [rabbitmq-discuss] New RabbitMQ 3.3.0 Web_stomp SSL problems In-Reply-To: References: Message-ID: please keep rabbitmq-discuss in CC.? On April 26, 2014 at 3:52:44 AM, Sebastien Dubois (dubois.sebastien66 at gmail.com) wrote: > > No we did not,? openssl s_client should be one of the first tools you use in investigating SSL-related issues. > However our setup used to work before on the patched > 3.2.3 as I said. The only difference being rabbitMQ and Erlang > versions. Sorry but your patched plugin never shipped with RabbitMQ so I?m not sure how this is relevant to investigating an SSL problem with 3.3.0. -- MK Software Engineer, Pivotal/RabbitMQ From sagu.prf1 at gmail.com Sun Apr 27 05:10:02 2014 From: sagu.prf1 at gmail.com (sagu prf) Date: Sat, 26 Apr 2014 21:10:02 -0700 Subject: [rabbitmq-discuss] 3 node rabbitmq cluster behind the load balancer Message-ID: Hello Team, How effieciently, we can setup 6 node rabbitmq cluster with HA mirror queues behinde the Load balancer? Loadbalancer will place the publishing and consuming request across the node and how does the rabbitmq works ? 6 reals (rabbitmq nodes) are configured in NS VIP ) Regards Sagu From mklishin at gopivotal.com Sun Apr 27 12:02:39 2014 From: mklishin at gopivotal.com (Michael Klishin) Date: Sun, 27 Apr 2014 15:02:39 +0400 Subject: [rabbitmq-discuss] 3 node rabbitmq cluster behind the load balancer In-Reply-To: References: Message-ID: On 27 April 2014 at 08:12:14, sagu prf (sagu.prf1 at gmail.com) wrote: > > How effieciently, we can setup 6 node rabbitmq cluster with > HA mirror > queues behinde the Load balancer? It depends on what exactly you mean by ?efficiently? but one good idea in a 6 node cluster is to not use ha-mode = ?all? for your queues. Use ha-mode = ?exactly? and mirror to 2 or 3 nodes (depending on how valuable their contents is for your system). Mirroring to more nodes will only make RabbitMQ do more work and increase intra-cluster traffic, at no real benefit for the described setup. See?http://www.rabbitmq.com/ha.html. > Loadbalancer will place the publishing and consuming request > across > the node and how does the rabbitmq works ? In a RabbitMQ cluster, clients can connect to and work with any node. However, queues have a master node that all requests are routed to. You will reduce intra-cluster traffic and thus improve throughput and reduce latency if you publish on a node that hosts a cluster for the queue(s) of interest. The same is true for consumers. There are multiple load balancing strategies available in modern tools but they are unaware of the above point. Which brings us to my (very general) recommendation: if you don?t have extreme throughput requirements, you can use virtually any load balancing strategy.? With HAproxy, leastconn [1] is a good idea if you expect a lot of client connections. If you do have extreme throughput requirements, you need to infer queue masters via HTTP API and make your clients connect to master nodes, and reconnect when it fails. 1.?http://haproxy.1wt.eu/download/1.5/doc/configuration.txt, search for ?balance " -- MK Software Engineer, Pivotal/RabbitMQ From scott.kendall.mcfadden at gmail.com Sat Apr 26 14:30:41 2014 From: scott.kendall.mcfadden at gmail.com (Scott McFadden) Date: Sat, 26 Apr 2014 06:30:41 -0700 (PDT) Subject: [rabbitmq-discuss] Cross rabbitmq server message routing? Message-ID: <54aeb918-b35c-409f-9738-4620d46a880f@googlegroups.com> Is there a good way to automatically route messages from Rabbit Server A / Queue A to Rabbit Server B / Queue B scott -------------- next part -------------- An HTML attachment was scrubbed... URL: From firesimon at googlemail.com Sun Apr 27 15:42:30 2014 From: firesimon at googlemail.com (kein namen) Date: Sun, 27 Apr 2014 07:42:30 -0700 (PDT) Subject: [rabbitmq-discuss] Server Clustering Message-ID: Hello, i have a question about Clustering RMQ Servers. I want to have multiple RMQ Servers that i can switch on and off if i want. But if there more than one server running in the LAN they should syncronize itself. I think clustering is the right setup. My question is how i can setup my client to use the right ip? factory.setHost("192.168.0.51"); At the moment there a running two instances with the same hostname: root at raspberrypi:/etc/rabbitmq# rabbitmqctl cluster_status > Cluster status of node rabbit at raspberrypi ... > Error: unable to connect to node rabbit at raspberrypi: nodedown > > DIAGNOSTICS > =========== > > nodes in question: [rabbit at raspberrypi] > > hosts, their running nodes and ports: > - raspberrypi: [{rabbitmqctl3224,39192}] > > current node details: > - node name: rabbitmqctl3224 at raspberrypi > - home dir: /var/lib/rabbitmq > - cookie hash: k3Y8zd/RVr1yZFvY2SoRPQ== > AND root at raspberrypi:/etc/rabbitmq# rabbitmqctl cluster_status Cluster status of node rabbit at raspberrypi ... Error: unable to connect to node rabbit at raspberrypi: nodedown DIAGNOSTICS =========== nodes in question: [rabbit at raspberrypi] hosts, their running nodes and ports: - raspberrypi: [{rabbitmqctl2964,40862}] current node details: - node name: rabbitmqctl2964 at raspberrypi - home dir: /var/lib/rabbitmq - cookie hash: hEhcDC1Mlicw6A9QfMhtaw== -------------- next part -------------- An HTML attachment was scrubbed... URL: From mklishin at gopivotal.com Sun Apr 27 17:19:55 2014 From: mklishin at gopivotal.com (Michael Klishin) Date: Sun, 27 Apr 2014 20:19:55 +0400 Subject: [rabbitmq-discuss] Cross rabbitmq server message routing? In-Reply-To: <54aeb918-b35c-409f-9738-4620d46a880f@googlegroups.com> References: <54aeb918-b35c-409f-9738-4620d46a880f@googlegroups.com> Message-ID: On 27 April 2014 at 20:18:54, Scott McFadden (scott.kendall.mcfadden at gmail.com) wrote: > > Is there a good way to automatically route messages from Rabbit > Server A / Queue A to Rabbit Server B / Queue B Shovel and [exchange] federation: http://www.rabbitmq.com/shovel.html http://www.rabbitmq.com/federation.html? -- MK Software Engineer, Pivotal/RabbitMQ From mklishin at gopivotal.com Sun Apr 27 17:23:22 2014 From: mklishin at gopivotal.com (Michael Klishin) Date: Sun, 27 Apr 2014 20:23:22 +0400 Subject: [rabbitmq-discuss] Server Clustering In-Reply-To: References: Message-ID: On 27 April 2014 at 20:21:00, kein namen (firesimon at googlemail.com) wrote: > > i have a question about Clustering RMQ Servers. I want to have > multiple RMQ Servers that i can switch on and off if i want. But > if there more than one server running in the LAN they should syncronize > itself. I think clustering is the right setup. > > My question is how i can setup my client to use the right ip? > > factory.setHost("192.168.0.51?); See Advanced Connection Options: http://www.rabbitmq.com/api-guide.html -- MK Software Engineer, Pivotal/RabbitMQ From qiujixiong at 163.com Mon Apr 28 03:38:52 2014 From: qiujixiong at 163.com (qiujixiong) Date: Mon, 28 Apr 2014 10:38:52 +0800 (CST) Subject: [rabbitmq-discuss] Why rabbitmq stop pushing message to consumer In-Reply-To: References: <6ed96fd1.d2d0.1459825dc8e.Coremail.qiujixiong@163.com> Message-ID: <48c50037.8ab2.145a63302e2.Coremail.qiujixiong@163.com> Hi Michael, Thanks for your response very much! There indeed were some RAM alarms logged while no disk alarm. The RAM alarms were raised after message accumulate for a long time. It should be the result, not the reason, right?(before the alarms raised, consumers could not receive message already) What I can saw at that time was that all consumer threads could not receive message from rabbitmq and gdb tell me they were blocked in tcp receive function. The connection usage currently is not very well and we will enhance it later. What I confused is the blocking of consumer(as you said, they should not be blocked). The reason is still uncertain. At 2014-04-25 19:58:36,"Michael Klishin" wrote: > >Resource-based flow control should only block publishers. Re-creating connections >won?t help: every publishing connection will be blocked. Consumers should not >be blocked but nearly running out of disk space sounds alarming and can prevent >RabbitMQ from moving messages to disk or very heavy OS paging . > >What?s in RabbitMQ log? There should be disk and RAM alarms logged. rabbitmqctl status >indicates disk limit alarm should be on. >-- >MK > >Software Engineer, Pivotal/RabbitMQ -------------- next part -------------- An HTML attachment was scrubbed... URL: From dubois.sebastien66 at gmail.com Mon Apr 28 04:08:02 2014 From: dubois.sebastien66 at gmail.com (Sebastien Dubois) Date: Sun, 27 Apr 2014 23:08:02 -0400 Subject: [rabbitmq-discuss] New RabbitMQ 3.3.0 Web_stomp SSL problems In-Reply-To: References: Message-ID: > openssl s_client should be one of the first tools you use in investigating SSL-related issues. Fair enough. We'll do more tests here. It's just that the patched solution (over 3.2.4) was working so well that we did not expect any problems with the systemized solution. Since you say that our config seems fine, we'll investigate more here > Sorry but your patched plugin never shipped with RabbitMQ so I?m not sure how > this is relevant to investigating an SSL problem with 3.3.0. We were using jshiell's solution, which in my understanding was the patch that was systemized in RMQ3.3.0. Best Regards, /Sebastien On Sat, Apr 26, 2014 at 10:58 AM, Michael Klishin wrote: > please keep rabbitmq-discuss in CC. > > On April 26, 2014 at 3:52:44 AM, Sebastien Dubois ( > dubois.sebastien66 at gmail.com) wrote: > > > No we did not, > > openssl s_client should be one of the first tools you use in investigating > SSL-related issues. > > > However our setup used to work before on the patched > > 3.2.3 as I said. The only difference being rabbitMQ and Erlang > > versions. > > Sorry but your patched plugin never shipped with RabbitMQ so I?m not sure > how > this is relevant to investigating an SSL problem with 3.3.0. > -- > MK > > Software Engineer, Pivotal/RabbitMQ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From loganathan.ms at gmail.com Mon Apr 28 08:41:59 2014 From: loganathan.ms at gmail.com (Loganathan Sellapa) Date: Mon, 28 Apr 2014 13:11:59 +0530 Subject: [rabbitmq-discuss] [RUBY-AMQP GEM] How do we can retry publish message on errors? Message-ID: Hi All, I would like to know how the error handling is done for "publish message". I want to retry the publish message when some exception/failure occurred on RabbitMq? I checked the examples on ruby amqp gem but unable to find the proper example for republishing the message on error. regards, Loganathan Mob: +91 7760780741 | +91 9944414388 Skype: loganathan.sellappa ViewMe -------------- next part -------------- An HTML attachment was scrubbed... URL: From subshiri at gmail.com Mon Apr 28 08:30:29 2014 From: subshiri at gmail.com (Subshiri S) Date: Mon, 28 Apr 2014 00:30:29 -0700 (PDT) Subject: [rabbitmq-discuss] How to stop consuming message from selective queue Message-ID: <11b16a32-75dc-4265-91d6-b3455351d0a8@googlegroups.com> > QueueingConsumer consumer = new QueueingConsumer(channel); > channel.basicConsume("queue1", consumer); > channel.basicConsume("queue3", consumer); Is it possible to stop consuming the messages from the queue "queue3" alone dynamically? -------------- next part -------------- An HTML attachment was scrubbed... URL: From subshiri at gmail.com Mon Apr 28 08:41:41 2014 From: subshiri at gmail.com (Subshiri S) Date: Mon, 28 Apr 2014 00:41:41 -0700 (PDT) Subject: [rabbitmq-discuss] Need a design suggestion - Limited Consumer handling Message-ID: <8f6881f3-b20d-4f3f-b6d7-e462a4139a02@googlegroups.com> Hi, I am a newbie. I would like to know , is it possible to design as mentioned below. For eg, There are 100 queues and 10 threads to consume messages from that 100 queue. Each thread will be consuming messages from 10 queue each. *Question 1 :* *How to dynamically assign the threads to queues ?* No more than one thread should consume from a queue (to maintain the order of processing the message in the respective queue) *Question 2 :* *When there is a need to increase the consumer threads while the system runs, How it can be done ?.* Thank you. Subshiri. -------------- next part -------------- An HTML attachment was scrubbed... URL: From videlalvaro at gmail.com Mon Apr 28 09:02:38 2014 From: videlalvaro at gmail.com (Alvaro Videla) Date: Mon, 28 Apr 2014 10:02:38 +0200 Subject: [rabbitmq-discuss] [RUBY-AMQP GEM] How do we can retry publish message on errors? In-Reply-To: References: Message-ID: Hi, Have you seen publisher confirms? https://www.rabbitmq.com/confirms.html Regards, Alvaro On Mon, Apr 28, 2014 at 9:41 AM, Loganathan Sellapa wrote: > Hi All, > > I would like to know how the error handling is done for "publish message". I > want to retry the publish message when some exception/failure occurred on > RabbitMq? I checked the examples on ruby amqp gem but unable to find the > proper example for republishing the message on error. > > regards, > Loganathan > Mob: +91 7760780741 | +91 9944414388 > Skype: loganathan.sellappa > ViewMe > > > _______________________________________________ > rabbitmq-discuss mailing list > rabbitmq-discuss at lists.rabbitmq.com > https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss > From videlalvaro at gmail.com Mon Apr 28 09:05:36 2014 From: videlalvaro at gmail.com (Alvaro Videla) Date: Mon, 28 Apr 2014 10:05:36 +0200 Subject: [rabbitmq-discuss] How to stop consuming message from selective queue In-Reply-To: <11b16a32-75dc-4265-91d6-b3455351d0a8@googlegroups.com> References: <11b16a32-75dc-4265-91d6-b3455351d0a8@googlegroups.com> Message-ID: Hi, Yes, you can use basicCancel for a specific consumer. See here: https://www.rabbitmq.com/amqp-0-9-1-quickref.html Regards, Alvaro On Mon, Apr 28, 2014 at 9:30 AM, Subshiri S wrote: > >> >> QueueingConsumer consumer = new QueueingConsumer(channel); >> channel.basicConsume("queue1", consumer); >> channel.basicConsume("queue3", consumer); > > > Is it possible to stop consuming the messages from the queue "queue3" alone > dynamically? > > _______________________________________________ > rabbitmq-discuss mailing list > rabbitmq-discuss at lists.rabbitmq.com > https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss > From bra at fsn.hu Mon Apr 28 09:22:15 2014 From: bra at fsn.hu (Nagy, Attila) Date: Mon, 28 Apr 2014 10:22:15 +0200 Subject: [rabbitmq-discuss] LIFO queue In-Reply-To: <4BCC459C.50303@250bpm.com> References: <4BCC459C.50303@250bpm.com> Message-ID: <535E0FB7.3040708@fsn.hu> Hi, On 04/19/10 13:59, Martin Sustrik wrote: >> Hi, I'm just learning about messaging and I need to create a LIFO >> message queue. I looked into it with ApacheMQ and it is apparently not >> easily possible. >> >> Does AMQP/RabbitMQ support LIFO / FILO? > No. There's no such thing in AMQP. You won't find it even in non-AMQP > solutions. > > It would be extremely interesting to know a bit more about your use case > though. > Although the message is quite old -and unanswered-, I can tell you about my case: I collect RADIUS accounting messages via queues for maintaining historic data (who used what port/IP) and actual data (who are using them now) too. Here, data with growing age becomes of less importance in the meaning of how quickly it gets processed. If the queue is FIFO and contains multi million records (due to something going bad) and processing it takes a half day, it means for a half day, I won't be able to serve actual session data, because they are at the tail of the queue, while the processing still happens on its head, which is possibly history now. With a LIFO queue, if the queue grows, history gets into the background and actual data in the foreground. Meaning: the actual data can be served with high possibility, while historic data will be eventually processed, which is the desired effect. Hence, I would need a LIFO queue in RabbitMQ too. :) From subshiri at gmail.com Mon Apr 28 09:50:40 2014 From: subshiri at gmail.com (Subshiri S) Date: Mon, 28 Apr 2014 01:50:40 -0700 (PDT) Subject: [rabbitmq-discuss] How to stop consuming message from selective queue In-Reply-To: References: <11b16a32-75dc-4265-91d6-b3455351d0a8@googlegroups.com> Message-ID: <82df2aea-1fd8-48e3-84a6-8f5ac6fb30ed@googlegroups.com> Thank you Alvaro. Here is the code QueueingConsumer consumer = new QueueingConsumer(channel); channel.basicConsume("queue1", consumer); channel.basicConsume("queue3", consumer); while(true){ Delivery delivery = consumer.nextDelivery(); String msg = new String(delivery.getBody()); channel.basicAck(delivery.getEnvelope().getDeliveryTag(), true); } I am using the same consumer object to retrieve the msgs from the queues. If I use the basicCancel, I cannot process the msg from other queues as well. Please correct me If am wrong. If I create new consumer for every queue and consume through the same channel. How do I retrieve the deliveries from more than one consumer. Please refer the code. On Monday, April 28, 2014 1:35:36 PM UTC+5:30, Alvaro Videla wrote: > > Hi, > > Yes, you can use basicCancel for a specific consumer. See here: > https://www.rabbitmq.com/amqp-0-9-1-quickref.html > > Regards, > > Alvaro > > On Mon, Apr 28, 2014 at 9:30 AM, Subshiri S > > wrote: > > > >> > >> QueueingConsumer consumer = new QueueingConsumer(channel); > >> channel.basicConsume("queue1", consumer); > >> channel.basicConsume("queue3", consumer); > > > > > > Is it possible to stop consuming the messages from the queue "queue3" > alone > > dynamically? > > > > _______________________________________________ > > rabbitmq-discuss mailing list > > rabbitmq... at lists.rabbitmq.com > > https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From subshiri at gmail.com Mon Apr 28 10:07:16 2014 From: subshiri at gmail.com (Subshiri S) Date: Mon, 28 Apr 2014 02:07:16 -0700 (PDT) Subject: [rabbitmq-discuss] How to stop consuming message from selective queue In-Reply-To: References: <11b16a32-75dc-4265-91d6-b3455351d0a8@googlegroups.com> Message-ID: Yes, I got it. the basicConsume() method returns the consumer tag corresponding to the queue. So I can call the basicCancel(consumertag) to unsubscribe from that queue alone. String consumerTag1 = *channel*.basicConsume("queue1", consumer); String consumerTag2= *channel*.basicConsume("queue3", consumer); *channel*.basicCancel(consumerTag2); Thank you. Subshiri On Monday, April 28, 2014 1:35:36 PM UTC+5:30, Alvaro Videla wrote: > > Hi, > > Yes, you can use basicCancel for a specific consumer. See here: > https://www.rabbitmq.com/amqp-0-9-1-quickref.html > > Regards, > > Alvaro > > On Mon, Apr 28, 2014 at 9:30 AM, Subshiri S > > wrote: > > > >> > >> QueueingConsumer consumer = new QueueingConsumer(channel); > >> channel.basicConsume("queue1", consumer); > >> channel.basicConsume("queue3", consumer); > > > > > > Is it possible to stop consuming the messages from the queue "queue3" > alone > > dynamically? > > > > _______________________________________________ > > rabbitmq-discuss mailing list > > rabbitmq... at lists.rabbitmq.com > > https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From haster2004 at yandex.ru Mon Apr 28 11:20:04 2014 From: haster2004 at yandex.ru (Haster) Date: Mon, 28 Apr 2014 03:20:04 -0700 (PDT) Subject: [rabbitmq-discuss] [rabbitmq_c] Infinite wait in login_with_properties function In-Reply-To: References: <1387786326818-32275.post@n5.nabble.com> <1388037935031-32306.post@n5.nabble.com> Message-ID: <1398680404528-35118.post@n5.nabble.com> Alan, But is it possible to limit such timeout or use nonblocked version of recv? And can you add this functionality to rabbitmq_c library? -- View this message in context: http://rabbitmq.1065348.n5.nabble.com/rabbitmq-c-Infinite-wait-in-login-with-properties-function-tp32275p35118.html Sent from the RabbitMQ mailing list archive at Nabble.com. From mklishin at gopivotal.com Mon Apr 28 11:23:26 2014 From: mklishin at gopivotal.com (Michael Klishin) Date: Mon, 28 Apr 2014 14:23:26 +0400 Subject: [rabbitmq-discuss] How to stop consuming message from selective queue In-Reply-To: References: <11b16a32-75dc-4265-91d6-b3455351d0a8@googlegroups.com> Message-ID: On 28 April 2014 at 13:56:29, Subshiri S (subshiri at gmail.com) wrote: > > Yes, I got it. the basicConsume() method returns the consumer > tag corresponding to the queue Correction: for the *consumer*. A queue can have many consumers. -- MK Software Engineer, Pivotal/RabbitMQ From mklishin at gopivotal.com Mon Apr 28 11:30:34 2014 From: mklishin at gopivotal.com (Michael Klishin) Date: Mon, 28 Apr 2014 14:30:34 +0400 Subject: [rabbitmq-discuss] [RUBY-AMQP GEM] How do we can retry publish message on errors? In-Reply-To: References: Message-ID: On 28 April 2014 at 11:44:52, Loganathan Sellapa (loganathan.ms at gmail.com) wrote: > > I would like to know how the error handling is done for "publish > message". I want to retry the publish message when some exception/failure > occurred on RabbitMq? I checked the examples on ruby amqp gem > but unable to find the proper example for republishing the message > on error. http://www.rabbitmq.com/confirms.html http://rubyamqp.info/articles/rabbitmq_extensions/? https://github.com/ruby-amqp/amqp/blob/master/examples/extensions/rabbitmq/publisher_confirmations_with_transient_messages.rb -- MK Software Engineer, Pivotal/RabbitMQ From simon at rabbitmq.com Mon Apr 28 11:47:03 2014 From: simon at rabbitmq.com (Simon MacMullen) Date: Mon, 28 Apr 2014 11:47:03 +0100 Subject: [rabbitmq-discuss] LIFO queue In-Reply-To: <535E0FB7.3040708@fsn.hu> References: <4BCC459C.50303@250bpm.com> <535E0FB7.3040708@fsn.hu> Message-ID: <535E31A7.6060703@rabbitmq.com> On 28/04/2014 09:22, Nagy, Attila wrote: > Hence, I would need a LIFO queue in RabbitMQ too. Feel free to provide an alternate implementation: http://hg.rabbitmq.com/rabbitmq-server/file/02d1d855e4b3/src/rabbit_backing_queue.erl On a slightly more serious note, you could use TTL + DLX to at least get "old" messages into a lower priority queue. But you probably already knew that. Cheers, Simon From mklishin at gopivotal.com Mon Apr 28 11:46:46 2014 From: mklishin at gopivotal.com (Michael Klishin) Date: Mon, 28 Apr 2014 14:46:46 +0400 Subject: [rabbitmq-discuss] Need a design suggestion - Limited Consumer handling In-Reply-To: <8f6881f3-b20d-4f3f-b6d7-e462a4139a02@googlegroups.com> References: <8f6881f3-b20d-4f3f-b6d7-e462a4139a02@googlegroups.com> Message-ID: On 28 April 2014 at 12:04:26, Subshiri S (subshiri at gmail.com) wrote: > > There are 100 queues and 10 threads to consume messages from > that 100 queue. > Each thread will be consuming messages from 10 queue each. > Question 1 : How to dynamically assign the threads to queues ? If you know queue names, it?s as easy as splitting a collection in groups of 10. If you don?t, do it when you have queue names. > No more than one thread should consume from a queue (to maintain > the order of processing the message in the respective queue) > Question 2 : When there is a need to increase the consumer threads > while the system runs, How it can be done ?. It depends on your client library. In general, concurrent processing of messages while preserving order is not trivial and will require consumer synchronization in the client, one way or another.? You can specify a custom concurrent pool in some clients, e.g. Java: http://rabbitmq.com/api-guide.html -- MK Software Engineer, Pivotal/RabbitMQ From loganathan.ms at gmail.com Mon Apr 28 12:16:53 2014 From: loganathan.ms at gmail.com (Loganathan Sellapa) Date: Mon, 28 Apr 2014 16:46:53 +0530 Subject: [rabbitmq-discuss] [RUBY-AMQP GEM] How do we can retry publish message on errors? In-Reply-To: References: Message-ID: HI Michael, Thanks, but I need some more clarification. Right now I have durable exchange and auto delete is set to false for the queue. So I assumed that RabbitMq will re-publish the message automatically, whenever RabbitMq recovered from failover case. My question is how can I make publish the message(from producer to RabbitMq broker), when some exceptions occurred while publishing the message to rabbitMq channel. Below is my sample code for it. *ch1 = AMQP::Channel.new($connection)* *@ns_x = ch1.direct(ns_exchange, :durable => true)* *@ns_queue = ch1.queue(ns_queue, :auto_delete => false)* *@ns_queue.bind(@ns_x, :routing_key => @ns_queue.name ).subscribe(:ack => true, &method(:handle_ns_message))* *## How can I redo this whenever some exceptions/failure occurred* *@ns_x.publish(payload, :routing_key =>@ns_queue.name , :headers => headers, :mandatory => true) * regards, Loganathan Mob: +91 7760780741 | +91 9944414388 Skype: loganathan.sellappa ViewMe On Mon, Apr 28, 2014 at 4:00 PM, Michael Klishin wrote: > On 28 April 2014 at 11:44:52, Loganathan Sellapa (loganathan.ms at gmail.com) > wrote: > > > I would like to know how the error handling is done for "publish > > message". I want to retry the publish message when some exception/failure > > occurred on RabbitMq? I checked the examples on ruby amqp gem > > but unable to find the proper example for republishing the message > > on error. > > http://www.rabbitmq.com/confirms.html > http://rubyamqp.info/articles/rabbitmq_extensions/ > > https://github.com/ruby-amqp/amqp/blob/master/examples/extensions/rabbitmq/publisher_confirmations_with_transient_messages.rb > -- > MK > > Software Engineer, Pivotal/RabbitMQ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mklishin at gopivotal.com Mon Apr 28 12:20:05 2014 From: mklishin at gopivotal.com (Michael Klishin) Date: Mon, 28 Apr 2014 15:20:05 +0400 Subject: [rabbitmq-discuss] [RUBY-AMQP GEM] How do we can retry publish message on errors? In-Reply-To: References: Message-ID: ?On 28 April 2014 at 15:17:34, Loganathan Sellapa (loganathan.ms at gmail.com) wrote: > > So I assumed that RabbitMq will re-publish the message automatically, > whenever RabbitMq recovered from failover case. That?s not true. > My question > is how can I make publish the message(from producer to RabbitMq > broker), when some exceptions occurred while publishing the > message to rabbitMq channel. Below is my sample code for it. Handle exceptions from Exchange#publish. Note that this does not cover cases when EventMachine event loop just dies under you but there?s nothing amqp gem can do. This is also why unless you already use EventMachine heavily, you should use Bunny or March Hare: https://github.com/ruby-amqp/amqp#a-word-of-warning-use-this-only-if-you-already-use-eventmachine -- MK Software Engineer, Pivotal/RabbitMQ From michael.s.klishin at gmail.com Mon Apr 28 12:43:50 2014 From: michael.s.klishin at gmail.com (Michael Klishin) Date: Mon, 28 Apr 2014 15:43:50 +0400 Subject: [rabbitmq-discuss] New RabbitMQ 3.3.0 Web_stomp SSL problems In-Reply-To: References: Message-ID: 2014-04-28 7:08 GMT+04:00 Sebastien Dubois : > We were using jshiell's solution, which in my understanding was the patch > that was systemized in RMQ3.3.0. It wasn't exactly the same patch. Here's the key commit: http://hg.rabbitmq.com/rabbitmq-web-stomp/rev/7c7c60d6ab05 Can you post a small HTML/JS example that replicates the issue? I can compare it with what we've used to QA WebSTOMP SSL support. -- MK http://github.com/michaelklishin http://twitter.com/michaelklishin -------------- next part -------------- An HTML attachment was scrubbed... URL: From loganathan.ms at gmail.com Mon Apr 28 13:46:43 2014 From: loganathan.ms at gmail.com (Loganathan Sellapa) Date: Mon, 28 Apr 2014 18:16:43 +0530 Subject: [rabbitmq-discuss] [RUBY-AMQP GEM] How do we can retry publish message on errors? In-Reply-To: References: Message-ID: Hi Michael, Ok, I can able to handle the exceptions while publishing the messages. As you mentioned I have set up the publisher confirm acknowledgement, and I published 10 messages to the exchange and I received all the 10 messages on my consumer but the acknowledgement is not coming for all the 10 messages.I tried multiple times and I never received the acknowledgment for all my messages at all. Copied below my code and output. how can I identify the message along with the message contents for which didn't get the acknowledgement ? *OUTPUT FOR CHANNEL ACKNOWLEDGEMENT: * Received basic_ack: multiple = #, delivery_tag = 5 Received basic_ack: multiple = #, delivery_tag = 6 Received basic_ack: multiple = #, delivery_tag = 7 Received basic_ack: multiple = #, delivery_tag = 8 Received basic_ack: multiple = #, delivery_tag = 10 Got acknowledgement for 5 messages only(I suspect RabbitMq sent acknowledgement for multiple messages on *5 & 10*) *OUTPUT FROM CONSUMER (I increment the number while receiving the message)* 1 2 3 4 5 6 7 8 9 10 # Received 10 messages in consumer *CODE* ch1 = AMQP::Channel.new($connection) @ns_x = ch1.direct(ns_exchange, :durable => true) @ns_queue = ch1.queue(ns_queue, :auto_delete => false) @ns_queue.bind(@ns_x, :routing_key => @ns_queue.name).subscribe(:ack => true, &method(:handle_ns_message)) ch1.confirm_select PUBLISH MESSAGE 10 times: message_headers = {"messageType" => "ipam/operations/ip/checkCanAllocateIP"} message_body = {"ipsToAllocate" => "1", "orgId" => "438c6cdd-aa2d-4f3c-bec3-aefb85d6fca5"} 10.times do @ns_x.publish(message_body, :routing_key => "ipamAgent.operations", :headers => message_headers, :mandatory => true) end ch1.on_ack do |basic_ack| puts "Received basic_ack: multiple = #{basic_ack.multiple}, delivery_tag = #{basic_ack.delivery_tag}" end regards, Loganathan Mob: +91 7760780741 | +91 9944414388 Skype: loganathan.sellappa ViewMe On Mon, Apr 28, 2014 at 4:50 PM, Michael Klishin wrote: > On 28 April 2014 at 15:17:34, Loganathan Sellapa (loganathan.ms at gmail.com) > wrote: > > > So I assumed that RabbitMq will re-publish the message automatically, > > whenever RabbitMq recovered from failover case. > > That?s not true. > > > My question > > is how can I make publish the message(from producer to RabbitMq > > broker), when some exceptions occurred while publishing the > > message to rabbitMq channel. Below is my sample code for it. > > Handle exceptions from Exchange#publish. Note that this does not cover > cases when EventMachine event loop just dies under you but there?s nothing > amqp gem can do. > > This is also why unless you already use EventMachine heavily, you > should use Bunny or March Hare: > > > https://github.com/ruby-amqp/amqp#a-word-of-warning-use-this-only-if-you-already-use-eventmachine > -- > MK > > Software Engineer, Pivotal/RabbitMQ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mklishin at gopivotal.com Mon Apr 28 13:52:16 2014 From: mklishin at gopivotal.com (Michael Klishin) Date: Mon, 28 Apr 2014 16:52:16 +0400 Subject: [rabbitmq-discuss] [RUBY-AMQP GEM] How do we can retry publish message on errors? In-Reply-To: References: Message-ID: On 28 April 2014 at 16:47:24, Loganathan Sellapa (loganathan.ms at gmail.com) wrote: > > @ns_x.publish(message_body, :routing_key => "ipamAgent.operations", > :headers => message_headers, :mandatory => true) > end > > ch1.on_ack do |basic_ack| > puts "Received basic_ack: multiple = #{basic_ack.multiple}, > delivery_tag = #{basic_ack.delivery_tag}" > end You need to define the handler before you publish, otherwise you have a race condition ?(it makes sense to do this right after calling confirm_select). -- MK Software Engineer, Pivotal/RabbitMQ From loganathan.ms at gmail.com Mon Apr 28 14:04:29 2014 From: loganathan.ms at gmail.com (Loganathan Sellapa) Date: Mon, 28 Apr 2014 18:34:29 +0530 Subject: [rabbitmq-discuss] [RUBY-AMQP GEM] How do we can retry publish message on errors? In-Reply-To: References: Message-ID: Ok, I updated the coed as below. ch1.on_ack do |basic_ack| puts "Received basic_ack: multiple = #{basic_ack.multiple}, delivery_tag = #{basic_ack.delivery_tag}" end 10.times do @ns_x.publish(message_body, :routing_key => "ipamAgent.operations", :headers => message_headers, :mandatory => true) end Can you let me know your comments for the below questions 1. How can I identify the message along with the message contents for which didn't get the acknowledgement ? 2. While publishing the 10 messages, I received acknowledgement for 5 messages only. I suspect RabbitMq sent acknowledgement for multiple messages on 5 & 10, if so then how can we make sure that the acknowledgement is meant for message 1,2,3,4 and 5 in "Received basic_ack: multiple = #, delivery_tag = 5" *Acknowledgement O/p for 10 messages:* Received basic_ack: multiple = #, delivery_tag = 5 Received basic_ack: multiple = #, delivery_tag = 6 Received basic_ack: multiple = #, delivery_tag = 7 Received basic_ack: multiple = #, delivery_tag = 8 Received basic_ack: multiple = #, delivery_tag = 10 3. How do we ensure that message is successfully published to the exchange.(I want some way to identify the message is sent from ruby client to RabbitMq exchange successfully) regards, Loganathan Mob: +91 7760780741 | +91 9944414388 Skype: loganathan.sellappa ViewMe On Mon, Apr 28, 2014 at 6:22 PM, Michael Klishin wrote: > On 28 April 2014 at 16:47:24, Loganathan Sellapa (loganathan.ms at gmail.com) > wrote: > > > @ns_x.publish(message_body, :routing_key => "ipamAgent.operations", > > :headers => message_headers, :mandatory => true) > > end > > > > ch1.on_ack do |basic_ack| > > puts "Received basic_ack: multiple = #{basic_ack.multiple}, > > delivery_tag = #{basic_ack.delivery_tag}" > > end > > You need to define the handler before you publish, otherwise you have > a race condition (it makes sense > to do this right after calling confirm_select). > -- > MK > > Software Engineer, Pivotal/RabbitMQ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mklishin at gopivotal.com Mon Apr 28 14:08:47 2014 From: mklishin at gopivotal.com (Michael Klishin) Date: Mon, 28 Apr 2014 17:08:47 +0400 Subject: [rabbitmq-discuss] [RUBY-AMQP GEM] How do we can retry publish message on errors? In-Reply-To: References: Message-ID: ?On 28 April 2014 at 17:05:09, Loganathan Sellapa (loganathan.ms at gmail.com) wrote: > > 1. How can I identify the message along with the message contents > for which didn't get the acknowledgement ? Channel keeps a counter of confirms: https://github.com/ruby-amqp/amqp/blob/master/lib/amqp/channel.rb#L187 Outstanding messages can be stored and identified however you like. > 2. While publishing the 10 messages, I received acknowledgement > for 5 messages only. I suspect RabbitMq sent acknowledgement > for multiple messages on 5 & 10, if so then how can we make sure that > the acknowledgement is meant for message 1,2,3,4 and 5 in "Received > basic_ack: multiple = #> @delivery_tag=5, @multiple=true>, delivery_tag = 5" delivery_tag = 5, multiple = true means "all unacknowledged messages up to 5 (inclusive)" -- MK Software Engineer, Pivotal/RabbitMQ From KarrasT at Pragmatics.com Mon Apr 28 14:12:00 2014 From: KarrasT at Pragmatics.com (Karras, Thomas) Date: Mon, 28 Apr 2014 09:12:00 -0400 Subject: [rabbitmq-discuss] Certificate with more than one intermediate CA Message-ID: <62573470865EAA41A09BB261D33F2DDF64148E@pragmail3.pragmatics.com> Hello, I have a question regarding a certificate with more than one intermediate CA and the RabbitMQ federation plugin. We have successfully been using the SSL connection setup with certificates with a single CA but a new security requirement is forcing us to switch to a certificate with 2 intermediary CA's. We are running into SSL errors when switching to this new cert with it printing "unknown POSIX error". We believe this is related to the depth setting in the erlang ssl_options section. We added the depth=2 option since the documentation states that the default is 1. Setting this on both servers did not fix the error. Does this setting need to be set somehow on the client connection the federation plugin creates? I could not find a way to specify client ssl_options for the federation plugin. The new certs work fine with all the openssl tests on the SSL troubleshooting page. Thanks, Thomas Karras -------------- next part -------------- An HTML attachment was scrubbed... URL: From mklishin at gopivotal.com Mon Apr 28 14:16:33 2014 From: mklishin at gopivotal.com (Michael Klishin) Date: Mon, 28 Apr 2014 17:16:33 +0400 Subject: [rabbitmq-discuss] Certificate with more than one intermediate CA In-Reply-To: <62573470865EAA41A09BB261D33F2DDF64148E@pragmail3.pragmatics.com> References: <62573470865EAA41A09BB261D33F2DDF64148E@pragmail3.pragmatics.com> Message-ID: ?On 28 April 2014 at 17:13:32, Karras, Thomas (karrast at pragmatics.com) wrote: > > We are running into SSL errors when switching to this new cert > with it printing ?unknown POSIX error?. Is it upstream or downstream that reports the error? -- MK Software Engineer, Pivotal/RabbitMQ From KarrasT at Pragmatics.com Mon Apr 28 14:22:27 2014 From: KarrasT at Pragmatics.com (Karras, Thomas) Date: Mon, 28 Apr 2014 09:22:27 -0400 Subject: [rabbitmq-discuss] [SPAM] Re: Certificate with more than one intermediate CA In-Reply-To: References: <62573470865EAA41A09BB261D33F2DDF64148E@pragmail3.pragmatics.com> Message-ID: <62573470865EAA41A09BB261D33F2DDF641490@pragmail3.pragmatics.com> We have them both federating to each other so both sides report the error. The error message comes from the federation plugin attempting to connect to the server and failing to successfully negotiate a SSL connection. -----Original Message----- From: Michael Klishin [mailto:mklishin at gopivotal.com] Sent: Monday, April 28, 2014 9:17 AM To: Karras, Thomas; Discussions about RabbitMQ Subject: [SPAM] Re: [rabbitmq-discuss] Certificate with more than one intermediate CA Importance: Low ?On 28 April 2014 at 17:13:32, Karras, Thomas (karrast at pragmatics.com) wrote: > > We are running into SSL errors when switching to this new cert > with it printing ?unknown POSIX error?. Is it upstream or downstream that reports the error? -- MK Software Engineer, Pivotal/RabbitMQ From martijn.otto at copernica.com Mon Apr 28 14:26:58 2014 From: martijn.otto at copernica.com (Martijn Otto) Date: Mon, 28 Apr 2014 15:26:58 +0200 Subject: [rabbitmq-discuss] Missing Cancel-OK frame Message-ID: <1398691618.17773.12.camel@martijn-laptop> Hello, I am working on an AMQP library for RabbitMQ and either I am missing something totally fundamental, or RabbitMQ is failing in its duty to send me a Cancel-OK frame to confirm consuming has stopped. I have attached a pcap file to illustrate this behavior. As you can see, the nowait option has been set to false which, according to the specification, should result in a Cancel-OK frame. Instead I get the confirmation for the next call. With regards, Martijn Otto Copernica B.V. -------------- next part -------------- A non-text attachment was scrubbed... Name: no-cancel-ok-frame Type: application/vnd.tcpdump.pcap Size: 3768 bytes Desc: not available URL: From matthias at rabbitmq.com Mon Apr 28 14:37:43 2014 From: matthias at rabbitmq.com (Matthias Radestock) Date: Mon, 28 Apr 2014 14:37:43 +0100 Subject: [rabbitmq-discuss] Missing Cancel-OK frame In-Reply-To: <1398691618.17773.12.camel@martijn-laptop> References: <1398691618.17773.12.camel@martijn-laptop> Message-ID: <535E59A7.8040007@rabbitmq.com> On 28/04/14 14:26, Martijn Otto wrote: > I am working on an AMQP library for RabbitMQ and either I am missing > something totally fundamental, or RabbitMQ is failing in its duty to > send me a Cancel-OK frame to confirm consuming has stopped. You are pipelining synchronous commands. The AMQP spec does not permit that, so the behaviour is undefined. See the pseudo code in section 2.2.2 of https://www.rabbitmq.com/resources/specs/amqp0-9-1.pdf. Matthias. From sp345s at att.com Mon Apr 28 16:01:33 2014 From: sp345s at att.com (PATAR, SAGAR) Date: Mon, 28 Apr 2014 15:01:33 +0000 Subject: [rabbitmq-discuss] dynamic binding Message-ID: <59BF56992750434F89F8EBE141E64B1D01876F97@GAALPA1MSGUSR9C.ITServices.sbc.com> We are using header exchanges and are using spring-rabbit mq for configuration and binding listeners to listen to specific messages with specific message header values. Ex: message header "srcId" has values "1001,1002,1003,1004" At runtime if the same listener bean has to listen to messages with message header "srcId" having value "2001,2002,2003" .. Is there a way to add these bindings dynamically at runtime ..?? Thanks Sagar -------------- next part -------------- An HTML attachment was scrubbed... URL: From grussell at gopivotal.com Mon Apr 28 17:09:26 2014 From: grussell at gopivotal.com (Gary Russell) Date: Mon, 28 Apr 2014 12:09:26 -0400 Subject: [rabbitmq-discuss] dynamic binding In-Reply-To: <59BF56992750434F89F8EBE141E64B1D01876F97@GAALPA1MSGUSR9C.ITServices.sbc.com> References: <59BF56992750434F89F8EBE141E64B1D01876F97@GAALPA1MSGUSR9C.ITServices.sbc.com> Message-ID: Yes; just add a second binding with the new header value(s); no need to do anything on the listener - it's all a function of the rabbitmq-server. On Mon, Apr 28, 2014 at 11:01 AM, PATAR, SAGAR wrote: > > > We are using header exchanges and are using spring-rabbit mq for > configuration and binding listeners to listen to specific messages with > specific message header values. > > Ex: message header "srcId" has values "1001,1002,1003,1004" > > > > At runtime if the same listener bean has to listen to messages with > message header "srcId" having value "2001,2002,2003" .. > > Is there a way to add these bindings dynamically at runtime ..?? > > > > Thanks > > Sagar > > _______________________________________________ > rabbitmq-discuss mailing list > rabbitmq-discuss at lists.rabbitmq.com > https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From srinatsr at cisco.com Mon Apr 28 19:14:36 2014 From: srinatsr at cisco.com (Srinath Sridharan -X (srinatsr - ZENSAR TECHNOLOGIES INC at Cisco)) Date: Mon, 28 Apr 2014 18:14:36 +0000 Subject: [rabbitmq-discuss] RabbitMQ Monitoring Queue Activites in Spring Project Message-ID: <1DB4F5481AB296448E8A5B5BC9D336541CB02E37@xmb-aln-x10.cisco.com> Hello, * I need API's to capture Queue Activites. * Status of Nodes and Queues. * Implementation of Monitoring Queue. * Logging using async logger. Please need help with some example. Regards ...* Srinath -------------- next part -------------- An HTML attachment was scrubbed... URL: From mklishin at gopivotal.com Mon Apr 28 19:26:02 2014 From: mklishin at gopivotal.com (Michael Klishin) Date: Mon, 28 Apr 2014 22:26:02 +0400 Subject: [rabbitmq-discuss] RabbitMQ Monitoring Queue Activites in Spring Project In-Reply-To: <1DB4F5481AB296448E8A5B5BC9D336541CB02E37@xmb-aln-x10.cisco.com> References: <1DB4F5481AB296448E8A5B5BC9D336541CB02E37@xmb-aln-x10.cisco.com> Message-ID: On April 28, 2014 at 10:15:58 PM, Srinath Sridharan -X (srinatsr - ZENSAR TECHNOLOGIES INC at Cisco) (srinatsr at cisco.com) wrote: > > > ? I need API?s to capture Queue Activites. > ? Status of Nodes and Queues. > ? Implementation of Monitoring Queue. RabbitMQ HTTP API likely provides what you want: http://www.rabbitmq.com/management.html http://hg.rabbitmq.com/rabbitmq-management/raw-file/rabbitmq_v3_3_0/priv/www/api/index.html? > ? Logging using async logger. Can you be more specific? -- MK Software Engineer, Pivotal/RabbitMQ From srinatsr at cisco.com Mon Apr 28 19:46:12 2014 From: srinatsr at cisco.com (Srinath Sridharan -X (srinatsr - ZENSAR TECHNOLOGIES INC at Cisco)) Date: Mon, 28 Apr 2014 18:46:12 +0000 Subject: [rabbitmq-discuss] RabbitMQ Monitoring Queue Activites in Spring Project In-Reply-To: References: <1DB4F5481AB296448E8A5B5BC9D336541CB02E37@xmb-aln-x10.cisco.com> Message-ID: <1DB4F5481AB296448E8A5B5BC9D336541CB030D4@xmb-aln-x10.cisco.com> How to do logging using async logger? How to hit those api's from spring project(java class)? Regards ?? Srinath -----Original Message----- From: Michael Klishin [mailto:mklishin at gopivotal.com] Sent: Monday, April 28, 2014 11:26 AM To: Srinath Sridharan -X (srinatsr - ZENSAR TECHNOLOGIES INC at Cisco); Discussions about RabbitMQ Subject: Re: [rabbitmq-discuss] RabbitMQ Monitoring Queue Activites in Spring Project On April 28, 2014 at 10:15:58 PM, Srinath Sridharan -X (srinatsr - ZENSAR TECHNOLOGIES INC at Cisco) (srinatsr at cisco.com) wrote: > > > ? I need API?s to capture Queue Activites. > ? Status of Nodes and Queues. > ? Implementation of Monitoring Queue. RabbitMQ HTTP API likely provides what you want: http://www.rabbitmq.com/management.html http://hg.rabbitmq.com/rabbitmq-management/raw-file/rabbitmq_v3_3_0/priv/www/api/index.html? > ? Logging using async logger. Can you be more specific? -- MK Software Engineer, Pivotal/RabbitMQ From mklishin at gopivotal.com Mon Apr 28 19:49:59 2014 From: mklishin at gopivotal.com (Michael Klishin) Date: Mon, 28 Apr 2014 22:49:59 +0400 Subject: [rabbitmq-discuss] RabbitMQ Monitoring Queue Activites in Spring Project In-Reply-To: <1DB4F5481AB296448E8A5B5BC9D336541CB030D4@xmb-aln-x10.cisco.com> References: <1DB4F5481AB296448E8A5B5BC9D336541CB02E37@xmb-aln-x10.cisco.com> <1DB4F5481AB296448E8A5B5BC9D336541CB030D4@xmb-aln-x10.cisco.com> Message-ID: ?On April 28, 2014 at 10:46:14 PM, Srinath Sridharan -X (srinatsr - ZENSAR TECHNOLOGIES INC at Cisco) (srinatsr at cisco.com) wrote: > > How to do logging using async logger? What do you want to log? Do you want to log application events over RabbitMQ, log RabbitMQ activity using a Java logger, or something else? Sorry but I expect engineers to do better than 1 line overly generic problem descriptions. > How to hit those api's from spring project(java class)? It?s HTTP API. Like you would work with any other HTTP API from Java (e.g. Spring?s REST template, Apache HTTP components, etc). -- MK Software Engineer, Pivotal/RabbitMQ From srinatsr at cisco.com Mon Apr 28 19:51:52 2014 From: srinatsr at cisco.com (Srinath Sridharan -X (srinatsr - ZENSAR TECHNOLOGIES INC at Cisco)) Date: Mon, 28 Apr 2014 18:51:52 +0000 Subject: [rabbitmq-discuss] RabbitMQ Monitoring Queue Activites in Spring Project In-Reply-To: References: <1DB4F5481AB296448E8A5B5BC9D336541CB02E37@xmb-aln-x10.cisco.com> <1DB4F5481AB296448E8A5B5BC9D336541CB030D4@xmb-aln-x10.cisco.com> Message-ID: <1DB4F5481AB296448E8A5B5BC9D336541CB030F4@xmb-aln-x10.cisco.com> I need to log RabbitMQ activity using a java logger but asynchronously Regards ?? Srinath -----Original Message----- From: Michael Klishin [mailto:mklishin at gopivotal.com] Sent: Monday, April 28, 2014 11:50 AM To: Discussions about RabbitMQ; Srinath Sridharan -X (srinatsr - ZENSAR TECHNOLOGIES INC at Cisco) Subject: RE: [rabbitmq-discuss] RabbitMQ Monitoring Queue Activites in Spring Project ?On April 28, 2014 at 10:46:14 PM, Srinath Sridharan -X (srinatsr - ZENSAR TECHNOLOGIES INC at Cisco) (srinatsr at cisco.com) wrote: > > How to do logging using async logger? What do you want to log? Do you want to log application events over RabbitMQ, log RabbitMQ activity using a Java logger, or something else? Sorry but I expect engineers to do better than 1 line overly generic problem descriptions. > How to hit those api's from spring project(java class)? It?s HTTP API. Like you would work with any other HTTP API from Java (e.g. Spring?s REST template, Apache HTTP components, etc). -- MK Software Engineer, Pivotal/RabbitMQ From srinatsr at cisco.com Mon Apr 28 19:52:50 2014 From: srinatsr at cisco.com (Srinath Sridharan -X (srinatsr - ZENSAR TECHNOLOGIES INC at Cisco)) Date: Mon, 28 Apr 2014 18:52:50 +0000 Subject: [rabbitmq-discuss] RabbitMQ Monitoring Queue Activites in Spring Project In-Reply-To: References: <1DB4F5481AB296448E8A5B5BC9D336541CB02E37@xmb-aln-x10.cisco.com> <1DB4F5481AB296448E8A5B5BC9D336541CB030D4@xmb-aln-x10.cisco.com> Message-ID: <1DB4F5481AB296448E8A5B5BC9D336541CB03105@xmb-aln-x10.cisco.com> I am using spring REST template. How to pass authentication username and password as guest/guest to these apis? Regards ?? Srinath -----Original Message----- From: Michael Klishin [mailto:mklishin at gopivotal.com] Sent: Monday, April 28, 2014 11:50 AM To: Discussions about RabbitMQ; Srinath Sridharan -X (srinatsr - ZENSAR TECHNOLOGIES INC at Cisco) Subject: RE: [rabbitmq-discuss] RabbitMQ Monitoring Queue Activites in Spring Project ?On April 28, 2014 at 10:46:14 PM, Srinath Sridharan -X (srinatsr - ZENSAR TECHNOLOGIES INC at Cisco) (srinatsr at cisco.com) wrote: > > How to do logging using async logger? What do you want to log? Do you want to log application events over RabbitMQ, log RabbitMQ activity using a Java logger, or something else? Sorry but I expect engineers to do better than 1 line overly generic problem descriptions. > How to hit those api's from spring project(java class)? It?s HTTP API. Like you would work with any other HTTP API from Java (e.g. Spring?s REST template, Apache HTTP components, etc). -- MK Software Engineer, Pivotal/RabbitMQ From mklishin at gopivotal.com Mon Apr 28 19:55:44 2014 From: mklishin at gopivotal.com (Michael Klishin) Date: Mon, 28 Apr 2014 22:55:44 +0400 Subject: [rabbitmq-discuss] RabbitMQ Monitoring Queue Activites in Spring Project In-Reply-To: <1DB4F5481AB296448E8A5B5BC9D336541CB030F4@xmb-aln-x10.cisco.com> References: <1DB4F5481AB296448E8A5B5BC9D336541CB02E37@xmb-aln-x10.cisco.com> <1DB4F5481AB296448E8A5B5BC9D336541CB030D4@xmb-aln-x10.cisco.com> <1DB4F5481AB296448E8A5B5BC9D336541CB030F4@xmb-aln-x10.cisco.com> Message-ID: On April 28, 2014 at 10:51:53 PM, Srinath Sridharan -X (srinatsr - ZENSAR TECHNOLOGIES INC at Cisco) (srinatsr at cisco.com) wrote: > > I need to log RabbitMQ activity using a java logger but asynchronously You can bind a queue to?amq.rabbitmq.log (a topic exchange in the default vhost ) and consume some events that happen in the broker, then log them using any logger you want. See also?http://www.rabbitmq.com/firehose.html. There is no other way to get a stream of broker activity in a JVM app. -- MK Software Engineer, Pivotal/RabbitMQ From mklishin at gopivotal.com Mon Apr 28 19:56:45 2014 From: mklishin at gopivotal.com (Michael Klishin) Date: Mon, 28 Apr 2014 22:56:45 +0400 Subject: [rabbitmq-discuss] RabbitMQ Monitoring Queue Activites in Spring Project In-Reply-To: <1DB4F5481AB296448E8A5B5BC9D336541CB03105@xmb-aln-x10.cisco.com> References: <1DB4F5481AB296448E8A5B5BC9D336541CB02E37@xmb-aln-x10.cisco.com> <1DB4F5481AB296448E8A5B5BC9D336541CB030D4@xmb-aln-x10.cisco.com> <1DB4F5481AB296448E8A5B5BC9D336541CB03105@xmb-aln-x10.cisco.com> Message-ID: On April 28, 2014 at 10:54:13 PM, Srinath Sridharan -X (srinatsr - ZENSAR TECHNOLOGIES INC at Cisco) (srinatsr at cisco.com) wrote: > > I am using spring REST template. How to pass authentication > username and password as guest/guest to these apis? As?http://hg.rabbitmq.com/rabbitmq-management/raw-file/rabbitmq_v3_3_0/priv/www/api/index.html demonstrates, using HTTP authentication and a RabbitMQ user credentials.? -- MK Software Engineer, Pivotal/RabbitMQ From srinatsr at cisco.com Mon Apr 28 19:58:04 2014 From: srinatsr at cisco.com (Srinath Sridharan -X (srinatsr - ZENSAR TECHNOLOGIES INC at Cisco)) Date: Mon, 28 Apr 2014 18:58:04 +0000 Subject: [rabbitmq-discuss] RabbitMQ Monitoring Queue Activites in Spring Project In-Reply-To: References: <1DB4F5481AB296448E8A5B5BC9D336541CB02E37@xmb-aln-x10.cisco.com> <1DB4F5481AB296448E8A5B5BC9D336541CB030D4@xmb-aln-x10.cisco.com> <1DB4F5481AB296448E8A5B5BC9D336541CB03105@xmb-aln-x10.cisco.com> Message-ID: <1DB4F5481AB296448E8A5B5BC9D336541CB03131@xmb-aln-x10.cisco.com> Using rest template from java? Not from command prompt Regards ?? Srinath -----Original Message----- From: Michael Klishin [mailto:mklishin at gopivotal.com] Sent: Monday, April 28, 2014 11:57 AM To: Srinath Sridharan -X (srinatsr - ZENSAR TECHNOLOGIES INC at Cisco); Discussions about RabbitMQ Subject: Re: [rabbitmq-discuss] RabbitMQ Monitoring Queue Activites in Spring Project On April 28, 2014 at 10:54:13 PM, Srinath Sridharan -X (srinatsr - ZENSAR TECHNOLOGIES INC at Cisco) (srinatsr at cisco.com) wrote: > > I am using spring REST template. How to pass authentication > username and password as guest/guest to these apis? As?http://hg.rabbitmq.com/rabbitmq-management/raw-file/rabbitmq_v3_3_0/priv/www/api/index.html demonstrates, using HTTP authentication and a RabbitMQ user credentials.? -- MK Software Engineer, Pivotal/RabbitMQ From srinatsr at cisco.com Mon Apr 28 19:58:44 2014 From: srinatsr at cisco.com (Srinath Sridharan -X (srinatsr - ZENSAR TECHNOLOGIES INC at Cisco)) Date: Mon, 28 Apr 2014 18:58:44 +0000 Subject: [rabbitmq-discuss] RabbitMQ Monitoring Queue Activites in Spring Project In-Reply-To: References: <1DB4F5481AB296448E8A5B5BC9D336541CB02E37@xmb-aln-x10.cisco.com> <1DB4F5481AB296448E8A5B5BC9D336541CB030D4@xmb-aln-x10.cisco.com> <1DB4F5481AB296448E8A5B5BC9D336541CB030F4@xmb-aln-x10.cisco.com> Message-ID: <1DB4F5481AB296448E8A5B5BC9D336541CB0313E@xmb-aln-x10.cisco.com> How to bind a queue to amq.rabbitmq.log? Any useful links? Regards ?? Srinath -----Original Message----- From: Michael Klishin [mailto:mklishin at gopivotal.com] Sent: Monday, April 28, 2014 11:56 AM To: Discussions about RabbitMQ; Srinath Sridharan -X (srinatsr - ZENSAR TECHNOLOGIES INC at Cisco) Subject: RE: [rabbitmq-discuss] RabbitMQ Monitoring Queue Activites in Spring Project On April 28, 2014 at 10:51:53 PM, Srinath Sridharan -X (srinatsr - ZENSAR TECHNOLOGIES INC at Cisco) (srinatsr at cisco.com) wrote: > > I need to log RabbitMQ activity using a java logger but > > asynchronously You can bind a queue to?amq.rabbitmq.log (a topic exchange in the default vhost ) and consume some events that happen in the broker, then log them using any logger you want. See also?http://www.rabbitmq.com/firehose.html. There is no other way to get a stream of broker activity in a JVM app. -- MK Software Engineer, Pivotal/RabbitMQ From mklishin at gopivotal.com Mon Apr 28 20:01:27 2014 From: mklishin at gopivotal.com (Michael Klishin) Date: Mon, 28 Apr 2014 23:01:27 +0400 Subject: [rabbitmq-discuss] RabbitMQ Monitoring Queue Activites in Spring Project In-Reply-To: <1DB4F5481AB296448E8A5B5BC9D336541CB0313E@xmb-aln-x10.cisco.com> References: <1DB4F5481AB296448E8A5B5BC9D336541CB02E37@xmb-aln-x10.cisco.com> <1DB4F5481AB296448E8A5B5BC9D336541CB030D4@xmb-aln-x10.cisco.com> <1DB4F5481AB296448E8A5B5BC9D336541CB030F4@xmb-aln-x10.cisco.com> <1DB4F5481AB296448E8A5B5BC9D336541CB0313E@xmb-aln-x10.cisco.com> Message-ID: On April 28, 2014 at 10:58:45 PM, Srinath Sridharan -X (srinatsr - ZENSAR TECHNOLOGIES INC at Cisco) (srinatsr at cisco.com) wrote: > > How to bind a queue to amq.rabbitmq.log? Any useful links? http://lmgtfy.com/?q=rabbitmq+tutorials? -- MK Software Engineer, Pivotal/RabbitMQ From srinatsr at cisco.com Mon Apr 28 23:06:41 2014 From: srinatsr at cisco.com (Srinath Sridharan -X (srinatsr - ZENSAR TECHNOLOGIES INC at Cisco)) Date: Mon, 28 Apr 2014 22:06:41 +0000 Subject: [rabbitmq-discuss] Why Unacked messages - I couldn't get messages Message-ID: <1DB4F5481AB296448E8A5B5BC9D336541CB0330A@xmb-aln-x10.cisco.com> ValidateSaveRequestQueue the last one. I am seeing unacked messages and I couldn't get those messages. [cid:image001.png at 01CF62F3.6EC8EBB0] Regards ...* Srinath -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image001.png Type: image/png Size: 256802 bytes Desc: image001.png URL: From mklishin at gopivotal.com Mon Apr 28 23:10:11 2014 From: mklishin at gopivotal.com (Michael Klishin) Date: Tue, 29 Apr 2014 02:10:11 +0400 Subject: [rabbitmq-discuss] Why Unacked messages - I couldn't get messages In-Reply-To: <1DB4F5481AB296448E8A5B5BC9D336541CB0330A@xmb-aln-x10.cisco.com> References: <1DB4F5481AB296448E8A5B5BC9D336541CB0330A@xmb-aln-x10.cisco.com> Message-ID: On 29 April 2014 at 02:08:09, Srinath Sridharan -X (srinatsr - ZENSAR TECHNOLOGIES INC at Cisco) (srinatsr at cisco.com) wrote: > > ValidateSaveRequestQueue the last one. I am seeing unacked > messages and I couldn?t get those messages. You need to acknowledge or reject deliveries, or use automatic acknowledgement mode. Closing the channel those messages were delivered on will re-queue them.? http://www.rabbitmq.com/tutorials/tutorial-two-java.html -- MK Software Engineer, Pivotal/RabbitMQ From jeffdoto at gmail.com Mon Apr 28 21:04:31 2014 From: jeffdoto at gmail.com (Jeff Doto) Date: Mon, 28 Apr 2014 13:04:31 -0700 (PDT) Subject: [rabbitmq-discuss] Writes to Rabbit Occasionally Hang Indefinitely Message-ID: <6ee00146-26d3-4a89-9c1f-5811d4ba058d@googlegroups.com> Hello, We have a distributed Java application talking to a clustered RabbitMQ instance, and we are encountering a situation where the message-producing Java client will occasionally hang indefinitely during queueing operations. It is not the result of server memory limits being exceeded, or delays from flow control, as the management console displays both as healthy. Attached is the output of 'rabbitmqctl report' on the cluster nodes, and a Java thread stack. Note the presence of the blocked threads attempting to write to Rabbit in the attached stack. Our application producer node is running RabbitMQ 3.1.3 Client through spring-rabbit 1.2.1.RELEASE, on CentOS release 5.6 (Final), Java(TM) SE Runtime Environment (build 1.6.0_45-b06). Our application is configured to talk directly to the Rabbit server at the moment, though we have previously connected to the cluster through a hardware load balancer. We see the same issue whether we connect directly, or through the load balancer. We have the following RabbitMQ ConnectionFactory settings in our application: requestedHeartbeat: 10 seconds connectionTimeout: 60 seconds This connectionFactory gets passed to a Spring CachingConnectionFactory with a channelCacheSize of 100. Messages are being sent to the default exchange on an application-specific virtual host; the queues are direct queues with ha-mode:all set as a policy. The application has a single master node with multiple threads writing to, and many consumer nodes, each with multiple threads consuming from the server. We are queueing between 25 to 75 million messages a day. We?re wondering if anyone has encountered an issue similar to this running the versions of Rabbit server / client we are. Are there any suggestions for what we can do to prevent this? Could upgrading the server version help? Thank you, Jeff -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- { STATUS: { threads: [ { name: "1036158690 at qtp-783328245-6", priority: 5, state: "TIMED_WAITING", trace: [ "java.lang.Object.wait(Native Method)", "org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:626)" ] }, { name: "122130810 at qtp-783328245-1", priority: 5, state: "TIMED_WAITING", trace: [ "java.lang.Object.wait(Native Method)", "org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:626)" ] }, { name: "1313257738 at qtp-783328245-2", priority: 5, state: "TIMED_WAITING", trace: [ "java.lang.Object.wait(Native Method)", "org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:626)" ] }, { name: "1437053119 at qtp-783328245-9 - Acceptor0 SelectChannelConnector at 0.0.0.0:9043", priority: 5, state: "RUNNABLE", trace: [ "sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)", "sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)", "sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)", "sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)", "sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)", "org.mortbay.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:498)", "org.mortbay.io.nio.SelectorManager.doSelect(SelectorManager.java:192)", "org.mortbay.jetty.nio.SelectChannelConnector.accept(SelectChannelConnector.java:124)", "org.mortbay.jetty.AbstractConnector$Acceptor.run(AbstractConnector.java:708)", "org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)" ] }, { name: "1613938021 at qtp-783328245-5", priority: 5, state: "TIMED_WAITING", trace: [ "java.lang.Object.wait(Native Method)", "org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:626)" ] }, { name: "1681889006 at qtp-783328245-4", priority: 5, state: "RUNNABLE", trace: [ "java.lang.Thread.dumpThreads(Native Method)", "java.lang.Thread.getAllStackTraces(Thread.java:1530)", "com.company.module.management.status.thread.ThreadStatusList.getThreads(ThreadStatusList.java:17)", "sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)", "sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)", "sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)", "java.lang.reflect.Method.invoke(Method.java:597)", "com.company.module.management.status.json.JSONObject.populateInternalMap(JSONObject.java:420)", "com.company.module.management.status.json.JSONObject.(JSONObject.java:371)", "com.company.module.management.status.json.JSONObject.(JSONObject.java:365)", "com.company.module.management.status.JsonReportFormatter.writeOutputForBean(JsonReportFormatter.java:60)", "com.company.module.management.status.ConfiguredBeanStatusReporter.writeOutputForBean(ConfiguredBeanStatusReporter.java:75)", "com.company.module.management.status.CompositeStatusReporter.writeOutputForBean(CompositeStatusReporter.java:76)", "com.company.module.management.status.StatusServlet.doGet(StatusServlet.java:183)", "javax.servlet.http.HttpServlet.service(HttpServlet.java:707)", "javax.servlet.http.HttpServlet.service(HttpServlet.java:820)", "org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)", "org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:401)", "org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)", "org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)", "org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)", "org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)", "org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)", "org.mortbay.jetty.handler.HandlerCollection.handle(HandlerCollection.java:114)", "org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)", "org.mortbay.jetty.Server.handle(Server.java:326)", "org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)", "org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)", "org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)", "org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)", "org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)", "org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)", "org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)" ] }, { name: "1851406627 at qtp-783328245-8 - Acceptor1 SelectChannelConnector at 0.0.0.0:9043", priority: 5, state: "RUNNABLE", trace: [ "sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)", "sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)", "sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)", "sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)", "sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)", "org.mortbay.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:498)", "org.mortbay.io.nio.SelectorManager.doSelect(SelectorManager.java:192)", "org.mortbay.jetty.nio.SelectChannelConnector.accept(SelectChannelConnector.java:124)", "org.mortbay.jetty.AbstractConnector$Acceptor.run(AbstractConnector.java:708)", "org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)" ] }, { name: "254300116 at qtp-783328245-3", priority: 5, state: "TIMED_WAITING", trace: [ "java.lang.Object.wait(Native Method)", "org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:626)" ] }, { name: "532405576 at qtp-783328245-7", priority: 5, state: "TIMED_WAITING", trace: [ "java.lang.Object.wait(Native Method)", "org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:626)" ] }, { name: "73152903 at qtp-783328245-0", priority: 5, state: "TIMED_WAITING", trace: [ "java.lang.Object.wait(Native Method)", "org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:626)" ] }, { name: "AMQP Connection 172.20.149.41:5672", priority: 5, state: "RUNNABLE", trace: [ "java.net.SocketInputStream.socketRead0(Native Method)", "java.net.SocketInputStream.read(SocketInputStream.java:129)", "java.io.BufferedInputStream.fill(BufferedInputStream.java:218)", "java.io.BufferedInputStream.read(BufferedInputStream.java:237)", "java.io.DataInputStream.readUnsignedByte(DataInputStream.java:271)", "com.rabbitmq.client.impl.Frame.readFrom(Frame.java:95)", "com.rabbitmq.client.impl.SocketFrameHandler.readFrame(SocketFrameHandler.java:131)", "com.rabbitmq.client.impl.AMQConnection$MainLoop.run(AMQConnection.java:515)" ] }, { name: "AsyncAppender-Worker-Thread-21", priority: 5, state: "WAITING", trace: [ "sun.misc.Unsafe.park(Native Method)", "java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)", "java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)", "java.util.concurrent.ArrayBlockingQueue.take(ArrayBlockingQueue.java:322)", "ch.qos.logback.core.AsyncAppenderBase$Worker.run(AsyncAppenderBase.java:225)" ] }, { name: "AsyncAppender-Worker-Thread-22", priority: 5, state: "WAITING", trace: [ "sun.misc.Unsafe.park(Native Method)", "java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)", "java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)", "java.util.concurrent.ArrayBlockingQueue.take(ArrayBlockingQueue.java:322)", "ch.qos.logback.core.AsyncAppenderBase$Worker.run(AsyncAppenderBase.java:225)" ] }, { name: "AsyncAppender-Worker-Thread-23", priority: 5, state: "WAITING", trace: [ "sun.misc.Unsafe.park(Native Method)", "java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)", "java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)", "java.util.concurrent.ArrayBlockingQueue.take(ArrayBlockingQueue.java:322)", "ch.qos.logback.core.AsyncAppenderBase$Worker.run(AsyncAppenderBase.java:225)" ] }, { name: "AsyncAppender-Worker-Thread-24", priority: 5, state: "WAITING", trace: [ "sun.misc.Unsafe.park(Native Method)", "java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)", "java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)", "java.util.concurrent.ArrayBlockingQueue.take(ArrayBlockingQueue.java:322)", "ch.qos.logback.core.AsyncAppenderBase$Worker.run(AsyncAppenderBase.java:225)" ] }, { name: "AsyncAppender-Worker-Thread-25", priority: 5, state: "WAITING", trace: [ "sun.misc.Unsafe.park(Native Method)", "java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)", "java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)", "java.util.concurrent.ArrayBlockingQueue.take(ArrayBlockingQueue.java:322)", "ch.qos.logback.core.AsyncAppenderBase$Worker.run(AsyncAppenderBase.java:225)" ] }, { name: "AsyncAppender-Worker-Thread-3", priority: 5, state: "WAITING", trace: [ "sun.misc.Unsafe.park(Native Method)", "java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)", "java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)", "java.util.concurrent.ArrayBlockingQueue.take(ArrayBlockingQueue.java:322)", "ch.qos.logback.core.AsyncAppenderBase$Worker.run(AsyncAppenderBase.java:225)" ] }, { name: "AsyncAppender-Worker-Thread-4", priority: 5, state: "WAITING", trace: [ "sun.misc.Unsafe.park(Native Method)", "java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)", "java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)", "java.util.concurrent.ArrayBlockingQueue.take(ArrayBlockingQueue.java:322)", "ch.qos.logback.core.AsyncAppenderBase$Worker.run(AsyncAppenderBase.java:225)" ] }, { name: "AsyncAppender-Worker-Thread-5", priority: 5, state: "WAITING", trace: [ "sun.misc.Unsafe.park(Native Method)", "java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)", "java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)", "java.util.concurrent.ArrayBlockingQueue.take(ArrayBlockingQueue.java:322)", "ch.qos.logback.core.AsyncAppenderBase$Worker.run(AsyncAppenderBase.java:225)" ] }, { name: "AsyncAppender-Worker-Thread-6", priority: 5, state: "WAITING", trace: [ "sun.misc.Unsafe.park(Native Method)", "java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)", "java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)", "java.util.concurrent.ArrayBlockingQueue.take(ArrayBlockingQueue.java:322)", "ch.qos.logback.core.AsyncAppenderBase$Worker.run(AsyncAppenderBase.java:225)" ] }, { name: "AsyncAppender-Worker-Thread-7", priority: 5, state: "WAITING", trace: [ "sun.misc.Unsafe.park(Native Method)", "java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)", "java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)", "java.util.concurrent.ArrayBlockingQueue.take(ArrayBlockingQueue.java:322)", "ch.qos.logback.core.AsyncAppenderBase$Worker.run(AsyncAppenderBase.java:225)" ] }, { name: "Checkpointer (imageWebServicePerm_hraq257l)", priority: 5, state: "WAITING", trace: [ "java.lang.Object.wait(Native Method)", "java.lang.Object.wait(Object.java:485)", "com.sleepycat.je.utilint.DaemonThread.run(DaemonThread.java:172)", "java.lang.Thread.run(Thread.java:662)" ] }, { name: "Cleaner-1 (imageWebServicePerm_hraq257l)", priority: 5, state: "WAITING", trace: [ "java.lang.Object.wait(Native Method)", "java.lang.Object.wait(Object.java:485)", "com.sleepycat.je.utilint.DaemonThread.run(DaemonThread.java:172)", "java.lang.Thread.run(Thread.java:662)" ] }, { name: "Cluster Socket Accept", priority: 5, state: "RUNNABLE", trace: [ "java.net.PlainSocketImpl.socketAccept(Native Method)", "java.net.PlainSocketImpl.accept(PlainSocketImpl.java:408)", "java.net.ServerSocket.implAccept(ServerSocket.java:462)", "java.net.ServerSocket.accept(ServerSocket.java:430)", "com.company.module.cluster.peer.socket.SocketPeerServer$Acceptor.run(SocketPeerServer.java:184)", "java.lang.Thread.run(Thread.java:662)" ] }, { name: "ClusterNode Master to Slave Monitor", priority: 5, state: "TIMED_WAITING", trace: [ "java.lang.Thread.sleep(Native Method)", "com.company.contrib.web.cluster.MerlinClusterNode$Monitor.run(MerlinClusterNode.java:72)", "java.lang.Thread.run(Thread.java:662)" ] }, { name: "DestroyJavaVM", priority: 5, state: "RUNNABLE", trace: [ ] }, { name: "Finalizer", priority: 8, state: "WAITING", trace: [ "java.lang.Object.wait(Native Method)", "java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118)", "java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:134)", "java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:171)" ] }, { name: "Format client refresh timer", priority: 5, state: "TIMED_WAITING", trace: [ "java.lang.Object.wait(Native Method)", "java.util.TimerThread.mainLoop(Timer.java:509)", "java.util.TimerThread.run(Timer.java:462)" ] }, { name: "INCompressor (imageWebServicePerm_hraq257l)", priority: 5, state: "TIMED_WAITING", trace: [ "java.lang.Object.wait(Native Method)", "com.sleepycat.je.utilint.DaemonThread.run(DaemonThread.java:174)", "java.lang.Thread.run(Thread.java:662)" ] }, { name: "InvokePhaseTaskStatusService-getUnfinishedTasks 1", priority: 5, state: "WAITING", trace: [ "java.lang.Object.wait(Native Method)", "java.lang.Object.wait(Object.java:485)", "com.company.module.pipeline.queue.EquitableBlockingQueue.take(EquitableBlockingQueue.java:175)", "com.company.module.pipeline.queue.EquitableBlockingQueue.take(EquitableBlockingQueue.java:38)", "com.company.module.pipeline.thread.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:970)", "com.company.module.pipeline.thread.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:930)", "java.lang.Thread.run(Thread.java:662)" ] }, { name: "InvokePhaseTaskStatusService-getUnfinishedTasks 2", priority: 5, state: "WAITING", trace: [ "java.lang.Object.wait(Native Method)", "java.lang.Object.wait(Object.java:485)", "com.company.module.pipeline.queue.EquitableBlockingQueue.take(EquitableBlockingQueue.java:175)", "com.company.module.pipeline.queue.EquitableBlockingQueue.take(EquitableBlockingQueue.java:38)", "com.company.module.pipeline.thread.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:970)", "com.company.module.pipeline.thread.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:930)", "java.lang.Thread.run(Thread.java:662)" ] }, { name: "InvokePhaseTaskStatusService-getUnfinishedTasks 3", priority: 5, state: "WAITING", trace: [ "java.lang.Object.wait(Native Method)", "java.lang.Object.wait(Object.java:485)", "com.company.module.pipeline.queue.EquitableBlockingQueue.take(EquitableBlockingQueue.java:175)", "com.company.module.pipeline.queue.EquitableBlockingQueue.take(EquitableBlockingQueue.java:38)", "com.company.module.pipeline.thread.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:970)", "com.company.module.pipeline.thread.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:930)", "java.lang.Thread.run(Thread.java:662)" ] }, { name: "InvokePhaseTaskStatusService-getUnfinishedTasks 4", priority: 5, state: "WAITING", trace: [ "java.lang.Object.wait(Native Method)", "java.lang.Object.wait(Object.java:485)", "com.company.module.pipeline.queue.EquitableBlockingQueue.take(EquitableBlockingQueue.java:175)", "com.company.module.pipeline.queue.EquitableBlockingQueue.take(EquitableBlockingQueue.java:38)", "com.company.module.pipeline.thread.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:970)", "com.company.module.pipeline.thread.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:930)", "java.lang.Thread.run(Thread.java:662)" ] }, { name: "InvokePhaseTaskStatusService-getUnfinishedTasks 5", priority: 5, state: "WAITING", trace: [ "java.lang.Object.wait(Native Method)", "java.lang.Object.wait(Object.java:485)", "com.company.module.pipeline.queue.EquitableBlockingQueue.take(EquitableBlockingQueue.java:175)", "com.company.module.pipeline.queue.EquitableBlockingQueue.take(EquitableBlockingQueue.java:38)", "com.company.module.pipeline.thread.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:970)", "com.company.module.pipeline.thread.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:930)", "java.lang.Thread.run(Thread.java:662)" ] }, { name: "InvokePhaseimage-computeCandidateMainImages 36", priority: 5, state: "WAITING", trace: [ "java.lang.Object.wait(Native Method)", "java.lang.Object.wait(Object.java:485)", "com.company.module.pipeline.queue.EquitableBlockingQueue.take(EquitableBlockingQueue.java:175)", "com.company.module.pipeline.queue.EquitableBlockingQueue.take(EquitableBlockingQueue.java:38)", "com.company.module.pipeline.thread.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:970)", "com.company.module.pipeline.thread.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:930)", "java.lang.Thread.run(Thread.java:662)" ] }, { name: "InvokePhaseimage-computeCandidateMainImages 37", priority: 5, state: "WAITING", trace: [ "java.lang.Object.wait(Native Method)", "java.lang.Object.wait(Object.java:485)", "com.company.module.pipeline.queue.EquitableBlockingQueue.take(EquitableBlockingQueue.java:175)", "com.company.module.pipeline.queue.EquitableBlockingQueue.take(EquitableBlockingQueue.java:38)", "com.company.module.pipeline.thread.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:970)", "com.company.module.pipeline.thread.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:930)", "java.lang.Thread.run(Thread.java:662)" ] }, { name: "InvokePhaseimage-computeCandidateMainImages 38", priority: 5, state: "WAITING", trace: [ "java.lang.Object.wait(Native Method)", "java.lang.Object.wait(Object.java:485)", "com.company.module.pipeline.queue.EquitableBlockingQueue.take(EquitableBlockingQueue.java:175)", "com.company.module.pipeline.queue.EquitableBlockingQueue.take(EquitableBlockingQueue.java:38)", "com.company.module.pipeline.thread.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:970)", "com.company.module.pipeline.thread.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:930)", "java.lang.Thread.run(Thread.java:662)" ] }, { name: "InvokePhaseimage-computeCandidateMainImages 39", priority: 5, state: "WAITING", trace: [ "java.lang.Object.wait(Native Method)", "java.lang.Object.wait(Object.java:485)", "com.company.module.pipeline.queue.EquitableBlockingQueue.take(EquitableBlockingQueue.java:175)", "com.company.module.pipeline.queue.EquitableBlockingQueue.take(EquitableBlockingQueue.java:38)", "com.company.module.pipeline.thread.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:970)", "com.company.module.pipeline.thread.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:930)", "java.lang.Thread.run(Thread.java:662)" ] }, { name: "InvokePhaseimage-computeCandidateMainImages 40", priority: 5, state: "WAITING", trace: [ "java.lang.Object.wait(Native Method)", "java.lang.Object.wait(Object.java:485)", "com.company.module.pipeline.queue.EquitableBlockingQueue.take(EquitableBlockingQueue.java:175)", "com.company.module.pipeline.queue.EquitableBlockingQueue.take(EquitableBlockingQueue.java:38)", "com.company.module.pipeline.thread.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:970)", "com.company.module.pipeline.thread.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:930)", "java.lang.Thread.run(Thread.java:662)" ] }, { name: "InvokePhaseimage-getQueueSizeByServiceAndFunction 274", priority: 5, state: "WAITING", trace: [ "java.lang.Object.wait(Native Method)", "java.lang.Object.wait(Object.java:485)", "com.company.module.pipeline.queue.EquitableBlockingQueue.take(EquitableBlockingQueue.java:175)", "com.company.module.pipeline.queue.EquitableBlockingQueue.take(EquitableBlockingQueue.java:38)", "com.company.module.pipeline.thread.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:970)", "com.company.module.pipeline.thread.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:930)", "java.lang.Thread.run(Thread.java:662)" ] }, { name: "InvokePhaseimage-getQueueSizeByServiceAndFunction 275", priority: 5, state: "WAITING", trace: [ "java.lang.Object.wait(Native Method)", "java.lang.Object.wait(Object.java:485)", "com.company.module.pipeline.queue.EquitableBlockingQueue.take(EquitableBlockingQueue.java:175)", "com.company.module.pipeline.queue.EquitableBlockingQueue.take(EquitableBlockingQueue.java:38)", "com.company.module.pipeline.thread.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:970)", "com.company.module.pipeline.thread.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:930)", "java.lang.Thread.run(Thread.java:662)" ] }, { name: "InvokePhaseimage-getQueueSizeByServiceAndFunction 276", priority: 5, state: "WAITING", trace: [ "java.lang.Object.wait(Native Method)", "java.lang.Object.wait(Object.java:485)", "com.company.module.pipeline.queue.EquitableBlockingQueue.take(EquitableBlockingQueue.java:175)", "com.company.module.pipeline.queue.EquitableBlockingQueue.take(EquitableBlockingQueue.java:38)", "com.company.module.pipeline.thread.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:970)", "com.company.module.pipeline.thread.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:930)", "java.lang.Thread.run(Thread.java:662)" ] }, { name: "InvokePhaseimage-isMaster 1", priority: 5, state: "TIMED_WAITING", trace: [ "java.lang.Object.wait(Native Method)", "java.lang.Object.wait(Object.java:443)", "com.company.module.pipeline.queue.EquitableBlockingQueue.poll(EquitableBlockingQueue.java:199)", "com.company.module.pipeline.queue.EquitableBlockingQueue.poll(EquitableBlockingQueue.java:38)", "com.company.module.pipeline.thread.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:968)", "com.company.module.pipeline.thread.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:930)", "java.lang.Thread.run(Thread.java:662)" ] }, { name: "InvokePhaseimage-isMaster 2", priority: 5, state: "TIMED_WAITING", trace: [ "java.lang.Object.wait(Native Method)", "java.lang.Object.wait(Object.java:443)", "com.company.module.pipeline.queue.EquitableBlockingQueue.poll(EquitableBlockingQueue.java:199)", "com.company.module.pipeline.queue.EquitableBlockingQueue.poll(EquitableBlockingQueue.java:38)", "com.company.module.pipeline.thread.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:968)", "com.company.module.pipeline.thread.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:930)", "java.lang.Thread.run(Thread.java:662)" ] }, { name: "InvokePhaseimage-isMaster 3", priority: 5, state: "TIMED_WAITING", trace: [ "java.lang.Object.wait(Native Method)", "java.lang.Object.wait(Object.java:443)", "com.company.module.pipeline.queue.EquitableBlockingQueue.poll(EquitableBlockingQueue.java:199)", "com.company.module.pipeline.queue.EquitableBlockingQueue.poll(EquitableBlockingQueue.java:38)", "com.company.module.pipeline.thread.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:968)", "com.company.module.pipeline.thread.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:930)", "java.lang.Thread.run(Thread.java:662)" ] }, { name: "InvokePhaseimage-isMaster 5", priority: 5, state: "TIMED_WAITING", trace: [ "java.lang.Object.wait(Native Method)", "java.lang.Object.wait(Object.java:443)", "com.company.module.pipeline.queue.EquitableBlockingQueue.poll(EquitableBlockingQueue.java:199)", "com.company.module.pipeline.queue.EquitableBlockingQueue.poll(EquitableBlockingQueue.java:38)", "com.company.module.pipeline.thread.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:968)", "com.company.module.pipeline.thread.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:930)", "java.lang.Thread.run(Thread.java:662)" ] }, { name: "InvokePhaseimage-isMaster 6", priority: 5, state: "TIMED_WAITING", trace: [ "java.lang.Object.wait(Native Method)", "java.lang.Object.wait(Object.java:443)", "com.company.module.pipeline.queue.EquitableBlockingQueue.poll(EquitableBlockingQueue.java:199)", "com.company.module.pipeline.queue.EquitableBlockingQueue.poll(EquitableBlockingQueue.java:38)", "com.company.module.pipeline.thread.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:968)", "com.company.module.pipeline.thread.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:930)", "java.lang.Thread.run(Thread.java:662)" ] }, { name: "InvokePhaseimage-isMaster 7", priority: 5, state: "TIMED_WAITING", trace: [ "java.lang.Object.wait(Native Method)", "java.lang.Object.wait(Object.java:443)", "com.company.module.pipeline.queue.EquitableBlockingQueue.poll(EquitableBlockingQueue.java:199)", "com.company.module.pipeline.queue.EquitableBlockingQueue.poll(EquitableBlockingQueue.java:38)", "com.company.module.pipeline.thread.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:968)", "com.company.module.pipeline.thread.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:930)", "java.lang.Thread.run(Thread.java:662)" ] }, { name: "InvokePhaseimage-queueSize 1", priority: 5, state: "WAITING", trace: [ "java.lang.Object.wait(Native Method)", "java.lang.Object.wait(Object.java:485)", "com.company.module.pipeline.queue.EquitableBlockingQueue.take(EquitableBlockingQueue.java:175)", "com.company.module.pipeline.queue.EquitableBlockingQueue.take(EquitableBlockingQueue.java:38)", "com.company.module.pipeline.thread.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:970)", "com.company.module.pipeline.thread.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:930)", "java.lang.Thread.run(Thread.java:662)" ] }, { name: "InvokePhaseimage-queueSize 2", priority: 5, state: "WAITING", trace: [ "java.lang.Object.wait(Native Method)", "java.lang.Object.wait(Object.java:485)", "com.company.module.pipeline.queue.EquitableBlockingQueue.take(EquitableBlockingQueue.java:175)", "com.company.module.pipeline.queue.EquitableBlockingQueue.take(EquitableBlockingQueue.java:38)", "com.company.module.pipeline.thread.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:970)", "com.company.module.pipeline.thread.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:930)", "java.lang.Thread.run(Thread.java:662)" ] }, { name: "InvokePhaseimage-queueSize 3", priority: 5, state: "WAITING", trace: [ "java.lang.Object.wait(Native Method)", "java.lang.Object.wait(Object.java:485)", "com.company.module.pipeline.queue.EquitableBlockingQueue.take(EquitableBlockingQueue.java:175)", "com.company.module.pipeline.queue.EquitableBlockingQueue.take(EquitableBlockingQueue.java:38)", "com.company.module.pipeline.thread.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:970)", "com.company.module.pipeline.thread.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:930)", "java.lang.Thread.run(Thread.java:662)" ] }, { name: "InvokePhaseimage-queueSize 4", priority: 5, state: "WAITING", trace: [ "java.lang.Object.wait(Native Method)", "java.lang.Object.wait(Object.java:485)", "com.company.module.pipeline.queue.EquitableBlockingQueue.take(EquitableBlockingQueue.java:175)", "com.company.module.pipeline.queue.EquitableBlockingQueue.take(EquitableBlockingQueue.java:38)", "com.company.module.pipeline.thread.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:970)", "com.company.module.pipeline.thread.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:930)", "java.lang.Thread.run(Thread.java:662)" ] }, { name: "InvokePhaseimage-queueSize 5", priority: 5, state: "WAITING", trace: [ "java.lang.Object.wait(Native Method)", "java.lang.Object.wait(Object.java:485)", "com.company.module.pipeline.queue.EquitableBlockingQueue.take(EquitableBlockingQueue.java:175)", "com.company.module.pipeline.queue.EquitableBlockingQueue.take(EquitableBlockingQueue.java:38)", "com.company.module.pipeline.thread.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:970)", "com.company.module.pipeline.thread.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:930)", "java.lang.Thread.run(Thread.java:662)" ] }, { name: "InvokePhaseimage-updateMainImages 1", priority: 5, state: "WAITING", trace: [ "java.lang.Object.wait(Native Method)", "java.lang.Object.wait(Object.java:485)", "com.company.module.pipeline.queue.EquitableBlockingQueue.take(EquitableBlockingQueue.java:175)", "com.company.module.pipeline.queue.EquitableBlockingQueue.take(EquitableBlockingQueue.java:38)", "com.company.module.pipeline.thread.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:970)", "com.company.module.pipeline.thread.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:930)", "java.lang.Thread.run(Thread.java:662)" ] }, { name: "InvokePhaseimage-updateMainImages 2", priority: 5, state: "WAITING", trace: [ "java.lang.Object.wait(Native Method)", "java.lang.Object.wait(Object.java:485)", "com.company.module.pipeline.queue.EquitableBlockingQueue.take(EquitableBlockingQueue.java:175)", "com.company.module.pipeline.queue.EquitableBlockingQueue.take(EquitableBlockingQueue.java:38)", "com.company.module.pipeline.thread.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:970)", "com.company.module.pipeline.thread.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:930)", "java.lang.Thread.run(Thread.java:662)" ] }, { name: "InvokePhaseimage-updateMainImages 3", priority: 5, state: "WAITING", trace: [ "java.lang.Object.wait(Native Method)", "java.lang.Object.wait(Object.java:485)", "com.company.module.pipeline.queue.EquitableBlockingQueue.take(EquitableBlockingQueue.java:175)", "com.company.module.pipeline.queue.EquitableBlockingQueue.take(EquitableBlockingQueue.java:38)", "com.company.module.pipeline.thread.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:970)", "com.company.module.pipeline.thread.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:930)", "java.lang.Thread.run(Thread.java:662)" ] }, { name: "InvokePhaseimage-updateMainImages 4", priority: 5, state: "WAITING", trace: [ "java.lang.Object.wait(Native Method)", "java.lang.Object.wait(Object.java:485)", "com.company.module.pipeline.queue.EquitableBlockingQueue.take(EquitableBlockingQueue.java:175)", "com.company.module.pipeline.queue.EquitableBlockingQueue.take(EquitableBlockingQueue.java:38)", "com.company.module.pipeline.thread.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:970)", "com.company.module.pipeline.thread.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:930)", "java.lang.Thread.run(Thread.java:662)" ] }, { name: "InvokePhaseimage-updateMainImages 5", priority: 5, state: "WAITING", trace: [ "java.lang.Object.wait(Native Method)", "java.lang.Object.wait(Object.java:485)", "com.company.module.pipeline.queue.EquitableBlockingQueue.take(EquitableBlockingQueue.java:175)", "com.company.module.pipeline.queue.EquitableBlockingQueue.take(EquitableBlockingQueue.java:38)", "com.company.module.pipeline.thread.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:970)", "com.company.module.pipeline.thread.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:930)", "java.lang.Thread.run(Thread.java:662)" ] }, { name: "JMX server connection timeout 14497", priority: 5, state: "TIMED_WAITING", trace: [ "java.lang.Object.wait(Native Method)", "com.sun.jmx.remote.internal.ServerCommunicatorAdmin$Timeout.run(ServerCommunicatorAdmin.java:150)", "java.lang.Thread.run(Thread.java:662)" ] }, { name: "MultiThreadedHttpConnectionManager cleanup", priority: 5, state: "WAITING", trace: [ "java.lang.Object.wait(Native Method)", "java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118)", "java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:134)", "org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$ReferenceQueueThread.run(MultiThreadedHttpConnectionManager.java:1122)" ] }, { name: "Multicast Heartbeat Receiver: imageWebService_merpocIngest", priority: 5, state: "RUNNABLE", trace: [ "java.net.PlainDatagramSocketImpl.receive0(Native Method)", "java.net.PlainDatagramSocketImpl.receive(PlainDatagramSocketImpl.java:145)", "java.net.DatagramSocket.receive(DatagramSocket.java:725)", "com.company.module.cluster.heartbeat.multicast.MulticastHeartbeatReceiver$HeartbeatReceiverThread.run(MulticastHeartbeatReceiver.java:221)" ] }, { name: "Multicast Heartbeat Sender: imageWebService_merpocIngest", priority: 5, state: "TIMED_WAITING", trace: [ "java.lang.Object.wait(Native Method)", "com.company.module.cluster.heartbeat.multicast.MulticastHeartbeatSender$HeartbeatSenderThread.run(MulticastHeartbeatSender.java:274)" ] }, { name: "Notification Client - DataServiceClient-MainImageType-172.20.153.35", priority: 5, state: "RUNNABLE", trace: [ "java.net.SocketInputStream.socketRead0(Native Method)", "java.net.SocketInputStream.read(SocketInputStream.java:129)", "java.io.BufferedInputStream.fill(BufferedInputStream.java:218)", "java.io.BufferedInputStream.read(BufferedInputStream.java:237)", "org.apache.commons.httpclient.ChunkedInputStream.getChunkSizeFromInputStream(ChunkedInputStream.java:250)", "org.apache.commons.httpclient.ChunkedInputStream.nextChunk(ChunkedInputStream.java:221)", "org.apache.commons.httpclient.ChunkedInputStream.read(ChunkedInputStream.java:176)", "java.io.FilterInputStream.read(FilterInputStream.java:116)", "org.apache.commons.httpclient.AutoCloseInputStream.read(AutoCloseInputStream.java:108)", "java.io.FilterInputStream.read(FilterInputStream.java:90)", "org.apache.commons.httpclient.AutoCloseInputStream.read(AutoCloseInputStream.java:127)", "com.company.data.api.marshalling.json.stream.JsonStreamReaderImpl.nextByte(JsonStreamReaderImpl.java:845)", "com.company.data.api.marshalling.json.stream.JsonStreamReaderImpl.read(JsonStreamReaderImpl.java:826)", "com.company.data.api.marshalling.json.stream.JsonStreamReaderImpl.scanToNext(JsonStreamReaderImpl.java:786)", "com.company.data.api.marshalling.json.stream.JsonStreamReaderImpl.scanToNext(JsonStreamReaderImpl.java:709)", "com.company.data.api.marshalling.json.stream.JsonStreamReaderImpl.initialize(JsonStreamReaderImpl.java:100)", "com.company.data.api.marshalling.json.stream.JsonStreamReaderImpl.nextElement(JsonStreamReaderImpl.java:172)", "com.company.data.api.marshalling.notification.NotificationUnmarshaller.unmarshalNotifications(NotificationUnmarshaller.java:105)", "com.company.data.api.marshalling.notification.NotificationUnmarshaller.unmarshal(NotificationUnmarshaller.java:78)", "com.company.data.notification.api.client.NotificationClient$Poller.parsePayload(NotificationClient.java:1170)", "com.company.data.notification.api.client.NotificationClient$Poller.get(NotificationClient.java:1039)", "com.company.data.notification.api.client.NotificationClient$Poller.run(NotificationClient.java:988)", "java.lang.Thread.run(Thread.java:662)" ] }, { name: "Notification Client - DataServiceClient-MainImageType-172.20.153.35", priority: 5, state: "RUNNABLE", trace: [ "java.net.SocketInputStream.socketRead0(Native Method)", "java.net.SocketInputStream.read(SocketInputStream.java:129)", "java.io.BufferedInputStream.fill(BufferedInputStream.java:218)", "java.io.BufferedInputStream.read(BufferedInputStream.java:237)", "org.apache.commons.httpclient.ChunkedInputStream.getChunkSizeFromInputStream(ChunkedInputStream.java:250)", "org.apache.commons.httpclient.ChunkedInputStream.nextChunk(ChunkedInputStream.java:221)", "org.apache.commons.httpclient.ChunkedInputStream.read(ChunkedInputStream.java:176)", "java.io.FilterInputStream.read(FilterInputStream.java:116)", "org.apache.commons.httpclient.AutoCloseInputStream.read(AutoCloseInputStream.java:108)", "java.io.FilterInputStream.read(FilterInputStream.java:90)", "org.apache.commons.httpclient.AutoCloseInputStream.read(AutoCloseInputStream.java:127)", "com.company.data.api.marshalling.json.stream.JsonStreamReaderImpl.nextByte(JsonStreamReaderImpl.java:845)", "com.company.data.api.marshalling.json.stream.JsonStreamReaderImpl.read(JsonStreamReaderImpl.java:826)", "com.company.data.api.marshalling.json.stream.JsonStreamReaderImpl.scanToNext(JsonStreamReaderImpl.java:786)", "com.company.data.api.marshalling.json.stream.JsonStreamReaderImpl.scanToNext(JsonStreamReaderImpl.java:709)", "com.company.data.api.marshalling.json.stream.JsonStreamReaderImpl.initialize(JsonStreamReaderImpl.java:100)", "com.company.data.api.marshalling.json.stream.JsonStreamReaderImpl.nextElement(JsonStreamReaderImpl.java:172)", "com.company.data.api.marshalling.notification.NotificationUnmarshaller.unmarshalNotifications(NotificationUnmarshaller.java:105)", "com.company.data.api.marshalling.notification.NotificationUnmarshaller.unmarshal(NotificationUnmarshaller.java:78)", "com.company.data.notification.api.client.NotificationClient$Poller.parsePayload(NotificationClient.java:1170)", "com.company.data.notification.api.client.NotificationClient$Poller.get(NotificationClient.java:1039)", "com.company.data.notification.api.client.NotificationClient$Poller.run(NotificationClient.java:988)", "java.lang.Thread.run(Thread.java:662)" ] }, { name: "Notification Client - DataServiceClient-MainImageType-172.20.153.35", priority: 5, state: "RUNNABLE", trace: [ "java.net.SocketInputStream.socketRead0(Native Method)", "java.net.SocketInputStream.read(SocketInputStream.java:129)", "java.io.BufferedInputStream.fill(BufferedInputStream.java:218)", "java.io.BufferedInputStream.read(BufferedInputStream.java:237)", "org.apache.commons.httpclient.ChunkedInputStream.getChunkSizeFromInputStream(ChunkedInputStream.java:250)", "org.apache.commons.httpclient.ChunkedInputStream.nextChunk(ChunkedInputStream.java:221)", "org.apache.commons.httpclient.ChunkedInputStream.read(ChunkedInputStream.java:176)", "java.io.FilterInputStream.read(FilterInputStream.java:116)", "org.apache.commons.httpclient.AutoCloseInputStream.read(AutoCloseInputStream.java:108)", "java.io.FilterInputStream.read(FilterInputStream.java:90)", "org.apache.commons.httpclient.AutoCloseInputStream.read(AutoCloseInputStream.java:127)", "com.company.data.api.marshalling.json.stream.JsonStreamReaderImpl.nextByte(JsonStreamReaderImpl.java:845)", "com.company.data.api.marshalling.json.stream.JsonStreamReaderImpl.read(JsonStreamReaderImpl.java:826)", "com.company.data.api.marshalling.json.stream.JsonStreamReaderImpl.scanToNext(JsonStreamReaderImpl.java:786)", "com.company.data.api.marshalling.json.stream.JsonStreamReaderImpl.scanToNext(JsonStreamReaderImpl.java:709)", "com.company.data.api.marshalling.json.stream.JsonStreamReaderImpl.initialize(JsonStreamReaderImpl.java:100)", "com.company.data.api.marshalling.json.stream.JsonStreamReaderImpl.nextElement(JsonStreamReaderImpl.java:172)", "com.company.data.api.marshalling.notification.NotificationUnmarshaller.unmarshalNotifications(NotificationUnmarshaller.java:105)", "com.company.data.api.marshalling.notification.NotificationUnmarshaller.unmarshal(NotificationUnmarshaller.java:78)", "com.company.data.notification.api.client.NotificationClient$Poller.parsePayload(NotificationClient.java:1170)", "com.company.data.notification.api.client.NotificationClient$Poller.get(NotificationClient.java:1039)", "com.company.data.notification.api.client.NotificationClient$Poller.run(NotificationClient.java:988)", "java.lang.Thread.run(Thread.java:662)" ] }, { name: "Notification Client - DefaultImageCriteriaCacheManager:Entity-172.20.153.35", priority: 5, state: "RUNNABLE", trace: [ "java.net.SocketInputStream.socketRead0(Native Method)", "java.net.SocketInputStream.read(SocketInputStream.java:129)", "java.io.BufferedInputStream.fill(BufferedInputStream.java:218)", "java.io.BufferedInputStream.read(BufferedInputStream.java:237)", "org.apache.commons.httpclient.ChunkedInputStream.getChunkSizeFromInputStream(ChunkedInputStream.java:250)", "org.apache.commons.httpclient.ChunkedInputStream.nextChunk(ChunkedInputStream.java:221)", "org.apache.commons.httpclient.ChunkedInputStream.read(ChunkedInputStream.java:176)", "java.io.FilterInputStream.read(FilterInputStream.java:116)", "org.apache.commons.httpclient.AutoCloseInputStream.read(AutoCloseInputStream.java:108)", "java.io.FilterInputStream.read(FilterInputStream.java:90)", "org.apache.commons.httpclient.AutoCloseInputStream.read(AutoCloseInputStream.java:127)", "com.company.data.api.marshalling.json.stream.JsonStreamReaderImpl.nextByte(JsonStreamReaderImpl.java:845)", "com.company.data.api.marshalling.json.stream.JsonStreamReaderImpl.read(JsonStreamReaderImpl.java:826)", "com.company.data.api.marshalling.json.stream.JsonStreamReaderImpl.scanToNext(JsonStreamReaderImpl.java:786)", "com.company.data.api.marshalling.json.stream.JsonStreamReaderImpl.scanToNext(JsonStreamReaderImpl.java:709)", "com.company.data.api.marshalling.json.stream.JsonStreamReaderImpl.initialize(JsonStreamReaderImpl.java:100)", "com.company.data.api.marshalling.json.stream.JsonStreamReaderImpl.nextElement(JsonStreamReaderImpl.java:172)", "com.company.data.api.marshalling.notification.NotificationUnmarshaller.unmarshalNotifications(NotificationUnmarshaller.java:105)", "com.company.data.api.marshalling.notification.NotificationUnmarshaller.unmarshal(NotificationUnmarshaller.java:78)", "com.company.data.notification.api.client.NotificationClient$Poller.parsePayload(NotificationClient.java:1170)", "com.company.data.notification.api.client.NotificationClient$Poller.get(NotificationClient.java:1039)", "com.company.data.notification.api.client.NotificationClient$Poller.run(NotificationClient.java:988)", "java.lang.Thread.run(Thread.java:662)" ] }, { name: "Notification Client - DefaultImageCriteriaCacheManager:Linear-172.20.153.35", priority: 5, state: "RUNNABLE", trace: [ "java.net.SocketInputStream.socketRead0(Native Method)", "java.net.SocketInputStream.read(SocketInputStream.java:129)", "java.io.BufferedInputStream.fill(BufferedInputStream.java:218)", "java.io.BufferedInputStream.read(BufferedInputStream.java:237)", "org.apache.commons.httpclient.ChunkedInputStream.getChunkSizeFromInputStream(ChunkedInputStream.java:250)", "org.apache.commons.httpclient.ChunkedInputStream.nextChunk(ChunkedInputStream.java:221)", "org.apache.commons.httpclient.ChunkedInputStream.read(ChunkedInputStream.java:176)", "java.io.FilterInputStream.read(FilterInputStream.java:116)", "org.apache.commons.httpclient.AutoCloseInputStream.read(AutoCloseInputStream.java:108)", "java.io.FilterInputStream.read(FilterInputStream.java:90)", "org.apache.commons.httpclient.AutoCloseInputStream.read(AutoCloseInputStream.java:127)", "com.company.data.api.marshalling.json.stream.JsonStreamReaderImpl.nextByte(JsonStreamReaderImpl.java:845)", "com.company.data.api.marshalling.json.stream.JsonStreamReaderImpl.read(JsonStreamReaderImpl.java:826)", "com.company.data.api.marshalling.json.stream.JsonStreamReaderImpl.scanToNext(JsonStreamReaderImpl.java:786)", "com.company.data.api.marshalling.json.stream.JsonStreamReaderImpl.scanToNext(JsonStreamReaderImpl.java:709)", "com.company.data.api.marshalling.json.stream.JsonStreamReaderImpl.initialize(JsonStreamReaderImpl.java:100)", "com.company.data.api.marshalling.json.stream.JsonStreamReaderImpl.nextElement(JsonStreamReaderImpl.java:172)", "com.company.data.api.marshalling.notification.NotificationUnmarshaller.unmarshalNotifications(NotificationUnmarshaller.java:105)", "com.company.data.api.marshalling.notification.NotificationUnmarshaller.unmarshal(NotificationUnmarshaller.java:78)", "com.company.data.notification.api.client.NotificationClient$Poller.parsePayload(NotificationClient.java:1170)", "com.company.data.notification.api.client.NotificationClient$Poller.get(NotificationClient.java:1039)", "com.company.data.notification.api.client.NotificationClient$Poller.run(NotificationClient.java:988)", "java.lang.Thread.run(Thread.java:662)" ] }, { name: "Notification Client - DefaultImageCriteriaCacheManager:Menu-172.20.153.35", priority: 5, state: "RUNNABLE", trace: [ "java.net.SocketInputStream.socketRead0(Native Method)", "java.net.SocketInputStream.read(SocketInputStream.java:129)", "java.io.BufferedInputStream.fill(BufferedInputStream.java:218)", "java.io.BufferedInputStream.read(BufferedInputStream.java:237)", "org.apache.commons.httpclient.ChunkedInputStream.getChunkSizeFromInputStream(ChunkedInputStream.java:250)", "org.apache.commons.httpclient.ChunkedInputStream.nextChunk(ChunkedInputStream.java:221)", "org.apache.commons.httpclient.ChunkedInputStream.read(ChunkedInputStream.java:176)", "java.io.FilterInputStream.read(FilterInputStream.java:116)", "org.apache.commons.httpclient.AutoCloseInputStream.read(AutoCloseInputStream.java:108)", "java.io.FilterInputStream.read(FilterInputStream.java:90)", "org.apache.commons.httpclient.AutoCloseInputStream.read(AutoCloseInputStream.java:127)", "com.company.data.api.marshalling.json.stream.JsonStreamReaderImpl.nextByte(JsonStreamReaderImpl.java:845)", "com.company.data.api.marshalling.json.stream.JsonStreamReaderImpl.read(JsonStreamReaderImpl.java:826)", "com.company.data.api.marshalling.json.stream.JsonStreamReaderImpl.scanToNext(JsonStreamReaderImpl.java:786)", "com.company.data.api.marshalling.json.stream.JsonStreamReaderImpl.scanToNext(JsonStreamReaderImpl.java:709)", "com.company.data.api.marshalling.json.stream.JsonStreamReaderImpl.initialize(JsonStreamReaderImpl.java:100)", "com.company.data.api.marshalling.json.stream.JsonStreamReaderImpl.nextElement(JsonStreamReaderImpl.java:172)", "com.company.data.api.marshalling.notification.NotificationUnmarshaller.unmarshalNotifications(NotificationUnmarshaller.java:105)", "com.company.data.api.marshalling.notification.NotificationUnmarshaller.unmarshal(NotificationUnmarshaller.java:78)", "com.company.data.notification.api.client.NotificationClient$Poller.parsePayload(NotificationClient.java:1170)", "com.company.data.notification.api.client.NotificationClient$Poller.get(NotificationClient.java:1039)", "com.company.data.notification.api.client.NotificationClient$Poller.run(NotificationClient.java:988)", "java.lang.Thread.run(Thread.java:662)" ] }, { name: "Notification Client - [Album] NotificationListener-172.20.153.35", priority: 5, state: "RUNNABLE", trace: [ "java.net.SocketInputStream.socketRead0(Native Method)", "java.net.SocketInputStream.read(SocketInputStream.java:129)", "java.io.BufferedInputStream.fill(BufferedInputStream.java:218)", "java.io.BufferedInputStream.read(BufferedInputStream.java:237)", "org.apache.commons.httpclient.ChunkedInputStream.getChunkSizeFromInputStream(ChunkedInputStream.java:250)", "org.apache.commons.httpclient.ChunkedInputStream.nextChunk(ChunkedInputStream.java:221)", "org.apache.commons.httpclient.ChunkedInputStream.read(ChunkedInputStream.java:176)", "java.io.FilterInputStream.read(FilterInputStream.java:116)", "org.apache.commons.httpclient.AutoCloseInputStream.read(AutoCloseInputStream.java:108)", "java.io.FilterInputStream.read(FilterInputStream.java:90)", "org.apache.commons.httpclient.AutoCloseInputStream.read(AutoCloseInputStream.java:127)", "com.company.data.api.marshalling.json.stream.JsonStreamReaderImpl.nextByte(JsonStreamReaderImpl.java:845)", "com.company.data.api.marshalling.json.stream.JsonStreamReaderImpl.read(JsonStreamReaderImpl.java:826)", "com.company.data.api.marshalling.json.stream.JsonStreamReaderImpl.scanToNext(JsonStreamReaderImpl.java:786)", "com.company.data.api.marshalling.json.stream.JsonStreamReaderImpl.scanToNext(JsonStreamReaderImpl.java:709)", "com.company.data.api.marshalling.json.stream.JsonStreamReaderImpl.initialize(JsonStreamReaderImpl.java:100)", "com.company.data.api.marshalling.json.stream.JsonStreamReaderImpl.nextElement(JsonStreamReaderImpl.java:172)", "com.company.data.api.marshalling.notification.NotificationUnmarshaller.unmarshalNotifications(NotificationUnmarshaller.java:105)", "com.company.data.api.marshalling.notification.NotificationUnmarshaller.unmarshal(NotificationUnmarshaller.java:78)", "com.company.data.notification.api.client.NotificationClient$Poller.parsePayload(NotificationClient.java:1170)", "com.company.data.notification.api.client.NotificationClient$Poller.get(NotificationClient.java:1039)", "com.company.data.notification.api.client.NotificationClient$Poller.run(NotificationClient.java:988)", "java.lang.Thread.run(Thread.java:662)" ] }, { name: "Notification Client - [ImageAssociation] NotificationListener-172.20.153.35", priority: 5, state: "RUNNABLE", trace: [ "java.net.SocketOutputStream.socketWrite0(Native Method)", "java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:92)", "java.net.SocketOutputStream.write(SocketOutputStream.java:136)", "java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65)", "java.io.BufferedOutputStream.flush(BufferedOutputStream.java:123)", "java.io.DataOutputStream.flush(DataOutputStream.java:106)", "com.rabbitmq.client.impl.SocketFrameHandler.flush(SocketFrameHandler.java:142)", "com.rabbitmq.client.impl.AMQConnection.flush(AMQConnection.java:488)", "com.rabbitmq.client.impl.AMQCommand.transmit(AMQCommand.java:125)", "com.rabbitmq.client.impl.AMQChannel.quiescingTransmit(AMQChannel.java:316)", "com.rabbitmq.client.impl.AMQChannel.transmit(AMQChannel.java:292)", "com.rabbitmq.client.impl.ChannelN.basicPublish(ChannelN.java:636)", "sun.reflect.GeneratedMethodAccessor118.invoke(Unknown Source)", "sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)", "java.lang.reflect.Method.invoke(Method.java:597)", "org.springframework.amqp.rabbit.connection.CachingConnectionFactory$CachedChannelInvocationHandler.invoke(CachingConnectionFactory.java:348)", "com.sun.proxy.$Proxy50.basicPublish(Unknown Source)", "org.springframework.amqp.rabbit.core.RabbitTemplate.doSend(RabbitTemplate.java:675)", "org.springframework.amqp.rabbit.core.RabbitTemplate$1.doInRabbit(RabbitTemplate.java:351)", "org.springframework.amqp.rabbit.core.RabbitTemplate.execute(RabbitTemplate.java:627)", "org.springframework.amqp.rabbit.core.RabbitTemplate.send(RabbitTemplate.java:349)", "org.springframework.amqp.rabbit.core.RabbitTemplate.send(RabbitTemplate.java:343)", "org.springframework.amqp.rabbit.core.RabbitTemplate.send(RabbitTemplate.java:335)", "com.company.web.tv.image.impl.messaging.Producer.queueMessage(Producer.java:40)", "com.company.web.tv.image.impl.util.IterativeEntityProcessor.queueEntity(IterativeEntityProcessor.java:149)", "com.company.web.tv.image.impl.util.IterativeEntityProcessor.queueMitsForOriginalEntity(IterativeEntityProcessor.java:71)", "com.company.web.tv.image.impl.util.IterativeEntityProcessor.queueItemsForEntity(IterativeEntityProcessor.java:36)", "com.company.web.tv.image.impl.notifications.DataServiceNotificationProcessor.process(DataServiceNotificationProcessor.java:58)", "com.company.web.tv.image.impl.notifications.DataServiceNotificationProcessor.process(DataServiceNotificationProcessor.java:27)", "com.company.web.tv.image.impl.notifications.ImageWebServiceNotificationListener.receiveNotifications(ImageWebServiceNotificationListener.java:221)", "com.company.data.notification.api.client.NotificationClient$Poller.parsePayload(NotificationClient.java:1240)", "com.company.data.notification.api.client.NotificationClient$Poller.get(NotificationClient.java:1039)", "com.company.data.notification.api.client.NotificationClient$Poller.run(NotificationClient.java:988)", "java.lang.Thread.run(Thread.java:662)" ] }, { name: "Notification Client - [ImageAssociation] NotificationListener-172.20.153.35", priority: 5, state: "RUNNABLE", trace: [ "java.net.SocketInputStream.socketRead0(Native Method)", "java.net.SocketInputStream.read(SocketInputStream.java:129)", "java.io.BufferedInputStream.fill(BufferedInputStream.java:218)", "java.io.BufferedInputStream.read(BufferedInputStream.java:237)", "org.apache.commons.httpclient.ChunkedInputStream.getChunkSizeFromInputStream(ChunkedInputStream.java:250)", "org.apache.commons.httpclient.ChunkedInputStream.nextChunk(ChunkedInputStream.java:221)", "org.apache.commons.httpclient.ChunkedInputStream.read(ChunkedInputStream.java:176)", "java.io.FilterInputStream.read(FilterInputStream.java:116)", "org.apache.commons.httpclient.AutoCloseInputStream.read(AutoCloseInputStream.java:108)", "java.io.FilterInputStream.read(FilterInputStream.java:90)", "org.apache.commons.httpclient.AutoCloseInputStream.read(AutoCloseInputStream.java:127)", "com.company.data.api.marshalling.json.stream.JsonStreamReaderImpl.nextByte(JsonStreamReaderImpl.java:845)", "com.company.data.api.marshalling.json.stream.JsonStreamReaderImpl.read(JsonStreamReaderImpl.java:826)", "com.company.data.api.marshalling.json.stream.JsonStreamReaderImpl.scanToNext(JsonStreamReaderImpl.java:786)", "com.company.data.api.marshalling.json.stream.JsonStreamReaderImpl.scanToNext(JsonStreamReaderImpl.java:709)", "com.company.data.api.marshalling.json.stream.JsonStreamReaderImpl.initialize(JsonStreamReaderImpl.java:100)", "com.company.data.api.marshalling.json.stream.JsonStreamReaderImpl.nextElement(JsonStreamReaderImpl.java:172)", "com.company.data.api.marshalling.notification.NotificationUnmarshaller.unmarshalNotifications(NotificationUnmarshaller.java:105)", "com.company.data.api.marshalling.notification.NotificationUnmarshaller.unmarshal(NotificationUnmarshaller.java:78)", "com.company.data.notification.api.client.NotificationClient$Poller.parsePayload(NotificationClient.java:1170)", "com.company.data.notification.api.client.NotificationClient$Poller.get(NotificationClient.java:1039)", "com.company.data.notification.api.client.NotificationClient$Poller.run(NotificationClient.java:988)", "java.lang.Thread.run(Thread.java:662)" ] }, { name: "Notification Client - [ImageAssociation] NotificationListener-172.20.153.35", priority: 5, state: "RUNNABLE", trace: [ "java.net.SocketInputStream.socketRead0(Native Method)", "java.net.SocketInputStream.read(SocketInputStream.java:129)", "java.io.BufferedInputStream.fill(BufferedInputStream.java:218)", "java.io.BufferedInputStream.read(BufferedInputStream.java:237)", "org.apache.commons.httpclient.ChunkedInputStream.getChunkSizeFromInputStream(ChunkedInputStream.java:250)", "org.apache.commons.httpclient.ChunkedInputStream.nextChunk(ChunkedInputStream.java:221)", "org.apache.commons.httpclient.ChunkedInputStream.read(ChunkedInputStream.java:176)", "java.io.FilterInputStream.read(FilterInputStream.java:116)", "org.apache.commons.httpclient.AutoCloseInputStream.read(AutoCloseInputStream.java:108)", "java.io.FilterInputStream.read(FilterInputStream.java:90)", "org.apache.commons.httpclient.AutoCloseInputStream.read(AutoCloseInputStream.java:127)", "com.company.data.api.marshalling.json.stream.JsonStreamReaderImpl.nextByte(JsonStreamReaderImpl.java:845)", "com.company.data.api.marshalling.json.stream.JsonStreamReaderImpl.read(JsonStreamReaderImpl.java:826)", "com.company.data.api.marshalling.json.stream.JsonStreamReaderImpl.scanToNext(JsonStreamReaderImpl.java:786)", "com.company.data.api.marshalling.json.stream.JsonStreamReaderImpl.scanToNext(JsonStreamReaderImpl.java:709)", "com.company.data.api.marshalling.json.stream.JsonStreamReaderImpl.initialize(JsonStreamReaderImpl.java:100)", "com.company.data.api.marshalling.json.stream.JsonStreamReaderImpl.nextElement(JsonStreamReaderImpl.java:172)", "com.company.data.api.marshalling.notification.NotificationUnmarshaller.unmarshalNotifications(NotificationUnmarshaller.java:105)", "com.company.data.api.marshalling.notification.NotificationUnmarshaller.unmarshal(NotificationUnmarshaller.java:78)", "com.company.data.notification.api.client.NotificationClient$Poller.parsePayload(NotificationClient.java:1170)", "com.company.data.notification.api.client.NotificationClient$Poller.get(NotificationClient.java:1039)", "com.company.data.notification.api.client.NotificationClient$Poller.run(NotificationClient.java:988)", "java.lang.Thread.run(Thread.java:662)" ] }, { name: "Notification Client - [Media] NotificationListener-172.20.153.35", priority: 5, state: "WAITING", trace: [ "java.lang.Object.wait(Native Method)", "java.lang.Object.wait(Object.java:485)", "com.rabbitmq.utility.BlockingCell.get(BlockingCell.java:50)", "com.rabbitmq.utility.BlockingCell.get(BlockingCell.java:65)", "com.rabbitmq.utility.BlockingCell.uninterruptibleGet(BlockingCell.java:111)", "com.rabbitmq.utility.BlockingValueOrException.uninterruptibleGetValue(BlockingValueOrException.java:37)", "com.rabbitmq.client.impl.AMQChannel$BlockingRpcContinuation.getReply(AMQChannel.java:349)", "com.rabbitmq.client.impl.ChannelN.close(ChannelN.java:569)", "com.rabbitmq.client.impl.ChannelN.close(ChannelN.java:501)", "com.rabbitmq.client.impl.ChannelN.close(ChannelN.java:494)", "org.springframework.amqp.rabbit.connection.CachingConnectionFactory$CachedChannelInvocationHandler.physicalClose(CachingConnectionFactory.java:398)", "org.springframework.amqp.rabbit.connection.CachingConnectionFactory$CachedChannelInvocationHandler.invoke(CachingConnectionFactory.java:331)", "com.sun.proxy.$Proxy50.close(Unknown Source)", "org.springframework.amqp.rabbit.connection.RabbitUtils.closeChannel(RabbitUtils.java:64)", "org.springframework.amqp.rabbit.connection.ConnectionFactoryUtils.releaseResources(ConnectionFactoryUtils.java:151)", "org.springframework.amqp.rabbit.core.RabbitTemplate.execute(RabbitTemplate.java:634)", "org.springframework.amqp.rabbit.core.RabbitTemplate.send(RabbitTemplate.java:349)", "org.springframework.amqp.rabbit.core.RabbitTemplate.send(RabbitTemplate.java:343)", "org.springframework.amqp.rabbit.core.RabbitTemplate.send(RabbitTemplate.java:335)", "com.company.web.tv.image.impl.messaging.Producer.queueMessage(Producer.java:40)", "com.company.web.tv.image.impl.util.IterativeEntityProcessor.queueEntity(IterativeEntityProcessor.java:149)", "com.company.web.tv.image.impl.util.IterativeEntityProcessor.queueChildren(IterativeEntityProcessor.java:123)", "com.company.web.tv.image.impl.util.IterativeEntityProcessor.queueMitsForChildren(IterativeEntityProcessor.java:62)", "com.company.web.tv.image.impl.util.IterativeEntityProcessor.queueItemsForEntity(IterativeEntityProcessor.java:39)", "com.company.web.tv.image.impl.notifications.DataServiceNotificationProcessor.process(DataServiceNotificationProcessor.java:58)", "com.company.web.tv.image.impl.notifications.DataServiceNotificationProcessor.process(DataServiceNotificationProcessor.java:27)", "com.company.web.tv.image.impl.notifications.ImageWebServiceNotificationListener.receiveNotifications(ImageWebServiceNotificationListener.java:221)", "com.company.data.notification.api.client.NotificationClient$Poller.parsePayload(NotificationClient.java:1240)", "com.company.data.notification.api.client.NotificationClient$Poller.get(NotificationClient.java:1039)", "com.company.data.notification.api.client.NotificationClient$Poller.run(NotificationClient.java:988)", "java.lang.Thread.run(Thread.java:662)" ] }, { name: "Notification Client - [Program] NotificationListener-172.20.153.35", priority: 5, state: "WAITING", trace: [ "java.lang.Object.wait(Native Method)", "java.lang.Object.wait(Object.java:485)", "com.rabbitmq.utility.BlockingCell.get(BlockingCell.java:50)", "com.rabbitmq.utility.BlockingCell.get(BlockingCell.java:65)", "com.rabbitmq.utility.BlockingCell.uninterruptibleGet(BlockingCell.java:111)", "com.rabbitmq.utility.BlockingValueOrException.uninterruptibleGetValue(BlockingValueOrException.java:37)", "com.rabbitmq.client.impl.AMQChannel$BlockingRpcContinuation.getReply(AMQChannel.java:349)", "com.rabbitmq.client.impl.ChannelN.close(ChannelN.java:569)", "com.rabbitmq.client.impl.ChannelN.close(ChannelN.java:501)", "com.rabbitmq.client.impl.ChannelN.close(ChannelN.java:494)", "org.springframework.amqp.rabbit.connection.CachingConnectionFactory$CachedChannelInvocationHandler.physicalClose(CachingConnectionFactory.java:398)", "org.springframework.amqp.rabbit.connection.CachingConnectionFactory$CachedChannelInvocationHandler.invoke(CachingConnectionFactory.java:331)", "com.sun.proxy.$Proxy50.close(Unknown Source)", "org.springframework.amqp.rabbit.connection.RabbitUtils.closeChannel(RabbitUtils.java:64)", "org.springframework.amqp.rabbit.connection.ConnectionFactoryUtils.releaseResources(ConnectionFactoryUtils.java:151)", "org.springframework.amqp.rabbit.core.RabbitTemplate.execute(RabbitTemplate.java:634)", "org.springframework.amqp.rabbit.core.RabbitTemplate.send(RabbitTemplate.java:349)", "org.springframework.amqp.rabbit.core.RabbitTemplate.send(RabbitTemplate.java:343)", "org.springframework.amqp.rabbit.core.RabbitTemplate.send(RabbitTemplate.java:335)", "com.company.web.tv.image.impl.messaging.Producer.queueMessage(Producer.java:40)", "com.company.web.tv.image.impl.util.IterativeEntityProcessor.queueEntity(IterativeEntityProcessor.java:149)", "com.company.web.tv.image.impl.util.IterativeEntityProcessor.queueMitsForOriginalEntity(IterativeEntityProcessor.java:71)", "com.company.web.tv.image.impl.util.IterativeEntityProcessor.queueItemsForEntity(IterativeEntityProcessor.java:36)", "com.company.web.tv.image.impl.notifications.DataServiceNotificationProcessor.process(DataServiceNotificationProcessor.java:58)", "com.company.web.tv.image.impl.notifications.DataServiceNotificationProcessor.process(DataServiceNotificationProcessor.java:27)", "com.company.web.tv.image.impl.notifications.ImageWebServiceNotificationListener.receiveNotifications(ImageWebServiceNotificationListener.java:221)", "com.company.data.notification.api.client.NotificationClient$Poller.parsePayload(NotificationClient.java:1240)", "com.company.data.notification.api.client.NotificationClient$Poller.get(NotificationClient.java:1039)", "com.company.data.notification.api.client.NotificationClient$Poller.run(NotificationClient.java:988)", "java.lang.Thread.run(Thread.java:662)" ] }, { name: "Notification Client - [Song] NotificationListener-172.20.153.35", priority: 5, state: "RUNNABLE", trace: [ "java.net.SocketInputStream.socketRead0(Native Method)", "java.net.SocketInputStream.read(SocketInputStream.java:129)", "java.io.BufferedInputStream.fill(BufferedInputStream.java:218)", "java.io.BufferedInputStream.read(BufferedInputStream.java:237)", "org.apache.commons.httpclient.ChunkedInputStream.getChunkSizeFromInputStream(ChunkedInputStream.java:250)", "org.apache.commons.httpclient.ChunkedInputStream.nextChunk(ChunkedInputStream.java:221)", "org.apache.commons.httpclient.ChunkedInputStream.read(ChunkedInputStream.java:176)", "java.io.FilterInputStream.read(FilterInputStream.java:116)", "org.apache.commons.httpclient.AutoCloseInputStream.read(AutoCloseInputStream.java:108)", "java.io.FilterInputStream.read(FilterInputStream.java:90)", "org.apache.commons.httpclient.AutoCloseInputStream.read(AutoCloseInputStream.java:127)", "com.company.data.api.marshalling.json.stream.JsonStreamReaderImpl.nextByte(JsonStreamReaderImpl.java:845)", "com.company.data.api.marshalling.json.stream.JsonStreamReaderImpl.read(JsonStreamReaderImpl.java:826)", "com.company.data.api.marshalling.json.stream.JsonStreamReaderImpl.scanToNext(JsonStreamReaderImpl.java:786)", "com.company.data.api.marshalling.json.stream.JsonStreamReaderImpl.scanToNext(JsonStreamReaderImpl.java:709)", "com.company.data.api.marshalling.json.stream.JsonStreamReaderImpl.initialize(JsonStreamReaderImpl.java:100)", "com.company.data.api.marshalling.json.stream.JsonStreamReaderImpl.nextElement(JsonStreamReaderImpl.java:172)", "com.company.data.api.marshalling.notification.NotificationUnmarshaller.unmarshalNotifications(NotificationUnmarshaller.java:105)", "com.company.data.api.marshalling.notification.NotificationUnmarshaller.unmarshal(NotificationUnmarshaller.java:78)", "com.company.data.notification.api.client.NotificationClient$Poller.parsePayload(NotificationClient.java:1170)", "com.company.data.notification.api.client.NotificationClient$Poller.get(NotificationClient.java:1039)", "com.company.data.notification.api.client.NotificationClient$Poller.run(NotificationClient.java:988)", "java.lang.Thread.run(Thread.java:662)" ] }, { name: "Notification Client - [SportsTeam] NotificationListener-172.20.153.35", priority: 5, state: "RUNNABLE", trace: [ "java.net.SocketInputStream.socketRead0(Native Method)", "java.net.SocketInputStream.read(SocketInputStream.java:129)", "java.io.BufferedInputStream.fill(BufferedInputStream.java:218)", "java.io.BufferedInputStream.read(BufferedInputStream.java:237)", "org.apache.commons.httpclient.ChunkedInputStream.getChunkSizeFromInputStream(ChunkedInputStream.java:250)", "org.apache.commons.httpclient.ChunkedInputStream.nextChunk(ChunkedInputStream.java:221)", "org.apache.commons.httpclient.ChunkedInputStream.read(ChunkedInputStream.java:176)", "java.io.FilterInputStream.read(FilterInputStream.java:116)", "org.apache.commons.httpclient.AutoCloseInputStream.read(AutoCloseInputStream.java:108)", "java.io.FilterInputStream.read(FilterInputStream.java:90)", "org.apache.commons.httpclient.AutoCloseInputStream.read(AutoCloseInputStream.java:127)", "com.company.data.api.marshalling.json.stream.JsonStreamReaderImpl.nextByte(JsonStreamReaderImpl.java:845)", "com.company.data.api.marshalling.json.stream.JsonStreamReaderImpl.read(JsonStreamReaderImpl.java:826)", "com.company.data.api.marshalling.json.stream.JsonStreamReaderImpl.scanToNext(JsonStreamReaderImpl.java:786)", "com.company.data.api.marshalling.json.stream.JsonStreamReaderImpl.scanToNext(JsonStreamReaderImpl.java:709)", "com.company.data.api.marshalling.json.stream.JsonStreamReaderImpl.initialize(JsonStreamReaderImpl.java:100)", "com.company.data.api.marshalling.json.stream.JsonStreamReaderImpl.nextElement(JsonStreamReaderImpl.java:172)", "com.company.data.api.marshalling.notification.NotificationUnmarshaller.unmarshalNotifications(NotificationUnmarshaller.java:105)", "com.company.data.api.marshalling.notification.NotificationUnmarshaller.unmarshal(NotificationUnmarshaller.java:78)", "com.company.data.notification.api.client.NotificationClient$Poller.parsePayload(NotificationClient.java:1170)", "com.company.data.notification.api.client.NotificationClient$Poller.get(NotificationClient.java:1039)", "com.company.data.notification.api.client.NotificationClient$Poller.run(NotificationClient.java:988)", "java.lang.Thread.run(Thread.java:662)" ] }, { name: "Notification Client - [StationCompany] NotificationListener-172.20.153.35", priority: 5, state: "RUNNABLE", trace: [ "java.net.SocketInputStream.socketRead0(Native Method)", "java.net.SocketInputStream.read(SocketInputStream.java:129)", "java.io.BufferedInputStream.fill(BufferedInputStream.java:218)", "java.io.BufferedInputStream.read(BufferedInputStream.java:237)", "org.apache.commons.httpclient.ChunkedInputStream.getChunkSizeFromInputStream(ChunkedInputStream.java:250)", "org.apache.commons.httpclient.ChunkedInputStream.nextChunk(ChunkedInputStream.java:221)", "org.apache.commons.httpclient.ChunkedInputStream.read(ChunkedInputStream.java:176)", "java.io.FilterInputStream.read(FilterInputStream.java:116)", "org.apache.commons.httpclient.AutoCloseInputStream.read(AutoCloseInputStream.java:108)", "java.io.FilterInputStream.read(FilterInputStream.java:90)", "org.apache.commons.httpclient.AutoCloseInputStream.read(AutoCloseInputStream.java:127)", "com.company.data.api.marshalling.json.stream.JsonStreamReaderImpl.nextByte(JsonStreamReaderImpl.java:845)", "com.company.data.api.marshalling.json.stream.JsonStreamReaderImpl.read(JsonStreamReaderImpl.java:826)", "com.company.data.api.marshalling.json.stream.JsonStreamReaderImpl.scanToNext(JsonStreamReaderImpl.java:786)", "com.company.data.api.marshalling.json.stream.JsonStreamReaderImpl.scanToNext(JsonStreamReaderImpl.java:709)", "com.company.data.api.marshalling.json.stream.JsonStreamReaderImpl.initialize(JsonStreamReaderImpl.java:100)", "com.company.data.api.marshalling.json.stream.JsonStreamReaderImpl.nextElement(JsonStreamReaderImpl.java:172)", "com.company.data.api.marshalling.notification.NotificationUnmarshaller.unmarshalNotifications(NotificationUnmarshaller.java:105)", "com.company.data.api.marshalling.notification.NotificationUnmarshaller.unmarshal(NotificationUnmarshaller.java:78)", "com.company.data.notification.api.client.NotificationClient$Poller.parsePayload(NotificationClient.java:1170)", "com.company.data.notification.api.client.NotificationClient$Poller.get(NotificationClient.java:1039)", "com.company.data.notification.api.client.NotificationClient$Poller.run(NotificationClient.java:988)", "java.lang.Thread.run(Thread.java:662)" ] }, { name: "Notification Client - [Station] NotificationListener-172.20.153.35", priority: 5, state: "RUNNABLE", trace: [ "java.net.SocketInputStream.socketRead0(Native Method)", "java.net.SocketInputStream.read(SocketInputStream.java:129)", "java.io.BufferedInputStream.fill(BufferedInputStream.java:218)", "java.io.BufferedInputStream.read(BufferedInputStream.java:237)", "org.apache.commons.httpclient.ChunkedInputStream.getChunkSizeFromInputStream(ChunkedInputStream.java:250)", "org.apache.commons.httpclient.ChunkedInputStream.nextChunk(ChunkedInputStream.java:221)", "org.apache.commons.httpclient.ChunkedInputStream.read(ChunkedInputStream.java:176)", "java.io.FilterInputStream.read(FilterInputStream.java:116)", "org.apache.commons.httpclient.AutoCloseInputStream.read(AutoCloseInputStream.java:108)", "java.io.FilterInputStream.read(FilterInputStream.java:90)", "org.apache.commons.httpclient.AutoCloseInputStream.read(AutoCloseInputStream.java:127)", "com.company.data.api.marshalling.json.stream.JsonStreamReaderImpl.nextByte(JsonStreamReaderImpl.java:845)", "com.company.data.api.marshalling.json.stream.JsonStreamReaderImpl.read(JsonStreamReaderImpl.java:826)", "com.company.data.api.marshalling.json.stream.JsonStreamReaderImpl.scanToNext(JsonStreamReaderImpl.java:786)", "com.company.data.api.marshalling.json.stream.JsonStreamReaderImpl.scanToNext(JsonStreamReaderImpl.java:709)", "com.company.data.api.marshalling.json.stream.JsonStreamReaderImpl.initialize(JsonStreamReaderImpl.java:100)", "com.company.data.api.marshalling.json.stream.JsonStreamReaderImpl.nextElement(JsonStreamReaderImpl.java:172)", "com.company.data.api.marshalling.notification.NotificationUnmarshaller.unmarshalNotifications(NotificationUnmarshaller.java:105)", "com.company.data.api.marshalling.notification.NotificationUnmarshaller.unmarshal(NotificationUnmarshaller.java:78)", "com.company.data.notification.api.client.NotificationClient$Poller.parsePayload(NotificationClient.java:1170)", "com.company.data.notification.api.client.NotificationClient$Poller.get(NotificationClient.java:1039)", "com.company.data.notification.api.client.NotificationClient$Poller.run(NotificationClient.java:988)", "java.lang.Thread.run(Thread.java:662)" ] }, { name: "Notification Client - [TagAssociation] NotificationListener-172.20.153.35", priority: 5, state: "RUNNABLE", trace: [ "java.net.SocketInputStream.socketRead0(Native Method)", "java.net.SocketInputStream.read(SocketInputStream.java:129)", "java.io.BufferedInputStream.fill(BufferedInputStream.java:218)", "java.io.BufferedInputStream.read(BufferedInputStream.java:237)", "org.apache.commons.httpclient.HttpParser.readRawLine(HttpParser.java:78)", "org.apache.commons.httpclient.HttpParser.readLine(HttpParser.java:106)", "org.apache.commons.httpclient.HttpConnection.readLine(HttpConnection.java:1116)", "org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.readLine(MultiThreadedHttpConnectionManager.java:1413)", "org.apache.commons.httpclient.HttpMethodBase.readStatusLine(HttpMethodBase.java:1973)", "org.apache.commons.httpclient.HttpMethodBase.readResponse(HttpMethodBase.java:1735)", "org.apache.commons.httpclient.HttpMethodBase.execute(HttpMethodBase.java:1098)", "org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:398)", "org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171)", "org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397)", "org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323)", "com.company.data.notification.api.client.NotificationClient$Poller.get(NotificationClient.java:1024)", "com.company.data.notification.api.client.NotificationClient$Poller.run(NotificationClient.java:988)", "java.lang.Thread.run(Thread.java:662)" ] }, { name: "Peer Client //172.20.153.34:35631", priority: 5, state: "WAITING", trace: [ "java.lang.Object.wait(Native Method)", "java.lang.Object.wait(Object.java:485)", "com.company.module.cluster.peer.socket.SocketPeerClient$Writer.run(SocketPeerClient.java:251)", "java.lang.Thread.run(Thread.java:662)" ] }, { name: "Peer Server Reader Socket[addr=/172.20.153.34,port=26135,localport=26080]", priority: 5, state: "RUNNABLE", trace: [ "java.net.SocketInputStream.socketRead0(Native Method)", "java.net.SocketInputStream.read(SocketInputStream.java:129)", "com.company.module.cluster.peer.socket.ByteCountingInputStream.read(ByteCountingInputStream.java:67)", "java.io.ObjectInputStream$PeekInputStream.read(ObjectInputStream.java:2288)", "java.io.ObjectInputStream$PeekInputStream.readFully(ObjectInputStream.java:2301)", "java.io.ObjectInputStream$BlockDataInputStream.readShort(ObjectInputStream.java:2772)", "java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:778)", "java.io.ObjectInputStream.(ObjectInputStream.java:278)", "com.company.module.cluster.peer.socket.Deserializer.readObjects(Deserializer.java:41)", "com.company.module.cluster.peer.socket.SocketPeerServer$Reader.run(SocketPeerServer.java:240)", "java.lang.Thread.run(Thread.java:662)" ] }, { name: "RMI Scheduler(0)", priority: 5, state: "TIMED_WAITING", trace: [ "sun.misc.Unsafe.park(Native Method)", "java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)", "java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025)", "java.util.concurrent.DelayQueue.take(DelayQueue.java:164)", "java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)", "java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)", "java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:957)", "java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:917)", "java.lang.Thread.run(Thread.java:662)" ] }, { name: "RMI TCP Accept-0", priority: 5, state: "RUNNABLE", trace: [ "java.net.PlainSocketImpl.socketAccept(Native Method)", "java.net.PlainSocketImpl.accept(PlainSocketImpl.java:408)", "java.net.ServerSocket.implAccept(ServerSocket.java:462)", "java.net.ServerSocket.accept(ServerSocket.java:430)", "sun.management.jmxremote.LocalRMIServerSocketFactory$1.accept(LocalRMIServerSocketFactory.java:34)", "sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTransport.java:369)", "sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:341)", "java.lang.Thread.run(Thread.java:662)" ] }, { name: "RMI TCP Accept-0", priority: 5, state: "RUNNABLE", trace: [ "java.net.PlainSocketImpl.socketAccept(Native Method)", "java.net.PlainSocketImpl.accept(PlainSocketImpl.java:408)", "java.net.ServerSocket.implAccept(ServerSocket.java:462)", "java.net.ServerSocket.accept(ServerSocket.java:430)", "sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTransport.java:369)", "sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:341)", "java.lang.Thread.run(Thread.java:662)" ] }, { name: "RMI TCP Accept-9243", priority: 5, state: "RUNNABLE", trace: [ "java.net.PlainSocketImpl.socketAccept(Native Method)", "java.net.PlainSocketImpl.accept(PlainSocketImpl.java:408)", "java.net.ServerSocket.implAccept(ServerSocket.java:462)", "java.net.ServerSocket.accept(ServerSocket.java:430)", "sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTransport.java:369)", "sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:341)", "java.lang.Thread.run(Thread.java:662)" ] }, { name: "RMI TCP Connection(idle)", priority: 5, state: "TIMED_WAITING", trace: [ "sun.misc.Unsafe.park(Native Method)", "java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)", "java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)", "java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)", "java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)", "java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:955)", "java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:917)", "java.lang.Thread.run(Thread.java:662)" ] }, { name: "RMI TCP Connection(idle)", priority: 5, state: "TIMED_WAITING", trace: [ "sun.misc.Unsafe.park(Native Method)", "java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)", "java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)", "java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)", "java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)", "java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:955)", "java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:917)", "java.lang.Thread.run(Thread.java:662)" ] }, { name: "Reference Handler", priority: 10, state: "WAITING", trace: [ "java.lang.Object.wait(Native Method)", "java.lang.Object.wait(Object.java:485)", "java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)" ] }, { name: "Signal Dispatcher", priority: 9, state: "RUNNABLE", trace: [ ] }, { name: "StopMonitor", priority: 5, state: "RUNNABLE", trace: [ "java.net.PlainSocketImpl.socketAccept(Native Method)", "java.net.PlainSocketImpl.accept(PlainSocketImpl.java:408)", "java.net.ServerSocket.implAccept(ServerSocket.java:462)", "java.net.ServerSocket.accept(ServerSocket.java:430)", "org.mortbay.start.Monitor.run(Monitor.java:81)" ] }, { name: "TaskManager Scheduler", priority: 5, state: "TIMED_WAITING", trace: [ "java.lang.Object.wait(Native Method)", "com.company.contrib.tasks.impl.TaskManagerImpl$SchedulerThread.run(TaskManagerImpl.java:597)" ] }, { name: "Thread-39", priority: 5, state: "TIMED_WAITING", trace: [ "java.lang.Thread.sleep(Native Method)", "com.company.module.management.status.alive.ThreadedCachingAliveChecker$AliveCheckerThread.run(ThreadedCachingAliveChecker.java:109)" ] }, { name: "Timer-1", priority: 5, state: "TIMED_WAITING", trace: [ "java.lang.Object.wait(Native Method)", "java.util.TimerThread.mainLoop(Timer.java:509)", "java.util.TimerThread.run(Timer.java:462)" ] }, { name: "Timer-2", priority: 5, state: "TIMED_WAITING", trace: [ "java.lang.Object.wait(Native Method)", "java.util.TimerThread.mainLoop(Timer.java:509)", "java.util.TimerThread.run(Timer.java:462)" ] }, { name: "Timer-3", priority: 5, state: "TIMED_WAITING", trace: [ "java.lang.Object.wait(Native Method)", "java.util.TimerThread.mainLoop(Timer.java:509)", "java.util.TimerThread.run(Timer.java:462)" ] }, { name: "Timer-4", priority: 5, state: "TIMED_WAITING", trace: [ "java.lang.Object.wait(Native Method)", "java.util.TimerThread.mainLoop(Timer.java:509)", "java.util.TimerThread.run(Timer.java:462)" ] }, { name: "Timer-5", priority: 5, state: "TIMED_WAITING", trace: [ "java.lang.Object.wait(Native Method)", "java.util.TimerThread.mainLoop(Timer.java:509)", "java.util.TimerThread.run(Timer.java:462)" ] }, { name: "Timer-6", priority: 5, state: "TIMED_WAITING", trace: [ "java.lang.Object.wait(Native Method)", "java.util.TimerThread.mainLoop(Timer.java:509)", "java.util.TimerThread.run(Timer.java:462)" ] }, { name: "Timer-7", priority: 5, state: "TIMED_WAITING", trace: [ "java.lang.Object.wait(Native Method)", "java.util.TimerThread.mainLoop(Timer.java:509)", "java.util.TimerThread.run(Timer.java:462)" ] }, { name: "authorize 1", priority: 5, state: "TIMED_WAITING", trace: [ "sun.misc.Unsafe.park(Native Method)", "java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)", "java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025)", "java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:347)", "com.company.module.pipeline.thread.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:968)", "com.company.module.pipeline.thread.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:930)", "java.lang.Thread.run(Thread.java:662)" ] }, { name: "authorize 10", priority: 5, state: "TIMED_WAITING", trace: [ "sun.misc.Unsafe.park(Native Method)", "java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)", "java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025)", "java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:347)", "com.company.module.pipeline.thread.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:968)", "com.company.module.pipeline.thread.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:930)", "java.lang.Thread.run(Thread.java:662)" ] }, { name: "authorize 2", priority: 5, state: "TIMED_WAITING", trace: [ "sun.misc.Unsafe.park(Native Method)", "java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)", "java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025)", "java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:347)", "com.company.module.pipeline.thread.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:968)", "com.company.module.pipeline.thread.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:930)", "java.lang.Thread.run(Thread.java:662)" ] }, { name: "authorize 3", priority: 5, state: "TIMED_WAITING", trace: [ "sun.misc.Unsafe.park(Native Method)", "java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)", "java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025)", "java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:347)", "com.company.module.pipeline.thread.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:968)", "com.company.module.pipeline.thread.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:930)", "java.lang.Thread.run(Thread.java:662)" ] }, { name: "authorize 4", priority: 5, state: "TIMED_WAITING", trace: [ "sun.misc.Unsafe.park(Native Method)", "java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)", "java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025)", "java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:347)", "com.company.module.pipeline.thread.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:968)", "com.company.module.pipeline.thread.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:930)", "java.lang.Thread.run(Thread.java:662)" ] }, { name: "authorize 5", priority: 5, state: "TIMED_WAITING", trace: [ "sun.misc.Unsafe.park(Native Method)", "java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)", "java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025)", "java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:347)", "com.company.module.pipeline.thread.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:968)", "com.company.module.pipeline.thread.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:930)", "java.lang.Thread.run(Thread.java:662)" ] }, { name: "authorize 6", priority: 5, state: "TIMED_WAITING", trace: [ "sun.misc.Unsafe.park(Native Method)", "java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)", "java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025)", "java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:347)", "com.company.module.pipeline.thread.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:968)", "com.company.module.pipeline.thread.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:930)", "java.lang.Thread.run(Thread.java:662)" ] }, { name: "authorize 7", priority: 5, state: "TIMED_WAITING", trace: [ "sun.misc.Unsafe.park(Native Method)", "java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)", "java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025)", "java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:347)", "com.company.module.pipeline.thread.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:968)", "com.company.module.pipeline.thread.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:930)", "java.lang.Thread.run(Thread.java:662)" ] }, { name: "authorize 8", priority: 5, state: "TIMED_WAITING", trace: [ "sun.misc.Unsafe.park(Native Method)", "java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)", "java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025)", "java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:347)", "com.company.module.pipeline.thread.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:968)", "com.company.module.pipeline.thread.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:930)", "java.lang.Thread.run(Thread.java:662)" ] }, { name: "authorize 9", priority: 5, state: "TIMED_WAITING", trace: [ "sun.misc.Unsafe.park(Native Method)", "java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)", "java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025)", "java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:347)", "com.company.module.pipeline.thread.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:968)", "com.company.module.pipeline.thread.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:930)", "java.lang.Thread.run(Thread.java:662)" ] }, { name: "dtautosensor", priority: 5, state: "RUNNABLE", trace: [ ] }, { name: "dtcontroller", priority: 5, state: "RUNNABLE", trace: [ ] }, { name: "dtmetricstimer", priority: 5, state: "RUNNABLE", trace: [ ] }, { name: "marshall 179", priority: 5, state: "TIMED_WAITING", trace: [ "java.lang.Object.wait(Native Method)", "java.lang.Object.wait(Object.java:443)", "com.company.module.pipeline.queue.EquitableBlockingQueue.poll(EquitableBlockingQueue.java:199)", "com.company.module.pipeline.queue.EquitableBlockingQueue.poll(EquitableBlockingQueue.java:38)", "com.company.module.pipeline.thread.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:968)", "com.company.module.pipeline.thread.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:930)", "java.lang.Thread.run(Thread.java:662)" ] }, { name: "marshall 184", priority: 5, state: "TIMED_WAITING", trace: [ "java.lang.Object.wait(Native Method)", "java.lang.Object.wait(Object.java:443)", "com.company.module.pipeline.queue.EquitableBlockingQueue.poll(EquitableBlockingQueue.java:199)", "com.company.module.pipeline.queue.EquitableBlockingQueue.poll(EquitableBlockingQueue.java:38)", "com.company.module.pipeline.thread.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:968)", "com.company.module.pipeline.thread.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:930)", "java.lang.Thread.run(Thread.java:662)" ] }, { name: "marshall 192", priority: 5, state: "TIMED_WAITING", trace: [ "java.lang.Object.wait(Native Method)", "java.lang.Object.wait(Object.java:443)", "com.company.module.pipeline.queue.EquitableBlockingQueue.poll(EquitableBlockingQueue.java:199)", "com.company.module.pipeline.queue.EquitableBlockingQueue.poll(EquitableBlockingQueue.java:38)", "com.company.module.pipeline.thread.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:968)", "com.company.module.pipeline.thread.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:930)", "java.lang.Thread.run(Thread.java:662)" ] }, { name: "marshall 193", priority: 5, state: "TIMED_WAITING", trace: [ "java.lang.Object.wait(Native Method)", "java.lang.Object.wait(Object.java:443)", "com.company.module.pipeline.queue.EquitableBlockingQueue.poll(EquitableBlockingQueue.java:199)", "com.company.module.pipeline.queue.EquitableBlockingQueue.poll(EquitableBlockingQueue.java:38)", "com.company.module.pipeline.thread.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:968)", "com.company.module.pipeline.thread.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:930)", "java.lang.Thread.run(Thread.java:662)" ] }, { name: "marshall 200", priority: 5, state: "TIMED_WAITING", trace: [ "java.lang.Object.wait(Native Method)", "java.lang.Object.wait(Object.java:443)", "com.company.module.pipeline.queue.EquitableBlockingQueue.poll(EquitableBlockingQueue.java:199)", "com.company.module.pipeline.queue.EquitableBlockingQueue.poll(EquitableBlockingQueue.java:38)", "com.company.module.pipeline.thread.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:968)", "com.company.module.pipeline.thread.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:930)", "java.lang.Thread.run(Thread.java:662)" ] }, { name: "marshall 201", priority: 5, state: "TIMED_WAITING", trace: [ "java.lang.Object.wait(Native Method)", "java.lang.Object.wait(Object.java:443)", "com.company.module.pipeline.queue.EquitableBlockingQueue.poll(EquitableBlockingQueue.java:199)", "com.company.module.pipeline.queue.EquitableBlockingQueue.poll(EquitableBlockingQueue.java:38)", "com.company.module.pipeline.thread.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:968)", "com.company.module.pipeline.thread.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:930)", "java.lang.Thread.run(Thread.java:662)" ] }, { name: "marshall 202", priority: 5, state: "TIMED_WAITING", trace: [ "java.lang.Object.wait(Native Method)", "java.lang.Object.wait(Object.java:443)", "com.company.module.pipeline.queue.EquitableBlockingQueue.poll(EquitableBlockingQueue.java:199)", "com.company.module.pipeline.queue.EquitableBlockingQueue.poll(EquitableBlockingQueue.java:38)", "com.company.module.pipeline.thread.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:968)", "com.company.module.pipeline.thread.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:930)", "java.lang.Thread.run(Thread.java:662)" ] }, { name: "marshall 203", priority: 5, state: "TIMED_WAITING", trace: [ "java.lang.Object.wait(Native Method)", "java.lang.Object.wait(Object.java:443)", "com.company.module.pipeline.queue.EquitableBlockingQueue.poll(EquitableBlockingQueue.java:199)", "com.company.module.pipeline.queue.EquitableBlockingQueue.poll(EquitableBlockingQueue.java:38)", "com.company.module.pipeline.thread.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:968)", "com.company.module.pipeline.thread.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:930)", "java.lang.Thread.run(Thread.java:662)" ] }, { name: "marshall 204", priority: 5, state: "TIMED_WAITING", trace: [ "java.lang.Object.wait(Native Method)", "java.lang.Object.wait(Object.java:443)", "com.company.module.pipeline.queue.EquitableBlockingQueue.poll(EquitableBlockingQueue.java:199)", "com.company.module.pipeline.queue.EquitableBlockingQueue.poll(EquitableBlockingQueue.java:38)", "com.company.module.pipeline.thread.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:968)", "com.company.module.pipeline.thread.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:930)", "java.lang.Thread.run(Thread.java:662)" ] }, { name: "marshall 205", priority: 5, state: "TIMED_WAITING", trace: [ "java.lang.Object.wait(Native Method)", "java.lang.Object.wait(Object.java:443)", "com.company.module.pipeline.queue.EquitableBlockingQueue.poll(EquitableBlockingQueue.java:199)", "com.company.module.pipeline.queue.EquitableBlockingQueue.poll(EquitableBlockingQueue.java:38)", "com.company.module.pipeline.thread.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:968)", "com.company.module.pipeline.thread.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:930)", "java.lang.Thread.run(Thread.java:662)" ] }, { name: "messageStoreTaskScheduler-1", priority: 5, state: "TIMED_WAITING", trace: [ "sun.misc.Unsafe.park(Native Method)", "java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)", "java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025)", "java.util.concurrent.DelayQueue.take(DelayQueue.java:164)", "java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)", "java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)", "java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:957)", "java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:917)", "java.lang.Thread.run(Thread.java:662)" ] }, { name: "messageStoreTaskScheduler-1", priority: 5, state: "TIMED_WAITING", trace: [ "sun.misc.Unsafe.park(Native Method)", "java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)", "java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025)", "java.util.concurrent.DelayQueue.take(DelayQueue.java:164)", "java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)", "java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)", "java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:957)", "java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:917)", "java.lang.Thread.run(Thread.java:662)" ] }, { name: "net.sf.ehcache.CacheManager at 13e4318d", priority: 5, state: "WAITING", trace: [ "java.lang.Object.wait(Native Method)", "java.lang.Object.wait(Object.java:485)", "java.util.TimerThread.mainLoop(Timer.java:483)", "java.util.TimerThread.run(Timer.java:462)" ] }, { name: "pool-1-thread-1", priority: 5, state: "WAITING", trace: [ "sun.misc.Unsafe.park(Native Method)", "java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)", "java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)", "java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)", "java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:957)", "java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:917)", "java.lang.Thread.run(Thread.java:662)" ] }, { name: "pool-10-thread-1", priority: 5, state: "WAITING", trace: [ "sun.misc.Unsafe.park(Native Method)", "java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)", "java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)", "java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)", "java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:957)", "java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:917)", "java.lang.Thread.run(Thread.java:662)" ] }, { name: "pool-10-thread-2", priority: 5, state: "WAITING", trace: [ "sun.misc.Unsafe.park(Native Method)", "java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)", "java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)", "java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)", "java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:957)", "java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:917)", "java.lang.Thread.run(Thread.java:662)" ] }, { name: "pool-10-thread-3", priority: 5, state: "WAITING", trace: [ "sun.misc.Unsafe.park(Native Method)", "java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)", "java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)", "java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)", "java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:957)", "java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:917)", "java.lang.Thread.run(Thread.java:662)" ] }, { name: "pool-10-thread-4", priority: 5, state: "WAITING", trace: [ "sun.misc.Unsafe.park(Native Method)", "java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)", "java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)", "java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)", "java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:957)", "java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:917)", "java.lang.Thread.run(Thread.java:662)" ] }, { name: "pool-10-thread-5", priority: 5, state: "WAITING", trace: [ "sun.misc.Unsafe.park(Native Method)", "java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)", "java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)", "java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)", "java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:957)", "java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:917)", "java.lang.Thread.run(Thread.java:662)" ] }, { name: "pool-11-thread-1", priority: 5, state: "BLOCKED", trace: [ "com.rabbitmq.client.impl.SocketFrameHandler.writeFrame(SocketFrameHandler.java:136)", "com.rabbitmq.client.impl.HeartbeatSender$HeartbeatRunnable.run(HeartbeatSender.java:133)", "java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:439)", "java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)", "java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)", "java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)", "java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:180)", "java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:204)", "java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)", "java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)", "java.lang.Thread.run(Thread.java:662)" ] }, { name: "pool-4-thread-105", priority: 5, state: "TIMED_WAITING", trace: [ "sun.misc.Unsafe.park(Native Method)", "java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)", "java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1011)", "java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1303)", "java.util.concurrent.CountDownLatch.await(CountDownLatch.java:253)", "com.company.contrib.tasks.impl.RunnableWrapper.await(RunnableWrapper.java:349)", "com.company.contrib.tasks.impl.RunnableWrapper$ContextImpl.waitForRunningSubTasks(RunnableWrapper.java:600)", "com.company.web.tv.image.impl.task.producer.UpdateMainImagesListTask.run(UpdateMainImagesListTask.java:41)", "com.company.contrib.tasks.impl.RunnableWrapper.run(RunnableWrapper.java:97)", "java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)", "java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)", "java.lang.Thread.run(Thread.java:662)" ] }, { name: "pool-4-thread-106", priority: 5, state: "WAITING", trace: [ "java.lang.Object.wait(Native Method)", "java.lang.Object.wait(Object.java:485)", "com.rabbitmq.utility.BlockingCell.get(BlockingCell.java:50)", "com.rabbitmq.utility.BlockingCell.uninterruptibleGet(BlockingCell.java:89)", "com.rabbitmq.utility.BlockingValueOrException.uninterruptibleGetValue(BlockingValueOrException.java:33)", "com.rabbitmq.client.impl.AMQChannel$BlockingRpcContinuation.getReply(AMQChannel.java:343)", "com.rabbitmq.client.impl.AMQChannel.privateRpc(AMQChannel.java:216)", "com.rabbitmq.client.impl.AMQChannel.exnWrappingRpc(AMQChannel.java:118)", "com.rabbitmq.client.impl.ChannelN.open(ChannelN.java:125)", "com.rabbitmq.client.impl.ChannelManager.createChannel(ChannelManager.java:121)", "com.rabbitmq.client.impl.AMQConnection.createChannel(AMQConnection.java:473)", "org.springframework.amqp.rabbit.connection.SimpleConnection.createChannel(SimpleConnection.java:42)", "org.springframework.amqp.rabbit.connection.CachingConnectionFactory$ChannelCachingConnectionProxy.createBareChannel(CachingConnectionFactory.java:416)", "org.springframework.amqp.rabbit.connection.CachingConnectionFactory$ChannelCachingConnectionProxy.access$000(CachingConnectionFactory.java:407)", "org.springframework.amqp.rabbit.connection.CachingConnectionFactory.createBareChannel(CachingConnectionFactory.java:209)", "org.springframework.amqp.rabbit.connection.CachingConnectionFactory.getCachedChannelProxy(CachingConnectionFactory.java:186)", "org.springframework.amqp.rabbit.connection.CachingConnectionFactory.getChannel(CachingConnectionFactory.java:180)", "org.springframework.amqp.rabbit.connection.CachingConnectionFactory.access$300(CachingConnectionFactory.java:52)", "org.springframework.amqp.rabbit.connection.CachingConnectionFactory$ChannelCachingConnectionProxy.createChannel(CachingConnectionFactory.java:420)", "org.springframework.amqp.rabbit.connection.ConnectionFactoryUtils$1.createChannel(ConnectionFactoryUtils.java:81)", "org.springframework.amqp.rabbit.connection.ConnectionFactoryUtils.doGetTransactionalResourceHolder(ConnectionFactoryUtils.java:129)", "org.springframework.amqp.rabbit.connection.ConnectionFactoryUtils.getTransactionalResourceHolder(ConnectionFactoryUtils.java:67)", "org.springframework.amqp.rabbit.connection.RabbitAccessor.getTransactionalResourceHolder(RabbitAccessor.java:102)", "org.springframework.amqp.rabbit.core.RabbitTemplate.execute(RabbitTemplate.java:618)", "org.springframework.amqp.rabbit.core.RabbitTemplate.send(RabbitTemplate.java:349)", "org.springframework.amqp.rabbit.core.RabbitTemplate.send(RabbitTemplate.java:343)", "org.springframework.amqp.rabbit.core.RabbitTemplate.send(RabbitTemplate.java:335)", "com.company.web.tv.image.impl.messaging.Producer.queueMessage(Producer.java:40)", "com.company.web.tv.image.impl.util.IterativeEntityProcessor.queueEntity(IterativeEntityProcessor.java:149)", "com.company.web.tv.image.impl.util.IterativeEntityProcessor.queueMitsForOriginalEntity(IterativeEntityProcessor.java:71)", "com.company.web.tv.image.impl.util.IterativeEntityProcessor.queueItemsForEntity(IterativeEntityProcessor.java:36)", "com.company.web.tv.image.impl.task.producer.UpdateMainImagesForAllMainImageTypesTask.queueEntity(UpdateMainImagesForAllMainImageTypesTask.java:63)", "com.company.web.tv.image.impl.task.producer.UpdateMainImagesForAllMainImageTypesTask.run(UpdateMainImagesForAllMainImageTypesTask.java:39)", "com.company.contrib.tasks.impl.RunnableWrapper.run(RunnableWrapper.java:97)", "java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)", "java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)", "java.lang.Thread.run(Thread.java:662)" ] }, { name: "pool-4-thread-107", priority: 5, state: "TIMED_WAITING", trace: [ "sun.misc.Unsafe.park(Native Method)", "java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)", "java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1011)", "java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1303)", "java.util.concurrent.CountDownLatch.await(CountDownLatch.java:253)", "com.company.contrib.tasks.impl.RunnableWrapper.await(RunnableWrapper.java:349)", "com.company.contrib.tasks.impl.RunnableWrapper$ContextImpl.waitForRunningSubTasks(RunnableWrapper.java:600)", "com.company.web.tv.image.impl.task.producer.UpdateMainImagesListTask.run(UpdateMainImagesListTask.java:41)", "com.company.contrib.tasks.impl.RunnableWrapper.run(RunnableWrapper.java:97)", "java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)", "java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)", "java.lang.Thread.run(Thread.java:662)" ] }, { name: "pool-4-thread-108", priority: 5, state: "WAITING", trace: [ "java.lang.Object.wait(Native Method)", "java.lang.Object.wait(Object.java:485)", "com.rabbitmq.utility.BlockingCell.get(BlockingCell.java:50)", "com.rabbitmq.utility.BlockingCell.get(BlockingCell.java:65)", "com.rabbitmq.utility.BlockingCell.uninterruptibleGet(BlockingCell.java:111)", "com.rabbitmq.utility.BlockingValueOrException.uninterruptibleGetValue(BlockingValueOrException.java:37)", "com.rabbitmq.client.impl.AMQChannel$BlockingRpcContinuation.getReply(AMQChannel.java:349)", "com.rabbitmq.client.impl.ChannelN.close(ChannelN.java:569)", "com.rabbitmq.client.impl.ChannelN.close(ChannelN.java:501)", "com.rabbitmq.client.impl.ChannelN.close(ChannelN.java:494)", "org.springframework.amqp.rabbit.connection.CachingConnectionFactory$CachedChannelInvocationHandler.physicalClose(CachingConnectionFactory.java:398)", "org.springframework.amqp.rabbit.connection.CachingConnectionFactory$CachedChannelInvocationHandler.invoke(CachingConnectionFactory.java:331)", "com.sun.proxy.$Proxy50.close(Unknown Source)", "org.springframework.amqp.rabbit.connection.RabbitUtils.closeChannel(RabbitUtils.java:64)", "org.springframework.amqp.rabbit.connection.ConnectionFactoryUtils.releaseResources(ConnectionFactoryUtils.java:151)", "org.springframework.amqp.rabbit.core.RabbitTemplate.execute(RabbitTemplate.java:634)", "org.springframework.amqp.rabbit.core.RabbitTemplate.send(RabbitTemplate.java:349)", "org.springframework.amqp.rabbit.core.RabbitTemplate.send(RabbitTemplate.java:343)", "org.springframework.amqp.rabbit.core.RabbitTemplate.send(RabbitTemplate.java:335)", "com.company.web.tv.image.impl.messaging.Producer.queueMessage(Producer.java:40)", "com.company.web.tv.image.impl.util.IterativeEntityProcessor.queueEntity(IterativeEntityProcessor.java:149)", "com.company.web.tv.image.impl.util.IterativeEntityProcessor.queueMitsForOriginalEntity(IterativeEntityProcessor.java:71)", "com.company.web.tv.image.impl.util.IterativeEntityProcessor.queueItemsForEntity(IterativeEntityProcessor.java:36)", "com.company.web.tv.image.impl.task.producer.UpdateMainImagesForAllMainImageTypesTask.queueEntity(UpdateMainImagesForAllMainImageTypesTask.java:63)", "com.company.web.tv.image.impl.task.producer.UpdateMainImagesForAllMainImageTypesTask.run(UpdateMainImagesForAllMainImageTypesTask.java:39)", "com.company.contrib.tasks.impl.RunnableWrapper.run(RunnableWrapper.java:97)", "java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)", "java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)", "java.lang.Thread.run(Thread.java:662)" ] }, { name: "pool-4-thread-111", priority: 5, state: "TIMED_WAITING", trace: [ "sun.misc.Unsafe.park(Native Method)", "java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)", "java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1011)", "java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1303)", "java.util.concurrent.CountDownLatch.await(CountDownLatch.java:253)", "com.company.contrib.tasks.impl.RunnableWrapper.await(RunnableWrapper.java:349)", "com.company.contrib.tasks.impl.RunnableWrapper$ContextImpl.waitForRunningSubTasks(RunnableWrapper.java:600)", "com.company.web.tv.image.impl.task.producer.UpdateMainImagesListTask.run(UpdateMainImagesListTask.java:41)", "com.company.contrib.tasks.impl.RunnableWrapper.run(RunnableWrapper.java:97)", "java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)", "java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)", "java.lang.Thread.run(Thread.java:662)" ] }, { name: "pool-4-thread-112", priority: 5, state: "WAITING", trace: [ "java.lang.Object.wait(Native Method)", "java.lang.Object.wait(Object.java:485)", "com.rabbitmq.utility.BlockingCell.get(BlockingCell.java:50)", "com.rabbitmq.utility.BlockingCell.uninterruptibleGet(BlockingCell.java:89)", "com.rabbitmq.utility.BlockingValueOrException.uninterruptibleGetValue(BlockingValueOrException.java:33)", "com.rabbitmq.client.impl.AMQChannel$BlockingRpcContinuation.getReply(AMQChannel.java:343)", "com.rabbitmq.client.impl.AMQChannel.privateRpc(AMQChannel.java:216)", "com.rabbitmq.client.impl.AMQChannel.exnWrappingRpc(AMQChannel.java:118)", "com.rabbitmq.client.impl.ChannelN.open(ChannelN.java:125)", "com.rabbitmq.client.impl.ChannelManager.createChannel(ChannelManager.java:121)", "com.rabbitmq.client.impl.AMQConnection.createChannel(AMQConnection.java:473)", "org.springframework.amqp.rabbit.connection.SimpleConnection.createChannel(SimpleConnection.java:42)", "org.springframework.amqp.rabbit.connection.CachingConnectionFactory$ChannelCachingConnectionProxy.createBareChannel(CachingConnectionFactory.java:416)", "org.springframework.amqp.rabbit.connection.CachingConnectionFactory$ChannelCachingConnectionProxy.access$000(CachingConnectionFactory.java:407)", "org.springframework.amqp.rabbit.connection.CachingConnectionFactory.createBareChannel(CachingConnectionFactory.java:209)", "org.springframework.amqp.rabbit.connection.CachingConnectionFactory.getCachedChannelProxy(CachingConnectionFactory.java:186)", "org.springframework.amqp.rabbit.connection.CachingConnectionFactory.getChannel(CachingConnectionFactory.java:180)", "org.springframework.amqp.rabbit.connection.CachingConnectionFactory.access$300(CachingConnectionFactory.java:52)", "org.springframework.amqp.rabbit.connection.CachingConnectionFactory$ChannelCachingConnectionProxy.createChannel(CachingConnectionFactory.java:420)", "org.springframework.amqp.rabbit.connection.ConnectionFactoryUtils$1.createChannel(ConnectionFactoryUtils.java:81)", "org.springframework.amqp.rabbit.connection.ConnectionFactoryUtils.doGetTransactionalResourceHolder(ConnectionFactoryUtils.java:129)", "org.springframework.amqp.rabbit.connection.ConnectionFactoryUtils.getTransactionalResourceHolder(ConnectionFactoryUtils.java:67)", "org.springframework.amqp.rabbit.connection.RabbitAccessor.getTransactionalResourceHolder(RabbitAccessor.java:102)", "org.springframework.amqp.rabbit.core.RabbitTemplate.execute(RabbitTemplate.java:618)", "org.springframework.amqp.rabbit.core.RabbitTemplate.send(RabbitTemplate.java:349)", "org.springframework.amqp.rabbit.core.RabbitTemplate.send(RabbitTemplate.java:343)", "org.springframework.amqp.rabbit.core.RabbitTemplate.send(RabbitTemplate.java:335)", "com.company.web.tv.image.impl.messaging.Producer.queueMessage(Producer.java:40)", "com.company.web.tv.image.impl.util.IterativeEntityProcessor.queueEntity(IterativeEntityProcessor.java:149)", "com.company.web.tv.image.impl.util.IterativeEntityProcessor.queueMitsForOriginalEntity(IterativeEntityProcessor.java:71)", "com.company.web.tv.image.impl.util.IterativeEntityProcessor.queueItemsForEntity(IterativeEntityProcessor.java:36)", "com.company.web.tv.image.impl.task.producer.UpdateMainImagesForAllMainImageTypesTask.queueEntity(UpdateMainImagesForAllMainImageTypesTask.java:63)", "com.company.web.tv.image.impl.task.producer.UpdateMainImagesForAllMainImageTypesTask.run(UpdateMainImagesForAllMainImageTypesTask.java:39)", "com.company.contrib.tasks.impl.RunnableWrapper.run(RunnableWrapper.java:97)", "java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)", "java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)", "java.lang.Thread.run(Thread.java:662)" ] }, { name: "pool-4-thread-113", priority: 5, state: "TIMED_WAITING", trace: [ "sun.misc.Unsafe.park(Native Method)", "java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)", "java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1011)", "java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1303)", "java.util.concurrent.CountDownLatch.await(CountDownLatch.java:253)", "com.company.contrib.tasks.impl.RunnableWrapper.await(RunnableWrapper.java:349)", "com.company.contrib.tasks.impl.RunnableWrapper$ContextImpl.waitForRunningSubTasks(RunnableWrapper.java:600)", "com.company.web.tv.image.impl.task.producer.UpdateMainImagesListTask.run(UpdateMainImagesListTask.java:41)", "com.company.contrib.tasks.impl.RunnableWrapper.run(RunnableWrapper.java:97)", "java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)", "java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)", "java.lang.Thread.run(Thread.java:662)" ] }, { name: "pool-4-thread-114", priority: 5, state: "BLOCKED", trace: [ "java.io.BufferedOutputStream.write(BufferedOutputStream.java)", "java.io.DataOutputStream.writeByte(DataOutputStream.java:136)", "com.rabbitmq.client.impl.Frame.writeTo(Frame.java:189)", "com.rabbitmq.client.impl.SocketFrameHandler.writeFrame(SocketFrameHandler.java:137)", "com.rabbitmq.client.impl.AMQConnection.writeFrame(AMQConnection.java:480)", "com.rabbitmq.client.impl.AMQCommand.transmit(AMQCommand.java:102)", "com.rabbitmq.client.impl.AMQChannel.quiescingTransmit(AMQChannel.java:316)", "com.rabbitmq.client.impl.AMQChannel.quiescingTransmit(AMQChannel.java:298)", "com.rabbitmq.client.impl.AMQChannel.quiescingRpc(AMQChannel.java:233)", "com.rabbitmq.client.impl.AMQChannel.rpc(AMQChannel.java:224)", "com.rabbitmq.client.impl.AMQChannel.privateRpc(AMQChannel.java:209)", "com.rabbitmq.client.impl.AMQChannel.exnWrappingRpc(AMQChannel.java:118)", "com.rabbitmq.client.impl.ChannelN.open(ChannelN.java:125)", "com.rabbitmq.client.impl.ChannelManager.createChannel(ChannelManager.java:121)", "com.rabbitmq.client.impl.AMQConnection.createChannel(AMQConnection.java:473)", "org.springframework.amqp.rabbit.connection.SimpleConnection.createChannel(SimpleConnection.java:42)", "org.springframework.amqp.rabbit.connection.CachingConnectionFactory$ChannelCachingConnectionProxy.createBareChannel(CachingConnectionFactory.java:416)", "org.springframework.amqp.rabbit.connection.CachingConnectionFactory$ChannelCachingConnectionProxy.access$000(CachingConnectionFactory.java:407)", "org.springframework.amqp.rabbit.connection.CachingConnectionFactory.createBareChannel(CachingConnectionFactory.java:209)", "org.springframework.amqp.rabbit.connection.CachingConnectionFactory.getCachedChannelProxy(CachingConnectionFactory.java:186)", "org.springframework.amqp.rabbit.connection.CachingConnectionFactory.getChannel(CachingConnectionFactory.java:180)", "org.springframework.amqp.rabbit.connection.CachingConnectionFactory.access$300(CachingConnectionFactory.java:52)", "org.springframework.amqp.rabbit.connection.CachingConnectionFactory$ChannelCachingConnectionProxy.createChannel(CachingConnectionFactory.java:420)", "org.springframework.amqp.rabbit.connection.ConnectionFactoryUtils$1.createChannel(ConnectionFactoryUtils.java:81)", "org.springframework.amqp.rabbit.connection.ConnectionFactoryUtils.doGetTransactionalResourceHolder(ConnectionFactoryUtils.java:129)", "org.springframework.amqp.rabbit.connection.ConnectionFactoryUtils.getTransactionalResourceHolder(ConnectionFactoryUtils.java:67)", "org.springframework.amqp.rabbit.connection.RabbitAccessor.getTransactionalResourceHolder(RabbitAccessor.java:102)", "org.springframework.amqp.rabbit.core.RabbitTemplate.execute(RabbitTemplate.java:618)", "org.springframework.amqp.rabbit.core.RabbitTemplate.send(RabbitTemplate.java:349)", "org.springframework.amqp.rabbit.core.RabbitTemplate.send(RabbitTemplate.java:343)", "org.springframework.amqp.rabbit.core.RabbitTemplate.send(RabbitTemplate.java:335)", "com.company.web.tv.image.impl.messaging.Producer.queueMessage(Producer.java:40)", "com.company.web.tv.image.impl.util.IterativeEntityProcessor.queueEntity(IterativeEntityProcessor.java:149)", "com.company.web.tv.image.impl.util.IterativeEntityProcessor.queueMitsForOriginalEntity(IterativeEntityProcessor.java:71)", "com.company.web.tv.image.impl.util.IterativeEntityProcessor.queueItemsForEntity(IterativeEntityProcessor.java:36)", "com.company.web.tv.image.impl.task.producer.UpdateMainImagesForAllMainImageTypesTask.queueEntity(UpdateMainImagesForAllMainImageTypesTask.java:63)", "com.company.web.tv.image.impl.task.producer.UpdateMainImagesForAllMainImageTypesTask.run(UpdateMainImagesForAllMainImageTypesTask.java:39)", "com.company.contrib.tasks.impl.RunnableWrapper.run(RunnableWrapper.java:97)", "java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)", "java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)", "java.lang.Thread.run(Thread.java:662)" ] }, { name: "pool-4-thread-115", priority: 5, state: "TIMED_WAITING", trace: [ "sun.misc.Unsafe.park(Native Method)", "java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)", "java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1011)", "java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1303)", "java.util.concurrent.CountDownLatch.await(CountDownLatch.java:253)", "com.company.contrib.tasks.impl.RunnableWrapper.await(RunnableWrapper.java:349)", "com.company.contrib.tasks.impl.RunnableWrapper$ContextImpl.waitForRunningSubTasks(RunnableWrapper.java:600)", "com.company.web.tv.image.impl.task.producer.UpdateMainImagesListTask.run(UpdateMainImagesListTask.java:41)", "com.company.contrib.tasks.impl.RunnableWrapper.run(RunnableWrapper.java:97)", "java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)", "java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)", "java.lang.Thread.run(Thread.java:662)" ] }, { name: "pool-4-thread-116", priority: 5, state: "BLOCKED", trace: [ "com.rabbitmq.client.impl.SocketFrameHandler.writeFrame(SocketFrameHandler.java:136)", "com.rabbitmq.client.impl.AMQConnection.writeFrame(AMQConnection.java:480)", "com.rabbitmq.client.impl.AMQCommand.transmit(AMQCommand.java:102)", "com.rabbitmq.client.impl.AMQChannel.quiescingTransmit(AMQChannel.java:316)", "com.rabbitmq.client.impl.AMQChannel.quiescingTransmit(AMQChannel.java:298)", "com.rabbitmq.client.impl.AMQChannel.quiescingRpc(AMQChannel.java:233)", "com.rabbitmq.client.impl.AMQChannel.rpc(AMQChannel.java:224)", "com.rabbitmq.client.impl.AMQChannel.privateRpc(AMQChannel.java:209)", "com.rabbitmq.client.impl.AMQChannel.exnWrappingRpc(AMQChannel.java:118)", "com.rabbitmq.client.impl.ChannelN.open(ChannelN.java:125)", "com.rabbitmq.client.impl.ChannelManager.createChannel(ChannelManager.java:121)", "com.rabbitmq.client.impl.AMQConnection.createChannel(AMQConnection.java:473)", "org.springframework.amqp.rabbit.connection.SimpleConnection.createChannel(SimpleConnection.java:42)", "org.springframework.amqp.rabbit.connection.CachingConnectionFactory$ChannelCachingConnectionProxy.createBareChannel(CachingConnectionFactory.java:416)", "org.springframework.amqp.rabbit.connection.CachingConnectionFactory$ChannelCachingConnectionProxy.access$000(CachingConnectionFactory.java:407)", "org.springframework.amqp.rabbit.connection.CachingConnectionFactory.createBareChannel(CachingConnectionFactory.java:209)", "org.springframework.amqp.rabbit.connection.CachingConnectionFactory.getCachedChannelProxy(CachingConnectionFactory.java:186)", "org.springframework.amqp.rabbit.connection.CachingConnectionFactory.getChannel(CachingConnectionFactory.java:180)", "org.springframework.amqp.rabbit.connection.CachingConnectionFactory.access$300(CachingConnectionFactory.java:52)", "org.springframework.amqp.rabbit.connection.CachingConnectionFactory$ChannelCachingConnectionProxy.createChannel(CachingConnectionFactory.java:420)", "org.springframework.amqp.rabbit.connection.ConnectionFactoryUtils$1.createChannel(ConnectionFactoryUtils.java:81)", "org.springframework.amqp.rabbit.connection.ConnectionFactoryUtils.doGetTransactionalResourceHolder(ConnectionFactoryUtils.java:129)", "org.springframework.amqp.rabbit.connection.ConnectionFactoryUtils.getTransactionalResourceHolder(ConnectionFactoryUtils.java:67)", "org.springframework.amqp.rabbit.connection.RabbitAccessor.getTransactionalResourceHolder(RabbitAccessor.java:102)", "org.springframework.amqp.rabbit.core.RabbitTemplate.execute(RabbitTemplate.java:618)", "org.springframework.amqp.rabbit.core.RabbitTemplate.send(RabbitTemplate.java:349)", "org.springframework.amqp.rabbit.core.RabbitTemplate.send(RabbitTemplate.java:343)", "org.springframework.amqp.rabbit.core.RabbitTemplate.send(RabbitTemplate.java:335)", "com.company.web.tv.image.impl.messaging.Producer.queueMessage(Producer.java:40)", "com.company.web.tv.image.impl.util.IterativeEntityProcessor.queueEntity(IterativeEntityProcessor.java:149)", "com.company.web.tv.image.impl.util.IterativeEntityProcessor.queueMitsForOriginalEntity(IterativeEntityProcessor.java:71)", "com.company.web.tv.image.impl.util.IterativeEntityProcessor.queueItemsForEntity(IterativeEntityProcessor.java:36)", "com.company.web.tv.image.impl.task.producer.UpdateMainImagesForAllMainImageTypesTask.queueEntity(UpdateMainImagesForAllMainImageTypesTask.java:63)", "com.company.web.tv.image.impl.task.producer.UpdateMainImagesForAllMainImageTypesTask.run(UpdateMainImagesForAllMainImageTypesTask.java:39)", "com.company.contrib.tasks.impl.RunnableWrapper.run(RunnableWrapper.java:97)", "java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)", "java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)", "java.lang.Thread.run(Thread.java:662)" ] }, { name: "task-scheduler-1", priority: 5, state: "TIMED_WAITING", trace: [ "sun.misc.Unsafe.park(Native Method)", "java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)", "java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025)", "java.util.concurrent.DelayQueue.take(DelayQueue.java:164)", "java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)", "java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)", "java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:957)", "java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:917)", "java.lang.Thread.run(Thread.java:662)" ] }, { name: "task-scheduler-10", priority: 5, state: "TIMED_WAITING", trace: [ "sun.misc.Unsafe.park(Native Method)", "java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)", "java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025)", "java.util.concurrent.DelayQueue.take(DelayQueue.java:164)", "java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)", "java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)", "java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:957)", "java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:917)", "java.lang.Thread.run(Thread.java:662)" ] }, { name: "task-scheduler-2", priority: 5, state: "TIMED_WAITING", trace: [ "sun.misc.Unsafe.park(Native Method)", "java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)", "java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025)", "java.util.concurrent.DelayQueue.take(DelayQueue.java:164)", "java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)", "java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)", "java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:957)", "java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:917)", "java.lang.Thread.run(Thread.java:662)" ] }, { name: "task-scheduler-3", priority: 5, state: "TIMED_WAITING", trace: [ "sun.misc.Unsafe.park(Native Method)", "java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)", "java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025)", "java.util.concurrent.DelayQueue.take(DelayQueue.java:164)", "java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)", "java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)", "java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:957)", "java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:917)", "java.lang.Thread.run(Thread.java:662)" ] }, { name: "task-scheduler-4", priority: 5, state: "TIMED_WAITING", trace: [ "sun.misc.Unsafe.park(Native Method)", "java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)", "java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025)", "java.util.concurrent.DelayQueue.take(DelayQueue.java:164)", "java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)", "java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)", "java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:957)", "java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:917)", "java.lang.Thread.run(Thread.java:662)" ] }, { name: "task-scheduler-5", priority: 5, state: "TIMED_WAITING", trace: [ "sun.misc.Unsafe.park(Native Method)", "java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)", "java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025)", "java.util.concurrent.DelayQueue.take(DelayQueue.java:164)", "java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)", "java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)", "java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:957)", "java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:917)", "java.lang.Thread.run(Thread.java:662)" ] }, { name: "task-scheduler-6", priority: 5, state: "TIMED_WAITING", trace: [ "sun.misc.Unsafe.park(Native Method)", "java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)", "java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025)", "java.util.concurrent.DelayQueue.take(DelayQueue.java:164)", "java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)", "java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)", "java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:957)", "java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:917)", "java.lang.Thread.run(Thread.java:662)" ] }, { name: "task-scheduler-7", priority: 5, state: "TIMED_WAITING", trace: [ "sun.misc.Unsafe.park(Native Method)", "java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)", "java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025)", "java.util.concurrent.DelayQueue.take(DelayQueue.java:164)", "java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)", "java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)", "java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:957)", "java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:917)", "java.lang.Thread.run(Thread.java:662)" ] }, { name: "task-scheduler-8", priority: 5, state: "TIMED_WAITING", trace: [ "sun.misc.Unsafe.park(Native Method)", "java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)", "java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025)", "java.util.concurrent.DelayQueue.take(DelayQueue.java:164)", "java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)", "java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)", "java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:957)", "java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:917)", "java.lang.Thread.run(Thread.java:662)" ] }, { name: "task-scheduler-9", priority: 5, state: "TIMED_WAITING", trace: [ "sun.misc.Unsafe.park(Native Method)", "java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)", "java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025)", "java.util.concurrent.DelayQueue.take(DelayQueue.java:164)", "java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)", "java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)", "java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:957)", "java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:917)", "java.lang.Thread.run(Thread.java:662)" ] }, { name: "unmarshall 165", priority: 5, state: "TIMED_WAITING", trace: [ "java.lang.Object.wait(Native Method)", "java.lang.Object.wait(Object.java:443)", "com.company.module.pipeline.queue.EquitableBlockingQueue.poll(EquitableBlockingQueue.java:199)", "com.company.module.pipeline.queue.EquitableBlockingQueue.poll(EquitableBlockingQueue.java:38)", "com.company.module.pipeline.thread.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:968)", "com.company.module.pipeline.thread.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:930)", "java.lang.Thread.run(Thread.java:662)" ] }, { name: "unmarshall 166", priority: 5, state: "TIMED_WAITING", trace: [ "java.lang.Object.wait(Native Method)", "java.lang.Object.wait(Object.java:443)", "com.company.module.pipeline.queue.EquitableBlockingQueue.poll(EquitableBlockingQueue.java:199)", "com.company.module.pipeline.queue.EquitableBlockingQueue.poll(EquitableBlockingQueue.java:38)", "com.company.module.pipeline.thread.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:968)", "com.company.module.pipeline.thread.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:930)", "java.lang.Thread.run(Thread.java:662)" ] }, { name: "unmarshall 168", priority: 5, state: "TIMED_WAITING", trace: [ "java.lang.Object.wait(Native Method)", "java.lang.Object.wait(Object.java:443)", "com.company.module.pipeline.queue.EquitableBlockingQueue.poll(EquitableBlockingQueue.java:199)", "com.company.module.pipeline.queue.EquitableBlockingQueue.poll(EquitableBlockingQueue.java:38)", "com.company.module.pipeline.thread.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:968)", "com.company.module.pipeline.thread.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:930)", "java.lang.Thread.run(Thread.java:662)" ] }, { name: "unmarshall 170", priority: 5, state: "TIMED_WAITING", trace: [ "java.lang.Object.wait(Native Method)", "java.lang.Object.wait(Object.java:443)", "com.company.module.pipeline.queue.EquitableBlockingQueue.poll(EquitableBlockingQueue.java:199)", "com.company.module.pipeline.queue.EquitableBlockingQueue.poll(EquitableBlockingQueue.java:38)", "com.company.module.pipeline.thread.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:968)", "com.company.module.pipeline.thread.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:930)", "java.lang.Thread.run(Thread.java:662)" ] }, { name: "unmarshall 171", priority: 5, state: "TIMED_WAITING", trace: [ "java.lang.Object.wait(Native Method)", "java.lang.Object.wait(Object.java:443)", "com.company.module.pipeline.queue.EquitableBlockingQueue.poll(EquitableBlockingQueue.java:199)", "com.company.module.pipeline.queue.EquitableBlockingQueue.poll(EquitableBlockingQueue.java:38)", "com.company.module.pipeline.thread.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:968)", "com.company.module.pipeline.thread.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:930)", "java.lang.Thread.run(Thread.java:662)" ] }, { name: "unmarshall 173", priority: 5, state: "TIMED_WAITING", trace: [ "java.lang.Object.wait(Native Method)", "java.lang.Object.wait(Object.java:443)", "com.company.module.pipeline.queue.EquitableBlockingQueue.poll(EquitableBlockingQueue.java:199)", "com.company.module.pipeline.queue.EquitableBlockingQueue.poll(EquitableBlockingQueue.java:38)", "com.company.module.pipeline.thread.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:968)", "com.company.module.pipeline.thread.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:930)", "java.lang.Thread.run(Thread.java:662)" ] }, { name: "unmarshall 174", priority: 5, state: "TIMED_WAITING", trace: [ "java.lang.Object.wait(Native Method)", "java.lang.Object.wait(Object.java:443)", "com.company.module.pipeline.queue.EquitableBlockingQueue.poll(EquitableBlockingQueue.java:199)", "com.company.module.pipeline.queue.EquitableBlockingQueue.poll(EquitableBlockingQueue.java:38)", "com.company.module.pipeline.thread.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:968)", "com.company.module.pipeline.thread.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:930)", "java.lang.Thread.run(Thread.java:662)" ] }, { name: "unmarshall 175", priority: 5, state: "TIMED_WAITING", trace: [ "java.lang.Object.wait(Native Method)", "java.lang.Object.wait(Object.java:443)", "com.company.module.pipeline.queue.EquitableBlockingQueue.poll(EquitableBlockingQueue.java:199)", "com.company.module.pipeline.queue.EquitableBlockingQueue.poll(EquitableBlockingQueue.java:38)", "com.company.module.pipeline.thread.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:968)", "com.company.module.pipeline.thread.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:930)", "java.lang.Thread.run(Thread.java:662)" ] }, { name: "unmarshall 176", priority: 5, state: "TIMED_WAITING", trace: [ "java.lang.Object.wait(Native Method)", "java.lang.Object.wait(Object.java:443)", "com.company.module.pipeline.queue.EquitableBlockingQueue.poll(EquitableBlockingQueue.java:199)", "com.company.module.pipeline.queue.EquitableBlockingQueue.poll(EquitableBlockingQueue.java:38)", "com.company.module.pipeline.thread.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:968)", "com.company.module.pipeline.thread.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:930)", "java.lang.Thread.run(Thread.java:662)" ] }, { name: "unmarshall 177", priority: 5, state: "TIMED_WAITING", trace: [ "java.lang.Object.wait(Native Method)", "java.lang.Object.wait(Object.java:443)", "com.company.module.pipeline.queue.EquitableBlockingQueue.poll(EquitableBlockingQueue.java:199)", "com.company.module.pipeline.queue.EquitableBlockingQueue.poll(EquitableBlockingQueue.java:38)", "com.company.module.pipeline.thread.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:968)", "com.company.module.pipeline.thread.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:930)", "java.lang.Thread.run(Thread.java:662)" ] } ] } } -------------- next part -------------- $ cat /etc/redhat-release? CentOS release 6.5 (Final) $ rpm -qa | grep -e rabbit -e erlang rabbitmq-server-3.2.2-1.noarch esl-erlang-R16B03-3.x86_64 vmware-tools-plugins-grabbitmqProxy-9.4.5-1.el6.x86_64 Reporting server status on {{2014,4,28},{17,3,45}} ... Status of node 'rabbit at ccpmer-po-cb176-p' ... [{pid,18776}, {running_applications, [{rabbitmq_management_visualiser,"RabbitMQ Visualiser","3.2.2"}, {rabbitmq_management,"RabbitMQ Management Console","3.2.2"}, {rabbitmq_web_dispatch,"RabbitMQ Web Dispatcher","3.2.2"}, {webmachine,"webmachine","1.10.3-rmq3.2.2-gite9359c7"}, {mochiweb,"MochiMedia Web Server","2.7.0-rmq3.2.2-git680dba8"}, {rabbitmq_management_agent,"RabbitMQ Management Agent","3.2.2"}, {rabbit,"RabbitMQ","3.2.2"}, {os_mon,"CPO CXC 138 46","2.2.14"}, {inets,"INETS CXC 138 49","5.9.7"}, {mnesia,"MNESIA CXC 138 12","4.11"}, {amqp_client,"RabbitMQ AMQP Client","3.2.2"}, {xmerl,"XML parser","1.3.5"}, {sasl,"SASL CXC 138 11","2.3.4"}, {stdlib,"ERTS CXC 138 10","1.19.4"}, {kernel,"ERTS CXC 138 10","2.16.4"}]}, {os,{unix,linux}}, {erlang_version, "Erlang R16B03 (erts-5.10.4) [source] [64-bit] [smp:2:2] [async-threads:30] [hipe] [kernel-poll:true]\n"}, {memory, [{total,89450216}, {connection_procs,8822888}, {queue_procs,1948648}, {plugins,1031840}, {other_proc,14366152}, {mnesia,291528}, {mgmt_db,5768400}, {msg_index,1269048}, {other_ets,6624968}, {binary,5845976}, {code,19745928}, {atom,703377}, {other_system,23031463}]}, {vm_memory_high_watermark,0.8}, {vm_memory_limit,13380295065}, {disk_free_limit,50000000}, {disk_free,29098586112}, {file_descriptors, [{total_limit,219900}, {total_used,55}, {sockets_limit,197908}, {sockets_used,13}]}, {processes,[{limit,1048576},{used,1831}]}, {run_queue,0}, {uptime,4650735}] Cluster status of node 'rabbit at ccpmer-po-cb176-p' ... [{nodes,[{disc,['rabbit at ccpmer-po-cb176-p','rabbit at ccpmer-po-cb177-p']}]}, {running_nodes,['rabbit at ccpmer-po-cb177-p','rabbit at ccpmer-po-cb176-p']}, {partitions,[]}] Application environment of node 'rabbit at ccpmer-po-cb176-p' ... [{auth_backends,[rabbit_auth_backend_internal]}, {auth_mechanisms,['PLAIN','AMQPLAIN']}, {backing_queue_module,rabbit_variable_queue}, {cluster_nodes,{['ccpmer-po-cb102-p','ccpmer-po-cb103-p','ccpmer-po-cb176-p'], disc}}, {cluster_partition_handling,ignore}, {collect_statistics,fine}, {collect_statistics_interval,5000}, {default_permissions,[<<".*">>,<<".*">>,<<".*">>]}, {default_user,<<"guest">>}, {default_user_tags,[administrator]}, {default_vhost,<<"/">>}, {delegate_count,16}, {disk_free_limit,50000000}, {enabled_plugins_file,"/etc/rabbitmq/enabled_plugins"}, {error_logger,{file,"/var/log/rabbitmq/rabbit at ccpmer-po-cb176-p.log"}}, {frame_max,131072}, {halt_on_upgrade_failure,true}, {heartbeat,580}, {hipe_compile,false}, {hipe_modules,[rabbit_reader,rabbit_channel,gen_server2,rabbit_exchange, rabbit_command_assembler,rabbit_framing_amqp_0_9_1, rabbit_basic,rabbit_event,lists,queue,priority_queue, rabbit_router,rabbit_trace,rabbit_misc,rabbit_binary_parser, rabbit_exchange_type_direct,rabbit_guid,rabbit_net, rabbit_amqqueue_process,rabbit_variable_queue, rabbit_binary_generator,rabbit_writer,delegate,gb_sets,lqueue, sets,orddict,rabbit_amqqueue,rabbit_limiter,gb_trees, rabbit_queue_index,rabbit_exchange_decorator,gen,dict,ordsets, file_handle_cache,rabbit_msg_store,array, rabbit_msg_store_ets_index,rabbit_msg_file, rabbit_exchange_type_fanout,rabbit_exchange_type_topic,mnesia, mnesia_lib,rpc,mnesia_tm,qlc,sofs,proplists,credit_flow,pmon, ssl_connection,tls_connection,ssl_record,tls_record,gen_fsm, ssl]}, {included_applications,[]}, {log_levels,[{connection,info}]}, {msg_store_file_size_limit,16777216}, {msg_store_index_module,rabbit_msg_store_ets_index}, {plugins_dir,"/usr/lib/rabbitmq/lib/rabbitmq_server-3.2.2/sbin/../plugins"}, {plugins_expand_dir,"/var/lib/rabbitmq/mnesia/rabbit at ccpmer-po-cb176-p-plugins-expand"}, {queue_index_max_journal_entries,65536}, {reverse_dns_lookups,false}, {sasl_error_logger,{file,"/var/log/rabbitmq/rabbit at ccpmer-po-cb176-p-sasl.log"}}, {server_properties,[]}, {ssl_apps,[asn1,crypto,public_key,ssl]}, {ssl_cert_login_from,distinguished_name}, {ssl_listeners,[]}, {ssl_options,[]}, {tcp_listen_options,[binary, {packet,raw}, {reuseaddr,true}, {backlog,128}, {nodelay,true}, {linger,{true,0}}, {exit_on_close,false}]}, {tcp_listeners,[5672]}, {trace_vhosts,[]}, {vm_memory_high_watermark,0.8}, {vm_memory_high_watermark_paging_ratio,0.5}] Status of node 'rabbit at ccpmer-po-cb177-p' ... [{pid,4882}, {running_applications, [{rabbitmq_management_visualiser,"RabbitMQ Visualiser","3.2.2"}, {rabbitmq_management,"RabbitMQ Management Console","3.2.2"}, {rabbitmq_web_dispatch,"RabbitMQ Web Dispatcher","3.2.2"}, {webmachine,"webmachine","1.10.3-rmq3.2.2-gite9359c7"}, {mochiweb,"MochiMedia Web Server","2.7.0-rmq3.2.2-git680dba8"}, {rabbitmq_management_agent,"RabbitMQ Management Agent","3.2.2"}, {rabbit,"RabbitMQ","3.2.2"}, {os_mon,"CPO CXC 138 46","2.2.14"}, {mnesia,"MNESIA CXC 138 12","4.11"}, {amqp_client,"RabbitMQ AMQP Client","3.2.2"}, {inets,"INETS CXC 138 49","5.9.7"}, {xmerl,"XML parser","1.3.5"}, {sasl,"SASL CXC 138 11","2.3.4"}, {stdlib,"ERTS CXC 138 10","1.19.4"}, {kernel,"ERTS CXC 138 10","2.16.4"}]}, {os,{unix,linux}}, {erlang_version, "Erlang R16B03 (erts-5.10.4) [source] [64-bit] [smp:2:2] [async-threads:30] [hipe] [kernel-poll:true]\n"}, {memory, [{total,104229600}, {connection_procs,1685248}, {queue_procs,1781608}, {plugins,1212960}, {other_proc,14832136}, {mnesia,291472}, {mgmt_db,11352}, {msg_index,6131472}, {other_ets,27521256}, {binary,7630456}, {code,19736328}, {atom,703377}, {other_system,22691935}]}, {vm_memory_high_watermark,0.8}, {vm_memory_limit,13380295065}, {disk_free_limit,50000000}, {disk_free,29138169856}, {file_descriptors, [{total_limit,219900}, {total_used,42}, {sockets_limit,197908}, {sockets_used,5}]}, {processes,[{limit,1048576},{used,588}]}, {run_queue,0}, {uptime,4575908}] Cluster status of node 'rabbit at ccpmer-po-cb177-p' ... [{nodes,[{disc,['rabbit at ccpmer-po-cb176-p','rabbit at ccpmer-po-cb177-p']}]}, {running_nodes,['rabbit at ccpmer-po-cb176-p','rabbit at ccpmer-po-cb177-p']}, {partitions,[]}] Application environment of node 'rabbit at ccpmer-po-cb177-p' ... [{auth_backends,[rabbit_auth_backend_internal]}, {auth_mechanisms,['PLAIN','AMQPLAIN']}, {backing_queue_module,rabbit_variable_queue}, {cluster_nodes,{['ccpmer-po-cb102-p','ccpmer-po-cb103-p','ccpmer-po-cb176-p', 'ccpmer-po-cb177-p'], disc}}, {cluster_partition_handling,ignore}, {collect_statistics,fine}, {collect_statistics_interval,5000}, {default_permissions,[<<".*">>,<<".*">>,<<".*">>]}, {default_user,<<"guest">>}, {default_user_tags,[administrator]}, {default_vhost,<<"/">>}, {delegate_count,16}, {disk_free_limit,50000000}, {enabled_plugins_file,"/etc/rabbitmq/enabled_plugins"}, {error_logger,{file,"/var/log/rabbitmq/rabbit at ccpmer-po-cb177-p.log"}}, {frame_max,131072}, {halt_on_upgrade_failure,true}, {heartbeat,580}, {hipe_compile,false}, {hipe_modules,[rabbit_reader,rabbit_channel,gen_server2,rabbit_exchange, rabbit_command_assembler,rabbit_framing_amqp_0_9_1, rabbit_basic,rabbit_event,lists,queue,priority_queue, rabbit_router,rabbit_trace,rabbit_misc,rabbit_binary_parser, rabbit_exchange_type_direct,rabbit_guid,rabbit_net, rabbit_amqqueue_process,rabbit_variable_queue, rabbit_binary_generator,rabbit_writer,delegate,gb_sets,lqueue, sets,orddict,rabbit_amqqueue,rabbit_limiter,gb_trees, rabbit_queue_index,rabbit_exchange_decorator,gen,dict,ordsets, file_handle_cache,rabbit_msg_store,array, rabbit_msg_store_ets_index,rabbit_msg_file, rabbit_exchange_type_fanout,rabbit_exchange_type_topic,mnesia, mnesia_lib,rpc,mnesia_tm,qlc,sofs,proplists,credit_flow,pmon, ssl_connection,tls_connection,ssl_record,tls_record,gen_fsm, ssl]}, {included_applications,[]}, {log_levels,[{connection,info}]}, {msg_store_file_size_limit,16777216}, {msg_store_index_module,rabbit_msg_store_ets_index}, {plugins_dir,"/usr/lib/rabbitmq/lib/rabbitmq_server-3.2.2/sbin/../plugins"}, {plugins_expand_dir,"/var/lib/rabbitmq/mnesia/rabbit at ccpmer-po-cb177-p-plugins-expand"}, {queue_index_max_journal_entries,65536}, {reverse_dns_lookups,false}, {sasl_error_logger,{file,"/var/log/rabbitmq/rabbit at ccpmer-po-cb177-p-sasl.log"}}, {server_properties,[]}, {ssl_apps,[asn1,crypto,public_key,ssl]}, {ssl_cert_login_from,distinguished_name}, {ssl_listeners,[]}, {ssl_options,[]}, {tcp_listen_options,[binary, {packet,raw}, {reuseaddr,true}, {backlog,128}, {nodelay,true}, {linger,{true,0}}, {exit_on_close,false}]}, {tcp_listeners,[5672]}, {trace_vhosts,[]}, {vm_memory_high_watermark,0.8}, {vm_memory_high_watermark_paging_ratio,0.5}] From amo.libertatem at gmail.com Mon Apr 28 21:36:05 2014 From: amo.libertatem at gmail.com (William Wallace) Date: Mon, 28 Apr 2014 13:36:05 -0700 (PDT) Subject: [rabbitmq-discuss] RabbitMQ on Azure VM Message-ID: <1dac0b0a-d7d9-4c5c-a928-0dcc15c536c2@googlegroups.com> We're trying to set up RabbitMQ on an Azure VM and are not having success. We have installed Erlang and RabbitMQ, can see the service is running, and get status from rabitmqctl on the command line. What we can't do, is get the management console to come up when we connect to port 15672 either locally or remotely. Anybody know what the trick is to make it work? -------------- next part -------------- An HTML attachment was scrubbed... URL: From mklishin at gopivotal.com Mon Apr 28 23:31:34 2014 From: mklishin at gopivotal.com (Michael Klishin) Date: Tue, 29 Apr 2014 02:31:34 +0400 Subject: [rabbitmq-discuss] Writes to Rabbit Occasionally Hang Indefinitely In-Reply-To: <6ee00146-26d3-4a89-9c1f-5811d4ba058d@googlegroups.com> References: <6ee00146-26d3-4a89-9c1f-5811d4ba058d@googlegroups.com> Message-ID: On 29 April 2014 at 02:29:30, Jeff Doto (jeffdoto at gmail.com) wrote: > > We?re wondering if anyone has encountered an issue similar > to this running the versions of Rabbit server / client we are. > Are there any suggestions for what we can do to prevent this? Could > upgrading the server version help? This is a known Java client (or, well, JDK Socket) limitation: http://markmail.org/thread/w7ay4hslxwqirfsc? -- MK Software Engineer, Pivotal/RabbitMQ From mklishin at gopivotal.com Mon Apr 28 23:33:21 2014 From: mklishin at gopivotal.com (Michael Klishin) Date: Tue, 29 Apr 2014 02:33:21 +0400 Subject: [rabbitmq-discuss] RabbitMQ on Azure VM In-Reply-To: <1dac0b0a-d7d9-4c5c-a928-0dcc15c536c2@googlegroups.com> References: <1dac0b0a-d7d9-4c5c-a928-0dcc15c536c2@googlegroups.com> Message-ID: On 29 April 2014 at 02:31:44, William Wallace (amo.libertatem at gmail.com) wrote: > > What we can't do, is get the management console to come up when > we connect to port 15672(tel://port%2015672) either locally > or remotely. Anybody know what the trick is to make it work? have you enabled the plugin and restarted RabbitMQ? Can port 15672 be blocked by a firewall? What exactly do you observe?? -- MK Software Engineer, Pivotal/RabbitMQ From Jesse.Smithnosky at emc.com Tue Apr 29 01:37:35 2014 From: Jesse.Smithnosky at emc.com (Smithnosky, Jesse) Date: Mon, 28 Apr 2014 20:37:35 -0400 Subject: [rabbitmq-discuss] ssl handshake failure Message-ID: Hi All, I am trying to connect from one broker to another using SSL. I am using these instructions as my primary guide. I have three certificates/keys involved: (1) client_ec_cert.pem (the client's public key) (2) client_ec_pri_key.pem(the client's private key) (3) root_ec_cert.pm (the authoritative certificate) First of all, note that we are using Elliptic curve certs and keys. I wouldn't have thought that would be a problem, but at this point I am not ruling anything out. I converted the .pem files from (1) and (3) to .der format using a command from this guide of the following form: Convert PEM to DER openssl x509 -outform der -in certificate.pem -out certificate.der I then successfully imported these two .der files into my keystore using a command that looks like this (except with the correct file names and paths, etc). # keytool -import -alias server1 -file /path/to/server/cert.der -keystore /path/to/rabbitstore This completed with no errors. For some reason, however, I was unable to convert the private key in (2) in such fashion, and had to do something like this: openssl pkcs12 -export -in client_ec_cert.pem -inkey client_ec_pri_key.pem > client.p12 I tried several versions of this conversion, and successfully imported a few of the results into my keystore. Each time, however, my java code results in a javax.net.ssl.SSLHandshakeException. public final boolean declareBrmExchangesAndQueuesSSL(String brmHostname) { try { char [] keyPassphrase = "password".toCharArray(); KeyStore ks = KeyStore.getInstance("PKCS12"); ks.load(new FileInputStream("/etc/rabbitmq/client.p12"), keyPassphrase); KeyManagerFactory kmf = KeyManagerFactory.getInstance("SunX509"); kmf.init(ks, keyPassphrase); char[] trustPassphrase = keyPassphrase; KeyStore tks = KeyStore.getInstance("JKS"); tks.load(new FileInputStream("/usr/local/avamar/lib/avamar_keystore"), trustPassphrase); TrustManagerFactory tmf = TrustManagerFactory.getInstance("SunX509"); tmf.init(tks); SSLContext context = SSLContext.getInstance("SSLv3"); context.init(kmf.getKeyManagers(), tmf.getTrustManagers(), new SecureRandom()); ConnectionFactory factory = new ConnectionFactory(); factory.setHost(brmHostname); factory.setPort(5671); factory.useSslProtocol(context); Connection conn = factory.newConnection(); Channel channel = conn.createChannel(); //rest of code omitted...failure happens right here. } catch (IOException e) { e.printStackTrace(); log.log(Level.WARNING, e.getMessage()); return false; } catch (KeyManagementException e) { e.printStackTrace(); log.log(Level.WARNING, e.getMessage()); return false; } catch (NoSuchAlgorithmException e) { e.printStackTrace(); log.log(Level.WARNING, e.getMessage()); return false; } catch (CertificateException e) { e.printStackTrace(); log.log(Level.WARNING, e.getMessage()); return false; } catch (KeyStoreException e) { e.printStackTrace(); log.log(Level.WARNING, e.getMessage()); return false; } catch (UnrecoverableKeyException e) { e.printStackTrace(); log.log(Level.WARNING, e.getMessage()); return false; } return true; } Am I doing anything obviously incorrectly? I also attempted adding the server's public key to my client's keystore, but the result was the same. Any help would be greatly appreciated. Thanks, Jesse Smithnosky -------------- next part -------------- An HTML attachment was scrubbed... URL: From fancyerii at gmail.com Tue Apr 29 02:27:12 2014 From: fancyerii at gmail.com (Li Li) Date: Tue, 29 Apr 2014 09:27:12 +0800 Subject: [rabbitmq-discuss] install rabbitmq problem in rhel 6.3 Message-ID: 1. I installed erlang from source 2. >sudo yum install rabbitmq-server-3.3.0-1.noarch.rpm Loaded plugins: fastestmirror, product-id, subscription-manager Updating certificate-based repositories. Unable to read consumer identity Loading mirror speeds from cached hostfile base | 3.7 kB 00:00 http://repos.fedorapeople.org/repos/peter/erlang/epel-%24releasever/x86_64/repodata/repomd.xml: [Errno 14] PYCURL ERROR 22 - "The requested URL returned error: 404" Trying other mirror. extras | 3.4 kB 00:00 updates | 3.4 kB 00:00 Setting up Install Process Examining rabbitmq-server-3.3.0-1.noarch.rpm: rabbitmq-server-3.3.0-1.noarch Marking rabbitmq-server-3.3.0-1.noarch.rpm to be installed Resolving Dependencies --> Running transaction check ---> Package rabbitmq-server.noarch 0:3.3.0-1 will be installed --> Processing Dependency: erlang >= R13B-03 for package: rabbitmq-server-3.3.0-1.noarch --> Finished Dependency Resolution Error: Package: rabbitmq-server-3.3.0-1.noarch (/rabbitmq-server-3.3.0-1.noarch) Requires: erlang >= R13B-03 You could try using --skip-broken to work around the problem ** Found 1 pre-existing rpmdb problem(s), 'yum check' output follows: rhn-check-1.0.0-87.el6.noarch has missing requires of yum-rhn-plugin >= ('0', '0.9.1', '35') the erlang version: [lili at DC-TEST-1 ~]$ erl Erlang/OTP 17 [erts-6.0] [source] [64-bit] [smp:4:4] [async-threads:10] [hipe] [kernel-poll:false] Eshell V6.0 (abort with ^G) 1> From carl.hoerberg at gmail.com Tue Apr 29 03:15:18 2014 From: carl.hoerberg at gmail.com (carlhoerberg) Date: Mon, 28 Apr 2014 19:15:18 -0700 (PDT) Subject: [rabbitmq-discuss] ssl_handshake error Message-ID: <1398737718676-35156.post@n5.nabble.com> With RabbitMQ 3.3.0 we sometimes see error like this: (I changed some of the numbers in there in case it's our private key...) This is our ssl_options config in rabbitmq.config: -- View this message in context: http://rabbitmq.1065348.n5.nabble.com/ssl-handshake-error-tp35156.html Sent from the RabbitMQ mailing list archive at Nabble.com. From srikanthtns at gmail.com Tue Apr 29 04:19:50 2014 From: srikanthtns at gmail.com (srikanth tns) Date: Mon, 28 Apr 2014 20:19:50 -0700 Subject: [rabbitmq-discuss] Rabbitmq cluster design and HA Message-ID: Hi we are thinking to setup a 3-node Rabbitmq cluster . each member in the cluster will have 5000 clients connected to it. The client is mcollective. We made sure the client would connect to only one node in the cluster . So, this would be approximately 50K queues on each member in the cluster.All of our hosts are RHEL machines. Can you let us know what would be optimal settings on rabbitmq cluster to prevent it from crashing and having reliable messaging? 1) sysctl configurations 2) Disk limitations to setup on cluster . the default is 50MB 3) Memory to used , right now its default 40% 4) any other settings to be done ? 5) mnesia / erlang settings 6) any disk space limitation on the hosts For HA , we are using ha-mode all to duplicate the queues across all nodes in the cluster. Do you think its reliable during the failover scenario ?During the failover it would like 100k queues on nthe new master? Or should we use ha-mode = exactly/nodes ? Thanks Srikanth -------------- next part -------------- An HTML attachment was scrubbed... URL: From alan.antonuk at gmail.com Tue Apr 29 05:10:50 2014 From: alan.antonuk at gmail.com (Alan Antonuk) Date: Mon, 28 Apr 2014 21:10:50 -0700 Subject: [rabbitmq-discuss] [rabbitmq_c] Infinite wait in login_with_properties function In-Reply-To: <1398680404528-35118.post@n5.nabble.com> References: <1387786326818-32275.post@n5.nabble.com> <1388037935031-32306.post@n5.nabble.com> <1398680404528-35118.post@n5.nabble.com> Message-ID: rabbitmq-c does have limited support for doing non-blocking operations on sockets, I do not surface it in the amqp_login() functions currently, as the the implementation of such a thing is not straight-forward to get right in all cases. If there's enough interest I would consider adding this, though I really haven't seen a huge demand for such a thing. -Alan On Mon, Apr 28, 2014 at 3:20 AM, Haster wrote: > Alan, > But is it possible to limit such timeout or use nonblocked version of recv? > And can you add this functionality to rabbitmq_c library? > > > > -- > View this message in context: > http://rabbitmq.1065348.n5.nabble.com/rabbitmq-c-Infinite-wait-in-login-with-properties-function-tp32275p35118.html > Sent from the RabbitMQ mailing list archive at Nabble.com. > _______________________________________________ > rabbitmq-discuss mailing list > rabbitmq-discuss at lists.rabbitmq.com > https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss > -------------- next part -------------- An HTML attachment was scrubbed... URL: From janets at nairial.net Tue Apr 29 00:55:22 2014 From: janets at nairial.net (Janet Sullivan) Date: Mon, 28 Apr 2014 23:55:22 +0000 Subject: [rabbitmq-discuss] RabbitMQ on Azure VM In-Reply-To: <1dac0b0a-d7d9-4c5c-a928-0dcc15c536c2@googlegroups.com> References: <1dac0b0a-d7d9-4c5c-a928-0dcc15c536c2@googlegroups.com> Message-ID: <4881b4186a7f4c918cfc8baf88e3a3c4@BY2PR07MB043.namprd07.prod.outlook.com> What happens if you telnet to localhost port 15672 from the machine itself? Do you have management configured? I run multiple RabbitMQ boxes on Azure without issue. From: rabbitmq-discuss [mailto:rabbitmq-discuss-bounces at lists.rabbitmq.com] On Behalf Of William Wallace Sent: Monday, April 28, 2014 1:36 PM To: rabbitmq-discuss at googlegroups.com Subject: [rabbitmq-discuss] RabbitMQ on Azure VM We're trying to set up RabbitMQ on an Azure VM and are not having success. We have installed Erlang and RabbitMQ, can see the service is running, and get status from rabitmqctl on the command line. What we can't do, is get the management console to come up when we connect to port 15672 either locally or remotely. Anybody know what the trick is to make it work? -------------- next part -------------- An HTML attachment was scrubbed... URL: From ngoc.tang at quant-edge.com Tue Apr 29 03:21:16 2014 From: ngoc.tang at quant-edge.com (ngoc.tang at quant-edge.com) Date: Mon, 28 Apr 2014 19:21:16 -0700 (PDT) Subject: [rabbitmq-discuss] RabbitMQ 3.3.0 doesn't working with OpenLDAP ???? Message-ID: Dear All I am setup RabbitMQ 3.3.0 anh config authentication with Open Ldap, but it not work. *This is the log file: *when i try logon =INFO REPORT==== 29-Apr-2014::08:48:36 === Starting RabbitMQ 3.3.0 on Erlang R14B04 Copyright (C) 2007-2013 GoPivotal, Inc. Licensed under the MPL. See http://www.rabbitmq.com/ =INFO REPORT==== 29-Apr-2014::08:48:36 === node : rabbit at jabber home dir : /var/lib/rabbitmq config file(s) : /etc/rabbitmq/rabbitmq.config cookie hash : 35evYSGOFlwFMikMqgLbqw== log : /var/log/rabbitmq/rabbit at jabber.log sasl log : /var/log/rabbitmq/rabbit at jabber-sasl.log database dir : /var/lib/rabbitmq/mnesia/rabbit at jabber =INFO REPORT==== 29-Apr-2014::08:48:37 === Limiting to approx 924 file handles (829 sockets) =INFO REPORT==== 29-Apr-2014::08:48:37 === Memory limit set to 395MB of 989MB total. =INFO REPORT==== 29-Apr-2014::08:48:37 === Disk free limit set to 40000MB =INFO REPORT==== 29-Apr-2014::08:48:37 === msg_store_transient: using rabbit_msg_store_ets_index to provide index =INFO REPORT==== 29-Apr-2014::08:48:37 === msg_store_persistent: using rabbit_msg_store_ets_index to provide index =INFO REPORT==== 29-Apr-2014::08:48:47 === started TCP Listener on 10.0.25.132:5672 =INFO REPORT==== 29-Apr-2014::08:48:52 === started SSL Listener on 10.0.25.132:5671 =INFO REPORT==== 29-Apr-2014::08:48:57 === Management plugin started. Port: 15672 =INFO REPORT==== 29-Apr-2014::08:48:57 === Statistics database started. =INFO REPORT==== 29-Apr-2014::08:48:57 === Server startup complete; 8 plugins started. * amqp_client * mochiweb * rabbitmq_auth_mechanism_ssl * rabbitmq_federation_management * rabbitmq_management * rabbitmq_management_agent * rabbitmq_web_dispatch * webmachine =ERROR REPORT==== 29-Apr-2014::08:58:10 === webmachine error: path="/api/whoami" {error,undef, [{rabbit_auth_backend_ldap,check_user_login, [<<"user.test">>, [{password,<<"abc123456">>}]]}, {rabbit_access_control,try_login,3}, {rabbit_access_control,'-check_user_login/2-fun-0-',4}, {lists,foldl,3}, {rabbit_access_control,check_user_login,2}, {rabbit_mgmt_util,is_authorized,6}, {webmachine_resource,resource_call,3}, {webmachine_resource,do,3}]} *And this is config file* [ > {rabbit, [ > {tcp_listeners, [{"10.x.x.x",5672}]}, > {ssl_listeners, [{"10.x.x.x",5671}]}, > {ssl_options, [{cacertfile,"/etc/rabbitmq/ssl/nvca/cacert.pem"}, > {certfile,"/etc/rabbitmq/ssl/server/cert.pem"}, > {keyfile,"/etc/rabbitmq/ssl/server/key.pem"}, > {verify,verify_none}, > {fail_if_no_peer_cert,false}]}, > {heartbeat, 15}, > {vm_memory_high_watermark_paging_ratio, 0.75}, > {vm_memory_high_watermark, 0.4}, > {auth_backends, [{rabbit_auth_backend_ldap, > rabbit_auth_backend_internal}, > rabbit_auth_backend_internal]}, > {disk_free_limit, 40000000000} > ]}, > {rabbitmq_auth_backend_ldap, > [ {servers, ["ldapserver.com"]}, > {dn_lookup_attribute, {"uid","%u"}}, > {dn_lookup_base, "dc=ldapserver,dc=com"}, > {other_bind, > ["cn=manager,cn=internal,dc=ldapserver,dc=com","sTwQCDWHTuuNO5bo"]}, > {use_ssl, false}, > {port, 389}, > {log, network}, > {tag_queries, [{administrator, {constant, true}}, > {management, {constant, true}}]} > ] > } > ]. *Information of LDAP server is*: LDAP Base DNdc=ldapserver,dc=comLDAP Bind DN cn=manager,cn=internal,dc=ldapserver,dc=comLDAP Bind Password Ad09DSJwidjdwf89D Can anybody help me troubleshoot problem, Please Thanks so much -------------- next part -------------- An HTML attachment was scrubbed... URL: From ngoc.tang at quant-edge.com Tue Apr 29 03:35:38 2014 From: ngoc.tang at quant-edge.com (=?UTF-8?Q?Ng=E1=BB=8Dc_T=C4=83ng_V=C4=83n?=) Date: Mon, 28 Apr 2014 19:35:38 -0700 (PDT) Subject: [rabbitmq-discuss] RabbitMQ 3.3.0 doesn't working with OpenLDAP ???? In-Reply-To: References: Message-ID: <7f038186-e1e7-4e48-99c4-a56997d5caac@googlegroups.com> Sorry. LDAP Bind Password in config file i was modified it. In real config file, It's match with OpenLDAP -------------- next part -------------- An HTML attachment was scrubbed... URL: From ewulf84 at gmail.com Tue Apr 29 05:03:08 2014 From: ewulf84 at gmail.com (ewulf84 at gmail.com) Date: Mon, 28 Apr 2014 21:03:08 -0700 (PDT) Subject: [rabbitmq-discuss] ConnectionFactory.CreateConnection() exception when using IPv4 address Message-ID: <54ce9c0e-843b-45bb-83ce-aa7d158678b8@googlegroups.com> I have a straightforward issue: If you supply an IPv4 address to a new AmqpTcpEndpoint in an IPv6-enabled Windows environment (Windows7) then pass that endpoint to a connection factory it will not be able to create connections. It seems that if the client environment supports IPv6 you have to use an IPv6 address or a hostname. My sample code is running the 3.3.0 RabbitMQ .Net client: var factory = new ConnectionFactory() { UserName = "eric", Password = "pwd" }; // WORKS (ipv6 addr) // var uri = new Uri("amqp-0-9://[2001:a9fe:3100:1:f06c:255a:43e2:1459]:5672"); // WORKS (hostname) // var uri = new Uri("amqp-0-9://s6-2k8r2-3:5672"); // Times out (mapped ipv4 [socket is probably not dual-stack?]) // var uri = new Uri("amqp-0-9://[0:0:0:0:0:ffff:172.31.195.246]:5672"); // Throws a System.ArgumentException in CreateConnection() var uri = new Uri("amqp-0-9://172.31.195.246:5672"); factory.Endpoint = new AmqpTcpEndpoint(uri); using (var connection = factory.CreateConnection()) { using (var channel = connection.CreateModel()) { channel.QueueDeclare("test", false, false, false, null); string message = "Hello World"; var body = Encoding.UTF8.GetBytes(message); channel.BasicPublish("", "hello", null, body); Console.WriteLine(" [x] Sent {0}", message); } } The problem appears to be in the way the SocketFrameHandler_0_9.ctor() determines the socket address family: if (Socket.OSSupportsIPv6) { try { this.m_socket = socketFactory(AddressFamily.InterNetworkV6); this.Connect(this.m_socket, endpoint, timeout); } catch (ConnectFailureException) { this.m_socket = null; } } This may not be the best way to determine whether to use an IPv6 socket or an IPv4 socket. The ArgumentException which is getting thrown from down in the Socket's DoDnsCallback is not caught in the FrameHandler ctor and instead makes its way up to the caller. You may want to consider IPAddress.TryParsing the host from the Amqp endpoint and then getting the address family from that. It would probably be faster than this method, too. As it stands I cannot figure out a workaround for this issue. If the only way you can access a machine from an IPv6-enabled Windows box is via its IPv4 address you would appear to be out of luck. In the .Net client, anyway. Thanks, Eric -------------- next part -------------- An HTML attachment was scrubbed... URL: From 77.ears at gmail.com Tue Apr 29 06:01:37 2014 From: 77.ears at gmail.com (77 ears) Date: Mon, 28 Apr 2014 22:01:37 -0700 Subject: [rabbitmq-discuss] Cannot start RabbitMQ server Message-ID: I cannot get RabbitMQ server started after configured the two nodes cluster. The error message is not much and unclear. Any help is appreciated. Status of node 'rabbit at qot-job-02' ... Error: unable to connect to node 'rabbit at qot-job-02': nodedown DIAGNOSTICS =========== attempted to contact: ['rabbit at xxx-xxx-01'] rabbit at xxx-xxx-01: * rabbit seems not to be running at all * no other nodes on xxx-xxx-01 current node details: - node name: 'rabbitmqctl21253 at xxx-xxx-01' - home dir: /var/lib/rabbitmq - cookie hash: 8fSuUG9sekKrk+JdL5S4VQ== -------------- next part -------------- An HTML attachment was scrubbed... URL: From loganathan.ms at gmail.com Tue Apr 29 09:05:40 2014 From: loganathan.ms at gmail.com (Loganathan Sellapa) Date: Tue, 29 Apr 2014 13:35:40 +0530 Subject: [rabbitmq-discuss] [RUBY AMPQ] SSL LOGIN ERROR EXTERNAL login refused Message-ID: Hi All, I tried to setup SSL onmy development machine by following the instructions provided in https://www.rabbitmq.com/ssl.html. And then created rabbitmq.cong file with the below settings. [ {rabbit, [ {ssl_listeners, [5671]}, {ssl_options, [{cacertfile,"/opt/ssl/testca/cacert.pem"}, {certfile,"/opt/ssl/server/cert.pem"}, {keyfile,"/opt/ssl/server/key.pem"}, {verify,verify_peer}, {fail_if_no_peer_cert,true}]}, {auth_mechanisms, ['EXTERNAL']}, {ssl_cert_login_from, common_name} ]} ]. After restarting the rabbitmq server I tried to connect to RabbitMq via AMQP Gem with the below code, but I am getting authentication failure error. *RUBY CODE:* RMQ_CONFIGURATIONS = {:host=>"127.0.0.1", :port=>5671, :vhost=>"/", :auth_mechanism=>"EXTERNAL", :ssl=>{:cert_chain_file=>"/opt/ssl/client/cert.pem", :private_key_file=>"/opt/ssl/client/key.pem"}} AMQP.start(RMQ_CONFIGURATIONS) do |connection| puts "Connected, authenticated. TLS seems to work." connection.disconnect { puts "Now closing the connection..."; EventMachine.stop } end AMQP::PossibleAuthenticationFailureError: AMQP broker closed TCP connection before authentication succeeded: this usually means authentication failure due to misconfiguration. Settings are {:host=>"127.0.0.1", :port=>5671, :user=>"guest", :pass=>"[filtered]", :auth_mechanism=>"EXTERNAL", :vhost=>"/", :timeout=>nil, :logging=>false, :ssl=>{:cert_chain_file=>"/opt/ssl/client/cert.pem", :private_key_file=>"/opt/ssl/client/key.pem"}, :frame_max=>131072, :heartbeat=>0} *RABBITMQ LOG:* ==> /var/log/rabbitmq/rabbit at ubuntu.log <== =INFO REPORT==== 29-Apr-2014::13:26:19 === accepting AMQP connection <0.862.0> (127.0.0.1:40657 -> 127.0.0.1:5671) =ERROR REPORT==== 29-Apr-2014::13:26:22 === closing AMQP connection <0.862.0> (127.0.0.1:40657 -> 127.0.0.1:5671): {handshake_error,starting,0, {amqp_error,access_refused, * "EXTERNAL login refused: user 'ubuntu' - invalid credentials",* 'connection.start_ok'}} I validated the SSL certificates using the below command and I got the VERIFY OK as response. *SSL VALIDATION I/P* openssl s_server -accept 443 -cert /opt/ssl/server/cert.pem -key /opt/ssl/server/key.pem -CAfile /opt/ssl/testca/cacert.pem openssl s_client -connect localhost:443 -cert /opt/ssl/client/cert.pem -key /opt/ssl/client/key.pem -CAfile testca/cacert.pem *SSL VALIDATION O/P* CONNECTED(00000003) depth=1 CN = MyTestCA verify return:1 depth=0 CN = ubuntu, O = server verify return:1 --- Certificate chain 0 s:/CN=ubuntu/O=server i:/CN=MyTestCA 1 s:/CN=MyTestCA i:/CN=MyTestCA --- Server certificate -----BEGIN CERTIFICATE----- MIIC3zCCAcegAwIBAgIBATANBgkqhkiG9w0BAQUFADATMREwDwYDVQQDEwhNeVRl c3RDQTAeFw0xNDA0MjkwNzIzMTFaFw0xNTA0MjkwNzIzMTFaMCIxDzANBgNVBAMM BnVidW50dTEPMA0GA1UECgwGc2VydmVyMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8A MIIBCgKCAQEAw7066GHqFC4xtEwi4tRTDscHCsuSXFne6wpPl3kQ/Cf+91YVosgC fNmsxj4MQQxiPFqr/l6Dhscsdd6OL3PuSfetu+MSS+lD+UmF5QZ5D3ezN9I35sVw NigQ9CCfsBxfGfQDc6llJpt/3V3p/ropijGO2FaIBGp/Xn42FQJSbe8wCwMlulBV 9rSPbFfo71OjXhHH9PcG+RL29omNDY6a2c0vCr9LOJZH8qpCPTFjO0ueFUqIpi31 uaZ/DojNHb/KfZOfaWXRn/oLknfbNdDnxyt+VwNGTJ7NlMo6PuhiB6H60kzZ+44V c8jtR+skt2MMo0WKWCeaqFzM/kWtmU2AzwIDAQABoy8wLTAJBgNVHRMEAjAAMAsG A1UdDwQEAwIFIDATBgNVHSUEDDAKBggrBgEFBQcDATANBgkqhkiG9w0BAQUFAAOC AQEAm1khz04OM/rssO4biVbMKv5g2d7skRByN1TuPVpnc34vVBTVErEBeL19S4VA PwhJm0+E+fVgdPPR/m3Yjh5LvXYpq4pkdFjvHmbdzj5CTiRRJ/SUdCGVKOFuFnU8 ecJtfjz9qK8BKvqlIb0J1FYmUalFaTvpqNYa96y1SmF4MFbcVgu0ZjqKXcHpU0vH 5rsX0VqbqbyQv3oI05BmuR0Ta0AGxMc/VDGRI25oCpcEjICa7shYZDVa8uISZMlc pnPLUxNBKjD7INqwx9y3Ikl70zou3xA6akZHnTFhcn0ak0WSob+O/L8rlNM4ANM/ n3fWiKRnyZ9zRPmx7cB6MsUx0w== -----END CERTIFICATE----- subject=/CN=ubuntu/O=server issuer=/CN=MyTestCA --- No client certificate CA names sent --- SSL handshake has read 2137 bytes and written 375 bytes --- New, TLSv1/SSLv3, Cipher is ECDHE-RSA-AES256-SHA Server public key is 2048 bit Secure Renegotiation IS supported Compression: NONE Expansion: NONE SSL-Session: Protocol : TLSv1.1 Cipher : ECDHE-RSA-AES256-SHA Session-ID: A74004CB078A50AFE2CDA0550BE606DB7747BBF9386ED348129CEDCADDB8B753 Session-ID-ctx: Master-Key: 366B450D2C99556844C47A27B280AAA1F7BA1CC5196B1BB9FC4E5FABC95493C873DBF9375116E3E3036DAC51D406741C Key-Arg : None PSK identity: None PSK identity hint: None SRP username: None TLS session ticket lifetime hint: 300 (seconds) TLS session ticket: 0000 - 4f 5f 65 19 7c 30 46 78-34 58 21 af 6b aa 4e 39 O_e.|0Fx4X!.k.N9 0010 - 58 4a a4 70 0d bb 00 fe-e4 8c 5e dd 9f 66 6d 33 XJ.p......^..fm3 0020 - bf 0a e6 37 a2 8c 07 a4-88 43 de c3 b0 77 2c 9d ...7.....C...w,. 0030 - f1 f4 8a 1b de d5 10 fd-7b 0f bc 91 91 3b ed ba ........{....;.. 0040 - fe bc 9a 35 75 a0 b9 b2-f3 a1 12 da 4c 7a 81 fb ...5u.......Lz.. 0050 - 42 c7 68 7c e8 38 16 d2-dc a9 c3 d6 6e 7a 5d 7d B.h|.8......nz]} 0060 - f6 51 7c ed 9f 0d fe 7d-f1 d2 65 03 3d 56 75 23 .Q|....}..e.=Vu# 0070 - 8c 70 de fb 40 a1 f8 9a-fa 28 85 7a 97 1f e0 5d .p..@ ....(.z...] 0080 - 9b d2 2c e4 67 91 e6 51-d0 05 c9 cc 8f 8e 38 a3 ..,.g..Q......8. 0090 - 9c b7 4d b1 8f 98 c8 1f-a6 59 26 fd fe e1 94 1b ..M......Y&..... Start Time: 1398758603 Timeout : 300 (sec) Verify return code: 0 (ok) --- regards, Loganathan Mob: +91 7760780741 | +91 9944414388 Skype: loganathan.sellappa ViewMe -------------- next part -------------- An HTML attachment was scrubbed... URL: From mklishin at gopivotal.com Tue Apr 29 09:06:47 2014 From: mklishin at gopivotal.com (Michael Klishin) Date: Tue, 29 Apr 2014 12:06:47 +0400 Subject: [rabbitmq-discuss] ConnectionFactory.CreateConnection() exception when using IPv4 address In-Reply-To: <54ce9c0e-843b-45bb-83ce-aa7d158678b8@googlegroups.com> References: <54ce9c0e-843b-45bb-83ce-aa7d158678b8@googlegroups.com> Message-ID: On 29 April 2014 at 11:29:34, ewulf84 at gmail.com (ewulf84 at gmail.com) wrote: > > The problem appears to be in the way the SocketFrameHandler_0_9.ctor() > determines the socket address family: > > if (Socket.OSSupportsIPv6) > { > try > { > this.m_socket = socketFactory(AddressFamily.InterNetworkV6); > this.Connect(this.m_socket, endpoint, timeout); > } > catch (ConnectFailureException) > { > this.m_socket = null; > } > } > > > > This may not be the best way to determine whether to use an IPv6 > socket or an IPv4 socket. The ArgumentException which is getting > thrown from down in the Socket's DoDnsCallback is not caught > in the FrameHandler ctor and instead makes its way up to the caller. > You may want to consider IPAddress.TryParsing the host from > the Amqp endpoint and then getting the address family from that. > It would probably be faster than this method, too. > > As it stands I cannot figure out a workaround for this issue. If > the only way you can access a machine from an IPv6-enabled Windows > box is via its IPv4 address you would appear to be out of luck. In > the .Net client, anyway. Thanks for reporting!? -- MK Software Engineer, Pivotal/RabbitMQ From mklishin at gopivotal.com Tue Apr 29 09:07:41 2014 From: mklishin at gopivotal.com (Michael Klishin) Date: Tue, 29 Apr 2014 12:07:41 +0400 Subject: [rabbitmq-discuss] Cannot start RabbitMQ server In-Reply-To: References: Message-ID: ?On 29 April 2014 at 11:31:40, 77 ears (77.ears at gmail.com) wrote: > > rabbit at xxx-xxx-01: > * rabbit seems not to be running at all Looking at the logs on xxx-xxx-01 should hint at why. -- MK Software Engineer, Pivotal/RabbitMQ From mklishin at gopivotal.com Tue Apr 29 09:13:54 2014 From: mklishin at gopivotal.com (Michael Klishin) Date: Tue, 29 Apr 2014 12:13:54 +0400 Subject: [rabbitmq-discuss] Rabbitmq cluster design and HA In-Reply-To: References: Message-ID: On 29 April 2014 at 07:22:03, srikanth tns (srikanthtns at gmail.com) wrote: > > Can you let us know what would be optimal settings on rabbitmq > cluster to prevent it from crashing and having reliable messaging? It's really hard to make general recommendations without knowing what kind of workload you use. > 1) sysctl configurations nlimit > 20,000 or so (definitely can't be less than ~ 6K) > 2) Disk limitations to setup on cluster . the default is 50MB I'd use 500 MB or so if RabbitMQ is the primary thing that runs on the machine. > 3) Memory to used , right now its default 40% Can be up to 90%, again, depending on what else may run on the same machine. > 6) any disk space limitation on the hosts Host is this different from 2) above? > For HA , we are using ha-mode all to duplicate the queues across > all nodes in the cluster. Do you think its reliable during the > failover scenario ?During the failover it would like 100k queues > on nthe new master? Or should we use ha-mode = exactly/nodes ? With 3 nodes ha-mode=all is a reasonable amount of duplication. If you are confident you will never lose more than 1 node at a time, use ha-mode=exactly with 2 nodes. Less duplication means better throughput. As to how well new master election will work with 100K queues, try it.? -- MK Software Engineer, Pivotal/RabbitMQ From mklishin at gopivotal.com Tue Apr 29 09:18:07 2014 From: mklishin at gopivotal.com (Michael Klishin) Date: Tue, 29 Apr 2014 12:18:07 +0400 Subject: [rabbitmq-discuss] [RUBY AMPQ] SSL LOGIN ERROR EXTERNAL login refused In-Reply-To: References: Message-ID: On 29 April 2014 at 12:08:34, Loganathan Sellapa (loganathan.ms at gmail.com) wrote: > > I tried to setup SSL onmy development machine by following the > instructions provided in https://www.rabbitmq.com/ssl.html. > And then created rabbitmq.cong file with the below settings. > > [ > {rabbit, [ > {ssl_listeners, [5671]}, > {ssl_options, [{cacertfile,"/opt/ssl/testca/cacert.pem"}, > {certfile,"/opt/ssl/server/cert.pem"}, > {keyfile,"/opt/ssl/server/key.pem"}, > {verify,verify_peer}, > {fail_if_no_peer_cert,true}]}, > {auth_mechanisms, ['EXTERNAL']}, > {ssl_cert_login_from, common_name} > ]} > ]. > > After restarting the rabbitmq server I tried to connect to RabbitMq > via AMQP Gem with the below code, but I am getting authentication > failure error. > > RUBY CODE: > > RMQ_CONFIGURATIONS = {:host=>"127.0.0.1", :port=>5671, > :vhost=>"/", :auth_mechanism=>"EXTERNAL", :ssl=>{:cert_chain_file=>"/opt/ssl/client/cert.pem", > :private_key_file=>"/opt/ssl/client/key.pem"}} Do your RabbitMQ server and client actually use different CA certificate files or is this a typo?? -- MK Software Engineer, Pivotal/RabbitMQ From mklishin at gopivotal.com Tue Apr 29 09:18:54 2014 From: mklishin at gopivotal.com (Michael Klishin) Date: Tue, 29 Apr 2014 12:18:54 +0400 Subject: [rabbitmq-discuss] ssl_handshake error In-Reply-To: <1398737718676-35156.post@n5.nabble.com> References: <1398737718676-35156.post@n5.nabble.com> Message-ID: ?On 29 April 2014 at 06:16:31, carlhoerberg (carl.hoerberg at gmail.com) wrote: > > With RabbitMQ 3.3.0 we sometimes see error like this: > > > > (I changed some of the numbers in there in case it's our private > key...) > > This is our ssl_options config in rabbitmq.config: > Carl, Maybe it's just my email client but I don't see any error message or ssl_options in your email. -- MK Software Engineer, Pivotal/RabbitMQ From mklishin at gopivotal.com Tue Apr 29 09:21:40 2014 From: mklishin at gopivotal.com (Michael Klishin) Date: Tue, 29 Apr 2014 12:21:40 +0400 Subject: [rabbitmq-discuss] install rabbitmq problem in rhel 6.3 In-Reply-To: References: Message-ID: ?On 29 April 2014 at 05:29:26, Li Li (fancyerii at gmail.com) wrote: > > Requires: erlang >= R13B-03 > You could try using --skip-broken to work around the problem http://markmail.org/thread/vy4cg55inipiusos -- MK Software Engineer, Pivotal/RabbitMQ From mklishin at gopivotal.com Tue Apr 29 09:22:50 2014 From: mklishin at gopivotal.com (Michael Klishin) Date: Tue, 29 Apr 2014 12:22:50 +0400 Subject: [rabbitmq-discuss] ssl handshake failure In-Reply-To: References: Message-ID: ?On 29 April 2014 at 04:39:01, Smithnosky, Jesse (jesse.smithnosky at emc.com) wrote: > > First of all, note that we are using Elliptic curve(http://en.wikipedia.org/wiki/Elliptic_curve_cryptography) > certs and keys. I wouldn?t have thought that would be a problem, > but at this point I am not ruling anything out. Jesse, ECC support in Erlang/OTP R16 is limited: http://erlang.org/pipermail/erlang-bugs/2013-October/003782.html I'm not sure if it's any better in R17 but can you please try it? -- MK Software Engineer, Pivotal/RabbitMQ From 0x6e6562 at gmail.com Tue Apr 29 09:28:10 2014 From: 0x6e6562 at gmail.com (Ben Hood) Date: Tue, 29 Apr 2014 09:28:10 +0100 Subject: [rabbitmq-discuss] HA load balancing Message-ID: Hi all, Is anybody aware of any load balancers that maintain the AMQP conversation when a node dies and the client has not been designed to detect failures and reconnect via the load balancer? Also, to what extent can the use of a network load balancer (either in software or hardware) be called an RabbitMQ load balancer when it is not aware of the location of the queue that a client is sending data to or receiving data from? Cheers, Ben From simon at rabbitmq.com Tue Apr 29 09:43:55 2014 From: simon at rabbitmq.com (Simon MacMullen) Date: Tue, 29 Apr 2014 09:43:55 +0100 Subject: [rabbitmq-discuss] [RUBY AMPQ] SSL LOGIN ERROR EXTERNAL login refused In-Reply-To: References: Message-ID: <535F664B.1070404@rabbitmq.com> On 29/04/2014 09:05, Loganathan Sellapa wrote: > After restarting the rabbitmq server I tried to connect to RabbitMq via > AMQP Gem with the below code, but I am getting authentication failure error. > > *_RUBY CODE:_* > * > * > RMQ_CONFIGURATIONS = {:host=>"127.0.0.1", :port=>5671, :vhost=>"/", > :auth_mechanism=>"EXTERNAL", > :ssl=>{:cert_chain_file=>"/opt/ssl/client/cert.pem", > :private_key_file=>"/opt/ssl/client/key.pem"}} Do you actually want to use EXTERNAL authentication? That's the authentication mode where you do not specify a username and password, but the authentication is done via SSL client certs and the username comes from the client cert DN / CN. > =ERROR REPORT==== 29-Apr-2014::13:26:22 === > closing AMQP connection <0.862.0> (127.0.0.1:40657 > -> 127.0.0.1:5671 ): > {handshake_error,starting,0, > {amqp_error,access_refused, > * "EXTERNAL login refused: user 'ubuntu' - invalid credentials",* > 'connection.start_ok'}} Note that the user is 'ubuntu' here, not 'guest'. I guess that's the CN of your client cert. RabbitMQ is complianing because although the SSL-based auth succeeded, it then couldn't find the user in the internal database (to find out what permissions it has). So if you *do* want to use EXTERNAL, you need to make sure that the user 'ubuntu' exists in the user database. The user can be set to have no password. You can also remove the If you do *not* want to use EXTERNAL and instead wanted to log in as 'guest' secured by SSL, remove 'auth_mechanisms' and 'ssl_cert_login_from' from the config. Cheers, Simon From simon at rabbitmq.com Tue Apr 29 09:46:54 2014 From: simon at rabbitmq.com (Simon MacMullen) Date: Tue, 29 Apr 2014 09:46:54 +0100 Subject: [rabbitmq-discuss] RabbitMQ 3.3.0 doesn't working with OpenLDAP ???? In-Reply-To: References: Message-ID: <535F66FE.2070202@rabbitmq.com> On 29/04/2014 03:21, ngoc.tang at quant-edge.com wrote: > =INFO REPORT==== 29-Apr-2014::08:48:57 === > Server startup complete; 8 plugins started. > * amqp_client > * mochiweb > * rabbitmq_auth_mechanism_ssl > * rabbitmq_federation_management > * rabbitmq_management > * rabbitmq_management_agent > * rabbitmq_web_dispatch > * webmachine You haven't enabled the LDAP plugin. Cheers, Simon From matthias at rabbitmq.com Tue Apr 29 09:49:21 2014 From: matthias at rabbitmq.com (Matthias Radestock) Date: Tue, 29 Apr 2014 09:49:21 +0100 Subject: [rabbitmq-discuss] install rabbitmq problem in rhel 6.3 In-Reply-To: References: Message-ID: <535F6791.9060600@rabbitmq.com> On 29/04/14 09:21, Michael Klishin wrote: > On 29 April 2014 at 05:29:26, Li Li (fancyerii at gmail.com) wrote: >>> Requires: erlang >= R13B-03 >> You could try using --skip-broken to work around the problem > > http://markmail.org/thread/vy4cg55inipiusos ...though really you shouldn't be installing Erlang from source anyway. If you want to run rabbit with the most recent Erlang, then follow the "Install Erlang from Erlang Solutions" instructions at https://www.rabbitmq.com/install-rpm.html Matthias. From loganathan.ms at gmail.com Tue Apr 29 09:54:33 2014 From: loganathan.ms at gmail.com (Loganathan Sellapa) Date: Tue, 29 Apr 2014 14:24:33 +0530 Subject: [rabbitmq-discuss] [RUBY AMPQ] SSL LOGIN ERROR EXTERNAL login refused In-Reply-To: References: Message-ID: Hi Michael, I followed the steps mentioned in the rabbitMq website. And I hope I have separate certificates for client & server. root at ubuntu:/opt/ssl/client# ls cert.pem keycert.p12 key-cert.pem key.pem req.pem root at ubuntu:/opt/ssl/server# ls cert.pem keycert.p12 key.pem req.pem regards, Loganathan Mob: +91 7760780741 | +91 9944414388 Skype: loganathan.sellappa ViewMe On Tue, Apr 29, 2014 at 1:48 PM, Michael Klishin wrote: > On 29 April 2014 at 12:08:34, Loganathan Sellapa (loganathan.ms at gmail.com) > wrote: > > > I tried to setup SSL onmy development machine by following the > > instructions provided in https://www.rabbitmq.com/ssl.html. > > And then created rabbitmq.cong file with the below settings. > > > > [ > > {rabbit, [ > > {ssl_listeners, [5671]}, > > {ssl_options, [{cacertfile,"/opt/ssl/testca/cacert.pem"}, > > {certfile,"/opt/ssl/server/cert.pem"}, > > {keyfile,"/opt/ssl/server/key.pem"}, > > {verify,verify_peer}, > > {fail_if_no_peer_cert,true}]}, > > {auth_mechanisms, ['EXTERNAL']}, > > {ssl_cert_login_from, common_name} > > ]} > > ]. > > > > After restarting the rabbitmq server I tried to connect to RabbitMq > > via AMQP Gem with the below code, but I am getting authentication > > failure error. > > > > RUBY CODE: > > > > RMQ_CONFIGURATIONS = {:host=>"127.0.0.1", :port=>5671, > > :vhost=>"/", :auth_mechanism=>"EXTERNAL", > :ssl=>{:cert_chain_file=>"/opt/ssl/client/cert.pem", > > :private_key_file=>"/opt/ssl/client/key.pem"}} > > Do your RabbitMQ server and client actually use different CA certificate > files > or is this a typo? > -- > MK > > Software Engineer, Pivotal/RabbitMQ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From loganathan.ms at gmail.com Tue Apr 29 09:58:21 2014 From: loganathan.ms at gmail.com (Loganathan Sellapa) Date: Tue, 29 Apr 2014 14:28:21 +0530 Subject: [rabbitmq-discuss] [RUBY AMPQ] SSL LOGIN ERROR EXTERNAL login refused In-Reply-To: <535F664B.1070404@rabbitmq.com> References: <535F664B.1070404@rabbitmq.com> Message-ID: Hi Simon, Yes, I want to use EXTERNAL authentication by giving SSL certificates instead of username & password. Can you let me know the way to set user name while generating the SSL certificates, since I didnt find any steps for this inhttps://www.rabbitmq.com/ssl.html regards, Loganathan Mob: +91 7760780741 | +91 9944414388 Skype: loganathan.sellappa ViewMe On Tue, Apr 29, 2014 at 2:13 PM, Simon MacMullen wrote: > On 29/04/2014 09:05, Loganathan Sellapa wrote: > >> After restarting the rabbitmq server I tried to connect to RabbitMq via >> AMQP Gem with the below code, but I am getting authentication failure >> error. >> >> *_RUBY CODE:_* >> * >> >> * >> RMQ_CONFIGURATIONS = {:host=>"127.0.0.1", :port=>5671, :vhost=>"/", >> :auth_mechanism=>"EXTERNAL", >> :ssl=>{:cert_chain_file=>"/opt/ssl/client/cert.pem", >> :private_key_file=>"/opt/ssl/client/key.pem"}} >> > > Do you actually want to use EXTERNAL authentication? That's the > authentication mode where you do not specify a username and password, but > the authentication is done via SSL client certs and the username comes from > the client cert DN / CN. > > =ERROR REPORT==== 29-Apr-2014::13:26:22 === >> closing AMQP connection <0.862.0> (127.0.0.1:40657 >> -> 127.0.0.1:5671 ): >> {handshake_error,starting,0, >> {amqp_error,access_refused, >> * "EXTERNAL login refused: user 'ubuntu' - invalid credentials",* >> 'connection.start_ok'}} >> > > Note that the user is 'ubuntu' here, not 'guest'. I guess that's the CN of > your client cert. RabbitMQ is complianing because although the SSL-based > auth succeeded, it then couldn't find the user in the internal database (to > find out what permissions it has). > > So if you *do* want to use EXTERNAL, you need to make sure that the user > 'ubuntu' exists in the user database. The user can be set to have no > password. You can also remove the > > If you do *not* want to use EXTERNAL and instead wanted to log in as > 'guest' secured by SSL, remove 'auth_mechanisms' and 'ssl_cert_login_from' > from the config. > > Cheers, Simon > -------------- next part -------------- An HTML attachment was scrubbed... URL: From simon at rabbitmq.com Tue Apr 29 10:04:04 2014 From: simon at rabbitmq.com (Simon MacMullen) Date: Tue, 29 Apr 2014 10:04:04 +0100 Subject: [rabbitmq-discuss] [RUBY AMPQ] SSL LOGIN ERROR EXTERNAL login refused In-Reply-To: References: <535F664B.1070404@rabbitmq.com> Message-ID: <535F6B04.2070403@rabbitmq.com> On 29/04/2014 09:58, Loganathan Sellapa wrote: > Can you let me know the way to set user name while generating the SSL > certificates, since I didnt find any steps for this > inhttps://www.rabbitmq.com/ssl.html Replace: -subj /CN=$(hostname)/O=client/ with -subj /CN=my-user-name/O=client/ (or similar) when generating the client cert. That's assuming you're using OpenSSL. Cheers, Simon From videlalvaro at gmail.com Tue Apr 29 10:11:35 2014 From: videlalvaro at gmail.com (Alvaro Videla) Date: Tue, 29 Apr 2014 11:11:35 +0200 Subject: [rabbitmq-discuss] Cannot start RabbitMQ server In-Reply-To: References: Message-ID: Hi, Make sure you are using the same user that started the rabbitmq-server as the user that called rabbitmqctl, since you need to be able to share the erlang.cookie among other things. Regards, Alvaro On Tue, Apr 29, 2014 at 10:07 AM, Michael Klishin wrote: > On 29 April 2014 at 11:31:40, 77 ears (77.ears at gmail.com) wrote: >> > rabbit at xxx-xxx-01: >> * rabbit seems not to be running at all > > Looking at the logs on xxx-xxx-01 should hint at why. > -- > MK > > Software Engineer, Pivotal/RabbitMQ > _______________________________________________ > rabbitmq-discuss mailing list > rabbitmq-discuss at lists.rabbitmq.com > https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss From loganathan.ms at gmail.com Tue Apr 29 10:13:05 2014 From: loganathan.ms at gmail.com (Loganathan Sellapa) Date: Tue, 29 Apr 2014 14:43:05 +0530 Subject: [rabbitmq-discuss] [RUBY AMPQ] SSL LOGIN ERROR EXTERNAL login refused In-Reply-To: <535F6B04.2070403@rabbitmq.com> References: <535F664B.1070404@rabbitmq.com> <535F6B04.2070403@rabbitmq.com> Message-ID: Hi Simon, Thanks, I can able to connect with RabbitMq. Also I hope password in the below line is rabbitMq password right? openssl pkcs12 -export -out keycert.p12 -in cert.pem -inkey key.pem -passout pass:*MySecretpassword (I replaced with guest user password)* regards, Loganathan Mob: +91 7760780741 | +91 9944414388 Skype: loganathan.sellappa ViewMe On Tue, Apr 29, 2014 at 2:34 PM, Simon MacMullen wrote: > On 29/04/2014 09:58, Loganathan Sellapa wrote: > >> Can you let me know the way to set user name while generating the SSL >> certificates, since I didnt find any steps for this >> inhttps://www.rabbitmq.com/ssl.html >> > > Replace: > > -subj /CN=$(hostname)/O=client/ > > with > > -subj /CN=my-user-name/O=client/ > > (or similar) when generating the client cert. That's assuming you're using > OpenSSL. > > Cheers, Simon > -------------- next part -------------- An HTML attachment was scrubbed... URL: From fancyerii at gmail.com Tue Apr 29 10:19:02 2014 From: fancyerii at gmail.com (Li Li) Date: Tue, 29 Apr 2014 17:19:02 +0800 Subject: [rabbitmq-discuss] install rabbitmq problem in rhel 6.3 In-Reply-To: <535F6791.9060600@rabbitmq.com> References: <535F6791.9060600@rabbitmq.com> Message-ID: thank you. I have used rabbitmq-server-generic-unix-3.3.0.tar.gz any differences between rpm and tar.gz? On Tue, Apr 29, 2014 at 4:49 PM, Matthias Radestock wrote: > On 29/04/14 09:21, Michael Klishin wrote: >> >> On 29 April 2014 at 05:29:26, Li Li (fancyerii at gmail.com) wrote: >>>> >>>> Requires: erlang >= R13B-03 >>> >>> You could try using --skip-broken to work around the problem >> >> >> http://markmail.org/thread/vy4cg55inipiusos > > > ...though really you shouldn't be installing Erlang from source anyway. If > you want to run rabbit with the most recent Erlang, then follow the "Install > Erlang from Erlang Solutions" instructions at > https://www.rabbitmq.com/install-rpm.html > > Matthias. From matthias at rabbitmq.com Tue Apr 29 10:24:01 2014 From: matthias at rabbitmq.com (Matthias Radestock) Date: Tue, 29 Apr 2014 10:24:01 +0100 Subject: [rabbitmq-discuss] install rabbitmq problem in rhel 6.3 In-Reply-To: References: <535F6791.9060600@rabbitmq.com> Message-ID: <535F6FB1.8010905@rabbitmq.com> On 29/04/14 10:19, Li Li wrote: > thank you. I have used rabbitmq-server-generic-unix-3.3.0.tar.gz > any differences between rpm and tar.gz? The rpm sets up a separate user account for rabbit, configures log rotation, and startup scripts, etc. i.e. does everything you'd expect a well-behaved service to do on an rpm-based system. The generic unix install does none of that. Unless there are compelling reasons not to, I strongly recommend you follow the installation instructions for the specific platform, i.e. in your case http://www.rabbitmq.com/install-rpm.html Matthias. From bra at fsn.hu Tue Apr 29 10:27:16 2014 From: bra at fsn.hu (Nagy, Attila) Date: Tue, 29 Apr 2014 11:27:16 +0200 Subject: [rabbitmq-discuss] LIFO queue In-Reply-To: <535E31A7.6060703@rabbitmq.com> References: <4BCC459C.50303@250bpm.com> <535E0FB7.3040708@fsn.hu> <535E31A7.6060703@rabbitmq.com> Message-ID: <535F7074.4040308@fsn.hu> On 04/28/14 12:47, Simon MacMullen wrote: > On 28/04/2014 09:22, Nagy, Attila wrote: >> Hence, I would need a LIFO queue in RabbitMQ too. > > Feel free to provide an alternate implementation: > > http://hg.rabbitmq.com/rabbitmq-server/file/02d1d855e4b3/src/rabbit_backing_queue.erl > I guess I would need to get some erlang skills first. :) > > On a slightly more serious note, you could use TTL + DLX to at least > get "old" messages into a lower priority queue. But you probably > already knew that. In this combination (mixing priorities in) I haven't thought about that. Sadly, to be somewhat useful, it will be much uglier and more complex than a LIFO queue. Is this much harder to implement on the server side, or you don't see enough interest? Thanks as always, From carl.hoerberg at gmail.com Tue Apr 29 10:28:38 2014 From: carl.hoerberg at gmail.com (carlhoerberg) Date: Tue, 29 Apr 2014 02:28:38 -0700 (PDT) Subject: [rabbitmq-discuss] ssl_handshake error In-Reply-To: References: <1398737718676-35156.post@n5.nabble.com> Message-ID: Oh, see here http://rabbitmq.1065348.n5.nabble.com/ssl-handshake-error-tp35156p35169.html Or I'll repost it later On Apr 29, 2014 4:22 PM, "Michael Klishin-2 [via RabbitMQ]" < ml-node+s1065348n35169h60 at n5.nabble.com> wrote: > On 29 April 2014 at 06:16:31, carlhoerberg ([hidden email]) > wrote: > > > With RabbitMQ 3.3.0 we sometimes see error like this: > > > > > > > > (I changed some of the numbers in there in case it's our private > > key...) > > > > This is our ssl_options config in rabbitmq.config: > > > > Carl, > > Maybe it's just my email client but I don't see any error message > or ssl_options in your email. > -- > MK > > Software Engineer, Pivotal/RabbitMQ > _______________________________________________ > rabbitmq-discuss mailing list > [hidden email] > https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss > > > ------------------------------ > If you reply to this email, your message will be added to the discussion > below: > > http://rabbitmq.1065348.n5.nabble.com/ssl-handshake-error-tp35156p35169.html > To unsubscribe from ssl_handshake error, click here > . > NAML > -- View this message in context: http://rabbitmq.1065348.n5.nabble.com/ssl-handshake-error-tp35156p35183.html Sent from the RabbitMQ mailing list archive at Nabble.com. -------------- next part -------------- An HTML attachment was scrubbed... URL: From simon at rabbitmq.com Tue Apr 29 10:55:31 2014 From: simon at rabbitmq.com (Simon MacMullen) Date: Tue, 29 Apr 2014 10:55:31 +0100 Subject: [rabbitmq-discuss] LIFO queue In-Reply-To: <535F7074.4040308@fsn.hu> References: <4BCC459C.50303@250bpm.com> <535E0FB7.3040708@fsn.hu> <535E31A7.6060703@rabbitmq.com> <535F7074.4040308@fsn.hu> Message-ID: <535F7713.2040906@rabbitmq.com> On 29/04/2014 10:27, Nagy, Attila wrote: > Sadly, to be somewhat useful, it will be much uglier and more complex > than a LIFO queue. Oh, of course. > Is this much harder to implement on the server side, or you don't see > enough interest? Bit of both. We've had a lot more requests for priority queues than LIFO over the years, and we're only just starting to work on them. LIFO seems like a pretty obscure request with only two people asking in 4 years I'm afraid. And it's not completely obvious how to implement it; if you wanted to do it in terms of the existing queue implementation you'd have the problem that it's designed to move queues through to the head; that one way movement is a fairly big assumption (and it's more code than you would think - due to persistence and paging large parts of the queue may not be in memory at any given time). Or you could do a new implementation; the in-memory version would be trivial but then you'd have to do persistence from scratch if you wanted it; also not very appealing. Or you could back to SQL or whatever; performance would be poor but you'd be able to do all sorts of random-access things and LIFO would fall out very easily. But these are all quite big tasks I'm afraid. Cheers, Simon From lepolac at gmail.com Tue Apr 29 10:59:21 2014 From: lepolac at gmail.com (lepolac) Date: Tue, 29 Apr 2014 02:59:21 -0700 (PDT) Subject: [rabbitmq-discuss] Federated queue and bindings Message-ID: <1398765561189-35186.post@n5.nabble.com> Hi, I have configured queue federation between 2 brokers, but can't see messages on my downstream broker. Here is the configuration on the downstream : eval 'rabbit_federation_status:status().' [[{queue,<<"federated-queue.test">>}, {upstream_queue,<<"federated-queue.test">>}, {type,queue}, {vhost,<<"/">>}, {upstream,<<"my-upstream">>}, {status,running}, {local_connection,<<"">>}, {uri,<<"amqp://host1:17580">>}, {timestamp,{{2014,4,29},{5,22,42}}}]] ...done. ./rabbitmqctl list_policies Listing policies ... / federate-me exchanges ^federated {"federation-upstream-set":"all"} 0 / federate-queue queues ^federated-queue {"federation-upstream-set":"all"} 0 Listing bindings ... exchange A queue A [] exchange federated-queue.test queue federated-queue.test [] myexchange exchange federated-queue.test queue [] ...done. Bindings on upstream host1 : Listing bindings ... exchange A queue A [] exchange federated-queue.test queue federated-queue.test [] myexchange exchange federated-queue.test queue [] ...done. I have a feeling this has to do with the way my publisher or consumer is binding. QUEUE_NAME being obviously federated-queue.test and EXCHANGE_NAME being myexchange Consumer : channel.queueDeclare(QUEUE_NAME, true, false, false, null); channel.exchangeDeclare(EXCHANGE_NAME, "direct", true); channel.queueBind(QUEUE_NAME,EXCHANGE_NAME,""); Publisher : channel.queueDeclare(QUEUE_NAME,true, false, false, null); channel.exchangeDeclare(EXCHANGE_NAME, "direct", true); channel.queueBind(QUEUE_NAME,EXCHANGE_NAME,""); .... channel.basicPublish(EXCHANGE_NAME,"",null, message.getBytes()); This works locally, I can publish/consume messages on the same broker, but if I run a consumer on the downstream I can't see messages. Could use some help here, thanks :) -- View this message in context: http://rabbitmq.1065348.n5.nabble.com/Federated-queue-and-bindings-tp35186.html Sent from the RabbitMQ mailing list archive at Nabble.com. From simon at rabbitmq.com Tue Apr 29 10:59:45 2014 From: simon at rabbitmq.com (Simon MacMullen) Date: Tue, 29 Apr 2014 10:59:45 +0100 Subject: [rabbitmq-discuss] [RUBY AMPQ] SSL LOGIN ERROR EXTERNAL login refused In-Reply-To: References: <535F664B.1070404@rabbitmq.com> <535F6B04.2070403@rabbitmq.com> Message-ID: <535F7811.9080508@rabbitmq.com> On 29/04/2014 10:13, Loganathan Sellapa wrote: > > Thanks, I can able to connect with RabbitMq. Also I hope password in the > below line is rabbitMq password right? > > openssl pkcs12 -export -out keycert.p12 -in cert.pem -inkey key.pem > -passout pass:*MySecretpassword (I replaced with guest user password)* > * > * No, that's the password OpenSSL should use for the output file. Unless OpenSSL ignores it in that context, I'm not sure. Certificate passwords and RabbitMQ passwords have nothing to do with each other. And when using EXTERNAL auth you don't need RabbitMQ passwords at all. Cheers, Simon From loganathan.ms at gmail.com Tue Apr 29 11:04:44 2014 From: loganathan.ms at gmail.com (Loganathan Sellapa) Date: Tue, 29 Apr 2014 15:34:44 +0530 Subject: [rabbitmq-discuss] [RUBY AMPQ] SSL LOGIN ERROR EXTERNAL login refused In-Reply-To: <535F7811.9080508@rabbitmq.com> References: <535F664B.1070404@rabbitmq.com> <535F6B04.2070403@rabbitmq.com> <535F7811.9080508@rabbitmq.com> Message-ID: Got it, thanks for the assistance. regards, Loganathan Mob: +91 7760780741 | +91 9944414388 Skype: loganathan.sellappa ViewMe On Tue, Apr 29, 2014 at 3:29 PM, Simon MacMullen wrote: > On 29/04/2014 10:13, Loganathan Sellapa wrote: > >> >> Thanks, I can able to connect with RabbitMq. Also I hope password in the >> below line is rabbitMq password right? >> >> openssl pkcs12 -export -out keycert.p12 -in cert.pem -inkey key.pem >> -passout pass:*MySecretpassword (I replaced with guest user password)* >> * >> * >> > > No, that's the password OpenSSL should use for the output file. Unless > OpenSSL ignores it in that context, I'm not sure. > > Certificate passwords and RabbitMQ passwords have nothing to do with each > other. And when using EXTERNAL auth you don't need RabbitMQ passwords at > all. > > Cheers, Simon > -------------- next part -------------- An HTML attachment was scrubbed... URL: From michael.laing at nytimes.com Tue Apr 29 11:19:57 2014 From: michael.laing at nytimes.com (Laing, Michael) Date: Tue, 29 Apr 2014 06:19:57 -0400 Subject: [rabbitmq-discuss] LIFO queue In-Reply-To: <535F7713.2040906@rabbitmq.com> References: <4BCC459C.50303@250bpm.com> <535E0FB7.3040708@fsn.hu> <535E31A7.6060703@rabbitmq.com> <535F7074.4040308@fsn.hu> <535F7713.2040906@rabbitmq.com> Message-ID: We do LIFO external to RabbitMQ with Cassandra. We ttl the data and never explicitly delete - we're interested mostly in the recent stuff so we do queries like 'give me the latest 200'. It's fast and we handle jillions of messages. You can use redis for this too, I believe, tho I am not sure about ttl. Or you could tinker with zmq. Good luck, Michael On Tue, Apr 29, 2014 at 5:55 AM, Simon MacMullen wrote: > On 29/04/2014 10:27, Nagy, Attila wrote: > >> Sadly, to be somewhat useful, it will be much uglier and more complex >> than a LIFO queue. >> > > Oh, of course. > > > Is this much harder to implement on the server side, or you don't see >> enough interest? >> > > Bit of both. We've had a lot more requests for priority queues than LIFO > over the years, and we're only just starting to work on them. LIFO seems > like a pretty obscure request with only two people asking in 4 years I'm > afraid. > > And it's not completely obvious how to implement it; if you wanted to do > it in terms of the existing queue implementation you'd have the problem > that it's designed to move queues through to the head; that one way > movement is a fairly big assumption (and it's more code than you would > think - due to persistence and paging large parts of the queue may not be > in memory at any given time). > > Or you could do a new implementation; the in-memory version would be > trivial but then you'd have to do persistence from scratch if you wanted > it; also not very appealing. Or you could back to SQL or whatever; > performance would be poor but you'd be able to do all sorts of > random-access things and LIFO would fall out very easily. > > But these are all quite big tasks I'm afraid. > > Cheers, Simon > > _______________________________________________ > rabbitmq-discuss mailing list > rabbitmq-discuss at lists.rabbitmq.com > https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss > -------------- next part -------------- An HTML attachment was scrubbed... URL: From Marek.Hudik at deutsche-boerse.com Tue Apr 29 14:05:17 2014 From: Marek.Hudik at deutsche-boerse.com (Marek.Hudik at deutsche-boerse.com) Date: Tue, 29 Apr 2014 15:05:17 +0200 Subject: [rabbitmq-discuss] rabbitmq_tracing documantation Message-ID: Hi, I am using rabbitmq_tracing plugin to log all message and have problem with setup and "pattern" field. Only the value "#" is working for me. There is no documentation for this field and I am not sure for which message field is this pattern used for. Probably for filtering. But according which field - exchange, routing key or a property? Could you add additional documentation. Thanks Marek Hudik ---------------------------------------------------------------------------- Deutsche B?rse Services s.r.o. Managing Directors/Gesch?ftsf?hrung: Michael Gassmann, Mats Andersson. Limited liability company with registered office at Sokolovsk? 662/136B, CZ-186 00 Prague 8 recorded in the Commercial Register IC: 275 77 015. Maintained by the city court in Prague, Sec. C, File No. 116874. ----------------------------------------- Diese E-Mail enthaelt vertrauliche oder rechtlich geschuetzte Informationen. Wenn Sie nicht der beabsichtigte Empfaenger sind, informieren Sie bitte sofort den Absender und loeschen Sie diese E-Mail. Das unbefugte Kopieren dieser E-Mail oder die unbefugte Weitergabe der enthaltenen Informationen ist nicht gestattet. The information contained in this message is confidential or protected by law. If you are not the intended recipient, please contact the sender and delete this message. Any unauthorised copying of this message or unauthorised distribution of the information contained herein is prohibited. Legally required information for business correspondence/ Gesetzliche Pflichtangaben fuer Geschaeftskorrespondenz: http://deutsche-boerse.com/letterhead -------------- next part -------------- An HTML attachment was scrubbed... URL: From simon at rabbitmq.com Tue Apr 29 14:08:11 2014 From: simon at rabbitmq.com (Simon MacMullen) Date: Tue, 29 Apr 2014 14:08:11 +0100 Subject: [rabbitmq-discuss] rabbitmq_tracing documantation In-Reply-To: References: Message-ID: <535FA43B.1000108@rabbitmq.com> On 29/04/2014 14:05, Marek.Hudik at deutsche-boerse.com wrote: > I am using rabbitmq_tracing plugin to log all message and have problem > with setup and "pattern" field. Only the value "#" is working for me. > There is no documentation for this field and I am not sure for which > message field is this pattern used for. Probably for filtering. But > according which field - exchange, routing key or a property? See "Firehose notification format" at http://www.rabbitmq.com/firehose.html Cheers, Simon From simon at rabbitmq.com Tue Apr 29 14:09:31 2014 From: simon at rabbitmq.com (Simon MacMullen) Date: Tue, 29 Apr 2014 14:09:31 +0100 Subject: [rabbitmq-discuss] RabbitMQ 3.3.1 released Message-ID: <535FA48B.6060800@rabbitmq.com> The RabbitMQ team is pleased to announce the release of RabbitMQ 3.3.1 This release fixes a number of bugs in 3.3.0 and earlier versions, including security bugs in the MQTT and shovel plugins. See the release notes at: http://www.rabbitmq.com/release-notes/README-3.3.1.txt for more information. The new release can be downloaded from: http://www.rabbitmq.com/download.html As always, we welcome any questions, bug reports, and other feedback on this release, as well as general suggestions for features and enhancements in future releases. Mail us via the RabbitMQ discussion list. Regards, The RabbitMQ Team (http://www.rabbitmq.com) From Marek.Hudik at deutsche-boerse.com Tue Apr 29 14:02:54 2014 From: Marek.Hudik at deutsche-boerse.com (Marek.Hudik at deutsche-boerse.com) Date: Tue, 29 Apr 2014 15:02:54 +0200 Subject: [rabbitmq-discuss] rabbitmq_tracing documantation Message-ID: Hi, I am using rabbitmq_tracing plugin to log all message and have problem with setup and "pattern" field. Only the value "#" is working for me. There is no documentation for this field and I am not sure for which message field is this pattern used for. Probably for filtering. But according which field - exchange, routing key or a property? Could you add additional documentation. Thanks Marek Hudik ---------------------------------------------------------------------------- Deutsche B?rse Services s.r.o. Managing Directors/Gesch?ftsf?hrung: Michael Gassmann, Mats Andersson. Limited liability company with registered office at Sokolovsk? 662/136B, CZ-186 00 Prague 8 recorded in the Commercial Register IC: 275 77 015. Maintained by the city court in Prague, Sec. C, File No. 116874. ----------------------------------------- Diese E-Mail enthaelt vertrauliche oder rechtlich geschuetzte Informationen. Wenn Sie nicht der beabsichtigte Empfaenger sind, informieren Sie bitte sofort den Absender und loeschen Sie diese E-Mail. Das unbefugte Kopieren dieser E-Mail oder die unbefugte Weitergabe der enthaltenen Informationen ist nicht gestattet. The information contained in this message is confidential or protected by law. If you are not the intended recipient, please contact the sender and delete this message. Any unauthorised copying of this message or unauthorised distribution of the information contained herein is prohibited. Legally required information for business correspondence/ Gesetzliche Pflichtangaben fuer Geschaeftskorrespondenz: http://deutsche-boerse.com/letterhead -------------- next part -------------- An HTML attachment was scrubbed... URL: From simon at rabbitmq.com Tue Apr 29 15:00:43 2014 From: simon at rabbitmq.com (Simon MacMullen) Date: Tue, 29 Apr 2014 15:00:43 +0100 Subject: [rabbitmq-discuss] Federated queue and bindings In-Reply-To: <1398765561189-35186.post@n5.nabble.com> References: <1398765561189-35186.post@n5.nabble.com> Message-ID: <535FB08B.6060805@rabbitmq.com> On 29/04/2014 10:59, lepolac wrote: > I have configured queue federation between 2 brokers, but can't see messages > on my downstream broker. How are you checking this? Messages will only be transferred to the downstream on demand - i.e. when there is a consumer there and inadequate consumers upstream. So you can't just expect to see messages piling up in the management UI. > Here is the configuration on the downstream : that all looks fine. > I have a feeling this has to do with the way my publisher or consumer is > binding. > QUEUE_NAME being obviously federated-queue.test and EXCHANGE_NAME being > myexchange No, queue federation has nothing to do with bindings; everything happens at the consumer end. Cheers, Simon From martijn.otto at copernica.com Tue Apr 29 15:14:15 2014 From: martijn.otto at copernica.com (Martijn Otto) Date: Tue, 29 Apr 2014 16:14:15 +0200 Subject: [rabbitmq-discuss] Missing Cancel-OK frame In-Reply-To: <535E59A7.8040007@rabbitmq.com> References: <1398691618.17773.12.camel@martijn-laptop> <535E59A7.8040007@rabbitmq.com> Message-ID: <1398780855.22177.0.camel@martijn-laptop> Ah, I had totally missed that part of the specification. The problem is indeed solved by waiting for the Cancel-OK frame to arrive before continuing. Thanks for pointing that out to me. I am puzzled, however, why the flow-ok method is not a synchronous method. All of the other '-ok' methods are marked as synchronous. I don't really get that concept. An OK frame should only be sent as a result of receiving a frame (in this case the 'flow' frame). If that other frame is synchronous, the server would have nothing else to send (because nothing else should have been received before sending the response). I don't see what is gained by making this method asynchronous and I don't understand how it would work. Sorry if I'm asking stupid questions here. On Mon, 2014-04-28 at 14:37 +0100, Matthias Radestock wrote: > On 28/04/14 14:26, Martijn Otto wrote: > > I am working on an AMQP library for RabbitMQ and either I am missing > > something totally fundamental, or RabbitMQ is failing in its duty to > > send me a Cancel-OK frame to confirm consuming has stopped. > > You are pipelining synchronous commands. The AMQP spec does not permit > that, so the behaviour is undefined. See the pseudo code in section > 2.2.2 of https://www.rabbitmq.com/resources/specs/amqp0-9-1.pdf. > > Matthias. From matthias at rabbitmq.com Tue Apr 29 15:20:02 2014 From: matthias at rabbitmq.com (Matthias Radestock) Date: Tue, 29 Apr 2014 15:20:02 +0100 Subject: [rabbitmq-discuss] Missing Cancel-OK frame In-Reply-To: <1398780855.22177.0.camel@martijn-laptop> References: <1398691618.17773.12.camel@martijn-laptop> <535E59A7.8040007@rabbitmq.com> <1398780855.22177.0.camel@martijn-laptop> Message-ID: <535FB512.1090902@rabbitmq.com> On 29/04/14 15:14, Martijn Otto wrote: > I am puzzled, however, why the flow-ok method is not a synchronous > method. All of the other '-ok' methods are marked as synchronous. It's likely a bug in the spec. In a way, the 'synchronous' marker is somewhat superfluous on '-ok' methods - the synchronous/asynchronous distinction doesn't really make sense for response methods. Matthias. From mklishin at gopivotal.com Tue Apr 29 15:32:30 2014 From: mklishin at gopivotal.com (Michael Klishin) Date: Tue, 29 Apr 2014 18:32:30 +0400 Subject: [rabbitmq-discuss] ssl_handshake error In-Reply-To: References: <1398737718676-35156.post@n5.nabble.com> Message-ID: ?On 29 April 2014 at 13:29:36, carlhoerberg (carl.hoerberg at gmail.com) wrote: > > Oh, see here http://rabbitmq.1065348.n5.nabble.com/ssl-handshake-error-tp35156p35169.html ** {function_clause,[{tls_v1,enum_to_oid, ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?[65282], ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?[{file,"tls_v1.erl"},{line,404}]}, ? ? ? ? ? ? ? ? ? ? ?{ssl_handshake,'-dec_hello_extensions/2-blc$^1/1-0-',1, ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? [{file,"ssl_handshake.erl"},{line,1657}]}, ? ? ? ? ? ? ? ? ? ? ?{ssl_handshake,'-dec_hello_extensions/2-blc$^1/1-0-',1, ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? [{file,"ssl_handshake.erl"},{line,1657}]}, ? ? ? ? ? ? ? ? ? ? ?{ssl_handshake,dec_hello_extensions,2, ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? [{file,"ssl_handshake.erl"},{line,1657}]}, ? ? ? ? ? ? ? ? ? ? ?{tls_handshake,decode_handshake,3, ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? [{file,"tls_handshake.erl"},{line,182}]}, ? ? ? ? ? ? ? ? ? ? ?{tls_handshake,get_tls_handshake_aux,3, ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? [{file,"tls_handshake.erl"},{line,153}]}, ? ? ? ? ? ? ? ? ? ? ?{tls_connection,next_state,4, ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?[{file,"tls_connection.erl"},{line,454}]}, ? ? ? ? ? ? ? ? ? ? ?{gen_fsm,handle_msg,7, ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? [{file,"gen_fsm.erl"},{line,505}]}]} suggests it may be an Erlang ssl app issue. How do you generate your keys and what Erlang version do you run? -- MK Software Engineer, Pivotal/RabbitMQ From carl.hoerberg at gmail.com Tue Apr 29 15:39:19 2014 From: carl.hoerberg at gmail.com (carlhoerberg) Date: Tue, 29 Apr 2014 07:39:19 -0700 (PDT) Subject: [rabbitmq-discuss] ssl_handshake error In-Reply-To: References: <1398737718676-35156.post@n5.nabble.com> Message-ID: Erlang R16B3-1. Key generated by Firefox, cert generated by StartSSL. Note that we haven't found a faulty client (yet), only this showing up in the log, ie. both AMQPS and HTTPS to the mgmt interface works in the normal case.. So maybe not high priority. On Tuesday 29 April 2014 at 22:33, Michael Klishin-2 [via RabbitMQ] wrote: > On 29 April 2014 at 13:29:36, carlhoerberg ([hidden email] (/user/SendEmail.jtp?type=node&node=35197&i=0)) wrote: > > > Oh, see here http://rabbitmq.1065348.n5.nabble.com/ssl-handshake-error-tp35156p35169.html > > ** {function_clause,[{tls_v1,enum_to_oid, > [65282], > [{file,"tls_v1.erl"},{line,404}]}, > {ssl_handshake,'-dec_hello_extensions/2-blc$^1/1-0-',1, > [{file,"ssl_handshake.erl"},{line,1657}]}, > {ssl_handshake,'-dec_hello_extensions/2-blc$^1/1-0-',1, > [{file,"ssl_handshake.erl"},{line,1657}]}, > {ssl_handshake,dec_hello_extensions,2, > [{file,"ssl_handshake.erl"},{line,1657}]}, > {tls_handshake,decode_handshake,3, > [{file,"tls_handshake.erl"},{line,182}]}, > {tls_handshake,get_tls_handshake_aux,3, > [{file,"tls_handshake.erl"},{line,153}]}, > {tls_connection,next_state,4, > [{file,"tls_connection.erl"},{line,454}]}, > {gen_fsm,handle_msg,7, > [{file,"gen_fsm.erl"},{line,505}]}]} > > suggests it may be an Erlang ssl app issue. How do you generate your keys > and what Erlang version do you run? > -- > MK > > Software Engineer, Pivotal/RabbitMQ > _______________________________________________ > rabbitmq-discuss mailing list > [hidden email] (/user/SendEmail.jtp?type=node&node=35197&i=1) > https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss > > > If you reply to this email, your message will be added to the discussion below: http://rabbitmq.1065348.n5.nabble.com/ssl-handshake-error-tp35156p35197.html > To unsubscribe from ssl_handshake error, click here (http://rabbitmq.1065348.n5.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=35156&code=Y2FybC5ob2VyYmVyZ0BnbWFpbC5jb218MzUxNTZ8LTEyNDcxMDc4NjM=). > NAML (http://rabbitmq.1065348.n5.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml) -- View this message in context: http://rabbitmq.1065348.n5.nabble.com/ssl-handshake-error-tp35156p35198.html Sent from the RabbitMQ mailing list archive at Nabble.com. -------------- next part -------------- An HTML attachment was scrubbed... URL: From matthias at rabbitmq.com Tue Apr 29 15:45:35 2014 From: matthias at rabbitmq.com (Matthias Radestock) Date: Tue, 29 Apr 2014 15:45:35 +0100 Subject: [rabbitmq-discuss] ssl_handshake error In-Reply-To: References: <1398737718676-35156.post@n5.nabble.com> Message-ID: <535FBB0F.9090802@rabbitmq.com> On 29/04/14 15:39, carlhoerberg wrote: > Erlang R16B3-1. The error looks quite similar to http://erlang.org/pipermail/erlang-questions/2014-March/078083.html. I think that may have been fixed in R17. In any case R17 contains a bunch of SSL/crypto fixes, so it's worth giving that a try. Matthias. From carl.hoerberg at gmail.com Tue Apr 29 15:49:43 2014 From: carl.hoerberg at gmail.com (carlhoerberg) Date: Tue, 29 Apr 2014 07:49:43 -0700 (PDT) Subject: [rabbitmq-discuss] ssl_handshake error In-Reply-To: <535FBB0F.9090802@rabbitmq.com> References: <1398737718676-35156.post@n5.nabble.com> <535FBB0F.9090802@rabbitmq.com> Message-ID: Ahh, good catch. Yes, will :) On Tuesday 29 April 2014 at 22:46, Matthias Radestock-3 [via RabbitMQ] wrote: > On 29/04/14 15:39, carlhoerberg wrote: > > Erlang R16B3-1. > > The error looks quite similar to > http://erlang.org/pipermail/erlang-questions/2014-March/078083.html. I > think that may have been fixed in R17. In any case R17 contains a bunch > of SSL/crypto fixes, so it's worth giving that a try. > > Matthias. > _______________________________________________ > rabbitmq-discuss mailing list > [hidden email] (/user/SendEmail.jtp?type=node&node=35199&i=0) > https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss > > > If you reply to this email, your message will be added to the discussion below: http://rabbitmq.1065348.n5.nabble.com/ssl-handshake-error-tp35156p35199.html > To unsubscribe from ssl_handshake error, click here (http://rabbitmq.1065348.n5.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=35156&code=Y2FybC5ob2VyYmVyZ0BnbWFpbC5jb218MzUxNTZ8LTEyNDcxMDc4NjM=). > NAML (http://rabbitmq.1065348.n5.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml) -- View this message in context: http://rabbitmq.1065348.n5.nabble.com/ssl-handshake-error-tp35156p35200.html Sent from the RabbitMQ mailing list archive at Nabble.com. -------------- next part -------------- An HTML attachment was scrubbed... URL: From carl.hoerberg at gmail.com Tue Apr 29 16:00:24 2014 From: carl.hoerberg at gmail.com (carlhoerberg) Date: Tue, 29 Apr 2014 08:00:24 -0700 (PDT) Subject: [rabbitmq-discuss] vhost in error reports Message-ID: <1398783624043-35201.post@n5.nabble.com> Would help us a lot if the vhost always was included (when possible) when an error occurs, like for this: =ERROR REPORT==== 19-Apr-2014::00:57:46 === connection <0.11988.251>, channel 2 - soft error: {amqp_error,access_refused, "queue name 'amq.gen-MekSW94u_f7itDHHZMdtNA' contains reserved prefix 'amq.*'", 'queue.declare'} -- View this message in context: http://rabbitmq.1065348.n5.nabble.com/vhost-in-error-reports-tp35201.html Sent from the RabbitMQ mailing list archive at Nabble.com. From lepolac at gmail.com Tue Apr 29 16:03:17 2014 From: lepolac at gmail.com (lepolac) Date: Tue, 29 Apr 2014 08:03:17 -0700 (PDT) Subject: [rabbitmq-discuss] Federated queue and bindings In-Reply-To: <535FB08B.6060805@rabbitmq.com> References: <1398765561189-35186.post@n5.nabble.com> <535FB08B.6060805@rabbitmq.com> Message-ID: <1398783797025-35202.post@n5.nabble.com> Hi, You described exactly the test case I run, a producer upstream, and a consumer downstream. But the consumer doesn't get any messages. "This works locally, I can publish/consume messages on the same broker, but if I run a consumer on the downstream I can't see messages. " Any ideas on what I might be doing wrong ? -- View this message in context: http://rabbitmq.1065348.n5.nabble.com/Federated-queue-and-bindings-tp35186p35202.html Sent from the RabbitMQ mailing list archive at Nabble.com. From carl.hoerberg at gmail.com Tue Apr 29 16:05:31 2014 From: carl.hoerberg at gmail.com (carlhoerberg) Date: Tue, 29 Apr 2014 08:05:31 -0700 (PDT) Subject: [rabbitmq-discuss] Error report explanation Message-ID: <1398783931040-35203.post@n5.nabble.com> Had some server crashes, got these error msgs in the log: =ERROR REPORT==== 19-Apr-2014::00:47:50 === webmachine error: path="/api/nodes/rabbit at node01" {error,badarg, [{ets,lookup,[rabbit_user,<<"admin">>],[]}, {rabbit_misc,dirty_read,1,[]}, {rabbit_auth_backend_internal,internal_check_user_login,2,[]}, {rabbit_access_control,try_login,3,[]}, {lists,foldl,3,[{file,"lists.erl"},{line,1261}]}, {rabbit_access_control,check_user_login,2,[]}, {rabbit_mgmt_util,is_authorized,6,[]}, {webmachine_resource,resource_call,3,[]}]} =ERROR REPORT==== 19-Apr-2014::00:48:32 === ** Generic server rabbit_error_logger_lifecycle terminating ** Last message in was {'EXIT',<0.157.0>,shutdown} ** When Server state == {rabbit_error_logger,stop,[]} ** Reason for termination == ** {{badmatch,{error,module_not_found}}, [{rabbit_error_logger,stop,0,[]}, {supervised_lifecycle,terminate,2,[]}, {gen_server,terminate,6,[{file,"gen_server.erl"},{line,721}]}, {proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,239}]}]} Possible to say what might have gone wrong? rabbitmq 3.2.4, erlang r16b3-1 -- View this message in context: http://rabbitmq.1065348.n5.nabble.com/Error-report-explanation-tp35203.html Sent from the RabbitMQ mailing list archive at Nabble.com. From Marek.Hudik at deutsche-boerse.com Tue Apr 29 16:05:42 2014 From: Marek.Hudik at deutsche-boerse.com (Marek.Hudik at deutsche-boerse.com) Date: Tue, 29 Apr 2014 17:05:42 +0200 Subject: [rabbitmq-discuss] bug: rabbitmq_tracing pattern with multiple entries Message-ID: Hi, I am using rabbitmq_tracing plugin to log message from multiple exchanges to one file. I found a bug regarding "pattern" field and specifying multiple fields. Trace with pattern "publish.exchange1, publish.exchange2" doesn't log anything. But pattern "publish.exchange1" or "publish.exchange2" works without any problems. If the not working pattern should not work by design, please mention it somewhere (best option is Example text under the pattern field). Otherwise please fix it. Thanks Marek Hudik ---------------------------------------------------------------------------- Deutsche B?rse Services s.r.o. Managing Directors/Gesch?ftsf?hrung: Michael Gassmann, Mats Andersson. Limited liability company with registered office at Sokolovsk? 662/136B, CZ-186 00 Prague 8 recorded in the Commercial Register IC: 275 77 015. Maintained by the city court in Prague, Sec. C, File No. 116874. ----------------------------------------- Diese E-Mail enthaelt vertrauliche oder rechtlich geschuetzte Informationen. Wenn Sie nicht der beabsichtigte Empfaenger sind, informieren Sie bitte sofort den Absender und loeschen Sie diese E-Mail. Das unbefugte Kopieren dieser E-Mail oder die unbefugte Weitergabe der enthaltenen Informationen ist nicht gestattet. The information contained in this message is confidential or protected by law. If you are not the intended recipient, please contact the sender and delete this message. Any unauthorised copying of this message or unauthorised distribution of the information contained herein is prohibited. Legally required information for business correspondence/ Gesetzliche Pflichtangaben fuer Geschaeftskorrespondenz: http://deutsche-boerse.com/letterhead -------------- next part -------------- An HTML attachment was scrubbed... URL: From mklishin at gopivotal.com Tue Apr 29 16:35:24 2014 From: mklishin at gopivotal.com (Michael Klishin) Date: Tue, 29 Apr 2014 19:35:24 +0400 Subject: [rabbitmq-discuss] bug: rabbitmq_tracing pattern with multiple entries In-Reply-To: References: Message-ID: ?On 29 April 2014 at 19:07:41, marek.hudik at deutsche-boerse.com (marek.hudik at deutsche-boerse.com) wrote: > > Trace with pattern "publish.exchange1, publish.exchange2" > doesn't log anything. But pattern "publish.exchange1" or "publish.exchange2" > works without any problems. > > If the not working pattern should not work by design, please mention > it somewhere (best option is Example text under the pattern field). > Otherwise please fix it. What leads you to believe your pattern should work? Is it something in the docs? -- MK Software Engineer, Pivotal/RabbitMQ From mklishin at gopivotal.com Tue Apr 29 17:51:38 2014 From: mklishin at gopivotal.com (Michael Klishin) Date: Tue, 29 Apr 2014 20:51:38 +0400 Subject: [rabbitmq-discuss] Cannot start RabbitMQ server In-Reply-To: References: Message-ID: On 29 April 2014 at 20:50:30, 77 ears (77.ears at gmail.com) wrote: > > Thanks Michael. the log is not very helpful. Well, can you post it? Is there nothing at all in the logs? What about syslog (or similar)? -- MK Software Engineer, Pivotal/RabbitMQ From 77.ears at gmail.com Tue Apr 29 17:56:55 2014 From: 77.ears at gmail.com (RabbitLove) Date: Tue, 29 Apr 2014 09:56:55 -0700 (PDT) Subject: [rabbitmq-discuss] Cannot start RabbitMQ server In-Reply-To: References: Message-ID: <1398790615777-35207.post@n5.nabble.com> Thank you both Alvaro and Michael. Thanks for your timely help. I guess cookie could be the issue. What I did was to delete the whole directory of DB, reinstall RabbitMQ it worked. Only reinstall didn't work without deleting the DB. if this happens again, I'll try to check user. Thanks. Emily -- View this message in context: http://rabbitmq.1065348.n5.nabble.com/Cannot-start-RabbitMQ-server-tp35163p35207.html Sent from the RabbitMQ mailing list archive at Nabble.com. From simon at rabbitmq.com Tue Apr 29 18:00:56 2014 From: simon at rabbitmq.com (Simon MacMullen) Date: Tue, 29 Apr 2014 18:00:56 +0100 Subject: [rabbitmq-discuss] Federated queue and bindings In-Reply-To: <1398783797025-35202.post@n5.nabble.com> References: <1398765561189-35186.post@n5.nabble.com> <535FB08B.6060805@rabbitmq.com> <1398783797025-35202.post@n5.nabble.com> Message-ID: <535FDAC8.6050109@rabbitmq.com> On 29/04/2014 16:03, lepolac wrote: > Any ideas on what I might be doing wrong ? No, it sounds like you are doing everything right. Some things it might be worth looking at: When you consume from the downstream queue, you should see a consumer appear on the upstream queue, with an "x-priority" argument of -1. Do you? Do messages get stuck in the upstream queue or disappear? Has an alarm gone off on the downstream? Cheers, Simon From lepolac at gmail.com Tue Apr 29 18:21:56 2014 From: lepolac at gmail.com (lepolac) Date: Tue, 29 Apr 2014 10:21:56 -0700 (PDT) Subject: [rabbitmq-discuss] Federated queue and bindings In-Reply-To: <535FDAC8.6050109@rabbitmq.com> References: <1398765561189-35186.post@n5.nabble.com> <535FB08B.6060805@rabbitmq.com> <1398783797025-35202.post@n5.nabble.com> <535FDAC8.6050109@rabbitmq.com> Message-ID: <1398792116861-35209.post@n5.nabble.com> No I don't see any. Interestingly I noticed that consumers is always 0, even when I put my publisher/consumer together on the upstream or on the downstream, whereas messages are being sent/received correctly. There definitely is the connection and the channel showing in the management interface, but not the consumer. I haven't seen this one before... -- View this message in context: http://rabbitmq.1065348.n5.nabble.com/Federated-queue-and-bindings-tp35186p35209.html Sent from the RabbitMQ mailing list archive at Nabble.com. From matthias at rabbitmq.com Tue Apr 29 18:27:17 2014 From: matthias at rabbitmq.com (Matthias Radestock) Date: Tue, 29 Apr 2014 18:27:17 +0100 Subject: [rabbitmq-discuss] Federated queue and bindings In-Reply-To: <1398792116861-35209.post@n5.nabble.com> References: <1398765561189-35186.post@n5.nabble.com> <535FB08B.6060805@rabbitmq.com> <1398783797025-35202.post@n5.nabble.com> <535FDAC8.6050109@rabbitmq.com> <1398792116861-35209.post@n5.nabble.com> Message-ID: <535FE0F5.2090507@rabbitmq.com> On 29/04/14 18:21, lepolac wrote: > No I don't see any. > Interestingly I noticed that consumers is always 0, even when I put my > publisher/consumer together on the upstream or on the downstream, whereas > messages are being sent/received correctly. > There definitely is the connection and the channel showing in the management > interface, but not the consumer. Are you perhaps consuming messages with basic.get instead of basic.consume? Matthias. From lepolac at gmail.com Tue Apr 29 18:31:10 2014 From: lepolac at gmail.com (lepolac) Date: Tue, 29 Apr 2014 10:31:10 -0700 (PDT) Subject: [rabbitmq-discuss] Federated queue and bindings In-Reply-To: <535FE0F5.2090507@rabbitmq.com> References: <1398765561189-35186.post@n5.nabble.com> <535FB08B.6060805@rabbitmq.com> <1398783797025-35202.post@n5.nabble.com> <535FDAC8.6050109@rabbitmq.com> <1398792116861-35209.post@n5.nabble.com> <535FE0F5.2090507@rabbitmq.com> Message-ID: <1398792670637-35211.post@n5.nabble.com> Yes, spot on, I am -- View this message in context: http://rabbitmq.1065348.n5.nabble.com/Federated-queue-and-bindings-tp35186p35211.html Sent from the RabbitMQ mailing list archive at Nabble.com. From lepolac at gmail.com Tue Apr 29 18:33:51 2014 From: lepolac at gmail.com (lepolac) Date: Tue, 29 Apr 2014 10:33:51 -0700 (PDT) Subject: [rabbitmq-discuss] Federated queue and bindings In-Reply-To: <1398792670637-35211.post@n5.nabble.com> References: <1398765561189-35186.post@n5.nabble.com> <535FB08B.6060805@rabbitmq.com> <1398783797025-35202.post@n5.nabble.com> <535FDAC8.6050109@rabbitmq.com> <1398792116861-35209.post@n5.nabble.com> <535FE0F5.2090507@rabbitmq.com> <1398792670637-35211.post@n5.nabble.com> Message-ID: <1398792831983-35212.post@n5.nabble.com> And the queue federation works when using basicConsume. Wow, what did I miss here ? -- View this message in context: http://rabbitmq.1065348.n5.nabble.com/Federated-queue-and-bindings-tp35186p35212.html Sent from the RabbitMQ mailing list archive at Nabble.com. From mklishin at gopivotal.com Tue Apr 29 18:33:26 2014 From: mklishin at gopivotal.com (Michael Klishin) Date: Tue, 29 Apr 2014 21:33:26 +0400 Subject: [rabbitmq-discuss] Federated queue and bindings In-Reply-To: <1398792670637-35211.post@n5.nabble.com> References: <1398765561189-35186.post@n5.nabble.com> <535FB08B.6060805@rabbitmq.com> <1398783797025-35202.post@n5.nabble.com> <535FDAC8.6050109@rabbitmq.com> <1398792116861-35209.post@n5.nabble.com> <535FE0F5.2090507@rabbitmq.com> <1398792670637-35211.post@n5.nabble.com> Message-ID: On 29 April 2014 at 21:32:22, lepolac (lepolac at gmail.com) wrote: > > Yes, spot on, I am See Limitations in http://www.rabbitmq.com/federated-queues.html? -- MK Software Engineer, Pivotal/RabbitMQ From lepolac at gmail.com Tue Apr 29 18:51:45 2014 From: lepolac at gmail.com (lepolac) Date: Tue, 29 Apr 2014 10:51:45 -0700 (PDT) Subject: [rabbitmq-discuss] Federated queue and bindings In-Reply-To: References: <1398765561189-35186.post@n5.nabble.com> <535FB08B.6060805@rabbitmq.com> <1398783797025-35202.post@n5.nabble.com> <535FDAC8.6050109@rabbitmq.com> <1398792116861-35209.post@n5.nabble.com> <535FE0F5.2090507@rabbitmq.com> <1398792670637-35211.post@n5.nabble.com> Message-ID: <1398793905739-35214.post@n5.nabble.com> good old rtfm, sorry, thanks :) -- View this message in context: http://rabbitmq.1065348.n5.nabble.com/Federated-queue-and-bindings-tp35186p35214.html Sent from the RabbitMQ mailing list archive at Nabble.com. From srinatsr at cisco.com Tue Apr 29 20:02:16 2014 From: srinatsr at cisco.com (Srinath Sridharan -X (srinatsr - ZENSAR TECHNOLOGIES INC at Cisco)) Date: Tue, 29 Apr 2014 19:02:16 +0000 Subject: [rabbitmq-discuss] RabbitMQ Monitoring Queue Activites in Spring Project In-Reply-To: References: <1DB4F5481AB296448E8A5B5BC9D336541CB02E37@xmb-aln-x10.cisco.com> <1DB4F5481AB296448E8A5B5BC9D336541CB030D4@xmb-aln-x10.cisco.com> <1DB4F5481AB296448E8A5B5BC9D336541CB030F4@xmb-aln-x10.cisco.com> Message-ID: <1DB4F5481AB296448E8A5B5BC9D336541CB03982@xmb-aln-x10.cisco.com> I have binded my queue to amq.rabbitmq.log. How to consume some events from there? Regards ?? Srinath -----Original Message----- From: Michael Klishin [mailto:mklishin at gopivotal.com] Sent: Monday, April 28, 2014 11:56 AM To: Discussions about RabbitMQ; Srinath Sridharan -X (srinatsr - ZENSAR TECHNOLOGIES INC at Cisco) Subject: RE: [rabbitmq-discuss] RabbitMQ Monitoring Queue Activites in Spring Project On April 28, 2014 at 10:51:53 PM, Srinath Sridharan -X (srinatsr - ZENSAR TECHNOLOGIES INC at Cisco) (srinatsr at cisco.com) wrote: > > I need to log RabbitMQ activity using a java logger but > > asynchronously You can bind a queue to?amq.rabbitmq.log (a topic exchange in the default vhost ) and consume some events that happen in the broker, then log them using any logger you want. See also?http://www.rabbitmq.com/firehose.html. There is no other way to get a stream of broker activity in a JVM app. -- MK Software Engineer, Pivotal/RabbitMQ From srinatsr at cisco.com Tue Apr 29 20:22:36 2014 From: srinatsr at cisco.com (Srinath Sridharan -X (srinatsr - ZENSAR TECHNOLOGIES INC at Cisco)) Date: Tue, 29 Apr 2014 19:22:36 +0000 Subject: [rabbitmq-discuss] Get Messages based on Exchange and Key Message-ID: <1DB4F5481AB296448E8A5B5BC9D336541CB039CD@xmb-aln-x10.cisco.com> Hello, I need to get messages from my queue based on exchange parameter and routing key parameter. How to achieve this? Regards ...* Srinath -------------- next part -------------- An HTML attachment was scrubbed... URL: From srinatsr at cisco.com Tue Apr 29 20:37:40 2014 From: srinatsr at cisco.com (Srinath Sridharan -X (srinatsr - ZENSAR TECHNOLOGIES INC at Cisco)) Date: Tue, 29 Apr 2014 19:37:40 +0000 Subject: [rabbitmq-discuss] Binding amq.rabbitmq.trace Message-ID: <1DB4F5481AB296448E8A5B5BC9D336541CB03A08@xmb-aln-x10.cisco.com> While binding my queue with rabbitmq.trace. What routing key should I use? I need published and delivered receipts Regards ...* Srinath -------------- next part -------------- An HTML attachment was scrubbed... URL: From mklishin at gopivotal.com Tue Apr 29 21:24:10 2014 From: mklishin at gopivotal.com (Michael Klishin) Date: Wed, 30 Apr 2014 00:24:10 +0400 Subject: [rabbitmq-discuss] Get Messages based on Exchange and Key In-Reply-To: <1DB4F5481AB296448E8A5B5BC9D336541CB039CD@xmb-aln-x10.cisco.com> References: <1DB4F5481AB296448E8A5B5BC9D336541CB039CD@xmb-aln-x10.cisco.com> Message-ID: On April 29, 2014 at 11:23:51 PM, Srinath Sridharan -X (srinatsr - ZENSAR TECHNOLOGIES INC at Cisco) (srinatsr at cisco.com) wrote: > > > I need to get messages from my queue based on exchange parameter > and routing key parameter.? > How to achieve this? Srinath, I highly recommend reading the tutorials [1]. They cover everything you need in order to build a monitoring tool using the amq.rabbitmq.log exchange and also 90% of the things you?d typically do with RabbitMQ in general. In 5 languages and code examples ported to a few more. And it takes less than 1 hour to read the first 5. Is this the kind of time you are willing to invest in order to complete your project?? Asking every little thing here is not going to scale: RabbitMQ team is tiny and every member but 1 is in European time zones (where it?s already well past business hours). There?s only so much hand holding we collectively can do. The tutorials that cover *everything* you need for your monitoring project: ?*?http://www.rabbitmq.com/tutorials/tutorial-one-java.html ?*?http://www.rabbitmq.com/tutorials/tutorial-two-java.html ?*?http://www.rabbitmq.com/tutorials/tutorial-three-java.html (this one can even be skipped) ?*?http://www.rabbitmq.com/tutorials/tutorial-five-java.html when in doubt, clarify protocol concepts and features in?http://www.rabbitmq.com/tutorials/amqp-concepts.html If this still feels too complicated for you and your team, Pivotal offers professional services. You can buy time of one or more engineers experienced with RabbitMQ, and they will build what you need. Contact info at rabbitmq.com if interested. 1.?http://www.rabbitmq.com/getstarted.html -- MK Software Engineer, Pivotal/RabbitMQ From sp345s at att.com Tue Apr 29 21:36:25 2014 From: sp345s at att.com (PATAR, SAGAR) Date: Tue, 29 Apr 2014 20:36:25 +0000 Subject: [rabbitmq-discuss] dynamic binding In-Reply-To: References: <59BF56992750434F89F8EBE141E64B1D01876F97@GAALPA1MSGUSR9C.ITServices.sbc.com> Message-ID: <59BF56992750434F89F8EBE141E64B1D01877432@GAALPA1MSGUSR9C.ITServices.sbc.com> I am using the below code and it is creating the binding object BUT is NOT reflecting on the admin console.. Attaching the src and context file for reference .. Binding b = BindingBuilder.bind(q).to(e).with(binding.getRoutingKey()).and(bArgs); Below is the configuration context file ... From: rabbitmq-discuss [mailto:rabbitmq-discuss-bounces at lists.rabbitmq.com] On Behalf Of Gary Russell Sent: Monday, April 28, 2014 12:09 PM To: Discussions about RabbitMQ Subject: Re: [rabbitmq-discuss] dynamic binding Yes; just add a second binding with the new header value(s); no need to do anything on the listener - it's all a function of the rabbitmq-server. On Mon, Apr 28, 2014 at 11:01 AM, PATAR, SAGAR > wrote: We are using header exchanges and are using spring-rabbit mq for configuration and binding listeners to listen to specific messages with specific message header values. Ex: message header "srcId" has values "1001,1002,1003,1004" At runtime if the same listener bean has to listen to messages with message header "srcId" having value "2001,2002,2003" .. Is there a way to add these bindings dynamically at runtime ..?? Thanks Sagar _______________________________________________ rabbitmq-discuss mailing list rabbitmq-discuss at lists.rabbitmq.com https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: BindingBean.java Type: application/octet-stream Size: 3944 bytes Desc: BindingBean.java URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: amqp-rabbitmq-consumer.xml Type: text/xml Size: 4548 bytes Desc: amqp-rabbitmq-consumer.xml URL: From grussell at gopivotal.com Tue Apr 29 21:50:14 2014 From: grussell at gopivotal.com (Gary Russell) Date: Tue, 29 Apr 2014 16:50:14 -0400 Subject: [rabbitmq-discuss] dynamic binding In-Reply-To: <59BF56992750434F89F8EBE141E64B1D01877432@GAALPA1MSGUSR9C.ITServices.sbc.com> References: <59BF56992750434F89F8EBE141E64B1D01876F97@GAALPA1MSGUSR9C.ITServices.sbc.com> <59BF56992750434F89F8EBE141E64B1D01877432@GAALPA1MSGUSR9C.ITServices.sbc.com> Message-ID: It looks like you are trying to modify an existing binding; I said... >add a second binding with the new header value(s)... Just use admin.declareBinding(newBinding), ...ending up with 2 bindings on the exchange, with the initial binding having the 1xxx matches and the new binding having the 2xxx matches. On Tue, Apr 29, 2014 at 4:36 PM, PATAR, SAGAR wrote: > I am using the below code and it is creating the binding object BUT is > NOT reflecting on the admin console.. Attaching the src and context file > for reference .. > > > > Binding b = BindingBuilder.*bind* > (q).to(e).with(binding.getRoutingKey()).and(bArgs); > > > > Below is the configuration context file ? > > > > addresses="${addresses}" channel-cache-size="${session.size}" /> > > > > > > durable="true" auto-delete="false" /> > > > > "org.springframework.amqp.rabbit.config.BindingFactoryBean"> > > > > > > > > > > > > > > > > > > > > "${listener.thread.size}" > > connection-factory="connectionFactory"> > > > > > > > > > > > > > > *From:* rabbitmq-discuss [mailto: > rabbitmq-discuss-bounces at lists.rabbitmq.com] *On Behalf Of *Gary Russell > *Sent:* Monday, April 28, 2014 12:09 PM > *To:* Discussions about RabbitMQ > *Subject:* Re: [rabbitmq-discuss] dynamic binding > > > > Yes; just add a second binding with the new header value(s); no need to do > anything on the listener - it's all a function of the rabbitmq-server. > > > > On Mon, Apr 28, 2014 at 11:01 AM, PATAR, SAGAR wrote: > > > > We are using header exchanges and are using spring-rabbit mq for > configuration and binding listeners to listen to specific messages with > specific message header values. > > Ex: message header ?srcId? has values ?1001,1002,1003,1004? > > > > At runtime if the same listener bean has to listen to messages with > message header ?srcId? having value ?2001,2002,2003? .. > > Is there a way to add these bindings dynamically at runtime ..?? > > > > Thanks > > Sagar > > > _______________________________________________ > rabbitmq-discuss mailing list > rabbitmq-discuss at lists.rabbitmq.com > https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss > > > > _______________________________________________ > rabbitmq-discuss mailing list > rabbitmq-discuss at lists.rabbitmq.com > https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From videlalvaro at gmail.com Tue Apr 29 21:50:38 2014 From: videlalvaro at gmail.com (Alvaro Videla) Date: Tue, 29 Apr 2014 22:50:38 +0200 Subject: [rabbitmq-discuss] RabbitMQ Monitoring Queue Activites in Spring Project In-Reply-To: <1DB4F5481AB296448E8A5B5BC9D336541CB03982@xmb-aln-x10.cisco.com> References: <1DB4F5481AB296448E8A5B5BC9D336541CB02E37@xmb-aln-x10.cisco.com> <1DB4F5481AB296448E8A5B5BC9D336541CB030D4@xmb-aln-x10.cisco.com> <1DB4F5481AB296448E8A5B5BC9D336541CB030F4@xmb-aln-x10.cisco.com> <1DB4F5481AB296448E8A5B5BC9D336541CB03982@xmb-aln-x10.cisco.com> Message-ID: Did you read the tutorials? Specifically tutorial one and the part about basicConsume? https://www.rabbitmq.com/tutorials/tutorial-one-java.html Regards, Alvaro On Tue, Apr 29, 2014 at 9:02 PM, Srinath Sridharan -X (srinatsr - ZENSAR TECHNOLOGIES INC at Cisco) wrote: > I have binded my queue to amq.rabbitmq.log. > > How to consume some events from there? > > Regards ?? > Srinath > > > -----Original Message----- > From: Michael Klishin [mailto:mklishin at gopivotal.com] > Sent: Monday, April 28, 2014 11:56 AM > To: Discussions about RabbitMQ; Srinath Sridharan -X (srinatsr - ZENSAR TECHNOLOGIES INC at Cisco) > Subject: RE: [rabbitmq-discuss] RabbitMQ Monitoring Queue Activites in Spring Project > > On April 28, 2014 at 10:51:53 PM, Srinath Sridharan -X (srinatsr - ZENSAR TECHNOLOGIES INC at Cisco) (srinatsr at cisco.com) wrote: >> > I need to log RabbitMQ activity using a java logger but >> > asynchronously > > You can bind a queue to amq.rabbitmq.log (a topic exchange in the default vhost ) and consume some events that happen in the broker, then log them using any logger you want. > > See also http://www.rabbitmq.com/firehose.html. There is no other way to get a stream of broker activity in a JVM app. > -- > MK > > Software Engineer, Pivotal/RabbitMQ > _______________________________________________ > rabbitmq-discuss mailing list > rabbitmq-discuss at lists.rabbitmq.com > https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss From videlalvaro at gmail.com Tue Apr 29 21:51:20 2014 From: videlalvaro at gmail.com (Alvaro Videla) Date: Tue, 29 Apr 2014 22:51:20 +0200 Subject: [rabbitmq-discuss] Binding amq.rabbitmq.trace In-Reply-To: <1DB4F5481AB296448E8A5B5BC9D336541CB03A08@xmb-aln-x10.cisco.com> References: <1DB4F5481AB296448E8A5B5BC9D336541CB03A08@xmb-aln-x10.cisco.com> Message-ID: This tutorial will show you how to use a topic exchange: https://www.rabbitmq.com/tutorials/tutorial-five-java.html On Tue, Apr 29, 2014 at 9:37 PM, Srinath Sridharan -X (srinatsr - ZENSAR TECHNOLOGIES INC at Cisco) wrote: > While binding my queue with rabbitmq.trace. What routing key should I use? > > > > I need published and delivered receipts > > Regards ?? > > Srinath > > > > > _______________________________________________ > rabbitmq-discuss mailing list > rabbitmq-discuss at lists.rabbitmq.com > https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss > From pmaisenovich at blizzard.com Tue Apr 29 22:12:34 2014 From: pmaisenovich at blizzard.com (Pavel) Date: Tue, 29 Apr 2014 14:12:34 -0700 (PDT) Subject: [rabbitmq-discuss] Possible memory leak in the management plugin In-Reply-To: <5347F359.7010504@rabbitmq.com> References: <1397020502914-34663.post@n5.nabble.com> <534521F1.9030707@rabbitmq.com> <1397058685615-34697.post@n5.nabble.com> <53457674.1040704@rabbitmq.com> <1397106237672-34725.post@n5.nabble.com> <53466D39.4050906@rabbitmq.com> <1397157592459-34779.post@n5.nabble.com> <5346F7AC.2090703@rabbitmq.com> <1397171854071-34790.post@n5.nabble.com> <5347F359.7010504@rabbitmq.com> Message-ID: <1398805954002-35223.post@n5.nabble.com> Thanks, Simon! I've seen your update to mgmt plugin GC in 3.3.1 (26110) and going to re-run my tests against new release when we'll be ready to upgrade to it. So far we've been running for several weeks with disabled fine stats without any issue. -- View this message in context: http://rabbitmq.1065348.n5.nabble.com/Possible-memory-leak-in-the-management-plugin-tp27414p35223.html Sent from the RabbitMQ mailing list archive at Nabble.com. From greg.poirier at opower.com Wed Apr 30 04:46:30 2014 From: greg.poirier at opower.com (Greg Poirier) Date: Tue, 29 Apr 2014 20:46:30 -0700 Subject: [rabbitmq-discuss] statistics_db_node not_running Message-ID: Is there a way to correct this state? I'm using RabbitMQ 3.2.3 with a 3-node cluster on centos. Our nodes were restarted poorly by a wayward sysadmin, and one of the nodes failed to come back up. It then came back up and was the stats node for a while, but when it was restarted just now the stats db did not come back up. My cluster is not partitioned. -------------- next part -------------- An HTML attachment was scrubbed... URL: From jpd750 at gmail.com Tue Apr 29 23:57:35 2014 From: jpd750 at gmail.com (JD) Date: Tue, 29 Apr 2014 15:57:35 -0700 (PDT) Subject: [rabbitmq-discuss] Failure to Connect RabbitMQ/Celery across two EC2 instances Message-ID: <5023f150-c7c2-4fec-a2a7-eafc94b56518@googlegroups.com> > > I keep getting:`*ERROR/MainProcess] consumer: Cannot connect to > amqp://ec2celeryuser*` when I run `celery -A tasks worker` on terminal. > Basically what I'm trying to do is get celery/rabbitmq working properly across (2) ec2 instances. To pass a silly task in `tasks.py` for processing to rabbitmq. * 1. Instance 1* - Houses rabbitMQ This currently runs RabbitMQ fine. If I run `sudo rabbitmqctl status` it outputs: Status of node 'rabbit at ip-xx-xxx-xxx-xx' ... > [{pid,786}, > *2. Instance 2* - Houses Celery I'm trying to run celery on `instance 2` against Instance 1 using the following in terminal: celery -A tasks worker > I have a file `*celeryconfig.py*`: BROKER_URL = > 'amqp://ec2celeryuser:mypasshere at xx.xxx.xx.xx:5672/celeryserver1/' > > #CELERY SETTINGS > CELERY_IMPORTS = ("tasks",) > > CELERY_RESULT_BACKEND = "amqp" > I have a file `*client.py*`: from tasks import add > > result = add.delay(4, 4) # call task > result_sum = result.get(timeout=5) # wait to get result for a maximum > of 5 seconds > I have a file `*tasks.py*`: from celery import Celery > > app = Celery('tasks', > broker='amqp://ec2celeryuser:mypasshere at xx.xxx.xx.xx:5672/celeryserver1/') > > @app.task > def add(x, y): > return x + y > I've properly setup a `vhost`, a user `ec2celeryuser`, and gave this user permissions of: sudo rabbitmqctl set_permissions -p /celeryserver1 ec2celeryuser ".*" > ".*" ".*" > if I do: `sudo rabbitmqctl list_users` on RabbitMQ (instance 1) it shows: > ec2celeryuser [] > guest [administrator > I've tried *both usernames* with their passwords, but no change. What am I doing wrong here? Clearly there is a connection issue, but what am I doing wrong? *I've been following these without much luck.* http://celery.readthedocs.org/en/latest/getting-started/first-steps-with-celery.html http://www.sammyliu.com/2011/06/23/celery-simple-distributed-tasks-with-celery-and-rabbitmq Thank you! -------------- next part -------------- An HTML attachment was scrubbed... URL: From dor.ben-dov at tikalk.com Wed Apr 30 05:55:51 2014 From: dor.ben-dov at tikalk.com (Dor Ben Dov) Date: Tue, 29 Apr 2014 21:55:51 -0700 (PDT) Subject: [rabbitmq-discuss] Rabbit MQ 3.3.1 and SSL Message-ID: Hi All, I Have two rabbits that communicate with the shovel. (one side has the shovel configured with the destination of the other side) On each side i have one queue and one exchange, while queue is defined as direct. Simple implementation. Now i want to use ssl between the rabbits. -> this is one. Can you assist ? Dor -------------- next part -------------- An HTML attachment was scrubbed... URL: From simon at rabbitmq.com Wed Apr 30 09:35:29 2014 From: simon at rabbitmq.com (Simon MacMullen) Date: Wed, 30 Apr 2014 09:35:29 +0100 Subject: [rabbitmq-discuss] Rabbit MQ 3.3.1 and SSL In-Reply-To: References: Message-ID: <5360B5D1.4080008@rabbitmq.com> On 30/04/2014 05:55, Dor Ben Dov wrote: > I Have two rabbits that communicate with the shovel. (one side has the > shovel configured with the destination of the other side) > On each side i have one queue and one exchange, while queue is defined > as direct. > Simple implementation. > > Now i want to use ssl between the rabbits. -> this is one. Read http://www.rabbitmq.com/ssl.html for details on setting up SSL, and http://www.rabbitmq.com/uri-query-parameters.html for details on making SSL client connections with the shovel. Cheers, Simon From simon at rabbitmq.com Wed Apr 30 12:27:25 2014 From: simon at rabbitmq.com (Simon MacMullen) Date: Wed, 30 Apr 2014 12:27:25 +0100 Subject: [rabbitmq-discuss] Possible memory leak in the management plugin In-Reply-To: <1398805954002-35223.post@n5.nabble.com> References: <1397020502914-34663.post@n5.nabble.com> <534521F1.9030707@rabbitmq.com> <1397058685615-34697.post@n5.nabble.com> <53457674.1040704@rabbitmq.com> <1397106237672-34725.post@n5.nabble.com> <53466D39.4050906@rabbitmq.com> <1397157592459-34779.post@n5.nabble.com> <5346F7AC.2090703@rabbitmq.com> <1397171854071-34790.post@n5.nabble.com> <5347F359.7010504@rabbitmq.com> <1398805954002-35223.post@n5.nabble.com> Message-ID: <5360DE1D.6080009@rabbitmq.com> On 29/04/14 22:12, Pavel wrote: > Thanks, Simon! I've seen your update to mgmt plugin GC in 3.3.1 (26110) and > going to re-run my tests against new release when we'll be ready to upgrade > to it. Cool. Of course, this will only mitigate the problem; mgmt will still be storing a lot of data in your case, it should just GC it better. In the future (3.4.0?) there will probably be a mode to avoid storing much of this data at all, which should help further. Cheers, Simon -- Simon MacMullen RabbitMQ, Pivotal From simon at rabbitmq.com Wed Apr 30 12:30:13 2014 From: simon at rabbitmq.com (Simon MacMullen) Date: Wed, 30 Apr 2014 12:30:13 +0100 Subject: [rabbitmq-discuss] vhost in error reports In-Reply-To: <1398783624043-35201.post@n5.nabble.com> References: <1398783624043-35201.post@n5.nabble.com> Message-ID: <5360DEC5.5020709@rabbitmq.com> On 29/04/14 16:00, carlhoerberg wrote: > Would help us a lot if the vhost always was included (when possible) when an > error occurs, like for this: > > =ERROR REPORT==== 19-Apr-2014::00:57:46 === > connection <0.11988.251>, channel 2 - soft error: > {amqp_error,access_refused, > "queue name 'amq.gen-MekSW94u_f7itDHHZMdtNA' contains reserved > prefix 'amq.*'", > 'queue.declare'} Good point. Bug filed. Might as well add the username too. Cheers, Simon -- Simon MacMullen RabbitMQ, Pivotal From simon at rabbitmq.com Wed Apr 30 12:40:20 2014 From: simon at rabbitmq.com (Simon MacMullen) Date: Wed, 30 Apr 2014 12:40:20 +0100 Subject: [rabbitmq-discuss] statistics_db_node not_running In-Reply-To: References: Message-ID: <5360E124.1030107@rabbitmq.com> On 30/04/14 04:46, Greg Poirier wrote: > Is there a way to correct this state? I'm using RabbitMQ 3.2.3 with a > 3-node cluster on centos. Our nodes were restarted poorly by a wayward > sysadmin, and one of the nodes failed to come back up. It then came back > up and was the stats node for a while, but when it was restarted just > now the stats db did not come back up. There are some fixes for this in 3.2.4, but if you can't upgrade, then you might be able to prod the database back into life with: rabbitmqctl eval \ 'application:stop(rabbitmq_management),application:start(rabbitmq_management).' People have reported varying levels of success with that command. Cheers, Simon -- Simon MacMullen RabbitMQ, Pivotal From simon at rabbitmq.com Wed Apr 30 13:14:33 2014 From: simon at rabbitmq.com (Simon MacMullen) Date: Wed, 30 Apr 2014 13:14:33 +0100 Subject: [rabbitmq-discuss] RabbitMQ 3.3.0 doesn't working with OpenLDAP ???? In-Reply-To: <000301cf641e$97f86a70$c7e93f50$@quant-edge.com> References: <535F66FE.2070202@rabbitmq.com> <000301cf641e$97f86a70$c7e93f50$@quant-edge.com> Message-ID: <5360E929.6070803@rabbitmq.com> Please keep rabbitmq-discuss CCed. On 30/04/14 03:47, Ngoc Tang (Quant Edge) wrote: > Dear Simon > > I have enable the Ldap Plugin, But I can not login with OpenLdap, this is > the contents of log file > ------ --- -- -------- ------------- > Server startup complete; 10 plugins started. > * amqp_client > * eldap > * mochiweb > * rabbitmq_auth_backend_ldap > * rabbitmq_auth_mechanism_ssl > * rabbitmq_federation_management > * rabbitmq_management > * rabbitmq_management_agent > * rabbitmq_web_dispatch > * webmachine > > =INFO REPORT==== 30-Apr-2014::09:31:56 === > LDAP CHECK: login for test.rabbit > =INFO REPORT==== 30-Apr-2014::09:31:56 === > LDAP DECISION: login for test.rabbit: {error,invalidDNSyntax} Since you have configured neither user_dn_pattern nor dn_lookup_attribute, the LDAP plugin treats the user name as provided over HTTP / AMQP as the DN for LDAP. And "test.rabbit" is not a valid DN. > And this is the contents of config file > > -------------- ------------------ > [ > {rabbit, [ > {tcp_listeners, [{"10.x.x.x",5672}]}, > {ssl_listeners, [{"10.x.x.x",5671}]}, > {ssl_options, [{cacertfile,"/etc/rabbitmq/ssl/nvca/cacert.pem"}, > {certfile,"/etc/rabbitmq/ssl/server/cert.pem"}, > {keyfile,"/etc/rabbitmq/ssl/server/key.pem"}, > {verify,verify_none}, > {fail_if_no_peer_cert,false}]}, > {heartbeat, 15}, > {vm_memory_high_watermark_paging_ratio, 0.75}, > {vm_memory_high_watermark, 0.4}, > {auth_backends, [{rabbit_auth_backend_ldap, > rabbit_auth_backend_internal}, > rabbit_auth_backend_internal]}, > {disk_free_limit, 40000000000} > ]}, > {rabbitmq_auth_backend_ldap, > [ {servers, ["openldap.com"]}, > %% {dn_lookup_attribute, > {"uid=${username},ou=allusers,dc=openldap,dc=com"}}, > {dn_lookup_base, {"dc=openldap,dc=com"}}, > {other_bind, ["cn=manager,cn=internal,dc=openldap,dc=com"," > Ad09DSJwidjdwf89D"]}, > {use_ssl, false}, > {port, 389}, > {log, network}, > {vhost_access_query, {exists, "cn=${username},ou=allusers, > dc=openldap,dc=com"}}, > {resource_access_query, > {for, [{resource, exchange, {for, [{permission, configure, > {in_group, "cn=${username}, > dc=openldap,dc=com "} > }, > {permission, write, {constant, true}}, > {permission, read, {constant, true}} > ]}}, > {resource, queue, {constant, true}}]}}, > {tag_queries, [{administrator, {constant, true}}, > {management, {constant, true}}]} > ] > } > ]. You haven't said what you are trying to do. The LDAP plugin can be configured to convert usernames into DNs and subsequently bind in a number of different ways - how are you expecting it to work? Cheers, Simon > And this is the Information of OpenLdap > > LDAP Base DN dc=openldap,dc=com > LDAP Bind DN cn=manager,cn=internal,dc=openldap,dc=com > LDAP Bind Password Ad09DSJwidjdwf89D > ---- > Help me Please. > Thanks & Regards. > Ngoc Tang. > -- Simon MacMullen RabbitMQ, Pivotal From loganathan.ms at gmail.com Wed Apr 30 13:32:46 2014 From: loganathan.ms at gmail.com (Loganathan Sellapa) Date: Wed, 30 Apr 2014 18:02:46 +0530 Subject: [rabbitmq-discuss] Frequent disconnection of rabbitmq Message-ID: Hi All, I am getting frequent ssl connection error while connecting to RabbitMq via RubyAmqp gem, sometimes getting connection issue after publishing the message. Copied below the RabbitMq log, I am using the self signed SSL certificates is it causing this issue? =ERROR REPORT==== 30-Apr-2014::17:42:04 === error on AMQP connection <0.29428.17>: {ssl_upgrade_error,timeout} (unknown POSIX error) =ERROR REPORT==== 30-Apr-2014::17:42:05 === error on AMQP connection <0.29434.17>: {ssl_upgrade_error,timeout} (unknown POSIX error) =ERROR REPORT==== 30-Apr-2014::17:42:05 === error on AMQP connection <0.29440.17>: {ssl_upgrade_error,timeout} (unknown POSIX error) =ERROR REPORT==== 30-Apr-2014::17:42:05 === error on AMQP connection <0.29444.17>: {ssl_upgrade_error,timeout} (unknown POSIX error) =ERROR REPORT==== 30-Apr-2014::17:42:05 === error on AMQP connection <0.29449.17>: {ssl_upgrade_error,timeout} (unknown POSIX error) =ERROR REPORT==== 30-Apr-2014::12:06:11 === error on AMQP connection <0.5206.3>: {ssl_upgrade_error,closed} (unknown POSIX error) regards, Loganathan Mob: +91 7760780741 | +91 9944414388 Skype: loganathan.sellappa ViewMe -------------- next part -------------- An HTML attachment was scrubbed... URL: From michael at rabbitmq.com Wed Apr 30 14:32:41 2014 From: michael at rabbitmq.com (Michael Klishin) Date: Wed, 30 Apr 2014 17:32:41 +0400 Subject: [rabbitmq-discuss] Frequent disconnection of rabbitmq In-Reply-To: References: Message-ID: <2DD08AA8-6D98-4C6C-9320-237C0245342B@rabbitmq.com> Are you sure you connect on the SSL port (5671 by default)? MK > On 30/04/2014, at 16:32, Loganathan Sellapa wrote: > > Hi All, > > I am getting frequent ssl connection error while connecting to RabbitMq via RubyAmqp gem, sometimes getting connection issue after publishing the message. Copied below the RabbitMq log, I am using the self signed SSL certificates is it causing this issue? > > =ERROR REPORT==== 30-Apr-2014::17:42:04 === > error on AMQP connection <0.29428.17>: {ssl_upgrade_error,timeout} (unknown POSIX error) > > =ERROR REPORT==== 30-Apr-2014::17:42:05 === > error on AMQP connection <0.29434.17>: {ssl_upgrade_error,timeout} (unknown POSIX error) > > =ERROR REPORT==== 30-Apr-2014::17:42:05 === > error on AMQP connection <0.29440.17>: {ssl_upgrade_error,timeout} (unknown POSIX error) > > =ERROR REPORT==== 30-Apr-2014::17:42:05 === > error on AMQP connection <0.29444.17>: {ssl_upgrade_error,timeout} (unknown POSIX error) > > =ERROR REPORT==== 30-Apr-2014::17:42:05 === > error on AMQP connection <0.29449.17>: {ssl_upgrade_error,timeout} (unknown POSIX error) > > > =ERROR REPORT==== 30-Apr-2014::12:06:11 === > error on AMQP connection <0.5206.3>: {ssl_upgrade_error,closed} (unknown POSIX error) > > > > > regards, > Loganathan > Mob: +91 7760780741 | +91 9944414388 > Skype: loganathan.sellappa > ViewMe > > _______________________________________________ > rabbitmq-discuss mailing list > rabbitmq-discuss at lists.rabbitmq.com > https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: From carl.hoerberg at gmail.com Wed Apr 30 14:50:29 2014 From: carl.hoerberg at gmail.com (carlhoerberg) Date: Wed, 30 Apr 2014 06:50:29 -0700 (PDT) Subject: [rabbitmq-discuss] vhost in error reports In-Reply-To: <5360DEC5.5020709@rabbitmq.com> References: <1398783624043-35201.post@n5.nabble.com> <5360DEC5.5020709@rabbitmq.com> Message-ID: Yes, awesome, thanks! :) On Wednesday 30 April 2014 at 19:31, Simon MacMullen-2 [via RabbitMQ] wrote: > On 29/04/14 16:00, carlhoerberg wrote: > > Would help us a lot if the vhost always was included (when possible) when an > > error occurs, like for this: > > > > =ERROR REPORT==== 19-Apr-2014::00:57:46 === > > connection <0.11988.251>, channel 2 - soft error: > > {amqp_error,access_refused, > > "queue name 'amq.gen-MekSW94u_f7itDHHZMdtNA' contains reserved > > prefix 'amq.*'", > > 'queue.declare'} > > Good point. Bug filed. Might as well add the username too. > > Cheers, Simon > > -- > Simon MacMullen > RabbitMQ, Pivotal > _______________________________________________ > rabbitmq-discuss mailing list > [hidden email] (/user/SendEmail.jtp?type=node&node=35231&i=0) > https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss > > > If you reply to this email, your message will be added to the discussion below: http://rabbitmq.1065348.n5.nabble.com/vhost-in-error-reports-tp35201p35231.html > To unsubscribe from vhost in error reports, click here (http://rabbitmq.1065348.n5.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=35201&code=Y2FybC5ob2VyYmVyZ0BnbWFpbC5jb218MzUyMDF8LTEyNDcxMDc4NjM=). > NAML (http://rabbitmq.1065348.n5.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml) -- View this message in context: http://rabbitmq.1065348.n5.nabble.com/vhost-in-error-reports-tp35201p35236.html Sent from the RabbitMQ mailing list archive at Nabble.com. -------------- next part -------------- An HTML attachment was scrubbed... URL: From sp345s at att.com Wed Apr 30 14:56:07 2014 From: sp345s at att.com (PATAR, SAGAR) Date: Wed, 30 Apr 2014 13:56:07 +0000 Subject: [rabbitmq-discuss] dynamic binding In-Reply-To: References: <59BF56992750434F89F8EBE141E64B1D01876F97@GAALPA1MSGUSR9C.ITServices.sbc.com> <59BF56992750434F89F8EBE141E64B1D01877432@GAALPA1MSGUSR9C.ITServices.sbc.com> Message-ID: <59BF56992750434F89F8EBE141E64B1D0187751B@GAALPA1MSGUSR9C.ITServices.sbc.com> Thanks.. It helped.. Just wondering what is the maximum number of binding Rabbit MQ supports and also does that depend on the resources like ram, disk space, processes rabbit mq is running From: rabbitmq-discuss [mailto:rabbitmq-discuss-bounces at lists.rabbitmq.com] On Behalf Of Gary Russell Sent: Tuesday, April 29, 2014 4:50 PM To: Discussions about RabbitMQ Subject: Re: [rabbitmq-discuss] dynamic binding It looks like you are trying to modify an existing binding; I said... >add a second binding with the new header value(s)... Just use admin.declareBinding(newBinding), ...ending up with 2 bindings on the exchange, with the initial binding having the 1xxx matches and the new binding having the 2xxx matches. On Tue, Apr 29, 2014 at 4:36 PM, PATAR, SAGAR > wrote: I am using the below code and it is creating the binding object BUT is NOT reflecting on the admin console.. Attaching the src and context file for reference .. Binding b = BindingBuilder.bind(q).to(e).with(binding.getRoutingKey()).and(bArgs); Below is the configuration context file ? From: rabbitmq-discuss [mailto:rabbitmq-discuss-bounces at lists.rabbitmq.com] On Behalf Of Gary Russell Sent: Monday, April 28, 2014 12:09 PM To: Discussions about RabbitMQ Subject: Re: [rabbitmq-discuss] dynamic binding Yes; just add a second binding with the new header value(s); no need to do anything on the listener - it's all a function of the rabbitmq-server. On Mon, Apr 28, 2014 at 11:01 AM, PATAR, SAGAR > wrote: We are using header exchanges and are using spring-rabbit mq for configuration and binding listeners to listen to specific messages with specific message header values. Ex: message header ?srcId? has values ?1001,1002,1003,1004? At runtime if the same listener bean has to listen to messages with message header ?srcId? having value ?2001,2002,2003? .. Is there a way to add these bindings dynamically at runtime ..?? Thanks Sagar _______________________________________________ rabbitmq-discuss mailing list rabbitmq-discuss at lists.rabbitmq.com https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss _______________________________________________ rabbitmq-discuss mailing list rabbitmq-discuss at lists.rabbitmq.com https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: From loganathan.ms at gmail.com Wed Apr 30 14:56:13 2014 From: loganathan.ms at gmail.com (Loganathan Sellapa) Date: Wed, 30 Apr 2014 19:26:13 +0530 Subject: [rabbitmq-discuss] Frequent disconnection of rabbitmq In-Reply-To: References: Message-ID: Hi Michael, Yes Micheal, We are using the port 5671, and connecting over SSL. Do you know why we are getting this error. Also on another RMQ, when we communicate we see the same behaviour of connection getting disconnected. However the error there was ssl_upgrade_error,closed =ERROR REPORT==== 30-Apr-2014::12:01:29 === error on AMQP connection <0.4491.3>: {ssl_upgrade_error,closed} (unknown POSIX error) regards, Loganathan Mob: +91 7760780741 | +91 9944414388 Skype: loganathan.sellappa ViewMe On Wed, Apr 30, 2014 at 6:02 PM, Loganathan Sellapa wrote: > Hi All, > > I am getting frequent ssl connection error while connecting to RabbitMq > via RubyAmqp gem, sometimes getting connection issue after publishing the > message. Copied below the RabbitMq log, I am using the self signed SSL > certificates is it causing this issue? > > =ERROR REPORT==== 30-Apr-2014::17:42:04 === > error on AMQP connection <0.29428.17>: {ssl_upgrade_error,timeout} > (unknown POSIX error) > > =ERROR REPORT==== 30-Apr-2014::17:42:05 === > error on AMQP connection <0.29434.17>: {ssl_upgrade_error,timeout} > (unknown POSIX error) > > =ERROR REPORT==== 30-Apr-2014::17:42:05 === > error on AMQP connection <0.29440.17>: {ssl_upgrade_error,timeout} > (unknown POSIX error) > > =ERROR REPORT==== 30-Apr-2014::17:42:05 === > error on AMQP connection <0.29444.17>: {ssl_upgrade_error,timeout} > (unknown POSIX error) > > =ERROR REPORT==== 30-Apr-2014::17:42:05 === > error on AMQP connection <0.29449.17>: {ssl_upgrade_error,timeout} > (unknown POSIX error) > > > =ERROR REPORT==== 30-Apr-2014::12:06:11 === > error on AMQP connection <0.5206.3>: {ssl_upgrade_error,closed} (unknown > POSIX error) > > > > > regards, > Loganathan > Mob: +91 7760780741 | +91 9944414388 > Skype: loganathan.sellappa > ViewMe > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mrtndimitrov at gmail.com Wed Apr 30 15:34:40 2014 From: mrtndimitrov at gmail.com (Martin Koroudjiev) Date: Wed, 30 Apr 2014 17:34:40 +0300 Subject: [rabbitmq-discuss] Declaring queues and exchanges on both sides Message-ID: <53610A00.5060307@gmail.com> Hello, I am new to RabbitMQ and sorry in advance if the question has trivial answer. I read the tutorials and something really bothers me: Why do we have to declare the queues and the exchanges both on the server and on the client? Is it expensive to do so? Best regards, Martin From videlalvaro at gmail.com Wed Apr 30 15:51:03 2014 From: videlalvaro at gmail.com (Alvaro Videla) Date: Wed, 30 Apr 2014 16:51:03 +0200 Subject: [rabbitmq-discuss] Declaring queues and exchanges on both sides In-Reply-To: <53610A00.5060307@gmail.com> References: <53610A00.5060307@gmail.com> Message-ID: Hi, Is not expensive to do so. Also it will be done just once, when the script/app starts. There's actually no consensus about who should setup the whole "messaging fabric", either the consumer, the producer, or both. For example the producer might be working in some kind of Pub/Sub scenario, where consumers come and go, then it's probably just required for the producer to declare the exchange, and then let the consumers declare their own queues and do the bindings. On the other hand, if you have a typical "worker" scenario, then the producer might want to make sure there's a queue where the messages are being routed to, say a "resize images" queue. Then the producer will declare the queue and the exchange, and will also perform the bindings. So basically, "it depends". Regards, Alvaro On Wed, Apr 30, 2014 at 4:34 PM, Martin Koroudjiev wrote: > Hello, > > I am new to RabbitMQ and sorry in advance if the question has trivial > answer. I read the tutorials and something really bothers me: > > Why do we have to declare the queues and the exchanges both on the > server and on the client? Is it expensive to do so? > > Best regards, > Martin > _______________________________________________ > rabbitmq-discuss mailing list > rabbitmq-discuss at lists.rabbitmq.com > https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss From marks at nationalfibre.net Wed Apr 30 16:14:02 2014 From: marks at nationalfibre.net (Mark Steele) Date: Wed, 30 Apr 2014 11:14:02 -0400 Subject: [rabbitmq-discuss] Mirrored HA queues disappeared on one out of two nodes restarting Message-ID: Hi all, When restarting a node in a cluster that contained mirrored queues, I just experienced a mirrored queue disappearing completely from the cluster. Both nodes in the cluster were both ram and disc nodes. This is extremely worrisome to say the least. -- =INFO REPORT==== 28-Apr-2014::14:17:07 === Synchronising queue 'affiliate_clicks' in vhost '/': 5565 messages to synchronise =INFO REPORT==== 28-Apr-2014::14:17:07 === Synchronising queue 'affiliate_clicks' in vhost '/': all slaves already synced =INFO REPORT==== 29-Apr-2014::22:19:06 === Mirrored-queue (queue 'affiliate_clicks' in vhost '/'): Slave saw deaths of mirrors =INFO REPORT==== 29-Apr-2014::22:19:06 === Mirrored-queue (queue 'affiliate_clicks' in vhost '/'): Promoting slave to master =INFO REPORT==== 29-Apr-2014::22:19:33 === rabbit on node rabbit at mq03 up =INFO REPORT==== 29-Apr-2014::22:19:33 === Synchronising queue 'affiliate_clicks' in vhost '/': complete =INFO REPORT==== 29-Apr-2014::22:19:33 === Synchronising queue 'affiliate_clicks' in vhost '/': 4696 messages to synchronise =INFO REPORT==== 29-Apr-2014::22:19:33 === Synchronising queue 'affiliate_clicks' in vhost '/': all slaves already synced lots of connection logs, then kaboom =INFO REPORT==== 29-Apr-2014::22:23:48 === Mirrored-queue (queue 'affiliate_clicks' in vhost '/'): Master saw deaths of mirrors =ERROR REPORT==== 29-Apr-2014::22:23:50 === ** Generic server <0.274.0> terminating ** Last message in was emit_stats ** When Server state == {q, {amqqueue, {resource,<<"/">>,queue,<<"affiliate_clicks">>}, true,false,none,[],<0.274.0>,[],[], [{vhost,<<"/">>}, {name,<<"affiliate_queues">>}, {pattern,<<"^affiliate_.*$">>}, {definition, [{<<"ha-mode">>,<<"all">>}, {<<"ha-sync-mode">>,<<"automatic">>}]}, {priority,0}], [{<2827.281.0>,<2827.280.0>}]}, none,false,rabbit_mirror_queue_master, {state, {resource,<<"/">>,queue,<<"affiliate_clicks">>}, <0.275.0>,<0.19739.588>,rabbit_variable_queue, {vqstate, {0,{[],[]}}, {0,{[],[]}}, {delta,undefined,0,undefined}, {0,{[],[]}}, {2660, {[{msg_status,2363798, <<117,194,172,33,185,58,225,43,141,116,31,73, 152,23,146,23>>, {basic_message, {resource,<<"/">>,exchange, <<"affiliate_clicks">>}, [<<"#">>], {content,60, {'P_basic',<<"application/json">>,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined}, <<128,0,16,97,112,112,108,105,99,97,116,105, 111,110,47,106,115,111,110>>, rabbit_framing_amqp_0_9_1, [<<"DATA SNIPPED OUT">>]}, <<205,79,109,87,12,83,109,226,230,122,218,63, 27,68,138,67>>, false}, false,false,false,false, 2363799, {0,nil}, {0,nil}, {qistate, "/var/lib/rabbitmq/mnesia/rabbit at mq04/queues/D8CDHLZOTXCZL6MJMMYRK9EAN", {{dict,0,16,16,8,80,48, {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[], []}, {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[], []}}}, []}, undefined,0,65536, #Fun, {0,nil}}, {{client_msstate,msg_store_persistent, <<69,37,230,131,60,26,47,62,12,194,26,130,4,129, 159,57>>, {dict,0,16,16,8,80,48, {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[], []}, {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[], []}}}, {state,356427, "/var/lib/rabbitmq/mnesia/rabbit at mq04/msg_store_persistent"}, rabbit_msg_store_ets_index, "/var/lib/rabbitmq/mnesia/rabbit at mq04/msg_store_persistent", <0.265.0>,360524,352330,364621,368718}, {client_msstate,msg_store_transient, <<140,110,236,52,188,182,217,136,180,245,92,51, 176,116,195,10>>, {dict,0,16,16,8,80,48, {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[], []}, {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[], []}}}, {state,335942, "/var/lib/rabbitmq/mnesia/rabbit at mq04/msg_store_transient"}, rabbit_msg_store_ets_index, "/var/lib/rabbitmq/mnesia/rabbit at mq04/msg_store_transient", <0.260.0>,340039,331840,344136,348233}}, true,0,2660,0,infinity,2660,2660,0,0,0, {rates, {{1398,824624,347232},0}, {{1398,824624,347232},84}, 0.0,17.611352475686193, {1398,824629,389132}}, {0,nil}, {0,nil}, {0,nil}, {0,nil}, 0,0, {rates, {{1398,824624,347232},6706}, {{1398,824624,347232},0}, 663.4928634941101,0.0, {1398,824629,389132}}}, {dict,0,16,16,8,80,48, {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]}, {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[], []}}}, [], {set,0,16,16,8,80,48, {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]}, {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[], []}}}}, {[],[]}, undefined,undefined,undefined,undefined, {state,fine,5000,#Ref<0.0.527.127396>}, {0,nil}, undefined,undefined,undefined, {dict,1,16,16,8,80,48, {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]}, {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[], [[<0.18782.588>|#Ref<0.0.524.252716>]]}}}, undefined,undefined,undefined,running} ** Reason for termination == ** {{badmatch,{error,not_found}}, [{rabbit_mirror_queue_master,stop_all_slaves,2}, {rabbit_mirror_queue_master,delete_and_terminate,2}, {rabbit_amqqueue_process,'-terminate_delete/3-fun-1-',6}, {rabbit_amqqueue_process,terminate_shutdown,2}, {gen_server2,terminate,3}, {proc_lib,wake_up,3}]} ** In 'terminate' callback with reason == ** {{badmatch,{error,not_found}}, [{rabbit_amqqueue_process,i,2}, {rabbit_amqqueue_process,'-infos/2-lc$^0/1-0-',2}, {rabbit_amqqueue_process,'-infos/2-lc$^0/1-0-',2}, {rabbit_amqqueue_process,emit_stats,2}, {rabbit_amqqueue_process,handle_info,2}, {gen_server2,handle_msg,2}, {proc_lib,wake_up,3}]} Here's the error in the SASL log: =SUPERVISOR REPORT==== 29-Apr-2014::22:23:55 === Supervisor: {local, rabbit_mirror_queue_slave_sup} Context: child_terminated Reason: {{badmatch,{error,not_found}}, [{rabbit_mirror_queue_master,stop_all_slaves,2}, {rabbit_mirror_queue_master,delete_and_terminate,2}, {rabbit_amqqueue_process,'-terminate_delete/3-fun-1-',6}, {rabbit_amqqueue_process,terminate_shutdown,2}, {gen_server2,terminate,3}, {proc_lib,wake_up,3}]} Offender: [{pid,<0.274.0>}, {name,rabbit_mirror_queue_slave}, {mfa, {rabbit_mirror_queue_slave,start_link, [{amqqueue, {resource,<<"/">>,queue,<<"affiliate_clicks">>}, true,false,none,[],<2827.280.0>,[],[], [{vhost,<<"/">>}, {name,<<"affiliate_queues">>}, {pattern,<<"^affiliate_.*$">>}, {definition, [{<<"ha-mode">>,<<"all">>}, {<<"ha-sync-mode">>,<<"automatic">>}]}, {priority,0}], [{<2827.281.0>,<2827.280.0>}]}]}}, {restart_type,temporary}, {shutdown,4294967295}, {child_type,worker}] Known issue? Need to update? Please advise. Cheers, Mark Steele, CISSP, CSM, GCIA, GPEN Director of development Instaclick Inc. marks at nationalfibre.net m: (416) 844-9221 -------------- next part -------------- An HTML attachment was scrubbed... URL: From simon at rabbitmq.com Wed Apr 30 16:24:07 2014 From: simon at rabbitmq.com (Simon MacMullen) Date: Wed, 30 Apr 2014 16:24:07 +0100 Subject: [rabbitmq-discuss] Mirrored HA queues disappeared on one out of two nodes restarting In-Reply-To: References: Message-ID: <53611597.6060100@rabbitmq.com> On 30/04/14 16:14, Mark Steele wrote: > Known issue? Need to update? Please advise. There was a known issue, fixed in 3.2.1, where a crashing slave could cause the master (and other slaves of the same queue) to crash with stack traces like the ones you posted. So that's definitely a reason to upgrade. Of course, that doesn't help us with why the first slave crashed. It's quite possibly another bug that has been fixed since, but just to be sure, could you look for and post any errors from the same time frame mentioning "affiliate_clicks" on mq03? Cheers, Simon -- Simon MacMullen RabbitMQ, Pivotal From greg.poirier at opower.com Wed Apr 30 16:31:53 2014 From: greg.poirier at opower.com (Greg Poirier) Date: Wed, 30 Apr 2014 08:31:53 -0700 Subject: [rabbitmq-discuss] statistics_db_node not_running In-Reply-To: <5360E124.1030107@rabbitmq.com> References: <5360E124.1030107@rabbitmq.com> Message-ID: Worked like a charm. I'm working on upgrading to 3.3.1 fairly soon. On Wed, Apr 30, 2014 at 4:40 AM, Simon MacMullen wrote: > On 30/04/14 04:46, Greg Poirier wrote: > >> Is there a way to correct this state? I'm using RabbitMQ 3.2.3 with a >> 3-node cluster on centos. Our nodes were restarted poorly by a wayward >> sysadmin, and one of the nodes failed to come back up. It then came back >> up and was the stats node for a while, but when it was restarted just >> now the stats db did not come back up. >> > > There are some fixes for this in 3.2.4, but if you can't upgrade, then you > might be able to prod the database back into life with: > > rabbitmqctl eval \ > 'application:stop(rabbitmq_management),application:start( > rabbitmq_management).' > > People have reported varying levels of success with that command. > > Cheers, Simon > > -- > Simon MacMullen > RabbitMQ, Pivotal > -------------- next part -------------- An HTML attachment was scrubbed... URL: From marks at nationalfibre.net Wed Apr 30 16:34:34 2014 From: marks at nationalfibre.net (Mark Steele) Date: Wed, 30 Apr 2014 11:34:34 -0400 Subject: [rabbitmq-discuss] Mirrored HA queues disappeared on one out of two nodes restarting In-Reply-To: <53611597.6060100@rabbitmq.com> References: <53611597.6060100@rabbitmq.com> Message-ID: <56058353-7C00-4AA6-85E7-38F17355E7D1@nationalfibre.net> MQ03 was the master for that queue, and it was cleanly restarted (service rabbitmq-server restart). So the chain of events was mq03 restarted, mq04 became master for that queue, mq03 rejoined after restart, then queue disappeared a few minutes later. The sasl log on 03: =CRASH REPORT==== 29-Apr-2014::22:24:15 === crasher: initial call: gen:init_it/6 pid: <0.277.0> registered_name: [] exception exit: {function_clause, [{gb_trees,delete_1,[4227,nil]}, {gb_trees,delete,2}, {rabbit_variable_queue,remove_pending_ack,2}, {rabbit_variable_queue,'-ack/2-fun-0-',2}, {lists,foldl,3}, {rabbit_variable_queue,ack,2}, {rabbit_mirror_queue_slave,process_instruction,2}, {rabbit_mirror_queue_slave,handle_cast,2}]} in function gen_server2:terminate/3 ancestors: [rabbit_mirror_queue_slave_sup,rabbit_sup,<0.168.0>] messages: [{'$gen_cast', {gm, {publish,<2828.25884.590>, {message_properties,undefined,false}, {basic_message, {resource,<<"/">>,exchange,<<"affiliate_clicks">>}, [<<"#">>], {content,60, {'P_basic',<<"application/json">>,undefined,undefined, undefined,undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined,undefined, undefined}, <<128,0,16,97,112,112,108,105,99,97,116,105,111,110,47, 106,115,111,110>>, rabbit_framing_amqp_0_9_1, [<<"snipped out">>]}, <<117,194,172,33,185,58,225,43,141,116,31,73,152,23,146, 23>>, false}}}}, {'$gen_cast', {deliver, {delivery,false,<2828.25884.590>, {basic_message, {resource,<<"/">>,exchange,<<"affiliate_clicks">>}, [<<"#">>], {content,60, {'P_basic',<<"application/json">>,undefined,undefined, undefined,undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined,undefined, undefined}, <<128,0,16,97,112,112,108,105,99,97,116,105,111,110,47, 106,115,111,110>>, rabbit_framing_amqp_0_9_1, [<<"snipped out}">>]}, <<117,194,172,33,185,58,225,43,141,116,31,73,152,23,146, 23>>, false}, undefined}, true,flow}}, {'$gen_cast',{gm,{sender_death,<2828.25884.590>}}}, {'$gen_cast', {run_backing_queue,rabbit_mirror_queue_master, #Fun}}, {'$gen_cast', {run_backing_queue,rabbit_mirror_queue_master, #Fun}}, {'EXIT',<0.278.0>,normal}, {'$gen_cast', {run_backing_queue,rabbit_mirror_queue_master, #Fun}}, {'$gen_cast', {run_backing_queue,rabbit_mirror_queue_master, #Fun}}] links: [<0.273.0>] dictionary: [{credit_blocked,[]}, {{xtype_to_module,direct},rabbit_exchange_type_direct}, {{xtype_to_module,topic},rabbit_exchange_type_topic}, {guid,{{4125214297,1894440844,1353716068,2733218191},1}}] trap_exit: true status: running heap_size: 317811 stack_size: 24 reductions: 4292143 neighbours: =SUPERVISOR REPORT==== 29-Apr-2014::22:24:15 === Supervisor: {local, rabbit_mirror_queue_slave_sup} Context: child_terminated Reason: {function_clause, [{gb_trees,delete_1,[4227,nil]}, {gb_trees,delete,2}, {rabbit_variable_queue,remove_pending_ack,2}, {rabbit_variable_queue,'-ack/2-fun-0-',2}, {lists,foldl,3}, {rabbit_variable_queue,ack,2}, {rabbit_mirror_queue_slave,process_instruction,2}, {rabbit_mirror_queue_slave,handle_cast,2}]} Offender: [{pid,<0.277.0>}, {name,rabbit_mirror_queue_slave}, {mfa, {rabbit_mirror_queue_slave,start_link, [{amqqueue, {resource,<<"/">>,queue,<<"affiliate_clicks">>}, true,false,none,[],<2828.274.0>,[],[], [{vhost,<<"/">>}, {name,<<"affiliate_queues">>}, {pattern,<<"^affiliate_.*$">>}, {definition, [{<<"ha-mode">>,<<"all">>}, {<<"ha-sync-mode">>,<<"automatic">>}]}, {priority,0}], [{<2828.275.0>,<2828.274.0>}]}]}}, {restart_type,temporary}, {shutdown,4294967295}, {child_type,worker}] Mark Steele, CISSP, CSM, GCIA, GPEN Director of development Instaclick Inc. marks at nationalfibre.net m: (416) 844-9221 On Apr 30, 2014, at 11:24 AM, Simon MacMullen wrote: > On 30/04/14 16:14, Mark Steele wrote: >> Known issue? Need to update? Please advise. > > There was a known issue, fixed in 3.2.1, where a crashing slave could cause the master (and other slaves of the same queue) to crash with stack traces like the ones you posted. So that's definitely a reason to upgrade. > > Of course, that doesn't help us with why the first slave crashed. It's quite possibly another bug that has been fixed since, but just to be sure, could you look for and post any errors from the same time frame mentioning "affiliate_clicks" on mq03? > > Cheers, Simon > > -- > Simon MacMullen > RabbitMQ, Pivotal -------------- next part -------------- An HTML attachment was scrubbed... URL: From mpietrek at skytap.com Wed Apr 30 18:49:14 2014 From: mpietrek at skytap.com (Matt Pietrek) Date: Wed, 30 Apr 2014 10:49:14 -0700 Subject: [rabbitmq-discuss] Best way to live migrate RabbitMQ consumer load from one cluster to another Message-ID: Apologies if this has been answered somewhere already, but my searching isn't turning up anything. I'm looking for advice on how to best move client traffic from one RabbitMQ cluster to another without service interruption. I suspect it can be done with HA-proxy, but I can't figure out the steps. I'm open to other ideas as well. Scenario: We have a 2-node cluster with all queues mirrored. Let's say it's running RabbitMQ 3.2.2. Clients connect to this cluster through a VIP maintained by keepalived on both RabbitMQ hosts. We now want to move to a different identical cluster, but running 3.3.1. I'm thinking I can bring up this new cluster "side-by-side" with the old cluster, such that they're running concurrently. All new connections would go to the new cluster, while already established connections to the 3.2.2 cluster will remain alive and working, until all clients have eventually connected to the new cluster. At that point the 3.2.2 cluster can be shut down. I have a hunch I can put HA-proxy in front of both clusters, and though live configuration changes have it migrate the traffic to the new cluster while leaving existing cluster connections alone. What I don't know is the commands/configuration options/steps to implement it. Thoughts and guidance on how to best do this? Thanks, Matt -------------- next part -------------- An HTML attachment was scrubbed... URL: From goyalk at vmware.com Wed Apr 30 21:23:35 2014 From: goyalk at vmware.com (Kapil Goyal) Date: Wed, 30 Apr 2014 13:23:35 -0700 (PDT) Subject: [rabbitmq-discuss] RabbitMQ backward compatibility policy Message-ID: Does RabbitMQ have a policy about when it can break backward compatibility with older clients, for example a major version update? Is this documented somewhere? Thanks Kapil -------------- next part -------------- An HTML attachment was scrubbed... URL: From srikanthtns at gmail.com Wed Apr 30 21:48:45 2014 From: srikanthtns at gmail.com (srikanth tns) Date: Wed, 30 Apr 2014 13:48:45 -0700 Subject: [rabbitmq-discuss] Rabbitmq cluster design and HA In-Reply-To: References: Message-ID: Thanks Micheal , here is the settings we are doing on rabbitmq cluster {rabbit, [{vm_memory_high_watermark_paging_ratio, 0.8},{vm_memory_high_watermark, 0.9},{disk_free_limit, 1000000000}]} Apart from this ,is there any QoS value that we need to set ? On Tue, Apr 29, 2014 at 1:13 AM, Michael Klishin wrote: > > > On 29 April 2014 at 07:22:03, srikanth tns (srikanthtns at gmail.com) wrote: > > > Can you let us know what would be optimal settings on rabbitmq > > cluster to prevent it from crashing and having reliable messaging? > > It's really hard to make general recommendations without knowing what > kind of workload you use. > > > 1) sysctl configurations > > nlimit > 20,000 or so (definitely can't be less than ~ 6K) > > > 2) Disk limitations to setup on cluster . the default is 50MB > > I'd use 500 MB or so if RabbitMQ is the primary thing that runs on the > machine. > > > 3) Memory to used , right now its default 40% > > Can be up to 90%, again, depending on what else may run on the same > machine. > > > > 6) any disk space limitation on the hosts > > Host is this different from 2) above? > > > For HA , we are using ha-mode all to duplicate the queues across > > all nodes in the cluster. Do you think its reliable during the > > failover scenario ?During the failover it would like 100k queues > > on nthe new master? Or should we use ha-mode = exactly/nodes ? > > With 3 nodes ha-mode=all is a reasonable amount of duplication. If you are > confident > you will never lose more than 1 node at a time, use ha-mode=exactly with 2 > nodes. > Less duplication means better throughput. > > As to how well new master election will work with 100K queues, try it. > -- > MK > > Software Engineer, Pivotal/RabbitMQ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From yguenane at gmail.com Wed Apr 30 16:43:14 2014 From: yguenane at gmail.com (Yanis Guenane) Date: Wed, 30 Apr 2014 08:43:14 -0700 (PDT) Subject: [rabbitmq-discuss] Core Dump when /proc is not mounted (chroot) Message-ID: <13002772-cda9-46c9-bd0e-aff6ff37ab30@googlegroups.com> Hi list, Today I tried to install rabbitmq-server on a debian chroot running on fedora and it crashed during package installation. This was due to the fact that my chroot did not have its /proc mounted. After mouting the host /proc to the chroot /proc I could install rabbitmq-server just fine. Before getting to this conclusion it took me a moment, not having any helpful information from the stack trace. I was wondering if this situation could be improved, by checking if /proc is mounted else reporting that it should be mounted ? # # For reproduction purpose # root at fedora:/ # mkdir /tmp/debian && cd /tmp root at fedora:/tmp # debootstrap --no-check-gpg --arch ${ARCH:=amd64} --variant=minbase wheezy debian/ http://http.debian.net/debian root at fedora:/tmp # chroot debian root at debian:/ # apt-get install rabbitmq-server ... Starting message broker: rabbitmq-serverFAILED - check /var/log/rabbitmq/startup_\{log, _err\} ... (warning). failed! ... root at debian:/ # exit root at fedora:/tmp # mount -t proc proc debian/proc root at debian:/ # apt-get install rabbitmq-server ... Starting message broker: rabbitmq-server. root at debian:/ # Hope it helps, -- Yanis Guenane -------------- next part -------------- An HTML attachment was scrubbed... URL: From srinatsr at cisco.com Wed Apr 30 23:39:34 2014 From: srinatsr at cisco.com (Srinath Sridharan -X (srinatsr - ZENSAR TECHNOLOGIES INC at Cisco)) Date: Wed, 30 Apr 2014 22:39:34 +0000 Subject: [rabbitmq-discuss] How to resolve this issue? Message-ID: <1DB4F5481AB296448E8A5B5BC9D336541CB041B5@xmb-aln-x10.cisco.com> Apr 30, 2014 3:38:24 PM org.apache.catalina.core.StandardWrapperValve invoke SEVERE: Servlet.service() for servlet [Q2OServiceEngine Servlet] in context with path [/QOTService-1.0] threw exception [Request processing failed; nested exception is org.springframework.web.client.ResourceAccessException: I/O error on POST request for "http://qot-job-03:15672/api/queues/%252F/validate.monitor/get": The target server failed to respond; nested exception is org.apache.http.NoHttpResponseException: The target server failed to respond] with root cause org.apache.http.NoHttpResponseException: The target server failed to respond at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:95) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:62) at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:254) at org.apache.http.impl.AbstractHttpClientConnection.receiveResponseHeader(AbstractHttpClientConnection.java:289) at org.apache.http.impl.conn.DefaultClientConnection.receiveResponseHeader(DefaultClientConnection.java:252) at org.apache.http.impl.conn.ManagedClientConnectionImpl.receiveResponseHeader(ManagedClientConnectionImpl.java:191) at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:300) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:127) at org.apache.http.impl.client.DefaultRequestDirector.tryExecute(DefaultRequestDirector.java:717) at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:522) at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:906) at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:805) at org.springframework.http.client.HttpComponentsClientHttpRequest.executeInternal(HttpComponentsClientHttpRequest.java:88) at org.springframework.http.client.AbstractBufferingClientHttpRequest.executeInternal(AbstractBufferingClientHttpRequest.java:46) at org.springframework.http.client.AbstractClientHttpRequest.execute(AbstractClientHttpRequest.java:49) at org.springframework.web.client.RestTemplate.doExecute(RestTemplate.java:509) at org.springframework.web.client.RestTemplate.execute(RestTemplate.java:472) at org.springframework.web.client.RestTemplate.exchange(RestTemplate.java:420) at com.cisco.ccrc.service.controller.PushRequestToQueueController.getMessagesFromQueue(PushRequestToQueueController.java:94) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.springframework.web.method.support.InvocableHandlerMethod.invoke(InvocableHandlerMethod.java:219) at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:132) at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:104) at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandleMethod(RequestMappingHandlerAdapter.java:745) at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:686) at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:80) at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:925) at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:856) at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:953) at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:855) at javax.servlet.http.HttpServlet.service(HttpServlet.java:647) at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:829) at javax.servlet.http.HttpServlet.service(HttpServlet.java:728) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:305) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:222) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:123) at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:502) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:171) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:99) at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:953) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:408) at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1023) at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:589) at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:310) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:724) Regards ...* Srinath -------------- next part -------------- An HTML attachment was scrubbed... URL: From michael at rabbitmq.com Wed Apr 30 23:42:21 2014 From: michael at rabbitmq.com (Michael Klishin) Date: Thu, 1 May 2014 02:42:21 +0400 Subject: [rabbitmq-discuss] RabbitMQ backward compatibility policy In-Reply-To: References: Message-ID: <935B4F66-3FB4-451F-AE66-E1AA2CA93BF5@rabbitmq.com> As far as the server features go, I don't think we've had anything that broke amqp 0-9-1 compatibility in quite a while. In general, versions roughly follow semantic versioning. HTH. MK > On 01/05/2014, at 00:23, Kapil Goyal wrote: > > Does RabbitMQ have a policy about when it can break backward compatibility with older clients, for example a major version update? Is this documented somewhere? From michael at rabbitmq.com Wed Apr 30 23:45:40 2014 From: michael at rabbitmq.com (Michael Klishin) Date: Thu, 1 May 2014 02:45:40 +0400 Subject: [rabbitmq-discuss] How to resolve this issue? In-Reply-To: <1DB4F5481AB296448E8A5B5BC9D336541CB041B5@xmb-aln-x10.cisco.com> References: <1DB4F5481AB296448E8A5B5BC9D336541CB041B5@xmb-aln-x10.cisco.com> Message-ID: <405CBD98-FB34-48C0-A3A8-F99064296439@rabbitmq.com> This means the server is not reachable (over TCP). Check if the IP is correct, possible firewall settings, if management plugin is enabled and binds to the IP address and port you expect, etc. Try connecting via telnet first. MK > On 01/05/2014, at 02:39, "Srinath Sridharan -X (srinatsr - ZENSAR TECHNOLOGIES INC at Cisco)" wrote: > > Apr 30, 2014 3:38:24 PM org.apache.catalina.core.StandardWrapperValve invoke > SEVERE: Servlet.service() for servlet [Q2OServiceEngine Servlet] in context with path [/QOTService-1.0] threw exception [Request processing failed; nested exception is org.springframework.web.client.ResourceAccessException: I/O error on POST request for "http://qot-job-03:15672/api/queues/%252F/validate.monitor/get": The target server failed to respond; nested exception is org.apache.http.NoHttpResponseException: The target server failed to respond] with root cause > org.apache.http.NoHttpResponseException: The target server failed to respond > at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:95) > at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:62) > at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:254) > at org.apache.http.impl.AbstractHttpClientConnection.receiveResponseHeader(AbstractHttpClientConnection.java:289) > at org.apache.http.impl.conn.DefaultClientConnection.receiveResponseHeader(DefaultClientConnection.java:252) > at org.apache.http.impl.conn.ManagedClientConnectionImpl.receiveResponseHeader(ManagedClientConnectionImpl.java:191) > at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:300) > at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:127) > at org.apache.http.impl.client.DefaultRequestDirector.tryExecute(DefaultRequestDirector.java:717) > at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:522) > at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:906) > at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:805) > at org.springframework.http.client.HttpComponentsClientHttpRequest.executeInternal(HttpComponentsClientHttpRequest.java:88) > at org.springframework.http.client.AbstractBufferingClientHttpRequest.executeInternal(AbstractBufferingClientHttpRequest.java:46) > at org.springframework.http.client.AbstractClientHttpRequest.execute(AbstractClientHttpRequest.java:49) > at org.springframework.web.client.RestTemplate.doExecute(RestTemplate.java:509) > at org.springframework.web.client.RestTemplate.execute(RestTemplate.java:472) > at org.springframework.web.client.RestTemplate.exchange(RestTemplate.java:420) > at com.cisco.ccrc.service.controller.PushRequestToQueueController.getMessagesFromQueue(PushRequestToQueueController.java:94) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.springframework.web.method.support.InvocableHandlerMethod.invoke(InvocableHandlerMethod.java:219) > at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:132) > at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:104) > at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandleMethod(RequestMappingHandlerAdapter.java:745) > at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:686) > at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:80) > at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:925) > at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:856) > at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:953) > at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:855) > at javax.servlet.http.HttpServlet.service(HttpServlet.java:647) > at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:829) > at javax.servlet.http.HttpServlet.service(HttpServlet.java:728) > at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:305) > at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210) > at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:222) > at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:123) > at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:502) > at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:171) > at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:99) > at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:953) > at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118) > at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:408) > at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1023) > at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:589) > at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:310) > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:724) > > > > Regards ?? > Srinath > > _______________________________________________ > rabbitmq-discuss mailing list > rabbitmq-discuss at lists.rabbitmq.com > https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: From Jeffery.Jochum at echostar.com Wed Apr 30 22:02:51 2014 From: Jeffery.Jochum at echostar.com (Jochum, Jeffrey) Date: Wed, 30 Apr 2014 21:02:51 +0000 Subject: [rabbitmq-discuss] Redelivery of unacked messages. Message-ID: <85BDDAE422FE734992DD233B5E1873C0766263D4@INWPIEXC02.SATS.CORP> Hello, I apologize if this is a pretty basic question, I couldn't find an answer to it. I have a queue that I am consuming many messages off of but I am not acking them immediately. I wait a period and then ack them all at once. However I notice I seem to be getting these messages again ever couple of minutes. I don't believe the publisher is republishing these messages, so I was wondering if the broker sends these out again, of if the messages go back to a recovery state at some point. Thanks, Jeff Jochum -------------- next part -------------- An HTML attachment was scrubbed... URL: From brad at debtpaypro.com Wed Apr 30 23:04:56 2014 From: brad at debtpaypro.com (brad at debtpaypro.com) Date: Wed, 30 Apr 2014 15:04:56 -0700 (PDT) Subject: [rabbitmq-discuss] New RabbitMQ 3.3.0 Web_stomp SSL problems In-Reply-To: References: Message-ID: <1e6f76f9-f85e-4f73-8b42-588e504a3c37@googlegroups.com> I'm having a similar, or possibly the same, problem. My config is basically the same as yours. I'm running version 3.3.1. When starting up, a message is logged that says "rabbit_web_stomp: listening for HTTPS connections on 0.0.0.0:15675", but netstat shows that nothing is listening on port 15675 and connections to my server on port 15675 fail. There aren't any other interesting log messages in any of the rabbit log files. I have tried changing the port several time but the same message is logged and nothing listens on the new port. I know the certificates are good since we're using the on our live sites. If I change the config to point to files that do not exist for the certificates, the same thing happens. rabbitmq.config: [ {rabbit, [ {auth_backends, [rabbit_auth_backend_internal]}, {log_levels, [ {connection, info}, {mirroring, info} ]}, {heartbeat, 10}, {collect_statistics_interval, 1000}, {delegate_count, 32}, {cluster_partition_handling, pause_minority} ]}, {rabbitmq_management, [ {sample_retention_policies, [ {global, [{3600, 5}, {86400, 60}, {604800, 600}]}, {basic, [{60, 5}, {3600, 60}]}, {detailed, [{30, 1}]} ]}, {http_log_dir, "/tmp/rabbit-mgmt"} ]}, {kernel, [ {net_ticktime, 5} ]}, {rabbitmq_web_stomp, [ {ssl_config, [ {port, 15674}, {backlog, 1024}, {certfile, "/etc/pki/tls/certs/dpp.crt"}, {keyfile, "/etc/pki/tls/private/dpp.key"} ]} ]} ]. status: {running_applications, [{rabbitmq_web_stomp,"Rabbit WEB-STOMP - WebSockets to Stomp adapter", "3.3.1"}, {ssl,"Erlang/OTP SSL application","4.1.6"}, {public_key,"Public key infrastructure","0.13"}, {crypto,"CRYPTO version 2","2.0.4"}, {rabbitmq_stomp,"Embedded Rabbit Stomp Adapter","3.3.1"}, {rabbitmq_management_visualiser,"RabbitMQ Visualiser","3.3.1"}, {rabbitmq_management,"RabbitMQ Management Console","3.3.1"}, {rabbitmq_management_agent,"RabbitMQ Management Agent","3.3.1"}, {rabbitmq_amqp1_0,"AMQP 1.0 support for RabbitMQ","3.3.1"}, {rabbit,"RabbitMQ","3.3.1"}, {os_mon,"CPO CXC 138 46","2.2.7"}, {rabbitmq_web_dispatch,"RabbitMQ Web Dispatcher","3.3.1"}, {webmachine,"webmachine","1.10.3-rmq3.3.1-gite9359c7"}, {mochiweb,"MochiMedia Web Server","2.7.0-rmq3.3.1-git680dba8"}, {xmerl,"XML parser","1.2.10"}, {cowboy,"Small, fast, modular HTTP server.","0.5.0-rmq3.3.1-git4b93c2d"}, {sockjs,"SockJS","0.3.4-rmq3.3.1-git3132eb9"}, {inets,"INETS CXC 138 49","5.7.1"}, {mnesia,"MNESIA CXC 138 12","4.5"}, {amqp_client,"RabbitMQ AMQP Client","3.3.1"}, {sasl,"SASL CXC 138 11","2.1.10"}, {stdlib,"ERTS CXC 138 10","1.17.5"}, {kernel,"ERTS CXC 138 10","2.14.5"}]}, {os,{unix,linux}}, {erlang_version, "Erlang R14B04 (erts-5.8.5) [source] [64-bit] [smp:24:24] [rq:24] [async-threads:30] [kernel-poll:true]\n"}, On Friday, April 25, 2014 4:34:22 PM UTC-5, Sebastien Dubois wrote: > > > Hi all, > > Up to recently we've been using RabbitMQ 3.2.3/Erlang R15B01 with the > web_Stomp plugin over SSL. Since this was not supported, we had to rebuilt > the web_stomp plugin using a patched branch as described at > https://gist.github.com/berico-rclayton/5475365 (also see > https://github.com/rabbitmq/rabbitmq-web-stomp/pull/3). This has been > working fine for months. > > We recently realized that the fix for SSL support in web_stomp was > systemized in RabbitMQ 3.3.0/Erlang R16B03. So we installed it and updated > our configuration. However, using the exact same setup and self-signed > keys/certificates, we cannot get the new version to work properly while > using SSL over stomp. Connecting to https://:15678/stomp > just fails, although from the log web_stomp seems to be listening correctly > on port 15678. We tried to regenerate a new set of keys, but it did not do > anything. > > Our old rabbitMQ config (patched 3.2.3 web_stomp) was: > > {rabbitmq_web_stomp, [ > > {ssl_enabled, true}, > > {https_port, 15678}, > > {ssl_key_file, "/usr/local/ssl/private/server.key"}, > > {ssl_key_password, "password"}, > > {ssl_ca_certificate_file, "/usr/local/ssl/crt/public.crt"}, > > {ssl_certificate_file, "/usr/local/ssl/crt/public.crt"} > > ] }, > > > And the corresponding new config on RabbitMQ 3.3.0 is > > {rabbitmq_web_stomp, > > [{ssl_config, [{port, 15678}, > > {backlog, 1024}, > > {certfile, "/usr/local/ssl/crt/public.crt"}, > > {keyfile, "/usr/local/ssl/private/server.key"}, > > {cacertfile, "/usr/local/ssl/crt/public.crt"}, > > {password, "password"} > > ]} > > ]}, > > Is our config ok? does anybody experienced similar problems or have any > idea what we could be doing wrong? > > > Thanks in advance, > > /Sebas > -------------- next part -------------- An HTML attachment was scrubbed... URL: From brad at debtpaypro.com Wed Apr 30 23:04:56 2014 From: brad at debtpaypro.com (brad at debtpaypro.com) Date: Wed, 30 Apr 2014 15:04:56 -0700 (PDT) Subject: [rabbitmq-discuss] New RabbitMQ 3.3.0 Web_stomp SSL problems In-Reply-To: References: Message-ID: <1e6f76f9-f85e-4f73-8b42-588e504a3c37@googlegroups.com> I'm having a similar, or possibly the same, problem. My config is basically the same as yours. I'm running version 3.3.1. When starting up, a message is logged that says "rabbit_web_stomp: listening for HTTPS connections on 0.0.0.0:15675", but netstat shows that nothing is listening on port 15675 and connections to my server on port 15675 fail. There aren't any other interesting log messages in any of the rabbit log files. I have tried changing the port several time but the same message is logged and nothing listens on the new port. I know the certificates are good since we're using the on our live sites. If I change the config to point to files that do not exist for the certificates, the same thing happens. rabbitmq.config: [ {rabbit, [ {auth_backends, [rabbit_auth_backend_internal]}, {log_levels, [ {connection, info}, {mirroring, info} ]}, {heartbeat, 10}, {collect_statistics_interval, 1000}, {delegate_count, 32}, {cluster_partition_handling, pause_minority} ]}, {rabbitmq_management, [ {sample_retention_policies, [ {global, [{3600, 5}, {86400, 60}, {604800, 600}]}, {basic, [{60, 5}, {3600, 60}]}, {detailed, [{30, 1}]} ]}, {http_log_dir, "/tmp/rabbit-mgmt"} ]}, {kernel, [ {net_ticktime, 5} ]}, {rabbitmq_web_stomp, [ {ssl_config, [ {port, 15674}, {backlog, 1024}, {certfile, "/etc/pki/tls/certs/dpp.crt"}, {keyfile, "/etc/pki/tls/private/dpp.key"} ]} ]} ]. status: {running_applications, [{rabbitmq_web_stomp,"Rabbit WEB-STOMP - WebSockets to Stomp adapter", "3.3.1"}, {ssl,"Erlang/OTP SSL application","4.1.6"}, {public_key,"Public key infrastructure","0.13"}, {crypto,"CRYPTO version 2","2.0.4"}, {rabbitmq_stomp,"Embedded Rabbit Stomp Adapter","3.3.1"}, {rabbitmq_management_visualiser,"RabbitMQ Visualiser","3.3.1"}, {rabbitmq_management,"RabbitMQ Management Console","3.3.1"}, {rabbitmq_management_agent,"RabbitMQ Management Agent","3.3.1"}, {rabbitmq_amqp1_0,"AMQP 1.0 support for RabbitMQ","3.3.1"}, {rabbit,"RabbitMQ","3.3.1"}, {os_mon,"CPO CXC 138 46","2.2.7"}, {rabbitmq_web_dispatch,"RabbitMQ Web Dispatcher","3.3.1"}, {webmachine,"webmachine","1.10.3-rmq3.3.1-gite9359c7"}, {mochiweb,"MochiMedia Web Server","2.7.0-rmq3.3.1-git680dba8"}, {xmerl,"XML parser","1.2.10"}, {cowboy,"Small, fast, modular HTTP server.","0.5.0-rmq3.3.1-git4b93c2d"}, {sockjs,"SockJS","0.3.4-rmq3.3.1-git3132eb9"}, {inets,"INETS CXC 138 49","5.7.1"}, {mnesia,"MNESIA CXC 138 12","4.5"}, {amqp_client,"RabbitMQ AMQP Client","3.3.1"}, {sasl,"SASL CXC 138 11","2.1.10"}, {stdlib,"ERTS CXC 138 10","1.17.5"}, {kernel,"ERTS CXC 138 10","2.14.5"}]}, {os,{unix,linux}}, {erlang_version, "Erlang R14B04 (erts-5.8.5) [source] [64-bit] [smp:24:24] [rq:24] [async-threads:30] [kernel-poll:true]\n"}, On Friday, April 25, 2014 4:34:22 PM UTC-5, Sebastien Dubois wrote: > > > Hi all, > > Up to recently we've been using RabbitMQ 3.2.3/Erlang R15B01 with the > web_Stomp plugin over SSL. Since this was not supported, we had to rebuilt > the web_stomp plugin using a patched branch as described at > https://gist.github.com/berico-rclayton/5475365 (also see > https://github.com/rabbitmq/rabbitmq-web-stomp/pull/3). This has been > working fine for months. > > We recently realized that the fix for SSL support in web_stomp was > systemized in RabbitMQ 3.3.0/Erlang R16B03. So we installed it and updated > our configuration. However, using the exact same setup and self-signed > keys/certificates, we cannot get the new version to work properly while > using SSL over stomp. Connecting to https://:15678/stomp > just fails, although from the log web_stomp seems to be listening correctly > on port 15678. We tried to regenerate a new set of keys, but it did not do > anything. > > Our old rabbitMQ config (patched 3.2.3 web_stomp) was: > > {rabbitmq_web_stomp, [ > > {ssl_enabled, true}, > > {https_port, 15678}, > > {ssl_key_file, "/usr/local/ssl/private/server.key"}, > > {ssl_key_password, "password"}, > > {ssl_ca_certificate_file, "/usr/local/ssl/crt/public.crt"}, > > {ssl_certificate_file, "/usr/local/ssl/crt/public.crt"} > > ] }, > > > And the corresponding new config on RabbitMQ 3.3.0 is > > {rabbitmq_web_stomp, > > [{ssl_config, [{port, 15678}, > > {backlog, 1024}, > > {certfile, "/usr/local/ssl/crt/public.crt"}, > > {keyfile, "/usr/local/ssl/private/server.key"}, > > {cacertfile, "/usr/local/ssl/crt/public.crt"}, > > {password, "password"} > > ]} > > ]}, > > Is our config ok? does anybody experienced similar problems or have any > idea what we could be doing wrong? > > > Thanks in advance, > > /Sebas > -------------- next part -------------- An HTML attachment was scrubbed... URL: From brad at debtpaypro.com Wed Apr 30 23:06:39 2014 From: brad at debtpaypro.com (brad at debtpaypro.com) Date: Wed, 30 Apr 2014 15:06:39 -0700 (PDT) Subject: [rabbitmq-discuss] New RabbitMQ 3.3.0 Web_stomp SSL problems In-Reply-To: <1e6f76f9-f85e-4f73-8b42-588e504a3c37@googlegroups.com> References: <1e6f76f9-f85e-4f73-8b42-588e504a3c37@googlegroups.com> Message-ID: The config I posted was from when I was testing some things and is not correct. The ssl port should be 15675. On Wednesday, April 30, 2014 5:04:56 PM UTC-5, br... at debtpaypro.com wrote: > > I'm having a similar, or possibly the same, problem. My config is > basically the same as yours. I'm running version 3.3.1. When starting up, > a message is logged that says "rabbit_web_stomp: listening for HTTPS > connections on 0.0.0.0:15675", but netstat shows that nothing is > listening on port 15675 and connections to my server on port 15675 fail. > There aren't any other interesting log messages in any of the rabbit log > files. I have tried changing the port several time but the same message is > logged and nothing listens on the new port. I know the certificates are > good since we're using the on our live sites. If I change the config to > point to files that do not exist for the certificates, the same thing > happens. > > rabbitmq.config: > > [ > {rabbit, [ > {auth_backends, [rabbit_auth_backend_internal]}, > {log_levels, [ > {connection, info}, > {mirroring, info} > ]}, > {heartbeat, 10}, > {collect_statistics_interval, 1000}, > {delegate_count, 32}, > {cluster_partition_handling, pause_minority} > ]}, > {rabbitmq_management, [ > {sample_retention_policies, [ > {global, [{3600, 5}, {86400, 60}, {604800, 600}]}, > {basic, [{60, 5}, {3600, 60}]}, > {detailed, [{30, 1}]} > ]}, > {http_log_dir, "/tmp/rabbit-mgmt"} > ]}, > {kernel, [ > {net_ticktime, 5} > ]}, > {rabbitmq_web_stomp, [ > {ssl_config, [ > {port, 15674}, > {backlog, 1024}, > {certfile, "/etc/pki/tls/certs/dpp.crt"}, > {keyfile, "/etc/pki/tls/private/dpp.key"} > ]} > ]} > ]. > > > status: > > {running_applications, > [{rabbitmq_web_stomp,"Rabbit WEB-STOMP - WebSockets to Stomp adapter", > "3.3.1"}, > {ssl,"Erlang/OTP SSL application","4.1.6"}, > {public_key,"Public key infrastructure","0.13"}, > {crypto,"CRYPTO version 2","2.0.4"}, > {rabbitmq_stomp,"Embedded Rabbit Stomp Adapter","3.3.1"}, > {rabbitmq_management_visualiser,"RabbitMQ Visualiser","3.3.1"}, > {rabbitmq_management,"RabbitMQ Management Console","3.3.1"}, > {rabbitmq_management_agent,"RabbitMQ Management Agent","3.3.1"}, > {rabbitmq_amqp1_0,"AMQP 1.0 support for RabbitMQ","3.3.1"}, > {rabbit,"RabbitMQ","3.3.1"}, > {os_mon,"CPO CXC 138 46","2.2.7"}, > {rabbitmq_web_dispatch,"RabbitMQ Web Dispatcher","3.3.1"}, > {webmachine,"webmachine","1.10.3-rmq3.3.1-gite9359c7"}, > {mochiweb,"MochiMedia Web Server","2.7.0-rmq3.3.1-git680dba8"}, > {xmerl,"XML parser","1.2.10"}, > {cowboy,"Small, fast, modular HTTP > server.","0.5.0-rmq3.3.1-git4b93c2d"}, > {sockjs,"SockJS","0.3.4-rmq3.3.1-git3132eb9"}, > {inets,"INETS CXC 138 49","5.7.1"}, > {mnesia,"MNESIA CXC 138 12","4.5"}, > {amqp_client,"RabbitMQ AMQP Client","3.3.1"}, > {sasl,"SASL CXC 138 11","2.1.10"}, > {stdlib,"ERTS CXC 138 10","1.17.5"}, > {kernel,"ERTS CXC 138 10","2.14.5"}]}, > {os,{unix,linux}}, > {erlang_version, > "Erlang R14B04 (erts-5.8.5) [source] [64-bit] [smp:24:24] [rq:24] > [async-threads:30] [kernel-poll:true]\n"}, > > > On Friday, April 25, 2014 4:34:22 PM UTC-5, Sebastien Dubois wrote: >> >> >> Hi all, >> >> Up to recently we've been using RabbitMQ 3.2.3/Erlang R15B01 with the >> web_Stomp plugin over SSL. Since this was not supported, we had to rebuilt >> the web_stomp plugin using a patched branch as described at >> https://gist.github.com/berico-rclayton/5475365 (also see >> https://github.com/rabbitmq/rabbitmq-web-stomp/pull/3). This has been >> working fine for months. >> >> We recently realized that the fix for SSL support in web_stomp was >> systemized in RabbitMQ 3.3.0/Erlang R16B03. So we installed it and updated >> our configuration. However, using the exact same setup and self-signed >> keys/certificates, we cannot get the new version to work properly while >> using SSL over stomp. Connecting to https://:15678/stomp >> just fails, although from the log web_stomp seems to be listening correctly >> on port 15678. We tried to regenerate a new set of keys, but it did not do >> anything. >> >> Our old rabbitMQ config (patched 3.2.3 web_stomp) was: >> >> {rabbitmq_web_stomp, [ >> >> {ssl_enabled, true}, >> >> {https_port, 15678}, >> >> {ssl_key_file, "/usr/local/ssl/private/server.key"}, >> >> {ssl_key_password, "password"}, >> >> {ssl_ca_certificate_file, "/usr/local/ssl/crt/public.crt"}, >> >> {ssl_certificate_file, "/usr/local/ssl/crt/public.crt"} >> >> ] }, >> >> >> And the corresponding new config on RabbitMQ 3.3.0 is >> >> {rabbitmq_web_stomp, >> >> [{ssl_config, [{port, 15678}, >> >> {backlog, 1024}, >> >> {certfile, "/usr/local/ssl/crt/public.crt"}, >> >> {keyfile, "/usr/local/ssl/private/server.key"}, >> >> {cacertfile, "/usr/local/ssl/crt/public.crt"}, >> >> {password, "password"} >> >> ]} >> >> ]}, >> >> Is our config ok? does anybody experienced similar problems or have any >> idea what we could be doing wrong? >> >> >> Thanks in advance, >> >> /Sebas >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From brad at debtpaypro.com Wed Apr 30 23:06:39 2014 From: brad at debtpaypro.com (brad at debtpaypro.com) Date: Wed, 30 Apr 2014 15:06:39 -0700 (PDT) Subject: [rabbitmq-discuss] New RabbitMQ 3.3.0 Web_stomp SSL problems In-Reply-To: <1e6f76f9-f85e-4f73-8b42-588e504a3c37@googlegroups.com> References: <1e6f76f9-f85e-4f73-8b42-588e504a3c37@googlegroups.com> Message-ID: The config I posted was from when I was testing some things and is not correct. The ssl port should be 15675. On Wednesday, April 30, 2014 5:04:56 PM UTC-5, br... at debtpaypro.com wrote: > > I'm having a similar, or possibly the same, problem. My config is > basically the same as yours. I'm running version 3.3.1. When starting up, > a message is logged that says "rabbit_web_stomp: listening for HTTPS > connections on 0.0.0.0:15675", but netstat shows that nothing is > listening on port 15675 and connections to my server on port 15675 fail. > There aren't any other interesting log messages in any of the rabbit log > files. I have tried changing the port several time but the same message is > logged and nothing listens on the new port. I know the certificates are > good since we're using the on our live sites. If I change the config to > point to files that do not exist for the certificates, the same thing > happens. > > rabbitmq.config: > > [ > {rabbit, [ > {auth_backends, [rabbit_auth_backend_internal]}, > {log_levels, [ > {connection, info}, > {mirroring, info} > ]}, > {heartbeat, 10}, > {collect_statistics_interval, 1000}, > {delegate_count, 32}, > {cluster_partition_handling, pause_minority} > ]}, > {rabbitmq_management, [ > {sample_retention_policies, [ > {global, [{3600, 5}, {86400, 60}, {604800, 600}]}, > {basic, [{60, 5}, {3600, 60}]}, > {detailed, [{30, 1}]} > ]}, > {http_log_dir, "/tmp/rabbit-mgmt"} > ]}, > {kernel, [ > {net_ticktime, 5} > ]}, > {rabbitmq_web_stomp, [ > {ssl_config, [ > {port, 15674}, > {backlog, 1024}, > {certfile, "/etc/pki/tls/certs/dpp.crt"}, > {keyfile, "/etc/pki/tls/private/dpp.key"} > ]} > ]} > ]. > > > status: > > {running_applications, > [{rabbitmq_web_stomp,"Rabbit WEB-STOMP - WebSockets to Stomp adapter", > "3.3.1"}, > {ssl,"Erlang/OTP SSL application","4.1.6"}, > {public_key,"Public key infrastructure","0.13"}, > {crypto,"CRYPTO version 2","2.0.4"}, > {rabbitmq_stomp,"Embedded Rabbit Stomp Adapter","3.3.1"}, > {rabbitmq_management_visualiser,"RabbitMQ Visualiser","3.3.1"}, > {rabbitmq_management,"RabbitMQ Management Console","3.3.1"}, > {rabbitmq_management_agent,"RabbitMQ Management Agent","3.3.1"}, > {rabbitmq_amqp1_0,"AMQP 1.0 support for RabbitMQ","3.3.1"}, > {rabbit,"RabbitMQ","3.3.1"}, > {os_mon,"CPO CXC 138 46","2.2.7"}, > {rabbitmq_web_dispatch,"RabbitMQ Web Dispatcher","3.3.1"}, > {webmachine,"webmachine","1.10.3-rmq3.3.1-gite9359c7"}, > {mochiweb,"MochiMedia Web Server","2.7.0-rmq3.3.1-git680dba8"}, > {xmerl,"XML parser","1.2.10"}, > {cowboy,"Small, fast, modular HTTP > server.","0.5.0-rmq3.3.1-git4b93c2d"}, > {sockjs,"SockJS","0.3.4-rmq3.3.1-git3132eb9"}, > {inets,"INETS CXC 138 49","5.7.1"}, > {mnesia,"MNESIA CXC 138 12","4.5"}, > {amqp_client,"RabbitMQ AMQP Client","3.3.1"}, > {sasl,"SASL CXC 138 11","2.1.10"}, > {stdlib,"ERTS CXC 138 10","1.17.5"}, > {kernel,"ERTS CXC 138 10","2.14.5"}]}, > {os,{unix,linux}}, > {erlang_version, > "Erlang R14B04 (erts-5.8.5) [source] [64-bit] [smp:24:24] [rq:24] > [async-threads:30] [kernel-poll:true]\n"}, > > > On Friday, April 25, 2014 4:34:22 PM UTC-5, Sebastien Dubois wrote: >> >> >> Hi all, >> >> Up to recently we've been using RabbitMQ 3.2.3/Erlang R15B01 with the >> web_Stomp plugin over SSL. Since this was not supported, we had to rebuilt >> the web_stomp plugin using a patched branch as described at >> https://gist.github.com/berico-rclayton/5475365 (also see >> https://github.com/rabbitmq/rabbitmq-web-stomp/pull/3). This has been >> working fine for months. >> >> We recently realized that the fix for SSL support in web_stomp was >> systemized in RabbitMQ 3.3.0/Erlang R16B03. So we installed it and updated >> our configuration. However, using the exact same setup and self-signed >> keys/certificates, we cannot get the new version to work properly while >> using SSL over stomp. Connecting to https://:15678/stomp >> just fails, although from the log web_stomp seems to be listening correctly >> on port 15678. We tried to regenerate a new set of keys, but it did not do >> anything. >> >> Our old rabbitMQ config (patched 3.2.3 web_stomp) was: >> >> {rabbitmq_web_stomp, [ >> >> {ssl_enabled, true}, >> >> {https_port, 15678}, >> >> {ssl_key_file, "/usr/local/ssl/private/server.key"}, >> >> {ssl_key_password, "password"}, >> >> {ssl_ca_certificate_file, "/usr/local/ssl/crt/public.crt"}, >> >> {ssl_certificate_file, "/usr/local/ssl/crt/public.crt"} >> >> ] }, >> >> >> And the corresponding new config on RabbitMQ 3.3.0 is >> >> {rabbitmq_web_stomp, >> >> [{ssl_config, [{port, 15678}, >> >> {backlog, 1024}, >> >> {certfile, "/usr/local/ssl/crt/public.crt"}, >> >> {keyfile, "/usr/local/ssl/private/server.key"}, >> >> {cacertfile, "/usr/local/ssl/crt/public.crt"}, >> >> {password, "password"} >> >> ]} >> >> ]}, >> >> Is our config ok? does anybody experienced similar problems or have any >> idea what we could be doing wrong? >> >> >> Thanks in advance, >> >> /Sebas >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: