<html><head><meta http-equiv="Content-Type" content="text/html charset=koi8-r"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; "><br><div><div>11.02.2013, в 12:49, Tim Watson <<a href="mailto:watson.timothy@gmail.com">watson.timothy@gmail.com</a>> написал(а):</div><br class="Apple-interchange-newline"><blockquote type="cite">Hi Vladimir,<br><br>On 11 Feb 2013, at 07:52, Бородин Владимир wrote:<br><br><blockquote type="cite">Hi all.<br><br>I'm testing mirrored queues in v. 3.0.1. If I stop node with 'rabbitmqctl stop_app', clients and other nodes behave normally (clients reconnect to other nodes because of a tcp balancer and other nodes continue to serve the queue). But if I close one node from others with iptables or kill it with Alt+SysRq+b, the cluster stops working for a long period of time.<br></blockquote><br>What exactly stops working? The whole cluster, all queues/exchanges are inaccessible? Or just this particular mirrored queue? <br></blockquote><div><br></div><div>Producers can't put messages in queue, consumers can't take them from queue. I can not even see the result of running 'rabbitmqctl list_queues' or 'rabbitmqctl cluster_status'.</div><br><blockquote type="cite"><br><blockquote type="cite">Is there any kind of a timeout, after which the node is considered to be dead by others?<br></blockquote><br>No, although the os networking stack can take a while to notice peers are gone. Erlang does have a kind of heartbeat mechanism though, which should notice in a fairly timely fashion that another node has gone away. How long does the 'long period of time' last exactly?<br></blockquote><div><br></div><div>Long period means about a minute for seeing the death of a node. I don't think this is normal for heartbeat messages. I'm using RHEL6 with default sysctl timeouts. I can show appropriate kernel parameters if it can help.</div><div><br></div><div>In the master log I can see that in period of a bit more than a minute (~ 70 sec) master saw death of a mirror:</div><div><div>=ERROR REPORT==== 11-Feb-2013::13:07:11 ===</div><div>** Node rabbit@loadtest04g not responding **</div><div>** Removing (timedout) connection **</div><div><br></div><div>=INFO REPORT==== 11-Feb-2013::13:07:11 ===</div><div>rabbit on node rabbit@loadtest04g down</div><div><br></div></div><div><div>=INFO REPORT==== 11-Feb-2013::13:07:19 ===</div><div>Mirrored-queue (queue 'loadtest01g.domain.com.celery.pidbox' in vhost '/'): Master <<a href="mailto:rabbit@loadtest03g.1.10550.4">rabbit@loadtest03g.1.10550.4</a>> saw deaths of mirrors <<a href="mailto:rabbit@loadtest04g.2.28501.0">rabbit@loadtest04g.2.28501.0</a>></div><div><br></div><div><br></div><div>And in the live slave log I see:</div><div><div>=INFO REPORT==== 11-Feb-2013::13:07:03 ===</div><div>rabbit on node rabbit@loadtest04g down</div><div><br></div></div><div><div>=INFO REPORT==== 11-Feb-2013::13:07:19 ===</div><div>Mirrored-queue (queue 'celery' in vhost '/'): Slave <<a href="mailto:rabbit@loadtest05g.1.18027.0">rabbit@loadtest05g.1.18027.0</a>> saw deaths of mirrors <<a href="mailto:rabbit@loadtest04g.2.28499.0">rabbit@loadtest04g.2.28499.0</a>></div></div><div><br></div><div>The clients are actually using queue with name 'celery'.</div><div><br></div><div>So I suppose there are two problems:</div><div>1. the timeout tuned by the kernel parameters of erlang (this is better) or OS (this is a bit worse because it can affect other applications). Is there a way to make it smaller in erlang?</div><div><div>2. if the live nodes saw the death of killed node, why commands like 'rabbitmqctl cluster_status' don't work? I can understand if the queue does not work due to wrong policy for a queue, but it should not affect the whole cluster, should it?</div></div><div><br></div></div><br><blockquote type="cite"><br><blockquote type="cite">It does not depend on which node I kill - the primary or one of the slaves. There are 3 nodes in a cluster, the queue is mirrored by a policy like that '/<span class="Apple-tab-span" style="white-space:pre"> </span>HA<span class="Apple-tab-span" style="white-space:pre"> </span>^(?!amq\\.).*<span class="Apple-tab-span" style="white-space:pre"> </span>{"ha-mode":"all"}<span class="Apple-tab-span" style="white-space:pre"> </span>0'.<br>If I should give extra info for understanding of a problem, tell me, please. Thanks.<br><br>--<br>Vladimir<br><br><br>_______________________________________________<br>rabbitmq-discuss mailing list<br><a href="mailto:rabbitmq-discuss@lists.rabbitmq.com">rabbitmq-discuss@lists.rabbitmq.com</a><br>https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss<br></blockquote><br>_______________________________________________<br>rabbitmq-discuss mailing list<br><a href="mailto:rabbitmq-discuss@lists.rabbitmq.com">rabbitmq-discuss@lists.rabbitmq.com</a><br>https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss<br></blockquote></div><br><div>
<div style="color: rgb(0, 0, 0); font-family: Helvetica; font-size: medium; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-align: -webkit-auto; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; "><div><br class="Apple-interchange-newline">--</div><div>Vladimir</div><div><br></div></div><br class="Apple-interchange-newline" style="color: rgb(0, 0, 0); font-family: Helvetica; font-size: medium; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-align: -webkit-auto; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; "><br class="Apple-interchange-newline">
</div>
<br></body></html>