No subject

Cluster status of node 'rabbit at domU-12-31-39-06-72-50' ...
[{nodes,[{disc,['rabbit at domU-12-31-38-07-18-A6','rabbit at ip-10-202-209-83',
                'rabbit at domU-12-31-39-06-72-50']}]},
 {running_nodes,['rabbit at domU-12-31-38-07-18-A6','rabbit at ip-10-202-209-83',
                 'rabbit at domU-12-31-39-06-72-50']}]
...done.

rabbitmqctl list_queues has the same behavior on the other nodes (never
returns).

How big were the queues? We recently fixed some bugs which had
> previously been causing queue recovery to take a _very_ long time so it
> might be one of those that's afflicting you. What is the CPU/disk doing
> of the "stuck" node? If it's spinning then it's probably just taking a
> very long time to recover.
>
> Matthew
>

Maybe 10-20 queues, probably about 15 messages queued at the time.  This
environment is a very *low* volume but very *critical* part of our
application.  I'd be surprised if the production servers saw more than a
couple hundred messages total per day and this was our test environment.

Right now the node is idle and it's been sitting there for four hours:

 01:08:33 up 8 days,  5:33,  1 user,  load average: 0.00, 0.01, 0.05

A bunch of messages were in an allocated but not-acked state as I was
testing some of our server processes at the time and they were crashing
before they could ack the messages.  I originally went to restart the node
to try and get those messages flowing again.

Thanks,
Bryan

--0015176f09c88e29da04ae82ded6
Content-Type: text/html; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

<div class=3D"gmail_quote">On Tue, Oct 4, 2011 at 4:59 PM, Matthew Sackman =
<span dir=3D"ltr">&lt;<a href=3D"mailto:matthew at rabbitmq.com">matthew at rabbi=
tmq.com</a>&gt;</span> wrote:<br><blockquote class=3D"gmail_quote" style=3D=
"margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;">
<div class=3D"im">Ok, well you can Ctl-C the list_queues. On one of the oth=
er nodes, what</div>
does rabbitmqctl cluster_status return?<br></blockquote><div><br></div><div=
>From another node:</div><div><br></div><div>Cluster status of node &#39;ra=
bbit at domU-12-31-39-06-72-50&#39; ...</div><div>[{nodes,[{disc,[&#39;rabbit@=
domU-12-31-38-07-18-A6&#39;,&#39;rabbit at ip-10-202-209-83&#39;,</div>
<div>=A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 &#39;rabbit at domU-12-31-39-06-72-50&#39=
;]}]},</div><div>=A0{running_nodes,[&#39;rabbit at domU-12-31-38-07-18-A6&#39;=
,&#39;rabbit at ip-10-202-209-83&#39;,</div><div>=A0 =A0 =A0 =A0 =A0 =A0 =A0 =
=A0 =A0&#39;rabbit at domU-12-31-39-06-72-50&#39;]}]</div>
<div>...done.</div><div><br></div><div>rabbitmqctl list_queues has the same=
 behavior on the other nodes (never returns).=A0</div><div><br></div><block=
quote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1px #ccc=
 solid;padding-left:1ex;">

How big were the queues? We recently fixed some bugs which had<br>
previously been causing queue recovery to take a _very_ long time so it<br>
might be one of those that&#39;s afflicting you. What is the CPU/disk doing=
<br>
of the &quot;stuck&quot; node? If it&#39;s spinning then it&#39;s probably =
just taking a<br>
very long time to recover.<br>
<br>
Matthew<br></blockquote><div><br></div><div>Maybe 10-20 queues, probably ab=
out 15 messages queued at the time. =A0This environment is a very *low* vol=
ume but very *critical* part of our application. =A0I&#39;d be surprised if=
 the production servers saw more than a couple hundred messages total per d=
ay and this was our test environment.</div>
<div><br></div><div>Right now the node is idle and it&#39;s been sitting th=
ere for four hours:</div><div><br></div><div>=A001:08:33 up 8 days, =A05:33=
, =A01 user, =A0load average: 0.00, 0.01, 0.05</div><div><br></div><div>A b=
unch of messages were in an allocated but not-acked state as I was testing =
some of our server processes at the time and they were crashing before they=
 could ack the messages. =A0I originally went to restart the node to try an=
d get those messages flowing again.</div>
<div><br></div><div>Thanks,</div><div>Bryan</div></div>

--0015176f09c88e29da04ae82ded6--