<div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;"><br>
<div class="im"><br>
> Trying to check if this is a plugin<br>
> bug with list_queues (list_queues name slave_pids synchronised_slave_pids),<br>
> I can't get the cluster to list them at all, it just sits there (for the<br>
> last 15 minutes).<br>
<br>
</div>Curious. Are any of the nodes hammering CPU or disk? Were any of the<br>
queues particularly long before shutdown?</blockquote></div><br>Have got a similar problem running tests this evening, although this time *without* any shutdowns (3 nodes all running fine since full restart). I've had some queues get fairly long (1/2 million messages or so, 3 of them), due to broken consumers. The management interface stopped updating message counts as CPU use hit the roof. I was testing so decided to delete the queues and start again. I stopped the producers and consumers running, so the queues were in a steady state and CPU came down. Deleting most queues worked fine, apart from the last one. The management interface just hangs (Chrome offers to kill the page), whether I try to delete or purge the queue. As can be seen below, beam processes are hogging two of the cores on the 'master' for this queue:
<div><br></div><div><a href="http://www.evernote.com/shard/s53/sh/1ba2b59b-913f-4a00-bf3a-03067c0a1f29/faa3fc9c1b250b10b5d8491c367aa991">http://www.evernote.com/shard/s53/sh/1ba2b59b-913f-4a00-bf3a-03067c0a1f29/faa3fc9c1b250b10b5d8491c367aa991</a></div>
<div><br></div><div>Beam processes are also hogging one core on the slave nodes.</div><div><br></div><div>The logs show all three nodes are switching between being clear of the watermark and over it, every few minutes.</div>
<div><br></div><div>Watermark is 5.9GB, management interface shows the slaves < 2GB, the master on 4.4GB. (Log snippet at the end of the email).</div><div><br></div><div><br></div><div><div><font color="#000099">=INFO REPORT==== 7-Nov-2011::22:20:01 ===</font></div>
<div><font color="#000099">vm_memory_high_watermark set. Memory used:6970525768 allowed:6296176230</font></div><div><font color="#000099"><br></font></div><div><font color="#000099">=INFO REPORT==== 7-Nov-2011::22:20:01 ===</font></div>
<div><font color="#000099"> alarm_handler: {set,{{vm_memory_high_watermark,clusterrabbit@hermes02},</font></div><div><font color="#000099"> []}}</font></div><div><font color="#000099"><br></font></div>
<div><font color="#000099">=INFO REPORT==== 7-Nov-2011::22:20:03 ===</font></div><div><font color="#000099">vm_memory_high_watermark clear. Memory used:5018976928 allowed:6296176230</font></div><div><font color="#000099"><br>
</font></div><div><font color="#000099">=INFO REPORT==== 7-Nov-2011::22:20:03 ===</font></div><div><font color="#000099"> alarm_handler: {clear,{vm_memory_high_watermark,clusterrabbit@hermes02}}</font></div><div><font color="#000099"><br>
</font></div><div><font color="#000099">=INFO REPORT==== 7-Nov-2011::22:22:00 ===</font></div><div><font color="#000099">vm_memory_high_watermark set. Memory used:6583230408 allowed:6296176230</font></div><div><font color="#000099"><br>
</font></div><div><font color="#000099">=INFO REPORT==== 7-Nov-2011::22:22:00 ===</font></div><div><font color="#000099"> alarm_handler: {set,{{vm_memory_high_watermark,clusterrabbit@hermes02},</font></div><div><font color="#000099"> []}}</font></div>
<div><font color="#000099"><br></font></div><div><font color="#000099">=INFO REPORT==== 7-Nov-2011::22:22:02 ===</font></div><div><font color="#000099">vm_memory_high_watermark clear. Memory used:5025939688 allowed:6296176230</font></div>
<div><font color="#000099"><br></font></div><div><font color="#000099">=INFO REPORT==== 7-Nov-2011::22:22:02 ===</font></div><div><font color="#000099"> alarm_handler: {clear,{vm_memory_high_watermark,clusterrabbit@hermes02}}</font></div>
<div><font color="#000099"><br></font></div><div><font color="#000099">=INFO REPORT==== 7-Nov-2011::22:24:50 ===</font></div><div><font color="#000099">vm_memory_high_watermark set. Memory used:6587145104 allowed:6296176230</font></div>
<div><font color="#000099"><br></font></div><div><font color="#000099">=INFO REPORT==== 7-Nov-2011::22:24:50 ===</font></div><div><font color="#000099"> alarm_handler: {set,{{vm_memory_high_watermark,clusterrabbit@hermes02},</font></div>
<div><font color="#000099"> []}}</font></div><div><font color="#000099"><br></font></div><div><font color="#000099">=INFO REPORT==== 7-Nov-2011::22:24:51 ===</font></div><div><font color="#000099">vm_memory_high_watermark clear. Memory used:5029821304 allowed:6296176230</font></div>
<div><font color="#000099"><br></font></div><div><font color="#000099">=INFO REPORT==== 7-Nov-2011::22:24:51 ===</font></div><div><font color="#000099"> alarm_handler: {clear,{vm_memory_high_watermark,clusterrabbit@hermes02}}</font></div>
</div>