<br clear="all"><div><div><font face="verdana,sans-serif">Hi Simon,</font></div><div><font face="verdana,sans-serif"><br></font></div><div><font face="verdana,sans-serif">Thanks for taking the time to look into this.</font></div>
<div><font face="verdana,sans-serif"><br></font></div><div><font face="verdana,sans-serif">I had a feeling this was an environmental issue because it only occurred on our QA and PreLive environments, we do not experience the same issue on our LIVE environment (hosted in another service, not Azure).</font></div>
<div><font face="verdana,sans-serif"><br></font></div><div><font face="verdana,sans-serif">I will get in touch with Microsoft and find out what our options are with Azure VMs and establish if there is anything we can do to eliminate network partitions. We do not currently use their Virtual Networks so we may try setting up the two VMs in a virtual Network to increase the stability between the two nodes. Failing that we may look at the Federation / Shovel solutions mentioned in the link.</font></div>
<div><font face="verdana,sans-serif"><br></font></div><div><font face="verdana,sans-serif">Cheers,</font></div><div><font face="verdana,sans-serif"><br></font></div><div><font face="verdana,sans-serif">Ben</font></div><div>
<font face="verdana,sans-serif"><br></font></div><div><font face="verdana,sans-serif"><br></font></div><div><div style="color:rgb(136,136,136);font-family:arial,sans-serif;font-size:13.333333969116211px;background-color:rgb(255,255,255)">
<font face="verdana,sans-serif">Ben West</font></div><div style="color:rgb(136,136,136);font-family:arial,sans-serif;font-size:13.333333969116211px;background-color:rgb(255,255,255)"><font face="verdana,sans-serif">Product Owner</font></div>
<div style="color:rgb(136,136,136);font-family:arial,sans-serif;font-size:13.333333969116211px;background-color:rgb(255,255,255)"><font face="verdana, sans-serif"> </font></div><div style="color:rgb(136,136,136);font-family:arial,sans-serif;font-size:13.333333969116211px;background-color:rgb(255,255,255)">
<font face="verdana,sans-serif">Mobile: +44(0)7824 617 813</font></div><div style="font-family:arial,sans-serif;font-size:13.333333969116211px;background-color:rgb(255,255,255)"><font face="verdana,sans-serif" color="#3366ff"><a href="mailto:ben.west@mobankgroup.com" target="_blank">ben.west@mobankgroup.com</a></font></div>
<div style="font-family:arial,sans-serif;font-size:13.333333969116211px;background-color:rgb(255,255,255)"><font color="#3366ff"><br></font></div><div style="font-family:arial,sans-serif;font-size:13.333333969116211px;background-color:rgb(255,255,255)">
<font face="verdana, sans-serif" color="#3366ff"><a href="http://goog_1166506706/" target="_blank">www.mopowered.co.uk</a></font></div><div style="color:rgb(136,136,136);font-family:arial,sans-serif;font-size:13.333333969116211px;background-color:rgb(255,255,255)">
<br></div></div></div>
<br><br><div class="gmail_quote">On 28 January 2013 10:55, Simon MacMullen <span dir="ltr"><<a href="mailto:simon@rabbitmq.com" target="_blank">simon@rabbitmq.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Hi, thanks. So it looks like the cluster is experiencing network partitions:<br>
<br>
<a href="http://www.rabbitmq.com/partitions.html" target="_blank">http://www.rabbitmq.com/<u></u>partitions.html</a><br>
<br>
since we can see (for example) that around 18-Jan-2013 03:30:21 both nodes saw the other one go down. RabbitMQ clusters do not tolerate partitions well, so this needs to be fixed I'm afraid.<br>
<br>
You should not need to rebuild the cluster when this happens however, just stopping and starting nodes should be enough to recover.<br>
<br>
Cheers, Simon<div class="im"><br>
<br>
On 25/01/13 17:04, Ben West wrote:<br>
</div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Hi Simon,<br>
<br><div class="im">
Please find attached.<br>
<br>
Thanks again,<br>
<br>
Ben<br>
<br>
<br>
On 25 January 2013 16:41, Simon MacMullen <<a href="mailto:simon@rabbitmq.com" target="_blank">simon@rabbitmq.com</a><br></div><div class="im">
<mailto:<a href="mailto:simon@rabbitmq.com" target="_blank">simon@rabbitmq.com</a>>> wrote:<br>
<br>
Could you send the log for MOPAYPL2 as well? It would be useful to<br>
correlate what's happening with it too.<br>
<br>
Cheers, Simon<br>
<br>
<br>
On 25/01/13 16:28, Ben West wrote:<br>
<br>
<br>
Hi Simon,<br>
<br>
Thanks for for coming back to me.<br>
<br>
Attached is a rabbit mq log from one of the nodes.<br>
<br>
As I mentioned before, this seems to happen once every week or<br>
two. I<br>
can be too specific on when it happened because i only noticed<br>
when I<br>
check the console, however I believe the last occurrence would<br>
have been<br>
yesterday (24th Jan).<br>
<br>
If you need further logs / info let me know and I'll dig it out<br>
for you.<br>
<br>
Kind regards,<br>
<br>
Ben<br>
<br>
<br>
On 25 January 2013 15:32, Simon MacMullen <<a href="mailto:simon@rabbitmq.com" target="_blank">simon@rabbitmq.com</a><br>
<mailto:<a href="mailto:simon@rabbitmq.com" target="_blank">simon@rabbitmq.com</a>><br></div><div class="im">
<mailto:<a href="mailto:simon@rabbitmq.com" target="_blank">simon@rabbitmq.com</a> <mailto:<a href="mailto:simon@rabbitmq.com" target="_blank">simon@rabbitmq.com</a>>>> wrote:<br>
<br>
On 25/01/13 09:52, <a href="mailto:ben.west@mobankgroup.com" target="_blank">ben.west@mobankgroup.com</a><br>
<mailto:<a href="mailto:ben.west@mobankgroup.com" target="_blank">ben.west@mobankgroup.<u></u>com</a>><br></div>
<mailto:<a href="mailto:ben.west@mobankgroup." target="_blank">ben.west@mobankgroup.</a>_<u></u>_com<div class="im"><br>
<mailto:<a href="mailto:ben.west@mobankgroup.com" target="_blank">ben.west@mobankgroup.<u></u>com</a>>> wrote:<br>
<br>
I'm more thank happy to provide logs if these may be<br>
useful?<br>
<br>
<br>
That would be useful. Preferably if you can also tell us<br>
about the<br>
time at which the cluster broke (even if it's just "some<br>
time on<br>
Wednesday" or similar).<br>
<br>
Cheers, Simon<br>
--<br>
Simon MacMullen<br>
RabbitMQ, VMware<br>
<br>
<br>
<br>
<<a href="http://www.mopowered.co.uk" target="_blank">http://www.mopowered.co.uk</a>><br>
<br>
<br>
<br>
--<br>
Simon MacMullen<br>
RabbitMQ, VMware<br>
<br>
<br>
<br>
<<a href="http://www.mopowered.co.uk" target="_blank">http://www.mopowered.co.uk</a>><br>
</div></blockquote><div class="HOEnZb"><div class="h5">
<br>
<br>
-- <br>
Simon MacMullen<br>
RabbitMQ, VMware<br>
</div></div></blockquote></div><br>
<br>
<a href="http://www.mopowered.co.uk" target="_blank"><img src="http://m.mopowered.co.uk/mopowered_email_footer.png"></a><br>