Tim,<br><br>Unfortunately, the RabbitMQ logs are rather silent around the time of this observed behavior. However, looking at other logs, I'm starting to suspect that something triggered rsyslog (running on the same box) to send hundreds of thousands of old log lines to another server, thus potentially choking the ability of other processes (e.g. Rabbit) from responding during this time.<br>
<br>I'll update this thread if I subsequently find something else beyond this. For now, I'm inclined to call this closed.<br><br><div class="gmail_quote">On Mon, Oct 15, 2012 at 12:25 AM, Tim Watson <span dir="ltr"><<a href="mailto:tim@rabbitmq.com" target="_blank">tim@rabbitmq.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hi Matt,<br>
<br>
Usual request for any scrubbed log(s) you can provide please! :)<br>
<br>
Also, if you could confirm that /api/queues every 5 seconds is the only non-amqp traffic the broker(s) will be subject to, that'd be quite useful to know as well.<br>
<br>
Cheers,<br>
<br>
Tim<br>
<div><div class="h5"><br>
<br>
On 13 Oct 2012, at 00:43, Matt Pietrek wrote:<br>
<br>
> We've hit an odd circumstance in production that we can't make heads or tails of. Our setup is two RabbitMQ 2.8.6 nodes running clustered with all HA queues.<br>
><br>
> In front of them is a VIP managed by a keepalive instances running on the same host as the RabbitMQ nodes. Every 5 seconds the keepalive instance runs a custom script that queries the node local broker via the HTTP API, �requesting the set of queues (/api/queues).<br>
><br>
> From our logs, I can see that there was a single interval where the HTTP request returned a 404 error. (The time was the same on both brokers) Previous to this moment in time, all HTTP queries were successful, and after this moment in time, all queries were sucessful - As I said, just one blip on each broker.<br>
><br>
> The only interesting thing we noticed in the logs was that the primary broker's rabbit@xxx-sasl file had this little snippet about 30 seconds after the hiccup occurred:<br>
><br>
> =SUPERVISOR REPORT==== 12-Oct-2012::15:06:47 ===<br>
> Supervisor: {<0.25499.41>, rabbit_channel_sup_sup}<br>
> Context: shutdown_error<br>
> Reason: shutdown<br>
> Offender: [{pid,<0.25503.41>},<br>
> {name,channel_sup},<br>
> {mfa,{rabbit_channel_sup,start_link,[]}},<br>
> {restart_type,temporary},<br>
> {shutdown,infinity},<br>
> {child_type,supervisor}]<br>
><br>
> I'm mostly at a loss to process what the snippet is telling me, nor can tell if the two things are related. Any help is appreciated!<br>
><br>
> Thanks,<br>
><br>
> Matt<br>
</div></div>> _______________________________________________<br>
> rabbitmq-discuss mailing list<br>
> <a href="mailto:rabbitmq-discuss@lists.rabbitmq.com">rabbitmq-discuss@lists.rabbitmq.com</a><br>
> <a href="https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss" target="_blank">https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss</a><br>
<br>
_______________________________________________<br>
rabbitmq-discuss mailing list<br>
<a href="mailto:rabbitmq-discuss@lists.rabbitmq.com">rabbitmq-discuss@lists.rabbitmq.com</a><br>
<a href="https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss" target="_blank">https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss</a><br>
</blockquote></div><br>