<div dir="ltr">Sure .. but I'm wondering if there is native syslog support in RabbitMQ. Its much cleaner that way.</div><div class="gmail_extra"><br clear="all"><div><div dir="ltr">Matt Wise<div>Sr. Systems Architect</div>
<div>Nextdoor.com</div></div></div>
<br><br><div class="gmail_quote">On Wed, Nov 6, 2013 at 8:18 AM, Frank Shearar <span dir="ltr"><<a href="mailto:frank.shearar@gmail.com" target="_blank">frank.shearar@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div class="HOEnZb"><div class="h5">On 6 November 2013 16:02, Matt Wise <<a href="mailto:matt@nextdoor.com">matt@nextdoor.com</a>> wrote:<br>
> See comments inline.<br>
><br>
><br>
> On Wed, Nov 6, 2013 at 2:37 AM, Tim Watson <<a href="mailto:tim@rabbitmq.com">tim@rabbitmq.com</a>> wrote:<br>
>><br>
>> Hi Matt,<br>
>><br>
>> Sorry to hear you've been running into problems.<br>
>><br>
>> On 5 Nov 2013, at 22:05, Matt Wise wrote:<br>
>><br>
>> > (sorry if this gets posted twice.. first email never seemed to make it<br>
>> > to the list)<br>
>> ><br>
>> > Hey... I had a pretty rough time today with a 3-node RabbitMQ 3.1.3<br>
>> > cluster thats under pretty heavy use (6-7 million messages per day -- 100MB<br>
>> > peak bandwidth per node). I want to pose a few questions here. First off,<br>
>> > here's the basic configuration though.<br>
>> ><br>
>> > Configuration:<br>
>> > � serverA, serverB and serverC are all configured with RabbitMQ 3.1.3.<br>
>> > They each are configured via Puppet ... and Puppet uses a dynamic node<br>
>> > discovery plugin (zookeeper) to find the nodes. The node lists are<br>
>> > hard-coded into the rabbitmq.config file. A dynamic server list generator<br>
>> > supplies Puppet with this list of servers (and is not really necessary to<br>
>> > describe here in this email).<br>
>> ><br>
>> > Scenario:<br>
>> > � A momentary configuration blip caused serverA and serverB to begin<br>
>> > reconfiguring their rabbitmq.config files... when they did this, they also<br>
>> > both issued a 'service rabbitmq restart' command. This command took<br>
>> > 40+minutes and ultimately failed. During this failure, RabbitMQ was<br>
>> > technically running and accepting connections to the TCP ports ... but it<br>
>> > would not actually answer any queries. Commands like list_queues would hang<br>
>> > indefinitely.<br>
>> ><br>
>><br>
>> What ha recovery policy (if any) do you have set up? A and B might get a<br>
>> different "view of the world" set up in their respective rabbitmq.config<br>
>> files (either to each other and/or to C) and then get restarted, but this<br>
>> should affect their view of the cluster, because as per<br>
>> <a href="http://www.rabbitmq.com/clustering.html" target="_blank">http://www.rabbitmq.com/clustering.html</a>:<br>
>><br>
>> "Note that the cluster configuration is applied only to fresh nodes. A<br>
>> fresh nodes is a node which has just been reset or is being start for the<br>
>> first time. Thus, the automatic clustering won't take place after restarts<br>
>> of nodes. This means that any change to the clustering via rabbitmqctl will<br>
>> take precedence over the automatic clustering configuration."<br>
>><br>
><br>
> So far we've taken the approach that clustering configuration should be<br>
> hard-coded into the rabbitmq.config files. This works well in explicitly<br>
> defining all of the hosts in a cluster on every machine, but it also means<br>
> that adding a 4th node to a 3-node cluster will cause the 3 running live<br>
> nodes to do a full service restart, which is bad. Our rabbitmq.config though<br>
> is identical on all of the machines (other than the server-list, which may<br>
> have been in-flux when Puppet was restarting these services)<br>
><br>
>> [<br>
>> � � � � {rabbit, [<br>
>> � � � � � � � � {log_levels, [{connection, warning}]},<br>
>> � � � � � � � � {cluster_partition_handling,pause_minority},<br>
>> � � � � � � � � {tcp_listeners, [ 5672 ] },<br>
>> � � � � � � � � {ssl_listeners, [ 5673 ] },<br>
>> � � � � � � � � {ssl_options,<br>
>> [{cacertfile,"/etc/rabbitmq/ssl/cacert.pem"},<br>
>> � � � � � � � � � � � � {certfile,"/etc/rabbitmq/ssl/cert.pem"},<br>
>> � � � � � � � � � � � � {keyfile,"/etc/rabbitmq/ssl/key.pem"},<br>
>> � � � � � � � � � � � � {verify,verify_peer},<br>
>> � � � � � � � � � � � � {fail_if_no_peer_cert,true}<br>
>> � � � � � � � � ]},<br>
>> � � � � � � � � {cluster_nodes,['rabbit@i-23cf477b', 'rabbit@i-07d8bc5f',<br>
>> 'rabbit@i-a3291cf8']}<br>
>> � � � � ]}<br>
>> ].<br>
><br>
><br>
>><br>
>> > Questions:<br>
>> > � 1. We only had ~2500 messages in the queues (they are HA'd and<br>
>> > durable). The policy is { 'ha-mode': 'all' }. When serverA and serverB<br>
>> > restarted, why did they never come up? Unfortunately in the restart process,<br>
>> > they blew away their log files as well which makes this really tough to<br>
>> > troubleshoot.<br>
>><br>
>> It's nigh on impossible to guess what might've gone wrong without any log<br>
>> files to verify against. We could sit and stare at all the relevant code for<br>
>> weeks and not spot a bug that's been triggered here, since if it were<br>
>> obvious we would've fixed it already.<br>
>><br>
>> If you can give us a very precise set of steps (and timings) that led to<br>
>> this situation, I can try and replicate what you've seen, but I don't fancy<br>
>> my chances to be honest.<br>
><br>
><br>
> Its a tough one for us to reproduce.. but I think the closest steps would<br>
> be:<br>
><br>
> � 1. Create a 3-node cluster... configured with similar config to the one I<br>
> pasted above.<br>
> � 2. Create enough publishers and subscribers that you have a few hundred<br>
> messages/sec going through the three machines.<br>
> � 3. On MachineA and MachineC, remove MachineB from the config file.<br>
> � 4. Restart MachineA's rabbitmq daemon using init script<br>
> � 5. Wait 3 minutes... theoretically #4 is still in process.. now issue the<br>
> same restart to MachineC.<br>
><br>
> � Fail.<br>
><br>
> Thats our best guess right now.. but agreed, the logs are a problem. Can we<br>
> configure RabbitMQ to log through syslog for the future?<br>
<br>
</div></div>Syslog-ng can tail logs, dumping the logs in some arbitrary<br>
destination (another file, Papertrail, etc.)<br>
<span class="HOEnZb"><font color="#888888"><br>
frank<br>
</font></span><div class="HOEnZb"><div class="h5"><br>
>> > � 2. I know that restarting serverA and serverB at nearly the same time<br>
>> > is obviously a bad idea -- we'll be implementing some changes so this<br>
>> > doesn't happen again -- but could this have lead to data corruption?<br>
>><br>
>> It's possible, though obviously that shouldn't really happen. How close<br>
>> were the restarts to one another? How many HA queues were mirrored across<br>
>> these nodes, and were they all very busy (as your previous comment about<br>
>> load seems to suggest)? We could try replicating that scenario in our tests,<br>
>> though it's not easy to get the timing right and obviously the existence of<br>
>> network infrastructure on which the nodes are running won't be the same (and<br>
>> that can make a surprisingly big difference IME).<br>
><br>
><br>
> The restarts were within a few minutes of each other. There are 5 queues,<br>
> and all 5 queues are set to mirror to 'all' nodes in the cluster. They were<br>
> busy, but no more than maybe 100 messages/sec coming in/out.<br>
><br>
>><br>
>><br>
>> > Once the entire RabbitMQ farm was shut down, we actually were forced to<br>
>> > move the rabbitmq data directory out of the way and start up the farm<br>
>> > completely with blank databases. It seemed that RabbitMQ 3.1.3 really did<br>
>> > not want to recover from this failure. Any thoughts?<br>
>> ><br>
>> > � 3. Lastly .. in the event of future failures, what tools are there for<br>
>> > recovering our Mnesia databases? Is there any way we can dump out the data<br>
>> > into some raw form, and then import it back into a new fresh cluster?<br>
>> ><br>
>><br>
>> I'm afraid there are not, at least not "off the shelf" ones anyway. If you<br>
>> are desperate to recover important production data however, I'm sure we<br>
>> could explore the possibility of trying to help with that somehow. Let me<br>
>> know and I'll make some enquiries at this end.<br>
><br>
><br>
> At this point we can move on from the data loss... but it does make for an<br>
> interesting issue. Having tools to analyze the Mnesia DB and get "most of"<br>
> the messages out in some format where they could be re-injected into a fresh<br>
> cluster would be an incredibly useful tool. I wonder how hard it is to do?<br>
><br>
>><br>
>> Cheers,<br>
>> Tim<br>
>><br>
>><br>
>> _______________________________________________<br>
>> rabbitmq-discuss mailing list<br>
>> <a href="mailto:rabbitmq-discuss@lists.rabbitmq.com">rabbitmq-discuss@lists.rabbitmq.com</a><br>
>> <a href="https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss" target="_blank">https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss</a><br>
><br>
><br>
><br>
> _______________________________________________<br>
> rabbitmq-discuss mailing list<br>
> <a href="mailto:rabbitmq-discuss@lists.rabbitmq.com">rabbitmq-discuss@lists.rabbitmq.com</a><br>
> <a href="https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss" target="_blank">https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss</a><br>
><br>
_______________________________________________<br>
rabbitmq-discuss mailing list<br>
<a href="mailto:rabbitmq-discuss@lists.rabbitmq.com">rabbitmq-discuss@lists.rabbitmq.com</a><br>
<a href="https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss" target="_blank">https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss</a><br>
</div></div></blockquote></div><br></div>