[rabbitmq-discuss] Clustered startup with multiple queues and multiple masters

Thu Jun 21 17:05:05 BST 2012

Thanks Francesco.

Your script looks reasonably close to what I'm doing, except for a couple
of key differences:

All of your nodes are running on the same host. In my setup, each node is
in its own VM. I imagine this introduces additional networking into the
communication between nodes.

When you start Rabbit instances, you're doing it sequentially. In my setup
I start up in parallel on all three VMs via Capistrano. If you can't go the
multi-VM route, you may be able to somewhat simulate this by starting the
instances with an '&' at the end.

I use "killall -9 beam.smp", rather than just "killall beam.smp"

In short, the goal of my test is to simulate a worst case powerout in the
datacenter. All of our Rabbit instances run on separate VMs for fault
tolerance. Hopefully we won't lose them all at once, but if we do, we need
to be able to reliably restart them.

On Thu, Jun 21, 2012 at 6:36 AM, Francesco Mazzoli
<francesco at rabbitmq.com>wrote:

> Hi Matt,
> At Tue, 19 Jun 2012 16:33:05 -0700,
> Matt Pietrek wrote:
> > Francesco,
> >
> > Thanks again for the valuable insight from your reply. I'm down to
> > one issue at this point.
> >
> > Given what you said earlier about it being OK to start the brokers
> > in any order, I wrote a simple "catastrophic stress" test. The good
> > news is that RabbitMQ does what's expected. The bad news: Only most
> > of the time, i.e. about 90%.
>
> First of all, this is more of an erlang question than a RabbitMQ one -
> not that this changes anything, but you could ask about it in
> erlang-questions as well. RabbitMQ clusters are mnesia clusters, so
> they offer the same guarantees.
>
> I can't think of a motivation of why this would be happening, but I'm
> no expert with mnesia. I've attached a script that reproduces your
> test, can you verify that that's more or less what you're doing? I'm
> using the puka python client to publish the messages. I've run it 50
> times but I wasn't able to reproduce your problem.
>
> In the case that my test is indeed accurate I think that the best
> thing is to ask about someone with more mnesia knowledge - I have CCed
> possible candidates :).
>
> Francesco.
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20120621/bbaa7888/attachment.htm>