[rabbitmq-discuss] Lost messages in cluster

[Jerry Kuch]

Hi, Artsiom:

> I mean a set of brokers, exchanges and queues and relations between them.

Gotcha.  Sometimes it's helpful to keep the actual network topology of
the brokers and how the networks connect them mentally separate from
the implicit topology of exchanges, bindings, and queues that live
within the brokers, since, as we've seen earlier in this thread, they
often aren't quite the same thing, depending on how things are
configured. :-)

> Not, I dont use mirrored queues.

Gotcha.

> Each node is physical/virtual machine with installed rabbitmq broker,
> consumer and producer. Consumer and producer always connect to local
> broker.
> Consumer receives messages only from queues that were
> declared on local broker.  Producer publishes messages to fanout
> exchange to which all queues are bound.

So your consumers and producers live on the same machine as the broker
they're using and essentially use Rabbit as an IPC mechanism?  Or am I
misreading you?  As for the fanout exchanges into which you're
publishing, are you expecting messages pushed into them ever to be
seen on other hosts/nodes?  I'm trying to guess what the nature of
your application is and make sure I'm not misunderstanding you.  It
sounds from the above paragraph that you're not currently looking at a
clustering scenario at all?

> All brokers are in the same
> datacenter, but any broker may become unavailable to others. And if
> that period of time is less than net_ticktime, queue, declared on that
> broker, misses messages.

So at minimum you have a network environment that you believe is
sufficiently hostile or unreliable that you can't cluster your brokers
using the regular Rabbit/Erlang distribution mechanisms?

> I tried to use active/active mirrored queues, declaring them
> with "x-ha-policy" = "all" and faced some unpleasant issue:
> If any node become unavailable, the command
> # rabbitmqctl list_queues
> hangs and I cant get any information.

Ah, now *that* shouldn't be happening.  Which version of RabbitMQ were
you using and are you sure your cluster was set up OK?  Does the
broker appear to actually be alive and listening on port 5672 (the
default AMQP port; elsewhere if you've changed it).  Having
rabbitmqctl hang is generally a sign that something is misconfigured...

> Thats why I looked at federation plugin.

With federation you'd be able to have messages published to one broker
or cluster thereof automatically sent downstream to other
brokers/clusters.  If you're all in a single LAN type deployment this
may be overkill-ish for what you need.

Can you say a bit more about your application and what sorts of things
are communicating and what they're passing amongst one another?  Are
you sure clustering, with an appropriate HA solution, whether the
original active-passive or the new active-active definitely doesn't
meet your needs?

> I'll continue playing with HA queues and hope this solution will help
> to solve my problem. I will keep you informed of the results of my
> experiments.

Please do.  If I'm missing the structure of your intended use class,
please also disabuse me of my misunderstanding so I can try to help
more coherently... :-)

Best regards,
Jerry