[rabbitmq-discuss] Distribution model advice sought

[Matthias Radestock]

Mark,

Mark J. Reed wrote:
> That's it.  Basically, there are N "topics", and each message gets
> published to all of those topics - but without the producer having to
> do N publishes, and with only one consumable copy of each message per
> topic.
> 
> Our initial design had a single exchange with N queues bound to it;
> each of the M consumers then subscribes to one of the N queues.
> Simple.

How do you avoid the producer having to publish the message N times in 
this setting? I am asking because there is a new feature appearing in 
the next release that addresses that very problem :)

> But not reliable.  The problem is that a queue lives on a single node,
> and if that node goes down, the queue is gone.  It's not the loss of
> data from the queue - no big deal in our use case - but the fact that
> the queue itself becomes unusable within the cluster, and that entire
> set of consumers stop working.

When a node goes down, taking all its queues with it, the clients 
connected to that node and consuming from the queues will have their 
connections torn down. They would notice that and could simply reconnect 
and redeclare the queues & bindings & their consumers.

The problem you are describing arises for clients that are connected to 
the other, surviving, nodes. They will have no idea that anything is 
amiss and that their consumers (in the AMQP sense, i.e. the things 
declared by basic.consume) are no longer functional.

We have yet another new feature in development which will inform clients 
when their consumers have been cancelled by the server, e.g. when the 
queue from which they were consuming has been deleted or destroyed.

In your setup, when a client receives that notification it could 
redeclare the queue & bindings & their consumers.

Until that feature makes it into a release, as a stop gap clients could 
periodically check whether the queue they are consuming from still 
exist, e.g. by issuing passive queue.declares.


Regards,

Matthias.