[rabbitmq-discuss] Advice needed on new 'feature'

[Matthias Radestock]

Nemanja,

Nemanja Stefanovic wrote:
> The auto_delete's per connection is a good idea. I had considered it but 
> it seemed like duplicating many bindings per connection per 'primary' 
> queue might have unnecessary overhead. I suppose there probably wouldn't 
> be more than a few per user, but how does creating new queues and ~15 or 
> so bindings for each affect system performance? The other reason for 
> avoiding re-creation of bindings each time is that the topics are 
> collected from a user-profile from a database. I really liked the 
> exchange for this reason because it allowed for minimizing DB-calls. I 
> realize this is probably a minor point though.

An alternative to duplicating the bindings would be to introduce 
exchange-to-exchange bindings; something that was originally envisaged 
in AMQP and wouldn't be too hard to add to RabbitMQ.

You would then be able to have a three-level topology:

1) a topic exchange, just as you have now
2) one fanout exchange per user, with your ~15 bindings to the topic 
exchange
3) the primary and, temporarily, the secondary per-user queues bound 
with a single binding each to the user's fanout exchange.

You end up with one exchange and 15+1(+1 temporarily) bindings per user, 
which RabbitMQ should cope with fine and efficiently.

> Lastly, the current 'primary' queue allows me to store messages/events 
> in it, so when the first client connects it can consume past messages 
> and display them in a "here's what you missed while you were away" kind 
> of case. I wouldn't be able to get this same behaviour with auto_delete 
> queue's since (as far as I know) there's no way to 'synchronize' their 
> contents with the ones of the' primary' queue.

So, if I understand you correctly ...

- when none of the user's clients are connected, messages for that user 
should be buffered

- when a user's client connects it should receive all buffered messages 
(i.e. messages that have not been sent to *any* of the user's clients), 
followed by any new messages

That's an interesting use case.

One possible approach is to have one permanent queue per user and then 
temporary queues for each client, with all queues having the same 
bindings (or being bound to the fanout exchange described above). 
Clients subscribe to to both the permanent queue and temporary queue and 
maintain state to allow them to spot duplicates, i.e. messages that came 
from both the permanent and temporary queue.

While that is not the most efficient of solutions, the cost is bounded. 
Each message is sent to a client at most twice. And the de-dup state is 
bounded too - in the worst case you need to remember the hashes of all 
messages received from the permanent queue that have not been received 
(yet, or possibly never) from the temporary queue.

> As for acknowledgements, I'm not using them in this case so I'm not too 
> worried about that.

Right, but it stops the broadcast-queue feature from being orthogonal. 
Other aspects that will break things are transactional acks, network 
flow control (which blocks consumers) and channel flow control (ditto).


Regards,

Matthias.