[rabbitmq-discuss] Advice needed on new 'feature'

Thu Apr 23 13:50:25 BST 2009

Matthias,

Exchange-to-exchange bindings sound very interesting, I didn't think about
that and didn't know it could be implemented easily.

And yes, your assumptions are correct, I would like the messages buffered
for the first client to connect and then as the others come on they just get
later ones. With the patch I submitted that is exactly how it works now and
I love it.
It also makes STOMP connections (since I'll be using STOMP as protocol for
client to subscribe and consume) very easy to handle since subscriptions can
be done easily and mapped directly to the queue. No need to create temporary
queue's on another exchange.

The second approach you gave about client handling de-dup with hashes and
feeding from primary and temporary I suppose could work, but would be much
more complicated in my case than the changes I had made to the server.
Because I don't plan on using transactional ack's or flow control, it is a
reasonable compromise for my purposes.

I'm by no means asking for this to get merged in or anything. I realize it
breaks AMQP specs; breaks all current client implimentations since it's
adding a new mandatory bit bool-field for broadcast in queue.declare, I had
to change py-amqplib and the STOMP adapter for this. I mainly sent to to the
list in case someone else might find it useful.
I realize this 'feature' isn't exactly welcome because of its effects on
ack's and flow/network controls (not mention AMQP spec), but I had just
hoped you guys who made the server to see if my changes make sense IF such a
feature was to be done; if only because I'm new to both Erlang and RabbitMQ.

Thank you though for all the other suggestions. One concern we have in the
team is users that don't come back and whose queue's as a result get filled
with many messages. Since the messages we're sending will not be persisent,
we're not worried about disk usage but more about memory. I'm not sure if
the garbage collector would purge old messages from such queues (I assume
no) and hence the node running out of memory. Some kind of cron-based
purging could also be possible but not very reliable. In which case we may
just go with auto_delete queue's per connection duplicating the bindings of
a 'primary' queue as you suggested in the first reply.

Thank you,
Nemanja

On Thu, Apr 23, 2009 at 1:39 AM, Matthias Radestock <matthias at lshift.net>wrote:

> Nemanja,
>
> Nemanja Stefanovic wrote:
>
>> The auto_delete's per connection is a good idea. I had considered it but
>> it seemed like duplicating many bindings per connection per 'primary' queue
>> might have unnecessary overhead. I suppose there probably wouldn't be more
>> than a few per user, but how does creating new queues and ~15 or so bindings
>> for each affect system performance? The other reason for avoiding
>> re-creation of bindings each time is that the topics are collected from a
>> user-profile from a database. I really liked the exchange for this reason
>> because it allowed for minimizing DB-calls. I realize this is probably a
>> minor point though.
>>
>
> An alternative to duplicating the bindings would be to introduce
> exchange-to-exchange bindings; something that was originally envisaged in
> AMQP and wouldn't be too hard to add to RabbitMQ.
>
> You would then be able to have a three-level topology:
>
> 1) a topic exchange, just as you have now
> 2) one fanout exchange per user, with your ~15 bindings to the topic
> exchange
> 3) the primary and, temporarily, the secondary per-user queues bound with a
> single binding each to the user's fanout exchange.
>
> You end up with one exchange and 15+1(+1 temporarily) bindings per user,
> which RabbitMQ should cope with fine and efficiently.
>
>  Lastly, the current 'primary' queue allows me to store messages/events in
>> it, so when the first client connects it can consume past messages and
>> display them in a "here's what you missed while you were away" kind of case.
>> I wouldn't be able to get this same behaviour with auto_delete queue's since
>> (as far as I know) there's no way to 'synchronize' their contents with the
>> ones of the' primary' queue.
>>
>
> So, if I understand you correctly ...
>
> - when none of the user's clients are connected, messages for that user
> should be buffered
>
> - when a user's client connects it should receive all buffered messages
> (i.e. messages that have not been sent to *any* of the user's clients),
> followed by any new messages
>
> That's an interesting use case.
>
> One possible approach is to have one permanent queue per user and then
> temporary queues for each client, with all queues having the same bindings
> (or being bound to the fanout exchange described above). Clients subscribe
> to to both the permanent queue and temporary queue and maintain state to
> allow them to spot duplicates, i.e. messages that came from both the
> permanent and temporary queue.
>
> While that is not the most efficient of solutions, the cost is bounded.
> Each message is sent to a client at most twice. And the de-dup state is
> bounded too - in the worst case you need to remember the hashes of all
> messages received from the permanent queue that have not been received (yet,
> or possibly never) from the temporary queue.
>
>  As for acknowledgements, I'm not using them in this case so I'm not too
>> worried about that.
>>
>
> Right, but it stops the broadcast-queue feature from being orthogonal.
> Other aspects that will break things are transactional acks, network flow
> control (which blocks consumers) and channel flow control (ditto).
>
>
> Regards,
>
> Matthias.
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20090423/1d1cec38/attachment.htm 