[rabbitmq-discuss] How to sanely bind one queue to many exchanges?

[Nathan Gray]

Hi Paul,

On Thu, Aug 13, 2009 at 2:54 AM, Paul Jones<pauljones23 at gmail.com> wrote:
> Hi Nathan,
>
> Would it be possible to detail a bit more about your use case, and thus why
> you don't know when exchanges will come and go? In most scenarios, we'd
> recommend that you just declare the exchanges in a non-passive manner - but
> I take it that there is something special about your use case that prevents
> this.

Maybe, maybe not.  I've written a multiplayer boardgame.  I used
Google App Engine for the server, which works fine but requires that
the client poll the server to detect when someone has made a move,
sent a chat, or otherwise generated an event.  (GAE doesn't allow
long-lived connections or raw socket access.)  I'm hoping to use
rabbitmq as an "event server" to get real-time updates to clients
without pounding GAE with HTTP requests.  When an event is generated
for a game, GAE publishes to the event server, which tells any
connected clients of the game to ask GAE for an update.

There will be many games running, and each game has 2-4 players.  The
natural mapping (IMHO) to AMQP is to have one fanout exchange per
game.  The server on GAE creates this (durable) exchange when a new
game starts.  When the game ends there will be no more events from it,
so the exchange is deleted.  It's possible for a game to end before
the client connects (the last player can play while the client is
offline).  Each player can play multiple games, thus the requirement
to connect to multiple exchanges.  Clients connect with auto-delete
queues, since message content is uninteresting -- it's only the timing
they care about.

If there's a better way to do this I'm happy to hear it.  I could use
one big direct exchange with player names as routing keys, but that
would require my publisher to send multiple messages for each game
event, one for each player in the game.  I could use one big topic
exchange with routing keys like "joe.sally.bob" and subscribe with
patterns like "#.bob.#", but putting all the messages through one
topic exchange doesn't seem like it would scale well.

Declaring exchanges non-passively could probably be made to work, but
my intention was to do my best to disallow creation of exchanges by
users in order to prevent abuse.  (I'll save my rant about rabbitmq's
permissions system for another time...)

> If your requirement definitely does stand, there isn't really a cleaner way
> to do this from the queue-binding end. Speculating a bit on your use case,
> if you're trying to bind every exchange in your system to a single queue,
> would it be possible to perform the binding at the point of exchange
> creation (ie, instead of a queue binding to a well-known exchange, bind the
> exchange to a well-known queue).
>
> Exchange-to-exchange bindings are still just a proposal at the moment I'm
> afraid.

That's a shame.  They're a perfect fit for many-to-many relationships
like mine.  I need one fan-out exchange per game and one fan-in
exchange per user.

Thanks,
-n8

-- 
http://n8gray.org

> On Tue, Aug 11, 2009 at 8:20 PM, Nathan Gray <n8gray at n8gray.org> wrote:
>>
>> Hi folks,
>>
>> I'm using rabbitmq as an event notification system.  In my application
>> each client will subscribe to a number of exchanges.  Exchanges may
>> come and go.  The client cannot be expected to know in advance which
>> exchanges no longer exist, so they will use exchange.declare(...,
>> passive=True,...) to find out which ones exist.  However,
>> exchange.declare in passive mode raises a channel exception and kills
>> the channel if the exchange doesn't exist.  Assuming I want to use one
>> channel per client I'm stuck with ugly, racy code to subscribe to
>> multiple exchanges.  (See the end of the post for the python code to
>> do it.)  And if something goes wrong and kills the channel I have to
>> go through the whole dance again.  This seems crazy.
>>
>> My question is, what's the right way for one client to get messages
>> from multiple exchanges?  Can one auto-delete queue be bound in
>> multiple channels, and if so is that the proper approach?  Should I be
>> making N queues & channels to subscribe to N exchanges?
>>
>> Also, exchange-exchange binding [1] would solve a lot of problems for
>> me -- does it work or is it just a proposal?
>>
>> Thanks,
>> -n8
>>
>> --
>> http://n8gray.org
>>
>> [1]: https://dev.rabbitmq.com/wiki/ExchangeToExchangeBindings
>>
>> while True:
>>    chan = connection.channel()
>>    found = []
>>    for xc in exchanges:
>>        try:
>>            chan.exchange_declare(exchange=xc, passive=True, ...)
>>            found.append(xc)
>>        except amqp.exceptions.AMQPChannelException, e:
>>            if e.amqp_reply_code == 404:
>>                # Channel got closed.  Open a new channel.
>>                chan = connection.channel()
>>            else:
>>                raise e
>>    # Sure hope nothing gets deleted at this point!
>>    qname, _, _ = chan.queue_declare(exclusive=True)
>>    try:
>>        for xc in found:
>>            chan.queue_bind(queue=qname, exchange=xc)
>>    except amqp.exceptions.AMQPChannelException, e:
>>        if e.amqp_reply_code == 404:
>>            # Oops, an exchange got deleted.  Start over.
>>            continue
>>        else:
>>            raise e
>>    # No exception raised, so we're done (until something
>>    # else goes wrong & kills the channel...)
>>    break
>>
>> _______________________________________________
>> rabbitmq-discuss mailing list
>> rabbitmq-discuss at lists.rabbitmq.com
>> http://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
>
>