[rabbitmq-discuss] Cluster, shovel, how to accomplish this scenario?

Mon May 9 16:10:19 BST 2011

FWIW- I hinted at this in my blog post about the RabbitMQ <-> Riak integration, but I envisioned a scenario just as you've described (one broker/exchange/whatever knows nothing about the other). I think it would be relatively straightforward to extend this to encompass your other problem (the cache exchange...where you get all the messages historically when binding a new queue) as well.

The Riak exchange type will route messages locally (it's an exchange like any other...you set the type-module argument when declaring the exchange to the erlang module to use when routing the message). When the message gets stored in Riak, the commit hook is triggered, which causes that message to be sent to the configured exchange. You can set what broker, exchange, etc... on a per-bucket basis, so your individual messages don't have to know anything about your routing architecture.

https://github.com/jbrisbin/riak-exchange
https://github.com/jbrisbin/riak-rabbitmq-commit-hooks

It would be good to implement something similar but that bypasses an external data store. An exchange type could handle this without forcing the application to know what's going on (you could put broker settings, exchange settings, etc... on the exchange declaration as arguments). It could "echo" the message into that other broker. Such a thing doesn't, as yet, exist. :)

Thanks!

Jon Brisbin

http://jbrisbin.com
Twitter: @j_brisbin

On May 8, 2011, at 5:29 PM, Simone wrote:

> Hello, I managed to set up a shovel correctly. I have some doubts about the usage scenarios of this feature. Ideally I would like the shovel to be transparent to my application, but I realized that since you have to create a queue on one side and an exchange on the other side then these two make the applications built to use Rabbit somewhat aware of the shovel. 
> With that I refer to the input queue of the shovel needding to be bound to an exchange, therefore the exchange has to exist and someone has to publish messages to it. The same holds true on the other side of the shovel, where clients need to bind to the shovel exchange.
> All this makes sense and works great, but makes my application aware of the exchanges used by the shovel. 
> 
> Here's a scenario. My application has some publishers publishing to a topic exchange named X and some subscribers with their queues bound to that exchange. Setting up a shovel to have the same subscriber clients running on both sides of the shovel with no client code changes and no configuration changes except for the node url requires that the shovel needs to bind its input queue to exchange X and that an equal exchange X is created to the other end of the shovel so that clients on either side bind their queues their node's X exchange.
> 
> This makes the shovel mechanism somewhat aware of my application internals (the exchanges it uses) and requires that the same exchanges exist as soon as the nodes are started, thus moving (or worse, duplicating) their definition in the shovel configuration file. In addition it limits the flexibility as a new exchange creation which needs to be "mirrored" on the other node requires a change in the configuration of the shovel addin.
> 
> As said this works fine, but ideally I would like this mechanism to be transparent to the application. I read about the federation addin, which from a brief read seems to do exactly this. Any ideas or suggestions?
> 
> Thanks
> _______________________________________________
> rabbitmq-discuss mailing list
> rabbitmq-discuss at lists.rabbitmq.com
> https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss

Thanks!

Jon Brisbin

http://jbrisbin.com
Twitter: @j_brisbin