[rabbitmq-discuss] Multiple shovels to increase create a LB queue?

[Michael Giagnocavo]

Hello,

We use shovel to pull messages from 20 RabbitMQ endpoints, but that'll grow to 40 to 100. At some point, we'll run into the limit of a single queue and eventually broker. Right now all these producing RMQ instances get shoveled into a single master exchange and fanout to 6 queues.

Our plan is to make multiple central RabbitMQ servers, and each one will run Shovel and pull from those producing RabbitMQ instances. The idea is that since consumers get load balanced, we can scale horizontally in a nearly "unlimited" fashion just by adding new "central" servers, each shoveling from each producer. And it just requires our consumers to connect to multiple queues, which is no big deal. And each server can run a mirrored queue giving us HA, and if any central server fails, only a part of the capacity is affected.

Is this a sound approach? Or is there some simpler way?

-Michael
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20140114/be6a7c83/attachment.html>