[rabbitmq-discuss] Design for global data ingestion

[Simon MacMullen]

On 07/11/13 17:47, Dustin wrote:
> We have ran into 2 problems which won't allow us to scale any further.
>   The first is the max bandwidth for a single TCP connection across the
> globe (specifically between the US and China).  The second is we have
> maxed out the CPU for the federation clients on the downstream (SSL is
> enabled, I'm not sure how much CPU overhead that adds).

First of all: federation has seen some optimisations going in to 3.2.x 
so hopefully your CPU use should drop somewhat just from an upgrade 
(I've seen throughput increase by 50% to 100%).

> For the CPU issue, I figured the newly added federated queues would be a
> perfect solution to the problem.  I can setup additional Rabbits on the
> downstream side, setup the federation links, and have everything load
> balance nicely.  The only thing it doesn't address is the max bandwidth
> for a single TCP connection.  Because of our initial design, we would
> run into max bandwidth problems for each queue.

Federated queues in the downstream might not be necessary to get more 
TCP connections in use. You should be able to get away with just 
defining multiple upstreams per real upstream each with the same URI. 
The only slight downside to that is that each upstream will think it 
"owns" the upstream internal exchange and will rewire it when it starts 
up. But in practice having that happen more than once should not be a 
problem.

Oh, and it will (obviously) break ordering, but you said you don't care 
about that.

However, I suspect the next bottleneck you might find would be the 
downstream queue(s) - each queue can only utilise a single CPU. So 
federating together multiple queues on your downstream is likely to be 
useful there.

> Our current objective is to be able to send 20k/sec messages from each
> datacenter for a single application.  In China, the most we can do is
> around 2.5k/sec (ends up being around 1.6MB/sec, this is on a good day).
> Because this message load will be from a single application, with the
> current design, it will be tied to a single routing key.  So for China,
> I would need around 8 TCP connections to do this.
>
> For this use case, message order doesn't matter.  Does anyone have any
> ideas on how I can setup multiple federation links that will be load
> balanced?  Here are some ideas I have, but they all feel hacky.
>
> 1) On the upstreams, use a consistent hash exchange, with exchange to
> exchange bindings to 8 direct exchanges, which would be federated.
> 2) Run multiple instances of RMQ on the downstream machines, and use
> federated queues.  Total number of instances across all machines should
> be greater than 8.

So as discussed above, I don't think this is needed. But if you do start 
needing federated queues in the downstream cluster then that does not 
mean you need to run multiple copies of RabbitMQ - you can federate 
queues with each other within the same cluster.

Just use a URI of "amqp://" and specify the "queue" key in the upstream 
to specify a different queue name (otherwise you federate the queue with 
itself, which is harmless but not exactly useful).

Cheers, Simon

-- 
Simon MacMullen
RabbitMQ, Pivotal