[rabbitmq-discuss] Scaling to large number of bindings

[Simon MacMullen]

On 27/05/14 22:17, Aaron Dufour wrote:
> On the other end, I would have an exchange on each outgoing server that
> the queues would bind to (a direct exchange - they are binding to exact
> routing keys).  However, when I federate this to the outgoing exchanges
> on the routing servers, every binding is stored on every routing server,
> which puts a hard ceiling on the number of bindings I can make.  Ideally
> the outgoing servers would only tell each routing server about the
> bindings it cares about (since each routing server is only dealing with
> a known subset of messages, based on routing keys), but I can't see a
> way to make this happen.

I am not sure I fully understand your explanation here, but: exchange 
federation will reproduce a binding on the upstream for every 
{exchange,routing key,arguments} combination on the downstream - i.e. it 
will only collapse bindings together if they are identical in everything 
but the destination.

The rationale for this is that it considers it more important that only 
messages which are known to be needed downstream get transmitted over 
the federation link than that the number of bindings in use be limited.

> Do I need something other than federation?

If you're trying not to transmit bindings upstream, then yes. You should 
be able to set up a very similar link (except without the automatic 
binding propagation) using the shovel.

> Am I thinking about the
> whole problem wrong?  I could use some help figuring out the right
> topology to allow the system to scale for the foreseeable future.

Your plan sounds reasonable assuming that you really do need to have 
hundreds of thousands of bindings around.

But since that number of bindings is awkward to deal with, it's really 
worth investigating whether that is true. Obviously I don't know why you 
need so many, but I can think of a couple of antipatterns that would 
lead to use of many bindings:

* Binding all possible routing keys with a fixed prefix / suffix 
instead of using wildcards (i.e. bind "a.b", "a.c", "a.d" etc instead of 
"a.*").

* Binding nearly all possible routing keys in order to exclude one and 
match everything else (investigate the alternate-exchange feature instead).

Cheers, Simon

-- 
Simon MacMullen
RabbitMQ, Pivotal