[rabbitmq-discuss] For speed, use Shovel or not?

[Matthew Sackman]

Hi,

On Sun, Apr 24, 2011 at 02:13:13PM -0700, ristretto.rb wrote:
> Suppose there are two servers connected across a WAN (read internet),
> sending transient messages, with all else held constant what is the
> faster method in terms of messages / sec:  Two brokers and a Shovel,
> one broker with a remote publisher, or one broker with a remote
> consumer.  (The last two cases might be really just one case.)
> 
> msg -> X -> Q -> Shovel -> X -> Q -> Consume
> vs
> msg ---------------------------> X -> Q -> Consume
> 
> What if durability is required?  Does this change the decision?

Regardless of whether you're using the shovel or not, over the WAN link
will flow AMQP msgs over TCP/IP. There is no compression or any batching
used by shovel, so your throughput and latency won't be different in any
of the combinations you suggest.

However, you might wish for, eg the publisher to very quickly be able to
publish a msg and get a confirmation back that the msg has been
published and accepted by the broker. If the latency of this operation
is important then it makes sense to have the broker near the publisher.

The next thing to consider is whether it's actually going to make any
difference to the consumer whether or not it's got a broker very near
it. For example, is it going to be doing any synchronous operations
regularly. If not (i.e. it's just receiving msgs and acking them), then
there probably isn't much difference whether it's connected to a local
broker or a remote one.

Matthew