[rabbitmq-discuss] For speed, use Shovel or not?

[Jon Brisbin]

It's been languishing in a corner for several months since I've been rather busy, but I wrote a "job scheduler" utility for RabbitMQ at a previous employer. It has a bunch of different things in it but one of them is a BatchMessageHandler. It listens for zip files coming in, unzips them, dumps them into a broker, collects the results, zips them up, then responds to the sender with a zip file of results.

This was to take the place of a somewhat similar system they have where requests from remote WAN nodes are batched, zipped and sent to the datacenter to be executed by a proxy (because of slow WAN pipes).

I'd like to revisit some of this code at some point because there's some pretty neat functionality here, I think, and it could use some love to update it to the latest spring-amqp version and whatnot.

It's not an "unzip it and run it" thing, but it might be a jumping-off point for something like what you describe.


Thanks!

Jon Brisbin

http://jbrisbin.com
Twitter: @j_brisbin


On Apr 28, 2011, at 1:01 PM, Gene Campbell wrote:

> Thank you for the response.    Great one for a FAQ.  The confirmation of batching and compression not being in Shovel is something we until now we couldn't confirm.  (Therefore it's usefulness to us just dropped a bit.)
> 
> The value of being able to ACK back fast, and let a client continue on, while giving some comfort going forward that the message will actually be consumed somewhere after some length of time.  For example, if you can stand 1 minute of latency from publish to consume, but you're client needs an ACK, it will need to store potentially large amounts of data (if your link goes down) until it gets the ACK (link comes back up and message consumed.)  With shovel, it can get a quicker ACK.
> 
> Thanks
> Gene
> 
> On Thu, Apr 28, 2011 at 5:22 AM, Matthew Sackman <matthew at rabbitmq.com> wrote:
> Hi,
> 
> On Sun, Apr 24, 2011 at 02:13:13PM -0700, ristretto.rb wrote:
> > Suppose there are two servers connected across a WAN (read internet),
> > sending transient messages, with all else held constant what is the
> > faster method in terms of messages / sec:  Two brokers and a Shovel,
> > one broker with a remote publisher, or one broker with a remote
> > consumer.  (The last two cases might be really just one case.)
> >
> > msg -> X -> Q -> Shovel -> X -> Q -> Consume
> > vs
> > msg ---------------------------> X -> Q -> Consume
> >
> > What if durability is required?  Does this change the decision?
> 
> Regardless of whether you're using the shovel or not, over the WAN link
> will flow AMQP msgs over TCP/IP. There is no compression or any batching
> used by shovel, so your throughput and latency won't be different in any
> of the combinations you suggest.
> 
> However, you might wish for, eg the publisher to very quickly be able to
> publish a msg and get a confirmation back that the msg has been
> published and accepted by the broker. If the latency of this operation
> is important then it makes sense to have the broker near the publisher.
> 
> The next thing to consider is whether it's actually going to make any
> difference to the consumer whether or not it's got a broker very near
> it. For example, is it going to be doing any synchronous operations
> regularly. If not (i.e. it's just receiving msgs and acking them), then
> there probably isn't much difference whether it's connected to a local
> broker or a remote one.
> 
> Matthew
> _______________________________________________
> rabbitmq-discuss mailing list
> rabbitmq-discuss at lists.rabbitmq.com
> https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
> 
> _______________________________________________
> rabbitmq-discuss mailing list
> rabbitmq-discuss at lists.rabbitmq.com
> https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20110428/b18c7003/attachment.htm>