[rabbitmq-discuss] Sending large files to a queue

Mon Jul 19 12:25:16 BST 2010

On Mon, Jul 19, 2010 at 12:20:48PM +0100, Alexis Richardson wrote:
> Carl, to add to Matthew's comment: yes, people have implemented 'file
> chunking' apps over RabbitMQ.  I think Nanite may have (ruby) libs for
> this.

Yeah, you can do that, but it's not necessarily a great idea due to
message reordering issues. However, if you use something like tornado
codes / erasure code, that might be one (very neat) way around the
problem. Pushing the message through Rabbit gives you the advantage that
you don't need some additional logic to work out when to delete the file
from, say, your shared NFS store. However, pushing the message through
Rabbit has the disadvantage that you may find for large files you need
to do chunking... swings, roundabouts and traffic lights.

Matthew