[rabbitmq-discuss] chunking a file to many messages

[Vlad Alexandru Ionescu]

Hi,

To answer 1:
A setup I could imagine would be to have a "work queue", in which each 
message would consist of a file identifier and a queue name. Your 
servers can pick up jobs from this queue in a round-robin fashion. 
Whenever you want to send a new file with your clients, you first 
declare a new queue for your file (let's call it "file queue"), and then 
publish a message to the "work queue" mentioned above (with the name of 
the "file queue"). Then you continue publishing bits of the file to the 
"file queue". The servers that serve a job from the "work queue" would 
subscribe to the "file queue", whose name was picked up from the job's 
message.

And to answer 2:
RabbitMQ guarantees ordering of the messages, as long as the messages in 
question follow the exact same path through the clients and the broker 
(same publisher, same exchange, same queue and same consumer). With the 
setup described above, you should be good to go without extra identifiers.

Hope that makes sense,
Vlad.


On 03/08/2011 09:52 PM, asaf david wrote:
> hello
> I'm a RabbitMQ beginner. I need to handle a case where several clients 
> can produce files for processing, which should be done in some 
> servers. I've realized there is no native support for files transfers, 
> so I'm going to chunk a produced file to many small messages are 
> transfer them (similar to the answer given here 
> <https://groups.google.com/d/msg/rabbitmq-discuss/-G39l-JJtp4/OTSd9FUJdfEJ>), 
> and reassemble them to a whole file at the server. However there are 
> two points i'm not sure about:
>
>    1. How do i make sure that different messages that belong to the
>       same file won't end up in different servers? I want a fair
>       dispatching between the servers on a per-file base, not per-message
>    2. How do I make sure that messages won't be reordered? should i
>       attach an identifier to each message to mark it's location in
>       the original file?
>
> thanks in advance :)
>
>
> _______________________________________________
> rabbitmq-discuss mailing list
> rabbitmq-discuss at lists.rabbitmq.com
> https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20110309/8aed0320/attachment.htm>