[rabbitmq-discuss] Problem transferring large message
matthias at rabbitmq.com
Wed Aug 22 12:46:30 BST 2012
On 22/08/12 12:22, Zabrane Mickael wrote:
>> On 22/08/12 10:34, Zabrane Mickael wrote:
>>> I asked this question long time ago and the idea was to break our
>>> big messages (~50Mb) in smaller ones (100Kb). Everything was fine
>>> after that with the default settings !!!
>> Fragmenting the message is fine, but the broker does not force you
>> to do this.
> You sure about stability? Sending 50Mb at high rate makes RabbitMQ
> very unhappy (lot of crashes and messages lost).
I have just tested twenty producers sending 50Mb messages to Rabbit as
fast as they can. The combined rate was ~30Hz. And that's on a fairly
modest machine. There were some long pauses when messages were getting
paged to disk, and memory peaked at about 80% (so twice the configured
threshold), but that's expected. No crashes.
With smaller messages the performance profile will be smoother, i.e. the
pauses will be shorter and memory peaks will be lower. But there are
downsides to fragmenting messages aside from the extra code involved...
> That's what we can't afford to have. RAM is limited and we had to
> deal with millions of very big messages.
When you say 'deal with millions', what do you mean? That your rabbit
needs to hold (in its queues) millions of 50Mb messages? If you fragment
messages at 100k that would translate to 500 million messages. You'd
need a fair amount of RAM for that since each message has some fixed
memory footprint (in the small hundreds of bytes, iirc) even when paged
More information about the rabbitmq-discuss