[rabbitmq-discuss] Unexaplainable behaviour with shovel plugin.

[Jason McIntosh]

We saw similar behavior mostly though related to latency issues between
remote sites which actually caused some crashes due to memory and load (we
had almost 3 million messages backlogged and shovel went "WTF" when we
started it up).  We ended up setting a prefetch of 1500 messages, and these
problems went away.  We also ended up using x-consistent-hash to multiplex
the shoveling to a remote system, which also helped.  We didn't have a
capacity issue so much as a latency issue.  SO I HIGHLY recommend setting a
prefetch that's not unlimited if you're doing WAN replication,

Jason


On Thu, Feb 27, 2014 at 6:31 AM, Simon MacMullen <simon at rabbitmq.com> wrote:

> On 27/02/14 12:11, Claire Fautsch wrote:
>
>> Probably this will not really solve the situation, as we will see then
>> the messages waiting as "Ready" instead of as "Unacknowledged"
>>
>
> Quite possibly. But it will keep memory use in your shovel much more
> controlled, which has to be a good thing.
>
>
>  but on
>> the other hand, maybe it avoids the shovel connections to the
>> destination brokers to get in a "flow" or even "blocked" state, where
>> the publishs are limited. (any opinion on this?)
>>
>
> Well, if the publishing connection is in "flow" state then that just means
> that it would like to publish faster, but something (the downstream queue
> probably) can't keep up.
>
> Ultimately you probably need more capacity in your destination brokers, if
> the source brokers are backing up.
>
>
> Cheers, Simon
>
> --
> Simon MacMullen
> RabbitMQ, Pivotal
> _______________________________________________
> rabbitmq-discuss mailing list
> rabbitmq-discuss at lists.rabbitmq.com
> https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
>



-- 
Jason McIntosh
https://github.com/jasonmcintosh/
573-424-7612
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20140227/18f12fd2/attachment.html>