[rabbitmq-discuss] Unexaplainable behaviour with shovel plugin.
mcintoshj at gmail.com
Thu Feb 27 12:50:33 GMT 2014
We saw similar behavior mostly though related to latency issues between
remote sites which actually caused some crashes due to memory and load (we
had almost 3 million messages backlogged and shovel went "WTF" when we
started it up). We ended up setting a prefetch of 1500 messages, and these
problems went away. We also ended up using x-consistent-hash to multiplex
the shoveling to a remote system, which also helped. We didn't have a
capacity issue so much as a latency issue. SO I HIGHLY recommend setting a
prefetch that's not unlimited if you're doing WAN replication,
On Thu, Feb 27, 2014 at 6:31 AM, Simon MacMullen <simon at rabbitmq.com> wrote:
> On 27/02/14 12:11, Claire Fautsch wrote:
>> Probably this will not really solve the situation, as we will see then
>> the messages waiting as "Ready" instead of as "Unacknowledged"
> Quite possibly. But it will keep memory use in your shovel much more
> controlled, which has to be a good thing.
> but on
>> the other hand, maybe it avoids the shovel connections to the
>> destination brokers to get in a "flow" or even "blocked" state, where
>> the publishs are limited. (any opinion on this?)
> Well, if the publishing connection is in "flow" state then that just means
> that it would like to publish faster, but something (the downstream queue
> probably) can't keep up.
> Ultimately you probably need more capacity in your destination brokers, if
> the source brokers are backing up.
> Cheers, Simon
> Simon MacMullen
> RabbitMQ, Pivotal
> rabbitmq-discuss mailing list
> rabbitmq-discuss at lists.rabbitmq.com
-------------- next part --------------
An HTML attachment was scrubbed...
More information about the rabbitmq-discuss