[rabbitmq-discuss] Unexaplainable behaviour with shovel plugin.

[Claire Fautsch]

Hi Simon

thanks for your feedback. This is also what we finally thought. I actually
made a small mistake in my comment above, delivery rate on source !=
publish rate on destination, and confirm and ack rate are equal (so the
other way round).

So we are currently thinking of setting the prefetch_count.

Probably this will not really solve the situation, as we will see then the
messages waiting as "Ready" instead of as "Unacknowledged", but on the
other hand, maybe it avoids the shovel connections to the destination
brokers to get in a "flow" or even "blocked" state, where the publishs are
limited. (any opinion on this?)

Thanks,
Claire


2014-02-27 13:03 GMT+01:00 Simon MacMullen <simon at rabbitmq.com>:

> On 26/02/14 08:57, Claire Fautsch wrote:
>
>> Hello,
>>
>
> Hi!
>
>  This leads to the fact, that we have on the destination servers a total
>> of two hanful of messages that are not yet confirmed, however on the
>> source servers we have millions of messages that are waiting for
>> confirmation (acknowledgement)
>>
>> We would expect with some threshold that
>> delivery rate on source = publish rate on destination (which is the case)
>> confirm rate on destination = acknowledge rate on source (which shows
>> considerable difference)
>>
>> Does anyone have an idea or suggestion what could be the reason for
>> this? Is it a bad idea to have load balancer as destination in the
>> shovel, or should that work fine? Network issue?
>>
>
> I doubt the load balancer is the problem. I think I have a reasonable idea
> where the problem lies.
>
> The issue is that the shovel does not enforce any form of flow control
> other than (optionally) using prefetch limiting, which you are not using.
>
> So your source servers are delivering messages into the shovel as fast as
> they can, and your destination servers are accepting messages as fast as
> *they* can, but they are ending up being a bit slower. Nothing is creating
> any back pressure on the source servers, and so messages are queuing up
> inside the shovel. Since you are using on_confirm ack mode, these show as
> unacknowledged messages on the source.
>
>  Here some more details on our shovel setup:
>> ack_mode=on_confirm
>> prefetch_count=0 (default)
>> reconnect_delay=5
>>
>
> I suspect that if you set prefetch_count to some high-but-not-insane
> number (exactly how high depends on your message size + rate but I might
> start the bidding at 1,000) this might solve your problem.
>
> Of course, if your destination servers are actually slower than your
> source ones, then you might need to do something about that. But turning on
> prefetch limiting would make the system better-behaved and make it clearer
> where your issues are.
>
> There might be another issue though - on all released versions of RabbitMQ
> turning on prefetch limiting reduces performance somewhat. This will get
> fixed in the next release.
>
> Cheers, Simon
>
> --
> Simon MacMullen
> RabbitMQ, Pivotal
>



-- 


*Claire Fautsch*
Server Developer
cfautsch at goodgamestudios.com

Goodgame Studios
Theodorstr. 42-90, House 9
22761 Hamburg, Germany
Phone: +49 (0)40 219 880 -0

*www.goodgamestudios.com <http://www.goodgamestudios.com>*


Goodgame Studios is a branch of Altigi GmbH
Altigi GmbH, District court Hamburg, HRB 99869
Board of directors: Dr. Kai Wawrzinek, Dr. Christian Wawrzinek, Fabian
Ritter
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20140227/c18c1295/attachment.html>