[rabbitmq-discuss] Unexaplainable behaviour with shovel plugin.

[Claire Fautsch]

Hello,

we are currently experiencing some (for us) unexaplainable behaviour with
shovel plugin.

Before, I enter into the details, let me first outline our setup.

We have a set of RabbitMQ Servers hosted in the cloud, where incoming
messages are queued. On those servers we then have the shovel plugin
running, which distributes the incoming messages to a set of RabbitMQ
Servers in our local datacenter.
The configuration of the shovel does not directly specifiy a RabbitMQ
Server as its destination, but a load balancer, which then distributes to
the servers in our local datacenter.

Since a few days we are now encountering the following Problem:
The queues on our cloud servers reach (and pass) the Memory watermarks,
because of Unacknowledged Messages. The queue size of ready messages is
almost constantly on 0 (i.e., messages are delivered as fast as published)
however the unacknowledged messages keep growing.
On the other side, at the same time the statistics on our local servers
show, that messages are confirmed almost as fast as published.

This leads to the fact, that we have on the destination servers a total of
two hanful of messages that are not yet confirmed, however on the source
servers we have millions of messages that are waiting for confirmation
(acknowledgement)

We would expect with some threshold that
delivery rate on source = publish rate on destination (which is the case)
confirm rate on destination = acknowledge rate on source (which shows
considerable difference)

Does anyone have an idea or suggestion what could be the reason for this?
Is it a bad idea to have load balancer as destination in the shovel, or
should that work fine? Network issue?

Here some more details on our shovel setup:
ack_mode=on_confirm
prefetch_count=0 (default)
reconnect_delay=5


Thanks for advance for any hint or discussion point,
Regards
Claire
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20140226/48ccfd3e/attachment.html>