[rabbitmq-discuss] RabbitMQ Producer Flow Control

[Aaron Blew]

Hey Simon,
Thanks for getting back to me.  After upgrading to 2.8.7 I'm still unable
to flow messages in to the head node faster than 4,000/sec while the
consumer is working.  I appreciate that flow control is trying to prevent
the server from being over-run, but in some situations it would be very
beneficial for an administrator to override this behavior.  Are there any
plans to add this kind of behavior?  I'm currently seriously considering a
switch to ActiveMQ/stomp since it gives me the knobs to tweak this behavior
both in real time and via configuration.

Thanks!
-Aaron


On Mon, Feb 4, 2013 at 3:30 AM, Simon MacMullen <simon at rabbitmq.com> wrote:

> Hi. There's a bug in versions of RabbitMQ prior to 2.8.4 where under some
> circumstances queues can over-prioritise getting rid of message to such an
> extent that they don't accept any more until empty.
>
> I recommend an upgrade to 2.8.7 at least, or preferably 3.0.2.
>
> And in this case flow control is just the messenger, so you don't want to
> shoot it :-)
>
> Cheers, Simon
>
>
> On 01/02/13 17:31, Aaron Blew wrote:
>
>> Hi all,
>> I'm currently using RabbitMQ 2.8.1 as the backbone of my Logstash
>> architecture.  Currently I've got 5 RabbitMQ servers running shovels
>> that push messages onto the "mothership", where data is consumed by a
>> Logstash agent that feeds into various endpoints.
>>
>> The issue is that under high load the Logstash consumer on the
>> mothership isn't fast enough at to keep up with the data coming in
>> (generally above 4,000 messages/sec).  Normally this wouldn't be a
>> problem since we've got capacity to buffer things on the mothership, but
>> RabbitMQ starts rate limiting the shovels that are sending data in,
>> which compounds the issue since the nodes shoveling data don't have as
>> much IO capacity as the mothership does.  If I stop the slow consumer on
>> the mothership, the shovels play data in as fast as they can acknowledge
>> it.
>>
>> Ideally I'd be able to turn off producer flow control on the mothership
>> side so the shovels wouldn't be slowed down except in cases where I'm
>> approaching high watermarks or something along those lines.
>>
>> Can anyone offer any suggestions on how I might be able to adjust the
>> behavior?
>>
>> Thanks,
>> -Aaron
>>
>>
>> ______________________________**_________________
>> rabbitmq-discuss mailing list
>> rabbitmq-discuss at lists.**rabbitmq.com<rabbitmq-discuss at lists.rabbitmq.com>
>> https://lists.rabbitmq.com/**cgi-bin/mailman/listinfo/**rabbitmq-discuss<https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss>
>>
>>
>
> --
> Simon MacMullen
> RabbitMQ, VMware
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20130204/5ceffeb2/attachment.htm>