[rabbitmq-discuss] Fairness wrt queues under heavy load

[Eugene Kirpichov]

Hi Matthias,

On Tue, Oct 11, 2011 at 12:09 PM, Matthias Radestock
<matthias at rabbitmq.com> wrote:
> Eugene,
>
> On 07/10/11 16:30, Eugene Kirpichov wrote:
>>
>> I've got a large cluster with many consumers, and several producers
>> who publish a very intense steady stream of messages in
>> publish/confirm mode.
>>
>> Some components occasionally submit a single but very important and
>> urgent message in transactional mode, which takes up to 5-10 minutes
>> to be delivered.
>>
>> Is this expected behavior?
>
> Depends.
>
>> How can I avoid these obscenely large delays?
>
> Most likely due to backlogs. The cure depends on where the backlog is. Let's
> follow the path of a message from producer to consumer...
>
> If the producer publishes at a high rate then backlogs will arise in the
> client buffers, network buffers, server-side network buffers and buffers
> inside rabbit. These backlogs can be substantial, though unlikely minutes
> worth. The way to avoid them is to ensure that high-priority messages are
> published on a separate connection from ordinary messages.
>
> If the publishing rate exceeds the rabbit capacity, the broker will
> eventually hit its memory threshold and throttle producers. The logs will
> tell you when that happens. Memory pressure can take a while to dissipate,
> sometimes several minutes. The cure for this is to set up a bigger rabbit.
This could be the case, actually, though I'm limiting the number of
unconfirmed publishes on each queue, but I don't limit it on a global
level. Thanks, I'll look at the logs.

What somewhat disproves this is that this *only* happens with
transactional messages (establish channel, start transaction, publish
message, commit, close channel - though the connection may be shared).

>
> Messages pile up in queues when consumers don't consume them fast enough.
> Since queues are FIFO structures that manifests itself as a delay. To avoid
> this delay for high priority messages, make sure they get routed to a
> separate queue and code consumers s.t. they consume from that queue in
> addition to the regular queues.
I am routing to a separate queue already.

>
> At the consuming end messages can pile up in network buffers and, depending
> on the client library and its use, in client-side buffers. To avoid that set
> a basic.qos prefetch limit on the consuming channels.
I have prefetch = 1.

>
>
> Regards,
>
> Matthias.
>



-- 
Eugene Kirpichov
Principal Engineer, Mirantis Inc. http://www.mirantis.com/
Editor, http://fprog.ru/