[rabbitmq-discuss] Ensuring low latency for publishers

[Simon MacMullen]

On 23/02/12 20:27, Eric wrote:
> Well... I would find it useful.  And the person in the thread I linked
> to would have found it useful as well.

Sure, I'm trying to gauge how widespread such a demand is. It looks like 
we could plausibly do something in this direction.

> It's pretty tough to go into a production environment with the
> statement "Publishing should be fast unless the broker is throttling.
> If the broker is throttling we won't be able to tell, we don't have a
> good way of knowing beforehand, and we don't know how long the threads
> will be blocked".  It's useful in any case where an application would
> rather not publish the message rather than waiting to publish it
> indefinitely.

Yeah, the problem is that we've historically optimised for Never 
Throwing Away Messages Ever Ever...

> The net result is that a struggling broker will cause a chain
> reaction; publishers will block indefinitely, which may bring the
> publishing application to a halt.  In that case I'd like to abandon
> the publishing process, but it's quite impossible at that point - it's
> too late, the broker is already throttling, and there's no timeout.
> The thread could block for seconds, or more.  I'm concerned about not
> throwing the message away, but I'm more concerned about being able to
> isolate services from RabbitMQ broker events.

If it's any consolation the flow control stuff that's coming in 2.8.0 
will make it much harder to make the memory alarm go off, so blocking 
for seconds should be much less likely.

> One more question along this line.  The flow control document says "By
> default, when the RabbitMQ server uses above 40% of the installed RAM,
> it raises a memory alarm... Connection heartbeat monitoring gets
> disabled too."  If I set up a connection heartbeat on the shared
> connection, will the heartbeat detect a broker alarm that exceeds the
> interval and disconnect?  A disconnect would terminate the publisher
> channels and free things up.

No - the whole point of disabling heartbeats is to make sure connections 
stay up when this happens...

Cheers, Simon

-- 
Simon MacMullen
RabbitMQ, VMware