[rabbitmq-discuss] Extremely uneven distribution of latencies under high load

[Matthias Radestock]

Eugene,

On 01/08/11 12:26, Eugene Kirpichov wrote:
> I'm publishing about 6000 messages/s, each about 10kb in size, in
> roundrobin to these 4 nodes. I have publisher confirms turned on.
> Autoack is turned off; I have a cycle of "get, process, publish
> result, ack". Each message takes ~800ms to process.

Where in the cycle do you wait for / process the publisher confirm?

Also, if you have a publishing rate of 6000Hz, but a consuming rate of 
1/0.8 = 1.25Hz, then you'll need thousands of consumers to keep up. 
Otherwise the number of messages in / disk usage of rabbit will just 
keep growing.

> I'm experiencing things like "rabbitmq not giving a message to a
> consumer for several minutes", i.e. a large portion of my cluster is
> idle waiting for messages.

I suggest you use the management plug-in to get a better insight into
what rabbit is doing. Also look at the rabbit logs for anything unusual.

> RabbitMQ is also rather frequently dropping connections but I'm
> reconnecting.

RabbitMQ will only drop connections for the following reasons:

- an error occurred, in which case an error message sent to the client
(which you should see there and in the rabbit logs) and a connection
shutdown handshake sequence is initiated

- heartbeats are enabled and a client didn't send anything (including
heartbeat frames) for a while; there should be a message in the logs for
when that happens

Any other connection drops will be due to the client, network,
firewalls, etc.

> I also graphed the publish confirmations from RabbitMQ (actually the
> number of unack'd messages remaining)

Those two are completely different things. Confirms relate to messages
published by the client to the broker, acks relate to messages delivered
by the broker to clients.


Regards,

Matthias.