[rabbitmq-discuss] Losing messages in a Cluster configuration of RabbitMQ

[Ask Solem]

On 28 Jun 2012, at 22:24, Upendra Sharma wrote:

> Hi all,
> 
> I am facing a weird problem of losing messages, even though I am publishing them as persistent messages with pika.BasicProperties(delivery_mode=2) . Here is the scenario:
> 
> I have two RabbitMQ brokers, namely rabbit at vm11 and rabbit at vm22;  these two are have been setup in a cluster configuration as shown below:
> [{nodes,[{disc,[rabbit at vm11,rabbit at vm22]}]},
>  {running_nodes,[rabbit at vm22,rabbit at vm11]}]
> 
> I have two clients (written using pika.BlockingConnection() ); lets call these clients C1 and N1.
> 1.) C1 creates an unique exchange (exchange name generated using UUID, say 3fe546be8aa341b7b174b29a56e63797).
> 2.) C1 then spawns a thread, say C1-T1, which connects to rabbitmq server and waits for a response on this exchange using channel.start_consuming(). 
> 3.) C1 then sends a message to N1 and in the message provides the name of the exchange (3fe546be8aa341b7b174b29a56e63797) where N1 should send the response.
> 4.) once C1-T1 gets the response, it hands over the response to C1 and dies.
> 
> In my current setup I have 100 clients process like C1, i.e. C1, C2 ... C100 and one N1. 
> 
> This setup works perfectly fine when RabbitMQ is in a single node setup but when I put it in a cluster setup, it starts to loose the messages. What I mean is that Thread C1-T1 never gets a response and time-outs writing an ERROR in my log file. 
> 
> The trouble is I am loosing as many as 50% of the messages.
> 

Rabbitmq delivers messages in round-robin, so when you say 50% of the messages
are lost it sounds like too much of a coincidence to me.

What can often happen is having a broken consumer process that just sits
there stealing messages, you can verify that by looking at the `rabbitmqctl list_queues`
output:

    $ sudo rabbitmqct list_queues -p $your_vhost name messages_ready messages_unacknowledged consumers

If you have more consumers and unacknowledged messages than you expected then
you probably have a stale consumer process that you have to kill.