[rabbitmq-discuss] Losing messages in a Cluster configuration of RabbitMQ

[Upendra Sharma]

Hi all,

I am facing a weird problem of losing messages, even though I am publishing 
them as persistent messages with pika.BasicProperties(delivery_mode=2) . 
Here is the scenario:

I have two RabbitMQ brokers, namely rabbit at vm11 and rabbit at vm22;  these two 
are have been setup in a cluster configuration as shown below:
[{nodes,[{disc,[rabbit at vm11,rabbit at vm22]}]},
 {running_nodes,[rabbit at vm22,rabbit at vm11]}]

I have two clients (written using pika.BlockingConnection() ); lets call 
these clients C1 and N1.
1.) C1 creates an unique exchange (exchange name generated using UUID, say 
3fe546be8aa341b7b174b29a56e63797).
2.) C1 then spawns a thread, say C1-T1, which connects to rabbitmq server 
and waits for a response on this exchange using channel.start_consuming(). 
3.) C1 then sends a message to N1 and in the message provides the name of 
the exchange (3fe546be8aa341b7b174b29a56e63797) where N1 should send the 
response.
4.) once C1-T1 gets the response, it hands over the response to C1 and dies.

In my current setup I have 100 clients process like C1, i.e. C1, C2 ... 
C100 and one N1. 

This setup works perfectly fine when RabbitMQ is in a single node setup but 
when I put it in a cluster setup, it starts to loose the messages. What I 
mean is that Thread C1-T1 never gets a response and time-outs writing an 
ERROR in my log file. 

The trouble is I am loosing as many as 50% of the messages.

could someone tell me what could be going wrong.

thanks in advance,
-upendra

Some more Details of setup: 
RabbitMQ Server: 2.8.4
Pika Library: 0.9.5
OS: Ubuntu 12.04 (Precise)
VM11 and VM22: quad-core machines with 8GB RAM

Q.) How do I know message is lost?
A.) I performed the following steps:
1.) log the message at N1 after successfully publishing it
2.) enable trace_on on rabbitMQ server and there too log the messages

What I found that the message was sent by N1, was received by rabbitMQ 
server and RabbitMQ server reports it to be published on the 
Excahnge 3fe546be8aa341b7b174b29a56e63797  and never reports it to be 
delivered. 
But when I look at the queue it reports "0" messages -- I am pasting the 
output of rabbitmqadmin command
root at vm11:~# rabbitmqadmin list queues name node messages | grep 
3fe546be8aa341b7b174b29a56e63797
| 3fe546be8aa341b7b174b29a56e63797 | rabbit at vm22 | 0        |

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20120628/5a7bca02/attachment.htm>