[rabbitmq-discuss] RabbitMQ Crashed! HA queue disappeared, all messages with delivery_mode=2 are gone!

[Sergey Dobrovolskiy]

Hi everybody,

I'm testing RabbitMQ and can't get it to work properly! 

So that's what I have: 

two c1.xlarge nodes running on EC2. One Ram node and one Disk node. 

I created a test queue. setted up it as a HA mirrored queue like this:

rabbitmqctl set_policy -p / HA "^ha.*" '{"ha-mode":"all", 
> "ha-sync-mode":"automatic"}'
>

Then ran this command to see if everything worked well:

[root at rabbita ~]$ rabbitmqctl list_queues name messages pid slave_pids 
synchronised_slave_pids
Listing queues ...
ha.api.postworker    0    <'rabbit at rabbit-master'.1.853.0>    
[<rabbit at rabbita.2.1122.0>]    [<rabbit at rabbita.2.1122.0>]


After this I was expecting that all the messages that will get to the queue 
will be automatically synchronised (mirrored) between all nodes. In my case 
there is only one additional node. So I would expect that even if one node 
goes down for some reason, another node would have this queue defined and 
all the messages will still be there. NOTE, I was sending messages with 
delivery_mode set to 2. Meaning they are durable and will be saved to 
disk!!!!

Than the actual test:

1 producer - was sending messages messages at a rate about 2000 msgs/s. 
Each message is about 1kB long. 

Saw that Rabbit setted "Flow control" periodically.

I tried to fill up the queue up to some point. So at the time when I added 
consumers, I had about 200 k msgs in the queue

had 6 consumers:

Each where using basic_consume() 
with prefetch count = 3000

There where about 4k unacknowledged messages  constantly.

THEN BOOM!

RAM node went down and ALL THE MESSAGES and even Queue where deleted, they 
just disappeared. 
here is the screenshot from UI:

You can see that there where 150K messages and then suddenly, 0. + all 
consumers are gone, because they can't connect to the Q.
Also NOTE that even that Disk node  (rabbita) is Red, it will be 
functioning after crash, but the node above, RAM node, will crash.

<https://lh6.googleusercontent.com/-z0jfOUTy5eA/UnpsY_34yHI/AAAAAAAAABA/EJQLb5ou7wA/s1600/rabbitmq.png>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20131106/bcb240d2/attachment.htm>