[rabbitmq-discuss] RabbitMQ Crashed! HA queue disappeared, all messages with delivery_mode=2 are gone!

[Simon MacMullen]

Do you have any logs?

Cheers, Simon

On 06/11/13 16:23, Sergey Dobrovolskiy wrote:
> Hi everybody,
>
> I'm testing RabbitMQ and can't get it to work properly!
>
> So that's what I have:
>
> two c1.xlarge nodes running on EC2. One Ram node and one Disk node.
>
> I created a test queue. setted up it as a HA mirrored queue like this:
>
>     rabbitmqctl set_policy -p / HA "^ha.*" '{"ha-mode":"all",
>     "ha-sync-mode":"automatic"}'
>
>
> Then ran this command to see if everything worked well:
>
>     [root at rabbita ~]$ rabbitmqctl list_queues name messages pid
>     slave_pids synchronised_slave_pids
>     Listing queues ...
>     ha.api.postworker    0    <'rabbit at rabbit-master'.1.853.0>
>     [<rabbit at rabbita.2.1122.0>]    [<rabbit at rabbita.2.1122.0>]
>
>
> After this I was expecting that all the messages that will get to the
> queue will be automatically synchronised (mirrored) between all nodes.
> In my case there is only one additional node. So I would expect that
> even if one node goes down for some reason, another node would have this
> queue defined and all the messages will still be there. NOTE, I was
> sending messages with delivery_mode set to 2. Meaning they are durable
> and will be saved to disk!!!!
>
> Than the actual test:
>
> 1 producer - was sending messages messages at a rate about 2000 msgs/s.
> Each message is about 1kB long.
>
> Saw that Rabbit setted "Flow control" periodically.
>
> I tried to fill up the queue up to some point. So at the time when I
> added consumers, I had about 200 k msgs in the queue
>
> had 6 consumers:
>
> Each where using basic_consume()
> with prefetch count = 3000
>
> There where about 4k unacknowledged messages  constantly.
>
> THEN BOOM!
>
> RAM node went down and ALL THE MESSAGES and even Queue where deleted,
> they just disappeared.
> here is the screenshot from UI:
>
> You can see that there where 150K messages and then suddenly, 0. + all
> consumers are gone, because they can't connect to the Q.
> Also NOTE that even that Disk node  (rabbita) is Red, it will be
> functioning after crash, but the node above, RAM node, will crash.
>
> <https://lh6.googleusercontent.com/-z0jfOUTy5eA/UnpsY_34yHI/AAAAAAAAABA/EJQLb5ou7wA/s1600/rabbitmq.png>
>
>
>
> _______________________________________________
> rabbitmq-discuss mailing list
> rabbitmq-discuss at lists.rabbitmq.com
> https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
>


-- 
Simon MacMullen
RabbitMQ, Pivotal