[rabbitmq-discuss] Strange message loss behavior with mirrored queues

[Matt Pietrek]

Hi Emile,

Thanks much for the reply. I certainly understand that in a mirrored
setup, message publishing will be slower. However, I still think there
some sort of issue here.

I could understand that I'd lose messages if I rapidly published 100,000
messages to the broker and then abruptly killed the broker. However,
that's not what happening. My app is writing the messages and then cleanly
existing without any errors, and nothing "bad" is happening to the broker.

While I would expect it might longer for the broker to catch up and
eventually process all 100,000 messages, it doesn't. Instead, it reports
that the queue has about 25,000 message (varies from run to run). I've
waited for 10 minutes and never seen any more messages appear in the queue.

Conversely, if I run the identical scenario without mirroring, all 100,000
message appear in the queue within seconds of my app exiting.

Also, I understand that with publisher-confirms I can wait till I see
100,000 messages written, but I wouldn't expect that *not* using them, and
simply exiting from my client process would cause messages to be lost. As
far as I can tell, all messages were sent on the wire to the broker, and
barring an error state in the broker, should eventually appear in the
queue.

I'll follow up offline with my exact code and config.

Thanks,

Matt

On 2/6/12 2:43 AM, "Emile Joubert" <emile at rabbitmq.com> wrote:

>Hi Matt,
>
>On 03/02/12 18:57, Matt Pietrek wrote:
>> I'm writing some basic throughput tests to verify that RabbitMQ will
>> scale to our needs in HA environments. In one scenario I'm finding
>> substantial message loss.
>
>I wouldn't describe what you're seeing as message loss, rather a
>demonstration that publishing in a HA environment with more nodes is
>slower. Publishing is slower because it takes longer to send the
>messages to a greater number of nodes in the cluster.
>
>You should make use of publisher confirms
>(http://www.rabbitmq.com/extensions.html#publishing)
>to make sure that published messages have been safely received by the
>broker.
>
>
>-Emile