[rabbitmq-discuss] Weird Crash - Recovery logic for durable messages/queues/exchanges?

Fri Aug 7 11:05:12 BST 2009

Darien,

Darien Kindlund wrote:
> Okay, your wording is a little vague, so I want to be crystal clear.
> Assume we have a single RabbitMQ server, with 3 consumers, where all 3
> are consuming messages off the same durable queue (with durable
> messages).

(minor quibble on terminology) resources can be made *durable*, messages 
can be made *persistent*.

> When all 3 start processing their respective messages RabbitMQ marks
> all 3 durable messages as 'unacknowledged'.  Then, let's assume
> RabbitMQ crashes (for some reason or another).  *STICKING POINT: Upon
> crash, all 3 consumers channels and connections have terminated -- I
> assume there's no way for any of the consumers to "reuse" their
> existing channels/connections because RabbitMQ server died.*

Correct.

> Therefore, when a sysadmin restarts RabbitMQ and the persister is
> recovered, will all 3 messages be marked 'ready' ?  Or will all 3 be
> marked 'unacknowledged' ?   Sorry to be pedantic, but your original
> reply was slightly unclear about this.

They will be marked as ready, unless some consumers have managed to 
connect again already and the messages have been sent out to them, in 
which case they are counted as 'unacknowledged'.

>> So what you are seeing is rather strange. Are you sure there aren't any
>> connected consumers?
> 
> I'm sure there are no connected consumers -- although I assume that
> when RabbitMQ crashes, all consumer channels/connections are
> terminated as well.  For good measure, I also had to terminate and
> restart epmd... otherwise, RabbitMQ would not start up properly via
> '/etc/init.d/rabbitmq start'.  FYI, this is on a stock Ubuntu
> distribution.

Your consumers aren't by any chance containing some reconnecting logic 
that tries to connect to the server again whenever a connection has been 
dropped? I seem to recall reports that, for example, the ruby client 
does something like that. Please check the server log to be sure - it 
contains a record of all connects/disconnects.

> As an interesting side case, is there any way to manually reset
> un-ack'd messages back into the ready state while RabbitMQ is running
> (and consumers/producers are active?).

Once the server has delivered a message to a consumer, it will only 
become available again to other consumers when then recipients 
channel/connection is closed.

> I'm trying to avoid having to shutdown the RabbitMQ server and
> obliterate the nmesia persister log in order to clear out these
> messages.

That would remove all messages completely, rather than just making the 
unacknowledged messages available to other consumers again. If the 
former is really what you want then the 'queue.purge' command is your 
friend.

Regards,

Matthias.