[rabbitmq-discuss] RabbitMQ Shovel Connections
alexandru at rabbitmq.com
Mon Nov 7 10:50:54 GMT 2011
Looking at report, I see the following:
* there's one channel with 1524 unconfirmed messages:
<'rabbit at server02'.3.374.49> <'rabbit at server02'.3.366.49> 1 guest / false false 1 1524 0 0 0 0 false
These are messages that were sent to a consumer but not ack'd (so
they can't be delivered to other consumers);
* the broker currently has 1524 messages sitting on two queues:
<'rabbit at server02'.3.241.0> number_3 true false  0 1524 1524 1 ...
<'rabbit at server02'.3.244.0> number_6 true false  1524 0 1524 0
Both queues have 1524 messages sitting on them, but the kinds of
messages are different. The messages on number_6 are all ready,
which means that once a consumer connects to the queue, they will
all be delivered (but no consumer was connected to the queues). The
messages on number_3, on the other hand, are unacknowledged; so,
they were delivered to the sole consumer, but it didn't acknowledge
This is consistent with what you describe, since a broker restart
will mark the unacknowledged messages as ready again, and deliver
them to a consumer.
This means that the consumer for number_3 has received messages, but
hasn't acknowledged them. This can only happen if you set ack_mode in
the shovel configuration:
* if ack_mode is on_publish, the shovel is having trouble forwarding
consumed messages to their destination;
* if ack_mode is on_confirm, the destination is not confirming the
messages the shovel sent it.
The former could be caused by a network issue (the logs list a
connection failing with a timeout), while the latter
shouldn't be possible (the RabbitMQ broker guarantees it will eventually
confirm all received messages).
So, to fix this, you could try:
* playing around with the ack_mode setting;
* upgrading (the other poster mentioned this fixed his problem).
What versions of RabbitMQ are you running (both on the source and
destination brokers)? Is there any chance you could send us your shovel
config? (or at least describe what it does)
On Wed, Nov 02, 2011 at 06:31:34AM -0700, Jelle Smet wrote:
> Hi Alex,
> The requested output is in attachment. I have string replaced server and
> queue names with fictional names.
> I could find following non INFO report in the rabbit at server02.log file:
> ... snip ...
> =WARNING REPORT==== 2-Nov-2011::13:56:10 ===
> exception on TCP connection <0.24639.172> from 10.96.72.91:58172
> ... snip ...
> Issuing "rabbitmqctl stop" gave following output and kept hanging endless:
> Stopping and halting node 'rabbit at server02' ...
> When doing a strace -p on the open running rabbitmq process I get following
> output for each one of them:
> #strace -p 26367
> Process 26367 attached - interrupt to quit
> select(0, NULL, NULL, NULL, NULL
> The only thing left at this stage is kill all processes and restart server02
More information about the rabbitmq-discuss