[rabbitmq-discuss] Shovel stops receiving acks from cluster

Wed Aug 15 15:00:54 BST 2012

Hi, thanks for the reply.

This is the shovel config on the broker (with credentials and hostnames replaced).
The broker nodes in destination are the cluster nodes, and source is the local broker
I'm shoveling messages from.

{foo_shovel, [
  {sources, [{broker, "amqp://localhost?heartbeat=5"}]},
  {destinations, [
    {brokers,
      ["amqp://user:pass@rabbit1?heartbeat=5", "amqp://user:pass@rabbit2?heartbeat=5"]
    }
  ]},
  {reconnect_delay, 5},
  {queue, <<"foo">>},
  {prefetch_count, 1000},
  %{ack_mode, on_publish},
  {publish_fields, [
    {exchange, <<"foo">>}
  ]}
]}

The problem appears when I shut down and then start rabbit1 or rabbit2 (using either
the debian init script or rabbitmqctl stop_app). All servers are VMs, and I've also
reproduced this problem by forcing a shutdown of the VM and then booting it up again.

I've checked the shovel status in the management interface and it always shows as
"running" even when the problem appears.

On a related note, when letting the broker sit idle for a while (approx. 24 hours by now)
the shovel seems to disconnect from the destination without reconnecting even with
messages still in the source queue (all of them shows as "ready" with none "unacked").

________________________________________
From: rabbitmq-discuss-bounces at lists.rabbitmq.com [rabbitmq-discuss-bounces at lists.rabbitmq.com] on behalf of Francesco Mazzoli [francesco at rabbitmq.com]
Sent: Wednesday, August 15, 2012 15:32
To: Discussions about RabbitMQ
Subject: Re: [rabbitmq-discuss] Shovel stops receiving acks from cluster

At Wed, 15 Aug 2012 14:19:41 +0100,
Francesco Mazzoli wrote:
> At Tue, 14 Aug 2012 12:27:26 +0000,
> Jon Bergli Heier wrote:
> > I have a cluster consisting of two nodes, and brokers shoveling messages to
> > the cluster using the shovel plugin. When testing failover with the shovel
> > plugins (by stopping the RabbitMQ nodes) I often see that when starting a node
> > after stopping it causes the shovel plugin to stop receiving acks for
> > messages. I have prefetch_count set to 1000 in the shovel config, and I see
> > that "unacked" in the management interface for the queue rises to 1000 and
> > sits there, with "ready" steadily increasing.  The connection seems to be
> > alive as I can see (via tcpdump) that there are heartbeats being sent.  All
> > RabbitMQ servers are 2.8.5 on Erlang R14A. This only seems to happen with
> > setting ack_mode set to on_confirm. The queues on the cluster are mirrored
> > using x-ha-policy: all.
>
> Could you post more details about your shovel config and your HA cluster?  In
> particular, how do you handle failover?  Do you have the cluster nodes all as
> sources in the shovel config?

Also, you might want to check the shovel status,
<http://www.rabbitmq.com/shovel.html#management-status>.

--
Francesco * Often in error, never in doubt
_______________________________________________
rabbitmq-discuss mailing list
rabbitmq-discuss at lists.rabbitmq.com
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss