[rabbitmq-discuss] Memory leak when using shovels?

Tue Dec 20 19:20:26 GMT 2011

Travis,

On 20/12/11 16:42, Travis wrote:
> Are there any known memory leaks when using shovels with 2.6.1?

Nope.

> We're seeing a slow leak that grows over 6-8 days on a few instances of
> rabbitmq that are using the shovel plugin and we're not exactly sure
> why.  I'm not sure if this is related to the memory leak we previously
> reported a few weeks ago that was fixed in 2.7.1

Is the shovel publishing to mirrored queues? If so then then it's 
probably the same problem.

> The shovel config (and rabbit config) we're using is:
>
>
> {rabbitmq_shovel,
>     [{shovels,
>       [{engine_my_prod_shovel,
>         [{sources,      [{brokers,
> ["amqp://my-user-prod:rabbitmq-my-prod@localhost/my-prod"]},
>                          {declarations,
>                             ['queue.declare',
>                              {'queue.bind',
>                                     [{exchange,<<"my-exchange">>},
>                                      {queue,<<"my-queue">>}]}
>                             ]}]},
>          {destinations, [{brokers, [
>
> "amqp://my-user-prod:rabbitmq-my-prod@aus-svc3/my-prod?heartbeat=5",
>
> "amqp://my-user-prod:rabbitmq-my-prod@aus-svc4/my-prod?heartbeat=5"
>                              ]},
>                          {declarations,
>                             [{'exchange.declare',
>                                     [{exchange,<<"my-exchange">>},
>                                      {type,<<"direct">>},
>                                      durable]},
>                              {'queue.bind',
>                                     [{exchange,<<"my-exchange">>},
>                                      {queue,<<"my-queue">>}]}
>
>                             ]}]},
>          {queue,<<"my-queue">>},
>          {prefetch_count, 1000},
>          {ack_mode, on_confirm},
>          {publish_properties, [{delivery_mode, 2}]},
>          {publish_fields, [{exchange,<<"my-exchange">>},
>                            {routing_key,<<"">>}]},
>          {reconnect_delay, 5}
>         ]}
>        ]
>      }]
> }
> ].

Two observations:

1) there is a stray 'queue.declare' in the first shovel config. This 
will cause a new queue to be created every time the shovel connection is 
interrupted. You would surely have seen those queues in the management 
UI and rabbitmqctl, so I doubt that's the cause of the leak.

2) Instead of the establishing a connection to 'localhost', specify no 
host at all, e.g.
"amqp://my-user-prod@/my-prod"; that way a more efficient and less fault 
prone direct connection is used instead of a network connection (which 
is also why no password is required).

Regards,

Matthias.