[rabbitmq-discuss] Shunting messages from one queue to another
tom.anderson at timgroup.com
Wed Jul 3 11:33:33 BST 2013
We have a queue which is used to send messages to an app, which then
processes them, and inserts some data into a database. Perhaps people
will have seen similar setups before :). Sometimes, the app cannot
process the message for some reason, and will use basic.reject with
requeue=false to discard it. We have a dead-letter queue, so discarded
messages end up there.
Sometimes (actually, always), failures to process are not really due to
errors in the messages, but in the processing code in the application.
For example, we had a case where a zero in a particular field in a
message led to an infinity after some calculations, which then caused
database inserts to blow up. As another, we have an ongoing epidemic of
database deadlocks causing transient failures to insert.
Our response to these problem is to fix the bug in the processing code,
and then re-run the messages. We do this by prodding the upstream
message-sending system to repeat all the relevant messages. This is
manual and error-prone (it involves deleting documents from a MongoDB
server by hand, then bouncing an app server >_<), and also rather
What i would like to do instead is move the discarded messages from the
dead-letter queue back to the main queue. In fact, we may want to move
messages from the dead-letter queue to a holding queue, and from there
back to the main queue, so that an infinite loop is never possible. What
is the best way to do this?
We could write code of our own that talks to RabbitMQ to move the
messages. I would like to avoid doing this if possible, because every
line of code we write is a line of code we can get wrong.
We could use the shovel plugin to do this - in some ways this seems
ideal. However, shovels are configured in RabbitMQ's configuration files
as essentially permanent features, which we don't want; we want to be
able to run a shovel-like process just once, rather than constantly.
Also, shovels seem to be geared towards moving messages between queues
in different brokers, whereas in our case, both queues are in a single
broker. Would it still be sensible to use a shovel?
If not, are there any other utilities in the RabbitMQ ecosystem which we
could use for this?
Thanks in advance,
Tom Anderson | Developer | +44 20 7826 4312 | timgroup.com
STATEMENT OF CONFIDENTIALITY: The information contained in this
electronic message and any attachments to this message are intended for
the exclusive use of the addressee(s) and may contain confidential or
privileged information. If you are not the intended recipient, please
notify Tom Anderson at TIM Group at tom.anderson at timgroup.com and
destroy all copies of this message and any attachments.
TIM Group is the trading name for YouDevise Limited. YouDevise Limited
is registered in England, No. 3331176. Registered office: 3 Copthall
Avenue, London, EC2R 7BH.
-------------- next part --------------
An HTML attachment was scrubbed...
More information about the rabbitmq-discuss