[rabbitmq-discuss] RabbitMQ and batch processing

Sun May 18 19:55:39 BST 2014

I mentioned this on Twitter and a couple of people have requested that I
bring this up on the mailing list.

It seems to be a given that RabbitMQ was not designed for the batch
processing use case (i.e. using RabbitMQ as a buffer between large serial
steps). We have a system in place that attempts to do just that, however.

I have been working with the developers of the software involved in an
attempt to help them redesign around a more ideal use of RabbitMQ (or to
help them move to a different bus altogether -- database or something like
kafka) and some of them have been able to simply operate in smaller batch
sizes (thus keeping their queues relatively small).

However, I cannot stem the tide of improper RabbitMQ use.

When things go poorly, millions of messages end up in the queues.

In 3.1.x we saw this regularly cause our clusters to partition.

In 3.1.x and 3.2.x when we would delete large queues (5+ million messages
enqueued), this would cause the cluster to become unresponsive, run out of
memory, and then crash.

During the 3.1 -> 3.2 upgrade, we had to completely rebuild our clusters.
When 3.2 came up, it soon crashed.

In the most recent upgrade, we saw a 3.2.3 cluster in our dev environment
crash. I performed an opportunistic upgrade to 3.3.1, because hey...
downtime already, so let's see if 3.3.1 addresses some of the issues we've
been seeing.

https://gist.github.com/grepory/384410ac90186ed0ce2a

After the upgrade, 3.3.1 would not startup at all. I removed
/var/lib/rabbitmq/mnesia on all of the nodes and brought RabbitMQ back up.

3.3.1 has been up and running alright so far, but we haven't done another
end-to-end test in our development environment in a while. One of these
tests can lead to at least a million messages in the queue over a period of
time on average.

So, I guess my question is:

If I know that I have people using RabbitMQ like this, and there is nothing
I can do to change that fact... what do I do?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20140518/071d73df/attachment.html>