[rabbitmq-discuss] 100% CPU usage and system crawls to a halt

[Matthew Sackman]

Hi Mark,

On Wed, Oct 21, 2009 at 08:57:02PM -0500, Mark Nutter wrote:
> A quick overview of what I have it doing: I'm taking a firehose of Twitter
> statuses and throwing every single one of them into a main queue from which
> they are thrown into a mySQL database. These statuses also get routed to
> other smaller queues based on different routing keys like twitter user ids
> and URLs. These messages come in fast, about 1000 per 10 seconds, and we
> pull the messages off the queue about as fast as they come in.

Are you setting QoS, and are you using Basic.Get or a subscription to a
queue? The latter will be better. Also, if you don't care about losing
messages, make sure you're not setting the persistent flag on the
publishes, and you may wish to set NoAck when pulling messages off the
queue. If you're not setting NoAck, make sure you are acking messages as
you receive them.

Finally, what type of exchange are you using? Topic exchanges are much
slower (especially with large numbers of bindings) than fanout.

> I've seen a few threads discussing the 100% CPU usage problem, but no real
> solutions.  Am I going about this the wrong way?  What might be causing this
> problem.  Is 1.7 a better option and is there a stable package out there
> that works on Hardy?  Thanks in advance for your input.

We always recommend using our latest and greatest release, but I'm not
sure if it's available for Hardy yet. I would also suggest that given
the current release of Erlang is R13B02 and the version in Hardy is
R11B5, you're somewhat behind the times there too.

Generally, if you have the logic in your consumers to be able to drop
messages out later on, then we tend to suggest that using a fanout
exchange to multiple queues is much better than using a topic exchange
with many many bindings. We do have plans to improve the efficiency of
the topic exchange, but it's a little way off just yet (unless people
wish to pay us to do it!).

On modern hardware, no persistence, fanout exchange, a real consumer
(not Basic.Get) and setting NoAck, we regularly see around 20kHz,
depending on the message size. Now obviously, that'll change easily with
a billion different variables, but given your message rate of 100Hz, I
would be surprised if you can't make it work.

Please let us know if you have any further salient details!

Matthew