[rabbitmq-discuss] direct queue throughput
cremes.devlist at mac.com
Fri Aug 28 23:38:58 BST 2009
On Aug 28, 2009, at 5:16 PM, Charity Majors wrote:
> Hi. I'm working on a distributed load testing tool for mysql, where
> we capture several hours worth of peak query load and replay the
> queries against different db versions and configurations. Rabbitmq is
> the pipeline between the central aggregator process and the workers.
> I have (essentially) two queues set up, one for farming query
> sequences out to workers and one for sending mysql return codes back
> to the central. Both queues are non-durable and use a direct
> So far, so good. But it's starting to look like this configuration
> isn't fast enough to let me test SSDs. Rabbitmq appears to top out at
> a throughput rate of around 7k msgs/sec for the two queues. Does that
> number sound right? The rate is the same whether I'm running on an 8
> core box with 32 gigs of memory or a 4 core with 4 gigs.
I don't have an answer for you but I do have a few things to point out.
1. RabbitMQ is written in Erlang. I am pretty sure the latest stable
releases are still a single-threaded process (though internally the
application uses hundreds or thousands of "green" threads). Therefore
you won't get much throughput increase by running on a larger multi-
2. You don't really describe what kind of data stream the queues
carry. A fanout exchange has higher throughput than a direct exchange
since it does fewer routing checks. That may or may not be a more
appropriate choice for you.
3. Are you certain that your worker processes are keeping up? It's
possible you are only seeing 7k msgs/sec because your worker processes
aren't consuming them fast enough (ie. back pressure on the queue).
4. It sounds odd that you have higher than 7k queries/sec in your
environment. Is your mysql system really capable of returning every
query within 140 microseconds? I'm probably just misunderstanding
something here so feel free to clarify what you mean by 7k queries/sec.
I hope some of this is useful to you.
More information about the rabbitmq-discuss