[rabbitmq-discuss] direct queue throughput

[Charity Majors]

Whoops, sorry -- I'm not getting 7k msgs/sec through rabbitmq.  I'm
getting closer to *2k*/sec through rmq.

7k/sec is the mysql query count.  But we serialize the queries into
sequences and submit those sequences into the worker-jobs queue, to
better mimic the way the queries were originally executed.

> 1. RabbitMQ is written in Erlang. I am pretty sure the latest stable
> releases are still a single-threaded process (though internally the
> application uses hundreds or thousands of "green" threads). Therefore
> you won't get much throughput increase by running on a larger multi-
> core box.

Good to know.  The CPU load does seem awfully low.

> 3. Are you certain that your worker processes are keeping up? It's
> possible you are only seeing 7k msgs/sec because your worker processes
> aren't consuming them fast enough (ie. back pressure on the queue).

Yeah, I'm sure.  I can run rabbitmqctl list_queues and look at the
message counts, and the consumers are keeping the queues basically
clear.

> 4. It sounds odd that you have higher than 7k queries/sec in your
> environment. Is your mysql system really capable of returning every
> query within 140 microseconds? I'm probably just misunderstanding
> something here so feel free to clarify what you mean by 7k queries/sec.

Yeah.  This wasn't really a problem until I started testing the SSDs.
With the SSDs and my current test setup, I can push the QPS up close
to 10k queries per second (according to innotop and my ganglia
graphs), and the box is still basically just yawning at me.  It's
still only using like 20% of the IO.

It's a nice problem to have, but still ... :)