[rabbitmq-discuss] direct queue throughput

[Charity Majors]

> Erlang has supported SMP for several releases, I believe first gaining
> support about 4 years ago. Initially, yes, we found that very early
> versions of Rabbit ran slower with the SMP support turned on that off.
> However, Erlang's support for SMP has improved substantially over the
> years and Rabbit benefits substantially from Erlang's SMP support.

Oh!  Hmm, okay, even better to know.  :)

> The rest of Chuck's points are absolutely valid though - we really need
> a fair bit more information - which clients are you using, what is the
> sustain network load, are the consumers, auto-acking, manually acking,
> is qos being used?

I'm using python-amqplib.  There's no flow control, so I have an
option implemented that works kind of like a poor man's throttle.  We
already keep state on how many jobs are running or waiting at a given
time, so our job minder thread doesn't submit any more jobs to the
queue if there's more than a certain number already running or waiting
to execute.

This works pretty well.  With the SSDs, however, it runs so fast that
I don't even have to turn on the throttle.  The workers consume the
jobs, execute the SQL, and return their status faster than the
publisher can stuff jobs into the queue.

We aren't acking at all.

Network load ... I would have to check on that.  But I don't think
that's the ceiling we're bumping up against.  I can run it with a null
plugin, so it just pushes the jobs through the rabbitmq queues and
fakes the mysql responses, and everything stays on the same box.  And
my throughput rate looks identical to the rates I see testing the
SSDs.

Does that answer your questions?

Thanks,
charity.