[rabbitmq-discuss] Disable Per-Connection Flow Control

[Simon MacMullen]

On 13/10/12 02:16, Dwayne Campbell wrote:
> Simon,
>
> Thank you very much for your reply.

No problem!

> Using HIPE compilation reduced the flow control significantly.I now get
> flow controlled at around 4.1 MB/S to 7.2 MB/S .However flow control
>   seems to steadily remain on at 3.8MB/s for the rest of the time
> RabbitMQ runs.

When sending tiny messages almost all of your costs are per message 
rather than per byte - so it makes more sense to think in terms of the 
number of messages / second rather than bytes / second.

> Each message is about approximately 300 bytes at most 800 bytes
>
>
> Multiple connections(independent applications) will be made from the
> same host to the RabbitMQ Server. Each connection will be sending small
> messages really fast to the broker(RabbitMQ Server)____

I'm not currently sure about whether you've been describing the 
behaviour of a live system or some tests you've been running - if you've 
been describing tests then bear in mind that RabbitMQ will exhibit a 
reasonable amount of parallelism - so just testing with one connection 
will not tell you the total message rate you can handle.

> What optimizations can be made to increase throughput on RabbitMQ Server
> or the routing part of the pipeline ?

It sounds like you're CPU bound in routing - so the next step is to 
ensure your routing is as simple as possible. Exchange type is the 
primary factor here. Headers is (a lot) slower than topic, topic is 
slower than direct and direct is slower than fanout (except in the 
default direct exchange, which is as fast as fanout).

> Would cluster be the next option to evaluate ?

Yes, it should scale pretty much linearly for this workload.

> Is there any output that i can gather to analyze which part of
> the pipeline is the bottleneck ?

Not at the moment within RabbitMQ; future releases might expose more 
details about this. But in the tiny-non-persistent-messages (I think 
your messages are not persistent?) case it's usually routing, with 
framing coming a close second.

Cheers, Simon

-- 
Simon MacMullen
RabbitMQ, VMware