[rabbitmq-discuss] Architecture Question

Thu May 30 17:49:37 BST 2013

On 28/05/13 18:11, Crash Course wrote:
> What I'm seeing as I go from
> 4 producers -> direct exchange -> single queue -> 6 consumers
> then repeating that layout two more times (in parallel) against a
> single broker is:
> 38000 -> 54000 -> 63000 messages per second
> and
> 14 -> 26 -> 34% CPU
>
> Would your recommendation then be to achieve scaling of RabbitMQ by
> having the application layer be able to send to multiple exchanges and
> brokers, and consumers likewise be able to listen to multiple queues
> and multiple brokers

The architecture will be determined by the application's requirements on 
the routing topology.

Inside the broker, messages are passed through a pipeline of processes, 
which roughly looks like this:

producer connection handler (one per connection) ->
protocol and routing logic (one per channel) ->
queue ->
protocol logic (one per channel) ->
consumer connection handler (one per connection)

This pipeline is never perfectly smooth - there's always some bottleneck 
somewhere.

Generally, the more you separate message flows - by making them go 
through separate connections, channels and queues - the smaller the 
impact of individual bottlenecks becomes and the better performance 
you'll get.

The "layout repetition" you did above is a good example of that.

Regards,

Matthias.