[rabbitmq-discuss] RabbitMQ/AMQP understanding

Fri Aug 22 21:49:23 BST 2008

Torben,

On Fri, Aug 22, 2008 at 9:05 PM, Torben Hoffmann
<torben.lehoff at googlemail.com> wrote:
> Okay, I did not think of this type of solution, but I think this would work
> just as fine. So either I get a lot of queues or a lot of exchanges
> depending on how I choose to represent the groups. I will let that be up to
> a practical test on a couple of machines.

Usually you would go for a queue per client, but this is not a rule,
since queues can be shared. YMMV.

>> Note that you do not necessarily need an AMQP instance on machine that
>> clients run on. Multiple remote clients can connect to a single broker
>> instance.
>
> Yes, but for the sake of performance I think that the more I can distribute
> the work load the better.

Sure, but wouldn't you want the main load to be with the consumers,
i.e. where the real work is being done? The broker is just an
intermdiary that routes messages in an efficient fashion. Under normal
circumstances I would want to be allocating a lot more resources to be
doing the real work rather than distributing it.

Ideally you should be using a small (relative to the workload) number
of Rabbit instances to farm out work to a large number of workers.

This is not to say that you can't have large Rabbit clusters, it's
just that Rabbit, as a communication fabric, shouldn't consume the
lion share of your resources.

> Indeed! The machines are already there and cannot be removed, but with AMQP
> I think that I can let them do more of the work in my system and remove all
> the machines that connect the machines where the consumer processes run.

You might be interested in a feature of the AMQP spec called basic.qos
which allows for better load balancing. We have not implemented it yet
in Rabbit because of two reasons:

1. It is tricky to get right and verify it's correctness;
2. Few people have asked for it yet.

But we do have a plan of how we'd like to implement it.

>> The size of your broker cluster will depend (amongst other factors) on
>> the amount of messages you need the system to be able to handle.
>
> A system today consisting of 100 "consumer" machines plus some inner
> machines handles in the neighbourhood of 3,000 message per second.
> So when I aim to remove the inner machines that does the most of the
> intelligence regarding routing of the messages I expect to get rid of the
> inner bottleneck and thus get a higher performance... or at least one that
> is closer to the limits of the "consumer" machines which I think is a better
> constraint.

You may want to check out the performance metrics of a single broker
with a single producer and a single consumer using the ProducerMain
and ConsumerMain test classes from the RabbitMQ Java library, just to
calibrate your expectations for yourself.

> Sorry if I sound a bit vague at times but I have only just read about AMQP
> and are now trying to get the concepts for a new AMQP based architecture in
> place.

Not a problem. There are various sources of information about the
concepts of AMQP, not least to mention the specification :-)

HTH,

Ben