[rabbitmq-discuss] RabbitMQ/AMQP understanding

[Torben Hoffmann]

On Fri, Aug 22, 2008 at 10:49 PM, Ben Hood <0x6e6562 at gmail.com> wrote:

> Torben,
>
> On Fri, Aug 22, 2008 at 9:05 PM, Torben Hoffmann
> <torben.lehoff at googlemail.com> wrote:
> > Okay, I did not think of this type of solution, but I think this would
> work
> > just as fine. So either I get a lot of queues or a lot of exchanges
> > depending on how I choose to represent the groups. I will let that be up
> to
> > a practical test on a couple of machines.
>
> Usually you would go for a queue per client, but this is not a rule,
> since queues can be shared. YMMV.


I will need queues for the individuals themselves plus for all the groups -
communication can be both 1:1 and 1:many.


>
>
> >> Note that you do not necessarily need an AMQP instance on machine that
> >> clients run on. Multiple remote clients can connect to a single broker
> >> instance.
> >
> > Yes, but for the sake of performance I think that the more I can
> distribute
> > the work load the better.
>
> Sure, but wouldn't you want the main load to be with the consumers,
> i.e. where the real work is being done? The broker is just an
> intermdiary that routes messages in an efficient fashion. Under normal
> circumstances I would want to be allocating a lot more resources to be
> doing the real work rather than distributing it.


I wasn't clear earlier: I want to avoid having a bottleneck in the middle as
today. But I am not sure what the best option is right now, but I will try
some scenarios to see what fits the best.

>
> Ideally you should be using a small (relative to the workload) number
> of Rabbit instances to farm out work to a large number of workers.
>
> This is not to say that you can't have large Rabbit clusters, it's
> just that Rabbit, as a communication fabric, shouldn't consume the
> lion share of your resources.


Having the workers and Rabbit share machines will allow me to make a very
elegant architecture that scales very well since adding a new machine does
not require more than adding it to the cluster. But you are right that there
could be some side effects in play here that I need to iron out before I get
a good solution.


>
>
> > Indeed! The machines are already there and cannot be removed, but with
> AMQP
> > I think that I can let them do more of the work in my system and remove
> all
> > the machines that connect the machines where the consumer processes run.
>
> You might be interested in a feature of the AMQP spec called basic.qos
> which allows for better load balancing. We have not implemented it yet
> in Rabbit because of two reasons:
>
> 1. It is tricky to get right and verify it's correctness;
> 2. Few people have asked for it yet.
>
> But we do have a plan of how we'd like to implement it.


That does sound interesting, but for my application I think that the current
Rabbit implementation will be fast enough since the task is to get a control
plane in place and let an underlying IP network carry the heavy duty traffic
between users.


>
>
> >> The size of your broker cluster will depend (amongst other factors) on
> >> the amount of messages you need the system to be able to handle.
> >
> > A system today consisting of 100 "consumer" machines plus some inner
> > machines handles in the neighbourhood of 3,000 message per second.
> > So when I aim to remove the inner machines that does the most of the
> > intelligence regarding routing of the messages I expect to get rid of the
> > inner bottleneck and thus get a higher performance... or at least one
> that
> > is closer to the limits of the "consumer" machines which I think is a
> better
> > constraint.
>
> You may want to check out the performance metrics of a single broker
> with a single producer and a single consumer using the ProducerMain
> and ConsumerMain test classes from the RabbitMQ Java library, just to
> calibrate your expectations for yourself.
>
> > Sorry if I sound a bit vague at times but I have only just read about
> AMQP
> > and are now trying to get the concepts for a new AMQP based architecture
> in
> > place.
>
> Not a problem. There are various sources of information about the
> concepts of AMQP, not least to mention the specification :-)
>
> I like the standard - way easier to read than others I have had to deal
with in the past...

Cheers,
Torben
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20080823/4f1cb1c4/attachment.htm