<div dir="ltr"><br><br><div class="gmail_quote">On Fri, Aug 22, 2008 at 10:49 PM, Ben Hood <span dir="ltr"><<a href="mailto:0x6e6562@gmail.com">0x6e6562@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
Torben,<br>
<br>
On Fri, Aug 22, 2008 at 9:05 PM, Torben Hoffmann<br>
<div class="Ih2E3d"><<a href="mailto:torben.lehoff@googlemail.com">torben.lehoff@googlemail.com</a>> wrote:<br>
</div><div class="Ih2E3d">> Okay, I did not think of this type of solution, but I think this would work<br>
> just as fine. So either I get a lot of queues or a lot of exchanges<br>
> depending on how I choose to represent the groups. I will let that be up to<br>
> a practical test on a couple of machines.<br>
<br>
</div>Usually you would go for a queue per client, but this is not a rule,<br>
since queues can be shared. YMMV.</blockquote><div><br>I will need queues for the individuals themselves plus for all the groups - communication can be both 1:1 and 1:many.<br> </div><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
<br>
<div class="Ih2E3d"><br>
>> Note that you do not necessarily need an AMQP instance on machine that<br>
>> clients run on. Multiple remote clients can connect to a single broker<br>
>> instance.<br>
><br>
> Yes, but for the sake of performance I think that the more I can distribute<br>
> the work load the better.<br>
<br>
</div>Sure, but wouldn't you want the main load to be with the consumers,<br>
i.e. where the real work is being done? The broker is just an<br>
intermdiary that routes messages in an efficient fashion. Under normal<br>
circumstances I would want to be allocating a lot more resources to be<br>
doing the real work rather than distributing it.</blockquote><div><br>I wasn't clear earlier: I want to avoid having a bottleneck in the middle as today. But I am not sure what the best option is right now, but I will try some scenarios to see what fits the best. <br>
</div><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
<br>
Ideally you should be using a small (relative to the workload) number<br>
of Rabbit instances to farm out work to a large number of workers.<br>
<br>
This is not to say that you can't have large Rabbit clusters, it's<br>
just that Rabbit, as a communication fabric, shouldn't consume the<br>
lion share of your resources.</blockquote><div><br>Having the workers and Rabbit share machines will allow me to make a very elegant architecture that scales very well since adding a new machine does not require more than adding it to the cluster. But you are right that there could be some side effects in play here that I need to iron out before I get a good solution.<br>
</div><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;"><br>
<div class="Ih2E3d"><br>
> Indeed! The machines are already there and cannot be removed, but with AMQP<br>
> I think that I can let them do more of the work in my system and remove all<br>
> the machines that connect the machines where the consumer processes run.<br>
<br>
</div>You might be interested in a feature of the AMQP spec called basic.qos<br>
which allows for better load balancing. We have not implemented it yet<br>
in Rabbit because of two reasons:<br>
<br>
1. It is tricky to get right and verify it's correctness;<br>
2. Few people have asked for it yet.<br>
<br>
But we do have a plan of how we'd like to implement it.</blockquote><div><br>That does sound interesting, but for my application I think that the current Rabbit implementation will be fast enough since the task is to get a control plane in place and let an underlying IP network carry the heavy duty traffic between users.<br>
</div><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;"><br>
<div class="Ih2E3d"><br>
>> The size of your broker cluster will depend (amongst other factors) on<br>
>> the amount of messages you need the system to be able to handle.<br>
><br>
> A system today consisting of 100 "consumer" machines plus some inner<br>
> machines handles in the neighbourhood of 3,000 message per second.<br>
> So when I aim to remove the inner machines that does the most of the<br>
> intelligence regarding routing of the messages I expect to get rid of the<br>
> inner bottleneck and thus get a higher performance... or at least one that<br>
> is closer to the limits of the "consumer" machines which I think is a better<br>
> constraint.<br>
<br>
</div>You may want to check out the performance metrics of a single broker<br>
with a single producer and a single consumer using the ProducerMain<br>
and ConsumerMain test classes from the RabbitMQ Java library, just to<br>
calibrate your expectations for yourself.<br>
<div class="Ih2E3d"><br>
> Sorry if I sound a bit vague at times but I have only just read about AMQP<br>
> and are now trying to get the concepts for a new AMQP based architecture in<br>
> place.<br>
<br>
</div>Not a problem. There are various sources of information about the<br>
concepts of AMQP, not least to mention the specification :-)<br>
<br>
</blockquote><div>I like the standard - way easier to read than others I have had to deal with in the past...<br> </div></div>Cheers,<br>Torben<br></div>