Hi Tony,<br><br>Thanks for the talk on Wednesday.<br><br>I'd rather not hack the kernel. If RabbitMQ can already handle the queues and messages, could I instead have a dozen multiplexer processes that do nothing but multiplex 1000 TCP AMQP connections into a single one? I don't know if the AMQP protocol makes this possible. It might need to keep track of virtual connections within the single connection.<br>
<br>Lawrence<br><br><br><br><div class="gmail_quote">On Fri, Apr 10, 2009 at 2:53 PM, Tony Garnock-Jones <span dir="ltr"><<a href="mailto:tonyg@lshift.net">tonyg@lshift.net</a>></span> wrote:<br><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
Hi Lawrence,<div class="im"><br>
<br>
Lawrence Kesteloot wrote:<br>
<blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
I need to have a large number of connections to the message broker,<br>
perhaps 10,000. There is typically (on Unix) a limit of 1024 open file<br>
descriptors per process.<br>
</blockquote>
<br></div>
Indeed. You will need to engage in kernel (or at least rlimit) tuning to raise the limit. (You may also need to increase the limit the erlang VM places on the maximum number of processes; "+P 1000000" argument to the VM)<br>
<br>
I've had upwards of 4000 simultaneous connections to a single node before, but haven't done any serious testing of it with so many connections.<div class="im"><br>
<br>
<blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
Multiple instances of the broker?<br>
</blockquote>
<br></div>
Clustering will help spread the load, so yes, maybe.<div class="im"><br>
<br>
<blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
Also, each connection will<br>
have its own queue. Can RabbitMQ deal with tens of thousands of<br>
queues? The total message throughput will be on the order of the<br>
number of connections per second (several thousand per second).<br>
</blockquote>
<br></div>
It can deal with tens or even hundreds of thousands of queues. The throughput may be a limiting factor, but again clustering may help there.<br>
<br>
Regards,<br><font color="#888888">
Tony<br>
<br>
</font></blockquote></div><br>