Hi,<br><br><div class="gmail_quote">2013/2/18 Emile Joubert <span dir="ltr"><<a href="mailto:emile@rabbitmq.com" target="_blank">emile@rabbitmq.com</a>></span><br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
Hi,<br>
<br>
On 18/02/13 10:53, Florian Müller wrote:<br>
> (Using only one exchange crashes when reaching 64k bindings)<br>
<br>
Do you have a crash report or a logfile entry from this crash? If you<br>
can explain the precise nature of the difficulty encountered at this<br>
point then it might apply in the 120k case as well.<br></blockquote><div><br></div><div>Sadly no system log. Just an own snippet:</div><div><br></div><div>17:57:12316599977 #13371 17:56:49: bind_65534<br>#13371 17:56:49: =============> EXCEPTION: Error: NOT_FOUND - no exchange 'getNotificaions' in vhost '/websocket'</div>
<div>(The exchange disappeared completely - It happened on Dec. 3rd 2012, it might be already fixed in newer version, we didn't test it)</div><div><br></div><div>When we started this project, we used only one exchange (direct) for all messages. At exactly the 64k th (+1?) Binding on this one exchange, the whole thing crashed. So we decided to split the exchange up to 100, and distribute the users to specific ones (for example User ID xxxxxxxx817 -> exchange17). That balances the bindings well.</div>
<div><br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
> Problem:<br>
> Queues are starting to fill up when reaching 120k Bindings (130-140k<br>
> unique Websocket-Connections). 4-6k delivered Msg/s.<br>
<br>
Queues getting longer is not normally a cause for concern, unless your<br>
hardware spec is not sufficient to deal with so many queues when they<br>
get longer. Queues will consume more broker resources when they get<br>
longer. Consider setting a TTL limit to make sure queues don't grow very<br>
long: <a href="http://www.rabbitmq.com/ttl.html" target="_blank">http://www.rabbitmq.com/ttl.html</a><br></blockquote><div><br></div><div>Hardware-Specs of the "big" one should be more than enough... 2* Xeon E5645 2.40 GHz, 24GB ram, 50G SSD (not in use), GBit Ethernet. Ram usage stays below 40% all the time.</div>
<div><br></div><div>We're already using TTL, every Messageis being redirected after 5 seconds to a separate Queue (for debugging).</div><div><br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
Queues getting longer could be a sign that consumers are not able to<br>
keep up, or some other bottleneck between the broker and consumers. Make<br>
sure the network is not saturated.<br></blockquote><div><br></div><div>Every consumer (Nodejs-Sockets, 16 per Host, 3 Hosts) is fine at crashing time. Load is below 30%. Ram is far away from limit.</div><div>(Network bandwidth needs to be checked, but reproduction is not a good idea, because we can only reach this load in production environment)</div>
<div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
Do changes to the consumption patterns make any difference to limit for<br>
maximum number of bindings? E.g. What happens when you change the<br>
prefetch count, or switch between server-side and client-side<br>
acknowledgements?<br></blockquote><div><br></div><div>Forwarding this to dev... </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<span class="HOEnZb"><font color="#888888"><br>
<br>
<br>
-Emile<br>
<br>
<br></font></span></blockquote><div>Florian</div><div> </div></div><br>