[rabbitmq-discuss] High Performance Setup Strategy

[Alvaro Videla]

I just published a message to an exchange with 2^16 bindings. My Macbook
fan is humming :) Beam is still using over 220% CPU.


On Mon, Feb 18, 2013 at 8:32 PM, Jerry Kuch <jerryk at rbcon.com> wrote:

> Was there really expected to be an exchange with the misspelled name "getNotificaions"
> or did a typo in something seeking it yield a surprise?
>
> On Mon, Feb 18, 2013 at 5:45 AM, Florian Müller <max06.net at googlemail.com>wrote:
>
>> Hi,
>>
>> 2013/2/18 Emile Joubert <emile at rabbitmq.com>
>>
>>>
>>> Hi,
>>>
>>> On 18/02/13 10:53, Florian Müller wrote:
>>> > (Using only one exchange crashes when reaching 64k bindings)
>>>
>>> Do you have a crash report or a logfile entry from this crash? If you
>>> can explain the precise nature of the difficulty encountered at this
>>> point then it might apply in the 120k case as well.
>>>
>>
>> Sadly no system log. Just an own snippet:
>>
>> 17:57:12316599977 #13371 17:56:49: bind_65534
>> #13371 17:56:49: =============> EXCEPTION: Error: NOT_FOUND - no exchange
>> 'getNotificaions' in vhost '/websocket'
>> (The exchange disappeared completely - It happened on Dec. 3rd 2012, it
>> might be already fixed in newer version, we didn't test it)
>>
>> When we started this project, we used only one exchange (direct) for all
>> messages. At exactly the 64k th (+1?) Binding on this one exchange, the
>> whole thing crashed. So we decided to split the exchange up to 100, and
>> distribute the users to specific ones (for example User ID xxxxxxxx817 ->
>> exchange17). That balances the bindings well.
>>
>>
>>> > Problem:
>>> > Queues are starting to fill up when reaching 120k Bindings (130-140k
>>> > unique Websocket-Connections). 4-6k delivered Msg/s.
>>>
>>> Queues getting longer is not normally a cause for concern, unless your
>>> hardware spec is not sufficient to deal with so many queues when they
>>> get longer. Queues will consume more broker resources when they get
>>> longer. Consider setting a TTL limit to make sure queues don't grow very
>>> long: http://www.rabbitmq.com/ttl.html
>>>
>>
>> Hardware-Specs of the "big" one should be more than enough... 2* Xeon
>> E5645 2.40 GHz, 24GB ram, 50G SSD (not in use), GBit Ethernet. Ram usage
>> stays below 40% all the time.
>>
>> We're already using TTL, every Messageis being redirected after 5 seconds
>> to a separate Queue (for debugging).
>>
>>
>>> Queues getting longer could be a sign that consumers are not able to
>>> keep up, or some other bottleneck between the broker and consumers. Make
>>> sure the network is not saturated.
>>>
>>
>> Every consumer (Nodejs-Sockets, 16 per Host, 3 Hosts) is fine at crashing
>> time. Load is below 30%. Ram is far away from limit.
>> (Network bandwidth needs to be checked, but reproduction is not a good
>> idea, because we can only reach this load in production environment)
>>
>>
>>>
>>> Do changes to the consumption patterns make any difference to limit for
>>> maximum number of bindings? E.g. What happens when you change the
>>> prefetch count, or switch between server-side and client-side
>>> acknowledgements?
>>>
>>
>> Forwarding this to dev...
>>
>>>
>>>
>>>
>>> -Emile
>>>
>>>
>>> Florian
>>
>>
>>
>> _______________________________________________
>> rabbitmq-discuss mailing list
>> rabbitmq-discuss at lists.rabbitmq.com
>> https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
>>
>>
>
> _______________________________________________
> rabbitmq-discuss mailing list
> rabbitmq-discuss at lists.rabbitmq.com
> https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20130218/47b0552d/attachment.htm>