[rabbitmq-discuss] High Performance Setup Strategy

Mon Feb 18 19:32:27 GMT 2013

Was there really expected to be an exchange with the misspelled name
"getNotificaions"
or did a typo in something seeking it yield a surprise?

On Mon, Feb 18, 2013 at 5:45 AM, Florian Müller <max06.net at googlemail.com>wrote:

> Hi,
>
> 2013/2/18 Emile Joubert <emile at rabbitmq.com>
>
>>
>> Hi,
>>
>> On 18/02/13 10:53, Florian Müller wrote:
>> > (Using only one exchange crashes when reaching 64k bindings)
>>
>> Do you have a crash report or a logfile entry from this crash? If you
>> can explain the precise nature of the difficulty encountered at this
>> point then it might apply in the 120k case as well.
>>
>
> Sadly no system log. Just an own snippet:
>
> 17:57:12316599977 #13371 17:56:49: bind_65534
> #13371 17:56:49: =============> EXCEPTION: Error: NOT_FOUND - no exchange
> 'getNotificaions' in vhost '/websocket'
> (The exchange disappeared completely - It happened on Dec. 3rd 2012, it
> might be already fixed in newer version, we didn't test it)
>
> When we started this project, we used only one exchange (direct) for all
> messages. At exactly the 64k th (+1?) Binding on this one exchange, the
> whole thing crashed. So we decided to split the exchange up to 100, and
> distribute the users to specific ones (for example User ID xxxxxxxx817 ->
> exchange17). That balances the bindings well.
>
>
>> > Problem:
>> > Queues are starting to fill up when reaching 120k Bindings (130-140k
>> > unique Websocket-Connections). 4-6k delivered Msg/s.
>>
>> Queues getting longer is not normally a cause for concern, unless your
>> hardware spec is not sufficient to deal with so many queues when they
>> get longer. Queues will consume more broker resources when they get
>> longer. Consider setting a TTL limit to make sure queues don't grow very
>> long: http://www.rabbitmq.com/ttl.html
>>
>
> Hardware-Specs of the "big" one should be more than enough... 2* Xeon
> E5645 2.40 GHz, 24GB ram, 50G SSD (not in use), GBit Ethernet. Ram usage
> stays below 40% all the time.
>
> We're already using TTL, every Messageis being redirected after 5 seconds
> to a separate Queue (for debugging).
>
>
>> Queues getting longer could be a sign that consumers are not able to
>> keep up, or some other bottleneck between the broker and consumers. Make
>> sure the network is not saturated.
>>
>
> Every consumer (Nodejs-Sockets, 16 per Host, 3 Hosts) is fine at crashing
> time. Load is below 30%. Ram is far away from limit.
> (Network bandwidth needs to be checked, but reproduction is not a good
> idea, because we can only reach this load in production environment)
>
>
>>
>> Do changes to the consumption patterns make any difference to limit for
>> maximum number of bindings? E.g. What happens when you change the
>> prefetch count, or switch between server-side and client-side
>> acknowledgements?
>>
>
> Forwarding this to dev...
>
>>
>>
>>
>> -Emile
>>
>>
>> Florian
>
>
>
> _______________________________________________
> rabbitmq-discuss mailing list
> rabbitmq-discuss at lists.rabbitmq.com
> https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20130218/148d4059/attachment.htm>