[rabbitmq-discuss] High Performance Setup Strategy

Mon Feb 18 13:45:36 GMT 2013

Hi,

2013/2/18 Emile Joubert <emile at rabbitmq.com>

>
> Hi,
>
> On 18/02/13 10:53, Florian Müller wrote:
> > (Using only one exchange crashes when reaching 64k bindings)
>
> Do you have a crash report or a logfile entry from this crash? If you
> can explain the precise nature of the difficulty encountered at this
> point then it might apply in the 120k case as well.
>

Sadly no system log. Just an own snippet:

17:57:12316599977 #13371 17:56:49: bind_65534
#13371 17:56:49: =============> EXCEPTION: Error: NOT_FOUND - no exchange
'getNotificaions' in vhost '/websocket'
(The exchange disappeared completely - It happened on Dec. 3rd 2012, it
might be already fixed in newer version, we didn't test it)

When we started this project, we used only one exchange (direct) for all
messages. At exactly the 64k th (+1?) Binding on this one exchange, the
whole thing crashed. So we decided to split the exchange up to 100, and
distribute the users to specific ones (for example User ID xxxxxxxx817 ->
exchange17). That balances the bindings well.

> > Problem:
> > Queues are starting to fill up when reaching 120k Bindings (130-140k
> > unique Websocket-Connections). 4-6k delivered Msg/s.
>
> Queues getting longer is not normally a cause for concern, unless your
> hardware spec is not sufficient to deal with so many queues when they
> get longer. Queues will consume more broker resources when they get
> longer. Consider setting a TTL limit to make sure queues don't grow very
> long: http://www.rabbitmq.com/ttl.html
>

Hardware-Specs of the "big" one should be more than enough... 2* Xeon E5645
2.40 GHz, 24GB ram, 50G SSD (not in use), GBit Ethernet. Ram usage stays
below 40% all the time.

We're already using TTL, every Messageis being redirected after 5 seconds
to a separate Queue (for debugging).

> Queues getting longer could be a sign that consumers are not able to
> keep up, or some other bottleneck between the broker and consumers. Make
> sure the network is not saturated.
>

Every consumer (Nodejs-Sockets, 16 per Host, 3 Hosts) is fine at crashing
time. Load is below 30%. Ram is far away from limit.
(Network bandwidth needs to be checked, but reproduction is not a good
idea, because we can only reach this load in production environment)

>
> Do changes to the consumption patterns make any difference to limit for
> maximum number of bindings? E.g. What happens when you change the
> prefetch count, or switch between server-side and client-side
> acknowledgements?
>

Forwarding this to dev...

>
>
>
> -Emile
>
>
> Florian
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20130218/75f6f4ea/attachment.htm>