[rabbitmq-discuss] RabbitMQ to the browser - scale question

Sat Jan 7 21:13:05 GMT 2012

Thanks for that analysis. It's clear to me now that RabbitMQ doesn't
work for push at the scale I would need - yet.

On Jan 5, 12:01 pm, Marek Majkowski <maje... at gmail.com> wrote:
> On Wed, Dec 21, 2011 at 01:28, monosodiumg <monosodi... at gmail.com> wrote:
> > My use case is distributing data updates  to user's browsers. Typical
> > initial load of 5KB and updates of <200B at a rate of 0.5 updates/s.
> > I have 500 topics and 1M browser clients, with a very uneven sub
> > profile  e.g. 30% of users subscribing to the same one topic. Most
> > users subscribe to just one topic. Clients using browsers are a bit
> > erratic.
>
> Outgoing messages = 1M * 0.5 msg/sec =500Kmsg/sec
>
> A normal RabbitMQ broker should handle around 30K msg/sec.
> With a strong box, new Erlang and HIPE enabled you should
> be able to get up to 50K msg/sec per box.
>
> So, for your needs you should consider partitioning the clients
> and using mulitple brokers. As far as I understand most messages
> are broadcasted (you mentioned 500 topics). Just forwarding
> all messages to all brokers should do (as far as the rate of
> produced messages is reasonable, let's say few K msg/sec).
>
> By using multiple brokers you also will gain a bit of fault tolerance.
>
> Outgoing throughput = 5KB * churn_rate_per_sec + 1M clients * 200B * 0.5 sec
>   - that's around 100MiB/s
>
> Yes, you should use multiple brokers :)
>
> > I have a few questions:
> > 1. Can RabbitMQ handle that load?
>
> Use mutliple brokers and publish all messages to all of them.
>
> > 2. What is the roadmap for rabbitmq_jsonrpc_channel?
>
> As you may see from the commit history we're not heavily
> working on it at the minute. We're considering creating a plugin
> that would combine SockJS and rabbitmq-stomp plugin, in order to
> allow some form of web messaging.
>
> > 3. How does RabbitMQ (+jsonrpc plugin) deal with browser clients
> > failing to disconnect explicitly?
>
> Frankly - I don't know!
>
> > 4. What does this mean for thread and socket counts server-side?
>
> Erlang/Rabbit don't use too much operating system threads, don't worry
> about that.
> But Erlang does use a thing called "erlang processes", and there is a
> preconfigured limit for that. If you're considering starting many connections
> against a single RabbitMQ broker you may consider updating that limit.
>
> Socket limit is on most unix-es set to 1024. Feel free to modify this
> number using `ulimit -n`.
>
> Additionally, rabbitmq-management plugin shows files/sockets counters,
> you can use that to see how rabbit is doing and how many file descriptors
> are still available.
>
> Cheers,
>    Marek
> _______________________________________________
> rabbitmq-discuss mailing list
> rabbitmq-disc... at lists.rabbitmq.comhttps://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss