[rabbitmq-discuss] RabbitMQ to the browser - scale question

[Marek Majkowski]

On Wed, Dec 21, 2011 at 01:28, monosodiumg <monosodiumg at gmail.com> wrote:
> My use case is distributing data updates  to user's browsers. Typical
> initial load of 5KB and updates of <200B at a rate of 0.5 updates/s.
> I have 500 topics and 1M browser clients, with a very uneven sub
> profile  e.g. 30% of users subscribing to the same one topic. Most
> users subscribe to just one topic. Clients using browsers are a bit
> erratic.

Outgoing messages = 1M * 0.5 msg/sec = 500K msg/sec

A normal RabbitMQ broker should handle around 30K msg/sec.
With a strong box, new Erlang and HIPE enabled you should
be able to get up to 50K msg/sec per box.

So, for your needs you should consider partitioning the clients
and using mulitple brokers. As far as I understand most messages
are broadcasted (you mentioned 500 topics). Just forwarding
all messages to all brokers should do (as far as the rate of
produced messages is reasonable, let's say few K msg/sec).

By using multiple brokers you also will gain a bit of fault tolerance.

Outgoing throughput = 5KB * churn_rate_per_sec + 1M clients * 200B * 0.5 sec
  - that's around 100MiB/s

Yes, you should use multiple brokers :)

> I have a few questions:
> 1. Can RabbitMQ handle that load?

Use mutliple brokers and publish all messages to all of them.

> 2. What is the roadmap for rabbitmq_jsonrpc_channel?

As you may see from the commit history we're not heavily
working on it at the minute. We're considering creating a plugin
that would combine SockJS and rabbitmq-stomp plugin, in order to
allow some form of web messaging.

> 3. How does RabbitMQ (+jsonrpc plugin) deal with browser clients
> failing to disconnect explicitly?

Frankly - I don't know!

> 4. What does this mean for thread and socket counts server-side?

Erlang/Rabbit don't use too much operating system threads, don't worry
about that.
But Erlang does use a thing called "erlang processes", and there is a
preconfigured limit for that. If you're considering starting many connections
against a single RabbitMQ broker you may consider updating that limit.

Socket limit is on most unix-es set to 1024. Feel free to modify this
number using `ulimit -n`.

Additionally, rabbitmq-management plugin shows files/sockets counters,
you can use that to see how rabbit is doing and how many file descriptors
are still available.

Cheers,
   Marek