[rabbitmq-discuss] Determining Cause of Flow-Control Invocation

[Michael Klishin]

On oct 4, 2013, at 9:37 p.m., Richard Raseley <richard at raseley.com> wrote:

> (*) When a RabbitMQ broker has multiple queues, will it automatically disperse them across the available cores? For example if I have 8 queues on a broker with 8 cores, will it automatically ensure a 1:1 mapping of queues:cores?

For all intents and purposes, you can think that the answer is "yes". 8 cores will be used,
but it's a little more complicated than "1 queue per core".

> 
> (*) In the situation where I have been using one queue that mapped well to a logical construct (e.g 1 queue for info, 1 queue for warning, 1 queue for errors) can you share your own personal opinions on the best way to divide that up to ensure distribution across all cores (e.g. one queue per type per core, one queue per producer per type, etc.)?

you can implement sharding in the client. That's basically how travis-ci.org worked
at some point (2012). To be more specific, you can have N queues named

info.1
info.2
…
info.N

and have consumers consume from all of them. Publisher can use any hashing
technique you like, e.g. consistent hashing or even basic module-based caching,
since there is no need to rebalance queue contents when you add or remove queues.

The key issue with this is ordering. E.g. if your stream must be ordered (e.g. CI build
log chunks must be), it is trickier but certainly doable. If not, you can parallelize the
stream between many queues pretty trivially.

See http://about.travis-ci.org/blog/2013-08-08-solving-the-puzzle-of-scalable-log-processing/ for
some more insights.

HTH.

MK



-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 495 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20131004/d12a480b/attachment.pgp>