[rabbitmq-discuss] questions about distributed queue

Mon Aug 17 15:34:53 BST 2009

Sorry for the confusion. I mean scalability on a single queue. Say I
want to push 20k messages per second through a single queue. If a
single node can't handle that it seems I'm out of luck. That is, if
I'm understanding how things work.

So I guess I'm not worried about total queue size, but queue
throughput (although size may become an issue, I'm not sure). It seems
the solution is to split out across multiple queues, but I was hoping
to avoid that since it will add a layer of complexity to my producers
and consumers.

I don't think I understand how using Linux-HA with clustering would
lead to a splitting a single queue across multiple nodes. I'm not
familiar with HA, but it looked like it was a solution to provide a
replicated failover.

Thanks again,
Paul

On Mon, Aug 17, 2009 at 10:24 AM, Tony Garnock-Jones<tonyg at lshift.net> wrote:
> Paul Dix wrote:
>> Do you have a roadmap for when a scalable queue
>> will be available?
>
> If by "scalable" you mean "replicated", then that's available now, by
> configuration along the lines I hinted at in my previous message. Adding
> clustering into the mix can help increase capacity, on top of that (at a
> certain cost in configuration complexity).
>
> If instead you mean "exceeding RAM+swap size", we're hoping to have that
> for the 1.7 release -- which ought to be out within a month or so.
>
>> Just to give you a little more information on what I'm doing, I'm
>> building a live search/aggregation system. I'm hoping to push updates
>> of a constant internet crawl through the messaging system so workers
>> can analyze the content and build indexes as everything comes in.
>
> Sounds pretty cool!
>
> Tony
> --
>  [][][] Tony Garnock-Jones     | Mob: +44 (0)7905 974 211
>   [][] LShift Ltd             | Tel: +44 (0)20 7729 7060
>  []  [] http://www.lshift.net/ | Email: tonyg at lshift.net
>