[rabbitmq-discuss] questions about distributed queue

Mon Aug 17 15:23:48 BST 2009

Paul

Can you explain what your scalability metric is please?

If you mean 'able to be more available' then RabbitMQ has been able to
cluster for some time.  Want more capacity?  Add more nodes.

If you mean the slow consumer problem, i.e. you have N queues, don't
want to add more, and don't want to buy more boxes, then the 'page to
disk' feature is currently in QA for RabbitMQ 1.7 which is due quite
soon.

alexis

On Mon, Aug 17, 2009 at 3:06 PM, Paul Dix<paul at pauldix.net> wrote:
> I guess Linux-HA takes care of the failover requirement. The 1s or
> more of downtime isn't really a big concern for me. The truth is that
> my bigger concern is scalability. I'm glad to hear that it's what your
> priority is right now. Do you have a roadmap for when a scalable queue
> will be available?
>
> Just to give you a little more information on what I'm doing, I'm
> building a live search/aggregation system. I'm hoping to push updates
> of a constant internet crawl through the messaging system so workers
> can analyze the content and build indexes as everything comes in.
> That's why 1s or so of downtime isn't that big of a concern while
> scalability is.
>
> Thanks,
> Paul
>
> On Mon, Aug 17, 2009 at 9:32 AM, Tony Garnock-Jones<tonyg at lshift.net> wrote:
>> Hi Paul,
>>
>> Paul Dix wrote:
>>> I've heard that there are workable solutions to these problems, but I
>>> wasn't able to dig up anything that made sense. Also, it's noted in
>>> the FAQ and a few discussions that work is being done on distributed
>>> queues. How close is this?
>>
>> The main solution is to separate the problem into two pieces: service
>> availability and data availability. Then, for data availability (i.e.
>> effectively replicating the contents of queues) use normal
>> high-availability network file systems to share the data directories
>> between nodes. For service availability, Linux-HA or similar can handle
>> failover and locking.
>>
>>  - the network filesystem ensures the data is replicated appropriately
>>
>>  - Linux-HA takes care of locking
>>
>>  - Linux-HA takes care of starting the standby service when the
>>   primary goes down
>>
>> This assumes that you can deal with a nonzero (but arbitrarily small)
>> failover window. If you absolutely must have 100% uptime (!) then there
>> are a bunch of other solutions that can be explored, involving redundant
>> data-paths, replication of message streams, and deduplication at the
>> client. We find that very few applications really need this.
>>
>> We do have some plans for simplifying that "100%" uptime solution and
>> embedding it into the server without need for as much client-side
>> support, but we're concentrating right now on the new scalable persister
>> QA. We're likely to address issues of HA once that's done.
>>
>> Regards,
>>  Tony
>> --
>>  [][][] Tony Garnock-Jones     | Mob: +44 (0)7905 974 211
>>   [][] LShift Ltd             | Tel: +44 (0)20 7729 7060
>>  []  [] http://www.lshift.net/ | Email: tonyg at lshift.net
>>
>
> _______________________________________________
> rabbitmq-discuss mailing list
> rabbitmq-discuss at lists.rabbitmq.com
> http://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
>