[rabbitmq-discuss] questions about distributed queue

[Paul Dix]

I guess Linux-HA takes care of the failover requirement. The 1s or
more of downtime isn't really a big concern for me. The truth is that
my bigger concern is scalability. I'm glad to hear that it's what your
priority is right now. Do you have a roadmap for when a scalable queue
will be available?

Just to give you a little more information on what I'm doing, I'm
building a live search/aggregation system. I'm hoping to push updates
of a constant internet crawl through the messaging system so workers
can analyze the content and build indexes as everything comes in.
That's why 1s or so of downtime isn't that big of a concern while
scalability is.

Thanks,
Paul

On Mon, Aug 17, 2009 at 9:32 AM, Tony Garnock-Jones<tonyg at lshift.net> wrote:
> Hi Paul,
>
> Paul Dix wrote:
>> I've heard that there are workable solutions to these problems, but I
>> wasn't able to dig up anything that made sense. Also, it's noted in
>> the FAQ and a few discussions that work is being done on distributed
>> queues. How close is this?
>
> The main solution is to separate the problem into two pieces: service
> availability and data availability. Then, for data availability (i.e.
> effectively replicating the contents of queues) use normal
> high-availability network file systems to share the data directories
> between nodes. For service availability, Linux-HA or similar can handle
> failover and locking.
>
>  - the network filesystem ensures the data is replicated appropriately
>
>  - Linux-HA takes care of locking
>
>  - Linux-HA takes care of starting the standby service when the
>   primary goes down
>
> This assumes that you can deal with a nonzero (but arbitrarily small)
> failover window. If you absolutely must have 100% uptime (!) then there
> are a bunch of other solutions that can be explored, involving redundant
> data-paths, replication of message streams, and deduplication at the
> client. We find that very few applications really need this.
>
> We do have some plans for simplifying that "100%" uptime solution and
> embedding it into the server without need for as much client-side
> support, but we're concentrating right now on the new scalable persister
> QA. We're likely to address issues of HA once that's done.
>
> Regards,
>  Tony
> --
>  [][][] Tony Garnock-Jones     | Mob: +44 (0)7905 974 211
>   [][] LShift Ltd             | Tel: +44 (0)20 7729 7060
>  []  [] http://www.lshift.net/ | Email: tonyg at lshift.net
>