[rabbitmq-discuss] Queue inspector.

Wed Nov 7 20:42:03 GMT 2007

Alexis,

On 11/7/07, Alexis Richardson <alexis.richardson at cohesiveft.com> wrote:
> Brian
>
>
> On 11/6/07, Brian Granger <ellisonbg.net at gmail.com> wrote:
> >
> > our system has a
> > few unusual properties (more on that below)
> >
> > > > - each worker needs a private queue
> > >
> > > Is that strictly needed for your use case?
> >
> > Yes, each worker does need a private queue so workers can directly
> > send messages to a given worker.  But, it gets more complicated than
> > that in a number of ways:
> >
> > - The master needs to be able to push messages to arbitrary sets of
> > workers.  Our general syntax (our master in written in python) looks
> > like this:
> >
> > push(list_of_workers, msg)
> >
> > So you see things like this:
> >
> > push(getIDs(), msg)  that does a broadcast or:
> >
> > push(range(10), msg) that pushes to workers 0,1,...10
>
> OK, this is good.  I assume you also have queues for work done
> (results) and work in progress (lease-based?).

Yep, these queues would also have to be there.

> > So, the next question is ..... how easy it is to build custom rabbitmq
> > servers that have custom queue logic.  Basically, I need queues to be
> > able to act as clients to other queues.
>
> OK, there are at least three answers to this question :-)
>
> 1. You can do this today by running a scheduler that is a regular
> client and acts both as a producer and a consumer.  You may have
> latency concerns if your units of work are fine grained (say <10ms
> total work time per task).  If you do not, then I would recommend
> looking at this as a solution.  If you do have latency concerns then I
> would still start here as a prototype since you will be able to verify
> that it scales.   By the way, you can also improve latency by
> colocating processes (or using a single process - see 3 below).

The other option that is related to this one would be to have each
worker pull from both its private queue and the shared one.  I would
just need to make some decisions about which has priority in given
situations.  Unfortunately, latency is a big issue for us - the lower
the better.

> 2. Future versions of AMQP will support 'chaining' of brokers and
> federation better.  Solutions of type 1 will, most likely, be able to
> gracefully adopt the likely new features at low cost.

I see.  That would be fantastic, as long as the chaining was
relatively flexible.

> 3. You can mod the broker by running 'in broker consumers' and 'in
> broker producers'.  This is a very interesting area for future work
> but it would require a knowledge of Erlang.  We would be happy to help
> on list.  However if this is your goal then I would still start at 1
> (which also lets you try out multiple brokers and remain
> interoperability-compliant).

I don't mind learning Erlang if it is needed.  My other option is to
just write a custom solution from scratch in Erlang.  But rabbitmq
seems to have many of the pieces already, so I would really like to
use it if possible.  Thanks for the offer of help - I will keep you
posted.

> > Currently we have implemented this system using twisted (an
> > asynchronous networking framework in python).  It works extremely
> > well....but it doesn't scale to  a large number of workers, which is
> > why we are looking at erlang/amqp.
>
> Interesting stuff - how many workers?  (If that's not an impertinent
> question ..)

We are using this architecture for parallel computing in Python.  Our
workers are basically Python interpreters exposed to the network.  Our
"clients" are also python scripts that submit python code/objects to
the workers to be performed.  As you can imagine I am looking at the
qpid python amqp client.  We have run already with up to 256 workers
and we already see scaling problems.  Our stuff works just fine on
supercomputers, so we would like to scale up to.....2**10, 2**11,
2**12, etc.  We hit scaling problems in a number of ways:

1) # of file descriptors per process.
2) broadcasting/multicasting to all (or large subsets) of workers.
3) Latency - the latency determines the grain size of work, which we
would like to be as small as possible.
4) Large messages.  In some cases, the messages can be very large
(files or large data sets).  It is not uncommon for people to want to
distribute 100 MB messages to subsets of workers.  Our current
solution generates a short lived temporary copy of the large objects
for each worker they are sent to.  This means if we have 128 workers,
we use 128*msg_size memory for a short period of time.  We know how to
fix this in Python, but it still doesn't solve the other scalability
problems.  All that to say, any solution that we come up with must
also be able to handle large messages well.  Does rabbitmq have any
streaming support?

>
> >  It might be the case that we need
> > to implement our server at a lower level in erlang, but I really like
> > rabbitmq/amqp.
>
> Thank-you very much.  I hope we can help you avoid having to care
> about the lower levels.

I would love to avoid the lower level stuff, but I also realize we
have a somewhat unusual set of design requirements (scalability,
low-latency && ability to handle large messages).  In various
prototypes we even implemented custom protocols on top of TCP.  Once
you start doing that Erlang becomes a "high-level" solution :)

>
> > > I would certainly do this if you want to 'push' work.  In effect you
> > > are making the registry explicit and this would be very useful for
> > > other implementation patterns, eg shared services 'fed' by queues (a
> > > bit like an 'ESB').
> >
> > OK, it is good to know that I am on the right track.
>
> Cool.  Please let us know if you need help, eg with the deregistration methods.

Ahh yes, the deregistration methods could be subtle.  The main concern
there is if a worker dies without notifying the server that they are
going to die.  What facilities does rabbitmq/amqp have to handle this
type of stuff?

Thanks much for your help!

Brian

> alexis
>
>
>
>
>
>
>
>
>
>
>
>
>
>
> >
> > Thanks
> >
> > Brian
> >
> > >
> > > > But, it would be nice to have more introspection capabilities.
> > >
> > > I wholly agree!
> > >
> > > alexis
> > >
> > >
> > >
> > >
> > >
> > > >
> > > > Thanks
> > > >
> > > > Brian
> > > >
> > > > On 11/6/07, Alexis Richardson <alexis.richardson at cohesiveft.com> wrote:
> > > > > Brian
> > > > >
> > > > > Not yet, I'm afraid :-(
> > > > >
> > > > > Management features are due in a future version of AMQP.  How
> > > > > important is this for you today?
> > > > >
> > > > > alexis
> > > > >
> > > > >
> > > > > On 11/5/07, Brian Granger <ellisonbg.net at gmail.com> wrote:
> > > > > > On a related note, is there a way of querying the server to get the
> > > > > > names of all the queues?
> > > > > >
> > > > > > Thanks
> > > > > >
> > > > > > Brian
> > > > > >
> > > > > > On 11/5/07, Matthias Radestock <matthias at lshift.net> wrote:
> > > > > > > Matt,
> > > > > > >
> > > > > > >
> > > > > > > matt wrote:
> > > > > > > > Getting the message count only appears to get updated when you declare
> > > > > > > > the queue.  What if you wanted to poll the queue and see how many
> > > > > > > > messages were on it every 15 seconds or so?
> > > > > > >
> > > > > > > queue.declare has a passive mode of operation where it doesn't actually
> > > > > > > create a new queue. That's what I was referring to when I wrote that "in
> > > > > > > combination with queue.declare's passive flag it allows you to determine
> > > > > > > the length of existing queues".
> > > > > > >
> > > > > > >
> > > > > > > Matthias.
> > > > > > >
> > > > > > > _______________________________________________
> > > > > > > rabbitmq-discuss mailing list
> > > > > > > rabbitmq-discuss at lists.rabbitmq.com
> > > > > > > http://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
> > > > > > >
> > > > > >
> > > > > > _______________________________________________
> > > > > > rabbitmq-discuss mailing list
> > > > > > rabbitmq-discuss at lists.rabbitmq.com
> > > > > > http://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
> > > > > >
> > > > >
> > > > >
> > > > > --
> > > > > Alexis Richardson
> > > > > +44 20 7617 7339 (UK)
> > > > > +44 77 9865 2911 (cell)
> > > > > +1 650 206 2517 (US)
> > > > >
> > > >
> > >
> > >
> > > --
> > > Alexis Richardson
> > > +44 20 7617 7339 (UK)
> > > +44 77 9865 2911 (cell)
> > > +1 650 206 2517 (US)
> > >
> >
>
>
> --
> Alexis Richardson
> +44 20 7617 7339 (UK)
> +44 77 9865 2911 (cell)
> +1 650 206 2517 (US)
>