[rabbitmq-discuss] Queue inspector.

Wed Nov 7 17:58:18 GMT 2007

Brian

On 11/6/07, Brian Granger <ellisonbg.net at gmail.com> wrote:
>
> our system has a
> few unusual properties (more on that below)
>
> > > - each worker needs a private queue
> >
> > Is that strictly needed for your use case?
>
> Yes, each worker does need a private queue so workers can directly
> send messages to a given worker.  But, it gets more complicated than
> that in a number of ways:
>
> - The master needs to be able to push messages to arbitrary sets of
> workers.  Our general syntax (our master in written in python) looks
> like this:
>
> push(list_of_workers, msg)
>
> So you see things like this:
>
> push(getIDs(), msg)  that does a broadcast or:
>
> push(range(10), msg) that pushes to workers 0,1,...10

OK, this is good.  I assume you also have queues for work done
(results) and work in progress (lease-based?).

> - Obviously, when pushing messages to specific workers, you can't do
> dynamic load balancing.  To support dynamic load balancing we have
> another single shared FIFO queue that sits on top of the private
> worker queues.  We have a scheduler that pulls things off the shared
> queue and puts them on the private queues, based on how many messages
> there are in the private queues at any given moment.

Got it.

> I am not sure if
> rabbitmq/amqp will be able to handle this more complex queue logic as
> it would require logic *in* the rabbitmq server......
>
> So, the next question is ..... how easy it is to build custom rabbitmq
> servers that have custom queue logic.  Basically, I need queues to be
> able to act as clients to other queues.

OK, there are at least three answers to this question :-)

1. You can do this today by running a scheduler that is a regular
client and acts both as a producer and a consumer.  You may have
latency concerns if your units of work are fine grained (say <10ms
total work time per task).  If you do not, then I would recommend
looking at this as a solution.  If you do have latency concerns then I
would still start here as a prototype since you will be able to verify
that it scales.   By the way, you can also improve latency by
colocating processes (or using a single process - see 3 below).

2. Future versions of AMQP will support 'chaining' of brokers and
federation better.  Solutions of type 1 will, most likely, be able to
gracefully adopt the likely new features at low cost.

3. You can mod the broker by running 'in broker consumers' and 'in
broker producers'.  This is a very interesting area for future work
but it would require a knowledge of Erlang.  We would be happy to help
on list.  However if this is your goal then I would still start at 1
(which also lets you try out multiple brokers and remain
interoperability-compliant).

> Currently we have implemented this system using twisted (an
> asynchronous networking framework in python).  It works extremely
> well....but it doesn't scale to  a large number of workers, which is
> why we are looking at erlang/amqp.

Interesting stuff - how many workers?  (If that's not an impertinent
question ..)

>  It might be the case that we need
> to implement our server at a lower level in erlang, but I really like
> rabbitmq/amqp.

Thank-you very much.  I hope we can help you avoid having to care
about the lower levels.

> > I would certainly do this if you want to 'push' work.  In effect you
> > are making the registry explicit and this would be very useful for
> > other implementation patterns, eg shared services 'fed' by queues (a
> > bit like an 'ESB').
>
> OK, it is good to know that I am on the right track.

Cool.  Please let us know if you need help, eg with the deregistration methods.

alexis

>
> Thanks
>
> Brian
>
> >
> > > But, it would be nice to have more introspection capabilities.
> >
> > I wholly agree!
> >
> > alexis
> >
> >
> >
> >
> >
> > >
> > > Thanks
> > >
> > > Brian
> > >
> > > On 11/6/07, Alexis Richardson <alexis.richardson at cohesiveft.com> wrote:
> > > > Brian
> > > >
> > > > Not yet, I'm afraid :-(
> > > >
> > > > Management features are due in a future version of AMQP.  How
> > > > important is this for you today?
> > > >
> > > > alexis
> > > >
> > > >
> > > > On 11/5/07, Brian Granger <ellisonbg.net at gmail.com> wrote:
> > > > > On a related note, is there a way of querying the server to get the
> > > > > names of all the queues?
> > > > >
> > > > > Thanks
> > > > >
> > > > > Brian
> > > > >
> > > > > On 11/5/07, Matthias Radestock <matthias at lshift.net> wrote:
> > > > > > Matt,
> > > > > >
> > > > > >
> > > > > > matt wrote:
> > > > > > > Getting the message count only appears to get updated when you declare
> > > > > > > the queue.  What if you wanted to poll the queue and see how many
> > > > > > > messages were on it every 15 seconds or so?
> > > > > >
> > > > > > queue.declare has a passive mode of operation where it doesn't actually
> > > > > > create a new queue. That's what I was referring to when I wrote that "in
> > > > > > combination with queue.declare's passive flag it allows you to determine
> > > > > > the length of existing queues".
> > > > > >
> > > > > >
> > > > > > Matthias.
> > > > > >
> > > > > > _______________________________________________
> > > > > > rabbitmq-discuss mailing list
> > > > > > rabbitmq-discuss at lists.rabbitmq.com
> > > > > > http://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
> > > > > >
> > > > >
> > > > > _______________________________________________
> > > > > rabbitmq-discuss mailing list
> > > > > rabbitmq-discuss at lists.rabbitmq.com
> > > > > http://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
> > > > >
> > > >
> > > >
> > > > --
> > > > Alexis Richardson
> > > > +44 20 7617 7339 (UK)
> > > > +44 77 9865 2911 (cell)
> > > > +1 650 206 2517 (US)
> > > >
> > >
> >
> >
> > --
> > Alexis Richardson
> > +44 20 7617 7339 (UK)
> > +44 77 9865 2911 (cell)
> > +1 650 206 2517 (US)
> >
>

-- 
Alexis Richardson
+44 20 7617 7339 (UK)
+44 77 9865 2911 (cell)
+1 650 206 2517 (US)