[rabbitmq-discuss] Queue inspector.

Tue Nov 6 20:02:47 GMT 2007

> Brian
>
> Thanks for your note - that helps.

No problem.

> On 11/6/07, Brian Granger <ellisonbg.net at gmail.com> wrote:
> > I am evaluating rabbitmq/amqp to see if it would do what we need it
> > to.  Our system has a worker/master architecture and we would like to
> > use rabbitmq for the master<-> worker messaging.
>
> Great :-)
>
> Work queues are a common pattern and we would love to hear how you get on.

As I move forward I will let you know how it goes.

>
>
> > There are a few
> > constraints in our system though:
> >
> > - workers come and go at any time.
>
> Indeed this is also a common situation when you want your system to be
> robust - as I assume you do.  Some of this will be supported more
> 'inherently' in future versions of AMQP, eg for federations of
> brokers.  But I think there is enough in RabbitMQ as it is today, and
> as it currently clusters, to get you going.

This is definitely a very common pattern - although our system has a
few unusual properties (more on that below)

>
> > - each worker needs a private queue
>
> Is that strictly needed for your use case?

Yes, each worker does need a private queue so workers can directly
send messages to a given worker.  But, it gets more complicated than
that in a number of ways:

- The master needs to be able to push messages to arbitrary sets of
workers.  Our general syntax (our master in written in python) looks
like this:

push(list_of_workers, msg)

So you see things like this:

push(getIDs(), msg)  that does a broadcast or:

push(range(10), msg) that pushes to workers 0,1,...10

- Obviously, when pushing messages to specific workers, you can't do
dynamic load balancing.  To support dynamic load balancing we have
another single shared FIFO queue that sits on top of the private
worker queues.  We have a scheduler that pulls things off the shared
queue and puts them on the private queues, based on how many messages
there are in the private queues at any given moment.  I am not sure if
rabbitmq/amqp will be able to handle this more complex queue logic as
it would require logic *in* the rabbitmq server......

So, the next question is ..... how easy it is to build custom rabbitmq
servers that have custom queue logic.  Basically, I need queues to be
able to act as clients to other queues.

Currently we have implemented this system using twisted (an
asynchronous networking framework in python).  It works extremely
well....but it doesn't scale to  a large number of workers, which is
why we are looking at erlang/amqp.  It might be the case that we need
to implement our server at a lower level in erlang, but I really like
rabbitmq/amqp.

Any ideas you have about how to approach this would be greatly appreciated.

> You could have workers in groups, sharing a queue on a round robin
> basis.  Then they can 'pull' work and no registry is needed.

We have thought about things like this, but we would still need to
have the push mode as well.

>
> > - the master process needs to be
> > able to see which workers are currently connected.
>
> Yes - if you want to 'push' work and each worker must have a private queue.
>
>
> > My initial thought
> > is that this could be accomplished by the master getting a list of
> > queues.
>
> Got it.
>
>
> > My current thought is to have a separate queue that workers use to
> > register/unregister themselves.  They could put the name of their
> > private queue on this registration queue.
>
> I would certainly do this if you want to 'push' work.  In effect you
> are making the registry explicit and this would be very useful for
> other implementation patterns, eg shared services 'fed' by queues (a
> bit like an 'ESB').

OK, it is good to know that I am on the right track.

Thanks

Brian

>
> > But, it would be nice to have more introspection capabilities.
>
> I wholly agree!
>
> alexis
>
>
>
>
>
> >
> > Thanks
> >
> > Brian
> >
> > On 11/6/07, Alexis Richardson <alexis.richardson at cohesiveft.com> wrote:
> > > Brian
> > >
> > > Not yet, I'm afraid :-(
> > >
> > > Management features are due in a future version of AMQP.  How
> > > important is this for you today?
> > >
> > > alexis
> > >
> > >
> > > On 11/5/07, Brian Granger <ellisonbg.net at gmail.com> wrote:
> > > > On a related note, is there a way of querying the server to get the
> > > > names of all the queues?
> > > >
> > > > Thanks
> > > >
> > > > Brian
> > > >
> > > > On 11/5/07, Matthias Radestock <matthias at lshift.net> wrote:
> > > > > Matt,
> > > > >
> > > > >
> > > > > matt wrote:
> > > > > > Getting the message count only appears to get updated when you declare
> > > > > > the queue.  What if you wanted to poll the queue and see how many
> > > > > > messages were on it every 15 seconds or so?
> > > > >
> > > > > queue.declare has a passive mode of operation where it doesn't actually
> > > > > create a new queue. That's what I was referring to when I wrote that "in
> > > > > combination with queue.declare's passive flag it allows you to determine
> > > > > the length of existing queues".
> > > > >
> > > > >
> > > > > Matthias.
> > > > >
> > > > > _______________________________________________
> > > > > rabbitmq-discuss mailing list
> > > > > rabbitmq-discuss at lists.rabbitmq.com
> > > > > http://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
> > > > >
> > > >
> > > > _______________________________________________
> > > > rabbitmq-discuss mailing list
> > > > rabbitmq-discuss at lists.rabbitmq.com
> > > > http://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
> > > >
> > >
> > >
> > > --
> > > Alexis Richardson
> > > +44 20 7617 7339 (UK)
> > > +44 77 9865 2911 (cell)
> > > +1 650 206 2517 (US)
> > >
> >
>
>
> --
> Alexis Richardson
> +44 20 7617 7339 (UK)
> +44 77 9865 2911 (cell)
> +1 650 206 2517 (US)
>