[rabbitmq-discuss] Creating a job only once (using Java Client API)

[Matthew Sackman]

On Mon, Apr 18, 2011 at 07:13:43PM +0300, Allan Kamau wrote:
> Almost. I am trying to process jobs level by level. Let's say I have a
> tree structure, where I need to create jobs one level at a time and
> completely process them then generate the next batch of jobs and
> process them totally and so on. Each job (level) creation depends on
> the successful processing of all the jobs of the preceding level as
> those results form the bases of the jobs being created for the new
> level.
> 
> So the key is for the client that obtained and successfully processed
> the last job to realize this fact and generate the next round of jobs.
> The jobs are to be generated and injected into the queue by a single
> client, process and thread.
> 
> One way would be to specially mark the last job as it is being
> generated in the job creation undertaking, then program the clients to
> be on the lookout for this "last" job as they process the jobs. But
> this may not really work well considering that some jobs may take
> longer to process than this job marked as "last" and that they may be
> some jobs still under processing or awaiting to be processed (if they
> were resent due to some failure of some kind) after the completion of
> processing of the "last" job.

Thanks for the explanation.

To start with, I'd probably do one queue per level. Eventually you may
be able to collapse that back down to one queue overall, but it might
help to start with the separation.

You are after some sort of global concensus that all clients have become
idle. I would do the following;

1. Is it actually important that the next set of jobs are created by the
client that processed the last msg enqueued at the previous level; or is
it enough for the next set of jobs to be created by the client that
finished processing last?

If the former, then this is harder - you'd probably need to use a
last-value-cache and hope that all msgs come from one source, or use
something in addition to Rabbit to achieve this (maybe something like
Zookeeper?).

If the latter:

All clients are consuming from queue X for level X. When each client
becomes idle, it does a basic.cancel on its consumer, and then does an
exclusive basic.consume on the same queue.

Exactly one client will find its later basic.consume will succeed and
this will indicate that all clients have become idle and cancelled their
consumer, and there is now exactly one consumer hanging off that queue.

The client that succeeds can then generate the next batch of jobs. This
will probably consist of creating a new queue, sending the jobs into
that queue, and then communicating the queue name to the other workers.
Thus there'll need to be both a "control" exchange which simply gets
msgs about which is the next queue to consume from (each client will
need to have a queue hanging off that exchange), and then "job" queues
with one queue for each level.

Something like that should, I think work. You're really trying to build
a semaphore though and using exclusive consumers is a very very crude
way of doing this. I'm not sure this is the right tool for the job, but
I think it should work.

Matthew