[rabbitmq-discuss] building system architecture

[Francesco Mazzoli]

Hi Anton,

At Fri, 13 Jul 2012 12:32:33 +0400,
Anton Kirilenko wrote:
> 1) Every building client can process its task for an hour or more. Is it 
> correct to send ACK only after the building is done?

That should not pose any particular problem, unless you have a considerable
amount of acks piling up and thus occupying memory on the broker.

> 2) Now I've got one queue for each combination of building platform and
> architecture. Lots of client consume from it. But I need a task scheduling
> mechanism (send all the task to queue when we get them is not a good idea),
> ability to stop building... It looks like I need separate queue for every
> client (it can be exclusive consumer for every queue).  Is it possible to
> configure RabbitMQ to close the queue and notify building server (that have
> sent the message) about it?

Check the `auto-delete' field in the AMQP `queue.declare' method. I am not sure
what you mean with "notifying the building server".

If you mean that you want to notify the publisher when the queue the published
on has been deleted, then you can't do that - the publisher has no knowledge
about which queue(s) he's publishing in anyway. In general the only messages
you'll get from the broker to a publisher are return messages after a publish.

You can, however, be notified if the message you are publishing could not be
routed using the `mandatory' flag.

You can also be notified on the consumer side about queue deletion, see
http://www.rabbitmq.com/extensions.html#consumer-cancel-notify .

> The important thing: it's very bad for the system to loose building 
> tasks. In case of client failure RabbitMQ should notify server,

As said before, publishers and consumers are very decoupled, so this is not
possible.

> in case of RabbitMQ failure (or connection failure) server should know that
> the task was not processed well and resend it to another client.

It seems here that you are referring to RabbitMQ and "server" as different
entities, when they are the same thing.

In any case, you can have strong guarantees about the delivery of messages, for
example using durable queues and publisher confirms:
http://www.rabbitmq.com/extensions.html#publishing .

> In other words - after building server have sent the message, it should be
> notified if the task was not (and will never be) processed.

Again it's not clear to me what you mean here. It seems to me that a
one-queue-per-client solution with publisher confirms would suffice in your
case. You have strong guarantees against the loss of messages, and you can ack
the messages once you've finished building. If rabbit fails, you know that the
messages are persisted and thus they won't be lost. If the consumer fails, the
messages pending acks will be redelivered to the next consumer (with a
`redeliver' flag to indicate that the messages was sent before).

--
Francesco * Often in error, never in doubt