[rabbitmq-discuss] pull task queue design

Wed Jul 25 11:20:55 BST 2012

On Tue, Jul 24, 2012 at 4:31 PM, Mark Ward <ward.mark at gmail.com> wrote:
> Hello,
>
> I am fairly new to rabbitMQ and AMQP.  I am wondering if it is possible to
> build a pull task queue with task lease capabilities.  Something like the
> Google App Engine pull queue
> https://developers.google.com/appengine/docs/python/taskqueue/overview-pull.

I strongly recommend taking a look at RabbitMQ tutorials:
    http://www.rabbitmq.com/getstarted.html

Second tutorial explains how RabbitMQ deals with "tasks" and "acknowledgments"
(similar to leases):
  http://www.rabbitmq.com/tutorials/tutorial-two-python.html

In rabbitmq you can't do time based 'leases', but 'acknowledgemnts'
are a similar concept, only without an explicit timeout.

> I am exploring the idea to have an http/rest interface between our worker
> and the server.  The server would connect rabbitMQ.  The workers are
> distributed to our client sites.  The reasoning behind this is due to being
> unsure how the persistent connection of AMQP will fair with customer’s
> firewall and proxy server configurations.  I cannot predict the combinations
> or limitations of outbound connections but typically http(s) 80/443 are
> guaranteed outbound connections.
>
> Possibly a reassurance of how well the AMQP protocol fairs across the
> internet may sway the design back to use the AMQP protocol over using the
> http/proxy idea.  The AMQP setup at the client site will need to function
> right from the start without having to involve network administrators
> altering configurations.  It would be possible for us to host rabbitMQ on
> ports 80/443.

Exposing AMQP protocol to the wide internet is not a recommended setup.
You would be better off by creating a custom application that would pull
messages from rabbitmq and expose them over http or something.

You might want to take a look at SockJS (http://sockjs.org) and
RabbitMQ-Web-Stomp plugin:
    http://www.rabbitmq.com/blog/2012/05/14/introducing-rabbitmq-web-stomp/

> It might be impossible if I am correct that AMQP does not provide message
> timeouts

AMQP does not provide message timeouts. But RabbitMQ has an extension
for that: http://www.rabbitmq.com/extensions.html#lifetimes

> and messages are redelivered if the worker’s connection dies

Correct. Unacknowledged messages will be redelivered when the client
connection dies, or when the client explicitly rejects them.

>  With an http/proxy implementation the connections would have to be stateless.

Marek