[rabbitmq-discuss] Writing applications using AMQP, basic design questions

[Laurens Van Houtven]

(Messages were cut liberally for brevity reasons.)

On Thu, Mar 4, 2010 at 12:32 AM, Garrett Smith <g at rre.tt> wrote:
> Are you talking about exposing your mobile apps to external users via AMQP?

Possibly. I'm definitely not considering letting the mobile users be
AMQP producers themselves for the reasons mentioned below. Whether or
not letting the monitoring users be actual AMQP consumers, or having
another abstraction layer there is a good idea, I'm not sure.

The people doing the monitoring will primarily use a webapp that does
two things:
1. Get previous data from the persistence layer (a classic webapp that
gets stuff from a database somewhere, not AMQP so not relevant to this
discussion)
2. Follow data coming in from the mobile clients live, through a Comet
implementation.

Alternatively, later, mobile apps could request this data too, but
there's no reason they couldn't use the same Comet implementation the
browsers of the webapp users do.

AMQP sounds very well suited to the latter. I'm not sure RabbitMQ's
plugins for letting browsers talk AMQP through JS. Because of the
sensitive nature of the data, I'm not sure if
rabbitmq-jsonrpc-channel/rabbitmq-jsonrpc are applicable. The
advantage of an AMQP->Comet bridge at the end (analogous to the
HTTP->AMQP bridge at the beginning) would be that I can do the same
pre-shared SSL certificate tricks for verifying and authenticating
clients.

> What's the nature of the public API. Is there a performance or
> throughput requirement?

Not a particularly big one. Lots of writers, very low amount of data,
regular reporting. Think less than a hundred bytes, every ten seconds
to a minute or so, for a few hours a day, times potentially a few
thousand mobile clients.

I know making guesses about bottlenecks in complex systems is stupid
without any data to back it up, but I'm guessing the proposed
HTTPS->AMQP bridge will start running out of steam long before the
broker will. Hopefully this can be fixed by adding more of them, and
using classic load balancing ideas for web servers.

> Working with an AMQP interface is considerably different from working
> with a REST or other RPC interface. If you're thinking along these
> lines, I'd float the idea by some of your users to see what they think
> of the approach. That's also assuming you have a technical requirement
> to use AMQP for the public interface.

I'm mostly interested in using it for the internal interfaces for
flexibility reasons. I don't think any of the properties of AMQP
particularly interest my monitoring users (which would be AMQP
consumers if we let them speak AMQP). The mobile clients just want
their data to be persisted and possibly reported, they don't really
care what I do internally to make it happen.

>> I'm interested in temporary queues, so that only active mobile clients
>> have queues running. I have read the AMQP specification (v0.8),
>> section 2.2.8, specifically the part about temporary queues.
>> Apparently any client can cause a queue to be created, and the client
>> creating it is responsible for creating a consumer on it.
>
> You can create your queues as exclusive, which tells rabbit to auto
> delete the queue once the declaring connection is closed.

In that case, the proposed HTTPS->AMQP bridge would be responsible for
creating the mandatory consumer responsible for persistence every time
a previously inactive client starts sending data? Does that mean the
bridge has to keep the connection open? (creating a queue and
attaching a consumer sounds like an expensive operation.) That would
have implications for the load balancing I can do later.

>> I take it that means letting the mobile clients (with untrustworthy
>> software) do AMQP is a bad idea. The first idea is to have an
>> HTTP(S)->AMQP bridge. Why?

> Running AMQP on mobile clients would make for a cool demo, but you're
> thinking is sound here.

Yay! I'm not insane.

> In my block-and-tackle understanding, you would opt for a message
> oriented approach for a few reasons:
>
> - Out-of-the-box support async application patterns

Definitely true. Everything is being powered by Twisted/txAMQP, so
everything else is already asynchronous in nature.

> - Latency requirements prohibit the use of expensive transports like HTTP

Internally? Sure. As long as the only HTTP latency being introduced is
that at the start (the HTTPS->AMQP bridge) and at the end (AMQP->Comet
like thing), I'm happy.

> - Flexibility in provisioning services

Definitely.

> Garrett

Thanks for your reply,
Laurens