[rabbitmq-discuss] [Minimum Air Induction] Introducing Shovel: An AMQP Relay

[Ben Hood]

Valentino,

On Sat, Sep 20, 2008 at 8:18 PM, Valentino Volonghi <dialtone at gmail.com> wrote:
> There is still the duplicates problem but this is a minor issue right now
> IMHO,
> at least it's solvable. (It wouldn't hurt to have 2PC :P).

Sure, but I think these days people are going for 1.5PC instead of 2PC
- it's just a cost-benefit comparison.

>> Rabbit will nuke it. Logging the message to disk is done by the queue
>> process, so if nothing gets routed, nothing gets persisted.
>
> Ok, so I need durable queues and will wait for the overflow on disk
> functionality.

Yes. AMQP is consumer driven, not producer driven.

>> Yep. Don't know what the exact factor is though.
>
> Ouch... I'll try later today or tomorrow to see what this value actually is
> without
> shovel in the middle... I also wonder which header I should use to make a
> message
> persistent, I'll look into this I guess.

I shouldn't imagine that this will cost you an order of magnitude -
the persister is write-optimized after all, though it does have to
synchronously write to disk.

> They look OK in the sense that even persistent messages can reach those
> rates?

I guess the IO bandwidth of your filesystem would be an important
factor in this equation.

>> I think that you need to work out what ingress requirements you have
>> (this will be determined by the capacity of the http server) and what
>> egress you need (so as to avoid stuff queuing up too much).
>>
>> Remember that ATM Rabbit does not implement QoS, so your egress will
>> be bound by the slowest consumer.
>
> What if I have multiple consumers on a queue? Does speed scales almost
> linearly?

No, because the current queue delivery mechanism takes a
robin-robin-ish approach, so the overall throughput is rate limited by
the slowest consumer subscribed to a queue.

This should be addressed when we implement QoS, which would take a
credit based approach that is determined by consumer acks.

If you are interested in the details of what is planned, there is a
design for it here: https://dev.rabbitmq.com/wiki/BasicQosDesign

As indicated before, there is no ETA on this feature.

Ben