[rabbitmq-discuss] Unexaplainable behaviour with shovel plugin.

[Jason McIntosh]

On our systems, we've seen consistent 400/sec on some queues. During a
heavy data load roughly 2500/sec per queue (these are short lived usually).
 Usually at that point flow control kicks in as our consumers can't quite
keep up.  We use x-consistent-hashes to get around network latency and
shovel each queue in a hash.  SO publishers publish to a fanout queue with
a random routing key, which is bound to an x-consistent-hash exchange bound
to 8 queues.  Each of the 8 queues is shoveled independently with a 1500
prefetch.  We've not been able to overload this mechanism easily - the
drive IO is typically our limiting factor.  Or the consumers as stated on
the remote side.  And that's because we're doing persistent messages,
publisher confirms, and a whole lot of checks to make sure we don't ever
lose anything.

Jason


On Sat, Mar 1, 2014 at 10:22 AM, Laing, Michael
<michael.laing at nytimes.com>wrote:

> Our volumes are quite variable on the shovels, representing a high overall
> degree of variability in our message volumes.
>
> Just looking over the last 24 hours, shovel volume ranged from 25/sec to
> 2,500/sec on our Oregon core cluster.
>
> Best,
>
> Michael
>
>
> On Fri, Feb 28, 2014 at 1:14 PM, Ben Hood <0x6e6562 at gmail.com> wrote:
>
>> Michael,
>>
>> On Fri, Feb 28, 2014 at 12:45 PM, Laing, Michael
>> <michael.laing at nytimes.com> wrote:
>> > So I turned to shovels for more simplicity and control at the expense of
>> > more difficult configuration.
>>
>> Yes, it is quite a low level tool, but I guess sometimes your
>> requirements are intricate enough to need to reach down to the lower
>> layer.
>>
>> > Some of our core clusters support the 'retail' layer of instances that
>> > gateway to clients (candles?). We are introducing federation into one of
>> > these communication links because we want the propagation of client
>> bindings
>> > from the gateway instance to the core - an excellent feature of
>> federation
>> > and an important refinement for us.
>>
>> Using federation to implement an AMQP gateway seems like a common
>> pattern. One wonders why it didn't go into the AMQP spec ....
>>
>> > Initially I had thought that the 'new' federation replaced the 'old'
>> shovel,
>> > but this is not true - each tool has its place although their
>> capabilities
>> > overlap.
>> >
>> > With easier configuration in 3.3, the lowly shovel may get its due!
>>
>> It's interesting to see that the shovel still lives on, despite it
>> being quite an agricultural component. What sort of message volumes
>> are you guys processing with this, BTW?
>>
>> Thanks for being so detailed about your experiences, it's much
>> appreciated.
>>
>> Cheers,
>>
>> Ben
>> _______________________________________________
>> rabbitmq-discuss mailing list
>> rabbitmq-discuss at lists.rabbitmq.com
>> https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
>>
>
>
> _______________________________________________
> rabbitmq-discuss mailing list
> rabbitmq-discuss at lists.rabbitmq.com
> https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
>
>


-- 
Jason McIntosh
https://github.com/jasonmcintosh/
573-424-7612
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20140301/48dbf3c4/attachment.html>