[rabbitmq-discuss] [SOLVED] Very slow get (dequeue) on node B if shovel on node A is configured with {ack_mode, on_confirm}

Thu Feb 21 17:40:33 GMT 2013

Hi,

Simon, as soon as you said 'fsync' i knew what the problem is. I'm testing
on KVMs with virtual disks - thus very slow fsync. Moving /var/lib/rabbitmq
to tempfs (in memory) on nodeB showed that there is no problem to speak of.
My production machines will have either real disks or LVM LVs exported to
XEN. Which means there should be
no problem.

My target design is a "star", local collectors publish to central node. The
idea is not to reconfigure the central node (NodeB in my test) every time a
new local collector is create or destroyed.

Simon, many thanks!

Boris.

On Thu, Feb 21, 2013 at 5:57 PM, Simon MacMullen <simon at rabbitmq.com> wrote:

> Hi. If the shovel is on node A then {ack_mode, on_publish} is not
> particularly safe - if the network connection goes down then you will lose
> messages that were on the wire.
>
> If the shovel were to be running on node B then {ack_mode, on_publish}
> would be safer, as it would tolerate network failures (but not a crash at
> node B).
>
> on_confirm would still be better. Of course, since you're consuming in
> autoack mode in the php script you can lose messages there anyway...
>
> You didn't say which version of RabbitMQ you were running. The script
> seems perfectly reasonable (I assume that you are not doing anything
> AMQPish in the part elided by "/* do stuff here, count messages, run mps
> stats etc... */").
>
> So it's a bit of a puzzle. If confirms + persistence are so much slower
> than persistence alone, then I wonder if somehow you have a machine that
> fsyncs very slowly, since that's the primary difference in what node B will
> be doing.
>
> Cheers, Simon
>
>
>
> On 21/02/13 15:10, bratner bratner wrote:
>
>> Hi!
>>
>> Sorry for the long delay. I did some more tests and found a way to speed
>> things up.
>> After changing {ack_mode, on_confirm} to {ack_mode, on_publish} things
>> started to work well.
>> Which means that the speed of reading the messages from the queue NodeB
>> was the speed they
>> were published to the queue on NodeA.
>>
>> Before this change (when 'on_confirm' was set) messages where piling up
>> in the queue on NodeB.
>> So if i published 100 messages per sec into a queue on nodeA, the queue
>> on nodeB would receive all of those and the queue on nodeA would be
>> empty (as it should). But the PHP script on nodeB (below) would read
>> something like 3-8 messages per sec and the rest (92-97 messages per
>> sec) would keep piling up in the queue on nodeB.
>> That is until i completely stopped writing new messages to the queue on
>> nodeA. Then this script will read messages like crazy until the queue on
>> nodeB was completely empty.
>>
>> So I left with a couple of questions. What is causing the write and the
>> read to be so uneven with 'on_confirm' given that there no real
>> resources problem? Is this a bug or a feature?
>> How much reliability do I lose if i use 'on_publish' ? Is there a chance
>> messages can just disappear with 'on_publish' ?
>>
>> Here is the outline of my php test reader, very simple:
>>
>> <?php
>> $connection = new AMQPConnection();
>> $connection->setLogin("<login_**nodeb>");
>> $connection->setPassword("<**password_nodeb");
>> $connection->setVhost("vhostB"**);
>> $connection->connect();
>>
>> if (!$connection->isConnected()) {
>>      die('Not connected :('. PHP_EOL);
>> }
>> $channel    = new AMQPChannel($connection);
>> $queue      = new AMQPQueue($channel);
>> $queue->setName('queueB');
>>
>> while(true) {
>>          while($queue->get(AMQP_**AUTOACK)) {
>>             /* do stuff here, count messages, run mps stats etc... */
>>          }
>>          /* sleep some not to eat the whole cpu */
>>          sleep(1);
>> }
>>
>>
>> Thanks,
>> Boris
>>
>> On Wed, Feb 13, 2013 at 5:52 PM, Simon MacMullen <simon at rabbitmq.com
>> <mailto:simon at rabbitmq.com>> wrote:
>>
>>     That sounds very wrong.
>>
>>     Which version of RabbitMQ are you using? Can you post (a cut down
>>     version of) your PHP script somewhere? (I have no familiarity with
>>     the PHP client but I'd like to see what it's doing...)
>>
>>     Cheers, Simon
>>
>>
>>     On 13/02/13 14:33, bratner bratner wrote:
>>
>>         Hi!
>>
>>         My setup includes a rabbitmq-c application that publishes
>>         messages on
>>         node A.
>>         The publishing rate is about 200mps, each message can be up to
>> 10Kb.
>>
>>         Shovel (node A) is configured to move them to node B.
>>         On node B i'm dequeuing the messages with a PHP test script with
>>         $queue->get(AMQP_AUTOACK).
>>         If i keep the publishing rate at 200mps then i can see that the
>>         queue on
>>         node A is empty and the Q on node B is
>>         filling up. The read-rate of my test script is really low.
>>
>>         If i stop the publishing, 3-5 seconds later, my test script starts
>>         reading like crazy until the queue on node B is empty.
>>
>>         Even If I slow down the publishing rate to 5mps , same is
>>         happening ,
>>         messages are piling up on node B until i dial down the pressure.
>>
>>         This problem disappears if I set ack_mode to on_publish or
>>         no_ack. In
>>         this case the reader script reads with the publishing speed.
>>
>>         My configuration :
>>
>>
>>
>>         [
>>         {rabbit, [
>>         {log_levels, [{connection, error}]}
>>         ]},
>>         {rabbitmq_shovel,
>>         [ {shovels, [ {messagemover, [
>>         {sources, [
>>         {broker, "amqp://usera:pass@localhost/_**_vhosta
>>         <amqp://cdrposter:4VkI6MKH@__**localhost/sipout>"},
>>
>>         {declarations, [
>>         {'exchange.declare',[{__**exchange, <<"my-fanout">>},{type,
>>         <<"fanout">>},durable]},
>>         {'queue.declare',[{queue,<<"__**messages">>},durable]},
>>
>>         {'queue.bind',[{exchange, <<"my-fanout">>},{queue,
>> <<"messages">>}]}
>>         ]}
>>         ]},
>>         {destinations, [
>>         {brokers, [ "amqp://userb:pass@nodeB/__**vhostb
>>         <amqp://cdr_manager:Ci2XOb3b@_**_10.200.10.218/cdrs
>>         <http://cdr_manager:Ci2XOb3b@**10.200.10.218/cdrs<http://cdr_manager:Ci2XOb3b@10.200.10.218/cdrs>>>"
>> ]},
>>
>>         {declarations, [
>>         {'exchange.declare',[{__**exchange, <<"my-fanout">>},{type,
>>         <<"fanout">>},durable]},
>>         {'queue.declare',[{queue,<<"__**messages">>},durable]},
>>
>>         {'queue.bind',[{exchange, <<"my-fanout">>},{queue,
>> <<"messages">>}]}
>>         ]}
>>         ]},
>>         {queue, <<"messages">>},
>>         {prefetch_count, 200},
>>         {ack_mode, on_confirm},
>>         {publish_properties, [{delivery_mode, 2}]},
>>         {reconnect_delay, 5}
>>         ]}
>>         ]}
>>         ]}
>>         ].
>>
>>
>>         Thank you,
>>         Boris.
>>
>>
>>         ______________________________**___________________
>>         rabbitmq-discuss mailing list
>>         rabbitmq-discuss at lists.__rabbi**tmq.com <http://rabbitmq.com>
>>         <mailto:rabbitmq-discuss@**lists.rabbitmq.com<rabbitmq-discuss at lists.rabbitmq.com>
>> >
>>         https://lists.rabbitmq.com/__**cgi-bin/mailman/listinfo/__**
>> rabbitmq-discuss<https://lists.rabbitmq.com/__cgi-bin/mailman/listinfo/__rabbitmq-discuss>
>>
>>         <https://lists.rabbitmq.com/**cgi-bin/mailman/listinfo/**
>> rabbitmq-discuss<https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss>
>> >
>>
>>
>>
>>     --
>>     Simon MacMullen
>>     RabbitMQ, VMware
>>
>>
>>
>
> --
> Simon MacMullen
> RabbitMQ, VMware
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20130221/a226789b/attachment.htm>