[rabbitmq-discuss] Testing Shovel on a WAN

Fri Mar 30 15:50:03 BST 2012

Hi 

To investigate this I set a small client in Python (spy.py) which subscribed the same exchange as my application, and 
traces some information from messages properties (code attached).
Before starting my test I run one spy.py on each end.
At the end of test I compare both information, to see if
there is a difference in the message_id sequences, or if messages have been redelivered.

I did a test first with the local broker clustered (as before):
- then I have a different sequence in messages
- but there is no redelivered messages 

Then I suspected that my problem could be in relation with the cluster configuration (why not), 
So I reconfigured the local end with a single node (no more cluster), leaving the remote end clustered.
- then I could not reproduce the difference in message_id sequences

May be I did something wrong when configuring Shovel with my local cluster ?
What I did is : 
- set a two node cluster on local end
- Have the Shovel plug-In configured on both nodes
  Each node has its one Shovel setting, with similar parameters except "sources [broker..]" and "destinations [broker...]", 
  That are specific to the node. 
  On each node I have defined one source broker and one destination broker.

- set a two node cluster on remote end, without Shovel setting.

I am wondering if this is correct with clusters ?

However, everything is working fine, when the WAN network is good enough.

Any Idea ?

Thank's a lot.

Denis

-----Original Message-----
From: Simon MacMullen [mailto:simon at rabbitmq.com] 
Sent: Wednesday, March 28, 2012 5:57 PM
To: MELIQUE Denis (MORPHO)
Cc: rabbitmq-discuss at lists.rabbitmq.com
Subject: Re: [rabbitmq-discuss] Testing Shovel on a WAN

On 28/03/12 14:47, MELIQUE Denis (MORPHO) wrote:
> When I slow the WAN to 1 Mbit/s, I see messages accumulating in the
> shovel queue, which is what as expected.
>
> Then once my throughput go to zero, messages are slowly transferred to
> the distant cluster, and that’s OK as well.
>
> The problem is that messages are not processed in the same order on the
> distant cluster than on the local.
>
> My feeling is that with 1 Mbit/s some messages fail to be transferred
> correctly, and are re-sent later.
>
> ( I have the same problem when I simulate a very high level of packet
> loss (30%), with a 10Mbit/s speed)
>
> Is there such a logic in Shovel ?

That seems unlikely. Packet loss should lead to redelivery at the TCP 
level, which the shovel should be unaware of.

Messages should only get redelivered if the consumer rejects them or the 
TCP connection is interrupted completely. Shovel won't reject messages, 
but maybe the connection is being interrupted and re-established? You 
should be able to see this in the logs. You would also see a 
"redelivered" message rate in mgmt.

> Is there a way to configured Shovel to guaranty the sequence of messages ?

This should already be happening.

> Second:
>
> ---------
>
> When a simulate a WAN outage lasting 2 hours, messages staying in the
> shovel queue have a status,
>
> first set to “unacknowledged”.
>
> Then, after about 900 sec, they all have their status changed to “ready”.
>
> When the network recovers, all the pending messages change back to
> “unacknowledged”, before being sent
>
> to the distant cluster.
>
> It does not seems to be a problem for us, just I’d like to understand
> this behavior, and if it could impact rabbit performances.

unacknowledged -> the message has been consumed by the remote end, but 
not acknowledged.

After 900 seconds presumably the connection is determined to be dead by 
RabbitMQ and then the messages go back into the queue ("ready").

Then when the network recovers, they are sent again and become 
"unacknowledged" again, until the acks come in.

Cheers, Simon

-- 
Simon MacMullen
RabbitMQ, VMware
#
" Ce courriel et les documents qui lui sont joints peuvent contenir des informations confidentielles ou ayant un caractère privé. S'ils ne vous sont pas destinés, nous vous signalons qu'il est strictement interdit de les divulguer, de les reproduire ou d'en utiliser de quelque manière que ce soit le contenu. Si ce message vous a été transmis par erreur, merci d'en informer l'expéditeur et de supprimer immédiatement de votre système informatique ce courriel ainsi que tous les documents qui y sont attachés."
******
" This e-mail and any attached documents may contain confidential or proprietary information. If you are not the intended recipient, you are notified that any dissemination, copying of this e-mail and any attachments thereto or use of their contents by any means whatsoever is strictly prohibited. If you have received this e-mail in error, please advise the sender immediately and delete this e-mail and all attached documents from your computer system."
#
-------------- next part --------------
A non-text attachment was scrubbed...
Name: spy.py
Type: application/octet-stream
Size: 6425 bytes
Desc: spy.py
URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20120330/e040b8e6/attachment.obj>