[rabbitmq-discuss] Federated exchange slowdown

[Busoli, Simone]

Hi Simon, I can't imagine why as well. The network connection is not that fast but it's a bunch of MBits, way more than is required to transfer those messages around.
The federation queue is indeed showing 100 unacked messages, with way more ready to be delivered. The downstream broker nonetheless doesn't give us any clue about that might be happening, although as I said the bidirectional federated exchange on B2 behaved way worse than B1 and the outgoing federation queue showed lots more ready messages then there were in B1.

-----Original Message-----
From: rabbitmq-discuss-bounces at lists.rabbitmq.com [mailto:rabbitmq-discuss-bounces at lists.rabbitmq.com] On Behalf Of Simon MacMullen
Sent: lunedì 19 marzo 2012 12:26
To: Simone Busoli
Cc: rabbitmq-discuss at lists.rabbitmq.com
Subject: Re: [rabbitmq-discuss] Federated exchange slowdown

Hmm. Well, this *should* be very simple - the federation plugin is a bit tricky in terms of how it gets everything set up, but once the link is up it really just consumes and republishes the messages.

Assuming you have an even slightly fast network link I can't imagine why either federation queue would end up with messages backing up.

So assuming you are not network bound we could start by figuring out what is stopping the messages getting delivered immediately. If the prefetch count is 100, does that mean the queue is showing 100 messages unacked (i.e. the delay is happening because the federated exchange is not consuming fast enough), or fewer (i.e. the broker is just not delivering the messages for some reason)?

Cheers, Simon

On 18/03/12 11:42, Simone Busoli wrote:
> Following up on this issue, the problem has worsened to a great extent.
> Here's a bit more information about the setup.
> Two brokers B1 and B2, running version 2.7.1 and Erlang R14B04 on 
> Windows, about 15000 km away from each other. Both machines running 
> the brokers are physical machines with several cores, lots of RAM, 
> fast disks, and reported no problems in disk access, CPU or memory consumption.
>
> We have two federated durable topic exchanges X1 and X2, the upstream 
> connection to the other broker is configured with an hearbeat of a few 
> seconds and the prefetch count is 100. X1 is bidirectionally federated 
> with max_hops = 1 to prevent messages from going in loops (which 
> nonetheless suffer the additional roundtrip before being discarded, a 
> problem I reported a while ago), X2 unidirectionally federated from B1 
> to B2. Messages are all published on B1, all those published on X1 are 
> persistent, while those published on X2 aren't. Messages are published 
> on X1 with an average rate of 3 msg/sec, and on X2 with an average 
> rate of 10 msg/sec.
>
> What we experienced is that messages published on X1 queue up on the 
> outgoing queue on B1, and even worse on the outgoing queue on B2 
> (although we don't care about them as the would be discarded once 
> delivered thanks to the max_hops setting). Looking at the Web UI it 
> appears that the acknowledges for those messages come in more slowly 
> than messages are published, but I really can't understand why this 
> might be the case, given the very low publish rate. The visible effect 
> is a delay of even several minutes between when a message is published 
> on B1 and received on B2, but again, only for the low-rate messages 
> published on X1 (perhaps even worse for the other direction if the 
> messages were not thrown away by the federation because of the 
> max_hops setting, as the outgoing queue on B2 has several thousands of 
> messages versus the few hundreds of that on B1). In other words, the 
> only difference between the two federated exchanges appears to be the 
> bidirectionality of the first versus the unidirectionality of the 
> second and the fact that those published on the first are persistent.
> The logs of the two brokers contain apparently no relevant 
> information, and the network link between the two has always appeared 
> to be in a good state.

--
Simon MacMullen
RabbitMQ, VMware
_______________________________________________
rabbitmq-discuss mailing list
rabbitmq-discuss at lists.rabbitmq.com
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss

_________________________________________________________________________________________________________________
Questo messaggio è da intendersi esclusivamente ad uso del destinatario e può contenere informazioni che sono di natura privilegiata, confidenziale
o non divulgabile secondo le leggi vigenti. Se il lettore del presente messaggio non è il destinatario designato, o il dipendente/agente responsabile
per la consegna del messaggio al destinatario designato, si informa che ogni disseminazione, distribuzione o copiatura di questa comunicazione è 
strettamente proibita anche ai sensi del decreto legislativo 196/03 . Se avete ricevuto questo messaggio per errore, vi preghiamo di notificarcelo
immediatamente a mezzo e-mail di risposta e successivamente di procedere alla cancellazione di questa e-mail e relativi allegati dal vostro sistema.
_________________________________________________________________________________________________________________
This message is intended only for the use of the addressee and may contain information that is privileged, confidential and exempt from 
disclosure under applicable law. If the reader of this message is not the intended recipient, or the employee or agent responsible for delivering the 
message to the intended recipient, you are hereby notified that any dissemination, distribution or copying of this communication is strictly 
prohibited. If you have received this e-mail in error, please notify us immediately by return e-mail and delete this e-mail and all attachments from 
your system.
_________________________________________________________________________________________________________________