[rabbitmq-discuss] Lower delivery rate than publish rate - why?
mike at meshfire.com
Fri Dec 13 21:50:22 GMT 2013
I noticed something else very odd.
Currently, one queue has 43,000 messages backed up in its queue. But when I
look at the exchange (there is only one exchange) I see that the message
rate in exactly matches the message rate out.
With such a huge backlog, why would that be? I would have thought that the
consumers (there are 16 total distributed across 4 systems for that queue
with a prefetch of 100) would run at a much higher steady state.
This exchange also seems to cycle regularly. It appears to run from a low
of around 60/s in and out to 500+ a second in and out.
*Head of Development*
T: @talkingfrog1950 <http://twitter.com/missdestructo>
T: @Meshfire <http://twitter.com/meshfire>
On Fri, Dec 13, 2013 at 10:40 AM, Mike Templeman <mike at meshfire.com> wrote:
> Also, from observing the Connections screen on the web UI shows that no
> flow control has been recently turned on for any of the four current
> connections (four app servers).
> *Mike Templeman *
> *Head of Development*
> T: @talkingfrog1950 <http://twitter.com/missdestructo>
> T: @Meshfire <http://twitter.com/meshfire>
> On Fri, Dec 13, 2013 at 10:17 AM, Mike Templeman <mike at meshfire.com>wrote:
>> Hi Alvaro
>> I would be more than happy to provide logs. But all they have in them is
>> connection and shutdown information. Nothing more. I have just enabled
>> tracing on the vhost and will send the logs shortly. We encounter this
>> issue when under load every day now.
>> Let me tell you our architecture and deployment:
>> - m1.large ec2 instance. Version: RabbitMQ 3.1.5, Erlang R14B04
>> - 23 queues (transaction and direct)
>> - 3 exchanges used, two fanout and one topic exchange
>> - topic exchange
>> - Topic exchange overview is attached.
>> - 46 total channels.
>> - m1.large tomcat servers running grails application
>> - 2-7 servers at any one time.
>> - Consume + publish
>> - On busy queues, each server has 16 consumers with prefetch at 100
>> - message sizes on busy queues are ~4KB.
>> - publishing rates on busiest queue ranges from 16/s to >100/s. (We
>> need to be able to support 1000/s).
>> Each AppServer connects to a sharded mongodb cluster of 3 shards. Our
>> first suspicion was that something in Mongo or AWS was causing the periodic
>> delay but AWS techs looked into our volume use and said we were only use
>> 25% of available bandwidth.
>> At this moment, we have a modest publish rate (~50-60/s) but a backlog of
>> 50,000 messages for the queue "user". You can see a 10 minute snapshot of
>> the queue and see the cycling.
>> I turned on tracing but the results don't seem to becoming into the log.
>> Is there another way to enable reporting of flow control?
>> Mike Templeman
>> *Mike Templeman*
>> *Head of Development*
>> T: @talkingfrog1950 <http://twitter.com/missdestructo>
>> T: @Meshfire <http://twitter.com/meshfire>
>> On Fri, Dec 13, 2013 at 6:03 AM, Alvaro Videla-2 [via RabbitMQ] <
>> ml-node+s1065348n32063h90 at n5.nabble.com> wrote:
>>> Would you be able to provide information more information to help us
>>> debug the problem?
>>> Tim (from the rabbitmq team) requested more info in order to try to
>>> find answers for this.
>>> For example, when consumption drops to zero, are there any logs on the
>>> rabbitmq server that might tell of a flow control mechanism being
>>> On Fri, Dec 13, 2013 at 2:19 AM, MikeTempleman <[hidden email]<http://user/SendEmail.jtp?type=node&node=32063&i=0>>
>>> > Tyson
>>> > Did you ever find an answer to this question? We are encountering
>>> > the exact same problem.
>>> > We have a variable number of servers setup as producers and consumers
>>> > see our throughput drop to zero on a periodic basis. This is most
>>> > when there are a few hundred thousand messages on rabbit.
>>> > Did you just drop Rabbit? Ours is running on an m1.large instance with
>>> > ephemeral drives, so size and performance of the disk subsystem is not
>>> > issue (we are still in beta). We have spent untold hours tuning our
>>> > mongodb subsystem only to find out that it is only being 25% utilized
>>> > least it will be blazing fast if we ever figure this out).
>>> > --
>>> > View this message in context:
>>> > Sent from the RabbitMQ mailing list archive at Nabble.com.
>>> > _______________________________________________
>>> > rabbitmq-discuss mailing list
>>> > [hidden email] <http://user/SendEmail.jtp?type=node&node=32063&i=1>
>>> > https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
>>> rabbitmq-discuss mailing list
>>> [hidden email] <http://user/SendEmail.jtp?type=node&node=32063&i=2>
>>> If you reply to this email, your message will be added to the
>>> discussion below:
>>> To unsubscribe from Lower delivery rate than publish rate - why?, click
View this message in context: http://rabbitmq.1065348.n5.nabble.com/Lower-delivery-rate-than-publish-rate-why-tp29247p32088.html
Sent from the RabbitMQ mailing list archive at Nabble.com.
-------------- next part --------------
An HTML attachment was scrubbed...
More information about the rabbitmq-discuss