[rabbitmq-discuss] Lower delivery rate than publish rate - why?

Fri Dec 13 21:50:22 GMT 2013

I noticed something else very odd.

Currently, one queue has 43,000 messages backed up in its queue. But when I
look at the exchange (there is only one exchange) I see that the message
rate in exactly matches the message rate out.

With such a huge backlog, why would that be? I would have thought that the
consumers (there are 16 total distributed across 4 systems for that queue
with a prefetch of 100) would run at a much higher steady state.

This exchange also seems to cycle regularly. It appears to run from a low
of around 60/s in and out to 500+ a second in and out.

-- 

*Mike Templeman*
*Head of Development*

T: @talkingfrog1950 <http://twitter.com/missdestructo>
T: @Meshfire <http://twitter.com/meshfire>

On Fri, Dec 13, 2013 at 10:40 AM, Mike Templeman <mike at meshfire.com> wrote:

> Also, from observing the Connections screen on the web UI shows that no
> flow control has been recently turned on for any of the four current
> connections (four app servers).
>
> --
>
> *Mike Templeman *
> *Head of Development*
>
> T: @talkingfrog1950 <http://twitter.com/missdestructo>
> T: @Meshfire <http://twitter.com/meshfire>
>
>
>
> On Fri, Dec 13, 2013 at 10:17 AM, Mike Templeman <mike at meshfire.com>wrote:
>
>> Hi Alvaro
>>
>> I would be more than happy to provide logs. But all they have in them is
>> connection and shutdown information. Nothing more. I have just enabled
>> tracing on the vhost and will send the logs shortly. We encounter this
>> issue when under load every day now.
>>
>> Let me tell you our architecture and deployment:
>>
>> rabbitMQ:
>>
>>    - m1.large ec2 instance. Version: RabbitMQ 3.1.5,  Erlang R14B04
>>    - 23 queues (transaction and direct)
>>    - 3 exchanges used, two fanout and one topic exchange
>>    - topic exchange
>>    - Topic exchange overview is attached.
>>    - 46 total channels.
>>
>>
>> AppServers
>>
>>    - m1.large tomcat servers running grails application
>>    - 2-7 servers at any one time.
>>    - Consume + publish
>>    - On busy queues, each server has 16 consumers with prefetch at 100
>>    - message sizes on busy queues are ~4KB.
>>    - publishing rates on busiest queue ranges from 16/s to >100/s. (We
>>    need to be able to support 1000/s).
>>
>>
>> Each AppServer connects to a sharded mongodb cluster of 3 shards. Our
>> first suspicion was that something in Mongo or AWS was causing the periodic
>> delay but AWS techs looked into our volume use and said we were only use
>> 25% of available bandwidth.
>>
>> At this moment, we have a modest publish rate (~50-60/s) but a backlog of
>> 50,000 messages for the queue "user". You can see a 10 minute snapshot of
>> the queue and see the cycling.
>>
>> I turned on tracing but the results don't seem to becoming into the log.
>> Is there another way to enable reporting of flow control?
>>
>> Mike Templeman
>>
>>
>> --
>>
>> *Mike Templeman*
>> *Head of Development*
>>
>> T: @talkingfrog1950 <http://twitter.com/missdestructo>
>> T: @Meshfire <http://twitter.com/meshfire>
>>
>>
>>
>> On Fri, Dec 13, 2013 at 6:03 AM, Alvaro Videla-2 [via RabbitMQ] <
>> ml-node+s1065348n32063h90 at n5.nabble.com> wrote:
>>
>>> Mike,
>>>
>>> Would you be able to provide information more information to help us
>>> debug the problem?
>>>
>>> Tim (from the rabbitmq team) requested more info in order to try to
>>> find answers for this.
>>>
>>> For example, when consumption drops to zero, are there any logs on the
>>> rabbitmq server that might tell of a flow control mechanism being
>>> activated?
>>>
>>> Regards,
>>>
>>> Alvaro
>>>
>>>
>>> On Fri, Dec 13, 2013 at 2:19 AM, MikeTempleman <[hidden email]<http://user/SendEmail.jtp?type=node&node=32063&i=0>>
>>> wrote:
>>>
>>> > Tyson
>>> >
>>> > Did you ever find an answer to this question? We are encountering
>>> virtually
>>> > the exact same problem.
>>> >
>>> > We have a variable number of servers setup as producers and consumers
>>> and
>>> > see our throughput drop to zero on a periodic basis. This is most
>>> severe
>>> > when there are a few hundred thousand messages on rabbit.
>>> >
>>> > Did you just drop Rabbit? Ours is running on an m1.large instance with
>>> RAID0
>>> > ephemeral drives, so size and performance of the disk subsystem is not
>>> an
>>> > issue (we are still in beta). We have spent untold hours tuning our
>>> sharded
>>> > mongodb subsystem only to find out that it is only being 25% utilized
>>> (at
>>> > least it will be blazing fast if we ever figure this out).
>>> >
>>> >
>>> >
>>> >
>>> >
>>> > --
>>> > View this message in context:
>>> http://rabbitmq.1065348.n5.nabble.com/Lower-delivery-rate-than-publish-rate-why-tp29247p32040.html
>>> > Sent from the RabbitMQ mailing list archive at Nabble.com.
>>> > _______________________________________________
>>> > rabbitmq-discuss mailing list
>>> > [hidden email] <http://user/SendEmail.jtp?type=node&node=32063&i=1>
>>> > https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
>>> _______________________________________________
>>> rabbitmq-discuss mailing list
>>> [hidden email] <http://user/SendEmail.jtp?type=node&node=32063&i=2>
>>> https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
>>>
>>>
>>> ------------------------------
>>>  If you reply to this email, your message will be added to the
>>> discussion below:
>>>
>>> http://rabbitmq.1065348.n5.nabble.com/Lower-delivery-rate-than-publish-rate-why-tp29247p32063.html
>>>  To unsubscribe from Lower delivery rate than publish rate - why?, click
>>> here<http://rabbitmq.1065348.n5.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=29247&code=bWlrZUBtZXNoZmlyZS5jb218MjkyNDd8MTYzNTUyMDM4MA==>
>>> .
>>> NAML<http://rabbitmq.1065348.n5.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>>>
>>
>>
>

--
View this message in context: http://rabbitmq.1065348.n5.nabble.com/Lower-delivery-rate-than-publish-rate-why-tp29247p32088.html
Sent from the RabbitMQ mailing list archive at Nabble.com.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20131213/d7369c36/attachment.html>