[rabbitmq-discuss] Lower delivery rate than publish rate - why?

[MikeTempleman]

Also, from observing the Connections screen on the web UI shows that no
flow control has been recently turned on for any of the four current
connections (four app servers).

-- 

*Mike Templeman*
*Head of Development*

T: @talkingfrog1950 <http://twitter.com/missdestructo>
T: @Meshfire <http://twitter.com/meshfire>



On Fri, Dec 13, 2013 at 10:17 AM, Mike Templeman <mike at meshfire.com> wrote:

> Hi Alvaro
>
> I would be more than happy to provide logs. But all they have in them is
> connection and shutdown information. Nothing more. I have just enabled
> tracing on the vhost and will send the logs shortly. We encounter this
> issue when under load every day now.
>
> Let me tell you our architecture and deployment:
>
> rabbitMQ:
>
>    - m1.large ec2 instance. Version: RabbitMQ 3.1.5,  Erlang R14B04
>    - 23 queues (transaction and direct)
>    - 3 exchanges used, two fanout and one topic exchange
>    - topic exchange
>    - Topic exchange overview is attached.
>    - 46 total channels.
>
>
> AppServers
>
>    - m1.large tomcat servers running grails application
>    - 2-7 servers at any one time.
>    - Consume + publish
>    - On busy queues, each server has 16 consumers with prefetch at 100
>    - message sizes on busy queues are ~4KB.
>    - publishing rates on busiest queue ranges from 16/s to >100/s. (We
>    need to be able to support 1000/s).
>
>
> Each AppServer connects to a sharded mongodb cluster of 3 shards. Our
> first suspicion was that something in Mongo or AWS was causing the periodic
> delay but AWS techs looked into our volume use and said we were only use
> 25% of available bandwidth.
>
> At this moment, we have a modest publish rate (~50-60/s) but a backlog of
> 50,000 messages for the queue "user". You can see a 10 minute snapshot of
> the queue and see the cycling.
>
> I turned on tracing but the results don't seem to becoming into the log.
> Is there another way to enable reporting of flow control?
>
> Mike Templeman
>
>
> --
>
> *Mike Templeman*
> *Head of Development*
>
> T: @talkingfrog1950 <http://twitter.com/missdestructo>
> T: @Meshfire <http://twitter.com/meshfire>
>
>
>
> On Fri, Dec 13, 2013 at 6:03 AM, Alvaro Videla-2 [via RabbitMQ] <
> ml-node+s1065348n32063h90 at n5.nabble.com> wrote:
>
>> Mike,
>>
>> Would you be able to provide information more information to help us
>> debug the problem?
>>
>> Tim (from the rabbitmq team) requested more info in order to try to
>> find answers for this.
>>
>> For example, when consumption drops to zero, are there any logs on the
>> rabbitmq server that might tell of a flow control mechanism being
>> activated?
>>
>> Regards,
>>
>> Alvaro
>>
>>
>> On Fri, Dec 13, 2013 at 2:19 AM, MikeTempleman <[hidden email]<http://user/SendEmail.jtp?type=node&node=32063&i=0>>
>> wrote:
>>
>> > Tyson
>> >
>> > Did you ever find an answer to this question? We are encountering
>> virtually
>> > the exact same problem.
>> >
>> > We have a variable number of servers setup as producers and consumers
>> and
>> > see our throughput drop to zero on a periodic basis. This is most
>> severe
>> > when there are a few hundred thousand messages on rabbit.
>> >
>> > Did you just drop Rabbit? Ours is running on an m1.large instance with
>> RAID0
>> > ephemeral drives, so size and performance of the disk subsystem is not
>> an
>> > issue (we are still in beta). We have spent untold hours tuning our
>> sharded
>> > mongodb subsystem only to find out that it is only being 25% utilized
>> (at
>> > least it will be blazing fast if we ever figure this out).
>> >
>> >
>> >
>> >
>> >
>> > --
>> > View this message in context:
>> http://rabbitmq.1065348.n5.nabble.com/Lower-delivery-rate-than-publish-rate-why-tp29247p32040.html
>> > Sent from the RabbitMQ mailing list archive at Nabble.com.
>> > _______________________________________________
>> > rabbitmq-discuss mailing list
>> > [hidden email] <http://user/SendEmail.jtp?type=node&node=32063&i=1>
>> > https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
>> _______________________________________________
>> rabbitmq-discuss mailing list
>> [hidden email] <http://user/SendEmail.jtp?type=node&node=32063&i=2>
>> https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
>>
>>
>> ------------------------------
>>  If you reply to this email, your message will be added to the
>> discussion below:
>>
>> http://rabbitmq.1065348.n5.nabble.com/Lower-delivery-rate-than-publish-rate-why-tp29247p32063.html
>>  To unsubscribe from Lower delivery rate than publish rate - why?, click
>> here<http://rabbitmq.1065348.n5.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=29247&code=bWlrZUBtZXNoZmlyZS5jb218MjkyNDd8MTYzNTUyMDM4MA==>
>> .
>> NAML<http://rabbitmq.1065348.n5.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>>
>
>




--
View this message in context: http://rabbitmq.1065348.n5.nabble.com/Lower-delivery-rate-than-publish-rate-why-tp29247p32086.html
Sent from the RabbitMQ mailing list archive at Nabble.com.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20131213/348b51f4/attachment.html>