[rabbitmq-discuss] Lower delivery rate than publish rate - why?

[MikeTempleman]

Hi Alvaro

I would be more than happy to provide logs. But all they have in them is
connection and shutdown information. Nothing more. I have just enabled
tracing on the vhost and will send the logs shortly. We encounter this
issue when under load every day now.

Let me tell you our architecture and deployment:

rabbitMQ:

   - m1.large ec2 instance. Version: RabbitMQ 3.1.5,  Erlang R14B04
   - 23 queues (transaction and direct)
   - 3 exchanges used, two fanout and one topic exchange
   - topic exchange
   - Topic exchange overview is attached.
   - 46 total channels.


AppServers

   - m1.large tomcat servers running grails application
   - 2-7 servers at any one time.
   - Consume + publish
   - On busy queues, each server has 16 consumers with prefetch at 100
   - message sizes on busy queues are ~4KB.
   - publishing rates on busiest queue ranges from 16/s to >100/s. (We need
   to be able to support 1000/s).


Each AppServer connects to a sharded mongodb cluster of 3 shards. Our first
suspicion was that something in Mongo or AWS was causing the periodic delay
but AWS techs looked into our volume use and said we were only use 25% of
available bandwidth.

At this moment, we have a modest publish rate (~50-60/s) but a backlog of
50,000 messages for the queue "user". You can see a 10 minute snapshot of
the queue and see the cycling.

I turned on tracing but the results don't seem to becoming into the log. Is
there another way to enable reporting of flow control?

Mike Templeman


-- 

*Mike Templeman*
*Head of Development*

T: @talkingfrog1950 <http://twitter.com/missdestructo>
T: @Meshfire <http://twitter.com/meshfire>



On Fri, Dec 13, 2013 at 6:03 AM, Alvaro Videla-2 [via RabbitMQ] <
ml-node+s1065348n32063h90 at n5.nabble.com> wrote:

> Mike,
>
> Would you be able to provide information more information to help us
> debug the problem?
>
> Tim (from the rabbitmq team) requested more info in order to try to
> find answers for this.
>
> For example, when consumption drops to zero, are there any logs on the
> rabbitmq server that might tell of a flow control mechanism being
> activated?
>
> Regards,
>
> Alvaro
>
>
> On Fri, Dec 13, 2013 at 2:19 AM, MikeTempleman <[hidden email]<http://user/SendEmail.jtp?type=node&node=32063&i=0>>
> wrote:
>
> > Tyson
> >
> > Did you ever find an answer to this question? We are encountering
> virtually
> > the exact same problem.
> >
> > We have a variable number of servers setup as producers and consumers
> and
> > see our throughput drop to zero on a periodic basis. This is most severe
> > when there are a few hundred thousand messages on rabbit.
> >
> > Did you just drop Rabbit? Ours is running on an m1.large instance with
> RAID0
> > ephemeral drives, so size and performance of the disk subsystem is not
> an
> > issue (we are still in beta). We have spent untold hours tuning our
> sharded
> > mongodb subsystem only to find out that it is only being 25% utilized
> (at
> > least it will be blazing fast if we ever figure this out).
> >
> >
> >
> >
> >
> > --
> > View this message in context:
> http://rabbitmq.1065348.n5.nabble.com/Lower-delivery-rate-than-publish-rate-why-tp29247p32040.html
> > Sent from the RabbitMQ mailing list archive at Nabble.com.
> > _______________________________________________
> > rabbitmq-discuss mailing list
> > [hidden email] <http://user/SendEmail.jtp?type=node&node=32063&i=1>
> > https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
> _______________________________________________
> rabbitmq-discuss mailing list
> [hidden email] <http://user/SendEmail.jtp?type=node&node=32063&i=2>
> https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
>
>
> ------------------------------
>  If you reply to this email, your message will be added to the discussion
> below:
>
> http://rabbitmq.1065348.n5.nabble.com/Lower-delivery-rate-than-publish-rate-why-tp29247p32063.html
>  To unsubscribe from Lower delivery rate than publish rate - why?, click
> here<http://rabbitmq.1065348.n5.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=29247&code=bWlrZUBtZXNoZmlyZS5jb218MjkyNDd8MTYzNTUyMDM4MA==>
> .
> NAML<http://rabbitmq.1065348.n5.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>


ploverExchange.tiff (39K) <http://rabbitmq.1065348.n5.nabble.com/attachment/32085/0/ploverExchange.tiff>
userqueue.tiff (94K) <http://rabbitmq.1065348.n5.nabble.com/attachment/32085/1/userqueue.tiff>




--
View this message in context: http://rabbitmq.1065348.n5.nabble.com/Lower-delivery-rate-than-publish-rate-why-tp29247p32085.html
Sent from the RabbitMQ mailing list archive at Nabble.com.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20131213/eed16b04/attachment.html>