[rabbitmq-discuss] How to handle extremely large queues

Sun Feb 16 19:57:37 GMT 2014

If Rabbit inaccurately reports the amount of memory it's using, how are we
to provision systems? With some understanding of normal load, however
suboptimal the use case may be, we should have an understanding of memory
requirements... Particularly when rabbit provided a reporting mechanism to
confirm out expected memory usage.

I guess I am confused.

On Sunday, February 16, 2014, Alvaro Videla <videlalvaro at gmail.com> wrote:

> Hi,
>
> Keep in mind that there's small memory footprint per message, even if
> the message has been paged to disk.
>
> Regards,
>
> Alvaro
>
> On Sun, Feb 16, 2014 at 8:28 PM, Greg Poirier <greg.poirier at opower.com<javascript:;>>
> wrote:
> > They are many small messages. Each node in the cluster has 8 gigs but is
> > only using maybe 2. Is memory really the problem?
> >
> >
> > On Sunday, February 16, 2014, Michael Klishin <
> michael.s.klishin at gmail.com <javascript:;>>
> > wrote:
> >>
> >>
> >> 2014-02-16 22:32 GMT+04:00 Greg Poirier <greg.poirier at opower.com<javascript:;>
> >:
> >>>
> >>> In our current configuration, we have a 3-node cluster with 2 disc and
> 1
> >>> ram node with HA mirroring to all nodes in the cluster. In periods of
> high
> >>> utilization of the cluster, we are noticing frequent partitioning. We
> have
> >>> narrowed it down to this particular use case as none of our other
> clusters
> >>> (running on the same physical hardware with the same cluster
> configuration)
> >>> experience this kind of partitioning.
> >>>
> >>> Is there some better way that we can configure RabbitMQ to handle this
> >>> kind of load pattern? I understand this is perhaps not the best way to
> use
> >>> RabbitMQ, but it is unavoidable for the time being. Any suggestions
> would be
> >>> appreciated.
> >>
> >>
> >> Short answer is: give it more RAM.
> >>
> >> Relevant blog posts:
> >>
> >>
> >>
> http://blog.travis-ci.com/2013-08-08-solving-the-puzzle-of-scalable-log-processing/
> >>
> >>
> http://www.rabbitmq.com/blog/2014/01/23/preventing-unbounded-buffers-with-rabbitmq/
> >> --
> >> MK
> >>
> >> http://github.com/michaelklishin
> >> http://twitter.com/michaelklishin
> >
> >
> > _______________________________________________
> > rabbitmq-discuss mailing list
> > rabbitmq-discuss at lists.rabbitmq.com <javascript:;>
> > https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
> >
> _______________________________________________
> rabbitmq-discuss mailing list
> rabbitmq-discuss at lists.rabbitmq.com <javascript:;>
> https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20140216/b5ea0e05/attachment.html>