[rabbitmq-discuss] How to handle extremely large queues

Sun Feb 16 22:22:47 GMT 2014

Erlang processes also consume memory, atoms consume memory as well,
plus a data structure here and there I would say.

On Sun, Feb 16, 2014 at 10:04 PM, Greg Poirier <greg.poirier at opower.com> wrote:
> So I am using the management interface to verify that the memory utilization
> is as I expect. We do not page messages to disk (set the paging watermark to
> 1.1) so the management plugin should be reporting a number similar to
> message size * number of messages.
>
>
> On Sunday, February 16, 2014, Alvaro Videla <videlalvaro at gmail.com> wrote:
>>
>> What do you mean by RabbitMQ inaccurately reporting the memory that is
>> using?
>>
>> On Sun, Feb 16, 2014 at 8:57 PM, Greg Poirier <greg.poirier at opower.com>
>> wrote:
>> > If Rabbit inaccurately reports the amount of memory it's using, how are
>> > we
>> > to provision systems? With some understanding of normal load, however
>> > suboptimal the use case may be, we should have an understanding of
>> > memory
>> > requirements... Particularly when rabbit provided a reporting mechanism
>> > to
>> > confirm out expected memory usage.
>> >
>> > I guess I am confused.
>> >
>> >
>> > On Sunday, February 16, 2014, Alvaro Videla <videlalvaro at gmail.com>
>> > wrote:
>> >>
>> >> Hi,
>> >>
>> >> Keep in mind that there's small memory footprint per message, even if
>> >> the message has been paged to disk.
>> >>
>> >> Regards,
>> >>
>> >> Alvaro
>> >>
>> >> On Sun, Feb 16, 2014 at 8:28 PM, Greg Poirier <greg.poirier at opower.com>
>> >> wrote:
>> >> > They are many small messages. Each node in the cluster has 8 gigs but
>> >> > is
>> >> > only using maybe 2. Is memory really the problem?
>> >> >
>> >> >
>> >> > On Sunday, February 16, 2014, Michael Klishin
>> >> > <michael.s.klishin at gmail.com>
>> >> > wrote:
>> >> >>
>> >> >>
>> >> >> 2014-02-16 22:32 GMT+04:00 Greg Poirier <greg.poirier at opower.com>:
>> >> >>>
>> >> >>> In our current configuration, we have a 3-node cluster with 2 disc
>> >> >>> and
>> >> >>> 1
>> >> >>> ram node with HA mirroring to all nodes in the cluster. In periods
>> >> >>> of
>> >> >>> high
>> >> >>> utilization of the cluster, we are noticing frequent partitioning.
>> >> >>> We
>> >> >>> have
>> >> >>> narrowed it down to this particular use case as none of our other
>> >> >>> clusters
>> >> >>> (running on the same physical hardware with the same cluster
>> >> >>> configuration)
>> >> >>> experience this kind of partitioning.
>> >> >>>
>> >> >>> Is there some better way that we can configure RabbitMQ to handle
>> >> >>> this
>> >> >>> kind of load pattern? I understand this is perhaps not the best way
>> >> >>> to
>> >> >>> use
>> >> >>> RabbitMQ, but it is unavoidable for the time being. Any suggestions
>> >> >>> would be
>> >> >>> appreciated.
>> >> >>
>> >> >>
>> >> >> Short answer is: give it more RAM.
>> >> >>
>> >> >> Relevant blog posts:
>> >> >>
>> >> >>
>> >> >>
>> >> >>
>> >> >> http://blog.travis-ci.com/2013-08-08-solving-the-puzzle-of-scalable-log-processing/
>> >> >>
>> >> >>
>> >> >>
>> >> >> http://www.rabbitmq.com/blog/2014/01/23/preventing-unbounded-buffers-with-rabbitmq/
>> >> >> --
>> >> >> MK
>> >> >>
>> >> >> http://github.com/michaelklishin
>> >> >> http://twitter.com/michaelklishin
>> >> >
>> >> >
>> >> > _______________________________________________
>> >> > rabbitmq-discuss mailing list
>> >> > rabbitmq-discuss at lists.rabbitmq.com
>> >> > https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
>> >> >
>> >> _______________________________________________
>> >> rabbitmq-discuss mailing list
>> >> rabbitmq-discuss at lists.rabbitmq.com
>> >>
>
>
> _______________________________________________
> rabbitmq-discuss mailing list
> rabbitmq-discuss at lists.rabbitmq.com
> https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
>