[rabbitmq-discuss] How to handle extremely large queues

Sun Feb 16 20:29:37 GMT 2014

What do you mean by RabbitMQ inaccurately reporting the memory that is using?

On Sun, Feb 16, 2014 at 8:57 PM, Greg Poirier <greg.poirier at opower.com> wrote:
> If Rabbit inaccurately reports the amount of memory it's using, how are we
> to provision systems? With some understanding of normal load, however
> suboptimal the use case may be, we should have an understanding of memory
> requirements... Particularly when rabbit provided a reporting mechanism to
> confirm out expected memory usage.
>
> I guess I am confused.
>
>
> On Sunday, February 16, 2014, Alvaro Videla <videlalvaro at gmail.com> wrote:
>>
>> Hi,
>>
>> Keep in mind that there's small memory footprint per message, even if
>> the message has been paged to disk.
>>
>> Regards,
>>
>> Alvaro
>>
>> On Sun, Feb 16, 2014 at 8:28 PM, Greg Poirier <greg.poirier at opower.com>
>> wrote:
>> > They are many small messages. Each node in the cluster has 8 gigs but is
>> > only using maybe 2. Is memory really the problem?
>> >
>> >
>> > On Sunday, February 16, 2014, Michael Klishin
>> > <michael.s.klishin at gmail.com>
>> > wrote:
>> >>
>> >>
>> >> 2014-02-16 22:32 GMT+04:00 Greg Poirier <greg.poirier at opower.com>:
>> >>>
>> >>> In our current configuration, we have a 3-node cluster with 2 disc and
>> >>> 1
>> >>> ram node with HA mirroring to all nodes in the cluster. In periods of
>> >>> high
>> >>> utilization of the cluster, we are noticing frequent partitioning. We
>> >>> have
>> >>> narrowed it down to this particular use case as none of our other
>> >>> clusters
>> >>> (running on the same physical hardware with the same cluster
>> >>> configuration)
>> >>> experience this kind of partitioning.
>> >>>
>> >>> Is there some better way that we can configure RabbitMQ to handle this
>> >>> kind of load pattern? I understand this is perhaps not the best way to
>> >>> use
>> >>> RabbitMQ, but it is unavoidable for the time being. Any suggestions
>> >>> would be
>> >>> appreciated.
>> >>
>> >>
>> >> Short answer is: give it more RAM.
>> >>
>> >> Relevant blog posts:
>> >>
>> >>
>> >>
>> >> http://blog.travis-ci.com/2013-08-08-solving-the-puzzle-of-scalable-log-processing/
>> >>
>> >>
>> >> http://www.rabbitmq.com/blog/2014/01/23/preventing-unbounded-buffers-with-rabbitmq/
>> >> --
>> >> MK
>> >>
>> >> http://github.com/michaelklishin
>> >> http://twitter.com/michaelklishin
>> >
>> >
>> > _______________________________________________
>> > rabbitmq-discuss mailing list
>> > rabbitmq-discuss at lists.rabbitmq.com
>> > https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
>> >
>> _______________________________________________
>> rabbitmq-discuss mailing list
>> rabbitmq-discuss at lists.rabbitmq.com
>> https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
>
>
> _______________________________________________
> rabbitmq-discuss mailing list
> rabbitmq-discuss at lists.rabbitmq.com
> https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
>