[rabbitmq-discuss] RabbitMQ memory management

Fri Sep 12 21:02:27 BST 2008

Ben,

Exact volumetrics are difficult to obtain. Averages don't mean much,
unfortunately, because when you look at the mean you also have to look at
the standard deviation, and the standard deviation in this case is I believe
- high. This is because the SMS vary widely in number and size, depending on
the content ("Hello" vs "War and Peace ";), the sending client and the type
of SMS (WAP push, plain text, proprietary attachments, etc). My
understanding is that the binary component of this (i.e. the sum total of
all messages if the exacty data as received over TCP/IP to a binary file)
would come to about 1 GB per day.

The million messages per day is in a way from day 1, since there is an
existing system handling this volume and the Erlang system I am writing is
replacing parts of it. However, only a fraction of that volume will come on
stream immediately; there will be a gradual cut-over once we are satisfied
with the functionality and performance, and we will monitor things to make
sure the system is coping before cutting over mroe clients.

At this stage of Real Life In Software Development (tm), I don't know
exactly how long the cutover will take or when it will start. In
business-driven things, I have learned to expect anything at any time and
not expect anything ever, if you get my drift. So it could happen in two
weeks or in 6 months. Sorry I can't be more precise, but with the "luxury"
of an existing system keeping the store going, the pressure to cut over is
driven by a number of factors over which I have no control and into which I
have little visibility. For example, if the company gets a large, rich, new
client tomorrow that MUST have the new functionality provided by the Erlang
system and wants to come on board starting yesterday (the usual scenario),
you can bet the cutover will be pretty soon ;-)

So it's a bit of a time bomb, but will not be the show-stopper I imagined,
because if the VM does crash for any reason, I will make sure it gets
restarted by a watchdog, so service won't be interrupted for long and nobody
besides myself and my client will know any better or even care. I'll also
have to make sure that my code notices that Rabbit died and reconnect when
it comes back up (which the code should be doing anyway). The worst that
will happen is that some in-flight messages that didn't make it to disk may
need to be re-sent.

I am not panicking, stuff happens, so I will wait until the desired
functionality is added and hope that YOU guys soon get a rich, impatient
client that wants this and can afford to pay for it right now. In the
meantime, I will probably just add some sort of disk spillover mechanism and
tell my client to get lots more memory for the production servers :) .

Bottom line: no serious worries. Let's work together on this as and when it
becomes a priority for you guys. I understand what it means to have limited
resources, and I really like your product and the way you support it, so
please don't stop and don't sweat it.

Regards,
Edwin

On Fri, Sep 12, 2008 at 2:30 PM, Ben Hood <0x6e6562 at gmail.com> wrote:

> Edwin,
>
> On Fri, Sep 12, 2008 at 6:36 PM, Edwin Fine
> <rabbitmq-discuss_efine at usa.net> wrote:
> > So... in the absence of the "feature" discussed above, do you have any
> > suggestions as to how I can dig myself out of this hole (without using a
> > different m&q system) ? :)
>
> Quick question: what are your exact volumetrics?
>
> You mentioned 1M SMS/day. Is this from day 1 or does this allow for
> growth? And how big is the average SMS inclusive of all of the
> application layer stuff you want to attach to each one?
>
> Ben
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20080912/9bd28649/attachment.htm 