<div dir="ltr">Ben,<br><br>Thanks for clarifying. Disclaimer: what follows is not an attack on Rabbit, or you, or anything like that. It's simply explaining my situation without judgment and asking for suggestions.<br>
<br>I now understand that in the worst-case scenario as discussed, the Rabbit Erlang VM could crash due to out of memory conditions. What I should then do is probably set the rabbit node up with a heartbeat so that in the extremely unlikely event of this happening, at least the system will auto-recover. All persistent messages will be on disk, so nothing crucial will be lost. Presumably, if there are a grillion (that's a metric SI term for "a lot of" ;)) persistent messages in Mnesia for a specific queue, RabbitMQ won't try to load these all into memory and cause the crash to repeat itself. <i>That</i> would be unfortunate.<br>
<br>I think it would be a really solid addition to RabbitMQ to limit the number of messages kept in memory and hence reduce considerably the likelihood of out of memory conditions. It is very difficult for me to know exactly when I will need this feature, because I don't yet know the volumes that could be hitting the system when it goes into full production. It's currently in limited production. However, the sooner the better. In my opinion, and I really like RabbitMQ, so don't get me wrong, the lack of this feature severely limits the usefulness of RabbitMQ in a store and forward scenario. Basically, it means that barring other solution, I would have to store the messages myself in a database, and dequeue them myself. This means I would have to write a mini-queuing system myself, which I was trying to avoid. In particular, I didn't want to write it because Mnesia has a serious limitation in that if your table is a set, it cannot be disk-only, which means... that all the messages in the database are kept in memory!! There are some ways around this, but I just didn't want to have to handle these issues myself.<br>
<br>Look, it's not your fault that I didn't understand that RabbitMQ keeps all the queued messages in memory until they are delivered (although this should be prominently featured in the documentation, if it is not already, but I haven't seen it). But now I do.<br>
<br>The system that I am using RabbitMQ is part of is an SMS message aggregator. It stores and forwards of the order of a million SMS text messages and other kinds of messages per day. I had a choice when designing the system: either write my own database queuing mechanism, or find a messaging and queuing system. I had used WebSphereMQ for very similar systems in the past, so I was averse to reinventing the wheel. When I evaluated RabbitMQ, I saw that it did persistent messaging and made an erroneous assumption that it worked the same way as WebSphereMQ in how it dealt with persistent messages. My testing never showed any different, but I probably never stressed RabbitMQ to the point where it would have shown up.<br>
<br>In fact, I originally had a problem, which I posted on this mailing list, where my consumers were being overwhelmed by the speed at which RabbitMQ was pushing messages to them. This caused the messages to be buffered in the consumer's Erlang queue, which made the consumer's memory usage go through the roof. There was no way to throttle the sender because QoS was not implemented, so I changed my consumers to use a basic.get. This, I thought, pushed the problem back into RabbitMQ's camp, which I thought would solve the problem because I had believed it was all going to disk and did not know it was being shadowed in memory too. So it didn't solve the problem, it simply pushed it to a different part of the system.<br>
<br>I put a great deal of effort into ensuring that the consumers stay running at all times, but the thing that is out of my control is whether they can keep up with the messages that are being put into the system. The reason for this is that the consumers deliver messages to various URLs over http, and the server behind the URL might be offline for any period of time, or delivery might be very slow. This can cause the messages to back up if they are being added at a high rate. Messages are added at a high rate sometimes when a client wants to send a batch of tens of thousands of messages in a store and forward mode.<br>
<br>So... in the absence of the "feature" discussed above, do you have any suggestions as to how I can dig myself out of this hole (without using a different m&q system) ? :)<br><br>Regards,<br>Edwin<br><br>
<br><div class="gmail_quote">On Fri, Sep 12, 2008 at 12:27 PM, Ben Hood <span dir="ltr"><<a href="mailto:0x6e6562@gmail.com">0x6e6562@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
Edwin,<br>
<div class="Ih2E3d"><br>
On Fri, Sep 12, 2008 at 2:31 AM, Edwin Fine<br>
<<a href="mailto:rabbitmq-discuss_efine@usa.net">rabbitmq-discuss_efine@usa.net</a>> wrote:<br>
> I think you misunderstood my message. I was saying RabbitMQ should be able<br>
> to store a non-memory-limited number of persistent messages in the absence<br>
> of a consumer draining the queue. So to be exact, these would be<br>
> non-transactional persistent messages. You replied to me about<br>
> non-persistent messages, which I know are memory-limited. I am confused,<br>
> because I was at one time (when I was doing my due diligence) convinced that<br>
> RabbitMQ had no practical limit (other than any set by Mnesia and hard disk<br>
> space) for storing non-transactional persistent messages. I am simply<br>
> concerned that perhaps I was mistaken and was asking you to elaborate.<br>
<br>
</div>Thanks for the clarification.<br>
<br>
The status quo is that irrespective of whether we are talking about<br>
persistent or non-persistent messages, messages are queued up in<br>
memory until they are drained. The difference between persistent and<br>
non-persistent is that with persistent messages a replica of each<br>
message is written to disk. This disk copy will only be used in a<br>
recovery situation for messages that have not been acknowledged.<br>
<br>
Obviously the limit to which you carry on queuing messages without<br>
draining them depends on the physical resources available to the<br>
Erlang VM and how virtual memory is utilized, which in turn depends on<br>
your OS and production setup.<br>
<br>
So to draw a line in the sand, you could go through and calibrate your<br>
setup wrt to the point at which it cannot allocate any more memory. By<br>
doing so, you've worked out approximately what your absolute bottom<br>
line will be. If, for example, your expected capacity is a small<br>
fraction of this known limit for your particular setup, and that in<br>
practice, you would notice undrained messages long before you hit the<br>
limit, you *might* decide that for all intents and purposes this is a<br>
acceptable and manageable risk.<br>
<br>
Whilst doing your testing, you may not have encountered any problems,<br>
because you may not have pushed it to the limit. All I am saying is<br>
that in doing stress testing for the scalability improvements I am<br>
working on, I have pushed Rabbit to its limits at various stages and<br>
am just informing you of the current theoretical worse case scenario -<br>
ATM queue depth is bounded by memory and there is no overflow<br>
facility.<br>
<br>
However, you may be more risk averse and not want to entertain the<br>
possibility of exhausting your system resources, ever.<br>
<br>
In that case, as you have rightfully pointed out, an obvious<br>
improvement to Rabbit is to overflow to disk when a certain queue<br>
depth is reached.<br>
<br>
This could apply to persistent and non-persistent messages alike.<br>
<br>
I'll take this down as an enhancement suggestion which may factored in<br>
the work that we have planned to make queues pluggable. An alternative<br>
is to incorporate this into the current queue implementation, but<br>
whether we do this would basically depend on what resources we have<br>
available and how acute it is time wise.<br>
<br>
Another related improvement includes implementing message expiry.<br>
<br>
Please let us what your time scales are for when you absolutely need<br>
this in production.<br>
<br>
HTH,<br>
<font color="#888888"><br>
Ben<br>
<br>
</font></blockquote></div><br></div>