<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
<html>
<head>
<meta content="text/html; charset=ISO-8859-1"
http-equiv="Content-Type">
</head>
<body bgcolor="#ffffff" text="#000000">
<font face="Calibri">Wayne, (and Mathew below please)<br>
<br>
If you send me your code for the pub's & sub's, I will test it on
one of our machines here. I'm on the road this weekend, but will have
access to a multi core 64 bit Ubuntu 10.04 with RabbitMQ 1.72, 4gig
ram, lots-o-disk, etc. to test on Sunday evening. <br>
<br>
Also, can you paint the complete picture of what you're doing - based
upon what I've read so far, you've got a couple of producers, they're
publishing, and then you've got a couple of subscribers dumping to
Cassandra. Is that correct?<br>
<br>
When you say 1000 messages below, is that per second? <br>
<br>
(I'm easily pushing ~20,000,000 messages a day with RabbitMQ and nada
problema and that's nothing compared to other peeps out there so I'm
sure that we can find something here)<br>
<br>
Mathew - I'm going to test with the latest rabbitmq server as is on Ubu
10.04 - if I need to rebuild with this new persistor, can you send me
some pointers please (I usually build rabbitmq from source anyway, so
maybe I've already got one laying around some where, but I want to make
sure)<br>
</font><br>
On 5/7/2010 3:23 PM, Wayne Van Den Handel wrote:
<blockquote cite="mid:4BE476C3.8040707@dataraker.com" type="cite">
<meta content="text/html; charset=ISO-8859-1"
http-equiv="Content-Type">
I tried the high water mark setting and it now prevents too much memory
from being used (barely) but as my test continued it eventually had a
hard crash for another reason. I was hitting it with 2 producers
throttled to 1000 messages each to a separate queue and 5 consumers.
There was always 1000 messages each in the 2 queues and after 15
minutes one of the producers crashed with an error (from the logs)
{amqp_error,not_found,"no queue 'xxx' in vhost '/'", queue.declare'}. I
figured this was a timeout or something as the queue did exist and was
being emptied by 3 different consumers actively.<br>
<br>
5 seconds later the entire process died with an error (from the logs)
"exception on TCP connection ... connection_closed_abruptly" and then
"closing TCP connection".<br>
<br>
I am new to queuing but frankly I don't know what to do. My Experience
with this and other MQ products is less than compelling. Does this take
a ton of tweaking to get stable? How do people manage to push 500k
messages a day? <br>
<br>
<br>
Wayne<br>
<br>
<br>
Matthew Sackman wrote:
<blockquote cite="mid:20100507120026.GJ2739@mrnibble.lshift.net"
type="cite">
<pre wrap="">Hi Wayne,
On Fri, May 07, 2010 at 07:48:54AM -0400, Wayne Van Den Handel wrote:
</pre>
<blockquote type="cite">
<pre wrap="">I will test the basic consume, thanks for the tip. The machine I am running
is a single socket 4 core with 4 gigs ram but there are other things running
and taking up memory. Is there a gc interval that can be set to get memory
back sooner? As I add messages and remove them I can see the exact memory
taken with the admin tool and it keeps going up. It seems the rate at which
I add to the queue exceeds the ability the clear memory. Basically even
though there is little in the queue at any time the memory use keeps going
up.
</pre>
</blockquote>
<pre wrap=""><!---->
Rabbit assumes it gets the whole machine's memory to itself. If that's
not the case then you need to reduce the memory high watermark from 0.4
to some much lower figure - say, 0.1.
See <a moz-do-not-send="true" class="moz-txt-link-freetext"
href="http://www.rabbitmq.com/extensions.html#memsup">http://www.rabbitmq.com/extensions.html#memsup</a>
</pre>
<blockquote type="cite">
<pre wrap="">At this point I am ready to give up until they harden the persister to
go to disk as I assume the persister is the problem.
</pre>
</blockquote>
<pre wrap=""><!---->
Yup. The new persister currently requires you build from source, but it
is actively being QA'd and great progress has been made getting it into
default. We've gone from about 100 hunks difference a couple of weeks
ago to now about 13 - so things are very much moving in this direction.
In the mean time, many people are already using the new persister, which
is on branch "bug21673" and if you're comfortable compiling from source
and using some code which has not yet completed our QA process, then we
would encourage you to use it, with the understanding that there may
still be bugs in it. Further battle testing is always welcome and should
you come across any bugs, any bug reports you make are very gratefully
received.
Matthew
</pre>
</blockquote>
<br>
<br>
<pre wrap="">
<fieldset class="mimeAttachmentHeader"></fieldset>
_______________________________________________
rabbitmq-discuss mailing list
<a class="moz-txt-link-abbreviated" href="mailto:rabbitmq-discuss@lists.rabbitmq.com">rabbitmq-discuss@lists.rabbitmq.com</a>
<a class="moz-txt-link-freetext" href="http://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss">http://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss</a>
</pre>
</blockquote>
</body>
</html>