[rabbitmq-discuss] Scalability?

Fri May 7 21:40:03 BST 2010

Wayne, (and Mathew below please)

If you send me your code for the pub's & sub's, I will test it on one of 
our machines here.  I'm on the road this weekend, but will have access 
to a multi core 64 bit Ubuntu 10.04 with RabbitMQ 1.72, 4gig ram, 
lots-o-disk, etc. to test on Sunday evening.

Also, can you paint the complete picture of what you're doing - based 
upon what I've read so far, you've got a couple of producers, they're 
publishing, and then you've got a couple of subscribers dumping to 
Cassandra.  Is that correct?

When you say 1000 messages below, is that per second?

(I'm easily pushing ~20,000,000 messages a day with RabbitMQ and nada 
problema and that's nothing compared to other peeps out there so I'm 
sure that we can find something here)

Mathew - I'm going to test with the latest rabbitmq server as is on Ubu 
10.04 - if I need to rebuild with this new persistor, can you send me 
some pointers please (I usually build rabbitmq from source anyway, so 
maybe I've already got one laying around some where, but I want to make 
sure)

On 5/7/2010 3:23 PM, Wayne Van Den Handel wrote:
> I tried the high water mark setting and it now prevents too much 
> memory from being used (barely) but as my test continued it eventually 
> had a hard crash for another reason. I was hitting it with 2 producers 
> throttled to 1000 messages each to a separate queue and 5 consumers. 
> There was always 1000 messages each in the 2 queues and after 15 
> minutes one of the producers crashed with an error (from the logs) 
> {amqp_error,not_found,"no queue 'xxx' in vhost '/'", queue.declare'}. 
> I figured this was a timeout or something as the queue did exist and 
> was being emptied by 3 different consumers actively.
>
> 5 seconds later the entire process died with an error (from the logs) 
> "exception on TCP connection ... connection_closed_abruptly" and then 
> "closing TCP connection".
>
> I am new to queuing but frankly I don't know what to do. My Experience 
> with this and other MQ products is less than compelling. Does this 
> take a ton of tweaking to get stable? How do people manage to push 
> 500k messages a day?
>
>
> Wayne
>
>
> Matthew Sackman wrote:
>> Hi Wayne,
>>
>> On Fri, May 07, 2010 at 07:48:54AM -0400, Wayne Van Den Handel wrote:
>>    
>>> I will test the basic consume, thanks for the tip. The machine I am running
>>> is a single socket 4 core with 4 gigs ram but there are other things running
>>> and taking up memory. Is there a gc interval that can be set to get memory
>>> back sooner? As I add messages and remove them I can see the exact memory
>>> taken with the admin tool and it keeps going up. It seems the rate at which
>>> I add to the queue exceeds the ability the clear memory. Basically even
>>> though there is little in the queue at any time the memory use keeps going
>>> up.
>>>      
>>
>> Rabbit assumes it gets the whole machine's memory to itself. If that's
>> not the case then you need to reduce the memory high watermark from 0.4
>> to some much lower figure - say, 0.1.
>> Seehttp://www.rabbitmq.com/extensions.html#memsup
>>
>>    
>>> At this point I am ready to give up until they harden the persister to
>>> go to disk as I assume the persister is the problem.
>>>      
>>
>> Yup. The new persister currently requires you build from source, but it
>> is actively being QA'd and great progress has been made getting it into
>> default. We've gone from about 100 hunks difference a couple of weeks
>> ago to now about 13 - so things are very much moving in this direction.
>>
>> In the mean time, many people are already using the new persister, which
>> is on branch "bug21673" and if you're comfortable compiling from source
>> and using some code which has not yet completed our QA process, then we
>> would encourage you to use it, with the understanding that there may
>> still be bugs in it. Further battle testing is always welcome and should
>> you come across any bugs, any bug reports you make are very gratefully
>> received.
>>
>> Matthew
>>
>>    
>
>
>
> _______________________________________________
> rabbitmq-discuss mailing list
> rabbitmq-discuss at lists.rabbitmq.com
> http://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
>    
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20100507/6d9ef7b1/attachment.htm 