[rabbitmq-discuss] server crashes with very fast consumers

[alex chen]

Hi Matthew,

Thanks for investigating this problem! I tried rabbitmq-server from hg tip, but 
it still crashed when the message consuming rate reached 300 MB/sec.  When you 
ran the test, did you load at least 60 GB messages to 1000 queues?  each message 
is 20 KB in size.

My machine is RAID-10 with 4 SSD disks:  
480GB (1%) RAID-10/2 == 4 x 240GB SSD SATA/300
Maybe your test machine has only 1 SSD disk so the consuming rate could not 
reach 300 MB/sec?

Thanks.

-alex



________________________________
From: Matthew Sackman <matthew at rabbitmq.com>
To: rabbitmq-discuss at lists.rabbitmq.com
Sent: Wed, March 16, 2011 3:49:57 AM
Subject: Re: [rabbitmq-discuss] server crashes with very fast consumers

Hi Alex,

On Thu, Mar 10, 2011 at 09:48:08PM -0800, alex chen wrote:
> I am testing rabbit server with SSD.  The broker process would crash when many 

> consumers start to consume backed up messages.  Because SSD if very fast, the 
> consuming rate reached more than 200 MB/sec.  After several minutes, the broker 
>
> process would use all cpu resources (800% on 8 cores) and crash.

I test all the time on an Intel G2 drive. I've never seen it cause a
crash, even in scenarios such as yours. Furthermore, I find it somewhat
unlikely that if the server process did die, it didn't spit out any sort
of error on its way out. There must be /something/ somewhere that
suggests a bit more as to what went wrong, whether it's in syslog, or
dmesg, or rabbit's various logs.

I did however yesterday find and fix a bug where inexplicably a function
that normally takes about 0.5s ended up in some very strange cases
taking > 80s. After a lot of head scratching and debugging, I never
understood why this variance existed. However, I rewrote the function
and it's now much faster all the time. This particular bug has been
through QA and is now on default and should be in the next release, thus
if you want to try compiling from default and repeating your test, we'd
be interested in knowing whether you see any change.

Matthew
_______________________________________________
rabbitmq-discuss mailing list
rabbitmq-discuss at lists.rabbitmq.com
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss



      
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20110316/2b4de9ad/attachment.htm>